When Worse is Better: Navigating the Compression Generation Trade-off In Visual Tokenization
Authors: Vivek Ramanujan
Kushal Tirumala
Armen Aghajanyan
Luke Zettlemoyer
Ali Farhadi
Venue: NeurIPS 2025 spotlight ()
Keywords: Representation Learning
Scaling Laws
Image Generation
Primary area: deep_learning
TL;DR: We study how different changes to the auto-encoder rate-distortion trade-off affect discrete auto-regressive generation performance through scaling laws.

Abstract: Current image generation methods are based on a two-stage training approach. In stage 1, an auto-encoder is trained to compress an image into a latent space; in stage 2, a generative model is trained to learn a distribution over that latent space. This reveals a fundamental trade-off, do we compress more aggressively to make the latent distribution easier for the stage 2 model to learn even if it makes reconstruction worse? We study this problem in the context of discrete, auto-regressive image generation. Through the lens of scaling laws, we show that smaller stage 2 models can benefit from more compressed stage 1 latents even if reconstruction performance worsens, demonstrating that generation modeling capacity plays a role in this trade-off. Diving deeper, we rigorously study the connection between compute scaling and the stage 1 rate-distortion trade-off. Next, we introduce Causally Regularized Tokenization (CRT), which uses knowledge of the stage 2 generation modeling procedure to embed useful inductive biases in stage 1 latents. This regularization improves stage 2 generation performance better by making the tokens easier to model without affecting the stage 1 compression rate and marginally affecting distortion: we are able to improve compute efficiency 2-3$\times$ over baseline. Finally, we use CRT with further optimizations to the visual tokenizer setup to result in a generative pipeline that matches LlamaGen-3B generation performance (2.18 FID) with half the tokens per image (256 vs. 576) and a fourth the total model parameters (775M vs. 3.1B) while using the same architecture and inference procedure.

When Worse is Better: Navigating the CompresiionGGnneraioon Trade-off for Visual Tokenization
Vivek Ramanujan #
Kushal Tirumala
Armen Aghajanyan?
Luke Zettlemoyer
Ali Farhadi
University of Washington
Meta FAIR
Work done while at Meta FAIR I ramanvêcs. washington. edu
Abstract
Current image generation methods are based on two-stage training approach. In stage 1, an auto-encoder is trained compress an image into a latent space: in stage 2, a generative model is trained to learn a distribution over that latent space. This reveals a fundamental trade-off, make the latent distribution easier for the stage 2 model to learn even if it makes reconstruction worse? We study this problem in the context discrete, auto- regressive image generation. Through the lens of scaling laws, we show that smaller stage 2 models can benefit from more compressed stage latents even if reconstruction performance worsens, demonstrating that generation modeling capacity plays a role in this trade-off. Diving deeper, we rigorously study the con- nection between compute scaling and the stage rate-distortion trade-off. Next, we introduce Causally Regularized Tokenization (CRT), which uses knowledge of the stage generation modeling procedure embed useful inductive biases in stage 1 latents. This regularization improves stage 2 generation performance better by making the tokens easier to model without affecting the stage compression rate and marginally affecting distortion:
we compress more aggressively
we are able to improve compute effi- ciency 2-3x over baseline. Finally, we use CRT with further optimizations the visual tokenizer setup to result in generative pipeline that matches LlamaGen-3B generation performance (2.18 FID) with half the tokens per image (256 vs. 576) and fourth the total model parameters (775M vs. 3.1B) while using the same architecture and inference procedure.
Introduction
Modern image generation methods use two-stage approach to training. In the first stage, model such as a VQGAN B3
46 is trained compress images to a latent representation. In the second stage, a generative model is trained these latent representations. This setup implies deep interaction between the stage and stage models. Since the stage model does not have to reconstruct the image perfectly, it can compress the image more aggressively to achieve simpler latent distribution for the stage 2 model to learn. However, we ultimately care about the quality of the generated images, which depends both the stage I model's 's ability to reconstruct and the stage 2 model's ability to learn the latent distribution. On one extreme, if the stage model compresses to constant, the resulting latent distribution is casy to model but can only generate one image. At the other extreme, the stage model does not compress at all and we get no benefit from latent distribution modeling.
63
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Stage Compression, Vector-Quantized Tokenization
Stage 2: Auto-regressive Generation
Gradient Flow
Gneil
Next-token Ez loss
Causal Transformer
Figure 1: Overview of our method. We introduce a small 2-layer causal transformer 41 51 trained optimize l2 next-token prediction the pre-quantized latents of the auto-encoder. This loss propagated through the encoder, producing tokens with a causal transformer inductive bias. Thus, call our method Causally Regularized Tokenization (CRT).
Figure 2: Qualitative examples from our best model (2.18 gFID). We show our model's ability to generate high-quality, diverse images on par with LlamaGen-3B [49] with only 775M parameters and 550% of the tokens per image. For more non-curated examples Figure and Appendix G
We view this as trade-off between rate, distortion, and distribution modeling, similar to the rate- iettortion-usfulnee trade-off discussed in (52). The important interacting variables for generation performance are: (1) The reconstruction performance of the stage 1 auto-encoder (its "distortion (2) The degree of compression of the stage auto-encoder (its "rate") and (3) The smallest achiev- able loss of the stage 2 model in learning distribution over this compressed representation (the "irreducible loss" of distribution modeling).
We study these factors using a vector-quantized GAN (VQGAN) stage model and an auto- regressive stage model trained on VQ tokens. We explore key interventions which affect the stage 1 rate (bits per image pixel) and examine how they affect stage 2 generation performance as function of model scale. Through the lens of compute scaling laws, we show that compression is not all you need - while small models benefit from more compressed representations, and this relationship also depends on stage training compute. Given this, we ask: can we design an optimal stage I tokenizer given that our stage 2 model is ruto-reggessive? Through simple model- causal loss, we learn a causal inductive bias on our stage 1 tokenizer. Intuitively, our loss attempts to make token i as predictable possible given tokens () through i 1. We call this training recipe Causally Regularized Tokenization (CRT). This regularization trades off reconstruction for distri- bution modeling, improving generation performance across ImageNet [47] and LSUN [61]. With CRT, we also show significant improvements in generation performance scaling laws (2-2.5x faster training). We demonstrate our improvements through comprehensive experiments across five orders of magnitude of training compute and two orders of magnitude of model scale.
Summary of contributions.
We study the complex trade-off between compression and generation. Our analysis shows that the ideal amount of image compression varies with generation model capacity.
We provide a principled framework for analyzing this trade-off through the lens scaling laws, showing consistent patterns across multiple orders of magnitude in computational budget.
We introduce method for training stage tokenizer with a causal inductive bias. This improves inference and training compute scaling of our stage 2 models, without any other interventions, lead-

---

ing to our key result: by making tokens easier to model, we improve compute efficiency 2-3x over baseline. We also match LlamaGen-3B 44), a prior SOTA auto-regressive discrete generation model (2.18 FID) with half the tokens per image (256 vs 576) and fourth the total model parameters (775M vs. 3.IB).
2 Related Work
For an extensive rundown of visual tokenization methods, refer the reader to Section |
Trade-off's between stage 1 and stage 2 performance. We are not the first to notice a discon- nect between stage and stage 2 performance. [52] extensively studies how different regularization methods affect the usefulness of learned VAE representations for downstream representations, intro- trade-off. Notably, they highlight that stage I loss (e.g. VAE
ducing the loss) is not representative downstream task usefulness. [53] also shows in Figure 4 demonstrates that increasing B in their 3-VAE (23] architecture worsens reconstruction but improves genera- tion performance. Recent advances in lookup-free quantization [37 64 achieve better codebook utilization, improving stage 1 performance. Both works observe that scaling up codebook size im- proves stage 1 performance but degrades stage performance (see Figure 3 in [37] and Figure I in (44). Outside visual tokenization, previous work in text tokenization observes that optimizing text tokenizer compression (i.e. stage performance) leads to worse perplexity [33] and downstream accuracy [48). JetFormer (54) an end-to-end training procedure for generation, using an auto- regressive (AR) model in conjunction with a flow-based model to perform generation and compres- sion at the same time. They balance this trade-off by using the flow-based model for compression, which by its invertible nature, does not allow for information collapse.
ID-tokenizers for Generative Visual Models. Many recent works have identified the issue that existing visual tokenizers are not built for auto-regressive generation and seek to rectify this issue
through the use of so-called "ID" tokenizers, which focus on global over local semantics. [17
constructs a discrete tokenizer that captures high-level semantics by using Stable Diffusion [46] as
stage 2 model. VAR and VQVAE-2 [50 43) modifies the tokenizer produce multi-scale tokens
for easier auto-regressive modeling. [66 and (56] use transformer register-basee 7 approach
learn global tokens with perceptual losses. SEED [6] constructs ID tokenizer by using Stable
Diffusion [46] as powerful decoder on quantized tokens. More recently, methods [2] 60, 38 extend
this strategy to create progressive tokens, which can be used flexibly refine generated images for sampling efficiency. In contrast to these works, we do not modify the common VQGAN tokenizer architecture and focus on constructing causal regularization that enforces compatibility with stage 2 auto-regressive model. This makes our work generally applicable, as variants of this architecture are common across modern generation systems. We further rigorously study scaling properties with respect to this regularization and other common interventions on tokenizer compression rate, which is not found in existing literature.
Experiments & Methodology
Structure of our study. Our goal to understand the variables involved tokenizer construction affecting generation performance at different model scales. First, we fix stage 1 tokenizer, and study the connection between generation performance (gFID on ImageNet) and compute scaling. We study how changing the stage I "rate" (bits per output pixel) affect stages 2 model performance at various scales. Then we examine a new angle on the rate- rate-distenn distortini which affects compute-optimal generation performance scaling: causally regularized tokenization (CRT).
3.1 VQGAN and Auto-regressive Image Generation
We summarize the standard auto-regressivg image generation procedure
63
Stage 1 (Tokenization). In this stage, the goal is to map an image x E #> to set of N discrete tokens X, E C, where is a codebook such that C C Rd. We also learn a decoder which can reconstruct the original image from a set of tokens. We use the VQGAN architecture [33, which is a ResNet-based architecture that is a mix of convolutions and self-attention layers both the encoder and decoder. During training, the encoder maps the input image x to continuous latents
3

---

X₁. Then, we use codebook look-up to determine the nearest embedding X, E C. Finally, these are passed to the decoder to get reconstruction.
Stage (Generation). In this stage, the goal is to learn a prior over the latent tokens learned in the previous stage. For each image in our train set, we use the encoder learned in stage to map it to set of discrete tokens. We fix an order with which to decode these tokens per image (raster-scan works best empirically (3), and treat these as sequences. Finally,
train transformer with causal attention to learn the conditional distributions p(z,|z0 II-1) 1) for all tokens I₁ by maximizing Ih ( (r.. Tx11 for our model qa- This is done by minimizing cross-entropy loss over the next-token prediction objective. We train our generation models in the class-conditional regime. To do this, append learned class token the start of the sequence. We also use classifier-fre free guidance (CFG) at inference time, where we predict a token based on both the class-conditional and unconditional distributions. These decisions are discussed in detail in Section 3.6 Details on architecture, training, and CFG discussed in Appendix A
Data. We train and evaluate primarily on ImageNet (1.28M images, 1000 classes) for class- conditional generation without introducing external data for both stage I and stage 2. To confirm general applicability of our method to large scale datasets, we also evaluate our results on several categories of the Large-Scale Scene Understanding Dataset. These categories are cats (1.66M im- ages), horses (2.0M images), and bedrooms (3.02M images).
3.2 Metrics
Reconstruction metries. We use Fréchet Inception Distance (FID) 122 on a validation set as our primary metric. We refer to this metric rFID (reconstruction FID). We also report the signal-to- noise ratio (PSNR) and sulti-scale structured similarity metric (MS-SSIM) (59]
Generation metries. We use FID between the generations and the whole validation set for our primary metric. For clarity, we refer to this as generation FID (gFID). To conform with existing literature, we use the OpenAl guided diffusion [10] repository to evaluate our models. We also report Inception Score, precision, and recall, which are proxies for generation fidelity and diversity.
3.3
Validation Loss and Generation Performance
3.3.1 Neural Scaling Laws for Image Generation
Scaling Laws for Generation Performance. We want study the relationship between validation loss scaling laws (as studied in 29 and (0) and generation performance scaling as measured by FID. For modeling scaling laws, we the common [29 25 functional form L(C) Limin + (C) where Limin is considered the irreducible loss of the problem and L' is the reducible loss as function of C (compute). In our case, we fit [C) C When graphed on log-log scale, a is the slope and À is the y-intercept of a line. For both gFID and validation loss scaling laws, we fit Liin empirically. We note that rFID usually equivalent to Lminn however this is not always the case in practice (see Figure 6 (far left)).
Experimental Setup. We fix stage tokenizer (16k token codebook size, 256 sequence length), vary stage 2 model parameters and compute dedicated to training. Fixing codebook size is important because, all else being equal, validation cross-entropy loss is proportional to the intrinsic entropy of the codes, which increases log-log-linearly by codebook size (see Figure 9). Each point in these scaling law plots is independent training run. We only plot the end point of each training run since our cosine learning rate decay schedule results in substantial validation loss changes in the last few iterations of training. Given that we are iterating on a fixed-size dataset (~~ 3 10 total tokens), we do not scale models past 775M parameters to avoid data scale bottlenecks.
Log-log- lineerr loss scaling for visual token modeling. In Figure
present our findings on the relationship between compute scaling and validation loss. These results corroborate scaling trends found in 220 [9]. We see consistent log-log- log-log-linear scaling with validation loss (Figure 4 (left)) on the compute validation loss Pareto curve across four orders of magnitude.
Validation loss and generation performance. In Figure (center), we plot the gFID score function of training compute for different model scales. We see that, in general, lower validation loss implies better generation performance (lower gFID). However, if we look at the upper Pareto frontier

---

0.76
Woreasang
Honeadng
Ioreasing
Inzuesiin [JC] 25394)
hncrewing 301- 38224)
0.74
Incressing 28354)
- Inzisso (591
220
PS2 2351
4-
0.70
21.0
0.68
20.5
0.66
20.0
0.64
2
Piool
Pixol
per Pixel
Figure 3: Relationship between different methods of scaling bits/pixel and reconstruction per- formance. Here we increase bits per pixel of either vary the codebook size |C| (210 227} with fixed number of tokens N 256 or by varying N E {196... 576} with fixed |C] = 2 using the strategy described in Section 3.4 Increasing token count is generally more effective the cost of increased inference compute. We also compare our VQ recipe with FSQ (37], and find it generally outperforms, so we use VQ for all of our experiments going forward.
40
40-
Val Cros-Entropy Loss
20
7.8
7.7
7.5-
7.43
100
2007
2.5
1013
10AT
102
1023
2.5
7.4
7.5
7.8 7.0 8.0 8.1 6.2 Val Eno Lass
7.6
77
Train FLOPs
Train FLOPs
Figure 4: Neural scaling laws for ruto-regressive image generation (ImageNet 256x256). We show validation cross-entropy loss (left) and gFID (center) as a function of total train FLOPs. Across model scales, we observe log-log linear scaling with train FLOPSs for both FID and cross- entropy loss across Pareto frontiers. Moreover, we observe validation cross entropy loss is generally predictive of gFID (right). See Section 3.3.1 for further discussion.
for Figure (right), see that individual model scales achieve lower gFIl) for correspondingly higher validation losses. This is considered the "overtraining" regime with respect to validation loss, described in (15). This implies that validation loss is only predictive of FID ordering when the number of training steps is held constant. More generally, models on the training compute pareto frontier of validation loss are also on the pareto frontier of gFID. This means that compute optimality with respect to validation loss also corresponds to compute optimality with respect to gFID, even if specific orderings are not respected between the two quantities.
3.4 Sequence Length and Compute Scaling
In this section, we investigate how scaling the tokens per image affects rFID and gFID.
Setup. To vary the number of tokens per image, we follow 49 to re-use our tokenizers trained 256 x 256 resolution and evaluate them at higher resolutions. Our base tokenizers are trained with a downsample factor of 16: if we process an R x R image, the number of tokens produced (R/16)2 R²/ /256. To measure rFID and PSNR, we take the R x R output and downsize to 256 256 with bicubic interpolation. In Figure 3 we observe the expected result that increasing sequence length improves reconstruction performance. It also outperforms increasing codebook size, however at the expense of linear increase in inference cost, We use this method opposed
to changing the number of encoder downsample layers (e.g. 13 69 32) to avoid introducing confounding variables related to architecture and provide more granular control over token counts.
Sequence Length and Scaling Laws. In Figure we see that for smaller models, training with a lower number of tokens per image is more compute efficient, whereas larger models take advantage of the greater representation capacity of longer sequence lengths. Notably in those results, we find that increasing sequence lengths makes the compute optimal Pareto curve better only close to the 256 token/image rFID lower bound. Our main conclusion is that more compressed sequences are generally more compute optimal before performance saturation. As gFID approaches the rFID barrier, using more tokens per image becomes compute optimal.
5

---

256 TD Tikees per Image
400 TD kenss per Image
576 Tokens Image
Comparing Parcto Frontiers
Tokers per Imagu
20
226
20
30
25
Training FLOPs
Training FLOPss
Training FLOPs
Training FLOPs
Figure 5: Relationship between compute scaling laws and tokens per image. We show compute scaling laws for stage 2 performance for different stage I tokens/image. We use the same base tokenizer, expanded to greater tokens/image counts using the strategy described in Section 3.4 We that fewer tokens per image is generally more compute efficient until we get close saturation (rFID lower bound), despite their much better reconstruction performance. Also, (far right) shows that the true trend deviates more from the predicted trend as tokens per image count increases close to saturation, implying larger models are needed to maintain compute optimality.
1024 codebook size
16384 radeback
131072 codebook size
Comparing Paroto Frontiors
[
20
201
10
grib
gRDb
BRD
30
3.0
10
10th
10
101
10
Training ELOPs
Training FLOPs
Training FLDPs
Training FLOPs
Figure 6: Relationship between compute scaling laws and stage 1 codebook size. We show individual model scaling laws at three different stage I codebook sizes. 1k outperforms 131k at low compute regimes and the trend reverses at high compute regimes. Note that 16k outperforms both even though it has worse rFID that the 131k codebook size tokenizer (2.21 rFID vs 1.72 rFID).
3.5 Codebook Size and Compute Scaling
Codebook size is the second main mechanism for changing the rate of the image tokenizer. This differs from changing sequence length because it has a minor effect on inference cost, both for the stage 1 and stage 2 models, so the effect on scaling is more subtle. This is in contrast to scaling sequence length, which increases inference cost linearly on both stage 2 and stage 1.
Stage scaling by codebook size. In Figure 3 show consistent improvements from 1 to 131k codebook size in stage 1 rFID, PSNR, and MS-SSIM without collapse in performance or codebook utilization. in contrast to prior work [13] 37 64]. We credit this scaling improvement to modern VQ recipes [62] [4) in conjunction with a long learning rate warm-up period (training recipe details can be found in the appendix). We also show that our VQ recipe outperforms finite-scalar quantization (FSQ) 37, a modern popular non- VQ quantization scheme.
Stage 2 model scale and codebook size. In Figure 3 we study scaling laws for the extremes of the codebook sizes we study (Ik and 131k), along with our baseline codebook size of 16k. When we use less training compute, we see that the Ik codebook outperforms 131k codebook by 5-10 gFID, even though the rFID is significantly lower. At the other extreme, 131k outperforms 1K, as the stage 2 model nears the gFID saturation point and the worse rFID becomes bottleneck. This crossover happens around 10-7 17 train FLOPs. Important to note is that the 16k codebook size outperforms both at almost every compute scale, implying that there is an optimal point for the rFID/gFID trade-off. Interestingly, the change in scaling is not nearly as extreme as that exhibited by the tokens per image ablation in Section 3.4 despite the massive difference in rFID between 1k, 16k and 131k (1.7vs 2.21 vs 3.0 rFID). We suggest that this is because changing codebook size leaves training and inference FLOPs largely unaffected, while compute scales linearly with increased token counts. We discuss this further in Section 4 This also notably differs with prior works 5 64), which show major differences between stage 2 performance when using VQ with large codebook sizes.
3.6 Causally Regularized Tokenization
We have shown how Pareto optimal scaling laws change based on bits per pixel (bpp) used for to- kenization. We have also shown that increasing bits per pixel above 16k codebook size and 256

---

40
20-
20
CTT led
val Cross-Entroyy
7.7
10-
10
7.6
3.0
7.4
2,5-
2,5
1337
4TRCERO paRt
2073
103H
10 2035 10Th
1077
10
0.0 10 3.5
2,5
Train FLOPs
Train FLOPs
Train FLOPs
lel8
Figure 7: Stage compute scaling laws with CRT tokens (ImageNet 256x256). We demonstrate the superior scaling of CRT tokens across four orders of magnitude with respect to validation cross entropy (Ieft) and FID (center). The trade-off between reconstruction performance and causal de- pendence means that CRT yields better stage 2 performance across many model scales with worse reconstruction (2.36 2.21 rFID). The result is improved absolute generation performance and 1.5x to 2.5x better scaling by training FL.Pss.
Table I: System-level and compute-controllce comparisons of autoregressive image-generation models. (a) System-level comparison on ImageNet 256x256. (b) Compute-controllle comparison against a luned VQ-GAN baseline on ImageNet. (c) Compute-controller comparison against a tuned VQ-GAN baseline across LSUN Cats, Horses, and Bedrooms. Across all settings our tokenizer (CRT) and its reconstruction performance optimized variant, CRT opt, outperform baselines.
(a) System-level comparison between methods on Ima- geNet 256x256: Our method outperforms other AR, discrete methods at equal parameter counts (8x less inference com- pute than LlamaGen-3B for equal performance). CRT opt de- notes our reconstruction optimized tokenizer variant.
(b) Comparison against tuned VQGAN baseline (ImageNet) at saturation. CRT uniformly out per-perfosms the baseline
Tuk. Type
4Params
Tok/Im
gilD.
LSt
Praz"
Reaf
111M
256
4.90
190.65
0181
0.51
Buscline
211M
256
3.32
251.53
0.84
0.52
Type
Model
*Para
TakIm]
FID]
1Sc
Pret
Reet
(221 JED)
300
256
2.89
253.25
0.83
0.54
550M
Dif.
ADM
10
55:M
10.91
101.0
0.69
0.65
775M
2.55
242.33
Dil..
LDM-IG
65
400M
3.60
247.7
DIT.
DIT-L/2
DIT
MAR-H
943M
675M
2.27
5.02
1.55
278.2
303.7
0.81
0.75
0.62
CRT (ours)
(2.5 -iT))
211M
550M
340M
111M
256
256
2.75
2.55
241.95
265.34
278.29
0.83
0.80
0.83
0.54
0.59
Mask.
MaskGIT
227M
256
6.18
182.1
0.80
0.51
773M
256
2.35
289.
0.81
0.39
Mask.
RCG (cond.)
84
502M
256
3.49
215.5
VAR
V1R-J16
50
310M
680
3.30
274.4
0.84
0.51
(c) Comparison against tuned VQ-GAN
VAR-120
600M
680
across LSUN datasets: Under the same compute constraints as (b), CRT uniformly surpasses the tuned baseline on LSUN Cats, Horses, and Bedrooms. gFIDruP is the pri- mary metric because Inception features are out-of-distribution for LSUN.
VAR
VAR-/24
1:0
680
2,09
312.9
0.82
VAR
VAR-300
50
2.08
680
1.92
323.1
0.82
0.59
AR
VQVAE-2
13
13.58
5120
31.11
.-15
0.36
0.57
AR
VQGAN
R
227M
256
18.65
80.4
0.78
0.26
AR
VOGAN
1.48
256
15.7H
AR
VITVQ
52
1024
1.17
AR
LlamaGen-BB
49
111M
256
6.10
182.54
0.85
0.42
AR
LlamaGen-1.
49
343M
256
3,80
24K:28
0.83
0.51
AR
AR
LlamaGen XXL
LlamaGen XL
29
19
775M
1.4B
256
256
3.09
3.39
253.61
227.08
0.81
0.83
0.53
0.54
Daraen
Tel. Tree
Toote
RecT
AR
LlamaKGen-B
111M
$76
5.16
24330
256
7.23
5.31
2.55
0.36
AR
LlamaGie-1..
343M
576
3.07
256.06
0.83
0.52
ESEN
1.11
AR
AR
LlamaGen-XL
LlamaGen-X0 XXL
29
19
775M
14B
576
576
2.62
2.34
253.90
240.08
0.80
0.80
0.57
0.59
11-4 rtth
HCM
796
0.29
1.80
3:11
0%
AR
LlamaGen- 3B
3.1B
576
2.18
263.33
0.81
0.58
MCM
766
5.00
1.80
3.6
Bnl
AR
CKT-AR-
111M
256
4.34
195.33
0.81
ESEN Roses
AR
AR
CRT-AR ART-AR-775M
340M
775M
300M
256
256
2.75
2.35
259.12
265.24
0.81
0.59
CRT
11.00 -Fib
HMM
796
3:60
2:11
0m
AR
AR
CRT AR 340M
-R-----RR-10 11IM
340M
111M
256
256
2.45
4.23
249.08
194.87
0.82
0.84
0.29
0.57
(2.91/10)
Budik
HMM
196
1.9
1%
0.8
AR
(validation data)
CRT 7R5775M
775M
256
2.18
1.78
268.38
236.9
0.82
0.58
LSUN Bobvens
C3T
11.39
34250
250
836
2.99
0.30
tokens per image generally makes scaling worse. Further, while reducing to 1K codebook size im- proves the scaling law marginally, explicitly reducing the reconstruction capacity to such a degree greatly harms generation performance close to saturation. Going past bpp measure of rate, we study an alternate rate-distortion trade-off based on optimizing stage I latents for stage 2 perfor- mance. Our guiding question is: since our stage 2 model is an auto-regressive transformer, can we imbue this inductive bias into the tokenizer? We show we can take our optimal baseline tok- enizer and apply a simple causal regularization that improves scaling without changing bpp used for compression.
regulle
Setup. Figure demonstrates our key intervention. We apply an l2 loss parameterized by a causal transformer to the tokens before quantization, and backpropagate this loss to the encoder of the our stage auto-encoder. We train with this model-based loss (CCRT) in conjunction standard VQGAN
7

---

3.0
2.0
6.50
0.30
2.8
6.25
2.6
2.6
6.00
6.00
2.4
2.4
5.75
5.50
2.2
3.50
10
12
0.0
2.5
5.0
7.5
16.0
12.5
15.0
of CRT lass layers
Causal loss weight (A)
Figure 8: Impact of CRT hyperparameter choices on reconstruction FID (rFID) and generation FID (gFID). (top): Increasing the number of causal regularizer layers worsens rFID and, beyond layers, also degrades gFID. (bottom): Higher causal loss weights (A) trade off rFID for improved next-token prediction, with optimal balance
0 layers and À correspond to baseline.
Further discussion in Section 3.6
losses, using the same hyperparameters as our baseline recipe. Applying this loss with cross-entropy after quantization is intuitive; however, we found that this damages reconstruction fidelity too much (see Section(4 4) since VQ tokens change discretely during training. If token i changes to token j. the cross-entropy loss treats it the same regardless of how similar the tokens and j are. Meanwhile, & on pre-quantized tokens are naturally similarity aware. We use a 2-layer causal transformer with the architecture of our stage model. This introduces 5% cost in training FLOPs, which we control for by training 5% less (2 fewer epochs). For both our baseline and CRT models we codebook of size 16k and 256 tokens per image.
Effects on stage 1 reconstruction. After applying this loss, our tokenizer is marginally worse in terms of reconstruction against the compute normalized baseline (2.36 rFID vs 2.21 rFID). Intu- itively, our loss attempts to make token as predictable as possible given tokens O through i 1. The most predictable set of tokens would if the encoder outputs a sequence of entirely dependent tokens (e.g., the constant token). However, this reduces the total information that can be conveyed through the tokens. Our CRT loss (a proxy for generation) conflicts with reconstruction, implying a trade-off. With this inductive bias, stage? 2 scaling is improved, which we discuss next.
Improvements in stage performance. In Figure we show the relative scaling properties of stage 2 models using our CRT tokenizer compared to the VQ-GAN baseline (in Figure 4.. For clarity, we also include Pareto frontier scaling trends from Figure 4 Figure 7 (left) Shows that stage models trained with our tokenizer achieve systematically lower validation losses. Figure 7 (center) shows that this translates to improvements generation performance on the training compute Pareto front. Namely, our compute scaling law is improved from a -0.65 to a -0.73. Finally. Figure 7 (right) shows the actual magnitude of improvement, which the log-log-linean plots obfuscate: CRT achieves gFID scores with 1.5 3x compute efficiency and improve over the baseline, even with worse rFID. We show comparisons at individual model scales in Ta, and detailed comparisons across training durations in Appendix F. Our model systematically outperforms the baseline in all compute and model size regimes.
Results on LSUN 61]. In Table Te. demonstrate our method's generality by evaluating several L.SUN categories. We demonstrate improvements in gFID and gFID across three model scales using CRT across all model scales. Note that we use CLIP gFID because these datasets are out of distribution for the Inception network, making FID more noisy metric.
Optimizing the CRT tokenizer for absolute performance. In the previous sections, we perform compute-controoleed comparisons of CRT and baseline. CRT outperforms the baseline despite the trade-off in reconstruction performance. However, we showed previously that generation perfor- mance close to the rFID barrier is affected by reconstruction quality. In this section, we apply simple optimizations to boost reconstruction performance, which do not modify our fundamental setup, push the limits of our method. These include: (1) increasing training duration from 38 epochs to 80 epochs, (2) doubling the size of the decoder, (3) increasing the codebook size from 16k 131k, which interestingly improves performance for CRT but not our baseline. With these changes, our tokenizer achieves 1.26 rFID. Our stage training procedures remain constant. In Ta- Tp. we show a sestem-level comparison of our method (RR to other existing methods in the field. The closest comparison to our method is LlamaGen, which uses the same architec- tures and inference method. Notably, we exceed the performance of the LlamaGen .B parameter model with our 775M parameter model, while using half the number of tokens per image. This is

---

Per Postiion Entropy
Total Entrosy Vacabulary S26
Skowness vs Vacabulary Size
Aug. Loss Tokoon Index
Excrpp 1pp
0.8
Entrapy (bins)
1224
Avcrage Men
Mean Loss
3240
0
0:52
34885
0.4
62336
0.2
232973
100
150
200
250
10
30t
10
107
10a
307
50
100
1.50
200
250
Posttion
Vacabulary Siza
Vocabulary Size
Position Indox traster ordert
Figure 9: Entropy analysis of image tokenizers. We do a deep analysis of how tokenizer entropy distributions. Per position and total entropy deviate with increasing codebook size, implying n n-gram modeling gains which are not reflected in codebook utilization. Skewness (center right) is another view of the concentration of codes property demonstrated by (center left). We compute skewness as
demonstrating that as the codebook size increases, so does codebook specialization per position. (far right) shows CRT tokenization uniformly reduce stage loss across all positions, improving stage 2 generation performance. See Section for more details.
2estapp prsition
gioal orrngg
an 8x reduction in inference compute. We show uncurated qualitative examples from select object categories in Figure 14 and further examples in Appendix G
Hyperparameter Ablations. There are two key choices involved with our causal loss: (1) the size of the causal model used during stage 1 training and (2) the weight À we assign the & next- token prediction loss (TC ). We include an ablation of these parameters in Figure We plot the relationship of rFID and gFID with respect these parameters, computing gFID over an average of stage 2 model scales (52M, 11 IM, 21 IM, 340M) trained for 250k iterations each. In Figure 8 (left), we increase the number of layers in our causal regularizer. For every 2-layers we add, we reduce the overall training time by 5% to compensate. We see that rFID uniformly increases with more layers, as expected. After 4 layers, this worsening rFID hurts average gFID. Thus, we chose to use two layers in all of our experiments, which has the added advantage of being very computationally cheap. In Figure X (right), we see a similar trend. Increasing À biases the encoder to focus on making representations easy for next-token prediction rather than reconstruction, making overall rFID worse. However, there's a sweet spot around À = 4 where this worsened reconstruction is within acceptable range and this also improves overall generation performance.
Loss Function Ablation (&g vs cross-entropy). In Appendix E we show comparisons between lg loss and cross-entropy (CE) loss. We also compare to the loss suggested by [56]. who propose a similar auto-regressive inductive bias in the video domain (based on CE not fg). In general, CE losses harm reconstruction performance in our setting but fail to provide substantial generation performance improvement to be worth the trade-off. This deviation from findings in (56) is likely due to two reasons: (1) They use an architecture that is substantially different from ours, focusing on global versus local tokens (we use the VQGAN architecture for compatibility with existing SOTA literature) (2) The video generation setting they studied is much further from performance saturation, meaning changes to tokenizer reconstruction are less likely to harm generation performance.
4 Analysis
Factors affecting validation loss. We saw in Section 3.4 that tokens per image affected compute scaling more significantly than changing the codebook size. Why that? We note that for a stage unigram model, the loss lower bound Limin is the empirical entropy of the codebook distribution per- position, since cross-entropy can decomposed into H (X) + DRr. (q(X)lp(X)) and DxL. - when y + p. In Figure 9 (far left), we measure this directly over the ImageNet train set over various codebook sizes.In In Figure 2 (center left), we see that the per position entropy converges, even as the overall codebook entropy increases log-linearly due to uniform codebook utilization. This implies that as codebook size increases, codes become more specialized by position. This has the effect of ensuring that Lmnin for unigram model does not increase, meaning that our stage 2 model can still learn even with very large codebook sizes.
What does CRT do? Lmin for an n-gram model is upper- bounded by this estimate, as H(X,) xxx)) x(x.)Xx<)) is not directly measurable, this quantity what we are trying to estimate with language modeling. We directly model and minimize this in stage with CRT, using the same architecture as in stage 2, to ensure that our inductive biases transfer. In Figure ((ar right), we plot the stage 2 losses per-position for 21 M parameter auto-regressive model between our baseline
9

---

arXiv preprint tokenizer and CRT. These tokenizers have the same codebook size, utilization, and per per-position entropy. However, we see that CRT lowers loss uniformly across token positions, with most of the loss reduction happening at the end of the sequence. This implies that the CRT tokenizer has less estimated entropy than baseline when using a causal entropy model.
Conclusion & Future Work
We systematically study the tradeoff between compression and generation for image tokenization and reveal several key insights: First, we demonstrate that the relationship between compression and generation quality is nuanced: smaller models benefit from more aggressive compression, even at the cost of reconstruction fidelity. Second, provide principled framework for analyzing this trade-off through the lens of scaling laws, showing consistent patterns across multiple orders of magnitude in computational budget. Finally, introduce Causally Regularized Tokenization (CRT), a method for optimizing tokenizers specifically for autoregressive generation. We add a causal inductive biases during tokenizer training which substantially improves compute efficiency (2-3x) and parameter efficiency (4x reduction) compared to previous approaches. Notably, these gains are most pronounced the resource- constrainnd regime, making the method particularly valu- able for practical applications. Our results highlight the importance of considering the interaction between different components for multi-stage machine learning pipelines: rather than optimizing each stage independently, we achieve significant gains by baking in inductive biases in early stages. This principle suggests several promising directions for future work, for example extending to other architectures (e.g. diffusion models) or other modalities (e.g. video and audio tokenizers).
6
Acknowledgments
The authors thank Jonathan Hayase, Matthew Wallingford, Ludwig Schmidt, Nicholas Lourie, Will Merrill, Mohammand Rastegari, Jaskirat Singh, Lili Yu, Chunting Zhou, Artidoro Pagnoni and members of the RAIVN Lab for helpful comments and discussions throughout the course of re- search. VR and AF acknowledge funding by NSF IIS 1652052, IIS 1703166, DARPA N66001- 19-2-4031, DARPA W911NFF15-1-0544 and gifts from Allen Institute for Artificial Intelligence, Google and Apple.
References
[1] Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXfv:1701.07875. 2017.
Wasserstein gan.
[2] Roman Bachmann, Jesse Allardice, David Mizrahi, Enrico Fini, Oguzhan Fatih Kar, Elmira Amirloo, Alaaeldin El-Nouby, Amir Zamir, and Afshin Dehghan. Flextok: Resampling im- ages into 1d token sequences of flexible length. In Forty-second International Conference on Machine Learning, 2025.
[3] Huiwen Chang. Han Zhang, Lu Jiang, Ce Liu, and William Freeman. Maskgit: Masked gen- erative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11315-11325 2022.
[4] Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein. Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, et al. Blip3-o: A family of fully open unified multimodal models-architecturr training and dataset. arXiv preprint XxXx:25505.0966. 2025.
[5] W Chen, X Du, F Yang, Beyer, Zhai, TY Lin, et al. simple single-scale vision trans- former for object localization and instance segmentation. 2022: doi: doi. org/10.48550. arXiv preprint arXiv.2112.09747
[6] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-iman model. arXiv preprint mrXiv:2209.06777 2022.
[7] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint 2309v:2309. 16588, 2023.
10

---

[8] Mostafa Dehghami, Josip Djolonga. Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scal- ing vision transformers to 22 billion parameters. arXiv preprint arXiv:2302.05442 2023.
[9] Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint 0rXiv:2005.00344 2020.
[10] Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. arXiv preprint arXiv:2105.05233, 2021.
[11] Alazeldin El-Nouby, Michal Klein, Shuangfei Zhai, Miguel Angel Bautista, Alexander Toshev, Vaishaal Shankar, Joshua M Susskind, and Armand Joulin. Scalable pre-training large autoregressive image models. arXiv preprint arXiv:2401 00541,, 2024.
[12] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow trans- formers for high-resolution image synthesis. arXiv preprint arXiv:2403.05206 2024.
[13] Patrick Esser, Robin Rombach, and Bjöm Ommer. Taming transformers for high-resolution image synthesis. arXiv preprini arXiv:2012.09841 2020.
[14] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 12873-12883, 2021.
[15] Samir Yitzhak Gadre, Georgios Smymis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, et al. Lan- guage models scale reliably with over-training and on downstream tasks. arXiv preprint arXiv:2403.088 085400, 2024.
[16] Y Ge, Y Ge, Z Zeng, X Wang, and Y Shan. Planting seed of vision in large language model. arxiv 2023, arXiv preprint arXiv:2307.08041.
[17] Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan. Planting a seed of vision in large language model, 2023.
[18] Robert Gray. Vector quantization. IEEE Assp Magazine, 1())4-29. 1984.
[19] Yuchao Gu, Xintao Wang, Yixiao Ge, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Re- thinking the objectives of vector-quantized tokenizers for image synthesis. arXiv preprint arXiv:2212.031855 2022.
[20] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Hee- woo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14700 2020.
[21] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08716 2021.
[22] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. arXiv preprint arXiv: 706.0000000 2017.
[23] Irina Higgins, Loic Matthey, Arka Pal, Christopher P Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual con- cepts with a constrained variational framework. ICLR (Poster), 3, 2017.
[24] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207. 12598, 2022.
[25] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
11

---

[26] Minyoung Huh, Brian Cheung, Pulkit Agrawal, and Phillip Isola. Straightening out the straight- estimator: Overcoming optimization challenges in vector quantized networks. In International Conference on Machine Learning, pages 14096-14113, PMLR, 2023.
[27] Phillip Isola, Jun- Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation 1.07004, 2016.
with conditional adversarial networks. arXiv preprint
[28] Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Quzhe Huang, Bin Chen, Chenyi Lei. An Liu, et al. Unified lanuuagevvisonn pretraining in Ilm with dynamic discrete visual tokenization. arxiv 2024. arXiv preprint arXiv:2309.04669.
[29] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint wwxiv:200.088300 2020.
[30] Tero Karras, Samuli Laine, Miik Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. arXiv preprint arXiv: 191.0099/, 2019.
[31] Diederik Kingma. arXiv:1412.6980, 2014.
Adam: A method for stochastic optimization. arXiv preprint
[32] Doyup Lee, Chiheon Kim, Sachoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. arXiv preprint arXiv:2203.0194) 2022.
[33] Brian Lester, Jaehoon Lee, Alex Alemi, Jeffrey Pennington, Adam Roberts, Jascha Sohl- Dickstein, and Noah Constant. Training lims over neurally compressed text. arXiv preprint NEKV:22004003626. 2024.
[34] Tianhong Li, Dina Katabi, and Kaiming He. Return of unconditional generation: A self- supervised representation generation method. Neural Information Processing Systems, 2024.
The Thirty-eighth Annual Conference on
[35] Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He, Autoregressive image generation without vector quantization. arXiv preprint wrXiv:2406.11838 2024.
[36] Zhuoyan Luo, Fengyuan Shi, Yixiao Ge, Yujiu Yang, Limin Wang, and Ying Shan. Open- magvit2: An
project toward democratizing auto-regressive visual generation. arXiv preprint (eX:::2409.04111 2024.
[37] Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple. arXiv preprint StIV:22309.15005 2023.
[38] Keita Miwa, Kento Sasaki, Hidehisa Arai, Tsubasa Takahashi, and Yu Yamaguchi. One-d-piece: Image tokenizer meets quality-controllable compression. arXiv preprint w.X1:22501.10044 2025.
[39] William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint 17X.4222 2022.
[40] Z Peng, L Dong, H Bao, Q Ye, and F Wei. Beit v2: Masked image modeling with vector- quantized visual tokenizers. arxiv 2022. arXiv preprint arXiv:2208.06.3 2022.
[41] Alec Radford. Improving language understanding by generative pre-training. 2018
[42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-87886 8763. PMLR, 2021.
[43] Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with va-vae-2. arXiv preprint
00446, 2019.
12

---

[44] Anton Razzhigaev, Arseniy Shakhmatov, Anastasia Maltseva, Vladimir Arkhipkin, Igor Pavlov, Ilya Ryabov, Angelina Kuts, Alexander Panchenko, Andrey Kuznetsov, and Denis Dimitrov. Kandinsky: an improved text-to-image synthesis with image prior and latent diffu- sion, arXiv preprint arXiv:2110.03502, 2023.
[45] Juan A Rodrigue7, David Vazquez, Issam Laradji, Marco Pedersoli, and Pau Rodriguez. Ocr- vqgan: Taming eext-within- generation. In Proceedings of the IEEE/CVF Winter Con- ference on Applications of Computer Vision, pages 3689-3698, 2023.
[46] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. arXiv e-prints, pages arXiv- 21 2112, 2021.
[47] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. arXiv preprint arXiv:
2014.
[48] Craig W Schmidt, Varshini Reddy, Haoran Zhang, Alec Alameddine, Omri Uzan, Yuval Pinter, and Chris Tanner. Tokenization is more than compression. arXiv preprint Sttiv::202.187766 2024.
[49] Peize Sun, Yi Jiang. Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint 265555 2024.
[50] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autore- gressive modeling: Scalable image generation via next-scale prediction. arXiv preprint arXiv:2404.00 02905, 2024.
[51] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiiv2302. 13971, 2023.
[52] Michael Tschannen, Olivier Bachem, and Mario Lucic. Recent advances in autoencoder-based representation learning. arXiv preprint
2018.
[53] Michael Tschannen, Cian Eastwood, and Fabian Mentzer. Givt: Generative infinite-vocabulary transformers. In European Conference on Computer Vision, pages 292-309, Springer, 2025.
[54] Michael Tschannen, André Susano Pinto, and Alexander Kolesnikov. Jetformer: An autore- gressive generative model of raw images and text. arXiv preprint arXiv:2411.19722. 2024.
[55] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[56] Hanyu Wang, Saksham Suri, Yixuan Ren, Hao Chen, and Abhinav Shrivastava. Larp: Tok- enizing videos with a learned autoregressive generative prior, 2024.
[57] Junke Wang, Yi Jiang, Zehuan Yuan, Binyue Peng, Zuxuan Wu, and Yu- Yu-Gang Jiang. Omnitok- enizer: A joint image-video tokenizer for visual generation. arXiv preprint arXiv:2406.09399 093999 2024.
[58] Luting Wang, Yang Zhao, Zijian Zhang, Jiashi Feng, Si Liu, and Bingyi Kang. Image under- standing makes for good tokenizer for image generation. arXiv preprint arXiv:2411.. 1.04006, 2024.
[59] Zhou Wang, Eero P Simoncelli, and Alan C Bovik. Multiscale structural similarity for im- age quality assessment. The Sevith- Asilomar Conference on Signals, Systems & Computers, 2003, volume 2, pages 1398-1402. leee, 2003.
(60] Xin Wen, Bingchen Zhao, Ismail Elezi, Jiankang Deng, and Xiaojuan Qi. principal compo- nents" enable new language of images. arXiv preprint
2025.
13

---

[61] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 033600336, 2015.
[62] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vggan. arXiv preprint arXiv: 10.04627, 2021.
[63] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Va- sudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv: 10789, 2(3):5, 2022.
[64] Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats iiffusion-tokenizee is key to visual generation. arXiv preprint 2rX1v:2310.057777, 2023.
[65] Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, al. Scaling autoregressive multi-modal models:
Pretraining and instruction tuning. arXiv preprint arXiv:2309.02550 0259999900299........ 2(3), 2023.
[66] Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chich Chen. An image is worth 32 tokens for reconstruction and generation. arXiv preprint vrXiv:2406.07555 2024.
[67] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[68] Yue Zhao, Yuanjun Xiong, and Philipp Krähenbühl. Image and video tokenization with binary spherical quantization. arXiv preprint arXiv:2406.07548, 2024.
[69] Chuanxia Zheng, Tung-Long Vuong. Jianfei Cai, and Dinh Phung. Movq: Modulating quan- tized vectors for high-fidelity image generation. Advances in Neural Information Processing Systems, 35:23412-23422 12-23425, 2022.
[70] Zhenhai Zhu and Radu Soricut. Wavelet-based image tokenizer for vision transformers. arXiv preprint arXiv:2405.1886 2024.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly state bulleted list of contributions the introduction, as well as datasets/metrics to qualify our claims.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Discussed in Appendix B
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: No real theory discussed except as intuition (Section We do not claim any novel theoretical results.
4. Experimental result reproducibility
14

---

Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide most details of our method paper, and extensive details of setup in Appendix A We will work to release code rost-acceptance, however cannot promise this due to corporate constraints.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We do not have permission to release code. We are working towards an open implementation. In the meantime, we provide all necessary details for reproduction in Appendix A
6. Experimental setting/details
Does paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: In Appendix
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: Partial [Yes]
Justification: A significant portion of our study is on scaling laws, for which we report best fit Rt FID and validation loss variances are low (<0.01 and <0.0001 respectively) but do not report these exactly due to computational cost.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Appendix
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:p///..aarr......cpppaaaaaaa
Answer: [Yes]
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Appendix
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We train specific academic datasets, limiting the risk of this artifact.
15

---

12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite where relevant for all of these asset types.
13. New assets
Question: Are new assets introduced in the paper well-documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
14. Crowdsourcing and research with human subjects
Question: For crowdsoureing experiments and research with human subjects, does the pa- per include the full text instructions given participants and sereenshots, applicable, well as details about compensation (if any)?
Answer: [NA]
15.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMSs if it is important, original, or non-standard component of the core methods in this research? Note that if the L.M used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
16

---

Structure of Appendix. Appendix A documents full training details for all experiments and ar- chitectures. Appendix discusses the limitations of our experimental methodology. Appendix provides further ablations to justify our choice of l2 over cross-entrope loss with CRT. Appendix
provides more detailed scaling analysis across range of experiments the main paper. Appendix
G
(Figures 15 28 provide uncurated qualitative generations across range of ImageNet categories for our best model.
A Further training details
In this section, we enumerate the complete training details for experiments Section hyperparameters are the same for both
These basic
A.1 Stage 1: VQGAN training
We largely follow the training recipes from 13 and 49 that we state here. The VQGAN architecture exactly that of [3] with the following standard loss terms (not including CRT loss):
train our tokenizer, with modifications
AveLro AGAN + MPenceptual + A12C2.
The perceptual loss comes from LPIPS 667). The l loss is defined as L2(x, = || u11 be- tween the reconstruction and the input. The weights for each loss are: AvQ = 1.0, AGAN 0.5, Aperceptual = 1.0, λL 1.0. We start AGAN at O. After 20k iterations, we anneal AGAN to 0.5 using a cosine schedule for 2k iterations.
GAN Loss Details. The GAN loss uses the PatchGAN 27 as a discriminator with the Wasserstein GAN loss function from [1] directly reconstructions from the auto-encoder to force the output to look "realistic." This is the only "reference-free" image loss used. Using improved StyleGAN and StyleGAN2 30 discriminators ended up with worse qualitative results in our setting.
we project from 256 to 8 dimensions before doing codebook lookup and perform codebook lookup with cosine distance. We use two losses, quan- tization error loss and a commitment loss, both proposed in (55). with a commitment = 0.25. We do not use entropy loss, as it was unnecessary to prevent codebook collapse in our setup. We find that training with long learning rate warmup period is crucial for scaling beyond 16k codes, otherwise performance saturates at 2.1 rFID and codebook utilization stagnates.
Codebook Loss Details. Following
62 49
Optimization Parameters. We use Adam 31 with learning rate Adam W with weight decay 0.1 performs identically. We use the same optimization parameters for the PatchGAN discriminator. We use with linear warmup from 1e-5 for 3k iterations, batch size 128 and constant learning rate. We found that this warmup is critical for good performance, especially for codebook length and training duration scaling. Cosine learning rate gave very similar results, and constant learning rate allows for continued training. We train for total of 400k iterations unless stated otherwise.
and f (0.9, 0.95).
CRT details. CRT is our L.2 next-token prediction loss applied to the raster-order tokens during stage 1 training. The weight for this loss for all of our experiments is 4.0 (we include a sweep in Figure 8 Like with the GAN discriminator, we anneal this weight from 0 to 4.0 for Ik iterations from the start of training. The duration of this annealing not important for final performance. The architecture we use is identical to our stage 2 model, except with no last classification layer and no token embedding layer (replaced with single linear layer). We use separate optimizer for this component: an AdamW optimizer with learning rate and 3 parameters matching the original VQGAN at all iterations. Our weight decay was 0.1. We normalize the loss by the annealed weight at all iterations so that the causal model training is not impacted by its weight. To control for extra training FL.OPs, we drop the number of training iterations to 380k. For CRT opts we train for 800k iterations.
17

---

Table 2: Stage model configurations. These are the parameters for the Llama architectures we use in our stage 2 experiment sweeps. To construct this sweep. we interpolated by head and layer counts with existing standard models (-S, -M, -I., etc.) and ensured 64 dimensions per head.
Parameters
Heads
Layers
Dimension
5.21 107
10
6
640
8,06 X 107
11
9
704
1.11 x 10
12
12
768
2.12 x 108
14
18
896
3.43 10
16
24
1024
5.17 X 10
18
30
1152
7.75 x 10
36
1280
A.2 Stage 2: Auto-regressive training
Architecture. Following 449, we use the Llama architecture [51 with QK-layer normalization 8 for stability. For some 576 tokens/image runs, also needed use z-loss [6] with coefficient of le 4 for stability. We find that this did not impact the performance of stable runs.
Optimization. We perform data augmentation using ten-crop, allowing us to pre-tokenize the train set during stage training. We use AdamW, betas (0.9. 0.95) and a cosine learning with schedule with linear warmup (5k iterations) and batch size of 256. Our max learning rate is 3e 3, with pre-warmup learning rate 3e 4, annealing our learning rate to O. We also include class token dropout rate of 0.1, where we replace the class-token with randomly initialized learnable dummy token to enable classifier-free guidance. We train for total of 375k iterations unless otherwise stated (we often sweep this parameter for stage scaling law experiments).
Class- fass-conditioning and Classifier-Free Guidance (CFG). For class-conditioning, special class token to image token sequences at train time. At inference time, we use the relevant class token to start the sequence to generate an image of particular class. We sample with next- token prediction without top-k or top-p sampling. All our results use CFG at inference time to further emphasize classifier guidance. To do this, we have two inference procedures, one with the class-token and one using the aforementioned dummy token. At each timestep, these yield logits f, for the unconditional logit, and le for the class-conditioned logit. To get the next token, we sample from the combined logit Lefa = f,, + (fc L,)a, where a is known at the CFG scale. This kind of sampling was initially used for diffusion [24], but has seen success in auto-regressive image generation as well 665 49]
prepend
and is used commonly.
CFG trades off fidelity and diversity (related precision and recall respectively) generation, and we found that the optimal CFG scale was also pareto optimal for precision and recall. To choose our CFG scale at inference time, we split a 50k subset of the train set (not held out during training) to compute FID against. We select the optimal FID against samples from a E (1.5, 1.75, 2.0, 2.25] and use those generations against the validation set. We do this because CFG can have large effect on FID score [35] 49 50], and optimizing directly against the ImageNet validation set can pollute our evaluation. However, in separate experiments, we find that optimal CFG against this train split and the validation set are identical in practice.
A.3 Scaling law sweep details
To produce scaling laws (e.g. Figure we sweep training compute by changing the num- ber of total training iterations. Because we are in the repeated data setting, unlike [29, and we are using a cosine learning rate schedule, we need to evaluate the end points of separate, independent training runs to produce these curves. The iteration counts we sweep over are (15k, 22.5k, 45k: 90k, 187.5k, 375k: 750k, 1.5M, 2.25M} for all models. We compute estimates for training FL.OPs using the approximation from 29 The architecture parameters for our sweep can be found in T
18

---

Parameters
8.4
111
Validation Loss
8.2
3.43 10"
5:17 101
8.0
7.8
7.6
10'
10
Dataset Size (tokens)
Figure 10: Stage Validation Loss by Dataset Size. Here we study the data scale bottleneck by varying the total size of the dataset. Every model was trained for 250k iterations at batch size 256. We see that our dataset is large enough to demonstrate scaling laws at the parameter range studied.
A.4 Computational Cost
Each experiment can be run on one node 80GB A100s. Training the tokenizer for 40 epochs takes 3 days in this setting. The largest stage 2 model (770M) trained for 375k iterations takes 3.5 days, and the cost decreases linearly with the number of parameters and training iterations.
B Limitations
Our work has few specific limitations, largely related to the availability of compute and data. First, the largest model we test is 775M, which we do in order to prevent overfitting (given that ImageNet is finite dataset). This is close to the range of current state-of-the of-the-att image generation models, such as [2] which ranges from 800M parameters to 8B parameters, but is smaller than modern large language models 25 51]. Future work could include scaling parameter counts. Second, primarily evaluate on class-to-image synthesis and not text-to-image synthesis. We do this because FID is much more well-behaved metric ImageNet (see inconsistent scaling in LSUN results by parameter count Table Tpp). However, this means we cannot train on web-scale datasets, and thus in the repeated data regime, deviating from (29]. In Figure [0] we study the effects of this but varying the stage 2 training set size (375k training iterations), showing that for existing model sizes are not yet the data bottleneck regime. Future work could include validating an distribution metric (e.g. CLIP FID) on large scale text-to-image dataset and then reproducing our scaling laws.
C Broader Impacts
Our method, as applied in this paper, makes training of image generation models more efficient and boosts overall performance. Image generation in general has broader social impacts. For example, Deepfakes can cause personal harm. Popular text-to-image models can be seen as stealing artist data for corporate gain. In its most benign form, this method can be used to accelerate research on fully legal datasets (e.g. Creative Commons images), however that may not be how it is used in the future.
D Further Baseline Tokenizer Ablations
Table shows the difference between native resolution training and resolution extrapolation (the method used in the paper). Native resolution training improves performance very little compared to extrapolating to higher resolution images. This allows to compare a fixed tokenizer at many different compression rates, removing the confounding aspect of higher resolution training.
Figure ?? shows the result of training with and without long learning rate warmup period. This was crucial for achieving high codebook utilization and good performance without collapse for large codebooks.
19

---

rFID Codabook Size
With Wormup
Without Warmup
2.75
2.25-
2
2/a
2
2.5
Codebooo Size
Figure 11: Effects of learning rate warmup tokenizer performance.
Table 3: Training Tokenizers at Native Resolution. Native resolution training improves perfor- mance very little compared to extrapolating to higher resolution images. This allows to compare a fixed tokenizer at many different compression rates, removing the confound of higher resolution training.
Tokens per Image
Native resolution training
rFID / gFID
400
1.30 / 3.76
400
yes
1.21 3.72
576
0.99 / 3.61
576
yes
0.93 / 3.66
E
Further CRT Ablations
Results for different causal loss application configurations are presented in Table 4
Table 4: Comparison between & loss and CE loss for causal regularization. The LARP loss is also compared with components relevant to this setup (CE + stochastic VQ cosine distance + scheduled sampling (56)). The setup from [56] is likely more successful in conjunction with global register tokens as opposed to patch tokens (as in our case). We report average gFID across stage 2 scales 52M, 11 111M, 21 IM, and 340M trained for 250k iterations each.
Causal Loss Form
rFID
Avg. gFID
Eg before quantization (CRT) &g after quantization
2.36
5.51
2.61
6.37
CE after quantization (A 0.03)
2.53
6.10
LARP (CE + SVQ + scheduled sampling)
2.67
6.08
Effects of decoding order on performance. As noted in prior work [13 raster-scan is empirically the best canonical ordering for image generation. Our results corroborate this. While CRT provides improvements in these settings, is not enough to overcome the difference in ordering performance (> (>IgFD). is possible that a more flexible base architecture (e.g. a ID transformer based tok- enizer) than that of VQGAN would allow for more flexible re-ordering, and we believe this to be an interesting direction for future work.
More Detailed Scaling Plots
In this section cover more detailed scaling law explorations expanding plots in the main paper.
Figure 13 provides different perspective on the comparison between Figures and 7 by comparing stage 2 model scales directly by gFID performance by train FLOPs Note that like the main figure, each point is an independent training run. We see that CRT tokens improve perfor- mance at all scales both for training efficiency and final performance. This difference especially pronounced at the smaller compute and parameter scales.
We do this
20

---

606
580
520
1.0
-4
100
E00
030-
030
030
Figure 12: Effects of CRT weight À on VQGAN losses. In general, increasing loss weight increases losses related to perceptual quantities (e.g. l2 reconstruction, discriminator loss, and perceptual LPIPS loss). It also seems to reduce codebook losses, like VQ and commitment.
Stage Param. Count: 5.21 x0'
Stage
Count: 8.06 100
Stage Param. Count: 1.11 10
CRT
CRT
20-
CRT
- Sasdin
20
Besaline
Saseing
gFID
10
5-
10
1013
10 Train FLOPs
1036
2007
Train FLOPs
Train FLOPs
Stage Param. Count: 2.12 x 10
Stage 2 Param. Count: 3.43 x 10
Stage Param. Count: 5.17 10
Saseine
Baseline
Bsselne
10
10
qFID
gFID
5
3.0
3,0
3.0
2.5
10:4
10
10
10
1046
10
10
Train FLOPs
Train FLOPs
Train FLOPs
Stage 2 Param. Count: 7.75 x 10
10
Basaiine
6h 3.0
2.5
20
1010
Train FLOPs
Figure 13: Individual model scale comparison for Figures 4 and 7 Here we show how stage 2 gFID scales with respect to train FI.OPs with both the CRT and baseline tokenizer. We see that with CRT tokens, stage learns faster and also ends up more performant across all model sizes, especially at smaller scales (note the log-scaled y-axis).
G Qualitative Examples
Figures 15 19 show non-cherrypicked generation examples from our best model CRT ARR-777M...
They demonstrate a high degree of diversity and image quality across a wide array of classes. Fig- ure 29 shows selected comparisons with and without CRT, constructed so that overall image structure comparable. CRT demonstrates better long-range and structural consistency.
H Text-to-image results
The main body of this work focused on ImageNet, since it is an established benchmark with good im- age diversity. However, much of the large-scale work on image generation focuses on text-to-image
21

---

Table 5: Text-to-image results on BLIP-3o. Re-using the CRT (opt variant) and baseline tokeniz- ers, we compare results on the BLIP-3o pre-training dataset for text-to- toiag performance (775M auto-regressive model for 600k iterations). We show improvements on both image-text alignment and distribution matching.
Tokenizer
gFID
CLIPScore
Baseline
4.61
0.33
CRT (ours)
4.15
0.36
GRT22/4R-112M
CRT 4R-3400
AR-77A
Figure 14: Qualitative examples (best viewed zoomed in). Random, non-curated samples from CRT CRTopt-AR op models of increasing scale (11IM to 775M parameters, ImageNet 256x256), Larger models demonstrate improved visual fidelity and coherence across diverse subjects including an- imals, landscapes, and objects across scales. CRT allows for even small models to produce high quality outputs.
generation. To show the generality of method, show results in the text-to-image domain. For this experiment, we took the BLIP-3o long-caption [4] pre-training dataset (27M image-text pairs vs 1.2M ImageNet images), tokenized the with OpenAI's CLIP L/14 text tower [42), and trained an auto-regressive image generation model conditioned on these text embeddings. This dataset is diverse, being combination of webcrawled images and high-quality data from JourneyDB, We trained our XL sized model (775M) for 600k iterations at batch size 256 using the CRT (opt variant) and baseline tokenizers from our study. The results from this experiment are in Table [ We use CLIPScore [2I] (higher is better) with OpenAl CLIP B/32 [42] to measure image-tex alignment. We see a solid improvement with our method, even though we are re-using the image tokenizer from our ImageNet experiments and this dataset is thus OOD.
Extended Related Work
Evolution of visual tokenization. The current paradigm of discrete visual tokenizers came from VQ-VAE [55], originally introduced as method for discrete latent representation learning in contin- uous domains such as image and audio. was introduced to prevent the posterior collapse problem in VAEs by separating the compression and generation phases of training. Section 4.2 of (55] notes
that their loss function leads to blurry reconstructions, and the authors suggest including perceptual metrics (metrics that align with human judgment of similarity) in the loss function.
[14] explores
this idea with VQGAN, which introduces a perceptual loss based on L.TPS [67. and GAN loss add realistic textures. Since the advent of VOGAN, there have been numerous attempts at improving visual tokenizers (62) 19 69 68 66 57 28, 45 588 40 70 44). Some works focus on improvements including ViT- VQGAN 62). which replaces CNN layers with ViT layers, and MoVQGAN [69] which adds adaptive instance normalization layers in the CNN decoder. Other works focus on mitigating "codebook collapse" (26) vector quantization (VQ) 18 by introduc- ing novel quantization schemes, for example by replacing dead codes with encoder outputs 99. We refer the reader to (26 for a thorough review of these methods. Other works replace VQ entirely with scalar quantization techniques [37), and demonstrates increased codebook utilization compared
VQ: for example, MAG-VIT (64 36 quantizes each dimension values and adds entropy loss
to encourage codebook utilization.
22

---

Figure 15: Non-cherrypicked generation examples from oR (ImageNet
256x256). CFG scale 4.0, class index 250, class name Siberian Husky
Figure 16: Non-cherrypicked generation examples from CRT CRT ARR-755 (ImageNet 256x256). CFG scale = 4.0, class index 972, class name cliff
J VQVAE comparisons
In Figure 30 we show detailed comparisons between reconstructions from baseline and various CRT configurations. CRT and baseline look quite similar, while CRT with CE (instead of 1.2) demonstrates some noticeable blocky artifacts as a result of the strength of regularization.
23

---

Figure 17: Non-cherrypicked generation examples from CRT AR (ImageNet
256x256), CFG scale = 4.0, class index 207, class name Golden Retriever
Figure 18: Non-cherrypicked generation examples from CRT (ImageNet
256x256). CFG scale = 4.0, class index 90, class name lorikeet
Figure 19: Non-cherrypicked generation examples from CRT (ImageNet
256x256). CFG scale 4.0, class index 29, class name axolotl
24

---

Figure 20: Non-cherrypicked generation examples from CRT ORT (ImageNet
256x256). CFG scale = 4.0, class index 297, class name sloth bear
Figure 21: Non-cherrypicked generation examples from CRT (ImageNet
256x256). CFG scale = 4.0, class index 248, class name husky
Figure 22: Non-cherrypicked generation examples from ARR-775M (ImageNet 256x256). CFG scale 4.0, class index 973, class name coral reef
25

---

Figure 23: Non-cherrypicked generation examples from CRT -R (ImageNet
256x256), CFG scale = 4.0, class index 993, class name gyromitra
Figure 24: Non-cherrypicked generation examples from
(ImageNet
256x256). CFG scale = 4.0, class index 388, class name giant panda
Figure 25: Non-cherrypicked generation examples from CRT CRT (ImageNet
256x256). CFG scale 4.0, class index 387, class name red panda
26

---

10 BB
LINIEE
ERIEE
GBA
220
LOGS
Figure 26: Non-cherrypicked generation examples from CRT AR- (ImageNet
256x256), CFG scale 4.0, class index 417, class name balloon
Figure 27: Non-cherrypicked generation examples from CRT (ImageNet
256x256). CFG scale = 4.0, class index 279, class name Arctic fox
Figure 28: Non-cherrypicked generation examples from CRT -RR (ImageNet
256x256). CFG scale 4.0, class index 291, class name lion
27

---

With CRT
Baseline
Figure 29: Selected generation comparisons with and without using the CRT tokenizer. Here we show direct comparisons between stage 2 generations with and without using the CRT tokenizer. CRT results greater long-range structural coherence and higher quality samples. To generate these direct comparisons, we provide the first 32 (of 256) tokens from a validation example for context to the stage auto-regressive model and filter for image similarity using CLIP B/32 embeddings. For more non-curated generations with our model, see Figure 14 and Appendix
Original
Deseline mDD)
CAT
with CE FIID)
Figure 30: Reconstruction comparison (best viewed zoomed in). We show that CRT applied to
baseline only minimally affects distortion, while CRT with CE introduces more noticeable blocky artifacts in reconstruction due to the strength of regularization.
28


Reviews:
[{"review_id": "wA96tBfcF4", "reviewer": "Reviewer_SM8A", "summary": "The submission has two main contributions.\n\nThe first is a thorough study on the design space of discrete image tokenizers (stage 1), and what effect it has on downstream generative modeling with vanilla autoregressive models (stage 2). To that end, the paper sets up scaling laws and studies the relation of tokenizer vocabulary size and token sequence length with stage 2 model scale & compute.\n\nThe second main contribution is a training modification on 2D tokenizers that regularizes the learned latent space to be more amenable to generative modeling with AR models. The authors call this \"Causally Regularized Tokenization\" (CRT) and it is performed by co-training a tiny AR model with the tokenizer whose objective it is to perform next-token prediction on the tokenizer latents. By doing that, the tokens are trained to be more predictable in an AR manner. The authors observe a slight reduction in reconstruction performance, but at the benefit of significantly stronger generative performance, with better scaling behavior compared to the non-CRT baseline.", "questions": "- The authors note that raster-scan order has been observed to work best for AR generation, but how would the use of CRT change this? Is this simply an inductive bias that may be arbitrarily changed by regularizing the tokenizer to encode information in a different order?\n- Fig 3: It may not be practical for stage 2 training, but it would be interesting to see the trendlines for scaling the codebook size extended beyond 2^17. It looks like for PSNR, scaling the codebook size may scale better.", "limitations": "The authors have adequately addressed potential limitations.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- The investigations into how a tokenizer's codebook size and token sequence length should be chosen for different sizes and compute budgets of downstream generative models are highly interesting and well-designed. \n- The design of CRT is simple and effective, and incurs a minimal compute overhead. The efficiency improvements / improved scaling law slope, and the demonstration of outperforming LlamaGen with smaller models and shorter sequence lengths are impressive.\n- The authors seem to have put great effort to taking into account the compute overhead of various interventions and to compute-match the various experiments. In general, the experimental setup and procedure appears very thorough and thought-through.\n- The ablations on CRT are thorough and answered all the key questions I had w.r.t. loss balancing, size of the AR regularizer, loss function, etc.\n- The paper shows sufficient training details to be able to reproduce the experiments and method\n- The paper's writing is very clear, the sections are well-formatted, and the figures are easy to parse. The paper is packed with interesting and useful insights, and it was a joy to read. It also challenges some common assumptions from previous works, which would be useful to study in more detail.\n\nWeaknesses:\n- Sec 3.4: The authors note that more compressed sequences are generally more compute optimal, but how much of this may be due to the tokenizer operating on out-of-distribution resolutions? It is noted that for these experiments, the same tokenizer is used, but at higher resolutions than what it was trained at. Would the results change if tokenizers were trained at those resolutions? Would we see further improvements for smaller models if the sequence length is further decreased, e.g. to 14*14 or below (with frozen or specially trained tokenizers)?\n- The paper focuses mostly on class-conditional ImageNet generation, which to me is one of its main weaknesses. There are some results on LSUN, but as for IN1K generation, the benchmark is somewhat close to saturation and susceptible to gaming if one tries to just overfit a model to the train set (since gFID is measured against it). The paper's findings are valuable, but could be significantly stronger if they were demonstrated to A) hold on more difficult datasets, B) on more nuanced tasks like text-to-image where the conditioning alignment could be measured, and C) in a data regime that is not data-limited, unlike ImageNet-1k. \n- Besides the main contributions, the VQ training recipe proposed by the authors seems to outperform simpler baselines like FSQ. This is somewhat surprising, as the VQ recipe in this paper seems similar to the one studied in FSQ, and it would be useful to investigate this further and show if the proposed CRT also works well for LFQ.\n- The paper could benefit from including a discussion about related work like JetFormer [1] that jointly trains a tokenizer with an AR model, as well as alternative methods of regularizing tokenizers with a 1D causal structure, like SEED [2], ElasticTok [3], One-D-Piece [4], FlexTok [5], or Semanticist [6]. I note that the submission already includes a discussion about LARP's use of AR regularization.\n\n[1] JetFormer: An Autoregressive Generative Model of Raw Images and Text, Tschannen et al. 2024\n\n[2] Planting a SEED of Vision in Large Language Model, Ge et al. 2023\n\n[3] ElasticTok: Adaptive Tokenization for Image and Video, Yan et al. 2024\n\n[4] One-D-Piece: Image Tokenizer Meets Quality-Controllable Compression, Miwa et al. 2025\n\n[5] FlexTok: Resampling Images into 1D Token Sequences of Flexible Length, Bachmann et al. 2025\n\n[6] \"Principal Components\" Enable A New Language of Images, Wen et al. 2025", "was_revised": true, "final_justification": "My initial review for this submission was positive and the author's rebuttal addressed most of my concerns, and even provided additional insights (especially with respect to the VQ training recipe and the order in which CRT is applied) which could further strengthen the paper. My main issues with the submission were that the main evaluation was on ImageNet generation, but the authors showed that CRT is effective for text-to-image generation on a more diverse and large-scale dataset too. For these reasons I will further increase my score.", "rebuttal": "We thank the reviewer for their in detailed and insightful review. We were particularly gratified that the reviewer found our paper \"a joy to read\" and that \"the experimental setup and procedure appears very thorough and thought-through\". We address the reviewers constructive feedback/criticisms below.\n\n### Addressing Weaknesses\n\n**The authors note that more compressed sequences are generally more compute optimal, but how much of this may be due to the tokenizer operating on out-of-distribution resolutions?**\n\nWe thank the reviewer for pointing this out! In order to address the point of operating OOD resolutions, we have results for:\n- _Tokenizers greater than 16x16 resolution_: We do not see significant improvement in rFID/other distortion metrics when training with natively higher input resolution, nor do we see stage 2 improvements. We ran these experiments with a 211M parameter stage 2 model for 500k iterations (batch size 256).\n| Tokens per Image | rFID / gFID | Native resolution training |\n|------------------|-------------|----------------------------|\n| 400 | 1.30 / 3.76 | no |\n| 400 | 1.21 / 3.72 | yes |\n| 576 | 0.99 / 3.61 | no |\n| 576 | 0.93 / 3.66 | yes |\n- _Tokenizers less than 16x16 resolution_: It is likely possible to see some improvement since training a lower resolution tokenizer often damages PSNR; however, we note that training such a tokenizer requires architectural modifications in order to avoid resizing the image below 256x256. This introduces an architectural confound outside the scope of this paper. We will include this comparison when the model is finished training. We thank the reviewer for their great observation.\n\n\n**The paper focuses mostly on class-conditional ImageNet generation ... The paper's findings are valuable, but could be significantly stronger if they were demonstrated to A) hold on more difficult datasets, B) on more nuanced tasks like text-to-image where the conditioning alignment could be measured, and C) in a data regime that is not data-limited, unlike ImageNet-1k.**\n\nWe thank the reviewer for pointing this out. To alleviate this concern and supplement our results, we ran a text-to-image experiment. For this experiment, we took the BLIP-3o long-caption [1] pre-training dataset (27M image-text pairs vs 1.2M ImageNet images), tokenized the text with OpenAI's CLIP L/14 text tower [2], and trained a auto-regressive image generation model conditioned on these text embeddings. This dataset is diverse, being a combination of webcrawled images and high-quality data from JourneyDB. We trained our XL sized model (775M) for 600k iterations at batch size 256. The results from this experiment are in the table below:\n\n| Tokenizer | gFID | CLIPScore |\n| --------------- | --------------- | --------------- |\n| Baseline | 4.61 | 0.33 |\n| CRT (ours) | 4.15 | 0.36 |\n\n\nWe use CLIPScore [3] (higher is better) with OpenAI CLIP B/32 to measure image-text alignment. We see a solid improvement with our method, even though we are re-using the image tokenizer from our ImageNet experiments and this dataset is thus OOD. We will include full details of this experiment in the final revision.\n\n\n**Besides the main contributions, the VQ training recipe proposed by the authors seems to outperform simpler baselines like FSQ ... it would be useful to investigate this further and show if the proposed CRT also works well for LFQ.**\n\nThis point has two parts, exploring the VQ recipe and discussion of LFQ.\n\n- **VQ recipe exploration.** The performance of the VQ recipe was also surprising to us. Our VQ recipe is similar to that of ViT-VQGAN and LlamaGen but ended up being more scalable than both. We credit this to a long learning rate warmup period, which is the only substantial deviation from the ViT-VQGAN/LlamaGen settings. [4] finds that long learning rate warmup periods lend greater stability through training runs even when instabilities appear far after the warmup period. It's possible this has a similar effect here. Not only is our codebook utilization ~100%, Figure 9 (center left) implies that the distribution of code usage is close to uniform _without any entropy losses_. This is a deviation from the result in FSQ, where both FSQ and VQ are substantially below the uniform compression cost threshold (see Figure 3 bottom right in FSQ). We show a the effect with and without learning rate warmup below for our setting, where we see a clear improvement with warmup across codebook sizes:\n\n\n| Codebook Size | With Warmup (rFID) | Without Warmup (rFID) |\n| --------------- | --------------- | --------------- |\n| $2^{12}$ | **2.60** | 3.04 |\n| $2^{13}$ | **2.53** | 2.95 |\n| $2^{14}$ | **2.21** | 2.24 |\n| $2^{15}$ | **1.91** | 2.29 |\n| $2^{16}$ | **1.76** | 2.45 |\n\n\n - **Regarding LFQ.** We have started training a tokenizer with LFQ, but properly training LFQ requires carefully tuning entropy loss and commitment loss correctly, and therefore it is not finished as of the rebuttal deadline. We will include this comparison when it is complete. We speculate that it will not outperform our VQ recipe in terms of rFID, as we are already achieving almost uniform codebook utilization (Figure 9). \n\n**The paper could benefit from a discussion of JetFormer and more 1D tokenization works.**\n\nWe thank the reviewer for pointing out these important works. We will include a discussion of them in the final revision under the 1D tokenizer section in our related work.\n\n\n### Addressing Questions\n\n**The authors note that raster-scan order has been observed to work best for AR generation, but how would the use of CRT change this?**\n\nWe have done experiments with alternate token orders (Hilbert space filling curve, spiral in, spiral out, snake order from top left) at small scale (<200M stage 2 parameters). In all of these experiments, raster scan ended up being the best empirically, often substantially (>1 gFID). When applying CRT to other orderings (at small scale), CRT would improve performance but not enough to overcome the absolute difference between raster-scan and worse orderings. We are not the first to note the superiority of raster-scan. Indeed a comparison exists in [5] in Figure 47 (although with a sliding window attention pattern instead of full causal). It is possible that a more flexible base architecture than that of VQGAN would allow for more flexible re-ordering. However our implementations of ViT-VQGAN underperformed VQGAN when compute-matched, so we focused on VQGAN for this study. \n\n\n**Fig 3: It may not be practical for stage 2 training, but it would be interesting to see the trendlines for scaling the codebook size extended beyond 2^17. It looks like for PSNR, scaling the codebook size may scale better.**\n\nThis is an interesting observation. In response, we trained a tokenizer with $2^{18}$ codes. Note that if we trained a stage 2 model, the embedding parameters for a 770M GPT model would be ~650M parameters, which as the reviewer notes is cumbersome. We actually do see a slight improvement over increasing tokens per image (rFID: 1.72, PSNR: 21.6, MS-SSIM: 0.695), however it's not great enough to justify the added compute. To stabilize this run during the GAN loss phase, we had to introduce warmup when increasing the weight of the discriminator loss (over 2000 iterations). We did not have to do this for codebook size 131k.\n\n\\[1\\] BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset, Chen et al 2025\n\n\\[2\\] Learning Transferable Visual Models From Natural Language Supervision, Radford et al 2021\n\n\\[3\\] CLIPScore: A Reference-free Evaluation Metric for Image Captioning, Hessel et al 2021\n\n[4] Small-scale proxies for large-scale Transformer training instabilities, Wortsman et al 2023\n\n[5] Taming Transformers for High-Resolution Image Synthesis, Esser et al 2020."}, {"review_id": "XpYe4uKhIX", "reviewer": "Reviewer_DoZ7", "summary": "This paper studies the fundamental trade-off between image compression (stage 1) and generative modeling (stage 2) in two-stage image generation pipelines. Specifically, it investigates how the design of the tokenizer—namely the number of tokens and the codebook size—interacts with compute scaling laws. Through comprehensive empirical analysis, the authors find that under limited compute, high compression (i.e., fewer tokens and smaller codebooks) is beneficial, as it reduces reconstruction quality (rFID) but improves generative performance (gFID). To better navigate this trade-off, the authors introduce Causally Regularized Tokenization (CRT), a simple yet effective method that incorporates the autoregressive inductive bias of stage-2 decoders into the stage-1 tokenizer. CRT achieves significant efficiency gains (2–3×) and matches the performance of LlamaGen-3B with far fewer tokens and reduced model complexity.", "questions": "1. The paper states (line 158) that models on the training compute Pareto frontier of validation loss are also on the gFID Pareto frontier. How can this be concluded directly from Figure 4? Can you elaborate?\n\n2. In line 220, you mention reducing training epochs by two to control training FLOPs. Was model convergence affected in those cases?\n\n3. In Figure 7, CRT's rFID appears to be 2.36, but why the fitted curve has an intercept of 2.21?\n\n4. CRT relies on the autoregressive modeling bias to guide tokenizer regularization. For other modeling paradigms (e.g., MaskGIT, diffusion), what inductive biases would be appropriate? Would CRT need to be redesigned accordingly? Any insights would be appreciated.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:** \n1. The paper provides lots of evidence of how stage-1 tokenization choices (number of tokens and codebook size) affect stage-2 modeling performance, and how these trade-offs evolve under different compute budgets.\n\n2. The proposed CRT mechanism is conceptually straightforward but effective, leveraging the AR modeling bias to regularize tokenizer outputs for better downstream performance.\n\n3. The authors conduct targeted ablations to isolate the effect of key CRT components (e.g., applying the L2 loss before quantization), offering actionable insights into its design.\n\n**Weakness:** \n\n1. The analysis is limited to a fixed tokenizer architecture (VQ-GAN) and autoregressive stage-2 decoders. Interestingly, the experiments suggest that the specific choice of token count and codebook size does not strongly affect the compute-scaling behavior. This raises the question of whether other, potentially more impactful factors—such as the type of tokenizer or the choice of decoder architecture—might play a larger role in shaping the trade-off. However, this dimension remains unexplored. \n\n2. CRT explicitly sacrifices reconstruction quality for better modeling ease. However, existing scaling laws suggest that rFID becomes a bottleneck at large model scales. Since the experiments are limited to models up to 800M parameters, it remains unclear whether CRT continues to help, plateaus, or even hurts performance in the high-capacity regime. \n\n3. The first half of the paper focuses on empirical analysis of scaling laws and tokenizer design, while the second half introduces CRT as a solution. However, the connection between these two parts feels somewhat loose—does the first half merely identify a design choice (fewer tokens), or does it provide deeper insight that motivates CRT?", "was_revised": true, "final_justification": "After reading the rebuttal and discussion, I have increased my score to a weak accept (5). The authors have adequately addressed my concerns regarding the scope of tokenizer choices, the effect of CRT near saturation, and the connection between the scaling analysis and the proposed method. The paper offers valuable empirical insights and a practical method (CRT) with strong efficiency gains. While some writing and structural issues remain—particularly the somewhat loose connection between the first half (scaling analysis) and the second half (CRT proposal)—the contributions are solid and of interest to the community.", "rebuttal": "### Addressing Weaknesses\n\n**The analysis is limited to a fixed tokenizer architecture (VQ-GAN) and autoregressive stage-2 decoders.**\n\nWe focused on VQGAN as it is the current SOTA tokenizer architecture for auto-regressive image generation. We note that the VQGAN architecture is still used for top discrete token models and has been dominant since it was proposed in \\[1\\]. We agree that tokenizer architecture would be an interesting avenue to study, however given the expense and depth of our scaling studies, we leave this axis of study to future work. We hypothesize more flexible (e.g. transformer-based) architectures can potentially gain more from causal regularization.  \n\n**CRT explicitly sacrifices reconstruction quality for better modeling ease...  it remains unclear whether CRT continues to help, plateaus, or even hurts performance in the high-capacity regime**\n\nWe agree that very close to saturation, reconstruction performance becomes important. We point out the following:\n\n1. CRT only harms reconstruction performance minimally (2.21 vs 2.36 rFID), which is hard to differentiate visually. \n2. rFID is not always the lower bound of generation model performance, although it usually serves as a good proxy. In scaling laws, $L_{min}$ (the scaling law's lower bound), $\\alpha$ (the scaling exponent), and $\\lambda$ (the intercept) are fit together [2, 3]. In our case, when we fit $L_{min}$ for gFID curves it generally matched rFID, so we used the convention of rFID as $L_{min}$. In two cases this was not true: 1. In Figure 6 (far left, 1k codebook size) and 2. In the CRT scaling law. In Figure 6 (far left) we used the fitted $L_{min}$ (2.7 instead of 3.0 rFID), and in the CRT scaling law, the fitted $L_{min} = 2.2$ which essentially matches the baseline tokenizer's rFID. Therefore, scaling laws suggest that CRT and the baseline tokenizer have the same saturation point. We will make this clear in the final revision.\n3. We introduce the $CRT_{opt}$ recipe in the paper for the very purpose of optimizing for both absolute reconstruction performance and causal regularization. This improves performance near saturation (2.18 vs 2.35 gFID). \n\n\n**However, the connection between \\[scaling laws and CRT\\] does the first half merely identify a design choice (fewer tokens), or does it provide deeper insight that motivates CRT?**\n\nWe thank the reviewer for this constructive observation. The first half of the paper shows that increasing bits per pixel above 16k codebook size and 256 tokens per image generally makes scaling worse. Further, while reducing to 1k codebook size improves the scaling law marginally, _explicitly_ reducing the reconstruction capacity to such a degree greatly harms generation performance close to saturation. This motivates us to look for a rate-distortion trade-off which does not directly inhibit the bottleneck size, of which CRT is one. Section 4 analysis expands on this concept. We will make this explicit in the text, which should improve the overall flow of the paper.\n\n\n### Addressing Questions\n\n**The paper states (line 158) that models on the training compute Pareto frontier of validation loss are also on the gFID Pareto frontier. How can this be concluded directly from Figure 4? Can you elaborate?**\n\nThis fact can be concluded directly from the figure, however it is difficult and we will improve the clarity in the final revision. To see this in the figure itself, each line is a specific parameter scale of increasing compute, so points in the validation loss curve directly correspond to points in the gFID curve. The pareto frontier are minimum points at a specific compute interval. For example, you see that in the validation loss curve, the last 6 points of the 775M parameter model lie on the pareto frontier. In the gFID curve this is the same for that model scale. To make this clear in the final revision, we will use stars to denote validation loss points which are on the pareto frontier and propagate those to the gFID curve.\n\n**In line 220, you mention reducing training epochs by two to control training FLOPs. Was model convergence affected in those cases?**\n\nModel convergence was not affected to any real degree. We trained versions at 38 and 40 epochs, and their rFIDs were 2.36 and 2.34 respectively, which we is within training noise.\n\n**In Figure 7, CRT's rFID appears to be 2.36, but why the fitted curve has an intercept of 2.21?**\n\nWe expand on this above (rebuttal point 2 with regards to weakness point 2). Note that varying $L_{min}$ between 2.2 and 2.3 for CRT does _not_ change the scaling exponent, so our conclusions remain the same. We will make this nuance (fitting $L_{min}$ vs rFID lower bound) clear in the text. \n\n**CRT relies on the autoregressive modeling bias to guide tokenizer regularization. For other modeling paradigms (e.g., MaskGIT, diffusion), what inductive biases would be appropriate? Would CRT need to be redesigned accordingly? Any insights would be appreciated.**\n\nYes, CRT would need to be re-designed if the stage 2 modeling design was changed. This is an active area of research for us. Looking towards diffusion, generally the closer the latent distribution is to a gaussian normal, the easier it is to do flow matching (since all the marginal flows cancel and you are left with no flow). However, if the latent prior is completely normal, and there is no flow, then only VAE decoder matters for generation. Further, the latent space would have a very low signal to noise ratio and the reconstruction would be blurry. This is one of the fundamental trade-offs for diffusion. This trade-off partially already exists in literature with the KL regularization term for the VAE prior, which pushes the latent distribution towards an isotropic normal, but there are other ways to accomplish this. For example, linearly interpolating the latent with noise during auto-encoder training accomplishes the same thing while allowing for a more controllable signal-to-noise ratio. Ultimately, for any stage 2 model, to introduce such an inductive bias into the stage 1 model requires analyzing the specific processes and assumptions inherent to stage 2 sampling.\n\n[1] Taming Transformers for High-Resolution Image Synthesis, Esser et al 2020.\n\n[2] Training Compute-Optimal Large Language Models, Hoffman et al 2022.\n\n[3] Scaling Laws for Neural Language Models, Kaplan et al 2020."}, {"review_id": "S4ATDVnAVm", "reviewer": "Reviewer_BfTw", "summary": "This paper presents a thorough scaling exploration of AR image generation models based on discrete tokens. The authors highlight the trade-off between the reconstruction performance of autoencoders and the generation performance of AR models. They further demonstrate that increasing computational scale can help overcome this trade-off and push the performance boundaries of the model. Additionally, the paper introduces Causally Regularized Tokenization, which incorporates a next-token prediction loss to instill an inductive bias into the tokenizer. This approach leads to improved generation performance.", "questions": "Please see the weaknesses.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- This paper provides extensive experiments demonstrating that: (1) a trade-off exists between reconstruction (or compression) performance and generation quality; (2) scaling compute helps push the model's limits by analyzing this trade-off through the lens of scaling laws; and (3) incorporating an autoregressive inductive bias improves generation performance.\n- These experimental results offer valuable insights for designing AR-based generative models.\n- The proposed CRT and its optimized version are intuitively sound and empirically effective.\n- The paper is well-written, logically structured, and supported by clear figures.\n\n**Weaknesses**\n\n- The authors could provide a qualitative comparison of the reconstructed results between the baseline VQGAN, CRT, and CRT with CE loss, as the rFID differences are quite subtle and may not translate into noticeable visual differences.\n- It would be better if the authors could conduct some T2I generation experiments, which would make the work more generalizable.", "was_revised": true, "final_justification": "The authors address my concerns by:\n\n- Committing to include qualitative visual comparisons in the final revision, explaining that larger rFID gaps cause more visible degradation, especially in small high-frequency regions.\n- Conducting additional large-scale T2I experiments, demonstrating gains in gFID and CLIPScore.\n\nTherefore, I keep my initial rating.", "rebuttal": "We thank the reviewer for their insightful review. We are glad that the reviewer appreciates our extensive experiments and writing. Below we respond to constructive feedback.\n\n**The authors could provide a qualitative comparison of the reconstructed results between the baseline VQGAN, CRT, and CRT with CE loss, as the rFID differences are quite subtle and may not translate into noticeable visual differences.**\n\nWe thank the reviewer for this suggestion. Given the limitations around sharing visual media during the rebuttal period, we commit to showing these comparisons in the final revision. Qualitatively, the difference between reconstruction performance of VQGAN and VQGAN + CRT is hard to discern visually, and this is reflected in their rFID (2.21 vs 2.36 with CRT). We see this as an advantage of our method, as it mitigates damage to visual reconstrction while improving scaling. For larger gaps, e.g. 2.2 to 2.6 (VQGAN + CE CRT), we see a greater degradation that becomes more noticeable. This is prevalent in small high frequency patches (e.g. grass or tree branches) and in the bottom right corner of images (as the AR bias is applied in a raster-scan).\n\n**It would be better if the authors could conduct some T2I generation experiments, which would make the work more generalizable.**\n\nWe thank the reviewer for pointing this out. To alleviate this concern and supplement our results, we ran a text-to-image experiment. For this experiment, we took the BLIP-3o long-caption [1] pre-training dataset (27M image-text pairs vs 1.2M ImageNet images), tokenized the text with OpenAI's CLIP L/14 text tower [2], and trained a auto-regressive image generation model conditioned on these text embeddings. This dataset is diverse, being a combination of webcrawled images and high-quality data from JourneyDB. We trained our XL sized model (775M) for 600k iterations at batch size 256. The results from this experiment are in the table below:\n\n| Tokenizer | gFID | CLIPScore |\n| --------------- | --------------- | --------------- |\n| Baseline | 4.61 | 0.33 |\n| CRT (ours) | 4.15 | 0.36 |\n\n\nWe use CLIPScore [3] (higher is better) with OpenAI CLIP B/32 to measure image-text alignment. We see a solid improvement with our method, even though we are re-using the image tokenizer from our ImageNet experiments and this dataset is thus OOD. We will include full details of this experiment in the final revision.\n\n[1] BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset, Chen et al 2025\n\n[2] Learning Transferable Visual Models From Natural Language Supervision, Radford et al 2021\n\n[3] CLIPScore: A Reference-free Evaluation Metric for Image Captioning, Hessel et al. 2021"}, {"review_id": "slQ9GKJRsv", "reviewer": "Reviewer_pH7h", "summary": "In this paper, the authors systematically investigate the trade-off between compression and generation, including neural scaling, sequence length scaling and codebook size scaling. Based on these analysis, the authors shows that the ideal amount of image compression varies with generation model capacity. In addition, they also introduce Causally Regularized Tokenization (CRT) to train tokenizers with a causal inductive bias. CRT improves generation performance across ImageNet and LSUN.", "questions": "See Weakness.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n+ This paper provides a systematically investigation about the trade-off between compression and generation. The analysis on sequence length scaling and codebook size scaling is somewhat new.\n+ This paper imbues the causal inductive bias of generation model into the tokenizer by  Causally Regularized Tokenization (CRT). This simple causal regularization boost generation performance across various datasets.\n\nWeaknesses\n- The link between the systematic investigation and CRT is not very strong, which makes their combination in one paper feel somewhat disjointed.\n-  Although this paper provides systematic investigation about the trade-off between compression and generation, these investigations are not absolutely new and insightful. Neural scaling is widely studied in previous works, like VAR. Sequence length scaling and codebook size scaling are implicitly studied in existing works, like LlamaGen.\n- Limited experiments about CRT. In this paper, the effectiveness of CRT is only validated on a codebook of size 16k and 256 tokens per image. CRT is designed for image tokenizers. It is essential to provide more experiments to show effectiveness with various codebook size and tokens per image.\n- Missing discussion about Figure 9 (center right). In the main text, there is no discussion about skewness vs. vocabulary size.  \n- Type Errors. 1) line 156, higher -> lower; 2) line 199, (1.7 vs 2.21 vs 3.0 rFID) - > (3.0 vs 2.21 vs 1.7 rFID).", "was_revised": true, "final_justification": "My concerns have been resolved, thus, I raise my score to Borderline accept.", "rebuttal": "### Addressing Weaknesses\n\n**The link between the systematic investigation and CRT is not very strong, which makes their combination in one paper feel somewhat disjointed.**\n\nThe first half of the paper shows that increasing bits per pixel above 16k codebook size and 256 tokens per image generally makes scaling worse. Further, while reducing to 1k codebook size improves the scaling law marginally, explicitly reducing the reconstruction capacity to such a degree greatly harms generation performance close to saturation. This motivates us to look for a rate-distortion trade-off which does not directly inhibit the bottleneck size, of which CRT is one. Section 4 analysis expands on this concept. We will make this explicit in the text, which should improve the overall flow of the paper.\n\n**Although this paper provides systematic investigation about the trade-off between compression and generation, these investigations are not absolutely new and insightful. \\[e.g. VAR or LlamaGen\\]**\n\nWe would like to clarify that the _existence_ of scaling laws in this scenario is not surprising. Scaling laws have been widely studied and validated since Kaplan et al. 2020. The connection between scaling laws and the rate-distortion trade-off (which we measure as bits per pixel) is not studied in prior art. The reviewer points to VAR and LlamaGen. We address our novelty to these individual studies below:\n\n\n**VAR.** VAR includes scaling laws with regard to a proposed generation procedure (next-scale generation) with a specific, fixed stage 1 tokenizer. We study the effect changing the tokenizer's compression rate has on stage 2 scaling, a dimension not explored in VAR. \n\n**LlamaGen.** LlamaGen includes some results with increasing token count and stage 2 model capacity, however, they do not attempt to establish scaling laws and thus miss much of the interplay between compute scale and model capacity. They also do not study the effect of codebook size on scaling. Further, their codebook scaling does not get 100% codebook utilization or uniform codebook distribution, resulting in subpar scaling beyond 16k codebook size. For scaling laws to be valid, components of the pipeline have to be tuned and optimized. \n\nWe emphasize that the setup of studying stage 1 rate-distortion in connection to stage 2 compute scaling laws is, to our knowledge, unique in literature. Having factors related to stage 1 bottleneck capacity studied together in a unified, systematic setting is crucial for understanding the interplay of rate-distortion and stage 2 modeling.\n\n**Limited experiments about CRT. In this paper, the effectiveness of CRT is only validated on a codebook of size 16k and 256 tokens per image. CRT is designed for image tokenizers. It is essential to provide more experiments to show effectiveness with various codebook size and tokens per image.**\n\n\nWe thank the reviewer for this important point. Given the depth of our scaling law study, we decided to focus CRT on the setting which demonstrated the best trade-off between reconstruction and generation in the first part of our paper (16k codebook size with 256 tokens per image). We show additional results at multiple scales with more tokens per image below (400 and 576). We observe that across tokens per image configurations and stage-2 model size, CRT outperforms the baseline.\n\n**400 tokens per image**\n\n| Parameter Count | Iterations | Method | FID |\n| :--- | :--- | :--- | :--- |\n| **111M** | $7.5 \\times 10^5$ | Baseline | 5.3745 |\n| | $7.5 \\times 10^5$ | CRT | 4.7140 |\n| | $2.25 \\times 10^6$ | Baseline | 5.4305 |\n| | $2.25 \\times 10^6$ | CRT | 4.3158 |\n| **211M** | $7.5 \\times 10^5$ | Baseline | 3.7601 |\n| | $7.5 \\times 10^5$ | CRT | 3.3526 |\n| | $2.25 \\times 10^6$ | Baseline | 3.4438 |\n| | $2.25 \\times 10^6$ | CRT | 3.0547 |\n| **775M** | $2.25 \\times 10^6$ | Baseline | 2.4247 |\n| | $2.25 \\times 10^6$ | CRT | 2.2492 |\n\n\n**576 Tokens per Image**\n\nOf course, here is the table with the **Precision** and **Recall** columns removed.\n\n| Parameter Count | Iterations | Method | FID |\n| :--- | :--- | :--- | :--- |\n| **111M** | $7.5 \\times 10^5$ | Baseline | 6.2174 |\n| | $7.5 \\times 10^5$ | CRT | 4.7268 |\n| | $1.5 \\times 10^6$ | Baseline | 4.7259 |\n| | $1.5 \\times 10^6$ | CRT | 4.5559 |\n| **211M** | $7.5 \\times 10^5$ | Baseline | 3.6164 |\n| | $7.5 \\times 10^5$ | CRT | 3.3401 |\n| | $1.5 \\times 10^6$ | Baseline | 3.6185 |\n| | $1.5 \\times 10^6$ | CRT | 3.1098 |\n| **775M** | $7.5 \\times 10^5$ | Baseline | 2.4453 |\n| | $7.5 \\times 10^5$ | CRT | 2.1984 |\n| | $1.5 \\times 10^6$ | Baseline | 2.2381 |\n| | $1.5 \\times 10^6$ | CRT | 2.0993 |\n\n\nWe also ran $CRT_{opt}$ in settings with greater codebook size. We show those results below. \n\n| Parameter Count | Codebook Size | Method | FID |\n| :--- | :--- | :--- | :--- |\n| **111M** | 16384 | Baseline | 4.8837 |\n| | 16384 | $CRT_{opt}$ | 4.2323 |\n| | 131072 | Baseline | 5.0036 |\n| | 131072 | $CRT_{opt}$ | 4.4945 |\n| **550M** | 16384 | Baseline | 2.8497 |\n| | 16384 | $CRT_{opt}$ | 2.5793 |\n| | 131072 | Baseline | 2.8454 |\n| | 131072 | $CRT_{opt}$ | 2.4488 |\n| **775M** | 16384 | Baseline | 2.5452 |\n| | 16384 | $CRT_{opt}$ | 2.2079 |\n| | 131072 | Baseline | 2.7080 |\n| | 131072 | $CRT_{opt}$ | 2.1833 |\n\n\nIt is computationally infeasible for us to generate full scaling law studies for each specific setting, but we hope the results above are evidence that our conclusion is not specific to a particular tokenizer configuration.\n\n\n**Missing discussion about Figure 9 (center right). In the main text, there is no discussion about skewness vs. vocabulary size.**\n\n\nWe thank the reviewer for pointing this out. Skewness is another view of the concentration of codes property demonstrated by Figure 9 (center left). We compute skewness as $1 - \\frac{2^\\text{entropy per position}}{2^\\text{total entropy}}$, demonstrating that as the codebook size increases, so does codebook specialization per position. We will include this discussion in the final revision.\n\n\nIf we have adequately responded to the concerns with our experiments, we request that the reviewer raise their score."}]

Author rebuttal:

Meta-review: This paper studies the trade-off between image tokenization and generative modeling in two-stage image generation methods, and proposes a regularization term for tokenization to benefit the training of the second stage. Reviewers acknowledged the major contribution and valuable insights of the proposed method, while initially raising concerns regarding its experiment designs and missing T2I results.

After the rebuttal, the authors addressed most of the concerns, and all reviewers agreed to accept this paper. AC read all the reviews, author rebuttals, and the paper, and believes this is a strong paper and recommends acceptance.
Decision: Accept (spotlight)