Title: When Worse is Better: Navigating the Compression Generation Trade-off In Visual Tokenization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Representation Learning

Scaling Laws

Image Generation
Abstract: Current image generation methods are based on a two-stage training approach. In stage 1, an auto-encoder is trained to compress an image into a latent space; in stage 2, a generative model is trained to learn a distribution over that latent space. This reveals a fundamental trade-off, do we compress more aggressively to make the latent distribution easier for the stage 2 model to learn even if it makes reconstruction worse? We study this problem in the context of discrete, auto-regressive image generation. Through the lens of scaling laws, we show that smaller stage 2 models can benefit from more compressed stage 1 latents even if reconstruction performance worsens, demonstrating that generation modeling capacity plays a role in this trade-off. Diving deeper, we rigorously study the connection between compute scaling and the stage 1 rate-distortion trade-off. Next, we introduce Causally Regularized Tokenization (CRT), which uses knowledge of the stage 2 generation modeling procedure to embed useful inductive biases in stage 1 latents. This regularization improves stage 2 generation performance better by making the tokens easier to model without affecting the stage 1 compression rate and marginally affecting distortion: we are able to improve compute efficiency 2-3$\times$ over baseline. Finally, we use CRT with further optimizations to the visual tokenizer setup to result in a generative pipeline that matches LlamaGen-3B generation performance (2.18 FID) with half the tokens per image (256 vs. 576) and a fourth the total model parameters (775M vs. 3.1B) while using the same architecture and inference procedure.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors systematically investigate the trade-off between compression and generation, including neural scaling, sequence length scaling and codebook size scaling. Based on these analysis, the authors shows that the ideal amount of image compression varies with generation model capacity. In addition, they also introduce Causally Regularized Tokenization (CRT) to train tokenizers with a causal inductive bias. CRT improves generation performance across ImageNet and LSUN.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n+ This paper provides a systematically investigation about the trade-off between compression and generation. The analysis on sequence length scaling and codebook size scaling is somewhat new.\n+ This paper imbues the causal inductive bias of generation model into the tokenizer by  Causally Regularized Tokenization (CRT). This simple causal regularization boost generation performance across various datasets.\n\nWeaknesses\n- The link between the systematic investigation and CRT is not very strong, which makes their combination in one paper feel somewhat disjointed.\n-  Although this paper provides systematic investigation about the trade-off between compression and generation, these investigations are not absolutely new and insightful. Neural scaling is widely studied in previous works, like VAR. Sequence length scaling and codebook size scaling are implicitly studied in existing works, like LlamaGen.\n- Limited experiments about CRT. In this paper, the effectiveness of CRT is only validated on a codebook of size 16k and 256 tokens per image. CRT is designed for image tokenizers. It is essential to provide more experiments to show effectiveness with various codebook size and tokens per image.\n- Missing discussion about Figure 9 (center right). In the main text, there is no discussion about skewness vs. vocabulary size.  \n- Type Errors. 1) line 156, higher -> lower; 2) line 199, (1.7 vs 2.21 vs 3.0 rFID) - > (3.0 vs 2.21 vs 1.7 rFID).\n\nQUESTIONS:\nSee Weakness.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper presents a thorough scaling exploration of AR image generation models based on discrete tokens. The authors highlight the trade-off between the reconstruction performance of autoencoders and the generation performance of AR models. They further demonstrate that increasing computational scale can help overcome this trade-off and push the performance boundaries of the model. Additionally, the paper introduces Causally Regularized Tokenization, which incorporates a next-token prediction loss to instill an inductive bias into the tokenizer. This approach leads to improved generation performance.

STRENGTHS AND WEAKNESSES:
**Strengths**

- This paper provides extensive experiments demonstrating that: (1) a trade-off exists between reconstruction (or compression) performance and generation quality; (2) scaling compute helps push the model's limits by analyzing this trade-off through the lens of scaling laws; and (3) incorporating an autoregressive inductive bias improves generation performance.
- These experimental results offer valuable insights for designing AR-based generative models.
- The proposed CRT and its optimized version are intuitively sound and empirically effective.
- The paper is well-written, logically structured, and supported by clear figures.

**Weaknesses**

- The authors could provide a qualitative comparison of the reconstructed results between the baseline VQGAN, CRT, and CRT with CE loss, as the rFID differences are quite subtle and may not translate into noticeable visual differences.
- It would be better if the authors could conduct some T2I generation experiments, which would make the work more generalizable.

QUESTIONS:
Please see the weaknesses.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the fundamental trade-off between image compression (stage 1) and generative modeling (stage 2) in two-stage image generation pipelines. Specifically, it investigates how the design of the tokenizer—namely the number of tokens and the codebook size—interacts with compute scaling laws. Through comprehensive empirical analysis, the authors find that under limited compute, high compression (i.e., fewer tokens and smaller codebooks) is beneficial, as it reduces reconstruction quality (rFID) but improves generative performance (gFID). To better navigate this trade-off, the authors introduce Causally Regularized Tokenization (CRT), a simple yet effective method that incorporates the autoregressive inductive bias of stage-2 decoders into the stage-1 tokenizer. CRT achieves significant efficiency gains (2–3×) and matches the performance of LlamaGen-3B with far fewer tokens and reduced model complexity.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:** \n1. The paper provides lots of evidence of how stage-1 tokenization choices (number of tokens and codebook size) affect stage-2 modeling performance, and how these trade-offs evolve under different compute budgets.\n\n2. The proposed CRT mechanism is conceptually straightforward but effective, leveraging the AR modeling bias to regularize tokenizer outputs for better downstream performance.\n\n3. The authors conduct targeted ablations to isolate the effect of key CRT components (e.g., applying the L2 loss before quantization), offering actionable insights into its design.\n\n**Weakness:** \n\n1. The analysis is limited to a fixed tokenizer architecture (VQ-GAN) and autoregressive stage-2 decoders. Interestingly, the experiments suggest that the specific choice of token count and codebook size does not strongly affect the compute-scaling behavior. This raises the question of whether other, potentially more impactful factors—such as the type of tokenizer or the choice of decoder architecture—might play a larger role in shaping the trade-off. However, this dimension remains unexplored. \n\n2. CRT explicitly sacrifices reconstruction quality for better modeling ease. However, existing scaling laws suggest that rFID becomes a bottleneck at large model scales. Since the experiments are limited to models up to 800M parameters, it remains unclear whether CRT continues to help, plateaus, or even hurts performance in the high-capacity regime. \n\n3. The first half of the paper focuses on empirical analysis of scaling laws and tokenizer design, while the second half introduces CRT as a solution. However, the connection between these two parts feels somewhat loose—does the first half merely identify a design choice (fewer tokens), or does it provide deeper insight that motivates CRT?\n\nQUESTIONS:\n1. The paper states (line 158) that models on the training compute Pareto frontier of validation loss are also on the gFID Pareto frontier. How can this be concluded directly from Figure 4? Can you elaborate?\n\n2. In line 220, you mention reducing training epochs by two to control training FLOPs. Was model convergence affected in those cases?\n\n3. In Figure 7, CRT's rFID appears to be 2.36, but why the fitted curve has an intercept of 2.21?\n\n4. CRT relies on the autoregressive modeling bias to guide tokenizer regularization. For other modeling paradigms (e.g., MaskGIT, diffusion), what inductive biases would be appropriate? Would CRT need to be redesigned accordingly? Any insights would be appreciated.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe submission has two main contributions.\n\nThe first is a thorough study on the design space of discrete image tokenizers (stage 1), and what effect it has on downstream generative modeling with vanilla autoregressive models (stage 2). To that end, the paper sets up scaling laws and studies the relation of tokenizer vocabulary size and token sequence length with stage 2 model scale & compute.\n\nThe second main contribution is a training modification on 2D tokenizers that regularizes the learned latent space to be more amenable to generative modeling with AR models. The authors call this \"Causally Regularized Tokenization\" (CRT) and it is performed by co-training a tiny AR model with the tokenizer whose objective it is to perform next-token prediction on the tokenizer latents. By doing that, the tokens are trained to be more predictable in an AR manner. The authors observe a slight reduction in reconstruction performance, but at the benefit of significantly stronger generative performance, with better scaling behavior compared to the non-CRT baseline.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The investigations into how a tokenizer's codebook size and token sequence length should be chosen for different sizes and compute budgets of downstream generative models are highly interesting and well-designed. \n- The design of CRT is simple and effective, and incurs a minimal compute overhead. The efficiency improvements / improved scaling law slope, and the demonstration of outperforming LlamaGen with smaller models and shorter sequence lengths are impressive.\n- The authors seem to have put great effort to taking into account the compute overhead of various interventions and to compute-match the various experiments. In general, the experimental setup and procedure appears very thorough and thought-through.\n- The ablations on CRT are thorough and answered all the key questions I had w.r.t. loss balancing, size of the AR regularizer, loss function, etc.\n- The paper shows sufficient training details to be able to reproduce the experiments and method\n- The paper's writing is very clear, the sections are well-formatted, and the figures are easy to parse. The paper is packed with interesting and useful insights, and it was a joy to read. It also challenges some common assumptions from previous works, which would be useful to study in more detail.\n\nWeaknesses:\n- Sec 3.4: The authors note that more compressed sequences are generally more compute optimal, but how much of this may be due to the tokenizer operating on out-of-distribution resolutions? It is noted that for these experiments, the same tokenizer is used, but at higher resolutions than what it was trained at. Would the results change if tokenizers were trained at those resolutions? Would we see further improvements for smaller models if the sequence length is further decreased, e.g. to 14*14 or below (with frozen or specially trained tokenizers)?\n- The paper focuses mostly on class-conditional ImageNet generation, which to me is one of its main weaknesses. There are some results on LSUN, but as for IN1K generation, the benchmark is somewhat close to saturation and susceptible to gaming if one tries to just overfit a model to the train set (since gFID is measured against it). The paper's findings are valuable, but could be significantly stronger if they were demonstrated to A) hold on more difficult datasets, B) on more nuanced tasks like text-to-image where the conditioning alignment could be measured, and C) in a data regime that is not data-limited, unlike ImageNet-1k. \n- Besides the main contributions, the VQ training recipe proposed by the authors seems to outperform simpler baselines like FSQ. This is somewhat surprising, as the VQ recipe in this paper seems similar to the one studied in FSQ, and it would be useful to investigate this further and show if the proposed CRT also works well for LFQ.\n- The paper could benefit from including a discussion about related work like JetFormer [1] that jointly trains a tokenizer with an AR model, as well as alternative methods of regularizing tokenizers with a 1D causal structure, like SEED [2], ElasticTok [3], One-D-Piece [4], FlexTok [5], or Semanticist [6]. I note that the submission already includes a discussion about LARP's use of AR regularization.\n\n[1] JetFormer: An Autoregressive Generative Model of Raw Images and Text, Tschannen et al. 2024\n\n[2] Planting a SEED of Vision in Large Language Model, Ge et al. 2023\n\n[3] ElasticTok: Adaptive Tokenization for Image and Video, Yan et al. 2024\n\n[4] One-D-Piece: Image Tokenizer Meets Quality-Controllable Compression, Miwa et al. 2025\n\n[5] FlexTok: Resampling Images into 1D Token Sequences of Flexible Length, Bachmann et al. 2025\n\n[6] \"Principal Components\" Enable A New Language of Images, Wen et al. 2025\n\nQUESTIONS:\n- The authors note that raster-scan order has been observed to work best for AR generation, but how would the use of CRT change this? Is this simply an inductive bias that may be arbitrarily changed by regularizing the tokenizer to encode information in a different order?\n- Fig 3: It may not be practical for stage 2 training, but it would be interesting to see the trendlines for scaling the codebook size extended beyond 2^17. It looks like for PSNR, scaling the codebook size may scale better.\n\nLIMITATIONS:\nThe authors have adequately addressed potential limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}