Title: Understand Before You Generate: Self-Guided Training for Autoregressive Image Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Autoregressive Models

Image Generation
Abstract: Recent studies have demonstrated the importance of high-quality visual representations in image generation and have highlighted the limitations of generative models in image understanding. As a generative paradigm originally designed for natural language, autoregressive models face similar challenges. In this work, we present the first systematic investigation into the mechanisms of applying the next-token prediction paradigm to the visual domain. We identify three key properties that hinder the learning of high-level visual semantics: local and conditional dependence, inter-step semantic inconsistency, and spatial invariance deficiency. We show that these issues can be effectively addressed by introducing self-supervised objectives during training, leading to a novel training framework, Self-guided Training for AutoRegressive models (ST-AR). Without relying on pre-trained representation models, ST-AR significantly enhances the image understanding ability of autoregressive models and leads to improved generation quality. Specifically, ST-AR brings approximately 42% FID improvement for LlamaGen-L and 49% FID improvement for LlamaGen-XL, while maintaining the same sampling strategy.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the limitations of autoregressive (AR) models in image generation, particularly their poor visual understanding (based on LlamaGen). The authors identify three key issues: (1) over-reliance on local and conditional information (each next token relies on only spatial neighbors), (2) semantic inconsistency across generation steps (seeing all tokens have lower linear probe accuracy than seeing partialy), and (3) lack of spatial invariance in visual tokens (e.g., VQGAN encodes differently if image is shifted). To address these, they propose ST-AR (Self-guided Training for AR), a training framework that incorporates masked attention modeling and two contrastive learning objectives (inter-step and inter-view). These augment the standard AR next-token loss without modifying inference. ST-AR significantly improves both visual representation quality (e.g., linear probing accuracy from 21% to 55%) and generation quality (e.g., up to 49% FID improvement), all without relying on external pretrained models.\n\nSTRENGTHS AND WEAKNESSES:\nQuality:\n- The paper is technically sound, well-formulated, and experimentally robust. Very insightful analysis. The integration of masked attention and contrastive learning into AR training is effective and carefully designed. The experiments on ImageNet are extensive, showing strong improvements across multiple AR model sizes and metrics, with clear ablation studies validating each component.\n\nClarity:\n- The writing is clear and well-organized. Visualizations and architecture diagrams support understanding. The authors provide precise explanations of why applying attention masking and contrastive objectives.\n\nOriginality:\n- While the self-supervised tools used (e.g., contrastive loss, masked modeling) are established, their application to AR image generation is novel and well-motivated. The diagnosis of AR's semantic weaknesses is insightful and targeted design of ST-AR is a key original contribution.\n\nSignificance:\n- This work has broad implications for scaling AR models across modalities. Representation learning is always the key to better model, by improving representation learning within AR models, ST-AR makes them more viable competitors to diffusion models, especially given the efficiency and modularity of AR transformers.\n\nWeaknesses:\n\n1. ST-AR increases training complexity and compute cost (EMA teacher, additional losses).\n2. The careful composition of known components is a bit trivial and straightforward, although effective.\n3. Text-to-image generation experiment detail in appendix is not clear enough and perhaps can move some results to main text.\n4. Fig. 3 can be improved, the loss parts can't be understood easily without reading the text.\n\nQUESTIONS:\n1. I might have missed this information from the paper, can you quantify the training overhead introduced by ST-AR?\n2. Since contrastive loss is applied, what is the effect of batch size? \n3. Can you discuss how ST-AR be extended to other AR domains like 3d or video?\n4. I notice the high norm issue on the first patch, would adding register tokens help? (Also, whether adding this register token have any new insight?)\n\nLIMITATIONS:\nyes. computation cost but it is unavoidable.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the underlying limitations of the standard next-token prediction paradigm for autoregressive (AR) image generation. The authors systematically identify three key properties that hinder the learning of high-level visual semantics: (1) over-reliance on local and conditional dependence, (2) semantic inconsistency across generation steps, and (3) a lack of spatial invariance in the visual tokens.\nTo address these issues, the paper proposes ST-AR (Self-guided Training for AutoRegressive models), a novel training framework that integrates self-supervised learning objectives directly into the AR training process. Specifically, ST-AR employs masked image modeling on attention maps to encourage a wider receptive field and uses two forms of contrastive learning (inter-step and inter-view) to enforce semantic consistency. A key advantage of this approach is that it enhances the model's understanding and generation quality without requiring pre-trained representation models or altering the standard AR sampling strategy at inference time. Experiments on ImageNet show that ST-AR significantly improves both image understanding (linear probing accuracy) and generation quality (FID scores) across various model scales.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n\n1. This work systematically investigate into why autoregressive models struggle with visual semantics. The authors go beyond simply proposing a new method and provide a clear, well-supported diagnosis of the core problems. This principled analysis provides a strong foundation and clear motivation for the proposed ST-AR framework.\n\n2. The proposed ST-AR framework is a novel and clever integration of self-supervised learning techniques (MIM and contrastive learning) into the autoregressive training loop. \n\nWeakness\n\n1. The authors correctly identify \"increased training costs\" as a limitation in the conclusion. However, this is a significant practical concern that warrants a more detailed discussion. The ST-AR framework introduces a teacher network, requires processing multiple views of each image, and computes three additional loss terms. This will substantially increase both the computational and memory requirements for training compared to the baseline. \n\n2. This paper positions itself as an approach that avoids reliance on pre-trained representation models (a key advantage). However, a stronger argument could be made by directly discussing or comparing against methods that do leverage such models (e.g., by using a pre-trained VAE or distilling knowledge from a model like DINO/CLIP). A brief discussion on the conceptual pros and cons (e.g., ST-AR's \"self-guided\" nature vs. the potential power of a large-scale pre-trained teacher) would better situate the work within the broader landscape of improving semantic understanding in generative models.\n\nQUESTIONS:\n1. In the final loss function (Eq. 8), the weights for the new loss terms are set to α = 1.0 and β = 0.5. How sensitive is the model's performance to these specific hyperparameter choices? Was a sweep performed to find these values, and could you provide some intuition on how the balance between the standard AR loss and the new self-supervised losses affects the training dynamic?\n\n2. The framework combines a reconstruction-style objective (L_MIM) with consistency-based contrastive objectives (L_step, L_view). Did you observe any interesting interplay or potential conflict between these losses during training? For example, does the MIM objective, which forces the model to use broader context, directly aid the inter-step consistency objective, or are they largely independent mechanisms for improvement?\n\n3. The contrastive losses are applied at a middle layer of the network (e.g., the 6th layer for LlamaGen-B). The ablation in Table 5 shows this is optimal. Do you have any intuition for why this \"middle-ground\" is more effective than applying the constraint on earlier (more textural) or later (more semantic) features? Does this suggest that ST-AR is primarily helping to structure the mid-level representations of the network?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the challenges in visual understanding for autoregressive (AR) image generation models and proposes a self-guided training framework, ST-AR (Self-guided Training for AutoRegressive models), to address these issues. The authors identify three key limitations in AR models:  \n1. Local and conditional dependence: AR models overly rely on initial conditioning tokens and spatially adjacent information, as shown by attention map analyses.  \n2. Inter-step semantic inconsistency: Semantic information varies across different generation timesteps, leading to degraded global modeling.  \n3. Spatial invariance deficiency: Visual tokenizers lack invariance, causing ambiguous tokenization for slightly perturbed images.  \n\nST-AR integrates self-supervised learning techniques to enhance AR models without pre-trained representations:  \n- **Masked image modeling (MIM)** on attention maps to expand the receptive field.  \n- **Inter-step contrastive loss** to ensure semantic consistency across timesteps.  \n- **Inter-view contrastive loss** to align representations from different image augmentations.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1/ The research motivation of this work is to enhance the performance of auto-regressive models in understanding tasks. The authors found that next token prediction-based AR models have obvious defects in understanding tasks (Autoregressive models primarily rely on local and conditional information, Causal Attention Challenges Bi-directional Image Context Modeling). Based on these findings, the authors introduced methods commonly used in traditional visual representation learning, such as MIM and contrastive learning, which can effectively improve the performance of ImageNet class-conditional generation and downstream understanding tasks, such as ImageNet linear probing.\n\n2/ The authors conducted detailed experiments. In Table 1 & Table 2, the performance has significantly improved compared with the LlamaGen baseline. This method of adding additional constraints to align representations is similar in spirit to REPA and VAVAE( Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models), and it indeed can effectively improve the convergence speed and effect. However, I wonder whether this will make the model fall into a local optimum, because the model needs to learn both representation and generation during training. My personal view is that aligning to a pre-trained representation is more stable for optimization.\n\nWeaknesses:\n\n1/ Personally, I think this work somewhat resembles stitching together multiple established approaches that are known to work—such as MIM and contrastive learning, which are effective for representation learning. As such, I believe the work lacks substantial innovation.  \n\n2/ I am skeptical about the upper limits of pure visual representation learning. First, the validation of understanding tasks should be conducted on a broader range of multimodal tasks and downstream visual tasks. Currently, the effectiveness of visual encoders is mostly verified for downstream MLLM (Multimodal Large Language Model) tasks like VQA, grounding, and reasoning. However, in such MLLM tasks, approaches like MAE and contrastive learning have been abandoned in favor of semantic-aligned models such as CLIP and the SigLIP series. Therefore, my view is that integrating MIM and contrastive learning into autoregressive models does not represent the correct path.\n\nQUESTIONS:\nrefer to the weakness\n\nLIMITATIONS:\nnone", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper presents Self-guided Training for AutoRegressive models (ST-AR), which explores using next-token prediction paradigm to the visual domain. It does not rely on pre-trained representation models (unlike existing approaches such as llava), it enhances image understanding ability, compared to LlamaGen-L and LlamaGen-XL.

STRENGTHS AND WEAKNESSES:
Strengths:
- The paper is well-written and easy to follow.
- The paper is insightful, identifying three properties that impact visual understanding, e.g. local and conditional dependence, inter-step semantic inconsistency, and spatial invariance deficiency, and build accordingly.
- The experiments results are great (Table 1).
- The ablation is comprehensive, covering effectiveness of training losses, mask ratio, contrastive loss depth and number of steps.
- The comparison (on imagenet) is comprehensive and well categorized (Table 2).


Weaknesses:
- The paper should could include comparison to more pre-trained based methods, and more baseline beyond LlamaGen.
- The paper could include more datasets than Imagenet.

QUESTIONS:
(1) The paper mostly compares with LLamaGen, could the author provide additional baseline?
(2) Could the author discuss how to apply ST-AR to pre-trained representation?
(3) Could the author additionally provide result on more dataset?
(4) On line 245 - 246 the author discusses the contrastive loss is added to the 6-th and 18-th layer, can the author provides rational how these are set?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-03