Understand Before You Generate: Self-Guided Training for Autoregressive Image Generation
Authors: Xiaoyu Yue
ZiDong Wang
Yuqing Wang
Wenlong Zhang
Xihui Liu
Wanli Ouyang
LEI BAI
Luping Zhou
Venue: NeurIPS 2025 poster ()
Keywords: Autoregressive Models
Image Generation
Primary area: deep_learning

Abstract: Recent studies have demonstrated the importance of high-quality visual representations in image generation and have highlighted the limitations of generative models in image understanding. As a generative paradigm originally designed for natural language, autoregressive models face similar challenges. In this work, we present the first systematic investigation into the mechanisms of applying the next-token prediction paradigm to the visual domain. We identify three key properties that hinder the learning of high-level visual semantics: local and conditional dependence, inter-step semantic inconsistency, and spatial invariance deficiency. We show that these issues can be effectively addressed by introducing self-supervised objectives during training, leading to a novel training framework, Self-guided Training for AutoRegressive models (ST-AR). Without relying on pre-trained representation models, ST-AR significantly enhances the image understanding ability of autoregressive models and leads to improved generation quality. Specifically, ST-AR brings approximately 42% FID improvement for LlamaGen-L and 49% FID improvement for LlamaGen-XL, while maintaining the same sampling strategy.

Understand Before You Generate: Self-Guided Training for Autoregressive Image Generation
Xiaoyu Yue
Zidong Wang
Yuqing Wang
Wenlong Zhang'
Xihui Liu
Wanli Ouyang!? 1.3
Lei Bai
Luping Zhou?
Shanghai Al Laboratory
University of Sydney
Chinese University of Hong Kong
*University of Hong Kong
Abstract
Recent studies have demonstrated the importance of high-quality visual representa- tions in image generation and have highlighted the limitations of generative models in image understanding. As a generative paradigm originally designed for natural language, autoregressive models face similar challenges. In this work, we present the first systematic investigation into the mechanisms of applying the next-token prediction paradigm to the visual domain. We identify three key properties that hinder the learning of high-level visual semanties: local and conditional depen- dence, inter-step semantic inconsistency, and spatial invariance deficiency. We show that these issues can be effectively addressed by introducing self-supervised objectives during training, leading Training for RutoRegres models (ST-AR). Without relying on pre-trained rep- resentation models, ST-AR significantly enhances the image understanding ability of autoregressive models and leads to improved generation quality. Specifically, ST-AR brings approximately 42% FID improvement for LlamaGen-L and 49% FID improvement for LlamaGen-XI, while maintaining the same sampling strategy
novel training framework, Self-guided
Introduction
The field of image generation has witnessed remarkable progress through various approaches, includ- ing diffusion models 227
4
32
52
18
37
34
41
33
38
20). Generative Adversarial Networks
44 Among these, autoregressive models, originally developed for natural language processing (NLP), have demonstrated exceptional genera- tive capabilities as the foundational paradigm for large language models (LLMs) such as GPT 2 and Llama [47 48). When adapted to image generation, autoregressive models achieve performance comparable to modality-specific methods, indicating their potential as a unified generative framework
(GANs) 21 22 42 and autoregressive models (AR) [19
across diverse data modalities 5 16 45 49).
Recent studies have highlighted the importance of image understanding in enhancing generation per- formance. For instance, REPA (54] enhances the generative capabilities of diffusion models [34] [1]
by distilling self supervupeee representations into their intermediate layers. Similarly, ImageFolder 31 introduces semantic regularization to the quantizer of the tokenizer to inject semantic constraints. These methods rely pre-trained representation models to provide additional semantic information, as denoising and compressing may not be appropriate tasks for learning semantically meaningful image representations [54]. In contrast, the next-token prediction paradigm used by autoregressive models has proven to an effective pre-training approach for capturing contextual information in natural language processing (39 28 40]. However, when adapted to vision, due to the inherent differences between image and text modalities, next-token prediction also faces challenges in learning high-level visual representations.
Project Page:
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

275
250
225
200
$
17.5
3 150
325
100
75
32
64
56
128
160
192 224 256
Timestep Index
(a)
(b)
(c)
Figure I: Illustration of three properties of LlamaGen-B model. (a) Attention map from the last layer, highlighting the current token in red and tokens with larger attention weights in yellow. (b) Linear probing results on features from the 6-th layer at 8 uniformly selected steps. (c) Visual token indices from two slightly different views of the same image.
In this work, we aim to enhance the learning of high-level visual representations autoregressive models to improve the generative capability. Employing the popular autoregressive model Llama- Gen [44], we first conduct an in-depth investigation into the intrinsic mechanisms of autoregressive image generation and identify three key properties that impact visual understanding:
(I) Local and conditional dependence. Autoregressive models predominantly depend on local and conditional information. Our analysis of attention maps, as shown in Figure (a), reveals strong dependence on the initial step (conditioning token) and spatially adjacent steps, highlighting that the model primarily utilizes conditional and local information for its predictions.
(2) Inter-step semantic inconsistency. Figure T (b) demonstrates inconsistent semantic information across different timesteps, as evidenced by the top-1 linear probing accuracy. Specifically, while accuracy increases in early timesteps with more visible image tokens, its subsequent decline reveals that autoregressive models fail to maintain previously learned semantic information, thereby limiting the global modeling capability.
(3) Spatial invariance deficiency. Autoregressive image generation models typically employ visual tokenizers, such VQ-GAN [19 29] to quantize images into discrete tokens. However, slight perturbations in image space can result in completely different tokens, as shown in (igure[[()) This ambiguity of objects significantly increases the difficulty for autoregressive models in encoding visual signals.
These three problems create bottlenecks for autoregressive models in learning high- image representations, mirroring the challenge faced by diffusion models as revealed by REPA (54). To this end, we propose ST-AR, short for Self-guided Training for AutoRegressive models, novel training paradigm that leverages techniques well-explored in self-supervised learning to enhance modeling of visual signals. Specifically, for property inspired by masked image modeling (53) 25
that forces the network to attend to larger regions of the image [36] (5), we randomly mask a portion of the tokens in the attention map of the transformer layers. Mcanwhile, for properties 2 & 3, employ contrastive learning 15] to ensure the consistency of feature vectors from different time steps and views, referred to inter-step contrastive loss and inter-view contrastive loss, respectively. The resulting training paradigm, ST-AR, incorporates a MIM loss and two contrastive losses in addition to the token prediction loss, forming an iBOT-style (56) framework. ST-AR is utilized only during training, and the trained models retain the autoregressive sampling strategy, thus preserving their potential for unification with other modalities.
By integrating visual self-supervised paradigms into next-token prediction, ST-AR eliminates the need for pre-trained representation learning models to provide additional knowledge, achicving stronger image understanding solely through self-guided training. Specifically, ST-AR significantly improves the linear probing top-1 accuracy of LlamaGen-B from 21.00% to 55. 23% and demonstrates image understanding
semantically meaningful attention maps. Furthermore, the enhancement facilitates image generation. On dlass-conditional ImageNet, ST-AR boosts LlamaGen-B -B by 7.82 FID score. Notably, LlamaGen-Xi trained with ST-AR for just 50 epochs achieves approximately a 49% improvement in FID over the baseline, and is even comparable LlamaGen-3B trained for 300 epochs, despite the latter having about 4x more parameters.
Our contributions can be summarized as follows:

---

Conceptually, we conduct an in in-depth investigation into the mechanisms of autoregressive image generation. identifying three key properties that hinder visual representation learning.
Technically, we propose novel training paradigm, ST-AR, which enhances image under- standing by integrating self-supervised training techniques into the next-token prediction paradigm.
Experimentally, conduct comprehensive experiments validate the design of each component of ST-AR, demonstrating its effectiveness in both image understanding and generation.
2 Related Work
Autoregressive Image Generation. The autoregressive (AR) generation paradigm has established itself as leading approach in language modeling [40 0 due to its simplicity, scalability, and zero- shot generalization capabilities. When extended to image generation, AR methods can be categorized into three types according to the sampling strategies. Causal AR methods, such as VQ-GAN [19 and LlamaGen 44. directly adapt AR architectures for image synthesis, utilizing the traditional raster- order next-token prediction paradigm as language models. Masked AR methods, like MaskGiT M and MAR [30], employ bi-directional attention within mncoder-decodee framework, supporting iterative generation with flexible orders. Parallelized AR methods introduce vision-specific designs to enhance visual signal modeling capability. VAR [46] proposes next-scale prediction that progressively generates tokens at increasing resolutions. PAR (00 and NPP 33 propose token grouping strategies to generate image tokens in parallel. Although masked and parallelized AR methods enhance the modeling of bidirectional image contexts, they require adjustments to sampling strategies. Our ST- AR focuses on improving the modeling of visual modalities within AR models without altering the sampling strategy, thereby enhancing image generation performance while preserving compatibility with language models.
Supe-Supe Learning. In the field of visual superviee learning, methods can be broadly categorized into two types: contrastive learning and masked image modeling. The first to emerge was contrastive learning, exemplified by methods such as SimCLR [2], B YOL MoCo 24 [5]. SwAV and DINO[1) These approaches typically employ image augmentation techniques to construct sets of positive samples and optionally use augmented views from other images as negative samples. They learn semantic information by aligning the representations of positive samples. Masked Image Modeling (MIM) 66 53 25] adapts the concept of Masked Language Modeling from NLP, training networks to reconstruct randomly masked portions of image content, thereby learning visual context. Some studies have shown that MIM primarily learns low-level pixel correlations and adjust the effective receptive field size of the network by modifying the mask ratio. Our ST-AR leverages the strengths of both contrastive learning and MIM, using random masking on attention maps increase the attention distance of autoregressive models, as well as employing like contrastive losses to align representations across different time steps and different views.
3 Method
3.1 Preliminaries
We provide brief review of visual autoregressive models operating in discrete space. Given an input image I, a quantized autoencoder is employed to convert a sequence of discrete tokens: [x1, I2, = where [ .... Ør] is the output token sequence, and q(-) denote the encoder and quantizer of the quantized autoencoder.
The autoregressive model is trained to maximize the joint conditional probability of predicting the token x, the current step i, based on the conditional vector and the preceding tokens [21, 11- The condition can be class label or text vector. The training objective can be formalized as:
T
max Pe (x) = II 0
pa (xe (c, 3t, 22, Tt-1)):
(I)
3

---

Layer Index
2
10
11
12
Figure 2: Attention maps of LlamaGen-B across layers and steps. These attention maps consis- tently show that conditional and spatially adjacent tokens receive the highest attention weights, while other tokens have significantly lower weights.
where pa is the autoregressive model parameterized by 0. And the token prediction loss is:
T CAR
T Σ
log pog
After training, pe iteratively generate new sequences. This process known as the next-token prediction, has been proven effective in text modeling.
3.2 Observations
We conduct an in-depth investigation into the intrinsic mechanisms of autoregressive models in image generation, evaluating visual understanding capabilities through two aspects: attention maps and linear probing. For the class-conditional LlamaGen-B model trained on ImageNet [77, we first analyze the behavior of the transformer module by visualizing attention maps. Attention maps reveal what the model relies on for predictions and whether it can capture image context. Then, we evaluate the quality of learned representations by comparing linear probing results across intermediate layers different time steps. Specifically, we closely followed the training protocol in MAE 25 and set the input class embedding the null unconditional embedding for classifier free guidance to prevent knowledge leakage. We uniformly select out of 256 steps and feed the features from the sixth layer at the corresponding steps into trainable linear layers. Our observations are as follows:
Obs. Autoregressive models primarily rely on local and conditional information. In Figure 2 we present attention maps across various depths and positions, all exhibiting a consistent pattern: the highlighted areas predominantly include spatially adjacent tokens and the first token. As indicated in Eq. [ the input at the initial step is the conditional token, which significantly influences subsequent sampling, thereby holding considerable importance the attention maps. Tokens surrounding the current token also receive elevated attention weights due to the inherent locality of images. Despite all preceding tokens being visible during training, the spatial proximity of tokens dictates that the most informative tokens for predicting the current token are typically those nearby. Excessive reliance on local information can impede the generation quality, as minor errors in adjacent tokens may be accumulated for subsequent steps.
Obs. 2. Causal Attention Challenges Bi-directional Image Context Modeling. The application of causal attention to images presents two critical challenges: semantic inconsistency across different steps and limited global modeling capability. The inherent sequential nature of causal attention, which restricts each step to accessing only previously generated content, fundamentally limits the model's 's capacity to capture comprehensive global information. As illustrated in Figure (b), the linear probing accuracy at the initial steps is extremely low, indicating that AR models struggle to establish the correct semantic context in the early steps. Furthermore, the observed deterioration in linear probing performance beyond the 192-th step indicates a progressive semantic misalignment in the learned representations as generation proceeds. This phenomenon underscores a critical limitation in the model's ability to maintain and leverage global contextual information effectivvly throughout

---

Trnsfomet
2002
EMA
Eat
atep
Transforma
Trassfor
Liver
stop grad
ground truth tokers tokens mesked ettention maps
Stap grad
Figure 3: Overview of Self-Guided Training Pipeline. We incorporate masked image modeling
(CMIM) to expand the effective field of visual autoregressive models. Additionally, we introduce inter-step contrastive learning (Cste) to ensure global consistency, as well as inter-view contrastive learning (Ceiew) for consistency in visual representations.
the generation process. Such constraints pose significant challenges for achieving coherent and semantically consistent image generation.
Obs. 3. Visual tokens lack invariance. Autoregressive models utilize a visual tokenizer like VQ-GAN transform continuous image signals into discrete tokens. However, visual tokenizers are primarily trained for image compression and reconstruction, lacking invariance constraints. Consequently, when transformations produce entirely different visual tokens, as demonstrated in Figure (c). This variability in visual signals confuse the model, resulting in redundant learning of identical semantic concepts.
applied to image of given object, the tokenizer may
3.3 Self-Guided Training for Autoregressive Models
Building upon these observations, we introduce Self-guided Training for RutoRegees models (ST-AR) to enhance the visual understanding capabilities of autoregressive models. ST-AR provides targeted solutions for the aforementioned challenges within unified training paradigm.
Overview. The overall pipeline of ST-AR is illustrated in Figure 3 ST-AR borrows ideas from self-supervised representation learning, employing masked learning to expand attention regions while utilizing contrastive learning to ensure feature alignment across both steps and views. A non-trainable teacher network [14] 24 I0] is employed to provide additional training objectives. It shares the same architecture as the autoregressive model (student model), and weights are updated through the Exponential Moving Average (EMA) of the student model parameters A. ST-AR integrates reconstruction loss and two contrastive losses into the training of autoregressive models, eliminating dependence on pretrained representation models. We refer it "Self-Guided Training".
3.3.1 Masked Learning for Longer Contexts
As revealed in 36 [55), masked image modeling (MIM) can expand the effective receptive field of image encoding models. This insight motivates our approach to leverage MIM for addressing the challenge AR models outlined in Obs. [ ie., the excessive dependence on local information. However, traditional MIM methods, which substitute input image tokens with a special mask token, is unsuitable for autoregressive models. This is because autoregressive models, unlike autoencoders, necessitate the use of image tokens from the preceding step for next-token prediction. To overcome this, ST-AR utilizes random masking directly on the attention maps within transformer blocks, rather than on the input tokens. sequence mask M applied the attention map. assigning negative infinity (-inf) to a ratio r of the total tokens (masked tokens), while normal tokens are assigned as zero. Formally:
(QiKe + +MMM Vi Vdk
Attn(Q,, K,, V/) Softmax
where Qi, Kú, and V& are the query, key, and value matrices for the i-th head.
As the masking operation on attention may lead to information loss for next-token prediction, we employ teacher model to extract features and align the student model accordingly. Specifically, for given input tokens, we solely mask the attention maps of the student model and align the final hidden states of the student network the teacher network. Given token length T, the MIM loss can be
5

---

formalized as:
T
CMIM D(he,hi)
where (), is the distance function, defaulting to cosine distance, h4 and he are the features extracted from the last transformer layer of the student and teacher networks.
3.3.2 Contrastive Learning for Consistency
The essence of Obs. and Obs. 3 glies in the inconsistency of representations during the autoregressive iterative process. Specifically, Obs. pertains to inconsistencies between different steps in the same image, while Obs. relates to inconsistencies between different augmented image views. Inspired by the SSL methods, we use contrastive learning paradigm solve such inconsistency.
16=1 Given batch of images [ ST-AR applies M random augmentations to each image, resulting in set of augmented views (b,m)) j.M m== These augmented images are then encoded by VQ-GAN g()), producing discrete token sequences X E nx M x7 where denotes the token sequence length fed into both the student network pa and EMA teacher network pgr. yielding token features h, Po (X( E 2BxxxXXX and h2 = (xx e RBxMxTxD where D) is the feature dimension. Following SimSiam[1X) [3], we employ a projector f()). which consists several MIPs, on the student features: 2, = f(h.). The projector helps prevent model collapse and enhances training stability.
(i.e., the steps of AR models). The resulting tokens
To compute the contrastive loss, we randomly select K token positions from the sequence length T, denoted as I ~ RandomK T"). The sampled features used for loss computation are:
2, = :, IR BXMXXXD h4 h,Is,;, BxMMXXX
(5)
We use inter-step contrastive loss Lstep enforce semantic consistency across different steps, addressing Obs. For each sampled student feature vector 2(b.m.1), we define the positive sample as the teacher feature F. f.b.m..) extracted from the same view but a different position, while the negative samples come from other images in the batch. Formally:
B
M
K
exp & (b.mm.)) ki(bom..i)
Lstep
(6)
B
2o exp(1 exp(s!0im)) (bim)))) (.u.m.)))
b=1
In addition, introduce inter-view contrastive loss Lriew
semantic consistency across different augmented views, addressing Obs. 3 Specifically, for a student feature =! =(b,i,k) the positive
sample is the teacher feature i. f.(b...k) extracted from the same token position & but different view of the same image. Negative samples come from other images in the batch. The loss defined as:
exp (h...) (((.J.A)))
Lview
B D
exp 2l R. (((.s.)))
(b.,k)
To improve training efficiency, we set the number of image views M = 2 in our implementation. We conduct an ablation study about the effects of the number of different steps K generation quality, which is detailed in Table
3.3.3 Training Losses
We incorporate masked image modeling (Eq. 4 and contrastive learning (Eq. 6 and Eq. 7 into the
conventional next-token prediction loss (Eq. 2 The final loss function can be formalized as:
CST-AR = LAR + aLMI + 3 (Cotep + Lview).
(8)
where and 3 are the weights for the reconstruction loss and contrastive losses, respectively.

---

Table I: Comparisons between LlamaGen model and ST-AR. All the results are evaluated without using CFG on ImageNet. imeans the model is trained 384 x 384 resolution and resized to 256 x 256 resolution for evaluation.
40
Top- Acc.1%)
Model
#Params Epochs
FID
sFID)
1S
Prec.
Rec.
LlamaGen-B
111M
50
31.35
8.75
39.58
0.57
0.61
+ STT-RR
11MM
50
26.58
7.70
49.91
0.60
0.62
20-
LlamaGen-B
111M
300
26.26
9.22
48.07
0.59
0.62
+ST-AR
111M
300
18.44
6.71
66.18
0.64
0.62
Limatizn- (0 (3000)
Lamstien B12e 1526)
LlamaGen-LL
343M
21.81
8.77
59.18
0.62
0.64
LiamaGan-
LiamaGan- 1o00
+TT-AR
343M
12.59
6.79
91.19
0.65
0.64
64
128
160 224 256
LlamaGen-L
343M
300
13.45
8.32
82.29
0.66
0.64
Timestap Index
+ ST-AR
343M
300
9.38
6.64
112.71
0.70
0.65
Figure 4: Linear probing results of LlamaGen-B and our ST-AR. Our method demonstrates consistent im- provements in image under- standing.
LlamaGen-XL
775M
300
15,55
7.05
79.16
0.62
0.69
LlamaGen-XXe
L4B
300
14.65
8.69
86.33
0.63
0.68
LlamaGen-3B
3.1B
300
9.38
8.24
112.88
0.69
LJamaGen-X0
775M
50
19.42
8.91
66,20
0.61
0.67
+ ST-AR
775M
50
9.81
6.94
109.77
0.71
0.63
+ ST-AR
775M
300
6.20
6.47
147.47
0.73
0.65
Figure 5: Attention maps of LlamaGen-B model trained with our ST-AR method. We utilize features from the final transformer layer, selecting random steps draw attention maps. These maps exhibit an expanded effective receptive field, moving beyond mere focus on spatially adjacent and conditional tokens, and reveal distinct semantic patterns.
Experiments
4.1 Implementation Details
Dataset. We evaluate the effectiveness of ST-AR the class-conditional image generation task using the widely adopted ImageNett2266 x 256 dataset. We employ the same VQGAN as LlamaGen [44] for tokenization, precomputing the image token sequences before training. Following LlamaGen, also compute tokens for ten crops of the original image.
Evaluation metrics. Since our ST-AR generative model is trained with self ceff-supervise losses to enhance its visual modeling capabilities, we holistically evaluate ST-AR on image understanding and generation. For image understanding, we use the top-1 accuracy of linear probing the primary metric. We adopt the linear probing setup of MAE[25), training linear layer for 90 epochs using the representations from the sixth layer. For image generation, we use the ADM evaluation suite and report Fréchet Inception Distance (FID) [26 as the main evaluation metric.
Training & Inference. All the models are trained with the same setting as LlamaGen: base learning rate of x 10-4 per 256 batch size, AdamW optimizer with 31 0.9, 3, 0.05, weight decay set to 0.05 and gradient clipping set to 1.0. We train our models on images with 256 x 256 resolution, rather than LlamaGen with 384 x 384 training images. The teacher model is updated through the exponential moving average of the student model with an E.A decay of 0.9999. The class token embedding dropout ratio 0.1 for classifier-free guidance. The contrastive loss is added on medium of the transformer network, i.e. the 6-/h layer for LlamaGie-BB, 18-th layer for LlamaGen-L. and 18-th layer for LlamaGen-X0 The masking ratio used for mask image modeling Eq. 3 is set 0.25. The number of steps used in Eq. 6 and Eq. 7 is set as K 4. The weights of

---

Table 2: Model comparisons ImageNNt-2256 x 256 Benchmark. All the results are evaluated with CFG. imeans the model trained on 384 x 384 resolution and resized to 256 x 256 resolution for evaluation. ST-AR consistently beats baseline LlamaGen all model sizes and training costs.
Type
Model
#Params
Epochs
FID
sFID
IST
Prec.
Rec.
BigGAN 7
112M
6,95
7,36
171.40
0.87
0.28
GAN
StyleGan-XI
42
166M
2.30
4.02
265.12
0.78
0.53
LDM-4 41
400M
3.60
5.12
247.67
0.87
0.48
Diff.
DIT-XL 37
675M
1400
2.27
4,60
278.24
0.83
0.57
SiT-XI
675M
14000
2.15
4.50
258.09
0.81
0.60
11
227M
300
6.18
182.10 10
0.80
0.51
Masked AR
MaskGIT-re
227M
300
4.02
355.60
0.83
0.50
VAR-d16 46
310M
3.30
274.4
0.84
0.51
Parallelized AR
VAR-d20 46
600M
2.57
302.6
0.83
0.56
1.4B
15.78
74.30
VQGAN-re
1.4B
5,20
280.30
Casual AR
LlamaGen-B4
111M
300
6.09
7.24
182.54
0.85
0.42
LlamaGen-Li +
44
343M
300
3,08
6,09
256.07
0.83
0.52
LlamaCin--I
775M
300
2.63
5.59
244.09
0.81
0.58
LlamaGen-B
111M
300
5,46
7.50
193.61
0.84
0.46
+ ST-AR
111M
300
4.09
6.72
246.29
0.86
0.47
LlamaGie--L.
343M
300
3.81
8.49
248.28
0.83
0.52
+ ST-AR
343M
300
2.98
6,44
264.11
0.85
0.53
LlamaGen-XI.
775M
50
3.39
7.02
227.08
0.81
0.54
Casual AR
ST-AR
775M
50
2.72
6.03
254.59
0.83
0.57
ST-AR
775M
300
2.37
6.05
270.59
0.82
0.58
Table 3: The effects of proposed losses. ST-AR improves linear probing and generation quality.
Model
LMIM
Cutep
Lviese
FID
sFID
1St
Prec.
Rec.t
LP Ac).
IlamaGee-B
31.35
8.75
39.58
0.57
0.61
18.68
30.58
8.94
41.95
0.59
0.59
22.71
V
28.02
8.21
46.20
0.59
0.61
27.73
27.78
7.52
45.88
0.60
0.61
38.31
ST-AR (Ours)
26.58
7.70
49.91
0.60
0.62
45.27
reconstruction loss and contrastive loss in Eq. inference, we use the same sampling strategy LlamaGen.
8 are set to = 1.0 and B = 0.5 by default. For
4.2 Main Results
Image understanding. The linear probing results are shown in Figure 4 ST-AR significantly enhances the linear probing performance of the bascline model, LlamaGen-B, across steps, demonstrating improved image understanding capabilities. Importantly, the accuracy does not degrade after the 192-th step, indicating that LlamaGen trained with ST-AR effectively preserves semantic information from previous iterations during the sampling process.
In Figure [] visualize the attention maps of the last layer at different steps. Compared to the baseline model (Figure 2). ST-AR not only significantly expands the scope of attention but also focuses semantically relevant regions, further demonstrating that ST-AR effectively enhances the learning of visual semantic representations.
Class-conditional image generation. As previously stated, the enhancement in image understanding also leads to higher generation quality. We first compare the LlamaGen models trained with ST- AR to their vanilla counterparts. As shown in Table T ST-AR achieves significant performance improvements across all LlamaGen variants. Specifically, for LlamaGen-Xe -LL, training with ST-AR for 50 epochs improves the FID score approximately 10, reducing from 19.42 9.81 compared to the vanilla counterpart. Further training for 300 epochs leads to an FID 6.20, which is even stronger than LlamaGen-3B with 4x parameters.
In T we provide results using classifier-fres guidance (CFG) and comparisons with methods from other paradigms, including GANs, diffusion models, masked AR, and parallelized AR. ST-AR

---

Table 4: Ablation on mask
Table 5: Ablation on contrastive loss depth.
Table 6: Ablation on the num- ber of selected steps.
ratio.
Ratio
FID4
sFID4
iSt
Depth
FID
sFID
1St
#Steps
FID
sFID)
ISt
0.15
28.62
7.28
44.58
3 (1/4-))
27.34
7.49
46.23
2
27.50
8.31
47.15
0.25
26.58
7.70
49.91
6 (1/2-d)
26.58
7.70
49,91
4
26.58
7.70
49.91
26.36
49.73
(3/4-d)
28.76
8.66
44.73
8
26.54
7.61
0.45
27.50
8.31
47.15
12 (1-d)
29.45
8.56
43.32
25.78
7.86
50.66
achieves consistent and significant improvements over LlamaGen while also delivering performance comparable other state-of-the-art methods.
Qualitative Comparison. In Figure 6 we generate images using the same random seed for LlamaGen- and LlamaGen- B ST-AR. It can be observed that, due to the lack of global and semantic infor- mation, images generated by LlamaGen exhibit distortions and object discontinuities, while images generated with ST-AR appear more natural. This also highlights the importance of incorporating high-level semantic information.
(a) LlamaGen-B
(b) LlamaGen-B + ST-AR
Figure 6: Qualitative comparison between (a) LlamaGen-B and (b) LlamaGen-B + ST-AR.
4.3
Ablation Studies
We conduct comprehensive experiments on different configurations of ST-AR. All reported results obtained using LlamaGen-B model trained for 50 epochs.
Effectiveness of Training Losses. We conduct experiments to validate the effectiveness of the three loss functions in ST-AR, namely LMIM. Cateps and Lview. The results are shown in Table All three losses improve linear probing accuracy, thereby enhancing generation quality. Among them, the inter-view contrastive loss Lview contributes more to the improvement in linear probing accuracy compared to the inter-step contrastive loss Lstep- Notably, equipping LlamaGen-B B with all three losses significantly increases its linear probing accuracy from 18.68% 45.27%.
In Figure we visualize the attention maps models trained with each of the three losses individually. The MIM loss significantly expands the attention range but fails to capture semantic information, consistent with the results in Table Both the inter-step contrastive loss and inter-view contrastive loss slightly expand the attention range but significantly highlight semantically relevant areas.

---

a
(c)
Figure 7: Attention maps of LlamaGen-B trained with (a) MIM loss, (b) inter-step contrastive loss, and (c) inter-view contrastive loss individually.
Effect of Mask Ratio. Masked image modeling is a key design in ST-AR, as discussed in Section 3.3.1 expands the effective receptive field of the network. In Table 4 we examine the effect of the mask ratio generation performance. The FID score is lowest when the mask ratio is 0.35. However, increasing the mask ratio leads to degradation in sFID, indicating that masking too many tokens can negatively affect the learning of low-level spatial structures.
Effect of Contrastive Loss Depth. We validate the impact of incorporating the two contrastive losses, Catep and Lviews at different depths of the network. There has long been view that image generators consist of an encoder and a decoder. The results shown in Table [ align with this perspective, demonstrating that applying contrastive losses at the th layer (half the depth) yields the best performance.
Effect of the Number of Steps. As described in Section 3.3.2
randomly select K different
steps for contrastive learning. In Table 6 we examine the impact of the number of steps K. Larger values of K lead better generation performance. However, the improvement becomes marginal for K > 4. Therefore, we set K 4 by default.
Conclusion
In this work, we focus on investigating the visual understanding capabilities of autoregressive models for image generation, offering in-depth analysis and identifying three fundamental challenges that hinder the learning high-level visual semantics. We demonstrate that these challenges can be effectively addressed by incorporating representation learning objectives, leading to novel training framework: Self-guided Training for AutoRegressive models (ST-AR). ST-AR employs masked image modeling to broaden attention regions while utilizing contrastive learning to maintain semantic consistency across steps and views. Extensive experiments validate ST-AR's effectiveness in enhancing visual understanding, which consequently improves image generation quality.
Limitations & societal impacts. The main limitation of this work lies in increased training costs, which we will address in future research. While ST-AR establishes novel training paradigm for autoregressive image generation with potential industry applications, may also raise concerns regarding image manipulation risks.
Acknowledgements
This work was supported by the JC STEM Lab of AI for Science and Engineering, funded by The Hong Kong Jockey Club Charities Trust, the Research Grants Council of Hong Kong (Project No. CUHK 14213244.
10

---

References
[1] Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahrce, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404. 2e.e244444.219, 2024.
[2] Josh Achiam, Steven Adler. Sandhini Agarwal, Lama Ahmad, Nige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint ar::: 087744 2023.
[3] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm technical report. arXiv preprint arXiv:2305. 10403, 2023.
[4] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang. Xiaodong Deng. Yang Fan, Wenbin Ge, Yu Han, Fei Huang, al. Qwen technical report. arXiv preprint arXiv:2309. 20w.:299..6609, 2023.
[5] Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan L Yuille, Trevor Darrell, Jitendra Malik, and Alexci A Efros. Sequential modeling enables scalable learning for large vision models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22861-22872, 2024,
[6] Hangbo Bao, Li Dong. Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint Xi1:22106.08254. 2021.
[7] Andrew Brock. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809 1096, 2018
[8] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are fow-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[9] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912-9999 12-9924, 2020.
[10] Mathilde Caron. Hugo Touvron. Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in supervidd vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[11] Huiwen Chang, Han Zhang, Lu Jiang. Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11315-11325, 2022.
[12] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. International conference on machine learning, pages 1597-1607, PMLR, 2020.
[13] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. arXiv preprint 1. 2020.
[14] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint 2XXi:2003304299) 2020.
[15] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEF/CVF international conference on computer vision, pages 9640-9649, 2021.
[16] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, al. Scaling instruction instruction-finctuned language models. Journal of Machine Learning Research, 25(70): 53, 2024.
[17] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In EEE/CVF Conference on Computer Vision and Pattern Recognition, 2009.
[18] Prafulla Dhariwal and Alexander Nichol, Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-877 8794, 2021.
[19] Patrick Esser, Robin Rombach, and Bjor Ommer. Taming transformers for synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873 12873-12883, 2021.
image

---

[20] Patrick Esser, Sumith Kulal, Andreas Blattmann. Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi. Dominik Lorenz, Axel Sauer, Frederic Boesel, al. Scaling rectified flow transformers for high-resolution image synthesis. 2024.
[21] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[22] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11): 139-144, 2020.
[23] Lean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21284, 2020.
learnnn
[24] Kaiming He, Hanqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv:1911.05722, 2019.
[25] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[26] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances neural information processing systems, 30, 2017.
[27] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. 2020.
[28] Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers language understanding. In Proceedings of naacL HLT, volume 1. Minncapolis, Minnesota, 2019.
[29] Doyup Lec, Chihcon Kim, Sachoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11523 11532, 2022.
[30] Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. arXiv preprint arXiv:2406. / 1838, 2024.
[31] Xiang Li, Kai Qiu, Hao Chen, Jason Kuen, Jiuxiang Gu, Bhiksha Raj, and Zhe Lin. Imagefolder: Autoregressive image generation with folded tokens. arXiv preprint 24100:2400001756, 2024.
[32] Yaron Lipman, Ricky TQ Chen, Heli generative modeling. arXiv preprint arXiv:2210.02744 2022.
Maximilian Nickel, and Matt Lc. Flow matching for
[33] Zeyu Lu, Zidong Wang. Di Huang, Chengyue Wu, Xihui Liu, Wanli Ouyang, and Lei Bai. Fit: Flexible vision transformer for diffusion model. In International Conference on Machine I Learning, 2024.
[34] Nanye Ma, Mark Goldstein, Michael Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. arXiv preprint ar 2401000008 2024.
[35] Yatian Pang, Peng Jin, Shuo Yang. Bin Lin, Bin Zhu, Zhenyu Tang, Liuhan Chen, Francis EH Tay. Ser- Nam Lim, Harry Yang, et al. Next patch prediction for autoregressive visual generation. arXiv preprint arXiv:2412.15321,, 2024,
[36] Namuk Park, Wonjae Kim, Byeongho Heo. Taekyung Kim, and Sangdoo Yun. What do self superviee vision transformers learn?? arXiv preprint arXiv:2305.00729, 2023.
[37] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4195-4496 4205- 2023.
[38] Dustin Podell, Zion English. Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01955 2023.
[39] Alec Radford. Improving language understanding by generative pre-training. 2018.
12

---

[40] Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning reseurch, 21(140):1-67, 2020.
[41] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[42] Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xI xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings, pages 1-10, 2022.
[43] Yang Song, Jascha Sohl-Dickstein. Diederik Kingma. Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint 0mXiv:2011.13455 2020.
[44] Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autore- gressive model beats difffusion: Llama for scalable image generation. arXiv preprint 1:Xi:22400006525 2024.
[45] Chamelcon Team. Chameleon: Mixed-mode carly-fusion foundation models. arXiv preprint orXiv:2405.09816 2024.
[46] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint arXiv:2404.02905 2024.
[47] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.3997 2023.
[48] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et Llama Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
[49] Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409,18880 18869, 2024.
[50] Yuqing Wang. Shuhuai Ren, Zhijie Lin, Yujin Han, Haoyuan Guo, Zhenheng Yang. Difan Zou, Jiashi Feng. and Xihui Liu. Parallelized autoregressive visual generation. arXiv preprint arX 19, 2024.
[51] Zidong Wang, Lei Bai, Xiangyu Yue, Wanli Ouyang, and Yiyuan Zhang. Native-rosolution image synthesis. arXiv preprint arXiv:2506.031 03131, 2025.
[52] Zidong Wang. Yiyuan Zhang, Xiaoyu Yue, Xiangyu Yue, Yangguang Li, Wanli Ouyang, and Lei Bai. Transition models: Rethinking the generative learning objective. arXiv preprint 2509v:2509.04333 2025.
[53] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9653 9663, 2022.
[54] Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jcong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is casier than you think. arXiv preprint Xww:::44400600000 2024.
[55] Xiaoyu Yue, Lei Bai, Meng Wei, Jiangmiao Pang, Xihui Liu, Luping Zhou, and Wanli Ouyang. Under- standing masked autoencoders from local contrastive perspective. arXiv preprint 1rx0::2310.01994 2023.
[56] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint 2 2021.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims made in the introduction reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of this paper are discussed in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

The conclusions of this paper are based on analyzing autoregressive models and do not include theoretical results.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: provide detailed hyperparameters for model training in Section and will release the code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We used public datasets and will release the code for reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ praip..cc/pblli/guuudee CoodeSumeeieeeeee ssi iessonPolll molyicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all training and test details in Section 4, along with comprehensive ablation studies.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Our experiments are conducted on large-scale datasets and show stable results through multiple reproductions, error bars not included.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Refer Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics Ehte:s/brrr.//rrrrrr...../ob.bbb http://arr.ppp...aaaoaacccoacaar
Answer: [Yes]
Justification:
The research in this paper fully complies with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Societal impacts are discussed Section 5.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The proposed training paradigm poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We strictly adhered to the licenses of the public datasets used.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
20


Reviews:
[{"review_id": "A6GnIjx0TE", "reviewer": "Reviewer_rKee", "summary": "This paper presents Self-guided Training for AutoRegressive models (ST-AR), which explores using next-token prediction paradigm to the visual domain. It does not rely on pre-trained representation models (unlike existing approaches such as llava), it enhances image understanding ability, compared to LlamaGen-L and LlamaGen-XL.", "questions": "(1) The paper mostly compares with LLamaGen, could the author provide additional baseline?\n(2) Could the author discuss how to apply ST-AR to pre-trained representation?\n(3) Could the author additionally provide result on more dataset?\n(4) On line 245 - 246 the author discusses the contrastive loss is added to the 6-th and 18-th layer, can the author provides rational how these are set?", "limitations": "Yes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper is well-written and easy to follow.\n- The paper is insightful, identifying three properties that impact visual understanding, e.g. local and conditional dependence, inter-step semantic inconsistency, and spatial invariance deficiency, and build accordingly.\n- The experiments results are great (Table 1).\n- The ablation is comprehensive, covering effectiveness of training losses, mask ratio, contrastive loss depth and number of steps.\n- The comparison (on imagenet) is comprehensive and well categorized (Table 2).\n\n\nWeaknesses:\n- The paper should could include comparison to more pre-trained based methods, and more baseline beyond LlamaGen.\n- The paper could include more datasets than Imagenet.", "was_revised": true, "final_justification": "As the initial review points out, the paper is in good quality and insightful. During rebuttal, the author further addressed my non-major concerns, Thus I would like keep my original rating and vote for acceptance.", "rebuttal": "We truly thank the reviewer for your detailed review and thoughtful comments in helping us enhance the quality of our paper. Below, we provide detailed point-to-point responses addressing each of your valuable suggestions:\n\n\n**Q1: Comparison on other baselines and datasets.**\n\nIn the supplementary material, we provide the results of applying ST-AR on the vanilla Transformer, also in Table A1. Besides, we provide the text-to-image generation results of ST-AR and LlamaGen in the supplementary material, also in Table A2.\n\nFor the vanilla autoregressive Transformer, we follow the training setup of LlamaGen and conduct comparative experiments on the ImageNet-256x256 benchmark. As shown in Table A1, ST-AR consistently and significantly improves the image generation performance of the vanilla Transformer.\n\n**Table A1: Results of vanilla Transformers on ImageNet-256x256 Benchmark.**\n||Model|Epochs|FID|sFID|IS|Prec.|Rec.|\n|-|-|-|-|-|-|-|-|\n|w/o CFG|Transformer|50|35.30|6.66|31.90|0.56|**0.59**|\n||**+ST-AR**|50|**29.37**|**6.08**|**38.88**|**0.60**|**0.59**|\n|w/ CFG|Transformer|50|9.67|6.94|129.50|**0.84**|0.37|\n||**+ST-AR**|50|**6.86**|**6.40**|**159.13**|0.83|**0.43**|\n\nFor text-to-image generation experiments, we train LlamaGen-XL and ST-AR on a 2M subset of the SAM-11M dataset with an image resolution of $256 \\times 256$. We use a pretrained FLAN-T5-XL to extract text embeddings, where the maximum embedding length is set to 120. \nWe use FID and CLIP Score to evaluate image quality and text-image alignment on the COCO-val-30K benchmark. As shown in Table A2, ST-AR also achieves significant improvements on text-conditional generation, further demonstrating the generalizability of ST-AR.\n\n**Table A2: Results of LlamaGen-XL on the text-conditional COCO-val benchmark.**\n|Model|Epochs|FID|CLIP|\n|-|-|-|-|\n|LlamaGen-XL|50|17.08|0.25|\n|**+ST-AR**|50|**13.52**|**0.29**|\n\n**Q2: Comparison with pretraining-based methods**\n\n**Aligning the intermediate features of autoregressive models with pretrained representations introduces spatial misalignment.** Specifically, for position $i$, pretrained vision encoders extract features of the $i$-th image patch, whereas autoregressive models take the $(i-1)$-th image token as input to predict the $i$-th token. Such behavior is fundamentally distinct from diffusion models. \nTherefore, directly applying representation alignment methods, such as REPA[1], to autoregressive models is **suboptimal**.\nTo validate this, we apply REPA to LlamaGen, strictly following the settings of REPA, aligning the intermediate features of LlamaGen-B with representations from DINOv2-B or MAE-B.\nAs shown in Table A3, while REPA achieves better generation performance compared to the baseline, it is still inferior to ST-AR.\n\n**Table A3: Comparison between ST-AR and pretraining-based methods.**\n|Model|Pretrained Encoder|Epochs|FID|sFID|IS|Prec.|Rec.|\n|-|-|-|-|-|-|-|-|\n|LlamaGen-B||50|31.35|8.75|39.58|0.57|0.61|\n|LlamaGen-B + ST-AR||50|26.58|7.70|49.91|0.60|**0.62**|\n|LlamaGen-B + REPA|DINOv2-B|50|28.50|7.78|44.72|0.59|**0.62**|\n||MAE-B|50|31.27|9.12|40.27|0.58|0.60|\n|LlamaGen-B + REPA + ST-AR|DINOv2-B|50|**25.11**|**7.37**|**50.52**|**0.61**|**0.62**|\n\n**Q3: Applying ST-AR to pretrained representations.**\n\nWe also apply ST-AR to pretrained representations (the last row of Table A3). We align the intermediate features from the $6$-th layer of LlamaGen-B with the visual representations extracted by pretrained DINOv2.\nWe retain the three losses of ST-AR and set the weight of the alignment loss to $0.5$.\nAs shown in Table A3, by incorporating additional pretrained representations, the performance of ST-AR is further improved. We attribute this improvement to the richer semantic information learned by DINOv2 on a larger-scale dataset.\n\n**Q4: Choice of contrastive loss depth.**\n\n**The contrastive losses in ST-AR are designed to align the high-level semantic information across different timesteps and views.**\nIn Table 5 of our manuscript, we explore the impact of contrastive loss depth and demonstrate that applying the losses at half of the model depth achieves the best performance for ST-AR. Here we provide more insights about this design. REPA [1] and REPA-E [2] propose to divide a diffusion model into an encoder (shallow layers) and a decoder (deeper layers),  where the encoder implicitly learns a representation that reconstructs the target.\nWe find that the above perspective also holds true in autoregressive models. Table A4 shows the linear probing performance of features extracted from different layers of LlamaGen-B, where the $6$-th layer achieves the best representation quality.\nThis demonstrates that autoregressive models exhibit behavior similar to diffusion models.\nSpecifically, for step $i$, the shallow subnetwork (encoder) of the autoregressive model maps preceding tokens into a high-level representation, which is then decoded by the deeper subnetwork (decoder) to predict the $i$-th token.\nTherefore, applying contrastive losses ($L_{step}$ and $L_{view}$) to the intermediate features is an effective strategy. \n\n**Table A4: Linear probing results of different layers in LlamaGen-B.**\n|Layer Index|3|6|9|12|\n|-|-|-|-|-|\n|LP Acc.(%)|8.42|**18.68**|16.90|12.39|\n\n*Reference*\n\n[1] Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think, ICLR 2025\n\n[2] REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformers, ICCV 2025"}, {"review_id": "9oo72iDOXq", "reviewer": "Reviewer_Paj7", "summary": "This paper investigates the challenges in visual understanding for autoregressive (AR) image generation models and proposes a self-guided training framework, ST-AR (Self-guided Training for AutoRegressive models), to address these issues. The authors identify three key limitations in AR models:  \n1. Local and conditional dependence: AR models overly rely on initial conditioning tokens and spatially adjacent information, as shown by attention map analyses.  \n2. Inter-step semantic inconsistency: Semantic information varies across different generation timesteps, leading to degraded global modeling.  \n3. Spatial invariance deficiency: Visual tokenizers lack invariance, causing ambiguous tokenization for slightly perturbed images.  \n\nST-AR integrates self-supervised learning techniques to enhance AR models without pre-trained representations:  \n- **Masked image modeling (MIM)** on attention maps to expand the receptive field.  \n- **Inter-step contrastive loss** to ensure semantic consistency across timesteps.  \n- **Inter-view contrastive loss** to align representations from different image augmentations.", "questions": "refer to the weakness", "limitations": "none", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1/ The research motivation of this work is to enhance the performance of auto-regressive models in understanding tasks. The authors found that next token prediction-based AR models have obvious defects in understanding tasks (Autoregressive models primarily rely on local and conditional information, Causal Attention Challenges Bi-directional Image Context Modeling). Based on these findings, the authors introduced methods commonly used in traditional visual representation learning, such as MIM and contrastive learning, which can effectively improve the performance of ImageNet class-conditional generation and downstream understanding tasks, such as ImageNet linear probing.\n\n2/ The authors conducted detailed experiments. In Table 1 & Table 2, the performance has significantly improved compared with the LlamaGen baseline. This method of adding additional constraints to align representations is similar in spirit to REPA and VAVAE( Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models), and it indeed can effectively improve the convergence speed and effect. However, I wonder whether this will make the model fall into a local optimum, because the model needs to learn both representation and generation during training. My personal view is that aligning to a pre-trained representation is more stable for optimization.\n\nWeaknesses:\n\n1/ Personally, I think this work somewhat resembles stitching together multiple established approaches that are known to work—such as MIM and contrastive learning, which are effective for representation learning. As such, I believe the work lacks substantial innovation.  \n\n2/ I am skeptical about the upper limits of pure visual representation learning. First, the validation of understanding tasks should be conducted on a broader range of multimodal tasks and downstream visual tasks. Currently, the effectiveness of visual encoders is mostly verified for downstream MLLM (Multimodal Large Language Model) tasks like VQA, grounding, and reasoning. However, in such MLLM tasks, approaches like MAE and contrastive learning have been abandoned in favor of semantic-aligned models such as CLIP and the SigLIP series. Therefore, my view is that integrating MIM and contrastive learning into autoregressive models does not represent the correct path.", "was_revised": true, "final_justification": "Thank you for the author's reply, which has basically solved my problem. I will raise the rating as accept.", "rebuttal": "We truly thank the reviewer for your detailed review and thoughtful comments in helping us enhance the quality of our paper. Below, we provide detailed point-to-point responses addressing each of your valuable suggestions:\n\n**Q1: Innovations of ST-AR**\n\nTo the best of our knowledge, we are the first to explore the representation learning capabilities of AR models and to identify the challenges of transferring next-token prediction to the visual domain.\nThe innovations of ST-AR can be summarized in two main aspects:\n- We introduce an effective methodology to analyze and identify the limitations of autoregressive models in visual representation learning by evaluating attention maps and linear probing performance.\n- We demonstrate that by appropriately incorporating well-established self-supervised learning techniques, the representation quality of AR models can be significantly improved, which subsequently enhances generation performance.\n\nFurthermore, the application of established self-supervised approaches to AR generation has not been explored before. ST-AR is the first unified framework that bridges representation learning and AR image generation.\n\nNotably, ST-AR outperforms methods that align AR features with pre-trained representations (e.g., REPA). In Table C1, we align the intermediate features of LlamaGen-B to those of either DINOv2-B or MAE-B, but both approaches perform worse than ST-AR. We attribute this to two reasons:\n- **Spatial misalignment between AR features and pre-trained representations.** In AR models, each time step uses the preceding token as input and predicts the next token as output, while pre-trained visual encoders focus on a single token. Directly aligning representations from these two paradigms is suboptimal. For more details, please refer to our response to **Reviewer rKee Q2**.\n- **Uncertainty in desired representations for AR models.** Different pre-trained visual encoders learn different types of representations, and it remains unclear what specific kinds of representations are optimal for AR models. ST-AR avoids introducing inductive bias by employing a self-guided approach that jointly trains representations and generation within a unified framework.\n\n**Table C1: Comparison between ST-AR and pretraining-based methods.**\n|Model|Pretrained Encoder|Epochs|FID|sFID|IS|Prec.|Rec.|\n|-|-|-|-|-|-|-|-|\n|LlamaGen-B||50|31.35|8.75|39.58|0.57|0.61|\n|LlamaGen-B + REPA|DINOv2-B|50|28.50|7.78|44.72|0.59|\n||MAE-B|50|31.27|9.12|40.27|0.58|0.60|\n|LlamaGen-B + ST-AR||50|**26.58**|**7.70**|**49.91**|**0.60**|**0.62**|\n\n**Q2: The importance of pure visual representation learning.**\n\nThe effectiveness of incorporating high-level visual representations into generative models has been well demonstrated[1][2].  Notably, REPA conducts the comparison between CLIP features and pure visual representations (e.g., DINOv2), showing that DINOv2 provides a stronger boost to generation performance than CLIP. **This suggests that the potential of pure visual representation learning is not inferior to vision-language alignment.**\n\nPure visual representation learning and vision-language alignment (e.g., CLIP) each have their own strengths and weaknesses, and they can be combined to obtain richer and more comprehensive representations. Enhancing image understanding through pure visual representation learning remains necessary and valuable.\n- **Representation quality.** Pure visual representation learning excels at capturing high-quality features of diverse objects. For example, DINO features have been effectively utilized for unsupervised instance segmentation[3][4]. In contrast, CLIP focuses on aligning image content with text embeddings, which can sometimes result in slightly inferior features.\n- **Data Requirements.** Pure visual representation learning requires only a large collection of images as training data, making it less resource-intensive in comparison to vision-language alignment methods like CLIP, which depend on costly textual annotations paired with images.\n\nIn practice, combining pure visual representation learning with vision-language alignment can significantly enhance visual understanding at various levels. For instance, SPHINX-X[5] leverages both DINOv2 and CLIP as its visual encoders to achieve superior performance.\n\n\n*Reference*\n\n[1] Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think, ICLR 2025\n\n[2] REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformers, ICCV 2025\n\n[3] Cut and Learn for Unsupervised Object Detection and Instance Segmentation, CVPR 2023\n\n[4] Freesolo: Learning to segment objects without annotations. CVPR 2022\n\n[5] SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models. ICML 2024"}, {"review_id": "vTbqhF7dNZ", "reviewer": "Reviewer_gtuF", "summary": "This paper investigates the underlying limitations of the standard next-token prediction paradigm for autoregressive (AR) image generation. The authors systematically identify three key properties that hinder the learning of high-level visual semantics: (1) over-reliance on local and conditional dependence, (2) semantic inconsistency across generation steps, and (3) a lack of spatial invariance in the visual tokens.\nTo address these issues, the paper proposes ST-AR (Self-guided Training for AutoRegressive models), a novel training framework that integrates self-supervised learning objectives directly into the AR training process. Specifically, ST-AR employs masked image modeling on attention maps to encourage a wider receptive field and uses two forms of contrastive learning (inter-step and inter-view) to enforce semantic consistency. A key advantage of this approach is that it enhances the model's understanding and generation quality without requiring pre-trained representation models or altering the standard AR sampling strategy at inference time. Experiments on ImageNet show that ST-AR significantly improves both image understanding (linear probing accuracy) and generation quality (FID scores) across various model scales.", "questions": "1. In the final loss function (Eq. 8), the weights for the new loss terms are set to α = 1.0 and β = 0.5. How sensitive is the model's performance to these specific hyperparameter choices? Was a sweep performed to find these values, and could you provide some intuition on how the balance between the standard AR loss and the new self-supervised losses affects the training dynamic?\n\n2. The framework combines a reconstruction-style objective (L_MIM) with consistency-based contrastive objectives (L_step, L_view). Did you observe any interesting interplay or potential conflict between these losses during training? For example, does the MIM objective, which forces the model to use broader context, directly aid the inter-step consistency objective, or are they largely independent mechanisms for improvement?\n\n3. The contrastive losses are applied at a middle layer of the network (e.g., the 6th layer for LlamaGen-B). The ablation in Table 5 shows this is optimal. Do you have any intuition for why this \"middle-ground\" is more effective than applying the constraint on earlier (more textural) or later (more semantic) features? Does this suggest that ST-AR is primarily helping to structure the mid-level representations of the network?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n\n1. This work systematically investigate into why autoregressive models struggle with visual semantics. The authors go beyond simply proposing a new method and provide a clear, well-supported diagnosis of the core problems. This principled analysis provides a strong foundation and clear motivation for the proposed ST-AR framework.\n\n2. The proposed ST-AR framework is a novel and clever integration of self-supervised learning techniques (MIM and contrastive learning) into the autoregressive training loop. \n\nWeakness\n\n1. The authors correctly identify \"increased training costs\" as a limitation in the conclusion. However, this is a significant practical concern that warrants a more detailed discussion. The ST-AR framework introduces a teacher network, requires processing multiple views of each image, and computes three additional loss terms. This will substantially increase both the computational and memory requirements for training compared to the baseline. \n\n2. This paper positions itself as an approach that avoids reliance on pre-trained representation models (a key advantage). However, a stronger argument could be made by directly discussing or comparing against methods that do leverage such models (e.g., by using a pre-trained VAE or distilling knowledge from a model like DINO/CLIP). A brief discussion on the conceptual pros and cons (e.g., ST-AR's \"self-guided\" nature vs. the potential power of a large-scale pre-trained teacher) would better situate the work within the broader landscape of improving semantic understanding in generative models.", "was_revised": false, "final_justification": "", "rebuttal": "We truly thank the reviewer for your detailed review and thoughtful comments in helping us enhance the quality of our paper. Below, we provide detailed point-to-point responses addressing each of your valuable suggestions:\n\n**Q1: Training cost of ST-AR.**\n\nIn Table C1, we provide a fair comparison of the training costs for LlamaGen, REPA [1], and ST-AR.\nAll experiments are conducted under the same training settings, using 8 A100 GPUs with a global batch size of $256$ and an input image resolution of $256\\times256$. We report the average runtime per training iteration and peak memory usage.\nCompared to the baseline LlamaGen and the REPA (which requires a pre-trained image encoder), ST-AR incurs slightly higher training costs but delivers significant performance improvements.\nReducing the training cost of ST-AR will be an important focus of our future work.\n\n**Table C1: Comparison of training costs and comparison between ST-AR and pretraining-based methods.**\n|Model|Pretrained Encoder|Epochs|FID|sFID|IS|Prec.|Rec.|Training Latency (s/iter)|Peak GPU Memory (GB)|\n|-|-|-|-|-|-|-|-|-|-|\n|LlamaGen-B||50|31.35|8.75|39.58|0.57|0.61|**0.0493**|**~7**|\n|LlamaGen-B + REPA|DINOv2-B|50|28.50|7.78|44.72|0.59|**0.62**|0.0658|~8|\n||MAE-B|50|31.27|9.12|40.27|0.58|0.60|0.0662|~8|\n|LlamaGen-B + ST-AR||50|**26.58**|**7.70**|**49.91**|**0.60**|**0.62**|0.0698|~10|\n\n**Q2: Comparison with methods requiring a pre-trained image encoder.**\n\nWe follow REPA[1] to align the intermediate features of LlamaGen-B with the representations from DINOv2-B and MAE-B, respectively. \nAs shown in Table C1, LlamaGen-B with REPA demonstrates better generation performance compared to the baseline but falls short of ST-AR.\n\nWe analyze the pros and cons of ST-AR and pretraining-based methods from two perspectives:\n1. **Spatial misalignment when aligning intermediate features from autoregressive models with pre-trained visual representations.** For position $i$, an autoregressive model involves two adjacent tokens: the $(i-1)$-th token as the input and the $i$-th token as the predicted output.\nIn contrast, visual encoders like DINOv2 extract features from the $i$-th image patch.\nForcing alignment may introduce inductive bias into the autoregressive model. In comparison, ST-AR does not rely on pretrained visual encoders, thereby avoiding the spatial misalignment.\n2. **Representation quality.** While visual encoders pretrained on larger-scale datasets may learn better semantic representations, these representations do not necessarily contribute to learning for image generation. In contrast, ST-AR unifies representation learning and image generation, allowing it to directly learn features that are more beneficial for autoregressive image generation.\n\n**Q3: The effect of $\\alpha$ and $\\beta$ and the balance between AR loss and self-supervised losses.**\n\nIn Table C2, we examine the effect of $\\alpha$ and $\\beta$ on generative performance.\nWhen training for 50 epochs:\n- ST-AR performs best when $\\alpha=1.0$ and $\\beta=0.5$.\n- $\\alpha$ is the weight of the MIM loss. Reducing the value of $\\alpha$ decreases the model's effective receptive field, leading to poorer generation quality.\n- $\\beta$ is the weight of the two contrastive losses. A lower value ($0.25$) makes it difficult to learn semantic information, while values greater than $0.5$ might hinder the learning of the autoregressive loss.\n\nFor training dynamics, we evaluate the performance of ST-AR with different $\\alpha$ and $\\beta$ at $20$ epochs and $50$ epochs. As in Table C2, the best performance is achieved by set $\\alpha=1.0$ and $\\beta=1.0$ at $20$ epochs. This indicates that larger $\\alpha$ and $\\beta$ contribute to learning better semantic representations, which accelerates the convergence of the autoregressive model. However, larger contrastive loss may hinder the learning of autoregressive loss, leading to inferior performance of setting $\\{\\alpha=1.0, \\beta=1.0\\}$ at $50$ epochs. \n\n**Table C2: Ablation study for $\\alpha$ and $\\beta$.**\n|Epochs|$\\alpha$|$\\beta$|FID|sFID|IS|Prec.|Rec.|\n|-|-|-|-|-|-|-|-|\n|50|1.00|0.25|26.72|8.17|48.55|**0.61**|0.61|\n||1.00|0.50|**26.58**|**7.70**|**49.91**|0.60|**0.62**|\n||1.00|0.75|26.67|7.71|48.41|0.60|0.61|\n||1.00|1.00|27.32|7.86|47.59|0.59|0.61|\n||0.75|0.50|27.07|**7.70**|47.56|**0.61**|0.61|\n||0.50|0.50|27.57|7.71|46.70|**0.61**|0.59|\n||0.25|0.50|27.92|8.08|44.54|**0.61**|0.59|\n|20|1.00|0.25|38.09|10.28|33.83|0.52|0.61|\n||1.00|0.50|36.88|9.12|35.38|0.52|0.62|\n||1.00|0.75|36.28|9.67|36.27|**0.54**|0.61|\n||1.00|1.00|**35.27**|**8.45**|**36.79**|0.53|**0.63**|\n||0.75|0.50|38.35|10.42|35.05|0.52|0.62|\n||0.50|0.50|38.63|9.96|34.54|0.52|0.62|\n||0.25|0.50|39.31|10.26|33.66|0.51|0.62|\n\n**Q4: The interaction between the reconstruction objective and the contrastive objectives.**\n\nThe interaction between the reconstruction objective and contrastive objectives exhibits both conflict and mutual benefits.\n\n**Conflict.** MIM and contrastive learning capture different levels of semantic information, resulting in a unique challenge of representation conflict. The inter-step and inter-view contrastive losses are designed to address semantic inconsistencies across steps and views, respectively, promoting the learning of global, image-level semantics. In contrast, the MIM loss trains the autoregressive (AR) model by reconstructing the correct semantics of masked tokens, emphasizing token-level semantics. To mitigate this conflict, we compute the contrastive losses on intermediate-layer features and the MIM loss on final-layer features. As in the Table 5 of our manuscript, this design can bring the best performance. \n\n**Mutual Benefits.** As shown in Figure A1 of the supplementary material, contrastive objectives face limitations in expanding the effective receptive field, which restricts the learning of wide-range high-level semantics (e.g., objects). The reconstruction objective effectively addresses this issue. On the other hand, the reconstruction objective alone struggles to capture precise semantic entities, whereas the contrastive objectives complement and expand the capabilities of MIM, leading to more robust learning of semantic representations.\n\n**Q5: Applying contrastive losses to the middle layer.**\n\nThe contrastive losses in ST-AR aim to align high-level semantic information across different timesteps and views. Notably, the most semantically meaningful features exist in the middle layer of the autoregressive model.\n\nThe autoregressive model processes preceding tokens as input to predict the next token, which can conceptually be divided into two stages:\n1. The shallow subnetwork maps the preceding tokens into a high-level representation.\n2. The deeper subnetwork decodes this high-level representation into the next token.\n\nThe most semantically meaningful features are produced in the middle layer rather than the final layer. This claim is supported by the linear probing results shown in Table C3.\n\n**Table C3: Linear probing results of different layers in LlamaGen-B.**\n|Layer Index|3|6|9|12|\n|-|-|-|-|-|\n|LP Acc.(%)|8.42|**18.68**|16.90|12.39|\n\n[1] Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think. ICLR 2025"}, {"review_id": "mv8oQy1yn4", "reviewer": "Reviewer_qqSW", "summary": "This paper investigates the limitations of autoregressive (AR) models in image generation, particularly their poor visual understanding (based on LlamaGen). The authors identify three key issues: (1) over-reliance on local and conditional information (each next token relies on only spatial neighbors), (2) semantic inconsistency across generation steps (seeing all tokens have lower linear probe accuracy than seeing partialy), and (3) lack of spatial invariance in visual tokens (e.g., VQGAN encodes differently if image is shifted). To address these, they propose ST-AR (Self-guided Training for AR), a training framework that incorporates masked attention modeling and two contrastive learning objectives (inter-step and inter-view). These augment the standard AR next-token loss without modifying inference. ST-AR significantly improves both visual representation quality (e.g., linear probing accuracy from 21% to 55%) and generation quality (e.g., up to 49% FID improvement), all without relying on external pretrained models.", "questions": "1. I might have missed this information from the paper, can you quantify the training overhead introduced by ST-AR?\n2. Since contrastive loss is applied, what is the effect of batch size? \n3. Can you discuss how ST-AR be extended to other AR domains like 3d or video?\n4. I notice the high norm issue on the first patch, would adding register tokens help? (Also, whether adding this register token have any new insight?)", "limitations": "yes. computation cost but it is unavoidable.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Quality:\n- The paper is technically sound, well-formulated, and experimentally robust. Very insightful analysis. The integration of masked attention and contrastive learning into AR training is effective and carefully designed. The experiments on ImageNet are extensive, showing strong improvements across multiple AR model sizes and metrics, with clear ablation studies validating each component.\n\nClarity:\n- The writing is clear and well-organized. Visualizations and architecture diagrams support understanding. The authors provide precise explanations of why applying attention masking and contrastive objectives.\n\nOriginality:\n- While the self-supervised tools used (e.g., contrastive loss, masked modeling) are established, their application to AR image generation is novel and well-motivated. The diagnosis of AR's semantic weaknesses is insightful and targeted design of ST-AR is a key original contribution.\n\nSignificance:\n- This work has broad implications for scaling AR models across modalities. Representation learning is always the key to better model, by improving representation learning within AR models, ST-AR makes them more viable competitors to diffusion models, especially given the efficiency and modularity of AR transformers.\n\nWeaknesses:\n\n1. ST-AR increases training complexity and compute cost (EMA teacher, additional losses).\n2. The careful composition of known components is a bit trivial and straightforward, although effective.\n3. Text-to-image generation experiment detail in appendix is not clear enough and perhaps can move some results to main text.\n4. Fig. 3 can be improved, the loss parts can't be understood easily without reading the text.", "was_revised": true, "final_justification": "thanks authors for their rebuttal. concern addressed and i am keeping my score as Accept.", "rebuttal": "We truly thank the reviewer for your detailed review and thoughtful comments in helping us enhance the quality of our paper. Below, we provide detailed point-to-point responses addressing each of your valuable suggestions:\n\n**Q1: Training overhead of ST-AR.**\n\nIn Table D1, we quantify the training overhead introduced by ST-AR. All experiments are conducted using 8 NVIDIA A100 GPUs (80GB each), with a global batch size of 256 and an input image resolution of $256 \\times 256$. We report the average training time per iteration as well as the peak memory consumption during training.\nThe training overhead of ST-AR is higher than the baseline LlamaGen but comparable to that of REPA. However, ST-AR achieves significantly better performance gains. Reducing the training cost of ST-AR will be a key direction for our future work.\n\n**Table D1: Comparison of training costs.**\n|Model|Epochs|FID|sFID|IS|Prec.|Rec.|Training Latency (s/iter)|Peak GPU Memory (GB)|\n|-|-|-|-|-|-|-|-|-|\n|LlamaGen-B|50|31.35|8.75|39.58|0.57|0.61|**0.0493**|**~7**|\n|LlamaGen-B + REPA|50|28.50|7.78|44.72|0.59|**0.62**|0.0658|~8|\n|LlamaGen-B + ST-AR|50|**26.58**|**7.70**|**49.91**|**0.60**|**0.62**|0.0698|~10|\n\n**Q2: Effect of Batch Size**\n\nWe evaluate the impact of batch size on generation performance and representation quality. As shown in Table D2, a larger batch size is more beneficial for semantic information learning, resulting in higher linear probing accuracy. However, in terms of generation quality, a batch size of 512 performs worse than 256. We attribute this to the reduced number of training iterations when using a larger batch size.\n\n\n**Table D2: Effect of batch size.**\n|Batch size|FID|sFID|IS|Prec.|Rec.|LP Acc.(%)|\n|-|-|-|-|-|-|-|\n|128|28.83|9.46|46.47|**0.60**|0.61|45.03|\n|256|**26.58**|**7.70**|**49.91**|**0.60**|**0.62**|45.27|\n|384|27.70|8.73|47.25|0.59|**0.62**|45.81|\n|512|29.29|8.49|45.38|0.57|**0.62**|**46.13**|\n\n**Q3: Extending ST-AR to other AR domains.**\n\nOne of the key contributions of our work is establishing a methodology for analyzing autoregressive generation models. By quantifying the scope of the attention map and evaluating linear probing performance, we gain insights into the intrinsic mechanisms of autoregressive models.\n\nTo extend ST-AR to other domains, such as video generation, conducting similar analyses is a crucial first step. Intuitively, for video generation, we would need to further examine the attention weights between frames and assess the semantic consistency across frames. Based on these insights, we could then introduce additional high-level inter-frame representation alignment losses to ST-AR, aiming to enhance the video generation performance of baseline models.\n\n**Q4: High norm issue on the first patch**\n\nThe input at the first time step is a conditional vector that guides the generation process. It is reasonable for this vector to always have a high norm. \n\nTo validate the effect of register tokens, we inserted 4 register tokens between the first token (conditional vector) and the second token (the first image patch). The image generation process starts from the last register token.\nAs shown in Table D3, the introduction of register tokens leads to a slight performance improvement. However, visualizations of the attention maps reveal that the conditional vector still exhibits a high norm.\n\n**Table D3: Effect of register tokens.**\n||FID|sFID|IS|Prec.|Rec.|\n|-|-|-|-|-|-|\n|w/o registers|26.58|7.70|49.91|**0.60**|0.62|\n|w/ registers|**25.98**|**7.69**|**50.78**|**0.60**|**0.63**|"}]

Author rebuttal:

Meta-review: This paper proposes a framework ST-AR for self-supervised training of autoregressive models in the image domain. The approach is motivated by three issues with existing approaches (local and conditional dependence; inter-step semantic inconsistency; and spatial invariance deficiency), which are addressed in ST-AR with corresponding losses. 

Reviewers are quite positive about this work, listing the motivation, writing, effectiveness of the method, and quality of the experiments as strengths. The main limitation of ST-AR is that it increases computational cost, which the authors further investigated in their rebuttal. Another concern is that, although ST-AR is effective, it is a recombination of existing ideas and therefore less novel. That said, after rebuttal, all of the reviewers are in favor of acceptance and I agree that this paper makes a valuable contribution.

I note that one reviewer, reviewer gtuF, did not participate in in the author-reviewer discussion, which was taken into account before making a recommendation.
Decision: Accept (poster)