reAR: Rethinking Visual Autoregressive Models via Token-wise Consistency Regularization
Authors: Qiyuan He
Yicong Li
Haotian Ye
Jinghao Wang
Xinyao Liao
Pheng-Ann Heng
Stefano Ermon
James Zou
Angela Yao
Venue: ICLR 2026 Poster (2026)
Keywords: Visual Generation
Autoregressive Model
Primary area: generative models
TL;DR: reAR is a simple regularization method that fixes generator-tokenizer inconsistency in visual autoregressive models, greatly improving quality without altering tokenizer or inference pipeline.

Abstract: Visual autoregressive (AR) generation offers a promising path toward unifying vision and language models, yet its performance remains suboptimal against diffusion models. Prior work often attributes this gap to tokenizer limitations and rasterization ordering. In this work, we identify a core bottleneck from the perspective of generator-tokenizer inconsistency, i.e., the AR-generated tokens may not be well-decoded by the tokenizer. To address this, we propose reAR, a simple training strategy introducing a token-wise regularization objective: when predicting the next token, the causal transformer is also trained to recover the visual embedding of the current token and predict the embedding of the target token under a noisy context. It requires no changes to the tokenizer, generation order, inference pipeline, or external models. Despite its simplicity, reAR substantially improves performance. On ImageNet, it reduces gFID from 3.02 to 1.86 and improves IS to 316.9 using a standard rasterization-based tokenizer.  When applied to advanced tokenizers, it achieves a gFID of 1.42 with only 177M parameters, matching the performance with larger state-of-the-art diffusion models (675M).

REAR: RETHINKING VISUAL AUTOREGRESSIVE MODELS VIA GENERATOR-TOKENTZA CONSISTENCY
REGULARIZATION
Qiyuan He², Yicong Li Haotian Ye², Jinghao Wang', Xinyao Liao
Pheng-Ann Heng?, Stefano Ermon², James Zou2, Angela Yao
National University of Singapore "Stanford University "The Chinese University Hong Kong
ABSTRACT
Visual autoregressive (AR) generation offers a promising path toward unifying vision and language models, yet its performance remains suboptimal against dif- fusion models. Prior work often attributes this gap to tokenizer limitations and rasterization ordering. In this work, we identify core bottleneck from the per- spective of generator-lokenizer inconsistency, i.c., the AR-generated tokens may not be well-decoded by the tokenizer. To address this, we propose reAR, simple training strategy introducing token-wise regularization objective: when predict- ing the next token, the causal transformer also trained to recover the visual em- bedding of the current token and predict the embedding of the target token under a noisy context. requires no changes to the tokenizer, generation order, inference pipeline, external models. Despite its simplicity, reAR substantially improves performance. On ImageNet, it reduces gFID from 3.02 to 1.86 and improves IS to 316.9 using a standard rasterization-batt based tokenizer. When applied advanced tokenizers, it achieves a gFID of 1.42 with only 177M parameters, matching the performance with larger state-of-the-art diffusion models (675M).
Sehe Sequence
Embedding Space
GFID Dower better)
Tokenizer Codsbook
ABSOCAAN
Tokemizer
vit
Ah-Geeereied Jecvence
17
19
X
Decoder-only
Decooder
Dovedivtt
17
42
60
98
(higher
(a) Visual autoregressive generation suffers from gencrator-tokee tokenizer in- consistency: (1) Due to exposure bias, the AR model is more likely to generate token sequences unseen by the tokenizer; (2) Being embedding unaware, the embedding sequence of the generated discrete tokens may in an unnatural pose,
(b) With generator-tokenizer con- sistency regularization, reAR with fewer parameters significantly im- proves vanilla AR (gFID: 3.02 1.86, 256.2 316.9) even surpasses methods based on advanced tokenization and sophis- ticated generative paradigm.
also be unfamiliar to the tokenizer, resulting with its lower body flipped and the belly facing upward. The top and bot- tom images can still appear similar despite differing token indices, since distinct token sequences may map to nearby embeddings.
Figure tenerator-tokennzer inconsistency bottleneck in the visual autoregressive model.
INTRODUCTION
Autoregressive (AR) models, using a decoder-only transformer with the objective of next token pre- diction, are state-of-the-art for natural language generation (Team al., 2023; Achiam et al., 2023). For image generation, however, AR models are less competitive than diffusion models (Dhariwal
"Corresponding Authors.

---

& Nichol, 2021; Peebles & Xie, 2023; Ma et al., 2024; Yu et al., 2024c). There is great interest in advancing visual autoregressive models to unify the language and visual modalities into a single generative framework (Bai et al., 2024; Team, 2024; Chung et al., 2024).
Scrutinizing the current design in visual AR, the dominant paradigm is to convert images into dis- crete tokens and train autoregressive model the converted token sequences. Specifically, tokenizer is trained to split an image (or the feature) into patches and utilizes them into a sequence of discrete tokens (Esser et al., 2021; Sun et al., 2024; Luo et al., 2024), which it can use to reconstruct the original image. A decoder-only transformer using a causal mask is then trained on this token se- quence in raster-scan order with the objective of next-token prediction. Unfortunately, this paradigm typically results in suboptimal performance compared to the diffusion model (Dhariwal & Nichol, 2021; Peebles & Xie, 2023; Ma al., 2024; Yu et al., 2024c). Previous works have analyzed the performance gap from the perspective of tokenization, including token decomposition (Tian et al., 2024; Yu et al., 2024b; Bachmann et al., 2025; Pan al., 2025) and sequence order (Pang et al., 2025; Yu et al., 2024a), rather than the whole system of visual autoregressive generation.
In this work, we provide a unified perspective on the key bottleneck of visual AR through the lens of generator-tokenizer inconsistency, which refers to the challenge that the autoregressive model might generate a token sequence that is hard for the tokenizer to decode back to an image. Specifi- cally, we examine two sources of the inconsistencies inherited from the generated token sequence.
Firstly, the generated token sequence can be unseen by the tokenizer due to exposure bias. In au- toregressive training, each token is predicted given the ground-truth context (teacher forcing), but at inference, the context consists of the model's own predictions. Early mistakes then compound and lead to sequences never observed during training. While exposure bias is well studied in lan- guage (Bengio et al., 2015; Wang & Sennrich, 2020), it is amplified in visual AR. Text tokens are themselves the final output, so even an unseen sequence may still be semantically coherent. By contrast, visual tokens are decoded into images: single wrong token can corrupt future predictions and decode into a token sequence never seen by the tokenizer during training. spreading structural artifacts across the image. As shown in Figure (a)) an early misprediction (e.g.. 42' -4"" cascades through subsequent tokens and yields a cat in an unnatural pose with a different coat color.
Secondly, the AR model suffers from embedding unawareness. During training. it optimizes only the discrete token indices without considering how these tokens are embedded by the tokenizer. However, the decoded image quality depends on the embeddings of the generated tokens rather than their indices alone, as shown in Figure 1(a). This unawareness leads to two issues: (i) even if two tokens are close in the embedding space, the model can only infer this relation indirectly from co-occurrence statistics, which is data-inefficient. and (ii) the embedding of an incorrect token is unconstrained by the ground-truth embedding, which can cause the overall sequence embedding to drift far from the training distribution of the tokenizer decoder. As illustrated in Figure ((a), although the purple and red sequences contain the same number of incorrect tokens, the one with embeddings closer to the ground truth generates a decoded image of higher quality.
In this regard, we propose re,R, unified training framework that explicitly regularizes the model toward pokenizer-friende behavior. Concretely, we introduce two complementary strategies: 1) Noisy Context Regularization that exposes the model to perturbed context during training, re- ducing its reliance on clean contexts and improving robustness to imperfect histories at test time, thereby alleviating the model's tendency to generate unseen token sequence; 2) Codebook Embed- ding Regularization that aligns the generator's 's hidden states with the tokenizer's embedding space, which encourages the generator to be aware of how tokens are decoded into visual patches. By learn- ing to predict the embeddings rather than only discrete indices, even if the generator generates an unseen token sequence, the corresponding embedding sequence is optimized to be more compatible with the tokenizer. Combining them together, the token-wise consistency regularization can guide visual AR to be friendly to the tokenizer by predicting the visual embedding in robust manner.
Building on reAR, conduct extensive experiments comparing it against other generative frame- works. To show that reAR generalizes beyond specific tokenizers, we apply it to non-standard designs such as TiTok (Yu et al., 2024b) and AliTok (Wu et al., 2025). When combined with stan- dard rasterization-order AR, reAR outperforms prior autoregressive methods even when those rely on sophisticated tokenizers, as Figure (b) shows. Under the same model size and training budget, it also surpasses alternative paradigms such as MAR (Li et al., 2024), VAR (Tian et al., 2024). and
2

---

Medl Pradicion
Tolenizer Decodar
Cudrbbok Entendiiig Royuletisiion
Toraforee Block
MLP
Couscl Transformar Black
Couccl Tendoree Block
Couacl Tranaformat Block
Courcl Foralorer Block
Couacl Porutoome Block
MLP
Courci Tenalomer Block
Cousol Transformer Block
Dodibook Leniup
Coussl Transformer Block
Crooder
tegulotbation
(a) reAR Training
(b) reAR Inference
Figure 2: Overview of reAR, plug-and-play framework that agnostic the visual tokenizer.
SiT (Ma et al., 2024). Furthermore, when paired with a tokenizer tailored for causal AR model- ing (Wu et al., 2025), reAR achieves FID 1.42 with only 177M parameters conerrtttii with the diffusion model REPA, which requires external representations and 675M parameters.
Our contributions can be summarized as follows:
We identify the inconsistency between generator and tokenizer, where tokenizer fails to decode the generated token sequence, as the bottleneck of visual autoregressive generation;
We propose reAR, plug-and-pla, training regularization that introduces visual inductive bias from the tokenizer and alleviates exposure bias to train the visual autoregressive model;
We demonstrate that reAR significantly improves visual autoregressive generation across different tokenizers (e.g., VQGAN, FID improves from 3.02 to 1.86) and surpasses more sophisticated generative models, using far fewer parameters.
2 PRELIMINARIES
Visual autoregressive generation is commonly divided into two components: (1) A visual tokenizer to tokenize the image; (2) An autoregressive model to sample the token sequence.
Visual Tokenizer. Visual tokenizers compress image pixels into discrete token sequences. The most commonly adopted methods are patch-based tokenizers (Esser et al., 2021; Sun et al., 2024; Yu et al., 2023; Chang et al., 2022). The tokenizer includes three parts: Encoder E, Quantizer Q and Decoder DD Formally, a given image I Hx H is converted to feature & E Rexhxw with the encoder where h < H, < It's then processed into quantized embedding 2 [xx via the quantizer Q and decoded back to reconstruct image i by the decoder D:
E(I, x = Q(2), i- D(22)
(1)
The vector quantization is performed element-wist with codebook 2 (21, ..., 2x] gexhxw by looking up the closest entry. Formally:
,,,
29, arg min 2. zx|l. 2
Xij arg min |2. zx/l. i= 1....,, = w. (2)
kE(I....KK
where Xij forms the discrete token (indices such 17 and 73). In the standard approach, it's arranged into ID token sequence via row-major rasterization order, i.e. (Xi....., Xie, X21......... Xpl............................. Xpw }. The autoregressive model can then be trained on it..
Autoregressive Model. To model the distribution of sequence of signal X = {x1, X2,....... X,), the autoregressive model pa aims maximize the likelihood of the next token under teacher forcing:

---

20 boteet <Te
Groznak-ruth (CTR- (00%))
Conteet (CTR 18.5%
Perkect Contect (CTR (l.4%))
Grooné-raat) 199%)
Tsneker Forong (CTI 1089%
Repleedd Prodiction (CTR 9U8%))
(a) Tokenizer is sensitive to the error of generated tokens from exposure bias
(b) Tokenizer is sensitive to the embedding of generated tokens
Figure 3: Token sequence with the same correct token ratio (CTR) under teacher forcing can be decoded into images with different quality. Under the same CTR, (a) The images decoded from imperfect context is much less similar the ground truth than the from perfect context; (b) Replacing incorrect token with other incorrect tokens but with more similar embedding of the correct token, the generated image can be more similar to ground truth than original prediction.
arg max log po (XI:N) = arg max
log po (x; X2,..., Xi-1)
During inference, the model then decodes the sequence one by one. The ith token sampled from the context X1:t-1 by x, ~ p(- X1:1-1) under free running. In visual autoregressive generation, after sampling sequence & from po, it's decoded into as the final generated image by the tokenizer decoder D.
3 REAR: REGULARIZING CONSISTENCY IN VISUAL AR
Different from natural language, & is not the final generated result in visual autoregressive gener- ation. Therefore, inconsistency between the generator and decoder can lead to unsatisfying results even if the autoregressive model is trained well. For example, when sampling an unseen or rare se- quence & in the training dataset of the tokenizer, it's possible that the sequence cannot be properly decoded by decoder D and affect the final generated results. We hypothesize that the inconsistency between the tokenizer and generator is the main obstacle to performance. A promising solution is to train the AR model such that it can generate token sequence that is friendly the tokenizer.
To verify our hypothesis, we investigate and quantitatively analyze how the existing visual autore- gressive model suffers from the inconsistency in Section 3.1. Based on the observations, propose reAR: R: regularizing token-wise consistency of visual AutoRegressive generation, a plug-and- play regularized training method designed for visual autoregressive model. In summary, reAR introduces visual embedding looked up from discrete tokenizer to the hidden feature of the gener- ator under a noisy context. Despite its simplicity, reAR allows the autoregressive model leverage visual signals that are compatible with the tokenizer and reduce inconsistent behavior significantly.
3.1 UNDERSTANDING THE BOTTLENECK OF VISUAL AUTOREGRESSIIN GENERATION
The performance of an autoregressive model can be assessed through the quality of generated tokens f1.n with the ground-truth sequence Tim by the correct token ratio (CTR), where 2, While CTR is widely used to indicate the performance,
CTR( Fi.( = n E the token sequence only an intermediate representation in visual autoregressive generation, and the final output is actually the decoded image. To evaluate end-to-end quality, we instead measure I.PISS (Zhang et al., 2018) between the images decoded from two token sequences. We consider that the inconsistencies between training and inference can be observed from inconsistencies be- tween CTR and LPIPS. the following, two controlled experiments demonstrate that generated token sequences with similar CTR result in images of different quality. This inconsistency is also reflected by other metrics for the AR model, such as perplexity, with details the Appendix B.
Amplified exposure bias. Exposure bias is well-known issue in sequence models (Bengio et al., 2015: Wang Sennrich, 2020): during training with teacher forcing, the model predicts the next token given the ground-truth context, whereas at inference must condition its own predictions,
4

---

which may contain errors. In visual autoregressive generation, we hypothesize that the visual tok- enizer amplifies this effect since exposure bias leads to more unseen token sequences and spreads structural error in the pixel space. To verify it, consider token sequence Ti:w decoded from an image with a ground-truth token ratio r E [0, 1]. We compare two decoding protocols: (1) Perfect context (front-loaded). Fix the first [rn] tokens to ground truth, i.e., [|:[[n] and let the AR model generate the remainder. This minimizes exposure bias for a given r, since the context remains clean until step [rn]. (2) Imperfect context (uniformly interleaved). Sample mask M S {L,....?7 n with M = |nn] uniformly at random. During decoding at the tth step, it uses ground truth token 2; if t E M, otherwise samples the token from the AR model. This introduces earlier contamination of the context, thereby increasing exposure bias compared to Perfect context with similar CTR.
Since both protocols fix the number of ground-truth tokens at [ra], any difference downstream quality reflects sensitivity to exposure bias rather than token-level accuracy. Results are shown in Figure (a). For comparable CTR, imperfect context consistently yields higher LPIPS than perfect context. Qualitatively, an imperfect context leads to images that deviate significantly from the original, whereas a perfect context yields better prediction, i.c., the layout of the dog is more similar. This highlights that alleviating exposure bias is essential in visual autoregressive generation.
Embedding unawareness. During training, the AR model is optimized only for token correct- ness, whereas the tokenizer decoder operates in embedding space. We hypothesize that even if predicted token is incorrect, if its embedding is close to that of the correct token, the decoded im- age may still retain high visual quality. To verify this, we introduce replacement ratio r'. Given a ground-truth sequence Iim. the AR model predicts ă Iim with teacher forcing. For each incor- rect prediction (#; of xz), we replace à; with probability r' by another incorrect token x² + I; whose embedding 2%, is closest to the correct embedding zº; under cosine similarity d(:,-), i.e., 2%, = arg min d(=", 2²)). This replacement leaves CTR unchanged.
Figure 3(b) presents the results. As r' increases, the average embedding similarity improves and LPIPS decreases markedly. Qualitatively, as shown on the right of Figure 3(b), such replacements without altering CTR can yield decoded images more faithful to the ground truth (e.g., clearer prediction of shirts and human legs). This suggests that incorporating tokenizer embeddings into the training of the AR model could potentially improve consistency between them.
A straightforward approach to increase generator-tokenizer inconsistency is to reuse the tokenizer's codebook embeddings in the embedding layer or prediction head of the AR model. However, this method commonly results in suboptimal performance without sophisticated design of the tok- enizer (Weber et al., 2024; Yu et al., 2023). We hypothesize that such rigid integration not ideal: it may constrain the scalability of a large AR model with smaller tokenizer, and the codebook embeddings themselves may not be the optimal representations for the primary task of next-token prediction. It's required to introduce the embedding into the model in less constrained manner.
3.2
CONSISTENCY REGULARIZATION
These findings reveal training-inference inconsistencies: maximizing correctness to predict token indices alone is insufficient for visual AR models. Proper inductive bias is required to train the gen- erator such that the generated token sequence is more consistent with the tokenizer during inference. Meanwhile, injecting this inductive bias should remain lightweight to ensure good cross- aros-trccitecture generalization and full compatibility with existing AR training and inference pipelines.
To address these challenges, reAR introduces token-wise consistency regularization during training of the visual AR model. Specifically, the decoder-only transformer trained to perform next-token prediction under noisy contexts, while its hidden representations are regularized by the visual em- beddings of the correct current token at shallow layer and the correct next token at a deep layer. This encourages the AR model to interpret current tokens similar to the tokenizer while improving robustness to exposure bias, then predicting the next token embedding compatible with the decoder. Below we denote the AR model as po. the tokenizer codebook as Z 21, .2x), the training dataset as X and the discrete token sequence as x = (x1, X2, x,).
Noisy Context Regularization. While techniques such as scheduled sampling (Bengio et al., 2015) can mitigate exposure bias, we choose a simple approach that preserves parallel training of the transformer. Specifically, we apply uniform noise to the input, denoted by (x x). Formally:
5

---

x; = (1 b;) X + by ug by ~ Bernoulli(c), u, Uniform ((1..... k)).
(4)
where b; is Bernoulli random variable with probability and u, is sampled uniformly from the codebook indices. practice, the choice strongly affects training stability. To ensure the AR model is exposed to sequences with varying noise levels, sample sequence, where t E [0, 1] denotes the normalized training progress. Here, f [0, 1] - [0, 1] is an annealing schedule that controls the maximum noise level over training. The AR model is then trained to predict the next correct token based on the noisy context. Formally:
(0, for each token
(A) = - (10)))
log po (X, & 1,,,,...,
(5)
Empirically, we found that the annealing uniform noisy augmentation can stabilize training com- pared noisy augmentation with a fixed ratio. We provide detailed ablation Section 4.3.
Codebook Embedding Regularization. Instead of directly applying codebook embedding, we propose add a regularization task recover current embedding and predict next embedding. Specifically, we apply a trainable MLP layer h& to project the hidden feature into the target space in the same dimension of visual embedding. For the simplicity of notation, we use w (X) to represent
the feature at the shallow layer and w (&) as the one at the deep layer 1'. To be aligned with the design of decoder-only transformer, the objective of the shallow layer (x)) is to predict the emhedding of current token and w (x) is to predict the next token. Formally:
(f, E Xyrinn Smag (0,)(())))
(()((( 2xx) + ((b (x)), 2x.11)
(6)
where d(.,-) is cosine distance to evaluate the distance between different features, ht refers the mapping from the feature of the jth current token the embedding space, 2x; is the embedding of current token and 2x;11 is the embedding of the next token looked up from the codebook. In the implementation, we apply the regularization bedding of the tokenizer in the vanilla AR (i.c, the 1st layer for encoding regularization and the 15th layer for decoding regularization) to avoid potential conflicts on the primary task of next-token pre- diction. Intuitively, we place the encoding regularization at the first layer to ensure no downstream information required for predicting the next token suppressed or overwritten, and apply the decod- ing regularization in a deep but not final layer, since the raw tokenizer embedding is not necessarily the best latent representation for prediction. By default, we regularize at three three- quee--uerrrss the model depth, which works well across architectures though the exact layer for decoding regularization is flexible. We provide more analysis on the codebook embedding regularization in Section 4.3 and Appendix C.2.
the layers that are originally most closely the em-
GeneratoorTokennizer Consistency Regularization, Combing Noisy Context Regularization and Codebook Embedding Regularization, the object of reAR is:
(0, L
(0; (0,
where À is the weight of the regularization term. Notice that we align the hidden feature of noisy to- kens to the embedding of the ground truth token well, which further encourages the autoregressive model to predict codebook embedding in a robust manner. This joint effect is important to boost the performance of visual autoregressive generation. We provide detailed ablation in Section 4.3.

---

Table 1: Results on 256x256 class-conditional generation on ImageNet-IK. "Mask." indicates masked generation; "Tok." denotes non-standard tokenization; "Rand." denotes randomized order; "Raster." denotes rasterization order. "f" indicates that the model is not provided and it's trained with our implementation. BPP 16 16 x BPP (bits per pixel) measures the compression rate of discrete tokenizers and not applicable ("N/A") to continuous tokenizers. "#Params" is the number of model parameters. "I" and "+" indicate whether higher or lower values are better, respectively.
Purdiigm
Training
Generation Model
Tokemizer
Type
Tekenizur BPP,s
Training Epocts
PParams.]
FID1
1ST
LDM-4 (Rombach et al., 3027)
Patch-VAF
N/A
200
400M
3,60
247.7
DCT (Peebles
Pach VAE
NAA
1400
615M
2.27
278.2
Diflision
SiT-XI.
2014)
Puck-VAE
N/A
675M
2.06
270.5
at. 2024c)
VAC
N/A
1.42
305.7
MAR
MAR-H et 2024)
MAR-L al.,
Patch- VAB
Pacc--VEE
NIA
NIA
800
943M
479M
1.58
1.55
290.3
MaskGäft-ee Chang al. (2022)
Pach-VQ VQ
0,625
300
227M
4.02
353
Mask.
Maskon (Weber et al, 2024)
MAGVIT-2 (Yo et 2023)
Patch LFQ
PacchVQQ
0,875
1.125
1080
1060
305M
307M
1.78
1.52
328.6
319
Mask-TiTok-64 (Yo et 20124))
TTlk
0.188
500
177M
2.48
214-7
Mask: TiTok-126 (Yo 2024b)
TTTok
0,375
800
287M
1.97
281.5
VAR
VAR-420
VAR-d30 atal, 2024))
2024)
VAR
2.57
1.90
eY at.,
Pach-VQ
261M
1.95
290.5
Cassal
AR
Rand.
RandAR-L, ot al., 2025)
RandAR-X XL. (Pung al., 2025)
RAR XL. al. 2024a)
RAR-L. al, al. 2024s)
Parch-V0 VQ
Pach VQ
Paich- VQ
Puch-VQ
0.625
0.875
0.875
0.625
400
400
300
300
955M
343M
775M
461M
2:25
2.55
1.50
1.70
288.5
317.8
305.9
RandAR-XM (Pang al., 2025)
PQ
0,875
300
1,4B
2.35
322.0
Tok.
AR-FlexTok-Ne (Bachmann al. 2025)
FlexTok
0.125
300
1.38
2.00
Causal
AR GignTok XXL (Xiong al. 2025)
Gigalok
0,875
300
1,98
256.8
WcTo----- (Zhuang 3025)
WeTok
6
VQGAN
et al., 2021)
5.20
Dyos-MAGiVTT-2 (1.00 2024)
PatchLLFQ
2.33
Rasser.
Cossal
LiamaGien XXL (Sun et 2014)
Luma (Smn et at. 202-0)
Pacc- VQ
Patch- VQ
0.875
0,875
300
300
775M
1,4B
2.60
2.34
264.1
253.9
AR
AR-L. (Yo 2004g)
PatchVVQ
0.625
400
461M
3,00
255
RAR
Patc
0,625
400
201M
2,00
295.7
WAR-8
Pucl-VV
400
261M
1.91
300.9
(cfg=1000/1..0
Patch
1.86/1.00
316.9/323.3
FID (lower is better)
reAR-S
Table 2: Superior generalization ability. reAR adapts to different tokenizers and achieves state-of-the-an per- formance with smaller models.
reAR
reAR-L
Model
Epochs
Params.
FID J
Maskhit (Weher et al., 2024) REPA (Yu al., 2024c)
1080
305M
1.52
800
675M
1.42
50
100 150
200
250
AR-TTTok-b64 (Yu 2024b)
400
261M
4.45
Training Steps (K)
RAR-TTTO ITFok-b64 (Yu et al., 2024a)
400
261M
4.07
leAR-TITok-be.
400
261M
4.01
Figure 4: Scaling Effect of reAR. As model size increases, the FID at each training step decreases consistently.
AR-AliTok-B B (Wu et al., 2025)
177M
1.50
B-A-TTiTkkk (Yu al., 2024a)
800
177M
1.52
-AR-B-AliTok
800
177M
1.42
4 EXPERIMENTS & ANALYSIS
4.1 EXPERIMENTAL SETUP
Below we provide brief of our experimental setup, and more details are Appendix A.
Dataset and evaluation. We evaluate reAR on ImageNet-IK IK at 256 x56 using the ADM proto- col (Dhariwal & Nichol, 2021). Each model generates 50k images with elassifier-free guidance (Ho & Salimans, 2022). We report FID (lower is better) (Heusel et al., 2017) and IS (higher is bet- ter) (Salimans et al., 2016), and compare training efficiency by epochs and parameters needed to reach the same quality. Baselines span diffusion, masked generation (continuous and discrete), VAR, randomized-order AR, siivanced-tokenizerr AR, and standard raster AR (see Table 1).
Model configuration. We use MaskGIT VQGAN (Chang al., 2022) (rFID= 1.97) as a tokenizer and a DiT-style (Peebles & Xie, 2023) AR backbone. We report reAR-S/B/L with 20/24/24 causal Transformer layers and hidden sizes 768/768/1024 To evaluate the generalization of reAR, also
7

---

pair it with TyTok (Yu et al., 2024b) and with AliTok (Wu et al., 2025) using their original setting. Additionally, we also verify the effectiveness our method on non-standale causal AR model such as VAR (Tian et al., 2024) with more details in the Appendix A.
Training. All models are trained for 400 epochs on 8 A800 GPUs (batch size 2048) with AdamW (Loshchilov & Hutter, 2017), gradient clipping (norm= 1), and accumulation. The learn- ing rate warms to x 10-4 over the first 100 epochs, then decays to x 10-6 for the remaining 300 epochs. Class labels are dropped with probability 0.1 enable classifier-free guidance inference.
reAR implementation. We apply a linear schedule for annealing noise augmentation. Embedding regularization is implemented using a 2-layer MIP (hidden size 2048, weight x=1): the shallow layer regularizes the current embeddings at /=0, while the deeper layer regularizes the decoding features at depth of the whole transformer (!' 15/18/18 18 for reAR-S/B/L).
4.2 MAIN RESULTS
Generation Quality. Table 1 shows that reAR achieves strong results even with standard raster- order AR model and simple 2D patch tokenizer. reAR-S outperforms prior raster AR models like LlamaGen-XL (Sun et al., 2024) (FID 2.00 vs. 2.34; IS 295.7 vs. 253.9) using only 14% of the parameters (201M vs. 1.4B), and surpasses advanced-tokenize AR models such as WeTok (Zhuang al.. 2025) with just 13-15% of their size. It matches RAR (Yu al., 2024a) and outperforms RandAR (Pang et al., 2025) under similar scales, and reAR-L exceeds MAR-L and VAR-d30 (Li et al., 2024; Tian et al., 2024). While diffusion and masked -enkee-genneattoon models remain strong, reAR narrows the gap with far fewer training epochs. More qualitative results are shown in Appendix F.
Generalization. We also evaluate reAR on non-standard tokenizers TYTok (Yu et al., 2024b) and AliTok (Wu et al., 2025). Unlike RAR (Yu et al., 2024a), which helps mainly bidirectional tok- enization, reAR consistently improves performance on both bidirectional (TYTok: 4.45 + 4.01) and unidirectional (AliTok: 1.50 1.42) tokenizers. Notably, it approaches iiffusion-basse REPA (Yu et al., 2024c) and outperforms Maskbit while using far fewer parameters (177M vs. 675M/305M).
Scaling Effect. We also study if the scaling behav- ior of the original AR model maintains with reAR. Specifically, we plot the FID under different train- ing epochs for each model size. As Figure 4 shows, the FID consistently decreases training iteration increase, revealing the potential of
FID - (lower is better)
EXos
SAM
2.
MAS-
RandAR-XI
model size and
BARR NOGGAA
reAR on large-scale visual AR models.
Sampling Speed. Like other autoregressive mod- els (Sun et al., 2024; Luo al., 2024), reAR bene- fits from KV-cache to achieve high sampling speed. We measure throughput on a single A800 GPU with batch size 128 (Figure 5). With KV-cache, autore- gressive models can run much faster than diffusion and MAR. Moreover, reAR-B-AliTE achieves lower FID with faster sampling speed even against parallel-decoding approaches such as Maskbit, TiTok, VAR, and RandAR.
MESIII
16
20
Throughput (Images/s) - (higher Is better)
Figure 5: Sampling Speed. Comparison of differ- ent methods on FID and throughput (images/sec).
4.3 ABLATION STUDIES
We conduct ablation studies on the key components reAR, focusing on the weighting and layer selection for encoding/decoding regularization, as well as the strategy for noise augmentation.
Regularization Layer. We analyzed the optimal layers for embedding regularization using reAR-S trained for 80 epochs without classifier-free guidance (Table 4). We regulariza-
Table 3: Ablation studies of noisy context regularization with annealing.
ablated both the presence and placement tion and compared with the naive tied embedding strat- egy (Press & Wolf, 2016; Weber et al., 2024). For decod- ing regularization, early layers (e.g., layer 10) offer little benefit, while layer 15 performs best; applying it deeper slightly degrades performance. For encoding regulariza-
Angmentation settings
0.0
2.12
0.5
3.15
<~ (700..5)
2.05
U(0, ((O). f()
2.02
C'O,f(t)) f(ti). f(t) mln(0.
40
2,00
wal embbacddng regularization
2.18
8

---

tion, the first layer is optimal it aligns best with the token embeddings, whereas deeper layers harm generation quality. Notably, applying regularization to the layers closest to the target embed- ding space in vamilla AR yields the best depth. We hypothesize this placement minimizes interference with next-token prediction. Based on these findings, we EN@0 DE@ 15 for reAR-S and EN@0 + DE@ 18 for reAR-B/L.. We provide a more detailed comparison of different choices of the decoding regularization layer in Ap- pendix C.2.
layer 0 and decoding at roughly }
Regularization Weight. As shown in Table 4, regular- ization weight has negligible impact on the quality of generation, likely because the AdamW optimizer is in- sensitive to the scale of the loss (Loshchilov Hutter, 2017; Zhuang et al., 2022). For simplicity, we use À = 1.
Table 4: Ablation studies of embedding regularization. We 'EN' as the encod- ing regularization and 'DN' as decod- ing regularization. For example, 'DN @ 15' means applying decoding regularization at the 15th layer of the transformer black.
Noise Augmentation. We further ablate the design of noise augmentation, exploring two strategies: (1) assign-
Regularization settings
F1D
IS
ing different noise levels to each token sequence, and (2)
Vanilla
21.32
+ -iod codebook embooking
21.08
57.2
annealing the maximum noise level during training. Re- sults are summarized in Table 3, based on the default
DDE
21.29
57.5
DDFR15
+ DDB220
20.28
61.2
setting with codebook embedding regularization (EN@0
-HN880 D85620
19.83
61.7
DE@ 15 for reAR-S). All models are trained for 400 epochs to evaluate the effect of different schedules. We
15
EN + DE (Final cholee)
19.72
61.3
A 0.5
19.79
60.9
find that a fixed noise level of = 0.25 improves FID
A
from 2.12 2.08, while higher level (e 0.5) leads to training collapse (FID- = 3.15). Randomizing the noise level within [0, 0.5] further improves FID to 2.05. Incorporating an annealing schedule, where f(t = 1 yields a stronger result (2.02 FID). Finally, using a truncated linear schedule f(t) max(0, 1 ft) achieves the best performance of 2.00 FID. These results highlight the effectiveness of proper annealing noise augmentation.
Joint Effect of Consistency Regularization. As shown in Table 3, using only embedding regular- ization (c=0) yields an FID of 2.12, while using only noise augmentation yields 2.18. In contrast, combining the two further improves performance, reducing the FID of reAR-S to 2.00. This indi- cates that both noisy context regularization and codebook embedding regularization are important.
RELATED WORK
Visual AR models generate images by predicting pixels or patch tokens sequentially, each condi- tioned on previous context (Gregor et al., 2014; Van den Oord et al., 2016; Van Den Oord et al., 2016; Parmar et al., 2018; Chen et al., 2020). In this paper, we refer specifically to the visual AR model as the family using unidirectional structure. Direct pixel-level modeling is expensive, so patch-based tokenizers (Van Den Oord et al., 2017: Esser et al., 2021) are used to compress local regions into discrete tokens. An AR model then predicts the token sequence (Esser et al., 2021; Sun et al., 2024; Luo et al., 2024). Prior work has focused on modular design, such as reducing quantization errors (Yu et al., 2023; Mentzer al., 2023; Ma et al., 2025; Li al., 2024) or ex- ploring tokenization beyond standard 2D grids (Yu et al., 2024b; Miwa et al., 2025; Sargent et al., 2025; Xiong et al., 2025). Others have studied sequence dependencies, imposing causality during tokenizer training (Wu et al., 2025; Bachmann et al., 2025; Pan et al., 2025) or randomizing token or- der (Pang et al., 2025; Yu et al., 2024a). While these works focus on the flaw of single component, we provide novel perspective on the inconsistency between the AR model and the tokenizer.
Other visual generation paradigm has advanced from Variational Autoencoders (VAEs) (Kingma & Welling, 2013) and Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) to modern approaches such as masked generative models (Chang et al., 2022; Yu et al., 2023; Weber et al., 2024) and diffusion-based models (Dhariwal & Nichol, 2021; Peebles & Xie, 2023; Ma et al., 2024; Yu et al., 2024c), apart from AR model. Recently, MAR (Li et al., 2024) was proposed to address quantization errors, and VAR (Tian et al., 2024) for next-scale prediction. However, they not implemented in a decoder-only transformer, making them harder to incorporate with the standard AR used in large language models. We provide more discussion in Appendix D.

---

Exposure bias has been extensively studied in the language domain, with methods such sched- uled sampling (Bengio et al., 2015). the visual domain, RQ-Transformer (Lee et al., 2022) applies scheduled sampling, and IQ-VAE (Zhan et al., 2022) uses Gumbel-softmax to mix ground-trute -oound-truth and predicted tokens, though both approaches compromise the parallel training efficiency of decoder- only Transformers. More recently, video generation works have addressed exposure bias in loregressive diffusion models (Zhou et al., 2025; Huang et al., 2025), but these strategies are not applicable discrete token prediction.
Representation Alignment. Representation alignment has been explored in visual generation (Yu al., 2024c; Leng al., 2025; Yao al., 2025; Xiong al., 2025). For example. REPA (Yu al., 2024c) incorporates DINO-v2 features to accelerate diffusion training. and Disperse Loss (Wang & He, 2025) applies self-supervised objectives to improve diffusion representations. However, these methods are either designed for encoder-only Transformers and diffusion models or often rely on external visual encoders. contrast, we aim align the representations of the tokenizer and the AR model itself, requiring external models and fitting naturally into the vanilla AR training pipeline.
CONCLUSION
In this paper, we identify the key bottleneck of visual autoregressive generation the mismatch between the generator and the tokenizer, where the AR model struggles to produce token sequences that can be effectively decoded back into images. To address this, propose reAR, simple regularization method that substantially improves visual AR performance while remaining agnostic to tokenizer design. We hope this work will encourage future research unifying generators and tokenizers within visual AR models, and more broadly, on developing unified multi-modal models.
10

---

ACKNOWLEDGMENT
We would like to acknowledge that computational work involved in this research is partially sup- ported by NUS IT's Research Computing group using grant number NUSREC-HPC-0001.
ETHICS STATEMENT
Our work introduces regularization strategy improve visual autoregressive models and tributes toward the broader goal of unifying vision and language generation. While these advances can benefit research on unified multimodal models, we acknowledge the potential risks associated with generative technologies. In particular, improvements in fidelity and scalability may also lower the barrier for misuse, such as the creation of misleading harmful synthetic media.
REPRODUCIBILITY STATEMENT
We include all experiment details sufficient for reproducibility in Section. 4, Appendix A, Ap- pendix B, and Appendix C. We provide the anonymous code here and will release the code once the paper is accepted.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303. 08774, 2023.
Roman Bachmann. Jesse Allardice, David Mizrahi, Enrico Fini, Oguzhan Fatih Kar, Elmira Amir- loo, Alaaeldin El-Nouby, Amir Zamir, and Afshin Dehghan. Flextok: Resampling images into 1d token sequences of flexible length. In Forty-second International Conference on Machine Learning. 2025.
Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan Yuille, Trevor Darrell, Jitendra Malik, and Alexei A Efros. Sequential modeling enables scalable learning for large vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 22861-22872, 2024.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. Advances in neural information processing systems, 28, 2015.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1 11315 1325, 2022.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pp. 1691 1703. PMLR, 2020.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. Scaling instruction-finetune lan- guage models. Journal of Machine Learning Research, 25(70): 2024.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Patrick Esser, Robin Rombach, and Bjom Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- fion, pp. 12873-12883 2021.

---

Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or, An image is worth one word: Personalizing text-to-image generation using textual inversion, 2022. URL https::tt////. htt.t::/taaaatttee http://axxivvvvovo /arggg/bb//aae /208 01
lan J Goodfellow, Jean Pouget-Abadee Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Karol Gregor, Ivo Danihelka, Andriy Mnih, Charles Blundell, and Daan Wierstra. Deep autore- gressive networks. In International Conference on Machine Learning, pp. 1242-1250. PMLR. 2014.
Qiyuan He and Angela Yao. Conceptrol: Concept control of zero-shot personalized image genera- tion, 2025. URL https: org/ abs 2503. 06568.
Qiyuan He, Jinghao Wang, Ziwei Liu, and Angela Yao. Aid: Attention interpolation of text-to-image diffusion, 2024. URL httttt:///aaiiiii org/ abs/243333333/ 17924.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-promet image editing with cross attention control, 2022. URL https://arxix org/abat 2208. 01626.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. arXiv:2207.12598, 2022.
Classifier-free diffusion guidance.
arXiv preprint
Xun Huang. Zhenggi Li, Guande He, Mingyuan Zhou, and Eli Shechtman. Self forcing: Bridg- ing the train-test gap in autoregressive vidco diffusion. arXiv preprint arXiv:2506. NXXV:22506008099, 2025. URL https: : / arxiv org/ abs 2500. 08009. Trains with AR rollout and KV-cached self-generated history to directly tackle exposure bias.
Diederik P Kingma and Max Welling.
Auto-encoding variational bayes.
arXiv preprint
1312v:1312.6114, 2013.
Simon Korbllith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pp. 3519- 3529. PMIR, 2019.
Doyup Lee, Chiheon Kim. Sachoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11523-11533 2022.
Xingjian Leng, Jaskirat Singh, Yunzhong Hou, Zhenchang Xing, Saining Xie, and Liang Zheng. Repa-e: Unlocking vae for end-to-end tuning with latent diffusion transformers. arXiv preprint arXiv:2504. 10483, 2025.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. Advances in Neural Information Processing Systems, 37: 56424-56445, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Zhuoyan Luo, Fengyuan Shi, Yixiao Ge, Yujiu Yang, Limin Wang, and Ying Shan. Open-magvit2: An open-source project toward democratizing ruto-regressive visual generation. arXiv preprint 24X1::2409.04440, 2024.
Chuofan Ma. Yi Jiang, Junfeng Wu, Jihan Yang, Xin Yu, Zehuan Yuan, Bingyue Peng, and Xiao- juan Qi. Unitok: A unified tokenizer for visual generation and understanding. arXiv preprint
arXivv2502.20321, 2025.
12

---

Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden- and Sain- ing Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In European Conference on Computer Vision, pp. 23 40. Springer, 2024.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In ICLR. Open- Review.net, 2022. URL httt:://bllp uni- meiittrier: de/ db/ conf/ htap///ccrricrr//// html #MengHSSWZE22.
Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantiza- tion: Vq-vae made simple. arXiv preprint arXiv:2309.15505, 2023.
Keita Miwa, Kento Sasaki, Hidehisa Arai, Tsubasa Takahashi, and Yu Yamaguchi. One-d-piece: Image tokenizer meets quatity- compression. arXiv preprint arXiv:2501.10065 2025.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.). Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, Pp. 16784-16804. PMLR, 17-23 Jul 2022. URL https:/ hettss:/proceedine /ppppoeeeeingsss.. mll.. .tr.spsss./ v162 (1162/nncho22e.. html.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.071900 071930 2023.
Kaihang Pan, Wang Lin, Zhongqi Yue, Tenglong Ao, Liyu Jia, Wei Zhao, Juncheng Li, Siliang Tang, and Hanwang Zhang. Generative multimodal pretraining with discrete diffusion timestep tokens. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 26136-26146, 2025.
Ziqi Pang, Tianyuan Zhang, Fujun Luan, Yunze Man, Hao Tan, Kai Zhang, William T Freeman, and Yu-Xiong Wang. Randar: Decoder-only autoregressive visual generation in random orders. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 45-55, 2025.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. International conference on machine learning, pp. 4055- 4064. PMIR, 2018
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEF/CV international conference on computer vision, pp. 4195 4195-4205, 2023.
Ofir Press and Lior Wolf. Using the output embedding to improve language models. arXiv preprint arXiv: 1608.0000000000 05859, 2016.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation, 2023.
URL httphttp//// hta//axivv.orrg abs//20s 12242.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Kyle Sargent, Kyle Hsu, Justin Johnson, Li Fei-Fei, and Jiajun Wu. Flow to the mode: Mode-seeking diffusion autoencoders for statc-of-the-a image tokenization. arXiv preprint arXiv:2503. 11056, 2025.
13

---

Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint 0rXXi:24006006555, 2024.
Zhenxiong Tan, Songhua Liu, Xingyi Yang, Qiaochu Xue, and Xinchao Wang. Ominicontrol: Mini- mal and universal control for diffusion transformer, 2025. URL http:: htt //tp://aaavv../////ee / 2411 15098.
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint
arXiv:2405.098/8 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Sean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.1
1805, /880, 2023.
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng. and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next- scale prediction. Advances in neural information processing systems, 37:84839-84888 2024.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Con- ditional image generation with pixelenn decoders. Advances in neural information processing systems, 29, 2016.
Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International conference on machine learning, pp. 1747-1756. PMLR, 2016.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Chaojun Wang and Rico Sennrich. On exposure bias, hallucination and domain shift in neural machine translation. arXiv preprint arXiv:2005.03644 2020.
Runqiam Wang and Kaiming He. Diffuse and disperse: Image generation with representation regu- larization. arXiv preprint arXiv:2506.0991 00550600022.... 2025.
Mark Weber, Lijun Yu, Qihang Yu, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang- Chieh Chen. Maskbit: Embedding-free image generation via bit tokens. arXiv preprint
arXiv:2409. 16211, 2024.
Pingyu Wu, Kai Zhu, Yu Liu, Longxiang Tang, Jian Yang, Yansong Peng. Wei Zhai, Yang Cao, and Zheng-Jun Zha. Alitok: Towards sequence modeling alignment between tokenizer and autore- gressive model. arXiv preprint arXiv:2506.055888 2025.
Tianwei Xiong, Jun Hao Liew, Zilong Huang. Jiashi Feng, and Xihui Liu. Gigatok: Scaling
visual tokenizers to billion parameters for autoregressive image generation. arXiv preprint arXiv:2504.0873: 2025.
Jingfeng Yao, Bin Yang, and Xinggang Wang. Reconstruction vs. generation: Taming optimization dilemma in latent diffusion models. In Proceedings of the Computer Vision and Pattern Recogni- tion Conference, pp. 15703-15712, 2025.
Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, al. Language model beats diffusion- tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737. 2023.
Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chich Chen. Randomized autoregres- sive visual generation. arXiv preprint axXiv:4411.0077,, 2024a.
Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. An image is worth 32 tokens for reconstruction and generation. Advances in Neural Information Processing Systems, 177118840-128946 2024b.
Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is casier than you think. arXiv preprint 1rXiv:2410.069400. 2024c.
14

---

Fangneng Zhan, Yingchen Yu, Rongliang Wu, Jiahui Zhang, Kaiwen Cui, Changgong Zhang, and Shijian Lu. Auto-regressive image synthesis with integrated quantization. In Proceedings of the European Conference on Computer Vision (ECCV), 2022. URL https: /tww//wwww// eeva.
wet.papprs//eeee /ccv _2ee 2222///pprr..e.. http:pppppppppppaaeepeppppp sampling with celiability-hased scheduling mitigate exposure bias in AR training.
r //13//1666000 601 06. 06.p.f.. Uses Gumbel
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 586-595, 2018.
Hongkai Zheng, Weili Nie, Arash Vahdat, and Anima Anandkumar. Fast training of diffusion models with masked transformers. arXiv preprint arXiv:2306..00 230.::23006003055 2023.
Deyu Zhou, Quan Sun, Yuang Peng, Kun Yan, Runpei Dong, Duomin Wang, Zheng Ge, Nan Duan, and Xiangyu Zhang. Taming teacher forcing for masked autoregressive video generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 7374-7384, 2025.
Shaobin Zhuang, Yiwei Guo, Canmiao Fu, Zhipeng Huang. Zeyue Tian, Ying Zhang, Chen Li, and Yali Wang. Wetok: Powerful discrete tokenization for high-fideitty visual reconstruction. arXiv preprint wrwiv:2508.0050000 2025.
Zhenxun Zhuang, Mingrui Liu, Ashok Cutkosky, and Francesco Orabona. Understanding adamw through proximal methods and scale-freeness. arXiv preprint arXiv:2202.00089 2022.
15

---

A ADDITIONAL EXPERIMENTAL DETAILS
Dataset and Evaluation Protocol. For ImageNet evaluation, we follow the ADM protocol (Dhari- wal & Nichol, 2021). Specifically, we compute both FID and IS using the ImageNet-I1 IK validation split (50,000 images), and we generate 50,000 synthetic images with our model. We then com- pute FID between the generated set and the real validation set. During sampling, for classifier-free guidance, we adopt a power-cosine schedule used in prior work (Zheng et al., 2023). For our reAR-S/B/LL models, we set the guidance scale to 22, 14.5, and 10, respectively, and corresponding power scales to 2.75, 2.25, and 1.75. Across all models, we keep the temperature at 1.0 and do not use top-p or top-k sampling, so that improvements reflect model quality rather than sampling tricks. All the images generated and evaluated are fixed at the resolution of 256 256.
Comparing methods. We divide the visual generation into seven classes in Table I: Diffusion model, MAR (continuous masked generation), Mask. (discrete masked generation), VAR (next scale prediction with encoder-only transformer), Rand. Causal AR (introduce randomized order of token sequence), Tok. Causal AR (use an advanced tokenizer that is not rasterization order), Raster. Causal AR (the most standard visual AR based on patch tokens and rasterization order).
Model Configuration. We use the same VQGAN tokenizer from MaskGIT (Chang et al., 2022), a pure CNN that produces feature maps which are patchified into 16 x 16 patches and quantized via a codebook of size 1024. For the autoregressive backbone, we follow the visual transformer (ViT)- based architecture of RAR (Yu et al., 2024a) and DiT (Peebles & Xie, 2023), further inserting class conditioning via AdaLN layers as in DiT. To ensure fair comparison with RAR, we use learnable positional embeddings throughout. We apply dropout with probability 0.1 both in the ferd-forward network and in attention layers. Additionally, the MLP ratio is kept as 4.0 in the feed-forward network, and the number of attention heads is fixed to 16 for all different settings. We also include QK-Norm in attention to enhance stability.
Training details. As we mentioned in Section 4.1. all models are trained for 400 epochs with a batch size of 2048 a single node of 8 A800 GPUs. For reAR-S and reAR-B, we use gradient accumulation as 1, and for reAR-L, we use gradient accumulation over 2 steps with a batch size of 1024 to achieve the same effective batch size. Following prior work (Yu et al., 2024a), we linearly warm up the learning rate to 4 x 10-4 over the first 100 epochs and apply a cosine decay schedule to decrease the learning rate to x 10-5 for the remaining 300 epochs. We
Table 5: Comparison of computation cost.
Method Time / Epoch (min)
FID
AR-B
8.11
3.12
reAR-B
8.14
1.91
AR-L
15,99
3.02
reAR-L
16.05
1.86
use A as the optimizer with B1 = 0.9, 32 = 0.96, and weight decay of 0.03, The gradient clipping is applied with maximum gradient norm of 1.0. We use mixed precision with bfloat16 to accelerate training.
Implementation details of reAR. Regarding noisy context regularization, the noise ratio is sampled from range that is determined by the training pro- cedure. Specifically, the noise ratio is sampled from (0, f(t). where f() = min
Table 6: Evaluation of reAR VAR
Method
FID
IS
{) and refers to the normalized training progress. For example, the noise ratio sampled from (0, )) at the 150 epoch where t - over total 400 epochs. Regard- ing codebook embedding regularization, the 2-layer MLP with hidden size as 2048 is equipped with GeLU and maps the generator feature into the dimension of the corresponding codebook embed- ding. The parameter overhead of the MI. is 2. M/2.. M//42M for reAR-S/BA Table 5 shows the training time cost on A800 GPUs This light design only brings minimal training overhead while achieving superior performance with the same inference cost.
VAR-d16
3.55
274.4
VAR-d16 (retrained w/ reAR)
3.39
276.6
Experiments on VAR (Tian et al., 2024). VAR differs from standard autoregressive models such as VQGAN, TiTok, and AliTok, as it predicts the next scale resolution and outputs multiple discrete tokens simultaneously rather than using a decoder-only transformer to predict a single next token. These differences lead to training and inference behaviors that diverge from standard AR, and we provide more details in Appendix D. Nevertheless, because VAR still generates discrete tokens autoregressively, it may also benefit from reAR. To test this, apply reAR to VAR-d16 6 using the
16

---

Figure 6: Qualitative Results VAR-d16 retrained with reAR.
Ground-truth
Replace Ratio: 0%
Replace Ratio: 20%
Replace Ratio: 40%
Replace Ratio: 60%
Figure 7: Visualization of analysis experiment on replacing tokens with more similar embedding.
same training settings in our main experiments and same inference settings as in the original VAR paper. As shown Table 6, reAR improves performance without tuning any training or inference hyperparameters, demonstrating its generalization ability.
B ANALYSIS DETAILS ON GENERATOR-TOKENZIA INCONSISTENCY
In this section, we present more details on the analysis experiments introduced in Section 3.1 on the generator-tokenize inconsistency, including (i) evaluation metric (Section B.1), (ii) experiment settings (Section B.2), and (ii) Findings (Section B.3).
B. EVALUATION METRIC FOR STUDYING INCONSISTENCY
We provide additional results on the quantitative evaluation of the inconsistency between token se- quences X1.V and the corresponding decoded images I. We adopt two groups of metrics: (i) for token sequence quality, we use the correct token ratio (CTR) and perplexity. and (ii) for image quality, we use PSNR and LPIPS. Here, the LPIPS and PSNR are different from those in the recon- struction task, since the decoded image is obtained from the generated token sequence under teacher forcing. While this is not direct evaluation of the generation quality, it serves as an intermediate proxy similar to the correct token ratio and perplexity, but in pixel space.
Evaluation on token sequence. CTR measures the fraction of correctly predicted tokens under teacher forcing, while perplexity reflects the uncertainty of the predicted token distribution. For- mally, given ground-truth sequence x1:N and autoregressive model pg. we define
N CTR
Σ
arg max po X1:1-1) = xz
(8)
N
1=1
Perplexity exp
1
po X15-2)
(9)
N
17

---

Imperfect Context
PONE (nigher in betorrs
Imperfeci Conlext
LMIPS (owee to betttor
Imperfect Conlext Pertioct Contest
Imperfect Context Perfect Conteat
Perfect Coctext
Pertect Context
PSNR (Oigher is
36
0.3
0.2
0.4
0.6
0.5
022
0.5
0.5
1500
1700
1900
1500
17:00
1,000
(higher better)
(highhr better)
Perplesity (lemerr beller)
Perplecity (Iossee belter)
1217
PANA
PUNK- (wweeeeeeeeeeeee IA beiteed
2:22 0317
1433
0.30
0.12
300
11:47
15477
0312
0.20
0.72
14:
90
50
Replazmtent
Replacvnent Rall
Replacument Hallo
Reglacstnntt Batto
(d)
(e)
Figure 8: Detailed analysis of generator- inconsistency. Results are evaluated with CTR (t). perplexity (L). PSNR (1), and LPIPS (t). (a-d): Exposure bias analysis- undsrrrr-ddrrrrereeereeee the same CTR and lower perplexity, imperfect context yields higher LPIPS and lower PSNR. (e-g): Embedding unawareness analysis -À Perplexity denotes the increase from original to replaced sequences; even with similar CTR and lower perplexity, original predictions can give worse PSNR/LPIPS, showing that higher-quality token sequences can decoded into worse images.
Evaluation on decoded image. To assess the quality of the decoded images i D(2)), we report peak signal-to-noise ratio (PSNR) and learned perceptual image patch similarity (LPIIS). PSNR is distortion-based metric that measures reconstruction fidelity relative the ground-truth image I:
11
W
MSE
ΣΣΣ
Los Less)
(10)
3HW
jml
L
PSNR 10 log10
(11)
MSE
where L is the maximum possible pixel value (e.g., 255 for 8-bit images), A higher PSNR indicates better pixel-wise reconstruction fidelity.
LPIPS, on the other hand, evaluates perceptual similarity by comparing deep features extracted from a pretrained network Φ:
LPIPS(I, = H,W, | |m )
(()0))((()))
(12)
where (-).) denotes the activation map from layer 1, and wy learned weights that calibrate the contribution of each layer. Lower LPIPS corresponds to higher perceptual similarity.
B.2 ANALYSIS EXPERIMENT SETTINGS
To analyze the inconsistency between token sequence behavior and decoded image quality, we study the relationship between token-level metrics (CTR, Perplexity) and image-level metrics (LPIPS, PSNR). The key challenge is to design controlled interventions such that one aspect of quality (to- ken sequence image) can be varied while holding the other approximately fixed, thereby revealing causal effects. all experiments, we treat correct token ratio (CTR) as the control variable, since it is the most straightforward to manipulate, while Perplexity, LPIPS, and PSNR dent variables. This setup allows us to investigate how changes in token correctness propagate to perceptual differences in reconstructed images.
depen-
18

---

Experiments amplified exposure bias. As discussed in Section 3.1, design two decoding protocols to vary the amount of exposure bias under the same CTR level:
Perfect Context (front-loaded); Given a target CTR r, we fix the first [ra] tokens to ground truth 21:[ru] and let the autoregressive model freely generate the remaining tokens. This minimizes exposure bias, since the context remains error-free until the switch point.
Imperfect Context (uniformly interleaved): For the same CTR T, we randomly select [rn] positions in the sequence and load ground-truth tokens only at those positions. At all other positions, tokens are sampled autoregressively. This introduces earlier corruption into the context and amplifies exposure bias.
Both settings guarantee the same number of ground tokens, so any difference downstream LPIPS/PSNR is attributable to the severity of exposure bias. This isolates the tokenizer's 's role in amplifying exposure bias during generation.
Experiments on embedding unawareness. While exposure bias focuses where ground-truth tokens are inserted, embedding unawareness examines what happens when incorrect tokens are re- placed by semantically similar alternatives. During training, the autoregressive model is optimized for exact token prediction, whereas the tokenizer decoder operates a continuous embedding space. To study this gap, we introduce replacement ratio r' [0. I]:
1. First, generate predictions 61.m with teacher forcing. Identify all positions i where á; +
2. For each such incorrect prediction, replace 2; with probability r' by another token z whose embedding 2 is the closest to the correct embedding =9; under cosine similarity, i.e.
2 arg min d(29,29,). 22.).
**d <
3. The CTR remains unchanged, since replacements are only among incorrect predictions, but the embedding similarity of the sequence is improved.
By varying r'. control the degree of embedding similarity while holding CTR constant, and then measure its effect on LPIPS and PSNR of the reconstructed images. This design allows us to directly test whether embedding closeness-rathe than token identity alfnc-affects perceptual quality.
Summary. For both experiments, we additionally evaluate the perplexity of the token sequence un- der the same CTR and study its correlation with LPIPS PSNR as well. Together, these controlled settings-Pertte vs. Imperfect Context for exposure bias, and embedding replacement for unaware- ness enable systematic evaluation of how token-level inconsistencies translate into perceptual pixel-level degradation in decoded images.
B.3 FINDINGS AND OBSERVATION
Results on exposure bias. As shown in Figure 8(a-b), under the same CTR, sequences generated with imperfect context lead to higher LPIPS and lower PSNR, indicating worse decoded images, es- pecially low CTR. A similar trend observed with perplexity. Although perplexity cannot be di- rectly controlled, varying CTR naturally induces different perplexity levels. Thus, in Figure 8(c-d), plot perplexity against PSNR/LPIPS under matched CTR. Even when the token sequence quality appears worse (higher perplexity). images decoded from tokens generated with perfect context still achieve better visual quality (lower LPIPS, higher PSNR) compared to those from imperfect context. This highlights that a token sequence favored by the autoregressive model does not necessarily yield a better decoded image.
Results on embedding unawareness. As shown in Figure 8(e-g), increasing the replacement ratio improves embedding similarity while keeping CTR unchanged. This leads to consistent improve- ments in decoded image quality: LPIPS decreases and PSNR increases more incorrect predictions are replaced with embedding- nearest tokens. Importantly, even though perplexity rises due to these replacements. the resulting images become visually closer to the ground truth. Figure 7 further illus- trates this effect-images reconstructed from sequences with higher replacement ratios (20-60%) recover clearer object structures (e.g., sharper outlines of the dog's and the butterfly's wings)
19

---

AR en Thntredd Data
AR on Usseen Duta
(neAlt) vos Enooded Tobmn
(mAllt xx. Decoded
Correct Ratio
12
roAR on Tratned Dusa
roAR Uhsoon Data
CKA Score
CKA Score
0.0
0.10
10
0.2
0.05
0.0
Choon
Clean
Noney
Noney
Layer
Layer Index
(a) Correct token ratio of clean and noisy input
(b) generator feature v.s.
(c) generator feature v.s. decoded embedding
encoded embedding
Figure 9: Mitigating inconsistencies in visual autoregressive generation. (a) reAR narrows the performance gap between trained and unseen data compared to vanilla AR and improves robustness under noisy inputs, indicating better generalization. (b, The CKA score demonstrates similarity between the feature and codebook embedding. reAR further aligns hidden features with the em- bedding of the current token in early layers and with the embedding of the next token in deeper layers.
compared the 0% baseline. These results demonstrate that token correctness alone is insuffi- cient to guarantee high-quality reconstructions; instead, embedding proximity plays a crucial role in aligning autoregressive predictions with tokenizer decoding.
C ANALYSIS ON THE EFFECT OF REAR
In this section, we present further analysis on the effect of reAR: (i) its effect on the token space (Section C.I) and (ii) its effect on the hidden features, which also includes the analysis on the choice of regularization layer as mentioned in Section 3.2 and Section 4.3.
C.1 IMPACT ON SAMPLED TOKEN SEQUENCE
We found that reAR improves the next token prediction on: (i) generalization and (ii) robustness.
Generalization. We compare the correct token ratio (CTR) of vanilla AR and reAR on both trained data' and unseen validation data from ImageNet- 1K as shown in Figure 9()) On clean inputs, reAR achieves nearly identical performance to vanilla AR on trained data (12.9 vs. 12.8), but obtains higher CTR unseen data (12.3 11.8), indicating improved generalization. These results sug- gest that incorporating codebook embeddings provides a stronger inductive bias for visual signals, enabling the AR model learn more generalizable representations.
Robustness. To examine the robustness gained from reAR, we randomly replace a fraction of cur- rent tokens with noise at controlled rate. Figure 9 (a) also compares the CTR for clean sequences and for sequences with 10% of tokens replaced uniformly. Compared to vanilla AR, reAR gains higher CTR compared AR on the noisy trained data (9.5 v.s. 9.1). On the noisy and unseen data, the performance gap is even larger: reAR substantially outperforms vanilla AR (9.0 vs. 8.3). This result shows that reAR more robust to the possible exposure bias.
C.2 IMPACT ON HIDDEN FEATURES OF DIFFERENT REGULARIZATION LAYER
To better understand how reAR interacts with hidden representations, evaluate the similarity be- tween generator features and tokenizer embeddings using centered kernel alignment (CKA) (Ko- mbblth al., 2019). Specifically, given two of feature representations X Raxd, and Y E paxd, we first compute their Gram matrices K Xx and L = YY and then cen- ter them K. HKH and L. HLH. where H I, 1, is the centering matrix. The
To avoid class-wise bias, we sample 1000 images per class match the validation setting.
20

---

CKA score is defined as
(Ker Lc) | L
CKA(X, Y)
(13)
where ( denotes the Frobenius inner product and || || is the Frobenius norm. Intuitively, CKA measures the alignment between the pairwise similarity structures two representations and invariant to isotropic scaling and orthogonal transformation. A higher CKA score indicates that the hidden features of the generator are more similar to the corresponding tokenizer embeddings.
Analysis Target. We aim to examine how hidden features within the decoder-only transformer correlate with two types of embeddings: the encoded embedding z,, representing the codebook vector of the current token, and the decoded embedding 1 corresponding the codebook vector of the next token. By comparing generator features against both embeddings, we can assess how the autoregressive model's 's hidden representations evolve capturing alignment with the tokenizer's codebook while simultaneously encoding the current token and preparing to decode the next one.
Correlation between hidden features and embeddings. To analyze how autoregressive representations evolve with depth, we compute CKA similarity between hidden features of vanilla AR model and tokenizer embeddings across layers (Figure 9(-c)). Four key trends emerge: (1) overall, CKA with the decoded embedding is lower than with the encoded embedding, since the current token known while the next token remains uncertain; (2) sim- ilarity to the encoded embedding is highest at the input layer and decreases monotonically with depth; (3) simi- larity to the decoded embedding gradually increases and peaks around layer 15, roughly three-quarters of the full architecture; and (4) similarity to the decoded embedding drops again in the final layers. Together, these patterns suggest natural progression: early layers focus on encoding the current token and aggregating contextual information, while deeper layers shift toward modeling the next-token embedding. The decline the final layers likely reflects the model's 's need to project features onto decision boundary for prediction, where the codebook em- bedding itself may not form an optimal target. This also explains why directly tying AR outputs to codebook embeddings can lead to suboptimal performance.
Regularization settings
FID
IS
DE@13 13
20.47
59.4
DE@ 14
20.17
60.8
DE@15 15
20.03
61.0
DE@16
20.11
60.5
17
20.25
61.1
Table 7: Analysis on nearby regulariza- tion layer. We 'EN' the encoding regularization and 'DN' the decoding reg- ularization. For example, 'DN@ 15' means applying decoding regularization at the 1,5th layer of the transformer block.
Choosing the regularization layer. Motivated by these observations, we design reAR to apply regularization layers where the CKA similarity is naturally high-eatt layers for encoded em- beddings and later layers for decoded embeddings. Intuitively, this choice minimizes conflict with the primary next-token prediction objective, since these layers are already aligned with the tokenizer. Importantly, we avoid imposing regularization at the very last layer. Instead, we place regulariza- tion near the three-quarter depth of the model, where decoded embedding similarity is maximized. Empirically, we find that applying reAR to nearby layers yields similar performance as Table 7, highlighting the flexibility of our method with respect to the choice of regularization layer.
Effeet of reAR on feature alignment. After introducing reAR, we observe consistent increases in CKA similarity between generator features and both encoded and decoded embeddings (Fig- ure 9(--c) at the target layer. In early layers, reAR strengthens alignment with encoded embed- dings, helping the generator encode current tokens similar to the tokenizer. In deeper layers, reAR improves similarity with decoded embeddings, ensuring that hidden features are better aligned with the next token. This result indicates that reAR directly improves the consistency between the hidden feature of the autoregressive model and the tokenizer.
D ADDITIONAL DISCUSSION ON THE RELATED WORK
In this section, we present a detailed discussion and comparison of related work. Diffusion models have achieved great success in many downstream visual tasks, including image editing (Nichol et al., 2022: Meng et al., 2022; He et al., 2024; Hertz et al., 2022) and personalized image generation (Gal et al., 2022; Ruiz et al., 2023; He & Yao, 2025; Tan et al., 2025). By contrast, visual autoregressive models are less frequently used in these domains, mainly because their generation quality often
21

---

lags behind that of diffusion models. A growing line of research aims to bridge this gap between visual autoregressive modeling and diffusion- based approaches. In the following, we mainly discuss that how these prior methods can be viewed through unified lens: they address the inconsistency between the tokenizer (or tokenization scheme) and the autoregressive model. We also discuss how MAR and VAR differ from other autoregressive approaches, and highlight the distinction between our method and REPA, a regularization technique proposed for visual generation.
D.1 TOKENIZATION WITH RANDOMIZED ORDER
RandAR (Pang et al., 2025) introduces a positional token in front of each patch token to let the token be aware of its position in terms of tokenization. Specifically, given 256 token sequence, it inserts additional 256 tokens, and the generator is required to learn the distribution of the total 512 tokens under permutation. During training and inference, the token sequence is always shuffled. It enables parallel decoding during inference by inserting multiple positional tokens simultancously. However, RandAR can double the context and significantly increase the computation budget.
RAR (Yu et al., 2024a) introduces a learnable embedding of target position over each token. During training. it randomly shuffles the token sequence given probability, and the token is aware of its own position with the additional positional embedding. It slowly decreases the probability of shuffling and returns to standard rasterization order during training. During inference, keeps the standard operation for the autoregressive generation.
Summary. Both RandAR and RAR use permutation during training so that the context of each token is not limited to the tokens that are on the left or the top of it, thereby introducing bidirec- tional context even using a decoder-only transformer. This mitigates the inconsistency between the tokenizer that also models bidirectional context, such as MaskGiT-VQGAl TiTok (Yu et al., 2024b). However, in terms of the advanced tokenizer already introduced, unidi- rectional dependency, such AliTok (Wu et al., 2025) and FlexTok (Bachmann et al., 2025), may further amplify the inconsistency as Table 2 in the main text shows.
(Chang al., 2022) or
D.2 TOKENIZATION WITH ID SEQUENCE OR UNIDIRECTIONAL DEPENDENCY
TYTok (Yu et al., 2024b) transforms an image into ID discrete token sequence with query token using VIT. It firstly decouples the number of tokens from the number of patches and can further compress the number of tokens. However, the reconstruction quality of TYTok remains suboptimal compared to the patchify tokenizer. Additionally, although the represented token sequence is 1-dimensional, it's still in a bidirectional context instead of modeling unidirectional dependency. Therefore, the autoregressive model trained on it remains suboptimal Table 2 in the main text shows.
GigaTok (Xiong et al., 2025) transforms the image into 1D discrete token sequence as well similar TiTok. Additionally, it introduces the feature from DINO-v2, similar to REPA (Yu al., 2024c) to regularize the hidden feature of the tokenizer decoder. This enables the tokenizer to scale up and stabilize training. However, it suffers from the same problem TiTok, which still models bidirectional dependency.
FlexTok (Bachmann et al., 2025) firstly Iearns a continuous VAE with high fidelity. It then fur- ther resamples ID discrete tokens from the 2D continuous token obtained from the VAE. Different from TYTok and Giga'Tok, it additionally employs causal mask on the 1D sequence to model the unidirectional dependency, which is more consistent with an autoregressive model.
AliTok (Wu et al., 2025) introduces an Aligned Tokenizer that uses ID sequences instead of the typi- cal 2D patch grid, but with novel training to better align the tokenizer with autoregressive generation. Unlike standard patchified tokenizers, AliTok uses a causal decoder during tokenizer training to en- force unidirectional dependency among encoded tokens, ones. After that, it freezes the encoder and then uses a bidirectional decoder to refine the reconstruc- tion quality. This unidirectional alignment improves compatibility with autoregressive models and leads to state-of-the the-----rttttttt art generation metrics - our method still further enhances performance.
that tokens depend only on preceding
Summary.
These works (e.g. TiTok (Yu al., 2024b) and AliTok (Wu al., 2025)) impose ID token sequence or enforce unidirectional dependency in the tokenization stage so that the tokenizer is more aligned with autoregressive models, which shows the importance of consistency between
22

---

the tokenizer and autoregressive model. In our experiments, we further demonstrate that using cenerator-tokenizer consistency regularization can further improve upon their performance.
D.3 REMARKS ON MAR AND VAR
MAR (Li et al., 2024) is a model paradigm that combines masked prediction and autoregressive gen- eration. Rather than generating tokens strictly in a raster (ID) order, MAR predicts multiple masked tokens in parallel across iterations, while still enforcing an ordering among iterations. Importantly MAR uses continuous tokens instead of discrete ones and employs a diffusion-based head model the continuous distribution of token predictions.
VAR (Tian al., 2024) proposes a coarse-to- next-scale prediction strategy in image generation: rather than predicting each patch or token in raster order, VAR generates images scale by scale, first at low resolution and then successively higher resolutions, where each finer scale conditioned on all previously generated coarser scales. Given tokens of previous scale, the model will provide multiple mask tokens corresponding to the next scale, and decode them in parallel.
Summary. Although MAR and VAR can be regarded as autoregressive since generation proceeds in an autoregressive manner, they implement with an encoder-only transformer or block causal transformer. In MAR, the model receives masked tokens as input and learns reconstruct the masked positions, rather than predicting the next token a decoder-only setup. In VAR, tokens from the previous resolution provide the context for predicting multiple tokens at the next resolution in parallel. Both model are different from standard AR paradigm of next token prediction.
D.4 REGULARIZATION ON GENERATION TECHNIQUE
REPA (Yu et al., 2024c) is regularization technique for diffusion traniforrrrnfforeer models that aligns noisy intermediate states in the denoising process with clean image features from pretrained visual encoder. Rather than forcing the model to learn image representations from scratch under noisy conditions, REPA adds a loss that encourages the hidden states of the diffusion model to match the semantic structure of an external teacher (e.g.. DINO, DINO-v2).
Comparison. Unlike REPA, which focuses on accelerating the training of diffusion models, reAR designed to address the inconsistency between autoregressive models and their tokenizers. More- over, while REPA relies on external feature extractors such as DINO-v2 (Oquab et al., 2023), reAR directly leverages features from the tokenizer, which is already an integral component of the visual generation pipeline. In addition, REPA tailored to bidirectional transformers and is restricted to 2D tokenizers. whereas reAR is compatible with decoder-only transformers. For these reasons, do not apply REPA to visual autoregressive models, as it is less generalizable to visual AR training.
E DISCUSSION ON THE LIMITATION
Our method has several limitations that suggest promising directions for future work. First, the choice of the decoding regularization layer is determined empirically. This issue is not unique to our approach, prior works that regularize intermediate representations, such REPA (Yu al., 2024c) and Dispersive Loss (Wang & He, 2025), also depend on empirically selected layers in the absence of a clear theoretical principle. Developing an adaptive or theoretically grounded strategy for layer selection remains an open challenge and is more closely aligned with ongoing research in automated architecture and hyperparameter search.
Second, our experiments focus primarily ImageNet, following common practice in foundational visual generative modeling (Esser al., 2021; Yu et al., 2024b; Wu et al., 2025). While this setup did not evaluate reAR on downstream text-guided generation
enables controlled comparisons, tasks. A comprehensive evaluation on standard text-to-image benchmarks would offer clearer as- sessment of practical utility, but is computationally demanding. We leave expanded downstream study for future work.
Finally, although empirical results demonstrate the effectiveness and generalization capability of reAR and we provide direct CKA analysis on the hidden feature of transformer layers before and after regularization, we do not provide deeper theoretical analysis of the geometric factors
23

---

alignment. Understanding properties such manifold structure distributional behavior could yield more principled perspective, but developing such a theoretical framework is non-trivial. We view this an important direction for future research.
underlying
F QUALITATIVE RESULTS
We present comprchensive generated results of reAR-B-AlToo, (Figure 10 to 18) and reAR-L- VQGAN (Figure 19 to 24). All results are generated with a constant guidance scale of 4.0.
24

---

Figure 10: Generated Results of reA reAR-B-AliTok of class 'Cliff'
Figure 11: Generated Results of reAR-B-AliTok of class 'Goldfish'
Figure 12: Generated Results of reAR-B-AliTok of class 'Labrador retriever'
25

---

Figure 13: Generated Results of reAR-B-Aliine of class 'Ice cream'
Figure 14: Generated Results of reAR-B-AliTok of class 'Lakeshore'
Figure 15: Generated Results of reAR-B-AliTok of class 'Cheeseburger'
26

---

Figure 16: Generated Results of reAR-B-AliTok of class 'Bridge'
3
Figure 17: Generated Results of reAR-B-AliTok of class 'Balloon'
Figure 18: Generated Results of reAR-B-AliTok of class 'Chihuahua'
27

---

Figure 19: Generated Results of reAR-L-VQGAN of class 'Cock'
Figure 20: Generated Results of reAR-L-VQGAN of class 'Green mamba"
Figure 21: Generated Results of reAR-L-VQGAN of class 'Hermit crab'
28

---

Figure 22: Generated Results of reAR-L-VQGAN of class 'Flamingo'
Figure 23: Generated Results of reAR-L-VQGAN of class 'Hourglass'
Figure 24: Generated Results of reAR-L-VQGAN of class 'Pirate'
29


Reviews:
[{"review_id": "05Il9DDmTl", "reviewer": "Reviewer_YZ8R", "summary": "This paper tries to improve (traditional, raster-order) autoregressive image generation.\n\nProblem statement: inconsistency between a generator and a tokenizer, i.e., the autoregressive model might generate a token sequence that is hard for the tokenizer to decode back to an image.\n\nLogical development\n* The generated token sequence can be unseen by the tokenizer due to exposure bias between training (teacher forcing) and inference (own predictions). It is more problematic in image models than language models because the possibility of the decoder not seeing the combination of generated visual tokens in the training phase.\n* AR models produce token indices and do not care the embeddings produced by the tokenizer.\n\n\nSolutions\n* Expose the model to perturbed context during training.\n    * (context = previous tokens)\n    * It encourages the model not to rely on clean contexts and improves robustness to imperfect histories at inference.\n* Align the generator's hidden states with the tokenizer's embedding space.\n    * It helps the visual embeddings being reconstructed from the unseen tokens.\n    * Method: SimSiam-like architecture and loss for regressing current embedding and next embedding.\n\nExperiments\n* Competitors: many.\n* ImageNet\n* FID 1.42 with 177M parameters (sota diffusion models = 675M)", "questions": "1. Figure 1 should be explained in more detail. What is wrong in the image of orange cat? Why are the top and bottom images similar although the token indices are different?\n2. Resolving weakness 1 will raise my rating regarding originality.\n3. Resolving weakness 2 will raise my rating regarding soundness.\n4. Resolving weakness 3 will raise my rating regarding clarity. If the proposed method is applicable to VAR, please help me find the statement in the paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Originality:\n1. The idea of aligning the embeddings of the generator and the tokenizer is original.\n2. Please see weakness 1.\n\nQuality:\n1. The experiments thoroughly compare many competitors in Table 1.\n2. Please see weakness 2. \n3. Ablation study is thorough.\n\nClarity:\n1. The problem statement, logical development, solution, and empirical supports are clear as described in Summary.\n2. Please see weakness 3.\n\nSignificance:\n1. The inconsistency between the embeddings of generator and tokenizer and its solution are important because they are the fundamental components of AR models.", "weaknesses": "1. Perturbing something is a common practice for robustness. Why is the proposed perturbation non-obvious compared to the literature?\n2. Experiments are conducted only on ImageNet.\n3. The scope should be specified in more detail such as raster-order autoregressive modeling because the paper does not tackle other autoregressive models such as [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction].\n\nminor\n\n1. The term \"context\" should be defined. It is okay-ish to be implied from context (lol) but it can be clearer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rojR59s3Bi", "reviewer": "Reviewer_Xk7f", "summary": "This work identifies a key bottleneck in visual autoregressive generation—the mismatch between the generator’s token sequences and how the tokenizer decodes them. The authors propose a plug‑and‑play regularization strategy that forces the autoregressive model to align its hidden representations with the tokenizer’s codebook embeddings and to remain robust under noisy contexts. This approach requires no changes to the tokenizer, inference pipeline or generation order, yet yields substantial gains. The method is validated across different tokenizers, showing that improving generator ‑ tokenizer consistency significantly boosts AR image generation.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper clearly identifies a fundamental issue in visual autoregressive models — the inconsistency between the generator and tokenizer — and presents a well-motivated solution.\n\n2. The proposed generator–tokenizer consistency regularization is simple, plug-and-play, and does not require changes to the tokenizer, inference pipeline, or generation order, making it broadly applicable.\n\n3. Extensive experiments demonstrate that reAR significantly enhances class-conditional image generation, yielding notable improvements in FID on ImageNet.", "weaknesses": "1. The paper does not report the impact of REAR on downstream multimodal understanding tasks or text-to-image generation benchmarks, leaving the broader utility of the method unclear.\n\n2. While the method improves generation quality, the novelty is somewhat limited, as the approach mainly introduces a regularization term rather than a fundamentally new architecture.\n\n3. The study only investigates reAR’s generalization across different VQ tokenizers, but does not examine its effectiveness when applied to different LLM backbones; additionally, the number of VQ tokenizers included in the experiments is limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1VpwEhhup5", "reviewer": "Reviewer_n9Nt", "summary": "This paper analyzes the performance bottleneck in visual autoregressive (AR) models, attributing it to \"generator-tokenizer inconsistency.\" To address this, the authors propose reAR, a plug-and-play training regularization strategy. Without altering the model architecture or inference pipeline, reAR introduces two auxiliary objectives: 1) Noisy Context Regularization, which mitigates exposure bias by training the model in corrupted contexts, and 2) Codebook Embedding Regularization, which forces the model's hidden states to predict the visual embeddings of both the current and next tokens, making the generator aware of the tokenizer's embedding space.", "questions": "1. reAR innovatively regularizes both the \"current token\" embedding (shallow layers) and the \"next token\" embedding (deep layers). Is there a potential conflict between these two objectives? (e.g., could the shallow layers lose information needed to predict the *next* token in their effort to align with the *current* token?).\n2. Could you provide an ablation study showing that only regularizing the \"next\" token's embedding (which seems like the more direct objective) performs worse than regularizing both?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. reAR delivers excellent results, significantly improving model performance without requiring architectural changes, and appears to be a general-purpose strategy for vision AR models.\n2. As a training-only strategy, reAR is highly versatile. Experiments show it works well not only with VQGAN but also significantly boosts other tokenizers like TiTok and AliTok.\n3. The paper clearly defines the \"generator-tokenizer inconsistency\" problem. It uses well-designed comparison experiments (e.g., \"perfect context\" vs. \"imperfect context\" and \"error token embedding replacement\") to convincingly validate that \"exposure bias amplification\" and \"embedding-unawareness\" are indeed critical bottlenecks.", "weaknesses": "1. reAR requires applying regularization at specific shallow (e.g., layer 0) and deep (e.g., layer 15) layers, chosen based on CKA analysis and ablations. This feels somewhat like \"alchemy\" or fine-tuning, lacking an adaptive or theoretically-driven mechanism to automatically determine the optimal layers for feature alignment.\n2. The method introduces additional MLP projection layers and an extra loss term, which inevitably increases training complexity and memory consumption. The paper doesn't explicitly quantify the additional training time overhead introduced by the reAR strategy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JIO4uEbDbN", "reviewer": "Reviewer_RcEf", "summary": "This paper proposes reAR, a new regularization framework designed to improve the generation quality of visual autoregressive (AR) models. The authors argue that existing visual AR methods suffer from a generator–tokenizer inconsistency, which manifests as exposure bias amplification and embedding unawareness during training and inference. reAR introduces two lightweight regularization techniques: Noisy Context Regularization, which injects random noise into the input token sequence to simulate imperfect contexts during training, mitigating exposure bias. Codebook Embedding Regularization, which aligns the generator’s hidden states with the tokenizer’s embedding space through cosine distance loss.", "questions": "1. How sensitive is reAR to the noise schedule or regularization strength λ across different datasets and architectures?\n2. Since the proposed method involves embedding alignment, how does it interact with tokenizers of different codebook geometries?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper identifies a genuine and underexplored problem: the misalignment between generator and tokenizer in visual AR models. This perspective is well-motivated and conceptually interesting.\n2. The proposed regularizations are straightforward, computationally light, and compatible with existing AR models and different tokenizers.\n3. The approach yields improvements across multiple tokenizers (e.g., TiTok, AliTok) and model sizes, and narrows the gap with diffusion-based methods.", "weaknesses": "1. While the paper frames generator–tokenizer inconsistency as a new perspective, the actual solutions (noise injection and embedding regularization) looks similar to existing techniques in language modeling.\n2. For a method positioned as “plug-and-play” and intended to generalize across tokenizers and datasets, the paper applies the embedding alignment regularization to specific layers but does not provide analysis behind this choice. The decision appears empirical, based on trying a few configurations and selecting the one yielding the best FID.\n3. While reAR demonstrates improvements across different tokenizers (e.g., TiTok, AliTok), the paper does not provide any analysis of how the proposed embedding regularization interacts with the underlying codebook geometry.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)