Towards Sequence Modeling Alignment between Tokenizer and Autoregressive Model
Authors: Pingyu Wu
Kai Zhu
Yu Liu
Longxiang Tang
Jian Yang
Yansong Peng
Wei Zhai
Yang Cao
Zheng-Jun Zha
Venue: ICLR 2026 Poster (2026)
Keywords: Image Generation
Autoregressive model
Tokenizer
Primary area: generative models

Abstract: Autoregressive image generation aims to predict the next token based on previous ones. However, this process is challenged by the bidirectional dependencies inherent in conventional image tokenizations, which creates a fundamental misalignment with the unidirectional nature of autoregressive models. To resolve this, we introduce AliTok, a novel Aligned Tokenizer that alters the dependency structure of the token sequence. AliTok employs a bidirectional encoder constrained by a causal decoder, a design that compels the encoder to produce a token sequence with both semantic richness and forward-dependency. Furthermore, by incorporating prefix tokens and employing a two-stage tokenizer training process to enhance reconstruction performance, AliTok achieves high fidelity and predictability simultaneously. Building upon AliTok, a standard decoder-only autoregressive model with just 177M parameters achieves a gFID of 1.44 and an IS of 319.5 on ImageNet-256. Scaling to 662M, our model reaches a gFID of 1.28, surpassing the SOTA diffusion method with 10x faster sampling. On ImageNet-512, our 318M model also achieves a SOTA gFID of 1.39. Code and weights at https://github.com/ali-vilab/alitok.

Published as a conference paper at ICLR 2026
TOWARDS SEQUENCE MODELING ALIGNMENT BETWEEN TOKENIZER AND AUTOREGRESSIVE MODEL
Pingyu Wu Wu Kai Zhu 12.,, Yu Liu?, Longxiang Tang² Jian Yang', Yansong Peng 12 Wei Zhai', Yang Cao', Zheng-Jun Zha 1:f
University of Science and Technology of China wpy3647555 620 20aai.mma......mm..mmm..mm.mmmmm ustc. edu.
Tongyi Lab
Figure 1: 256 samples of class-conditional generation on ImageNet using our AliTok-XL model (662M).
ABSTRACT
Autoregressive image generation aims predict the next token based on previ- ous ones. However, this process is challenged by the bidirectional dependencies inherent in conventional image tokenizations, which creates a fundamental mis- alignment with the unidirectional nature of autoregressive models. To resolve this, we introduce AliTok, a novel Aligned Tokenizer that alters the dependency struc- ture of the token sequence. AliTok employs a bidirectional encoder constrained by a causal decoder, a design that compels the encoder to produce a token sequence with both semantic richness and forward-dependency. Furthermore, by incorpo- rating prefix tokens and employing two-stage tokenizer training process to en- hance reconstruction performance, AliTTok achieves high fidelity and predictabil- ity simultaneously. Building upon AliTok, standard decoder-only autoregressive model with just 177M parameters achieves gFID of 1.44 and ImageNet-256 Scaling to 662M, our model reaches gFID of 1.28, surpassing the SOTA diffusion method with 10x faster sampling. On ImageNet-512, our 318M model also achieves SOTA gFID of 1.39. Code and weights at AliTok
of 319.5 on
INTRODUCTION
Autoregressive models, particularly GPT-style decoder-only transformers (Achiam et al. 2023 Tou- vron al. 2023a b). have achieved revolutionary success in natural language processing owing their simple yet scalable next-token prediction paradigm. Inspired by this triumph, the research com- munity actively exploring the application of this powerful paradigm image generation (Team) et al. 2025 Han et al. 2024: Wei et al. 2025) by sequentially predicting a stream of tokens, show- casing a promising path for multi-moda unification (Team 2024 Wu et al., 2025 Chen al. 2025).
"Equal contribution.
Corresponding author.

---

Published as a conference paper at ICLR 2026
Imgs y
Conventional Approach
Imgs
Intuitive Approach
Imas
Our Approach
Bidirectional
Transformer
Tol. Enc
Tol. Ene
Tok Enc
Causal
AR Generation:
AR
AR Generation: Asct: 10.7% g GFID 1.88
Tranffrmer
Tok. Dec
Tol: Dec
Generation:
Tok Dec
5.4%
Auc 11.2%
Encoded
Revonstmuctio::
gFID 2.96
Reconstruttoo: 1FIDI:1.56
gFID 2.19
Recurstruction: rFID; L.07
Tokens
IFIDI:
(a)
(b)
(c)
Figure 2: Reconstruction us generation with different transformer- tokenizers. Images are compressed into raster-scan order ID sequences by the tokenizers. AR are standard decoder-only autoregressive models. green for poor results and red for good results. The best results are bolded. Tok. Tokemizer. Acc: Training accuracy. Fair setup with matched parameter counts and computational loads. See Appendix A.2 for details.
However, as demonstrated in previous literature (Pang et al. 2024; Li ct al. 2024b), the inherent
bi-directional property of visual sequences makes it difficult for the raster-scan decoder-only autore- gressive model (e.g., LlamaGen (Sun et al., (024g) to achieve excellent performance, since the uni- directional transformer is hard to model bi-directional sequences. Consequently, recent works have
shifted focus towards paradigms like masked autoregressive (Li et al.] [024b)] Chang et al. 2022
Yu et al. 2024c Fan al., 2024) and next-scale prediction (Tang et al. 2024] Tian et 2024
Han et al. 2024), which employ bidirectional attention in autoregressive models and demonstrate a
superior choice for visual modeling. However, these approaches complicate visual generation and
diverge from the traditional autoregressive paradigm, increasing the challenges for multi-modal uni- fication. In contrast to existing works that modify the model to fit the data properties, we propose an alternative perspective: Can we instead instill forward-dependent align with the powerful simplicity of decoder-only AR models?
into the token sequence, to
Unlike natural language, which is inherently compact and allows for one-to-one mapping between words and indices with a non-parametric tokenizer, images are high-dimensional and redundant, requiring a learnable tokenizer for effective compression. In pursuit of maximal reconstruction fi- delity, conventional tokenizer implicitly incentivizes global collaborative encoding among all tokens to efficiently eliminate redundancy. This means the representation of a token becomes functionally dependent on its non-causal context, particularly on subsequent tokens in the raster-scan order. This bidirectional dependency of representations creates fundamental conflict with the strictly unidirec- tional predictive paradigm of autoregressive models. Since the target token depends on unseen future content, the conditional probability distribution learned by the AR model exhibits extremely high uncertainty, rendering the learning task exceedingly complex and limiting its generative capability.
To empirically validate this analysis, we construct an intuitive method where the encoder is forced to adopt a causal structure, strictly forbidding the preceding image patches access to future infor- mation and thus ensuring a purely unidirectional dependency. As shown in Fig. b)) the results are instructive: the training accuracy of AR model skyrockets from 5.4% to 11 11.2%, demonstrating that the predictability of sequence is substantially improved. However, this loss of global receptive field catastrophically degrades reconstruction quality (rFID: 1.58.) A core challenge there- fore emerges: Can we preserve the efficient compression bidirectional encoder while ensuring its output token sequence possesses the high predictability required by AR models?
To address this dilemma, we propose AliTok, whose core insight is to decouple the process of global semantic construction from the causal constraints of sequence. Specifically, AliTok lever- ages the global receptive field of a bidirectional encoder to build semantically rich representations, but critically, couples it with causal decoder that acts a powerful implicit regularizer. During reconstruction, the visibility of encoded tokens is strictly confined to the causal context along a raster-scan order. This architectural constraint, in turn, forces the encoder to suppress non-causal dependencies from its representations, thereby ensuring that the relevant contextual information re- quired for reconstruction is efficiently organized within each token's causal history. Ultimately, this process yields a token sequence that is both semantically rich and highly predictable. Experiments Fig. (cc) demonstrate that this approach achieves high reconstruction fidelity (rFID: 1.07) while dramatically enhancing generative performance (gFII: 2.96-+1.88)
Building upon this core design, we introduce several key implementation details address its prac- tical challenges and maximize performance. While the causal, raster-scan decoding is effective at enforcing forward-depeeneency. leads to poor reconstruction of the initial tokens (i.e., the first row
2

---

Published as a conference paper at ICLR 2026
of the image) due to lack of preceding context.
4.0
To address this problem, we introduce prefix to- kens specific to the first row, guided by dedi- cated auxiliary loss to provide the necessary con- textual priors for compensation. Finally, further enhance reconstruction quality without compromis- ing its friendiiosssiennnieesss,, we introduce a two- stage tokenizer training strategy. The second stage retrains a powerful bidirectional decoder over the frozen encoder and codebook, significantly boosting visual continuity and detail consistency.
Circles indicate model size
RAR
30 GEID wio
3.2
1.58
MAR
4T9M
17M
94M
2.4
15x Faster
675M
31834
LightningDir
1.6
Ours
5s
7s
9s
1is
13s
100s
200s
300s
2.4
20AM
Circles indicate
VAR
model size
gFID wi die
2.0
479M
To validate the effectiveness of AliTok, we employ a standard decoder-only autoregressive model as the generative model and conduct experiments. The re- sults, illustrated in Fig. 3 demonstrate a remark- able leap in performance and efficiency. Even with a 318M parameter model, our method already sur- passes all methods on gFID (w/o efg), including state-of-the-art diffusion model LightningDiT (Yao) ct al.] (025). Upon increasing the parameter count to 662M, we achieve a better gFID (w/ cfg) than LightningDiT (1.28 us 1.35), while providing a 10x faster sampling speed. In summary, the contributions of this paper include:
2B
MAR
1.6
461M
915M
RAR
1.3
043M
177M
LightningDit
Ours
663M
10x Faster
675M
1.2
7s
96
11s
13s
100s
200s
300s
Sampling time (total batch-128)
Figure 3: Sampling time and gFID (w/o efg and w/ cfg). Sampling time is evaluated on an A800,
1. We reveal key factor restricting autoregressive model efficacy: conventional tokenizers tend to establish bidirectional dependencies within encoded tokens, which fundamentally conflicts with the unidirectional nature of autoregressive models.
2. We propose simple yet effective design for an image tokenizer that enables the encoded tokens to more easily modeled by autoregressive models while ensuring high reconstruction fidelity.
3. Based on the proposed tokenizer, a standard decoder-only autoregressive model beats state-of- the-art diffusion models on ImageNet benchmark.
2 RELATED WORK
Autoregressive visual generation. Inspired by GPT-style language models (Achiam et al. 2023
Touvron et al. 2023a b). early visual autoregressive methods (Van Den Oord et 20.17 Lee et al. 2022) predict image tokens along the raster-scan order. However, this strict unidirectional paradigm struggles to effectively model the bidirectional spatial dependencies inherent in visual sequence, thereby limiting its performance ceiling. To resolve this fundamental conflict, research has pivoted to re-engineering the models or generation paradigms for stronger bidirectional context awareness. One prominent approach is masked autoregressive modeling, including MaskGIT (Chang et al.) 2022) and MAR (Li et al., (024b)), which leverages bidirectional attention for multi-round parallel generation. Another category explores hierarchical or next-scale prediction, such as VAR (Tian et al., 2024], which proceeds autoregressively across scales yet bidirectionally within each scale. Further- more, other efforts enhance global perception by maximizing all-directional generation probabilities (RAR, Yu al. ( (2024b)) or adopting random generation orders SanndAR,Paagge al. (0244). While effective, these approaches compromise the simplicity of the classical autoregressive paradigm by introducing complex generation mechanisms or training objectives. Unlike these strategies that adapt models to data, this paper focuses on reshaping the data itself. We propose a novel tokenizer that instills a causal dependency structure into the token sequence during the encoding stage, thereby enabling a standard decoder-only autoregressive model to unleash its full potential.
Image tokenizer, a type of Variational Autoencoder (VAE) (Kingma et al. 2013), plays an impor- tant role in visual generation domain (Wan et al. 2025 Wu et al. 2024 Wei et al.) 2024 Tang
et al. 2025 Peng et al.) 2025). By mapping sigh-dimensional image pixels to a low- dinddmniiiooal latent space, the image tokenizer significantly enhances the training efficiency of generative models. Among them, VQ-VAE (Van Den Oord et al. (017)) utilizes quantization techniques to discretize continuous latent features for autoregressive image generation. Subsequently, VQ-GAN (Esser et al.)
3

---

Published as a conference paper at ICLR 2026
substantially improves visual fidelity by incorporating adversarial loss. MagViT-v2 Yu et al. 2024a) and FSQ (Mentzer et al., 2023) focus on algorithmic improvements to efficiently scale the codebook size. Besides, TiTok Yu et al. 2024c) proposes transformer-basee tokenizer, encoding 2D image patches into a ID sequence. FlexTTok (Bachmann et al. 2025) explores multi- granularity semantic encoding, ranging from coarse to fine-grained. Despite numerous efforts, there has been limited exploration of how to design a tokenizer that simultaneously achieves high-quality reconstruction while being more conducive to subsequent autoregressive generation. Bridging this critical gap is the core goal of our work.
2021
METHOD
3.1 PRELIMINARIES
The modeling assumption of the autoregressive model is to factorize the joint probability distri- bution of sequence x [ri,..., into product of unidirectional conditional probabilities:
(= II Po (2; I
2
(1)
where pe is the parameterized autoregressive model. Consequently, the learning efficiency of an autoregressive model fundamentally depends on whether its modeling target, the token sequence x, possesses strong unidirectional dependency structure. When the critical information required to predict z; is predominantly contained within its preceding context ,2,...,,,;;11, the learning task for the model becomes significantly more well-defined and efficient.
Image tokenizer. The inherent spatial continuity in images leads to redundancy not only within individual patches but also extensively among adjacent ones. To achieve effective compression, multiple tokens must collaborate to eliminate overall redundancy and construct a compact represen- tation, which inevitably creates strong bidirectional representational dependencies among encoded tokens. This directly conflicts with the autoregressive learning paradigm: the model aims to learn the conditional distribution p(z; x=)a), while the ground-trun -ruth token z; has representation that implic- itly depends on the future context æ>i. Consequently. the target conditional distribution p(z; (Ees)) becomes marginalization over all possible unseen futures Z> Such distribution is inherently high-entropy, severely limiting the model's convergence and generation quality.
3.2 ALITOK TOKENIZER
To resolve the dilemma posed in Introduction (Sec. 1. we introduce AliTok, novel tokenizer. Its core insight to employ a causal decoder to constrain the training of bidirectional encoder, com- pelling it to produce highly predictable token sequence while retaining its efficient compression capabilitics. Furthermore, to address the issue insufficient initial context arising from the causal constraint, we introduce extra prefix tokens for contextual priming. Finally, mitigate the trade-off by adopting a two-stage tokenizer training process: the first stage learns a generation- friendly encoder, while the second stage retrains a separate, high-fidelity bidirectional decoder.
Definition,
As depicted in Fig. the architecture of AliTok is built upon a vanilla vision transformer. An input image E R fxxwxx is first processed into sequence of patch tokens P E R Here, H h/ f. W = w/f, D is the number of channels, and f is patch size (set to 16). Simultane- ously, we introduce K+ x W latent tokens in the input space (with K = W = 16, H 256) information carriers, with the first K tokens serving as prefix tokens, and the remaining H W latent tokens corresponding to H x W patch tokens. After concatenating the latent tokens with the patch tokens, the sequence is fed into the encoder (Enc) for tokenization, which compresses the in- formation of the image patches into ID latent tokens. Subsequently, the patch tokens are discarded, and we retain only the encoded latent tokens, denoted as Z E HH x Finally, these tokens Z undergo vector quantization (Quant) (Van Den Oord et al. 2017) (see Appendix A.3 for details) and pass through the decoder (Dec) reconstruct the image patch sequence } F
Causal decoder. The core of our method lies in the first stage of training, which aims to produce generation-friendyy encoder and codebook. To achieve this, introduce causal decoder
4

---

Published as a conference paper at ICLR 2026
Stage
Stage
Vector Quantization
63
21
58
[S]
92
63
[B]
92
63
58
Transformer (Bidirectional)
Transformer
Transformer
Transformer
Enc
Dec (Causal)
AR
(Causal)
Dec (Bidirectional)
5B
Ignoce
toleens
Patch tokens
Aut Lows
Reconataction Lous
Choss-Entnger Lose
Meconstrsectin Loss
(a) Tokenizer Encoder
(b) Tokenizer Decoder
(c) Image Generation
(d) Bidirectional
Tokenizer Decoder
Prefix token
Start token
Buffer token
Freeze the encoder and codebook
Figure 4: Two-stage training process the proposed AliTok, Stage 1: Training an image tokenizer with a causal decoder. Stage 2: Freezing the encoder and codebook of the tokenizer, training the autoregressive model and retraining a bidirectional tokenizer decoder.
critical architectural constraint. Unlike conventional bidirectional decoder, the causal decoder's visibility of the encoded tokens is strictly confined to a causal, raster-scan order during reconstruc- tion. Specifically, the reconstruction of the i-th image patch Pr is conditioned only on its causal context. This process is formally defined as:
{pu Deceausal (Quant(zz))H))) (z+)
(2)
This architectural constraint acts powerful implicit regularizer. It compels the bidirectional encoder to alter its encoding behavior in order to minimize reconstruction loss under this limited re- ceptive field, ultimately learning an antoregressively-strrturme information required to reconstruct patch p: is strategically organized within the causal sequence 21...i- This learned strategy establishes a forced alignment between the token dependency structure and the autoregressive generation process (Fig. b) and (c)). As result, the next-token prediction task becomes substantially more well-defined for the AR model, enabling more stable and effective training process that directly leads to higher generation quality.
representation, where the contextual
The reconstruction loss Lrecon for the first stage, follows standard practices Yuu et al. 2024c), combining a mean squared eтror (MSE) loss Lmse, perceptual loss Lpere: quantization loss (Zheng Vedaldi, 2023) Liuantt and adversarial loss Lindo:
Lypecon
Limse + Lapere + Liquant + ALeadu
where À is set to 0.1. For adversarial loss, we employ the GAN in Open-MagViT2 (uo al. 2024
Prefix token. However, a causal decoder forces latent tokens to only reference features from pre- ceding tokens, leading to poor reconstruction of the first row (16x256, pixels) of the image. To solve this problem, an intuitive solution is to add an extra row to the training image, i.e., using images with 272x256 256 resolution for training, and crop out the unsatisfactory first row in the final gener- ated result. But this simple approach may result in generating images with incomplete objects, as information in the topmost row is lost during cropping.
To provide contextual priors for the problematic first row, we introduce 16 prefix tokens as supple- mentary aids, each dedicated a patch the first row. These tokens are optimized via specialized auxiliary loss, Liwas which incorporates both MSE and perceptual losses. Since the perceptual loss requires full image for accurate assessment, we form a complete image by concatenating the first row, reconstructed from the prefix tokens, with the remaining 15 rows, reconstructed from the last 240 latent tokens. Critically, we detach the gradient from the latter component before concatenation. This design allows the perceptual network to evaluate a spatially coherent image while ensuring the optimization signal is backpropagated only to the reconstruction result of prefix tokens.
Retraining a bidirectional decoder. In the second stage, we freeze the encoder and codebook of the tokenizer and retrain a bidirectional decoder to improve detail consistency. As shown in Fig. d)) in addition to converting the attention mechanism in the decoder, we integrate 64 buffer tokens (Li et al. 2024b) to enhance modeling capabilities by increasing computational load. Meanwhile, the loss previously imposed prefix tokens is removed to allow the decoder to focus reconstruc- tion quality. This two-stage training strategy enables our AliTok tokenizer to not only produce generation-friende encoded tokens, but also maintain good reconstruction performance.
5

---

Published as a conference paper at ICLR 2026
Table 1: ImageNet 256 256 conditional generation. "Diff":: Diffusion. "Mask.": Masked transformer Precision. Rec.; Recall. RAR does not report results w/o efg. Thus, we test using the weights provided the original paper, adjusting the temperature at intervals of 0.01 to select the best gFID w/o cfg.
models, Pre.:
Training #Para. epochs
w/o cfg
w/ cfg
Type
Generator
gFIDL
1St
gFID
1St
Pre.
Rec.
SiT-XL Ma et al. REPA
2024
800
675M
8.61
131.7
2.06
270.3
0.82
0.59
Diff.
Yu ct all
2024d
800
675M
5.90
157.8
1.42
305.7
0.80
0.65
LightningDii
Yao al. 2025
800
675M
2.17
205.6
1.35
295.3
0.79
0.65
VAR-d24 TTan et al. 2024
350
1.0B
2.09
312.9
0.82
0.59
VAR
VAR-d30 TTan et al.
2024
350
2.0B
1.92
323.1
0.82
0.59
MaskGIT Chang al. MAGVIT- vZ
2022
300
227M
6.18
182.1
Mask.
Yu
2024a
1080
307M
3.65
200.5
1.78
319.4
TYTok-S- 128 Yu
2024c
800
287M
4.44
168.2
1.97
281.8
MAR-B
Li
2024b
800
208M
3,48
192.4
2.31
281.7
0.82
0.57
MAR
MAR-L
al.
20246
800
479M
2.60
221.4
1.98
290.3
0.81
0.60
MAR-H H
ct al.
2024b
800
943M
2.35
227.8
1,55
303.7
0.81
0.62
LlamaGen-XI.
Sun et al. 2024)
775M
2.62
244.1
0.80
0.57
LlamaGen-XXy
Sun et al 2024
300
1.4B
2.34
253.9
0.80
0.59
LlamaGen-3B
Sun et al. 2024
300
3B
9.38
112.9
2.18
263.3
0.81
0.59
Causal
AR
RAR-B Yu et al.
2024b
400
261M
7.12
124.8
1.95
290.5
0.82
0.58
RAR-L L
Yu et al. 2024b)
400
461M
5.39
149.1
1.70
299.5
0.81
0.60
RAR-XI XI.
Yu al.
20244
400
955M
3.72
179.9
1.50
306.9
0.80
0.62
RAR-XXI.
Yu
2024b)
400
1.5B
3.26
193.3
1.48
326.0
0.80
0.63
AliTok-B
800
177M
2.40
177.1
1.44
319.5
0.77
0.65
Causal AR
AliTok-L
800
318M
1.98
200.8
1.38
326.2
0.78
0.65
AliTok-XL
400
662M
1.88
228.4
1.28
306.3
0.79
0.65
3.3
AUTOREGRESSIIE MODEL.
Our decoder-only autoregressive architecture follows the standard design in LlamaGen (Sun al. 2024], which uses RMSNorm (Zhang & Sennrich 2019) for pre-normalization and applies 2D rotary positional embeddings (RoPE) (Su et al.) 2024) each layer. Building upon LlamaGen, we make only few modifications. First, since our method has 16 extra prefix tokens and needs to model 272 tokens. 2D RoPE cannot used directly. Therefore, we use 1D RoPE for the prefix tokens and 2D RoPE for the remaining 256 tokens. Secondly,
introduce the QK-Norm operation (Team 2024) in the attention module to stabilize the large-scale model training, aligning with recent multi- modal autoregressive models (Xie et al., 2024; Ma et al., 2025 Team 2024). We do not conduct extensive exploration of autoregressive models and only use the conventional architecture to verify the effectiveness of the proposed tokenizer.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
The design of the proposed tokenizer based on TA-TiTok (Kim al., 2025), utilizing ViT-B (Yu et al.) 2024c) for the encoder and ViT-L for the decoder. We set the vocabulary size to 4096 and adopt the online feature clustering method (Zheng & Vedaldi 2023)
Models and evaluations.
to ensure that the codebook utilization is 100%. For simplicity, we call the autoregressive models trained using our AliTok tokenizer as AliTok-B/L/XL, comprising three different sizes, more details provided in Appendix A.4 Following common practice, FID (including rIFD and gFID) (Heusel et al. 2017). IS (Salimans et al. 2016), Precision, and Recall are adopted as metrics. We generate 50,000 images to test the gFID and adopt the evaluation code from RAR (Yu et al. 2024b). For rFID, we use the ImageNet- 1K val set for evaluation, consistent with other methods (Sun et al.] [2024] Yu
al. 2024c Yao et al. 2025). During the test, KV-cache is employed to enhance sampling speed.

---

Published as a conference paper at ICLR 2026
Beacun (437)
Cural (973)
Maczu (88)
cream (928)
Fountain (582)
Spuce shutdl (812)
Altar (406)
AllTo- (177M)
AlTok-XL (662M)
Figure 5: 256 256 samples generated by our models of different sizes.
Dataset and training details. We train our image tokenizer from scratch on ImageNet-IK (Rus-) sakovsky et al. 2013). The tokenizer is trained for 600K steps in the first stage and 300K steps in the second stage 32 A800-80G GPUs. For the autoregressive model training,
employ ten- crop (Szegedy et al. 2015) for data augmentation and cache (Li et al. (024b) the encoded results of the tokenizer reduce training time. We train the base model and large model for 800 epochs and the XL model for 400 epochs, with a batch size of 2048 and learning rate of 4e-4. For the first 100 epochs, the learning rate is linearly warmed up, then decayed using a cosine decay schedule down to 1e-5, following RAR (Yu et al. 2024b). During training, class conditioning is randomly dropped with a probability of 0.1 to support the use of classifier-fres guidance (cfg) (Ho Salimans 2022)
4.2
MAIN RESULTS
Generation comparison. We report a comprehensive comparison with state-of-the-arr methods on the ImageNet- IK 256x256 benchmark in Table ! Utilizing our AliTok image tokenizer, even standard autoregressive model demonstrates exceptional performance. Specifically, our AliTok-B model achieves gFID (w/ cfg) of 1.44, using less than 6% of the parameter count compared to its raster-order counterpart LlamaGen-3B (Sun et al.] 2024), which achieves a gFID of 2.18 with 3B model. Furthermore, our AliTok-L model, with parameter count of 318M, attains a gFID (w/ efgg of 1.38, eclipsing all existing autoregressive methods and outperforming the sfate-of-the-aat autoregressive model RAR XXL (1.5B parameters) (Yu al.] 2024b) in both IS and gFID. Upon increasing the parameter count to 662M, our Ali beats all competing methods, particularly in gFID w/o efg, markedly outperforming the previous best method LightningDiTT (Yao et al. 2025) (1.88 us 2.17). When using the efg, our Al. achieves a gFll) of 1.28, surpassing Light- DiT (1.35) while also achieving a superior IS. To the best of our knowledge, this represents the first time standard autoregressive model beats mate-of-the-att diffusion models.
Sampling efficiency comparison. Beyond its excellent generation quality, AliTok also demonstrates a significant advantage in sam- pling efficiency benefiting from the of KV-cache. As shown in Table [ with simi- lar gFID scores, AliTok-L improves through- put by 33.7x and 2.0x compared to MAR- H (Li al. 20246) and RAR-XXL (Yu et al. 2024b), respectively. Compared to the state- th- diffusion model LightningDiT (Yao)
Table 2: Sampling speed comparison. Throughput (images/sec) measured on an A800 GPU (FP32, batch size=64), using the original codebases for all methods.
Method
Type
#Para. gFIDJ images/sect
VAR-d30
VAR
2.0B
1.92
12.3
AliTok-B B (ours)
AR
177M
1.44
10.9
MAR-H RAR XXL
MAR
943M
1.55
0.3
AR
1.5B
1.48
5,0
AliTok-L (ours)
AR
318M
1.38
10.1
et al. (025), our AliTok-XI. requires less than 10% of the time needed by LightningDiT to
LightningDiT
Diff.
675M
1.35
0.6
AliTok-XL (ours)
AR
662M
1.28
6.3
produce image, establishing a substantial
improvement in image generation efficiency.
Generation visualization. We further show the generation results under different model sizes in Fig. 5 It can be observed that even the smallest model is capable of producing high-quality visual results and reasonably generating complex structures, such as the coral reef and altar. As the model size increases, the generated images exhibit enhanced detail and more intricate structures, illustrated by the realistic texture of the feathers of the parrot, the fine-grained structure of the space shuttle
7

---

Published as a conference paper at ICLR 2026
Traiiing Emer Rahe (hi))
Altik & (1/M)
Aatck (1/M
Alfobd (27M)
(0196
JPID wio
10
230000
Traistig
Trinnng (c) 2FID elz
Traintig
(9) Trxining Lins
(b) Training Error Rate (%6
(4) wi dg
Figure 6: Training curves. (a) training loss (b) training error rate (%) (c) gFID scores w/o efg (d) gFID scores w/ ofg varies with training steps and model parameters. 250,000 training steps corresponds to 400 epochs.
Table 3: Ablation studies of main components on Ali Base model. Experiment (A) employs a bidi- rectional Transformer as the baseline. Subsequent experiments progressively incorporate a causal tokenizer decoder (Causal Dec), prefix tokens (Prefix), auxiliary loss (Tauz), and two-stage tokenizer training strategy (Two-stage). Key metrics including AR training loss (Loss), AR training accuracy (Acc.), generation gFID, and reconstruction rFID are reported to show the impact of each component.
Tokenizer Setting
AR Training
Evaluation
Training Setting
Causal Dec Prefix Luwxe Two-stage
Loss
Acc.t
gFID]
1st
rFID]
(A)
5.75
5.4%
2.96
270.6
0.98
Tokenizer trained for 300K steps
(B)
5.18
10.7%
1.88
285.6
1.07
(C)
5.20
9.7%
1.86
283.5
1.01
AR for 200 epochs
(D)
5.06
10.2%
1.82
286.5
1.02
Best rFID & AR for 800 epochs
(E)
4.98
10.5%
1.47
316.3
0.91
(F)
V
4.98
10.5%
1.44
319.5
0,86
and the altar, demonstrating a superior level of realism in image quality, More results generated by the different model sizes are provided in the Appendix (Fig. [4]] [6]
17 and [8].
Training curves. As shown in Fig. 0 we explore the training behavior of the proposed AliTok. With the increase in model parameters and training steps, both the training loss and training error rate decrease significantly and steadily. Additionally, larger models achieve lower loss and superior gFID scores with fewer training steps. Experiments verify that our generative model effectively in- herits the scaling ability of autoregressive models. However, the gFID curves indicate that after 400 training epochs, neither our AliTok-B model nor AliTok-L. model shows clear signs of convergence. Therefore, extend the training duration for these two models 800 epochs to further improve the performance. More analysis is provided in Appendix A.4
Reconstruction performance. We report the reconstruc-
Table 4: Reconstruction quality, Size: vocabulary size.
tion performance of various tokenizers Yu et al., 2024c
Zheng & Vedaldi 2023 Xiiong et al.; (025) in Table 4
By introducing a powerful bidirectional decoder in the sec- ond stage, Ali'Tok significantly improves the rFID to 0.86. This result remains competitive with GigaTok-B-L (rFID: 0.81), particularly given that the latter relies on substan- tially larger vocabulary size (16384 us 4096). Experiments demonstrate that our proposed AliTok successfully achieves an ideal balance between the goals of generation -riinndly and high- fidelity through its ingenious decoupling design.
Tokenizer
Tokens
Size
rFID
CVQ-VAR TiTok- B-64
256
1024
1.57
64
4096
1.70
GigaTok- B-L
256
16384
0.81
AliTok (Stage 1)
272
4096
0.91
AliTok (Stage 2)
272
4096
0.86
4.3 ABLATION STUDIES
Ablation studies of main components. In Table 3 we ablate the main components of the proposed AliTok. First, introducing a causal decoder (B) dramatically improves predictability over the bidi- rectional baseline (A). This evidenced by the AR model's training accuracy soaring from 5.4% to 10.7% and its gFID dropping from 2.96 to 1.88, confirming the critical role of aligning token depen- dencies with the model's ' causal nature. Next, we examine the effect of prefix tokens. Simply adding them (C) not only increases the modeling burden due to a longer sequence, but these unconstrained tokens also tend to learn global features. This makes the prediction task for the autoregressive model more difficult and fails alleviate the poor first-row reconstruction (Fig. 7 In contrast, when
8

---

Published as a conference paper at ICLR 2026
()) Origina image
(b) Deseline
Baneline Cacsal Erc (4) Baseline Causal Dec (6) Pietii Tokens
+ Ayys
(g) + Two Stage
Figure 7:
Reconstruction visualization comparison. (a) Original image. (b, c) Comparison results. (d-g) Ablation study showing the progressive addition our components. red boxes highlight noticeable artifacts.
guided by the auxiliary loss Lauz (D), the prefix tokens are forced to represent the features of the first image row. This creates more progressive and consistent learning process, as the AR model consistently predicts token based on adjacent context, unifying the prediction task and leading to rebound in training accuracy (10.2%) and better gFID of 1.82. Finally, by reintroducing bidi- rectional decoder (F) through a two-stage training process, AliTok further enhances reconstruction fidelity and improves generative quality.
Ablation studies of reconstruction visualization. We visualize the impact of each component on reconstruction quality in Fig. First, adopting a causal encoder (c) introduces noticeable visual discontinuities and grid-like artifacts (highlighted by red boxes), as its unidirectional structure limits compression efficiency. In contrast, causal decoder (d) yields generally natural reconstructions, but its sequential reconstruction process leads to poor quality in the first row, marked by severe blurring and distortion the top of the bookshelf. To address this, we introduce prefix tokens with Lauz (f) to provide the necessary contextual prior, leading to significant restoration of detail in the first row. Finally, retraining bidirectional decoder in the second stage (g) smooths patch boundaries and rectifies detail inaccuracies, such as the lack of clarity on the metal mesh. Through these sequential designs, our final reconstruction (g) becomes visually comparable to the baseline (b).
Visualization of attention maps. To demonstrate the intrinsic dependency structure within AliTok's s latent tokens, we analyze the attention maps from the bidirectional decoders of both baseline model and AliTok (Stage 2) in Fig. We first examine the mean local x3) attention pattern averaged across all tokens in Fig. b). As expected, the baseline's attention is symmetrically distributed, indicating a balanced reliance on past and future context. In stark contrast, AliTok's 's mean attention map reveals a
6.545
6:34
6.115
Image
patterns
clear systemic shift towards the top-left, demonstrat- ing a strong preference for causally preceding tokens even with full access to future ones. To demon- strate the generality of this behavior, Fig. (fc) dis- plays attention maps for several uniformly sampled tokens, which consistently exhibit this same causal bias. More examples are provided in the Appendix (Fig. [T]. This striking asymmetry provides com- pelling visual evidence that our training strategy ef- fectively regularizes the encoder. It compels the en- coder to learn nutoregressively--stutturr tations, where the necessary context for reconstruct-
(c) Attention map the selenne sakenc the tokentur decoder
Figure 8: Visualization of attention maps from bidirectional decoders. (a) The input image and selected tokens (red boxes) for analysis. (b) The mean local (3x3) attention patterns across all to- kens. (c) Attention maps for the selected tokens.
represen-
ing patch is primarily packed into its causal tokens. This process instills a strong causal bias sequence
into the token sequence itself, shaping to be inherently amenable forward prediction. Such fundamental alignment unlocks the high performance of standard autoregressive model.
Scalability in high-resolution generation. To verify the scalability of our method on the high-resolution image gen- cration task, we trained AliTok on the ImageNet dataset at 512x512 resolution. For this experiment, both the to- kenizer and autoregressive model are fine-tuned from pre- trained weights at 256 resolution, where the embedding of tokenizer is linearly interpolated to match the shape. The gen- eration results shown in Table 5 Our A achieves
Table 5: 512x512 generation quality.
Method
#params gFIDJ
ISt
REPA MAR-L
675M
2.08
274.6
481M
1.73
279.9
AliTok-L.
318M
1.39
295.3

---

Published as a conference paper at ICLR 2026
Figure 9: 512 samples of class-conditional generation ImageNet using our AliTok-L model (318M).
gFII of 1.39 and an IS of 295.3, significantly surpassing MAR-L eLi 2024b) and REPA et al., 2024d). The result highlights the strong scalability and robustness of our AliTok. Fig. provides generated samples, visually confirming the high quality of the outputs.
Scalability codebook size. To further validate the adaptability of the AliTok framework larger discrete vocabularies, we conducted an experiment expanding the codebook size from 4096 16384. As shown in Table 6 the 16384 codebook achieves a lower reconstruction FID (from 0.86 to 0.81), in- dicating higher upper bound for potential genera- tion quality. On the generation side, after training our Xizsized model for 400 epochs, the generative performance with the 16384 codebook (gFID: 1.30) is slightly inferior to that of the 4096 codebook (gFID: 1.28). We attribute this temporary performance gap the increased difficulty of the next-token prediction task within 4x larger vo- cabulary space, which clearly evidenced by the sharp drop in training accuracy and the rise in loss. Consequently, posit that with extended training epochs tokenizer with 16384 codebook size is expected to achieve better generation performance.
Table 6: AliTok-XL reconstruction and gener- ation performance at different codebook sizes. AR models are trained for 400 epochs.
Size
rFID.
gFID
1St
Loss
Acct
4096
0.86
1.28
306.3
4.67
12.2%
16384
0.81
1.30
304.9
6.42
4.3%
increase in model parameters, the
CONCLUSION
This paper introduces AliTok, novel tokenizer designed to mitigate the fundamental misalignment between conventional tokenizers and autoregressive models. By employing causal decoder to strain a bidirectional encoder, AliTok instills a significant causal bias into the image token sequence, making it inherently compatible with the autoregressive paradigm. This strategy enables stan- dard decoder-only generator to surpass state-of-the-arr diffusion models in both generation quality and sampling efficiency. Our work confirms that when data and the model paradigm properly aligned, the concise autoregressive approach remains a powerful pathway toward building efficient, gigh-perforancce generative models, paving the way for future multimodal unification.
ACKNOWLEDGGENNTS
This work is supported by the National Natural Science Foundation of China (NSFC) under Grants 62225207, 62436008, 62306295, and 62576328. The AL-driven experiments, simulations and model training performed on the robotic Al-Scientist platform of Chinese Academy of Sciences.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.08777 2023.
Roman Bachmann, Jesse Allardice, David Mizrahi, Enrico Fini, Oguzhan Fatih Kar, Elmira Amir- loo, Alaaeldin El-Nouby, Amir Zamir, and Afshin Dehghan. Flextok: Resampling images into Id token sequences of flexible length. arXiv preprint arXiv:2502. 13967, 2025.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In CVPR, 2022.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3558 3558-3568, 2021.
Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling, arXiv preprint arXiv:2501. 11 17811, 2025.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
Lijie Fan, Tianhong Li, Siyang Qin, Yuanzhen Li, Chen Sun, Michael Rubinstein, Deqing Sun, Kaiming He, and Yonglong Tian. Fluid: Scaling autoregressive text-to-image generative models with continuous tokens. arXiv preprint arXiv:2410. 13863, 2024.
Jian Han, Jinlai Liu, Yi Jiang, Bin Yan, Yuqi Zhang, Zehuan Yuan, Bingyue Peng, and Xiaobing Liu. Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis. arXiv preprint arXiv:2412.0444 241..24421,44 2024.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. 120.5 2022.
Classifier-free diffusion guidance.
arXiv preprint
Dongwon Kim, Ju He, Qihang Yu, Chenglin Yang. Xiaohui Shen, Suha Kwak, and Liang-Chich Chen. Democratizing text-to-image masked generative models with compact text-aware one- dimensional tokens. arXiv preprint arXivv2501.07730, 2025.
Diederik P Kingma, Max Welling, al. Auto-encoding variational bayes, 2013.
Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In CVPR, 2022.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.00 03326, 2024a.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. In NeurlPS, 2024b.
Zhuoyan Luo, Fengyuan Shi, Yixiao Ge, Yujiu Yang. Limin Wang, and Ying Shan. Open-magvit2: An open-source project toward democratizing auto-regressive visual generation. arXiv preprint arXiv:2409.04410. 2024.
Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden- Eijnden and Sain- ing Xie. SIT: Exploring flow and diffusion- based generative models with scalable interpolant transformers. arXiv preprint 2rXiv:2401.08744 2024.

---

Published as a conference paper at ICLR 2026
Xu Ma, Peize Sun, Haoyu Ma, Hao Tang, Chih- Yao Ma, Jialiang Wang, Kunpeng Li, Xiaoliang Dai, Yujun Shi, Xuan Ju, et al. Token-shuffle: Towards high-resolution image generation with autoregressive models. arXiv preprint arXiv:2504.1778, 2025.
Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantiza- tion: Vq-vae made simple. arXiv preprint urXiv:2309.1500 2023.
Ziqi Pang. Tianyuan Zhang, Fujun Luan, Yunze Man, Hao Tan, Kai Zhang, William T. Freeman, and Yu-Xiong Wang. Randar: Decoder-only autoregressive visual generation in random orders. arXiv preprint arXiv:2412.01822 2024.
Yansong Peng, Kai Zhu, Yu Liu, Pingyu Wu, Hebei Li, Xiaoyan Sun, and Feng Wu. Faem: Flow- anchored consistency models. arXiv preprint arXiv:2507.0377 03738, 2025.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-555 2015.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06522 2024.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015.
Haotian Tang, Yecheng Wu, Shang Yang, Enze Xie, Jumsong Chen, Junyu Chen, Zhuoyang Zhang, Han Cai, Yao Lu, and Song Han. Hart: Efficient visual generation with hybrid autoregressive transformer. arXiv preprint arXiv:2410. 108/2, 2024.
Longxiang Tang, Ruihang Chu, Xiang Wang, Yujin Han, Pingyu Wu, Chunming He, Yingya Zhang, Shiwei Zhang, and Jiaya Jia. Exploiting discriminative codebook prior for autoregressive image generation. arXiv e-prints, pp. arXiv-2508, 2025.
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.098/8, 2024.
NextStep Team, Chunrui Han, Guopeng Li, Jingwei Wu, Quan Sun, Yan Cai, Yuang Peng, Zheng Ge, Deyu Zhou, Haomiao Tang, et al. Nextstep-1: Toward autoregressive image generation with continuous tokens at scale. arXiv preprint arXiv:2508. 1001111181111 2025.
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng. and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. In NewrIPS, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint
2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint ArXiv:2307.09288 2023b.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In NeurIPS, 2017.
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang. et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.203/4 2025.
12

---

Published as a conference paper at ICLR 2026
Yujie Wei, Shiwei Zhang, Zhiwu Qing, Hangjie Yuan, Zhiheng Liu, Yu Liu, Yingya Zhang, Jingren Zhou, and Hongming Shan. Dreamvideo: Composing your dream videos with customized sub- ject and motion. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6537-6549, 2024.
Yujie Wei, Shiwei Zhang, Hangjee Yuan, Yujin Han, Zhekai Chen, Jiayu Wang, Difan Zou, Xihui Liu, Yingya Zhang. Yu Liu, al. Routing matters in moe: Scaling diffusion transformers with explicit routing guidance. arXiv preprint arXiv:2510.24711 2025.
Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 12966- 12977, 2025.
Pingyu Wu, Kai Zhu, Yu Liu, Liming Zhao, Wei Zhai, Yang Cao, and Zheng-Jun Zha. Improved video vae for latent video diffusion model. arXiv preprini arXiv:241 1.06449, 2024.
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. 12528, 2024.
Tianwei Xiong, Jun Hao Liew, Zilong Huang, Jiashi Feng, and Xihui Liu. Gigatok: Scaling visual tokenizers to billion parameters for autoregressive image generation. arXiv preprint arXiv:2504.08736, 2025.
Jingfeng Yao, Bin Yang, and Xinggang Wang. Reconstruction vs. generation: Taming optimization dilemma in latent diffusion models. arXiv preprint arXiv:2501.01423, 2025.
Lijun Yu, José Lezama, Nitesh B. Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, Alexander G. Hauptmann, Boqing Gong, Ming-Hsuan Yang. Irfan Essa, David A. Ross, and Lu Jiang. Language model beats diffusion- tokenizer is key to visual generation. In ICLR, 2024a.
Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chieh Chen. Randomized autoregres- sive visual generation. arXiv preprint arXiv:2411 1,007776, 2024b.
Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang- Chinh Chen. An image is worth 32 tokens for reconstruction and generation. arXiv preprint arXiv:2406.0755) 2024c.
Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is casier than you think. arXiv preprint 24X1v:2410.06940. 2024d.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Infor- mation Processing Systems, 32, 2019.
Chuanxia Zheng and Andrea Vedaldi. Online clustered codebook. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2023.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 LIMITATION AND FUTURE WORK
Benefiting from the AliTok tokenizer, our 662M AR model reaches a significantly high training ac- curacy of 12.2%. In this context, the reconstruction performance of the discrete tokenizer becomes a critical bottleneck for the generative performance. To test the limiting value of gFID that can be achieved under this tokenizer, calculate the gFID after reconstruction (differently from rFID evaluation), obtaining a FID of 1.15. This implies that even if the accuracy in the generation phase is close 100% (which is not possi- ble), the generation performance still cannot reach the upper limit of 1.15. This experimental result explains the phenomenon observed in Fig. [ where the training loss and error rate of AliTok-XL continue to decrease steadily, yet the gFID metric struggles to achieve further improvement. While our experiments with 16384-size codebook have demonstrated a higher ceiling for reconstruction quality (as shown in Table 5) they also revealed an increased learning challenge for the autore- gressive model. This paper does not explore generative models larger than our XL variant, and we leave the comprehensive exploration, such training larger-scale models or extending the training duration on the 16384 codebook to fully unlock its generative potential to future work.
randomly select 50 real images per class from the training set and
A.2 EXPERIMENTAL DETAILS IN INTRODUCTION
The experiments depicted in Fig. 2 of the main paper are conducted under controlled setting to ensure a fair comparison. For all three configurations, the tokenizer is trained for 300K steps, and the corresponding AliTok-Base autoregressive model is trained for 200 epochs. The conventional approach serves as our baseline. It employs a standard architecture with a bidirectional Transformer for both the encoder and decoder. The tokenizer for our approach uses bidirectional encoder but is constrained by a causal decoder. This configuration is identical to the model presented in Table of our ablation study. For the intuitive approach using a causal encoder, we consider the following factors to design the tokenizer: Firstly, it is challenging to compress the information within image patches into latent tokens with causal logic. Secondly, for a causal encoder, adding more learnalle tokens does not effectively increase computational load, as the direction of interaction is limited. Therefore, we remove the 256 latent tokens from the input space of the encoder and treat the encoded patch tokens as the encoding result. To ensure faimesss, we relocate the 256 removed latent tokens to the input space of the bidirectional decoder as buffer tokens. Concurrently, to maintain the computational load of the encoder, we reverse the model sizes of the encoder and decoder by using a ViT-L model for the encoder and ViT-B model for the decoder. In this way. we ensure that the computational effort of each tokenizer is similar and the design is reasonable. Finally, the codebook utilization of each tokenizer is 100% to avoid the impact on the training accuracy.
A.3 METHOD DETAILS
Vector quantization. After the latent tokens pass through the encoder, continuous encoded latent vector is produced. The nearest vector to z, based on Euclidean distance, is then queried from the discrete codebook Z to serve as the quantized result. This vector quantization process can be formalized by the following equation:
Quant(z) = argmin ||2 12-22 2
(4)
2, 2
Hyperparameters, In the first stage, we define the total loss Leorel as straightforward sum of the reconstruction loss (Iyee) and the auxiliary loss (Laux), treating them with equal importance:
Listal Liree + Lavx
(5)
Positional embedding in autoregressive models. In autoregressive models, employ ID RoPE for the 16 prefix tokens, with positions ranging from to 15. Following this, we use 2D RoPE for the remaining 16x 16 tokens, with positions extending from 16 to 31.
14

---

Published as a conference paper at ICLR 2026
Table Improvements and curves of training 800 epochs. 500,000 training steps corresponds to 800 epochs. Acc.: final training accuracy. Since we employ cosine learning rate decay instead of fixed learning rate, we train the models for 800 epochs from scratch rather than continuing training based on the weights from the 400 epochs.
gFID gFID
A (177M)
Aliokk (177M)
Model
Epochs Loss Acc.
w/o cfg w/ cfg
A (314M)
AlTok (314M)
400
5.02
10.3%
2.69
1.61
AliTok-B
800
4.98
10.5%
2,40
1.44
400
4.90
10.9%
2,09
1.49
AliTok-L
68.5
800
4.85
11.1%
1.98
1.38
Training Losa x Training Step
Training Frror Rote (96 xs Training Step
A.4 EXPERIMENTS
Model configurations. For simplicity, we call the autoregressive models trained using our AliTok tokenizer as BiToo-B//LXLL comprising three different sizes. For all autoregressive models, we set the number of attention heads 16. Specifically, AliTok-B has depth of 24 and a width of 768, AliTok-L has depth of 24 and width of 1024, and AliTok-XL has a depth of 32 and width of 1280, following the RAR (Yu al. 2024b) configuration.
Improvements of training 800 epochs. In Table 7 we show the enhancements achieved by in- creasing the training epochs from 400 to 800, which significantly reduces gFID scores both w/o cfg and w/ cfg. We further plot the training curves on the right side of Table 7 It is evident that the training loss and error rate continue to decrease linearly without convergence, suggesting that extending the number of training epochs may yield further improvements. However, since cosine learning rate decay rather than a fixed learning rate, additional epochs require training from scratch. Consequently, do not explore the improvements that might result from extending the training period.
60 Origgial Image
Buselie
Beseliee Carsil Eit
An Momiagge kow ofio)
6d Baselie Dessel
de AlTiok
An miarged View of
Figure 10: More reconstruction result comparisons. The same regions in (c) and (e) enlarged for detailed comparison.
More reconstruction results. Fig. 10 presents additional comparisons of reconstruction results, with enlarged views of the detailed regions from both the intuitive approach (baseline + causal encoder) and our AliTok approach. From the enlarged region, it obvious that the causal encoder causes texture mismatches between different reconstructed patches with discontinuities and detail deviations. Notably, significant grid artifacts persist even after the usage of adversarial loss. In contrast, our AliTok method achieves continuous and realistic reconstruction results. Furthermore, as seen in Fig. d)), solely applying the causal decoder on the baseline leads to color deviation and blurriness in the first of the reconstructed images due to poor reconstruction quality, especially in the first image patch. In certain scenes, like the edge of the plate the first row, unnatural transitions may also be present. However, after implementing our proposed series of improvements,
15

---

Published as a conference paper at ICLR 2026
Table 8: Comparison of training configurations and model specifications.
Tokenizer
TYTok-B
TYTok-L
GigaTok-B-A
AliTok (Stage 1)
AliTok (Stage 2)
Codebook size
4096
4096
16384
4096
4096
Codebook dimension
12
12
32
32
Training epochs
200
200
200
120
60
#Param
204.8M
641.1M
621.6M
389.8M
390.0M
Total batch size
256
256
Base learning rate
le-4
le-4
Minimum learning rate
le-5
1e-5
Table 9: Hyperparameters for sampling include without and with cfg.
Model
Temperature
Scaler Power
Guidance Scale
AliTok-B
0.95
cfg w/o cfg
AliTok-L
0.95
AliTok-XL
0.95
w/o cfg
AliTok-B
1.00
1.3
AliTok-L
1.00
0.6
AliTok-XL
1.00
1.4
including prefix tokens, auxiliary loss, and two-stage training strategy, this issue is significantly alleviated. Our AliTok (Fig. e)) can ultimately achieve similar reconstruction quality and visual perception as the baseline approach (Fig. b)).
More generation results, We present more generation results our AliTok-XL model in Fig. 14 As can be seen, method is capable of producing detailed textures and realistic image quality. In Fig. 15 Fig. 16, Fig. 17] Fig. 18 we present various images generated by AliTok-B and AliTok-XI. across different categories. Even smallest model can generate great visual outcomes.
Tokenizer training specifications. This subsection details the training configurations and model specifications for AliTok and other tokenizers, summarized in Table g As shown, our comparative experiments are conducted under fair conditions, with aligned batch sizes and learning rate sched- ules, ensuring the validity of our system-level performance analysis.
Hyperparameters for sampling. We list the sampling hyperparameter settings in Table 9 When cfg used, we employ a temperature hyperparameter, When cfş used, we apply pow-cosine as the guidance schedule, following RAR (Yu et al. 2024b).
Table 10: System-level reconstruction performance comparison across different tokenizers.
Tokenizer
Tokens
Size
PSNRT
SSIM
LPIPSJ
rFID,
#Param
TYTok-L-32
32
4096
16.15
0.5098
0.3232
2.21
641.1M
TYTok-B-64
64
4096
17.39
0.5491
0.2539
1.70
204.8M
AliTok (Stage 1)
272
4096
20.66
0.6571
0.1659
0.91
389.8M
AliTok (Stage 2)
272
4096
20.97
0.6610
0.1573
0.86
390.0M
GigaTok B-L
256
16384
21.24
0.6873
0.1284
0.81
621.6M
AliTok (Stage 1)
272
16384
21.08
0.6805
0.1367
0.84
390.2M
AliTok (Stage 2)
272
16384
21.30
0.6851
0.1303
0.81
390.4M
System-level comparison of reconstruction performance. To comprchensively evaluate the per- formance of our proposed AliTok on the image reconstruction task, we detail the performance of Al- iTok against other tokenizers across several standard reconstruction metrics, including Peak Signal- to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), Learned Perceptual Image Patch Similarity (LPIPS), and reconstruction FID (rFID).
The results in Table 10 highlight AliTok's competitiveness. In the comparison with GigaTok-B-I --L using 16384 codebook, method achieves on-par reconstruction fidelity, matching its rFID of
16

---

Published as a conference paper at ICLR 2026
0.81, while using approximately 37% fewer parameters (390.4M 621.6M). This performance also validates our two-stage design, as Stage significantly enhances the reconstruction quality from Stage to reach this state-of-the-an level. This demonstrates that AliTok maintain great reconstruction performance while simultaneously creating a generation-frienlly token sequence.
Oragnal tmage
Bandie
ATT
Soberad bld loa)
(4) Image tokens
(b) Antmtion mpp of the selected tokens the tosemizer decoder
Figure 11: More attention maps for selected tokens in the bidirectional decoder. In contrast to the symmetric attention distribution of the baseline, AliTok's attention exhibits strong causal bias (concentrating on the top-left), even within a fully bidirectional decoder.
Visualization of more attention maps. To confirm the generality of the attention pattern revealed in the main text, we provide additional attention maps in Fig. [1 for individual tokens various spatial locations, particularly from the edges and the center of the image. For the baseline model, its bidirectional decoder widely references the global context during reconstruction, with attention freely dispersed across both causal history and future regions. In contrast, the attention in AliTok's bidirectional decoder heavily skewed towards drawing information from historical tokens, largely ignoring the available context from future regions. This indicates that the necessary contextual in- formation for each token has been effectively organized within its causal history, thereby facilitating the subsequent unidirectional generation process.
Figure 12: t-SNE visualization of codebook vectors. The plot compares the feature distributions of codebook vectors used for single ImageNet class, generated by the baseline ViT-VQ tokenizer (blue) and AliTok (red).
Visualization of codebook. We performed a t-SNE analysis to visualize the codebooks learned by a standard ViT-VQ (conventional approach our paper) and AliTok. We encoded 200 images from single ImageNet class with each tokenizer, and collected the utilized codebook vectors for 2D projection. The resulting visualization in Fig. 12 shows a clear topological divergence. The base- line's feature space (blue) is diffuse and continuous, characterized by ambiguous cluster boundaries and significant feature entanglement. This suggests that the unconstrained bidirectional encoder creates non-causal dependencies across tokens achieve high reconstruction fidelity. AliTok (red),
17

---

Published as a conference paper at ICLR 2026
by contrast, learns well-structured discrete manifold of compact. clearly separated semantic clus- ters. This demonstrates that AliTok's causal decoder constraint forces the encoder to learn more deterministic and disentangled representations. This structured and predictable feature distribution presents a more learnable target for the autoregressive model, which partly explains the significant improvement in generation accuracy observed with AlíTok.
Table 11: KV-Caching Ablation (256x256, w/ cfg, batch-64).
Setting
Sampling Time (batch=64)
Speedup
AliTok-XI. (w/ KV-Cache) AliTok-XI. (w/ KV-Cache)
107.76
1.0x
10.07s
10.7x
Table 12: Sampling cost at different resolutions.
256x256 (batch=64, w/ cfg)
512x512 (batch=64, w/ cfg) Sampling time Memory
Tokenizer
Tokens
Sampling time Memory
Tokens
Baseline-Xi XL
256
9.20s
14.81G
1024
95.17s
46.98G
AliTok-XL A
272
10.07s
15.48G
1056
100.67s
48.33G
(+6.3%)
(+9.5%)
(+4.5%)
(+3.1%)
(+5.8%)
(+2.9%)
Detailed Analysis of Sampling Efficiency and Scalability. we provide a comprehensive analysis of the sampling efficiency of AliTok, explaining the primary driver behind its significant speed advantage and detailing the scalability of our approach with respect to the prefix tokens. First, AliTok's s 10x speedup over masked models and diffusion based models primarily stems from its full compatibility with Key-Value (KV) caching.
sequential generation, KV-caching stores past states to avoid re-computation at each step. Our ablation study in Table 11 confirms its dominant role: disabling KV-caching skyrockets inference time from 10.07s to 107.76s.
Regarding scalability, we further quantified the overhead of additional prefix tokens at higher resolu- tions. As shown in Table [2 the relative impact of these tokens decreases significantly as resolution increases. For instance, scaling from 256x256 to 512x512 reduces the extra sampling time over- head from 9.5% to 5.8%. This favorable trend occurs because prefix tokens grow linearly (O(W) while total tokens grow quadratically (O(Wx xH), making the overhead asymptotically negligible.
Curve stiching
Outdocr portmit girt year
Lego Graplic T-Shirt
A bost on the lake
The BMW 5 Series 2017
circle
Janus-pro-IB (Trained for 460K steps)
ChiRat
(Trained for 30K steps)
Figure 13: Comparison of text-to-image generation between Janus-pro-18 1B and ours-0.5B. Note that our model was trained for merely 30K steps.
Generalization to Multimodal Text-to- Image Generation. To investigate AliTok's generalization
capability, we conducted preliminary text-to-image experiment. Following the unified paradigm of Janus-pro (Chen et al. 2025), we built 0.5B parameter model where single autoregressive (AR) architecture handles both text understanding and visual generation. This model is based on LLaVA- OV-0.5B (Li et al. 2024a) and incorporates AliTok its visual tokenizer. The model was trained
18

---

Published as a conference paper at ICLR 2026
on the CC12M dataset (Changpinyo al. 2021 at 512x512 resolution for only 30,000 steps 64 A100 GPUs.
Fig. 13 shows qualitative comparison against the 1B-parameter Janus-pro model (trained for 460K steps). Despite the significant disparity in model scale and training duration, our model demonstrates the ability generate coherent images corresponding the text prompts. This result serves as initial validation AliTok's ss effectiveness and efficiency a challenging multimodal autoregressive setting, highlighting its potential for broader applications.
A.5 USE LARGE LANGUAGE MODELS (LLMS)
In accordance with ICLR 2026 policy, we report our use of large language model (LLM) during the preparation of this manuscript. The LLM's role was strictly confined to language polishing, such as correcting grammar and refining word choices. The authors are solely responsible for all scientific contributions, including the ideation, methodology, experimental design, and final conclusions. The authors have reviewed and take full responsibility for the final manuscript.
19

---

Published as a conference paper at ICLR 2026
Figure 14; More generation results the ImageNet 256x256 benchmark using AliTok-XI.
20

---

Published as a conference paper at ICLR 2026
AlITokBB (177M)
A (662M)
Bald cagle (22)
Goldfish
Check (436)
Figure 15: More generation results. The left images are generated by AliTok-B (177M), and the right images are generated by AliTok-XL (662M).
21

---

Published as a conference paper at ICLR 2026
AlToo-BB (177M)
AliTok-XL (662M)
Coral reef (973)
Schooner (780)
Boathouse (449)
Figure 16: More generation results. The left images generated by AliTok-B (177M), and the right images are generated by AliTok-XL. (662M).
22

---

Published as a conference paper at ICLR 2026
AliTok-B (177M)
AliTok-XL (662M)
Fountain (562)
Ice cream (780)
Agaric (980)
Figure 17: More generation results. The left images generated by AliTok-B (177M), and the right images are generated by AliTok-XL. (662M).
23

---

Published as a conference paper at ICLR 2026
AliTo-BB (177M)
AlTol-XL (662M)
Space (812)
Lorikeet (90)
Knot (616)
Figure 18: More generation results. The left images generated by AliTok-B (177M), and the right images are generated by AliTok-XL. (662M).
24


Reviews:
[{"review_id": "orfctfeTnO", "reviewer": "Reviewer_ubw6", "summary": "This paper proposes AliTok, a discret image tokenizer designed to match the unidirectional nature of autoregressive models. AliTok is trained in a two-stage manner, the first stage uses causal decoder to regularize the encoder, and the second stage uses bidirectional decoder to improve reconstruction. AliTok also employs prefix tokens to fix the \"first row\" issue in autoregressive models. AliTok achieves comparable or better performance on ImageNet dataset with a faster sampling speed.", "questions": "1. A system-level comparison of reconstruction performance is a must for this paper. Especially with codebook size / codebook dim / training epochs / number of parameters.\n2. In Tab. 2, the paper uses original codebases for all methods. Does this mean other methods do not use KV-Cache?\n3. In Tab. 2, the performance of GigaTok is attributed to its codebook size. How does AliTok perform with different vocabulary size?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The intuition of tailoring the tokenizer to suit the AR generation is well-established. Image as 2-d grid does not really have a naturally direction. Several existing method uses random order AR. AliTok solves this with 1d tokenizer and a causal decoder.\n2. The fix for \"first row\" using prefix tokens is smart. As stated, image does not really have direction, this makes the first row hard to predict. Using prefix row to encode global semantic information provides guidance for this with minimal cost (256 + 17 prefix).\n3. The experiments results demonstrate the superior generation performance with extensive ablations and visualization.", "weaknesses": "1. No system-level comparison of reconstruction performance. Despite the final goal for tokenizer is to enable better generation, a detailed comparison and analysis of reconstruction performance cannot be neglected. This is a huge missing in a paper focusing on tokenizer.\n2. Several experimental details are missing, especially for reconstruction. For example, what is the batch size / epochs used in two-stage training? What are the size of the tokenizer compared to other model? Without those details, it is hard for apples-to-apples comparison.\n3. Sampling speed is not really a vital contribution of AliTok. It naturally fits in autoregressive models which originates from LlamaGen.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zJpZfDTe6q", "reviewer": "Reviewer_u2Nd", "summary": "The paper proposes AliTok, an image tokenizer aimed at reducing the misalignment between conventional bidirectional visual tokenization and the unidirectional nature of autoregressive (AR) models.\nAliTok trains a bidirectional encoder under a causal decoder constraint, encouraging the latent tokens to exhibit forward-dependency so that an AR decoder can better model them. The authors further introduce prefix tokens for first-row reconstruction and a two-stage training scheme to restore image fidelity.\nOn ImageNet-256, AliTok enables a standard AR transformer to outperform diffusion models (gFID 1.28 vs 1.35) and sample about 10× faster.", "questions": "While raster-scan decoding is the traditional form of autoregressive image generation, it is widely recognized as suboptimal for visual data due to its rigid ordering and limited global context.  \nRecent works such as **MaskGIT**, **MAR** have successfully relaxed this constraint using masked or random-order prediction, enabling bidirectional context while maintaining autoregressive structure.\n\nGiven this progress, why does your method **intentionally return to a strict raster-scan decoding scheme** instead of adopting or extending these more flexible paradigms?  \nDoesn’t enforcing raster causality reintroduce the very inefficiencies and spatial myopia that modern AR methods have been trying to overcome?  \nHow can you justify this design choice as a step forward rather than a regression in autoregressive modeling?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Clear motivation and writing. The authors communicate the misalignment problem and causal-decoder idea clearly.\n\n- Comprehensive ablations. The study isolates each design component’s effect on AR accuracy and gFID.\n\n- Interpretability. Attention-map visualization nikens learn a causal bias.\n\n- Empirical competence. Experiments are reproducible, metrics are standard, and odel sizes.", "weaknesses": "- **Limited conceptual novelty:**  \n  The core idea—adding a causal constraint to the tokenizer—is elegant but incremental, extending existing causal-structure ideas (e.g., masked or random-order AR) rather than offering a fundamentally new paradigm.\n\n- **Order-specific and brittle design:**  \n  The method hard-codes raster order and needs ad-hoc fixes (prefix tokens, auxiliary loss) for top-row reconstruction. It is unclear if the alignment holds under different scan orders, resolutions, or tasks.\n\n- **Two-stage compromise undermines the claim:**  \n  The encoder is trained causally but later paired with a bidirectional decoder, partially reversing the intended unidirectional alignment.\n\n- **Narrow experiments:**  \n  Results are limited to ImageNet-256 class-conditional generation. No tests on higher resolutions, text-conditional, or multimodal setups—so generality is unproven.\n\n- **Marginal quantitative gains and unclear fairness:**  \n  The gFID improvement over diffusion (1.35 → 1.28) is small and possibly within evaluation noise; speedups rely on different kernel and caching settings without matched baselines.\n\n- **Tokenizer bottleneck:**  \n  The discrete tokenizer itself caps achievable FID (~1.15), limiting scalability unless codebook size or representation capacity increases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O2vcedhP8P", "reviewer": "Reviewer_5Mwz", "summary": "This paper tackles the misalignment between conventional bidirectional image tokenizers and unidirectional autoregressive (AR) models in image generation by introducing AliTok, an aligned tokenizer. AliTok uses a bidirectional encoder constrained by a causal decoder to produce semantically rich, forward-dependent token sequences, plus prefix tokens and a two-stage training process to boost reconstruction and generation performance. On ImageNet-256, AR models with AliTok excel: a 177M-parameter model achieves 1.44 gFID and 319.5 IS, while the 662M-parameter AliTok-XL hits 1.28 gFID  with 10× faster sampling.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is clearly written and well-structured, making it easy to read and understand.\n2. The work provides a compelling analysis of the fundamental conflict between conventional bidirectional tokenizers and unidirectional autoregressive (AR) models, and addresses it through a novel design that combines a bidirectional encoder with a causally constrained decoder to facilitate AR image generation.\n3. The proposed framework is thoroughly validated through extensive experiments, ablation studies, and well-motivated design choices, including the use of a causal decoder, prefix tokens with an auxiliary loss, and a two-stage tokenizer training strategy.\n4. The proposed AliTok approach achieves state-of-the-art performance on the ImageNet-256 benchmark, surpassing leading diffusion-based methods while offering significantly faster sampling speed.", "weaknesses": "1. The experimental evaluation is limited to ImageNet-256; results on higher-resolution datasets such as ImageNet-512 are missing, which leaves uncertainty about the scalability and robustness of the proposed method to larger image resolutions.\n2. The paper does not provide a detailed analysis or explanation for the reported 10× sampling speedup. My understanding is that the method introduces additional buffer tokens and prefix tokens, which could potentially incur higher computational cost compared to the baseline. A clear breakdown of the factors contributing to the acceleration would strengthen the claims.\n3. The approach may face challenges when scaling to higher resolutions, as the number of prefix tokens is likely to grow with image size (e.g., for ImageNet-512), which might impact efficiency and memory usage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OZVb2jCnVG", "reviewer": "Reviewer_yYv4", "summary": "This paper tackles an fundamental issue in visual autoregressive modeling: the mismatch between the bidirectional structure of visual tokenizers and the unidirectional nature of autoregressive models. The authors propose AliTok, a new tokenizer that introduces a causal decoder during training, enforcing the encoder to produce tokens with forward-dependency patterns more aligned with autoregressive modeling. A two-stage training strategy and prefix tokens are used to balance generation-friendliness and reconstruction fidelity.", "questions": "1. Can you visualize or quantify how the learned codebook differs from standard VQGAN/ViT-VQ tokens?\n2. How does AliTok affect token entropy compared to standard bidirectional tokenizers? A measurable drop would suggest over-regularization, the model becomes easier to train but less expressive.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation of the paper is clearly articulated and the proposed approach is effective.\n2. The paper is well-organized and clearly written. The framework is described in detail and the authors provide sufficient information about it.\n3. Instead of modifying the AR model, the authors redesign the tokenizer training objective to produce causally-aligned tokens. This “tokenizer alignment” perspective is novel and could generalize to other modalities.", "weaknesses": "1. AliTok forces tokens to be easier to predict, but that very constraint reduces representational richness.\n2. It seems that there is no guarantee that the learned codebook remains causally coherent after AR fine-tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)