Stabilize the Latent Space for Image Autoregressive Modeling: A Unified Perspective
Authors: Yongxin Zhu
Bocheng Li
Hang Zhang
Xin Li
Linli Xu
Lidong Bing
Venue: NeurIPS 2024 poster (2024)
Keywords: Latent space
Image sequential modeling
Image tokenizer
Primary area: generative_models

Abstract: Latent-based image generative models, such as Latent Diffusion Models (LDMs) and Mask Image Models (MIMs), have achieved notable success in image generation tasks. These models typically leverage reconstructive autoencoders like VQGAN or VAE to encode pixels into a more compact latent space and learn the data distribution in the latent space instead of directly from pixels. However, this practice raises a pertinent question: Is it truly the optimal choice? In response, we begin with an intriguing observation: despite sharing the same latent space, autoregressive models significantly lag behind LDMs and MIMs in image generation. This finding contrasts sharply with the field of NLP, where the autoregressive model GPT has established a commanding presence. To address this discrepancy, we introduce a unified perspective on the relationship between latent space and generative models, emphasizing the stability of latent space in image generative modeling. Furthermore, we propose a simple but effective discrete image tokenizer to stabilize the latent space for image generative modeling by applying K-Means on the latent features of self-supervised learning models. Experimental results show that image autoregressive modeling with our tokenizer (DiGIT) benefits both image understanding and image generation with the next token prediction principle, which is inherently straightforward for GPT models but challenging for other generative models. Remarkably, for the first time, a GPT-style autoregressive model for images outperforms LDMs, which also exhibits substantial improvement akin to GPT when scaling up model size. Our findings underscore the potential of an optimized latent space and the integration of discrete tokenization in advancing the capabilities of image generative models. The code is available at \url{https://github.com/DAMO-NLP-SG/DiGIT}.

Stabilize the Latent Space for Image Autoregressive Modeling: A Unified Perspective
Yongxin Zhu 1.3 Bocheng Li Hang Zhang', Xin Li, Linli Xu* Lidong Bing
School of Data Science, University of Science and Technology of China
Schooo of Computer Science and Technology, University of Science and Technology of China Sttate Key Laboratory of Cognitive Intelligence, *Zhejiang University
zyx201 10:00660ail..em wdu..uuuuuuuuuuuuuuuuuoouuuuuu.. edu. en, bcl @mail. www.ustcc..... edu. en haagchhagg__suuffoxaal1 com ixii4ever@@mail.. com, wanniixuuustc..m edu. en, binglidong@@mail.cc com
Abstract
Latent- based image generative models, such Latent Diffusion Models (LDMs) and Mask Image Models (MIMs), have achieved notable success in image gen- eration tasks. These models typically leverage reconstructive autoencoders like VQGAN or VAE to encode pixels into more compact latent space and learn the data distribution in the latent space instead of directly from pixels. However, this practice raises pertinent question: Is it truly the optimal choice? In response, we begin with an intriguing observation: despite sharing the same latent space, autoregressive models significantly lag behind LDMs and MIMs in image genera- tion. This finding contrasts sharply with the field of NLP, where the autoregressive model GPT has established commanding presence. To address this discrepancy, we introduce a unified perspective on the relationship between latent space and generative models, emphasizing the stability of latent space in image generative modeling. Furthermore, we propose a simple but effective discrete image tokenizer stabilize the latent space for image generative modeling by applying K-Means on the latent features self-supervised learning models. Experimental results show that image autoregressive modeling with our tokenizer (DiGIT) benefits both image understanding and image generation with the next token prediction principle, which is inherently straightforward for GPT models but challenging for other generative models. Remarkably, for the first time, GPT-style autore- gressive model for images outperforms LDMs. which also exhibits substantial improvement akin to GPT when scaling up model size. Our findings underscore the potential of an optimized latent space and the integration of discrete tokenization in advancing the capabilities of image generative models. The code is available at https://gi tht.b.....tu.../////g
NLP SCCIIIII
Introduction
In recent years, remarkable advancements have been achieved the field of image generation, principally propelled by the development of latent-based generative models, such Latent Diffusion Models (LDMs) [34] 30) and Mask Image Models (MIMs) 77 [6). By employing reconstructive autoencoders such as VQGAN [5] VAE 23 to compress images into manageable low dimen- sional latent space, these models can generate highly realistic and imaginative samples. Concurrently, in light of the transformative impact of autoregressive (AR) generative models, such as Large Lan- guage Models [1 32 5 27 in NLP, it becomes compelling to investigate the feasibility of similar paradigms images. Despite the advances in image autoregressive pre-training, exemplified by
"Correspondngg author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

the investigation regarding Model Performance: FID vs. Top-I Accuracy
Model Performance: I8 vs. FID
DINOV2
DGIT)-waamm
MAGVIE
54.5
DIGIT
82.0
HOOAAN
Fréchet Iriception Distance (FID)
McCa
L-NT-16
DeGAAN
Top- Acouracy (%)
iB0T
24.5
MAE
22.0
VIN-VII-NQGAN
10
LDM-
DIT-XU/2
AIM
ADM
69.5
67.0
VII-VQGAN
GAN
64.1
VIMI WEEVOGAN
VIM-IVAE
GIVT
Mark Modele
Models
63.0
VIM VOGAN
Aurocgreesive Models
IGPEL
16
RQ) kamee
MaaGGIT
LDM
ADM
N/
VOGAN
VOGAN
120
180
Frechet Inception Distance (FID) L
Inception Score (IS)
(n)
(b)
Figure 1: (a): Linear probe and class-unconditional generation performance of different methods trained and evaluated on ImageNet- IK. (b): Class-conditional generation performance of different methods ImageNet-Ik: The size of the bubbles indicates the number of parameters in the models. DiGIT achieves SOTA performance in linear probing and establishes a new SOTA in image generation within a single model.
models such as iGPT 88, AIM 14] and GIVT [66 in the pixel space. VIM and LVM in the latent space, their performance are still inferior to the leading models [4 12 in image generation, or self-supervised learning models [88
T0. 28
in image understanding tasks.
An intriguing observation emerges regarding the presumed optimality of current practices in latent space: as illustrated in Figure [b), though sharing the same latent space, autoregressive models significantly lag behind LDMs and MIMs in the image generation task. This discrepancy prompts reevaluating our understanding of latent spaces and their interaction with generative models, sug- gesting unexplored avenues worth investigating. A central premise in learning theory 20 is that the latent distribution should retain as much critical information of the data distribution as possible, akin to compression goal. This leads to a common misconception that an optimal latent space for re- construction equates to optimal generative performance. Nevertheless, the reconstruction and generation ability with the popular VQGAN model [42] it is observed that the generation FID will deteriorate when the reconstruction FID becomes lower (where lower value indicates better performance), challenging the above assumption.
To address these intriguing discrepancies. we introduce a unified perspective on the relationship between latent spaces and generative models to analyze what constitutes an optimal latent space for generative models. Our findings reveal that, beyond the compression techniques employed by prevalent latent generative models, an optimal latent space should also aim to minimize the distance between distributions under the condition of incorporating generative model, which is an aspect often overlooked. We critically assess prevalent methodologies and reveal that the stability of latent space is important for generative models. We argue that the reason why autoregressive models underperform iterative models such as LDMs and MIMs is that iterative models can correct errors brought by the instability of latent space.
Drawing from this insight, we propose a straightforward method to stabilize the existing latent space methods for image autoregressive generative models. Unlike conventional autoencoder-style ap- proaches, our approach disentangles the concurrent training of encoders and decoders and commences with encoder-only training through discriminative self-supervised model 28]. This phase does not necessitate a decoder for pixel reconstruction, enabling the encoder to discern the intrinsic and distin- guishable features present within the data. Subsequently, a separate decoder of the autoencoder [15) trained and tasked solely with the pixel reconstruction process, conditioned on the features identified by the encoder. By focusing initially on the encoder's capability to extract meaningful data features independently of pixel reconstruction, lay foundation for more stable and feature-rich rich latent

---

space. The subsequent independent training phase of the decoder ensures that these captured features can be accurately translated back to pixels.
In support of the autoregressive generative model, which requires discrete tokens for next token prediction, we employ a strategy inspired by VQGAN [5] discretize the encoder's latent feature space with the K-Means clustering algorithm. With this novel image tokenizer induced from the stabilized latent space, the performance of autoregressive generative models in images is enhanced significantly in both image understanding and image generation tasks. We refer to this approach as call Discriminative Generative Image Transformer (DiGIT). Notably, when scaling up the model size, substantial improvements can be achieved. To the best of our knowledge, this is the first evidence that image autoregressive generative models behave analogously to GPT. essence, this work endeavors to redefine the boundaries of what is possible in image autoregressive modeling through unified perspective of latent space.
In summary, our contributions to the field of image generative models include:
We introduce a unified perspective on the relationship between latent space and generative models, emphasizing the stability of latent space in image generative modeling.
We propose novel method to stabilize latent space by disentangling the encoder and decoder training processes. Furthermore, simple yet effective discrete image tokenizer is proposed to improve the image autoregressive generative model's performance under the philosophy of next token prediction.
The experimental results show that the image autoregressive modeling with our tokenizer leads to SOTA performance in image understanding and generation, with further improve- ments witnessed when scaling up the model size.
Problem Formulation
In section we formalize the latent space requirements for generative models and categorize current latent- based generative models. Furthermore, in section 2.2 we analyze the stability of different induced latent spaces and propose to stabilize the latent space for autoregressive generative models instead of stabilizing the generation process with an iterative decoding strategy like LDMs.
2.1 Latent Space for Generative Models
Drawing inspiration from the complexity perspective of latent space induced from autoencoders in Hu et al. [2I]. we delve into the latent space for generative models. Generative models aim at learning distribution to approximate the data distribution Px. Formerly, given a tractable prior distribution Pz and distance metric -::-) between distributions, the purpose of generative model g E G is to minimize the distance between data distribution Px and the distribution generated by g(Z):
min D(P D(Pa)z),PP Px).
(1)
For example, GANs [16] employ the Gaussian distribution their prior and utilize discriminator network as the distance metric. However, the optimal strategy for data representation in generative models is still under-explored. Recent studies on latent diffusion models (34] have identified that direct learning the pixel space of images is suboptimal. They propose to learn latent space
induced by constrained autoencoder model such VAE 23 or VQGAN [15], which has been demonstrated to improve the perceptual quality.
A simple method to construct the latent space is using an encoder f F :R IR - Rdd to map raw data samples x E ddd into a latent space f(X) of dimension d.. Consequently, the goal of latent-based generative models is to learn the distribution as per the following formula:
min D(P Pr())),
where Pς(x) denotes the data distribution in the latent space induced by the encoder f. Despite these advances, determining the optimal latent space configuration for generative models remains unresolved.
3

---

Distance between distributions in different spaces. Given the ultimate goal of the generative model to produce image pixels, a decoder model h E H R4. + R. paired with the encoder model f, is necessary to convert latent representations back into pixels. We define generalized distance between different spaces associated with a decoder h E H as:
Dd (P((x))) Px)= == inf D(PR(S(X))) Px). hEH
By employing D to compare distributions across different spaces, we can define the ideal latent space f(X) as the one that minimizes (Pxx), Px). This implies selecting a latent representation that minimizes the empirical objective [(x)))(x)) with the same family H. of decoder models. Such latent configuration depends both the data and the decoder training methodology. We can formalize with an autoencoder framework by looking at the encoder and decoder together. and the primary goal becomes the reconstruction of the input sample formulated min [f (f(((())))))))))())) z). Once a generative model ! successfully approximates the latent distribution Pς(x), the generated sample be efficiently transformed back into pixels using the decoder h.
Similarly, we can define the distance between distributions in the latent space f(X) E Rds and data space X Id conditioned on the latent-based generative model g g.
(P Px) inf D(Ps(2), gEg
(4)
which aims to minimize the empirical objective with the generative model family G.
Optimal Latent Space for Generative Models. Now that have characterized the ideal latent distribution given the family of generative models and the data, the next step is to determine how to find the optimal latent space. At the population level, the objective for the latent-based generative models with a decoder is:
min D(Ph(o(Z)) Px)= min D(PN(S())) Px) + ((P bEH.gE9
(5)
g
where the first term focuses on optimizing the decoder to enhance the reconstruction quality and the second one is directed towards optimizing the generative model to more accurately approximate the latent space distribution. Inspired by this observation, we can characterize the optimal latent distribu- tion Pf(x) for given Px from the perspective of minimizing the distance between distributions in different spaces by defining f* as
argmin Diatent (P (xxxx)xx) = fx) = argmin (Px(), + (P JEF
(6)
JEF
Notice that (x)(x) depends on multiple factors, including Px, the distance metric D, and the constructed model families G and H. By integrating D and DHH we arrive at a comprehensive measure of the distance between the distribution in the latent space and the original data distribution as Diatent (())). Px). The second term, (P(((X()) Px). is exactly the objective of reconstructive autoencoders. Ültimately, from examining the learning objective pertinent to identifying the optimal latent space for generative models, it becomes evident that:
A reconstructive autoencoder does not necessarily establish an advantageous latent space for generative models.
Two Pathways of the Latent Space Construction. Although we theoretically analyze the opti- mization of the optimal latent space for generative models, it is challenging to implement in practice because optimizing (f, g. hi) simultaneously computationally complex. A practical solution is to optimize D and Dd separately, allowing for tractable training.
When (P((() Px) is not a target for optimization, it implies that the optimization of the decoder within the generative model framework is bypassed. The encoder independently forms a latent space, aligning with self-supervised learning (SSL) strategies aimed at un- covering lower-dimensional features from unlabeled data. However, learning the generative models in the latent space induced by SSL models remains relatively unexplored.
On the other hand, when D (P Px) remains fixed, the primary objective becomes optimizing the encoder and decoder to effectively learn and represent the latent space,

---

34 and where Dlatent (P(()), (ix)).Px Px) degrades into an autoencoder learning objective. This approach is evident in recent seeeent-ppae-rrienntee generative models, such as LDM 30), VQGAN [5), and MaskGIT (MIM) 77 all of which concentrate on learning g in the latent space with the encoder and decoder frozen.
While latent generative models such as LDM 34 MaskGIT 77, and VOGAN [5] share the same latent space induced by reconstructive autoencoder [S]]] to minimize (P Px), their image generation performances differ significantly. the next section, we analyze the reason behind from the perspective of the latent space.
2.2 Stability of the Latent space
We first describe the decoding mechanism of various latent generative models. Both LDM MaskGIT 7 can be depicted as an iterative sampling procedure given by:
p(*")
II T(ax (=)))
The intermediate states x² in LDMs represent images infused with Gaussian noise of varying variance, whereas for MaskGIT, they denote discretely tokenized images augmented with masks. In contrast, the autoregressive framework of VQGAN (AR) described as:
N
p(x) = p(x1,...... NN) = II ((aylaes))
(8)
where x; represents the i-th patch in the image sequence. Notice that xf denotes the entire image while Ty means the local patch tokens. In the autoregressive decoding process, if the previously sampled tokens are incorrect, the accuracy of subsequent tokens would be affected due to error aggregation. In contrast, the iterative decoding approach allows for the revision of earlier misjudged tokens. When the latent space unstable that small perturbation in pixels can change the latent distribution significantly, the iterative decoding mechanism employed by LDM and MaskGIT can alleviate the error aggregation problem by allowing for revision of earlier misjudged tokens while autoregressive models cannot. Consequently, stable latent space is required to reduce errors introduced in the generation process of autoregressive models.
This principle forms the foundation of our methodology for developing metric to evaluate latent spaces with an emphasis on the stability of the latent representations. We examine two primary types of latent spaces: (1) autoencoder induced by minimizing D* Px) and (2) self-supervised learning (SSL) model induced by minimizing D$(P (P)(x): Px). Px). By analyzing network parameters of these models in a linear regime, we derive the following propositions.
Proposition 2.1. The latent space spanned by a linear autoencoder is congruent with that spanned by the principal component loading vectors derived in Principal Component Analysis (PCA). Fur- thermore, the principal component loading vectors can be elucidated from the autoencoder's 's weights. Proposition 2.2. The discriminative self-supervieed model learns to separate data distributions in the latent space as Linear Discriminant Analysis in principle.
Motivated by these theoretical insights, we introduce metric to assess the stability of the latent space induced by different encoder models. To exemplify these concepts, we refer to an example consisting of two Gaussian distributions in wwo-dimensional space, as depicted in Figure 6(a). The results attained from applying the PCA and LDA algorithms are visually depicted in Figure 6(b) and (c) respectively. The distribution embedded by the LDA model exhibits greater separability than that by the PCA model. To quantitatively evaluate stability, we add Gaussian noise of different variances to the original 2D data and subsequently train a linear classifier on the latent space. As Tigure/olal illustrates, the accuracy of the LDA model consistently surpasses that of PCA.
2D and D can achieve zero simultancously. For example, nomenon in the VAE literature, the latent space f f(X ) is a tractable Gaussian distribution and D (Px(), can be zero by simply setting the generative models as Gaussian sampler. If the decoder is strong enough and directly generates samples without conditioning on the encoder output, D (P)(()))) Px) can be well.
well known posterior collapse phe-
5

---

Table I: The stability of latent spaces induced from VQ Token and Discriminative Token (introduced in Section assessed across different Signal-to-Noise Ratio (SNR) levels to evaluate performance under varying signal and noise conditions.
SNR
30
25
20
10
0.01
VQ Token change +
0.187
0.317
0.487
0.663
0.805
0.901
0.948
0.956
Dise Token change J
0.114
0.178
0.260
0.355
0.457
0.570
0.687
0.721
VQ Token cos-sim
0.972
0.949
0.910
0.853
0.777
0.682
0.594
0.571
Dise Token cos-sim
0.975
0.960
0.940
0.916
0.888
0.855
0.816
0.803
Tratofomne Blakk
Discriminative
Tranffomr Blakk
SSL Model
Tokenize
Tntofonec Black
Image Patches
Visual Features
K-Means
Discriminative Tokens
Visual Features
Trainable
Figure 2: The architecture of DIGIT.
To evaluate the stability of latent space induced from autoencoders and SSL models, we add Gaussian noise to image pixels and then feed the noisy images to VQGAN encoder and an SSL encoder DINOv2 228] This experiment aims to examine the resilience of the latent spaces induced by these encoders to such disturbances. We measure the rate of change in discrete tokens, specifically VQ tokens for the latent space induced from the VQGAN encoder and discriminative tokens for the latent space induced from the SSL model, and the cosine similarity in conjunction with the strength of the noise introduced. The experimental results in Table T demonstrate that the latent space induced from the SSL model exhibits heightened stability compared to that derived from the VQGAN autoencoder. Therefore, we propose to replace the unstable latent space induced by the reconstructive model with stable latent space induced by the discriminative self-supervised model for autoregressive models.
Stabilize the Latent Space with Self-supervised Learning Model
In this section, we present a simple but effective image tokenizer that discretizes the feature rep- resentations of discriminative SSL models to form discrete tokens for autoregressive models. The architecture of our model is illustrated in Figure
Discrete Image Discriminative Tokenizer Drawing inspiration from the VQGAN tokenizer [15 which employs an implicit K-Means clustering algorithm within the latent space to generate discrete tokens for autoregressive modeling, we propose straightforward approach perform K-Means clustering on the feature space of discriminative SSL models to obtain discrete tokens. To process a given dataset, our initial step involves gathering the features of image patches, akin the hidden states produced by SSL models. Then we employ a clustering algorithm to group these patches, resulting a collection of K clustering centers. These centers constitute the codebook for the discrete tokenizer. To determine the discrete tokens for an image patch at inference, we identify its nearest neighbor in the codebook, which then designated as the discrete token for the respective patch.
Image Autoregressive Modeling After converting images into discrete tokens with the discrimina- tive tokenizer, we treat each image sequence by flattening the discrete tokens from images into ID sequence raster order. We train a causal Transformer [38] model with the next token prediction objective, which is the same as the standard approach for language models.

---

Table 2: Linear-pa Ainear-probe accuracy of image autoregressive generative models on ImageNet
Methods
# Tokens
Features
# Params
Top-1 Acc.1
iGPT-L.
32
1536
1362M
60.3
iGPT-XL 8
64 x 64
3072
6801M
68.7
VIM+ 41
32 x 32
1024
650M
61.8
VIM+dVAE 41
32 32
1024
650M
63.8
GIVT [36]
16
1024
304M
65.1
VIM-VIT-VOGAN
32 x
650M
65.1
VIM-VIT-VGGAN
41
32 x 32
2048
1697M
73.2
AIM [+]
16 x 16
1536
0.6B
70.5
DIGIT (Ours)
16 16
1024
219M
71.7
DIGIT (Ours)
16
1536
732M
80.3
4 Experiments
4.1
Implementation Details
We take the discriminative SSL model DINOv2 (28) as the encoder for all experiments. The K-Means
model trained on the randomly selected 10% subset of the ImageNet [T]] training set. use autoregressive model with the same architecture as GPT-2 [32]. We train the DiGIT models with the base and large sizes. The vocabulary size the tokenizer for the base is 8192 and 16000 for the large size. More implementation details and
are provided in Appendix(A.3 A.3
4.2 Image Understanding
The GPT model is famous for learning semantic features by a generative training objective of next token prediction. We compare the image understanding ability of different image autoregressive models with linear-probe described in iGPT 88. We train linear classifier on top of the frozen features average from each layer on the ImageNet training set. We report the Top-1 accuracy compared with other image autoregressive models in Table 2 Remarkably, with only 219M parameters, DiGIT achieves Top-1 accuracy of 71.7%, surpassing both iGPT and VIM-Base, which have greater number of parameters and operate at more visual tokens. Despite representing images with smaller token grid size (16 x 16 opposed to 32 x 32), DiGIT still delivers superior top-1 accuracy, demonstrating the effectiveness our tokenizer. Morcover, when we scale DiGIT's parameters from 219M 732M, the Top-1 accuracy shows additional increase of 8.6% and reaches 80% for the first time. The improvement indicates that DiGIT with the proposed discriminative tokenizer has the potential for the development of large vision models.
4.3 Image Generation
Since the SSL models not have a paired decoder to recover pixels from latent space, the generative models trained with our discriminative tokenizer require auxiliary image decoder to render pixels. The discriminative tokenizer can be seamlessly integrated with any existing image generative models trained with tokenizer induced from reconstructive autoencoder. In our experiment, train an autoregressive model VQGAN, and an MIM model MaskGIT as pixel decoder respectively. The results presented in Table 3 and Table 4 The autoregressive model equipped with discriminative tokenizer achieves the SOTA performance with FID reaching 3 for the first time. Furthermore, the performance significantly improves as the model size increases. demonstrating the potential of a large vision model with next token prediction. Interestingly, when utilizing the DIGIT as the conditioning factor, the performance of both the autoregressive and MaskGIT decoders becomes close (4.6 and 4.79). This observation suggests that stabilizing the latent space produces effects analogous to the iterative stabilization decoding mechanism.
4.4 Ablation Study
We conduct the ablation study to present a comprehensive analysis of the proposed discriminative tokenizer in image generation and understanding. The results are illustrated in Table a[a and

---

Table 3: Class-und cnisssuncondiiionll image generation on ImageNet with resolution 256 256. DiGIT + VQ indicates that we utilize golden discriminative tokens alongside VQ generated by autoregressive models.
Type
Methods
#Param
#Epoch
FID
ISt
GAN
BigGAN
70M
38.6
24.70
Diff.
LDM
34
395M
39.1
22.83
Diff.
ADM
12
554M
26.2
39.70
MIM
MAGE
26
200M
1600
11.1
81.17
MIM
MAGR
26
463M
1600
9.10
105.1
MIM
MaskGIT 7
227M
300
20.7
42.08
MIM
DIGIT (+MaskGIT)
219M
200
9.04
75.04
AR
VQGAN [5
214M
200
24.38
30.93
AR
GIVT 36
304M
500
17.70
AR
GIVT 36
1.67B
500
11.02
AR
DIGIT (+VQGAN)
219M
400
9.13
73.85
AR
DIGIT (+VQGAN)
732M
200
4.59
141.29
validation data
DiGIT + VQ
1.92
184.40
validation data
VQ only
1.67
175.56
Table 4: Class-conditional image generation on ImageNet with resolution 256 x 256. f denotes the model is trained with classifier-free guidance while all the other models not.
Type
Methods
#Param
#Epoch
FID
IS
GAN
BigGAN
160M
6.95
198.2
Diff.
ADM
554M
10.94
DifI.
LDM-4
400M
10.56
103.5
Diff.
DFT-XL/2
675M
9.62
121.50
Diff.
L-DīT-7B
7B
6.09
153.32
MIM
Contextual
RQ-Trans
371M
300
5.45
172.6
MIM+AR
VAR
310M
4.64
MIM+AR
VAR
310M
200
3.601
257.51
MIM+AR
VAR
35
600M
250
2.951
306.1
MIM
MAGVIT-v2 42
307M
1080
3.65
200.5
AR
VQVAE-2
33
13.5B
31.11
45
AR
RQ-Trans
24
480M
15.72
86.8
AR
RQ-Trans
24
3.8B
7.55
134.0
AR
ViTVQGAN
41
650M
360
11.20
97.2
AR
ViTVQGAN
41
1.7B
360
5.3
149.9
AR
GIVT 33
304M
500
5.67
AR
GIVT 36
1.67B
500
3.46
MIM
MaskGIT f
227M
300
6.18
182.
MIM
DIGIT (+MaskGIT)
219M
200
4.62
146.19
VOGAN 15
227M
300
18.65
80.4
AR
DIGIT (+VQAAN)
219M
400
4.79
142.87
DIGIT (+VQGAN)
732M
200
3,39
205.96
validation
DIGIT + VQ
1.92
184.40
validation
VQ only
1.67
175.56
Figure Bb. For image generation tasks, we take the autoregressive model trained with the VQGAN tokenizer as the baseline. Introducing discriminative tokens leads to a significant improvement, reducing FID 9.66 and increasing IS to 69.15, underscoring the effectiveness of stabilizing latent space for autoregressive models. Further extending the training duration to 400 epochs yielded additional improvements of 0.53. A substantial advancement is observed when scaling up the model size to 732M, resulting in FID dropping dramatically to 4.59 and IS more than doubling to 141.29. This indicates that increasing the model's capacity significantly enhances ability to model complex relationships within the data, which is similar phenomenon in GPT models. Overall, the study highlights the latent space stabilization and the potential of large-scale training of autoregressive modeling in images with our discriminative tokenizer.

---

72.5
70.0-
FID
1ST
67.5
65.0
VQ Token
24.38
30.93
62.5
60.0
K- 8192
Discriminative Token
9.66
69.15
2 57.5
Means 4096
Longer Training (400 epoch)
9.13
73.85
55.0
K-Maans 1024
Scale up (732M)
4.59
141.29
12 14
Layer Index
(b)
Ablation study of DIGIT. (a) The comparison of tokenizer, training steps, and model size in the image generation task. (b) Linear- probe accuracy from different layers the pre-trained DiGIT-base with different number of K-Means clusters.
Figure 3:
For the image understanding task, investigate the effect of K-Means clusters and features learned in the different layers of DiGIT. We can see that increasing the cluster number can further improve the accuracy of the linear probe, which means the image autoregressive model can benefit from larger vocabulary. The linear probe accuracy increases quickly from the first transformer block, reaches its peak the middle layers, and finally decreases little bit for the last few blocks. This observation connects the image autoregressive model to the text language model where the semantic information learned in the middle layers of the transformer.
4.5 Comparison of Discrete Tokenizers
We conduct an experiment to investigate the effect of different SSL models on latent space. We generally categorize the SSL models into two types according to their pre-training objectives: (I) Global level (MoCo) and patch level (MAF,iBOT), (2) reconstructive (MAE) and discriminative (MoCo, iBOT). At the global level, the loss function is computed using an aggregate output such as [CLS) token or mean pooling. In contrast, patch-level models involve patches directly in loss computation. Reconstructive models, such as MAE, aim recover image pixels in manner akin to autoencoders, while discriminative models are optimized to learn the distinguishable features. As demonstrated in Table 4 a), the discriminative objective plays a pivotal role in image generation in that it can stabilize the latent space. Furthermore, because generative models need to predict patches, the inclusion of a patch-level loss function can enhance performance.
To assess the stability of latent space induced by our discriminative tokenizer and reconstructive tok- enizer. We pre-train two auto-regressive generative models the ImageNet dataset [11], employing the proposed discriminative tokenizer and VQGAN tokenizer respectively. We provide each model with the upper half of the target image conditional prompt for generation, challenging them to complete the lower half of the image. stable latent space should be able to help the autoregressive model generate the lower half more robustly, maintaining thematic and aesthetic coherence. As shown in Figure bb), the FID decreases for both models when given a longer prefix context. However, when the prefix length is reduced from 75% to only 12.5% of the image, the model trained with the VQGAN tokenizer encounters difficulties in producing images that adhere to the specified prompt. In contrast, the model utilizing the discriminative tokenizer effectively continues produce congruent visual tokens, maintaining low FID scores even with significantly truncated prefix.
Related Work
The image tokenizer [37 33 15 is essential in converting pixels into discrete tokens for autoregressive generative modeling. VQVAE (37 first proposes to assign latent features learned by an encoder to the nearest entry the learnable codebook embeddings. followed by
Image Tokenizer
decoder to reconstruct the original image pixels. VQGAN [15] further incorporates adversarial loss and perceptual loss improve the image synthesis quality. RQ-Transformer (24) extends the
single-layer quantizer to the multi-layer residual quantizer to augment the visual tokenizer's 's ability to
capture fine-grained details. ViT-VQGAN 41 incorporates the modern Vision-Transtone Transformer [13
into VQGAN to enhance the reconstruction quality. MAGVIT-v2 [2] substitute the online update codebook VQGAN with lookup-free quantizer to enable a larger vocabulary for generative language models.

---

20.0
SSL
Type
FID4
1St
Acc@ LP
Acc@OLi
17.5
15.0
VOCAN Takenizer
MAE
P+R
45.51
18.39
31.40
75.8
12.5
10.0
MoCo
G+D
20.38
45.02
59.22
76.7
iBOT
P+D
16.81
57.88
61.10
76.0
2.5
VOGAN
24.38
30.93
Prefix Length %
60
10
(a)
(b)
Figure (a): The comparison of tokenizers induced from different SSL models. is obtained by linear probing on the autoregressive model (model size of 39M for 100 epochs) trained with tokenizers. A OL the linear probe score of the SSL model. "P": patch, "D": discriminative. "R": reconstructive. (b): Generation quality curves in FID on ImageNet 256 x 256 valid set when scaling the prefix length with discriminative tokenizer and reconstructive VQGAN tokenizer. Both are autoregressive models with 219M parameters.
Image Autoregressive Modeling Inspired by the success of autoregressive Transformer 38 in text generation tasks, there have been several efforts to replicate image generation tasks. One of the pioneering works is iGPT [), which pre-trains autoregressive model pixels with the same architecture as GPT2 32] achieving promising results in unsupervised visual representation learning.
LVM [I] proposes large-scale vision dataset composed of images and videos, based which large vision model is trained. Empirical observations indicate that the model scales effectively across various tasks with in-context learning [40]. Similarly, AIM [14 follows ViT [13] and represents
images in the patch. It is observed that the performance of image recognition continues to increase as
the model size scales up. VAR [35] proposes next-scale prediction generate images from coarse to fine in hybrid of autoregressive and nonautoregressive manner.
Self-supervised Learning Models Self-supervised learning (SSL) 9 plays important role in learning fundamental visual representations for downstream tasks. Among them, SimCLR (9]. MoCo
p T8 I0] compute losses the image level through CCSS) token aggregation pooling
operations with contrastive learning. iBOT-style models 43 28 extend the loss to the patch
level, achieving improved performance in dense prediction tasks. BEiT uses VQGAN tokenized sequences as the training target. MAE [19] randomly masks some patches of images and reconstructs the pixels with unmasked patches the condition.
6 Conclusion
In this paper, we make an exploration in the latent space for generative modeling. We introduce unified perspective on the relationship between latent space and generative models, emphasizing the stability of latent space in image generative modeling, Subsequently, we propose a simple but effective discriminative image tokenizer, followed by an image autoregressive generative model DIGIT. Empirical results indicate that our tokenizer achieves superior performance across both image understanding and image generation tasks. Notably, when DiGIT is scaled up in model size, exhibits even greater enhancements, indicating the potential for the development of large vision models. Our findings challenge the conventional wisdom that proficiency in reconstruction equates to an effective latent space for auto-regressive generation. Through this work, we aim to rekindle interest in the generative pre-training of image auto-regressive models and encourage a reevaluation of the fundamental components that define latent space for generative models.
Acknowledgments and Disclosure of Funding
This research was supported by the National Key Research and Development Program of China (Grant No. 2022YFB3103100) the National Natural Science Foundation of China (Grant No. 62276245).
References
[1] Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan Yuille, Trevor Darrell, Jitendra Malik, and Alexei A. Efros. Sequential modeling enables scalable learning for large
10

---

vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22861-22872, June 2024.
[2] Pierre Baldi and Kurt Hornik. Neural networks and principal component analysis: Learning from examples without local minima. Neural Networks, ((1):53-58, 1989. ISSN 0893-6080. doi:
URL https::/:// btttpp::/www.ssiiie www.sccienceeirecttttt
[3] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transformers. In International Conference on Learning Representations, 2022. URLptttps
//openreview.o h-t. Bh2sp-942
[4] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019, URL
htttps://peenreriew.nttt wtt..ntee/ooommddddttteesas
[5] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
[6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self- superviedd vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9650-9666 October 2021.
[7] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11315-118 11325, June 2022.
[8] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Pro- ceedings of Machine Learning Research, pages 1691-1703, PMLR, 13-18 Jul 2020. URL https: http:e//prroceeings mlr ppwss//119/ chwph//ccceecss.cm html
[9] Ting Chen, Simon Korblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Pro- ceedings of Machine Learning Research, pages 1597-1607 PMLR, 13-18 Jul 2020. URL Ittps: httpss://prroeedingss.o
.mm.diinnnnmmm.......mm.mm.m mlr
chtne0jeennjcce .html
[10] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint wxxv:2003300229, 2020.
[11] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hicrarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009. doi: 10. 10.1109/CVPR. 200955000848.....
[12] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 9780-879A, Curran Associates, Inc., 2021. URL httpttpppppp/ppeeeeeneeeeeeeeeee 0011:11118880021aeh////000066063
cc/paper http/apprrfffee/eeeeaeeapepee files/paper/
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021. URL. https:///penreeiee..o wtt.://iww.....oooooooocooeooooa
11

---

[14] Alaaeldin El-Nouby, Michal Klein, Shuangfei Zhai, Miguel Angel Bautista, Vaishaal Shankar, Alexander T Toshev, Joshua M. Susskind, and Armand Joulin. Scalable pre-training of large autoregressive image models. In Forty-first International Conference on Machine Learning, 2024. URL
ret//orrmmiid=<2X0EEE77
[15] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR). pages 12873-1888,, June 2021.
[16] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sher- jil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, 2014. URL http::::s///prroeennngggg htwp://rooeediinnne.eeeirrpppppp wwtpapcc_peeeeeeeeeee filef wallespaapee patee////ppr/21///i2eeee .pff
[17] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent- new approach to self-supervised learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 21271-21284 Curran Associates, Inc., 2020. URL http::ts neurips cc/ pdf
paper_ httpa///aapppeeoo//eeeeaaaaaaaaa
[18) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[19] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16000-16009, June 2022.
[20] G. E. Hinton and R. R. Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313 2006. doi: 10.1 /2t.////eecccc URL https
/ /www. www.:sciencc..cm www.:/ciennee.orr/dooooooooo httgg//oi//// aba/aaaaaa110// 11t.:...1/ssiiieeeeeeeo iocienne..127644 1127647
[21] Tianyang Hu, Fei Chen, Haonan Wang, Jiawei Li, Wenjia Wang, Jiacheng Sun, and Zhen- guo Li. Complexity matters: Rethinking the latent space for generative modeling. In A. Oh, Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neu- ral Information Processing Systems, volume 36, pages 29558-29579. Curran Associates, Inc., 2023. URL http:tttt//oocoeeaeiiiiii wwwp:eeddddiinnnneeneeeeeeeeeeee httpc/ppprrffeeeeeeeeeeeeeeeseee filefflllppppe httpaeeffleesspppeeeeeeeee ftt.ees/ppppee222///11... 05802330762577461115330008888883
Ceafrr-ooffeeeene pdf
[22] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6000 2014.
[23] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 14, 2013.
[24] Doyup Lee, Chiheon Kim, Sachoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11523-11532. June 2022.
[25] Doyup Lee, Chiheon Kim, Sachoon Kim, Minsu Cho, and WOOK SHIN HAN. Draft- and-revise: Effective image generation with contextual tr-traasformmr. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho. and A. Oh, editors, Advances in Neu- ral Information Processing Systems, volume 35, pages 30127-30138. Curran Associates, Inc., 2022. URL htw.httppss/prroeeddngg. www.wnn......e wwcpppppe_reeeeeee files/ wtt..e///aaprr/22///112221 227630000/266333bb544aa1ld623222 Poper .df
[26) Tianhong Li, Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. Mage: Masked generative encoder to unify representation learning and image synthesis. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2142-2152, June 2023.
12

---

[27] OpenAl. Gpt-4 technical report, 2023.
[28]
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL httptt://ppeeeeererreee wttp://wwwweeeooooidddedeeiaidd a68SUt66FFF
[29] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
[30] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4195-4205, October 2023.
[31] Alec Radford. Improving language understanding by generative pre-training. 2018.
[32] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9 2019.
[33] Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Bir -uce E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https:: ptttoee//proceedingg htw.:eeeeddinnnnn.nueipeeee http//aper_flle//pppppeeeeeeeepp
[34] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, June 2022.
[35] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. 2024.
[36] Michael Tschannen, Cian Eastwood, and Fabian Mentzer. Givt: Generative infinite- vocabulary transformers. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol, editors, Computer Vision - ECCV 2024, pages 292-309, Cham, 2025. Springer Nature Switzerland. ISBN 978-3-001--72998-0.
[37] Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learning. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30, Curran Associates, Inc., 2017. URL https h/pooceepp///rroeeeiiii...ms neurips. wwtppppp_pfiiiieeeeeee files/paper/ pdf
[38] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, f. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, ed-
U. Von Luxburg, itors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL http/atttt///prrceeinngge wwwp://ooooeeiionne.eeeeeeeeepep neurips. ct/ fiaape_filees/pperr/00////// #
Paper .dd
[39] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Pro- ceedings of Machine Learning Research, pages 9929-9939. PMLR, 13-18 Jul 2020. URI. https://procesee
mlr. wwwppp/eseveee.... watees/v119//e watsss//v11//aagg000 html
13

---

[40] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoe V Le, and Denny Zhou. Shain-of-thougee prompting elicits reasoning large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances Neural Information Processing Systems, volume 35, pages 24824-24837, Curran Associates, Inc., 2022. URL htkk.ttttts///prcceediigg neur ips cc/ palpp_apppleeeeeere /aper// 1o2p2/aae666005554ccc/000///1660 Cow.aperrerrooeeeeeeeeeeeeee pdf
[41] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang. James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved VQGAN. International Conference on Learning Representations, 2022. URL
https://openreviexx net.fforumndddpdlyy33/33
[42] Lijun Yu, Jose Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng. Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, Boqing Gong, Ming-Hsuan Yang. Irfan Essa, David Ross, and Lu Jiang. Language model beats diffusion tokenizer is key to visual generation. In The Twelfth International Conference on Learning Representations, 2024. URL /ttp: www.::nenefooooee iaaimmiidddgggii
het. id=gzqrANCFF4
[43] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image BERT pre-training with online tokenizer. In International Conference on Learning Representations, 2022. URL
14

---

A Appendix
A.1 Limitation and Broader Impact
Our proposed discriminative tokenizer exhibits remarkable capabilities in both the image under- standing and the image generation tasks, which is challenging for single model. However, the discriminative tokenizer induced by the SSL model cannot directly render pixels. Consequently, we need to train another decoder model to convert tokens from discriminative tokenizers to VQ- GAN tokens. The potential for a direct generation of RGB pixels from the tokens produced by the discriminative tokenizer remains an uncharted avenue, which we leave for future work.
Image generative models possess a dichotomous nature, particularly within the realm of visual media. On the positive side, they foster myriad of creative endeavors, and methodologies aim minimize the costs of training and inference, holding the potential to broaden access and democratize the use of this technology. On the negative side, the simplicity with which manipulated content can be crafted and propagated raises serious concerns. This includes the proliferation of misinformation and spam. Furthermore, generative models may inadvertently unveil aspects of their training data, particularly troubling issue when that data includes sensitive or personal information collected without explicit consent.
A.2 Proof of claims in Section 2
Proposition A.1. Consider the optimization problem for X E R":
min
|X W2WiX
WIERm W2
which is a linear autoencoder. W2 is minimizer of the problem if and only if its column space is spanned by the first m loading vectors of X.
Proof. First, we derive the condition for the optimal W, in the context of this optimization problem. Setting the gradient of the objective function with respect to Wi to zero leads to W1 being the left Moore-Penrose pseudoinverse of W₂ (2 Similarly, setting the gradient with respect to W₂ to zero would identify W2 as the right pseudoinverse of W1:
W1 w (W2 W
(10)
This finding indicates that the optimization can be simplified to focus single matrix (either W1 or W2), hence removing the redundancy in the parameters:
min W2ERnxm
|| W2W
(11)
The term W2W, = W2 WWWWTW " is recognized as the matrix form of the orthogonal projection operator onto the column space of W2. This property holds true even when the column vectors of W2 are not orthonormal.
By performing QR decomposition on W2, W2 QR where Q is an orthogonal matrix (Q /) and R is an upper triangular matrix, we effectively transform the problem into optimizing over orthogonal matrices. The objective can thus be restated as:
min WER
X WXXI,
subject to WWT Inx
(12)
This revelation explicitly demonstrates that minimizing the reconstruction error in the space mx demands that W (equivalent to W2 in our context) projects X onto a space spanned by its most significant structural components (in terms of variance), which precisely the first m loading vectors, or principal components, of X.
Proposition A.2. The discriminative self-supervised model learns to separate data distributions in latent space as LDA in principle.
Proof. We first consider the objective of Fisher LDA for two-class classification. LDA seeks to find linear projection that maximizes the separation between the two classes:
15

---

w" Sgw
(ww w" Sww"
(13)
where Sn the between-class scatter matrix, Sw is the within-class scatter matrix, and w is the projection vector.
We take contrastive learning with InfoNCE for analysis because it is the most popular learning objective in discriminative SSL models. For contrastive learning using the asymptotic form of the InfoNCE objective [39], we have two components:
4- -- -=--B(aa,ttppapr [f(z) [xx)]] E Ex-paatt log Ex--rpeeee (e)la) f(x [
(14)
where the first term encourages similarity between positive pairs. In LDA, this is analogous to minimizing Sw because we want points from the same class to be close each other when projected onto w. The second term, when expanded using Jensen's 's inequality, represents an upper bound on the regularized sum of all pairwise inner products between different embeddings, effectively encouraging dissimilarity between all samples:
Excepauea [og Exg BagEE--na Mauulo (e f(e ]/r]]
1 m
log
1 m
h hy/T
hg.
T
(15)
When h, = f(r;) are normalized, optimizing this term aims to decrease Sum WW E mm h2 hy. This term, being an upper bound for the largest eigenvalue of WWT, when minimized, encourages a "flatter" singular value distribution of the embedding space. This flattening makes the embedding space more isotropic, which in turn increases the between-class scatter SB.
=
To make the connection explicit, consider that LDA, we want to maximize SB, the between-class scatter, which is typically represented as:
SB = ((11112)44 -22)²
(16)
where μl and (l2 are the means of the two classes.
Analogously, in the contrastive learning framework, minimizing the second term scatters the embed- dings more uniformly in the high-dimensional space, akin maximizing between-class scatter. The uniform distribution the embeddings across the space increases the separation between classes or
clusters of points, akin to the effect of maximizing Sg.
A.3
Implementation Details
We extract the hidden states from the third-to-last layer of DINOv2 for discriminative tokenizer training, it suggests more generalized representations in the intermediate layers compared the last layer. All DiGIT models are trained from seratch with a batch size of 2048 for the base model and 1024 for the large model, over duration of 200 epochs. For the image generation task, the decoder model for pixel rendering is trained with a batch size of 2048 for 400 epochs, while the large model uses batch size of 1024 for 200 epochs. The base model consists of 16 layers with a dimension of 1024 and a hidden dimension of 4096. The large model consists of 24 layers with dimension of 1536 and hidden dimension of 6144. Both configurations feature 16 attention heads. For all models, we utilize the Adam [22 optimizer with 3 = (0.9. 0.98). We employ an inverse square root decay schedule for the learning rate with a peak value of 3e for epochs warm-up. We use the same VQGAN model as MAGE Li et al. [26]. During training, we apply data augmentations including random crops resized the short edge and horizontal flips. For the decoding strategy, we apply top-k sampling with K = 400 for the large model and K 200 for the base model. In the autoregressive decoder for pixel rendering, we use a probability of 0.8 with temperature of 1.0 for top-p sampling. All experiments are conducted using the FAIRSEQ 22 library.
16

---

200.0
FID DIG
140.0
18.0
DIGIT
sage?
-140.0
16.0
-120.0
Inception Score
150.0
Scoen
100.0
14.0
100.00
80.0
Incoption
Inception Score stage?
8000
40.0
60.0
20.0
0.0
0.0
0.1
0.3
0.5
0.7
0.9
1.0
300 400
700 800 900 1000
top-p value
top-k value
(a)
(b)
Figure 5: FID and Inception Score as function top-k, top-p sampling on the image generation task with DiGIT-base. The decoding temperature is fixed to 1.0. The "stage2" denotes the autoregressive model for pixel rendering.
Original Data
PCA Reduction
LDA Reduction
Classification Accuracy
PCA LDA Acc.
Cless
Gass Olss 2
2.2
Clss
Feature 2
Frequency
Frequency
6.*
6.-
Feature
Principal Component
LDA Component
Noise Level
Figure 6: Toy example of PCA and LDA.
17

---

A.4 Qualitative Cases
Figure 7: Classinnco image generation results on ImageNet 256x256 by DIGIT.
18

---

Figure 8: Class- conditionii image generation results on ImageNet 256x256 by DiGIT, where the images the same row share the same class label.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to Section 6
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification:
Please refer to Section 2 and Appendix A.2
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer Appendix
A.3
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification: A primary reason for not providing open access to the code is at the request of our collaborators.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /nipss.cc/pulli//guu guwde//ddee/ee/ Cot//ooodeeesiseeeeeee hwtpbbmmsssioooooooooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer to Section and Appendix
A.3
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Error bars are not reported because computing them would be too computa- tionally expensive, given the large-scale nature of our experiments, which involved multiple large models and extensive datasets.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
22

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Appendix A.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with NeurlPS Code of Ethics https: http://uwrrpp..ccppppllctetcccca
Answer: [Yes]
The research aligns with the NeurlPS Code Ethics by ensuring ethical treat- ment of participants, maintaining data privacy, avoiding harmful applications, considering environmental impacts, providing transparency for reproducibility, and complying with legal and ethical standards.
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer to Section A.I
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
23

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: Implementing effective safeguards is challenging now due to limited computa- tional and human resources. We plan to develop and integrate appropriate measures prior to release to ensure responsible usage.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Please refer Section 4.5
we have appropriately cited the ImageNet dataset and adhered strictly to its terms of access, as outlined at pottps://imee net org/ download. php
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "MtcyxOSUF7", "reviewer": "Reviewer_Vc7B", "summary": "The paper propose to disentangle the encoder and decoder learning for image tokenzier which ultimately will be used for providing the latent space of AR generative model. In particular, SSL model such as DinoV2 is used for encoder (plus k-means clustering).", "questions": "1. In section 2.2, you mentioned that a drawback of auto-regressive image modeling is that each iteration only generate a patch so error in the previous generated patch will accumulate. How is this related to your method? IIUC, your tokenizer is still patch based, so it does not resolve the issue mentioned here.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "1. The idea of disentangling the encoder and decoder learning for image tokenizier is interesting and novel.\n\n2. Strong empirical results can be obtained from the method. The fact that by changing a tokenizer and training the same AR model, FID can be halved to half is really impressive.", "weaknesses": "1. The motivation for adopting self-supervised model as encoder/tokenizer is not very clear. Since the method is easy (DinoV2 + kmeans), the motivation of why doing so is the most critical part of the paper. However, I don't think this is presented very clearly and explicitly. Large improvements of the presentation is needed. \n\n2. The term \"stability\" or \"\"stablize\" is a bit confusing. Explicit explanation is needed. When is a latent space not stable? If it means hard to learn an AR model, probably a better term such as learnability is better. \n\n3. While the argument of \"iterative decoding process can stabilize the sampling process by correcting the data falling in the low-density overlap between distributions\" makes sense, it still requires justification and evidence, not just conceptual analysis.\n\n4. If you use SSL model as encoder, you need to train a decoder. Not much explicit detail is presented for this part.\n\n5. The metric is not very clearly defined. What's the name of the metric? What is the definition? How to compute it? All these information should be highlighted.\n\nOverall the presentation and organization is not very clear, some major rewrite is needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful consideration of the paper and your constructive feedback.\n\n## 1. Motivation for Adopting a Self-Supervised Model as Encoder/Tokenizer\n\nIn section 2.1, we first conduct a theoretical analysis that demonstrates the necessity of considering both $\\mathcal{D}^{H}$ and $\\mathcal{D}^{G}$ simultaneously in the latent space for generative models. Current VQ-based latent generative models represent one pathway for constructing latent space by optimizing $\\mathcal{D}^{H}$, an autoencoder objective. In contrast, our approach focuses on optimizing the latent space with $\\mathcal{D}^{G}$, which is a SSL objective. Following this, in section 2.2, we explore the differences in latent space induced by various learning objectives and find that the success of combining iterative generative models with VQ latent space is attributable to the stabilization ability of the iterative decoding strategy. In contrast, our proposed discriminative tokenizer directly stabilizes the latent space, which can naturally benefit auto-regressive models. Therefore, the above analysis and findings provide the foundation for the proposed method: SSL + Kmeans.\n\n## 2. Clarification of \"Stability\" and \"Learnability\" of the Latent Space\n\nWe measure the stability of the latent space quantitatively in Table 1, incorporating metrics such as the rate of changed tokens and cosine similarity when varying levels of noise are introduced. It intuitively validates that SSL models exhibit greater noise stability because they learn high-level semantic features of images, rather than low-level appearance features as in traditional autoencoders. \n\nWe plan to include the negative likelihood loss as a measure of learnability in our revised version (3.347 for the discriminative tokenizer compared to 7.104 for the VQ tokenizer, using the same vocabulary size and model size). The lower NLL loss represents more learnable latent space. \n\n## 3. Addressing the Argument Regarding Iterative Decoding\n\nThe iterative decoding strategy serves as the foundation for iterative generative models, including Masked Image Modeling (MIM) and diffusion models. For MIM [1], low-density tokens are revised during the iterative decoding process. Similarly, in diffusion models [2], low-density data undergo perturbation through Gaussian noise and are subsequently denoised in an iterative manner.\n\n[1] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, William T. Freeman. MaskGIT: Masked Generative Image Transformer. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2022.\n\n[2] Y. Song, S. Ermon. Generative Modeling by Estimating Gradients of the Data Distribution. In Advances in Neural Information Processing Systems, pp. 11895--11907. 2019.\n\n## 4. The Decoder for Pixel Rendering\n\nThe decoder setup is elaborated upon in Section 4.3, where we detail the additional training of a decoder for pixel rendering. To showcase the generalization and robustness of our discriminative tokenizer, we employ both autoregressive models (VQGAN) and MIMs (MaskGIT) as decoders. The experimental results presented in Tables 2 and 3 indicate that the performance gap between AR and MIM decoders is minimal, demonstrating the efficacy of the discriminative tokenizer.\n\n## 5. Clarification of the Metric\n\nThe metrics used to assess stabilization in Table 1 are the rate of token changes and the cosine similarity between perturbed and original tokens. We will ensure that these metrics are presented more clearly in the revised version.\n\n## 6. Connection Between Our Method and Error Accumulation in Auto-Regressive Image Models\n\nIn Section 2.2, we address the potential for error accumulation due to unstable VQ tokenizers across all latent generative models, including diffusion models, MIMs, and AR models. While iterative models like diffusion models and MIMs can rectify errors through their iterative decoding strategies, autoregressive models do not possess this capability. Therefore, we propose a direct approach to stabilize the latent space for autoregressive models, effectively reducing errors caused by unstable tokenizers in the decoding process.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yeECdhlS7O", "reviewer": "Reviewer_eGbB", "summary": "This paper tries to understand why latent autoregressive image models perform worse than latent diffusion models. The key insight is that existing tokenizers are trained primarily with the reconstruction objective, whose latent space is unstable and thus may not be easy to model autoregressively. To solve this issue, the authors propose first learning a stable latent space, which autoregressive models can model easily, and then learning to reconstruct pixels from this latent space. Experimental results show that this modification enables latent autoregressive image models to match latent diffusion models' performance in terms of image understanding and image generation.", "questions": "How negatively does the proposed method impact reconstruction?", "limitations": "I think the authors adequately addressed the limitations", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposed a new perspective—latent space stability—on understanding latent autoregressive image models, which was neglected in previous works. I think this explanation is intuitive since a fixed-depth autoregressive model may not be able to model very noisy distributions (e.g., the language data have high regularity)\n2. The proposed solution is straightforward -- just let image features with similar semantics share the same token.\n3. The experiments are comprehensive and interesting. Both image understanding and image generation are evaluated; improvements over previous latent autoregressive models are significant. The ablation study also makes sense to me.", "weaknesses": "1. I think there is a tension between how stable the latent space is and how easily we can reconstruct the latent codes to pixels. The impact of the proposed method on reconstruction is not elaborated in this paper. For example, if we only care about reconstruction, how badly does the proposed method perform? This matters greatly if we are modeling high-resolution images and care about the visual details.\n2. The theoretical analysis and the proposed algorithm seem loosely connected to me -- I don't see the proposed algorithm as a direct result of the theoretical analysis. The stability analysis is more straightforward, though.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful consideration of the paper and your constructive feedback.\n\n## 1. Performance of the Proposed Method on Image Reconstruction\n\nWe conduct an experiment to assess the reconstruction performance of the proposed discriminative tokenizer. We use the golden tokens obtained from the discriminative tokenizer to reconstruct the images and calculate the rFID score, which yields a result of 1.92. In comparison, the rFID score for the corresponding VQ tokenizer is 1.67. This indicates that the impact of our proposed tokenizer on reconstruction quality is minimal. We appreciate your suggestions and will include these reconstruction results in Table 2 of the revised paper.\n\n## 2. Regarding the Stability of the Proposed Method\n\n(1) Stability of Latent Space Induced by Tokenizers\n\nWe quantitatively measure the stability of the latent space in Table 1, incorporating metrics such as the rate of token changes and cosine similarity when varying levels of noise are introduced. The results align with intuition well that SSL models demonstrate greater stability against noise, as they learn high-level semantic features of images rather than low-level appearance features as in traditional autoencoders.\n\n\n(2) Stability of Autoregressive Models\n\nTo evaluate the stability of autoregressive models, we compare the negative log-likelihood (NLL) loss of models using different tokenizers, thereby demonstrating the nondeterministic nature of the latent space. We monitor the NLL loss values for both the discriminative tokenizer and VQ tokenizer, both using a vocabulary size of 1024 and the same model size: 3.347 for the discriminative tokenizer versus 7.104 for the VQ tokenizer. A lower NLL loss indicates a more stable latent space for autoregressive models. We will include the loss curve comparison in the revised version.\n\n\n## 3. Connection Between the Theoretical Analysis and the Proposed Algorithm\n\nIn section 2.1, we first conduct a theoretical analysis that demonstrates the necessity of considering both $\\mathcal{D}^{H}$ and $\\mathcal{D}^{G}$ simultaneously in the latent space for generative models. Current VQ-based latent generative models represent one pathway for constructing latent space by optimizing $\\mathcal{D}^{H}$, an autoencoder objective. In contrast, our approach focuses on optimizing the latent space with $\\mathcal{D}^{G}$, which is a SSL objective. Following this, in section 2.2, we explore the differences in latent space induced by various learning objectives and find that the success of combining iterative generative models with VQ latent space is attributable to the stabilization ability of the iterative decoding strategy. In contrast, our proposed discriminative tokenizer directly stabilizes the latent space, which can naturally benefit auto-regressive models. Therefore, the above analysis and findings provide the foundation for the proposed method: SSL + Kmeans.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w4ygBjMa3l", "reviewer": "Reviewer_Sn2s", "summary": "Latent-based image generative models, such as LDMs and MIMs, have achieved success, but autoregressive models lag behind in image generation. Our research introduces a unified perspective on latent space stability and proposes a discrete image tokenizer, DiGIT, that significantly improves autoregressive image modeling, outperforming LDMs and benefiting from scaling similar to GPT in NLP.", "questions": "See above.", "limitations": "The writing & presentation of this paper seems too rush and lacks rigor. I recommend the authors to refine and polish this paper. The current draft may not be qualified for the publication of NeurIPS.", "rating": 3, "confidence": 5, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "- The results beat some baseline models, though under a specific (and somewhat confused) experimental setting.\n- The topic of latent space property is worth investigating.", "weaknesses": "The paper has several weaknesses:\n\n1. **Factual Errors**:\n\n    1.1. The cited MIM models, such as MaskGIT and MAGE, cannot revise previously predicted tokens. This contradicts the claim in line 53 that \"iterative models like LDMs and MIMs can correct errors.\" I recommend the authors to their papers for more details.\n\n    1.2. In lines 72-73, the authors state that this work provides \"the first evidence that image autoregressive generative models behave analogously to GPT.\" However, the Parti[1] paper has already demonstrated that image autoregressive models have similar scalability to GPT and successfully scaled the model to 20B. The authors have not cited this work.\n\n2. The writing is poor and lacks rigor. For example, the discussion on the so-called \"common misconception\" in line 41 is not well-supported. What exactly is meant by the \"optimal latent space for reconstruction\"? How many studies hold this view? There are no citations provided.\n\n3. The quantitative comparisons are also peculiar. The authors cite many paper results without using CFG, while CFG has become a de-facto practice for augmenting generative models. Why not adopt CFG and perform more apples-to-apples comparisons to other SOTA methods with CFG?\n\n4. Presenting two tables (Table 2 lower and Table 3) for image generation performance is confusing. Why not consolidate the results into a single, clear table?\n\n[1] Yu, Jiahui, et al. \"Scaling autoregressive models for content-rich text-to-image generation.\" arXiv preprint arXiv:2206.10789 2.3 (2022): 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful consideration of the paper and your constructive feedback.\n\n## 1. Factual Clarification\n\nWe respectfully disagree with the points raised and would like to clarify our positions.\n\n**1.1** Regarding the MIM models like MaskGIT and MAGE, as well as diffusion models, it is important to note that they can indeed revise the predicted tokens from previous iterations. As outlined in Section 2.2, the decoding mechanism of iterative generative models can be expressed as:\n\\begin{equation}\n    p(x^T)=\\prod^{T}_{i=1} p(x^i|x^{i-1}),\n\\end{equation}\nwhere $x^{i}$ represents the predicted tokens (the entire image) in the i-th iteration. In MIM models, tokens with low probability (from the softmax of logits) are replaced with UNK tokens and re-generated in subsequent iterations. For diffusion models, the core mechanism involves iterative denoising, which is an established concept, where predicted tokens in one iteration are further modified and re-generated in the next iteration. Thus, we believe our statement in the paper is accurate and we encourage the reviewer to reconsider the interpretation in light of this clarification.\n\n**1.2** Regarding the Parti model, it is important to emphasize that it is a VQ-based text-to-image generation model that benefits from extensive training data and a large model size. However, it would be misleading to assert that Parti demonstrates success in image auto-regressive generative models. When we talk about GPT model, we are usually interested in the learning efficiency from data rather than merely enlarging dataset size and model size. By removing the influence of dataset size and model size, we note that the core architecture of Parti, which is a VQ-based AR model, does not perform as well compared to other generative models, limiting its scalability [2]. Our research specifically targets autoregressive image generative models operating without text guidance, evaluated on the ImageNet benchmark under **rigorous** experimental settings. In addition, methods later than Parti like MagVITv2 [1] and VAR [2] also claim to be the \"first\" language models outperforming diffusion models, yet they do not qualify as genuine autoregressive models. In contrast, our proposed DiGIT model is based on a pure GPT architecture, without any modifications to the decoding strategy. Therefore, we do not agree with the assertion that \"the Parti paper has already demonstrated that image autoregressive models have similar scalability to GPT\".\n\n## 2. The statement of \"common misconception\"\n\nWe cited relevant work [1] in Line 43 to support the statement regarding the \"common misconception.\" This is a well-recognized aspect within the research community, as pointed out by reviewer K39P as well. The \"optimal latent space for reconstruction\" refers to the latent space achieved by an autoencoder model that can yield the lowest rFID, which is a straightforward statement that does not require further elaboration.\n\n\n## 3. The CFG in the experiment\n\nWe choose not to employ CFG as a default method due to its tendency to sacrifice diversity in generated images. Instead, we compare all models using class labels as conditions. It is worth noting that autoregressive models can benefit a lot from CFG [3] as well. However, autoregressive models are renowned for their prompt engineering capabilities and our philosophy is to revive the GPT model with as few modifications as possible. Therefore, we intentionally do not adopt CFG as the default method for all the models in our experiments.\n\n## 4. The table in the paper\n\nWe attempted to consolidate the tables in the submission version but exceeded the page limit. In the camera-ready version, one additional page will be allowed and we could improve the table presentation if accepted.\n\n**After addressing these potential misunderstandings of the paper, we kindly request a reevaluation of our paper.**\n\n\n[1] Lijun Yu, Jos’e Lezama, Nitesh B. Gundavarapu, Luca Versari, Kihyuk Sohn, David C. Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G. Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A. Ross, and Lu Jiang. Language model beats diffusion - tokenizer is key to visual generation. ArXiv, abs/2310.05737, 2023.\n\n[2] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. 2024.\n\n[3] Sun, Peize and Jiang, Yi and Chen, Shoufa and Zhang, Shilong and Peng, Bingyue and Luo, Ping and Yuan, Zehuan. Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation. arXiv:2406.06525", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IHelEQhpV9", "reviewer": "Reviewer_K39P", "summary": "This paper designs a better quantized autoencoder on top of VQGAN. It builds an image autoencoder which is able to both achieve good recognition performance for linear probing, and have a latent space which is suitable for training a generative model. It studies the existing autoencoders from a high-level theoretical perspective and proposes design ideas which are targeted to improve them. The paper claims that the modern autoencoders ignore the fact that they will be utilized for downstream generative modelling tasks and mainly focus on reconstruction. The paper argues that adding recognition losses on top of the encoder would help. To fulfill this desiderata, the model takes DINOv2 features and discretizes them via k-means. Then it trains a translation model into VQ-GAN decoder features. For image generation, it trains a LLM in the discrete token space. For classification, it does linear probing on top of discretized DINOv2 features. As a result, it attains reasonable generative capabilities while being able to keep a latent space suitable for linear probing classification.", "questions": "- The transition \"Drawing from these theoretical insights, we introduce a metric to assess the stability of the latent space induced by different encoder models\" on L196 is extremely unclear. How exactly do theoretical results suggest that one should focus on stability of the latent space? Why would LDA lead to a better generative model? Why \"separating\" distributions class-wise would lead to a better generative model? What exactly do you mathematically define \"separation of distributions\" for an encoder?\n- Is linear probing done on top of discretized or non-discretized DINOv2 features?", "limitations": "One limitation that is not explored is whether the model is upper-bounded by the performance of the underlying SSL classifier. In other words, what would be the greater source of improved performance in the future — improving SSL or decoder?", "rating": 6, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "- In terms of the scores, the paper achieves very good results in the sense of discrimination/generation tradeoff (judging by figure 1).\n- It's an interesting finding that one can discretize dino-v2 via K-means and train a strong generative model on top of such tokens.\n- The paper studies an important problem of more rigorous understanding of modern autoencoders", "weaknesses": "- The paper shows an equivalence between a linear AE and PCA, but it's a well known fact: https://arxiv.org/abs/1804.10253. One can also just google \"equivalence between a linear autoencoder and PCA\", and find a ton of resources on that.\n- \"A reconstructive autoencoder does not necessarily establish an advantageous latent space for generative model\". That's a very well-known fact in the community (e.g., see Fig 5 in https://arxiv.org/pdf/2312.02116). The paper should not claim this observation as a novel contribution.\n- The proposed stability metric is interesting, but it's unclear whether it will correlate with downstream generative models performance\n- Proposition 2.4 is extremely vague and seems to be very different from its \"rigorous\" analogue in the supplementary.\n- FID metrics for VQGAN on ImageNet are much higher than in the original paper.\n- It's delusive to compare performance of the developed model vs those trained from scratch, since the developed model starts from strong pre-trained models.\n- For image generation, the paper shows just 16 random samples, which is extremely little to get a good understanding of the model. It's better to show much more (e.g. stylegan2 provides 100k samples for FFHQ: https://github.com/NVlabs/stylegan2).\n- Why DiT-XL/2 is included for comparison but not its guided variants? Why more recent papers are not included for comparison? (e.g., EDMv2).\n- The logical transitions in the paper are unclear, e.g., it's unclear why the proposed training improves D^G, it's unclear, why it follows from the propositions that we should improve the stability of the latent space (where stability is also not defined well), etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive feedback and would like to clarify several points to enhance the understanding of our contributions.\n\n## 1. Regarding the analysis of AE and PCA and the claim of the observation\n\nWe want to emphasize that the analysis of AE and PCA is not the core focus of our theoretical framework. We mention it to provide intuitive insights into the latent space induced by VQ autoencoder and our proposed discriminative tokenizer. \n\nWhile we acknowledge that the distinction between reconstruction and generation has been explored in prior works, our primary contribution lies in offering a theoretically unified perspective on this issue and proposing a novel method to disentangle the encoder and decoder processes for a more stable latent space, which is neglected in previous work. \n\n## 2. Regarding the stability \n\nTo further evaluate the stability of latent space, we analyze the Negative Log-Likelihood loss for AR models associated with different tokenizers. Our results highlight the nondeterministic nature of the latent space, showing an NLL loss of 3.347 for the discriminative tokenizer versus 7.104 for the VQ tokenizer, where a lower NLL loss indicates a more stable latent space, underscoring the efficacy of our approach. The resulting FID score also demonstrates that stable latent space can improve the performance of AR models.\n\n## 3. The FID score of VQGAN\n\nWe adopt the VQ tokenizer used in the MaskGIT mdoel, which is an improved version and performs better than the one used in the original VQGAN. We reproduced the **class-unconditional** results by ourselves, yielding an FID score of 24.38, which is reasonably within range and slightly worse than MaskGIT's score of 20.7. Moreover, we check the experiments in the original VQGAN paper and it only presents the results of **class-conditional** image generation results (FID 15.78), which is **not** the same experimental results as above and should not be directly compared. We will clarify these experimental settings in the revised paper.\n\n## 4. Regarding the pretrained SSL encoder in our method\n\nWe recognize the dataset difference used in SSL encoders and have included an experimental comparison of various SSL encoders in Fig 3(a). We adopt three SSL models and **all** of them are trained **only** on ImageNet, which is the same as the VQ model. As detailed in section 4.5, the **small** size autoregressive model trained with tokenizer induced by the iBOT SSL model, which shares a similar learning objective with DINOv2 (patch level discriminative objective), **already achieves superior** performance compared to a much larger model trained with VQ tokenizer. The comparison demonstrates the efficacy of the discriminative tokenizer. \n\nWe adopt a more powerful SSL encoder DINOv2 to demonstrate that the discriminative tokenizer with more powerful SSL representations can boost the performance of auto-regressive generative models by a large margin, which is not observed in existing AR + VQ models, highlighting the main limitation of image auto-regressive models is the encoder used for latent space construction. \n\n## 5. Sample Cases\n\nThe FID and IS scores are the established metrics for image-generative models. The cases shown in the paper are just for reference. We can provide more samples in the revised pdf.\n\n## 6. Regarding the baselines\n\nThere exist too many diffusion variants and most of them are based on the LDM or DiT architecture, therefore we opt to focus on foundational works, such as LDM and DiT, as baselines. Furthermore, our proposed method is built upon a different theoretical framework, aimed primarily at surpassing widely accepted baseline models.\n\n## 7. The logical transitions and question on stability\n\nIn section 2.1, we first conduct a theoretical analysis that demonstrates the necessity of considering both $\\mathcal{D}^{H}$ and $\\mathcal{D}^{G}$ simultaneously in the latent space for generative models. Current VQ-based latent generative models represent one pathway for constructing latent space by optimizing $\\mathcal{D}^{H}$, an autoencoder objective. In contrast, our approach focuses on optimizing the latent space with $\\mathcal{D}^{G}$ to learn lower-dimensional separable features for generative models, which is an SSL objective. Following this, in section 2.2, we explore the differences in latent space induced by various learning objectives and find that the success of combining iterative generative models with VQ latent space is attributable to the stabilization ability of the iterative decoding strategy. In contrast, our proposed discriminative tokenizer directly stabilizes the latent space, which can naturally benefit auto-regressive models. Therefore, the above analysis and findings provide the foundation for the proposed method: SSL + Kmeans.\n\n## 8. Question on Linear probing\n\nThe linear probing is done on the hidden states of auto-regressive models, which is the same as the evaluation protocol of iGPT and VIM.\n\n## 9. Question on limitation\n\nAs discussed in the paper, the latent generative models are affected by both the encoder and decoder. Our strategy to disentangle these components aims to strengthen the encoder’s capabilities with the integration of pre-trained SSL methods unsupervised. We use the VQGAN model as the decoder to compare with existing VQ-based models. Although improvements to both the encoder and decoder could yield further benefits, our findings indicate that the encoder's ability is currently the primary limiting factor for autoregressive models since the rFID score of VQGAN is low enough. Improvements using SSL encoders and K-means considerably elevate performance, as demonstrated in Table 2, Fig 2(a), and Fig 3(a).\n\n**We kindly ask you to reconsider the evaluation of our work in light of these clarifications. We believe that our research contributes significantly to the understanding and development of image tokenizer and latent autoregressive models.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their thoughtful comments. We appreciate that reviewers highlight the novelty and effectiveness of our method, e.g. "The idea is interesting and novel... Strong empirical results ... really impressive" (Vc7B), "A new perspective...which was neglected in previous works ... explanation is intuitive and the proposed solution is straightforward ... The experiments are comprehensive and interesting ... improvements are significant...ablation study makes sense" (eGbB), "The paper achieves very good results ... It's an interesting finding ... The paper studies an important problem" (K39P). 

The main topics the reviewers commented on were the explanation of metrics on stabilization and the connection between our theoretical analysis and the proposed method. We respond to each reviewer individually about these topics and others. 

We believe the proposed discriminative tokenizer is a non-trivial leap that indeed explores a new perspective of latent space for image auto-regressive models. The proposed method shows the promising potential of large-scale pre-training with next token prediction akin to Language Language Models (LLMs) in the visual domain.

Meta-review: This paper proposes a novel approach for disentangling the encoder and decoder learning processes in image tokenizers, with the aim of improving the stability of the latent space in autoregressive (AR) generative models. The reviewers have generally acknowledged the originality of the method and the strong empirical results it presents, particularly noting the impressive reduction in FID scores achieved through the proposed technique (Vc7B, eGbB). However, significant concerns were raised regarding the clarity and logical structure of the paper, particularly in how the concept of latent space stability is defined and connected to the proposed method (K39P, Vc7B).

One reviewer, Sn2s, recommended rejection based on what they identified as factual errors in the paper, specifically regarding the capability of MaskGIT and MAGE models to revise previously predicted tokens and the claim that this paper provides “the first evidence that image autoregressive generative models behave analogously to GPT.” During the rebuttal process and subsequent AC-reviewer discussion, the issue concerning MaskGIT and MAGE was largely resolved, with the AC and others recognizing that Sn2s’s critique was based on a narrow interpretation. However, I find that the claim about the paper providing “the first evidence” of GPT-like scalability in AR models is potentially ambiguous in the way it was presented in the manuscript. Although the authors provided a consistent explanation during the rebuttal period, I believe that the original manuscript, as written, might leave some ambiguity regarding the intended meaning of this claim. It would be beneficial for the paper to more clearly articulate what is meant by this claim to avoid any misunderstanding.

Given these considerations, I am inclined to recommend acceptance of this paper. The contributions are novel and valuable, but it is imperative that the authors address the concerns raised during the rebuttal period and the subsequent AC-reviewer discussion. The final manuscript should include a clearer articulation of the logical flow in the introduction and method sections, as suggested by K39P. Additionally, the manuscript should provide accurate definitions of key concepts such as stability and the metrics used, as emphasized by K39P and Vc7B. Furthermore, the authors should refine the claim about being the first to demonstrate GPT-like scalability in AR models, ensuring that the intended meaning is unambiguously communicated in the revised paper. These revisions will be crucial in enhancing the paper’s clarity and ensuring its impact on the field.
Decision: Accept (poster)