Stabilize the Latent Space for Image Autoregressive Modeling: A Unified Perspective

Authors: yongxin  zhu
bocheng  li
hang  zhang
xin  li
linli  xu
lidong  bing
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 19:10:01

Abstract: Latent-based image generative models, such as Latent Diffusion Models (LDMs) and Mask Image Models (MIMs), have achieved notable success in image generation tasks. These models typically leverage reconstructive autoencoders like VQGAN or VAE to encode pixels into a more compact latent space and learn the data distribution in the latent space instead of directly from pixels. However, this practice raises a pertinent question: Is it truly the optimal choice? In response, we begin with an intriguing observation: despite sharing the same latent space, autoregressive models significantly lag behind LDMs and MIMs in image generation. This finding contrasts sharply with the field of NLP, where the autoregressive model GPT has established a commanding presence. To address this discrepancy, we introduce a unified perspective on the relationship between latent space and generative models, emphasizing the stability of latent space in image generative modeling. Furthermore, we propose a simple but effective discrete image tokenizer to stabilize the latent space for image generative modeling by applying K-Means on the latent features of self-supervised learning models. Experimental results show that image autoregressive modeling with our tokenizer (DiGIT) benefits both image understanding and image generation with the next token prediction principle, which is inherently straightforward for GPT models but challenging for other generative models. Remarkably, for the first time, a GPT-style autoregressive model for images outperforms LDMs, which also exhibits substantial improvement akin to GPT when scaling up model size. Our findings underscore the potential of an optimized latent space and the integration of discrete tokenization in advancing the capabilities of image generative models. The code is available at \url{https://github.com/DAMO-NLP-SG/DiGIT}.

Main paper: 
 introduction: 
 In recent years, remarkable advancements have been achieved in the field of image generation, principally propelled by the development of latent-based generative models, such as Latent Diffusion Models (LDMs) [34, 30] and Mask Image Models (MIMs) [7, 26] . By employing reconstructive autoencoders such as VQGAN [15] or VAE [23] to compress images into a manageable low dimensional latent space, these models can generate highly realistic and imaginative samples. Concurrently, in light of the transformative impact of autoregressive (AR) generative models, such as Large Language Models [31, 32, 5, 27] in NLP, it becomes compelling to investigate the feasibility of similar paradigms to images. Despite the advances in image autoregressive pre-training, exemplified by models such as iGPT [8] , AIM [14] and GIVT [36] in the pixel space, VIM [41] and LVM [1] in the latent space, their performance are still inferior to the leading models [34, 12, 7] in image generation, or self-supervised learning models [18, 10, 6, 28] in image understanding tasks.An intriguing observation emerges regarding the presumed optimality of current practices in latent space: as illustrated in Figure 1 (b), though sharing the same latent space, autoregressive models significantly lag behind LDMs and MIMs in the image generation task. This discrepancy prompts reevaluating our understanding of latent spaces and their interaction with generative models, suggesting unexplored avenues worth investigating. A central premise in learning theory [20] is that the latent distribution should retain as much critical information of the data distribution as possible, akin to a compression goal. This leads to a common misconception that an optimal latent space for reconstruction equates to optimal generative performance. Nevertheless, in the investigation regarding the reconstruction and generation ability with the popular VQGAN model [42] , it is observed that the generation FID will deteriorate when the reconstruction FID becomes lower (where a lower value indicates better performance), challenging the above assumption.To address these intriguing discrepancies, we introduce a unified perspective on the relationship between latent spaces and generative models to analyze what constitutes an optimal latent space for generative models. Our findings reveal that, beyond the compression techniques employed by prevalent latent generative models, an optimal latent space should also aim to minimize the distance between distributions under the condition of incorporating a generative model, which is an aspect often overlooked. We critically assess prevalent methodologies and reveal that the stability of latent space is important for generative models. We argue that the reason why autoregressive models underperform iterative models such as LDMs and MIMs is that iterative models can correct errors brought by the instability of latent space.Drawing from this insight, we propose a straightforward method to stabilize the existing latent space methods for image autoregressive generative models. Unlike conventional autoencoder-style approaches, our approach disentangles the concurrent training of encoders and decoders and commences with encoder-only training through a discriminative self-supervised model [28] . This phase does not necessitate a decoder for pixel reconstruction, enabling the encoder to discern the intrinsic and distinguishable features present within the data. Subsequently, a separate decoder of the autoencoder [15] is trained and tasked solely with the pixel reconstruction process, conditioned on the features identified by the encoder. By focusing initially on the encoder's capability to extract meaningful data features independently of pixel reconstruction, we lay a foundation for a more stable and feature-rich latent space. The subsequent independent training phase of the decoder ensures that these captured features can be accurately translated back to pixels.In support of the autoregressive generative model, which requires discrete tokens for next token prediction, we employ a strategy inspired by VQGAN [15] to discretize the encoder's latent feature space with the K-Means clustering algorithm. With this novel image tokenizer induced from the stabilized latent space, the performance of autoregressive generative models in images is enhanced significantly in both image understanding and image generation tasks. We refer to this approach as call Discriminative Generative Image Transformer (DiGIT). Notably, when scaling up the model size, substantial improvements can be achieved. To the best of our knowledge, this is the first evidence that image autoregressive generative models behave analogously to GPT. In essence, this work endeavors to redefine the boundaries of what is possible in image autoregressive modeling through a unified perspective of latent space.In summary, our contributions to the field of image generative models include:• We introduce a unified perspective on the relationship between latent space and generative models, emphasizing the stability of latent space in image generative modeling.• We propose a novel method to stabilize latent space by disentangling the encoder and decoder training processes. Furthermore, a simple yet effective discrete image tokenizer is proposed to improve the image autoregressive generative model's performance under the philosophy of next token prediction.• The experimental results show that the image autoregressive modeling with our tokenizer leads to SOTA performance in image understanding and generation, with further improvements witnessed when scaling up the model size.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: Please refer to Section 6.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
problem formulation: 
 In section 2.1, we formalize the latent space requirements for generative models and categorize current latent-based generative models. Furthermore, in section 2.2, we analyze the stability of different induced latent spaces and propose to stabilize the latent space for autoregressive generative models instead of stabilizing the generation process with an iterative decoding strategy like LDMs.Drawing inspiration from the complexity perspective of latent space induced from autoencoders in Hu et al. [21] , we delve into the latent space for generative models. Generative models aim at learning a distribution to approximate the data distribution P X . Formerly, given a tractable prior distribution P Z and a distance metric D(•, •) between distributions, the purpose of a generative model g ∈ G is to minimize the distance between data distribution P X and the distribution generated by g(Z):EQUATIONFor example, GANs [16] employ the Gaussian distribution as their prior and utilize a discriminator network as the distance metric. However, the optimal strategy for data representation in generative models is still under-explored. Recent studies on latent diffusion models [34] have identified that direct learning in the pixel space of images is suboptimal. They propose to learn in a latent space induced by a constrained autoencoder model such as VAE [23] or VQGAN [15] , which has been demonstrated to improve the perceptual quality.A simple method to construct the latent space is using an encoder f ∈ F : R d → R dz to map raw data samples x ∈ R d into a latent space f (X) of dimension d z . Consequently, the goal of latent-based generative models is to learn the distribution as per the following formula:EQUATIONwhere P f (X) denotes the data distribution in the latent space induced by the encoder f . Despite these advances, determining the optimal latent space configuration for generative models remains unresolved.Distance between distributions in different spaces. Given the ultimate goal of the generative model is to produce image pixels, a decoder model h ∈ H : R dz → R d , paired with the encoder model f , is necessary to convert latent representations back into pixels. We define a generalized distance between different spaces associated with a decoder h ∈ H as:EQUATIONBy employing D H to compare distributions across different spaces, we can define the ideal latent space f (X) as the one that minimizes D H (P f (X) , P X ). This implies selecting a latent representation that minimizes the empirical objective L(h|P f (X) ) with the same family H of decoder models. Such a latent configuration depends on both the data and the decoder training methodology. We can formalize it with an autoencoder framework by looking at the encoder and decoder together, and the primary goal becomes the reconstruction of the input sample x, formulated as min f,h L(h(f (x)), x).Once a generative model g successfully approximates the latent distribution P f (X) , the generated sample can be efficiently transformed back into pixels using the decoder h.Similarly, we can define the distance between distributions in the latent space f (X) ∈ R dz and data space X ∈ R d conditioned on the latent-based generative model g ∈ G,EQUATIONwhich aims to minimize the empirical objective with the generative model family G.Optimal Latent Space for Generative Models. Now that we have characterized the ideal latent distribution given the family of generative models and the data, the next step is to determine how to find the optimal latent space. At the population level, the objective for the latent-based generative models with a decoder is:EQUATIONwhere the first term focuses on optimizing the decoder to enhance the reconstruction quality and the second one is directed towards optimizing the generative model to more accurately approximate the latent space distribution. Inspired by this observation, we can characterize the optimal latent distribution P * f (X) for a given P X from the perspective of minimizing the distance between distributions in different spaces by defining f * asEQUATIONNotice that P f * (X) depends on multiple factors, including P X , the distance metric D, and the constructed model families G and H. By integrating D G and D H , we arrive at a comprehensive measure of the distance between the distribution in the latent space and the original data distribution as D latent (P f (X) , P X ). The second term, D H (P f (X) , P X ), is exactly the objective of reconstructive autoencoders. Ultimately, from examining the learning objective pertinent to identifying the optimal latent space for generative models, it becomes evident that:A reconstructive autoencoder does not necessarily establish an advantageous latent space for generative models.Two Pathways of the Latent Space Construction. Although we theoretically analyze the optimization of the optimal latent space for generative models, it is challenging to implement in practice because optimizing (f, g, h) simultaneously is computationally complex. A practical solution is to optimize D G and D H separately, allowing for tractable training.• When D H (P f (X) , P X ) is not a target for optimization, it implies that the optimization of the decoder within the generative model framework is bypassed. The encoder independently forms a latent space, aligning with self-supervised learning (SSL) strategies aimed at uncovering lower-dimensional features from unlabeled data. However, learning the generative models in the latent space induced by SSL models remains relatively unexplored. • On the other hand, when D G (P f (X) , P X ) remains fixed, the primary objective becomes optimizing the encoder and decoder to effectively learn and represent the latent space, where D latent (P f (X) , P X ) degrades into an autoencoder learning objective. This approach is evident in recent latent-space-oriented generative models, such as LDM [34, 30] , VQGAN (AR) [15] , and MaskGIT (MIM) [7] , all of which concentrate on learning g in the latent space with the encoder and decoder frozen. 2While latent generative models such as LDM [34] , MaskGIT [7] , and VQGAN [15] share the same latent space induced by a reconstructive autoencoder [15] to minimize D H (P f (X) , P X ), their image generation performances differ significantly. In the next section, we analyze the reason behind it from the perspective of the latent space.We first describe the decoding mechanism of various latent generative models. Both LDM [34] and MaskGIT [7] can be depicted as an iterative sampling procedure given by:EQUATIONThe intermediate states x i in LDMs represent images infused with Gaussian noise of varying variance, whereas for MaskGIT, they denote discretely tokenized images augmented with masks. In contrast, the autoregressive framework of VQGAN (AR) is described as:EQUATIONwhere x i represents the i-th patch in the image sequence. Notice that x i denotes the entire image while x j means the local patch tokens. In the autoregressive decoding process, if the previously sampled tokens are incorrect, the accuracy of subsequent tokens would be affected due to error aggregation. In contrast, the iterative decoding approach allows for the revision of earlier misjudged tokens.When the latent space is unstable that small perturbation in pixels can change the latent distribution significantly, the iterative decoding mechanism employed by LDM and MaskGIT can alleviate the error aggregation problem by allowing for revision of earlier misjudged tokens while autoregressive models cannot. Consequently, a stable latent space is required to reduce errors introduced in the generation process of autoregressive models.This principle forms the foundation of our methodology for developing a metric to evaluate latent spaces with an emphasis on the stability of the latent representations. We examine two primary types of latent spaces: (1) autoencoder induced by minimizing D H (P f (X) , P X ) and ( 2) self-supervised learning (SSL) model induced by minimizing D G (P f (X) , P X ). By analyzing network parameters of these models in a linear regime, we derive the following propositions. Proposition 2.1. The latent space spanned by a linear autoencoder is congruent with that spanned by the principal component loading vectors derived in Principal Component Analysis (PCA). Furthermore, the principal component loading vectors can be elucidated from the autoencoder's weights. Proposition 2.2. The discriminative self-supervised model learns to separate data distributions in the latent space as Linear Discriminant Analysis in principle.Motivated by these theoretical insights, we introduce a metric to assess the stability of the latent space induced by different encoder models. To exemplify these concepts, we refer to an example consisting of two Gaussian distributions in a two-dimensional space, as depicted in Figure 6 (a). The results attained from applying the PCA and LDA algorithms are visually depicted in Figure 6 (b) and (c) respectively. The distribution embedded by the LDA model exhibits greater separability than that by the PCA model. To quantitatively evaluate stability, we add Gaussian noise of different variances to the original 2D data and subsequently train a linear classifier on the latent space. As Figure 6(d) illustrates, the accuracy of the LDA model consistently surpasses that of PCA. To evaluate the stability of latent space induced from autoencoders and SSL models, we add Gaussian noise to image pixels and then feed the noisy images to a VQGAN encoder and an SSL encoder DINOv2 [28] . This experiment aims to examine the resilience of the latent spaces induced by these encoders to such disturbances. We measure the rate of change in discrete tokens, specifically VQ tokens for the latent space induced from the VQGAN encoder and discriminative tokens for the latent space induced from the SSL model, and the cosine similarity in conjunction with the strength of the noise introduced. The experimental results in Table 1 demonstrate that the latent space induced from the SSL model exhibits heightened stability compared to that derived from the VQGAN autoencoder. Therefore, we propose to replace the unstable latent space induced by the reconstructive model with a stable latent space induced by the discriminative self-supervised model for autoregressive models.3 Stabilize the Latent Space with Self-supervised Learning ModelIn this section, we present a simple but effective image tokenizer that discretizes the feature representations of discriminative SSL models to form discrete tokens for autoregressive models. The architecture of our model is illustrated in Figure 2 .Discrete Image Discriminative Tokenizer Drawing inspiration from the VQGAN tokenizer [15] , which employs an implicit K-Means clustering algorithm within the latent space to generate discrete tokens for autoregressive modeling, we propose a straightforward approach to perform K-Means clustering on the feature space of discriminative SSL models to obtain discrete tokens. To process a given dataset, our initial step involves gathering the features of image patches, akin to the hidden states produced by SSL models. Then we employ a clustering algorithm to group these patches, resulting in a collection of K clustering centers. These centers constitute the codebook for the discrete tokenizer. To determine the discrete tokens for an image patch at inference, we identify its nearest neighbor in the codebook, which is then designated as the discrete token for the respective patch.Image Autoregressive Modeling After converting images into discrete tokens with the discriminative tokenizer, we treat each image as a sequence by flattening the discrete tokens from images into a 1D sequence in raster order. We train a causal Transformer [38] model with the next token prediction objective, which is the same as the standard approach for language models. We take the discriminative SSL model DINOv2 [28] as the encoder for all experiments. The K-Means model is trained on the randomly selected 10% subset of the ImageNet [11] training set. We use the autoregressive model with the same architecture as GPT-2 [32] . We train the DiGIT models with the base and large sizes. The vocabulary size of the tokenizer for the base is 8192 and 16000 for the large size. More implementation details and hyper-parameters are provided in Appendix A.3.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Please refer to Section A.1.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
theory assumptions and proofs: 
 Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
image understanding: 
 The GPT model is famous for learning semantic features by a generative training objective of next token prediction. We compare the image understanding ability of different image autoregressive models with linear-probe as described in iGPT [8] . We train a linear classifier on top of the frozen features average from each layer on the ImageNet training set. We report the Top-1 accuracy compared with other image autoregressive models in Table 2 . Remarkably, with only 219M parameters, DiGIT achieves a Top-1 accuracy of 71.7%, surpassing both iGPT and VIM-Base, which have a greater number of parameters and operate at more visual tokens. Despite representing images with a smaller token grid size (16 × 16 as opposed to 32 × 32), DiGIT still delivers superior top-1 accuracy, demonstrating the effectiveness of our tokenizer. Moreover, when we scale DiGIT's parameters from 219M to 732M, the Top-1 accuracy shows an additional increase of 8.6% and reaches 80% for the first time. The improvement indicates that DiGIT with the proposed discriminative tokenizer has the potential for the development of large vision models.Since the SSL models do not have a paired decoder to recover pixels from latent space, the generative models trained with our discriminative tokenizer require an auxiliary image decoder to render pixels. The discriminative tokenizer can be seamlessly integrated with any existing image generative models trained with a tokenizer induced from a reconstructive autoencoder. In our experiment, we train an autoregressive model VQGAN, and an MIM model MaskGIT as the pixel decoder respectively. The results are presented in Table 3 and Table 4 . The autoregressive model equipped with our discriminative tokenizer achieves the SOTA performance with FID reaching 3 for the first time. Furthermore, the performance significantly improves as the model size increases, demonstrating the potential of a large vision model with next token prediction. Interestingly, when utilizing the DiGIT as the conditioning factor, the performance of both the autoregressive and MaskGIT decoders becomes close (4.62 and 4.79). This observation suggests that stabilizing the latent space produces effects analogous to the iterative stabilization decoding mechanism.We conduct the ablation study to present a comprehensive analysis of the proposed discriminative tokenizer in image generation and understanding. The results are illustrated in Table 3 (a) and For the image understanding task, we investigate the effect of K-Means clusters and features learned in the different layers of DiGIT. We can see that increasing the cluster number can further improve the accuracy of the linear probe, which means the image autoregressive model can benefit from a larger vocabulary. The linear probe accuracy increases quickly from the first transformer block, reaches its peak at the middle layers, and finally decreases a little bit for the last few blocks. This observation connects the image autoregressive model to the text language model where the semantic information is learned in the middle layers of the transformer.We conduct an experiment to investigate the effect of different SSL models on latent space. We generally categorize the SSL models into two types according to their pre-training objectives: (1) Global level (MoCo) and patch level (MAE,iBOT), ( 2) reconstructive (MAE) and discriminative (MoCo, iBOT). At the global level, the loss function is computed using an aggregate output such as [CLS] token or mean pooling. In contrast, patch-level models involve patches directly in loss computation. Reconstructive models, such as MAE, aim to recover image pixels in a manner akin to autoencoders, while discriminative models are optimized to learn the distinguishable features. As demonstrated in Table 4 (a), the discriminative objective plays a pivotal role in image generation in that it can stabilize the latent space. Furthermore, because generative models need to predict patches, the inclusion of a patch-level loss function can enhance performance.To assess the stability of latent space induced by our discriminative tokenizer and reconstructive tokenizer. We pre-train two auto-regressive generative models on the ImageNet dataset [11] , employing the proposed discriminative tokenizer and VQGAN tokenizer respectively. We provide each model with the upper half of the target image as a conditional prompt for generation, challenging them to complete the lower half of the image. A stable latent space should be able to help the autoregressive model generate the lower half more robustly, maintaining thematic and aesthetic coherence. As shown in Figure 4 (b), the FID decreases for both models when given a longer prefix context. However, when the prefix length is reduced from 75% to only 12.5% of the image, the model trained with the VQGAN tokenizer encounters difficulties in producing images that adhere to the specified prompt. In contrast, the model utilizing the discriminative tokenizer effectively continues to produce congruent visual tokens, maintaining low FID scores even with a significantly truncated prefix. 
related work: 
 Image Tokenizer The image tokenizer [37, 33, 15] is essential in converting pixels into discrete tokens for autoregressive generative modeling. VQVAE [37] first proposes to assign latent features learned by an encoder to the nearest entry in the learnable codebook embeddings, followed by a decoder to reconstruct the original image pixels. VQGAN [15] further incorporates adversarial loss and perceptual loss to improve the image synthesis quality. RQ-Transformer [24] extends the single-layer quantizer to the multi-layer residual quantizer to augment the visual tokenizer's ability to capture fine-grained details. ViT-VQGAN [41] incorporates the modern Vision-Transformer [13] into VQGAN to enhance the reconstruction quality. MAGVIT-v2 [42] substitute the online update codebook in VQGAN with a lookup-free quantizer to enable a larger vocabulary for generative language models. Image Autoregressive Modeling Inspired by the success of autoregressive Transformer [38] in text generation tasks, there have been several efforts to replicate it in image generation tasks. One of the pioneering works is iGPT [8] , which pre-trains an autoregressive model on pixels with the same architecture as GPT2 [32] , achieving promising results in unsupervised visual representation learning. LVM [1] proposes a large-scale vision dataset composed of images and videos, based on which a large vision model is trained. Empirical observations indicate that the model scales effectively across various tasks with in-context learning [40] . Similarly, AIM [14] follows ViT [13] and represents images in the patch. It is observed that the performance of image recognition continues to increase as the model size scales up. VAR [35] proposes a next-scale prediction to generate images from coarse to fine in a hybrid of autoregressive and nonautoregressive manner.Self-supervised Learning Models Self-supervised learning (SSL) [9, 17] plays an important role in learning fundamental visual representations for downstream tasks. Among them, SimCLR [9] , MoCo [9, 18, 10 ] compute losses at the image level through [CLS] token aggregation or pooling operations with contrastive learning. iBOT-style models [43, 6, 28] extend the loss to the patch level, achieving improved performance in dense prediction tasks. BEiT [3] uses VQGAN tokenized sequences as the training target. MAE [19] randomly masks some patches of images and reconstructs the pixels with unmasked patches as the condition. 
conclusion: 
 In this paper, we make an exploration in the latent space for generative modeling. We introduce a unified perspective on the relationship between latent space and generative models, emphasizing the stability of latent space in image generative modeling. Subsequently, we propose a simple but effective discriminative image tokenizer, followed by an image autoregressive generative model DiGIT. Empirical results indicate that our tokenizer achieves superior performance across both image understanding and image generation tasks. Notably, when DiGIT is scaled up in model size, it exhibits even greater enhancements, indicating the potential for the development of large vision models. Our findings challenge the conventional wisdom that proficiency in reconstruction equates to an effective latent space for auto-regressive generation. Through this work, we aim to rekindle interest in the generative pre-training of image auto-regressive models and encourage a reevaluation of the fundamental components that define latent space for generative models. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Please refer to Section A.1.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [No] Justification: Implementing effective safeguards is challenging now due to limited computational and human resources. We plan to develop and integrate appropriate measures prior to release to ensure responsible usage. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: Please refer to Section 4.5. we have appropriately cited the ImageNet dataset and adhered strictly to its terms of access, as outlined at https://image-net.org/ download.php. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Our proposed discriminative tokenizer exhibits remarkable capabilities in both the image understanding and the image generation tasks, which is challenging for a single model. However, the discriminative tokenizer induced by the SSL model cannot directly render pixels. Consequently, we need to train another decoder model to convert tokens from discriminative tokenizers to VQ-GAN tokens. The potential for a direct generation of RGB pixels from the tokens produced by the discriminative tokenizer remains an uncharted avenue, which we leave for future work.Image generative models possess a dichotomous nature, particularly within the realm of visual media. On the positive side, they foster a myriad of creative endeavors, and methodologies aim to minimize the costs of training and inference, holding the potential to broaden access and democratize the use of this technology. On the negative side, the simplicity with which manipulated content can be crafted and propagated raises serious concerns. This includes the proliferation of misinformation and spam. Furthermore, generative models may inadvertently unveil aspects of their training data, a particularly troubling issue when that data includes sensitive or personal information collected without explicit consent.Proposition A.1. Consider the optimization problem for X ∈ R n :EQUATIONwhich is a linear autoencoder. W 2 is a minimizer of the problem if and only if its column space is spanned by the first m loading vectors of X.Proof. First, we derive the condition for the optimal W 1 in the context of this optimization problem.Setting the gradient of the objective function with respect to W 1 to zero leads to W 1 being the left Moore-Penrose pseudoinverse of W 2 [2] . Similarly, setting the gradient with respect to W 2 to zero would identify W 2 as the right pseudoinverse of W 1 :EQUATION)This finding indicates that the optimization can be simplified to focus on a single matrix (either W 1 or W 2 ), hence removing the redundancy in the parameters:EQUATIONThe termW 2 W † 2 = W 2 (W T 2 W 2 ) -1 W T2 is recognized as the matrix form of the orthogonal projection operator onto the column space of W 2 . This property holds true even when the column vectors of W 2 are not orthonormal.By performing QR decomposition on W 2 , W 2 = QR where Q is an orthogonal matrix (Q T Q = I) and R is an upper triangular matrix, we effectively transform the problem into optimizing over orthogonal matrices. The objective can thus be restated as:EQUATION)This revelation explicitly demonstrates that minimizing the reconstruction error in the space R m×n demands that W (equivalent to W 2 in our context) projects X onto a space spanned by its most significant structural components (in terms of variance), which are precisely the first m loading vectors, or principal components, of X.Proposition A.2. The discriminative self-supervised model learns to separate data distributions in latent space as LDA in principle.Proof. We first consider the objective of Fisher LDA for two-class classification. LDA seeks to find a linear projection that maximizes the separation between the two classes:EQUATION)where S B is the between-class scatter matrix, S W is the within-class scatter matrix, and w is the projection vector.We take contrastive learning with InfoNCE for analysis because it is the most popular learning objective in discriminative SSL models. For contrastive learning using the asymptotic form of the InfoNCE objective [39] , we have two components:EQUATION)where the first term encourages similarity between positive pairs. In LDA, this is analogous to minimizing S W because we want points from the same class to be close to each other when projected onto w. The second term, when expanded using Jensen's inequality, represents an upper bound on the regularized sum of pairwise inner products between different embeddings, effectively encouraging dissimilarity between all samples:EQUATION)Whenh i = f (x i ) are normalized, optimizing this term aims to decrease Sum(WW ⊤ ) = m i=1 m j=1 h ⊤ i h j .This term, being an upper bound for the largest eigenvalue of WW ⊤ , when minimized, encourages a "flatter" singular value distribution of the embedding space. This flattening makes the embedding space more isotropic, which in turn increases the between-class scatter S B .To make the connection explicit, consider that in LDA, we want to maximize S B , the between-class scatter, which is typically represented as:EQUATION)where µ 1 and µ 2 are the means of the two classes.Analogously, in the contrastive learning framework, minimizing the second term scatters the embeddings more uniformly in the high-dimensional space, akin to maximizing between-class scatter. The uniform distribution of the embeddings across the space increases the separation between classes or clusters of points, akin to the effect of maximizing S B .We extract the hidden states from the third-to-last layer of DINOv2 for discriminative tokenizer training, as it suggests more generalized representations in the intermediate layers compared to the last layer. All DiGIT models are trained from scratch with a batch size of 2048 for the base model and 1024 for the large model, over a duration of 200 epochs. For the image generation task, the decoder model for pixel rendering is trained with a batch size of 2048 for 400 epochs, while the large model uses a batch size of 1024 for 200 epochs. The base model consists of 16 layers with a dimension of 1024 and a hidden dimension of 4096. The large model consists of 24 layers with a dimension of 1536 and a hidden dimension of 6144. Both configurations feature 16 attention heads. For all models, we utilize the Adam [22] optimizer with β = (0.9, 0.98). We employ an inverse square root decay schedule for the learning rate with a peak value of 3e -4 for 10 epochs warm-up. We use the same VQGAN model as MAGE Li et al. [26] . During training, we apply data augmentations including random crops resized to the short edge and horizontal flips. For the decoding strategy, we apply top-k sampling with K = 400 for the large model and K = 200 for the base model. In the autoregressive decoder for pixel rendering, we use a probability of 0.8 with a temperature of 1.0 for top-p sampling. All experiments are conducted using the FAIRSEQ [29] library. NeurIPS Paper ChecklistD G and D H can achieve zero simultaneously. For example, in the well-known posterior collapse phenomenon in the VAE literature, the latent space f (X) is a tractable Gaussian distribution and D G (P f (X) , PX ) can be zero by simply setting the generative models as a Gaussian sampler. If the decoder is strong enough and directly generates samples without conditioning on the encoder output, D H (P f (X) , PX ) can be zero as well. 


Decision: This paper proposes a novel approach for disentangling the encoder and decoder learning processes in image tokenizers, with the aim of improving the stability of the latent space in autoregressive (AR) generative models. The reviewers have generally acknowledged the originality of the method and the strong empirical results it presents, particularly noting the impressive reduction in FID scores achieved through the proposed technique (Vc7B, eGbB). However, significant concerns were raised regarding the clarity and logical structure of the paper, particularly in how the concept of latent space stability is defined and connected to the proposed method (K39P, Vc7B).

One reviewer, Sn2s, recommended rejection based on what they identified as factual errors in the paper, specifically regarding the capability of MaskGIT and MAGE models to revise previously predicted tokens and the claim that this paper provides “the first evidence that image autoregressive generative models behave analogously to GPT.” During the rebuttal process and subsequent AC-reviewer discussion, the issue concerning MaskGIT and MAGE was largely resolved, with the AC and others recognizing that Sn2s’s critique was based on a narrow interpretation. However, I find that the claim about the paper providing “the first evidence” of GPT-like scalability in AR models is potentially ambiguous in the way it was presented in the manuscript. Although the authors provided a consistent explanation during the rebuttal period, I believe that the original manuscript, as written, might leave some ambiguity regarding the intended meaning of this claim. It would be beneficial for the paper to more clearly articulate what is meant by this claim to avoid any misunderstanding.

Given these considerations, I am inclined to recommend acceptance of this paper. The contributions are novel and valuable, but it is imperative that the authors address the concerns raised during the rebuttal period and the subsequent AC-reviewer discussion. The final manuscript should include a clearer articulation of the logical flow in the introduction and method sections, as suggested by K39P. Additionally, the manuscript should provide accurate definitions of key concepts such as stability and the metrics used, as emphasized by K39P and Vc7B. Furthermore, the authors should refine the claim about being the first to demonstrate GPT-like scalability in AR models, ensuring that the intended meaning is unambiguously communicated in the revised paper. These revisions will be crucial in enhancing the paper’s clarity and ensuring its impact on the field.

Reviews: review: paper_summary: paper_summary: The paper propose to disentangle the encoder and decoder learning for image tokenzier which ultimately will be used for providing the latent space of AR generative model. In particular, SSL model such as DinoV2 is used for encoder (plus k-means clustering).
strength_weakness: strength_weakness: 1. The motivation for adopting self-supervised model as encoder/tokenizer is not very clear. Since the method is easy (DinoV2 + kmeans), the motivation of why doing so is the most critical part of the paper. However, I don't think this is presented very clearly and explicitly. Large improvements of the presentation is needed. 

2. The term "stability" or ""stablize" is a bit confusing. Explicit explanation is needed. When is a latent space not stable? If it means hard to learn an AR model, probably a better term such as learnability is better. 

3. While the argument of "iterative decoding process can stabilize the sampling process by correcting the data falling in the low-density overlap between distributions" makes sense, it still requires justification and evidence, not just conceptual analysis.

4. If you use SSL model as encoder, you need to train a decoder. Not much explicit detail is presented for this part.

5. The metric is not very clearly defined. What's the name of the metric? What is the definition? How to compute it? All these information should be highlighted.

Overall the presentation and organization is not very clear, some major rewrite is needed.
questions: questions: 1. In section 2.2, you mentioned that a drawback of auto-regressive image modeling is that each iteration only generate a patch so error in the previous generated patch will accumulate. How is this related to your method? IIUC, your tokenizer is still patch based, so it does not resolve the issue mentioned here.
limitations: limitations: N/A
paper_summary: paper_summary: This paper tries to understand why latent autoregressive image models perform worse than latent diffusion models. The key insight is that existing tokenizers are trained primarily with the reconstruction objective, whose latent space is unstable and thus may not be easy to model autoregressively. To solve this issue, the authors propose first learning a stable latent space, which autoregressive models can model easily, and then learning to reconstruct pixels from this latent space. Experimental results show that this modification enables latent autoregressive image models to match latent diffusion models' performance in terms of image understanding and image generation.
strength_weakness: strength_weakness: 1. I think there is a tension between how stable the latent space is and how easily we can reconstruct the latent codes to pixels. The impact of the proposed method on reconstruction is not elaborated in this paper. For example, if we only care about reconstruction, how badly does the proposed method perform? This matters greatly if we are modeling high-resolution images and care about the visual details.
2. The theoretical analysis and the proposed algorithm seem loosely connected to me -- I don't see the proposed algorithm as a direct result of the theoretical analysis. The stability analysis is more straightforward, though.
questions: questions: How negatively does the proposed method impact reconstruction?
limitations: limitations: I think the authors adequately addressed the limitations
paper_summary: paper_summary: Latent-based image generative models, such as LDMs and MIMs, have achieved success, but autoregressive models lag behind in image generation. Our research introduces a unified perspective on latent space stability and proposes a discrete image tokenizer, DiGIT, that significantly improves autoregressive image modeling, outperforming LDMs and benefiting from scaling similar to GPT in NLP.
strength_weakness: strength_weakness: The paper has several weaknesses:

1. **Factual Errors**:

    1.1. The cited MIM models, such as MaskGIT and MAGE, cannot revise previously predicted tokens. This contradicts the claim in line 53 that "iterative models like LDMs and MIMs can correct errors." I recommend the authors to their papers for more details.

    1.2. In lines 72-73, the authors state that this work provides "the first evidence that image autoregressive generative models behave analogously to GPT." However, the Parti[1] paper has already demonstrated that image autoregressive models have similar scalability to GPT and successfully scaled the model to 20B. The authors have not cited this work.

2. The writing is poor and lacks rigor. For example, the discussion on the so-called "common misconception" in line 41 is not well-supported. What exactly is meant by the "optimal latent space for reconstruction"? How many studies hold this view? There are no citations provided.

3. The quantitative comparisons are also peculiar. The authors cite many paper results without using CFG, while CFG has become a de-facto practice for augmenting generative models. Why not adopt CFG and perform more apples-to-apples comparisons to other SOTA methods with CFG?

4. Presenting two tables (Table 2 lower and Table 3) for image generation performance is confusing. Why not consolidate the results into a single, clear table?

[1] Yu, Jiahui, et al. "Scaling autoregressive models for content-rich text-to-image generation." arXiv preprint arXiv:2206.10789 2.3 (2022): 5.
questions: questions: See above.
limitations: limitations: The writing & presentation of this paper seems too rush and lacks rigor. I recommend the authors to refine and polish this paper. The current draft may not be qualified for the publication of NeurIPS.
paper_summary: paper_summary: This paper designs a better quantized autoencoder on top of VQGAN. It builds an image autoencoder which is able to both achieve good recognition performance for linear probing, and have a latent space which is suitable for training a generative model. It studies the existing autoencoders from a high-level theoretical perspective and proposes design ideas which are targeted to improve them. The paper claims that the modern autoencoders ignore the fact that they will be utilized for downstream generative modelling tasks and mainly focus on reconstruction. The paper argues that adding recognition losses on top of the encoder would help. To fulfill this desiderata, the model takes DINOv2 features and discretizes them via k-means. Then it trains a translation model into VQ-GAN decoder features. For image generation, it trains a LLM in the discrete token space. For classification, it does linear probing on top of discretized DINOv2 features. As a result, it attains reasonable generative capabilities while being able to keep a latent space suitable for linear probing classification.
strength_weakness: strength_weakness: - The paper shows an equivalence between a linear AE and PCA, but it's a well known fact: https://arxiv.org/abs/1804.10253. One can also just google "equivalence between a linear autoencoder and PCA", and find a ton of resources on that.
- "A reconstructive autoencoder does not necessarily establish an advantageous latent space for generative model". That's a very well-known fact in the community (e.g., see Fig 5 in https://arxiv.org/pdf/2312.02116). The paper should not claim this observation as a novel contribution.
- The proposed stability metric is interesting, but it's unclear whether it will correlate with downstream generative models performance
- Proposition 2.4 is extremely vague and seems to be very different from its "rigorous" analogue in the supplementary.
- FID metrics for VQGAN on ImageNet are much higher than in the original paper.
- It's delusive to compare performance of the developed model vs those trained from scratch, since the developed model starts from strong pre-trained models.
- For image generation, the paper shows just 16 random samples, which is extremely little to get a good understanding of the model. It's better to show much more (e.g. stylegan2 provides 100k samples for FFHQ: https://github.com/NVlabs/stylegan2).
- Why DiT-XL/2 is included for comparison but not its guided variants? Why more recent papers are not included for comparison? (e.g., EDMv2).
- The logical transitions in the paper are unclear, e.g., it's unclear why the proposed training improves D^G, it's unclear, why it follows from the propositions that we should improve the stability of the latent space (where stability is also not defined well), etc.
questions: questions: - The transition "Drawing from these theoretical insights, we introduce a metric to assess the stability of the latent space induced by different encoder models" on L196 is extremely unclear. How exactly do theoretical results suggest that one should focus on stability of the latent space? Why would LDA lead to a better generative model? Why "separating" distributions class-wise would lead to a better generative model? What exactly do you mathematically define "separation of distributions" for an encoder?
- Is linear probing done on top of discretized or non-discretized DINOv2 features?
limitations: limitations: One limitation that is not explored is whether the model is upper-bounded by the performance of the underlying SSL classifier. In other words, what would be the greater source of improved performance in the future — improving SSL or decoder?

Comments: title: Follow-up on Our Rebuttal Submission
Rebuttal Review Required for Accurate Assessment
After rebuttal
comment: Thank you for your thoughtful feedback and for taking the time to engage with our work. We appreciate your acknowledgment of our detailed responses. We are committed to making improvements in clarity and coherence to enhance reader comprehension during the revision process.
I am thankful to the authors for elaborated responses to my pesky inquiries. In terms of writing, my main concern not the absence of some metrics or experiments, but the high-level exposition: I had difficulties figuring out the logical transitions and relating theoretical claims to the main text. As to CFG, I apologize for requesting the results with it, since I realized that it is less common for autoregressive models (i no longer have the concern regarding CFG).
## Revision of the Manuscript
We will make revisions to the manuscript to address the comments raised during the review process in the camera-ready submission, should our paper be accepted. The main changes will include:
1. Metrics: We will add a description of the stability metrics used in Table 1.

2. Experiments: We will include the stability of latent space for autoregressive models by adding NLL loss curve comparisons.

3. Experiments: We will add the rFID score of the VQ tokenizer to Table 2 and enhance the overall presentation of the table.

3. Appendix: We will provide more qualitative samples to strengthen our findings.

## Question about CFG

We believe that autoregressive models can also significantly benefit from CFG, as demonstrated in LLaMAGen [1], which was submitted to ArXiv in June 2024, but we did not try it in our experiments. LLaMAGen is a VQ + AR model that follows the ViTVQGAN model but explores modern llama architecture and CFG. In the table below, we present experimental results from LLaMAGen and compare them with our model DiGIT. LLaMAGen only reports the results of class-conditional generation, but we contend that unsupervised class-unconditional generation is a compelling feature of GPT, and our model excels in it.

As illustrated, our DiGIT model, without CFG, **already** outperforms LLaMAGen with CFG. This outcome highlights the effectiveness of our proposed tokenizer. Additionally, LLaMAGen without CFG still struggles with the VQ tokenizer, as evidenced by significantly worse FID scores even with the larger 3.1B model size, which fails to demonstrate the potential of scaling laws. In contrast, our proposed tokenizer demonstrates significant improvement with increased model size In Table 2 class-unconditional setting, as evidenced by the reduction in FID from 9.13 at 219M parameters to 4.59 at 732M parameters.


| Type | Methods | CFG | Params | Epoch | FID | IS |
|--|--|--|--|--|--|--|
|AR | LLaMAGen | No | 300 | 343M | 13.452 | 82.289 |
|AR | LLaMAGen | Yes | 300 | 343M | 3.805 | 248.280 | 
|AR | **DiGIT (ours)** | **No** | 200 | 219M | **4.79** | 142.87 |
|AR | LLaMAGen | No | 50 | 775M | 19.417 |  66.196 |
|AR | LLaMAGen | Yes | 50 | 775M | 3.391 | 227.081
|AR | LLaMAGen | No | 50 | 3.1B | 13.581 |  87.902 |
|AR | LLaMAGen | Yes | 50 | 3.1B | 3.050 | 222.330 | 
|AR | **DiGIT (ours)** | **No** | 200 | 732M | **3.39** | 205.96 |


[1] Sun, Peize and Jiang, Yi and Chen, Shoufa and Zhang, Shilong and Peng, Bingyue and Luo, Ping and Yuan, Zehuan. Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation. arXiv:2406.06525
Dear Reviewer eGbB,

I hope this message finds you well. We are grateful for your valuable feedback on our submission and are pleased to see your positive score. In our responses, we have addressed the points you raised in detail.

As the discussion period is coming to a close soon, we kindly ask if you could review our responses at your earliest convenience. We are eager to know if our explanations have alleviated your concerns. If there are still areas needing improvement, your insights would be greatly appreciated and instrumental in enhancing our work.

Thank you once again for your thoughtful review and support.

Warm regards, Authors
Dear Reviewer Sn2s,

I hope this message finds you well. The discussion period is ending soon, I am writing to emphasize the importance of your review for our submission. Your score is significantly lower than the other three reviewers, and we believe this discrepancy may indicate a misunderstanding or oversight.

We have addressed all the concerns in our detailed rebuttal and would appreciate your prompt attention to it. A thorough reassessment is crucial to ensure a fair evaluation.

Your expertise is highly valued, and we trust that a reconsidered review will reflect the true merit of our work.

Thank you for your immediate attention to this matter.

Best regards, Authors
Thank you again for the detailed clarification, it helped me to improve my understanding. The two remaining concerns for me are:
- [major] Writing could be improved. Not sure how to fix this in the scope of a rebuttal since one cannot update the manuscript.
- [minor] Will the scores improve with CFG? Have you tried it?
Thank you for the time and effort you have dedicated to reviewing our paper. Your thorough review and insightful suggestions have significantly contributed to improving the quality of our work.
Thank you for your comments. We are happy that we were able to clarify some points and thank you for acknowledging it. We will address each question below

## 1. Definition of Stability in Latent Space

The stability of latent space can be understood from two perspectives:

1. **Stability of Latent Space for Discrete Tokenizers**: This approach, utilized in our paper, can be mathematically expressed as:
   $
   e = \text{Enc}(x + \epsilon), \quad q = \text{Quantizer}(e)
   $
   where $\epsilon \sim \mathcal{N}(0, \sigma)$ and $\sigma$ can be adjusted to vary the signal-to-noise ratio. Here, the Quantizer may refer to the VQ module or K-means module in our method. The stability metrics can be quantified using $\delta \cos(e)$ and $\delta \mathbb{I}(q) $, which correspond to token cosine similarity and token change in Table 1. Our results align with the intuition that SSL models exhibit greater stability against noise, as they capture high-level semantic features of images, unlike traditional autoencoders that focus on low-level appearance features.

2. **Stability of Latent Space for Autoregressive Models**: As the reviewer Vc7B's suggested, we evaluate the stability of latent space for autoregressive models (learnability) by comparing the negative log-likelihood (NLL) loss of models using different tokenizers, thereby demonstrating the nondeterministic nature of the latent space. We monitor the NLL loss values for both the discriminative tokenizer and VQ tokenizer, both using a vocabulary size of 1024 and the same model size: 3.347 for the discriminative tokenizer versus 7.104 for the VQ tokenizer. A lower NLL loss indicates a more stable latent space for autoregressive models. We will include the loss curve comparison in the revised version.

## 2. VAE Objective in the Framework of $\mathcal{D}^H$ and $\mathcal{D}^G$

In **Remark 2.1** in the paper, we discussed that "the latent space $f(X)$ of the VAE is modeled as a tractable Gaussian distribution, and $D^{\mathcal{G}}(P_{f(X)}, P_{X})$ can be zero by setting the generative model as a Gaussian sampler. If the decoder is sufficiently strong and generates samples independently of the encoder output, $D^{\mathcal{H}}(P_{f(X)}, P_{X})$ can also be zero." While the VAEs can generate images by sampling from the prior Gaussian distribution, it significantly lags behind the learned generative model. Consequently, the VAE has largely evolved into a compression model, losing its generation efficacy, particularly in modern latent diffusion technologies.

## 3. Clarification on PCA and AEs in Our Analysis Framework

As emphasized in our rebuttal, the discussion of PCA and AEs serves merely for intuitive understanding of latent space, aiming to make our work more accessible to readers. This part is **NOT** a core contribution of our theoretical framework.

To summarize our contributions succinctly: 
1. A unified perspective on the relationship between latent space and generative models.
2. A novel method to stabilize latent space by disentangling the training processes of the encoder and decoder, leading to a simple yet effective tokenizer.
3. Remarkable performance of our proposed tokenizer in image autoregressive modeling.

We appreciate the response and the discussion and hope we provided information that is helpful to clarifying the points made.
Thank you for the detailed response. There are still some parts which I do not understand:
- What exactly is "stability of latent space"? How do you define it?
- Modern autoencoders are typically trained with a VAE objective, which is a generative objective and takes care of both $\mathcal{D}^H$ and $\mathcal{D}^G$. Or am I missing something?

Several of my concerns have been resolved: regarding the SSL pretraining, regarding the small amount of qualitative samples (please, include more in the earliest revision), regarding the VQGAN baseline, and more or less regarding the guided DiT scores. However, I'm still concerned that the paper is written in a fashion like it's the one proposing the equivalence between PCA and linear AEs and also the overall writing quality could be improved.
Thanks the authors for the response. I will keep my judgement as my concerns are relatively minor and the authors did a good job clarifying.
Dear reviewers, towards the end of the discussion phase, we trust that our response has successfully addressed your inquiries. We look forward to receiving your feedback regarding whether our reply sufficiently resolves any concerns you may have, or if further clarification is needed.