Title: Stabilize the Latent Space for Image Autoregressive Modeling: A Unified Perspective
Venue: NeurIPS 2024
Accepted: yes
Keywords: Latent space

Image sequential modeling

Image tokenizer
Abstract: Latent-based image generative models, such as Latent Diffusion Models (LDMs) and Mask Image Models (MIMs), have achieved notable success in image generation tasks. These models typically leverage reconstructive autoencoders like VQGAN or VAE to encode pixels into a more compact latent space and learn the data distribution in the latent space instead of directly from pixels. However, this practice raises a pertinent question: Is it truly the optimal choice? In response, we begin with an intriguing observation: despite sharing the same latent space, autoregressive models significantly lag behind LDMs and MIMs in image generation. This finding contrasts sharply with the field of NLP, where the autoregressive model GPT has established a commanding presence. To address this discrepancy, we introduce a unified perspective on the relationship between latent space and generative models, emphasizing the stability of latent space in image generative modeling. Furthermore, we propose a simple but effective discrete image tokenizer to stabilize the latent space for image generative modeling by applying K-Means on the latent features of self-supervised learning models. Experimental results show that image autoregressive modeling with our tokenizer (DiGIT) benefits both image understanding and image generation with the next token prediction principle, which is inherently straightforward for GPT models but challenging for other generative models. Remarkably, for the first time, a GPT-style autoregressive model for images outperforms LDMs, which also exhibits substantial improvement akin to GPT when scaling up model size. Our findings underscore the potential of an optimized latent space and the integration of discrete tokenization in advancing the capabilities of image generative models. The code is available at \url{https://github.com/DAMO-NLP-SG/DiGIT}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper designs a better quantized autoencoder on top of VQGAN. It builds an image autoencoder which is able to both achieve good recognition performance for linear probing, and have a latent space which is suitable for training a generative model. It studies the existing autoencoders from a high-level theoretical perspective and proposes design ideas which are targeted to improve them. The paper claims that the modern autoencoders ignore the fact that they will be utilized for downstream generative modelling tasks and mainly focus on reconstruction. The paper argues that adding recognition losses on top of the encoder would help. To fulfill this desiderata, the model takes DINOv2 features and discretizes them via k-means. Then it trains a translation model into VQ-GAN decoder features. For image generation, it trains a LLM in the discrete token space. For classification, it does linear probing on top of discretized DINOv2 features. As a result, it attains reasonable generative capabilities while being able to keep a latent space suitable for linear probing classification.\n\nSTRENGTHS:\n- In terms of the scores, the paper achieves very good results in the sense of discrimination/generation tradeoff (judging by figure 1).\n- It's an interesting finding that one can discretize dino-v2 via K-means and train a strong generative model on top of such tokens.\n- The paper studies an important problem of more rigorous understanding of modern autoencoders\n\nWEAKNESSES:\n- The paper shows an equivalence between a linear AE and PCA, but it's a well known fact: https://arxiv.org/abs/1804.10253. One can also just google \"equivalence between a linear autoencoder and PCA\", and find a ton of resources on that.\n- \"A reconstructive autoencoder does not necessarily establish an advantageous latent space for generative model\". That's a very well-known fact in the community (e.g., see Fig 5 in https://arxiv.org/pdf/2312.02116). The paper should not claim this observation as a novel contribution.\n- The proposed stability metric is interesting, but it's unclear whether it will correlate with downstream generative models performance\n- Proposition 2.4 is extremely vague and seems to be very different from its \"rigorous\" analogue in the supplementary.\n- FID metrics for VQGAN on ImageNet are much higher than in the original paper.\n- It's delusive to compare performance of the developed model vs those trained from scratch, since the developed model starts from strong pre-trained models.\n- For image generation, the paper shows just 16 random samples, which is extremely little to get a good understanding of the model. It's better to show much more (e.g. stylegan2 provides 100k samples for FFHQ: https://github.com/NVlabs/stylegan2).\n- Why DiT-XL/2 is included for comparison but not its guided variants? Why more recent papers are not included for comparison? (e.g., EDMv2).\n- The logical transitions in the paper are unclear, e.g., it's unclear why the proposed training improves D^G, it's unclear, why it follows from the propositions that we should improve the stability of the latent space (where stability is also not defined well), etc.\n\nQUESTIONS:\n- The transition \"Drawing from these theoretical insights, we introduce a metric to assess the stability of the latent space induced by different encoder models\" on L196 is extremely unclear. How exactly do theoretical results suggest that one should focus on stability of the latent space? Why would LDA lead to a better generative model? Why \"separating\" distributions class-wise would lead to a better generative model? What exactly do you mathematically define \"separation of distributions\" for an encoder?\n- Is linear probing done on top of discretized or non-discretized DINOv2 features?\n\nLIMITATIONS:\nOne limitation that is not explored is whether the model is upper-bounded by the performance of the underlying SSL classifier. In other words, what would be the greater source of improved performance in the future — improving SSL or decoder?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nLatent-based image generative models, such as LDMs and MIMs, have achieved success, but autoregressive models lag behind in image generation. Our research introduces a unified perspective on latent space stability and proposes a discrete image tokenizer, DiGIT, that significantly improves autoregressive image modeling, outperforming LDMs and benefiting from scaling similar to GPT in NLP.\n\nSTRENGTHS:\n- The results beat some baseline models, though under a specific (and somewhat confused) experimental setting.\n- The topic of latent space property is worth investigating.\n\nWEAKNESSES:\nThe paper has several weaknesses:\n\n1. **Factual Errors**:\n\n    1.1. The cited MIM models, such as MaskGIT and MAGE, cannot revise previously predicted tokens. This contradicts the claim in line 53 that \"iterative models like LDMs and MIMs can correct errors.\" I recommend the authors to their papers for more details.\n\n    1.2. In lines 72-73, the authors state that this work provides \"the first evidence that image autoregressive generative models behave analogously to GPT.\" However, the Parti[1] paper has already demonstrated that image autoregressive models have similar scalability to GPT and successfully scaled the model to 20B. The authors have not cited this work.\n\n2. The writing is poor and lacks rigor. For example, the discussion on the so-called \"common misconception\" in line 41 is not well-supported. What exactly is meant by the \"optimal latent space for reconstruction\"? How many studies hold this view? There are no citations provided.\n\n3. The quantitative comparisons are also peculiar. The authors cite many paper results without using CFG, while CFG has become a de-facto practice for augmenting generative models. Why not adopt CFG and perform more apples-to-apples comparisons to other SOTA methods with CFG?\n\n4. Presenting two tables (Table 2 lower and Table 3) for image generation performance is confusing. Why not consolidate the results into a single, clear table?\n\n[1] Yu, Jiahui, et al. \"Scaling autoregressive models for content-rich text-to-image generation.\" arXiv preprint arXiv:2206.10789 2.3 (2022): 5.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nThe writing & presentation of this paper seems too rush and lacks rigor. I recommend the authors to refine and polish this paper. The current draft may not be qualified for the publication of NeurIPS.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tries to understand why latent autoregressive image models perform worse than latent diffusion models. The key insight is that existing tokenizers are trained primarily with the reconstruction objective, whose latent space is unstable and thus may not be easy to model autoregressively. To solve this issue, the authors propose first learning a stable latent space, which autoregressive models can model easily, and then learning to reconstruct pixels from this latent space. Experimental results show that this modification enables latent autoregressive image models to match latent diffusion models' performance in terms of image understanding and image generation.\n\nSTRENGTHS:\n1. The paper proposed a new perspective—latent space stability—on understanding latent autoregressive image models, which was neglected in previous works. I think this explanation is intuitive since a fixed-depth autoregressive model may not be able to model very noisy distributions (e.g., the language data have high regularity)\n2. The proposed solution is straightforward -- just let image features with similar semantics share the same token.\n3. The experiments are comprehensive and interesting. Both image understanding and image generation are evaluated; improvements over previous latent autoregressive models are significant. The ablation study also makes sense to me.\n\nWEAKNESSES:\n1. I think there is a tension between how stable the latent space is and how easily we can reconstruct the latent codes to pixels. The impact of the proposed method on reconstruction is not elaborated in this paper. For example, if we only care about reconstruction, how badly does the proposed method perform? This matters greatly if we are modeling high-resolution images and care about the visual details.\n2. The theoretical analysis and the proposed algorithm seem loosely connected to me -- I don't see the proposed algorithm as a direct result of the theoretical analysis. The stability analysis is more straightforward, though.\n\nQUESTIONS:\nHow negatively does the proposed method impact reconstruction?\n\nLIMITATIONS:\nI think the authors adequately addressed the limitations", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper propose to disentangle the encoder and decoder learning for image tokenzier which ultimately will be used for providing the latent space of AR generative model. In particular, SSL model such as DinoV2 is used for encoder (plus k-means clustering).\n\nSTRENGTHS:\n1. The idea of disentangling the encoder and decoder learning for image tokenizier is interesting and novel.\n\n2. Strong empirical results can be obtained from the method. The fact that by changing a tokenizer and training the same AR model, FID can be halved to half is really impressive.\n\nWEAKNESSES:\n1. The motivation for adopting self-supervised model as encoder/tokenizer is not very clear. Since the method is easy (DinoV2 + kmeans), the motivation of why doing so is the most critical part of the paper. However, I don't think this is presented very clearly and explicitly. Large improvements of the presentation is needed. \n\n2. The term \"stability\" or \"\"stablize\" is a bit confusing. Explicit explanation is needed. When is a latent space not stable? If it means hard to learn an AR model, probably a better term such as learnability is better. \n\n3. While the argument of \"iterative decoding process can stabilize the sampling process by correcting the data falling in the low-density overlap between distributions\" makes sense, it still requires justification and evidence, not just conceptual analysis.\n\n4. If you use SSL model as encoder, you need to train a decoder. Not much explicit detail is presented for this part.\n\n5. The metric is not very clearly defined. What's the name of the metric? What is the definition? How to compute it? All these information should be highlighted.\n\nOverall the presentation and organization is not very clear, some major rewrite is needed.\n\nQUESTIONS:\n1. In section 2.2, you mentioned that a drawback of auto-regressive image modeling is that each iteration only generate a patch so error in the previous generated patch will accumulate. How is this related to your method? IIUC, your tokenizer is still patch based, so it does not resolve the issue mentioned here.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}