Stabilize the Latent Space for Image Autoregressive Modeling: A Unified
  Perspective
Authors: name: Yongxin Zhu, name: Bocheng Li, name: Hang Zhang, name: Xin Li, name: Linli Xu, name: Lidong Bing

Latent-based image generative models, such as Latent Diffusion Models (LDMs)
and Mask Image Models (MIMs), have achieved notable success in image generation
tasks. These models typically leverage reconstructive autoencoders like VQGAN
or VAE to encode pixels into a more compact latent space and learn the data
distribution in the latent space instead of directly from pixels. However, this
practice raises a pertinent question: Is it truly the optimal choice? In
response, we begin with an intriguing observation: despite sharing the same
latent space, autoregressive models significantly lag behind LDMs and MIMs in
image generation. This finding contrasts sharply with the field of NLP, where
the autoregressive model GPT has established a commanding presence. To address
this discrepancy, we introduce a unified perspective on the relationship
between latent space and generative models, emphasizing the stability of latent
space in image generative modeling. Furthermore, we propose a simple but
effective discrete image tokenizer to stabilize the latent space for image
generative modeling. Experimental results show that image autoregressive
modeling with our tokenizer (DiGIT) benefits both image understanding and image
generation with the next token prediction principle, which is inherently
straightforward for GPT models but challenging for other generative models.
Remarkably, for the first time, a GPT-style autoregressive model for images
outperforms LDMs, which also exhibits substantial improvement akin to GPT when
scaling up model size. Our findings underscore the potential of an optimized
latent space and the integration of discrete tokenization in advancing the
capabilities of image generative models. The code is available at
https://github.com/DAMO-NLP-SG/DiGIT.