A Spark of Vision-Language Intelligence: 2-Dimensional Autoregressive
  Transformer for Efficient Finegrained Image Generation
Authors: name: Liang Chen; name: Sinan Tan; name: Zefan Cai; name: Weichu Xie; name: Haozhe Zhao; name: Yichi Zhang; name: Junyang Lin; name: Jinze Bai; name: Tianyu Liu; name: Baobao Chang
Published: 2024-10-02 18:10:05

Abstract: This work tackles the information loss bottleneck of vector-quantization (VQ)
autoregressive image generation by introducing a novel model architecture
called the 2-Dimensional Autoregression (DnD) Transformer. The DnD-Transformer
predicts more codes for an image by introducing a new autoregression direction,
model depth, along with the sequence length direction. Compared to
traditional 1D autoregression and previous work utilizing similar 2D image
decomposition such as RQ-Transformer, the DnD-Transformer is an end-to-end
model that can generate higher quality images with the same backbone model size
and sequence length, opening a new optimization perspective for autoregressive
image generation. Furthermore, our experiments reveal that the
DnD-Transformer's potential extends beyond generating natural images. It can
even generate images with rich text and graphical elements in a self-supervised
manner, demonstrating an understanding of these combined modalities. This has
not been previously demonstrated for popular vision generative models such as
diffusion models, showing a spark of vision-language intelligence when trained
solely on images. Code, datasets and models are open at
https://github.com/chenllliang/DnD-Transformer.