DART: Denoising Autoregressive Transformer for Scalable Text-to-Image
  Generation
Authors: name: Jiatao Gu; name: Yuyang Wang; name: Yizhe Zhang; name: Qihang Zhang; name: Dinghuai Zhang; name: Navdeep Jaitly; name: Josh Susskind; name: Shuangfei Zhai
Published: 2024-10-10 17:41:54

Abstract: Diffusion models have become the dominant approach for visual generation.
They are trained by denoising a Markovian process that gradually adds noise to
the input. We argue that the Markovian property limits the models ability to
fully utilize the generation trajectory, leading to inefficiencies during
training and inference. In this paper, we propose DART, a transformer-based
model that unifies autoregressive (AR) and diffusion within a non-Markovian
framework. DART iteratively denoises image patches spatially and spectrally
using an AR model with the same architecture as standard language models. DART
does not rely on image quantization, enabling more effective image modeling
while maintaining flexibility. Furthermore, DART seamlessly trains with both
text and image data in a unified model. Our approach demonstrates competitive
performance on class-conditioned and text-to-image generation tasks, offering a
scalable, efficient alternative to traditional diffusion models. Through this
unified framework, DART sets a new benchmark for scalable, high-quality image
synthesis.