Denoising Autoregressive Transformers for Scalable Text-to-Image Generation

Authors: Jiatao Gu, Yuyang Wang, Yizhe Zhang, Qihang Zhang, Dinghuai Zhang, Navdeep Jaitly, Joshua M. Susskind, Shuangfei Zhai
Primary area: generative models
Keywords: diffusion models, autoregressive models, Transformer

Abstract:
Diffusion models have become the dominant approach for visual generation. They are trained by denoising a Markovian process which gradually adds noise to the input. We argue that the Markovian property limits the model’s ability to fully utilize the generation trajectory, leading to inefficiencies during training and inference. In this paper, we propose DART, a transformer-based model that unifies autoregressive (AR) and diffusion within a non-Markovian framework.  DART iteratively denoises image patches spatially and spectrally using an AR model that has the same architecture as standard language models. DART does not rely on image quantization, which enables more effective image modeling while maintaining flexibility. Furthermore, DART seamlessly trains with both text and image data in a unified model. Our approach demonstrates competitive performance on class-conditioned and text-to-image generation tasks, offering a scalable, efficient alternative to traditional diffusion models. Through this unified framework, DART sets a new benchmark for scalable, high-quality image synthesis.

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1730033676239

thread_type:
official_review

<thread_object>
creator: reviewer_z9qz

<content>
summary:
This paper presents DART, a model that integrates autoregressive and diffusion methods within a non-Markovian framework for efficient, scalable text-to-image generation. DART uses autoregressive denoising to retain the full generation trajectory, improving image coherence and generation quality. Key features include DART-AR for token-level autoregressive modeling and DART-FM for flow-based refinement, both enhancing image fidelity and flexibility. DART beats standard DiT in class-conditioned and text-to-image tasks and adapts well to multimodal generation, providing a unified and scalable approach for high-quality visual synthesis.

strengths:
Overall I find that the writing is clear and well-structured, making it easy for readers to follow the arguments and understand the key points. This paper starts by revisiting key design choices of diffusion models and naturally denoising autoregressive transformer, which is a relatively novel framework aiming to integrate the strength of both diffusion and autoregressive models. Besides, this paper provides plenty of experiments and demos in terms of class/text-conditional image generation.

weaknesses:
* My main concern is that while the paper’s title is Denoising Autoregressive Transformers for **Scalable** Text-to-Image Generation, I am worried about whether the proposed method can truly scale to higher-resolution images. In Line 264, it mentions using  T = 16 , which results in around 4000 tokens for a 256x256 image. So, for resolutions like 512 or 1024, can it really be trained and sampled efficiently using this framework? Although VAR also adopts a similar progressive generation approach, it uses different resolutions in its generation trajectory, whereas DART’s generation trajectory consists of full-sized, noisy images. I would like to see the experiments on higher resolutions (512 and 1024).
* The paper presents many different models (DART, DART-AR, DART-FM, Kaleido-DART), without providing a sufficient explanation of these models, such as their strengths and weaknesses. After reading this paper, I do not know how to choose between them since there is no consistent winner in the provided experiments. I would like to see a table to clearly describe differences, strengths, and weaknesses of each DART variant.
* The paper only compares the DART family with the original DiT on the ImageNet benchmark. However, DiT is originated from a paper two years ago, and there are a bunch of related works with improvements in both diffusion/autoregressive transformers that should be discussed and compared, e.g., SiT [1], VAR [2], MAR [3], etc... It is better to provide comparisons with these sota models in terms of FID scores and inference efficiency.

[1] Ma, Nanye, et al. "Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers." arXiv preprint arXiv:2401.08740 (2024).

[2] Tian, Keyu, et al. "Visual autoregressive modeling: Scalable image generation via next-scale prediction." arXiv preprint arXiv:2404.02905 (2024).

[3] Li, Tianhong, et al. "Autoregressive Image Generation without Vector Quantization." arXiv preprint arXiv:2406.11838 (2024).

questions:
* Is the proposed DART compatible with existing techniques for diffusion models, e.g., DDIM, DPM-Solver, etc. I think it is a bit unfair to directly compare with original DiT with 16/256 steps without using any of these advanced samplers.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730573121949

thread_type:
official_review

<thread_object>
creator: reviewer_7f3i

<content>
summary:
DART unifies autoregressive (AR) and diffusion models within a non-Markovian framework, addressing the limitations of traditional diffusion models' Markovian property.

strengths:
1. DART has well-grounded theoretical framework connecting non-Markovian and Markovian processes.

2. It can handle multiple modalities. 

3. The experiments across different tasks are comprehensive.

weaknesses:
1. While the technical contributions are solid, the paper's narrative structure prioritizes theoretical formulation over an intuitive explanation of the approach. The image is first transformed into several patches in the latent space through a Variational Autoencoder (VAE). The basic version of DART processes these patches simultaneously, predicting their denoised mean values, and then continues denoising in a manner similar to traditional diffusion models. The authors can include a more accessible explanation of how DART works in practice.

2. Even though, I think this paper's results are good, and show generalization ability, I am not sure whether this AR+diffusion method become new trend for future visual generation. Is is better than transfusion, show-o, and mar manner? (I recognize those should be concurrent work. This is just for curiosity, and I will not penalize the paper in terms of scoring).

questions:
see Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730586117293

thread_type:
official_review

<thread_object>
creator: reviewer_afu4

<content>
summary:
The paper, introduces DART, a novel generative model that integrates autoregressive and diffusion approaches within a non-Markovian framework. Key contributions of the work include:
- DART Model: A non-Markovian diffusion model that uses autoregressive techniques to capture the full denoising trajectory, improving generation efficiency and flexibility.
- DART-AR and DART-FM Modules: DART-AR enables token-level autoregressive modeling for enhanced control and quality, while DART-FM employs flow-based refinement to increase model expressiveness.

strengths:
1. The integration of autoregressive and non-Markovian diffusion in a unified framework is innovative, particularly by moving beyond the Markovian assumptions that traditionally constrain diffusion models. DART's AR and FM extensions add novel approaches for token-based and flow-matching refinement in image generation.
2. Experiments are carried out on low-resolution class-conditional and text-to-image generation as well as multimodal generation, demonstrating its effectiveness on different classes.

weaknesses:
1. My biggest concern is the lack of comparison to SOTA approach, either in the form of diffusion models or auto-regressive models, for image generation tasks. The whole paper literally only shows comparison to the baseline DiT model which is published in 2022. There're a flourished set of literature for transformer-based image generation (SD-3[1], PixArt[2], etc.) and multimodal generation (Chameleon[3], Transfusion[4], etc.), yet none of these is compared in the paper. As such, it is really hard to understand the performance and actual benefits this approach brings us.  
2. The efficiency of the model is a big concern. As shown in Fig 7 and Fig. 9(a), although a finer grain modeling by DART-AR could bring a marginal improvement, it would introduce 100x more latency, making it hard to justify whether this efficiency-performance tradeoff is ideal. Scaling up a diffusion model that consume 100x more latency may have better performance. This comparison would be good to show in Fig 7.

[1] "Scaling rectified flow transformers for high-resolution image synthesis." ICML 2024.
[2] "Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis". ICLR2024
[3] "Chameleon: Mixed-Modal Early-Fusion Foundation Models". ArXiv
[4] "Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model". ArXiv

questions:
Please refer to the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730588098947

thread_type:
official_review

<thread_object>
creator: reviewer_xqrw

<content>
summary:
The paper presents DART, al generative model combining diffusion and autoregressive (AR) frameworks for efficient and scalable image synthesis. DART addresses limitations in traditional diffusion models, particularly their Markovian process, which restricts generative efficiency by only leveraging information from the previous denoising step.

strengths:
The idea is interesting, exploring if we can use AR to model the image diffusion and denoising processes to generate image.

1. The technique is well-supported by a theory. 

2. The results are good on border benchmarks. The T2I results seem promising.

weaknesses:
My major concern is that the comparison is not comprehensive. DART integrates diffusion in the AR model, using the AR to model the diffusion. MAR [1] uses diffusion for the AR model, using the diffusion to model the tokenization. Even though their two different approaches, the comparison should include MAR, as MAR has already shown good generalization ability on multimodal generation [2]. It is becoming a trend. With this comparison, we can see if DART has this potential. 

[1] MAR, NeurIPS 2024

[2] Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokens

questions:
1. In the inference stage, does the DART-FM also show the short-step sampling property? How to realize short-step sampling with DART-FM.

2. I still have no idea why DART-AR is better than DART. For me, the whole image processing mechanism should work better. Could the authors give me more detailed explanation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730675193355

thread_type:
official_review

<thread_object>
creator: reviewer_74aa

<content>
summary:
This paper proposes DART, a new method to learn a distribution over images by autoregressively denoising a noise image to obtain high-quality generated images. They claim that DART leverages all the latents within the trajectory in oppose to existing methods that only use the previous latent. They compare their method with DiT and observe improvements in generated contents as evaluated by FID and CLIP Score.

strengths:
The method is scalable and efficient as explained in the paper.

weaknesses:
I think that the comparison to existing work is not comprehensive. Is DiT the only other model that this work should be compared with?

questions:
I want the authors to clarify the comprehension of their evaluation. Also, I'd want to see how the autoregressive generation is the cause of improvements. Can authors run any experiments supporting the fact that autoregressive generation, where attention happens with all previous tokens, has a huge role?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732309916704

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Official Response to All Reviewers

comment:
We thank all reviewers for their comments that help substantially improve the quality of our manuscript. We have updated the manuscript with changes highlighted. We here list the major updates and clarifications in response to the reviews. 

1. First of all, we want to highlight the scope of this work is to build a **novel and potentially more scalable generative framework beyond existing popular autoregressive (AR) models and diffusion models**. In light of this, we propose DART, a Transformer-based generative model that implements non-Markovian diffusion process. DART generates images through iterative refinement like diffusion models. However, DART differs from diffusion model in the way that it allows access to the full denoising trajectory rather than a Markovian process which allows access to one previous state. Also, unlike AR models which usually relies on discrete tokens, our DART directly operate on continuous latents. DART adapts the iterative refinement through denoising which conventional AR lacks. DART showcases its capabilities in class-conditioned image generation, text-to-image generation, multi-resolution generation, and multimodal co-generation. We believe this work paves a way for developing novel generative models beyond AR and diffusion models and may lead to a more unified framework. 

2. We want to further emphasize that our proposed model is different from several recent works on visual generation with autoregressive models. 
    - MAR [1] is an AR model on continuous image tokens. It autoregressively predicts a latent for each patch and applies an additional diffusion model to generate the image tokens conditioned on the predicted latent. It differs from DART from two aspects. (1) MAR applies a different factorization as DART. Namely, MAR only partially denoise one patch at a time, while DART iteratively denoise the whole image which better leverages the global information. (2) MAR relies on an additional diffusion module alongside the AR model to denoise patches. Whereas DART organically integrates the denoising process into AR model through non-Markovian diffusion, which offers a more simplified and unified framework. 
    - VAR [2] is an AR model that works on discrete tokens. It progressively predicts tokens of different resolutions and relies on a curated multi-resolution autoencoder tokenizer trained beforehand. Unlike VAR, our DART works on continuous latent space and doesn’t require curated multi-resolution tokens. Moreover, to generate at different resolutions, VAR requires re-train the autoencoder tokenizer while DART can adapts to higher-resolution generation easily by concatenating denoising steps of higher-resolution images (see Matryoshka-DART in point 4 below).  
    - There are also families of vision-language model (VLM) like Chameleon [3] and Transfusion [4]. Chameleon models both text and image generate as autoregressive next-token prediction. And Transfusion trains a VLM but applying AR for text while applying diffusion for image generation separately. Unlike these models which develop multimodal generation in a brutal way, DART models image generation as autoregressive denoising which can be integrated seamlessly with AR text generation as shown in some preliminary results of Kaleido-DART. Besides, our model doesn’t focus on multimodal generation and just showcase the capability of text-image co-generation using our proposed framework. We believe DART shows the potential of building more unified and efficient multimodal generative models.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes an interesting formulation using a non-Markovian diffusion model that uses autoregressive techniques to capture the full denoising trajectory, improving generation efficiency and flexibility. The paper also proposes DART-AR and DART-FM Modules: DART-AR enables token-level autoregressive modeling for enhanced control and quality, while DART-FM employs flow-based refinement to increase model expressiveness. Experimental results show promise in this formulation.