Denoising Autoregressive Transformers for Scalable Text-to-Image Generation
Authors: Jiatao Gu
Yuyang Wang
Yizhe Zhang
Qihang Zhang
Dinghuai Zhang
Navdeep Jaitly
Joshua M. Susskind
Shuangfei Zhai
Venue: ICLR 2025 Poster (2025)
Keywords: diffusion models
autoregressive models
Transformer
Primary area: generative models
TL;DR: DART unifies autoregressive and diffusion models for efficient, high-quality image generation.

Abstract: Diffusion models have become the dominant approach for visual generation. They are trained by denoising a Markovian process which gradually adds noise to the input. We argue that the Markovian property limits the model’s ability to fully utilize the generation trajectory, leading to inefficiencies during training and inference. In this paper, we propose DART, a transformer-based model that unifies autoregressive (AR) and diffusion within a non-Markovian framework.  DART iteratively denoises image patches spatially and spectrally using an AR model that has the same architecture as standard language models. DART does not rely on image quantization, which enables more effective image modeling while maintaining flexibility. Furthermore, DART seamlessly trains with both text and image data in a unified model. Our approach demonstrates competitive performance on class-conditioned and text-to-image generation tasks, offering a scalable, efficient alternative to traditional diffusion models. Through this unified framework, DART sets a new benchmark for scalable, high-quality image synthesis.

Published as a conference paper at ICLR 2025
DENOISING AUTOREGRESSIVE TRANSFORMERS FOR SCALABLE TEXT-TO-IMAG GENERATION
Jiatao Gul, Yuyang Wang', Yizhe Zhang', Qihang Zhang"; Dinghuai Zhang"*, Navdeep Jaitly', Josh Susskind', Shuangfei Zhail Apple, "hhe Chinese University of Hong Kong, "Mila
(igg33,, yuyangw, yizzhang, njaitly, jsusskind, szhai @apple. com "ghzhang@ 1 i ink. cuhk. edu. hk 7dinghuai. zhang@mi la quebec
DART
GENA
Figure 1: Curated examples of images generated by DART 256 512 and 1024 pixels.
ABSTRACT
Diffusion models have become the dominant approach for visual generation. They are trained by denoising a Markovian process which gradually adds noise the input. We argue that the Markovian property limits the model's ability fully utilize the generation trajectory, leading to inefficiencies during training and inference. In this paper, propose DART, transformer-basee based model that unifies autoregressive (AR) and diffusion within a non-Markovian framework. DART iteratively denoises image patches spatially and spectrally using AR model that has the same architecture standard language models. DART does not rely on image quantization, which enables more effective image modeling while maintaining flexibility. Furthermore, DART seamlessly trains with both and image data unified model. Our approach demonstrates competitive performance on class-conditioned and text-to-image generation tasks, offering a scalable, efficient alternative to traditional diffusion models. Through this unified framework, DART sets new benchmark for scalable, high-quality image synthesis.
INTRODUCTION
Recent advancements in deep generative models have led to significant breakthroughs in visual synthesis, with diffusion models emerging as the dominant approach for generating high-quality images (Rombach et al., 2022: Esser et al., 2024). Diffussin models (Sohl-Dickstein et al., 2015; Ho et al., 2020) operate by progressively adding Gaussian noise to an image and learning to reverse this process in sequence of denoising steps. Despite their success, these models difficult to
Work done part of internship Apple.

---

Published as a conference paper at ICLR 2025
Image
Output Module
"Golden Retriever"
FFN Black with GLG
Context
Denoising Auto-Regressive Transformer (DART)
croee--attettion (optional)
Encoder
+
Soff anff-attetsion Block with KV Cashe
Imput Module
Figure 2: (+) A general illustration of the proposed DART. The model autoregressively denoises image through a Transformer until clean image is generated. Here, whole images are shown for visualization purpose; (-+) We show the architecture details which integrates state-of-the-an designs similar to common language models (Dubey et al., 2024).
train on high resolution images directly, requiring either cascaded models (Ho et al., 2022), or multiscale approaches (Gu al., 2023) or preprocessing of images autoencoder codes at lower resolutions (Rombach et al., 2022). These limitations can stem from their reliance on the Markovian assumption, which simplifies the generative process but restricts model only to see the generation from the previous step. This often leads to inefficiencies during training and inference, the individual steps of denoising are unaware of the trajectory of generations from prior steps.
In parallel, autoregressive models, such GPT-4 (Achiam et al., 2023), have shown great success in modeling long-rang dependencies sequential data, particularly in the field of natural language processing. These models efficiently cache computations and manage dependencies across time steps, which has also inspired research into adapting autoregressive models for image generation. However, early efforts such PixelCNN (Oord et al., 2016), while promising, suffered from high computational costs due to pixel-wise generation. More recent models like VQ-GAN (Esser et al., 2021a) and related work (Yu al., 2022: Team, 2024: Tian al., 2024) learn models of quantized images in a compressed latent space: Li et al. (2024) propose to generate directly in such space without quantization by employing diffusion- based loss function. However, these methods fail to fully leverage the progressive denoising benefits of diffusion models, resulting in limited global context and error propagation during generation.
To address these limitations, we propose Denoising AutoRegees Transformer (DART), novel generative model that integrates autoregressive modeling within a non- Markovian diffusion frame- work (Song et al., 2021) (Fig. 2). The non-Markovian formulation in DART enables the model to leverage the full generative trajectory during training and inference, while retaining the progressive modeling benefits of diffusion models, resulting more efficient and flexible generation compared to traditional diffusion and autoregressive approaches. Additionally, DART introduces two key improvements to address the limitations of the Mar-Markov approach: (1) token-level autoregres- sive modeling (DART-AR), which captures dependencies between image tokens autoregressively, enabling finer control and improved generation quality, and (2) flow-based refinement module (DART-FM), which enhances the model's expressiveness and smooths transitions between denoising steps. These extensions make DART flexible and efficient framework capable of handling wide range of tasks, including class conditional, text-to-image, as well as multimodal generation.
DART offers a scalable, efficient alternative to traditional diffusion models, achieving competitive performance on standard benchmarks for class-conditioned (e.g., ImageNet (Deng et al., 2009)) and text-to-image generation. To summarize, major contributions of work include:
We propose DART, novel non-Markovian diffusion model that leverages the full denoising trajec- tory, leading to more efficient and flexible image generation compared to traditional approaches.
We propose two key improvements: DART-AR and DART-FM, which improve the expressiveness and coherence throughout the Markovann generation process.
DART achieves competitive performance in both class-conditioned and text-to-image generation tasks, offering scalable and unified approach for high-quality, controllable image synthesis.
2

---

Published as a conference paper at ICLR 2025
2 BACKGROUND
2.1 DIFFUSION MODELS
Diffusion models (Sohl- Dich--Dickstein et al., 2015; Ho et al., 2020; Song al., 2020) are latent variable models given a pre-defined posterior distribution, and trained with a denoising objective. These models have gained widespread use in image generation (Rombach et al., 2021; Peebles & Xie, 2022; Podell et al., 2023; Esser et al., 2024). Diffusion models produce the entire image in a non- autoregressive manner through iterative processes. Specifically, given an image æo RR*H* we define a series of latent variables (t = I.... T) with a Markovian process which gradually adds noise the original image Zo- The transition ( and the marginal ( ( probabílities are defined as follows, respectively:
((20s N( v 1,T)) ((22)
(a Varao, -+) I),
(1)
where ār = II= 3, < Be are determined by the noise schedule. The model learns to reverse this process with a backward model po (2 which aims to denoise the image. The training objective for the model
min Co
( Jay | t) xollé].
(2)
where ao (Za, 1) is sime-conditioned denoiser that learns to map the noisy sample to its clean version zo; We a time-dependent loss weighting. which usually uses SNR (Ho et al., 2020)
or SNR+1 (Salimans & Ho, 2022). Practically, 2, can be re-parameterized with noise- prediction (Salimans & Ho, 2022) for enhanced performance, and can be applied on pixel space (Gu al., 2023; Saharia et al., 2022) or latent space, encoded by VAE encoder (Rombach et al., 2021). However, standard diffusion models are computationally inefficient, requiring numerous denoising steps and extensive training data. Moreover, they lack the ability leverage generation context effectively, hindering scalability to complex scenes and long sequences like videos.
2.2 AUTOREGRESSIVE MODELS
In the field of natural language processing. Transformer models have achieved notable success in autoregressive modeling (Vaswani et al., 2017; Raffel et al., 2020). Building on this success, similar approaches have been applied image generation (Parmar et al., 2018; Esser et al., 2021a; Chen et al., 2020: Yu et al., 2022; Sun et al., 2024: Team, 2024). Different from diffusion- based methods, these methods typically focus on learning the dependencies among discrete image tokens (e.g., through Vector Quantization ( (Van Den Oord et al., 2017)). To elaborate, consider an image E IR H The process begins by encoding this image into sequence of discrete tokens 21:N = {(x). These tokens are designed to approximately reconstruct the original image through learned decoder = (2 An autoregressive model is then trained by maximizing the cross-entropy as follows:
max CCE
log (2(zu|zunn-1
n=1
where zo is the special start token. During the inference phase, the autoregressive model is first used to sample tokens from the learned distribution, and then decode them into image space using D.
As discussed in Kilian al. (2024), autoregressive models offer significant efficiency advantages over diffuusin models by caching previous steps in memory and enabling the entire generation process to be computed in single parallel forward pass. This reduces computational overhead and accelerates training and inference. However, the reliance on quantization can lead to information loss, potentially degrading generation quality. Additionally, the linear, step-by-step nature of token prediction may overlook the global structure, making it challenging to capture long-range dependencies and holistic coherence complex scenes or sequences.
DART
3.1 NON-MARKOVIAN DIFFUSION FORMULATION
We start by revisiting the basics diffusion models from the perspective of hierarchical variational auto-encoders (HVAEs, Kingma& & Welling, 2013; Child, 2021). Given data-point æt, a hierarchical
3

---

Published as a conference paper at ICLR 2025
VAE models generation process pe of sequence of latent variables æz((t = T... 1)' by maximizing a evidence lower bound (ELBO):
(o( log qo (2) (1) 1200)
LELBO max 8.6
Buirrras (x))
log po ( + log po log
(4)
where Zy is the real data, and 4s is learnable inference model. As pointed out in VDM (Kingma et al., 2021), diffusion models can essentially be seen as HVAEs with three specific modifications:
1. A fixed inference process q which gradually adds noise to corrupt data Zo:
2. Markovian forward and backward process where 2; depends only on 2++1 (Eq (1));
3. Noise-dependent loss weighting that reweighs ELBO with a focus on perception.
Only with all above simplifications combined, standard diffusion models can be formulated as Eq (2) where the generator becomes Markovian pe (xt-1 ( = Pe (#+=1 so that one can randomly sample t to learn each transition independently. This greatly simplifies modeling, enabling training models with sufficiently large number of steps (e.g.. 1000 for original DDPM (Ho et al., 2020)) without suffering from memory issues.
In prior research, these aspects are highly coupled, and few works attempt to disentangle them. We speculate that the Markovian assumption might not be a necessary requirement for a high generation quality, as long as the fixed posterior distribution and flexible loss weightings are maintained, As side evidence, one can achieve reasonable generation with much fewer steps (e.g., 100) inference time using non-Markovian HVAE. On the contrary, the Markovian modeling forces all information compressed solely in the corrupted data from previous noise level which could be an obstacle preventing efficient learning and require more inference steps.
MOn-MArkovian Diffusion Models (NOMAD) In this paper, we reconsider the original form of generator Po - ( of HVAEs while maintaining the modifications, 1 and 3, made by diffusion models. More precisely, we learn the following weighted ELBO loss (Eq (4)) with adjustable Wy:
T
max L NOMAD (((o) We log (
where is the pre-defined inference process. This formulation shares many similarities autoregres- sive (AR) models in Eq (3), where in our case, each token represents a noisy sample 2;. Therefore, natural to implement such process with autoregressive Transformers (Vaswani et al., 2017).
However, the Markovian inference process of standard diffusion models makes impossible for the generator 0 to use the entire context except for æe, That is say, even if initiate our generator as ( [. the model only needs information in a, in order to best denoise 0 Therefore, critical to design a fixed and non-Markovian' inference process (30 (20:--1) 1) to sample the noisy sequences 11.7 ~ Zp. The simplest approach is to perform an independent noising process:
q( ( [0: 1) = (=0))=o) = A((:;; Viiao, (1 I)) Vt E [1,7).
(6)
where "i represents the non-Markovian noise schedule. Note that, while Eq (6) may look close the marginal distribution of the original diffusion models (Eq (I)), the underlying meaning is different I, is conditionally independent given Eg. In practice, one also choose more complex non non-Markovian process as presented in DDIM (Song et al., 2021), and we leave this exploration in future work. Additionally, we can show the following proposition:
Proposition 1. A non-Markovian diffusion process {at)): with independent noising T/t has a bijection to Markovian diffusion process [y:): with the same number of steps and noise level {0r)6 {yche achieves the maximal signal-to-noise ratio when its noise level satisfies
6r T-a-
-
We defer the proof to Appendix A. The above proposition indicates that we can carefully choose a set of noise level 7 model the same coommation-destroiing process as a comparable diffusion baseline, while modeling with the entire generation trajectory.
Wee follow the same notation of time indexing in diffusion models for consistency. "The term non-Marte Markovian refers to x not only related the previous step #t-1-
4

---

Published as a conference paper at ICLR 2025
0.200
DART
DART-AR
0.33
0.500
1055
0.185
0.175
90
100
150
200
250
350
Thaining Ioar Itaratiorn
(a)
(b)
Figure 3: Attention masks for (a) DART and (b) DART-AR, highlighting their different structures.
(c) Comparison of learning curves, demonstrating the superior performance of DART-AR.
3.2 PROPOSED METHODS
Denoising AugoRegres Transformer (DART) We propose DART a Transformer- generative model that implements non Mor-Mvakovi diffusion with an independent noising process (see Figure 2). First, following DiT (Peebles Xie, 2022), we represent each image by first extracting the latent map with pretrained VAE (Rombach et al., 2021), patchify, and flatten the map into sequence of continuous tokens 2, E RR xC where K the length, and
the channel dimension. concat tokens along the length dimension. Then, DART
When considering multiple noise levels, models the generation as po (- ( = N (x ( ), (1-7-)))), where xz(() is Transformer network that takes in the concatenated sequence oer E B (T) and predicts the "mean" of the next noisy image. By combining with Eq (6), we simplify Eq (5) as:
7-))
T
min CP P
CAART
We (e - xol1
where we define We Te-1 Be to simplify the notation. Similar to standard AR models, training of T denoising steps is in parallel, where computations across different steps are shared. A chunk-based causal mask is used to maintain the autoregressive structure (see Figure 3(a)).
1-7--1
It is evident from Eq (7) that the objective is similar to the original diffusion objective (Eq (2)), demonstrating that DART can be trained as robustly as standard diffusion models. Additionally, by leveraging the diffusion trajectory within an autoregressive framework, DART allows us to incorporate the proven design principles of large language models (Brown et al., 2020; Dubey et al., 2024). Furthermore, Proposition I indicates that we select wa according to its associated diffusion process. For instance, with SNR weighting (Ho et al., 2020), wi can be defined as w/ Σ 1-7+
T
Sampling from DART is straightforward: we simply predict the mean we (R..), add Gaussian noise to obtain the next step ø;-1, and feed that the following iteration. Unlike diffusion models, no complex solvers are needed. Similar to diffusion models, classifier-fre guidance (CFG, Ho & Salimans, 2021) is applied to the prediction of wy for improved visual quality. Additionally, KV-cache is employed to enhance decoding efficiency.
Limitations of Naive DART Unlike diffusion models, which can be trained with a large number of steps T, non Mon-vvakovi modeling is constrained by memory consumption as T increases. For example, using T 16 on 256 256 images will easily create over 4000 tokens even in the latent space. This fundamentally limits the modeling capacity complex tasks such as text-to-image generation. However, the flexibility of the autoregressive structure allows us to enhance the capacity without compromising scalability. In this work, we propose two methods on top of DART:
1. DART with Token Autoregressive (DART-AR) As similarly discussed by Xiao et al. (2021),
the independent Gaussian assumption of po (z,-1 (xTT)) ) inaccurate to approximate the complex true
5

---

Published as a conference paper at ICLR 2025
Algorithm DART-FM
I: +- (V---i (Zeer) 7-
cu(murr). 7
DART
FlowNet
3: while
4:
2, uu( A-
6: end while
Xp-1
7: #b-1 2z
Figure 4: An illustration for the generation process of DART-FM.
"Golden Retriever*
A puppy looking camera. 

Context Encoder
DART (Text-Image Joint Modeling)
puppy looking the camera.
next takan prediction (sptiona)
chanolsing
Figure 5: Illustrations of Kaleido-DARR an application for multi-modal generation.
distribution of q(<,-1/<x.)) especially when I' is small. A straightforward solution to model image
denoising additional autoregressive model po ( I po( ول 1 T.:r):
T
K
min C9
CARRTAAR E (g((o)
[ x..))
2 112
(8)
where 2.-11 -.............. - are the K flatten tokens of ΦT-1. The autoregressive decomposition ensures each tokens are not independent, which is strictly stronger than the original DART. We demonstrate this by visualizing the training curve in Figure (c). Training of DART-AR takes essentially the same amount of computation standard DART with two additional modifications at the input and attention masks (see comparison Figure (a) (b)). At sampling time, DART-AR is relatively much more expensive it requires K T AR steps before it outputs the final prediction.
2. DART with Flow Matching (DART-FM) The above approach models dependency across tokens, while maintaining Gaussian modeling each step. Alternatively, can improve the expressiveness of pa( (It-1 by abandoning the Gaussian assumption, similar to Li et al. (2024). More precisely, we first sample Φ; as regular DART. Then, we recursively apply a continuous flow network, U. (Z,, C,, Tr), over multiple iterations to bridge the gap between ž, and æ;-1 (see Figure 4). Here, Uo (ā,, Cr, T) models the velocity field for the probabality flow between the distributions of ãt and 2, E [0, 1] denotes the auxiliary flow timestep, and features from the last Transformer block, providing contextual information across the noisy image. Consequently, a simple MLP suffices model Vg. adding only minimal overhead to the total training cost. We train Ds via flow matching (Liu et al., 2022; Lipman et al., 2023; Albergo et al., 2023) due its simplicity:
co co(#:r) represents the
min 6000 Ex E (2o)
E E 11 T) T3,-1, C. (x,-1 2.)1
(9)
where a; = ISG (x(T)) 7-16 and c ~ (N. SG is the stop-gradient operator avoid trivial solutions optimization. In practice, we combine Eq (9) with the original DART objectives, which can be seen as an additional refinement on top of the Gaussian-based prediction.
3.3 MULTI-MODAL GENERATION
Our proposed framework, built on an autoregressive model, naturally extends to discrete token modeling tasks. This includes discrete latent modeling for image generation (Gu al., 2024) and multi-moda generation (Team, 2024). By leveraging a shared architecture, we jointly optimize for both continuous denoising and next-token prediction loss using cross-entropy for discrete latents

---

Published as a conference paper at ICLR 2025
which we call Kaleido DART considering its architectual similarity as Gu et al. (2024) (see Figure 5). To balance inference across modalities, we reweight the discrete loss (Eq (3)) according to the relative lengths between the discrete and image tokens:
Lo [kaleido ACS CB CDART
(10)
where À text tokens
It is important to note that our approach is markedly distinct from several concurrent works aimed at unifying autoregressive and diffusion models within a single parameter space (Zhou et al., 2024; Xie et al., 2024; Zhao et al., 2024; Xiao et al., 2024). In these efforts, the primary goal is to adapt language model architectures to perform diffusion tasks, without modifying the underlying diffusion process itself to account for the shift in model design. As have discussed, the Markovian nature of diffusion models inherently limits their ability to leverage generation history, a feature that lies at the core of autoregressive models. In contrast, DART is designed to merge the advantages of both autoregressive and diffusion frameworks, fully exploiting the autoregressive capabilities. This formulation also allows for seamless integration into LLM pipelines.
# image tokens"
EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Dataset We experiment with DART on both class-conditioned image generation on ImageNet (Deng et al., 2009) and texi-to-image generation on CC12M (Changpinyo et al., 2021), where each image is accompanied by descriptive caption. All models are trained to synthesize images at 256 256. For multimodal generation tasks, we augment CC12M with synthetic captions as additional ground-trum
Evaluation In line with prior works, we report Fréchet Inception Distance (FID) (Heusel et al., 2017) to quantify the the realism and diversity of generated images. For text-to- generation, also use the CLIP score (Hessel et al., 2021) to measure how well the generated images align with the given text instructions. To assess the zero-shot capabilities of the models, we report scores based on the MSCOCO 2017 (Lin et al., 2014) validation set.
Architecture Following 3.2, we experimented with three variants of the proposed model: the default DART, along with two enhanced versions. DART-AR and DART-FM. All variants are implemented using the same Transformer blocks for consistency. As illustrated in Figure 2, our design is similar to Dubey et al. (2024), incorporating rotary positional encodings (RoPE, Su et al., 2024) within the self-attention layers and SwiGL.U activation (Shazeer, 2020) in the FFN layers. For class-conditioned generation, we follow (Peebles & Xie, 2022), adding an AdaLN block to each Transformer block to integrate class-label information. For text-to-image generation, we replace AdaLN with additional cross-attention layers over pretrained TL-XL encoder (Raffel et al., 2020). Since DART uses fixed noise schedule, there is need for extra time embeddings as long as RoPE is active. In addition, for DART-FM, we incorporate a small flow network, implemented as a 3-layer MLP, which increases the total parameter count by only about 1%
Training Proposition I not only sets a connection between DART and standard diffusion models, but also allows us to define the noise schedule based on any existing diffusion schedule {ārh. which can be inversely mapped to the DART schedule {ye) using the bijection. In this paper, we adopt the cosine schedule ā; = cos ( We set T = 16 while K = 256 throughout all experiments unless otherwise specified. We train all models with a batch size of 128 images, resulting in a total of 0.5M image tokens per update. We use the Adam W optimizer (Loshchilov & Hutter, 2017) with cosine learning rate schedule, setting the maximum learning rate to 3e-4.
4.2 RESULTS
Class-conditioned Generation We report the FIID scores for conditional ImageNet generation in Figure 7(a). following the approach of previous works. In line with Li et al. (2024), we apply linear CFG scheduler to DART and its variants. As shown in the figure, both -AR -FM variants consistently outperform the default DART across all guidance scales, demonstrating the effectiveness
3 K = 256 from encoding 256 256 images with StahleDiffusion v1.4 VAE co/ sttbbiiii 1 ityai/ / vae- ft ema) with patch size of and C = 16 channels.
htthhhgggngffcc....
7

---

Published as a conference paper at ICLR 2025
(4) Random samples DART-AR trained on Clus-conditioned Image Generation
CCR
human depicted entirely out froctale
This dreamilke digital captunes a vibrant, kaledoccopic bird with fire in the background
a flufyy red panda reading newepapor with ICLR writton on
cat rising actopus through the clouds in the cky
blue standing
A traditional Chinese garden with impressionist painting
An armchai- the shape of
pig all wet by heavy
basket
END
desert landscape on an tllen
T-shirt
hippo compased entirely of thick, flowing caramel, with sticky
Detaied pen and drewing of happy beer lady selling flowers in her shop.
planet, giant pacti, and a gss giant low the horizon
drips oazing from its body. Parts
the caramel stretch and melt.
(b) Selected samples of DART-FM trained Text-to-Image Generution
Figure (a) Uncurated samples generated hy class-conditioned image DART trained on ImageNet. (b). Selected samples generated by text-to-image DART trained on CC12M.
of the proposed improvement strategies. We also compare our methods with DiT (Peebles & Xie, 2022), using both 16 sampling steps (to match DART) and 250 steps (the suggested best setting). Notably, DART-AR achieves the best FID score of 3.98 among all variants and significantly surpasses DiT when using steps, highlighting its advantage leveraging generative trajectories, particularly when the number of sampling timesteps is limited. Not surprisingly. DiT with 250 steps performs better than DART. However, is important to note that the official DiT model trained for 7M steps, which is substantially more training iterations than those used for DART.
Figure presents examples of generated results on ImageNet from all models using 16 sampling steps. DART-FM tends to produce sharper images with higher fidelity at higher CFG values. In contrast, DART-AR demonstrates an ability to generate more realistic samples at lower CFG values when compared to both the baselines and other variants.
8

---

Published as a conference paper at ICLR 2025
CART
DUAT
TIGATAL
TOAT
2
23
DART
53
12
1.6
2.0
1/0
1.25 1.00
1.31
2.50
2.75 100
1.0
1.25 1.50
1.75 2/00
201
2:50 2.75 a.00
Guidance weight
Guidtince weight
Guisancee weght
(a) FIDSOK on ImageNet
(b) FID30K on COCO
(c) CLIP score on COCO
Figure 7: Comparison of DART, DART-AR, DART-FM and baseline models with different CFG guidance scale on different benchmarks. denotes models implemented trained by us.
CIG-1 L7G-2
CIG
CIG-1
B3G-2
CIG-5
60 D&T (Pablla & Xis, 2022) - 16 sttpps
(b) DART (Oun)
(Ckm)
Figure 8: Sample comparison of DARTs and baseline DiT models with different CFG gudidance scale on ImageNet with the class label golden retriever.
Text-to-Image Generation To demonstrate the capability of DART at scale, we train the model for text-to-image generation. We also implement an in in-house DiT with cross attention to text condition in comparison to our models, where we evaluate the performance on both 16 and 250 steps. As shown in Figure 7, while both -AR and -FM variants still show clear improvements against the default DART, FM achieves the best FID of 11.12, indicating its ability of handling diverse generation tasks.
Efficiency We compare both the actual inference speed (measured by wall-clock time with batch size 32 on a single H100) as well as the thcoretical computation (measured by GFlops) in 9()). Since DART, DART-AR, DART-FM share the same encoder-decoder Transformer architecture, their flops are roughly the same. However, DART-AR has high wall clock inference time due to its large autoregressive steps, which have not been well parallelized in current implementation. Integrated with recent advances in autoregressive LLMs, DART-AR can be deployed in a more efficient and we leave this for future investigations. DART-FM also have inference time overhead due to the iterations of flow net in inference. Compared to DART, DiT has less flops for a single pass. However, requires sufficient many number of iterations to generate high quality samples. DART has comparable flops and inference time as DiT with 16 sampling timesteps, while DART achieves better performance than DiT (16) on ImageNet and COCO, showing the efficiency benefits of DART.
Scalibility We show the scalability of our DART by training models of different sizes including small (S), base (B), large (L), and extra large (XL) on CC12M, where the configurations are listed in Appendix B.I. Figure 9b) illustrates how the performance changes as model size increase. Across all the four models. CLIP score significantly improves by increasing the number of parameters in DART. Besides, the perform steadily increases as more training iterations are applicd. This demonstrates that proposed generative paradigm benefits from scaling previous generative models like diffusion (Peebles& & Xie, 2022) and autoregressive models (Li et al., 2024).
Effects of noise levels. We experiment with different noise levels, where we vary the number of total noise levels in our denoising framework (Figure 9c)). A total of T 4 or noise levels

---

Published as a conference paper at ICLR 2025
53
DARTAR (1:14)
Model
[lop((10))
time(s)
43
DARTAX (Tw4)
DART
2.157
2.4
43
DATTLL
DAVE XL (Seftee))
(T-4)
35
DART-AR
2.112
238.1
DART-FM
2.177
14.4
DET (16)
2.073
2.14
DIT (250)
32.384
26.38
120 125
175
750
12
1.6
1.8
32
244
Poining Reradions (K]
Guissince weight
(a) Inference speed & flops
(b) Comparison of model sizes
(c) Comparison of steps
Figure 9: (a) Inference flops and wall clock time of different models. (b) Performance of DART of different
sizes. (c) Effect number of noise levels on DART.
Agudden
13 comee mumkin looding face carved ioo and paeto bodogrourd la whito and lpriina teggn compooi pentened
griken seeee ang
andirg brightty
and and
Ageidon HEROVV weesin and Danten 11 Fa
weiting sedlice tackgmmnn out reess dirrinne The lighing neger ai rosoppiene edio
and the sumpain, with papee trme The ovendll poans cudee Hallowoon
ioou onl wopling a hot.
Bil magas there to onl cnd over uip ned and white The becloyonn caoe gaay The The fotl mus nat tano around the rild beak
Ii the mage, in us evening in
amrall aud Dewaing: black
la depicaddi black and att
perchied the persorts haand
hal clrss be drape and tas of the ret the book I'n
reciggrann of koreet perior
burigronnd plan white, vaite. otpade
black
den The cadtt wesher ar hea pelloa apes. The atyle of unimiics playfil
loon oxpoeas artaing
out pose and semple crede senae of myster and prriek en journe oho
somaliiing
Cive tac'a tora
eegreiitttee appeat cllo
Figure 10: Examples of multi-modal generation with Kaleido-DART.
are trained in comparison with standard 16 noise levels. Not surprisingly, less noise levels lead to gradually degraded performance. However, with as few 4 noise levels, DART can still generate plausible samples, indicating the capabilities of deploying DART in a more efficient way. Variants in noise levels provides a potential option for finding the optimal computation and performance trade-off' especially when compute is limited.
Multimodal Generation We showcase the capabilities of our proposed method in the joint genera- tion of discrete text and continuous images, as introduced in $ 3.3. Figure 10 provides examples of multimodal generation using the Kaleido-DART framework. Given an input, the model generates rich descriptive texts along with corresponding realistic images, demonstrating its ability to produce diverse samples with intricate details. Notably, unlike Gu et al. (2024), our approach processes both text and images through the same model, utilizing unified mechanism to handle both modalities. This unified framework can potentially be integrated into any multimodal language models.
CONCLUSION
We presented DART, a novel model that integrates autoregressive denoising with Man-Miakovian diffusion to improve the efficiency and scalability of image generation. By leveraging the full generation trajectory and incorporating token-level autoregression and flow matching, DART achieves competitive performance class- class-conditioned and text-to-image tasks. This approach offers unified and flexible solution for high-quality visual synthesis.
Our current model is restricted by the number of tokens in the denoising process. One direction is to explore architectures for long context modeling which enables application to problems like video generations. Also, current work only conduct preliminary investigates multimodal generation. Future work may train a multimodal generative model based on the framework of the decoder-only language model to handle wide variety of problems with unified model.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Michael Albergo, Nicholas Boffi, and Eric Vanden-Eijnden. Stochastic interpolants: unifying framework for flows and diffusions. arXiv preprint arXiiv2303.08779, 2023.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. Advances in neural information processing systems, 2020.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Πyy Sutskever. Generative pretraining from pixels. In International conference on machine learning, pp. 1691- 1703. PMLR, 2020.
Rewon Child. Very deep vaes generalize autoregressive models and can outperform them images. In International Conference on Learning Representations, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A Large-scale Hierarchical Image Database. IEEE Conference Computer Vision and Pattern Recognition, pp. 248 255, 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.2 1783, 2024.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873-12883, 2021a.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution im- age synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12873-12888 2021b.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederie Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning. 2024.
Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Miguel Angel Bautista, and Josh Susskind. f-dm: A multi- stage diffusion model via progressive signal transformation. arXiv preprint 10X.v:2210.04955, 2022.
Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Joshua M Susskind, and Navdeep Jaitly. Matryoshka diffusion models. The Twelfth International Conference on Learning Representations, 2023.
Jiatao Gu, Ying Shen, Shuangfei Zhai, Yizhe Zhang, Navdeep Jaitly, and Joshua M Susskind. Kaleido diffusion: Improving conditional diffusion models with autoregressive latent modeling. arXiv preprint arXiv:2405.2 2024.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference- free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7514-7528, 2021.
11

---

Published as a conference paper at ICLR 2025
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. Classifier-rree free diffusion guidance. In NeurlPS 202/ Workshop Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 53:6840-6888 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res., 23:47-1, 2022.
Maciej Kilian, Varun Japan, and Luke Zettlemoyer. Computational tradeoffs in image synthesis: Diffusion, tasked,token, and next-token prediction. arXiv preprint arXiv:2405. 2024.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 19211696221701 2021.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114, 2013.
arXiv preprint
Doyup Lee, Chiheon Kim, Sachoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11523-11532 2022.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. arXiv preprint arXiv:2406.11888 1838, 2024.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common Objects in Context. European Conference Computer Vision, pp. 740-755, 2014.
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. The Eleventh International Conference on Learning Repre- sentations, 2023. URL https: / //te:://ppeeeee eeeeeeeeiieeeeeeeee .ttee/eewweeeeeeeeeeeee.eeeeeeee forum? d= PVVVVRDDCT...
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning generate and transfer data with rectified flow. arXiv preprint arXiv:2209.084 03000, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rXiv:1711.05101, 2017.
Nanye Ma. Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden and Saining Xic. Sit: Exploring flow and diffusion-barte generative models with scalable interpolant transformers. arXiv preprint arXi::240.008740 2024.
Aaron den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel Recurrent Neural Networks. ICML, 2016.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International conference on machine learning, pp. 6055-4064, PMLR, 2018.
William Peebles and Saining Xic. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748. 2022.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01993 00952/ 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal machine learning research, 21(140): 2020.
12

---

Published as a conference paper at ICLR 2025
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding, Advances in Neural Information Processing Systems, 35:36479-36494 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512. 2022.
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202 2020.
Jascha Sohl-Dicke bohl-Dicksteen, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference Machine Learning. pp. 2256-2265 PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2021.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13455, 2020.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06555, 2024.
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.098/5 2024.
Scalable image generation via next-scale prediction. arXiv preprint arXiv:2404.00 02905, 2024. Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling:
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xingrun Xing, Ruiran Yan, Shuting Wang, Tiejun Huang, and Zheng Liu. Omnigen: Unified image generation. arXiv preprint arXiv:2409.1 1340, 2024.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. arXiv preprint 2rXiv:2112.07810 2021.
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. 12528. 2024.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. Transactions on Machine Learning Research, 2022.
13

---

Published as a conference paper at ICLR 2025
Chuyang Zhao, Yuxing Song, Wenhao Wang, Haocheng Feng, Errui Ding, Yifan Sun, Xinyan Xiao, and Jingdong Wang. Monoformer: One transformer for both diffusion and autoregression. arXiv preprint ar arXiv:2409.16280, 2024.
Huangjie Zheng, Zhendong Wang, Jianbo Yuan, Guanghan Ning, Pengcheng He, Quanzeng You, Hongxia Yang, and Mingyuan Zhou. Learning stackable and skippable lego bricks for efficient, reconfigurable, and rarouble-resolution diffusion modeling. The Twelfth International Conference on Learning Representations, 2023.
Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-mon modal model. arXiv preprint
2024.
14

---

Published as a conference paper at ICLR 2025
A PROOF OF PROPOSITION 1
Proof. Due g( (m)) (a vneao - -+))),, we can write a, v/.20 + v1 -
With certain coefficient {Xg) let us define:
y/
(11)
We study when the signal-to -gmiie noise ratio of y: achieves its maximal value:
T
y1:
Σ
To
2o
X v 766
(12)
SNR(
7s Ys
(13)
E. 23 21 %ss
which follows from Titu's lemma and the Cauchy-Schwarr Schwarz inequality. It becomes an equality when:
X
vs E T]
(14)
Next, we demonstrate that {3e follows Markov property when achieving the maximal signal-to- To
noise ratio for each From Eq (14), let X Pt Pti- Pr O, and T-76 -7 have:
-I=e Us. we
Y: Pt T0
Σ
T.E
(15)
a
Pr (1:30 + VTE,)),
(16)
where we use e/ (0, I) equivalently simplify the noise term. When Pt 1/v + nr fyche is variance preserving (VP). Next, let us assume
P++1
1.tE,E~N(0,)
(17)
PElt:
Pt4 + Pc+1 VTlee
(18)
T
o²
=
Pr+1
T1+120 +
V
c"
(19)
7t
Pi+
where we use e" to replace the noise term. So we let ý match the distribution of y:+1, then
ni+1
a²
+
7c+1
(20)
Pi
2
p
2
n+
> 0
(21)
πt
The above equation has root that = Pt++1 7 / Tr. This implies that we can find an independent noise term added to Ye obtain Bt++1, establishing that ye, constitutes Markovian forward process.
ply
(yr) /
(P++1T++1
Yts 2
Tc+
(22)
Petje
What's more, (æ:) sequence can be uniquely determined from {ye via
Vt yi+1
ift < T
P: P1+1
VT.
(23)
y: 1 7
=ft=
Pt. VT,
Therefore, the two processes
) and {yr}e has a one-to-one correspondence.
15

---

Published as a conference paper at ICLR 2025
B IMPLEMENTATION DETAILS
B.1 ARCHITECTURE
Table lists the parameters of different model sizes for DART. For DART-FM, we implements the flow net as three feed-for weed-forward networks (FFNs) with additional adaptive LayerNorm for modulation. Also unlike common FFNs in Transformer, the hidden size stays unchanged in our implementation.
Table 1: Configurations of DART.
Model
Layers
Hidden dim
# Heads
# Params
DART-S
12
384
6
48M
DART-B B
12
768
12
141M
DART-L
24
1024
16
464M
DART-XL
28
1280
20
812M
B.2 TRAINING
All models are trained with the following settings.
default training config:
batch_size=la Dotimizer=/Adami AdamW'
adan_beta1=0.9
adam_beta2=0.95 95
adan_eps=le-8
Learning_rate=3e-4
warmup__ttep===00000
weight_deccy=0..01
gracient_clip_noomen
ema_decay=
mixe__precissoon_trrnnnnnnggii
C RESULTS ON IMAGENET-256
Table 2 lists the performance of our proposed DART in comparison to recent generative models on ImageNet-256. The reproduced results use the official codebase and checkpoint and we follow the best performing ofg scale reported in the original papers. We report reproduced results of DiT (Peebles & Xie, 2022), SiT (Ma et al., 2024), and MAR et al. (2024) with 16 sampling times which is the same vanilla DART. Our model achieves competitive performance when compared with diffusion models like LDM (Rombach et al., 2021) and AR models like VQGAN (Esser et al., 2021a) and Tr-nssssformer (Lee et al., 2022). Admittedly, there is gap between DART and SOTA visual generative models (like VAR (Tian et al., 2024) and MAR (Li et al., 2024)). However, we want to point out that many basclines are trained with sigmificantly more FLOPs. For example, DiT (Poebles & Xie, 2022) is trained for 7M iterations whereas DART is only trained for 500k iterations. Also, baselines like VAR and MAR employ larger models than our DART. In particular, VAR deploys 2B model while the largest DART model is approximately 800M. Besides, in our reproduced results, when using only 16 sampling steps as the setting of our vanilla DART, our model show significantly better performance than DIT, SiT and MAR. Also, report MAR-AR (Li et al., 2024), a variant of MAR which generate tokens an autoregressive manner instead of masked modeling which is applied in standard MAR models. DART which generates samples through autoregressive denoising shows better performance than MAR-AR. These results further validate the effectiveness of leveraging the whole denoising trajectory.
16

---

Published as a conference paper at ICLR 2025
Table 2: Generative models class-conditional ImageNet 256 266. *: reproduced from official codebase and checkpoints.
Type
Model
FID
IS
#params
Steps
ADM (Dhariwal Nichol, 2021)
10.94
101.0
554M
250
CDM (Ho et al., 2022)
4.88
158.7
8100
Diff.
LDM (Rombach et al., 2021)
3.60
247.7
400M
250
DiT (Peebles & Xie, 2022)
2.27
278.2
675M
250
SiT (Ma et al., 2024)
2.06
277.5
675M
250
VQGAN (Esser et al., 2021b)
15.78
74.3
1,4B
256
RQTran (Lee et al., 2022)
3.80
323.7
3.8B
68
AR
MAR-AR (Li et al., 2024)
4.69
244.6
479M
256
MAR (Li et al., 2024)
1.55
303.7
943M
256
VAR (Tian et al., 2024)
1.73
350.2
2.0B
10
DiT* (Peebles & Xie, 2022)
19.52
125.9
675M
16
Reprod.
SiT* (Ma et al., 2024)
6.98
122.9
675M
16
MAR (Li al., 2024)
6.37
221.3
943M
16
DART
5.62
231.7
812M
16
Ours
DART-AR
3.98
256.8
812M
4096
DART-FM
3.82
263.8
820M
D ADDITIONAL TEXT-TO-IMAGE RESULTS
We here show more text-to-image generative examples from DART-AR and DART-FM at resolution 256 x 256 in Figure 13.
E MULTI-RESOLUTION GENERATION
DART (including both the -AR and -FM variants) is a highly flexible framework that can easily extended and applied in various scenarios with minimal changes in the formulation. For example, instead of learning a fixed resolution of images, one can learn a joint distribution of ps (())) 1 where x IR K2XC X 20 with different resolution. Following the approaches proposed in (Gu et al., 2022; 2023; Zheng et al., 2023) for diffusion models, a single DART referred to as Matryoshka- DART - can model multiple resolutions by representing each image with corresponding noisy sequence {x/] J separately, then flattening and concatenating these sequences for sequential prediction. As shown in Figure 11, we model the NOMAD objective in Eq (5) as
N
T
Matryoshka max Cg
E
ΣV
of log po (x 1 x (ir)
(24)
(a(mo)
where the above can be implemented using any DART variant. Note that anj is not directly conditioned on 2 which not only avoids the need to handle shape changes the boundaries but also mitigates potential eтror propagation, common issue in learning cascaded diffusion (Ho et al., 2022). All low-resolution information is processed through self-attention.
By this approach, the model can balance the number of noise levels with the total number of tokens to achieve better efficiency. Additionally, learning resolutions can be progressively increased by finetuning low-resolution models with extended sequences.
Figure 12 visualizes the generative process an image at resolution 256 x 256 and its upsampling to 512 x 512 by Matryoshka-DARR DART first iteratively refines the generative results for 16 denoising steps at resolution 256 x 256. then upsamples images at resolution 512 512 and 1024 x 1024 through iterative denoising well. Since the generation of high resolution is conditioned on the previous low-resolution samples, it needs less demoising steps at high resolution to generate realistic images. In particular, add 4 denoising steps for generating 512 512 images and further add
17

---

Published as a conference paper at ICLR 2025
DART (Multi-Scale Modeling)
Figure 11: Illustrations of Matryoshka-DART, By joint training models, DART perform multi-
resolution.
Table 3: Comparison of DART variants.
Model
Attn Mask
#AR steps
#FM steps
DART
Block-wise Causal
16
0
DART-AR
Causal
4096
0
DART-FM
Block-wise Causal
16
1600
denoising steps for generating 1024 x 1024 images. Figures and 15 show examples resolution 512 x 512 and 1024 x 1024 from Matryoshka-DAre -ART finetuned from DART-FM at 256 256.
F COMPARISON OF DART VARIANTS
We here further clarify the differences and connections between the variants of DART. Table lists the major comparison between DART, DART-AR, and DART-FM. Conceptually, DART predicts the denoised value and adds independent noise to acquire less noisy image at each step. It conducts this denoising process autoregressively until the clean image is generated. DART-AR applies token- wise autoregression instead of block-wise autoregressive in vanilla DART, which conduct denoising generation in a more fine-grained granularity. DART-FM, on the other hand, keeps the block-wise autoregression while introduces additional flow network to conduct Now-mating-baser refinement for generated tokens. Both DART-AR and DART-FM improve the performance over vanilla DART. In general, DART-FM demonstrates better tradeoff between generation quality and inference efficiency.
Kaleido-DART is for multimodal text-image generation, which integrates next-token prediction for text and next-denoising prediction for images (proposed in our DART). In image generation, it can seamlessly adapt all three variants of proposed methods: DART, DART-AR, DART-FM. Similarly, Matryoshka-Daa DART, which enables multiitesolluion generation, also adapts to all the three DART variants. Since in Matryoshka- -ART, one can simply concatenate high-resolution image tokens after the low-resolution ones, which doesn't it affect the denoising modeling in these variants.
18

---

Published as a conference paper at ICLR 2025
Step
Step 9
Step 17
Step - 2
Step 10
Stop 33
Step 11
Stop
Step 12
Step 18
Stop 5
Step = 13
Step
Step 14
Step = 19
Step 7
Step 15
Step
Step 16
Step 20
Figure 12: Visualization of the generation process (left: latent maps, right: decoded RGB images) for 256 256 (T =16) and its upsampling to 512 x (T 4) using Matryoshka-DARi DART..
19

---

Published as a conference paper at ICLR 2025
(a) Generated text-to-image samples from DART-AR 256x256 pixels
A corg
An angry duck doing heavy weightifting at the gym
A horse roading bookin a library
light bulb containing a salibott floats through the galexy
supressive all chip being dipped glass depicled as explesion of flevors
phull island, waterfall piselated cascacing down into pixelated cloud below.
crewting meticulousty
wateriall pioslatad wister cascading down into picelsted cloud below.
(b) Generated text-lo-image samples from DART-FM FM 256x256 pixels
Downtown NYC sunrise,
A smail book largo
monarch buttertty.
grafthi of funny dog on street wall.
detalled ink
A rowbcat lake with bike
A photo of person with the head coww weering fuoedo and black bowtie, Beach wallpaper in beckground.
yallow wall with two frames of tamous paintings.
An old rusted robet wearing pants and jacket rifing skis supermarket.
DART
entrely switing starding majestially andscape surflowers.
mountain village dawn, wooden golden surlight illuinnting snow-coovred pesks
mechanical emerging ocean, rising lightning.
hypar- hypar-realiste sign prominently displaying (DART in bold.
Figure 13: Additional samples from DART varints on text-to-image generation at 256 x 256 pixels given various captions.
20

---

Published as a conference paper at ICLR 2025
An all painting of ariat painting andacape coool oot park ot aut.mn.
A stack of pancakes with syrup dripp ng down the sides, topped with butter and few blusbarie les.
Heat death of universe line
pair sung esses resting the sond the beach, with a gentle wave spproaching.
spaceehio landing on wilen planet with purple skiee, glowing and toneaiin crystal formationa sumour ding ancinng
swimming in through room filld with hanging paper lanters, Lantern painted with waleecciors
Figure 14: Samples from DART-FM with Matryoshka-DARR fine-tuning on text-to-image generation at 512 x 512 and 1024 x 1024 pixels given various captions.
21

---

Published as a conference paper at ICLR 2025
crab made chasse clade
U
deer sunit forest clearing,
light ref refracting through transparent
magnityin glase
Dycladic
magenta bougairill
setting.
megical portal opening many clocks surrounding portal.
sofill panda sitting wincowsll rany day, looking raindrops the glass reatmi through.
Figure 15: Samples from DART-FM with Matryoshka-DAae -DRR fine-tuning on text-to-image generation at 512 x 512 and 1024 x 1024 pixels given various captions.
22


Reviews:
[{"review_id": "g7BKhjA3ka", "reviewer": "Reviewer_74aA", "summary": "This paper proposes DART, a new method to learn a distribution over images by autoregressively denoising a noise image to obtain high-quality generated images. They claim that DART leverages all the latents within the trajectory in oppose to existing methods that only use the previous latent. They compare their method with DiT and observe improvements in generated contents as evaluated by FID and CLIP Score.", "questions": "I want the authors to clarify the comprehension of their evaluation. Also, I'd want to see how the autoregressive generation is the cause of improvements. Can authors run any experiments supporting the fact that autoregressive generation, where attention happens with all previous tokens, has a huge role?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The method is scalable and efficient as explained in the paper.", "weaknesses": "I think that the comparison to existing work is not comprehensive. Is DiT the only other model that this work should be compared with?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2S4xKAeDfs", "reviewer": "Reviewer_xQrW", "summary": "The paper presents DART, al generative model combining diffusion and autoregressive (AR) frameworks for efficient and scalable image synthesis. DART addresses limitations in traditional diffusion models, particularly their Markovian process, which restricts generative efficiency by only leveraging information from the previous denoising step.", "questions": "1. In the inference stage, does the DART-FM also show the short-step sampling property? How to realize short-step sampling with DART-FM.\n\n2. I still have no idea why DART-AR is better than DART. For me, the whole image processing mechanism should work better. Could the authors give me more detailed explanation?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The idea is interesting, exploring if we can use AR to model the image diffusion and denoising processes to generate image.\n\n1. The technique is well-supported by a theory. \n\n2. The results are good on border benchmarks. The T2I results seem promising.", "weaknesses": "My major concern is that the comparison is not comprehensive. DART integrates diffusion in the AR model, using the AR to model the diffusion. MAR [1] uses diffusion for the AR model, using the diffusion to model the tokenization. Even though their two different approaches, the comparison should include MAR, as MAR has already shown good generalization ability on multimodal generation [2]. It is becoming a trend. With this comparison, we can see if DART has this potential. \n\n[1] MAR, NeurIPS 2024\n\n[2] Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokens", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "85uluUq9rp", "reviewer": "Reviewer_AfU4", "summary": "The paper, introduces DART, a novel generative model that integrates autoregressive and diffusion approaches within a non-Markovian framework. Key contributions of the work include:\n- DART Model: A non-Markovian diffusion model that uses autoregressive techniques to capture the full denoising trajectory, improving generation efficiency and flexibility.\n- DART-AR and DART-FM Modules: DART-AR enables token-level autoregressive modeling for enhanced control and quality, while DART-FM employs flow-based refinement to increase model expressiveness.", "questions": "Please refer to the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The integration of autoregressive and non-Markovian diffusion in a unified framework is innovative, particularly by moving beyond the Markovian assumptions that traditionally constrain diffusion models. DART's AR and FM extensions add novel approaches for token-based and flow-matching refinement in image generation.\n2. Experiments are carried out on low-resolution class-conditional and text-to-image generation as well as multimodal generation, demonstrating its effectiveness on different classes.", "weaknesses": "1. My biggest concern is the lack of comparison to SOTA approach, either in the form of diffusion models or auto-regressive models, for image generation tasks. The whole paper literally only shows comparison to the baseline DiT model which is published in 2022. There're a flourished set of literature for transformer-based image generation (SD-3[1], PixArt[2], etc.) and multimodal generation (Chameleon[3], Transfusion[4], etc.), yet none of these is compared in the paper. As such, it is really hard to understand the performance and actual benefits this approach brings us.  \n2. The efficiency of the model is a big concern. As shown in Fig 7 and Fig. 9(a), although a finer grain modeling by DART-AR could bring a marginal improvement, it would introduce 100x more latency, making it hard to justify whether this efficiency-performance tradeoff is ideal. Scaling up a diffusion model that consume 100x more latency may have better performance. This comparison would be good to show in Fig 7.\n\n[1] \"Scaling rectified flow transformers for high-resolution image synthesis.\" ICML 2024.\n[2] \"Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis\". ICLR2024\n[3] \"Chameleon: Mixed-Modal Early-Fusion Foundation Models\". ArXiv\n[4] \"Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model\". ArXiv", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HBrSDeJUOs", "reviewer": "Reviewer_7f3i", "summary": "DART unifies autoregressive (AR) and diffusion models within a non-Markovian framework, addressing the limitations of traditional diffusion models' Markovian property.", "questions": "see Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. DART has well-grounded theoretical framework connecting non-Markovian and Markovian processes.\n\n2. It can handle multiple modalities. \n\n3. The experiments across different tasks are comprehensive.", "weaknesses": "1. While the technical contributions are solid, the paper's narrative structure prioritizes theoretical formulation over an intuitive explanation of the approach. The image is first transformed into several patches in the latent space through a Variational Autoencoder (VAE). The basic version of DART processes these patches simultaneously, predicting their denoised mean values, and then continues denoising in a manner similar to traditional diffusion models. The authors can include a more accessible explanation of how DART works in practice.\n\n2. Even though, I think this paper's results are good, and show generalization ability, I am not sure whether this AR+diffusion method become new trend for future visual generation. Is is better than transfusion, show-o, and mar manner? (I recognize those should be concurrent work. This is just for curiosity, and I will not penalize the paper in terms of scoring).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EHKQiFYZvk", "reviewer": "Reviewer_Z9qZ", "summary": "This paper presents DART, a model that integrates autoregressive and diffusion methods within a non-Markovian framework for efficient, scalable text-to-image generation. DART uses autoregressive denoising to retain the full generation trajectory, improving image coherence and generation quality. Key features include DART-AR for token-level autoregressive modeling and DART-FM for flow-based refinement, both enhancing image fidelity and flexibility. DART beats standard DiT in class-conditioned and text-to-image tasks and adapts well to multimodal generation, providing a unified and scalable approach for high-quality visual synthesis.", "questions": "* Is the proposed DART compatible with existing techniques for diffusion models, e.g., DDIM, DPM-Solver, etc. I think it is a bit unfair to directly compare with original DiT with 16/256 steps without using any of these advanced samplers.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall I find that the writing is clear and well-structured, making it easy for readers to follow the arguments and understand the key points. This paper starts by revisiting key design choices of diffusion models and naturally denoising autoregressive transformer, which is a relatively novel framework aiming to integrate the strength of both diffusion and autoregressive models. Besides, this paper provides plenty of experiments and demos in terms of class/text-conditional image generation.", "weaknesses": "* My main concern is that while the paper’s title is Denoising Autoregressive Transformers for **Scalable** Text-to-Image Generation, I am worried about whether the proposed method can truly scale to higher-resolution images. In Line 264, it mentions using  T = 16 , which results in around 4000 tokens for a 256x256 image. So, for resolutions like 512 or 1024, can it really be trained and sampled efficiently using this framework? Although VAR also adopts a similar progressive generation approach, it uses different resolutions in its generation trajectory, whereas DART’s generation trajectory consists of full-sized, noisy images. I would like to see the experiments on higher resolutions (512 and 1024).\n* The paper presents many different models (DART, DART-AR, DART-FM, Kaleido-DART), without providing a sufficient explanation of these models, such as their strengths and weaknesses. After reading this paper, I do not know how to choose between them since there is no consistent winner in the provided experiments. I would like to see a table to clearly describe differences, strengths, and weaknesses of each DART variant.\n* The paper only compares the DART family with the original DiT on the ImageNet benchmark. However, DiT is originated from a paper two years ago, and there are a bunch of related works with improvements in both diffusion/autoregressive transformers that should be discussed and compared, e.g., SiT [1], VAR [2], MAR [3], etc... It is better to provide comparisons with these sota models in terms of FID scores and inference efficiency.\n\n[1] Ma, Nanye, et al. \"Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers.\" arXiv preprint arXiv:2401.08740 (2024).\n\n[2] Tian, Keyu, et al. \"Visual autoregressive modeling: Scalable image generation via next-scale prediction.\" arXiv preprint arXiv:2404.02905 (2024).\n\n[3] Li, Tianhong, et al. \"Autoregressive Image Generation without Vector Quantization.\" arXiv preprint arXiv:2406.11838 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)