ACDiT: Interpolating Autoregressive Conditional Modeling and Diffusion Transformer
Authors: Jinyi Hu
Shengding Hu
Yuxuan Song
Yufei Huang
Mingxuan Wang
Hao Zhou
Zhiyuan Liu
Wei-Ying Ma
Maosong Sun
Venue: Accepted by TMLR ()

Abstract: Autoregressive and diffusion models have achieved remarkable progress in language models and visual generation, respectively. We present ACDiT, a novel Autoregressive blockwise Conditional Diffusion Transformer, that innovatively combines autoregressive and diffusion paradigms for continuous visual information. By introducing a block-wise autoregressive unit, ACDiT offers a flexible interpolation between token-wise autoregression and full-sequence diffusion, bypassing the limitations of discrete tokenization. The generation of each block is formulated as a conditional diffusion process, conditioned on prior blocks. ACDiT is easy to implement, as simple as applying a specially designed Skip-Causal Attention Mask on the standard diffusion transformer during training. During inference, the process iterates between diffusion denoising and autoregressive decoding that can make full use of KV-Cache. We validate the effectiveness of ACDiT on image, video, and text generation and show that ACDiT performs best among all autoregressive baselines under similar model scales on visual generation tasks. We also demonstrate that, benefiting from autoregressive modeling, pretrained ACDiT can be transferred in visual understanding tasks despite being trained with the generative objective. The analysis of the trade-off between autoregressive and diffusion demonstrates the potential of ACDiT to be used in long-horizon visual generation tasks. We hope that ACDiT offers a novel perspective on visual autoregressive generation and sheds light on new avenues for unified models.

Published in Transactions on Machine Learning Research (01/2026)
ACDiT: Interpolating Autoregressive Conditional Modeling and Diffusion Transformer
Jinyi Hul
Shengding Hul*
Yuxuan Song'
Yufei Huang'
Mingxuan Wang"
Hao Zhou
Zhiyuan Liu'
Wei- Ying Ma
Maosong Sun
TTinghaa University
2 ByleDance
(hu-jy21 (hsd23) @mmilss www.giuul..tiiitaa..o.o. edu.cn
Reviewed on OpenReview: https: // openrevee.. net/ forum? id- OuFNXESOCO
Abstract
Autoregressive and diffusion models have achieved remarkable progress in language models and visual generation, respectively. We present ACDrT, novel Autoregressive block- wise Conditional Diffusion Transformer, that innovatively combines autoregressive and diffusion paradigms. By introducing a block-wise autoregressive unit, ACDIT offers a flexi- ble interpolation between token-wise autoregression and full-sequence diffusion, bypassing the limitations of discrete tokenization. The generation of each block formulated as conditional diffusion process, conditioned prior blocks. ACDIT is casy to implement, simple applying specially designed Skip-Causal Attention Mask on the standard diffuuson transformer during training. During inference, the process iterates between diffusion denoising and autoregressive decoding that can make full use of KV-Cache. We validate the effectiveness of ACDIT on image, video, and text generation and show that ACDrT performs best among all autoregressive baselines on similar model scales on visual generation tasks. We also demonstrate that benefiting from autoregressive modcling, pretrained ACDIT can be transferred to visual underslanding lasks despite being trained with the generalive objective. The analysis of the trade-off between autoregressive and diffusion demonstrates the potential of ACDIT to be used in long- horizon visual generation tasks. We hope that ACDIT offers novel perspective on visual autoregressive generation and sheds light on new avenues for unified models.
1 Introduction
Autorcgressive modeling has been central paradigm in artificial intelligence, most notably driving the success of large language models through next-token prediction (Touvron et al., 2023a;b; Brown, 2020; Achiam el al., 2023; Jiang et al., 2023; Bai et al., 2023; Yang et al., 2024). By decomposing complex distributions into a sequence of conditional predictions, autoregressive models provide a natural mechanism for modeling long- range dependencies and structured generation. Similar predictive formulations also underpin reinforcement learning (Mnih, 2013; Schulman et al., 2017; Chen al., 2021; Reed et al., 2022) and world models (Ha. & Schmidhuber, 2018), where future states predicted conditioned past observations. These successes motivate us to revisit autoregressive factorization beyond language and ask how its advantages can be effectively extended to high- -igh-dimensional visual generation.
In the realm of visual generation, diffusion models (Ho et al., 2020; Song et al., 2020a; Dhariwal Nichol, 2021; Song el al., 2020b) have demonstrated superior generative capabilities, producing creative outputs that are
"Equal contribution i Correspooding authors

---

Published in Transactions on Machine Learning Research (01/2026)
Diffusion Process
Diffusion Process
Diffusion Process
dutoregressie
(a) DIT: Image generation with diffusion process
31
X1
Autoregressive
(b) AR: Image generation with next-image-toker generation
Image
Video
ACDi:: Image/Video generation generaaiooo with next-black generation via conditional diffusion process
Figure 1: The generation process of ACDIT. Pixels in each block are denoised simultaneously conditioned on
clean context which are autoregressively generated. The block can be flexibly defined, such as part of pixels in images or several frames in videos.
virtually indistinguishable from human-generated content, as evidenced by innovations like Sora (Brooks et al., 2024) and Stable Diffusiin (Rombach et al., 2022; Podell et al., 2023; Esser et al.). Despite their remarkable success, diffusion models operate in a non-autoregressive manner. They take as input corrupted target sequences of the full length and reconstruct the intended output through in-place iterative refinement. The term "in-place" highlights a fundamental distinction from autoregressive models, which provide subsoquent prediction, thereby extending the sequence and progressing lowards the future. Such variation makes the diffusion model less effective at capturing long-range temporal dependencies in visual data and poses difficulties for developing integrated frameworks capable of seamlessly bridging the vision foundation model with unified multimodality modeling (Dong et al., 2024; Team, 2024; Zhou et al., 2024a; Wang et al., 2024b; Xie et al., 2024; Wu et al., 2024b) and world model (Yang et al., 2023; Du et al., 2023; Bruce et al., 2024; Zhou et al., 2024b; Wu et al., 2024a).
To build the visual autoregressive model, existing works convert visual generation tasks into discrete token prediction tasks with vector quantization techniques (Esser et al., 2021; Team, 2024; Wang et al., 2024b; Tian al., 2024) and training with next-token prediction objective. However, approaching the continnous distribution requires huge vocabulary sizes and high utilization rale (Yu et al., 2023b; Weber et al., 2024), which is a complex objective.
In this paper, we propose ACDIT, an Autoregressive blockwise Conditional Diffusion Transformer that fuses the diffusion process with the autoregressive paradigm. Instead of relying on discrele tokens, we demonstrate that visual autoregressive generation can also be achieved using continuous visual features. At a high level, extend the autoregressive units from the individual discrete token to blocks, where cach block consists of visual patches of flexible size. The generation of each block can be formulated as conditional diffusion process based on the previous block. As Fig. shows, for image generation, block represents small region of the image, whereas in video generation, a block can correspond to single frame or multiple frames.
ACDIT easy to implement, as simple as adding a tailored Skip- CappCCusal Attention Mask to the current diffusion transformer (Peebles & Xie, 2022) during training. The inference process is formatted as an iteration between the conditional diffusion denoising process within block, conditioned on the complete clean context, and antoregressive generation of new block appended as the new context. In this way, KV-Cache can be used for faster inference. In general, ACDrT offers the following inherent advantages: (i) ACDIT simultancously learns the causal interdependence across visual blocks with autoregressive modeling and the non-causal dependence within blocks with diffusion modeling. (ii) ACDIT is endowed with clean and continuous visual input, which eliminates the requirement for vector quantization and improves the transfer visual understanding tasks after being trained with generative objectives. (iii) ACDIT makes full use of KV-Cache for flexible autoregressive generation of any length and can potentially benefit from other latest techniques in text for long video generation.
2

---

Published in Transactions on Machine Learning Research (01/2026)
The effectiveness of ACDiT is primarily validated visual generation tasks, including both image and video generation tasks, respectively. To further demonstrate its versatility, we also perform experiments on text generation. Experimental results demonstrate that ACDIT outperforms all antoregressive baselines of the same model scale, and achioves visual quality comparable to full-sequence diffusion models while exhibiting higher inference speed when extended to long sequences. It also achieves strong performance on text gencration following the discrete diffusion formulation without any specific customization. We hope ACDIT presents fresh perspective on the visual autoregressive model and paves the way for exploring unified multimodal and world models. The codes and models are available at https:/ http:/ggtbbbbo//eeeee /ttunlpt/l//////// ACDiT.
2 Related Work
Diffusion Models. The field of image generation has witnessed remarkable advancements with the rndue tion of diffusion models (Ho et al., 2020; Song et al., 2020a; Dhariwal & Nichol, 2021; Nichol & Dhariwal, 2021). U-Net (Ronneberger et al., 2015) is the early mainstream choice of network architccture (Song et al., 2020b; Nichol & Dhariwal, 2021; Rombach et al., 2022; Podell al., 2023). Following that, Transformer (Vaswani et al., 2017) is applied to diffusion models for image generation, with groundbreaking work such as DiT (Pee- bles & Xie, 2022) and U-ViT (Bao et al., 2023). A series work, including PixArt-(a, 6, } (Chen et al., 2023; 2024c;b), demonstrate the capability of DiT on text-to-image tasks. Several studies also apply DiT to video generation, such as Lumiere (Bar-Tal et al., 2024) and Movie Gen (Polyak et al., 2024).
Autoregressive Generation on Discrete Tokens. The iGPT (Chen et al., 2020) first proposes autore- gressively generating raw image pixels as raster-scan sequence. Most visual autoregressive models (Ramesh et al., 2021; Wang et al., 2024b; Yu et al., 2022; Ding et al., 2021; 2022) follow VQGAN (Fsser et al., 2021), which pioneers this direction by training an autoregressive transformer on discrete tokens produced by VQVAE (Van Den Oord et al., 2017). LlamaGen (Sun et al., 2024) enhances the image lokenizer and scales up the autoregressive transformers building on the latest Llama architecture (Touvron et al., 2023a). Inspired by RQ- and obtains good improvement. Some subsequent works (Tian et al., 2024; Li et al., 2024c; Yao et al., 2024) extend next-scale prediction to sot-to--maae generation.
(Lee et al., 2022), VAR (Tian et al., 2024) proposes the next-scale prediction
Autoregrossive Generation Continuous Features. Some recent works make some carly exploration achieving visual autoregressive generation without discrete tokens and combining the advantages of diffusion and autoregressive. Diffusion Forcing (Chen et al., 2024a) trains a causal autoregressive model to generate blocks without fully diffusing past ones and implements it on small RNN, MAR (Li et al., 2024a) proposes the diffusion loss to learn the autorogressive conditional distribution on the head of the main Transformer with a small MLP network. Transfusion (Zhou et al., 2024a) and Monoformer (Zhao et al., 2024a) conduct joint training with language modeling loss and diffusion loss a single transformer, while within image generation, their design is identical to the standard diffusion model. Dart (Gu et al., 2024) proposes to autoregressively denoise the complete image within in Transformer. Different from these works, ACDIT redefines the autoregressive unit and generates each block based on the clear past. Block Diffusion (Arriola et al., 2025) also explores an interpolation between autoregressive and diffusson modeling. Their approach focuses solely on text generation, building on discrete diffusion objectives proposed in prior work (Sahoo el al., 2024; Shi et al., 2024), and introduces lariahle-length generation and KV-caching within diffusion language models. In contrast, ACDIT is designed for visual generation, and motivated by cnabling autoregressive modeling over continuous visual data, yet still achieves better results on text generation using our simple and generally effective model architecture.
3 Prerequisite
3.1 Autoregressive Modeling
Autoregression asserts that the value at each timestep is contingent upon its preceding values. This principle is exemplified in autorogressive language models, which iteratively predict the probability distribution of subsequent tokens. Given a sequence of tokens (#1, z2, a salient characteristic of autoregression is that the prediction of 2; is only dependent on its prefix (x1, (2,,22,..... fi-.).. Upon determining I;; it is
3

---

Published in Transactions on Machine Learning Research (01/2026)
concatenated with the preceding sequence, thereby forming the conditioning context (2,,22,...,, to predict #i+1- Therefore, the likelihood of sequence can be factorized as:
p(21, 22... IN) II p(a.(re))
(1)
Thanks to the flexibility of self-attention in Transformers, autoregressive models can be effectively implemented by adding causal attention mask in the Transformer attention block (Vaswani et al., 2017).
3.2 Diffusion
Diffusion models, in contrast, conceptualize moise-infusion and denoising process, which is defined by gradually adding noise to the initial data z0 p(z) and training the model to learn the inverse mapping. Formally, the noised data 1x(*) at each step sampled by q(x(*)/2(t (z((-1)) NN(x()), vall)210) (1- (I) which is equivalent to add a Gaussian noise to the previons samples x01) = (a(l)((t + vi- (1-a00000 c00) ~ N/ I).
po is trained to learn the reverse process po (x(((1)(z<()) = (N(o(>(()), 3). With the reparameterization trick, the network (z()()) can be reparameterized as noise prediction network es(x(t) and the training objective can be simple as:
Co = -=
(2)
During the inference phase, the denoising process is initialized with a random Gaussian noise sample a!T), followed by T" demoising steps, ultimately yielding single deterministie sample () from its underlying distribution. Typically, the denoising process in diffusion models operates "in-place", meaning that each new denoising step directly replaces the previous step's is input. This differs from autoregressive modeling, where the value of a subsequent step is appended to the existing sequence.
4 ACDiT
4.1
Desiderata for Autoregressive Diffusion Model
Rather than modeling the full data distribution p(x) directly, autoregressive diffusson aims to learn the conditional distributions for the continuous visual feature p(=;(=<>> through a diffusion process, which is typically represented as categorical distributions over discrete token vocabulary in language models. Combining the (1. and (2), the learning objective of autoregressive diffusion is:
La = EB- Σ (=) it, x2x) - 12)
(3)
where {Ø1, X2, ..., [N} are N autoregressive units. In the context of visual generation, each unit can flexibly correspond the continuous visual information patch pixels in an image or set of frames in video, depending on the granularity of the representation.
The key difference between (3) and (2) is that for each input noisy block, the noise prediction network c(x!") predicts the noise conditioned on their previous clean block. To effectively learn this objective, we need a framework that integrates the stretts of both autoregressive and diffusion models. We identify three critical desiderata the framework should meet:
1.
The generation of future elements should be predicated on a precise representation of antecedent sequences. This imperative because any ambiguity in the past inevitably complicates future predictions. This approach preserves the efficacy of autoregressive modeling and potentially facilitates the development of the world model. Furthermore, adherence to this principle enhances performance in discriminative tasks (e.g., visual understanding), these tasks necessitate the input of all observable features into the model.
For clarification, use subscript denote timesteps in antoregressive models and superscript (t) denote the timesteps in diffusion models.
4

---

Published in Transactions on Machine Learning Research (01/2026)
Attention Diffuzion
Autoregressve
Order
Order
Fl Go Cy 2g
Co
Ngg Ra
Diffusion Step
Autoregressive Step
11y
60
11g
group
C4
Ro
Cached Past
7g
62
C2
CO
C1
C2
C3
No mask pesition
Mask position
(a) Skip-Causal Attention Mask.
(b) Inference process of ACDIT.
(c) 3D view of ACDIT.
Figure 2: (a): For each noised block n,, it can only attend previous clean block {co, Ci.... G-13 and itself. Each clean block C; only attends to the previous clean blocks. (b): ACDIT can effectively utilize the KV-Cache during inference. (c): the 3D view of ACDIT, where B, L, T denote the block size, number of blocks, and denoising time steps, respectively. The darker color denotes higher noise.
2. Both the antoregressive modeling and the denoising process should optimally utilize the entóre parameter space of the newral network. In an elegant fusion of autoregressive models and diffusion, neither component should be relegated to an auxiliary role. Instead, they should function as integral and complementary elements of the system.
3. The denoising process should directly allend comprehensively to the entire sequence of past sequences. Otherwise, the conditioning imput to the denoising process would be required to fully encapsulate all prior information, imposing an unrealistic burden on the bottleneck context vector to losslessly compress the context feature into one vector. A holistic input of all past information in each denoise step ensures a Forcing (Chen et al., 2024a) proposes to use different-level random noise in different positions of a sequence. Thus, in the inference process, denoising subsequent positions from a clean past can be seen as special case of denoising different levels of noise. Their method does not meet. the first desideratum. In the training process, the future is not predicted from the precise representation the past. In MAR (Li et al., 2024a), the diffusion process is trained based on the block in the last position, which does not satisfy the third desideratum. Moreover, the diffusion process sorely leverages the head part of the network, which conflicts with the second desideratum. To address suboptimal performance due to the causal direction based on this design, MAR incorporates bidirectional attention, which requires recomputing attention at each step and prevents the use of KVccache during inference. In Transfusion (Zhou et al., 2024a) and Monoformer (Zhao et al., 2024a), the autoregressive generation is restricted to the textual modality and is not applied within the visual modality. Image gencration in these models follows the standard diffusion process conditioned on preceding text inputs, without incorporating autoregressive prediction over visual patches. Moreover, in visual understanding tasks or multi-image generation settings, subsequent text and images attend the noisy version of the preceding image, which conflicts with the first desideratum. To mitigate this issuc, these models reduce the noise schedule, i.e., halving the maximum number of diffusion steps in 20% of image-captioning pairs, which is not optimal strategy. See the visual comparison of these models in Fig. 3a.
more effective processing of temporal dependencies.
Based on these desiderata, analyze representative autoregressive diffusion methods. Diffusion-
4.2 Framework
To satisfy the desiderata discussed above, we propose versatile framework for autoregressive diffusion called ACDIT. For generality, ACDIT runs block-wise autoregression instead of token-wise autoregression. We identify two kinds of blocks e; and n4. For each autoregressive unit 2;, G corresponds to the clean version 24, n, corresponds to the corrupted version 2!"). ACDIT learns the conditional noise prediction network cn(n (0, ces) with Eq. 3).
To effectively learn this objective within one transformer network, we can conceptualize all n, and all e as separate positions, effectively transforming the dependency structure into an attention pattern between different positions. We designate this attention pattern as the Skip-Causal Attention Mask, shown in Figure 2a.
5

---

Published in Transactions on Machine Learning Research (01/2026)
Clean Visual Patch
Noised Visual Patch
Condition
ni
nz
na
Noise
Scale
S
Pointwise
Transformer
Diffusion Transformer
Feedforward
Linear and Reshape
YerP2
Shift
X1
X/
72
71
715
7
o
62 12
Layer Norm
Layer Norm
Transfusion
Diffusion Forcing
ACDiT Block
Diff Head
Scale
n'
Skip- Meip-Causal
h2z
1
Patchify
Embed
Diffusion
Shift
Transformer
Transformer
Layer Norm
Latent
Latint
Labely y
Noisy
x
Co
C1
C2
x
Latents
Latents
Autoregressive Conditional Diffusion Transformer
Diffusion Head MAR
ACDiT
ACDIT Block with adaLN-Zero
(a)
Comparison of model design.
(b) The model architecture of ACDIT.
Figure 3: (a): Comparison between ACDIT with Transfusion (Zhou et al., 2024a), Diffusion Forcing (Chen et al., 2024a) and MAR (Li et al., 2024a). Transfusion does not perform autoregressive modeling within the visual modality. Diffusion Forcing introduces different noise levels across the autoregressive inputs. MAR applies diffusion only on lop of the backbone model. ACDIT utilizes the full parameters for both the autoregressive and diffusion process with clean input context. (b): The model architecture of ACDrT. Both the clean and noisy latent imput, while only the noisy latent interacts with the conditioning information. The Skip-Causal Attention Mask pattern, shown in Figure 2a, will be applied in the attention block.
The figure elucidates that n, attends to all preceding clean blocks {cylj 0, ..., 1} and itself, while e, also attends all precoding clean blocks {cylj = ...,, i- 1} and itself. In training, for simplicity, we can group attention mechanisms illustrated in the right matrix of Figure 2a. Suppose the number of blocks is N, then the unmasked positions form two triangular matrices with length N complemented by diagonal matrix with length N.
During the inference phase, each autoregressive step executes a conditional diffusion process for n, based on fco, Co-, KV-Cache. Upon finishing denoising, it appended to the clean sequence as ej followed by the maximal noisy version of the next block n4+1- The key-value tensor will be computed for these two blocks, and the key-value tensor of the clean block e; will be kept in KV-Cache. All noisy version of n, is disregarded. The process is visualized in Figure 2b. A three-dimensional view presented in Figure 2c. By interpolating between full-sequence diffusion and autoregressive paradigms, ACDIT enjoys flexibility and expressivity, enabling it to generate video of any length utilizing the latest long-context techniques developed for language models.
4.3 Positional Encoding
ACDIT is designed to be versatile, capable of handling one-, two-, three-, even higher-dimentn dimbesroiimeniionnll data, including but not limited to text (1D), images (2D), and vidco (3D). For any given dimension of data, the position of that data a critical altribute that must be made known to the model. This positional awareness enables the model to contextualize its current focus relative to historical data. In the domain textual data, the Rotary Position Embodding (RoPE) (Su al., 2024) has gained widespread adoption as an effective relative positional encoding method. To address the challenges posed by multi-dimensional positional indices, introduce RoPF-ND, natural extension of RoPF

---

Published in Transactions on Machine Learning Research (01/2026)
For token of D dimensional data, its positional index is [m1, [ my,..., mp). Given qucry and key vectors in the Transformer's attention module, we partition the hidden dimension into D segments. Tt is imperative that each segment's hidden dimension be an even number. For each segment j, we apply a RoPE with a specific base by. as defined in the following equation:
Rdh m
0
d2 2
R
(4)
R dp
ppp
d, where each R R.,,, represents dy-dimensional rotary matrix² with rotation angles θ = {b) 10)-2(1-1// 1)/4,
[1, 2,. 4.. The base b, is empirically determined as 100 100 7, where Lj denotes the maximum position index in dimension j. This formulation ensures that the highest wavelength of RoPE is approximately eight times the maximum position, thereby mitigaling rapid decay in long-term dependencies. It is worth noting that ACDIT inherently supports length extrapolation (Su et al., 2024), although comprehensive exploration of this falls beyond the present work.
4.4 Efficiency Analysis and Block Size Choice
We provide brief analysis of the computational efficiency of ACDIT in terms of floating-poins operations (FLOPS). The full derivation and more analysis are discussed in the Appendix A. We first assume that denoising each block requires the same number of time steps T as the full sequence diffusion, despite that when the block size is small, it may require fewer denoising steps, thus making ACDIT potentially more efficient. Let denote the sequence length, the hidden dimension, the number of parameters in a transformer layer, and B the block size used in ACDIT. The FLOPP for standard full-sequence diffusion is Fiull 2L0 + 4hL2, With KV-Cache, ACDIT reduces the attention cost to 2hL 2hLB. The relative
B
FLOPS savings is: Egged Fial
1-T T.. This indicates that ACDIT can reduce per-layer FLOPS by up to 50% k+
when L >>>>> B.
4.5 Model Architecture and Implementation
As shown in Fig. 3b, ACDIT mainly inherits the main architecture of DiT. We replace the original bidirectional attention with the proposed SCAM attention pattern to process the clean and noisy latent. In addition, we replace the absolute position embedding and Layer Normalization with RoPE (Su el al., 2024) and RMSNorm (Touvron al., 2023a), respectively. We use QK-norms for stable training. The additional condition, including timesteps and labels, is injected into the model with AdaLN-Zero only the noise part.
We explore 4 different model sizes, as shown in Table 5. In the image generation task, we set the patch size as 1 and the autoregressive unit block size as 256 = 16 x 16. ACDIT is trained on ImageNet for 1.2M iterations with a batch size of 1024. We use the AdamW optimizer (Loshchilov, 2017) and WSD (Warmup Steady Decay) learning rate scheduling (Hu et al., 2024). We sample images with DPM-Solver (Lu et al., 2022) for 25 steps within each block and use classifier-free guidance (Ho & Salimans, 2022) with a guidance scale of 1.5. In video generation, we sample 16 frames from each video and set the patch size as 2 and the block size 1024 = 256 x 4. We train ACDIT on UCF-101 for 400K iterations with batch size of 96. The Dassifier-frre guidance scale is 2.5. Unless otherwise specified, ablation and analysis experiments are conducted on ACDIT-B, See more details at Appendix 4.5.
5 Experiments
We mainly experiment on the ImageNet (Russakovsky et al., 2015) dataset with 256x256 resolution and the UCF-101 (Soomro et al., 2012) dataset with 16 frames for image generation and video generation, respectively. In addition, also validate the generality of ACDIT on text generation.
See detailed explanation in the Equation 15 (Su al., 2024).

---

Published in Transactions on Machine Learning Research (01/2026)
Table 1: Image generation results on ImageNet 256x256. We report FID score, Inception Score (IS), Precision
(Pre.), and Recall (Rec.).
Model
Type
Latent
KV-Cache
Params
FID4
IST
Pre.t
Rec.t
ADM (Dhariwal & Nichol, 2021)
Diff.
554M
10.94
101.0
0.69
0.63
LDM-I-G (Rombach et al., 2022)
Diff.
Cont.
100M
3.60
217.7
D/T-L/2 (Peebles & Xie, 2022)
Diff.
Cont.
458M
5.02
167.2
0.75
0.57
DiT-XL/2 (Peebles & Xie, 2022)
Diff.
Cont.
675M
2.27
278.2
0.83
0.37
MaskGIT (Weber al., 2024)
Mask.
Disc.
227M
6.18
316.2
0.83
0.58
MAGE (Li et al., 2023)
Mask.
Disc.
230M
6.93
195.8
VQGAN (Esser et al., 2021)
AR
Disc.
1.4B
15.78
78.3
RQTran (Lee et al., 2022)
AR
Disc.
<
3.8B
7.55
134.0
VAR-d16 (Tian et al., 2024)
VAR
Disc.
310M
3.30
274.4
0.84
0.51
VAR-d20 (Tian et al., 2024)
VAR
Disc.
600M
2.57
302.6
0.83
0.56
LlamaGen-L (Sun et al., 2024)
AR
Disc.
343M
3.07
256.1
0.83
0.52
LlamaGen-XL (Sun et al., 2024)
Disc.
775M
2.62
244.1
0.57
LlamaGen-XXI, (Sun et al., 2024)
AR
Disc.
14BB
2.34
253.9
0.80
0.59
ImageFolder (Li et al., 2024b)
AR
Disc.
362M
2.60
295.0
0.75
0.63
LlamaGen+NPP (Pang et al,, 2024)
AR
Cont,
1.4B
2.55
282.0
0.84
0.56
MAR-L-Causal (Li et al., 2024a)
Cont.
479M
4.07
232.4
MAR-L (Li et al., 2024a)
Mask.
Cont.
179M
1.78
296.0
0.81
0.60
MonoFormer (Zhao et al., 2024a)
DRfD
Cont.
1.1B
2.57
272.6
0.84
0.56
Dart (Gu et al., 2025)
ARf
Cont.
820M
3.82
263.8
ACDIT-L
AR+Diff
Cont.
460M
2.53
262.9
0.82
0.55
ACDIT-XL
AR+Diif
Cont.
677M
2.45
267.4
0.82
0.57
ACDIT-H
ARfD
Cont.
954M
2.37
273.3
0.82
0.57
5.1
Main Results
Image Generation. We report the FID-50K (Heusel al., 2017), Inception Score (Salimans et al., 2016), Precision and Recall (Kynkåänniemi al., 2019) of ACDIT and baselines in Table 1. Compared with previous autoregrossive models and masked generative models utilizing discrete tokens, such VQGAN, VAR, LlamaGen, and MaskGIT, ACDIT consistently achieves superior performance with lower FID scores at comparable model scales. Notably, ACDIT-XL achieves 2.45 FID scores, outperforming both LlamaGen- XXL and VAR-d20 with similar parameters. Additionally, when compared to the MAR-L-Causal variant that does not recompute attention, ACDrT-L significantly improves performance across all metrics, specifically improving FID from 4.07 to 2.53. Compared with other autoregressive diffusson methods, such as Monoformer (Zhao et al., 2024a) and Dart (Gu et al., 2024), ACDrT has significantly superior performance. When compared with leading diffusion-basod methods, ACDIT also demonstrates competitive performance. For instance, despite not employing full-sequence attention, ACDIT models achieve results close to DiT-XL. In general, these results highlight the distinct ad advantages of ACDIT over other baselines with the continuous latent representation and KV-Cache. Qualitative results are presented in the Appendix E.
Video Generation. Different from image generation, video inherently includes a temporal dimension, making more well-suited autoregressive modeling. The FVD metric on UCF-101 for class-conditional video generation is reported in Table 2. With hybrid AR+Diff architecture, ACDIT-H achieves much lower FVD than other diffusion-based and autoregressive methods, even outperforming MAGVIT-AR and MAGVITV2-AR, which utilize closed-source, specially designed video tokenizer. In contrast, ACDIT simplifies the process by directly using an open-soureed image VAE. Although MAGVIT with masked generative methods has a lower FVD than ACDIT, they rely on in-place operation to generate a video similar to the diffusion model and require repeatedly processing global context during inference. This constraint limits their ability and efficiency to generalize to long vidco generation and build world models. We further analyze the efficiency implications autoregressive versus masked generative formulations in the following sention, with quantitative inference-time comparisons reported in Table 6. Compared to image generation,
8

---

Published in Transactions on Machine Learning Research (01/2026)
Table 2: Video generation results on the UCF-101 dataset. ACDIT-XL-LI means ACDIT-XL trained for
longer epochs.
Model
Type
Params
FVD
LVDM (He et al., 2022b)
Diff.
437M
372
Latte (Ma et al., 2024)
Diff.
674M
478
Vidco-LaVII (Jin et al., 2024)
Diff.
7B
281
MagDiff (Zhao et al., 2024b)
Diff.
2B
340
Matten (Gao et al., 2024)
Diff.
853M
211
VidcoFusion (Luo et al., 2023)
Diff.
510M
173
MMVG (Fu et al., 2023)
Mask.
230M
328
MAGVIT (Yu et al., 2023a)
Mask.
306M
76
MAGVTTv2 (Yu et al., 2023b)
Mask.
307M
58
TATS (Ge et al., 2022)
AR
331M
332
CogVideo (Hong et al., 2022)
AR
9.4B
626
MAGVIT-AR (Yu et al., 2023a)
AR
306M
265
MAGVITv2-AR (Yu et al., 2023b)
AR
307M
109
OmniTokenizer (Wang et al., 2024a)
AR
650M
191
ACDIT-XL
DRfDiff.
677M
111
ACDIT-H
AR+Diif
954M
104
ACDIT-H-LT
DRfD
954M
90
Table 4: Test perplexities on OpenWeb
Table 3: Classification accuracy on ImageNet.
PPL (L)
Autoregressive AR (Sahoo al., 2024)
Model
Type
Params
Top-1 Acc
17.54
ViT-H (Dosovitskiy, 2020)
Supervised
632M
83.1
Diffusion
MAGE (I. al., 2023)
Masked
328M
84.3
SEDD (Lou et al., 2023)
24.10
MAF (He et al., 2022a)
Masked
632M
85.9
MDLM (Sahoo et al., 2024)
22.98
1GPT (Chen et al., 2020)
Generative
1.4B
72.6
Block-wise diffusion
DiT-XL (Peebles & Xie, 2022)
Cenerative
675M
82.8
BD3-LMs (Arriola et al., 2025)
T/ = 16
< 22.27
ACDIT-XL
Generative
677M
84.0
T/ 8
S 21.68
ACDIT
L' 256
S 22.98
L' 128
S 22.82
1'
< 21.59
ACDIT demonstrates greater potential in the more complex domain of video generation, where longer visual sequences and temporal information are critical. Qualitative results are presented in the Appendix E.
Text Generation. ACDIT is a general, modality-agnostic framework. At the architectural level, ACDIT adopts a unified block-wise autoregressive formulation and a shared attention pattern that conditions each block on clean past context while allowing non-causal modeling within the current block. This design is independent of the data modality and remains unchanged when applied text generation. For text generation, which operates discrete tokens, the differences from visual generation arise only in the noise process used during training and inference. Specifically, we follow discrete diffusion language modeling objectives and the same experimental setup previous work (Sahoo et al., 2024; Shi et al., 2024). Importantly, these choices alfect only the modalityyspecific corruption and denoising operators, while the model architecture, atlention mask, and block-wise autoregressive structure remain identical. Specifically, we use a 12-layer Transformer with a hidden dimension of 768, 12 attention heads, and 128-dimensional timestep embedding as our model backbone. We conduct all training and validation on the OeenWebText dataset (Gokaslan & Cohen, 2019), open-source replica of the WebText dataset. To evaluate the model's ability in probabilistic modeling and

---

Published in Transactions on Machine Learning Research (01/2026)
256,
256,
Lan
300
Le
254,
Geg 1024,
700
3250
600
500
25
335 3006
3306
2026
2536
3006
2006
400K
2006
150%
2315
2506
3300
AR Lengtt
Figure 4: FID and FVD curves of ACDIT-B over training steps Figure 5: The change of inference time in with different sequence lengths and autoregressive lengths. PS different autoregressive lengths under varied means patch size.
total sequence length.
60
DIT ACDIT Ae lon ACDIT, de lon
ACDTT ACDIT REPT-ND
ACOIT w/ 10 RUPE
50
ACDIT, Len
ACDIT we dleat cornee
Begin LR decay
RI-20k
40
20
EMA FID Ourve
<0K
300K
2004
503c
900K 950K 1000K D. 1.00K 1350K 1200K Training Steps
Rodel Size (M)
Elozs
(a) Scaling performance of ACDIT.
(b) Ablation for ROPE-ND and SCAM.
(c) FID curve of last 30% training.
Figure 6: (a): ACDIT shows scaling performance similar to DIT. (b): Ablation for ROPE-ND and using clean contexts. (c): FID score sharply drops with the learning rate decaying when using the WSD scheduler.
compression, we report Perplexity (PPL) on the validation set. PPL is defined as:
Bao [- -oog po (2o))
PPL exp
D
where D is the data dimension. For models where the exact likelihood po (2r)) is intractable, we report the PPL calculated using the variational lower bound the logelikelihood. Despite conceptual similarity to blockwise discrete approaches (Arriola et al., 2025), our effective positional embedding implementation (e.g., RoPE-ND; Sec. 4.3) enables ACDIT to achieve comparable performance across various block sizes as shown in Tab. 4 without any customized variance reduction or noise schedules as proposed in (Arriola et al., 2025). These advantages highlight the potential of ACDIT as a general and powerful generative modeling framework.
Image Representation. We assess the capability of ACDIT in image representation, which is essential for building unified visual understanding and generation model. We finetune ACDIT-XT, and DiT-XL ImageNet using classification loss for 100 epochs and report the Top-1 accuracy in Table 3. ACDIT outperforms DiT-XL, highlighting the advantage of using clean latents, which help the model to learn better representations compared to using only noisy latent inputs. Additionally, ACDrT matches the Top-1 accuracy of MAGE (Li et al., 2023), while offering superior generation performance.
5.2 Analysis
Trade off of block size. Fig. 4 illustrates the trend of trade-off under different sequence lengths and block sizes in image and video generation tasks on ACDIT-B, The FID curve indicates that for image generation, directly increasing the autoregressive length leads to decline in image quality, since each patch receives less attention information on average. However, we can mitigate this decline by increasing the total sequence length, which means reducing the patch size. For video generation, ACDIT shows more advantages due to the inherent temporal dependence of videos. The FVD curve demonstrates that increasing autoregressive length has minimal effect on the vidco quality, even with a slight improvement. As for efficiency, we test the sampling time for various sequence lengths with a batch size of 4 on an NVIDIA A100 GPU. Fig. 5 shows that as the sequence length increases, particularly beyond 16k, full-sequence attention (AR length of 1)
10

---

Published in Transactions on Machine Learning Research (01/2026)
Table 5: Configuration details of ACDIT.
Table 6: Inference cost comparison with MAR.
Model
#Layers
Hidden
MLP
# #Heads
Params
Model
Params
Image
Video
ACDIT-B
12
768
3072
12
132M
MAR-B
322M
2.74s
49.27s
ACDIT-L
24
1024
4096
16
460M
MAR-L
479M
3.58s
63.82s
ACDIT-XL
28
1152
4608
18
677M
ACDIT-L
460M
2.29s
4.08s
ACDIT-H
32
1280
5120
20
954M
ACDIT-XL
677M
3.22s
5.16s
becomes very tine--onsuming, necessitating the autoregressive generation. Empiriaally, blocks spanning 2-4 frames provide good quality efficiency trade-off for video generation. Dynamic or content-adaptive block sizing is promising extension, and we leave it for future work.
Inference Cost Comparison with MAR. We further compare the average inference time of ACDIT and MAR for image generation and video generation in Table 6. Since MAR is not designed or trained for video generation, both models are evaluated under a controlled setting with a fixed sequence length 4096 tokens to reflect inference cost. a long-horizon setting. Notably, ACDIT consistently faster than MAR even for standard image generation. This shows that KV-cached block-wise autoregression already improves efficiency for common image-generation workloads. The advantage becomes substantially larger for long sequences. While MAR recomputes full bidirectional attention at each generation step, ACDIT caches key-value pairs of clean past blocks and only performs attention on newly generated blocks, avoiding redundant computation and achieving much lower (10x) latency for long-horizon horizon vidco generation.
Scaling Performance. We present the scaling performance of ACDIT in Fig. 6a. For a fair comparison with DiT, we use the same batch size and learning rate as DiT in these training sessions. When increasing the model size, ACDrT shows consistent improvement image quality across all autoregressive lengths, sharing similar scaling trend with DiT. Notably, the performance gap between different autoregressive lengths shrinks as model capacity grows. This suggests that larger models more accurately fit each autoregressive conditional distribution, thereby reducing the accumulation of errors over long autorcgressive horizons.
We ablate the effectiveness of ROPE-ND on image generation. As shown in Fig 6b, both removing ROPE-ND and replacing RoPE-ND with standard ID-RoPE consistently degrade FID.
ROPE-ND.
Effect of clean-past conditioning. We further ablate the effect of clean-past conditioning by replacing clean context latents with noisy ones when modeling autoregressive conditionals. As shown in Fig. 6b, removing clean-past conditioning leads consistently worse generation quality. Beyond generation, using clean latents also provides benefits for representation learning, as evidenced by the improved downstream classification performance discussed in Table 3. These results indicate that clean-past past conditioning is beneficial for both autorogressive generation and learning informative visual representations.
Training dynamics of WSD scheduler. We utilize the WSD learning rale scheduler (Hu el al., 2024). WSD scheduler uses constant learning rate during main training, while allowing divergence at any point with a rapidly dccaying learning rate based compute budget. As Fig. 6c shows, the FID remains almost converged during the constant learning rate state, while sharply dropping after the learning decays, similar to the loss curve when using the WSD scheduler in LLM training. To the best of our knowledge, we are the first. to validate the effectiveness of the WSD scheduler in visual generation.
6 Conclusion
In this paper, we propose ACDIT that interpolates the autoregressive modcling and diffusion transformers. With a simple but novel design of atlention mask, ACDIT can achieve auloregressive generation on any length while maintaining a clear latent input potentially for adding a visual understanding task. By combining the advantages of both autoregressive and diffusion models, we demonstrate the performance and efficiency of ACDIT in both image and video generation tasks. We hope ACDrT can shed light on designing the new visual autoregressive model and building a unified multimodal model in the future.
11

---

Published in Transactions on Machine Learning Research (01/2026)
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tgge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXic preprint arXiv:2303.08774, 2023.
Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sckhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models. arXiv preprint arXim:2503.0955 2025.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, al. Qwen technical report. arXiv preprint (rXiv2230.16600) 2023.
Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In CVPR, 2023.
Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michacli, et al. Lumiere: A space-time diffusion model for video generation.
Xrxiv preprint (tt:2401.129444 2024.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schmurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Vidco generation models as world simulators. 2024. URL https://openal com/research/ srggeerrttoormmdell------rrrrrrr
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14163 2020.
Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In Forty-first International Conference Machine Learning, 2024.
Boyuan Chen, Diego Marti Monso, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. arXiv preprint ar 2070:33,07 2024a.
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-a: Fast training of diffusion transformer for photorcalistic text-to-image synthesis, 2023.
Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Ti.. Pixart-\sigma: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. arXiv preprint arXiv:4403.46692 2024b.
Junsong Chen, Yue Wu, Simian Luo, Enze Xie, Sayak Paul, Ping Luo, Hang Zhao, and Zhenguo Li. Fixart-('ydllta) Fast and controllable image generation with latent consistency models. ar ar preprint or arx:iv:441 002252, 2024c.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Picter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-1555 15097, 2021.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Tyy Sutskever. Generative pretraining from pixels. In International conference on machine learning, pp. 1691 1703. PMLR, 2020.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780 8794, 2021.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering -o-tto- generation via transformers. Advances in 2021.
neural information processing systems,
12

---

Published in Transactions on Machine Learning Research (01/2026)
Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang. Cogview2: Faster and better text-to-image generation via hierarchical transformers. Advances in Neural Information Processing Systems, 35:16890-16902, 2022.
Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, Xiangwen Kong, Xiangyu Zhang, Kaisheng Ma, and Li Yi. DreamIIIM: Synergistic multimodal comprehension and creation. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreviin net/forum?id
Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint AXtt:2010.11929, 2020.
Yilun Du, Mengjaa Yang, Pete Florence, Fei Xia, Ayzaan Wahid, Brian Ichter, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua B Tenenbaum, et al. Video language planning, arXiv preprint orXic:2310.10625, 2023.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Froderic Boosel, et al. Scaling rectified flow transformers for high-rcsolution image synthesis. In Forty-forst International Conference on Machine Learning.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVE conference on compuler vision and pallern recognition, pp. 12873 12883,
2021.
Tsu-Jui Fu, Licheng Yu, Ning Zhang, Cheng- Cheng-Yang Fu, Chyg Su, William Yang Wang, and Sean Bell. Tell me what happened: Unifying text-guided video completion via multimodal masked video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Poltern Recognition, pp. 10681 10692, 2023.
Yu Gao, Jiancheng Huang, Xiaopeng Sun, Zequn Jie, Yujie Zhong, and Lin Ma. Matten: Video generation with mamba-attention. arXiv preprint arXiv:2405.0300 2024.
Songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vegan and time-sensitive transformer. In European Conference on Computer Vision, pp. 102-118. Springer, 2022.
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. 2019.
Jiatao Gu, Yuyang Wang, Yizhe Zhang, Qihang Zhang, Dinghuai Zhang, Navdeep Jaitly, Josh Susskind, and Shuangfei Zhai. Dart: Denoising autoregressive transformer for scalable toxt-to-image generation. ar preprint arXit:2410.0811 2024.
Jiatao Gu, Yuyang Wang, Yizhe Zhang, Qihang Zhang, Dinghuai Zhang, Navdeep Jaitly, Joshua M. Susskind, and Shuangfei Zhai. Denoising autoregressive transformers for scalable text-to-image generation. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreviews
forum?id=anDDMMPVic
David Ha and Jürgen Schmidhuber. World models. arXiv preprint 1rX3 2018
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. Proceedings of the SEEE/CVE conference on computer vision and pattern recognition, pp. 16000 16009, 2022a.
Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent vidco diffusion models for high-fidelity long video generation. 2022b.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. Xix preprint ar Xcxx:2207..2595 2022.
13

---

Published in Transactions on Machine Learning Research (01/2026)
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistie models. Advances in newral informalion processing systems, 33:6840-660 (1884006551, 2020.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv: 15868, 2022.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint ar orXix:2404.0033 06395, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXtx:2310.066 06825, 2023.
Yang Jin, Zhicheng Sun, Kun Xu, Liwei Chen, Hao Jiang, Quche Huang, Chengru Song, Yuliang Liu, Di Zhang, Yang Song, al. Video-lavit: Unified video-language pre-training with decoupled visual- motional tokenization. In International Conference Machine Learning, pp. 22185-22209. PMLR, 2024.
Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint drX1::1312.6114 2013.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems, 32, 2019.
Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11523 11532, 2022.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp.
PMLR, 2023.
Tianhong Li, Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. Mage: Masked generative encoder to unify representation learning and image synthesis. In Proceedings of the IEEE/CV Conference on Computer Vision and Pottern Recognition, pp. 2142-2152, 2023.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. arXie preprint or Nrx:::2006.1 1188, 2024a.
Xiang Li, Hao Chen, Kai Qiu, Jason Kuen, Jiuxiang Gu, Bhiksha Raj, and Zhe Lin. Imagefolder: Autoregres- sive image generation with folded tokens. arXiv preprint arXiv:2410. 10t-22400.11666 01756, 2024b.
Xiang Li, Kai Qiu, Hao Chen, Jason Kuen, Zhe Lin, Rita Singh, and Bhiksha Raj. Controlvar: Exploring controllable visual autoregressive modeling. arXic preprint arXiv:2406. 09750, 2024c.
Loshchilov. Decoupled weight decay regularization. arXie preprint arXiv: 1 0711.05101 2017.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310.1683, 2023.
Cheng Tu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775 5787, 2022.
Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. arXiv preprint XrXi:22033.080 08320, 2023.
Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint S5a:2401.03044 2024.
Volodymyr Mnih. Playing atari with deep reinforcement learning. arXiv preprint ar 1312 2013.
14

---

Published in Transactions on Machine Learning Research (01/2026)
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pp. 1162-8171. PMTR, 2021.
Yatian Pang, Peng Jin, Shuo Yang, Bin Lin, Bin Zhn, Zhenyu Tang, Liuhan Chen, Francis EH Tay, Ser-Nam Lim, Harry Yang, et al. Next patch prediction for autoregressive visual generation. ar preprint WrXiv:2412113322 2024.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
William Peebles and Saining Xie. Scalable diffussio models with transformers. arXiv preprint XrW:222212009748 2022.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXie preprint arXix:2307.011 01952, 2023.
Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yaa Ma, Ching-Yau Yao Chuang, et al. Movic gen: A cast of media foundation models. arXiv preprint or Xiv: 10. 10.11720.0000 13720, 2024.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Πyya Sutskever. Zero-shot, text-to-image generation. In International conference on machine learning, pp- 8821 8831. Pmir, 2021.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth- Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. ar preprinl ar Xiv:2200. 06175. 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffussin models. In Proceedings of the IEEE/CVF conference on compuler vision and pattern recogmition, pp. 10684-066000 10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. ArXiv, abs/ bbs//1555.0455 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Audrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-252. 2015.
Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 011130136-130184 2024.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. XiXiv preprint arXiv: 10TT::777703344 7034777 2017.
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and generalized masked diffusion for discrete data. Advances in neural information processing systems, 37:103131-103167, 2024.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint ar arXiv:2010.025002 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint 2xXi:2011..3455 2020b.
15

---

Published in Transactions on Machine Learning Research (01/2026)
Khurram Soomro, Amir Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. Arxiv, abs/1212.0402, 2012. URL https://api semanticscholas wwt.scchooo.....oorroioieeoee
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:122 167:127063, 2024.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng. Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. ar preprint urXiv:2406.0033 06525, 2024.
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024.
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprinl or Xic:2404.02905, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient. foundation language models. arXiv preprint Xix:22200. 18971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. Xix preprint or Xii:23077002288 2023b.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural informalion processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NewrIPS, 30, 2017.
Junke Wang, Yi Jiang, Zehuan Yuan, Binyue Peng, Zuxuan Wu, and Yn-Gang Jiang. Ommitokenizer: joint vmdge-video tokenizer for visual generation. arxiv preprint arXiv:2406.0988 2024a.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. Xi preprint 18880 2024b.
Mark Weber, Lijun Yu, Qihang Yu, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang- Chieh Chen. Maskbit: Embedding-fiee free image generation via bit tokens. arXio preprint orXiv:2409.162 16211, 2024.
Jialong Wu, Shaofeng Yin, Ningya Feng, Xu He, Dong Li, Jianye Hao, and Mingsheng Long. ivideogpt: Interactive videogpts are scalable world models, 2024a.
Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xic, Hongxu Yin, Li Yi, et al. Vila-u: a unified foundation model integrating visual understanding and generation. orXiv preprint XrXi::2409.0 0499,, 2024b.
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. 12528, 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint Xrx:22447 10671, 2024.
Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning intcractive real-world simulators. XrXiv preprint arXiv:2310.06114, 2023.
Ziyu Yao, Jialin Li, Yifeng Zhou, Yong Liu, Xi Jiang, Chengjie Wang, Feng Zheng, Yuexian Zou, and Lei Li. Car: Controllable autoregressive modeling for visual generation. arxio preprint ar 10.0664,, 2024.
16

---

Published in Transactions on Machine Learning Research (01/2026)
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint (tto:2200.108899 2(3):5, 2022.
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Mfin-Hsuua Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative vidco transformer. In Proceedings of the IEEE/CVF Conference Compuler Vision and Pallern Recognilion, pp. 10559-10699,
2023a.
Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion-tokenizz is key to visual generation. arXiv preprint arXiv:2310.05797, 2023b.
Chuyang Zhao, Yuxing Song, Wenhao Wang, Haocheng Feng, Errui Ding, Yifan Sun, Xinyan Xiao, and Jingdong Wang. Monoformer: One transformer for both diffusion and autoregression. ar Xio preprinl 0rXhv:4409.16280 2024a.
Haoyu Zhao, Tianyi Lu, Jiaxi Gu, Xing Zhang, Qingping Zheng, Zuxuan Wu, Hang Xu, and Yu-Gant Jiang. Magdiff: Multi-alignment diffusion for high-fidelity video generation and editing. In Europeon Conference on Computer Vision, pp. 205 221. Springer, 2024b.
Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408. 11039, 2024a.
Siyuan Zhou, Yilun Du, Jiaben Chen, Yandong Li, Dit-Yan Yeung, and Chuang Gan. Robodreamer: Learning compositional world models for robot imagination. arXiv preprint (rXiv:2444.2237, 2024b.
17

---

Published in Transactions on Machine Learning Research (01/2026)
A Detailed Discussion about Block Size
In this scction, we provide detailed analysis of the computational cost of ACDIT in terms of FLOPS, comparing it to standard full-sequence diffusion in transformer layers. Let L denote the sequence length, h the hidden size, n the number of attention heads (assumed n < h), 0 the number of parameters in a single transformer layer, and the block size used in ACDIT.
Full-Sequence FLOPS. Each transformer layer has two major computational components:
Linear projections and cerdffrward layers: 2L0
Q K altention dot-products: 4(h + n) zs 4hT,
Hence, the total FLOPS per layer is:
Fiul = 2L0 + 4hL2
ACDIT with KV-Cache. In ACDIT, attention is computed block wise using KV-Cache. The i-th block (of size B) attends to (i B) cached tokens, requiring i Q K dot-products. Summing over all blocks:
L/B Σ
L
L B
/
2 +
i B²
+1
B
So the attention FLOPS becomes:
AllentionACDIT = L LB
2h + 2hLB
Including projection and FFN cost, the total FLOPS under ACDIT
FACDIT = 210 + 2h 2hLB
FLOPS Savings. The absolute FLOPS saved per layer is:
F Faueed Ffull
FACDIT = 22hL2 P,)
The relative savings are:
Fraved 21 - B)
Fial 2L0 + 4hL2
2 hl
When L >> h, the savings approaches [1 R), implying up to 50% FLOPS reduction when B <
In practice, is not always beneficial for small B. Setting B to an excessively small value may not fully leverage the iterative modification inherent in the diffusion process, potentially compromising generation quality. Purthermore, given the parallel computing nature of computational kernels, very small B may not yield speed improvements, phenomenon analogous to the rationale behind speculative decoding (Leviathan et al., 2023). Conversely, setting B to a very large value diminishes efficiency both in terms of attention calculation. It also fails to capitalize on the strengths auto-rogressive generation. Indeed, when is set equal to L, ACDIT reverts to the original DiT model.
B Details of Model Architecture and Implementation
ACDIT mainly inherits the main architecture of DIT. We replace the original bidirectional attention with the proposed SCAM attention pattern to process the clean and noisy latent. Since we want to keep the architecture as simple and unified as possible, we use linear layers instead of convolution in the input layer and final layer. Besides, replace the position embedding and Layer Normalization with RoPE (Su et al.,
18

---

Published in Transactions on Machine Learning Research (01/2026)
2024) and RMSNorm (Root Mean Square Layer Normalization) (Touvron et al., 2023a), respectively. We find that QK-norm is important to stabilizing the video generation training, thus we use QK-norms in all experiments. The additional conditional information, including timesteps and labels, is injected into the model with AdaLN-Zero only on the noise part. For both image and vidco generation, we follow DiT and leverage the pre-trained image VAE (Kingma, 2013) from Stable Diffusion (Rombach et al., 2022), whose downsample factor is For image generation under B block size, we group square latent representation patches with vB x vB shape as block. We do not further train the tokenizer in the target dataset since it's out of the scope of our work, despite that further training might yield better results.
In the image gencration task, we set the patch size as and the autoregressive unit block size as 256 = 16 x 16. Therefore, for 256 x 256 x 3 image in 32 x 32 x latent shape, the total sequence length and autoregressive length are 1024 and 4, respectively. We explore different model sizes, as shown in Table 5. ACDIT-B is used for design verification and analysis. ACDIT is trained on ImageNet for 1.2M iterations with a batch size of 1024. We use the AdamW optimizer (Loshchilov, 2017) and WSD (Warmup Steady Docay) Icarning rate scheduling (Hu et al., 2024) with the peak learning rate 3e-4 and weight decay. The learning rale begins to decay in the last 15% training iterations. Following the common training recipe of generative models, we keep exponential moving average (EMA) of the ACDIT weights during training using decay rate of 0.9999. We sample images with DPM-Solver (Lu et al., 2022) for 25 steps within each block and use classifier-fres guidance (Ho & Salimans, 2022) with a guidance scale of 1.5. In video generation, we sample 16 frames from each video and set the patch size as 2 and the block size as 1024 = 256 For a 16 x 256 x 256 video in latent shape, the sequence length of each frame is 256 and the total sequence length is 4096, with 4 frames grouped into block. We train ACDIT UCF-101 for 400K iterations with batch size of 96. The classifier-free guidance scale is 2.5. Other training configs are the same image training. All models are implemented with Py PyTorch (Paszke et al., 2019). Specifically, use Altentttnn to implement the SCAM for both customization and efficiency. We use 256 and 96 NVIDIA H100 GPUs to train the image and video generation model, respectively.
16 x 32 x
C ACDiT PyTorch Code Example
In this section, provide key implementation details of ACDIT, including (1) how clean and noisy blocks are arranged in the sequence, (2) how the Skip-Causal Attention Mask (SCAM) is constructed, and (3) how SCAM integrates with the attention module and KV-cache for efficient autoregressive inference.
For N autoregressive blocks, we build a sequence of length 2N containing all clean blocks followed by all noisy blocks:
idx cdx_cceenn()) = idx adx_noisy(i)-aa = N +
The following code shows how to construct SCAM, where noisy blocks are only allowed to attend to all previous clean blocks and themselves, while clean blocks attend to all preceding clean blocks. This attention mask enforces clean-past conditioning and enables KV-cache reuse across blocks.
def build cecteentionmaak(eeee
т, B, device):
size B
m_ m__oise_noise = torch. zeros size)
for range(T):
start_idh = - B
end idx start_ids
g_moise_noiseli iesstart_ixxeedd_itttt idx, start_ida sarerend_dde = torch. ones B) m. s__noieeclean torch. zeros size)
for i in range(T):
10
for j in range(1 1, T): start_col = end_col. = start_co col
12
start_row end = start_roo row
14
9htpp://ppttorch.on///loggggg
19

---

Published in Transactions on Machine Learning Research (01/2026)
m_ ___oise__tt clean (start, enntart_row:edd...m start_col: start_col:end_col]
18
m_ clean clean_noise moise = torch. zeros (size, size)
m_ clean clean torch zeros (size, size) for i in range(T):
18
18
start_ide idx
end_idx iix = start_ idx + B m_ (ltarcllaaassar_iaaaaaaaa
21
:end_idx]
23
attn_mask corch.zeros size, size)
attn attn_mask[::s122 :size]
clean
25
attn_mask sisze, size:] m_clean _oise
attn_mask [size:, saiee m_noise_clean
27
attn_mask (size; size:] m_noise_noiis
return attn_mask: (onn_mask.bboo()) bool() (device)
If using PyTorch FlexAttention, the same causal dependency can be implemented through its index-based mask generation. Below is the corresponding function:
def casp_ccasal___ae lait___ssss______e__ mod_gen h, q_idx, kv kv_idx, block _aze_ len1):
__idx = q_idx // block block_size size
kv kv_idx kv_idx // block_size
mask torch. where (((q_ idx < (een) & (kv_idx idx len1) & (q_ idx >= kv [w_idx))] ((q_ idx len1)
(kv_idx len1) & ((q_ idx len1) kv xv_id))) {q_idx kv idx) True, False)
mask
Once the mask constructed, is passed the attention module, KV-cache is applied only clean blocks, allowing us to cache context features while discarding the noisy-block activations after each diffusion step.
class SteppCuusalAttentionn (Atteet)
def forward(self,
Deeition_idssttone
Iiteettion_mss-Hoone block_ Back__size-one, cache=False):
B, N, C x. shape
qkv self oolf.gkv(x) (alf.ggkxx)..rehaaeede reshape N, 3, self num heads, self head head_dim)) permute 0, 3, 4)
9. k, v (kv.unbindd0))
9, self. self.q.norm(q), self. kkommk))
if self is not None:
9, self.rope k, position_ids)
if self. caching:
10
cache:
11
if self cached_ k is None:
12
self colf.cacio cached k[:, :, :block :block_size, :] aaloc_siize, :]
selhecached.3 = v[:,
14
self.cached_x else:
16
cached k torch. cat (sself.ccched_.. cached_k, k[;, :, :block_sssiaeeeeeeeeceeeeeeeebed size, :)), dim=2)
self. cached_ torch. cat
v[:, i, blockk :]), dim=2)
18
if self cached is not None:
18
x torch. carch.cat((e cat cat((f hwweefffccceeee....kk k, k[:, i, -block jacck_iize, :)) dim=2)
v = torch. cat (self. cached_v, v[:, :, bblockssize: ;]), dim=2)
21
not USE FLEX FLEX_ATTENTION
x = torch, nn
dot crot__prodct_eee produuct_attentinn
23
9. k, v, bett__assratteetion..em mask, tropout_p=selff... attnn_doopppppp drop.p
25
else:
flex_attee _ax__tteentio(q, k, block_mand wwt.nnkkaattettin__aaaaa
27
x x. transspseee1, 2) reshape N, C)
self.proj (x)
29
x self. prtf..oo..co holt.proj.drop(xxx
20

---

Published in Transactions on Machine Learning Research (01/2026)
ACDIT uses standard Transformer block design in DiT, with key modification: only the noisy latent. tokens interact with the conditional information (diffusion timestep, class label). Clean latents remain unconditioned to preserve their deterministic role as the cached context.
class ACDiTBlock (nn. Module):
def forward x, c, Actention_mask-Noont cond_length-O, block size=None, cache=False, position_ ids=None):
N, C c.ahape
N, TB, C x(:, lengh:eeehhh
//
ada c list self adaLN (oddllatont)))) c6 dim=-1)
(shift_man msa, scale_mss, m_sa, gate_msa, shift_mlp, scale_mlp, gate_mlp) [ada_c.repeatee 1, reshape (N, TB, C)for ada_c ada_c_list]
norm_xl = self normi (x (torch.. flocct2))))))) (oattt3)))ttodyyyp))) (dtype))
10
attn_input_x torch. cat (norm_noti xi[:, :cond_ nengthen modulate (norm_xim___) xi[; [;, cond Length])
shift_mss
(amee_ssa)aee ddmmm)
attn_output_= = self attt bictaatttninnu________ attention wtwp:oonnsssaattttttnmaaaaaeaaee
block_ swwe=slocecs_eeeee size, cache=cache, position_ ids=position_ide
= x + torch. cat ((aat(( cond_length:]] dim=1)
:cond_length), gate_msa attn_output_xl:;
14
norm_ x2 self norm2 to(torch...... (((torch.llott2))) to (dtype)
gate_input, torch. cat (norm__ea x2[:, :cnnd length], modulate modulate(norm_xcd _2__x cond length:], shift_mlp peptt_mppp,.. scale_mlp)), dim=1)
gate_ cste_output_x = self (gate ([(gate_innuxxx))
17
x x + torch. cat
scond_lengtin gate_mlp gate cate_output_x(::
cond cond_length:]) dim-1)
18
return x
Broader Impact
This work advances the efficiency and scalability of generative models for images, vidcos, and text by combining autoregressive modeling with diffusion. While such models enable positive applications in creative content generation, simulation, and reprosentation learning, they may also be misused generate misleading harmful visual textual content, including deepfakes. As with other general-purpose generative models, responsible deployment requires appropriate safeguards, dataset curation, and usage policies. We encourage future work to explore detection, watermarking, and alignment techniques alongside improvements in generative quality and efficiency.
E Qualitative Results of ACDiT
We show the qualitative results of ACDIT in Figure 7.
21

---

Published in Transactions on Machine Learning Research (01/2026)
Figure 7: Sample images from ACDIT-H on ImageNet 256x and sample videos from ACDIT-XL trained
UCF-101.
22


Reviews:
[{"review_id": "PkIGbeI5FJ", "reviewer": "Reviewer_Qdvv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary:\nACDiT fuses block-wise autoregression and conditional diffusion in a single Transformer via a lean SCAM mask, delivering high-quality, KV-cached generation of any length.\nOperating on continuous visual tokens, it outperforms same-scale AR baselines and rivals full-sequence diffusion on ImageNet/UCF-101, offering a new baseline for unified multimodal modeling.\n\nStrengths:\n1. The method is the first to unify block-autoregression and intra-block diffusion in an end-to-end vision framework without relying on vector quantization.\n2. The combination of KV-cache and block-wise attention provides substantial reductions in FLOPs and wall-clock time, especially for long sequences.\n3. A single architecture is shown to work competitively across image, video, and text generation, demonstrating strong cross-modal scalability. Generative pre-training produces representations that transfer effectively to ImageNet classification, achieving accuracy comparable to masked-generation approaches.\n\nWeaknesses:\n1. The selection of key hyper-parameters, such as block size B and patch size, is fully empirical, and no adaptive or dynamic scheduling strategy is explored.\n2. The benchmarks are restricted to ImageNet and UCF-101, and the paper does not report quantitative results at higher resolutions (≥512²) or for longer videos (≥64 frames).\n3. The FID and FVD scores remain behind state-of-the-art diffusion models such as SDXL and Sora, and it is unknown whether scaling the model would narrow this performance gap.\n4. The text-generation section is overly brief, providing no details on the experimental setup and no description or definition of the PPL metric, and the presented text-generation results are only small-scale demonstrations without any evaluation on downstream NLP tasks such as summarization or machine translation.\n5. The theoretical analysis is limited, offering no examination of error propagation in block-wise autoregression and no formal discussion of the expressive capacity or sparsification limits of the SCAM masking mechanism.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The core claims are supported by clear and sufficient evidence. Results on ImageNet and UCF-101 across multiple model sizes, together with standard metrics and ablations on block size and RoPE-ND, consistently show ACDiT outperforming AR baselines while approaching full-sequence diffusion. The KV-cache and downstream classification results further substantiate the claims. Aside from missing higher-resolution or longer-sequence evaluations, the evidence provided is accurate, convincing, and well presented within the stated scope.", "audience_yn": "Yes", "audience_explanation": "Yes. TMLR readership includes a large community working on generative vision, autoregressive modeling, and unified multimodal transformers; the paper shows how to fuse diffusion with AR using only an attention mask and offers reproducible gains on ImageNet/UCF-101 while enabling KV-cache acceleration—practical techniques that many researchers in that audience actively seek.", "requested_changes": "1. The model’s effectiveness has not been validated on larger or more diverse datasets, and the reliance on UCF-101—which contains low-resolution and relatively simple videos—limits the strength of the empirical claims for real-world video generation.\n\n2. Clarify how block size B should be chosen when scaling to higher resolutions (512²/1024²/1080p).\nA brief guideline or empirical observation would help readers understand how to maintain a good quality–efficiency trade-off. If dynamic or content-adaptive block sizes are feasible, mentioning this would further clarify future directions.\n\n3. Discuss model-scaling effects on AR error accumulation.\nSince the current FID is slightly above DiT-XL, explaining whether larger models (e.g., 2B–4B parameters) may worsen AR error propagation—and whether you plan to use regularizers or noise-schedule adjustments—would strengthen the claims.\n\n4. Address memory scalability for long videos.\nKV-cache grows linearly with block count for >100-frame generation. Commenting on possible solutions such as windowed attention or memory compression would help clarify long-form applicability.\n\n5. Provide some discussion on exposure bias introduced by clean-past conditioning.\nThe training–inference mismatch may accumulate errors during generation. Potential fixes (e.g., adversarial or RL-based fine-tuning) could be briefly mentioned as future work.\n\n6. Expand on the robustness of RoPE-ND under different aspect ratios.\nSince the wavelength design depends on per-dimension maxima, clarifying how well it generalizes to varied spatial–temporal shapes, and whether additional ablations will be included, would improve completeness.", "broader_impact_concerns": "No more broader impact concerns needed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FZ1qW1LC6E", "reviewer": "Reviewer_BCof", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes ACDiT, a simple yet powerful method for interpolating AR and diffusion models by introducing block-wise autoregressive conditional diffusion. The core technical contribution is the Skip-Causal Attention Mask (SCAM) applied to a standard Diffusion Transformer (DiT), which allows clean past blocks to be cached while each new block is generated via a conditional diffusion process. The approach works without discrete tokens, supports images, videos, and text with the same architecture, enables KV-caching for arbitrary-length generation, and shows promising transfer to visual understanding.\n\n\nOverall, this paper is well-written, and the motivation is effectively presented. The introduced method is elegant, minimal modification to DiT, offers a theoretical complexity advantage for long sequences, and avoids the pitfalls of discrete tokenization.\n\nTo me, this paper can be accepted with minor revisions regarding the following suggestions:\nThe current submission completely lacks experiments, quantitative results, ablation studies, and inference speed measurements, making all performance claims unsubstantiated.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper makes very strong claims, such as “performs best among all autoregressive baselines on similar model scales”, “achieves visual quality comparable to full-sequence diffusion models while exhibiting higher inference speed when extended to long sequences”, and “achieves strong performance on text generation”.  There are no experimental settings, datasets, model sizes, FID/FVD scores, speed measurements, or ablation studies. Without empirical evidence, the central claims remain unsupported, which prevents acceptance in their current form.", "audience_yn": "Yes", "audience_explanation": "Combining autoregressive and diffusion modeling in a clean, cacheable, tokenization-free way is currently one of the hottest topics in generative modeling (see concurrent works such as Block Diffusion, MAR, Diffusion Forcing, Transfusion, etc.). If the empirical claims hold even approximately, ACDiT would be a significant advance, especially for long video and world-model-style applications. The idea is simple enough to be widely adopted and has clear scaling implications.", "requested_changes": "1. Add a complete experimental section with quantitative results on image generation (e.g., class-conditional ImageNet 256×256 and zero-shot COCO FID/IS), video generation (e.g., UCF-101 or something comparable to OpenSORA), and text generation (e.g., Zero-shot LM perplexity or OpenWebText). Include main comparison tables against DiT-scale diffusion baselines, VAR/LlamaGen-style pure AR baselines, and recent hybrid methods (MAR, Diffusion Forcing, Block Diffusion, etc.). Without numbers, the paper cannot be accepted.\n\n2. Provide theoretical and empirical inference complexity analysis. The main selling point (“higher inference speed when extended to long sequences” + full KV-cache support) currently has zero supporting data. Please add:\n- A clear complexity table (FLOPs or attention cost) comparing DiT, token-wise AR, and ACDiT as a function of sequence length and block size\n\n- Measured generation time (seconds per image or seconds per frame) for different block sizes and diffusion steps T\n- Speed vs. quality curves for long videos (≥100 frames) compared to full-diffusion baselines\nThis is essential to substantiate the long-sequence advantage.\n\n3. Significantly strengthen the comparison with the most related concurrent work Block Diffusion (Arriola et al., 2025). The current treatment (“they focus solely on text”) is insufficient, given the near-identical high-level idea. Provide a detailed side-by-side comparison of training objectives, attention masks, and inference procedures, and include empirical numbers (at least on text, ideally also vision) showing where and why ACDiT outperforms it.\n\n4. Clarify implementation details of SCAM and KV-caching. Specify exactly how clean and noisy blocks are concatenated (single sequence of length ≈2×blocks or separate streams?), provide pseudocode or a short PyTorch snippet showing how to apply the mask in an existing DiT code base (<20 lines as claimed), and confirm compatibility with FlashAttention-2/3 and exact cached KV shape.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "frR7H5NlUj", "reviewer": "Reviewer_9z82", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Brief Summary**\n\nThe paper proposes a transformer architecture called ACDiT to jointly perform diffusion and block-wise autoregressive decoding for image and video generation tasks.\n\n**Strengths**\n- S1. The underlying motivation to leverage the success of autoregressive decoding from LLMs for visual generation tasks definitely makes sense and is an interesting direction to explore.\n- S2. The paper does a good analysis of what's missing from the other recent autoregressive diffusion models with respect to the criterion defined in Sec. 4.1 and how ACDiT can achieve them.\n- S3. The image generation performance is very competitive across different model sizes which demonstrates the efficacy of the model architecture, training and inference strategy.\n\n**Weaknesses**\n- W1. For short-form video generation, the rationale for lower performance than some baselines with smaller model sizes is unclear. The authors mention that ACDiT may perform better for longer-form videos but there's no evidence presented for it.\n  - W1.1 Is ACDiT truly modality-agnostic as the subsection for text generation mentions? It's unclear how the ACDiT architecture / training / inference would need to be modified for doing this and what's the magnitude of such modifications.\n  - W1.2 The image classification performance in Table 3 doesn't describe the model sizes considered for the baselines which is a very important factor for performance variations on a competitive benchmark.\n\n- W2. ACDiT involves many design choices such as Skip-Causal Attention Mask, RoPE-ND and while they're meant for meeting the criterion defined in Sec 4.1 / multi-dimensional data, it's unclear whether these are the only feasible options.\n  - W2.1 Fig. 6(b) only seems to ablate RoPE-ND v/s no RoPE at all but not vanilla RoPE. Also there are several popular variants of multimodal RoPE such as those adopted by Qwen-VL series, is there a reason why those aren't applicable for ACDiT?\n\n- W3. The authors can provide more insights into their claim of better inference compute utilization in Fig. 5 by comparing against popular methods (e.g. DiT-XL/2, MAR-L) especially for longer visual sequences.", "claims_and_evidence_yn": "Yes", "claims_explanation": "- The overall claim of joint autoregressive diffusion modeling is met through the proposed Transformer-based ACDiT architecture and most of the experiments and theoretical justifications shown.\n- However, some other claims such as autoregressive modeling being as beneficial for video generation (W1 above) and some design choices (W2 above) can be better justified through more extensive and controlled experiments.", "audience_yn": "Yes", "audience_explanation": "- Yes, the paper will be useful for practitioners working on visual generation tasks. Some suggestions (W3 above) would further help practitioners on when to opt for ACDiT instead of other available methods.\n- Researchers would also find this paper insightful overall as it's well grounded in the limitations of existing literature on autoregressive diffusion modeling.", "requested_changes": "- [Critical] Better justification for lower performance on short-form video generation than baseline methods. If their limitation is long-form video generation, some early quantiative / qualitative evidence should be included.\n  - Alternatively, their image generation performance can be included so that the claim can be made in favor of ACDiT being a single model effective for both image and video generation tasks.\n- [Strengthening] Other suggestions listed in the weaknesses section above.", "broader_impact_concerns": "- The authors should include a broader impact section to discuss the potentially negative effects of automated visual generation, related but not limited to generation of harmful images and videos, intended audience, etc.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qcoF3N3Vcb", "reviewer": "Reviewer_T9BM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "#### **Summary**\n\nThis manuscript proposes ACDiT, a visual generative model combining auto-regressive model and diffusion model. Though there were several works in this direction, the authors proposed three research questions: 1) The generation of future elements should be predicated on a precise representation of antecedent sequences (not noisy inputs), 2) both the auto-regressive modeling and the denoising process should optimally utilize the entire parameter space of the neural network (not utilizing a small submodule), 3) the denoising process should directly attend comprehensively (not a compact hidden state). To implement these points, the authors introduced a Skip-Causal Attention mask structure (SCAM), which attends to the clean previous patches, and RoPE-ND, a multi-dimensional extension of RoPE positional encoding. They compared the performance on image, video, and language generation tasks.\n\n#### **Strength**\n\n- Though the motivation for adopting auto-regressive structure is vague for now, I guess there should be application domains where combining auto-regressive and diffusion structures is necessary.\n- Overall performance is comparable to the other SOTA models.\n\n#### **Weakness**\n\n- The main weakness of the current form of this manuscript is that it does not fully address the main research questions proposed. For each question:\n  - *1) The generation of future elements should be predicated on a precise representation of antecedent sequences* $\\rightarrow$ **how the overall performance changes if ACDiT utilizes the noisy input patches?**\n  - *2) both the auto-regressive modeling and the denoising process should optimally utilize the entire parameter space of the neural network* $\\rightarrow$ **how model can become faulty if we use only small submodule?**\n  - *3) the denoising process should directly attend comprehensively* $\\rightarrow$ **what happen if we use only a compact hidden state?**\n- These questions should be proven either theoretically or empirically, not just depending on words. Authors could answer these questions by comparing the other models opposing these statements. But MAR, which opposes the 2) and 3), achieves a much better performance with a much smaller model size. Though authors tried to justify the benefit of ACDiT with the KV-Cache argument, it does not directly address the main research questions above.\n- I'm also not sure about the KV-Cache argument. Though MAR utilizes the bi-directional attention structure, so re-computing the representation is necessary for each patch, it seems to only use a small submodule in diffusion steps. Meanwhile, though ACDiT uses the KV-Cache for each patch, it should do diffusion for each patch with the whole transformer model. It would be more persuasive if an empirical computational budget (like flops) were compared.\n- The motivation of the work is not that convincing. In the Introduction, the authors mentioned the 'predicting the future' argument, but I'm not sure whether auto-regression+diffusion is necessary in the image generation task. (But as stated in the strength part, I guess some domains would require this structure.)\n- Some additional arguments are not justified: authors iteratively mention the world model and long-video generation, but it only remains as a possibility, and not verified. Additionally, RoPE-ND is proposed for D>4 dimensions, but tested only D<4, which seems simply an original RoPE.\n\n#### **minor points**\n- I don't think the abbreviation 'SCAM' is good for the overall credibility of the manuscript.\n- Please add explanations on unexplained arguments of Table 1, e.g. VAR, MAR, Pre, and Rec.", "claims_and_evidence_yn": "No", "claims_explanation": "There were three scientific questions: 1) The generation of future elements should be predicated on a precise representation of antecedent sequences, 2) both the auto-regressive modeling and the denoising process should optimally utilize the entire parameter space of the neural network, and 3) the denoising process should directly attend comprehensively. But authors did not directly answer these questions, and MAR, which opposes these questions, even outperforms the proposed ACDiT.", "audience_yn": "Yes", "audience_explanation": "Yeah, sure. Image generation is a key area in modern deep learning. I also believe the auto-regression and diffusion structure can be beneficial for some domains.", "requested_changes": "- Please directly address the three main research questions. Maybe an ablation study would be necessary.\n- It would be great to see a key application domain of ACDiT. Currently, the benefit of merging two structure is vague.", "broader_impact_concerns": "I don't think there is an ethical concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)