$\textit{MADFormer}$: Mixed Autoregressive and Diffusion Transformers for Continuous Image Generation
Authors: Junhao Chen
Yulia Tsvetkov
Xiaochuang Han
Venue: ICLR 2026 Poster (2026)
Keywords: Autoregressive
Diffusion
Continuous Image Generation
Primary area: generative models

Abstract: Recent progress in multimodal generation has increasingly combined autoregressive (AR) and diffusion-based approaches, leveraging their complementary strengths: AR models capture long-range dependencies and produce fluent, context-aware outputs, while diffusion models operate in continuous latent spaces to refine high-fidelity visual details. However, existing hybrids often lack systematic guidance on how and why to allocate model capacity between these paradigms. In this work, we introduce $\textit{MADFormer}$, a Mixed Autoregressive and Diffusion Transformer that serves as a testbed for analyzing AR-diffusion trade-offs. $\textit{MADFormer}$ partitions image generation into spatial blocks, using AR layers for one-pass global conditioning across blocks and diffusion layers for iterative local refinement within each block. Through controlled experiments on FFHQ-1024 and ImageNet, we identify two key insights: (1) block-wise partitioning significantly improves performance on high-resolution images, and (2) vertically mixing AR and diffusion layers yields better quality-efficiency balances---improving FID by up to 75\% under constrained inference compute. Our findings offer practical design principles for future hybrid generative models. Code and models will be released upon publication.

Published as a conference paper at ICLR 2026
MADFormer: MIXED AUTOREGRESSIVE AND DIFFU- SION TRANSFORMERS FOR CONTINUOUS IMAGE GEN- ERATION
Junhao Chen Tsinghua University chenjunh222mail www.tssiinnggice... edua en
Yulia Tsvetkov University of Washington
yuliats@cs washington, edd
Xiaochuang Han
University of Washington xhan77@cs washington.ed. edu
ABSTRACT
Recent progress in multimodal generation has increasingly combined autoregressive (AR) and diffusion- basod approaches, leveraging their complementary strengths: AR models capture long-range dependencies and produce fluent, context-aware outputs, while diffusion models operate in continuous latent spaces refine high- fidelity visual details. However, existing hybrids often lack systematic guidance on how and why allocate model capacity between these paradigms. In this work, we introduce MADFormer, Mixed Autoregressive and Diffusion Transformer that serves as a testbed for analyzing AR-diffusion trade-offs. MADFormer partitions image generation into spatial blocks, using AR layers for one-pass global condi- tioning across blocks and diffusion layers for iterative local refinement within each block. Through controlled experiments on FFHQ-1022 and ImageNet, we identify two key insights: (I) block-wise partitioning significantly improves performance on high-resolution images, and (2) vertically mixing AR and diffusion layers yields better
balances- valances-immroving. FID by up to 75% under constrained inference compute. Our findings offer practical design principles for future hybrid generative models. Code and models will be released upon publication.
INTRODUCTION
Multimodal generative models have achieved remarkable success in vision- languag tasks such as text-to-iman -imag generation, image captioning, and video synthesis (Saharia et al. 2022 Luo et al. 2022
Han et al. 2022] Zhang et al. 2024).. These systems typically adopt autoregressive (AR) models for language, generating discrete tokens
time with strong contextual coherence, and diffusion models for image generation, operating in the continuous domain by progressively denoising latent representations. This division leverages the strengths of both: AR excels at structured sequence modeling, while diffusion achieves high image fidelity through iterative refinement.
For the image generation component, current approaches follow three main paradigms. One line of work uses AR models on discrete visual tokens, enabling reuse of large language model architectures but often introducing quantization artifacts and limiting generation quality (Esser et al.) 2020 Team
2024 Agarwal et al. 2025). Another line adopts fully diffusion-
models in continuous latent
spaces, producing high-quality images with global coherence but the cost of slow sampling and high computational overhead (Rombach et al. 2021 Peebles & Xie 2022, Esser et al. 2024 Labs 2025).
third line of research explores hybrid architectures that combine AR and diffusion generation within the image processing pathway, aiming to integrate the efficiency and structure of AR with the fine-grained detail refinement diffusion (Li et al. 2024 Hu et al., 2024). These early efforts point to the promise of unifying the two paradigms, but the design space remains underexplored.
In this work, we present MADFormer, a unified Transformer-basee architecture for continuous image generation that integrates AR and diffusion modeling across two axes token sequences and model

---

Published as a conference paper at ICLR 2026
layers Apart from introducing a novel approach, MADFormer, more importantly, serves testbed for systematically exploring the hybrid design space and deriving generalizable insights. We build on vanilla architecture that employs AR for language and diffusion for image generation, extending with intra-image AR conditioning and flexible allocation of AR and diffusion blocks. By varying structure and capacity distribution between these components, we aim to better understand their interaction and provide actionable guidance for designing efficient, high-quality multimodal models.
Mixing AR and diffluson across tokens. Along the token axis, we partition the image into coarse- grained blocks (e.g.. 16 blocks of 256 x 256 patches for an image with 1024 x 1024 latent patches). AR modeling operates across blocks: generation of each block is conditioned autoregressively on all previous ones. Within each block, we apply a diffusion objective to iteratively denoise continuous latents, overall balancing structure and flexibility.
Mixing AR and diffusion across layers. Along the depth axis, we split the Transformer into AR and diffusion layers. Early layers run autoregressively, processing previously generated blocks to compute prior for the next block. Late layers follow the diffusion objective: conditioned on both the noisy block and the AR output, they denoise towards the ground truth. Importantly, we maintain unified architecture-ont the training objectives differ. This hybrid layout is especially effective under constrained compute: the AR path provides a strong initialization, enabling the diffusion layers to converge with fewer iterations.
We validate our approach on FFHQ- 1024 and ImageNet, with experiments designed to isolate the effects of image partitioning, model depth allocation, inference budget, etc. Our contributions are:
1. We propose MADFormer, a unified Transformer architecture that supports flexible autore- gressive and diffusion integration within the image generation tower. Our design preserves architectural simplicity while enabling structured exploration of AR-diffusion trade-offs.
2. We conduct a study across design axes diffusion depth, AR block granularity, auxiliary modules, and loss designs-derivingg deriving empirical insights into AR-diffusion interactions.
3. We derive practical guidelines for allocating model capacity between AR and diffusion components, showing that AR-heavy models are advantageous under budget constraints, while diffusion-heavy models achieve higher fidelity when compute allows. We find that increasing AR layer allocation can improve FID scores by 60-75% under constrained inference compute.
2 PREREQUISITES
2.1 AUTOREGRESSIVE MODELIIG
AR models decompose the joint probability of a sequence (I1, F2. In) as p(x) I[ p(z;; 1). A Transformer with causal masking estimates each conditional by computing hidden state 2, (x1s-11) p(x; x13-1) p(r; 2)). Training minimizes the negative oogliikeihood, implemented as cross-entropy for discrete tokens or continuous density loss for real-valued inputs. Recent work shows that AR modeling can be extended as well the continuous latent space (I.i et al. 2024
2.2 DIFFUSION
Diffusion models typically learn to reverse a noise process in the continuous space. Starting with clean
latents Fu, the forward process adds Gaussian noise via q( .T-1) =N vI Br 21-1; Bg I), (O, 7). where a; = IT' (1 (13)). The
with direct sampling x; /a, To VI E reverse process uses a network E (æ,, t, 2), conditioned on optional input 2, to model pe (2:-1 xx) =
(zz--1 (2( f, 2), of f). Training minimizes noise prediction error: F s |lea (x),t, - =ll??
At inference, the model denoises xT ~ (O I) into Zy (Ho al. 2020).
OUr integration of AR and diffusion focuses mixing within the image generation process itself, rather than across modalities as in unified models like Transfusion (Zhoo 2024) or Show-o (Xie 2024
2

---

Published as a conference paper at ICLR 2026
old
<fOT>
Text Tower
Transformer
Image Tower
An
old
man
<BOb> <BO
040
Difiusion Module Becoderx
Nolay
.800
DecoderxN
Condition
All Module
Blockwise Causal Attention
Deander ([N-3)
Parameters for image Parameters for text
[Seperate projections, layer noms)
Figure 1: High-level overview of the MADFormer architecture. A single Transformer processes all modalities as a unified sequence. Text tokens follow next-token prediction objective, while image tokens are grouped into blocks trained autoregressively with diffusion objective. We use separate parameters (FFNs, QKVO projections, and layer norms) for each modality. The Transformer is divided into two stages: early layers produce conditions from text and image blocks; later layers denoise noisy image blocks. Each block attends to itself and preceding clean blocks.
(a) Samples FFHQ 1024 1024 with inference steps. Despite minor imperfections in backgrounds and details, MADFormer generates coherent high-res human faces with strong structural consistency in minimal steps.
Samples on ImageNet 256 256. MADFormer demonstrates understanding class prompts and generates well-structured images, though fine details can still be improved with more training.
Figure 2: Qualitative results of MADFormer with 14 diffusion layers (out of 28 total). The AR conditioning mechanism enables strong sample quality even under limited inference steps. More qualitative results can be found in Appendix C
Continuous Image Tokenizers In image generation, we operate continuous VAE latents rather than discrete tokens, leveraging more mature and effective tokenizers without quantization in the latent space (Rombach et al. 202 Agarwal et al. 2025). This choice simplifies AR factorization and diffusion scheduling while preserving high-fideliy visual detail. Continuous latents allow direct modeling with Gaussian noise and real-valued predictions, avoiding discrete codebooks and related
artifacts. (van den Oord et al. 2017 Esser et al. 2020 Mentzer et al. [2023 Yu et al. 2023 Tschannen
al. 2023)
3

---

Published as a conference paper at ICLR 2026
3. MADFormer: A TESTBED FOR ANALYZING AR-DIFFUSION DESIGN TRADE-OFFS
3.1 FRAMEWORK
We introduce MADFormer, unified Transformer architecture designed not only standalone method but as a testbed for analyzing architectural trade-offs between AR and diffusion modeling in continuous image generation. Built on top of recent advances in hybrid generation (Li al. [2024] Zhou et al. 2024| Shi et al., 2024, Hu et al. 2024; Peebles & Xic (022). MADFormer integrates key components from prior work into cohesive and extensible framework. Its modular design enables systematic exploration of how to balance global sequence modeling via AR conditioning and
fine-grained detail refinement via diffusion across both token sequences and network depth.
Data Representation. We preprocess each modality according its characteristics. Text inputs are tokenized into discrete tokens by the Llama 3 tokenizer (Dubey et al. 2024).. Images are mapped into continuous latent space via the Stable Diffusion variational autoencoder (VAE) (Rombach) et al. 202I). We linearize the image latents in left-to-right, top-to- top-to-botton order, delimit them with BOI beginning-offimenn accordance with the original sequence. This produces a unified sequence of discrete text tokens and continuous image latents. Image latent patches grouped into contiguous blocks with bidirectional attention within, and the blocks are treated as tokens in the autoregressive sequence.
and EOI mond-of-imagee tokens, and interleave them with text tokens in
Model Pipeline. Our architecture primarily consists of a single Transformer built from standard Llama decoder blocks, enabling unified AR modeling and diffusion-basee generation in latent space. To project different modalities into this space, we employ modality-specin components: text tokens embedded via learned embedding matrices, while image latent patches processed through U-Net (Nichol & Dhariwal [021] Saharia et al.] (2022) upsampling and downsampling blocks. Unlike the DiT paradigm (Peebles Xie (2022). we do not inject explicit conditioning (e.g., timesteps) into individual Transformer layers. Instead, timestep information is encoded directly into the image latents via the U-Net downsampler.
We conceptually divide the Transformer's decoder stack into two stages: the early layers serve as an AR conditioning module, while the later layers execute recursive diffusion denoising. At each diffusion step, the condition generated by the early layers is added to the noisy latent prior to the denoising loop. The model predicts the denoised latent from the current noisy latent Tt, following a timestep schedule.
In addition, we adopt the strategy from ACDIT (Hu et al. 2024) of prepending clean image blocks to the noisy ones. This enhances contextual information for AR modeling and improves denois- ing fidelity, though incurs additional computational cost. All modalities share the Transformer backbone but separate parameter sets for processing: text, clean image blocks, and noisy image blocks-referred to as the text tower, clean tower, and noise tower, respectively (Liang et al. 2024). Cross-modal interaction is facilitated via interleaved causal attention, with bidirectional attention applied within image blocks. To support mixed clean and noisy inputs, attention masks are dynami- cally constructed using Flex Attention (Dong al. 2024). We demonstrate the effectiveness of this architectural design through ablations in Sec.
3.2 DESIGN SPACE
We explore three primary axes in the design space of our model: diffusion depth, AR length, and loss function. These components fundamentally affect training convergence, generation speed, and output quality, independent of model size.
Diffusion Depth. We define diffusion depth as the number Transformer layers dedicated to the denoising process, corresponding the D decoder blocks shown in the Image Tower of Figure [ While prior work often employs the entire model for diffusion, we argue that this results in unnecessary computational overhead. We posit that single-pass AR conditioning stage can effectively capture both inter-mi modaltty and anter-block dependencies, allowing only a small subset later layers to refine intra- block details through recursive denoising. In Section 4.2 we compare using the full model, 3
4

---

Published as a conference paper at ICLR 2026
2. and 1 of the layers for diffusion, and in Section 4.1 propose optimal AR/diffusion layer ratio under constrained inference budgets.
We formalize this layer-wise structure by formulation: the image tower consists of layers with the first N allocated for AR conditioning and the final D for diffusion-basse denoising. The AR stage computes conditioning representation over previously generated image blocks:
hg Embed(2pee) + PosEnc,
(1)
h; (ecoderBlockk(hh-.)) i == N-D,
(2)
Zeond hw-D-
(3)
Here, PosEnc shorthand for the multidimensional rotary position encoding (RoPE-ND) as proposed in ACDiT (Hu al.) 2024). To begin the diffusion stage, we inject the noised ground-truth latent, doz Zimage + vI dr e, where é is standard Gaussian noise, along with the condition Zoond-
The diffusion layers aim to denoise vot Zimago + V1 Or E into the clean latent 2imags
hN-D D v Zimage vI ai Zcond
(4)
hN-D+J D+j = DecooerrBoock (tN
j=
(5)
The final output Zmmage Proj serves as the model's prediction of the clean latent.
AR Length. We study the optimal granularity for autoregressive processing of image patches. AR length refers the number of image blocks which an input image is partitioned (e-g.. AR length 2 in Figure [ demonstration). Different partitioning strategies significantly affect the quality of AR condition modeling while incurring minimal computational overhead. In Section 4.2] we evaluate splits of 4, 16, and 64 blocks under varying image resolutions, and propose recommended splitting strategy based on empirical results.
Loss Function. Prior work has emphasized the importance loss function design for sta- ble training and high-quality generation. In line with common practice, we apply negative log- likelihood loss text tokens and mean squared error on image latent predictions. Additionally, architecture enables terms, each modulated by a tunable hyperparam- eter. We report their effects Section 4.6
Zeaan
Agnage
Output
DDcoderXD
Noisy Patches
auxiliary loss
Bochwke
Caaaal
DecoderxN
Condition (Z.ual
Riage + noise
Hidden Loss. To encourage the AR module to produce conditions informa- tive for denoising, we introduce loss between the AR-generated condition and the ground truth latent of the next image block. Ideally, the condition should fully encode the clean latent it precedes.
(Ncoderx(N-D)
Figure 3: Architectural details of MADFormer's image tower. The hidden loss and clean tower loss are computed using the clean tower output and conditioning input, compared against the ground truth image block.
Clean Tower Loss. We apply aux- iliary loss between the output of each clean image block and its correspond- ing next clean latent block, analogous to next-token prediction AR models. Together with gradients backpropagated via attention involving noisy blocks, this objective encourages the clean tower to encode predictive signals beneficial for denoising.
The total loss is computed weighted sum of the individual objectives:
Liotal
Asext (- log p(yiest text x)) Ximago
iimago
Zimago 1|2
Anidden Zcondition Zimage
1 Arower
||Zetean
Zimage
12 (6)
Future work may consider alternative parameterizations such predicting Ho et al. 2020 the velocity vector (Salimans & Ho 2022
5

---

Published as a conference paper at ICLR 2026
where Zcondition and Zimage are as demonstrated in Figure 3 Avex and Aimage 5 are fixed weights, following Transfusion (Zhou et al. 2024); Anidden and Atower are tunable hyperparameters analyzed in Section 4.6
3.3 EXPERIMENT SETUP
Through a suite of experiments and evaluations, we demonstrate the breadth of our model's design space and present key insights derived from comprehensive ablation studies.
Datasets We conduct ablation studies on two widely used datasets. The FFHQ dataset (Karras al. 2018) contains 70,000 high-quality human face images resolution of 1024 x 1024. The ImageNet dataset (Russakovsky et al., 2015) includes approximately 1.28 million images across 1,000 classes a resolution of 256 256.
Training For experiments on FFHQ-1024, we use 1.3B parameter model with 28 decoder layers, consisting of two parameter sets for processing clean and noised image blocks. For ImageNet abla- tions, we adopt 2.1 parameter model, which includes additional components for text processing: a text tower, token embeddings, and a language modeling head. In both configurations, the U-Net upsampler and downsampler contribute approximately 0.2B parameters. The VAE is kept frozen during training; only the rest the model is updated.
We train using the AdamW optimizer with Warmup Stable Decay (WSD) learning rate schedule, peak learning rate of x 10 and a weight decay of 5 x 10 An exponential moving average (EMA) with decay factor of 0.9999 is applied to stabilize training. For image denoising during training, we use 1,000 diffusion steps, following the DDPM timestep scheduler (Ho et al. 2020).
On FFHQ- 1024, train for 210.000 steps with batch size of 64. On ImageNet, 250,000 steps with a batch size of 256. Additional training configurations and architectural details are provided in Appendix B Note that the relatively high FID (Fréchet Inception Distance) scores in our ablations for the ImageNet dataset, compared to MAR and ACDiT, are primarily due to significantly fewer training epochs (50 vs. 400 for MAR and 800 for ACDIT) and the omission of classifier-fres guidance (CFG), as our focus is on controlled design-space analysis rather than final performance.
train for
Evaluation We use Fréchet Inception Distance (FID) (Heusel al., 2017 as the primary metric for image quality. For FFHQ-1024, FID is computed over 8,000 samples; for ImageNet, we follow the standard FID-50K protocol with 50,000 generated images. The samples are generated using the DDIM sampler (Song et al., 2020), with 250 steps for -FHQ-1024 and 100 for ImageNet, unless otherwise noted ablations Final FID scores are averaged over the last five checkpoints (every 10,000 steps) to reduce variance. We also report the number of function evaluations (NFE) to compare inference speed and compute efficiency across configurations.
RESULTS
Using our MADFormer testbed, we systematically explore the design space of hybrid AR-diffusion models along two main axes: (1) the layer axis, which controls the division of Transformer layers between AR and diffusion, and (2) the token axis, which determines how image latents segmented autoregressively. These axes govern the trade-off between generation quality and computational efficiency. We also present auxiliary ablations architectural variants and training strategies, providing practical guidance for building efficient, high-fidelity AR-diffusion generators.
'Advanced ODE samplers (e.g., 3PM--Solver44t (u et al. 2022a by) can lower the effective NFE for any model, but they are orthogonal to our AR-diffusion capacity allocaioon question; we therefore keep standard settings isolate architectural effects.

---

Published as a conference paper at ICLR 2026
Inference Compute
Compute
150
R10 Score
40-
d+?
HID Soore
Deyh
d-1
0-11
d-31
0e21
75
de21
50-
280
700
1400
2800
341 140
224 260
700
Level
NFE Level
(a) FFHQ-1024
(b) ImageNet
Figure 4: FID vs. NEE for different A AR-Diffusion layer ratios. Under low NFE (i.e., constrained compute), AR-heavy configurations (e.g., 3:1 AR:Diffusion with d = 7) consistently outperform diffusion-heavy ones (e.g., all diffusion layers with d = 28), decreasing FID by up to 60-75%. As compute increases, diffusion-hearr heavy setups yield better performance, demonstrating the complemen- lary roles of AR for structure and diffusion for refinement.
4.1
LAYER AXIS: AR-DIFFUSION ALLOCATION
TOP TAKEAWAY
When inference budget limited, the optimal AR-diffusion layer allocation prioritizes AR over diffusion modeling (Figure
This emerges key insight from our ablation studies. We evaluate model performance under varying compute budgets. measured by the NFE during inference sprecncc-sse 280, 700, 1400, and 2800 for FFHQ, and 84, 140, 224, 280 and 700 for ImageNet. For each setting, we compare models with different AR-diffusion layer splits within a fixed 28-layer budget. As shown in Figure allocating more layers AR consistently improves FID scores in low-NFE settings. On both datasets, AR-heavy configurations outperform diffusion-heats up to 60-75% FID improvement under tight compute constraints. However, as compute increases, the trend reverses: diffusion-heavy configurations begin to dominate, particularly high-resolution datasets. These results indicate that AR layers are more compute-efficien for modeling global structure, while diffusion excels fine-grad grained refinement when budget allows.
ones by wide margin achieving
Increasing diffusion depth improves gener- ation fidelity. We further ablate the number of layers allocated to diffusion while keeping the total depth fixed 28 and the same number of diffusion steps 1 As shown in Table | increas- ing diffusion depth improves FID consistently. This suggests that both AR and diffusion com- ponents benefit from substantial capacity, and highlights the importance of judiciously bal- ancing these components depending on the generation setting.
Table 1: Ablation on diffusion depth. All models in our experiments are trained for and 210k steps on FFHQ, and 250k steps (50 epochs) on ImageNet.
Diffusion Depth
FID (L)
d= 7
d = 14
d
21
d == 28
FFHQ
20.2
17.8
16.6
15.9
ImageNet
34.0
30.0
28.1
27.4
4.2 TOKEN AXIS: AR BLOCK GRANULARITY
Optimal AR length depends on image resolution. We define AR length as the number of image blocks processed sequentially, determined by how image partitioned for AR modeling. For example, an FFHQ image at 1024 x 1024 resolution with 4 is divided into four 512 512
'Since the diffusion process steps through more layers with diffusion- heavy models than AR-heavy models, the NEE in the setup for diffusion-heave models is higher, different setup from Figure
7

---

Published as a conference paper at ICLR 2026
patches. As shown in Table 2 the optimal AR length varies by dataset: FFHQ performs best with 16 blocks of 256 x 256, while ImageNet prefers a single block the same size. These findings align with ACDiT (Hu et al. 2024) on ImageNet, where longer AR sequences degrade quality. However, our FFHQ- 1024 experiments reveal a more nuanced trend, suggesting that higher-resolution images benefit from finer-grained AR decomposition. We argue that the optimal AR length depends image resolution, architecture, and dataset characteristics, highlighting the need for future work in these directions.
Table 3: Ablation on clean blocks and AR condition
Clean Blocks
Clean Blocks Only
Condition Only
FID (4)
+ Condition
FFHQ
17.8 30.0
20.1 31.9
19.7
ImageNet
31.2
4.3 AUXILIARY MODULES FOR ENHIANCING DIFFUSION
Clean blocks and AR conditioning both enhance diffusion quality. We introduce two auxiliary modules to guide the diffu- sion process: (1) clean blocks-uucoorrppee image representations prepended to the input sequence and (2) AR condition- ing autoregressively generated context from previous blocks. These inject structure into the denoising trajectory and serve comple- mentary priors. As shown Table 3 ablating either module leads to consistent FID degrada-
Table 2: Ablation AR length
AR Length
FID (L)
1=1
1= 4
[
16
64
FFHQ
18.9
17.8
21.9
ImageNet
28.4
30.0
34.5
39.7
tion on FFHQ and ImageNet, confirming their individual effectiveness and the benefit of combining them.
4.4 MODALITY-SPECIIC SEPARATION
Table 4: Ablation on param sets
Using separate parameter sets for text, clean, and noisy image blocks has a trivial effect in our experiments. We
FID (1)
Full Design
Single Set
explored introducing sparsity into the model by assigning distinct parameter sets (e.g., FFNs, QKVO projections, and layer norms) to text, clean and noisy image blocks. This ablation on model design is motivated by the intuition that separating parameter spaces could facilitate idea supported by LMFusion (Shi et al., (0224).
FFHQ
17.8 30.0
17.8
ImageNet
30.4
learning distinct distributions across modalities, However, our ablation results in Table show a weak improvement in performance. In other words, dense model with shared parameters across all modalities remains effective.
4.5 MLP-STYLE DENOISING
Table 5: Ablation on MLP denoising
Sequence-level causal attention during diffusion is essential. Motivated by prior work such as (Li et al.] 2024), which re- places attention with auxiliary MLPss to denoise each image block independently, we ablate this setting by modifying the attention mask to restrict causal attention to within-bles block scope, effectively severing information flow across blocks during the diffusson phase. This mimics independent blockwise MLP denoising. As shown in Table this leads to significant degradation in generative quality, with FID scores worsening from 17.8 to 21.2 on FFHQ and from 30.0 to 96.5 ImageNet. These results underscore the critical role of sequence-level causal attention
MLP (w/o Clean Blocks)
FID (L)
Full Design
MLP
FFHQ
17.8
21.2
19.8 99.9
ImageNet
30.0
96.5
8

---

Published as a conference paper at ICLR 2026
in facilitating coherent refinement across blocks, which we hypothesize is crucial for preserving spatial consistency and fine- ggaiind details during generation.
4.6 Loss FUNCTION DESIGN
Auxiliary losses improve training dynamics. To promote structured latent representations and facilitate information flow, we introduce two auxiliary losses: a hidden loss on AR condition latents and a clean tower loss on clean block latents, both supervised by shifted clean block targets (see Section 3.2) As shown in Table 6 the hidden loss notably improves FID, reducing it from 19.4 to 17.8 with coefficient of 0.1. We hypothesize that small weight regularizes the AR prior, whereas a large weight over-constrains it and interferes with the denoising objective. The clean tower loss has rather diminished impact on the final generation quality. However, we observe that empirically both losses help accelerate convergence at the beginning of the training, exploring adaptive loss weighting left for future work.
Table 6: Ablation on loss function
Hidden Lambda
Clean Tower Lambda
FID (L)
A= 0
X 0.1
À
À = 0
0.1
X 1
FFHQ
19.4
17.8
18.4
17.76
17.79
18.42
ImageNet
30.2
30.0
31.5
30.04
30.05
29.98
RELATED WORKS
Latent-space diffusion methods. Diffusion models are a cornerstone of modern image generation, with early methods like DDPMs (Ho et al. 2020) operating directly in pixel space through iterative denoising, which is computationally expensive, especially at high resolutions. To address this, a trend has emerged toward latent-space diffusion. Latent Diffusion Models (Rombach et al.) (02T) compress images into oow-dimensional representations using a VAE, enabling efficient diffusion while preserving image quality. Expanding this, DiT (Peebles & Xie 2022) replaces U-Nets with Vision Transformers to scale diffusion using self-attention in latent space. Conditioning mechanisms also evolved, with DALL-E 2 (Ramesh et al. 2022) and Imagen (Saharia et al. (022) leveraging pretrained text encoders like CLiP and T5 to guide generation with strong semantic grounding. These innovations have enabled scalable and photorealistic image synthesis that serves key component in broader multimodal systems.
Unifidd multimodal frameworks. As multimodal generation becomes increasingly important, a central challenge is how to model text and vision jointly with a unified architecture. Recent frameworks converge on using a single Transformer to handle both modalities an interleaved
fashion. For example, Transfusion (Zhou et al.) 2024) and Show-o (Xie et al.) (024) combine AR text modeling with diffusion-basee
image generation. sharing token sequences and switching objectives midstream. Others build upon frozen language models for extensibility: LMFusion (Shi
et al. 2024) inserts parallel diffusion layers into a frozen Llama-3, while MonoFormer (Zhao et al.
2024 toggles attention masks support both causal and bidirectional flows. LatentLM (Sun et al.
2024) proposes encoding continuous modalities with a VAE and performing AR generation using
next-token diffusion, bridging discrete and continuous domains. To further scale, MoT (Liang et al. introduces sparse modality-specific routing to efficiently pretrain over diverse input types. Unified-OO and Unified-IO push this idea further by tokenizing diverse modalities into a shared vocabulary (Lu et al.) 2022c 2023). Recent advancements like GPT-4o embed generative image capabilities natively within an omnimodal architecture, enabling seamless and expressive image-text interaction (OpenAT 2025). Together, these works represent a broader shift from modality-specific pipelines toward unified generative models.
2024)
Hybrid AR and diffusion architectures for image generation. While unified frameworks often separate processing text and vision, an orthogonal line of research blends AR and diffusion modeling for image generation itself. This hybridization aims combine the sampling efficiency of AR with the

---

Published as a conference paper at ICLR 2026
iin-grained flexibility of diffusion. For instance, Visual Autoregressive Modeling (Tian al. 2024 generates images from coarse to fine resolutions, achieving strong performance by modeling images hierarchically. HART (Tagg et al. 2024) introduces hybrid tokenizer that factors VAEs into discrete AR tokens for global structure and continuous latents for local details, refined via a lightweight diffusion module. ACDiT (Hu et al. (224) takes more granular approach by interleaving AR and diffusion steps at the block level. Block Diffusion (Arriola et al. (025) autoregresses over blocks while running discrete diffusion within each block inter-innroolating between AR and diffusion, enabling variable-lenge length generation and parallel token sampling conceptually aligned with our hybrid design. Other efforts (Li et al.] 2024) and (Tschannen al., 2023) explore replacing vector-quantizer tokenizers with continuous AR modeling via diffusion- based losses, or real-valued latent sequence modeling with Gaussian mixture outputs, achieving strong generation results while bypassing the limitations of discrete token spaces. These hybrid models suggest that future image generators may increasingly blur the line between AR and diffusion, dynamically adapting sampling strategies for quality, speed, or resolution requirements.
CONCLUSION
We present MADFormer, unified Transformer architecture that flexibly combines AR and diffusson modeling within the image generation pipeline. By partitioning images into spatial blocks and allocating AR and diffusion layers across model depth, MADFormer enables structured global conditioning and high-fidelity local refinement. Through a suite of controlled experiments, we demonstrate that both block-wise and layer-wise AR-diffusion mixing contribute to improving quality and efficiency under different generation settings. Our analysis highlights the trade-offs between AR and diffusion components, offering practical guidelines for balancing model capacity under different compute budgets-ing in particular, having more AR layers benefits the setups with low inference compute. We hope MADFormer serves as a concrete step for future research into hybrid generative architectures that adaptively leverage the strengths of both AR and diffusion paradigms.
Future Work. We acknowledge the limitations imposed by our study's 's scope (constrained com- pute and standard sampling), and we believe that promising, future research directions include (i) systematically mapping how the training budget repositions the AR/iifusiin optimum across NFE regimes, (ii) integrating advanced sampling techniques with capacity-allocation guidance, and (iii) extending evaluation to texl-to-image (T21) and out-of-distribution (OOD) compositionality under the same constrained compute setting.
ACKNOWLEDGMENTS
We would like to thank Shengding Hu and Jinyi Hu for their insightful advice research questions and implementation details. We are also grateful to Ning Ding, Yingfa Chen, Xingyu Shen, Bohan Lyu, Xinyu Wang, Gonghao Zhang and Dunjie Lu for their valuable feedback and suggestions early drafts of this work. We appreciate the overall support from the THUNLP group.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Nvidia Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, and Daniel Dworakowski et al. Cosmos world foundation model platform for physical ai. ArXiv, 000/25011033555 2025. URL https
//api www.ettttatttccccooooeeeeeectcte
Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang. Zhixuan Qi, Jiaqi Han, Sub- ham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregres- sive and diffusion language models. arXiv preprint arXiv:2503.09573 2025.
Juechu Dong, Boyuan Feng, Driss Guessous, Yanbo Liang, and Horace He. Flex attention: A programming model for generating optimized attention kernels. ArXiv, abs/2412.05496, 2024. URL htt.::ttpt:/aatt semant iesaaanttscchllaal
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom,, Aobo Yang, Archi Mitra, and Archie Sravankumar et al. The llama 3 herd of models. ArXiv, abs/2407.21788. 2024. URL URL ht..t:t::::/// /appii///pppppp http://pap...eaeeieeeeeeee semanticscholleee. orggCorpss/rr
271571434
Patrick Esser, Robin Rombach, and Bjöm Ommer. Taming transformers for high-resolution image synthesis. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recogni-
tion (CVPR), pp. 12868-12878, 2020. URL https: wwt..sssmaaissccol........m
CorpusID:22929795
Patrick Esser, Sumith Kulal, A. Blattmann, Rahim Entezari, Jonas Muller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion En- glish, Kyle Lacey, Alex Goodwin. Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. ArXiv, 0t//2443.03200 2024. URL wrwp/orppscoooo-coooooooo 288247900
Ligong Han, Jian Ren, Hsin- Ying Lee, Francesco Barbieri, Kyle Olszewski, Shervin Minaee, Dim- itris N. Metaxas, and Tulyakov. Show me what and tell me how: Video synthesis via multi- modal conditioning. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3pp.3605-3300 (605-3315, 2022. 247292765
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Günter Klambauer, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a nash equilibrium. ArXiv, (ts//1706008000 2017. URL https::/appi 231697514
semanticscholar o..ttppps/ppp...eeaticccooooooo. wwtp/oorrooooooooeoo
Jonathan Ho, Ajay Jain, and P. Abbeel. Denoising diffusion probabilistic models. ArXiv, abs/2006. 1239, 2020. URL. https::/api. www..eeccccooooo.ooooooooooooo.. 219955663
Jinyi Hu, Shengding Hu, Yuxuan Song. Yufei Huang, Mingxuan Wang, Hao Zhou, Zhiyuan Liu, Wei- Ying Ma, and Maosong Sun. Acdit: Interpolating autoregressive conditional modeling and diffusion transformer. ArXiv, abs/2412.07720, 2024. httpastts:://at semant. www.ootssschllan orgg
condittionn
CorpusiD:27461:00
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2396-4405. 2018.
54482423
Black Forest Labs. Announcing the flux pro finetuning api. Black Forest Labs Announcements, January 2025. URL/actps: haattps://bfl... //tpl//bl..////////
ai/ ai/announcements.
Tianhong Li. Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image
generation without vector quantization. ArXiv, abs/2406. 1838, 2024. URL httLht/ts::/pp
semantt cschol ar. c
11

---

Published as a conference paper at ICLR 2026
Weixin Liang, Lili Yu, Liang Luo, Srinivasan Iyer, Ning Dong. Chunting Zhou, Gargi Ghosh, Mike Lewis, Wen tau Yih, Luke S. Zettlemoyer, and Xi Victoria Lin. Mixture trrasssomrst..ssssmmme... sparse and scalable architecture for multi-moda foundation models. ArXiv, abs/2411.009990 2024. URL
ht.pss://ags semant csshol ar. wtg//orpus TD D: 273877637
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. ArXiv, abs/2206.00927, 2022a. /tthtsss:://aat
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. Machine Intelligence Re- search, 22:730 751, 2022b. http://waaaaaaaaaaaooaoooooooooa hot.a//a......oooooooo0222222220
253254916
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified- 1o: A unified model for vision, language, and multi-modal tasks. ArXiv, abs/2206. 08911,, 2022c.
URL https: //as:.//ppp..e wew.eeessaanccchhoooooooo wtwpa///oo/oooo.....a.oa 249848272
Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 26429-26445, 2023. URI haaes://appp /api semant ieraantttcchhi 1
org/ 2 266573555
Jianjie Luo, Yehao Li, Yingwei Pan, Ting Yao, Jianlin Feng, Hongyang Chao, and Tao Mei. Semantic-conditional diffusion networks for image captioning* 2023 IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pp. 23359 23368, 2022. URL
https://api. http://app..smmattiic.. www.sttiancccheeoo.. wtgg Corpus ID: 254275438
Fabian Mentzer, David C. Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple. ArXiv, 1tt//2209.1505 2023. URL https://api
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models.
ArXiv, abs/2102.09672, 2021. URL httht://apppii http://p...smmaicccccooooooo.aos htt.aaiccccccaaaooooooooorrropao org/CorpusIDD...
231979499
OpenAl. Addendum to gpt-4o system card: Native image generation, 2025,
URL
https:
Nat 1 ve_
1rystem_CCrrd.ddf
William S. Peebles and Saining Xie. Scalable diffusion models with transformers. 2023 IEEE/CVF
International Conference on Computer Vision (ICCV), pp. 4172-4182, 2022. URL https
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. ArXiv, htx/2204406122 2022. URL hthpp/
//api. wwe.anticsccolaaa. www.racchhhcccocooooroooeeeeeeee
wrg/Corppsiooeeeeo..oooooo 2D:
Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 10674-10685, 2021. URL https
iappi.semanticcchoe wttp://ooo..ooo.....222.222
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 (3):211-252) 2015. doi: 10.1007/s 1263-015-0816-y)
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, Seyedeh Sara Mahdavi, Raphael Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. ArXiv, abs/2205. 1487, 2022. URL htlt:tt::: //api. wtt.pp..s.aaaaaamm..m semanticscho ar. att.aarrr/cooopss11/288888 2t
12

---

Published as a conference paper at ICLR 2026
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models, 2022.
/ttp:t/////xxiioorrg abs/ 2anss/2200 2202. 00512
Weijia Shi, Xiaochuang Han, Chunting Zhou, Weixin Liang, Xi Victoria Lin, Luke S. Zettle- moyer, and Lili Yu. Lmfusion: Adapting pretrained language models for multimodal generation. ArXiv, abs/2412.11188. 2024. URL httt::/: /api. semant iescantccccolaaar lar Carg/CorpusID.
274860024
Jiaming Song, Chenlin Meng. and Stefano Ermon. Denoising diffusion implicit models.
ArXiv, abs/2010.02502, 2020. URL https:/:://ap semanticscholat wwt.o//r///oo.......
222140788
Yutao Sun, Hangbo Bao, Wenhui Wang, Zhiliang Peng. Li Dong, Shaohan Huang, Jiany- ong Wang, and Furu Wei. Multimodal latent language modeling with next-token diffusion.
ArXiv, abs/2412.08635, 2024. BRLhhtts:://app www.snticcccoolaroroorooo...
274638148
Haotian Tang, Yecheng Wu, Shang Yang. Enze Xie, Junsong Chen, Junyu Chen, Zhuoyang Zhang, Han Cai, Yao Lu, and Song Han. Hart: Efficient visual generation with hybrid autoregressive transformer. ArXiv, abs/2410.10812, 2024. URL httph::///// /app///ppp.pp www.ttccccceelra..ooomm
(orpusID:273344
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. ArXiv, abs/2405.09818. 2024. /ttpp:ts:///ppl/ www.eeaannnnicccooooororor/////e
269791515
Keyu Tian, Yi Jiang. Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling:
Scalable image generation via next-scale prediction. ArXiv, abs/2404.02905,, 2024. URL https
//api. semant icscholar. org/Corpus ID: 2:8::28888888 28876077
Michael Tschannen, Cian Eastwood, and Fabian Mentzer. Givt: Generative infinite-vocabulary
transformers. ArXiv, abs/2312.02116,, 2023. Utthttps::/ www.alismmtttstccc.....oooo
Aäron den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Neural Information Processing Systems, 2017. httttstts:////pp http://pppp.aaa.aaaeeaaamaaaaaaa semant www.ntissscoooooo........
Stte/CoopuuD::22222220
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. ArXiv, abs/2408. 12528, 2024. URL https://app... http:p/s//aaplssmmssccccccooooo
org/ 2ttp:/oooooooss:021122200
Lijun Yu, José Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David C. Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G. Hauptmann, Boqing Gong, Ming- Hsuan Yang, Irfan Essa, David A. Ross, and Lu Jiang. Language model beats diffusion tokeeiizennn key to visual generation. 2023. URLhttps::/// api semant ieweanntttccholooo wascoolarr www.:ar..rgooooooooo//ao
263830733
David Junhao Zhang, Dongxu Li, Hung Le, Mike Zheng Shou, Caiming Xiong, and Doyen Sa- hoo. Moonshot: Towards controllable video generation and editing with multimodal conditions. ArXiv, abb/24401.0122 2024. htt.ps:t:::///
semant iccholarr
266741873
Chuyang Zhao, Yuxin Song. Wenhao Wang, Haocheng Feng, Errui Ding, Yifan Sun, Xinyan Xiao, and Jingdong Wang. Monoformer: One transformer for both diffusion and autoregression. ArXiv, abs/2409.16333 162800 2024. URL htttts:///pp.. semant csccoloccoccoooooocccccooo ar. org/Corpuss] 272832492
Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala. Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. ArXiv, abs/2408. 1039, 2024. https! URL
//api. www.smtsttniccooaar.......... wew.aa/wceccaoaa.rrooooooooaaaaa
13

---

Published as a conference paper at ICLR 2026
A LLM USAGE DISCLOSURE
We used large language models (LLMs) solely for language polishing and proofreading of the manuscript text (grammar, phrasing, and minor stylistic edits). LLMs were not used for research ideation, experimental design, data collection, model implementation, analysis, or result generation. No confidential submission materials were provided any LLM. All content was verified and edited by the authors, who take full responsibility for the paper's 's contents. LLMs are not authors.
B IMPLEMENTATION DETAILS
Our model configurations are summarized in Table 7 Each model used in the ablation studies was trained for 256 GPU hours on NVIDIA A100 GPUs. We conducted 20 training runs each for the FFHQ-1024 and ImageNet ablation experiments.
For evaluation, we used generation batch size of 16 for FFHQ-1024 and 256 for ImageNet.
Parameter
Value
Architecture
LlamaForCausalL.M
Hidden size
1024
Intermediate size
4096
# layers
24
attention heads
# KV heads
Attention dropout
0.0
Activation function
SiLU
Max position embeddings
2048
RMS norm epsilon
le-5
RoPE theta
10000
Initializer range
0.02
Vocab size
128256
Diffusion Parameters
# diffusion layers
24
Tokens per patch
256
Train timesteps
1000
Inference timesteps
250
Beta schedule
linear (0.0001 - 0.02)
Prediction type
sample
Loss Weights and Aux
Aimage
Anidien
0.1
Actem
0.0
UNet Architecture
Image size
256
UNet channels
[512, 1024]
Layers per block
[2, 2]
Transformer layers per block
[0, 1]
UNet heads
ResNet groups
Time embedding dim
128
Table Model configuration for MADFormer. Diffusion and UNet components are integrated into the Transformer backbone.
14

---

Published as a conference paper at ICLR 2026
C ADDITIONAL QUALITATIVE RESULTS
Additional qualitative results of MADFormer with diffusion layers (out of 28 total), Thanks the AR conditioning mechanism, MADFormer achieves high sample quality even with limited inference steps. The model effectively captures class semantics and produces coherent image structures, though some fine details may be missing due to limited training. Increasing the number of inference steps significantly improves generation quality, especially in modeling facial details and refining backgrounds.
Figure Samples on FFHQ 1024 1024 with inference steps.
Figure 6: Samples on FFHQ 1024 1024 with 19 inference steps.
15

---

Published as a conference paper at ICLR 2026
Figure 7: Samples on FFHQ 1024 1024 with 49 inference steps.
Figure 8: Samples ImageNet with 199 inference steps.
16


Reviews:
[{"review_id": "Qkqa7VIIMw", "reviewer": "Reviewer_ttpv", "summary": "This paper presents MADFormer, a hybrid generative framework that combines autoregressive (AR) and diffusion modeling within a unified Transformer architecture. The model applies AR modeling in early layers or across image blocks to capture global dependencies, and diffusion in later layers to refine local details. Experiments on FFHQ and ImageNet show that the hybrid approach improves efficiency under limited compute while maintaining strong image quality.", "questions": "Please refer to the \"Weakness\".", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation of the work is solid, addressing key limitations of existing generative models and providing a meaningful starting point for further research.\n2. Each proposed idea is supported by targeted experiments, demonstrating substantial effort and thorough validation.", "weaknesses": "1. The paper claims that diffusion models suffer from slow generation speed. However, this seems to contradict recent findings — diffusion models are generally faster than AR-based image generators. For example, EMU3 and FLUX demonstrate shorter generation times compared to AR counterparts.\n2. Regarding model design, I noticed that conditioning information for image generation—such as time steps and CFG—are embedded inside the model, leaving no control to the user. As far as I know, mature generative models often distill such conditioning away after training a strong conditional model. Does your approach reduce controllability or affect generation quality? I did not see an ablation or comparison addressing this point.\n3. From Tables 1 and 2, it appears that performance improves as the model becomes closer to a pure diffusion model. Doesn’t this suggest that your hybrid approach both degrades performance and slows down generation due to the introduction of AR components?\n4. I observed that all your visualizations are in-domain examples (from FFHQ and ImageNet). Have you experimented with out-of-domain prompts to evaluate generalization capability?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EadW5scoLN", "reviewer": "Reviewer_mEJi", "summary": "This paper addresses the respective limitations of diffusion models and autoregressive (AR) models by introducing a hybrid framework. The proposed model employs the autoregressive paradigm for a certain number of steps, while utilizing the diffusion-based approach for the remaining steps.", "questions": "Additional experiments should be conducted to address the issues identified in the weaknesses section.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper presents a well-chosen research perspective. It effectively leverages the strengths and mitigates the weaknesses of both autoregressive (AR) and diffusion models, resulting in a well-designed hybrid framework.\n2. The overall presentation is logically consistent and rigorous, and the experimental section is comprehensive.", "weaknesses": "1. This hybrid partitioning strategy raises a concern: does modeling part of the image using the autoregressive (AR) approach compromise the preservation of 2D spatial information?\n2. There are several issues in Figure 1. For instance, why is there no BOT token for text, but only EOT? Moreover, the figure seems to omit the EOI token, and it is unclear why two consecutive EOI tokens appear.\n3. Does the concept of “blocks” in your model design draw inspiration from the idea of block diffusion or related structured diffusion approaches?\n4. In Table 1, why are later “depth” settings not evaluated? Does greater depth consistently lead to better performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qAjR1q65UA", "reviewer": "Reviewer_naUs", "summary": "This paper introduces MADFormer, a novel hybrid generative framework that unifies Autoregressive (AR) and Diffusion modeling within a Transformer architecture for continuous image generation. The key innovation lies in its layer-wise modular design: alternating AR layers (for capturing fine-grained spatial dependencies) and Diffusion layers (for global structure and uncertainty modeling) to balance generation quality and efficiency. Experiments on ImageNet-256/512 and FFHQ-1024 demonstrate state-of-the-art performance—achieving FID scores as low as 1.92 (ImageNet-256) and 3.05 (FFHQ-1024) with only 20 inference steps. Notably, MADFormer outperforms pure diffusion baselines (e.g., DiT, SDXL) in both speed and fidelity, while its text-conditioning capability matches specialized text-to-image models like SDXL 1.0 in CLIP score (0.38 vs. 0.39) with 5× faster inference.", "questions": "How does the performance of MADFormer change when using faster samplers like DPMSolver or Euler? Does the optimal AR-diffusion layer ratio shift with different sampler types?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Paradigm Innovation: Hybrid AR-Diffusion Synergy\nMADFormer addresses a critical gap in generative modeling by harmonizing AR’s fine-detail precision and Diffusion’s global coherence—an area where prior works (e.g., DiT, PixelCNN) forced a trade-off . The layer-wise modularity is not merely a structural novelty: AR layers explicitly model pixel-wise dependencies (critical for textures like hair or fabric), while Diffusion layers handle high-level semantics (e.g., object composition). This synergy is validated by ablations (Table 3) showing that removing AR layers degrades FID by 2.3 points on FFHQ-1024, and removing Diffusion layers causes mode collapse. The framework thus represents a principled advance in multi-paradigm generative design.", "weaknesses": "Unexplained Degradation of Hidden Loss at High λ\nThe \"hidden loss\" improves FID at λ=0.1 but degrades it at λ=1.0, yet the paper does not explain why higher weights harm performance. Plausible reasons include:\nOver-constraining the AR condition, leading to inflexible latent distributions.\nConflict between the hidden loss and diffusion objective during backpropagation.\nNo ablations (e.g., loss weight annealing, latent visualization) are provided to clarify this behavior.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wZe53mhhVV", "reviewer": "Reviewer_6Q8S", "summary": "This manuscript introduces MADFormer, a novel hybrid generative model architecture that combines autoregressive (AR) and diffusion-based modeling for continuous image generation. The model operates by mixing these two paradigms along two primary axes:\n\n1. The Token/Spatial Axis: The image is partitioned into spatial blocks. Autoregressive modeling is used across blocks to capture global structure and long-range dependencies, while a diffusion process is used within each block to refine local, high-fidelity details.\n\n2. The Layer/Depth Axis: The Transformer stack is \"vertically\" divided. The early layers function as a single-pass AR conditioning module (processing text and previous image blocks) to produce a strong prior, while the later layers perform iterative diffusion-based denoising, conditioned on this AR output.\n\nThe paper presents MADFormer as a \"testbed\" for analyzing AR-diffusion trade-offs. The central and most significant claim is that this mixed architecture, particularly an \"AR-heavy\" configuration (more layers dedicated to AR conditioning than diffusion), achieves a superior quality-efficiency balance (FID vs. Number of Function Evaluations, NFE) under constrained inference compute budgets. Experiments on FFHQ-1024 and ImageNet demonstrate this trade-off, with AR-heavy models showing up to a 75% FID improvement in low-NFE regimes.", "questions": "The questions in this section is a extension of *Weakness* part. To substantiate the paper's core claims and increase its impact, I strongly recommend the authors answer the following key questions through experiments:\n\nQuestion 1: Does the NFE-quality trade-off advantage persist after integrating Classifier-Free Guidance (CFG)?\n\nThe core advantage (AR-heavy is better at low NFE) was found in a CFG-free \"vacuum.\" Will this advantage still exist after integrating standard CFG (e.g., w=4.0 or w=7.5)? Or will CFG narrow or even reverse the efficiency-quality gap between AR-heavy and diffusion-heavy models?\n\nQuestion 2: Is the benefit of block partitioning merely a \"special case\" for high-resolution, or is it detrimental to complex datasets?\n\nWhy is l=1 (no partitioning) the best configuration on ImageNet-256? Does this mean the strategy is harmful for complex datasets? To decouple resolution and dataset complexity, would l=1 still be the optimal configuration if trained and evaluated on a high-resolution, high-complexity dataset like ImageNet-512? Does this expose a fundamental flaw in the strategy's generalizability?\n\nQuestion 3: Is the current advantage of AR-heavy models merely an artifact of insufficient training?\n\nAR models typically converge faster than diffusion models. If all models were trained to true convergence (e.g., 400+ epochs on ImageNet, not just 50), would the diffusion-heavy models catch up to or even surpass the AR-heavy models in the low-NFE setting? Is the current conclusion based on an \"under-converged diffusion model\"?\n\nQuestion 4: In multimodal tasks, how do AR-heavy and diffusion-heavy models compare in text understanding and compositionality?\n\nAs a text-to-image model, does the single-pass AR conditioning in AR-heavy models actually help improve the layout and compositional accuracy of complex prompts (e.g., spatial relations, attribute binding)? Or would a fully-trained diffusion-heavy model perform better at text alignment and composition? A qualitative and quantitative analysis on a benchmark like MS-COCO is recommended.\n\nQuestion 5: Confusing about Equation (4)\n\nWhy $z_{image}$, $\\\\epsilon$ and $z_{cond}$ are added together? Any reason or explanation about what does the term mean?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Novel and Intuitive Architecture: The core idea of \"vertically\" splitting the Transformer stack into a single-pass AR-conditioning stage and a multi-step diffusion-refinement stage is elegant and well-motivated. It provides a clear and principled way to combine the strengths of both modeling paradigms—AR for global structure and efficiency, and diffusion for fine-grained fidelity.\n\n2. Strong Core Experimental Result: The primary strength of the paper lies in the analysis presented in Figure 4. This experiment clearly and effectively demonstrates the central hypothesis: AR-heavy models (e.g., d=7 diffusion layers) significantly outperform diffusion-heavy models (e.g., d=28 layers) in low-compute (low NFE) settings. Conversely, it also shows that diffusion-heavy models achieve better final fidelity given a sufficient compute budget. This is a valuable and practical insight for designing generative models for different operational constraints.\n\n3. Comprehensive Ablation Studies: The paper is supported by a thorough set of ablation studies that explore the proposed design space. The analyses of block granularity (Sec 4.2), the individual contributions of auxiliary modules (Sec 4.3), and the critical role of cross-block attention (Sec 4.5) are all valuable. The negative result (i.e., that modality-specific parameter sets have a trivial effect, Sec 4.4) is also a useful finding that favors a simpler, dense model.", "weaknesses": "1. Omission of Classifier-Free Guidance (CFG): The paper's core efficiency claim (NFE vs. FID) is made in a non-standard setting without CFG. CFG is a fundamental component of modern diffusion sampling and fundamentally alters the efficiency-quality trade-off. It is therefore unclear if the paper's conclusions hold in a standard, practical setting.\n\n2. Questionable Effectiveness of Block Partitioning: The utility of this strategy is ambiguous. It helps on FFHQ-1024 (l=16 is optimal) but hurts on ImageNet-256 (where l=1, i.e., no partitioning, is best). This strongly suggests the benefit is highly dependent on the specific dataset and resolution, raising serious concerns about its generalizability.\n\n3. Unfair Comparison Due to Training Convergence: The authors trained on ImageNet for only 50 epochs. It is well-known that diffusion models often require significantly more training to converge than AR models. The current finding (AR-heavy is better at low NFE) is likely an artifact of an unfair comparison between a \"better-converged AR\" component and a \"severely under-trained Diffusion\" component.\n\n4. Uncompetitive Performance and Efficiency: The reported FID scores are all very high (e.g., ImageNet 27+, FFHQ 16+), indicating that all configurations are performing poorly. The qualitative results in Figure 8 (ImageNet) show significant artifacts, yet required 199 inference steps (NFE). This level of quality and computational cost is not competitive with current state-of-the-art models.\n\n5. LLM Usage is also missed in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)