HART: Efficient Visual Generation with Hybrid Autoregressive Transformer
Authors: Haotian Tang
Yecheng Wu
Shang Yang
Enze Xie
Junsong Chen
Junyu Chen
Zhuoyang Zhang
Han Cai
Yao Lu
Song Han
Venue: ICLR 2025 Poster (2025)
Keywords: autoregressive models
image generation
text-to-image
Primary area: foundation or frontier models, including LLMs
TL;DR: An autoregressive model that can directly generate 1024x1024 images. It takes advantage of hybrid tokenization and residual diffusion to model both continuous and discrete tokens.

Abstract: We introduce Hybrid Autoregressive Transformer (HART), the first autoregressive (AR) visual generation model capable of directly generating 1024x1024 images, rivaling diffusion models in image generation quality. Existing AR models face limitations due to the poor image reconstruction quality of their discrete tokenizers and the prohibitive training costs associated with generating 1024px images. To address these challenges, we present the hybrid tokenizer, which decomposes the continuous latents from the autoencoder into two components:  discrete tokens representing the big picture and continuous tokens representing the residual components that cannot be represented by the discrete tokens. The discrete component is modeled by a scalable-resolution discrete AR model, while the continuous component is learned with a lightweight residual diffusion module with only 37M parameters. Compared with the discrete-only VAR tokenizer, our hybrid approach improves reconstruction FID from 2.11 to 0.30 on MJHQ-30K, leading to a 31% generation FID improvement from 7.85 to 5.38. HART also outperforms state-of-the-art diffusion models in both FID and CLIP score, with 4.5-7.7$\times$ higher throughput and 6.9-13.4$\times$ lower MACs. Our code is open sourced at https://github.com/mit-han-lab/hart.

Published as a conference paper at ICLR 2025
HART: EFFICIENT VISUAL GENERATION
WITH
HYBRID AUTOREGRESSIVE TRANSFORMER
Haotian Tang Yecheng Wu Shang Yang!
Enze Xie Yao Lu
Junsong Chen? Song Han
Junyu Chen 13
Zhuoyang Zhang'
Han Cai
MIT NVIDIA² Tsinghua University
attps://hande /hanlaaalabbba/eaa mit. fects jects/hart
SD XLL FLUX--chheil
PorArt Sigmatiiaaaa HART
SD3 Mediuum
28.6
Throushpt (hmppta
2.4
1.8
7.7x
1.2
higher
0.6
0.5
Figure 1: HART early autoregressive model that can directly generate 1024 1024 images with quality comparable to diffusion models, while offering significantly improved efficiency. achieves 4.5-7.7x higher throughput, 3.1-5.9x lower latency (measured on A100), and 6.9-13.4x lower MACs compared to state-of-the-art diffusion models. Check out our online demo and video.
ABSTRACT
We introduce Hybrid Autoregressive Transformer (HART), autoregressive (AR) visual generation model capable directly generating 1024x 1024 images, rivaling diffusion models in image generation quality. Existing AR models face limitations due to the poor image reconstruction quality of their discrete tokeniz- associated with generating 1024px images.
ers and the prohibitive training To address these challenges, we present the hybrid tokenizer, which decomposes the continuous latents from the autoencoder into two components: discrete to- kens representing the big picture and continuous tokens representing the residual components that cannot be represented by the discrete tokens. The discrete com- ponent is modeled by scalable- resolutton discrete AR model, while the continu- ous component learned with a lightweight residual diffussion module with only 37M parameters. Compared with the discrete-only VAR tokenizer, hybrid ap- proach improves reconstruction FID from 2.11 to 0.30 on MJHQ-30K, leading 31 % generation FID improvement from 7.85 5.38. HART also outper- forms state-of-the-art diffusion models in both FID and CLIP score, with 4.5- 7.7x higher throughput and 6.9-13.4x lower MACs. Our code is open sourced :ettts::/gitie cwm www/motttt..coooooo..oooococ..mt han lab a hhrrt.
indicates equal contribution. Part of the work was done when Haotian Tang and Shang Yang were summer interns at NVIDIA. Correspondence (kentang, songhan @mi eddu.

---

Published as a conference paper at ICLR 2025
HART
Playground v2.5
PixArt-E
SD-XL
Prompt: A small cactus with a happy face in the Sahara desert.
HART
Playground v2.5
PixArt-E
SD-XL
FAT
Prompt: A panda that has been cybemetically enhanced. Playground v2.5
HART
PixArt-E
SD-XL
Prompt: An astronaut riding a horse on the moon, oil painting by Van Gogh. Playground v2.5
HART
PixArt-E
SD-XL
Prompt: A lighthouse in giant wave, origami style. Playground v2.5
HART
PixArt-E
SD-XL
Prompt: A beautiful cabin in A ttersee, A ustria, 3d animation style.
Figure 2: HART generates 1024px images with quality comparable to state-of-the-ar diffusion models such Playground v2.5 (Li et al., 2024a), PixArt-Y (Chen al., 2024a), and SDXL (Podell et al., 2023) while being 4.6-5.6x faster.
2

---

Published as a conference paper at ICLR 2025
INTRODUCTION
The rapid advancement of large language models (LLMs) is pushing artificial intelligence into new era. At the core of LLMs are autoregressive (AR) models, which have gained popularity due to their generality and versatility. These models typically predict the next token in a sequence based on the previous tokens as input. While originating from natural language processing, autoregres- sive models have also recently been adopted for visual generation tasks. These approaches utilize a visual tokenizer to convert images from pixel space into discrete visual tokens through vector quantization (VQ) (Van Den Oord et al., 2017). These visual tokens are then processed in the same manner as language tokens. Benefiting from techniques proven successful in the LLM field, autore- gressive visual generation methods have demonstrated their effectiveness in diverse tasks, including mext-to-image generation, rext-to-video generation, and image editing (Van Den Oord et al., 2017; Esser et al., 2021; Chang et al., 2022; Yu et al., 2022b; Kondratyuk et al., 2023; Tian et al., 2024). Autoregressive image generation models have also demonstrated significant potential for building unified visual language models (Gemini Team, Google, 2023; OpenAl, 2024), such as Emu3 (Emu3 Team, BAAI, 2024), VILA-U (Wu et al., 2024), and Show-o (Xie et al., 2024).
Concurrently, another major trend in visual generation from Ho et al. (2020); Rombach et al. (2022); Chen al. (2024a); BlackForest Labs (2024) has centered diffusion models. These models em- ploy a progressive denoising process, beginning with random Gaussian noise. Diffusion models achieve better generation quality compared with autoregressive models, but they can be computa- tionally expensive to deploy: even with an efficient DPM-Solve sampler from Lu et al. (2022), it still takes D/T-XL//2 (Peebles Xie, 2023) 20 denoising steps to generate an image, which translates to 86.2T MACs at 1024x 1024 resolution. In contrast, generating comparable image using a sim- ilarly sized AR model capable of predicting multiple tokens in parallel (Tian et al., 2024) requires only 10.1T MACs at the same resolution, which is 8,5x less computationally intensive.
This paper addresses the following question: Can we develop an autoregressive model that matches the visual generation quality of diffusion models while still being significantly faster?
Currently, visual generation AR models lag behind diffusion models two key aspects:
1. Discrete tokenizers in AR models exhibit significantly poorer reconstruction capabilities com- pared to the continuous tokenizers used by diffusion models. Consequently, AR models have a lower generation upper bound and struggle to accurately model fine image details.
Diffusion models excel high-resolution image synthesis, but no existing AR model can directly generate 1024x 1024 images.
To address these challenges, we introduce HART (Hybrid Autoregressive Transformer) for effi- cient high-resolution visual synthesis. HART bridges the reconstruction performance gap between discrete tokenizers in AR models and continuous tokenizers in diffusion models through hybrid tok- enization. The hybrid tokenizer decomposes the continuous latent output of the autoencoder into two components: one as the sum of discrete latents derived from a VAR tokenizer (Tian et al., 2024), and the other as the continuous residual, representing the information that cannot be captured by discrete tokens. The discrete tokens captures the big picture, while continuous residual tokens focus on fine details (Figure 3). These two latents are then modeled by our hybrid transformer: the discrete latents VAR transformer, while the continuous latents are predicted by
are handled by a a lightweight residual diffusion module with 5% parameter and 10% runtime overhead.
HART achieves significant improvements in both image tokenization and generation over its discrete-only baseline. Compared with VAR, it reduces the reconstruction FID from 2.11 0.30 at 1024x 1024 resolution on MJHQ-30K (Li et al., 2024a), enabling HART to lower the 1024px gen- eration FIID on the same dataset from 7.85 to 5.38 (a 31% relative improvement). Furthermore, we demonstrate that HART achieves up to 7.8% improvement in FID over VAR for class-conditioned generation on ImageNet (Deng et al., 2009). HART also outperforms MAR on this task with 13x higher throughput.
Notably, HART closely matches the quality of of diffusion models in multiple text-to- image generation metrics. Simultaneously. HART achieves 3.1-5.9x faster inference latency, 4.5. x higher throughput, and requires 6.9-13.4x less computation compared with diffusion models.
3

---

Published as a conference paper at ICLR 2025
HA discrete toleens
HA residual tokens
HART
HT discrete takess
HART meiduni tokens Prompt: salfie of woman and her fion cub on the plains
HART
Prompt: close of helmet on a person, digital art, victorian armor
Figure 3: HART synergizes discrete and continuous tokens. The discrete tokens capture the overall image structure, while the fine details (e.g. eyes, eyebrows and hair) are reflected in the residual tokens, which is modeled by residual diffusion (introduced in Section 3.2).
RELATED WORK
Visual generation has become key focus in machine learning research. Initial work by Kingma & Welling (2013) introduced variational autoencoders (VAEs) for image synthesis. Subsequently, Goodfellow et al. (2014) proposed generative adversarial networks (GANs), which were further improved by Brock al. (2018); Karras et (2019); Kang et al. (2023).
Diffusion models from Ho et al. (2020); Nichol & Dhariwal (2021); Dhariwal & Nichol (2021); Ramesh et al. (2022); Betker et al. (2023) have emerged as the state-of-the approach for gen- erating high-quality images after VAE and GAN. The latent diffusion model from Rombach et al. (2022); Podell et al. (2023) applies U-Net to denoise the Gaussian latent input, and is succeeded by DiT from Peebles & Xie (2023) and U-VIT from Bao et al. (2023) which replaces the U-Net with transformers. Chen et al. (2023; 2024b;a) scale up DiTs to text-to-image (T2I) generation. Con- currently, Kolors Team (2024); Ma et al. (2024a); Li et al. (2024a) further scaled up T21 diffusion models to billions of parameters. Recent research from Esser et al. (2024); Auraflow Team (2024); BlackForest Labs (2024) also explored rectified flow for fast sampling.
Autoregressive models pioneered by Chen et al. (2020) generate images as pixel sequences, rather than denoising an entire latent feature map simultaneously. Early research VQVAE and VQGAN from Van Den Oord et al. (2017); Esser et al. (2021) quantize image patches into discrete tokens and employ decoder-only transformer to predict these image tokens, analogous to language modeling. VQGAN was subsequently enhanced in several aspects: Yu et al. (2022a) improved its autoencoder modeling, Chang et al. (2022); Yu al. (2023a); Li al. (2023) increased its sampling speed with MaskGIT, while Mentzer et al. (2023), Yu et al. (2023b), and Yu et al. (2024) enhanced its tok- enization performance and efficiency. Lee et al. (2022) introduced residual quantization reduce tokenization error. Building on this, Tian et al. (2024) developed VAR, which innovatively trans- formed next-token prediction in RQVAE to next-scale prediction, significantly improving sampling speed. There were also efforts that scaled up autoregressive models to text-conditioned visual gen- eration: Ramesh et al. (202 1); Ding et al. (2021; 2022); Liu et al. (2024); Sun et al. (2024); Crowson et al. (2022); Gafni et al. (2022); Emu3 Team, BAAI (2024) were T2I generation methods based on VQGAN, and Chang et al. (2023); Villegas et al. (2022); Kondratyuk et al. (2023); Xie et al. (2024) extended MaskGIT. STAR, VAR-CLIP and ControIVAR from Ma et al. (2024b); Zhang al. (2024); Li et al. (2024c) were extensions of VAR.
class of visual generative models that synergize discrete and con- tinuous image modeling approaches. GIVT from Tschannen et al. (2023) predicted continuous vi- sual tokens with autoregressive models while VQ-Diffusion from Gu et al. (2022) extended diffusion to discrete latents. MAR from Li et al. (2024b) and DisCo-Diff from Xu et al. (2024) conditioned a diffusion model with autoregressive prior. This idea was also concurrently explored in visual lan- guage models by Ge et al. (2024); Jin et al. (2023). Transfusion (Zhou et al., 2024) fuses DIT and LLMM into a single model, and is natively capable of multi-modal generation.
Hybrid models represent
HART: HYBRID AUTOREGRESSIVE TRANSFORMER
We introduce Hybrid Autoregressive Transformer (HART) for image generation. Unlike all existing generative models that operate exclusively either discrete continuous latent spaces, HART
4

---

Published as a conference paper at ICLR 2025
models both discrete and continuous tokens with unified transformer. The key factors enabling this are a hybrid tokenizer and residual diffusion.
3.1
HYBRID VISUAL TOKENIZATION
Refermce (256x256)
VAR (discrete)
HART (hybric)
Reference (1lxx1k)
VAR (discrete))
HART (hybrid)
Figure 4: Reconstruction quality comparison between VAR and HART tokenizers. The discrete
tokenizer employed by VAR will lose some details have some distortion during the reconstruction, which is solved by hybrid tokenization in HART. Please zoom in for details in 1k images.
Residual tokens
(to be modeled by resichal cliffusson
Resice and Sum up
Continuuus tokens
Discrete
Visual Encoder
VQ Quantizer
Visual Decoder
-50%
50% (@ tokenizer training)
100% (@ generation decoding)
Tokenization
Generation
Discrete
Continuous
Figure 5: Unlike conventional image tokenizers that decode either continuous discrete latents, the hybrid tokenizer HART is trained to decode both continuous and discrete tokens. At inference time, we only decode continuous tokens, which the of discrete tokens and residual tokens. The residual tokens will be modeled by residual diffusion (introduced in Figure 6).
Conventional autoregressive visual generation encodes images into discrete tokens using trained tokenizers. These tokens map to entries in vector-quantized (VQ) codebook and can reconstruct the original images from the VQ tokens. This approach transforms text-to-image generation into requence-to-sequee
problem, where decoder-only transformer, LLM, predicts image tokens from text input. The tokenizer's reconstruction quality sets the upper limit for image generation. Constrained by their finite vocabulary codebooks, discrete tokenizers often struggle to faithfully reconstruct images with intricate, high-frequency details such as human faces, as Figure 4.
Hybrid tokenization. We introduce our hybrid tokenizer in Figure 5. The primary goal of hybrid tokenization is to enable the decoding of continuous features during generation, thereby overcoming the poor generation upper bound imposed by finite VQ codebooks. We begin with a CNN-based vi- sual encoder that transforms the input image into continuous visual tokens in the latent space. These tokens are then quantized into discrete tokens across multiple scales, following VAR (Tian et al., 2024). The multi-scale vector quantization process results difference between the accumulated discrete features and the original continuous visual features, which can not be accurately represented using VQ codebook elements. We term this difference residual tokens, which are subsequently mod- eled by residual diffusion,, as detailed in Section 3.2.
Alternating training. To train our hybrid tokenizer, we begin by initializing the visual encoder, quantizer (i.e., codebook), and visual decoder from a pretrained discrete VAR tokenizer. We then freeze the visual encoder and quantizer, allowing only the visual decoder to be trained. During each training step, we randomly choose with equal probability (50%) whether to provide the decoder with discrete continuous visual tokens for reconstructing the input image. Specifically, when the continuous path is selected (lower red path in Figure 5), it bypasses the VQ quantizer, effectively turning the model into conventional continuous autoencoder. Otherwise. if the discrete path is selected (upper red path in Figure 5), we are essentially training a standard VQ tokenizer. Empirical results show that the HART tokenizer achieves comparable continuous rFID (i.e., reconstruction FII)

---

Published as a conference paper at ICLR 2025
when the continuous path is activated) to the SDXL tokenizer (Podell et al., 2023), while its discrete rFID matches the performance of the original VQ tokenizer. As a result, the generation upper bound of HART remains consistent with state-of-the ahe- diffusion models. This alternating training strategy also ensures that the continuous and discrete latents remain sufficiently similar from the decoder's perspective, facilitating easier modeling of continuous latents.
Discussions. It's important to note that low rFID does not necessarily indicate better generation FID. The alternating approach in training our hybrid tokenizer is crucial for high-quality generation. In Section 4.3, we demonstrate that other methods, such using separate decoders for continuous and discrete tokens, may achieve similar continuous reconstruction FII) but significantly compro- mise generation FID. Furthermore, the next subsection explains why autoregressive methods utiliz- ing continuous tokenizers, like MAR (Li et al., 2024b), are less efficient than HART.
3.2
HYBRID AUTOREGRESSIVE MODELING WITH RESIDUAL DIFFUSION
Désrrdee takens
Prompt: heautiful eybory with golden hair
2
hidden
Resichal Diffusion,
Candiramasstkemss
Scalable-Resolee Resolution
stales
MLP
uutorerressive Transfommer
1
Resichal tolens
L.MM
Stop
Step to N-1 sum
HART allention mask
text takens
Sum of Disordettaens
Figure 6: HART is an efficient hybrid autoregressive image generation framework. decom- poses continuous image tokens into two components: 1) a series of discrete tokens modeled by a rreouuttnn (up to 1024px) autoregressive transformer, and 2) residual tokens modeled by a lightweight residual diffusion (37M parameters and 8 steps) module. The final image representa- tion is the sum of these two components.
Hybrid tokenization offers superior rFID and better generation upper bound compared to discrete tokenization. We introduce HART (Figure 6) to efficiently translate this improved upper bound into real enhancements in generation quality. HART models the continuous image tokens the sum of two components: (1) discrete tokens, modeled by a scalable-resolution autoregressive transformer, and (2) residual tokens, fitted by an efficient residual diffusion process.
Scalable-resolution autoregressive transformer. Our discrete token modeling extends VAR to text-to-ine text-to-image generation and improves scalability at higher resolutions. HART concatenates text tokens with visual tokens during training. in contrast VAR which use single class token. The text tokens are visible to all visual tokens, as in Figure 6 (right). Our approach is 25% more parameter- efficient than STAR (Ma et al., 2024b)'s 's cross-attention method (Chen et al., 2023).
Unlike Parti, the only prior bR-based method achieving 1024px generation through super-resolution with Imagen (Saharia et al., 2022), HART directly generates 1024px images with a single model. To mitigate the O training cost for high-resolution AR transformers, we finetune from pre- trained low-resolution checkpoints. We convert all absolute position embeddings (PEs) in VAR to relative embeddings. including step (indicating the resolution each token belongs to) and token index embeddings. We utilize sinusoidal PE for step embeddings, which naturally accommodates varying sampling steps in 256/51 1px (10 steps) and 1024px (14 steps) gen- cration. For token index embeddings, we implement a hybrid approach: ID rotary embeddings for text tokens and 2D rotary embeddings (Sun et al., 2024; Ma et al., 2024a; Wang et al., 2024) for visual tokens. The position indices of visual tokens directly continue from those of text tokens. We found these relative embeddings significantly accelerates HART convergence at higher resolutions.
interooltiionccmmpttiii
Residual diffusion. We employ diffusion to model residual tokens, given their continuous nature. Similar to MAR (Li et al., 2024b), we believe that a full DiT is unnecessary for learning this residual. Instead, lightweight (37M parameters) residual diffusion MLP would be sufficient. This MLP is conditioned on the last layer hidden states from our scalable-resolution AR transformer, as well as the discrete tokens predieted in the last VAR sampling step.
Despite similar denoising MLP model architectures, HART differs fundamentally from MAR. While MAR predicts full continuous tokens, HART models residual tokens crucial distinction for effi-

---

Published as a conference paper at ICLR 2025
cient diffusion modeling. Although both trained with 1000-step noise schedule, HART achieves optimal quality with just sampling steps at inference, compared to MAR's s 30-50, resulting in a 4-6x reduction in diffusion module overhead. This demonstrates that HART's residual tokens significantly easier to learn than MAR's full tokens.
Other differences. The AR transformers in HART and MAR differ significantly in their formu- lation. MAR's AR transformer generates only conditions for its diffusion MLP, lacking a discrete codebook for token generation. In contrast, HART's AR transformer produces both discrete tokens and diffusion conditions. These discrete tokens can be decoded into meaningful, albeit less detailed, images using our hybrid tokenizer design (Figure 3). This approach reduces the burden on residual diffusion, which only needs to model fine details rather than the overall image structure. Further- more, HART supports KV caching for faster inference, significantly reducing computational costs. MAR's transformer, based on MaskGIT (Chang et al., 2022), lacks this capability.
In contrast to other representative AR+diffusion methods such as VIT (Jin et al., 2023) and SEED- X (Ge et al., 2024), which employ complete diffusion models (IB parameters, 20 steps) for full continuous tokens, HART provides significant efficiency gains through the use of a tiny diffusion MLP (37M parameters, steps) that models only residual tokens.
3.3 EFFICIENCY ENIIANCEMENTS
AR transformer and residual diffusion designs are crucial for high- quality, high-resolution image generation, they inevitably introduce inference and training overhead. We address these efficiency challenges in this section.
While our
Training. Naively adding the residual diffusion module incurs both computational and memory overhead during training. To address this, we found that discarding 80% of the tokens (on average) in the final step and applying supervision only to the remaining tokens during training does not de- grade performance. This approach accelerates training by 1.4x at 5 512px and 1.9x at 1024px, while also reducing training memory usage by 1.1 x. In the appendix, we explain the effectiveness this method by demonstrating that the attention pattern in our autoregressive transformer is mostly lo- cal. Consequently, although token subsampling during training may compromise global interactions between tokens, it has small impact attention calculation.
For inference, we observed that relative position embeddings introduced multiple memory-bound GPU kernel calls, in contrast to the single call required for absolute position em- beddings in VAR (Tian et al., 2024). To optimize performance, we fused these computations into two kernels: one for sinusoidal calculation and another for rotary embedding. This optimization resulted in a 7% improvement in end-to-end execution time. Additionally, fusing all operations in RMSNorm into a single GPU kernel also improved total runtime by 10%.
Inference.
EXPERIMENTS
In this section, evaluate HART's performance in tokenization and generation. For generation, we present both text-to-image and class-conditooned image generation results.
4.1 SETUP
Models. For class-conditioned image generation models, we follow VAR (Tian et al., 2024) to construct HART models with varying parameter sizes in the AR transformer: 600M, IB, and 2B. The diffusion MLP contains an additional 37M parameters. We replace VAR's attention and FFN blocks with Llama-style (Touvron et al., 2023) building blocks. For eoxtctondtioned image generation, we start with the 1B model and remove all AdaLN (Peebles & Xie, 2023) layers. resulting in 30% reduction in parameters. We employ Qwen2-1.50 (Yang et al., 2024) as our text encoder and follow LI-DiT (Ma et al., 2024a) to reformat user prompts.
Evaluation and Datasets. We evaluate HART ImageNet (Deng et al., 2009) for class- conditioned image generation, and on MJHQ 30K (Li et al., 2024a), GenEval (Ghosh et al., 2024), and
(Hu et al., 2024) for text-to-image generation. The HART tokenizer is trained on OpenImages (Kuznetsova et al., 2020). For HART transformer training, we utilize ImageNet, Jour- neyDB (Pan et al., 2023), and internal MidJourne--style synthetic data, All text-to-image generation
7

---

Published as a conference paper at ICLR 2025
MJHQO30K
GenEval Overall
DPG-Bench
Type
Model
#Params
Resolution
FIDI
CLor--core"
AverageT
Diff.
SI v2.1
860M
768x 768
26.96
25.90
0.50
68.09
Diff.
SD-XL
2.6B
x024x1022
8.76
28.60
0.55
74.65
Difl.
PixArt-a
630M
512x512
6.14
27.55
0,48
71.11
Diff.
PixArt-E
630M
1024x
6.34
27.62
0.52
79.46
Diff.
Playground v2.5
2B
1024x
6.84
29.39
0,56
76.75
Diff.
SD3-medium
2B
1024x
11.92
27.83
0.62
85.80
AR
LlamaGen
775M
512x512
25.59
23,03
0.32
65.16
AR
Show-o
1.3B
256x256
14.99
27.02
0,53
67.48
512x512
5.22
29.01
0.56
80.72
AR
HART
732M
1024x 1024
5.38
29.09
0.56
80.89
The performance of HART MJHQ-30K, GenEval and DPG-Bench benchmarks. We compare HART with open-source diffusion models and autoregressive models. Results demonstrate that HART can achieve comparable performance to state-of-the-art diffusion models with < pa- rameters, surpassing prior autoregressive models by large margin.
Table 2:
Model
#Params
#Steps
Latency
Thruughut (imagels)
512x512
MACs
Latency
1024 1024 Throughpu (imagek)
MACs
SDXL
2.68
40
20
2.5
2.1
1.4
61.4
30.7
4.3
0.49
0.25
239
120
PixArt-)
630M
86.2
Playgroumd v2.5
2B
20
2.3
0.49
239
120
SDS-medium
2B
28
14
1.1
51.4
4.4
0.29
168
LlamaGion
775M
1024
0.4
HART
732M
10
0.3
10.6
14
0.75
2.23
12,5
Table 3: Compared to state-of-the-ar diffusion models, HART achieves 5.0-9.6x higher throughput and 4.0-4.7x lower latency at 512x512 resolution. 1024x 1024 resolution, demonstrates 4.5- 7.7x x higher throughput and 3.1-5.9x lower latency.
data are recaptioned using VIL.A1.--1BB (Lin et al., 2024). We measure all quality and efficiency metries using open-source models with recommended sampling parameters as released by their au- thors. Latency and throughput (batch=8) measurements are conducted on NVIDIA A100.
4.2 MAIN RESULTS
Hybrid tokenization. We evaluate the HART hybrid tokenizer on ImageNet and MJHQ-30K, two datasets not observed during training. As shown in Table 1, our hybrid tokenization offers significant advantages over discrete tokenization, reducing the 1024px rFID from 2.11 to 0.30. This matches the performance level of the SDXL tok- enizer, indicating that the generation upper bound of HART is comparable to that of diffusion mod-
Method
MJHQ-30K rFID..
ImageNet rFID.,
256px
512px
1024px
256px
512px
VAR
1.42
1.19
2.11
0.92
0.58
SDXL.
1.08
0.54
0.27
0.69
0.28
Ours (dis.)
1.70
Ours
0.78
0.67
0.30
0.41
0.33
Table 1: HART significantly outperforms VAR and matches SDXL tokenizer performance on MJHQ- 30K and ImageNet datascts.
els. The discrete rFID of our hybrid tokenizer also ensures that the discrete tokens still capture the majority of image structure, so that the residual tokens remain easily learnable.
Text-to-image generation. We present quantitative text-to-image generation results in Table 2. On MJHQ-30K, our method achieved superior FID compared to all diffusion models. HART also demonstrates better image-text alignment than the 3.6x larger SD-XL (Podell et al., 2023), as indi- cated by the CLIP score on the same dataset. On GenEval and DPG-Bench, HART achieves results comparable to diffusion models with <2B parameters. Importantly, HART achieves this genera- tion quality at significantly lower computational cost. As shown in Table 3, HART achieves 9.3x higher throughput compared to SD3-medium (Esser et al., 2024) at 512x512 resolution. For x024x 1024 generation, HART achieves at least 3.1x lower latency than fate-of-the-as diffusion
8

---

Published as a conference paper at ICLR 2025
Type
Model
FTD4
1st
#Params
#Step (AR.)
#Step (diff.)
MACs
Time (8)
Diff.
DiT-XL/2
2.27
278.2
675M
250
57.2T
113
AR
VAR-d20
2.57
302.6
600M
10
412G
1.3
AR
VAR-d24
2.09
312.9
1.0B
10
709G
1.7
AR
VAR-d30
1.92
323.1
2.0B
1.4T
2.6
AR
MAR-B
2.31
281.7
208M
100
7.0T
26.1
AR
MAR-L
1.78
296.0
479M
64
100
16.0T
34.9
AR
HART-A20
2.39
316.4
649M
10
8
579G
1.5
AR
HART-d24
2.00
331.5
1.0B
10
8
858G
1.9
AR
HART-d30
1.77
330.3
2.0B
10
8
1.5T
2.7
Table 4: HART achieves better class-conditioned image generation results compared to MAR (Li al., 2024b) with 10.7x lower MACs and 12.9x faster runtime. It also offers 7.8% FID reduction with 4 % runtime overhead compared with VAR (Tian et al., 2024). Time: bs=64 on A100.
models. Compared to the similarly sized PixArt-E (Chen et al., 2024a), our method achieves 3.6x faster latency and 5.6x higher throughput, which closely aligns with the theoretical 5,8% reduction MACs. Compared to SDXL, HART not only achieves superior quality across all benchmarks in Table but also demonstrates 3.1x lower latency and 4.5x higher throughput.
Class-conditioned generation. Table 4 presents our class-to-image generation results. HART outperforms MAR-L (Li et al., 2024b) terms of FID and inception score, while requiring 10.7 x fewer MACs and achieving 12.9x lower latency. Across all model sizes, HART demonstrates 4.3-7.8% % improvement in FID and consistent enhancement (d 224), the residual diffusion overhead accounts for only 4-11 % of the total runtime. HART also compares favorably to DiT-XL/2 (Peebles & Xie, 2023), with our largest model being 3.3x faster, when DiT employs 20-step sampler.
inception score. For larger models
4.3
ABLATION STUDIES AND ANALYSIS
Resolution Res. tokens
FID,
CLIPt
Time (s)
Depth Res. tokens
FIDJ
1S1
Time (s)
256px
X
6.11
27.96
2.23
20
x
2.67
297.3
1.3
256px
/
5.52
28.03
2.42
20
/
2.39
316.4
1.5
512px
X
6.29
28.91
5.62
24
x
2.23
312.7
1.7
512px
/
5.22
29.01
6.04
24
/
2.00
331.5
1.9
1024px
x
5.73
29.08
25,9
30
x
2.00
311.8
2.5
1024px*
X
7.85
28.85
25.9
30
/
1.77
330.3
2.7
1024px
5.38
29.09
28.7
Table 5: HART learns residual lokens, which enhance conditioned image generation as evidenced by improvements in FID, inception score, and CLIP score. The HART-VAS VAR results are obtained by omitting residual diffusion from the full HART model. Left: class-to-image, right: text-to-image. results obtained using the official VAR quantizer.
MAR
HART
72
Separale Decodas
Image 15
300
54
Single Decodir Alt Trining
Taiddgg 2713 100 trys, 2004
ImageNet
200
36
Seperte Decoder, 24000 FID=21
18
0
25
50
75
100
4000
9000
14000
19000
24000
Ditfusion Sampling Steps
Training Steps
Single Det: (AIL Training), 24000 strps, FTD=5.30
Figure 7: Left: residual tokens in HART much easier to learn than full tokens in MAR. Mid- dle/Right: Despite achieving similar reconstruction FID, single decoder with alternating training enables faster and better generation convergence.

---

Published as a conference paper at ICLR 2025
pooteell phecoo egg.. photoomph, deptt bad
Self-gonted od jeeiting oybom with
SActial pholo aigil photoggah dooth hoid
Selfgrettal oal paining
Dog
DoxT
Official VAR, 256 512pk, 4000 steps
HART (scalable 256px-112x 512pK, 4000 steps
Figure Scalable-reoltution transformer accelerates convergence when finetuning HART at
higher resolution thanks relative position embeddings that supports resolution interpolation.
We evaluate the key design choices in HART by examining: the effectiveness and efficiency of residual diffusion, the impact of our alternating training strategy for the hybrid tokenizer, and the importance of the scalable-resolution AR transformer.
Residual diffusion: effectiveness. Table 5 demonstrates the effectiveness of learning residual to- kens in HART. For ImageNet 256 generation, residual diffusion yields 10-14% improvement in FID and up to 6.4% increase in inception score compared to the baseline model, HART- VAR. We constructed HART-VAL using the publicly available VAR codebase, which resulted slightly lower discrete-only performance than reported in Tian et al. (2024). For coxt-conitioneed generation on MJHQ-30K, FID improved by 11.1% at 256px and 17.0% at 512px. At 1024px. where the orig- inal VAR tokenizer shows poor reconstruction performance (Table 1). HART achieves 31 % FID) improvement. Even compared to the stronger discrete-only HART tokenizer, residual diffusion still offers 6.1% FID improvement. Figure visualizes the residual tokens, illustrating how residual diffusion enhances discrete tokens with high-resolution details.
Residual diffusion: efficiency. Figure (left) demonstrates that HART's approach of learning residual tokens is significantly more efficient than MAR's method of learning full tokens. Notably, HART achieves higher inception score with just diffusion sampling steps compared MAR's 60 denoising steps, resulting in a 20x reduction in runtime for continuous token learning.
Alternating training in hybrid tokenizer. We explored various strategies train the hybrid tok- enizer while maintaining similar continuous rFID. Figure (middle and right) compares our current approach (single decoder with alternating training) using separate decoders for continuous and discrete latents, with the discrete decoder frozen. Our design offers faster, better convergence for class-conditioned image generation. Alternative strategies, such as finetuning the entire hybrid tok- enizer from a pretrained continuous tokenizer or decoding only continuous latents during training, are proven to be as bad as the separate decoder solution.
Scalaabe-rresoutionn transformer. Lastly, Figure 8 illustrates that substituting all absolute PEs in VAR with relative PEs significantly enhances convergence when fine-tuning HART higher resolu- tions from pretrained row-uuouttiion checkpoints. Given that the token count in HART increases by 4x (resulting a 16x increase attention computation) when the output image resolution doubles, this accelerated convergence is crucial for maintaining manageable training costs.
5 CONCLUSION
We introduce HART (Hybrid Autoregressive Transformer), the first autoregressive model capable of directly generating 1024x 1024 images from text prompts without super-resolution. НART achieves quality comparable to diffusion models while being 3.1-5.9x faster and offering 4.5-7.7 x higher throughput. Our key insight lies in the decomposition of continuous image latents through hybrid tokenization, producing discrete tokens that capture the overall structure and residual tokens that refine image details. We model the discrete tokens using a saalable-resolutin while lightweight residual diffusion module with just 37M parameters and 8 sampling steps leams the residual tokens. We believe HART will catalyze new research into modeling both discrete and continuous tokens for sequence-based visual generation.
AR transformer,
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENTS: DISCUSSION OF POTENTIAL MISUSE OF HART
Misuse of generative AI for creating NSFW (not safe for work) content remains a significant concern in the community. To enhance safety, we have integrated HART with ShieldGemma-2B (Zeng et al., 2024), powerful LLM-based safety check model. In our implementation, user prompts are first evaluated by the safety check model determine whether they contain NSFW content, including harmful, abusive, hateful, sexually explicit, or otherwise inappropriate material targeting individuals or protected groups. If the prompt deemed appropriate, it proceeds to HART for image generation. Otherwise, the prompt is rejected and replaced with our default prompt ("A red heart"). Through extensive testing, we have confirmed that our safety check model effectively filters out NSFW prompts under strict thresholds, ensuring our pipeline does not generate harmful AI content.
ACKNOWLEDGMENT
We thank MIT-IBM Watson AI Lab, MIT and Amazon Science Hub, MIT AI Hardware Program, National Science Foundation for supporting this research. We thank NVIDIA for donating the DGX server. We would like also thank Tianhong Li from MIT, Lijun Yu from Google DeepMind, Kai- wen Zha from MIT and Yunhao Fang from UCSD for helpful technical discussions, and Paul Palei, Mike Hobbs, Chris Hill and Michel Erb from MIT for helping us set up the online demo and maintain the servers.
REFERENCES
Auraflow Team. Introducing auraflow v0.1, an open exploration of large rectified flow models, 2024.
URL https:t /ttl:///bog fal.al. aarra
Fan Bao, Shen Nie, Kaiwen Xue. Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 22669-22677 2023. 4
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang. Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://edn. openai. com/papers/dallee-,, pdf. 2(3):8, 2023. 4
BlackForest Labs. 2024. URL http::/:://ggttt..coooooo 3,4
hut. www..///ww...ootoottooooa.aaaae. wwweeooortttee--e-- laba/ flux.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1096, 2018. 4
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11315-1 11325, 2022. 4,7
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, José Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image gen- eration via masked generative transformers. Proceedings of the 40th International Conference Machine Learning, pp. 4055-4075, 2023. 4
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.0042
2023. 4,
Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-a: Weak-to-strong training of diffusion transformer for 4k texi-to-image generation. arXiv preprint arXiv:2403.0m 04692, 2024a. 2, 3, 4,
Junsong Chen, Yue Wu, Simian Luo, Enze Xie, Sayak Paul, Ping Luo, Hang Zhao, and Zhenguo Li. Pixant-6: Fast and controllable image generation with latent consistency models. arXiv preprint arXiv:2401.088 05555,, 2024b. 4
11

---

Published as a conference paper at ICLR 2025
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In International conference on machine learning. pp. 1691 1703. PMLR. 2020. 4
Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan. Louis Cas- tricato, and Edward Raff. Vggan-clip: Open domain image generation and editing with natural language guidance. In European Conference on Computer Vision, pp. 88-105. Springer, 2022. 4
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009. 3, 7, 19
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8799 2021.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, al. Cogview: Mastering text-to-image generation via transformers. Advances in neural information processing systems, 34: 19822-19835, 2021. 4
Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang. Cogview2: Faster and better text-to-image generation via hierarchical transformers. Advances in Neural Information Processing Systems, 15:16890-16902 2022. 4
Emu3 Team, BAAL Emu3: Next token prediction is all you need, 2024. URL https://
baai-solut coluution.oo www.oon.n-k--..... kss--n--al -ne-enn-eeeeeeeeeeeee jing ksyuncs com/emu3/ / Emu-3 tech-eehhheee tech-report.
pdf. 3,4
Patrick Esser, Robin Rombach, and Bjom Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, pp. 12873-12833 2021. 3,
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müllrr, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024. 4, 8
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make- a-scene: Scene-based text-to-image generation with human priors. In European Conference Computer Vision, pp. 89-106. Springer, 2022. 4
Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song. Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi multi-granularity comprehension and generation. arXiv preprini arXiv:2404. 14396, 2024.
Gemini Team, Google. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. 2023. 3
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object- framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36, 2024. 7
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farke Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for eoxt-toimagee synthesis. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 10696-10706. 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-68511 2020. 3,
12

---

Published as a conference paper at ICLR 2025
Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with Ilm for enhanced semantic alignment. arXiv preprint arXiv:2403.05133, 2024.
Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Yadong Mu, et al. Unified language-vision pretraining in Ilm with dynamic discrete visual tokenization. arXiv preprint arXiv:2309.04660 04669, 2023. 4,7
Minguk Kang, Jun- Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for rext-to-image symthesis. Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 10124-10134, 2023.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019. 4
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint
arXiv:1312.611, 2013. 4
Kolors Team. Kolors: Effective training of diffusion model for photorealistic texa-to-image syn- thesis, 2024. URL https:/ /ttp::///httbb/ com/ Kwaii Kollors//ooors////a htallo/ommaae///////
Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. Videopoet: A large language model for zero-shot video generation. arXiv preprint urXiv:2312.14125 2023.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Sha- hab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection scale, In- ternational journal of computer vision, 128(7):1566-1911, 2020. 7
Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 11523-11532, 2022.
Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. Playground v2. 5: Three insights towards enhancing aesthetic quality in text-to-image generation. arXiv preprint arXiv:2402. 17245, 2024a. 3,
Jiacheng Li, Longhui Wei, Zong Yuan Zhan, Xin He, Siliang Tang, Qi Tian, and Yueting Zhuang. Lformer: Text-to-image generation with I-shape block parallel decoding. arXiv preprint arXiv:2303.03800, 2023. 4
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. arXiv preprint arXiv:2406. 1838, 2024b. 4, 6,
Xiang Li, Kai Qiu, Hao Chen, Jason Kuen, Zhe Lin, Rita Singh, and Bhiksha Raj. Controlvar: Exploring controllable visual autoregressive modeling. arXiv preprint arXiv:2406.099. 09750, 2024c.
Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26689- 26699, 2024.
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on Kallloon-enngte video and language with blockwise ringattention. arXiv preprint urXiv:2402.08268, 2024.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
13

---

Published as a conference paper at ICLR 2025
Bingqi Ma, Zhuofan Zong, Guanglu Song, Hongsheng Li, and Yu Liu. Exploring the role of large language models in prompt encoding for diffusion models. arXiv preprint arXiv:2406. 11831, 2024a. 4, 6,7
Xiaoxiao Ma, Mohan Zhou, Tao Liang, Yalong Bai, Tiejun Zhao, Huaian Chen, and Yi Jin. Star: Scale-wise text-to-image generation via auto-regressive representations. arXiv preprint arXiv:2406. 10797, 2024b. 4,
Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantiza- tion: Vq-vae made simple. arXiv preprint arXiv:2309. 15505, 2023. 4
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusson probabilistic models. In International conference on machine learning, pp. 8162- 8171. PMLR, 2021.
OpenAl. Hello gpt-4o, 2024. URL httph:ttp//// httpp:::///pmaa.... http://nai..cmm///////e///e/e//e -ateeell--pp------ 3pt-4o/.
Junting Pan, Keqiang Sun, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. Journeydb: A benchmark for generative image understanding, 2023. 7
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195 4195-4225, 2023. 3, 4, 7,9
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 2, 4, 6, 8
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pp. 8821-8831. Pmlr, 2021. 4
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- computer vision and pattern recognition, pp.
2022. 3,4
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- tion processing systems, 35:36479-36494 2022.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Ilama for scalable image generation. arXiv preprint arXiv:2406.06525 2024. 6
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng. and Liwei Wang. Visual autoregressiie modeling: Scalable image generation via sext- prediction. arXiv preprint arXiv:2404.02905, 2024. 3, 4, 10, 16
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprini arXiv:2302. 13971, 2023. 7
Michael Tschannen, Cian Eastwood, and Fabian Mentzer. Givt: Generative infinite-vocabulary transformens. arXiv preprint arXiv:2312.02116 2023. 4
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017. 3,4
14

---

Published as a conference paper at ICLR 2025
Ruben Villegas. Mohammad Babaeizadeh, Pieter-Jan Kindermans, Herann Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference Learning Representations, 2022. 4
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et Qwen2-vl: Enhancing lision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191 2024. 6
Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u; a unified foundation model integrating visual understanding and generation. arXiv preprint ar arXiv:2409.044: 2024. 3
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang. Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou, Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. /2528, 2024. 3, 4
Yilun Xu, Gabriele Corso, Tommi Jaakkola, Arash Vahdat, and Karsten Kreis. Disco-diff: Enhanc-
ing continuous diffusion models with discrete latents. In ICML, 2024. 4
An Yang. Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10667 2024. 7
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge. and Yonghui Wu. Vector-quantized image modeling with improved vggan. In International Conference on Learning Representations, 2022a. 4
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:
2(3):5, 2022b.
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recog- nition, pp. 10459-10469, 2023a.
Lijun Yu. José Lezama, Nitesh Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, al. Language model beats diffusion- tokenizer key to visual generation. arXiv preprint arXiv:23 2023b.
Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chich Chen. An image is worth 32 tokens for reconstruction and generation. arXiv preprint arXiv:2406.07550) 2024. 4
Wenjun Zeng, Yuchi Liu, Ryan Mullins, Ludovic Peran, Joe Fernandez, Hamza Harkous, Karthik Narasimhan, Drew Proud, Piyush Kumar, Bhaktipriya Radharapu, al. Shieldgemma: Genera- tive ai content moderation based on gemma. arXiv preprint arXiv:2407.21772, 2024. 11
Qian Zhang, Xiangzi Dai, Ninghua Yang, Xiang An, Ziyong Feng, and Xingyu Ren. Var-clip: Text- to-image generator with visual auto-regressive modeling. arXiv preprint arXiv:2408.01 181, 2024.
Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408. //039, 2024.
15

---

Published as a conference paper at ICLR 2025
A
APPENDIX
Stagg 8
Stagge 9
Stagg 10
Layer (Sage 10)
Layer (Saage 10)
Layer 11 (Stage 10)
1.9%
The VAR ittietiioni HART follows the stage-wise "sink +local pattem (b) Within the last stage, the attention pattem locaal
Figure 9: Left: The VAR attention HART exhibits a sink local pattern: for stages 8-10 visual- ized here, attention scores concentrate in the most recent two stages and the first three stages, akin to StreamingLLM Right: Within the final stage, the attention score distribution predominantly local. Note: For clearer visualization, we apply sigmoid function to the attention scores in the rightmstt three subfigures.
1003
G122K
Prompt: 3D render of coffee placed window sill during stormy day, The storm outside the window reflasted in coffee,, with ministure. lighting boits and turbulent atmosphion seen mug. The diorma cafe adomedd Indoor plants. Wonden crtsarmes above, and cold hime station stands out tiny bottles and glasses.
Prompt: Hiffel Tower was M ade up of more than 2 million translucent straws to like a cloud, with the hell tower the top the building, Miche installed ad hage foom- meking machines in the forest to blow huge amounts of unpredictable wet clouds in the building' classic architecture.
dimiy
Figure 10: Direct high-resolution (1024x 1024) image generation yields significantly more detailed results compared to low-resolution (512x512) generation.
A.1 ATTENTION PATTERN ANALYSIS
We visualize the attention patterns of a pretrained VAR (Tian et al., 2024) in Figure 9, Our empirical analysis reveals that for each VAR stage (i.e.. sampling step), the attention score predominantly concentrated on three key areas: the current stage, the preceding stage, and the initial three stages.
Within the current stage, where the attention score is highest, we further examine the spatial atten- tion map. as depicted in the rightmost three subfigures of Figure 9. Interestingly, despite the VAR attention mechanism allowing all tokens within the last stage to interact, the attention map exhibits a surprisingly localized pattern: each token primarily attends to its immediate neighbors, similar to convolution operations.
This observation has important implications. Even when significantly reduce the number of tokens in the last stage during training (by up to 80%), the fundamental attention pattern remains intact due to the limited global interaction between tokens. This explains why the partial supervision approach during training (discussed Section 3.3) does not compromise generation quality.
We have also empirically verified that explicitly restricting attention patterns to the first stages plus 2 local stages during training does not impact final results. Consequently, implementing a sparse attention kernel to further accelerate training is feasible, which we leave future direction.
16

---

Published as a conference paper at ICLR 2025
HART
Playground v2.5
PixArt-E
SD-XL
Prompt: boy and girl fall in love.
HART
Playground v25
PixArt-E
SD-XL
Prompt: Full body shot, French woman, Photography, French streets...
HART
Playground PixArt-E
SD-XL
Prompt: Drone view of waves crashing against the rugged cliffs Big Sur
HART
Playground v2.5
PixArt-E
SD-XL
Prompt: A dog that has been meditating all the time
HART
Playground v2.5
PixArt-E
SD-XL
Prompt: Editorial photoshoot of old woman, high fashion 2000s fashionn
Figure 11: Additional 1024x 1024 text-to-image generation results with HART. Full prompt for example 2: Full body shot, French woman, Photography, French Streets background, backlighting. rim light, Fujifilm. Full prompt for example 3: Drone view of waves crashing against the rugged cliffs along Big Sur's Garay Point beach. The crashing blue waters create white-tipped waves, while the golden light of the setting sun illuminates the rocky shore.
17

---

Published as a conference paper at ICLR 2025
HART
Playground v2.5
PixArt-E
SD-XL
Prompt: A alpaca made of colorful building blocks, cyberpunk.
HART
Playground v2.5
PixArt-E
SD-XL
Prompt: A man looks up at the starty sky, lonely and ethereal, Minimalism..
HART
Playground v25
PixArt-E
SD-XL
Prompt: The subject a woman. She wore blue coat with a gray dress Playground v2.5
HART
PixArt-E
SD-XL
Prompt: A melting apple
HART
Playground v2.5
PixArt-E
SD-XL
Prompt: beautiful lady, freckles, big smile, blue eyes, short ginger hair
Figure 12: Additional 1024x 1024 text-to-image generation results with HART. Full prompt for example 2: 8k uhd A man looks up at the starry sky, lonely and ethereal, Minimalism, Chaotic composition Op Art. Full prompt for example 3: A close-up photo of person. The subject is woman. She wore a blue coat with a gray dress underneath. She has blue eyes and blond hair, and wears pair of earrings. Behind beautiful lady, freckles, big smile, blue eyes, short ginger hair, dark makeup, wearing floral blue vest top. soft light, dark grey background.
blurred city buildings and streets. Full prompt for example 5:
18

---

Published as a conference paper at ICLR 2025
Figure 13: 256x256 class-conditional generation results from HART on ImageNet (Deng et al., 2009).
19

---

Published as a conference paper at ICLR 2025
A.2
MORE VISUALIZATIONS
Reference
VAR
MAR (KI-16)
HART
F P O Z
2012
F P Z
20138
F P T
2000
F
P Z
2PRR
T
T O
T O
512x512
P EDPCR?
PECFD
200
E
FD
2000
RECTD
300
P P
B EUPER
200
RDTCRP
300
7OECTEO
7ULO7ES
200
PRLCEED
199
ANTTOTRA
D00
AAPOOTDS
300
SCPFATED
2901
F P T O L P E
F
P Z D
F
P E D
F P Z
Z
T O E C
L
2000
200
L P
1024x1021
PEO EDFORR
EDTCHT
309
E
7O10750
7DSO78R
PELORDO
Additional image reconstruction comparison among VAR (discrete), MAR (KL-16, con- tinuous) and HART (hybrid) tokenizers.
Figure 14:
Figure 10 demonstrates the significant impact of direct synthesis at 1024 1024 resolution: the 1024px generated images exhibit substantially more details compared to their 512px counterparts. Figures 11 and 12 demonstrate text-conditioned generation. HART produce these images with com- parable quality to diffusion models, while offering up to 7.7x higher throughput. In Figure 13, also showcase additional visualizations of HART-generated images for class-conditioned gen- eration. Finally, Figure 14 presents additional image reconstruction results for various tokenizers. The HART tokenizer demonstrates reconstruction performance comparable MAR's continuous tokenizer, while significantly outperforming VAR's R's discrete tokenizer.
20


Reviews:
[{"review_id": "WyvVsrWHkE", "reviewer": "Reviewer_5LBN", "summary": "This paper addresses limitations in autoregressive models (AR models), particularly the challenges in image reconstruction due to discretized tokens and the high computational costs of generating high-resolution images. To overcome these issues, the authors introduce a novel hybrid tokenizer that combines both discrete and continuous tokens. Specifically, discrete tokens are generated in an AR style, while continuous tokens are generated in a diffusion style conditioned on the discrete tokens. Compared to the VAR tokenizer, this approach demonstrates enhanced reconstruction FID performance. Moreover, in generative modeling tasks, it shows superior efficiency compared to current state-of-the-art models.", "questions": "In Table 4, I am interested to know if increasing the diffusion steps for HART during inference—similar to the approach used in MAR—would lead to an improvement in generation quality. Clarifying this aspect would help in understanding the impact of diffusion steps on HART's performance.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The hybrid tokenization approach presented in this paper demonstrates impressive reconstruction quality, outperforming methods that rely solely on discrete tokens. Additionally, the necessity of residual tokens is convincingly validated through detailed ablation studies, which provide strong empirical support for their inclusion. The model architecture—an autoregressive model with discrete tokens and conditioned diffusion on continuous tokens—exemplifies an effective design choice that balances efficiency and performance. This hybrid approach not only achieves comparable speeds to VAR in class-conditional image generation tasks but also scales effectively to large datasets, as evidenced by its application to tasks like text-to-image generation.", "weaknesses": "While Figure 7 demonstrates that residual tokens in HART are indeed easier to learn than full tokens in MAR, the results from the class-conditional image generation task seem to contradict this intuition. MAR, which generates full continuous tokens, achieves competitive FID performance despite having fewer parameters than HART, which focuses on generating residual tokens—ostensibly a less complex task. \n\nThis raises questions about the specific aspects that contribute to the observed performance gap between the two models. It would be interesting to explore whether this difference could be mitigated by increasing the diffusion steps for HART, similar to the approach in MAR. Further clarification or analysis in this area would provide valuable insights into the comparative efficiency and effectiveness of the models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NS5wIb859d", "reviewer": "Reviewer_yud3", "summary": "The paper presents HART, a Hybrid Autoregressive Transformer model for efficient, high-resolution text-to-image generation. Key innovations include a hybrid tokenization approach that combines discrete and continuous tokens, enabling finer detail capture while reducing computational overhead. HART achieves better efficiency, with lower latency and higher throughput than comparable models, and directly generates 1024x1024 images without super-resolution. However, the paper could benefit from a stronger theoretical foundation, exploration of alternative text conditioning methods, more comparative analysis with similar models like VAR-CLIP and STAR, and clearer differentiation from MAR, especially regarding inference optimizations. Overall, HART is a promising work in autoregressive T2I generation.", "questions": "See the weaknesses, and my major concern is that: \n \nCan you provide a more in-depth theoretical explanation of HART's hybrid tokenization and why it outperforms traditional discrete-only AR approaches? Not only to put some tricks to improve the performance. This would help clarify whether the hybrid approach could serve as a generalized framework for other AR T2I tasks.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The model demonstrates efficiency improvements over state-of-the-art diffusion models. The residual diffusion approach minimizes memory and processing costs, achieving a reduction in latency and MACs. As a result, HART’s latency and throughput metrics are efficient, offering a speed and computational advantage.\n\n- the hybrid tokenizer combining discrete and continuous tokens is efficient in autoregressive (AR) image generation, providing higher fidelity in image reconstruction and generating fine details often missed by discrete-only models. This method addresses the usual limitations of discrete tokenizers by retaining important image details, especially at high resolutions (1024×1024）", "weaknesses": "- **Limited Theoretical Foundation**:\n    - The paper relies predominantly on experimental results, without providing a thorough theoretical basis for the proposed methodology. A more detailed theoretical analysis would strengthen the paper’s rigor and enhance the broader applicability of the approach.\n- **Ambiguity in Differentiation Between HART and other work, e.g. MAR**:\n    - The distinction between HART and MAR is not fully clear, especially concerning optimizations like sampling efficiency and inference techniques. For example, while HART achieves optimal quality with just 8 sampling steps due to its diffusion setup, it is unclear why MAR could not potentially achieve similar results with similar diffusion adaptations. Additionally, some statements (lines 329-335) imply that certain optimizations, such as KV-caching, are unique to HART, but MAR could likely implement these as well. Further clarification on these points would provide a more accurate comparison between the two models and help readers understand the unique contributions of HART\n\n- **Unexplored Alternatives for Text Conditioning**:\n    - The current approach employs text tokens as the sequence start token, but it does not explore or compare this choice with other established methods, such as cross-attention mechanisms commonly used in diffusion models for text-to-image (T2I) tasks. Including comparisons with alternative conditioning methods would clarify the advantages or limitations of the current setup.\n- **Insufficient Comparative Analysis**:\n    - The paper would benefit from a broader evaluation against similar autoregressive T2I models, such as VAR-CLIP and STAR. A horizontal comparison would add depth to the experimental results, positioning HART more clearly within the landscape of current models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "47CXlZkU4i", "reviewer": "Reviewer_nfS4", "summary": "This paper proposes a feasible scheme for AR model to produce high quality images effectively, and has a great improvement in performance and throughput.", "questions": "Please explain the above questions.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper proposes a feasible scheme to solve the key problem of AR model: effective modeling of image quality, which is ahead of other methods in FID.\n2. The paper is logical, the process of problem solving is very smooth, and the experimental setting is also reasonable.", "weaknesses": "1. There is doubt about the use of 50% for modeling discrete and continuous tokens, what happens if it is alternate, please explain the necessity of 50% here, and if there is a 30% accident, whether the other probability will break the balance, in other words, whether 50% is the best balance between the probabilities of the two modeling methods.\n2. The overall solution of this paper is based on the random selection of two mature pipelines. There is a lot of randomness in training in this way, will it lead to poor stability of the network? Besides, this kind of scheme is not the most ideal for balancing the combination of continuous tokens and the separation of tokens, and it is a cheating scheme.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6J1jvNeKrC", "reviewer": "Reviewer_y4uu", "summary": "This paper introduces HART, an innovative autoregressive transformer designed for efficient visual generation. It employs a hybrid token space to represent images, with the discrete component modeled by a VAR model and the continuous component modeled by a compact diffusion model. Extensive experiments demonstrate that HART outperforms previous baselines in both image quality and efficiency.", "questions": "see Weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. The idea of using a compact diffusion model to model the residual continuous token is both simple and effective. I believe this design will offer valuable insights to the community.\n3. The experiments are carefully designed. The results and conclusions are convincing.", "weaknesses": "The paper notes that a lower rFID does not necessarily indicate a better gFID, which is also supported by the ablation experiments. However, I feel that there is a lack of analysis regarding this phenomenon. For instance, it would be beneficial to provide insights into which design elements are crucial for obtaining tokens that achieve both good rFID and gFID simultaneously, and the other designs lead to a bad gFID.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gWPVoEsvAI", "reviewer": "Reviewer_a2Ay", "summary": "This paper presents an efficient and high-resolution autoregressive visual generation model with a hybrid tokenizer. The hybrid tokenizer includes a discrete tokenizer and a continuous tokenizer. The discrete tokenizer that is widely used in existing AR models is applied to model image structure, while the continuous tokenizer is used to represent residual image details. To model these image token priors, this paper also introduces a scalable discrete AR model and a light-weight residual diffusion model. Based on above modules, the proposed method beats VAR in rFID and outperforms sota diffusion models in both FID and CLIP score.", "questions": "- The illustration of Figure 6. First, the arrow direction of scalable-resolution autoregressive transformer is wrong. Second, the last layer hidden states are not figured out in Figure 6. Third, the HART attention mask (right) does not contain much information. The attention mechanism of each stage is not easy to obtain for the readers.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ This paper proposes a hybrid tokenizer, addressing the poor image reconstruction quality of existing discrete tokenizer. Also, the hybrid tokenizer is simple and easy to implement.\n+ To model tokens generated by the hybrid tokenizer, this paper introduce a scalable-resolution discrete AR model and a light-weight residual diffusion model. Based on these modules, the proposed method can generate 1024px image efficiently.\n+ The authors conduct exhaustive experiments to validate the proposed method. Compared with existing AR and diffusion models, the proposed method achieves superior performance on text-to-image and class-conditioned image generation tasks.", "weaknesses": "- The condition of residual diffusion. As shown in line 320~321, the condition consists of last layer hidden states from AR transformer and the discrete tokens predicted in the last sampling step. To my knowledge, the discrete tokens predicted in the last sampling step only contain residual image details. It is not enough to predict the residual tokens. Thus, the hidden states are important. Does the hidden states come from all sampling steps?\n- Efficiency enhancements (Training). In line 346~349, the fist sentence is about the overhead of residual diffusion module. But the following solution (discarding 80% tokens) is used to mitigate training overhead of VAR, which is also illustrated in Appendix A.1. In my opinion, there is a mismatch between these two parts.\n- Alternating training in hybrid tokenizer. To my knowledge, with this training scheme, visual decoder is robust to the error of predicted tokens. For generation model, the error is large in the early training. Thus, the decoder with this scheme performs better than others, as shown in Figure 7 (middle and right). Does the great advantage of alternating training exist after generation model training?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)