Hierarchical Masked Autoregressive Models with Low-Resolution Token Pivots
Authors: Guangting Zheng
Yehao Li
Yingwei Pan
Jiajun Deng
Ting Yao
Yanyong Zhang
Tao Mei
Venue: ICML 2025 poster (2025)
Keywords: Masked Autoregressive Models
Image Generation
Primary area: applications->computer_vision

Abstract: Autoregressive models have emerged as a powerful generative paradigm for visual generation. The current de-facto standard of next token prediction commonly operates over a single-scale sequence of dense image tokens, and is incapable of utilizing global context especially for early tokens prediction. In this paper, we introduce a new autoregressive design to model a hierarchy from a few low-resolution image tokens to the typical dense image tokens, and delve into a thorough hierarchical dependency across multi-scale image tokens. Technically, we present a Hierarchical Masked Autoregressive models (Hi-MAR) that pivot on low-resolution image tokens to trigger hierarchical autoregressive modeling in a multi-phase manner. Hi-MAR learns to predict a few image tokens in low resolution, functioning as intermediary pivots to reflect global structure, in the first phase. Such pivots act as the additional guidance to strengthen the next autoregressive modeling phase by shaping global structural awareness of typical dense image tokens. A new Diffusion Transformer head is further devised to amplify the global context among all tokens for mask token prediction. Extensive evaluations on both class-conditional and text-to-image generation tasks demonstrate that Hi-MAR outperforms typical AR baselines, while requiring fewer computational costs.

Hierarchical Masked Autoregressive Models with Low-Resolution Token Pivots
Guangting Zheng
Yehao Li²
Yingwei Pan
Jiajun Deng?
Ting Yao?
Yanyong Zhang' Tao Mei?
Abstract
1. Introduction
In recent years, GPT-style Autoregressive (AR) models have brought a powerful revolution in Natural Language Process- ing (NLP), and become the model of choice in designing Large Language Models (LLMs) (Achiam et al., 2023; Team et al., 2023; Bai et al., 2023; Dubey et al., 2024; Team et al., 2024). The dominant training paradigm in AR models is to predict the next word/token in sequence conditioned on previous estimated words, i.c., next token prediction. This prevailing paradigm has shown excellent scalability via scal- ing laws and generalization in zero-shot settings, paving a reliable way towards Artificial General Intelligence (AGI).
Autoregressive models have emerged as a pow- erful generative paradigm for visual generation. The current de-facto standard of next token pre- single-scale
diction commonly operates sequence of dense image tokens, and is incapable of utilizing global context especially for early tokens prediction. In this paper, we introduce a new autoregressive design to model a hierar- chy from a few low-resolution image tokens to the typical dense image tokens, and delve into a thorough hierarchical dependency across multi- scale image tokens. Technically, present a Hierarchical Masked Autoregressive models (Hi- MAR) that pivot on low-resolution image tokens to trigger hierarchical autoregressive modeling in a multi-phase manner. Hi-MAR learns to pre- dict few image tokens low resolution, func- tioning as intermediary pivots reflect global structure, in the first phase. Such pivots act the additional guidance to strengthen the next au- toregressive modeling phase by shaping global structural awareness of typical dense image to- kens. A new Diffusion Transformer head is fur- ther devised to amplify the global context among all tokens for mask token prediction. Extensive evaluations on both class-conditional and text-to- image generation tasks demonstrate that Hi-MAR outperforms typical AR baselines, while requiring fewer computational costs. Code is available at http:://gttuu..com// Doa.ee/oo/miieaeaaiiiaiiaaa Dream-a//himar.
Inspired by the scaling successes of AR models in NLP, a steady of attempts have been attained scale up AR models in CV field for visual generation (e.g., text-to-image gen- eration (Sun et al., 2024; Tian et al., 2024; Li et al., 2024; He et al., 2025; Lu et al., 2024a) and text-to-video genera- tion (Ren et al., 2024b; Wang et al., 2024b;a)). Specifically, after decomposing input image into a sequence of image patches/tokens, one direction (Esser et al., 2021; Lee et al., 2022; Sun al., 2024; Zhuo et al., 2024) directly adopts GPT-style AR models with causal attention (see Figure 1 (a)), which enforces next token prediction by only attending to preceding tokens. Another direction commonly adopts BERT-style AR models with bidirectional attention (see Fig- ure (b)), e.g., Masked Autoregressive model (MAR) (Li et al., 2024; Fan et al., 2024), that simultaneously predicts multiple masked tokens in a random order by attending to all masked and unmasked tokens. These AR models with next token prediction objective, while being dominant in NLP, have not yet been scaled effectively in visual gener- ation, especially for visual content creation with complex semantics. Accordingly, in large-scale text-to-image/vider generation, typical AR models tend to be less effective when compared to other generative models (e.g.. Diffusion mod- els (Rombach al., 2022: Dhariwal & Nichol, 2021; Esser et al., 2024: Zhang al., 2024; Chen et al., 2023) and Dif- fusion Transformer (Peebles & Xie, 2023; Bao et al., 2023; Lu et al., 2024b; Zhu et al., 2024)). This might be attributed to two inherent limitations in typical AR models: (1) Most AR models capitalize on vector quantization process to de- compose images into discrete tokens, thereby inevitably resulting in information loss. (2) Both GPT-style and BERT- style AR models solely hinge on single-scale sequence of
1 This work was performed when Guangting Zheng was visit- ing HiDream.ai ai as a research intern. 'University of Science and Technology of China, Anhui, China 'HiiDremm.ai Inc, Beijing, China "The University of Adelaide, Adelaide, Australia. Corre- spondence to: Yingwei Pan <panyw.uste @ com,, Yanyong Zhang <yanyongz@ust 2....oongeust...cu........... edu.en>..
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Hierarchical Masked Autoregressive Models with Low-Resolution Token Pivots
Add Masks
Add Masks
5
7
8
5
8
6
5
7
11
12
10
13
14
1.3
14
16
14
15
Add Masks
14
15
C
2
13
14
15
C
14
15
3
14
15
Causal Transformer
Bidirectional Transformer
Bidirectional Transformer
Bidirectional Transformer
16
(a) Autoregressive Model
(b) Masked Autoregressive Model
[c] Hierarchical Masked Autoregressive Model
Figure 1. Next-token autoregressive (AR). GPT-style autoregressive models process 2D image tokens ID sequence, predicting tokens in raster order using causal attention to ensure each token depends only on preceding ones. b) Next-token masked autoregressive model (MAR). BERT-style autoregressive models initially consider all tokens be masked, subsequently predicting each masked token based on the known tokens in a random order, leveraging bidirectional attention to enable parallel prediction of a subset of tokens. c) Hierarchical mask autoregressive model (Hi-MAR). Hicrarchical mask autoregressive models adapt hierarchical prediction strategy to address the lack of global context in the next-token prediction. Hi-MAR first predicts a low-resolution image token sequence, which contains few tokens, to reflect the global structure, and then pivots on these tokens to enhance and refine the next-resolution prediction.
image tokens for predicting next tokens in autoregressive form. Such single-shot autoregressive design makes it dif- ficult capture global context for facilitating early token prediction, resulting in sub-optimal autoregressive modeling among all dense image tokens.
Moreover, in the second phase, we design new Diffusion Transformer head mine global context among all masked and unmasked tokens, aiming further strengthen autore- gressive modeling over dense image tokens.
The main contribution of this work is the mining of global context masked autoregressive models for image gener- ation. The solution also leads to the elegant view of how to build and interpret the global structure of an image, and how to nicely integrate such global structural awareness into typical masked autoregressive models, which are problems not yet fully understood in the literature. Through extensive experiments on ImageNet and MS-COCO, we demonstrate the effectiveness of our proposal, for example, achieving 0.38 FID performance boost over MAR and only requiring 54% computational costs.
In an effort to mitigate the above limitations, we start from the recent pioneering work of MAR that nicely sidesteps vector quantization and triggers autoregressive modeling in a continuous-valle space. This lossless image tokenization seems more suitable for autoregressive modeling in vision data. It motivates us to delve into the potential of global con- text mining in MAR for boosting visual generation, thereby further alleviating the second limitation. In order to upgrade MAR with additional global context, we derive a particu- lar form for autoregressive modeling named Hierarchical Masked Autoregressive models (Hi-MAR). Our launching point is to build top-down hierarchical structure from the root of low-resolution image tokens to the leaf layer of typ- ical dense image tokens. Figure 1 (c) conceptualizes such construction of hierarchical structure in Hi-MAR, which en- ables hierarchical autoregressive modeling in a multi-phase fashion. Specifically, the first phase performs bidirectional autoregressive modeling over low-resolution image tokens. Such learnt low-resolution image tokens naturally reflect global structural information of the whole image. After that, we take the low-resolution image tokens as intermediary piv- ots to guide the next autoregressive modeling over typical dense image tokens. In this way, Hi-MAR could elegantly trigger global context propagation in a top-down manner, and the second- -hasse autoregressive modeling over typical dense image tokens does benefit from this additional guid- ance of global context. Meanwhile, this additional global context propagation significantly eases the autoregressive modeling over dense image tokens, and thus requires fewer generation steps, thereby harboring an innate agency that remains advantageous in the speed of sequence prediction.
2. Related Works
Diffusion models. In the domain of image generation, dif- fusion models (Ho et al., 2020; Rombach et al., 2022; Pee- bles & Xie, 2023; Esser et al., 2024; Qian et al., 2024; Wan et al., 2024) have emerged as a powerful paradigm, achiev- ing impressive results across diverse tasks. These models conceptualize image generation
denoising process (Ho et al., 2020), where images progressively reconstructed from noise through a multi-step denoising strategy. A com- mon backbone for diffusion models is the CNN-based U-Net (Song & Ermon, 2019; 2020), which effectively captures both local and global features during denoising. To improve scalability and flexibility, DiT (Peebles & Xie, 2023) re- places the U-Net backbone with transformer (Waswani et al., 2017). Building on this, U-ViT (Bao et al., 2023) incorporates long skip connections inspired by U-Net into the transformer architecture, and treats all inputs as tokens to enhance performance.
2

---

Hierarchical Masked Autoregressive Models with Low-Resolution Token Pivots
Next-token autoregressive models. Autoregressive mod- els (AR) have achieved significant success in natural lan- guage processing (Achiam et al., 2023; Team et al., 2024;
model cross-scale dependencies. VAR (Tian et al., 2024) and HART (Tang al., 2025) employ shared Transformer to autoregressively model tokens across scales without ex- plicit scale-specific guidance, which may underutilize the unique characteristics of each resolution level. Muse (Chang et al., 2023) adopts two separately trained models for low- resolution and high-resolution generation, increasing param- eter count. FlowAR (Ren et al., 2024a) introduces spatially adaptive layer normalization in the flow matching head, facilitating position-by cot-tsttopooooonnnoooee semantic injection and en- abling scale-wise adjustments; however, it lacks an explicit mechanism for encoding scale-level information within the Transformer backbone. In contrast, our method mitigates this by introducing an elegant scale-aware Transformer back- bone that incorporates explicit resolution information while maintaining parameter efficiency. To explicitly encode scale information, we introduce a learnable scale vector for each resolution, which is injected into the Transformer backbone via adaLN-Zero operations (Peebles & Xie, 2023), further enhancing awareessss. This explicit scale encoding allows the model to better capture essolution-speciffi char- acteristics and improves multi-scale generation quality.
Bai et al., 2023; Dubey et al., 2024; Brown et al., 2020: Radford et al., 2019) and are increasingly applied to im- age generation (Zhuo et al., 2024; Team, 2024; Sun et al., 2024; Tian et al., 2024; Chang et al., 2022). Early works, such as VQVAE (Esser et al., 2021) and RQ-Transformer (Lee et al., 2022), serialize 2D images into ID token se- quences and predict tokens in raster order. LlamaGen (Sun et al., 2024) adapts this next-token prediction paradigm and scales up models to achieve quality comparable to diffusion models. Due to the fact that the dependencies between 2D image tokens do not naturally follow a raster order, some studies (Chang et al., 2022: Yu et al., 2023) suggest that causal attention may not be ideal for image generation. To address this, models like MaskGIT (Chang et al., 2022), MagViT ViT (Yu et al., 2023), Mag V (Yu et al., 2024), and MUSE (Chang et al., 2023) adopt bidirectional attention and masked prediction strategies used in BERT (Kenton & Toutanova, 2019). enabling non -on-sequential token prediction.
Compared to mainstream continuous-valued diffusion mod- els, traditional autoregressive models, inherited from lan- guage autoregressive models, predict over discrete tokens, which may introduce information loss, ultimately limiting the generation quality. Recently, GIVT (Tschannen et al., 2025) and MAR (Li et al., 2024) replaced discrete tokens with continuous tokens. MAR introduces a diffusion loss function to model per-token probabilities, replacing the cat- egorical cross-entropy loss used in discrete-valuee models. Furthermore, Fluid (Fan et al., 2024) extends continuous- valued autoregressive models to text-to-image generation.
In addition to the information loss caused by token dis- cretization and the challenges in modeling cross-scale de- pendencies, a critical limitation shared by existing multi- scale models (Chang et al., 2023; Tian et al., 2024; Tang et al., 2025; Ren et al., 2024a) is the reliance on ground- truth low-resolution tokens during training to supervise high-resolution generation. This introduces discrepancy between training and inference since ground-truth tokens are not available at inference time. Our approach mitigates this issue by conditioning high-resolution generation on predicted low-resolution tokens from the Transformer back- bone, ensuring consistency between training and inference and improving generation robustness. Furthermore, meth- ods like HART adopt an MLP-based diffusion head that treats each token independently, leading to loss of global information during the denoising process. In contrast, method introduces a Diffusion Transformer head that lever- ages self-attention to capture inter-token dependencies dur- ing diffusion, resulting in more coherent generation.
Next-scale autoregressive models. Multi-scale autore- gressive generation has been explored in various forms to enhance image synthesis quality (Chang et al., 2023; Tian et al., 2024; Tang et al., 2025; Ren et al., 2024a). Among them, Muse (Chang et al., 2023) generates high-resolution discrete tokens conditioned on low-resolution tokens and text inputs, while VAR (Tian et al., 2024) replaces next- token prediction with a next-scale prediction strategy by retraining multi-scale quantization autoencoder to encode an image into K discrete token maps at different resolutions. However, such approaches often rely on VQ-based quan- tization, which can introduce substantial information loss and limit the fidelity of generated outputs (Li et al., 2024). Moreover, retraining multi-scale autoencoders, as in VAR, increases model complexity and training costs. In contrast, our approach generates continuous- from multiple resolutions of images, eliminating the need for additional autoencoder training and improving the upper bound of generation quality.
3. Method
In this work, we devise Hierarchical Masked Autoregres- sive models (Hi-MAR) that pivot low-resolution image tokens to trigger hierarchical autoregressive modeling in a multi-phase manner. This section starts with the preliminar- ies of MAR (Li et al., 2024), which triggers autoregressive modeling in continuous- chical masked autoregressive model is elaborated. After that, we detail the integration of newly proposed Diffusion Transformer head, which mines the global context among
token sequences
space. Then, the hicrar-
Another limitation of existing approaches lies in how they
3

---

Hierarchical Masked Autoregressive Models with Low-Resolution Token Pivots
First Phase
Second Phase
Miriused DiL
10
Masked Visual Tokens
14
Conventional Hierarchical MAR
First Phase
Second Phase
3
10
HESVAR
Macked
Tosers
14
16
(b) HI-MAR
Scale
MX
Fabeis Takm
Proj
Proj
Febediig Toknn
Proj
(Pid)
Pr
Proj Nx
apps
appis
pisweig
appss
spps
(c) Swale Transformer Block
(d) MLP-based Diffusion
(e) Diffusion Transformer
Figure 2. (a) Pipeline of conventional hierarchical MAR. Conventional hierarchical uses shared Transformer for both phases and directly leverages low-resolution visual tokens to guide second predictions. (b) Pipeline of Hi-MAR. An image and its low-resolution counterpart converted into token sequences at two scales. During inference, both sequences initially masked. In the first phase, masked low-resolution tokens are processed by the Transformer to predict conditional tokens, followed by an MIPP-hased diffusion head for token reconstruction. In the second phase, the masked high-resolution tokens and the predicted conditional tokens in first phase are fed into the Transformer with the Diffusion Transformer head predicting the full high-resolution token sequence. (c) Scale-aware Transformer blocks consist of adaLN-Zere layemorm, self-attention, and berd-forward layers. (d) MLP-based Diffusion head blocks include adaLN, layernorm, and ferw-forward layers. (e) Diffusion Transformer head blocks are composed of adaLN, layernorm, self-attention, and feed-fore forward layers.
all tokens to strengthen image generation. An overview of Hi-MAR is shown in Figure 2 (b).
the noise schedule, and Ey denotes the diffusion head.
Nevertheless, MAR suffers from two limitations: (1) MAR solely hinges on single-scale sequence of visual tokens for predicting next tokens, which combines global structure construction and local details refinement into a single stage. This contrasted to typical human perception, which first captures the global structure and then the local details in a hierarchical manner (Tian et al., 2024). (2) MAR utilizes MLP-based diffusion head to model the masked token probability distribution individually instead of all tokens together as in conventional Diffusion Transformer (Peebles & Xie, 2023; Bao et al., 2023; Lu et al., 2024b). This way ignores the inherent structure of natural images and leaves the interdependency among all visual tokens under- exploited, resulting in a sub-optimal solution for masked token prediction. Such MLP-based diffusion head could produce abnormal bright spots and fail generate correct images (Fan et al., 2024).
3.1. Preliminary
We build our Hi-MAR upon the recent pioneering work of MAR (Li al., 2024). Specifically, given image I E R Xx MAR first utilizes the pre-trained varia- tional autoencoder (VAE) to encode / into latent represen- tations I' E 4xxxd, where h/w/d is the dimension of height/width//hanneh for the latent. Then, the latent 7' is continuous-valued visual tokens X = (11, T2., }. During training, MAR randomly selects [r N] visual tokens and replaces them with masked tokens, where denotes the masking ratio sampled from a pre-defined distribution p(r). The masked sequence X' = {2/, 2 .... x is fed into the masked au- toregressive Transformer. To predict the ground-truth token Ii from the masked one o at position i, diffusion head devised to model the probability distribution of x; con- ditioned on the output of the masked autoregressive Trans- former 2;. The objective of the diffusion head is defined the standard denoising process:
reshaped into sequence of N
3.2. Hierarchical Masked Autoregressive Transformer
To mitigate the first limitation of MAR, we build top-down hierarchical masked autoregressive Transformer (Hi-MAR Transformer) with two phases. The first phase performs bidirectional autoregressive modeling over low-resolution (1)
2() E., [||= co(a]tt,2)) 24) ==)]
where of is the noise-corrupted vector of 24, E is the noise vector sampled from distribution (0, I), / is a time step of
visual tokens to capture the global structure. the second phase, we take the output of the first phase as intermediary

---

Hierarchical Masked Autoregressive Models with Low-Resolution Token Pivots
pivots guide the next autoregressive modeling over typ- ical dense visual tokens for local details refinement. One typical variant (Tian et al., 2024) is to use a shared Trans- former for the two phases and utilize the low-resolution visual tokens directly to guide the second phase prediction, as shown in Figure 2 (a). However, this training strategy of- ten introduces a rraining-inferenc discrepancy. Specifically, during training, the models are conditioned on ground- -tuth low-resolution tokens X predict high-resolution tokens X During inference, since ground-truth tokens are un- available, the models have to first generate low-resolution tokens X=, which may contain errors, and then use these noisy predictions as conditions to predict the high-resolution tokens X This mismatch between clean ground-truth low- resolution tokens X* used in training and noisy predicted low-resolution tokens X' used in inference inevitably re- sults in a discrepancy, resulting in performance degradation. To mitigate such training-inferene discrepancy, we take the conditional tokens output from the Hi-MAR Transformer of low-resolution visual tokens for the second phase instead.
and FFN denote the layernorm, self-attention and feed- forward layer, respectively. The output of the final scale- aware Transformer block acts as conditional tokens for the diffusion head.
3.3. Diffusion Transformer Head
To address the second limitation of MAR, we design new Diffusion Transformer head by exploiting the self-attention to model the interdependency among tokens. In contrast to MLP-based diffusion head that only takes the conditional tokens of masked tokens as conditions, the Diffusion Trans- former head considers all the masked and unmasked condi- tional tokens, as illustrated in Figure 2 (e). The Diffusion Transformer head contains stack of Transformer blocks, and the i-th block with input y; is computed as:
01 71, 71, 02, 32, 72 = sc)it(c),
= + 7 Attention(a) LNN(')) B1), y yo + 72 FFN((2 (N. + :22,,
(3)
The overview of our Hi-MAR is shown in Figure 2 (b). Specifically, in the first phase, the masked low-resolution visual tokens along with the context tokens (e.g., class to- kens or text tokens) are fed into the Transformer, which outputs the conditional tokens z = { 25. ..... zNx } on small scale. An additional diffusion head conditioned on 7a is adopted for the small scale optimization as in MAR (Li et al., 2024). In the second phase, the Transformer takes the concatenation of context tokens, small scale conditional tokens and the masked dense visual tokens as input to gen- erate dense conditional tokens, which are further fed into Diffusion Transformer head for token prediction.
where e denotes the context vector obtained by summating the time step embedding and the conditional tokens, and the input of the first block is the noise- corrupted vector. Note that we only adopt Diffusion Transformer head in the second phase while the first phase still utilizes MLP--baed diffusion head, since the diffusion head on the first phase mainly aims to optimize the low-resolution conditional tokens instead of providing intermediary pivots for the next phase. At inference, we use much fewer steps (e.g., 4 steps) in the second phase considering that the Diffusion Transformer head is much heavier than the MI.P-based diffusion head. With the global structure provided by the first phase, the second phase can focus on the local fine-grained details and requires much fewer steps to generate satisfied results.
Conventional hicrarchical autoregressive variants (Tian et al., 2024) model multi-scale probability distribution by a shared Transformer without additional guidance. This way could make the Transformer ambiguous and is harmful for token prediction. To alleviate the issue, we design scale- aware Transformer block shown in Figure 2 (c). Inspired by the udaLN-Zerv operations adopted by DiT (Peebles & Xie, 2023), we first represent the scale information by sinusoidal embedding. The sinusoidal embedding is fed into MLP layers to generate scale vector v. which is leveraged to regress the scale and shift parameters of layer norm as well as the scaling parameters for residual connection. Specifi- cally, the i-th scale-awar Transformer block with input at is computed as:
4. Experiments
4.1. Datasets
We empirically verify the merit of hierarchical masked au- toregressive models for image generation in comparison with state-of-the-art approaches on two datasets, i.e., Ima- geNet (Deng et al., 2009) and MS-COCO (Lin et al., 2014). For class-conditional image generation, we validate Hi- MAR on ImageNet at 256 x 256 resolution, which con- sists of 1,281,167 training images from IK different classes. For text-to- image generation, we evaluate Hi-MAR on MS- COCO at 256 256, which is composed of 82,783 train- ing images and 40,504 validation images. Each image is annotated with five captions. Following Stable Diffusion (Rombach et al., 2022), we convert captions into sequence of text embeddings with CLIP text encoder. Then the text embeddings act as context tokens and are fed into Hi-MAR for autoregressive modeling.
0 = =a.u+b.
01 31 71, 022 B2, 72 = split(i),
za = 2² + Attention(a) L2' + (1),,
_411_ 2. + 72- FFN LN(zz) + 52),
where a and b are the parameters, split denotes the split operation along the channel dimension, LN, Attention
5

---

Hierarchical Masked Autoregressive Models with Low-Resolution Token Pivots
4.3. Results on ClassCooniiiooaa Image Generation
Table 1. The architecture configurations of the family of Hi-MAR in three different scales (i.e., Base, Large. and Huge). Diff. 1/ /2 denotes the diffusion head on the first/second phase.
Table 2 summarizes the performance comparisons of dif- ferent methods for class-conditional image generation on ImaegNet dataset. All runs are grouped into four categories: the Generative Adversarial Network(GAN) based models, diffusion-basee models, autoregressive models, and masked autoregressive models. Except for the GAN based models, we report the performances of the runs under two different inference settings i.e. with or without Classifier-Fre Free Guid- ance (CFG) (Ho & Salimans, 2021). For our method under the w/o CFG setting, the CFG is only turned off during the prediction of dense tokens, as the first-stage output quality significantly affects Hi-MAR performance.
TE-MAR Transfommrr
Dift Tlend
Ditl. Deadz
Model
#Layers
Hidden sixe
Vlayers
Hidden
ALayers
Hidken
SParams
Hi-MAR-B
24
768
6
1024
512
244M
HIMMR-L
32
1024
8
1280
8
512
529M
Hi M
1280
12
1536
768
1090M
4.2. Experimental Settings
Image Tokenizer. We employ the variational autoencoder (KL-16 version) trained by MAR (Li et al., 2024) to encode low-resolution (128x x 128) and high-resolution (256x 256) images into latent representations for the two phases.
Overall, under the two different settings, the results across most metrics consistently indicate that our Hi-MAR achieves superior performances against the state-of-the-ar models among all the four categories with comparable pa- rameter size. In particular, the FID score of Hi-MAR-B on the base scale achieves 1.93 under CFG, making the absolute improvement over the best competitor MAR-B by 0.38. The results generally highlight the key advantage of exploiting hierarchical autoregressive and modeling inter- dependency among tokens. Specifically, U-ViT and DiT upgrade the conventional U-Net structure diffusion model with Transformer, resulting in remarkable scaling property with superior performances than all GAN based and U-Net structure diffusion models. Note that the use of CFG gener- ally improves the FID, IS and Precision scores for diffusion and AR models across different scales. But the Recall scores decrease due to CFG tends to improve the generation quality at the cost of sacrificing diversity. Compared diffusion models, autoregressive models (c.g.. VQGAN (Esser et al., 2021) and LlamaGen (Sun et al., 2024)) regard images as a sequence of discrete tokens, achieving comparable image generation results. Furthermore, the mask autoregressive models introduce mask tokens into autoregressive modeling, facilitating bidirectional contextual information learning along generative modeling, and thus improve performance. Instead of quantizing images into discrete tokens, MAR utilizes a continuous tokenizer via the diffusion loss, lead- ing to significant performance boosts. But the performance of MAR across different model sizes is still inferior to our Hi-MAR. This validates the effectiveness of hicrarchical au- toregressive modeling and the Diffusion Transformer head for enhanced masked token prediction with richer context information among all tokens.
Network Architectures. Following the architecture con- figurations of MAR family (Li et al., 2024), we build three variants of our Hi-MAR in three different scales (i.e., Base, Large and Huge). To compare with MAR-B/L/H, the num- ber of Transformer blocks in masked autoregressive Trans- former of 1-MM--Bass (Hi-MAR-B), Hi-MAR-Large (Hi- MAR-L), and Hi-MARR-uge (Hi-MAR-H) are set as 24,32, and 40 respectively, The number of Transformer blocks in the diffusion head on both phases of Hi-MMAR-B/L/H is 6/8/12. Table shows the detailed configurations (e.g., layer number, hidden size) of three Hi-MAR variants.
Training Setup. At training stage, we conduct all exper- iments on 80GB-H100 GPUs. For class-conditional im- age generation on ImageNet, we follow MAR (Li et al., 2024) and train the models using AdamW optimizer (8) = 0.9, B2 0.95) with 0.02 weight decay for 800 epochs. We use the constant Ir schedule with le-4 learning rate and 100-epoch linear warmup. In the first phase, the masking ratio is randomly sampled in [0.7, 1.0] as MAR, while the second phase uses the cosine masking strategy following MaskGIT (Chang et al., 2022). For text-to-image genera- tion on MS-COCO, we follow AutoNAT-L. (Ni et al., 2024) and randomly sample the masking ratio by Beta distribution (a = 4, 1). The AdamW optimizer is adopted with an 8c-4 learning rate, 0.03 weight decay and 8K-step linear warmup. The exponential moving average is adopted with momentum of 0.9999. At inference, we use 32 and 4 steps for the first and second phases with a cosine schedule.
Evaluation Metries. For evaluation, we use Fréchet In- ception Distance (FID) (Heusel et al., 2017), Inception Score (IS) (Salimans et al., 2016) and Precision/Recall (Kynkäänniemi et al., 2019) on 50K generated samples to measure the image quality on ImageNet. On MS-COCO, we randomly draw 30K prompts from the validation set and generate samples on these prompts as U-VIT (Bao et al., 2023). We report the FID score as the main metric.
4.4. Results on Text-to-Image Generation
Table shows the performance comparison on MS-COCO for text-to-image generation. For fair comparison, we follow the configuration of U-ViT-S/2 (Deep) (Bao et al., 2023) and build a light-weight version of our Hi-MAR with compara-
6

---

Hierarchical Masked Autoregressive Models with Low-Resolution Token Pivots
Table Generative model family comparison on class-conditional ImageNet 256 "4" "1" indicate lower or higher values are better. Metrics include Fréchet inception distance (FID), inception score (IS), precision and recall. Models with the suffix "-re" used rejection sampling.
w/o CFG Precisionf
w/ CHG Precisionf
Type
Model
#Para.
FID
1S1
Recalff
FID
IST
Recalff
BigGAN (Brock et al., 2019)
112M
6.95
224.5
0.38
GAN
GigaGAN (Kang et al., 2023)
569M
0.84
0.61
StyleGan-XT. (Sauer et al., 2022)
166M
2.30
265.1
0.78
0.53
ADM (Dhariwal & Nichol, 2021)
554M
10.94
101.0
0.69
0.63
4.59
186.7
0.82
0.52
CDM (Ho al., 2022)
4.88
158.7
Diff.
LDM-4-G (Rombach al., 2022)
400M
10.56
103.5
0.71
0.62
3.60
247.7
0.87
0.48
U-VIT-II/2 (Bao al., 2023)
501M
2.29
263.88
0.82
0.57
DiT-XL/2 (Peebles Xie, 2023)
675M
9.62
121.5
0.67
0.67
2.27
278.2
0.83
0.57
VQGAN (Esser ct al., 2021)
227M
18.65
80.4
0.78
0.26
VQGAN-re (Esser ct al., 2021)
1.4B
5,20
280.3
RQTran. (Lec ct al., 2022)
3.8B
7.55
134.0
RQTran.-re (Lee et al., 2022)
3.8B
3.80
323,7
GIVT (Tschannen et al., 2025)
304M
5.67
0.75
0.59
3.35
0.84
0.53
AR
LlamaGen-L (Sun et al., 2024)
343M
19.07
64.3
0.61
0.67
3,07
256.06
0.83
0.52
LlamaGen-XLL (Sun al., 2024)
775M
15.54
79.2
0.62
0.69
2.62
244.08
0.80
0.57
LlamaGen-XXX (Sun al., 2024)
1.4B
14.65
86.3
0.63
0.68
2.34
253.90
0.80
0.59
VAR-d16 (Tian et al., 2024)
310M
3.30
274.4
0.84
0.51
VAR-d20 (Tian et al., 2024)
600M
2.57
302.6
0.83
0.56
VAR-d24 (Tian al., 2024)
1.0B
2.09
312.9
0.82
0.59
MaskGIT (Chang et al., 2022)
227M
6.18
182.1
0.80
0.51
AutoNAT-L (Ni et al., 2024)
422M
2.68
278.8
Mask.
MAR-B (Li ct al., 2024)
208M
3.48
192.4
0.78
0.58
2.31
281.7
0.82
0.57
MAR-L (Li et al., 2024)
479M
2.60
221.4
0.79
0.60
1.78
296.0
0.81
0.60
MAR-H (I.i et al., 2024)
943M
2.35
227.8
0.79
0.62
1.55
303.7
0.81
0.62
Hi-MAR-B B Hi-MAR-L
244M
2.11
251.46
0.80
0.59
293.0
0.81
0.59
Hi-MAR
529M
278.63
0.79
0.62
322.3
0.79
0.61
Ili-MAR-I
1090M
0.80
0.63
322.78
0.80
Similar to the observation in ImageNet, diffusion models with convolutional structure generally exhibit more flexible and scalable generative modeling and thus achieve better performance. U-VIT further obtains better results by placing convolutional structure with Transformer, basically validating the effectiveness of Diffusion Transformer as higher-capacity backbone. There is a large performance gap between MAR and our Hi-MAR. Though both runs belong to masked autoregressive models with continuous tokenizer, Hi-MAR upgrades MAR with hicrarchical masked autore- gressive modeling that creates images from global structure to local details and Diffusion Transformer head mine context among all tokens, yielding apparent improvements.
Table FID results different models on MS-COCO 256 X
256 validation. "I" indicates lower values better.
Type
Model
FID |
AttnGAN (Xu al., 2018)
35.49
DM-GAN (Zhu et al., 2019)
32.64
GAN
DF-GAN (Tao et al., 2022)
19.32
XMC-GAN (Zhang al., 2021)
9.33
LAFITE (Zhou et al., 2022)
8.12
VQ-Diffusion (Gu al., 2022)
19.75
Diffusion
Friro (Fun al., 2023)
8.97
U-ViT-S/2 (Deep)
5.48
AutoNAT-S (Ni al., 2024)
5.36
Mask.
MAR (Li al., 2024)
6.36
Furthermore, we assess the compositional alignment between generated images and input text using T2I- CompBench (Huang al., 2023), a comprehensive bench- mark designed evaluate fine-graine ine-ggrined compositional under- standing in text-to-image generation. We compare Hi- MAR with other state-of-the-ar methods which are trained MS-COCC COCC with similar parameter size. As shown in Table 4, Hi-MAR outperforms existing baselines across several crucial aspects, including attribute binding, object
Hi-MAR
Hi-MAR-S
4.77
ble model size. We group all runs in three categories, i.e., GAN based models, diffusion models, and masked autore- gressive models. In general, our Hi-MAR outperforms other baselines on this challenging dataset. In particular, the FID score of Hi-MAR can achieve 4.77, making the absolute improvement over the best competitor AutoNAT-S by 0.59.

---

Hierarchical Masked Autoregressive Models with Low-Resolution Token Pivots
Table 1ee-CompBench evaluation of different models. indicates higher values are better.
4.5
MAR
Hi-MR
AMtillote Boodng
Object Reletoeship
Medel
Spatil
Syral
Complex
U.N0E-5/2 (Deast (Ban al. 2023)
0.3626
0,2682
0.3474
0,353
0,3003 IL2469
0.219
AsteNAU S EN al, 2004)
0.3225
0.3466
0.338)
00453
0.3024
FID
MARS
0.3663
9/MM
(.200)
02313
23-
Table 5. Ablation study of Hi-MAR. Pivots denote the first phase generated tokeas that act as additional guidance for the next phase. Diff. Head denotes the diffusion head the first/second phase.
2.
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Inference Time (sec per image)
Pivats
Diff. Head,
Dift. Headz
Scale vector
APara.
FID
Figure 3. Speed/accuracy trade-off.
x
MLP-based
208M
visual tokens
MI P. Mlbse
MLP- Phused
245M
2.28
conditional tokens
MLP P-ased
Ml.e-
X
245M
2.07
conditional tokens
MLP-based
Transformer
239M
conditional tokens
Transformer
Transormer
X
235M
1.98
irbesece Time
conditional tokens
MLP-bused
Transformer
242M
relationships, and complex compositions, demonstrating its capability to generate semantically aligned images.
autorgeesstte
[a] low-resolution
(b) high-resolution
Figure 4. Impact of autoregressive steps. The experiments are conducted using the Hi-MAR-B model. For experiments varying low-resolution inference steps, typical dense inference steps are fixed at 4. Similarly, for experiments varying typical dense infer- ence steps, low-resolution inference steps are fixed 32.
4.5. Experimental Analysis
Ablation Study. Here we study how each design in our Hi- MAR framework influences the overall performance, Recall that our Hi-MAR upgrades typical MAR with three novel designs, i.e., Hi-MAR Transformer that creates images from global structure to local details, scale-aware Transformer block that provides scale guidance to the Transformer on the two phases, and Diffusion Transformer head that models the interdependency among tokens and strengthens masked token prediction. Table 5 details the performance across different ablated runs of Hi-MAR on ImageNet for class- start from
of our Hi-MAR (the last row) is finally benefited from our three key designs, and achieves the best performances across most metrics. The results basically validate our design.
Qualitative Results. To qualitatively evaluate our Hi- MAR, we showcase 24 image results generated by MAR and Hi-MAR on ImageNet and MS-COCO in figure Figure 5, We clearly observe that Hi-MAR generates higher-quality images with less distortions and better aligned semantics with input class/caption, validating the effectiveness of ex- ploiting hierarchical autoregressive and modeling interde- pendency among tokens.
conditional image generation. In particular, the base model of typical MAR (the first row in this table), which enables masked autoregressive modeling through a space. By incor-
diffusion loss within a continuous- porating Hi-MAR Transformer and using low-resolution visual tokens to guide the second phase prediction (the sec- ond row), FII) only improves 0.03 due the discrepancy between training and inference as mentioned in Section 3.2. The masked autoregressive Transformer relies too much on the low-resolution tokens and tends to merely resize them into larger resolution. Thus, it fails to correct the flawed tokens predicted by the first phase. When adopting the con- ditional tokens to mitigate such discrepancy (the third row), the FID notably improves over MAR by 0.24. Next, we upgrade the MLP-based diffusion head to Diffusion Trans- former head (the fourth row) for the second phase to model interdependency among tokens, the FID score further im- proves to 1.98. Nevertheless, there is no large improvement when the diffusion head in the first phase is replaced by Diffusion Transformer head (the fifth row). The full version
Speed/accuracy Trade-off. Following MAR (Li et al., 2024), we plot the speed/accuracy trade-off curves of DiT- XL/2, MAR-B and Hi-MAR-B in figure Figure 3. The curve of DiT-XL/2 is obtained by different diffusion steps (50, 75, 100, 250), while the curve of MAR-B is measured by different autoregressive steps (16, 32, 64, 128, 256). For Hi-MAR-B B, we fix the steps on the first phase as 32 and varying the number of steps on the second phase (1,2,4,6,8). We measure the speed on ImageNet 256x using one H100 GPU with batch size 128. is clear that our Hi-MAR has better trade-off than MAR and DIT-XL/2.
Impact of Autoregressive Steps. We analyze the impact of autoregressive steps on both phases, As shown in figure
8

---

Hierarchical Masked Autoregressive Models with Low-Resolution Token Pivots
Hi-MAR
European fire salamander
Cock
Indian cobra
White stark
Maltese
Husky
Panda
Barn
Hi-MAR
Ice bear
Tiger
Beacon
Proboscis monkey
Zebra
EntleBucher
Drake
Rock crab
STOP
Hi-MAR
Dijht
STOP
Yellow umbrellas scattered around the beach along with one chair.
large bast filled wheels.
Sheep grazing in large grassy field near hillsides.
small white bird standing top pond of water.
white plate of food with and veggies.
A guy takes picture car's rear view mirror.
planes window of clouds and
picture of tall stop sign.
Figure 5. Qualitative results on class-conditional image generation and text-to-image generation. The top two rows show class- conditional generation, while the bottom show text-to-image generation.
Figure (a), the FID decreases the step number on the first phase increases and reaches optimal value 32 steps. With the global structure provided by the first phase, the second phase can focus more on the fine-grained local details and manage to achieve nearly saturated FID with just 4 autoregressive steps, as illustrated figure Figure 4 (b). Therefore, we the number of autoregressive steps on both phases 32 and 4 for better trade-off between generation quality and inference speed.
global context. To overcome this, propose new hierar- chical autoregressive prediction framework that establishes hierarchy from low-resolution image tokens, which cap- ture global structure with coarse details, to high-resolution image tokens, which provide fine-grained details. Building on this framework, we introduce the Hicrarchical Masked Autoregressive Model (Hi-MAR), which demonstrates supe- rior performance compared widely-used diffusion models and conventional autoregressive baselines. We hope that work will highlight the importance of utilizing global con- text in visual autoregressive modeling and hope it inspires further research in this direction.
5. Conclusion
In this work, we discuss the limitations next-token predic- tion visual autoregressive modeling caused by the lack of

---

Hierarchical Masked Autoregressive Models with Low-Resolution Token Pivots
Acknowledgments.
Chen, J., Pan, Y., Yao, T., and Mei, T. Controlstyle: Text- driven stylized image generation using diffusion priors. In Proceedings of the 31 ACM International Conference on Multimedia, pp. 7540-7548, 2023.
This work was supported in part by the Beijing Municipal Science and Technology Project No. Z241 100001324002, Beijing Nova Program No. 20240484681 and National Natural Science Foundation of China (No. 62332016) and the Key Research Program of Frontier Sciences, CAS (No.
Deng, J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pat- tern Recognition, pp. 248 255. leee, 2009.
ZDBS-LY-ISCWI),
Impact Statement
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle,
A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.,, Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Esser, P., Rombach, R., and Ommer, B. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12873-12883, 2021.
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang. F., et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F,, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning. 2024.
Bao, F., Nie, S., Xue, K., Cao, Y., Li, C.,, Su, H., and Zhu, J. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22669- 22679, 2023.
Fan, L., Li, T., Qin, S., Li, Y., Sun, C., Rubinstein, M., Sun, D., He, K., and Tian, Y. Fluid: Scaling autoregressive text-to-image generative models with continuous tokens. arXiv preprint arXiv:2410. 13863, 2024.
Brock, A., Donahue, J., and Simonyan, K. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019. URL
Fan, W.-C., Chen, Y.-C., Chen, D., Cheng, Y., Yuan, L., and Wang, Y.-.. F. Frido: Feature pyramid diffusion for complex scene image synthesis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 579-587, 2023.
id-Blxsqj09Pn
Brown, T., Mann, B., Ryder, N., Subbiah, M.. Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33: 1877-1901, 2020.
Gu, S., Chen, D., Bao, J., Wen, F,, Zhang, B., Chen, D., Yuan, L., and Guo, B. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10696-10706, 2022.
Chang, H., Zhang, H., Jiang, L., Liu, C., and Freeman, W. T. Maskgit: Masked generative image transformer. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 11315-11325, 2022.
He, W., Fu, S., Liu, M., Wang, X., Xiao, W., Shu, F., Wang, Y., Zhang, L., Yu, Z., Li, H., et al. Mars: Mixture auto- regressive models for fine- mne-grained text-to- -mage synthe- sis. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 17123-17131, 2025.
Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama, J., Jiang. I.,, Yang, M.-H., Murphy, K., Freeman, W. T., Rubinstein, M., et al. Muse: Text-to-image generation via masked generative transformers. In Proceedings of the 40th International Conference on Machine Learning, pp. 4055 4075, 2023.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in Neural Information Processing Systems, 30, 2017.
10

---

Hierarchical Masked Autoregressive Models with Low-Resolution Token Pivots
Ho, J. and Salimans, T. Classifier-free diffusion guidance. In NeurlPS 2021 Workshop Deep Generative Models and Downstream Applications, 2021.
Lu, J., Clark, C., Lee, S., Zhang, Z., Khosla, S., Marten, R., Hoiem, D.,, and Kembhavi, A. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Puttern Recognition, pp. 26439-26455, 2024a.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in Neural Information Process- ing Systems, 33:6840-6851, 2020.
Lu, Z., Wang. Z., Huang, D., Wu, C., Liu, X., Ouyang, W., and Bai, L. Fit: Flexible vision transformer for diffusion model. In International Conference Machine Learning, pp.
Ho, J., Saharia, C.. Chan, W., Fleet, D. J., Norouzi, M., and Salimans, T. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23347) 2022.
PMLR, 2024b.
Ni, Z., Wang. Y., Zhou, R., Guo, J., Hu, J., Liu, Z., Song, S., Yao, Y., and Huang, G. Revisiting non-autoregressive transformers for efficient image synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7007-7016, 2024.
Huang. K., Sun, K., Xie, E., Li, Z., and Liu, X. T2i- comphench: A comprehensive benchmark for open-world compositional text-to-image generation. Advances in Neu- ral Information Processing Systems, 36:78723-78747, 2023.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. (195-4205, 2023.
Kang, M., Zhu, J.,Y., Zhang, R., Park, J., Shechtman, E., Paris, S., and Park, T. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10124 10134, 2023.
Qian, Y., Cai, Q. Pan, Y., Li, Y., Yao, T. Sun, Q. and Mei, T. Boosting diffusion models with moving average sampling in frequency domain. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19910,,,, 2024.
Kenton, J. D. M.-W. C. and Toutanova, L. K. Bert: Pre- training of deep bidirectional transformers for language understanding. In Proceedings of naacI.-HLT, volume Minneapolis, Minnesota, 2019.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 188)99, 2019.
Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., and Aila, T. Improved precision and recall metrie for assess- ing generative models. Advances in Neural Information Processing Systems, 32, 2019.
Ren, S., Yu, Q., He, Shen, X., Yuille, A., and Chen, L.- C. Flowar: Scale-wise autoregressive image generation meets flow matching. arXiv preprint arXiv:2412.15205, 2024a.
Langley, P. Crafting papers machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning, pp. 1207-1216, Stanford, CA, 2000. Morgan Kaufmann.
Ren, S., Zhu, H., Wei, C., Li, Y., Yuille, A., and Xie, C. Arvideo: Autoregressive pretraining for self- supervised video representation learning. arXiv preprint MXXiv:4455111600 2024b.
Lee, D., Kim, C., Kim, S., Cho, M., and Han, W.S. Autore- gressive image generation using residual quantization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 11523-11532, 2022.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedingy of the IEEF/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10685- 2022.
Li, T., Tian, Y., Li, H., Deng, M., and He, K. Autoregressive image generation without vector quantization. Advances in Neural Information Processing Systems, 37:56424- 56445, 2024.
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for training gans. Advances in Neural Information Processing Systems, 29, 2016.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. I.. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Sauer, A., Schwarz, K., and Geiger, A. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings, pp. 1-10, 2022.

---

Hierarchical Masked Autoregressive Models with Low-Resolution Token Pivots
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Wang, Y., Xiong, T., Zhou, D., Lin, Z., Zhao, Y., Kang, B., Feng, J., and Liu, X. Loong: Generating minute-level long videos with autoregressive language models. arXiv preprint arXiv:2410.02757, 2024b.
Song. Y. and Ermon, S. Improved techniques for train- ing score-based generative models. Advances in Neural Information Processing Systems, 33:12438 12448, 2020.
Waswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A., Kaiser, L.,, and Polosukhin, Attention all you need. Advances in Neural Information Processing Systems, 2017.
Sun, P., Jiang, Y., Chen, S., Zhang, S., Peng, B., Luo, P. and Yuan, Z. Autoregressive model beats diffu- sion: Llama for scalable image generation. CoRR, abs/2406.06525, 2024. URL https: htt/ddi.org////// 48550/arXiv www.exiii......... 06525.
Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., and He, X. Attngan: Fine-graines text to image gener- ation with attentional generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1316-1324, 2018.
Tang. H., Wu, Y., Yang. S., Xie, E.,, Chen, J., Chen, J., Zhang, Z., Cai, H., Lu, Y., and Han, S. Hart: Efficient visual generation with hybrid autoregressive transformer. In The Thirteenth International Conference on Learning Representations, 2025.
Yu, L., Cheng, Y., Sohn, K., Lezama, J., Zhang. H., Chang, H., Hauptmann, A. G., Yang. M.-H., Hao, Y., Essa, I., et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10459-10469, 2023.
Tao, M., Tang, H., Wu, F., Jing. X.-Y., Bao, B.-K., and Xu, C. Df-gan: A simple and effective baseline for text- to-image synthesis. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 16515-16525, 2022.
Yu, L.,, Lezama, J., Gundavarapu, N. B., Versari, I.., Sohn, K., Minnen, D., Cheng, Y., Gupta, A., Gu, X., Haupt- mann, A. G., Gong, B., Yang, M.-H., Essa, I., Ross, D. A., and Jiang, L. Language model beats diffusion tokenizerr is key to visual generation. In The Twelfth International Conference on Learning Representations, 2024. URL https: /owprevveeeeeeeeeeee/eeeeeeeeeee net forum? id g2ggggCA 4g.
Team, C. Chameleon: Mixed-modee carly-fusion foundation models. arXiv preprint arXiv:2405. 098/8, 2024.
Team, G., Amil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Sori- cut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., al. Gemini: a family of highly capable multimodal 2023.
models. arXiv preprint a3Xh::2312.11
Zhang, H., Koh, J. Y., Baldridge, J., Lee, H., and Yang. Y. Cross-modal contrastive learning for text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 833- 842, 2021.
Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahri- ari, B., Ramé, A., et al. Gemma 2: Improving open language models at practical size. arXiv preprint arXiv:2408.001 18, 2024.
Zhang. Z.., Long, F., Pan, Y., Qiu, Z., Yao, T., Cao, Y., and Mei, T. Trip: Temporal residual learning with image noise prior for image-to-vider diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8671 1 11,18,8,2000 2024.
Tian, K., Jiang, Y., Yuan, Z., Peng, B., and Wang, L. Visual autoregressive modeling: Scalable image generation via next-scale prediction. Advances in Neural Information Processing Systems, 37:84839-8443 84865, 2024.
Tschannen, M., Eastwood, C., and Mentzer, F. Givt: Gen- erative ddfiite-vocabulart transformers. In European Conference on Computer Vision, pp. 292-309 Springer, 2025.
Zhou, Y., Zhang, R., Chen, C., Li, C., Tensmeyer, C., Yu, T., Gu, J., Xu, J., and Sun, T. Towards language-free training for text-to-image generation. In Proceedings of the IEEE/CVF conference Computer Vision and Pattern Recognition. pp. 17907-17917, 2022.
Wan, S., Li, Y., Chen, J., Pan, Y., Yao, T., Cao, Y., and Mei, T. Improving virtual try-on with garment-focused diffuuion models. In European Conference on Computer Vision, pp. 184-199. Springer, 2024.
Zhu, M., Pan, P., Chen, W., and Yang, Y. Dm-gan: Dy- namic memory generative adversarial networks for text- to-image synthesis. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 5802-5810, 2019.
Wang, J., Jiang, Y., Yuan, Z., Peng, B., Wu, Z., and Jiang, Y.-G. Omnitokenizer: A joint image-vides tokenizer for visual generation. Advances in Neural Information Processing Systems, 37:28281-28294 2024a.
Zhu, R., Pan, Y., Li, Y., Yao, T., Sun, Z., Mei. T., and Chen, C. W. Sd-dit: Unleashing the power of self-supervised
12

---

Hierarchical Masked Autoregressive Models with Low-Resolution Token Pivots
discrimination in diffusion transformer. In Proceedings of the IEFE/CVE Conference Computer Vision and Pattern Recognition, pp. 8435-8445, 2024.
Zhuo, L., Du, R., Xiao, H., Li, Y., Liu, D., Huang, R., Liu, W., Zhu, X., Wang, F.-Y., Ma, Z., Luo, X., Wang. Z., Zhang, K., Zhao, I.., Liu, S., Yue, X., Ouyang, W., Qiao, Y., Li, H., and Gao, P. Lumina-next: Making lumina-12x stronger and faster with nixt-dit. In The Thirty- Annual Conference on Neural Information Processing Systems, 2024. URL https:/ / http:///ppeeeeeeee/e/eeeeeoo/ee. n Forum?id-iie -i
13


Reviews:
[{"review_id": "1lxH4HX6gO", "reviewer": "Reviewer_S3ag", "summary": "This paper improves Masked Autoregressive models (MAR) by introducing hierarchical modeling, specifically, low resolution is used as pivots. Additionally, MLP-based Diffusion is changed to global Diffusion to further improve the performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:  Add a new row of \"Pivots + MLP-based diffusion heads\" in Table 4**\n\nAppreciate this comment. As suggested, we conducted experiments by including a new ablated run of \"Pivots + MLP-based diffusion heads\" in Table 4. This ablated run enables hierarchical modeling with shared MLP-based diffusion heads, without using any additional Diffusion module. As shown in this table, the FID of this new ablated run (the third row) manages to outperform MAR (the first row) by 0.17, which clearly validates the effectiveness of the hierarchical modeling. We will add the discussion in revision.\n\n|       Pivots       |  Diff. Head_1 |  Diff. Head_2  | Scale vector |  #Para. | FID  |\n| ------------------ | ----------------------- | ------------------------ | ------------ | ------- | ---- |\n|        &cross;     |         &cross;         |         MLP-based        |    &cross;   |  208M   | 2.31 |\n|   visual tokens    |        MLP-based        |         MLP-based        |    &cross;   |  245M   | 2.28 |\n| conditional tokens |       &cross;   |          Shared MLP-based        |    &cross;   |  208M   | 2.14 |\n| conditional tokens |        MLP-based        |         MLP-based        |    &cross;   |  245M   | 2.07 |\n| conditional tokens |        MLP-based        |         Transformer      |    &cross;   |  239M   | 1.98 |\n| conditional tokens |        Transformer      |         Transformer      |    &cross;   |  233M   | 1.98 |\n| conditional tokens |        MLP-based        |         Transformer      |    &check;   |  242M   | 1.93 |", "claims_and_evidence": "The claims made in this paper are supported by both qualitative and quantitative results.", "methods_and_evaluation_criteria": "The proposed method is evaluated on class-conditional image generation and text-to-image generation.", "theoretical_claims": "Not applicable.", "experimental_designs_or_analyses": "Section 4.3 to 4.5 give a comprehensive and solid evaluation on the proposed method.", "relation_to_broader_scientific_literature": "This paper is closely related to autoregressive visual generation, which has broad impacts.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "My biggest concern about this paper is the introduction of global Diffusion. \n\nI admit it helps improve the overall performance, but it is well-known that an additional Diffusion module will improve the image generation ability no matter what methods are used before it.  The focus of this paper is supposed to prove the effectiveness of Hierarchical modeling, so my suggestion to Table 4 is to add a new row of \"Pivots + MLP-based diffusion heads\" to show the gain from Hierarchical modeling only.", "other_comments_or_suggestions": "None.", "questions_for_authors": "This paper provides an intuitive method to improve MAR and shows its effectiveness on a range of tasks, I tend to accept this paper.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OGc17zy9j0", "reviewer": "Reviewer_kLHw", "summary": "The paper introduces Hi-MAR, a hierarchical masked generative model for visual generation. Hi-MAR first predicts low-resolution image tokens as global structural pivots, which then guide the next phase of dense token prediction, enhanced by a Diffusion Transformer head for better global context modeling. Experiments on image generation tasks show that it outperforms baselines and is computationally efficient.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Discussion on VAR and comparisons to \"VAR w/ diffusion heads\"**\n\nThanks. We summarize the contributions of our Hi-MAR against VAR in two points:\n\n1) VAR utilizes the low-resolution visual tokens directly to guide the next phase prediction, which would cause training-inference discrepancy as discussed in section 3.2. During training, VAR takes the ground-truth low-resolution tokens as the condition for the next phase. Since no ground-truth token is available at inference, VAR has to take the predicted noisy low-resolution tokens as the condition, resulting in training-inference discrepancy. Instead, to mitigate such discrepancy, Hi-MAR takes the conditional tokens estimated from the Hi-MAR Transformer of low-resolution visual tokens to trigger the second phase. As shown in Table 4, when replacing the pivots of visual tokens (similarly used in VAR) with our conditional tokens, the FID score clearly improves from 2.28 to 2.07, which validates the effect of the conditional tokens that mitigate training-inference discrepancy.\n2) VAR models multi-scale probability distributions via a shared Transformer without additional guidance. This way leaves the inherent different peculiarities of each scale in autoregressive modeling not fully exploited, resulting in a sub-optimal solution for multi-scale token prediction. In contrast, our Hi-MAR incorporates a scale-aware Transformer block that provides scale guidance to the Transformer tailored to each phase.\n\nMoreover, as suggested, we experimented by implementing VAR with diffusion heads, and \"VAR w/ diffusion heads\" (FID: 2.67) manages to outperform VAR-d16 (FID: 3.30). Nevertheless, the performance of \"VAR w/ diffusion heads\" (FID: 2.67) is still inferior to our Hi-MAR (FID: 1.93), which demonstrates the effectiveness of hierarchical autoregressive modeling. We will add all discussions in revision.\n\n**Q2: Would further stages yield additional improvements**\n\nAppreciate this comment. We experimented by stacking one more stage (64x64 resolution) ahead of the low-resolution stage (128x128 resolution), and the FID score only fluctuates within the range of 0.03. We speculate that the low-resolution stage (128x128 resolution) has already provided sufficient global structure guidance for the next high-resolution stage. The use of an additional stage (64x64 resolution) might introduce unnecessary/redundant global structure information. We will discuss this in the revised version.\n\n**Q3: Autoregressive relationship of Figure 1 (a)**\n\nThanks. To be clear, Figure 1(a) correctly illustrates the left-to-right autoregressive relations among the image token sequence. That is, each predicted token at position *i* (see the bottom output sequence) can only be emitted conditioned on the previous input tokens at positions less than *i* (see the top input sequence). We will clarify this in revision.", "claims_and_evidence": "Fine. The claims are intuitive and easy to follow.", "methods_and_evaluation_criteria": "Fine.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Given the success of recent next-scale prediction approaches, such as VAR, it is not surprising that the proposed cascaded method could be beneficial.\nHowever, its practical advantage over VAR remains unclear.\nAdditionally, the method explicitly adopts a \"two-stage\" approach—would further stages yield additional improvements?\nMore experiments and analyses may be required to address the two concerns.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The reviewer acknowledges that the paper's technical contributions to visual generation are acceptable but not particularly significant, considering the success of VAR. The approach appears to be a specific \"two-scale\" variant of VAR, replacing discrete tokenization (i.e., VQ) with continuous diffusion, inspired by MAR.\nThe authors are encouraged to provide deeper insights into the discussion and include necessary comparisons to \"VAR w/ diffusion heads\" to assess whether additional stages could lead to further improvements.\nAs I am not an expert in this area, I will seek input from other reviewers for a more objective evaluation. This recommendation is not final.", "other_comments_or_suggestions": "Has the autoregressive relationship in Figure 1(a) been drawn incorrectly? Is the causal relationship represented by the black connecting lines reversed? It seems like a left-right mirror flip of the figure would be correct.\n\n##### Post-rebuttal: Thanks to the authors for the detailed responses. I am ok with the rebuttal. I hope the discussions can be incorporated into the revision. Good luck.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p4TpcHWJ3u", "reviewer": "Reviewer_SRgF", "summary": "This paper proposes a hierarchical masked AR visual generation model with low-resolution tokens as pivots. By first generating low-resolution image tokens, which provide a global structure, the second generation phase can benefit from the global context. Besides, the proposed diffusion transformer head to further improve the results. Experimental results show that Hi-MAR can obtain better performance than baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Novelty**\n\nThanks. We summarize the differences between Hi-MAR and conventional multi-scale generation models as follows:\n\n1) During training, the conventional models (i.e., Muse, VAR, Hart) commonly utilizes the ground-truth low-resolution visual tokens directly to guide the next phase prediction. Instead, Hi-MAR takes the conditional tokens estimated from the Hi-MAR Transformer of low-resolution visual tokens as the condition. Such design can mitigate the training-inference discrepancy as discussed in section 3.2. As shown in Table 4, the FID score improves from 2.28 to 2.07 when adopting the conditional tokens to mitigate such discrepancy.\n2) Both VAR and Hart models multi-scale probability distribution via a shared Transformer without additional guidance. This way leaves the inherent different peculiarities of each scale in autogressive modeling not fully exploited, resulting in a sub-optimal solution for multi-scale token prediction. In contrast, our Hi-MAR incorporates a scale-aware Transformer block that provides scale guidance to the Transformer tailored to each phase.\n3) Both VAR and Muse discretize images by VQGAN, resulting in severe information loss. Instead, Hi-MAR adopts continuous tokenizer via a diffusion loss, overcoming the poor generation upper bound caused by the vector quantization.\n4) Muse utilizes two different models for low/high-resolution image generation and the two models are trained separately. Instead, our Hi-MAR jointly optimizes the probability distribution for low/high-resolution tokens with a shared scale-aware masked autoregressive Transformer and two small diffusion heads, which is more parameter-efficient.\n5) In contrast to Hart that utilizes an MLP-based diffusion head to model the each token probability distribution individually, Hi-MAR devises Diffusion Transformer head by exploiting the self-attention to model the interdependency among tokens. Note that we appreciate the suggested reference of concurrent work Hart (pulished on ICLR 2025 with camera ready deadline on Mar 14, 2025). We will add the discussion.\n\n\n\n\n**Q2: Speed comparison with VAR and HART**\n\nAs suggested, we compare the speed of our Hi-MAR with mentioned VAR and HART in the following table. The results basically demonstrate that Hi-MAR achieves superior performance against VAR and HART with comparable computational costs. We will add this in revision.\n\n| Method   | #Para. | FID  | Phase_1 Steps     | Phase_2 Steps  | Diff. Head_1 Step | Diff. Head_2 Step | Inference Time/per image |\n| -------- | ------ | ---- | ------------ | ------------ | ------------------------ | ------------------------ | ------------------------ |\n| VAR-d20  | 600M   | 2.57 | 10           | &cross;      | &cross;                  | &cross;                  | 0.14489                  |\n| HART-d20 | 649M   | 2.39 | 10           | &cross;      | 8                        | &cross;                  | 0.15401                  |\n| DiT-XL/2 | 675M   | 2.27 | 250          | &cross;      | &cross;                  | &cross;                  | 0.78970                  |\n| MAR-B    | 208M   | 2.31 | 256          | &cross;      | 100                      | &cross;                  | 0.52641                  |\n| Hi-MAR-B | 244M   | 2.00 | 32           | 4            | 100                      | 50                       | 0.13587                  |\n| Hi-MAR-B | 244M   | 1.93 | 32           | 4            | 100                      | 250                      | 0.28552                  |\n\n**Q3: Other resolution settings**\n\nThanks. We experimented by equipping Hi-MAR-L with larger resolution (i.e., 256 -> 512), and the FID score achieves 1.62, which outperforms MAR-L with 512 resolution by 0.11. The result again validates the effectiveness of exploiting hierarchical autoregressive and modeling interdependency among tokens. We will add this.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No theoretical claims", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "I think the key problem with Hi-Mar is the novelty improvement with Muse[1], VAR[2], and Hart[3]. \n1. Muse also uses a super-resolution transformer to generate the final image with low-resolution information fused by cross-attention. \n2. VAR proposes to use multi-scale generation to image generation. Hi-Mar only uses one low-resolution scale. I think this is a special case  for VAR.\n3. In Hart (missing reference), they use residual diffusion to improve the performance of VAR. \nConsidering these three papers, the novelty of Hi-Mar is limited enough.\n\n\n[1] Muse: Text-to-image generation via masked generative transformers. ICML 2023.\n[2] Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction. NeurIPS 2024.\n[3] HART: Efficient Visual Generation with Hybrid Autoregressive Transformer. ICLR 2025.", "essential_references_not_discussed": "See Relation To Broader Scientific Literature.", "other_strengths_and_weaknesses": "Strengths: \n1. The paper is easy to follow, and the idea is intuitively easy to understand. \n2. The performance is impressive.\n3. Reduce the ar steps in the second step and improve the generation speed.\nWeakness: \n1. Limited novelty. See Relation To Broader Scientific Literature\n2. Missing speed comparison with VAR and HART.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Have you tried other resolution settings, like 128->512,  256->512?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2HEjowo1Dd", "reviewer": "Reviewer_dCxF", "summary": "This paper proposed a Hierarchical Masked Autoregressive Model based on MAR (Li et al., 2024) by introducing a low-resolution modeling phase, which is claimed to provide global structure guidance for generating dense image tokens. Specifically, x2-lower-res image tokens are first modeled by a scale-aware transformer block in a bi-directional masked-modeling manner, followed by a MLP diffusion head to predict the continuous latents. Then, at the second phase, the predicted latents, rather than the ground-truth tokens, are used as additional conditions for generating high-res tokens with the same scale-aware transformer but a different transformer-based diffusion head. The proposed model is tested on ImageNet and MS-COCO for C2I and T2I generation and compared with existing approaches.\n\n**update after rebuttal**: Based on the reviews, rebuttal, and discussions, I find this paper to be borderline. While the additional information and experiments provided in the rebuttal help strengthen the paper within its reasonable scope, I remain unconvinced by its level of novelty. That said, I have raised my final rating to a weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Training-inference discrepancy issue**\n\nYes. We mainly focus on the design of hierarchical masked autoregressive model which addresses the training-inference discrepancy for passing tokens from phase 1 to phase 2, while the discrepancy caused by the inherent autoregressive process in each stage still remains. Such discrepancy is also occurred on most existing autoregressive models. We will discuss this.\n\n**Q2: Speed and accuracy behave for longer visual sequences**\n\nAs suggested, we experimented on larger 512 resolution and Hi-MAR-L (FID: 1.62) outperforms MAR-L (FID: 1.73), while its computational cost is 20.9% less than MAR-L. We will add this.\n\n**Q3: More comprehensive benchmarks**\n\nThanks. As suggested, we evaluate T2I model on T2I-CompBench and GenEval benchmarks. Due to limited computational resources, here we compare our Hi-MAR with other state-of-the-art methods (e.g., U-ViT-S and AutoNAT-S) which are trained on MS-COCO with similar parameter size. The larger models with billions of parameters trained on billions of images (e.g., SDXL, SD3) are not included. As shown in the following tables, our Hi-MAR consistently outperforms other baselines with comparable parameter size. We will add this.\n\n| Method    | Single Obj. | Two Obj. | Counting | Colors | Positions | Color Attri. | Overall |\n| --------- | ----------- | -------- | -------- | ------ | --------- | ------------ | ------- |\n| U-ViT-S   |   83.75     |  18.69   |  16.25   | 38.03  |  3.00     |   0.75       |  26.75  |\n| AutoNAT-S |   81.25     |  18.18   |  16.56   | 36.70  |  3.75     |   1.25       |  26.28  |\n| Hi-MAR-S  |   89.06     |  22.73   |  17.50   | 44.41  |  2.50     |   2.25       |  29.74  |\n\n| Method    | Color  | Shape  | Texture | Spatial | Non-Spatial | Complex |\n| --------- | ------ | ------ | ------- | ------- | ----------- | ------- |\n| U-ViT-S   | 0.3626 | 0.2682 | 0.3474  | 0.0353  | 0.2693      | 0.2219  |\n| AutoNAT-S | 0.3225 | 0.2466 | 0.3389  | 0.0453  | 0.2468      | 0.2024  |\n| Hi-MAR-S  | 0.3862 | 0.2782 | 0.3945  | 0.0409  | 0.2690      | 0.2313  |\n\n**Q4: Configuration of Hi-MAR-S**\n\nFor fair comparison, Hi-MAR-S follows the configuration of U-ViT-S/2 (Deep) and its exact configuration is shown in the following table. We will add this in Table 1.\n\n|          |         | Hi-MAR Transformer |         | Diff. Head1 |         | Diff. Head2 |         |\n| -------- | ------- | ------------------ | ------- | ----------- | ------- | ----------- | ------- |\n| Model    | #Layers | Hidden size        | #Layers | Hidden size | #Layers | Hidden size | #params |\n| Hi-MAR-S | 17      | 512                | 5       | 512         | 5       | 512         | 108M    |\n\n**Q5: Training setup**\n\nTo be clear, we only employ the diffusion loss as the training objective on both stages. Following MAR, we set the maximum timestep as 1,000 for the Diffusion heads. The models in Figure 4 follow this training setup.\n\n**Q6: The gap between Hi-MAR and MAR reduces. The scalablility of Hi-MAR to larger images and larger models**\n\nThe performance in ImageNet is almost saturated and it is relatively difficult to introduce large margin of improvement. Considering this comment, we conducted the suggested experiments on larger resolution (i.e., 256 -> 512) and the FID achieves 1.62, which improves over MAR-L by 0.11. Furthermore, we scale both MAR and Hi-MAR to 2B parameters, and the FID of MAR and Hi-MAR is 1.49 and 1.45 respectively. The results basically demonstrate that Hi-MAR is scalable to both larger images and larger models. We will add this.\n\n**Q7: Concurrent works**\n\nWe appreciate the suggested concurrent works and we are also happy to discuss them in revised version.\n\n**Q8: Safe innovation**\n\nPlease refer to Q1 of Reviewer SRgF for more discussion on technical contribution against existing works.\n\n**Q9: More visualizations**\n\nAs suggested, we provide more visualization results on the [link](https://anonymous.4open.science/r/HiMAR_Visual/README.md). We will add this.\n\n**Q10: Impact on phase 1 resolution**\n\nWe experimented with smaller resolution (i.e., 64x64) on phase 1 and the FID score is degraded into 2.06 due to the quality of images generated in such smaller resolution is relatively low. Therefore, we choose 128x128 resolution for the first phase. We will add this.\n\n| Phase 1 Resolution | FID  |\n| ------------------ | ---- |\n| 64x64              | 2.06 |\n| 128x128            | 1.93 |\n\n**Q11: Other design choices of the diffusion heads**\n\nWe also experimented by replacing the self-attention layer with cross-attention in the Diffusion Transformer head to mine the context among all tokens, and the FID is slightly dropped by 0.05 compared to the final version of Hi-MAR. We will discuss this in ablation study.", "claims_and_evidence": "The essential claims, including the lack of global context, training-inference discrepancy, independent sampling issue, and speed/accuracy trade-off, are well-justified and largely supported by empirical results within the scope of this paper.\n\nHowever, whether these claims are conclusive for large-scale problems or models is unclear. For example, \n  - The **training-inference discrepancy** issue is only addressed for passing tokens from phase 1 to phase 2, while during the autoregressive process of stage 1 and stage 2, teacher-forcing learning is still applied.\n  - It is unclear how **speed and accuracy** behave for longer visual sequences. For example, will the additional phase 1 and the Diffusion Transformer head in phase 2 lead to much higher computational cost for higher-res images?", "methods_and_evaluation_criteria": "The proposed method is largely based on MAR (and incorporates many ideas from the VAR paper about scales). It is reasonable, well-motivated, and clearly feasible.\n\nThe paper follows MAR and uses the most widely applied ImageNet dataset to evaluate 256x256-res C2I generation. The paper also uses MS-COCO to evaluate T2I results. Both are very common practices and reasonable.", "theoretical_claims": "The essential claims, including the lack of global context, training-inference discrepancy, independent sampling issue, and speed/accuracy trade-off, are well-justified and *largely supported* by empirical results *within the scope of this paper*. I have mentioned potential concerns in Claims And Evidence and Experimental Designs Or Analyses in my review, please refer to those sections for details.", "experimental_designs_or_analyses": "1. I am not satisfied with the T2I experiment. The model is only tested on MS-COCO and only compares FID, which can hardly reflect the actual performance. \n - Other more comprehensive benchmarks should be evaluated, such as T2I-CompBench (Huang et al., 2023) and GenEval (Ghosh et al., 2023).\n - Only a small-scale model (Hi-MAR-S) is tested. By the way, the exact configuration of Hi-MAR-S is not specified in Table 1.\n\n2. Is the diffusion loss at the second stage the only training objective? How many steps are the Diffusion heads trained for? Is it the same as MAR (1000 steps)? And do the models in Figure 4 follow this training setup?\n\n3. Table 2 (w/ CFG) shows that as the model size increases, the gap between Hi-MAR and MAR reduces. I am concerned that the proposed method is not scalable to larger images and large models.", "relation_to_broader_scientific_literature": "The proposed method is largely based on MAR (Li et al., 2024) and incorporates many ideas about scales from the VAR (Tian et al., 2024) paper. In short, MAR proposes a diffusion head based on masked-modeling bidirectional autoregressive model for predicting continuous latents instead of discrete code. VAR proposes a scale-wise instead of token-wise autoregressive paradigm. The key contribution of this paper, introducing an extra smaller-scale autoregressive modeling phase to the single-stage MAR framework, is highly relevant to the VAR's scale-wise idea.", "essential_references_not_discussed": "This paper includes and clearly refers to most of the essential literature, and I have no problem with this. There are many concurrent works about AR+continuous latent and AR+multi-scale (e.g., Fluid, Infinity, FlowAR, HART, FlexVAR, FractalAR, ...) that can be added to the later version of this paper.", "other_strengths_and_weaknesses": "I like the overall idea of introducing scale-wise modeling to MAR, along with other nice adaptations. To me, this is a safe innovation but somewhat incremental. My major concern is that the scope and depth of this study is too small or shallow to reveal the potential of the approach.", "other_comments_or_suggestions": "At the current stage, I think the most doable items are expanding the T2I experiment, providing more visualizations, and improving the analysis to include more insightful studies, such as the impact on phase 1 resolution and other design choices of the diffusion heads.\n\nTo strengthen this paper, I expect experiments on longer visual sequences (higher-resolution images) and larger models to justify the proposed method's scalability, efficiency, and generalization ability.", "questions_for_authors": "I have specified my concerns and questions in the sections above, please refer to those parts.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors present Hierarchical Masked Autoregressive models (Hi-MAR) for image generation. Specifically, Hi-MAR first generates low-resolution tokens, which are used to guide the generation of high-resolution tokens. The authors also propose Diffusion Transformer head to improve global context modeling. The effectiveness of the proposed method is demonstrated on ImageNet (class-condition) and COCO (text-to-image).

In the initial reviews, the reviewers expressed several concerns:

* Reviewer dCxF: Clarification of training-inference discrepancy issue; speed and accuracy behave for longer visual sequences; more Text-to-Image benchmark evaluations; inclusion of related works (Fluid, Infinity, FlowAR, HART, FlexVAR, FractalAR); ablation on the effect of phase 1 resolution, and so on.

* Reviewer SRgF: Limited novelty due to the incremental improvement over Muse, VAR, and HART; missing speed comparison with VAR and HART; ablation on different resolutions.

* Reviewer kLHw: Detailed comparison with VAR; ablation on additional stages; effectiveness-efficiency trade-off

* Reviewer S3ag: Additional ablation study for Tab. 4.

The provided rebuttal effectively assuaged the reviewers' concerns to some degree, while Reviewers kLHw and dCxF are still on Borderline. After carefully considering the reviews, author rebuttal, discussions, and the draft, the AC agrees with the reviewers that the proposed method is somewhat incremental, but appreciates the authors' efforts. As a result, the AC recommends acceptance of the paper.

However, the authors are strongly encouraged to incorporate feedback from the reviews and discussion into the final version. For example, it is critical to clearly discuss the difference between the proposed method and existing works (e.g., the discussion/comparison with VAR/Muse/HART/FlowAR). As a side note, the AC observes that the proposed scale-aware Transformer shares similarities with the scale-wise Flow Matching module in FlowAR; a more detailed discussion on this point would be greatly appreciated.
Decision: Accept (poster)