ICML 2025

Hierarchical Masked Autoregressive Models with Low-Resolution Token Pivots
Guangting Zheng 1 † Yehao Li 2 Yingwei Pan 2 Jiajun Deng 3 Ting Yao 2 Yanyong Zhang 1 Tao Mei 2
Abstract
Autoregressive models have emerged as a pow-
erful generative paradigm for visual generation.
The current de-facto standard of next token pre-
diction commonly operates over a single-scale
sequence of dense image tokens, and is incapable
of utilizing global context especially for early
tokens prediction. In this paper, we introduce
a new autoregressive design to model a hierar-
chy from a few low-resolution image tokens to
the typical dense image tokens, and delve into a
thorough hierarchical dependency across multi-
scale image tokens. Technically, we present a
Hierarchical Masked Autoregressive models (Hi-
MAR) that pivot on low-resolution image tokens
to trigger hierarchical autoregressive modeling
in a multi-phase manner. Hi-MAR learns to pre-
dict a few image tokens in low resolution, func-
tioning as intermediary pivots to reflect global
structure, in the first phase. Such pivots act as
the additional guidance to strengthen the next au-
toregressive modeling phase by shaping global
structural awareness of typical dense image to-
kens. A new Diffusion Transformer head is fur-
ther devised to amplify the global context among
all tokens for mask token prediction. Extensive
evaluations on both class-conditional and text-to-
image generation tasks demonstrate that Hi-MAR
outperforms typical AR baselines, while requiring
fewer computational costs. Code is available at
https://github.com/HiDream-ai/himar.
† This work was performed when Guangting Zheng was visit-
ing HiDream.ai as a research intern. 1University of Science and
Technology of China, Anhui, China 2HiDream.ai Inc, Beijing,
China 3The University of Adelaide, Adelaide, Australia. Corre-
spondence to: Yingwei Pan <panyw.ustc@gmail.com>, Yanyong
Zhang <yanyongz@ustc.edu.cn>.
Proceedings of the 42 nd International Conference on Machin...