Autoregressive Image Generation with Randomized Parallel Decoding
Authors: Haopeng Li
Jinyue Yang
Guoqi Li
Huan Wang
Venue: ICLR 2026 Poster (2026)
Keywords: autoregressive image generation
parallel decoding
next-token prediction
Primary area: generative models
TL;DR: parallel autoregressive image generation in random-order

Abstract: We introduce ARPG, a novel visual Autoregressive model that enables Randomized Parallel Generation, addressing the inherent limitations of conventional raster-order approaches, which hinder inference efficiency and zero-shot generalization due to their sequential, predefined token generation order. Our key insight is that effective random-order modeling necessitates explicit guidance for determining the position of the next predicted token. To this end, we propose a novel decoupled decoding framework that decouples positional guidance from content representation, encoding them separately as queries and key-value pairs. By directly incorporating this guidance into the causal attention mechanism, our approach enables fully random-order training and generation, eliminating the need for bidirectional attention. Consequently, ARPG readily generalizes to zero-shot tasks such as image in-painting, out-painting, and resolution expansion. Furthermore, it supports parallel inference by concurrently processing multiple queries using a shared KV cache. On the ImageNet-1K 256 benchmark, our approach attains an FID of 1.83 with only 32 sampling steps, achieving over a 30 times speedup in inference and and a 75 percent reduction in memory consumption compared to representative recent autoregressive models at a similar scale.

Published as a conference paper at ICLR 2026
AUTOREGRESSIVE IMAGE GENERATION WITH RANDOMIZED PARALLEL DECODING
Haopeng Li
Jinyue Yang²
Guoqi Li
Huan Wang'
WWstlaake University Institute of Automation, Chinese Academy of Sciences
https: / /ttgt/ttttttttt//tt com hp- 133/ ARPG
L
108
13
Thrsuphpe Saughi
Trichet laceptinn Dusuace,
(CB)
Class-Coniiional Generation Test-lo-lmage Generation
Quality and Fificiincy Comperixa
Controlabee Image Geserttinn
(6) Class-Cnndtionull Editing, Inpaining & Outpainting
(c) Resolution Expansion
Visual Autoregressive Model with Randomized Parallel Generation (ARPG): A high- quality and efficient framework for image synthesis. ARPG enables (a) class-conditional and text-to- image generation with just 64-step parallel decoding and (b) outperforms recent representative works (e.g., VAR (Tian et al., 2024), LlamaGen (Sun et al., 2024a)) in throughput, memory consumption, and quality. It further supports (c) controllable generation and zero-shot generalization, including (d) class-conditional editing, inpainting, outpainting, and (e) resolution expansion.
Figure 1:
ABSTRACT
We introduce ARPG, a novel visual Autoregressive model that enables Randomized Parallel Generation, addressing the inherent limitations of conven- tional raster-order approaches, which hinder inference efficiency and zero-shot generalization due to their sequential, predefined token generation order. Our key insight is that effective random-order modeling necessitates explicit guidance for determining the position of the next predicted token. To this end, we propose novel decoupled decoding framework that decouples positional guidance from content representation, encoding them separately as queries and key-value pairs. By directly incorporating this guidance into the causal attention mechanism, approach enables fully random-order training and generation, eliminating the need for bidirectional attention. Consequently, ARPG readily generalizes zero-shot inference tasks such image inpainting, outpainting, and resolution expansion. Furthermore, it supports parallel inference by concurrently processing multiple queries using shared KV cache. On the ImageNet- IK 256x256 benchmark, our approach attains an FID of 1.83 with only 32 steps, achieving a 30x and speedup in inference over raster-order and recent parallel AR models, respectively, while reducing memory consumption by 75 % at a similar scale.
Work done during the author's research internship. "orresponning authors: ewanghuan@wess ake.edu.ccm <guogi 1110ia en>.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Autoregressive (AR) models have demonstrated remarkable performance and scalability (Henighan al., 2020; Kaplan et al., 2020), particularly in the domain of large language models, where the next-token prediction paradigm has driven significant advancements (Achiam et al., 2023; et al., 2024). This success has extended to visual synthesis, enabling breakthroughs image generation with methods like VQGAN (Esser al., 2021; Ramesh et al., 2021; Lee et al., 2022). However, directly applying next-token prediction to images presents fundamental challenges. Unlike text, which possesses an inherent causal structure, images defined over two-dimensional spatial domain. AR models necessitate flattening this spatial information into a sequence, typically following rigid, predefined order (e.g., raster-scan). This strictly sequential generation process is not only inefficient, especially for high-resolution images, but also fundamentally limits the model's ability to perform zero-shot generalization to tasks requiring non-causal dependencies.
autoregressive
To address these challenges, alternative approaches, such as MaskGIT (Chang et al., 2022), have adopted masked modeling (Devlin al., 2019) approach for parallel token generation in random order. However, the reliance on bidirectional attention prevents the use of the KV cache, resulting in high computational overhead. Block-wise AR (Tian et al., 2024; Liu et al., 2024; Wang et al., 2025; Chang et al., 2023) enable block-wise parallel decoding, however, they are constrained by fixed block orderings and sampling schedules, which limit their flexibility and fidelity. Recent work RandAR (Pang et al., 2025) enables fully random-order training and inference with causal attention via positional instruction tokens, but incurs significant memory and computational costs due to the increased sequence length. Therefore, fundamental challenge persists: how to achieve flexible- order and parallel generation without compromising computational efficiency.
In this work, we introduce ARPG, novel visual AR model that enables training and inference in fully random token orders through a decoupled decoding process. Unlike the standard decoder-only Transformers (Vaswani et al., 2017), our approach decouples the prediction process into two distinct passes: (1) The content refinement pass utilizes causal self-attention over random-order token sequence construct tabel-leakage-feee content representations as key-value pairs, without directly predicting tokens. (2) In the position-guided prediction pass, data-independent [MASK] tokens endowed with positional information corresponding right-shift of the input, act as target-aware queries. These queries use causal cross-attention to predict their respective target tokens based on the content key-value pairs processed the first pass. This design allows the model to be trained within fully causal paradigm while enabling generalization to block-wise parallel decoding with flexible token orders, as multiple queries can be processed independently in a single step.
Extensive experiments demonstrate the superiority of ARPG. On the ImageNet- IK 256 256 bench- mark (Deng et al., 2009), ARPG at various scales achieve FID (Heusel et al., 2017) of 2.30, 1.93, and 1.83. This performance surpasses recent works while achieving nearly 30x and 3x speedup over raster-order and parallel AR models, respectively, and reducing memory usage by 75%.
Furthermore, we extend our method more complex tasks, including controllable generation, zero- shot generalization, and text-to-image generation, as shown in Fig. 1. In summary:
1. We propose novel visual autoregressive framework that enables parallel image generation with a random token order using a decoupled two-pass decoding mechanism, overcoming the ineffi- ciencies and poor generalization of traditional next-token prediction methods.
2. We explore the zero-shot generalization ability of our method and further extend it to versatile controllable generation and text-to-image generation.
Extensive experiments demonstrate that our approach achieves competitive generation quality while simultaneously excelling in throughput and memory consumption, setting new bench- mark for efficient, high-performance autoregressive image generation.
2 RELATED WORK
2.1 AUTOREGRESSIVE IMAGE GENERATION
Sfate-of-in Srate-of-the-ae the-art large language models (Achiam et al., 2023; et al., 2024) adopt a decoder-only Trans- former (Vaswani et al., 2017) for causal modeling of language sequences and autoregressive gener-
2

---

Published as a conference paper at ICLR 2026
99
MAR
Block-AR
RandAR
ARPG
Image Token
Ohaipat Tiikers
Figure 2: Methods for representing the position next token. MAR (Chang et al., 2022; et al., 2024b) indicates the position via masking out image token; Block-AR models (Tian et al., 2024; Wang et al., 2025; He et al., 2025) use predefined positions: RandAR (Pang et al., 2025) intersperses position tokens throughout the sequence; and ARPG integrates
query cross-attention mechanism.
ation, method commonly known as the GPT (Achiam et al., 2023) style approach. In the vision domain, images be quantized into discrete tokens (Van Den Oord al., 2017) and flattened from 2D to ID, enabling generation via the next-token prediction paradigm, as seen in models like VQGAN (Esser al., 2021), LlamaGen (Sun al., 2024a), etc. (Lee al., 2022: Wang al., 2024; Li et al., 2024a; Wang et al., 2024; Yu et al., 2024). These methods have demonstrated impressive generative performance. However, this token-by-token image generation approach inefficient, especially when dealing with high-resolution images. Additionally, since the generation can only proceed in a specific token order, it encounters difficulties zero-shot inference that require non- causal dependencies, such as inpainting and outpainting.
2.2 ACCELERATION OF VISUAL AUTOREGRESSIVE MODELS
Another mainstream approach to sequence modeling is the encoder-only architecture. This method, widely used in language models like BERT (Devlin et al., 2019), involves randomly masking and then predicting multiple tokens sequence. In the vision domain, this paradigm is adopted by models such as MaskGIT (Chang et al., 2022) for image generation. By leveraging bidirectional methods eliminate causal dependencies, enabling multi-token
attention, these masseed-generatii generation in a single step and thus significantly faster inference. However, due the absence of KV cache, their overall inference efficiency remains limited. Beyond this, other parallel generation techniques face their own challenges. Many block-wise autoregressive methods (Tian et al., 2024; Wang et al., 2025; He et al., 2025) are constrained by predefined orders and schedulers, while various training-fres free methods (Teng et al., 2025) achieve acceleration at the cost of degraded output quality. A recent work, RandAR (Pang et al., 2025), inspired by XI.Net (Yang et al., 2019), implements permuted autoregressive model for parallel decoding by interspersing position tokens throughout the sequence. This strategy, however, incurs heavy penalty: it doubles the sequence length, thereby increasing both the computational load and the memory required for the KV cache.
METHOD
3.1 RETHINKING VISUAL AUTOREGRESSIVE MODELING
Preliminaries. Given sequence X {I1, 22,.., ), causal autoregressive model predicts the next token based on all preceding tokens, following the probabilistic formulation:
p(x) II p(z: F1,22. 22,, (..))......)
(I)
The formulation for block-wise AR models (Tian et al., 2024; Wang et al., 2025) is analogous, substituting a single token æ; with a set of tokens (z;,
Ft+)).
In contrast, masked sequence modeling (Devlin et al., 2019) processes sequences with certain tokens masked out and predicts them based on the unmasked context. The objective to minimize the negative logelikelidood of the tokens at masked positions (Devlin et al., 2019):
Σ m,=1
logp(z; XM)]. Vi (1,2, .n..
where M (m1, m2,... m., E (0, 1)" is masked positions and XM is the corrupted sequence.
3

---

Published as a conference paper at ICLR 2026
Mass
Mark
Mack
18
16
16
Unmuaa
Uniquak
12
Figure 3: Analysis of attention scores. Normalized attention maps from multiple distinct heads
in the final layer of RandAR (Pang et al., 2025). The maps, partitioned by token type (masked vs. unmasked), reveal that attention weights are predominantly concentrated on unmasked tokens.
Key Insights. Based on the characteristies of the methods discussed, we find:
Insight 1: Breaking the order-specific constraints of model requires explicit positional guidance.
This insight stems from the fundamental difference in how models determine the prediction target. As shown in Fig. 2. causal autoregressive models are bound to a strict, predefined generation order (e.g., raster-scan for standard AR. scale or diagonal patterns for block-wise AR), which inherently limits their flexibilityy. Masked modeling using position-awarr [MASK] tokens to act as explicit instructions, directing the model the specific locations that need to be filled. This mechanism of providing direct positional guidance is what allows the model to predict tokens a flexible manner.
Insight 2: Masked modeling is inherently training-inee inefficient due to its sparse prediction objective.
This inefficiency arises because the loss function is calculated exclusively on [MASK] tokens, which represent only fraction of the input sequence. More precisely, the query vectors corresponding unmasked tokens receive no direct gradients from the optimization objective. Consequently, in any given training step, the model parameters suboptimally updated using only subset of the tokens, yet this process incurs the full computational cost equivalent to processing the entire sequence. The related proof is provided in Appendix C.
Insight 3: Attention directed towards [MASK] tokens is redundant.
This redundancy is empirically evident, exemplified by the RandAR (Pang et al., 2025) attention scores in Fig. 3, which show that [MASK] tokens contribute minimally to the attention mechanism. The vast majority of attention scores are allocated to unmasked tokens. We argue this also con- ceptually sound based on the roles of different tokens. An unmasked token should attend to other unmasked tokens enrich semantic representation. A [MASK] token, tasked with reconstructing the original token at its position, should attend to semantic-rich unmasked tokens to gather contex- tual information. neither case necessary to attend to other [MASK] tokens, which contain no native semantic content.
3.2 VISUAL AUTOREGRESSIVE MODEL WITH RANDOMIZED PARALLEL DECODING
Reformulation. We observe that the essential components for predicting token Tr, at position тe within an arbitrary permutation T indices are: (1) the set of already known tokens {x+, 1f-1 and (2) the target position itself. The state of other unknown positions is irrelevant. This motivates reformulation where the prediction is explicitly conditioned only on these necessary elements:
P Iri Try Ir. fo fo (L Tt)
1=1
Here, fo is model parameterized by 0 that takes the known tokens and the target position as separate inputs. Based on Insights 2 and 3, we configure the softmax attention (Vaswani al., 2017) such that queries are derived exclusively from data data-independent [MASK] tokens, while the keys and values originate entirely from unmasked content tokens. This decoupling yields three key advantages. (1) The content representations can be learned independently. (2) Decoding is guided by positional queries that attend to rich, fully-learned KV representations, rather than the less informative ones from shallow layers. (3) The projection parameters of query not shared with key-value, enabling full learning in single training step without redundancy computation. These points constitute the key distinctions from prior works (Yang et al., 2019; Pang et al., 2025; Liu al., 2024).
4

---

Published as a conference paper at ICLR 2026
Cached
Ri
Step
Food-Forward
Step 2
Cross-Atientie
900
Q KV
Step
[AAKK]
Az
Cached
Aq
Ra
Feed-Forwart Forward
Step {
Self-Artemtion
04
Step
{
13
Aa
96
Aug
Step
Architecture
Teacher-foreing training
Paraliel decoding
The 1tt decoder extract representations of image tokens. The 2nd decoder use target- aware [MASK] tokens as queries that attend to key-value pairs from the output of the 1st decoder. Teacher- forcing training is performed under causal attention. Parallel decoding achieved by inputting multiple queries single step. with each query independently attending to existing KV cache (omit value for clarity).
Figure 4: Architecture:
Two-Pass Decoder Architecture. Based on the preceding discussion, we employ a two-pass de- coder architecture to decouple the prediction of target token Fr, from the representation learning of known tokens {xr, M=1 The equivalence of the probabilistic model is maintained through the application of a suitable causal mask. As illustrated in Fig. and formulated as:
1. Pass-1: Content Representation Learning. The first decoder consists of standard causal self- attention. Unlike typical autoregressive models that predict the next token, its sole purpose is process the sequence of known content token embeddings {ær,, |=1 generate set of rich, context-aware representations ) These representations then projected into key-value pairs, which serve as comprehensive summary of all historical information.
2. Pass-2: Position-Guided Decoding. The second decoder is comprised of causal cross- coossaattention. In this pass, query vector (r, are derived from the [MASK] token, and infused with positional information for specific target location. This target-aware query attends to key-value pairs that are derived from [h,, i=1 produced by Pass-1, ultimately predicting predict C.,.
1--1
Mathematically, let m E 1 be the embedding of [MASK] token, for a given target position 7: and content tokens represented by (hr from Pass-1. Using the rotary position embedding (RoPE) (Su et al., 2024), the attention operations layer of Pass-2 are:
d RoPE(o (1-1) w Te),
VT. o
(4)
k (1) RoPE(h,, w Ti); w! h, W(1)
Z + Attention (1 (k4)
1--11 (m )-1))
(6)
where W,, W., W. E IR dxd are learnable projection matrices.
Training and Decoding. (1) During training, the Pass-1 decoder learns causal representations from shuffled sequence. Then the positional information of the sequence right-shifted by one position and embedded into [MASK] tokens serve as target-aware queries, as illustrated in Fig. 4. (2) At inference time, we first compute the KV cache from the known tokens using the self-attention Pass-1. 1. Next, we select multiple target-aware queries. These queries can simultaneously attend to the KV cache via cross- attss-atttntion in Pass-2, thereby enabling multi-token prediction within a single inference step, as illustrated in Fig. 4. In this setting, the causal attention at step t Pass-1 can be generalized to block-wise attention
that the model learns local bidirectional representations for the tokens generated in step t 1. This mismatch attention patterns does not compromise the causal conditional probability model. Instead, it improves image fidelity (validated in Sec.4) and increases robustness to diverse sampling schedules, including the cosine and arccos schemes used in methods such MaskGiT (Chang al., 2022) and MUSE (Chang et al., 2023).
5

---

Published as a conference paper at ICLR 2026
Decoder
K.V
Pass 2 Decuder
Garies
Gaage
decoding
(0) Controllable
(b)
Inference
Figure 5: Implementation details. (a) Conditional inputs provide the queries. (b) For zero-shot inpainting, known regions are pre-filled in Pass-1, while masked regions are generated Pass-2.
Controllable Generation & Zero-shot Inference. (1) Controllable generation is analogous to class-conditional generation, but uses representations from conditional inputs (e.g., depth maps) as queries instead of positional [MASK] tokens. As illustrated in Fig. 5. (2) Zero-shot inference is achieved via a two-stage process. In the case of inpainting (Fig. 5), the Pass-I decoder is first pre- filled with tokens from the known image regions. Subsequently, the target regions replaced by [MASK] tokens, and the Pass-2 decoder generates them using the key-value pairs from the Pass- 1. This process also upplicable outpainting and resolution expansion.
Comparison with Other Methods. In contrast to recent studies and their notable limitations, ARPG is fundamentally distinct. The key differences detailed below and summarized in Tab. 1.
Different block-wise AR methods introduce unique limitations. VAR (Tian al., 2024) lies on a multi-scale image tokenizer for coarse- to-fine prediction, a strategy that significantly increases the token count and computational overhead. PAR (Wang et al., 2025), SAR (Liu et al., 2024), and NAR (He et al., 2025) predict parallel spatial blocks but are constrained by rigid, predefined ordering or scheduler that im- pairs sample quality and scheduling flexibility.
Table 1: Summary existing methods.
Method
Attention Pattern
Scheduler
Zero-shol
MAR
Bidirectional
Cosine
VAR
Block-wise
Prodetined
PAR
Block-wise
Predefined
NAR
Block-wise
Predefined
X
RAR
Causal
N/A
RandAR
Causal
Flexible
ARPG
Causal
Block-wisc
Flexible
RAR (Yu al., 2024) randomly permutes sequences during training but progressively anneals them to raster-scan order, meaning it does not support random-order parallel generation in practice.
RandAR (Pang et al., 2025) couples the position and content tokens. This method not only doubles the computational cost but also leads to suboptimal fidelity, as the less-informative content repre- sentations in shallow layers are forced to co-learn with [MASK] token. Furthermore, necessitates extra memory to cache [MASK] tokens and retains a causal attention during parallel decoding.
EXPERIMENTS
4.1
IMPLEMENTATION DETAILS
Model. We implement three models of different scales, all of which follow the LlamaGen de- sign (Sun et al., 2024a) and use its tokenizer. All layers in the Pass-2 decoder share a global key- value projection to improve efficiency (Sun et al., 2024b). More details shown in Appendix A.
Training. We train the class-conditional model on ImageNet- K (256 x 256) (Deng et al., 2009) for 400 epochs and the text-to-image model on 4M subset of BLIP-30 (Chen et al., 2025) for 50 epochs. All models are optimized with the Adam (Loshchilov & Hutter, 2019) optimizer using learning rate of le-4 per 256 batch size. Further configuration details are in Appendix A.
We use FID (Heusel et al., 2017) the primary evaluation metric, and additionally port Inception Score (IS) (Salimans et al., 2016), precision, and recall (Kynkäänniemi et al., 2019), following the protocol of ADM (Dhariwal & Nichol, 2021). For text-to-image generation, eval- uate our models using selected metrics from GenEval (Ghosh et al., 2023). The efficiency profile is evaluated on NVIDIA A800-80GB GPU, considering only the token generation process.
Metrics.

---

Published as a conference paper at ICLR 2026
Table 2: Overall comparisons on ImageNet benchmarks. Arrows indicate whether lower or higher is better. Efficiency was profiled with a batch size of 64 and 6 precision.
Throughput (img/s)
Mem. (GB))
Type
Model
Param.
Steps
FID,
1ST
Pre.f
Rec.
Resolution: x56
Diffusion
DiT-L/2 (Peebles Xie, 2023)
458
250
1.32
1.62
5.02
167.2
0.75
0.57
DiT-XL/2 (Peebles & Xie, 2023)
675 M
250
0.91
2.14
2.27
278.2
0.83
0.57
MaskGIT (Chang et al., 2022)
227 M
8
46.18
1.71
6.18
182.1
0.80
0,51
Mask
MAR-B al., 2024b)
208
1.71
1.47
2.31
281.7
0.82
MAR-L (L) al., 2024b)
479 M
100
1.27
2.32
1.78
296.0
0.81
0.60
VAR-d16 (Tian et al., 2024)
310 M
10
123.21
10.85
3,30
274.4
0.84
0.51
VAR
VAR-d20 (Tian et al., 2024)
600 M
75.38
15.97
2.57
302.6
0.83
VAR-224 (Tian2 2024)
1.0
49.94
22.29
2.09
312.9
0.82
0,59
LamaaGen-L. (Sun
2024a)
343 M
576
4.33
10.23
3.07
256.1
0.83
0.52
LlamaGen-XL (Sun et al,, 2024a)
775 M
576
2.46
17.11
2.62
244.1
0.80
0.57
LamaaGen-XX0 (Sun 2024s)
2.62
244.1
AR
RAR-B (Yu 2024)
261
256
14.12
4.65
1.95
290.5
0.82
0.58
RAR-L (Yu et al., 2024)
461 M
256
12.08
6.37
1.70
299.3
0.81
0.60
RAR-XL (Yu et al., 2024)
8.00
10.55
1.50
306.9
0.80
PAR-L al., 2025)
3.76
218.9
PAR-XL (Wang al., 2025)
775 M
147
7.91
17.13
2.61
259.2
0.80
0.62
PAR-XXI. (Wang et al., 2025)
1.4 B
147
5.23
26.25
2.35
263.2
0.80
0.62
AR (Parallel)
NAR-L (Ho al., 2025)
372
31
42.71
10.25
3.06
263.9
0.81
NAR-XL (Ho al., 2025)
816 M
31
23.97
17.13
2.70
277.5
0.81
0.58
NAR-XXL (Ho et ol., 2025)
1.5 B
31
15.23
26.25
2.58
293.5
0.82
0.57
RandAR-L (Pang et al., 2025)
343 M
88
25.30
7.32
2.55
268.8
0.81
RandAR XI. (Pang al., 2025)
775 M
88
15.51
13.52
2.25
317.8
0.80
0.60
RandAR-XXI (Pang et al., 2025)
1.4 B
88
10.46
21.77
2.15
322.0
0.79
0.62
ARPG-L
320 M
32
130.14
2.78
2,30
297.7
0.82
0.56
ARPG-XL
719
80.56
4.65
349.2
0.80
AR (Parallel)
ARPG-XXL
1.3 B
32
55.28
7.22
1.83
336.1
0.80
0.60
ARPG-L
320 M
64
67.47
2.64
2.37
293.7
0.82
0.55
ARPG
4.57
ARPG-XXL
1.3 B
64
31.65
7.18
1.86
339.7
0.81
0.59
Resolution: 512x512
DIT XL/2 (Peebles & Xie, 2023)
675 M
250
0.18
4,70
3.04
240.8
0.84
0.54
MaskGIT (Chang et al., 2022)
227 M
12
4.48
7.63
7.32
156.0
0.78
VQGAN (Esser al. 2021)
1.4 B
1024
0.63
44.12
26.52
66.8
0.73
0.31
VAR-d36 (Tian et al., 2024)
2.0 B
10
OOM
2.63
303.2
ARPG-XL
719
35.53
2.82
277.5
0.82
(a) Class-conditional generation.
(b) Controllable generation.
Figure 6: Generation samples. ARPG can efficiently generate high-fidelity images with 64 steps
4.2 IMAGE GENERATION
Hass-Condittnal Generation. We compare the results of ARPG with existing methods on the ImageNet- IK, as shown in Tab. 2. For 256x256 benchmarks, present two sampling settings: 32 steps for optimal FID and 64 steps for the best visual quality. Samples are shown in Fig. 6a.
Compared to LlamaGen (Sun et al., 2024a), ARPG-XXL achieves better FID of 1.83 with only 32 sampling steps, representing a 30 x speedup with superior fidelity. Compared to VAR (Tian et al., 2024), ARPG achieves higher throughput while reducing 75% memory consumption. Further- more, compared to recent parallel decoding methods (Wang et al., 2025; He al., 2025; Pang et al., 2025), ARPG achieves better generation quality with superior efficiency.

---

Published as a conference paper at ICLR 2026
Figure 7: Text-to-image generation. ARPG-XL can efficiently generate high-quality images with only 64 steps based on the given image captions. Prompt omitted for brevity.
Table 3: Quantitative evaluation of text-to-image generation at 512 resolution.
Model
Params.
Data
Single Obj.
Two Obj.
Colors
Overall
Throughput
LlamaGen-Xi (Sun al., 2024a)
0.8 B
60 M
0.87
0.34
0.64
0.32
0.83 img/s
Chameleon (Team, 2024)
7.0
1.4 B
0.39
SD-v1.5 (Romhach et al., 2022)
0.9 B
2B
0.97
0.38
0.76
0.43
4.32 img/s
ARPG-XL
0.8
M
0.87
0.30
0.68
0.31
30.11 img/s
For 512x512 resolution, fine-tune the XL model (pre-trained at 256x resolution) instead of training from scratch to save computational resources. With only 50 epoch fine-tuning, it achieves competitive quality while surpassing others in throughput, shown in Tab.
Text-to-Imaze Generation. We use the pre-trained FLAN-TS-XL (Chung et al., 2024) generate prompt embeddings that condition the token sequence. As shown in Tab. 3, our method achieves performance comparable LlamaGen (Sun al., 2024a) using only 7% of the training data while offering significantly higher throughput. Generated samples are shown in Fig. 7.
Controllable Generation. Following Con- pre-trained
Table 4: Controllable generation on ImageNet.
trolAR (Li et al., 2024d), al., 2021) adapter to pro- cess conditional inputs, such Canny edges and depth maps. These conditional inputs are used target-aware queries fine-tune the pre-trained class-conditional model. As shown in Tab. 4 and Fig. 6b, ARPG significantly out- performs recent works such as ControlVAR (Li al., 2024c) and ControlAR (Li al., 2024d).
VIT (Dosovitskiy
FID
Method
Model
Param
Canny
Depth
VAR-d16
310M
16,20
13.80
Contro\VRR
VAR-d30
2.08
7.85
6,50
ControlAR
AiM-I.
350M
9.66
7.39
LlamaGen-L
343M
7.69
4.19
ControARRGG
ARPG-1
320M
7.39
4.06
Zero-shot Generalization. We assess the zero-shot generalization capabilities of ARPG on a variety of image manipulation tasks, with qualitative results presented in Fig. It demon- strates that ARPG excels at inpainting (class- conditional editing) and outpainting without any task-specific fine-tuning, generating con- tent that both high-fidelity and semantically consistent with the surrounding context. This strong performance highlights a key advantage of our flexible, random-order generation. More zero-shot inference samples of different tasks provided in the Appendix D.
(a) Left: image inpainting.
Right: image editing.
(b) Outpainting: from 256x x024x256
Figure 8: Zero-shot inference of different tasks.
8

---

Published as a conference paper at ICLR 2026
Table 5: Ablation study of model design. The baseline model is ARPG-L, trained for 150 epochs with 64 sampling steps. "Rand. & Parall." denotes support for randomized parallel generation.
Description
Parameters
Layers
Rand. Parall.
Steps
Throughput"
Memory
FID
ARPG-L
320 M
12+12
64
67.47 imgls
2.64 GB
3.51
282.7
w. CosinePE
320 M
12+12
64
70.63 imgls
2.64 GB
3.61
262.4
w/o Shared KV
343 M
12+12
64
48, img/s
3,83 GB
3,46
228.1
Fower Pass-2 Decoder
332 M
18+ 6
64
62.35 imgls
3,34 GB
3.82
223.0
More Pass Decoder
307 M
6+18
64
71.24 img/s
1,93 GB
3.51
242.5
Fully Pass-2 Decoder
295 M
0+24
64
72.26 img/s
0.91 GB
4.57
255.9
w/o. Pass-2 Decoder
343 M
24+ 0
X
256
11.70 imgis
4.96 GB
>90
<.50
1.2 0
200
APPSX
2.6
MPPG
(RFG (auoii
ARP5X00
APPG
3.0
150
Throughpat
2.6
200
2.4
50-
2.0
2.2
16
128
236 16
32
64
122
256
16
24
32
64
128
256
Step
Step
Stop (c)
(a)
(b)
Figure 9: Ablation study of parallel decoding. The impact of decoding steps speed (a) and
quality (b). Generalized attention patterns enhance quality with fewer decoding steps (c).
4.3 ABLATION STUDY
We report several key ablation studies. Additional experiments are presented in the Appendix B.
Effeet of Decoding Steps. We evaluated how the number of decoding steps affects both generation quality and efficiency. As shown in Fig. 9, reducing the number of sampling steps substantially improves inference efficiency while not significantly degrading generation quality. In some cases, quantitative quality metrics fewer steps
outperform those obtained with more steps.
Effect of Attention Pattern. We additionally evaluated setting in which the attention pattern is prevented from generalizing from causal to block-wise attention at inference. The results in Fig. show the opposite trend the permissive setting, namely that fewer steps produce markedly worse quality. These findings support our claim in Sec. that generalizing causal attention to block-wing wise attention does not invalidate the underlying probabilistic model. On the contrary, the resulting local bidirectional awareness helps the model better predict future tokens.
Effect of Architecture Design. We explored the impact of architecture design, shown in Tab. 5.
1. Decoder. The higher the proportion of the Pass-2 decoders, the higher the inference efficiency, but the more severe the deterioration of the generation quality. Reducing the proportion of the Pass-2 decoder not only reduces inference efficiency but also degrades the generation quality. When there are no Pass-2 decoders at all, the model degenerates into a standard AR model and loses the ability of randomized parallel decoding.
KV Projection. We also examined the impact of using shared KV, discussed in the Sec. 3. Without shared KV, although the generation quality of the model is slightly improved, it signif- icantly affects the inference speed and memory consumption. To balance the generation quality and inference efficiency, we choose to use the shared KV design in subsequent experiments.
Position Encoding. We further examine the effect of different positional encodings. Replacing RoPE with cosine positional encodings leads to degradation in performance.
CONCLUSION
In this work, propose novel autoregressive image generation framework that can parallelly generate images in random token orders. breaking the limitations of the inefficiency of the next token prediction paradigm and its poor zero-shot generalization ability.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work is supported by the Young Scientists Fund of the National Natural Science Foundation of China (NSFC) (No. 62506305), the Zhejiang Leading Innovative and Entrepreneur Team Introduc- tion Program (No. 2024R01007), the Key Research and Development Program of Zhejiang Province (No. 2025C01026), the Scientific Research Project of Westlake University (No. WU2025WF003), and the Chinese Association for Artificial Intelligence (CAAT) & Ant Group Research Fund AGI Track (No. 2025CAAI-ANT-13), is also supported by the research funds of the National Talent Program and the Hangzhou Municipal Talent Program.
Additionally, this work is partially supported by the CAS Project for Young Scientists in Basic Re- search (YSBR-116), the National Natural Science Foundation of China (NSFC) (No. 62325603, No. 62236009, No. U22A20103), the Beijing Science and Technology Plan (No. 2241100004224011, and Shanghai NeuHelium Neuromorphic Technology Co., Ltd.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303. 08774, 2023.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In CVPR, 2022.
Huiwen Chang. Han Zhang, Jarred Barber, AJ Maschinot, José Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image gen- eration via masked generative transformers. In ICML, 2023.
Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, et al. Blip3-o: A family of fully open unified multimodal archtts-achittcetuee, training and dataset. arXiv preprint arXsv:2505.09568 2025.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instraction-finetunee lan- guage models. JMLR, 25(70):1-533 2024.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. In NeurIPS, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Jacob Devlin, Ming- Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In NeurlPS, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
DeepSeek-Al et Deepseek Ilm: Scaling open-source language models with longtermism, 2024.
Sicheng Feng, Keda Tao, and Huan Wang. Is oracle pruning the true oracle? arXiv preprint arXiv:2412.00113,, 2024.
Ziteng Gao and Mike Zheng Shou. D-ar: Diffusion via autoregressive models. arXiv 2505.23660, 2025.
10

---

Published as a conference paper at ICLR 2026
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An objuct-focused framework for evaluating text-to-image alignment. In NewrlPS, 2023.
Yefei He, Yuanyu He, Shaoxuan He, Feng Chen, Hong Zhou, Kaipeng Zhang, and Bohan Zhuang. Neighboring autoregressive modeling for efficient visual generation. arXiv preprint arXiv:2503.10696 2025.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14700 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In NeurlPS, 2017.
Jonathan Ho and Tim Salimans. arXiv:2207.122 12..29987.11988,.. 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Tuomas Kynkäääniemi. Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In NeurIPS, 2019.
Doyup Lee, Chiheon Kim, Sachoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In CVPR, 2022.
Haopeng Li, Jinyue Yang, Kexin Wang, Xuerui Qiu, Yuhong Chou, Xin Li, and Guoqi Li. Scalable autoregressive image generation with mamba. arXiv preprint arXiv:2408. 12245, 2024a.
Muyang Li*, Yujun Lin*, Zhekai Zhang Tianle Cai, Xiuyu Li, Junxian Guo, Enze Xie, Chenlin Meng, Jun-Yan Yan Zhu, and Song Han. Svdquant: Absorbing outliers by low-rank components for 4-bit diffusion models. In ICLR, 2025.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. In NeurlPS, 2024b.
Xiang Li, Kai Qiu, Hao Chen, Jason Kuen, Zhe Lin, Rita Singh, and Bhiksha Raj. Controlvar: Exploring controllable visual autoregressive modeling. arXiv preprint arXhv:2406.09750 2024c.
Zongming Li, Tianheng Cheng, Shoufa Chen, Peize Sun, Haocheng Shen, Longjin Ran, Xiaoxin Chen. Wenyu Liu, and Xinggang Wang. Controlar: Controllable image generation with autore- gressive models. arXiv preprint drXiv:4410.02705, 2024d.
Jiacheng Liu, Chang Zou, Yuanhuiyi Lyu, Junjie Chen, and Linfeng Zhang. From reusing to fore- casting: Accelcrating diffusion models with taylorseers. arXiv preprint arXiv:2503.06922, 2025.
Wenze Liu, Le Zhuo, Yi Xin, Sheng Xia, Peng Gao, and Xiangyu Yue. Customize your visual autoregressive recipe with set autoregressive modeling. arXiv preprint 101iv:2410.10511, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deepcache: Accelerating diffusion models for free. In CVPR, 2024.
Ziqi Pang, Tianyuan Zhang, Fujun Luan, Yunze Man, Hao Tan, Kai Zhang, William T Freeman, and Yu-Xiong Wang. Randar: Decoder-only autoregressive visual generation in random orders. In CVPR, 2025.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.

---

Published as a conference paper at ICLR 2026
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NeurlPS, 2016.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525 06525, 2024a.
Yutao Sun, Dong, Yi Zhu, Shaohan Huang, Wenhui Wang. Shuming Ma. Quanlu Zhang, Jianyong Wang, and Furu Wei. You only cache once: Decoder-decoder architectures for language models. arXiv preprint arXiv:2405.0555 245505554, 2024b.
Keda Tao, Haoxuan You, Yang Sui, Can Qin, and Huan Wang. Plug-and-play 1. x-bit kv cache quantization for video large language models. arXiv preprint 1rXiv:2503.16257 2025.
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint
arXiv:2405.09811 2024.
Yao Teng, Han Shi, Xian Liu, Xuefei Ning, Guohao Dai, Yu Wang, Zhenguo Li, and Xihui Liu, Ac- celerating suto-regressive text-to-image generation with training- free speculative jacobi decoding. In ICLR, 2025.
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng. and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. NeurlPS, 2024.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. In NeurIPS, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, Yingli Zhao, Yulong Ao, Xuebin Min, Tao Li, Boya Wu, Bo Zhao, Bowen Zhang, Liangdong Wang, Guang Liu, Zheqi He, Xi Yang, Jingjing Liu, Yonghua Lin, Tiejun Huang, and Zhongyuan Wang. Emu3: Next-token prediction is all you need. arXiv preprint arXhv:2409.18800 2024.
Yuqing Wang, Shuhuai Ren, Zhijie Lin, Yujin Han, Haoyuan Guo, Zhenheng Yang, Difan Zou, Jiashi Feng, and Xihui Liu. Parallelized autoregressive visual generation. In CVPR, 2025.
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang. Kevin Qinghong Lin,
Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. /2528, 2024.
Jinheng Xie, Zhenheng Yang, and Mike Zheng Shou. Show-o2: Improved native unified multimodal models. arXiv preprint arXiv:2506. 15564, 2025.
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoe V. Lc. Xlnet: Generalized autoregressive pretraining for language understanding. In NeurlPS, 2019.
Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chieh Chen. Randomized autoregres- sive visual generation. arXiv preprint arXiv:2411.0007 2024.
Runpeng Yu, Xinyin Ma, and Xinchao Wang. Dimple: Discrete diffusion multimodal large language model with parallel decoding, arXiv preprint arXiv: 16990, 2025.
Junhan Zhu, Hesong Wang. Mingluo Su, Zefang Wang, and Huan Wang. Obs-diff: Accurate pruning for diffusion models in one-shot. arXiv preprint 1rXiv:2510.0675) 2025.
12

---

Published as a conference paper at ICLR 2026
A IMPLEMENTATION DETAILS
Table 6: ARPG model architecture and hyperparameters configuration. X+Y layers: X layers in the first-pass decoder and layers in the second-pas. decoder.
Table ARPG-L 256
Table 2 ARPG-XL 256
Table 2 ARPG-XXL
Table 2 ARPG-XL
Table 4 ARPG-L
256 256
x
Architecture
Layer
12.+
18
24 24
18 1280
12.
Hidden Size
1024
1280
1536
1024
Heads
20
20
16
Parameters
320 M
719 M
1.3 B
719 M
320 M
Optimization
Truining Heration
500 K
670 K
I 512
500 K
K
Batch Size Optimizer
1024
768
128
1024
AdamW
Adam
AdamW
Adam
AdamW
4e-4
3e.4
5e-5
4e
Ir scheduler
Cosine
Cosine
Cosine
Linear
Constant
warmup ratio
0,25
0.25
1.0
0.0
weight decay
0,05
0.05
0.05
0.05
0,05
(()))9))
(0.9.0.95)
(0.9.0.95)
(0.9.0.95)
(0.9 (0.9.0.95)
(0.9.0.95)
Data Augmentation
Ten-cmp
Ten-crop
Ten-crop
Ten-crop
Ten-cmp
Sampling
Steps Scheduler
Arccos Linear
Arccos Lineur
Arceos Lineur
Arcens Lineur
Arecos Linear
CFG Scheduler
Temperature
1.0 None
1.0
1.0
1.0
1.0
Top-K
None 1.0
None
None
None
Top-P
1.0
1.0
Architecture Configuration. We show the model architecture configuration in three different scales in Tab. 6. The notation X+Y layers represents the number of layers in the first-pass and second-pass decoders, respectively. For the parameter counts of the models in the controllable gen- eration, we only report the parameters of the base model. The parameters of the additionally intro- duced ViT-Adapter (Li et al., 2024d), used for processing conditional information, are excluded to maintain consistency with the representation in ControlAR (Li et al., 2024d).
Training Hyperparameters. We detail the hyperparameters for the experimental setups to ensure the precise reproducibility of findings. The AdamW (Kingma, 2014; Loshchilov & Hutter, 2019) optimizer, with a consistent configuration, was utilized in all experiments. The batch size and learning rate were tailored for each model based on computational resources.
Sampling Hyperparameters. For the sampling process, all models employ arccos schedule to determine the number of tokens to decode at each step. A linear scheduler utilized for the classifier-free guidance (CFG) (Ho & Salimans, 2022) scale. Optimal performance is achieved by setting both the temperature and top-p to 1.0, while top-k sampling is not applied (denoted as None in the Tab. 6, which is equivalent setting top-k to the vocabulary size of 16,384).
B ADDITIONAL EXPERIMENTS
Effects of Sampling Order. We also evaluated the performance of ARPG under specific orders, in-
Table 7: Effect of generation orders.
cluding raster order and several alternatives (Esser et al., 2021), as shown in Tab. 7. While random- order modeling is more challenging due to the n! possible orderings, it still outperforms fixed orders. The constraint of a fixed order impedes effective parallel decoding and zero-shot tasks.
Steps
FID.
ISt
Pre.
Rec.1
Ruster
256
2.49
277.6
0.79
0.58
Spiral-in
256
3.71
221.1
0.75
0.57
Spiral-out
256
4.11
210.5
0.74
0.56
Z-curve
256
2.56
278.2
0.78
Alternate
256
2.56
279.4
0.78
0.54
Random
287.1
0.81
Effects of Sampling Steps. In Tab. 8 we present more detailed set of experiments that quantify
the effects of decoding steps and attention pattern generalization on model performance. For each carried out fine- -ee-graindd search the classifier-free guidance (CFG) scale to ensure that the reported FID values are optimal. FID is used as the primary quantita- tive metric and the Inception Score (IS) is treated as secondary metric. Accordingly, the IS values
decoding step configuration,
13

---

Published as a conference paper at ICLR 2026
Table 8: Ablation study of parallel decoding, Wofgi the optimal classifier-free guidance scale.
FID
ISt
Throughput (img/s) t
Memory
Model
Steps (Wegg)
causal
block-wise
causal
block-wise
(GB) 4
16
(7.8)
3.51
2.67
351.09
302.50
223.18 18
3.00
24
(6.1)
2.96
2.37
320.73
293.72
164.80
2.87
(5.4)
2.69
2.30
305.71
287.90
130.14
2.78
ARPG-L
(5.1)
2.37
302.63
297.72
67.47
2.64
128
(4.6)
2.45
2.44
286.82
289.36
33.44
2.53
256
(4.3)
2,43
2,44
277.59
278.55
17.45
2.32
(8.4)
2.68
2.38
364.46
331.83
126.66
4.78
24
(7.0)
2.33
2.01
353.89
339.06
99.74
4.71
32
(6.6)
2.24
1.93
352.62
342.43
80.56
4.65
ARPG-XI.
(5.3)
327.57
323,92
45.09
4.57
128
(5.1)
2.02
325.86
322.69
22.43
4.49
256
(5.0)
2.03
2.01
318.77
318.94
11.60
4,41
(9.3)
2.49
2.38
353.07
326.11
79.97
7.25
24
(8.1)
2.18
1.89
350.20
341.00
64.70
7.20
32 (6.5)
2.04
1.83
337.26
328.89
55.28
7.22
ARPG-XXL
64
(6.3)
1.94
1.86
337.76
334.57
31.65
7.18
128
(6.1)
1.93
336.41
336.34
17.20
7.16
256
(6.0)
1.90
1.88
334.96
337.34
8.41
7.23
shown in Tab. 8 correspond to the operating points that minimize FID. Finally, although causal and block-wise causal attention (in Fig. 2) patterns exhibit minor differences in inference throughput and memory footprint, we report only the block-wise resource measurements in Tab. 8.
Our results demonstrate that generalizing from causal attention to block-wisc causal attention during inference does not compromise the probabilistic model, but instead significantly improves perfor- mance. This finding validates claim in Sec. that the local bidirectional context is beneficial. This marks a key distinction from previous works (Pang et al., 2025; Wang et al., 2025; He et al., 2025), which are restricted to the same attention pattern for both training and inference.
Analysis Scalability. To analyze the scal- ability of models, present their train- ing loss curves on class-conditional generation tasks in Fig. 10. A clear performance hierarchy emerges based on model scale.
8.0
ARPG-XX XXL
ARPG-XL
ARPG-L
Loss
The largest model (1.3B parameters) achieves the lowest final loss of approximately 7.03 after 1M iterations. The model with 719M parame- ters converges to a loss of 7.29, while the small- est model (320M parameters) finishes with the highest loss at 7.50.
7.5
7.0
200K
400K
GOOK
BOOK
1000K
Iteration
This scaling trend strong indicator of the robustness of our model and suggests potential for further improvement with even larger scale.
Figure 10: Training loss curves. All models were trained for the same epochs.
Analysis of Attention Scores. As we mentioned in Sec. 2, decoupling the representation learning of unmasked tokens from the prediction [MASK] tokens addresses the computational redun- dancy issue. This problem arises in coupled architectures where [MASK] tokens receive negligible attention. We visualize the final-layer attention maps ARPG's two decoders in Fig. 11. The visu- alization reveals that this decoupled structure leads more uniformly distributed attention scores in both decoders. The attention patterns primarily exhibit a natural decay over long distances, avoiding the issue seen in Fig. 3 where significant number of tokens assigned extremely low weights.
14

---

Published as a conference paper at ICLR 2026
18
Mact
Mark
Nock
16
96
12
Uriaack
Unmask
Unmask
(a) Attention map of cross-attention decoder.
2.0
3.0
Unenaze
Unrraak
Unimaak
Unimaok
3A
(b) Attention map of self- olf-attention decoder.
Figure 11: Attention maps of different decoders. Visualization of normalized attention maps from different decoders, Each column corresponds to the same attention head.
C THEORETICAL PROOF
As mentioned in Sec. 3, we claimed that the masked modeling is inefficient due to the non non-directi -in-ireetlly gradient flow caused by sparse optimization objects. Here is the theoretical proof:
Proof. the final layer the Transformer (Vaswani et al., 2017) blocks, given input sequences with length and q, k, Rixd, The softmax attention (Vaswani et al., 2017) mechanism (the scale term omitted for clarity) operates as:
Ej ex-ixxp(ckk))o Σj k))(q.k))
IR Ixd
Let S.y = q:k3. Ps softmax(S.)). is evident that dP, = d and the derivative of the softmax function is its Jacobian matrix. Using the fact that the Jacobian of y = softmax(x) is diag(y) (Dao et al., 2022), have:
dS. (PPa P.) -P. P.)
P. ② dP: (do:0 T Pi:
(9)
dS P.j(PP.j - do.o,))
(10)
where ( denotes the Hadamard product. Then we derive gradient for q using chain rule:
dq: Σ dS
Pojdo. do,o, )k,
(11)
1=1
Obviously, when o, corresponds to unmasked token, it does not contribute to the loss calculation, resulting in do; 0, and consequently, dq. O.
The query vectors for unmasked tokens in shallower layers updated via indirect gradients propa- gated through key-value pairs from deeper layers, which provide a partial learning signal not directly tied to the optimization objective.
15

---

Published as a conference paper at ICLR 2026
forest giant, qlowing masheoons,
beautiful butserfly with fairy walking clockwork vings.
girl glowing atazs petal of a giant ziowez, are captive in glass
giant mechanicsi bootle in polished obsidian
& nan who resembles Zecko takes the ot
gray-tone image teaturing magni ticent
A pertrait or boautitul ancient lgyptian chi goddees.
girl driftina dwwign of fantaay wales, sursoarded by travelor.
fantasy Bluating islanda and waterfalls.
A looking out the window, painted reslistic styla.
nimbus of blorde hair.
Figure 12: Text-to-image generation. Samples generated by ARPG-XL with 64 sampling steps. ARPG achieve high-quality and semantically well-aligned images based given prompt through training on a small amount of data.
D GENERATION SAMPLES
To further showcase the versatility of ARPG, examples of text-to-image generation, class- conditional image generation, controllable image generation, and zero-shot inference are provided.
Text-to-Image Generation. Additional visual results for text-to-image generation, including the prompts used, are presented in Fig. 12. All images were sampled with the following parameters: 64 steps, CFG scale of 8.0, top-k 900, with temperature and top-p both at 1.0. These examples illustrate that ARPG is capable of producing high-quality images in various styles with training dataset. of only 4M samples. our future work, we will further investigate the effectiveness of ARPG with larger number of parameters and an increased volume of training data.
Class-Conditional Generation. We present uncurated examples of class-conditional image gen- eration in Fig. 13, using the same sampling parameters set in Tab. 6. ARPG excels in both quan- titative metrics and visual quality. Additionally, examples of controllable generation and zero-shot inference based on the class-conditional model also presented in Fig. 13.
16

---

Published as a conference paper at ICLR 2026
Figure 13: Uncurated Samples in different generation tasks produced by ARPG.
17

---

Published as a conference paper at ICLR 2026
E PSEUDO-CODE
We provide PyTorch-style pseudo code for the training and parallel decoding of our model. The classifier-free guidance is omitted for clarity.
def forward dcc_oddddrdddce (model, input_ids, freqs_cis)
x model embed (input_ids)
model [edel.decoder_11 (x, freqs_cis)
k, model.kv_nee (x)
koobelkky_rorf (x) chunk (2, dim=-1)
k rearrange (k, "b t (h d) -> b h d", h=model. inme nun )
k apply_rotary (k, (reqs_cis) wtt.ssstsssssseeeeeee....seessse (1, 2)
if model. caching: k, v model. wpdattpdlacceeaeeeeeeeeee (k, v)
return k,
def forward _nd decodddeceerreeee (model, k, v, freqs_cis, batch_size num cum__)err))
q model.ember (torch. full (batch_size, num_query), BASK_TOKEN_DD) q model.decoder 2 (g, k, v, freqs_cis)
return model. head hede (model. norom)))
def forwarddmodee input_ids, condition):
8, innut_npsssss.. shape
shuffle input and RoPB using the same order.
shuffled ids, orders = batch seq shuffle (input ids)
freqs cis model. freqs_ wwt.eci............ uns.uuuuueeeee( repeat (3, 1)
fixed. freqs_ cis treqs_cis(:: :1, ...] shuff_freqs_cis = sttp__ecesshufleeeeeeeeeeeeee freqs_cis = torch. cat ([fixcd_ shuff_froqs_cis) dim=1)
1;, ...l, orders)
prepare teacher-forcing input
x Corch.cat(leooditte (Icondition, shuffled_ide dim=-1) k, v gerward_lst_decoder (madel, x(:, :-1], freqs_cis(;; :-1, ....)
logits Sorward__2ndddecode model, k, v, freqs_cisl:; 1;, ...,, pateh_ num query=))
return cross_ent (logits, shuffled_ids.c () ddetach ())
def generate (model, condition, seg_len-256, num
num samples condition.shape fregs_cct cis_ model. cis. unsqueeze (0)
orders torch. raed .rggort (dim=0) +
last_pos, last_range 0, sequences [condition] for step in range (num_iter))
num_pred = sample_schedule (step, num_iter)
next_range orders (range (last_pos, last_pos num_pred) last_pos += num_pred
the classifier-free guidance is omitted for clarity
k, - forward_ist_decodd
model, sequences[-11, freqs_cis_li: last_range, ...])
logits
k, v, freqs_cis_ls next_range, ...), num_samples, num_preed
tckens sample_token (lcgits)
sequences. append wawpencces.appcndttooccee last_range next_range
sequences torch. cat (sequences dim--1)
return sequences I orders. arders.argsert (dim=0)]
18

---

Published as a conference paper at ICLR 2026
F DISCUSSION
In future work, we plan to extend ARPG to large-scale text-to-image synthesis and unified models for understanding and generation (Xie al., 2024; 2025). Additionally, we will explore architectures and methods that combine the strengths of AR and diffusion models (Yu et al., 2025; Chen et al., 2025; Gao & Shou, 2025), incorporating techniques such feature caching Ma al. (2024); Liu et al. (2025), quantization Tao et al. (2025); Li et al. (2025) and pruning Zhu et al. (2025); Feng al. (2024) to further enhance efficiency.
G THE USE OF LARGE LANGUAGE MODELS (LLMS)
Our use of LLMs was strictly limited to proofreading for grammatical and spelling errors. LLMs made no contribution to the ideation process or the writing of the content.
19


Reviews:
[{"review_id": "2oRe6KVeaR", "reviewer": "Reviewer_SipR", "summary": "This work introduces ARPG (Autoregressive Randomized Parallel Generation), a method that incorporates a decoder-decoder architecture design and decouples positional guidance from content representation.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation is clear, where the input sequence redundancy is a bottleneck for RandAR.\n\n2. The experiments conducted on class-to-image generation, controllable generation, zero-shot generalization, and text-to-image generation verify the effectiveness of ARPG.", "weaknesses": "1. At the same level of model size, e.g. 320M in Table 2, ARPG outperforms RandAR significantly in both efficiency (throughput and memory consumption) and output quality. What is the underlying reason for such improvement? Does most of the improvement come from YOCO architecture design? \n\n2. RandAR is trained with a batch size of 1024 for 300 epochs, but ARPG is trained for 400 epochs with different batch sizes. I am wondering whether ARPG will keep the same advantage under the same training setting.\n\n3. As a minor concern, I am curious about how ARPG with fewer sampling steps (32 steps) outperforms ARPG with more steps (64 steps)？For 256x256 image reconstruction, LlamaGen tokenizer achieves an rFID of 2.19 at 256×256 resolution. How can the gFID of ARPG surpass this upper bound?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nbrIjXgbRL", "reviewer": "Reviewer_ePaj", "summary": "This paper proposes ARPG, an efficient architecture for Random AR Image generation. First, the paper found that  (i) Rand AR require explicit positional information for the token to be predicted, and (ii) using position information tokens at the input token level hinders training efficiency and fails to achieve meaningful attention. Based on this, this paper proposes an efficient architecture for RandAR, which is built with a Pass-1 decoder trained only with visual tokens, and a Pass-2 decoder trained with cross-attention with position tokens. Specifically, the Pass-1 decoder outputs the KV cache of the current image token sequences, and the Pass-2 decoder uses this KV cache with position token queries for decoding. Experimental results showed that ARPG outperforms existing AR, PAR, and RandAR frameworks in terms of speed and quality across several benchmarks.", "questions": "- What are the advantages of AR modeling of image tokens over multi-modal discrete diffusion models like [1]? They also support the parallel generation of visual tokens. While I am aware that AR has advantages in KV-cache efficiency, I am just curious about the author's perspective.\n\n- In Fig. 9 (b) and (c), what specific reason makes fewer generation steps actually improve the generation quality? Doesn't this result indicate that the current ARPG training is unstable?\n\n[1] Yu, Runpeng, Xinyin Ma, and Xinchao Wang. \"Dimple: Discrete diffusion multimodal large language model with parallel decoding.\" arXiv preprint arXiv:2505.16990 (2025).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to understand. The introduction, observations, and methodology are intuitive and clear.\n- The paper presents interesting insights and observations regarding the characteristics of RandAR. To the best of my knowledge, these findings are novel.\n- The proposed method is intuitively derived from the observations and simple.\n- Experimental results show that ARPG outperforms existing AR models across multiple benchmarks.", "weaknesses": "- **Integrability with LLMs** : A key advantage of (raster) image AR or RandAR, which rely solely on self-attention decoder architecture, is that they can be easily integrated with existing LLM architectures without any changes. However, ARPG's reliance on cross-attention requires LLM architecture modification or, at least, fine-tuning by attaching Pass-2 decoder in parallel to the existing models.\n\n- **T2I evaluation** : Experimental results on T2I evaluation are limited. In Tab.3, the model was only trained on a relatively small number of samples (4M) and actually shows worse performance. It is difficult to evaluate whether this architecture can be extended to large-scale T2I.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NDWFUBtHPE", "reviewer": "Reviewer_5zjF", "summary": "This paper proposes ARPG, a novel autoregressive image generation framework that enables random-order, parallelizable token generation via a two-pass decoupled decoding mechanism: (1) a content refinement pass builds a causal, label-leakage-free memory from randomly ordered tokens; (2) a position-guided prediction pass uses [MASK] queries to decode multiple tokens in parallel. This design allows training under full AR causality while supporting flexible inference orders and block-parallel decoding. ARPG achieves good performance on ImageNet with significantly improved throughput and reduced memory, and demonstrates compelling zero-shot generalization and controllable generation.", "questions": "I don't think this article has any fatal flaws. I only have a few minor questions to raise.\n1. The resolution in the T2I generation does not seem to be mentioned in the paper.\n2. I have some questions about the throughput in Table 3. Firstly, the throughput of the c2i model of LlamaGen-XL in Table 2 is 2.46, but it is 4.32 in Table 3. Why is the T2I model faster instead? I think this is abnormal. Moreover, the throughput of SD1.5 is too low. In fact, the inference efficiency of SD1.5 is higher than that of LlamaGen.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1.Decoupling content modeling from position prediction is a conceptual advance that breaks the raster-scan bottleneck in AR vision models.\n2.This paper has sufficient experimental verification and has verified the effectiveness of the method on various generation tasks such as c2i, t2i, and controllable generation.\n3. This paper makes a detailed comparison of the existing AR-based models, which is helpful for understanding the uniqueness of the method proposed in this paper.", "weaknesses": "1. This paper contains ambiguous statements about efficiency improvement. For example, in the last sentence of the abstract section, the author claims: \"achieving over a 30× speedup ... compared to representative recent autoregressive models ... \". In fact, the 30× speedup here is relative to the naive autoregressive model LlamaGen from a year ago. Compared to recent autoregressive models such as RandAR, the real advantage of ARPG in throughput is ~3× speedup. I think the author should have a more rigorous expression in the paper, especially in the explanation and comparison of quantitative indicators.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e4hTbX1EeG", "reviewer": "Reviewer_XPoH", "summary": "The paper introduces ARPG, a model designed to improve the efficiency of autoregressive image generation. It separates the generation process into two stages: Pass-1, which generates content representations using causal self-attention, and Pass-2, which decodes these representations using causal cross-attention. This separation allows for parallelization and significant improvements in inference speed and memory efficiency—up to 30× faster and 75% less memory. ARPG also maintains competitive image quality, with minimal loss of fidelity. The model excels in both speed and quality, making it suitable for real-time image generation tasks.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation and observations in the paper are well-founded, highlighting the need for more efficient autoregressive image generation models.\n2. The proposed method effectively decouples the generation process into two stages (Pass-1 and Pass-2) to improve efficiency without compromising quality.\n3. The performance of ARPG is impressive, offering significant speedup and memory reduction while maintaining competitive image quality.", "weaknesses": "1. The description of Figure 3 is unclear, as it doesn't explicitly mention that it shows multiple attention heads, which may lead readers to think they are the same. \n\n2. In Table 2, the citation for **NAR** appears to be incorrect. It should be updated to reference the correct source, as the current citation does not align with the intended reference or context.\n\n3. Due to ARPG separating Pass-1 and Pass-2 into two distinct stages, each handling different tasks, the model's interpretability is limited, making it difficult to pinpoint performance issues during debugging.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)