Locality-aware Parallel Decoding for Efficient Autoregressive Image Generation
Authors: Zhuoyang Zhang
Luke J. Huang
Chengyue Wu
Shang Yang
Kelly Peng
Yao Lu
Song Han
Venue: ICLR 2026 Oral (2026)
Keywords: Efficient Autoregressive Image Generation
Parallel Decoding
Primary area: generative models

Abstract: We present Locality-aware Parallel Decoding (LPD) to accelerate autoregressive image generation. Traditional autoregressive image generation relies on next-patch prediction, a memory-bound process that leads to high latency. Existing works have tried to parallelize next-patch prediction by shifting to multi-patch prediction to accelerate the process, but only achieved limited parallelization. To achieve high parallelization while maintaining generation quality, we introduce two key techniques: (1) Flexible Parallelized Autoregressive Modeling, a novel architecture that enables arbitrary generation ordering and degrees of parallelization. It uses learnable position query tokens to guide generation at target positions while ensuring mutual visibility among concurrently generated tokens for consistent parallel decoding. (2) Locality-aware Generation Ordering, a novel schedule that forms groups to minimize intra-group dependencies and maximize contextual support, enhancing generation quality. With these designs, we reduce the generation steps from 256 to 20 (256×256 res.) and 1024 to 48 (512×512 res.) without compromising quality on the ImageNet class-conditional generation, and achieving at least 3.4× lower latency than previous parallelized autoregressive models.

Published as a conference paper at ICLR 2026
LOCALITY-AWARE PARALLEL DECODING FOR EFFICIENT AUTOREGRESSIVE IMAGE GENERATION
Zhuoyang Zhang" Luke J. Huang"
Chengyue Wu
Shang Yang Kelly Peng
Yao Lu Song Han
MIT NVIDIA First Intelligence
https://githube hwm.thh...ccoo...coommmm www.eommtitttttttteeeeee--aaaaet lab/lpd
ABSTRACT
We present Locality-wware Parallel Decoding (LPD) to accelerate autoregressive image generation. Traditional autoregressive image generation relies on next- patch prediction, memory- process that leads to high latency. Existing works have tried to parallelize next-patch prediction by shifting to multi-pane -atch prediction accelerate the process, but only achieved limited parallelization. To achieve high parallelization while maintaining generation quality, we introduce key techniques: (1) Flexible Parallelized Autoregressive Modeling, novel architecture that enables arbitrary generation ordering and degrees of parallelization. It learnable position query tokens guide generation target positions while ensuring mutual visibility among concurrently generated tokens for consistent parallel decoding. (2) Locality-aware Generation Ordering, novel schedule that forms groups to minimize intra-group dependencies and maximize contextual support, enhancing generation quality. With these designs, we reduce the generation steps from 256 20 (256 x256 res.) and 1024 to 48 (512x512 res.) without compromising quality on the ImageNet class-conditional generation, and achieving at least x lower latency than previous parallelized autoregressive models.
INTRODUCTION
Autoregressive modeling has achieved state-of-the-art results in large language models in terms of scalability and generalizability (Brown et al., 2020; OpenAl, 2023; Touvron et al., 2023a;b; Grattafiori et al., 2024; Jiang al., 2024; Yang et al., 2024; 2025; Liu et al., 2024a).
Naturally, many works have applied this power- ful paradigm visual generation (Esser al., 2021; Lee et al., 2022; Ramesh et al., 2021; Yu al., 2022; Sun et al., 2024; Tian al., 2024). Moreover, this autoregressive formulation of vi- sual generation has become increasingly cru- cial for unified multimodal generation (OpenAl, 2025; Wang et al., 2024a; Wu et al., 2024c;a; Chen et al., 2025a; Ma et al., 2025; Jiao et al., 2025; Song et al., 2025; Chen et al., 2025b; Zhao et al., 2025; Lin et al., 2025; Deng et al., 2025; Liao et al., 2025; Xie et al., 2025) since it is highly compatible with language modeling,
Frechet Incepption Ditence (FI))
3.5
703M
1.4B
NAR
2.5
2- fastter
steps)
RandAR
LPO (32 staps)
1.8
0.1
10
Latenoyls, batch bice-11
Prevailing autoregressive visual generation methods typically follow two paradigms: (1)
Figure 1: Performance comparison among par- allelized autoregressive models on ImageNet 256 x 256. We significantly reduce the generation steps and achieve at least 3.4x lower latency com- pared with previous models.
next-patch prediction by flattening the image into sequence of patches (Esser al., 2021)
and (2) next-scale prediction via coarse-to-fine multi-scale representations (Tian et al., 2024).
'Equal Contribution.

---

Published as a conference paper at ICLR 2026
Head
Head 2
Head 17
Head
Hedd
Locality
Hend
Head 21
Locality
Figure 2: Visualization of attention maps in the LLAMAGEN-I.AE model. There is strong spatial locality, as the attention encodes images into 24 x 24 tokens, where token that is 24 positions earlier in the attention map corresponds to the token directly above it in the 2D) grid.
decoding token concentrated on nearby spatial tokens. LLAMAGEN
In the first formulation, generating one token per step creates a memory-bound workload', causing latency to scale with the number of steps. The second formulation substantially reduces generation steps and thus latency. However, its multi-scale token representation fundamentally differs from the universal flat token representation, making it incompatible with widely used flat vision perception foundation models (e.g., CLIP (Radford et al., 2021; Zhai et al., 2023), DINO (Caron et al., 2021; Oquab et al., 2023)) and thereby limiting interoperability with perception backbones that have been proven critical for unified multimodal systems (Wu et al., 2024c; Ma et al., 2025; Jiao et al., 2025; Song et al., 2025; Chen al., 2025b; Zhao al., 2025; Lin al., 2025; Tong et al., 2024; Wu et al., 2025; 2024b).
Thus, autoregressive visual generation should be (1) highly efficient: minimizing latency and maxi- mizing throughput; (2) remain flat token representations for universality and compatibility with vision backbones and, by extension, unified multimodal models. Recent works (Wang et al., 2024b; Pang al., 2024; Li et al., 2025a) have tried to parallelize next-patch prediction by shifting to multi matli-patchh prediction to accelerate the process, but only achieved limited parallelization. Non-autoregressive mask-prediction models like MASKGIT (Chang et al., 2022) enable multi-pater prediction but require full attention for bidirectional context, making them less efficient than autoregressive methods.
To address the challenges, introduce Locality-aware Parallel Decoding (LPD), framework that consists of a novel flexible parallelized autoregressive modeling architecture and a novel locality- aware generation order schedule. We design a new modeling architecture as conventional decoder-only autoregressive models struggle with flexible generation order and parallelization, limiting efficiency. In contrast, ours enables arbitrary generation order and degrees of parallelization. This is achieved by using learnable position query tokens to guide the model in generating tokens at target positions. Moreover, the generation is parallel-aware, as we leverage specialized attention mechanism to ensure mutual visibility among tokens generated concurrently. Notably, our design also inherits the K caching mechanism, avoiding redundant computation.
Furthermore, we observe strong spatial locality in image generation attention where tokens predomi- nantly attend to nearby regions as shown Figure 2. This indicates a high dependency among nearby tokens, meaning that spatially closer tokens provide stronger conditioning. Recent works (Wang et al., 2024b; Besnier et al., 2025) also identify that minimizing mutual dependency among simultaneously generated tokens is essential to maintain sample consistency. With these insights, we introduce locality-aware generation order schedule that selects parallel decoding groups to maximize contextual support while minimizing intra-group dependencies, enabling higher degrees of parallelization.
We examine the effectiveness of our proposed method on ImageNet class-conditional image genera- tion. Our results reveal that we reduce the generation steps of traditional raster-order autoregressive generation from 256 to 20 (256 x 256 res.) and 1024 to 48 51225112 res.) without compromising quality, and achieving at least 3.4 lower latency (Figure 1) than previous parallelized autoregressive models. Thanks to the design of flexible autoregressive modeling, our models are also capable of zero-shot image editing including class-conditional editing, inpainting and outpainting.
A memory- bound workload refers to the scenario where the efficiency is limited by memory access speed rather than computation speed. In this context, each generation step requires loading the entire model parameters into GPU registers, making the pracess hottlenecked by memory handwidth rather than computational power.
2

---

Published as a conference paper at ICLR 2026
4
5
5
Autoregressive Transformer
Autoregressive Transformer
Image Token
3
Stepe
Slare
No Prediction
(a) Raster Order Autoregressive Modeling
(b) Flexible Parallelized Autoregressive Modeling
Figure 3: Raster Order Flexible Parallelized Autoregressive Modeling. (a) In raster order, each token simultaneously provides context and predicts the next token, restricting flexibility and efficiency. (b) Our approach decouples these roles: previously generated tokens supply context, while position query tokens drive parallel generation at arbitrary target positions. This separation enables both flexible order and efficient parallelization.
2
METHOD
2.1
RETHINKING AUTOREGRESSIV MODELING
In next-patch autoregressive modeling, images are split into patches and usually discretized via tokenizer into image tokens. While the joint distribution of the N tokens Z1,- NN and condition extremely high dimensional and therefore hard to model directly, the autoregressive framework makes this amenable by factorizing the total joint distribution as
N
p(r1, ... C0.,,,, NN:C) II p(en|(enic))
The training objective of the autoregressive model therefore to optimize parametric approximations p for those one-step conditionals. This factorization needs predefined order, typically raster order, as shown in Figure (a). However, during sampling, this leads to N sequential steps, creating major efficiency bottleneck.
To reduce the number of sequential generation steps, we can partition tokens into G disjoint groups (X1,... .x), where each group Xg {Ig..' Ff is predicted jointly, resulting in the following:
p(11, 2....., NNiC) = II p(X, Xegic)
(2)
g=1
The training objective hecomes optimizing Po (Xg X c). Previous work has shown that directly
grouping tokens in raster order causes significant performance degradation (Wang al., 2024b; Pang et al., 2024). This is because spatially adjacent tokens exhibit strong mutual dependencies, and independent sampling usually leads generation inconsistencies inside group. is essential to break the raster order when grouping. In addition, the size of the prediction group XX, should gradually increase. As the context size Xx grows, offers stronger conditioning, allowing more tokens to be predicted in parallel. Previous work using masked transformers (Chang et al., 2022) also mirrors this intuition by predicting fewer tokens early when context sparse and predicting more tokens over time. Therefore, effective parallelized autoregressive model should support: (1) Flexible generation order alleviate the issue caused by mutual interdependency of concurrently predicted tokens and (2) Dynamic group sizes increasing the number of tokens predicted per step with available context.
However, it is diffieult to achieve these within the standard decoder-only autoregressive models, which are inherently designed with a fixed onput-output structure, e.g. next-token prediction. this modeling, each token simultaneously serves two roles: it provides context via its hidden state and enables generation via its output logits. This coupling limits flexibility in the the generation order and output size. To address these challenges, propose novel flexible parallelized autoregressive modeling which is able to support arbitrary generation order and degrees of parallclization.
2.2 FLEXIBLE PARALLELIZED AUTOREGRESSIVE MODELING
Our core idea is to decouple the context representation and token generation by leveraging separate tokens. We illustrate this Figure (b). In this formulation, previously generated tokens are encoded

---

Published as a conference paper at ICLR 2026
Key
Figure 4: Illustration of the training attention mask. Context Attention allows subsequent tokens to attend to the context tokens causally. Query Attention ensures mutual visibility among the po- sition query tokens within the same step, and pre- vents any subsequent tokens from attending the query tokens. For example, image token can be attended to by all subsequent tokens, including im- age tokens and position query tokens, to provide context information. The two position query to- kens P3 and Ps in the same generation step attend to the condition, the image token 4, and each other, while ignoring the earlier query P4.
Context Attantion
Query
Attertion
Cache
Key
Figure 5: Illustration of the inference attention mask. Encoding with image tokens and Decoding with position query tokens can be fused into single step. Taking step 2 in Figure 3 (b) the example, simultaneously encodes the previously generated image tokens 3, update the K V-ccche and decodes the desired image tokens 1, 2 and in parallel.
Quiry
9
to provide context and the generation driven by learnable position query tokens corresponding to the desired target positions. These position query tokens constructed by adding the positional embedding of the target location to shared learnable embedding. By directly inputting these position- specific queries, the model can generate tokens at arbitrary target positions in parallel. This design allows the model to leverage positional information in both the context and generation pathways, enabling arbitrary generation order.
Training formulation. We train the model to transform each position query token into the cor- responding ground-truth image token, conditioned on all ground-truth tokens that precede it. To preserve teacher-forcing while allowing parallel prediction, we interleave position query tokens with ground-truth tokens and apply specialized training attention mask as shown in Figure 4 that contains two attention patterns:
1. Context Attention allows subsequent tokens to attend to context tokens causally. Query Attention ensures mutual visibility among the position query tokens within the same step, and prevents any subsequent tokens from attending the query tokens.
Inference formulation. At test time we alternate between encoding the generated image tokens and decoding with position query tokens.
1. Encoding. Sampled image tokens go through a forward pass to store the KV cache, providing context for future decoding steps.
Decoding, Learnable position query tokens attend to all previously generated tokens in the KV cache, and the forward pass outputs logits for each target position in parallel. KV cache for query tokens is not stored.
However, sequentially execute these two operations double the generation steps. As shown in Figure (b), these two operations can be fused into a single step via a specialized inference attention mask as shown in Figure 5.
Comparison with other methods. Recent efforts have also pursued parallel generation in autore- gressive modeling, yet each carries inherent limitations. One line of work, exemplified by SAR (Liu et al., 2024b) and ARPG (Li et al., 2025a), adopts an eecoder-decoder architecture where target-aware query tokens attend to the encoder's 's key-value cache via cross- aotss-ttennion. However, as illustrated in Figure 6 (a), the target positions themselves do not contribute any key-value pairs, resulting in the tokens generated within the same parallel step being produced independently of one another.
4

---

Published as a conference paper at ICLR 2026
Encoder Cache
Cache (Include Positional Instruction Token)
Cache
3
Pr
O2
Ps
Po
(a) SAR, APRG
RandAR
LPD
Figure 6: Comparison with other methods. (a) Encoder-decodee approaches such as SAR and ARPG generate tokens independently, since query tokens contribute no key-value pairs. (b) Decoder- only methods like RANDAR rely on positional instruction tokens, but the causal mask reduces parallel generation batched next-token prediction and forces instruction tokens be cached, doubling memory. (c) In contrast, our method employs specialized training mask that ensures mutual visibility among concurrently predicted tokens while caching only the generated tokens.
Another approach, represented by RANDAR (Pang et al., 2024), adheres to the prevailing decoder- only architecture. achieves arbitrary order by inserting positional instruction tokens to designate target positions. However, it still leverages a standard causal mask during training. This strategy, as depicted in Figure 6 (b), leads to two notable issues: (1) the parallel generation degenerates into a batched next-token prediction instead of joint prediction and (2) the positional instruction tokens must be stored in the KV cache during inference, doubling the memory consumption. Compared with these two methods, method as shown in Figure (c) guarantees the visibility among all concurrently predicted target positions and only stores the generated tokens the KV cache.
PAR (Wang al., 2024b), NAR (He al., 2025), and ZipAR (He al., 2024) preserve the standard decoder-only architecture and increase the number of tokens generated per step. Although they guarantee mutual visibility among concurrently generated tokens, they rely fixed parallel generation order, which prevents them from supporting arbitrary generation orders. This limits the generation flexibility thus achieved limited parallelization and generation quality, ACDIT (Hu et al., 2024) shares similar attention scheme with us, yet it was used for evenly interpolating between autoregressive and diffusion modeling.
2.3 LOCALITY-AWARD GENERATION ORDER SCHEDULE
To fully leverage our flexible parallelized autoregressive modeling architecture, we introduce locality-aware generation order schedule. This schedule is guided by two key principles (1) High proximity previously generated tokens: target positions should be spatially close to existing context to ensure strong conditioning and (2) Low proximity among concurrently generated tokens: tokens predicted the same parallel step should be spatially distant to reduce mutual dependency.
These principles are derived from systematic analysis of the attention patterns in autoregressive image generation by the widely adopted LLLAMAGEN (Sun et al., 2024) model. Using LLAMAGEN, generate 50,000 images and collect attention scores each decoding step. Qualitative attention patterns are shown in Figure 2, and quantitative results are presented in Figure 7. To quantify locality, we define the Per-Token Attention (PTA) to neighborhood of radius as:
N ΣS Attention(T. T3) I[d((,, PTA,
(3)
N
Σ, [d(T,
s]
where Attention(T;, TT) denotes the attention weight from token T; to token Tj. and d(T;, Tj) is their Euclidean distance on the 2D image grid.
As shown in Figure 7 (a). PTA decreases sharply with increasing distance, indicating a strong spatial locality in the attention mechanism. This suggests that nearby tokens carry significantly more useful information during decoding, and that spatially adjacent tokens are highly dependent on one another for accurate prediction. This locality pattern consistently observed across all attention heads. In Figure 7 (b), we visualize the Attention Sum, defined as the total attention score decoding token assigns to tokens within a relative distance 8. The plot uses s = 3 and confirms that most attention is concentrated within local neighborhoods, reinforcing the importance of spatial locality.
"The neighhorhood defined the of tokens whose centers are exactly euclidean distance of away.
5

---

Published as a conference paper at ICLR 2026
This analysis supports two principles: de- coding tokens should remain close to previously generated tokens to maximize contextual sup- port, and distant from concurrently generated tokens to minimize intra-group dependency.
75
LiamaGensL
LlamaGen-L Liamatsen XL
Liamatsen XL
LlamaGen-X0
Attenton Sum (96)
ElemaGen-X0C
To-Towen Attention
8
165
Based on these principles, we implement a locality-aware generation order schedule de- scribed in Algorithm I. Suppose we use K' de- coding steps generate 22 tokens, with group sizes O = [01 .2..... ox]. where is the number of tokens generated step k. typically increasing via cosine schedule. At each step k, we compute the euclidean distance between unselected and already selected tokens to mea-
36
Relative Distance
Head ID
(6)
(b)
Figure 7: Attention Analysis of LLAMAGEN.
(a) Attention diminishes with distance (b) Spatial locality is consistently observed in all heads.
sure spatial proximity, where closer distance leads to higher proximity, sort unselected tokens by proximity and split them into two sets: are tokens with sufficient proximity larger than the threshold which are eligible for the following high-proximity selection, and C2 are the rest. We sequentially select tokens from Cı, adding each to the selected set while filtering that the relative distance is smaller than the repulsion threshold p. which are added to c2. all the grids in Cj are considered and the number of selected grids is less than Ok, we use farthest point sampling (Qi et al., 2017) 7) to select the remaining grids from C2 to ensure spatial low dependency. It is worth noting that the generation order can be precomputed and stored for direct use during inference, incurring no additional latency. We provide the PyTorch implementation in Appendix B.I. We further clarify the key distinction between our method and prior work Appendix B.2.
nearby tokens
Algorithm 1: Locality-awar Generation Order Schedule
Input: decoding steps K, group sizes O = [o1, .z..... ox), grids G = {(ij)) proximity threshold T. repulsion threshold p:
schedule S =
for i 1.... K do
8 []: p = 1/ euclidean S.S)
proximity measurement
sorted(G1) S, key = p, reverse True);
C1. C2 = cutoff((c, r):
while len(s) оş len()) do
s = queue quuuue_euuhhh queue peceue_pop(c__111 1))
high-proximity selection
C1 f filter(c1, p): C2 = queuu__pssh(c._1)
len 8) оş then
8 = queue_push(s, farthest_poinn prrthest_pint_ssmplnnggg sampling(e,, 8, len (+)))
low-dependency selection
S queue queue_push(SS.)) push(S,
return S
For intuitive understanding, we illustrate an example of our generation order schedule in Figure 8. We also plot the schedule for raster order, random order and Halton order (Besnier al., 2025) for comparison. The raster order generates tokens in a raster-scan manner and the random order generates tokens random manner. The Halton order is discrecrypancy sequence to arrange the generation positions which spreads out the tokens to achieve uniform image coverage step by step.
3 EXPERIMENT
3.1 SETUP
Models. For fair comparisons with existing autoregressive image generation methods, the LLAMAGEN tokenizer (Sun et al., 2024) with codebook size 16384 and downsample factor 16.
We train three models of different sizes: 337M, 752M, and 1.4B parameters. We use standard

---

Published as a conference paper at ICLR 2026
A) Rodler
By Rendom
63 Halto
15
Steps
StepO
Step112
Figure 8: Illustration different generation order schedules. All schedules leverage 20 decoding steps for 162 tokens. Dark green marks newly selected grids and light green marks those already selected. Compared to others, our schedule selects grids close to previous ones and far from concurrent ones, maximizing the contextual support and minimizing the mutual dependency,
decoder-only transformer architecture, and refer to them as LPD-L, LPD-XL, and LPD-XXL, respectively. Please refer to the Appendix A.1 for more details.
Training and Evaluation. We train and evaluate our models the class-conditional ImageNet (Rus- sakovsky et al., 2015) 256 and ImageNet 512x512 datasets. We first train all models on ImageNet 256 for 450 epochs, with epochs of learning rate warmup followed by constant learning rate and finally 50 epochs of cosine decay. For 512-rsollution models, we load the pre-trained 256-resolution models and interpolate the positional embeddings and continue training on ImageNet 512x512 for another 50 epochs. During training, the image tokens randomly shuffled while the class token is kept at the beginning. We train on range of predefined decoding steps where the tokens per step follows cosine schedule. We reportuse Fréchet Inception Distance (FID) (Heusel et al., 2017) as the primary metric computed on 50k,000 generated samples as the primary metric as well asnd also report Inception Score (IS) (Salimans al., 2016), Precision, and Recall (Kynkäänniemi et al., 2019). Please refer to the Appendix A.2 for more details.
Efficiency Profiling. We profile all the efficiency results BFloat16 precision. We measure the latency with batch size of I and throughput with a batch size of 64. We report the average latency over 500 inference steps, with a 100-step warm-up period.
single NVIDIA A100 GPU with
3.2 MAIN RESULTS
We compare models against broad set of generative baselines ImageNet 256x 256 (Table 1). For a fair comparison, we also create a raster order counterpart following the same setup. As shown in the table, we reduce the generation steps from 256 20, achieving 12.8x generation steps reduction, without sacrificing the generation quality. Compared with other parallelized autoregressive models, achieve significantly better image generation quality and efficiency. Taking LPD-XL model as an example, it achieves FID of 2010 with only 20 steps, reducing the number of generation steps by 3.2x compared to ARPG and achieving 4.2x lower latency. Increasing the steps slightly to yields FID of 1.92, even matching ARPG-XXL. while reducing latency by 3.4x. We further report our results on ImageNet 512x512 (Table 2). As shown the table, we reduce the generation steps from 1024 to 48, achieving 21.3x generation steps reduction, without sacrificing the generation quality. These results validate the effectiveness of our flexible parallelized autoregressive modeling and the locality- generation order schedule. We also provide qualitative generation and zero-shot editing results in Figure 12.
Our method is general and can be readily extended to higher- reshurrrssolution text-to-in tinttto-mmmee generation (e.g., 1024x 1024). We provide a detailed description this extension in the Appendix A.3. We evaluate the widely used GenEval (Ghosh et al., 2023) benchmark and report results in Table 3.

---

Published as a conference paper at ICLR 2026
Table 1: System-level comparison on ImageNet 256 class-conditioaal generation. We
evaluate the generation quality by metrics including Fréchet inception distance (FID), inception score (IS), precision and recall. #Steps is the number of model runs needed generate an image. We measure latency with batch size of and throughput with a batch size of 64 on single NVIDIA A100 GPU under 6 precision, with classifier-free guidance (CFG) for both.
Type
Model
#Para.
FID
1ST
Precisiont
Recallf
#Steps
Lstency(s)
ThroughputlingsSf
ADM-G [17]
554M
4.59
186.7
0.82
0.52
250
CDM [29]
4.88
158.7
8100
Diffusion
LDM-4 [56]
400M
3.60
247.7
250
DFT-XL/2 [50]
675M
2.27
278.2
0.83
0.57
250
4.34
0.58
XLT- [44]
675M
2.06
270.3
0.82
0.59
250
MaskGIT [8]
227M
6.18
182.1
0.80
0.51
8
MAGVIT-v2
307M
1.78
319.4
64
Mask
MAR-B [37]
MaskBit [72]
208M
2.31
281.7
0.82
0.57
64
18.14
1.03
2.93
MAR L [37]
479M
1.78
2966.0
0.81
0.60
64
20.80
2.11
303.7
25.96
274.4
10
VAR
VAR-420 [64]
600M
2.57
302.6 6
0.83
0.56
10
0.15
52.53
YAR-424 [64]
L,B
2.09
312.9
0.82
0,59
10
0,17
39.30
[64]
VQGAN-re
280.3
256
RQTran.-re [34]
3,8B
3.80
323.7
256
LlamaGen-L. [61]
345M
3.07
256.
0.83
0.52
576
12.22
2,08
LlamaGien-X
AR
LlamaGen-38 3B [6]]
LlamaGen XXLL [XL,
3.1B
1.4B
2.18
2.34
253.9
263.3
0.80
0.81
0.59
0.58
576
576
24.40
12.37
0,58
0.72
RAR-B [84]
261M
RAR-L [84]
461M
1.70
299.5
0.81
0.60
256
4.04
12.63
RAR-XI [84]
955M
1.50
306.9
0.80
0.62
256
5.47
8,76
RAR-XXL [84]
326.0
256
6.72
PAR-L-4x [71]
343M
3.76
218.9
0.84
0.50
147
3.16
6.83
PAR-XL-4x
775M
2.61
259.2
0.82
0.56
147
4.79
3.69
PAR XRR-XLL-44X [71]
1.4B
2.35
2633.2
0.82
0.57
147
6.26
2.33
PAR-3B-4x
3.18
2.29
255.5
0.82
0.58
147
3.29
2.32
RandAR-L [48]
343M
2.55
0.58
1,97
28.59
RandAR- XL. [48]
775M
2.25
317.8
0.80
0.60
88
2.78
17.06
Paralletized
RandAR-XXX
1.4B
322.0
0.79
0.62
88
3.58
11.49
AR
0.58
ARPG-L [36]
320M
2.44
287.
0.82
0.55
64
1.15
$4.70 70.70
ARPG-XL [36]
719M
2.10
351.0
0.79
0.61
64
1.71
36.53
ARPG XXL [36]
2.24
NAR-I. [27]
372M
3.06
263.9
0.81
0.53
1.01
41.03
NAR-XL [27]
816M
2.70
277.5
0.81
0.58
31
1.42
23.36
Counterpart-L
337M
278.0
256
3.73
17.53
AR
Raster Counterpart XL.
752M
2.12
307.4
0.81
0.60
256
5.29
12.31
Raster Counterpart XXI.
1.4B
2.01
316.0
0.80
0.59
256
7.10
8.99
LPD-L
337M
2.40
284.5
0.81
0.57
20
0,28
139.11
XL
Parallelized
IPD.XXX
1.4B
2.00
337.6
0.80
0.60
20
0.55
45.07
AR
LPD-L LPD-XL
337M
282.7
0.81
0.58
32
0.46
110.34
752M
1.92
319.4
0.79
0.61
32
0.66
61.24
As demonstrated in the table, LPD reduces the sampling steps for 1024x 1024 image generation from 4096 to just 64 and simultaneously improves the GenEval score. This provides compelling evidence that LPD is generalizable method capable of supporting high-resolution text-to-image generation. We also include qualitative generation results in Figure in the appendix.
3.3 EFFICIENCY ANALYSIS
Our method introduces position query tokens to enable flexible generation. These tokens add extra queries and thereby increase FLOPs. However, the resulting computational overhead has negligible impact on wall-clock latency in memory-boume settings such as small-batch inference. In these scenarios, the reduction in generation steps translates almost linearly into latency reduction. As the batch size increases, the system progressively shifts toward a compute-bound regime, where the additional overhead begins matter and diminish the speedup. We provide quantitative analysis in Figure 14 to illustrate this trend. By gradually increasing the batch size until reaching the memory limit, we observe that the model transitions from memory- to compute- bound when the batch size exceeds 16. Nevertheless, even at the maximum feasible batch size, our method retains a throughput advantage of approximately 3x over the raster-order baseline.
8

---

Published as a conference paper at ICLR 2026
Table System-level comparison on ImageNet 512x512 class-conditional generation. Metrics
and evaluation setup are the same as in Table 1.
Type
Model
@Para.
FID
1S1
Precisiont
Recallt
#Steps
Latency(s))
Throughuut(mgg/1
ADM-G [17]
554M
7.72
172.71
0.87
0.42
250
Diffusion
DIT-XL/2 [50]
240.82
0.84
0.54
11.32
0.10
SIT XL/2 [44)
675M
2.62
252.21
0.84
0.57
250
MaskGIT 18]
227M
7.32
156.0
0.78
0.50
12
Mask
MAGVIT-v2 [83]
307M
1.91
324.3
64
MAR-L [37]
481M
1.73
279.9
VAR
VAR-d36-s [64]
2:3B
2.63
303.2
10
0.45
O0M
AR
VQGAN [18]
227M
26.52
66.8
0.73
0.31
1024
Parallelized AR ARPG-XI. [36]
719M
3.38
257.8
Raster Counterpart- I.
337M
2.54
278.5
0.80
0.58
1024
14.25
3.79
Raster Counterpar-XL
752M
2.09
315.0
0.81
0.57
1024
20.93
2.36
Parallelized AR
LPD-L
337M
2.54
292.2
0.81
0.55
48
0.69
35.16
LPD.XI.
752M
2.10
326.0
0.80
0.63
48
1,01
18.18
Table 3: System-level comparison on GenEval 1024x 1024 text-to-image generation. Detailed
per-category GenEval scores are provided in Table 7.
Type
Model
#Para.
GenEval Scoret
#Steps
Lateney(s))
Throughput(mngs)t
Raster Counterpart-,
344M
0.55
4096
64.7
0.20
Raster Counterpen-XX XL
760M
0.60
4096
93.8
0.14
Parallelized AR
LPD-XL
LPD-L
344M
760M
0.62
0.58
64
64
1.53
1.01
2.85
2.4
2,2
RandAR-XL
Random Halton
Random Princagle 2
Princiato LPD
2.8
2:11
2:1
2.8
2.9
2.4
1.82
2.2
1.8
16
20
24
64
128
250
16
20
24 Generatton Stops
32
64
Generation Stapa
LFD-XL 32 Steps
0 Figure Ablation Studies. All ablation experiments are conducted with XL size models on 256x256 resolution. (a) Effectiveness of flexible parallelized autoregressive modeling, (b) Effectiveness of locality-aware generation order schedule. (c) Effectiveness of the locality principles.
(b)
ABLATION
Effectiveness of Flexible Parallelized Autoregressive Modeling. One key design of our flexible parallelized autoregressive modeling is the guarantee of the mutual visibility among all concurrently generated tokens. This is critical to maintain the consistency in the same group when the degree of the parallelization is high. We show the effectiveness of this design in Figure 9 (a). We compare our model with RANDAR and ARPG which lack this design. To only ablate the effectiveness of our flexible parallelized autoregressive modeling, we use random generation order for all models without our locality-aware parallel generation order schedule. As shown in the figure, with the generation steps decrease and the parallelization increases, our model exhibits a smaller FID increase compared with the other two models. For example, with 32 steps, our model almost maintain the performance with 256 steps but ARPG and RANDAR have a significant FID increase. This design is crucial for us to achieve fewer generation steps while maintaining the generation performance.
Effectiveness of Locality-aware Generation Order Schedule. We compare our schedule with another two generation order schedules as shown in Figure 9 (b). Random order just arrange the generation positions randomly. Halton order leverages the Halton low-discrepancy sequence to arrange the generation positions which spreads out the tokens to achieve uniform image coverage step by step. Intuitively it mainly focus reducing the dependency inside parallel group which shares the same insight with our second principle that low proximity is needed among concurrently generated tokens. However, the low-discrepancy sequence omits the importance of the already

---

Published as a conference paper at ICLR 2026
generated context which is our first principle that need to maintain high proximity to previously generated tokens. As shown in the figure, our locality- aware parallel decoding order consistently outperforms the other two orders, showing the effectiveness of our method.
Effectivvness of the Locality Principles. As introduced in Section 2.3, our locality- -aware generation order schedule is guided by two principles. We ablate the effectiveness of these two principles in Figure 9 (c). As shown, the random order baseline yields an FID of 2.11. We first apply Principle I only, selecting points close previously generated tokens without considering their mutual dependency. This improves the performance to 2.00. We then apply Principle 2 alone, using farthest point sampling each step to ensure concurrently generated tokens are well separated, without considering context from previously generated tokens. This improves the FID to 2.06. Combining both in our locality-aware generation order achieves 1.92, highlighting the synergy of both principles. We further provide a sensitivity analysis of the hyperparameters 7 and in Appendix B.3.
5 RELATED WORKS
5.1 AUTOREGRESSIVE IMAGE GENERATION
Autoregressive models generate the current output conditioned only on previous outputs. Usually this dependency is captured by causal attention mechanisms, enabling efficient inference via KV caching. Autoregressive modeling with GPT-style "next-token-predictinn (Brown et al., 2020; OpenA1, 2023; Touvron et al., 2023a;b; Chiang et al., 2023; Jiang et al., 2024) has dominated the field of language generation. Inspired by this success, autoregressive visual generation has shifted from operating on sequences of pixels (Van Den Oord et al., 2016; Van den Oord et al., 2016; Parmar et al., 2018; Chen al.. 2018: Salimans
al., 2017; Yu et al., 2021; Li et al., 2025b) to sequences of latent discrete tokens (Esser et al., 2021; Lee et al., 2022; Ramesh et al., 2021; Razavi et al., 2019; Yu et al., 2021; 2022; Sun et al., 2024; Yu et al., 2024; Wang et al., 2024a; Teng et al., 2024; Ren et al., 2025; He et al., 2025; 2024). However, the token-by-toker btokeen decoding strategy is often bottlenecked by memory bandwidth. This limitation prevents full utilization of computation and results high latency. Recently, "next-scale- seettssale-rrediction" (Tian et al., 2024; Han et al., 2024) has emerged predict the next scale of the image instead of the next token thus accelerates the generation process. However, its multi-scale token representation fundamentally differs from the universal flat token representation, making it incompatible with widely used flat vision perception foundation models.
5.2 PARALLEL GENERATION IN SEQUENCE MODELING
Parallel generation has been widely studied in the field of language modeling. Prior to the era of large language models, masked prediction architectures (Gu et al., 2017; Ghazvininejad et al., 2019; masked-prediction
Gu et al., 2019) were commonly used to do parallel generation and iterative refinement. Recently, with the rapid success of large language models, speculative decoding (Chen et al., 2023; Leviathan et al., 2023) and its derivatives (Cai et al., 2024; Ankner et al., 2024) employ a draft model to generate the next few tokens and then the main model conducts the verification. In visual generation, masked
models (Chang et al., 2022; Yu et al., 2023a;b; Chang et al., 2023) are widely used generate masked tokens step by step leveraging a masked prediction transformer similar to BERT (Devlin et al., 2019; Bao et al., 2021; He et al., 2022), which are able to generate multiple tokens in parallel. However, they are non-autoregressive models and need bidirectional attention which is computationally expensive and KV cache is not applicable to accelerate the inference. Recent works (Wang et al., 2024b; Pang et al., 2024; Li et al., 2025a; He et al., 2025) have explored parallel generation in autoregressive models, but with limited parallelization and generation quality. Our proposed method enables greater parallelization without sacrificing performance.
CONCLUSION
Our contributions lie in two key aspects: (1) flexible parallelized autoregressive modeling and (2) locality-aware generation order schedule. We significantly reduce the generation steps required by the traditional autoregressive models without compromising the generation quality and achieve at least 3.4x lower latency than previous parallelized autoregressive models.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard, Jonathan Ragan- Kelley, and William Brandon. Hydra: Sequentially-deeendmm draft heads for medusa decoding. arXiv preprint arXi::2422.055 109, 2024.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL https: / arxiv.ivv.. org/ abs//ab///// 13922..1119....
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2/06.082: 2021.
Victor Besnier, Mickael Chen, David Hurych, Eduardo Valle, and Matthieu Cord. Halton scheduler for masked generative image transformer. arXiv preprint arXiv:2503. 17076, 2025.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason Lee, Deming Chen, and Tri Dao. Medusa: Simple I1m inference acceleration framework with multiple decoding heads. arXiv preprint wwxiv:2401.10777 2024.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. Proceedings of the IEEE/CVF international conference computer vision, pp. 9650-9660, 2021.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11315 11325, 2022.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint 1xXiv:2301100000. 2023.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318. 2023.
Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. Pixelsnail: An improved autore- gressive generative model. In International conference on machine learning, pp. 864-872. PMLR, 2018.
Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling, arXiv preprint ar0iv:2501. 17811, 2025a.
Zisheng Chen, Chunwei Wang. Xiuwei Chen, Hang Xu, Jianhua Han, and Xiaodan Liang. Semhitok: unified image tokenizer via semantic-guided hierarchical codebook for multimodal understanding and generation. arXiv preprint urXiv:2503.06764 2025b.
Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%. chatgpt quality. See hutp::/vvicua. Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie. Ziang Song, et al. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505. 14683, 2025.
11

---

Published as a conference paper at ICLR 2026
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873-12883, 2021.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. arXiv preprint 10X.vv:90.009322 2019..
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36: 52132-52152, 2023.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama herd of models. arXiv preprint arXiv:2407.21783, 2024.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. Non-autoregressive neural machine translation. arXiv preprini 17X1v:1711.0228//, 2017.
Jiatao Gu, Changhan Wang, and Junbo Zhao. Levenshtein transformer. Advances in neural informa-
tion processing systems, 32, 2019.
Jian Han, Jinlai Liu, Yi Jiang, Bin Yan, Yuqi Zhang, Zehuan Yuan, Bingyue Peng, and Xiaobing Liu. Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis. arXiv preprint arXiv:2412.0 222414 2024.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Yefei He, Feng Chen, Yuanyu He, Shaoxuan He, Hong Zhou, Kaipeng Zhang, and Bohan Zhuang. Zipar: Accelerating autoregressive image generation through spatial locality. arXiv preprint arXiv:2412.04002, 2024.
Yefei He, Yuanyu He, Shaoxuan He, Feng Chen, Hong Zhou Kaipeng Zhang, and Bohan Zhuang. Neighboring autoregressive modeling for efficient visual generation. arXiv preprint arXiv:2503. 10696, 2025.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge a local nash equílibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1-33 2022.
Jinyi Hu, Shengding Hu, Yuxuan Song, Yufei Huang, Mingxuan Wang, Hao Zhou, Zhiyuan Liu, Wei-Ying Ma, and Maosong Sun. Acdit: Interpolating autoregressive conditional modeling and diffusion transformer. arXiv preprint arXiv:2412.077200 2024.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint 1xXii:2401.040080 2024.
Yang Jiao, Haibo Qiu, Zequn Jie, Shaoxiang Chen, Jingjing Chen, Lin Ma, and Yu- Yu-Gang Jiang. Unitoken: Harmonizing multimodal understanding and generation through unified visual encoding. arXiv preprint 2r54::2504.04423, 2025.
12

---

Published as a conference paper at ICLR 2026
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems, 32, 2019.
Doyup Lee, Chiheon Kim, Sachoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 11523-11532, 2022.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning pp. 19274-19286. PMLR, 2023.
Haopeng Li, Jinyue Yang. Guoqi Li, and Huan Wang. Autoregressive image generation with randomized parallel decoding. arXiv preprini arXiv:2503. 105668 2025a.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. Advances in Neural Information Processing Systems, 37: 36424-56445 2024.
Tianhong Li, Qinyi Sun, Lijie Fan, and Kaiming He. Fractal generative models. arXiv preprint arXiv:2502.17447. 2025b.
Chao Liao, Liyang Liu, Xun Wang. Zhengxiong Luo, Xinyu Zhang, Wenliang Zhao, Jie Wu, Liang Li, Zhi Tian, and Weilin Huang. Mogao: An omni foundation model for interleaved multi-modal generation. arXiv preprint arXiv:2505.088 55555555.7, 2025.
Haokun Lin, Teng Wang, Yixiao Ge, Yuying Ge, Zhichao Lu, Ying Wei, Qingfu Zhang, Zhenan Sun, and Ying Shan. Toklip: Marry visual tokens to clip for multimodal comprehension and generation. arXiv preprint ar wwxh::205500525 2025.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepscek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024a.
Wenze Liu, Le Zhuo, Yi Xin, Sheng Xia, Peng Gao, and Xiangyu Yue. Customize your visual autoregressive recipe with set autoregressive modeling. arXiv preprint arXiv:2410. 105/1, 2024b.
Chuofan Ma, Yi Jiang, Junfeng Wu, Jihan Yang, Xin Yu, Zehuan Yuan, Bingyue Peng. and Xiao- juan Qi. Unitok: A unified tokenizer for visual generation and understanding. arXiv preprint arXiv:2502.20331, 2025.
Nanye Ma. Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnde and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In European Conference on Computer Vision, pp. 23 40. Springer, 2024.
OpenAI. Chatgpt. https: http://ppeaaa/
htmm//boog/chattptttee 2023.
OpenAl. Introducing 4o image generation, Mar 2025. URL. https: / oww.appeeaaooooo com/ index/ Accessed 2025-06-20.
introducing-Ao- ieto-image image- generation/. -tte:----mmm---geeaaiiiinoo////
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa. Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2344.0719. 2023.
Ziqi Pang, Tianyuan Zhang, Fujun Luan, Yunze Man, Hao Tan, Kai Zhang, William T Freeman, and Yu-Xiong Wang. Randar: Decoder-only autoregressive visual generation in random orders. arXiv preprint arXiv:24/2.01827, 2024.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International conference on machine learning, pp. 4055-4064.
PMLR, 2018
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEFE/CVF international conference on computer vision, pp. 4195 4195-4205, 2023.
13

---

Published as a conference paper at ICLR 2026
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. Advances in neural information processing systems, 30, 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PmLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot tex--to-image generation. In International conference on machine learning, pp. 8821-8831. Pmlr, 2021.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
Sucheng Ren, Qihang Yu, Ju He, Xiaohui Shen, Alan Yuille, and Liang-Chieh Chen. Beyond next- token: Next-x prediction for autoregressive visual generation. arXiv preprint arXiv:2502.20388; 2025.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-22 1-22, 2015.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelenn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv: 170 2017.
Wei Song, Yuran Wang, Zijia Song, Yadong Li, Haoze Sun, Weipeng Chen, Zenan Zhou, Jianhua Xu, Jiaqi Wang, and Kaicheng Yu. Dualtoken: Towards unifying visual understanding and generation with dual visual vocabularies. arXiv preprint arXiv:2503. 14324, 2025.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint 2665:22006006255, 2024.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at practical size. arXiv preprint arXiv:2408.00118 18, 2024.
Yao Teng, Han Shi, Xian Liu, Xuefei Ning, Guohao Dai, Yu Wang, Zhenguo Li, and Xihui Liu. Ac- celerating auto-regressive text-to-image generation with training-free speculative jacobi decoding. arXiv preprint arXiv:2410.01699, 2024.
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. Advances in neural information processing systems, 37:84839 84865, 2024.
Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. Metamorph: Multimodal understanding and generation via instruction tuning. arXiv preprint arXiv:2412. /4164, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Ama Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023a.
14

---

Published as a conference paper at ICLR 2026
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXhv:2307.09288 2023b.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelenn decoders. Advances in neural information processing systems, 29, 2016.
Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In Interaational conference on machine learning, pp. 1747-1756. PMLR, 2016.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang. Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409. 188.6 2024a.
Yuqing Wang, Shuhuai Ren, Zhijie Lin, Yujin Han, Haoyuan Guo, Zhenheng Yang. Difan Zou, Jiashi Feng, and Xihui Liu. Parallelized autoregressive visual generation. arXiv preprint wrXiv224/2.15//9 2024b.
Mark Weber, Lijun Yu, Qihang Yu, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chich Chen. Maskbit: Embedding-free free image generation via bit tokens. arXiv preprint (XXi:24499.162111 2024.
Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint arXiv:2410. 13848, 2024a.
Junfeng Wu, Yi Jiang, Chuofan Ma, Yuliang Liu, Hengshuang Zhao, Zehuan Yuan, Song Bai, and Xiang Bai. Liquid: Language models are scalable multi-modal generators. arXiv preprint 043.032,, 2024b.
Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Zhonghua Wu, Qingyi Tao, Wentao Liu, Wei Li, and Chen Change Loy. Harmonizing visual representations for unified multimodal understanding and generation. arXiv preprint arXiv:2503.21979. 2025.
Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: a unified foundation model integrating visual understanding and generation. arXiv preprint arXiv:2409.044: 04429,449999. 2024c.
Jinheng Xie, Zhenheng Yang, and Mike Zheng Shou. Show-o2: Improved native unified multimodal models. arXiv preprint 2rxhv:2506.55564 2025.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 15155:24/2.15115, 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vggan. arXiv preprint arXiv:2110.0462. 2021.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zinu Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint ar arXiv:2206. 10789, 2(3):5, 2022.
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang. Alexander G Hauptmann, Ming- Hsuan Yang, Yuan Hao, Irfan Essa, al. Magvit: Masked generative video transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14499-10469 2023a.
15

---

Published as a conference paper at ICLR 2026
Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion- tokenizer is key to visual generation. arXiv preprint arXtv:2310.057.// 2023b.
Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chich Chen. Randomized autoregressive visual generation. arXiv preprini urXiv:2411.00776. 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. Proceedings of the IEEE/CVF international conference on computer vision, pp. 11975-11986, 2023.
Yue Zhao, Fuzhao Xue, Scott Reed, Linxi Fan, Yuke Zhu, Jan Kautz, Zhiding Yu, Philipp Krähenbühl, and De-An Huang. Qlip: Text-aligned visual tokenization unifies auto-regressive multimodal understanding and generation. arXiv preprint ar0iv:2502.05/ 2025.
16

---

Published as a conference paper at ICLR 2026
APPENDIX
A ADDITIONAL IMPLEMENTATION DETAILS
A.1 MODEL ARCHITECTURE
We provide the model architecture configurations in Table 4. All models use standard decoder-only transformer architecture. We vary model scale by adjusting the number of layers, the hidden size, and the number of attention heads.
Model
Parameters
Layers
Hidden Size
Heads
LPD-L
111M
12
1024
12
LPD-XL
775M
36
1280
20
LPD-XXL
1.4B
48
1536
48
Table 4: Model architecture configurations.
A.2 TRAINING AND EVALUATION DETAILS
We train all models on ImageNet 256x 256 for 450 epochs, with 50 epochs of learning rate warmup followed by constant learning rate and finally 50 epochs of cosine decay. For 512-resolution models, load the pre-trained 256-resolution models and interpolate the positional embeddings and train on ImageNet 512 for 50 epochs. The continued training is conducted for 50 epochs using a cosine learning rate decay schedule, preceded by 1 epoch of warm-up. We use batch size 512 for LPD-L and 256 for LPD-XL.
We take the training of LPD-L model 256 x 256 resolution example and list all the training hyper-parameters in Table For LPD-XI. and LPD-XXI, we use batch size 1024 and the same base learning rate.
Hyper-parameters for 256x256 training Configuration
optimizer
Adam
81
0.9
32
0.95
learning rate?
8 x 10-1
batch size
2048 (64 x 32 GPUs)
training precision
BFloat16
total epochs
450
warm-up epochs
50
constant LR epochs
350
cosine decay epochs offsets
random per-sample
Table 5: Training
for LPD-L on 256 256 resolution.
We train on a range of predefined decoding steps where the number of tokens in each step is determined by cosine schedule. For the 256 x 256 resolution, the decoding steps are randomly selected from the set {8, 12, 16, 20, 24, 32, 64, 128, 256]. For the 512 x 512 resolution, the decoding steps are randomly selected from the set (32, 40, 48, 56, 64, 80, 96, 128, 160, 192, 224, 256, 512, 1024]. Take 20 steps in the 256 x 256 resolution an example, the number of tokens in each step is [1, 2, 4, 5, 7, 8, 10, 11, 12, 14, 15, 16, 17, 18, 18, 19, 19, 20, 20, 20].
For evaluation, we sweep the optimal frree guidance scale with an interval 0.1 and follow the Locality-aware Generation Order Schedule.
A.3 TEXT-TO-IMAGE GENERATION
We provide implemenation details in terms of model architecture, data, and training procedure.
Model. To process user prompts rather than class labels, we incorporate the Gemma-2-2B (Team et al., 2024) model as the text encoder, which converts prompts into text embeddings. These embeddings
Effeettive LRR computed hase (global batch size/256) with base Ir x 10-4
17

---

Published as a conference paper at ICLR 2026
passed through a linear projection layer and then concatenated with the image embeddings, with the text embeddings placed at the beginning of the sequence. All other components remain identical to our class consiionnd models. Our locality-awar generation order schedule continues to apply, as only the spatial resolution needs to be increased.
Data. We use internal MidJourrey-stylee synthetic dataset containing approximately 10M images. All images are re-captioned using Qwen2.5- VL-32B (Bai et al., 2025), which produces both concise caption and a detailed caption for each image. During training, we randomly sample one of the two captions to serve as the text prompt.
Training Recipe. We train models, LPD-L (344M) and LPD-XL (760M), using three-stage progressiverrcsolution schedule. We first train at x56x256 resolution for 40 epochs with batch sizes of 2048/1024 for the L/XL models. We then increase the resolution x56 checkpoint and interpolating positional embeddings, followed by 5 epochs of training with batch sizes of 1024/512 for L/XL. Finally, we increase the resolution 1024x 1024 by loading the 512x512 checkpoint and again interpolating positional embeddings, and train for 2 epochs with batch sizes of 256/128 for L/XL. For a fair comparison, we also create a raster order counterpart following the same setup.
512x512 by loading the
B LOCALITY-AWARE GENERATION ORDER
B.I PYTORCH IMPLEMENTATION
import numpy as np
import random
from ecipy.spatial.distane import edist
from wpapiappaaall..aa...
import euclidean
def 1pd_
sires-pne,, grid_size=16,
proximity__threshol... sepulsion__threshood-]]
if group_sizes is None:
10
group_sizes = [1] (grid size_
12
gric_coorde 11i, jl for in range for in (grid_ggrd_ sel selected_coords - []
13
15
for step, group size in enumerate (group_sizer sizes) if step 0:
16
17
For the first step, select a random coord. We always assumo the group size for the first step is 1. sclected_coords.. append (random. choice (grid_coords)) continue
19
20
21
Calculate the proximity score for all remaining grid coords candidates [] for coord in grid_coords:
24
if coord in selected_ccords
25
continue
76
Calculate the proximity score based on euclidean distance to already selected grid coords proximity_score
28
29
for selected_come coord in selected coorrs:: if abs (coord[0) abs (coord selected [crecteed_cord(1]] distance euclidean (coord, selected_coord) if distance 0:
selected_coorde
<= and
32
33
proximity_score += 1.0 / distance candidates. (proxxmity_sccree, coord])
18

---

Published as a conference paper at ICLR 2026
Shuffle candidates that grid coords with the same proximity score are randomly ordered
37
random, random.shufflo (canddiates) candidates,
38
sort sort (kcy=lambda x: x[0], reverse=True)
candidatesl = [item[1] for item in candidates if item [0]
>=
proximity_threehhol
40
candidates2 [item[]] for item candidates if item
sroximity_thresholke
step selected [] step_filtered []
44
45
Proxinity-based selection
while (stes sstep_selected] group size and candidatesl:
candidate candidatesl.popt0
48
too_close False for selected in step_selected:
49
30
if abs (candidate(0) - selected(0)) <=
pouiision_thrsholl and abs (1)
selected[1]) <= repulsion_ttreeholle
51
too_close True
52
step_fill hwl.ffleeeedappeee.. (candi break
53
55
if not too too__cooette close: step_selected.append (candidatt))
56
57
step_filtered extend (candidates1)
candidates2. ltssspfflleeerreee
60
61
Lox-dependency selection remaining group_size (sn(sttp_selected)
62
if remaining 0:
64
candidates?, remaining))
61
sel eelected eeeeeed_coords.... extend (step_selectes
return np. ravel multi index (ap. wrwaarayyaaoo.m...........oaaaaa (selected grid size) tollstt ()
hnddex/g...rrry/eeeeeee....eeee coordss) I, (grid_size,
def (archess_ppointt.aaannnnnnnn (cr.smpppoinggccxxttstooooootoo candidate_points,
num_to_ nun_to_select):
72
if len leaandddate_pinttee <= num num_to_select:
return padidateeppinnt..
75
Convert to numpy arrays for efficient computation existing_ag = np.array (existing_points)
76
77
cancidates_np aprarray (candidate_pointte
/nitialize with existing points selected_np - existing_ng.copy!
so
81
selected indices = []
for
in range (num select):
if len (sclected_ng)
0:
85
If no existing points, select randomly
86
idx = np. random. random. (len (candidates_np)) selected_np = candidates_mp [idx] newaxis,, :] else:
89
Calculate distances from all candidates to selected points distances cdist (candidates selected_ap) min distances np np.min (distances, axis=1)
90
19

---

Published as a conference paper at ICLR 2026
93
Set already selected candidates to distance min_distances (ereeatnnnneeeeleetteeiidddiiii [sclected_indices]]
94
95
96
Soloct the candidate with maximum minimum distance idx = np. argmax (min mnin_distances)
98
selected_np np.vstack ([eelected_np, [andidates_nppidxx)
99
100
selected indices append (idx)
return pondidaaee_pinntee for i in selected indices]
B.2 DISTINCTION FROM PREVIOUS WORK
The key distinction and primary advantage of our ordering mechanism that we turn both principles into single, explicit proximity objective. While previous works have observed each principle separately, none provide a way to quantify and jointly optimize them. our method, we define proximity metric that simultaneously (i) measures proximity to already generated context tokens and (ii) measures proximity among concurrently generated tokens, and we design an algorithm that optimizes generation orders with respect both. For example, (Wang et al., 2024b) aim reduce dependencies among concurrently generated tokens, but rely on fixed region-wise parallel scheme, which inherently cannot both maximize proximity previously generated tokens and minimize proximity within each concurrent group, Similarly, (Besnier et al., 2025) use Halton-based ordering decorrelate concurrent tokens; however, without proximity metric their method cannot incorporate our first principle of staying close to existing context.
B.3 SENSITIVITY ANALYSIS OF AND p
We conduct the sensitivity analysis using the LPD-XL model with 32 decoding steps on ImageNet x56x256. Our default configuration uses p 1 and t = 2.
Sensitivity analysis of r. We fix no repulsion applied, causing concurrently sampled tokens to be placed too close to another and violating the requirement that concurrently generated tokens should maintain low proximity. Conversely, when too large, concurrently sampled tokens are forced to be overly distant, making it difficult to find tokens that are both close to the already generated tokens and sufficiently far from each other. This violates the principle that newly generated tokens should preserve high proximity to existing tokens. Therefore, moderate repulsion threshold is essential.
and vary T to evaluate its impact (Table 6, left). When 7 = 0.
Sensitivity analysis of p. We fix T = 2 and vary & to evaluate its impact (Table 6, right), A larger p requires tokens to have very high proximity in order to be considered sufficiently close to the already generated tokens. This makes the condition increasingly difficult to infinitely large, token would meet the criterion, rendering the condition meaningless. Conversely, when p = 0, no threshold is imposed and all tokens are treated as sufficiently close to the generated ones. As shown in the results, the FID increases only slightly in this case. This is because algorithm still sorts tokens by their computed proximity and selects them descending order, so the first crinciple-prioritizing
indeed, if were
prooiitt remainss intact. This also indicates that FID is relatively insensitive to smaller values of p.
Table 6: Sensitivity analysis of and p. All experiments LPD-XL with 32 decoding steps ImageNet 256 256. Default values (7-2, p=1) are underlined.
(n) Varying (fx p 1)
(h) Varying (fix
0
2
3
4
0.2
0.5
1.5
2
FID
2.00
1.94
1.92
1.97
2.00
FID
1.94
1.94
1,93
1.92
2.00
2.04
C MORE VISUALIZATION OF ATTENTION MAPS
We provide partial visualization of the attention maps in Figure 2 and we provide more here. We select two layers each consists of 24 attention heads during the decoding and visualize them in Figure 10 and Figure 11.
20

---

Published as a conference paper at ICLR 2026
Head
Head
Head
Herd 4
Head 5
Head 6
Head
Head
Head
Head 10
Head 11
Head 32
Head
Head
Head 15
Head 16
Head
Head 18
Figure 10: More visualization of attention maps in the L.L.MAGEN-J. 1.4B model.
17
Hcad 28
Head 1.0
Head 20
Head 23
Head 22
Head 23
Head 24
Figure 11: More visualization of attention maps in the LLAMAGEN-L.4B model.
21

---

Published as a conference paper at ICLR 2026
D MORE VISUALIZATION OF GENERATION EXAMPLES
Figure 12: Generation Examples of Our Model. We show 512x512 generation samples (top), 256x256 generation samples (middle) and zero-shot image editing results including class-conditional editing, inpainitng and outpainting (bottom).
Our model can naturally perform zero-shot editing tasks since we support image generation in arbitrary order. For image inpainting and outpainting, we prefill the KV cache with all tokens from the non-repaint regions along with class token and generate the masked region in random order. For class-conditional editing, we substitute the class embedding with a new class embedding and generate the edited region in random order.
22

---

Published as a conference paper at ICLR 2026
small cactus with happy face in the Sahara desert.
Ultra-detalled wildllife photo of zebra standing in golden savanna light.
A melting apple.
orange sliding down aqua slide, happy excited. Vibrant colors, water splashing the
alpaca made colorful building blocks, cyberpunk.
elephant walking water.
handsome 24 years old boy the middle with sky color background wearing eye glasses, ss super detailed with anime style.
Ultra-detall detailed portrait of old man long beard, weathered skin, deep expressive eyes.
astronaut ricling moon, oil painting by Van Gogh.
A realistic landscape shot the Northem Lights dancing over a snowy mountain range in loeland.
New England fall with leaves, and river.
crystal shimmering under starry sky.
Figure 13: Generation Examples of Our Model. We show 1024x 1024 text-to-image generation samples.
23

---

Published as a conference paper at ICLR 2026
Table 7: Detailed per-calegory GenEval scores for text-to-image generation 1024x 1024 resolution.
Single Obj.
Ohj.
Counting
Position
Counterpart-L
Raster Couneepart-Xi:
0.60
0.99
0.71
0.46
0.82
0.25
0.36
LPD-L LPD-XL
0.58
1.00
0.06
0.39
0.85
0.17
0.42
0.62
0.99
0.76
0.45
0.82
0.34
0.37
E EFFICIENCY ANALYSIS
Raatee-1
BaxisrXI
LPD-L
16.09
400
Theughrininh
100
Ther hiw.spp--itmgg mott
10
0.1
0.1
0.10
(1)
(9)
Figure 14: Throughput vs. Batch Size on ImageNet 256 256 Class-Conditte Generation. For LPD, we use 20 generation steps. Raster refers the traditional fixed-raster-order generation model. We progressively increase the batch size until the process runs out of memory. The throughput values on the y-axis are plotted on a logarithmic scale.
As shown Figure 14, LPD models are memory- when the batch size is 16 or smaller, as indicated by the linear inerease in throughput with respect to batch size. When the batch size exceeds 16, the process gradually transitions from being memory-bound to compute-bous bound. For the traditional fixed-rster-orrder models, this transition occurs batch size around 128. Notably, when both models operate in the memory-bound regime, LPD consistently achieves nearly 12x higher throughput than the raster-order model-roughly roughly matching the reduction the number of generation steps. When at the maximum batch size, LPD still maintains a throughput ad advantage of approximately 3x.
24


Reviews:
[{"review_id": "MZCLZEONMi", "reviewer": "Reviewer_z7UP", "summary": "The paper introduces Locality-aware Parallel Decoding (LPD), a method designed to speed up autoregressive image generation. LPD uses two key techniques—Flexible Parallelized Autoregressive Modeling for arbitrary parallelization and Locality-aware Generation Ordering for quality enhancement—and reduces generation steps significantly (e.g., 256 to 20 for 256×256 images) on ImageNet while cutting latency by at least 3.4× compared to prior parallel models.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* Sound Method Design：The learnable position query tokens decouple context modeling from decoding, enabling generation at arbitrary target positions and boosting flexibility. The exploration of two locality principles, particularly the second one, offers meaningful insights for the community.\n\n* Strong Performance：The method achieves clear reductions in generation steps and latency with good quality.", "weaknesses": "1. Overclaimed Contributions in Writing\n\n   - For \"Flexible Parallelized Autoregressive Modeling\", decoder-only works like PAR/ZipAR/NAR already treat previously decoded tokens as KV Cache and the queries are decoded in parallel ensuring the mutual visibility among tokens generated concurrently; the key difference lies only in LPD’s position query tokens (enabling arbitrary target positions), which should be clarified to avoid overstating contributions.\n   - For \"Locality-aware Generation Ordering\", the first principle is well-studied (Sec. 3.2 descriptions can be simplified), while the second principle (\"low proximity among concurrent tokens\") has been explored by Wang et al. (2024b) and Besnier et al. (2025)—its underexploration in prior work needs more detailed explanation in Sec. 3.2.\n\n2. Insufficient Ablation Studies\n   - Sensitivity analysis of critical thresholds $\\tau$ and $\\rho$ is missing, despite their importance to query position performance.\n   - A key ablation is absent: for Flexible Parallelized AR Modeling (excluding adaptive generation order), testing whether replacing query tokens with LPD’s \"position query tokens\" would clarify the source of performance gains.\n\n3. Confusing Details\n    - Inference process ambiguity: It is unclear if sampled tokens need a forward pass to store KV Cache, and if this can be done in the final decoding step.\n   - Figure 3 confusion: The figure shows queries attending only to the latest decoded tokens, conflicting with the claim that queries causally attend to all previously generated tokens.\n\n4. Minor Structural Issue：The dynamic generation order, a core technical contribution, should be included in the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VBx0wQFrlJ", "reviewer": "Reviewer_Gaej", "summary": "This paper proposes a method to accelerate AR visual generation, which is typically bottlenecked by sequential token prediction and memory bandwidth. It introduces two core innovations: (1) Flexible Parallelized Autoregressive Modeling, which allows arbitrary generation order and parallel prediction using learnable position query tokens and (2) Locality-aware Generation Ordering, a scheduling strategy that groups spatially related tokens to minimize dependencies and maximize contextual support. These techniques reduce generation steps dramatically and achieve at least 3.4× lower latency than prior parallel AR models without compromising image quality on ImageNet benchmarks.", "questions": "Could the proposed LPD framework be extended or adapted to multimodal generation (e.g., image–text, video, or audio tokens)? Given its reliance on spatial locality, how might token dependencies behave for non-visual modalities?\n\nHow would LPD compare in efficiency and quality against speculative decoding or speculative Jacobi decoding? Including at least one experiment could clarify whether LPD complements or surpasses these methods.\n\nSince the fusion of encoding and decoding steps introduces additional query tokens, could you quantify peak GPU memory usage and performance scaling with batch size and resolution (beyond batch 64)?\n\nThe method partially decouples context and generation tokens. Could the authors clarify how this affects the probabilistic consistency of the AR factorization (Eq. 2 in the paper)? In addition, the paper highlights speedups, but how does varying the group size quantitatively affect image fidelity and diversity?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper conducts a genuinely deep analysis of the key factors that affect both performance and generation quality in parallel autoregressive decoding (e.g., group size, dependency structure, attention visibility), and then turns those observations into a coherent, end-to-end parallelization method rather than a single heuristic component.\n\nThrough careful comparisons with recent parallel AR implementations (e.g., encoder–decoder style SAR/ARPG and decoder-only RANDAR), the authors show that their design, especially the “mutual visibility among concurrently generated tokens + cache only generated tokens” part, is not just faster but architecturally better motivated, and they support this with latency as well as quality numbers.\n\nThe paper is very readable: the motivation is clearly laid out, figures are aligned with the text (the attention-mask figures in particular), and the training vs. inference formulations are described in a way that makes reproduction and reimplementation realistic even for non-authors.", "weaknesses": "Experiments are limited to image generation; since current AR models are increasingly used for multimodal I/O (image–text, video tokens, layout, even audio tokens), it would strengthen the claim of “general AR parallelization” to show at least one non-image setting (e.g., CLIP-conditioned image tokens, image+text joint decoding, or video latents).\n\nThe paper does not compare against the newest AR acceleration lines such as speculative decoding, speculative Jacobi-style decoding, or draft/verify variants; even a small-scale experiment would help position LPD as complementary vs. strictly better.\n\nWhile there is throughput analysis at batch 64, the work does not fully characterize memory consumption and scaling beyond that point; since the method introduces extra query tokens and fused encode–decode steps, reporting peak GPU memory and how it scales with batch size and resolution would make the efficiency story more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "loPV4OHH2f", "reviewer": "Reviewer_mUfX", "summary": "The paper introduces Locality-aware Parallel Decoding (LPD) with two techniques to accelerate autoregressive image generation. First, Flexible Parallelized Autoregressive Modeling leverages position-aware query tokens to indicate the tokens to be generated, enabling arbitrary generation ordering and degrees of parallelization. Second, a Locality-aware Generation Ordering is proposed to minimize mutual dependencies during parallel generation. Experiments on ImageNet 256$\\times$256 and 512$\\times$512 demonstrate the effectiveness of the proposed LPD.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed Flexible Parallelized Autoregressive Modeling overcomes the constraint of a fixed generation order by allowing images to be synthesized in an arbitrary sequence. This capability holds the potential for discovering more effective generation orders in the future.\n2. When equipped with the proposed Locality-aware Generation Ordering strategy, LPD demonstrates improved FID scores and greater generation efficiency on the ImageNet dataset.\n3. The paper is easy to read and the figures are informative.", "weaknesses": "1. The paper's core algorithm (Algorithm 1) is presented in the appendix. While space constraints are understandable, the most critical algorithm should ideally be included in the main text, or at the very least, its underlying principles should be explained there.\n2. The computational cost of the model increases compared to traditional fixed-order autoregressive models due to the use of additional positional query tokens. However, an analysis of this overhead is absent from the paper.\n3. In line 208, the authors state that in previous methods, \"tokens generated within the same parallel step are produced independently of one another.\" However, the paper does not rigorously analyze the issue of conditional independence in LPD's parallel sampling. It appears that while LPD ensures visibility among all target positions predicted concurrently, it may not fully resolve the underlying conditional independence of the parallel-generated tokens. A theoretical justification for this aspect would be beneficial.\n4. The authors train the LPD model for 450 and 500 epochs on ImageNet at 256×256 and 512×512 resolutions, respectively. In contrast, most baseline methods (e.g., LlamaGen, PAR, and NAR) are trained for only 300 epochs. This discrepancy in training budgets may lead to an unfair comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LVQcKISZ4x", "reviewer": "Reviewer_axY1", "summary": "This paper proposes Locality-aware Parallel Decoding (LPD), an efficient method for parallel autoregressive image generation. By introducing position query tokens, it enables flexible, location-independent parallel decoding, while a locality-aware generation order improves consistency and quality. Experiments show that LPD achieves comparable or better image quality with up to 10× fewer generation steps and 3–4× faster decoding speed.", "questions": "see above", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Here are the main strengths of the paper:\n\nSignificant Speedup – LPD greatly reduces the number of decoding steps (up to 10–20× fewer) while maintaining or improving image quality.\n\nHigh-Quality Generation – The locality-aware scheduling preserves spatial coherence, producing consistent and detailed images even under high parallelism.\n\nFlexible Decoding Framework – The position query token design allows generation in arbitrary orders, enabling diverse tasks like inpainting and outpainting without retraining.\n\nStrong Empirical Validation – Extensive experiments on ImageNet (256×256 and 512×512) demonstrate clear improvements in both FID and latency over prior autoregressive baselines.", "weaknesses": "The paper is technically strong and the proposed method is clearly effective for autoregressive image generation. However, I think the evaluation could be further strengthened by extending it beyond ImageNet. In particular, it would be interesting to test the method on text-conditioned generation at higher resolutions (e.g., 1024²) to see whether the parallel decoding and locality assumptions still hold under more complex, long-range dependencies.\nMoreover, applying LPD to video generation or temporal sequence modeling could highlight its scalability in spatiotemporal domains. Even a small-scale video experiment (e.g., UCF-101) would make the paper more convincing in terms of generality and impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)