ZipAR: Parallel Autoregressive Image Generation through Spatial Locality
Authors: Yefei He
Feng Chen
Yuanyu He
Shaoxuan He
Hong Zhou
Kaipeng Zhang
Bohan Zhuang
Venue: ICML 2025 poster (2025)
Keywords: Auto-regressive (AR) visual generation
Parallel decoding
Efficient AI
Primary area: deep_learning->generative_models_and_autoencoders

Abstract: In this paper, we propose ZipAR, a training-free, plug-and-play parallel decoding framework for accelerating autoregressive (AR) visual generation. The motivation stems from the observation that images exhibit local structures, and spatially distant regions tend to have minimal interdependence. Given a partially decoded set of visual tokens, in addition to the original next-token prediction scheme in the row dimension, the tokens corresponding to spatially adjacent regions in the column dimension can be decoded in parallel. To ensure alignment with the contextual requirements of each token, we employ an adaptive local window assignment scheme with rejection sampling analogous to speculative decoding. By decoding multiple tokens in a single forward pass, the number of forward passes required to generate an image is significantly reduced, resulting in a substantial improvement in generation efficiency. Experiments demonstrate that ZipAR can reduce the number of model forward passes by up to 91% on the Emu3-Gen model without requiring any additional retraining.

ZipAR: Parallel Autoregressive Image Generation through Spatial Locality
Yefei He
Feng Chen
Yuanyu He
Shaoxuan He
Hong Zhou'
Kaipeng Zhang?
Bohan Zhuang
Prompt: bustling downtown street in Tokyo at night, with neon signs, sidewalks, and tall skyscrapers
SO
Prompt: A mountain lake at sunrise, with mist rising off, and snow-capped peaks in the background
Steps: 8190
Steps: 2849 (-65%)
Steps: 1425 (-82%)
Steps: 713 (-91%)
Figure 1: Up to 91 % forward step reduction with ZipAR. Samples are generated by Emu3-Gen model with next-token prediction paradigm (the first column) and ZipAR (the right three columns).
Abstract
corresponding to spatially adjacent regions in the column dimension can be decoded in parallel. To ensure alignment with the contextual require- ments of each token, we employ an adaptive local window assignment scheme with rejection pling analogous to speculative decoding. By de- coding multiple tokens in single forward pass, the number of forward passes required to generate an image is significantly reduced, resulting in a substantial improvement in generation efficiency. Experiments demonstrate that ZipAR can reduce the number of model forward passes by up 91% % on the Emu3-Gen model without requiring any additional retraining.
In this paper, propose ZipAR, training-free. plug-and-play parallel decoding framework for accelerating autoregressive (AR) visual genera- tion. The motivation stems from the observation that images exhibit local structures, and spatially distant regions tend have minimal interdepen- dence. Given a partially decoded set of visual tokens, in addition to the original next-token pre- diction scheme in the row dimension, the tokens
'Zhejiang University, China 'Shanghai Al Laboratory, China The University of Adelaide, Australia. Correspondence to: Hong Zhou <zhouhong zju @ju@zju.cuu..cn>>>>. Kaipeng Zhang <kp_zhang foxmail.com....
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

ZipAR; Parallel Autoregressive Image Generation through Spatial Locality
(a) Raster order AR Modeling
1
2
3
4
Vector Quantizer
VQ Encoder
5
7
8
Decoder
VQ
...
Input (training)
Training pipeline
Rearange
Flatten
AR Model
Decoding pipeline
3
Output
(b) Medusa/Jacobi
(c) MAR
(d) ZipAR
Generated tokens
Tokens
decode
Flattened view
Flattened view
Flattened view
Figure 2: (a) An overview of the training and decoding pipeline for autoregressive (AR) visual generation models. For models trained with next-token prediction objective, each forward pass generates single visual token. (b) Medusa (Cai et al., 2024) and Jacobi (Santilli et al., 2023) decoding predict multiple adjacent tokens in sequence order. (c) MAR (Li et al., 2024) predicts multiple tokens a random order. (d) The proposed ZipAR predicts multiple spatially adjacent tokens.
1. Introduction
reduce the number of forward passes required for gen- erating lengthy responses, several studies (Cai al., 2024; Santilli al., 2023; Chen et al., 2023) have proposed the "next-set prediction" paradigm for LLMs, as depicted in Fig- ure 2(b). These approaches involves introducing multiple de- coding heads (Cai et al., 2024) or small draft models (Chen et al., 2023), which generate several candidate tokens that are later evaluated by the original model. However, these methods incur additional costs, as they require extra draft models or the training of new decoding heads. Another approaches the jacobi decoding methods (Santilli et al., 2023; Fu et al., 2024; Teng al., 2024), iteratively updates sequences of tokens until convergence. However, in practice, the acceleration achieved by these methods is marginal, as LLMs often fail to generate correct tokens when errors exist in preceding ones. Furthermore, none of these approaches exploit the unique characteristies of visual content, and parallel decoding framework specifically tailored for AR visual generation has yet to be developed.
Recent advancements in large language models (LLMs) with the "next-token prediction" paradigm (Achiam et al., 2023; Vavekanand & Sam, 2024; Team et al., 2023) have demonstrated remarkable capabilities in addressing text- related tasks. Building these successes, many stud- ies (Liu al., 2024a; Wang al., 2024b; Team, 2024; Ge al., 2024; Wu et al., 2024a) have extended this paradigm to the generation of visual content, leading to the development of autoregressive (AR) visual generation models. These models not only produce high-fidelity images and videos that rival or even exceed the performance of state-of-the-ar diffusion models but also facilitate unified multimodal un- derstanding and generation (Wang et al., 2024a; Chen al., 2025; Wu et al., 2024a;b). However, their slow generation speed remains significant barrier to widespread adoption. To generate high-resolution images or videos, these mod- els must sequentially produce thousands of visual tokens, requiring numerous forward passes and resulting in high latency.
In this paper, we introduce ZipAR, parallel decoding framework designed to accelerate AR visual generation.
2

---

ZipAR; Parallel Autoregressive Image Generation through Spatial Locality
100
0
100
100
500
110
200
200
1000
300
300
1500
400
400
10-3
2000
500
500
0
500 1000 1500 2000
100
200
300 400
500
100 200 300 400 500
Layer 10, Lumina-mGPT
(b) Luyer 10, LlamaGen
(c) Layer 20, LlamaGen
Figure 3: The attention scores of visual tokens in the Lumina-mGPT-78 (Liu et al., 2024a) and LlamaGen-XII (Sun et al., 2024) models. Slash lines indicate that significant attention scores are allocated to tokens at fixed intervals, corresponding to tokens in the same column of previous rows. The full attention scores are presented by storing the attention scores of each visual token during decoding and concatenating them.
As depicted in Figure 2(a), common AR visual generation models produce visual tokens in raster order, where the first token in cannot be generated until the last to- ken in the preceding row is decoded despite their spatial separation. However, visual content inherently exhibits strong locality, which is a widely utilized inductive bias for visual tasks (Liu et al., 2021; Zhang et al., 2022; Le- Cun et al., 1989; Krizhevsky et al., 2012: Zeiler & Fergus, 2014). Specifically, there are significant spatial correla- tions between spatially adjacent tokens (e.g., token and token I in Figure 2(a)) compared to tokens that are adjacent only in the generation order (e.g., token 5 and token 4), which makes the raster-order sequential dependency sub- optimal. Empirical evidence, as shown in Figure 3, further supports this observation, with significant attention allo- cated to tokens in the same column of the previous row. This motivates to propose decoding tokens from the next row without waiting for the full decoding of the current row, enabling the parallel decoding of multiple tokens in a single forward pass. Specifically, predefined window size determines whether two tokens are spatially adjacent. Tokens outside this window in adjacent rows are considered irrelevant. Consequently, once the number of generated tokens in a row exceeds the window size, decod- ing of the next row begins in parallel with the current row. With an appropriately chosen window size, multiple rows can be decoded simultaneously. Unlike Medusa (Cai et al., 2024), which employs auxiliary heads, all tokens generated in parallel by ZipAR are produced using the original model head. Moreover, address the limitation that manually tuned window size may not optimally adapt to varying at- tention distributions across tokens, introduce adaptive window size assignment scheme. This scheme dynamically adjusts the local window size during generation, ensuring that each token is generated with a window size tailored
to its contextual requirements. As result, ZipAR can be seamlessly implemented in training-free plug-and-play manner for autoregressive visual generation models, without introducing additional overhead. Experiments across mul- tiple autoregressive visual generation models demonstrate the effectiveness and robustness of ZipAR, achieving for- ward steps reductions of 91%, 75%, and 81% on Emu3-Gen, Lumina-mGPT-7B. and LlamaGen- XL models, respectively, with minimal degradation in image quality.
In summary, our contributions follows:
We propose spatially-aware parallel decoding strategy that enables inter-row token generation by leveraging the inherent spatial locality of visual content. Once the number of generated tokens in a row exceeds a window size, decoding of the next row begins in parallel.
We propose adaptive window size assignment scheme that dynamically adjusts the local window size for each token during generation, optimizing decoding efficiency while ensuring the contextual information essential for producing high-quality tokens.
By integrating these techniques, we present ZipAR, training-free, alug-and-play framework that achieves significant acceleration in autoregressive visual gener- ation. Extensive experiments demonstrate its effective- ness and robustness across multiple AR visual genera- tion models.
2. Related Work
2.1. Autoregressive Visual Generation
The success of Transformer models in text-based tasks has inspired studies (Van Den Qord al., 2017; Esser et al.,
3

---

ZipAR; Parallel Autoregressive Image Generation through Spatial Locality
2021; Yu al., 2023) to apply autoregressive modeling to visual content generation. These methods can be classified into two main categories: GPT-style approaches that uti- lize the next-token prediction paradigm (Esser et al., 2021; Wang et al., 2024b; Liu et al., 2024a; Sun et al., 2024) and BERT-style approaches that employ masked prediction models (Chang et al., 2022; 2023; Li et al., 2024; Yu et al., 2023). More recently, VAR (Tian et al., 2024) modified the traditional next-token prediction paradigm to next-scale pre- diction, resulting in faster sampling speeds. Models trained using next-token prediction can leverage the infrastructure and training techniques of large language models (LLMs) and pave the way towards unified multi-modal understand- ing and generation. However, they are generally less ef- ficient during sampling compared to models that predict multiple tokens in single forward pass. In this paper, we focus on accelerating visual generation models trained with the next-token prediction objective, hereafter referred to as autoregressive visual generation models unless otherwise specified.
points on the Jacobi trajectory to a fixed point. However, none of these approaches are designed for autoregressive visual generation or incorporate visual inductive biases. In contrast, the proposed ZipAR takes advantage of the spatial locality inherent in visual content, offering significant accel- cration without the need for retraining. Moreover, ZipAR is orthogonal to the aforementioned methods, and can be combined with them achieve even greater acceleration.
3. Method
3.1. Preliminaries
Autoregressive (AR) visual generation models with the next- token prediction paradigm have shown exceptional versa- tility across various vision-language tasks, including gen- erating high-quality images and videos. As shown Fig- ure 2(a). pre-trained VQ-VAE models (Van Den Oord et al., 2017; Esser al., 2021) are commonly employed to convert images or videos into visual tokens. The process begins with a visual encoder that extracts feature maps at a reduced spatial resolution. These feature maps are then subjected to vector quantization to produce discrete latent representa- tions, known as visual tokens. These tokens are arranged in one-dimensional sequence to serve as input for AR models. Although various methods exist to flatten these tokens, the major order (raster order) is empirically validated to offer the best performance (Esser et al., 2021), making it the prevalent method for visual generation. During the image generation phase, AR models generate visual tokens sequen- tially this raster order. Finally, the complete sequence of visual tokens is rearranged into a two-dimensional structure and processed through a visual decoder to reconstruct the images.
2.2. Efficient Decoding of LLMs.
Efforts to reduce the number of forward passes required for LLMs to generate lengthy responses can be broadly categorized into two main approaches. The first approach involves sampling multiple candidate tokens before verify- ing them with the base LLM. Speculative decoding (Chen et al., 2023; Liu et al., 2024b; Spector & Re, 2023; Gui et al., 2024) utilizes small draft L.M to generate can- didate tokens, which are then verified in parallel by the base LLM. While this approach can potentially generate multiple tokens in a single evaluation, deploying multiple models introduces significant memory overhead and engi- neering challenges. Medusa (Cai et al., 2024) addresses this by employing multiple decoding heads for the base LLM, enabling self-speculation. However, due to the large vocabulary size of LLMs, the parameters in each decoding head can be substantial. The second approach, Jacobi de- coding (Santilli et al., 2023; Teng et al., 2024), involves randomly guessing the next n tokens in a sequence, which are iteratively updated by the LLMs. Over time, the n-token sequence converges to the same output as that generated by the next-token prediction paradigm. However, in prac- tice, vanilla Jacobi decoding offers only marginal speedup over autoregressive decoding. This limited improvement is largely due to the causal attention mechanism, which rarely produces a correct token when preceding tokens are incorrect. Lookahead (Fu et al., 2024) decoding enhances efficiency by leveraging n-grams generated from previous Jacobi iterations, which are verified in parallel during the decoding process. CLLMMs (Kou et al., 2024) further im- proves the efficiency of Jacobi decoding by fine-tuning the model with a consistency loss, requiring it to map arbitrary
3.2. Inference with ZipAR
As analyzed in Section 3.1, AR visual generation mod- els with a raster order generate visual tokens row by row, completing each row sequentially from left to right before proceeding to the next. However, images inherently exhibit strong spatial locality. Intuitively, in a high-resolution im- age, the starting pixel of a row is more closely related to the starting pixel of the preceding row than to the ending pixel of the preceding row due to their spatial proximity. Em- pirical evidence, as shown in Figure 3, also indicates that significant attention scores are allocated to tokens within the same column of the previous row. Building on these obser- vations, we propose ZipAR, a simple yet effective parallel decoding framework for autoregressive visual generation models. Unlike conventional parallel decoding methods that predict multiple consecutive tokens in a single forward pass, our approach decodes tokens from different rows in parallel. The key idea is that it is unnecessary wait for entire now to be generated before initiating the decoding of the
4

---

ZipAR; Parallel Autoregressive Image Generation through Spatial Locality
Xo,0
X,,1
X,,2
X
X
Xp,
X,,
X0,3
X,,0
X,,
X,,
Generated tokens
1,,0
X1,
X,2
X13
forward
110
X1,
x12
X,3
forward
x1.0
X11
X12
X13
22,0
x21
22,
X2.3
22,0
2,1
X2,2
X2,
X2,0
X2,1
X22
x2,3
Tokens to decode
X3,0
X3,1
X3,2
X3,3
x3,0
X3,1
X3,2
X3,3
X3,0
X3,1
X3,2
X3,3
Flattened view
Figure 4: A toy example of the ZipAR framework. The window size is set to in this toy example.
next row, as spatially distant tokens contribute minimally to attention scores.
COCO
40
Parti
To formalize this, we define a local window size s. Given the tokens Fi.j located in row and column j. we assume that tokens beyond Et-1,j+s in the previous row have negligible impact the generation of vijj based the
Window Size (s)
30
25
spatial locality of visual tokens. Consequently, the criterion for initiating the generation of token Fi can be formulated as:
20
11,
if li
10
C(i.j) = 0, otherwise
(L)
10
20
30
40
50
Row Index
Figure 5: The local window size required retain 95% of attention scores across different rows and input prompt. Data is collected from the first token of each row in Lumina- mGPT-7B model with input prompt from COCO (Lin et al., 2014) and Parti (Yu et al., 2022) dataset.
Here, denotes the set decoded tokens, and C(i,j) = indicates that token Ti is ready to be generated. Once the first token in a row generated, subsequent tokens in the row be generated sequentially, along with the unfinished portion of the preceding row, following next- token prediction paradigm. An illustration of the ZipAR framework with window size of 2 is shown in Figure 4.
3.3. Adaptive Window Size Assignment
However, to initiate the decoding the first token 2;,0 in row i, the last token of the row is required as input to the autoregressive model, despite it has not yet been gener- ated in the ZipAR framework. To address this, we propose several solutions tailored to different types of AR visual generation models. Some methods (Liu et al., 2024a; Wang et al., 2024b) support generating images with dynamic res- olutions, typically by appending extra end-of-row tokens at the end of each row. With these special tokens placed at fixed positions, we can insert the end-of-row tokens in ad- vance when initiating the generation of the next row. Since the values of these tokens are predetermined, there is need to update them subsequently. Conversely, for models that lack end-of-row tokens (Sun et al., 2024), we temporar- ily assign values to the last token in row i to decode token 24,0- This value can be derived from the most spatially adjacent token that have been decoded.
While ZipAR with a predefined local window size demon- strates improved efficiency, the window size remains a hy- perparameter that requires manual tuning to balance image fidelity and generation efficiency. Moreover, using fixed window size for all token positions is suboptimal, as the attention distributions vary significantly across tokens. As illustrated in Figure 5, the local window size needed to retain 95% of attention scores differs across token positions and input prompts. Consequently, maintaining fixed window size throughout the image generation process can lead suboptimal results, potentially compromising image fidelity.
To address this, we propose an adaptive window size assign- ment scheme that dynamically adjusts the local window size during the generation process. Given a minimum window size Smin, after generating token Is, -1 in row we attempt to generate the first token row i Unlike the
5

---

ZipAR; Parallel Autoregressive Image Generation through Spatial Locality
fixed window size approach, we do not immediately accept this newly generated token, as the current local window size may provide insufficient information. Instead, in the subsequent step, with the addition of new token from the previous row, we regenerate the token using a slightly larger window size Smin + and apply an acceptance criterion to evaluate its validity based on the predictions from both steps. If the criterion is satisfied, subsequent tokens in row i ++1 1 can be generated sequentially, following a next-token prediction paradigm. Otherwise, the current window size is deemed inadequate, and we iteratively expand it until the criterion is met or the previous row is fully generated.
4.2. Main Results
4.2.1. CLASS-CONDITIONAL IMAGE GENERATION
In this subsection, we quantitatively evaluate the perfor- mance of class-conditional image generation on the Ima- geNet 256 x 256 benchmark using the LlamaGen model, as summarized in Table 1. The model processes a 24 x 24 feature map and requires 576 forward passes to generate an image under the next-token prediction (NTP) paradigm. For the LlamaGen-L. model, integrating ZipAR with a minimal window size of 16 reduces the number of forward passes by 26.7% 667%% without increasing the FID score. For the LlamaGen- XL model, ZipAR- 12 achieves a lower FID (3.67 vs. 3.87) while requiring fewer steps than the previous parallel decod- ing algorithm, SJD (Teng et al., 2024) (331 steps vs. 335 steps). This highlights the efficiency of ZipAR in decoding spatially adjacent tokens in parallel.
Specifically, we adopt a rejection sampling scheme analo- gous to speculative decoding (Leviathan al., 2023; Chen et al., 2023). For consecutive window sizes k + 1 and k in row i, we compute the ratio between their predictions pf -..,,,,..,,,,,kk) and p(z/200,0 ..... -...,-,,.611).. 11))) which quanti- fies how well the token sampled under the smaller window size. Formally, the criterion for initiating the generation of token 2,410 with window size l can be formulated as:
4.2.2. TEXT-GUIDED IMAGE GENERATION
In this subsection, we expand our evaluation by assess- ing ZipAR's performance using multiple metrics, including VQAScore (Lin et al., 2024), Human Preference Score v2 (HPSv2) (Wu et al., 2023), ImageReward (Xu et al., 2023), and Aesthetic Score, across three models: LlamaGen-XL 512, Lumina-mGPT-768, and Lumina-mGPT-1022 as pre- sented in Table 2. For the LlamaGie- model, ZipAR-15 reduces the number of generation steps by 45.1% 1% with- out any decline in the VQAScore, Image Reward and Aes- thetic Score. Similarly, for the Lumina-mGPP-766 model, ZipAR-20 achieves a 54.8% reduction in generation steps while improving VQAScore, HPSv2, and Aesthetic Score. When evaluating the CLIP Score over the LlamaGen-Xi model, compared to the previous parallel decoding algo- jithhmm SJD (Teng et al., 2024), ZipAR-7 significantly im- proves efficiency (324 steps vs. 635 steps) while achieving higher CLIP score (0.285 vs. 0.283). Moreover, we observe that the acceleration ratio for both text-to-image models is higher than that for the class-conditional LlamaGen-L model. This is primarily attributed to the larger spatial reso- lution the feature maps and the generated images. These results suggest that ZipAR provides greater efficiency gains when generating higler-resolution images.
1, ifr ifr Fin min (1, plu (o.
1(i+
0, otherwise
Here, we sample r UO0, 1] from uniform distribution. ((i+1,0) 1 indicates that token 2,+1,0 is ready to be generated. If the criterion is not met, we resample 2;+1,0 from the following distribution:
max(0, p(z (/.000,0-00 2.,5) p(z 20.0 0: Li,k- 1))
1,+1,0
Σx max p( xi,k) - 00.00 .... Di.k
(3)
The resampled token subsequently verified in the next step.
4. Experiments
4.1. Implementation Details
To assess the effectiveness of our proposed method, we in- tegrate it with three state-of-thh--ars autoregressive visual generation models: LlamaGen (Sun et al., 2024), Lumina- mGPT (Liu et al., 2024a) and Emu3-Gen (Wang et al., 2024b). All experiments are conducted with Nvidia A100 GPUs and Pytorch framework. For class-conditional im- age generation with LlamaGen on ImageNet, we report the widely adopted Frechet Inception Distance (FID) to evaluate the performance. We sample 50000 images and evaluate them with ADM's TensorFlow evaluation suite (Dhariwal & Nichol, 2021).
4.3. Ablation Study
1.3. 1. EFFECT OF ADAPTIVE WINDOW SIZE ASSIGNMENT
In this subsection, we evaluate the effectiveness of the pro- posed adaptive window size assignment scheme. Specifi- cally, we compare the performance of ZipAR with fixed and adaptive window sizes over class-conditional LlamaGen- L model, respectively. As shown in Figure 6, under sim- ilar generation steps, ZipAR with adaptive window size
6

---

ZipAR; Parallel Autoregressive Image Generation through Spatial Locality
Table 1: Quantitative evaluation on ImageNet 256 x 256 benchmark. The generated images are 384 x 384 and resized to 256 x 256 for evaluation. Here, "NTP" denotes the next-token prediction paradigm. "ZipAR-n" denotes the ZipAR paradigm with a minimal window size of n. "Step" is the number model forward passes required generate an image. The latency is measured with batch size of
Model
Method
Step
Latency (s)
FID_
NTP
576
15,20
3.16
SJD (Teng et al., 2024)
367 (-36.3%)
10.83 (-28.8%)
3.85
LlamaGen-L (cfg=2.0)
ZipAR-16 16
422 (-26.7%)
11.31 (-25.6%)
3.14
ZipAR-14
378 (-34,4%)
10.16 (-33.2%)
3.44
ZipAR-12 12
338 (-41.3%%
9.31 (-38.8%)
3.96
NTP
576
22.65
2.83
SJD (Teng et al., 2024)
335 (-41.8%)
13.17 (-41.8%)
3.87
LlamaGen-XX XL XL (cfg=2.0)
ZipAR-16
423 (-26.6%)
16.46 (-27.3%)
2.87
ZipAR-14 14
378 (-34,4%)
14.89 (-34,3%)
3.16
ZipAR-12
331 (-41.8%)
13.17 (-41.8%)
3.67
Table 2: Quantitative results on diverse automatic evaluation approaches. Here, "NTP" denotes the next-token prediction paradigm. "ZipAR-n" denotes the ZipAR paradigm with a minimal window size of n. "Step" is the number of model forward passes required to generate an image.
Model
Method
Steps
VQAScore"
HPSv2f
Image Rewardf
Aesthetic Scoref
NTP
1024
0.6439
0.2647
-0.0818
5.38
75pAR-15
562
0.6534
0,2637
-0.0690
5.39
LlamaGen-XIL-512
ZipAR-11
451
0.6581
0.2630
-0.0982
5.37
ZipAR-7
324
0.6410
0,2625
-0.1683
5.33
ZipAR-3
185
0.6343
0,2599
-0.3121
5.32
NTP
2352
0.6579
0,2743
0.4164
6.10
ZipAR-20
1063
0.6595
0.2747
0.3971
6.13
Lumina-mGPT-766
ZipAR-17
915
0.6433
0,2732
0.3049
6.12
ZipAR-14
740
0.6589
0.2739
0.3646
6.10
ZipAR-11
588
0.6490
0,2730
0.2861
6.10
NTP
4160
0.6718
0.2762
0.4232
5.97
ZipAR-20
1331
0.6705
0,2761
0.3913
5.95
Lamina-mGPP-1022
ZipAR-17
1150
0.6797
0.2761
0.4018
5.94
ZipAR-14
964
0.6732
0,2747
0.3298
5.94
ZipAR-11
772
0.6723
0.2746
0.3222
5.95
consistently achieves lower FID than its fixed-window counterpart, which suggests that dynamically adjusting the window size based on token position and context enhances the fidelity generated images.
(NTP) paradigm and the proposed ZipAR framework over Emu3-Gen (Wang et al., 2024b) and Lumina-mGPT-7B (Liu et al., 2024a), as shown in Figures and 7. Notably, ZipAR can reduce the number model forward steps by up to for Emu3-Gen, while still producing high-fidelity images rich in semantic information.
4.3.2. IMPACT ON OPTIMAL SAMPLING HYPERPARAMETERS
As presented in Tables 4-5, we performed grid search to determine the optimal token-sampling hyperparameters, namely, sampling temperature and classifier-free guidance scale, for ZipAR. The results are shown below. These results indicate that ZipAR sampling does not alter the optimal sampling temperature and classifier-fres guidance scale.
5. Conclusion
In this paper, we have proposed ZipAR, a new parallel decoding framework designed to accelerate autoregressive visual generation. ZipAR leverages the spatial locality inher- ent in visual content and predicts multiple spatially adjacent visual tokens in single model forward pass, thereby sig- nificantly enhancing generation efficiency compared the traditional next-token prediction paradigm. An adaptive local window assignment scheme with rejection sampling is employed, ensuring that each token generated with
4.4. Qualitative Visualizations
next-token-prediction
In this subsection, we present non-cherry-pic picked visualiza- tions of images generated using the next-token prediction

---

ZipAR; Parallel Autoregressive Image Generation through Spatial Locality
Table 3: Quantitative evaluation on MS-COCO dataset. Here, "NTP" denotes the next-token prediction paradigm. "ZipAR-n" denotes the ZipAR paradigm with a minimal window size of n. "Step" is the number model forward passes required to generate an image. The latency measured with a batch size of
Model
Method
Step
Latency (s)
CLIP Score?
NTP
1024
33.17
0.287
SJD (Teng et al., 2024)
635 (-38.0%)
24.80 (-25.2%)
0.283
Z5pAR-15
562 (-45.1%)
18.98 (-42.7%)
0.287
LlamaGen-XL-5111
ZipAR-11
451 (-55.9%)
14.65 (-55.8%)
0.286
ZipAR-7
324 (-68.4%)
10.24 (-69.1%)
0.285
ZipAR-3
185 (-81.9%)
5.86 (-82.3% )
0.281
NTP
2352
91.70
0.313
SJD (Teng et al., 2024)
1054 (-55.2%)
60.27 (-34.2%)
0.313
Luming-mGPT-78--668
ZipAR-20
1063 (-54.8%)
63.28 (-31.0%)
0.314
ZipAR-17
915 (-61)
58.54 (-36.2%%
0.314
ZipAR-14
740 (-68.5%)
53.41 (-118%%
0.313
ZipAR-11
588 (-75.0%)
50.32 (-45.1 %))
0.312
8
Table 4: The performance of LlamaGen and ZipAR under different classifier-free guidance. Here, "** denotes results obtained from LlamaGen's 's paper.
Fixed Window Size
Adaptive Window Size
Model
Classifier- Guidance
FID]
4.74
1.75
FID
LlamaGen-L*
2.25
3.62
5
1.5
6.18
1.75
3.72
ZipAR-16
2.0
3.14
2.25
3.44
Table 5: The performance of LlamaGen and ZipAR under different sampling temperatures. Here, "** denotes the results obtained from LlamaGen's paper.
250
275
300
325 Step
350
375
400
Figure 6: Performance comparisons of ZipAR over class- conditional LlamaGen-L model with fixed window size and adaptive window size. Under similar step budget, ZipAR with adaptive window size always achieves lower FID.
Model
Temperature
FID.
0.96
3.53
3.24
LlamaGen-L
1.0*
3.07
3.14
0.96
3.35
sufficient contextual information. Extensive experiments demonstrate that ZipAR reduce the number of model forward steps by up 91% on the Emu3-Gen model with minimal impact on image quality.
0.98
3.25
ZipAR-16
1.0
3.14
1.02
3.34
In the future, we anticipate that integrating ZipAR with other methods that employ the sext-set-prediction paradigm, such as Medusa (Cai et al., 2024) and Jacobi decoding (Santilli al.. 2023), will further enhance acceleration ratios.
Impact Statement
The proposed ZipAR framework stands out for its high effi- ciency, which earry significant implications reducing the carbon emissions attributed to the widespread deployment of deep generative models. However, similar to other deep generative models, ZipAR has the potential to be utilized for producing counterfeit images and videos for malicious purposes.
Acknowledgements
This work was supported by the National Key Research and Development Program of China (2022YFC3602601) and the National Key Research and Development Program of China (20222)0160102).
8

---

ZipAR; Parallel Autoregressive Image Generation through Spatial Locality
Prompt: image of dog playing water, and water fall the background
Prompt: Image a magical fairy tale castle on a hilltop surrounded mystical forest
Prompt: bustling downtown street in Tokyo night, with neon signs, sidewalks, and tall skyscrapers
Prompt: serene beach sunset, with palm trees swaying the breeze and sailboat in the distance
Prompt: medieval knight standing guard front of a grand castle, with dragon flying overhead
Steps: 2355
Steps: 1508 (-36%)
Steps: 1132 (-52%)
Steps: 756 (-67%)
Figure 7: Samples generated by the Lumina-mGPT-7B- model with next-token prediction paradigm (the first column) and ZipAR under different configurations (the right three columns). The classifier-free guidance is set to 3.

---

ZipAR; Parallel Autoregressive Image Generation through Spatial Locality
References
LeCun, Y., Boser, B., Denker, J., Henderson, D., Howard, R., Hubbard, W., and Jackel, L.. Handwritten digit recog- nition with back-propagation network. NeurIPS, 1989.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding. In ICML, 2023.
Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J. D., Chen, D., and Dao, Medusa: Simple Ilm inference acceleration framework with multiple decoding heads, arXiv preprint arXiv:2401.10774. 2024.
Li, T., Tian, Y., Li, H., Deng, M., and He, K. Autoregres- sive image generation without vector quantization. arXiv preprint arXiv:2406.11838 2024.
Chang, H., Zhang, H., Jiang, L., Liu, C., and Freeman, W. T. Maskgit: Masked generative image transformer. In CVPR, 2022.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft coco:
Common objects in context. In ECCV, 2014.
Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama, J., Jiang, L.. Yang, M.-H., Murphy, K.. Freeman, W. T., Rubinstein, M., et al. Muse: Text-to-image genera- tion via masked generative transformers. arXiv preprint 3XXv:2301.007000 2023.
Lin, Z., Pathak, D., Li, B., Li, J., Xia, X., Neubig, G., Zhang, P., and Ramanan, D. Evaluating text-to-visual generation with image-to-tens to-text generation, 2024. URL ftteps//arxiie org/ wwwgg/ab////aa 2tt..///244.0010 01291.
Liu, D., Zhao, S., Zhuo, L., Lin, W., Qiao, Y., Li, H., and Gao, P. Lumina- mgpt: Illuminate flexible photorealis- tic text-to-image generation with multimodal generative pretraining. arXiv preprint arXiv:2408. 02657, 2024a.
Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.0133 0188, 2023.
Liu, X., Hu, I.., Bailis, P., Cheung, A., Deng. Z., Stoica, I., and Zhang, H. Online speculative decoding. In ICML, 2024b.
Chen, X., Wu. Z., Liu, X., Pan, Z., Liu, W. Xie, Z., Yu, X., and Ruan, C. Janus-pro: Unified multimodal un- derstanding and generation with data and model scaling, 2025.
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z.,, Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 2021.
Santilli, A., Severino, S., Postolache, E., Maiorca, V., Man- cusi, M., Marin, R., and Rodolå, E. Accelerating trans- former inference for translation via parallel decoding. arXiv preprint arXiv:2305. 10427, 2023.
Esser, P., Rombach, R., and Ommer, B. Taming transformers for high-resolution image synthesis. CVPR, 2021.
Fu, Y., Bailis, P., Stoica, I., and Zhang, H. Break the se- quential dependency of IIm inference using lookahead decoding, arXiv preprint arXiv:2402.02057 2024.
Spector, B. and Re, C. Accelerating Ilm inference with staged speculative decoding. arXiv preprint arXiv:2308.044 046233 2023.
Ge, Y., Zhao, S., Zhu, J., Ge, Y., Yi, K., Song. L., Li, C., Ding, X., and Shan, Y. Seed-x: Multimodal models with unified multi-granularity comprehension and generation. arXiv preprint ar 4404 10.0044.4439.. /4396, 2024.
Sun, P., Jiang. Y., Chen, S., Zhang, S., Peng, B., Luo, P., and Yuan, Z. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint 24Xiv:2406.06555, 2024.
Gui, L., Xiao, B., Su, L., and Chen, W. Boosting lossless speculative decoding via feature sampling and partial alignment distillation. arXiv preprint arXiv:2408. 15562, 2024.
Team, C. Chameleon: Mixed-modal early- fusion foundation models. arXiv preprint ArXiv:2405.0988. 2024.
Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac J.-B., Yu,
J., Soricut, R., Schalkwyk. J., Dai. A. M.. Hauth, A., et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
Kou, S., Hu, L., He, Z., Deng, Z., and Zhang, H. Clims: Consistency large language models. arXiv preprint arXiv:2403.00835, 2024.
Teng, Y., Shi, H., Liu, X., Ning, X., Dai, G., Wang. Y., Li, Z., and Liu, X. Accelerating auto-regressive text-to-image generation with training faeni speculative jacobi decoding. arXiv preprint arXiv:2410. 100::2400001999 2024.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. NeurlPS, 2012.
10

---

ZipAR; Parallel Autoregressive Image Generation through Spatial Locality
Tian, K., Jiang, Y., Yuan, Z., Peng, B., and Wang, L. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint arXiv:2404.02905. 2024.
Van Den Oord, A., Vinyals, O., al. Neural discrete repre- sentation learning. NeurlPS, 2017.
Vavekanand, R. and Sam, K. Llama 3.1: An in-depth analy- sis of the next-generation large language model, 2024.
Wang, C., Lu, G., Yang, J. Huang, R., Han, J., Hou, L., Zhang, W., and Xu, H. Illume: Illuminating your Iims to see, draw, and self-enhance. arXiv preprint arXiv:2412.06675 2024a.
Wang. X., Zhang, X., Luo, Z., Sun, Q. Cui, Y., Wang, J.. Zhang, F. Wang. Y., Li, Z., Yu, Q., et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409, 18869, 2024b.
Wu, C., Chen, X., Wu, Z., Ma, Y., Liu, X., Pan, Z., Liu, W., Xie, Z., Yu, X., Ruan, C., et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint arXiv:2410. 13848, 2024a.
Wu, J., Jiang, Y., Ma, C., Liu, Y., Zhao, H., Yuan, Z., Bai, S., and Bai, X. Liquid: Language models are scalable multi- modal generators. arXiv preprint arXiv:2412.04332, 2024b.
Wu, X., Hao, Y., Sun, K., Chen, Y., Zhu, F., Zhao, R., and Li, H. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09344, 2023.
Xu, J., Liu, X., Wu, Y., Tong, Y., Li, Q., Ding, M., Tang, J., and Dong, Y. Imagereward: learning and evaluat- ing human preferences for text-to-image generation. In NeurIPS, 2023.
Yu, J., Xu, Y., Koh, J. Y., Luong, T., Baid, G., Wang, Z., Va- sudevan, V., Ku, A., Yang, Y., Ayan, B. K., et al. Scaling autoregressive models for content-rich text-to-image gen- eration. arXiv preprint arXiv:2206.
2(3):5, 2022.
Yu, L.. Cheng, Y., Sohn, K., Lezama. J., Zhang, H., Chang, H., Hauptmann, A. G., Yang, M.-H., Hao. Y., Essa, I., et al. Magvit: Masked generative video transformer. In CVPR, 2023.
Zeiler, M. D. and Fergus, R. Visualizing and understanding convolutional networks. In ECCV, 2014.
Zhang, Z., Zhang, H., Zhao, L., Chen, T., Arik, S. Ö., and Pfister, T. Nested hierarchical transformer: Towards accu- rate, data-efficient and interpretable visual understanding.
In AAAI, 2022.


Reviews:
[{"review_id": "VS1xxhkKED", "reviewer": "Reviewer_ELf9", "summary": "This paper introduces a novel technique to conduct parallel decoding in AR-based image generation. The proposed approach can be directly applied to off-the-shelf pretrained AR-based image generation models, speeding up the generation with small performance drop.\n\n## update after rebuttal\nGiven the updated results with more evaluation metrics, I would like to keep my score of weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for the valuable comments.\n\n**Q1:More diverse automatic evaluation approach should be considered.**\nTo address this concern, we have expanded our evaluation by assessing ZipAR’s performance using multiple metrics, including VQAScore, Human Preference Score v2, ImageReward, and Aesthetic Score, across three models: LlamaGen-XL-512, Lumina-mGPT-768, and Lumina-mGPT-1024. The results presented below demonstrate that our method significantly improves generation efficiency with little impact on output quality across various benchmarks.\n| Model | Method | Steps | VQAScore | HPSv2 | Image Reward | Aesthetic Score |\n| ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| LlamaGen-XL | NTP | 1024 | 0.6439 | **0.2647** | -0.0818 | 5.38 |\n| LlamaGen-XL | ZipAR-15 | 562 | 0.6534 | 0.2637 | **-0.0690** | **5.39** |\n| LlamaGen-XL | ZipAR-11 | 451 | **0.6581** | 0.2630 | -0.0982 | 5.37 |\n| LlamaGen-XL | ZipAR-7 | 324 | 0.6410 | 0.2625 | -0.1683 | 5.33 |\n| LlamaGen-XL | ZipAR-3 | 185 | 0.6343 | 0.2599 | -0.3121 | 5.32 |\n| Lumina-mGPT-768 | NTP | 2352 | 0.6579 | 0.2743 | **0.4164** | 6.10 |\n| Lumina-mGPT-768 | ZipAR-20 | 1063 | **0.6595** | **0.2747** | 0.3971 | **6.13** |\n| Lumina-mGPT-768 | ZipAR-17 | 915 | 0.6433 | 0.2732 | 0.3049 | 6.12 |\n| Lumina-mGPT-768 | ZipAR-14 | 740 | 0.6589 | 0.2739 | 0.3646 | 6.10 |\n| Lumina-mGPT-768 | ZipAR-11 | 588 | 0.6490 | 0.2730 | 0.2861 | 6.10 |\n| Lumina-mGPT-1024 | NTP | 4160 | 0.6718 | **0.2762** | **0.4232** | **5.97** |\n| Lumina-mGPT-1024 | ZipAR-20 | 1331 | 0.6705 | 0.2761 | 0.3913 | 5.95 |\n| Lumina-mGPT-1024 | ZipAR-17 | 1150 | **0.6797** | 0.2761 | 0.4018 | 5.94 |\n| Lumina-mGPT-1024 | ZipAR-14 | 964 | 0.6732 | 0.2747 | 0.3298 | 5.94 |\n| Lumina-mGPT-1024 | ZipAR-11 | 772 | 0.6723 | 0.2746 | 0.3222 | 5.95 | \n\n**Q2: Ablation studies on whether ZipAR affects the optimal token-sampling-hyperparameters.**\nWe performed a grid search to determine the optimal token-sampling hyperparameters, namely, sampling temperature and classifier-free guidance scale, for ZipAR. The results are shown below. Here, \"*\" denotes the results obtained from LlamaGen's paper. These results indicate that ZipAR sampling does not alter the optimal sampling temperature and classifier-free guidance scale.\n\n| model | cfg | FID |\n| ---- | ---- | ---- |\n| LlamaGen-L* | 1.5 | 4.74 |\n| LlamaGen-L* | 1.75 | 3.15 |\n| LlamaGen-L* | 2.0 | **3.07** |\n| LlamaGen-L* | 2.25 | 3.62 |\n| ZipAR-16 | 1.5 | 6.18 |\n| ZipAR-16 | 1.75 | 3.72 |\n| ZipAR-16 | 2.0 | **3.14** |\n| ZipAR-16 | 2.25 | 3.44 |\n\n| model | Temperature | FID |\n| ---- | ---- | ---- |\n| LlamaGen-L | 0.96 | 3.53 |\n| LlamaGen-L | 0.98 | 3.24 |\n| LlamaGen-L* | 1.0 | **3.07** |\n| LlamaGen-L | 1.02 | 3.14 |\n| ZipAR-16 | 0.96 | 3.35 |\n| ZipAR-16 | 0.98 | 3.25 |\n| ZipAR-16 | 1.0 | **3.14** |\n| ZipAR-16 | 1.02 | 3.34 |", "claims_and_evidence": "Yes, claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation makes sense.", "theoretical_claims": "Not applicable.", "experimental_designs_or_analyses": "Yes, I have checked the soundness of all experimental designs and analyses. Overall, the experiments can validate the effectiveness of the proposed method. However, one issue is that this paper could benefit from more numerical results. Currently, only FID and CLIP-scores are provided.\n\nThe reviewer believe that some human evaluation results would enhance the significance of the paper. If human evaluation is not feasible, then at least more diverse automatic evaluation approach such as VQA-score [1], image reward [2] should be considered.\n\n[1] Evaluating Text-to-Visual Generation with Image-to-Text Generation. Lin. et al.\n[2] ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation. Xu. et al.", "relation_to_broader_scientific_literature": "The key contributions can be related to the autoregressive-based image generation models. These models are known for their low generation speed. The proposed approach could alleviate such problem, and thus incentivize more researchers to explore AR-based image generation.", "essential_references_not_discussed": "Related works are properly discussed.", "other_strengths_and_weaknesses": "Strength:\n1) The proposed algorithm is simple and can be applied without the need of retraining.\n2) The proposed approach is well-motivated and demonstrate promising results.\n\nWeakness:\n1) Please see \"Experimental Designs Or Analyses\".\n2) The paper could also benefit from more ablation studies or discussions. For example, the author could study/discuss whether the proposed approach affects the optimal token-sampling-hyperparameters such as sampling temperature or CFG scale.", "other_comments_or_suggestions": "Not applicable.", "questions_for_authors": "Please see weakness (1), (2)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hsztCizlN2", "reviewer": "Reviewer_TPry", "summary": "This paper proposes ZipAR, a training-free method to accelerate the decoding speed of the AR image generation model. They first show that significant attention scores are allocated to tokens in the same column of previous rows. Therefore, decoding the next row is not necessary to wait for the finishing of the last row. Based on this idea, they design the ZipAR method and use an adaptive window size to control the number of tokens in one step. The results show that this method can accelerate the generation speed without a significant performance drop.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for the valuable comments.\n\n**Q1: Essential reference VAR is not discussed.**\nAs noted in our related work section (lines 157-160 in the paper), we do discuss VAR and its approach to visual generation. Moreover, it should be noted that VAR requires specialized multi-scale tokenizers and must be trained from scratch as a complete generation framework. In contrast, our proposed ZipAR is a training-free, plug-and-play parallel decoding solution for existing vanilla (raster order) next-token autoregressive visual generation models without any architectural modifications or retraining.\n\n**Q2: Missing speed comparison with VAR and MaskGIT.**\nAs noted in Q1, ZipAR aims to accelerate existing vanilla next-token AR visual generation models, which is not directly comparable with VAR or MaskGIT. However, to address this concern, we have evaluated the generation efficiency of ZipAR, VAR and MaskGIT with similar model sizes. The results are presented below. Compared with vanilla next-token AR models, ZipAR greatly improves generation efficiency and narrows the efficiency gap with VAR and MaskGIT without any additional training.\n\n| Resolution | Model | Throughput (img/s) |\n| ---- | ---- | ---- |\n| 256x256 | LlamaGen-L | 40.9 |\n| 256x256 | ZipAR-11 | 47.0 |\n| 256x256 | ZipAR-7 | 58.1 |\n| 256x256 | ZipAR-3 | 80.8 |\n| 256x256 | MaskGIT | 120.0 |\n| 256x256 | VAR-d16 | 126.7 |\n| 512x512 | LlamaGen-L | 6.1 |\n| 512x512 | ZipAR-11 | 12.4 |\n| 512x512 | ZipAR-7 | 16.5 |\n| 512x512 | ZipAR-3 | 22.9 |\n| 512x512 | MaskGIT  | 50.8 |\n| 512x512 | VAR-d16  | 55.3 |\n\n**Q3: More benchmarks are needed to prove the robustness of ZipAR.**\nTo address this concern, we have expanded our evaluation by assessing ZipAR’s performance using multiple metrics, including VQAScore, Human Preference Score v2, ImageReward, and Aesthetic Score, across three models: LlamaGen-XL-512, Lumina-mGPT-768, and Lumina-mGPT-1024. The results presented below demonstrate that our method significantly improves generation efficiency with little impact on output quality across various benchmarks.\n\n| Model | Method | Steps | VQAScore | HPSv2 | Image Reward | Aesthetic Score |\n| ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| LlamaGen-XL | NTP | 1024 | 0.6439 | **0.2647** | -0.0818 | 5.38 |\n| LlamaGen-XL | ZipAR-15 | 562 | 0.6534 | 0.2637 | **-0.0690** | **5.39** |\n| LlamaGen-XL | ZipAR-11 | 451 | **0.6581** | 0.2630 | -0.0982 | 5.37 |\n| LlamaGen-XL | ZipAR-7 | 324 | 0.6410 | 0.2625 | -0.1683 | 5.33 |\n| LlamaGen-XL | ZipAR-3 | 185 | 0.6343 | 0.2599 | -0.3121 | 5.32 |\n| Lumina-mGPT-768 | NTP | 2352 | 0.6579 | 0.2743 | **0.4164** | 6.10 |\n| Lumina-mGPT-768 | ZipAR-20 | 1063 | **0.6595** | **0.2747** | 0.3971 | **6.13** |\n| Lumina-mGPT-768 | ZipAR-17 | 915 | 0.6433 | 0.2732 | 0.3049 | 6.12 |\n| Lumina-mGPT-768 | ZipAR-14 | 740 | 0.6589 | 0.2739 | 0.3646 | 6.10 |\n| Lumina-mGPT-768 | ZipAR-11 | 588 | 0.6490 | 0.2730 | 0.2861 | 6.10 |\n| Lumina-mGPT-1024 | NTP | 4160 | 0.6718 | **0.2762** | **0.4232** | **5.97** |\n| Lumina-mGPT-1024 | ZipAR-20 | 1331 | 0.6705 | 0.2761 | 0.3913 | 5.95 |\n| Lumina-mGPT-1024 | ZipAR-17 | 1150 | **0.6797** | 0.2761 | 0.4018 | 5.94 |\n| Lumina-mGPT-1024 | ZipAR-14 | 964 | 0.6732 | 0.2747 | 0.3298 | 5.94 |\n| Lumina-mGPT-1024 | ZipAR-11 | 772 | 0.6723 | 0.2746 | 0.3222 | 5.95 | \n\n**Q4: Sec 3.3 can be more informative.**\nTo clearly demonstrate the details of ZipAR, we have provided a pseudo code for ZipAR's sampling process. Please refer to Q4 in our response to Reviewer CC9X due to the character limit here. We will include it in the revised version.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "I think only evaluating text-to-image on MSCOCO with CLIP-Score is not robust enough. The quality results show no distinct difference between different models.", "theoretical_claims": "There is no proof of theoretical claims.", "experimental_designs_or_analyses": "1. Missing speed comparison with VAR and MaskGIT.\n2. Need more benchmark to prove the robustness of ZipAR", "relation_to_broader_scientific_literature": "Accelerating current AR image generation", "essential_references_not_discussed": "VAR published in NeurIPS 2024.", "other_strengths_and_weaknesses": "Sec 3.3 is a little hard to understand intuitively the details.", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MzDHyJBgP1", "reviewer": "Reviewer_CC9X", "summary": "This paper presents ZipAR, a training-free framework for accelerating autoregressive visual generation. It leverages the local structure of images by allowing parallel decoding of spatially adjacent tokens, alongside the standard next-token prediction. An adaptive local window assignment with rejection sampling ensures contextual alignment. This method significantly reduces the number of forward passes needed for image generation—by up to 91% on the Emu3-Gen model—without requiring retraining, thus enhancing efficiency in visual generation tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for the valuable comments.\n\n**Q1: The lack of experiments with 512 or higher resolution.**\nFor clarity, we would like to highlight that our experimental results already include higher-resolution evaluations, as referred to Table 2 in the paper. Specifically, the LlamaGen-XL model operates at 512x512 resolution and the Lumina-mGPT model operates at 768x768 resolution. To improve transparency, we will explicitly state these resolutions in the revised Table 2 caption. Moreover, to further address your point, we have conducted additional experiments using the Lumina-mGPT-1024 model at 1024x1024 resolution on various benchmarks. Due to the character limit here, please refer to Q1 in our response to Reviewer ELf9 for the evaluation results. These results demonstrate our method's effectiveness across multiple resolution scales.\n\n**Q2: Experiments are mainly implemented using the LlamaGen model.**\nAs referred to lines 299-300 in the paper, we integrate ZipAR with three state-of-the-art next-token AR visual generation models: LlamaGen, Lumina-mGPT and Emu3-Gen. Quantitative results can be found in Table 1-2 in the paper and the visualization results of these models can be found in Figures 1, 8 in the paper and Figures 9, 10 in the supplementary material. \n\n**Q3: Lack verification of other models, such as MAR.**\nIt should be noted that ZipAR is a training-free, plug-and-play parallel decoding framework for **vanilla next-token AR** visual generation models. However, MAR does not follow a next-token prediction generation paradigm. \n\n**Q4: Provide a brief pseudo code for the sampling process.**\nThanks for your valuable comment. We have provided a pseudo code for the sampling process, as shown below\n\n```python\n# Pytorch-style Pseudo Code for ZipAR Sampling Process\n\n# Image latent dimensions: H x W\n# Minimum window size: s_min\n\n# Initialize variables\ntotal_columns = W                           # Number of columns\ntotal_rows = H                              # Number of rows\ndecoding_rows = [0]                         # Rows actively being decoded\ndecoded_tokens = [[] for _ in range(H)]     # Decoded tokens for each row\npending_starts = []                         # Tentative new rows awaiting validation\n\nwhile decoding_rows or pending_starts:\n    # --- Step 1: Decode one token in each active row ---\n    for row in decoding_rows:\n        if len(decoded_tokens[row]) < total_columns:\n            # Decode next token using AR model\n            new_token = generate_token(row, len(decoded_tokens[row])) ## Generate token at position (row, len(decoded_tokens[row]))\n            decoded_tokens[row].append(new_token)\n\n    # --- Step 2: Process pending_rows\n    new_pending = []\n    for (new_row, old_token, old_prob) in pending_starts:\n        new_prob, new_token = speculative_generate(new_row, 0) ## Tentative generation of token at position (new_row, 0)\n        # Calculate acceptance probability\n        r = uniform_sample()\n        \n        if r < min(1, new_prob[old_token] / old_prob[old_token]): ## Eq. 2 in the paper\n            # Accept token and start decoding\n            decoding_rows.append(new_row)\n            decoded_tokens[new_row].append(new_token)\n        else:\n            # Resample from difference distribution\n            resampled_prob, resampled_token = resample_distribution(new_prob, old_prob) ## Eq. 3 in the paper\n            new_pending.append( (new_row, resampled_token, resampled_prob) )\n            \n    pending_starts = new_pending\n\n    # --- Step 3: Check for completed s_min-1 tokens to initiate new rows ---\n    for row in list(decoding_rows):\n        if len(decoded_tokens[row]) == s_min and \\\n           row+1 < total_rows and \\\n           row+1 not in decoding_rows and \\\n           row+1 not in [p[0] for p in pending_starts]:\n            # Generate tentative token with window size s_min\n            prob, pend_token = speculative_generate(row+1, 0) ## Tentative generation of token at position (row+1, 0)\n            pending_starts.append( (row+1, pend_token, prob) )\n\n    # --- Step 4: Cleanup completed rows ---\n    decoding_rows = [r for r in decoding_rows \n                        if len(decoded_tokens[row]) < total_columns]\n```", "claims_and_evidence": "The claims made in the paper are clear and supported by experiments.", "methods_and_evaluation_criteria": "Perhaps we can add experiments with ImageNet 512 × 512, because generating images with higher resolutions often requires more acceleration. In addition, we can add more AR models to the evaluation, such as MAR.", "theoretical_claims": "There is nothing wrong with the theoretical statement.", "experimental_designs_or_analyses": "I think the experiments are a bit too few, because for the accelerated experiments, it would be better to add higher resolution tests to prove the effectiveness of the method in high resolution experiments. In addition, it would be good if the method can be effectively applied to other AR models.", "relation_to_broader_scientific_literature": "The paper's use of rejection sampling and adaptive local window assignment draws from the concept of speculative decoding, which has been explored in natural language processing (NLP) to improve efficiency. Previous approaches, such as those used in language models, have shown that generating multiple tokens simultaneously can lead to significant efficiency gains. ZipAR applies this principle to visual generation, demonstrating that similar techniques can be effective beyond text.", "essential_references_not_discussed": "I think related work has been discussed in the paper.", "other_strengths_and_weaknesses": "I think the advantage of this paper is that it provides motivation for the acceleration of AR model reasoning through methods such as attention map visualization analysis. And the acceleration module designed in this paper is reasonable and the effect is significant.\n\nI think the disadvantage of this paper is mainly the lack of experiments with 512 resolution, which cannot illustrate the acceleration ability of the model at higher resolutions. In addition, the experiments in this paper are mainly implemented using the llama gen model, and lack verification of other models.", "other_comments_or_suggestions": "No other suggestions.", "questions_for_authors": "Please provide a brief pseudo code to facilitate understanding of the sampling process.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ncp883AX6u", "reviewer": "Reviewer_b59B", "summary": "The paper proposes ZipAR, a training-free plug-and-play decoding method for accelerating auto-regressive visual generation models. It decodes spatially adjacent tokens in the column dimension in parallel. It emplys an adaptive local window assignment scheme with reject sampling strategy. Experimental results demonstrate that ZipAR can decrease the number of required forward passes without compromising the quality of generation.\n\n## Update after rebuttal\n\nThe rebuttal satisfactorily addresses my concerns regarding the evaluation and theoretical justification. However, I remain concerned about the noticeable artifacts in the images. As a result, I am increasing my score to Weak Accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for the valuable comments.\n\n**Q1: Utilize additional metrics to fully evaluate the method.**\nTo address this concern, we have expanded our evaluation by assessing ZipAR's performance using multiple metrics, including VQAScore, Human Preference Score v2, ImageReward, and Aesthetic Score, across three models: LlamaGen-XL-512, Lumina-mGPT-768, and Lumina-mGPT-1024. The results presented below demonstrate that our method significantly improves generation efficiency with little impact on output quality across various benchmarks.\n\n| Model | Method | Steps | VQAScore | HPSv2 | Image Reward | Aesthetic Score |\n| ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| LlamaGen-XL | NTP | 1024 | 0.6439 | **0.2647** | -0.0818 | 5.38 |\n| LlamaGen-XL | ZipAR-15 | 562 | 0.6534 | 0.2637 | **-0.0690** | **5.39** |\n| LlamaGen-XL | ZipAR-11 | 451 | **0.6581** | 0.2630 | -0.0982 | 5.37 |\n| LlamaGen-XL | ZipAR-7 | 324 | 0.6410 | 0.2625 | -0.1683 | 5.33 |\n| LlamaGen-XL | ZipAR-3 | 185 | 0.6343 | 0.2599 | -0.3121 | 5.32 |\n| Lumina-mGPT-768 | NTP | 2352 | 0.6579 | 0.2743 | **0.4164** | 6.10 |\n| Lumina-mGPT-768 | ZipAR-20 | 1063 | **0.6595** | **0.2747** | 0.3971 | **6.13** |\n| Lumina-mGPT-768 | ZipAR-17 | 915 | 0.6433 | 0.2732 | 0.3049 | 6.12 |\n| Lumina-mGPT-768 | ZipAR-14 | 740 | 0.6589 | 0.2739 | 0.3646 | 6.10 |\n| Lumina-mGPT-768 | ZipAR-11 | 588 | 0.6490 | 0.2730 | 0.2861 | 6.10 |\n| Lumina-mGPT-1024 | NTP | 4160 | 0.6718 | **0.2762** | **0.4232** | **5.97** |\n| Lumina-mGPT-1024 | ZipAR-20 | 1331 | 0.6705 | 0.2761 | 0.3913 | 5.95 |\n| Lumina-mGPT-1024 | ZipAR-17 | 1150 | **0.6797** | 0.2761 | 0.4018 | 5.94 |\n| Lumina-mGPT-1024 | ZipAR-14 | 964 | 0.6732 | 0.2747 | 0.3298 | 5.94 |\n| Lumina-mGPT-1024 | ZipAR-11 | 772 | 0.6723 | 0.2746 | 0.3222 | 5.95 | \n\n**Q2: Show some qualitative comparisons of different window sizes.**\nWe have provided qualitative visualizations of different window sizes, as referred to Figures 1, 8 in the paper and Figures 9, 10 in the supplementary material. Since we can not update PDF version in the current phase, we will add more qualitative comparisons of different window sizes in the revised version.\n\n**Q3: The effectiveness of speculative decoding in identifying short local window sizes that lead to insufficient information.**\nLet $p_s$ and $p_{s+1}$ be the token distributions for window sizes $s$ and $s+1$. As referred to Eq. 2 in the paper, the acceptance probability for a candidate $x_s \\sim p_s$ is defined as \n$$\n\\alpha(x_s)= \\min\\left(1, \\frac{p_{s+1}(x_s)}{p_s(x_s)}\\right)\n$$ \n\nThe expectation of the acceptance can be formulated as:\n$$\n\\mathbb{E}_{x \\sim p_s}\\left[\\alpha(x_s)\\right]\n$$\n\n$$\n= \\sum_x p_s(x) \\min(1, \\frac{p_{s+1}(x)}{p_s(x)}) = \\sum_x \\min(p_s(x), p_{s+1}(x))\n$$\n\n**Theorem 1** (Relationship between Pairwise Minimum and Total Variation).  \nLet $p$ and $q$ be two probability distributions over the same discrete support $\\mathcal{X}$. The sum of their element-wise minima satisfies:  \n$$\n\\sum_{x \\in \\mathcal{X}} \\min(p(x), q(x)) = 1 - \\text{TV}(p, q),  \n$$  \nwhere $\\text{TV}(p, q) = \\frac{1}{2} \\|p - q\\|_1$ is the total variation distance between $p$ and $q$.\n\n**Proof**  \n\nFor any $x \\in \\mathcal{X}$, observe that:  \n$$\n\\max(p(x), q(x)) + \\min(p(x), q(x)) = p(x) + q(x).  \n$$  \nSumming over all $x$ yields:  \n$$\n\\sum_{x} \\max(p(x), q(x)) + \\sum_{x} \\min(p(x), q(x)) = \\sum_{x} p(x) + \\sum_{x} q(x) = 2.  \n\\quad (1)  \n$$\n\nBy definition of the L1-norm, we have:  \n$$\n\\|p - q\\|_1 \n$$\n\n$$\n= \\sum_{x} \\|p(x) - q(x)\\| \n$$\n\n$$\n= \\sum_{x} \\max(p(x), q(x)) - \\sum_{x} \\min(p(x), q(x)).  \n\\quad (2)  \n$$  \n\nLet $S_{\\min} = \\sum_{x} \\min(p(x), q(x))$ and $S_{\\max} = \\sum_{x} \\max(p(x), q(x))$. \nThen Equation (1) can be reformulated as:  \n$$\nS_{\\max} = 2 - S_{\\min}. \\quad (3)  \n$$  \n\nBy substituting Equation (3) into (2), we have:  \n$$\n\\|p - q\\|_1 \n$$\n\n$$\n= (2 - S_{\\min}) - S_{\\min} \n$$\n\n$$\n= 2 - 2S_{\\min}.  \n$$  \nRearranging and using $\\text{TV}(p, q) = \\frac{1}{2} \\|p - q\\|_1$:  \n\n$$\nS_{\\min} = 1 - \\text{TV}(p, q). \\quad\n$$  \n\nTherefore, the expected acceptance rate is formulated as:\n\n$$\n\\mathbb{E}_{x \\sim p_s}\\left[\\alpha(x_s)\\right] = \n$$\n\n$$\n\\sum_x \\min(p_s(x), p_{s+1}(x))=1-\\text{TV}(p_{s+1},p_s) \\quad (3)\n$$\n\nIf $s$ is insufficient, the distributions $p_s$ and $p_{s+1}$ diverge significantly, implying $\\text{TV}(p_s, p_{s+1}) \\geq \\Delta$ for a threshold $\\Delta > 0$. By (3), the expected acceptance rate is upper bounded by:  \n$$\n\\mathbb{E}[\\alpha] \\leq 1 - \\Delta.\n$$  \nA low acceptance rate ($\\leq 1 - \\Delta$) prompts the algorithm to increase the window size to $s+1$.\n\nIf $s$ is sufficient, $p_s$ and $p_{s+1}$ are statistically indistinguishable ($\\text{TV}(p_s, p_{s+1}) \\approx 0$). By (3), the expected acceptance rate approaches 1:  \n$$\n\\mathbb{E}[\\alpha] \\geq 1 - \\epsilon \\quad (\\epsilon \\approx 0),\n$$  \nallowing immediate sampling from $p_{s+1}$ without window expansion.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "1. I think the evaluation is not enough. Authors only use CLIP to evlauate the quality of text-to-image generation, which fails to evaluate the image appearance. I would suggest authors to consider Aesthetic Score, Human Preference Score v2, and ImageReward to fully evaluate the method.\n\n2. I see that ZipAR with diverse window sizes (for example from 3 to 15 in LlamaGen-XL) achieves similar CLIP score. Can you also show some qualitative comparison to see if different window sizes will lead to different behaviors?", "theoretical_claims": "A theoretical proof demonstrating the effectiveness of speculative decoding in identifying short local window sizes that lead to insufficient information is currently lacking.", "experimental_designs_or_analyses": "I check the soundness/validity of experimental designs and analyses. Please find my concerns in Evaluation Criteria setting.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "As the number of steps decreases, some artifacts become noticeable in Fig. 8. Does ZipAR enhance efficiency at the cost of visual quality? I recommend that the authors utilize additional metrics, as outlined in the Methods and Evaluation Criteria section, to further investigate this trade-off.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors propose ZipAR, a training-free parallel decoding framework for accelerating autoregressive visual generation by leveraging parallel decoding of spatially adjacent tokens. The effectiveness of the proposed method is validated with three state-of-the-art autoregressive visual generation models (LlamaGen, Lumina-mGPT, and Emu3-Gen).

Initially, the paper received mixed scores. The reviewers were mainly concerned about the lack of more evaluation metrics, clarification of large resolution experiments, and speed comparisons with VAR and MaskGIT. The provided rebuttal effectively resolved the reviewers' concerns (with one minor unaddressed concern from Reviewer b59B regarding the artifacts in the generated results). As a result, the paper received all accept recommendations. After considering the author rebuttal and reviewer discussion/comments, the area chair agrees with this recommendation. The authors are encouraged to incorporate the rebuttal and reviewer suggestions to further improve the final version of the paper.
Decision: Accept (poster)