Hawk: Leveraging Spatial Context for Faster Autoregressive Text-to-Image Generation
Authors: Zhi-Kai Chen
Jun-Peng Jiang
Han-Jia Ye
De-Chuan Zhan
Venue: NeurIPS 2025 poster ()
Keywords: Autoregressive image generation
Inference acceleration
Spatial speculative decoding
Primary area: deep_learning

Abstract: Autoregressive (AR) image generation models are capable of producing high-fidelity images but often suffer from slow inference due to their inherently sequential, token-by-token decoding process. Speculative decoding, which employs a lightweight draft model to approximate the output of a larger AR model, has shown promise in accelerating text generation without compromising quality. However, its application to image generation remains largely underexplored. The challenges stem from a significantly larger sampling space, which complicates the alignment between the draft and target model outputs, coupled with the inadequate use of the two-dimensional spatial structure inherent in images, thereby limiting the modeling of local dependencies. To overcome these challenges, we introduce Hawk, a new approach that harnesses the spatial structure of images to guide the speculative model toward more accurate and efficient predictions. Experimental results on multiple text-to-image benchmarks demonstrate a 1.71× speedup over standard AR models, while preserving both image fidelity and diversity.

Hawk: Leveraging Spatial Context for Faster Autoregressive Text-to-lmage Generation
Zhi-Kai Chen Jun-Peng Jiang! Han-Jia Ye
De-Chuan Zhan 1.+
School of Artificial Intelligence, Nanjing University, China National Key Laboratory for Novel Software Technology. Nanjing University, China jchenz,,jaangip yehj, zhande) @ lamdaanju.eedu.cn
Abstract
Autoregressive (AR) image generation models are capable of producing high- fidelity images but often suffer from slow inference due to their inherently sequen- tial, loken-by-token decoding process. Speculative decoding, which employs lightweight draft model to approximate the output of a larger AR model, has shown promise in accelerating text generation without compromising quality. However, its application to image generation remains largely underexplored. The challenges stem from significantly larger sampling space, which complicates the alignment between the draft and target model outputs, coupled with the inadequate use of the two-dimensional spatial structure inherent in images, thereby limiting the modeling of local dependencies. To overcome these challenges, we introduce Hawk, a new approach that harnesses the spatial structure of images guide the speculative model toward more accurate and efficient predictions. Experimental results on multiple text-to-image benchmarks demonstrate a 1.71x speedup over standard AR models, while preserving both image fidelity and diversity.
Introduction
The field of image generation has witnessed significant progress [37 20 45 especially with the emer-
gence of autoregressive (AR) models that demonstrate strong capabilities generating high-quality images [48] These models have been applied in various domains, including super-resolution [3]
image completion
43
style transfer 77. image segmentation
28 and image editing (
Despite the impressive performance of autoregressive (AR) models in generating high-fidelity images, their slow inference speed remains significant obstacle for real-world deployment. This limitation primarily stems from their inherently sequential nature-generatinn images token by token-which leads low throughput and high latency. While existing approaches, such non-autoregressive generation methods [14] 20, 38 and model compression techniques like pruning 44] 25 or quantiza- tion 22 50), can accelerate inference, they often do the cost of degraded image quality. This raises the question: is it possible to speed up inference without compromising generation fidelity?
Speculative decoding has already demonstrated its effectiveness in accelerating autoregressive text generation models without compromising output quality 19 33 12 10). This technique employs two-model framework: lightweight draft model first generates sequence of tokens, which then partially or fully verified by a larger, high-quality target model. the target model confirms the draft's predictions, multiple tokens can be accepted in single step, thus reducing the number of sequential forward passes and accelerating inference. Given its success in the text domain, is natural to consider applying speculative decoding AR image generation means achieve faster inference without sacrificing image quality.
*Corresponding author, email: zhande @wwddalllaamauuuu.....
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Although AR-based text and image generation share many underlying principles, key differences between the two domains hinder the direct transfer of speculative decoding techniques to image generation. First, the significantly larger sampling space in image generation presents key challenge for applying speculative decoding. To capture fine-grained visual details, image generation models typically sample 20 to 400 times more candidates from the vocabulary than text models [35] 23 (4), introducing greater variability in token prediction. This makes it much harder for the draft model to align with the target model, limiting the effectiveness of speculative decoding when applied directly,
From another perspective, unlike sequential text, images are naturally two-dimensional, exhibiting complex spatial dependencies that are largely absent in text generation. Our iieention-sinking experiments [47] in Section revea that the generation of a pixel-level token influenced not only by its horizontal neighbors within the same row, but also by vertical dependencies across rows. These findings suggest that both horizontal and vertical spatial contexts provide informative signals, which can be effectively leveraged to guide and accelerate AR image generation. However, existing speculative decoding algorithm designed for text tends to overlook the potential vertical information, and are not well suited to capture two-dimensional spatial context.
To address the aforementioned challenges, we introduce Hawk, new framework based on Spatial Speculative Decoding that leverages the two-dimensional spatial structure of images to accelerate autoregressive (AR) image generation. By explicitly incorporating spatial dependencies both horizontal and vertical Hawk significantly improves alignment between the draft and target models. This not only expands the effective sampling space but also enables more informed speculative predictions, leading to faster generation without compromising image fidelity.
Specifically, Hawk introduces a pair of draft heads that independently speculate in both horizontal and vertical directions. Given the inherent asymmetry between these two spatial dimensions, the horizontal and vertical heads produce complementary speculative results, partieularly enhancing diversity in edge contours and central image regions. We then integrate these outputs to construct a unified spatial sampling space, which serves as the foundation for candidate token selection. These candidates are subsequently verified by the target model. Empirically, we find that this merged spatial space aligns more closely with the target model's ' sampling distribution, improving the acceptance rate of speculative tokens. By enabling multiple tokens to be generated and validated in a single pass, Hawk effectively speeds up inference without compromising generation fidelity. Experimental results on multiple text-to-image benchmarks demonstrate a 1.71x speedup over standard AR models. The main contributions of this work are summarized as follows:
We highlight the importance of accelerating image generation without compromising output quality, and analyze the unique challenges specific to image generation with speculative decoding.
We propose Hawk, a novel strategy that combines speculative decoding with spatial information derived from the attention mechanism. To the best of our knowledge, this is the first work to leverage spatial information for accelerating autoregressive image generation.
By integrating optimized Draft Heads and Spatial Speculative Decoding into large-scale text-to- image model, Hawk achieves 1.71x speedup without compromising output quality.
2 Related Works
Autoregressive Text-to-Image Generation. Autoregressive (AR) models were initially developed for time series tasks. However, subsequent research revealed that AR models could also applied to spatial data, such as image generation [II. Early works, including PixeIRNN [4] and PixeICNN 40). leveraged RNNs and CNNs to perform
image inference. However, this approach had its limitations. Later, methods such as vector quantization [11] were introduced to convert images into discrete tokens. Early approaches like VQ-VAE (42 explored this concept, and as the capabilities of autoregressive models became more apparent, research shifted toward developing more advanced models based on the autoregressive framework. Related works include 9 30 49).
In the era of large language models, multimodal generation, which combines both text and image generation, has become one of the key capabilities of large models. Chameleon (35 was proposed as a multimodal model capable of generating both images and text. Lumina-mGPT 2 is a fine-tuned model based on Chameleon. However, due to the reliance on AR inference, these methods inherently suffer from slow inference speeds.
2

---

Speculative Decoding, Speculative Decoding was initially proposed as an optimization strategy 13 [5] and later widely adopted as an acceleration technique in large language model inference [9] [
It achieves acceleration while maintaining output quality, making it a promising method for fast inference. Building on the speculative framework, several enhancements have been proposed to address the challenge of difficulty in obtaining drafter models. For instance, Medusa 4 and Eagle [2I] introduced mechanisms such as draft heads, enabling the model to perform speculative generation and verification simultaneously within single inference pass. Additionally, tree-based attention mechanisms [26] have been employed to increase the number of sampling candidates, enhancing the success rate of speculative decoding.
Further advancements include works like Hydra [ and KOALA [5]]. which optimize the training of draft heads to enhance speculative decoding performance. Due to the lack of spatial information utilization in image generation, these acceleration algorithms did not achieve the expected perfor- mance gains. In our work, we adopt a similar draft head structure to that in Medusa, enhancing its effectiveness by incorporating spatial information.
Autoregressive Image Generation Acceleration. The use of speculative decoding to accelerate image generation remains an underdeveloped area, with no standardized methodology across existing works. Different approaches have been proposed in the literature. For example, the ZipAr [14] adopts a aoo-eutoreer generation strategy, avoiding the traditional raster scan order. In contrast, the LANTERN++ (18) employs a more relaxed speculative decoding scheme, increasing the acceptance probability of the small model's speculative outputs. The SJD 36 integrates speculative sampling with Jacobi decoding to achieve acceleration.
In Hawk method, propose spatial speculative sampling method. While preserving the autoregressive inference paradigm, we adopt the original speculative decoding strategy to maintain sampling performance, thereby achieving acceleration without compromising the model's accuracy.
3 Preliminary
Autoregressive Image Generation. An Autoregressive (AR) model follows a next-token prediction process, where the prediction of x4 depends only its preceding tokens T2..,2,.... 1t-1)) The probability of generating a sequence of discrete tokens can be formulated as: p(r1, I2, ZT) =
Ti=1((a .,,
In the context of image generation, vector quantization is employed to convert an image into a sequence of tokens. Vector quantization consists of an encoder and quantizer. The encoder maps an image of shape x Rhxwx3 into a latent representation 2 kxxxxxd, where dl is the dimension of the latent space. Formally, this can be written as E(x). In the process of computing the quantizer, we use a pretrained codebook = {*x |= consisting of K elements, each of dimension d. The quantizer then searches for the closest codebook vector for each feature element Zig. Specifically, for each feature point, the quantizer computes: =g = arg minz, ||= =x|l. where zę is a vector the codebook Z. The quantizer's 's result is then derived by combining the indices of the nearest neighbors in the codebook for each element in the feature map.
Subsequently, 2D n X image tokens are reshaped into a ID sequence of length n², enabling next-token generation. As result, AR-based image generation typically follows raster scan order, which involves sequential inference of the image, row by row and column by column. The most common pattern is from left to right and top to bottom.
Speculative Sampling. The core of speculative decoding lies in its use of the speculative sam- pling strategy sample and validate the results generated by the draft model. Suppose the target model has sampling distribution Prarger (x) and the draft model has a sampling distribution Parate Parate(afx ((())) where represents the current inference step. The process consists of two phases: drafting and verification. In the drafting phase, we use the draft model to generate 7 E Z tokens from the draft sampling space P Parart (xxe However, these tokens are not directly reliable. In the verification phase, we use single forward pass of the target model to verify the speculative tokens + 1: t + in parallel. For each speculative token, the target model has probability of min (1, Pearget (z) /Pt///aaaeeeaee (z)) to accept the token. If no speculative tokens are accepted, the target
model resamples the token with the probability Plarger (z) norm max(0, Ptarget (z) Paratt (z))
3

---

Utilization of Draft Heads Structure. Since speculative sampling relies on smaller model for speculation, acquiring such model can be challenging. The smaller model must maintain good consistency with the target model. However, not all models have smaller parameter versions available. In this context, the draft heads structure 4 serves as an alternative the draft model. Draft heads refer to additional decoding heads incorporated after the final hidden layers to generate speculative outputs. In the Hawk method, we employ spatial draft heads: horizontal draft heads predict the token at position (T' + Horizontal Depth), while vertical draft heads predict the token at position (I++ +ertical Depth X Width). Here, T represents the current position in the autoregressive (AR) decoding process, Horizontal Depth refers to the horizontal speculative distance, and Vertical Depth refers to the vertical speculative distance. By leveraging the draft heads structure, we eliminate the need for separate, smaller speculative model, allowing to complete both the draft and verification phases in a single forward pass.
Attention Sinking. The concept of "attention sinking" was first introduced in the context of large language decoding [47. In their study, the authors observed that during the inference process of an autoregressive model, the model's attention tends to concentrate on the first token and a few tokens close to the current inference point. This indicates that the first token and its neighboring tokens contain critical information for the model's reasoning. The loss of this information can lead to a significant decline in the model's performance during inference. To better understand the mechanism of attention sinking in image generation, introduce two key concepts that will be referenced throughout this paper: siference-neighbone Inference-neighhorinn neighboring points are defined as the set of points that are sequentially adjacent to the current point in raster order (e.g. IT-1, -2, -3, TT- Spatial-neighhoring points, on the other hand, are defined in terms of the geometric structure of the image. These include the points spatially
points and spatial-neighhong
points.
adjacent to the current point, such as those directly above or to the left of it in the image space (e.g.
-T-1, -mag width, 22, JT -2 Image width).
4 Spatial Speculative Decoding
In this section, first analyze the unique characteristics of the attention sinking phenomenon in autoregressive image generation and observe its reliance on spatial information. We then introduce our proposed Hawk method.
4.1 Preliminary Experiments
To analyze the differences in inference depen- dencies between image and text autoregressive generation, we conduct an attention sinking ex- periment to compare the attention focus of differ- ent models during the generation process. This analysis helps us identify potential strategies for inference accelcration. Our experiment com-
Lumina-mGPP
ox5-
042
ANNPRN www. Atantoon Logit
18
182
2300
200
12.00
1220
2540
pares the average attention logits across different models to highlight their attention patterns dur- ing generation. For the Lumina-mGP1 model, we generate a 48 x 48 image and examine the av- erage attention logits at its central point. For the text model, we use LLaMA 2 [39], instructing it to generate the same number of tokens as the image central point while observing the average attention logits at the most recently generated token. Our findings reveal that, in image gen- eration, attention is not solely concentrated on inference-neighboriin neighboring points but also extends to Spatial-ned neighboring points within the image space. Figure I illustrraess this relationship.
5000
1158
11.0
Postton
Figure 1: The difference in average attention log- its between image generation and text generation, displayed with Lumina-mGPT in a 1D raster or- der (top), LlaMA 2 (bottom). The average atten- tion logits of the image generation model exhibit a strong dependency The red box highlights the previous row the im- age generation process.
spatially neighboring points,
The figure clearly shows that in text generation, attention primarily focuses on inference-neighhoring points, with attention scores dropping sharply after approximately five tokens from the current

---

One Step in Model Inferencing
Update Vertical Spaculation Cache
Spatial Speculative Decoding
irage (Mertcal
Inflvanee
(Row 2
Varification
shrough cath canditaus crmned in Skop? lee tyundkttre te (a
Tx2
(Roc- 13 Depth
Spacolltion
- (pp I]
crrpling poot
Vedical Specolation
Verical Dapth cache
Kix wertnst sangying poot Iven
ifr oole RagelCC)
entece pas ventfronton
oade
Raticced for T Spoccaation
troa
Adjat targal space rejucted
Po Poraeto) narri Paj Oe
Generate Speculation Candidates
sca Rapeliill Pil
Canditate
vortccatio
Cpecuaaion
Gurnr
dbs
imag
Aerioal
Condraas
dionengl Seaar and carel spiies
IFAR Cancilan Rejeced
Taur Parrl))
Sumpling pool
Sampping
Samplnng
Sumpling pool
Figure 2: An overview of our Hawk method is presented. During each iteration of the inference pro- cess, horizontal and vertical speculations are generated using the draft head. The vertical speculations stored in the Speculation Cache for future use when processing subsequent lines. Meanwhile, the horizontal speculations are combined with the previous vertical speculations to create the speculation sampling pool. From this pool, tree decoding candidates generated, followed by a verification step akin to tree speculative decoding.
point. In contrast, for image generation, attention extends beyond the immediate tokens to include spatially adjacent tokens from previous rows, resulting attention logits with multiple peaks. The red box in the figure highlights the previous row of the image generation. Notably, when inferring the current row, the average attention logits from the previous row surpass those from the current row, suggesting that attention from the preceding row plays a crucial role in image autoregressive inference, even contributing to the majority of the attention. This phenomenon underscores the distinctive dependencies in image generation compared to text generation.
Speculative sampling employs speculation heads to predict future tokens, but these heads often result in a performance drop compared to the original model's inference. We attribute this performance degradation to insufficient attention information. For text models, key attention information is primarily concentrated around the current inference point. In text-based speculative generation, when attempting predict tokens farther away, much of the critical attention information lost, leading to poor performance. In contrast, for image generation models, important information also resides in the spatially adjacent regions around the speculative point. Therefore, even when speculating on distant points (e.g.. T + Image Width), the existing inference information from the current row (highlighted by the red box our attention sinking experiment) helps mitigate the speculation loss. As result, the draft heads can still maintain relatively good performance despite the absence of some neighboring information. Additional experiments on the attention sinking phenomenon are provided in Appendix B
4.2 Enhancing Speculative Decoding with Spatial Information
Analysis. To achieve better speculative inference acceleration, we aim to expand the sampling space of the draft model, increasing its candidate selection options. With a larger sampling space, the likelihood of smaller model successfully aligning with the sampling distribution of larger model improves. Simply increasing the number of candidates for speculative decoding in a mie-iimnniional manner does not inherently enhance sampling diversity, as its effectiivnness remains constrained by the limited information available within a single-dimensional space.
Our preliminary experiment demonstrates that directly speculating on tokens below can yield promis- ing results. A potential approach to expand the sampling direction of draft heads to introduce additional draft heads in speculative sampling, enabling the prediction of the next row or even multiple subsequent rows. By adopting this approach, we effortlessly double the sampling space dimension of the draft model.
5

---

Specifically, our strategy consists of three key steps: First, we use dual-direction draft heads to generate an additional sampling space and cache the vertical sample space for use in subsequent inferences. During the speculation process, we combine the cached vertical sampling space from the previous row with the horizontal sampling space at the current speculative point to form a spatial sampling pool, from which we generate our speculative candidates. In this generation process, a tree structure is employed to expand the number of candidates that can be verified simultancously. Finally, speculative sampling and tree decoding are applied to verify whether the forward pass produces an acceptable speculative result. Figure 2 provides an overview of the method.
Dual Direction Drafting Heads. To more effectively explore the target model's 's sampling space, we believe incorporating spatial information is crucial. We use Dual Direction Drafting Heads to simultaneously speculate in both vertical and horizontal directions, thereby expanding the sampling capacity of our draft heads and improving the success rate of our speculation. Let represent the current inference position along the horizontal axis (e.g.. the current image token in a row). Instead of speculating horizontally on positions T + 1, T + 2, ..... + n, where n is the horizontal speculation length, Hawk also makes predictions for positions further down the vertical axis. Specifically, we introduce an additional drafter heads that target positions of the form:
T + IW X VSD
Where Vertical Speculation Depth (VSD) refers to the number of rows below the current position that explored, and Image Width (IW) represents the number of tokens per row in the generated image. In practical usage, we need to follow the raster order for image generation. As a result, the outputs from the vertical draft heads are not utilized immediately; instead, we maintain cache of size
IW x (VSD X (VSD + 1)) /
(2)
to store vertically speculative predictions, which will be utilized in the generation of vertical sampling space for subsequent inferences.
Spatial Sampling Pool. To optimize our draft sampling space and better align it with the target sampling space, we build candidate sampling pool using wwo-dimensional speculating information at each speculation point. We reuse vertical predictions already computed from cache when speculating horizontally to point T + n in the candidates generation step.
When the model computes speculative outputs for T + n, it utilizes vertical predictions from up to Vertical Speculation Length subsequent rows at T + n IW x A. position, where A. represents the subsequent row number. The prediction results at these positions will be combined to form our vertical sampling space.
VertSpec(T + +) (Cache(T + n., Av. d) | dd= A.. A. E (1,2,..., VSD))
(3)
Here, d represents the depth index of the vertical speculation. Specifically, for the vertical speculation results of the previous kes row, we utilize the speculative result with a depth of k at that position. Although these speculative results have different speculation depths and come from different previous rows, all of them point to the same current inference point. These predictions are retrieved from the cache, which we maintained in the previous inference. For constructing the horizontal sampling space, we directly use the speculation from the current point to the T + n point.
HT n) = ( J S T)
When speculation point T n, we combine the vertical sampling space and the horizontal sampling space form wwo-dimensional spatial sampling pool of next-step candidates. We merge:
HorizSpec(T + n)) + VertSpec(T +
(5)
This merged set forms the new candidate pool at the speculation location T + n, incorporating the candidate tree expansion. The candidate tree is generated using the Cartesian product of the sampling results from the sampling pool at each speculation position. By adopting this approach, each newly expanded node in the candidate tree benefits from both the current horizontal speculation and the cached vertical speculation, enabling a more comprehensive exploration of the image space and resulting in richer, more diverse candidate generation.
Sampling and Verification. The core principle of speculative decoding lies in preserving the distribution of the original model. Our algorithm focuses validating this design through

---

verification-baset decoding process. Specifically, we employ tree-structured decoding strategy (3tep-3 in Figure 2 to generate and verify speculative drafts.
For each draft result, candidates at the same depth (e.g., 1+1, 1+2, (++) are verified sequen- tially. If candidate passes verification, decoding proceeds the next depth; otherwise, the target distribution is updated iteratively. Specifically, if candidates are consecutively rejected, we perform k updates:
(++1) Ptarget (x) norm( max(0, Ptarget (1) Qarart (z))), i= i=0,...,k-1
(6)
and resample from the last refined distribution Piarget. where qarart may come from either the horizontal or vertical draft head. Despite involving two distinct draft sampling distributions, our approach still preserves the statistical behavior of the original model (see Appendix [ for proof).
(k)
Formally, our final formulation for approximating the original distribution is expressed as:
((x)=
Σ
a+b
II (1 (1-05(0)) ((()) ar(x))
o
+
II (I -o((a))))
a+b
Presia(a)
where a and denote the numbers of samples drawn from the two sampling spaces, and a represents the acceptance rate at each verification step.
Here, (resia()) (2) denotes the residual probability of failure after all candidates have been evaluated. In conventional speculative sampling, multiple candidates are drawn from single latent space, yielding identical candidate distributions. As a result, when all candidates fail verification, the residual probability (I)) cannot be further reduced. In contrast, by introducing multiple sampling heads that operate on distinct spaces, our approach diversifies the verification distributions, thereby mitigating the residual probability and improving overall sampling efficiency.
5
Experiment
Model and Benchmark. We evaluate our proposed Hawk method on the Mumina-mGGP" model [33 For testing, we generate 768 x 768 images and use top-k sampling with = 2000 and temperature of 1.0. As for how these parameters affect the generated images, we provide a discussion in Appendix We also employ classifier-free guidance [ with a guidance scale of 3.0. For the benchmark, we
use the test sets from COCO 2017 [22] and Flickr30K [29]. From these datasets, sample 500 examples from each for evaluation.
Metrics. We evaluate inference acceleration using two metrics: Inference Acceleration Rate (original inference time l'accelerated inference time), where higher better, and Accept Length, the average number of successful speculative predictions per step. For non-speculative baselines, the accept length 1. To evaluate image quality, we employed two widely used metrics. The FID [I7] score measures the similarity between the generated images and real images, where lower scores indicate better quality. Additionally, the CLIP [6] score assesses the semantic alignment between the generated images and their corresponding text prompts, with higher scores reflecting better alignment. All experiments are conducted on RTX 3090 GPUs
Training. To train our Spatial Draft Head, we largely follow the same procedure used for Medusa and Lumina-mGPT. We only train the draft head, while keeping the rest of the model frozen. We use the Adam W 24
optimizer with weight decay of 0.1 and 8 (0.9, 0.95). The base learning rate is set to 10 We set the draft head balance weight A to Structural details of the draft heads and memory considerations are provided in Appendix C We use 6, 000 images sampled from
the LAION aesthetic training set (32 to fine-tune our drafter head. This training procedure typically takes 8-12 12 hours to converge on single RT'X 3090 GPU.
5.1 Quantitative and Qualitative Evaluation
We evaluate five different methods by analyzing their performance across various test datasets: vanilla autoregressive decoding, Medusa (horizontal draft heads), Hawk with vertical draft heads, Medusa with LANTERN++, and Hawk with spatial draft heads. this context, spatial draft heads refer the combination of both vertical and horizontal draft heads.
7

---

Table 1: Experiment result for different speculation methods on COC02017 and Flickr30K validation dataset. We sampled 500 images from the validation set and evaluated the hardware environment of a single RTX 3090 GPU. Spatial Draft heads: combination of vertical draft heads and horizontal draft heads. For LANTERN++, we use the hyperparameter setting of (I: 10, X = 2).
Accepted Length
Dataset
Method
Speed up
FID
CLIP Score
Vanilla AR
1.00x
90.68
33,43
Medusa 4
1 1.58x
1.728
92.51
33.34
COCO2017
Hawk (Vertical Draft Heads)
1,58x
1.726
91.46
33.33
LANTERN+ [18]
1.69x
1.863
93.22
33.08
Hawk (Spatial Draft Heads)
1.71x
1.890
90.71
33.39
Vanilla AR
1.00x
104.80
34.49
Medusa 4
1.55x
1.704
103.51
34.48
Flickr30K
Hawk (Vertical Draft Heads)
1,59x
1.738
104.21
34.45
LANTERN+ [8]
1,67x
1.842
106.10
34.25
Hawk (Spatial Draft Heads)
1.69x
1.871
104.50
34.45
Table presents the quantitative results comparing different approaches in the image speculative decoding task. Our vertical and horizontal draft heads demonstrate varying effectiveness depending on the dataset. This discrepancy is likely attributed the distinct image distribution characteristies of each dataset. Due to varying spatial dependencies within the images across datasets, performance differences arise between the different draft heads. Our Hawk method achieves 1.71x speed-up compared to the baseline method, aligning with our expectations. Since our sampling space is combination vertical and horizontal sampling spaces, overall performance is constrained by the summation of the performance gains from both heads.
From the perspective of FID (lower is better) and CLIP score (higher is better), approach achieves substantial acceleration while effectively preserving the image quality of the original model. We com- pare method with LANTERN++, strategy, and integrate their algorithm into our Medusa-based framework for a fair comparison. Our approach achieves a consistently higher accepted length during speculative verification, indicating more efficient and stable speculative decoding. Although the overall acceleration gain is moderate due the additional computation from dirl--irectional draft heads, this improvement is obtained without relaxing verification thresholds, maintaining strict adherence to the verification criteria.
existing multi-draft image speculative decoding acceleration
In contrast, LANTERN+ adopts a relaxed speculative decoding strategy from min (1.
(()a))
p())s)
to min (1, Exedy, Sta) ((a)s)) that substantially lowers the verification threshold, increasing the p(2]a)
acceptance rate of inaccurate predictions from the small model. This often results in degraded generation quality, as reflected in the fact that LANTERN++ exhibits higher FID and lower CLIP scores compared Hawk. By maintaining more conservative and accurate verification process, Hawk achieves superior overall performance in both efficiency and output fidelity.
For the qualitative evaluation, visually compared images generated by different methods identify any noticeable differences in quality. This analysis aimed to assess whether the accelerated methods introduced any degradation compared the baseline other approaches. Given that speculative sampling strategy is mathematically grounded, it theoretically preserves the same sampling space as the target model. As shown in Figure we conducted extensive observational experiments our test set and found no significant degradation in image quality. These results suggest that our Hawk method produces images that maintain the same quality as the original model.
5.2 Effectiveness of Vertical Draft Head
We aim to demonstrate that, compared to traditional horizontal speculative sampling, vertical heads in- troduce new information that improves the spatial sampling space. We tested performance differences between vertical and horizontal draft heads during training and inferencing.

---

sprte
Pornecingg GEOMR SOPOIN DUNOO
FOPU YAU ww FIOM RORM
Figure 3: Quality evaluation of Hawk method. For each image type, the images, from left to right, are generated by the base model, Medusa, Hawk with vertical heads, and Hawk with spatial heads. The Hawk method maintains the performance of the baseline model while enhancing inference speed.
Promot Gar crato en imag 105s 705 according to the lollowing promptinnnggs cla large seaved walking in woodoo and
Hortzontal Speculation
5.043
6,850
7,010 -30,05
7.177
-42.2%
Draft Heads
5,672
7.038
7,200
Spacoration
32.3%
-0.0%
-22,05
6.880
7,250
33.44%
-0.446
Horizontal Heads
4. 8818
7,068
Spania Specu kaiive Deooding
HL. diverggnne buttooon vertical and henizorial draft
brfeemning
Figure 5: The KL divergence between the verti- cal and horizontal draft heads during speculative decoding for the given prompt. The difference between the vertical and horizontal draft heads more pronounced when generating complex areas of the image.
Figure 4: The training loss of draft heads at dif- ferent locations is related to the current decoding point. Vertical heads experience relatively less performance decay as the speculation depth in- creases.
During training, we observed discrepancy the speculative capabilities vertical and horizontal heads. We compared the training losses of draft heads at different positions. Intuitively, since vertical speculation head is positioned decoding point-much larger than that of the horizontal head to have a higher training loss. However, our experiments yielded the opposite result. In Figure 4 show comparisons of training losses of draft heads at various positions.
distance equal to the image width from the current we would expect the vertical draft
The green sections represent the fine-tuning loss of the model on the training dataset, while the other sections show the training loss of the speculative draft heads. For speculative points at the same Euclidean distance from the current decoding position, the vertical draft head demonstrates lower training loss. The performance decay rate shown in the figure illustrates this phenomenon. With increasing speculative depth, the vertical direction shows a smaller performance decay rate (3.90%) compared to the horizontal direction (4.88%). Our experiments reveal that, ideally, vertical heads should produce better speculative results than horizontal heads, even though they involve significantly larger speculative distance.

---

In Figure 5 we compare the Seulbacck-eibble (KL) divergence between horizontal and vertical draft heads during inferencing. The plot reveals that the divergence gap becomes more pronounced when generating fine-grained details, such as an elephant's ' eyes, nose, teeth, and overall contour. These regions, due to their inherent ambiguity, permit multiple potential outcomes, resulting in a broader sampling space for the target model and greater KI. differences between vertical and horizontal heads. Conventional draft-based speculative sampling struggles to align with the target model in such cases. In contrast, our Hawk method leverages integrated sampling spaces, with additional contributions from the vertical head, which broadens the range of speculative choices and significantly increases the acceptance rate of our sampling strategy.
To further quantify the benefits of this in- creased diversity among draft heads, we an-
Speculative Rejection Rate vs. Candidate Size
Average Speculative Rejection Rate (96)
alyze its theoretical impact on sampling ef-
ficiency. As shown in Fi we illustrate the theoretical speedup effects introduced by the diversity among different draft heads.
0.6
Dual direction draft heads
Horizontal Draft Heads
We evaluate the theoretical speculative rejec- tion probability (r)) (see Eq, 7. where lower Presid (z) indicates greater potential for acceleration. The term Presia (z) represents the residual probability of resampling after rejection.
0.4
0.3
0.2
10
As defined in Eq. [6] the residual distribution (a+1) (z) is recursively updated by subtract- Pearget ing the draft distribution Garafe (x) from the target distribution. When the same draft dis- tribution (rrart(r) is repeatedly used across iterations, the residual mass converges to fixed point where further subtraction yields no meaningful reduction Seriiion--eneeeeeee (r)) cannot further converge. However, by introducing multiple sampling spaces with distinct draft (1)
Candidate Size
Figure 6: Comparison of dual-direction and horizontal draft heads across varying numbers of candidates. The dual-direction design yields lower speculative de- coding rejection probability, indicating more efficient utilization of draft tokens.
distributions, the residuals each step are computed with respect different garart (I)) enabling
additional reduction of the overall Presid (x) and thus yielding greater theoretical speedup.
6 Conclusion
In this work, we introduced Hawk, an acceleration framework for autoregressive image generation that leverages spatially aware speculative decoding. By integrating spatial information into the sampling process, Hawk effectively enlarges the candidate space and achieves substantially higher acceptance rates, leading to faster inference without compromising image quality. Extensive experiments demonstrate that Hawk maintains visual fidelity comparable to baseline while delivering notable speedups. Future work includes integrating Hawk with more advanced speculative backbones (e.g.. Eagle (2] or Hydra (I] to further enhance efficiency and performance.
7 Acknowledgements
This work supported by the National Key R&D Program of China (No. 2024YFF0202800), the National Natural Science Foundation of China (NSFC) under Grant No. 62476123, and NSFC under Grant No. 62376118.
10

---

References
[1] Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard, Jonathan Ragan-Kelley, and William Brandon. Hydra: Sequentially-dependemn draft heads for medusa decoding. arXiv preprint urXiv:2402.000 05 109, 2024.
[2] Aishwarya Bhandare, Vamsi Sripathi, Deepthi Karkada, Vivek V. Menon, Sun Choi. Kushal
Datta, and Vikram A. Saletore. Efficient 8-bit quantization of transformer neural machine language translation model. ArXiv, abs//906.000 1906.0053, 2019.
[3] Warren Burton. Speculative computation, parallelism, and functional programming. IEEE Transactions on Computers, 100(12): 190-1199--93 1985.
[4] Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri
Dao. Medusa: Simple Ilm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774 2024.
[5] Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, -ean-Baptiste Lespiau, Laurent Sifre, and
John Jumper. Accelerating large language model decoding with speculative sampling, arXiv preprint arXiv:2302.00 08..11.1...... 2023.
[6] Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff. Vqgan-clip: Open domain image generation and editing with natural language guidance. In European Conference on Computer Vision, pages 80-105. Springer, 2022.
[7] Yingying Deng. Fan Tang, Weiming Dong, Chongyang Ma, Xingjia Pan, Lei Wang. and Changsheng Xu. Stytr2: Image style transfer with transformers. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 1 11326-1 1276--13366 1136,, 2022.
[8] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis.
Advances in neural information processing systems, 34:8780-8788 2021.
[9] Ming Ding, Zhuoyi Yang. Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang. Cogview: mastering text-to-image generation via transformers. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS '21, Red Hook, NY, USA, 2024. Curran Associates Inc.
[10] Jiale Fu, Yuchu Jiang, Junkai Chen, Jiaming Fan, Xin Geng, and Xu Yang. Speculative ensemble: Fast large language model ensemble via speculation. arXiv preprint arXiv:2502.01662 2025. [11] R. Gray. Vector quantization. IEEE ASSP Magazine, 1(2):4-99, 1984.
[12] Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. Non- autoregressive neural machine translation. arXiv preprint arXiv:1711.0222
2017. [13] Baisong Guo, Xiaoyun Zhang, Haoning Wu, Yu Wang. Ya Zhang. and Yan-Feng Wang. Lar-sr: local autoregressive model for image super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1909-1918, 2022.
[14] Yefei He, Feng Chen, Yuanyu Hc, Shaoxuan He, Hong Zhou, Kaipeng Zhang, and Bohan Zhuang. Zipar: Accelerating auto-regressive image generation through spatial locality, 2024.
[15] John L Hennessy and David A Patterson. Computer architecture: a quantitative approach. Elsevier, 2011.
[16] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A 08718,
reference-free evaluation metric for image captioning, arXiv preprint arXiv:2104 104.0818,,8788, 2021. [17] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[18] Doohyuk Jang, Sihwan Park, June Yong Yang. Yeonsung Jung, Jihun Yun, Souvik Kundu, Bung-Yub Kim, and Eunho Yang. Lantern: Accelerating visual autoregressive models with relaxed speculative decoding, 2025.
[19] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In Proceedings the 40th International Conference on Machine Learning, ICML'23. JMLR.org .org, 2023.
[20] Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. CoRR, abs/2406. 1838, 2024.
[21] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: speculative sampling requires rethinking feature uncertainty. In Proceedings of the 41st International Conference on Machine Learning, 244 JMLR.org, 2024.
[22] Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár, Microsoft coco: Common objects in context, 2015.
11

---

[23]
Dongyang Liu, Shitian Zhao, Le Zhuo, Weifeng Lin, Yu Qiao, Hongsheng Li, and Peng Gao. Lumina-mgpt: Illuminate flexible photorealistic eext-to-image generation with multimodal generative pretraining, arXiv preprint
Sitiv:2408.026555 2024.
[24]
Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv::711.05100 2017. Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm- Ammpruner: On the structural pruning of large language models. Advances in neural information processing systems. 36:21702-21720 2023, Xupeng Miao. Gabriele Oliaro, Zhihao Zhang. Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. Specinfer: Accelcrating large language model serving with tree- bese speculative inference and verification. In Proceedings of the 29th ACM International Conference Architectural Support for Programming Languages and Operating Systems, Volume 3. ASPLOS '24, page 932-949. ACM, Apr. 2024.
[25]
[26]
[27] James M Ortega and Werner C Rheinboldt. Iterative solution of nonlinear equations in several variables. SIAM, 2000.
[28] Yassine Ouali, Céline Hudelot, and Myriam Tami. Autoregressive unsupervised image segmen- tation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VII 16. pages 142-158. Springer, 2020.
[29] Bryan A Plummer, Liwei Wang. Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to- -o-phrass correspondences for richer image-to-sene mmage-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2649, 2015.
[30] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 20Xiv:2204.06/55, 1(2):3, 2022.
[31] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8821-8831. PMLR, 18-24 Jul 2021. [32] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25294, 2022.
[33] Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling. Advances in Neural Information
Processing Systems, 35:17456-17472, 2022.
[34] Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan.
Autoregressive model beats difffuson:: Llama for scalable image generation, 2024.
[35] Chameleon Team. Chameleon: Mixed-modal early- foundation models. arXiv preprint
arXiv:2405.0 000.088888 2024.
[36] Yao Teng, Han Shi, Xian Liu, Xuefei Ning. Guohao Dai, Yu Wang, Zhenguo Li, and Xihui Liu. Accelerating auto-regressive text-to-image generation with training- free speculative jacobi decoding, 2025.
[37] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive mod- eling: Scalable image generation via next-scale prediction. arXiv preprint 2404:2240. 2024.
[38] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. Advances in neural information processing systems, 37:84839-84865, 2024.
[39] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Ilaaaa open and efficient foundation language models. arxiv. arXiv preprint arXiv:2302. 230, 2023.
[40] Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, al. Con- ditional image generation with pixelcnn decoders. Advances in neural information processing systems, 29, 2016.
[41] Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International conference on machine learning, pages 1747-1756. PMLR, 2016.
[42] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[43] Ziyu Wan, Jingbo Zhang, Dongdong Chen, and Jing Liao. High-fidelity pluralistic image completion with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages
2021.
12

---

[44] Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Structured pruning of large language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLPP. Association for Computational Linguistics. 2020.
[45] Yongliang Wu, Shiji Zhou, Mingzhuo Yang, Lianzhe Wang, Heng Chang, Wenbo Zhu, Xinting Hu, Xiao Zhou, and Xu Yang. Unlearning concepts in diffusion model via concept domain correction and concept preserving gradient. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pages (596-8504, 2025.
[46] Heming Xia, Zhe Yang. Qingxiu Dong, Peiyi Wang. Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. Unlocking efficiency in large language model inference: comprehensive survey of speculative decoding. arXiv preprint arXiv:2401 .07851. 1, 2024.
[47] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming
language models with attention sinks. arXiv preprint arXiv:2309. 1755,, 2023.
[48] Jing Xiong. Gongye Liu, Lun Huang, Chengyue Wu, Taiqiang Wu, Yao Mu, Yuan Yao, Hui Shen, Zhongwei Wan, Jinfa Huang. et al. Autoregressive models in vision: survey. arXiv preprint 2rXiv:2411.05900 2024.
[49] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.1078, 2(3):5, 2022.
[50] Ali Hadi Zadeh, Isak Edo, Omar Mohamed Awad, and Andreas Moshovos. Gobo: Quantizing attention-based nlp models for low latency and energy efficient inference. In 2020 53rd Annual IEEE/ACM International Symposium Microarchiteclure (MICRO), page 81 811-824. IEEE, Oct. 2020.
[51] Kaiqi Zhang, Jing Zhao, and Rui Chen. Koala: Enhancing speculative decoding for Ilm via multi-layer draft heads with adversarial learning. arXiv preprint arXiv:2408. 08/ 46, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, our abstract and introduction summarize the main contributions of our work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, added an limitation section at the end of the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
Yes, we have complete proof for each theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, the paper has contain all the details to reproduce the result.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
The code for this paper will be released upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
wwt.Suubmisioooooiiooooooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, we have included the hyperparameters of our training and evaluation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Papers on large language models (LLMs) typically do not report error bars in their evaluation results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, we have included the training and inferencing computation resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://ps///acce....p/o////////e
Answer: [Yes]
Justification: Yes, we fellow the code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper presents work whose goal is to advance the field of Machine Learning, especially image generation acceleration. There are many potential societal consequences in our work, none of which we feel must be specifically highlighted here.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper has no safeguards risks
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, we have cite all the original owners of assets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code for this paper will be released upon acceptance.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
20

---

A Theoretical Justification
Assume we are given two speculative sampling spaces: one from the vertical draft head and the other from the horizontal draft head. Let a denote the number of speculative tokens sampled from the vertical head, and b denote the number of speculative tokens sampled from the horizontal head. In the simplest case, we set for validation. We adopt a sequential validation strategy in which the token proposed by the vertical head is evaluated first. If it is rejected, we then proceed to validate the token from the horizontal head. Under this setup, we define the following notation: p(x) be the true distribution defined by the base AR model. Ovent (2)) and (z) be the proposal distributions from the vertical and horizontal draft heads, respectively.
and b = 1, meaning that we draw one candidate token from each head
Pven(1) is the adjusted base model distribution after vertical head rejection. Calculated by
p(x) min (gven (x), p(x))
(max(1a p(x) Oven (x)))
(8)
Σx (p(x') minen (z'), p(x'))]
Pvert-horiz (x) is the residual distribution after both vertical and horizontal rejections. Calculated by
(r) min (I)), (x))( Σx' [Pvon (a') min min (vene())))))))) Pvent (1¹))
norm(max(0, Pven(()) Oven (2)))
(9)
Oven is the acceptance probability of the vertical head. Calculated by
Σ (p(x') min(qvel(z'). p(x
(10)
Avert-horiz is the acceptance probability of the horizontal head conditioned vertical rejection. Calculated by
- Σ (pven (x') - min(gvon(a) P. (x)))
(11)
Our ultimate goal is to approximate the original model's distribution p(x) by combining samples from two draft heads operating along different spatial dimensions. For given point x', our proof proceeds as follows:
p(x') = Pr(vertical accept, x = x') Pr(vertical reject, horizontal accept, x = x')
+ Pr(vertical reject, horizontal reject, x = x')
p(x')
= [venical (2') min
Avertical (vertica((x")
P' Plvenical (z.")
+ (1 - avvrtical) Qborizoutal (x') min
Qhorizontal (x")
+ (1 - avertical) (1
(kyertical-horizoonaa))
Pvertical- Prertical-booizootal (x¹)
= Overtieal (z') min 1, p(x')
Overtical (x²)
Aborizooutal (x') min "'enica(("")
+ (1 - O'vertica))
Thorizontal (x²)
+ (1 - avertical) (Prentical (x') min (Dvorizontal (2'), (2'),
Pventical (x²)))
plx') )
= Ovenical (x') min gventicat (xx) + (1 Ovenical) Phertical (x')
= =(aa))
(12)
Let ग [81, 82, Sa+6] denote an ordered sequence of speculative tokens, where each 8; is generated from either vertical horizontal draft head. The total number of speculative tokens + with a sampled from horizontal heads and from vertical heads. Each sample result 8; corresponds to proposal posbility (xx). At each step i, the acceptance probability is defined
21

---

as a; =1 (x') - min(q((')) p(()))))) where (x) represents the residual target
density after the first i proposals have been rejected. It is recursively defined as: pl (x)
norm (max (0., (z) qi q (z))) The sampling probability specific point x' under the generalized formulation is given by:
a+b
a+b
p(a) -
II ay(x)) q(x)) ar(z) a(l))a(())) + ([a=)y()))) Presia(
(13)
This section provides the theoretical justification needed address the reviewer's concern about potential distributional shift. It shows that our dual-direction speculative decoding preserves the original output distribution.
B More Attention Sinking Experiment
We conducted further analysis of attention sinking in image generation. In Figure 7 we reshaped the ID image attention logits into 2D spatial representation for visualization. The results reveal that during the generation of an image, significant attention is concentrated on the first token of the image and the tokens spatially adjacent to the current inference point. Additionally, tokens the end of each row tend to have slightly higher weights, while the rest of the attention is evenly distributed across the entire image. Although image generation still follows a raster order, inherently encodes spatial information.
10
30.
40
Position of 2D tokens
Figure 7: Lumina-mGPT average attention logits converted into a 2D spatial representation.
C Draft Heads Structure and Memory Overhead
For single Medusa head, we employ a linear layer with both input and output dimensions equal to the model's hidden size, followed by SiLU activation function to introduce non-linearity.
The original Medusa strategy introduces only minimal memory overhead (less than 0.5%). Our proposed SSD strategy results in relatively small increase in the total number of parameters, accounting for less than % the base model (Lumina-mGGPT)) During inference, the actual memory overhead ratio is even lower due to the use of KV caching, which helps reduce the memory cost of additional components. In the case of FP16, our draft heads occupy approximately 134 MB of memory.
22

---

Figure 8: The performance of the base model under different top-k parameters is as follows: from top bottom, the values top-k are 2000, 1000, 200, 50, 10, and 1.
D Different Generation Parameters A Affect Image Quality
In AR text generation, the top-k parameter is typically set between 5 and 50, representing the maximum number of elements considered during sampling at each step. However, AR image generation, the choice of top-k has significantly different impact. Larger top-k values generally result in more detailed images and higher image quality. However, as top-k increases, speculative sampling becomes
challenging: fewer speculative predictions pass the model's validation, leading to slower speculative decoding speeds. Figure compares how different top-k values affect the image quality of the baseline model.
Temperature plays a crucial role in autoregressive image generation, as the quality of the generated images varies significantly with changes in temperature. In Figure
tested the performance of the base model under different temperature settings. Due to the strong impact of temperature on image quality, certain algorithms, such as jacobi decoding (27] (which only applicable in greedy decoding), are not suitable for accelerating autoregressive image generation.
E Limitation and Future works
In current implementation, employ Medusa as the baseline for speculative decoding. How- ever, according to prior evaluations [46]. Medusa is not among the most efficient or state-of-the. algorithms for text speculative decoding. Consequently, our reported acceleration results may under- estimate the potential performance achievable with more advanced speculative decoding methods. In future work, we plan to integrate stronger baseline algorithms to further improve inference efficiency and validate the generality of our proposed approach.
23

---

Figure 9: The performance of the base model under different temperature parameters is as follows:
from top to bottom, the values temperature are 1, 0.8, 0.6, 0.4, 0.2.
24


Reviews:
[{"review_id": "YWA4fTHfY6", "reviewer": "Reviewer_g8rp", "summary": "This paper introduces a spatial speculative decoding framework that leverages both horizontal and vertical draft heads to accelerate autoregressive text-to-image generation. By explicitly leveraging the two-dimensional spatial structure of images, the proposed method achieves substantial inference speedup while preserving image fidelity.", "questions": "- Does expanding the sampling space via dual-direction speculation provide theoretically justified gains in acceptance probability and sampling efficiency, or does it primarily amplify speculative variance absent formal guarantees?\n\n- How does the proposed spatial speculative framework control for potential shifts in the output distribution, especially under repeated rejection and reweighting, and is there a formal calibration or uncertainty analysis?\n\n- In scenarios where spatial local correlations are less critical for prediction, could the computational overhead introduced by dual-direction speculative sampling outweigh the gains in speed or acceptance efficiency, thereby reducing overall cost-effectiveness?", "limitations": "This paper lacks formal analysis of how 2D sampling impacts acceptance or output calibration, inherits known speculative biases, and may be inefficient when local spatial correlations are weak.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strenths\n\n- This paper addresses an important practical challenge of slow sequential inference in AR image generation.\n\n- The proposed method employs dual-direction draft heads and spatial caches that explicitly align with the two-dimensional image structure, offering an interesting way to leverage spatial dependencies.\n\n### Weaknesses\n\n- The contribution of the proposed method is largely an extension of Medusa-style speculative decoding into a second spatial dimension.\n\n- The paper lacks a formal theoretical justification for why vertical speculative heads, despite involving longer dependency paths, exhibit lower performance degradation.\n\n- The paper does not provide a formal analysis of how the expanded 2D sampling space affects acceptance rates or output distribution.\n\n- The proposed method inherits known speculative decoding issues like sampling bias after rejection.", "was_revised": true, "final_justification": "The theoretical justification has been strengthened, and my concern regarding weak spatial correlation has been addressed.", "rebuttal": "We sincerely thank the reviewer for the constructive and encouraging feedback. We appreciate your recognition of both the practical significance of our problem setting and the novelty of our spatial speculative decoding formulation. \n\nIn response to your valuable suggestion for stronger theoretical support, we now provide a formal analysis (outlined below) demonstrating that our dual-direction speculation preserves the output distribution and improves acceptance rates.  We acknowledge that our original submission lacked clarity in this regard, and we will include two new sections in the appendix of the final version: **(1) a proof of output distribution preservation**, and **(2) a theoretical justification of efficiency gains**. These will expand on the analysis provided here in response to your comments.\n\n### 1. Dual-direction speculative decoding preserves the original output distribution\n\nTo address the reviewer’s concern regarding potential shifts in the output distribution, we have provided a theoretical justification of our algorithm. This analysis demonstrates that using multi-directional draft heads (e.g., vertical and horizontal) does not alter the target output distribution. In particular, our framework maintains the same acceptance formulation and sampling guidance as standard speculative decoding. \n\nAssume we are given two speculative sampling spaces: one from the vertical draft head and the other from the horizontal draft head. Let $a$ denote the number of speculative tokens sampled from the vertical head, and $b$ denote the number of speculative tokens sampled from the horizontal head.\n\nIn the simplest case, we set $a = 1$ and $b = 1$, meaning that we draw one candidate token from each head for validation. We adopt a sequential validation strategy in which the token proposed by the vertical head is evaluated first. If it is rejected, we then proceed to validate the token from the horizontal head. \nUnder this setup, we define the following notation: \n$p(x)$ be the true distribution defined by the base AR model. \n$q\\_{\\text{vert}}(x)$ and $q\\_{\\text{horiz}}(x)$ be the proposal distributions from the vertical and horizontal draft heads, respectively.\n\n$p^{\\prime}\\_{\\text{vert}}(x)$ is the adjusted base model distribution after vertical head rejection. Calculated by  \n$\\text{norm}\\left(\\max\\left(0, p(x) - q\\_\\text{vert}(x)\\right)\\right) = \\frac{p(x) - \\min(q\\_\\text{vert}(x), p(x))}{\\sum\\_{x'} \\left(p(x') - \\min(q\\_\\text{vert}(x'), p(x'))\\right)}$\n\n$p^{\\prime}\\_{\\text{vert-horiz}}(x)$ is the residual distribution after both vertical and horizontal rejections. Calculated by $\\text{norm}\\left(\\max\\left(0, p^{\\prime}\\_{\\text{vert}}(x) - q\\_\\text{vert}(x)\\right)\\right) = \\frac{p^{\\prime}\\_{\\text{vert}}(x) - \\min(q\\_\\text{vert}(x), p^{\\prime}\\_{\\text{vert}}(x))}{\\sum\\_{x'} \\left(p^{\\prime}\\_{\\text{vert}}(x') - \\min(q\\_\\text{vert}(x'), p(x'))\\right)}$\n\n$\\alpha\\_{\\text{vert}}$ is the acceptance probability of the vertical head. Calculated by \n$1- \\sum\\_{x'} \\left( p(x') - \\min(q\\_\\text{vert}(x'), p(x')) \\right)$\n\n$\\alpha\\_{\\text{vert-horiz}}$ is the acceptance probability of the horizontal head conditioned on vertical rejection. Calculated by \n$1- \\sum\\_{x'} \\left( p\\_\\text{vert}(x') - \\min(q\\_\\text{vert}(x'), p^{\\prime}\\_\\text{vert}(x')) \\right)$\n\nOur ultimate goal is to approximate the original model’s distribution p(x) by combining samples from two draft heads operating along different spatial dimensions. For a given point x’, our proof proceeds as follows: \n$$\\begin{aligned}\np(x') &= \\Pr(\\text{vertical accept}, x = x') \n  \\+ \\Pr(\\text{vertical reject, horizontal accept}, x = x') \\\\\\\\\n&\\quad \\+ \\Pr(\\text{vertical reject, horizontal reject}, x = x') \\\\\\\\\n&= q\\_{\\text{vertical}}(x') \\cdot \\min\\left(1, \\frac{p(x')}{q\\_{\\text{vertical}}(x')} \\right) \\\\\\\\\n&\\quad \\+ (1 \\- \\alpha\\_{\\text{vertical}}) \\cdot q\\_{\\text{horizontal}}(x') \\cdot \\min\\left(1, \\frac{p^{\\prime}\\_{\\text{vertical}}(x')}{q\\_{\\text{horizontal}}(x')} \\right) \\\\\\\\\n&\\quad \\+ (1 \\- \\alpha\\_{\\text{vertical}})(1 \\- \\alpha\\_{\\text{vertical-horizontal}}) \\cdot p^{\\prime}\\_{\\text{vertical-horizontal}}(x') \\\\\\\\\n&= q\\_{\\text{vertical}}(x') \\cdot \\min\\left(1, \\frac{p(x')}{q\\_{\\text{vertical}}(x')} \\right) \\\\\\\\\n&\\quad \\+ (1 \\- \\alpha\\_{\\text{vertical}}) \\cdot q\\_{\\text{horizontal}}(x') \\cdot \\min\\left(1, \\frac{p^{\\prime}\\_{\\text{vertical}}(x')}{q\\_{\\text{horizontal}}(x')} \\right) \\\\\\\\\n&\\quad \\+ (1 \\- \\alpha\\_{\\text{vertical}}) \\cdot \\left(p^{\\prime}\\_{\\text{vertical}}(x') \\- \\min(q\\_{\\text{horizontal}}(x'), p^{\\prime}\\_{\\text{vertical}}(x'))\\right) \\\\\\\\\n&= q\\_{\\text{vertical}}(x') \\cdot \\min\\left(1, \\frac{p(x')}{q\\_{\\text{vertical}}(x')} \\right)  \\+ (1 \\- \\alpha\\_{\\text{vertical}}) \\cdot p^{\\prime}\\_{\\text{vertical}}(x') \\\\\\\\\n&= p(x')\\end{aligned}$$\n\nLet $\\pi = [s\\_1, s\\_2, \\dots, s\\_{a+b}]$ denote an ordered sequence of speculative tokens, where each $s_i$ is generated from either a vertical or horizontal draft head. The total number of speculative tokens is $a + b$, with $a$ sampled from horizontal heads and $b$ from vertical heads. Each sample result $s_i$ corresponds to a proposal posbility $q_i(x)$\n\nAt each step i, the average acceptance probability is defined as\n$\\alpha_i = 1- \\sum_{x'} \\left( p^{(i)}(x') - \\min(q_i(x'), p^{(i)}(x')) \\right)$,\nwhere $p^{(i)}(x')$ represents the residual target density after the first i - 1 proposals have been rejected. It is recursively defined as:\n$p^{(i)}(x') = \\text{norm}\\left( \\max\\left(0,  p^{(i-1)}(x') - q\\_{i-1}(x') \\right) \\right)$,\nThe sampling probability at a specific point $x = x’$ under the generalized formulation is given by:\n$$p(x') = \\sum_{i=1}^{a+b} \\left( \\prod_{j=1}^{i-1} (1 - \\alpha_j) \\cdot q_i(x') \\cdot \\alpha_i \\right) + \\left( \\prod_{j=1}^{a+b} (1 - \\alpha_j) \\right) \\cdot p_{\\text{resid}}(x')$$\nThis section provides the theoretical justification needed to address the reviewer’s concern about potential distributional shift. \n\n### 2. Does dual-direction speculation offer provable improvements in acceptance and efficiency, or does it just increase variance without guarantees?\nIn one iteration of speculative decoding, a key inefficiency arises when all speculative candidates are rejected. In this worst-case scenario, the algorithm incurs additional computational overhead compared to standard autoregressive (AR) decoding, as both the speculative sampling and verification steps are wasted. The probability of encountering such an inefficiency is given by:\n$P(\\text{all reject}) = \\left( \\prod\\_{j=1}^{a+b} (1 - \\alpha\\_j) \\right) \\cdot p\\_{\\text{resid}}(x)$\n\nOur method significantly mitigates this inefficiency compared to conventional single-head speculative decoding. In particular, we observe a **16.2% relative reduction** in the rejection probability term $\\prod_{j=1}^{a+b} (1 - \\alpha_j(x))$, highlighting a more robust acceptance behavior. Recall that the acceptance probability at each step is defined as: $\\alpha_j = 1 - \\sum_{x’} \\left( p(x’) - \\min(q(x’), p(x’)) \\right)$. The target distribution p(x’) is dynamically updated during verification by removing the mass assigned to rejected proposals $q_i(x’)$. \n\nWhen all speculative tokens are drawn from a single proposal space, the **diversity of samples is limited**, and $\\min(q(x’), p(x’))$ tends to **quickly decay to zero with successive rejections**. This leads to rapidly vanishing acceptance probabilities $\\alpha_j$, increasing the chance of complete rejection.\n\nBy contrast, our dual-direction approach samples from two orthogonal proposal distributions (horizontal and vertical). Due to the **inherent diversity between these directions**, the proposals $q_j(x)$ differ significantly across heads. This variation delays the convergence of $\\min(q(x’), p(x’))$ toward zero, **slowing the decay of $\\alpha\\_j$** and thereby reducing the probability of total rejection.\n\n### 3. Is dual-direction speculation still cost-effective in scenarios with weak spatial correlation?\nWe thank the reviewer for raising this important concern. At its core, our method adheres to the speculative decoding framework, which can introduce computational overhead when speculative predictions are repeatedly rejected. This scenario typically arises when the proposal distribution diverges significantly from the base model’s target distribution, resulting in frequent rollbacks and redundant computations. \n\nSuch challenges have been documented in earlier speculative decoding methods, particularly those relying on **smaller auxiliary models** to approximate the base model, where **distributional mismatch can be substantial**. In contrast, our method leverages **draft heads** that are jointly trained with the base model, with the **explicit objective of approximating the token distribution at specific spatial locations**. This position-aware training aligns the speculative and target distributions more closely, thereby reducing rejection rates and mitigating unnecessary overhead.\n\nFurthermore, in scenarios where the input images lack strong spatial correlations, this limitation would typically manifest as elevated training loss or unstable draft head convergence. However, based on our empirical observations, we have not encountered real-world examples that exhibit a complete absence of spatial structure, and our method has demonstrated stable performance across diverse samples.\n\nIn summary, while speculative decoding inherently carries the risk of inefficiency due to distribution mismatch, our approach effectively addresses this by training draft heads that are well-aligned with the base model.\n\n\nOnce again, we sincerely thank the reviewer for recognizing the significance of our problem and for providing constructive and actionable feedback. We warmly welcome any additional questions or suggestions you might have, and would be happy to provide further clarification."}, {"review_id": "5kAA5muMoS", "reviewer": "Reviewer_UpDY", "summary": "This paper introduces Hawk, an autoregressive T2I framework. It presents a new speculative decoding method considering the two-dimensional spatial context of images. It uses spatial draft heads (horizontal and vertical) for parallel decoding. It uses the Lumina-mGPT as the base model to verify the effectiveness of the proposed method. According to the experiments, it outperforms related methods (Medusa and LANTERN) slightly. The core insight is that leveraging spatial dependencies in parallel decoding and improving the acceptance rate of speculative sampling in image generation.", "questions": "The training time is too short. Is the model converged?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. It is easy to follow the motivation and the main idea. The predication redundancy of vanilla AR is clear, and this paper introduces an improved speculative decoding method to tackle the redundancy.\n2. The implementation is easy to follow with pseudo code. And the overall presentation is clear.\n\nWeaknesses\n1. The method is similar to ZipAR and NAR (https://arxiv.org/abs/2503.10696). Please identify their differences in the main paper. This is the biggest problem. I will change my rating after reading the response.\n2. The improvements of results for vanilla AR are limited, compared with other parallel decoding methods, such as ZipAR, NAR, and PAR (https://arxiv.org/abs/2412.15119).\n3. Comparisons with mask modelling methods and diffusion methods are missing.", "was_revised": true, "final_justification": "I maintain my positive score considering comments from other reviewers and the potential of parallel decoding. However, I still think the novelty and comparisons against other parallel decoding methods (e.g., ZipAR, NAR, and PAR)are not enough. This paper does not bring new ideas to the community and just makes incremental improvements to an existing parallel decoding paradigm.", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback, as well as for acknowledging the importance of the problem and the quality of our empirical results. We greatly appreciate your openness to revisiting the evaluation upon clarification, and we would like to take this opportunity to respond to your main concerns in detail.\n\n### 1. On the differences between our method and ZipAR / NAR / PAR\nWe appreciate the request to clarify the distinctions between our approach and prior works such as ZipAR, NAR, and PAR. We acknowledge that our initial submission may not have made these differences sufficiently explicit, and we are grateful for the opportunity to elaborate here. The key differences can be grouped into two dimensions: **(1) Algorithmic design**, and **(2) High-level motivation**\n\n(1) Algorithmic differences: Speculative validation vs. non-AR generation\n\nOur method fundamentally differs from ZipAR and similar approaches in its inference paradigm. While ZipAR and related methods follow a **non-autoregressive (NAR)** decoding paradigm, our method remains **strictly autoregressive**, employing **speculative decoding with validation**. Our core decoding process is:\n  1. At decoding step n, we speculatively generate t future tokens using a spatial draft module.\n  2. These tokens are then validated using the base AR model.\n  3. If validation fails, we roll back to the last verified position.\n\nOur algorithm is **theoretically guaranteed to preserve the output distribution** of the original model, adhering to the fundamental principles of speculative decoding. As outlined in our response to **reviewer g8rp** (response 1), we have provided a formal proof demonstrating that our method maintains distributional consistency.  By contrast, ZipAR and similar methods **directly accept unvalidated predictions** for multiple future positions, jumping from n to n + t in a single step. This can accelerate decoding but often comes at the cost of generation quality, as predictions are made with insufficient contextual grounding. While their approach only performs the parallel decoding step, our method **additionally incorporates verification and rollback procedures**, which are essential for maintaining the model’s output fidelity and preventing performance degradation.\n\nTo give a concrete example, ZipAR adopts a different acceleration strategy: once a token’s preceding context reaches a certain threshold (though still smaller than what standard AR decoding requires), the model begins decoding that token early. These early-decoded tokens are then grouped with the current token and processed in parallel. It also introduces two sources of potential degradation. First, early decoding may suffer from insufficient attention context, as some crucial preceding tokens are not yet available. Second, ZipAR accepts parallel predictions without full autoregressive verification, which can further compromise generation quality. In our method, speculative outputs at positions like n + row_size are cached but not accepted immediately. They are only used when the AR decoding process naturally reaches that position, thereby preserving temporal consistency while still leveraging spatial context.\n\n\n(2) High-level motivation: Preserving fidelity vs. maximizing speed\n\nOur approach **prioritizes maintaining the fidelity of the original AR inference process**, rather than achieving maximum speedup. While methods like ZipAR and NAR allow more aggressive approximations, this often leads to degraded performance—especially in tasks requiring fine-grained consistency. In contrast, our speculative decoding strategy validates all generated tokens before accepting them, thereby ensuring that quality is not compromised. This leads to a more conservative speedup but better preserves model behavior. We believe this fidelity is particularly important for downstream applications where consistent generation is critical. For instance, in a real-world deployment scenario such as a generative image editing app, introducing a faster but unstable decoding strategy could result in noticeable quality drops, which may in turn degrade user experience and lead to user attrition.\n\n### 2. On the limited improvements over vanilla AR\n\nThis concern is closely related to the previous point, as it reflects a natural trade-off between generation fidelity and decoding speed. While approaches such as ZipAR, NAR, and PAR demonstrate higher raw speedups over vanilla AR models, these gains are often achieved by relaxing constraints on prediction verification, which can result in noticeable quality degradation. In general, more aggressive approximations tend to yield faster inference, but often at the cost of fidelity. Without **a unified performance-quality trade-off metric**, it becomes **difficult to directly compare their results against ours in a fair and meaningful way**.\n\n\n### 3. On comparisons with mask modeling and diffusion-based methods\nWe thank the reviewer for highlighting this point. We would like to respectfully clarify that our decision not to compare with mask modeling or diffusion-based methods stems from a **fundamental difference in task formulation and generation paradigm**. \nOur work focuses on **accelerating unified multimodal foundation models** that follow a next-token autoregressive generation framework, which supports **interleaved text-image generation**. Specifically, our acceleration algorithm targets the image generation component of this process. This modeling paradigm has gained significant traction in the community, with prominent examples including Chameleon [4], Lumina-mGPT [5,6], BLIP-3-o [7], Emu3 [8], Bagel [9], and Uni-world [10]. These models share a common generation pipeline. In this setting, the typical generation flow involves:\n\n  1. An LLM performs autoregressive inference over the text portion.\n  2. **The model continues autoregressively to generate image tokens (often in a raster order).**\n  3. A decoder, such as a VAE or diffusion model, reconstructs the final image from these tokens.\n\nOur work specifically targets the **second step**, which is often the **computational bottleneck**, and aims to accelerate the token-level generation process within these foundation models. Importantly, models like **diffusions** are used primarily for decoding images in the **final step** and are not responsible for autoregressive token generation in the latent space. As such, they operate in a **different part of the pipeline** and are not the focus of the acceleration we propose.\n\nThis evaluation practice is consistent with recent works that **address the same generation setting**—such as LANTERN++[1] (ICLR 2024), SJD[2] (ICLR 2024), and  ZiPar[3] (cvpr)—which also focus on token-level generation and do not include diffusion methods as a baseline due to the fundamental differences in modeling scope.\n\n\n### 4. Concern regarding model convergence due to limited training time\n\nWe appreciate the reviewer’s concern regarding whether the model has fully converged given the short training time. In our design, low training cost is not only expected but also intentional. Our goal is to enable practical deployment by accelerating inference with minimal additional training overhead.\n\nSpecifically, the draft head is a lightweight module inserted between the transformer blocks and the final language modeling (LM) head. It consists of a single linear layer followed by a SiLU activation. With a hidden size of 4096, the draft head introduces **less than 1% additional parameters** relative to the full model. Due to its compact size and initialization as an identity mapping, it typically converges within a few training steps in practice.\n\nOnce again, we sincerely thank the reviewer for recognizing the significance of our problem and for providing constructive and actionable feedback. We warmly welcome any additional questions or suggestions you might have, and would be happy to provide further clarification.\n\nReference\n\n[1]LANTERN++: ENHANCING RELAXED SPECULATIVE DECODING WITH STATIC TREE DRAFTING FOR VISUAL AUTO-REGRESSIVE MODELS\n\n[2]Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding\n\n[3]ZipAR: Parallel Auto-regressive Image Generation through Spatial Locality\n\n[4]Chameleon: Mixed-Modal Early-Fusion Foundation Models \n\n[5]Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining\n\n[6]Lumina-mGPT 2.0:Stand-Alone AutoRegressive Image Modeling\n\n[7]BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset\n\n[8]Emu3: Next-Token Prediction is All You Need\n\n[9]Emerging Properties in Unified Multimodal Pretraining (Bagel)\n\n[10]UniWorld-V1: High-Resolution Semantic Encoders for Unified Visual Understanding and Generation"}, {"review_id": "Ltc8FkSYln", "reviewer": "Reviewer_LJzH", "summary": "This paper addresses the slow inference issue in autoregressive (AR) text-to-image generation models, noting that while speculative decoding accelerates text generation, its application to images is underexplored due to larger sampling spaces and unexploited 2D spatial dependencies. The authors propose Hawk, a framework that leverages horizontal and vertical spatial contexts via dual draft heads, constructing a unified sampling pool by merging horizontal speculations with cached vertical predictions. Empirically, Hawk achieves a 1.71× speedup over standard AR models on benchmarks like COCO2017 and Flickr30K, preserving image fidelity and diversity.", "questions": "See Weaknesses", "limitations": "See Weaknesses", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The problem is timely and important.\n2. The empirical performance seems good.\n\nWeaknesses:\n1. This paper does not consider state-of-the-art Autoregressive (AR) image generation models, e.g., VAR. Moreover, the proposed technique cannot be applied to other types of generation models, such as diffusion models, flow-matching, and consistency models.\n2. The proposed technique does not introduce many innovations on top of the vanilla version of speculative decoding (the current 1D --> 2D may not be novel enough).\n3. Given 2, the experiments are not extensive enough. The authors should consider state-of-the-art generative fundation models learned on significantly larger datasets (e.g., at least at the level of Stable Diffusion).", "was_revised": true, "final_justification": "I thank the authors for the rebuttal. Most of my concerns have been addressed. Hence, I decide to increase my score to 4.", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback. We appreciate your recognition of both the problem’s significance and our empirical results. Below, we would like to clarify a few key aspects of our setting, comparison scope, and technical contributions that may not have been fully explained in the original submission.\n\n### 1. Clarification on Comparison Scope:\n\nRegarding the comment:\n“This paper does not consider state-of-the-art Autoregressive (AR) image generation models,”\n\nWe believe the apparent discrepancy may stem from **differing assumptions about the target setting**. While VAR focuses on **autoregressive image generation in isolation**, our work is situated in a different and increasingly popular setting: **unified multimodal foundation models** that support next-token generation across both text and image modalities.\n\nThis line of research has recently attracted substantial interest in the multimodal community, as it enables interleaved text-image generation through a unified autoregressive backbone. Representative examples include Chameleon[6], Lumina-mGPT[10,11], BLIP3-o[12], Emu3[13], Bagel[14], and Uni-World[15], all of which follow similar generation pipelines.\n\nIn this setting, the typical generation flow involves:\n\n1. An LLM performs autoregressive inference over the text portion.\n2. **The model continues autoregressively to generate image tokens (often in a raster order).**\n3. A decoder, such as a VAE or diffusion model, reconstructs the final image from these tokens.\n\nOur work specifically targets the **second step**, which is often the **computational bottleneck**, and aims to **accelerate the token-level generation process within these foundation models**. Importantly, models like VAR[16], Diffusion, or Consistency Models are used primarily for decoding images in the **final step** and are not responsible for autoregressive token generation in the latent space. As such, they operate in a **different part of the pipeline** and are not the focus of the acceleration we propose.\n\nThis evaluation practice is consistent with recent works that address the same generation setting—such as LANTERN++[1] (ICLR 2024), SJD[2] (ICLR 2024), and  ZiPar[3] (cvpr)—which also focus on token-level generation and do not include VAR as a baseline due to the fundamental differences in modeling scope.\n\n### 2. Clarification on Application Scope:\n\n\nRegarding the comment: “The proposed technique cannot be applied to other types of generation models, such as diffusion models, flow-matching, and consistency models.”\n\nWe sincerely thank the reviewer for raising this important point. Indeed, we fully agree that our proposed method is tailored to the autoregressive (AR) generation paradigm, and it may not be directly applicable to other generation frameworks such as diffusion models, flow-matching, or consistency models. This is **not a limitation unique to our work**, but rather reflects **a broader landscape** where **acceleration techniques are often specialized to the structural properties of the target model class**.\n\nAs also noted in recent surveys [4,5], acceleration methods for AR transformers and diffusion models tend to **evolve independently**, given their **fundamentally different generation processes**—sequential token prediction vs. iterative denoising. For instance, techniques such as Speculative Diffusion Decoding [8] and Cache Me If You Can [9] are powerful in the diffusion setting but are tightly coupled with the nature of score-based or noise-conditioned sampling. These cannot be directly transferred to token-level AR models.\n\nSimilarly, our method—like prior AR acceleration work including LANTERN++ [1], SJD [2], and Zipar [3]—is specifically designed to leverage the autoregressive sampling structure. While we believe bridging the gap across paradigms is a compelling future direction, it would likely require fundamentally new algorithmic tools beyond the scope of the current work.\n\nWe hope this clarification helps contextualize our design decisions, and we truly appreciate the reviewer’s thoughtful feedback on this point.\n\n### 3. Clarifying the Novelty Beyond 1D-to-2D Extension\n\nRegarding the comment: “The proposed technique does not introduce many innovations on top of the vanilla version of speculative decoding (the current 1D --> 2D may not be novel enough)”\n\nThank you very much for your thoughtful feedback. We understand your concern that the transition from 1D to 2D speculative decoding may seem incremental at first glance. However, we respectfully argue that adapting speculative decoding to the two-dimensional structure of image generation is **non-trivial both algorithmically and conceptually**, and introduces several new challenges that we address in our work.\n\nUnlike prior speculative decoding approaches that operate on linear (1D) sequences, our method is designed specifically for 2D spatial data, where information flows both horizontally and vertically. To effectively handle this structure while preserving the autoregressive (AR) generation order, we propose several key innovations:\n\n- We design **vertical draft heads** to enable speculative decoding along the vertical axis, which is entirely absent in prior methods.\n\n- We introduce a **spatial cache mechanism** that stores vertical speculative results across multiple future rows, allowing reuse of these intermediate computations **without violating the AR generation order**.\n\n- We further propose a **fused speculative sampling algorithm** that jointly leverages both cached vertical information and horizontal predictions, **balancing efficiency with consistency** in the 2D decoding trajectory.\n\nWe believe these contributions go beyond a simple dimensional extension and represent a meaningful step toward adapting speculative decoding to structured generation domains such as vision. Moreover, several reviewers have recognized the novelty of this contribution:\n\n- **Reviewer g8rp** noted that our method “employs dual-direction draft heads and spatial caches that explicitly align with the two-dimensional image structure, offering **an interesting way to leverage spatial dependencies.**”\n\n- **Reviewer httE** described it as “**the first** to leverage two-dimensional spatial information for accelerating autoregressive image generation using speculative decoding.”\n\n- **Reviewer UpDY** commented that our method “presents **a new speculative decoding method** considering the two-dimensional spatial context of images.”\n\nWe are grateful that these reviewers found our contributions promising, and we hope that this clarification better highlights the novelty and technical depth of our proposed method. We also believe this work may open new directions for applying speculative decoding in other structured domains beyond language.\n\n### 4. Clarifying the Choice of Foundation Models\n\nRegarding the  comment: “ The authors should consider state-of-the-art generative fundation models learned on significantly larger datasets (e.g., at least at the level of Stable Diffusion)”\n\nTo clarify, the foundation model we use, Lumina-mGPT [10,11], is actually a derivative of Chameleon [6], a large-scale vision-language model developed by Meta. Since Meta has not released the VAE component (which handles image tokenization), the Lumina-mGPT project was created to fill in that missing piece and make the model accessible for academic use. That said, most of the model’s parameters are directly inherited from Chameleon, and only the VAE was newly trained.\n\nIn terms of data scale, we’d like to point out that Lumina-mGPT was trained on about **1.9 trillion image-text tokens**. For comparison, Stable Diffusion 3.0[7], the latest version based on the Stable Diffusion Transformer architecture, was trained on roughly **1 billion image-text pairs**. The exact number of tokens may vary depending on the tokenizer used. While the data formats differ, we believe it is difficult to say that SD3.0 was trained on significantly more data, especially when considering the total token volume.\n\nFinally, we’d like to mention that **Lumina-mGPT is increasingly being used in the research community** as a practical foundation model. Two recent papers with settings similar to ours, SJD (ICLR) and LANTERN++ (ICLR), both adopt Lumina-mGPT as their main base model.\n\nWe respectfully hope that these clarifications provide additional context for our design choices and contributions. We remain open to further discussion and sincerely appreciate the opportunity to engage with your valuable feedback.\n\nReference\n\n[1]LANTERN++: ENHANCING RELAXED SPECULATIVE DECODING WITH STATIC TREE DRAFTING FOR VISUAL AUTO-REGRESSIVE MODELS\n\n[2]Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding\n\n[3]ZipAR: Parallel Auto-regressive Image Generation through Spatial Locality\n\n[4]A Comprehensive Survey of Accelerated Generation Techniques in Large Language Models\n\n[5]Efficient Diffusion Models: A Comprehensive Survey from Principles to Practices\n\n[6]Chameleon: Mixed-Modal Early-Fusion Foundation Models \n\n[7]Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (stable diffusion 3.0)\n\n[8] Speculative Diffusion Decoding: Accelerating Language Generation through Diffusion\n\n[9] Cache Me if You Can: Accelerating Diffusion Models through Block Caching\n\n[10]Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining\n\n[11]Lumina-mGPT 2.0:Stand-Alone AutoRegressive Image Modeling\n\n[12]BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset\n\n[13]Emu3: Next-Token Prediction is All You Need\n\n[14]Emerging Properties in Unified Multimodal Pretraining (Bagel)\n\n[15]UniWorld-V1: High-Resolution Semantic Encoders for Unified Visual Understanding and Generation\n\n[16]Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction"}, {"review_id": "0biP5gGh7r", "reviewer": "Reviewer_httE", "summary": "This paper introduces Hawk, a novel approach designed to accelerate autoregressive text-to-image generation by leveraging the two-dimensional spatial structure inherent in images. The core idea is to enhance speculative decoding by incorporating both horizontal and vertical spatial information, which helps improve the alignment between draft and target models, leading to faster inference without compromising image quality.", "questions": "1. The fundamental concept presented in this paper is highly dependent on raster order generation. This raises a question about whether the observed attention sink exists in other non-raster autoregressive generation order.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper proposes a novel method, Hawk, which is the first to leverage two-dimensional spatial information for accelerating autoregressive image generation using speculative decoding. \n2. Hawk demonstrates a notable 1.71x speedup over standard AR models on multiple text-to-image benchmarks (COCO2017 and Flickr30K), while effectively preserving both image fidelity (lower FID scores) and diversity (higher CLIP scores). The quantitative results presented in Table 1 provide clear evidence of its performance.\n\nWeakness:\n\n1. It is advisable to display the acceptance rate at each location, similar to Figure 4. This will provide a more straightforward illustration of the effectiveness of the proposed method.\n2. Formatting questions: the equations in this paper lack tags, which makes it difficult to locate the corresponding equations in the context.", "was_revised": true, "final_justification": "The author addressed my concerns in their rebuttal and I maintain my positive rating.", "rebuttal": "We sincerely thank the reviewer for recognizing the timeliness and importance of the problem we aim to address, as well as for acknowledging the strong empirical performance of our proposed method. We deeply appreciate your constructive feedback, which has helped us identify several areas for improvement.\n\nIn response to your suggestion on visualizing spatial acceptance behavior, we fully agree that examining acceptance length across different spatial locations can provide valuable insights into the dynamics of our acceleration process. Accordingly, we plan to include an acceptance heatmap in the “Additional Experimental Results” section of the final version, which will visually demonstrate how acceptance varies across image regions and reinforce the effectiveness of our method. Regarding the formatting issue of missing equation numbers, we apologize for the oversight. We will ensure that all mathematical expressions are properly tagged in the camera-ready version to improve clarity and ease of reference.\n\nIn response to your question about whether the attention sink phenomenon exists in non-raster autoregressive generation orders (which we interpret as generation paradigms that produce complete images progressively rather than token-by-token in raster order, such as VAR or ViDT), we note that similar attention behaviors have indeed been reported. For example, the paper Analysis of Attention in Video Diffusion Transformers observes attention sinking in ViDT models. However, these patterns exhibit some differences from those in traditional autoregressive models. Specifically, attention sinks in ViDTs are typically (1) **spatially random**—tokens associated with sink behavior are scattered rather than clustered in semantically significant regions, and (2) **temporally biased**—they appear more frequently in early latent frames and gradually diminish in later ones. In other words, they show temporal regularity but lack consistent spatial structure. To the best of our knowledge, while attention sinking has been observed in non-raster generation settings, no existing acceleration method has systematically leveraged this behavior for inference speedup. We agree that this is a compelling and underexplored direction, and we thank the reviewer for bringing it to our attention.\n\nOnce again, we sincerely thank the reviewer for recognizing the significance of our problem and for providing constructive and actionable feedback. We warmly welcome any additional questions or suggestions you might have, and would be happy to provide further clarification."}]

Author rebuttal:

Meta-review: This paper introduces Hawk, a novel spatial speculative decoding framework for accelerating autoregressive (AR) text-to-image generation by leveraging 2D spatial dependencies. The method employs dual-directional draft heads (horizontal and vertical) alongside a spatial cache mechanism, achieving a 1.71x speedup over standard AR models while preserving image fidelity and diversity across benchmarks.

#### Strengths

* Tackles the critical issue of slow AR inference by adapting speculative decoding to 2D image structures.
* Strong empirical performance and theoretical grounding, including formal proofs of distribution preservation.
* High practical efficiency (<1% parameter overhead, minimal additional training).
* First approach to exploit spatial context for AR acceleration, making it directly deployable for unified multimodal models.

#### Resolved Issues

1. Theoretical justification (Reviewer g8rp): Proofs confirm that dual-direction speculation preserves output distributions and lowers rejection rates, mitigating concerns about distribution shift.
2. Novelty and scope (Reviewer LJzH): Clarifications distinguished Hawk from non-AR approaches (e.g., ZipAR, PAR) by highlighting its AR-compatible verification mechanism and lightweight training.
3. Comparisons (Reviewer UpDY): Authors committed to include ZipAR baselines and contextualized trade-offs between fidelity and speed.


Reviewer UpDY maintained a borderline stance, citing incremental contribution despite positive shifts from others. Authors are encouraged to incorporate the rebuttal into the final revision and address this lingering concern.


Hawk is a conceptually and empirically solid contribution to AR acceleration with clear implications for multimodal foundation models. The rebuttal effectively resolved methodological and evaluative concerns, reinforcing consensus toward acceptance.
Decision: Accept (poster)