Title: Hawk: Leveraging Spatial Context for Faster Autoregressive Text-to-Image Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Autoregressive image generation

Inference acceleration

Spatial speculative decoding
Abstract: Autoregressive (AR) image generation models are capable of producing high-fidelity images but often suffer from slow inference due to their inherently sequential, token-by-token decoding process. Speculative decoding, which employs a lightweight draft model to approximate the output of a larger AR model, has shown promise in accelerating text generation without compromising quality. However, its application to image generation remains largely underexplored. The challenges stem from a significantly larger sampling space, which complicates the alignment between the draft and target model outputs, coupled with the inadequate use of the two-dimensional spatial structure inherent in images, thereby limiting the modeling of local dependencies. To overcome these challenges, we introduce Hawk, a new approach that harnesses the spatial structure of images to guide the speculative model toward more accurate and efficient predictions. Experimental results on multiple text-to-image benchmarks demonstrate a 1.71× speedup over standard AR models, while preserving both image fidelity and diversity.


Peer reviews:
comments: SUMMARY:
This paper introduces Hawk, a novel approach designed to accelerate autoregressive text-to-image generation by leveraging the two-dimensional spatial structure inherent in images. The core idea is to enhance speculative decoding by incorporating both horizontal and vertical spatial information, which helps improve the alignment between draft and target models, leading to faster inference without compromising image quality.

STRENGTHS AND WEAKNESSES:
Strengths:

1. The paper proposes a novel method, Hawk, which is the first to leverage two-dimensional spatial information for accelerating autoregressive image generation using speculative decoding. 
2. Hawk demonstrates a notable 1.71x speedup over standard AR models on multiple text-to-image benchmarks (COCO2017 and Flickr30K), while effectively preserving both image fidelity (lower FID scores) and diversity (higher CLIP scores). The quantitative results presented in Table 1 provide clear evidence of its performance.

Weakness:

1. It is advisable to display the acceptance rate at each location, similar to Figure 4. This will provide a more straightforward illustration of the effectiveness of the proposed method.
2. Formatting questions: the equations in this paper lack tags, which makes it difficult to locate the corresponding equations in the context.

QUESTIONS:
1. The fundamental concept presented in this paper is highly dependent on raster order generation. This raises a question about whether the observed attention sink exists in other non-raster autoregressive generation order.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

comments: SUMMARY:
This paper addresses the slow inference issue in autoregressive (AR) text-to-image generation models, noting that while speculative decoding accelerates text generation, its application to images is underexplored due to larger sampling spaces and unexploited 2D spatial dependencies. The authors propose Hawk, a framework that leverages horizontal and vertical spatial contexts via dual draft heads, constructing a unified sampling pool by merging horizontal speculations with cached vertical predictions. Empirically, Hawk achieves a 1.71× speedup over standard AR models on benchmarks like COCO2017 and Flickr30K, preserving image fidelity and diversity.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The problem is timely and important.
2. The empirical performance seems good.

Weaknesses:
1. This paper does not consider state-of-the-art Autoregressive (AR) image generation models, e.g., VAR. Moreover, the proposed technique cannot be applied to other types of generation models, such as diffusion models, flow-matching, and consistency models.
2. The proposed technique does not introduce many innovations on top of the vanilla version of speculative decoding (the current 1D --> 2D may not be novel enough).
3. Given 2, the experiments are not extensive enough. The authors should consider state-of-the-art generative fundation models learned on significantly larger datasets (e.g., at least at the level of Stable Diffusion).

QUESTIONS:
See Weaknesses

LIMITATIONS:
See Weaknesses
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-01

comments: SUMMARY:
This paper introduces Hawk, an autoregressive T2I framework. It presents a new speculative decoding method considering the two-dimensional spatial context of images. It uses spatial draft heads (horizontal and vertical) for parallel decoding. It uses the Lumina-mGPT as the base model to verify the effectiveness of the proposed method. According to the experiments, it outperforms related methods (Medusa and LANTERN) slightly. The core insight is that leveraging spatial dependencies in parallel decoding and improving the acceptance rate of speculative sampling in image generation.

STRENGTHS AND WEAKNESSES:
Strengths
1. It is easy to follow the motivation and the main idea. The predication redundancy of vanilla AR is clear, and this paper introduces an improved speculative decoding method to tackle the redundancy.
2. The implementation is easy to follow with pseudo code. And the overall presentation is clear.

Weaknesses
1. The method is similar to ZipAR and NAR (https://arxiv.org/abs/2503.10696). Please identify their differences in the main paper. This is the biggest problem. I will change my rating after reading the response.
2. The improvements of results for vanilla AR are limited, compared with other parallel decoding methods, such as ZipAR, NAR, and PAR (https://arxiv.org/abs/2412.15119).
3. Comparisons with mask modelling methods and diffusion methods are missing.

QUESTIONS:
The training time is too short. Is the model converged?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a spatial speculative decoding framework that leverages both horizontal and vertical draft heads to accelerate autoregressive text-to-image generation. By explicitly leveraging the two-dimensional spatial structure of images, the proposed method achieves substantial inference speedup while preserving image fidelity.\n\nSTRENGTHS AND WEAKNESSES:\n### Strenths\n\n- This paper addresses an important practical challenge of slow sequential inference in AR image generation.\n\n- The proposed method employs dual-direction draft heads and spatial caches that explicitly align with the two-dimensional image structure, offering an interesting way to leverage spatial dependencies.\n\n### Weaknesses\n\n- The contribution of the proposed method is largely an extension of Medusa-style speculative decoding into a second spatial dimension.\n\n- The paper lacks a formal theoretical justification for why vertical speculative heads, despite involving longer dependency paths, exhibit lower performance degradation.\n\n- The paper does not provide a formal analysis of how the expanded 2D sampling space affects acceptance rates or output distribution.\n\n- The proposed method inherits known speculative decoding issues like sampling bias after rejection.\n\nQUESTIONS:\n- Does expanding the sampling space via dual-direction speculation provide theoretically justified gains in acceptance probability and sampling efficiency, or does it primarily amplify speculative variance absent formal guarantees?\n\n- How does the proposed spatial speculative framework control for potential shifts in the output distribution, especially under repeated rejection and reweighting, and is there a formal calibration or uncertainty analysis?\n\n- In scenarios where spatial local correlations are less critical for prediction, could the computational overhead introduced by dual-direction speculative sampling outweigh the gains in speed or acceptance efficiency, thereby reducing overall cost-effectiveness?\n\nLIMITATIONS:\nThis paper lacks formal analysis of how 2D sampling impacts acceptance or output calibration, inherits known speculative biases, and may be inefficient when local spatial correlations are weak.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-10"}