Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding

Authors: Yao Teng, Han Shi, Xian Liu, Xuefei Ning, Guohao Dai, Yu Wang, Zhenguo Li, Xihui Liu
Primary area: generative models
Keywords: auto-regressive image generation, acceleration, training-free, image generation

Abstract:
The current large auto-regressive models can generate high-quality, high-resolution images, but these models require hundreds or even thousands of steps of next-token prediction during inference, resulting in substantial time consumption. In existing studies, Jacobi decoding, an iterative parallel decoding algorithm, has been used to accelerate the auto-regressive generation and can be executed without training. However, the Jacobi decoding relies on a deterministic criterion to determine the convergence of iterations. Thus, it works for greedy decoding but is incompatible with sampling-based decoding which is crucial for visual quality and diversity in the current auto-regressive text-to-image generation. In this paper, we propose a training-free probabilistic parallel decoding algorithm, Speculative Jacobi Decoding (SJD), to accelerate auto-regressive text-to-image generation. By introducing a probabilistic convergence criterion, our SJD accelerates the inference of auto-regressive text-to-image generation while maintaining the randomness in sampling-based token decoding and allowing the model to generate diverse images. Specifically, SJD facilitates the model to predict multiple tokens at each step and accepts tokens based on the probabilistic criterion, enabling the model to generate images with fewer steps than the conventional next-token-prediction paradigm. We also investigate the token initialization strategies that leverage the spatial locality of visual data to further improve the acceleration ratio under specific scenarios. We conduct experiments for our proposed SJD on multiple auto-regressive text-to-image generation models, showing the effectiveness of model acceleration without sacrificing the visual quality. The code of our work is available here: https://github.com/tyshiwo1/Accelerating-T2I-AR-with-SJD/.

Decision: accept (poster)
Reviewer scores: 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1730018187604

thread_type:
official_review

<thread_object>
creator: reviewer_espc

<content>
summary:
SJD iteratively decodes multiple tokens in parallel within a sliding window. The model computes the conditional probability for a sequence of draft tokens in a single forward pass. A probabilistic criterion then determines which tokens to accept, adding them to the fixed pre-filling sequence. Unaccepted tokens are resampled for the next iteration. The spatial locality-aware token initialization strategy leverages the spatial relationships in images for initializing new tokens. SJD effectively accelerates auto-regressive text-to-image generation models, achieving approximately 2× speedup for Lumina-mGPT and Anole without significant visual quality degradation.

strengths:
This paper introduces reject sampling to accelerate auto-regressive text-to-image generation.
SJD enables the model to predict multiple tokens at each step, accepting them based on a probabilistic criterion, reducing the number of inference steps compared to conventional next-token-prediction.
The paper proposes a spatial locality-aware token initialization strategy to further enhance acceleration in specific scenarios.
Experiments on multiple auto-regressive text-to-image generation models demonstrate SJD's effectiveness in accelerating inference without compromising visual quality.

weaknesses:
1) SJD relax the greedy acceptence criteron of next token by reject sampling. Such acceleration will sample results with smaller probabilities and JD is also able to update multiple tokens at a time.  And relaxation of acceptence criteron lead to suboptimal results. Moreover, this paper is infact very simple with little experiments. It's more like a trick rathor than a paper.
2) The results that JD only accelerate t2i models by a rate of 1.04 is not convicing, which is much worse than results provided by JD and other papers.
3) The evaluation details are not provided. As far as I'm concerned, SJD may leads The nolvelty seems incremental. to suboptimal image quality and better image diversity. Hence, the FID score would be better if the image quality decrease a little and the image diversity increase.
4) There are too little models tested. It's better to apply SJD to more large models and small models.
Not that, I'm not familar to large model accelaration, I will consider other reviews' comments.

questions:
Q1: Could the authors provide the conparison of perplex between SJD and other decoding methods?


Q2: I'm curious about how many image samples are used to compute the FID scores. Could the authors provide more evaluation details?
I doubt that the authors have use some tricks in evalution.  For example, the image quantity maybe different to compared method
Q3: Could the author provide more  results on small  models?
Not everyone has access to large models, it's more convicing to conduct experiments on small models
</content>
</thread_object>

</thread>
<thread>
creation date:
1730601892912

thread_type:
official_review

<thread_object>
creator: reviewer_3y1d

<content>
summary:
Recent auto-regressive models generate quality images but require thousands of next-token prediction steps during inference, which is significant in terms of inference speed. To address this, this paper proposes Speculative Jacobi Decoding (SJD), a training-free probabilistic parallel decoding algorithm. SJD introduces a probabilistic convergence criterion, allowing for sampling-based token decoding that maintains diversity and accelerates inference while overcoming the limitation of original Jacobi Decoding - deterministic, greedy decoding, which limits diversity to be incorporated into the image generation framework. This approach enables multiple token predictions per step in a probabilistic way (i.e., inserting randomness), reducing required steps while preserving image quality.

strengths:
- Clear motivation about the limitation of existing performant auto-regressive T2I generation models and Jocobi decoding strategy.
- Achieving about 2 times faster inference latency with fewer iteration steps, with a small drop in image quality compared to original T2I backbones.
- This approach can be applied to multiple different auto-regressive T2I generation models.
- Provide several meaningful ablations to better understand the robustness of the proposed methods with varying scales of hyperparameters.

weaknesses:
- The proposed approach requires over-computation in their Jacobi iteration. It computes conditional prediction scores for all tokens in the window but only accepts several earlier tokens by design - so resources used for computing later tokens are the first unaccepted tokens and later ones, even if some of them are technically accepted. This is inevitable since next-token prediction should be conditioned on all previous tokens that should be accepted (except for the beginning) and cannot allow some undesired holes of unaccepted tokens. While the approach proceeds these unaccepted tokens to new draft tokens for the next iterations, the effectiveness of this technique is not studied throughout the paper. At least, comparing the impact of this reuse of over-computed tokens with several token initializations with spatial priors (and random as well) would be meaningful.

- Initialization with spatial prior (ISP) doesn't show real acceleration in both terms of latency and steps compared to SJD w/o ISP. Figure 9 shows some different behavior, but the gap looks marginal in the main table (Table 1). I would say this is not appealing in terms of idea since this is from a random idea without strong grounding, less relevant to the overall story of this paper, and the performance is also less impactful.

- Analyses/ablations look incomplete. For Figures 6,7,8 and 9, I believe this should also be meaningful to provide quantitative performance of image quality rather than reporting only step compression rate. Reporting only the efficiency of the proposed approach does not give meaningful insight.

- The experimental results with Anole in Table 1 also look incomplete. I think more rows (i.e., B and C in Lumina-mGPT cases) are necessary for this table.

- (minor) typo: Each column presents --> Each row presents

questions:
- The acceptance is determined if it improves prediction compared to the conditions of the previous Jacobi iteration. I wonder if the conditional probability given the draft tokens from the previous iteration is too low; it may have a chance to accept tokens even if it still has a very low conditional probability but a little bit higher than that, which may result in inaccurate token acceptance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730777424715

thread_type:
official_review

<thread_object>
creator: reviewer_8dqw

<content>
summary:
This paper proposes Speculative Jacobi Decoding (SJD) for accelerating the inference of auto-regressive (AR) image generation. SJD introduces a probabilistic acceptance criterion, using the ratio of the probability conditioned on the current draft tokens to that of the previous iteration for speculative verification, enabling faster convergence while preserving diversity in sampling-based decoding. This approach reduces the number of steps required, improving inference speed without compromising image quality.

strengths:
1. This paper adapts speculative decoding to auto-regressive (AR) image generation, achieving substantial speed improvements with minimal impact on image quality, addressing a key efficiency challenge in this area.

2. The proposed method demonstrate performance gains on state-of-the-art models, including Lumina-mGPT and Anole, underscoring its applicability and effectiveness.

weaknesses:
1. Need for analysis on image randomness in SJD. The randomness in SJD introduces variability in generated images, but there is limited analysis on the extent of this randomness. Unlike speculative decoding in the language domain, which offers some guarantees on reproducibility, SJD would benefit from theoretical bounds or deeper empirical analysis to clarify the extent of variation in its results.

2. Demonstrating similar speed-up and preservation of image quality on additional models, such as LlamaGen, would strengthen this paper.

3. Need for additional qualitative evaluation: Additional qualitative results, as in Fig. 5, comparing original images with those generated by the proposed method, along with steps and latency comparisons on Lumina-mGPT and Anole, would be beneficial. Particularly, a detailed discussion on representative failure cases and their causes is needed to clarify potential issues in image quality.

3. While this paper tends to emphasize step counts in terms of speed-up, latency is a more accurate metric for evaluating it. Therefore, using latency as the primary measure would enable a clearer comparision. 

4. Improved clarification in Fig. 4 would enhance understanding of the proposed method.

questions:
Please see the Weaknesses
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes Speculative Jacobi Decoding, a training-free probabilistic parallel decoding algorithm for accelerating auto-regressive text-to-image generation. The core novelty lies in introducing a probabilistic convergence criterion, which allows for faster decoding without sacrificing visual quality or diversity. The authors conduct multiple experiments demonstrating speed-ups across multiple state-of-the-art models while maintaining image quality. Overall, this work makes a meaningful contribution to auto-regressive visual generative models, and thus I recommend acceptance.