Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding
Authors: Yao Teng
Han Shi
Xian Liu
Xuefei Ning
Guohao Dai
Yu Wang
Zhenguo Li
Xihui Liu
Venue: ICLR 2025 Poster (2025)
Keywords: auto-regressive image generation
acceleration
training-free
image generation
Primary area: generative models

Abstract: The current large auto-regressive models can generate high-quality, high-resolution images, but these models require hundreds or even thousands of steps of next-token prediction during inference, resulting in substantial time consumption. In existing studies, Jacobi decoding, an iterative parallel decoding algorithm, has been used to accelerate the auto-regressive generation and can be executed without training. However, the Jacobi decoding relies on a deterministic criterion to determine the convergence of iterations. Thus, it works for greedy decoding but is incompatible with sampling-based decoding which is crucial for visual quality and diversity in the current auto-regressive text-to-image generation. In this paper, we propose a training-free probabilistic parallel decoding algorithm, Speculative Jacobi Decoding (SJD), to accelerate auto-regressive text-to-image generation. By introducing a probabilistic convergence criterion, our SJD accelerates the inference of auto-regressive text-to-image generation while maintaining the randomness in sampling-based token decoding and allowing the model to generate diverse images. Specifically, SJD facilitates the model to predict multiple tokens at each step and accepts tokens based on the probabilistic criterion, enabling the model to generate images with fewer steps than the conventional next-token-prediction paradigm. We also investigate the token initialization strategies that leverage the spatial locality of visual data to further improve the acceleration ratio under specific scenarios. We conduct experiments for our proposed SJD on multiple auto-regressive text-to-image generation models, showing the effectiveness of model acceleration without sacrificing the visual quality. The code of our work is available here: https://github.com/tyshiwo1/Accelerating-T2I-AR-with-SJD/.

Published as a conference paper at ICLR 2025
ACCELERATING AUTO-REGRESSIVE TEXT-TO-IMAGE GENERATION WITH TRAINING- SPECULATIVE JA-
COBI DECODING
Yao Teng Han Shi²
Xian Liu
Xuefei Ning*
Guohao Dai
Yu Wang*
Zhenguo Li²
Xihui Liu'-
'The University of Hong Kong Huawei Noah's 's Ark Lab *Tsinghua University SShnghai Jiao Tong University IIninigence AI
CUHK
ABSTRACT
The current large auto-regressive models can generate high-quality, high-resolution images, but these models require hundreds or even thousands of steps of next- token prediction during inference, resulting in substantial time consumption. In existing studies, Jacobi decoding, an iterative parallel decoding algorithm, has been used to accelerate the auto-regressive generation and can be executed without training. However, the Jacobi decoding relies on deterministic criterion to determine the convergence of iterations. Thus, it works for greedy decoding but is incompatible with sampling-bas baaed decoding which is crucial for visual quality and diversity the current auto-regressive text-to-image generation. In this paper, we propose training-fii -ree probabilistic parallel decoding algorithm, Speculative Jacobi Decoding (SJD), to accelerate auto-regressive text-to-image generation. By introducing a probabilistic convergence criterion, our SJD accelerates the inference of auto-regressive text-to-image generation while maintaining the randomness in campling-basee based token decoding and allowing the model generate diverse images. Specifically, SJD facilitates the model to predict multiple tokens at each step and accepts tokens based on the probabilistic criterion, enabling the model to generate images with fewer steps than the conventional next-token prediction paradigm. We also investigate the token initialization strategies that leverage the spatial locality of visual data to further improve the acceleration ratio under specific scenarios. We conduct experiments for our proposed SJD on multiple auto-regressive text-to- image generation models, showing the effectiveness of model acceleration without sacrificing the visual quality. The code of our work is available here: https //github. htwpitttu..cmmttttttoto toscm/ysshii woll Acce erat ing-tii-----n T2I BETT-R-aa AR- wi SJ/J//
INTRODUCTION
auto-regressive models enable generative tasks by performing next-token prediction, which is widely used in multiple domains such the language (Bubeck al. 2023), image (Yu al. 2022), and video (Kondratyuk et al. 20233 Wang et al. (2024b) generation. Notably, auto-regressive text-to-image generation models (Ding et al. 202 2021] Ramesh et al. 202T Yu al. 2022) have shown promising results in generating high-quality images. Auto-regressive text-to-image generation models have better potential in scalability and pave the way for native multi-ma multi-modal models (Team) 2024). However, paradigm creates high latency during inference because it necessitates the decoding of tokens in a sequential, token-by-toke manner. Therefore, the models have to sequentially go through hundreds even thousands of forward passes to generate single image. Unlike diffusion models of which the inference acceleration methods have been extensively investigated (Song et al., Luo et al., 2023 Yin et al. 2024b), there has been limited previous work exploring the text-to-imate generation models. Moreover, those auto-regressive models that capable of text-to-image generation typically have several billions of parameters, making the common baining-based generative model acceleration techniques such as self-consistency
the auto-regressoe
2023
acceleration of auto-regressee
"Corresooniing Author

---

Steps: 844 (2. 2.8 Faster) Published as a conference paper at ICLR 2025
Steps: 2357
Steps: 1012 ( Faster)
Steps: 2357
Steps: 959 (2.5 Faster)
Prompt: hawk man with a
Prompt: masterpiece oil painting about starry sky
Steps: 2357
Steps: 1085 (2. 2 x Faster)
Steps: 2357
Prompt: Macro photography transparent drop shape of cat.
Prompt: Corgi dog logo style, simple texture, background, facial- and eye symmetr.
Steps: 2357
Steps: 934 (2. x Faster)
Steps: 2357
Steps: 1075 ( x Faster)
Prompt: most beautiful nnme a rtwork, a most cute anime girl, double exposure, iridesceut sebula galaxy, black background, ethereal glow, bloom, hdr, sigh-quality..
Prompt: cool furry black monkey meditalze tates on the clean wet ground, in the dusk, the goldea sunset shining
ground on
one side and the other side, high-quality, 8K, facial symmmeyyy
We propose Speculative Jacobi Decoding, training-free multi-token prediction algorithm, to accelerate auto-regressive text-to-image generation by reducing the number of model forward passes (denoted as steps) during inference. We perform our algorithm Lumina-mGPT. and the reduced steps are marked in red. The original steps are marked in black.
Figure 1:
distillation computationally expensive (Kou et al. 2024). Therefore, our work aims to accelerate the auto-regressive text-to-image generation models in training-free manner.
An intuitive approach is to enable the auto-regressive models to decode multiple tokens in parallel early research on auto-regressive image generation, Jacobi decoding (Or) tega & Rheinboldt 2000) has been employed to achieve this objective (Song al. 2021). Jacobi decoding is an iterative algorithm starting from a sequence of randomly initialized tokens, and this algorithm be executed directly pre-trained auto-regressive models in training-free way. In each Jacobi iteration, the model performs a single forward pass on the input sequence with a causal mask, thus decoding tokens in parallel. The decoded tokens would converge after multiple iterations of parallel decoding. The criterion for this convergence is defined as follows: the difference between the values of decoded tokens remains within sufficiently small threshold over two consecutive iterations. Since the number of iterations required for convergence is typically smaller than the sequence length and the parallel forward pass runs fast in GPUs, the generation can be accelerated with Jacobi Decoding.
within a forward pass.
However, Jacobi decoding faces significant challenges when applied recent auto-regressive text- to-image generation models. We observe that the recent auto-regressive text-to-image generation models (Liu et al., 2024b Chern et al.) 2024 Sun et al. 2024a) greatly rely on sampling-be decoding with high randomness to generate diverse images. We present the generated images using top-K K sampling with various K values, where larger K indicates higher randomness. As shown in Fig. 2 the model with high randomness in sampling generates images with diverse and high- fidelity details and structures, whereas it outputs monotonous or even incomprehensible images with greedy decoding. Unfortunately, Jacobi decoding with the deterministic criterion of convergence is
2

---

Published as a conference paper at ICLR 2025
incompatible with such highly random sampling (analyzed in Sec. 5.4), i.e., and it cannot accelerate the inference given such sampling decoding.
In this work, we propose to probabilistic Jacobi decoding algorithm to accelerate the inference of auto-regressive text-to-image generation models and to support the sampling decoding methods for those models. We observe that the acceleration of Jacobi decoding relies the assumption that multiple consecutive tokens can be correctly decoded in each Jacobi iteration (shown by green stepped area in Fig. 3). Similar ideas have been applied in other probabilistic algorithms for accelerating the decoding of large language models. For example, in speculative sampling (Leviathan al., 2023 Chen al., 2023), an additional small model is trained for rapidly generating draft sequences, and then the large language model probabilistically accepts subset of draft tokens from left to right. Drawing from the above analysis, in this paper, we directly advance the deterministic Jacobi decoding into a probabilistic algorithm, coined as Speculative Jacobi Decoding (SJD). Our method allows the auto-regressive text-to-image generation models to decode multiple tokens within one forward pass in training-free manner. In SJD, the model computes the conditional probability for a sequence draft tokens with a single forward pass. Then, we define a probabilistic criterion to determine which draft tokens to accept, from left to right. The accepted tokens appended to the fixed pre- pre-filiing sequence. The remaining tokens are concatenated with a set of newly initialized tokens, serving as the draft tokens for the next decoding iteration. Our SJD accelerates the inference of auto-regressive text-to-image generation models without requiring additional training or tuning of separate modules. Moreover, we propose the spatial locality-aware token initialization strategy to accelerate the generation process further.
We perform quantitative and qualitative experiments demonstrate the effectiveness of our method. Results show that our method can accelerate several auto-regressive text-to-image generation models without sacrificing the quality of generated images. For example, it can accelerate Anole (Chern et al., 2024) and Lumina-mGPP mGPT (Liu et al. (024b) by about 2 with almost no loss in visual quality. Moreover, the acceleration ratio can be beyond 3x in certain scenarios containing simple patterns.
To the best of our knowledge, SJD is the first method for accelerating the inference of auto-regressive text-to-image models that rely on sampling decoding. We summarize our contributions as follows:
We propose new probabilistic multi-token decoding algorithm, coined as Speculative Jacobi Decoding (SJD). By improving the previous Jacobi decoding with probabilistic criterion for token acceptance, we can accelerate the recent regressegee exxt-to-image generation models that rely heavily on random token samplers.
* Compared with previous Speculative Decoding to accelerate language models, our approach is training- and does not require training an extra model to predict draft tokens.
Experiments demonstrate that our method can accelerate auto-regressive text-to-image generation by around 2x with almost no sacrifice in visual quality.
2 RELATED WORK
Auto-regressive image generation. Auto-regressive image generation models have two features: cer-tokeen-rredictinn and discrete image tokenization. Early works including PixeICNNs (Van den Oord et al., 2016; Salimans al.) 2017) and PixeISNAIL (Chen et al., 2018) use the auto-regressive strategy to model the image generation with the convolutional neural networks on the discretized pixels. These works generate pixels in the raster-scan ordering or the zigzag ordering. DALL- E (Ramesh et al., 202T) and View (Ding et al., 202T) pave the way for the pipeline of the auto-regressive image generation: discrete autoencoder compresses RGB images into image tokens and a large auto-regressive model makes predictions based on these image tokens. Parti (Yu al. 2022) uses a transformer encoder (Vaswani et al., (017) to provide the textual features for the regressgee model to perform the next image token prediction, thereby achieving text-to- image generation. LlamaGen (Sun et al., 2024a) acts as a class-to-image auto-regressive baseline on ImageNet dataset (Deng et al. 2009). MARS (He et al. 2024) performs multi-modal generation with a mixture of auto regresseeeesive models, where its image model is initialized with the pre-trained large language model and is fine-tuned to perform image generation. Chameleon (Team] 2024 aims to unify all modal tasks with discrete tokens and perform the next token prediction on these tokens with large auto-regressive model. Lumina-mGPT (Liu et al., 2024b) and Anole (Chem et al.)
3

---

Published as a conference paper at ICLR 2025
2024 fine-tune Chameleon for better text-to-image generation. In this paper, we conduct experiments mailly on Lumina-mGPt and Anole to verify the effectiveness of our method.
Acceleration of image generation models. The iterative image generation requires acceleration. For instance, the diffusion model, originally trained on a denoising trajectory with one thousand steps, has been accelerated to perform inference using just dozens
a few steps. Given that the diffusion model has emerged as a leading approach in text-to-image generation (OpenAl 2023 Rombach et al.) 2022 Esser et al.] 2024), most acceleration methods in image generation are built upon it. Many acceleration methods focus on shortening the denoising trajectory by distillation technique (Salimans & Ho 2022 Song et al. 2023 Wang et al., 2024a] Kim et al. [023] Xu et al. 2024, Yin al. 2024b a) while some other studies focus on reducing the computational complexity (Yuan et al.
2024| //hao et al. 2024 Ma et al. 2024). In contrast to the diffusion model, acceleration methods for auto-regressive image generation have not been extensively explored, primarily due to the absence of powerful base models. Jacobi decoding is applied to PixeICNNs for inference acceleration in the early research (Song et al. 2021), yet it lacks careful design for the random token sampling, significantly impacting its acceleration on current auto-regressive models. In this paper, we enhance Jacobi decoding to be compatible with this random sampling. Also, the inference process of each iteration in our method is similar to that of non-auto-regrestee
models (Chang et al. [022: Tian et al., 2024: Li al. 2024b). Nevertheless, unlike these models, our approach only modifies the inference schedule of pre-trained auto-regressive models instead of training a separate non-auto-regressin model, thereby preserving the performance and scalability of the auto-regressive models.
Acceleration of language models. Different from image generation, the auto-regressive paradigm prevails in language processing. A lot of works (Zhou et al.) 2024 Devoto et al., 2024 Liu et al.
2024a c Yang et al. 2024a; DeepSeek-AT 2024; Zhang el al. 2024, Fu et al.) 2024a Li et al. 2024a) focus on compressing the models by weight pruning, activation sparsification, quantization, factorization, but the paradigm of token-by-toker prediction remains unchanged. There are also works fine-tuning the auto-regressive models to predict multiple tokens in parallel with several decoding heads (Gloeckle et al.] 2024). However, these works require more memory to load these additional heads in GPUs The speculative sampling (Leviathan et al., 2023 Chen et al.) 2023 Li et al. 2024c] (unn et al.) (024b) uses small language model to assist the large language model in sequence generation. This model is trained on the same domain as the large model and is small enough for faster generation. It first generates sequence with its own inference paradigm. Then, the large model verifies and samples only one prefix of this sequence to serve as part of the final output by executing single forward pass. The verification phase is well-designed to guarantee that each sampled token theoretically satisfies the conditional probability parameterized by the large model. Jacobi decoding (Song et al., 2021 Santilli et al. 2023) allows the model to iteratively decode multiple tokens in fewer steps than the token counts with the deterministic greedy sampling but without auxiliary modules. CL.LM (Kou et al. 2024 fine-tunes the collected Jacobi trajectories into large language models for acceleration. Lookahead Decoding (Fu et al., (024b) adapts training- free Jacobi decoding in large language models by using a pool of rt-grams obtained via Jacobi iterations with greedy sampling. In this work, we directly adapt the probabilistic verification of speculative sampling into Jacobi decoding to advance it into a probabilistic algorithm without any additional auxiliary designs and training.
3 PRELIMINARIES
3.1 AUTO-REGRESSIVE TEXT-TO-IMAGE GENERATION
The auto-regressive text-to-image imae generation models are composed of three components: a discrete image tokenizer that encodes images into discrete tokens, an auto-regressive transformer-based generator that generates discrete image tokens with eext-token-predictie conditioned on the text prompts, and an image decoder that decodes the predicted image tokens to the images in pixel space. The most time-consuming component for ruto-regressive text-to-image generation is the auto-regressive transformer, and our work aims at accelerating the inference of the auto-regressive transformer to predict discrete image tokens based on text prompts.
During each inference step of the auto-regressive transformer, the model predicts the probability distribution of the next token over the entire vocabulary of the tokenizer (implemented through a softmax classifier) and then samples from this distribution to generate the token. Specifically, given
4

---

Published as a conference paper at ICLR 2025
Previous Tokens
Initial Candidate Tokens
Decoding
Inputs
2 Paratlel Forward
-2
Probability Po(x)e en
X Sample
The results the greedy de- coding (no randomness), top-10, top- 100, and top-2000 sampling (high randomness) of Lumina-mGPT (Liu et al. 2024b). Each row presents the images generated with the same ran-
Figure 2:
Accepted
Outputs
(1+1)
Acceptance
Iterations
Figure 3: The pipeline of the vanilla Jacobi decoding on an auto-regressive model. The prediction with sampling is performed in parallel at each Jacobi iteration. We use different shades of blue to indicate the differences between the tokens that have not been accepted.
dom seeds.
sequence of pre-filled or already decoded tokens (21, z.). the auto regreseeressive model predicts a categorical distribution po (0/001 [r)a)s)) where we denote the input token sequence T2, .. .x;)) as æ1: for simplicity, / denotes the auto-regressive model parameters, and x is the random variable representing the next token (category). Then, a token is sampled according to p=<x[ I1+1, and is subsequently appended auto-regressive generation, the above process starts with sequence of text tokens and special token represent the beginning of image token prediction. To facilitate the generation of diverse images, top-K K sampling is commonly employed as the token sampling strategy for text-to-imane generation.
treated as #2, ... 3i) for the next decoding step. In text-to-image
3.2 JACOBI DECODING
Jacobi decoding deems the ruto-regressive inference as a process of solving the fixed point of a nonlinear equation in triangular system (Song al.) (021). h. This decoding algorithm iteratively performs multi-token decoding and can be executed without fine-tuning or auxiliary modules. We show the specific process of decoding one sequence of tokens in Fig. 3 First, given the previously pre-filled or decoded tokens, we randomly initialize a sequence of candidate tokens. Then, in each iteration, execute one forward pass of the auto-regressive model for all the candidate tokens with a causal mask. The predicted probabilities then generate the tokens via greedy sampling, and these sampled tokens are taken as the inputs of the next iteration. This process can be formulated as: [f+11) = arg max, Pe (x](=())
where i denotes the token index and j denotes the iteration index. The Jacobi decoding process continues iterating until the convergence is reached, as determined by deterministic criterion where these tokens remain unchanged between consecutive iterations.
Discussion. The acceleration of Jacobi decoding derives from an assumption that multiple tokens can be correctly decoded within one forward pass in Jacobi iteration. Fig. 3 Billustrates this scenario, where the accepted tokens (green stepped area) extend beyond the dashed green triangle outline. Specifically, the model accepts two consecutive tokens after the first Jacobi iteration. Thus, it can generate at least four tokens through three forward passes. In the worst case, only three tokens can be generated through three forward passes (Song al. 2021). Note that the number of forward passes in the worst case of Jacobi decoding is equal to that in the original auto-regressive case.
SPECULATIVE JACOBI DECODING
The vanilla Jacobi decoding incorporates a deterministic criterion for determining the convergence, which works well with greedy sampling (no randomness) in language models (Fu et al.) 2024b Kou et al.) 2024). In contrast, in auto-regressive text-to-image generation, randomness plays crucial role in the sampling based decoding process, i.e., higher randomness corresponds to highly diverse details and structures in the generated images. As shown in Fig. 2 we use the text prompt "a mat" to generate images with different sampling strategies including greedy decoding and
Analysis.
5

---

Published as a conference paper at ICLR 2025
Vectors of Probability: Po (=(
Speculative Jacobi Iteration
Inputs
Draft Tokens:
Jacobi Window)
Parallel Forward
For
r-U[0,1], r- min(1, Pela
Vectors of Probability Pe (x1x
Pa
Acceptance:
(+D
Jacobi Convergence
Verification Sampling
Else:
Rejection, Resample i with Calibrated Prob:
x ((+ (po(+lx -Po0-)))))
Then, Sample 1' (++)) Po (x1xx
Accepted
Unaccepted x xf(
New Init
Xt ((os)
Break
TTuss, (1+3) -p (x)x 1 kept for all
Outputs for Next Iter
Figure 4: Overview of one iteration of our speculative Jacobi decoding (SJD). First, sequence of draft tokens and the corresponding probabilities are taken as the inputs. Second, we perform a forward pass with the auto-regressive model the draft tokens, obtaining the probabilities of these tokens. Third, we perform the verification according to these two types of probabilities, accepting a subset of tokens and (re- Jsamplig the remaining tokens. Last, the accepted tokens are appended to the pre-filling tokens and fixed, while the resampled tokens, along with newly initialized tokens, will serve as the draft tokens for the next iteration.
top-K decoding with different values of K. We observe that the generated images would contain more details and diverse structures the K increases. The greedy decoding (K = 1 with no equal probabilities) leads to suboptimal performance with low quality and no diversity in generated images Therefore, random sampling-bee paasiing-baeddd decoding is important to image generation, but the original Jacobi decoding is incompatible with such randomness in sampling.
To address the aforementioned issue, we advance the deterministic Jacobi iteration into new training- free probabilistic parallel decoding algorithm, inspired by speculative sampling (Leviathan et al.. 2023). Specifically, in each iteration, decode multiple tokens in parallel and utilize a probabilistic criterion to accept multiple decoded tokens from the outputs the previous iteration. Moreover, to reduce the number of iterations required for inference, propose new image token initialization strategy incorporating spatial priors.
4.1 SPECULATIVE JACOBI ITERATION
After pre-filling the tokens of text prompts, perform speculative Jacobi decoding for the image tokens. Acknowledging the computational expense of decoding all image tokens simultaneously. we decode multiple tokens per iteration in a blddng-windoww manner, termed the Jacobi window. Our method starts with sequence of initialized candidate tokens, the length of which corresponds to the window size. During each decoding iteration, we predict the token probabilities for the draft token sequence within the current window. Then, subset of these tokens is accepted based probabilistic criterion, and these accepted tokens are added to the fixed pre-filling tokens for the next iteration. The remaining unaccepted tokens resampled for the next iteration. In the next iteration, the Jacobi windows slides to include the unaccepted tokens from the previous iteration plus some newly initialized tokens to maintain the Jacobi window size during decoding. The process of our iteration is illustrated in Fig. 4 Assuming that we have pre-filled or accepted tokens and the Jacobi window size is W. would decode the next W tokens. The iteration be described as follows:
Step 1: At the j-th iteration, we have the input tokens that are either predicted (but not accepted) in the previous (j 1)-th iteration or newly initialized (Sec. 4.2). These tokens serve as the draft tokens ..)
-ntw-1 We denote the probability corresponding to the draft token (6)) as pa (x (1))) This probability is set to be conditioned on the input tokens of
in this iteration, denoted as (xr 2n (6)
())
the previous iteration (Step ensures this setting).

---

Published as a conference paper at ICLR 2025
We execute single forward pass of the auto-regressive model to obtain the conditional probability for the draft tokens in parallel. The probability for x;(2) is denoted as po(=)
Step 2:
We conduct the speculative verification between the conditional probability given the draft tokens from the previous iteration, (= and the conditional probability from the current iteration, ( | ( In this verification process, scan the draft sequence from left to right and determine the acceptance of each token based on a probabilistic threshold. We measure the ratio of the probability conditioned on the draft tokens in the current iteration that conditioned on the tokens from the previous iteration. Intuitively, this ratio measures how well the token is decoded from the previous iteration (the draft token) and whether further decoding is necessary. The acceptance criterion of a token n,(5)
Step 3:
can be formulated as follows:
U+11 - 1j () if r ~ U[0 r min (1. (a( < 1+--1
pe -ل-يي
where is random variable and ([00 1] represents uniform distribution between 0 and 1. If a token meets the above criterion, it is accepted and appended to the pre-filling token sequence for the next iteration, i.e., i.e.,
20(5)) V''= > j. After accepting one token, we continue this scan until a token is rejected. If the token x; is rejected, i.e., the inequality in Equ. 1 is not true, we resample new token by a calibrated distribution:
max(0, max(0.Po(z) ( -P
Po (o(2|=))1111
(j+1)
max(0, Pe (x (1) pe (=(
Then, unlike the vanilla speculative sampling, we do not end this scan, but sample the tokens at the remaining indexes with the conditional probability calculated in this iteration. This sampling process is consistent with the original Jacobi iteration, and the specific process is as follows:
(++1) 36
(o( ())
Vi' >
(3)
can be proven that all the accepted and sampled tokens the Jacobi iteration satisfy x; (6+1)
[ (the proof is in the appendix). This conditional probability (= ze (2)))_)) is exactly the probability predicted by the parallel forward pass on the input draft tokens, and is passed to the next iteration together with the sampled tokens.
Step 4: we append the unaccepted tokens with newly initialized candidate tokens, forming a new Jacobi window with W tokens, as the draft tokens for the next iteration. We use this fixed window size instead of the whole sequence to save the memory usage and accelerate the inference speed.
4.2 TOKEN INITIALIZATION WITH SPATIAL PRIOR
Vanilla Jacobi decoding methods sample the initial candidate tokens from a uniform distribution. However. 2D images exhibit unique characteristics of spatial locality, i.e., spatially adjacent tokens tend to share similar semanties and textures. Leveraging these characteristies for token initialization may enable faster convergence. Considering that auto-regressive models generate image tokens in a raster scan order (from the top-left to the bottom-right in 2D space), we propose the following strategies for initializing new tokens: (a) repeating the previously generated left adjacent token; (b) repeating the previously generated above adjacent token; (c) resampling from the predicted probability from the left adjacent token; (d) resampling from the predicted probability from the above adjacent token. Experimental results demonstrate that these strategies provide greater acceleration than random initialization under certain scenarios.
EXPERIMENTS
5.1 IMPLEMENTATION DETAILS
We experiment with two recent and representative auto-regressive text-to-image generation models. Lumina-mGPT (Liu et al. 2024b) and Anole (Chern et al.) 2024). For Lumina-mGPT (Liu et al.
7

---

Published as a conference paper at ICLR 2025
Table
The evaluation on the validation set of MSCOC02017 with A100. JD: Jacobi decoding.
ISP: initialization with spatial prior. SJD: Speculative Jacobi decoding.
Configuration
Average Latency (.)
Acceleration (1)
FID (L) CLIP-Score (t)
Latency
Step
Lumina-mGPT (Liu et al.
2024b)
87.23s
1.00x
1.00x
30.76
31.29
B JD (Song al. 2021
85.20s
1.02x
1.04x
30.66
31.38
C
SJD
42.73s
2.04x
2.22x
30.85
31.35
w. SJD (ISP)
42.49s
2.05x
2.23x
31.13
31.33
Anole (Chern et al. 2024
48.96s
1.00x
1.00x
28.87
30.59
w. (JD'ISP)
26.18s
1.87x
1.97x
29.14
30.61
Table 2: The evaluation on the validation set of Parti parti-prompt with RTX4090. JD: Jacobi decoding. ISP: initialization with spatial prior. SJD: Speculative Jacobi decoding.
Average Latency (U)
Acceleration (t)
Configuration
CLIP-Score (t)
Latency
Step
Lumina-mGPT (Liu et al. 2024b)
100.69s
1.00x
1.00x
32.13
B w. JD (Song et al. 2021
100.00s
1.01x
1.04x
32.17
C w. SJDD
47.52s
2.12x
2.26x
32.13
D
w. SJD (ISP)
47.35s
2.13x
2.28x
32.06
E Anole (Chern et al. 2024
48.24s
1.00x
1.00x
30.46
(ID(
25.12s
1.92x
2.11x
30.48
2024b), by default,
its 7B version generate 768 x 768 images for evaluation, and measure
the sampling randomness by the value of its top-K logit sampler. Following the basic setting of Lumina-mGPT, K is set to 2000 and the classifier-free guidance weight is set to 3.0. Anole (Chern
et al. 2024) is another 7B auto-regressive generation model finetuned from Chameleon (Team 2024)
that generate 512 512 images.
Metries. For visual quality, we use FID (Heusel et al. 2017 and CLIP-Score (Radford et al. 2021 as the metrics for evaluation. We use the step compression ratio (Fu et al. 2024b): S # generated Takens
decoding steps to show the theoretical acceleration ratio. For each benchmark, we report the average of the step compression ratio on all generated images. We also attach this ratio each image sample in the qualitative comparison of our method with other approaches. Moreover, we also report the latency acceleration of the model forward passes on single GPU for testing the actual speedup.
Benchmark. The parti-promplo Yu et al. 2022) and the validation set of MS-COCO 2017 (Lin et 2014) are taken as the benchmarks of image generation. On parti-prompts, we use the Score and the acceleration of latency and steps excluding FID for evaluation because this benchmark only provides prompts without ground-trun -ruth images.
5.2 QUANTITATIVE RESULTS
As shown in Tab. D and Tab. 2 our speculative Jacobi decoding accelerates the auto-regressive text-to-image generation nearly without sacrificing visual qualities. When comparing our SJD (config C and D) with the vanilla Jacobi decoding (config B) on Lumina-mGPT, we observe that our probabilistic method greatly accelerates the generation by more than 2x while the Jacobi decoding method can provide step compression of about 2x for Anole. We observe
cannot. Moreover, that the token initialization with spatial priors has marginal influence on the speed general image generation. We further analyze the specific scenarios of this modification in our ablation studies,
5.3 QUALITATIVE RESULTS
As shown in Fig. 5 we present the images generated with different configurations. For comparison, we set the same random seed for each image sample. According our observation, the visual qualities of the images generated by different methods are similar, illustrating that our method can keep the visual quality for multiple styles of images. More importantly, our speculative Jacobi decoding with or without our spatial initialization can greatly reduce the inference steps by more than 2x for each case, and thus accelerate the inference process.
8

---

Published as a conference paper at ICLR 2025
Jacobi
Speculative Jacobi Decoding Steps: 1012 (2.3 Faster)
Speculative Jacobi Decoding Spatial Init Steps: 1001 (2. Faster)
Original Steps: 2357
Decoding
Steps: 2257
Faster)
Promst:a
wearing yellow
Professional ligh-qualily BK pholographi
Steps:
Faster)
Steps:
Faster)
Steps: 1049 (2. Faster)
Prompi: Most beautifll gtzl deep
wearing purple clothes symmmmrrrrrr
symmetry. high quality. realistic
Steps:
Steps: 2293 (1.0 Faster)
Steps: (2.2 Faster)
Steps: (2. Faster)
Prompt: a giazt golden Hylng saucer Bring lasers from bottom, soorching grouzd, the most Fanlasy high-qualily photos
Staps: 2357
Staps:
Faster)
Faster)
Steps: 1032 Faster)
Prompt: typica zobra's bady the grassand, not acchsoed by grass, the nebra has only woo cars, dlear strijio patter,, The MOSE Professional qughiuality BK photograph, clear lestune, sharp focus, Exquisite te detzils
Figure 5: The images generated by Lumina-mGPT with different acceleration methods.
5.4 ABLATION STUDIES
We perform ablation studies on Lumina-mGPT 7B. Except for the experiments involving multiple resolutions, we use this model to generate 768 x 768 images for evaluation.
The correlation between the sampling strategy in decoding and the acceleration ratio. We compare the deterministic Jacobi decoding our method under various randomness of the logit sampling. In Fig. 6 We show the correlation between their acceleration ratio and the randomness of logit sampling. According to this figure, method is stable across multiple randomness and can achieve more than 2x step compression ratio. On the contrary, the Jacobi decoding can only accelerate the greedy sampling (top-1 sampling), which is useless for image generation.
The relationship between the image resolution and the acceleration ratio. We employ the 7B Lumina-m to generate images with the resolutions 512 512 (about 1,000 tokens), 768 x 768 (about 2,300 tokens), and 1024 x 1024 (about 4,100 tokens). We calculate the average step compression ratio for each resolution given the same set of text prompts. Then, present these

---

Published as a conference paper at ICLR 2025
Stap Compression
Jacobi Dacoding
Step Compression
Ours
2.5
2.5
2
1.5
1.5
10
2000
1024
Top-K
Resolution
Figure 6: Our method beats the vanilla Jacobi decoding under various sampling randomness.
Figure 7: Higher image resolution can result in a slightly larger acceleration in our method.
Step Compression
Step Compresssion
3.5
2
I Random
1.5
3
0.5
2.5
Window 16 32
Horizontal Vertical Horizontal
Vertical
Sample Sample
Repeat
Repeat
Examples
Figure 8: The acceleration ra- tio is the largest when the Jacobi window size is least 16.
Figure 9; The token initialization strategy impacts the acceler- ation ratio image generation that contains simple and repeat patterns (examples of generated images the right side).
ratios in Fig. The results demonstrate that method is stable across multiple resolutions, ie.. the acceleration on each resolution larger than 2x. Moreover, with higher resolutions, the acceleration can be slightly better. For example. SJD achieves 2.43x acceleration for 1024 x 1024 images.
Studies on the window size of each iteration. As mentioned in Sec. 4.1 we append newly initialized tokens onto the unaccepted tokens in each iteration, fixing the Jacobi window size. Accordingly, we perform the ablation studies on the size of the window. We report the accelcration ratios under various sequence lengths in Fig. 8 The results show that our acceleration ratio reaches almost the maximum when the number of input tokens is greater than or equal to 16 tokens.
Studies on the initialization of candidate tokens. The acceleration ratio of our speculative Jacobi decoding is also related to the application scenarios. For example, when generating images composed of many simple and repeating patterns, token initialization correlated with the already sampled tokens can provide more precise guess than the random initialization. As shown in Fig. 9 adopt an extreme case, the textual prompt "2D logo of pure white box a pure black background" for evaluation. We run the accelerated forward passes ten times with different random seeds for each initialization. The results show that the average step compression with the spatial-prior-awant initialization is much greater than that using the random initialization. Also, Fig. 1 shows that generating 2D logo requires fewer steps than generating images containing exquisite details.
6 CONCLUSION
This paper proposes a new training-free free probabilistic parallel decoding algorithm, called Speculative Jacobi Decoding (SJD), to accelerate auto-regressive text-to-image generation. The sampling-bas decoding is critical for image generation models, which prevents naive Jacobi Decoding from being applied to accelerate uuto-regressive text-to-image generation models. By introducing a probabilistic convergence criterion, our SJD allows the model to iteratively predict-then -henn-smplee multiple tokens in fewer steps than the token counts for auto-regressive lext-to-image generation models with sampling- based decoding rather than greedy decoding. We also propose the spatial-awar token initialization to reduce the number of iterations under specific scenarios. We conduct experiments verify the effectiveness of SJD on multiple auto-regressive text-to-image generation models, and it accelerates models without sacrificing visual quality.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGE
This work is supported by the National Nature Science Foundation of China (No. 62402406) and HKU IDS research Seed Fund. This work is partly supported by HKU Shanghai Intelligent Computing Research Center.
REFERENCES
Emu3 Team BAAL Emu3: Next-token prediction is all you need, 2024. URL https://em/
baai. ac. en/l
Shaojie Bai, Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in neural information processing systems, 32, 2019.
Shaojie Bai, Zhengyang Geng, Yash Savani, and J Zico Kolter. Deep equilibrium optical flow estimation. In Proceedings the IEEE/CVF conference computer vision and pattern recognition, pp. 620-630, 2022.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Túlio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4. CoRR, abs/2303.12213 2023.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11315-111 11325, 2022.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, fean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling, arXiv preprint 01310: 2023.
Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. Pixelsnail: An improved autore- gressive generative model. In International conference machine learning, pp. 864-872. PMLR, 2018.
Ethan Chern, Jiadi Su, Yan Ma, and Pengfei Liu. Anole: An open, autoregressive, native large multimodal models for interleaved image-text generation. arXiv preprint arXiv:2407.06133, 2024.
DeepSeek-AL. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Alessio Devoto, Yu Zhao, Simone Scardapane, and Pasquale Minervini. A simple and effective 1_2 norm-based strategy for kv cache compression. arXiv preprint arXiv:2406.1144 2024.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. Advances in neural information processing systems, 25:19222-19835 2021.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint 2:X3::24033032006 2024.
Tianyu Fu, Haofeng Huang, Xuefei Ning, Genghan Zhang, Boju Chen, Tianqi Wu, Hongyi Wang. Zixiao Huang, Shiyao Li. Shengen Yan, et al. Moa: Mixture of sparse attention for automatic large language model compression. arXiv preprint arXiv:2406. 4909,, 2024a.
Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of Ilm inference using lookahead decoding. arXiv preprint arXiv:2402.02057 2024b.

---

Published as a conference paper at ICLR 2025
Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction. arXiv preprint arXiv:2404.1973 19737, 2024.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, arXiv preprint arXiv:2312.00752, 2023,
Wanggui He, Siming Fu, Mushui Liu, Xierui Wang, Wenyi Xiao, Fangxun Shu, Yi Wang, Lei Zhang, Zhelun Yu, Haoyuan Li, et al. Mars: Mixture of ougo-regressvve models for fine-grained text-to-image synthesis. arXiv preprint arXiv:2407.07614 2024.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Berhaard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-l: A generative world model for autonomous driving, arXiv preprint arXiv:2309. 17880, 2023.
Dongjun Kim, Chich-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Ue- saka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. arXiv preprint arXiv:2310.0227, 2023.
Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. Videopoet: A large language model for zero-shot video generation. arXiv preprint arXiv:2312.1411 141225,, 2023.
Siqi Kou, Lanxiang Hu, Zhezhi He, Zhijie Deng, and Hao Zhang. Cllms: Consistency large language models. arXiv preprint 2rXXi:2403.00833 2024.
LAION. Laion-coco 600m, 2022. URL httppe::/laaooo.aa///ool/aoooooe
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274-19286. PMLR, 2023.
Shiyao Li, Xuefei Ning. Luning Wang, Tengxuan Liu, Xiangsheng Shi, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. Evaluating quantized large language models. International Conference on Machine Leurning, 2024a.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. arXiv preprint arXiv:2406. 1838, 2024b.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401.1500 2024c.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Akide Liu, Jing Liu, Zizheng Pan, Yefei He, Gholamreza Haffari, and Bohan Zhuang. Minicache: Ky cache compression in depth dimension for large language models. arXiv preprint arXiv:2405. 14366. 2024a.
Dongyang Liu, Shitian Zhao, Le Zhuo, Weifeng Lin, Yu Qiao, Hongsheng Li, and Peng Gao. Lumina- mgpt: Illuminate flexible photorealistic text-to-image generation with multimodal generative pretraining, arXiv preprint arXiv:2408.0200 02655, 2024b.
Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Seissorhands: Exploiting the persistence of importance hypothesis for 1lm kv cache compression at test time. Advances in Neural Information Processing Systems, 36, 2024c.
Simian Luo, Yiqin Tan, Longbo Huang. Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arX.v:2310.04378, 2023.
12

---

Published as a conference paper at ICLR 2025
Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deepeache: Accelerating diffusion models for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15762-15772, 2024.
OpenAl. Dalle-2, 2023. URL https:t//ppeaaaii com/ dall-e-2
James M Ortega and Werner C Rheinboldt. Iterative solution of nonlinear equations in several variables. SIAM, 2000.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 8748-8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine Pmlr, 2021.
learning, pp. 8821-8831
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pp. 10674-10685. IEEE, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00:: 2022.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelenn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv: 2017.
Andrea Santilli, Silvio Severino, Emilian Postolache, Valentino Maiorca, Michele Maneusi, Riccardo Marin, and Emanuele Rodolà. Accelerating transformer inference for translation via parallel decoding. arXiv preprint arXiv:2305. J0427, 2023.
Yang Song. Chenlin Meng, Renjie Liao, and Stefano Ermon. Accelerating feedforward computation via parallel nonlinear equation solving. In International Conference on Machine Learning, pp- 9791-9800. PMLR, 2021.
Yang Song, Prafulla Dhariwal, Mark Chen, and llya Sutskever. Consistency models. arXiv preprint arXiv:2303.01466 2023.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2400.06555, 2024a.
Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. Spectr: Fast speculative decoding via optimal transport. Advances in Neural Information Processing Systems, 36, 2024b.
Chameleon Team. Chameleon: Mixed-modal carly-fusion foundation models. arXiv:2405.098//, 2024.
arXiv preprint
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint arXiv:2404.022 02905, 2024.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelenn decoders. Advances in neural information processing systems, 29, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Fu- Yun Wang. Zhaoyang Huang, Alexander William Bergman, Dazhong Shen, Peng Gao, Michael Lingelbach, Keqiang Sun, Weikang Bian, Guanglu Song. Yu Liu, et al. Phased consistency model. arXiv preprint a 240. 2024a.
13

---

Published as a conference paper at ICLR 2025
Shuai Wang, Yao Teng, and Limin Wang. Deep equilibrium object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6296-6306, 2023.
Yuqing Wang, Tianwei Xiong, Daquan Zhou, Zhijie Lin, Yang Zhao, Bingyi Kang, Jiashi Feng, and Xihui Liu. Loong: Generating minute-level long videos with autoregressive language models. arXiv preprint arXiv:2410.02757, 2024b.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint (rXiv:2306.0944 2023.
Chen Xu, Tianhui Song, Weixin Feng, Xubin Li, Tiezheng Ge, Bo Zheng. and Limin Wang- Accelerat- ing image generation with sub-path linear approximation model. arXiv preprint arXiv:2404. 13903, 2024.
Dongjie Yang, XiaoDong Han, Yan Gao, Yao Hu, Shilin Zhang, and Hai Zhao. Pyramidinfer: Pyramid kv cache compression for high-throughput Ilm inference. arXiv preprint arXiv:2405. 12532, 2024a.
Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. arXiv preprint arXiv:2406. 06484, 2024b.
Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and William T Freeman. Improved distribution matching distillation for fast image synthesis. arXiv preprint 2rXiv:2405. 2024a.
Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6613-6623, 2024b.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint ar 2rXiv:2206.1078, 2022.
Zhihang Yuan, Pu Lu, Hanling Zhang, Xuefei Ning, Linfeng Zhang, Tianchen Zhao, Shengen Yan, Guohao Dai, and Yu Wang. Ditfastattn: Attention compression for diffusion transformer models. Advances in Neural Information Processing Systems, 2024.
Yichi Zhang, Bofei Gao, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, Wen Xiao, et al. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling, arXiv preprint 1xXiv:2406.02066 2024.
Tianchen Zhao, Xuefci Ning, Tongcheng Fang. Enshu Liu, Guyue Huang, Zinan Lin, Shengen Yan, Guohao Dai, and Yu Wang. Mixdq: Memory-efficient few-step text-to-image diffusion models with metric-decoupled mixed precision quantization. ECCV, 2024.
Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang. Zhihang Yuan, Xiuhong Li, al. A survey on efficient inference for large language models. arXiv preprint 4404v:2404. 14294, 2024.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A PROOFS
Theorem (The correctness of speculative Jacobi decoding) The token sampled in each speculative Jacobi iteration satisfies po 13--)> 1) where x denotes a token, j denotes the index of iteration, i denotes the token index, and 0 denotes the auto-regressive model parameters.
Proof. The main process of speculative Jacobi iteration decomposed into two cases: (a) obtaining the token sampled in the previous iteration and then accepting according to acceptance probability; (b) rejecting the sampled token and resampling a new token according to a calibrated probability. Thus, like the proof of the vanilla speculative sampling (Leviathan et al., 2023), to prove the correctness of speculative Jacobi decoding, we verify that the conditional probability of a token sampled following the above two cases, alongside the manually designed acceptance and resampling probability, remains Pa pa(r) (r)))
For simplicity, by default, we omit the token index i and denote the token category of (f) as x. We denote the condition of token ( the j-th Jacobi iteration (i.e., the tokens "is-1 and model weights 0) to Jy. Thus, the condition of the (j 1)-th Jacobi iteration is denoted as J-11- Thus, we can denote the probability po po(=<=12 as Jf) and denote po (<r [e|20_111 as p( T-1)- We use random boolean variable to represent the acceptance. With these notations, the proof is follows:
First, the acceptance probability on the token category xr is manually set as follows:
p
p(r is true 2 Jy, = min
(4)
p(
and the calibrated resampling probability subsequent to the rejection is set as follows:
- p(=
p(a/r is false, Jz. Jz-1) = Σn max p(x1|J)-1))
(5)
Next, make assumption that Jj and are conditionally independent given Jz-1:
(6)
This assumption is reasonable due the properties of the Jacobi iteration and the auto-regressive paradigm, i.e., with the observation of the sequence x i:a-1* one of the tokens in x 1:--1 (denoted as ( can be determined by x f(a 0) (I < i) where the function indicates the prediction-thes wwwnontttte-----iini of auto-regressive models, the variable 2.()) is redundant as one of the conditions in the probability p(Jrla,, pr, TT Thus, Equ. (6) is reasonable.
(j-1)
Then, with Bayes rule, Equ. (6) has the following equivalence:
+ p(o)J,,T-1))= =(a)))))))]J]]] p(x(J
Hence, according to Equ. 4 and Equ. the probability that token category x sampled in the previous iteration and subsequently accepted can be computed as:
p(r is true, xI [(J,,
= p(x: J, J--1) p(r is truel Jee Jj.JJ-J
= =p(z (((x)),11) min (1, p(z(TT)
(8)
p(x|J
15

---

Published as a conference paper at ICLR 2025
Steps: 8193 3515 (2.3 Faster)
Steps: 8193 + 3561 (2. x Faster)
Steps: 8193 + 3472 Faster)
Figure 10: The images generated by Emu3 BAAI 2024 with our acceleration method.
With Equ. 8 we can calculate the probability of rejection with the law of total probability on the token categories:
p(r is false Jy. Jg-1 = - p(r is true J--1)
Σ plr true, x' Jn-))
Σ p(a'(J,) min (1)ppp(("()),((((()__)_
Σ max (0x p(a'lJg-1)}-
Then, with Equ. 5 and Equ. 9 we get the following equation:
p(arr is false, Jn, p(r is false Jy, J--1)
max (0. p(x[[]]])) p(x] E. max 1() Emax((,p(('133) p(a' (('1(T3-1)) (10)
max(0,p(z|TT)) - p(x) J-1)}
Since
Ya E DUERDER a= min(a, b) max b),
(11)
we can decompose p( as follows:
p(*|J,) = min min max - p(x(J,-1)}
(12)
With Equ. (8. Equ. 10 and Equ. 12). we can compute:
p(z]J,)
min [T,, p(=]),-))} + max p( p( J-11}
-(=])T-11) p(r is true xr J-1)
(13)
+ p(r is false JT,) p(0// is false, J,, J).
According to Equ. 13). the conditional distribution p()(TT)) can exactly represent (a) obtaining the token sampled in the previous iteration and then accepting according to acceptance probability: (b) rejecting the sampled token and resampling new token according to a calibrated probability. In conclusion, the token sampled in each speculative Jacobi iteration satisfies po(x (
B MORE QUALITATIVE RESULTS
In Fig. 11 we showcase more generated images with Lumina-mGPT accelerated by our method. These results illustrate that method functions well the image contents including humans, animals, and landscapes. Recently, new powerful regr-ssgre model, Emu3 (BAAI (BAAT[024)) (024), has been released. We also explore method on Emu3 for text-to-image generation, and find it
16

---

Published as a conference paper at ICLR 2025
Figure 11: The images generated by Lumina-mGPT Liu et al. 2024b) with our acceleration method.
still leads great step compression, shown in Fig. 10 We leave the quantitative results of Emu3 for future work.
We have included additional qualitative results for Lumina-mGP? and Anole in the supplementary material of the revised paper. specifically in Fig. [5] and Fig. [6] and we report both the steps and latency. According the reported latency and step compression in these figures, our SJD outperforms other decoding methods while maintaining visual quality. Furthermore, spatial token initialization can further enhance the acceleration of our SJD. Additionally, we observe that Anole exhibits significantly higher image diversity compared to Lumina-mGPT. Despite the fixed random seed, it remains challenging for Anole to generate similar images due to the differences among the decoding methods.
C INFERENCE LATENCY
In addition to reporting the step compression ratio, we also report the practical latency of SJD servers. We set the batch size as 1 for testing, and report the latency of the accelerated Lumina-mGPT 7B excluding the pre- and post-processing operations. For 768 x 768 image generation (the number of generated tokens 2357), we perform the experiments on one RT' 4090 GPU. For 1024 X 1024 image generation (the number of generated tokens is at least 4165), we perform the experiments on one A 100 GPU. In these settings, the latency of Lumina-mGHH
156.76
768
150.00
1024
Latency(s)
100.70
100.00
76.53
least
47.36
0.00
Baseline
Ours
with and without method is presented in Fig. [2 Our method significantly accel- erates the auto-regressive image generation.
Figure 12: The latency of Lumina-mGPT generating 768 x 768 and 1024 x 1024 images without or with our method.
17

---

Published as a conference paper at ICLR 2025
Table 3: The evaluation LlamaGen Sun et al. 2024a) with or without our method on
MSCOC02017
(Lin et al. 2014 4) and Parti parti-prompt Yu et al. 2022).
Accelcration (1)
Dataset
Configuration
FID (L)
CLIP-Score (1)
Latency
Step
LlamaGen-stagel
1.00x
1.00x
28.54
30.87
Jaaaaeen-ttagel + Ours
1.56x
1.63x
29.00
30.82
COCO
LlamaGen-stagge
1.00x
1.00x
56.21
28.26
LlamaGen-stagee + Ours
1.54x
1.63x
57.02
28.33
LlamaGen-stagell
1.00x
1.00x
30.22
LlamaGen-stagee + Ours
1.57x
1.73x
30.29
Parti
LlamaGen-stagee
1.00x
1.00
28.14
LlamaGen-stage22 + Ours
1.62x
1.69x
28.16
D MORE QUANTITATIVE RESULTS
More Results. We further compare SJD other decoding methods Anole (Chern et al. 2024). As shown in Tab, and Tab. | consistent with the results on Lumina- Commina-mGPT, SJD with spatial token initialization create larger acceleration ratios than other decoding methods Anole, and the cost of visual quality is small. In addition to Anole (Chern et al.) 2024) and Lumina-mGPT (Liu al. 2024b), we evaluate our method with the text-to-image LlamaGen (Sun al.) 2024a). This model adopts a two-stage training strategy: (a) stagel: LlamaGen is first trained on subset of LAION-COCO (LAION) 2022) (50M 256 256 images); (b) stage2: it is then fine-tuned on 10M high aesthetic quality internal data with a resolution of 512 x 512. In Tab. B we evaluate our method with the two versions of LlamaGen. The results show that our method can still accelerate this model without sacrificing the visual quality. However, in comparison to the experiments conducted on Lumina-mGPT and Anole, the acceleration ratios on LlamaGen are lower. We hypothesize that this discrepancy attributed to the model size, some existing works for multi-token prediction demonstrate that the model size has a great influence on the effectiveness of acceleration (Gloeckle We leave this investigation to future work.
al. 2024).
More results about visual quality. We take the CLIP-Score and the human preference score (HPSv2) (Wu et al. (023)) the metries for evaluating the visual quality for our ablation studies (the step compression ratios are reported in Sec.B.44 We present the results in Tab. 7 Tab. 8 Tab, 9 and Tab. T0 From Tab. 7 given any K values in the top-K sampling strategies, we observe that the human preferences are also not much different among the original auto-regressive decoding, the original Jacobi decoding, and our SJD.
Perplexity. We also compare the perplexities between SJD and other decoding methods on Lumina- mGPT, as detailed in Tab. 6 Since the perplexities influenced by the sampling strategies (Hu et al. 2023), 3 we report the perplexities under various K values. Given an identical K value, the perplexities between our method and other decoding methods are close. Furthermore, we note that K 2000 results in perplexity higher than that of large language models (Gu & Dao 2023 Yang et al. (0244) on language processing tasks. Despite this high value, the text-to-image auto-regressive model can still generate high-quality images. This indicates that image generation can tolerate wide range of image tokens.
Statistics of model outputs. We compute the statistics of the logarithm of the token probability for both auto-regressive decoding and our method. The average and standard deviation all image tokens presented in Tab. [1 The results demonstrate that the image tokens accepted by method exhibit similar statistics to those accepted by the original rego-segeessive decoding. Consequently, method generally does not mistakenly accept tokens with lower probabilities.
E VISUALIZATION OF ACCELERATION IN 2D SPACE
We visualize the impact of multi-token prediction in 2D space. As illustrated in Fig. [3] the color each long strip area represents the length of accepted tokens from that area, with darker colors indicating longer sequences of accepted tokens, ie.,, higher acceleration. We observe that
18

---

Published as a conference paper at ICLR 2025
Figure 13: The visualization of the accelerated tokens on 2D space.
high acceleration tends to occur in the background, particularly on the left and right sides of images. Additionally, while some high acceleration is observed on foreground objects, it is relatively sparse 2D space.
F ANALYSIS ON THE EFFECTIVENESS OF OUR METHOD
This section analyzes the acceleration mechanism of our speculative Jacobi decoding in image generation. empirically find that this acceleration stems from the resampling of unaccepted tokens. Specifically, some tokens are continuously resampled (i.e., their positions within the entire sequence are reused for multiple forward passes) according to Equ. (3) over iterations until they are accepted. refer to this process of a token being continuously resampled by Equ. (3) (except the possible rejection resampling) as refinement, following the terminology in fixed-poing iteration (Bai al. 2019 2022: Wang et al. 2023). Consequently, Equ. (3) is the main operation of every refinement step. In the following paragraphs, we explore the influences of this refinement.
For clarity and simplicity,
The acceleration originates from the refinement of unaccepted tokens. In verification phase, there are three treatments for the tokens: acceptance, rejection, and refinement, corresponding Equ. (I). Equ. (2). and Equ. (3). respectively. We empirically find that only the first treatments are insufficient to support acceleration. We conduct the following experiment to demonstrate that our method makes hard achieve acceleration without refinement: when we deactivate the refinement (i.e., using the newly initialized tokens to replace the unaccepted tokens as the draft tokens in the next iterations), we observe that the model requires over two thousand forward passes to generate images rather than one thousand forward passes. Although our token initializations with spatial prior (e.g., horizontal repeat) are slightly better than the random token initialization replacing the unaccepted tokens, its performance is still much worse than directly refining the unaccepted tokens. The examples of the generated images under such setting are shown Fig. 14 This phenomenon illustrates that the acceleration of our method originates from refining unaccepted tokens.
G QUALITATIVE ANALYSIS OF IMAGE RANDOMNESS ON OUR METHOD
Like Fig. 2 we also examine the image randomness with both the auto-regressive decoding and our speculative Jacobi decoding. As shown in Fig. 17 first, we find that SJD does introduce some randomness into image generation (the random variable r in Equ. (T). so the images generated with auto-regressive decoding cannot exactly align those generated with SJD, even when the random seed is fixed. Therefore, in Fig. [7] given a column, two images with the same K value cannot be exactly identical. However, in general, the diversity of the set of images is not influenced much. Fig. [7] we present the images generated based on three textual prompts. Given the same prompt and K value from top-K sampling, the model with different decoding methods generates images with many similarities. For example, when K 2000, for the first prompt "an apple of strange color", the images the identical columns show the apples with similar color patterns and styles, Also, for the third prompt "pumpkin the table", the frequency of faces carved on the pumpkins is similar for these two decoding methods.
Moreover, the K value in top-K sampling still dominates the image randomness in terms of texture, color, and local structure details. With larger K, the image details about textures, colors, and local structures increase. Such image randomness still largely comes from the random token sampling.
19

---

Published as a conference paper at ICLR 2025
Examples of Images Generated (Without Refinement; With random initialization)
Latency: 107.35s
Latency: 107.49s
Latenty: 107.96s Steps:
Latency: 107.84s Steps:
Lateney: 107.800
Steps:
Steps: 2356
Steps:
Examples of Images Generated (Without Refinement; With horizontal repeat initialization)
Latancy: Steps: 2273
Latency: 0..88
Latency, Steps: 2285
Latency: 95.60s
Letency:
Strgs: 2254
Steps: 2300
Stops: 2291
Examples of Images Generated (With Refinement; With horizontal repeat initialiization only for New Tokens)
Latency: 42.49s
Latency: 42.52s
Latency: 41.95s
Latency: 41.23s
Latency: 42.44s
Steps: 1032
Steps: 1049
Steps: 969
Steps: 947
Steps: 1029
Figure 14: Ablation studies acceleration mechanism: examples of images generated by our SJD without or with refining unaccepted tokens. When the refinement defined by Equ. (3) is NOT applied (i.e., using the newly initialized tokens to replace the unaccepted tokens as the draft tokens in the next iterations), there is almost no acceleration (though one our token initializations with spatial prior, horizontal repeat, can slightly reduce the steps in these images), This illustrates that refining unaccepted tokens are essential to the acceleration mechanism SJD.
H ANALYSIS ON FAILURE CASES
As shown in Fig. 18 when generating the images with exquisite details, although auto-regressive decoding can produce artifacts, SJD seems to generate continuous tokens that cause the artifacts, as highlighted by the red boxes in this figure. The pre-trained auto-regressive model is not sufficiently robust to handle such complex images. Consequently, it may mistakenly accept a sequence of draft tokens that contain artifacts.
I LIMITATION AND FUTURE WORK
Since our speculative Jacobi decoding training-free, the accelerated model itself still not spe- cialized for multi-token prediction. Therefore, the acceleration ratio has the potential to be further improved. In the future, we believe that fine-tuning the regresseeee models for fast image genera- tion is promising direction. Also, acceleration is important for long-sequent generation, like video generation. Since videos contain more redundancy than images, the initialization of candidate tokens should be carefully designed if applying speculative Jacobi decoding to video generation.
20

---

Published as a conference paper at ICLR 2025
Table 4: The evaluation Anole on the validation set of MSCOC02017. JD: Jacobi decoding. ISP:
initialization with spatial prior. SJD: Speculative Jacobi decoding.
Average Latency (J)
Acceleration (t)
Configuration
FID (4) CLIP-Score (1)
Latency
Step
Anole (Chern al. 2024
48.96s
1.00x
1.00x
28.87
30.59
w. JD (Song el al. 2021)
47.60s
1.03x
1.06x
29.34
30,64
w. SJD
27.08s
1.81x
1.94x
29.04
30.54
D w. SJD (ISP)
26.18s
1.87x
1.97x
29.14
30.61
Table 5: The evaluation of Anole the validation set of Parti-prowe prompt. JD: Jacobi decoding. ISP: initialization with spatial prior. SJD: Speculative Jacobi decoding.
Average Latency (4)
Acceleration (t)
Configuration
CLIP-Score (1)
Latency
Step
A
Anole (Chern et al. 2024)
48.24s
1.00x
1.00x
30.46
B w. JD (Song et al. 2021
44.65s
1.08x
1.14x
30.57
C
SJD
26.77s
1.80x
2.00x
30.55
D
w. SJD (ISP)
25.12s
1.92x
2.11x
30.48
J BROADER IMPACTS
Image generation offers extensive utility in helping users, designers, and artists produce fantastic content. Nonetheless, these models could be exploited create deceptive content. Thus, it is crucial for the users including researchers and developers to acknowledge the potential negative social impact of image generation models.
21

---

Published as a conference paper at ICLR 2025
Table 6: The comparison of perplexity on Lumina-mGPT.
Perplexity with Top-K K sampling
Configuration
K 10
K
100
2000
Lumina-mGPT (Liu et al. 2024b)
7.31
43.37
204.06
w. JD (Song al. 2021)
7.20
43.85
197.64
w. SJD
7.34
43.87
217.96
w. SJD (ISP)
7.26
44.03
199.70
Table 7: CLIP-Score of various decoding methods on Lumina-mGPT with different top- values. The image qualities for Jacobi Decoding and our method correspond Fig. (6] The image qualities for Auto-regression
only for the comparison in this table. Note that the image quality score with greedy sampling is extremely poor, as this setting leads to meaningless images for lot of prompts (analyzed in Fig. 2
Decoding Methods
Sampling
CLIP-Score
HPSv2
Auto-regression
Top-1 Sampling
26.40
0.1976
Auto-regression
Top-10 Sampling
32.83
0,2950
Auto-regression
Top-100 Sampling
32,41
0.3020
Auto-regression
Top-2000 Sampling
32.00
0,2965
Jacobi Decoding
Top-1 Sampling
26.34
0.1413
Jacobi Decoding
Top-10 Sampling
32.75
0.2960
Jacobi Decoding
Top-100 Sampling
32,46
0,3089
Jacobi Decoding
Top-2000 Sampling
31.68
0.3103
Ours
Top-1 Sampling
26.16
0,1695
Ours
Top-10 Sampling
32.27
0,2942
Ours
Top-100 Sampling
32.65
0,2977
Ours
Top-2000 Sampling
31.83
0.3020
Table 8: CLIP-Scores on Lumina-mGPT with various resolutions. The image qualities of our method under different settings correspond Fig. The image qualities for Auto-regression are only for the comparison in this table.
Decoding Methods
Resolutions
CLIP-Score
HPSv2
Auto-regression
512
29.49
0,2503
Auto-regression
768
32.00
0.2965
Auto-regression
1024
31.41
0.2961
Ours
512
29.69
0.2558
Ours
31.83
0.3020
Ours
1024
31.11
0.2935
Table 10: CLIP-Score of our method on Lumina-
Table 9: CLIP-Scort of method Lumina-mGPT with various Jacobi win- dow sizes. The image qualities correspond to Fig.
mGPT with various token initialization when gen- erating images with simple patterns. The image qualities correspond Fig, 9
Token Initialization
CLIP-Scom
HPSv2
Window Size
CLIP-Score
HPSv2
Horizontal Sample
31.52
0.2567
32,00
0.2965
Vertical Sample
30.91
0.2622
4
31.91
0.3046
Horizontal Repeat
31.17
0.2616
16
31.83
0.3020
Vertical Repeat
31.15
0.2651
32
31.55
0.3045
Random
31.37
0.2681
22

---

Published as a conference paper at ICLR 2025
Jacobi Decoding
Speculative Jacobi Decoding
Speculative Jacobi Decoding Spatial Init
Original
Latency:91., 91.38s Stops: 2857
Latenoy: 91, 10s 11.0 Faster)
Latenoy: 47, 13s (1.9 x Faster)
Latenay: 45, 97s (2.0 Faster) Stops: 1094 (2. Fawer)
Staps: 2293 (1.0 Fower)
Stops: 1115 (2.1 Fower))
Prompt: giant golden- hained indige gale of
Latency: 90.05s Stepe:
Latency: 88.31s 11.0 Faster)
Latency: 43.09s (2.1 x Faster)
Latency:441 41.75s (2.2 Faster) Steps: Fower)
Stepe: Fewer)
Stepe: Fewwrr)
Prompt: Portrail of the most besutiful Asian women, Weering dress and beaddress decoraled with pescock feelbrr
Latency:
Latency: 99.74s Faster)
Latency: (2.1 Fasteri
Latency: 40.39s (2. Faster)
Steps: 2357
Steps: 2270 11.0 X Fewer)
Steps: 1009 (2.3 Fewer)
Steps: 962 (2. Fewer)
Prompt:AM painting Iady
Latency: 92 92.475
Latency: 89.84s [1.0 Faster)
Latency: 42.31s (2.3 x Faster) Steps: Fewwrr)
Latency: 41.95s (2 3 Faster) Steps: Fewer)
Steps:
Steps: Fewer)
Prompt: One the forest illuminsted by gloomy strong light, must Professional high quality SK photograph
Latency: 33.63s Staps: 2357
Latenoy: 92, 00s Faster)
Latenoy: 42. (2.3 Faster)
Latency: 41.23s (2.3 Faster)
Staps: 2304 (1.0 Fower)
Stops: 1015 (2.3 x Fowrr)
Steps: 947 (2.5 Fewer)
Prompt: Atlants, most Fantasy high-quality photos
Figure 15: The qualitative comparison of different decoding methods on Lumina-m mGGT 2024b).
(Liu et al.
23

---

Published as a conference paper at ICLR 2025
Jacobi Decoding
Speculative Jacobi Decoding
Speculative Jacobi Decoding Spatial
Original
Latemcy: 47.90s Steps: 1026
Laterny:: 46.36s (1.0 Faster)
Latency; 24.14s (2.0 x Faster) Steps: 483 Fewer)
Latenny: 23.55s (2.0 x Faster)
Steps: 955 (1.1 Fewer)
Steps: (2. Fewer)
Prompt: Generate an image of cute Beagle dog.
Latency. 47.54s Steps: 1026
Latenoy: 48.638 63.63s (1.0 Faster)
Latency: 26.71s (1.8 x Faster)
Latency: 21.38s (2. x Faster) Steps: (2, Fewer)
Steps: 969 (1.1 Fewer)
Steps:548 Fewer)
Prompt: G amerate an image a square box of cooldies
Latenoy: 47.40s
Latency: 46.21s 1100x Faster) Steps: 966 [1.1 x Fewer)
Letency: 25.918 (1.8 x Faster) Steps: 526 (2.0 Fewer)
Latency: 25.05s (1.9 Faster) Steps: 505 (2. Fewer)
Steps: 1026
Prompt: Gemerate an image a blue Porsche 356 pariad in frunt ofa yellow brick wall.
Figure 16: The qualitative comparison of different decoding methods on Anole (Chern et al. 2024). Considering the high image diversity of Anole, although the random seed is fixed, it still hard for Anole to generate similar images with different decoding methods.
Table 11: The comparison of token statisties on Lumina-mGPT.
Logarithm of Token Probability Average
Decoding Methods
Standard Deviation
Auto-regression
4.8950
2.3457
Ours
-49007
2.3275
24

---

Published as a conference paper at ICLR 2025
'Teere apple
table
T. besutiful cat the pillow"
Pumplin talle Hallowers'
K-10
K-100
K-2000
Auto-regressive Decoding with Top-K sampling
K-10
K-2000
Speculative Jacobi Decoding with Top-K sampling
Figure 17: Comparing our method to the original auto-regressive decoding on the image randomness. First, considering the random variable in SJD, given a column, two images with the same K value cannot be exactly identical. Second, changing the decoding method from auto-regression to SJD has little influence the image diversity for each prompt (e.g., given K 2000 for each decoding method, the color patterns and styles of the generated apples are similar, and the frequency of the carved faces pumpkins is also similar). Third, the top-K sampling still dominates the image randomness about texture, color, and local structure details. The images in each column share single random seed.
Prompt: Image of downtown sidewalks, and tall skyscrapers.
downtown street in Tokyo at night, with neon signs, crowded
Speculative Jacobi Decoding
Figure 18: Failure Cases. complex image scenarios. our method generates some continuous tokens that result in artifacts, as highlighted by the red boxes. The pre-trained model inaccurately accepts large sequence of the tokens that cause the artifacts.
25


Reviews:
[{"review_id": "KDXNOgjpC6", "reviewer": "Reviewer_8dQw", "summary": "This paper proposes Speculative Jacobi Decoding (SJD) for accelerating the inference of auto-regressive (AR) image generation. SJD introduces a probabilistic acceptance criterion, using the ratio of the probability conditioned on the current draft tokens to that of the previous iteration for speculative verification, enabling faster convergence while preserving diversity in sampling-based decoding. This approach reduces the number of steps required, improving inference speed without compromising image quality.", "questions": "Please see the Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper adapts speculative decoding to auto-regressive (AR) image generation, achieving substantial speed improvements with minimal impact on image quality, addressing a key efficiency challenge in this area.\n\n2. The proposed method demonstrate performance gains on state-of-the-art models, including Lumina-mGPT and Anole, underscoring its applicability and effectiveness.", "weaknesses": "1. Need for analysis on image randomness in SJD. The randomness in SJD introduces variability in generated images, but there is limited analysis on the extent of this randomness. Unlike speculative decoding in the language domain, which offers some guarantees on reproducibility, SJD would benefit from theoretical bounds or deeper empirical analysis to clarify the extent of variation in its results.\n\n2. Demonstrating similar speed-up and preservation of image quality on additional models, such as LlamaGen, would strengthen this paper.\n\n3. Need for additional qualitative evaluation: Additional qualitative results, as in Fig. 5, comparing original images with those generated by the proposed method, along with steps and latency comparisons on Lumina-mGPT and Anole, would be beneficial. Particularly, a detailed discussion on representative failure cases and their causes is needed to clarify potential issues in image quality.\n\n3. While this paper tends to emphasize step counts in terms of speed-up, latency is a more accurate metric for evaluating it. Therefore, using latency as the primary measure would enable a clearer comparision. \n\n4. Improved clarification in Fig. 4 would enhance understanding of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E9XRGGZRPX", "reviewer": "Reviewer_3y1d", "summary": "Recent auto-regressive models generate quality images but require thousands of next-token prediction steps during inference, which is significant in terms of inference speed. To address this, this paper proposes Speculative Jacobi Decoding (SJD), a training-free probabilistic parallel decoding algorithm. SJD introduces a probabilistic convergence criterion, allowing for sampling-based token decoding that maintains diversity and accelerates inference while overcoming the limitation of original Jacobi Decoding - deterministic, greedy decoding, which limits diversity to be incorporated into the image generation framework. This approach enables multiple token predictions per step in a probabilistic way (i.e., inserting randomness), reducing required steps while preserving image quality.", "questions": "- The acceptance is determined if it improves prediction compared to the conditions of the previous Jacobi iteration. I wonder if the conditional probability given the draft tokens from the previous iteration is too low; it may have a chance to accept tokens even if it still has a very low conditional probability but a little bit higher than that, which may result in inaccurate token acceptance.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Clear motivation about the limitation of existing performant auto-regressive T2I generation models and Jocobi decoding strategy.\n- Achieving about 2 times faster inference latency with fewer iteration steps, with a small drop in image quality compared to original T2I backbones.\n- This approach can be applied to multiple different auto-regressive T2I generation models.\n- Provide several meaningful ablations to better understand the robustness of the proposed methods with varying scales of hyperparameters.", "weaknesses": "- The proposed approach requires over-computation in their Jacobi iteration. It computes conditional prediction scores for all tokens in the window but only accepts several earlier tokens by design - so resources used for computing later tokens are the first unaccepted tokens and later ones, even if some of them are technically accepted. This is inevitable since next-token prediction should be conditioned on all previous tokens that should be accepted (except for the beginning) and cannot allow some undesired holes of unaccepted tokens. While the approach proceeds these unaccepted tokens to new draft tokens for the next iterations, the effectiveness of this technique is not studied throughout the paper. At least, comparing the impact of this reuse of over-computed tokens with several token initializations with spatial priors (and random as well) would be meaningful.\n\n- Initialization with spatial prior (ISP) doesn't show real acceleration in both terms of latency and steps compared to SJD w/o ISP. Figure 9 shows some different behavior, but the gap looks marginal in the main table (Table 1). I would say this is not appealing in terms of idea since this is from a random idea without strong grounding, less relevant to the overall story of this paper, and the performance is also less impactful.\n\n- Analyses/ablations look incomplete. For Figures 6,7,8 and 9, I believe this should also be meaningful to provide quantitative performance of image quality rather than reporting only step compression rate. Reporting only the efficiency of the proposed approach does not give meaningful insight.\n\n- The experimental results with Anole in Table 1 also look incomplete. I think more rows (i.e., B and C in Lumina-mGPT cases) are necessary for this table.\n\n- (minor) typo: Each column presents --> Each row presents", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VTFQMd8LEW", "reviewer": "Reviewer_EsPc", "summary": "SJD iteratively decodes multiple tokens in parallel within a sliding window. The model computes the conditional probability for a sequence of draft tokens in a single forward pass. A probabilistic criterion then determines which tokens to accept, adding them to the fixed pre-filling sequence. Unaccepted tokens are resampled for the next iteration. The spatial locality-aware token initialization strategy leverages the spatial relationships in images for initializing new tokens. SJD effectively accelerates auto-regressive text-to-image generation models, achieving approximately 2× speedup for Lumina-mGPT and Anole without significant visual quality degradation.", "questions": "Q1: Could the authors provide the conparison of perplex between SJD and other decoding methods?\n\n\nQ2: I'm curious about how many image samples are used to compute the FID scores. Could the authors provide more evaluation details?\nI doubt that the authors have use some tricks in evalution.  For example, the image quantity maybe different to compared method\nQ3: Could the author provide more  results on small  models?\nNot everyone has access to large models, it's more convicing to conduct experiments on small models", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper introduces reject sampling to accelerate auto-regressive text-to-image generation.\nSJD enables the model to predict multiple tokens at each step, accepting them based on a probabilistic criterion, reducing the number of inference steps compared to conventional next-token-prediction.\nThe paper proposes a spatial locality-aware token initialization strategy to further enhance acceleration in specific scenarios.\nExperiments on multiple auto-regressive text-to-image generation models demonstrate SJD's effectiveness in accelerating inference without compromising visual quality.", "weaknesses": "1) SJD relax the greedy acceptence criteron of next token by reject sampling. Such acceleration will sample results with smaller probabilities and JD is also able to update multiple tokens at a time.  And relaxation of acceptence criteron lead to suboptimal results. Moreover, this paper is infact very simple with little experiments. It's more like a trick rathor than a paper.\n2) The results that JD only accelerate t2i models by a rate of 1.04 is not convicing, which is much worse than results provided by JD and other papers.\n3) The evaluation details are not provided. As far as I'm concerned, SJD may leads The nolvelty seems incremental. to suboptimal image quality and better image diversity. Hence, the FID score would be better if the image quality decrease a little and the image diversity increase.\n4) There are too little models tested. It's better to apply SJD to more large models and small models.\nNot that, I'm not familar to large model accelaration, I will consider other reviews' comments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)