Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding

## Abstract

Abstract The current large auto-regressive models can generate high-quality, high-resolution images, but these models require hundreds or even thousands of steps of next-token prediction during inference, resulting in substantial time consumption. In existing studies, Jacobi decoding, an iterative parallel decoding algorithm, has been used to accelerate the auto-regressive generation and can be executed without training. However, the Jacobi decoding relies on a deterministic criterion to determine the convergence of iterations. Thus, it works for greedy decoding but is incompatible with sampling-based decoding which is crucial for visual quality and diversity in the current auto-regressive text-to-image generation. In this paper, we propose a training-free probabilistic parallel decoding algorithm, Speculative Jacobi Decoding (SJD) , to accelerate auto-regressive text-to-image generation. By introducing a probabilistic convergence criterion, our SJD accelerates the inference of auto-regressive text-to-image generation while maintaining the randomness in sampling-based token decoding and allowing the model to generate diverse images. Specifically, SJD facilitates the model to predict multiple tokens at each step and accepts tokens based on the probabilistic criterion, enabling the model to generate images with fewer steps than the conventional next-token-prediction paradigm. We also investigate the token initialization strategies that leverage the spatial locality of visual data to further improve the acceleration ratio under specific scenarios. We conduct experiments for our proposed SJD on multiple auto-regressive text-to-image generation models, showing the effectiveness of model acceleration without sacrificing the visual quality.

## 1 Introduction

Auto-regressive models enable generative tasks by performing next-token prediction, which is widely used in multiple domains such as the language (Bubeck et al., ), image (Yu et al., ), and video (Kondratyuk et al., ) generation.
Notably, auto-regressive text-to-image generation models (Ding et al., ; Ramesh et al., ; Yu et al., ) have shown promising results in generating high-quality images. Auto-regressive text-to-image generation models have better potential in scalability and pave the way for native multi-modal models (Team, ).
However, the auto-regressive paradigm creates high latency during inference because it necessitates the decoding of tokens in a sequential, token-by-token manner.
Therefore, the models have to sequentially go through hundreds or even thousands of forward passes to generate a single image.
Unlike diffusion models of which the inference acceleration methods have been extensively investigated (Song et al., ; Luo et al., ; Yin et al., ), there has been limited previous work exploring the acceleration of auto-regressive text-to-image generation models.
Moreover, those auto-regressive models that are capable of text-to-image generation typically have several billions of parameters, making the common training-based generative model acceleration techniques such as self-consistency distillation computationally expensive (Kou et al., ).
Therefore, our work aims to accelerate the auto-regressive text-to-image generation models in a training-free manner.

An intuitive approach is to enable the auto-regressive models to decode multiple tokens in parallel within a forward pass.
In the early research on auto-regressive image generation, Jacobi decoding (Ortega & Rheinboldt, ) has been employed to achieve this objective (Song et al., ).
Jacobi decoding is an iterative algorithm starting from a sequence of randomly initialized tokens, and this algorithm can be executed directly on pre-trained auto-regressive models in a training-free way.
In each Jacobi iteration, the model performs a single forward pass on the input sequence with a causal mask, thus decoding tokens in parallel.
The decoded tokens would converge after multiple iterations of parallel decoding.
The criterion for this convergence is defined as follows: the difference between the values of decoded tokens remains within a sufficiently small threshold over two consecutive iterations.
Since the number of iterations required for convergence is typically smaller than the sequence length and the parallel forward pass runs fast in GPUs, the generation can be accelerated with Jacobi Decoding.

However, Jacobi decoding faces significant challenges when applied to recent auto-regressive text-to-image generation models.
We observe that the recent auto-regressive text-to-image generation models Liu et al. (); Chern et al. (); Sun et al. () greatly rely on sampling-based decoding with high randomness to generate diverse images.
We present the generated images using top-$K$ sampling with various $K$ values, where a larger $K$ indicates higher randomness.
As shown in Fig. 3, the model with high randomness in sampling generates images with diverse and high-fidelity details and structures, whereas it outputs monotonous or even incomprehensible images with greedy decoding.
Unfortunately, Jacobi decoding with the deterministic criterion of convergence is incompatible with such highly random sampling (analyzed in section 5.4), i.e., and it cannot accelerate the inference given such sampling decoding.

Figure: Figure 1: We propose Speculative Jacobi Decoding, a training-free multi-token prediction algorithm, to accelerate auto-regressive text-to-image generation by reducing the number of model forward passes (denoted as steps) during inference. We perform our algorithm on Lumina-mGPT, and the reduced steps are marked in red. The original steps are marked in black.
Refer to caption: x1.png

In this work, we propose a probabilistic Jacobi decoding algorithm to accelerate the inference of auto-regressive text-to-image generation models and to support the sampling decoding methods for those models.
We observe that the acceleration of Jacobi decoding relies on the assumption that multiple consecutive tokens can be correctly decoded in each Jacobi iteration (shown by green stepped area in Fig. 3). Similar ideas have been applied in other probabilistic algorithms for accelerating the decoding of large language models.
For example, in speculative sampling (Leviathan et al., ; Chen et al., ), an additional small model is trained for rapidly generating draft sequences, and then the large language model probabilistically accepts a subset of draft tokens from left to right.
Drawing from the above analysis, in this paper, we advance the deterministic Jacobi decoding into a probabilistic algorithm, coined as Speculative Jacobi Decoding (SJD). Our method allows the auto-regressive text-to-image generation models to decode multiple tokens within one forward pass in a training-free manner.
In SJD, the model computes the conditional probability for a sequence of draft tokens with a single forward pass. Then, we define a probabilistic criterion to determine which draft tokens to accept, from left to right. The accepted tokens are appended to the fixed pre-filling sequence. The remaining tokens are concatenated with a set of newly initialized tokens, serving as the draft tokens for the next decoding iteration. Our SJD accelerates the inference of auto-regressive text-to-image generation models without requiring additional training or tuning of separate modules.
Moreover, we propose the spatial locality-aware token initialization strategy to accelerate the generation process further.

We perform quantitative and qualitative experiments to demonstrate the effectiveness of our method.
Results show that our method can accelerate several auto-regressive text-to-image generation models without sacrificing the quality of generated images.
For example, it can accelerate Anole (Chern et al., ) and Lumina-mGPT (Liu et al., ) by about $2\times$ with almost no loss in visual quality.
Moreover, the acceleration ratio can be beyond $3\times$ in certain scenarios containing simple patterns.

To the best of our knowledge, SJD is the first method for accelerating the inference of auto-regressive text-to-image models that rely on sampling decoding. We summarize our contributions as follows:

- •
We propose a new probabilistic multi-token decoding algorithm, coined as Speculative Jacobi Decoding (SJD), to speed up the inference of auto-regressive image generation. Our approach addresses the problem that previous Jacobi decoding cannot be applied to recent auto-regressive text-to-image generation models, which rely on sampling-based decoding rather than greedy decoding.
- •
Compared with previous Speculative Decoding to accelerate language models, our approach is training-free and does not require training an extra model to predict draft tokens.
- •
Experiments demonstrate that our method can accelerate auto-regressive text-to-image generation by around $2\times$ with almost no sacrifice in visual quality.

## 2 Related Work

Auto-regressive image generation.
Auto-regressive image generation models have two features: next-token-prediction and discrete image tokenization.
Early works including PixelCNNs (Van den Oord et al., ; Salimans et al., ) and PixelSNAIL (Chen et al., ) use the auto-regressive strategy to model the image generation with the convolutional neural networks on the discretized pixel space. These works generate the pixels in the raster-scan ordering or the zigzag ordering.
DALL-E (Ramesh et al., ) and CogView (Ding et al., ) pave the way for the pipeline of the auto-regressive image generation:
A discrete autoencoder trained to compress RGB images into image tokens and a large auto-regressive model trained to make predictions based on these image tokens.
Parti (Yu et al., ) uses a transformer encoder (Vaswani et al., ) to provide the textual features for the auto-regressive model to perform the next image token prediction, thereby achieving text-to-image generation.
LlamaGen (Sun et al., ) acts as a class-to-image auto-regressive baseline on ImageNet dataset (Deng et al., ).
MARS (He et al., ) performs multi-modal generation with a mixture of auto-regressive models, where its image model is initialized with the pre-trained large language model and is fine-tuned to perform image generation.
Chameleon (Team, ) aims to unify all multi-modal tasks with discrete tokens and perform the next token prediction on these tokens with a large auto-regressive model.
Anole (Chern et al., ) and Lumina-mGPT (Liu et al., ) fine-tune Chameleon for better text-to-image generation.
In this paper, we conduct experiments mainly on Lumina-mGPT and Anole to verify the effectiveness of our method.

Acceleration of image generation models.
The iterative image generation requires acceleration.
For instance, the diffusion model, originally trained on a denoising trajectory with one thousand steps, has been accelerated to perform inference using just dozens or even a few steps.
Given that the diffusion model has emerged as a leading approach in text-to-image generation (OpenAI, ; Rombach et al., ; Esser et al., ), most acceleration methods in image generation are built upon it.
Many acceleration methods focus on shortening the denoising trajectory by distillation technique (Salimans & Ho, ; Song et al., ; Wang et al., ; Kim et al., ; Xu et al., ; Yin et al., ; ) while some other studies focus on reducing the computational complexity (Yuan et al., ; Zhao et al., ; Ma et al., ).
In contrast to the diffusion model, acceleration methods for auto-regressive image generation have not been extensively explored, primarily due to the absence of powerful base models.
Jacobi decoding is applied to PixelCNNs for inference acceleration in the early research (Song et al., ), yet it lacks a careful design for the random token sampling, significantly impacting its acceleration on current auto-regressive models. In this paper, we enhance Jacobi decoding to be compatible with this random sampling.
Also, the inference process of each iteration in our method is similar to that of non-auto-regressive models (Chang et al., ; Tian et al., ; Li et al., ). Nevertheless, unlike these models, our approach only modifies the inference schedule of pre-trained auto-regressive models instead of training a separate non-auto-regressive model, thereby preserving the performance and scalability of the auto-regressive models.

Acceleration of language models.
Different from image generation, the auto-regressive paradigm prevails in language processing.
A lot of works (Zhou et al., ; Devoto et al., ; Liu et al., ; ; Yang et al., ; DeepSeek-AI, ; Zhang et al., ; Fu et al., ; Li et al., ) focus on compressing the models by weight pruning, activation sparsification, quantization, factorization, but the paradigm of token-by-token prediction remains unchanged.
There are also works fine-tuning the auto-regressive models to predict multiple tokens in parallel with several decoding heads (Gloeckle et al., ). However, these works require more memory to load these additional heads in GPUs.
The speculative sampling (Leviathan et al., ; Chen et al., ; Li et al., ; Sun et al., ) uses a small language model to assist the large language model in sequence generation. This model is trained on the same domain as the large model and is small enough for faster generation.
It first generates a sequence with its own inference paradigm. Then, the large model verifies and samples only one prefix of this sequence to serve as part of the final output by executing a single forward pass.
The verification phase is well-designed to guarantee that each sampled token theoretically satisfies the conditional probability parameterized by the large model.
Jacobi decoding (Song et al., ; Santilli et al., ; Fu et al., ; Kou et al., ) allows the model to iteratively decode multiple tokens in fewer steps than the token counts with the deterministic greedy sampling but without auxiliary modules.
In this work, we adapt the probabilistic verification of speculative sampling into Jacobi decoding, thereby advancing it into a probabilistic algorithm.
Additionally, unlike existing speculative decoding methods, our method utilizes the intermediate results of Jacobi iterations as draft sequences without requiring any additional training.

## 3 Preliminaries

### 3.1 Auto-regressive Text-to-image Generation

The auto-regressive text-to-image generation models are composed of three components: a discrete image tokenizer that encodes images into discrete tokens, an auto-regressive transformer-based generator that generates discrete image tokens with next-token-prediction conditioned on the text prompts, and an image decoder that decodes the predicted image tokens to pixel-space images. The most time-consuming component for auto-regressive text-to-image generation is the auto-regressive transformer, and our work aims at accelerating the inference of the auto-regressive transformer to predict discrete image tokens based on text prompts.

During each inference step of the auto-regressive transformer, the model predicts the probability distribution of the next token over the entire vocabulary of the tokenizer (implemented through a softmax classifier) and then samples from this distribution to generate the token.
Specifically, given a sequence of pre-filled or already decoded tokens $(x_{1},x_{2},\cdots,x_{i})$, the auto-regressive model predicts a categorical distribution $p_{\theta}(x|{\bm{x}}_{1:i})$, where we denote the input token sequence $(x_{1},x_{2},\cdots,x_{i})$ as ${\bm{x}}_{1:i}$ for simplicity, $\theta$ denotes the auto-regressive model parameters, and $x$ is the random variable representing the next token (category).
Then, a token is sampled according to $p_{\theta}(x|{\bm{x}}_{1:i})$, treated as $x_{i+1}$, and is subsequently appended to $(x_{1},x_{2},\cdots,x_{i})$ for the next decoding step.
In text-to-image auto-regressive generation, the above process starts with a sequence of text tokens and a special token to represent the beginning of image token prediction.
To facilitate the generation of diverse images, top-$K$ sampling is commonly employed as the token sampling strategy for text-to-image generation.

Figure: Figure 2: The results of the greedy decoding (no randomness), top-$10$, top-$100$, and top-$2000$ sampling (high randomness) of Lumina-mGPT (Liu et al., ). Each column presents samples with the same prompt but different random seeds.
Refer to caption: x2.png

### 3.2 Jacobi Decoding

Jacobi decoding deems the auto-regressive inference as a process of solving the fixed point of a nonlinear equation in a triangular system (Song et al., ).
This decoding algorithm iteratively performs multi-token decoding and can be executed without fine-tuning or auxiliary modules.
We show the specific process of decoding one sequence of tokens in Fig. 3.
First, given the previously pre-filled or decoded tokens, we randomly initialize a sequence of candidate tokens.
Then, in each iteration, we execute one forward pass of the auto-regressive model for all the candidate tokens with a causal mask.
The predicted probabilities then generate the tokens via greedy sampling, and these sampled tokens are taken as the inputs of the next iteration. This process can be formulated as: $x_{i}^{(j+1)}=\arg\max_{x}p_{\theta}(x|{\bm{x}}_{1:i-1}^{(j)}),$
where $i$ denotes the token index and $j$ denotes the iteration index.
The Jacobi decoding process continues iterating until the convergence is reached, as determined by a deterministic criterion where these tokens remain unchanged between consecutive iterations.

Discussion.
The acceleration of Jacobi decoding derives from an assumption that multiple tokens can be correctly decoded within one forward pass in Jacobi iteration.
Fig. 3 illustrates this scenario, where the accepted tokens (green stepped area) extend beyond the dashed green triangle outline.
Specifically, the model accepts two consecutive tokens after the first Jacobi iteration. Thus, it can generate at least four tokens through three forward passes.
In the worst case, only three tokens can be generated through three forward passes (Song et al., ).
Note that the number of forward passes in the worst case of Jacobi decoding is equal to that in the original auto-regressive case.

## 4 Speculative Jacobi Decoding

#### Analysis.

The vanilla Jacobi decoding incorporates a deterministic criterion for determining the convergence, which works well with greedy sampling (no randomness) in language models (Fu et al., ; Kou et al., ).
In contrast, in auto-regressive text-to-image generation, randomness plays a crucial role in the sampling-based decoding process, i.e., higher randomness corresponds to highly diverse details and structures in the generated images.
As shown in Fig. 3, we use the text prompt “a cat on a mat” to generate images with different sampling strategies including greedy decoding and top-$K$ decoding with different values of $K$.
We observe that the generated images would contain more details and diverse structures as the $K$ increases. The greedy decoding ($K=1$ with no equal probabilities) leads to suboptimal performance with low quality and no diversity in generated images
Therefore, random sampling-based decoding is important to image generation, but the original Jacobi decoding is incompatible with such randomness in sampling.

To address the aforementioned issue, we advance the deterministic Jacobi iteration into a new training-free probabilistic parallel decoding algorithm, inspired by speculative sampling (Leviathan et al., ).
Specifically, in each iteration, we decode multiple tokens in parallel and utilize a probabilistic criterion to accept multiple decoded tokens from the outputs of the previous iteration.
Moreover, to reduce the number of iterations required for inference, we propose a new image token initialization strategy incorporating spatial priors.

Figure: Figure 4: Overview of one iteration of our speculative Jacobi decoding (SJD). First, a sequence of draft tokens and the corresponding probabilities are taken as the inputs. Second, we perform a forward pass with the auto-regressive model on the draft tokens, obtaining the probabilities of these tokens. Third, we perform the verification according to these two types of probabilities, accepting a subset of tokens and (re-)sampling the remaining tokens. Last, the accepted tokens are appended to the pre-filling tokens and fixed, while the resampled tokens, along with newly initialized tokens, will serve as the draft tokens for the next iteration.
Refer to caption: x4.png

### 4.1 Speculative Jacobi Iteration

After pre-filling the tokens of text prompts, we perform speculative Jacobi decoding for the image tokens.
Acknowledging the computational expense of decoding all image tokens simultaneously, we decode multiple tokens per iteration in a sliding-window manner, termed the Jacobi window.
Our method starts with a sequence of initialized candidate tokens, the length of which corresponds to the window size.
During each decoding iteration, we predict the token probabilities for the draft token sequence within the current window. Then, a subset of these tokens is accepted based on a probabilistic criterion, and these accepted tokens are added to the fixed pre-filling tokens for the next iteration.
The remaining unaccepted tokens are resampled for the next iteration.
In the next iteration, the Jacobi windows slides to include the unaccepted tokens from the previous iteration plus some newly initialized tokens to maintain the Jacobi window size during decoding.
The process of our iteration is illustrated in Fig. 4.
Assuming that we have pre-filled or accepted $n$ tokens and the Jacobi window size is $W$, we would decode the next $W$ tokens.
The iteration can be described as follows:

Step 1: At the $j$-th iteration, we have the input tokens that are either predicted (but not accepted) in the previous $(j-1)$-th iteration or newly initialized (Sec. 4.2).
These tokens serve as the draft tokens in this iteration, denoted as $(x_{n}^{(j)},x_{n+1}^{(j)},\cdots,x_{n+W-1}^{(j)})$.
We denote the probability corresponding to the draft token $x_{i}^{(j)}$ as $p_{\theta}(x|{\bm{x}}_{1:i-1}^{(j-1)})$. This probability is set to be conditioned on the input tokens of the previous iteration (Step 3 ensures this setting).

Step 2:
We execute a single forward pass of the auto-regressive model to obtain the conditional probability for the draft tokens in parallel. The probability for $x_{i}^{(j)}$ is denoted as $p_{\theta}(x|{\bm{x}}_{1:i-1}^{(j)})$.

Step 3:
We conduct the speculative verification between the conditional probability given the draft tokens from the previous iteration, $p_{\theta}(x|{\bm{x}}_{1:i-1}^{(j-1)})$, and the conditional probability from the current iteration, $p_{\theta}(x|{\bm{x}}_{1:i-1}^{(j)})$.
In this verification process, we scan the draft sequence from left to right and determine the acceptance of each token based on a probabilistic threshold.
We measure the ratio of the probability conditioned on the draft tokens in the current iteration to that conditioned on the tokens from the previous iteration.
Intuitively, this ratio measures how well the token is decoded from the previous iteration (the draft token) and whether further decoding is necessary.
The acceptance criterion of a token $x_{i}^{(j)}$ can be formulated as follows:

$$ $x_{i}^{(j+1)}\leftarrow x_{i}^{(j)}\text{~{}~{}if~{}~{}}r\sim{\mathcal{U}}[0,1 ],~{}r<\min\left(1,\frac{p_{\theta}(x_{i}^{(j)}|{\bm{x}}_{1:i-1}^{(j)})}{p_{ \theta}(x_{i}^{(j)}|{\bm{x}}_{1:i-1}^{(j-1)})}\right),$ (1) $$

where $r$ is a random variable and ${\mathcal{U}}[0,1]$ represents a uniform distribution between 0 0 and $1$.
If a token meets the above criterion, it is accepted and appended to the pre-filling token sequence for the next iteration, i.e., $x_{i}^{(j^{\prime})}=x_{i}^{(j)}~{}\forall j^{\prime}>j$.
After accepting one token, we continue this scan until a token is rejected.
If the token $x_{i}^{(j)}$ is rejected, i.e., the inequality in Equ. 1 is not true,
we resample a new token by a calibrated distribution, and the resampled token will serve as the draft token for the next decoding iteration:

$$ $x_{i}^{(j+1)}\sim\frac{\max(0,p_{\theta}(x|{\bm{x}}_{1:i-1}^{(j)})-p_{\theta}( x|{\bm{x}}_{1:i-1}^{(j-1)}))}{\sum_{x}\max(0,p_{\theta}(x|{\bm{x}}_{1:i-1}^{(j )})-p_{\theta}(x|{\bm{x}}_{1:i-1}^{(j-1)}))}.$ (2) $$

Then, unlike the vanilla speculative sampling, we do not end this scan, but sample the tokens at the remaining indexes with the conditional probability calculated in this iteration.
This sampling process is consistent with the original Jacobi iteration, and the specific process is as follows:

$$ $x_{i^{\prime}}^{(j+1)}\sim p_{\theta}(x|{\bm{x}}_{1:i^{\prime}-1}^{(j)}),~{}~{ }\forall i^{\prime}>i.$ (3) $$

It can be proven that all the accepted and sampled tokens in the Jacobi iteration satisfy $x_{i}^{(j+1)}\sim p_{\theta}(x|{\bm{x}}_{1:i-1}^{(j)})$ (the proof is in the appendix).
This conditional probability $p_{\theta}(x|{\bm{x}}_{1:i-1}^{(j)})$ is exactly the probability predicted by the parallel forward pass on the input draft tokens,
and is passed to the next iteration together with the sampled tokens.

Step 4: we append the unaccepted tokens with newly initialized candidate tokens, forming a new Jacobi window with $W$ tokens, as the draft tokens for the next iteration. We use this fixed window size instead of the whole sequence to save the memory usage and accelerate the inference speed.

### 4.2 Token Initialization with Spatial prior

Vanilla Jacobi decoding methods sample the initial candidate tokens from a uniform distribution. However, 2D images exhibit unique characteristics of spatial locality, i.e., spatially adjacent tokens tend to share similar semantics and textures.
Leveraging these characteristics for token initialization may enable faster convergence.
Considering that auto-regressive models generate image tokens in a raster scan order (from the top-left to the bottom-right in 2D space),
we propose the following strategies for initializing new tokens:
(a) repeating the previously generated left adjacent token;
(b) repeating the previously generated above adjacent token;
(c) resampling from the predicted probability from the left adjacent token;
(d) resampling from the predicted probability from the above adjacent token.
Experimental results demonstrate that these strategies provide greater acceleration than random initialization under certain scenarios.

## 5 Experiments

### 5.1 Implementation Details

We experiment with two recent and representative auto-regressive text-to-image generation models, Lumina-mGPT (Liu et al., ) and Anole (Chern et al., ).
For Lumina-mGPT (Liu et al., ), by default, we use its 7B version to generate $768\times 768$ images for evaluation, and we measure the sampling randomness by the value $K$ of its top-$K$ logit sampler.
Following the basic setting of Lumina-mGPT, $K$ is set to $2000$ and the classifier-free guidance weight is set to $3.0$.
Anole (Chern et al., ) is another 7B auto-regressive generation model finetuned from Chameleon (Team, ) that can generate $512\times 512$ images.

**Table 1: The evaluation on the validation set of MSCOCO2017. JD: Jacobi decoding. ISP: initialization with spatial prior. SJD: Speculative Jacobi decoding.**
| Configuration | Acceleration ($\uparrow$) | FID ($\downarrow$) | CLIP-Score ($\uparrow$) |  |
| --- | --- | --- | --- | --- |
| Latency | Step |  |  |  |
| A    Lumina-mGPT (Liu et al., ) | $1.00\times$ | $1.00\times$ | 30.76 | 31.29 |
| B    w. JD (Song et al., ) | $1.02\times$ | $1.04\times$ | 30.66 | 31.38 |
| C    w. SJD | $2.04\times$ | $2.22\times$ | 30.85 | 31.35 |
| D    w. SJD (ISP) | $\mathbf{2.05\times}$ | $\mathbf{2.23\times}$ | 31.13 | 31.33 |
| E    Anole (Chern et al., ) | $1.00\times$ | $1.00\times$ | 28.87 | 30.59 |
| F    w. SJD (ISP) | $1.87\times$ | $1.97\times$ | 29.14 | 30.61 |

**Table 2: The evaluation on the validation set of Parti-prompt. JD: Jacobi decoding. ISP: initialization with spatial prior. SJD: Speculative Jacobi decoding.**
| Configuration | Acceleration ($\uparrow$) | CLIP-Score ($\uparrow$) |  |
| --- | --- | --- | --- |
| Latency | Step |  |  |
| A    Lumina-mGPT (Liu et al., ) | $1.00\times$ | $1.00\times$ | 32.13 |
| B    w. JD (Song et al., ) | $1.01\times$ | $1.04\times$ | 32.17 |
| C    w. SJD | $2.12\times$ | $2.26\times$ | 32.13 |
| D    w. SJD (ISP) | $\mathbf{2.13\times}$ | $\mathbf{2.28\times}$ | 32.06 |
| E    Anole (Chern et al., ) | $1.00\times$ | $1.00\times$ | 30.46 |
| F    w. SJD (ISP) | $1.92\times$ | $2.11\times$ | 30.48 |

Metrics.
For visual quality, we use FID (Heusel et al., ) and CLIP-Score (Radford et al., ) as the metrics for evaluation.
We use the step compression ratio (Fu et al., ): ${\mathcal{S}}=\frac{\text{\# generated tokens}}{\text{ \# decoding steps}}$ to show the theoretical acceleration ratio.
For each benchmark, we report the average of the step compression ratio on all generated images.
We also attach this ratio to each image sample in the qualitative comparison of our method with other approaches.
Moreover, we also report the latency acceleration of the model forward passes on a single GPU for testing the actual speedup.

Benchmark.
The parti-prompts (Yu et al., ) and the validation set of MS-COCO 2017 (Lin et al., ) are taken as the benchmarks of image generation.
On parti-prompts, we use the CLIP-Score and the acceleration of latency and steps excluding FID for evaluation because this benchmark only provides prompts without ground-truth images.

### 5.2 Quantitative Results

As shown in Tab. 1 and Tab. 2, our speculative Jacobi decoding accelerates the auto-regressive text-to-image generation nearly without sacrificing visual qualities.
When comparing our SJD (config C and D) with the vanilla Jacobi decoding (config B) on Lumina-mGPT, we observe that our probabilistic method greatly accelerates the generation by more than $2\times$ while the Jacobi decoding cannot.
Moreover, our method can provide a step compression of about $2\times$ for Anole.
We observe that the token initialization with spatial priors has a marginal influence on the speed of general image generation. We further analyze the specific scenarios of this modification in our ablation studies.

Figure: Figure 5: The images generated by Lumina-mGPT with different acceleration methods.
Refer to caption: x5.png

### 5.3 Qualitative Results

As shown in Fig. 5, we present the images generated with different configurations.
For comparison, we set the same random seed for each image sample.
According to our observation, the visual qualities of the images generated by different methods are similar, illustrating that our method can keep the visual quality for multiple styles of images.
More importantly, our speculative Jacobi decoding with or without our spatial initialization can greatly reduce the inference steps by more than $2\times$ for each case, and thus accelerate the inference process.

### 5.4 Ablation Studies

We perform ablation studies on Lumina-mGPT 7B.
Except for the experiments involving multiple resolutions, we use this model to generate $768\times 768$ images for evaluation.

The correlation between the sampling strategy in decoding and the acceleration ratio.
We compare the deterministic Jacobi decoding to our method under various randomness of the logit sampling.
In Fig. 7, We show the correlation between their acceleration ratio and the randomness of logit sampling.
According to this figure, our method is stable across multiple randomness and can achieve more than $2\times$ step compression ratio.
On the contrary, the Jacobi decoding can only accelerate the greedy sampling (top-$1$ sampling), which is useless for image generation.

The relationship between the image resolution and the acceleration ratio.
We employ the 7B Lumina-mGPT to generate images with the resolutions $512\times 512$ (about 1,000 tokens), $768\times 768$ (about 2,300 tokens), and $1024\times 1024$ (about 4,100 tokens).
We calculate the average step compression ratio for each resolution given the same set of text prompts.
Then, we present these ratios in Fig. 7. The results demonstrate that our method is stable across multiple resolutions, i.e., the acceleration on each resolution is larger than $2\times$. Moreover, with higher resolutions, the acceleration can be slightly better. For example, SJD achieves $2.43\times$ acceleration for $1024\times 1024$ images.

Studies on the window size of each iteration.
As mentioned in section 4.1, we append newly initialized tokens onto the unaccepted tokens in each iteration, fixing the Jacobi window size. Accordingly, we perform the ablation studies on the size of the window.
We report the acceleration ratios under various sequence lengths in Fig. 9.
The results show that our acceleration ratio reaches almost the maximum when the number of input tokens is greater than or equal to 16 tokens.

Studies on the initialization of candidate tokens.
The acceleration ratio of our speculative Jacobi decoding is also related to the application scenarios.
For example, when generating images composed of many simple and repeating patterns, a token initialization correlated with the already sampled tokens can provide a more precise guess than the random initialization.
As shown in Fig. 9, we adopt an extreme case, the textual prompt “2D logo of a pure white box in a pure black background”, for evaluation. We run the accelerated forward passes ten times with different random seeds for each initialization.
The results show that the average step compression with the spatial-prior-aware initialization is much greater than that using the random initialization.
Also, Fig. 1 shows that generating 2D logo requires fewer steps than generating images containing exquisite details.

Figure: Figure 6: Our method beats the vanilla Jacobi decoding under various sampling randomness.
Refer to caption: x6.png

Figure: Figure 8: The acceleration ratio is the largest when the Jacobi window size is at least 16.
Refer to caption: x8.png

## 6 Conclusion

This paper proposes a new training-free probabilistic parallel decoding algorithm, called Speculative Jacobi Decoding (SJD), to accelerate auto-regressive text-to-image generation.
The sampling-based decoding is critical for image generation models, which prevents naive Jacobi Decoding from being applied to accelerate AR-based text-to-image generation models.
By introducing a probabilistic convergence criterion, our SJD allows the model to iteratively predict-then-sample multiple tokens in fewer steps than the token counts for AR-based T2I models with sampling-based decoding rather than greedy decoding.
We also propose the spatial-aware token initialization to reduce the number of iterations under specific scenarios.
We conduct experiments to verify the effectiveness of SJD on multiple auto-regressive text-to-image generation models, and it accelerates the models without sacrificing visual quality.

## Appendix A Proofs

Theorem 1 (The correctness of speculative Jacobi decoding) The token sampled in each speculative Jacobi iteration satisfies $p_{\theta}(x|{\bm{x}}_{1:i-1}^{(j)})$, where $x$ denotes a token, $j$ denotes the index of iteration, $i$ denotes the token index, and $\theta$ denotes the auto-regressive model parameters.

Proof.
The main process of speculative Jacobi iteration is decomposed into two cases:
(a) obtaining the token sampled in the previous iteration and then accepting it according to an acceptance probability;
(b) rejecting the sampled token and resampling a new token according to a calibrated probability.
Thus,
like the proof of the vanilla speculative sampling (Leviathan et al., ),
to prove the correctness of speculative Jacobi decoding,
we verify that the conditional probability of a token sampled following the above two cases, alongside the manually designed acceptance and resampling probability, remains $p_{\theta}(x|{\bm{x}}_{1:i-1}^{(j)})$.

For simplicity, by default, we omit the token index $i$ and denote the token category of ${\bm{x}}_{i}^{(j)}$ as $x$.
We denote the condition of token ${\bm{x}}_{i}^{(j)}$ at the $j$-th Jacobi iteration (i.e., the tokens ${\bm{x}}^{(j)}_{1:i-1}$ and model weights $\theta$) to ${\mathcal{J}}_{j}$. Thus, the condition of the $(j-1)$-th Jacobi iteration is denoted as ${\mathcal{J}}_{j-1}$.
Thus, we can denote the probability $p_{\theta}(x|{\bm{x}}_{1:i-1}^{(j)})$ as $p(x|{\mathcal{J}}_{j})$, and denote $p_{\theta}(x|{\bm{x}}_{1:i-1}^{(j-1)})$ as $p(x|{\mathcal{J}}_{j-1})$.
We use a random boolean variable $r$ to represent the acceptance.
With these notations, the proof is as follows:

First, the acceptance probability on the token category $x$ is manually set as follows:

$$ $\displaystyle p(r\text{ is true}|x,{\mathcal{J}}_{j},{\mathcal{J}}_{j-1})=\min \{1,\frac{p(x|{\mathcal{J}}_{j})}{p(x|{\mathcal{J}}_{j-1})}\},$ (4) $$

and the calibrated resampling probability subsequent to the rejection is set as follows:

$$ $\displaystyle p(x|r\text{ is false},{\mathcal{J}}_{j},{\mathcal{J}}_{j-1})= \frac{\max\{0,p(x|{\mathcal{J}}_{j})-p(x|{\mathcal{J}}_{j-1})\}}{\sum_{x^{ \prime}}\max\{0,p(x^{\prime}|{\mathcal{J}}_{j})-p(x^{\prime}|{\mathcal{J}}_{j- 1})\}}.$ (5) $$

Next, we make an assumption that ${\mathcal{J}}_{j}$ and $x$ are conditionally independent given ${\mathcal{J}}_{j-1}$:

$$ $\displaystyle p({\mathcal{J}}_{j}|x,{\mathcal{J}}_{j-1})=p({\mathcal{J}}_{j}|{ \mathcal{J}}_{j-1})$ (6) $$

This assumption is reasonable due to the properties of the Jacobi iteration and the auto-regressive paradigm, i.e., with the observation of the sequence ${\bm{x}}_{1:i-1}^{(j-1)}$, one of the tokens in ${\bm{x}}_{1:i-1}^{(j)}$ (denoted as ${\bm{x}}_{k}^{(j)}$) can be determined by ${\bm{x}}_{k}^{(j)}=f({\bm{x}}_{1:k-1}^{(j-1)},\theta)~{}(k<i)$ where the function $f$ indicates the prediction-then-sampling of auto-regressive models, so the variable ${\bm{x}}_{i}^{(j)}$ is redundant as one of the conditions in the probability $p({\mathcal{J}}_{j}|x,{\mathcal{J}}_{j-1})$. Thus, Equ. 6 is reasonable.

Then, with Bayes rule, Equ. 6 has the following equivalence:

$$ $\displaystyle p({\mathcal{J}}_{j}|x,{\mathcal{J}}_{j-1})=p({\mathcal{J}}_{j}|{ \mathcal{J}}_{j-1})~{}~{}\Leftrightarrow~{}~{}p(x|{\mathcal{J}}_{j},{\mathcal{ J}}_{j-1})=p(x|{\mathcal{J}}_{j-1})$ (7) $$

Hence, according to Equ. 4 and Equ. 7, the probability that a token category $x$ is sampled in the previous iteration and subsequently accepted can be computed as:

$$ $\displaystyle p(r\text{ is true},x|{\mathcal{J}}_{j},{\mathcal{J}}_{j-1})$ $\displaystyle=p(x|{\mathcal{J}}_{j},{\mathcal{J}}_{j-1})\cdot p(r\text{ is true}|x,{\mathcal{J}}_{j},{\mathcal{J}}_{j-1})$ (8) $\displaystyle=p(x|{\mathcal{J}}_{j-1})\cdot\min\{1,\frac{p(x|{\mathcal{J}}_{j} )}{p(x|{\mathcal{J}}_{j-1})}\}$ $\displaystyle=\min\{p(x|{\mathcal{J}}_{j}),p(x|{\mathcal{J}}_{j-1})\}$ $$

With Equ. 8, we can calculate the probability of rejection with the law of total probability on the token categories:

$$ $\displaystyle p(r\text{ is false}|{\mathcal{J}}_{j},{\mathcal{J}}_{j-1})$ $\displaystyle=1-p(r\text{ is true}|{\mathcal{J}}_{j},{\mathcal{J}}_{j-1})$ (9) $\displaystyle=1-\sum_{x^{\prime}}p(r\text{ is true},x^{\prime}|{\mathcal{J}}_{ j},{\mathcal{J}}_{j-1})$ $\displaystyle=\sum_{x^{\prime}}p(x^{\prime}|{\mathcal{J}}_{j})-\min\{p(x^{ \prime}|{\mathcal{J}}_{j}),p(x^{\prime}|{\mathcal{J}}_{j-1})\}$ $\displaystyle=\sum_{x^{\prime}}\max\{0,p(x^{\prime}|{\mathcal{J}}_{j})-p(x^{ \prime}|{\mathcal{J}}_{j-1})\}.$ $$

Then,
with Equ. 5 and Equ. 9,
we get the following equation:

$$ $\displaystyle p(x|r\text{ is false},{\mathcal{J}}_{j},{\mathcal{J}}_{j-1}) \cdot p(r\text{ is false}|{\mathcal{J}}_{j},{\mathcal{J}}_{j-1})$ (10) $\displaystyle=\frac{\max\{0,p(x|{\mathcal{J}}_{j})-p(x|{\mathcal{J}}_{j-1})\}} {\sum_{x^{\prime}}\max\{0,p(x^{\prime}|{\mathcal{J}}_{j})-p(x^{\prime}|{ \mathcal{J}}_{j-1})\}}\cdot\sum_{x^{\prime}}\max\{0,p(x^{\prime}|{\mathcal{J}} _{j})-p(x^{\prime}|{\mathcal{J}}_{j-1})\}$ $\displaystyle=\max\{0,p(x|{\mathcal{J}}_{j})-p(x|{\mathcal{J}}_{j-1})\}.$ $$

Since

$$ $\displaystyle\forall a\in{\mathbb{R}},b\in{\mathbb{R}},~{}~{}a=\min\{a,b\}+ \max\{0,a-b\},$ (11) $$

we can decompose $p(x|{\mathcal{J}}_{j})$ as follows:

$$ $\displaystyle p(x|{\mathcal{J}}_{j})=\min\{p(x|{\mathcal{J}}_{j}),p(x|{ \mathcal{J}}_{j-1})\}+\max\{0,p(x|{\mathcal{J}}_{j})-p(x|{\mathcal{J}}_{j-1})\}.$ (12) $$

With Equ. 8, Equ. 10 and Equ. 12, we can compute:

$$ $\displaystyle p(x|{\mathcal{J}}_{j})$ $\displaystyle=\min\{p(x|{\mathcal{J}}_{j}),p(x|{\mathcal{J}}_{j-1})\}+\max\{0, p(x|{\mathcal{J}}_{j})-p(x|{\mathcal{J}}_{j-1})\}$ (13) $\displaystyle=p(x|{\mathcal{J}}_{j-1})\cdot p(r\text{ is true}|x,{\mathcal{J}} _{j},{\mathcal{J}}_{j-1})$ $\displaystyle~{}~{}~{}~{}+p(r\text{ is false}|{\mathcal{J}}_{j},{\mathcal{J}}_ {j-1})\cdot p(x|r\text{ is false},{\mathcal{J}}_{j},{\mathcal{J}}_{j-1}).$ $$

According to Equ. 13, the conditional distribution $p(x|{\mathcal{J}}_{j})$ can exactly represent
(a) obtaining the token sampled in the previous iteration and then accepting it according to an acceptance probability;
(b) rejecting the sampled token and resampling a new token according to a calibrated probability.
In conclusion, the token sampled in each speculative Jacobi iteration satisfies $p_{\theta}(x|{\bm{x}}_{1:i-1}^{(j)})$.

Figure: Figure 10: The images generated by Lumina-mGPT (Liu et al., ) with our acceleration method.
Refer to caption: x10.png

## Appendix B More Qualitative Results

In Fig. 10, we showcase more generated images with Lumina-mGPT accelerated by our method. These results illustrate that our method functions well on the image contents including humans, animals, and landscapes.
Recently, a new powerful auto-regressive model, Emu3 (BAAI, ), has been released.
We also explore our method on Emu3 for text-to-image generation, and we find it still leads to great step compression, shown in Fig. 11.
We leave the quantitative results of Emu3 for future work.

Figure: Figure 11: The images generated by Emu3 (BAAI, ) with our acceleration method.
Refer to caption: x11.png

## Appendix C Inference Latency

Figure: Figure 12: The latency of Lumina-mGPT on generating $768\times 768$ and $1024\times 1024$ images without or with our method.
Refer to caption: x12.png

In addition to reporting the step compression ratio, we also report the practical latency of SJD on servers.
We set the batch size as 1 for testing, and report the latency of the accelerated Lumina-mGPT 7B excluding the pre- and post-processing operations.
For $768\times 768$ image generation (the number of generated tokens is at least 2357), we perform the experiments on one RTX 4090 GPU.
For $1024\times 1024$ image generation (the number of generated tokens is at least 4165), we perform the experiments on one A100 GPU.
In these settings, the latency of Lumina-mGPT with and without our method is presented in Fig. 12.
Our method significantly accelerates the auto-regressive image generation.

## Appendix D More Quantitative Results

In addition to Anole (Chern et al., ) and Lumina-mGPT (Liu et al., ), we evaluate our method with the text-to-image LlamaGen (Sun et al., ).
This model adopts a two-stage training strategy:
(a) stage1: LlamaGen is first trained on a subset of LAION-COCO (LAION, ) (50M $256\times 256$ images);
(b) stage2: it is then fine-tuned on 10M high aesthetic quality internal data with a resolution of $512\times 512$.
In Tab. 3, we evaluate our method with the two versions of LlamaGen. The results show that our method can still accelerate this model without sacrificing the visual quality.
However, in comparison to the experiments conducted on Lumina-mGPT and Anole, the acceleration ratios on LlamaGen are lower.
We hypothesize that this discrepancy is attributed to the model size (LlamaGen has 3.1B model parameters while Lumina-mGPT has 7B parameters), as some existing works for multi-token prediction demonstrate that the model size has a great influence on the effectiveness of acceleration (Gloeckle et al., ).
We leave this investigation to future work.

**Table 3: The evaluation of LlamaGen (Sun et al., ) with or without our method on MSCOCO2017 (Lin et al., ) and Parti-prompt (Yu et al., ).**
| Dataset | Configuration | Acceleration ($\uparrow$) | FID ($\downarrow$) | CLIP-Score ($\uparrow$) |  |
| --- | --- | --- | --- | --- | --- |
| Latency | Step |  |  |  |  |
| COCO | LlamaGen-stage1 | $1.00\times$ | $1.00\times$ | 28.54 | 30.87 |
| LlamaGen-stage1 + Ours | $1.56\times$ | $1.63\times$ | 29.00 | 30.82 |  |
| LlamaGen-stage2 | $1.00\times$ | $1.00\times$ | 56.21 | 28.26 |  |
| LlamaGen-stage2 + Ours | $1.54\times$ | $1.63\times$ | 57.02 | 28.33 |  |
| Parti | LlamaGen-stage1 | $1.00\times$ | $1.00\times$ | - | 30.22 |
| LlamaGen-stage1 + Ours | $1.57\times$ | $1.73\times$ | - | 30.29 |  |
| LlamaGen-stage2 | $1.00\times$ | $1.00\times$ | - | 28.14 |  |
| LlamaGen-stage2 + Ours | $1.62\times$ | $1.69\times$ | - | 28.16 |  |

## Appendix E Visualization of Acceleration in 2D space

Figure: Figure 13: The visualization of the accelerated tokens on 2D space.
Refer to caption: x13.png

We visualize the impact of multi-token prediction in a 2D space. As illustrated in Fig. 13, the color of each long strip area represents the length of accepted tokens from that area, with darker colors indicating longer sequences of accepted tokens, i.e., higher acceleration. We observe that high acceleration tends to occur in the background, particularly on the left and right sides of images. Additionally, while some high acceleration is observed on foreground objects, it is relatively sparse in 2D space.

## Appendix F Limitation and future work

Since our speculative Jacobi decoding is training-free, the accelerated model itself is still not specialized for multi-token prediction. Therefore, the acceleration ratio has the potential to be further improved.
In the future, we believe that fine-tuning the auto-regressive models for fast image generation is a promising direction.
Also, acceleration is important for long-sequence generation, like video generation. Since videos contain more redundancy than images, the initialization of candidate tokens should be carefully designed if applying our speculative Jacobi decoding to video generation.

## Appendix G Broader Impacts

Image generation offers extensive utility in helping users, designers, and artists produce fantastic content. Nonetheless, these models could be exploited to create deceptive content.
Thus, it is crucial for the users including researchers and developers to acknowledge the potential negative social impact of image generation models.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: Please see the Weaknesses
soundness: 3
strengths: 1. This paper adapts speculative decoding to auto-regressive (AR) image generation, achieving substantial speed improvements with minimal impact on image quality, addressing a key efficiency challenge in this area.

2. The proposed method demonstrate performance gains on state-of-the-art models, including Lumina-mGPT and Anole, underscoring its applicability and effectiveness.
summary: This paper proposes Speculative Jacobi Decoding (SJD) for accelerating the inference of auto-regressive (AR) image generation. SJD introduces a probabilistic acceptance criterion, using the ratio of the probability conditioned on the current draft tokens to that of the previous iteration for speculative verification, enabling faster convergence while preserving diversity in sampling-based decoding. This approach reduces the number of steps required, improving inference speed without compromising image quality.
weaknesses: 1. Need for analysis on image randomness in SJD. The randomness in SJD introduces variability in generated images, but there is limited analysis on the extent of this randomness. Unlike speculative decoding in the language domain, which offers some guarantees on reproducibility, SJD would benefit from theoretical bounds or deeper empirical analysis to clarify the extent of variation in its results.

2. Demonstrating similar speed-up and preservation of image quality on additional models, such as LlamaGen, would strengthen this paper.

3. Need for additional qualitative evaluation: Additional qualitative results, as in Fig. 5, comparing original images with those generated by the proposed method, along with steps and latency comparisons on Lumina-mGPT and Anole, would be beneficial. Particularly, a detailed discussion on representative failure cases and their causes is needed to clarify potential issues in image quality.

3. While this paper tends to emphasize step counts in terms of speed-up, latency is a more accurate metric for evaluating it. Therefore, using latency as the primary measure would enable a clearer comparision. 

4. Improved clarification in Fig. 4 would enhance understanding of the proposed method.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: - The acceptance is determined if it improves prediction compared to the conditions of the previous Jacobi iteration. I wonder if the conditional probability given the draft tokens from the previous iteration is too low; it may have a chance to accept tokens even if it still has a very low conditional probability but a little bit higher than that, which may result in inaccurate token acceptance.
soundness: 3
strengths: - Clear motivation about the limitation of existing performant auto-regressive T2I generation models and Jocobi decoding strategy.
- Achieving about 2 times faster inference latency with fewer iteration steps, with a small drop in image quality compared to original T2I backbones.
- This approach can be applied to multiple different auto-regressive T2I generation models.
- Provide several meaningful ablations to better understand the robustness of the proposed methods with varying scales of hyperparameters.
summary: Recent auto-regressive models generate quality images but require thousands of next-token prediction steps during inference, which is significant in terms of inference speed. To address this, this paper proposes Speculative Jacobi Decoding (SJD), a training-free probabilistic parallel decoding algorithm. SJD introduces a probabilistic convergence criterion, allowing for sampling-based token decoding that maintains diversity and accelerates inference while overcoming the limitation of original Jacobi Decoding - deterministic, greedy decoding, which limits diversity to be incorporated into the image generation framework. This approach enables multiple token predictions per step in a probabilistic way (i.e., inserting randomness), reducing required steps while preserving image quality.
weaknesses: - The proposed approach requires over-computation in their Jacobi iteration. It computes conditional prediction scores for all tokens in the window but only accepts several earlier tokens by design - so resources used for computing later tokens are the first unaccepted tokens and later ones, even if some of them are technically accepted. This is inevitable since next-token prediction should be conditioned on all previous tokens that should be accepted (except for the beginning) and cannot allow some undesired holes of unaccepted tokens. While the approach proceeds these unaccepted tokens to new draft tokens for the next iterations, the effectiveness of this technique is not studied throughout the paper. At least, comparing the impact of this reuse of over-computed tokens with several token initializations with spatial priors (and random as well) would be meaningful.

- Initialization with spatial prior (ISP) doesn't show real acceleration in both terms of latency and steps compared to SJD w/o ISP. Figure 9 shows some different behavior, but the gap looks marginal in the main table (Table 1). I would say this is not appealing in terms of idea since this is from a random idea without strong grounding, less relevant to the overall story of this paper, and the performance is also less impactful.

- Analyses/ablations look incomplete. For Figures 6,7,8 and 9, I believe this should also be meaningful to provide quantitative performance of image quality rather than reporting only step compression rate. Reporting only the efficiency of the proposed approach does not give meaningful insight.

- The experimental results with Anole in Table 1 also look incomplete. I think more rows (i.e., B and C in Lumina-mGPT cases) are necessary for this table.

- (minor) typo: Each column presents --> Each row presents

---

confidence: 2
contribution: 2
initial_rating: 6
presentation: 2
questions: Q1: Could the authors provide the conparison of perplex between SJD and other decoding methods?


Q2: I'm curious about how many image samples are used to compute the FID scores. Could the authors provide more evaluation details?
I doubt that the authors have use some tricks in evalution.  For example, the image quantity maybe different to compared method
Q3: Could the author provide more  results on small  models?
Not everyone has access to large models, it's more convicing to conduct experiments on small models
soundness: 2
strengths: This paper introduces reject sampling to accelerate auto-regressive text-to-image generation.
SJD enables the model to predict multiple tokens at each step, accepting them based on a probabilistic criterion, reducing the number of inference steps compared to conventional next-token-prediction.
The paper proposes a spatial locality-aware token initialization strategy to further enhance acceleration in specific scenarios.
Experiments on multiple auto-regressive text-to-image generation models demonstrate SJD's effectiveness in accelerating inference without compromising visual quality.
summary: SJD iteratively decodes multiple tokens in parallel within a sliding window. The model computes the conditional probability for a sequence of draft tokens in a single forward pass. A probabilistic criterion then determines which tokens to accept, adding them to the fixed pre-filling sequence. Unaccepted tokens are resampled for the next iteration. The spatial locality-aware token initialization strategy leverages the spatial relationships in images for initializing new tokens. SJD effectively accelerates auto-regressive text-to-image generation models, achieving approximately 2× speedup for Lumina-mGPT and Anole without significant visual quality degradation.
weaknesses: 1) SJD relax the greedy acceptence criteron of next token by reject sampling. Such acceleration will sample results with smaller probabilities and JD is also able to update multiple tokens at a time.  And relaxation of acceptence criteron lead to suboptimal results. Moreover, this paper is infact very simple with little experiments. It's more like a trick rathor than a paper.
2) The results that JD only accelerate t2i models by a rate of 1.04 is not convicing, which is much worse than results provided by JD and other papers.
3) The evaluation details are not provided. As far as I'm concerned, SJD may leads The nolvelty seems incremental. to suboptimal image quality and better image diversity. Hence, the FID score would be better if the image quality decrease a little and the image diversity increase.
4) There are too little models tested. It's better to apply SJD to more large models and small models.
Not that, I'm not familar to large model accelaration, I will consider other reviews' comments.

Meta-review: This paper proposes Speculative Jacobi Decoding, a training-free probabilistic parallel decoding algorithm for accelerating auto-regressive text-to-image generation. The core novelty lies in introducing a probabilistic convergence criterion, which allows for faster decoding without sacrificing visual quality or diversity. The authors conduct multiple experiments demonstrating speed-ups across multiple state-of-the-art models while maintaining image quality. Overall, this work makes a meaningful contribution to auto-regressive visual generative models, and thus I recommend acceptance.

Decision: Accept (Poster)