Speculative Jacobi-Denoising Decoding for Accelerating Autoregressive Text-to-image Generation
Authors: Yao Teng
Fu-Yun Wang
Xian Liu
Zhekai Chen
Han Shi
Yu Wang
Zhenguo Li
Weiyang Liu
Difan Zou
Xihui Liu
Venue: NeurIPS 2025 poster ()
Keywords: Acceleration
Auto-regressive Model
Text-to-image Generation
Primary area: deep_learning

Abstract: As a new paradigm of visual content generation, autoregressive text-to-image models suffer from slow inference due to their sequential token-by-token decoding process, often requiring thousands of model forward passes to generate a single image. To address this inefficiency, we propose Speculative Jacobi-Denoising Decoding (SJD2), a framework that incorporates the denoising process into Jacobi iterations to enable parallel token generation in autoregressive models. Our method introduces a next-clean-token prediction paradigm that enables the pre-trained autoregressive models to accept noise-perturbed token embeddings and predict the next clean tokens through low-cost fine-tuning. This denoising paradigm guides the model towards more stable Jacobi trajectories. During inference, our method initializes token sequences with Gaussian noise and performs iterative next-clean-token-prediction in the embedding space. We employ a probabilistic criterion to verify and accept multiple tokens in parallel, and refine the unaccepted tokens for the next iteration with the denoising trajectory. Experiments show that our method can accelerate generation by reducing model forward passes while maintaining the visual quality of generated images.

Speculative Jacobi-Denoising Decoding for Accelerating Autoregressive Text-to-image Generation
Yao Teng'
Fuyun Wang²
Xian Liu
Zhekai Chen'
Han Shi Xihui Liu'*
Yu Wang
Zhenguo Li
Weiyang Liu?
Difan Zou'
'The University of Hong Kong
CCUHK
Huawei Noah's 's Ark Lab
"Tsinghua University
Abstract
As new paradigm of visual content generation, autoregressive text-to-image models suffer from slow inference due to their sequential token-by-token decoding process, often requiring thousands of model forward passes generate single image. To address this inefficiency, we propose Speculative Decoding (SJD2), framework that incorporates the denoising process into Jacobi iterations to enable parallel token generation in autoregressive models. Our method introduces next-clean-toker prediction paradigm that enables the pre-trained autoregressive models to accept noise- perturrert token embeddings and predict the next clean tokens through low-cost fine-tuning. This denoising paradigm guides the model towards more stable Jacobi trajectories. During inference, our method initializes token sequences with Gaussian noise and performs iterative next-clean- coken-predictto in the embedding space. We employ probabilistic criterion to verify and accept multiple tokens in parallel, and refine the unaccepted tokens for the next iteration with the denoising trajectory. Experiments show that our method can accelerate generation by reducing model forward passes while maintaining the visual quality of generated images.
1 Introduction
Autoregressive models have emerged as cornerstone of visual generative tasks through next-token prediction [1-4]. However, the autoregressive paradigm suffers from significant inference latency due to its sequential, token-by-token decoding process. For instance, generating a single high-resolution image often requires thousands of sequential forward passes. To address this challenge, we focus on accelerating autoregressive text-to-image generation models via parallel token decoding.
Jacobi decoding [5] is an iterative method that accelerates the inference of autoregressive models through parallel token decoding without any training. This method operates on a sequence of randomly initialized tokens and iteratively calls the neural network refine the tokens until the convergence (i.e., tokens are correctly decoded). Its variant, Speculative Jacobi Decoding (SJD) [6]. improves Jacobi decoding with a probabilistic criterion tailored for accelerating text-to-image generation using discrete tokens. The core of SJD is a reffiatttin--efeemmeett process. Specifically, given sequence of tokens, in each Jacobi iteration, SJD first predicts the probability for each input token, and then these probabilities enable the criterion to determine the acceptance of a prefix of the tokens (i.e., verification) while also guiding the resampling of unaccepted tokens for the next iteration (i.e., refinement). The above verification-refiennn Jacobi window where the accepted tokens are removed and newly initialized tokens are appended. By accepting multiple tokens (at least one token) per iteration, SJD reduces model forward passes, speeding up generation compared to token-by-token decoding.
process operates within fixed-length sliding
"Corresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Steps: 2357
Steps: Fewer)
Steps: 2357
Steps: (4. Fewer)
Prompt: A buge black bear wearing red cassock, in front blazing fire
Prompt: An image of hamster with grasping trident and pointing the other Enger forward in the desort
Steps: 2357
Steps: 578 (4. 1 X Fewer)
Steps: 2357
Steps: 549 (4. 3 x Fewer)
Prompt: Jars in the laboratory with yellow crystals growing inside, film grained
Prompt: An oil painting of beautiful young lady with crimson eyes, short white hair, sly smile on face, and an violet outfit
Steps: 2357
Steps: 600 9 Fewer)
Steps: 2357
Steps: 564 (4. Fewer)
Prompt: An imaginary picture of cyborg German shepherd, standing in the ruins of modern city under red sunset sky, film grained
Prompt: muscular warrior wearing dark brown armor with helmet, mask over his face (black mask covering his eyes), and a huge oxygen tube plugged into the mask
We propose Speculative Jacobi-Denoising Decoding to accelerate autoregressive text-to- image generation via multi-token prediction. On Lumina-mGPT. the number of model forward passes for inference (denoted as steps) reduced. The inference step for our decoding is marked in green.
Figure 1:
Although SJD accelerates autoregressive text-to-image generation by a non-negligible margin, the token refinement process in SJD is inherently unconstrained, making it difficult to control the refinement to achieve the correct token predictions. Consequently, some tokens undergo many refinement iterations before being accepted, resulting in low speedup ratio. In contrast, diffusion models explicitly define a trajectory for the iterative input refinement, known as the denoising process, which is governed by the principles of stochastic differential equations [7]. More importantly, previous works [8-10] have demonstrated that this trajectory can be remarkably short (as few as tens of iterations) for any length of inputs (i.e., various image resolutions). Inspired by this intuition, to further accelerate the autoregressive text-to-image generation, we leverage the denoising process from the diffusion models to assist the Jacobi decoding and fine-tune the autoregressive model to adapt to the noise perturbation.
In this paper, we propose Speculative Deoobi--eniiingg Decoding (SJD2), along with fine-tuning strategy, enable pre-trained autoregressive text-to-image generation models to perform parallel token-level denoising decoding. To achieve this, we introduce task called next-clean-toke pre- diction, where an autoregressive model accepts noisy input tokens and predicts the clean tokens at one nne-posiioon offset (i.e., noise-free next tokens). Our noise-augmented fine-tuning strategy

---

equips pre-trained autoregressive models with the denoising ability, i.e., we randomly select segments of imput tokens and add Gaussian noise to their token embeddings during training. The training supervision remains consistent with conventional autoregressive models, i.e., the one position-offset discrete tokens are taken as ground-truth supervision and the model trained with cross-entropy loss. After fine-tuning the model has the ability to process noisy inputs and thus perform Jacobi-Denoising decoding during inference. The decoding process begins with a sequence (Jacobi window) of token embeddings randomly initialized with Gaussian noise. In each Racobi-Denoisiln sequence is fed into the autoregressive model to predict the probability distribution clean tokens at each token position. The noisy tokens refined for the next iteration through the denoising process using the probability predicted in the current iteration. If a token is sufficiently denoised, it becomes clean token and undergoes the standard Jacobi refinement. The iterative process is repeated until all tokens are accepted.
iteration, this
To validate our method, we conduct experiments on two open-source large autoregressive models, Lumina-mGP7 [11] and Emu3 [4]. Experimental results demonstrate that our method reduces the number of forward passes by about 4x on Lumina-mGPT and more than 5x on Emu3 and thus achieves latency speedup by more than 2x. We further verify image quality to show that our method accelerates autoregressive text-to-image generation without compromising image quality.
2 Related Work
Integration of Autoregression and Continuous Diffusion The AR-diffusion language model [12] integrates the concept of autoregression into an embedding diffusion language model [13-18]. The embedding diffusion performs the standard denoising process on (normalized) token embeddings instead of next-token prediction, and use the regression loss [12] or cross-entropy loss [14] for training. The recent diffusion-foree AR-diffusion model and introduce the autoregressive sampling into the temporal dimension of the continuous latent video diffusion models. They employ causal masks to support history condtro--ndd video generation, but have independent noise levels across input tokens. Moreover, for the diffusion process on the temporal dimension, works like the rolling diffusion model [26] further introduce the sliding window mechanism on the time series. Transfusion and its variants [27-29] utilize a single backbone model to jointly perform diffusion-based image generation in a continuous latent space and autoregressive language generation in a discrete space, using separate loss, different lightweight decoders, and specific attention masks for training. MAR and its variants [30-32] propose the use of an efficient MLP diffusion head to decode features from autoregressive backbones, enabling autoregression in continuous latent space. Several multimodal large language models (MLLMs) [33- 38] generate images via pre-trained diffusion models such as SDXL [39] which take the output features of autoregressive models conditions.
model and its variants [19-25] share similar pipeline as the
Parallel Token Decoding in Autoregressive Models Blockwise Parallel Decoding [40] and Medusa [41] employ auxiliary modules to predict multiple next tokens simultancously, thereby accelerating language models. Speculative decoding [42, 43, 41, 44-47 enhances inference effi- ciency by using smaller model to generate candidate tokens, which are then verified and accepted by the larger model in parallel. Difful.LaMA [48] fine-tunes the large autoregressive language models into discrete diffusion models [49] for parallel decoding. Jacobi Decoding [5], initially applied to pixel-level autoregressive generation models, iteratively decodes tokens in parallel until their values converge, often in training-free manner. Lookahead Decoding [50] employs the token trajectories of Jacobi decoding to form a pool of n-grams generated using greedy sampling to accelerate language models. CLLMs [5]] collect the Jacobi trajectories into a dataset and then distill the language models with it. Speculative Jacobi Decoding [6] revisits the original Jacobi Decoding in image generation and adapts it for modern autoregressive -at-to-imaae generation based on discrete tokens by simply introducing a probabilistic criterion. Spatially parallel image autoregressive decoding [52, 53] de- codes multiple tokens simultaneously by leveraging spatial dependencies. Distilled- -iiilleind- [17] distills an autoregressive model into consistency model [54] via an embedding- prediction head with millions of prepared noise-token pairs and hundreds of training epochs.
embedding-prediction
Discussion In contrast to the above works, this paper integrates the denoising process into discrete autoregressive models while preserving the properties like next-token prediction. Our method enables flexible modulation between the autoregression, the speculative decoding, and the denoising process.
3

---

We fine-tume pre- pre-trained autoregressive text-to-ine rext-to-image generation models to achieve our goal with a few epochs and off-the- shethe--shelf image data while avoiding adding additional modules.
3 Preliminaries
Autoregressive Generation Let (uN,...,IN)} denote a sequence of discrete tokens, where N is the sequence length and 22 E is an integer from a vocabulary of size [V In autore- gressive models, the joint probability of the sequence is factorized as: P(r1, ......,......................... ZN)
P(x;) N
which assumes each discrete token z; depends only on its pre- ceding tokens, following a causal structure. Autoregressive models parameterize the conditional probability as [[..,_11.....2 and sequentially decode (i.e. sampling token based on the predicted probability) each discrete token 2; based on the preceding outputs.
P(x, :2
Jacobi Decoding and Speculative Jacobi Decoding Jacobi Decoding treats the autoregressive decoding as solving a non-linear equation in a triangular system via fixed-point iteration [5]. Instead of sequentially generating tokens by the rule 2; Pal .2y1),, Jacobi Decod- ing introduces an iteration index j to enable parallel updates across all the sequential positions: a (((+)) ~ Pa(x] ..... -x.") where x!()) denotes a token at position at iteration j. Jacobi Decod- ing starts with randomly initialized tokens and iterates across the dimension until convergence, i.e.
the tokens remain unchanged between two consecutive iterations. Since the number of iterations is proven to be not greater than the number of tokens [5], the acceleration can be achieved.
To adapt Jacobi Decoding for the modern autoregressive text-to-image generation which is based on a large range of discrete tokens, Speculative Jacobi Decoding (SJD) [6] improves it by introducing the probabilistic criterion from speculative sampling [42, 43] determine the convergence of tokens:
converged if r min (1, Po Po(x (1) (110 11
r 1~~21 21 1),
(1)
(o(o (1) [ ,0,,,,9))) (2))))
where 24(0, 1] denotes uniform distribution between and 1, and j' is set to by default, i.e., the tokens generated in previous Jacobi iterations serve as draft tokens for the current Jacobi iteration. SJD also maintains a fixed-length Jacobi window within which Jacobi iterations are performed and token convergence is determined. In each iteration, a prefix of the tokens is determined to be converged and removed from the window (i.e., the tokens are accepted), while the remaining tokens are resampled and newly initialized tokens are appended to the window.
Continuous Diffusion Models Continuous diffusion models [55-59, 8, 60] generate data by learning to reverse a noise-corruption process. In this process, clean continuous input zg is gradually corrupted into pure Gaussian noise This can be formulated as: at any timestep (normalized the range of [0, 1)) the noise perturbation can be written as x2 Qg dzE where a2 and a are manually defined functions. As increases, 04 monotonically decreases but de increases. The reverse denoising
Or. Øt Do 8) [61]. Here, d xr, 0ss a, trajectory between two timesteps can be solved by: æe d,
Da is a neural network trained via regression loss to predict xn given a noisy input and a function of timestep. this paper, we incorporate the denoising process into the Jacobi decoding.
4 Method
In Jacobi Decoding, the refinement of tokens is unconstrained and difficult to control, with no guarantee that tokens will follow the fastest path to reach the correct values. contrast, diffusion models have been demonstrated to generate high-resolution, high-quality images through short trajectories (as few as dozens of iterations) [8-10]. Motivated by this, we propose integrating the denoising process from the diffusion models into Jacobi Decoding.
Overview Our decoding process, illustrated in Figure 2, comprises the following steps: (1) Token initialization: Given sequence of noisy normalized token embeddings (illustrated as blue-bordered patches in the first row) and weefllinn/aaready-acceeepp first row), the noise levels of the token embeddings configured to be non-strictly monotonically
tokens (depicted as green circles in the

---

Tokers
PrefilUng & Decodod
Input Embaddings
Token
Normalized Embeddlng
Perturbation (Normalized Embedding Space)
Noisy Inputs
Append Timestopa
De-Norme
X
Transformer
Blocks
Prediction Head
X
Taken-Sample
Training Target (Logits)
1-4
Embeddling
Layer
*
Ri
R
Normalization
Casual Attention
Normalized
Loco
Acceptance
Jacobi-Diffusion Iterations
Embedding
Figure 2: Overview of our decoding process. The noisy token embeddings with increasing noise levels undergo parallel forward pass with a causal attention mask, predicting conditional probabilities and then sampling clean tokens. A probabilistic criterion selects a prefix of tokens for acceptance (green area). For unaccepted to- kens, if clean, the next-token prediction performs on them (green solid arrows marked with AR). If noisy, they are denoised with one- me--ooitionn offset (blue solid go arrows and dash arrows).
Figure 3: Overview of our training strategy and model process. The input token embedding se- quence is randomly divided into segments and the adjacent segments are perturbed with the noise of consecutive levels. The noisy embed- dings with timestep tokens are fed into trans- former blocks and prediction head. During training, the predicted probability is used to com- pute the cross-entropy loss for next-token pre- diction. During inference, the probability is for token sampling and then generating embeddings.
increasing. Before the iterations start, the sequence is initialized with pure Gaussian noise. (2) Parallel forward: After initialization, these token embeddings undergo Jacobi-denoising iteration, where they are fed into the neural network along with timestep encodings for single parallel forward pass using a causal attention mask. The network predicts conditional probability and performs token sampling for the next clean token at each position. Sampled tokens from noisy embedding inputs are denoted as the one-posiie nneepooition--fffee 80" -predictions (marked by the down-right blue solid arrows) while those from prefilling or accepted token inputs are denoted as autoregressive (AR) predictions, i.e., the standard next-token prediction (marked by green solid arrows). (3) Verification: After the parallel forward pass, a prefix of sampled tokens is accepted based on the probabilistic criterion outlined in Equation (1). For example, in the second row of Figure 2, the first two sampled tokens are accepted and marked with green borders. (4) Refinement for unaccepted tokens: After the verification,
the refinement is performed for unaccepted tokens. For each unaccepted noisy token, a denoising step, outlined in Equation (3), is performed on its embedding. Specifically. a linear combination is performed on the token embedding from the previous iteration the same spatial position (indicated by vertical blue dashed lines) and the embedding of the predicted clean tokens from the one-pose one-position offset (the down-right blue solid arrows). If an unaccepted token has been sufficiently denoised to be clean in previous iterations, there is no further denoising needed and the refinement follows the standard next-token prediction. This iterative process repeats until all required tokens are accepted, serving as the final outputs.
4.1 Model Parameterization
To seamlessly integrate the denoising process with next-token prediction, we propose a paradigm of next-clean-loken prediction based noisy normalized token embeddings. In this paradigm, the autoregressive neural network learns to accept noisy input tokens and predict the clean tokens at a one-position offset (i.e., noise-free next tokens). Let denote normalized token embedding and 0
5

---

denote the autoregressive model, our paradigm can be formulated as follows:
2 ~ Po(zle , -r .. ,,,))
...
m i ++1
0 W. One-hot(22411)
i+1 (ormalize(mh).:1)
where et is the normalized token embedding at spatial position in the sequence and at timestep
t, and e represents the predicted clean normalized token embedding at position i + 1, Po (arlef. - e. fo e denotes the predicted conditional probability of discrete to- e as conditions, One-hot(-) denotes transforming token category into one-hot vector, We E DD denotes the learned embedding weight matrix of the model, and m RD denotes the D-dimensional predicted token embedding at position i + 1. Normalize(-) is normalizing embeddings with their statistics (details in Section 4.3). The timesteps ffu... ) i.e, noise levels, are configured be non-strictly monotonically increas- ing, and we discuss the specific selection of timesteps for decoding in Section 4.2. In this paradigm, the pre-trained autoregressive models still predict the categorical probabilities of tokens instead of the continuous token embeddings, and we employ these probabilities to generate the embeddings.
ken x with the embedding sequence {ef, ... e, ,...,...,,,..........
4.2 Speculative Jacobi-Denoising Decoding
Jacobi Window and token initialization In practice, our method employs a sliding Jacobi window during the decoding phase, rather than directly performing the decoding process at fixed token positions. The Jacobi window a fixed-length sliding window containing noisy normalized token embeddings as the draft. Initially, the window is filled with pure Gaussian noise. In each iteration, a prefix of accepted tokens is removed, and an equal number of new tokens, sampled from pure Gaussian noise, is appended. Consequently, the noise levels of the tokens within the window are non-strictly monotonically increasing across iterations.
Parallel Forward and Verification The model takes draft token embeddings the inputs and predicts the conditional probability of their next clean tokens in parallel. Then, Equation (1) is employed to determine whether to accept or reject each draft token which is transformed from the input token embedding. This transformation finding the nearest discrete tokens the vocabulary through cosine similarity [47].
Refinement with Denoising After verification, the unaccepted tokens will go through refinement process to refine the token embeddings for the next iteration. The refinement of noisy tokens is realized by the denoising process. For denoising, we first define a fixed monotonically decreasing timestep sequence {!x, ['..... 44-
lo, 0}. where tu is set very close to zero [62]. The values of these timesteps can follow the Karras timestep scheduler [62]. Then, we show the specific denoising
formula based on these timesteps when k 0:
1K-1
Oty-1 ett + a't
Otta-1
Ote-1
Oits
Tty
where éi is the token embedding predicted according Equation (2). In the above equation, perform a denoising step like the diffusion models to obtain the embedding with a smaller noise level position i with timestep 42- If k 0, i.e., the denoising process just complete, Equation (3) reduces into the standard Jacobi iteration: ee == 0 + 20. Additionally, for the unaccepted sufficiently denoised tokens, as no further denoising required, resample the tokens whose threshold from Equation (1) is below 0.5 but retain the others for the next Jacobi iteration.
By introducing the denoising trajectories into the decoding process of autoregressive models, our method stabilizes the token trajectories in the decoding process, accelerating the token convergence.
4.3 Fine-tuning Strategy
In this section, we introduce the strategy of fine-tuning a pretrained autoregressive text-to-image generation model to accept noisy input tokens for Speculative Deooii-Deniisnggg Decoding. Figure illustrates an overview of our training strategy and the specific process of the neural network: During

---

training, the normalized embeddings (initially clean) are first transformed into noisy embeddings. For example, in the right-side image of Figure 3, noise levels increase non- monotonically across patches (i.e., token positions) in raster scan order (left to right, top to bottom). When reaching randomly determined position, the noise level stops increasing and the noise level of the next position resets to zero, forming segments with oon-moootonically increasing noise levels in the token sequence. Next, these noisy embeddings are appended with timestep encodings, which indicate the noise level at each position. Together, the embeddings and encodings are denormalized and are fed into transformer blocks and a prediction head to produce logits for each position. The cross-entropy loss then applied to each position, using the clean token indices as labels, with mee-postioon offset (shown by prediction. During inference, the input normalized
the dotted frame in Figure 3) for next-clean-toke embeddings are already noisy and are not further perturbed. These embeddings are also appended with timestep encodings and processed to generate logits through the denormalization, the transformer blocks and the prediction head. As described in Equation (2), these logits are used for token sampling, and these sampled clean tokens are then transformed into normalized token embeddings.
Noise Perturbation We add noise to the embeddings of these discrete input tokens, bypassing the discrete values. Although we can directly perform linear combination of the embedding and Gaussian random variable, the distribution of the pre-trained embeddings may deviate from the scale of the standard Gaussian distribution. For instance, if the variance of the embedding is small, the pre- trained transformer blocks may only handle small values, making the values from a standard Gaussian distribution unsuitable for these blocks. To address this issue, we add noise to the normalized token embeddings. Subsequently, these noisy normalized embeddings are de-normalized and then fed into the transformer blocks. The detailed procedure is as follows:
e" = Normalize(m"))
@ 6m" -Me),
oe
(4)
e. a + O,E, m f = De-normalize(e") = o. ① e' pe:
where m RP denotes the D D-dimensional embedding of a ground-truth token. Me R and Te E E denote the mean and standard deviation of the learned embedding weight W. E RIV respectively. In practice, we directly average this weight across its first dimension to compute the mean, and the standard deviation is also obtained across this dimension. _ E RD represents the element-wise reciprocal of de, and ( is the element-wist product. 0g E lR and Ø, E IR are the hyper-parameters for the denoising timestep f, and 6 E TR." denotes the standard Gaussian noise. With Equation (4), we can transform a clean embedding m into noisy embedding m For fine-tuning, the input sequence is divided into randomly sized segments, and we add the identical level of noise to the tokens from the same segment.
Finetuning Objective and Loss Function We fine-tune a pre-trained autoregressive text-to-image model predict the next clean token from the inputs of noisy token embeddings. The model accepts noisy token embeddings and outputs logits representing the categorical probability distribution of the next clean token. The cross-entropy loss computed between these logits and the ground truth token categories, optimizing the model to denoise inputs while maintaining autoregressive prediction. Specifically, analogous to standard next-token prediction, the cross-entrope loss is performed as follows: C = ENO Cross-Entropy (xi 1: p)+1). where of denotes the one-hot label of the ground- truth token at position i, N is the total number of tokens, and pP., denotes the conditional probability in Equation (2). Here, we assume that only the first token is prefilled with text conditioning. With this training objective and parameterization, the model learns to decode or verify clean tokens even when noisy tokens are present in the input conditions.
Timestep Injection Injecting the information of timesteps into noisy inputs is a common design for the denoising process [56, 63, 64]. To avoid introducing additional modules like AdaLN [56], we take the sinusoidal encodings of timesteps as a sequence of special token embeddings and append them to the sequence of input token embeddings during fine-tuning and decoding. Then, the sequence, which comprises input token embeddings and timestep encodings, is fed into the transformer blocks. Within the attention modules of these blocks, we use the attention mask to force each noisy token embedding to attend to the corresponding timestep encoding which indicates its noise level. To ensure
7

---

Latency
Denoising Iteratins
18
10
80 120
180
Jacobi Window Length
Prompt: Miss Mexiro portrait of the most bosutiful mexican woman, Exquisito detail
Figure 4:
Correlation between de- noising iterations and Jacobi window length on the latency. Circle areas rep- resent absolute latency values, and the lowest latency (a difference within 3 seconds is allowed) is in orange.
Figure 5: Study on embedding normalization for de- noising process. Left: Denoising output without embed- ding normalization, failing to generate coherent image. Right: Denoising output with embedding normalization, generating a semantically meaningful image.
Table Evaluation on the validation sets of MSCOCO [65].
COC02017 (Sk)
COC02014 (30k)
Configuration
Average Steps (1)
Step Comp- ression (t)
CLIP-
Average
Step Comp-
CLIP-
FID
FID
Score
Steps (4)
ression (?)
Score
Lumina-mGPT [11]
2357
1.00
30.8
31.3
2357
1.00x
21.0
31.3
SJD [6]
1060
2.23x
31.1
31.3
1057
2.23x
20.8
31.3
SJD2
592
4.02x
31.4
31.8
599
3.93x
21.1
31.9
Emu3 [4]
8193
1.00x
31.1
31.0
8193
1.00x
19.3
31.2
SJD [6]
3528
2.32x
30.6
30.9
3535
2.32x
21.4
31.0
SJD2
1461
5.62x
31.5
30.4
1461
5.63x
21.8
30.8
that the distribution of the timestep encodings aligns with that of the token embeddings, we apply process these encodings similar to Equation (4).
Experiments
5.1
Implementation Details
We perform experiments on two baselines, Lumina-mGPT [11] and Emu3 [4]. When generating an image least 720 720, Lumina-mGPT only needs about 2k tokens while Emu3 requires more than 8k tokens because of the difference between their tokenizers. For each fine-tuning, 8 GPUs with 80G memory are required for each model. Since all model parameters leverage DeepSpeed ZeRO-3 or FSDP with gradient checkpointing to save GPU memory at the cost of increased training time. The global batch size is set to 64, and the learning rate set to 2 x 10-5 with the AdamW W optimizer. We tune each model only within 6 epochs, costing approximately 14 x 8 A100 hours for Lumina-mGPT and 26 8 H100 hours for Emu3. By default, we set classifier-fres guidance to 3.0 and use top-2000 for the quantitative results of our method.
used for fine-tuning, we
Evaluation metrics. To assess visual quality, we employ two key metrics: FID [66] and CLIP- Score [67] on COCO benchmark [65] and GenEval benchmark [68]. To quantify the efficiency of the geaerated tokens which serves as decnding steps
decoding process, we introduce the step compression ratio [50]: S =
theoretical measure of parallelization and thus reflects the acceleration. For each benchmark, we compute the average step compression ratio across all generated images. Additionally, this ratio included alongside individual image samples in qualitative comparisons to highlight the performance differences between our method and other approaches. More importantly, we provide the latency of model forward passes a single GPU to evaluate the practical speedup achieved.

---

Table 2:
Visual quality GenEval benchmark [68].
Method
Colors Position Counting
Two Color Attri
Single Overall
Lumina-mGPT [11]
0.83
0.09
0.26
0.60
0.15
0.96
0,48
Lumina-mGPT + SJD [6]
0,82
0.08
0.23
0.63
0.12
0.96
0.47
Lumina-mGPT (tuned)
0.81
0.13
0.27
0.65
0.27
0.98
0.52
Lumina-mGPT (tuned) + SJD [6]
0.81
0.12
0.31
0.68
0.24
0.97
0,52
Lumina-mGPT (tuned) + SJD2
0.79
0.11
0.31
0.64
0.23
0.96
0.51
Emu3 [4]
0.78
0.15
0.33
0.69
0.16
0.98
0.52
Emu3 SJD [6]
0.79
0.12
0.28
0.61
0.13
0.97
0.48
Emu3 (tuned) + SJD2
0.73
0.14
0.28
0.61
0.24
0.96
0.49
5.2 Qualitative Results
In Figure and Figure 6, we compare the standard autoregressive decoding and our method on Lumina-mGPP [11]. The results show that our method can achieve about 4x fewer steps for autoregressive text-to-image generation and the visual quality preserved. also compare different decoding methods on Emu3 [4] in Figure 7.
5.3 Quantitative Results
Our SJD2 significantly reduces the steps for autoregressive pokt-to-image generation while maintaining visual quality, demonstrated by our evaluation MS-COCO [65] validation sets in Table 1. We also find that our method achieves a higher step compression ratio on Emu3 [4] than that on Lumina- mGPT [11]. Specifically, SJID2 can achieve step compression about 4.0x Lumina-mGPT and about 5.6x x on Emu3. For visual quality, we further compare our method to autoregressive decoding and SJD [6] on the GenEval benchmark [68] with Lumina-mGPT [11] as the baseline in Table 2. Our method achieves an overall score of 0.51, nearly matching the 0.52 of tuned AR and SJD, demonstrating preserved visual quality. More importantly, following [47], we select 100 COCO prompts on the identical A100 server to compare the practical average speedup and visual quality among these decoding methods. According to the latency reported Table 3, our method still faster than other decoding methods on the real server by more than 2x. We also provide the GPU memory usage during inference. While our parallel decoding method achieves significant latency reductions, we acknowledge it incurs an additional memory overhead of about 3GB compared to autoregressive decoding, because of the variables for denoising like the timestep tokens.
5.4 Ablation Studies
We perform experiments for method with Lumina-mGPT the baseline and on RTX 4090 by default. The selected 100 prompts used in Table 3 are for the evaluation in the ablation studies. We also include specific and detailed analysis for our denoising process in Appendix B.
Study on the sampling timesteps and Jacobi window length. While contemporary diffusion models typically require dozens of denoising iterations to achieve satisfactory results, extending Jacobi window lengths introduces computational overhead. This establishes a trade-off between denoising iteration counts and Jacobi window length for the minimization of latency. According to the results in Figure 4, when constraining denoising steps to 20 while maintaining Jacobi window lengths beyond 80, the latency reduction converges to a minimum.
Study on the embedding normalization. When verifying the usefulness of embedding normalization, we focus on our denoising process by enforcing the denoised tokens to be immediately accepted (as detailed in Appendix B). As shown Figure 5, deactivating embedding normalization results in complete failure of the denoising process, yielding pure noise instead of coherent images. Conversely, activating normalization enables the denoising process to generate reasonable outputs, demonstrating the critical role of embedding normalization in denoising process.

---

Table 3: The computational of decoding methods subset of COCO prompts.
Methods
Steps
Latency
CLIP-Score
GPU Memory
Lumina-mGPT [11]
2357
88.55s
32.0
17G
SJD [6]
1058
41.99s
31.5
17G
SJD2
596
33.64s
32.2
20G
Emu3 [4]
8193
375.29s
30.9
20G
SJD [6]
3537
207.60s
31.2
20G
SJD2
1470
147.65s
30.7
23G
Original
Original Shepe0113
SID Sheee 33842240 Faves
Suppe 2007
Shene TOBBI2 Fewe
5001l Faver)
Steper 189912 26 Fomer
po poeeeoaaaioo puclyy
Supe 2000
Steci FA501- a Fineeo
Stepe G81 el 66 Faver)
Shape 5490 Faver
Stipii 1981 28 Fower)
plccce.pppcoo..ooo aptlin corciing peteed Aps www.ppllo..cc.
Supe 2007
Stece 90 x Finee)
Supe o2d 20% Faner)
Siew 54307720 va Fower
Rlepe 1870ys to Power
Unapticry do
by propa pepple
118 Fnd Scaa Sohk sOllc GIN 9L707 ald A0/A///////
Figure 6: The comparison of the original au- toregressive decoding, SJD [6], and our method with Lumina-mGPT [11] as the baseline and on RTX 4090.
Figure 7: The comparison of the original au- toregressive decoding, SJD [6], and our method with Emu3 [4] as the baseline and on one RTX 4090.
Conclusion
This paper introduces Speculative Secobi-Denoising Decoding, a new algorithm integrating con- tinuous denoising process into Jacobi decoding accelerate autoregressive text-to-image generation. By extending next-token prediction into next-clean-token prediction with noisy inputs, we enable pre-trained autoregressive models to learn denoise noise- peiterperturbed token embeddings through a fine-tuning strategy. The proposed Jacobi- ceoobi-Deniisiggg decoding initializes token sequences with Gaussian noise. It then iteratively refines them using process that combines denoising steps with Jacobi decoding and an improved probabilistic prefix acceptance criterion. Experiments show that method reduce the number of model forward passes for acceleration while keeping the visual quality generated images.
Acknowledgment
This work is supported by the National Nature Science Foundation of China (No. 62402406).
10

---

References
[1] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:220.. 10789, 2022.
[2] Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. Videopoet: A large language model for zero-shot video generation. arXiv preprint (rXiv:2312.14125, 2023.
[3] Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.095 098888 2024.
[4] Emu3 Team BAAL Emu3: Next-token prediction is all you need, 2024. 1, 8, 9, 10, 23, 24
[5] Yang Song, Chenlin Meng, Renjie Liao, and Stefano Ermon. Accelerating feedforward compu- tation via parallel nonlinear equation solving. In International Conference Machine Learning, pages 9791-9800. PMLR, 2021. 1, 3, 4, 24, 25
[6] Yao Teng, Han Shi, Xian Liu, Xuefei Ning, Guohao Dai, Yu Wang, Zhenguo Li, and Xihui Liu. Accelerating auto-regressive text-to-image generation with training-free frree speculative jacobi decoding. arXiv preprint arXiv:2410.01699, 2024. 1. 4, 8, 9, 10, 24, 25
[7] Yang Song, Jascha Sohl-Dickttein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR. OpenReview.net, 2021.
[8] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR. OvenReview.net, 2021. 2.
[9] Qiang Liu. Rectified flow: A marginal preserving approach to optimal transport. arXiv preprint 2rXiv:2209.14577 2022.
[10] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In NeurlPS, 2022. 2, 4
[11] Dongyang Liu, Shitian Zhao, Le Zhuo, Weifeng Lin, Yu Qiao, Hongsheng Li, and Peng Gao. Lumina-mgpt: Illuminate flexible photorealistic text-to-image generation with multimodal generative pretraining. arXiv preprint arXiv:2408. 02657, 2024. 8, 9, 10, 23, 24, 25
[12] Tong Wu, Zhihao Fan, Xiao Liu, Hai-Tao Zheng, Yeyun Gong, Jian Jiao, Juntao Li, Jian Guo, Nan Duan, Weizhu Chen, al. Ar-diffusion: Auto-regressive diffusion model for text generation. Advances in Neural Information Processing Systems, 36:39957-39974, 2023. 3
[13] Xiang Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. Diffusion-Im improves controllable text generation. Advances in Neural Information Processing Systems, 35:4328 4343, 2022.
[14] Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, et al. Continu- diffusion for categorical data. arXiv preprint
1. 2022. 3
[15] Zhujin Gao, Junliang Guo, Xu Tan, Yongxin Zhu, Fang Zhang, Jiang Bian, and Linli Xu. Difformer: Empowering diffusion models on the embedding space for text generation. arXiv preprint arXiv:2212.094122 2022.
[16] Ishaan Guirajani and Tatsunori B Hashimoto. Likelihood baasee diffusion language models. Advances in Newral Information Processing Systems, 36, 2024.
[17] Enshu Liu, Xuefei Ning, Yu Wang, and Zinan Lin. Distilled decoding 1: One-step sampling of image auto-regressive models with flow matching, 2024. 3
11

---

[18] Vincent Hu, Di Wu, Yuki Asano, Pascal Mettes, Basura Fernando, Bjöm Ommer, and Cees Snoek. Flow matching for conditional text generation in a few sampling steps. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguisties (Volume 2: Short Papers), pages 380-392, 2024.
[19] Boyuan Chen, Diego Marti Monso, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. arXiv preprint arXiv:2407.01392, 2024. 3
[20] Kaifeng Gao, Jiaxin Shi, Hanwang Zhang. Chunping Wang, Jun Xiao, and Long Chen. Ca2- vdm: Efficient autoregressive video diffusion model with causal generation and cache sharing. arXiv preprini arXiv:2411. 2024.
[21] Jiatao Gu, Yuyang Wang. Yizhe Zhang, Qihang Zhang, Dinghuai Zhang, Navdeep Jaitly, Josh Susskind, and Shuangfei Zhai. Dart: Denoising autoregressive transformer for scalable text-to-image generation. arXiv preprint arXiv:2410.08155 2024.
[22] Chaorui Deng, Deyao Zh, Kunchang Li, Shi Guan, and Haoqi Fan. Causal diffusion transformers for generative modeling. arXiv preprint arXiv:2412. 220. 2024.
[23] Xun Huang, Zhengqi Li, Guande He, Mingyuan Zhou, and Eli Shechtman. Self forcing: Bridging the train-test gap in autoregressive video diffusion. arXiv preprint arXiv:2506.08009, 2025.
[24] Hansi Teng, Hongyu Jia, Lei Sun, Lingzhi Li, Maolin Li, Mingqiu Tang, Shuai Han, Tianning Zhang, WQ Zhang. Weifeng Luo, et al. Magi-l: Autoregressive video generation at scale. arXiv preprint arXiv:2505.13211 2025.
[25] Tianwei Yin, Qiang Zhang, Richard Zhang, William T Freeman, Fredo Durand, Eli Shechtman, and Xun Huang. From slow bidirectional to fast autoregressive video diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 22974422974 2025.
[26] David Ruhe, Jonathan Heek, Tim Salimans, and Emiel Hoogeboom. Rolling diffusion models. In Proceedings of the s1 International Conference on Machine Learning, 2024. 3
[27] Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408. 10099, 2024. 3
[28] Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Liang Zhao, et al. Janusflow: Harmonizing autoregres- sion and rectified flow for unified multimodal understanding and generation. arXiv preprint Stti:2411.07997. 2024.
[29] Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, al. Emerging properties in unified multimodal pretraining, arXiv preprint arXiv:2505. 146.. 2025. 3
[30] Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. arXiv preprint arXiv:2406. 1138, 2024. 3
[31] Lijie Fan, Tianhong Li, Siyang Qin, Yuanzhen Li, Chen Sun, Michael Rubinstein, Deqing Sun, Kaiming He, and Yonglong Tian. Fluid: Scaling autoregressive text-to-image generative models with continuous tokens. arXiv preprint arXiv:2410.1386: 2024.
[32] Haoge Deng, Ting Pan, Haiwen Diao, Zhengxiong Luo, Yufeng Cui, Huchuan Lu, Shiguang Shan, Yonggang Qi, and Xinlong Wang. Autoregressive video generation without vector quantization. arXiv preprint arXiv:2412.1416) 2024.
[33] Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan. Planting a seed of vision in large language model. arXiv preprint arXiv:2307.08041 2023. 3
12

---

[34] Runpei Dong, Chunrui Han, Yuang Peng. Zekun Qi, Zheng Ge, Jinrong Yang. Liang Zhao, Jian- jian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal comprehension and creation. arXiv preprint arXiv: 1499, 2023.
[35] Canyu Zhao, Mingyu Liu, Wen Wang, Weihua Chen, Fan Wang. Hao Chen, Bo Zhang, and Chunhua Shen. Moviedreamer: Hierarchical generation for coherent long visual sequence. arXiv preprint arXiv:2407. 16655, 2024.
[36] Yuying Ge, Sijie Zhao. Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi-granularity comprehension and generation. arXiv preprint arXiv:24044 /4396, 2024.
[37] Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14398-14409 2024.
[38] Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, et al. Blip3-o: A family of fully open unified multimodal models-architecctue,, training and dataset. arXiv preprint arXiv:250S.00 09568, 2025. 3
[39] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952. 2023. 3, 25
[40] Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit Blockwise parallel decoding for deep autoregressive models. Advances in Neural Information Processing Systems, 31, 2018. 3
[41] Tianle Cai, Yuhong Li, Zhengyang Geng. Hongwu Peng. Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple Ilm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401. 2024. 3
[42] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274-19286.
PMLR, 2023. 3.
[43] Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean- -Baptittt Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.013118 2023. 3.4 4
[44] Jacob K Christopher, Brian R Bartoldson, Bhavya Kailkhura, and Ferdinando Fioretto. Specula- tive diffusion decoding: Accelerating language generation through diffusion. arXiv preprint arXiv:2408.0550 056366 2024. 3
[45] Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. Spectr: Fast speculative decoding via optimal transport. Advances in Neural Information Processing Systems, 36, 2024.
[46] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401.15077 2024. 23, 25
[47] Doohyuk Jang, Sihwan Park, June Yong Yang, Yeonsung Jung, Jihun Yun, Souvik Kundu, Sung-Yub Kim, and Eunho Yang. Lantern: Accelerating visual autoregressive models with relaxed speculative decoding. arXiv preprint arXiv:2410.03335 2024. 3, 6, 9. 23, 25
[48] Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, et al. Scaling diffusion language models via adaptation from autoregressive models. arXiv preprint arXiv:2410. 17891, 2024.
[49] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances Neural Information Processing Systems, 34: 14:17981-17999 2021. 3
13

---

[50] Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of Ilm inference using lookahead decoding. arXiv preprint arXiv:2402.02057, 2024. 3,
[51] Siqi Kou, Lanxiang Hu, Zhezhi He, Zhijie Deng, and Hao Zhang. Cllms: Consistency large language models. arXiv preprint arXiv:2403.0088. 2024. 3
[52] Yuqing Wang, Shuhuai Ren, Zhijie Lin, Yujin Han, Haoyuan Guo, Zhenheng Yang, Difan Zou, Jiashi Feng, and Xihui Liu. Parallelized autoregressive visual generation. arXiv preprint arXiv:2412.15119, 2024. 3
[53] Yefei He, Feng Chen, Yuanyu He, Shaoxuan He, Hong Zhou, Kaipeng Zhang, and Bohan Zhuang. Zipar: Accelerating autoregressive image generation through spatial locality. arXiv preprint arXiv:24/2.04062, 2024. 3, 23,25 25
[54] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint 200.:3333.01469, 20213 3
[55] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pages 10674-10685, IEEE, 2022. 4, 25
[56] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195-4205, 2023. 7
[57] Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. arXiv preprint urXiv:2401.085 .08740, 2024.
[58] Shuai Wang, Zexian Li, Tianhui Song, Xubin Li, Tiezheng Ge, Bo Zheng, and Limin Wang. Flowden: Exploring dcn-like architectures for fast image generation with arbitrary resolution. arXiv preprint a1Xiv:2410.22665, 2024.
[59] Shuai Wang, Zhi Tian, Weilin Huang, and Limin Wang. Ddt: Decoupled diffusion transformer. arXiv preprint arXiv:2504.05741, 2025. 4
[60] Yao Teng, Yue Wu, Han Shi, Xuefei Ning, Guohao Dai. Yu Wang, Zhenguo Li, and Xihui Liu. Dim: Diffusion mamba for efficient high-resolution image synthesis. arXiv preprint
14X1v:2405.14224, 2024. 4
[61] Cheng Lu and Yang Song. Simplifying, stabilizing and scaling continuous-time consistency models. arXiv preprint arXi::4410.10081, 2024. 4
[62] Tero Karras, Mika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In NeurlPS, 2022. 6
[63] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint 10.00423 2023. 7
[64] Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 22669 22679,, 2023. 7
[65] Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014. 8, 9,
[66] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[67] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICML. volume 139 of Proceedings of Machine Learning Research, pages 8748-8788 8763. PMLR, 2021. 8
14

---

[68] Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems,36:521322
2023. 8, 9, 23, 24
[69] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu. Jialin Wang. Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision- language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 23
[70] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow trans- formers for high-resolution image synthesis. arXiv preprint arXiv:2403.03206 2024. 23, 25
[71] Peng Gao, Le Zhuo, Ziyi Lin, Chris Liu, Junsong Chen, Ruoyi Du, Enze Xie, Xu Luo, Longtian Qiu, Yuhang Zhang, et Lumina-12x: Transforming text into any modality, resolution, and duration via flow-based large diffusion transformers. arXiv preprint 2eXXx:2400.05550 2024. 23
[72] Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint axXiv:5501118111, 2025. 23, 24, 25
15

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the
relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). ). general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: This paper introduces a novel paradigm for autoregressive generation, which has been outlined in the abstract and introduction regarding the contributions of this paradigm.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
16

---

Justification:
Please refer to our appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
This paper does not include theoretical proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The model details and training details are outlined in Section 5,
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We only utilized publicly available datasets. We will release the code and models as soon as possible.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https: /ttp///npps.... cc/
public/ /tclc//guisssseee /oodeSuodiiieeeec ss ssionoolicyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NewrIPS code and data submission guidelines (https: / /nipss cttccc/ppaaee wwtubicc/guiie
Code Subeeuubmssee iimisssions MonPolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
18

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Section
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: We conducted multiple replications and found very minimal deviation in the experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section 5.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
19

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /n/.riuuspppppp/ http.//ccwcccsstacccccclllllllll
Answer: [Yes]
Justification: We confirm that the research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: We exclusively employ verified and secure data for training the generative model.
20

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: License: Custom (research, non-commereial)
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper swi thoode com/ dataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
21

---

Answer:
[NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: The LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standart components.
Please refer to our LLM policy (https: /wtprrieupppp.pp// cc/ Conferencese /2025/ LLM) for what should or should not described.
22

---

Appendix
A More Implementation Details
Experimental settings. We perform experiments on two baselines, Lumina-mGPT [11] and Emu3 [4]. We have collected approximately 80k high-resolution (at least 720 x 720) images from the Internet. For images lacking text descriptions, we caption them with Qwen2-VLL VL. [69]. We follow the advanced flow matching setting [70, 71] with a; 1 - t and or for our denoising process. By default, the length of the Jacobi window of our method set to 96 for Lumina-mGPT and 128 for Emu3, The number of denoising steps in SJD2 is set to 25.
B Further Analysis
The pure denoising process also generates reasonable images. In our approach, the speculative Jacobi decoding is applied after the denoising steps, so we need to recognize that the acceleration benefits stem not only from the Jacobi iterations but also from the effectiveness of the denoising process. To systematically verify that the denoising stage alone can produce valid tokens, we conduct the following experiment: when a token is successfully denoised and its distance from the first point of the Jacobi window is smaller than & (where I. represents the length of the Jacobi window and T denotes the number of denoising iterations), that the last token in the Jacobi window completes exactly T denoising iterations when it reaches the left part of the window. The results of this experiment are presented in Figure 9a. According to these results, we observe that our denoising process can generate reasonable images in certain cases, particularly those featuring characters. Additionally, the immediate acceptance of denoised tokens significantly reduces the number of steps and overall latency. In comparison, as shown in Figure 9b, we reintroduce Jacobi iterations into the decoding process. These iterations serve as a refinement mechanism, enhancing the image generation with more intricate details and reducing the artifacts. However, this improvement comes at the cost of increased computational steps. Therefore, to preserve image quality, we opt to keep the Jacobi iterations in our method.
immediately accept it. The ratio 4. guarantees
Analysis of unifying noise perturbation for discrete and continuous inputs. We demonstrate the feasibility of unifying noise perturbation for both the discrete and continuous inputs in the state-of- the-art transformer-basee models. Since the noise perturbation is commonly used in diffusion models, we first analyze the hehavior of noisy inputs in diffusion transformer (DiT) architecture. Although noise perturbation appears to occur in the latent space, the learnable linear transformation W RdH W + IR g (usually implemented by a 2D convolution without receptive field overlapping) before the transformer blocks causes the perturbation to actually happen in the feature space: W x; ar (Wx'* ) + F, where DHs Ws is random Gaussian variable formed by a linear weighted sum of independent Gaussian noises with the elements of W as weights, and D, Hs, Ws, d denote feature dimension, latent height, latent width and latent dimension, respectively. Therefore, based on the above equation, the noise perturbation can be interpreted as a linear combination of a clean feature vector and noise vector. Since both DiTs and autoregressive models rely on transformer blocks operating in the feature space, we aim to align their noise perturbation on the input embedding.
Analysis of Flops in inference. We present the average Flops per output token in Table 4. The results reveal that autoregressive decoding requires fewer Flops than SJD2. Although more Flops are used for decoding, the practical latency becomes lower. Actually, this Flops overload stems from the paradigm of all the speculative decoding methods. Their drafting-an ven-feiification mechanism, which inherently introduces computational overhead: the number of accepted tokens per sampling step is substantially lower than the number of input draft tokens.
Smaller baseline. We also implement SJD2 on Yanus-pro-1B 1B [72], an advanced autoregressive model much smaller than Lumina-mGPT [11]. The results in Table 5 and Table 6. From the results, we observe that SJD2 still can accelerate Janus-pro without sacrificing on generated image quality, as evidenced by the following Geneval metrics [68].
Comparison to other accelerating methods. We compare our method with the recent and classic speculative/parallel decoding methods, including Lantern [47], ZipAR [53], Eagle [46] and Jacobi
23

---

Table 4: The inference Flops of autoregressive models with different decoding methods.
Method
GFlops (L)
Latency (1)
Lumina-mGPT [11]
18.72
88.55s
Lumina-mGPT + SJD2
219.60
33.64s
Emu3 [4]
18.15
375.29s
Emu3 + SJD2
465.92
147.65s
Table 5: Inference performance comparison Manus-Pro-Bi [72].
Method
Latency (4)
Steps (4)
Kanus-Pro-lle IB [72]
9.1s
576
Lamus-Pro- 1B + SJD2
2.5s
144
Decoding [5], on COC02017 validation set [65] with Lumina-mGPT [11] as baseline. As shown Table 7, our approach achieves superior acceleration while maintaining comparable visual quality.
Comparison to diffusion models. While many autoregressive models currently underperform state-of-the-as diffusion models in image quality and face acceleration challenges, our SID2 narrows the speed gap. In Table 8, we evaluate inference latency for several commonly-used diffusion models (smaller than 3B) and Janus-Pro [72] at the same resolution (384 x 384). We set the number of sampling steps for diffusion models as 50. Results demonstrate that our SJD2 reduces latency of Janus-Pro-Ile IB from 9.1s to 2.5s, narrowing the gap between Janus-pre pro and the advanced diffusion models like SD3. Moreover, this result means Janus-Pro- IB with SJD2 already outperforms SDXL in speed (2.5s vs. 4.3s).
Investigation the refinement. In this paragraph, we demonstrate that SJD2 stabilizes the refinement trajectory, as illustrated in Figure 8, Specifically, we apply SJD2 and SJD [6] on Emu3 [4] respectively, and then we examine the first five tokens from the Jacobi window in one iteration, computing the times of the token change between adjacent steps and the cumulative changes. As shown in the first five figures, SJD2 yields identical initial predictions across the 25 sampling steps. As the noise level decreases, token predictions diversify but then become unchanged at several steps, indicating the stabilization of the token trajectory. In contrast, for SJD, the tokens consistently change, appearing to oscillate and remain unstable. The last figure of Figure 8 also shows that the SJI) causes more times of token change than SJD2.
C Limitations and Future Work
Although SJD2 can achieve similar step compression in various models, the improvements on actual latency not consistent, shown by Figure and Figure 7. We speculate that this is caused by the different sizes KV cache different autoregressive models (the model whose tokenizer has a low image compression ratio leads to large KV cache). A promising direction is to stabilize the latency acceleration of Jacobi-bare based acceleration methods across the models with different KV caches.
Table 6: Visual quality Geneval benchmark [68] with Janus-pro-1B [72]
the baseline.
Method
Colors
Position
Counting
Two
Color Attr
Single
Overall
Janus-Pro-113 1B (tuned) [72]
0.82
0.39
0.39
0.55
0.42
0.94
0.59
Janus-Pro- IB (tuned) + SJD2
0.83
0.45
0.37
0.59
0.38
0.96
0.60
24

---

Table 7: Comparison to other accelerating methods with Lumina-mGPT [11] as baseline.
Configuration
Acceleration Latency (1)
Acceleration Step (f)
CLIP-Score (t)
Autoregressive Decoding
x.00
1.00x
31.3
Jacobi Decoding [5]
1.02x
1.04x
31.4
SJD [6]
2.05x
2.23x
31.3
EAGLE [46]
2.10x
2.94x
33.3
LANTERN [47]
2.56x
3.63x
32.7
ZipAR [53]
1,82x
4.00x
31.2
SJD2
2.63x
4.02x
31.8
Table 8: Efficiency comparison with the diffusion model.
Method
Latency (L)
Steps (4)
Janus-Pro 1B [72]
9.1s
576
SD1.5 [55]
1.7s
50
SDXL [39]
4.3s
50
SD3-Medium [70]
1.7s
50
Janus-Pro- Paoss-rroBBBl IB + SJD2
2.5s
144
Tikent Canuativ Taken Differences
Teeez Cimulleive Tiken D Berennes
Tokmn Damilative "heen Differnnees
402
S82
Samdinn
Semsilig
Tokend Camuative Taken Differunces
Wouns Curulativee Token D iffernnoes
Stab Riy Compurison: Toxal Changes
Sanslin 300e
Samr Steps
Figure The trajectories of token difference.
25

---

D Impact Statement
Our paper proposes new method of autoregressive texi-to-image generation for research purposes. Real-world deployment would require additional safeguards beyond technical implementation. We recognize that this open technical accessibility could lead to potential societal risks, including misuse for generating misleading content or harmful biases. Fortunately, this problem can be alleviated by strict dataset filtering to exclude harmful content.
Latency: 19.49s
Lstency: Steps: 305
Latency: 19.40s
Steps: 306
Steps: 306
Prompt: Miss portrsitofdh rexicos werrais, Eaguiitt
Prompt elephant wearing armor, Eqquisila detall 3l-rraaapinsl
Prompt: mest Fantsoy high-quilit photos
(a) The images generated with the immediate acceptance of denoised tokens (without Jacobi iterations).
Latency: 32.80s
Latency: 34.83s Steps: 575
Latency: 35.18s Steps: 587
Steps: 538
Prompt: Niss Nesco portralt of besuzful meicat Exquiste drtail
Prompt: An old Chinese Esquiile deadl 32 ixepapinel 1k
Mastis the pest Fantass high
(b) The images generated with the combination of Jacobi itcrations and denoising.
Figure 9: Analysis of our denoising process. (a) Our denoising process without Jacobi iterations can generate reasonable images with few model forward passes and small latency. (b) The further Jacobi iterations refine the results of our denoising process, resulting in more details and fewer artifacts.
26


Reviews:
[{"review_id": "5hJtpmDq7a", "reviewer": "Reviewer_ucHV", "summary": "Speculative Jacobi-Denoising Decoding is the next step in proposing novel improvements to Speculative Jacobi Decoding for parallel token generation in autoregressive models. These changes result in faster convergence, reducing computational overhead, as well as equivalent or higher quality on several metrics demonstrated on image generation tasks.", "questions": "Do the authors have any user preference A/B tests to complement the GenEval result? It would be great to correlate these quantitative metrics to human A/B.\n\nIn eq(1), if the 1 in the lefthand side is decreased, what happens to the resulting quality? Do the authors have any study, reference, or ablation on this - I think it might also improve the convergence speed of Speculative Jacobi Decoding, but maybe at the expense of quality?\n\nDo the authors have any high-order statistics, like skewness, kurtosis, and so on about the distribution of embeddings before and after normalization? It is slightly surprising to me that simple mean standard deviation normalization is enough to Gaussianize sufficiently to prevent the failure modes discussed in the paper. It would be good to have some information, if people want to apply this method but have different embedding distributions up front.\n\nIs there any potential of a \"training free\" version of this method, if not what are the key roadblocks (technical, computational, theoretical) to such a version?\n\nWill the authors release code to help reproduction in open source tools?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\nDetailed ablations and metric studies, as well as a clear description of the approach. I think this is a good, informative paper which is useful for people to get introduced to Jacobi decoding methods, as well as see the line of research leading up to this new method. The experimental section is focused on images, but clear in the results and goals. Descriptions of the inference time procedure are high quality, and presented with great figures and examples.\n\nWeaknesses:\nGiven the widespread applicability of Jacobi decoding and descendants, it would be really interesting to see this approach applied to other generative applications, like text or audio token generation. It may be out of scope for this review cycle, but really worth considering for broad appeal of the method.\n\nIt is not clear to me if the full model is trained during the finetuning procedure, or only a small part. There are some other complaints below, but a bit more indepth treatment of the tuning stage, and what parameters are involved would be useful especially for followup work.\n\nThere is a strong conceptual convergence between the approach detailed here, and the paper \"Rolling Diffusion Models\", which adds a rolling time-window approach into continuous diffusion. It it is probably worth referencing or briefly discussing in the background material.\n\nThere was some mention of compute costs, but it would be useful to detail how long the finetuning takes in terms of approximate wall-clock time, as well as further details on the finetune hyperparameters in general.\n\nThere is a statement \"For fine-tuning, we divide the input sequence into randomly sized segments and add noise to each segment.\", it would be useful to have details about the procedure to select the randomly sized adjacent segments, and how the noise level is chosen and applied over training. There is \"We follow the advanced flow matching setting with αt = 1 − t and σt = t for our denoising process\", but more details would be better.\n\nSimilarly, there is a statement \"we take the normalized Fourier encodings of timesteps\" - is this different than a standard positional encoding? If so, it would be good to add more detail, or cite a relevant paper defining this. Otherwise, change the writing slightly to tie in to the more standard naming with \"Fourier positional encodings\" or \"Transformer sinusoidal position encodings\", or something like this.", "was_revised": true, "final_justification": "I like this paper and think it has large applicability with broad interest for a variety of autoregressive modeling approaches. Given the focused scope of the work and the clarity of discourse, I think it should be a \"strong accept\" for any conference. I really like the clear explanation and practicality of the method.", "rebuttal": "# Response to Reviewer ucHV\n\nThank you for your thoughtful and constructive feedback on our paper. We particularly appreciate your recognition of our detailed ablation studies, clear methodological explanations, and high-quality experimental demonstrations.\n\n## **1. About potential application on other domains like text/audio**\n\nThank you for your thoughtful suggestion about extending Jacobi decoding to text and audio generation. While such expansion falls beyond the scope of current review cycle, we plan future research to pursue this direction, including curating diverse text/audio datasets and exploring integrations with autoregressive LLMs like Llama and Qwen by using SJD2.\n\n\n## **2. About fine-tuning details and the training/inference cost**\nAll model parameters are used for fine-tuning. To build the dataset, we collect about 80,000 synthesized images from huggingface and recaption them with Qwen-VL. We perform fine-tuning on 8 NVIDIA GPUs (80GB memory each) with a global batch size of 64, a learning rate of 2e-5, and the AdamW optimizer ($\\beta_1$=0.9, $\\beta_2$=0.95), while leveraging DeepSpeed ZeRO-3 or FSDP with gradient checkpointing to save GPU memory at the cost of increased training time. The fine-tuning requires 6 epochs, which costs about $14 \\times 8$ A100 hours for Lumina-mGPT and $26 \\times 8$ H100 hours for Emu3. The inference cost is in the following table:\n\n| Method | GPU memory | Latency|\n|---------|---------------|---------------|\n| Lumina-mGPT (AR) | 17G | 88.55s |\n| Lumina-mGPT (SJD2) | 20G  | 33.64s |\n| Emu3 (AR) | 20G | 375.29s |\n| Emu3 (SJD2) | 23G  | 147.65s |\n\n## **3. About discussion on Rolling Diffusion Models**\n\n\nThank you for your suggestion. SJD2 accelerates autoregressive text-to-image generation by integrating denoising with Jacobi iterations, using a fixed sliding window for parallel token prediction in discrete token space. It fine-tunes pre-trained models to predict conditional probability given the input of noisy token embeddings, and uses the cross-entropy loss for supervision. During inference, the token embedding is first denoised and then seamlessly enters the standard Speculative Jacobi iterations draft and verify its corresponding discrete token and probability. In contrast, Rolling Diffusion Model applies a continuous diffusion process with a rolling time-window for time-series data, trained with diffusion loss to predict noise in continuous space. While both use window-based denoising for efficiency, SJD2 focuses on the discrete tokens in autoregressive models, whereas Rolling Diffusion Model focuses on adapting continuous diffusion for temporal sequences. We will add the discussion in the camera-ready version.\n\n\n## **4. About segment size and noise level selection**\nThe process of sequence segment dividing and noise level selection is as follows: First, we sample a monotonically increasing timestep sequence $( 0, t\\_{0}, \\cdots, t\\_{K'} , 1 )$ from the range of $[0,1]$, where $K'$ is randomly determined in range $[10, 50]$ (the typical number of steps in diffusion sampling). The elements $t\\_i$ in this timestep sequence can adhere to the Karras timestep schedule [A]. Then, for an input token sequence of length $N$ ($N > K'$), we uniformly partition it into $n$ segments ($n$ is randomly chosen from range $[1, \\frac{N}{K'}]$), and fill each segment uniformly with the timestep sequence $( 0, t\\_{0}, \\cdots, t\\_{K'} , 1 )$ across all positions. These timesteps are used to calculate the parameters $\\alpha_t=1-t$ and $\\sigma_t=t$ for noise perturbation on token embeddings.\n\n[A] Karras T, Aittala M, Aila T, et al. Elucidating the design space of diffusion-based generative models[J]. Advances in neural information processing systems, 2022, 35: 26565-26577.\n\n## **5. About the writting of positional encoding**\n\nThank you for your suggesion. Our position encoding is the same as the sinusoidal position encodings used in standard Transformers. We will add this explanation in the camera-ready version.\n\n## **6. About the user preference A/B tests**\n\nWe conduct user preference on the images generated by autoregressive decoding and SJD2. We use Lumina-mGPT as baseline and select 16 pairs of images with 10 users for study. The results are in the following table. We observe that the preference of the two decoding methods is close and the images generated by autoregressive decoding is slightly more preferred.\n\n| Method  | Preference |\n|---------|--------------|\n| AR | 54.7\\% |\n| SJD2 | 45.3\\%   |\n\n## **7. The influence of decreasing the 1 in the lefthand side in Equation 1**\n\nThank you for your question. First, for simplicity, let's consider replacing the constant \"1\" on the left-hand side of Equation 1 with a variable $k$. Actually, decreasing this variable $k$ would reduce the acceptance rate, as a lower $k$ makes the acceptance condition $r <  \\min \\left( k,  \\frac{\\mathcal{P}\\_\\theta(x\\_i^{(j)} | x\\_{1}^{(j)}, \\cdots, x\\_{i-1}^{(j)})}{\\mathcal{P}\\_\\theta(x\\_i^{(j)} |  x\\_{1}^{(j')}, \\cdots, x\\_{i-1}^{(j')}  )}  \\right) , ~~ r \\sim   \\mathcal{U}[0,1]$ harder to satisfy. We also conduct corresponding experiments and the results aligns with our hypothesis. Thus, $k=1.0$ not only conforms to the theory of speculative sampling, but also guarantees a relatively large acceptance rate.\n\n| Method  | Steps |\n|---------|--------------|\n| Emu3 + SJD ($k=1.0$) | 3537 |\n| Emu3 + SJD ($k=0.5$) | 4969   |\n\n## **8. About relationship between normalization and the failure mode** \n\nWe do not utilize high-order statistics like skewness or kurtosis for normalization and denormalization. Actually, the failure mode depicted in Figure 5 arises when the noise-perturbation-based fine-tuning (and the denoising sampling) applied *directly* to token embeddings *without normalization and denormalization*. As we discussed in lines 213-220, normalization and denormalization primarily serve to *scale* input values appropriately for noise perturbation and transformer blocks, respectively. The noise perturbation formula $x\\_t = \\alpha\\_t \\cdot x\\_0 + \\sigma\\_t \\cdot \\epsilon$, where $\\epsilon \\sim N(0, I)$, can produce extreme values, i.e., $\\epsilon$ may reach magnitudes of 6 or -6, because $\\epsilon$ is sampled from a standard Gaussian distribution. However, we observe that the token embedding statistics are consistently small across many models (e.g., mean = 0.0179, std = 0.2709). It is challenging for models to adapt to large-value inputs with few fine-tuning epochs. Consequently, we implement noise perturbation on normalized embedding values and subsequently rescale them for model input.\n\n\n## **9. The possibility of a training-free version of our method** \nCurrently, it is challenging to achieve a training-free variant of our method. The difficulty stems from the fact that autoregressive models are not exposed to Gaussian-perturbed tokens during their standard training process. These models are typically optimized for next-token prediction using discrete tokens generated by tokenizer encoder (e.g., VQGAN with CNN encoders), where the inputs are not intentionally corrupted with Gaussian noise throughout pre-training. Consequently, the pre-trained autoregressive models may not encounter noisy token representations, making it difficult to generalize to such inputs without explicit training adaptation.\n\n\n## **10. About the open-source**\nWe will release our code upon paper acceptance."}, {"review_id": "nO4TGF38OV", "reviewer": "Reviewer_Q75X", "summary": "This paper introduces Speculative Jacobi-Denoising Decoding (SJD2), a method for enabling parallel token generation in autoregressive models. The authors propose a next-clean-token-prediction task that allows a model to predict multiple future tokens with only light fine-tuning. Experiments on MSCOCO and GenEval show that this approach can achieve a speedup of around 4x without a significant drop in performance.", "questions": "- Why was there no comparison with the Emu3 model on the GenEval benchmark, even though it was used in other experiments?\n\n- Are there any comparisons of computational cost in terms of FLOPs? This would provide a more detailed efficiency analysis beyond latency.\n\n- How does the proposed SJD2 approach compare to other parallel decoding methods from recent literature? A broader comparison is needed to position this work.\n\n- Is SJD2 compatible with autoregressive models that already use a form of parallel decoding, such as MAR models?\n\n- Could you provide an analysis or comparison of the training cost required for the proposed SJD2 approach?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- The paper is well-written, and the ideas and methods are described clearly. The main figure illustrates the core idea very well.\n\n- The evaluation on MSCOCO shows that the method can speed up models like Lumina-mGPT and Emu3 by about 4x without harming performance too much.\n\n- The qualitative results look good and effectively support the main experimental claims.\n\nWeaknesses\n\n- Limited Scope of Model Compatibility: It is unclear if the proposed SJD2 method works with a wider range of autoregressive models. For example, compatibility with other advanced autoregressive approaches like VAR or Janus Pro is not discussed.\n\n- Insufficient Baseline Comparisons: The main experiments lack comparisons with other modern speculative or parallel decoding methods. The only baseline compared against is the original SJD, which makes it hard to judge SJD2's performance relative to the state of the art.\n\n- Missing Comparison with Diffusion Models: The paper does not include a comparison of the efficiency/performance trade-off against state-of-the-art diffusion models.\n\n- Weak Performance on GenEval: The overall performance on the GenEval benchmark does not appear to be competitive with many existing methods.", "was_revised": true, "final_justification": "The author's rebuttal resolves most of my concerns and I'm willing to increase the score. Authors are encouraged to update the manuscript to reflect the comparison with more baselines.", "rebuttal": "# Response to Reviewer Q75X\n\n\nThank you for your thoughtful and constructive feedback on our paper. We particularly appreciate your recognition of the clear methodological presentation and our performance of acceleration. Our primary contribution lies in enabling efficient parallelization of existing autoregressive architectures through lightweight fine-tuning, rather than pursuing state-of-the-art performance.\n\n\n## **1. Results on more autoregressive models** \n\nThank you for your suggestion. As our SJD2 is developed for the paradigm of next-token prediction, we choose Janus-pro-1B for experiments. The paradigm of VAR is next-scale prediction which could require different ways of acceleration and thus is beyond the scope of our paper. Specifically, we employ our pairwise text-image data to tune Janus-pro-1B to be compatiable with noisy embedding inputs and use our SJD2 for decoding. The results are in the following tables. From the results, we observe that SJD2 still can accelerate Janus-pro without sacrificing on generated image quality, as evidenced by the following Geneval metrics.\n\n| Method | Latency | Steps |\n|---------|---------------|------------------|\n| Janus-Pro-1B | 9.1s | 576 |\n| Janus-Pro-1B + SJD2 | 2.5s | 144  |\n\n| Method | Colors | Position | Counting | Two | Color Attri| Single | Overall |\n|---------|---------------|------------------|------------------|---------|----------------|----------------|----------------|\n| Janus-Pro-1B (AR) |  0.82 | 0.39   |  0.39 | 0.55  |  0.42 | 0.94 | 0.59  |\n| Janus-Pro-1B (SJD2) | 0.83 | 0.45   | 0.37  | 0.59  | 0.38  |0.96 | 0.60 |\n\n## **2. Comparison to modern parallel decoding methods**\nWe compare our method with the recent state-of-the-art and classic speculative/parallel decoding methods, including Lantern (ICLR 2025) [A], ZipAR (ICML 2025) [B], Eagle [C] and Jacobi Decoding [D], on COCO2017 validation set with Lumina-mGPT as baseline. As shown in the table below, our approach achieves superior acceleration while maintaining comparable visual quality.\n\n| Configuration | Acceleration Latency $(\\uparrow)$ | Acceleration Step $(\\uparrow)$ | CLIP-Score $(\\uparrow)$ |\n|---------------|------------------|------------------|---------|\n|  AR | $1.00 \\times$ | $1.00 \\times$ | $31.3$ |\n|  Jacobi Decoding | $1.02 \\times$ | $1.04 \\times$ |$31.4$ |\n|  SJD | $2.05 \\times$ | $2.23 \\times$ |  $31.3$ |\n| EAGLE | $2.10 \\times$ | $2.94 \\times$ | $33.3$ |\n|  LANTERN | $2.56 \\times$ | $3.63 \\times$ | $32.7$ |\n|  ZipAR | $1.82 \\times$ | $4.00 \\times$ |$31.2$ |\n|  Our SJD2 | $2.63 \\times$ | $4.02 \\times$ |  $31.8$   |\n\n[A] Jang D, Park S, Yang J Y, et al. Lantern: Accelerating visual autoregressive models with relaxed speculative decoding[C]. ICLR 2025.\n[B] He Y, Chen F, He Y, et al. ZipAR: Parallel Autoregressive Image Generation through Spatial Locality[C]. Forty-second International Conference on Machine Learning (ICML 2025).\n[C] Li Y, Wei F, Zhang C, et al. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty[C]. Forty-first International Conference on Machine Learning (ICML 2024).\n[D] Song Y, Meng C, Liao R, et al. Accelerating feedforward computation via parallel nonlinear equation solving[C]. PMLR 2021.\n\n\n## **3. About the lack of comparison to diffusion models**\n\nActually, our work focuses on improving the inference efficiency of standard autoregressive (AR) models for image generation. Thus, we choose these baselines: Lumina-mGPT, Emu3, and Janus-Pro that is supplemented in this rebuttal. The AR models have advantages in the unified modeling of multimodal data (predicting discrete tokens for both the linguistic and visual tasks, different from the diffusion models) and thus it is worth studying [A, B, C]. While many AR models currently underperform state-of-the-art diffusion models in image quality and face acceleration challenges, our SJD2 narrows the speed gap. In the following table, we evaluate inference latency for several commonly-used diffusion models (smaller than 3B) and Janus-Pro at the same resolution ($384 \\times 384$). We set the number of sampling steps for diffusion models as 50. Results demonstrate that our SJD2 reduces latency of Janus-Pro-1B from 9.1s to 2.5s, narrowing the gap between Janus-pro and the advanced diffusion models like SD3. Moreover, this result means Janus-Pro-1B with SJD2 already outperforms SDXL in speed (2.5s vs. 4.3s):\n\n| Method | Latency | Steps |\n|---------|---------------|------------------|\n| Janus-Pro-1B | 9.1s | 576 |\n| SD1.5 | 1.7s |  50  |\n| SDXL | *4.3s* | 50  |\n| SD3-Medium | *1.7s* | 50  |\n| Janus-Pro-1B + SJD2 | *2.5s* | 144  |\n\n[A] Chen X, Wu Z, Liu X, et al. Janus-pro: Unified multimodal understanding and generation with data and model scaling[J]. arXiv preprint arXiv:2501.17811, 2025.\n\n[B] Wang X, Zhang X, Luo Z, et al. Emu3: Next-token prediction is all you need[J]. arXiv preprint arXiv:2409.18869, 2024.\n\n[C] Team C. Chameleon: Mixed-modal early-fusion foundation models[J]. arXiv preprint arXiv:2405.09818, 2024.\n\n## **4. Regarding Performance on GenEval**\nThis work focuses on **accelerating existing pre-trained autoregressive models while preserving visual quality**, not on achieving state-of-the-art image quality. As shown in Table 2, our method successfully maintains visual fidelity on GenEval. While current autoregressive baselines like Lumina-mGPT generally underperform compared to advanced diffusion-based generation models, our proposed method has the potential to be applied to stronger autoregressive base models in the future.\n\n\n## **5. Performance of Emu3 on Geneval benchmark**\n\nIn the following table, we compare our SJD2 to autoregressive decoding (AR) and SJD on the GenEval benchmark with Emu3 as the baseline. Our method achieves an overall score of 0.49, which is close to the 0.52 of AR and the 0.48 of SJD, demonstrating preserved visual quality.\n\n| Method | Colors | Position | Counting | Two | Color Attri| Single | Overall |\n|---------|---------------|------------------|------------------|---------|----------------|----------------|----------------|\n| AR | 0.78 | 0.15   | 0.33  | 0.69  | 0.16  |0.98 | 0.52 |\n| SJD | 0.79 | 0.12   | 0.28  | 0.61  | 0.13  |0.97 | 0.48 |\n| SJD2 | 0.73 | 0.14  | 0.28  | 0.61 | 0.24  |0.96 | 0.49 |\n\n\n## **6. Flops in inference**\n\nWe present the average Flops per output token in the table below. The results reveal that autoregressive decoding requires fewer Flops than SJD2. *Although more Flops are used for decoding, the practical latency becomes lower*. Actually, this Flops overload stems from the paradigm of all the speculative decoding methods. Their drafting-and-verification mechanism, which inherently introduces computational overhead: the number of accepted tokens per sampling step is substantially lower than the number of input draft tokens. Previous studies have also confirmed this observation. For example, as demonstrated in Tables 4 and 5 of [A], Medusa (a speculative decoding method) consumes 403 GFlops per token, whereas vanilla autoregressive decoding requires only 19 GFlops.\n\n| Method | GFlops |  Latency|\n|---------|---------------|-------------|\n| Lumina-mGPT (AR) | 18.72 | 88.55s |\n| Lumina-mGPT (SJD2) | 219.60 | 33.64s |\n| Emu3 (AR) | 18.15 |  375.29s |\n| Emu3 (SJD2) |  465.92 | 147.65s |\n\n[A] Lin C H, Tuli S, Smith J, et al. SLiM: Speculative decoding with hypothesis reduction[C]//Findings of the Association for Computational Linguistics: NAACL 2024. 2024: 1005-1017.\n\n\n\n## **7. About compatibility of parallel-decoding models**\nThank you for your question. Actually, speculative decoding was originally designed to accelerate standard autoregressive models that decode tokens sequentially (one token per forward pass). Its lossless acceleration relies on optimizing GPU parallelism utilization. However, the models employing parallel decoding techniques (e.g., MaskGit or the next-set-prediction of MAR) inherently utilizes GPU parallelism well. Consequently, speculative decoding methods like SJD2 would require redesign and new theoretical foundations to assist these frameworks, which can be explored for future works. Regarding the diffusion head of MAR which operates on continuous latents, the speculative decoding should be redesigned to compatiable with Gaussian inputs, and some concurrent efforsts have been conducted and still in exploration [A].\n\n[A] Wang Z, Zhang R, Ding K, et al. Continuous speculative decoding for autoregressive image generation[J]. arXiv preprint arXiv:2411.11925, 2024.\n\n## **8. Comparison of the training cost of SJD2**\nRegarding the fine-tuning cost, the baseline, Lumina-mGPT, requires 10 million image-text pairs for pre-training. In contrast, our fine-tuning uses only 80,000 images over 6 epochs, totaling approximately 0.5 million image samples used during training. This indicates that at most 5% of the original computational resources are used for fine-tuning our method. We only use $14\\times8$ A100 hours for fine-tuning, highlighting the efficiency."}, {"review_id": "LHzr8dRE5G", "reviewer": "Reviewer_eMe4", "summary": "This paper proposed Speculative Jacobi-Denoising Decoding (SJD2) to accelerate autoregressive text-to-image model. By incorporating diffusion denoising with speculative decoding for AR model, the paper reaches 4x speed up on common AR image generation model.", "questions": "1. The index of embedding $ e $ in Eq.(2) is confused, please explain why the superscript skips from 0 to $ t_0 $, and what is $ i $ and $ i' $ in subscript.  \n2. Does the original AR model utilize normalized embedding as input? \n3. Please explain the training objective and process, I only see a cross-entropy loss for AR model, how did this work for the diffusion? If there is a multi-step denoising process, would this involve a high-order gradient?\n4. What is the finetuning effort of SJD2, like the dataset size and training time.\n5. Please compare with more parallel decoding methods, like Jacobi Decoding, Lookahead Decoding, CLLMs, and ZipAR.\n6. How effective is the Refinement with Denoising? Is the difference between SJD and SJD2 this refinement? If no, there is a lack of relevant ablation experiments.", "limitations": "See Weaknesses and Questions", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The background of SJD2 is well introduced in the first section.\n2. SJD2 gets good acceleration with almost no loss on image quality.\n3. The method works on both text-to-image and text-to-video models.\n\n**Weaknesses**\n1. Based as pervious work SJD, this work is only a minimal piece of contribution, besides, as stated in related work, the integration of diffusion has been studied in many visual generation works.\n2. The method section is not well organized, needs more details, like the finetuning objective, the timestep injection and so on.\n3. The difference between SJD2 and AR+diffusion methods are not introduced in related works.\n4. The method comparison is not effective, only SJD are compared.", "was_revised": true, "final_justification": "My final rating is 4 Borderline accept. The authors proposed a decoding method to accelerate the AR image generation model, which shows benefits. My concerns are answered, the paper should be revised to make it clearer for audience.", "rebuttal": "# Response to Reviewer eMe4\n\nThank you for your thoughtful and constructive feedback on our paper. We appreciate your recognition of our introduction and the significant acceleration.\n\n## **1. About our contribution**\n\nWe feel it necessary to clarify the contribution of our work: our SJD2 introduces a new advancement by integrating denoising process into autoregressive (AR) models to **stabilize the Jacobi iteration in SJD**. We achieve this through **fine-tuning** AR models to be **compatible with noisy embeddings** (thus the denoising can be integrated in Jacobi iteration, shown in Eq. 3, lines 191-202), while distinctively **preserving core AR mechanisms** (discrete tokenization, next-token prediction with cross-entropy loss, and Jacobi iteration) via the proposed techniques like **noise perturbation on normalized embeddings**. Unlike existing AR+diffusion models that rely on diffusion losses and auxiliary decoders for image generation, SJD2 **maintains the standard AR components for image generation** (i.e. **predicting the probability of discrete tokens**, and **training with cross-entropy loss**) while delivering significant acceleration gains (shown in Table 1).\n\n## **2. About details of our finetuning objective and timestep injection**\n\nThank you for your suggestion. We will improve the clarity of the method section in the camera-ready version and we detail our finetuning objective and timestep injection here: **(A) timestep injection**: Since we avoid introducing additional adapters, we utilize a flexible operator, i.e., the attention mechanism, for our token-wise timestep injection. Specifically, we take the sinusoidal encodings of timesteps as a sequence of special token embeddings and append them to the sequence of input token embeddings. Then, the sequence, which comprises clean input token embeddings, noisy input token embeddings, and timestep encodings, is fed into the transformer blocks. Within the attention modules of these blocks, we use the attention mask to force each noisy token embedding to attend to the corresponding timestep encoding which indicates its noise level. To ensure the distribution of the Fourier encodings of timesteps aligns with that of the token embeddings, we apply a normalization-then-de-normalization process to these encodings (normalization with statistics of the sinusoidal encodings and denormalization with the statistics of the token embeddings), similar to Eq 4. **(B) finetuning objective**: SJD2 fine-tunes a pre-trained autoregressive text-to-image model to predict the next clean token from noisy token embeddings. During fine-tuning, input token embeddings are perturbed with Gaussian noise and then fed into transformer blocks followed by a prediction head which outputs logits representing the categorical probability distribution of the next clean token. The cross-entropy loss is computed between these logits and the ground truth token categories, optimizing the model to denoise inputs while maintaining autoregressive prediction. This objective enables SJD2 to handle noisy inputs during inference.\n\n\n## **3. Difference between SJD2 and AR+diffusion methods**\n\nIn our related work section (Integration of autoregression and continuous diffusion), we have discussed models like Diffusion-forcing and Transfusion. However, with the recent publication of numerous new AR+Diffusion works, we now provide an updated version of the related work part: Recent diffusion forcing models (e.g. Self-forcing [A], MAGI-1 [B], and Causvid [C]) introduce the autoregressive sampling into the temporal dimension of the continuous video diffusion models. They employ causal masks to support history-conditioned video generation. The multimodal large language models (MLLMs) (e.g. BLIP-3o [D], Emu2 [E], and Seed-X [F]) generate images via pre-trained diffusion models (e.g., SDXL) which take the output features of autoregressive models as conditions. Moreover, the unified models like Bagel [G] and Janus-flow [H] directly integrate diffusion process into the autoregressive backbones. These models train the autoregressive backbones together with lightweight decoders through diffusion loss (e.g., noise prediction or velocity prediction) for image generation, and the backbone can perform both autoregressive decoding and continous diffusion sampling. In contrast, SJD2 uniquely integrates denoising into pre-trained autoregressive models without modifying their core: **preserving discrete tokenizers, next-token prediction with cross-entropy loss, and the established inference mechanisms**. Our innovation lies in **fine-tuning with noise-perturbed normalized embeddings**, enabling ODE-based denoising in autoregressive models. We do **not** relay on diffusion loss and additional decoders.\n\n[A] Huang X, Li Z, He G, et al. Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion[J]. arXiv 2025.\n\n[B] Teng H, Jia H, Sun L, et al. MAGI-1: Autoregressive Video Generation at Scale[J]. arXiv 2025.\n\n[C] Yin T, Zhang Q, Zhang R, et al. From slow bidirectional to fast autoregressive video diffusion models[C]. CVPR 2025.\n\n[D] Chen J, Xu Z, Pan X, et al. Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset[J]. arXiv 2025.\n\n[E] Sun Q, Cui Y, Zhang X, et al. Generative multimodal models are in-context learners[C]. CVPR 2024.\n\n[F] Ge Y, Zhao S, Zhu J, et al. Seed-x: Multimodal models with unified multi-granularity comprehension and generation[J]. arXiv 2024.\n\n[G] Deng C, Zhu D, Li K, et al. Emerging properties in unified multimodal pretraining[J]. arXiv 2025.\n\n[H] Ma Y, Liu X, Chen X, et al. Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation[C]. CVPR 2025.\n\n\n## **4. Comparison to modern parallel decoding methods** \n\nWe compare our method with the recent state-of-the-art and classic speculative/parallel decoding methods, including Lantern (ICLR 2025) [A], ZipAR (ICML 2025) [B], Eagle [C] and Jacobi Decoding [D], on COCO2017 validation set with Lumina-mGPT as baseline. As shown in the table below, our approach achieves superior acceleration while maintaining comparable visual quality.\n\n| Configuration | Acceleration Latency $(\\uparrow)$ | Acceleration Step $(\\uparrow)$ | CLIP-Score $(\\uparrow)$ |\n|---------------|------------------|------------------|---------|\n|  AR | $1.00 \\times$ | $1.00 \\times$ | $31.3$ |\n|  Jacobi Decoding | $1.02 \\times$ | $1.04 \\times$ |$31.4$ |\n|  SJD | $2.05 \\times$ | $2.23 \\times$ |  $31.3$ |\n| EAGLE | $2.10 \\times$ | $2.94 \\times$ | $33.3$ |\n|  LANTERN | $2.56 \\times$ | $3.63 \\times$ | $32.7$ |\n|  ZipAR | $1.82 \\times$ | $4.00 \\times$ |$31.2$ |\n|  Ours | $2.63 \\times$ | $4.02 \\times$ |  $31.8$   |\n\n[A] Jang D, Park S, Yang J Y, et al. Lantern: Accelerating visual autoregressive models with relaxed speculative decoding[C]. ICLR 2025.\n\n[B] He Y, Chen F, He Y, et al. ZipAR: Parallel Autoregressive Image Generation through Spatial Locality[C]. Forty-second International Conference on Machine Learning (ICML 2025).\n\n[C] Li Y, Wei F, Zhang C, et al. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty[C]. Forty-first International Conference on Machine Learning (ICML 2024).\n\n[D] Song Y, Meng C, Liao R, et al. Accelerating feedforward computation via parallel nonlinear equation solving[C]. PMLR 2021.\n\n\n## **5. About the superscript and subscript in equation 2**\n\nThe superscript in Equation (2) indicates the noise level applied to the embedding $e$, with the skip from $0$ to $t_0$ reflecting a deliberate design in the application of noise perturbation. As demonstrated in Figure 3, some regions of the embeddings remain unperturbed by noise (denoted by superscript 0), while their adjacent regions are perturbed to a minimal noise level (denoted by superscript $t_0$). The subscript $i$ identifies the position of a specific token embedding, and $i'$ is an integer in the range $[1, i]$, used to specify a certain position where the noise level shifts from $0$ to $t_0$.\n\n## **6. About inputs in original autoregressive models**\n\nThe pre-trained autoregressive models used as baselines are incompatible with normalized embedding inputs. Consequently, after performing denoising on normalized embedding inputs, we employ a **denormalization** step (Equation 4) to restore these noisy normalized embeddings to the original embedding space.\n\n\n## **7. About the training process**\n\nUnlike standard diffusion models that rely on diffusion loss (i.e. $\\mathbb{E}\\_{t,\\epsilon,x_0} [ ||v\\_\\theta(x\\_t,t)-(\\epsilon-x\\_0)|| ]$), the training objective of SJD2 is based **solely on cross-entropy loss**. This is because SJD2 is **not** a standard diffusion model: it only **leverages the denoising concept from diffusion models** while **not** using the diffusion loss. Specifically, during training, we perturb token embeddings with noise and the cross-entropy loss is applied on the noisy embeddings. During inference, the ODE denoising process is fused with the Jacobi decoding which is also a multi-step decoding algorithm (line 191-202 with equation 3). There is no high-order gradient computations in our fine-tuning.\n\n\n## **8. About the fine-tuning efforts**\n\nTo build the training dataset for fine-tuning, we collect about 80,000 synthesized images from huggingface and recaption them with Qwen-VL. We perform fine-tuning on 8 NVIDIA GPUs (80GB memory each) with a global batch size of 64, while leveraging DeepSpeed ZeRO-3 or FSDP with gradient checkpointing to save GPU memory at the cost of increased training time. All model parameters are used for fine-tuning. The fine-tuning requires 6 epochs, which costs about $14 \\times 8$ A100 hours for Lumina-mGPT and $26 \\times 8$ H100 hours for Emu3.\n\n## **9. About the refinement with denoising** \n\nThis refinement with denoising is the key distinction between SJD2 and SJD. As shown in Table 1, SJD2 achieves greater acceleration than SJD."}, {"review_id": "44MPxSDtAn", "reviewer": "Reviewer_3CPQ", "summary": "This paper proposes Speculative Jacobi-Denoising Decoding to enable the control of refinement in SJD. A novel and sophisticated fine-tuning strategy by next-clean-token prediction is designed to predict the clean token at the next position with noised tokens as input. Then a part of perfix tokens would be selected after verification and the remaining tokens would be refined. The experiments show that the proposed method largely accelerates the inference speed and maintains the generation quality.", "questions": "1. Is the proposed SJD2 able to perform batch inference?\n2. What's the specific choice of $\\alpha_t$ and $\\sigma_t$ in Equation (3)?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n1. The proposed SJD2 is novel and shows impressive acceleration and maintains the generation quality.\n2. The proposed fine-tuning strategy enables low-cost fine-tuning for the consecutive acceleration method.\n\nWeakness:\n\n1. The method's description is dense and requires prior familiarity with Jacobi decoding, speculative decoding, and diffusion models. This may hinder readability on readers who are not familiar with some of these topics.\n2. While the denoising process is justified through analogy to diffusion models, the behavior of denoising in discrete token space is not fully analyzed. More investigations on the claim that diffusion models improve the refinement, like showing how the output tokens converge, can be helpful.", "was_revised": true, "final_justification": "The author's clever rebuttal addressed my main concerns, and I tend to remain positive rating.", "rebuttal": "# Response to Reviewer 3CPQ\n\nThank you for your thoughtful and constructive feedback on our paper. We are particularly grateful for your recognition of our novelty and high performance.\n\n## 1. **Enhanced prelimiary on speculative decoding and Jacobi decoding**\nThank you for your suggestion. To improve clarity and readability, we improve preliminary section and add more knowledge details related to our core design. Specifically, we refine the descriptions of speculative decoding and Jacobi decoding: **(A) speculative decoding**: Speculative decoding employs a small model to accelerate the sequential generation of large autoregressive model. This model is trained on the same domain as the large model and is small enough for faster generation. In each step of inference, the small model first generates a sequence with its own inference paradigm. The large model then verifies this sequence in a single forward pass, selecting a prefix to serve as part of the final output. This verification ensures that each sampled token adheres to the probability distribution parameterized by the large model. Since multiple tokens can be taken as the final output by only one forward pass of the large model, the acceleration can be achieved; **(B) Jacobi decoding**: Jacobi decoding deems the auto-regressive inference as a process of solving the fixed point of a nonlinear equation in a triangular system. This decoding algorithm iteratively performs multi-token decoding and can be executed without fine-tuning or auxiliary modules. The specific process of Jacobi decoding is as follows: First, given the previously pre-filled or decoded tokens, we randomly initialize a sequence of candidate tokens. Then, in each iteration, we execute one forward pass of the auto-regressive model for all the candidate tokens with a causal mask. the predicted probabilities then generate the tokens typically via greedy sampling, and these sampled tokens are taken as the inputs of the next iteration. This process can be formulated as: $x\\_{i}^{(j+1)} = \\arg \\max\\_x  p\\_\\theta(x|\\mathbf{x}\\_{1:i-1}^{(j)})$, where $i$ denotes the token index and $j$ denotes the iteration index. The Jacobi decoding process continues iterating until the convergence is reached, as determined by a deterministic criterion where these tokens remain unchanged between consecutive iterations.\n\n\n\n## 2. **Investigations on the refinement**\n\nThank you for your suggestion. Since we cannot include links in this rebuttal, we use a pair of tables to show the change of token categories at each sampling step and token position, so each table includes an example of the token trajectory. In SJD2 (shown by each column of Table A), over 25 sampling steps, the predicted tokens start identical (e.g., token value 167282). As the noise level decreases, the token categories become diverse, but some token categories begin to repeat, which means the trajectory becomes stabilize. In contrast, for SJD (the columns in Table B), the token categories change irregularly, appearing to oscillate and remain unstable. We will provide more detailed visualizations using figures in the camera-ready version.\n\n**Table A: The discrete token trajectory in SJD2 with Emu3 as baseline:**\n\n| Sampling Steps \\   Token index | Token 1 | Token 2 | Token 3 | Token 4 | Token 5 |\n|---------------------------------|---------|---------|---------|---------|---------|\n| Step 1                          | 167282  | 167282  | 167282  | 167282  | 167282  |\n| Step 2                          | 167282  | 167282  | 167282  | 167282  | 167282  |\n| Step 3                          | 167282  | 167282  | 167282  | 167282  | 167282  |\n| Step 4                          | 167282  | 167282  | 167282  | 167282  | 167282  |\n| Step 5                          | 167728  | 167728  | 167728  | 167728  | 167728  |\n| Step 6                          | 167728  | 159459  | 159459  | 167728  | 159459  |\n| Step 7                          | 167282  | 167282  | 167282  | 167282  | 167282  |\n| Step 8                          | 167282  | 167282  | 163124  | 167282  | 152950  |\n| Step 9                          | 154915  | 153542  | 163124  | 156927  | 160877  |\n| Step 10                         | 155499  | 158046  | 158230  | 153542  | 152491  |\n| Step 11                         | 154578  | 157671  | 152153  | 153840  | 153399  |\n| Step 12                         | 154794  | 155092  | 152611  | 154282  | 153088  |\n| Step 13                         | 153692  | 153700  | 158037  | 156095  | 154951  |\n| Step 14                         | 152784  | 160040  | 156856  | 152551  | 155736  |\n| Step 15                         | **152388**  | 153700  | **155986**  | 152395  | 151927  |\n| Step 16                         | 158046  | 160040  | 165298  | 159796  | 151980  |\n| Step 17                         | **152388**  | *152357*  | 153737  | 155415  | *153182*  |\n| Step 18                         | 158046  | 160040  | **155986**  | **152006**  | *153182*  |\n| Step 19                         | **152388**  | *152357*  | **155986**  | **152006**  | **152268**  |\n| Step 20                         | 154500  | **155680**  | 153737  | 155493  | **152268**  |\n| Step 21                         | **152388**  | *152357*  | **155986**  | 154177  | **152268**  |\n| Step 22                         | 158046  | **155680**  | **155986**  | **152006**  | **152268**  |\n| Step 23                         | **152388**  | *152357*  | **155986**  | **152006**  | **152268**  |\n| Step 24                         | **152388**  | **155680**  | 153216  | **152006**  | 152519  |\n| Step 25                         | **152388**  | **155680**  | 153951  | 157603  | **152268**  |\n\n\n**Table B: The discrete token trajectory in SJD with Emu3 as baseline:**\n\n| Sampling Steps \\   Token index | Token 1 | Token 2 | Token 3 | Token 4 | Token 5 |\n|---------------------------------|---------|---------|---------|---------|---------|\n| Step 1                          | 178678  | 154122  | 180405  | 156118  | 153692  |\n| Step 2                          | 165560  | 164685  | 162805  | 167164  | 155025  |\n| Step 3                          | 166380  | 154949  | 164287  | 166096  | 157596  |\n| Step 4                          | 156660  | 159530  | 152575  | 152503  | 166994  |\n| Step 5                          | 158407  | 151855  | 153356  | 157268  | 165140  |\n| Step 6                          | 160136  | 152594  | 160547  | 153105  | 152290  |\n| Step 7                          | 162159  | 156850  | 156404  | 159634  | 155860  |\n| Step 8                          | 165406  | 154488  | 158930  | 159965  | 153749  |\n| Step 9                          | 164959  | 151927  | 160406  | 165482  | 154677  |\n| Step 10                         | 154998  | 155867  | 166882  | 159530  | 166918  |\n| Step 11                         | 155948  | 159306  | 153403  | 156096  | 162431  |\n| Step 12                         | 160407  | 152980  | 155008  | 157592  | 152012  |\n| Step 13                         | 160393  | 156175  | 153776  | 156225  | 165458  |\n| Step 14                         | 155559  | 157817  | 164655  | 166620  | 159092  |\n| Step 15                         | 161548  | 153057  | 164198  | 155867  | 167001  |\n| Step 16                         | 152190  | 155639  | 164741  | 165587  | 151964  |\n| Step 17                         | 155733  | 163798  | 153648  | 163956  | 160777  |\n| Step 18                         | 152029  | 156397  | 154102  | 153091  | 155513  |\n| Step 19                         | 165985  | 152276  | 154563  | 159234  | 153776  |\n| Step 20                         | 161888  | 154110  | 167323  | 155040  | 154248  |\n| Step 21                         | 160647  | 152453  | 157631  | 153105  | 157318  |\n| Step 22                         | 161724  | 154949  | 159608  | **154419**  | 155366  |\n| Step 23                         | 155774  | 152065  | 164738  | 157863  | 157972  |\n| Step 24                         | 152548  | 153464  | 155459  | **154419**  | 156705  |\n| Step 25                         | 164009  | 158589  | 160031  | 153100  | 155766  |\n\n## 3. **About the possibility of batch inference** \n\nIn principle, our SJD2 can inherently support batch inference as a variant of speculative decoding [A]. However, our current implementation has not yet been optimized for batch processing due to engineering challenges such as synchronization overhead. We are actively exploring engineering optimizations, including dynamic batching and asynchronous verification, to enable efficient batch inference in future versions.\n\n[A] Qian H, Gonugondla S K, Ha S, et al. BASS: Batched attention-optimized speculative sampling[J]. arXiv preprint arXiv:2404.15778, 2024.\n\n## 4. **About details of Equation 3**\nAs shown in Appendix A, we set $\\alpha_t=1-t$ and $\\sigma_t=t$ for Equation 3."}]

Author rebuttal:

Meta-review: This paper introduces Speculative Jacobi-Denoising Decoding (SJD2), a novel acceleration framework for autoregressive text-to-image generation. The core scientific claim is that by integrating denoising into Jacobi iterations and introducing a next-clean-token prediction task, SJD2 enables parallel token generation, achieving up to 4x inference speedup while largely preserving visual quality. The paper demonstrates this on multiple recent generative models (Lumina-mGPT, Emu3, Janus-Pro) with both quantitative and qualitative results, including user preference studies. The strengths of the work lie in its clear motivation, methodological novelty, careful experimental validation, and practical acceleration gains. The method is also relatively lightweight, requiring only modest fine-tuning compared to full pretraining.

The weaknesses identified by reviewers include incrementality over prior speculative Jacobi decoding, limited baseline comparisons (initially only against SJD), and questions about generalizability to other AR paradigms (e.g., VAR, MAR). During rebuttal, the authors responded constructively: they added experiments on Janus-Pro, compared against modern speculative/parallel methods (Lantern, ZipAR, Eagle), provided detailed fine-tuning and FLOP/latency analysis, clarified noise segmentation and timestep injection strategies, and conducted user preference A/B testing. The rebuttal substantially mitigated the main criticisms, with reviewers acknowledging improved clarity and breadth of evaluation. Considering the strong technical contribution, convincing empirical validation, and convincing rebuttal, the AC recommends acceptance. The paper makes a solid and timely contribution to the acceleration of autoregressive generative models.
Decision: Accept (poster)