Title: Speculative Jacobi-Denoising Decoding for Accelerating Autoregressive Text-to-image Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Acceleration

Auto-regressive Model

Text-to-image Generation
Abstract: As a new paradigm of visual content generation, autoregressive text-to-image models suffer from slow inference due to their sequential token-by-token decoding process, often requiring thousands of model forward passes to generate a single image. To address this inefficiency, we propose Speculative Jacobi-Denoising Decoding (SJD2), a framework that incorporates the denoising process into Jacobi iterations to enable parallel token generation in autoregressive models. Our method introduces a next-clean-token prediction paradigm that enables the pre-trained autoregressive models to accept noise-perturbed token embeddings and predict the next clean tokens through low-cost fine-tuning. This denoising paradigm guides the model towards more stable Jacobi trajectories. During inference, our method initializes token sequences with Gaussian noise and performs iterative next-clean-token-prediction in the embedding space. We employ a probabilistic criterion to verify and accept multiple tokens in parallel, and refine the unaccepted tokens for the next iteration with the denoising trajectory. Experiments show that our method can accelerate generation by reducing model forward passes while maintaining the visual quality of generated images.


Peer reviews:
comments: SUMMARY:
This paper proposes Speculative Jacobi-Denoising Decoding to enable the control of refinement in SJD. A novel and sophisticated fine-tuning strategy by next-clean-token prediction is designed to predict the clean token at the next position with noised tokens as input. Then a part of perfix tokens would be selected after verification and the remaining tokens would be refined. The experiments show that the proposed method largely accelerates the inference speed and maintains the generation quality.

STRENGTHS AND WEAKNESSES:
Strength:

1. The proposed SJD2 is novel and shows impressive acceleration and maintains the generation quality.
2. The proposed fine-tuning strategy enables low-cost fine-tuning for the consecutive acceleration method.

Weakness:

1. The method's description is dense and requires prior familiarity with Jacobi decoding, speculative decoding, and diffusion models. This may hinder readability on readers who are not familiar with some of these topics.
2. While the denoising process is justified through analogy to diffusion models, the behavior of denoising in discrete token space is not fully analyzed. More investigations on the claim that diffusion models improve the refinement, like showing how the output tokens converge, can be helpful.

QUESTIONS:
1. Is the proposed SJD2 able to perform batch inference?
2. What's the specific choice of $\alpha_t$ and $\sigma_t$ in Equation (3)?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed Speculative Jacobi-Denoising Decoding (SJD2) to accelerate autoregressive text-to-image model. By incorporating diffusion denoising with speculative decoding for AR model, the paper reaches 4x speed up on common AR image generation model.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The background of SJD2 is well introduced in the first section.\n2. SJD2 gets good acceleration with almost no loss on image quality.\n3. The method works on both text-to-image and text-to-video models.\n\n**Weaknesses**\n1. Based as pervious work SJD, this work is only a minimal piece of contribution, besides, as stated in related work, the integration of diffusion has been studied in many visual generation works.\n2. The method section is not well organized, needs more details, like the finetuning objective, the timestep injection and so on.\n3. The difference between SJD2 and AR+diffusion methods are not introduced in related works.\n4. The method comparison is not effective, only SJD are compared.\n\nQUESTIONS:\n1. The index of embedding $ e $ in Eq.(2) is confused, please explain why the superscript skips from 0 to $ t_0 $, and what is $ i $ and $ i' $ in subscript.  \n2. Does the original AR model utilize normalized embedding as input? \n3. Please explain the training objective and process, I only see a cross-entropy loss for AR model, how did this work for the diffusion? If there is a multi-step denoising process, would this involve a high-order gradient?\n4. What is the finetuning effort of SJD2, like the dataset size and training time.\n5. Please compare with more parallel decoding methods, like Jacobi Decoding, Lookahead Decoding, CLLMs, and ZipAR.\n6. How effective is the Refinement with Denoising? Is the difference between SJD and SJD2 this refinement? If no, there is a lack of relevant ablation experiments.\n\nLIMITATIONS:\nSee Weaknesses and Questions", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Speculative Jacobi-Denoising Decoding (SJD2), a method for enabling parallel token generation in autoregressive models. The authors propose a next-clean-token-prediction task that allows a model to predict multiple future tokens with only light fine-tuning. Experiments on MSCOCO and GenEval show that this approach can achieve a speedup of around 4x without a significant drop in performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- The paper is well-written, and the ideas and methods are described clearly. The main figure illustrates the core idea very well.\n\n- The evaluation on MSCOCO shows that the method can speed up models like Lumina-mGPT and Emu3 by about 4x without harming performance too much.\n\n- The qualitative results look good and effectively support the main experimental claims.\n\nWeaknesses\n\n- Limited Scope of Model Compatibility: It is unclear if the proposed SJD2 method works with a wider range of autoregressive models. For example, compatibility with other advanced autoregressive approaches like VAR or Janus Pro is not discussed.\n\n- Insufficient Baseline Comparisons: The main experiments lack comparisons with other modern speculative or parallel decoding methods. The only baseline compared against is the original SJD, which makes it hard to judge SJD2's performance relative to the state of the art.\n\n- Missing Comparison with Diffusion Models: The paper does not include a comparison of the efficiency/performance trade-off against state-of-the-art diffusion models.\n\n- Weak Performance on GenEval: The overall performance on the GenEval benchmark does not appear to be competitive with many existing methods.\n\nQUESTIONS:\n- Why was there no comparison with the Emu3 model on the GenEval benchmark, even though it was used in other experiments?\n\n- Are there any comparisons of computational cost in terms of FLOPs? This would provide a more detailed efficiency analysis beyond latency.\n\n- How does the proposed SJD2 approach compare to other parallel decoding methods from recent literature? A broader comparison is needed to position this work.\n\n- Is SJD2 compatible with autoregressive models that already use a form of parallel decoding, such as MAR models?\n\n- Could you provide an analysis or comparison of the training cost required for the proposed SJD2 approach?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSpeculative Jacobi-Denoising Decoding is the next step in proposing novel improvements to Speculative Jacobi Decoding for parallel token generation in autoregressive models. These changes result in faster convergence, reducing computational overhead, as well as equivalent or higher quality on several metrics demonstrated on image generation tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nDetailed ablations and metric studies, as well as a clear description of the approach. I think this is a good, informative paper which is useful for people to get introduced to Jacobi decoding methods, as well as see the line of research leading up to this new method. The experimental section is focused on images, but clear in the results and goals. Descriptions of the inference time procedure are high quality, and presented with great figures and examples.\n\nWeaknesses:\nGiven the widespread applicability of Jacobi decoding and descendants, it would be really interesting to see this approach applied to other generative applications, like text or audio token generation. It may be out of scope for this review cycle, but really worth considering for broad appeal of the method.\n\nIt is not clear to me if the full model is trained during the finetuning procedure, or only a small part. There are some other complaints below, but a bit more indepth treatment of the tuning stage, and what parameters are involved would be useful especially for followup work.\n\nThere is a strong conceptual convergence between the approach detailed here, and the paper \"Rolling Diffusion Models\", which adds a rolling time-window approach into continuous diffusion. It it is probably worth referencing or briefly discussing in the background material.\n\nThere was some mention of compute costs, but it would be useful to detail how long the finetuning takes in terms of approximate wall-clock time, as well as further details on the finetune hyperparameters in general.\n\nThere is a statement \"For fine-tuning, we divide the input sequence into randomly sized segments and add noise to each segment.\", it would be useful to have details about the procedure to select the randomly sized adjacent segments, and how the noise level is chosen and applied over training. There is \"We follow the advanced flow matching setting with αt = 1 − t and σt = t for our denoising process\", but more details would be better.\n\nSimilarly, there is a statement \"we take the normalized Fourier encodings of timesteps\" - is this different than a standard positional encoding? If so, it would be good to add more detail, or cite a relevant paper defining this. Otherwise, change the writing slightly to tie in to the more standard naming with \"Fourier positional encodings\" or \"Transformer sinusoidal position encodings\", or something like this.\n\nQUESTIONS:\nDo the authors have any user preference A/B tests to complement the GenEval result? It would be great to correlate these quantitative metrics to human A/B.\n\nIn eq(1), if the 1 in the lefthand side is decreased, what happens to the resulting quality? Do the authors have any study, reference, or ablation on this - I think it might also improve the convergence speed of Speculative Jacobi Decoding, but maybe at the expense of quality?\n\nDo the authors have any high-order statistics, like skewness, kurtosis, and so on about the distribution of embeddings before and after normalization? It is slightly surprising to me that simple mean standard deviation normalization is enough to Gaussianize sufficiently to prevent the failure modes discussed in the paper. It would be good to have some information, if people want to apply this method but have different embedding distributions up front.\n\nIs there any potential of a \"training free\" version of this method, if not what are the key roadblocks (technical, computational, theoretical) to such a version?\n\nWill the authors release code to help reproduction in open source tools?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}