Accelerating Diffusion LLMs via Adaptive Parallel Decoding
Authors: Daniel Mingyi Israel
Guy Van den Broeck
Aditya Grover
Venue: NeurIPS 2025 spotlight ()
Keywords: LLM
discrete diffusion
autoregression
sequential
fast
throughput
speculative decoding
Primary area: deep_learning
TL;DR: Substantially faster diffusion LLMs using a small auxiliary autoregressive model

Abstract: The generation speed of LLMs are bottlenecked by autoregressive decoding, where tokens are predicted sequentially one by one. Alternatively, diffusion large language models (dLLMs) theoretically allow for parallel token generation, but in practice struggle to achieve the speed of autoregressive models without significantly sacrificing quality. We therefore introduce adaptive parallel decoding (APD), a novel method that dynamically adjusts the number of tokens sampled in parallel. We achieve this by defining a multiplicative mixture between the dLLM marginal probabilities and the joint probability of sequences under a small auxiliary autoregressive model. This inverts the standard setup of speculative decoding, where the goal is to sample from a large autoregressive verifier by drafting from a smaller model. We further optimize APD by enabling KV caching and limiting the size of the masked input. Altogether, our method puts forward three tunable parameters to flexibly tradeoff throughput and quality. We show that APD provides markedly higher throughput with minimal quality degradations on downstream benchmarks.

Accelerating Diffusion LLMs via Adaptive Parallel Decoding
Daniel Israel
Guy Van den Broeck
Department of Computer Science University of California, Los Angeles disrael@cs. ucla.edu
Department of Computer Science University of California, Los Angeles guyvdbêcs ucla.edu
Aditya Grover
Department of Computer Science University of California, Los Angeles adityag0cs ucla.edi
Abstract
The generation speed current LLMs is bottlenecked by autoregressive decoding. where tokens are predicted sequentially one by one. Alternatively, diffusion large language models (dI.LMs) theoretically allow for parallel token generation, but practice struggle to achieve the speed of autoregressive models without significantly sacrificing quality. We therefore introduce adaptive parallel decoding (APD), a novel method that dynamically adjusts the number of tokens sampled in parallel. We achieve this by defining a multiplicative mixture between the dLLM marginal probabilities and the joint probability of sequences under small auxiliary autore- gressive model. This inverts the standard setup of speculative decoding, where the goal is to sample from large autoregressive verifier by drafting from smaller model. We further optimize APD by enabling KV caching and limiting the size the masked input. Altogether, our method puts forward three tunable parameters to flexibly tradeoff throughput and quality. We show that APD provides markedly higher throughput with minimal quality degradations downstream benchmarks.
Introduction
Large language models (LLMs) have remarkable text generation capabilities and have attracted evergrowing interest and widespread adoption. However, a significant impediment to their deployment lies in the speed of text generation 43). The dominant paradigm, autoregressive models [32], generates tokens one by one a sequential manner. While this approach has yielded state-of-the-art results in terms of quality, the inherent sequentiality of generation limits throughput and hinders real-time applications, especially as models continue to scale in size [19. addition, the recent phenomenon of test-time scaling [35] and reasoning models [13] suggest that generation speed will be an important bottleneck for increasing LLM capabilities. This challenge has spurred research into
alternative approaches beyond autoregressive sequential sampling.
One promising alternative to purely sequential generation is offered by diffusion large language models (dLLMs) [42). These models, inspired by successes in image generation (36), theoretically permit the parallel generation of multiple tokens simultaneously, offering path towards significantly faster inference. In this work, challenge the assumption that dLLMs, exemplified by the open- source models Dream [41] and Llada (2),, can be practically used for parallel generation without additional modifications. We find that the quality of these models is best achieved by generating tokens one at a time (one timestep per token), and attempts to exploit the parallelizability of diffusion
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Prompt
Charleston. Charleston has 4 times many sheep Seattle. How many sheep do Toulouse, Charleston, and Seattle have together if Seattle has sheep?
Toulouse twice
Autoregressive Decoding
Let's by defining number of sheep each city has
variables and then use given information to find the exact numbers,
Let V $ () represent the number of sheop in Seattle.
According the problem, Seattle has 20 sheep, so V
(). Charleston has 4 times as many sheep as Seatcle. So, let V( C ( represent the number of sheep in Charleston:
M C - 4S 4 tiimes 20
80
V]
Toulouse has
many sheep as Charleston. \( T represent the sheop Toulouse:
M T - 2C =22 1ttmmss 80 - 160
To find the total number sheap Taulouse, Charleston, and Seattle together, add number of sheap each city:
M T C-S 160 + + 20 260 Al
Therefore, the total number of sheep in Toulouse, Charleston, and Seattle together 260.
Latency: 7.08 seconds
Throughput: 37 tokens per second
Adaptive Parallel Decoding
Let's break down the information:
Seattle has 20 sheep.
Charleston has times many sheep Seattle, Charleston has 20 sheep.
Toulouse has twice as many sheep as Charleston, so Toulouse has 2 80 = 160 sheep.
Now, let's add up number sheep each city has:
Seattle: 20 shoep
Charleston: sheep
Toulouse: 160 sheep
Total number of sheep: 20 an + 160 100 + 160 260
So, Toulouse, Charleston, Seattle total (260) sheep together
Latency: 2.75 seconds
Throughput: 59 tokens per second
Figure 1: Autoregressive vs Adaptive Parallel Decoding (APD). We visualize a sample from Qwen2.5 7B decoded autoregressively (baseline) and Dream 7B with APD (our method). Contiguous text of the same color indicates tokens sampled in parallel. Here, we show that APD generates the solution much faster than autoregressive sampling, which is sequential and slow.
models suffer reduction in quality. Thus, current state-of-the-art dLLMs, namely Dream and Llada as currently conceived, fail to match the speed and quality of autoregressive LLMs.
This gap between theoretical and practical performance creates an opportunity for novel decoding mechanisms that can effectively harness the parallel generation capabilities of dLLMs while main- taining high fidelity to the target text distribution. The core challenge is that when sampling multiple tokens in parallel, one only has access to the marginal distribution of each token, which ignores inter-token dependencies (23]. A decoding algorithm that increases parallelism when sampling from dLLM and maintains generation quality must consider the joint distribution that captures these dependencies.
To address this challenge, we introduce Adaptive Parallel Decoding (APD), a novel decoding algorithm designed to dynamically modulate the number of tokens sampled in parallel during the generation process with dLLMs. In APD, we first fix the generation order of the dLLM to be left to right. This unexpected change makes the dLLM autoregressive, but we empirically show that it maintains generation quality and in some cases can improve it. By using the dLLMM autoregressively: we can then also use a smaller autoregressive model to determine which subset of tokens sampled in parallel adequately captures joint dependence. Our criterion depends on a multiplicative mixture between the diffusion model and autoregressive model. Note that while speculative decoding uses a large model to verify samples from a small draft model, our task is to use a smaller verification model to check the quality of a larger model. This dynamic leads to a fundamentally new problem statement.
Our contributions extend beyond the conceptual framework of APD. We further optimize the decoding process by incorporating practical improvements to diffusion model sampling such as Key- Value (KV) caching. a technique traditionally associated with autoregressive models [31]. We also gain significant speed by limiting the size of the masked input to the model. These simple changes substantially increase the computational efficiency of diffusion model inference.

---

Collectively, we introduces three distinct tunable parameters. These parameters provide practitioners with the flexibility navigate the inherent trade-off between generation throughput and output quality, allowing for tailored configurations that can meet diverse application-spect This paper will demonstrate that Adaptive Parallel Decoding offers a significant step forward in accelerating dLLMM inference (and L.M inference in general). We present empirical evidence showing that APD achieves substantially higher throughput compared to existing LLM decoding strategies, all while incurring only minimal degradations in quality across range of downstream benchmark tasks. The subsequent sections will discuss the technical details of APD, present a comprehensive set of experiments validating claims, and discuss the broader implications of our findings for the future of efficient LLM generation.
requirements.
2 Background
In the following sections, we will define notation to be used throughout the paper. We consider data point as a sequences
tokens x = (Z1,.., Tn). "T,TT) ). For sets of indices 2, O S {I..., n for which Q n O 0, masked language model pn( :: computes the marginal probabilities of tokens with query indices Q conditioned on tokens with observed indices O.
po(#o so; 0)
Il (a(a; wo) E
where pe is learned conditional distribution parameterized by 0. this work, a diffusion language model is a masked language model, and these terms can be used interchangeably.
2.1 Discrete Diffusion Language Models
Discrete diffusion models [26] have recently emerged as promising alternative to traditional autoregressive approaches for language generation, offering benefits such as non-autoregressive generation capabilities and inherent iterative refinement (39]. Discrete diffusion models are masked language models [34] trained to reverse data corruption process that stochastically converts a clean sequence of tokens z0 to noisy x', gradually converting clean tokens to [MASK] over time t
4. 1-4,
7 [MASK] if of x
2ej(aa x0))
II
(e x
(2)
otherwise
Given this noise process, dLLMMs are trained to maximize a lower bound on the log-likelihood computed using conditional distributions of clean data (30
1
log po po()) E,,
[lo (F ((AAS)))))))) 0)]
While dLLMss are trained in theory to randomly unmask tokens inference time, in practice decoding heuristics are used to determine the order of tokens to unmask (20). For example, to achieve the best results, Dream unmasks tokens according to lowest entropy as assessed by the diffusion model at each timestep. Llada performs the best when unmasking according to the highest probability, which
referred to as "confidence" based decoding (29]. In light of these inference time modifications, diffusion language models are able to compete with autoregressive model in terms of generation quality. However, in realistic scenarios, the quality of a generation must be weighed against the speed. When we consider both of these factors, current open-source dLLMs are far from competitive with their open-source autoregressive counterparts. In Table I we find that dLLMMs can perhaps achieve competitive quality, but their throughput is fraction of Qwen2.5 7B [40] Another observation that in order to achieve competitive performance on GSM8K (6) dLLMS must generate 256 tokens in 256 timesteps, which is actually sequential. We see that decreasing the number of timesteps can increase throughput at the expense of quality. Nonetheless, the throughput will not approach autoregressive speed without drastic reduction in quality. Interestingly, we find the simplest decoding order, just left to right, yields good results in most scenarios, and in the case of GSM8K the best results.
3

---

Dream 7B
Liada 8B
Throughpus (tokens/sec)
1.0
100
GSMBK Accuracy
0.0
D22
GSMBK Accuracy
0.0
10
0.6
Accuracy Thrughoir
Accurccy Thirushpue
Throughput
0.
0.0
K (# Tokens Sampled in Parallel per Iteration)
K (# Tokens Sampled in Parallel per Iteration)
Figure 2: Naive Parallel Generation. Fixing the number tokens to semi--uuorrerresivell generate (left to right) in parallel per iteration, we observe clear tradeoff between parallelization and generation quality. This tradeoff exists with open source diffusion models: Dream and Llada.
2.2 From Sequential to Parallel Sampling
In this work, we utilize diffusion models with a particular noise schedule: denoising left to right. This is equivalent to sampling autoregressively and can be used to compute an exact autoregressive likelihood.
PAR (x;
II po(as x<
(4)
To further parallelize this process, one can decode multiple tokens at a time semi-autoregrressiely 38 from groups of size k. Decoding in this manner samples from the following distribution
L(n 1)/kJ+1
PSAR (v; 0, k))
II
PD(#((A-R+144 T=((h-++1);0)
(5)
where it follows that PAR (a; 0) pSaR (2;; 0, 1) for 1. A diffusion model can parallelize the inside the product by sampling from each marginal distribution independently. Although this will lead faster generations, generations. We observe this empirically in Figure 2 as we increase the number of tokens sampled time, throughput increases, but downstream accuracy decreases. Thus, there exists tradeoff between parallelization and speed with sample quality. However, instead of fixing k. our core insight that the number of tokens to generate parallel can be chosen adaptively. In this work, we argue that doing so significantly reduces the sharp tradeoff between speed and quality.
intra-group independence assumption poor and leads to lower quality
3 Method
3.1 Problem Statement
Our goal to sample from groups g = {($1, e1), (82, (8), (.....,,,) (8t, Et) where (st, e.) is a tuple that denotes start and end indices inclusive and e; 8141 for all Note that Equation 5 a special
Table dLLM Quality and Throughput with Different Decoding Approaches
Model
GMS8K Accuracy
Throughput (tokens/sec)
Dream 7B (Random, 256 Steps)
0.404 0.021
3.31 + 0.068
Dream 7B (Entropy, 128 Steps)
0.708 0.020
7.57 + 0.157
Dream 7B (Entropy, 256 Steps)
0.804 = 0.017
4.28 + 0.080
Dream 7B (Left to Right, 256 Steps)
0.832 0.016
10.1 0.015
Llada 8B (Random, 256 Steps)
0.456 = 0.022
5.07 + 0.168
Llada 8B (Confidence, 128 Steps)
0.526 = 0.022
13.6 0.284
Llada 8B (Confidence, 256 Steps)
0.534 = 0.022
6.63 = 0.143
Llada 8B (Left to Right, 256 Steps)
0.712 = 0.020
9.33 + 0.016
Qwen2.5 7B (Autoregressive)
0.854 0.015
38.6 + 0.004

---

case where g = ={1,,), (k + 1, + .)..... (n..,nn-kk) k, n)}
We shall sample from
(on(x;0,g)- - II
(e xiss)
(6)
(a.6)EG
In choosing G. we must balance the following two goals (1) Speed: minimize | and (2) Quality: minimize the distance between pAPD and PAR.
Achieving only one out of two goals is trivial. For maximum speed, we can sample from all the marginals of the diffusion model in one shot such that & = {(1, n)} and |g] This would result in significant drop in quality. In fact, the drop in quality can be quantified by mutual information
PAR (x; 0) = (L(AAR((Imm)
PD( 0))
Alternatively, we set G {(1,1)...(n, n)]. in which case [9] n and PAPD PAR- This is very slow because maximizes the number of sequential iterations. For architectural reasons to be discussed in Section 3.3 it is also very slow to sample autoregressively from diffusion model because it cannot perform KV caching. This architectural difference distinguishes our problem setting from speculative decoding [22), because computing an autoregressive likelihood in diffusion model is slow and sequential, whereas in an autoregressive model it is fast and parallelizable. While speculative decoding does not map onto our problem, shall argue that adaptive parallel decoding achieves both goals together.
3.2 Adaptive Parallel Decoding
In adaptive parallel decoding, assume to have access small autoregressive model pAr that can compute the likelihood of sequences in parallel. While pD only computes marginal probabilities over tokens, PAR computes joint probability, allowing it to model dependencies between tokens. For brevity, we shall omit model parameters 0.
To select G. we shall focus our analysis
subproblem. Given samples å,, .... S. pn( xege), we must select & such that fuz+ are close to PAR. We do not have access to PAR because evaluating its likelihood for each token will give us the worst-case speed. Instead, we have access to small, inaccurate model PAR that can quickly evaluate likelihood in parallel. To approximate PAR. we shall to determine how many tokens k are accepted. This target
define target distribution which distribution pr should satisfy the following desiderata:
(1) if px (X c) 1 and PAR X c) < 1, then pr (X =c)=1 =1
(2) if pD = c) < and PAR X c) = 1, then pr (X =
Property (1) follows from the fact that PAR PAR( (xI1k) 2 Ei= pD( k by Bonferroni's inequality [10], which means if pD;x;) = 1 for all i, then PAR (T1:k) = 1. Put verbally, if the marginals given by the diffusion model are 1, its joint distribution
Par will also have probability 1. Property (2) is difficult to justify formally because we do not have a relationship between PAR and PAR- Assuming PAR reasonably approximates the joint distribution, property (2) is a desirable heuristic because token with joint probability should be accepted.
multiplicative mixture of distributions, also known product of experts [6). fulfills the two requirements above. We define the multiplicative mixture of pD and yAR as follows
1 Pr (x) ( > (AR(()''
(8)
where Z is the normalizing constant. Our mixture is defined in terms of a hyperparameter R. E [0,1]. When is high, it gives the diffusion model more weight. We may now give an accept criteria based on target distribution pr. We adopt accept criteria similar to existing parallel sampling methods based on universal coupling [.. Formally, a universal coupler is a function g which for a distribution p and source of randomness satisfies
Pr-Ujo. (e(p.))))))) r) = x) -(xx)
For categorical distributions, the Gunbee--Softax Trick [8] a universal coupler D. important
property of universal coupler is that for two distributions p. p', their samples from universal
5

---

Algorithm Adaptive Parallel Decoding
1: Input: Diffusion model pn. Autoregressive model PAR, Mixture Weight Parameter R, Maximum
sequence length
2: Output: Generated token sequence
3: I x+(
Stores the accepted tokens > Index of token to generate
4: t+1
5: while 1 do
6:
Serginal_logitan < PD( x<())
7:
r (-Gubell(0,1)
8:
iu.n samplel_gmbelllarginaaaaaa
9:
joint_logits,,, + PAR (ft. r())
10:
product_logite, + softmax (oaaaaaaannaaaagaaaaaaooii
11:
Dun + gwwbellproduutt_eaatttpp
12:
k sum (um(cupprod(<=11 )t+1m)) +
13:
I concat((z, (xt4kk1))
Append accepted tokens
14: t
15:
end while
16: return
coupler with shared source of randomness are likely to the same given that the distributions and p' are similar (21].
Pr(o(p,r) & ,(p'rr) s TV(p,p')
(10)
where TV is total variation distance. Thus, in our algorithm sample from the diffusion model using Gumbel-Softmax trick as a universal coupler y and source of randomness r: ft,..., &n g(PD: r). We then sample from our target: Vt..... Un ~ o(Pr.r), and we accept all tokens that are the same between dJ and & until the first disagreement. Because the goal is to sample from PAR. we always accept the first proposed token f,. The full algorithm is given in Algorithm [ While the same algorithm is valid even when using different Gumbel randomness to sample from the target and proposal distributions, intuitively using the same randomness will maximize the number of accepted tokens.
Algorithm | provides a sampling procedure which takes input diffusion model pD. small autoregressive model PAR, and tunable parameter R E [0, 1]. When R 1, the target distribution yn and the algorithm will accept every token from the diffusion model in one shot. When It = 0, the algorithm does not "trust" the diffusion model and instead only accepts tokens that PAR accepts.
Our sampling algorithm is agnostic to the implementation of PD and PAR, but in practice architectural optimizations are used for additional speed. For example. when computing the joint logits of the autoregressive model, we can use KV caching [3] to avoid redundant computation on & previous tokens. The only requirement for PD is that can draw samples from the marginal distributions of the suffix in parallel. However, if we consider that the implementation of is Transformer-bas masked language model, further speedups can be achieved.
3.3 Recompute KV Window
In above method, center sampling parallel as the primary way to improve decoding speed. However, overall we aim to maximize the throughput of diffusion models by all means necessary, also must consider architectural factors. We enable KV caching for tokens outside sliding window of size W. For a detailed explanation of KV caching, see Appendix B Although KV caching with diffusion model trained with bidirectional masking can induce arbitrary out-of-distribution behavior, empirically we observe very little performance degradation (Appendix BB. The intuitive reason is that tokens sufficiently far from the rightmost token will have small attention weight, so inaccuracies in their KV will not dramatically change the overall attention computation.
3.4 Maximum Masked Lookahead
Because we are sampling from a diffusion model left to right autoregressively, we can exploit the fact that the input will contain large block of contiguous [MASK] tokens the suffix. The simple fix is

---

1.0
120
Throughput (Sokens/sec)
Average # Tokens Sampled In Parallel per Iteration
GSM8K Accuracy
Acuracy
80
02
20
0.4
0.5
0.7
(Multiplicative Mixture Weight)
(Multiplicative Mixture Weight)
Figure 3: Multiplicative Mixture Weight. In Adaptive Parallel Decoding (APD), smaller R values result fewer parallel tokens per iteration but maintain high quality. In particular, achieve far better tradcoff compared to naively parallelizing sone-euoorerrssiively... Notably, generating over tokens per iteration on average is possible with APD while maintaining ~80% accuracy on GSM8K, At the expense of some quality, even over 100 tokens per second is possible.
to set a maximum length M for this suffix. Because the attention computation has O(n²) complexity, decreasing the size of the input lead to significant speed improvements. Unfortunately, the size of the masked lookahead M can change the output distribution by changing the probability of the end of sentence token [EOS]. Empirically, the maximum masked lookahead M can be tuned to gain significant speedup with minimal quality loss.
4 Experiments
In our method, we define three tunable parameters, which we will briefly summarize:
1. Multiplicative Mixture Weight R: Higher results in higher throughput, lower quality
2. Recompute KV Window W: Lower results higher throughput, lower quality
3. Maximum Masked Lookahead M: Lower results in higher throughput, lower quality
Our experimental goal is to empirically analyze the tradeoff in speed and generation quality these parameters vary.
4.1 Implementation
In our experiments, we use Dream 7B Instruct 41 as the diffusion model pD and Qwen2.5 0.5B 40 as the approximate autoregressive model PAR- Both models have demonstrated impressive capabilities on math, science, and reasoning benchmarks. Conveniently, Dream 7B is diffusion model that has been distilled from Qwen2.5 7B, so it is more likely to have distributional overlap with Qwen2.5 0.5B. These models satisfy another prerequisite; they share the same tokenizer. Although our method is theoretically applicable to Llada 7B, which exhibits strong left to right generation capabilities, Llada 7B is trained with a non-standard tokenizer that no existing autoregressive model utilizes.
4.2 Experimental Configuration
For the following experiments, load the models in BF16 precision and them single NVIDIA 24GB A5000 GPU connected to Colfax CXX1060s-EK9 4U Rackmount Server with AMD EPYC (Genoa) 9124 processors. We operate using the LM Evaluation Harness (T1] standard implementation of benchmarks with few modifications and evaluate on GSM8K [6), GPQA [33], and MATH [5].
and HumanEval [4] See Appendix for precise details.
4.3 Tradeoffs
For analyzing tradeoffs, vary each parameter in isolation to measure the impact of each on performance. later experiments, we pl configurations of the parameters jointly. Each plot shows the Grade School Math 8K (GSM8K) (6] accuracy with 500 samples, and the throughput measured by

---

40
120
15
Throughpur (tokens/Seo
Average # of Generated Tokens
GSMBK Accuracy
0.8
13
Accuracy Throughput
28
0.2
16
64
M. (Maximum Masked Lookahead)
M (Maximum Masked Lookahead)
Figure 4: Maximum Masked Lookahead. We illustrate the off between speed and quality when adjusting the maximum masked lookahead (M). Decreasing M significantly increase throughput (speed) but will also reduce quality by shortening the generation length. Thus, this parameter should not be made excessively small.
number of tokens generated divided by generation time. We measure the standard error throughput and accuracy with respect the variation in samples, i.e. different math problem.
Figure shows that we vary the multiplicative mixture weight R. we achieve a range of speed, quality outcomes. We see for small R, we accept fewer tokens per iteration but maintain high quality. Remarkably, is possible to generate over 5 tokens an iteration on average and achieve close to the same accuracy as generating I token per iteration. As increase R, the accuracy drops but not drastically. Constrast this with Figure [] where increasing the number tokens sampled parallel leads to precipitous drop in generation quality. With method, by dynamically choosing when to sample from the diffusion model in parallel, we are able to achieve much higher token acceptance rates without losing generation quality. In Table 2 track statistics of APD and find that the parallel acceptance rate is high.
We also analyze the impact of changing the recompute KV window W for left to right autoregressive generation from the diffusion model, sampling with with PAR (the same as K 1). In Figure (Appendix [). we observe the same relationship between accuracy and throughput, but the tradeoff not as strong. As we decrease the window W, we can achieve a nontrivial speedup at almost no expense to quality.
Finally, we examine the speed-quality throughput exhibited by altering the maximum masked lookahead M. We again observe that there is no free lunch that can increase speed with zero degradations in quality. In this case, we find that decreasing M increases throughput but can significantly alter quality by virtue of the fact that the generation length decreases. In general, works have shown a relationship between increased "thinking time" (the number of generated tokens) and reasoning strength (27). Thus, changing the output distribution of Pp in this way can damage generation quality, especially for complex reasoning tasks.
All together, the parameters in question each exhibit a unique tradeoff between speed and quality. We believe giving the user or practitioner more flexibility to balance these factors inference time an undoubted strength of our approach.
4.4
Pareto Frontier
The Pareto frontier characterizes the set of optimal configurations when faced with the inherent trade-off between two or more conflicting objectives, which the case of LLM inference is speed and quality. A specific LLM configuration is considered Pareto optimal [25) if it is impossible to
improve its performance one objective without simultaneously incurring a detrimental effect on
Table APD Generation Statisties. Mean number of tokens sampled in parallel.
Model
GMSSK
GPQA
MATH
Dream 7B (R 0.7, W
16, M 100)
7.62 + 0.14
4.35 + 0.03
6.45 + 0.04
Dream 7B (R 0.6, W 32, M 200)
5.75 0.05
4.31 + 0.02
6.33 + 0.03

---

1.0
Nocal
(K-21
Accuracy
(R 0.0 Wi-3 M0-201
Dreure 70
Mw1001
GENEK
CANA
MATTH
Humantbal
02
0.0
20
40
80
100
10 15
20
Throughput (tokens/second)
Latency (seconds/query)
Figure 5: The Pareto Frontier. Dream 7B contigured with ADP achieves substantially higher speeds with negligible impact performance compared to the K 1 base model. Notably, Dream with ADP surpasses the speed of autoregressive Qwen 7B and even Qwen 0.5B. This establishes Dream with ADP Pareto-optimal as no other model configuration superior both speed and quality. We show hyperparameter configurations APD, showing we can flexibly tradeoff speed and quality.
another objective, such as increased latency. The Pareto frontier is, therefore, the collection of all such non-dominated configurations, representing the attainable boundary of performance.
Figure 5 we show this frontier over several tasks and model configurations. Dream 7B (K 1) is naive baseline that decodes one token at time, and it therefore occupies low throughput and high density regions. According to our previous assumptions, it will also upper bound the quality of Dream with other decoding parameters. We observe that Dream 7B with our configuration of parameters given by ADP, achieves much greater speed with minimal performance degradation from the base K performance. We also find that Dream with ADP is faster than the autoregressive Qwen 7B and even Qwen 0.5B. Note that if Qwen 7B used speculative decoding with a Qwen 0.5B draft model, it will still never exceed the throughput of ADP. Thus, Dream with ADP is Pareto-optimal, because no model can dominate it in speed and quality. Similar to throughput, ADP also much faster in terms of latency. Diffusion models benefit from the fact that they generate fewer tokens stay within a fixed context window, while autoregressive models can generate much longer reasoning traces that may ramble and not improve quality.
4.5 Qualitative Examples
Though evaluate standard benchmarks, APD can also quickly generate for more open- ende-ended tasks. In Appendix [ we show examples with average number of parallelized tokens on a dataset of persuasive writing prompts {
5
Related Work
Several works propose architectural modifications to LLMs to enable multi-token prediction: Mask- Predict [12), Medusa R. and DynaMo 37 While these works are promising, our focus is multi- token prediction in dI..Ms. Other recent works seek to improve dI.LM speed. For example, block diffusion 2 enables KV caching, but requires training, unlike our method. Discrete copula diffusion 233 is an inference time method that aims to reduces the number of denoising steps required, but does not offer tunable speed, quality tradeoff. More generally, algorithmic approaches to speeding up LLM inference include cascades (28), lookahead decoding
(9]. and speculative decoding (8). which are methods that are generally only applicable in autoregressive models. Building on speculative decoding, works have used different architectures as the draft model including any- any-order autoregressive models [14] and dLLMS (5) We emphasize that using large dLLM to draft in parallel presents fundamentally distinct challenges that are not solved with speculative decoding. Finally, we highlight that quantization is another promising and orthogonal approach for offering a strong speed
quality tradcoff in LLMs 24 (7].

---

Conclusion
In this work, we pursue the significant challenge of increasing inference speed in large language mod- els. We introduced Adaptive Parallel Decoding (APD), a novel algorithm that enables substantially faster sampling from dLLMs. APD uniquely restructures the dLLM into left-to-right autoregressive process and leverages a smaller autoregressive model to assess the quality of parallel- parallel-generated generated token candidates. Complemented by optimizations such as KV caching and limited masked inputs, APD offers tunable parameters that allow a flexible trade-off between generation speed and output quality. This research offers significant advancement in making dLLMs more viable and efficient alternative for fast text generation.
7 Acknowledgment
Special thanks to Oliver Broadrick, who played an important role in brainstorming. This work was funded in part by the DARPA ANSR, CODORD, and SAFRON programs under awards FA8750- 23-2-0004, HR0011259008 12590089, and HR001 12530141, NSF grant IIS ISS194361,, and gifts from Adobe Research, Cisco Research, and Amazon. We also received funding from NSF CAREER Grant #2341040, Schmidt Sciences Early Career Fellowship, and Okawa Foundation Research Award. Approved for public release; distribution unlimited.
10

---

References
[1] Nima Anari, Ruiquan Gao, and Aviad Rubinstein. Parallel sampling via counting. In Proceed- ings of the 56th Annual ACM Symposium Theory of Computing, pages 537-548, 2024.
[2] Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models. arXiv preprint urXiv:2003.0577. 2025.
[3] Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng. Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple Ilm inference acceleration framework with multiple decoding heads. arXiv preprint arXi::2401. 2024.
[4] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint wrXiv2217.003774 2021.
[5] Jacob K Christopher, Brian R Bartoldson, Tal Ben-Nun, Michael Cardei, Bhavya Kailkhura, and Ferdinando Fioretto. Speculative diffusion decoding: Accelerating language generation through diffusion. arXiv preprint arXiv:2408.0550 2024.
[6] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint 10.v:2110.141666 2021.
[7] Tim Dettmers and Luke Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws. In International Conference on Machine Learning, pages 7750-7774. PMLR, 2023.
[8] Esin Durmus, Liane Lovitt, Alex Tamkin, Stuart Ritchie Jack Clark, and Deep Ganguli. Measuring the persuasiveness of language models, 2024.
[9] Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of Ilm inference using lookahead decoding. arXiv preprint arXiv:2402.02057, 2024.
[10] Janos Galambos. Bonferroni inequalities. The Annals of Probability, pages 577-581, 1977.
[11]
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite. Ben Wang. Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024.
[12] Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. arXiv preprint 1904.:990.00333 2019.
[13] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et Deepseek-rl. Incentivizing reasoning capability in Iims via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
[14] Gabe Guo and Stefano Ermon. Reviving any-subset autoregressive models with principled parallel sampling and speculative decoding. arXiv preprint arXiv:2504.20556 2025.
[15] Dan Hendryeks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurlPS, 2021.
[16] Geoffrey E Hinton. Products of experts. 1999 ninth international conference on artificial neural networks ICANN 99. (C.(C. Publ. No. 470), volume pages 1-6. IET, 1999.
[17] Daniel Israel, Aditya Grover, and Guy Van den Broeck. Enabling autoregressive models to fill in masked tokens, 2025.
[18] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax arXiv preprint 1rX14:161..01144, 2016.

---

[19] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001008661, 2020.
[20] Jaeyeon Kim, Kulin Shah, Vasilis Kontonis, Sham Kakade, and Sitan Chen. Train for the worst, plan for the best: Understanding token ordering in masked diffusions. arXiv preprini arXiv:2502.06768, 2025.
[21] Jon Kleinberg and Eva Tardos. Approximation algorithms for classification problems with pairwise relationships: Metrie labeling and markov random fields. Journal of the ACM (JACM), 49(5):616-635 (399,66666622000 2002.
[22] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. International Conference on Machine Learning, pages 19274-19286. PMLR, 2023.
[23] Anji Liu, Oliver Broadrick, Mathias Niepert, and Guy Van den Broeck. Discrete copula diffusion. arXiv preprini urXiv:2410.0194, 2024.
[24] Zechun Liu, Changsheng Zhao, Hanxian Huang, Sijia Chen, Jing Zhang, Jiawei Zhao, Scott Roy, Lisa Jin, Yunyang Xiong, Yangyang Shi, et al. Paretoq: Scaling laws in extremely low-bit Ilm quantization. arXiv preprini arXiv:2502006631, 2025.
[25] Alexander V Lotov and Kaisa Miettinen. Visualizing the pareto frontier. In Multiobjectivve optimization: interactive and evolutionary approaches, pages 213-243. Springer, 2008.
[26] Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion language modeling by estimating the ratios of the data distribution. 2023.
[27] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, 2025.
(28) Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Seungyeon Kim, Neha Gupta, Aditya Krishna Menon, and Sanjiv Kumar. Faster cascades via speculative decoding. arXiv preprint arXiv:2405. 19261, 2024.
[29] Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint arXiv:2502.09999. 2025.
[30] Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. arXiv preprint arXiv:2406.0331 037333 2024.
[31] Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference, 2022.
[32] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9 2019.
[33] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-prood qa benchmark, 2023.
[34] Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 37:1 20116-133188 2024.
[35] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling 11m test-time compute opti- mally can be more effective than scaling model parameters. arXiv preprint arXiv:2408. 03314, 2024.
12

---

[36] Jascha Sohl- Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265 pmlr, 2015.
[37] Shikhar Tuli, Chi-Heng Lin, Yen-Chang Hsu, Niraj Jha, Yilin Shen, and Hongxia Jin. Dy- naMo: Accelerating language model inference with dynamic multi-token sampling. Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference the North American Chapter of the Association for Computational Linguistics: Human Language Technologies ( (Volume I: Long Papers), pages 2922-3345, Mexico City, Mexico, June 2024. Association for Computational Linguistics.
[38] Chunqi Wang, Ji Zhang, and Haiqing Chen. Semi-autoregressive neural machine translation, 2018.
[39] Guanghan Wang, Yair Schiff, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Remasking discrete diffusion models with inference-time scaling, arXiv preprint arXiv:2503.00307, 2025,
[40] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.1511 15115, 2024.
[41] Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b, 2025.
[42] Siyan Zhao, Devaansh Gupta, Qinqing Zheng, and Aditya Grover. dl: Scaling reasoning in diffusion large language models via reinforcement learning, 2025.
[43] Zixuan Zhou, Xuefei Ning. Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, Shengen Yan, Guohao Dai, Xiao-Ping Zhang, Yuhan Dong, and Yu Wang. survey on efficient inference for large language models, 2024.
13

---

A Limitations
Adaptive Parallel Decoding offers a tradeoff, not free lunch. Higher throughput will lead to lower quality, which is a limitation. We also do not claim to improve over the base diffusion model that we are sampling from. Dream 7B is weak in particular domain, our method will also perform poorly.
B Recompute KV Tradeoff
major architectural difference between diffusion language models and autoregressive models is that diffusion models are trained with a bidirectional attention mask to predict masked tokens [7]. and autoregressive models trained predict the token with causal mask. Because of this fundamental difference, autoregressive models have an invariant property: the attention matrix over a fixed of tokens never changes over the course of a generation. Thus. previous "keys" and "values" (KV) used for computing attention can be cached, a process known as KV caching. While the same invariant property does not hold in diffusion language models, in practice, we can still perform KV caching. To adapt KV caching for diffusion language models, we relax the condition that every sampled token will immediately cache its keys and values. Instead we adopt sliding window of size W for which token's 's KV will be recomputed. Once sampled token is more than W entries away from the most recently sampled tokens, we cache its KV.
Here, we show modest gain in throughput for very minimal loss in accuracy by enabling KV caching in diffusion model.
1.0
Troughhe (tokens/sec)
35
GSMBK Accuracy
0.8
30
0.6
25
Accuracy Throughput
20
0.4
15
10
0.2
None
256
128
64
W (Recompute KV Window)
Figure 6
C Experimental Details
Because our models instruction tuned, evaluate the model zero-shot with chat format and system prompt. For GSM8K 6 and HumanEyal (4, the standard system prompt is "You are a helpful
assistant". For GPQA B3 and MATH [5 we modify the system prompt to specify the reasoning
task and answer format.
Also, we set maximum generation length of 256 512 tokens for the diffusion models and 16384 (the maximum context length) for the autoregressive Qwen models. This is a necessary difference, because diffusion models must materialize the maximum context length, unlike autoregressive models. In practice, the diffusion models generate [EOS] before the maximum generation length, at which point end the generation.
When sampling from Dream 7B, we use the hyperparameters of temperature 0.2 and top-p 0.95, as these are set default.
14

---

D Qualitative Examples
Though we evaluate on standard benchmarks, APD can also quickly generate for more open-ended tasks. We show qualitative examples on dataset of persuasive writing prompts g and sample from Dream 7B with APD parameter R 0.7. With APD, we can perform well on open-ended tasks, but compared to reasoning, which has more textual regularities, we quantitatively do not observe the same rate of parallel generation as grade school math questions, for example.
Prompt:
Write a persuasive argument for the following claim: 'Gas car sales should be hanned after a set phase-out date.'
Answer:
The use of gas cars major contributor to the emission of greenhouse gases, which are causing the climate change. Banning gas car sales after a set phase-out date would help to reduce the emission of greenhouse gases and the overall impact of climate change. Additionally, it would encourage the development of electric and hybrid vehicles, which are more environmentally friendly and help to reduce the reliance on fossil fuels.
Average Parallel Tokens per Iteration: 3.38
Prompt:
Write a persuasive argument for the following claim: 'Social media should verify user identities.'
Answer:
Social media platforms should verify user identities to ensure safe and secure environment for all users. By verifying user identities, social media platforms can prevent the spread of fake accounts, cyberbullying, and other harmful activities. Additionally, verifying user identities can help protect the privacy of users by ensuring that their personal information not shared with unauthorized individuals.
Average Parallel Tokens per Iteration: 2.92
15

---

Prompt:
Write persuasive argument for the following claim: 'Virtual emotional companion Al should be allowed and unrestricted'.
Answer:
Virtual emotional companion Al should be allowed and unrestricted because it can provide valuable support system for individuals who are struggling with mental health issues. These AI can offer emotional support, companionship. and sense of connection, which can be especially beneficial for those who are isolated or isolated. Additionally, virtual Al can provide safe and non-judgmental space for individuals to express their feelings and thoughts, which can be especially helpful for those who are struggling with anxiety or depression. By allowing virtual emotional companion Al to be unrestricted, can provide support system for those who need the most and help them to overcome their mental health challenges.
Average Parallel Tokens per Iteration: 2.88
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract, we promise to speed up dLLM generation and provide tunable
parameters. We discuss all of these in the experimental section of our paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The experimental section discusses speed and quality tradeofTs, which in- herently involve discussing limitations. addition, we have Limitations sections in the Appendix.
Justification:
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: While our work involves many theoretical discussions, we do not prove own propositions. Any theoretical claims made are either trivial or can be sourced to other works.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Our experimental results are run with LM Eval Harness, which is highly standardized test bed. Any deviation from it we disclose in the Appendix. Although throughput results are hardware dependent and difficult to reproduce exactly, we document the hardware used and model configuration.
5. Open access to data and code
Question: Does the paper provide open the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code reproduce the method and experiments is available publicly attps://githin com www.d//eeeareeaaaaaaaaaaaaa
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Again, our experiments will be reproducable using our algorithm and the LM Evaluation Harness.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Every plot and table is accompanied by standard error shading and numbers.
17

---

8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We include the hardware specifications needed to run our inference experiments. The experiments do take significant time run (hours days).
9. Code of ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httppp////eurrrsssccco/ooooo/clt
Question:
Answer: [Yes]
Justification: We have taken every possible effort to conform, in every respect, with the NeurIPS Code of Ethics.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: While we discuss positive side effects in the introduction of faster inference algorithms, have not discussed negative impacts because it is difficult think of negative societal impacts of faster LLM generation. Since it is generic improvement, we believe the discussion is not warranted.
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Because do acknowledge negative societal impacts of our work, also do not believe any safeguards are necessary on our work.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Every work that this paper builds on is source and properly cited. We are compliant with all licenses.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our contribution is purely algorithmic and does not introduce any assets.
14. Crowdsourcing and research with human subjects
Question: crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects. Institutional review board (IRB) approvals or equivalent for research with human subjects
18

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [No]
Justification: While the core method involves algorithmically speeding up LLMs, LLMs did not contribute any important, original, or non-standard components.
19


Reviews:
[{"review_id": "LOHg6iZiya", "reviewer": "Reviewer_SxPY", "summary": "This paper introduces Adaptive Parallel Decoding (APD), a novel decoding strategy designed to accelerate inference in diffusion-based large language models (dLLMs). Unlike traditional autoregressive decoding, which is inherently sequential, APD enables dynamic parallel token generation by combining the marginal probabilities from a diffusion model with the joint probabilities from a smaller auxiliary autoregressive model. The method introduces three tunable parameters—mixture weight (R), KV cache window (W), and maximum masked lookahead (M)—to balance the trade-off between speed and quality. Empirical results show that APD significantly improves throughput with minimal degradation in output quality, outperforming both naive diffusion decoding and autoregressive baselines on benchmarks like GSM8K, GPQA, and MATH.", "questions": "- In Table 1, why is random decoding always slower than entropy/confidence-base decoding for Dream/Llada models?", "limitations": "The authors have addressed the limitations of their method.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Paper is well motivated: Table 1 demonstrates that open source autoregressive LLMs still outperform the recent dLLMs in terms of performance v.s token/rate.\n- Paper is well written and easy to read.\n- The proposed technique is novel (as far as I know), the idea of adaptively choosing how many tokens to generate in parallel with a dLLM is new.\n- The paper presents extensive empirical results on recent LLMs to back their claims.\n\nWeaknesses:\n- The dependence on an auxiliary approximate autoregressive model limits its applicability. At the very least, ADP requires that for each dLLM, a small LLM with the same tokenizer exists. \n- I found sampling from $p_T$ as presented in eq (8) to be slightly confusing. In practice (as presented in Algorithm 1), $p_D$ is first used to generate $n$ tokens, which are then subsequently used by $\\hat{p}_{AR}$. Does this not impose some conditional dependence?", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer SxPY,\n\nWe thank the reviewer for their positive feedback regarding our writing and empirical results. We hope to use this opportunity to clarify questions and concerns.\n\n>\"The dependence on an auxiliary approximate autoregressive model limits its applicability. At the very least, ADP requires that for each dLLM, a small LLM with the same tokenizer exists.\"\n\nWhile ADP, as currently formulated, requires a smaller model with the same tokenizer, this problem is generally recognized to be the same in speculative decoding. However, a recently published work [1] shows that it is possible to perform speculative decoding over different tokenizers by performing verification over strings. This same idea naturally extends to ADP, where in our accept criterion, we can accept strings that the diffusion model and product mixture agree on.\n\n>\"I found sampling from $p_T$ as presented in eq (8) to be slightly confusing. In practice (as presented in Algorithm 1), $p_D$ is first used to generate $n$ tokens, which are then subsequently used by $\\hat{p}_{AR}$. Does this not impose some conditional dependence?\"\n\nIntuitively, $p_D$ captures the marginals and $p_T$ can capture joint dependencies. We sample from $p_D$ initially because it can be performed in parallel. We then resample each token conditioned on the previous tokens in the sequence using $p_T$ and check that the original samples $p_D$ are valid under $p_T$. Thus, you are correct that $\\hat{p}_{AR}$ (which is used to compute $p_T$) is adding conditional dependence between tokens. This is desirable because the inter-token dependencies are necessary to verify whether a sample should be accepted.\n\n>In Table 1, why is random decoding always slower than entropy/confidence-base decoding for Dream/Llada models?\n\nThank you for the perceptive question. When a diffusion model outputs the EOS token with everything to its left sampled, the generation can stop. A random generation order is not likely to satisfy this condition early, while entropy/confidence/left-right are much more likely. Thus, generation order and answer length can change the throughput significantly, as the attention operation grows O(n^2) and the diffusion model is not KV caching by default.\n\n[1] Timor, Nadav, et al. \"Accelerating llm inference with lossless speculative decoding algorithms for heterogeneous vocabularies.\" arXiv preprint arXiv:2502.05202 (2025)."}, {"review_id": "QXze6qpQMo", "reviewer": "Reviewer_mTc7", "summary": "LLMs today excel at text generation but remain bottlenecked by the strictly sequential nature of AR decoding. Diffusion-based LLMs offer parallel sampling in theory, yet in practice—exemplified by Dream and Llada—they only match AR quality when run one token at a time. To bridge this gap, this paper introduces APD: a left-to-right diffusion schedule that turns the model autoregressive, combined with a lightweight AR verifier to decide how many tokens to accept in parallel via a multiplicative product-of-experts criterion. They further accelerated inference with KV-caching beyond a sliding window and by setting a threshold for the masked lookahead. Together, these three tunable knobs let APD dramatically boost throughput—up to nearly twice the speed of pure AR—while incurring only minimal quality loss across three benchmarks.", "questions": "NA", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strength:\n\n1) Detailed analysis of the performance of the dLLMs in terms of accuracy and decoding strategies, showing the limitations of purely autoregressive generation and propose a novel method to maintain high quality while increasing the throughput.\n\n2) Proposing enabling KV caching for tokens outside a pre-specified window together with maximum masked lookahead to increase the throughput of dLLMs.\n\n3) APD consistently maintains near-autoregressive accuracy—even when accepting on average 5–7 tokens per iteration—while delivering up to a 2× speed-up over pure AR sampling on GSM8K, GPQA, and MATH.\n\nWeaknesses:\n\n1) I would appreciate a similar accuracy vs throughput comparison table similar to table 1, comparing the performance of Dream 7B + Qwen 2.5 0.5 with Qwen 7B to be able to observe the direct comparison of auto-regressive decoding and dLLMs with APD.\n\n2) All experiments focus on three reasoning benchmarks (GSM8K, GPQA, MATH) and a single diffusion backbone (Dream 7B). Without additional evaluations—e.g., text summarization, code generation, translation—and tests on other dLLMs, it remains unclear whether APD’s speed-quality trade-offs generalize beyond the presented setup.", "was_revised": true, "final_justification": "This paper presents a significant advancement by introducing APD, a novel decoding strategy that achieves up to a 2x inference speed-up on challenging reasoning benchmarks while impressively maintaining near-autoregressive accuracy. The work provides a robust and practical solution to the inherent latency of diffusion-based LLMs, and while further generalization is future work, the existing results convincingly validate the method's core contribution to efficient text generation.", "rebuttal": "Dear Reviewer mTc7,\n\nWe are thankful that the reviewers recognize the novelty and impact of our work. We are also thankful for their insightful suggestions, which we aim to incorporate into the paper.\n\nAs recommended, here is a table summary similar to Table 1 for the respective benchmarks:\n\n\n| Model | GSM8K Accuracy | GSM8K Throughput | GPQA Accuracy | GPQA Throughput | MATH Accuracy | MATH Throughput |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: |\n| Qwen 7B | **0.85** | 38.6 | **0.31** | 38.6 | **0.64** | 38.7 |\n| Qwen 0.5B | 0.34 | 55.0 | 0.24 | 54.7 | 0.19 | 55.8 |\n| Dream 7B K=1 | 0.83 | 10.1 | 0.26 | 7.66 | 0.40 | 6.15 |\n| ADP Config 1 | 0.76 | 53.3 | 0.30 | 43.2 | 0.34 | 56.7 |\n| ADP Config 2 | 0.71 | **104.3** | 0.30 | **56.9** | 0.30 | **83.3** |\n\n**ADP Config 1**: Dream 7B + Qwen 2.5 0.5B, (R = 0.6, W = 32, M = 200)\n\n**ADP Config 2**: Dream 7B + Qwen 2.5 0.5B, (R = 0.7, W = 16, M = 100)\n\nWe will include these table results in the Appendix for more direct comparisons. \n\nTo test for generalization to the other domains, we will also include coding in our evaluation suite. These results were obtained on NVIDIA A6000 GPUs due to memory errors, so they may not be directly comparable to the rest of the paper.\n\n| Model | HumanEval Accuracy | HumanEval Throughput |\n| :--- | :---: | :---: |\n| Qwen 7B | **0.76** | 37.6 |\n| Qwen 0.5B | 0.25 | 81.1 |\n| Dream 7B K=1 | 0.55 | 2.88 |\n| ADP Config 1 | 0.39 | 66.5 |\n| ADP Config 2 | 0.40 | **100.1** |\n\nAs can be seen in this table, ADP results generalize beyond the existing benchmark scores. While it does not obtain the same quality as autoregressive due to a worse base model (as can be seen in the K=1 result), it is firmly on the Pareto frontier in terms of quality and throughput.\n\nWe thank the reviewer for their generous feedback, and we hope that these updates have improved the paper significantly."}, {"review_id": "LUMzjsptyC", "reviewer": "Reviewer_4Kag", "summary": "This paper introduces Adaptive Parallel Decoding (APD), a novel method that accelerates diffusion-based LLMs by dynamically adjusting parallel token generation per decoding step. APD uses a multiplicative mixture between dLLM marginal probabilities and a small autoregressive model's joint distribution to determine token acceptance, enabling adaptive parallelization while maintaining quality. The framework integrates KV caching and masked lookahead constraints for acceleration and offers tunable parameters to balance throughput versus quality. APD achieves 59 tokens/sec (vs. 37 tokens/sec for autoregressive baselines) on reasoning tasks like GSM8K with minimal quality degradation, outperforming naive parallel approaches.", "questions": "Q1: How much latency does the small AR model introduce? Does KV caching sufficiently offset its computational overhead to justify inclusion?\n\nQ2: Given Table 1 shows left-to-right decoding yields optimal dLLM performance, can APD effectively integrate with entropy/confidence-based decoding schemes (e.g., Llada’s approach) without compromising throughput gains?\n\nQ3: Can APD achieve high parallelism (>5 tokens/step) in open-ended generation? Fig 2 suggests quality drops sharply for k>5 in naive parallelization, does APD overcome this limitation for non-reasoning domains?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n+ This paper proposes the APD framework, by using a small AR model to validate large dLLM samples, dynamically adjust the number of parallel tokens, and solve the contradiction between parallelism and quality of dLLM.\n+ The desigh integrates KV caching and mask lookahead restrictions, significantly accelerating dLLM inference without the need for re-training the model.\n+ The effect is good. It achieves 59 tokens per second on tasks such as GSM8K, with a 60% increase in throughput and a quality loss of less than 2%.\n\nWeaknesses:\n\n- APD requires dLLMs specifically trained for left-to-right decoding, with non-left-to-right implementations such as Llada exhibiting substantially weaker performance as evidenced in Table 1.\n- The small AR model adds latency; its impact on end-to-end throughput isn’t quantified.\n- Evaluation focused on math/reasoning (GSM8K, GPQA, MATH). Open-ended generation shows lower parallelism (2.9–3.4 tokens/step) as evidenced in Appendix D, suggesting task-dependent efficacy.", "was_revised": true, "final_justification": "In the paper, the authors propose APD to accelerate diffusion LLMs by dynamically adjusting parallel token generation per decoding step. The paper is interesting and the authors have addressed all the concerns from the reviewer. Hence, I give the paper a rating of 5.", "rebuttal": "Dear Reviewer 4Kag,\n\nThank you for your positive feedback. We hope to also answer your insightful questions and attend to your concerns.\n\n>APD requires dLLMs specifically trained for left-to-right decoding, with non-left-to-right implementations such as Llada exhibiting substantially weaker performance as evidenced in Table 1.\n\nTo be clear, ADP does not *require* a dLLM trained for left to right decoding. Dream 7B’s training objective trains for random decoding, but because it was initialized from an autoregressive model it will have better representations in a left to right setting than Llada. However, Table 1 shows that Llada can also perform left to right decoding, and on GSM8K left to right decoding is even better than confidence based decoding. While ADP does not require any left to right training, the success of ADP does indicate that that parallel prediction can be performed left to right, and additional left-right fine tuning on top of dLLMs will likely improve the results of ADP.\n\n>Q1: How much latency does the small AR model introduce? Does KV caching sufficiently offset its computational overhead to justify inclusion?\n\nThank you for the suggestion. We have compiled a table for GSM8K to show the impact of the small AR model.\n\n| Configuration | Latency (s) | AR Verification (s) | Ratio (%) |\n| :--- | :---: | :---: | :---: |\n| Dream K=1 | 11.6 | 0 | 0% |\n| ADP Config 1 | 2.18 | 1.09 | 50% |\n| ADP Config 2 | 1.40 | 0.73 | 52% |\n\n**ADP Config 1**: Dream 7B + Qwen 2.5 0.5B, (R = 0.6, W = 32, M = 200)\n\n**ADP Config 2**: Dream 7B + Qwen 2.5 0.5B, (R = 0.7, W = 16, M = 100)\n\nOverall, the latency using ADP is much lower than the default diffusion generation in Dream. However, the impact of the AR verification step is significant, and the model spends roughly half the time in this step. We see this as an opportunity for further optimizations. More efficient implementations of Gumbel sampling and better KV cache management using Paged Attention would likely reduce the latency significantly. The current implementation of ADP is purely PyTorch.\n\n>Q2: Given Table 1 shows left-to-right decoding yields optimal dLLM performance, can APD effectively integrate with entropy/confidence-based decoding schemes (e.g., Llada’s approach) without compromising throughput gains?\n\nAs currently formulated, ADP can only work for left to right decoding. This is the case because the autoregressive verifier computes a left to right likelihood. Consider the following example where tokens are separated by commas\n\n>Prompt: The, capital, of \n\n>Diffusion Sample: France, is, is, is, is, is\n\nThe diffusion model may sample this because it will be marginally likely due to there being many continuations besides \"France\" that can span multiple tokens. The autoregressive verifier will find the final “is” very likely due to the repetition, so we need to reject samples after the first disagreement between the diffusion samples and verifier (only accept up to the first “is”). This accept strategy produces a left to right decoding order. Adapting ADP for arbitrary decoding orders is an interesting open research question and may require a different auxiliary model such as an Any-Order Autoregressive model [1].\n\n>Q3: Can APD achieve high parallelism (>5 tokens/step) in open-ended generation? Fig 2 suggests quality drops sharply for k>5 in naive parallelization, does APD overcome this limitation for non-reasoning domains?\n\nThe problem of parallel sampling is inherently probabilistic and therefore will be data dependent. It is likely that non-reasoning domains are less parallelizable in a fundamental way due to more interdependent tokens. Because open-ended generation is more difficult to evaluate, we do not include it in our benchmarks, but as Appendix D shows we can still achieve significant speed up over baseline diffusion models. The strength of ADP is that it can adaptively find opportunities to parallelize generation, and where this cannot be achieved, it automatically applies less parallelization so that quality is not severely impacted.\n\n[1] Shih, Andy, Dorsa Sadigh, and Stefano Ermon. \"Training and inference on any-order autoregressive models the right way.\" Advances in Neural Information Processing Systems 35 (2022): 2762-2775."}, {"review_id": "zJb9yJvaH4", "reviewer": "Reviewer_ezv3", "summary": "The authors propose Adaptive Parallel Decoding (APD), which improves sampling speed for existing diffusion-based large language models (dLLMs) compared to original methods like Dream and Llada. By introducing KV cache and limited masked inputs, APD enables flexible trade-offs between generation speed and quality.", "questions": "1. Llada’s reference paper tested more diverse datasets. Why restrict evaluation to only three?  \n2. The reference to \"Dream 7B\" (Citation 39) appears incorrect—no matching work was found. Is this a citation formatting error?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:  \n1. APD adaptively determines the number of tokens generated per step via a small autoregressive model to evaluate output quality, mitigating the quality degradation caused by fixed-length parallel generation.  \n2. APD can be directly applied to existing dLLMs with significant speedup benefits.  \n3. The use of KV cache and limited masked inputs further enhances dLLM inference efficiency.  \n  \nWeaknesses:  \n1. The proof relying on Bonferroni’s inequality (Property 1) is flawed. As noted in Line 122, $P_{APD}=P_{AR}$ holds only if $|g|=n$; when $|g|<n$, they are not equivalent. Thus, applying Bonferroni’s inequality is mathematically unsound. This also invalidates the heuristic justification for Property 2.  \n2. The author directly uses the KV cache method, but this method has already been proposed and there are different variants, so it cannot be considered an innovative point.  \n3.  Experiments are confined to GMS8K, GPQA, and MATH without theoretical guarantees, leaving generalization to other datasets unverified.", "was_revised": true, "final_justification": "The author's rebuttal has basically resolved my question.", "rebuttal": "Dear Reviewer ezv3,\n\nThank you for your feedback. We hope to address your questions and concerns.\n\n>The proof relying on Bonferroni’s inequality (Property 1) is flawed. As noted in Line 122, $P_{APD}=P_{AR}$ holds only if $|g|=n$; when $|g|<n$, they are not equivalent. Thus, applying Bonferroni’s inequality is mathematically unsound. This also invalidates the heuristic justification for Property 2.\n\nWe are not making a formal statement about $p\\_{APD}$ or $\\hat{p}\\_{AR}$. Our proof shows that if $p\\_D(x)=1$, then $p\\_{AR}(x)=1$. Because the goal of $p_T$ is to approximate $p_{AR}$, we are stating that the same property must hold: if $p\\_D(x)=1$, then $p\\_{T}(x)=1$. This is our justification of Property 1.\n\n>The author directly uses the KV cache method, but this method has already been proposed and there are different variants, so it cannot be considered an innovative point.\n\nWe are unsure which papers are being referred to here and respectfully ask for clarification. If the reviewer is referring to the use of KV cache more broadly for AR models, indeed there are many works which use it. KV caching in dLLMs is very natural in APD because we are decoding left to right, and the decoding order is similar to autoregressive. The challenge of KV caching in a dLLM is that because of bidirectional attention, previously computed keys and values can change. In practice, we show in the paper that only KV from a small window from the most recently generated tokens need to be updated to maintain performance. We do not consider KV caching the central point of the paper, but an optimization to achieve better results.\n\n>Experiments are confined to GMS8K, GPQA, and MATH without theoretical guarantees, leaving generalization to other datasets unverified.\n\nBecause ADP exploits conditional dependence, which is a property of the data, to further parallel decoding, it is difficult to give theoretical guarantees on the speedups achievable. We emphasize that the advantage of ADP is that it dynamically adapts to the data to speed up decoding when it is possible without significantly degrading quality, but we are unable to formalize the precise impact on performance. \n\n>Llada’s reference paper tested more diverse datasets. Why restrict evaluation to only three?\n\nMany of the datasets for evaluating Llada such as MMLU and Hellaswag follow likelihood-based evaluation (eg, using multiple-choice questions) and are not generative, which is a requirement for evaluating our method. In light of the reviewers’ concern, we added a coding benchmark: Humaneval.\n\n| Model | HumanEval Accuracy | HumanEval Throughput |\n| :--- | :---: | :---: |\n| Qwen 7B | **0.76** | 37.6 |\n| Qwen 0.5B | 0.25 | 81.1 |\n| Dream 7B K=1 | 0.55 | 2.88 |\n| ADP Config 1 | 0.39 | 66.5 |\n| ADP Config 2 | 0.40 | **100.1** |\n\n**ADP Config 1**: Dream 7B + Qwen 2.5 0.5B, (R = 0.6, W = 32, M = 200)\n\n**ADP Config 2**: Dream 7B + Qwen 2.5 0.5B, (R = 0.7, W = 16, M = 100)\n\nAs can be seen in this table, ADP works well for code. It maintains most of the accuracy of the underlying base model at K=1 but can achieve 34x the throughput of the base model. Additionally, it is firmly Pareto optimal by improving on Qwen 7B in throughput and Qwen 0.5B in quality.\n\n>The reference to \"Dream 7B\" (Citation 39) appears incorrect—no matching work was found. Is this a citation formatting error?\n\nWe are citing the bibtex directly from the Dream 7B blog. There is currently no paper associated with the model.\n\n@misc{dream2025,\ntitle = {Dream 7B},\nurl = {_omitting link due to rebuttal rules_},\nauthor = {Ye, Jiacheng and Xie, Zhihui and Zheng, Lin and Gao, Jiahui and Wu, Zirui and Jiang, Xin and Li, Zhenguo and Kong, Lingpeng},\nyear = {2025}\n}\n\nWe thank the reviewer for their valuable critiques, and we hope that our response provides enough assurance to increase our score."}]

Author rebuttal:

Meta-review: The paper presents a clear, useful decoding method for dLLMs with strong empirical throughput gains and modest quality costs, supported by additional analyses introduced during rebuttal. While broader task coverage, cross-tokenizer implementation, and stronger theory would further strengthen the work, the contribution is substantive and relevant for the community.
Decision: Accept (spotlight)