ES-dLLM: Efficient Inference for Diffusion Large Language Models by Early-Skipping
Authors: Zijian Zhu
Fei Ren
Zhanhong Tan
Kaisheng Ma
Venue: ICLR 2026 Poster (2026)
Keywords: Diffusion Large Language Model
Inference Acceleration
KV Caching
Primary area: foundation or frontier models, including LLMs

Abstract: Diffusion large language models (dLLMs) are emerging as a promising alternative to autoregressive models (ARMs) due to their ability to capture bidirectional context and the potential for parallel generation. Despite the advantages, dLLM inference remains computationally expensive as the full input context is processed at every iteration. In this work, we analyze the generation dynamics of dLLMs and find that intermediate representations, including key, value, and hidden states, change only subtly across successive iterations. Leveraging this insight, we propose ES-dLLM, a training-free inference acceleration framework for dLLM that reduces computation by skipping tokens in early layers based on the estimated importance. Token importance is computed with intermediate tensor variation and confidence scores of previous iterations. Experiments on LLaDA-8B and Dream-7B demonstrate that ES-dLLM achieves throughput of up to 226.57 and 308.51 tokens per second (TPS), respectively, on an NVIDIA H200 GPU, delivering 5.6$\times$ to 16.8$\times$ speedup over the vanilla implementation and up to 1.85$\times$ over the state-of-the-art caching method, while preserving generation quality.

Published as a conference paper at ICLR 2026
ES-DLLM: EFFICIENT INFERENCE FOR DIFFUSION LARGE LANGUAGE MODELS BY EARLY-SKIPPING
Zijian Zhu
Fei Ren
III,, Tsinghua University zhuzj22@maill als...l...... l...maal...ssa.lliii........ www.ttinnngg.... edu.cn
IIS, Tsinghua University
ren f25èma 1s. tsinghua edu.
Zhanhong Tan
Kaisheng Ma
Polar Bear Tech. tanzha515@gmail.ccm com
III, Tsinghua University
kaishengemail tsinghua, edu. en
ABSTRACT
Diffusion large language models (dLLMs) are emerging as a promising alterna- tive autoregressive models (ARMs) due their ability to capture bidirectional context and the potential for parallel generation. Despite the advantages, dLLM inference remains computationally expensive as the full imput context is processed at every iteration. In this work, we analyze the generation dynamics of dLLMs and find that intermediate representations, including key, value, and hidden states, change only subtly across successive iterations. Leveraging this insight, we pro- pose ES-LLLM, a training-free inference acceleration framework for dLLM that reduces computation by skipping tokens in early layers based the estimated im- portance. Token importance is computed with intermediate tensor variation and confidence scores of previous iterations. Experiments LLaDA-8B and Dream- 7B demonstrate that ES-dLLMM achieves throughput of up to 226.57 and 308.51 tokens per second (TPS), respectively, on NVIDIA H200 GPU, delivering 5.6x to 16.8x speedup over the vanilla implementation and up to 1.85 x over the state- of-the-art caching method, while preserving generation quality. The source code is available at httpp:: httt:///ithubb com/
dLLM
INTRODUCTION
Autoregressive models (ARMs) have been the dominant paradigm in large language models (LLMs), achieving remarkable success in a wide range of applications (Cao et al. 2023). ARMS generate text in left-to-right pattern, producing tokens sequentially. Recently, diffusion- LLMs (dLLMs) have emerged as promising alternative. Unlike ARMs, dLLMs generate text through an iterative denoising process over sequence of mask tokens, enabling bidirectional attention and offering the potential for parallel decoding (Yu et al. 2025). Industrial dLLMs such as Mercury (Inception Labs) ctt al. 2025) and Gemini Diffusion (Deepmind 2025) have drawn significant attention for their ultra- fast generation speed. Despite this progress, open-source dLLMs, such as LLaDA (Nie et al.) 2025) and Dream Ye et al. 2025), remain far less efficient, even slower than ARMs of comparable size.
A key factor contributing to the inefficiency of dLLMMs is that each iteration processes the entire sequence as input, introducing substantial computational overhead. To mitigate this cost, recent studies (Ma et al. 2025 Wu al. 2025 Liu al. 2025) have proposed techniques such caching and parallel decoding mechanisms to improve generation efficiency. Nevertheless, more opportuni- ties remain for further acceleration. In each iteration during dLLM generation, only tokens with the highest confidence are unmasked, while the majority of mask tokens are processed without producing useful results. Moreover, since the inputs of adjacent iterations differ only in the positions of newly unmasked tokens, the intermediate states of most tokens remain almost un- changed. Despite this redundancy, conventional inference procedures still compute logits for all token positions, leading to excessive and unnecessary computation.
a few
"Corresoonding Author.

---

Published as a conference paper at ICLR 2026
To better understand these inefficiencies, conduct series of experiments analyze the charac- teristics during the dLLM generation process. The results reveal that both intermediate tensors and confidence scores exhibit only subtle variation across successive iterations. Therefore, identify opportunities to predict the importance token positions and to eliminate redundant computation in early layers for tokens that contribute little the outcome.
Motivated by these observations, we propose ES-dLLM a training-free inference acceleration framework designed for diffusion LLMs. Specifically, it focuses on optimizing redundant token computation in each iteration, estimating token importance using intermediate variations and prior confidence scores, and skipping low- importanccc positions in early layers of the inference process. ES-dLLM consists of two key components:
Importance Score Estimation: ES-dLLM skips irrelevant token positions by estimating their importance scores in early layers of the model, based the variation of intermediate tensors and confidence from previous iterations. Using importance scores, only the top-k positions are selected for further inference, while the remaining positions are skipped in the current iteration.
Partial Cache Update and Early Skip: ES-dLLM maintains intermediate tensors as caches to be reused in subsequent iterations, including key and value tensors for atten- tion layers and hidden states as the indicator for variation estimation in importance score. Since only subset of tokens passes through the inference, caches corresponding to the se- lected positions are updated with an in-place seatter operation, while the others are reused without recomputation.
The experimental results show that ES-dLLM substantially accelerates inference, achieving through- put of up to 226.57 and 308.51 51 tokens per second (TPS) on the LLaDA-88 and Dream-78 -BB models, respectively, using NVIDIA H200 GPU, without compromising generation quality.
In summary, this paper makes the following contributions:
1. We analyze the characteristics during the dLL.M generation process and observe that inter- mediate tensors and confidence scores of most positions exhibit only minor variation across iterations, thereby revealing opportunities to eliminate redundant computation.
2. We propose ES-dLLM, training-free inference acceleration framework that reduces per-iteration computation by early-skipping low-importance token positions.
3. We conduct extensive experiments and ablation studies, demonstrating that ES-dLLM achieves significant speedups of 5.6x-16.8x over the original implementation and up to 1.85 x compared saate-of-the-ane caching methods, all without sacrificing gener- ation quality.
PRELIMINARY ON DIFFUSION LARGE LANGUAGE MODELS
Diffusion large language models introduce new paradigm in natural language processing that adopts the diffusion mechanism. Unlike traditional autoregressive models, which generate se- quentially in token-by- by-token manner, dLLMs through series of denoising iterations, progres- sively unmasking tokens appended to the input.
Let V denote the vocabulary that includes special mask token [mask] E V. Given an input sequence (I1, F2.., Fn.) where x; E V, text generation begins by appending I mask tokens to form (0) (II, 22, Iny [mask]; [mask]), where specifies the desired generation length. In the 1-th iteration, a Transformer-baaee token predictor f is used to estimate the probability distribution over the vocabulary for token positions (rather than just the next token ARMs) in the sequence.
po ( (1)
'The name is inspired by early-exit (EE) Chen et al. 20244). which accelerates inference by exiting "easy" tokens early via trained EE layers. In contrast, ES M.M.M early-skip (ES) inrelevnt positions without additional training.
2

---

Published as a conference paper at ICLR 2026
From this distribution, subset of mask tokens is replaced with sampled tokens, producing the up- dated sequence (0) Several replacement strategies exist, and widely used approach is to unmask positions with the highest confidence (Yu al. 2025), where confidence for each position is defined as the maximum probability over the vocabulary. This iterative process continues until all mask tokens are replaced.
Compared to ARMs, which condition predictions solely on preceding tokens, dLLMs leverage the entire sequence context in each iteration. This enables them to capture bidirectional dependencies and generate more coherent text. However, it also introduces computational challenges due to bidi- rectional attention and flexible, non-sequential generation order. At the same time, computing entire logits creates opportunities for parallel decoding Wu et al. (2255)
RELATED WORKS FOR DLLM INFERENCE ACCELERATION
Semr-autoregressive Generation. LLaDA (Nie et al. 2025) adopts a semi-autorrerressive genera- tion strategy that partitions the output sequence into multiple blocks, each generated in diffusion manner while preserving sequential order across blocks. This approach constrains the generation order, but provides better performance. Furthermore, Arriola al. (2025) trained the BD3-LM model, which applies unidirectional attention between output blocks. This design enables lossless Key-Value (KV) caching of previous blocks, but sacrifices the ability to exploit bidirectional context.
Accelerating using KV Caching, In ARMs, KV caching is a common technique (Pope et al., 2023), where intermediate key and value tensors are stored and reused across decoding steps, since atten- tion depends solely on past tokens. In diffusion LLMs, however, the situation is more challenging due the bidirectional attention: newly unmasked tokens can influence all preceding tokens. To mitigate computational cost, several works have explored approximate KV caching strategies. dKV- Cache (Ma et al. 2025) delays the KV update for newly unmasked tokens to reduce errors, while dLL.M- (L.iu et al. 2025) caches all intermediate tensors and adaptively updates them in each layer using a V-verify mechanism. However, these methods do not leverage the cemi-autoregressvve generation paradigm, which is beneficial for both efficiency and generation quality. Fast-dLLM (Wu et al. 2025) proposed DualCache for semi- aatttrrurrreeesiiveeee generation, which maintains separate KV caches for context on both sides and computes attention only for the current block during inference.
Sparse Attention. Recent works such as Sparse-dLLM, (Song et al. 2025) and DPad [Chnn et al. (2025) exploit sparsity in attention scores and accelerate inference by modifying the set of attended tokens, either by sparsifying historical tokens or dropping distant suffix tokens. The techniques are orthogonal our approach, which optimizes the of processed tokens.
Parallel Decoding, In diffusion LLMs, logits are computed for all token positions in each iteration, allowing multiple tokens be unmasked simultaneously. [W al. (2025) proposed confidence- aware parallel decoding, which dynamically adjusts the number of tokens to be unmasked per itera- tion based on confidence threshold. This strategy effectively reduces the number of iterations for generation while preserving quality.
OBSERVATIONS ON DIFFUSION LLM CHARACTERISTICS
We conduct series of experiments to analyze the characteristics of diffusion LLMs during gener- ation, with particular attention to variations across successive iterations. For this study, we use two pre-trained diffusion LLMs, LA-DA-B--Insttt (Nie et al., (025) and Dream-7B-Instrr (Ye al., and perform inference on sequences sampled from widely used benchmark datasets.
2025
4.1 CONFIDENCE VARIATION ACROSS ITERATIONS
In dLLM generation, common strategy for selecting tokens to be unmasked is to choose posi- tions with the highest confidence (i.e., the maximum softmax probability over the vocabulary for each position). Figure Ia visualizes the confidence variation, measured the absolute difference between consecutive iterations, during generation for a sample from the BBH dataset using LLaDA-
Results for Dream-7B TB-Instruct provided in Appendix A.2
3

---

Published as a conference paper at ICLR 2026
10a
51
10-1
Confidenc Dift.
Iteration
Prequenty
10-4
201
10-4
50
100
150
200
250
0.4
0.5
0.8
1.
Position
Confidence Dift.
(a) Single-Sample heatmap.
(b) Log-scale distribution.
0.20
Proportonn
0.15
0.10
0.05
0.00
lteration
(c) Proportion of variation 0.05 at each iteration.
Figure 1: Confidence variation statistics using -LaDA-8B-Instrus (a) uses sample from the BBH dataset, while (b) and (c) present results using 100 samples from multiple datasets.
8B-Instruct. We observe that the confidence variation is subtle for most cases. To quantify this observation, Figures [bb and Ic present the distribution of confidence changes across all positions and iterations, and the proportion of positions with confidence variation greater than 0.05 at each iteration, based on 100 samples from multiple datasets. The results reveal that confidence changes approximately follow an exponential distribution. with the majority concentrated near zero. Except for some initial iterations, fewer than 10% of positions show confidence variation greater than 0.05.
The results suggest that most tokens experience only minimal confidence fluctuations across itera- tions. Consequently, confidence scores from previous iterations can serve as reliable predictors, helping identify the positions that are likely to be unmasked in the current iteration.
4.2 HIDDEN STATE VARIATION
51
10"
Hidden State Dm
30:1
10x4
201
16-
Zt0
450
905
0.0
0.2
04
3.5
Posttion
Hidden State Dm
(a) Single-sample heatmap.
(b) Log-scale distribution.
Figure 2: Hidden state variation layer 10 using LLaDA-82 SAAID---B-Inntrues (a) uses single sample from the BBH dataset, while (b) presents results using 100 samples from multiple datasets. The vertical line in (a) separates prompt and output tokens, and the distribution in (b) includes only output tokens.
Since the inputs of adjacent iterations differ only by the token that has just been unmasked, further analyze how this input change affects the intermediate tensors. Specifically, we measure the variation of hidden states (i.e., the output after the feed-forward of Transformer block) in selected
4

---

Published as a conference paper at ICLR 2026
Transformer layer between consecutive iterations. The variation is guantified the normalized L1- norm the difference, consistent with the latter term in Equation
As shown in Figure 2 the variation is minor for most positions, and only small fraction of positions exhibit noticeable changes. This indicates that input changes generally have limited impact intermediate states. Consequently, it is feasible skip computation for tokens with small variations without degrading generation quality. Similar patterns are observed for other layers and intermediate tensors, detailed in Appendix A.1
METHODOLOGY
Transformer Layer 31
Transformer Layer 31
Transformer Layer 31
Transformer Layer
Transformer Layer
Transformer Layer
Transformer Layer
Transformer Layer
Transformer Layer 4
Transformer Layer 0
Transformer Layer 0
Transformer Layer 0
Promet
Blook-0
Block-1
Block-2
Cached Taken
Reused Position
Skipped Position
(a) Vanilla
(b) DualCache
(c) ES-dLLM
Figure 3: Illustration of ES-dLLM compared with the vanilla implementation and DualCache, as- suming block-1 I is under processing. The figure presents only tokens per block, while the actual block length can be much larger (e.g., 32 or 64).
Motivated by the observations in Section celeration framework for diffusion LLM. ES-dLLM reduces computational overhead by selectively skipping redundant token computations during inference. Specifically, it estimates the importance score for each token position using both the confidence from previous iterations and the variation of intermediate tensors, and applies an early-skip mechanism to bypass tokens with low importance in early layers. To enable this process, ES-dLLM maintains cache that stores the necessary in- termediate tensors and confidence scores for all positions. The cache is partially updated only for non-skipped tokens in each layer, while the others are reused directly. Figure 3 compares the in- ference process of ES-aLLM with the vanilla implementation and the state-of-the-art DualCache method. DualCache caches key and value tensors for tokens outside the current processing block, thus computing for the entire block. contrast, ES-dLLM further reduces computation by skipping unimportant positions within the block at selected Transformer layers, achieving efficiency gains beyond DualCache.
propose ES-dLLM, training- frree inference
5.1 IMPORTANCE SCORE ESTIMATION
We estimate the importance tion. The score is calculated based on two criteria: First, mask tokens with higher confidence scores are more likely be selected for unmasking in subsequent iterations; thus, we prefer compute the results for these positions. Second, tokens that exhibit larger variations in intermediate tensors re- flect semantic or positional dependencies on newly generated tokens; updating these positions helps capture contextual changes and mitigate error accumulation.
of each position to determine which tokens retain for computa-
Formally, let the confidence score of position at iteration be denoted by o The intermediate tensor, selected as the variation indicator, which can be query, key, value tensors, or hidden states of position& in layer 1 iteration I, is denoted H(!) In layer 1, the importance of position i

---

Published as a conference paper at ICLR 2026
at iteration t is caleulated
[|
h = (t-1) a)
where is a hyperparameter that weighs two criteria and is the hidden dimension of The variation term is measured the LI-norm of the difference in indicator tensors across adjacent iterations, normalized by the L2-norm in the last iteration and by v to align the scale. In this work, we adopt the hidden state of each Transformer layer as the indicator tensor Section 6.3 also provides ablation study that compares the choices of the variation indicator.
5.2 PARTIAL CACHE UPDATE AND EARLY SKIP
Algorithm ES-dLLM Inference Process within Transformer Block
Require: Layer input x(I position set S, skip ratio TJ. caches of key, value, hidden states, and confidence Ck, Cv. CH. Cc. layer index I
Ensure: Layer output x ((++) on updated position set S', updated caches
1: Anorm + Normm((1)) (x)))
2: Qg, Ks, Vs Q-proj (xproj(Xnorni) Xnorm) K-ojj (Xnor)) ) V V-proj (Xoom)
3: Update caches Ck [kllls + Ks, Cvllls t- Vg for positions using scatter operation
4: K, Ck]1], [1] [v[[]
5: AttnOuts <- Attention Qe, K, V)
6: AttnOuts & AttnOuts + x(I)
7: Hg FFF((Nomm AttnOOtss) AttnOuts
8: Update cache CHllls + Hs
9: Initialize importance score list I
10: for position in E do
Calculate [i]] = It,4 using Equation
I with of (1=11) Ces. H
CH!!],, H H;
end for
13: Select tokens of top-k (k: = (1 - r: ( importance based on I, denote positions as S'
14: ((-11 + Hg.
return x(++1),3
We apply skipping for positions with low importance scores to reduce computational overhead; hence, only a subset of tokens is processed in each Transformer block. Therefore, we need to compute importance scores for position selection and update caches for subsequent usage. The inference process of a Transformer block with carly-skipping is outlined in Algorithm [] Given the input tensor (!) and the corresponding token position set S fed into layer 1, we first project the normalized input to query, key, and value tensors. We use them to update the KV cache for positions in S and then retrieve the full KV for attention. The attention output is passed through the feed- forward layer obtain hidden states Hg. which are also used calculate importance scores. We select the positions with top- (I--(1))) scores, where is the skip ratio that controls the fraction of skipped tokens layer 1. The output tensor, coupled with the updated position set, is propagated to the next layer.
The pseudo-code illustrates the case where hidden states are used the variation indicator for im- portance score calculation. If other tensors, such as query, key, or value, are chosen, the importance computation and cache update steps are adjusted accordingly.
The early-skip mechanism can reduce the size of intermediate tensors by ratio of rı, which propor- tionally decreases the computational cost of matrix multiplication in all subsequent layers. There- fore, skipping in the earlier layer could save more FLOPs, but the reliability of tensor variation is generally better in deeper layers. We discuss this trade-off in Appendix C.2
During generation, the cache is initialized by performing full forward pass for all prompt and output tokens. In subsequent iterations, the model is only fed with tokens of the current block for unmasking, applying carly-skipping in designated layers to skip unimportant positions and reduce overhead. To prevent error accumulation, we periodically refresh the cache for prompt tokens or the current block, where these tokens go through the entire inference process without skipping.

---

Published as a conference paper at ICLR 2026
EXPERIMENTS
6.1
EXPERIMENTAL SETUP
The experiments are conducted on an NVIDIA H200 GPU. We evaluate ES-dLLM on two repre- sentative diffusion LLMs: LLaDA-8B and Dream- 7B. Performance assessed in terms of both generation quality and throughput improvement across five benchmark datasets spanning diverse LLM tasks: GSM8K (Cobbe et al. (02T) and MATH (Hendrycks et al. 202T) for mathematical questions, HumanEval (Chen et al. 2021 and MBPP (Austin et 2021) for code generation, and BBH (Suzgun et al. 2022) for commonsense reasoning, All evaluations are carried out using the
LM-Eval framework (Gao et al. 2024).
Unless otherwise specified, the skip ratio in ES-dLLM set to 0.5 at positions of and 1/4 of all layers (i.e., F4 rs 0.5 for LLaDA and configuration reduces approximately 60% of total FLOPs during inference. The parameter a is set to 0.5, assigning equal importance to confidence and tensor variation. To mitigate error accumulation across iterations, we refresh the caches for all preceding tokens or all tokens in the current block with specific period, respectively, motivated by Liu et al. (202)). The settings of generation length and block length for each benchmark refer to those in LLaDA. And we use a batch size of 8 for better weight reuse and hardware utilization. More details are provided in Appendix
77 0.5 for Dream, while other r; 0). This
We compare ES-dLLM with two baseline methods. The first is the original implementation of LLaDA and Dream. The second is DualCache from Fast-dLLM (Wu al. 2025). which maintains KV caches for tokens outside the current block to allow cache reuse in the attention layer and refreshes the entire cache after processing each block.
6.2 MAIN RESULTS
Table 1: Performance comparison using LLaDA-8B- Insttuuuctttttttt five benchmark datasets. TPS (to- kens per second) measures throughput, calculated as the generated token count divided by total time. Speedup reports relative throughput improvement in terms of TPS compared to the vanilla imple- mentation (i.e., LLaDA). Performance score in percentage indicates the accuracy or pass rate. The number in parentheses of each benchmark indicates the number of shots for evaluation. Bold num- bers highlight the best performance on each benchmark.
Benchmark
Method
TPS
Speedup
Performance Score
LLaDA
8.56
1.0x
76.72
GSMBK(S)
DualCache
112.15
13.1x
76.35
ES-dLLM
143.93
16.8 x
76.95
LLaDA
14.04
1.0x
28.14
MATH(4)
DualCache
56.02
4.0%
26.94
ES-dLLM
103.63
7.4x
27.24
LLaDA
11.06
1.0x
56.75
DualCache
130.14
11.8x
53.29
BBH(3
ES-dLLM
159.89
14.5x
54.51
ES-dLLM
133.48
12.1
56.66
LLaDA
23.65
1.0x
36.59
HumanEval(O)
DualCache
176.85
7.5x
35.37
ES-dLLM
226.57
9.6x
37.8
LLaDA
8.98
1.0x
41
MBPP(3)
DualCache 117.85
13.1
38.4
ES-dLLLM
145.99
16.3x
39.4
Tables and present the performance comparison of ES-dLLM with the baselines LLaDA- 8B-Instruct and (ream-7B-Instruce respectively. ES-dLLM achieves substantial efficiency gains, with speedups ranging from 5.6x to 16.8x over the original implementations and from 1.20x to

---

Published as a conference paper at ICLR 2026
Table 2: Performance comparison using -nnstrt-- on five benchmark datasets. Metrics and notations are the same as in Table
Benchmark
Method
TPS
Speedup
Performance Score
Dream
19.80
1.0x
79.00
GSM8K(5)
DualCache
209.88
10.6x
77.94
ES-dLLM
267.13
13.5
77.94
Dream
26.38
1.0x
33.94
MATH(4)
DualCache
86.38
3.3x
33.60
ES-dLLM
147.44
5.6x
33.44
Dream
24.84
1.0x
61.48
DualCache
226.89
9.1x
58.27
BBH(3)
ES-dLLM
292.23
11.8x
57.84
ES-dLLM
196.50
7.9x
60.36
Dream
44.34
1.0x
46.95
HvmlnEvva(O))
DualCache
258.10
5,8x
45.12
ES-dLLM
308.51
7.0x
45.12
Dream
21.68
1.0x
60.4
DualCache
214.14
9.9x
56.8
MBPP(3)
ES-dLLM
276.12
12.7x
ES-dLLM#
130.49
6.0x
59
1.85 over DualCache. In terms of generation quality, ES-dLLM generally matches DualCache and surpasses it on several benchmarks. This result suggests that frequent updates of larger set of tokens may not always be beneficial and may introduce noise. These findings highlight both the computational redundancy inherent during dLLM generation and the effectiveness of ES-dLLM's early-skip strategy.
Moreover, observe that DualCache suffers from degraded accuracy the BBH and MBPP datasets compared to the original implementation, which we attribute to error accumulation in the KV cache of the prompt region. To address this, we evaluate ES-dLLM with KV cache refreshment for prompt tokens (i.c., multiple times within each block), denoted as ES- dLLM As reported in the tables, this adjustment effectively mitigates the accuracy drop while retaining decent speedups. Overall, ES-dLLM, delivers speedup of at least 5.6x, with accuracy differences ranging between -1.83 and +1.21 compared to the vanilla implementations, demon- strating both the efficiency and robustness of ES-dLLM in accelerating diffusion LLM inference.
frequent
6.3 ABLATION STUDY
Effect of a in Importance Score Estimation. The parameter a in Equation iss used to weigh the contributions of confidence and variation in importance scores. We evaluate different a values on LLaDA-8B-Insee
across multiple benchmarks, with the results shown in Figure 4a It indicates that combining both factors (a 0.5) achieves the best overall performance, while relying on one of them leads to noticeable accuracy degradation, particularly when using the confidence score alone (a 1). which likely stems from neglecting contextual updates from previously generated tokens. Interestingly, = (i.e., using only variation scores) performs well on the MATH dataset, even surpassing the original baseline, suggesting that the generation process may not strictly follow the confidence order for this task.
Selection of Intermediate Tensors for Variation Indicator. our main experiments, we use hid- den states as the variation indicator: To evaluate the sensitivity of this choice, we compare alternative indicators, including key, query, and value tensors, in Figure 4b The results show that generation quality generally robust to the choice of variation indicator. Hidden states provide slightly bet- ter performance overall, but key and value tensors achieve comparable results with lower memory overhead, offering practical alternative.
8

---

Published as a conference paper at ICLR 2026
100
100
0=0.0
Hidden
90
g-0.5
Performance (%)
Querr
Value
Performance Score
GSMBK
MATH
BBH
HumannEel
MВPP
GSMBK
MATH
BBH
HumanEval
MВPP
Datasets
Datasets
(a) Effect of a.
(b) Effect of variation indicator.
Figure 4: Ablation studies on importance estimation configurations using LLaDA-8B -nsttrutt
Additional analyses, including results on Base models, ablations on skip ratio and position, and the compatibility of ES-dLLM with sparse attention and parallel decoding techniques, are presented in Appendix C
7 DISCUSSION
Memory Overhead of ES-LLLM. ES-dLLM maintains caches for key, value, and variation indica- tor (e.g.. hidden state) tensors for each token. The cache the variation indicator only needed for output positions in layers where skipping applied. Therefore, the additional memory overhead of ES-dLLM 528KB for LLaDA-8B and 70KB for Dream-7B per output token in BF16 format, of which hidden states contribute only 16KB and 14KB, respectively. Considering sample with 1024 prompt tokens and a generation length of 256, the total memory overhead of ES-dLLM 644MB for LLaDA-8B and 73.5MB for Dream-7B per sample. This cost is modest and with negligible over- head beyond the KV caches, which acceptable for modern GPUs with large memory capacities, especially in comparison the over 10GB required for model weights.
Speedup Potential of ES-dLLM. With the chosen skip configuration. ES-dLLM reduces FLOPs by approximately 60% compared to DualCache. However, the observed speedup ranges from 1.20x to 1.85x, smaller than the theoretical computation reduction. This discrepancy can be attributed to the fact that LLM inference with fewer tokens (c.g., the decode phase autoregressive LLM) is often
rather than compute-bount According to the roofline model (Williams et al. 2009), efficiency is constrained by memory bandwidth and operational intensity. While ES-dLLMM lowers FLOPs by skipping computation, memory accesses for model weights and KV tensors re- main largely unchanged, shifting the workload from compute-bound to memory-bound. Therefore, this provides opportunities for system-level optimizations (Patel et al. 2024 Agrawal et al.) 2024) that better integrate compute-bound and memory-bound workloads, potentially unlocking the full speedup potential of ES-dLLM.
Limitation and Future Work. Although ES-dLLM effectively exploits redundancy in token com- putation, the estimation of the importance score relies simple heuristics, and partial KV updating diverges from the training process for dLLM that assumes complete state updates. Future work could explore lightweight model to predict token importance, adaptively adjusting skip ratios based on real-time variation, and investigating training strategies that align with the early-skipping mechanism fur- ther enhance both efficiency and generation quality.
sophisticated importance estimation and skipping methods, such training
8 CONCLUSION
In this work, we propose ES- ES-LLLM, a training-free free inference acceleration framework for diffusion large language models that reduces computational overhead by selectively skipping redundant com-

---

Published as a conference paper at ICLR 2026
putations in early layers. ES-dLLM achieves up to 16,8x speedup over the original implementation and up to 1.85x speedup compared to the state-of-thh-an DualCache method on LLaDA-8B and Dream-7B models, while maintaining comparable generation quality. Our results reveal redundancy in token computation during dL.LM generation, and ES-dLLM provides a practical direction to op- timize the inference process of dI.LM.
REPRODUCIBILITY STATEMENT
For reproducibility, we open-source our code in htttttts//gtthhuitt http://thhuu.ccmm//u////////// dLLM accompanied by README file with detailed instructions. The experimental settings are described in Section aan Appendix B.I In addition, we provide bash scripts in the supplementary material for reproducing the main results reported in this paper.
REFERENCES
Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Taming shoughput-latencyy tradeoff in Ilm infer- with sarathi-serve. In 18th USENIX Symposium on Operating Systems Design and Imple- mentation (OSDI 24), pp. 117-134, 2024.
Marianne Arriola, Aaron Gokaslan, Justin T. Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Sub- ham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autore- gressive and diffusion language models. In The Thirteenth International Conference Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, 2025.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski. David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoe Le, al. Program synthesis with large language models. arXiv preprint arXiv: 102.000..0002222 077332, 2021.
Yihan Cao, Siyu Li, Yixin Liu, Zhiling Yan, Yutong Dai, Philip S Yu, and Lichao Sun. com- prehensive survey of ai-generated content (aigc): A history of generative ai from gan to chatgpt. arXiv preprint arXiv:2003.0422 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin. Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. arXiv preprint /00.:22007.0337.. 2021.
Xinhua Chen, Sitao Huang, Cong Guo, Chiyue Wei, Yintao He. Jianyi Zhang, Hai "Helen" Li, and Yiran Chen. DPad: Efficient Diffusion Language Models with Suffix Dropout. arXiv e-prints, art. arXiv:2508. 14148, August 2025. doi: 10.48550/arxiv25000 14148.
Yanxi Chen, Xuchen Pan, Yaliang Li, Bolin Ding, and Jingren Zhou. Ee-lim: large-scale training and inference of early-exit large language models with 3d parallelism. In Proceedings of the 41st International Conference on Machine Learning, pp. 7163-7189, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano. Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint 1..iv:2110.1416. 2021.
Deepmind.
Gemini diffusion, 2025.
gemi ni -i f ffusion]
10

---

Published as a conference paper at ICLR 2026
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024.
Dan Hendrycks. Collin Burns. Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
Inception Labs, Samar Khanna, Siddhant Kharbanda, Shufan Li, Harshit Varma, Eric Wang, Sawyer Birnbaum, Ziyang Luo, Yanis Miraoui, Akash Palrecha, et al. Mereury: Ultra-fast language models based on diffusion. arXiv preprint (rXXi:2506.17290 2025.
Zhiyuan Liu, Yicun Yang, Yaojie Zhang, Junjie Chen, Chang Zou, Qingyuan Wei, Shaobo Wang, and Linfeng Zhang. dllm-cache: Accelerating diffusion large language models with adaptive caching, arXiv preprint (rxxi:2500600295, 2025.
Xinyin Ma, Rumpeng Yu, Gongfan Fang, and Xinchao Wang. dkv-cache: The cache for diffusion language models. arXiv preprint urXiv: 15781, 2025.
Shen Nic, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint arXiv:2502.09992, 2025.
Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Iñigo Goiri, Saeed Malcki, and Ri- cardo Bianchini. Splitwise: Efficient generative I1m inference using phase splitting. In 2024 ACM/EEE 5/st Annual International Symposium on Computer Architecture (ISCA), pp. 118- 132. IEEE, 2024.
Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. In D. Song, M. Carbin, and Chen (eds.), Proceedings of Machine Learning and Systems, vol- ume 5, pp. 606-624 624. Curan, 2023.
Yuerong Song, Xiaoran Liu, Ruixiao Li, Zhigeng Liu, Zengfeng Huang, Qipeng Guo, Ziwei He, and Xipeng Qiu. SLa: Accelerating Diffusion LLMs with Dynamic Cache Eviction. arXiv e-prints, art. arXiv:2508.0255 025558, August 2025. doi: 10.48550/axxi.2250..022 025558.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, , and Jason Wei. Challenging big- bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.
Samuel Williams, Andrew Waterman, and David Patterson. Roofline: an insightful visual perfor- mance model for multicore architectures. Communications of the ACM, 52(4))65-76, 2009.
Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao. Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. Fast-dllm: Training- free acceleration of diffusion 1lm by enabling kv cache and parallel decoding. arXiv preprint arXiv:2505.2260 2025.
Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b: Diffusion large language models. arXiv preprint arX1v:2508.15487, 2025.
Runpeng Yu, Li, and Xinchao Wang. Discrete diffusion in large language and multimodal models: A survey, arXiv preprint Miiv:2506.13754 2025.
11

---

Published as a conference paper at ICLR 2026
A ADDITIONAL EXPERIMENTS ON DLLM GENERATION CHARACTERISTICS
A.1 INTERMEDIATE TENSOR VARIATION IN LLADA
In addition to hidden states, we also examine the variation of other intermediate tensors, namely the key, value, and query tensors inside the attention. As shown in Figure 5 their variations relatively small for most tokens in most iterations, consistent with the observations on hidden states discussed in Section
We further plot the variation distribution of hidden states in other layers in Figure 6 which shows exponential distributions those observed for layer 10 in the main text. We perform normalization for values greater than I, producing a slight spike at I in the distribution for layer 30. The variation tends to increase in deeper layers, likely because of multiple interactions among tokens in attention, which can amplify variation in intermediate results. Nevertheless, even in the layer approaching the end, large portion positions still exhibit small variations.
Key Diff. Heatmap
Key Diff. Log Frequency
201
331
204
430
exc
31M
8.2
1.3
Value Diff. Heatmap
Velue Diff. Log Praquency
13
167
terottnn
234-
EEC
Fretuancy
201-
63
19
Query DIff. Heatmap
Query Diff. Log Prequency
10
334.
234
Postich
Figure 5: Variation statistics of key, value, and query tensors in layer 10 using LLaDA-8B-Instruue Left: mange-ssamle heatmap from BBH; red line separates prompt and output tokens. Right: log- scale distribution for output tokens using 100 samples from multiple datasets.
Layer
Layer
Layer
Hidden State
hioden State
55
18:den State DMY
Figure 6: Log-scale distributions of hidden state variation layers 10, 20, and 30 using LLaDA- 8B-Instruct.
A.2 EXPERIMENTS GENERATION CHARACTERISTICS IN DREAM
We perform the same set of experiments on Dream-7B- In-struttt to validate the generality of the ob- servations presented Section 4 The results are shown in Figures 7 Jndd The confidence change and intermediate tensor variation are slightly larger, but follow trends similar to those observed in LLaDA-8B- -LnstssB-Instrucct, demonstrating that the characteristics are consistent across different diffusion LLMs. For the specific sample presented in the heatmaps, variation at the iteration corresponding to the generation of the EOS token, which also influences the intermediate tensor variation in subsequent iterations.
observe pronounced confidence
12

---

Published as a conference paper at ICLR 2026
101
Confidence Ditt.
Iteration
10-4
201
10
251
50
1000
150
200
250
0.0
0.2
0.4
0.6
0.8
1.0
Position
Confidence Diff.
(a) Single-sample heatmap.
(b) Log-scale distribution.
0.30
0.25
Proportion
0.15
0.05
51
101
152
201
251
Iteration
(c) Proportion of variation 0.05 at each iteration.
Figure 7: Confidence variation statistics using Dream- -eeam-BB-une 77B-Instruct (a) uses sample from the BBH dataset, while (b) and (c) present results using 100 samples from multiple datasets.
Hidden State Diff. Heatmap
Hidden State Dift. Fraquency
1100en Slate
Kay Heatmap
Key Diff. Frequency
251
ste
890
1b6c
1200
14
Vallue Diff. tmop
Value Difl. Log Froquency
Pusidion
Volue Diff.
DifT. Hestmap
Diff. Frequency
Figure 8: Variation statistics hidden states and key, query, and value tensors in layer 10 using Dream-7B-Instre Left: single-sample heatmap from BBH; red line separates prompt and output tokens. Right: log-scale distribution for output tokens using 100 samples from multiple datasets.
A.3 CORRELATION BETWEEN VARIATIONS OF INTERMEDIATE TENSOR AND CONFIDENCE
We further analyze the relationship between intermediate tensor variation and confidence difference using 100 samples from multiple datasets. As shown in Table 3 the correlation coefficients range from 0.15 to 0.66. reflecting a positive correlation. The correlation tends to be stronger in later layers, which expected since the final predictions rely the hidden states of the last layer. How-
13

---

Published as a conference paper at ICLR 2026
Table 3: Pearson correlation between intermediate tensor variation and probability change (confi-
dence is the maximum probability) for all mask tokens in selected layers using LLaDA-8B -B-struutt..
Layer
4
8
16
24
31
Hidden
0.15
0.22
0.27
0.38
0.48
0.51
Query
N/A
0.18
0.24
0.39
0.45
0.66
Key
N/A
0.18
0.24
0.38
0.42
0.66
Value
N/A
0.16
0.25
0.43
0.45
0.65
ever, applying skipping in earlier layers could save more computation, creating a trade-off between efficiency and the reliability of tensor variation an indicator. We further discuss this trade-off by evaluating different skipping positions Appendix C.2 Notably, the correlation is not applicable for query, key, and value tensors in layer O. they are directly projected from the input embeddings without inter-token interaction, thus only differ in the newly generated tokens.
B EXPERIMENT DETAILS
B. PARAMETER SETTINGS
Table 4: Generation length and block length for each benchmark.
Benchmark
Generation Length
Block Length
GSM8K
256
64
MATH
256
256
BBH
256
64
HumanEval
512
MBPP
512
64
In addition to the parameters described in the main text, we provide more details of the experimental settings ensure reproducibility. Following Nie et al. (2025), we adopt their choices of genera- tion and block lengths for each benchmark, with some adjustments. The specific configurations are listed in Table 4 Although Dream does not inherently employ semi-autoregrressvve generation strategy, we found that constraining the generation order with a block length can improve perfor- mance. Therefore, we adopt the semi-autoregressive block setting for both models.
Table 5: Cache refresh periods in ES-dLLM for each benchmark and model.
(a) Instruct
(b) Dream-78B -B-nsrrutttt
Benchmark
Prompt
Block
Benchmark
Prompt
Block
GSM8K
64
16
GSM8K
64
8
MATH
256
8
MATH
256
4
BBH
64
4
BBH
64
8
HumanEval
64
HumanEval
64
2
MBPP
64
4
MBPP
256
2
Table Cache refresh periods in ES-dLLM* for each benchmark and model.
Model
Benchmark
Prompt
Block
LLaDA
BBH
4
BBH
16
4
Dream
MBPP
2
14

---

Published as a conference paper at ICLR 2026
Moreover, mentioned in the main text, periodically refresh the cache for the prompt tokens for the entire current block during generation. We employ different cache refresh frequency settings for each benchmark and model to achieve better performance, listed in Table On the BBH and MBPP datasets, we observe a performance gap between DualCache and the original implementation. This difference arises because DualCache intrinsically applies prompt refresh period equal the block size and a block refresh period of 1, whereas the original implementation updates both at every step (i.e., period of 1). To mitigate this accuracy loss, we increase the prompt refresh frequency in ES ES-dLLM, resulting in variant denoted as ES-dLLM* in the main text. The specific refresh period configurations of E are listed T
For the sampling strategy, we adopt low-confidence remasking for LLaDA and maskgit-plus with top-k (A: = 50) and top-p (p = 0.95) sampling for Dream. The temperature set to 0 for both mod- els. For evaluation metrics to get the performance score of each dataset, we use exa exactuma cc for filter), math_veri Fy metric for MATH, and pass_at_1 for the two coding datasets HumanEval and MBPP.
GSM8K and BBH (with GSM8K using the flexible-exte
B.2 IMPLEMENTATION DETAILS
We implement ES-dLLM and the DualCache baseline top of the open-source codebases of LLaDA (Nie et al. 2025) and Dream (Ye et al.] 2025). To enable the early-skipping mechanism, modify several modules, including tensor caching, importance score computation, and position selection logic. For LLaDA, we extend the framework to support batch inference. KV caching is ap- plied after the RoPE operation reduce the computation overhead. We notice that operating on full logits consumes a significant portion of memory during generation, especially for top-k sampling. To alleviate this issue, we truncate the logits of prompt tokens before sampling. In addition, we ob- serve that the EOS token is sometimes generated prematurely, producing incomplete outputs across all baselines. To address this, we disallow EOS generation when the last token is still a mask token to improve stability, which we found beneficial for the performance, especially on coding datasets.
C SUPPLEMENTARY EXPERIMENTS
C.1 MAIN RESULTS ON BASE MODELS
Table Performance comparison using LLaDA-SB-Barl on five benchmark datasets. Metrics and notations follow Table
Benchmark
Method
TPS
Speedup
Performance Score
LLaDA
9.20
1.0x
71.11
GSM8K(5)
DualCache
116.46
12.7x
66.19
ES-dLLM
140.45
15.3x
67.63
LLaDA
15.12
1.0x
32.38
MATH(4)
DualCache
57.89
3,8x
30.48
ES-dLLMM
94.46
6.2x
30.60
LLaDA
11.68
1,0x
45.26
BBH(3)
DualCache
133.95
11.5x
43.59
ES-dLLM
162.30
13.9
44.08
LLaDA
23.84
1.x
32.32
HumanEval(O)
DualCache
177.40
7.4x
32.32
ES-dLLM
232.66
9,8x
31.71
LLaDA
9.53
1.0x
39.6
MBPP(3)
DualCache
120.12
12.6x
38.6
ES-dLLM
158.81
16.7x
38
We further evaluate ES-dLLM using the base models LLaDA-8B. -B-D-----Base and Dream- (a-rn--B-Basee across five benchmark datasets, with results reported in Tables 7 and 8 ES-dLLM achieves speedups
15

---

Published as a conference paper at ICLR 2026
Table 8: Performance comparison of Dream-7B- on five benchmark datasets.
Benchmark
Method
TPS
Speedup
Performance Score
Dream
21.12
1.0x
74.15
GSM8K(5)
DualCache
211.67
10.2x
73.16
ES-dLLM
282.67
13.4x
73.92
Dream
27.98
1.0x
40.90
M4TH(4)
DualCache
86.58
3.1x
40.10
ES-dLLM
167.03
6.0x
39.60
Dream
25.94
1.0x
49.65
BBH(3)
DualCache
228.09
8.8x
45.34
ES-dLLM
281.65
10.9
44.28
Dream
44.26
1.0x
42.07
(uma)Eval
DualCache
257.42
5.8x
40.24
ES-dLLM
305.37
6.9
38.41
Dream
22.73
1.0x
55
MBPP(3)
DualCache
216.09
9.5x
56.2
ES-dLLM
301.43
13.3x
55.2
of up to 16.7x and 1.93x over the original implementations and DualCache, respectively, while maintaining comparable performance scores, demonstrating the effectiveness and generality ES- dLLM on base models.
C.2 ABLATION STUDY ON SKIP RATIO AND POSITION
Table 9: Ablation study of skip ratio and position configurations on MATH using LLaDA-8B- Instruct. No skipping represents the DualCache baseline. Speedup is calculated against the TPS of DualCache, and the FLOPs proportion is normalized to the no-skipping baseline.
Skip Ratio & Position
FLOPs Prop.
TPS
Speedup
Performance Score
No skipping
100%
56.02
1,0x
26.94
14 rs 0.5
40%
103.63
1.85x
27.24
rg 0.75
46%
95.26
1.70x
27.26
= 0.5
64%
77.66
1.39x
27.36
rs 0.25
82%
63.49
1.13x
27.58
To = 0.5
52%
89.40
1.60
26.76
Fa 0.5
58%
83.17
1.48
27.28
7в = 0.5
64%
77.66
1.39x
27.36
F16 0.5
77%
68.50
1.22x
27.52
Table 10: Ablation study on skipping times across five datasets using JLaDA-8B-Instruuc
Skip Ratio& Position
FLOPs Prop.
GSM8K
MATH
BBH
HumanEval
MBPP
0.7
40%
76.27
27.44
53.95
35.98
39.4 .4
F4 = rg = 0.5
40%
76.95
27.24
54.51
37.8
39.4
Ts T12 0.405
40%
76.35
27.48
53.86
36.59
39.8
In the main experiments, apply early-skipping two positions (1/8 and 1/4 of all layers) with skip ratio of 0.5. To further investigate the impact of the skip ratio and skip position, we evaluate range of configurations, with results reported in Tables and T0
16

---

Published as a conference paper at ICLR 2026
As shown in T varying the skip ratio at fixed position reveals trade-off between efficiency and generation quality. Larger ratios yield greater speedup by saving more computation, but also adversely affect performance. A similar trade-off is observed for skip position: applying skipping in earlier layers brings greater speedup, but leads to performance degradation due to insufficient intermediate information.
We also examine the effect of applying different times of skipping while keeping the overall FLOPs proportion roughly the same (about 40%). As shown in Table [T]] skipping two positions (r4 0.5) achieves the best balance, progressively refining token selection and avoiding overly aggressive pruning in early layers. In contrast, applying too many skips could reduce the number of remaining tokens in the final layer, which is also detrimental to performance.
C.3 INTEGRATION WITH EXISTING METHODS
Performance comparison for parallel decoding using LLaDA-8B-Instur Speedup is compared with the DualCache baseline without parallel decoding.
Table 11:
Benchmark
Method
TPS
Speedup
Performance Score
DualCache+PD
172.02
1.53x
76.57
GSM8K(5)
ES-dLLM+PD
201.60
1.80
75.74
DualCache+PD
85.05
1.52x
26.94
MATH(()
ES-dLLM+PD
152.19
2.72x
27.48
DualCache+PD
269.48
2.07x
52.31
BBH())
ES-dLLM+PD
302.95
2.33x
53.26
DualCache+PD
271.64
1,54x
34.76
HumanEEal(O)
ES-dLLM+PD
349.26
1.97
37.8
DualCache+PD
367.82
3.12x
38.4
MBPP(3)
ES-dLLM+PD
413.82
3.51x
37.8
Table 12: Performance comparison for parallel decoding using Dream-7B-Instrre
Benchmark
Method
TPS
Speedup
Performance Score
DualCache+PP
364.48
1.74x
77.94
GSM8K(5)
ES-dLLM+PD
425.27
2.03x
78.01
DualCache+PD
128.02
1.48x
33.58
MATH(4)
ES-dLLM+PD
215.61
2.50
33.48
DualCache+PD
546.51
2.41 x
57.18
BBH(3)
ES-dLLM+PD
615.51
2.71
57.07
DualCache+PD
513.85
1.99
44.51
HumanEval(O)
+S-
710.33
2.75x
46.34
DualCache+PD
829.71
3.87
56.8
MBPP(3)
ES-dLLM+PD
1469.15
6,86
57.4
As mentioned in Section 3 ES-dLLM orthogonal and complementary to existing acceleration techniques, including parallel decoding (Wu et al.) 2025) and sparse attention (Song et al. 2025). In this section, we explore the compatibility and try to integrate ES-JLLM with these methods.
C.3.1 INTEGRATION WITH PARALLEL DECODING
In the main text, evaluate ES- under the one-token serestpp generation scheme. How- ever, dLLM has the potential to generate multiple tokens in parallel at each iteration, offering fur- ther acceleration. To explore this, we integrate ES- with confidence-aware parallel decoding
17

---

Published as a conference paper at ICLR 2026
(PD) (Wu al. 2025), using confidence threshold of 0.9. The cache refresh period settings for ES ES-dLLM follow the same configuration as Table
Tables 11 and 12 present the results. We find that ES-dLLM can be seamlessly integrated with parallel decoding, delivering additional speedups of up to 1.79x for LLaDA and 1.68x for Dream compared to DualCache with parallel decoding, while maintaining comparable performance scores. These results confirm that ES-dLLM fully compatible with parallel decoding and can further enhance the efficiency of diffusion LLM inference.
C.3.2 INTEGRATION WITH SPARSE ATTENTION
Table 13:
Performance comparison for sparse attention using LLaDA-8B-Instrucc.. Speedup is com-
pared to DualCache without sparse attention technique.
Benchmark
Method
TPS
Speedup
Performance Score
Sparse-dLLM
140.26
1.25
76.80
GSM8K(5)
SS-dLLMsSparse
172.55
1.54x
75.97
Sp
66.80
1.19x
28.10
MATH(4)
SSprse
121.91
2.18x
28.06
Sparse-dLLM
160.35
1,23x
49.13
BBH(3)
MS-dLLLM+Sparse
191.56
1.47x
51.13
Sparse-dLLMM
207.95
1.18x
35.37
(umanEval())
ES-dLLMMSparree
261.63
1.48x
34.76
Sparse-dLLM
145.77
1,24x
40.2
MBPP(3)
ES-dLLM+Sparse
179.57
1.52x
38.8
Table 14: Performance comparison for sparse attention using Dream-7B-Instruct
Benchmark
Method
TPS
Speedup
Performance Score
Sparse-dLLM
227.68
1.08 x
77.94
GSM8K(5)
ES-dLLM++ppare
287.68
1.37 x
77.94
Sparse-dLLM
89.56
1.04
35.60
MATH(4)
MS-asee
157.19
1.82x
36.10
Sparse-dLLM
243.35
1.07
57.52
BBH(3)
M-
305.35
1.35x
57.09
Sparse-dLLM
270.53
1.05
45.73
HumanEval(O)
SSparee
322.05
1.25x
46.95
Sparse-dLLM
233.86
1.09
56.6
MBPP(3)
-S-dLLM
309.74
1.45
57.2
Sparse-dLLM (Song et al., 2025) another technique that accelerates dLLM inference by pruning KV caching of less important tokens from being attended in the attention operation, while ES- dLLLM focuses on reducing the number of tokens going through inference using early skipping. To investigate their compatibility, integrate ES-dLLM with sparse attention, using the settings from the original paper: a retention ratio 0.5, a kernel size of 3, and a delayed step of I.
The results in Tables and 14 show that ES-dLLM can be effectively combined with sparse atten- tion, achieving additional speedup while maintaining comparable performance, further validating the flexibility of ES-dLLM. Sparse-dLLM achieves less significant speedup Dream-7B-Iin -nIstrrcct compared to LLaDA-8B -B---BIttuct,, likely due to Dream's 's GQA mechanism that already reduces KV cache overhead in attention, and the proportion of attention in the overall inference time smaller.
18

---

Published as a conference paper at ICLR 2026
Table 15: Performance ES-dLLM when combined with both parallel decoding and sparse atten- tion.
Benchmark
GSM8K
MATH
BBH
HumanEval
MBPP
LLaDA- -A---A-----mmae Instructt
TPS
241.03
175.53
331.30
407.86
483.35
(vs. DualCache)
2.15x
3.13x
2.55x
2.31
4.10x
Performance
74.30
28,46
52.17
34,76
38.6
(vs. DualCache)
-2.05
+1.52
-1.12
-0.61
+0.2
Dream- 7B-nn--B--ssttt
TPS 459.63
228.00
645.27
763.21
1618.92
(vs. DualCache)
2.19x
2.64x
2.84x
2.96x
7.56x
Performance
77.63
35.64
56.47
45.12
56.8
(vs. DualCache)
-0.31
+2.04
-1.8
0
0
C.3.3 INTEGRATION WITH BOTH METHODS
Moreover, ES-dLLM can be integrated with both parallel decoding and sparse attention simultane- ously to achieve full speedup potential. As shown in Tables [5] this combined approach achieves speedups of 2.15-4.100 10x on -LaDA-8B-Instruu and 2.19-7.56x on Dream-7B-Instruc relative to the DualCache baseline, while maintaining competitive performance scores. These findings demon- strate that redundancy in vanilla dLLM inference can be effectively exploited from multiple dimen- sions, and ES-dLLM serves as versatile component that can be seamlessly combined with other acceleration techniques to enable simple and efficient deployment.
D THE USE OF LARGE LANGUAGE MODELS
In preparing this paper, we employed LLMs to assist with paper writing. Specifically, GitHub Copilot was used help with some word and sentence drafting, and ChatGPT was used refine and polish the language.
19


Reviews:
[{"review_id": "N0GAEowAmO", "reviewer": "Reviewer_ybg1", "summary": "This paper proposed an efficient inference technique for diffusion LLM. This method leverage intermediate tensor variation and confidence\nscores to determine which token to skip. Experiments show significant speedup while maintaining the task performance.", "questions": "- Roughly how many tokens are generated in each dataset? Do you know why some datasets show more speedup while others show less?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This work propose a method well-substantiated by empirical findings.\n- Consistent and significant speedup across all the datasets tested.", "weaknesses": "- The experiment session only compared to two other methods. Are there any other methods addressing the same problem?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rSz5O2JBeR", "reviewer": "Reviewer_id9o", "summary": "Diffusion large language models (dLLMs) have bidirectional context and parallel generation advantages but suffer from high inference latency due to full-sequence computation per iteration. ES-dLLM, a training-free framework, addresses this by leveraging the observation that most tokens’ intermediate states (hidden states, KV tensors) and confidence scores vary slightly across iterations. It estimates token importance via prior confidence and normalized tensor variations, skips low-importance tokens in early layers, updates caches only for selected tokens (with periodic refreshes to avoid errors), and achieves 5.6×–16.8× speedup over original dLLMs (LLaDA-8B, Dream-7B on NVIDIA H200) and 1.2×–1.85× over DualCache, while maintaining generation quality; it also integrates with parallel decoding for further speedups.", "questions": "None", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "+ **Training-free design for easy deployment**: ES-dLLM requires no model fine-tuning or structural modification. It only optimizes the inference process through early-skipping and partial cache updates, which can be directly integrated into open-source dLLMs (e.g., LLaDA, Dream) without reconstructing the underlying framework.", "weaknesses": "+ **Fixed heuristic for importance score, lacking adaptability**：The importance score of ES-dLLM relies on a fixed linear weighting of prior confidence and intermediate tensor variation (default α=0.5), without considering task-specific differences. Ablation experiments show that using only tensor variation (α=0) performs better on the MATH dataset than the default α=0.5, while relying solely on confidence (α=1) leads to noticeable quality degradation. However, ES-dLLM does not propose a dynamic adjustment mechanism (e.g., adapting α based on real-time token variation or task type), which may cause misjudgment of early-skipping in complex scenarios.  \n+ **Missing validation in extreme scenarios**：\n  + **Ultra-long sequence generation**：The experiments only cover sequence lengths up to 512 tokens (HumanEval, MBPP) and do not test ultra-long sequences (>1K tokens). It remains unproven whether ES-dLLM’s cache refresh frequency and accumulated memory overhead (e.g., hidden state cache) can be controlled in longer sequences. \n  - **Comparison with more SOTA methods**：The baseline only includes the original dLLM implementation and DualCache, without comparing with cross-paradigm acceleration schemes (e.g., D2Cache for dLLMs, ReSA for sparse attention), making it impossible to fully reflect ES-dLLM’s relative advantages in the broader dLLM acceleration field.  \n+ **Insufficient analysis of the rationality of early-skipping layer selection**：ES-dLLM only tests early-skipping at \"1/8 and 1/4 of all layers\" (e.g., layers 4 and 8 for LLaDA) but does not explain why these layers are optimal. Although ablation experiments involve different layer configurations (e.g., layer 16, layer 0), they fail to quantify the trade-off between \"layer position, efficiency, and quality\" or provide general guidelines for layer selection, which is unfavorable for adapting ES-dLLM to other dLLMs with different depths.  \n+ **Incomplete details on memory overhead**：While the paper mentions that memory overhead is controllable (≤644MB per sample for LLaDA-8B), it does not report the memory growth curve under different sequence lengths or compare memory usage with DualCache. This makes it impossible to evaluate ES-dLLM’s applicability in memory-constrained scenarios (e.g., edge devices with limited VRAM).  \n+ **Unverified adaptability to ultra-large dLLMs**：The experiments are only conducted on mid-sized models (LLaDA-8B, Dream-7B) and do not involve ultra-large dLLMs (70B+ parameters). The dynamics of KV caches and intermediate states in ultra-large models may differ significantly from mid-sized ones, and it is unclear whether ES-dLLM’s early-skipping strategy can maintain efficiency and quality in such models.\n+ **Rough graphical and textual expression**: The paper’s data visualization and textual description of figures lack clarity and detail, affecting result interpretability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rZr5kE7wZ7", "reviewer": "Reviewer_dw3w", "summary": "This paper proposes ES-dLLM for efficient inference for diffusion LLMs. The motivation is clear for slow DLLM inference, and similarly for activations during different diffusion steps. The idea of skipping low-importance tokens in early layers is effective. Results show up to 16.8× speedup with negligible accuracy drop.\n\nThe paper is well organized, experimental results are extensive, and the method is training-free, which leads to weak accept.", "questions": "How ES-dLLM perform on MMLU?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "The motivation is strong that the hidden-state variation statistics convincingly demonstrate redundancy. We believe it has been observed in diffusion video generation models and image generation models [1], but it is new in DLLM. This paper clearly demonstrates its motivation. I like that.\n\nCompromising generation quality is important for real-world applications, which helps a lot for this paper.\n\nWe believe training-free is essential for effective inference, which this paper achieves.\n\n\n[1] Silveria A, Govande S V, Fu D Y. Chipmunk: Training-Free Acceleration of Diffusion Transformers with Dynamic Column-Sparse Deltas[C]//ES-FoMo III: 3rd Workshop on Efficient Systems for Foundation Models. 2025.", "weaknesses": "Novelty risk: This method appears to be an extension of the original implementation and DualCache. Fortunately, DualCache is not a sparsity work, which distinguishes the two works, but we hope to see more discussion on the degree of differentiation. Also, as we claimed in the Strengths, a similar idea has been proposed in the traditional diffusion model. \n\nComparison and related work: The paper compares only with the original method and DualCache, limiting the scope of the comparison. However, this paper only talks about PRELIMINARY with DLLM, DualCache, and does not talk about related work (I can't see any section on related work) using a lossy method to accelerate DLLM. No related work and no comparison with related work are the main problems of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G59M6ERkm3", "reviewer": "Reviewer_6Gok", "summary": "To solve the high inference latency of diffusion-based LLMs (dLLMs), caused by full-sequence computation in each iteration, the paper proposes ES-dLLM, a training-free acceleration framework. It is built on two key observations: (1) Intermediate states (hidden/key/value tensors) of most tokens have subtle variations across consecutive iterations; (2) Confidence scores of token positions change minimally. ES-dLLM implements two core components: Importance Score Estimation and Partial Cache Update & Early Skip. Experiments on LLaDA-8B/Dream-7B (5 benchmarks: GSM8K/MATH/BBH/HumanEval/MBPP)  show ES-dLLM achieves speedup over vanilla dLLMs.", "questions": "1. How does ES-dLLM perform on 13B/70B dLLMs (e.g., Dream-13B)? Does the overhead of importance score calculation (L1-norm of hidden states) scale linearly with the number of layers/heads? Can lightweight optimizations (e.g., sparse hidden state comparison, only checking top-20% varying dimensions) reduce this overhead? Please provide TPS/accuracy data for Dream-13B across GSM8K/MATH and analyze layer-wise computation time.\n2. For 1k/2k-token sequences (e.g., LongBench benchmarks), how to adjust skip positions and ratios dynamically? Does a variation-aware skipping strategy maintain accuracy while increasing speedup? Please compare fixed vs. dynamic skipping on 1k-token PubMed abstracts (generation length 512) in terms of FLOP reduction and performance on benchmarks.\n3. How to adjust prompt/block cache refresh periods dynamically based on sequence type? For example, legal documents (long sentences) may need shorter prompt refresh periods (16 vs. 64) to retain context, while short math problems (GSM8K) can use longer periods.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "1.  Unlike heuristic acceleration methods, ES-dLLM’s logic is rooted in quantitative analysis of dLLM generation characteristics. For example, PCA and L1-norm metrics confirm hidden state variation follows an exponential distribution (concentrated near 0) across layers 10–30 of LLaDA-8B; log-scale distribution of confidence scores shows 80% of positions have variation <0.05 after iteration 50. These observations directly validate the \"early-skipping\" rationale, low-variation tokens contribute little to final outputs, justifying computational reduction.\n2. ES-dLLM can be seamlessly integrated with confidence-aware parallel decoding (PD), further boosting speedup.", "weaknesses": "1. ES-dLLM is only tested on 7B/8B models, for 13B/70B dLLMs, the computational overhead of importance score calculation (layer-wise L1-norm of hidden states) may scale sharply with model size (more layers/heads increase tensor comparison cost), offsetting speed gains. For sequences >512 tokens (e.g., 1k legal documents), fixed skip positions (r₄=r₈) are suboptimal: deep layers (e.g., layer 16+) have higher tensor variation, but ES-dLLM does not adjust skipping layers dynamically, which may lead to under-skipping in deep layers or over-skipping in early layers. It needs some discussions here.\n2. Theoretical 60% FLOP reduction only translates to 1.85× speedup over DualCache in practice, as LLM inference shifts to memory-bound (model weights/KV cache access unchanged). ES-dLLM does not integrate system-level optimizations  to unlock full speedup potential. For example, on Dream-7B/HumanEval, ES-dLLM achieves 308.51 TPS, but memory bandwidth constraints (H200’s 3.35TB/s) limit it to 40% of theoretical throughput (771 TPS).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)