Dynamic-dLLM: Dynamic Cache-Budget and Adaptive Parallel Decoding for Training-Free Acceleration of Diffusion LLM
Authors: Tianyi Wu
Xiaoxi Sun
Yanhua Jiao
Yulin Li
Yixin Chen
Yun-Hao Cao
Yi-Qi Hu
Zhuotao Tian
Venue: ICLR 2026 Poster (2026)
Keywords: dLLMs
Inference Acceleration
Primary area: foundation or frontier models, including LLMs

Abstract: Diffusion Large Language Models (dLLMs) offer a promising alternative to autoregressive models, excelling in text generation tasks due to their bidirectional attention mechanisms. However, their computational complexity, scaling as $\mathcal{O}(L^3)$ with sequence length $L$, poses significant challenges for long-sequence and real-time applications, primarily due to the lack of compatibility with key-value caching and the non-autoregressive nature of denoising steps. Existing acceleration methods rely on static caching or parallel decoding strategies, which fail to account for the dynamic behavior of token properties across layers and decoding steps. We propose \textbf{Dynamic-dLLM}, a training-free framework that enhances dLLM inference efficiency through two components: Dynamic Cache Updating (DCU), which adaptively allocates cache-update budgets based on layer-wise token dynamics, and Adaptive Parallel Decoding (APD), which dynamically calibrates decoding thresholds to balance generation quality and efficiency. Extensive experiments on models like LLaDA-8B-Instruct, LLaDA-1.5, and Dream-v0-7B-Instruct across benchmarks such as MMLU, GSM8K, and HumanEval demonstrate that Dynamic-dLLM significantly improves inference speed, attaining an average speedup of exceeding 3$\times$ while maintaining performance. Dynamic-dLLM outperforms state-of-the-art acceleration methods and provides a plug-and-play solution for efficient dLLM deployment without compromising performance. Code and models will be made publicly available.

Published as a conference paper at ICLR 2026
DYNAMIC-DLLM: DYNAMIC CACHE-BUDGET AND ADAPTIVE PARALLEL DECODING FOR TRAINING- FREE ACCELERATION OF DIFFUSION LLM
Tianyi Wu
Xiaoxi Sun" "1
Yanhua Jiao
Yulin Li
Yixin Chen² Yunhao Cao
Yiqi Hu²
Zhuotao Tian
Harbin Institute of Technology. Shenzhen
Huawei
Shennhen Loop Area Institute
ABSTRACT
Diffusion Large Language Models (dLLMs) offer a promising alternative au- toregressive models, excelling in text generation tasks due to their bidirectional attention mechanisms. However, their computational complexity, scaling as ((L3) with sequence length L, poses significant challenges for long-sequence and real-time applications, primarily due to the lack of compatibility with key-value caching and the non-autoregressive nature of denoising steps. Existing accelera- tion methods rely on static caching or parallel decoding strategies, which fail to account for the dynamic behavior of token properties across layers and decod- ing steps. We propose Dynamic-dLLMM a training- free framework that enhances dLLM inference efficiency through two components: Dynamic Cache Updating (DCU), which adaptively allocates cache-update budgets based on layer-wise to- ken dynamics, and Adaptive Parallel Decoding (APD), which dynamically cali- brates decoding thresholds to balance generation quality and efficiency. Extensive experiments on models like Instruct across benchmarks such as MMLU, GSM8K, and HumanEval demon- strate that Wynamic-dLLM, significantly improves inference speed, attaining an average speedup of exceeding 3x while maintaining performance. Dynamic- dL.LM outperforms state-of-the-an acceleration methods and provides a plug-and- play solution for efficient dLLM deployment without compromising performance. The code is available at https//giiibbbcomm//aaa////2/2//
LLaDA-1.5, and Dream-v0-77
ABCC 4913
Aq.
5201
12.30
.0/62
22.36
PR
2211
Cache
Pysami--111
BLUM
Tas-LLM
dL1M
(b) Dream-vo- Instruct Figure 1: The comparison in terms of tokens-per-second (TPS)
(a) LLaDA-8B-Instrus
INTRODUCTION
Diffusion Large Language Models (dLLMs) have emerged compelling alternative to autoregres- sive models (ARMS), demonstrating strong performance in text generation tasks. Notable examples such LLaDA (Nic al. 2025 Zhu et al. 2025) and Dream Ye et al. 2025) highlight the rapid
"Equal contribution. The work is done during Tianyi's internship at CBG Celia DeviceAl Team, Huawei. Corresponning author (tianzhuotaode hit.edu.cn).

---

Published as a conference paper at ICLR 2026
Masked
Token Token
Unmaaked
Pecodins
Confidence
ux
[EOS] him for
him
bad
good
fareeheedd
100
205
(a) Layer imput
(b) Attention output
Serted very
Sartid
very
302
200
is hat
Desired
is
202
Candidate
hat
His
His
232
3.50
150
Step
Step t-n
(c) Layer imput
(d) Attention output
Wrong prediction using fixed threshold
Figure 2: (a-b) Layer input similarity and attention output similarity across adjacent denoising steps. The brighter region denotes a higher similarity, indicating most tokens are stable across steps. (c-d) The number of tokens requiring updates across different steps. Differences across layers indicate varying demands for the token update budget. (e) Existing parallel decoding methods may yield wrong predictions as potential candidates have been discarded by the fixed threshold.
progress in this direction. A key advantage of dL.LMs lies in their bidirectional attention mecha- nisms, which enhance scalability and enable superior performance in handling complex scenarios, such as the "reversal curse" (Berglund al.),, where traditional ARMs often struggle. This allows dLLMs to capture richer contextual dependencies in challenging scenarios.
However, despite their strong performance certain domains, dLLMs face a fundamental challenge: their computational complexity scales as OLL3) with respect to sequence length L, significantly exceeding the O(L?) cost autoregressive models (ARs). This cubic scaling imposes a severe bottleneck for long-sequence and real-time generation tasks, limiting the practical deployability of dLLMMs in latency-sensitive applications. The root cause lies the con-autoregressvve nature of dLLMs, where each denoising step requires updating all tokens in parallel across the full sequence. Besides, this paradigm hinders the caching of key-value activations from previous steps, rendering dLLMs incompatible with the widely used KV-Cache mechanism.
Key observations. To address this issue, recent work has explored strategies for dLLM acceler- ation. For example, (Liu al. 2025b Ma et al. 2025 Song et al., 2025) reduce redundancy by caching internal token representations across decoding steps. Concurrently, (Wu et al. 2025) ac- celerates inference by enabling parallel unmasking of multiple tokens within single step. These methods implicitly rely on specific token properties, such as feature stability and confidence, to identify opportunities for optimization. However, they all rely a static strategy across all lay- ers and decoding steps, applying the same caching or unmasking criteria throughout the model and generation process, thus overlooking the dynamic nature of token behavior during generation.
As illustrated in Figure 2 aa-d), the token properties vary across different layers and steps. The frequency of changes the internal features of tokens differs across layers, while the distributions of token confidence fluctuate across decoding steps. The static strategies adopted by existing methods may fail to account for this dynamic behavior, leading to performance degradation. Therefore, this observation prompts critical question: how to design an adaptive method that dynumically aligns with the model's s intrinsic layer-wise and step-wise token dynamics improve the efficiency?
Our solution. In this work, we propose Dynamic-dLLM. a training-free framework for acceler- ating dLL.M inference. Dynamic-dI.LM consists of two key components: Dynamic Cache Updating (DCU) and Adaptive Parallel Decoding (APD).
Specifically, as tokens may exhibit heterogeneous dynamics across layers, instead of a static cache updating strategy across all layers, we propose Dynamic Cache Updating (DCU) that allocates cache-update budgets adaptively, ensuring that layers requiring frequent updates are prioritized, while computational overhead is reduced in stable layers. In addition, the existing parallel decoding strategy with fixed thresholds risks committing to tokens prematurely, as confidence estimates can shift over time, leading to error propagation. To mitigate this, introduce Adaptive Parallel De-
2

---

Published as a conference paper at ICLR 2026
coding (APD) that dynamically calibrates decoding thresholds by tracking the evolving distribution of prediction confidence, achieving a decent trade-off between the degradation of generation quality caused by low threshold and the inefficiency resulting from high threshold.
Extensive experiments across LLaDA-8B-Inse Instrrcc,, LLa Dream v--7B-On-sttie and bench- marks covering mathematics, science, coding, and general tasks demonstrate the effectiveness and strong generalization capabilities
the proposed method. Notably, Dynamic-dLLnn achieves a maximum acceleration of up to 4.48x, with an average speedup exceeding 3x while still maintain- ing performance, making it a plug-and-play training-free solution for enhancing the efficiency of dLLMs without compromising performance. In summary, our contributions are as follows:
In this study, we observe that the variations across layers and decoding steps of dLLM may undermine the effectiveness of existing static rule-based acceleration methods.
We propose Dynamic-dLLM. dating (DCU) and Adaptive Parallel Decoding (APD), DCÚ adaptively allocates cache- update budgets across layers, while APD dynamically calibrates decoding thresholds across steps, jointly enabling efficient yet robust acceleration of dI..Ms.
training-free free framework composed of Dynamic Cache Up-
Extensive experiments across diverse models and tasks show that Dynamic-dLLLM substan- tially improves inference efficiency while preserving the accuracy, outperforming state-of- the-art acceleration methods.
BACKGROUND AND MOTIVATION
2.1 PRELIMINARIES OF DLLM
In this section, we introduce preliminaries regarding the inference process of dLLM (Nie et al. 2025). The introduction of related work is presented in the Appendix D due to the page limit.
Given a prompt of length Lprompt tokens and a target generation length of Lgen tokens, let L pepoopp + Leen- The dLLM generates the output in 7 iterative decoding steps, producing approx- imately Lgen/ tokens per step. Let V denote the model's vocabulary, and let [MASK] special placeholder token indicating positions to be predicted. Denote by xf E VL the token sequence at step f, where t = T,T 1.. The initial sequence is constructed as:
V be
x" (Jo, ......... 2 2 |AAK] ... [MASK]), (MASKI),
where v; are the given prompt tokens. At each step I, the mask predictor So computes a distribution over the vocabulary for each position:
- fa(x') E R "xIv]
(2)
Using greedy decoding, we can obtain the most probable token at each masked position:
# arg max (Softmax(2))). if af [MASK].
VEV
transition function S then updates the sequence to x by selectively replacing tokens based on confidence scores, re-masking low-confidence predictions to refine them in subsequent steps: S(X',
x 1-
x t). The final output sequence x" is yielded when =
2.2
KEY OBSERVATIONS
Despite recent progress in accelerating diffusion-style LLMs (dLLMs) (Liu et al. 2025b: Wu et al. Ma et al. 2025 Song et al. 2025), two critical inefficiencies remain unaddressed.
2025
Layer-wise Cache Update Needs Vary Significantly. Existing methods exploit temporal redun- dancy by reusing cached intermediate features (e.g., query, key, value, attention output, FFN output) from the previous step for subset of tokens, assuming high feature similarity across steps. However, as illustrated in Figure 2(add), the proportion of tokens requiring cache updates varies substantially across layers, increasing monotonically from shallow deep layers. This suggests that uniform or heuristic caching strategies are suboptimal. Instead, a dayer-adaptive cache update policy is essen- tial for dynamically allocating computation budgets where they matter most.
3

---

Published as a conference paper at ICLR 2026
Step
key token
Step t-1
key token
Fenture Computed
Luyer
Feature Rossed
Window
I Layer
XX
Budget reallocation
XX
X
X
X
Macked Tocka
Chehe updaringg
Unmasked Tockn
Rudger
XX
M
X
X
KX
Thresholds
Layer
Confidence
Infeasily
Sorted
Xpai
24 ust
Consine Similarity
Dynamic-dLLAm LLM consists of two key components: Dynamic Cache Updating (DCU, upper part) and Adaptive Parallel Decoding (APD, lower part). DCU reallocates cache update budget for each layer each step, while APD dynamically adjusts the decoding thresholds for all tokens.
Figure 3:
Static Thresholding Hinders Parallel Decoding Efficacy. Parallel decoding strategy (e.g.. Wu
jet al. (2025)) unmask tokens once their confidence exceeds fixed threshold. Yet, as shown in Figure e)), the token with the highest confidence at early step may be the desired output and will be revised later, often replaced by its "runner-up" prediction with the second-highess confidence initially. Conversely, tokens whose top prediction exhibits clear dominance over alternatives, i.e. low entropy or large margin, can be safely finalized earlier, even if absolute confidence remains below a static threshold. Therefore, to enable earlier commitment to stable predictions, thereby expediting convergence without compromising accuracy, exploring the feasibility of dynamic per- token threshold, udjusting adaptively based on the predicted distribution (e.g., entropy or probability margin), becomes essential.
METHOD
To overcome the limitations of existing approaches, we propose Dynamic-dLLMM, a training- acceleration framework that dynamically optimizes dLLM inference along two dimensions: cache- update management and parallel decoding scheduling.
Regarding cache-update management, introduce dynamic allocation mechanism for managing cache updates, recognizing the varying dynamics across layers. This approach dynamically dis- tributes the update budget among layers. prioritizing layers that require more frequent cache updates. On the other hand, for optimizing the parallel decoding, we replace fixed confidence thresholds with an adaptive per-token unmasking strategy, based on the predicted distribution of each token. This strategy facilitates early commitment to confident predictions while postponing uncertain ones, achieving more balanced trade-off between speed and output quality.
The overview is presented in Figure Sections
and 3.2 detail each component, respectively.
3.1 DYNAMIC CACHE UPDATING
Recent works (Liu et al. 2025b Ma et al. 2025 Song et al. 2025 update fixed uniform number of token caches across all layers. However, as demonstrated Section 2.2 the demand for cache up- dates varies significantly across layers. This observation motivates the need for dynamic allocation strategy that adapts the cache-update budget per layer according to the specific requirement.
In this section, we propose the Dynamic Cache Updating (DCU) strategy, which selectively updates only those tokens whose representations undergo significant changes between consecutive inference steps. Prior work (Liu et al.) 2025b) identifies such tokens by measuring the cosine similarity be- tween the current and cached Value vectors. While effective, this approach incurs non-negligible computational overhead due to the explicit recomputation and comparison of Value vectors. Ideally,

---

Published as a conference paper at ICLR 2026
(a) Attention: 0.94
(b) FFN: 0.97
(c) Key: 0.99
(d) Value: 0.99
Spearman correlation values of layer inputs with intermediate features, including Key, Value, Attention Output, and FFN Ouiput. We visualized the cosine similarity between tokens' fea- ture vectors and their cached counterparts at adjacent steps, and compared the relationship between layer input and (a) Attention Output, (b) FFN Output, (c) Key, (d) Value.
Figure 4:
token dynamics could be estimated without recomputing these vectors, cached values could be safely reused, thereby reducing redundancy.
Inspired by Liu et al. (2025a), who observed a strong correlation between model inputs and outputs in diffusion transformers (DiT) (Peebles & Xie 2023a), investigate the relationship between layer inputs and intermediate features in dsLMs. As shown in Figure ! the features cached (e.g., Key, Value, Attention Output, and FFN Output) exhibit high correlation with the corresponding layer inputs. This implies that changes in layer inputs across steps serve as reliable proxy for the un- derlying dynamics of intermediate activations. Consequently, input-level differences can effectively inform vache-update decisions without accessing or recomputing the cached features themselves.
Layer-Adaptive Cache Budget Allocation. To dynamically allocate the cache update budget across layers, we first define a token-level dissimilarity metric, d'd,, estimating the change in the representation of token z; at layer between consecutive inference steps t and + 1. This metric is computed using the cosine distance between the normalized token inputs at the respective steps:
(x x 1.
ted = --
(4)
||x] |||xx +1
higher value of d'd denotes a greater change in the token's representation, suggesting higher need for cache update. Then, we aggregate the token-level variations into a layer-wise metric ghd This metric represents the average change in token representations within layer I:
N-1
Σ
d
(5)
where N is the sequence length. Subsequently, the cache update budget for layer / at step I, denoted as D is then allocated proportionally to its measured dynamism the previous step (t+-1), s'+1, This allocation is normalized across all layers using the total available budget, Buyer X LayerNum:
Blyer (Blayer x LayerNum)
(6)
Layer
Et=D
For each layer the set of tokens scheduled for cache update at step f, denoted 4td.1 is initialized as an empty set at the start of the step: 4rd 0. Then, layer identifies the set St. comprising the top-Bi.y tokens with the highest variation d. These selected tokens are then added to the update set: (1t. + 44h St.
Token Stuck in the Mud. Nevertheless, the layer-adaptive cache budget allocation strategy may potentially make some tokens stuck in the mud. Specifically, if token T; is not selected for an update layer 1, its cached representation remains unchanged. Consequently, its input to layer | ++ also remains static, leading to zero variation score d:++1 = 0 for that layer. As the allocation strategy prioritizes tokens with high df the token 2, will only be updated in layer I + 1 if the number of tokens exhibiting non-zero variation is insufficient fill the allocated budget B1,++1 Should this
5

---

Published as a conference paper at ICLR 2026
0.14
0.150
50
50-
0.125
hregend
0.3
100
100
150
150
0.1
0.34
200
200
0.025
0.02
250
250
136
186
236
206
336
306
136
186
2356
286
336
306
distence
taken
taken
(a) Analysis of Distance
(b) Layer 15
(c) Layer 21
Figure 5: Local property analysis of dLLMs (a) Relationship between the distance from key token and the frequency of being decoded in the current step. The closer the token is to the key token, the higher the probability of being decoded. (b) The last two images respectively represent the attention of response tokens to key token in layer 15 and 24. Red dot is the key token at this step. The illustration shows that the tokens around the key token have higher attentions, which means that the changes caused by decoding the key token affect those tokens more than others.
occur, and if z; is again not selected (e.g., chosen randomly among the low-priority tokens), will remain unchanged entering layer 1++2, perpetuating the cycle. We refer to this phenomenon, where token fails to be updated across multiple consecutive layers due to consistently low variation scores induced by prior missed updates, as a token becoming stuck in the mud.
Mandatory Update Window. As illustrated in Figure 5 there exists spatial locality in the update pattern: tokens surrounding the one unmasked in the previous step (the key token) are statistically more likely to be updated in the current step. Let the position of the key token be To mitigate the risk of the next key token (the token with the highest confidence to unmasked in the current step) becoming stuck in the mud, we introduce Mandatory Update Window, This mechanism ensures that a local region around the key token is always updated, regardless the adaptive budget allocation. Formally, we define window of fixed size Bwindow centered on the key token's position
The set of token positions covered by this window at a given step is [p Baanaoo P Bassaon
For each layer 1, the caches for all tokens within this window are compulsorily added to the layer's update set L
Bwindow 2
Bwindow
Sispt
2
This updated set 24 then constitutes the final list of tokens whose caches will be recomputed for layer in the current step. By ensuring continuous updates within this local window, we reduce the likelihood of critical tokens being overlooked and retain the response local changes. The global budget is subsequently distributed adaptively among the remaining tokens based on the layer-specific variation metrics ghd for the following step.
3.2 ADAPTIVE PARALLEL DECODING
Section 2.2
highlights that the peak confidence of a token can vary significantly across decoding steps in ds..Ms. This inherent dynamism poses challenge for fixed-threshold parallel decoding methods (Wu et al., 2025). which rely on a static criterion and consequently suffer from decoding inaccuracies due to mispredictions at certain steps.
To address this, we introduce the Adaptive Parallel Decoding mechanism that dynamically adjusts the masking threshold for each token based on its local prediction stability. Each token æ; starts with an initial threshold 7." The threshold step t, denoted r/. is adapted from the threshold used the previous step t + 1, T ++1
Adaptive Threshold via Confidence Concentration. The core idea is to modulate the threshold based the concentration of the token's 's predicted probability distribution. Intuitively, diffuse distribution (a small gap between the highest and second-hiie -ighhsst probabilities) suggests lower con- fidence in the current prediction, warranting stricter (higher) threshold to reduce unnecessary up-

---

Published as a conference paper at ICLR 2026
MMLU
ARC-C
GSMSK
GPQA
IIE
Avg.
L.DDA-SD-Intr
60.92
88.53
78.21
32,17
38.54
89.67
Throughput (TPS,
10.19(1.0))
25.49 (1.0%)
(1.00)
(1.0X)
(1.00)
1.0x
dLLM Cache
61.33
88.49
78.78
31.84
37,99
59.69
Throughput (TPS, 1)
(2.31x)
33.64 (L.32x)
(3.0x)
2.90x)
28.44(183x))
Cache
61.37
32.25
Throughput (TPS,
17.73 x.74x)
29.82 (L.17)
15.14 (1.82)
14,44( x,90))
20.82 x1.34x)
1.59
Fast-dLLM
61.39
88.05
78.44
32.01
38.21
59.62
Throughput (TPS, 1)
26.29 (2.58x)
32.88 (L.29x)
22.74 (2.73x)
22.63( 2,98x)
27.82 (1.79
2.27x
DDnamicc-LLLM (Ouss
60.95
88.09
78.24
31.98
38,33
59.51
Thrroghput (TPS, ?)
M0.1 (2.96x)
40.27 (1.58x)
27.21 (3.27
25.46 (3.35x)
30.61 (L37x)
2.63x
L'sst-LLLM*
61.08
88.32
76.62
32,21
37.87
59.22
Throughput (TPS,
32.30 (3.17x)
41.55 (1665
31.36 (3.77x)
27.06/3.56
32.94 (2
2
1ynamii-dl L (Ours)
60.89
87.79
78.01
31.89
38,08
59.33
Throoghput (TPS,
34.14
42.31 (1.6x)
37.29 (4.48
31.84(4.19
36.83/2.37 X)
3.21x
Table Results on LLaDA-8BB 8B-Instruct (Nie al. 2025). Each cell includes the accuracy, decoding throughput (TPS), with relative efficiency enhancement to the baseline. Best values in bold, subop- timal values underlined. Results with * are obtained with parallel decoding.
dates. Conversely, concentrated distribution indicates stability, permitting reduced threshold for early decoding. Let zł be the probability distribution over the vocabulary for token x, at step t, the index of the most likely token is: arg maxvev (z)), Thus, the concentration of this distribution is quantified using the second-highest probability score:
1
max eEV\(u)
(z)).-
A larger value of of signifies more peaked and confident distribution. Based on this measure, the decoding threshold for token x; at step is adjusted as follows:
7 = -..
where a positive hyperparameter controlling the sensitivity of the threshold adaptation. This formulation ensures that tokens with highly concentrated distributions (large c)) have their thresh- olds decreased, allowing for early decoding, while tokens with diffused distributions have increased thresholds to prevent decoding errors.
Integration with Temporal Instability. In addition, the magnitude of historical shifts in token's confidence distribution provides a strong signal for its likelihood of future revision. We quantify this shift via the cosine distance between the token's contidence distributions at adjacent steps:
(2)
H
(10)
A larger H indicates greater instability in the prediction, suggesting that the token may still be undergoing refinement and thus warrants stricter (higher) threshold to prevent early decoding. By combining et and H;, the decoding threshold for token at step is updated as:
7j + 1+1 a ci + 3 H
(11)
where 3 2 0 are hyperparameters balancing the influence of prediction confidence and temporal instability.
[nd in the Appendix outline the core mechanisms of Dynamic-dLLM for accelerating dynamic LLMs (dLLMs) via Feature-Caching and Parallel Decoding, respectively. By explicitly accounting for dynamism along both the layer and step dimensions, Dynamic-dLLM minimizes redundant computation and thereby significantly accelerates the inference process of dLLMs.
EXPERIMENT
4.1 EXPERIMENT SETTINGS
We assessed the performance of Dynamic-dLLM using three typical dLLMs baselines: LLaDA- SB-Instruct, LLaDA-1.5 and Dream- Bream-7B-Inma In-Itruuuc.. not otherwise specified, we default Blyer to 32, and B Bwindow to 32. More experimental details are shown in Appendix B

---

Published as a conference paper at ICLR 2026
MMLU
ARC-C
GSMSK
GPQA
HB
LLaDA-15
61,42
88,51
81.62
33.61
40.24
61.08
Throughput (TPS,
10.18 (1.0%)
(1.00)
8.30(LX)
(1.0
15.54 (1.0%)
1.0%
JLLM-Cashe
61.39
88.04
81.64
33,45
40.18
60.94
Throughpot (TPS,
23.62 (2.32xx
(1.33x)
(3.19x
21.88
(1.82x
UKVVCCache
61.47
88.23
81,53
33.54
39.81
60.92
Throughpat (TPS.
(1.71x)
29,33(1.15x)
15.27 (184X)
14.46 (1.9 X)
20.67 (1.33x)
1.59x
Fast-dLLM
61.46
$8.18
81.21
33,63
40.21
60.94
Throughput (TPS.
26.47 (2.60x)
33.15 (1.30x)
23.07 (2.78x)
22. 22.56 (2.98
28.13 (1.81x)
2.29X
+Dynamic-LLLM (Ours)
61,03
88.29
80,98
33.37
39,93
60.72
Throughput (TPS.
30.03(2.95x)
(L.1
27.31 (3.299%)
28 (3.34xx
(1.98x)
2.63x
Fasst-LLLM*
61,22
80.94
33.43
Throughpat (TPS, +)
(3.16x)
(3.75x)
17.00 (3.58
32,93 (2.12x)
2.85x
+DynamicddLLM (Ours)
61.34
88.02
81.03
32.97
40.01
60.67
Throughput (TPS, 7)
34,20 (3.36x)
42.59(1.67 x)
37.02 (4.46x)
31.57 (4.17x)
36.67(2.36 x)
3.20
Table 2: Results on LLaDA-1.5 (huu et al. 2025). Each cell includes the accuracy, decoding throughput (TPS), with relative efficiency enhancement to the baseline. Best values in bold, subop- timal values underlined. Results with are obtained with parallel decoding.
MMLU
ARC-C
GSMSK
GPQA
Avg.
Beamm-vt--B-Insteeee
73.34 9.97 (1.x)
89.63 20.44 (L0X)
77.47
34,08
56.82
66.27
Throughput (TPS,
8.05
7.13
14.95 (1.0))
1.0%
+LLLM-Cache
73,08
90.04
76.64
34,75
$4,31
65.76
Throughpat (TPS,
(1.95x))
(1.32(
22.46 (2.79x)
(2.74x)
24.22 (1.62x)
2.06
<KK--Cache
72.93
89.40
77,32
33,87
$4.69
65.64
Throughpat (TPS.
16.75( 1.68X)
22.07 (1.08
12,80 (1.59x)
(1.49
(1.28x)
1.42X
La Throughpat (TPS,
72.14
90.11
76.81
33.99
55.70
65.75
18.74 (1.888]
(1.47x)
21.50 (2.67))
17.04 (2.39x)
20.18 (1.35
DDnnmiccdLLMM (Ours)
72.09
33.17
TTrrughppt (TPS. ?))
25.82 (2.59
31.89 (1.56x)
25.52 (3.17x)
21.90
27.81 (1.86x)
2.45x
Fast L
71.97
89.98
76.95
33.34
56.78
65.80
Throughpat (TPS, 19
10.11(3.02)
32.50 32.50x1,59x)
28.90 (3.59x)
23.81 (1.34x)
31.99 11,99(2.44 (214x)
2.7x
Dynamic-al J.M.M L (Ours) Throughput (TPS.
72.10
89.38
77,52
32.02
$4.05
65.01
31.31 (3.14x)
32.91 (1.6x)
31.48 (3.91))
25.88 (3.63x)
36.93
2.95
Table 3: Results Dream-v0-7B ---trrru Ye et al. coding throughput (TPS), with relative efficiency enhancement to the baseline. Best values bold, suboptimal values underlined. Results with are obtained with parallel decoding.
2025
Each cell includes the accuracy, de-
To comprehensively evaluate a model's performance and efficiency, we employ two key metrics: accuracy benchmarks and throughput, with the latter measured in Tokens Per Second (TPS).
The benchmarks includes MMLJU (5-shot)(Hendryyks et al., 2020). ARC-challenge (ARC-c, 0- Shor) al. 2018). GPQA (5-shot)(Rein al.] 2024), GSM8k (4-shot)(oobbe et al.) 2021). and HumanEval (HE, (-shot) et al.] 2021). For fair comparison, we divided the methods into
two groups, one using Feature-Cache(Liu et al. 2025b Ma et 2025 Wu et al. 2025) and the
other using KV-Cache and parallel decoding(W) Wu et al. 2025). All experiments were performed on NVIDIA Pro6000 GPUs.
4.2
MAIN RESULTS
Our results (baseline vs. alternative methods vs. our Dynamic-dILMM) are presented in Table I These results show that DyLamic-dI.LM not only achieves the most significant throughput improvement but also maintains performance.
and 3
With only feature cache enabled, Dynamic-dLLM delivers substantial speedups for high-priority tasks without accuracy degradation. It achieves notable throughput boosts benchmarks with an average speedup of over 2.5x across all evaluated tasks for LLaDA-8B. Instruct, while maintaining accuracy. When combined with parallel decoding, Dynamic-dLLLn scales speedups. For LLaDA- SB-Instruct on GSM8k, throughput hits 37.29 TPS (4.48x faster than the baseline's 8.32 TPS), with average speedup across tasks reaching 3.21x and robust accuracy.
8

---

Published as a conference paper at ICLR 2026
AGo RCTosd ToMAto lite Tirelodd Tirshood
100
Theighe Doan
625
266
12
254
Duspet
(a) Ablation of Buyer
(b) Ablation of Bwindow
(c) Ablation of Threshold
Figure Ablation studies on key hyperparameters, investigating the respective effects on the model's performance (accuracy) and efficiency (throughput).
This superiority persists across models: LLaDA-1.5 achieves 4.46x speedup on GSM8k (37.02 vs. 8.30 TPS) with near-baseline accuracy (60.67% 61.08%); Dream-v0-7B- v0-7B-Instrucc gains 3.9 speedup on GSM8K (31.4F vs. 8.05 TPS). These cross-model results demonstrate its generalization capabilities.
4.3 ABLATION STUDIES
In this section, we present the ablation studies regarding the core designs of our method.
fix the windoww to 32 and do not use parallel decoding, and explore the impact of Buyer on accuracy and throughput. With the gradual increase of Bwindow. the accuracy shows an upward trend, reaching a plateau around 32. On the other hand, the throughput also rapidly decreases with the increase of Bwindow- Based on observations, a value of 32 for Bwindow is more trade-off choice.
Impact of Blayer Accuracy and Throughput. As shown in Figure 6a
Impact of Bwindow on Accuracy and Throughput. Similarly, we discussed the impact of Bwindow on accuracy and throughput in Figure 6b Bwindow is fixed 32 and parallel decoding is disabled. The impact of Bwintow on accuracy and throughput is roughly the same as that of Buayer. but the smaller Bwindow has more severe reduction in accuracy than Blayer- ensure that the accuracy is basically par with the baseline, we have chosen 32 the optimal value for Bwindow-
Dynamic Threshold vs. Fixed Threshold. We discussed the difference between fixed threshold and dynamic threshold in Figure The accuracy of both is the same under all initialization. How- ever, dynamic thresholds bring fewer inference steps than fixed thresholds in higher initialization. With the maximum initialization of 0.9, which does not excessively descend performance, dynamic thresholds can reduce inference steps by approximately 30% compared to the fixed thresholds.
CONCLUDING REMARKS
Summary. We present Dynamic-dLLM, a training-free framework for accelerating diffusion LLMs by adapting the dynamic behavior of tokens across layers and decoding steps. By troducing dynamic cache updating and adaptive parallel decoding, our method significantly reduces redundant computation while preserving generation quality. Extensive experiments across various models and benchmarks covering mathematics, science, coding, and general tasks demonstrate the effectiveness and strong generalization capabilities of the proposed method. In summary, Dynamic- dLLM offers general, sluu-and--layy solution toward efficient dLLM inference, highlighting the importance of adaptive strategies in non-eevoreeressive generation.
Limitation & Future Work. While Dynamic-dLLM, demonstrates strong performance across standard language generation benchmarks, capabilities in multi-modal understanding and com- plex reasoning scenarios remain largely unexplored. In particular, the model's current design is tailored to unimodal textual inputs, and it is unclear how its core mechanisms generalize to set- tings involving heterogeneous data modalities (e.g., vision, audio, structured knowledge). Future work should investigate how these principles can be reformulated or extended to address the unique challenges of cross-modal alignment, representation fusion, and modality-speciff computational demands. Such extensions could unlock new avenues for building more flexible and efficient foun- dation models capable of robust reasoning across diverse input types.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This work was supported by the Guangdong Basic and Applied Basic Research Foun- dation (2025A1515011546) and by the Shenzhen Science and Technology UCY220240813100500000,, w1222222000010000000022222222222
Program
BIBLIOGRAPHY
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems,
2021.
Berglund, M Tong, M Kaufmann, Balesni, AC Stickland, T Korbak, and O Evans. The reversal curse: Lims trained on ia is b" fail to learn "b is a". arxiv 2023. arXiv preprint arXiv:2309. Stt::2229..2888
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the al2 reasoning challenge. arXiv preprinl arXiv: 1803.0547, 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.144 14168, 2021.
Jiequan Cui, Shu Liu, Zhuotao Tian, Zhisheng Zhong, and Jiaya Jia. Reslt: Residual learning for long-tailed recognition. IEEE transactions on pattern analysis and machine intelligence, 45(3): 3695-3706, 2022.
Jiequan Cui, Zhisheng Zhong, Zhuotao Tian, Shu Liu, Bei Yu, and Jiaya Jia. Generalized parametric contrastive learning. IEEE Transactions on Pattern Analysis and Machine Intelligence. 46( (6):: 7463-7474, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusson probabilistic models. Advances in neural information processing systems, 33:6840-683 68511 2020.
Junchao Huang, Xinting Hu, Boyao Han, Shaoshuai Shi, Zhuotao Tian, Tianyu He, and Li Jiang. Memory forcing: Spatio-temporal memory for consistent scene generation on minecraft. arXiv preprint arXiv:2510.03198 2025a.
Junchao Huang, Xinting Hu, Shaoshuai Shi, Zhuotao Tian, and Li Jiang. Edit360: 2d image edits to 3d assets from any angle. In ICCV, 2025b.
Li Jiang, Shaoshuai Shi, Zhuotao Tian, Xin Lai, Shu Liu, Chi- thi-Wing Fu, and Jiaya Jia. Guided point contrastive learning for supe-supe point cloud semantic segmentation. Proceedings of the IEEE/CVF international conference on computer vision, pp. 6423-6432, 2021.
Xin Lai, Zhuotao Tian, Jiang, Shu Liu, Hengshuang Zhao, Liwei Wang, and Jiaya Jia. Semi- supervised semantic segmentation with directional context-aware consistency. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 1205-1214, 2021.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Rea- soning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9579-9589. 2024a.
10

---

Published as a conference paper at ICLR 2026
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. Step-dpo: Step- wise preference optimization for long-chain reasoning llms. arXiv preprint arXiv:2406.1862, 2024b.
Yunxin Li, Zhenyu Liu, Zitao Li, Xuanyu Zhang. Zhenran Xu, Xinyu Chen, Haoyuan Shi. Shenyuan Jiang. Xintong Wang, Jifang Wang, et al. Perception, reason, think, and plan: A survey large multimodal reasoning models. arXiv preprint arXiv:2505.0492. 0492/, 2025.
Feng Liu, Shiwei Zhang, Xiaofeng Wang. Yujie Wei, Haonan Qiu, Yuzhong Zhao, Yingya Zhang, Qixiang Ye, and Fang Wan. Timestep embedding tells: It's time to cache for video diffusion model. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 7353- 7363, 2025a.
Zhiyuan Liu, Yieun Yang, Yaojie Zhang, Junjie Chen, Chang Zou, Qingyuan Wei, Shaobo Wang, and Linfeng Zhang. dllm-cache: Accelerating diffusion large language models with adaptive caching. arXiv preprint arXiv:2506.0660 06295, 2025b.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion language modeling by estimating the ratios of the data distribution. 2023.
Xinyin Ma. Runpeng Yu, Gongfan Fang, and Xinchao Wang. dkv-cache: The cache for diffusion language models. arXiv preprint arXiv:2555.15781.222 2025.
Shen Nic, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint arXiv:2502.09992 2025.
Zhenhua Ning, Zhuotao Tian, Guangming Lu, and Wenjie Pei. Boosting few-shot 3d point cloud segmentation via query- -eeidd enhancement. In Proceedings of the 31st ACM international con- ference on multimedia, pp. 1895- 1904, 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4195-4205, 2023a.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4105-4205, 2023b.
Bohao Peng, Zhuotao Tian, Xiaoyang Wu, Chengyao Wang, Shu Liu, Jingyong Su, and Jiaya Jia. Hierarchical dense correlation distillation for few-shot segmentation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 23641-23651, 2023. 2023.
Bohao Peng, Zhuotao Tian, Shu Liu, Mingchang Yang, and Jiaya Jia. Scalable language model with generalized continual learning. arXiv preprint arXiv:2404.07470, 2024a.
Bohao Peng Xiaoyang Wu, Li Jiang, Yukang Chen, Hengshuang Zhao, Zhuotao Tian, and Jiaya
Jia. Oa-cnns: Omni-adaptive sparse cnns for 3d semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21305-21315, 2024b.
Shangpin Peng, Senqiao Yang. Li Jiang, and Zhuotao Tian. Mitigating object hallucinations via sentence-level early intervention. arXiv preprint arXiv:2507. 12455, 2025.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel Bowman. Gpqa: A graduate-level google-proof q&a bench- mark. In First Conference Language Modeling, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Tong Shao, Zhuotao Tian, Hang Zhao, and Jingyong Su. Explore the potential of clip for training- free open vocabulary semantic segmentation. In European Conference on Computer Vision, pp- 139-156. Springer, 2024.
11

---

Published as a conference paper at ICLR 2026
Jascha Sohl-Dickstein, Eric A Weiss, Niru Maheswaranathan. and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. JMLR.org. 2015.
Yuerong Song, Xiaoran Liu, Ruixiao Li, Zhigeng Liu, Zengfeng Huang, Qipeng Guo, Ziwei He, and Xipeng Qiu. Sparse-dllm: Accelerating diffusion Ilms with dynamic cache eviction. arXiv preprint arXiv:2508.023: 2025.
Zhuotao Tian, Hengshuang Zhao, Michelle Shu, Zhicheng Yang, Ruiyu Li, and Jiaya Jia. Prior guided feature enrichment network for few-shot segmentation. IEEE transactions on pattern analysis and machine intelligence, 44(2):1050-106; 2020.
Zhuotao Tian, Pengguang Chen, Xin Lai, Li Jiang, Shu Liu, Hengshuang Zhao, Bei Yu, Ming- Chang Yang, and Jiaya Jia. Adaptive perspective distillation for semantic segmentation. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(2):1372 (5(2):1322-1377, 2022.
Zhuotao Tian, Jiequan Cui, Li Jiang. Xiaojuan Qi, Xin Lai, Yixin Chen, Shu Liu, and Jiaya Jia. Learning context-aware classifier for semantic segmentation. In Proceedings of the AAAI confer- ence on artificial intelligence, volume 37, pp.
2023.
Chengyao Wang, Li Jiang, Xiaoyang Wu, Zhuotao Tian, Bohao Peng. Hengshuang Zhao, and Jiaya Jia. Groupcontrast: Semantic-aware self-supervised representation learning for 3d understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 4917 4928, 2024.
Junjie Wang, Bin Chen, Yulin Li, Bin Kang, Yichi Chen, and Zhuotao Tian. Declip: Decoupled learning for open-vocabulary dense perception. In Proceedings of the Computer Vision and Pat- tern Recognition Conference, pp. 14824-14834, 2025a.
Junjie Wang, Keyu Chen, Yulin Li, Bin Chen, Hengshuang Zhao, Xiaojuan Qi, and Zhuotao Tian. Generalized decoupled learning for enhancing open-vocabulary dense perception. arXiv preprint arXiv:2508. 11256, 2025b.
Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao. Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. Fast-dllm: Training-free acceleration of diffusion Ilm by enabling kv cache and parallel decoding. arXiv preprint arXiv 2025.
Senqiao Yang, Tianyuan Qu, Xin Lai, Zhuotao Tian, Bohao Peng. Shu Liu, and Jiaya Jia. Lisa++: An improved baseline for reasoning segmentation with large language model. arXiv preprint arXiv:2312.1 17240, 2023.
Senqiao Yang, Zhuotao Tian, Li Jiang, and Jiaya Jia. Unified language-driven zero-shot domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, pp. 23407-23415, 2024.
Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 19792-19802, 2025.
Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong, Dream 7b: Diffusion large language models. arXiv preprint arXiv:2508. 15487, 2025.
Yujia Zhang, Xiaoyang Wu, Yixing Lao, Chengyao Wang, Zhuotao Tian, Naiyan Wang, and Heng- shuang Zhao. Concerto: Joint 2d-3d self-supervised learning emerges spatial representations. In NeurlPS, 2025.
Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chunwei Wu. Jun Hu, Jun Zhou, Jianfei Chen, Yankai Lin, Ji-Rong Wen, et al. Llada 1.5: Variance-reduced preference optimization for large language diffusion models. arXiv preprint arXiv:2505 19223, 2025.
12

---

Published as a conference paper at ICLR 2026
CONTENTS
Contents
13
Algorithm Supplement
14
Experiment Details
14
B. Benchmarks and Settings
14
8. Implementation Details
14
Example Description
15
Related Work
16
Experimental Supplements
17
Stability Analysis of Hyperparameters
17
Analysis Additional Latency
17
Performance in Low-resource Hardware
18
13

---

Published as a conference paper at ICLR 2026
A ALGORITHM SUPPLEMENT
The pseudocode of Dynamic-dLLM is shown in Algorithm I
and
Algorithm Dynamic Cache Updating
Require:
Mask predictor fe. prompt с and initial masked sequence x² with length L, denoising
steps 7,, cache update budget Bwindow and Blayers initial threshold 7
Ensure: Final prediction x"
Initialize caches step f T */ D Cache Key, Value, Attention Output and FFN Output of T,
1: C SaitiaalizeCace(T...XX
tokens.
2: Generate prediction x² using model fa
> Needs initial pass separate handling
3: x S(x xT, .T)
4: fort=T= T
to do
5:
Xlayer f x"
Initial input for layer at step
6: for 1 in each layer in the Transformer network do x <- TayerNorm(x (x(ayerni)
7:
8:
4rd - +- 4td t- u ue, U f2; p-
9:
Buintre
S Sis S
Bande
D / p is the position of token
2
unmasked in last step */ for each token j in sequence do
10:
(x x
dy =
11:
|xx |1| |
12:
end for
1h, Mean (obt, chi ..... dii
14:
B + (Blayer x LayerNumi)
Layerraa -,--1 kez
15:
4 t- u indices of Biyye tokens with heightest dy Xuyerauut C RefreshCache (x* xtumm 1, C,U,))
17:
Xlayer.in Xlayer.oul
b Update input for the next layer
end for
End layer loop
19:
Generate prediction z' using final layer output Xlayer out x -' + ParallelDecoding22
20:
>> Adaptive Parallel Decoding shown in
if all x unmasked then
22:
break
23:
end
24:
end for
P End step loop
25:
return final prediction x"
B EXPERIMENT DETAILS
B.1 BENCHMARKS AND SETTINGS
Table 4 shows the specific setups for each benchmark, involving the count of decoding steps, block length. and generation length. The benchmarks encompass MMLU (5-shot), ARC-C (0-shot), GSM8K (4-shot), Math (4-shot), and HumanEval (0-shot). To examine the generalization and ro- bustness of diverse approaches, use a uniform setup for all benchmarks except HvmanEval. Owing to its unique task characteristic, HumanEyal demands greater number of decoding steps and longer generation length.
reduce task-dependent hyperparameter adjustments and instead
B.2 IMPLEMENTATION DETAILS
We offer thorough explanation of the parameter setups for the compared methods dK V-Cahh and dLLM-Cache across various models. According to the suggested configurations in the dKV-Cache paper, the cache update interval to 8 for the L.LaDA series and 4 for the Dream series.
14

---

Published as a conference paper at ICLR 2026
Algorithm 2 Adaptive Parallel Decoding
Require: Prediction z', parameters a,,3, initial threshold Tn for every masked token n, masked
sequence x!
1: p' + Softmax (z')
> Probability distribution over
2: for 0 to L 1 do
3:
Puotte f- Sort(p,,. descending)
4: C -
Psoed
D Calculate peak concentration */ End token loop
5: end for
> / Calculate confidence fluctuation (for adjacent steps) */
6:
i < T then
7:
for i = 0 to L do
8:
H
(2:) x.
Cosine similarity
9: end for
10: end if
/ Adaptive threshold adjustment */
11: if
< T then
12:
7. =/=11
of + 8 H
13: end if
Unmask all with p r', always unmask p
15:
return X₁-1
For dLLM-Cacke the paper presents multiple parameter configurations, where K, denotes the prompt refresh interval and K, represents the response refresh interval. specifically, For LLaDA- 8B-Instruct: K,- 50, K, 7; For LLaDA-1.5: K,- 100, K, 6; For Dream-v0-77BInntrrec, K₂ = 50, K₂ = 2.
In addition, for Adaptive Parallel Decoding (APD) in Dynamic-dLLM, we set a 0.001 and 3 = 0.0008 based on extensive statistical analysis.
Datasets
Steps
Block Len
Gen Len
MMLU
256
32
256
ARC-C
256
256
GSMBK
256
32
256
Math
256
32
256
HumanEval
512
32
512
Table 4: Configuration of Benchmarks
C EXAMPLE DESCRIPTION
As shown in Figure 7 in the absence of candidate, fixed threshold can actually hinder early de- coding of correct predictions, while Adaptive Parallel Decoding monitors the status candidates in real time and ends unnecessary steps early.
15

---

Published as a conference paper at ICLR 2026
Musked
Decoding
Confidence
Token
Token
Locked
good
threshold:
thueshuld:
0.8
good
Sorted
g0od Sorted
the
Non Candidate
Step
Step
Figure 7: Fixed threshold may hinder the early output of correct predictions, as shown in the figure. The correctly predicted "good" cannot be decoded until its confidence exceeds the threshold 0.8, which wastes steps.
D RELATED WORK
Large Language Models. Driven by transformer-based architectures and large-scale pretraining.
Large Language Models (LLMs) have achieved remarkable success, demonstrating exceptional ca-
pabilities(Tian Tian et al.] 2020; Lai al. 2021 Jiang et al. 2021 Peng al. 2023 Cui et al., 2023 ?:
Shao et al. 2024 Peng ct al. 2024a). While these models primarily excel in textual processing Tran
al. 2022 2023] Ning et al.) 2023 Wang al. 2024), their robust architectural foundation has
paved the way for various functional extensions Cui et al. 2022 2023 Peng et al. 2024b Yang
et al. 2024 Wang et al., 2024). such semantic segmentation and object detection Yang et al.
2024
2025 Lai et al. 2024b] Peng et al.
20255 Wang et al./ 22025b] Huang et al.) (022aa). One no-
table derivative branch the development of Multimodal Large Language Models (Wang
et al. 2025a et al., [0255] Zhang et al., 2025 Huang et al. 2025b), which expand the core LLM utility by integrating modality encoders to process ipputs beyond text, such as image, audio, and video. Through this extension, the reasoning power of the central LIM backbone has been adapted assist other domains, including traditional computer vision tasks, as illustrated by specific applications like LISA(Lai et al. 2024a) and LISA+ (Yan+ et al. 2023) in reasoning segmentation.
Diffusion Large Language Models. Diffusion models, which excel continuous data generation through iterative denoising processes (Sohl-Dickstein al., 2015 Ho et al. 2020), have recently shown promising potential in natural language processing. Unlike their success image domains (Rombach et al. 2022 Peebles & Xie 2023b), adapting these models to text generation faces fun- damental challenges arising from the discrete token space and sequential dependencies inherent in language. A key advancement in addressing these issues comes from discrete diffusion frameworks, particularly Masked Diffusion Models (MDMs) that operate by progressively refining sequences through context-aware mask prediction (Austin al. [021] Lou et al. 2023). Recent methodolog- ical innovations have significantly expanded the capabilities of diffusion-
language models. Scaling efforts have produced foundation models like LLaDA(Nie et al. (025), an 8B parameter bidirectional Transformer trained from scratch, and Dream Ye et al. (2025) which leverages pre- trained autoregressive weights, both achieving performance parity with similarly sized autoregres- sive models.
Inference acceleration methods of dLLMs. Multiple studies have investigated strategies for speed- ing up discrete diffusion large language models (dLLMs). Some studies using feature caching cut down on computational costs by caching the internal features of tokens across different diffusion steps. dLLM-Cache (Liu et al. 2025b) selects fixed proportion of tokens for cache update for each layer by sorting the cosine similarity of Value vector between adjacent steps. dKV-Cache (Ma et al. 2025) puts the tokens decoded each step into the cache and doeses not update them in later steps. L (Wu et al. 2025) puts tokens outside the current block to the cache and updates tokens within the current block. The similarity of these methods is that they adopt the same cache update
16

---

Published as a conference paper at ICLR 2026
Bwindow
32
64
128
192
64
0
256
Blayer
32
64
128
64
192
256
0
Score
73.92
77.62
79.15
78.03
78.92
74.07
75.74
Table 5: Performance of DCU with different settings of Bwindow and Blayer, using 1024 generated tokens on the GSM8K dataset with the LLaDA-8B-Instrun model.
gen
256
512
1024
score(dynamic dLLM)
78.01
78.31
78.15
TPS
37.29
35.41
33.19
score(baseline)
78.21
78.98
79.11
TPS
8.78
7.62
6.21
Table 6: Performance scores under different generation lengths with 0.001, = 0.0008
strategy for all layers, which ignores the different requirements of each layer. In addition, Fast- dLLM proposes the parallel decoding strategy, unmasking the tokens with confidence exceeding a predetermined threshold, which has difficulty balancing quality and efficiency.
E EXPERIMENTAL SUPPLEMENTS
E.I STABILITY ANALYSIS OF HYPERPARAMETERS
Stability Analysis of Blayer and Bwiiddwr- To investigate the stability parameter settings across diverse scenarios, we conducted additional experiments. As shown in Table when the sequence length exceeds 1k, the accuracy of the default settings (Btayer = 32, B Bwindow: = 32) exhibits slight decline. However, with appropriate increases these two parameters, the accu- racy gradually recovers. Furthermore, when the sum of Blayer and Buindow is fixed, different proportional allocations between them lead varying impacts on performance. Through these experiments, we confirmed that setting Blayer equal to Bwindow yields optimal results. To en- sure the method's stability across different generation lengths, we propose an auto-tuning strat- egy: Blayer Bwindow = gen len (where gen len denotes the generation length).
Stability Analysis a and B. For the hyperparameters formed experiments under different output length settings (256, 512, and 1024), while maintain- ing Burindow = x gen_len and Bloger = x gen_len (see Table 6). The results indicate that there is almost no degradation in accuracy across these different generation lengths, verifying the strong stability of a and B.
0.001 and B 0.0008, we per-
As per Equation 10 and 3 regulate threshold adaptation to prediction confidence and temporal instability, respectively. We conducted ablation experiments on GSM8K (gen len=256), with re- sults in Table 7 (score=accuracy; NIS=Number of Inference Steps, fewer faster). The optimal range for a/ is the 10-3 order of magnitude. The model sensitive to order-of-ne maggnnuudee scaling (severe quality loss with over-scaling) but stable small variations within this range.
E.2 ANALYSIS OF ADDITIONAL LATENCY
We present comprehensive metrics in Table S which details the performance scores (accuracy), av- erage single-inference latency, and additional latency introduced by DCU and APD separately- evaluated on the GSM8K dataset across different generation lengths (eenJen: 256, 512, 1024). As shown in the Table 8 both DCU and APD introduce minimal additional latency, and this overhead exhibits a clear linear scaling trend with generation length, which only accounts for a small portion of the inference latency, far less than the gain it brings.
17

---

Published as a conference paper at ICLR 2026
3
Score
NIS
0.001
0.0008
78.01
95
0.01
0.008
69.75
54
0.1
0.08
59.76
16
0,001
0.0007
78.75
96
0,001
0.0005
78.85
95
0.001
0.0017
78.93
97
Table 7: Performance scores and Number of Inference Steps (NIS) under different combinations of
and 5, with 256 generate length on GSM8K
Baseline
DCU
APD
gen len
Score
Time (s)
Extra Time (s) Score
Time (s)
Extra Time (s)
Score
Time (s)
Extra Time (s)
256
78.21
29.16
None 78.85
9.41
1.94
78.01
6.87
0.11
512
78.98
67.19
None 79.31
20.68
4.31
78.31
14.46
0.19
1024
79.11
164.90
None 79.15
47.36
8.90
78.15
30.85
0.33
Table 8: different generation lengths (gen.len). Time values are rounded to two decimal places.
Performance metrics (scores and time-related data) of baseline, DCU, and APD under
E.3 PERFORMANCE IN LOW- LOW-RESOURCE HARDWARE
To address low-resource hardware deployment, tested our method with the LLaDDA-8B-Instruett model on the GSMSK and HumanFval datasets using an RTX 4090 GPU. As shown in the Table our method still maintains strong demonstrating its practicality for deployment on non-high-end hardware.
trade-off under such cost-constrained settings,
18

---

Published as a conference paper at ICLR 2026
LLaDA 8 Instruct
Dynamic-dLLM(DCU) Dynamic-die dLLM
Dataset
Score
TPS
Score
TPS
Score
TPS
GSM8K
78.34
5.48
77.89 37.24
14.73
77.91
29.77
HumanEval
37.47
9.74
24.32
37.15
35.19
Table 9: Performance Scores and Inference Speed (TPS) of Different Methods on RTX 4090
19


Reviews:
[{"review_id": "AGW62aQ9e8", "reviewer": "Reviewer_m7U5", "summary": "The paper introduces Dynamic-dLLM, a training-free way to speed up diffusion LLMs at inference. The idea is twofold: first, Dynamic Cache Updating—it watches how layer inputs change and only refreshes the cache where it matters, with a small must-update window around key tokens so you don’t get stuck. Second, Adaptive Parallel Decoding—it decides, token by token, when it’s safe to decode in parallel based on how sharp the distribution is and how much things are shifting over time. In short: fewer wasted updates, smarter parallel steps, similar quality, much faster runs.\nAcross LLaDA-8B/1.5 and Dream-7B on MMLU, ARC-C, GSM8K, GPQA, and HumanEval, the method reports 3-4x speedups with near-baseline accuracy, outperforming prior dLLM acceleration methods (dLLM-Cache, dKV-Cache, Fast-dLLM).", "questions": "1. How stable are the chosen defaults (Blayer=32, Bwindow=32; α=0.001, β=0.0008) across sequence lengths >1k and other decoding schedules? Any auto-tuning strategy? \n2. How is the “key token” window identified?\n3. How does parallel decoding length and context length impact the performance?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Elegant DCU proxy (measuring changes in layer inputs instead of recomputing K/V/attn/FFN), plus a practical “mandatory window” to avoid tokens getting “stuck.”\n2. Solid empirical gains across three models/5 tasks; headline result: 37.29 TPS on GSM8K (4.48×) with parallel decoding on LLaDA-8B; similar trends for LLaDA-1.5 and Dream-7B.\n3. APD is simple and principled\n4. The two proposed components target each failure mode.", "weaknesses": "1. Using input-change proxies is cheaper than feature recomputation, but the paper could more explicitly break down wall-clock costs for computing dt and maintaining per-layer budgets.\n2. Limited discussion of long-form coherence/failures or quality trade-offs.\n3. Limited novelty vs. Fast-dLLM and prior caching work", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "omrw1Dgri3", "reviewer": "Reviewer_sidX", "summary": "Diffusion Large Language Models (dLLMs) have bidirectional attention strengths but face $O(L^3)$ computational complexity and inefficient static acceleration methods. Dynamic-dLLM, a training-free framework, solves this via two components: Dynamic Cache Updating (DCU) which adaptively allocates layer-wise cache budgets using layer input cosine distance (proxying intermediate changes) and a mandatory update window to avoid \"token stuck\", and Adaptive Parallel Decoding (APD) which adjusts thresholds via prediction concentration and temporal stability. Tested on NVIDIA Pro6000 with LLaDA-8B-Instruct/1.5, Dream-v0-7B-Instruct across 5 benchmarks, it achieves average 3× speedup (max 4.48× on GSM8K) while maintaining accuracy (difference ≤1% vs. baselines), outperforming dLLM-Cache, dKV-Cache, and Fast-dLLM.", "questions": "I would be happy to increase my rating if my views are given a thorough discussion.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ **Targeted Dynamic Adaptation Addresses Core Pain Points**：The paper is the first to systematically capture two key dynamic characteristics of dLLMs—heterogeneous inter-layer cache update demands and step-wise fluctuations in token confidence distribution—overcoming the limitations of existing static acceleration methods. For Dynamic Cache Updating (DCU), it uses layer inputs as a proxy for intermediate feature changes (Spearman correlation with Key/Value features > 0.94), avoiding the computational overhead of recomputing intermediate tensors while ensuring accurate update decisions. For Adaptive Parallel Decoding (APD), it adjusts thresholds by combining prediction distribution concentration and historical stability, solving the misjudgment issue of fixed-threshold methods (e.g., preventing premature unmasking of unstable tokens).  \n+ **Training-Free Design Enables Strong Engineering Practicality**：Dynamic-dLLM requires no model fine-tuning or architectural modifications; it only optimizes inference via dynamic strategies, allowing direct integration as a \"plug-and-play\" module into open-source dLLMs (e.g., LLaDA-8B-Instruct, Dream-v0-7B-Instruct). The \"Mandatory Update Window\" (solving the \"token stuck in the mud\" problem) enhances engineering robustness, and no additional large caches are introduced, keeping memory overhead manageable and reducing deployment costs.  \n+ **Comprehensive Experiments with High Persuasiveness**：Experiments cover 3 representative dLLMs, 5 task types (general reasoning, math, coding, QA), and 4 SOTA baselines (dLLM-Cache, dKV-Cache, Fast-dLLM). Results consistently show 2.45×–4.48× speedups with ≤1% accuracy loss (e.g., LLaDA-1.5’s average accuracy 60.67% vs. baseline 61.08%). Ablation studies (on $B_{\\text{layer}}$, $B_{\\text{window}}$, threshold types) fully validate the necessity of each component, and hyperparameter guidelines (e.g., optimal $B_{\\text{window}}=32$) improve result reproducibility.  \n+ **Tight Integration of Empirical Observations and Theoretical Foundations**：The method design is rooted in solid empirical insights—such as the monotonic increase in inter-layer update demand from shallow to deep layers, and the local update pattern around key tokens—rather than pure heuristics. Quantitative metrics (cosine distance for token variation, distribution concentration for confidence) provide mathematical grounding, enhancing the method’s generality and interpretability.", "weaknesses": "+ **Key parameters lack adaptability**：Mandatory update window size ($B_{\\text{window}}=32$) and APD hyperparameters ($\\alpha=0.001$, $\\beta=0.0008$) are fixed, failing to adapt to sequence length (e.g., redundant for short sequences, insufficient for ultra-long ones) or task differences, leading to suboptimal performance.  \n+ **Extreme scenarios untested**：No validation on ultra-long sequences (>1K tokens) (to verify cache/memory stability) or low-resource hardware (e.g., RTX 3090) (to check cost-constrained deployment), limiting practicality.  \n+ **Incomplete baseline comparisons**：Missing cross-paradigm methods (e.g., ES-dLLM’s early skipping) and industrial-grade dLLMs (e.g., Mercury), failing to fully prove relative advantages.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t9dLdgqKzx", "reviewer": "Reviewer_Pz1R", "summary": "This paper tackles the $\\mathcal{O}(L^{3})$ inference cost of dllms. It argues that existing training-free methods use static caching and decoding strategies, which fail to account for the dynamic behavior of tokens across layers and steps. The authors propose Dynamic-dLLM, a training-free framework with two components. First, Dynamic Cache Updating adaptively allocates cache-update budgets to layers based on token input similarity, a proxy for feature change. It introduces a Mandatory Update Window to prevent token stagnation. Second, Adaptive Parallel Decoding dynamically adjusts the unmasking threshold for each token based on its prediction confidence concentration and temporal instability. Experiments on dLLMs like LLaDA and Dream show an average 3x speedup while maintaining accuracy.", "questions": "- Can the authors provide a detailed breakdown of the computational overhead (e.g., in latency) introduced by the DCU and APD  mechanisms at each step? How does this overhead scale with sequence length $L$?\n- Please provide the extensive statistical analysis or an ablation study used to determine the APD hyperparameters $\\alpha=0.001$ and $\\beta=0.0008$. How sensitive is the model's quality/speed trade-off to these two values?\n- The \"stuck in the mud\" phenomenon is mitigated by a $B_{window}$ around the previous key token. What is the failure rate if the next most confident token falls outside this window? Have the authors explored alternative, non-heuristic solutions to this problem?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- DCU is well-motivated, using layer input similarity as an efficient proxy for feature dynamics , and thoughtfully addresses the stuck in the mud failure case with a mandatory update window. \n- APD is also a strong contribution, moving beyond static thresholds to a dynamic policy based on both confidence concentration and temporal instability. \n- The empirical results are excellent, demonstrating a >3x average speedup with no performance degradation, and clearly outperforming SOTA baselines .", "weaknesses": "- While the method is training-free, both DCU and APD introduce non-trivial computations at each step (e.g., cosine similarities for all tokens/layers, probability sorting, and instability calculations). The paper does not quantify this overhead relative to the computation saved. \n- The method introduces several new, sensitive hyperparameters (e.g., $B_{layer}$, $B_{window}$, $\\alpha$, $\\beta$). The paper provides an ablation for the budgets but states the APD parameters ($\\alpha$, $\\beta$) were set based on extensive statistical analysis, which is not shown. The Mandatory Update Window is also a heuristic whose robustness is not fully ablated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UiNqS28HHd", "reviewer": "Reviewer_qCPz", "summary": "Diffusion-based Large Language Models (dLLMs) face high computational complexity (scaling with the cube of sequence length) and inefficiency from static acceleration strategies. To solve this, the paper proposes Dynamic-dLLM—a training-free framework that boosts dLLM inference efficiency via two core components: Dynamic Cache Updating (DCU): It adaptively distributes cache-update budgets across layers based on how much token inputs change between steps. To prevent \"stuck tokens\" (tokens never updated due to low variation), it adds a mandatory update window around \"key tokens\". Adaptive Parallel Decoding (APD): Instead of fixed thresholds for unmasking tokens, it adjusts thresholds dynamically using two signals. Experiments on three dLLMs (LLaDA-8B-Instruct, LLaDA-1.5, Dream-v0-7B-Instruct) across five benchmarks (MMLU, ARC-C, GSM8K, GPQA, HumanEval) show Dynamic-dLLM achieves an average speedup of 3.21 times.", "questions": "1. For long sequences (e.g., extended GSM8K with 1k tokens), do adjustments like a larger mandatory window (64 instead of 32) improve accuracy? What’s the TPS and accuracy for 1k-token tasks, and how does APD’s overhead change with longer sequences?\n2. What percentage of inference time is spent calculating token variation and layer metrics? Can sampling fewer tokens (e.g., 50% of tokens) for variation checks reduce overhead without losing accuracy?\n3. For tasks with >256 steps (e.g., HumanEval’s 512 steps), do APD’s threshold adjustments get too strict/lenient over time? Does resetting thresholds periodically (e.g., every 64 steps) help?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Unlike baselines (e.g., dLLM-Cache, Fast-dLLM) that use the same update budget for all layers, DCU allocates budgets based on actual layer needs. It leverages a strong correlation (0.94–0.99) between layer inputs and intermediate features (like Key/Value tensors) to avoid redundant calculations—instead of checking expensive Value vectors, it uses simpler input changes to judge if a token needs updating. The mandatory update window (default size 32) ensures critical tokens near recently unmasked ones are always updated, solving the \"stuck token\" problem common in static methods.\n2. APD replaces fixed confidence thresholds (used by Fast-dLLM) with thresholds that adjust per token and per step. Tokens with clear, stable predictions get lower thresholds (unmasked earlier for speed), while uncertain tokens get higher thresholds (avoiding errors). At a high initial threshold (0.9), APD cuts inference steps by 30% vs. fixed thresholds without accuracy loss. For example, on LLaDA-8B-Instruct/GSM8K, APD pushes throughput to 37.29 tokens per second (TPS), much faster than the baseline.", "weaknesses": "1. All experiments use sequences up to 512 tokens (HumanEval) or 256 tokens (other benchmarks). For sequences longer than 1k tokens: (1) The fixed 32-size mandatory window may miss long-range dependencies (key tokens’ influence could span more than 32 positions); (2) Calculating layer-wise variation for thousands of tokens becomes slower, possibly offsetting speed gains; (3) APD’s temporal stability checks (comparing token distributions across steps) get more computationally heav. It is better providing more dicussion here.\n2. While DCU avoids recomputing Value vectors, it still spends time calculating token input variation and layer-wise metrics for every layer and token. For deep dLLMs (e.g., 128 layers), this adds extra work per step, but the paper seemingly ignores to present how much time this takes, or if it slows down inference compared to baselines like dLLM-Cache.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)