Attention Is All You Need for KV Cache in Diffusion LLMs
Authors: Quan Nguyen-Tri
Mukul Ranjan
Zhiqiang Shen
Venue: ICLR 2026 Poster (2026)
Keywords: Diffusion LLMs
Attention-aware KV Cache Update
Layer-aware KV Cache Update
Primary area: generative models

Abstract: This work studies how to adaptively recompute key–value (KV) caches for diffusion large language models (DLMs) to maximize prediction accuracy while minimizing decoding latency. Prior methods' decoders recompute QKV for all tokens at every denoising step and layer, despite KV states changing little across most steps, especially in shallow layers, leading to substantial redundancy. We make three observations: (1) distant MASK tokens primarily act as a length-bias and can be cached block-wise beyond the active prediction window; (2) KV dynamics increase with depth, suggesting that selective refresh starting from deeper layers is sufficient; and (3) the most-attended token exhibits the smallest KV drift, providing a conservative lower bound on cache change for other tokens. Building on these, we propose Elastic-Cache, a training-free, architecture-agnostic strategy that jointly decides ${when}$ to refresh (via an attention-aware drift test on the most-attended token) and ${where}$ to refresh (via a depth-aware schedule that recomputes from a chosen layer onward while reusing shallow-layer caches and off-window MASK caches). Unlike fixed-period schemes, Elastic-Cache performs adaptive, layer-aware cache updates for diffusion LLMs, reducing redundant computation and accelerating decoding with negligible loss in generation quality. Experiments on LLaDA-Instruct, LLaDA-1.5, and LLaDA-V across mathematical reasoning and code generation tasks demonstrate consistent speedups: $8.7\times$ on GSM8K (256 tokens), $45.1\times$ on longer sequences, and $4.8\times$ on HumanEval, while consistently maintaining higher accuracy than the baseline. Our method achieves significantly higher throughput ($6.8\times$ on GSM8K) than existing confidence-based approaches while preserving generation quality, enabling practical deployment of diffusion LLMs.

Published as a conference paper at ICLR 2026
ATTENTION Is ALL YOU NEED FOR KV CACHE IN DIFFUSION LLMS
Quan Nguyen- Nguyen-Tri'..2 Mukuul Ranjan' Zhiqiang Shen't
VILA Lab, MBZUAI, Abu Dhabi, UAE 3FPT Software Al Center, Vietnam
(quan.tris mukul ranjan, zhigiang wmb..neaaaa.em.................. ac
Project http::///.ii.......tt..aaaaaaaaa atg.eeaneee...............aaaaar
ABSTRACT
This work studies how to adaptively recompute key-value (KV) caches for diffu- sion large language models (DLMs) to maximize prediction accuracy while mini- mizing decoding latency. Prior methods' decoders recompute QKV for all tokens at every denoising step and layer, despite KV states changing little across most steps, especially in shallow layers, leading to substantial redundancy. We make three observations: (1) distant MASK tokens primarily act as a length-bias and be cached block-wise beyond the active prediction window; (2) KV dynamics increase with depth, suggesting that selective refresh starting from deeper layers is sufficient; and (3) the most-attended token exhibits the smallest KV drift, pro- viding a conservative lower bound on cache change for other tokens. Building on these, we propose Elastie-Cache that jointly decides when to refresh (via an attention-awar drift test on the most- attended token) and where to refresh (via a depth-aware schedule that recomputes from a chosen layer onward while reusing shallow-layer caches and off-window MASK caches). Unlike fixed- fixed-period schemes, Clastic-Cache performs adaptive, layer-aware cache updates for diffusion LLMs, reducing redundant computation and accelerating decoding with negligible loss in generation quality. Experiments LLaDA--nstrus LLaDA-1.5, and LLaDA-V across mathematical reasoning and code generation tasks demonstrate consistent speedups: 8.7x on GSM8K (256 tokens), and 45.1x on longer sequences, while consistently maintaining higher accuracy than the baseline. Our method achieves significantly higher throughput (6.8x on GSMSK) than existing confidence-bait based approaches while preserving generation quality, enabling practical deployment of diffusion LLMs.
training- free, architecture agnossic strategy
INTRODUCTION
Diffusion large language models (DLMs) (Li et al., 2025) have recently emerged as compelling alternative to autoregressive Transformers (Radford et al. 2018 Achiam et al.) 2023), yet their iter- ative denoising procedure makes inference particularly compute-intensie tations, each decoding step recomputes queries, keys, and values (QKV) for every token at every layer, even though the underlying key-value (KV) states change only marginally across most steps. This all-tokens, all-layers recomputation incurs substantial latency and memory traffic, ultimately limiting practical deployment. Our goal in this study is to determine how and when to adaptively recompute the KV cache during decoding so as to maximize prediction quality while minimizing wall-clock latency.
In standard implemen-
A defining property of diffusion L.M decoding is the progressive unmasking of tokens under a length- and structure-aware attention pattern. This induces heterogeneous KV dynamics: shallow layers tend to stabilize quickly as they encode local lexical structure, whereas deeper layers continue adjust global, semantic dependencies. We formalize this with a notion of KV drift: the step-to- step change in cached keys and values, and observe two consistent trends: (i) drift is small for most
"Equal contribution. 'Corresponding author.

---

Published as a conference paper at ICLR 2026
steps, and (ii) drift grows with layer depth. These trends suggest that indiscriminate recomputation is wasteful, and that targeted refreshes could preserve accuracy while slashing cost.
Prior acceleration methods for diffusion (and related) decoders typically refresh the KV cache fixed schedule, c.g., every & iterations without regard to instance difficulty, current attention patterns, or layerwise variability. Such fixed-period policies leave performance the table: they recompute when nothing has changed and miss updates precisely when rapid semantic revisions occur. More- over, by treating all layers uniformly, they over-service shallow layers whose representations have already converged, while under-servicing deeper layers where changes matter most. This motivates an adaptive, attention-awan aware alternative.
Our approach is built on three empirical observations. First, distant MASK tokens exert negligible influence on unmasking the current token and behave primarily as a length- -ias prior; thus, their KV can be block-cached outside the active prediction window to avoid redundant work. Second, KV drift increases with depth, so refreshes should start at an automatically learned boundary layer Ex (determined by attention threshold and adapted to each input decoding step) and apply only to deeper layers, reusing shallow-layer caches. Third, the most-attended token at step typically exhibits the smallest drift, providing a conservative lower bound on KV changes across the context. Monitoring this drift yields reliable, low-overhead trigger for deciding whether a global refresh is warranted.
Based on these ideas, propose Elastic-Cacher training frre, architecture-agnostic strategy that couples Attention-Aware KV Cache Update with Layer-Aware KV Cache Update. The attention- aware module computes lightweight drift statistic on the most-attended token; if the statistic ceeds a threshold, a refresh is triggered, otherwise cached KVs are reused. The layer-aware module then refreshes only layers É > e*, while shallow layers retain their caches, and off-window MASK tokens remain block-cached. Together, these mechanisms align recomputation with where and when the model's beliefs actually change, minimizing unnecessary QKV work.
In contrast to fixed-period baselines, our Elastic-Cache adapts to the input, step, and layer granu- larity together. It reduces compute by skipping recomputation during stable phases, focuses effort on deeper layers during semantic revisions, and leverages block-wise caching for distant MASK tokens. Conceptually, the method reframes KV management as an attention-guided control prob- lem: attention estimates which tokens matter; drift detects how much the state has changed; and the layer boundary encodes where updates pay off. This yields a practical pathway to low-latency diffusion LLM decoding without modifying training or the base architecture.
Our contributions of this work:
We diagnose redundancy in diffusion LLM decoding and introduce KV drift as a principled signal for adaptive cache management.
We propose Elastic-Cacke refresh policy for diffusion LLMs that jointly decides when to recompute (attention-awanl drift test) and where to recompute
the first (to our best knowledge) adaptive, layer-aware KV
selective updates).
We develop block-wise MASK caching to eliminate needless updates outside the prediction window. We provide comprehensive empirical experiments and ablations showing that our Elastic-Cache preserves generation quality while substantially reducing decoding latency across tasks and model scales.
2 PRELIMINARY
2.1 MASKED DIFFUSION MODELS
Masked Diffusion Models (MDMs), absorbing-state discrete diffusion, build on D3PM (Austin al.) 222 la) and its continuous-time variant (Campbell al. 2022). replacing tokens with a special MASK along a forward process (Sahoo et al. 2024 Shi et al. 2024) at timestep t:
(x)
II (1) = II Cat(a); (1 -) tŌMASK)
where t [0, 1] controls interpolation between the original data zu (at 0) and fully masked sequence (at t = 1), Cat(-) denotes the categorical distribution. A parametric model Po learns the reverse denoising; generation starts from all MASK and iteratively unmasks by sampling po (ar]aa)).
2

---

Published as a conference paper at ICLR 2026
(a) Accantion weights
Value states Casine similarity
Atzantinn weight Cosine amilark
(d) The mest-attended toxand
Prompt tokena
Deep layers
within Sliding Window
Cache changes
Fara WT
35
Co colena
Tromn Prxition
Layer
Layer
Layer
Figure 1: Visualization of our motivation. (a) MASK tokens located near each other receive high attention, while those situated far apart have minimal influence. (b) Over time, the representations in the KV states of cached tokens evolve, with deeper layers experiencing more substantial changes. (c) The changes in attention weights of most-attended tokens exhibit similar patterns to the changes in KV states of all cached tokens. (d) KV states of the most-attended tokens have the least changes.
Recent theory (MDLM (Shi al. 2024 Sahoo et al. 2024). RADD (Ou et al. 2024)) simplifies training from a variational bound to a reweighted cross-entropy over masked positions:
CMDM = [.' -
Σ
- log po (zg
(2)
Fa0t= MASK
This formulation scales LLMs as diffusion language models (DLMs), with LLaDA (Nie al. 2025b) and Dream-7B (Ye et al. 2025) matching autoregressive performance while enabling parallel decoding and flexible infilling.
2.2 KEY- VALUE CACHE IN TRANSFORMERS
Transformer- based language models achieve computational efficiency during autoregressive genera- tion through Key- Value (KV) caching (Pope et al. 2023). In causal attention, each layer projects the current hidden state H' into query, key. and value representations using learned projection matrices WQ. WK. Wv. .. At decoding step t, the attention computation for the current token follows:
Lig (K(
(K.() (1:9)" Vdx
K [::] concat K t-1 1:t-1]: K
(i);
A/1 softmax
KV cache:
(3)
V
[1:4] concat
[1=t-1] (4)
To avoid redundant computation, previous key-value pairs are cached and reused. This caching strat- egy is effective because in causal attention, previously computed key-value pairs remain invariant
throughout decoding (K (-1
[1:t-1] ( enabling efficient reuse without affecting model output.
KV-Cache in Bidirectional Attention. However, in diffusion models, bidirectional attention allows all positions to attend to each other, invalidating the invariance assumption of traditional KV-cache. As dKV-Cache (Ma al. 2025) observes, token representations evolve across denoising steps, making cached keys/values stale. This dynamic behavior necessitates rethinking caching strategies for diffusion language models.
3
METHODOLOGY
3.1 OUR FRAMEWORK OVERVIEW AND MOTIVATION
Diffusion LLMs differ from autoregressive decoders in that their key-value (KV) states evolve across denoising steps due to bidirectional dependencies. Our objective is to adaptively decide when and where to recompute the KV cache to preserve accuracy while minimizing latency. Baseline decoders recompute QKV for all tokens and layers at every step, despite negligible KV changes for most steps and especially in shallow layers (Fig. [b); deeper layers exhibit larger drift. Rather than
fixed-penne seriop refreshes (Wu al. 2025 Ma et al. 2025 [iu et al. (202)), we propose Elastic-Cacha
the first (to our knowledge) adaptive, layer-aware KV update policy for diffusion LLMS that jointly optimizes timing and location of recomputation.
Our design is driven by three observations. (1) Distant MASK tokens mainly act as length prior and exert minimal influence the current unmasking, therefore block-cache their KV beyond the
3

---

Published as a conference paper at ICLR 2026
Most stentive
Cachee
MASK
Decoded
Decod:
Fully compose coche for layers
Punialy coeputt cacht from layer to Resperne
Sis Ameniin wrightx
token
taken
bolio
Iokzn Priingi
Cesine Similarity 1s 8
Frompi
Titeck
Weeck
1 18/1
change soo aeatloonatt
Window
Layer; gett coched store
layero
Recompitee
layerr+
from
laper 2
layer
(a) Fast LM: Dual Cache
(b) Elastc-Cache (Chns)
Figure 2: Illustration of the Key-Value cache method for diffusion LLMs. (a) The fast-dLLM (Wu et al., 2025) block-wise decoding method caches the Key- Value
tokens outside the current block each step. The KV cache is updated after completing block of decoding.
(b) Our proposed method, Elastic-Cache, caches the key-value of tokens outside a sliding window that flexibly moves through the sentence from left right each iteration. When the attention weights corresponding to the most-attended tokens (one for each layer) change significantly at a layer i, we start recomputing the KV cache from layer + 1 the last layer.
active prediction window (Fig. D.. (2) KV drift grows with depth, refresh should start at a boundary layer and apply only to deeper layers (Fig. [D). (3) The most-attended tokens typically shows the smallest KV change (Fig. Id), giving conservative lower bound for others,
its drift lightweight trigger for refresh (Fig. [T]. Fig. summarizes the pipeline. To the end, we proposed Elastic-Cache, flexible method for key-value caching in diffusion large language models. Fig. provides a visual representation of the overall pipeline of our proposed method.
3.2 SLIDING WINDOW DECODING AND KV CACHING
Formally, let I {1, 2,.,, N} represent all positions. At decoding step 1, let Dr denote newly decoded positions and M denote remaining masked positions, where M = M U Denotes
D* UH(Di as the set of all decoded tokens up to time step t. Initially, at = 0 we compute the attention for each layer
Q D.L. 12
K V..
0.1 K
0.1
A X
softmax
VO1 [
initialize KV cache:
(4)
dx
For each subsequence iteration / raagingg from to T, The model perform prediction for newly decoded position De and the remaining masked position M To enhance efficiency, we only per- form predictions for masked positions that are closest to the left and form a sliding window of size 3. denoted as M' Mf We also have that M tokens within the sliding window attend closely to one another, while those outside receive little attention. This allows safe reuse of cached KV for oft-of-window MASKs without affecting current predictions, At step i, attention computed only for tokens sliding window M¹
MY U D'. We observe that masked
Q Pivs-
(K
t.l (mT)
K My- Vil
K [M
A 6, [Mg-1] = softmax
a. 2
update cache:
(5)
Vdg.
V
While sliding window decoding shares similarities with Fast-dLLM's block-wise KV caching (wu al. 2025) (see Fig. it offers key improvements. By predicting nearby tokens together, it reduces cache loss for distant MASK tokens. In contrast, block-wisg decoding may miss MASK tokens near block ends, resulting in less efficient predictions due to overly aggressive context caching.
ATTENTION-AWARE KV CACHE UPDATE 3.3
The most important novelty of our proposed method is to automatically determine whether to update the KV cache to preserve accuracy while minimizing latency. Our method leverages the awareness of the model's attention weights identify when the KV cache undergoes significant changes. At
4

---

Published as a conference paper at ICLR 2026
Algorithm The Elastic-Cache Algorithm
1: Require: Prompt Xprompt, Sliding window size B. Update threshold 7, Generation length N.
2: Initialize: x"
P [MASK] ..... [MASK]]:
}:
P 4- (ength(Xuump))
3: t 1; D
p): M (P
P+N]: -
while M + do
M - M
@' 7 MM
H 1o [qq
f Embedding(xfog)) t
1..... do
7: if
then
I| Cache Update
H K.d Vid cache(T):
O K V PFN(H)))
9:
H H4l+1 st.
"IT-1 + MHA(Q Q2
Kdd vil
10:
else
I/ Cache Reuse
11:
H Ktl V - cacho(Q)):
Jop lep
Q
Kid
Tey 10y
V
FFN(H (H
H441 sti
[t) [TI-1 + (
Rd Vid
13:
-ossne.similarityyy
(ri 4)
14:
th << then
Start update cache from layer
15:
c- H get vet_cccched state(H (+)1
[2
16:
end if
17:
end
the most-attended token: T arg maNkEDs Egenp St. [g.h)]
19:
end for
20:
Decode new tokens: x*+1, Dt+1 + decode(x'
21:
Update state: M <- MM' Dett. T U'= 6+1+1
State Update
22:
end while
23:
return -1
time step and layer 1, we determine the token that receives the most frequent attention from other tokens based on the attention weights corresponding to the current model's 's prediction for M
Q JMM
f. [
T arg
Σ
Sti la.pp
where: std (M3)
softmax
(6)
REDSE
GEMs
Here, we focus solely on the most-attended token among the current decoded tokens D' This is because the remaining MASK tokens either fall within the sliding window of predictions or have negligible influence on the unmasking tokens (Fig. [T]. We obtain one most-attended token per layer and compile the of most-attended tokens, denoted T most-attended token for layer often overlaps with tokens from other layers, resulting in a relatively limited number of most-attended tokens being available at any given time.
(T In practice, the
T', besides being the tokens that have the most influence on the predictions' outcome, also signify the least changes among the cached decoded tokens (Fig. [t.. Therefore, we use T lightweight trigger for our cache update mechanism. Without updating all cached tokens, we only frequently update the most-attended tokens T to measure the degree of changes for all other cached tokens. Ideally, since T have the least change among the decoded tokens, we expect that when 7 change significantly, the rest the decoded tokens will also change significantly. Therefore, we add T to the sliding window at step t: T U M We then measure the changes in attention weights of T between the current and previous steps, and t 1, using cosine similarity.
||
|||T
St. 7
1 r=
if: ofd <>. Cosine Similarity: gel othewise
The changes in attention St directly affect the output of the current layer or the input of the next layer H4,+1 This implies that our cached values are diverging from the actual values, necessitating an update. When layer (" observes significant changes attention weights tf < 7, we initiate the update of the KV cache for the subsequent layers, starting from /" + and continuing until the last layer L. To achieve this, we initialize the hidden states of all cached tokens with the states H f4.11

---

Published as a conference paper at ICLR 2026
which have been saved and updated using patterns similar to K
k(,++
and VLL+1
Update state: H H [M
H
Initialize:
Q
a++1
K
L1+1
VLH+1
linear H (1+1)
(8)
[
We then update and overwrite the KV cache using the same process as initially 0, as described
in Eq. 4 If none of the layers satisfy ard < we continue to reuse our KV cache for future predictions.
We didn't directly compare the hidden state H and H H because their changes depend on various network components, The error in measurement could amplified by the divergence between the cached value and the actual value (including Key- Key-Value states).
On the other hand, the changes in attention weights are closely linked to the source of the change in Key-Value states, which is the bidirectional attention mechanism in diffusion LLMs. Intuitively, the changes in attention weights become significant when new decoded tokens receive high attention and alter the attention output computed in the past when they were still masked. Consequently, the changes in attention weights exhibit very similar patterns to the changes in Key- Value states during decoding, as illustrated in Fig. DP and Fig. Te.
Our approach is formally grounded in Theorem A.9 (Appendix), which proves that the most- attended token Tt. has KV drift bounded by Apc. S A + vdg
where the error term scales negligibly with typical transformer dimensions. This establishes that monitoring attention patterns of most-attended tokens provides a computationally efficient and theoretically sound proxy for overall KV drift.
R.VN
We use the syper-parameter y to set the trigger for automatic cache updates. As shown in Fig. the attention weights' cosine similarity landscapes influence this. A higher y results in more frequent and extensive cache updates across multiple layers, while lower y triggers updates less frequently. This flexibility allows us to effectively manage the trade-off between accuracy and latency. Our overall algorithm described in Algorithm I
4
EXPERIMENTS
4.1
EXPERIMENTAL SETUP
Implementation Details. All
single NVIDIA A100 80GB. We evaluate Elastic-
Cache on LLaDA-Instruc (Nie et al., 2025a). LLaDA-1.5 (Zhu et al. 2025), and multimodal 2021b), HumanEval (Chen et al. [02T).
LLaDA-V YYuu et al., 2025) across MBPP (Austin et al. MATH Hendryycks et al. 2021). GSMSK (Cobbe et al. Math (Zhang al. 2024b). Default hyperparameters: attention threshold 9=0.9 parallel-
2021J. Math Vista (Lu et al. (202)). and
decoding confidence c=0.9, cache block size 32. For fair comparison, we re-run LLaDA (Nie)
et al. 2025a) and Fast-dL.LM (Wu al. 2025) under the same hardware/software. Evalua- tion Framework and Metrics. We use 1m eval wal-harness (Gao et al. [2024). Throughput
is tokens/sec averaged until emitting, matching Fast-dLLM's protocol (Wu et al. (025).. Ac-
curacy metrics: GSM8K: 5-shot flexible.extrace
(Cobbe et al., 2021); MATH: 4-shot
math_verify (minerva.math) (Hendrycks et al., 2021); HumanEval O-shot with the Fast-
dLLM post-processing (Cen al. 2021 Wu et al. 2025]; MBPP 3-shot pass@l (Austin al.
2021b For LLaDA-V, we adopt the official pipeline with 1mms-eval 1 (Zhang et al.) 2024a
2024 Math Vista: gpt_eva 1 _score (Lu et al. (023);; Math Verse: gpt_eval _score
math htt.vverss.eetmmmiiiiiiio...oooo
(Zhang et al. 2024b).
Confidence-Aware Decoding. We employ confidence-aware decoding strategies from Fast- dLLM (Wu et al., 2025), which select only tokens with confidence scores exceeding a specified threshold (c), instead of unmasking a fixed number of tokens per step. as in the baseline Diffusion LLM. This straightforward yet effective approach accelerates Diffusion LLM inference by enabling more tokens to be predicted concurrently at each iteration, contingent upon the model's performance. Consequently, we concentrate on comparing the acceleration achieved by the KV caching method under the same decoding strategies.
4.2 PERFORMANCE AND EFFICIENCY EVALUATION
Across Tables I and 5 our proposed Elastic-Cacht delivers substantial throughput gains for diffusion LLMSS with minimal accuracy loss. By adaptively updating the cache only when neces-

---

80.97 Published as a conference paper at ICLR 2026
Table 1: Comprehensive benchmark results on the LLaDA-Instruc suite. Each cell shows accuracy (top) and decoding throughput in tokens/sec with relative speedup to the LLaDA baseline (bottom, blue: t/s orange: speedup). Highlighted cells denote the highest throughput and speedup per configuration. The highest accuracy is bolded.
Confident-Awar Decoding
Benchmark
Gen Length
LLaDA
LLaDD
Fast-dLL.M
Flastie-Cache
78.01
78.62
77.94
78.24
256
GSM8K (5-shot)
(1.0x)
(3.1x)
53.7 (7.7)
(8.2
77.10
77.33
74.83
77.71
512
3.6 (1.00)
(8.6
(1.3
90.1 (25.2X)
33.58
33.28
32.50
33.14
256
(1.00))))
25.8
(6.0x) x)
48.7 (5.1x)
MATH (4-shot)
37.20
36,82
35,70
36,60
512
(1.0%)
(344%)
52.8 (7.4%)
(7.9%)
40.85
42.07
37.20
40.24
256
33.0(100))
102.1 (3.1
99.8 (3.0x)
160.5 (1.8x)
HumanEval (0-shot)
43.90
43.29
45.73
46.34
512
(1.0%)
(2.x))
(4.3x)
100.7 (5.0x)
29.80
30.00
25.40
32,2
256
(1.0x))))
(3.6
(7.0x)
46.9 (7.3x)
MBPP (3-shot)
15.0
15.0
13.6
15.6
512
4.7 (1.0%)
(4.4x)
44.7 (9.5x)
63.0 (13.4x)
Table 2: Comparison with additional KV caching methods on GSM8K (5-shot, 512 tokens) using LLaDA-1.5. Each cell shows accuracy (top) and throughput in t/s with relative speedup (bottom, blue: t/s; orange: speedup).
LLaDA-1.5 dKV-Cache dLLM-Cache DeepCache (N=10) DeepCache (N=20) Elastie-Cache
81.35
67.02
83.1
81.4 60.9 (234x)
83.7
21.0)
(5.7%)
(6.5%)
58.4 (22.5x)
139.4
sary, it achieves a speedup of up to 45.1 x over the standard baseline. While maintaining accuracy within 1~2%% MATH and HumanEval, it also achieves higher accuracy GSM8K and MBPP. Compared to Fast-dI.LM (Wu et al. 2025), Elastic-Cache consistently attains greater tokens/sec at better accuracy. Elastic-Cacha also generalizes Dream-7B (Table achieving 21.4x and 5.5x speedups on GSMSK and HumanEval respectively.
As presented in Table D on LLaDA-Instruct Elastic-Cache reaches 90.1 t/s on GSMSK (512 tokens; 25.2x over baseline) at 77.71 % accuracy, surpassing Fast-dLLM' 44.0 t/s @ 74.83%. On LLaDA- 1.5 (Table 3), our approach yields even greater gains, including 4 on GSM8K with 512 Gen Length, with an accuracy of 81.35% (baseline 81.35%). This observation indicates that Flastic- Cache performs better when the model's predictions are more accurate. The reason behind this could be the close relationship between approach and attention scores. Intuitively, accurate predictions are associated with meaningful attention scores with fewer outliers, which makes our approach operate more smoothly.
We also observed that in most settings, Elastic-Cache provides higher throughput for longer gener- ation lengths, whereas this is the opposite for Fast-dLLM (Wu al. 2025). as it often experiences reduced throughput as the generation length increases. The advantages of our approach stem from the fixed-size sliding window and automatic cache update, which minimizes the dependency of throughput on the generation length.
In the multimodal setting (LLaDA-V; Table Elastic-Cache raises Math Verse-256 throughput to 32.3 t/s from Fast-dLLM's 30.3 t/s while maintaining 29.19% accuracy, demonstrating robustness beyond text-only tasks. The significant improvement of Elastic-Cache over baselines across various settings suggests that our method is broadly applicable and has high scalability potential.
Comparison with Additional KV Caching Methods. To further validate the effectiveness of Elastic-Cache, we compare against dLLM-Cache (Ma et al. 2025) and DeepCache (Ma et al. (0244) on GSM8K (512 tokens) with LLaDA-1.5 in Table DeepCache uses fixed-interaal cache updates with intervals N = 10 and N = 20. Our method achieves 139.4 t/s at 83.7% acc, significantly out- performing dLLM-Cache (16.84 84 t/s. 80.97%) and DeepCache variants (58.4-60.9 t/s, 81.4-83.1 demonstrating the advantages of adaptive, attention-aware cache over fixed-schedule approaches.
7

---

Published as a conference paper at ICLR 2026
Table Comprchensive benchmark results on the LLaDA-1.5 suite. Each cell shows accuracy (top)
and decoding throughput in tokens/sec with relative speedup to the LLaDA baseline (bottom, blue: t/s; orange: speedup). Bold cells denote the highest throughput and speedup per configuration.
Confident-Aware Decoding
Benchmark
Gen Length
LLaDA-1.5
LLaDA-1.5
Fast-dLLM
Elastie-Cache
80.36
80,44
80.59
81,50
256
6.7
22.5 (3.3%)
(7.6
(8.0()
GSM8K (5-shot)
81.35
81.88
80.82
81.35
512
(L.0%)
(6.66)
36.8 (14.1x)
(45.
33,52
33,60
32.74
33,50
256
(L.00
(2.6%)
(5.2x)
(6.5
MATH (4-shot)
35.63
35,56
33.68
35,36
512
(1.00))))
20.3 (4.0x)
(8.88)
(14.9)
43.29
42.68
34.75
36,59
256
7.0 (L.0x)
(2.5x)
(27xx)
(3,00)
HuamaEval (0-shot)
40.85
39.63
36,59
37,80
512
31200).)))
(.1
(4.8%)
16.8 (5.3%)
38.00
38.00
34.60
41,20
256
(1.0x))))
(5.88)
(1.6()
32.7 (13,5x)
MBPP (3-shot)
38.20
38.60
36.20
39,00
512
(.0().)))
(11.5)))))
(17.8))
32.8 (32.88)
Table 4:
Comprehensive benchmark results the "Dream-vOl (0-Base-7B" suite. Each cell shows accuracy (top) and decoding throughput in tokens/sec with relative speedup to the Dream baseline (bottom, blue: t/s; orange: speedup).
Confident-Aware Decoding Fast-dLLM Elastic-Cache
Benchmark
Gen Length
Dream
76.0
74.1
75.6
GSM8K (5-shot)
512
(1.0%)
(5.9
169.4 (21.4)
54.3
51.2
56.7
HumanEval (0-shot)
512
(1.0
50.1 (2.9x)
95.2 (5.5%)
Table 5: Performance and Speedup Comparison of LLaDA-V on Math Vista and Math Verse. Each benchmark presents results from LLaDA-V (base) using Fast-dLLM, and our method.
Length
Vash
Math Verse
Base Model (LLaDA-V)
Fast-dLLM
Elastic-Cacht (Ours)
Buse Model (LLaDA-V)
Lust-ALLMM
Elustie-Cache (Ours)
256
54.6 (2.6(
55.9 (28.7)
55.9 (29.))
30.1 (2.1)
26.8 (30.3)
29.2 (32.3)
512
(1.9)
54.1 (23.7)
55.8 (24.1)
26.9 (2.0)
25.5 (28.1)
29.2 (308)
4.3
ABLATIONS
We ablate key choices: 1) Cache update threshold y. 2) sliding window size B, and 3) prefill and generation length, to expose speed/accuracy trade-off's and justify defaults.
Cache Update Threshold (7). Table illustrates the sensitivity of our proposed method to the parameter As y is used to control the frequency of cache updates, a consistent decrease in y lcads to an increase in throughput. However, there is also a trend of decreasing accuracy as throughput increases. The trend is more consistent for the LLaDA-1.5 model, while for LLaDA, the accuracy at peak (? 0.9) is higher, but the throughput is lower.
Sliding Window Size (3). Fig. 3a shows that our accuracy is stable across various / and close No-Cache until 3 =64; beyond that LLaDA's tendency to emit EOS early degrades results (You et al. 2025). Throughput, however, is sensitive to f: larger windows enable more parallel prediction (fewer iterations, lower latency), but overly large reduces cacheable MASK tokens, raising per- step compute and latency.
Sliding Window vs. Block-Wise. When switching Elastic-Cache to block-wise decoding (Fast- dLLM-style) (Fig. 3a). our accuracy is often similar to No-Cache, but short blocks hurt accuracy and throughput diverges. Our sliding window groups nearby MASK tokens that strongly attend to each other, whereas block-wise caching over-aggressively freezes distant MASKs, harming small- block predictions. Our Elastic-Cacht ' automatic cache refresh detects divergent tokens and updates them, preserving accuracy at the cost of some throughput.
8

---

Published as a conference paper at ICLR 2026
027.75
-130
GSHBK Accuracy
BG CECT
1.0%
Throughouu (to (asina
10.25
43.5%
7 Spenets
Througho (tos
GSMUK (--hhet)
6.6x Spesoup
72
Marry dar Forglied
sliding kize
Ga mma
Threshold
(a) Sliding window and block size
(b) Cache update frequency.
(c) Confident-aware decoding.
Figure 3: Ablation study and analysis of our proposed method. (a) Ablation study of our sliding window mechanism compared to block-wise decoding, (b) Analysis of cache update frequency under varying 7. The blue and orange lines represent accuracy and throughput, respectively. The numbers along the lines indicate the frequuncy of cache updates, assuming no baseline. (c) Analysis of cache update frequency under confident-awand decoding with varying c.
Table 6: Performance under different denoising schedules (LLaDA-1.5, GSM8K).
Method
tok/step
tok/step
tok/step
3.25 tok/step (c=0.9)
5.12 tok/step (c=0.7)
81.4
79.8
67.5
81.9
79.6 26.6 (1.0x)
Baseline
(1.0%)
(1.0%)
(1.0%)
(1.0))
80.5
77.3
64.7
80.8
80.0
Fast-dLLM
8.5 (3.3%)
(3.0%)
27.3 (2.7x)
36.8 (2.1x)
46.9 (1.8x)
82.6
78.1
69.9
81.4
81.2
Elastic-Cuche
86.5 (18.
149.8 (14.5x)
117.2 (6.8x)
167.6 (6.3%)
Prefill and Generation Length. Table 8a and Table 8b provide insights into the impact of prefill length and generation length on the overall speedup. Notably, both Fast-dLLLM and Elastic-Cache experience a decrease in throughput the prefill length increases from 3-shot to 8-shot. However, Elastic-Cache exhibits a remarkable speedup and consistently high accuracy across different prefill lengths. Moreover, the throughput of Elastic-Cache increases with generation length, highlighting its unique scaling properties.
4.4 ANALYSIS
Cache update frequency. Fig. 3b and Fig. 3e illustrate the frequency of cache updates performed and e. The proposed method maintains a very
by Elastic-Cache under varying hyper-parameters low cache update frequency across different values of y (Fig.3b). the cache update frequency increases to only 20% compared to the baseline without cache. More- over, increasing the model's confidence and accuracy (with c, Fig. 3cb enhances Elastic-Cache's effectiveness, and reduces the cache update frequency.
extreme cases, with y = 0.95,
Tunable Speed-Accuracy Tradc-off. The cache update threshold y directly determines the balance (Table 7). An excessively high y could lead unnecessary cache updates, resulting in decrease in speedup without any improvement in accuracy. Conversely, smaller y value could guaran- tee speedup while sacrificing accuracy. The optimal value for 7 to maximize both accuracy and throughput depends on the model's prediction. Models with higher accuracy tend to have the best value, which closer to 1.0 (Table
Scaling Properties. Elastic-Cache scales greatly with the generation length and the power of the base model. Increasing the generation length slows down the baseline performance but speeds up Elastic-Cache (Tables (b). Moreover, Flastic-Cache effectiveness is highly dependent on the accu- racy of the model's predictions (Table [ Table 3 Fig. 3c). This indicates that Elastie-Cache can effectively scale with the size of the model and the size of the training data, LLMs generally improve when they scale up.
Robustness Across Denoising Schedules. We test Elastic-Cache LLaDA-1.5, GSMSK under varying denoising schedules by controlling average tokens decoded per step (Table G While the baseline decodes 1 token/step, confidence-awee decoding (Wu et al. 2025 increases this 3.25 (e 0.9) and 5.12 (e = 0.7). As decoding becomes more aggressive, KV drift grows, demand- ing more frequent cache updates. Flastic-Cache adapts by raising update frequency from 5.6% to

---

Published as a conference paper at ICLR 2026
Table Impact of attention threshold on accuracy and speedup under GSM8K (5-Shot) for LLaDA and LL with generation length of 512.
Elastic-Cache (Ours)
Model
No Cache
Fast-dLLLM
?
y = 0.7
0.8
0.85
Y = 0.9
y= 0.95
77.10
74.83
71.57
73,46
74.30
74,68
77.71
76.72
LLaDA
3,6(1.x))
(1..x))
109.9 ()0.5)
108.7 (30.2x)
103.9 (28.9x)
99.1 (27.5x)
91.5 (25.4x)
0x)
81.35
80,82
76.04
77.63
79.45
80.21
81.35
83.02
LLaDA-1.5
2.6(1.0X)
36.8 (14.2))
142.7
138.6 (53.3%)
(50.5)
129.9 (50.0x)
117.2 (15.1
98.1 (37.8x)
Table 8: Comparison between Elastic-Cache and Fast-dLLM when varying Prefill and Gen. Length.
(a) Impact of few-shots Accuracy and Speedup Under GSM8K (1024) for LLaDA.
(b) Impact of generation length on Accuracy and Speedup Under GSM8K (5-Shot) for LLaDA.
Model
3-shot
5-shot
8-shot
Model
256
512
1024
73.77
76.04
75.36
77.94
74,83
76.04
Fast-dLLLM
Fast-dI.LM
28.5 (1.0))
(1.0x)
20.8 (1.0%)
53.7(1.0x)
(1.00)
(1.00)
75.13
75.21
75.28
78.24
77.71
75.21
Elastic-Cache
Clash
(6.5%)
169.8 (6.8x)
143.9 (6.9%)
58.0(1.130)
(2.5x(
169.8 (6.8%)
17.2%, preserving accuracy, unlike Fast-dLLM, which suffers under fixed schedules. This highlights Elastic-Cache's robustmess to denoising variations without manual tuning.
RELATED WORK
Diffusion Language Models. Classical diffusion models excel in continuous domains: im- (Ho et al., 2020: Dhariwal & Nichol 2021 Rombach et al.
ages
2022). audio ( YYang et al.] 2022a b). building
2023
Huang al. (023), and video (Xing et al.; 2024 Ho et al.
the
seminal formulation of (Soh--Dickstein et al. (2015). Adapting diffusion to discrete text has fol- lowed
paths (Austin et al. 2021a Hoogeboom et al. 202 a Campbell et al., 2022, Sun et al. 2022), refined via score matching, ratio methods, and reparame- terization (Meng et al. [202 Lou & Ermon 2023 Zheng et al. 2023). with recent work unifying
these views (Sahoo et al. 2024 Shi et al. 2024 Ou et al. 2024: ZZeng et al., (024). Early NLP
systems validated these ideas (He et al. 2022; Li et al. 2022 Gong et al. 2022) and explored semi-autoregression (Han et al. 2002). Masked diffusion approaching autoregressive quality (Sa- hoo et al. 2024) enabled scalable models (LLaDA) competitive with LLaMA (Nie et al., 2025a 2024 Touvron et al. [2023a] Dubey et al.| (024)), with further gains from AR adaptation and in- struction tuning (Gong et al., 2024| Zhu et al. 2025 Ye et al. 2025) ) The paradigm now spans
multimodal/strietureed domains (You et al. 2025 Yang al. 2025 Yu et al. 2025 Wang et al.
2024a]b Kitouni et al. (023).
Acceleration Techniques for Large Language Models. KV caching underpins efficient trans- former inference (Vaswani et al., 2017: Poope et al.] (202)) complemented by GQA, RoPE, and modern LLM optimizations (Ainslic
2023] Su al. 2024 Touvron et al. 2023a b Dubey
et al. 2024). Diffusion LLMs complicate caching due to bidirectional attention and evolving repre-
sentations; dedicated methods include Fast-dILLM (Wu et al. 2025), dKV-Cache (Ma et al. 20255
and DeepCache (Ma et al. (0024). Orthogonal accelerations exploit parallel/non-AM generation (Gu
cet al. 2017|Xiao et al. 2023), block-wise diffusion (Arriola et al. 2025), fast sampling (Chen et al.] 2023). test-time scaling (Ramesh & Mardani] 2025), and consistency models (Kou et al.) 2024))
However, most rely on temporal heuristics or fixed thresholds, leaving attention patterns underused.
6 CONCLUSION
We presented Elastic-Cache. a training- free, architecture-agnosii policy that makes KV caching in diffusion LLMs adaptive along two axes: when to refresh (via an attention-awar aware drift test) and where to refresh (via a depth-selective update starting at learned boundary layer). By block-caching distant MASK tokens, reusing shallow-layer caches, and refreshing only when the most-attended token indicates meaningful state change, Elastic-Cache removes large amounts of redundant QKV work. Across decoding steps, this yields substantial latency reductions with negligible impact on generation quality, addressing a key deployment bottleneck for diffusion decoders. Looking ahead, plan to refine drift thresholds with learned predictors, formalize guarantees linking attention pat- terns to KV drift, and explore interplay with speculative decoding or other hardware-ane schedul- ing, extending the same principles autoregressive L.MMs and multimodal diffusion frameworks.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work targets inference-time efficiency for diffusion LLMs and does not introduce new data collection or model training. All evaluations use publicly available datasets and third-party check- points under their original licenses, no personally identifiable information is processed. While faster decoding can lower the cost generation and thus broaden access, may also amplify misuse. We neither change safety filters nor attempt to bypass alignment constraints of the underlying models. We will document evaluation prompts and tasks, follow the usage policies of model providers, and encourage human oversight for downstream deployments, especially in high-stake applications.
REPRODUCIBILITY STATEMENT
Clastic-Cache is training-free free and defined by small set of inference hyperparameters: the attention similarity threshold 7. block size and generation length. We will release code, configs, and scripts to reproduce all results: (i) reference implementations of Attention-Aware and Layer-Aware KV updates with ablation; (ii) exact prompts/datasets, metrics, and other criteria; and (iii) environment specs (CUDA/driver, framework versions) and hardware details (GPU type, batch sizes). We report wall-clock latency and accuracy metrics for each setting, and provide logs to tables/figures from raw traces.
ACKNOWLEDGEMENTS
This research was supported by the MBZUAI-WIS Joint Program for Al Research.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.00 087744 2023.
Joshua Ainslie, James Lee-Thorp Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. Gqu: Training generalized multi-query transformer models from multi-head check- points. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Pro- cessing, pp. 4895-4901, 2023.
Marianne Arriola, Aaron Gokaslan, Justin T. Chiu, Zhihan Yang. Zhixuan Qi, Jiaqi Han, Sub- ham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregres- sive and diffusion language models, 2025. httt/htps://aart
org htt.abb/./55552....
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:1 1798--17997 202
Jacob Austin, Augustus Odena, Maxwell Nye. Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoe Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.0777 2021b.
Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. Advances in Neural Information Processing Systems, 35:28266-28279, 2022.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph. Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint 2rXiv:2107.03374 2021.
Zixiang Chen, Huizhuo Yuan, Yongqian Li. Yiwen Kou, Junkai Zhang, and Quanquan Gu. Fast sampling via de-randomization for discrete diffusion models. arXiv preprint arXiv:2312.09193, 2023.
11

---

Published as a conference paper at ICLR 2026
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. 2021.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Leuman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21755 2024.
Bolin Gao and Lacra Pavel. On the properties of the softmax function with application in game theory and reinforcement learning. arXiv preprint vrXiv:/704..00 00805, 2017.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter; Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL httthtttt///eeette
org/ records/
12608602
Shansan Gong, Mukai Li, Jiangtao Feng. Zhiyong Wu, and LingPeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models. arXiv preprint arXiv:22/0.08933, 2022.
Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, al. Scaling diffusion language models via adaptation from autoregressive models. arXiv preprint arXiv:2410. 17891, 2024.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. Non-autoregressivv neural machine translation. arXiv preprint arXiv:1711. 011.7711.02212222 2017.
Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. Ssd-Im: Semi-autoregressive simplex- based diffusion language model for text generation and modular control. arXiv preprint arXiv:2210.174322 2022.
Zhengfu He, Tianxiang Sun, Kuanning Wang, Xuanjing Huang, and Xipeng Qiu. Diffusion- bert: Improving generative masked language models with diffusion models. arXiv preprint arXiw:2211.15024 2022.
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Mcasuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2 103.03/03333874 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint 10..03220.002303, 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. Advances in neural information processing systems, 35:8633- 8646, 2022b.
Emiel Hoogeboom, Alexey Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. arXiv preprint arXiv:2110.02037, 2021a.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in neural information processing systems, 14:12444-1246 2021b.
Rongjie Huang, Jiawei Huang. Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an- anddiio:: Text-to-audio generation with prompt-enhanced diffusion models, 2023. URI /tt///aavvvv///
12661
12

---

Published as a conference paper at ICLR 2026
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does bert learn about the structure of language? In ACL 27t9-57th Annual Meeting of the Association for Computational Linguistics, 2019.
Ouail Kitouni, Niklas Nolte, James Hensman, and Bhaskar Mitra. Disk: A diffusion model for structured knowledge. arXiv preprint urXiv:23/2.05253, 2023.
Siqi Kou, Lanxiang Hu, Zhezhi He, Zhijie Deng. and Hao Zhang. Cllms: Consistency large language models. arXiv preprint arXiv:2403.008333 2024.
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. Revealing the dark secrets of bert. arXiv preprint arXiv: 085933, 2019.
Bo Li, Peiyuan Zhang, Kaichen Zhang, Fanyi Pu, Xinrun Du, Yuhao Dong, Haotian Liu, Yuanhan Zhang, Ge Zhang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Accelerating the development of large multimodal models, March 2024. URL [tttps://iithut com/
EvolvingLMMs- Lab/ Imms-eval
Tianyi Li, Mingda Chen, Bowei Guo, and Zhiqiang Shen. A survey on diffusion language models. arXiv preprint arXiv:2508. 10875, 2025.
Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. Diffusion- Im improves controllable text generation. Advances in neural information processing systems, 35: 4328 4344-433.220 2022.
Zhiyuan Liu, Yieun Yang, Yaojie Zhang, Junjie Chen, Chang Zou, Qingyuan Wei, Shaobo Wang, and Linfeng Zhang. dllm-cache: Accelerating diffusion large language models with adaptive caching. arXiv preprint 0et.:22506006295. 2025.
Aaron Lou and Stefano Ermon. Reflected diffusion models. In International Conference on Machine Learning, pp. 22675-22701 PMLR, 2023.
Guanxi Lu, Hao Mark Chen, Yuto Karashima, Zhican Wang, Daichi Fujiki, and Hongxiang Fan. Adablock-dllm: Semantic-aware diffusion Ilm inference via adaptive block size. arXiv preprint arXiv:2509.26432. 2025.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deepeache: Accelerating diffusion models for free. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15762-15772, 2024.
Xinyin Ma, Rumpeng Yu, Gongfan Fang, and Xinchao Wang. dkv-cache: The cache for diffusion language models. arXiv preprint arXiv:2505. /5781,
Chenlin Meng, Kristy Choi, Jiaming Song, and Stefano Ermon. Concrete score matching: General- ized score matching for discrete data. Advances in Neural Information Processing Systems. 35: 14532-34545, 2022.
Shen Nie, Fengqi Zhu, Chao Du, Tianyu Pang, Qian Liu, Guangtao Zeng, Min Lin, and Chongxuan Li. Scaling up masked diffusion models on text. arXiv preprint arXiv:2410.18514, 2024.
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffuuson models. arXiv preprint MXXv:25522.0992 2025a.
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Rong Wen, and Chongxuan Li. Large language diffusion models, 2025b. URL https:
//axxiv.or www.v.rrrg/oo abs / 2202 09992
Jingyang Ou, Shen Nie. Kaiwen Xue, Fenggi Zhu, Jiacheng Sun, Zhenguo Li. and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. arXiv preprint X4X::24066037366 2024.
13

---

Published as a conference paper at ICLR 2026
Reiner Pope. Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek. Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. Proceedings of machine learning and systems, 5:606-624, 6246666622226 2023.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language under- standing by generative pre-training. 2018.
Vignav Ramesh and Morteza Mardani. Test-time scaling of diffusion models via noise trajectory search. arXiv preprint arXiv:2506.05. 164, 2025.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology: What we know about how bert works. Transactions of the association for computational linguistics, 8:842-866, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models, 2022. URL pattps://arxiv.cr
abs/2112, 217.
Subham Sekhar Sahoo, Mariannn Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. arXiv preprint arXiv:2406.07524, 2024.
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K Titsias. Simplified and gener- alized masked diffusion for discrete data. arXiv preprint (Bt.:2406.04329 2024.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing. pp. 2256-2265, PMLR, 2015.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En-
hanced transformer with rotary position embedding. Neurocomputingg, 568: 127063, 2024.
Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based continuous-timg discrete diffusion models. arXiv preprint arXiv:2211.16750,, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint urXiv:2302.13971 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Xinyou Wang. Zaixiang Zheng, Fei Ye, Dongyu Xue, Shujian Huang, and Quanquan Gu. Diffusion language models are versatile protein learners. arXiv preprini 288. 2024a.
Xinyou Wang, Zaixiang Zheng, Fei Ye, Dongyu Xue, Shujian Huang, and Quanquan Gu. Dplm-2: A multimodal diffusion protein language model. arXiv preprint arXiv:2410.13782. 2024b.
Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. Fast-dllm: Training- acceleration of diffusion 1lm by enabling kv cache and parallel decoding. arXiv preprint arXiv:2505. 2025.
Yisheng Xiao, Lijun Wu, Junliang Guo, Juntao Li, Min Zhang, Tao Qin, and Tie yan Liu. A survey on aun-auuorreresssiv generation for neural machine translation and beyond, 2023. URL URLhttps:
/aaxiiv.org/abs//200000 09269
Zhen Xing, Qijun Feng. Haoran Chen. Qi Dai, Han Hu, Hang Xu, Zuxuan Wu, and Yu-Gang Jiang. A survey video diffusion models. ACM Computing Surveys, 57(2):1-42, 2024.
14

---

Published as a conference paper at ICLR 2026
Dongchao Yang, Jianwei Yu, Helin Wang. Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. Diffsound: Discrete diffusion model for text-to-sound generation, 2023. URL hetps:
httpg//bb//207.c.. 2207.09983
Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang. Mmada: Multimodal large diffusion language models. arXiv preprinl arXiv:2505. 15809, 2025.
Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang. Zhenguo Li, and Lingpeng dream
Kong. Dream 7b, 2025. URL http:: /twpa///////pppppp github
Zebin You, Shen Nic, Xiaolu Zhang, Jun Hu, Jun Zhou, Zhiwu Lu, Ji-Rong Wen, and Chongxuan Li. Llada-v: Large language diffusion models with visual instruction tuning. arXiv preprint arXiv:2505.16933, 2025.
Runpeng Yu, Xinyin Ma, and Xinchao Wang. Dimple: Discrete diffussion diffusion multimodal large language model with parallel decoding, 2025. pRRtp:::s://axtit org/ /2055. 16990
Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang. Jingkang Yang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Reality check on the evaluation of large multimodal models, 2024a. http://pp////aiiooooo//o/aaa///s
12772
Renrui Zhang. Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal I1m truly see the diagrams in visual math problems? In European Conference Computer Vision, pp. 169-186.
Springer, 2024b.
Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming- Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. arXiv preprint DrXiv:2409.02908, 2024.
Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. A reparameterized discrete diffusion model for text generation. ArXiv, abs/2302.057311 2023.
Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chunwei Wu, Jun Hu, Jun Zhou, Jianfei Chen, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Llada 1.5: Variance-reduced preference optimization for large language diffusion models, 2025. URL
org/ abs/
22555. 19223
15

---

Published as a conference paper at ICLR 2026
APPENDIX
CONTENTS
Theoretical Validation for Elastic-Cachel
17
A.I
Notation and Sctup
17
Background Lemmas and Assumptions
17
A.3
Layer-Wise KV Drift Monotonicity
18
Attention Concentration and Drift
21
A.5
Implications for Elastic-Cache
24
Detailed Experiment Setup
24
Extended Experimental Analysis
25
Comprchensive Hyperparameter Sensitivity
25
Memory and Computational Overhead
26
Scalability Analysis
27
C.4
Block-Caching Mechanism
27
Runtime Adaptation of Layer Boundary
28
C.6
Validation of Most-Attended Token Heuristic
28
Multimoall Extensions
28
C.8
Comparison with Consistency Models
28
Use Large Language Models
29
Sample Response
29
16

---

Published as a conference paper at ICLR 2026
A THEORETICAL VALIDATION FOR ELASTIC-CACHE
A.1 NOTATION AND SETUP
L: number of transformer layers, indexed by e f1..... L
T: total denoising steps, indexed by t E (0...., ,
N: sequence length
d: hidden dimension; dg. dy: key and value dimensions
H E IR:: hidden state of token i at step f, layer &
K V. Rd IR:: key and value of token
S E N attention weights at step t, layer
Det: decoded token positions up to step 1
M masked token positions at step
My sliding window size over masked positions
0% EqEMt
St. 9.1 total attention token receives
AH; H H H.-L. 1,f change in hidden state
At. := N Σ | ||2 average hidden state drift
A max == max;
AH, ||e maximum hidden state change
t.f. D4. == Oipi.e ma O
t.f 2 0: Attention Gap
A.2 BACKGROUND LEMMAS AND ASSUMPTIONS
Lemma A.1 (Lipschitz Continuity of Softmax). Based on the Proposition 2 in Gao Pavel 2017).
the softmax function "R" n-1 defined by
exp(2;)
o(z);
(9)
E exp(zy)
L-Lipschitz continuous with respect to the &g norm:
[|o(()) azz))( ||z z/112
V2, E IR"
(10)
Assumption A.2 (Bounded Representations). At each layer and step t: H t.
S Rs
Assumption A.3
(Lipschitz Network Components). The projection matrices satisfy [ ( W z Wmax- The feedforward network at layer is Trnx-Lipschiz
continuous.
Assumption A.4 (Progressive Unmasking). At each step t, non-empty subset D C M is unmasked: |D<t| increases and M 1
Assumption A.5 (Layer- Wise Representation Dynamics), There exists 8 E (I. and func- tions ft(t) + as T for < l" such that:
Shallow layers (l ))): The expected hidden state change for decoded tokens vanishes: For L E":
E H
H
liE ((( +
Deep layers (E > l²): The expected change remains bounded away from zero:
lim inf E8 H
-
) >
This reflects that early layers encode local lexical patterns that stabilize quickly, while deep layers encode semantic relationships that continue evolving (Kovaleva et al.] 2019 Dawahar et al. 2019 Rogers al. 202T). Our experiments validate this (Fig.
17

---

Published as a conference paper at ICLR 2026
Assumption A.6 (Attention Concentration). The attention gap is non-negligible fraction of total attention mass:
r ス c MM
(11)
for some constant e > 0 independent of N,t,e. Definition A.7 (KV Drift). The KV drift at layer l, step for token is:
A
|
1,8
11 + Vi v
(12)
Average drift over decoded tokens: A
EiEDe
A
A.3 LAYER- WISE KV DRIFT MONOTONICITY
This theorem formalizes the observation that KV drift increases with layer depth, providing theoret- ical justification for our layer-aware cache refresh strategy that selectively recomputes deeper layers while reusing shallow-laye -laer caches. Fig. Th empirically validates this monotonicity property.
Theorem A.8 (Layer-Wise KV Drift Monotonicity). Under Assumptions A.2 A.5 there exists transition layer E (1,....,L) L} such that for sufficiently large (when most tokens are decoded):
E - 1,
(13)
Proof. Step 1: Relating KV Drift to Hidden State Drift.
The key-value projections at layer 8 are:
K.C W H
(14)
V H.
(15)
By the triangle inequality and Assumption
A.3
(I ||, [ ||2 S W Wmax):
K
t.
K
I|W (H)
H 1 ll2
||W ll H H 1
W max
AH
(16)
Similarly for values:
lIV v.- = maxx AH
(17)
Therefore:
K
K
212 + IIve V2-L1112 S 2W max AAH
(18)
Step 2: Layer Recursion for Hidden States.
At layer f, the transformer block computes:
H +-- H + Attnn (Q. K4C (
- FFN' + Att'(.)) (-))
(19)
where the attention output is:
Atn' (Q. k v(d Σ st
(20)
The change in hidden state at layer e satisties:
|A ||2 [++ H -1-11++11 1|2
S |AH 1 + ||Autn' (t) - Attn (t 1)112
FFFN@(input" FFN (inputt-)
(21)
18

---

Published as a conference paper at ICLR 2026
By Assumption A.3
the FFN is LivN- -ipsccitt
|FFN FFN (input' ll2 I LIEN LoeN||inpue |lipput' input"-11] 12
(22)
The FFN input H. + Atun A)tn(()) (-). so:
|input imput' S |AH 112 + ||Att((t) Attn' 1)ll2
(23)
Therefore:
|A 112 S (1 +1+ LYEN) | AH + (1 Tren)llAtt((()) Attn' (t
1)
(24)
Step 3: Bounding Attention Output Change.
Denote Altii
||Attn Attn' (t- 1)ll.. We decompose:
Σ S 1.
V
N
"
(25)
f=1
Taking norms and applying triangle inequality:
A
Σ S
+
S
'112
(26)
Step 3a: First term (value changes). Since
E3 S
(attention weights sum 1):
Σ L I
1,
S W max
AH ll
(by Assumption A.3
W Wmaxx B B ||A [|]]
$ W Wmax A
(27)
Step 3b: Second term (attention weight changes).
By Mtuchy-Schwarc
E, laylbs I
CΣ, la,1) max, bj
By Assumption A.2
1|a S
Wmax
Therefore:
N
N
1 1 1 Σ
S W Wmasc R
Σ 1.
-
11-
(28)
jml
By the inequality ||v|]| S vnlllvll2:
Σ S 1 VN -
(29)
By Lemma A.I (softmax is 1-Lipschitz in E2):
| - 112 S ||z L. 2 1- 1
(30)
2. 1. with ff 1 Q. K.C. where 2," (2,,
2
2. Va
19

---

Published as a conference paper at ICLR 2026
Step 3c: Bounding logit changes. For each component:
1-1 1Q
t.
K
Q
111 k;-1.9)
dd
1Q (Kj
k
(Q,
-
Kg-
(31)
By Cauchy-Schwee Schhar- and the bounds from Assumptions A.2 A.3
2/,
[W mamax
Re
Wmax
AH 12 + Wmax AHH W Wmax
vd
W max Rs
[| 1. 12 + |AH f. [|2]
dd
2W max Re
<
max |A 1l
(32)
vd
Taking & norm of the logit vector:
12
2;
1.6
12
1. 1,1,112
2i.j
2W2
Re 2
max
(max || AH 112)
(33)
dl
Therefore:
2W2 RAVN
2;
2.-1.61
112 5
max AH
(34)
Vdh.
For typical sequences where max; AH
112
O(A:)):
2W2
R. R&VN
1
112
max
Are
(35)
ddg
Step 3d: Combining, Combining the bounds from Steps 3a-3c:
2W2
R. REVN
tt,s atm
I W WmaxxA Are + Wmax Re R
A
Vdl
W Wmaxx Ale (1+
2W2
R&N
=
max
vdk
(36)
Define:
2W2 max
RRN O
W2 max
R2N
Caun (f)
(37)
Vile
Vile
Then:
A
1.8 aim
W Wmnxx1 + Catt Canm())) (
(38)
Step 4: Recursive Bound on Hidden State Drift.
Substituting equation 38 into equation 24
AH lll < + L AH . 112 + (1 + LFEX) W Wmax(1 (1 + Cat (2))
(39)
Taking averages over all tokens:
A 111+1 & [(1 TTIN) (1 + TTIN) W Caun (4)
(40)
20

---

Published as a conference paper at ICLR 2026
Define the layer-dependent amplification factor:
Ae = (1 + LeEN) [1 + Wmaxx( (1 + Caun Caun(e)) (f))
(41)
Then:
A A
(42)
Step Layer-wise Accumulation by Induction.
By induction on &
4
<
A 41
A
(43)
Since A 1, drift accumulates multiplicatively across layers.
Step 6: Applying Layer- Wise Specialization.
By Assumption A.5
Shallow layers (L < {*): A 1.4 Se((() + +
Deep layers (E > {"): lim inf, -T A 20>0 >0
By equation 18
A 6.
E
S 2W max A
(44)
Dect
iEDKI
Therefore, for sufficiently large and any l S 8" < Cl:
E - 2W max (1) 0
(45)
E 4 2 2W Cr >
(46)
This establishes:
E E < EEA'
ve <l
(47)
A.4 ATTENTION CONCENTRATION AND DRIFT
Theorem A.9 (Attention Concentration and Drift). Let T arg maX Ege St..t be the
most-attended token at layer L, step t. Under Assumptions A.2 A.3 drift bounded by:
the most-attended token has
A
ALE +
(48)
where ALe
DES EiEDa Alt is the average drift and ee
O
Vdd ReVN
Proof. Step 1: Bounding Attention Weight Changes.
We derive how attention weights S 9.6 change when hidden states change.
Step la: Logit change. The attention logits are Zq.h vdu
1 Qg K₁ where:
Q = -W 9: K₁ = WK H
(49)
The change in logits between steps t and is:
f. 2
1-1,f
1
Q ktt
Qt Kt-,g
(50)
Vdk
21

---

Published as a conference paper at ICLR 2026
Using the identity ab d'b' a(b 8) + (a a')8':
10 (K) k + (Q, -Q-1o)) k-
(51)
VT
Step 1b: Apply Cauchy-Schwant inequality. Taking absolute value and applying Cauchy-Schward
[||
2|2
K
13-1.111 --
q.k
9.
Vdkk
=1QQ,
Q
Q 1la
K 11 [|2]
(52)
Step Ic: Bound projection norms. By Assumption
A.2
H 1l2 S Re for all i,
By Assumption
A.3
[|oll2, [ S Wmax.
Therefore:
1 2 - [Well la
2 1l2 2 Wmax
(53)
K L
k 2 S [W laa
H f 112 S W W,
(54)
K
K (-(-, 2 E [W 2l
H
H
H
H S W max | ll2
(55)
Q Q Q 2 S W max AHHll2
(56)
Substituting these bounds:
1
29
[ Waxxx Rs W Wmax |HH + W Wmax |AH2l2 W Wmax
vdl
W Rs
|| 2 + |HH
(57)
vdg.
Step 1d: Use maximum drift. Since
|AH, S Aax for all i:
2W2 Rs
q.k
1-18
g
dd
max
Max
(58)
Step le: Compute E norm of logit vector The logit vector for query is 2g = (2g.1...,, E R.
By the previous bound applied to each component:
12
12-2
3g
2W2
Rd
max
A 2
ITy
max
k=
4W R2
N
(59)
dk
max
Taking square root:
2w2 max
R REVN
1-11111/
<
ddk
max
(60)
Step 1f: Apply softmax Lipschitz property. property. By Lemma
A.I (softmax is I-Lipschitz in lg norm):
2W2 max
Re RIVN
|| 9.:
4.:
-
<
2
1,4
1
Vdl
Amax
(61)
22

---

Published as a conference paper at ICLR 2026
Step Ig: Convert lo norm. Since l|v/loo I livl22 for any vector
2W max ReVN
S -s qube
$
A
max
(62)
vdx
Step Change Total Attention Received.
For token k. the change in total attention received is:
1o
0kk
1-114
Σ
(S q.
-S--1.6) q.k
REMM
S qik
S
(triangle inequality)
geMs
- |M
max max
/ 4.k
St- 1.1 q.
(bound by max)
(63)
k
Using equation 62
2W RA R
0 a
1-1,8 - |M's.
max
(64)
ddn
Step 3: Relating to KV Drift.
Recall that KV drift is A/ K K 112 lI V
By Assumption A.3
A 6 S W max
AH 12 + Wmax AH: = max |H
(65)
Ale
Therefore: |AH 12 2 2W
In partieular: Amax max, Ard
2Wmax
Substituting into equation 64
2W2 RVV
max, 4
W Wmax WeaaReVi REVN
f.f 1 [ap a
< MMa
max Vds
max A.
(66)
max
Vdk
Step Stability Constraint and Excess Drift.
Suppose The has drift ALL = Ate + where & > 0 is excess drift beyond average.
Then:
W WeaxReVN (Ale +E)
la a S [M2l
(67)
Vdk
While tokens with average drift have:
W max REVN N Vdn.
lo oz 1-1,1 $ |M
Ate
(68)
The differential attention change is:
W Wmax Ri REVN
! differential [M/l
(69)
Vdg.
For t. to remain most-attended, the gap at step t - 1 must absorb this differential:
W Wiaxx REVN
1- >
Adfffernntial
|M'l
8
(70)
dd
23

---

Published as a conference paper at ICLR 2026
Step 5: Assuming Bounded Attention Gap.
Applying the assumption A.6
W max Re REVN
MM 2 [M
(71)
vdk
Canceling [Mysl (assuming [M& > 0):
W
max R&v N
(72)
ddz
Solving for e:
dv
ddl ReVN
85
0
(73)
W maxxhe Rs ReV.
Therefore:
dd ReVN
A S SA +0++0
(74)
A.5
IMPLICATIONS FOR ELASTIC- CACHE
These results provide theoretical justification for our design:
Theorem A.8
Deeper layers have larger KV drift, justifying layer-aware refresh starting
from e
Theorem A.9
Most-attended tokens have minimal drift, validating their use cache stal-
eness indicators
B DETAILED EXPERIMENT SETUP
Implementation Details. We conduct all the experiments on single NVIDIA A100 80GB GPU to ensure consistent hardware environment. We evaluate our proposed method, Elastic-Cachs on three large scale DLMs: ALaDA--Intruce (Nie et 2025a). LLaDA-1.5 (Zhu al. 2025), and the multimodal LLaDA-V (You et al.) (025). Our evaluation spans both language and multimodal reasoning tasks including MBPP (Austin al. (021b), HumanEval (Chen al. 202T) for coding tasks, MATH (Hendrycks et al., 2021). GSMSK (Cobbe et al., 2021) for Maths related tasks and Vista (Lu et al.] (023) Math Verse (Zhagg et 20044b)) for mu multimodal mathematical reasoning tasks. The major hyperparameters for Elastic-Cache. unless otherwise specified in ablation studies, are set to a attention threshold of y = 0.9, a confidence threshold for parallel decoding of = 0.9, and cache block size of 32. To establish rigorous and fair comparison for all baseline methods, were re-evaluate all the methods including the original diffusion model L.LaD [Nie et al. (2025a) and Fast-dLLM (Wu et al. 2025). This process eliminates confounding variables from hardware or software discrepancies and ensures that all observed performance differences are attributable to the methods themselves.
Evaluation Framework and Metrics. Our evaluation protocol comprehensively assesses both in- ference efficiency and the preservation of model performance across a variety of tasks. For standard- ization and reproducibility, we conduct all task-specific evaluations using the tir-eval-harnee library (Gao et al. 2024). We measure inference speed by throughput tokens per second (t/s), which we calculate as the average number of tokens the model generates over the entire sequence until it produces an end-of-sequence (<eos>) token. We keep our calculation methodology con- sistent with that of Fast-dLLM (Wu et al. 2025) ensure comparable speed benchmarks. We measure task-specific performance using established metrics appropriate for each benchmark: for GSM8K (Cobbe et al. (2021), report 5-shot f flexible.extract exact match accuracy: for the MATH dataset (Hendrycks et al. (02T), we report the 4-shot math verify score using the mi nerva math h variant; for Human (Chen et al. 2021), evaluate 0-shot accuracy using
24

---

Published as a conference paper at ICLR 2026
Table 9: The hyper-parameters Elastic-Cache under various settings.
Model
Benchmark
Gen Length
256
32
0.9
GSM8K (5-shot)
512
16
0.9
256
16
0.9
MATH (4-shot)
512
16
0.9
LLaDA
256
32
0.9
Humancval (0-shot)
512
32
0.9
256
0.9
MBPP (3-shot)
512
16
0.9
GSM8K (5-shot)
512
0.9
MATH (4-shot)
512
LLaDA-1.5
0.9
Humaneval (0-shot)
256
0.9
MBPP (3-shot)
512
0.9
256
16
0.7
Mathvista
512
0.7
LLaDA-V
256
16
0.7
Mathverse
512
16
0.7
post-processing script consistent with the Fast-dLLM implementation ensure fair comparison; and for MBPP (Austin et al.) 202Ib), we report the 3-shot p metric. For multimodal eval- utilize evaluation suite adapted from its official im- plementation using the Imms eval framework (Zhang et al.) 2024a Li et al., 2024) to test on the Math (Lu et al. 2023 and Math Verse (Zhang et al.] (024b) benchmarks. For Math Vista, we report the gpt.eval.sseeeeee ..core, and for Verse, we report the gpt.eval.score on the mathverse.testmini. wtt.:/stmmmiiiiiiiiooooooi...ii.
uation on LLaDA-V You al. 2025),
subset.
Hyperrparammters: The hyper-parameters used for Elastic-Cache are provided in Table Specifi- cally.
For LLaDA and LLaDA-1.5 7 0.9 everywhere; /3 is mostly 16, except GSM8K (B 32 at 256, 16 at 512) and HumanEval (3 = 32 at both 256/512).
For LLaDA-V (Math Vista/Mahh Verse), 0.7 and B 16 for both 256 and 512 token lengths.
All tasks are reported generation lengths 256 and 512.
Our Motivation and Perspective. We close the gap with attention-aware and layer-aware caching for diffusion LLMs: tracking most-attended tokens and depth-varying KV dynamics to guide re- computation, complementary interval-based (Ma et al.) 2025) and confidence-based (Wu al.
2025
policies and compatible with the broader acceleration toolkit (Ainslic et al. 2023 Su et al.
2024
Touvron al. 2023a Dubey al. 2024 Gu al.
2017 Xiao et al. 2023 Arrola al.
2025
Chen et al. 2023 Ramesh & Mardani 2025
Kou et al. 2024).
C
EXTENDED EXPERIMENTAL ANALYSIS
C.I
COMPREHENSIVE HYPERPARAMETER SENSITIVITY
We provide extensive ablation studies to understand the interaction between window size 3 and attention threshold 7. Table 10 presents results on GSM8K with 512 Gen Length with LLaDA-1.5. systematically varying both parameters. The results demonstrate that / 16 with y 0.9 provides the best balance between accuracy and throughput for most applications. When maximum accuracy is required, with 7 0.95 achieves 83.2% accuracy 79.1 t/s. For throughput-critical
25

---

Published as a conference paper at ICLR 2026
deployments, 3 16 with y 0.7 delivers 138.6 t/s while maintaining 77.6% accuracy. Larger window sizes (3 = 32) do not consistently improve performance, likely because they cache too many MASK tokens that eventually become relevant, forcing more frequent cache invalidations.
Table 10: Joint sensitivity analysis of window size 3 and attention threshold on GSM8K with 512 Gen Length (LLaDA-1.5). Each cell shows accuracy (top) and throughput in t/s (bottom).
B/y
0.7
0.8
0.9
0.95
79.8
81.3
81.1
83.2
8
103.9
103.5
109.3
79.1
77.6
79.5
81.4
83.0
16
138.6
131.2
117.2
98.4
77.5
77.7
81.0
81.7
32
118.7
116.6
104.1
88.8
Beyond single token tracking, we evaluate strategies that monitor the top-k most-attended tokens per layer. Table 11 shows results for k (1,5, 5, 10, 15, 20) across different values. Tracking more tokens improves accuracy slightly but adds overhead. Top-10 and Top-15 strategies achieve the best accuracy, reaching up 84.7% GSM8K with 512 Gen Length. However, the throughput gains diminish as more tokens require drift computation. For most deployments, tracking a single most-attended token (Top-1) provides sufficient signal while minimizing overhead.
Table 11: Sensitivity to the number of tracked tokens (Top-k) across different attention thresholds on GSM8K with 512 Gen Length (LLaDA-1.5), Each cell shows accuracy (top) and throughput in 1/s (bottom).
Top-k/7
0.8
0.85
0.9
0.95
79.5
80.2
81.4
83.0
Top-1
131.2
129.9
117.2
98.4
81.5
81.4
83.5
82.7
Top-5
130.5
122.4
109.9
88.0
81.4
82.6
84.1
82.9
Top-10
121.1
118.4
100.7
77.6
82.5
83.2
83.7
84.7
Top-15
167.4
159.7
139.4
103.0
81.6
82.3
83.9
84.2
Top-20
162.4
154.1
136.3
102.0
C.2 MEMORY AND COMPUTATIONAL OVERHEAD
Memory Footprint Analysis. Table 12 reports peak GPU memory usage across generation lengths on LLaDA-1.5 with GSMSK. Elastic-Cache achieves lower memory consumption than both base- lines, with savings of 0.93-1.42 GB compared to the baseline and 2.38-3.89 GB compared to Fast- dLLM. This reduction stems from: (I) selective layer-wise cache updates that avoid storing inter- mediate states for all layers, and (2) block-wise caching of distant MASK tokens outside the sliding window. These memory savings enable deployment on resource-constrained devices while main- taining high throughput.
Table 12: Peak GPU memory footprint (GB) on LLaDA-1.5, GSM8K across generation lengths.
Method
256 tokens
512 tokens
1024 tokens
Baseline
19.04
19.62
20.79
Fast-dLLM
20.49
21.42
23.26
Elastic-Cache 18.11
18.13
19.37
The computational overhead of our attention-aware cache update mechanism is minimal compared to standard attention computation. Table [3 compares the complexity and multiply-accumull
op-
erations (MACs) for cache update triggers versus full QKV attention at sequence length K=1024. Finding the most-attended token requires OKK² H) operations, which translates to 6.7 x 107 MACs. In contrast, full attention computation scales O(K2 with 1.3 x 10t0 MACs. Our cache
26

---

Published as a conference paper at ICLR 2026
trigger introduces less than 0.5% overhead relative to attention, making it negligible in the over- all inference budget. The cosine similarity computation for drift detection adds another O(KH) operations, which is even cheaper.
Table 13: Computational overhead comparison. Cache update trigger has negligible cost compared full attention computation (K=1024, H=32, D=128).
Operation
Complexity
MACs (K=1024)
Cache Update Trigger Attention (QKV)
O(K2H)
6.7 x 107
O(KHHD)
1,3 x 1020
C.3 SCALABILITY ANALYSIS
We validate multi-GPU scalability by comparing throughput and latency under different hardware configurations. Table 14 shows results for LLaDA-1.5 on GSM8K with 512 Gen Length using and 2 A100 GPUs with data parallelism. With GPUs and batch size 8, Elastic-Cache achieves 225.5 t/s compared to Last-dI-LM's 68.0 t/s, maintaining the 3.3x throughput advantage observed in single-GPU settings. The latency improvement is more dramatic: our method reduces end-to-end inference time from 1.86 hours to 0.83 hours single GPU, and from 1.00 hours to 0.56 hours on two GPUs. These results confirm that Elastic-Cache scales effectively with additional hardware without requiring architecture-specffic optimizations.
Table 14: Multi-GPU UPU scalability on GSM8K with 512 Gen Length (LLaDA-1.5), Throughput in t/s and latency in hours for full benchmark evaluation.
Configuration
Accuracy (%)
Throughpul (Us)
Latency (h)
GPU, batch size
Fast-dLLM
1.86
Elastic
81.9
117.2
0.83
2 GPUs, batch size 8
Fast-dLLM
80.3
68.0
1.00
Elastic- 81.9
225.5
0.56
C.4 BLOCK-CACHING MECHANISM
To validate the effectiveness of our block-wise caching strategy for distant MASK tokens, we com- pare Elastic-Cache with and without this mechanism across different window sizes. Table 15 shows that block-caching provides substantial throughput gains with minimal impact on accuracy. At 16, removing block-caching reduces throughput from 119.8 t/s to 82.7 t/s while maintain- ing similar accuracy (81.4% 444 vs 80.6%). The benefits increase at larger window sizes, demonstrating that caching distant MASK tokens effectively eliminates redundant computation without harming prediction quality.
Table 15: Ablation of block-caching mechanism on GSM8K with 512 Gen Length (LLaDA-1.5,
7 0.9). Each cell shows accuracy (top) and throughput in t/s (bottom).
Method
B
B
16
3 = 32
3 = 64
81.1
80.6
80.6
74.3
w/o block-caching
77.0
82.7
84.3
67.1
81.1
81.4
81.0
75.7
Elastic-Cache
109.3
119.8
118.1
88.6
We further explore integrating adaptive block sizing using AdaBlock (Lu al. 2025), which dynam- ically adjusts window size based on semantic coherence. Table 16 shows results when combining AdaBlock with Elastic-Cache Starting from default window sizes Po. AdaBlock adjusts to average sizes 3 during decoding. However, this adaptive approach introduces overhead without accuracy gains, confirming our observation that fixed window sizes suffice when combined with attention- aware cache updates.
27

---

Published as a conference paper at ICLR 2026
Table 16: Integration with AdaBlock adaptive window sizing on GSM8K with 512 Gen Length (LLaDA-1.5,7 = 0.9). Format: accuracy throughppt (t/s).
Method
= = 16, = 15.6
8 = 27.1
Elastic-Cache
81.4/ 119.1
81.0/ 118.1
AdaBlock
81.9/ 87.1
80.7 85.5
C.5 RUNTIME ADAPTATION OF LAYER BOUNDARY
The layer boundary L is determined automatically at runtime based on observed attention drift. Ta- ble 17 shows how cache update frequency p L- varies with threshold y on GSM8K with 512 Gen Length. Lower y values trigger updates less fireuuenlyy (p = 0.47% at 7 = 0.5), maximizing throughput but sacrificing accuracy. Higher 7 values increase update frequency (p 20.02%
0.95), preserving accuracy at reduced throughput. This adaptive behavior demonstrates that f effectively responds imput difficulty without manual tuning.
Table 17: Cache update frequency and performance layer boundary l adapts to different thresh- olds on GSM8K with 512 Gen Length (LLaDA-1.5))
7
0.5
0.7
0.8
0.85
0.9
0.95
P (%)
0.47
1.81
4.17
6.50
10.23
20.02
Accuracy (%)
76.0
77.6
79.5
83.0 80.2
81.4
Throughput (t/s)
142.7
138.6
131.2
129.9
117.2
98.4
C.6 VALIDATION OF MOST-ATTENDER TOKEN HEURISTIC
Our method relies on the assumption that most-attended tokens exhibit minimal drift and serve as conservative indicators for cache staleness. Table 18 validates this assumption empirically by measuring average cosine similarity between consecutive steps for most-attended tokens versus all cached tokens across benchmarks. The most-attended tokens consistently maintain higher similarity (0.948-0.985) confirming they change less than average tokens and provide reliable lower bounds for drift detection.
Table 18: Empirical validation of most-attended token stability. Higher cosine similarity indicates lower drift. Results on LLaDA-1.5 with 0.9.
Token Type
GSM8K
MATH
HumanEval
MBPP
Most-attended
0.974
0.978
0.985
0.948
Average (all cached)
0.973
0.977
0.980
0.947
C.7 MULTIMODAL EXTENSIONS
For multimodal tasks on LLaDA-V evaluate two configurations: single-token prediction per step (matching our text-only setup) and parallel multi-token prediction (matching the original LLaDA-V implementation). Table 19 shows that Plastic-Cache with single-token prediction already surpasses the original LLaDA-V baseline in throughput. When extended to parallel prediction, throughput increases further to 44.2 t/s on Math Vista and 42.2 on Math Verse, demonstrating that our cache management strategy complements parallel decoding effectively in multimodal settings.
C.8 COMPARISON WITH CONSISTENCY MODELS
We compare against consistency-base acceleration methods for diffusion models. Table 20 shows results GSM8K with 512 Gen Length using Consistency LLMs (Kou et al.) 2024) versus our approach on 1... While consistency models achieve reasonable throughput (35.5 t/s), they
suffer significant accuracy degradation (56.5% vs 78.5% baseline). Elastic- Cachhe maintains higher accuracy (79.2%) while delivering substantially greater throughput (109.6 t/s), demonstrating the advantage of training-free cache management over distillation-based approaches.
28

---

Published as a conference paper at ICLR 2026
Table 19: Multimodal performance with parallel decoding on LLaDA-V (generation length 512).
Format: accuracy throughput (t/s).
Method
Math Vista
MathVerss
LLaDA-V baseline
53.0 1.9
26.9 / 2.0
Elastic-Cache (single-token)
55.8 / 24.1
29.2/
Elastic-Cache (parallel)
50.6 44.2
26.3 / 42.2
Table 20: Comparison with consistency-based acceleration on GSM8K with 512 Gen Length. Our training-free approach preserves accuracy while achieving higher throughput.
Method
Accuracy (%)
Throughput (ts)
Consistency LLMs
56.5
35.5
LLaDA-1.5 baseline
78.5
2.6
Elastic-Cache
79.2
109.6
D USE OF LARGE LANGUAGE MODELS
An LLM was used to help solely polish the writing of the paper, while all method, idea and experi- ments were prepared and carried out entirely by the authors.
E SAMPLE RESPONSE
In the following, we present several examples of actual generation results produced by our approach under different parameter settings.
Table 21: Example Qualitative comparison for different 7, LLaDA-1.5
Prompt: A store sells 3 types of fruit: apples at $2 each, bananas at $1 each, and oranges at $3 each. If Sarah buys 4 apples, bananas, and 2 oranges, how much does she spend total?
7= 7=0.5
0.95
To determine how much Sarah spends total, we need to calculate cost of each type of fruit separately and then sum these amounts.
To determine how much Sarah spends total, need to calculate cost each type of fruit separately and then sum these amounts.
To determine how much Sarah spends in total, we need to calculate cost of each type of fruit separately and then sum these amounts.
1. **Cost of apples:**.. Sarah buys apples. Each apple costs $2 Total cost for apples = (1 x = 8) dollars.
First, let's calculate the cost of the ap- ples: Sarah buys 4 apples. Each ap- ple costs $2. Therefore, the cost of apples is (4 x 2 = 8) dollars.
First, let's caleulate the cost of the ap- ples: Sarah buys 4 apples. Each ap- ple costs $2. Total cost for apples = (4 x 2 8) dollars.
2. **Cost of bananas: Sarah buys
6 bananas. Each hanana costs $1.
Next, let's calculate the cost of the ba- nanas: Sarah buys bananas. Each banana costs $1. Therefore, the cost bananas (6 x 6) dollars.
Next, let's calculate the cost of the ba- Sarah buys 6 bananas. Each banana costs $1. Total cost for ba- nanas = (6 x 1 = 6) dollars.
Total cost for bananas (6 x 1 6) dollars.
**Cost of oranges:** Sarah buys oranges. Each orange costs $3. Total
Finally, let calculate the cost of the or- anges: Sarah buys oranges. Each orange costs $3 Therefore, the cost of oranges is (2 x 3 6) dollars.
Then, let's calculate the cost of the or- anges: Sarah buys 2 oranges. Each orange = (2 x = 6) dollars.
cost for oranges= (2 3 = 6) dollars. Now, we sum the costs of all the fruits to find total amount Sarah spends:
$3 -Total cost for oranges
8 (apples) + 6 (bananas) +
Now, we sum the costs of all the fruits to find the total amount Sarah spends:
Now, we sum the costs of all the fruits to find the total amount spent: Total Cost of apples + Cost of bananas + Cost of oranges Total cost
6 (oranges) = 20 dollars
Therefore, Sarah spends total of 20 dollars.
Total cost Cost of apples Cost of bananas Cost of oranges Total cost
= (8 + 6 - 6 20) dollars
6 Total cost = 20 dollars
Time: 3.1s
So, Sarah spends total of 20 dollars.
Therefore, Sarah spends a total of 20 dollars.
Time: 3.4s
Time: 3.5s
29

---

Published as a conference paper at ICLR 2026
Table 22: Example Qualitative comparison for different 7, LLaDA-1.5
Prompt: Tom has 24 marbles. He gives 1/3 of them to his sister and 1/4 of the remaining marbles his brother. How many marbles does Tom have left?
0.5
0.7
0.95
To determine how many marbles Tom has left, will follow these steps: Calculate the number of marbles Tom gives to his sister. 2. Subtract the marbles given to his sister from the original number of marbles. 3. Calcu- late the number of marbles Tom gives brother. Subtract marbles given to his brother from the remaining marbles.
To determine how many marbles Tom has left, we will follow these steps: Calculate the number Tom gives to his sister. the marbles given to his sister from the original number of marbles. 3. Calcu- late the numher of marbles Tom gives his brother. 4. Subtract the marbles given to his brother from the remaining marbles.
To determine how many marbles Tom has left, will follow these steps: Calculate the number of marbles Tom gives to his sister. 2. Subtract the marbles given to his sister from the to- tal number of marbles. 3. Calculate the number of marbles Tom gives his brother. Subtract marbles given to his brother from the remaining mar- bles.
marbles Subtract
Let's start with the first step: **Step Calculate the number of bles Tom gives to his sister.
Let's start with the first step: = Step Calculate the number of mar- bles Tom gives to his sister.
*tStep 1: Calculate the number of mar- bles Tom gives his sister. ** Tom gives of his marbles to sis- Marbles given sister = x2 8 * **Step 2: Subtract the marbles given to his sister from the total number mar- bles,** **
Tom gives 3 of his 24 marbles to his
Tom gives of his 24 marbles to his sis-
sister. 1/2 x 24 = So, Tom gives marbles to his sister.
ter. Marbles given to sister= A x24 **Seep 2: Subtract the marbles given to his sister from the original number of marbles. #*
**Step 2: Subtract the marbles given to his sister from the original number of marbles.***
Remaining marbles after giving to sister 16
Remaining marbles after giving to sister = 24 - 16
24
24 - 8 16 So, after giving marbles to his sister, Tom has 16 marbles left. "SStep 3: Calculate the number of mar- bles Tom gives his * Tom gives bles to his brother. x 16 So, Tom gives 4 marbles his brother. **Step 4: Subtract the marbles given to his brother from the remaining mar- bles. **
""Step 3: Calculate the number mar- bles Tom gives to his brother." Tom gives 1 of the remaining 16 mar- bles to his brother. Marbles given brother = x 16
"Stepp 3: Calculate the number of mar-
bles Tom gives his brother." ** Tom gives of the remaining bles to brother. Marbles given to brother = 16 =4
remaining mar-
**Step 4: Subtract the marbles given to his brother from the remaining mar- bles.** **
*teppp 4: Subtract the marbles given his brother from the remaining bles. **
Remaining marbles after giving brother = 16 12
Remaining marbles after giving to brother = 16 - = 12
16 12 So, after giving marbles his brother, Tom has marbles left. Therefore, the number of marbles Tom left 12
Therefore, the number of marbles Tom has left is
Therefore, the number of marbles Tom has left 12.
Time: 3.2s
Time: 3.1s
Time: 3.3s
30

---

Published as a conference paper at ICLR 2026
Table 23: Example- Qualitative comparison of responses under different 7.
Prompt: Points A and B are on parabola y = 4x² + Tu and the origin the midpoint of AB. Find the square of the length of AB.
Y= 0.5
0.7
? 0.9
Let A = (21, W1) and B = (x2, V2). Since the origin the midpoint of AB, we have:
Let A = (I1, V1) and B = (I2, V2)- Since the origin is the midpoint of AB, we have:
Let A = (x1, V1) and B = (x2, V2)
Since the origin is the midpoint AB, we have:
91 y2
(0,
22 y1 41
(0,0) 0).
21
22
21
+ 1/2
(0,0).
2
2
2
2
2
This implies:
This implies:
This implies:
+ 22 = and y2 0.
42 0 and
0.
21
0 and
31 = 0.
Since A and B on the parabola y
Since
lie on the parabola y
lie on parabola y 4x 7 - 1, we have:
Since
4x² + 7x 1, we have:
4x² + 7x 1, we have:
y1+ [4z]+7m1- - (1)3
V1 4xf - 7x.
y1 4x + 7x
Now, using the equations #1 + x2 and y1 + 1/2 = 0, we get:
and
and
1/2 4. 702 1.
4r 7:0
4x + 7 - 4m² 7.0 1 0.
From 21 - x2 = 0, we get 51 = -22-
Using y1 + = 0, we get: yi + y/2 (4x + 7x, -1)+ (4r} + 7 -
This simplifies to:
Substituting æ = u2 into the equa-
tions for y1 and yz:
Simplifying, we have:
4 7 + 103 + 7 2 O.
91 4x² 7x2
1x + 7 112 + 7xz 2 = 0.
Thus. square the length AB is:
and
Since + 22 0. get:
(#1 x2 2 + (v1 y2)² 02 + 0. Therefore, the square the length of AB is:
1/2 4x² 702
4x² + 7 4x 7 2 = 0.
Since + = 0, we have:
Now, we need to find the square the length of AB. The length of AB is given by:
0
V1 1/2
Time: 8.1s
Thus,
AB ( - 21 (22 )
#1 (4x3 7 -4x3 -
Using midpoint formula and the properties of the parabola, we can sim- plify the expression. However, since the problem asks for the square of the length, we can directly use result. Thus, the square of the length of AB is:
Similarly.
8/2 = 40² + 7x2
Therefore,
11 = -/2
50
- (4x2 7 = 42 72a +1
Time: 8.7s (Correct)
Thus, the square of the length of AB is given by:
Time: 9.3s
31

---

Published as a conference paper at ICLR 2026
Table 24: Coding Example Close Elements
Prompt:
from typing import List
def has_close_elemee (numbers: List [float], threshold: float) -> bool: "*" Check if in given list of numbers, any two numbers closer to
each other than
given threshold.
>>> cecccoss__eemeetee ([1.0, 2.0, 3.01, 0.5)
Palse
>>> close elements (12.0, 2.8, 3.0, 4.0, 5.0, 2.01, 0.3)
True
FAST-DLLM
OURS
sorted_number sorted (numbers)
sorted_number sorted (numbers) for i in range (len (sorted, numbers)
for i in range (len (sorted (tttred__umbbrr) if abs (sorted rumbers [i]
1):
sorted Sitieerd_mberrreeeeee [i + 1]) <
if abs ptorted_numbeet corted_nurbers/ [i + 1]]
threshold:
return True
threshold:
return False
return True
return False
Fast-DLLM generated incorrect solution
Table 25: Coding Example Truncate Number
Prompt:
def truncate number (number: float) -> float:
Given positive floating point number, it can be decomposed into and integer part (largest integer smaller than given number) and decimals
(leftover part always smaller than 1).
Return the decimal part of the number. Seuncate_number (3.5)
0.5
FAST-DLLM
OURS
if number
if number raise ValueError ("meer .ce..pcsitive")
raise ValueError ("The_nunber__uuu www.positieee.... integer_part int (number)
integer_part int (number) decimal_part number integer_part return decimal part
decimal_part number integer_part return decimal_parr
Both solutions are functionally equivalent with minor differences
32


Reviews:
[{"review_id": "0TKQSigOKy", "reviewer": "Reviewer_FYhQ", "summary": "This paper introduces Elastic-Cache, a training-free, architecture-agnostic strategy to accelerate decoding in diffusion large language models (DLMs) by adaptively managing key-value (KV) cache recomputation. The method leverages three key observations: (1) distant MASK tokens act primarily as length priors and can be block-cached; (2) KV drift increases with layer depth, justifying depth-selective updates; and (3) the most-attended token exhibits minimal KV drift, serving as a reliable proxy for global cache stability. Elastic-Cache combines an attention-aware drift test (triggering refresh when cosine similarity of attention weights drops below threshold γ) with a layer-aware update policy (refreshing only from a boundary layer ℓ* onward). Experiments on LLaDA variants show substantial speedups—up to 45.1× on GSM8K-512, 8.7× on GSM8K-256, and 4.8× on HumanEval—while maintaining or even improving accuracy over baselines like Fast-dLLM.", "questions": "* How does Elastic-Cache perform under extreme KV drift (e.g., highly ambiguous prompts or adversarial inputs)?\n* Could the attention-drift threshold γ be learned or adapted per token/layer instead of being global?\n* What is the overhead of computing attention drift? Does it scale linearly with context length?\n* Is the method applicable to block diffusion?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Strong empirical results: Consistent throughput gains across models (LLaDA-Instruct, LLaDA-1.5, LLaDA-V) and tasks (math, code, multimodal), with negligible accuracy loss.\n* Principled design: The use of attention-weight drift on the most-attended token as a low-overhead trigger is well-motivated and novel for DLMs.\n* Practical impact: Training-free, compatible with existing confidence-based decoding, and enables real-world deployment of DLMs.\n* Comprehensive ablation: Thorough analysis of γ, sliding window size, and generation length validates design choices.", "weaknesses": "* Limited theoretical grounding: While empirically sound, the link between attention drift and KV state change lacks formal justification.\n* Hyperparameter sensitivity: Performance depends on γ and β; optimal values vary across tasks (e.g., γ=0.9 for text, γ=0.7 for multimodal).\n* Baseline scope: Comparisons focus on Fast-dLLM and dKV-Cache; broader context with other diffusion acceleration methods (e.g., consistency models) is sparse.\n* Hardware assumption: All experiments on single A100; scalability to multi-GPU or devices is unverified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AUBOjoNOMs", "reviewer": "Reviewer_9k5q", "summary": "The paper diagnoses redundancy in diffusion‑style LLM decoding: most methods recompute QKV for all tokens and all layers at every denoising step, even though KV states often change little—especially in shallow layers and for faraway MASK tokens. It proposes Elastic‑Cache, a training‑free, architecture‑agnostic policy that (i) block‑caches distant MASK tokens, (ii) triggers cache refresh when needed via an attention‑aware drift test on the most‑attended token, and (iii) refreshes where needed by recomputing only from a learned boundary layer ℓ^⋆ upward (layer‑aware schedule). Figure 2 contrasts the pipeline against blockwise caching; Algorithm 1 gives the full procedure. Reported results show up to 45.1× speedups on long sequences and 6.8× higher throughput on GSM8K while maintaining or improving accuracy versus baselines (e.g., Fast‑dLLM).", "questions": "1. How sensitive are results to the cosine‑similarity threshold γ and to choosing the single most‑attended token vs. a small set? Can you report AUROC of the drift test to predict accuracy deltas on held‑out runs?\n\n2. How is ℓ^⋆ determined at runtime? Static per‑model, or adaptive per input? Please include ablations showing latency/quality as ℓ^⋆ varies.\n\n3. In sequences where distant MASKs become influential (e.g., closing brackets, long‑range constraints), how often does block‑caching degrade predictions? Any fallback to expand the window β on‑the‑fly?\n\n4. Please provide per‑sequence memory footprints and DRAM traffic for Elastic‑Cache vs. baselines at 256/512/1024 tokens so practitioners can size deployments.\n\n5. Have you tested on masked‑diffusion LLMs with different denoising schedules or bidirectional attention variants? Any caveats for multimodal diffusion decoders?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "++ The paper clearly explains why AR KV caching fails in diffusion LLMs (bidirectional attention ⇒ evolving KVs) and motivates KV drift as the right signal. Figure 1 visualizes depth‑dependent drift and shows the most‑attended token’s attention changes track KV change, justifying the drift test.\n\n++ Three components—block‑caching far MASK tokens, sliding‑window decoding for active MASKs, and a layer‑aware refresh boundary—are well matched to diffusion masking dynamics and are easy to integrate.\n\n++ The method does not alter training or model internals; it only changes cache policy, which supports deployment claims and broad applicability across diffusion LLM variants. The motivation and contributions list emphasize this.", "weaknesses": "-- The most‑attended‑token drift is a clever heuristic, but the paper lacks sensitivity/theory: when does it fail (e.g., diffuse attention or multi‑focus regimes)? A risk analysis or bounds linking attention drift to KV error would strengthen the claim. Algorithm 1 hints at cosine‑similarity thresholds but ablations are not detailed.\n\n-- While Fast‑dLLM and related works are discussed, it’s unclear if comparisons include recent adaptive policies (e.g., learned thresholds) under identical hardware/token budgets and for very long masks; more head‑to‑head with dKV‑Cache/DeepCache over a shared protocol would improve fairness.\n\n-- The method introduces β (window size), γ (drift threshold), and ℓ^⋆ (boundary). The paper should show cross‑task robustness and guidance for setting them vs. accuracy/latency trade‑offs. (§3.2–3.3 describe β/γ, but systematic sweeps are not reported.)\n\n-- The narrative focuses on compute and latency; a detailed profile of memory bandwidth, cache footprint (esp. with sliding windows and partial refresh), and impact on KV tensor residency would help practitioners plan deployments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k8mr2rlCcu", "reviewer": "Reviewer_Gh7c", "summary": "This paper proposes Elastic-Cache, a training-free KV cache refresh strategy for diffusion LLMs. Based on KV cache drift statistic, Elastic-Cache adaptively controls when to recompute the KV cache and selectively updates only deeper layers. By removing redundant cache computation, it achieves significant speedups (up to 45×) with minimal accuracy loss across diverse tasks.", "questions": "Please refer to the weaknesses", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea is novel and well-motivated. The core idea is built upon a key observation: significant redundancy exists in DLLM's KV cache computation across denoising steps and layers, a majority of the KV cache can be reused. Similar phenomena have been discussed in KV cache acceleration for LLMs (Duoattention, Razorattention, CateKV, etc.)\n- Strong performance across different models: Elastic-Cache consistently outperforms baselines in performance and speedup, with up to 45.1× speedup over LLaDA-1.5. It also maintains similar accuracy, demonstrating efficient computation reduction without sacrificing quality.\n\nEffective Adaptive Mechanism: The attention-aware drift test (using most-attended tokens) and layer-aware updates reduce cache refresh frequency to as low as 20% of baseline (Fig. 3b), validating its lightweight and adaptive design.", "weaknesses": "- The author claims that Elastic-Cache architecture-agnostic. I suggest adding experiments on other DLLMs like Dream to prove it.\n\n- Limited Multimodal Speedup: On LLaDA-V, speedup is modest (29.7 t/s), suggesting less cache redundancy in multimodal tasks or architecture-specific limitation of the method. And why not report the performance and tps of original LLaDA-V in Table 3?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4ZaxMkcTUp", "reviewer": "Reviewer_SHc1", "summary": "This paper addresses the computational overhead in Diffusion Large Language Model inference, which stems from recomputing the full QKV cache for all tokens at all layers during each denoising step. The authors propose Elastic-Cache that adaptively decides when and where to recompute the KV cache.", "questions": "see weaknesses", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The method is training-free and architecture-agnostic, making it a drop-in component that can be easily adopted.\n\n2. Comprehensive experiments and ablations showing the effectness of Elastic-Cache", "weaknesses": "1. It seems that the authors used the wrong template.\n\n2. The primary assumption is that the most-attended token \"typically exhibits the smallest drift\" and thus serves as a \"conservative lower bound\" . This is a strong claim that is central to the method's success, but it is presented as an observation without comprehensive supporting data.\n\n3. Would the authors clarify the exact mechanics of this window? Does it advance one token at a time, or does it process a full block of $\\beta$ tokens and then move to the next block (which would make it very similar to Fast-dLLM, just with a different caching rule)?\n\n4. At each step and each layer, the algorithm must find the most-attended token. What is the computational overhead of this process? a brief analysis of the trigger's cost relative to the cost of a single QKV projection would better for analysing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)