dKV-Cache: The Cache for Diffusion Language Models
Authors: Xinyin Ma
Runpeng Yu
Gongfan Fang
Xinchao Wang
Venue: NeurIPS 2025 poster ()
Keywords: Diffusion Language Model
Inference Efficiency
KV-Cache
Primary area: deep_learning
TL;DR: We propose the delayed KV-Cache for diffusion language models.

Abstract: Diffusion Language Models (DLMs) have been seen as a promising competitor for autoregressive language models (ARs). However, diffusion language models have long been constrained by slow inference. A core challenge is that their non‑autoregressive architecture and bidirectional attention preclude the key–value cache that accelerates decoding. We address this bottleneck by proposing a KV-cache-like mechanism, **d**elayed **KV-Cache**, for the denoising process of DLMs. Our approach is motivated by the observation that different tokens have distinct representation dynamics throughout the diffusion process. Accordingly, we propose a delayed and conditioned caching strategy for key and value states. We design two complementary variants to cache key and value step‑by‑step: (1) dKV-Cache-Decode, which provides almost lossless acceleration, and even improves performance on long sequences, suggesting that existing DLMs may under‑utilise contextual information during inference. (2) dKV-Cache‑Greedy, which has aggressive caching with reduced lifespan, achieving higher speed-ups with quadratic time complexity at the cost of some performance degradation. dKV-Cache, in final, achieves from 2-10$\times$ speedup in inference, largely narrowing the gap between ARs and DLMs.  We evaluate our dKV-Cache on several benchmarks, delivering acceleration across general language understanding, mathematical, and code‑generation benchmarks. Experiments demonstrate that cache can also be used in DLMs, even in a training-free manner from current DLMs.

dKV-Cache: The Cache for Diffusion Language Models
Xinyin Ma
Runpeng Yu Gongfan Fang Xinchao Wang"
National University of Singapore maxinyindu mus. edu, xinchaolmus edu.ag
Abstract
Diffusion Language Models (DLMs) have been a promising competitor for autoregressive language models (ARs). However, diffusion language mod- els have long been constrained by slow inference. A core challenge is that their non-autoregressive architecture and bidirectional attention preclude the key-value cache that accelerates decoding. We address this bottleneck by proposing KV- cache-like mechanism, delayed KV-Cache, for the denoising process of DLMs. Our approach is motivated by the observation that different tokens have distinct representation dynamics throughout the diffusion process. Accordingly, we pro- pose a delayed and conditioned caching strategy for key and value states. We design two complementary variants to cache key and value step-by-stepp (1) dKV- Cache-Decode, which provides almost lossless acceleration, and even improves performance on long sequences, suggesting that existing DLMs may under-utilise contextual information during inference. (2) SEVV-Cace--Greedy which has ag- gressive caching with reduced lifespan, achieving higher speed-ups with quadratic time complexity at the cost of some performance degradation. final, achieves from 2-10x speedup in inference, largely narrowing the gap be- tween ARs and DLMs. We evaluate our dKV-Cache on several benchmarks, delivering acceleration across general language understanding, mathematical, and sode-generation benchmarks. Experiments demonstrate that cache can also be used at
in
in DLMs. even in training-firee manner from current DLMs. The code is available cactuuu..oommaaaa cww/horseee/ dKV Cache
1 Introduction
Diffusion language models (DLMs) B have recently emerged an alternative paradigm for text generation, inspired by the success of diffusion models in continuous domains like images [[5] 48 and videos 28 5. Unlike autoregressive transformers (ARs) 14
43 [2 that generate text left-
to-right at time, diffusion language model produces text by gradually refining a sequence of initially noisy tokens masked tokens into coherent output (24
29 Recent advancements in diffusion-based language models have underscored their versatility across both continuous [2]] and discrete formulations [40] [] particular, discrete diffusion models have shown competitive performance in language modeling (37), attracting growing interest for their potential to achieve faster decoding than traditional ARs while maintaining comparable generation quality.
One notable advantage of diffusion language models is their potential to decode an arbitrary number of tokens in parallel 49], whereas ARs require one forward pass per generated token. This parallel decoding paradigm offers the potential for improved wall-clock inference time and higher throughput. However, despite their theoretical efficiency, current DLMs remain substantially slower than ARs in practice. This inefficiency is primarily due to two factors: the incompatibility of DL.Ss with the
Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

KV-Cache mechanism B3 13,, and the large number of network evaluations for denoising [77. Specifically, generating a sequence of length L typically entails L denoising steps, each involving a full bidirectional attention pass, leading to a cubic time complexity of (1..). In contrast, AR models utilize KV-Cache to reduce per-step complexity to ((T,), achieving much faster inference overall.
In this paper, we tackle the challenge of integrating the KV-Cache mechanism into diffusion language models and operate without autoregressive or semi-autoregressive structures 22] We identify two core reasons that prevent the direct usage of KV-Cache in DLMs. (1) KV-Cache hinges on the assumption that the key and value states of previously generated tokens remain fixed during subsequent decoding steps. This property is preserved in autoregressive models through the use of a causal attention mask, which restricts each token attend only to earlier positions [T.. However, DLMs adopt a bidirectional attention mechanism, similar to -ooorerrreeee models (20). allowing every token to attend to all others in the sequence and making nontrivial to reuse previously cached keys and values. (2) KV-Cache presupposes a fixed and left-to-right sequential decoding, where the position of the next token is deterministic. This assumption enables ARs to compute QKV states selectively only at the current decoding position. However, DLMs break this paradigm by supporting flexible generation orders. At each denoising step, any token position may be selected for update, which is a key advantage of DLMs for tasks involving long-range dependencies and holistic planning [51]
To solve this problem, we propose the delayed KV-Cache, dKV-Cache, the KV-Cache for diffusion language models. The core design of our proposed dKV-Cache centers on how to enable caching of key and value states acToss denoising steps in diffusion language models. key insight motivating this design is that, although DLMS employ bidirectional attention, intuitively incompatible with caching, the representations of key and value are not fundamentally unreusable, but rather require delayed and conditioned reuse. In particular, we observe that the evolution of key and value states strongly influenced by whether a token has been decoded. This behavior motivates a delayed caching strategy, wherein only the key and value states of decoded tokens would be cached, delayed from the ARs that caching occurs immediately upon input. We further propose a one-step delayed caching mechanism, in which the caching of key/value states is postponed by another denoising step. This intentional delay substantially boosts the performance and also reduces memory overhead for KV storage. Besides the delayed caching, we further propose a more aggressive decoding strategy to reduce the computational complexity of diffusion language models from ((L3) to (L2) by restricting the caching to compact subset of tokens: the delayed tokens and the current to-be de-oe-de tokens, and the window tokens.
Our experiments demonstrate that the proposed method achieves 2-10x speedup on existing 7B- scale diffusion language models, including LLaDA (37 and Dream (52), across broad range of benchmarks such as general language understanding, code generation, and mathematical problem solving. These efficiency gains come with only minor and often negligible performance degradation, highlighting the practical value of our approach for accelerating DLM inference without training. Furthermore, we demonstrate that JKV-Cache is robust across variations in prefill length, output length, and the number of sampling steps.
Contributions. (1) We propose the first KV-Cache mechanism for diffusion language models by leveraging the evolving dynamics of token representations. We introduce a delay caching strategy compatible with bidirectional attention. (2) We propose two practical variants of our method: dK V- Mache-Decode, which enables long-term cache reuse, and dKV-Cache-GG Greedd, which reduces the per-step time complexity for faster decoding. (3) Extensive experiments DLMs demonstrate that our approach achieves 2-10x inference speedup with minimal or negligible performance loss.
2 Related Work
2.1 Diffusion Language Models
Diffusion models (4 model the data generation as the inversion of a forward-noiss process and demonstrated impressive generation quality in image [38], video [3], and audio generation [16]. For diffusion models on language generation, 225 3 extends DDPM to categorical data and defines the transition matrices for the corruption and denoising. 20 introduces the continuous-time diffusion
over the continuous word-embedding space, and (2] closes the quality of generation on par with GPT-
2 via simplex defined over the vocabulary. Besides the diffusion in the continuous space 47 53
2

---

for discrete distribution, another approach seeks the path in discrete language diffusion models. [22 training BERT to learn the reverse process of discrete diffusion process with an absorbing state in D3PM. SEDD 33 introduces score entropy training, novel loss extending score- more-matching to
discrete data and MDLM [40] shows that simple masked discrete diffusion is competitive to all previous kinds of DLMs. Block diffusion
2 extends the current non-autogressive [20] discrete language diffusion models into a cumi-eutoregressivee one [5], making it feasible to generate sequences of arbitrary length. [37 [
scaling the masked diffusion language models to billions of parameters, achieving performance comparable to leading autoregressive LLMs.
2.2 Cache in Generative Models
Cache 45 is small, fast memory that stores frequently accessed data, reducing the time that the CPU needs to fetch data from slower memory. Cache is first introduced in deep neural networks in transformers
where the KV-Cache caches previous tokens' key and value tensors. KV-
Cache becomes fundamental technique in transformers, and several improved techniques are
to reduce the memory consumption of KV-Cache for long-context generation. Besides this, caches are also been explored in diffusion models for image [36] 46]. Those work leverage the temporal similarities between high-level features [35] 9 attention map [54] 31 to
proposed [88 32 26
achieve faster inference of diffusion generation. This also has been explored in 3D generative
modeling [50] and video generation 34]. However, the cache for diffusion language models is less explored, especially the KV-Cache for diffusion language models.
(2) explores the kv- cache in autoerruooeer diffusion language models. This requires considering the KV-Cache in training, making twice the forward computation in the training and its form is still constrained in the autoregressive formula. have been calculated.
40) also considers cache, but under a strict condition that no new tokens
3 Methods
3.1 Preliminary
We primarily focus on continuous-time discrete language models, with particular attention masked diffusion language models (40 37/. which have shown strong scalability to billion billion-paramesn parameter scales with high generation quality.
Consider a text sequence with I. tokens xb- sampled from the target distribution Pdatt (x Each token is represented by one-hot vector with V categories, where V is the vocabulary size. The forward process adds noise in the original sequence Xo. which, in the discrete diffusion models here, takes the form of masking some of the tokens randomly. The masking process can be controlled by transition matrix Ut, where each element in this matrix [U 1l, represents the probability transition from token i to token j at step / (3. The denoising process is discretized into T steps, and we define the continuous timestep e(l) 1/T, where i {0, 1,..,, T}. We use timestep in the continuous temporal space of the denoising process and step in the discrete space. The forward diffusion can be modelled as:
(xe(t) (o)) = Cat (Ze(0i P xQU.), where U, U,=Iu
Then we can get the corrupted Xp. The absorbing form for the transition matrix is used here, where each token either remains the same transfers to the special [MASK] token at the probability of 3 The cumulated transition matrix U,, as defined in the masked diffusion models, can be formulated as:
1 if
i i=j = [MASK]
U
α+ if
i i=j [MASK]
with ā
(1 (-3)) 3x) 1=1
1 04.
if
- o [MASK]
where Cr linearly decrease to 0 approaches T. The reverse process is learned by the models 0, where po Po(xx(t1)) (X(()) ) learned to approximate q(xe(t-1) ())()))) Xo). (o(xx(t-1) Xx(e)) the neural network with parameters 0 is optimized to predict the clean tokens xo given Xe(t)-
3

---

D2
Projection
Cache
Reorder
k (1r-11)
MASK Token
Decoded Token
Compute KV
Cache for KV
Step
Step e-1
Compute Q
Step
Step
(a) KV-Cache in AR
(b) V-C-Ce
Figure 1: Illustration of dk V-Cche. At step t. no prior cache would be activated though the token D--1 has been decoded. K and V are delayed the next step to be reordered and reused.
Sampling Process of DLMs. Given the noisy sequence 2!:", which is consisted only with the masked token. Each timestep, the denoising model po (xe(tt11)) Xc(e)) would be called, with zo predict first and then remasking the rest by g(Xc(t-1))XX) Xx(t)) The unmasked tokens would remain unchanged during the later denoising process. Several strategies are used in the remasking stage, e.g.,
random remasking [3. keeping the most confident ones [ or by selecting the topk positions with the largest margin between the two most probable values [27].
Formulation of KV-Cache. Since diffusion language models still use the transformer architecture (with or without GQA [ would not affect our method), we simply grab the formulation of KV-Cache ARs first. In transformer decoder, each layer would project the current hidden states h, into query-key-value triplet (Q,, Kr, Vt) via learned projection WQ, WK and Wv. At step t, only the hidden states of the t-th tokens h.d would be calculated. The recursive KV-Cache update is appending the new key-value pair the running buffered K KV-Cache:
Q (K(u.))
K k(!tdl concat
[It-1]] k!!) (-1
2, = softmax
v[iid]
with
(1)
dd
v.!::]] concat
(v [1:-1 vv!!}
where 2; is the output of the attention head at step f and the dot products are scaled down by / vd..
3.2 Why KV-Cache Cannot be Used in DLMs?
The effectiveness of KV-Cache can be attributed to the reuse of previously computed K and V states, and the targeted computation only for the current decoding token. We conclude that standard KV-Cache is fundamentally incompatible with diffusion language models for two reasons:
Timestep-variant key and value states. In the autoregressive setting, every time step shares a single, causally growing set key and value tensors. K!!lt-1] and V[lit-I] are the same each step from (- 1 and later with the help of causal attention mask. By contrast, DLMs employ a bidirectional attention mask; consequently, the key and value representations that each token can attend to at timestep m, K(1:t [1=t-1] and [il=-1] differ m from those timestep n. Put differently, k[ * k[::t-1] and Vi:--11 * v[I:-11] if n. * m. The bidirectional attention introduced by diffusion language models therefore breaks the global reuse assumption that supports conventional KV-Cache.
Non-sequential decoding order. Generation in DLMs does not follow strictly left-to- right order. Instead, the model dynamically fills masked positions based on probabilities computed at each denoising step. As result, the positions of decoded tokens only revealed after the model forward pass, and the subsequent update may target any position in the sequence, rather than progressing sequentially. This uncertainty prevents from pre- deeedeternining which token i will require the computation of its hidden states h( and its Qli,, K( and Vll

---

Similarity Heatmap
Ropressentation Distance vs. After)
before decade
decode
175
F: Eeftre dacods
<. Abur decook
Defare decodk
decade
Tiissip the maximnn
With Dae Daroded Meen
3:11
124
Lat [ckitt]
2nd mas (jaie)
27-
2nd (sttat)
20
Decoded Tolles
Nith Decoded Thlles
N-th Decodee Trioen
Figure 2: (a) We present heatmap illustrating the pairwise similarities among the key states across different timesteps. Here we use LLaDA with I. 128, T= 128 and the block size set to 64. We then compute the Euclidean distance between consecutive steps 1 and i + 1 to analyze the dynamics of intermediate representations. (b) We report the average distance measured before and after the decoding of each token. (c) We highlight the 2op-2 steps exhibiting the largest and smallest changes in the key and value states for each token their decoded order.
Representation Dynamics for tokens in Diffusion Sampling We investigate DLMS whether K and V can be reused. We focus the dynamics of K and V for each token, and the results are shown in Figure 2 Interestingly, we observe several noteworthy patterns in the dynamics of QKV states: (1) Despite step-to-step differences, the key and value embeddings, K and V. exhibit consistently high similarity across timesteps, as shown in Figure aa. (2) Once a token is decoded, its representation becomes relatively stable in subsequent steps, whereas the representations of still-masked tokens continue to fluctuate significantly. This phenomenon evident in Figure b)), where QKV fluctuations are more pronounced prior to decoding than thereafter.
(3) The most substantial changes in K and V occur at the decoding step of each token and then in the early stages of the demoising process. This is reflected by prominent changes along the diagonal in FigureZ[o) corresponding to the i-th decoding step which decodes the i-th token.
These observations provide key insights into the temporal structure of discrete diffusion models and motivate the design of our KV-Cache mechanism for diffusion language modeling.
3.3 Delayed KV-Cache for Masked Diffusion Language Models
We first present a more general non-sequential KV-Cache formulation that replaces the contiguous slice (1itt11] used in Eq with an aubirrrryordee index set S, I (1,.... ... L} and the next token position from the fixed i D,. The cached keys and values gathered from previous steps are now K which retrieves cached states at the positions specified by indexes in Sy:
Q (K.SUID.)
(.S with {
K S.(tt) concat_reorder S. K.)) S. concat_rrordue (- VDr)
softanax
Vde
(2) where De is the denoised token at step f. If we use random sampling for diffusion, then before any inference, we can know the decoding order of the sequence. We here use the notation of D,, and later would not depend on knowing the decoding order. The operator concat. reorder proposed to enhance the efficiency of the indexing and gathering of K and V here. We explain in the appendix about this operator and how it works with ROPE 42 and how accelerates.
We first extend the formulation in Eq by generalizing the query input Q. from the single-token decoding to the multiple and arbitrary token decoding. Specifically, at decoding step f, we construct a dynamic set M,, where M, denotes the subset of tokens that are not yet finalized during generation. And with the h:M: calcuated, also can get the corresponding Q Q.Mi K.M, and V.M. We delay the caching of each token, not the time it is appended in the input, but rather on the step it is decoded:
concat concat_reorder
K
KP D
= concat_reorcee
TLMs
K M.
(3)
Cache from 8-1 Calcualte
Cache from t-1 Calculate
where I denotes the set of all tokens involved in the denoising process. This design reflects our core observation in the previous section: only the decoded tokens eligible for caching, while
5

---

the remaining masked tokens must be re-encoded at each step. Besides, this method solves the problem that we need to predefine or predict the denoising order, as M, is explicitly known each step. Cached keys and values corresponding to I \ .M, are reused across steps, while non non-finalized positions are recomputed at each step to ensure correctness under bidirectional attention masking.
One-step Delayed Caching, As our analysis in F c), the most significant change in K and V occurs exactly at the step where token transitions from [MASK] its decoded form. Currently, K. would be used for caching, it is longer in the masked set M,, However, K. can differ substantially from K.D,, and prematurely reusing it lead severe performance degradation. To address this, we introduce one-step delayed caching. At timestep t, we use the masking state from the previous step, MM-1, determine which tokens are cacheable. The method, named dKV-Cache- Decode, is finally formalized as:
Q QQ (Kt)
K concat_reorder
K Ki K.M 1-1 D.Me- vel--1
2, = softmax
V? with
dg
V concat_reorder
While this slightly reduces efficiency, find it to be critical for maintaining accuracy and stability in the proposed V-Cace mechanism for diffusion language models.
Cache Refreshing Mechanism. While it is possible apply caching after each token is decoded and reuse it throughout the denoising process, in practice, when the sequence sufficiently long, occasionally recomputing the cache incurs small computational overhead. To maintain consistency and improve correctness during decoding, we add a cache refreshing mechanism. Every N steps, the stored cache would be discarded and refreshed. The calculation would revert back to the normal calculation, resulting in an empty set to replace M for this refresh step in Eq
dk UKV-Cache-Prefill and dKV-Cache-PD. The set M can be further divided into two subsets: decoded tokens and always-decode tokens, i.e., prefill tokens. Our experiments show that prefill tokens primarily attend to each other, indicating limited influence from later tokens. Based on this, we adopt a special strategy, dK V-Cache-PPreflle that caches prefill tokens without refreshing. This design aligns with the disaggregation of prefill and decoding phases [57] for serving DLMs. Building on this, we have another variant, dK UKCChcce--PD, which intermittently refreshes only the newly decoded tokens, while keeping the key and values of prefill tokens without any recomputation.
3.4 dk dKV-Cache-Greddy: Greedy Formulation of dk V-CChhe
However, the above method still incurs ((T.)) complexity, which is less efficient than the ( complexity of ARs. This is primarily because M, initially consists of the entire sequence with tokens and only narrows
single token at the end. To improve efficiency, it is essential to decouple [.M]] for each step from the sequence length L.
Instead of the minimally refreshed caches proposed above, adopt more relaxed cache mechanism to refresh more so as to mitigate the performance degradation caused by stale K and V. Building on earlier observation that token representations undergo significant changes at their decoding step, we define Me to include only three components: the token at the current step D., the token from the previous step D(t 1) (motivated by one-step delayed caching), and local window W(t). For this local window, we extend it to include the token itself and its neighboring tokens within a fixed-size D.
where is the window size. We evaluated local window W; {>; i E D. D D. and found that the latter yields better performance. Since the windows centered at both Dj and window size [W] fixed (set at most 6 in our experiments), this strategy introduces additional computation but retains an overall time complexity of O())).
Experiments
4.1 Experimental Setup
We tested our method under the original evaluation benchmark of LLaDD [37] and Dream 52 Datasets: We conduct comprehensive evaluations across diverse set of benchmarks that assess

---

Table 1: Benchmark results on LLaDA-8B Insttusttct. We use zero-shot evaluation here Detailed
configuration is listed in the Appendix. We set the cache refresh step for Www-Cache-DDoooeee to be 8 to be 2. The window size of EEV-Cache-Greedy
and dk V-Caachh--Gresss
listed the bracket.
Base (mandoos))
Few-Steps
dKV.CCache
WWV-Cache-Grese w/ Window (aadom)
Base (coufidence)
Half-Steps (confieence)
dKV-Cache- Decdde
Remasking
(random)
(randoo))
(confidence)
MMLU
51.79
43.19
45.77
47.70 (4) 45.72 (1.513)
51.11
51.11
51.00
30.30
47.49 (1,678)
50.56 (1:57k)
28,27
55,80 (1.978)
66.52 (2.35k)
GSMSK
72.25
65.58
67.93
68 (4)
77.56
77.91
78.85
15.36
24.08
(1.59x)
25.47 (1.68)
24.76 (1.63))
14.31
28.71
(2.008)
27.50 (1.92)
Math500
27.4
21.8
26.0
27.0 (4)
36.6
34.2
36.8
19:36 (1.61x)
(1.70x)
19.86 (1.66K)
23.10
(2.00x)
24.46 (2.12%)
GPQA
27.46
24,78
26.79
28.35 (4)
30.80
27.68
28.13
11.40
18.59 (1.688)
19:27 (1.680)
18.26 (1.600)
11.86
23,88 (2,0130
28.75 (2.42%)
HvaunEval
19.88
15.61
15.13
15.37 (4)
39.63
33.54
46.34
7.50
12:50
(1,67x33
12.31 (1661))
12.13 (1.62x)
7,08
14.18 (2.00x)
13.76 (1.83)
MBPP
21.4
15.6
17.8
20.4 (2)
40.4
33.8
40.4
7.51
12.97
(1.73%)
12:55 (1.628)
12.44 (1.668)
7.50
15.01
(2.00x)
13.95 (1.86)
Table 2: Benchmark results on Dream- Baam-BBase-BB- is in Appendix. We set the cache refresh interval for dK Decccee and dKV-Cache-PD to 4.
7B. We use the few-shot ICL here and the configuration
Dream-7B
Half-Steps
dKV.Cache-Desue
dKV-Cache-Prefill dKV-Cache-PD
76.88
68.08
76.57
75.66
74.07
GSM8K (8-shot) L= 256
T = 256
15.1 (1.00))
30,3 (2.00x)
31.6 (2.09x)
53.6 (3.55x)
50,2 (3.32x)
68.81
46.63
65.35
65.96
63.31
T 128
30.3 (2.01%)
60.5 (4.01x)
62.31 (4.13%)
107.4 (7.11x)
99.5 (6.6%)
55.8
45,2
53.4
55.2
51.0
MBPP (3-shot) L= 512
T = 512
5.4 (1.00x)
10.8 (2.00x)
10.4 (1.93%)
13.6 (2,52x)
14,5 (2.69%)
45.2
26.2
43.4
41.8
42.6
T 256
10.8 (2.00%)
21.5 (3.98%)
20.6 (3818)
27.1 (5.02x)
28.9 (5.35 x)
$7.93
37.20
$7.32
56.10
59.76
HumanEval (0-shot)
T 512
10.3 (1.00x)
20.5 (1.99x)
15.5 (1.50x)
14.4 (1.4xx)
17.4 (1.69%)
515
37:20
18.29
31,70
33.54
31.70
T= 256
20.5 (1.99x)
40.9 (3.97x)
31.1 (3.02x)
28.7 (2.79x)
34.8 (3,38%)
general language understanding [23), mathematical reasoning [10 30 39], and code generation 8 4
Since the multi-choice evaluation based on the token likelihood doesn' it require more than one step for inference, we request the models to generate the answer letter and match the generated answer with the ground-truth answer. Evaluation: We follow the prompt in simple-evaly for LLaDA, making the model reason step by step. On Dream, we follow the evaluation setting of Dream to conduct few-shot in- in-context learning 66 Other implementation details are listed in the Appendix. Baseline: We choose the few-step sampling method (50% steps for Half-Steps and 62.5% steps for Few-Steps) as our baseline and select the number of steps such that their sampling speed is comparable to or slower than ours, and showing that our method can have better performance.
Metric. We report the accuracy for performance and token/s for speed. We tested the speed on A6000 (for LLaDA) and H20 (for Dream). Besides this, we use one more metric to show the cache ratio, calculated as: + > (Teache //[T,], where T is the total number of decoding steps. T, denotes the number of tokens processed step i (normally the whole sequence) and T,ache is the subset of tokens whose KV pairs are reused from cache at timestep i, which is [M,].
4.2 Performance and Speed with dKV-Cache
We begin by addressing the central question: What are the performance trade-offs and speedups introduced by applying dKV-Cache? For LLaDA, we evaluate two variants, dk UKK-Cache-Green Greeedy-rreyyy and UKV-Cache-Decool against baselines that accelerate generation by halving or reducing the number of denoising steps. As shown in Table [ dKV-Cache-Greedh consistently outperforms few-step
Phurps//githhubb. "hanps//gituuh.com/ppnnna cwmp/oaa//////mim/ecsaas https///gthbbocoom//etttteett//s
7

---

Table 3: Results long prefill settings. dKV-Cache-Decode refresh step of 4; UKV-Cache-Pree Prefill never refreshes.
GSMBK Accurary
Dream-78 7B
Half-Steps
dKV-Cache Decode
dKV.Cache Prefill
40
72.19
72.21
71.74
71.76
MMLU
T=8
(5-shor) 1. 8
(1.00R)
18.1 (1.99k)
25.2 (2.778)
(6.333)
T=4
18.1 (1.9%)
72.21
36.1 (3.97x)
71.63
49.2 (5.418)
71.69
67.3 (7.40x)
71.71
No Dewy +
Delay
0
GPQA
T- 128
35.49
35.71
35.27
Cache Radio
0.4
(5-shoo) 1 128
744 (1.00K
14.7 (1.99X)
18.2 (2.46x)
75.40 (10.19
35.49
35.27
34.15
35.27
Figure 3: Effect of one- step delayed caching
T 64
(1.99)
(3.97%)
3688 (4.97%)
139.9 (18.918))
baselines across most benchmarks, except for HumanEval. Notably, integrating a lightweight cache window yields substantial gains with negligible computational overhead. For dk IKCC-ache- Deccde, IKY-Cache-Decode achieves near-lossless performance with a high cache ratio and only a few refresh steps. Among all strategies, IKV-Cache-Decs and the baselines. Crucially, it maintains accuracy nearly indistinguishable
delivers the best trade-off, outperforming
both dk UKV-Cache-Gread from the full model, demonstrating that KV-Cache can also be applied in diffusion language models without sacrificing performance. Since dK UKV-Cache-GGi -reeddy relies on a predefined (e.g., random) decoding order, which results in marked accuracy drop relative low-confidence remasking, we concentrate our experiments more on dK V-Cache-Decode. We provide several case studies in the Appendix that compare the generated text before and after applying dK V-Cache.
The results on Dream are shown in Table 2 and Table 3 There is a small difference in the position of the decoded token since Dream is adapted from auto-regressive models and shifts the token position. We provide a detailed illustration in the appendix for this. Due to the use few-shot in-context learning, the model requires long input context, leading significant overhead from encoding those tokens repeatedly. In this setting, dKV-Cache-Prefif provides substantial speed improvements; for instance, on MMLU and GPQA, it achieves up to 10x acceleration. Across all tested datasets, observe that dKV-Cache largely outperforms the baseline under different prefilling and decoding lengths. We further evaluate the impact of applying dKV-Cache few-step diffusion models and observe consistent trends: as the number of diffusion steps increases, our method yields even larger gains over the baseline. For example, on GSM8K with a decoding length of 256, the baseline model with 64 steps achieves 46.63 Pass @ with a 4x speedup, whereas dK V-Cace attains 6.6x speedup while significantly improving performance 63.31 (+16.68).
4.3 Analysis
The one-step delay in dKV-Cache. Figure Baaure]illusraaaa illlutraates the impact of applying a one-step delay to the cache mechanism. Without the delay, performance remains acceptable at low cache ratios but degrades rapidly as the cache ratio increases. Introducing a one-step delay stabilizes the generation quality, enabling the model maintain nearly lossless performance even under high cache ratios.
Performance on different decoding length, denoising steps and refreshing steps. Figure presents the results of applying dK IKV-Cache-Decode and rations, including different decoding lengths, refresh intervals, sampling steps, and window sizes. Overall, our method consistently achieves performance comparable the original model without dK dKV-Cache, effectively pushing the Pareto front forward. We observe the following findings: (1) Decoding robustness: The performance impact of our method is largely insensitive the number of decoding steps, indicating strong robustness across varying generation lengths and sampling steps. (2) Enhanced long-form generation: In tasks involving longer outputs (e.g.. L 512), our method outperforms the baseline, even improving generation quality from 80.97% to 83.13% on GSM8K and from 39.63% to 46.34% for HumanEval. The results imply potential inefficiency bidirectional attention aggregates contextual signals, pointing to redundancy or underutilization in long-context modeling. (3) Effectiveness with only a few refreshing: Even with infrequent refreshes (e.g., every 16 steps), the performance degradation remains small. (4) Effectiveness of local windows: Incorporating a local window notably enhances the performance dk V-Ca-Chhh-Greee -eeeeyy-Gyeyyyy with minimal additional computational cost.
under various configu-
how
8

---

L-512 5-512
Lm514, 5-256
65
2.3%
Refresh =
Refresh
Performance (Accuracy ho
535 5m122
L- 125, 5-54
60
Refresh
Refresh
5-32
Performance (Accuracy
Fareto
55
Cacho
Window
50
1,6x
Window
who Cach
45
40
Cache
Befrwshm2
35
65
&efreshind
Refrezh-8
30
L-128,
S-128
Belree-15
20
40
60
80
1000
120
140
1.00
100
120
Speed (lokensss))
Speed (Tokans/s)
Figure 4: decoding length L, sampling steps S, refresh intervals and the window size.
Decfde(left) and dK VKV-Cache- UKV-Cacce-Greedyytigh on GSM8K with different settings:
Speci (Pro
Natrory y Roulpe It. 21of
Spocdd 5321
Merrar Fockerint (P22
22%
230
Deculi
Thiugha boomun
40088
800000
20000
256
255
L-15
256
L--34
224
Figure 5: Speed and memory for CKV-Cache-Deoool and dK --Cache-Greedy. The number (2 and 8) means that in every n steps, the cache would be refreshed.
Memory and speed analysis. We analyze the speed and memory footprint of dKV-Cache-Dcood and Greedy across varying decoding and prefill lengths. Our method achieves substantial inference acceleration, ranging from 1.75x to 3.3x, while introducing only a modest increase in memory usage. Notably, dKV-Cache- Brece-yreedy demonstrates greater potential for accelerating inference while dKV-Cache-DD -eccod would be capped. In our main experiments, we observed that setting the refresh interval larger than for dk IKV-Cache-Greedd under the same refresh interval, dK IKV-Cache-Greade consistently achieves higher speedups than dK V-Cache-Decode, highlighting its potential advantage when refresh frequency is relaxed.
may degrade performance. However,
5 Conclusions and Limitations
In this work, we explore the feasibility of incorporating the caching mechanism into diffusion language models. Specifically, we propose delayed VV-Ccche for DLMs, motivated by our empirical observations on the dynamics of the token representations throughout the diffusion process. We introduce two cache variants, dK dKV-Cache-Decool delayed caching for improved compatibility with diffusion-basel generation. Our analysis reveals that introducing delay is crucial for the cache to function effectively in this setting. Extensive experiments demonstrate that our approach largely accelerates inference while maintaining model performance. One primary limitation of this work lies in its focus on algorithmic design in isolation. While our proposed method introduces an effective caching mechanism from purely algorithmic perspective, diffusion language models also exhibit substantial room for improvement at the system level. We believe that future research integrating algorithmic innovations with system-level optimization, such as memory management, parallelism, and hardware-awan execution, could unlock further efficiency gains and performance improvements for DLMs.
and dKV-Cache-Grea -reede- each designed to leverage
Acknowledgment
This project supported by the Ministry of Education, Singapore, under its Academic Research Fund Tier 2 (Award Number: MOE-T2EP20122-00065

---

References
[1] Joshua Ainslie, James Lee-Thorp. Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint Arv::2305 13245, 2023.
[2] Marianne Arriola, Aaron Gokaslan, Justin Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshow. Block diffusion: Interpolating between autoregressive and diffusion language models. arXiv preprint (St::2503300573.. 2025.
[3] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 14:17981-17993, 2021.
[4] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprint Xiv: 100:21088.77355 07732, 2021.
[5] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[7] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 13355, 2022.
[8] Mark Chen, Jerry Tworek. Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrow, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizaheth Barnes, Ariel Herhert- Voss, William Hehgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder. Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. 2021.
[9] Pengtao Chen, Mingzhu Shen, Peng Ye, Jianjian Cao, Chongjun Tu, Christos-Savvas Bouganis, Yiren Zhao, and Tao Chen. 5-dit: A training-free acceleration method tailored for diffusion transformers, 2024.
[10] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchrr Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:21 141668 2021.
[11] Zihang Dai, Zhilin Yang. Yiming Yang. Jaime G. Carbonell, Quoe V. Le, and Ruslan Salakhutdinov. Transfomer-xl: Attentive language models beyond fixed-length context. ArXiv, abs/1901 022860 2019.
[12] DeepSeek-AL. D technical report. ArXiv, abs/2412. 19437, 2024.
[13] Justin Deschenaux and Caglar Gulcehre. Promises, outlooks and challenges of diffusion language modeling. arXiv preprint arXiv:2406.1 1473, 2024.
[14] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony S. Hartshorn, Aobo Yang. Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aur' elien Rodriguez, Austen Gregerson, and Ava Spataru et al. The llama herd of models. ArXiv, abs/2407.2111 21783, 2024.
[15] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
[16] Zach Evans, CJ Carr, Josiah Taylor. Scott H. Hawley, and Jordi Pons. Fast coniin-condiiioneed latent audio diffusion. ArXiv, abs/2402.00825, 2024.
10

---

[17] Guhao Feng, Yihan Geng, Jian Guan, Wei Wu, Liwei Wang, and Di He. Theoretical benefit and limitation of diffusion language model. ArXiv, abs/2502.09622, 2025.
[18] Suyu Ge, Yuman Zhang. Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive KV cache compression for LLMs. In The Twelfih International Conference on Learning Representations, 2024.
[19] Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, et al. Scaling diffusion language models via adaptation from autoregressive models, arXiv preprint arXiv:2410. 17891, 2024.
[20] Jiatao Gu, James Bradbury, Caiming Xiong. Victor OK Li, and Richard Socher. Non-autoregrrssive neural machine translation. arXiv preprint arXiv: 1..222,, 2017.
[21] Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. Ssd-Im: Semi eemn--uutoeerresivve simplex-based diffusion language model for text generation and modular control. arXiv preprint arXiv:2210.17432, 2022.
[22] Zhengfu He, Tianxiang Sun, Kuanning Wang. Xuanjing Huang. and Xipeng Qiu. Diffusionbert: Improving generative masked language models with diffusion models. arXiv preprint arXiv:2211.
2022.
[23] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song. and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[24] Jonathan Ho, Ajay Jain, and Picter Ahbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840 6851, 2020.
[25] Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances neural information processing systems, 12444 2444-12464544----1111111 2021.
[26] Coleman Hooper, Schoon Kim, Hiva Mohammadzadch, Michael W Mahoney, Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length Ilm inference with kv cache quantization. Advances in Neural Information Processing Systems, 02:1127--300 2024.
[27] Jacyeon Kim, Kulin Shah, Vasilis Kontonis, Sham Kakade, and Sitan Chen. Train for the worst, plan for the best: Understanding token ordering in masked diffusions. arXiv preprint arXiv:2502.0676 2025.
[28] Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvidea: A systematic framework for large video generative models. arXiv preprint XrX:v:2412.00000 033603 2024.
[29] Xiang Li, John Thickstun, Ishaan Guirajani, Percy S Liang, and Tatsunori Hashimoto. Diffusion-Im improves controllable text generation. Advances in neural information processing systems, 0144328-4443 2022.
[30] Hunter Lightman, Vincet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lec, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint 2.0.::::3352000500 2023.
[31] Haozhe Liu, Wentian Zhang, Jinheng Xie, Francesco Faccio. Mengmeng Xu, Tao Xiang, Mike Zheng Shou, Juan-Manuel Perez-Rua, and Jürgen Schmidhuber. Faster difffusion via temporal attention decomposition. arXiv preprint 240.::2000.02747, 2024..
[32] Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence importance hypothesis for I1m kv cache compression at test time. Advances in Neural Information Processing Systems, 36:52342-52664 2023.
[33] Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution, arXiv preprini arXiv:2310.16834 2023.
[34] Zhengyao Lv, Chenyang Si, Junhao Song, Zhenyu Yang, Yu Qiao, Ziwei Liu, and Kwan- Yee Wong. Fastercache: Training-fre video diffiusion model acceleration with high quality. arXiv preprint MiXiv:2410.19355. 2024.
[35] Xinyin Ma, Gongfan Fang, Michael Bi Mi, and Xinchao Wang. Learning to cache: Accelerating diffusson transformer via layer caching. Advances in Neural Information Processing Systems, 37:133282-13323 2024.
11

---

[36] Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deepcache: Accelerating diffusion models for free. arXiv preprini arXiv:2312.0085, 2023.
[37] Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models, arXiv preprint ar Arii::2502.088 09999, 2025.
[38] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4195-4205, 2023.
[39] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024.
[40] Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 37:130136-130 130114,44 2024.
[41] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[42] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127064 2024.
[43] Qwen Team. Qwen2.5 technical report. ArXiv, abs/2412.15115,, 2024.
[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[45] Maurice V Wilkes. Slave memories and dynamic storage allocation. IEEE Transactions on Electronic Computers, (2)2770-71,, 2006.
[46] Felix Wimhauer, Bichen Wu, Edgar Schoenfeld, Xiaoliang Dai, Ji Hou, Zijian Hc, Artsiom Sanakoyeu, Peizhao Zhang, Sam Tsai, Jonas Kohler, et al. Cache me if you can: Accelerating diffusion models through block caching. arXiv preprint arXiv:2312.03209, 2023.
[47] Tong Wu. Zhihao Fan, Xiao Liu, Hai-Tao Zheng. Yeyun Gong. Jian Jiao, Juntao Li, Jian Guo, Nan Duan, Weizhu Chen, et al. Ar-diffusion: Auto-regressvee diffusion model for text generation. Advances in Neural Information Processing Systems, 36:39957-399744 2023.
[48] Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu,, and Song Han. SANA: Efficient high-resolution text-to-image synthesis with linear diffusion transformers. In The Thirteenth International Conference on Learning Representations, 2025.
[49] Minkai Xu, Tomas Geffner, Karsten Kreis, Weili Nie, Yilun Xu, Jure Leskovee, Stefano Ermon, and Arash Vahdat. Eaaee diffusion language models for text generation. In The Thirleenth International Conference on Learning Representations, 2025.
[50] Xingyi Yang and Xinchao Wang. Hash3d: Training- free acceleration for 3d generation. arXiv preprint (xXv:240440660)) 2024.
[51] Jiacheng Ye, Zhenyu Wu, Jiahui Gao, Zhiyong Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Implicit search via discrete diffusion: A study chess. ArXiv, abs/2502. 19805, 2025.
[52] Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zinu Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b, 2025.
[53] Jiasheng Ye, Zaixiang Zheng, Yu Bao, Lihua Qian, and Mingxuan Wang. Dinoiser: Diffused conditional sequence learning by manipulating noises. arXiv preprint arXiv:2302.1002. 2023.
[54] Zhihang Yuan, Hanling Zhang, Lu Pu, Xuefei Ning, Linfeng Zhang. Tianchen Zhao, Shengen Yan, Guohao Dai, and Yu Wang. Ditfastattn: Attention compression for diffusion transformer models. Advances in Neural Information Processing Systems, 37:1 196-1219, 2024.
[55] Lingxiao Zhao, Xueying Ding, and Leman Akoglu. Pard: Permutation-invarian autoregressive diffusion for graph generation. arXiv preprint 240.022402.036687 2024.
[56] Xuanlei Zhao, Xiaolong Jin, Kai Wang, and Yang You. Real-time video generation with pyramid attention broadcast. arXiv preprint Xr0: 12588 2024.
[57] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. Distserve: Disaggregating prefill and decoding for goodput large language model serving. In USENIX Symposium on Operating Systems Design and Implementation, 2024.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the claims made, and experiments are conducted to support those claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
In Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
13

---

Justification:
No theory is proposed in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Experimental details are listed in Section 4.1 and the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer: [Yes]
Justification: The code would be submitted in the supplementary files and we would release the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https ioweeeomiisiiiioooio
//nips. //nips.cc/pil wwwppbblccgguiicuceccccc
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental details are listed in Section 4.1 and the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification:
No training is involved in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
15

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://uwwrp...c/.p...../aaaaaaa
Answer: [Yes]
Justification:
The research conforms to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The research poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: They are properly cited and the license and terms of use are properly respected.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
17

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
18

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
19

---

A Design for concat_reorder
concat reorder is our implementation of dKV-Cache designed to improve the speed of dKV-Cache in diffusion language models. Unlike standard KV-Cache used in autoregressive models, dKV-Cache requires gathering and scattering keys and values from arbitrary positions, introducing indexing operations that are less efficient than the simple concatenation in contiguous space used in ARs.
In dK V-Cache, the cache process involves two additional indexing operations: (1) At the cache step: After computing keys and values, we need to gather the corresponding states of cached tokens at non-continuous positions. (2) At the reuse step: obtain the whole matrices for key and value, we need to scatter these vectors back to their original positions in the sequence, In contrast, KV-Cache in ARs only requires matrix slicing and concatenation, making it significantly more efficient.
To minimize the overhead of gathering and scattering, we propose an algorithm similar to that of standard KV-Cache to avoid too many indexing operations. The key idea is to reorder token positions during the forward calculation of the Transformer, placing all cached tokens contiguously on one side (e.g., left) and newly decoded tokens on the other. This allows us to move parts of the indexing operation to the token level (matrices with shape [B, L.)) instead of the intermediate states (matrices with shape [B, L, DJ):
T.Mi and value states V A.Me based on the At Step t-1: Gather the cached key K K-M position index I \\ M with one indexing operation.
At Step t:
Reorder the sequence, making the cached tokens (at position I \ MM-1) at the left, and uncached tokens (at position My-1) at the right.
At Step t: Using concat_reorder for K M. K, K-M-) and for (V11 (VTMM--, V.M--
:-1
First, concatenate the cached and current key/value states directly without further gath- ering/scattering (concat, for getting all K and V to calculate attention), and reorder the whole KV matries based on V.T. T to get the cached states for the next step (reorder, for obtaining the cache).
The reorder operation is to know the position mapping from [I M, to [I Mr For example, if the unmasked position t 1 is [2, 4, 5] from a sequence of tokens, and at step ++ is [2, 4, 5, 7J. Then [Z \ Mr-1;M,-3 would be [2, 4, 5, 0, 1,3, 6, 7], and [Z \ Mr;Ma would be [2. 4, 5, 0, 1, 3, 6]. The mapping would be [0. 1, 2, 3, 4, 5, 6]. and we only need get
the corresponding entries [0, 1, 2, 7] from K K M
and v MM-1, V. Me-1
The only remaining thing is that the change of token position would impact the positional encoding. However, this is easy to solve; we can also reorder the positional embedding. Reordering positional embeddings is required only once per model evaluation and can be shared across layers, thus would not cost much time.
Furthermore, since our method introduces a one-step shift in caching, the position of cached tokens at step corresponds to the token positions decoded from step t - 1. This alignment allows to track which key and value entries need be cached without storing the entire key/value matrices, which, to cache which tokens, can only be known after the decoding results at step t.
We present the pseudo-algorithm of our approach in Algorithm While it largely improves inference speed over the naive implementation, the concat and reorder operations still introduce some overhead. We believe there substantial potential for further optimization.
B Design for Dream
Dream has different caching strategy with LLaDA. The main reason for this is that Dream is adapted from pre-trained autoregressive models, which would make the position of output align with the probability of the next token, instead of the current token in the traditional setting of masked diffusion models. This would make difference in the caching strategy and we investigate between different designs of those caching strategies:
20

---

Algorithm Pseudo code for dKV-Cache-Decode. We take step I as an example.
Require: Sequence x 1:L at step (Simplied x), position index of masked tokens M,, cached Key
c(t)
1- MeMe and Value 7-1
M
x' f x[M[-1]
D M for one-step shift
2: PE -PE[IY M PE[M,I]
Positional embeddings: cached on left, uncached on right b T: Calculation in Transformer to get Q, b Get all K and
3: MMK. V.M. t- T(x))
4: KS + Concat K- MM-M K ( V + Concat
( V-M.1 r-1
5: K, K,MMe Reorder K 1)), V.T.M Reorder V7, 1')
I The index of \ M, in the
[x[I \ [M-i]; x[M--i]]
6: p' A(Qu S,M,,KK Kr, V?)
7: p + Scatter(p' .M-1))
Put token logits back the original position
8: Return p. K, K.M, V.M
Al.
00
Sky
Un Stift
(8) Righn Shift
(c) Und Shift
KAsK Toker
Decoded Tales
Capst
Cangaat
Orrhe
Compatr
Preactos
Figure 6: Three variants for the caching strategy for diffusion language models adapted from auto- regressive language models, which would have shifted output position.
Un-Shift, Figure aa): We cache the unshifted token representations. Specifically, for the t-th token, we store its key and value as K' and V at position
Right-Shift, Figure 6 (b): Given that the hidden state is highly sensitive changes in input,
we also explore right-shifted variant. Here, for the t-th token, we cache K and V
UngRight-Sift, Figure(ojc:: We introduce a stricter variant where eaching conditioned on both input stability and decoding completion. For the t-th token, we cache its features only after its input is fixed and has been decoded.
The one-step shift is still used here. For example, in the right-shift variant, the t-th token is fed into the model at position t+1 in the next step, and we cache its output K'+1 and Vi+1 then. The results shown in Table 4 where Un&right-shift would have the best performance, and the right shift would largely harm the model performance. However, we use the Un-Shift in our main experiment, since Un&right-shift is incompatible with the above concaf_reorder.
Table 4: Comparison between different types of caching strategy for Fream-Base--e
Un-Shift
Right-Shift
Unghi Shift
MMLU
71.78 76.34
64.60
71.73
GSM8K
32.68
77.71
C Evaluation Details
C.1 For LLaDA
We re-implemented the evaluation of LLaDA on those reported datasets. We generate and extract the final answer instead of comparing the log prob the multiple-choice question. Thus, the result of MMLU and GPQA is lower than reported since the model sometimes cannot generate the answer in
21

---

the given format or does not generate the answer. We show the configuration of each experiment in T
Table 5: Configurations of experiments LLaDA-Instruct.
Base
Few Steps
dKV-Cache Greedy
dKV-Cache Greedy
Half Stefs (oonlidence)
dKchCache Decode
Remasking
(random/ contidence)
(randoen)
(random)
(randon) Window Size
(ountidence)
Confgurstion
Steps T
Cache Interval
Steps T
Cache Interval
MMLU
L=32, T=32, B-16
T-20
2
T-16
8
GSMSK
252
B=32
T=160
2
Math500
1 T=256, B=64
T=160
2
4
T=128
8
GPQA
L=128, Tw128, B-64
T=80
2
Tw64
HumanFval
La512, Tw512, Be32
T-320
2
4
Tw256
8
MBPP
L=512, T=512, B=32
T=320
2
2
T-256
8
C.2 For Dream
We follow the original evaluation pipeline for Dream and we also adopt two datasets, MMLU and GPQA, to generate the answer instead of comparing the probabilities. We follow all the hyperparameters set in the evaluation script, including the temperature, the remasking strategy, top_p and the number of few-shot in in-context learning.
D Impact of batch size on speed
175
150
Takan/s
11683
75
50
120
Batch Size
Figure 7: Impact of batch size on decoding speed. Evaluated on LLaDA with a single NVIDIA H20; prefill length fixed at 100 tokens.
Our inference pipeline relies heavily on indexing operations, gathers and scatters, that generate stream of small, non-contiguous memory accesses. At batch size of one, these uncoalesced reads make the inference memory-bound. As result, the GPU's compute units sit idle waiting for data. In some cases with the batch size equal to I, inference with caching can actually underperform the unaccelerated baseline. By contrast, scaling up the batch size can solve this problem and deliver far greater, more stable speed-up ratios over the baseline.
Case Study
We provide the study of our method in Table and Table Jnd Table g
http://hhhhccoo//to///Peeeeoooon ddeaalddddmmmmee gen.sh
22

---

Table 6: Case study on LLaDA--B-Instrrutt. Question: Lily can run 12 kilometers per hour for hours. After that, she runs 6 kilometers per hour. How many kilometers she in 8 hours?
Model
Generate Text (L=128, T=128, B-64)
LLaDA-8B-
1. Calculate the distance Lily runs in the first 4 hours:
Instruct (low
Speed: 12 kilometers per hour
confidence)
Time: hours
Distance: 12 km/h 4 = 48 km
2. Calculate the distance Lily runs in the next 4 hours:
Speed: 6 kilometers per hour
Time: hours
Distance: 6 km/h 4 h = 24 km
3. Add the distances together:
Total distance:
\boxed km
dKV-Cache-
1. Calculate the distance Lily runs in the first 4 hours:
Decode, Refresh 4
Distance = Speed x Time 12 km/h x 4 = 48 km
2. Calculate the distance Lily runs in the next 4 hours:
Distance = Speed x Time 6 km/h x h = 24 km
3. Add the distances from both periods to find the total distance Lily
runs in 8 hours:
Total Distance 48 km + 24 km 72 km
\boxed (72 km}
dKV-Cache- Decode, Refresh=16
1. Calculate the distance Lily runs in the first 4 hours:
km/hour * hours = 48 km
2. Calculate the distance Lily runs in the next 4 hours:
km/hour 4 hours 24 km
3. Add the distances from the first 4 hours and the next 4 hours to find the total distance she runs:
48 km 24 km 72 km
Therefore, Lily can run \boxed(72) kilometers in 8 hours.
dKV-Cache- Greedy, Refresh = 2
To determine how many kilometers Lily can run in 8 hours, we need to break down her rumning schedule and sum the distances. First, calculate the distance she runs at 12 kilometers per hour for 4 hours:
12 4 = 48
After 4 hours, she runs 6 kilometers per hour for the remaining 4 hours:
= 24
Now, add the distances:
48 24 72
\boxed [72)
dKV-Cache- Greedy,
For the first 4 hours Lily runs at 12 kilometers per hour, she runs 12 * = 48 kilometers. 8 hours hours are left after the first 4 (8 = hours).
Refresh
4
(random)
For the next hours, she runs at 6 kilometers per hour, so she runs 6 24 kilometers.
Adding the two distances together, Lily can run 48 + 24 = 72 kilometers in 8 hours.
Answer: \boxed(72)
23

---

Table 7: Case study on Dream-Instruct-7B.. Request: Write a story that ends with "Finally, Joey and Rachel get married."
Model
Generate Text (L=256, T=256)
Dream-
Once upon time, there was a young town in the middle of nowhere. He was a kind and gentle soul who loved to sing and play the guitar. One day, he met a beautiful girl named Rachel who was also a talented singer and musician. They fell in love and decided to get married. However, there was problem. Joey's parents were against the marriage because they didn't ' know Rachel well enough. They thought that she was not a good match for their son. Rachel, parents" opinion. She knew that she loved Joey and that he was the one for her. She decided to try and win over Joey's parents by showing them how much she cared about him. She started by cooking for them and inviting them to her house for dinner. She also took them on trip to the mountains where they could enjoy the scenery and the fresh air. After a while, Joey's 's parents started to see Rachel in a different light. They realized that she was a good person and that she loved Joey very much. They decided to give their blessing to the marriage. Finally, Joey and Rachel got married and lived happily ever after.
named Jocy who lived in small
Instruct-7B (Entropy)
the other hand, was not worried about Joey's s
dKV-Cache-
Once upon a time, in a small town called New York, there lived a young man named Joey and a young woman named Rachel. They were both very happy with their lives, but they were also very different from each other. Joey was a very outgoing person who loved to be around people, while Rachel was very introverted person who loved to be alone. One day, Joey and Rachel met at a coffee shop and fell in love. They spent hours talking and laughing together, and they realized that they were perfect for each other. They decided to be together and build their lives together. As the years went by, Joey and Rachel grew closer and closer. They shared their dreams and their fears, and they supported each other through thick and thin. They were perfect match, and they knew that they were meant to be together. Finally, Joey and Rachel got married. They had a beautiful wedding with all their friends and family in attendance. They were so happy to be together, and they knew that they would always be together.
Decode, Refresh = 16
dKV-Cache- Prefill
Once upon a time, in a small town called New York, there lived young man named Joey. Joey was a kind and caring person who always tried to help others. He had beautiful girlfriend named Rachel, who was also kind and caring. They were in love with each other and wanted to spend their lives together. One day, Joey and Rachel decided to get married. They planned their wedding for long time, and everyone in the town was excited to attend. On the day of the wedding, Joey and Rachel exchanged vows and promised love each other forever. After the wedding, Joey and Rachel moved into a beautiful house in the countryside. They were happy and content with their lives, and they had two children, a boy and a girl. The children grew up to be kind and caring people, just like their parents. As the years went by. Joey and Rachel continued love each other deeply. They had many ups and downs, but they always found a way overcome their challenges and stay together. Finally, after many years of marriage, Joey and Rachel decided to retire and live in small cottage in the countryside. They were happy and content with their lives, and they knew that they had found the love of their lives in each other.
24

---

Table Case study on Dream-Instruct-7B. Question: Lily can run 12 kilometers per hour for hours. After that, she runs 6 kilometers per hour. How many kilometers she run in 8 hours?
Model
Generate Text (I.=128, T=128)
Dream-
In the first 4 hours, Lily runs 12 kilometers per hour, she runs total of 12 4 48 kilometers. In the next hours, Lily runs 6 kilometers per hour, so she runs total of * 4 24 kilometers. Therefore, in 8 hours, Lily run total 48 + 24 = 72 kilometers. ##### 72
Instruct-7B
(Entropy)
dKV-Cache- Decode, Refresh =
In the first 4 hours, Lily can run 12 kilometers per hour, she can run total of 12 * 48 kilometers. In the next 4 hours, Lily can run 6 kilometers per hour, so she can run a total of run a total of 48 24 72 kilometers. #### 72
= 24 kilometers. Therefore, in 8 hours, Lily can
dKV-Cache- Decode, Refresh= 16
the first 4 hours, Lily runs 12 kilometers per hour, so she runs a total 12 4 == 48 kilometers. In the next 4 hours, Lily runs 6 kilometers per hour, she runs a total of 4 = 24 kilometers. Therefore, in 8 hours, Lily can run a total of 48 24 72 kilometers. The answer is: 72
dk dKV-Cache- Prefill, No refresh
the first 4 hours, Lily runs 12 kilometers per hour, so she runs total of 12 x 4 = 48 kilometers. In the next 4 hours, she runs 6 kilometers per hour, so she runs a total 6 4 = 24 kilometers. Therefore, in hours, Lily can run a total of 48 + 24 (oozre((72) kilometers. The answer is: 72
25


Reviews:
[{"review_id": "igfk0c1Tr0", "reviewer": "Reviewer_TZvD", "summary": "This paper proposes a KV-Cache mechanism specifically designed for Diffusion Language Models (DLMs) to address their slow inference problem. The authors introduce \"delayed KV-Cache\" (dKV-Cache), which enables caching of key-value states in bidirectional attention models by leveraging the observation that token representations stabilize after being decoded. The method achieves 2-10× speedup with minimal performance degradation across various benchmarks.", "questions": "See weaknesses", "limitations": "See weaknesses", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n1. This paper focuses on a practical problem of DLMs, which show promise but suffer from significantly slower inference compared to autoregressive models due to the lack of KV cache. Therefore, DLMs require significantly higher computational costs.\n2. The paper provides a clear identification of why standard KV-Cache fails for DLMs and an empirical analysis of token representation dynamics throughout the diffusion process.\n3. The delayed caching strategy is intuitive and well-motivated while two complementary variants (dKV-Cache-Decode and dKV-Cache-Greedy) provide different trade-offs. Moreover, all methods require no further training\n4. Experiments show the practical acceleration over existing SOTA DLMs on various tasks.\n\n**Weaknesses:**\n1. The experiments remain somewhat confusing, as they utilize different variants and baselines across various figures for different base models. I would expect the authors to explain why we can only combine dKV-Cache-Greedy with the random version of LLada and why it cannot be used for Dream.\n2. From the experiment results, it appears that dKV-Cache-Greedy still works in limited cases; therefore, it remains a question of how to reduce complexity while maintaining performance in real-world applications.", "was_revised": true, "final_justification": "Based on the information, I think it is a valuable work for accelerating dLLMs. Therefore, I think a weak accept should be good.", "rebuttal": "We sincerely thank the reviewer for the constructive and valuable feedback. We would revise and include the below experiments and analysis in our manuscript.\n\n---\n\n**W1: Why cannot dKV-Cache-Greedy be used on Dream?**\n\n> *W1: The experiments remain somewhat confusing, as they utilize different variants and baselines across various figures for different base models. I would expect the authors to explain why we can only combine dKV-Cache-Greedy with the random version of LLada and why it cannot be used for Dream.*\n\nThanks for your question. dKV-Cache-Greedy can also be applied to Dream, and we provide the corresponding results below. We selected hyperparameters for dKV-Cache-Greedy to achieve approximately 2× acceleration.\n\n| |            Entropy |   Random  |  Random w/ dKV-Cache-Greedy  |\n| --|  --|  --|  --| \n| gsm8k    |    76.88  |   8.11    |   8.58    |        \n| humaneval |   57.93  |   1.22   |    1.83  |       \n| mbpp      |   55.8   |  17.4   |    17.4        |  \n| math       |  39.92  |  4.02    |   4.20         | \n| GPQA     |    36.83 |   33.7  |     34.15      |  \n\nDream's **random sampling yields significantly worse performance compared to entropy-based sampling**. In some datasets, we observed over a 60\\% drop in accuracy, with overall accuracy falling below 10\\%. We hypothesize that this is because DREAM is adapted from an auto-regressive model and thus inherently favors left-to-right generation. As a result, it is less compatible with random sampling strategies. \n\nWe think evaluating our method on top of DREAM with random sampling makes it difficult to draw some reliable and generalized conclusions to show the effectiveness of our approach. Thus, we didn't put the paper in the main body of the paper. We would add these results in the Appendix.\n\n---\n\n**W2: dKV-Cache-Greedy still works in limited cases**\n> *W2: From the experiment results, it appears that dKV-Cache-Greedy still works in limited cases; therefore, it remains a question of how to reduce complexity while maintaining performance in real-world applications.*\n\nThank you for your insightful question. We agree that dKV-Cache-Greedy, as presented in this paper, has limitations in real-world applications. We first identify that the core reason for these limited cases comes from **the unknown decoding order** and then propose some potential ways to solve this problem. \n\n* **Problem: Unknown decoding order limits the application of dKV-Cache-Greedy**\n\nHere, we try another sampling method, AR sampling, to decode LLaDA-instruct. AR sampling refers to enforcing a deterministic left-to-right decoding order, just as the AR models. Here are the experimental results on GSM8K:\n\n| Method | Token/s | Accuracy |\n| --- | --- | --- | \n|LLaDA (AR sampling)             |  24.6              | 82.8 |\n|LLaDA w/ dKV-Cache-Greedy       |  174.4 (7.08x)     | 83.5 (+0.7) |\n||||\n|LLaDA (random sampling)         |  23.2              | 79.5 |\n|LLaDA w/ dKV-Cache-Greedy       |  91.7 (3.95x)      | 77.4 (-2.1)|\n\nUnder this sampling method, we observe that model performance remains largely unaffected while enabling the use of a significantly higher cache ratio (less refresh of the cache). This, in turn, leads to improved inference speed without compromising accuracy.\n\n* **Solution: How to know the decoding order in advance?**\n\nThus, the primary challenge lies in its reliance on knowing the decoding order in advance, prior to generating each token. \nTo address this limitation and enhance the applicability of dKV-Cache-Greedy, we propose several potential solutions:\n\n1. **Decoding Position Estimation** \nAlternatively, we can design a lightweight prediction module to estimate the most likely position to decode in the next step, based on information from the previous iteration. By learning or heuristically approximating the next decoded position, we can use it instead of the random sampling method to achieve better performance and higher cacheability, with the $\\mathcal{O}(N^2)$ complexity during inference.\n\n2. **Relaxed Decoding Order via Window Tolerance**\nRather than requiring the exact decoding position, we can instead predict an approximate decoding range, leveraging the inherent window mechanism in dKV-Cache-Greedy to provide tolerance for the imprecise prediction of the decoding position.  This relaxes the constraint from precise ordering to coarse localization."}, {"review_id": "gbXjxqPS3W", "reviewer": "Reviewer_vSQt", "summary": "KV-Cache is a widely adopted technique for accelerating sampling in large language models (LLMs). However, the recently emerging Diffusion Language Models (DLMs) do not support KV-Cache. This paper analyzes the reasons behind this limitation and systematically investigates the dynamic behaviors that occur during the sampling process of DLMs. Based on these observations, the authors propose the dKV-Cache mechanism. Experiments on two popular open-source models, LLaDA and Dream, as well as a wide range of downstream tasks, demonstrate that dKV-Cache can boost inference speed by 2–10×.", "questions": "1. Could you provide more explanation for Figure 2(b)? It appears that the feature distances both before and after decoding are relatively high and fairly similar. However, the paper claims that representations become stable after decoding, while they fluctuate more before decoding. I find this somewhat confusing and would appreciate a more detailed clarification.\n\n2. Why do LLaDA and Dream use different evaluation settings? LLaDA is evaluated on an A6000 GPU, while Dream is evaluated on an H20. Also, since Dream uses a few-shot in-context learning setup, did LLaDA use few-shot as well? Moreover, why does this paper use the Instruct model for LLaDA but the base model for Dream?\n\n3. Could you clarify what T refers to in Table 2? It doesn't seem to represent the number of sampling steps. Also, why is T not reported in Table 1?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. This paper is the first to demonstrate that KV-Cache can be integrated into DLMs in a training-free manner, which is a novel and meaningful contribution.\n\n2. The paper begins with a careful analysis of the unique characteristics of the DLM sampling process and then designs the method accordingly—the motivation is clear and well-justified.\n\n3. The proposed method shows consistent improvements across both open-source models (LLaDA and Dream) and a wide range of tasks, including mathematical reasoning and code generation.\n\n## Weaknesses\n1. The paper lacks analysis of some key factors, such as the effect of batch size on acceleration. I’m particularly curious how the speedup of dKV-Cache changes under varying batch sizes (i.e., different GPU utilization levels).\n\n2. The paper does not include comparisons with autoregressive baselines. After applying dKV-Cache, is the sampling speed of DLMs actually faster than that of autoregressive models?", "was_revised": true, "final_justification": "I consider this work a new attempt at adding cache to diffusion language models. Most of my concerns have been addressed in the rebuttal. However, I remain doubtful about some experimental results, which appear inconsistent with concurrent studies (see Replying to Rebuttal by Authors). Therefore, I will keep my score at 4.", "rebuttal": "We deeply appreciate the reviewer’s constructive feedback, and we will revise the manuscript accordingly\n\n---\n**W1: Lack of analysis of batch size**\n\n> *W1: The paper lacks analysis of some key factors, such as the effect of batch size on acceleration. I’m particularly curious how the speedup of dKV-Cache changes under varying batch sizes (i.e., different GPU utilization levels).*\n\nThanks for your great question. We also think this is a crucial problem and thus we have conducted an analysis on the effects of batch size and decoding length in Appendix D. Please refer to Figure 7 in the Appendix for full details. Below, we summarize the key observations and conclusions:\n\n* At low GPU utilization (e.g., small batch size or short decoding length), the GPU is underutilized, and therefore the speedup is limited.\n\n* When batch size = 1, the improvement in latency is marginal due to insufficient utilization of the GPU. \n\n* When batch size >= 2, the speedup becomes pronounced, reaching approximately 1.4×.\n\n* When batch size >= 64, the acceleration ratio stabilizes at around 2.5×, indicating that our method has better speedup with full GPU utilization.\n\n| Method | bsz=1 | bsz = 2 | bsz = 4 | bsz = 16 | bsz = 64 | bsz = 256 | \n| --     | -- |  -- |  -- |  -- | -- |  -- |  \n| Origin           | 23.1 | 30.7 | 44.9 | 56.6  | 59.7  | 60.7 |\n| dKV-Cache-Decode | 24.5 (1.06x) | 43.3 (1.41x) | 71.9 (1.60x) | 121.3 (2.14x) | 145.7 (2.44x) | 153.3 (2.52x) |\n\n---\n**W2:  Comparisons with autoregressive baselines**\n\n> *W2. The paper does not include comparisons with autoregressive baselines. After applying dKV-Cache, is the sampling speed of DLMs actually faster than that of autoregressive models?*\n\nThanks for your great question. Despite incorporating dKV-Cache, current Diffusion Language Models still fail to outperform AR models in terms of decoding speed. Below, we present a speed and complexity analysis comparing LLaDA and LLaMA3 under two scenarios: long decoding length (L=1024) and short decoding length (L=32).\n\n* Long Decoding (L=1024, zero-shot, LLaDA-Instruct)\n\n|   | Token/s | Accuracy |\n| -- |  -- |  -- |\n| LLaDA (entropy-based sampling)  |  22.4      | 82.1 | \n| LLaDA w/ dKV-Cache-decode       |  56.0      | 82.5 | \n| LLaDA (random sampling)         |  23.2      | 79.5 | \n| LLaDA w/ dKV-Cache-Greedy       |  91.7      | 77.4 | \n| LLaMA3-8B                       |  1463.9    | 73.7 | \n\n* Short Decoding (L=32, zero-shot, LLaDA-Instruct)\n\n|   | Token/s | Accuracy |\n| -- |  -- |  -- |\n| LLaDA                        |  149.7 |  38.6  | \n| LLaDA with dKV-Cache-decode  |  230.4 |  38.4  | \n| LLaDA   (random sampling)    |  151.1 |  34.4  | \n| LLaDA with dKV-Cache-greedy  |  190.1 |  28.2  | \n| LLaMA-8B                     |  1524.9 |  1.74 | \n\n* FLOPs analysis\n\nWe provide a FLOPs analysis for attention computation per token during generation:\n1. For DLMs, the FLOPs is $ 4L^2d + 4Ld^2$ per token\n2. For ARs, the FLOPs is $ 4Ld + 4d^2$ per token\n3. With dKV-Cache-decode, the expected FLOPs per token is: $ 2L^2d + 2Ld^2$ \n4. With dKV-Cache-greedy, the FLOPs is $ 4wLd + 4wd^2$, where $w$ denotes the attention window size (typically $w \\leq 4$).\n\nThis indicates that **unaccelerated DLMs require approximately $L\\times$ more FLOPs than ARs for attention computation**, leading to significantly slower inference on long sequences. The proposed **dKV-Cache-decode** method mitigates this overhead by reducing the FLOPs **by half**, while **dKV-Cache-greedy** further improves efficiency by reducing the cost **by a factor of L/w**.\n\n \n**Summarization**: \n\n1. Despite utilizing dKV-Cache to narrow the gap, **LLaDA still lags significantly behind ARs in terms of decoding speed**, although it achieves competitive performance in terms of generation quality. \n\n2. One scenario where **DLMs outperform AR models is under short decoding lengths**. Specifically, when the decoding length is limited to L=32, AR models like LLaMA struggle to produce coherent or complete answers, achieving only a score of 1.74, whereas LLaDA maintains strong performance with a score of 38.7. This demonstrates that DLMs are more effective under constrained generation budgets.\n\n**Solutions**: \n\nTo make DLMs as fast or faster than ARs, several key components are needed:\n\n1. dKV-Cache-Greedy to match ARs' complexity with respect to sequence length $L$.\n2. Parallel decoding to reduce the number of sampling steps.\n3. Semi-autoregressive decoding to bound $L$ and limit computation.\n4. System-level optimizations such as vLLM and FlashAttention for improved runtime efficiency.\n\n--- \n**Q1: Explanation for Figure 2(b)**\n> *Q1：Could you provide more explanation for Figure 2(b)? It appears that the feature distances both before and after decoding are relatively high and fairly similar. However, the paper claims that representations become stable after decoding, while they fluctuate more before decoding. I find this somewhat confusing and would appreciate a more detailed clarification.*\n\nSorry for the unclear explanation here. For Figure 2(b), the calculation pipeline is as follows:\n\n1. We compute the pairwise K/V distance between two adjacent decoding steps, resulting in an array dis[:] of length T-1 for a sequence of T steps.\n\n2. To analyze the impact of decoding for the i-th token, we compute:\n      * Before Distance: avg(dis[:i]), \n      * After Distance: avg(dis[i:])\n\nFrom this, we obtain the curve shown in Figure 2(b). The key observations are:\n\n* The **before-decoding distance is generally larger than the after-decoding distance**, suggesting that the hidden states become more stable as decoding progresses.\n\n* We observe a fluctuation pattern, and in particular, the last step shows after-distance > before-distance. This is because the maximum change in hidden states occurs exactly at the step when a token is decoded. Hence, the ranking of distance magnitudes is:\n              at decoded step > before decode > after decode\n\nFor the last token, there’s no step beyond it, so avg(dis[-1:]) = dis[-1], which is exactly at the decode step and has a large value. \nWe think that there is room to improve the presentation of Figure 2(b). Following your suggestion, we plan to enhance it by adding separate curves for before, at decoded, and after distances. Thank you for the valuable feedback.\n\n---\n**Q2:  LLaDA and Dream with different evaluation settings**\n> *Q2. Why do LLaDA and Dream use different evaluation settings? LLaDA is evaluated on an A6000 GPU, while Dream is evaluated on an H20. Also, since Dream uses a few-shot in-context learning setup, did LLaDA use few-shot as well? Moreover, why does this paper use the Instruct model for LLaDA but the base model for Dream?*\n\nThanks for your question. Since there is no big difference between using a base/instruct model regarding the proposed dKV-cache algorithm, we choose the following two settings for Dream and LLaDA in our submission to cover as many cases as possible (whether ICL, Instruct/base, different prefill length):\n* LLaDA: LLaDA-Instruct, no in-context learning, A6000.\n* Dream: Dream-base, in-context learning, H20.\n\nHere we also show another group of results on Dream-Instruct-7B, with similar experimental results as in Dream-base-7B (no ICL, H20, same hyper-parameters as in Dream-Base-7B).\n\n|         |    Origin  |   Half-Steps  | dKV-Cache-Decode   |   dKV-Cache-Prefill        |\n| -- |  -- |  -- |  -- |  -- |\n| GSM8K     |     81.8    |    75.9    |         81.2         |         79.9 | \n| MBPP     |      58.2  |      43.8    |         57.0       |           56.2 | \n| HumanEval  |    52.7  |      28.7    |         53.0    |              57.3 | \n| MMLU     |      69.7   |     69.7     |        69.7       |           69.8 | \n| GPQA    |       31.5   |     30.4     |        30.8     |             30.8 |  \n\nWe provide here another comparison of Dream and LLaDA on the same device (H20) to directly compare their throughput (token/s), where LLaDA-8B is slightly slower than Dream-7B.\n\n| Decode Length | 32 | 64 | 128 | 256 | \n| -- | -- | -- | -- | -- |\n| LLaDA-8B | 82.9 | 60.8 | 41.4 | 24.6 |\n| Dream-7B | 86.9 | 63.8 | 42.1 | 25.1 | \n\n---\n**Q3: What T refers to in Table 2**\n>*Q3. Could you clarify what T refers to in Table 2? It doesn't seem to represent the number of sampling steps. Also, why is T not reported in Table 1?*\n\nThank you for your question and for pointing out the unclear expressions. In Table 2, T refers to the total number of decoding steps (as defined in Line 123). We will clarify this in the caption of the table for better readability.\n\nRegarding Table 1, the experimental setup follows the hyperparameter configuration used in LLaDA, which involves configuring T(sampling steps), L(decoding length), block size, cache interval and window size. For clarity, we listed the detailed hyperparameters, including T, in Table 5 in the appendix."}, {"review_id": "OaHz0DjSD1", "reviewer": "Reviewer_2PJU", "summary": "The paper introduces a caching mechanism, dKV-Cache, tailored for Diffusion Language Models (DLMs), which traditionally lack the ability to reuse key/value states during inference. The method is motivated by the empirical observation that token representations tend to stabilize after decoding, allowing delayed reuse of key and value vectors.\n\nTwo variants are proposed:\n\ndKV-Cache-Decode: A one-step delayed caching strategy that prioritizes accuracy and stability.\n\ndKV-Cache-Greedy: A more aggressive approach using a local window around the current token to reduce complexity from O(L³) to O(L²), trading off some accuracy for greater speed.\n\n\nExtensive experiments on benchmarks such as MMLU, GSM8K, and HumanEval show 2–10× inference speedups with minimal performance degradation. The method requires no model retraining, making it a lightweight and practical optimization for accelerating inference in existing DLMs.", "questions": "see cons", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Pros\n\n1. Well-defined research problem: The paper addresses a fundamental bottleneck in diffusion language models (DLMs)—the lack of key-value caching—which is the main reason for their slow inference compared to autoregressive models.\n\n\n2. Simple yet practical idea: The proposed method is straightforward and easy to implement. It provides a lightweight inference-time optimization without requiring any retraining or architectural changes.\n\n\n3. Clear writing and presentation: The paper is well-organized and the method is clearly explained, making it easy to follow and reproduce.\n\n\n4. Broad applicability: The technique works across various tasks (e.g., QA, math, code), different model sizes, and generation lengths, showing strong robustness and generality.\n\n\n\n\n---\n\nCons\n\n1. Lack of theoretical justification: The method is based purely on empirical observations rather than formal theoretical guarantees. In particular, the reuse of key/value states lacks a rigorous foundation, unlike AR models where caching is mathematically supported by causal masking. The use of the term “KV-Cache” may therefore be misleading.\n\n\n2. Method is somewhat incremental: Similar ideas (e.g., exploiting representation stability across steps) have already appeared in the diffusion literature, particularly in text-to-image generation. The contribution here mainly lies in applying them to DLMs.\n\n\n3. Missing comparison to AR models: The paper does not show how the proposed caching mechanism compares against autoregressive models in terms of both throughput and generation quality, which is crucial for justifying practical relevance.\n\n\n4. No architectural improvements for lossless caching: The work focuses solely on inference-time heuristics. A more ambitious direction—modifying DLM architecture to enable theoretically lossless caching—would have been more compelling and impactful.", "was_revised": true, "final_justification": "Thank you for the authors’ detailed and thoughtful responses, which addressed several of my earlier points. However, my primary concern remains, and I am still not fully convinced about the level of novelty. Accordingly, I will keep my score unchanged.", "rebuttal": "We are thankful for the reviewer’s insightful comments and suggestions, and we will incorporate the following analysis and provide further clarification in the revised manuscript.\n\n------\n**W1: Lack of theoretical justification**\n> *W1: The method is based purely on empirical observations rather than formal theoretical guarantees. In particular, the reuse of key/value states lacks a rigorous foundation, unlike AR models where caching is mathematically supported by causal masking. The use of the term “KV-Cache” may therefore be misleading.*\n\n\nThank you for the insightful question. While diffusion language models do not naturally admit autoregressive causal masking, we provide a theoretical justification for the validity of reusing stale key/value states from earlier steps\n\nOur intuition is to treat the cached key/value states as propagation states across decoding steps, similar to the hidden states in RNNs or in linearized attention variants. We aim to derive the bound if we reuse the key/value states from previous steps, e.g., using $k_i^{t-k}$ and $v_i^{t-k}$ at step $t$ instead of freshly computed ones $k_i^{t}$ and $v_i^{t}$. $L$ below means the sequence length.\n\nConsider the standard attention output at decoding step $t$. Following prior work on kernelized attention, we approximate the softmax with a positive kernel $\\phi(\\cdot)$:\n\n$$\no_t=\\frac{\\phi\\left(q_t\\right) \\sum_{i=1}^L \\phi\\left(k_i^t\\right)^{\\top} v_i^t}{\\phi\\left(q_t\\right) \\sum_{i=1}^L \\phi\\left(k_i^t\\right)^{\\top}}\n$$\nAnd we have:\n\n$$\nZ_t = \\sum_{i=1}^L \\phi(k_i^t), P_t = \\sum_{i=1}^L \\phi(k_i)^\\top v_i^t, o_t = \\frac{\\phi(q_t)P_t}{\\phi(q_t)Z_t}\n$$\n\nHere for dKV-Cache:\n* Some $k_i^t$ and $v_i^t$ are fresh, while others are replaced with cached $\\tilde{k}_i^t = {k}_i^{t-k}$ and $\\tilde{v}_i^t = {v}_i^{t-k}$.\n* We denote $\\tilde{P}_t$ as $P_t$ cached counterpart; similarly $\\tilde{Z}_t$ and $Z_t$.\n\nLet’s define the output error for attention as: $\\delta_t=\\left\\|| o_t- \\tilde{o}_t\\right\\|| $, we would get:\n\n$$\n\\delta_t = \\left\\|| o_t- \\tilde{o}_t\\right\\|| =  \\left\\|\\left\\|  \\frac{\\phi(q_t)P_t}{\\phi(q_t)Z_t} - \\frac{\\phi(q_t)\\tilde{P}_t}{\\phi(q_t)\\tilde{Z}_t} \\right \\|\\right \\|\n\\leq \\frac{\\left\\||\\phi\\left(q_t\\right)\\right\\|| \\cdot || P_t-\\tilde{P}_t || }{\\left|\\phi\\left(q_t\\right) Z_t\\right|} + \\left\\||\\phi\\left(q_t\\right)\\right\\|| \\cdot || \\tilde{P}_t || \\cdot\\left|\\frac{1}{\\phi\\left(q_t\\right) Z_t}-\\frac{1}{\\phi\\left(q_t\\right) \\tilde{Z}_t}\\right|\n$$\n\nFirst, for \n\n$$ \n|| P_t-\\tilde{P_t} || =\\left\\|\\left|\\sum_{i=1}^L\\left(\\phi\\left(k_i^t\\right)^{\\top} v_i^t-\\phi\\left( \\tilde{k}_i^t\\right)^{\\top} \\tilde{v}{ }_i^t\\right)\\right\\|\\right|\n$$ \n\nLet’s denote $\\mathcal{M}_t \\subseteq \\\\{1,\\dots,t\\\\}$ as the set of indices for which cached states are used, i.e., $\\tilde{k}_i^t = k_i^{t-k}$ and $\\tilde{v}_i^t = v_i^{t-k}$. For fresh tokens ($i \\notin \\mathcal{M}_t$), $\\tilde{k}_i = k_i^t$ and $\\tilde{v}_i = v_i^t$, so the difference is zero. The sum becomes:\n\n$$ \n|| P_t-\\tilde{P_t} || =\\left\\|\\left\\| \\sum_{i \\in \\mathcal{M}_t}^L\\left(\\phi\\left(k_i^t\\right)^{\\top} v_i^t-\\phi\\left( \\tilde{k}_i^t\\right)^{\\top} \\tilde{v}{ }_i^t\\right)\\right\\|\\right\\|   \\leq \\left\\| \\mathcal{M}_t \\right\\| \\cdot (C_k\\epsilon_v + C_v\\epsilon_k)\n$$ \n\nwith $||\\phi(k_i^t) - \\phi(\\tilde{k}_i)|| \\leq \\epsilon_k$, $||v_i^t - \\tilde{v}_i|| \\leq \\epsilon_v$, $||\\phi(k_i^t)|| \\leq C_k$, $||\\tilde{v}_i|| \\leq C_v$.\n\nFor the denominator drift part, we aim to bound:\n\n$$\n\\left|\\frac{1}{\\phi\\left(q_t\\right) Z_t}-\\frac{1}{\\phi\\left(q_t\\right) \\tilde{Z}_t}\\right|=\\left|\\frac{\\phi\\left(q_t\\right)\\left(\\tilde{Z}_t-Z_t\\right)}{\\phi\\left(q_t\\right) Z_t \\cdot \\phi\\left(q_t\\right) \\tilde{Z}_t}\\right|\n$$\n\nAssume $\\phi(q_t) Z_t \\geq \\gamma$ and $\\phi(q_t) \\tilde{Z}_t \\geq \\gamma'$, similar to the previous one, we can get\n\n$$\n\\left|\\frac{1}{\\phi\\left(q_t\\right) Z_t}-\\frac{1}{\\phi\\left(q_t\\right) \\tilde{Z}_t}\\right| \\leq \\frac{\\left\\||\\phi\\left(q_t\\right)\\right\\|| \\cdot\\left\\||\\Delta Z_t\\right\\||}{\\gamma \\cdot \\gamma^{\\prime}} \\leq \\frac{\\left\\||\\phi\\left(q_t\\right)\\right\\|| \\cdot\\left\\|\\mathcal{M}_t\\right\\| \\cdot \\epsilon_k}{\\gamma \\cdot \\gamma^{\\prime}}\n$$ \n\nThus, summarizing the above two, we would get $\\delta_t$ is bounded by\n\n$$\n\\delta_t \\leq \\frac{||\\phi(q_t)|| \\cdot\\left|\\mathcal{M}_t\\right|\\left(C_v \\epsilon_k+C_k \\epsilon_v\\right)}{\\gamma} + \\frac{\\left\\||\\phi\\left(q_t\\right)\\right\\||^2 \\cdot ||\\tilde{P}_t|| \\cdot\\left\\|\\mathcal{M}_t\\right\\| \\cdot \\epsilon_k}{\\gamma \\cdot \\gamma^{\\prime}}\n$$\n\n**Interpretation of the bound**:\n* The more stable the cached $k, v$, the smaller $\\epsilon_k, \\epsilon_v$ → smaller error.\n* The fewer tokens we cache ($|\\mathcal{M}_t|$), the smaller the error.\n* If dKV-Cache avoids caching tokens (how to select $\\mathcal{M}_t$) with high dynamics (i.e., large $|\\phi(k_i^t) - \\phi(k_i^{t-k})|$), the smaller the error.\n\n---\n**W2: Method is somewhat incremental**\n> *W2: Similar ideas (e.g., exploiting representation stability across steps) have already appeared in the diffusion literature, particularly in text-to-image generation. The contribution here mainly lies in applying them to DLMs.*\n\nThank you for your question. While our method does use the representation stability across steps, the core contribution lies in the **delayed cache mechanism**, which is specifically designed for discrete diffusion language models. **This delayed caching strategy is unique to the discrete nature of text-based diffusion and is not applicable to standard text-to-image diffusion models**. This comes from the special evolving dynamics for the DLMs, and importantly, this design has a substantial impact on performance. As illustrated in Figure 3, when the cache ratio is fixed at 48\\%, **the model’s accuracy improves dramatically, from 3.33\\% to 74.3\\%**, when using the delayed cache compared to a naïve caching strategy. \n\nBuilding on this, we propose two variants for dKV-Cache: the greedy one, which uses a tolerant window to allow approximate caching, and the decode one, which aligns caching with decoding dynamics for better performance. **Both of these two designs are designed for discrete diffusion language models and cannot be used on image diffusion models.**\n\n---\n**W3: Missing comparison to AR models**\n> *W3: The paper does not show how the proposed caching mechanism compares against autoregressive models in terms of both throughput and generation quality, which is crucial for justifying practical relevance*\n\nThanks for your great question. Despite incorporating dKV-Cache, current Diffusion Language Models still fail to outperform AR models in terms of decoding speed. Below, we present a speed and complexity analysis comparing LLaDA and LLaMA3 under two scenarios: long decoding length (L=1024) and short decoding length (L=32).\n\n* Long Decoding (L=1024, zero-shot, LLaDA-Instruct)\n\n| | Token/s | Accuracy |\n| -- | -- | -- |\n| LLaDA (entropy-based sampling)  |  22.4 | 82.1 | \n| LLaDA w/ dKV-Cache-decode  |  56.0 | 82.5 | \n| LLaDA (random sampling)  |  23.2 | 79.5 | \n| LLaDA w/ dKV-Cache-Greedy  |  91.7 | 77.4 | \n| LLaMA3-8B  | 1463.9  | 73.7 | \n\n* Short Decoding (L=32, zero-shot, LLaDA-Instruct)\n\n|  | Token/s | Accuracy |\n| -- | -- | -- |\n| LLaDA  |  149.7 |  38.6  | \n| LLaDA with dKV-Cache-decode |  230.4 | 38.4 | \n| LLaDA (random sampling) | 151.1 |  34.4  | \n| LLaDA with dKV-Cache-greedy  | 190.1 | 28.2 | \n| LLaMA-8B  | 1524.9 | 1.74 | \n\n* FLOPs analysis\n\nWe provide a FLOPs analysis for attention computation per token during generation:\n1. For DLMs, the FLOPs is $ 4L^2d + 4Ld^2$ per token\n2. For ARs, the FLOPs is $ 4Ld + 4d^2$ per token\n3. With dKV-Cache-decode, the expected FLOPs per token is: $ 2L^2d + 2Ld^2$ \n4. With dKV-Cache-greedy, the FLOPs is $ 4wLd + 4wd^2$, where $w$ denotes the attention window size (typically $w \\leq 4$).\n\nThis indicates that **unaccelerated DLMs require approximately $L\\times$ more FLOPs than ARs for attention computation**, leading to significantly slower inference on long sequences. The proposed **dKV-Cache-decode** method mitigates this overhead by reducing the FLOPs **by half**, while **dKV-Cache-greedy** further improves efficiency by reducing the cost **by a factor of L/w**.\n\n \n**Summarization**: \n\n1. Despite utilizing dKV-Cache to narrow the gap, **LLaDA still lags significantly behind ARs in terms of decoding speed**, although it achieves competitive performance in terms of generation quality. \n\n2. One scenario where **DLMs outperform AR models is under short decoding lengths**. Specifically, when the decoding length is limited to L=32, AR models like LLaMA struggle to produce coherent or complete answers, achieving only a score of 1.74, whereas LLaDA maintains strong performance with a score of 38.7. This demonstrates that DLMs are more effective under constrained generation budgets.\n\n**Solutions**: \n\nTo make DLMs as fast or faster than ARs, several key components are needed:\n\n1. dKV-Cache-Greedy to match ARs' complexity with respect to sequence length $L$.\n2. Parallel decoding to reduce the number of sampling steps.\n3. Semi-autoregressive decoding to bound $L$ and limit computation.\n4. System-level optimizations such as vLLM and FlashAttention for improved runtime efficiency.\n\n---\n**W4: No architectural improvements for lossless caching**\n> *The work focuses solely on inference-time heuristics. A more ambitious direction—modifying DLM architecture to enable theoretically lossless caching—would have been more compelling and impactful*\n\n\nThanks for your suggestion. We fully agree that this problem maybe can be resolved from the model architecture, such as by redesigning the attention mechanism or the causal mask in a way that ensures strict equivalence for attention calculation and then pretrain this model. It would be highly interesting, impactful and also an ambitious idea and of significant theoretical and practical value. We hope to explore it in the future, making the architecture of DLMs to natively support the KV-Cache mechanisms."}, {"review_id": "OfJy8CfjJc", "reviewer": "Reviewer_P5ym", "summary": "This paper addresses the problem of slow inference in Diffusion Language Models (DLMs),  as unlike autoregressive models, DLMs cannot use the standard KV-Cache because their bidirectional attention and non-sequential generation process mean that key and value states are constantly changing for all tokens. The authores propose a new caching method called dKV-Cache to solve this. The core idea is based on their observation that a token's representation becomes stable only after it has been decoded. Therefore, they cache a token's key and value states with a one-step delay, which is critical for maintaining performance. The paper presents two main versions: dKV-Cache-Decode, which offers a large speedup with almost no drop in quality, and dKV-Cache-Greedy, which is even faster but trades a small amount of accuracy for higher speed. Overall, their training-free approach acheives speedup on various benchmarks narrowing the inference speed gap between DLMs and autoregressive models.", "questions": "a very interesting result is that dKV-Cache-Decode can improve performance on long sequences (e.g., HumanEval). The paper suggests this might be due to inefficiencies in how bidirectional attention aggregates context. This finding is quite interesting, what is the authors' thoughts on this? Why does freezing older representations help? Does this imply that the model is \"overthinking\" or that later-stage token refinements are destructively interfering with earlier, stable parts of the sequence?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- The paper is well executed, with a clear motivation and a practical algorithm that fits seamlessly into existing transformer architectures. The experimental protocol is thorough, covering multiple benchmarks (MMLU, GSM8K, HumanEval, MBPP) and two distinct 7B-parameter models (LLaDA, Dream).\n- Core ideas—delayed caching, one-step delay, and the two variants (Decode vs. Greedy)—are explained with simple diagrams and step-by-step equations. Readers can follow the transition from standard KV-Cache to the proposed dKV-Cache without undue effort.\n- Adapting key–value caching to bidirectional, non-sequential diffusion workflows is novel. The insight that token representations stabilize after decoding—and can thus be safely cached—breaks new ground in diffusion model efficiency.\n\nWeaknesses\n\n\n- The \"2-10x speedup\" claim while technically supported by the data, could be nuanced. The 10x (or even 18x) speedup appears in specific long-prefill scenarios, which are important but not universal. In more general cases, the speedup is closer to 2-4x. It would strengthen the paper to more clearly delineate the conditions under which different levels of speedup are achieved, perhaps by discussing the speedup as a function of the prefill-to-generation ratio.\n\n- The chosen baseline (reducing the number of sampling steps) is reasonable as it's a common way to trade quality for speed. However, the paper could be strengthened by briefly discussing how dKV-Cache compares to or complements other DLM acceleration strategies mentioned in the related work, such as semi-autoregressive approaches. This would better position the work within the broader landscape of DLM inference optimization.\n\n- The `concat_reorder` operator is critical for implementing a non-sequential cache efficiently, especially with rotational position embeddings (RoPE). While it's deferred to the appendix, a sentence or two in the main text explaining its function and importance would be beneficial for clarity.", "was_revised": true, "final_justification": "I remain positive for the final evaluation of this submission, as it's done meaningful and sound contributions for a practical important aspect for dLLMs.", "rebuttal": "We are deeply grateful for the reviewer’s insightful comments and thoughtful suggestions, which have helped us improve the quality of our work. We would revise our manuscript accordingly.\n\n---\n**W1: Speedup v.s. prefill-to-generation ratio**\n> *W1. The \"2-10x speedup\" claim while technically supported by the data, could be nuanced. The 10x (or even 18x) speedup appears in specific long-prefill scenarios, which are important but not universal. In more general cases, the speedup is closer to 2-4x. It would strengthen the paper to more clearly delineate the conditions under which different levels of speedup are achieved, perhaps by discussing the speedup as a function of the prefill-to-generation ratio.*\n\n Thanks for your question and your great suggestion. The acceleration ratio is relevant to both the generation length and also the prefill-to-generation ratio (ratio in the below table). We here give more results about the acceleration ratio regarding this two factors:\n\n* decode length = 32\n| ratio | 0.25 | 0.5 | 1 | 2 | 4 | 8 | 16 |32 | 64 |\n| -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | \n| dKV-Cache-Prefill | 1.15$\\times$ | 1.31$\\times$ | 1.68$\\times$ | 2.34$\\times$ | 3.51$\\times$ | 5.48$\\times$ | 8.21$\\times$ | 11.40$\\times$ | 13.99$\\times$ | \n| dKV-Cache-Decode  | 1.41$\\times$ | 1.48$\\times$ | 1.66$\\times$ | 1.83$\\times$ | 2.04$\\times$ | 2.18$\\times$ | 2.30$\\times$ | 2.37$\\times$ | 2.43$\\times$ |\n\n* decode length = 256\n| ratio |  0.03125 | 0.0625 | 0.125 |0.25 | 0.5 | 1 | 2 | 4 | 8 |\n| -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | \n| dKV-Cache-Prefill  | 1.00$\\times$ | 1.03$\\times$ | 1.07$\\times$ | 1.18$\\times$ | 1.39$\\times$ | 1.78$\\times$ | 2.52$\\times$ | 3.92$\\times$ | 6.58$\\times$ |\n| dKV-Cache-Decode   | 1.39$\\times$ | 1.40$\\times$ | 1.44$\\times$ | 1.52$\\times$ | 1.62$\\times$ | 1.78$\\times$ | 1.97$\\times$ | 2.18$\\times$ | 2.39$\\times$ |\n\nAccording to the results, dKV-Cache-Prefill can achieve a significantly higher speedup when the prefill-to-generation ratio is large, due to its ability to reuse cached key/value representations for the long prompt. dKV-Cache-Decode is less influenced by the prefill-to-generation ratio. \n\nIn our experiments, we extensively evaluated the method under diverse configurations, including a wide range of zero-shot in-context learning settings, to systematically vary the prefill-to-generation ratio and generation length. \n\n---\n\n**W2: Relationship with semi-autoregressive approaches**\n> *W2. The chosen baseline (reducing the number of sampling steps) is reasonable as it's a common way to trade quality for speed. However, the paper could be strengthened by briefly discussing how dKV-Cache compares to or complements other DLM acceleration strategies mentioned in the related work, such as semi-autoregressive approaches. This would better position the work within the broader landscape of DLM inference optimization.*\n\nThanks for your valuable suggestion. We identify two key connections between our method and semi-autoregressive approaches:\n\n1. **Complementarity**. Given a pre-trained semi-autoregressive model (e.g., Block Diffusion[1]), our approach can be seamlessly integrated to provide intra-block KV-Cache, while the semi-autoregressive design itself naturally induces inter-block KV-Cache. \n\n2. **Transformation from a non-autoregressive DLMs toward semi-autoregressive DLMs**. This is achieved by applying the proposed dKV-Cache-Greedy strategy and replaces random sampling with a deterministic, left-to-right generation order with a fixed-size window. The left-to-right sampling means that we decoded the n-th token at the n-th step, which has the same decoding order as an auto-regressive model, but with a sliding window sliding from left to right in dKV-Cache-greedy. This makes the model act as a semi-autoregressive one, constraining the decoding and the non-cached tokens inside the window. \n\nWe conduct exploratory experiments for this (LLaDA-instruct, GSM8K, official performance = 78.6%):\n\n| Decoding Length | Original | dKV-Cache-Greedy (left-to-right sampling) |\n| --| --|  --|  \n| 256 | 78.8 (70.0 token/s) | 79.5 (244.4 token/s, 3.5x) |\n| 512 | 81.5 (42.8 token/s) | 83.9 (220.8 token/s, 5.2x) | \n| 1024 | 82.7 (22.4 token/s) | 83.5 (147.4 token/s, 6.6x) | \n\nThe experimental results show that with this special sampling strategy under dKV-Cache-Greedy, the model performance remains largely unaffected while enabling the use of a significantly higher cache ratio. This, in turn, leads to a semi-autoregressive model with improved inference speed without compromising accuracy.\n\nWe would add this paragraph into the related work section and also add another analysis section for dKV-Cache-Greedy in the revised manuscript.\n \n---\n\n **W3: Explanations for concat_reorder**\n> *W3. The concat_reorder operator is critical for implementing a non-sequential cache efficiently, especially with rotational position embeddings (RoPE). While it's deferred to the appendix, a sentence or two in the main text explaining its function and importance would be beneficial for clarity.*\n\nThanks for your suggestion. We would modify and add this short description about the concat_reorder into Line 184-186 to introduce more about this operator:\n\nThe concat_reorder operator first reorders the input sequence by placing cached tokens on the left and newly generated (uncached) tokens on the right before passing it into the model. This reordering affects RoPE, so we also reorder the corresponding rotary position matrix accordingly prior to the model forward pass. This design minimizes token-level reordering overhead by applying it only once per decoding step. During attention computation, each attention module then concatenates the cached and new key/value states, applies the appropriate index reordering, and retrieves the updated cached representations from the combined key/value matrices.\n\n---\n\n **Q1: Why dKV-Cache-Decode can improve performance on long sequences**\n\n> *Q1. A very interesting result is that dKV-Cache-Decode can improve performance on long sequences (e.g., HumanEval). The paper suggests this might be due to inefficiencies in how bidirectional attention aggregates context. This finding is quite interesting, what is the authors' thoughts on this? Why does freezing older representations help? Does this imply that the model is \"overthinking\" or that later-stage token refinements are destructively interfering with earlier, stable parts of the sequence?*\n\nThanks for your insightful question. We also think that this phenomenon is very interesting.\n\nThis observation, as you noted, suggests that updating a token and thereby updating the key and value states can in fact degrade the prediction quality of future tokens. We attribute this to **error accumulation during intermediate generation stages**, where incorrectly generated tokens propagate and negatively impact subsequent decoding. Unlike some autoregressive models with potential for correction (e.g., reflection), current DLMs ( LLaDA and Dream ) **lack any refinement mechanism**, meaning that early mistakes cannot be revised and persist throughout the sequence. The cache mechanism helps mitigate this issue by using the previous key and value states, effectively bypassing the influence of erroneous tokens and leading to more stable decoding.\n\n\n[1] Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models"}]

Author rebuttal:

Meta-review: This paper proposes a caching mechanism for diffusion language models (DLMs). As DLMs have emerged as an interesting LM alternative, inference efficiency becomes a critical issue and this paper is very timely in this regard. All but one reviewer find the proposed approach interesting and results solid. The authors did a great job during the rebuttal which helped further strengthen the work. After going through all the discussions, I believe that the concerns raised by reviewer 2PJU have been addressed by the authors, and it's unfortunate that the reviewer did not further engage in the discussions. I think this work would be very interesting to the DLM community and recommend accept.
Decision: Accept (poster)