SparseD: Sparse Attention for Diffusion Language Models
Authors: Zeqing Wang
Gongfan Fang
Xinyin Ma
Xingyi Yang
Xinchao Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Diffusion Language Models
Sparse Attention
Primary area: generative models

Abstract: While diffusion language models (DLMs) offer a promising alternative to autoregressive models (ARs), existing open-source DLMs suffer from high inference latency. This bottleneck is mainly due to the attention’s quadratic complexity with respect to context length in computing all query–key pairs.
Intuitively, to reduce this complexity, a natural strategy is to restrict attention to sparse patterns that retain only the most relevant connections. 
Such approaches are well-established in ARs, where attention follows fixed and clearly defined sparse patterns.
However, in DLMs, we observe distinct sparsity behaviors: (1) attention patterns vary across heads, (2) attention patterns in each head remain highly similar across denoising steps, and (3) early denoising steps are critical for generation.
These findings render sparse attention methods designed for ARs largely incompatible with DLMs, as they fail to capture head-specific structures and risk degrading generation when applied in early denoising steps.
To address these challenges, we propose **SparseD**, a novel sparse attention method for DLMs. 
Leveraging the observations, SparseD only requires pre-computing head-specific sparse patterns one time, and reuses them across all steps. This prevents recomputing sparse patterns at each denoising step.
Meanwhile, SparseD uses full attention in the early steps, then switches to sparse attention later to maintain generation quality. 
Together, these establish SparseD as a practical and efficient solution for deploying DLMs in long-context applications. Experimental results demonstrate that SparseD  achieves lossless acceleration, delivering up to $1.50\times$ speedup over FlashAttention at a 64k context length with 1,024 denoising steps. Code is available at https://github.com/INV-WZQ/SparseD.

Published as a conference paper at ICLR 2026
SPARSED: SPARSE ATTENTION FOR DIFFUSION LAN- GUAGE MODELS
Zeqing Wang', Gongfan Fang', Xinyin Ma', Xingyi Yang"* Xinchao Wang National University of Singapore, 'The Hong Kong Polytechnic University zeging, wang@u. nus edu, xingyi yang@polyu. hetppolyluuu.........mmm edu. hk. x xinchao@nus egu.sg
ABSTRACT
While diffusion language models (DLMs) offer promising alternative autore- gressive models (ARs), existing open-source DLMs suffer from high inference latency. This bottleneck mainly due the attention's 's quadratic complexity with respect to context length in computing all query-key pairs. Intuitively, to reduce this complexity, a natural strategy is to restrict attention to sparse patterns that retain only the most relevant connections. Such approaches are well-established in ARs, where attention follows fixed and clearly defined sparse patterns. How- ever, in DLMs, we observe distinct sparsity behaviors: (1) attention patterns vary across heads, (2) attention patterns in each head remain highly similar across de- noising steps, and (3) early denoising steps are critical for generation. These find- ings render sparse attention methods designed for ARs largely incompatible with DLMs, as they fail to capture head-specific structures and risk degrading gen- eration when applied in early denoising steps. To address these challenges, we propose SparseD, a novel sparse attention method for DLMs. Leveraging the observations, SparseD only requires pre-computing head-specific sparse patterns one time, and reuses them across all steps. This prevents recomputing sparse patterns at each denoising step. Meanwhile. SparseD uses full attention in the early steps, then switches to sparse attention later to maintain generation qual- ity. Together, these establish Sparsel) as a practical and efficient solution for de- ploying DLMs in long-context applications. Experimental results demonstrate that SparseD achieves lossless acceleration, delivering up to 1.50x speedup over FlashAttention at a 64k context length with 1,024 denoising steps. Code is avail- able at https:s:: tiggtttuuua com/
SparseD
INTRODUCTION
Recently, diffusion language models (DLMs) have achieved significant progress in the area of natu- ral language processing (Nie al. 22025] Ye et al. 2025). Unlike traditional autoregressive models (ARs) TTuvrron et al. 2023; Yang et al. 2025), which generate tokens sequentially from left to right, DLMs generate the entire context in parallel. Leveraging this capability, DLMs achieve strong per- formance in language generation and represent a promising alternative to ARs.
Despite the advantages of parallel decoding, DLMs suffer from high-generation latency Ma et al. 2025 Wu et al.) 2023). This bottleneck arises mainly from the bidirectional attention mecha- nism Vaswani et al. 2017), which is central DLMs. This mechanism computes attention over all query-key token pairs simultaneously, including both prefill (prompt) and all generation tokens. As context length increases, the complexity of this mechanism grows quadratically, leading high latency in generation and limiting the efficiency of DLMs in real-world applications.
To reduce this complexity, sparse attention methods (Xiao et al. 2023 Lai al. 2025) have emerged as an effective solution. These methods lower the cost of standard attention by restricting compu- tations sparse patterns that include only subset of important query-key pairs, i.c., important attention scores. Such approaches have been widely adopted in ARs, as attention in ARs exhibits
"Corresponding Author.

---

Published as a conference paper at ICLR 2026
prominent and fixed sparse patterns (Xiao et al. 2023). Therefore, applying such methods to DLMs first requires verifying whether sparse patterns also exist in their attention mechanisms.
In this paper, investigate attention patterns in DLMs and find that they also exhibit clear sparse patterns, making sparse attention feasible in theory. However, we identify three unique observations in DLMs: (1) attention patterns vary significantly across attention heads, showing head-specific pat- terns, (2) attention patterns within each head remain highly consistent across denoising steps, (3) early diffusion steps are critical for generation, rendering sparse attention unsuitable at this stage. These unique observations make sparse attention methods designed for ARs largely incompatible with DLMs. Widely used fixed patterns in ARs, such as the -window scheme (Jiang et al.) 2023) and sink attention (Xiao et al. 2023), fail to capture head-speciff patterns of DLMs. More- over, applying sparse attention ML in the early steps leads degradation in generation quality.
To tackle these problems, we introduce SparseD, novel sparse attention approach tailored for DLMs. Its core principle to efficiently handle the unique attention patterns of DLMs without degrading generation quality. To achieve this goal, the three empirical observations outlined above serve as the foundation. Specifically, SparseD pre-computes and selects important query-key pairs for each head once to construct head-specific sparse patterns. These sparse patterns are then reused for sparse attention across denoising steps without the need to recompute. To enable hardware- friendly acceleration, we select important pairs block-wise query pairs (Dao 2023) rather than individual pairs. Besides, SparseD applies full attention in the early steps to prevent degradation in generation quality. These designs enable SparseD to capture head-specific dynamics without incurring significant latency in recomputing sparse patterns at every denoising step, while also preventing the generation degradation caused by sparse attention in the early steps.
preventsigniffican significant
To further preserve accuracy, we adopt an isolated selection strategy in computing sparse patterns. Specifically, we observe that attention scores for generation tokens are relatively low during the early steps but gradually increase in later steps. Since SparseD computes sparse patterns in the early steps and reuses them in subsequent steps, this will cause the selection concentrate primarily on prefill tokens with high attention scores. To address this issue, we separately select important scores for prefill and generation tokens, ensuring that both receive sufficient attention in selection.
Together, these establish SparseD as practical and efficient solution for deploying DLMs, particu- larly long-context applications. Experiments on recent DLMs, including Dream-7B-Instrue (Ye) et al., (025) and LLaDA-1.5 (Zhu et al. 2005), demonstrate that Sparsel) greatly preserves the orig- inal accuracy with negligible loss while achieving up to 1.50x speedup over FlashAttention (Dao) 2023) at 64k context length with 1,024 diffusion steps.
In summary, our main contributions are as follows:
We identify three key observations in DLMs: (1) attention patterns vary across heads, (2) attention patterns remain highly consistent across denoising steps, and (3) early diffusion steps are crucial for effective language generation.
We propose an effective and efficient sparse attention method that skips sparse attention during early denoising steps and reuses sparse patterns in the subsequent steps.
Extensive experiments show that SparseD greatly maintains accuracy on the evaluated benchmarks while achieving up 1.50x speedup at 64k context length with 1,024 steps.
RELATED WORKS
Diffusion Language Models (DLMs) Diffusion models (Ho al. 2020 Rombach et 2022) have emerged as a powerful paradigm in generative modeling, framing data generation as the in- version of a forward-noise process. They have achieved remarkable success in continuous domains such as images (Peebles & Xie 2023) and videos (Kong et al., (024)). More recently, diffusion mod- els have also advanced the natural language processing area. DLMs (Ye et al. 2025 Nie et al. 2025) 2025
extend diffusion to discrete sequences by redefining noise injection and denoising Ou et al.
Zheng et al. 2023). Unlike conventional autoregressive models (ARs) (Touvron et al. 2023 Yang let al. 2025) that generate tokens sequentially. DLMs denoise all tokens jointly in bidirectional manner. This parallel, bidirectional generation enables DLMs to achieve strong performance in both language understanding and generation, establishing them promising alternative to ARs.
2

---

Published as a conference paper at ICLR 2026
Sparse Attention Despite their success, DLMs suffer from high inference latency, which remains a major bottleneck. This issue is primarily due to the quadratic complexity of the core attention mechanism (Vaswani et al. 2017). This challenge has been extensively studied in traditional ARs. To address it, sparse attention has emerged as a promising and mature solution. In ARs, many meth- ods restrict attention computation to fixed patterns, such as sink attention (Xiao et al. 2023) and
the sliding-window approach (Jiang et al. (202)). Some approaches, such as those in BigBird (Za-
hherr et al. 2020) and Longformer (Beltagy al. 2020), utilize global and sliding-window attention schemes. Other approaches (Zhang et al. 2025 Lai et al. 2025) identify distinct fixed patterns in
ARs and dynamically select them for each head by computing approximate attention scores during
inference. However, these methods still rely on patterns from ARs, and sparse attention in DLMs remains largely unexplored.
Efficient DLMs Prior works accelerating DLMs inference primarily focuses on sache-based
approaches (Ma et al. 2025; Wu et al. 2025). For example, dKV-Cache (Ma et al. 2025) exploits the stability of activations in decoded tokens by caching their key-value states to reduce redundant computation. Fast-dLLM (Wu et al.] 2025) further introduces a block-wise caching scheme that caches both prefix and suffix tokens for improved efficiency. While these methods achieve substan- tial latency reduction, they suffer from noticeable accuracy degradation, especially in long-context scenarios. In this paper, instead of relying on cache-based techniques, we propose a new sparse attention method that reduces inference latency with lossless accuracy.
3 METHOD
3.1 PRELIMINARY
Diffusion language models (DLMs) generate text via an iterative unmasking process over discrete denoising steps, gradually transforming a masked sequence into the final output. Formally, let V denote the vocabulary, and let æ', E yv denote the sequence state of length at step 6, where (SK))) where
1 O..., T. (e...., Cp) represents the prompt (prefill tokens), and the remaining / - positions are occupied by mask tokens be generated (generation tokens). Through iterative denoising of both prefill and all generation tokens, DLMs achieve strong performance on language understanding and generation.
The initial state is defined as a! (ci,-...: Cp. [M ASK]......
However, denoising all tokens across all diffusion steps incurs substantial computational overhead due to the quadratic complexity of the attention mechanism with respect to sequence length 1. This challenge becomes even more severe in the long-contex. setting. To tackle this challenge, sparse attention becomes promising solution. The sparse attention mechanism reduces redundant compu- tation by focusing only on the most important query-key pairs, i.e., important attention scores. The attention score A R. is computed as the scaled dot product between the query matrix Q E Rixd and the key matrix K E gixd, normalized by the square root of the head dimension d. Formally, the attention score is defined as:
= A(Q, K) Softmar
Vd QQ (Q-KK))
(1)
Each A., can be viewed as the dot product between a query-key pair Q and K To improve computational efficiency, sparse attention restricts computation to a subset of query-key pairs. A simple strategy is to retain only the top-p% pairs with the highest scores Ai, for each query i as sparse patterns. The overall index set for selected query-key pairs defined as
S
U
Top s (1,5)) E Z, ranked by Aas
(2)
EE2
Then, the sparse attention mechanism is defined as:
A = A(Q, K, Ms) = Softmaz( 1 (Q (Q.K" + Ms)).
(3)
Here, Ms is sparse attention pattern based on S, defined as:
Msli.j] 0,
if (f. E8
-c, otherwise.
3

---

Published as a conference paper at ICLR 2026
Layer arcae
(a) step1
(b) step15
(c) step30
(d) similarity
Figure 1: Attention score across denoising steps using L
32, block lock-lenghh =
32). Rows correspond to different attention heads. Red lines divide key tokens in prefill and gen- eration tokens. The result shows pronounced similarity across denoising steps. More visualized aftention patterns from different DLMs provided in the Appendix A.I
minimize the discrepancy between A(Q. K Ms) V and A(Q, K) V, where V R'xd is the value matrix in the attention mechanism. Existing approaches in ARs achieve this goal either by using fixed sparse patterns (Xiao et al. (023) or by dynamically selecting suitable patterns for each attention head (Lai et al. 2025 Zhang al. 2025 Xiao et al. 2025). The commonality among these methods that they all rely on attention patterns observed ARs. Although the attention mechanism in DLMs also exhibits clear sparse patterns (Figure I strategies developed for ARs are not well suited to DLMs. This incompatibility primarily arises
The goal sparse attention
from the distinct attention patterns observed in DLMs, discussed in Section 3.2 To address this
issue, we build on these unique observations and propose our method in Section 3.3
3.2 OBSERVATIONS
To enable sparse attention to accelerate DLMs' inference while preserving accuracy, we conduct a systematic analysis of attention patterns, which reveals three fundamental properties:
Head-Specific Attention Patterns In the attention mechanism of DLMs, attention scores vary heads, shown in Fi-ure For example, the second row exhibits column-wise pat- tern, while the third row shows sliding-window pattern. In the first row, the upper part follows sliding-window window structure, whereas the lower part displays column-wise pattern. Such inconsis- tencies render widely used sparse attention methods in ARs, such as sliding- wiidiowwwindow (Jiang et al., 2023) and sink attention (Xiao et al. 2023), unsuitable for DL.M.
Attention Similarity Across Time Although attention scores differ across heads, each of them re- main highly consistent across denoising steps. shown Figure (d), the attention scores in each head exhibit high similarity across steps. Since sparse attention patterns are directly derived from attention scores, this consistency suggests that the sparse attention patterns for each head are also largely stable across steps, motivating the sparse reusing strategy in SparseD, detailed in Section 3.3 More analysis of attention similarity is provided Appendix A.2
Significant Impact of Early Steps Generation Unlike traditional AR models that generate each token sequentially from scratch, DLMs decode all tokens simultaneously. In this process, all tokens undergo denoising across every denoising step. This raises natural question: do all denois-
4

---

Published as a conference paper at ICLR 2026
50
Sparse
Sparse
Dheeklne
30
50
Sparse
Sparst-Full
Boseline
10
10
Steps
Stegs
(a) steps
(b) 128 steps
Figure 2: Influence of sparse attention across denoising steps. Experiments are conducted on LLaDA-1.5 (1 = 256 and llock.ength = 256) with denoising steps 32 and 128. 'Full-Sparree denotes applying full attention in the first steps and sparse attention in the remaining steps, while "Sparse-=Flll is the opposite. Sparse attention retains only the top 30% of attention scores per query token (Equation 2). Results highlight the importance of early denoising steps.
ing steps contribute equally? In other words, from a sparse attention perspective, which diffusion steps can apply sparse attention with minimal impact on generation quality?
To investigate this, we evaluate loss changes under different sparse attention configurations (Fig- ure 2. As shown by the green and gray dashed lines, applying sparse attention in the early steps results in significant loss increase (left side of the gray dashed line), while extending it to additional steps causes only marginal further degradation (right side of the gray dashed line). This indicates that early steps are particularly sensitive to sparse attention. Conversely, the blue line shows that gradually transitioning from sparse full attention in the early steps substantially reduces loss, fur- ther underscoring the critical role of early denoising steps in DLM text generation. These findings demonstrate that directly applying sparse attention methods from ARs in early steps leads to severe degradation in generation quality.
In summary, the above findings show that widely used sparse patterns ARs fail capture the head- weadd-pecifiic patterns of DLMs, and applying sparse attention to DLMs in the early steps leads to degradation in generation quality. To address these challenges, we propose SparseD for DLMs, based on these unique observations, as discussed in Section 3.3
3.3 SPARSED
At high level, SparseD is able to efficiently handle the unique attention patterns of DLMs with- out degrading generation quality. Specifically, SparseD uses full attention in early steps, and then pre-computes and reuses head-specific sparse patterns for sparse attention in subsequent steps. An overview SparseD is shown Figure 3 and this section details each of its components.
Isolated Selection As discussed in Section 3.2 DLMs exhibit head-specife attention patterns. This makes fixed sparse patterns, e.g., the sliding-window scheme, insufficient to capture important attention scores in DLMS. To address this problem, we compute and select important attention scores for each attention head to form head-specific sparse patterns using Equation Equation[]aadda [nd Moreover, since some heads gradually increase attention score on generation tokens in the key dimension (right side of the red line in Figure[[a-c), selecting dominant attention scores in the early stages may overlook the contribution from generation tokens. To address this issue, we separately select attention scores for prefill and generation tokens in the key dimension, applying the same selection ratio p% to both. Then, the index set for selected indices can be formulated as:
S SE?.
(SP
(5)
where spre and Sgen refer to the index set in prefill and generation tokens, respectively. Consid- ering hardware-frit frienddly acceleration, we select important attention scores a block-wise manner. Specifically, first apply the average pooling A, formally A' avgpool A. block size)
5

---

Published as a conference paper at ICLR 2026
Isolated Selection
PROMPT
Average
Queey
Select
Spars Patter
Score
Skipping Sparse
Xy-k
Sparse Reusing
X
Laver
Layer
Lager
Layer
Laver
Layet
Laver
Lawar
Layer
Layer
Laver
X--k
X7-t-1
Xn
Step
Step
Step k+
Soep T
Figure 3: Overview of SparselD. Sparsel) first applies full attention during the early diffusion steps. then pre-computes attention scores and selects the important scores using a block-wir wise scheme, while performing isolated selection for prefill and generation tokens. The resulting sparse patterns are reused the subsequent steps.
R// ht//bbock.sizexl//bbeeee /lock Then the selecting sets can be formulated
8pre Top [(t,5) S S p, ranked by Si
(6)
geen Top,, { (1. 1p <isı, ranked by Aly
However, computing the full A(Q, K) incurs substantial memory overhead. To address this, we partition Q I into smaller blocks Q' |MMock.sizeed and sequentially compute A' (espool(A(Q) K), blokc_size) for each block, thereby reducing memory usage.
Sparse Reusing As shown in Section 3.2 attention scores within each head show strong similarity over denoising steps. Leveraging this, we can only compute the sparse patterns once and reuse them in subsequent steps. Specifically, we calculate attention scores and select the top-p% important attention score (Equation 5). The resulting selection defines the sparse pattern M, in Equation which is then reused across denoising steps as shown in Equation 3
Skipping Sparse As discussed in Section 3.2 the early denoising steps are critical for language generation DLMs, and applying sparse attention at this stage leads to substantial degradation in quality. To address this issue, apply full attention during the initial skip% of denoising steps, thereby preserving generation performance. Specifically, full attention is applied during the first x skip% steps. At step T' skip%. SparseD computes and selects head-specific sparse patterns, which are then reused for sparse attention throughout the remaining T' (1 skip%) steps. In summary, the overview of SparseD is shown in Figure I and the pseudo code is shown in Algorithm I
Algorithm Sparsel)
1: Input: Q K, V E R Rlxdd current step, T, p%,, skip%, block size
2: if current step T skip% then
Skipping Sparse
Attention Output <- Full Attention( Q, K, V)
4: if currend_step = skip% then do
b. Isolated Selection
5:
for i in range(1/block..siz.
6:
start = i bll.kk.sczzeeetee end (i (i+ 1) block_size
Q' Qatartiend:
A' < avgpool(A(( Q'' K), block_size)
9:
5 + IndexSelection A', p%)
b Equation and & Equation
10:
Ms [start: end, t
11: end for
end if
13: else
14:
Attention.Output + A(Q,K,
P Sparse Reusing
15: end if
16: return Attention Output

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTING
Models: We evaluate all comparing methods recent DLMs, including both LLaDA-1.5 ZZuu et al., 2025) and Dream-7B-Instruce (Ye et al.] 2025) models. Baselines: We compare Sparsel) against the original models, the widely used sparse attention methods from ARs (Slide Window and StreamingLLM (Xiao et al. ), and efficient DLM methods (dKV-Cache (Ma et al.
2025 and
Fast-dLLM (Wu et al.J (025 ). Datasets: Experiments are conducted on a diverse set bench-
marks, including general language understanding (MMLU (Hendrycks et al. (202]), mathemati- cal reasoning (GSM8K (Cobbe et al. 202T)), code generation (HumanEval (Chen al. (02T )
and long-context evaluation (RULER (Hsieh et al. 2024)). We use 5-shot for MMI.U, 4-shot for GSM8K, and 0-shot for the other datasets.
Implementation Details All experiments were conducted on NVIDIA A800 (80 GB) GPUs. The original DLMs were accelerated with FlashAutention (Dao) 2023). For the sliding-window method, set the window size (ws) to 256 for short-contex. evaluations (MMLU, GSM8K, HumanEval) and to 2048 or 4096 for RULER with 4k and 8k contexts, respectively. For StreamingLLMM, we set the same dKV-Cache, we set the cache refresh interval to 2 for the LLaDA-1.5 and to for the Dream-7B- Instruct. For Fast-dLLM, we set the threshold of 0.9, and block size 8 for MMLU and 32 for other datasets. For SparseD, we set block_size = 32 and block_size 128 with p 30% for RULER. In all SparseD settings, use skip 20%. During the early steps employing full attention, we use FlashAttention switch FlexAttention (Dong et al.) 2024), which supports customized sparse patterns. We evaluate accuracy across all datasets and measure latency by processing individual input samples of varying lengths from RULER. Details for datasets, models, and methods are provided in the Appendix A.3
with the sliding-window method and initial 10% key tokens sink tokens. For
50% for short-context tasks, and
the accelerator, and afterward
4.2
MAIN RESULTS
Table 1: Comprehensive benchmark results on LLaDA-1.5 and Dream- -eeee-B--nnttteee Instruct.
MMLU
GSM8k
HE
RULER-4k
RULER-8K
Avg.
Dream-7B-Instruc"
66.42
80.74
53,05
90.13
71.79
72.42
dK V-Cchee
66.32
80.67
54.88
81.41
55.08
67.67
Fast-dLLLM
65.51
78.17
48.78
81.68
55.64
65.95
+Slide Window
63.45
70.20
34.76
41.46
34,36
48.84
StreamingLLM
64.19
72.86
33.54
43.94
36.36
50.17
SparseD
66.34
80.29
53.05
89.76
72.47
72.38
LLaDA-1.5
64.24
80.38
40.85
90.45
60.73
67.33
dK V-Cahhe
63.45
79.98
40.85
88.18
$7.11
65.91
Fast-dLLM
63.17
82.64
40.24
86.64
47.76
64.09
Slide Window
63.72
57.77
27.44
39.20
36,32
44.89
StreamingLLM
63.52
52.01
37.20
40.39
36.62
45.94
SparseD
64.14
79.80
40.85
90.89
62.44
67.62
This section presents a comparative evaluation of SparseD from accuracy and latency perspectives.
Accuracy As shown in Table I SparseD achieves lossless performance compared with the orig- inal models. On average, it incurs only 0.04% accuracy drop on Dream-78 (aan-BBaaa Instruct and even yields 0. 0.29% improvement on LLaDA-115. In contrast, compared with sparse attention methods in ARs, the sliding-windon method and Streamingl_LM, struggle to handle head-specific attention pat- terns in DLMs, whereas SparseD delivers great performance in maintaining original capacity. The underperformance of StreamingLLM indicates the incompatibility of attention sinks in ARs with DLMs, a conclusion that aligns with the finding in Rulli et al. (2025). Compared with efficient DLM approaches such dK VaChhe and FLMM SparseD shows clear advantages in long-come cong-context scenarios. Although cache-basee methods perform well on chort-context tasks, they experience sig- nificant accuracy degradation with long contexts. Compared with Sparsel on RULER-8K, both

---

Published as a conference paper at ICLR 2026
dKV-Cache and Fast-dLLM show approximately 16% accuracy reduction Dream- (eeam-7B-aa 7 Additionally, they exhibit 5.3% and 14.6% accuracy reductions on LLaDA-1.5, respectively, high- lighting their limitations. Detailed accuracy comparisons on the long-context RULER dataset provided in T of Appendix A.4] In addition to Dream-7B and L.aDA-1.5, we also evaluate the effectiveness of Sparsel) on the multimodal DLM, LLaDA-V You et al. 2025). for more comprehensive assessment in Appendix A.6
Labency Comparison Dream 7B-Instruct
Latendy Compartonn 1LaDM-1.5
1930
Spone
Sparn
1500
1000
730
1000
520
230
31k
Figure 4: Latency comparison (T=1288 for Dream-7B-Instruct and LLaDA-1.5, evaluated on a sin- gle sample from the RULER dataset with varying sequence lengths.
Latency To evaluate the inference latency of SparseD, we compare it with the sparse attention method (StreamingL.LMM) and FlashAttention across different context lengths with 128 steps. For StreamingLLMM the window size is set
1.- and sink token length is set to sink 10% l. As shown in Figure 4 SparseD matches FlashAttention at 4k and 8k length, and demonstrates clear advantages beyond 16k length. In particular, 64k, SparseD achieves 1.23x and 1.25 speedups over FltshAttention on Dream-7B: IB-Isrruct and LLaDA- model, respectively. Although SparseD achieves similar acceleration compared with StreamingLLMM, our method maintain accuracy with lossless loss while Streaming LLM lead to great degradation in generation.
Beyond evaluation 128 diffusion steps, further assess Sparsel) under varying numbers of dif- fusion steps. As shown in Figure 5 the results demonstrate gradual increase in acceleration com- pared FlashAttention. tention on Dream- Fream-7B-iie 7B-Instruct and LLaDA-1.5. respectively. At 1024 steps, the speedups increase 1.50x and 1.48x, x, respectively. This efficiency gain arises because sparse attention patterns are pre-computed only once and reused across all denoising steps. Consequently, in scenarios with long contexts and many diffusion steps, SparseD effectively amortizes the computational cost.
128 steps, SparseD achieves 1.23x and 1.25x speedups over FlashAt-
Dream-788 Dream-7B-Instan BIIntructt
LADA-1155
FlashAttention
Flasklitention
200
SperaeD
250
-
SparseD
Time (min)
150
Time toino)
150
100
100
50
64
128
256
512
1024
64
128
256
512
1024
Diffusion Steps"? T
Diffusion Steps"? T
Figure 5: Latency comparison of SparseD on Dream-7B-Instrue and LLaDA-1.5 across varying diffusion steps, evaluated on single RULER sample with a 64k context length.
4.3 ABLATIONS
In this section, we conduct extensive ablation studies to evaluate the components of SparseD and their effects under different configurations. We first validate the effectiveness of its core compo- skitpiini sparse, sparse reusing, and isolated selection. Next, quantitatively evaluate the similarity of sparse patterns in SparseD. Finally, we report the resource overhead during the pre- compute stage to demonstrate the efficiency of Sparsel..
8

---

Published as a conference paper at ICLR 2026
Effectiveness of Each Component As shown in 2 we conduct an ablation study to evaluate the effectiveness of each compo- nent in SparseD. Removing skipping sparse attention (third row) causes severe accuracy drop, highlighting its importance in preventing degradation during early steps. Recomputing sparse patterns at every denoising step (fourth row) introduces substantial latency due to repeated computations, whereas reusing sparse patterns (second row) achieves comparable accuracy with far lower latency. Furthermore, excluding isolated selection (last row) de-
Table 2: Ablation study of SparseD on LLaDA-
1.5. Each component is excluded individually to assess its contribution. Accuracy is measured on RULER at 4k length, and latency is evaluated 64k-1ength RULER sample. The relative changes compared to SparseD are highlighted in gray.
LLLDA-1.5
RULER (%)
Latency (8)
FlashAttention SparseD
90.45
2127
90.89
1695
Skkppiig Sparse 87.91 (-3.07%)
1552 (-8.43%)
Sparse Reusing
90.82 (-0.07%)
30020 (+1671%)
creases accuracy, while enabling it (second row) improves accuracy with negligible latency overhead. In summary, these results collectively confirm that all three components are crucial for making SparseD both effective and efficient for DLMs.
Isolated Selection
90.53 (-0.36%)
1687 (-0.47%)
Jeccard Similarity Across Heads (LLaD4-1.5)
Quantitative Analysis of Attention Similarity In Section 3.2 we observe that attention patterns in DLMS exhibit strong temporal similarity. Sparsel) leverages this property by reusing the sparse atten- tion patterns in the subsequent steps of T x skip%. To further validate the effectiveness of this strategy, we evaluate the similarity of the computed sparse at- tention patterns Ms during the reuse phase, i.e., af- ter step T x skip%
1.00
Jaccard Siillarity
0.90
0.85
0.50
0.75
0.70
0.65
As shown in Figure 66 we compute the Jaccard sim- ilarity of the selected top-p blocks between the pat- tern obtained at step T x skip% and the "ground- truth" top-p blocks at each subsequent step. The experimental setting is:
Generation Step
Figure 6: Jaccard similarity across attention heads on LLaDA-1.5. The light blue curves correspond to ten randomly selected atten- tion heads, while the orange curve repre- sents the average similarity across all atten- tion heads.
32, skip 0.2, block_size = 32, and p = 0.3. The results show that the selected blocks across all attention heads achieve an average similarity above 90%, demonstrating the high consistency of the selected blocks throughout the reuse steps.
the pre-compute overhead, including the storage and la- tency costs of computing the sparse attention pattern Ms, as well as the memory savings enabled by block-wise selection.
Pre-compute Overhead
Table 3: Pre-compute overhead. 'OOM' represents out of memory.
LL (T=128)
4k
8k
16k
32k
64k
Original Memory (MB)
18709
21977
29197
44989
75571
Mg Storage (MB)
0.48
2.84
13.37
59.10
246.11
w/o Block-wise selection (MB)
21445
39065
OOM
OOM
OOM
w/ Block-wise selection (MB)
18713
22031
29219
44997
75583
Original Latency (s)
38.91
94.65
245.33
690.79
2127.72
SparseD Latency (s)
41.32
95.81
230.11
596.70
1695,49
Precompute Time (s)
3.18
6.48
17.16
60.10
233.30
The storage cost of Ms arises from storing each pattern a boolean mask of size {0, 1) st/slock.iisxxx//oookk.. For Numbead attention heads across all layers, the total storage amounts to Numrread 12 /loock size? boolean entries. As shown in Table 3 the overall storage

---

Published as a conference paper at ICLR 2026
of Mg remains modest. Even 246 MB for 64k-length setting, it is still negligible compared to the runtime memory (75,583 MB).
As shown in Table pre-computation accounts for roughly 6.3%-13.7% of the total time in SparseD. Since this stage is executed only once, its relative cost diminishes as the number of de- noising steps T increases, effectively amortizing the pre-computation overhead.
Block-wise selection effectively reduces the memory cost during the pre-computation of Ms. Com- puting attention scores requires full attention, incurring O((2) memory cost per attention map, which prevents leveraging the memory-efficient implementation of FlashAttention. To mitigate this, adopt block-wise selection, loading only O(1 x block_size) attention scores into memory time, thereby significantly lowering the memory footprint. As shown in Table without bioe selection, it can run out of memory for sequences longer than 16k. In contrast, with block-wiss selection, SparseD maintains memory cost comparable to the original FlashAttention implemen- tation.
4.4 HYPE-PARAMETER ANALYSIS
In this section, analyze the two key hyperparameters of SparseD: the skipping ratio (skip) and the selection ratio (p).
26
28.)
13313
1700
73912
Divan
Lekrog
Latraa
Ricaning Barke
Bocering Bute
Selering Sato
(a) Analysis of skip%
(b) Analysis of p% on Dream.
(c) Analyssss of p% on LLaDA.
Figure 7:
analysis in Dream-7B-Instrrn and LLaDA-1.5 with RULER-4k dataset.
Skipping Ratio As shown in Figure 7a for both LLaDA-1.5 and Dream-Instruct, increasing the skipping ratio in SparseD improves accuracy. LLaDA-1.5 shows a stronger gain, reaching above 90.89% accuracy at skip 20%. Dream-Instrue steadily increases and plateaus 90.00% accu- racy once the skipping ratio exceeds skip = 30%. This suggests a moderate skipping ratio (20-30%) achieves the best balance. This result further verifies the observation in Figure 2 For an optimal balance between accuracy and efficiency, we set skip 20% in all experiments for both models.
Selecting Ratio
For the Dream-7B- Crestrct- model, as shown Figure 7b accuracy rises sharply from around 40% at p 5% to nearly 89.76% at p 30% after which it saturates. In contrast, similar trend observed for LLaDA-1.5 in Figure 77
latency increases steadily with higher with accuracy increasing until p 20% and then saturating. To achieve balanced trade-off between accuracy and efficiency, we set p = 30% for long-context experiments.
5 CONCLUSIONS
In this paper, we propose a novel sparse attention method, Sparse for DLMs. The design of SparseD based on three key observations in DLMs: (1) attention patterns vary across attention heads, showing head-specific patterns, (2) attention patterns remain highly consistent across denois- ing steps, and (3) early diffusion steps are crucial for effective language generation. Leveraging these insights. Sparsel) pre-computes sparse attention patterns for each head once and reuses them across diffusion steps. Additionally, SparseD applies full attention the early steps and skips sparse atten- tion to preserve generation quality. These designs enable SparseD efficiently handle head-specific patterns while avoiding degradation in generation quality, making a practical and effective solution for deploying DLMs in long-com context applications. Extensive experiments demonstrate that SparseD achieves lossless performance on all tested benchmarks while delivering up to 1.50x speedup over FlashA Attention 64k context length with 1,024 diffusion steps.
10

---

Published as a conference paper at ICLR 2026
6 ACKNOWLEDGMENTS
This project is supported by the Ministry of Education, Singapore, under its Academic Research Fund Tier 2 (Award Number: MOE-T2EP20122-0006 and the Hong Kong Polytechnic University under the Presidential Young Scholars Scheme (Project ID: PO058232).
REFERENCES
Beltagy, Matthew Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004005150, 2020.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained code. 2021.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano. Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint 1X.1:22100.11665 2021.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023.
Juechu Dong, Boyuan Feng, Driss Guessous, Yanbo Liang, and Horace He. Flex attention: A programming model for generating optimized attention kernels, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS), 2020..
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv.2404.06654 2024.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Ana Lachaux. Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR, 00s/2310.0088// 2023. doi: 1044550/ARXIV231006625
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03600 2024.
Xunhao Lai, Jianqiao Lu, Yao Luo, Yiyuan Ma, and Xun Zhou. Flexprefill: A context-aware sparse attention mechanism for efficient long- cong-sequence inference. In The Thirteenth International Con- ference Learning Representations, 2025.
Xinyin Ma, Rumpeng Yu, Gongfan Fang, and Xinchao Wang. dkv-cache: The cache for diffusion language models, 2025.
11

---

Published as a conference paper at ICLR 2026
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A bench- mark for question answering about charts with visual and logical reasoning. Smaranda Mure- san, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguistics: ACI. 2022, pp. 2263-2279, Dublin, Ireland, May 2022. Association for Computa- tional Linguistics. doi: 10.18653/v1/2022 http:///////222222nnnnnnaaaa...
org erg/2022.co fiw.inggniinn.. www.addiingaalll.111. 171.
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint arXiv:2502.09999, 2025.
Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. In The Thirteenth International Conference Learning Representations, 2025.
William Peebles and Saining Xie. Scalable diffusion models with transformers In Proceedings of the JEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Maximo Eduardo Rulli, Simone Petruzzi, Edoardo Michiclon, Fabrizio Silvestri, Simone Scar- dapane, and Alessio Devoto. Attention sinks in diffusion language models, 2025. URL
https://arxii: org/ abs/ 15731
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, f. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2017.
Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. Fast-dllm: Training- accelcration of diffusion 1lm by enabling kv cache and parallel decoding, 2025.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv, 2023.
Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, junxian guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. Duoattention: Efficient long-context LLM inference with retrieval and streaming heads. The Thirteenth International Conference on Leurning Representations, 2025. URL https: /ttp:///eeeerrvieeeeeme net/ wwrummid=cuuuuiu
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue. Mingze Li, Pei Zhang. Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin. Xingzhang Ren, Xinyu Wang, Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. arXiv preprint arXiv:2505. 258.2850500988. 2025.
Jiacheng Yc, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong, Dream 7b: Diffusion large language models. arXiv preprint arXiv: 2508. 2025.
Zebin You, Shen Nie, Xiaolu Zhang, Jun Hu, Jun Zhou, Zhiwu Lu. Ji-Rong Wen, and Chongxuan Li. Llada-v: Large language diffusion models with visual instruction tuning. arXiv preprint arXiv:2505.1692: 16933, 2025.
12

---

Published as a conference paper at ICLR 2026
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 9556-9567, June 2024.
Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang. Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang. Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. MMMU-pro: A more robust multi-discipline multimodal understanding benchmark. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilchvar (eds.). Proceedings of the 63rd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15134-15188 Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979- 8-89176-251-00 doi: 10. 116533v1120255 aln5so/o/222 2025.ac1-long 7367
RRLLatttss://clatthooooooo org/
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Trans- formers for longer sequences. In Advances in Neural Information Processing Systems, volume 33, pp. 17283-17297 Curran Associates, Inc., 2020.
Yu Zhang, Dong Guo, Fang Wu, Guoliang Zhu, Dian Ding, and Yiming Zhang. Anchorattention: Difference-aware sparse attention with stripe granularity, arXiv, 2025.
Lin Zheng, Jianho Yuan, Lei Yu, and Lingpeng Kong. A reparameterized discrete diffusion model for text generation. arXiv preprint 2rxivv2302.05737, 2023. 2023.
Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chunwei Wu, Jun Hu, Jun Zhou, Jianfei Chen, Yankai Lin, Ji-Rong Wen, et al. Llada 1.5: Variance-reduced preference optimization for large language diffusion models. arXiv preprint 2..Xiv:2505.122... 2025.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 ATTENTION PATTERNS
In this section, we visualize a broader range of attention patterss DLMs further support the observations discussed in Section 3.2 Specifically, the attention patterns of LLaDA-SB-Base and respectively.
Dream-7B-Instruee are shown in Figures g and |
illustrated in Figures X a-c)
and 9 a-c), both models display distinct head-specific attention patterns. Moreover, Figures 8 d) and 9 d) show strong consistency across denoising steps. These findings align well with the obser-
vations in Section 3.2
Admittedly, certain corner cases exist. For example, the last rows of Figures (id) and d) reveal instances with reduced similarity across steps. In Figure d), although the early steps deviate, the later steps still maintain strong similarity. In Figure d), the similarity appears in a blsck-wise manner; nevertheless, it still remains above 60% similarity across steps. Moreover, such cases rare among all attention heads.
141
Cly Peten
60
Inded
Lopn Hea
Liver (iuutee
Scsmiin 20 Si
26m
Iabil
stepl
(b) step15
(c) step30
(d) similarity
Figure Attention score across denoising steps using LLaDA-SB-BAL (1 102, 32, block length 32). Rows correspond to different attention heads. Red lines divide key tokens prefill and generation tokens. The result shows pronounced similarity across denoising steps.
A.2 ANALYSIS OF ATTENTION SIMILARITY
This section provides more detailed analysis of the strong similarity observed in the attention maps DLMs. In Section 3.2 we demonstrated this phenomenon by directly measuring the cosine similarity of the attention scores A across denoising steps for each attention head. Since A is computed as QK (Fquation 1. the high similarity of may originate from the intrinsic temporal consistency of the query (Q) and key (K) representations. Therefore, in this section, further analyze the cross-step similarity of both Q and K.
As shown in Figure 10 scores, queries, and keys across different layers and heads of LLaDA-1.5. Across all three com- ponents attention scores (column a), queries (column b). and keys (column c)-we sistently high temporal similarity. The high similarity between Q and K provides evidence that the similarity in attention scores primarily originates from and K.
visualize the cross-step cosine similarity matrices of the attention
we observe
14

---

Published as a conference paper at ICLR 2026
Lowar mecratt
Liaan steatt
Lowel Hon
Chyce Nadee
Guy dera maler
(a) step1
(b) step15
(c) step30
(d) similarity
Figure 9: Attention score across denoising steps using Pream-7B-Instrus (1 83, T 32). Rows correspond to different attention heads. Red lines divide key tokens in prefill and generation tokens. The result shows pronounced similarity across denoising steps.
Notably, in rows 1 and 2, Q and K exhibit similarity trends that closely mirror those of A, indicating that the stability of attention scores stems from the consistency of the underlying query and key representations. In 3, although the query and key do not follow the same similarity pattern, their product- the attention scores still exhibits high similarity.
A.3 EXPERIMENTAL DETAILS
In this section, we provide the detailed evaluation settings described in Section Specifically, we report the parameters used for different models, methods, and datasets.
Datasets Experiments conducted diverse set of benchmarks, including general lan- guage understanding (MMLU (Hendrycks et al.] (202]).. mathematical reasoning (GSM8K (Cobbe al. 2021)), code generation (HumanEval (Chen et al., 2021)). and long-context evaluation
(RULER (Hsieh et al., 2024)).
The RULER dataset consists of 13 subtasks for comprehensive evaluation, including niah fiihhsingge__, 2. niah_single. Single
mullinil, niah_multikey- 2, multikel, niah multiquery, niah multivalue, ruler-vi, ruler ruler-fwe, ruler-qa qaa--aaadd and ruler paler--a_hooo hotpo.. Depending on the spe- cific subtask, we set different evaluation parameters-pprticulllee T and loto enable efficient eval- uation. The details are shown the Table
Methods We compare Sparsel with the windoww approach, StreamingI.LMM, dKV-Cache, and Fast-dLLM. For SparseD, the slide-window method and StreamingLLM. Details of SparseD, the wlidew approach, and StreamingL are shown in Table Note that StreamingLLMM, sink refers the ratio of initial tokens designated sink tokens. For the slide-window approach and StreamingLLMM we use FlexAttention for acceleration.
For dk dKV-Cache, employ the dKV-Cache-PD variant the Dream- IB-Instruct model, setting the cache refresh interval to 4. For the LLaDA-1.5 model, we adopt the Ce--acch--Greeee variant, with cache refresh interval of 2 and window size of 4. For Fast-dl. L.M. we use the Prefix KV
15

---

Published as a conference paper at ICLR 2026
Cracs ctep mtwrtin Sin Simbbrty
Crce-tteo qurry Simbert
Carseiiig ciyy Eriirrty
60
Coreaton Doee
Coneuten Soee
02
Gonruden Suppo
Gurauler
Comoaiion
Cony 2oy Sitedaa Sendurty
Treaestes Upery Sandurty
Cons- day Seniodyy
Generston Seeps
Geraniton Sect
Cenerition Stare
Liyeri Mouad
Sinflert
Cross-itko Siiibutyy
Copst-tap Siiibrity
Gencubion Stups
(a) Attention score
(b) Query
(c) Key
Figure 10: The similarity analysis of attention score, query, and key. The experiment is conducted on LLaDA-1.5 (setting the same Figure
Cache version, setting the threshold to 0.9, with a block size of 8 for MMLUU and for the other datasets.
A.4 EVALUATION DETAILS ON RULER DATASET
Since we use different configurations for evaluating the subtasks the RULER dataset, in this section, present the detailed accuracy results for each subtask. Specifically, Table provides the detailed breakdown corresponding to Table [] while Table pressent the detailed results corre- sponding Table 2
A.5 DISCUSSION WITH CACHE-BASED METHODS
In this section, we provide a discussion of the relationship between SparseD and cache-based ac- celeration methods. Although SparseD and cache-based approaches belong fundamentally different categories, both aim to improve the efficiency of DLMs. In Section 4.21 we primarily compare their accuracy demonstrate the lossless performance of SparseD.
However, a direct comparison of latency between the two is inherently unfair and conceptually mismatched. Cache-based methods accelerate inference by storing and reusing the key-value (KV) representations generated in previous steps. In contrast, Sparsel concentrates on sparsifying the attention computation, without relying on any form of caching. Consequently. the two approaches operate under fundamentally different computational assumptions and incur distinct workloads.
For a fair comparison, we combine SparseD with Prefill-Cache (Ma et al. 2025), i.c., caching prompts, to compare with other cache-based methods, dKV-Cache and Fast-dLLM. As shown in Table 7 integrating SparseD with caching can further accelerate inference but causes performance drop, indicating degree of incompatibility between the two approaches. In fact, since sparse at- tention and KV-cache methods are fundamentally different and our work mainly focuses on sparse attention, such incompatibility is expected-esppecially especially in the context of DLMs, which remain un-
16

---

Published as a conference paper at ICLR 2026
Table 4: Parameters Evaluation. 'BS" represents the batch size. For the RULER dataset with varying lengths, certain parameters (ws, BS) are configured differently, with specific changes indi- cated parentheses denote the version being used.
nigh single.1 nish singtt_ malhmelllkey_ miah.mihikle..2 ruler.vt
nish single_3 niahlymuinike nish.muliiqurry nish, mulliilll ruler.fwe
MMLU
GSMBK
HE
reke..owe
nuler-qp.squal
ruler qu hoipot
Dream-7I -B
few, shoted
shhted
few. shot-O
few_shot-O)
few srw_shoowl) BS-16(4k)8(SS3)
US=1 T-
USS=1
(S=16(4438(Sk)
BS-16i (k-44k
7 T2256
2'=512
T-32
T-664
2 128
I=256
2w512
1-32
1-64
I=128
lemperatureea..
lemperatureaa..
Iemperaturead.
1emperattue=00.1
Semperature-0.11
top-p-0.9
topup=0.9
top-p-0.9
top-p-D099
top --00
top 11p-p-0.9
Slide Window
15x=256
25e
w==:204B(44
(==2204844k)
MM=2048(443
1004000 (K))
StecmingLL.M
15.2556
wam256
to w=00009011k
w wawntthee Sk))
5118==005 uas
sink=10%ed
sink=1088
simik=1016
simk-= 10%
simk=10%%
SparseD
block_size=32
Block Blockssiae:33
block_size:32
Blockusize=128
Mook_aise=ll
block.rtzeel.2
0.5
0.5
0.5
0.3
0.3
0.3
akip 20%
akip 20%
skip 20%
skip
skip 20%
skip
LLaDA-155
few shoted
lew shoted BS-1
few shot=O T-512
few shot=))
few shos=!)
few shoote)
BS-1
I-8 block. longtii88
I-256
I-512
2-12
[-64
I=128
block longth-32
black leagtt-52
black lnigh-22
blok longth-64
block longthr122
Slide Window
ws=2556
w5==256
we=2566
ws=204B(lk)
w===2048/4k)
2>=200S(44k
(s=1096688k
wS=60006
+ SMenmingLLM
mra2256
kinde200% vore
aink-108
wink=108%
10%
sink 10%
-SparseD
block_size=32 o = 0.5
Block_size=32 p 0.5
Block_size=32
block_rize=125
biccksisee=20
block.sizes1.2
o = 0.5
p 0.3
p 0.3
p 0.3
skip = 20%
skip 20%
skip = 20%
skip 20%
skip 20%
akipp 20%
Table 5: Evaluation details of RULER in main results (Table 1b.
nish
$.1
S.3
$.3
MK.1
MK 2
MK.3
MQ
MV
VT
CWE
FWE
QS
OFF
AVG
4K Length
Treeam-7B-lnsteee
100.00
100.00
79,80
-JKV-Cache
100.00
98.88
$1.40
83.40
90 90.60
44.00
87.10
$2.10
91.12
72.48
75.40
78.75
64.20
$1.41
Fast-df.1M
10000
98.20
78.
83.40
99.60
44.20
58.
68.75
00.00
73.58
74.47
78.62
63.80
31.68
Window
Stresming!
32.00
29.60
31.00
28.30
22.20
27. 00.90
28.45
86.78
85.13
$1.98
SperseT)
100.00
100.00
93.60
97.40
90.60
79.60
97.00
97.15
96.80
83.50
80.07
70.22
63.00
89.76
L.L.DA--15
100000
100.00
100.00
100.00
10000
100.00
95.05
99.80
100.00
56.24
63.30
83.17
77.80
90.45
dK.Y C.CV-CCac
100000
100.00
96.60
100.00
100000
96.20
99.40
94.20
96.60
50.88
56.67
90.47
75.50
33.18
Fast dLLM
100.00
100.00
100.00
95.60
99.95
98.05
$8.16
45.32
51.13
76.80
StrzamingL.LM
25,80
24.40
28.20
30.40
28:20
42.00
27.70
27.50
38.08
38.62
73.87
82.30
32.60
40.39
SparseD
100000
100.00
100.00
100.00
100000
100.00
100.00
98.66
100.00
53.74
68.37
$3.00
77.00
90.89
8K: Lemgthh
Dream- Iistuuct
99.80
90.80
67.20
75.40
71.30
28.40
93.10
92.70
63.08
57.50
79:27
39.45
35.80
71.79
Cache
$9.20
11.20
40.00
48.40
71.00
16.20
62.45
47.90
32.00
49.08
76.53
50.55
55.50
55.08
dL.LMMMMMM
39.00
39.20
67.25
56.68
StreamingL.LM
37.00
29.50
28.60
33.00
20.40
21.20
29.95
28.70
31.48
61.04
72.53
29.70
49.50
36.36
SparseD
99.90
91.40
71.60
76.00
72.00
25,40
93.55
93.75
65.00
57.42
79.53
56.39
56.20
72.47
I.L.DA-LS
63.00
71.40
$8.20
64.40
55.80
41,40
69.00
64.55
63.
67.12
61.60
49.95
59.20
60.73
dKVV-Cachee
63.20
74.40
54 60
64.60
$6.60
27.60
59.15
59.60
54.84
64.92
55.13
48.87
59.00)
57.11
Slide
26.30
79.60
20.20
28.11
71.47
27.27
StreamingL.LM
26.20
29.60
28. 60
31.80
26.00
20.20
30.45
28.15
38.64
56.62
76.90
27.37
$5.60
36.62
SparseD
74.00
75.20
58.40 40
67.20
60.60
41.60
69
63.35
66.60
57.44
68.60
30.68
59.60
62.44
derexplored. future work, we will investigate how to better integrate these methods to achieve lossless acceleration.
17

---

Published as a conference paper at ICLR 2026
Table 6: Evaluation details of RULER-4K in ablation study (Table 2
nish
niler PWE
LLaDA-15
S.1
S.2
S.3
MX.1
MK.
MX.
MQQ
MV
VI
CWE
QS
QH
AVG
FlachAticntion
100000
100.00
100.00
100.90
100.00
100.00
95.05
99.80
100.00
56.24
63.80
83.17
77.80
90:45
SpunsD
100.00
106.00
100.00
100.00
100.00
100.00
100.00
98. 60
100,00
53.74
68.87
83.40
77.00
90.89
Skipping Spurse
100000
Sparse Reusing
100000
100.00
100.00
100.00
100000
100.00
100.00
98.30
100.00
53.25
68.
60
76.80
90.82
bolated Selection
100000
100.00
100.00
100.00
100.00
99.00
99.95
97.65
95.88
53.88
68.53
82.47
76.60
90.53
Table 7: Comparison with cache-based methods. 'OOM' refers of memory.
Accuracy (%)
Latancy (s)
RULER-4K
4k
8k
16k
32k
64k
LLaDA-1.5
90.45
38
94
24.5
690
2127
VKV-Cache
88.18
28
64
1.54
409
OOM
+ Last-dLIIM
9
14
OOM
OOM
+ SparseD + Prefill Cache
77.42
13
19
32
86
OOM
A.6 EVALUATION ON LLADA-V
Table 8: Evaluation on LLaDA-V.
MMMU
MMMU-Pro standard
ChariQA
val
LLaDA-V + SparseD
48.0
35.20
78.32
48.56
34.39
78.12
To extend the evaluation to broader range models (excluding Dream- 7B and LLaDA-1.5), we tested our method on the multimodal DLM, LLaDA-V (You et al., 2025). As shown in Table 8 compared the results between the original version and the Sparsel) version. We evaluated them on
the MMMU (Yue al.) 2024). MMMU-Pro (uue al.) 22255,, and ChartQA (Masry et al., 2022)
benchmarks. For SparseD, we used 0.3 and skip 20%. The setup for all benchmarks strietly follows You et al. (2025).
The results demonstrate that our method achieves lossless performance on three benchmarks. This further confirms the effectiveness of our approach, extending its success beyond text-only DLMs.
A.7 LIMITATIONS
This work presents novel sparse attention method for DLMs that efficiently adapts head-specific patterns and avoids generation degradation in early denoising steps. However, several limitations remain. One primary limitation of this work lies in its focus algorithmic design. Future work could explore further system-level optimizations, both for computing sparse attention patterns and for accelerating head-specific sparse patterns. Another aspect is combining our method, sparse attention- baaeet approach, with cache-based methods. The key to implementation is how to retain the advantages of both the lossless sparse attention method and the fast rache-based method simul- taneously.
A.8 THE USE OF LARGE LANGUAGE MODELS (LLMS)
In our work, Large Language Models (LLMs) are primarily employed to support the writing process, particularly during paper refinement. Specifically, use LLMs improve grammar, correct word usage, and polish overall expression. This ensures that our manuscripts maintain both clarity and fluency, making the presentation of our ideas more precise and professional. In contrast, the research ideas and experimental design of this paper were conceived entirely by the authors without the use of LLMs.
18


Reviews:
[{"review_id": "NGB4JtZDAM", "reviewer": "Reviewer_ibCB", "summary": "This paper proposes **SparseD**, a **training-free, inference-time** sparse attention method for **Diffusion Language Models (DLMs)**. The key observations are: (1) **head-specific** attention patterns, (2) **high step-wise consistency** within a head, and (3) **early steps are sensitive** to sparsification. SparseD thus applies **full attention in early steps**, then **pre-computes head-specific sparse patterns once** using **block-wise top-ρ% selection with isolated selection for prefill vs. generation tokens**, and **reuses** the pattern thereafter. Experiments on Dream-7B-Instruct and LLaDA-1.5 show **up to 1.50× speedup** over FlashAttention at 64k context and 1,024 steps with negligible accuracy loss.", "questions": "1. **Skip ratio sensitivity:** How does accuracy/speed trade-off vary beyond 20–30% and across different total steps (e.g., 256 vs. 2048)? Any adaptive criterion?  \n2. **Dynamic prompts/streaming:** If prefill/generation boundaries change mid-run, can the mask be updated without losing reuse benefits?  \n3. **Precompute overhead:** What is the exact one-time cost for mask construction at 64k and how does it scale with heads/layers? (A timing table would help.)  \n4. **Composability with caches:** Any synergy or redundancy when combining SparseD with dKV-Cache/Fast-dLLM for short vs. long contexts?  \n5. **Very long contexts (128k+):** Does block granularity dominate selection quality; how does isolated selection behave there?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Practical & training-free:** No retraining; one-shot sparse mask + reuse; early-step full attention preserves quality.  \n- **Simple, hardware-friendly design:** Block-wise avg-pool → top-ρ% selection; **isolated selection** for prefill/generation.  \n- **Consistent wins at scale:** Clear latency gains at long contexts; speedups increase with more steps due to reuse.", "weaknesses": "## Major\n- **Theoretical under-pinning is light.** No bounds/guarantees on error from mask reuse or head-specific sparsity; argument is largely empirical.  \n- **Model diversity is limited.** Only two DLMs (Dream-7B, LLaDA-1.5); generality to other DLM families or hybrid architectures is unproven.  \n- **Early-step sensitivity.** Skip ratio is tuned (20–30%) rather than principled; transferability across schedules/steps remains unclear.\n\n## Minor\n- **Amortized precompute cost** is not fully quantified in wall-clock breakdowns, though reuse intuition is sound.  \n- **Context vs. cache baselines.** A deeper head-to-head with cache-based methods across sequence lengths would help.  \n- **AR-sparse comparisons.** Positioning vs. structured AR patterns (e.g., Longformer/BigBird) could be elaborated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L0zJmyRaXi", "reviewer": "Reviewer_C5KP", "summary": "This paper proposes SparseD, a sparse-attention framework specifically designed for diffusion language models (DLMs).  \nThe authors first present three empirical observations:  \n1. Attention patterns are head-specific rather than uniform across heads.  \n2. Within each head, the attention pattern is stable across denoising steps.  \n3. Early denoising steps are critical, and applying sparsity too early degrades quality.  \n\nBased on these insights, SparseD adopts a simple and practical design:  \n- Skip sparsity during the early diffusion steps.  \n- At a representative step, compute full attention once, select block-wise Top-ρ attention regions per head, and store these as reusable sparse masks.  \n- Reuse the same mask across later steps, with isolated selection for prefill and generation tokens to better handle different distributions.  \n\nExperiments on Dream-7B-Instruct and LLaDA-1.5 show that SparseD maintains almost identical accuracy  while achieving significant speedup over FlashAttention at 64 k context and up to 1024 diffusion steps.  \nAblation studies confirm that skipping early steps and reusing fixed patterns are both necessary for preserving model quality.  \n\nOverall, SparseD offers a simple yet effective mechanism to accelerate DLM inference while retaining quality, contributing a valuable empirical and practical perspective to efficient diffusion-based text generation.", "questions": "1. What is the per-layer/head memory footprint of the stored sparse masks under your standard setting?  \n2. Have you tried computing masks at multiple checkpoints (e.g., twice in the diffusion process) and reusing them thereafter?  \n3. If the prompt distribution shifts (e.g., code → narrative), do the pre-computed masks still perform well?  \n4. How do latency and accuracy trade off with smaller or larger block sizes for a fixed ρ?  \n5. Can heads with low cross-step similarity be selectively excluded from mask reuse to improve robustness?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Novel DLM-specific insights: Identifies and validates three key empirical properties (head-specific, step-stable, early-step-sensitive) that distinguish DLM attention from AR models.  \n- Simplicity and practicality: The “skip-early + reuse mask + isolated selection” design is lightweight, easily implementable on top of Flash/FlexAttention, and hardware-friendly.  \n- Strong empirical results: Demonstrates consistent latency improvements with negligible performance drop across multiple DLMs and tasks (MMLU, GSM8K, HumanEval, RULER).  \n- Comprehensive ablations: Validates each component’s necessity and explores sensitivity to skip ratio and sparsity rate ρ.  \n- Clear presentation: Includes concise pseudo-code and diagrams explaining mask generation and reuse; writing is crisp and reproducible.", "weaknesses": "1. Overhead accounting clarity.  The paper amortizes sparse-pattern pre-computation but does not quantify the actual wall-clock fraction of that cost across steps.  \n\n2. Storage footprint not quantified.  The memory cost for storing per-head, per-layer sparse masks (after block-wise Top-ρ selection) is not reported, leaving uncertainty about scalability under 64 k contexts and many layers.  \n\n3. Comparison to dynamic sparsity baselines.  The paper argues autoregressive (AR) sparsity patterns do not transfer to diffusion models, but does not include an adapted dynamic head-wise Top-ρ baseline for DLMs to confirm the advantage of static reuse.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GR4dk3F824", "reviewer": "Reviewer_tBmW", "summary": "This paper introduces a sparse attention mechanism designed to accelerate Diffusion Language Model (DLM) inference, particularly in long-context, high-step settings. While it is the first or one of the first approaches to apply sparse attention for DLMs, it is based on mainly *empirical* observations and shows only incremental improvements.", "questions": "-  I think the choice of the baselines is not clear. This is a sparse attention method but the chosen baselines for efficiency include KVCaching methods for DLMs. However, KVCaching methods mainly aim at storing pre-computed values, while sparse attention aims at reducing computations. I think the paper would benefit from a clearer discussion of these two different aspects. Are they complementary, orthogonal ? How do they relate ? \n-  For the same reason, it is not clear to me if this method is reducing memory footprint other than improving latency ? In line 294 the authors claim that the method reduced memory footprint but not experiments or comparisons are shown in this direction.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Empirical Validation of DLM Dynamics: The analysis highlighting the temporal consistency and the importance of early denoising steps is highly insightful and provides foundation for a DLM-specific acceleration technique. The ablation study confirming the necessity of the \"Skipping Sparse\" component is well-executed and validates the central hypothesis regarding diffusion step sensitivity. \n\n- State-of-the-Art Accuracy: the method achieves an almost lossless performance, competitive with other SOTA methods.", "weaknesses": "- *(General)* The paper is based mainly on very empirical observations concerning the distribution of attention scores. There is no validation of these properties through a quantitative analysis. \n- *(General)* The method per-se is not very novel, as it applies principles that are well known already in ARMs (top-k selection based on attention scores in blocks) to DLMs.\n- *(General)* The observation that attention patterns are head-specific is not a new empirical finding in the literature, and was already noticed in ARMs (cf. \"Duo Attention\", ICLR 2025 for KV Cache compression ARMs). \n- *(Results)* The comparison against StreamingLLM looks ill-posed. Streaming LLM was developed for ARMs and is based on attention sinks in ARMs, but other work shows sinks in DLMs behave very differently (e.g. \"Attention Sinks in Diffusion Language Models\" Arxiv Preprint) so it is not surprising that it underperforms greatly in this scenario.\n- *(Results)* The comparison with other baselines does not look very clear. Sparse methods are strongly dependent on the sparsity threshold, but the main results are displayed with a fixed chosen threshold for each method, making it difficult to understand the effective trade-off between accuracy and sparsity. A proper comparison should include different sparsity levels, which imply different FLOPs and memory footprint tradeoffs.\n- *(Results)* Accuracy and Latency are evaluated against different sets of baselines which makes the results part a bit confusing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HVHQqj4Qnz", "reviewer": "Reviewer_Q4L3", "summary": "This paper addresses the high inference latency of diffusion language models (DLMs), which stems from the quadratic complexity of the attention mechanism applied over all tokens at each denoising step. The authors propose SparseD, a sparse attention method tailored for DLMs. The method is motivated by three key empirical observations: (1) DLM attention patterns are \"head-specific,\" meaning they vary significantly across different attention heads; (2) patterns *within* a given head are \"highly similar\" across all denoising steps; and (3) the early denoising steps are \"critical\" for generation quality, and applying sparsity too early causes performance degradation.\n\nBased on these findings, SparseD's design is threefold:\n1.  **Skipping Sparse:** It applies full (dense) attention for an initial percentage of denoising steps (e.g., first 20%) to preserve generation quality.\n2.  **Isolated Selection:** At the end of this initial phase, it computes the full attention matrix *once* and selects a head-specific, block-wise top-$\\rho\\%$ sparse pattern. This selection is done \"isolating\" prefill and generation tokens to ensure both are adequately represented.\n3.  **Sparse Reusing:** This single, pre-computed head-specific sparse pattern is then cached and reused for all remaining denoising steps, leveraging the observed temporal stability.\n\nExperiments on models like Dream-7B and LLaDA-1.5 show that SparseD achieves \"lossless\" accuracy on various benchmarks (MMLU, RULER) while delivering significant speedups over standard FlashAttention (e.g., up to 1.50x at 64k context with 1024 steps), as the one-time pre-computation cost is amortized.", "questions": "1.  Could the authors provide a more quantitative analysis of \"Attention Similarity Across Time\" (Observation 2)? For example, by plotting the Jaccard similarity of the selected top-$\\rho\\%$ blocks between the pattern computed at step $T \\times \\text{skip}\\%$ and the \"ground truth\" top-$\\rho\\%$ blocks at each subsequent step $t$? This would be much stronger evidence than the current heatmaps.\n2.  How robust is the \"lossless\" claim to the temporal similarity assumption? What is the performance impact (e.g., on RULER) if the sparse pattern from one prompt is (incorrectly) reused for a different, unseen prompt? This would stress-test the model's reliance on the exact pattern.\n3.  Regarding the $O(N^2)$ pre-computation step: How does the wall-clock latency of this *single step* scale with context length (e.g., from 4k to 64k)? At what context length $N$ (assuming $T=1024$) would the authors project this one-time $O(N^2)$ cost to become larger than the cumulative $O(T \\cdot N \\cdot \\rho N)$ cost of the remaining sparse steps?\n4.  The \"Isolated Selection\" heuristic uses the same selection ratio $\\rho\\%$ for both prefill and generation tokens. Is this an optimal choice? Have the authors experimented with different ratios (e.g., $\\rho_{\\text{prefill}}$ and $\\rho_{\\text{gen}}$)? One might hypothesize that static prefill tokens and dynamic generation tokens could benefit from different sparsity levels.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  **Novel Empirical Insights:** The paper's main strength is its clear identification and empirical validation of three specific attention behaviors in DLMs: (1) head-specific patterns, (2) temporal stability of patterns, and (3) high sensitivity of early denoising steps. This analysis is novel and provides a solid foundation for future work on DLM efficiency.\n2.  **Well-Motivated Design:** The SparseD method is a strong example of principled system design. Each of its three components (Skipping Sparse, Isolated Selection, Sparse Reusing) is a direct and logical solution to one of the three identified empirical observations.\n3.  **Strong Experimental Validation:** The evaluation is comprehensive. The authors demonstrate that SparseD achieves its \"lossless\" claim by benchmarking against the original models. Crucially, they also show its superiority to two distinct and relevant classes of baselines: (a) AR-based sparse attention (which fails on accuracy) and (b) other DLM acceleration methods (which degrade on long-context tasks).\n4.  **Good Ablation Study:** The ablation in Table 2 is a model of clarity. It individually removes each component of SparseD and shows the precise, and severe, negative impact: removing \"Skipping Sparse\" destroys accuracy (Observation 3 validated), removing \"Sparse Reusing\" explodes latency (Observation 2 validated), and removing \"Isolated Selection\" hurts accuracy (heuristic validated).\n5.  **Practicality and Scalability:** The method provides a practical speedup (up to 1.5x) that compellingly scales with the number of denoising steps $T$ (Figure 5). This makes it particularly attractive for high-quality generation, which often requires many steps.", "weaknesses": "1.  **Lack of Deeper Analysis:** The paper is purely empirical. It does not offer any hypothesis or theoretical exploration for *why* these attention patterns are temporally stable. The denoising process is defined by $T$ steps of iterative refinement, so the $Q$, $K$, and $V$ representations are constantly changing. It is non-obvious and fascinating why the resulting $QK^T$ patterns should remain static. While a full theoretical proof is not required, some discussion or preliminary analysis (e.g., measuring the cosine similarity of $Q$/$K$ vectors over time) would have strengthened the paper.\n2.  **Scalability of the $O(N^2)$ Pre-computation Step:** The method requires a single, full $O(N^2)$ attention computation at step $T \\times \\text{skip}\\%$ to build the sparse pattern. While the authors use a memory-efficient block-wise implementation, the *computational cost* of this step is still $O(N^2)$. The experiments stop at 64k context. For truly massive contexts (e.g., 256k, 1M), this single pre-computation step could become the new latency bottleneck, potentially dominating the cost of all subsequent sparse steps. The paper does not analyze the scaling of this specific step or discuss this potential limitation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)