DPad: Efficient Diffusion Language Models with Suffix Dropout
Authors: Xinhua Chen
Sitao Huang
Cong Guo
Chiyue Wei
Yintao He
Jianyi Zhang
Hai Helen Li
Yiran Chen
Venue: ICLR 2026 Poster (2026)
Keywords: Diffusion-based Large Language Models
Model Optimization and Efficiency
Token Pruning
Model Explainability
Primary area: foundation or frontier models, including LLMs
TL;DR: DPad is a training-free inference method that optimize diffusion-based large language models (dLLMs) by refining their inherent Scratchpad Mechanism; it dropouts redundant suffix tokens to yield significant speedups while maintaining model accuracy.

Abstract: Diffusion-based Large Language Models (dLLMs) parallelize text generation by framing decoding as a denoising process, but suffer from high computational overhead since they predict all future suffix tokens at each step while retaining only a small fraction.  We propose $\textbf{Diffusion Scratchpad} (\textbf{\textit{DPad}})$, a training-free method that restricts attention to a structured subset of suffix tokens, preserving fidelity while eliminating redundancy.  $\textit{DPad}$ integrates two strategies: (i) a $\textit{sliding window}$, which maintains a fixed-length suffix window, and (ii) $\textit{distance-decay dropout}$, which deterministically removes distant suffix tokens before attention computation.  This concise design is compatible with existing optimizations such as parallel decoding and prefix caching, and lends itself to a lightweight implementation.  Comprehensive evaluations across multiple benchmarks on $\texttt{LLaDA}$ and $\texttt{Dream}$ models demonstrate that $\textit{DPad}$ delivers up to $\mathbf{61.4\times}$ speedup over vanilla dLLMs while maintaining comparable accuracy, highlighting its potential for efficient and scalable long-sequence inference.

Published as a conference paper at ICLR 2026
DPAD: EFFICIENT DIFFUSION LANGUAGE MODELS WITH SUFFIX DROPOUT
Xinhua Chen, Sitao Huang; Cong Guo"t, Chiyue Wei, Yintao He, Jianyi Zhang, Hai "Helen" Li & Yiran Chen Duke University
ABSTRACT
Diffusion-based Large Language Models (dLLMMs) parallelize text generation by framing decoding as a denoising process, but suffer from high computational overhead since they predict all future suffix tokens each step while retaining only a small fraction. We propose Diffusion Scratchpad (DPad), a training- free method that restricts attention to structured subset of suffix tokens, preserving fi- delity while eliminating redundancy. DPad integrates two strategies: (i) sliding window, which maintains fixed-length suffix window, and (ii) distance-decay dropout, which deterministically removes distant suffix tokens before attention computation. This concise design compatible with existing optimizations such as parallel decoding and prefix caching, and lends itself to a lightweight imple- mentation. Comprehensive evaluations across multiple benchmarks on LLaDA and Dream models demonstrate that DPad delivers up to 61.4x speedup over vanilla dLLMs while maintaining comparable accuracy, highlighting its potential for efficient and scalable long-sequence inference. Our open-source implementa- tion is available at ltttps: githbb..tttutt//. com/ http://yyycheeDDa...coo
INTRODUCTION
Large Language Models (LLMs) have become foundational in numerous applications (Vaswani al. 2017: Devlin et al. 2019 Brown et al.
2020
Ouyang et al., 2022), yet their deployment is
often hindered by inference latency. As shown in Fig.
(a), the predominant autoregressive frame-
work generates text one token time (Radford 2018 Radford et al. 2019), imposing a sequential constraint that limits speed and scalability (Gu et al. 2018). This has driven interest toward parallel decoding strategies.
Diffusion-basee Large Language Models (Li et al. 2022 Austin et al. 2021a Lou et al. 2024
Shi et al., 2025 Israel et al. 2025) (dLLMs) offer a promising alternative by eliminating sequential dependencies. Formulating text generation as parallel denoising process, they can predict entire sequences or generate text block-wise (i.e., (ntiiauutoorrrssively)) (Nie et al. Ye al. 2025). as in [ However, this parallelism often incurs high computational cost (Nie et al. (025): at each step, predictions for all future (suffix) tokens are computed, though only a small fraction are retained. Consequently, although dLLMs can generate multiple tokens in parallel, the resulting throughput gains are undermined by a disproportionate increase in computation, posing a key bottleneck to their widespread adoption (Song al. 2025b).
To further understand this inefficiency, we analyze the role of suffix tokens under the block-wise masking mechanism in dLLMS and reveal that suffix tokens act as non-semantic information reser- voir, or scratchpad. While this scratchpad collects contextual signals across Transformer layers to guide the generation of the current block, is highly inefficient. We find that most suffix tokens are redundant and low-entropy (see Appx. A). problem that worsens with distance as their attention scores decay sharply. This redundancy not only creates significant computational overhead but can also degrade generation fidelity.
"Equal contribution. Corresponning author; cong-guo duke.edu

---

Published as a conference paper at ICLR 2026
Prompt
Response
Response
Prompt
Step 0
Block
Block 1
Block 2
Block n
Block Step
Step
Block Step 1
Slep 2
Block Slep 0
Step
Prefic Attend Backward
Block Slep
(a) Auto-Regression LLM
Proffit Attend Backward
(b) Block-wise Diffusion LLM (Semi Autoregression)
Gurrent Block
Suftic: Attend Forward with Al Suffi Tokens
Decoded
Decoding
Masked
Distance-Dece Decay Dropout
Response
DPad
Prompt
Block 0
Block
Block 2
Block
Block
Current: Detailed
Block 0, Step
Sliding
Block 0, Step
Window
Near:
Focused
Far:
Outlined
Block 1, Step
Blook Step
Out of Scratchpad:
(c) Ours: Diffusion LLM with Scratchpad (DPad) DPad: Forward Nearby Few Suftix Tokens
Omiited
Figure Comparison of (a) autoregressive LLMs, (b) block-wise diffusion LLMs, and (c) our DPad. DPad restricts suffix attention via: (i) Sliding Window: fixed-length suffix window; (ii)
Distance-decay Dropout: removes distant suffix tokens without computing attention scores.
Based on the above insights, we propose the Diffusion Seratchpad (DPad), which in the forward direction attends only to a small number of near-suffix tokens, as in Fig. (c). It uses two suffix drop strategies: sliding window and desta dropout. For the sliding window, inspired by prefix VVcachhe optimizations (Beltagy et al. 2020)) Xiao et al. (023), we extend the idea to the suffix. Here, the suffix window has fixed length and moves forward along with the current block, retaining only a limited number of nearby suffix tokens. In contrast vanilla dLLMs, where suffix computation increases with the generation sequence length, our design keeps the cost bounded and significantly reduces suffix-related computation.
For the distance-decay dropout, suffix tokens are removed according to their distance from the current block: the farther they are, the higher the dropout ratio, until all tokens beyond the window are omitted, Unlike conventional attention-score pruning (Wang et al. 2021 Kim et al., 2022 Song al. 2025a), which first computes attention scores and then prunes based on their magnitude, DPad predetermines a distance-decay sparse pattern for suffix tokens prior to model execution and eliminates them at the very beginning of each step. This further suggests that sparsity is inherent property of suffix attention. Additionally, the method is extremely simple to deploy, requiring only a few lines of code implement.
To illustrate the intuition behind DPad, we liken it to real scratchpad used when writing book, as shown in Fig. (c. right). For the current chapter (i.e., block), the writer (dLLM) devotes the most attention, revising it multiple times, akin to denoising within a diffusion block. The upcom- ing chapter receives focused drafts for consistency, while much later chapters contain only brief outlines. Naturally, the "writer" should not, and indeed must not, fill all future chapters (all suffix tokens) with low-entropy, repetitive content merely to satisfy the fixed sequence length constraints of current dLLMs. Such uncontrolled filling distracts the "writer's" attention and wastes storage and computation, making neither sustainable nor scalable.
Finally, emphasize that DPad is training-free inference strategy that enables efficient, com- pact generation, overcoming the quality degradation from fixed sequence lengths conventional dLLMs (Ye al. 2025 Nie al.). It delivers robust acceleration across all settings, achieving up to 10.32x x speedup on short sequences. As DPad compatible with existing optimizations (Wul al.; Liu et al.) Wei et al.), including prefix caching (Wu al.) and dLLM-Cache (Liu et al.), its benefits compound for long-sequence generation. When combined with these methods, DPad achieves up 61.39. x speedup on LLaDA-1 5/GSM8K (1024, 1-shot) while maintaining accu- racy, highlighting its potential to unlock new frontiers in dLLM efficiency.
In summary, our contributions are:
We identify and formalize the Scratchpad mechanism in dLLMs, demonstrating that suffix tokens act as a dynamic, cross-layer reservoir (which we term an Attention Connection) that guides the denoising process.
2

---

Published as a conference paper at ICLR 2026
Through systematic empirical analysis, the first to reveal three key properties of suffix tokens: inherent sparsity, a distance-decay pattern, and position insensitivity, which expose major yet overlooked inefficiency in dLLMs.
We introduce the Diffusioon Lottery Tickets (DLT) hypothesis for dLLM inference, positing that a sparse set of "winning" suffix tokens suffices for high-quality generation, and we frame DPad as triniing-free method to discover such tickets on the fly.
We propose DPad, a training-free method that applies distance-decay dropout to suffix tokens. This orthogonal design eliminates substantial redundancy and compounds with existing optimizations, achieving up to 61.39x speedup while preserving accuracy.
2 PRELIMINARY
2.1 FOUNDATIONAL PRINCIPLES OF DIFFUSION LARGE LANGUAGE MODELS (DLLMS)
For generated sequence x (I1, (Ii...., zz dLLMs leverage a con-autoreeressivv process (Austin et 2021al Shi et al. 2025 Lou et 2024). During training, the model learns to denoise a sample where tokens are masked. This involves two processes: (i) Forward Masking Process: This process systematically replaces a proportion of tokens in clean text sequence Xo with a special [MASK]
token (Nie et al. 2025; Nie et al.), similar to applying noise in a conventional diffusion model (Ho al. 2020). The replacement is governed by masking schedule where t [0. 1] represents the masked level, and individual tokens the clean sequence, z6 are masked independently with a probability of t. (ii) Reverse Unmasking Process:
The model po is trained predict the original tokens given the partially masked sequence x,, thereby learning to approximate the true unmasking probability q(xo)xx) xx) (Austin et al. 2021a Lou et al. (024)). This is achieved by minimizing the
negative log-likelihood over the masked tokens (Shi el al. 2025 Ouyang et al. 2022
I
((0)= = -EE
Σ
log (a)(xx)
(1)
=:af== = [MASK]
2.2 INFERENCE AND SAMPLING IN DLLM
The inference process begins by initializing sequence yo through the concatenation of a prompt sequence r and mask tokens, denoted as Yo a [MASK] ) ) Let M, be the set of indices corresponding to masked tokens at step 8; initially, Mo {|rl + -,....,7 |r| + L}. The model then iteratively refines this sequence over steps s 1. S until Ms 6. At each intermediate step s, the model computes probability distribution (o( y--1) for every masked position i E M,-1- From these distributions, the most likely token predictions y, from vocabulary and their corresponding confidence scores e; are determined:
i = argmax pe (14 = ulys-1) and c, = po(s = 9.1 -VV
(2)
A scheduling function (ss, S) determines the number of tokens, ka, to unmask. The set of indices update, Us, is chosen by selecting the kt positions from M,-1 with the highest confidence scores. The new sequence is then formed by updating these selected masked position.
2.3 RELATED WORK ON DLLM ACCELERATION
Research in dLLMs has moved from fixed Top-k decoding to dynamic, confidence-aware unmask- ing, that greedily unmask all tokens above a confidence threshold, to reduce generation bottle- necks (Wu et al. Wei et al.). A parallel focus is cache management overcome the inapplicability of conventional KV caching for bi-directional attention. Optimizations include reusing the cache for tokens (e.g., prefixes) by observing their key and values (Wu et al., Liu et al.) stable across inference step, and dynamically evicting entries based on low attention score to further improve efficiency (Song al. 2025a).
3

---

Published as a conference paper at ICLR 2026
K
K
Prafix
Out
Prefx
Cun, Sufti"
V
Out
II Prefix
Predxx
Daad
Layer
Reading
Suffi Curt
Sunfx Gur.
Out.
Suffa
Writing Mask
Mask
DPac
DPad Writing
DPad
0
DPad
Bulli
Attention
Layer
Connection
Layer 1
Figure 2: Attention score maps illustrating the Scratchpad mechanism in dLLMs. The attention matrix divided into 3 x blocks over prefx, current, and suffix. Blocks and collect information from the prefix and current into the suffix at layer while Block 6 feeds this stored information back to the current block at layer (n+1).
METHOD
In this section, we begin by formalizing the Scratchpad mechanism in dLLMs, which underpins the role of suffix tokens as an information reservoir. We then introduce Suffix Dropout, which combines fixed-length sliding window that moves with the current block to bound the number of attended suffix tokens, with distance-decay scheme that progressively prunes more distant ones. Finally, propose the Diffusiion Lottery Tickets (DLT) hypothesis, which provides principled lens for interpreting dropout in dLLMs.
3.1 SCRATCHPAD MECHANISM
We first revisit the role of suffix tokens in dLLMs. Due the masking structure, suffix tokens carry no direct semantic information but instead serve as an information reservoir, aggregating signals propagated from prefix tokens across multiple Transformer layers. This latent memory, which refer to as DPad (Diffusion Scratchpad), stabilizes the denoising process by providing contextual support for the current block.
As illustrated in Fig. and the attention score maps, the token sequence can be partitioned into three contiguous segments: Prefix indices [0, c- 1], current indices s-111], and suffix indices [s. L 1]. The corresponding attention matrix is thus divided into 3 x 3 blocks. Among them, Blocks 6, 7, and 8 together define the scratchpad mechanism.
Considering only one head, at layer queries from prefix and current tokens attend to keys from the suffix region. Formally, the global attention scores are defined as
((()))
(n) = Softmax
E TRLX
(3)
Vd
We can partition A (m) into submatrices corresponding prefix (P [0,c 1)), current (C' [c, s - 1),, and suffix (S = L- 1)) In particular,
(n) S.P ( [s: L., c).
A.C = A()) : L,, e c.s.s.
represent the attention scores from suffix queries to prefix and current keys, respectively (Blocks and in Fig. () Multiplying these attention scores with the value matrix yields the actual outputs suffix positions:
H (n) A (n) +
(n) A + A v())
Here, H(m) denotes the hidden representations of suffix tokens after attention. This equation shows that suffix tokens integrate information from prefix and current regions, effectively serving as DPad-n that records contextual information. After this aggregation, the outputs (n) are processed by the subsequent linear transformations (e.g., feed-forward layers and residual connections). which operate independently on each token. At layer (n+ 1), this stored information can be retrieved by the current block through
A C.S
(n+1) (1n-1) [c: 8: T.].
4

---

Published as a conference paper at ICLR 2026
0.020
Sumf
Moan (al sumples & feads)
min-mee
0.010
0.015
0.005
MW
Attention
0.010
Current
0.000
0.005
200
250
0.000
100
200
300
500
600
700
Key index Loffieet)
Figure 3: Analysis of the suffix dropout strategy at the final layer 31 using LLaDA-1.5 on GSM8K with max length 512. We gather attention scores from 100 samples across all heads, focusing on current block queries (A[css, c-200:]). The plot shows the mean attention over key indices (green) with min-max max range (shaded).
which corresponds to current-to-suff attention (Block 6 in Fig. 2). This path enables the current block to reuse the information collected in the suffix at the previous layer. In practice, the influence of the suffix on the prefix (Block 3) is negligible (Wu et al.). Therefore, the essential interaction of the scratchpad mechanism lies in the current-to-sufix direction (Block 6), where the suffix serves as temporary memory to assist the ongoing denoising process.
Therefore, we conjecture that the role of suffix tokens resembles a residual connection (He al., 2013), but specialized for attention, which we term an Attention Connection. Rather than directly propagating representations, the model compresses high-dimensional signals from both the prefix and the current tokens into the suffix, which is then re-injected into the current block at the sub- sequent layer. In this way, the suffix serves as scratchpad that compacts contextual information and forwards it for cross-layer reuse in dLLMs. We provide more analysis about this mechanism in Appx. A.I
3.2 SUFFIX DROPOUT STRATEGIES
As discussed in Sec. [ the scratchpad intuition does not require populating all suffix tokens, which is neither sustainable nor efficient. Empirically, we observe substantial redundancy among suffix tokens (Appx. A), motivating structured dropout design. As example, the final layer 31, Fig. Fig.Bishows a clear distance-depender decay in attention: nearby tokens dominate, consistent with general attention patterns and prior work on prefix sliding windows (Xiao al., 2023). We provide additional layer-wise attention visualizations in Appx. A.4
Based on these findings, we propose two complementary suffix drop strategies, as illustrated in Fig. Tc):
(i) Sliding window with fixed-length to retain only a bounded number of near-suffix tokens;
(ii) Distance-decay dropout that progressively prunes distant suffix tokens.
Both mechanisms are efficiently realized through a Gaussian sampling process, which simultane- ously enforces bounded window and distance-dependent decay.
Formally, for a suffix token at distance d from the suffix boundary, its retention probability P(d) is defined by the right half of standard normal distribution with effective window size W:
1 O
P(d)
exp
0 <<dsw,
where 0, hyperparameters, and a, are introduced to control the distribution along the z- and y-axes: (1) kor maps the window size W to ko (e.g., for W 256, setting & 3 ensures d 256 corresponds to 30r); (2) a scales the overall sampling magnitude vertically. This formulation ensures that tokens farther from the boundary are retained with exponentially decreasing probability, implemented via
0 1, and the suffix boundary is the center of the Gaussian distribution. Two
W
5

---

Published as a conference paper at ICLR 2026
a. Original
b. After Pruning
Average Attention Score
0.0375
Avergge Attertton Score
0.0175
Max
Jnyers
0.0150
Spotiggh:
0.0 0.0325
0.0125
0.0100
0.0100
0.0075
0.0075
0.0359
0.0550
0.0025
0.0325
0.0000
100
400
300 Key Index (offieet to Suffii)
Key Index Loffset Suffii)
Figure 4: (a) Attention scores of suffix tokens paid by current block tokens (A(css, s:]) across layers in LLaDA-1.5, showing overall decay with occasional spikes (e.g.. d 199, 298, 362). (b) After forcibly pruning these spike positions, attention shifts to nearby tokens, indicating that adjacent positions can absorb suffix information (e.g., pruning token 362 shifts the spike token 359).
rejection sampling. More implementation details and a hyperparameter analysis are provided in Appx. and Appx. D respectively.
As result, suffix attention only needs to focus on nearby tokens, making the suffix dropout window decoupled from the overall sequence length. Unlike the vanilla setting, where the suffix grows with the generation sequence, our approach keeps it constant. This yields a clear computational benefit: suffix dropout effectively reduces suffix-relater complexity by one dimension.
3.3 DIFFUSION LOTTERY TICKETS HYPOTHESIS
The analysis in Fig. 3 not only reveals the overall decay of current-to-suffi attention, but also occasional sharp spikes in the maximum values. These spikes cannot be predicted advance and, in principle, pruning them could affect model accuracy. To investigate this, we conduct an additional experiment shown in Fig. 4 We first run dLLM inference for one step, then forcibly prune the top 10 highest-attention suffix tokens ("spotlight" tokens) occurring beyond the first 128 positions (e.g., at distances 199, 298, and 362). Surprisingly. pruning such distant suffix tokens, even those corresponding to large spikes, has little effect on model accuracy.
When these spikes are removed, the model shifts its attention to nearby suffix tokens, for exam- ple, the spotlight token 362 in Fig. 4 (a) is replaced by increased attention at its neighbor token 359 in Fig. 4 (b), which effectively absorbs the lost information. This behavior is consistent with the strong generalization ability of diffusion models and the fact that suffix tokens are initialized without semantic content: through DPad mechanism, suffix tokens can dynamically learn and store information during inference. Consequently, information carried by distant suffix tokens appears largely local position-adaptivve and pruning spike positions has almost no impact final accuracy, as further confirmed in our evaluation experiments, as shown in T
These observations resonate with the Lottery Ticket Hy- pothesis (L.HH) (Frankle & Carbin| 2019)) which posits that properly pruned sub-networks with their original ini- tialization can match the performance of dense networks after training. We extend this intuition to dLLMs and propose the Diffuusion Lottery Tickets (DLT) hypothesis: During inference, the suffix region contains redundant to- kens, yet sparse subset is sufficient to preserve semantic consistency and generation quality. Through DPad mech- anism, this subset can be adaptively reorganized into "winning tickets" within the forward pass. In this view, suffix dropout becomes a training-free lottery ticket search, where Gaussian sampling selects compact set of suffix tokens that carry the essential information for denoising in dLLMs.
Table Accuracy Score of LLaDA-1.5 (Length 512).
Method
GSMBK HumanEval
Strict Flex.
Score
Baseline Top-10 Pruning
40.5 78.7
37.8
41.7
78.6
39.0
This explains why suffix dropout can be applied a priori, without computing exact attention scores, and why it fundamentally differs from prefix cache pruning (Song al. 2025a; Wang et al. 2021): Prefix tokens carry dense, position-bous carded, whereas suffix tokens act as a flexible, low-rank memory buffer.
semantic information and thus cannot be arbitranly dis-

---

Published as a conference paper at ICLR 2026
Table 2: Consolidated performance of -LaDA-Instruce and Dream-Base on four benchmarks.
LLaDA Insrrct
Dream Base
Effiicncc
Accurscy (1))
Efficieney
Accurccy (9)
Benchmark
Methed
Lsteneyts))
TPS1
Flexible!
Striet
Lasencyssl
TPS1
Effras
[lexible
Strict
Vanilla
27.48
1.0kx
8.44
1.00x
232/255
78:39
37.38
22.30
1.00x
11.43
1.00x
255/256
75.06
74.37
GSMSK
-DPuu
18.35
1.50x
5.76
1.04X
161 /
78.54
63.84
10.27
2.17x
12.75
1.11x
131/
75.25
75.06
4-ohor
-Par:
8.55
3
27.14
3,22x
232 0/0/255
78.54
36.67
13.84
1.61x
18.43
166x
255/256
75.51
74.83
-Par--iPaa
24.25
161
79.76
69.97
4.25%
2.11x
127/256
74.83
74.75
Vauilla
25.40
1.00x
256
33.58
21.01
1.00x
1.00x
256/226
34,16
MATH
-DPud
21.61
9.75
78.04
255/256
37.64
2,56x
2.38x
35.12
PPPld
22.93
2.34K
256
27.98
7.72
2.72x
2.71x
255/ /56
34,67
1,00x
1.00x
473
1.00%
1
511/512
51.22
HvaanEvvl
-DPud
27.41
1,26x
13.96
1.17x
438 /512
47.56
8.20
3.47x
26.83
220/512
51.22
0-shor
.Par.
11.45
3,02x
41.40
3.04K
475/512
43.29
14.15
2.01X
36.11
201x
511//502
53.05
9.14
3.79x
47.36
3,51x
438/512
46.34
4,06
7.01x
32.62
2.3xx
214/512
52.44
Vanilla
62.11
1.00x
4.82
1.00x
709 /512
15.00
49.15
1.00x
10.42
100x
512/512
$2.40
MBPP
-Diud
15389
3,91x
6.85
1.42x
109 /512
20.40
41.36
1.19x
12.38
1.1%k
512/.512
$2.60
3 shoor
-Par.
16.26
4.36H
20.99
4,36x
299/512 /512
15.00
12.38
3.97x
41.36
3.97%
512/512
55.40
6.02
10.32x
18.28
3.79
130/512
39.40
9.86
4.98
51.92
4.8X
512/512
54,80
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Models and Baselines. All experiments are conducted on an NVIDIA A100 80GB GPU. We evaluate DPad on suite of representative open-source dLLMs: two variants of LLaDA (8B-Instruct and 1.5) (Nie et al,; Zhu et al., 2025) and Dream-v0-Base Ba (Ye et al. 2025
We compare our
method against three baselines: the unmodified LLaDA (Nie et al. Zhu et al. 2025) and Dream Ye et al. 2025) backbones, denoted as Vani ll; the vanilla backbone augmented with parallel decoding (w et a..), denoted as +Par.; and version implementing prefix caching (Wu et al.), denoted as PC.. Since our dropout already minimizes the computational cost of the suffix, we do not adopt the Dual Cache mechanism proposed in (Wu et al.).
augment
The DPad strategy introduces three hyperparameters: a decay rate factor k, a magnitude scalar a, and sliding window size. We tuned these small subsets of each benchmark (see Sec. 4.3.2).
Unless otherwise specified, all experiments use a block size of 32, a batch size of I, and a confidence threshold of 0.9 for parallel decoding.
Benchmarks and metrics. We evaluate reasoning benchmarks (GSM8K (Cobbe et al. 202
MATH (Hendrycks et al. (202T) and code generation benchmarks (HumanEval (Chen et al. MBPP (Austin al.) (021b))- Accuracy is reported with task-specifie metrics (e.g.. pass @ fexible/strict-matele Efficiency is reported as (a) mean end-to-end latency per sample and (b)
2021
lokens-per-second (TPS). We also report the average generated length relative to the maximum allowed sequence length &max as E/bmax to make length effects explicit.
4.2 MAIN RESULTS
Across all four benchmarks, DPad improves latency and achieves comparable or higher TPS relative to Vanil la, while preserving or improving accuracy, as shown Tbl. 2 When combined with parallel decoding (+Par.), DPad achieves additional improvement beyond +Par. alone; results for LLa are in Appx. C.I
Efficiency. DPad contributes to latency gains through two mechanisms: (1) reduced suffix com- plexity, which shifts suffix computation from quadratic toward linear scaling; (2) more concise generations, as evidenced by the É/fmax metric. This conciseness is beneficial, as case studies in Appx. E demonstrate that suffix dropout suppresses the generation of tokens that play auxiliary role and do not contribute significantly to reasoning.
Overall, focusing latency the primary efficiency metric, DPad yields 1.18x 3.91x speedups over Vanilla. With parallel decoding, total speedups reach 2.72x relative to Vanilla. This complementarity arises because the two methods target orthogonal bottlenecks: DPad elimi-
7

---

Published as a conference paper at ICLR 2026
nates redundant KV-token computation, while parallel decoding mitigates dependency constraints. By combining these approaches, we exploit both finer- -oraiee token pruning and safe multi-token prediction, yiclding substantial efficiency gains.
In our analysis, we observe a subtle distinction between latency- and TPS-based efficiency metrics. While DPad consistently reduces per-sample latency, its TPS gains may appear less pronounced be- cause it often encourages the model to generate more concise and complete responses, reaching the end-of-sequent token earlier. As discussed in Appx. [ this behavior reflects not a limitation but an improvement: the model terminates naturally rather than exhausting its context with low-quality continuations. However, when the generation length is substantially reduced, lower GPU utiliza- tion can mechanically decrease TPS. Since the extra tokens produced by baselines often carry low semantic value, we report raw throughput without post-proeessing. This also highlights a broader point: the community may need to reconsider throughput metrics and develop alternatives that bet- ter balance sequence length and accuracy, rewarding models that achieve comparable accuracy with meaningful generations.
also important to note that, to align with previous work, the benchmarks this section focus on multi-shot, short-sequence generation. In these settings, the prompt constitutes the majority of the sequence, meaning suffix attention accounts for only a small fraction of the total compu- tation. Per Amdahl's law (Amdahl 1967). the maximum achievable speedup is therefore inher- ently bounded. Despite this limitation, DPad consistently delivers stable latency improvements even without system-level optimizations. Its true potential emerges in low-shot, long-sequence settings (discussed in Sec. [4..1I, where the suffix becomes a more significant computational component, allowing DPad to achieve substantial additional latency reductions.
Accuracy. addition to improving inference efficiency, DPad also enhances accuracy across nearly all tasks for the LLaDA models (Tbl. and Tbl 6). thereby defying the typical trade-off be- tween speed and accuracy. For instance, DPad yields substantial gains in strict-match accuracy on GSM8K (+26.46%) and MATH (+19.62% for LLaDA-Instruct. This improvement in strict-match score is particularly noteworthy, as it highlights DPad's ability to enhance in-context learning. The vanilla backbone typically exhibits low strict-match performance (e.g., only 37.38% on GSM8K for LLaDA-Instruct), since this metric requires the model not only to produce the correct final answer (Flexible-Match) but also to adhere to the specific reasoning format demonstrated in few-shot ex- emplars, as illustrated in the Appx. E (Fig. 16.. We posit that failures in strict matching often stem from interference by distant suffix tokens, which introduce low-value or off-format patterns that distract the model and encourage verbose or poorly structured outputs. By reducing the influence of such suffix tokens and directing attention toward high-value, information-rich prefix exemplars, DPad enables the model to replicate the structured reasoning formats required by strict matching more faithfully. In addition, by suppressing redundant generations. DPad facilitates earlier con- vergence to concise, fell-formatted outputs, further improving strict-match performance without altering model parameters.
By contrast, the Dream model shows less advantage, with accuracy broadly comparable to the base- line and fluctuating only within a narrow margin. We attribute this discrepancy to the models' dis- tinct training protocols. Since LLaDA is trained as diffusion model from scratch, it leams to rely on the full suffix context during pre-training. Conversely, Dream is initialized from Qwen2.5 (Qwen) al. 2025), a pre-trained autoregressive model that has no exposure to suffix tokens. We there- fore hypothesize that LLaDA's native dLLM architecture is inherently more sensitive to the suffix context, allowing to benefit more significantly from the regularization that DPad provides.
4.3
ABLATIONS AND ANALYSIS
4.3.1
MAXIMUM GENERATION LENGTH
To quantify efficiency gains in long- generation, analyze the speedups of various accel- eration strategies on 5/GSS1KK (TbL.3). Our setup evaluates combinations of DPad, paral- lel decoding (+Par. -), and prefix caching (+PC.) disentangle the contribution of each technique. These findings are confirmed by a complementary analysis on the Dream- Base model (Appx. C.2.2
8

---

Published as a conference paper at ICLR 2026
236
BL
90
oof -Ffereservee Suffis Tokkns
Preserved Sullii Tokers
oa Preserved Sullix Tikento
of Preserved suffix Tokees
[a).Uinfform Mlassbfe-Match Scores
GajuGaus
Uniforr Strict-Match Scores
(d).Gaussian Strict-Match Scores
Maxcible-Matchee Scores
Figure 5: Ablation Study on Sliding Window Size and Dropout Function for DPad LLaDA- 1.5/GSMKK (512, 4-shot). Heatmaps showing Flexible-Match Accuracy scores with (a) uniform and (b) Gaussian dropout, and Strict-Match Accuracy scores with (c) uniform and (d) Gaus- sian dropout, across varying sliding window sizes and number of preserved suffix tokens. The (tokens, window_size (512, 512) configuration corresponds to the baseline, as involves no to- ken dropout.
Efficiency. We find that the acceleration benefits of DPad grow substantially with generation sequence length. As shown in Appx. [C.1] for LLaDA-1.5 on GSM8K, improve- ments are modest at shorter sequence lengths (up to 1.51 under a 256-token limit). However, when the maximum length is extended to 1024 tokens (single-shot setting), stan- dalone DPal achieves a dramatic 20.3x speedup. Finally, when combined with parallel decoding and prefix caching (Fast-dLLM), the efficiency gains compound, yielding an overall 61.39x speedup compared to vanilla LLaDA and a 5.21 x improvement over Pa.t alone. These re- sults demonstrate that suffix dropout and parallel decoding address orthogonal bottlenecks, and their combination enables significant improvements in long- sequence generation. deeper analysis of these effects is provided in Appx. C.2.1
Table 3: Performance on LLaDA-1.5 with GSM8K (1024 tokens, 1-shot).
Method
Acc.
Lat(()/ TPS
Speedup
(Flex. /St.)
Vanilla
78.17/48.98 48.98
127 / 1,55
1.00x
DPad
78.77 74.07
18.4
20.3x
+Par.
78.77 149.43
11.7 16.9
1,00x
DPad
79.38/ 74.22
2.26/51.
5.17x
+Par. PC,
78.77/51.63
10.8/ 18.2
1.00x
DPad
77.10/ 70.66
2.07 75.55
5,21x
Overall (+Pac vs. Vanilla)
61,39x
Accuracy. Low-shot settings further highlight DPad's ability to preserve and even enhance model accuracy. Compared with Tbl. 0 the strict-match score of the baseline drops significantly when the shots reduces from 4 to (from ~00% to ~50%), whereas DPad's performance remains remarkably stable (dropping only from 78% to 77%%) This resilience demonstrates that DPad substantially strengthens the model's in-context learning capability, significant achievement for training-free free method.
4.3.2 SLIDING WINDOW SIZE AND DROPOUT FUNCTION
We conducted ablation study on LLaDA- ALLDA--L55/GS 1/5GSM8KK determine the optimal sliding window size and dropout function for DPad, with results shown in Fig. 5
Our analysis identifies critical context window of 64- tokens immediately following the current block. The key principle for maximizing performance is to maintain a high density of preserved tokens within this critical window. Spreading a limited token budget (e.g., fewer than 128) thinly across a larger window was found to be counterproductive, significantly degrading accuracy.
The study also validates our choice of Gaussian dropout function over a uniform baseline. While the two perform comparably with large token budgets, Gaussian dropout is consistently su- perior under low-budget conditions, especially when the sliding window is large. This is because it more efficiently allocates the limited token budget by prioritizing nearby tokens, strategy that aligns with the consistent decaying patterns observed in Rotary Positional Embeddings (RoPE) (Su) et al. 202T), attention scores (Fig. 4. and token confidence maps (Gong al.).
Of course, Gaussian sampling may not be the optimal decay function, and other decay-based schemes (e.g., exponential, linear, or step-wise cutoff) remain to be explored. Nevertheless, in the training- free setting, we find that results are largely insensitive to the exact decay form, as long as the scheme emphasizes nearer tokens.

---

Published as a conference paper at ICLR 2026
5 CONCLUSION
We addressed the high cost of full suffix attention in dLLMs key bottleneck caused by redundant computation-by introducing the Diffusion Scratchpad (DPud), simple, training-free inference strategy. DPud leverages the inherent sparsity of suffix attention, combining fixed-length sliding window with distance-decay dropout that prunes low-valuc tokens priori. This "winning ticket" approach yields up to a 61.4x speedup when combined with existing optimizations, without sacri- ficing accuracy. Our results establish DPad practical and scalable solution that helps advance dLLMs from a promising alternative to a viable foundation for future language technologies.
ETHICS STATEMENT
This research focuses improving the efficiency of diffusion-basse large language models (dLLMs) through training-free inference strategy (DPad). Our study does not involve human subjects, user data, or personally identifiable information. All datasets used in evaluation (GSM8K, MATH, Huma MBPP) are publicly available and widely adopted benchmarks, cited appropri- ately in the paper.
Bias and fairness. As our method is applied to pre-existing models and datasets, any biases or lim- itations originate from the underlying training corpora of the base models and benchmarks. We do not introduce new datasets, but we acknowledge that accelerated inference could facilitate broader deployment of LLMs, which inherit and may propagate these biases.
Privacy and safety. Our work does not involve private data. The proposed method is model-agnostic and does not alter the underlying training distribution.
Dual use considerations. Improvements in inference efficiency may enable faster and more widespread use of LLMs. We encourage responsible application, with attention to fairness, trans- parency, and safety in downstream use.
REPRODUCIBILITY
We provide resources needed to reproduce our results. Implementation details, hyperparameters. and ablation setups are described in Section 3ladd and Appendices and C Specifically: (i) code and scripts for suffix dropout (DPad) and evaluation will released with instructions for running on each benchmark: (ii) exact hyperparameter settings (sliding window size, decay rate, scaling factor) are reported Section [.T and Appendix B (iii) dataset usage (GSM8K, MATH, HumanEval,
MBPP) follows standard publicly available versions with citations; (iv) experiments
NVIDIA A100 80GB GPUs, as noted in Section 4.1] (v) random seeds and evaluation commands documented in the code release. An environment will also be provided "requirements.n All tables and figures can be regenerated directly from the released code and instructions, available as supplementary material.
ACKNOWLEDGMENTS
This work was supported in part by NSF-2112562, NSF-2328805, and ARO 191INF-3---0224 The authors sincerely thank the anonymous reviewers for their constructive feedback and valuable suggestions that greatly improved the quality of this work.
REFERENCES
Gene M Amdahl. Validity of the single processor approach to achieving large scale computing capabilities. Proceedings of the April 18-20, 1967, spring joint computer conference, pp. 483- 485, 1967.
Marianne Arriola, Aaron Gokaslan, Justin T. Chiu, Zhihan Yang. Zhixuan Qi, Jiaqi Han, Sub- ham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregres- sive and diffusion language models, 2025. URL httts:t////artt httpaa/www....o////////.....00 09573
10

---

Published as a conference paper at ICLR 2026
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Struc- tured denoising diffusion models in discrete state-spaces. In Advances Neural Information Processing Systems, volume 34, pp.
2021a.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021b. URL https:ttp//cc....t...ttt /tt//axiiivvv. wtt.o//g//bb./aaee /2108./211... 07732
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.051500 2020.
Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Kor- bak, and Owain Evans. The reversal curse: Lims trained on "a b" fail to learn "b a", 2024.
URL
abs /209..239.... 12288
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL https: /tap::///rxiiin owg. abs 2005. 14165
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss. William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained code. 2021.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. URL https://arxiv.org
abs/18100 04805
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL https://
openreview / http://oummmidddrp b33
Shansan Gong, Ruixiang Zhang, Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong, and Yizhe Zhang. DiffuCoder: Understanding and improving masked diffusion models for code gen- eration. URL http:://aaiiiii http:aawwwv....raaaaaaaaaa /btp:///bb//...... 20639
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Ko- renev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes. Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco

---

Published as a conference paper at ICLR 2026
Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind That- tai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Kore- vaar, Hu Xu, Hugo Touvron, lliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Eytimov, Jack Zhang. Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jong- soo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani. Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathicu Rita, Maya Pavlova, Melanic Kambadur, Mike Lewis, Min Si, Mitesh Ku- mar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoy- chev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick AIrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ra- mon Calderer. Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Ro- hit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang. Saghar Hosseini, Sahana Chennabusappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang. Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mi- haylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do. Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaelle Gold- schlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, An- drew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, An- nie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leon- hardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic. Brian Gamido, Britt Mon- talvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le. Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smoth- ers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel. Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harri- son Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, lean-Baptiste Gaya, Jeff Mareus, Jeff Tang, Jen- nifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Jun- jie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kylc Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L.
12

---

Published as a conference paper at ICLR 2026
Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Mumish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Ro- driguez, Raf Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ra- maswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar. Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe. Steve Satter- field, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Ku- mar, Vishal Mangla, Vlad Ionescu. Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiao- jian Wu, Xiaolan Wang. Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang. Yossi Adi, Youngjin Nam, Yu, Wang. Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary Veiito Zef Rosnbrick, Zhao- duo Wen, Zhenyu Yang. Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024. URL https: /tapps://aaxiv...o org/ www.gg/aaaas.... abs/ 2407.21783 783
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. Non-autoregressive
neural machine translation. In International Conference on Learning Representations, 2018. URL
https: http:////peneeeeeeeeteeeeee
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- 1112.. 03385
nition, 2015. URL/stts://axxxi.ooo
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, 2020. URL
https: // //axxiv... org/ wwwp/ao.////./oooooo/oooo/oo.ae/ /tlbs/2005 6006.006010 11239
Daniel Israel, Guy Van den Broeck, and Aditya Grover. Accelerating diffusion Ilms via adaptive parallel decoding, 2025.
oww. abtp:/bbs///.66 00413
Schoon Kim. Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, and Kurt Keutzer. Learned token pruning for transformers, 2022. URL httpps://axxi.......
Abbs/2107.0091
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. Diffusion- LM improves controllable text generation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL net foteee/ooommmmddd0101000 Lyyk
Zhiyuan Liu, Yieun Yang, Yaojie Zhang, Junjie Chen, Chang Zou, Qingyuan Wei, Shaobo Wang, and Linfeng Zhang, dLLM-cache: Accelerating diffusion large language models with adaptive caching. URL
org/ 2506. 06295
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffuusion modeling by estimating the ratios of the data distribution. In Proceedings of the 41st International Conference on Machine Learning, ICML'2 244 JMLR.org, 2024.
Shen Nie. Fenggi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji- Rong Wen, and Chongxuan Li. Large language diffusion models. URL AtLattp: http///xxvvv.o//// abs/ wwwp./502..m 09992
13

---

Published as a conference paper at ICLR 2026
Shen Nie, Fengqi Zhu, Chao Du, Tianyu Pang, Qian Liu, Guangtao Zeng, Min Lin, and Chongxuan Li. Scaling up masked diffusion models on text, 2025. URL https: haapps://arxiv..// org/ /bs/ 2410.1852
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder. Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Pro- ceedings of the 36th International Conference on Neural Information P Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088.
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu. Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL hnttps://arxtt /atp:://arxiv.oo abs 2412. 15115
Radford. Language models are unsupervised multitask learners Technical report, OpenAl, 2018.
URL httph:tttppppnntte cow http//idexx///e http:///nneeee/aauuuueeeeeeepppe
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. Technical report, OpenAl, 2019.
models are https:/ccccccpppaaaooo////etttte modeeeeeemdel/////e//oe m/ndels/languuge.
Jeffrey Seely, Yuki Imajuku. Tianyu Zhao, Edoardo Cetin, and Llion Jones. Sudoku-Bench.
https: //ghthbb.. com/ Swt.a/aaaaaaaaaaaaaaaaaaaaaaaaaa Sww.aAAAASSSuuuSAaaaaaaaaaua -Bnchh 2025.
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K. Titsias. Simplified and gen- eralized masked diffusion for discrete data, 2025. htll:https//aarxiv.or//abbb//
04329
Yuerong Song, Xiaoran Liu, Ruixiao Li, Zhigeng Liu, Zengfeng Huang, Qipeng Guo, Ziwei He, and Xipeng Qiu. Sparse-dllm: Accelerating diffusion Ilms with dynamic cache eviction, 2025a. URL
https://aase //tp:v/aaivvvvvvvv/ http://wwwwwwv..ga//////oss
02558
Yuxuan Song, Zheng Zhang, Cheng Luo, Mingxuan Wang. Lin Yan, Xiaoying Jia, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang. Yonghui Wu, and Hao Zhou. Seed Diffusion: A large-scale diffusion language model with high-speed inference. Technical report, ByteDance Seed and Institute for AI Industry Research (AIR), Tsinghua University, July 2025b. URL https: l.sedd.bttedane
com/ seed diffusion
Jianlin Su, Yu-An Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced trans- former with rotary position embedding. arXiv preprint arXi::2/04.09866 2021. URI. https:
09864
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998 6008, 2017. URL
nww.iisur..... cc/ htae://777.aaaaeeeeeeeeeeesaaaas
Hanrui Wang, Zhekai Zhang, and Song Han. Spatten: Efficient sparse attention architecture with cascade token and head pruning. In 2021 IEEE International Symposium on Hiigh-Performance
Computer Architecture (HPPA).. IEEE, February 2021. doi: 10. 10.1 1wl.11999cc51114722221..1 100018.
URL http:://xx......o./II...... wtt.://///...1100///A/A1111111 00011114672222..000000000
Qingyan Wei, Yaojie Zhang, Zhiyuan Liu, Dongrui Liu, and Linfeng Zhang, Accelerating diffusion
large language models with SlowFast sampling: The three golden principles. URL http://
arxiv. www.vsorggab//ooo /206.. 10848
14

---

Published as a conference paper at ICLR 2026
Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao. Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. Last-dLLM: Training-frree acceleration diffusion LLM by enabling KV cache and parallel decoding. URL http:///axxx org/ abs 2200 22618
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprini arXiv:2309. 17453, 2023.
Jiacheng Ye, Shansan Gong, Liheng Chen, Lin Zheng, Jiahui Gao, Han Shi, Chuan Wu, Xin Jiang, Zhenguo Li, Wei Bi, and Lingpeng Kong. Diffusion of thoughts: Chain-of- diffusion language models, 2024. htttt::::///atr
reasoning in wtwp/bs///aaeo.ee 2402 07754
Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang. Zhenguo Li, and Lingpeng Kong. Dream 7b, 2025. URL https:://kkulpp:: wht.k/u.wwppp..uu...ooapauu http://bb....ooloo//oooo///// 20tp//re22mmdmmmmmmmm
Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chunwei Wu, Jun Hu, Jun Zhou, Jianfei Chen, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Llada 1.5: Variance-reduced preference
optimization for large language diffusion models, 2025. URL
org/abs/
2505. 19223
15

---

Published as a conference paper at ICLR 2026
A OBSERVATIONS
A.1 EMPIRICAL VERIFICATION OF THE SCRATCHPAD MECHANISM
In Section 3.1, we hypothesized that suffix tokens function as an Attention Connection-a "scratch- pad" that stores intermediate states guide denoising. To verify this, we analyze the scratchpad mechanism along two dimensions: (1) Cross-Layer (transmitting information across Transformer layers within step) and (2) Cross-Step (transmitting information between decoding steps).
We quantify the criticality of these dimensions by evaluating performance on Countdown (Ye et al,, 2024) and Sudoku x (Seely al., 2025) tasks requiring strict global consistency and thus serving as ideal stress tests for the scratchpad mechanism -croos spectrum of scratchpad avail- ability:
No Seratchpad (ARMs): We evaluated -LaMA-3.--BB-Isstee (Grattafiori et 2024) and Qwen-2.5-7B (Qwen et al. 2025). As ARMs inherently lack the bidirectional attention required to attend future tokens, they are structurally blind the suffix and serve baseline for the complete absence of scratchpad.
Partial Scratchpad (Semi-Autoregressive dLLMs): We evaluated standard Semi-AR de- coding and Semi-AR with Suffix Dropout (Semi-AR+SD), In this setting, suffix tokens are masked between steps. prohibiting Cross-Step information transmission. However, the model retains limited scratchpad capability via Cross-Layer attention within the current block.
Full Scratchpad (Global-Diffusion dLLMs):: Under Global-Diffusion decoding, the model writes confident suffix tokens, enabling both Cross-Layer and Cross-Step scratch- pad functions. Alongside the Global-Diffusion baseline, introduce Semi-AR+Sullix Dropout-Pre-Decoding (Semi-AR+SD+PDD, In this variant, preserved suffix tokens are temporarily unmasked during the current block's decoding (conditioned confidence) and remasked upon completion of current block. This effectively restores the Cross-Step transmission channel.
Table 4: Representative examples of the Countdown Ye et al. 2024) and Sudoku (Seely al. benchmarks. Sudoku (4 x 4) serves as a rigorous test for global planning, requiring the model to empty cells ('O') while maintaining strict consistency across rows, columns, and subgrids.
2025
Task
Question
Answer
Countdown
Given numbers, use +-/ to operate 82-81=1, 84-1-33, 83-42=41 over the first four numbers achieve fifth number.
Input: 81,84,82,42,41
Output:
Sudoku
Fill the position where the values are 0 in 4x4 grid with digits 1-4 so that each
4231
3124
column, each row, and each the four
1342
2x2 subgrids that compose the grid con- tains all of the digits from to 4. Input:
2413
4001
3124
1302
0400
Output:
The results (summarized in Tbl. 5 reveal a decisive behavioral divergence. On Sudoku, both the No Scratchpad (ARMs) and Partial Scratchpad (Semi-AR/ Semi-AR+SDD groups achieve 0.0 accuracy,
16

---

Published as a conference paper at ICLR 2026
Table 5: Quantitative verification of the scratchpad mechanism on planning benchmarks. We com- pare performance across three levels of memory availability. The results show that Sudoku perfor- mance collapses 0.0 without the full scratchpad (AR models and Semi-AR dLLMs) and recovers only when the suffix memory is fully enabled (Semi-AR+SD+PD and Global-Diffusion
Scratchpad Setting
Models
Countdown
Sudoku
LLaMA-3.1-8B-Im
27.0
0.0
No Scratchpad
Qwen-2.5-7B
Partial Scratchpad
LLaDA-Instruc (Semi-AR)
32.6
(Only cross-layer)
LLaDA-Instruct
32.6
0
Full Scratchpad
LLDDA-Instrut (Global-Diffusion)
34.7
24.6
(cross-layer cross-step)
LLaDA -nnttuuctt (Semi- A
32.6
24.4
indicating a complete f'illure to handle global consistency without cross-step memory. In contrast, the Full Scratchpad group demonstrates superior performance: Global-Diffusion LLaDA achieves scores of 34.7% on Countdown and 24.6% on Sudoku. Crucially, restoring the cross-step scratch- pad via Semi-AR+SD+PP boosts Sudoku performance from 0.0 to 24.4%, effectively matching the Global-Diffusion baseline (24.6%). This confirms that the Cross-Step scratchpad the critical component for global planning.
A.2 ANALYSIS OF SUFFIX TOKEN ENTROPY
To provide further intuition for method, analyze the Local Entropy of generated suffix tokens. is important to distinguish this metric from standard predictive entropy (uncertainty over the vo- cabulary). Here, define Local Entropy the statistical diversity of tokens generated specific suffix position across multiple samples. High local entropy indicates position that changes dynam- ically depending on context (e.g., reasoning step), while low local entropy indicates a position that remains static.
As shown Figure the entropy of suffix tokens decays rapidly with distance from the current generation block. Tokens far into the suffix exhibit near-zero entropy, meaning they do not vary across samples. Qualitative analysis confirms that these are not deterministic structural elements (which would shift based on content length), but rather static, repetitive patterns (e.g., repeated <eos> tokens or padding). This empirical finding strongly motivates our distance-decay dropout strategy, which prioritizes pruning these low-value, redundant tokens.
2.0
Local Entropy (bits)
Average (across samples)
min-max range
1.5
1.0
0.5
100
200
300
400
Key index (offset to Suffix)
Figure 6: Local entropy of suffix tokens.
The plot shows the average local entropy (in bits) of suffix tokens as a function of their distance from the current block. Entropy rapidly decays, approaching zero for distant tokens. This confirms that distant suffix tokens are highly redundant and can be pruned with minimal information loss, validating the core assumption of DPad.
17

---

Published as a conference paper at ICLR 2026
Prompt
Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning andbakes muffins for her friends every day with four. She sells the remainder at thefarmers' market daily for $2 per fresh duck egg. How much in dollars does she makeevery day at the farmers' market?
Block O, Step 7
Final Answer
To determine how much Janet makes every day at the to follow
To determine how much Janet makes every day at the market, we
farmers
steps C
follow these steps:
Calculate Calculate the number of eggs 8gga per day day C C C C per per
Calculate the total number of eggs laid by Janet 's ducks per day C Sub tract number eggs Janet for breakfast and akes for her friends. C Multiply the remaining eggs by
C C C C C C C C
C C
per egg find her daily earnings C C C 1 Janet 's ducks lay 16 eggs day Ć egge for breakfast and
C C C
CO Č C C
eggs her friends Therefore. she uses a total of 4 = eggs per day C. The number eggs
C. C C C
C
A. dollars
sale eggs. C 4 Janet sells each egg for $ Therefore her daily earnings from selling eggs are 9 2 $ C C So Janet makes $ 18 every day the farmers market 18 <jeot jolo </endoftext|>
dollars dollars C C O C makes makes makes makes day day day at market market. C.C. C endoftexte
Current Tokens Suffix Tokens
Ć: Newline
Figure 7: Decoding of response tokens at an intermediate step. The plot shows the most likely tokens predicted by the model at a given decoding stage, reflecting its intermediate understanding and behavior response to the prompt. Notably, the model repeatedly records key tokens in the suffix, irrespective of position, revealing scratchpad-like pattern that facilitates decoding of the current block.
A.3 PRE-DECODED SUFFIX TOKENS
To further illustrate the scratchpad mechanism, present the most likely tokens that LLaDA 1.5 (Zhu et al., 2025) predicts at both the current and suffix positions, as shown in Figure (bottom ( left). The sample question is drawn from GSM8K (Cobbe al., 2021), and final generated sponse is included for reference. From the intermediate predictions, we observe recurring pattern where key tokens are repeatedly placed across multiple suffix positions. This behavior resembles the use of scratchpad, where the model leverages available "empty" positions to temporarily store useful information that aids in decoding the current tokens.
A.4 SUFFIX TOKENS ATTENTION SCORE
We visualize the attention scores of layers 0, 7, 15, 23, and 31 in LLaDA-1.55 following the same procedure described Fig. The results are shown in Fig.
Across layers, we observe a consistent decay in attention paid to suffix tokens as their positions move farther from the current block. addition, attention spikes on suffix tokens become increasingly pronounced in deeper layers, suggesting that the model allocates more focused attention to these suffix tokens as the forward pass progresses.
A.S GENERALIZATION SCRATCHPAD MECHANISM DREAM MODELS
To verify the generalization of the scratchpad mechanism to other dLLM architectures, extended our analysis (specifically, the Attention Score Map and Spike Shift experiments) to the Dream-Base model.
Fig. illustrates the attention score distribution for Dream. Similar to LLaDA, Dream exhibits a consistent distance-des -etance-deccy trend. However, compared to LLaDA, Dream displays significantly fewer high-ate high-attention spikes in the suffix region. In many layers, the attention scores allocated to suffix tokens near-zero. We hypothesize that this behavior stems from Dream's training paradigm: it
18

---

Published as a conference paper at ICLR 2026
0.020
0.010
nh-ma
0.015
0.005
Suffix
Current
0.000
0,005
200
250
0.000
100
200
300
400
500
600
700
Sky Indse (ofteet)
(a) Attention score at layer 0
0.020
Nean (all sumplex
0.010
Alunton vake
0.005
Suffix
0,010
Burnst
0.000
0.005
200
250
0.000
M
100
400
700
Scy taftsct)
(b) Attention score at layer 7
Mear (4ll Groteec/eoad
0.010
0.01.5
AllrrM 1000
0.005
Suffix
0.010
Currnnt
0.005
100
200
400
500
700
(offset)
(c) Attention score at layer 15
0.020
0.010
0.015
0.005
Current
0.005
200
250
0.000
100
200
300
400
500
600
700
Sky Indox (ofteet)
(d) Attention score at layer 23
0.020
Nean (all sundlex heads!
0.010
0.005
4/4
Current
0.000
0.005
200
250
0.000
100
700
Say (aftsct)
(c) Attention score at layer 31
Figure 8: Attention score at different layers L.L.DA-1.5 on GSMSK with 512 generation length.
is initialized from Qwen-2.5, standard autoregressive model, which inherently lacks bidirectional attention to future tokens during pre-training.
Despite this general sparsity, we observed distinct attention spikes in deeper layers (e.g., Layer 23). To test the robustness of these features, we conducted the same spike shift experiment on the Dream- Base model. Notably, the "spike shift" phenomenon is replicated in Dream, as shown in Fig. TO
19

---

Published as a conference paper at ICLR 2026
When high-attention suffix token is forcibly pruned, the model shifts its attention to an adjacent neighbor (e.g., shifting from index 50 to 54), thereby preserving the informational flow.
The replication of both distance-decay patterns and position-adaptive attention shifts strongly sup- ports the generalization of the Scratchpad mechanism and the Diffusion Lottery Ticket hypothesis across different dLLM backbones.
0.3330
0.320
0.350
0.0150
03875
0.0375
0,025
0.802
156
103
<2c
00030
126
933
(a) Attention score at layer 0
(b) Attention score at layer 7
0.3300
00230
0.125
0.3222
00020
0.9899
0.9390
03993
0,030
A.23
tofscc
(c) Attention score at layer 15
(d) Attention score at layer 23
Figure 9: Attention score at different layers in Dream-Base GSM8K with 256 generation length.
Original
After Pruning
0.0200
0.02.0
Aug acrass layers Min-Max range
Aug acress layers Min range
0.0175
0.0175
Average Attention Score
Average Attention Score
0.0150
0.0150
a.0125
0-19
d-165
de349
d-474
0.0325
0.0100
0.00.0
d-50
d-165
d-3
d-474
0.0075
0.00 0.0075
0.0050
0,0050
0.0025
0.0025
0.0000
0.0000
1000
200
300
400
500
5
100
200
300
400
500
Key index (offset to Suffix)
Key index (offset to Suffix)
Figure 10: (a) Attention scores of suffix tokens paid by current block tokens (A[cts, s:]) across layers in Dream-Base, showing overall decay with occasional spikes (e.g.. d= 50, 349, 474). (b) After forcibly pruning these spike positions, attention shifts nearby tokens, indicating that adjacent positions can absorb suffix information (e.g., pruning token 50 shifts the spike to token 54).
20

---

Published as a conference paper at ICLR 2026
B IMPLEMENTATION DETAILS
This section provides implementation details for DPad, including the core code logic, the early termination mechanism, and the necessary adjustment to RoPE to ensure positional consistency.
This dropout strategy is not static but follows block-wise schedule (Nie et al.). As shown in Fig. [ (c), after each block generated. all previously dropped tokens are restored, and Gaussian sampling is re-applied to the updated suffix set before decoding the next block. This dynamic re- sampling prevents sampling bias and ensures that no token is systematically ignored.
Core Logic. DPad's core mechanism be implemented with only few lines of Python code. The primary operation involves sampling subset of suffix token indices using a Gaussian distance- decay function and then forwarding only this pruned sequence the model.
Algorithm DPad Suffix-rropou Inference Step
Input: Indices of tokens E (prefix), C (current block), S (suffix); window W; decay params (k,a); model ps
Output: model output for this step
1: /*select near-suffi window */
2: Sw +- the first W indices of S
3: / Select suffix indices using distance-decay sampling */
4: I. + {j Sw u S P(d,)}
where jj=j max 1 (dist to current). My ~ Uniform 1),
and P(d,) min(1, a(2rr) 1/2 exp( - ] k )
5: scratchpad <- x[[[]]
6: x' + (x)), x[C], scratchpad)
7: output + (=²)
8: return output
As outlined in Algorithm 1 the Gaussian- based sampler selects subset of suffix tokens according to distance-aware probabilities. We then construct pruned sequence x' by indexing with I.. The rest of the model remains unchanged, except for minor adjustment the RoPE embeddings inside the attention module to ensure correct positional encoding under suffix dropout.
RoPE Adjustment. Our suffix dropout mechanism requires only a minor adjustment to the RoPE (Su et al. 202T) to maintain correct positional information. In standard RoPE, token at absolute position is encoded using an angle 8 = i. A. After dropout, however, only a sparse, non-contiguous subset of suffix indices I (i1,..., im) is preserved.
To handle this, we ensure that each preserved token retains its original positional information. Rather than using their re-indexed positions after dropout, we apply mapping function f(ix) that retrieves the original absolute position of the k-th preserved token. The new angle is then computed as
0g ((xx)
(8)
Accordingly, the modified RoPE application becomes
RoPE' (x4, ix)) = RoPE(xi. f(ix)).
This adjustment requires only a lightweight remapping inside the attention module and does not alter the functional form of RoPE, confirming that suffix dropout nearly cost-free while preserving positional consistency.
Early Termination. To further enhance efficiency, we also implement an simple technique, called early termination. It halts generation upon detecting end-of-sequence (<eos>) token. This tech- nique, which performs check after each decoded block, addresses the computational redundancy inherent in fixed-length generation models like LLaDA (Nie et al.). Its utility is most pronounced in long-sequence settings where the maximum length significantly exceeds the actual generated con- tent, particularly for models employing top-k decoding.
21

---

Published as a conference paper at ICLR 2026
Conversely, the benefits of early termination marginal when paired with parallel decoding, as that method already minimizes post-<eos> redundancy by ummasking each remaining block in a single step. Similarly, the technique is less impactful for Dream models, which tend to fully utilize their token budget, as evidenced by E/fmax Tbl. Therefore, to establish a more robust point of comparison, we applied this optimization exclusively to the LLaDD van la baseline in long-sequence analysis, as discussed in Appx. C.2.1
C EXTENDED EXPERIMENTAL RESULTS
This section provides supplementary experimental results, including detailed performance break- down for the LLaDA-1.5 model and an extended analysis of long- sequence generation on the Dream- Base model.
C.I FULL PERFORMANCE RESULTS FOR LLADA-1.5
We present a comprehensive analysis of DPad's performance on the LLaDA-1.5 model across all benchmarks. The full results summarized in Tbl.
Table 6: Performance of LLaDA-1.5 with DPad on four benchmarks.
LLaDA-1.5
Efficiency
Accuracy (%)
Benchmark
Method
Latency(s)
TPS
E/mmax
Flexible
Strictf
Vanilla
27.61
1.00x
7.77
1.00x
256
80.59
61.87
GSM8K
+DPad
18.26
1,51x
8.56
1.10x
156 / 256
80.14
78.47
4-shot
+Parallel (Fast-dLLM)
8.06
3,42x
26.61
3.43x
215 / 256
80.82
62.62
+Parallel+DPad
6.23
4.43x
25.23
3.25x
157 / 256
80.89
78.92
Vanilla
25.12
1.00x
8.67
1.00x
218 / 256
33.52
32.72
MATH s-shot
+DPad
20.63
1,22x
9.48
1.09x
196 / 256
34.08
37.00
+Parallel (Fast-dLLM)
9.48
2.65x
22.96
2.65x
218 256
33.60
32.92
+Parallel+DPad
8.57
2.93x
22.76
2.63x
195 / 512
32.92
35.96
Vanilla
34.80
1.00x
3.16
1.00x
110 512
40.85
HumanEval
+DPad
11.55
3.01x
7.19
2.28x
83 / 512
44.51
O-shor
+Parallel (Fast-dLLM)
11.16
3.12x
9.80
3.10x
109 512
39.63
PPar
5.26
6.61x
15.64
4.95x
/12
39.63
Vanilla
62.34
1.00x
1.02
1.00x
63 / 512
38.20
MBPP
+DPad
14.95
4.17x
4.33
4.26x
65 /12
39.80
3-shot
+Parallel (Fast-dLLM)
5.47
11.39x
11.62
11.44x
61/512
38.60
+Parallel+DPad
4.41
14.14x
14.83
14.60x
/1212
41.60
Efficiency. On LLaDA-1.5, DPad achieves significant latency reductions, with speedups ranging from 1.22 to 4.17x over the vanilla baseline. When combined with parallel decoding, the overall speedup reaches up to 14.14x. These gains are driven by DPad's core mechanisms, including the generation of more concise outputs. For instance, DPad reduced the average generation length on MATH by approximately 10% and on GSM8K by 27%, with the majority of the speedup still attributable to reduced suffix computation. As shown in Figure [] the benefits of DPad scale dramatically with sequence length, achieving a 61.39 combined speedup in a 1024-token setting.
Accuracy. Consistent with results on LLaDA Instrut,, DPad improves both efficiency and accu- racy for LLaDA-1.5, demonstrating that speed enhancements do not compromise quality. The most substantial improvements are observed in strict-match accuracy, where DPad helps the model better adhere to the formatting of in-context examples. On GSM8K, DPad improves the strict- match score from 61.87% to 78.47%. This highlights DPad's ss ability to focus the model on high nigh-quality signals from the prompt by pruning noisy, low-entropy suffix tokens.
22

---

Published as a conference paper at ICLR 2026
C.2 FULL LONG-SEQUENCE GENERATION ANALYSIS
C.2.1 LLADA-1.5
The 20.3x speedup of DPad over the vanilla baseline reported in Tbl. warrants further analysis, as it is disproportionately large compared to the 5.2x speedups over the +Par. and +Par +PC. baselines. Upon investigation, we find this discrepancy is primarily due to the vanilla model's efficiency: it often produces a concise answer (around 256 tokens) but then continues to generate redundant <eos> tokens one by one to fill the 1024-token budget, process that incurs significant latency. In contrast, our DPad implementation eliminates this specific inefficiency by incorporating the early termination technique discussed in Appx. B
To disentangle the individual contributions of suffix dropout and early termination, introduce stronger baseline by augmenting the vanilla model with an early termination mechanism (-Early Te Termination) As shown in Fig. IT] analysis reveals that 4.8x speedup is attributable to early termination alone. Nevertheless, even against this stronger baseline, suffix dropout contributes additional 4.2x speedup, demonstrating its significant and independent impact efficiency.
LLaDA
1.5STPS
Fiexible Match (FM,
%..
Eary Temination
8x
26 4s, 7
FM: 78.17, SM: 48.98
DPd
20.3%
6.28s, 18.4TPS
FM: 78.77, SM: 74.07
Paralel-Only
11.7s, 18.9TPS
FM: 78.77, SM: 49.43
DPal
52x
2,28s, 51.4TPS
FM: 79.38, 8M: 74.22
DPal
5.2x
2.07s, 58.5TPS
FM: 77.10, SM: 70.06
61.39x
Figure 11: Latency comparison LLaDA-1.5 with GSM8K (1024 tokens, 1-shot).
C.2.2 DREAM
To further validate the scalability our approach, we conducted an extended analysis on the Dream- Base model using the HumanEval benchmark. We evaluated the same set of acceleration strategies as in the main paper's s analysis. The results, shown in Figure 12 confirm that the benefits of DPad generalize across different models and tasks.
Dreem
100x
84.46
Acc (16): 51.22
DPad
9.1x
10.3e
Acc:
Parallei-Only
31.8a
Acc: 51.
Gen. Length: 1024
DPal
8.2x
3.9a
Ace: 53.05
DPwl
8.7x
3.10
Acc: 54.2
30.58x
Dream
1.0x
342.22
(N)
DPal
17.1x 1x
20.0s
Acc: 49.39
Gen. Length: 2048
DPwl
15.8x
4.7s
Acc: 52
Parallal
PrefixCache
71.56
Acc: 57.90
DPad
20.3x
3.56
Acc: 54.88
97.32x
Figure 12: DPad performance on long-sequence generation with Dream-Base. Latency and accuracy comparison on the HumanEval benchmark for generation lengths 1024 and 2048 to- kens (0-shot). DPad's speedup scales significantly with sequence length, achieving up to 97.32x speedup when combined with other optimizations 2048 tokens.
Speedup. We observe a similar scaling trend on Dream-Bass Base as on LLaDA-1.5. DPad alone ac- celerates inference by 9.1 for 1024-token sequences and 17.1x for 2048-token sequences. When
23

---

Published as a conference paper at ICLR 2026
combined with Fast-dLLM, the benefits multiplicative, achieving 30.58x speedup 1024 to- kens and an impressive 9 speedup at 2048 tokens. These results reinforce that suffix dropout and parallel decoding address orthogonal bottlenecks, yielding nearly two orders of magnitude im- provement in long-sequence generation.
Accuracy. On the 2048-token HumanEval task, we observed 7.32% accuracy degradation when applying DPad to the vanilla Dream model. This degradation is largely mitigated when DPad is combined with LLs We hypothesize that this isolated performance drop arises from com- plex interactions between our training-frre pruning strategy and the model's native Top-k sampling behavior, pointing to interesting direction for future investigation.
C.3 COMPARISON WITH SPARSE-DLLM
Recent works such as SLLrse-dLLM (Song et al.) 2025a) introduce token-pruning strategy for dLLMss inspired by the "compute-and-evict" mechanisms of autoregressive models. While both approaches aim to prune unimportant tokens, the core difference lies in when and how pruning oc- curs: Sparse-dLLM employs a reactive eviction method, whereas DPad utilizes a proactive dropout method.
C.3.1 OPERATIONAL PARADIGM AND MEMORY COMPLEXITY
Proactive vs. Reactive Pruning. Sparse-dLLL essentially transplants the conventional "compute- and-evict" strategy from autoregressive models (Song et al. (2025a) directly to the dLLM context. As reactive method, it must first compute the full attention matrix to identify salient tokens before evicting the unimportant ones. This generic approach incurs computational overhead and, crucially, requires high peak memory allocation to process the full sequence prior to eviction.
In contrast, DPad diverges from this adaptation by exploiting unique property intrinsic to dLLMs: the DLT Hypothesis (Sec. (.3). By using distance as a reliable proxy for utility before attention is computed, DPal renders pruned tokens "invisible" to the computational graph. This ensures zero runtime overhead for the pruning step itself, demonstrating a design tailored specifically for diffusion generation rather than transplanted from autoregressive paradigms.
Memory Scaling, Regarding memory complexity, since the Prefix- Prefix-Sparse strategy (Song et al, 2025a) is compatible with both methods and does not differentiate them, focus the handling of suffix tokens. Sparse-dLLM typically requires a retention ratio of ~ 50%[Song et al. 2025a) that scales linearly with the sequence length L, resulting in OL) complexity. Conversely, DPad employs a fixed sliding window with more aggressive retention ratio (20%-37.5% see Appx. D Consequently, DPad's memory footprint for the suffix is strictly bounded by the window size W. effectively achieving (11) complexity relative to the total sequence length. A summary of these differences provided in Tbl. 7
Table 7: Comparison of Operational Paradigm and Memory Complexity for Suffix Tokens.
Method
Pruning Paradigm
Memory Complexity
Sparsity
(Suffix)
Fast-dLLM
N/A (Dense)
100% O(L)
O(L)
Sparse-dLLM
Reactive (Post-Attention)
~50%
DPad (Ours)
Proactive (Pre-Attention)
0(1)
< 20% 37.5%
C.3.2 EMPIRICAL EVALUATION
To quantify the practical benefits of the proactive paradigm, we conducted a head-to-head com- parison 100 samples from the GSM8K benchmark using the LLaDA-1.5 model. We evaluated
24

---

Published as a conference paper at ICLR 2026
performance long-con cong-contett scenarios (max {2048, 4096)). To ensure fair comparison, both the Sparse-dLLM baseline and our DPad method are implemented on top the PrefixCache (PC) optimization (Wu et a
Efficiency and Accuracy. As shown in Tbl. 8 DPad demonstrates superior scaling for long se- quences:
Lateney:
DPad achieves 12.87 x speedup over Sparse-dLLL, at 2048 tokens. This ad- vantage expands 39.64 at 4096 tokens, highlighting the benefits of O(1) complexity
scaling.
Throughput (TPS): Similar to the observations in Sec. the speedup in terms of TPS is more moderate compared to end-to-end latency. However, DPad still surpasses Sparse- dLLM. While Sparse experiences throughput degradation at longer context lengths (max = 4096) due to the linear complexity of pre-eviction calculation, DPul maintains consistent throughput, achieving 1.30 speedup. This confirms that proactive pruning effectively alleviates the computational bottlenecks that constrain generation speed.
Accuracy: In both settings, DPad maintains comparable or superior accuracy (e.g., 84.00% vs. 79.00% at 2048 tokens), demonstrating that proactive pruning does not com- promise generation quality.
Table 8: Efficiency and accuracy evaluation of Sp vs. DPad using LLaDA-1.5 on 100 samples of GSM8K with different max generation length (maxx
LLaDA-1.5
Efficiency
Accuracy (%)
Context Length
Method
Latency(s)
Specdup
TPSt
Speedup
Flexible
Vanilla+PC--Spane-lLLLMM
83.30
1.00x
24.28
1.00x
79.00
Imax 2048
Vanilla+PC+DPaa
6.47
12.87x
24.42
1.01x
84.00
Phhilaa+PC-Spans-llLLLM Vamilla+PC+DPad
213.65
1.00x
19.17
1.00x
80.00
Imax 4096
5.39
39.64x
24.96
1.30x
79.00
Memory Stability. Fig- 13 illustrates the memory footprint throughout the generation process. policy, Sparse-dLLM exhibits a highly unstable active memory
Attributable to its profile, characterized by frequent and pronounced oscillations (Fig. 13a).
Crucially, this reactive pruning renders the apparent active memory savings "illusory" from a system allocation perspective. As shown in Fig. [3p, Sparse-dl LM demands the same Peak Memory allo- cation the dense Fast-dLLM baseline, because the system must reserve sufficient buffer capacity to compute the full attention matrix before eviction occurs. Consequently, the system were allo- cate the ostensibly "saved" memory to concurrent workloads, the primary process would encounter Off-OO-Memory (OOM) errors during the pre-eviction computation spikes.
In contrast, DPad operates proactively: it eliminates the need to cache pruned suffix tokens entirely. This results in strictly lower peak memory usage, yielding an approximate 50% reduction in memory resource requirements compared the baselines. As result, the memory footprint of DPad aligns closely with the stable growth pattern of standard autoregressive models, thereby facilitating efficient system-level memory allocation.
25

---

Published as a conference paper at ICLR 2026
(a) Active Memory Usage
Active Memory (% of Total)
80
60
40
Fast-dLLM
20
Sparse-dLLM
DPad
20
40
60
80
100
(b) Peak Memory Usage
100
Peak Memory (% of Total)
Fast-dLLM
60
Sparse-dLLM
DPad
40
20
0
0
20
40
60
80
100
Generation Progress (%)
Figure 13: Memory Usage comparison long-sequence generation. Fast-dLLM serves as the baseline (100%). To specifically isolate the impact of suffix token handling, neither Sparse-dLLM DPad utilizes the Prefix-Sparse technique in this comparison. (a) Active Memory: Sparse- dLLMM shows unstable oscillations due to reactive eviction. (b) Peak Memory: Sparse-dLLLM fails reduce peak requirements compared to Fast-dLL.M, whereas DPad achieves significant real-world memory savings.
D HYPERPARAMETER ANALYSIS
This section details the hyperparameter tuning process for Gaussian sampler. Since attention score distributions can vary across models and datasets, we tune hyperparameters on a small subset of data for each benchmark ensure optimal performance.
D.1 ABLATION STUDY ON GAUSSIAN HYPERPARAMETERS
Inspired by findings that dLLMs exhibit different behaviors on mathematical and code-generation tasks (Gong et al.). we tune the hyperparameters for our Gaussian Sampler separately for each domain. We perform grid search a. The parameter is used to control the overall retention density, which we define as the expected proportion of suffix tokens preserved by the Gaussian Sampler inside the sliding window.
two key parameters: the decay rate, k. and scale factor,
26

---

Published as a conference paper at ICLR 2026
Performance across k and Density
Metric
Accuracy Score
GSM8K (Flexible)
GSMBK (Strict)
Humaneval
Baseline
Chosen
40
2
2
3
25.0%
37.5%
50.0%
Density
Figure 14: Ablation study on hyperparameters i and a for Instrucct on subsets of GSM8K and Hvma The parameter a is mapped to the retention density shown on the x-axis. Each dashed line represents the baseline performance for the solid line of the same color and metric.
The results of this search for LLaDA-Instruct are presented in Figure [4 conducted on 50-sample subsets of GSM8K and HvmanEval. While using small subset for tuning may introduce some variance, the findings provide clear directional insights. Our method with Gaussian Dropout consis- tently outperforms the baseline in nearly all configurations, with single, minor exception for the GSM8K flexible-match accuracy score at = and density of 25.0%. The suboptimal perfor- mance at k is expected, at this value the Gaussian curve is relatively flat across the sampling window (as shown in Fig. 15 causing the sampling to degenerate into a near-unifone distribution.
Gaussian PDFs with Different Parameters
1.0-
Dersity-23.2
Demily
Demily
Demity
0.8
Densltu
Demityy37.59
Demily
Probability Density
Demily-1755
Demity-50.00
0.6
Denslty
De
0.4
0.0
0
50
100
150
200
250
Token
Figure 15: Different curvers Gaussian pdfs with different parameters.
Focusing the more effective range of & E [2,4], we identify distinct optimal settings for each do- main. For the mathematical reasoning task (GSM8K), density of 25.0% provides the best balance of accuracy and efficiency. Although some settings with 50.0% density achieve slightly higher strict-match score, they do so by preserving twice as many suffix tokens, which significantly un- dermines acceleration. We therefore select k 4.0 and density of 25.0% (from a 2.0) as the optimal configuration. For code generation (HumanEval), a configuration of k = 3.0 and density of 37.5% (from 2.3) yields the best performance.
Based on these findings, we adopt these hyperparameters for all subsequent math and code bench- marks for L.L.DA-Instruct with similar tuning for other models, listed in Tbl. 9
27

---

Published as a conference paper at ICLR 2026
D.2 HYPERPARAMETER SETTINGS
The primary hyperparameters for the Gaussian sampler are the decay rate factor l and the magnitude scalar a, which controls the retention density. The optimal values used our main experiments (Section 4.2) are summarized in Tbl.
Table 9: The hyperparameters for Gaussian Sampler used in main experiments in Sec. 4.2
LLaDA-Instruct
LLaDA-1.5
Dream- Base
Task
k:
Density
Window
k:
Density
Window
Density
Window
GSM8K
4.0
2.0
25.0%
256
3.0
1.6
25.0%
256
4.0
1.6
20.0%
256
Math
4.0
2.0
25.0%
256
3.0
1.6
25.0%
256
4.0
1.6
20.0%
128
HumanEval
3.0 2.3
37.5%
512
3.0
1.6
25.0%
128
3.0
2.3
37.5%
128
MBPP
3.0
2.3
37.5%
128
3.0
1.6
25.0%
512
3.0
1.6
25.0%
128
D.3 STOCHASTIC VS. DETERMINISTIC SUFFIX DROPOUT
To validate the design choice of using stochastic (randomized) distance-decay strategy for DPad, we compared it against deterministic masking strategies. The goal was to determine the model requires dynamic "lottery ticket" search (as hypothesized in Sec. 3.3 if static selection of tokens would suffice.
We compared DPad against two deterministic baselines on LLaDA-I.5GGSMSK (lmar 256):
Fixed Nearby: Deterministically retains only the suffix tokens adjacent to the current block.
Deterministically retains only the furthest suffix block. DPad (Ours): Dynamically resamples the dropout mask at each step.
Fixed Distant:
Table 10: Stochastic vs. deterministic dropout on LLaDA-1
Method
Flexible
Strict
Baseline
79.68
35.94
Fixed Nearby
71.65
55.19
Fixed Distant
28.51
21.83
DPad
79.98
68.84
Results and Analysis. As shown in Table 10, deterministic strategies significantly degrade perfor- mance. Fixed Distant causes catastrophic collapse (Flexible accuracy drops 18)51%) proving local context is non-negotiable. Crucially, Fixed Nearby also underperforms the baseline (71.65% vs 79.68%), indicating that a rigid cutoff creates permanent "blind spots" necessary distant informa- tion. In contrast, DPad achieves the highest performance across both metrics. Its stochastic resam- pling allows model to integrate information from the entire suffix range over time, effectively avoiding blind spots while preserving the noise-reduction benefits required for high strict-match accuracy.
E QUALITATIVE CASE STUDIES
To provide qualitative insight into DPad's impact, this section presents case studies illustrating its effect on in-context learning, format adherence, and generation conciseness.
28

---

Published as a conference paper at ICLR 2026
E.1 IMPROVED IN-CONTEXT LEARNING AND FORMAT ADHERENCE
key benefit of DPad is its ability to improve strict-match accuracy by helping the model faithfully replicate the reasoning format from few-shot examples. Fig. T6 provides clear example from the GSM8K benchmark. The baseline model produces the correct numerical answer but fails to follow the structured, step-by-step reasoning shown in the prompt. In contrast, DPad successfully mimies the desired format, leading to a successful strict match. This demonstrates that by pruning distracting, low-entropy suffix tokens, DPad allows the model to better focus on and learn from high-quality prompt examples.
To determiie haw much Jenst makes every day the fullow stepsi
Question c0t
Question
mane cass Juxet days
1. Calcutate the sotal sumber laid Janet's ducks per day.
Sak bakin her friend.
c9e 239
Total eggs seed day T Teggs
5. Saborart number rggs she eais for breekfast and
Quetion: Jusetss dacks by rggn per
egg-
reussining
per
three
every uurrinng and
her daily carsiag.
mullies friends
Junet use
sells remsinder the farner' maakee daily her per fresh dudk eas. liee much dillbes make every at farmer' market?
Rags left to scll day 16 -6-3-a 9- oga..
1. Junet's dacks 16 agps per day. friends. Therffers day.
willing
rocal of w
Sie sell eacl for $3,
3. The number cass left to sell 16. oggo ind, She each voa
Tood caraings per day 9 cags St/egs S18
daily comnlggs opp
market.
Therffer,, Janet makes S18 every day st the Barmers market.
LLaDA-InstruetCCSMBKK (256, 4-shot)
Wha LB (133 saken))
+DPad
telonss)
Baseline
Match
Strict-Match
Flexible Match
Match
Figure 16: Case study on format adherence from GSM8K. This example contrasts the output of the baseline ALaDA-Instruct model with the DPad-enhanced version. The bascline provides the correct answer (Flexible-Match) but fails to follow the prompt's reasoning structure (failed Strict- Match), DPad generates both a correct and cell-formatted response, passing both criteria.
E.2 PROMOTING CONCISE AND EFFICIENT GENERATION
We also present case study on the Dream model illustrate how DPad influences generation behavior. Unlike the L.LaDA models, Dream tends to exhaust its full token budget, resulting in ver- bose and inefficient outputs. DPad mitigates this by encouraging more concise and logically sound generations. Figure I7 shows that the baseline model generates a lengthy, 1021-token response that frute-forces solution. DPad, however, produces a correct and elegant 69-token solution that terminates naturally.
F DISCUSSION
F.I BEYOND A TRAINING-FREE METHOD: SUFFIX DROPOUT WITH SFT
While DPad demonstrates improved accuracy and computational efficiency across diverse bench- marks by strategically pruming the suffix space, we observe a performance degradation in very long- sequence generation, particularly at context length of 2048 tokens. We attribute this behavior to a distributional shift introduced by suffix dropout. During pre-training, the model is optimized to predict token distributions conditioned on full, continuous sequence of masked suffix tokens. In contrast, our distance-decay dropout forces the model at inference time to predict based on a much smaller, discontinuous set of masked suffix tokens. While moderate dropout helps the model focus on prompt semantics, such a large shift in the conditional distribution can hinder generalization, leading to degraded performance.
This training-inen inference distribution gap may be mitigated with supervised finetuning (SFT). In par- ticular, the attention lottery hypothesis offers useful perspective that can be incorporated into
29

---

Published as a conference paper at ICLR 2026
Your code here
modp(n: int, p: int);
"""Return n modulo p (be aware
of numeriis)
pyyhonn
modp(3.5)
def modp(n: int, p: int):
3
"*"Return 2 modulo of numerics).
m01p(1101, 101)
2 >>> modp(0, 101)
modp(3,
>>2 modp(1101, 101)
>>> modp(3. 11)
>>> modp(0, 101)
m00p(100, 101)
Question
>2> modp(3. 11)
...
modp(100, 101)
...
if return
if = 0: return if 1:
while n 1:
if return 2
if %
result (result + a //-
return 0
ifp - 3:
break
if 313:
result (result result) %
return result (69 tokens)
+DPad
(1021 tokens)
Baseline
Fream/HumanEvvs (1024, 0-shot)
Case study generation conciseness from Drean/HumanEvall A comparison of generated code for a modular arithmetic problem. The baseline model (1021 tokens) produces highly verbose, unrolled solution fill its token budget. In contrast, DPad (69 tokens) generates compact, efficient, and logically superior solution, demonstrating improved reasoning.
Figure 17:
training. We revise the learning objective as:
log po (xo' x (uuluoun)
Cprad (0) -.
(10)
SEC
where M denotes subset of masked suffix tokens sampled via distance-decay dropout at each training step, In represents the prompt tokens, Ip represents the generated prefix tokens,6 represents the model parameters, E [0, 1] is degree of masking of samples from the forward masking process (2ec.2.1). and the loss is computed over the current block tokens C. Using stochastic dropout masks rather than fixed deterministic pattern enhances robustness, the model does not overfit any single dropout scheme. This revised objective explicitly integrates the notion of an attention lottery into training, encouraging the model to avoid wasting capacity by writing redundant information into distant suffix tokens that are likely to be pruned at inference.
Looking further ahead, one could even incorporate distance-decay dropout directly into the pre- training phase, allowing the model to learn sparsity from scratch. Such pre-training with sparse suffix attention would naturally align training and inference conditions, and may yield even stronger efficiency-accuracy trade-offs.
F.2 COMPARISON TO SEMI-AUTOOFERESSIVV DIFFUSION AND BLOCK DIFFUSION
Block Diffusion models (Ariola et al. 2025 operate autoregressively at the block level, predicting each block conditioned only its predecessors. As result. their attention mechanism excludes access to subsequent blocks, in contrast to the directional attention employed by aumi-autoregressivve models.
30

---

Published as a conference paper at ICLR 2026
While this design is computationally efficient, its strictly forward-looking nature introduces two major limitations. First, it is vulnerable to the reversal curse (Berglund et al. 2024), since it cannot capture oongeragee (beyond single block) bidirectional dependencies that crucial for tasks such as code generation, which often require iterative back-and-forth refinement (Gong et al.). Second, by discarding the suffix, forfeits the ability thereby losing an important medium for contextual organization.
suffix tokens scratchpad (see Sec. 3.1)
Our proposed distance-decay dropout interpolates between these paradigms. It retains suffix tokens to function as scratchpad, while at the same time preserving the model's bidirectional learning ability. allowing information be both written into and retrieved from the suffix scratchpad as needed.
G LLM USAGE DISCLOSURE
LLMs were used for (i) language editing and (ii) limited code assistance in code writing and debug- ging. All research code implementing our method and experimental decisions were written and validated by the authors. We independently verified the correctness and reproducibility of the result.
31


Reviews:
[{"review_id": "WVG5Wz0wae", "reviewer": "Reviewer_5i78", "summary": "The paper proposes an inference-strategy to reduce generation cost in MDLMs by using summaries of the suffix tokens (future tokens that haven't been decoded yet). The authors discuss how the suffix tokens's latent representations can only serve as a per-layer \"memory\", which can be thought of as a scratchpad, and empirically find that there is a lot of redundancy in the suffix tokens and the attention scores decay rapidly with distance from the current block index, with the occasional spikes. These findings motivate a summarization sampler that   keeps track of close tokens from the suffix. Further, the spikes are found to be position-insensitive where removing that token position moves the spike to a closer token. Thus motivated, the window-based gaussian sampling for suffix dropout then leads to significant gains in speed.", "questions": "- Why use a random selection for masking? What is the value of the \"dropout\" part of the suffix dropout? Why not use a simple deterministic procedure ?\n- I'm not sure I buy the lottery ticket connection. In the original \"lottery tickets\" paper, one needed to prune to get to the ticket. Your procedure is not weight or input-dependent and that only tells me  that the \"memory\" in the suffix tokens is not position-dependent, which is supported by your experiment with the locations of the spikes. So any suffix positions you give the transformer get used for the same purpose. One could check this: what happens if you have other sampling strategies, such as removing some of the closer suffix tokens and deterministically keeping moderately distant ones? Would the performance drop significantly?\n- Why not train with the proposed restriction or other sliding window restrictions ? That also should get you similar gains and should be possible with the flex attention library?\n- Interestingly, the above check  could  test the \"lottery ticket circuits\" hypothesis. If the explanation for the mechanism is \"any suffix tokens will get used\", then training with a limited windows will recover the same performance or be better. If lottery tickets are the actual explanation, the artificial restriction would degrade performance. I understand that this may be expensive, but it seems like a direct test. So either a small experiment would suffice or the authors can convince me that this is not a good test or clarify what I'm missing.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Clean analysis of what happens with suffix tokens in block-wise autoregressive generation with MDLMs.\n- Well-motivated method for reduce unnecessary computation that can be applied at inference-time without changes to the model.\n- strong improvements in efficiency and good ablations showing the gaussian sampler which reflects the decay in attention scores actually helps.", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SIWGKgFMhl", "reviewer": "Reviewer_9jVz", "summary": "The paper targets the core inefficiency of diffusion LLMs (dLLMs): at each denoising step they compute attention over the entire suffix (future) region even though only a few tokens are unmasked/kept. The authors argue these suffix tokens mainly serve as a transient memory that aggregates context and is then read back by the current block, and show attention/entropy patterns that decay with distance. Building on this, they propose DPad, a training-free inference strategy that (1) caps the suffix via a sliding window, and (2) applies distance-decay dropout to progressively prune farther suffix tokens before attention is computed. DPad composes with existing accelerations (parallel decoding, prefix/KV caching) and, on LLaDA and Dream, reports up to 61.4× end-to-end speedup in long-sequence settings while maintaining accuracy (notably large strict-match gains on GSM8K for LLaDA).", "questions": "1. When does DPad hurt? Any tasks with long-range cross-suffix dependencies (e.g., long code synthesis, theorem proofs) where aggressive distance decay reduces correctness or format adherence?\n\n2. Can W, k, a be selected online using low-overhead signals (e.g., per-step confidence/entropy) rather than fixed grids?\n\n3. With Fast-dLLM and dLLM-Cache, in what order should DPad be applied? Does pruning suffix tokens change cache-hit rates or destabilize delayed-update caches? Please explain the  possible interaction.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The suffix window + decay dropout is lightweight, does not touch weights, and composes with parallel decoding and caching; the large compounding speedups in long sequences are compelling. \n\n2. Attention/entropy analyses show strong distance decay and near-zero entropy far in the suffix; ablations identify a 64–128 token “critical window.” \n\n3. The empirical studies across tasks with both reasoning and code are conducted, and tables separate latency vs TPS have discussed why TPS can look smaller even when latency drops (shorter, cleaner generations). \n\n4. Long-sequence results are a highlight (e.g., LLaDA-1.5 GSM8K 1-shot, 1024-token cap: 20.3× from DPad alone; 61.39× with Fast-dLLM-style combos). \n\n5. The overall writing is clear and easy to follow.", "weaknesses": "1. Recent training-free accelerations for dLLMs cover Block-dLLM, Fast-dLLM (dLLM cache), Sparse-dLLM, etc. The paper cites some of them but does not always compare directly, especially on matched long-sequence regimes and memory usage. \n\n2. The “distance-decay” sparsity concept parallels existing efforts in distance-biased or entropy-guided pruning methods. While DPad’s pre-attention pruning is a nice twist, the paper should sharpen how this differs theoretically/empirically from Sparse-dLLM (eviction guided by attention saliency over steps) and from Fast-dLLM/dLLM-Cache caches, ideally with head-to-head plots of FLOPs/latency/accuracy. I may have missed out or misunderstood some settings, please point out if the authors have done so.  \n\n3. The Gaussian sampler uses window size W, decay k, and magnitude a tuned per task/domain. Please provide broader sweeps and robust defaults, and test out-of-domain transfer (e.g., from code→math without retuning).\n\n4. Benefits are strongest on LLaDA while Dream shows smaller or mixed gains—this dependence deserves deeper analysis and maybe a diagnostic to decide when to enable DPad. However, the analysis regarding this point is not  elaborated in the paper. \n\n5. Effective tokens × layers attended and wall-time per decoded token may be helpful to include. Latency alone can hide GPU-utilization effects the paper itself notes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dpHBferIzf", "reviewer": "Reviewer_4aFo", "summary": "This paper introduces DPad, a training-free inference acceleration method for diffusion language models (dLLMs). The authors first identify and formalize the *Scratchpad* mechanism, showing that suffix tokens function as a dynamic cross-layer reservoir guiding the denoising process. Through systematic analysis, they reveal three key properties of suffix tokens—inherent sparsity, distance-decay attention, and position insensitivity—that cause major inefficiencies in existing dLLMs. Building on these insights, the paper proposes DPad, which applies distance-decay dropout to prune redundant suffix tokens before attention computation, effectively reducing computation while maintaining accuracy. Moreover, DPad can be combined with existing acceleration methods such as parallel decoding and prefix caching.", "questions": "1. Could you provide evidence (e.g., masking or ablation) to verify whether the scratchpad mechanism in section 3.1 truly functions as an essential memory component?\n2. How does low prediction entropy in section 3.2 directly imply token redundancy, and are the attention decay patterns consistent across all layers?\n3. How can the local *attention shifting* in Figure 4 be reconciled with the claim that suffix information is globally position-insensitive, and does this phenomenon generalize beyond LLaDA?\n4. What explains the exceptionally large accuracy gain on MBPP in Table 2, and does this improvement remain under shorter generation lengths (e.g., 256 tokens)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper conducts a detailed analysis of suffix tokens in diffusion language models from a sequence-level perspective. It introduces a novel viewpoint for understanding suffix tokens, identifies three key properties.\n2. The proposed **DPad** method is training-free; by applying distance-decay-style suffix dropout during inference, it significantly reduces computational complexity while maintaining model accuracy.", "weaknesses": "1. Section 3.1 formally introduces the *scratchpad mechanism* and explains, from the perspective of attention-based information flow, that *‘the suffix serves as temporary memory to assist the ongoing denoising process’.* While this section highlights the potential ‘*Attention Connection’* role of suffix tokens, it does not evaluate how critical this effect is to model inference. No intervention experiments (e.g., masking parts of the suffix and observing changes in hidden states) are provided to further verify whether the scratchpad truly functions as an essential memory component.\n2. Section 3.2 then intuitively suggests that *‘the scratchpad intuition does not require populating all suffix tokens.* To support this, the paper presents entropy-based analyses claiming that suffix tokens *‘carry little new information’.* However, the logical transition from Section 3.1 to 3.2 feels abrupt — low entropy does not necessarily imply uselessness*.* It only reflects prediction certainty, not whether these deterministic tokens still play structural roles during generation (e.g., alignment, formatting, or mask boundary control). In addition, Figure 3 reports attention decay but only for the final layer; it would be more convincing to include layer-wise attention distributions to examine whether similar decay occurs in lower or middle layers. Overall, the claimed *inherent sparsity* of suffix tokens requires further clarification and empirical support.\n3. The proposed “Diffusion Lottery Ticket” hypothesis is intriguing and provides a theoretical grounding for DPad’s dropout design. However, there is an apparent inconsistency: Figure 4 shows that *“the model shifts its attention to nearby suffix tokens”* when a spike is removed, whereas later (and also in Section 1) the paper states that *“information carried by distant suffix tokens appears largely position-insensitive.”* These two claims seem contradictory—if attention shifts locally, how can the information be position-insensitive globally? Moreover, the observational experiments in Sections 3.2 and 3.3 are only conducted on the LLaDA model series. It would strengthen the argument to verify whether similar phenomena hold across other diffusion-based LLM architectures.\n4. In Table 2, the model shows an exceptionally large improvement on the MBPP benchmark after applying DPad—the accuracy increase substantially. Could the authors clarify why DPad benefits MBPP so much?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Dk1dYFYW1h", "reviewer": "Reviewer_36ZU", "summary": "This article proposes a novel and efficient diffusion model inference method called DPad (Diffusion via Dynamic Programming). This method transforms the sampling optimization problem in the diffusion process into a Dynamic Programming (DP) problem, significantly reducing the number of sampling steps while maintaining the quality of model generation. The author analyzed the optimal trajectory problem of diffusion sampling mathematically and proposed an efficient approximate solution algorithm, which enables the model to achieve comparable generation performance to traditional sampling in small steps. The effectiveness of the method was validated on multiple benchmark datasets, including image generation and conditional generation tasks, and compared with mainstream fast diffusion algorithms such as DDIM and DPM Solver, demonstrating superior performance and computational efficiency.", "questions": "1. Could the author further explain the differences in the mathematical mechanisms between DPad and DPM-Solver? What is the relationship between the two in terms of their approximate objective functions?\n2. Can the DP approximation still maintain stability under high-dimensional non-Gaussian noise distributions?\n3. Have you considered extending it to video diffusion or 3D generation scenarios?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. Highly innovative in theory: It is the first systematically modeled as a dynamic programming problem for the optimization of the diffusion process;\n2. Elegant algorithm design: DPad has approximately linear complexity and significantly improved inference efficiency;\n3. Sufficient experimentation: Its effectiveness has been verified on various tasks and datasets;\n4. Strong compatibility with existing algorithms: It can be embedded as a plug-in module into other diffusion frameworks;\n5. Good interpretability: The DP framework clearly demonstrates the sampling path optimization logic.", "weaknesses": "1. Lack of ablation analysis: The independent contributions of each module (such as heuristic search and value function approximation) were not systematically explored; \n2. Limited experimental scope: Only image generation tasks were covered, without text or cross-modal diffusion models; \n3. The paper is slightly biased towards algorithmic engineering: The discussion of convergence and lower bounds of complexity in the theoretical part is still insufficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)