DPad: Efficient Diffusion Language Models with Suffix Dropout

## Abstract

Abstract Diffusion-based Large Language Models (dLLMs) parallelize text generation by framing decoding as a denoising process, but suffer from high computational overhead since they predict all future suffix tokens at each step while retaining only a small fraction.
We propose Diffusion Scratchpad ( DPad ), a training-free method that restricts attention to a small set of nearby suffix tokens, preserving fidelity while eliminating redundancy. DPad integrates two strategies: (i) a sliding window , which maintains a fixed-length suffix window, and (ii) distance-decay dropout , which deterministically removes distant suffix tokens before attention computation.
This simple design is compatible with existing optimizations such as prefix caching and can be implemented with only a few lines of code.
Comprehensive evaluations across multiple benchmarks on LLaDA-1.5 and Dream models demonstrate that DPad delivers up to 61.4 × \mathbf{61.4\times} speedup over vanilla dLLMs while maintaining comparable accuracy, highlighting its potential for efficient and scalable long-sequence inference.
Our code is available at https://github.com/Crys-Chen/DPad .

## 1 Introduction

Large Language Models (LLMs) have become foundational in numerous applications [31, 10, 7, 24], yet their deployment is often hindered by inference latency.
As shown in Fig. [1](https://arxiv.org/html/2508.14148v2#S1.F1) (a), the predominant autoregressive framework generates text one token at a time [25, 26], imposing a sequential constraint that limits speed and scalability [13].
This has driven interest toward parallel decoding strategies.

Diffusion-based Large Language Models [19, 3, 21, 27, 17] (dLLMs) offer a promising alternative by eliminating sequential dependencies.
Formulating text generation as a parallel denoising process, they can predict entire sequences or generate text block-wise (i.e., semi-autoregressively) [23, 36], as in Fig. [1](https://arxiv.org/html/2508.14148v2#S1.F1) (b).
However, this parallelism often incurs high computational cost [22]: at each step, predictions for *all* future (suffix) tokens are computed, though only a small fraction are retained.
Consequently, although dLLMs can generate multiple tokens in parallel, the resulting throughput gains are undermined by a disproportionate increase in computation, posing a key bottleneck to their widespread adoption [29].

Figure: Figure 1: Comparison of (a) autoregressive LLMs, (b) block-wise diffusion LLMs, and (c) our DPad. DPad restricts suffix attention via: (i)Sliding Window: fixed-length suffix window; (ii) Distance-decay Dropout: removes distant suffix tokens without computing attention scores.
Refer to caption: x1.png

To further understand this inefficiency, we analyze the role of suffix tokens under the block-wise masking mechanism in dLLMs. Interestingly, these suffix tokens contribute no direct semantic information to the attention; instead, they function as an information reservoir that collects signals from the already decoded prefix tokens, much like a scratchpad.
Through multiple Transformer layers, this scratchpad can guide the diffusion process to produce a more organized and contextually consistent current block.
Yet, most suffix tokens are redundant and low-entropy, often forming long runs of similar content.
This redundancy becomes more severe as the distance to the current block increases, with attention scores for suffix tokens dropping sharply from near to far positions.
Such redundancy not only imposes unnecessary computational overhead, but can also degrade the fidelity of the generated content.

Based on the above insights, we propose the Diffusion Scratchpad (DPad), which attends forward only to a small number of near suffix tokens, as in Fig. [1](https://arxiv.org/html/2508.14148v2#S1.F1) (c).
It uses two suffix drop strategies: sliding window and distance-decay dropout.
For the sliding window, inspired by prefix KV-cache optimizations [5, 35], we extend the idea to the suffix.
Here, the suffix window has a fixed length and moves forward along with the current block, retaining only a limited number of nearby suffix tokens.
In contrast to vanilla dLLMs, where suffix computation increases with the generation sequence length, our design keeps the cost bounded and reduces suffix-related computation by one dimension.

For the distance-decay dropout, suffix tokens are removed according to their distance from the current block: the farther they are, the higher the dropout ratio, until all tokens beyond the window are omitted.
Unlike conventional attention-score pruning [32, 18, 28], which first computes attention scores and then prunes based on their magnitude, DPad predetermines a distance-decay sparse pattern for suffix tokens *prior to* model execution and eliminates them in a single operation at the beginning of each step.

Inspired by the Lottery Ticket Hypothesis (LTH) [11], which posits that properly pruned sub-networks with their original initialization can match the performance of dense networks after training, we observe that the strong generalization ability of dLLMs enables the *training-free* construction of “winning tickets” for suffix tokens during inference.
Although this predetermined dropout strategy may appear intrusive, it consistently achieves strong empirical performance, with only a small subset of suffix tokens required to maintain accuracy comparable to vanilla dLLMs.
This further suggests that sparsity is an inherent property of suffix attention.
Additionally, the method is extremely simple to deploy, requiring only a few lines of code to implement.

To illustrate the intuition behind DPad, we liken it to a real scratchpad used when writing a book, as shown in Fig. [1](https://arxiv.org/html/2508.14148v2#S1.F1) (c, right).
For the current chapter (i.e., block), the writer (dLLM) devotes the most attention, revising it multiple times, akin to denoising within a diffusion block.
The upcoming chapter receives focused drafts for consistency, while much later chapters contain only brief outlines.
Naturally, the “writer” should not, and indeed must not, fill all future chapters (all suffix tokens) with low-entropy, repetitive content merely to satisfy the fixed sequence length constraints of current dLLMs.
Such uncontrolled filling distracts the “writer’s” attention and wastes storage and computation, making it neither sustainable nor scalable.

Finally, we emphasize that DPad enables efficient and compact generation without enforcing a fixed sequence length.
Unlike conventional dLLMs, where mandatory sequence length and full suffix attention introduce redundancy and degrade output quality [36, 23], DPad produces concise results with minimal overhead.
It is entirely training-free, saving substantial memory and computation, and achieves stable acceleration even at shorter generation sequence lengths (e.g., $256$ and $512$), outperforming both vanilla and optimized variants such as Fast-dLLM [34].
DPad is also compatible with existing optimizations [34, 20, 33], including prefix caching [34], dual caching [34], and dLLM-Cache [20].
It further scales naturally to long sequences: when combined with parallel generation and prefix caching, it achieves a $\mathbf{30.58\times}$ speedup on Dream/HumanEval ($1024$, 0-shot) and a $\mathbf{61.39\times}$ speedup on LLaDA-1.5/GSM8K ($1024$, 1-shot) over vanilla, while maintaining comparable model accuracy.
These results highlight the strong potential of DPad for long-sequence inference and demonstrate its ability to unlock new efficiency frontiers in dLLMs.
Although dLLMs remain in an early exploratory stage, with many researchers striving to assemble the final puzzle, we believe that DPad represents one of the key components in completing the whole picture and paves the way toward practical, scalable diffusion-based language modeling.

## 2 Preliminary

### 2.1 Foundational Principles of Diffusion Large Language Models (dLLM)

Conventional Autoregressive Models (ARMs) define the joint probability of a generated sequence $x=(x_{1},\dots,x_{L})$ by factoring it into a product of conditional probabilities in a fixed sequential order [31]. The model predicts the next token $x_{i}$ based on the conditional probability of the preceding tokens $x_{1},\dots,x_{i-1}$:

$$ $p_{\theta}(x)=p_{\theta}(x_{1})\prod_{i=2}^{L}p_{\theta}(x_{i}|x_{1},\dots,x_{i-1})$ (1) $$

In contrast, Diffusion Language Models (dLLMs) leverage a non-autoregressive process  [3, 27, 21]. During training, the model learns to denoise a corrupted sample. This involves two processes:

##### Forward Masking Process

This process systematically replaces a proportion of tokens in a clean text sequence $x_{0}$ with a special [MASK] token  [22, 23], similar to applying noise in a conventional diffusion model  [16]. The replacement is governed by a masking schedule where $t\in[0,1]$ represents the masked level. For a given $t$, the probability of an individual token $x_{0}^{i}$ being masked is:

$$ $q(x_{t}^{i}|x_{0}^{i})=\begin{cases}1-t,&\text{if }x_{t}^{i}=x_{0}^{i}\\ t,&\text{if }x_{t}^{i}=\text{{[MASK]}}\end{cases}$ (2) $$

##### Reverse Unmasking Process

The model $p_{\theta}$ is trained to predict the original tokens given the partially masked sequence $x_{t}$, thereby learning to approximate the true unmasking probability $q(x_{0}|x_{t})$  [3, 21]. This is achieved by minimizing the negative log-likelihood over the masked tokens [27, 24]:

$$ $\mathcal{L}(\theta)=-\mathbb{E}_{t,x_{0},x_{t}}\left[\sum_{i:x_{t}^{i}=\text{{[MASK]}}}\log p_{\theta}(x_{0}^{i}|x_{t})\right]$ (3) $$

This process makes use of a Transformer architecture to learn the distribution $p_{\theta}(x_{0}|x_{t})$. During training, bidirectional multi-head attention on all tokens within a sequence—masked and unmasked tokens alike—is employed to predict the probability distribution for each masked token.

### 2.2 Inference and Sampling in dLLM

The inference process begins by initializing a sequence $y_{0}$ through the concatenation of a prompt $r$ and $L$ mask tokens, denoted as $y_{0}=r\circ(\texttt{[MASK]})_{i=1}^{L}$. Let $M_{s}$ be the set of indices corresponding to masked tokens at step $s$; initially, $M_{0}=\{|r|+1,\dots,|r|+L\}$. The model then iteratively refines this sequence over steps $s=1,2,\dots,S$. At each step $s$, the model $p_{\theta}$ computes a probability distribution $p_{\theta}(y_{i}|y_{s-1})$ for every masked position $i\in M_{s-1}$. From these distributions, the most likely token predictions $\hat{y}_{i}$ and their corresponding confidence scores $c_{i}$ are determined:

$$ $\hat{y}_{i}=\underset{v\in V}{\text{argmax}}\,p_{\theta}(y_{i}=v|y_{s-1})\quad\text{and}\quad c_{i}=p_{\theta}(y_{i}=\hat{y}_{i}|y_{s-1}).$ (4) $$

where $V$ is model’s vocabulary.
A scheduling function $\mathcal{G}(s,S)$ determines the number of tokens, $k_{s}$, to unmask. The set of indices to update, $U_{s}$, is chosen by selecting the $k_{s}$ positions from $M_{s-1}$ with the highest confidence scores. The new sequence $y_{s}$ is then formed by updating these positions:

$$ $(y_{s})_{i}=\begin{cases}\hat{y}_{i}&\text{if }i\in U_{s}\\ (y_{s-1})_{i}&\text{otherwise}\end{cases}$ (5) $$

The set of masked indices is updated via $M_{s}=M_{s-1}\setminus U_{s}$. This iterative refinement continues until $M_{s}=\emptyset$.

### 2.3 Related work on dLLM acceleration

While early work in dLLMs employed Top-$k$ sampling to unmask a fixed number of high-confidence tokens at each step, later methods introduced more dynamic, confidence-aware techniques [34, 33] that greedily unmask all tokens above a threshold, thereby alleviating the generation bottleneck.
Another line of research focuses on cache management.
Because attention in dLLMs is bi-directional, conventional KV caching is not directly applicable.
To address this, several optimizations [34, 20] exploit the observation that prefix tokens and distant suffix tokens exhibit little variation in their attention values across inference steps, allowing them to be reused and reducing expensive recomputation.
A more fine-grained approach [28] further improves efficiency by dynamically evicting cache entries based on attention scores, leveraging the persistence of attention patterns across steps.

## 3 Method

In this section, we begin by formalizing the *Scratchpad* mechanism in dLLMs, which underpins the role of suffix tokens as an information reservoir.
We then introduce *Suffix Dropout*, which combines a fixed-length sliding window that moves with the current block to bound the number of attended suffix tokens, with a distance-decay scheme that progressively prunes more distant ones.
Finally, we propose the *Diffusion Lottery Tickets* (DLT) hypothesis, which provides a principled lens for interpreting dropout in dLLMs.

Figure: Figure 2: Attention score maps illustrating the Scratchpad mechanism in dLLMs. The maps were generated by the LLaDA-1.5 model [37] on prompt and 512-token sequences from the GSM8K dataset [9]. The attention matrix is divided into $3\times 3$ blocks over prefix, current, and suffix. Blocks 7 and 8 collect information from the prefix and current into the suffix at layer $n$, while Block 6 feeds this stored information back to the current block at layer $(n\!+\!1)$. This write–store–read cycle makes suffix tokens a dynamic scratchpad for cross-layer contextual reuse.
Refer to caption: x2.png

### 3.1 Scratchpad Mechanism

We first revisit the role of suffix tokens in dLLMs.
Due to the masking structure, suffix tokens carry no direct semantic information but instead serve as an *information reservoir*, aggregating signals propagated from prefix tokens across multiple Transformer layers.
This latent memory, which we refer to as DPad, stabilizes the denoising process by providing contextual scaffolding for the current block.

As illustrated in Fig. [2](https://arxiv.org/html/2508.14148v2#S3.F2) and the attention score maps, the token sequence can be partitioned into three contiguous segments: Prefix indices $[0,c-1]$, current indices $[c,s-1]$, and suffix indices $[s,L-1]$.
The corresponding attention matrix is thus divided into $3\times 3$ blocks.
Among them, Blocks 6, 7, and 8 together define the scratchpad mechanism.

Considering only one head, at layer $n$, queries from prefix and current tokens attend to keys from the suffix region.
Formally, the global attention weights are defined as

$$ $A^{(n)}=\text{Softmax}\!\left(\frac{Q^{(n)}(K^{(n)})^{\top}}{\sqrt{d_{k}}}\right)\in\mathbb{R}^{L\times L}.$ (6) $$

We can partition $A^{(n)}$ into submatrices corresponding to prefix ($P=[0,c-1]$), current ($C=[c,s-1]$), and suffix ($S=[s,L-1]$).
In particular,

$$ $A^{(n)}_{S,P}=A^{(n)}[s:L,\,0:c],\qquad A^{(n)}_{S,C}=A^{(n)}[s:L,\,c:s],$ (7) $$

represent the attention weights from suffix queries to prefix and current keys, respectively (Blocks 7 and 8 in Fig. [2](https://arxiv.org/html/2508.14148v2#S3.F2)).
Multiplying these attention weights with the value matrix yields the actual outputs at suffix positions:

$$ $H^{(n)}_{S}=A^{(n)}_{S,P}V^{(n)}_{P}\;+\;A^{(n)}_{S,C}V^{(n)}_{C}\;+\;A^{(n)}_{S,S}V^{(n)}_{S}.$ (8) $$

Here, $H^{(n)}_{S}$ denotes the hidden representations of suffix tokens after attention.
This equation shows that suffix tokens integrate information from prefix and current regions, effectively serving as a *Scratchpad* that records contextual signals.
After this aggregation, the outputs $H^{(n)}$ are processed by the subsequent linear transformations (e.g., feed-forward layers and residual connections), which operate independently on each token.
Thus, information written into the suffix tokens will not interact with other regions until the next layer’s attention, where prefix and current queries can *read* back the stored information from DPad.
This establishes the complete write–store–read cycle across consecutive layers.
At layer $(n\!+\!1)$, this stored information can be retrieved by the current block through

$$ $A^{(n+1)}_{C,S}=A^{(n+1)}[c:s,\,s:L],$ (9) $$

which corresponds to current-to-suffix attention (Block 6 in Fig. [2](https://arxiv.org/html/2508.14148v2#S3.F2)).
This path enables the current block to reuse the information collected in the suffix at the previous layer.

Through this mechanism, the stored information in suffix tokens at layer $n$ can be retrieved by the prefix and current blocks at layer $(n\!+\!1)$.
However, in practice, the influence of the suffix on the prefix is negligible: Prefix tokens correspond to already generated content, and their representations can be refined directly from themselves and the current block, without requiring additional signals from the suffix.
Indeed, prior work [34] has shown that caching prefix states during inference barely impacts accuracy, implying that the suffix-to-prefix pathway is effectively redundant.
Therefore, the essential interaction of the scratchpad mechanism lies in the current-to-suffix direction (Block 6), where the suffix serves as temporary memory to assist the ongoing denoising process.

Therefore, we conjecture that the behavior of suffix tokens resembles a *residual connection* specialized for attention.
Instead of directly propagating representations, the model compresses high-dimensional signals from the prefix and current into the suffix and re-injects them into the current block at the next layer.
This is reminiscent of the *bottleneck residual block* in ResNets [14], where high-dimensional features are projected, propagated, and then expanded.
Similarly, the suffix acts as a scratchpad that compresses contextual information and delivers it forward for cross-layer reuse in dLLMs.

### 3.2 Suffix Dropout Strategies

Building on the scratchpad perspective, we analyze redundancy in suffix attention to motivate a structured dropout design.
At the final layer 31, Fig. [3](https://arxiv.org/html/2508.14148v2#S3.F3) shows a sharp distance-dependent decay: nearby suffix tokens dominate, consistent with general attention behavior and prior work on prefix sliding windows [5, 35].
Moreover, suffix attention is one to two orders of magnitude smaller than that of prefix or current, indicating that suffix representations are information-light and only a small subset is needed to serve as the scratchpad memory.

Based on these findings, we propose two complementary suffix drop strategies:
(a) fixed-length *Sliding Window* to retain only a bounded number of near-suffix tokens,
and (b) *Distance-decay Dropout* that progressively prunes distant suffix tokens with minimal impact on model quality,
as illustrated in Fig. [1](https://arxiv.org/html/2508.14148v2#S1.F1)(c).
Both mechanisms are efficiently realized through a Gaussian sampling process, which simultaneously enforces a bounded window and distance-dependent decay.

As a result, suffix attention only needs to focus on nearby tokens, making the suffix dropout window decoupled from the overall sequence length.
Unlike the vanilla setting, where the suffix grows with the generation sequence, our approach keeps it constant.
This yields a clear computational benefit: suffix dropout effectively reduces suffix-related complexity by one dimension.
Formally, let $L$ denote the suffix length and $B$ the block size.
In vanilla block-wise dLLMs, suffix computation requires $\tfrac{L}{B}$ steps, each involving $O(L)$ suffix token operations, whereas suffix dropout restricts this to a constant number of tokens, independent of $L$.

Figure: Figure 3: Analysis of the Suffix Drop strategy at the final layer (Layer 31), generated by the LLaDA-1.5 model [37] on GSM8K dataset [9] with a max length of 512. We collect attention weights from 100 samples across all heads, focusing on the current block queries ($A$[$c$:$s$, $c$-200:]). To align positions, we truncate $200$ tokens before $c$ as the prefix and align all current blocks at $c$. The plot shows the mean attention distribution over key indices (green curve), together with the min–max range (shaded area). Current and suffix boundaries are marked, showing that attention on far suffix tokens rapidly decays, motivating our distance-decay dropout design.
Refer to caption: x3.png

##### Gaussian Sampling.

Inspired by the smooth decay patterns observed in Fig. [3](https://arxiv.org/html/2508.14148v2#S3.F3), we adopt a Gaussian sampling strategy to implement distance-aware token dropout.
Formally, for a suffix token at distance $d$ from the suffix boundary, its retention probability $P(d)$ is defined by the right half of a standard normal distribution with effective window size $W$:

$$ $P(d)=a\cdot\frac{1}{\sigma\sqrt{2\pi}}\exp\!\left[-\tfrac{1}{2}\left(\tfrac{\tfrac{k\sigma}{W}\cdot d-\mu}{\sigma}\right)^{2}\right],\quad 0<d\leq W,$ (10) $$

where $\mu=0$, $\sigma=1$, and the suffix boundary is the center of the Gaussian distribution.
Two hyperparameters control the distribution along the $x$- and $y$-axes:
(1) $\tfrac{k\sigma}{W}$ maps the window size $W$ to $k\sigma$ (e.g., for $W=256$, setting $k=3$ ensures $d=256$ corresponds to $3\sigma$);
(2) $a$ scales the overall sampling magnitude vertically.
This formulation ensures that tokens farther from the boundary are retained with exponentially decreasing probability, implemented via rejection sampling.

This dropout strategy is not static but is dynamically re-applied in a block-wise way. As shown in Fig. [1](https://arxiv.org/html/2508.14148v2#S1.F1) (c), after a block is generated, all previously dropped-out tokens are restored. The Gaussian sampling is then invoked again on the updated set of suffix tokens before the next block is decoded. This dynamic resampling is crucial for preventing sampling bias and ensuring that no token is systematically ignored during the generation process.

##### Implementation.

Our method can be implemented in just a few lines of code:

Here, the Gaussian-based sampler gaussian_suffix_dropout selects a subset of suffix tokens according to distance-aware probabilities.
We then construct a pruned sequence x_pruned by indexing with keep_idx.
The rest of the model remains unchanged, except for a minor adjustment to the RoPE embeddings inside the attention module to ensure correct positional encoding under suffix dropout.

Besides, we also introduce an early termination mechanism to prevent inefficient computation after a <eos> token is generated, a common issue in fixed generation length strategies like LLaDA [23]. Our method performs a conditional check after each decoded block; if an <eos> token is found, we fill the remaining sequence with <eos> and immediately halt generation.

##### RoPE Adjustment.

Our suffix dropout mechanism requires only a minor adjustment to the Rotary Positional Embedding (RoPE) [30] to maintain correct positional information.
In standard RoPE, a token at absolute position $i$ is encoded using an angle $\theta_{i}=i\cdot\Delta$.
After dropout, however, only a sparse, non-contiguous subset of suffix indices $\mathcal{I}=\{i_{1},\dots,i_{m}\}$ is preserved.

To handle this, we ensure that each preserved token retains its original positional information.
Rather than using their re-indexed positions after dropout, we apply a mapping function $f(i_{k})$ that retrieves the original absolute position of the $k$-th preserved token.
The new angle is then computed as

$$ $\theta^{\prime}_{i_{k}}=f(i_{k})\cdot\Delta.$ (11) $$

Accordingly, the modified RoPE application becomes

$$ $\text{RoPE}^{\prime}(\mathbf{x}_{i_{k}},i_{k})=\text{RoPE}(\mathbf{x}_{i_{k}},f(i_{k})).$ (12) $$

This adjustment requires only a lightweight remapping inside the attention module and does not alter the functional form of RoPE, confirming that suffix dropout is nearly cost-free while preserving positional consistency.

Figure: Figure 4: (a) Average, maximum, and minimum attention scores of suffix tokens paid by current block tokens ($A$[$c$:$s$, $s$:]) across layers in LLaDA-1.5, showing overall decay with occasional spikes (e.g., d = 199, 298, 362). (b) After forcibly pruning these spike positions, attention shifts to nearby tokens, indicating that adjacent positions can absorb suffix information (e.g., pruning token 362 shifts the spike to token 359).
Refer to caption: x4.png

### 3.3 Diffusion Lottery Tickets Hypothesis

The analysis in Fig. [3](https://arxiv.org/html/2508.14148v2#S3.F3) not only reveals the overall decay of current-to-suffix attention, but also occasional sharp *spikes* in the maximum values.
These spikes cannot be predicted in advance and, in principle, pruning them could affect model accuracy.
To investigate this, we conduct an additional experiment shown in Fig. [4](https://arxiv.org/html/2508.14148v2#S3.F4).
We first run dLLM inference for one step, then forcibly prune the top 10 highest-attention suffix tokens (“spotlight” tokens) occurring beyond the first 128 positions (e.g., at distances 199, 298, and 362).
Surprisingly, pruning such distant suffix tokens, even those corresponding to large spikes, has little effect on model accuracy.
When these spikes are removed, the model shifts its attention to nearby suffix tokens, for example, the spotlight token at 362 in Fig. [4](https://arxiv.org/html/2508.14148v2#S3.F4) (a) is replaced by increased attention at its neighbor token 359 in Fig. [4](https://arxiv.org/html/2508.14148v2#S3.F4) (b), which effectively *absorbs* the lost information.
This behavior is consistent with the strong generalization ability of diffusion models and the fact that suffix tokens are initialized without semantic content: through DPad mechanism, suffix tokens can dynamically learn and store information during inference.
Consequently, information carried by distant suffix tokens appears largely *position-insensitive*, and pruning spike positions has almost no impact on final accuracy, as further confirmed in our evaluation experiments.

This leads to an important intuition: information collected by distant suffix tokens is insensitive to their exact positions.
When writing a book, one may use a scratchpad to record an event in the “middle chapters,” but it is not necessary to pin it down to precisely token 186; recording it at token 152 or 198 suffices.
This explains why suffix dropout can be applied *a priori*, without computing exact attention scores, and why it fundamentally differs from prefix cache pruning [28, 32]:
Prefix tokens carry dense, position-bound semantic information and thus cannot be arbitrarily discarded, whereas suffix tokens act as a flexible, low-rank memory buffer.

These observations resonate with the *Lottery Ticket Hypothesis (LTH)* [11], which posits that properly pruned sub-networks with their original initialization can match the performance of dense networks after training.
We extend this intuition to dLLMs and propose the Diffusion Lottery Tickets (DLT) hypothesis:
During inference, the suffix region contains redundant tokens, yet a sparse subset is sufficient to preserve semantic consistency and generation quality.
Through DPad mechanism, this subset can be adaptively reorganized into “winning tickets” within the forward pass.
In this view, suffix dropout becomes a *training-free lottery ticket search*, where Gaussian sampling selects a compact set of suffix tokens that carry the essential information for denoising in dLLMs.

##### Summary.

We showed that suffix tokens function as a scratchpad, enabling a high-dimensional residual pathway.
This insight motivates two lightweight strategies, Sliding Window and Distance-decay Dropout, together with the DLT hypothesis, which interprets suffix pruning as a training-free lottery ticket search.
These methods are simple, training-free, and easy to integrate.
Next, we evaluate the efficiency and accuracy of DPad-enhanced dLLMs across standard benchmarks.

## 4 Experiments

### 4.1 Experimental Setup

##### Models and Baselines.

All experiments are conducted on an NVIDIA A100 80GB GPU.
We evaluate DPad on a suite of representative open-source dLLMs: two variants of LLaDA (8B-Instruct and 1.5) [23, 37] and Dream-v0-Base-7B [36].
We compare against the following baselines:

- •
Vanilla: the unmodified LLaDA [23] and Dream [36] backbones, which compute attention over all tokens in the sequence, including prefix, current, and suffix tokens.
- •
+Parallel (Fast-dLLM): a parallel decoding strategy applied to the vanilla backbone, where tokens are unmasked whenever their confidence exceeds a predefined threshold [34], instead of the fixed top-$k$ unmasking used in Vanilla.
- •
+Prefix Cache: applying KV caching on prefix tokens, leveraging the observation that their key–value states remain nearly static across inference steps [34], thereby avoiding redundant computations.

When combining DPad with caching, we adopt a standard Prefix Cache.
Since our dropout already minimizes the computational cost of the suffix, the Dual Cache mechanism proposed in [34] is unnecessary.

The DPad strategy introduces three hyperparameters: a decay rate factor $k$, a magnitude scalar $a$, and a sliding window size.
We tuned these on small subsets of each benchmark (see Sec. [4.3.2](https://arxiv.org/html/2508.14148v2#S4.SS3.SSS2)).
Unless otherwise specified, we use a block size of 32 and set the confidence threshold for parallel decoding to 0.9.

##### Benchmarks and Metrics.

Our evaluation spans multiple domains.
For reasoning, we use GSM8K [9] and MATH [15];
for code generation, we use HumanEval [8] and MBPP [4].
We report two aspects of performance:
We report two aspects of performance:

- •
Accuracy: evaluated with task-specific metrics, including pass@1 for code generation and accuracy measures for reasoning (e.g., flexible-extract and strict-match accuracy on GSM8K). All accuracies are reported as percentages.
- •
Efficiency: measured by average inference latency per sample and Tokens Per Second (TPS). TPS is computed over the entire generated sequence until an <eos> token is produced.
We also use $a$ / $b$ to denote the generation (Gen.) length, where $a$ is the average real generation length and $b$ is the benchmark setting length.

### 4.2 Main Results

We evaluate the efficiency and accuracy of DPad on three models and four benchmarks, as reported in Tbl. [1](https://arxiv.org/html/2508.14148v2#S4.T1), Tbl. [2](https://arxiv.org/html/2508.14148v2#S4.T2), and Tbl. [3](https://arxiv.org/html/2508.14148v2#S4.T3).
We compare against both the vanilla LLaDA baseline [23] and the parallel decoding baseline Fast-dLLM [34].
Our results reveal three consistent trends:

- •
Latency: consistently improved across all settings.
- •
Throughput (TPS): shows higher variance, with fluctuations across benchmarks and models.
- •
Accuracy: Flexible Match remains comparable, while Strict Match achieves significant gains.

We analyze each of these aspects in detail below.

**Table 1: Performance of LLaDA-Instruct with DPad on four benchmarks.**
| LLaDA-Instruct |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Benchmark | Method | Efficiency | Accuracy (%) |  |  |  |  |  |
| Latency(s)↓ | TPS↑ | Gen. Length | Flexible↑ | Strict↑ |  |  |  |  |
| GSM8K<br>4-shot | Vanilla | 27.48 | 1.00× | 8.44 | 1.00× | 232 / 256 | 78.39 | 37.38 |
| +DPad | 18.35 | 1.50× | 8.76 | 1.04× | 161 / 256 | 78.54 | 63.84 |  |
| +Parallel (Fast-dLLM) | 8.55 | 3.21× | 27.14 | 3.22× | 232 / 256 | 78.54 | 38.67 |  |
| +Parallel+DPad | 6.64 | 4.14× | 24.25 | 2.87× | 161 / 256 | 79.76 | 64.97 |  |
| MATH<br>4-shot | Vanilla | 25.40 | 1.00× | 9.79 | 1.00× | 249 / 256 | 33.58 | 8.42 |
| +DPad | 21.61 | 1.18× | 9.75 | 1.00× | 211 / 256 | 33.42 | 28.04 |  |
| +Parallel (Fast-dLLM) | 9.91 | 2.56× | 25.09 | 2.56× | 249 / 256 | 33.40 | 8.76 |  |
| +Parallel+DPad | 9.20 | 2.76× | 22.93 | 2.34× | 211 / 256 | 33.30 | 27.98 |  |
| HumanEval<br>0-shot | Vanilla | 34.67 | 1.00× | 13.64 | 1.00× | 473 / 512 | 43.90 | – |
| +DPad | 27.41 | 1.26× | 15.96 | 1.17× | 438 / 512 | 47.56 | – |  |
| +Parallel (Fast-dLLM) | 11.48 | 3.02× | 41.40 | 3.04× | 475 / 512 | 43.29 | – |  |
| +Parallel+DPad | 9.14 | 3.79× | 47.86 | 3.51× | 438 / 512 | 46.34 | – |  |
| MBPP<br>3-shot | Vanilla | 62.11 | 1.00× | 4.82 | 1.00× | 299 / 512 | 15.00 | – |
| +DPad | 15.89 | 3.91× | 6.85 | 1.42× | 109 / 512 | 40.40 | – |  |
| +Parallel (Fast-dLLM) | 14.26 | 4.36× | 20.99 | 4.36× | 299 / 512 | 15.00 | – |  |
| +Parallel+DPad | 6.02 | 10.32× | 18.28 | 3.79× | 110 / 512 | 39.40 | – |  |

**Table 2: Performance of LLaDA-1.5 with DPad on four benchmarks.**
| LLaDA-1.5 |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Benchmark | Method | Efficiency | Accuracy (%) |  |  |  |  |  |
| Latency(s)↓ | TPS↑ | Gen. Length | Flexible↑ | Strict↑ |  |  |  |  |
| GSM8K<br>4-shot | Vanilla | 27.61 | 1.00× | 7.77 | 1.00× | 215 / 256 | 80.59 | 61.87 |
| +DPad | 18.26 | 1.51× | 8.56 | 1.10× | 156 / 256 | 80.14 | 78.47 |  |
| +Parallel (Fast-dLLM) | 8.06 | 3.42× | 26.61 | 3.43× | 215 / 256 | 80.82 | 62.62 |  |
| +Parallel+DPad | 6.23 | 4.43× | 25.23 | 3.25× | 157 / 256 | 80.89 | 78.92 |  |
| MATH<br>4-shot | Vanilla | 25.12 | 1.00× | 8.67 | 1.00× | 218 / 256 | 33.52 | 32.72 |
| +DPad | 20.63 | 1.22× | 9.48 | 1.09× | 196 / 256 | 34.08 | 37.00 |  |
| +Parallel (Fast-dLLM) | 9.48 | 2.65× | 22.96 | 2.65× | 218 / 256 | 33.60 | 32.92 |  |
| +Parallel+DPad | 8.57 | 2.93× | 22.76 | 2.63× | 195 / 512 | 32.92 | 35.96 |  |
| HumanEval<br>0-shot | Vanilla | 34.80 | 1.00× | 3.16 | 1.00× | 110 / 512 | 40.85 | – |
| +DPad | 11.55 | 3.01× | 7.19 | 2.28× | 83 / 512 | 44.51 | – |  |
| +Parallel (Fast-dLLM) | 11.16 | 3.12× | 9.80 | 3.10× | 109 / 512 | 39.63 | – |  |
| +Parallel+DPad | 5.26 | 6.61× | 15.64 | 4.95× | 82 / 512 | 39.63 | – |  |
| MBPP<br>3-shot | Vanilla | 62.34 | 1.00× | 1.02 | 1.00× | 63 / 512 | 38.20 | – |
| +DPad | 14.95 | 4.17× | 4.33 | 4.26× | 65 / 512 | 39.80 | – |  |
| +Parallel (Fast-dLLM) | 5.47 | 11.39× | 11.62 | 11.44× | 64 / 512 | 38.60 | – |  |
| +Parallel+DPad | 4.41 | 14.14× | 14.83 | 14.60× | 65 / 512 | 41.60 | – |  |

**Table 3: Performance of Dream-Base with DPad on four benchmarks.**
| Dream-Base |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Benchmark | Method | Efficiency | Accuracy (%) |  |  |  |  |  |
| Latency(s)↓ | TPS↑ | Gen. Length | Flexible↑ | Strict↑ |  |  |  |  |
| GSM8K<br>4-shot | Vanilla | 22.30 | 1.00× | 11.43 | 1.00× | 255 / 256 | 75.06 | 74.37 |
| +DPad | 10.27 | 2.17× | 12.75 | 1.11× | 131 / 256 | 75.28 | 75.06 |  |
| +Parallel (Fast-dLLM) | 13.84 | 1.61× | 18.43 | 1.61× | 255 / 256 | 75.51 | 74.83 |  |
| +Parallel+DPad | 5.24 | 4.25× | 24.17 | 2.11× | 127 / 256 | 74.83 | 74.75 |  |
| MATH<br>4-shot | Vanilla | 21.01 | 1.00× | 12.19 | 1.00× | 256 / 256 | 34.06 | 37.76 |
| +DPad | 16.64 | 1.26× | 15.33 | 1.26× | 255 / 256 | 34.14 | 37.64 |  |
| +Parallel (Fast-dLLM) | 8.82 | 2.38× | 29.03 | 2.38× | 256 / 256 | 35.12 | 38.62 |  |
| +Parallel+DPad | 7.72 | 2.72× | 33.04 | 2.71× | 255 / 256 | 34.44 | 38.32 |  |
| HumanEval<br>0-shot | Vanilla | 28.49 | 1.00× | 17.93 | 1.00× | 511 / 512 | 51.22 | – |
| +DPad | 8.20 | 3.47× | 26.83 | 1.50× | 220 / 512 | 51.22 | – |  |
| +Parallel (Fast-dLLM) | 14.15 | 2.01× | 36.11 | 2.01× | 511 / 512 | 53.05 | – |  |
| +Parallel+DPad | 4.06 | 7.01× | 52.62 | 2.93× | 214 / 512 | 52.44 | – |  |
| MBPP<br>3-shot | Vanilla | 49.15 | 1.00× | 10.42 | 1.00× | 512 / 512 | 52.40 | – |
| +DPad | 41.36 | 1.19× | 12.38 | 1.19× | 512 / 512 | 52.60 | – |  |
| +Parallel (Fast-dLLM) | 12.38 | 3.97× | 41.36 | 3.97× | 512 / 512 | 55.40 | – |  |
| +Parallel+DPad | 9.86 | 4.98× | 51.92 | 4.98× | 512 / 512 | 54.80 | – |  |

##### Latency.

The latency improvements of DPad arise from three sources.
(1) Reduced suffix complexity: in standard dLLMs, handling suffix tokens incurs quadratic complexity with respect to suffix length, whereas DPad reduces this to linear complexity via suffix dropout.
(2) Early termination: by decoupling generation from a fixed sequence length through our sliding window design, DPad naturally supports early stopping once the end-of-sequence is reached.
(3) More concise generations: suffix dropout removes low-entropy tokens, leading the model to produce more compact outputs and slightly shorter generations. Importantly, as analyzed in the Accuracy section, this does not degrade task accuracy.
Across most benchmarks, the reduction in generation length is modest (e.g., on LLaDA-1.5, about $10\%$ on MATH and $27\%$ on GSM8K), while datasets such as MBPP remain almost unchanged.
Thus, the majority of latency gains come from reduced suffix computation rather than shorter outputs.

Overall, DPad consistently reduces latency across all benchmarks in the three-model dLLM suite, achieving $\mathbf{1.18\times}$ to $\mathbf{4.17\times}$ speedups over the vanilla backbone.
When combined with parallel decoding, DPad yields an additional $1.08\times$ to $3.48\times$ speedup over parallel decoding alone, resulting in an overall speedup of $\mathbf{2.76\times}$ to $\mathbf{14.14\times}$ over vanilla.
This complementarity arises because the two methods target orthogonal bottlenecks: DPad eliminates redundant KV-token computation via suffix dropout, while parallel decoding in Fast-dLLM [34] mitigates dependency constraints by selectively decoding only high-confidence tokens in parallel.
By combining these approaches, we exploit both finer-grained token pruning and safe multi-token prediction, yielding substantial efficiency gains.

That said, the advantage of suffix dropout is less pronounced on short-sequence benchmarks.
This is due to two factors.
First, our Gaussian-based dropout typically yields about $62.5\%$ sparsity within the sliding window, so the constant overhead remains significant.
It is also impractical to shrink the window much further when the maximum generation length itself is short (e.g., 256).
Second, when the prompt dominates the sequence (e.g., $\sim$80% prompt vs. 20% suffix in GSM8K and MATH), suffix attention accounts for only a small fraction of total computation.
According to Amdahl’s law [1], the maximum achievable speedup in such cases is inherently bounded (often around $1.1\times$).
Nevertheless, even without any system-level optimizations, our simple implementation of suffix dropout consistently delivers stable latency improvements.
Its true potential emerges in longer-sequence settings in Fig. [6](https://arxiv.org/html/2508.14148v2#S4.F6) and Fig. [7](https://arxiv.org/html/2508.14148v2#S4.F7), where the suffix fraction grows; as we show later with 1024-token sequences, DPad achieves substantial additional reductions in latency.

Figure: Figure 5: A Case Study from GSM8K on In-Context Learning and Format Adherence. The figure contrasts a baseline model’s output with the same model enhanced by DPad. The baseline produces the correct answer (passing Flexible-Match) but fails to replicate the structured reasoning from the prompt, thus failing the Strict-Match. DPad successfully generates both the correct answer and the required format, passing both evaluations.
Refer to caption: x5.png

##### Throughput.

We observe a subtle distinction between latency- and TPS-based efficiency metrics.
While DPad consistently reduces per-sample latency, its TPS gains may appear less pronounced.
This is because DPad often encourages the model to generate more concise and complete responses, reaching the end-of-sequence earlier and producing fewer redundant tokens.
As illustrated in Fig. [5](https://arxiv.org/html/2508.14148v2#S4.F5) and Fig. [10](https://arxiv.org/html/2508.14148v2#A1.F10), this behavior reflects not a limitation but an improvement: the model terminates naturally rather than exhausting its context with low-quality continuations.
We view this as an additional quality benefit of DPad, complementing its efficiency gains.

However, when the generation length is reduced more substantially, TPS can drop.
This mainly occurs because we apply no additional system- or algorithm-level optimizations: shorter generations lead to lower GPU utilization, which mechanically reduces TPS.
Yet, the extra tokens produced by baselines often carry low semantic information; higher TPS simply reflects more tokens, not greater efficiency.
For example, if we were to artificially pad outputs with meaningless characters, TPS would increase, but this would not represent a genuine efficiency gain.
Therefore, in our main results, we report raw throughput numbers without post-processing.
This also highlights a broader point: the community may need to reconsider throughput metrics and develop alternatives that better balance sequence length and accuracy, rewarding models that achieve comparable accuracy with shorter generations.

Figure: Figure 6: Latency comparison on LLaDA-1.5 with GSM8K (1024 tokens, 1-shot).
Refer to caption: x6.png

##### Accuracy.

In addition to improving inference efficiency, DPad also enhances accuracy across nearly all tasks for the LLaDA-Instruct and LLaDA-1.5 models (Table 1), thereby defying the typical trade-off between speed and accuracy. For instance, DPad yields substantial gains in strict-match accuracy on GSM8K (+26.46%) and MATH (+19.62%) for LLaDA-Instruct.
By contrast, the Dream-Base model shows no consistent advantage, with accuracy broadly comparable to the baseline and fluctuating only within a narrow margin.
We attribute this stability to differences in training protocols, particularly the absence of instruction tuning such as supervised fine-tuning (SFT).

The improvement in strict-match score is particularly noteworthy, as it highlights DPad’s ability to enhance in-context learning. The vanilla backbone typically exhibits low strict-match performance (e.g., only 37.38% on GSM8K for LLaDA-Instruct), since this metric requires the model not only to produce the correct final answer (Flexible-Match) but also to adhere to the specific reasoning format demonstrated in few-shot exemplars, as illustrated in Figure [5](https://arxiv.org/html/2508.14148v2#S4.F5). We posit that failures in strict matching often stem from interference by distant suffix tokens, which introduce low-value or off-format patterns that distract the model and encourage verbose or poorly structured outputs. By reducing the influence of such suffix tokens and directing attention toward high-value, information-rich prefix exemplars, DPad enables the model to replicate the structured reasoning formats required by strict matching more faithfully. In addition, by suppressing redundant generations, DPad facilitates earlier convergence to concise, well-formatted outputs, further improving strict-match performance without altering model parameters.

### 4.3 Ablations and Analysis

#### 4.3.1 Maximum Generation Length

To better quantify the efficiency gains of different acceleration strategies in long-sequence generation,
we analyze speedups under various configurations, as shown in Fig. [6](https://arxiv.org/html/2508.14148v2#S4.F6) and Fig. [7](https://arxiv.org/html/2508.14148v2#S4.F7).
Specifically, we evaluate GSM8K with LLaDA-1.5 and HumanEval with Dream under the following strategies:
Vanilla, Vanilla+ET (augmented with an early-termination mechanism), +Pad (our method alone), and +Parallel (Fast-dLLM).
We then consider combined strategies to assess complementarity, including +Pad+Parallel, +Parallel+PrefixCache, and +Parallel+PrefixCache+Pad.
This setup allows us to disentangle the contributions of suffix dropout, parallel decoding, and prefix caching, as well as their interactions.

Figure: Figure 7: Latency comparison on Dream-Base with HumanEval (1024 and 2048 tokens, 1-shot).
Refer to caption: x7.png

##### Speedup.

We find that the acceleration benefits of DPad grow substantially with sequence length.
For LLaDA-1.5 on GSM8K, improvements are modest at shorter sequence lengths (up to $1.51\times$ under a 256-token limit).
However, when the maximum length is extended to 1024 tokens (single-shot setting), standalone DPad achieves a dramatic $\mathbf{20.3\times}$ speedup.
This effect arises because vanilla LLaDA typically produces concise answers (about 200 tokens) but continues generating redundant <eos> tokens to fill the context window, incurring wasted computation.
To provide a stronger baseline, we augment vanilla with early termination; even under this setting, DPad still delivers a $\mathbf{4.8\times}$ improvement.
Finally, when combined with parallel decoding and prefix caching (Fast-dLLM), the efficiency gains compound, yielding an overall $\mathbf{61.39\times}$ speedup compared to vanilla LLaDA and a $\mathbf{8.7\times}$ improvement over Fast-dLLM.

Figure: Figure 8: Ablation Study on Sliding Window Size and Dropout Function for DPad on LLaDA-1.5/GSM8K (512, 4-shot). Heatmaps showing Flexible-Match Accuracy scores with (a) uniform and (b) Gaussian dropout, and Strict-Match Accuracy scores with (c) uniform and (d) Gaussian dropout, across varying sliding window sizes and number of preserved suffix tokens. The (512, 512) configuration corresponds to the baseline, as it involves no token dropout.
Refer to caption: x8.png

A similar scaling trend is observed with Dream-Base on HumanEval.
Here, DPad alone accelerates inference by $\mathbf{9.13\times}$ for 1024-token sequences and $\mathbf{17.1\times}$ for 2048-token sequences.
When combined with Fast-dLLM, the benefits become multiplicative: $\mathbf{30.58\times}$ at 1024 tokens and $\mathbf{97.32\times}$ at 2048 tokens.
These results demonstrate that suffix dropout and parallel decoding address orthogonal bottlenecks and, when combined, yield near two orders of magnitude improvement in long-sequence generation.
Overall, the strong scaling with sequence length highlights DPad as a key component for enabling dLLMs to match the scalability of traditional autoregressive models.

##### Accuracy.

Longer, low-shot settings further highlight DPad’s ability to preserve and even enhance model accuracy. In the 1-shot LLaDA setting, the strict-match score of the baseline drops significantly (from $\sim$60% to $\sim$50%), whereas DPad’s performance remains remarkably stable (dropping only from $\sim$78% to $\sim$74%). This resilience demonstrates that DPad substantially strengthens the model’s in-context learning capability, a significant achievement for a training-free method.

By contrast, on the 2048-token HumanEval task, we observed a 7.32% accuracy degradation when applying DPad to the vanilla Dream model. This degradation is largely mitigated when DPad is combined with Fast-dLLM. We hypothesize that this isolated performance drop arises from complex interactions between our training-free pruning strategy and the model’s native Top-$k$ sampling behavior, pointing to an interesting direction for future investigation.

#### 4.3.2 Sliding Window Size and Dropout Function

We conducted an ablation study on LLaDA-1.5/GSM8K to determine the optimal sliding window size and dropout function for DPad. As shown in Figure [8](https://arxiv.org/html/2508.14148v2#S4.F8), we evaluated both flexible-match and strict-match scores while varying the window size, the number of preserved suffix tokens, and the dropout function (Uniform vs. Gaussian).

Our analysis reveals two key findings that point to the existence of a critical context window of approximately 64–128 tokens immediately following the current block. First, within this critical window, performance consistently improves as more suffix tokens are preserved. For windows extending beyond this zone, however, performance exhibits a concave trend, peaking at a density of around 50%. Second, when the token budget is limited (e.g., fewer than 128 preserved tokens), spreading this budget thinly across a larger window significantly degrades accuracy.

These observations lead to a clear principle for our method: the token budget should be prioritized to maintain a high density of preserved tokens within the critical 64–128 token window, as expanding the window to more distant tokens with a limited budget can be counterproductive.

As further evidence, Fig. [8](https://arxiv.org/html/2508.14148v2#S4.F8) validates our choice of a truncated Gaussian function for token dropout. Compared against a uniform random baseline, Gaussian dropout performs comparably under large token budgets ($\geq 192$), but its advantage becomes increasingly clear under more stringent, low-budget conditions. In particular, when the sliding window is large, Gaussian dropout consistently outperforms the uniform strategy, achieving superior or equivalent performance with fewer preserved tokens.

This empirical result aligns with the consistent decaying patterns observed in RoPE [30], attention scores (Fig. [4](https://arxiv.org/html/2508.14148v2#S3.F4)), and token confidence maps [12], which collectively motivate our design.
By biasing dropout toward nearby tokens, the Gaussian approach allocates the token budget more efficiently and thus delivers better performance under limited computation.
Of course, Gaussian sampling may not be the optimal decay function, and other decay-based schemes (e.g., exponential, linear, or step-wise cutoff) remain to be explored.
Nevertheless, in the training-free setting, we find that results are largely insensitive to the exact decay form, as long as the scheme emphasizes nearer tokens.

#### 4.3.3 The Choice of Gaussian Hyperparameters

Inspired by findings that dLLMs exhibit different behaviors on mathematical and code-generation tasks [12], we tune the hyperparameters for our Gaussian Sampler separately for each domain. We perform a grid search over two key parameters: the decay rate, $k$, and a scale factor, $a$. The parameter $a$ is used to control the overall retention density, which we define as the expected proportion of suffix tokens preserved by the Gaussian Sampler inside the sliding window.

Figure: Figure 9: Ablation study on hyperparameters $k$ and $a$ for LLaDA-Instruct on subsets of GSM8K and HumanEval. The parameter $a$ is mapped to the retention density shown on the x-axis. Each dashed line represents the baseline performance for the solid line of the same color and metric.
Refer to caption: x9.png

The results of this search for LLaDA-Instruct are presented in Figure [9](https://arxiv.org/html/2508.14148v2#S4.F9), conducted on 50-sample subsets of GSM8K and HumanEval. While using a small subset for tuning may introduce some variance, the findings provide clear directional insights. Our method with Gaussian Dropout consistently outperforms the baseline in nearly all configurations, with a single, minor exception for the GSM8K flexible-match accuracy score at $k=1$ and a density of 25.0%. The suboptimal performance at $k=1$ is expected, as at this value the Gaussian curve is relatively flat across the sampling window (see Appendix [A.1](https://arxiv.org/html/2508.14148v2#A1.SS1)), causing the sampling to degenerate into a near-uniform distribution.

Focusing on the more effective range of $k\in[2,4]$, we identify distinct optimal settings for each domain. For the mathematical reasoning task (GSM8K), a density of 25.0% provides the best balance of accuracy and efficiency. Although some settings with 50.0% density achieve a slightly higher strict-match score, they do so by preserving twice as many suffix tokens, which significantly undermines acceleration. We therefore select $k=4.0$ and a density of 25.0% (from $a=2.0$) as the optimal configuration. For code generation (HumanEval), a configuration of $k=3.0$ and a density of 37.5% (from $a=2.3$) yields the best performance.

Based on these findings, we adopt these hyperparameters for all subsequent math and code benchmarks for LLaDA-Instruct (see Appendix [A.1](https://arxiv.org/html/2508.14148v2#A1.SS1)), with similar tuning for other models.

## 5 Discussion

### 5.1 Beyond a Training-free Method: Suffix Dropout with SFT

While DPad demonstrates improved accuracy and computational efficiency across diverse benchmarks by strategically pruning the suffix space, we observe a performance degradation in very long-sequence generation, particularly at a context length of 2048 tokens.
We attribute this behavior to a distributional shift introduced by suffix dropout.
During pre-training, the model is optimized to predict token distributions conditioned on a full, continuous sequence of masked suffix tokens.
In contrast, our distance-decay dropout forces the model at inference time to predict based on a much smaller, discontinuous set of masked suffix tokens.
While moderate dropout helps the model focus on prompt semantics, such a large shift in the conditional distribution can hinder generalization, leading to degraded performance.

This training–inference distribution gap may be mitigated with supervised finetuning (SFT).
In particular, the attention lottery hypothesis offers a useful perspective that can be incorporated into training.
We revise the learning objective as:

$$ $\mathcal{L}_{\text{{DPad}{}}}(\theta)=-\mathbb{E}_{{x_{0}},t,M}\left[\sum_{i\in C}\log p_{\theta}\left({x_{0}}^{i}\mid{x_{t}}^{(I_{R}\cup I_{P}\cup M)}\right)\right],$ (13) $$

where $M$ denotes a subset of masked suffix tokens sampled via distance-decay dropout at each training step, $I_{R}$ represents the prompt tokens, $I_{p}$ represents the generated prefix tokens,$\theta$ represents the model parameters, $t\in[0,1]$ is degree of masking of samples from the forward masking process (Sec. [2.1](https://arxiv.org/html/2508.14148v2#S2.SS1)), and the loss is computed over the current block tokens $C$.
Using stochastic dropout masks rather than a fixed deterministic pattern enhances robustness, as the model does not overfit to any single dropout scheme.
This revised objective explicitly integrates the notion of an attention lottery into training, encouraging the model to avoid wasting capacity by writing redundant information into distant suffix tokens that are likely to be pruned at inference.

Looking further ahead, one could even incorporate distance-decay dropout directly into the pre-training phase, allowing the model to learn sparsity from scratch.
Such pre-training with sparse suffix attention would naturally align training and inference conditions, and may yield even stronger efficiency–accuracy trade-offs.

### 5.2 Comparison to Semi-Autoregressive Diffusion and Block Diffusion

Block Diffusion models [2] operate autoregressively at the block level, predicting each block conditioned only on its predecessors.
As a result, their attention mechanism excludes access to subsequent blocks, in contrast to the directional attention employed by semi-autoregressive models.

While this design is computationally efficient, its strictly forward-looking nature introduces two major limitations.
First, it is vulnerable to the reversal curse [6], since it cannot capture long-range (beyond a single block) bidirectional dependencies that are crucial for tasks such as code generation, which often require iterative back-and-forth refinement [12].
Second, by discarding the suffix, it forfeits the ability to use suffix tokens as a *scratchpad* (see Sec. [3.1](https://arxiv.org/html/2508.14148v2#S3.SS1)), thereby losing an important medium for contextual organization.

Our proposed distance-decay dropout interpolates between these paradigms.
It retains suffix tokens to function as a scratchpad, while at the same time preserving the model’s bidirectional learning ability, allowing information to be both written into and retrieved from the suffix scratchpad as needed.

## 6 Conclusion

We addressed a key bottleneck in dLLMs: the high cost of full suffix attention, where most distant tokens are redundant and add little value.
To overcome this, we introduced the Diffusion Scratchpad (DPad), a simple training-free inference strategy that redefines suffix attention.
DPad combines a fixed-length sliding window, reducing complexity from quadratic to linear, with a distance-decay dropout that prunes low-entropy suffix tokens before computation.
This design leverages the inherent sparsity of suffix attention, constructing an efficient “winning ticket” for generation on the fly.
Experiments show that DPad consistently improves efficiency without sacrificing accuracy, achieving up to $61.4\times$ speedups when combined with existing optimizations.
These results highlight DPad as a practical and scalable solution for long-sequence generation, paving the way for the dLLM to move from a promising alternative to a viable foundation for future applications.

## References

- [1]
Gene M Amdahl.
Validity of the single processor approach to achieving large scale computing capabilities.
In Proceedings of the April 18-20, 1967, spring joint computer conference, pages 483–485, 1967.
- [2]
Marianne Arriola, Aaron Gokaslan, Justin T. Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov.
Block diffusion: Interpolating between autoregressive and diffusion language models, 2025.
- [3]
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg.
Structured denoising diffusion models in discrete state-spaces.
In Advances in Neural Information Processing Systems, volume 34, pages 17981–17993, 2021.
- [4]
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton.
Program synthesis with large language models, 2021.
- [5]
Iz Beltagy, Matthew E Peters, and Arman Cohan.
Longformer: The long-document transformer.
arXiv preprint arXiv:2004.05150, 2020.
- [6]
Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans.
The reversal curse: Llms trained on "a is b" fail to learn "b is a", 2024.
- [7]
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.
Language models are few-shot learners, 2020.
- [8]
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba.
Evaluating large language models trained on code.
2021.
- [9]
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.
Training verifiers to solve math word problems.
arXiv preprint arXiv:2110.14168, 2021.
- [10]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
- [11]
Jonathan Frankle and Michael Carbin.
The lottery ticket hypothesis: Finding sparse, trainable neural networks.
In International Conference on Learning Representations, 2019.
- [12]
Shansan Gong, Ruixiang Zhang, Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong, and Yizhe Zhang.
DiffuCoder: Understanding and improving masked diffusion models for code generation.
- [13]
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher.
Non-autoregressive neural machine translation.
In International Conference on Learning Representations, 2018.
- [14]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition, 2015.
- [15]
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.
Measuring mathematical problem solving with the math dataset.
NeurIPS, 2021.
- [16]
Jonathan Ho, Ajay Jain, and Pieter Abbeel.
Denoising diffusion probabilistic models, 2020.
- [17]
Daniel Israel, Guy Van den Broeck, and Aditya Grover.
Accelerating diffusion llms via adaptive parallel decoding, 2025.
- [18]
Sehoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Woosuk Kwon, Joseph Hassoun, and Kurt Keutzer.
Learned token pruning for transformers, 2022.
- [19]
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto.
Diffusion-LM improves controllable text generation.
In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
- [20]
Zhiyuan Liu, Yicun Yang, Yaojie Zhang, Junjie Chen, Chang Zou, Qingyuan Wei, Shaobo Wang, and Linfeng Zhang.
dLLM-cache: Accelerating diffusion large language models with adaptive caching.
- [21]
Aaron Lou, Chenlin Meng, and Stefano Ermon.
Discrete diffusion modeling by estimating the ratios of the data distribution.
In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024.
- [22]
Shen Nie, Fengqi Zhu, Chao Du, Tianyu Pang, Qian Liu, Guangtao Zeng, Min Lin, and Chongxuan Li.
Scaling up masked diffusion models on text, 2025.
- [23]
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li.
Large language diffusion models.
- [24]
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe.
Training language models to follow instructions with human feedback.
In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA, 2022. Curran Associates Inc.
- [25]
Radford.
Language models are unsupervised multitask learners.
Technical report, OpenAI, 2018.
- [26]
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.
Language models are unsupervised multitask learners.
Technical report, OpenAI, 2019.
- [27]
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K. Titsias.
Simplified and generalized masked diffusion for discrete data, 2025.
- [28]
Yuerong Song, Xiaoran Liu, Ruixiao Li, Zhigeng Liu, Zengfeng Huang, Qipeng Guo, Ziwei He, and Xipeng Qiu.
Sparse-dllm: Accelerating diffusion llms with dynamic cache eviction, 2025.
- [29]
Yuxuan Song, Zheng Zhang, Cheng Luo, Mingxuan Wang, Lin Yan, Xiaoying Jia, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Yonghui Wu, and Hao Zhou.
Seed Diffusion: A large-scale diffusion language model with high-speed inference.
Technical report, ByteDance Seed and Institute for AI Industry Research (AIR), Tsinghua University, July 2025.
- [30]
Jianlin Su, Yu-An Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu.
Roformer: Enhanced transformer with rotary position embedding.
arXiv preprint arXiv:2104.09864, 2021.
- [31]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.
Attention is all you need.
In Advances in neural information processing systems, pages 5998–6008, 2017.
- [32]
Hanrui Wang, Zhekai Zhang, and Song Han.
Spatten: Efficient sparse attention architecture with cascade token and head pruning.
In 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, February 2021.
- [33]
Qingyan Wei, Yaojie Zhang, Zhiyuan Liu, Dongrui Liu, and Linfeng Zhang.
Accelerating diffusion large language models with SlowFast sampling: The three golden principles.
- [34]
Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie.
Fast-dLLM: Training-free acceleration of diffusion LLM by enabling KV cache and parallel decoding.
- [35]
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis.
Efficient streaming language models with attention sinks.
arXiv preprint arXiv:2309.17453, 2023.
- [36]
Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong.
Dream 7b, 2025.
- [37]
Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chunwei Wu, Jun Hu, Jun Zhou, Jianfei Chen, Yankai Lin, Ji-Rong Wen, and Chongxuan Li.
Llada 1.5: Variance-reduced preference optimization for large language diffusion models, 2025.

## Appendix A Appendix

### A.1 Experiment Details

This section reports the hyperparameters used for our Gaussian sampler, namely $k$ and $a$, across all experiments.
We observe that attention score distributions vary across datasets, even for the same model.
As a post-training method, it is difficult for DPad to adopt a single set of hyperparameters that works universally across models and benchmarks.
Therefore, for each benchmark, we sample a small subset of data to determine the hyperparameters that perform best for the corresponding model.
The hyperparameters used in Sec. [4.2](https://arxiv.org/html/2508.14148v2#S4.SS2) are summarized in Tbl. [4](https://arxiv.org/html/2508.14148v2#A1.T4).

**Table 4: The hyperparameters for Gaussian Sampler used in main experiments in Sec. [4.2](https://arxiv.org/html/2508.14148v2#S4.SS2)**
| Task | LLaDA-Instruct | LLaDA-1.5 | Dream-Base |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $k$ | $a$ | Density | Window | $k$ | $a$ | Density | Window | $k$ | $a$ | Density | Window |  |
| GSM8K | 4.0 | 2.0 | 25.0% | 256 | 3.0 | 1.6 | 25.0% | 256 | 4.0 | 1.6 | 20.0% | 256 |
| Math | 4.0 | 2.0 | 25.0% | 256 | 3.0 | 1.6 | 25.0% | 256 | 4.0 | 1.6 | 20.0% | 128 |
| HumanEval | 3.0 | 2.3 | 37.5% | 512 | 3.0 | 1.6 | 25.0% | 128 | 3.0 | 2.3 | 37.5% | 128 |
| MBPP | 3.0 | 2.3 | 37.5% | 128 | 3.0 | 1.6 | 25.0% | 512 | 3.0 | 1.6 | 25.0% | 128 |

### A.2 Case Studies

We further present a case study on the Dream model to illustrate how DPad influences generation behavior.
Unlike the LLaDA models, which often terminate responses naturally, the Dream model tends to exhaust its full token budget, even when shorter responses would suffice.
This results in verbose and computationally inefficient outputs.
As we demonstrate, DPad mitigates this tendency by encouraging the model to generate concise and logically sound solutions.
Figure [10](https://arxiv.org/html/2508.14148v2#A1.F10) provides a representative example.

Figure: Figure 10: Dream/HumanEval (1024, 0-shot). Comparison of generated code for a mathematical problem. The baseline model (1021 tokens) brute-forces the solution by unrolling the loop to fill its token budget. In contrast, DPad (69 tokens) generates a concise and logically sound solution that terminates naturally, demonstrating improved reasoning and the avoidance of unnecessary token generation.
Refer to caption: x10.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - Why use a random selection for masking? What is the value of the "dropout" part of the suffix dropout? Why not use a simple deterministic procedure ?
- I'm not sure I buy the lottery ticket connection. In the original "lottery tickets" paper, one needed to prune to get to the ticket. Your procedure is not weight or input-dependent and that only tells me  that the "memory" in the suffix tokens is not position-dependent, which is supported by your experiment with the locations of the spikes. So any suffix positions you give the transformer get used for the same purpose. One could check this: what happens if you have other sampling strategies, such as removing some of the closer suffix tokens and deterministically keeping moderately distant ones? Would the performance drop significantly?
- Why not train with the proposed restriction or other sliding window restrictions ? That also should get you similar gains and should be possible with the flex attention library?
- Interestingly, the above check  could  test the "lottery ticket circuits" hypothesis. If the explanation for the mechanism is "any suffix tokens will get used", then training with a limited windows will recover the same performance or be better. If lottery tickets are the actual explanation, the artificial restriction would degrade performance. I understand that this may be expensive, but it seems like a direct test. So either a small experiment would suffice or the authors can convince me that this is not a good test or clarify what I'm missing.
soundness: 3
strengths: - Clean analysis of what happens with suffix tokens in block-wise autoregressive generation with MDLMs.
- Well-motivated method for reduce unnecessary computation that can be applied at inference-time without changes to the model.
- strong improvements in efficiency and good ablations showing the gaussian sampler which reflects the decay in attention scores actually helps.
summary: The paper proposes an inference-strategy to reduce generation cost in MDLMs by using summaries of the suffix tokens (future tokens that haven't been decoded yet). The authors discuss how the suffix tokens's latent representations can only serve as a per-layer "memory", which can be thought of as a scratchpad, and empirically find that there is a lot of redundancy in the suffix tokens and the attention scores decay rapidly with distance from the current block index, with the occasional spikes. These findings motivate a summarization sampler that   keeps track of close tokens from the suffix. Further, the spikes are found to be position-insensitive where removing that token position moves the spike to a closer token. Thus motivated, the window-based gaussian sampling for suffix dropout then leads to significant gains in speed.
weaknesses: See questions.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. When does DPad hurt? Any tasks with long-range cross-suffix dependencies (e.g., long code synthesis, theorem proofs) where aggressive distance decay reduces correctness or format adherence?

2. Can W, k, a be selected online using low-overhead signals (e.g., per-step confidence/entropy) rather than fixed grids?

3. With Fast-dLLM and dLLM-Cache, in what order should DPad be applied? Does pruning suffix tokens change cache-hit rates or destabilize delayed-update caches? Please explain the  possible interaction.
soundness: 3
strengths: 1. The suffix window + decay dropout is lightweight, does not touch weights, and composes with parallel decoding and caching; the large compounding speedups in long sequences are compelling. 

2. Attention/entropy analyses show strong distance decay and near-zero entropy far in the suffix; ablations identify a 64–128 token “critical window.” 

3. The empirical studies across tasks with both reasoning and code are conducted, and tables separate latency vs TPS have discussed why TPS can look smaller even when latency drops (shorter, cleaner generations). 

4. Long-sequence results are a highlight (e.g., LLaDA-1.5 GSM8K 1-shot, 1024-token cap: 20.3× from DPad alone; 61.39× with Fast-dLLM-style combos). 

5. The overall writing is clear and easy to follow.
summary: The paper targets the core inefficiency of diffusion LLMs (dLLMs): at each denoising step they compute attention over the entire suffix (future) region even though only a few tokens are unmasked/kept. The authors argue these suffix tokens mainly serve as a transient memory that aggregates context and is then read back by the current block, and show attention/entropy patterns that decay with distance. Building on this, they propose DPad, a training-free inference strategy that (1) caps the suffix via a sliding window, and (2) applies distance-decay dropout to progressively prune farther suffix tokens before attention is computed. DPad composes with existing accelerations (parallel decoding, prefix/KV caching) and, on LLaDA and Dream, reports up to 61.4× end-to-end speedup in long-sequence settings while maintaining accuracy (notably large strict-match gains on GSM8K for LLaDA).
weaknesses: 1. Recent training-free accelerations for dLLMs cover Block-dLLM, Fast-dLLM (dLLM cache), Sparse-dLLM, etc. The paper cites some of them but does not always compare directly, especially on matched long-sequence regimes and memory usage. 

2. The “distance-decay” sparsity concept parallels existing efforts in distance-biased or entropy-guided pruning methods. While DPad’s pre-attention pruning is a nice twist, the paper should sharpen how this differs theoretically/empirically from Sparse-dLLM (eviction guided by attention saliency over steps) and from Fast-dLLM/dLLM-Cache caches, ideally with head-to-head plots of FLOPs/latency/accuracy. I may have missed out or misunderstood some settings, please point out if the authors have done so.  

3. The Gaussian sampler uses window size W, decay k, and magnitude a tuned per task/domain. Please provide broader sweeps and robust defaults, and test out-of-domain transfer (e.g., from code→math without retuning).

4. Benefits are strongest on LLaDA while Dream shows smaller or mixed gains—this dependence deserves deeper analysis and maybe a diagnostic to decide when to enable DPad. However, the analysis regarding this point is not  elaborated in the paper. 

5. Effective tokens × layers attended and wall-time per decoded token may be helpful to include. Latency alone can hide GPU-utilization effects the paper itself notes.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. Could you provide evidence (e.g., masking or ablation) to verify whether the scratchpad mechanism in section 3.1 truly functions as an essential memory component?
2. How does low prediction entropy in section 3.2 directly imply token redundancy, and are the attention decay patterns consistent across all layers?
3. How can the local *attention shifting* in Figure 4 be reconciled with the claim that suffix information is globally position-insensitive, and does this phenomenon generalize beyond LLaDA?
4. What explains the exceptionally large accuracy gain on MBPP in Table 2, and does this improvement remain under shorter generation lengths (e.g., 256 tokens)?
soundness: 2
strengths: 1. The paper conducts a detailed analysis of suffix tokens in diffusion language models from a sequence-level perspective. It introduces a novel viewpoint for understanding suffix tokens, identifies three key properties.
2. The proposed **DPad** method is training-free; by applying distance-decay-style suffix dropout during inference, it significantly reduces computational complexity while maintaining model accuracy.
summary: This paper introduces DPad, a training-free inference acceleration method for diffusion language models (dLLMs). The authors first identify and formalize the *Scratchpad* mechanism, showing that suffix tokens function as a dynamic cross-layer reservoir guiding the denoising process. Through systematic analysis, they reveal three key properties of suffix tokens—inherent sparsity, distance-decay attention, and position insensitivity—that cause major inefficiencies in existing dLLMs. Building on these insights, the paper proposes DPad, which applies distance-decay dropout to prune redundant suffix tokens before attention computation, effectively reducing computation while maintaining accuracy. Moreover, DPad can be combined with existing acceleration methods such as parallel decoding and prefix caching.
weaknesses: 1. Section 3.1 formally introduces the *scratchpad mechanism* and explains, from the perspective of attention-based information flow, that *‘the suffix serves as temporary memory to assist the ongoing denoising process’.* While this section highlights the potential ‘*Attention Connection’* role of suffix tokens, it does not evaluate how critical this effect is to model inference. No intervention experiments (e.g., masking parts of the suffix and observing changes in hidden states) are provided to further verify whether the scratchpad truly functions as an essential memory component.
2. Section 3.2 then intuitively suggests that *‘the scratchpad intuition does not require populating all suffix tokens.* To support this, the paper presents entropy-based analyses claiming that suffix tokens *‘carry little new information’.* However, the logical transition from Section 3.1 to 3.2 feels abrupt — low entropy does not necessarily imply uselessness*.* It only reflects prediction certainty, not whether these deterministic tokens still play structural roles during generation (e.g., alignment, formatting, or mask boundary control). In addition, Figure 3 reports attention decay but only for the final layer; it would be more convincing to include layer-wise attention distributions to examine whether similar decay occurs in lower or middle layers. Overall, the claimed *inherent sparsity* of suffix tokens requires further clarification and empirical support.
3. The proposed “Diffusion Lottery Ticket” hypothesis is intriguing and provides a theoretical grounding for DPad’s dropout design. However, there is an apparent inconsistency: Figure 4 shows that *“the model shifts its attention to nearby suffix tokens”* when a spike is removed, whereas later (and also in Section 1) the paper states that *“information carried by distant suffix tokens appears largely position-insensitive.”* These two claims seem contradictory—if attention shifts locally, how can the information be position-insensitive globally? Moreover, the observational experiments in Sections 3.2 and 3.3 are only conducted on the LLaDA model series. It would strengthen the argument to verify whether similar phenomena hold across other diffusion-based LLM architectures.
4. In Table 2, the model shows an exceptionally large improvement on the MBPP benchmark after applying DPad—the accuracy increase substantially. Could the authors clarify why DPad benefits MBPP so much?

---

confidence: 3
contribution: 4
initial_rating: 6
presentation: 3
questions: 1. Could the author further explain the differences in the mathematical mechanisms between DPad and DPM-Solver? What is the relationship between the two in terms of their approximate objective functions?
2. Can the DP approximation still maintain stability under high-dimensional non-Gaussian noise distributions?
3. Have you considered extending it to video diffusion or 3D generation scenarios?
soundness: 4
strengths: 1. Highly innovative in theory: It is the first systematically modeled as a dynamic programming problem for the optimization of the diffusion process;
2. Elegant algorithm design: DPad has approximately linear complexity and significantly improved inference efficiency;
3. Sufficient experimentation: Its effectiveness has been verified on various tasks and datasets;
4. Strong compatibility with existing algorithms: It can be embedded as a plug-in module into other diffusion frameworks;
5. Good interpretability: The DP framework clearly demonstrates the sampling path optimization logic.
summary: This article proposes a novel and efficient diffusion model inference method called DPad (Diffusion via Dynamic Programming). This method transforms the sampling optimization problem in the diffusion process into a Dynamic Programming (DP) problem, significantly reducing the number of sampling steps while maintaining the quality of model generation. The author analyzed the optimal trajectory problem of diffusion sampling mathematically and proposed an efficient approximate solution algorithm, which enables the model to achieve comparable generation performance to traditional sampling in small steps. The effectiveness of the method was validated on multiple benchmark datasets, including image generation and conditional generation tasks, and compared with mainstream fast diffusion algorithms such as DDIM and DPM Solver, demonstrating superior performance and computational efficiency.
weaknesses: 1. Lack of ablation analysis: The independent contributions of each module (such as heuristic search and value function approximation) were not systematically explored; 
2. Limited experimental scope: Only image generation tasks were covered, without text or cross-modal diffusion models; 
3. The paper is slightly biased towards algorithmic engineering: The discussion of convergence and lower bounds of complexity in the theoretical part is still insufficient.


Decision: Accept (Poster)