Title: DPad: Efficient Diffusion Language Models with Suffix Dropout
Primary area: foundation or frontier models, including LLMs
Keywords: Diffusion-based Large Language Models, Model Optimization and Efficiency, Token Pruning, Model Explainability

Abstract: Diffusion-based Large Language Models (dLLMs) parallelize text generation by framing decoding as a denoising process, but suffer from high computational overhead since they predict all future suffix tokens at each step while retaining only a small fraction.  We propose $\textbf{Diffusion Scratchpad} (\textbf{\textit{DPad}})$, a training-free method that restricts attention to a structured subset of suffix tokens, preserving fidelity while eliminating redundancy.  $\textit{DPad}$ integrates two strategies: (i) a $\textit{sliding window}$, which maintains a fixed-length suffix window, and (ii) $\textit{distance-decay dropout}$, which deterministically removes distant suffix tokens before attention computation.  This concise design is compatible with existing optimizations such as parallel decoding and prefix caching, and lends itself to a lightweight implementation.  Comprehensive evaluations across multiple benchmarks on $\texttt{LLaDA}$ and $\texttt{Dream}$ models demonstrate that $\textit{DPad}$ delivers up to $\mathbf{61.4\times}$ speedup over vanilla dLLMs while maintaining comparable accuracy, highlighting its potential for efficient and scalable long-sequence inference.

TL;DR: DPad is a training-free inference method that optimize diffusion-based large language models (dLLMs) by refining their inherent Scratchpad Mechanism; it dropouts redundant suffix tokens to yield significant speedups while maintaining model accuracy.