Sequential Diffusion Language Models
Authors: name: Yangzhou Liu; name: Yue Cao; name: Hao Li; name: Gen Luo; name: Zhe Chen; name: Weiyun Wang; name: Xiaobo Liang; name: Biqing Qi; name: Lijun Wu; name: Changyao Tian; name: Yanting Zhang; name: Yuqiang Li; name: Tong Lu; name: Yu Qiao; name: Jifeng Dai; name: Wenhai Wang
Published: 2025-09-28 17:59:15

Abstract: Diffusion language models (DLMs) have strong theoretical efficiency but are
limited by fixed-length decoding and incompatibility with key-value (KV)
caches. Block diffusion mitigates these issues, yet still enforces a fixed
block size and requires expensive training. We introduce Next Sequence
Prediction (NSP), which unifies next-token and next-block prediction, enabling
the model to adaptively determine the generation length at each step. When the
length is fixed to 1, NSP reduces to standard next-token prediction. Building
on NSP, we propose Sequential Diffusion Language Model (SDLM), which can
retrofit pre-trained autoregressive language models (ALMs) at minimal cost.
Specifically, SDLM performs diffusion inference within fixed-size mask blocks,
but dynamically decodes consecutive subsequences based on model confidence,
thereby preserving KV-cache compatibility and improving robustness to varying
uncertainty and semantics across the sequence. Experiments show that SDLM
matches or surpasses strong autoregressive baselines using only 3.5M training
samples, while achieving 2.1 higher throughput than Qwen-2.5. Notably, the
SDLM-32B model delivers even more pronounced efficiency gains, demonstrating
the strong scalability potential of our modeling paradigm. Project page and
codes: https://github.com/OpenGVLab/SDLM