Ultra-Fast Language Generation via Discrete Diffusion Divergence Instruct
Authors: Haoyang Zheng
Xinyang Liu
Xiangrui Kong
Nan Jiang
Zheyuan Hu
Weijian Luo
Wei Deng
Guang Lin
Venue: ICLR 2026 Poster (2026)
Keywords: diffusion language models
distillation
integral KL divergence
large language models
generative modeling
Primary area: generative models
TL;DR: We distill efficient few-step discrete diffusion language model, achieving fast inference and lower perplexity than state-of-the-art baselines.

Abstract: Fast and high-quality language generation is the holy grail that people pursue in the age of AI. In this work, we introduce **Di**screte **Di**ffusion Divergence **Instruct** (**DiDi-Instruct**), a training-based method that initializes from a pre-trained diffusion large language model (dLLM) and distills a few-step student for fast generation. The model distilled with DiDi-Instruct matches or surpasses its dLLM teacher and the GPT-2 baseline while providing up to **64$\times$ acceleration**. The theoretical foundation of DiDi-Instruct is a novel framework based on integral KL-divergence minimization, which leads to a practical training algorithm. We further introduce *grouped reward normalization, intermediate-state matching, and the reward-guided ancestral sampler* to improve *training stability, model coverage, and inference quality*. On the OpenWebText benchmark, DiDi-Instruct achieves perplexity ranging from 62.2 (8 NFEs) to 18.4 (128 NFEs), outperforming prior accelerated dLLMs and the GPT-2 baseline. These gains incur a negligible entropy loss (around $1$%) and reduce additional training wall-clock time by **more than $20\times$** compared to competing dLLM distillation methods. We further validate the robustness and effectiveness of DiDi-Instruct through extensive ablation studies, model scaling, downstream task evaluations, and unconditional protein sequence generation. In conclusion, DiDi-Instruct enables efficient and effective distillation for language generation in the blink of an eye.

Published as a conference paper at ICLR 2026
ULTRA-FAST LANGUAGE GENERATION VIA DISCRETE DIFFUSION DIVERGENCE INSTRUCT
Haoyang Zheng', Xinyang Liu?, Cindy Xiangrui Kong', Nan Jiang", Zheyuan Hu', Weijian Luo"r, Wei Deng"r Guang Lin'
'Purdue University, University of Texas at Austin, University of Texas at El Paso, "National University of Singapore, hi--La, Xiaohongshu Inc, ML Research, Morgan Stanley
ABSTRACT
Fast and high-quality language generation is the holy grail that people pursue the age of AL In this work, we introduce Discrete Diffusion Divergence Instruct (DiDi-Instruct), training-based method that initializes from a pre-trained dif- fusion large language model (dLLM) and distills a few-step student for fast gen- eration. The resulting DiDi-Instruct model achieves comparable or superior per- formance to its dLLM teacher and the GPT-2 baseline while enabling up to 64x acceleration. The theoretical foundation of DiDi Instruct is a novel framework based on integral KL-divergence minimization, which yields a practical training algorithm. We further introduce grouped reward normalization, intermediate- state matching, and the ewarrd-gided ancestral sampler that significantly im- prove training stability, model coverage, and inference quality. On Open Web DiDi-Instruct achieves perplexity from 62.2 (8 NFEs) 18.4 (128 NFEs), which outperforms prior accelerated dLLMs and GPT-2 baseline. These gains come with a negligible entropy loss (around 1%) and reduce additional training wall-clock time by more than 20x compared to competing dLLM distillation baselines. We further validate the robustness and effectiveness of DiDiiInstruct through exten- sive ablation studies, model scaling, downstream tasks, and the generation of dis- crete protein sequences. In conclusion. distillation method, enabling language generation in the blink of an eye.
is an efficient yet effective
Code: github.com/haon guttub.com/haooynggzeeng aiddd--i-idddddiiiiee
Project Page: haoyangzheng. wwwpeennnggggtuuiiieee hot.eaaciio/reeaaccddiiiiiiiiiii instruct/
Sampled text from 162M (1022 NFEs). Generative Perplexity=37.5m 50.
GPT-2
Working with kids in schoci, Solding Pride evert. creating, mo, a really axecone community for averyone to play past in.
hard,
MDLM
140
DUO
Perplexity
SDTT
Sampled text from 169M base model (1024 NFEs). Generative Perplexity=38. 53.
DiDi-Instruct (Ours)
I thought that when I vaa in hich schcol, it would be too hard people
80
1ike my clasenatos, but twel 21ke can talk to zhen bettez.
40
Sampled text 169M Instract (16 NTTs), Generative Perplexity= 30.90.
20
The sana thing is happening wish classnates. As eaz older, + chink the best part is kaen spe people ny ages I quesz borod with that.
16
128
256
512 1024
Figure 1: Perplexity vs. NFES
INTRODUCTION
Fast language sequence generation has been a long-standing goal for large-scale AI systems. Auto- regressive (AR) large language models (LLMs) have achieved remarkable success across wide range of natural language tasks (Brown et al., 2020; Meta AI, 2023; Achiam et al., 2023; Team
Corresponding authors: pkulwj1994@icc 1911lji1994@cllud.ccommm weideng056 Searrnngg@@@@@@mm..cmmmm @maillmmmammmmmmmmm
Baselines include GPT-2, masked diffusion language models (MDLM; Sahoo et al. (2024)), diffusion duality (DUO; Sahoo et al. (2025)), and self-distillation through time (SDTT; Deschenaux & Gulcehre (2025)).

---

Published as a conference paper at ICLR 2026
et al., 2024). By predicting the next token from left to right using causal attention of Transform- ers (Vaswami et al.. 2017), AR models can scale to billions or even trillions of parameters while delivering state-of-the-ar art performance (Touvron al.. 2023). However, the interior mechanism that enables this success also imposes an inherent bottleneck: tokens must be generated sequen- tially, which limits parallelism and constrains throughput at scales (Leviathan al., 2023; Kim et al., 2023; Ning et al., 2024). Even with recent computation optimization, such as KV caches, AR models still face a significant throughput ceiling.
Inspired by continuous diffusion models for images (Song et al., 202 Ho et al., 2020), diffusion large language models (dLLMs) (Li al., 2022) reinterpret text generation as an iterative denoising process over token sequences, offering competitive alternative to the AR model. Concretely, a dI.LM starts from a fully corrupted sequence and learns to progressively denoise it into clean text sequence. This inference paradigm leverages bidirectional attention in the Transformer, allowing dLLMMs generate sequences with fewer numbers of function evaluations (NFEs) than AR models.
Despite their strong performance and improved efficiency, dLLMs still require many denoising steps during inference, creating an inference time bottleneck. For example, on the Wex bench- mark, dLLMs still need up to 256 NFEs to match the performance of GPT2 model when generating sequences of length 1024. To mitigate this limitation, several distillation methods for dLLMs have been proposed to train few-step generators to produce high-quality samples without performance degradation. Notably, Deschenaux & Gulcehre (2025) trains a student to match a teacher's predic- tions across time-steps, enabling the student to generate at least 32 tokens per step and achieving up to 8x speedups over AR models. Later, Sahoo et al. (2025) relates dLLMs to Gaussian dif- fusion and introduces curriculum learning and discrete consistency distillation support few-step sampling. Zhu et al. (2025b) matches the token-level distribution of dLLMs and uses a token ini- tialization strategy to obtain a one-step generator. We defer a detailed, side-by-side introduction and discussions of their distillation mechanisms to Appendix A.3. While these studies advance the frontier of fast language generation, their generation throughput still leaves substantial room for im- provement: on
even the most recent SDTT can not match the performance of GPT2 baseline within 32 NFEs. Morcover, existing distillation methods are motivated by heuristic design choices, lacking well-established theoretical justification.
In this work, we introduce Discrete Diffusion Divergence Instruct (DiDi-Instruct), a novel distil- lation framework for fast language generation. The learning objective of DiDi-Instruct is to mini- mize the Integral Kullback-Leible (IKL) divergence, introduced by the seminal work of Luo et al. (2023b) for the continuous diffusion model. By minimizing IKL. between the distributions of a few- step student generator and a pre-trained teacher d.L.M, the student is trained to match the teacher's generation distribution, while achieving substantially improved inference efficiency. Directly adapt- ing IKL to masked diffusion models (MDMs), however, is nontrivial due to the discrete and non- differentiable operations inherent in dLLM generation. To overcome these challenges, we develop comprehensive solution spanning objective design, training stability, and inference efficiency. Specifically, we make the following contributions:
Principled Training for Fast Sequence Generation: We reformulate the distillation objective from a general policy gradient perspective, deriving simple yet tractable update rule for the few-step stu- dent according to some reward function. Using an adversarial language discriminator to estimate the log-density ratio (reward) between the teacher dLLM and the student, introduce a practical InDi-lnstruct algorithm that trains the few-step student, paired with an assistant discriminator.
Simple yet Effectivve Techniques in Training and Inference: We introduce grouped reward nor- malization, intermediate-state matching, and the reward-guided ancestral sampler (RGAS) that substantially improve the training stability, the model coverage, and the inference performances, reducing the perplexity of generated sequences by 30%.
State-of-the Art Fast Sequence Generation: InDtitnstruct achieves new state-of-the-ee perfor- mance on the Open WebText benchmark: consistently lower PPL across 8 to 128 NFEs (Figure 1), negligible entropy loss, and over 20x faster distillation; detailed ablations, model scaling, down- stream tasks, and protein sequence generation further confirm its robustness.
To the best of our knowledge, DiDiiInstruct is the first framework to successfully apply distribution- matching distillation to MDM-based text generation, along with record-breaking performances in few-step language sequence generation. A discussion of related work is deferred to Appendix A.
2

---

Published as a conference paper at ICLR 2026
2 PRELIMINARY
This section introduces the notations for MDMs (Sec. 2.1) and the integral KI. divergence (Sec. 2.2), for preparation of the DiDi-Instruct algorithm in Section 3. We consider vocabulary of size K, where each token in the vocabulary is represented by one-hot vector. The set of all such vectors y = {x E {0, 1}x| Σÿ= 3jj = }}. The K-th token is reserved for the special [MASK] token, whose one-hot encoding is noted by vector m with the K-th entry being and all others being O. To generate length-L sequence x1:
(x! ..... xx) E V assume the process factorizes across positions, i.e., each index & {l..... L} evolves independently. We adopt a continuous time index t E [0, 1] with a monotonically decreasing noise schedule ar 00 satisfying og 1 and ai O. Finally, we denote K -impleex as A
2.1 MASKED DIFFUSION MODELS
Forward process. In the forward corruption process, once token transitions the masked state m. it never reverts. Under independent per-token transitions, x x1:2 converges to the fully masked sequence (m.... m) with probability E V transitions to a latent state Z, E V at time t E (0, 1]. This absorbing-stare process implies that 2, either stays as the original token x with probability a or transitions to m with probability 1 oz. The forward corruption kernel is:
- (Sahoo al., 2024). Concretely, a clean token
Qzz x) (zz
(1 -1-00 0z m))
(1)
where Cab 2;; π denotes a Categorical distribution over the one-hot vector V with probability vector π A K For < t, the exact posterior has two cases:
Cat(Zz;Zz)
Z₁ * m,
Q(z, x),
(1
0x2 )m + (as - (a))
Cat
Zz;
24 = m.
1 ag
The intuition of Equation (2) is: time t, an unmasked token implies a deterministic pash, whereas masked token considers a probabilistic mixture governed by the noise schedule.
Backward process. The reverse process of MDMs iteratively reconstructs the original uncorrupted sequence from the noised input by replacing masked tokens. This is achieved through a learned neural network po at time t. The output of py (z( t) satisfies that (i) the predicted distribution forms valid categorical distribution that sum to 1, (ïi) the prediction pa (Z,, t) assigns zero probability mass to the [MASK] token, and (iii) once token is unmasked, its state remains fixed in all subsequent steps.
[o, 1] - A that predicts the clean token x from its corrupted version z,
When applied to sequences x I:L, the denoising process is assumed to token-wiig independent. A network (z!:" t) predicts the distribution over the entire sequence, where (z. ,t) specifies the distribution for the f-th token. The reverse transition from 2, to 2, is defined as Po 2.) the sequence of length L: (z 22 = 11 Q(z) x PV (z)=2, t)) continuous time, minimizing the negative evidence lower bound (NELBO) provides tractable training objective (Shi et al., 2024):
Q(z,, 2,, (2 1)). We extend this to sequences by factorizing the process
da: 1
Look LNELBO = Ey
Σ
xx log (6(2) 1.t. ())
di
di
Ott.
A: =
2.2 INTEGRAL KI. DIVERGENCE
Let (z( t) denote the forward teacher marginal and qu (z, t) the student marginal parameterized by v'. To distill knowledge from qo to q across different noise levels 1. E [o, 1). we minimize the integral Kullhack-Leibles (IKI.) divergence (introduced in Luo et al. (2023b)), which aggregates the discrepancy between qu and qe over the time domain:
DIKI. (qullg)) = /'' (())lL(
w())Baraa log (z,, go
dl,
(4)
Unless otherwise specified, qe and q denote the full collections q "
e t) and qi (z,,
3

---

Published as a conference paper at ICLR 2026
where (t) is positive weighting function". Intuitively, both qe and q are full-support for any t E [0, due to diffusion, making each term (L(qu/lq)) finite and smooth. IKL integrates this continuum of reliable comparisons, ensuring the student learns the teacher's complete denoising behavior, which leads to more stable and effective training than only matching the final output.
It is clear that if we minimize the IKL divergence (4) between a few-step student language model and a pre-trained dLLM teacher model, the student is able to match the teacher's generation ability with much improved efficiency.
3 METHODOLOGY
The extension of IKL to distill MDMs presents several inherent challenges. This section details the comprehensive solution via advances in objective design, training stability, and inference efficiency.
3.1 DISCRETE DIFFUSION DIVERGENCE INSTRUCTION
primary challenge in distilling MDMs is the discrete nature the state space. The gradient formulation Luo al. (2023b) relies on differentiating through ž₁. In MDMs, the forward process involves non-differentiable operations (e.g., arg max), making this gradient inapplicable (Zekri & Boullé, 2025). To address this, Zhu al. (2025b) proposed proxy model approximate z, and reported promising results for text-to-image tasks. However, its effectiveness in text generation remains underexplored. Instead, we draw inspiration from the policy gradient method (Schulman et al., 2017; Fan et al., 2023) to obtain a rigorous solution. By decomposing the objective gradient as score function weighted by a log-density ratio, we derive the following gradient estimation.
Theorem 3.1 (Score-Function Identity). Let the objective L(v) be the weighted IKL divergences between student and teacher marginals, qu and qa. The gradient of the objective admits:
w(t)
Vol(v) (x-r(Ux
R(ze, 1) V. log m. 1m)
(5)
((1)
where time is sampled from distribution (t), and R(z,, t) == log q (z,, log qo (zc, the reward (i.e., the log-density ratio between the student and teacher) evaluated at 2,.
The proof is deferred to Appendix B. Theorem 3.1 shows that the IKL gradient admits score- function form that does not differentiate through the discrete sampling path. We sample x ~ (zr m, 1). corrupt it to 2, Q, and differentiate only log (z m, =). The reward R(z,0), 1) weights the score to guide distillation toward matching the teacher's 's marginal at 4.
3.2 DENSITY-RATU ESTIMATION
The reward is mathematically natural choice for distribution matching. However, both log qu and log qe are intractable to compute directly. To address this, we avoid estimating the individual den- sities and instead approximate their ratio. Inspired by Goodfellow al. (2014); Wang et al. (2025), we train an auxiliary discriminator for this purpose. Intuitively. the optimal discriminator, which dif- ferentiates qu and qo. implicitly encodes their density ratio, as its output probability directly reflects the relative likelihood of the two distributions. We thus model the reward based on the discriminator.
Lemma 3.2 (Density Ratio Representation). Let Dx VL x [0, 1] (0, 1) be a parameterized discriminator to distinguish samples from the student marginal qu and the teacher marginal qg. For the optimal discriminator Dx (Ze, 1), the density ratio satisfies
(z(, t)
Dx (2-(2i, t)
qo (2.()
1- Dx-((2,)))
Following Lemma 3.2, we construct tractable reward signal R(z,, 1) by aggregating the log-density ratios across all masked positions. Let M denote the number of masked tokens in the sequence. The reward can be estimated using the discriminator DA as follows:
R(z,,t)
1
Σ
log (Z(,))))
(6)
M
DJ (Z,,t)
E, m
Too simplify notation, abhreviate 2!! 24, and x as unless stated otherwise.
4

---

Published as a conference paper at ICLR 2026
where (1(-;)) denotes the ê-th element of Dx(:.). This formulation provides tractable and com- putable reward signal for our objective based on Dx. For more details, please check Appendix C.
3.3 GROUPED REWARD NORMALIZATION
Although the reward estimator in (6) is tractable, its direct use in fcore-function gradients exhibit high variance (Williams, 1992). We therefore adopt Group Relative Policy Optimization (Shao et al., 2024; Team et al., 2025) to standardize rewards within each mini-batch. Given mini-batch of size G, we draw {4+]g from I(), estimate (Z,, ) and (R(z,, 44) 4))== correspondingly The final stabilized reward R2 is obtained by normalizing Rù with the mean Mg and variance og.
C Σ
G 2 - G
R Mg Jg
R =
where Mg
C
R
and
R
-2)) ng)2
1=1
Here, a small constant > 0 is used for numerical stability. Replacing the raw reward R2 with this stabilized reward RG in the gradient estimator yields a more robust and stable update rule.
3.4 THE PROPOSED ALGORITHMS
Adversarial reward estimation. We train parameterized discriminator D> separate corrupted samples from qu and qo at time t. For each mini-batch index i, we start from the all-masked sequence sample x Py and x' ~ Po. then corrupt both to the same noise level t via (2) to obtain Q and 2i Q. The discriminator is optimized with balanced binary cross-entropy:
G
1 log Dx t.) + log(1 Dx(z), t.))
(D(A) = - Σ
At optimality (i.e., for X*), the discriminator satisfies (ze, (2(2).)) ())(22).))))) This indicates that logit. Dx provides a tractable estimate of the log-density ratio used in (6).
qu(a;.t)
Remark 3.3. Compared with regression-basse distillation (e.g.. SDTT and DUO), distillation is more delicate to optimize and requires proper regularization, but it is also crucial for achieving high- fidelity generation with very few diffusion steps (Dhariwal & Nichol, 2021). To promote training stability, we (i) initialize both Pu and D from the pre-trained teacher, (ii) warm up Dx, while freezing Pu for an initial phase, and (üi) clip both rewards and gradients in the policy update to avoid exploding updates. Empirically, we monitor discriminator accuracy and observe that consistently stays in a non-saturated regime.
Score function decomposition. While sampling directly from 2,=m (t = 1) to x is suitable for one-step generators, it induces collapse in multi-step regimes: the student never conditions on inter- mediate states and tends toward low-entropy, mode-seeking behavior (Zhu et al., 2025b). To expose the student intermediate corruption levels, we approximate the score from (5) by decomposing at a randomly sampled time 4 ~ (l) and its corresponding state 2;, which gives:
V. log Pa (z. 1) 2 V. log Po (u(z.(z:=m) + Vp log (2,,
Training with the split score (9) exposes the student to a distribution of intermediate states and mitigates entropy collapse, while remaining compatible with the IKL. estimator.
End-to-end training algorithm. The training procedure of DiDi-Instruct alternates between up- dating the discriminator D2 and the student Pa. which is summarized in Algorithm and visualized in Figure 2. Each iteration consists of two phases: First, the discriminator is updated to better dis- tinguish between corrupted samples from pa and pu: We then apply (5) with the normalized reward and the decomposed score to stabilize gradient updates. This alternating scheme yields robust
For clarity and concision, denote R(z); ;t),
(t)) ))
Rd, and Dx (2.. tt) as D, throughout.
5

---

Published as a conference paper at ICLR 2026
Policy Gradient Update
Clean Sample X
Corrupted Sample Zi
Student P
Fully Masked Sequence
Student
[MASK]
Update V,C(v)
[MASK]
Forward Corruption
Discriminator D
Sample 4 ~ (t))
[MASK]
Discriminator Loss
[MASK]
Backprop
CD(A)
Reference Sample
Corrupted
Teacher P8
Sample
x'
Z;
The visualized pipeline of DiDi-Instruct (summarized in Algorithm 1). Given a fully masked input 2, (t 1), both the student Р. and the teacher (assistant model) Pe produce clean samples x and x', which are corrupted at t, ~ π (t) to form z; and z/- The discriminator Dx is trained classify these outputs, while its reward signal (6) enables the gradient update (5) for the student. The red line represents the gradient flow for the student's update step, and the blue line represents the gradient flow for the auxiliary model's update step.
Figure 2:
few-step student generator that closely matches the teacher's marginals over different corrupt levels.
Algorithm InDtruntt Training
Algorithm 2 RGAS Inference
for each training step do
Initialize ZN (m..... (m..... m). for n N..... do if n in early stage then
Sample t, (t). and
X ~ Pu (z..t Corrupt x, x' to partially masked z;, z/.
1). ~ (z, t 1).
Sample Z..- with (10) (h: > 0, M 1). else
Update discriminator DA with (8).
Update student Рa with (5)-(7), and (9).
Sample 2. with (10)-(11) 0. 2 1). Set Zg and return sequence x.
return student P. and discriminator Dx.
Reward-guided ancestral sampler. We further propose decoding strategy that leverages the trained discriminator to guide ancestral sampling (AS) (Shi et al., 2024; Zheng et al., 2025). Start- ing from fully masked sequence nN = (m,.. .., m) at ix 1, the procedure generates samples by iteratively denoising from t, to in-1 for n N...., 1, following the student's 's backward distri- bution p.. (2n- 2.). Specifically, for z!, E V (L =1, L), consider tilted transition:
* m,
z
(1 (n-am + (an-1 an P Znn tw)
(10)
Cat
if = m,
On
where the logits for tokens to unmasked are augmented with the reward gradient:
Czf.
if 2 * m
(z (n)
( tn) AVR(2, t).
if z m.
Here, ez E V denotes the one-hot vector with 1 at index z and O elsewhere, and h denotes the tilting scale. RGAS adjusts and the number of candidates M across the denoising process: For early steps (t,, ss 1), we use gradient tilting (h 0, AM 1) to steer global structure toward high- reward regions. For late steps (tn, RS 0), we switch to multi-candidate re- re-ranking (h = M, 1); M from (10) with h = 0. then select 2n according to:
we draw M candidates (2(mm) mm1=l
m==
exp [[
(m)
Zn-1 ~ Cat
(1I)
M exp R(z
t
FFor simplicity, we denote the m-th candidate of 1:d. (m)
zn

---

Published as a conference paper at ICLR 2026
To decompose estimation in (9) and sample the intermediate state z, t,. set h = 0 and M = 1 to mitigate the risk of reward hacking (Skalse et al., 2022; Gao et al., 2023). The complete procedure is summarized in Algorithm 2.
EXPERIMENTS
Experimental setup. We distill pre-trained teacher model into an efficient few-step student gener- ator with DiDi-Instruct. All models are trained on OWT (Gokaslan et al., 2019). Following standard practices (Sahoo et al., 2025), tokenize the corpus using the GPT-2 tokenizer, pack sequences to a context length of 1024, and hold out the last 100,000 documents for validation.
The teacher is 169M parameter MDLM with Diffusion Transformer (Peebles & Xie. 2023) (12 layers, 12 attention heads, 768 hidden dimension). We pre-trained this model from scratch for 1024 NFEs, achieving perplexity of 38.53 and entropy of 5.22. The student model shares an identical architecture to ensure a fair comparison. The reward model is a 131M parameter network based on the same backbone, but with new randomly initialized classification head. This head consists of two linear layers with spectral normalization and a SiLU activation function. During distillation, the reward model and the student generator are trained adversarially in an alternating fashion.
The distillation process runs 10,000 iterations using the AdamW optimizer with learning rate of 10 and no warm-up. The teacher model was pre-trained 8 NVIDIA H100 GPUs. Subsequently, our DiDii-nstruct distillation is highly efficient, requiring only single H100 GPU. All training procedures leverage bf boat1 for acceleration. Additional details are provided in Appendix D.
4.1 EXPERIMENTAL RESULT AND ANALYSIS
This section shows that DiDi-Instruct not only achieves state-of-the oftt---rrr performance in sample qual- ity, particularly in the few-step generation regime, but also offers substantial improvements in train- ing and inference efficiency.
Generation results. We evaluate generative quality by sampling text from the distilled student from 8 to 128 NFEs and measuring GPT-2 Large generative PPL and average sequence entropy. Figure shows that DiDi-Instruct consistently outperforms all baselines in PPL across all sampling steps. Notably, with only 16 NFEs, our model's PPL already surpasses that of the 1024-step teacher model. At 1024 NFEs, DiDii-nstruct achieves a final PPL of 15.62, reduction of over 24% compared to the strongest baseline. These performance gains are achieved with negligible loss in diversity; the generative entropy from 8 to 128 NFEs 5.17, 5.21, 5,18, 5.15, and 5.15, respectively (c.g.. 1024-step teacher model is 5.22), indicating sample diversity is well- well-preserved.
We further assess sample fidelity and diversity using MAUVE (Pillutla et al., 2021) and Self BeUf (Papineni et al., 2002; Montahaei et al., 2019). In appendix E.1, DiDi-Instruct achieves superior distribution matching (higher MAUVE) and reduced mode collapse (lower Self-BLEU) compared to SDTT and DUO, confirming the improvement comes without compromising generative diversity.
tradeoff. DiDi-Instruct offers substantial computational advantages in both training and inference. Our single-round distillation framework completes training in around one H100 GPU hour, in contrast to the 20+ GPU hours required by multi-round methods (Sahoo al., 2024; Deschenaux & Gulcehre, 2025). During inference, DiDi-Instruct demonstrates supe- rior throughput-latengy profile. Under a standardized benchmark on a single H100 GPU, it achieves 2366 tokens/sec, representing 13.2x speedup over an AR model of the same size at matched perplexity. We provide detailed benchmarks and setup configurations in Appendix E.2.
Zero-shot likelihood. Distilling model for few-step generations can risk degrading its core lan- guage understanding and causing mode collapse. We assess this trade-off by evaluating zero-shot PPL on seven out-of-domain corpora (PTB, WikiText, LMIB, LAMBADA, AG News, PubMed, and ArXiv), with results presented in Table 8. The evaluation shows that DiDi-Instruct strikes an effective balance. It consistently improves upon the DUO distilled baseline, validating our supe- rior distillation process. Crucially, while it trails some of the full, undistilled models as expected, it maintains a highly competitive level of performance. These results confirm that DiDi- achieves its goal of sampling efficiency while preserving robust zero-shot generalization.
7

---

Published as a conference paper at ICLR 2026
Sample quality. Appendix E.10 presents full text excerpts with per-sample metrics (PPL and en- tropy) for the 1024-step teacher and clear improvement in narrative quality. The 8-step student exhibits expected repetition, common artifact of rapid sampling. However, this issue is resolved by 16 NFEs, at which point the student model already surpasses the 1024-step teacher in paragraph-level coherence and topic adherence. This ability to construct focused and specific narratives strengthens as NFEs increase to 128, in- dicating that DiDi-Instruct not only preserves fluency but actively enhances the model's ability to generate structured, coherent text.
students from 8 to 128 NFEs, which confirms a
4.2 ABLATION STUDIES
We conduct comprehensive ablation studies to validate the contribution of each component in DiDi- Instruct. We perform two types of analyses: cumulative study (Table 1) that progressively adds techniques to a baseline, showing their synergistic benefits, and leave-one-out study (Table 2) that removes individual components to confirm their necessity. Implementation details can refer to E.5,
Table 1: Cumulative ablation study. We start from a baseline model and progressively add the listed tricks on top of the previous row (top>bootomm) Metrics are reported as PPL. and Entropy? for different NFEs. The teacher with 1024 NFEs yields entropy 5.22.
8 NFEs
16 NFEs
32 NFEs
64 NFEs
128 NFEs
Configurations
PPL.
Entropyt
PPL1
Entropy
PPL1
Entropyt
PPL.
Entropy?
PPL.
Entropy
Baseline (no tricks)
803.922
5.85
311.450
5.76
174,789
5.70
113.112
5.61
96.649
5.59
+ Score Decompose
667.830
5.83
289.720
5.76
165.809
5.70
105 880
5.61
89.350
5,59
+ Coupled Time
101.019
5.16
75.188
5.46
48.441
5.35
35.833
5.37
30.574
5.33
w(f)) Correction
94,955
5.21
75.607
5.22
31.651
5.20
25.271
5.16
20.980
5.12
+ x(t) Weighting
92,100
5.15
43.997
5.17
32.276
5.21
26.079
5.21
21.377
5.13
Regularization
88.274
5.11
43,980
5.16
28.444
5.12
21.946
5.06
18.325
5.00
+ Guided Inference
62.236
5.17
38.188
5.21
24.971
5.18
21.905
5.15
18.446
5.15
Table 2: Leave-one-out ablation study. Each row shows performance without (w/o) one trick while keeping the others unchanged. Metrics are PPL. and Entropyt over different NFEs. The lowest PPL in each NFE column is underlined.
NFEs
16 NFEs
NPES
64 NFEs
128 NFEs
Configurations
PFL.
Entropyt
PPL.
Entropy
PPL
Entropyt
PPL.
Entropyt
PPL.
Entropy
w/o Score Decompose
33584
6.77
28962
6.77
23134
6.75
14634
6.64
7983
6.51
wlo Coupled Time
360.75
5.42
159.43
5.43
94.859
5.45
64.639
5.35
51.121
5.39
wlo (!)) Correction
82.489
5.12
41.034
5.13
30.313
5.09
25.125
5.04
18,806
5.02
w/o (() Weighting
5.22
40.499
25.799
5.15
21.503
5.16
19,616
5.14
who Regularization
84.594
5.20
30.994
5.22
23.603
5.20
19.609
5.18
17.499
5.17
Guided Inference
5.11
21.946
5.06
18.325
Baseline (with all tricks)
62.256
5.17
38.188
5.21
24.971
5.18
21.905
5.15
18.446
5.15
Our ablation studies reveal distinct roles for each component in our framework. We find that a two-step score decomposition is non-negotiable cornerstone, providing essential stability without which the model fails to train. The most significant performance gains are driven by coupling in (5) and (9). Loss shaping with w({) and π (1) further smooths optimization (especially around 16 NFEs, while effects at very small/large budgets are modest). Finally. we identify two budget-dependent components: Regularization is crucial for stability at very few NFEs (< 8 NFEs) but detrimental at higher budgets, while Guided Inference boosts quality at low NFEs and enhances diversity at high NFEs. These highlight a hierarchy of importance and nuanced interactions between the techniques.
To further explore RGAS, we analyze the hyperparameters h and M through performance landscape seans and Functional ANOVA (Hutter et al., 2014). Our analysis reveals that low NFEs (< 16), dominates performance by guiding global structure; the computational budget increases (e.g., 32 NFEs), the importance of M rises significantly. More details are presented Appendices E.5-E.7.
4.3 DOWNSTREAM TASK EVALUATION
To validate the practical utility. we evaluate DiDi-Instruct and baselines on downstream tasks. Specifically, conduct experiments Domain Adaptation (where we fine-tune models the MMLU (Hendrycks et al., 2021) and PubMed benchmarks), and Frozen Feature Extraction (using the GL GLUE MRPC dataset (Dolan & Brockett, 2005)).
8

---

Published as a conference paper at ICLR 2026
On MMLU fine-tuning, DiDi-Instruct achieves the largest reduction in negative log-likelihood after 5,000 fine-tuning steps while matching teacher accuracy. On PubMed, DiDi-Instruct nearly matches the teacher's s perplexity while outperforming other distillation methods. Moreover, in frozen-feature evaluation on GLUE MRPC, DiDiiInstruct achieves the best accuracy and F1 score, demonstrating superior semantic representation quality. These results confirm that DiDtitnstruct maintains strong downstream performance while providing substantial efficiency gains. See Appendix E.4 for details.
4.4 SCALING UP TEACHER MODEL SIZE
We conduct an incremental scaling study 424M parameters while keeping the training and infer- ence pipeline identical to the 169M configuration. The teacher model a pre-trained 424M MDLM, which is DiT-style transformer with hidden size 1024, 24 blocks, and 16 attention heads (See Ta- ble for network architecture details). The student is distilled on a single H100 using the same data, masking policy, optimizer, and schedule. To balance the training, we also scale the reward discriminator to 373M parameters by adopting a deeper network with more trainable parameters.
Figure 4 reports generative PPL and entropy. We report improvements relative to the teacher matched NFEs. From 8 to 1024 NFEs, DiDi-Instruct yields large and consistent PPL reductions rel- ative to the teacher: 88.5% (8 NFEs), 87.7% (16 NFEs), 85.3% (32 NFEs), 82.7% (64 NFEs), and 79.0% (128 NFEs). Notably, with only 16 NFEs, the distilled model reaches a perplexity of 32.79, marking an 1 11.4% improvement over the 1024-step teacher baseline. Entropy remains comparable under the same NFE budgets, indicating that diversity is preserved while accuracy improves. Over- all, these results confirm that the quality-efficieney advantages of DiDii-nstruct persist at a larger scale with minimal procedural changes. Additional details are provided in Appendix E.8.
4.5 ANOTHER APPLICATION: PROTFIN SEQUENCES GENERATION
To demonstrate the applicability of our distillation framework beyond natural language generation, we apply DiDi-Instruct to unconditional protein sequence generation. We adopt the Diffusion Pro- tein Language Model (DPLM) (Wang al., 2024). pretrained on UniRef50 (Suzek al., 2014) with 150M parameters, as the teacher model and distill it into a few-step student generator. Follow- ing (Wang et al., 2024), we evaluate sequence quality using the predicted local distance difference test (pLDDT) score, which reflects structural plausibility and foldability. The distilled model re- tains the teacher's ability to generate variable-length protein sequences while substantially reducing inference cost, as shown in 5igure
Our results demonstrate that the distilled student consistently achieves superior pLDDT scores across generation settings ranging from 8 to 512 NFEs. Compared to the teacher model, the stu- dent not only preserves the ability to generate variable-length protein sequences but also enhances structural quality in most cases. Moreover, (pLDDT > 70) with as few as 8-32 NFEs, while the teacher requires substantially more NFEs to reach a comparable level. These results highlight that distillation framework not only improves structural confidence but also delivers stable performance across sequence lengths and generation budgets. Appendix E.9 includes detailed setup, and visual comparisons between low-confidence outputs of DPLM at small NFEs and high-quality samples produced by DiDi-Instruct (Figures 6-7).
model surpasses the high-confidence threshold
To ensure that the observed improvements in structural confidence (pLDDT) do not stem from mode collapse, we evaluate sequence diversity using MMsegs2 clustering (Steinegger & Söding, 2017). Quantitative results confirm that ter sizes comparable to the teacher model, particularly in few-step regimes. This indicates that our method generates diverse, biologically meaningful sequences without collapsing few high- confidence modes. Detailed results are provided in Appendix F.9.
maintains competitive cluster entropy and low clus-
CONCLUSION AND FUTURE WORK
In this work, we introduced DiDi-Instruct a craining-based acceleration framework for fast lan- guage generation that distills high-quality teacher into few-step student. Our design targets three axes simultaneously: (i) objective design via a tractable policy-gradice update driven by a discrim- inator-estimated reward; (ii) training stability through score decomposition and grouped reward

---

Published as a conference paper at ICLR 2026
normalization; and (iii) inference efficiency through seward-guides ancestral sampling with gradient tilting and re-ranking. Experiments demonstrate strong gains in generation quality, large reductions in training/Inference time, and competitive zero-shot generalization, corroborated by comprehensive cumulative and leave-one-out ablations.
We plan to scale DiDiiInstruct due to the memory requirements of concurrently maintaining the teacher, student, and discrimina- tor. Nevertheless, our findings already establish new state-of-the-art trade-off among comparable methods, with the student model excelling in quality at low NFEs and maintaining diversity at higher computational budgets. We posit that DiDi-Instrue offers foundational recipe (principled objec- tives, training stability. and efficient guidance) for developing high-performance generative models.
iillion-paramedd models, which presents a practical challenge
ACKNOWLEDGMENTS
We thank the Core AI cloud platform for supporting part of the computational resources used in this work. Nan Jiang acknowledges support from the Texas Advanced Computing Cen- ter (TACC) under award CCR25054. Guang Lin acknowledges support from the National Science Foundation (NSF) under grants DMS-253388
2MS-2053746, DMS-21 DMM-2214409, ECCS-2328241, CBET-2347401, and OAC-231 1848; the U.S. Department of Energy (DOE) Office of Science, Ad- vanced Scientific Computing Research program under award DE-SC0023161; the SciDAC LEADS Institute; and the DOE Fusion Energy Sciences program under grant DE-SC0024583.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 Techni- cal Report. arXiv preprint wrxhv:2303.08774 2023.
Marianne Arriola, Aaron Gokaslan, Justin T. Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Sub- ham Sekhar Sahoo, and Volodymyr Kuleshov. Block Diffusion: Interpolating Between Autore- gressive and Diffusion Language Models. In Proc. of the International Conference on Learning Representation (ICLR), 2025.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Struc- tured Denoising Diffusion Models in Discrete State-Spaces. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pp. 17981-17999 2021.
Nicholas Matthew Boffi, Michael Samuel Albergo, and Eric Vanden- Flow map matching with stochastic interpolants: A mathematical framework for consistency models. Transactions on Machine Learning Research, 2025. ISSN 2835-8856.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language Models Few-Shot Learners. In Advances in Neural Information Processing Systems (NeurIPS), vol- ume 33, pp. 1877-1901, 2020.
Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A Continuous Time Framework for Discrete Denoising Models. Advances in Neural Information Processing Systems (NeurIPS), 35:28266-28279, 2022.
Tianyu Chen, Zhendong Wang, and Mingyuan Zhou. Diffusion Policies Creating Trust Region for Offline Reinforcement Learning. In Advances Neural Information Processing Systems (NeurIPS), volume 37, pp. 50098-50125, 2024.
Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog Bits: Generating Discrete Data using Diffusson Models with Self-Conditioning Proc. of the International Conference on Learning Representation (ICLR), 2023.
Justin Deschenaux and Caglar Gulcehre. Beyond Autoregression: Fast LLMs via Self-Distillation Through Time. In Proc. of the International Conference on Learning Representation (ICLR). OpenReview.net, 2025.
10

---

Published as a conference paper at ICLR 2026
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proc. of the Annual Meeting of the Association Computational Linguistics (ACL), pp. 4171-4186, 2019.
Prafulla Dhariwal and Alexander Nichol. Diffusion Models Beat GANs Image Synthesis. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pp. 8780-8794, 2021.
Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer. Conor Durkan, Conor Durkan, Curtis Hawthorne, Rémi Leblond, Will Grathwohl, and Jonas Adler. Continuous Diffusion for Categor- ical Data, arXiv preprint arXiv:2211.15089 2022.
William B Dolan and Chris Brockett. Automatically Constructing a Corpus of Sentential Para- phrases. In Proceedings of the Third International Workshop on Paraphrasing (/WP), 2005.
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. DPOK: Reinforcement Learning for Fine- Text-to-Image Diffusion Models. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pp. 79858-79885, 2023.
Leo Gao, John Schulman, and Jacob Hilton. Scaling Laws for Reward Model Overoptimization. In Proc. of the International Conference Machine Learning (ICML), volume 202, pp. 10835- 10866. PMLR, 2023.
Zhengyang Geng, Ashwini Pokle, and Zico Kolter. One-Step Diffusion Distillation via Deep Equilibrium Models. Advances in Neural Information Processing Systems (NeurIPS), 36:419944 36:41914- 41931, 2023.
Zhengyang Geng, Mingyang Deng, Xingjian Bai, Zico Kolter, and Kaiming He. Mean flows for one-step generative modeling. Advances in Neural Information Processing Systems (NeurIPS), 2025a.
Zhengyang Geng, Ashwini Pokle, Weijian Luo, Justin Lin, and J. Zico Kolter. Consistency Mod- els Made Easy. Proc. of the International Conference on Learning Representation (ICLR). OvenRevie 2025b.
Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. OpenWebText Corpus. http: //Skylion007 gi tuu.huu..... io/ Owwp///eeeeetoopppaaiioa 2019.
Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models. In Proc. of the International Conference Learning Representation (ICLR), 2023.
lan J Goodfellow, Jean Pouget-Abadlis Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 27, 2014.
Jiatao Gu, Chen Wang, Shuangfei Zhai, Yizhe Zhang, Lingjie Liu, and Joshua M. Susskind. Data- free distillation of diffusion models with bootstrapping. In Proc. of the International Conference on Machine Learning (ICML), volume 235, pp. 16622-16646. PMLR, 2024.
Ishaan Guirajani and Tatsunori Hashimoto. Likelihood- -ased Diffusion Language Models. In Ad- vances in Neural Information Processing Systems (NeurIPS), volume 36, pp. 16693-16715, 2023.
Zhengfu He, Tianxiang Sun, Qiong Tang, Kuanning Wang, Xuanjing Huang, and Xipeng Qiu. Dif- FuSionBERT: Improving Generative Masked Language Models with Diffussinn Models. In Proc. of the Annual Meeting of the Association Computational Linguistics (ACL), pp. 4521-4534, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazcika, Dawn Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understanding. In Proc. of the International Conference on Learning Representation (ICLR), 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffussin Probabilistic Models. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 6840-6851, 2020.

---

Published as a conference paper at ICLR 2026
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions. Advances in Neural Information Processing Systems (NeurIPS), 14:12444-124655 2021.
Zemin Huang, Zhengyang Geng, Weijian Luo, and Guo-jun Qi. Flow Generator Matching. arXiv preprint arXiv:2410. 19310, 2024.
Frank Hutter, Holger Hoos, and Kevin Leyton- An Efficient Approach for Assessing Hyper- parameter Importance. In Proc. of the International Conference on Machine Learning (ICML), volume 32, pp. 754-762. PMLR, 2014.
Tero Karras, Mika Aittala, Timo Aila, and Samuli Laine. Elucidating the Design Space of Seffusiio-Based Generative Models. Advances in Neural Information Processing Systems (NeurIPS), 35:26565-2653 2022.
Samar Khanna, Siddhant Kharbanda, Shufan Li, Harshit Varma, Eric Wang, Sawyer Birnbaum, Ziyang Luo, Yanis Miraoui, Akash Palrecha, Stefano Ermon, Aditya Grover, and Volodymyr Kuleshov. Mercury: Ultra-Fast Language Models Based on Diffusion. arXiv preprint vrXiv:2506.17299 2025.
Dongjun Kim, Chich-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency Trajectory Models: Learning Prob- ability Flow ODE Trajectory of Diffusion. In Proc. of the International Conference on Learning Representation (ICLR), 2024.
Jaeyeon Kim, Kulin Shah, Vasilis Kontonis, Sham Kakade, and Sitan Chen. Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions. In Proc. of the International Conference on Machine Learning (ICML), volume 267, 2025.
Schoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W. Mahoney, Amir Gholami, and Kurt Keutzer. Speculative Decoding with Big Little Decoder. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pp.
2023.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast Inference from Transformers via Specula- tive Decoding. In Proc. of the International Conference on Machine Learning (ICML), volume 202, pp. 19274-19286, 2023.
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. Diffusion- LM Improves Controllable Text Generation. Advances in Neural Information Processing Sys- tems (NeurIPS), volume 35, pp. 4328 4343, 2022.
Shanchuan Lin, Anran Wang, and Xiao Yang. SDXL-Lightning: Progressive Adversarial Diffusion Distillation. arXiv preprint arXiv:2402. 13929, 2024.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, Allan dos Santos Costa, Maryam ZaraniZarandi, Tom Sercu, Salvatore Candido, and Alexander Rives. Evolutionary- Prediction of Atomic-Level Protein Structure with Language Model. Science, 379(6637):11233
2023.
Sulin Liu, Juno Nam, Andrew Campbell, Hannes Stärk, Yilun Xu, Tommi Jaakkola, and Rafael GömeezBombaree Think While You Generate: Discrete Diffusion with Planned Denoising. In Proc. of the International Conference Learning Representation (ICLR), 2025.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning generate and transfer data with rectified flow. In Proc. of the International Conference on Learning Represen- tation (ICLR), 2023.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete Diffusson Modeling by Estimating the Ratios of the Data Distribution. In Proc. of the International Conference on Machine Learning (ICML), volume 235, pp. 32819-32848, 2024.
Cheng Lu and Yang Song. Simplifying, Stabilizing and Scaling Continuous-tim time Consistency Models. In Proc. of the International Conference on Learning Representation (ICLR). Open- Review.net, 2025.
12

---

Published as a conference paper at ICLR 2026
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-Solver: A Fast ODE Solver for Diffusion Probabilisticc Model Sampling in Around 10 Steps. Advances in Neural Information Processing Systems (NeurlPS), 35:5775 35:5775-57877 2022.
Eric Luhman and Troy Luhman. Knowledge Distillation in Iterative Generative Models for Im- proved Sampling Speed. arXiv preprint arX1v:2101.02888 2021.
Shanchuan Luo, Xiaohui Wang, Fan Zhang, et al. Latent Consistency Models: Synthesizing High- Resolution Images with Few-Step Inference. arXiv preprint arXiv:2310.0433 04.7788 2023a.
Weijian Luo. Diff-instruct++: Training one-step text-to-image generator model to align with human preferences. Transactions on Machine Learning Research, 2025. ISSN 2835-8856.
Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang. Diff- Instruct: A Universal Approach for Transferring Knowledge From Pre- Trained Diffusion Models. In Advances Neural Information Processing Systems (NeurIPS), volume 36, pp. 76525-76546, 2023b.
Weijian Luo, Ting Zhu, Shifeng Zhang. Zhenguo Li, and Zhihua Zhang. One-Step Diffusion Distil- lation through Score Implicit Matching. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, pp. 115377-1 15408, 2024.
Weijian Luo, Colin Zhang, Debing Zhang, and Zhengyang Geng. David and Goliath: Small One- step Model Beats Large Diffusion with Score Post-training. In Proc. of the International Confer- ence on Machine Learning (ICMI),, volume 267, pp. 41520-41539 PMIR, 2025a.
Yihong Luo, Xiaolong Chen, Xinghua Qu, Tianyang Hu, and Jing Tang. You Only Sample Once: Taming One-Step Text-to-lmage Synthesis by Self-Cooperative Diffusion GANs. In Proc. of the International Conference on Learning Representation (ICLR). OpenReview.net, 2025b.
Yihong Luo, Tianyang Hu, Weijian Luo, Kenji Kawaguchi, and Jing Tang. Reward-Instruct A Reward-Centric Approach to Fast Photo-Realistic Image Generation. arXiv preprint arxti:orgg/bs//250..1100 2025c.
Chenlin Meng. Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In The IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 14297-14306, 2023.
Meta AL Introducing LLaMA: A foundational, a5-billion-paramed large language model. https://aa meta. hamm//llog/largeeeeeee 2023. Accessed: 2025-09-14.
model -ama-meta----
Ehsan Montahaei, Danial Alihosseini, and Mahdich Soleymani Baghshah. Jointly Measuring Di- versity and Quality in Text Generation Models. In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pp. 90-98, 2019.
Thuan Hoang Nguyen and Anh Tran. Swiftbrush: One-step text-to-image diffusion model with vari- ational score distillation. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7807-7816, 2024.
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large Language Diffusion Models. In Advances in Neural Information Processing Systems (NeurIPS), 2025.
Xuefei Ning, Zinan Lin, Zixuan Zhou, Zifu Wang, Huazhong Yang, and Yu Wang. Skeleton-of. Thought: Prompting LLMs for Efficient Parallel Generation. In Proc. of the International Con- ference on Leurning Representation (ICLR), 2024.
Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data. In Proc. of the International Conference Learning Representation (ICIR), 2025.
13

---

Published as a conference paper at ICLR 2026
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Yong-Hyun Park, Chich-Hsin Lai, Satoshi Hayakawa, Yuhta Takida, and Yuki Mitsufuji. Jump Your Steps: Optimizing Sampling Schedule of Discrete Diffusion Models. In Proc. of the International Conference on Learning Representation (ICLR). 2024.
William Peebles and Saining Xie. Scalable Diffusion Models with Transformers. In Proc. of the International Conference on Computer Vision (ICCV), pp. 4195-4205, 2023.
Krishna Pillutia, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. MAUVE: Measuring the Gap Between Neural Text and Human Text us- ing Divergence Frontiers. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pp. 1816-4828 2021.
Yuxi Ren, Xin Xia, Yanzuo Lu, Jiacheng Zhang, Jie Wu, Pan Xie, Xing Wang, and Xuefeng Xiao. Hyper-SD: Trajectory Segmented Consistency Model for Efficient Image Synthesis, In Advances in Neural Information Processing Systems (NeurIPS), volume 37, pp. 117340-117362, 2024.
Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and Effective Masked Diffusion Language Models. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, pp- 130136-130184 2024.
Subham Sekhar Sahoo, Justin Deschenaux, Aaron Gokaslan, Guanghan Wang, Justin Chiu, and Volodymyr Kuleshov. The Diffusion Duality, In Proc. of the International Conference on Learn- ing Representation (ICLR), 2025.
Tim Salimans and Jonathan Ho. Progressive Distillation for Fast Sampling of Diffusion Models. Proc. of the International Conference on Learning Representation (ICLR), 2022.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial Diffusion Distillation. arXiv preprint arX14:2311.17042, 2023.
Yair Schiff, Subham Sekhar Sahoo, Hao Phung, Guanghan Wang, Sam Boshar, Hugo Dalla-torre, Bernardo P. de Almeida, Alexander Rush, Thomas Pierrot, and Volodymyr Kuleshov. Simple Guidance Mechanisms for Discrete Diffusion Models. In Proc. of the International Conference Learning Representation (ICLR), 2025.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv preprint a 1ee010077777777 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv preprint arXiv:2402.033300 2024.
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and General- ized Masked Diffusion for Discrete Data. In Advances Neural Information Processing Systems (NeurIPS), volume 37, pp. 103131-103167, 2024.
Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and Charac- terizing Reward Gaming. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pp. 9460-9471, 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proc. of the International Conference on Machine Learning (ICML), pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng. and Stefano Ermon. Denoising Diffusion Implicit Models. In Proc. of the International Conference on Learning Representation (ICLR), 2021a.
Yang Song and Prafulla Dhariwal. Improved Techniques for Training Consistency Models. In Proc. of the International Conference Learning Representation (ICLR). OpenReview.net, 2024.
14

---

Published as a conference paper at ICLR 2026
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-Based Generative Modeling through Stochastic Differential Equations. In Proc. of the International Conference Learning Representation (ICLR), 2021b.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency Models. In Proc. of the International Conference Machine Learning (ICML), volume 202, pp. 32211-32223 PMLR, 2023.
Yuxuan Song, Zheng Zhang. Cheng Luo, Pengyang Gao, Fan Xia, Hao Luo, Zheng Li, Yuchang Yang, Hongli Yu, Xingwei Qu, et al. Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference. arXiv preprint arXiv:2508.00 02193, 2025.
Martin Steinegger and Johannes Söding. MMseqs2 Enables Sensitive Protein Sequence Searching for the Analysis of Massive Data Sets. Nature Biotechnology, 35(11):1026-100) 2017.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer: En- hanced transformer with Rotary Position Embedding. Neurocomputingg 568: 127062, 2024. ISSN 0925-2312.
Baris E. Suzek, Yuqi Wang, Hongzhan Huang, Peter B. McGarvey, Cathy H. Wu, and the UniProt Consortium. Uniref Clusters: Comprehensive and Scalable Alternative for Improv- ing Sequence Similarity Searches. Bioinformatics, 31(6):926-93 932, 2014.
Gemini Team, Petko Georgiev. Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan. Shibo Wang, et al. Gemini 1.5: Unlocking Multimodal Under- standing Across Millions of Tokens of Context. arXiv preprint arXiv:2403.055300 2024.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl.5: Scaling Reinforcement Learning with LLMs. arXiv preprint arXiv:2501.12550 2025.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothéc Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Foundation Language Models. arXiv preprint ar 233v 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In Advances in Neural In- formation Processing Systems (NeurIPS), pp. 5998-6008, 2017.
Xinyou Wang, Zaixiang Zheng, Fei Ye, Dongyu Xue, Shujian Huang, and Quanquan Gu. Diffu- sion language models are versatile protein learners. In Proc. of the International Conference on Machine Learning (ICML), volume 235, pp. 52309-52333, 2024.
Yifei Wang, Weimin Bai, Colin Zhang, Debing Zhang, Weijian Luo, and He Sun. Uni-Instruct: One- step Diffusion Model through Unified Diffusion Divergence Instruction. In Advances in Neural Information Processing Systems (NeurIPS), 2025.
Zhendong Wang, Huangjie Zheng, Pengeheng He, Weizhu Chen, and Mingyuan Zhou. Diffusion- GAN: Training GANs with Diffusion. In Proc. of the International Conference on Learning Representation (ICLR), 2023.
Ronald J Williams. Simple Statistical Fralient-Following Algorithms for Connectionist Reinforce- ment Learning. Machine Learning, 8(3-4):229-255 1992.
Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. Fast-dLLM: Training-fre free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding. In ICLR, 2026.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the Generative Learning Trilemma with Denoising Diffusion GANs. In Proc. of the International Conference on Learning Representation (ICLR), 2021.
15

---

Published as a conference paper at ICLR 2026
Sirui Xie, Zhisheng Xiao, Diederik P. Kingma, Tingbo Hou, Ying Nian Wu, Kevin P. Murphy, Tim Salimans. Ben Poole, and Ruiqi Gao. EM Distillation for One-step Diffusion Models. In Advances in Neural Information Processing Systems (NeurlPS), volume 37, pp. 45073-45104 2024.
Zhihui Xie, Jiacheng Ye, Lin Zheng, Jiahui Gao. Jingwei Dong, Zirui Wu, Xueliang Zhao, Shansan Gong, Xin Jiang, Zhenguo Li, et al. gream-Coder 7B: An Open Diffusion Language Model for Code. arXiv preprini arXiv:2509.01 142, 2025.
Yanwu Xu, Yang Zhao, Zhisheng Xiao, and Tingho Hou. Ufogen: You forward once large scale text-to-image generation via diffusion gans. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8196-8206, 2024.
Yilun Xu, Weili Nie, and Arash Vahdat. One-step Diffusion Models with f-Divergence Distribution Matching. arXiv preprint arXiv:2502.15681 2025.
Shuchen Xue, Mingyang Yi, Weijian Luo, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhi-Ming Ma. SA-Solver: Stochastic Adams Solver for Fast Sampling of Diffusion Models. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pp. 77632-77674, 2023.
Hanshu Yan, Xingchao Liu, Jiachun Pan, Jun Hao Liew, Qiang Liu, and Jiashi Feng. Perflow: Piecewise rectified flow as universal alug-and-play accelerator. In Advances in Neural Informa- lion Processing Systems (NeurIPS), volume 37, pp. 78630-78652, 2024.
Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7B: Diffusion Large Language Models. arXiv preprint arXiv:2508. 15487, 2025.
Tianwei Yin, Michaäl Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Frédo Durand, and Bill Freeman. Improved Distribution Matching Distillation for Fast Image Synthesis. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, pp. 47455-47487, 2024a.
Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Frédo Durand, William T. Freeman, and Taesung Park. One-Step Diffusion with Distribution Matching Distillation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6613-6623, 2024b.
Oussama Zekri and Nicolas Boullé. Fine-Tuning Discrete Diffusion Models with Policy Gradient Methods. In Advances in Neural Information Processing Systems (NeurIPS), 2025.
Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. In Proc. of the International Conference on Learning Representation (ICLR), 2023.
Yixiu Zhao, Jiaxin Shi, Feng Chen, Shaul Druckmann, Lester Mackey, and Scott Linderman. In- formed Correctors for Discrete Diffusion Models, In Advances in Neural Information Processing Systems (NeurIPS), 2025.
Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming-Yi Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling. In Proc. of the International Conference on Learning Representation (ICLR), 2025.
Mingyuan Zhou, Huangjie Zheng, Zhendong Wang, Mingzhang Yin, and Hai Huang. Score Identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Gener- ation. In Proc. of the International Conference on Machine Learning (ICML), volume 235, pp. 62307-62331. PMLR, 2024.
Mingyuan Zhou, Huangjie Zheng, Yi Gu, Zhendong Wang, and Hai Huang. Adversarial Score identity Distillation: Rapidly Surpassing the Teacher in One Step. In Proc. of the International Conference on Learning Representation (ICLR). OpenReview.net, 2025.
Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chunwei Wu, Jun Hu, Jun Zhou, Jianfei Chen, Yankai Lin, Ji-Rong Wen, et al. LLaDA 1.5: Variance- Preference Optimization for Large Language Diffusion Models. arXiv preprint urXiv:2505. 12.5505..2225 19223, 2025a.
Yuanzhi Zhu, Xi Wang, Stéphane Lathuilière, and Vicky Kalogeiton. DIMO: Distilling Masked Diffusson Models into One-step Generator. In Proc. of the International Conference on Computer Vision (ICCV), pp. 18606-18618, 2025b.
16

---

Contents
1
Introduction
2
Preliminary
2.1
Masked Diffusion Models
2.2
Integral KI. divergence
3
Methodology
3.1
Discrete Diffusion Divergence Instruction
3.2
Density-Ratio Estimation
3.3
Grouped Reward Normalization
3.4
The Proposed Algorithms
Experiments
4.1
Experimental Result and Analysis
4.2
Ablation Studies
8888
4.3
Downstream Task Evaluation
4.4
Scaling Up Teacher Model Size
4.5
Another Application: Protein Sequences Generation
5
Conclusion and Future Work
Related Works
18
A.1
Related Work Continuous-space Diffusion Distillation
18
A.2 Related Work on Discrete Diffusion Models
18
A.3 Related Work Few-Step Diffusion Large Language Models
19
B Student Objective Derivation
20
Auxiliary Discriminator for Density Ratio Estimation
22
Additional Experimental Setup
23
D.I Baseline Setup
23
D.2
Model Architecture
D.3
Dataset Details
................................
24
D.4
Training and Distillation Hyperparameters
25
D.5
Evaluation Details..
25
E
Additional Experimental Results
25
E.
Diversity and Fidelity Analysis
25
E.2
Practical Latency and Throughput
................................
26
E.3
Results on zero-shot perplexities
27
E.4
Downstream Task Evaluation
27
E.5
Implementation details the ablation studies
28
E,6
Ablation study result analysis.
29
E.7
Sensitivity Analysis of RGAS
30
E.8
Model Scaling Up...
E,9
Protein Sequence Generation
32
E.10
Text examples
34
17

---

Published as a conference paper at ICLR 2026
A RELATED WORKS
A.1 RELATED WORK ON CONTINUOUSSPPACE DIFFUSION DISTILLATION
Diffusion models (Song et al., 2021b; Ho et al., 2020; Sohl-Dickstein et al., 2015) perturb data by adding Gaussian noise in a forward diffusion process and then learn the reverse-time dynamics (formulated as a stochastic differential equation) using score networks. Early efforts to acceler- ate sampling reduced the number of calls iterative samplers using training-free high-order ODE solvers (Song et al., 2021a; Lu et al., 2022; Karras et al., 2022: Xue et al., 2023). However, such solver-based acceleration still suffered from discretization error in less than ten generation steps.
To achieve few-step generation, Luhman & Luhman (2021) and Salimans & Ho (2022) first intro- duce training the few-step student models by learning the consecutive trajectory mapping of diffu- sion solvers. Subsequently, the seminal work of Song et al. (2023) and Luo et al. (2023b) opens the one-step generation of diffusion models through different distillation principles. Specifically, consistency models (Song et al., 2023) distill the few-step models with the trajectory consistency principle, resulting in strong performances (Song & Dhariwal, 2024; Lu & Song, 2025; Geng et al., 2025b; Luo et al., 2023a; Kim et al., 2024; Geng et al., 2025a). Diff-Instruct (Luo et al., 2023b) introduces the distribution matching principle that distills one-step generative models, resulting in leading efficient image generative models (Wang et al., 2025; Luo et al., 2025a; 2024; Luo, 2025; Xu et al., 2025; Yin et al., 2024b;a; Xie et al., 2024; Fan et al., 2023; Zhou et al., 2024: Huang et al., 2024; Luo al., 2025b;c). Later, many other works have also studied the few-step continuous-space generative models from different perspectives (Liu et al., 2023; Geng et al., 2023; Zhou et al., 2025; Nguyen & Tran, 2024; Lin al., 2024; Boffi et al., 2025; Xu al., 2024; Xiao et al.. 2021; Meng et al., 2023; Zhang & Chen, 2023; Sauer et al., 2023; Ren et al., 2024; Yan et al., 2024; Gu et al., 2024: Chen al., 2024).
Inspired by Luo et al. (2023b), this work leverages the distribution matehing principle to tackle the more challenging problem of discrete language sequence generation.
A.2 RELATED WORK ON DISCRETE DIFFUSION MODELS
Early studies of discrete diffusion modeled categorical data via multinomial or argmax-based tran- sitions (Hoogeboom et al., 2021). This line of work was later generalized by D3PM, which intro- duced structured transition matrices (such as discretized Gaussian kernels, nearest-neighbor transi- tions, and absorbing states) together with an auxiliary cross-entropy objective (Austin et al., 2021). Continuous-time perspectives and relaxations further clarified the connection between discrete cor- ruption processes and stochastic dynamics (Campbell et al., 2022; Dieleman et al., 2022: Chen et al., 2023). In parallel, diffusion-based ideas were also explored for controllable text generation and sequencceto-sequenee modeling (Li et al., 2022; Gong et al., 2023).
For language modeling. two principal families of dLLMs have emerged. MDMs (Lou et al., 2024; Sahoo et al., 2024; Shi et al., 2024: Ou et al., 2025) treat the forward process as progressive masking of tokens. At the highest noise level, every token is replaced by special [MASK] symbol; the reverse process gradually unmask tokens. Once a token is unmasked, it remains fixed, but many to- kens can be denoised simultaneously, Intuitively, MDMs essentially train BERT (Devlin al., 2019) under a hierarchy of noise levels, motivated by scaling-based rationale for generation (He et al., 2023; Sahoo al., (0224),niffomm.
diffuuson models (USDMs) (Austin et al., 2021: Gulrajani & Hashimoto, 2023; Schiff et al., 2025; Sahoo et al., 2025) instead corrupt tokens by sampling from a uniform distribution over the vocabulary. Consequently, each token can change multiple times dur- ing sampling, enabling self-correction and strong theoretical connections to continuous Gaussian diffusion.
Additionally, recent work on dLLMs has advanced along several fronts, including training objectives and theoretical foundations (Lou et al., 2024; Sahoo et al., 2024; Shi et al., 2024; Ou et al., 2025; Zheng et al., 2025), decoding and sampling efficiency (Zheng et al., 2025; Kim et al., 2025; Zhao et al., 2025; Park et al., 2024: Liu et al., 2025), and large-scale model development (Nie et al., 2025: Zhu et al., 2025a; Song et al., 2025; Ye et al., 2025; Xie al., 2025; Khanna et al., 2025). Together, these studies have substantially improved our understanding of dLLMs and have made baased language generation increasingly effective and scalable.
18

---

Published as a conference paper at ICLR 2026
Our work is most closely related to this recent line of dLLMs, but differs in focus. Rather than de- signing a new corruption process or decoding heuristic, we develop a distillation framework for few- step generation based on distribution matching. Inspired by the continuous-ting IKL perspective, our method is tailored to models so as to maintain tractable training and inference while improving generation efficiency and quality.
A.3 RELATED WORK ON FEW-STEP DIFFUSION LARGE LANGUAGE MODELS
To distill dLLMs into few-step generators for fast language generation, a natural approach is to ex- tend the well-devcloped distillation techniques already established for continuous diffusion models to the discrete domain. However, in MDMs, all tokens eventually mapped to the masked state, meaning the prior distribution collapses to the fully masked sequence. As result, generation under MDMs is inherently stochastic, in contrast to the deterministic trajectories available in the contin- uous case. Consequently, the stochasticity in Masked Diffusion Models (MDMs) arises from the choice of the masking schedule; once a schedule is fixed, the reverse path is deterministic. This stands in sharp contrast to the inherently stochastic reverse dynamics of continuous diffusion. This determinism problematic for distillation, as training a student model solely on these fixed trajec- tories risks mode collapse and failure to capture the teacher's generalization capabilities. This fundamental difference makes the extensive body of work on continuous-stan state diffusion distillation not directly applicable. To make this distinction concrete, we first show how the following meth- ods address these challenges in discrete diffusion distillation, followed by a comparison of these representative approaches.
SDTT (Deschenaux & Gulcehre, 2025). SDTT progressively distills a multistep teacher diffusion into a fewer-step student model. Due the absence of ODE trajectory in continuous cases, it matches the teacher-student distribution via KL divergence minimization. Similar to DUO (Sahoo al., 2025), needs decrease the number of student steps for training stability gradually. Such curriculum- berriuulumbbased strategy can be inefficient and computationally costly.
DUO (Sahoo et al., 2025). Continuous-space consistency distillation (CD) requires probability flow ODE (PF-ODE) trajectories, while MDMs are inherently stochastic, making direct CD inappli- cable. DUO overcomes this by connecting Gaussian diffusion with Uniform-state Diffusion Model (USDM), enabling CD in the discrete setting. By applying arg max to the latent vectors of Gaus- sian diffusion, continuous vectors are mapped into discrete one-hot tokens. It is proven that the resulting marginal distribution exactly matches that of a USDM under suitably transformed noise schedule. Consequently, the evolution of a USDM can be described by an ODE, which makes CD feasible. However, performing CD on USDM requires multiple rounds of distillation and carefully annealed discretization schedule for training stability. Specifically, the CD procedure starts from the USDM diffusion and gradually learns to map two points, t and t At, along the same ODE tra- jectory back to the same origin. Initially, since the diffusion model cannot take large steps without losing accuracy, At must be kept small. Once the model learns to consistently align and t At, the step size can be increased, eventually reaching the largest possible step At t, i.e., the distilled model can jump large steps for few-step inference. A small step is safe but biased, while a large step is unbiased but unstable. This tradcoff is similar to the observation of CD in the continuous domain.
DIMO (Zhu et al., 2025b). DiMO distills multi-step MDMs into a one-step generator. The key idea is augment the prior distribution: instead of restricting the initial state fully-masked sequence, DiMO samples a subset of tokens from the entire vocabulary. This relaxation enriches the prior, allowing the model to leverage partial information during generation and improving both efficiency and diversity of outputs. During training, DiMO employs an on-policy distillation strat- egy. The one-step generator is supervised to match the teacher's conditional prediction distribution, not its final output distribution. Specifically, it uses the generator's 's own one-step output to cre- ate a pseudo-intermediate state, and then minimizes the divergence between the student's and the teacher's predicted token distributions conditioned on this state. To avoid mode collapse and reduce mismatch with the teacher's training distribution, DiMO introduces a token initialization strategy that mixes mask tokens with random tokens and adds Gaussian perturbations to embeddings.
While both our method and DiMO employ an auxiliary network, its function is fundamentally dif- ferent. DiMO utilizes an auxiliary model to approximate the intractable gradients of its token-level
19

---

Published as a conference paper at ICLR 2026
distribution matching objective. In contrast, draw inspiration from policy gradient (Williams, 1992: Schulman et al., 2017; 7; Fan et al., 2023) and use our auxiliary network to evaluate log-density ratio. This ratio serves reward signal, guiding the generator's updates and cireumventing the need for direct gradient approximation.
Although DiMO shows promise in text-to-image generation, it should be noted that extending it to our setting reveals a significant modality gap. DiMO is specialized for image codebook tokens char- acterized by 2D spatial dependencies and visual coherence. This differs fundamentally from pure language generation, which necessitates modeling ID sequential dependencies and strict syntactic constraints. our preliminary experiments. a direct adaptation of DiMO yielded comparable gener- ative perplexity but significantly degraded diversity (entropy drops to 4.0), which suggests training instability and potential mode collapse under naive adaptation. Consequently, transferring architec- tural choices tuned for visual latents to the text domain is non-trivial, and we consider a principled, text-specific adaptation of DiMO to be a valuable area for future work.
Our advantages over three related methods. Our work presents a straightforward and intuitive framework for distilling dL by directly matching teacher and student distributions, addressing several key limitations of recent approaches.
Unlike rajectooy-bbsee methods (Sahoo et al., 2025; Deschenaux & Gulcehre, 2025) that rely on supervised trajectory matching (forcing the student mimic the teacher's logits along rigid, prede- termined paths), DiDi-Instruct is a direct, single-stage process derived from Integral KL minimiza- tion. By reformulating distillation as policy gradient problem, DiDi-Instruct avoids the heuristic mappings and expensive multi-stage training typical of trajectory- based methods. Crucially, this ob- jective exhibits natural explore-exploit mechanism: the stochastic sampling of intermediate states and in (9) facilitates exploration of the discrete tokens, while the student's few-step backward update promotes exploitation. This natural mechanism effectively mitigates the mode-collapse is- sues inherent in trajectory wajectory-matching matching baselines.
Furthermore, by aligning the student with the teacher's entire stochastic process, our framework of- fers broad applicability to general dLLMs, in contrast methods tailored for specific architectures, e.g., DUO (Sahoo et al., 2025) relies on USDM for duality. Finally, we provide a mathematically rigorous solution to the problem of non-differentiabbiiiy in discrete spaces. This avoids the need for biased moxxy-gradient estimators, whose performance in purely textual domains remains underex- plored (Zhu al., 2025b), and instead offers principled path for distillation.
B STUDENT OBJECTIVE DERIVATION
Our goal is to train masked-diffussin student whose forward-time marginals match those of teacher across the entire time horizon. The training objective is the IKL divergence between the student and teacher forward marginals, denoted by qr and qa. In the masked setting, the forward- time corruption kernel is forward absorbing process, e.g.,
(ze x) = Cat (a.x + (1 0z m)
where m is a fixed masked tokens and S a₁ S 1. Crucially, this kernel independent of the parameter only enters through the initial student distribution pr(z, m, 1). Nevertheless, the forward marginal (z, = Exx=p.[((e) x)] still inherits v-dependence from P.. which is the key to the gradient calculation below.
We write the student and teacher forward-time marginals as
(z t) Ex-p. [Qzz x)]
qo (2,,, = Ex--po Q(z, x)).
Our goal is to derive a tractable, low-variance gradient for the IKI. objective between these two distributions. We here proceed with the detailed derivation of Theorem 3.1.
Proof of Theorem 3.1. We begin by formally defining the objective C(v) the Integral KL diver- gence. We assume the mask prior m has full support on the token simplex. Since the teacher uses the same absorbing kernel, q and qo share support (in particular, supp(qu (Zz, ()) S supp(go(z.)) ()),
20

---

Published as a conference paper at ICLR 2026
so KL (z,(z,,t) (z),))t) is well-defined and finite for all t E [0, 1]. We define the objective ((V) as the Integral KI. Divergence between the teacher and the student:
((v) DKL 90) i= / w()) (qu(ze, t) qo(zz, t)) dt
S/ w(t)Ez,nqu [lo qu (2(,,)) 1) - log df
(12)
T1
Under mild regularity conditions (bounded w(t), dominated convergence, and differentiabillty of (z,,, 1) w.r.t. V), we can interchange Vo with the expectation and the time integral (and apply Fu- bini/Tonelli theorem as needed). To evaluate the gradient of Bz,ng. llo 9 (Z,, t) log qo (ze. t)]
w.r.t. v, we use the identity Volšy~po [f Ey-ro [f V log po)) + Eyyro [Vo( Apply- ing it to T2 yields
E. Bu
[T1] = V
[log qu (2, () log qo (z,, [)]
(log (u log qu (zr, ()) V. log qu
reward
scure
+ Ezinqu
Vv (log qu t) - log qo t))
(18)
Eza (log (og, t) log (z(())))) V, log qu (zc.
(itt)
Ex -a [Mog qu (z,, () log qu ()) Ex-pa V. Po (2: m, 1)]] Exxxpu, a [log q (zz, t) log qo (z,, t)) V, Pa (z, = m, = 1)]
(13)
where step (1) applies the score- funetiin (log-derivative) identity and is justified by moving V under the expectation. Step (ii) uses V, log (o(z,, t) 0 and the fact that
Ex, Ex VV log qr (2), = Σ (z,,t) V.gu (2,,1) (2,)
Σ (zi, = V, Σ (z(,))
-1. =0
Step (iii) rewrites V, log qu (2,, 1) as Bxmp, V. Vogogpolz m, 1)]. A detailed derivation is as follows:
V. log (2,,))
(2r)(21,t)
(z(,)
V ExExp, 12(ze x]]
qu (ze,t)
(1)
Ex=p, 12 x) V. log Pr (z: = m, = 1)]
(14)
qu (2r, 1)
2 (Zt x)
Ex-pr
Vo log Pa (Zt = m,,
qu (2+,t)
(11) Exxpu
V log pa (z. m, l 1)]
where step (i) uses q (2,,t) Ex-p, [Q(z, 1x>p,1[(z,, x)] and pass V, through the expectation by linearity,
step (ii) applies (ep m, = 1) (2; = m, / = 1)V, log (o m, 1 1)
factor out log-gradient, and step (iii) follows from Bayes' rule (z,, (x 2,) ((z.|x)px (2. (2).)
We denote R (zz, log qu (zz,t) t) log qo (z,, t). Incorporating (12)-(13), we finally derive the objective:
V.L(v)= = V, //'' ((t) [log qu (z,, f) log qo (Z,, t)] dt
21

---

Published as a conference paper at ICLR 2026
! (t)) V V, [log qu (ze.t) t) - log 90 (ze, dt
Exx))xxp, 2n (log qu (ze, t) log qo (2) Vy log Pr (2. m, 1)] dr,
w())Ex-pe, a~2 [R (z..t) V. log Pv (z, m, dt,
(15)
where step (i) substitutes (13) into the time-weighter IKL integral and uses Fubini/Tonelli theorem to swap V. and So dt.
Remark B.1. In practice, the objective in (15) is approximated using a Monte Carlo estimator: The integral over time t is replaced with expectation over sampling distribution (t), resulting in single expectation over all random variables (t, x, z.). This unified expectation is then estimated by taking the sample mean over a mini-batch of size N:
w (1)) (t.)
Vol(v) N
Σ
(2),, V.P. (2+ (2==m,t=1)
(16)
This final expression is the practical Monte Carlo estimator of the full gradient. Conceptually. the gradient is single expectation over all random variables (t. x, zi), and this estimator approximates that expectation by taking the sample mean over mini-batch drawn from the respective distribu- tions.
C AUXILIARY DISCRIMINATOR FOR DENSITY RATIO ESTIMATION
Our student objective, derived in Appendix B, relies on the reward term R(Z () log (z,, t) - log qe (Z,, t), which intractable due to the unknown marginal distributions qu and qn. To overcome this, we introduce a tractable estimator for the density ratio g_(mz)) by training an auxiliary discriminator network. This approach is inspired by the principles of Generative Ad- versarial Networks (GANs) (Goodfellow et al., 2014; Wang et al., 2023) and has been successfully applied in Wang al. (2025). We hereby adapt the following lemma to MDM setting.
q2(2,
For clarity, we adopt a time-agnostic notation where the corruption level is parameterized by the masking ratio E [0, 1], which corresponds the schedule or via I a sequence with a mask ratio at time 1.
at- Let 24 VL denote
Lemma C.1 (Density Ratio Representation). models respectively, over the discrete state space V at a mask ratio t. Consider a discriminator Dx 1dh x [0, 1] - (0, 1)", which outputs probability for each position {1...., L). The discriminator Dy, is trained to minimize the objective
Let qo(zi,t) t) and qu (z,, t) be the teacher and student
1 (D(A)
I Σ Exrqul log (x t)] Ee log(1 - (z(z,,t) t))
(17)
The unique optimal discriminator Dx (Zi, t) that minimizes this objective given by:
4o (z(, 1) (zi, qo(ze,t) + qu(Z,,t)
Consequently, the density ratio can be expressed directly in terms of the optimal discriminator's 's's output:
(z,,t) Dx (Zz, qo (24.t)
1- Dx- (Zz, t)
Proof. We here provide a detailed derivation for the optimal discriminator in the context of our discrete state space. The objective J(D) is expectation over the discrete random variable 2,- We can express this expectation as summation over all possible sequences 2, E V4:
J(D)= Σ (2, t) log (z(z,, t) qo (z, t) log(1 - (x
EVL
22

---

Published as a conference paper at ICLR 2026
The loss is of terms, where each term depends only the value of (z(z,, t) for specific sequence Zg. Therefore, we can find the optimal discriminator Dx (Zz, t) by minimizing the sum- mand pointwise for each E V independently. For an arbitrary sequence z,, we find the optimal value Dx (z) 1) by taking the derivative of the summand with respect to Dx Dx 1) and setting it to
qu qu log D2(e(: q0(2), log(1 Dx(2,, t))]
0Dx t)
(18)
(z(
qo g0(2),
(2(z), t)
1- 1- Dx(z,tt)
where step (i) follows from standard differentiation of the logarithm function. We (18) to be zero and get:
(2(211)) g0(2),6)
(x 1 Dx(z,,,
(1- ())
Dx
(19)
qu (z(z),
(x-(z.))=
(2,2 ( (1) + qo (Z/ 1')
which represents an algebraic rearrangement solve for D Dx l). Since this holds for any arbitrary sequence z, we have established the general form of the optimal discriminator Dx (Zr, t).
Finally, rearrange the expression for Dx (Ze, to recover the density ratio:
x (Zz, t) (Ze, t) + go 2 t)) = (z,
Dx (Ze, 1) (z, 1) (ue, () Dx- (x,,)))))
qu (ze, D2 (2,,1)
(z),t)
Dx- (z,,
The point-wise nature of the derivation ensures its validity for discrete probability mass functions.
Remark C.2. The proof confirms that Lemma C.I is robust the specific properties of our MDM framework, including the discrete state space and the use of mask ratio t as the conditioning variable instead of continuous time t. By training an auxiliary discriminator network Dx to approximate D*, we obtain tractable, principled estimator for the density rutio, which in turn provides a computable reward signal
ilzze 1)
1 M.
Σ log m
II- (I,D)(Z).t) (2(,t)
for our student objective, where M denotes the number of masked tokens in the sequence.
D ADDITIONAL EXPERIMENTAL SETUP
D. BASELINE SETUP
To ensure a fair and reproducible comparison, we strictly follow the official implementations and experimental configurations of both SDTT and DUO. For SDTT (Deschenaux Gulcchre, 2025), we adopt the same 169M MDM teacher used in our method, pretrained for 1024 NFEs, and perform seven rounds of distillation with 10K steps each (70K steps in total) obtain an 8-step student, exactly matching the protocol in the official repository. For DUO (Sahoo et al., 2025), 169M USDM teacher (1024 NFEs) released in their codebase and conduct five rounds distillation (50K total steps). We additionally evaluate their pretrained few-step student model included in the repository.
the
In practice, observe non-negligible variation in the performance of these baselines across dif- ferent training runs. To provide the most stable and equitable comparison, we report the metrics published in their original papers, which also match the strongest results reproduced internally.
23

---

Published as a conference paper at ICLR 2026
D.2 MODEL ARCHITECTURE
Our model architectures are based on the MDLM setting (Sahoo et al.. 2024), which utilizes Diffusion Transformer (Peebles Xie, 2023) with Rotary Position Embeddings (RoPE) (Su al., 2024). We conduct experiments at two different scales. The first is 169M parameter setting, and the second is a larger 424M parameter setting. The detailed hyperparameters for each are listed in Table 3 and Table 4, respectively.
Table 3: Model Architecture Specifications (169M).
Hyperparameter
Teacher/Student
Discriminator Model
Model Type
MDLM
MDLM
Total Parameters
169M
131M
Num Layers
12
12
Hidden Size
768
768
Num Attention Heads
12
12
Positional Encoding
RoPE
RoPE
Context Length
1024
1024
Vocabulary Size
50257
50257
Classification Head
2 linear layers (SpecNorm)
Table 4: Model Architecture Specifications (424M).
Hyperparameter
Teacher/Student
Discriminator Model
Model Type
MDLM
MDLM
Total Parameters
424M
373M
Num Layers
24
24
Hidden Size
1024
1024
Num Attention Heads
16
16
Positional Encoding
RoPE
RoPE
Context Length
1024
1024
Vocabulary Size
50257
50257
Classification Head
2 linear layers (SpecNorm)
D.3 DATASET DETAILS
We use the WebTeet corpus for experiments. The raw text is tokenized using the standard GPT-2 tokenizer. Following Sahoo et al. (2025), we concatenate all documents and pack them into fixed-length sequences of 1024 tokens, adding an < enddo ftext >> token between documents. The final 100,000 documents of the corpus reserved as the validation set.
For zero-shot evaluation, we assess the model's 's generalization on the following seven diverse, out- of-domain datasets to measure its broader language understanding capabilities.
PTB:
The Penn Treebank dataset is a widely used benchmark in natural language processing, com- posed of articles from the Wall Street Journal.
Wikitext: The Wikitext- 103 dataset is large corpus of high-quality, long-form articles extracted from Wikipedia, serving standard for language modeling.
LMIB: The One Billion Word Benchmark a large-scale dataset derived from a news crawl, often used for training and evaluating large language models.
Lambada: The LAMBADA dataset is designed to evaluate a model's ability to comprehend long- range dependencies in text, requiring the prediction of the final word in passage.
AG News: The AG News dataset a collection news articles classified into four categories. For experiments, the raw text for perplexity evaluation.
24

---

Published as a conference paper at ICLR 2026
This dataset consists of abstracts from biomedical literature, representing specialized scientific domain.
Pubmed:
This dataset comprises scientific preprints from various quantitative fields available on the arXiv server, offering another domain of specialized, formal text.
Arxiv:
Table 5: Pre- Tre-Training and Distillation Hyperparameters.
Hyperparameter
Pre-training (Teacher)
Distillation (Student)
Optimizer
AdamW
AdamW
Learning Rate
1.0 x 10-4
1.0 x 10-6
LR Schedule
Cosine Decay
Linear Decay
Warm-up Steps
2,500
Decay Rate 81
0.9
0.9
Decay Rate 3g
0.95
0.999
Weight Decay
0.05
0.01
Global Batch Size
336 (42 per GPU)
8
Training Iterations
200,000
10,000
EMA Decay
0.9999
0.9999
DiDi-Instruct Specific Parameters
(I) weight schedules
We test linear schedule (()) and w(I) = aa - ar + 10-
sampling schedules
We test linear schedule Beta(1, 1, 1), Beta(2, 5), and Beta(5, 2).
D.4 TRAINING AND DISTILLATION HYPERPARAMETEN
The detailed hyperparameters teacher model pre-training and student model distillation are pro- vided in Table 5. For our 169M models, we pre-trained the teacher model for around 200,000 steps. The subsequent distillation process was completed in about one hour on a single H100 GPU. For the 424M models, the teacher was pre-trained for around 350,000 steps, while the corresponding distillation took approximately two hours on a single H100 GPU.
D.5 EVALUATION DETAILS
Generative perplexity: We use the implementation from Hugging Face's rrans formers library compute the perplexity of generated samples under frozen GPT-2 Large model. Text is processed identically to the training data.
Entropy: To assess sample diversity, we generated 40 samples for each configuration. Each sample has fixed length of 1024 tokens. We then computed the average token-level entropy across all generated samples.
Floating- point precision: All sampling and evaluation procedures were conducted using float 64 precision. This is to avoid potential artifacts and misleading scores associated with lower-precision arithmetic, a practice highlighted as important for dLLMs (Sahoo et al., 2025).
E ADDITIONAL EXPERIMENTAL RESULTS
E. DIVERSITY AND FIDELITY ANALYSIS
To provide comprehensive comparison beyond perplexity, evaluate sample diversity using MAUVE (for distribution similarity) and Self-BLEU (for diversity). All models generate 1,024 unconditional samples with identical decoding configurations (no prompt, max length 1024). Base- lines use the same sampling budgets (8, 16, 32, 64, and 128 NFEs), and all samples are processed using the same GPT-2 tokenizer.
MAUVE measures the similarity between the generated distribution Q and the reference distribution P by analyzing the trade-off between Type and Type II errors. Let Rx AP + X)Q be
25

---

Published as a conference paper at ICLR 2026
mixture distribution for À (0, 1). We compute the divergence frontier as curve (a(A), y(A)) in the 2D plane, where the coordinates are exponentially scaled Kullback-Leiller (KL) divergences:
2 <<A) exp (KI()) and y(A) exp (-c (P)))))))
(20)
where 0 is scaling factor (we 5). As À varies, these coordinates trace within the unit square [0, 1|2 The MAUVE score calculated as the area under this curve using numerical integration (trapezoidal rule) over discretized of values. Normally, texts truncated 256 tokens before embedding.
Self-BLEU evaluates diversity through n-gram overlap. For N samples {x:}
5 Σ
Self-BLEU BLEU(x;; {x))ste)
where BLEU = BP exp
Wi logPr Pn.
where Pn. the n-gram precision, w/ 0.2, and BP is the brevity penalty. We set smoothing method 1, and the tokenization of NLTK.
5, use the
Table 6: Additional generation metries: MAUVE (f, higher is better) and Self-BLEU (l, lower indicates more diversity). Comparison across varying NFEs.
Metrics
Methods
8 NFEs
NFFF
32 NFEs
64 NFEs
128 NFEs
MAUVE (t)
MDLM
4.77
6.76
6.46
5.50
5.97
(x10-3)
SDTT
5.60
5.38
5.70
5.04
5.96
DUO
5.75
5.15
5.99
5.85
6.05
DiDi-Instruct (Ours)
5.84
6.55
6.54
6.14
6,31
Self-BLEU (4) (x10-2)
MDLM
2.91
4.58
6.36
7.97
9.33
SDTT
5.02
7.98
10.09
12.49
12.92
DUO
8.26
9.28
9.62
8.84
8.94
DiDi-Instruct (Ours)
4.82
4.64
5.24
6.25
8.24
Table presents the full results. DiDi-Instruct consistently achieves the highest MAUVE scores across all NFEs, which validates that our adversarial distribution-matching objective successfully aligns the student's marginals with the teacher's. Self-BLEU values remain competitive, demon- strating that diversity is not sacrificed for fidelity. Intuitively, baselines such as SDTT and DUO tend to prioritize pointwise accuracy, leading mode-seeking behavior and worse Self-BLEU. In contrast, score decomposition forces the student to model noisy state evolution, preserving diverse modes, while adversarial training ensures marginal distribution matching for higher fidelity.
E.2 PRACTICAL LATENCY AND THROUGHPUT ANALYSIS
To complement the NFEEbased analysis in the main text, conduct practical latency evaluation.
All experiments use single H100 GPU with bf oat:1 6 precision, batch size 16, and sequence length 1024. All models share the same architecture as demonstrated Table 3. The AR baseline uses greedy decoding with standard optimizations such as K KV-aching and FlashAttention.
Table 7: Practical latency and throughput comparison one H100 GPU.
Models
NFEs
Tokens/sec
Latency per 1K tokens (sec)
AR
1024
179.042
5.719
MDLM
512
111.146
9.213
SDTT
64
806.498
1.269
DiDi-Instruct (Ours)
16
2366.604
0.432
The latency across different methods is reported in Table 7. At matched perplexity, DiDi-Instruct achieves a 13.2x reduction latency per 1k tokens relative to AR, while maintaining high sam- ple quality. This confirms that the advantages of distilled diffusion models persist under realistic
26

---

Published as a conference paper at ICLR 2026
decoding settings and not merely under NFE-based comparisons. Note that the measured latency improvement is a conservative estimate: DiDi-Instruct fully compatible with hybrid Ai-diffusion architectures (Arriola et al., 2025), and benefits from recent advances enabling KV-cache reuse and parallel diffusion decoding (Wu et al., 2026), which offer additional opportunities for further accel- eration.
E.3 RESULTS ON ZERO-SHOT PERPLEXITIES
As detailed in Table 8, DnDirtnst demonstrates superior generalization capabilities compared to the distilled baseline. achieves lower perplexity across different datasets relative DUO distilled. On benchmarks like Wikitext, it notably surpasses the autoregressive GPT-2 baseline. These re- sults confirm that DiDi-Instruct effectively preserves teacher's semantic coverage, mitigating mode collapse typically associated with few-step diffusion generation.
Table 8: Evaluation of zero-shot generalization on out-of-domain datasets. We report PPL to assess whether DiDi-Instruct preserves general language understanding after being distilled for sampling efficiency. This test is crucial to verify that the model avoids mode collapse. Lower PPL (L) indi- cates better performance. All models of comparable 170M parameter size, and perplexities for diffusion models are variational upper bounds.
Models
PTB
Wikitext
LMIB
Lambada
AG News
Pubmed
Arxiv
AR
GPT2
82.05
41.60
51.25
45.04
52.09
49.01
41.73
SEDD
100.09
40.62
68.20
50.92
62.09
44.53
MDMs
M
95.26
32.83
67.01
47.52
61.15
41.89
37.37
SEDD
105.51
49.60
82,62
65.40
82.64
$5.89
50.86
USDMs
UDLM1
112.82
39.42
77.59
53.57
80.96
50.98
44.08
DUO buse
89.35
33.57
73.86
49.78
67.81
44.48
40.39
DUO distilleds
112.27
42.53
89.45
61.17
88.70
56.17
50.27
Distilled
Instuunstruct (Ours)
107.03
35.20
80.58
53.62
78.36
47.56
45.35
E.4 DOWNSTREAM TASK EVALUATION
We conduct three downstream evaluations used verify whether DiDi-Instruct preserves the task capability of the teacher's 's semantic representations while enabling few-step generation. All experi- ments use the same 169M architecture unless otherwise specified.
Domain adaptation on MMLU. We fine-tune MDLM, SDTT, and DiDi-Instruct on the MMLU benchmark for 5,000 supervised fine-tuning (SFT) steps. Training uses the AdamW optimizer with base learning rate of 10-5, which is linearly warmed up for the first 2,500 steps and kept constant thereafter. We employ batch size of and an evaluation batch size of DiDiutnstruct not only matches the teacher's downstream accuracy but also achieves the most sig- nificant improvement in Negative Log-Likelihood (ANLL). This indicates robust adaptability to general knowledge domains.
shown Table 9,
Table 9: Domain adaptation on MMLU dataset.
Models
NLL (pre)
NLL (post)
ANLL
Ace (pre)
Ace (post) 1 AAce
MDLM
9,879 9,773
2.814
7.065
20.6%
25.3%
4.7%
SDTT
2.844
6,929
21.2%
24.3%
3.1%
DiDi-Instrucl (Ours)
9,960
2.815
7.145
21.6%
25.1%
3.5%
*Results are compiled from prior work a comprehensive comparison.
Values from Lou et al. (2024).
Values from the evaluation suite in Sahoo et al. (2024).
Values reported in Sahoo et al. (2025).
Result obtained by evaluating the DUO distilled model from Sahoo al. (2025).
27

---

Published as a conference paper at ICLR 2026
Scientific domain adaptation. We further evaluate domain transfer to scientific text by conducting 5,000 SFT steps on the PubMed corpus. All models are fine-tuned using Adam with a learning rate of 5 10-6 and constant schedule with 1,000 warmup steps. Training of 36 with gradient accumulation (effective batch size 72). As reported in Table 10, DiDiiInstruct significantly outperforms the SDTT and DUO baselines. Notably, it achieves final perplexity of 24.894, which is practically indistigguishable from the teacher's perplexity, which validates its effectiveness in specialized domains.
per-GPU batch size
Table 10:
Domain adaptation on PubMed dataset.
Models
NLL (pre)
PPL (pre)
NLL (post)
PPL (post)
MDLM
3.761
42.983
3.217
24.944
SDTT
3,938
51.336
3.243
25.602
DUO
4.028
56.174
3.338
27.734
DiDi-Instruct (Ours)
3.879
47.563
3.215
24.894
Frozen feature extractor evaluation. To assess the quality of the learned latent representations, we evaluate the models on the GLUE MRPC task (Dolan & Brockett, 2005). The pretrained backbones (MDLM, SDTT, DiDii-nstruct) are kept fixed, and a lightweight classification head is added on top of the encoder's pooled hidden representation. The head consists of a single linear layer mapping the pooled vector to two logits, and we additionally unfreeze the final encoder block to allow light adaptation. Training uses AdamW with learning rate of x 10-5, weight decay 0.01, a batch size of 8, and 10,000 training steps. We report Accuracy and FI on the validation split.
Results in Table 11 show that DiDi-Instruct attains the highest Accuracy and F1 scores. This con- firms that Instrutt preserves and potentially improves semantic discriminability.
Table 11: Frozen feature evaluation on GLUE MRPC dataset.
Models
Accuracy t
F1 Score f
MDLM
59.6%
72.2%
SDTT
59.1%
72.0%
DiDi-Instruct (Ours)
62.3%
73.3%
Across all downstream tasks, DiDi-Instruct matches or exceeds the teacher's performance, estab- lishing that few-step distillation maintains strong semantic representation quality while delivering significant inference speedups.
E.5 IMPLEMENTATION DETAILS OF THE ABLATION STUDIES
To provide a clear basis for interpreting our experimental results, we now elaborate on the precise implementation of each component tested in our ablation studies. This includes the mechanisms and key hyperparameters that were modified between the baseline and full models. Below, we detail the function of each component in our ablation studies.
The baseline model a one-step generator, mapping a fully masked input z, 1) directly the final output x. With this technique, we decompose the generation into a two- step process z, - Z, + x where & = 1 and 7 E (0, 1). The model first generates an intermediate state z, randomly sampled time following 7 (t) and then generates the final output x from z,. The ablation ('w/o Score Decompose') reverts to the direct, single-step generation.
Score decompose.
Coupled time This component synchronizes the time steps used for student generation and discriminator evaluation. In the standard setup, the intermediate denoising time for the student's two-step generation (z, - z,) and the corrppiion time 7' for preparing the discriminator's 's input (x Z,r for corrupted student generations, and x' - z!., for corrupted teacher generations) coupled, i.., - r'. The ablation ('w/o Coupled Time t') decouples them by sampling 7 and 7 independently.
28

---

Published as a conference paper at ICLR 2026
Weight function w(t) correction. The gradient of the objective (5) is weighted by time- dependent factor w(t). The ablation ('woo w(t) Correction") uses a constant weight, i.e, w() = 1. Our full model applies theoretically-motiviin ot, setting w(t) x - -a//(1 0t), which re-weights the objective based on the difficulty of the denoising step at time f.
correction based on the noise schedule's 's signal rate
Time scheduler t()) weighting. This controls how T is sampled and (optionally) whether we apply importance weighting, which also refers to the sampling distribution ((7) for the intermediate time step т. The ablation setting ('woo x(T) Weighting') samples from uniform distribution. Our method employs a non-uniform Beta distribution as x()) to focus the student's training on specific, more informative intervals of the denoising trajectory. Specifically, we consider heavy- early (Beta(2, 5)), and heavy-late (Beta(5,2)) schedules to improve sample quality, Practically, for smaller NFEs (e.g.. 8 and 16 NFEs) we bias toward earlier times (e.g., 52); for larger NFEs (e.g.. 32, 64, and 128 NFEs) we bias toward later times (e.g., (,
Regularization. To stabilize training and maintain generalization, we introduce two regularization terms to the student's 's loss. The first is KL divergence loss, which aligns the student's 's output logits with those of the frozen teacher both steps of the generation process (z, + Z, and z, + x, where implementations, we found forward KL
t == 1 and 7 E (0, 1)). Among consistently yielded the most stable optimization and best validation in our cases, so we adopt for all ablations and apply it with a weight of 0.05 in the student objective. The second term is an entropy bonus on the student's 's output distribution to encourage exploration, which weighted by 0.0005. The ablation ('w/o Regularization') removes both the KL--divergecce and entropy terms from the student's objective.
hrrward/Backward/elliiin
Guided inference. As elaborated in Section 3.4, this technique is applied only during sampling to leverage the trained discriminator and improve sample quality. We employ a hybrid strategy that divides the denoising process into two phases. For the first 50% of NFEs, we use gradient filting (M = 1). where the guidance scale h is linearly increased from 30.0 to 40.0 to steer the generation. For the remaining 50% of NFEs, we switch multi-candidate re-ranking = 0), sampling . candidates at each step and selecting the one with the highest reward. The ablation ('w/o Guided Inference') ) disables this process (h 0, M = 1), reverting to standard unguided ancestral sampling-
F.6 ABLATION STUDY RESULT ANALYSIS.
We now interpret the ablation results, breaking down our analysis by the function of each compo- nent group. We will discuss how the model is stabilized, how performance is driven by loss and time considerations, the conditional role of regularization, and finally, how inference-time guidance improves the final output.
Baseline and score decomposition. The baseline model (without tricks) performs poorly, with Gen PPL of 803.9 at 8 NFEs. As shown in Table 1, incorporating only the Score Decompose pro- vides moderate initial improvement. However, its indispensable role is starkly revealed in our leave-one-out study (Table 2). Removing this component from the full model results in a catas- trophic performance degradation (PPL> 30, 000). This indicates a critical interplay: while a single- step gradient estimation is unstable within the complex training landscape created by our other optimizations, the two-step decomposition is essential for stabilizing the entire framework.
Time coupling and loss weighting, The most significant performance gain is achieved by intro- ducing Coupled Time which dramatically reduces the 8-step PPL from 667.8 to 101.0. This highlights the importance of aligning the temporal context between the reward signal and the score function. Loss reweighting then refines convergence: the w()) Correction improves mid/late-step budgets (c.g., 32 NFEs: from 48.4 to 31.7; 64 NFEs: from 35.8 to 25.3), while T()) Weighting is especially effective at 16 NFEs (75.6-44)) +44.0), with neutral to slight trade-offs elsewhere. The leave- one-out analysis confirms that removing any of these components leads to a severe degradation in performance.
The dual role of regularization. Our study reveals that KL/entropy regularization plays a dual role depending on the sampling budget: it is helpful at very small budgets, where discretization error might be severe. For very small NFEs (e.g., 8 NFFs), where large discretization errors can
29

---

Published as a conference paper at ICLR 2026
destabilize training, regularization acts crucial stabilizer. The full model with regularization outperforms the version without (PPL of 62.2 vs. 88.3). However, for more NFEs (2 16), Table shows that removing regularization yields superior results, achieving the best PPLs in these set- tings (e.g., 30.99 vs. 38.19 at 16 NFEs). This suggests that for finer sampling schedules, the implicit stability of the chain is sufficient, and strong explicit regularization can over-constrain the model, leading to an accumulation of bias that harms the final generation quality.
Guided inference. Guidance is most effective at small NFES, where it markedly lowers PPL. Specif- ically, at 8 NFEs, guidance reduces PPL from 88.3 to 62.2, a relative improvement of 29.6%. This trend continues at 16 NFEs, where PPL drops by 13.2% (from 44.0 to 38.2), and at 32 NFEs. where it decreases by 12.0% (from 28, 4 to 25.0). Conversely, at high NFEs, the effect on PPL is negligible (e.g., from 21.95 to 21.91 at NFEs). In this regime, however, guidance substantially improves sample diversity. We observe that entropy increases from 5.06 to 5.15 at 64 NFEs and from 5.00 to 5.15 at 128 NFEs, indicating that guidance can enhance variety without sacrificing quality. This pat- tern matches our hybrid schedule: early gradient tilting improves accuracy small budgets, while late multi-candidate re-ranking expands support and boosts diversity at larger budgets.
E.7 SENSITIVITY ANALYSIS OF RGAS
32
32
Candidate count M
5.4
50.0
5.3
16
16
PPL
5,2
5.1
40.0
8
8
5.0
3
4.9
20 40
60
80
100
0
20
60 80 100
30.0
Gradient scale h
Gradient scale h
Figure 3: Sensitivity analysis of RGAS hyperparameters at 16 NFEs. We visualize the landscape of PPL (left) and Sequence Entropy (right) across varying gradient scales h. and candidate counts M. The heatmap indicates that increasing both h and M consistently reduces PPI., improving generation fidelity. However, larger gradient scales tend to lower sequence entropy, indicating trade-off between sample quality and diversity.
To understand the interaction between the tilting scale h and candidate number M in RGAS, conduct a comprehensive grid search to visualize the performance landscape, and a global sensitiv- ity analysis using Functional ANOVA (FANOVA) (Hutter al., 2014) quantify hyperparameter importance.
Performance landscape analysis. We first study the joint effect of the RGAS hyperparameters h and M on generation quality. At fixed budget of 16 NFEs and batch size 16, we perform dense grid search, sweeping h E [O, 100) and M E {1, 2, 4, 8, 16, 32). Figure 3 presents the resulting PPL and Entropy. The resulting landscapes show a broad valley: increasing f from 0 to 100 consistently reduces PPL. while entropy only gradually decreases and remains close to the teacher's entropy. This indicates that RGAS is relatively robust across wide range of (h, M) values and that there is no narrow "knife-edge" region required for good performance.
Global sensitivity with FANOVA. To formally quantify the relative importance of h and M, we conduct a global sensitivity analysis using Functional ANOVA. We adopt a Latin Hypercube Sam- pling strategy and generate 1,000 hyperparameter pairs, with h sampled continuously from [20, 60] (a range that balances PPL and entropy) and M sampled as an integer from [1, 32]. For each config- uration, run RGAS with 16 generated sequences and record the resulting PPL. This experiment is repeated separately for NFEs = 8, 16, 32, and the resulting performance is decomposed into vari- ance contributions from h and M.
Table 12 summarizes the FANOVA results. At and 16 NFEs, explains around 90% of the variance in PPI., while M accounts for less than 10%. At 32 NFEs, the importance of M increases, but
30

---

Published as a conference paper at ICLR 2026
Table 12: Variance decomposition PPL via IANOVA.
Parameter
NFEs
16 NFES
32 NFES
Gradient Scale (h)
92.7%
90.0%
63.7%
Candidate Count (M)
7.3%
6.0%
36.3%
still contributes the majority of the variance. These findings suggest that: (i) in the low-NFE regime, is the dominant hyperparameter and should be prioritized during tuning; and (ii) the sampling budget increases, M becomes more relevant knob for refining performance.
E.8 MODEL SCALING UP
MDLM
5.7
DiDi-Instruct (169M)
5.6
Perplexity
10a
DiDi-Instruct (424M)
Entropy
5.5
5.4
5.3
5.2
16
32
64
128
256
512
1024
32
128
256
512
1024
NFEs
NFEs
Figure 4: Scaling results for the 424M models. DiDi-Instruct significantly lowers PPL compared to the MDLM baseline across all NFEs.
To validate the effectiveness and scalability of DiDi-Instruct we extended our experiments to 424M parameter setting. In this evaluation, pre-trained 424M MDLM served as the baseline model, where the teacher uses DiT+RoPE (more details can be found in Table 4) and is pre-trained for around 350,000 steps on 8xH100; the 424M student then distilled on single H100 in about 2 hours. We keep data, masking, optimizer, and schedule aligned with the 169M setting, and scale the discriminator to 373M with deeper, then applied DiDi-Instruct to produce the distilled student model. We assessed performance using two key metrics: PPL for predictive accuracy and token-level entropy for model confidence.
head for per-token logits. We
The results demonstrate substantial improvement in language generation capabilities. As illus- trated in Figure 4, our model achieves a significantly lower perplexity than the pre-trained baseline across all evaluated sequence lengths. The 424M distilled student model demonstrates remarkable efficiency, significantly outperforming the pre-trained base model's best-case performance (at 1024 NFEs). With only 16 NFEs, the student model already achieves a PPL that is 11.4% lower than the base model at 1024 NFEs. This performance gap widens substantially with more inference steps: at 64 NFEs, the student's PPL 56.8% lower, and 128 NFEs, it is 61 .1% lower. At the maximum of 1024 NFEs, the distilled model's PPL is 64.2% lower than the base model's 1024-step result, showcasing a dramatic improvement both generative quality and efficiency.
Entropy remains comparable while quality improves. The distilled 424M student tracks the base model closely across NFEs: the absolute gap is modest at low NFEs (e.g., 5.52 vs. 5.75 16 NFEs; A -0.22) and narrows NFEs increase (5.43 5.43 at 512 NFEs; A 0), eventually turning slightly higher at 1024 NFEs (5.40 vs. 5.32; A Rs +0.08). Overall, diversity preserved while achieving markedly better PPL.
This incremental scaling experiment indicates that IDDi-Instruct quality-efficieny advantages per- sist at 424M with minimal procedural changes: large PPL reductions matched NFEs and near- constant entropy. summary, the 424M seale experiment confirms the robustness and scalability of DiDiiInstruct. Its ability to substantially improve upon already capable, larger-scale base
31

---

Published as a conference paper at ICLR 2026
model underscores its potential an effective technique for distilling powerful and efficient student models.
E.9 PROTEIN SEQUENCE GENERATION
L=100
L=200
L=300
L=400
L=500
PLDDT
conderce
OLDOT
HODI
32 26
14
126
230
3:2
26
32
14
120
250
532
30
52
120
250
532
36
120
258
5:2
29
1.00
299
512
DPLM-150M
Ours
Figure 5: pLDDT comparison between DiDi-Instruct and the DPLM-150M across different se- quence lengths (L 100, 200, 400, 500) and numbers of function evaluations (NFEs). Our method consistently outperforms the teacher, achieving up to +10 pLDDT gains at shorter sequence lengths (e.g., I 100) and maintaining superior structural confidence across all lengths, even with substantially fewer sampling steps. Moreover, the distilled student exhibits more stable performance across NFEs, whereas the teacher shows larger variability as the number of steps increases.
Dataset and evaluation metrie. The UniRef50 dataset (Suzek et al., 2014) contains approximately 45 million protein sequences, comprising roughly 14 billion amino acid tokens. Following previous work (Wang et al., 2024), we adopt a vocabulary of 33 tokens and a maximum sequence length of 1024, with longer proteins chunked into shorter chunks. We compute the predicted local distance difference test (pLDDT) score using the ESMFold model (Lin et al., 2023) to assess the foldability of the generated protein sequences, with values above 70 indicating high structural confidence. We report the pLDDT score across different sequence lengths and varying NFEs.
Experiment details. Our model architecture follows the DPLM- 150M setting (Wang et al., 2024), with the student model and discriminator matched in size, while the discriminator incorporates modified prediction head that outputs a scalar value. We set the learning rate of the student model to 1 x 10-3 and that of the discriminator to 5 x 10-6, with 1000 warm-up steps for each. All other training hyperparameters and sampling strategies follow those used in the text modeling task. The distillation of DPLM was completed in approximately two hours on single H100 GPU, using batches of up to 4096 tokens.
Protein sequence examples. We provide visualizations of generated protein sequences to compare structural plausibility between the teacher and our distilled model. Figure 6 shows low-confidence outputs from the teacher under limited NFEs, while Figure 7 presents high-confidence examples produced by DiDi-Instruct in Figure
Sequence diversity analysis. High predicted structural confidence (pLDDT) can sometimes indi- cate a risk of mode collapse, where the model generates repetitive samples. Therefore, we evaluate sequence-level diversity using MMseqs2 clustering (Steinegger & Söding, 2017) with a 30% se- quence identity and 80% coverage threshold-standar ingful variation. Across low-NFE settings (32-64), DiDi-Instruct exhibits diversity comparable to DPLM, which reflects the stochastic nature of early denoising. As the number of sampling steps in- creases (128-51) 212 NFEs), DiDi- yields slightly lower entropy and moderately larger cluster sizes, indicating a more concentrated output distribution. Importantly, diversity levels remain com- petitive with DPLM across all settings, and no regime shows abnormal cluster growth or collapse of entropy. Noted that genuine mode collapse in this setting would correspond to near-zero cluster entropy together with an average cluster size approaching the full sample count, i.c., all sequences falling into single MMseqs2 cluster. This collapse behavior is not observed in any configuration.
criteria for determining biologically mean-
This behavior is consistent with our text-generation ablations, where diversity naturally decreases as NFEs grow and the denoising trajectory becomes more deterministic. Moreover, DiDi-Instruct is explicitly designed for few-step generation; mild diversity reduction at very large NFEs does not contradict our intended operating regime. Overall, these results confirm that DiDii-nstruct maintains
32

---

Published as a conference paper at ICLR 2026
8-steps
16-steps
32-steps
Length
Low contidence!
Low confidence!
Low confidencel
100
Kazes
200
300
400
500
Figure 6: Visualization examples of low-confidenee protein sequences (pL.DTT 70) generated by the teacher model with lengths ranging from 50 to 500, under 8, 16, and 32 NFEs
8-steps
16-steps
32-steps
64-steps
128-steps
256-steps
512-steps
Length
100
200
300
400
500
Figure 7: Visualization examples of high-confidence protein sequences (pLDDT > 70) generated by the DiDi-Instruct model, with lengths ranging from 50 to 500 across different numbers of function evaluations (NFEs).
33

---

Published as a conference paper at ICLR 2026
biologically meaningful variability across the generation budget while avoiding collapse to small subset of high-pLDDTT sequences.
Table 13: Protein sequence diversity evaluation using MMseqs2 clustering. We compare the Average
Cluster Size and Cluster Entropy across different NFEs.
Cluster Entropy ?
NFEs
Methods
Average Cluster Size +
100
200
300
400
500
DPLM
3.68
3.68
3.55
3.65
3.68
1.02
32
DiDi-Instruct (Ours)
3.68
3.52
3.15
3.58
3.55
1.11
DPLM
3.68
3.51
3,45
3.68
3,53
1.07
64
DiDi-Instruct (Ours)
3.68
3.65
3.45
3,45
3.54
1.09
DPLM
3,68
3.65
3,50
3,52
3,61
1.05
128
DiDi-Instruet (Ours)
3.62
3,20
3.42
3,45
3.38
1.21
DPLM
3.61
3,42
3.41
3.19
3.44
1.19
256
DiDi-Instruct (Ours)
3.61
3.11
3,41
2.87
3.07
1.40
DPLM
3.58
3.23
3.02
3.11
2.97
1.41
512
DiDi-Instruct (Ours)
3.54
3.11
2.95
2.77
2.79
1.53
E.10 TEXT EXAMPLES
To provide more intuitive understanding of the performance of our distillation framework, this section contains qualitative examples of generated text. We present side-by-side comparison of samples generated by the original 1024-step pretrained teacher model and our distilled DiDii student. For the student, we showcase generations from range of few-step sampling configurations: 8, 11,632, 32, 64, and 128 NFEs. This allows for a direct visual inspection of the trade-off's between computational cost (i.e., NFEs) and sample quality.
We here provide qualitative analysis of generated texts to evaluate our models across several key linguistic dimensions. The results indicate that the number of function evaluations (NFEs) in- creases, DiDi-Instruct students show marked improvements in coherence and narrative consistency, eventually outperforming the teacher. Specifically, evaluate the generated texts with respect to the following three criteria:
Fluency and repetition. All models (including the teacher and students) generally produce fluent and grammatically correct sentences. The most significant exception is the 8-step student, which suffers from severe phrasal repetition (e.g., repeating "affirmative action program"), common tifact of extreme model compression. This issue is effectively eliminated in student models with 16 NFEs or more, which exhibit fluency comparable to the teacher.
Coherence and topic adherence. This is where the most critical differences emerge. The 1024- step teacher struggles with global coherence, frequently shifting between unrelated topies within single output. In contrast, while the 8-step student is incoherent due to repetition, the 16-step student already demonstrates stronger paragraph-level topic adherence than the teacher. This ability to maintain a consistent narrative thread strengthens progressively. The 128-step student excels at this, developing a central theme with supporting details over longer text, showcasing high global coherence.
Informativeness and specificity. The informativeness the generated text correlates strongly with coherence. The teacher's outputs, despite containing specific details, are uninformative as whole due to topic shifts. The 8-step student's 's text has very low information content due to repetition. As NFEs increase from 16 to 128, the students' generations become increasingly specific and informa- tive. For example, the 64-step model constructs detailed news-style report, and the 128-step model builds multi-faceta story, both rich in contextual details.
34

---

Published as a conference paper at ICLR 2026
Overall, the generated text samples show that DiDi-Instruct successfully distills the teacher's lin- guistic capabilities while simultaneously improving its ability to construct coherent and focused narratives.
Text obtained with 1024 NFEs teacher model. Perplexity= 10.44, Entropy=5.222
He mic-morning early, and started forming motorcade for him. He stopped evezy NYPD car, and I radioed back to every bicycle officer that was in there, to protect him. I did not hear you said + to be involved in it.
That you were involved with that campaign. never asked anybody to + test it. You did need help you leaked it, not made it for yourself.
4 He parked and it was then that I saw that president and ended up + there, speaking with a journalist.
The Prince was granted anonymity and noted that the FEI was aware the fami Samilian liriity with Russia and was conducting the investigation as 4 appropriate. In some accounts, Casey wilson, senior counsel for the Honden Bisson, told The Post that the President is aware of the inquiry.
But Mr. Baker's testimony is the first time someone has long taken
+ something of concern into the hands of their officials probing Trump's s personal rolationships with foreign officials. Former FBI
regional director James Kogan, former officer who met with President Trump, was once briefed about the inquiry from the
Russian District Attorney General.
Victor Atkinson, FBI chief-officer who is retired, hacked the personal PINs and cell phone lines of the two Trump officials to conceal the information, as shown inside the country's electronic networks. Anonymous messages can appear in our apps like Twitter
and Facebook and if one's political actions resists they become blocked.
Text obtained with 8 NFEs distilled by DiDi-Instruct. Perplexity=62.24 Entropy=5.17.
An engineer's a run up against system of dynamic memory-prodee ppoourigo + technology. examiner's mechaniam of memory has to reduce property crimes against the technology.
India's memory system has run into a deficit of memory and that
C economic damage in India has run up against a system of Now and economically secure and
Then. The system will be designed efficient system in deternining how residents receive support.
A loss of support from memory management systems won't t too far from + promise. With deficit of million for residents, the Navajo Nation testing new system that won't prevent you from receiving your financial stream.
The Navajo Nation faces deficit of 1.88 million in systers that are designed to enhance what residenta recsive. It's potential loss support from system that has been receiving both positive and negative feedback from the community.
A presidential candidate who once served during four tours in Iraq and pop
Afghanistan refers his record figurative culture. got disappointed order from state Department, the new record is to go against the American health care system.
35

---

Published as a conference paper at ICLR 2026
Text obtained with 16 NFEs distilled by InDi-lnstruct. Perplexity=30.99. Entropy=5.22.
by a bi omedical engineer and Justin ketter, researcher at Case
+ Western University who deve_oped a non-profit. Zetter says that the + sperm the new 1ab produces is quickly going to a place better stores and than human sperm.
A year and a half years ago Zetter took out of this lab and coated it
+ with a chicken, and he is optimistic it will come in next year. "It
+ 's very exciting to see aninal entrepreneurship in the future, he says.
Foreign Minister Chid wants to use a Frieses system for links to Saudi
+ Arabia. The Saudi interior minister has announced plan to allow imports to Frieses system system that links the Gul nation's ferries.
while the government had initially rejected the system, the Saudi + Statesport Service post has announced that it is considering amalganating two ferries including EiKhasi Oera Tries, which run-owned by Sad Cera Qatar, to avoid links to Riyadh.
Mohsen Chid, spokesman for Saudi Arabia's Public Security Ministry, said Monday that the ministry is considering plan to use a Frieses and trip management system known ElKhasi Oera Tries, which was founded 1927.
Text obtained with 32 NFEs distilled by DiDi-Instruct. Perplexity=23.60 Entropy=5.20.
car crash last month + initially thought a first-year student in therapy during a blocked
Virginia man accused of being involved
+ light video, according to a North Carolina grand jury. Khalid
< Anthony Matteo told investigators he was spending night in therapy.
Mattco was arrested after being preseribed a Virginia painkiller for + blocking someone's s vehicle during a display of headlights. Federal
+ prosecutors have recommended nearly 30 months of outpatient
treatment for Matteo, who in the first year of an outpatient treatment program.
Prosecutors initially said a woman had spent the night in therapy to + avoid the video, but Matteo was given month of outpatient
treatment and is up to nearly 30 years of probation. Democratic nominee for the presidential nomination sought the support of state lawmakers.
The nominee won't block any state cuts that they promise otherwise. Del Tom Perez will give testinony before the bench of the District Court, after a group of Virginia lawmakers would cut short the state's responsibility for the federal health care plan.
The plan designed to
that seniors receive access to health care. Virginia lawmakers say the federal plan would allow all
seniors to have access to health care, but 1awmakers have
4 threatened to block its imp' empntatin over claims that it would reduce the deficit.
36

---

Published as a conference paper at ICLR 2026
Text obtained with 64 NFEs distilled by DiDi-Instruc.. Perplexity=19.61, Entropy=5.18.
People gathered at the scene of an autopsy of a dead Rgyptian army
+ soldier near the northern Sinai town of Khaled el-Zawiya, east of
+ the Egyptian capitai, Cairo, November 9, 2017. REUTERS/String not immediately released.
Sundays
A police official in Cairo, Mchamed Hussein, said that the soldier was + involved in confrontation with a group of protesters acting + counter-daty and fired projectiles at him, Reuters said. The U.N. investigators said that Egyptian forces fired tear gas.
When hundreds of protesters gathered in the desert near Sheikh el-
+ Zawiya, near the hometown of the late Egyptian Abdel Fattah el-Sisi The UN initially accused the Egyptian forces of using excessive force during the confrontation. on, but later apologized.
kharyl Hershehey claims he was driving night-in his Mercedes-Renz Audi
+ over night to avoid trip. An Ohio court-appointer attorney is Virginia accused for drinking someone while he tried drive his green car over parked near stop.
The trip was leading to a risk of head injury. Kharyl Joshua Hershhhey is accused of drinking someone while driving his car in real green engine avoid trip, say court documents. The drunken driver claims a wonan was killed.
Text obtained with 128 NFEs distilled by Inst-Instruc.. Perplexity=1.55) Entropy=5.17.
officials at the U.S. District of Virginia claim "fatality epidemic" 4 is causing deaths in school classrooms around the U.S. The Virginia school district is investigating the death of student Tamey Nelson + after she wielded a knife to kill a classmate in class.
A school in Virginia is investigating after state claims that fatality epidemic death" is causing deaths among students classrooms around the world. Officials at the U.S. district disputed the school's clains of an outbreak.
Two students
killed and three others a news report obtained by NBC News. "I am concerned about the + safety of children and students in classrooms in the United States
hospitalized, according
+ the District of Virginia District Attorney James Trotsma told
+ NBC News.
Parents of a Virginia school claim that "child's life of death" was + caused by an epidemic of abuse in classrooms around the U.S.
+ district en Sunday, according to a news report. & video surfaced of 12-year-old student being struck and killed.
The student was killed by a knife wielded by others they identified as + "revered students," similar incident described by a teacher that was caused by the same incident in class at the school's Iron Arlington Branch Learning Center.
37


Reviews:
[{"review_id": "5yQPbw7jtH", "reviewer": "Reviewer_Qgt2", "summary": "This paper introduces Discrete Diffusion Divergence Instruct (DiDi-Instruct), a training-based distillation method that initializes from a pre-trained dLLM to create a fast, few-step student generator. The framework is founded on integral KL-divergence minimization, which is reformulated from a policy gradient perspective to derive a tractable update rule for the discrete state space of language.", "questions": "- Could you clarify the experimental setup for the SDTT and DUO baselines?   \n- Could you provide a sensitivity analysis for the RGAS hyperparameters ($h$, $M$) and elaborate on the stability of the adversarial training?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The paper proposes a principled training method by reformulating the IKL distillation objective from a policy gradient perspective, which provides a tractable update rule for discrete spaces.   \n- DiDi-Instruct achieves new state-of-the-art results on the OpenWebText benchmark, consistently delivering lower PPL across 8-128 NFEs with over 20x faster distillation time than competing methods.", "weaknesses": "- The proposed Reward-Guided Ancestral Sampler (RGAS) introduces new hyperparameters ($h$, $M$), but there is no sensitivity analysis.\n- It is unclear how the baseline results like DUO or SDTT are obtained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cKatyPVvd4", "reviewer": "Reviewer_HAtL", "summary": "This paper introduces a training-based distillation framework DiDi-Instruct to accelerate generation for discrete diffusion language models (dLLMs). The core idea of DiDi-Instruct is to distill a pre-trained teacher dLLM into a student model that can generate high-quality text in very few steps. The method is centered on minimizing the Integral KL (IKL) divergence between the teacher and student models, which forces the student to match the teacher's data distribution. The author proposes comprehensive solutions to make the training on discrete data tractable, concluding: 1) inspired by policy gradient, decomposing the gradient of the objective as a score function weighted by a log-density ratio to avoid non-differentiable issue. 2) using an auxiliary discriminator to approximate the log-density ratio because it is intractable to compute directly. 3) leveraging grouped reward normalization to stabilize reward. The training is end-to-end and jointly for discriminator and student model. During inference, the author uses the trained discriminator to guide the sampling process and improve output quality. Experiments show the effectiveness of the proposed method that DiDi-Instruct significantly outperforms existing accelerated models and the GPT-2 baseline in perplexity, and the distilled model surpasses the 1024-step teacher's quality with only 16 steps.", "questions": "See above weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents a mathematically rigorous solution to the difficult problem of distillation in a discrete state space. Instead of relying on heuristics, it adapts the IKL divergence objective from continuous models and  reformulates it as a tractable policy gradient problem.\n2. The method consistently outperforms other distillation methods and the GPT-2 baseline in perplexity across all sampling budgets.\n3. The proposed method demonstrates high efficiency. The training only needs one GPU, compared with 8 GPU for training teacher model. For generation, it is able to surpass the text quality of its 1024-step teacher model with only 16 inference steps, representing a significant acceleration.\n4. The proposed method avoids the mode collapse problem, which is a common problem in distillation. Experimental results on text generation demonstrate negligible entropy loss.", "weaknesses": "1. The author performs distillation on a self-trained teacher model. This raises questions about the method's transferability and effectiveness when applied to other pre-trained, open-source dLLMs, such as LLaDA [1] or Dream [2].\n2. In the text generation experiments, the evaluation is limited to perplexity and entropy. These metrics alone cannot accurately reflect the true text generation capabilities. The authors should conduct a more comprehensive evaluation on standard text generation benchmarks (e.g., MMLU, GSM8K, Humaneval) to demonstrate that the student model can indeed achieve performance comparable to the teacher model while using fewer steps.\n3. In the protein sequence experiments, the pLDDT score increases significantly after distillation. However, excessively high pLDDT scores might indicate a potential risk of mode collapse (e.g., collapsing to a few sequences with the highest pLDDT, thereby failing to sample other valid sequences), which will lead to a reduction in generation diversity. The authors need to clarify the generation results from the perspective of protein sequence diversity (e.g., by using MMseqs clustering) to address this concern.\n\n[1] Dream 7B: Diffusion Large Language Models\n\n[2] Large Language Diffusion Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hdhFk53Q81", "reviewer": "Reviewer_n4tg", "summary": "The paper proposes Discrete Diffusion Divergence Instruct (DiDi-Instruct), a distillation framework that trains a few-step discrete diffusion language model (student) to match a pre-trained masked diffusion language model (teacher) by minimizing an integral KL divergence (IKL) across noise levels. The core gradient is rewritten as a score-function estimator weighted by a log density-ratio reward between student and teacher; the reward is estimated via an auxiliary discriminator. Training is stabilized by grouped reward normalization and a score decomposition that exposes the student to intermediate corruption states. At inference, a Reward-Guided Ancestral Sampler (RGAS) uses discriminator signals for gradient tilting in early steps and re-ranking in later steps. On OpenWebText, the model achieves perplexity 62.2- 18.4 at 8 - 128 NFEs, with small entropy loss and lower distillation cost, and it scales to larger models and a protein-sequence task.", "questions": "1. Can you report tokens/sec (or sequences/sec) and latency per 1k tokens at matched perplexity versus a similarly sized AR model with and without speculative decoding, on the same hardware and kv-cache settings? This would clarify the practical speed–quality trade-off beyond NFEs.\n2. Can you include downstream tasks? A single perplexity metric is not enough to establish practical advantage of the new method.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper adapts IKL-based distribution matching to masked discrete diffusion and derives a policy-gradient form that bypasses non-differentiable paths, with a discriminator estimating the log density ratio used as reward. This method is different from time-matching distillations and provides a route to few-step discrete generation.\n2. The derivation gives the IKL objective and yields a score-function identity in which the reward is the marginal log-density ratio between student and teacher; the proof sketch and assumptions are stated, and the link to a tractable discriminator estimator is explicit.\n3. On OpenWebText the method reports consistently lower PPL across 8–128 steps, with faster distillation and small entropy loss. The paper includes cumulative and leave-one-out ablations indicating gains from the proposed components, and shows transfer to protein sequence generation.\n4. The paper is clearly written and well structured.", "weaknesses": "1. The comparison to the baselines are is too abbreviated with only one figure of perplexity. It would be better to have a table presenting more metrics besides the ppl.\n2. DiMO is discussed in the related work in appendix but is not include as an experiment baseline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)