Learnable Sampler Distillation for Discrete Diffusion Models
Authors: Feiyang Fu
Tongxian Guo
Zhaoqiang Liu
Venue: NeurIPS 2025 spotlight ()
Keywords: Discrete diffusion models
Efficient sampling
Sampler distillation
Primary area: deep_learning

Abstract: Discrete diffusion models (DDMs) have shown powerful generation ability for discrete data modalities like text and molecules. However, their practical application is hindered by inefficient sampling, requiring a large number of sampling steps. Accelerating DDMs by using larger step sizes typically introduces significant problems in generation quality, as it amplifies the impact of both the compounding decoding error due to factorized predictions and discretization error from numerical approximations, leading to a significant decrease in sampling quality. To address these challenges, we propose learnable sampler distillation (LSD), a novel approach to train fast and high-fidelity samplers for DDMs. LSD employs a distillation approach where a student sampler with a few steps learns to align its intermediate score trajectory with that of a high-quality teacher sampler with numerous steps. This alignment is achieved by optimizing learnable sampler coefficients that adaptively adjust sampling dynamics. Additionally, we further propose LSD+, which also learns time schedules that allocate steps non-uniformly. Experiments across text generation, image generation, and synthetic tasks demonstrate that our proposed approaches outperform existing samplers for DDMs, achieving substantially higher sampling quality with significantly fewer sampling steps.

Learnable Sampler Distillation for Discrete Diffusion Models
Feiyang Fu, Tongxian Guo, Zhaoqiang Liu" University of Electronic Science and Technology of China
Abstract
Discrete diffusion models (DDMs) have shown powerful generation ability for discrete data modalities like text and molecules. However, their practical appli- cation is hindered by inefficient sampling, requiring large number of sampling steps. Accelerating DDMs by using larger step sizes typically introduces significant problems generation quality, as it amplifies the impact of both the compound- ing decoding стrоr due to factorized predictions and discretization error from numerical approximations, leading to a significant decrease sampling quality. To address these challenges, we propose learnable sampler distillation (LSD), a novel approach to train fast and high-fidelity samplers for DDMs. LSD employs a distillation approach where a student sampler with a few steps learns to align its intermediate score trajectory with that of a high-quality teacher sampler with numerous steps. This alignment is achieved by optimizing learnable sampler coeffi- cients that adaptively adjust sampling dynamics. Additionally, we further propose LSD+, which also learns time schedules that allocate steps non-uniformly. Experi- ments across text generation, image generation, and synthetic tasks demonstrate that our proposed approaches outperform existing samplers for DDMs, achieving substantially higher sampling quality with significantly fewer sampling steps. Our code is available at https://iiuub.co//flyyggggi
Introduction
Diffusion models have demonstrated remarkable success across various generative tasks, particularly excelling in the synthesis of data within continuous domains like images, audio, and videos [ 2 3
5). These models frame the data generation process as a gradual denoising procedure in a continuous latent space. However, many other important data modalities, such as natural language, molecular sequences, and categorical data, inherently possess discrete structures. Applying diffusion models directly to these discrete spaces is challenging, as the standard formulation relies on continuous state transitions. Recently, discrete diffusion models (DDMs) 6. to address this issue. DDMs are specifically designed to operate on discrete data, adapting the core diffusion idea to categorical variables and enabling principled generation. Recent advances DDMs have shown promising results, achieving competitive performance in generating high-fidelity discrete data. Despite their promising applicability, DDMs face an important challenge in sampling efficiency, and they typically require substantial number of function evaluations (NFEs), e.g., 1024 or more, making inference computationally expensive.
10
have been developed
Current sampling methods for DDMs are mainly divided into two categories: 1) Exact simulation methods [3 provide unbiased samples from the target distribution but suffer from high sampling times and expensive computational costs due to numerous model evaluations, leading to poor scaling with dimensionality. 2) Approximate methods like r-leaping [55 [6] are designed for parallelization
"Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

and potentially faster sampling. However, such methods are first-order accurate and require small step sizes to ensure sampling quality.
Directly accelerating the sampling of DDMs through reducing NFEs typically produces unsatisfactory results, since this amplifies the impact of the compounding decoding error [17] and discretization error. Compounding decoding error arises since DDMs employ a factorized parameterization for computational efficiency, predicting the denoised state of each token independently, and ignoring inherent dependencies between tokens in the sequence. Consequently, the learned factorized denoising distribution differs from the true reversal process. This discrepancy exacerbated when reducing NFEs, as the approximation quality degrades over larger intervals. Discretization error occurs since large step sizes make inaccurate for numerical methods like Euler (18) and r-leaping [TS] to approximate the reverse dynamics. Moreover, these two errors accumulate over the sampling trajectory, severely degrading sampling quality when using small NFEs. Throughout the following, we call the combination of compounding decoding error and discretization error as accumulated error for brevity. To address the issue incurred by large accumulated error, we propose learnable sampler distillation (LSD) and its improved version for efficient sampling of DDMs.
1.1 Related Work
Efficient sampling in continuous diffusion models Recent efforts to accelerate sampling in
continuous diffusion models largely focus on reducing the NFEs for solving the reverse-time time ordinary
differential equation (ODE) or stochastic differential equation (SDE) 19 20
21
22
23
24].
One major direction involves designing advanced ODE solvers. Some works 20
25
26
provide efficient sampling methods by establishing high-order numerical ODE solvers for continuous diffusion models.
There are also approaches that learn or optimize various components of the sampling process. AYS (8 seeks non-uniform time step schedules specific to given models and datasets, though their optimization can be computationally intensive. DMN 29 proposes a general framework for designing optimization problem that seeks more appropriate time steps by minimizing the distance between the ground-truth solution to the ODE and an approximate solution corresponding to the numerical solver. AMED-Solver (30) learns adaptive mean estimation directions based on the observation that trajectories often reside in low-dimensional subspaces, which typically involves
training an auxiliary network with high costs.
The most relevant works to us are perhaps LD3 [31] and S4S [32. both proposing learning diffusion model solvers via distillation in the continuous domain. LD3 efficiently learns the time discretization by backpropagating through the ODE-solving procedure using the proposed surrogate loss. $4S further learns the coefficients of the student solver by minimizing the distance between the final samples generated by the student and teacher solvers using learned non-uniform time schedules. However, these approaches face challenges when applied DDMs. We highlight several distinctions in our learnable sampler distillation (LSD) approach (and its improved version) designed to address these challenges. 1) DDM sampling involves nonfdifferentiable categorical sampling at each step, obstructing direct gradient flow from the final discrete output back to the sampler parameters. The reliance of S4S on final sample comparison is thus infeasible. We address this issue by aligning the intermediate score trajectories between the student and teacher samplers. This provides a viable path for gradient-based optimization of the learnable coefficients within the discrete sampling methods. 2) The work for S4S uses the final sample matching error to learn the time schedules, which may ignore dynamic changes of the accumulated error in the intermediate steps. We instead learn the time steps by aligning the effective transition term at intermediate stages during the reverse process. The effective transition term in the reverse process incorporates step sizes and concrete scores, which are tailored for DDMs. 3) The work for S4S optimizes the continuous initial noise using projected stochastic gradient descent (SGD) within an L2 ball. This is inapplicable to DDMs where the initial state is often a discrete sequence, e.g., all masked tokens, which lacks a continuous gradient. To address this issue, we adapt the approach by measuring proximity using Hamming distance, which is suitable for discrete spaces and does not perform gradient updates on itself.
Distillation in Diffusion Models The distillation of continuous diffusion models is rapidly advancing field. A prominent direction related to the consistency model [33], which aims to learn function that maps any point ODE trajectory to its origin, enabling one-step or few-

---

step generation. This paradigm has been extended to multi-step variants [34 35] for improved performance. Other significant works focus on directly matching student and teacher distributions. such as distilling guided diffusion models [36], proposing simplified and faster matching objectives 37,, recursively distilling a deterministic diffusion sampler into a new model 38 or concentrating on one-step distillation (3)]. While these methods are highly effective for continuous models, they usually rely on continuous paths in the sense that the sampling process of each step is differentiable. Our work diverges by proposing a distillation framework specifically for the discrete diffusion model, which does not assume such a continuous path, and addressing a different set of challenges like the non-differentiability of the outputs. A recent work for Di[M]O [40] also involves distilling discrete diffusion models. It distills a multi-step masked diffusion model into a one-step generator. This is achieved by training new student model from scratch, using sophisticated proxy objective that involves creating "pseudo-intermediatee states" and training an auxiliary model to match conditional output distributions. Our approaches are significantly different with Di[M]O in both goal and mechanism. Similarly to LD3 and S4S, we focus on a ftw-step sampler distillation. We tackle the challenge of non-differentiability of sampling from categorical distributions, and we enhance an existing sampler rather than replacing the model, which avoids the complexity of training a new generator and an auxiliary model.
Discrete diffusion models DDMs have emerged and undergone substantial development recently, SEDD 6 proposes score entropy, a novel loss that naturally extends score matching to discrete spaces and integrates seamlessly to build DDMs. RADD [9] reveals that the concrete score in absorbing diffusion can be expressed as conditional probabilities of clean data, multiplied by a time-dependent scalar in an analytic form and it unifies absorbing DDMs and any-order autoregressive models.
Various strategies have been proposed accelerate the sampling of DDMs while maintaining quality. Among these, approximate simulation methods [8] 16 13 g are widely used due to their potential for parallelization. A prominent example is the r-laaping algorithm [8] that is adapted for DDMs. 7-leaping simulates the process by taking an approximate Euler-like step at each data dimension simultaneously and independently. Tweedie r-leaping 6 4T is an extension to r-leaping and is proposed to improve accuracy by specifically considering how the rate matrix changes according to the noise schedule throughout the reverse process. While these r-leaping variants offer the advantage of parallelization, the inherent approximation error still necessitates using many small steps to achieve high sampling quality.
The recent work for JYS 17 attempts to accelerate the sampling process of DDMs by focusing on optimizing the time steps of the sampling schedule. It minimizes Kullback-Leible divergence upper bound (KLUB) that implicitly captures the overall impact of the compounding decoding error and strategically allocates sampling steps. However, JYS operates by optimizing when sample, rather than how to sample. At each chosen time, it still relies on the intrinsically biased model and employs standard large-step approximations that suffer from a significant discretization error.
1.2 Contributions
To address limitations mentioned above, we move beyond fixed or hand-tuned inference strategies. We introduce a novel learnable sampler distillation (LSD) approach specifically for DDMs. We employ a teacher sampler using small step sizes to approximate a high-quality trajectory. A student sampler is then trained with larger step sizes. Instead of mimicking only the final output of the teacher sampler, which is challenging due to non-differentiabiiiiy in the discrete pipeline, the student sampler learns to align its intermediate score trajectory with that of the teacher sampler. This alignment is achieved by optimizing the learnable sampler coefficients, which provide the ability to adaptively adjust the sampling dynamics at each step and potentially compensate for the accumulated error given larger step sizes. Furthermore, we propose LSD+ that also learns sampling time schedules. This is done by comparing the effective transition term in the reverse process at intermediate stages and empirically works better compared with uniform sampling schedules used by LSD. We also utilize a relaxed objective during the learning process alleviate the difficulty of hard alignment between the teacher and student samplers.
Overall, our contribution can be summarized as follows:
3

---

We propose the LSD approach. Inspired by the insight of aligning intermediate score trajectories, LSD trains an efficient student sampler via distillation by optimizing learnable sampler coefficients and incorporating a relaxed training objective for improved feasibility.
We further introduce LSD+, an extension to LSD that additionally learns non-uniform time schedules. This allows for adaptive allocation of sampling steps, offering mechanism to potentially better capture varying dynamics and further reduce accumulated errors compared to using uniform time schedules.
Extensive experiments across text generation, image generation, and synthetic data tasks demonstrate that our proposed approaches achieve significantly higher sampling quality compared to existing baselines at reduced NFEs.
2 Preliminaries
2.1 Continuous time discrete diffusion models
DDMs model the generative process that can be expressed as continuous time Markov chain (CTMC) on a finite state space X' fI,...,Ny 8 42). The forward process describes how data is corrupted. Specifically, the probability of transitioning from state time to state y after a small time interval At is denoted by Pr+ Pr+ (t)))))) This is characterized by [9]
(Q y) + of (A)) 1 (1(z,))(1r)))))))))) o(At),
y リ y ==
Pi+ P++
(I)
where (z, y) is the (a. y) element of the transition rate matrix Q.. The transition rate matrix Q is usually formed as (ft) 8 where (t) is scalar factor, Q is a pre-defined standard matrix with special structures 88] Let Pa denote the marginal distribution of states time t. In particular,
Po Paata is the true distribution of the data. Additionally, for the terminal time T, pr approaches a distribution æ. Depending on Q, π can mainly be modeled as two distributions, namely a uniform distribution distribution that converts samples into masked tokens. For the reverse process that transfers pr back to Po. the inverse CTMC can be characterized as follows 9
(z. yt + o(At),
y ッ
Pt- Arje (y)ax)
+
+ o(
y -
where Qe is the reverse transition rate matrix
41, which can be parameterized by:
(x) (y,
y y x, y =
( v) = 1 E +0 Qx)
(3)
The concrete score term (() needs to be estimated, as Pa is generally unknown. Therefore, the goal (x)
of training a score network 89 X' x IR + x|x is to approximate these score values. For instance, [P(())
SEDD (6] provides effective method that learns so such that it satisfies sp(x,t) R
pr(x) yg
3 Methods
Achieving efficient inference in DDMs with high sampling quality necessitates accurate approxi- mation of the reverse CTMC using significantly fewer steps than traditional high-fidelity samplers. Numerical samplers like the Fuler sampler approximate this process by taking discrete steps guided by the concrete score from the model. However, increasing the step size for faster inference alters the discrete transition dynamics, leading to increased accumulated eπrors and degraded sampling quality. To enable accurate sampling with large step sizes, we propose learnable sampler distillation (LSD) to make some components of the numerical sampler learnable. Specifically, LSD employs learnable coefficients to dynamically adjust the influence of the concrete score at each time step, allowing the sampler to compensate for large-step discretization errors. Furthermore, while LSD could lead to significant improvement in the generation quality, we further propose LSD+. Instead of learning the coefficients using a fixed uniform schedule, LSDD+ further learns a sequence of non-uniform time steps. By training these parameters through distillation from high-quality teacher sampler, our method learns an optimized discrete-time trajectory. Figure the pipeline of our method, and the details of the method are described in the following subsections.

---

Teacher Trajectory
Leamable time schedules
Initial samplo
Student Trajectory
Denosing
Sampling space
earnable sampler cuefficients
ourput
Tre
Fanilla Trajectory
((4)
(b4)
(6s)
Alternative sample training only)
0.754
0.56g
0.256
Figure 1: Illustration of our LSD+ approach. During training, the student sampler starts from an alternative initial sample Stg within the sample space (close to the original zt2)) During inference, sampling starts from the initial sample Zt,- We can see that the vanilla sampling trajectory often introduces significant discretization errors with large step sizes. In contrast, LSD+ employs learnable coefficients (14c) and learnable time schedules 7. to adaptively adjust its trajectory. This enables the L.SD) sampler to more accurately mimic the trajectory of the teacher sampler, effectively compensating for errоrs inherent in accelerated sampling.
3.1 LSD: Coefficients
In this subsection, we illustrate our LSD approach that learns time-dependent coefficients for accel- erating DDMs. Given pre-trained score network so(:,), transition rate matrix Qt, and an initial state UT sampled from pr at the initial time T, the reverse sampling process for DDMs generates a sample by iteratively applying an update rule. For Euler-type sampler, the transition probability for the i-th token from its current state 2; at time to the next state Tt-At at time At can be parameterized as:
p(ai-Ar()) 65g (1)-Ar))) + A A. (0 1i
Here, xf denotes the i-th token of the current state sequence Ie, ( (a))_a)) is the Kronecker delta function, At represents the time step size, Qe of-ad denotes the (xt, (r),"(-A)) element of the transition rate matrix Qt, and (z t) is the (i, x/)))) element of the concrete score so s0(2,, t).
We apply a fixed teacher sampler that approximates the true reverse process with high fidelity using time schedules [ comprising steps, with T = to > t > ... > tyy = c > 0 The state generated by the teacher sampler at time 1ī along its trajectory is denoted o7.- The sampling process of a teacher sampler V yields a high-quality final sample x.- = T (xr, (1 80, (Qtg )o) which is abbreviated as x = " (IT) for simplicity in notation.
We apply a student sampler that operates with time schedule {4 comprising M steps, where tM
M & N and T to f1 > goal to learn set of time-dependent coefficients the output of the student sampler The state generated by the student sampler at time 4, along its trajectory is denoted as Zt,- The sampling process of student sampler W yields a final sample =
0. [ k=o is subsequence of (t7) 1- Our
(T(( to improve the quality of
v (IT, {1x As o se, (Q Jk o { 1). which is abbreviated as I, V (xT) to highlight the dependence on the coefficients (+(4)))
MM
The update rule for the i-th token within the student sampler incorporating & becomes:
p(a (xr,)) = 5xt (xin,)) AtQ (oi ot
(#(+)) (o (x))
(5)
Similar to strategies in some learning methods for continuous ODE solvers [32] direct objective is to minimize the distance d(x,, 2;), where d(:.)) :) is a certain distance metric. However, it is
2 Here, a positive value close to to avoid stability issues as discussed Here, (tto is fixed to 1.
5

---

generally infeasible for DDMs to directly minimize d(z,; 2²) since the non con-differentiable categorical sampling at each step obstructs gradient propagation. Instead, we propose to align intermediate score predictions. At each time step (x, the student sampler computes its score teacher sampler evolves its state to the same time step l (i.e., for certain j such that 13 = (g) using its more accurate sampling process and caches its score 8% = se (zt (x). The states Tte and vi, differ due to the distinct sampling paths taken to reach lx- Then, our objective is to minimize the discrepancy between & and ((xx) (8) for all (,, M). This can be expressed as:
-sp(2 (Ztt la) The
(x()(t))) Bayg ~a (d (st, (14))84]]
(6)
This intermediate score trajectory alignment provides differentiable path for optimizing (*(4))
and ensures the student sampler mimics the trajectory of the teacher sampler across the full denoising path, not just at the final output. We present details of the sampling and training processes for LSD in Algorithms and respectivvl.
3.2 LSD+: Coefficients with Icarnable time schedules
While LSD improves the sampler by learning the sequence of coefficients {+x [tt)) time schedule, the reverse diffusion dynamics vary significantly across time. We additionally propose LSD+ to also learn non-uniform time schedules. The intuition is that, by learning from high-fidelity teacher sampler, the student sampler implicitly learns to allocate its limited steps a manner that best approximates the trajectory of the teacher. Specifically, given time steps for the student sampler (tx I== k= the uniform time schedule uses a step size at Åt T'ce Our goal is to learn customized step sizes {
k=1 under fixed
M
LM which are initialized as At. The learnable time steps are calculated by:
T
t=1
At each learned time step the student sampler computes its score so (x Tx). To learn the step size Kb, we utilize the so-called effective transition term in the reverse process. Specifically, for the student sampler, this is proportional to Ka80( Tk), for the teacher sampler, this is proportional to T (0 fa.), where "H" is the step size for the teacher sampler and sp s0 44) is the cached score of teacher sampler. By caleulating the distance of the effective transition terms between the student sampler and teacher sampler, we effectively update Ką considering the unique characteristics of DDMs. This allows the time schedule to adaptively allocate step sizes based on the specific transition structures in DDMs. The updating process can be parameterized follows:
(x(ss) = Beery ~ [a (muso(en Tk),
T -
se se( (n)
)
(8)
N
We present details of the training and sampling processes for LSD+ in the supplementary material.
Algorithm 1 Sampling process of LSD
Require: Score network se, time schedule (tx (
k= for the student sampler, learned coefficients of the student sampler (1x)) transition rate matrices
1: Sample Eto
2: for k == 0 to M - do
3: Sample Feet: based on Ta, and ((tx):
4: p(atar (xtt,) 5xt (#t (tx tx+1) Qc, (at (ott, (0(t))so (((((1t,)))) 14) (t))
5:
si. ~ plat, (x(, for all
6: end for
7: return 2.

---

Algorithm 2 Training process of LSD
Require:
Score network 8e, frozen teacher sampler y with N steps, learnable student sampler V with M steps, learning rate 7,, distance metric d, time schedule (1))== for the teacher sampler, time schedule [ for the student sampler (a subsequence of (=))))) transition
rate matrices {Q2; 1z=0
1: Initialize (t+)) = 1 for k = 1,..,; M
2: while not converged do
3: Sample Iig n. set sio Ttt
4: for k: == to M do
5: Calculate the state It, generated by the student sampler at time and calculate the score
= se( fx)
6: Calculate the state ot, generated by the teacher sampler at time ty and calculate the score
$% so so tk)
7:
end for
8:
for
1 M do ((
9:
La 4- d( (1x))xx))) )8k oi)
10: (14x) (11)) (bt2)) lxk
11: end for
end while
13:
return {*(t)))
3.3 Relaxed objective
For a student sampler which typically has lower NFEs compared to a teacher sampler, non-trivial to force it to accurately match the output of the teacher sampler given the same initial input 2io Thus, we adopt a relaxed training objective for both LSD and LSD+. We take LSD as an example for further presentation. Instead of strictly requiring the score of the student sampler sø(It,, to) to match the score the teacher sampler sæ (z to), we only require that there exists an alternative input tt. sufficiently close the original 2t, (within small Hamming distance 43 in discrete token space). Specifically, it satisfies:
dH 3
where .:. denotes the Hamming distance between two sequences, represents positive integer threshold that defines the maximum allowed Hamming distance between Vt, and Itg where we set it as around 5% of the sequence length.
Therefore, the output of the student sampler at this perturbed score should approximately match the score of the teacher sampler at the original input such that s0 to) sa(zt, to). Moreover, the relaxed objective function for LSD can be expressed as:
Lyslaxod, (tx))) Ez.g 2no (d (s tx), ( (stt))s(fr... tx))]
(10)
where Σt, and satisfies Eq. 4 and it, is sampled starting from Zt.. This relaxation makes the optimization task more feasible for the conntity-oosttrained student sampler by alleviating the rigorous matching requirement. Notably. this input perturbation Tt, is only used during training, inference time, the student sampler receives the original and unperturbed input Itg- We provide further discussion on the reasonableness of the relaxed objective in the supplementary material.
4 Experiments
In this section, we empirically evaluate the performance of our proposed LSD approach and its improved version LSD+. Our goal is to validate their ability to generate high-quality samples at low NFEs. We conduct evaluations across diverse settings, including text generation, image generation, and a synthetic sequence task, comparing against various baselines. We highlight that our LSD+ provides an efficient learning process for the coefficients and time schedules. typically requiring minutes on NVIDIA RTX4090 GPU, compared to around 10 minutes of training time for JYS under the same environment. And the learned student sampler introduces no additional computational burden during sampling.
7

---

4.1 Text generation
For the text generation task, we employed three pre-trained DDM backbones for validation, namely SEDD-small [6], SEDD-medium [6], and RADD [9]. These are absorbing DDMs of GPT-2 level for text generation, trained on the Open WebTWe dataset 44 For the uniform DDMs, please refer the supplementary material. We compare L.SD and LSD+ against standard Fuler and Tweedie samplers [6] and the JYS method [17]. For the RADD baseline, we also compare with higher-order samplers, the 0-RK-2 and d-trapezoidal generated text using GPT2-large model. We generated 1024 samples, each containing 1024 tokens.
45 We evaluate the generative perplexity of unconditionally
The results are presented in Tables 1 2
and 3 LSD (LSD+)-Euler (Tweedie) denotes that
implement LSD (LSD+) based on the Euler (Tweedie) sampler [6]. The empirical results show that methods significantly outperform the baseline methods across all three backbones and all tested NFEs. Moreover, we find that LSD+ generally outperforms LSD, which indicates that the learned non-uniform time schedules help further reduce accumulated errors. Therefore, only present the results for LSD+ for the experiments in Sections 4.2 and 4.3
Table I: Comparison of generative perplexity (s) on the SEDD-small backbone, Best performances bolded.
NFEs
16
32
64
Sampler
Buler
423.109
215.472
72.820
56.218
Tweedie
404.881
177.539
64.347
50.151
JYS-Euler
308.123
125.283
55.842
32.943
JYS-Tweedie
307.382
127.232
56.382
31.192
LSD-Euler
145,490
88.564
31,235
21.956
LSD-Tweedic
168. 186
86.282
35.786
21.981
LSD+-Euler
128.413
51.769
36.800
20.728
LSD+-Tweedie
137.862
60.970
38.157
20.473
CTAC taker
LDD-Teeede
28-
230
Arte
(b)
Figure 2: Validation on (a) the image generation task and (b) the synthetic countdown task. Our LSD+ method shows superior performance.
4.2 Image generation
We also validate our LSD+ approach on the image generation task for the CIFAR-10 dataset [46
We utilize CTMC [ as the baseline, which employs Gaussian transition matrix and denoising
parameterization. Each data sample is flattened image with a size of 32 32, composed of
"Since the source code of (45) is inaccessible, we could only compare our method with these high-order samplers on the RADD backbone. The results are reported from the paper. "/" in Table 3 that this paper does report the results when NFEs is 8.

---

Table 2: Comparison of generative perplexity (5) on the SEDD-medium backbone, Best performances are bolded.
NFEs
32
Sampler
Fuler
399.315
184.603
77.925
44.370
Tweedie
394.470
178.485
67.114
40.487
JYS-Euler
299.394
115.853
43,958
25.545
JYS-Tweedie
300.492
118.218
48.430
28.539
LSD-Euler
125,769
54.865
28.001
19,886
LSD-Tweedie
98.209
51.223
26.668
20.794
LSD+-Euler
121.583
46.145
25.144
15,929
Tweedie
90.033
50.765
26.799
16.239
Table 3: Comparison of generative perplexity (5) bolded.
on the RADD backbone. Best performances are
NFEs
8
16
32
64
Sampler
Fuler
670.977
282.115
152.403
113.913
Tweedie
648.736
285.471
155.472
98.879
9-RK2
127.363
109.351
66.549
0-Trapezoidal
123.585
89.912
66.549
LSD-Euler
121.420
60.219
46.268
32.817
LSD- -weedie
122.167
67.176
40.274
28.644
LSD+-Fuler
89.830
36.106
33.234
29.115
L.weeeeeeee
90.364
40.263
36.129
24.312
tokens with values ranging from 0 to 255. We evaluate the FID score using 50k samples with the NFEs selected from {8, 16, 32, 64). Figure 2(a) shows the results and we observe that method provides better FID scores compared to the baseline method.
4.3 Synthetic countdown task
We follow [47 to evaluate LSD+ approach synthetic sequence task with strong dependencies. The dataset features 256-token sequences (with values 0-31) where non-zero tokens must strictly decrease by one. We trained an absorbing SEDD [6] model and measured performance by the error rate, which the proportion of generated samples violating this countdown rule. As shown in Figure (
(), method achieves lower enror rates across various NFEs compared to baselines.
5 Ablation Study
The learnable coefficients form the core of our LSD approach and have demonstrated significant performance improvements. Also, we observe that LSD+ generally outperforms LSD seen in Tables [ 2 and which indicates the benefit of the learned non-uniform time schedules. Therefore, our ablation study aims assess the contributions of the relaxed objective during training.
Benefit of the relaxed objective We proposed relaxed objective, allowing the student sampler to match the trajectory of the teacher sampler originating from It, by using a perturbed starting point Rt, that is close to It, during the training process. compares the performance of L.D+ trained with and without this relaxation. The results clearly indicate that employing the relaxed objective generally yields better performance than training with the strict objective. This validates the benefit the relaxation, confirming that makes the trajectory alignment task more feasible and leads to better convergence.
Impact of Hamming distance threshold To investigate the robustness of the algorithm to the Hamming distance threshold, conduct the ablation the SEDD-small backbone using the Euler

---

sampler with 32 inference steps. We train our LSD+ method using several different values for the Hamming distance threshold, specifically 0%, 1%, 5%, 10%. 20% of the sequence length, while keeping all other hyperparameters unchanged. The performance, measured by Perplexity, is reported below T
Table 4: Ablation study on the RADD backbone validating the importance of the Relaxed Objective (RO), "w/o RO" indicates parameters learned without relaxed objective, while "w/ RO" denotes parameters learned using proposed relaxed objective. Both settings evaluated using either Euler or Tweedie as the base sampler for our LSD+ method. Best performances are bolded.
NFEs
16
32
64
Sampler
LSD+ w/o RO-Euler
95. 95,943 89.830
38. 192
34,983
31.392
LSD+ w/ RO-Euler
36.106
33.234
29.115
Table 5: Ablation study on the Hamming distance threshold for the relaxed objective. Threshold(%))
O
(Our choice)
10
20
Perplexity())
35.98
32.15
31.24
39.97
51.52
Conclusion
This paper aims to address the challenge of inefficient sampling in DDMs, a major obstacle to their practical deployment. While reducing the NFEs accelerates inference, previous accelerating methods suffer from accumulated compounding decoding error and discretization errors, significantly degrading sampling quality. We introduce LSD, novel approach that leverages distillation from high-fidelity teacher sampler. Instead of merely matching final outputs, LSD trains a student sampler with few steps to align its entire intermediate score trajectory with that of the teacher sampler. This is achieved by optimizing learnable, time-dependent coefficients. And we additionally propose LSD+ that also learns non-uniform sampling schedules and this allows the sampler to adaptively compensate for errors induced by larger step sizes. Extensive experiments demonstrate that our methods significantly outperform the baseline samplers across diverse tasks, achieving high sampling fidelity at low NFEs.
A promising direction for future research is to provide theoretical guarantees regarding the distri- butional discrepancy between the outputs of teacher and student samplers, potentially building on existing theoretical findings related to discrete diffusion models 48 49 50
5
52
45 53 54
Acknowledgment. We sincerely thank the five anonymous reviewers and the area chair for their meticulous reading of our work and their constructive comments, which have substantially enhanced the quality and rigor of our study.
10

---

References
[1] Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. In International Conference Learning Representa- rions, 2022.
[2] Jonathan Ho. Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models, In Advances in Neural Information Processing Systems, 2020.
[3] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. DiffWave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021.
[4] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps. In Advances in Neural Information Processing Systems, 2022.
[5] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, 2021.
[6] Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In International Conference on Machine Learning, 2024.
[7] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Struc- tured denoising diffusion models in discrete state-spaces. In Advances in Neural Information Irocessing Systems, 2021.
[8] Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. In Advances Neural Information Processing Systems, 2022.
[9] Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. In International Conference on Learning Representations, 2025.
[10] Chenlin Meng, Kristy Choi, Jiaming Song, and Stefano Ermon. Concrete score matching: Generalized score matching for discrete data. In Advances in Neural Information Processing Systems, 2022.
[11] Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky TQ Chen, Gabriel Synnaeve. Yossi Adi, and Yaron Lipman. Discrete flow matching. Advances in Neural Information Processing Systems, 2024.
[12] Zixiang Chen, Huizhuo Yuan, Yongqian Li, Yiwen Kou, Junkai Zhang, and Quanquan Gu. Fast sampling via discrete non-Markov diffusion models with predetermined transition time. In Advances in Neural Information Processing Systems, 2024.
[13] Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming-Yu Yu Liu, Jun Zhu, and Qinsheng Zhang Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. In International Conference on Learning Representations, 2025.
[14] Jiaxin Shi, Kehang Han, Zhe Wang, Arnau Doucet, and Michalis Titsias. Simplified and generalized masked diffusion for discrete data. In Advances in Neural Information Processing Systems, 2024.
[15] Daniel T Gillespie. Approximate accelerated stochastic simulation of chemically reacting systems. The Journal of Chemical Physics, 2001.
[16] Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association, 2011.
[17] Yong-Hyun Park, Chieh-Hsin Lai, Satoshi Hayakawa, Yuhta Takida, and Yuki Mitsufuji. Jump your steps: Optimizing sampling schedule of discrete diffusion models. International Conference Learning Representations, 2025. 2025.

---

[18] David F Anderson, Arnab Ganguly, and Thomas G Kurtz. Error analysis of tau-leap simulation methods. Annals of Applied Probability, 2011.
[19] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[20] Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. DPM-Solver-w3: Improved diffusion ODE solver with empirical model statistics. In Advances in Neural Information Processing Systems, 2023.
[21] Shuchen Xue, Mingyang Yi, Weijian Luo, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhi-Ming Ma. SA-Solver: Stochastic Adams solver for fast sampling of diffusion models. In Advances Neural Information Processing Systems, 2023.
[22] Gen Li, Yu Huang, Timofey Efimov, Yuting Wei, Yuejie Chi, and Yuxin Chen. Accelerating convergence of score-based diffusion models, provably. In International Conference on Machine Learning, 2024.
[23] Jiajun Ma. Shuchen Xue, Tianyang Hu, Wenjia Wang, Zhaoqiang Liu, Zhenguo Li, Zhi-Ming Ma, and Kenji Kawaguchi. The surprising effectiveness of skip-tuning in diffusion sampling. In International Conference on Machine Learning. 2024.
[24] Mingyuan Zhou, Huangjie Zheng, Zhendong Wang. Mingzhang Yin, and Hai Huang. Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. In International Conference on Machine Learning, 2024.
[25] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM- Solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095 2022.
[26] Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. In International Conference on Learning Representations, 2023.
[27] Tim Dockhorn, Arash Vahdat, and Karsten Kreis. GENIE: Higher- denoising diffusion solvers. In Advances in Neural Information Processing Systems, 2022.
[28] Amirmojtaba Sabour, Sanja Fidler, and Karsten Kreis. Align your steps: Optimizing sampling schedules in diffusion models. In International Conference on Machine Learning, 2024.
[29] Shuchen Xue, Zhaogiang Liu, Fei Chen, Shifeng Zhang, Tianyang Hu, Enze Xie, and Zhenguo Li. Accelerating diffusion sampling with optimized time steps. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[30] Zhenyu Zhou, Defang Chen, Can Wang. and Chun Chen. Fast ODE-based sampling for diffusion models in around steps. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[31] Vinh Tong, Trung- Hoang. Anji Liu, Guy Van den Broeck, and Mathias Niepert. Learning to discretize denoising diffusion ODEs. In International Conference Learning Representa- tions, 2025.
[32] Eric Frankel, Sitan Chen, Jerry Li, Pang Wei Koh, Lillian Ratliff, and Sewoong Oh. S4S Solving for diffusion model solver. In International Conference on Machine Learning, 2025.
[33] Yang Song, Prafulla Dhariwal, Mark Chen, and llya Sutskever. Consistency models. In International Conference on Machine Learning, 2023.
[34] Nicholas M Boffi, Michael S Albergo, and Eric Vanden-Fijnden How to build a consistency model: Learning flow maps via self-distillation. arXiv preprint arXiv:2505. 18825, 2025.
[35] Jonathan Heek, Emiel Hoogeboom, and Tim Salimans. Multistep consistency models. arXiv preprint arX1v:2403.06807, 2024.
12

---

[36] Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[37] Zhenyu Zhou, Defang Chen, Can Wang, Chun Chen, and Siwei Lyu. Simple and fast distillation of diffusion models. In Advances in Neural Information Processing Systems. 2024.
[38] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022.
[39] Sirui Xie, Zhisheng Xiao, Diederik Kingma, Tingbo Hou, Ying Nian Wu, Kevin P Murphy, Tim Salimans, Ben Poole, and Ruiqi Gao. EM distillation for one-step diffusion models. In Advances in Neural Information Processing Systems, 2024.
[40] Yuanzhi Zhu, Xi Wang, Stéphane Lathuilière, and Vicky Kalogeiton. DiMMO: Distilling masked diffusion models into one-step generator. In IEEE International Conference Computer Vision, 2025.
[41] Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based continuous-ting discrete diffusion models. In International Conference on Learning Representations, 2023.
[42] Andrew Campbell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. Gener- ative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. In International Conference on Machine Learning, 2024.
[43] Mohammad Norouzi, David Fleet, and Russ R Salakhutdinov. Hamming distance metric learning. In Advances in Neural Information Processing Systems, 2012.
[44]
Aaron Gokaslan and Vanya Cohen. WebTTet Corpus. lattp://Skylion007/ wwt.::/.iioo0.....tiii
i ///DpenWWebeeextorppuus 2019.
[45] Yinuo Ren, Haoxuan Chen, Yuchen Zhu, Wei Guo, Yongxin Chen, Grant M Rotskoff, Molei Tao, and Lexing Ying. Fast solvers for discrete diffusion models: Theory and applications of high-order algorithms. arXiv preprint arXiv:2502.0023 002344 2025.
[46] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
[47] Yixiu Zhao, Jiaxin Shi, Feng Chen, Shaul Druckmann, Lester Mackey, and Scott Linderman. Informed correctors for discrete diffusion models. arXiv preprint 2rXiv:2407.21243 2024.
[48] Yinuo Ren, Haoxuan Chen, Grant M Rotskoff, and Lexing Ying. How discrete and continuous diffusion meet: Comprehensive analysis of discrete diffusion models via a stochastic integral framework. In International Conference on Learning Representations, 2025.
[49] Hongrui Chen and Lexing Ying. Convergence analysis discrete diffusion model: Exact imple- mentation through uniformization. In International Conference on Learning Representutions, 2025.
[50] Zikun Zhang, Zixiang Chen, and Quanquan Gu. Convergence of score-based discrete diffusion models: A discrete-time analysis. In International Conference on Learning Representations, 2025.
[51] Guhao Feng, Yihan Geng, Jian Guan, Wei Wu, Liwei Wang, and Di He. Theoretical benefit and limitation of diffusion language model. arXiv preprint arXiv:2502.09622, 2025.
[52] Gen Li and Changxiao Cai. A convergence theory for diffusion language models: An information-theorete perspective. arXiv preprint arXiv:2505.21440, 2025.
[53] Leo Zhang. The cosine schedule is Fisher-Rao-onn Rashor-Rao-ppiimal for masked discrete diffusion models. arXiv preprint arXiv:2508. 2025.
[54] Ruixiang Zhang, Shuangfei Zhai, Yizhe Zhang. James Thornton, Zijing Ou, Joshua M Susskind, and Navdeep Jaitly. Target concrete score matching: holistic framework for discrete diffusion. In International Conference on Machine Learning, 2025.
13

---

[55] Yuchen Liang, Renxiang Huang, Lifeng Lai, Ness Shroff, and Yingbin Liang. Absorb and converge: Provable convergence guarantee for absorbing discrete diffusion models. arXiv preprini 1xXiv:2506.023/8, 2025.
[56] Zikun Zhang, Zixiang Chen, and Quanquan Gu, Convergence of score-based discrete diffusion models: A discrete-time analysis. In International Conference on Learning Representations. 2025.
[57] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. MaskGIT: Masked generative image transformer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[58] Victor Besnier, Mickael Chen, David Hurych, Eduardo Valle, and Matthieu Cord. Halton scheduler for masked generative image transformer. arXiv preprint arXiv:2503. 170766633..777,.. 2025.
[59] Justin Deschenaux and Caglar Gulcehre. Beyond autoregression: Fast LLMs via self-distillation through time. In International Conference on Learning Representations, 2025.
[60] Guanghan Wang, Yair Schiff, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Remasking discrete diffusion models with inference-time scaling. arXiv preprint arXiv:2503.00337, 2025.
[61] Subham Sekhar Sahoo, Marianne Arriola, Aaron Gokaslan, Edgar Mariano Marroquin, Alexan- der M Rush, Yair Schiff, Justin T Chiu, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. In Conference on Neural Information Processing Systems, 2024.
[62] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
[63] Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. Fast-dLLM: Training-frre acceleration of diffusion LLM by enabling KV cache and parallel decoding. arXiv preprint arXiv:2505.22611 2025.
[64] Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, al. Scaling diffusion language models via adaptation from autoregressive models. In International Conference on Learning Representations, 2025.
[65] Myle Ott, Sergey Edunov, Alexci Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. FAIRSEQ: A fast, extensible toolkit for sequence modeling. arXiv preprint WrXiv:1904.01035 2019.
[66] Andrzej Mačkiewicz and Waldemar Ratajczak. Principal components analysis. Computers & Geosciences, 1993.
[67] Zhaoqiang Liu and Vincent YF Tan. The informativeness of k-meaan for learning mixture models. IEEE Transactions Information Theory, 65(11):7460-7477) 2019.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We provide brief and clear main claims in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in the supplementary material.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
Our paper does not involve theoretical proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the implementation details in Section 4 and the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer:
[No]
Justification: The data used in the experiments are open- and our code will be released upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
httpollliigggddde/oooooeo http://uddde/ooeSuubnrb ssicnpolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 4 and the supplementary material provide the experimental settings. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We follow closely relevant works such as SEDD 6 and RADD 9 the quantitative results without reporting error bars.
to calculate
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide sufficient information on the computer resources in the supple- mentary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///wwwwp..cccaaaaccccccaaaa
Answer: [Yes]
Justification:
We have read and conformed with the NeurIPS Code of Ethics every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
We provide discussions on potential societal impacts in the supplementary
material.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the original paper that produced the code package dataset.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
21

---

Supplementary Material
Learnable Sampler Distillation for Discrete Diffusion Models (NeurIPS 2025)
Feiyang Fu, Tongxian Guo, Zhaoqiang Liu
A Limitations and Broader Impact
While our proposed learnable sampler distillation (LSD) approach significantly enhances the sampling quality of discrete diffusion models (DDMs) at low NFEs, there exists a limitation that warrants discussion. Specifically, the performance ceiling of the student sampler inherently tied to the quality of the teacher sampler. Consequently, while our student sampler efficiently achieve the performance of the teacher sampler with very few NFEs, surpassing it is challenging. However, this dependence the teacher is intrinsic characteristic of knowledge distillation paradigms and LSD compatible with future advancements, since more sophisticated samplers can serve as improved teacher samplers.
Our approaches can be integrated with various DDM samplers, offering a general path to enhance their efficiency, and hold significant promise for accelerating scientific discovery, such as by facilitating the design of candidate DNA and protein sequences. However, the increased power of generative Al also necessitates a commitment to responsible development. This includes proactive efforts to mitigate societal risks, notably the potential for generating misinformation or amplifying existing biases, and underscores the importance of ethical guidelines and detection research.
B Discussion
B.1 Intuitive explanation
Given that several theoretical works on DDMs have been conducted [4 53 56 48), we provide an intuitive explanation for the effectiveness of our method, which may help lay the groundwork for subsequent theoretical framing. Specifically, the final discrepancy between the outputs of the student and teacher samplers stems from the accumulation of small local errors made each step. Fach of these local errors, in turn, tied to the specific score predicted by the model at that step. Our training objective directly enforces alignment between the score predictions of the student sampler and those of the teacher sampler at every step. By implicitly correcting these small local errors throughout the process, our approach guides the student sampler produce final outputs that closely match the high-quality results of the teacher sampler.
B.2 Relaxed objective
Our use of a relaxed training objective enhances training feasibility compared to strict alignment. However, the theoretical guarantees that the resulting student distribution closely matches the teacher distribution might be less rigorous than what could potentially be argued for in continuous spaces. To the best of our knowledge, establishing such rigorous thcoretical bounds in discrete spaces faces several challenges not present in their continuous counterparts. Specifically, (1) In continuous diffusion, initial state perturbations can often be controlled and analyzed via differentiable operations (e.g., Lozconntraieee gradient steps) (32) In contrast, discrete initial states (e.g., token sequences) and their perturbations that are measured by Hamming distance lack this differentiabilityy This precludes similar continuous optimization and analysis pathways. (2) The analytical tools used for relaxed objective continuous diffusion ODE/SDE, such as perturbation analysis for smooth dynamical systems (31) do not directly translate to the discrete dynamics of CTMCs and their approximations. Therefore, our proposed relaxed objective serves as an empirically effective solution for training fcasibility in LSD.
22

---

C Details for discrete diffusion models
In the main text, we illustrate the mechanism by which the forward process of CTMC introduces corruption into the data. Within Eq. ① the term Qx(2, 2) is delineated as the transition rate from state z to state at time t. Consistent with this definition, Qe(a, y) can be articulated as follows:
Qx(2,v) = limar-+o (xla)
y リ I: y == =
At [limar+0 Pr (1))))
(11)
A
For instances where > define (s), y) == (tlo))))) Drawing upon Kolmogorov's forward equation g and the definition of Qt, we derive
d
Piss = P.
(12)
dt
The analytical solution to Eq. (12) is given by Pris exp ((ō(t) (s)))),, with ā (t) denoting the integral So (s)dss and exp denoting the matrix exponential function. This solution facilitates the direct sampling of dt from x, in single step for all / > scenarios.
The transition rate matrix Q, formulated as (t)Q, where Q represents a pre-specified standard matrix. In the context of defining matrix Q. two principal alternatives are presented: A uniform distribution or a MASK absorbing state. When the base transition matrix Q is selected be uniform matrix, it simulates a fully connected graph structure. Within this framework, each state is interconnected with all other states, implying that transitions from any given state any other state are possible. The definition of this transition matrix ensures extensive exploratory capabilities of the state space, permitting the model account for all potential state transitions during simulation. Specifically, the construction of Quiiform is as follows:
1
1
1 1-
N
1
1
Quiiform
(13)
1-N
1
1
1-N
An alternative approach in the construction of the transition matrix Q includes the formulation of a matrix that encompasses absorbing states. An absorbing state refers to state, where the system will no longer undergo state transitions once it enters this state, and such design facilitates the rapid convergence of the model to a stable state. Recent works [6 9 have also demonstrated that the adoption of an absorbing matrix is associated with better performance and serves to accelerate the sampling process. The construction of Qahsarb is defined as:
Jabsorb
(14)
To effectively simulate the reverse process, the common practice is to train neural network sa(2, () approximate the required concrete score ps(v) In this process, optimize the training of network so (r, 1), SEDD 6 introduces an effective loss function:
Pe(m))
1 E Em-pyo(2)2o)
EQe (v. so((x,t) Pejo (y) mo) log so( t)y c) dt, goffe
(15)
Pijo (x z0)
where is constant with C' K estimate by the neural network of the probability of transitioning from state to state y at time t.
Pejo(z and K(a) == aloga a, so(x, t)y represents an
23

---

D Additional method details
D.1 LSD using Tweedie 7-leaping
In the main text, we detailed the application of our LSD appoach to an Euler-type discrete sampler. Notably, LSD can also be readily integrated with other sampling methods, such Tweedie r-leaping.
The Tweedie 7-leaping update rule for DDMs leverages Tweedic's formula to relate the conditional expectation of cleaner state to the score function. Given time schedules (tx k==0 for the student sampler. The transition probability p( ( for the i-th token from state 2 at time ly state Fty11 at the subsequent time step 1a+1 is constructed as follows:
p(riers (2bta) = -exp ((o(t++)) - ((x)))Q) so( 1))aa
x (exp ((o(tx) - ((tx+1))
(16)
where Q is the predefined standard matrix with special structure as mentioned in Section C
(())
be the cumulative noise schedule, which is non-decreasing function of s0 (x), 4x); is the i-th
element of the score 8o ( 4.). The subscript s on the first main term denotes the æt -th element of the resulting vector. Similarly, the subscript (2t, #/e1)) the second main term denotes selecting the corresponding element from the second main term.
To incorporate our LSD approach, we introduce the learnable time-dependent coefficient (+x to modulate the score network within the Tweedie update. The LSD-modified Tweedie r-leaping update rule is as follows:
p( (2//)) (exp ((o(t++1) {(t)))Q) (F(tx)so tx)))) oi
x (exp ((t(t)) ((tx+1)))) of
(17)
This formulation allows the student sampler using Tweedie 7-leaping learn an adaptive scaling { (1 of the score guidance.
D.2 Pseudocode for LSD+
In this section, we present the sampling and training processes of LSD+ in Algorithms and Empirically, we do not learn the coefficients and time schedules in the same training epoch, which empirically leads training instability issues. Instead, we learn them separately different training epochs. For brevity, we only list the algorithm that learns the time schedules, rather than the whole learning process.
Algorithm 3 Sampling process of LSD+
Require: Score network sp, learned time schedule {Tx (M for the student sampler with M steps, learned coefficients of the student sampler (T((T)) M transition rate matrices (Qr, I
1: Sample It, ~
2: Sample Iru+i based on Hr, and ((TT):
3: p(atee, (xte) = dns, (rth.) + (Th Te+i) 1 Qr (1 stre)) (TT))( T
Te)li
4: 21 ~ p(z= lof for all
5: return Z.
24

---

Algorithm 4 Training process of LSD+ that learns the time schedules
Require:
Score network sp. frozen teacher sampler V with steps. learnable student sampler V with M steps, learning y,, distance metric d. time schedule {4j]5= for the teacher sampler, original time schedule {t for the student sampler (a subsequence of {tz (15))=)) transition rate matrices {Q2; T==0
1.N
1:
Initialize step sizes кą = la-dur for = 1,2....; M
2: Initialize learnable time schedule Tę = -it Ke for k = 1,,,..., M
3: while not converged do
4: Sample Ftt π. set zio 2to
5: for k to M do
6:
Calculate the state Ir, generated by the student sampler time т and calculate 8
so (0 Th.
7:
Calculate the state ot, generated by the teacher sampler at time and calculate the score
st so(x7,, .k)
8:
end for
9:
for to M do
15 15-1x st LA f- Sk:
10:
11:
Kk Kk V/ n2 Lik
Tk = to Σ i== Ki
13:
end for
end while
15:
IMM return {NA [==
E Additional empirical details
E.1
Training settings
In this subsection, provide training settings and implementation details for reproducing empirical results. Specifically, we set the terminate time as 0.0001, the total sampling steps N of the teacher sampler as 1024, and the distance metric d as the KL divergence. We the number of training samples as 64, the training epoch as 20, and the learning rate as 0.001. All experiments are conducted on an NVIDIA RTX4090 GPU.
E.2
More results on large-scale image datasets
We conduct experiments on ImageNet (256x256) using the MaskGIT 57
architecture as the back- bone, incorporating the recently proposed advanced Halton sampler (58 the sampling method. The results are reported 6 which demonstrates that our LSD+ method yields improved generation performance measured by the FID metric. We also study the effect of softmax temperature in MaskGIT on ImageNet (256x256) for the image generation task, with the results listed in Table 7
Table 6: Comparison on ImageNet 256x256 (in terms of FID.)
NFEs
32
Sampler
Halton
14.16
10.15 8.66
8.89
6.92
LSD+-Halton
12.78
7.17
6.32
Table 7: Comparisons on ImageNet 256x256 (in terms of FIDJ when NFE=4).
Temperature
0.6 0.8
1.0
Sampler
Halton
54.05 48.29
26.45 24.51
14.16
LSD+-Halton
12.78
25

---

E.3 More results on text generation.
We provide more results on the task of text generation.
We first conduct experiments predictor-corrector solvers in -LDR-10 small backbone, with the results presented in Table [
8 sampler on the SEDD
To further break through the limitation of only relying on perplexity to assess generation quality and provide a more thorough evaluation, we conduct experiments on the SDTT-KLD [59] backbone. which utilizes Ancestral as the sampler. We report MAUVE, Perplexity and Entropy scores in Table 9 and these results demonstrate that our LSD+ method usually yields improved generation performance.
Meanwhile, we also integrate LSD+ with ReMasking (ReMDM) [60] on the MDLM [6]] backbone, with relevant findings reported in Table IO The results show that LSD+ effectively learn to work in conjunction with this method, further improving performance.
To ensure the generalization of our main results, we further re-evaluate our findings on the SEDD and RADD backbones, using Llama-3-88 [62] to assess perplexity. The re-evaluation results are reported in Tables II and 12 Morcover, also integrate our LSD+ with multi-token unmasking heuristics in
|63 and report the corresponding accuracy and throughput metrics Table These metrics indicate that our methods can improve both sampling quality and speed.
FastDLLMs
To demonstrate that our method is not limited smaller models, we conduct experiments on applying LSD+ to larger-scale models including DiffuLLaMA and DiffuGPT [64]. Specifically, we perform sampler distillation on pre-trained DiffuGPT-S and D checkpoints, and the results in terms of Perplexity are presented in Tables 14 and 15
Finally, conduct experiments the sampler of DNDM
12 which uses the FairSeq
65 the
backbone, the corresponding perplexity results are reported in Table I6
Table 8: Comparisons on the SEDD-small backbone.
Sampler NFEs
Perplexity())
Entropy(t)
16
32
64
128
16
32
64
128
-LDR-10
443.17
318.44
277.16
199.51
5.63
3.69
5.57
5.24
LSD+~1DD--10
205.42
143.58
114.93
90.43
5.58
5.49
5.59
5.44
Table 9: Comparisons the SDTT-KLD backbone.
Sampler
MAUVE(1)
Perplexity())
Entropy(1)
NFES
16
32
32
16
Ancestral
0.884
0.912
0.943
110.391
56.652
42.128
5.331
5.285
5.222
ASDetAneestral
0.905
0.928
0.951
68.130
36.577
31.597
5.298
5.239
5.226
Table 10: Comparisons on the MDLM backbone.
Sampler
Perplexity())
Entropy(t)
NFEs
16
32
64
128
16
32
64
128
ReMDM
434.08
174.72
85.15
62.33
5.73
5.66
5.48
5,55
LSD+-ReMDM
201.52
102.02
62.97
49.33
5.41
5.42
5.52
5.33
Table 11: Comparisons on SEDD-small backhone (in terms of Perplexity.). by LLLMAA3-8B.
NFEs
8
32
64
Sampler
Euler
116.93
67.43
49.81
46,88
LSD+-Euler
74.65
33.25
30.70
21.64
26

---

Table 12: Comparisons on RADD backbone (in terms of Perplexity(), judged by LLaMA-3-8B.
NFEs
16
32
64
Sampler
Buler
337.04
216.01
119.25
95.06
LSD+-Euler
130.00
60.90
42.53
35.65
Table 13: Integration with Fast-dLLM on LLaDA, We report accuracy (throughput, token/s).
Samplers
LLaDA
+Cache
+Parallel
La
LSD+-Fast-dLLM
Benchmark
(SMSK(5-shot)
79.3(6.7)
79.5(21.2)
79.2(16.5)
78.5(54.4)
79.0(62.5)
MATH(4-shot)
33.5(9.1)
33.3(23.7)
33.4(24.8)
33.2(51.7)
33.4(58.1)
Table 14: Comparisons on the DiffuGPT-S backbone (in terms of Perplexity
NFEs
16
32
64
128
Method DiffuGPT-S
117.32
75.19
58.34
37.16
LSD+-DiffGPT-S
53.95
41.37
32.10
22.25
Table 15: Comparisons on the DaA backbone (in terms of Perplexity.),
NFEs
16
32
64
128
Method
DiffoLLaMA
100.04
69.11
42.17
30.55
LLMA
49.83
34.32
29.18
24.72
Table Comparisons on the FairSeq backbone (in terms of Perplexity
NFEs
8
16
32
64
Method
DNDM
919.23
774.92
748.41
622.14
LSD+DNDM
601.22
554.19
477.10
403.13
E.4 More results on uniform discrete diffusion models
In the main text, we provide comparisons with existing samplers on the absorbing DDMs. In this subsection, provide additional empirical results uniform DDMs in Tables 18 and 19 We oberseve that our methods also outperforms existing sampling methods, validating the superiority and robustness of our methods.
Table 17: Comparison of generative perplexity (5) on the uniform SEDD-small backbone. Best performances are bolded.
NFEs
32
64
Sampler
Euler
467.832
224.954
76.364
54.293
Tweedie
433,590
215.233
70.361
52.364
JYS-Euler
310.329
130.034
60.574
33.951
JYS-Tweedie
308.732
129.843
55.293
30.675
LSD-Euler
160.811
103.346
38.059
23.365
LSD-Tweedic
177.728
101.213
37.682
25.675
LSD+-Fuler
157.535
46.448
39.850
23,938
LSD+-Tweedie
128.910
47.080
38.058
21.724
27

---

Table 18: Comparison generative perplexity (5) on the uniform --eD-um backbone. Best
performances are bolded.
NFEs
32
64
Sampler
Euler
403.654
190.784
80.094
47.885
Tweedie
387.743
182.312
65.853
44.754
JYS-Euler
311.427
121.954
49.912
32.934
JYS-Tweedie
306.089
116.233
45.287
28.192
LSD-Euler
132.209
60.823
28.283
22,956
LSD-Tweedie
118.138
53.591
27.162
21.308
LSD+-Euler
111.870
49,427
25.732
19.623
93.196
47.932
27.381
18.119
Table 19: Comparison of generative perplexity (5) on the uniform RADD backbone. Best perfor-
mances are bolded.
NFEs
8
16
32
64
Sampler
Euler
657.732
280.743
157.825
115.743
Tweedie
652.381
277.195
160.045
108.730
LSD-Euler
126.394
65.923
51.197
34.764
LSD-Tweede
127.098
71.034
44.936
29,834
LSD+-Euler
94.554
41.986
32.832
27.283
LSD+-Tweeke
95.029
45.823
37.900
26.883
F PCA analysis of coefficients
To analyze the the training evolution of learned sampler coefficients { we performed Principal Component Analysis (PCA) (66 667 on A -dimensional coefficient vectors collected at each epoch. F visualizes projections onto the top principal components. The trajectories of learned coefficients show substantial divergence from the vanilla baseline point, empirically demonstrating that optimization yields configurations distinct from fixed scaling. This dynamic learning process, while sensitive to initialization across runs, explores configurations enabling high- fidelity sampling at low NFEs, and the difference between different training is relatively small.
Learned Conficients During Training
5.05
gue't
PC3.0..00
-300
-0.35
Wrib Fanelne (uter
-013
Figure 3: PCA analysis of learned coefficients at each epoch of training.
G Text generation results
In this section, we provide text generation results from L.SD+. All results are generated using the Euler sampler on the SEDD-medium backbone.
28

---

swagqell on Twitter and for reading! Heath and Hello, that's this information in Continue reading. This's a good idea, I've going to have some ask you who. On asked you to come in powerful is, with for then, here were about some of the reasons for that. By any time in Baltimore, in Camden, or Washington, I'll sure you if some sources would and, well, Ld be in was in Soltas, of Philadelphia (Washington and elsewhere) ahout the game that's process and his way on the foothall team. For instance, cornerback Scott Snyder on Washington called "", the the 49ers' 2012 coach (if "" of those season said more byand (that). I'm curious you would ""-to have a pragmatic thing that actually assures a "" the of the eleccioon.rticll level activists and that it's not, to to the system of American columns, to this qualified follows in If I've the Before the election for the days where the didn't ask, and the Republican candidates to lead wall-withhou clear direction and without first getting a record in light of the constituents' from which I've taken my time to think about such thing and I'd not be too unapologetic about spending few days (Don't know, the rise of the headed system, I's in curious whether he wasn't 't in. Here thought was in response case. This is the same is of one's attempt to so funny enough, long as it remains without a". The 55 lot the plans, so more of an in in-house plan than irmative", a plan to interest increase and and saying, that should function as thought. As for whether Obama won. he's currently third in the, in Texas, nominee. I wrote about the new bill in an article-ed. Even if the "If" you believe in him" is. you be the the, which terms of the game being can self-destruct of Romney's ss spokesperson, Mitt Romney. was you who did it. case, it was passed last which was the new health insurance law. The law was Obamacare, the American Health Care Act. think was about to pre the Obamacar
in favor of repealing it. In
Figure 4: The generated text for NFE=8.
and that the federal programs have heen pay out for additional much. was agreed at the summit that there will be increases in the mean time and that they will need to look back and make full pay. have got stop before you get started," Boehner said. "And not a few of us are going to have budget in the some, because what we have't started looking at that's just beginning, We need to pay for all those programs - our normal budget plus we will make work toward the end the year."In order to be trusted in conjunction with the austerity of the entitlement will begin almost immediately, We the nation President Obama will have said it and the Republicans, for that matter, won't have said i- to have to face. Obama has done everything in his power not to acknowledge this possibility. There maintains a veneer of honestness his administration, but else he displayed almost entirely to blame on the cliff.On a united loan from the House Party, Obama has paid waid-Both Both parties in ownership the national dpec.Specificall:: the companies that profit from, and through whose contracts, the money government owes, and is owed more. has given a flying strip limit's. It also has debt small companies, the United States on this earth.In America, the government ent has had enormous responsibility to America. can tell Washington DC really has done the least for America over the world.Perhaps. it Washington, it will he able to assist and alter the government whether or not they have created something to the taxpayer the financial representation of this debt feels byllcacallly self sufficient, But their future goes far beyond the effects of this austerity many ways. For one, of the debt will create an opportunity for a new generation to out of the burden of the debt. a de-biz for all tax plan is publicly Manoounced.Beyonn that is however, self- gelf-governance will take, and the consequences mean be- and is obvious. from here to America, as wrote here, recently, debt
Figure 5: The generated text for NFE=16.
29

---

it's just me, it's bad feelings. He tells aware of sadistic behavior that is a joy to watch as a missionary and me when see some right-handed people validating Mormonism to fall in love, and some I know are not, as tell him. I teach me the left- hands, so does he. I love him for having his son. "He's often running riot," he says." "Like a kid bit in a get-out-of-wap mode." helpp him study a lot about that and I teach him about Ahem.As missionary I'm all out in search of sacred books, literally and figuratively, Every. I help him learn and learn of in the way we prepare for mission. Again, these are complements of his ministry. "Man is man and she is fairly simple too, because can see the power of that saying, when I'm all over him to evangel really struggle to deal with But, feel grounded in my own life. Though may know what to read, I study scriptures, read sacred books and now when your eyes see this as clearly as I do, that counts as difference. that you feel he lonely. It, beautiful and is that, when you never, you realize he is brokenhearted. But from the best his, it's just beautiful reflection of desire to experience in his daily life and the desires and needs those condemns and tries to project them onto young
people.Either way, welcome Ht. a human being and-I him just as well as can. I will not deny his character as kind, humble, and hospitable man who loves us in many different ways, tender-hearted and will be some the greatest on the feet of the eerth. loves us all, but as discerns more and more about people.That said, I'll love him a part of his missionary call. He knows shouldn't change my mind and when he has come to be the Savior, every response creates rays of change every day in the work of God, through his mission "correct the whole world." The better he can, the more he can come to understand Sometime, he says that he casts shadow for th
my heart.He loves about people
Figure 6: The generated text for NFE=32.
we're going to let the players know that some new players have a responsibility to the fans. you have idea of it like this, how to the teams with the fans?No, before that, went into this season, we were at the moment we were heading for the playoffs in this game.The The only difference is that, at the beginning, we had seven to go. in that moment, you put on the match with the players that popped up from the field with the players only, which is hard to do, because there are so many to deal with that we had to mess around with. And there, makes it over to the training room "the training I think everyone on the team is the training room.In the training room, the players come out here, the players come to the standing on the ground, the pass the ball.When the head the player passes, the player has to push-up their head on the ground, if from the pass. thenall the players, who are going to pass when the touch the ball, have to push- up their heads on this when they receive the ball before they receive the The organization knows that the match is not won not just by the fans' support, so that first half the fans' fans decided to become fans in the training room, where the players can help provide the imput by looking at the defender's tips.The defense going to score the goal's at the match's end. That will be based on the first half. This when we know it's going to be difficult for them. The team's title going people.Q: That's the talk that you gave the players this past week for in Tuesday. number of time they didn't really set up routines for the yes..Yes. They didn't know. mean: I don't know, in the team group, just think that they all the whatever it is, they just got prepare themselves for the easiest any time, and when play the game for week, then we'll go to something to the rest.For the players, to have the preparation routine, the ones to
for the people. That's the title, for the
the ones
Figure 7: The generated text for NFE=64.
30


Reviews:
[{"review_id": "zhACszncoY", "reviewer": "Reviewer_9x6X", "summary": "This paper proposed learnable sampler distillation (LSD), which is a novel approach to train fast and high-fidelity samplers for discrete diffusion models. The training method is based on the teacher-student model, where the student sampler is trained via distillation by optimizing learnable sampler coefficients with respect to a fixed teacher sampler of high fidelity. An extension of LSD, which is named as LSD+, is also proposed to learn non-uniform time schedules in an adaptive way. Numerical experiments on the generation of synthetic data, text and images are provided to justify the effectiveness of the proposed method.", "questions": "For the setting of distilling continuous diffusion models/consistency models, a few papers like [11, 12] have provided theoretical justifications. The reviewer is wondering whether it would be possible to provide theoretical justification for the distillation method proposed in this paper by referring to exist work on the theoretical analysis of discrete diffusion models like [8,13,14,15]?\n\nMoreover, for Algorithm 3 in Appendix D.2, it seems that some ambiguity exists. Specifically, as the time schedules $\\tau_{k}$'s depend on the stepsizes $\\kappa_{k}$'s, would it be possible for the authors to clarify whether the stepsizes $\\kappa_{k}$'s will be updated accordingly after the $\\tau_{k}$'s get updated in line 10 of Algorithm 3? (It seems that it doesn't make sense to fix all stepsizes $\\kappa_{k}$'s)\n\nFurthermore, for the comparison between LSD/LSD+ and other baseline methods, the reviewer is wondering whether it would be fair to fix the number of evaluations (NFEs) and compare the performance since the distillation step take nontrivial amount of time. Given that LSD and LSD+ both accelerate the inference via training, maybe the authors need to fix the running time and compare LSD/LSD+ with other methods like [8]?", "limitations": "The reviewer is satisfied with the limitations that the authors briefly discussed in Appendix A of the manuscript. For instance, one limitation that the authors mentioned is that the performance ceiling of the student sampler is highly related to the quality of the teacher samplers.\n\nReferences:\n\n[1] Song, Y., Dhariwal, P., Chen, M., & Sutskever, I. (2023). Consistency models. International Conference on Machine Learning (ICML), PMLR, 2023.\n\n[2] Boffi, N. M., Albergo, M. S., & Vanden-Eijnden, E. (2025). How to build a consistency model: Learning flow maps via self-distillation. arXiv preprint arXiv:2505.18825.\n\n[3] Heek, J., Hoogeboom, E., & Salimans, T. (2024). Multistep consistency models. arXiv preprint arXiv:2403.06807.\n\n[4] Meng, C., Rombach, R., Gao, R., Kingma, D., Ermon, S., Ho, J., & Salimans, T. (2023). On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 14297-14306).\n\n[5] Zhou, Z., Chen, D., Wang, C., Chen, C., & Lyu, S. (2024). Simple and fast distillation of diffusion models. Advances in Neural Information Processing Systems (NeurIPS), 37, 40831-40860.\n\n[6] Salimans, T., & Ho, J. (2022). In The Tenth International Conference on Learning Representations (ICLR), 2022.\n\n[7] Xie, S., Xiao, Z., Kingma, D., Hou, T., Wu, Y. N., Murphy, K. P., ... & Gao, R. (2024). Em distillation for one-step diffusion models. Advances in Neural Information Processing Systems (NeurIPS), 37, 45073-45104.\n\n[8] Ren, Y., Chen, H., Zhu, Y., Guo, W., Chen, Y., Rotskoff, G. M., ... & Ying, L. (2025). Fast solvers for discrete diffusion models: Theory and applications of high-order algorithms. arXiv preprint arXiv:2502.00234.\n\n[9] Karras, T., Laine, S., & Aila, T. (2019). A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 4401-4410).\n\n[10] Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., & Fei-Fei, L. (2009, June). Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 248-255). \n\n[11] Jain, N., Huang, X., Ma, Y., & Zhang, T. (2025). Multi-Step Consistency Models: Fast Generation with Theoretical Guarantees. arXiv preprint arXiv:2505.01049.\n\n[12] Chen, Y., Zhang, Y., Oertell, O., & Sun, W. (2025). Convergence of consistency model with multistep sampling under general data assumptions. arXiv preprint arXiv:2505.03194.\n\n[13] Ren, Y., Chen, H., Rotskoff, G. M., and Ying, L. How discrete and continuous diffusion meet: Comprehensive analysis of discrete diffusion models via a stochastic integral framework. In The Thirteenth International Conference on Learning Representations (ICLR), 2025\n\n[14] Zhang, Z., Chen, Z., & Gu, Q. (2024). Convergence of score-based discrete diffusion models: A discrete-time analysis. In The Thirteenth International Conference on Learning Representations (ICLR), 2025\n\n[15] Liang, Y., Huang, R., Lai, L., Shroff, N., & Liang, Y. (2025). Absorb and Converge: Provable Convergence Guarantee for Absorbing Discrete Diffusion Models. arXiv preprint arXiv:2506.02318.\n\n[16] Zhu, Y., Wang, X., Lathuilière, S., & Kalogeiton, V. (2025). Di $\\mathtt {[M]} $ O: Distilling Masked Diffusion Models into One-step Generator. arXiv preprint arXiv:2503.15457.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Pros: The reviewer finds that the manuscript is presented in a clear and straightforward way. Also, this paper studies an important question, since how to accelerate the inference time is currently one of the bottlenecks of discrete diffusion models. Experiments on multiple generation tasks are provided to justify how effective the proposed methodology is. In addition, a wide range of baselines and implementation details is also provided. \n\nCons:  The reviewer's major concern is that the literature review part of the manuscript seems to be incomplete. To the best of the reviewer's knowledge, lots of paper have already studied the idea of combining knowledge distillation with continuous diffusion models or its variants like latent diffusion models and consistency models [1] (In fact, the main idea behind consistency model is also similar to that of distillation, as both methodologies essentially use training techniques to speed up the inference process). Examples of work that probably should be cited and briefly discussed here include but not limited to [2,3,4,5,6,7,16]. More essentially, the authors probably also need to compare the distillation procedure proposed in this work with that of [16] (Or maybe consider including [16] as one of the baselines).  \n\nAlso, compared to existing work on accelerating the inference of discrete diffusion models like [8], the scale of the numerical experiment on image generation included in this manuscript might be a bit small. Specifically, the authors are encouraged to test the proposed distillation method on large-scale images like FFHQ [9] or ImageNet [10] to better justify its validity.", "was_revised": true, "final_justification": "After reviewing the authors' rebuttal, the reviewer is satisfied with nearly all of their responses. However, currently no theoretical analysis or experiments on large-scale images are included in the paper. Also, the idea of this paper partially overlaps with that of Di[M]O [16] (Both are based on distillation, even though detailed methodologies are different), which undermines the novelty of the manuscript. Therefore, the reviewer is inclined to maintain the original score of 3 (\"Borderline Reject\") for this paper.", "rebuttal": "Thanks for your useful comments and questions. Our responses to the main concerns are given as follows. All citations refer to the reference list provided by the reviewer. \n \n(**Examples of work that probably should be cited and briefly discussed here include but not limited to [2,3,4,5,6,7,16].** )\n\nThanks for the comment. We will add a paragraph discussing the mentioned papers in the revised manuscript, and we can consider citing others if the reviewer has additional suggestions. The paragraph to be added is as follows:\n\nThe distillation of continuous diffusion models is a rapidly advancing field. A prominent direction is related to the consistency model [1], which aims to learn a function that maps any point on an ODE trajectory to its origin, enabling one-step or few-step generation. This paradigm has been extended to multi-step variants [2, 3] for improved performance. Other significant works focus on directly matching student and teacher distributions, such as distilling guided diffusion models [4], proposing simplified and faster matching objectives [5], recursively distilling a deterministic diffusion sampler into a new model [6], or concentrating on one-step distillation [7]. While these methods are highly effective for continuous models, they usually rely on continuous paths in the sense that the sampling process of each step is differentiable. Our work diverges by proposing a distillation framework specifically for the discrete diffusion model, which does not assume such a continuous path, and addressing a different set of challenges like the non-differentiability of the outputs. A recent work for Di[M]O [16] also involves distilling discrete diffusion models. It distills a multi-step masked diffusion model into a one-step generator. This is achieved by training a new student model from scratch, using a sophisticated proxy objective that involves creating \"pseudo-intermediate states\" and training an auxiliary model to match conditional output distributions. Our approaches are significantly different with Di[M]O in both goal and mechanism. Similarly to LD3 and S4S, we focus on a few-step sampler distillation. We tackle the challenge of non-differentiability of sampling from categorical distributions, and we enhance an existing sampler rather than replacing the model, which avoids the complexity of training a new generator and an auxiliary model.\n\n(**The authors probably also need to compare the distillation procedure proposed in this work with that of [16].**)\n\nThank you for highlighting this work. As mentioned above, we will add a detailed comparison with Di[M]O [16] in the Related Work Section of our paper to clearly position our contribution.\n\nWhile both Di[M]O and our LSD+ method employ distillation for discrete diffusion models, a direct empirical comparison of the outcomes is not straightforward due to several significant differences between the two methods. Di[M]O trains a new student model from scratch to achieve one-step generation, primarily for image generation tasks. In contrast, our method is a few-step sampler distillation process that trains scalar-form coefficients and time steps for an existing sampler, avoiding the need to train a new model, and is evaluated mainly on text generation tasks. Given these differences in objective (one-step vs. few-step), mechanism (model training vs. sampler training), and application domain (image vs. text), currently, we do not include a comparison with Di[M]O. \n\nAdditionally, we note that the work for Di[M]O is publicly available on arXiv on March 19th, 2025. In this context, we would like to politely draw attention to NeurIPS policy, which states: “Papers appearing online after March 1st, 2025 are generally considered concurrent to NeurIPS submissions.  Authors are not expected to compare to those”. (https://neurips.cc/Conferences/2025/PaperInformation/NeurIPS-FAQ)\n\n(**The authors are encouraged to test the proposed distillation method on large-scale images like FFHQ [9] or ImageNet [10] to better justify its validity.**)\n \nThank you for your suggestion. We conduct experiments on ImageNet (256x256) using the  MaskGIT architecture as the backbone, incorporating the recently proposed advanced Halton sampler (https://arxiv.org/abs/2503.17076) as the sampling method. The results are reported in Table A1, which demonstrates that our LSD+ method yields improved generation performance as measured by the FID metric.\n \n**Table A1: Comparisons on ImageNet 256x256 (in terms of FID$\\downarrow$).**\n|Sampler\\NFE|4|8|16|32|\n|:---|:--:|:--:|:--:|:--:|\n|Halton|14.16|10.15|8.89|6.92|\n|LSD+-Halton|12.78|8.66|7.17|6.32|\n\n(**Whether it would be possible to provide theoretical justification for the distillation method proposed in this paper by referring to exist work on the theoretical analysis of discrete diffusion models like [8,13,14,15]?**)\n \nThank you for this question. We agree that providing theoretical justification for our distillation methods would strengthen the work. However, our current focus is primarily empirical: We aim to develop practical and effective techniques for accelerating discrete diffusion models. Given the time constraints of the short rebuttal period, we are unable to establish rigorous theoretical guarantees for our distillation methods at this stage, and we plan to address this with more thorough theoretical analysis in future work.\n\nThat said, we would like to offer an intuitive explanation for the effectiveness of our method, which may help lay the groundwork for subsequent theoretical framing. Specifically, the final discrepancy between the outputs of the student and teacher samplers stems from the accumulation of small local errors made at each step. Each of these local errors, in turn, is tied to the specific score predicted by the model at that step. Our training objective directly enforces alignment between the score predictions of the student sampler and those of the teacher sampler at every step. By implicitly correcting these small local errors throughout the process, our approach guides the student sampler to produce final outputs that closely match the high-quality results of the teacher sampler.\n\n(**For Algorithm 3 in Appendix D.2, it seems that some ambiguity exists.**)\n \nThank you for your careful reading and for bringing this to our attention. We will revise Algorithm 3 to correct the typos therein. The parameters $\\tau_k$'s will be updated accordingly after the $\\kappa_k$'s get updated. The update rules for $\\kappa_k$'s and $\\tau_k$'s are as follows:\n$$\\kappa\\_k \\gets \\kappa\\_k - \\eta \\nabla\\_{\\kappa\\_k} \\tilde{L}\\_k$$\n$$\\tau\\_k = t\\_0 - \\sum_{l=1}^{k} \\kappa\\_l$$\n \n(**Whether it would be fair to fix the number of evaluations (NFEs) and compare the performance since the distillation step take nontrivial amount of time?**)\n \nThank you for the comment and question. Our primary focus is on accelerating the inference stage, where a one-time and offline training cost is traded for significant, repeated latency gains. The training-stage cost is relatively low. For example, under the experimental setup detailed in the Experiment Section of our main paper, distilling a 1024-step teacher sampler to a 16-step student sampler using the SEDD-small backbone takes approximately 4 minutes on a single NVIDIA RTX4090 GPU. By contrast, inference time is substantially higher. Specifically, with a batch size of 1, evaluating perplexity via the generation of 1024 unconditional text samples requires around 300 minutes of inference time on the same GPU. Given that the one-time, offline training cost is orders of magnitude smaller than even a single moderately sized evaluation, we omit training time for our distillation methods and instead focus on reporting performance metrics with fixed NFEs.\n\nThis paradigm is not unique to our work. For example, the closely related acceleration method JYS also compares performance under fixed NFEs, despite involving a one-time training cost to learn time schedules.\n\nRegarding high-order solvers, we have included direct comparisons against the $\\theta$-RK2 and $\\theta$-Trapezoidal methods from [8] in Table 3 of the main paper. However, since the code for [8] is not yet open-sourced, a direct comparison of running times is currently not feasible."}, {"review_id": "iuJUcCzRyj", "reviewer": "Reviewer_61AF", "summary": "This work focuses on improving sampling efficiency of discrete diffusion models by optimizing when to query diffusion model i.e., noise level and coefficiencts of solvers, an interesting direction not explored in case of discrete diffusion models.  They demonstrate good performance  in case of text generation with discrete diffusion models at lower NFEs.", "questions": "Following Weakness\n\nIt would be informative for readers and broader community to see integration of LSD with\n- Effect of Temperature/Confidence\n- Multi-token unmasking heuristics like in FastDLLMs, etc.\n- How well LSD complement and effective speedup for distilled discrete diffusion models e.g., SDTT Checkpoints\n- ReMasking\n- Predictor-Corrector Solvers\n\n\nAlso additional metrics like use state-of-the-art language models (LLaMA-3 or alternatives) to evaluate generated text quality independently of the diffusion model.\n\nLooking forward to rebuttal.\n\n\n**References** \n\n1. G. Wang et al. Remasking Discrete Diffusion Models with Inference-Time Scaling (https://arxiv.org/abs/2503.00307)\n2. J. Deschenaux et al. Beyond Autoregression: Fast LLMs via Self-Distillation Through Time (https://arxiv.org/html/2410.21035v1)\n3. C. Wu et al. Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding (https://arxiv.org/html/2505.22618v1)", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n* Optimizing coefficients, solver properties in addition to where to query diffusion/flow model is interesting in context of discrete diffusion models.\n* The paper demonstrates meaningful reductions in NFE while maintaining generation quality, addressing a key limitation of discrete diffusion models.\n* The paper is well-written and accessible to readers.\n\n\n**Weakness**\nLimited Scope of Experimental Evaluation\n\nThe paper lacks comprehensive analysis of how different solver configurations perform under varying conditions, limiting understanding of when and where LSD can be effective and relative peroformance boost. \nAs paper lacks any significant analysis or theoretical formulation too and authors point out is similar to S4S but applied to discrete setting. \n*Lack of extensive empericial analysis and settings* limit potential insights and adoption guidelines for readers and broader community.\n\n- Currently this work does not compare against several important recent methods and settings which focus on quality vs latency of discrete diffusion models.\n- Currenlty, authors do not consider predictor-corrector solvers (like in DFM) nor higher order solvers like Runge Kutta or Trapezoidal though they cite relevant work.\n- How effective LSD is with other heuristics in multi-token sampling like for e.g., in FastdLLMs and also setting with Remasking during sampling which shows enhanced generation quality.\n- Absence of semantic quality metrics: No evaluation using ModernBERT-based MAUVE scores or similar metrics that assess semantic coherence and diversity (entropy)\n- Minimal Performance gain in case of Image Generation (but didn't consider this much for rating)", "was_revised": true, "final_justification": "The rebuttal key concerns and demonstrates broader practical potential of proposed approach.", "rebuttal": "Thanks for your comprehensive comments and questions. Our responses to the main concerns are given as follows. All citations refer to the reference list at the end of the responses.\n\n(**This work does not compare against several important recent methods and settings which focus on quality vs latency of discrete diffusion models.**)\n \nWe thank the reviewer for the comment. We have included comparisons with several recent methods that focus on the quality vs latency of discrete diffusion models, such as JYS and $\\theta$-trapezoidal (see Tables 2 and 3 in the main paper). \nAt this stage, we would like to clarify that we are uncertain about the specific works the reviewer is referring to. Additionally, we would like to politely draw attention to NeurIPS policy, which states:“Papers appearing online after March 1st, 2025 are generally considered concurrent to NeurIPS submissions.  Authors are not expected to compare to those” (https://neurips.cc/Conferences/2025/PaperInformation/NeurIPS-FAQ). \nThat said, we are fully committed to strengthening the empirical rigor of our study. If the reviewer could kindly specify the relevant works or settings they have in mind, we would be happy to conduct additional comparisons regarding the quality vs latency of discrete diffusion models and incorporate them into our revised submission.\n \n(**Authors do not consider predictor-corrector solvers like in DFM nor higher order solvers like Runge Kutta or Trapezoidal though they cite relevant work.**)\n\nThank you for your comment. We conduct experiments on predictor-corrector (PC) solvers in Discrete Flow Matching (DFM) [1] using MDLM [2] as the backbone, and find that the improvement is limited. This might be attributed to the specific dynamics of the flow-matching process in DFM, whose error characteristics may not align as well with our error correction mechanism. We report the corresponding results in Table B1. However, we successfully achieve good results on $\\tau$LDR-10 which also utilizes PC solvers [3] on the SEDD-small backbone, with the results presented in Table B2.\n\n**Table B1: Comparisons on the MDLM backbone.**\n|Sampler||Perplexity($\\downarrow$)||||Entropy($\\uparrow$)|||\n|:---|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n|NFE|16|32|64|128|16|32|64|128|\n|DFM|312.83|314.01|100.53|37.90|5.38|5.31|5.33|5.31|\n|LSD+-DFM|310.71|316.72|90.15|35.48|5.34|5.25|5.27|5.34|\n\n**Table B2: Comparisons on the SEDD-small backbone.**\n|Sampler||Perplexity($\\downarrow$)||||Entropy($\\uparrow$)|||\n|:---|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n|NFE|16|32|64|128|16|32|64|128|\n|$\\tau$LDR-10|443.17|318.44|277.16|199.51|5.63|5.69|5.57|5.24|\n|LSD+-$\\tau$LDR-10|205.42|143.58|114.93|90.43|5.58|5.49|5.59|5.44|\n\nRegarding higher-order solvers like Runge-Kutta or Trapezoidal methods, we would like to politely clarify that we have already included a comparison in Table 3 of our main paper. We directly compare our results on the RADD backbone against the numbers reported in their work. While we are unable to directly integrate LSD+ on top of their methods due to the lack of a public implementation, this comparison demonstrates the competitive performance of our approach.\n \n(**Absence of semantic quality metrics: MAUVE scores and entropy**)\n \nThanks for your suggestion. To provide a more thorough assessment of generation quality beyond perplexity, we conduct experiments on the SDTT-KLD backbone, which utilizes Ancestral as the sampler.  We report MAUVE, Perplexity and Entropy scores in Table B3, which demonstrates that our LSD+ method usually yields improved generation performance.\n \n**Table B3: Comparisons on the SDTT-KLD backbone.**\n|Sampler||MAUVE($\\uparrow$)|||Perplexity($\\downarrow$)|||Entropy($\\uparrow$)||\n|:---|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n|NFE|8|16|32|8|16|32|8|16|32|\n|Ancestral|0.884|0.912|0.943|110.391|56.652|42.128|5.331|5.285|5.222|\n|LSD+-Ancestral|0.905|0.928|0.951|68.130|36.577|31.597|5.298|5.239|5.226|\n \n(**Minimal Performance gain in case of Image Generation**)\n\nThank you for the comment. To better demonstrate the effectiveness and scalability of our method, we conduct experiments on ImageNet (256x256) using the MaskGIT [4] architecture as the backbone, incorporating the recently proposed advanced Halton [5] sampler as the sampling method. The results are reported in Table B4, which demonstrates that our LSD+ method yields improved generation performance as measured by the FID metric.\n\n**Table B4: Comparisons on ImageNet 256x256 (in terms of FID$\\downarrow$).**\n|Sampler\\NFE|4|8|16|32|\n|:---|:--:|:--:|:--:|:--:|\n|Halton|14.16|10.15|8.89|6.92|\n|LSD+-Halton|12.78|8.66|7.17|6.32|\n\n\n(**Integration of LSD with Effect of Temperature/Confidence**)\n \nThank you for this question. Due to the time limit of the rebuttal period, we study the effect of softmax temperature in MaskGIT on ImageNet256x256 for the image generation task. The evaluations are produced on 50000 generated images using the Halton sampler. The reported FIDs are given in Table B5, which indicates that our method could lead to improved performance under the tested temperature conditions.\n \n**Table B5: Comparisons on ImageNet 256x256 (in terms of FID$\\downarrow$ when NFE=4).** \n|Sampler\\Temperature | $\\tau$ = 0.6| $\\tau$ = 0.8 | $\\tau$ = 1.0 | \n| :--- | :---: | :---: | :---: | \n|Halton|54.05| 26.45|14.16|\n |LSD+- Halton|48.29 |24.51|12.78|\n\n(**Integration of LSD with Multi-token unmasking heuristics like in FastDLLMs.**)\n \nThank you for this suggestion. This is a promising direction as the two approaches are highly complementary. LSD enhances the quality of score predictions in few-step sampling, which can directly empower the confidence-based parallel decoding of Fast-dLLM. We report the accuracy and throughput metrics in Table B6, which indicate our methods improve both sampling quality and speed.\n \n**Table B6: Integration with Fast-dLLM on LLaDA. We report accuracy (throughput, token/s). Higher is better.**\n|Benchmark\\ Samplers|LLaDA|+Cache|+Parallel|Fast-dLLM|LSD+-Fast-dLLM|\n|:---|:--:|:--:|:--:|:--:|:--:|\n|GSM8K(5-shot)|79.3(6.7)|79.5(21.2)|79.2(16.5)|78.5(54.4)|79.0(62.5)|\n|MATH(4-shot)|33.5(9.1)|33.3(23.7)|33.4(24.8)|33.2(51.7)|33.4(58.1)|\n \n(**Integration of LSD with distilled discrete diffusion models e.g., SDTT Checkpoints.**)\n \nFollowing your suggestion, to show the complementarity of model distillation (SDTT) and our sampler distillation (LSD+), we apply LSD+ to an SDTT-KLD checkpoint, with the results presented in Table B3 above, which demonstrates that our LSD+ method generally yields improved generation performance.\n \n(**Integration of LSD with Remasking**)\n \nFollowing your suggestion, we also integrate LSD with ReMasking (ReMDM) on the MDLM backbone in Table B7. The results show that LSD effectively learns to work in conjunction with this method, further improving performance.\n\n**Table B7: Comparisons on the MDLM backbone.**\n|Sampler||Perplexity($\\downarrow$)||||Entropy($\\uparrow$)|||\n|:---|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n|NFE|16|32|64|128|16|32|64|128|\n|ReMDM|434.08|174.72|85.15|62.33|5.73|5.66|5.48|5.55|\n|LSD+-ReMDM|201.52|102.02|62.97|49.33|5.41|5.42|5.52|5.33|\n \n(**Additional metrics using LLaMA-3**)\n \nFollowing your suggestion, we re-evaluate our main results on the SEDD and RADD backbone using Llama-3-8B to evaluate the perplexity. The results are reported in Tables B8 and B9.\n \n**Table B8: Comparisons on SEDD-small backbone (in terms of Perplexity$\\downarrow$), judged by LLaMA-3-8B.**\n|Sampler\\NFE|8|16|32|64|\n|:---|:--:|:--:|:--:|:--:|\n|Euler|116.93|67.43|49.81|46.88|\n|LSD+-Euler|74.65|33.25|30.70|21.64|\n\n**Table B9: Comparisons on on RADD backbone (in terms of Perplexity$\\downarrow$), judged by LLaMA-3-8B.**\n|Sampler\\NFE|8|16|32|64|\n|:---|:--:|:--:|:--:|:--:|\n|Euler|337.04 |216.01 |119.25 |95.06|\n|LSD+-Euler|130.00 |60.90 |42.53 |35.65 |\n\nReferences:\n\n[1] Gat el al. Discrete Flow Matching, NeurIPS’24\n\n[2] Sahoo et al. Simple and Effective Masked Diffusion Language Models, ICLR’25.\n\n[3] Chang et al. A continuous time framework for discrete denoising models, NeurIPS’22\n\n[4] Chang el al. Masked Generative Image Transformer, CVPR’22.\n\n[5] Besnier et al. Halton Scheduler for Masked Generative Image Transformer, ICLR’25."}, {"review_id": "0CrEeDe3AJ", "reviewer": "Reviewer_VCFx", "summary": "This paper proposes a framework for accelerating discrete diffusion models by distilling a fast, high-fidelity sampler, named LSD.\n\nIn general, there is a trade-off between generation speed and generaion quality. \n\nLSD trains a student sampler with a small number of steps to align its intermediate score trajectories with those of a high-quality teacher sampler. By introducing learnable, time-dependent coefficients, the student dynamically adjusts the influence of scores to compensate for errors.\n\n In addition, LSD+, additionally learns non-uniform time schedules, allocating steps adaptively over time.\n\nExperiments demonstrate that LSD/LSD+ significantly improve the trade-off between sampling speed and fidelity across text generation (OpenWebText, GPT-2-based DDMs), image generation (CIFAR-10), and a synthetic sequence task.", "questions": "How sensitive are the results to the choice of the relaxed objective hyperparameter (Hamming distance threshold)? Could tuning it improve results further?", "limitations": "The authors did not discuss the limitations.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n- The paper designs a novel distillation framework for fast inference of DDMs.\n- The paper Introduces learnable per-step coefficients and time schedules, providing adaptive control over dynamics.\n- Experimental results on three diverse tasks with improvements in perplexity, FID.\n\nWeaknesses:\n- Evaluations are limited to relatively small-scale datasets (OpenWebText, CIFAR-10) and short sequences (1024 tokens)\n- The robustness of the algorithm to parameters has not been discussed.", "was_revised": true, "final_justification": "The authors have provided extra comparisons during the rebuttal, and resolved all my concerns. Therefore I maintain my recommendation for acceptance.", "rebuttal": "Thanks for your recognition of this paper and the valuable feedback and suggestions. Our responses to the main concerns are given as follows. All citations refer to the reference list at the end of the responses.\n \n(**Evaluations are limited to relatively small-scale datasets (OpenWebText, CIFAR-10) and short sequences (1024 tokens).**)\n \nWe thank the reviewer for the comment. To address this, we perform experiments on larger-scale datasets. For image generation, we conduct experiments on ImageNet (256x256) using the  MaskGIT architecture [1] as the backbone, incorporating the recently proposed advanced Halton sampler [2] as the sampling method. The results are presented in Table C1.\n\nFor text generation, we scale up our experiments to include the comparison to the sampler with respect to a larger backbone DiffuGPT-S [3]. The results are presented in Table C2, where we report perplexity on 1024 unconditionally generated text samples.\n\n**Table C1: Comparisons on ImageNet 256x256 (in terms of FID$\\downarrow$).** \n|Sampler\\NFE|4|8|16|32|\n|:---|:--:|:--:|:--:|:--:|\n|Halton|14.16|10.15|8.89|6.92|\n|LSD+-Halton|12.78|8.66|7.17|6.32|\n\n**Table C2: Comparisons on the DiffuGPT-S backbone (in terms of Perplexity$\\downarrow$).**\n|Sampler\\NFE|16|32|64|128|\n|:---|:--:|:--:|:--:|:--:|\n|DiffuGPT-S|117.32|75.19|58.34|37.16|\n|LSD+-DiffuGPT-S|53.95|41.37|32.10|22.25|\n \nWe also present results for larger-scale experiments on the SDTT-KLD backbone [4], which utilizes Ancestral as its sampler, and the MDLM backbone [5], which utilizes ReMDM [6] as the sampler. The experimental results are presented in Tables C3 and C4.\n\n**Table C3: Comparisons on the SDTT-KLD backbone.**\n|Sampler||MAUVE($\\uparrow$)|||Perplexity($\\downarrow$)|||Entropy($\\uparrow$)||\n|:---|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n|NFE|8|16|32|8|16|32|8|16|32|\n|Ancestral|0.884|0.912|0.943|110.391|56.652|42.128|5.331|5.285|5.222|\n|LSD+-Ancestral|0.905|0.934|0.961|68.130|36.577|31.597|5.298|5.239|5.226|\n\n\n**Table C4: Comparisons on the MDLM backbone.**\n|Sampler||Perplexity($\\downarrow$)||||Entropy($\\uparrow$)|||\n|:---|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n|NFE|16|32|64|128|16|32|64|128|\n|ReMDM|434.08|174.72|85.15|62.33|5.73|5.66|5.48|5.55|\n|LSD+-ReMDM|201.52|102.02|62.97|49.33|5.41|5.42|5.52|5.33|\n\nThese results show that LSD+ maintains its performance and acceleration benefits when applied to more complex datasets and larger models. We will integrate these results into the Experiments Section of our revised manuscript.\n \n(**The robustness of the algorithm to parameters (e.g., Hamming distance threshold) has not been discussed.**) \n\nThank you for the comment. To investigate the robustness of the algorithm to the Hamming distance threshold, we conduct the ablation on the SEDD-small backbone using the Euler sampler with 32 inference steps. We train our LSD+ method using several different values for the Hamming distance threshold, specifically {0%, 1%, 5%, 10%, 20%} of the sequence length, while keeping all other hyperparameters unchanged. The performance, measured by Perplexity, is reported below in Table C5. \n \n**Table C5: Ablation study on the Hamming distance threshold for the relaxed objective.**\n|Threshold(%)|Perplexity($\\downarrow$)|\n|:-:|:-:|\n|0 |35.98|\n|1|32.15|\n|5 (Our choice)|31.24|\n|10|39.97|\n|20|51.52|\n\nWe empirically choose 5% of the sequence length as the Hamming distance threshold in our main paper.\n\n(**The authors did not discuss the limitations.**) \n\nThanks for the comment. We politely clarify that as mentioned in our NeurIPS Paper Checklist, we discussed the limitations in the supplementary material (see Appendix A). \n\nReferences:\n\n[1] Chang et al. Masked Generative Image Transformer, CVPR’22.\n\n[2] Besnier et al. Halton Scheduler for Masked Generative Image Transformer, ICLR’25.\n\n[3] Gong et al. Scaling Diffusion Language Models via Adaptation from Autoregressive Models, ICLR’25.\n\n[4] Deschenaux et al. Beyond Autoregression: Fast LLMs via Self-Distillation Through Time, ICLR’25.\n\n[5] Sahoo et al. Simple and Effective Masked Diffusion Language Models, ICLR’25.\n\n[6] Wang et al. Remasking Discrete Diffusion Models with Inference-Time Scaling, ICLR’25."}, {"review_id": "XNGQtcE4B9", "reviewer": "Reviewer_ACPU", "summary": "This paper proposes a learnable sampler for discrete diffusion models (DDMs) that can achieve performance comparable to a multi-step teacher model through distillation. The proposed sampler is optimized using a novel, differentiable loss based on the discrepancy between the intermediate score trajectories of the student and a high-quality teacher. Extensive experiments demonstrate the effectiveness of the method in accelerating DDM sampling.", "questions": "1.  The experiments are conducted on relatively small-scale models (GPT-2 level, CIFAR-10). For larger models, e.g., DiffuLLaMA and DiffuGPT, how well can the proposed method perform? Is LSD or LSD+ compatible with those larger-scale discrete diffusion models?\n2.  The training objective aligns scores at each step independently, even though the student and teacher states are on different trajectories. Could the authors provide more intuition or analysis on why this myopic alignment is sufficient to guide the student sampler effectively over the full generation process?\n3.  Could the authors comment on the choice of a scalar coefficient for the sampler? Have more expressive alternatives (e.g., vector-based coefficients) been explored, and if so, how do they compare in performance and efficiency?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n1. This work addresses a critical and well-known problem in discrete diffusion models: their slow and computationally expensive sampling process. An effective solution to this problem would significantly broaden the practical applicability of DDMs.\n2. The proposed score trajectory loss is intuitive and provides a clever solution to the non-differentiability problem in discrete sampling pipelines. Additionally, the learnable sampler is lightweight with only a few parameters, making the distillation process highly efficient. The empirical results are strong and consistent across multiple tasks and model backbones.\n\n**Weaknesses:**\n1. **Lack of justification for the training objective:** It is not entirely clear why the training objective is effective in practice. The loss only aligns the scores at each step k independently, but fails to account for the dependency of a state on the full history of previous states and sampler coefficients. The student's state $x_{t_k}$ and the teacher's state $x*_{t_k}$ are generated from different trajectories, so simply aligning their scores at that instant does not guarantee a better final sample. Could the authors provide more insights into this phenomenon?\n\n2. **Underexplored design choices:** The form of the learnable sampler coefficients needs further justification. The current sampler introduces scalar-form multiplicative coefficients (`Φ(t_k)`) to modify the student's score. This assumes a single scalar is sufficient to correct for the complex, high-dimensional accumulated error. A pending ablation study comparing this design to more expressive forms, such as vector-form element-wise multiplicative coefficients or other parametric affine transformations, would be valuable.\n\n3. **Lacking theoretical grounding:** The trajectory loss appears to be more of a well-motivated heuristic rather than a principled solution derived from a solid mathematical foundation. While the method is empirically successful, a theoretical analysis (e.g., analyzing the distributional divergence between the optimal student sampler and the teacher sampler) would foster a deeper understanding of the proposed method and its limitations.", "was_revised": true, "final_justification": "I maintain my original recommendation of weak acceptance. While empirical results show that the proposed learnable sampler approach is more effective than baseline samplers, it is still unclear if this method can hold its relevance to discrete diffusion model sampling, especially when theoretical grounding is missing and generation diversity is not guaranteed.", "rebuttal": "Thanks for your positive assessment of this paper and the valuable comments and suggestions. Our responses to the main concerns are given as follows.\n \n(**Lack of justification for the training objective.**)\n\nThank you for your comment and question. In our view, the training objective that aligns scores at corresponding time steps functions as a mechanism for continuous error correction, addressing two main sources of error. First, by aligning scores at each step, we mitigate the accumulation of discretization errors that arise from large step sizes. Second, this approach helps alleviate the training-sampling mismatch, a common issue that leads to compounding decoding errors. In our method, the teacher sampler that operates with a large number of sampling steps effectively emulates the dynamics of the training process. Conversely, the student sampler, which uses far fewer steps, is designed to mimic the inference process. By enforcing that the scores of the student sampler match the scores of the teacher sampler at each corresponding time step, we partially bridge the gap between training and sampling phases. Additionally, as noted in the Related Work Section of our main paper, aligning scores across the full trajectory is computationally intractable due to the non-differentiable nature of sampling from categorical distributions. Thus, we adopt per-step alignment as a practical alternative to circumvent this non-differentiability, and empirical results demonstrate its effectiveness in enhancing sample quality at low NFEs. \n \n(**Underexplored design choices.**)\n \nThank you for raising this point about our design choice for the learnable coefficients.\nOur decision to follow prior works like S4S and LD3 in adopting a scalar form for the coefficient is a trade-off between expressiveness and computational feasibility.\nWhile a more expressive vector-form coefficient is theoretically feasible, directly modulating transition probabilities for each token would require the vector to match the vocabulary size (e.g., 50258 for GPT-2). This would result in a large number of learnable parameters, calculated as the product of the number of sampling steps and the vocabulary size. Such a surge would significantly increase memory usage and heighten the risk of overfitting during distillation. Thus, we opted for the scalar formulation, which offers simplicity, greater computational efficiency, and empirical effectiveness across our experiments.\n \n(**Lacking theoretical grounding.**)\n \nThank you for this question. We agree that analyzing the distributional divergence between the optimal student sampler and the teacher sampler is beneficial for fostering a deeper understanding of the proposed method and its limitations. However, our current focus is primarily empirical: We aim to develop practical and effective techniques for accelerating discrete diffusion models. Given the time constraints of the short rebuttal period, we are unable to establish rigorous theoretical guarantees for our distillation methods at this stage, and we plan to address this with more thorough theoretical analysis in future work.\nThat said, we would like to offer an intuitive explanation for the effectiveness of our method, which may help lay the groundwork for subsequent theoretical framing. Specifically, the final discrepancy between the outputs of the student and teacher samplers stems from the accumulation of small local errors made at each step. Each of these local errors, in turn, is tied to the specific score predicted by the model at that step. Our training objective directly enforces alignment between the score predictions of the student sampler and those of the teacher sampler at every step. By implicitly correcting these small local errors throughout the process, our approach guides the student sampler to produce final outputs that closely match the high-quality results of the teacher sampler.\n \n(**For larger models, e.g., DiffuLLaMA and DiffuGPT, how well can the proposed method perform?**)\n \nThank you for your questions regarding the scalability of our proposed methods. To demonstrate that our framework is not limited to smaller models, we conduct experiments on applying LSD+ to DiffuLLaMA and DiffuGPT. We perform sampler distillation on pre-trained DiffuGPT-S and DiffuLLaMA checkpoints. The results in terms of Perplexity are presented in Tables D1 and D2.\n \n**Table D1: Comparisons on the DiffuGPT-S backbone (in terms of Perplexity$\\downarrow$).**\n|Method\\NFE|16|32|64|128|\n|:---|:--:|:--:|:--:|:--:|\n|DiffuGPT-S|117.32|75.19|58.34|37.16|\n|LSD+-DiffuGPT-S|53.95|41.37|32.10|22.25|\n\n**Table D2: Comparisons on the DiffuLLaMA backbone (in terms of Perplexity$\\downarrow$).**\n|Method\\NFE|16|32|64|128|\n|:---|:--:|:--:|:--:|:--:|\n|DiffuLLaMA|100.04|69.11|42.17|30.55|\n|LSD+-LLaMA|49.83|34.32|29.18|24.72|\n \nThis experiment confirms that our approach is compatible with DiffuGPT and DiffuLLaMA. We will add these results to the Experiments Section of our paper."}, {"review_id": "HlTQZeLKZ1", "reviewer": "Reviewer_odGL", "summary": "This paper proposes LSD and LSD+, methods to accelerate the sampling of discrete diffusion models (DDMs). The main contributions are:\n\n* *Learnable sampling coefficients*: making the scaling factors applied to scores learnable.\n* *Learnable timesteps*: making the sampling schedule learnable.\n\nUsing these techniques, the method achieves comparable or superior performance to baselines (Euler, JYS) with fewer NFEs.\n\nSimply generating similar outputs by increasing NFEs does not enable learning solver parameters. Therefore, the paper introduces loss functions (Eq. 6 and 8) tailored for DDMs. The core idea is to make the scores at the intermediate states $x_k$ (student solver) and $x_k^*$ (teacher solver) similar.", "questions": "* As noted in the weaknesses, could you provide a theoretical guarantee or an intuitive explanation of why having similar scores at different locations is helpful?\n* What do the learned coefficients and timesteps look like? As in JYS, are they more concentrated toward $T$? Are the coefficients distributed to be greater than 1 in the early timesteps?\n* Additional baselines: it would be helpful to include comparisons with recent methods such as \\[1] and \\[2]. I would be satisfied even with results applied only to SEDD.\n\n\\[1]: Fast Sampling via Discrete Non-Markov Diffusion Models with Predetermined Transition Time ([https://arxiv.org/abs/2312.09193](https://arxiv.org/abs/2312.09193))\n\\[2]: Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithms ([https://arxiv.org/abs/2502.00234](https://arxiv.org/abs/2502.00234))", "limitations": "* The supplementary material acknowledges that the student sampler has fundamental limitations, such as an upper bound on performance.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "*Strengths*\n\n* The results significantly outperform baseline methods (Euler sampler and JYS). This is impressive.\n\n*Weaknesses*\n\n* Theoretical justification is lacking. Even if the learning of the solver is perfectly successful, is it guaranteed that the same distribution is recovered? It would be sufficient to at least show that it contributes to reducing the gap between the teacher distribution $p^*$ and the student distribution $p$.\n* The method is not intuitively easy to understand. The current objective enforces that the outputs of the student model at $x_k$ match those of the teacher model at $x_k^*$. \n* However, it is unclear why having similar scores at different points is beneficial. My intuition was that the student model should be trained to produce $x_k^*$ itself (as in SDTT \\[1]), though in that case there would be the drawback of sacrificing diversity.\n\n\\[1]: Beyond Autoregression: Fast LLMs via Self-Distillation Through Time ([https://arxiv.org/abs/2410.21035v1](https://arxiv.org/abs/2410.21035v1))", "was_revised": true, "final_justification": "I intend to raise my score for this paper to **Accept**. While the lack of a theoretical guarantee that the learnable solver approximates the teacher distribution remains a significant drawback, the empirical performance gains are noteworthy.", "rebuttal": "Thank you for your positive evaluation of our paper and for your valuable comments and suggestions. Below are our responses to the main points raised.\n \n(**Theoretical justification is lacking.** )\n\nThank you for the comment and question. We agree that the theoretical guarantee concerning the matching of teacher and student distributions is beneficial and providing theoretical justification for our distillation methods would strengthen the work. However, our current focus is primarily empirical: We aim to develop practical and effective techniques for accelerating discrete diffusion models. Given the time constraints of the short rebuttal period, we are unable to establish rigorous theoretical guarantees for our distillation methods at this stage, and we plan to address this with more thorough theoretical analysis in future work.\nThat said, we would like to offer an intuitive explanation for the effectiveness of our method, which may help lay the groundwork for subsequent theoretical framing. Specifically, the final discrepancy between the outputs of the student and teacher samplers stems from the accumulation of small local errors made at each step. Each of these local errors, in turn, is tied to the specific score predicted by the model at that step. Our training objective directly enforces alignment between the score predictions of the student sampler and those of the teacher sampler at every step. By implicitly correcting these small local errors throughout the process, our approach guides the student sampler to produce final outputs that closely match the high-quality results of the teacher sampler.\n\n(**The method is not intuitively easy to understand. It is unclear why having similar scores at different points is beneficial.**)\n\nThank you for the comment. An intuitive explanation of why having similar scores at different locations is helpful is provided as follows.\n\nIn our view, the training objective that aligns scores at corresponding time steps functions as a mechanism for continuous error correction, addressing two main sources of error. First, by aligning scores at each step, we mitigate the accumulation of discretization errors that arise from large step sizes. Second, this approach helps alleviate the training-sampling mismatch, a common issue that leads to compounding decoding errors. In our method, the teacher sampler that operates with a large number of sampling steps effectively emulates the dynamics of the training process. Conversely, the student sampler, which uses far fewer steps, is designed to mimic the inference process. By enforcing that the scores of the student sampler match the scores of the teacher sampler at each corresponding time step, we partially bridge the gap between training and sampling phases. Additionally, as noted in the Related Work Section of our main paper, aligning scores across the full trajectory is computationally intractable due to the non-differentiable nature of sampling from categorical distributions. Thus, we adopt per-step alignment as a practical alternative to circumvent this non-differentiability, and empirical results demonstrate its effectiveness in enhancing sample quality at low NFEs. \n\n \n(**What do the learned coefficients and timesteps look like?**)\n \nThank you for the question. We provide a visualization of the learned parameters for a 16-step student sampler, which is distilled from a 1024-step teacher sampler on the SEDD-small backbone.\n\nThe learned timesteps are:\n`[1.0000, 0.9219, 0.8459, 0.7719, 0.7000, 0.6302, 0.5625, 0.4969, 0.4334, 0.3719, 0.3125, 0.2552, 0.2000, 0.1469, 0.0959, 0.0469]`\nNotably, similar to JYS, the learned schedule concentrates more steps near $t=1$. However, the learned distribution of time step is not as drastic as that of JYS, since we also adjust the student sampler through coefficients. \n\nThe learned coefficients for these timesteps are:\n`[1.1313, 1.1305, 1.1289, 1.1270, 1.1245, 1.1210, 1.1178, 1.1141, 1.1102, 1.1068, 1.1031, 1.0999, 1.0962, 1.0925, 1.0881, 1.0839]`\nThe coefficients are all greater than 1. Additionally, the coefficients decrease with respect to timesteps, which indicates the sampler learns to apply a stronger corrective amplification to the score when taking larger steps at earlier stages, effectively compensating for the increased discretization error.\n\n**(Additional baselines.)**\n\nThank you for this suggestion. \n\nWe conduct experiments on the sampling method of DNDM [1] you mentioned, adhering to the original experimental settings outlined in the DNDM paper, and the perplexity results for unconditional text generation using the FairSeq backbone are reported in Table E1. As for the SEDD backbone, we have not performed comparable experiments, as no public implementation currently exists for applying DNDM to SEDD. \n\n**Table E1: Comparisons on the FairSeq backbone (in terms of Perplexity$\\downarrow$).**\n|Sampler\\NFE|8|16|32|64|\n|:---|:--:|:--:|:--:|:--:|\n|DNDM|919.23|774.92|748.41|622.14|\n|LSD+-DNDM|601.22|554.19|477.10|403.13|\n\nRegarding the specific work [2] you mentioned, we politely clarify that we included a direct performance comparison with the reported results of it in Table 3 of our main manuscript.\nWe could only compare our results with their work on the RADD backbone since the public implementation of their work is currently unavailable."}]

Author rebuttal:

Meta-review: This paper introduces a novel method, Learnable Sampler Distillation (LSD) and its extension LSD+, to accelerate the sampling process of discrete diffusion models (DDMs). The core idea is to train a lightweight "student" sampler to mimic the behavior of a high-fidelity, multi-step "teacher" sampler. The approach achieves this by optimizing a small number of learnable parameters: per-step coefficients that modify the score predictions and, for LSD+, an adaptive time schedule. The paper demonstrates that LSD/LSD+  improves the trade-off between generation speed and quality across multiple tasks, including text and image generation, and on various model architectures.

The proposed method is intuitive, and its empirical results are impressive. As noted by Reviewers 9x6X, 61AF, and VCFx, the paper shows performance gains over strong baselines, with fewer sampling steps (NFEs). The method's low computational overhead for training and its compatibility with various backbones and other acceleration techniques (e.g., remasking, parallel decoding) make it valuable to the community.

The primary weakness, highlighted by Reviewers 9x6X, ACPU, and odGL, is the lack of a strong theoretical foundation. The paper's core objective, which aligns scores at each step, is presented as an effective heuristic without a formal analysis of why this approach guarantees a better final sample distribution. A related point raised by Reviewer ACPU is the underexplored design choice of using scalar coefficients instead of more expressive forms, such as vector-based coefficients. Although the authors provide an intuitive explanation, the absence of rigorous theoretical guarantees remains a notable limitation.

I recommend this paper for spotlight acceptance because it offers an exceptionally strong and practical solution to a critical problem in the DDM space. The empirical results are not only compelling but also thoroughly validated through the rebuttal process. The authors' commitment to rigorous empirical evaluation, as demonstrated by the extensive new experiments, elevates the paper to "spotlight" status, provided that all promised improvements are fully integrated into the camera-ready version.


The author-reviewer discussion was productive, with the authors effectively addressing nearly all concerns. Reviewers 9x6X and ACPU initially raised questions about the paper's limited scope and scalability. The authors responded by providing extensive new results on larger datasets, which successfully demonstrated the method's robustness and scalability. Reviewer 61AF inquired about integrating the method with other techniques and solvers, and the authors provided new data showing successful integration with predictor-corrector solvers, remasking, and parallel decoding. The lack of a theoretical foundation was a key concern for Reviewers 9x6X, ACPU, and odGL. While the authors could not provide a formal proof during the rebuttal period, their intuitive explanation for the method's effectiveness was well-received and helped all reviewers feel confident in the empirical results. Reviewer VCFx raised concerns about the small scale of the experiments and the robustness to hyperparameters; the authors provided a new ablation study on the Hamming distance threshold that addressed those concerns. Most of the reviewers either maintained their positive score or raised their rating, leading to a  consensus for acceptance.
Decision: Accept (spotlight)