Beyond Autoregression: Fast LLMs via Self-Distillation Through Time
Authors: Justin Deschenaux
Caglar Gulcehre
Venue: ICLR 2025 Poster (2025)
Keywords: language modeling
LLM
diffusion models
discrete diffusion models
diffusion language models
distillation
Primary area: generative models
TL;DR: Discrete diffusion language models can be made faster than auto-regressive models at sampling, while also generating higher quality text and achieving better performance on NLU benchmarks

Abstract: Autoregressive (AR) Large Language Models (LLMs) have demonstrated significant success across numerous tasks. However, the AR modeling paradigm presents certain limitations; for instance, contemporary autoregressive LLMs are trained to generate one token at a time, which can result in noticeable latency. Recent advances have indicated that search and repeated sampling can enhance performance in various applications, such as theorem proving, code generation, and alignment, by utilizing greater computational resources during inference. In this study, we demonstrate that diffusion language models are capable of generating at least 32 tokens simultaneously, while exceeding the performance of AR models in text quality and on the LAMBADA natural language understanding benchmark. This outcome is achieved through a novel distillation method for discrete diffusion models, which reduces the number of inference steps by a factor of 32-64. Practically, at the 1.3B parameters scale, diffusion models, even without caching, can generate tokens at a rate that is up to 8 times faster than AR models employing KV-caching, and we anticipate further improvements with the inclusion of caching. Moreover, we demonstrate the efficacy of our approach for diffusion language models with up to 860M parameters.

Published as a conference paper at ICLR 2025
BEYOND AUTOREGRESSION: FAST LLMs VIA HELF-DISTILATTON THROUGH TIME
Justin Deschenaux, Caglar Gulcehre
School Computer and Communication Sciences
CLAIRE, EPFL
Lausanne, Switzerland
{justin. deschenaux, gualaar.guccehreeeppfL..
ABSTRACT
Autoregressive (AR) Large Language Models (LLMs) have demonstrated sig- nificant success across numerous tasks. However, the AR modeling paradigm presents certain limitations; for instance, contemporary autoregressive L.LSs are trained to generate one token at a time, which can result in noticeable latency. Recent advances have indicated that search and repeated sampling can enhance performance in various applications, such as theorem proving, code generation, and alignment, by utilizing greater computational resources during inference. In this study, we demonstrate that diffusion language models are capable of gener- ating least 32 tokens simultaneously, while exceeding the performance of AR models in text quality and on the LAMBADA natural language understanding benchmark. This outcome achieved through a novel distillation method for dis- crete diffusion models, which reduces the number of inference steps by factor 32-64. Practically, at the 1.3B parameters scale, diffusion models, even with- out caching, can generate tokens at a rate that is up to 8 times faster than AR models employing KV-caching, and we anticipate further improvements with the inclusion of caching. Moreover, demonstrate the efficacy of our approach for diffusion language models with up to 860M parameters.
INTRODUCTION
In recent years, autoregressive (AR) large lan- guage models (LLM) have exceeded expectations (Vaswani al., 2017: Devlin al., 2018; Radford et al., 2019; Brown et al., 2020b; Kaplan et al. 2020; Raffel et al., 2020; Fedus et al., 2022: Hoffmann et al., 2022; Chowdhery et al., 2023; Google, 2023; Touvron et al., 2023). Importantly, many break- throughs in coding (Chen et al., 2021). mathemat- ics, and reasoning (Trinh et al., 2024b;a; Romera- Paredes et al., 2024; Hosseini
4P1 20
180 di
vsi
TVD
20
al., 2024: Wang et al., 2024) were achieved based on decoding large amounts of completions from a base LLM.
10
Labancy taec)
Figure 1: Perplexity versus latency. The diffusion models (169M) use 16, 32, 64, 128 and 256 decoding step.
Importantly, the benefits of repeated sampling can be so significant that often more efficient use a smaller. faster model rather than a larger, slower one.
More generally, one can improve the performance of a fixed model by scaling up computational re- sources at inference time (Madaan et al., 2023; Yao et al., 2023; Snell et al., 2024; Wu et al., 2024; Chen et al., 2024; Brown et al., 2024; Goyal et al., 2024), phenomenon that was previously ob- served for games (Campbell et al., 2002; Silver et al., 2016; Lerer et al., 2019 Brown et al., 2020a; Jones, 2021). Hence, when tackling reasoning tasks, major bottleneck the latency of the model. In this work, build on recent breakthroughs discrete diffusion (Lou et al., 2023; Sahoo et al., 2024: Shi et al., 2024; Ou et al., 2024). Our approach generate text up to 8 times faster than AR models that
improve the decoding speed of LLMs by moving away from AR modeling. We

---

Published as a conference paper at ICLR 2025
LAMBADA
LANBADA
0.50
100
0.40
Porplottt
0.25
20
Number of minds of SDIT
Number rounds SDIT
(a) Accuracy of the correct last word decoded from our model. Distillation with KLD loss leads the student model outperform teacher terms of accuracy on LAMBADA.
(b) Perplexity of the last word. The KLD preserves performance best, and when the student is trained to sample with 16 instead of 1024 steps, the student still matches AR baselines.
Figure 2: Performance LAMBADA after multiple rounds of SDTT with different distillation
losses. We pre-train with the masked diffusion language modeling objective (MDLM) (Sahoo et al., 2024) and distill with 7 rounds of SDTT. Note that single word in the LAMBADA data set often consists of multiple tokens. We greedily decode all tokens a single forward pass for the diffusion models and decode autoregressively for the AR models.
use KV caching (Pope al., 2022). Diffusion models are typically trained maximize the evi- dence lower bound (ELBO), which does not consider the desired number of inference steps. Hence, vanilla diffusion models typically require thousands of decoding steps. Fortunately, is possible to drastically reduce the inference costs of continuous diffusion models via distillation (Luhman & Luhman, 2021; Salimans & Ho, 2022). Continuous distillation methods rely on deterministic map- pings from noise to data, such as DDIM (Song et al., 2022). The deterministic mappings can be efficiently learned by student diffusion model to sample in fewer steps. We hypothesize that such deterministic map cannot exist for the diffusion language models studied in this work. Indeed, those models always initialize the denoising process with a sequence of masked token, hence a determin- istic algorithm can only generate a single sample. As such, we devise a distillation method that does not does depend on deterministic maps. This is a significant finding because faster decoding mechanisms allow exploring a larger search space in applications that require search, planning, and reranking. In summary, our core contributions are as follows:
We introduce Delf-Distillation Through Time (SDTT), which allows generating at least 32 tokens time, while achieving better perplexity than GPT-2 with nucleus sampling for conditional and unconditional generation. Unlike many distillation methods for continuous diffusion models, SDTT does rely on deterministic mappings such as DDIM (Song et al., 2022). SDTT is very simple and easy to implement.
We show that SDTT generate tokens up to 8 times faster than AR models that use KV caching, for models with 1.3B parameters, in 16 decoding steps. Importantly, the discrete diffusion model does not rely activation caching, suggesting that there is potential for even greater efficiency gains. The latency gains for smaller models are even greater.
We demonstrate the effectiveness of SDTT for models with up to 860M parameters. To the best of our knowledge, this represents the largest publicly available discrete diffusion language model.
We evaluate the distilled students LAMBADA (Paperno al., 2016) and 6 multiple- choice questions benchmarks from Gao et al. (2021). We find that SDTT preserves the natural language understanding performance of the teacher.
2

---

Published as a conference paper at ICLR 2025
100
Distili, target
140
rounds) rounds) rounds) rounds)
Logits
AR 1p 0.95)
Input tokens
Selt
MMAkk
Thrugg
Time
Logits
OAOG Cup BAARS
Input tokens
Selt
eMAdIo
Through
dARGO
sampling
(a) The distillation targets the log probabilitics that lead to token being denoised, concatenated with log probabilities the last step for tokens remain masked.
(b) SDTT small models trained for 1M steps. Suc- cessive lines correspond additional SDTT rounds. SDTT can outperform the teacher GPT-2 with nu- cleus sampling.
Figure 3: SDTT. figure (a), we illustrate how we prepare the distillation targets, In figure (b), display the generative perplexity of samples after distillation.
2 BACKGROUND
2.1 MASKED DIFFUSION LANGUAGE MODELING
We follow the notation of Sahoo al. (2024) to introduce masked diffusion language model- ing (MDLM). Language modeling can framed as the sequential prediction task of discrete tokens (I;) coming from vocabulary X' z<N {0, ... N - 1} that can take N possi- ble discrete values. A language model would predict sequences of length L, which can be de- x(() (=)
fined as the sequences of x;'ss originating from x
= iezek" Let D = (x(0), ...., (K- : x() E xz} denote the training set. The goal of language modeling sample from the unknown distribution Po x - [0, 1] that generated the samples in D.
Similarly to continuous diffusion, we sample from an approximation of po by learning to denoise corrupted examples. One can sample from the model through ancestral sampling, starting from a stationary distribution. The stationary distribution of Sahoo et al. (2024) is such that all tokens of the sentence are replaced with special MASK token like the MASK token used for pre-training BERT models. However, key difference between BERT and MDLM is that MDLM is trained on sequences with varying levels of corruption, while BERT uses a fixed ratio.
Discrete absorbing diffusion process MDLM defines forward process to corrupt data and a backward process to learn to recover data. MDLM uses continuous-time formulation, with the data distribution denoted as Po and the stationary noise distribution as P1 æ. The forward process linearly interpolates between the one-hot distribution defined by the original document and the stationary distribution π, which places all mass on the MASK token. Mathematically,
q(z,|x) = Cat(z,; a.x (1 a,)
where the noise injection schedule is defined by dr, for L E [0, 1]. The constraints on aa are that ae E 1], aa should be strictly decreasing function of t, and op x 1, 1, s 0. The forward process is called absorbing because once a token is assigned to a MASK token, it cannot be reverted to real token.
We can derive the analytical form of the reverse process q(z, Ze x), with t s and azls = 2a as
[an + (1 - -(t)e) 1 ④ [0,,x a.)
g(z,/Ze, x) = Cat Zsi
azz, x (1 2z
3

---

Published as a conference paper at ICLR 2025
Algorithm Computing the Self-Distillation Through Time targets X Steacher (Zt, L, m//2)
1:
Inputs: Noisy tensor X₁ RRNLL Starting sampling time Estart E [0, 1]N, Number of sampling
steps m//k 2 2, such that m//k E N4, Sampling step size E (0, 1), Mask token index M E N, Minimal sampling time e.
2: Output: Distillation targets xhacher (Zz, m/kk
3:
4: target f- zeros(N, L, K)
Allocate empty tensor for xclacher (Zz, , t, m/k)
5: 2
<- Xt
6: for = 0, m/k do
7: teurr + max(I i (.
Sampling step for the current time P Updated z & log-probabillties
8:
Znows lrencher + reverse.samplet. teurrs A)
Fg 12, teurr)
9:
U Zneo + z target[Ui] t- freacher [U]
Create mask U of tokens that were denoised Extract oog-probs for the denoised tokens Update for the next iteration
10:
t- Znew
12:
end for
target[z M] lreacher 1z M] tokens
Use log-probs of the last denoising step for masked > Target log-probe for all masked tokens in X;
return target
Objective and parameterization To generate new samples, we can simulate the reverse process from eq. (2). Since the ground-truth sample unknown, Sahoo et (2024) learn an approxima- tion Xş using neural network with parameters 0. Sahoo et al. (2024) then use instead of x to sim- ulate the reverse process. The sampling distribution is denoted as po 2e) := gZ,,] Ze, X (Z,, t)). Sahoo et al. (2024) optimize O using continuous version of the negative evidence lower bound (NELBO) of Sohl-Dickstein et al. (2015a). Previous research has shown that continuous time objec- tives optimize the data likelihood better (Kingma et al., 2023). Due to the definition of the absorbing diffusion process, the NELBO simplifies to a weighted cross-entropy loss between the ground-truth and the model predictions Xo:
of Oit
2 NELBO
Ey 1=
(xg(xx (), x/.
To derive eq. (3), Sahoo et al. (2024) impose two properties (2 22). First, denoised tokens are never re-masked during sampling. Practically, this achieved by manipulating the output of the neural network xe( (z, t) to ensure that probability mass is assigned to the MASK token. Secondly, already-denoise tokens are carried-over to the next sampling step. Sahoo et al. (2024) showed that both constraints lead to improved likelihood.
2.2 KNOWLEDGE DISTILLATION
Knowledge distillation (Bucila al., 2006; Hinton et al., 2015) technique where student neural network is trained to imitate the predictions of a more complex teacher model. One of the main advantages distillation the ability to reduce the inference cost associated with sampling from large LLMs while surpassing the performance of smaller models trained without distillation (Gu al., 2024; Agarwal et al., 2024). The most relevant to our work are the distillation methods that match the predictions of the teacher and the student using divergence measure d:
Ex-D (x(u((x((xx))) (x)))))))))
Where Mes Me the AR distributions of the student and teacher, respectively, and D represent the training dataset. Common divergence measures include f-divergences (Wen et al., 2023) such as the Bollback-Leible divergence (KID) or the total variation distance (TVD).
4

---

Published as a conference paper at ICLR 2025
Algorithm 2 One training round of Self-Distillation Through Time
1: Inputs: Training set D. Teacher x8, Divergence measure 6, Number of sampling steps m/k, Sampling step size A, Mask token index M, Total number of training steps H
2: Output: Distilled student X..
3:
4: --
Initialize the student with the teacher weights
5: for i 0, H - do
6: Xa f- www.aa....aaappp...mmmamm...p (D)
> Sample a training example > Sample uniformly at random Forward diffusion process. See eq. (1)
7: Estart ~ [0,
8:
Xx ~ 4r (x:(xx/xx (xx
9:
Xstudent t X (xt,
10:
Xacnchor f- www.thee.ssTTeTTS
statt m/kk A, 1e-5)
> See algorithm > Compute divergence between student and SDTT targets. > Update the parameters of the student with AdamW
11: C 8 (Xttudent Ktacher)
12:
1/ t t-backprop opt im (C, v)
13: end for
14: return X,
METHOD
3.1 SELF-DISTILLATION THROUGH TIME
As explained in section 2.1, discrete diffusion language models optimize the NELBO over the train- ing examples. Fewer decoding steps typically lead to lower sample quality because the approxima- tion of the reverse process is less accurate, as visible in the teacher curve in fig. 4.
To address the issue of low sample quality with fewer decoding steps, we propose Self-Distillation Through Time (SDTT). SDTT fine-tunes pre-trained MDLM to allow decoding with significantly fewer steps. Interestingly, steps than the teacher would with 1024 forward passes. In short, SDTT improves the sampling speed by distilling the inference time computation to sample multiple steps into the student.
final model decodes samples with lower generative perplexity in 32
(m)
Let Pa be the distribution of samples generated with m steps, using a denoiser with parameters 0. SDTT trains denoiser with parameters minimize divergence between pb and pba) Here < m, and divides m (e.g., m 1024 and k 512):
min d (p!"" (0))))
(5)
Since X9 and x, are the only learnable elements of the sampling process, they completely determine the sampling distributions Pg (m) and pbk) As such, training x,, match the predictions of x9 with
(Zi, fewer steps minimizes eq. (5). We now present method for generating targets x xetacherr (Z,, m/ls) to
train X.. Mathematically, we optimize the following objective:
min Pan.D.eerg..a.le (xxxx(20)
| teacher (2t, m/xx)]
where divergence measure between the student and the teacher targets x xeeaaber (Zz, m/k)). We consider the Kullhack-Leibler divergence (KID), Total Variation Distance (TVD), and Mean- Squared Error (MSE). See appendix B for details on those divergence measures.
Generating the Teacher Targets Following the terminology of knowledge distillation, we call the denoiser used for many steps decoding as the teacher and the denoiser x, used for few steps de- coding as the student. To train x, to match the predictions xa, we sample from the teacher for m/x steps. Whenever MASK token is denoised, we collect the log probabilities predicted by the teacher for this MASK token. These log- probabilities become the distillation targets x xtencher (zz. t, m²/k). Al- gorithm 1 outlines this process and fig. 3a presents it visually. While fig. 3a shows how distill two decoding steps in one, the procedure can be extended to larger values of m/k. The complete SDTT training loop is presented in algorithm 2.
5

---

Published as a conference paper at ICLR 2025
Iterated SDTT SDTT reduces the number of decoding steps by factor m//k. If we want to reduce the number of decoding steps further, we can apply SDTT with k' < k. or alternatively apply SDTT times, using the newly distilled student as teacher for the next round, which we refer to as iterated SDTT. Instead of directly optimizing the divergence in eq. (5), we introduce n intermediate distributions pr: such that m/k; is an increasing sequence function of i. In practice, we choose m 20 and k; 200 with S S 7 and sequentially minimize the objective
min d (1+44) (Puy+1 lIb!)))
for 0 J 7, where vj denotes the parameters of the j-th denoiser, with Vo = 0 (teacher). the minimization procedure was perfect, minimizing eq. (5) or eq. (7) should result in the same solution. However in practice, we observe that it is easier to minimize eq. (7) sequentially for increasing values of i, in progressive fashion, similar to Salimans & Ho (2022).
As an alternative to iterated SDTT, we tried using a single model and slowly growing the step size used to generate X xilacher t, m/k). Unfortunately, this approach was unstable and the loss diverged after 30-50 steps, irrespective of how small the sampling step size was. Similar behavior was observed by Norouzi et al. (2023).
4 EXPERIMENTS
We distill MDLMs on the Open WebText dataset (Gokaslan & Cohen, 2019) as it was used to train recent discrete diffusion language models (Lou al., 2023; Sahoo et al., 2024). We use the Adam optimizer with learning rate of 6e 5, a batch size of 128 and no weight decay. We linearly increase the learning rate for 500 training steps and keep constant afterwards. As a base model, reuse the checkpoint released by Sahoo et al. (2024). See appendix C for more details.
In section 4.1, we evaluate distillation divergences and show that iterated SDTT can reduce the number of sampling steps by factor 16-32. In section 4.2, we ablate on the importance of hyperpa- rameters, including the duration of each round of iterated SDTT and the number of sampling steps to generate the targets 2gacher(z,, t, m/k). In section 4.3, scale SDTT to models with of up to 860M parameters. Finally, in section 4.4, we compare the latency of SDTT against autoregressive models that use KV caching.
Generative perplexity Following prior work (Dieleman et al., 2022: Lou et al., 2023; Sahoo et al., 2024), we use a larger model to compute the generative perplexity of unconditional and conditional samples. We evaluate the smallest students using GPT-2 (large) (Radford et al., 2019). In the scaling experiments, we use Llama3 8B (Touvron al., 2023), since we compare models with up to 860M parameters. As noted by Zheng et al. (2024a), the generative perplexity is sensitive to the floating-poii precision. In this section, we sample using bfloat16, and report results using float64 in appendix A. The conclusion are similar.
MAUVE
We evaluate conditional generation using the MAUVE score (Pillutla et al., 2021). MAUVE measures how well model follows prompt by comparing multiple generations with a reference continuation. We use the first 1024 samples with at least 1024 tokens from the WebText dataset (OpenAI, 2019), take the first 50 tokens as prompt, and generate 50 tokens of continuation. For each prompt, we generate 5 contimuations, as done in Lou et al. (2023).
Sample diversity Post-training can drastically reduce the diversity of language models (Kirk al., 2024: Agarwal et al., 2024; Li et al., 2024). Hence, we measure the diversity of samples using the self-BLEd score (Zhu et al., 2018) with the same completions used compute MAUVE.
Downstream performance We measure the downstream performance using the LAMBADA dataset (Paperno et al., 2016), as well as multiple-choice question (MCQ) tasks from Gao et al. (2021). On LAMBADA, we report an upper bound on the perplexity, computed using the ELBO (3). We also report the suffix accuracy by masking all tokens of the last word and predicting all of them in single forward pass, using the argmax of the predictions. The diffusion model is correct only all the masked tokens are decoded correctly in a single decoding step. The other benchmarks from Gao et al. (2021) evaluate the MCQ accuracy.

---

Published as a conference paper at ICLR 2025
125 siopa Round Round Round
Tound CPT2 ingl
Small (2800)
sipps
1024
Mid (424N)
32 32.04p5 stops
6PT2 (pm0:)5)
Large (855M)
Student
512
Teacher
Candboral soslla
1.50
Parpoodor
128
1.00
66-
32.5
32.0
52.5 32.0
12.0
sampling
(a) Diversity of conditional generation (small scale). We measure the trade-off between quality and diver- sity using self-BI.EU (Zhu et al., 2018). Deterministic
(b) Scaling SDTT 860M parameters. The plot compares the performance of the teacher and final stu- dent rounds). The student and teacher have the same size. The small distilled student reaches lower perplex- ity than the large teacher.
sampling yields a score of 1. The diversity minimally decreases after distillation.
Figure 4: Sampling step ablations on perplexity. Perplexity of samples after each round of iterated SDTT. (a): Iterated SDTT on small model trained for 1M step. (b): Scaling SDTT to larger models trained for 400K steps.
4.1 ABLATION ON THE TRAINING DIVERGENCE
SDTT requires choosing a divergence and we study the Mean dean-Squared Error (MSE), Total Vari- ation Distance (TVD) and (reverse) Kullhack-Leibler Divergence (KLD). We apply iterated SDTT for 7 rounds of 10k: training iterations and generate xteachor I, m/xx) with 2 sampling steps from the teacher (algorithm 1). We use an exponential moving average (EMA) of the weights with a decay of 0.9999 that we do not reset between rounds.
Figure shows that students distilled with the KLD clearly outperform students trained using the MSE and TVD on LAMBADA. The LAMBADA accuracy of students tuned with the KLD slightly improves over the teacher. while the perplexity remains better or matches the AR baselines for all but the last round of SDTT. The improved accuracy LAMBADA suggests that the model is better at predicting multiple tokens in parallel after distillation with SDTT, since we evaluates the accuracy by decoding all tokens of the last word simultaneously.
Figure shows that the KLD seem to outperform the MSE and TVD objectives on MAUVE. Since generate sequences of 100 tokens only for MAUVE, following (Lou et al., 2023), sample with at most 128 steps, and use samples generated with 128 sampling steps from the teacher as baseline. Note that as observed by Deschenaux & Gulcehre (2024), discrete diffusion models typically achieve slightly lower MAUVE scores than AR models. Nonetheless, distillation with the KLD objective improves the MAUVE score the students. Similarly fig. 18 shows that continuations from the student distilled with the KLD reaches the lowest perplexity and match GPT-2 with nucleus sampling in 32 forward passes.
In table 1, we compare the downstream performance on the tasks of Gao et al. (2021) before and after distillation. We observe that SDTT minimally affects the results, and that student distilled with the KLD objective reaches higher accuracies than other students in all but one task
Figure 4a measures the diversity of samples using the self-BLEU score (7hu et al., 2018), for the students distilled with the KLD objective. See appendix A for results with the MSE and TVD. We find that SDTT minimally decreases the diversity. Compared to distilling autoregressive models (Agarwal al., 2024), SDTT minimally reduces the diversity. For reference, Agarwal et al. (2024) routinely observes an increase of 15 in BeLf-BLEU while we observe a change at most 2 for the KLD student. See appendix A for more results and details the self-BLEU score.
Figure 6 shows that students distilled with KLD have higher unconditional generative perplexity than those distilled with the MSE However, KLD is the only objective that preserves performance

---

Published as a conference paper at ICLR 2025
round
2 rounds
3 rounds
MAUVE
0.90
MAUVE
0.90
MAUVE
0.90
0.85
0.85
0.85
0.80
0.80
0.80
16
32
64
128
16
32
64
128
16
32
64
128
Num. sampling steps
Num. sampling steps
Num. sampling steps
4 rounds
5 rounds
6 rounds
MAUVE
MAUVE
MAUVE
0.85
0.85
0.85
0.80
0.80
0.80
16
32
64
128
16
32
64
128
16
32
64
128
Num. sampling steps
sampling steps
Num. sampling steps
Teacher
KLD
MSE
TVD
Figure 5: MAUVE performance of the student after each round of SDTT. The teacher performance is computed using samples generated with 128 decoding steps.
in the LAMBADA data set while still significantly reducing the generative perplexity compared to the teacher. Therefore, in the remainder of this work, we focus on the KL.D.
4.2 ADDITIONAL ABLATIONS
Number of steps in each SDTT round In section 4.1, each round of SDTT consists of 10k train- ing iterations. Since the magnitude of the distillation loss does not reliably indicate convergence, experiment with shorter rounds. We find that reducing the number of training iterations 5k or 2.5k negatively impacted conditional generation performance, as shown in fig. 7. However, shorter rounds slightly improved the final generative perplexity (fig. 8) and resulted marginally better LAMBADA perplexity (fig. 10). Since SDTT does not directly optimize the ELBO, an increase in perplexity is expected. Interestingly, the LAMBADA accuracy remains unchanged with shorter rounds.
Number of sampling steps to generate the targets In section 4.1, the targets x xelacher (zz, t, m/k) are generated using 2 sampling steps from the teacher. We explore distilling a larger number of sam- pling steps once (4 or 8), since using more rounds of SDTT may induce more error accumulation in approximating the original teacher. Figure 13 shows that distilling more than two steps at a time difficult and results in weaker results on LAMBADA. This suggests that the higher stochasticity of the targets generated with four or eight steps makes the task too difficult for the student.
Generating targets with the analytical sampler Lou et al. (2023) observe that using analytical sampler (Campbell et al., 2022) results higher quality samples compared to ancestral sampling. However, when generating targets x xeencher (Z, (z,,t, m/kk with analytical sampling, observed minimal difference with ancestral sampling, as shown in fig. 11 and 12.
Resetting the optimizer and Exponential Moving Average between rounds Using an Expo- nential Moving Average (EMA) of the weights known to improve the quality of samples from diffusion models (Nichol & Dhariwal, 2021). However, when applying SDTT for multiple rounds, it unclear whether the EMA or current weights should be used as the teacher for successive rounds. Additionally, it could be favorable to reset the optimizer state between rounds as we grow the de- coding step size. We experiment with two approaches: either resetting the optimizer state only, or resetting both the EMA and optimizer state. Figure 14 shows the generative perplexity when reset- ting the optimizer state and using the EMA as the teacher instead of the current weights, while fig. 15 presents the corresponding results for MAUVE. When using the EMA teacher, since accumu- late updates in the EMA over 10k training iterations only, we use a slightly lower decay rate of 0.999. We find that using the EMA of the weights the teacher may slightly improve performance.
8

---

Published as a conference paper at ICLR 2025
Baselines
160
100
Racher
Racher
240
SOIT 11 munds)
SDTT mundsl
SOIT 12 rounss)
SDIT rounds)
120
SDIT (4 roundsi
SDIT 13 rounss)
S0IT rounds)
S0IT rounds)
SDIT 15 rounds)
SOIT rounds
100
SDIT 16 rounds)
SOIT rounds
SDIT rounds
SDTT rounds
MLD
80
MLD
MBE
TVC
Ro traincs AA
Rc: Vtincd AA
25
32
1248
256
1024
25
32
1024
sampping
sampling steps
(a) KLD MSE
(b) KLD vs TVD
Figure 6: Perplexity for different losses and decoding step size. Generative perplexity over rounds of SDTT with MSE, TVD and KLD. While the KLD leads to a higher perplexity than the MSE; we focus on the KLD because it is the only divergence that retains the performance on the LAMBADA dataset.
4.3 SCALING SDTT TO 860M PARAMETERS
We apply SDTT to larger discrete diffusion models with up to 860M parameters. In this experiment, train the models from scratch for 400k steps with a batch size of 512, context length of 1024 and the Adam optimizer. We reuse the training configuration of Sahoo et al. (2024) and scale the models to larger sizes. We train 3 model sizes, small (169M), medium (424M) and large (863M). Details of the model architecture for each scale are shown in table 2. As for the other experiments, the models are diffusion transformers (Peebles Xie, 2023) and we use an EMA with decay of 0.9999. Although the results in section 4.2 suggest that short distillation rounds might be sufficient, it is unclear whether this result also holds on larger scales. Therefore, we use 10k steps per round of SDTT. For simplicity, generate targets using 2 teacher ancestral decoding steps and do reset the optimizer state or EMA between rounds.
Since train larger models, evaluate the generative perplexity using Llama3 8B (Touvron et al., 2023). The generative perplexity over the 3 model sizes is shown in fig. 4b. Interestingly, the smaller diffusion model (169M) sampled from with 64 steps more after distillation achieves better generative perplexity than the largest model (863M) when sampling with 1024 steps. In fig. 16, we show that the MAUVE performance also improves after distillation for the medium and larger model. Finally, in fig. 17, we see that the LAMBADA accuracy improves after distillation, similar in the smaller scale, when using the KLD objective.
4.4 LATENCY WITH SDTT
While SDTT allows sampling from discrete diffusion models with 32-64 times less decoding steps, quantity of interest to practitioners the actual latency of text generation. Indeed, while the reduc- tion in the number of sampling steps is large, since discrete diffusion uses non-causal architecture, we cannot use KV caching (Pope et al., 2022). KV caching improves the inference performance drastically for AR models, hence we compare the latency of SDTT with GPT-2 with KV caching. We successfully reproduce the results of Deschenaux & Gulcehre (2024), which showed a 4x im- provement when sampling with 32 steps, and measure an 8x improvement with 16 decoding steps. We compute the latency using untrained models with around 1.3B parameters, using the same hy- perparameters as Deschenaux & Gulcehre (2024). We use batch size of 8 and time the sampling 10 times after one warm-up step on a single A100 GPU with 80 GiB of RAM. All models use Attentinn (Dao et al., 2022). See Appendix A for additional experiments the latency.

---

Published as a conference paper at ICLR 2025
5 RELATED WORK
Diffusion Models Diffusion models (Sohl-Dickstein et 2015b; et al., 2020; Song & Ermon, 2020) are the basis of many state-of-the-art lext-to-image models (Ramesh al., 2022: Rombach et al., 2022; Saharia et al., 2022). After their introduction by Sohl-Dickstein et al. (2015b), Ho et al. (2020) showed that diffusion models can achieve FID scores (Heusel et al., 2017) comparable to GANs (Goodfellow et al., 2014; Arjovsky et al., 2017).
Discrete Diffusion & Difffusion Language Models Prior to Sahoo et al. (2024); Shi et al. (2024); Ou et al. (2024), Lou et al. (2023) introduced novel discrete diffusion language model called SEDD. When decoding with a large number of steps, SEDD can match or surpass GPT-2 in uncon- ditional text generation. The model of Lou et al. (2023) learn discrete generalization of the score of continuous diffusion models (Song & Ermon, 2020; Song et al., 2021). Campbell et al. (2022); Zhao et al. (2024) developed the continuous-time discrete diffusion framework. Hoogeboom et al. (2021) extended Bernoulli diffusion (Sohl-Dickstein et al., 2015b) to categorical distributions, and Austin et al. (2023) generalized the work of Hoogeboom al. (2021) more general corruption processes, including absorbing diffusion. Zheng et al. (2024b) develop family of re-parameterized discrete diffusion models to enhance the training and decoding efficiency. parallel, several studies have explored continuous diffusion for language modeling (Li et al., 2022: Dieleman et al., 2022; Han et al., 2023; Chen et al., 2023; Gulrajani & Hashimoto, 2024). Despite recent breakthroughs, diffusion language models still have some drawbacks (Deschenaux & Gulcehre, 2024). Ye et al. (2024) adapt Shain-of-Thougeee reasoning (Wei et al., 2023) to diffusion models.
Distillation of Continuous Diffusion models Distilling continuous diffusion models is well- studied area. For a comprehensive survey, see Luo (2023). Many distillation methods rely on De- noising Diffusion Implicit Models (DDIM) (Song al., 2022), which showed that diffusion mod- els can be sampled deterministically. Luhman & Luhman (2021) unroll trajectories sampled with DDIM and train a student map noise directly images. Luhman & Luhman (2021) pre-compute a dataset of noise-image pairs. Close to our work, Salimans & Ho (2022) teaches the student to match multiple sampling steps of the teacher, given corrupted training examples. However, unlike Salimans & Ho (2022), we cannot rely on the existence of a deterministic map via DDIM. Con- sistency distillation (Song et al., 2023) fine-tunes a pre-trained diffusion model to predict the final sample from intermediate points of the sampling trajectory, which enable faster sampling. Luo et al. (2024) distills a pre-trained diffusion model into single-step generator through a novel loss, Integral Kullback- Leibler divergence. SD-XL Turbo (Sauer al., 2023) uses adversarial formulation to sample with 1-4 steps from a latent diffusion model (Rombach et al., 2022).
Masked & Non Auto- Lugo-Regressive Language Modeling BERT (Devlin al., 2018) introduced the masked language modeling objective. While BERT focuses on representation learning, discrete dif- fusion language models are generative. XLNet (Yang et al., 2020) uses generalized AR pretrtaining method to model the text distribution over all permutations of the training sequences, outperforming BERT on downstream tasks. Pannatier al. (2024) adopt a similar objective to XLNet for generative modeling instead of natural language understanding.
6 DISCUSSION
In this work, we introduce Self-Distillation Through Time (SDTT), distillation method for discrete diffusion models. Recent works (Lou et al., 2023; Sahoo et al., 2024: Shi et al., 2024; Ou et al., 2024) suggest that discrete diffusion models can match or outperform autoregressive models in text quality. However, those models require more inference resources than AR models to achieve good performance, because of the non-causal architecture of the neural network that prevents the use of KV caching. We show that SDTT can reduce the number of decoding steps while retaining performance. Our final student is up 8x faster than AR models that KV caching and demonstrate that SDTT is applicable to larger models as well. In future work, we plan to evaluate SDTT on tasks that involve generating large number of completions from base language model.
10

---

Published as a conference paper at ICLR 2025
7 REPRODUCIBILITY STATEMENT
We provide details on model architectures, hyperparameters, and provide pseudocode for our algo- rithm. We built on top of the open source model of Sahoo et (2024), which makes relatively easy for researchers to reproduce our results. Additionally, upon de-anonymization, we will release our code and artifacts.
ETHICS STATEMENT
Overall, language models are dual-use technologies, and thus, they can have unethical uses, such as fake content generation, and they can suffer from bias if applied to data sets that are not carefully curated. This paper focuses specifically on speeding up discrete diffusion language models at test time to reduce their computational demands; we do not have specific concerns with regard to this contribution.
9 ACKNOWLEDGEMENTS
We thank the ICLR'25 reviewers, area chairs, and organizers for their valuable feedback and support.
We acknowledge the SCITAS team EPFL for providing access to their beta cluster, and Karin Gétaz for her administrative assistance. This work was supported by the Swiss AI Initiative through a grant from the Swiss National Supercomputing Centre (CSCS), project ID al0 on Alps. Special thanks to Skander Moalla for providing reproducible compute infrastructure code template.
REFERENCES
Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self-ggnerated mistakes, 2024. URL https: htap:s:///rxxii http://axxvvv.rree org alos 22022.33.111 13649.
Martin Arjovsky, Soumith Chintala, and Léon Bottou.
Wasserstein gan, 2017. URL https:
//arxiv. (arrg//bs//300
Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne den Berg. Structured denoising diffusion models discrete state-spaces, 2023. URL https: peattts://erxiv... http://wxxvv.....aa.//. 21 2107.00 03006.
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoe V. Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. 2024. URL https: arxiv. org/ /220//7..2222... 217877.
Noam Brown, Anton Bakhtin, Adam Lerer, and Qucheng Gong, Combining deep reinforcement learning and search for imperfect-information games, 2020a. URL aitpss:/aaree org/ abs/2000 13544.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020b. 2020b.
Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. Knowl- edge Discovery and Data Mining, 2006. URL http:::/ppp..ssmattttaaaa ses.tt.sattttcsccoll....
CorpusID: :
Andrew Campbell, Joe Benton, Valentin De Bortoli, Tom Rainforth, George Deligiannidis, and Arnaud Doucet. continuous time framework for discrete denoising models, 2022.
Murray Campbell, A.Joseph Hoane, and Feng hsiung Hsu. Deep blue. Artificial Intelli- gence, 134(1):57-83, 2002. ISSN 0004-3702. doi: https///oioong//0000060044440000 00129-1. URL http://wwwwiieeeeerrre...c.... http://wwwecaaeccooooooo////aaaa 5000437020100 001291..

---

Published as a conference paper at ICLR 2025
Lingjiao Chen, Jared Quincy Davis, Boris Hanin, Peter Bailis, lon Stoica, Matei Zaharia, and James Zou. Are more Ilm calls all you need? towards scaling laws of compound inference systems, 2024. URL https: / wtwpaaxivvv....../..a aass//////////e/ /ta.//2403. 024.12229.........
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint urXiv:2107.03377 2021.
Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240): 1-113, 2023.
Tri Dao, Daniel Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022. URL 2205. 14135.
org/ abs/
Justin Deschenaux and Caglar Gulcehre. Promises, outlooks and challenges of diffusion language modeling, 2024. URL https://arxiisi wtwp:/iw.rrr/ab//440000 11..0.6..14333 11473.
Jacob Devlin, Ming-Weet Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H. Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, Curtis Hawthorne, Rémi Leblond, Will Grathwohl, and Jonas Adler. Continuous diffusion for categorical data, 2022.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 2311))):1-,, 2022.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot lan- guage model evaluation, September 2021. URL httt:s///ddd..orr/00.22//////// 5371629.
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. CpenWebTTettCoppuuu 2019.
io/
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014. URL https: //arxiv. wwwpeiiv.rrraaooooe abs 1106 2661.
/
Google. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 2023.
Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens, 2024. URL https:/ //arii////vvv//// org/ abs 2210.10... 02226.
Yuxian Gu, Li Dong. Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large lan- guage models, 2024. URL https: /ttpivaivvv////////////a////aaaa org/ wtgg/bs//////oooooo/o/..p//...rp abs/2306 08543.
Ishaan Guirajani and Tatsunori B Hashimoto. Likelihood based diffusion language models. Ad- vances in Neural Information Processing Systems, 36, 2024.
Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. Ssd-Im: Semi-autoregressive simplex- based diffusion language model for text generation and modular control, 2023. URL https: / //arxiv.co www.vvorg/abaa. abtp///bb//22/ 17432.
12

---

Published as a conference paper at ICLR 2025
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochre- iter. Gans trained by two time-scale update rule converge to a local nash equilibrium. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: pttpss://rrcceeedngggs.... newrips..eerppppccmmmm cttpssccppppppeeeeeeeeee Paperrdd..
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network, 2015.
URL https:: http:::/aaxxvv.ooo wttp:///.rrga/b////////oa// http//bas/111... 02531.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, 2020.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203. 2022.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions, 2021. URL https: :ettps://arxx //axxiv.. org/ 220.0.112000 05379.
Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh
Agarwal. V-star: Training verifiers for self-taught reasoners. arXiv preprint arXiv:2402.06457, 2024.
Andy L. Jones. Scaling scaling laws with board games, 2021. URL https://arxiv.ort abs/2100 03113.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models, 2023. URL https: https://arxiv..om www.aaawwi.agaaoeeaaaaaaa 22107220000000
Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects rlhf on Ilm generalisation and diversity, 2024. URL :ttppp:://axxae hbsrgg/bba//3// 06452.
Adam Lerer, Hengyuan Hu, Jakob Foerster, and Noam Brown. Improving policies via search in cooperative partially observable games, 2019, URL htttp:://axxiw.rrg//att 02318.
Xiang Lisa Li, John Thickstun, Ishaan Guirajani, Percy Liang, and Tatsunori B. Hashimoto. Diffusion-Im improves controllable text generation, 2022. URL ptsttpss//axxv.ooot /bbs/ 2205.14211
Ziniu Li, Congliang Chen, Tian Xu, Zeyu Qin, Jiancong Xiao, Ruoyu Sun, and Zhi- Zhi-Quan Luo. En- tropic distribution matching in supervised fine-tuning of lims: Less overfitting and better diversity, 2024. URL https: / arxiv. org/ abs /2008. 16673.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion language modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310.16834, 2023.
Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed, 2021. URL https:://// //rxivixivv owg/abs/ /2101.11111 02388.
Weijian Luo. A comprehensive survey knowledge distillation of diffusion models, 2023. URL https: /ttps////axvv.../ http://b///300/0 04262.
Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang. Diff- instruct: A universal approach for transferring knowledge from pre-trained diffusion models, 2024. URL hotttt:t/://arte wwwpiio.rrrrbbaa.be/ea /23llb/...2........ 22005 8455.
13

---

Published as a conference paper at ICLR 2025
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye. Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder. Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self- refine: Iterative refinement with self-feedback, 2023. URL https: htt.tt:////xxii..oooo //trxi..crgg//aaaeeeeeee / 2303. 17651.
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models, 2021.
Sajad Norouzi, Rasa Hosseinzadeh, Felipe Perez, and Maksims Volkovs. DIMS: Distilling multi- ple steps iterative non-autoregressive transformers for machine translation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 8538-8588 8553, Toronto, Canada, July 2023. Association for Computa- tional Linguistics. doi: N.L.8865/v112222 aindings-acl.542. URL http:: / org/2023. f nindi ngs- acl. .cl.542/..
OpenAl.
https://githi:n cttl:lccoo//p///////// com/openai/ spt-2-outpur-datasee 2019. Accessed: 2024-09-30.
Gpt-2
output
dataset.
Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data, 2024.
URL htt.h:tpp///.. //tp/aaxiivi/// org/ abs 2406 03736.
Arnaud Pannatier, Evann Courdier, and Francois Fleuret. Sigma-gpts: A new approach to autore- 2404/ 09562.
gressive models, 2024. URL
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoe Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context, 2016. URL https: attts:://axiiv http///xxvvvvoo..//// abs/1606 06031.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. Mauve: Measuring the gap between neural text and human text using diver- gence frontiers, 2021.
Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Lev- skaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling trans- former inference, 2022. URL https: /ttp/aaavvvv/v/////////e.//..avl org/ hbtp//bs/2220 22wps///11.....m 05102..
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents, 2022. URL https::t/////...... http::ss///xxiv.orggg.o 2204. 06125.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models, 2022. URL https: hettps://axxvv.ott /arxiv.org// abs/2112. 10752.
Bernardino Komera-Paredess Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. Mathematical discoveries from program search with large language models. Nature, 62597995):468 475, 2024.
14

---

Published as a conference paper at ICLR 2025
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Sal- imans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic rext-to-image dif- fusion models with deep language understanding, 2022. URL pestts://arxxx org/ abs// 2205. 11457.148..
Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models, 2024. URL https: www.aaaivvvi......... abs 2406. 07524.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models, 2022. URL 2ttp://aai.vv..r//a////......... 00512.
abs/2202.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion dis- tillation, 2023. URL https: org/ abs /bb1/ 17042.
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K. Titsias. Simplified and gen- eralized masked diffusion for discrete data, 2024. URL https: /aappi////xvvvoo// www.wiiwww..a..o..oooooooooooooe abs///bb//44 04329.
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game go with deep neural networks and tree search. Nature, 529(7587):484 489, Jan 2016. ISSN 1476-4687. doi: 10. 10.1038//ature1666
URL https: doi. org 1038 wtt.:/00////neeeeeeeeeee http//ntuuree666
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling IIm test-time compute optimally can be more effective than scaling model parameters, 2024. URL https: /aaxxiaxvvvv..//. abs /408 03314.
Jascha Sicl-Dicksttin, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei (eds.), Pro- ceedings of the 32nd International Conference on Machine Leurning, volume of Proceedings of Machine Learning Research, pp. 2256 2256-2265, Lille, France, 07-09 Jul 2015a. PMLR. URL html.
htpp:///ppcceedingg .wm. press vv3 htt7///shhaiicttiinmnm
Jascha Sohl-Dickstein, Eric A. Weiss, Ninu Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamies, 2015b. URL https: /ttp::///xxxvvoor/// abs/1503. 03585.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, 2022. URL https: //axxiv.///iiv////aa httgg//bs221000 02502.
Yang Song and Stefano Ermon. Generative modeling modellig by estimating gradients of the data distribution, 2020. URL https: //tp://////ivvaa/ae wwwp:iii.org//bb...o 1907_ 05600.
Yang Song. Jascha Sohl- Dicc-tiictt,,n, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations, 2021.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models, 2023. URL https: //arxiv..iiv///// org/ abs /2003 .1136301.633 619
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding, 2023. https: http::::///xxvvo//////o abs/ 2104. 09864.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023.
15

---

Published as a conference paper at ICLR 2025
Trieu H. Trinh, Yuhuai Wu, Quoe V. Le, He He, and Thang Luong. Solving olympiad ge- ometry without human demonstrations. Nature, 62577995):476-482. Jan 2024a. ISSN 1476-4687. doi: 10. 10.10388441566-02300044444 1118860023-0777--11
URL https://aci.org/// 10338/////
Trieu H Trinh, Yuhuai Wu, Quoe V Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625(7995):476-482 2024b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations, 2024. abs 22...3222200 08935.
URL wtthtttp//// /ettps//arxiv..ooe
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning large language models, 2023. URL https: / a/aaaiivv.orr..... abs 2201.11903.
Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. f-divergence minimization for sequence-level knowledge distillation, 2023, URL ht http//axxvvv///// org/abs /2077 .15190.
Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. An empirical analysis of compute-optimal inference for problem-solving with language models, 2024. URL https: www.aiiwwgg/.aa.aeaaaaooaaooee hblgg//bs/2ae 2000:/2240000..4....
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov. and Quoe V. Le. XInet: Generalized autoregressive pretraining for language understanding, 2020. URL https:
wwwpiiiv.rrgabbb/eeeeeee /0tp:/9966...00 08237.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023.
URL httpt::tp/// /atp:///axivvvv/ wex.iv.orr org/ abs 2305 0601.
Jiacheng Ye, Shansan Gong, Liheng Chen, Lin Zheng, Jiahui Gao, Han Shi, Chuan Wu, Xin Jiang, Zhenguo Li, Wei Bi, and Lingpeng Kong. Diffusion of thoughts: Chain-of offin-of-thoughh reasoning in diffusion language models, 2024. URL https: /wwpiva.iivvvv.////i///eaa/aaaia org/ abs/245 07754.
Lingxiao Zhao, Xueying Ding, Lijun Yu, and Leman Akoglu, Unified discrete diffusion for categor- ical data, 2024. URL https:/ http///axvvv//e//////ee//a.a org aww.abs/b/00 .00000072077300 03701.
Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming- Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. arXiv preprint arXiv:2409.000 249.000800 2024a.
Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. A reparameterized discrete diffusion model for text generation, 2024b. URL https:/ /ttp///aavvvvvaaa/eaa/ee.e.awav. org/ abtaaa////22.../ 05737.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models, 2018. URL https: /ttps:///axvvvco/o org// abs/1802 01886.
16

---

Published as a conference paper at ICLR 2025
Table Downstream evaluation results. We report the accuracy of GPT-2, the teacher and students after 7 rounds of SDTT. Distillation seems to minimally affect the downstream performance.
Task
GPTT-
Teacher
KLD student
MSE student
TVD student
ARC-Easy
43.81
40.91
40.57
40.45
40.32
ARC-Challenge
19.03
21.08
20.73
19.28
20.05
HellaSwag
28.92
30.50
29.65
29.10
29.18
MathQA
21.21
21.78
21.47
22.28
21.84
PIQA
62.89
59.74
59.85
58.11
58.16
WinoGrande
51.62
50.91
50.75
49.57
50.36
A ADDITIONAL ABLATION RESULTS
In this section, we show additional plots on the ablations we conducted. Because the KLD was best in retaining the performance on the LAMBADA dataset, used it in most the ablations. Hence, unless specified, the following experiments distill using the KLD.
Generative perplexity and precision of the floating-poit point operations. Zheng et al. (2024a) ob- served that low-precision sampling can be problematic in masked diffusion models, leading re- duced diversity and potentially misleading generative perplexity scores. As such, in addition to b6loat16, we try distilling (i.e. computing the backward KI.) and sampling using 64 bits precision. Overall, it does lead to higher generative perplexity, however the conclusions remain similar, as the final student achieves lower generative perplexity than GPT-2 with nucleus sampling (p=0.95) in 64 sampling steps, as shown in fig. 9.
Ablations the number of steps per round of SDTT In fig. 7 show the MAUVE perfor- mance. In fig. 8 we show the generative perplexity, and fig. 10, we show results on LAMBADA.
Ablation on the analytic sampler In fig. 11 we show results on LAMBADA, and on fig. 12 the MAUVE score.
Distilling more than steps at once In fig. 13, we show the generative perplexity.
Ablation on the optimizer state and exponential moving average of the weights In fig. 14 we show the generative perplexity when resetting the EMA and optimizer state. In fig. 14, compare the generative perplexity when resetting the optimizer state only, and when resetting the EMA state. Finally, in fig. 15, we show the MAUVE score.
Plots for scaled SDTT In fig. 16 we show the MAUVE score and in fig. 17, we show results on LAMBADA.
Conditional perplexity with TVD In fig. 18c, show the conditional perplexity (prompt cluded) on the small scale, for models trained for 1M steps. Empirically, the TVD performs worse than the KLD and MSE.
Measuring the diversity We evaluate the generation diversity using the self-BLEU score (Zhu et al., 2018). The self-BLEU score averages the BLEU score between one completion and the others. Therefore, when the sampling algorithm is deterministic, the self-BLEU score 1, and lower BLEU score denotes a more diverse set of samples. Formally, let X = { Fr be conditionally-genee -enerrted sequences, starting with the same prompt. The self-BLEU score can be computed as
self-BLEU = n
BLEU(I; XX (xx)).
(8)
17

---

Published as a conference paper at ICLR 2025
round
2 rounds
3 rounds
0.95
0.95
0.95
MAUVE
MAUVE
MAUVE
0.85
0.85
0.85
16
32
64
128
16
32
64
128
16
32
64
128
sampling steps
Num. sampling steps
Num. sampling steps
4 rounds
5 rounds
6 rounds
0.95
0.95
0.95
MAUVE
MAUVE
MAUVE
0.85
0.85
0.85
16
64
128
16
32
64
128
16
64
128
sampling steps
sampling steps
sampling steps
Teacher
10k
5k
(a) 10k vs 5k iter/round.
round
rounds
3 rounds
0.95
0.95
MAUVE
MAUVE
MAUVE
0.90
0.90
0.90
0.85
0.85
0.85
64
128
16
32
64
128
16
64
128
Num. sampling steps
Num. sampling steps
Num. sa sampling steps
4 rounds
rounds
6 rounds
MAUVE
MAUVE
MAUVE
0.90
0.90
0.90
0.85
0.85
0.85
32
64
128
16
32
64
128
16
32
64
128
Num. sampling steps
Num. sampling steps
Num, sampling steps
Teacher
10k
2.5k
(b) 10k vs 2.5k iter/round.
Figure 7: MAUVE performance with fewer steps per distillation round. It seems that using 5k or 2.5k distillation steps instead of 10k per round is detrimental the MAUVE performance.
We compute the BeLf score using 1000 prompts, as for MAUVE, and generate continua- tions per prompt. Figure 4a, fig. 19a and fig. 19b show the self-bleu score after distillation with the KLD, MSE and TVD objectives. Each objective only minimally decrease the diversity after distillation. Compared on-policy distillation of autoregressive models (Agarwal al., 2024), the decrease is marginal, as Agarwal et al. (2024) observe an increase of self-BLEU of the order of 10-20, demonstrating more significant decrease in diversity.
Decoding latency In addition to the results on the 1.3B scale, we report the latency for models with 169M, 424M, 863M, 3B and 8B parameters. We compute the latency with a batch size of 8 and 4. Figure 20 shows the latency with batch size of 8 and fig. 21 using batch size of 4. Figure 22 shows the trade-off between latency and perplexity. We measure the latency at the small model size and compare GPT-2 with the final students after 7 rounds of distillation.
Additional downstream evaluation results We show the performance of GPT-2, the teacher and distilled students additional downstream benchmarks from Gao et al. (2021) in table 1.
18

---

Published as a conference paper at ICLR 2025
160
Teacher
Teacher
140
SOIT 11 rounds)
140
SDFT munds)
SDIT 12 rounds)
SDFT rounds)
SPIT 13 munss)
SDTT mundst
120
SPIT (4 munts]
SDTT munds)
SDIT 15 munds]
SDTT munds)
100
SOIT (6 munds)
SDTT mundsl
SDIT IT rounds)
SDTT munds)
10k rounds
10% rounds
munds
Sc monnds
Re insined AR
Be traine AR
16
32
128
2.56
512
1024
16
32
1248
256
53.
1024
Numn samp Ing stops
samp shops
10k iter/round.
(b) 10k 2.5k iter/round.
Figure 8: Generative perplexity with fewer steps per distillation round. Using 5k or 2.5k steps per round yields slightly improved perplexity after the latest distillation rounds while being a slightly worse in intermediate ones.
160
140
rounds)
rounds)
(p=2.05)
munds)
120
100
Figure 9: Generative perplexity when distilling and sampling with 64 bits precision. Namely, sample from the teacher and students in float64, and compute the backward KI. in float64.
LAMBADA Accuracy
LAMBADA Perplexity
70
0.40
0.28
Perploity
0.30
30% Bertround
2.5% Remround
Waacher
Number rounds of SDIT
Figure 10: Performance LAMBADA when distilling with fewer steps per distillation round.
19

---

Published as a conference paper at ICLR 2025
Anyiycical sampling
140
Tescher
140
SOFT rounda)
SOFT rounda)
SDFT roundo)
SDFT roundi)
SOFT rounds)
100
SOFT
40-
(a) Generative perplexity.
LAMBADA Accuracy
LAMBADA Porpiexity
0.10
80
60-
0.25
40
Number
TAU
GP122 [smal))
See- Re-tramed AR
LAMBADA.
Figure 11: Generative perplexity and performance on the LAMBADA dataset when using the ana- lytical sampler. We find clear benefit over the ancestral sampler.
round
rounds
3 rounds
MAUVE
MAUVE
MAUVE
0.85
0.85
0.85
16
32
64
128
16
32
64
128
8
16
32
64
128
Num. sampling steps
Num. sampling steps
Num. sampling steps
4 rounds
rounds
6 rounds
0.95
0.95
MAUVE
MAUVE
MAUVE
0.85
0.85
0.85
16
32
64
128
16
32
64
128
16
32
64
128
Num. sampling steps
Num. sampling steps
Num. sampling steps
Teacher
Euler
Analytical
Figure 12: MAUVE performance when distilling using the ancestral sampler used by Lou et al. (2023). We find no clear benefit over the ancestral sampler.
20

---

Published as a conference paper at ICLR 2025
Distilling samping 110k train iters/ound)
sampling
Bers/round]
360
100
Reacher
Racher
SDIT munds)
rounds)
SDIT munds)
munds)
SDIT
munds)
120
SDIT
120
munds)
SDIT rounds
SOTT roundst
SDIT munds)
SOTT rounds)
SDIT munds)
munds)
80
steos
40
16
12.8
(a) 4 sleps.
(b) 4 steps and 15k iter/round.
Distiling 8 sampling steps at once (10k train iters/round)
140
SBTT
SBTT
mundi
SBIT
150-
SBIT
Perpecty
SBTT
steps
60-
Retraised
20
256
512
1024
Num sampling steps
(c) steps.
Figure 13: Trying distill more than 2 teacher steps once. (a): Distilling 4 steps at once. (b): Distilling 4 teacher sampling steps at once wit more training iterations per round (15k). (c): Distilling 8 sampling steps per iteration. Overall, distilling more than steps time to hurt performance. One could expect that distilling more steps at once would require longer rounds to train, hence tried growing the round to 15k steps per round, which hurt the performance of the student.
Reset optimizer state between rounds
Use EMA weights as teacher and reset optimizer state
Teacher
Teather
140
SDIT munds)
SD1T
140
SD1T munds)
munds)
SD1T rounds)
5D1T mundss
SDIT rounds)
SDTT rounds)
SDIT rounds)
SDTT rounds)
100
SDIT munte)
100
SDIT munds)
Koploxkt
SDIT mundss)
Reset
Rexet
trwnteeeddeeo Al
60
tuined AA
20
16
32
54
128
22.6
512
1024
15
32
12.0
25.
512
1024
Num. $ampping Staps
Num. Sampp Stops
(a) Resetting the optimizer state between rounds.
(b) Reset the optimizer state use EMA weights as teacher.
Figure 14: Generative perplexity when resetting optimizer EMA state between rounds of SDTT.
21

---

Published as a conference paper at ICLR 2025
round
rounds
3 rounds
MAJVE
MAUVE
MAUVE
0.90
0.90
0.90
0.85
0.85
0.85
64
128
16
32
64
128
16
32
64
128
Num. sampling steps
Num. sampling steps
Num. sampling steps
4 rounds
rounds
6 rounds
MAUVE
MAUVE
MAUVE
0.90
0.90
0.90
0.85
0.85
0.85
32
64
128
16
32
64
128
16
32
64
128
Num, sampling steps
Num. sampling steps
Num.
sa sampling steps
Teacher
Baseline
Reset optimizer state between rounds
(a) Resetting the optimizer state only.
round
2 rounds
3 rounds
0.95
0.95
0.95
MAUVE
MAUVE
MAUVE
0.90
0.90
0.90
0.85
0.85
0.85
Num. sampling steps
Num. sam sampling steps
Num. mapling steps
4 rounds
5 rounds
6 rounds
0.95
0.95
0.95
MAUVE
MAUVE
MAUVE
0.90
0.90
0.90
0.85
0.85
0.85
Num. sampling steps
Num. sampling steps
Num. sampling steps
Baseline
optimizer and rounds
(b) Reset the optimizer state use EMA of weights as teacher.
Figure 15: MAUVE performance when resetting optimizer EMA state between rounds of SDTT.
round
2 rounds
3 rounds
MAUVE
0.9
MAUVE
0.9
MAUVE
0.9
16
64
128
16
64
128
16
32
64
128
Num. sampling steps
Num. sampling steps
Num. Num. sampling steps
rounds
5 rounds
6 rounds
MAUVE
0.9
MAUVE
0.9
MAUVE
0.9
16
32
64
128
16
32
64
128
16
32
64
128
Num. sampling steps
Num. sampling steps
Num. sampling steps
Teacher
large
Figure 16: MAUVE performance of medium and large models pretrained for 400k steps. This experiment supports our claims that SDTT helps the final models approach the performance of the teacher with less sampling steps.
22

---

Published as a conference paper at ICLR 2025
LDM Accuracy
LAMBADA Perplexity
0.50
80
0.40
Forplentt
60
0.30
0.20
MOLN (sm)
HDLM (arge)
Teacher Re-izained Al
MDLM (md)
EP12 (small)
rounds SDDT
Accuracy.
(b) Perplexity.
Accuracy and perplexity on LAMBADA when scaling SDTT to larger models. All models are trained for 400k steps before distillation. On the small scale, training for 400k steps instead of 1M yields a weaker model. Interestingly, the perplexity can improve after distillation when the models are undertrained.
Figure 17:
Conditional perplexity (completion only)
perplexity (completion only)
100
SDIT 11 rounds
SOm 11 munntt
140
sam 13 rounds
sam 13 munddi
sam 15 roundul
som 15 monncal
5am 17 roundsl
SOIT 17 rouncal
120
CPT2 the ameai))
.... GPT2 uns ameau))
GPT2 ip-0 95)
GPT2 (p-0
100
100
Conditira
128
samppigg steps
samppig steps
(a) Perplexity of completions when distilling with the (b) Perplexity of completions when distilling with the KLD objective.
MSE objective.
Conditional perplexity (completion only)
150
munda
roundal rounda)
Candttie pepiedty
120
ameel) (p-3)
10
40
sampling
(c) Perplexity of completions when distilling with the TVD objective.
Figure 18: Conditional perplexity. Perplexity of the completions using GPT-2 large, excluding the prompt. SDTT with TVD performs worse. The final student distilled with KLD matches GPT-2 with nucleus sampling. Ground-truth continuations have perplexity ss 13.11.
23

---

Published as a conference paper at ICLR 2025
Steps stepe
stass
250
stups
stops
(reg)
stops
tropt
1p-0
steps
utupa
200
200
1.0
Cinditann
1000
X
51.5
52.0
32.5
53.0
53.5
$4.0
34.5
55.0
52.6
51.88
$2.0
52.2
52.4
52.8
52.8
53.0
Sel-9l T1
SeE-BLEU
(a) Distillation with the MSE loss.
(b) Distillation with the TVD loss.
Figure 19: Diversity of conditional generation (small scale). We measure the trade-off between quality and diversity using Self-BLEU (Zhu et al., 2018). Deterministic sampling yields score of The diversity minimally decreases after distillation.
2034
1034
2748
1024
(a) Small (169M.
(b) Medium (424M).
(c) Large (863M).
2/5
1027
275
312
2027
1000
125
(d) 1.3B.
(e) 3B.
(f) 8B.
Figure 20: Additional latency experiments with a batch size of 8.
24

---

Published as a conference paper at ICLR 2025
sate unall 214M
Pastcse (1208)
PhAT DONANS
10-110
130
Small (169M.
(h) Medium (424M).
(c) Large (863M).
Pidele 2a 1.30
Padelaee 28
2012
(d) 1.3B.
(e) 3B.
Figure 21: Additional latency experiments with batch size of 4.
X GPT2 (p-0.35) eWD KL MSE
TVD
55
23
Latency (ec.)
Figure 22: Perplexity vs wall-time latency (in seconds) for small models. We 16, 32, 64, 128 ans 256 decoding step for the diffusion models.
25

---

Published as a conference paper at ICLR 2025
Model size
small
medium
large
1.3B
3B
8B
# params
169M
424M
863M
1.3B
3B
8B
Num Layers
12
24
24
24
26
40
Embedding dim.
768
1024
1536
2048
3072
4096
Num. heads
12
16
16
32
32
32
Table 2: Hyperparameters of the diffusion models at different scales. All models use RoPE posi-
tional encoding (Su al., 2023).
B ADDITIONAL DETAILS ON THE DIVERGENCE MEASURES
In this work, teach the student to match the teacher targets x xheacher (Zzy L, m/k) generated by algo- rithm 1. We penalize the student deviating from the targets using one of three divergence measure: the Kullback-Leille Divergence (KLD), the Total Variation Distance (TVD), and the Mean-Square Sqqared Error (MSE). We now describe each of them.
B.I KULLBACK-LEIBLER DIVERGENCE
The Hollback-Leiblee Divergence (KLD) between two discrete distributions p and q defined on same finite sample space S2 is computed as
DEL (pll)) = Σ p(x) log p(x) q(x)
sE
The KLD has unique minimum when and q are equal, however the KLD is not symmetric, meaning that DxL(pl9g o DRL (4llp) in general. In this work, we train the student with the reverse KLD D Dz (pel Preacher). In the next paragraphs, we present differences between DKL (Preacher [pa] (forward KLD) and D (pell (reecher))rr) (reverse KLD).
The forward KLD is called zero-avoiding because if Ptarger()) is non-zero but (1)) is close to zero, then Pharget (xr Preyula) will be large. To minimize the forward KLD, Po will try to assign non-zero probability to all points where Purget is non-zero.
The Forward KLD
Po(z)
The Reverse KLD
The reverse KLD is called zero-foreing because if Parger (1)) close zero but (1( is not, pa pz)) pala) will be large. To minimize the reverse KLD, pe will try to assign zero Piege probability points where Praget is close zero.
B.2 TOTAL VARIATION DISTANCE
The total variation distance (TVD) is metric used to compare two probability distributions. For two discrete probability distributions p and yg defined on the same finite sample space Ω the TVD computed as:
drv (p. q)
Σ p(xx) g(x)).
(10)
The factor of 1/2 ensures that the TVD ranges between 0 and 1, where drv (p. 0 if and only
B.3 MEAN- ERROR
Unlike the Kullback-Leihhe divergence (KLD) and Total Variation Distance (TVD), the MSE can be used to compare any scalar quantities, not just probability distributions. For numerical stability, we compute the MSE in log space:
MSE(p.g)
(log ((x) log 2
(11)
Ω
ZEB
26

---

Published as a conference paper at ICLR 2025
B.4 X² DIVERGENCE
The x² divergence can be used to compare two probability distributions. For two discrete probability distributions p and y defined the same sample space Qm the x² divergence is computed
(((x) (())) ((x)) -11
(x=(p, = Σ ((x)) ZE
E mEn
g(x) (o(x) q(x))2
(12)
As such, we see that the x² divergence is related to the MSE. Note that when using the MSE for distillation, we penalize the eтrоr log space, while the x2 penalizes error in probability space. Additionally, the MSE uses a uniform weight factor [ for each term of the sum, while the x² divergence uses a weight of ((x))
C IMPLEMENTATION DETAILS
Architecture To compare with Sahoo et al. (2024), we trained the diffusion models using their code and pre-processing steps the OpenWebText dataset (Gokaslan & Cohen, 2019). As Sahoo et al. (2024), our models are not conditioned on the noise level. Nonetheless, Sahoo et al. (2024) kept the architecture of Lou al. (2023) unchanged and makes the model unconditional by feeding it zero tensor instead of the noise level. Removing the adaptive layers could improve the sampling speed further, but we avoided modifying the architecture prevent potential problems. See table for the hyperparameters of our models.
D TEXT EXAMPLES
We include non-cherry picked text generated from the small distilled model with KLD loss from the last round of distillation via unconditional sampling with varying number of steps. We show the first 512 tokens to so that the text fits on one page. Remember that those models are small and not fine-tuned for text quality. They can also start generating in the middle sentences, since they are trained on a concatenated corpus of documents.
27

---

Published as a conference paper at ICLR 2025
Text generated with steps (1/3)
invite to the gathering, because he was invited in 2008, probably on regular basis s thereafter. But to become scientist, to verify those cred veracity is important, inlamali said.
CNN is thus creating a monster that has supporting cascade of other grand jury investigations, he said.
"In the case of Mr. Eliaschis, wrote in today to number of everyone involved consideration of this matter; these people are invited, named and considered ccommiten tted to the process and trust of the Nation, he said.
There have been no complaints or formal complaints and this longer be CNN's standard and indepth
will directly coverage. <oogge<</endoff </endoftext /bbecause my office cherish diversity this is the approach we have come across. their subscription model is great, and right now there are folks our office that want to help to promote diversity. so re locking forward to hearing those responses from them. although we realize it is different place than we run it. but outside of this, I think we've never had a lot of conversations (especially here) about community-based leadership being happier than market-based leadership: the leader is fantastic, the person is valuable and talented.in the UK things don't that way. it has leaderless culture which has not been well with hierarchical planning and organizing process. we have a very specific image of this kind of organization. but one such qualities is the image of someone accomplished like everybody else does, which interests us as the talks of one our public figures
part of what we'd love to do here endeavors. recognize that most of the work here, we' re in the midst of the first day of the interview (which Prof. Garry had posted to the blog). Garry was kind enough to come participate in the interview and perusing on his and the next few competitors, in order get valuable feedback. wanted to feel enthusiastic about the process, eager to share feedback, and expect to have very professional experience.
social impact
well as conducting
but, broadly speaking, more than lot of the things that we struggle with the capacity to report, we just made a draft, and then got the post called. there is one example of things throughout the draft that made me the most-- The media is a small part in Far Left.
it is really important to have a relationship with your employer. At a high
28

---

Published as a conference paper at ICLR 2025
Text generated with steps (2/3)
kids not being in our schools as a result. The kids put their Camilies in Florida schools in this district, this is Georgia school, and not only do they have enough time to work for Florida firm, it's not desirable for kids to be in Florida school anymore."
The brothers turned to public education and the governors quickly asked them if they wanted to. Bonding Aid then was contacted once asked for special order from the administration setting aside $524, 000, but they were denied despite the requestBy Bill Othelle, according to government spokesman. The brothers wrote letters to the governors numerous times claiming the information was false, including one letter, which suggested that they funnel $2,000 to the Slothouse clearing House schools through the University of Florida. However, one of the brothers, Chris Yates, told Bloomberg News that he still was shocked and horrified by the correspondence, saying, "Obviously, felt like a coward uncomfortable situation."
in this very
Instead however, Yates said, he reacted very much like he pissed off at one of his favorite politicians, Bush.
Florida's two Republican leaders have strained relations, in particular with recent governors Doug Ducey, Republican and Jeb Bush, a Republican, addressing his concerns. "I do not think the other leaders will do this, but do have to work to make sure that is how we have to do it, something that's s something that we need to be doing, and that there will be always a need for better quality education, too," " Bush also said. "We not want Florida to stop funding education and essentially contradict the fact that what has provenance in this planet is that killing was at least 10 percentage and many people got dead. We I beginning to have problem with that and think that's the first where re going to eventually share that information with the public, and agree to the efforts that we are engaged in and also Haley and I] are ultimately going We will have the heavy lifting to do whatever we do. I think this is irresponsible but also that we are not going to start having real conversation because we' ve got lot to do, but that was eye opener to us."
know, for sure, when wer
feel in order to deal with that have to
have to provide that
When asked on Tuesday if was dazed and that he regretted attending the meeting
29

---

Published as a conference paper at ICLR 2025
Text generated with steps (3/3)
was subjected to will then acidize in the form of foam. The pH and pitch of foam create a, too substantial gap between an egg and tissue, which will drive it to accumulate ate faster, and can thus cause irritation to the sensitive parts of the body. It's penalties are well as analgesia, and shortens chances of learning how the material works.
Mr. Segal, who was involved in the study, did not acknowledge the simitations of the study to treat his own specifi facllly painful dental fracture, but added that it succeeds in all aspects of the process. "This is the first time that it might make significant contribution to improving dental health. Until then we will have to get better at adjusting what have changed to make sure that it is effective. "< ennooftex |>Yet, in June 2013, Laquan Phillips, 20, promising medical student Jackson State University School of Medicine and the son of cyclist Philando Castile, killed two or four weeks earlier, was forced to give a the dozens of officers. officer just about three seconds to pull up on the vehicle trying to lock the black car into silos, and Fewell, the neighborhood managed by the officer who arrested Phillips, was refusing to go all the way. The officers were concerned about the direction of bullets so that they could hit casting bit.
the
While police believed the bullet hit and belief that had been consistent since police had been in deep denial, it failed to hit the man on shot into
the left side
right side of the table. When you interjected the man's s first body, the face mostly rolled down the throat crushing moment of motherhood and last sweat for a father of two and thankfully the other one was stopped in his tracks.
"We believe him, said John Milliken, police officer at the time of his shooting, who could not confirm other deaths but said that 0li firing gun. Others would say it was the product of head trauma.
The policeants in Roswell could rely on variety of lethal weapons, they said, including anmunition that police had accessed. I want to thank the people for the first- aid the people for
kit for the family, and the people Justice Jesse James also, at hand members of the Black community the 71st.
An anonymous person was having a phone conversation with the Buckeyes's interim president, who was to take part in the participants of
30

---

Published as a conference paper at ICLR 2025
Text generated with steps (1/3)
Wilkins was he committed the acts of vandalism as juvenile
Woodward said he had "since confessed the crime based on information and an explanation for what he did. But police spokesman told the news outlet that investigators work for the government and it is the responsibility of whether it is the individual with knowledge of the crime or that should be punished as well as those involved with the system.
"Liberals withhold confidential information from the public on the basis of any reason or whether such information is public interest, the spokesman said condition of the anonymity because of the investigation.
Fox News reporter Brian Stelter, reporting the agency's s look the alleged Hanmond scheme, took high-profile line on political campaigning in Russia and current affairs on Fox News Channel's s propaganda channel, The Which?, " for instance.
The Rasmussen Reports cited concerns of spike in voter fraud, citing the divide between the Democrats and voters many of whom Republicans outside the party vowed to lose in the election.
The Washington-based advocacy group We Are Ameri car, which publishes figures from the polls, said the report was based on the promises of conservative news outlets, independent organization, and the use of pollsoddy reporting.
"The communication and reporting of American media has been critical," We Are Russia said in a statement following the report.
The United States media has been biased against the election, which Russian officials linked to hacking on Democratic computers and other promotional efforts that seem aimed at raising security fears in Moscow and toppling many of the Kremlin's s market allies in the U.S, which is moving closer to winning the election before referendum begin taking place.
Russian officials have attributed their bias opinion polls toward the 2016 elections to uptick in unemployment statistics, with figures they' re publishing being forecast on the
of their presidential elections.
Russian officials say they would like to prevent fraud, and polls show that they may have aggressive on potentially maintaining narrow five-point lead.
The Rasmussen poll report came days after the state Attorney General filed a lawsuit in general against the Rasmussen Reports' report, and asked the state to look to the opinion polls and determing, the Rasmussen poll did good job."
Democrats announced plans to use a federal court case to

---

Published as a conference paper at ICLR 2025
Text generated with steps (2/3)
Sherman was a city historian for 28 years before 1988 and former relations adviser spoke with more than dozen Asheville staff members during an interview outside the hotel early Monday morning. In brief interview, Sherman gave employees an overview of what they seerved from the two Highlanders electric paper and pictures of the cut and logings, perhaps sweeter than the compensation Yancey and Davis did compensation settlement for.
including a handwritten piece
Photo: Courtesy U.S. BAG via Jan. 30; The documents provided in the report stem from a link to a minor change in the law, according to the report.
The 1aw officially establishes enforcement and oversight process (PDF) for transferring payments to the other committee, or the City or State Board, that has the mandate to require the cash transfers from the other side
The legislation, upon meeting each committee, instead directs the legislature to set its own financial policy in the other committee;
The method suggests payments to the other committee can form the legislature's
committee pass enforcement and oversight legislation, and that each of the committees may vote motions for resolution and give the public ic a vote or motion to approve legislation by the Joint Finance Committee.
That, the city and state will follew the same process as New Hampshire City and Lodging Act. However, in the Nov. election the Legislature prepares to decide whether to modify the rules and procedures surrounding such Senate bill, so it would be unclear how long the interim Legislature policy to be implemented once it's adoption is in place.
The terms of the report are not necessarily overarching, some of the other ways are specifically saying that proposed legislation that would transfer payments to the other committee must be amended to include a general bill before it would be recognized as legislative body.
in Khan at editors@time.com
Twitter:
independent party New Greens has urged Parliament to declare that there was no reason for the war. Many Australians who believe that it would be better to their children going to war are handed local police officers the chance only for training next week. The leader of South's s Republic of the African state, Chri stopher Robertson, said that the government's refusal to give any status to the force is the worst for the Liberal party,
a statement Mr Robertson said it is important to keep a police force on ground as it sees fit. This has caused panic among all 16 states, in particulag the growing
Australia's s history.

---

Published as a conference paper at ICLR 2025
Text generated with steps (3/3)
the police, media and public
International Council of the Community of Europe (TECE) is preparing ng the next steps, being taken Monday that could the potential to make firm changes to Britain and the EU, including the introduction and procedures on trans financial transactions from ordinazy individuals private corporations.
modern regulations
However, senior mainstream EU members told the Telegraph that some of the changes in the legislation at the moment are not clearly relevant to that role" for regulatory oversight, as well legal procedures, for the provision of business and financial industry, social, welfare, police, promotions, public health and economic activities and finally, to give the EU the right to make the legal and regulatory decision that is needed by the wider market.
A senior Council official in the UK, speaking on condition of anonymity for the sensitivity the meeting with his staff, said there had been little progress in negotiation negotiations and how to implement the changes because the legislation
still on the way. The senior official said both countries had worked fully with each other so the legislation could meet the basic existing EU regulations.
"We agree then that we will need to amend the legislati on in a way that is appropriate for EU law in the parts the UK. So review and consultation is something we are looking very closely to to ensure that our new amendments to the legislation give additional leadership and framework
to verify we can implement those measures and will provide appropriate support of expertise to the UK in relation to getting them.
However, the official said could take some time still to implement the legislation and that any changes put in place require a complete clean review of the wording of legislation, and part with EU institutions in that way.
understand the process that is underway on how to implement those measures and implementing those measures is going rapidly, so we can't yet start assess the situation," he said.
The Council has agreed January 25 to pressure the UK government on the decision to leave the EU to good effect
"The previous government had helped reform the law around the UK and contributed to that change, said Matthew Kennedy official for the Council.
The Council will work on the new amendments as part of two the terms of engagement with the British government, signed by David Cameron.
"They are part of a broader A and to do so has not been determined, so the Prime Minister and those

---

Published as a conference paper at ICLR 2025
Text generated with steps (1/3)
create big gas storage wells locally. BP has proposed investing billions of dollars to build new big storage wells, which will raise gas production to new levels of production.
The research has been co co- sponsored by Congress lobbying oppose sil and gas drilling (FAPL), but the administration has yet to outline how far oil and gas drilling and the Eagle Ford expansion will take its new forms, with new risks rising.
SUS energy industry officials say changes in the measures ordered and new funding levels have been little bearing on oil production, capped in about 60% last year, but some proposals are still receiving orders still others have yet to be overcome by the Department of Energy to produce package
proposed Dakota Access oil pipeline to the West in the West Coast.
The distinction between fossil fuel and gas is also blurred on energy measures that have been laid out to Congress about year; most are now in the process to agree on the proposed Keystone XL pipeline, and the Dakota Access pipeline.
The US energy market is fast moving, the focus of the Obama administration has been to push jobs to the US and to encourage Americans to have better financial prospects for creating their new jobs
Gans to set low-oarbon renewable energy targets will soon launch at the White House. Since 2009, infrastructure projects have established several significant initiatives restructuring the coal sector in the West of Europe, reducing the tariffs renewable energy in Britain, and implementing zero-carbon policies in South Africa, despite high difficulties in succeeding in implementing EU emissions standards.
The infrastructure initiatives, announced and unveiled by the president on Wednesday, are focused on the aim of finding effective ways to build on competitiveness at long-term scales and to help around the world create a sustainable energy portfolio in the areas needed to mitigate the risks local economies.
But institutions as big and large as central banks infrastructure plans to support renewables have been especially critical in recent years. A number of EU states such as Finland and Sweden were taking initiative with their plans. The states, which will also launch a number of the other initiatives ceremony, said that "the energy policy environment will receive high profile development in the UK and the economy of the EU in mest of them".
the White House
The far-reaching strategy has been to place low-carbon targets the UK's renewable energy sector, of course, with push of the private sector much of which is being led by governments and businesses to scale their renewable energy

---

Published as a conference paper at ICLR 2025
Text generated with steps (2/3)
exchange Bitcoin, it should be able to deliver such services. Rasically, a service that hinges Coinbase make payments for those bitcoin users who use this unique means of purchasing and storing bitcoin, and that it's able to reach retailers as wide as possible, which makes it so attractive for merchants for transaction speeds. Once they've taken the step of overcoming a long wait for accept
an WePay service, they're permitting merchants Coinbase to begin. What explains the delay is that even if they decide moving forward with an offer to pay for ma maintainion the price of bitcoin and the impressive growth of bitcoin, it remains a high barrier and additional cost for some users. What's sure for certain is that merchants will accept the service, but that it method of payment.
will deliver version of bitcoin
They also plan allow Coinbase to maintain control of the new address, to ensure the security of the integrity of Coinbase, as well to further expand the service available to Mac users. And besides, Coinbase is still investing in BitPay, a leading bitcoin exchange offering for free an alternative to fiat value, which would be a perfect fit for a marketplace for all that bitcoin in a heavily regulated world.
We have reached to WePay Bitcoin for response and for their views on our WePay service.< ///oo///ext////////// |>How do you describe the typical user of mari Your questions are always open to the cipscoop. net email team. Today is Day Cannabis, an event celebrating the global outside of the marijuana industry. Let us know your favorite questions.
K: Let's s talk about the for-profit group that supports mari juana, Gives of Health, which established the first nationwide standard price system for marijuana. Can you name name?
its first statement, Gives of Health defined itself committed to supporting a national free market system of marijuana, promoting policies to reduce the supply of legal prescription drugs, reduce tobacco prices, reduce wasteful government spending, and taxing and regulating marijuana. We believe marijuana marketbased tax system is good for our public health and for the economy by helping people circumvent drug laws and purchase marijuana, reducing tax costs associated with prescription analgesics and avoiding tax evasion.
K: Leon and several advocacy groups that try to crack down marijuana, as the Gang of Eight, introduced bills to make cannabis and medical marijuana for adults in more than dozen states. What is that to say about addressing lot of the problems?
35

---

Published as a conference paper at ICLR 2025
Text generated with steps (3/3)
by the military. The army (which elected its President) saw the most important coup in its history.
The broader American Catholic Church claim that one of the first indications of the twentieth century thought of liberated Christians" not earlier than July, 1960 was that the Protestant church would be leveled within their territory. Protestant mainline hierarchies in the late 1950, and early 1960, which had witnessed the life of Calvinist Protestant movement, built the Protestant image of new creed that preached tolerance and loyalty to the Church, desperately looking for a new denomination. Although the political movements had been building up for a long time, there was a Protestant political movement that thwarted the church and introduced sense of impending doom specifically to certain movement, belief that while the Reagan government's attack the Church made it Americans personally and at the center of society, and eventually dissolved and destroyed Protestant institutions. The church, with the exception of the United Methodist Episcopal Church (MCCA), heavily engaged in Latin American, Australian, and indigenous communities, is recognized as having one of the first Protestant hegemonies in the United States, following the formation of St. Mary Catholic Church in Louiaville in 1963.
The ci ccumstancess of the resurgence of radicalism have led to a political movement that introduced a distinctly Protestant identity and spread feelings of insecurity and alienation that today plague converts to Christianity, This unusually widespread within the broader American Catholic Church. Often associated with radical movements, such trends may signal a significart cant shift in the Catholic Church from the separation techniques throughout the history of the United States Church. The Southern Baptist Convention Conference, established its Center for the Restoration of Christ of God, in America. Later, it established Adventist church in Texas. The Maclean Church professes and operates another Adventist church, the Church of America, which began forming one the most centrist and largest denominations in the country in 1960. a result this, the largest denomination would soon be Holy Cross Church in North Carolina. Nick Sheymia, president
Faith in America, which maintaing a hard line between conservative and fundamentaliss Christianity on the left, told the New Apostolic Arrangement Network that the church has always "walked in the overall shadow of the much broader church that attempted its rule and control of the nation."
01. Howard, an educator and the author of the American Catholic Church: A Study in Truth, said, "The church seems inept at the end of the time. But to most Americans
it
36

---

Published as a conference paper at ICLR 2025
Text generated with 128 steps (1/3)
most countries, but in the UK, where computers are run by the many as 15 people involved is complicated Many experts think that governments may need to overhaul their machines because they influence because budgets are stretched too thin. In such complicated situaticn, they say, public sector efforts to move ahead in technology, particularly climate change would have to put in real action.
unable to retain their
Another major concern was a government effort to undermine the private sector in the construction sector and financial services industry. Ben Gove, chief executive of the House of Commons, said the government needed to address the role of the public service sector, and made explicit that the role of the labour force has become effective vehicle to cut income from the poorest working people the richest.
20%: the economic consequences mass industrialisation Read more
In joint statement with several coalition partners, the Conservative party also called for changes in the public sector to ensure the return on public investment, as the budget. Labour pushed back against some reforms, which involve modernising the benefits system for older people to make it easier for businesses to compete to win jobs. However, these reforms critics say apply only the people who need them.
well
Ben Smith, the shadow environment minister with a record of climate change on the Labour side, expressed "general disappointment that this report shows significant shortcomings". In practice, the Commons report is not meant to advocate for such reforms. Labour would have hoped to represent a party with a strong alliance with existing policies and government that has committed to investing in more workers overall.
Labour responded by saying: "We' re pleased the MPs voted to put in place further policies to help the workforce and benefit everyone."
Labour paid for the narrow, decisive vote with a single vote.
They say the findings show the government has missed the importance of its work for helping the working class within the economy, so that the enormous impact automation benefits is linked with the exploitation and marginalisation of the working class.
The government has failed to recognise the fact that energy profits have been rising. Since 2008, bosses have tripled the ranks of the British electricity and gas companies, helping their annual profits rise by 3. 3.1bn previous decade.
the
The biggest companies have made more profits over the decade before. This has created an incentive to make for more secure and sustainable energy security. But as well as bosses, they must collectively recognise what is happening,

---

Published as a conference paper at ICLR 2025
Text generated with 128 steps (2/3)
many really exciting, non- independent artists, and all these indie bands, jazz and folk-rock, and rock-and-roll jazz bands, and you see a let of new artists making these kinds of fantastic music. What do you think about these artists in Seattle?
M: I my whole 1ife have never really wanted to do anything outside indie music, so wanted go way mainstream. It 's much of role model to try and spread the word. I think everyone gets invested in it so that people can use their music, but even back then, nobody had tried to bring these kinds of music to that community, and saw that it was good way you should find someone who's going to do that in the right way. This city can be an extremely helpful tool for developers to come up with the right media resources to develop quality music, and now there are many indie studios over the country just attracting artists, and I think it's very helpful, and so there are a lot more open tools available.
S. Norman, what probably the biggest impediment you put terms of the music scene, and what's really important for you and I to bring your music to audiences in Portland or New York?
:: I would say the only really obstacle to me, it's the fact that I have to work with people in Seattle, and I try to involve them in it much as possible because of where they actually come from. Many of the people we were talking are basically part timers in the industry, and that I and I have had to work with, so many people are also part of the community, among all of the audiences we've ve been meeting. And I really came out of this scene with lot of excitement about it. And because grew up growing up here in Seattle that was crowded and full of loud noise, it had just turned out to this place with so many people in this scene, that found myself moving more to the music that I come from, at least to me, in a small, vibrant, and growing community, and it makes that sound more accessible to everyone. And what is really so encouraging is that while most of the people that were talking to at events, they are also taking part in actions within the community, supporting our music through connections and sharing it with each other in our community and the city. Right now, there's s really not a
38

---

Published as a conference paper at ICLR 2025
Text generated with 128 steps (3/3)
to write this because it blew me out of my seat. I'vv ve been getting few negative reviews in the past couple of days and I realize that people are much harder than ever to read them. Well, everything I've heard to this is overwhelming, and I'm happy to say that there am so much that know that has really blown me out of my seat that this situation is much worse than anything I have told you. want to be very direct to you, regarding even some of the things that you have heard. Feel free to let me know what you feel like whether have a personal concern or sincere thing you would like to hear about this topic.
The response to this story is overwhelming support and love! I just want to reach out to my Facebook page to tell you what I wish to do for you and really appreciate. I don' want to talk to anyone just to tell you how I'm very sorry. I want to tell you about all the things that I've been through so far and what I've done for you. I'm really sorry for everything that I have done to this community, and I am really appreciative for my support. Not to even go into detail the issue too much, please do head over to the next page in the post.
ultimately don't t want to tell you that this is your biggest issue I don't assume that it help you much. Instead, I want to say that you should just get some things out of the way. I've encountered many other people just coming forward to this, and the fear of losing anything that I try say about these issues can only interfere with such process. However, now I like being out there trying to help the world, and am so thankful that the community might help me too. Although I try to read this as about once a day, just appreciate the amount of your feedback and hopefully I can build more positive ones to read by the evening.
Lastly, I want to inform everyone regarding this topic in case that other readers want to buy my blog here.... hopefully when I write something to my readers in order help them out, like your support. every reader so that I can talk to other readers for any information that can help me, to further tell my story.
want support
C: still have the horrible feeling that
39


Reviews:
[{"review_id": "9481kAVA2v", "reviewer": "Reviewer_LQck", "summary": "The paper introduces a novel distillation method for discrete diffusion models: SDTT (Self distillation through time), which significantly reduces inference steps, leading to a generation process that is up to 8 times faster than AR models with KV-caching. They ablate with different distillation alternatives. They evaluate the effectiveness of MDLMs with model sizes up to 860M parameters under unconditional generation, conditional generation, and downstream LAMBADA benchmark.", "questions": "- Does the self-distillation process affect the generation diversity? There is an ongoing discussion regarding potential model collapse when AI models are trained on recursively generated data, which may be relevant here.\n- How was the decision made to set the number of rounds to 7? \n- The paper focuses on discrete diffusion models and generation speed for reasoning tasks but fails to cite these related works: \n(1) A Reparameterized Discrete Diffusion Model for Text Generation;\n(2) Diffusion of Thoughts: Chain-of-Thought Reasoning in Diffusion Language Models.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Improved Generation Quality and Speed: The paper achieves remarkable improvements in both generation quality and efficiency, with fewer decoding steps required. This is a significant advancement for the field.\n- Clarity of Writing: The paper is well-written, with clear explanations of the methodologies and results, making it accessible to a wide audience.", "weaknesses": "- Insufficient Validation on Generation Speed: While generation speed is a key advantage highlighted in the paper, the validation seems inadequate in Section 4.4 and Figure 2b. The experimental settings lack clarity. For instance, it's not clear if the reported 8x speedup is based on 32 steps, 1.3B, and a batch size of 8. Additionally, the paper does not clarify whether the quality (generation perplexity) is comparable between 1.3B DLM and AR models.\n- Impact of Batch Size and Model Size: There is a need for further validation on how batch size affects memory usage and decoding speed for both diffusion and AR models. Similarly, the paper should explore how model size impacts generation speed.\n- Effect of Flash-Attention: The paper does not address whether the advantage of DLMs over AR models persists when flash-attention is employed.\n- More downstream tasks such as math reasoning are encouraged.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HPQQ0LIyEv", "reviewer": "Reviewer_7pDo", "summary": "This paper proposes self-distillation through time to speed up sampling from text diffusion models. The method trains student reconstruction models to predict ahead in time, given a teacher model. This method can be applied iteratively, using students to train even coarser students. Experiments demonstrate that the method results in achieving a given generative perplexity at fewer function evaluations.", "questions": "1. Notation nit: In the discrete case, with fully-factored reverse model, $p(z_s \\mid z_t) = \\sum_x p(z_s \\mid x)p(x \\mid z_t)$ can be computed efficiently (see D3PM) and (also in the discrete, fully-factored case) $x_\\theta(z_t, t) = p(x \\mid z_t)$.\n2. Notation nit: Equation 3 can be written as $\\log p(x \\mid z_t)$, rather than writing a probability as the inner-product of the mean-parameters with a one-hot vector.\n3. Writing: Please move iterated SDTT to the methods section and formally describe it with one equation and a few sentences. Something like a telescoping sum of divergences. Be sure to introduce terminology like \"rounds\" formally and do not use both \"times\" and \"rounds\" in order to minimize terms.\n4. Writing: Are figure 2 and 3 referred to in the body of the text?\n5. Writing: Notation such as `dt` should be formally defined in the methods section. Could you explain what `dt` is? Is it $\\Delta$? Could you also define $\\Delta$? In MDLM, $\\Delta$ refers to a simplex rather than time-step or increment.\n6. Writing: In general, \"steps\" can be ambiguous and can refer to sampling steps, decoding steps, or training steps. Please be clear which one is being used, for example in the beginning of section 4.1 it should be training steps. Also, likely sampling = decoding steps.\n7. Question: In this setting, is MSE roughly chi-square divergence? It's surprising that using MSE results in better generative perplexity. Any thoughts on why?\n8. Experiment: Could you also report the conditional likelihoods / perplexity alongside the MAUVE scores for conditional generation? It's not completely clear what MAUVE measures [1].\n9. Question: It's very interesting that distilling more than 2 steps at a time, as well as more training steps per round hurt the student (Figure 11). \n\n10: Figure request and section 4.4: In section 4.4, the following claim is made: \"We successfully reproduce the results of Deschenaux & Gulcehre (2024), which showed a 4x improvement when sampling with 32 steps, only this time, we retain the text quality because of SDTT.\" Please add a figure that clearly shows this. A figure with latency vs generative perplexity is extremely important for this paper. Additionally, Section 4.4 is likely the most important experimental section, and should be the first result discussed.\n\n[1] Pimentel, Tiago, Clara Meister and Ryan Cotterell. “On the Usefulness of Embeddings, Clusters and Strings for Text Generator Evaluation.” ICLR 2023.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The method is simple and intuitive. The results for generative perplexity and number of function evaluations look promising. However, wall-clock time / latency is most important.", "weaknesses": "The primary weakness is presentation and writing. The abstract claims that the generates tokens 8x faster than an AR baseline with KV-caching. This must be presented in a figure as early as possible (see comment 10). See comments below for more suggestions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mhXar5n2Cw", "reviewer": "Reviewer_ijRT", "summary": "In this work, the authors propose Self-Distillation Through Time (SDTT), a training strategy for decreasing the number of sampling steps required during discrete text diffusion. Given a pretrained discrete diffusion model, the method works by using the token distribution of the base model at later denoising timesteps as the target for a student network at an early denoising timestep. The student model is initialized to be the same as the base diffusion model. The authors demonstrate that through multiple rounds of SDTT they can sample with 32x fewer steps while maintaining quality. They also compare to standard auto-regressive decoding, finding that they can achieve the same perplexity with an 8x reduction in wall clock time during generation.", "questions": "* It would be interesting to look at why more SDTT rounds work but distilling a larger number of steps in a single round doesn’t. Did you investigate progressively growing the distillation step size throughout training?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* The SDTT methodology appears to significantly improve the decoding speed of discrete diffusion models, and the authors provide evidence of this over a large set of experiments. Compressing discrete diffusion sampling to this degree seems like an important contribution that will be used by future text diffusion works.\n\n* The paper conducts a thorough set of ablations justifying key design choices such as the particular divergence measure and the number of steps compressed per SDTT round.\n\n* The authors demonstrate that text diffusion models can scale up to 860 million parameters and promise to release said model which is a valuable contribution for future research on text-based diffusion models.", "weaknesses": "* Performing multiple rounds of SDTT seems integral to the empirical success the paper has but the detail that SDTT is performed over multiple rounds seems to be introduced in the experiments section. It would help the clarity of the paper if this technique was introduced as a core part of the algorithm.\n\n* As mentioned by the authors, previous work has explored distilling multiple diffusion steps in the continuous diffusion setting ([1], [2]). While the technique in SDTT has slight differences and is applied to the discrete diffusion setting, its similarity to [2] in particular may limit the novelty of SDTT.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NLugMXJ9DN", "reviewer": "Reviewer_fzmS", "summary": "This paper explores progressive distillation, a technique commonly used in continuous image diffusion models, in the context of discrete diffusion language models. Starting with a trained discrete diffusion model that might take many denoising steps to generate a text sequence, the proposed framework iteratively distills the slower teacher into a faster student diffusion model, effectively halving the number of sampling steps in each distillation iteration. The distillation process involves collecting predicted logits for all tokens from the teacher model and then training the student diffusion model to match the teacher’s logit predictions according to a defined divergence measure. This self-distillation yields a significantly faster diffusion model, capable of generating text with up to 32 times fewer sampling steps, all while improving generation quality.", "questions": "1. Typos:\n    1. Line 96: `(MDLM.)` → `(MDLM).`\n    2. Line 785: `the the MAUVE` → `the MAUVE`\n    3. Line 785 `resuts` → `results`\n2. Clarifications on contributions:\n    1. In Line 79, “Unlike many distillation methods for continuous diffusion models, SDTT does not rely on deterministic mappings such as DDIM” might require further elaboration. Given that discrete diffusion models inherently lack a clear connection to deterministic processes akin to DDIM, what is the implication of this distinction? Specifically, how does the absence of deterministic mappings influence the theoretical foundations or empirical performance of SDTT?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This work makes a valuable contribution by extending progressive distillation techniques to diffusion language models, achieving much faster generation with improved generation quality.\n- This work also includes extensive ablation experiments that carefully examine each design component of the distillation framework, demonstrating the solid empirical performance of SDTT. These results offer useful insights for future advancements in discrete diffusion language models.", "weaknesses": "- The paper appears to be written hastily and lacks clarity and organization. Beyond typos and formatting issues (examples below), key concepts are introduced ambiguously. For instance, only in the experiments section is it clarified that SDTT consists of multiple distillation rounds rather than a direct distillation; and the progressive nature of SDTT is not immediately evident in the main text or Algorithms 1 and 2, becoming clear only in experimental results. In addition, the number of distillation iterations can only be inferred indirectly from `dt`, which is quite unintuitive and causes more confusion. It would greatly improve understanding by emphasizing the progressive formulation early in the methods section, and annotating the number of distillation rounds explicitly instead of relying on `dt=1/k` .\n- Inconsistent use of x-axis labels in figures also adds to the confusion. Some figures, such as Figures 1(a), 1(b), 8, 9(b), 15(a), and 15(b), use “sampling step size” as the x-axis (e.g., `1/512`) while others use the “number of sampling steps” (e.g., 1024). If these two metrics are equivalent to each other (e.g., 512 sampling steps correspond to a sampling step size of 1/512), unifying the x-axis to the “number of sampling steps” across all figures would greatly enhance clarity; and also prevent confusion between terms like “sampling step size = 1/512” and “dt=1/512”, which, despite similar notation, have different meanings.\n- In Figure 2(a), the relationship between the figure and various distillation rounds is ambiguous. If I understand correctly, “Round 1” and “Round 2” in the figure actually denote different denoising steps within a single distillation round, rather than separate distillation rounds.\n- The concept of progressive self-distillation is not new within diffusion models. The proposed framework SDTT largely resembles progressive distillation [1] with modifications such as the construction of targets and distillation objectives, both of which are straightforward following the formulation of discrete diffusion. While SDTT adapts distillation techniques to a different application and presents strong results, this paper does not sufficiently discuss its related work [1] throughout the introduction and the methods section, making it difficult to evaluate how SDTT differs from previous contributions [1].\n\n[1] Salimans, T., & Ho, J. (2022). Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)