Distillation of Discrete Diffusion through Dimensional Correlations
Authors: Satoshi Hayakawa
Yuhta Takida
Masaaki Imaizumi
Hiromi Wakaki
Yuki Mitsufuji
Venue: ICML 2025 poster (2025)
Keywords: diffusion model
discrete diffusion
distillation
consistency model
dimensional correlation
convergence analysis
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: We conduct theoretical analysis of current discrete diffusion models and propose a method to effectively capture element-wise dependency that is ignored in conventional models.

Abstract: Diffusion models have demonstrated exceptional performances in various fields of generative modeling, but suffer from slow sampling speed due to their iterative nature. While this issue is being addressed in continuous domains, discrete diffusion models face unique challenges, particularly in capturing dependencies between elements (e.g., pixel relationships in image, sequential dependencies in language) mainly due to the computational cost of processing high-dimensional joint distributions. In this paper, (i) we propose "mixture" models for discrete diffusion that are capable of treating dimensional correlations while remaining scalable, and (ii) we provide a set of loss functions for distilling the iterations of existing models. Two primary theoretical insights underpin our approach: First, conventional models with element-wise independence can well approximate the data distribution, but essentially require *many sampling steps*. Second, our loss functions enable the mixture models to distill such many-step conventional models into just a few steps by learning the dimensional correlations. Our experimental results show the effectiveness of the proposed method in distilling pretrained discrete diffusion models across image and language domains. The code used in the paper is available at https://github.com/sony/di4c.

Distillation of Discrete Diffusion through Dimensional Correlations
Satoshi Hayakawa'
Yuhta Takida Masaaki Imaizumi 3
Hiromi Wakaki
Yuki Mitsufuji'
Abstract
2022; Ho al., 2022; Blattmann et al., 2023). Recent ad- vancements in diffussin models often outperform traditional generative models, such as variational autoencoders (VAEs, Kingma & Welling, 2014; Higgins et al., 2017: Zhao et al., 2019) and generative adversarial networks (GANs, Good- fellow et al., 2014), in terms of sample quality and the controllability of the generated results. Furthermore, dif- fusion models are not limited learning continuous data; they can also be applied to discrete or categorical data with modifications (Hoogeboom et al., 2021; Austin et al., 2021) and offer a promising approach for discrete generative mod- eling (Gu et al., 2022; Lou et al., 2024). Such discrete diffusion models are the main topic of this paper.
Diffusion models have demonstrated exceptional performances in various fields of generative mod- cling, but suffer from slow sampling speed due to their iterative nature. While this issue is be- ing addressed in continuous domains, discrete diffusion models face unique challenges, particu- larly in capturing dependencies between elements (e.g., pixel relationships in image, sequential de- pendencies in language) mainly due to the com- putational cost of processing high-dimennsonal joint distributions. In this paper, (i) we propose "mixture" models for discrete diffusion that are capable of treating dimensional correlations while remaining scalable, and (ii) we provide set of loss functions for distilling the iterations of exist- ing models. Two primary theoretical insights un- derpin our approach: First, conventional models with dementtwise independence can well approx- imate the data distribution, but essentially require many sampling steps. Second, our loss functions enable the mixture models to distill such many- step conventional models into just few steps by learning the dimensional correlations. Our ex- perimental results show the effectiveness of the proposed method in distilling pretrained discrete diffusion models across image and language do- mains. The code used in the paper is available at https:// http//gittttt con/ sony/ / /ii 4c.
A notable drawback of diffusion models is their slow sam- pling speed due requiring many sampling steps (Xiao et al., 2022; Zhang & Chen, 2023). In continuous do- mains, various approaches have been proposed to reduce the number of steps, including well-designed forward pro- cesses (Song et al., 2021a) and fast solvers of stochastic/ordi- nary differential equations (SDEs/ODEs, Lu et al., 2022a;b; Zheng et al., 2023b). Another notable approach is knowl- edge distillation, which significantly reduces the number of sampling steps compared with earlier attempts by com- pressing pretrained diffusion models into single- or few-step generative models (Luhman & Luhman, 2021; Salimans & Ho, 2022; Meng et al., 2023; Zheng et al., 2023a). An emerging sub-family of distillation is the consistency-type models (Song et al., 2023; Song & Dhariwal, 2023; Kim et al., 2024), which exploit the fact that samples generated via different paths from the same initial noise should coin- cide.
1. Introduction
However, discrete diffusion models face a fundamental chal- lenge when attempting to reduce the number of sampling steps. Conventional approaches use "product" models that treat each dimension independently as sampling distribu- tions (Figure 1, center), since high-dimensional joint dis- tributions intractable. While this has been successful with hundreds of sampling steps, ignoring element-wise de- pendencies (which we refer to as dimensional correlations) causes non-negligible approximation errors (Figure 1). This ignoring is also pointed out in some concurrent works (Park et al., 2025; Liu et al., 2024; Xu et al., 2025).
Diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021b) have demonstrated excellent per- formance in generative modeling, particularly for continu- ous data such as images (Nichol al., 2021; Rombach et al., 2022; Saharia al., 2022), audio (Kong al., 2021; Chen et al., 2021; Evans et al., 2024), and video (Harvey et al.,
'Sony Group Corporation, Tokyo, Japan "Sony AI, Tokyo, Japan 'The University of Tokyo, Tokyo, Japan. Correspondence to: Satoshi Hayakawa "sstoshi.a.hayakwaal com
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In this paper, we propose Di4C (Distilling Discrete Diffu-

---

Distillation of Discrete Diffusion through Dimensional Correlations
Mixture model:
Ground truthe p(a,y)
Product model: (1))
(1txoalaa slaa + (ar1la
(1.30) (g (g) (L (1.9))dd
1st marginal: pl(x)
FA(x)
() ()(;;)) (A(o)))
Figure 1. Illustration of dimensional correlations. (Left) Distribution p(r, 3) is two-dimensional categorical distribution. (a) and ("() are its marginals. (Center) Conventional denoiser in discrete diffusion uses product model, which is simply product of marginal distributions. It fails to approximate ground truth distribution. (Right) Our mixture model is given by expectation of product model p(z, y: = p (x; A)p²(y: X) for random A. In figure, takes a; 18,7 y in equal probabilities, and model reconstructs p(z,
=p'
sion through Dimensional Correlations) to overcome this limitation. Our key insight is that while individual steps in conventional models are dimensionally independent, their composition over multiple steps can implicitly capture cor- relations (Section 3. On the basis of this observation, we develop (1) "mixture" model that explicitly represents di- mensional correlations while remaining computationally tractable (Figure 1, right), and (2) novel loss functions that effectively distill the many-step denoising of a product model into fewer steps. Our contribution is as follows:
ImageNet class-conditional generation with masked diffu- sion, our method achieves 2x speed-up while maintaining a comparable sample quality to the teacher model (Besnier Chen, 2023). In addition, (3) on masked diffusion language modeling with Open WebTee,,ext, we show that Di4C can fur- ther distill well-distileed model (Deschenaux & Gulcehre, 2025) by capturing dimensional correlations, without much harming of sampling diversity. These results consistently demonstrate that Di4C can effectively compress the sam- pling steps of discrete diffusion models while maintaining improving generation quality.
Theoretical analysis: In Theorem 1, we show that N-step sampling with product models can approximate data distri- butions in (L//N) total variation eиror. We also prove that this bound cannot be improved in a simple two-dimensional example. It underpins the empirical effectiveness of discrete diffusion models with many steps and, at the same time, shows the importance of modeling dimensional correlations reduce the number of sampling steps.
Finally, the remainder of this paper is organized follows: Section 2 gives preliminaries on discrete diffusion models and explains the dimensionality issue in discrete diffusion. We then explain the central idea of Di4C Section 3 and show theoretical results Section 4, which are partially described above as our contribution. Section provide experimental results with image and language tasks. After discussing related works in Section 6, we conclude the paper with discussions on its limitations and future work in Section 7.
also
To capture the aforementioned dimensional correlations, we propose a "mixture" model that can represent dimensional correlations (Section 3.2). To distill a many-step discrete diffusion model (teacher) into a few-step model (student), we also propose Di4C loss functions for compressing the iterative process of the teacher (Section 3.3). In theory, prove that the loss functions in Di4C can upper-bound the distance between the output distributions of the N N-step teacher and the student with just one step (Theorem 2). In combination with Theorem 1, this provides an overall theoretical guarantee for Di4C.
Model and loss design:
2. Preliminaries
2.1. Discrete diffusion models
Suppose we have data distribution go :- Qdata over the space X. In diffusion models -Schk-Dickstein al., 2015; Ho et al., 2020), with Zo go and ær Gr, where the time t can be ei- ther discrete or continuous. In this paper, we follow the notational convention that qejs and qa represent the true conditional and joint distributions defined by this Markov process, respectively. This process designed so that the terminal distribution Φr is tractable distribution. Follow- ing the convention, regard this forward process as adding
consider Markov process (æ:)
Finally, we demonstrate that our approach is general and applicable multiple settings. (1) On CIFAR- 10 with a bisel-based discretized Gaussian diffusion, sub- stantially improve the sample quality metrics of the teacher model (Campbell et al., 2022) in few-step sampling. (2) On
Experiments:
2

---

Distillation of Discrete Diffusion through Dimensional Correlations
noise to the data distribution. Our aim is generate samples approximately from the conditional distribution (o)r(-)ar) with æT 4r, which is a generative model for qdata- To this end, we introduce a model or denoiser, which is repre- sented as Paje (for 8 < 1). to approximate Zs|s-
Proposition 3) or Sun et al. (2023, Eq. 16)). Thus, lear- ing backward transition rates does not lead to capturing dimensional correlations, even if we use model capable of representing them.
Our primary interest is in the discrete diffusion models (Austin et al., 2021; Campbell al., 2022), where the space X is a finite set. In this case, a probability distribution p on X' can be regarded as function p : X' IR, and we will sometimes abuse the notation by treating p as just an ordinary function. We are given a finite set S and con- sider a diffusion process over the product space X' = S for a large D. Each state æ E X' can thus be written as 2 = (a )== where nd indicates the entry of x in the d-th dimension. Given probability distribution p = p(x) on X, let p" = (==") be its d-th marginal distribution, i.e., the distribution of xd given x p. To enjoy scalability, the for- ward process is usually set to be factorized over dimensions, i.e., [t)s (2, at.) = II 1 48/. (x]|a²) holds for < (Gu et al., 2022; Campbell et al., 2022).
3. Di4C for distilling discrete diffusion models
This section describes our proposed method, Di4C. We first show that the composition of well-trained discrete diffusion models can represent the dimensional correlation in Sec- tion 3.1, and in the later sections we discuss how to distill the multi-step denoising of a teacher model into student model that requires fewer steps. In particular, Section 3.2 and Section 3.3 try to solve the existing limitation described in Section 2.2 terms of modeling and loss design, respec- tively. See Section A for more technical details of Di4C.
3.1. Composition of diffusion denoisers for inducing dimensional correlations
We introduce the notion of composition, which plays a sig- nificant role in representing the dimensional correlations to be learned. Consider two general conditional distribu- tions p(x) y) and p(y|z) over finite sets, We define their composition as
2.2. Dimensional correlations in discrete diffusion
The common practices in modeling and training discrete diffusion models lead them to ignore the dimensional cor- relations within a data distribution, First, under the afore- mentioned problem setting, for the sake of scalability, the denoiser is usually defined as product model that satisfies
p o p(x) z) := Ey-p( = Σ ((aly))((lz> y
where this definition can be extended to the continuous case in a straightforward way. Although this is just a convolu- tion of two functions, it can be viewed as composition of denoising operators in the context of diffusion models. Specifically, given a single-step denoiser Pate and the finite timesteps 0 to < t1 < < ty = T, we typically use Ptalt, e ..... ltx (-10T) with the terminal noise 2T ~ QT as a generative sampler.
(x))(2,))=)) = II (=))0))) S sct.
Namely, the distribution P-je(-|a,,) is dimensionally inddpen- dent. This product modeling is common if not particularly highlighted (Campbell et al., 2022, Section G), due to the combinatorial explosion of the product discrete state. In- deed, adopting a product model significantly reduces the output length from O(DiS) to O(D)S)) at the cost of rep- resentational capacity. This limited expressive power can be crucial for considering few-step discrete diffusion models. As an extreme example, consider doing one-step denoising in the case of masked (absorbing-state) diffusion (Austin et al., 2021): there is no chance we can approximate com- plex joint distribution (as in Figure 1) in one step when er is a completely masked sentence (i.e., following a delta distribution) and Pojr( (-1z2) is dimensionally independent. See Section 1.1.1 for more examples.
Notably, the composition can serve as a source of dimen- sional correlation in discrete diffusion models. Even if one- step denoisers, Paju and Puje (s < u < f), are dimensionally independent, their composition is generally not. Indeed, dimensionally independent denoisers are successful given hundreds of sampling steps (Austin et al., 2021; Gu et al., 2022). Our method aims at compressing the composition of well-trained denoisers into few-step sampling by learning dimensional correlations.
Let pd be a pretrained teacher model with a product struc- Pix_ be sufficiently good ap-
ture and phj,, proximation of gojT. where 0 < t1 <... < ty T are timesteps. In our distillation, we would like the student model p" to approximate the teacher compositions as
Another potential factor making the learning of dimensional correlations infeasible in discrete diffusion models is that some of the existing loss functions are not well prepared for learning dimensional correlations. Most notably, in the continuous- time score- based discrete diffusion, we need only the marginal p (1 or its variant to compute the infinitesimal transition rate (see, e.g., Campbell al. (2022,
Polt. Poje, P.
n .e...... N.
(2)
To achieve this, we provide a way of modeling De that is
3

---

Distillation of Discrete Diffusion through Dimensional Correlations
capable of representing dimensional correlations in Sec- tion 3.2, and propose set of loss functions distill dimensional correlation represented by the compositions of a teacher model in Section 3.3.
time
Caittil
teacher
L 2
3.2. Mixture models to capture dimensional correlations
As an effective instance to represent correlated multivariate categorical distributions, we propose a mixture model. We define it as a family of conditional probability distributions that have the following representation for s < t:
Locnsial 2
student
Figure 2. Illustration of how our loss functions work. Through Lawan and Leonsis, we distill multiple teacher denoising steps into fewer steps of student denoiser.
(3)
D Paje (2
where (000(00001111 A) = I1
small 8; the dimensional correlation is mainly incorporated in the following consistency loss (see also Section A.
Here, À is a random variable with an arbitrary distribution. This distribution can be viewed as convex mixture of product models indexed by A. See Figure 1 (righ() for an intuitive illustration. Despite the fact that Pole (2o) X) is dimensionally independent for each given point X, this mixture representation is universal in the following sense:
Consistency loss. We then propose a consistency loss, which allows the student model learn the dimensional cor- relation represented by the teacher denoiser compositions:
Lconsis (0; 1hs 8r, u1,()
Proposition 1. For any probability distribution p over SD there exist a probability distribution ι and a family of prod- uct distributions p(æ; A) = [T-1p" (a(a X) indexed by X satisfying p(x)) = -EA= (p(;;; X)] for all SD.
:= EzErr [KL(P]]u o P (-Ize) ll (122)((12))))
where ra is reference distribution over X' at time t ap- proximating 4- While this loss is not straightforward to compute, we discuss how to approximate it in practice with Monte Carlo or control variates in Section A.2. Note that the idea of mixing the teacher denoiser and student denoiser in Ccousis can also be found in the continuous- regarding ODE trajectories (Kim al., 2024, Fig. 3), but our loss is different in that we work on the compositions of conditional probabilities in (2).
Indeed, have p(æ:)= = (=x=))) where 82 is the delta distribution at z, which is certainly product distribution. Although the proof is not very informative, the assertion itself implies that the mixture model has sufficient expres- sive power to capture dimensional correlations. It should also be noted that sampling from this mixture model during the inference has almost no extra computational overhead compared with the conventional product model, since it just requires sampling and insertion of X (see Section 5.4).
setting
Figure 2 shows the intuition behind our loss functions. As reference distributions rs and rr, we can either use g gen- crated from data or the distribution obtained by applying multiple teacher denoising steps. See Section 4 for their roles and theoretical guarantees on Ldisti and Lcousis-
3.3. Consistency for distilling dimensional correlations
We present set of (two) loss functions that take dimensional correlation into account. Consider we are given product teacher model, which is denoted as p". Let p" be a general student model (with enough expressive power; an example is given in Section 3.2) that we want to train based on p
4. Theoretical analysis
In this section, we present an overall theoretical analysis on our distillation method. In Section 4.1, we show that the conventional product model (1) can approximate data distribution if the model's marginal is perfectly trained and given many steps (O(1/N) upper bound). We also show that its N-step total variation error can be lower bounded by ((1/N) even for simple two-dimensional example. Both bounds support the empirical evidences of cxisting models that work (only) under many steps. In Section 4.2, we prove that the proposed objective functions enable the many-step
Distillation loss. We first introduce a distillation loss, which forces the student model to approximate the teacher model time 8 (< T))
Laistu (0: v. r), ; Euswrrs DKL (P) (<sss) 1 (<125)
where T4 (=x g5) is a reference distribution over X' at time 5 and DKL is the Kullback-Leii (KL) divergence. We expect that single teacher denoising step enough for a
denoising with a teacher model to be distilled into a few-step
4

---

Distillation of Discrete Diffusion through Dimensional Correlations
student model, provided that the student model has enough expressive power.
as + 0. The exploits the factorization quja (2r (m.))
Ihe Lf s (xffa:3) and is based on additional continuity as- sumptions. We then decompose the left-hand side of (4) into N terms by using a triangle-like inequality on drv between compositional distributions (Proposition 4). The i-th term essentially measures the distance between 9t, alt; and Pt,-1 and thus is bounded by (1/N22 from (5) with = T/N. By summing up the N terms, we obtain the desired estimate for the first part. For the second part, we actually construct concrete example in Section C.3.
By combining the upper bounds in these results, we in- formally obtain the following estimate when distilling an N-step sampling process of a teacher model that learns the marginals perfectly:
(rv(stLs student), qo) (< Theorem I, upper bound)
I <div(p(L-step student), -step teacher)) + (1/NN)
(Theorem 2) Here, p(n-step model) represents the resulting distribution of n-step sampling with the model starting from qr, and drv denotes the total variation distance.
< (Di4C losses in (6)) + (1/NN)
See Theorem 3 for a formal version. Theorem 1 is important as it underpins the use of dimensionally parallel denoising given sufficient steps, which has been claimed as an advan- tage of discrete diffusion over autoregressive models whose sampling is sequential (Lou et al., 2024). However, it still requires (11/e) steps in order to have uniform error bound e, according to the latter half of the assertion. We show next that we can further reduce the number of steps with our loss functions, by distilling the distribution of an N-step teacher model into a few-step student model by learning dimensional correlations.
4.1. Product models with multi-step sampling can approximate data distribution
We first show that dimensionally independent denoisers with many steps are capable of approximately recovering a data distribution, which has already been empirically observed in existing studies. To consider varying the number of de- noising steps, let us work on the continuous-ting time setting. Let (ax )osssr follow a continuous-time Markov chain over [0, I] and the space X = S with a factorized forward pro- cess. i.c. Htjs (2 (x.) = IT 9/, (x+] (24) for < t. See Section C for more details.
4.2. Our losses can distill multi-step denoising models
Let p" and y" respectively be the teacher and student models given in Sections 3.1 & 3.3. The following statement gives theoretical guarantee for using the proposed loss functions at the appropriate time and distribution settings.
Theorem shows the capability and limitation of a dimen- sionally independent sampling scheme called analytical sampling (Sun al., 2023) or Tweedie r-leaping (Lou et al., 2024; Ou et al., 2025). where we use product-model de- noiser P=)e(ar.(22:) = I[ (xg (..)) (xz)))) approximating the true marginal as pa [x,) x gaje (x (z)). Although com- monly used, there has been only empirical evidence for the overall efficiency of this dimensionally independent method.
Theorem (Di4C student approximates N-step teacher). Let 0 = to < <... < ty T be timesteps and rT be probability distribution X. For each n, let rt,
d E ExTwet Pinlto+i o.... ....oPiN-ile (( Then, we have
Note that Campbell et al. (2022) provides a guarantee for an- other dimensionally independent method called r-leaping. Theorem 1 (N-step analytical sampling approximates data, informal). Let gels be forward transition probabilities that factorize as above and Patt be a product model with the correct marginals, i.e., (=))( -II oh((a" xx)) for s < t. Under regularity conditions, given timesteps 4 = =iT/N for = O,..., N, we have, as N + 00,
drv ro, Berory l ( ((T)
S v2
(dtssui(: v/ re, 12)
N-1 Lconsia (f: th, Pints O, tni tn, 1/+2)
(6)
We can prove this by formalizing the intuition behind Fig- ure 2 (see Section D.I). Note that the right-hand side of inequality (6) becomes zero (so does the left-hand side) if the student model perfectly learns the composition of the teacher as in (2), so learning with these loss functions is fea- sible in theory if the student model has enough expressive power Existing theoretical guarantees in consistency- distillation of continuous-state diffusions typically discuss the case when consistency losses are exactly zero (Song et al., 2023; Daras et al., 2024; Lai et al., 2023), so our guarantee would be interesting that it explicitly shows the
dTv (go. Exr-or [Peoles .... PeN- (- ((T))) = (1/N). (4)
Moreover, there is an example with |S|= = D 2 such that the left-hand side of (4) is lower-bounded by c/N with some constant e >>0 0 for sufficiently large N.
Proof (sketth). We first prove the following estimate for 0 5 t- e < t T and x E S (Lemma l):
dr drv((e-))((1a)) Pt- ((-e((())))) = O(2),
(5)
5

---

Distillation of Discrete Diffusion through Dimensional Correlations
relationships between the magnitude of loss functions and the total variation bound between the teacher and student.
original implementation of time conditioning. In training, the student model was initialized by the teacher network parameters with additional zero-initialized subnetworks con- cerning A. See Section F.1.3 for implementation details.
Regarding the choice of rr, we should take Py = 9r if we would like to combine Theorem 2 with Theorem 1 to evalu- ate Di4C's overall performance against the data distribution. For Fr with & < T. though we can generate samples dt ~ T't by using the teacher model, it might be expensive due to the multi-step inference required. Instead, we can use q; if we have access to data, which is given by just one-step forward sampling from (q with the data 2n ~ go. Since T is an approximation of (Theorem 1), it would not harm the training quality as long as the teacher model is well-trained.
Table 1. Comparison of models on CIFAR-10 dataset. Fréchet inception distance (FID 4) against training dataset and inception score (IS f) measured with 50,000 samples are shown in this order (FID / IS).
10 steps
20 steps
40 steps
teacher
32.61 / 7.59
12.36 / 8.55
8.01 /8.77
student
20.64 8.29
9.77 / 8.52
9.66 8.28
5. Experimental results
hybrid
25.54 8.00
9.47 / 8.56
8.02 8.44
We evaluated our Di4C method in three different diffu- sion settings, each with distinct teacher models and state spaces. First, we examined continuous-time discrete-state diffusion with pixel-space representations using CIFAR-10, where we distilled from a well well-trained U-Net teacher model (Section 5.1). Second, we explored masked diffusion on uuetor-quattized (VQ) space using ImageNet, working with hansformer-based teacher model designed for masked im- age generation (Section 5.2). Third, we tested our approach on masked diffusion language models trained on Open- WebText, demonstrating Di4C's effectiveness in distilling transformer-h hased diffusion language models (Section 5.3). Finally, we demonstrate that our mixture modeling causes minimal latency overhead in all the above experiments (Sec- tion 5.4). These experimental results, spanning different domains, architectures, and diffusion processes, showcase the broad applicability of our method while highlighting its consistent ability to achieve faster sampling.
Table I shows the results. The "hybrid" model used the student model for the first half of the denoising process and then switched to the teacher model for the remaining steps during inference. The student substantially improved the metrics compared with the teacher in 10-step sampling, while the benefits of our method diminished as the number of steps grew. In contrast, the hybrid model was the best at 20 steps and on par with the teacher in 40-step FID. We hypothesize that Di4C is particularly effective when using fewer sampling steps, as this is where capturing complex di- mensional correlations crucial (Theorem 1). We also give additional results for different sampling steps in Table 4.
Table 2. Comparison of teacher and student with PC steps. FID against CIFAR- 10 training dataset measured with 10,000 gener- ated samples are displayed (so there is discrepancy from Table 1). Number of function evaluations (NFEs) is adjusted 20, with varying budget allocations between denoising and PC steps.
5.1. Discretized Gaussian diffusion on pixel space
NFE
6-747
10+5+5
14+343
20+0+0
In our first experiment, we adopted the same setting as Campbell et al. (2022): continuous- time discrete-state Markov process (of discretized Gaussian transition) with the CIFAR-I0 image dataset (Krizhevsky, 2009). We used the well-trained model checkpoint provided by Campbell et al. (2022) as our product teacher model (p*)) This model outperforms previous discrete-time discrete-state models such as Austin et al. (2021). As in the original paper, we worked directly with the discrete pixel channel values (0 to 255) on 32 x 32 x 3 entries (i.c., |S| = 256, D = 3072). The teacher model pd has a U-net architecture (Ho et al., 2020) tailored for discrete diffusion, which is fed time stage.
teacher 57.57
32.74
21.32
14.42
student
44.57
22.63
14.25
11.81
In Table 2, we also compare the teacher and student models in combination with the predictor-corrector (PC, Camp- bell et al., 2022) steps. The notation Meuta indicates that we used n corrector steps before each of the final two out of m denoising steps, imitating the approach in Campbell et al. (2022). For each setting, the best PC step-size among [10 -3 10 100-5 }} was chosen. Since PC step requires an additional network evaluation (as expensive as one step of denoising), we adjusted the total NFE in the table. While adding PC steps shows improvement upon the vanilla m- step denoising (see also Table 4) in both teacher and student models, allocating all the NFE budget to denoising steps performs better in this small NFE regime; this observation aligns with Campbell et al. (2022, Fig. 4).
feature at each
To obtain an architecture for student mixture model (3), we slightly extended the teacher's architecture so that it accepts a conditioning with A ~ (I0, 1I) (uniform distribution over [0, 1) in this experiment, by following the
6

---

Distillation of Discrete Diffusion through Dimensional Correlations
5.2. Masked image modeling on VQ space
10.5
teacher (4 steps)
Next, we evaluated our method by applying it larger- scale image generation model. For this purpose, adopted the framework of MaskGIT (Chang et al., 2022) and worked on the ImageNet dataset (Deng et al., 2009) at 256 x 256 resolution. MaskGIT is one of the saae-offthheart image generation methods, based on masked diffusion modeling. Its generative (backward) process relies on heuristics includ- ing confidence-based sampling (Section F.2.2). The variant used also uses discrete classifier-free guidance (CFG; Section F.2.3). We demonstrate that Di4C can enhance image generation even in combination with such heuristics.
10.0
di4c (4 steps)
di4c-d (4 steps)
9.5
teacher (8 steps)
9.0
8.5
8.0
7.5
7.0
6.5
160
180
200
220
In our setting, the model comprised two main components: a VQGAN (Esser et al., 2021) pre-trained on the ImageNet dataset and a masked diffusion model trained on the VQ space. The VQGAN encodes a 256 256 resolution image into 16 x 16 256 tokens, each drawn from shared codebook S of size 1024. The forward process of our diffusion modeling, denoted as (ejo(-(zo), independently replaces =0f with [MASK] at certain probability m, for each d (i.e., S : S U [[MASK]], | 1025, and D 256). The masking probability my increases monotonically from my = to m_ = 1, resulting in completely masked sequences t = regardless Zg.
Inception Score
Figure 3. FID-IS curves of 4/ 4/8-step teacher and 4-step Di4C mod- els ImageNet 256 x 256 when varying CFG coefficients. Ar- rows connect experimental results (dots) at each CFG coefficient in ascending order.
See Section F.2 for further details of this experiment. In par- ticular, Table 8 shows the detailed quantitative performance (including precision and recall) of each model at its best CFG coefficient.
As the teacher model, we used the PyTorch- -aseed imple- mentation by Besnier & Chen (2023), which replicates the performance of the original MaskGIT. It uses bidirectional transformer yd that estimates the distribution of each token given an ImageNet label e and a partially masked sequence Ie E SP: vo.d (- c) = (- c). The input sequence length is 257 including the embedding of c. Additionally, the model supports unconditional generation, enabling dis- crete CFG match the original MaskGIT's performance. In this configuration, the teacher model generates high-quality samples with only 8 steps, which is significantly faster than typical diffusion-based generative models (see, e.g., Chang et al., 2022, Table 1). In the implementation of our mixture (student) model, we simply added a single token embedding coming from (n0 1)). so the input sequence length 258 instead of 257 in the student models.
5.3. Masked diffusion language models
Finally, we examined Di4C in language modeling. As teacher models, we adopted two versions of pretrained SDTT models (Deschenaux & Gulcehre, 2025, with a KL target), which we refer to as sdtt-6 and sdtt-7. They were obtained after respectively and rounds of distillation of masked diffusion language model (MDLM, Sahoo et al., 2024. See also its concurrent works: Shi et al. 2024; Ou et al. 2025) trained on the OpenWebText dataset (Gokaslan & Cohen, 2019). As a forward process, they use a masked diffusion with a GPT-2 tokenizer (Radford et al., 2019), which is essentially the same as the one explained in Sec- tion 5.2, while we had D 1024 and [8*| 50257 this time. In the mixture modeling for Di4C, added one token from À ~ Unif( 1J) to the transformer, similarly to the previous section. See Section F.3 for details.
Figure shows FID-IS curves of 4-step sampling with var- ious CFG coefficients: Weig E {5, 6, 7, 8, 9, 10, 11, 12} for 4 steps and Weig E (2, 2.5, 3, 3.5, 4} for 8 steps (teacher only). In the figure, di4c represents a model trained using the standard Di4C method, while di4c-d represents a model trained with an additional datapoint loss (see Section F.2.4 for details). As shown in the figure, the FID-IS performance of the 4-step Di4C models closely matched that of the 8- step sampling of the teacher model. This result indicates that Di4C can achieve an approximate 2x speed-up in the sampling process.
Figure 4 shows the results of applying Di4C to the SDTT models. For n = 6, 7, sdit-n + di4c represents the model obtained by fine-tuning sdtt-n using Di4C. Similarly, sdtt- n + di4c² represents the model obtained by applying an- other round of Di4C to sdit-n di4c (see Section A.4).
Unconditional generation. Figure 4(a) shows perfor- mance comparison for unconditional generation. Each model generated 1024 samples of 1024 tokens, with the number of sampling steps in {8, 16, 32, 64, 128). Follow- ing the previous work (Lou et al., 2024; Sahoo et al., 2024;
7

---

Distillation of Discrete Diffusion through Dimensional Correlations
180
sdtt-6
5dtt-6
4 steps
sdit-6 di4c
Cond. Generative Perplexity
sdit-6 6 di4c
B steps
Generative Perplexity
160
sdtt-6 + di4c²
250
sdit-6 di4c?
steps
140
sdtt-7
sdtt-7
32 steps
sdtt-7 + di4c
200
sdit-7 di4c
64 steps
120
sdtt-7 + di4c²
sdtt-7 di4c?
128 steps
100
150
80
100
60
40
50
20
23
2
2
2
2
55.5
56.0 56.5 57.0
57.5
58.0
58.5
59.0 59.5
Num. Sampling Steps
Cond. Sclf-BLEU
(a) Gen. PPL vs Num. Steps in unconditional generation
(b) Gen. PPL vs Self-BLBU in conditional generation Figure 4. Comparison of SDTT checkpoints (Deschenaux Gulcehre, 2025) and their Di4C distillations.
Deschenaux & Gulcehre, 2025), we used the GPT-2-large model (Radford et al., 2019) to evaluate the generative per- plexity. The results show that applying one round of Di4C lead similar level of improvement as applying one round of SDTT (compare sdtt-6 di4c with sdtt-7). Also, rounds of Di4C (+ di4c²) showed speed-ups of more than 2x to match the teacher's performance of 64 or 128 steps.
pared with its performance gain. Table 3 shows the quantita- tive results on the latency. The experiments listed in Table correspond to those described Sections 5.1-5.3, where the chosen student models are the best models (di4c-d in ImageNet and sdtt-7 di4c2 in OpenWeeText), The table shows the average runtime and standard deviation, calcu- lated over 10 batches with batch sizes of 50, 64, and 16, respectively. We tested unconditional generation for CIFAR- and OpenWeb WebText. and a conditiona generation with a uniform random class for ImageNet with a classifier- free guidance.
Conditional generation. We also tested the quality- diversity tradeoff in conditional generation. We used 256 samples from the WebText dataset (OpenAI, 2019), and each model, on the basis of the first 50 tokens of the sam- ple, generated continuations of 50 tokens, following De- schenaux Gulcehre (2025). We computed the generative perplexity of the generated continuations and the Self-BLEU score (Zhu et al., 2018) of the 5 completed samples starting from the same prompt, and averaged them over the 256 prompts. Self-BLEU with equal weights on n-gram for 1, 2, 4 and scaled from 0 to 100 (lower Self-BLEU indicates higher diversity). Note that (Self-)BLEU has several parameters including tokenization and maximum length for n-grams (Post, 2018 2018), so our numbers are not directly comparable to those from an existing work such Agarwal et al. (2024).
Table 3. Latency comparison between teacher (product) and stu- dent (mixture) models.
Experiment
# steps
teacher (scc.)
student (sec.)
CIFAR-I0 ImageNet
10
10..004
0.5786+000017
computed by GPT-2 tokenizer
2.074 +0.003
2.0734400045
WeenWetText
16
3.3409
3.4817+0.0
In the CIFAR-10 and OpenWebText experiments, the over- head from using mixture models was up to 5%, which is negligible compared with their performance gain. In the ImageNet experiment, the difference between the student and teacher was within the statistical noise. This is likely because the computational boutleneck in this case was not the logit inference itself but rather the sampling process that followed it.
Figure 4(b) shows the quality-diversig tradeoff curves of each model with various numbers of sampling steps. As shown with solid lines, both sdit-6 + di4c and sdtt-6 + di4c² consistently achieved higher diversity than sdtt-7 with the same number of sampling steps while maintaining comparable or better generative perplexity.
6. Related work
5.4. Latency overhead of mixture modeling
Speeding up continuous diffusion models. In continuous diffusion models, knowledge distillation (Luhman Luh- man, 2021; Salimans Ho, 2022; Meng et al., 2023: Zheng
In this section, we demonstrate that the computational over- head in introducing mixture modeling negligible com-
8

---

Distillation of Discrete Diffusion through Dimensional Correlations
al., 2023a) and consistency-type techniques (Song al., 2023; Song & Dhariwal, 2023; Kim al., 2024) enable single- or few-step sampling, most of which are tailored for probability flow ODEs. Among the studies on continuous diffusion, the work by Li et al. (2024) is particularly relevant to our mixture modeling approach. They highlight the lim- ited expressive power of unimodal Gaussian distributions in denoising continuous diffusions and demonstrate that using Gaussian mixtures can substantially reduce the number of sampling steps required in non-ODE diffusions.
can indeed recover data distribution, even without mod- eling dimensional correlations. We also proved that such many-step models can be distilled into few-step ones, if we use the Di4C loss functions with a model that has enough expressive power, such as a mixture model. In numerical experiments, we confirmed the efficiency of our framework upon teacher models across multiple domains: improving sample quality in 10-step sampling on CIFAR-I0 with a discretized Gaussian diffusion, achieving a 2x speed-up in ImageNet 256 256 generation with masked image mod- eling, and successfully distilling already-distille masked diffusion language models on OpenWeeTexx while main- taining generation diversity.
Speeding up discrete diffusion models. For faster sample generation in discrete diffusion models, Park et al. (2025) propose a post-hov hoc optimization of sampling schedules and outperforms the uniform partitioning of [0, 7). To speed up MDLMs (Sahoo et al., 2024); SDTT (Deschenaux & Gulcehre, 2025) gathers the logits at unmasked tokens from a pretrained teacher MDLM throughout the denoising pro- cess, achieving speed-up of approximately 32x compared with the teacher model. In addition, the confidence- baaed sampling by MaskGIT (Chang et al., 2022) is essentially a heuristic to enable faster sampling in masked diffusions. These approaches can be combined with Di4C; we indeed have demonstrated it for the latter two.
However, there are still problems to be solved. For example, although we can distill many-step models into one-step ones in theory (Theorem 2), our empirical results show only 2x fewer steps or so. To address this point, we need to further optimize the architecture (mainly concerning A) and training hyperparameters, while the iterated Di4C training (Section A.4) can be a promising candidate.
Impact Statement
This paper presents novel method for reducing the number of sampling steps required for high-quality generation in discrete data domains. While there are many potential social impacts as usual in the generative Al field, we do not foresee any specific problems particularly caused by this (mostly theoretical) work.
Two concurrent works are particularly relevant to our re- search, both of which attempt to incorporate dimensional correlations into discrete diffusion, specifically within the MDLM context. Liu et al. (2024) combine pretrained au- toregressive and masked diffusion models, achieving supe- rior performance compared with using either model alone. Meanwhile, Xu et al. (2025) propose an energy- -based model to modify the dimensionally independent sampler. While both approaches improve upon the vanilla MDLM, their sampling processes can incur some time and memory over- head. This is due to the use of an additional non non-diffusion model in both methods, and the reliance on importance sampling in the latter. In this regard, our mixture model- ing allows for a simpler sampling process with minimal modification to the original diffusion. However, since the should explore
A Acknowledgments
We would like to thank Masato Ishii, Chich-Hsin Lai, Naoki Murata, Bac Nguyen, and Toshimitsu Uesaka for their con- structive feedback. We also thank the anonymous reviewers for their valuable comments and suggestions.
References
Agarwal, R., Vieillard, N., Zhou, Y., Stanezyk, P., Garea, S. R., Geist, M., and Bachem, O. On-policy distillation of language models: Learning from self-generated mistakes. In Proceedings of the Twelfth International Conference on Learning Representations, 2024.
Di4C loss functions are madel-agnostic, the combination of Di4C losses and other proposed models capturing dimensional correlations in future research.
7. Conclusion
Anderson, W. J. Continuous-tim Markov chains: An applications-orrentee approach. Springer Science & Business Media, 2012.
In this paper, as current discrete diffusion models ignore the dimensional correlations that need to be incorporated to realize few-step models, we proposed Di4C, method for distilling pretrained discrete diffusion models. Di4C provides a set of loss functions for models that can cap- ture dimensional correlations, an example of which is the mixture model. As a theoretical contribution, we proved that the existing discrete diffusion models with many steps
Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and Van Den Berg, R. Structured denoising diffusion models in discrete state-spaces. In Advances in Neural Information Processing Systems, volume 34, pp. 17981-17992) 2021.
Besnier, V. and Chen, M. pytorch reproduction of

---

Distillation of Discrete Diffusion through Dimensional Correlations
masked generative image transformer. arXiv preprint arXiv:2310. 14400, 2023.
Elfwing, S., Uchibe, E., and Doya, K. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Networks, 107:3-11, 2018.
Bird, S., Klein, E., and Loper. Natural language process- ing with Python: analyzing text with the natural language toolkit. O'Reilly Media, Inc., 2009.
Esser. P., Rombach, R., and Ommer, B. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12873- 12973, 2021.
Blanchard, Higham, D. J., and Higham, N. J. Accurately computing the log-sum-exp and softmax functions. IMA Journal of Numerical Analysis, 41(4):2311-2333 --330, 2021.
Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z.., Voleti, V., Letts, A., Jampani, V., and Rombach, R. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint 1..Xv:23311151227 2023.
Evans, Z., Carr, C., Taylor, J., Hawley, S. H., and Pons, J. Fast conditioned latent audio diffusion. In Proceedings of the Forty-First International Conference on Machine Learning, pp. 12652-12665, 2024.
Gillespie, D. T. Approximate accelerated stochastic sim- ulation of chemically reacting systems. The Journal of Chemical Physics, 115(4):1716-1733 2001.
Campbell, A., Benton, J., De Bortoli, V., Rainforth, T., Deligiannidis, G., and Doucet, A. A continuous time framework for discrete denoising models. In Advances in Neural Information Processing Systems, volume 35, pp. 28266-28279, 2022.
Glasserman, Monte Carlo methods in financial engineer- ing, volume 53, Springer, 2004.
Canonne, C. L. short note on an inequality between KL and TV. arXiv preprint urXiv:2202.07/99 2022.
Gokaslan, and Cohen, V. Openwebtext cor- pus. http://Skylinn www.iooo077......i..... github.io/oo/ OpenWebTextCorpuas 2019.
Chang, H., Zhang, H., Jiang, L., Liu, C., and Freeman, W. T. Maskgit: Masked generative image transformer. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 11315-11325, 2022.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2672-2688 2014.
Chen, N., Zhang, Y., Zen, H., Weiss, R. J., Norouzi, M., and Chan, W. Wavegrad: Estimating gradients for waveform generation. In Proceedings of the Ninth International Conference Learning Representations, 2021.
Gu, S., Chen, D., Bao, J., Wen, F. Zhang, B., Chen, D., Yuan, L., and Guo, B. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10696-10706, 2022.
Comunità, M.. Zhong, Z.. Takahashi, A., Yang. S., Zhao, M., Saito, K., Ikemiya, Y., Shibuya, T., Takahashi. S., and Mitsufuji, Y. Specmaskgit: Masked generative modeling of audio spectrograms for efficient audio synthesis and beyond. arXiv preprint arXiv:2406. 17672, 2024.
Harvey, W., Naderiparizi, S., Masrani, V., Weilbach, C., and Wood, Flexible diffusion modeling long videos. In Advances in Neural Information Processing Systems, volume 35, pp. 27953 22965,,,,6,-- 2022.
Cover, T. M. and Thomas, J. A. Elements of information theory. John Wiley & Sons, second edition, 2006.
Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. beta- VAE: Learning basic visual concepts with a constrained variational framework. In Proceedings of the Fifth Inter- national Conference on Learning Representations, 2017.
Daras, G., Dagan, Y., Dimakis, A., and Daskalakis, C. Con- sistent diffusion models: Mitigating sampling drift by learning to be consistent. In Advances in Neural Informa- tion Processing Systems, volume 36, 2024.
Deng, J., Dong, W., Socher, R., Li, L.-J,, J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pat- tern Recognition, pp. 248-255. leee, 2009.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion prob- abilistic models. In Advances in Neural Information Processing Systems, volume 33, pp. 6840-6851, 2020.
Deschenaux, J. and Gulcehre, C. Beyond autoregression: Fast LLMS via diff-distillation through time. In Pro- ceedings of the Thirteenth International Conference on Learning Representations, 2025.
Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, D. Video diffusion models. In Advances in Neural Information Processing Systems, volume 35, pp. 8633-8646 2022.
10

---

Distillation of Discrete Diffusion through Dimensional Correlations
Hoogeboom, E., Nielsen, D., Jaini, P., Forré, P., and Welling, M. Argmax flows and multinomial diffusion: Learning categorical distributions. In Advances in Neural Informa- tion Processing Systems, volume 34, pp. 12454-12486 2021.
Luhman, E. and Luhman, T. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint 10t.0:22000002888. 2021.
Meng, C., Rombach, R., Gao, R., Kingma, D., Ermon, S., Ho, J., and Salimans, T. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 14297-14306, 2023.
Kim, D., Lai, C.-H., Liao, W.-H., Murata, N., Takida, Y., Uesaka, T., He, Y., Mitsufuji, Y., and Ermon, S. Consis- tency trajectory models: Learning probability flow ODE trajectory of diffusion. Proceedings of the Twelfth International Conference on Learning Representations, 2024.
Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., and Chen, M. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In Proceedings of the Thirtt-Eighth International Conference on Machine Learning, 2021.
Kingma, D. P. and Welling, M. Auto-encoding variational Bayes. In Proceedings of the Second International Con- ference Learning Reprexentations, 2014.
Kong, Z., Ping, W. Huang, J., Zhao, K., and Catanzaro, B. Diffwave: A versatile diffusion model for audio synthesis. In Proceedings of the Ninth International Conference on Learning Representations, 2021.
Oates, C. J., Girolami, M., and Chopin, N. Control func- tionals for monte carlo integration. Journal of the Royal Statistical Society Series B: Statistical Methodology, 79 (3):695-718, 2017.
OpenAL. URL
GPT-2 output dataset, 2019. https: /ttt:://gtttbbbt www.ttuu.coooooooooo com/openail 10s-2-ootpuu-dataass
Krizhevsky, A. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009.
Lai, C.-H., Takida, Y., Uesaka, T., Murata, N., Mitsufuji, Y., and Ermon, S. On the equivalence of consistency- -tpp models: Consistency models, consistent diffusion models, and fokker-plande regularization. In ICML 2023 Work- shop on Structured Probabilistic Inference & Generative Modeling, 2023.
Ou, J., Nie, S., Xue, K., Zhu, F., Sun, J., Li, Z., and Li, Your absorbing discrete diffusion secretly models the conditional distributions of clean data. In Proceedings of the Thirleenth International Conference on Learning Representations, 2025.
Park, Y.-H., Lai, C.-H., Hayakawa, S., Takida, Y., and Mit- sufuji, Y. Jump your steps: Optimizing sampling sched- ule of discrete diffusion models. Proceedings of the Thirteenth International Conference on Learning Repre- sentations, 2025.
Li, Y., van Breugel, B., and van der Schaar, M. Soft mixture denoising: Beyond the expressive bottleneck of diffu- sion models. In Proceedings of the Twelfth International Conference on Learning Representations, 2024.
Liu, A., Broadrick, O., Niepert, M., and Broeck, G. V. d. Dis- crete copula diffusion. arXiv preprint 24X.0:2410.01999 2024.
Pillutla, K., Swayamdipta, S., Zellers, R., Thickstun, J., Welleck, S., Choi, Y., and Harchaoui, Z. Mauve: Mea- suring the gap between neural text and human text using divergence fronticrs. In Advances in Neural Information Processing Systems, volume 34, pp. 4816 4816-4828, 2021.
Lou, A., Meng, C., and Ermon, S. Discrete diffusion mod- eling by estimating the ratios of the data distribution. In Proceedings of the Forty-Firs. First International Conference Machine Learning, 2024.
Post, M. A call for clarity in reporting BLEU scores. In Pro- ceedings of the Third Conference on Machine Translation: Research Papers, pp. 186-191, 2018.
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, DPM-solver: A fast ODE solver for diffusion probabilis- tic model sampling in around 10 steps. In Advances in Neural Information Processing Systems, volume 35, pp. 5775-5787, 2022a.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, Language models are unsupervised multitask learners. OpenAl blog, 2019.
Rombach, R., Blattmann,
A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, J. DPM- solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint 20X.022211.01095, 2022b.

---

Distillation of Discrete Diffusion through Dimensional Correlations
Saharia, C., Chan, W., Saxena, S., Li, L.,, Whang, J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., Ho, J., Fleet, D. J., and Norouzi, M. Photorealistic text-to- text-to-image diffusion models with deep language understanding. In Advances in Neural Informa- tion Processing Systems, volume 35, pp. 36479-36494, 2022.
van den Oord, A., Vinyals, O., and Kavukcuoglu, K. Neural discrete representation learning. In Advances in Neural Information Processing Systems, volume 30, 2017.
von Eitzen, H. Continuous function with continuous one-sided derivative. Mathematics Stack Exchange. 2014. URL https: https////maat..tccccccchhce com/ /t/////// /79750 (version: 2014-10-16).
Sahoo, S. S., Arriola, M., Schiff, Y., Gokaslan, A., Marro- quin, E., Chiu, J. T., Rush, A., and Kuleshov, V. Simple and effective masked diffusion language models. Ad- vances in Neural Information Processing Systems, vol- 36, 2024.
Williams, R. J. Simple statistical gradient- algo- rithms for connectionist reinforcement learning. Machine Learning, 8:229-256, 1992.
Xiao, Z., Kreis, K., and Vahdat, A. Tackling the genera- tive learning trilemma with denoising diffusion gans. In Proceedings of the Tenth International Conference on Learning Representations, 2022.
Salimans, T. and Ho, J. Progressive distillation for fast sampling of diffusion models. In Proceedings the Tenth International Conference on Learning Representations, 2022.
Xu, M., Geffner, T., Kreis, K., Nie, W., Xu, Y., Leskovec, J., Ermon, S., and Vahdat, A. Energy-based diffusion language models for text generation. In Proceedings of the Thirteenth International Conference on Learning Representations, 2025.
Shi, J., Han, K., Wang. Z., Doucet, A., and Titsias, M. Simplified and generalized masked diffusion for discrete data. In Advances in Neural Information Processing Systems, volume 37, pp. 103131-103167, 2024.
Sohl-Dickstein, J.. Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequilib- rium thermodynamies. In Proceedings of the Thirty-Sixth International Conference Machine Learning, pp. 2256- 2265, 2015.
Zhang, Q and Chen, Y. Fast sampling of diffusion mod- els with exponential integrator. In Proceedings of the Eleventh International Conference on Learning Repre- sentations, 2023.
Zhao, S., Song. J., and Ermon, S. InfoVAE: Balancing learning and inference in variational autoencoders. In AAAI Conference on Artificial Intelligence, pp. 5885- 5892, 2019.
Song. J., Meng, C., and Ermon, S. Denoising diffusion the Ninth International
implicit models. In Proceedings the Conference on Learning Representations, 2021a.
Song, Y. and Dhariwal, P. Improved techniques for train- ing consistency models. In Proceedings of the Eleventh International Conference on Learning Representations, 2023.
Zheng. H., Nie, W., Vahdat, A., Azizzadenesheli, K., and Anandkumar, A. Fast sampling of diffusion models via operator learning. In Proceedings of the Fortieth Inter- national Conference on Machine Learning, pp. 42390- 42402, 2023a.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score- based generative modeling through stochastic differential equations. In Proceed- ings of the Ninth International Conference on Learning Representations, 2021b.
Zheng, K., Lu, C., Chen, J., and Zhu, J. DPM-solver-v3: Improved diffusion ODE solver with empirical model statistics. In Advances in Neural Information Processing Systems, volume 36, pp. 55502-55542, 2023b.
Song. Y., Dhariwal, P., Chen, M., and Sutskever, I. Con- sistency models. In Proceedings of the Fortieth Inter- national Conference on Machine Learning, pp. 32211 32252. PMLR, 2023.
Zhu, Y., Lu, S., Zheng, L.,, Guo, J., Zhang, W., Wang, J., and Yu, Y. Texygen: benchmarking platform for text generation models. In The Forty-First International ACM SIGIR Conference Research & Development in Information Retrieval, pp. 1097-1100, 2018.
Sun, H., Yu, L., Dai, B., Schuurmans, D., and Dai, H. Score- based continuous-time discrete diffusion models. In Pro- ceedings of the Eleventh International Conference on Learning Representations, 2023.
Tang, Z., Gu, S., Bao, J., Chen, D., and Wen, F. Im- proved vector quantized diffusion models. arXiv preprint arXiv:2205. 16007, 2022.
12

---

Distillation of Discrete Diffusion through Dimensional Correlations
A. Training techniques for Di4C
In this section, we review the novel loss functions of Di4C and the mixture model given Section 3.3 from an algorithmic perspective, and we provide a set of techniques to stably train it. Specifically, we introduce techniques to make the computation of the loss functions scalable through Monte Carlo integration and control variate methods.
Before going into the details of the training techniques, we introduce two auxiliary loss functions, which we can use in addition to Ldistil and Lronsis for practical improvements. One is the datapoint loss that directly computes the negative log- ligliike with respect to the data distribution (e.g., Austin et al., 2021, Eq. 5), which we can use when we have access to data qu:
Caata (0;t) == (e ) log Poje (30) (200)2)))
The other is the following marginal loss, which is easier to compute, under the assumption that the teacher model sufficiently w.d learns the true marginal, i.c., Poje = aop."
D (. (-12x) D ((laz))
mmarginal (0; 4b, (t, = Exprr
Σ
DKt. (Pop
(8)
A.1. Surrogate of distillation loss
Since the exact evaluation of Ldistil with mixture model seems intractable, we consider an upper bound of Laistil as practical alternative:
Laistil (0; rb, d) Enggrrs DKi (P (2ss)) (x Jars; A)D)
E Eus Ex
DKL (Pojs of
(xs)) |l P8 (- A)]
D
< Ex
DKL (poj and (+10s)
Pojo (-106 21) =: Eaisaa (0; vi rá, rs.
Here, the inequality is given by the convexity of KL divergence (see Proposition 3). The upper bound Edistil (and then Edisti)) becomes zero if the student denoiser coincides with the teacher for the time interval [0, 8). regardless of X. Therefore, the use of this upper bound feasible if pd has enough expressive power.
A.2. Surrogate of consistency loss
We consider Lconsis in this section. As p²1,. is more "reliable" than Ph (since & < u < f), we consider only the gradient of Ceonsis concerning 1 and ignore the gradient coming from ph Therefore, we conduct stochastic gradient descent on with the loss
mg(0) (r (-). || (- H (x o w., (-12x),, (ax)) + const.,
where sg(+) is the stop-gradient operator (van den Oord et al., 2017) and (p, q) = Egpep - log ((x)) the cross entropy between p and q. We hereby ignore the constant term in (9) and consider how efficiently compute the cross entropy term.
Most naively, by using finite samples (!) ......"! (M) rid Palu p xx) and A,...., AN rid d, we can approximate this cross entropy by two-fold Monte Carlo:
sk(())
H (p sx(") a/u
Puje( x.), p.)(((((()))))
M
log 2 (2x(1) (x)) =
Σ J=1
log
N Σ poje (a060)
laze; 2x)
(10)
M
Although the value of each (a. Iar Ao)) ( we can exploit the log-sum-exp structure:
(ax
I Pale
odd (ar ( Im Ax) can be extremely small due to the D-fold product,
Σ Pageearl\paree poi Ai)) log im
N
(2:) (os::>)))) 2
log
Σ imm
exp
log Pajt Eog
0.d
log-sum-cxp
13

---

Distillation of Discrete Diffusion through Dimensional Correlations
which is implemented function with additional stabilization avoid under/overflows in some of the common numerical packages including Py PyTorch. See (Blanchard et al., 2021) for details of numerical properties associated with the log-sum-exp structure.
Dimensionally independent control variate. Although naive Monte Carlo sampling with a sufficiently large sample size can approximate the left-hand side of Eq. (10) well, a small batch can cause high variance in the evaluation of the expected values. An established way of stabilizing Monte Carlo integration is to use so-called control variates (Glasserman, 2004; Oates et al., 2017), also known as baseline in reinforcement learning (Williams, 1992). To estimate an expectation TE. [f], we can subtract another function/random variable g. called a control variate, whose integral we know or can compute more precisely than Monte Carlo, and execute the Monte Carlo for / f- 9 by using the decomposition E[ E[s g] Elg). See Section E for more detailed explanation. As concrete application of this technique, we below propose the use of dimensionally independent control variate.
We first exploit the compositional form of pg( Psju Puje( (1)))), which is more informative than x. the pure samples in the Monte Carlo approach. We can write it in expectation follows:
Palu
Puje (-1,) = E Aat Aauerp (1o0)) ( (-Ixx;
(11)
To simplify (11), let us denote q2 = Paju (- Rai : A) and q == [0"] with 1j = (I A). To construct an efficient control
25(
variate given g, need function g such that (i) it reasonably approximates 12 (+(ee) and (ii) Enzin [g(x)] is easy to compute/aprroximate. One such example is the product model defined as
D od Paje
12 (x = II
(+ke),
(-2x)) = Paje (-(z,) ) = Ex
[xie 9.d Pale (-120; A)]
(12)
d=1
We defer the explanation of how (i) and (ii) are satisfied to Section E. Given control variate p4 (1 we can decompose the loss computation:
H Hqq (+(x)) Be,og log ps1 (x. (z;) + log P. (x, [2:)
+ Eny [H( (a.
(13)
Monte Carlo by sampling a,
Monte Carlo by sampling
Here, the first term can be treated similarly to (10), and we approximately compute the second term by sampling rj and using the identity (y", p (+lax)) = Σd=, S22_,H(Ou H(gnd, pal (a:)) (see (64) in Section E. this decomposition, we expect the mixture model to explicitly learn the dimensional correlation with the first term, while the second term stabilizes the overall
approximation, as we use more detailed information on q than just its samples. See also Section E.2 for more background on how we derive 7" and another possible choice of control variate.
A.3. Auxiliary losses
While we can use similar Monte Carlo estimate for Ldats (with random samples of æo, R;, A), we can regard Euarginal as possible control variate for it. Indeed, if the teacher network is well-trained, we can expect that its marginal approximates the true marginal as p.dd x yd, Thus, for the marginal-matching product model z given in Eq, (12), we have
Mg H (1xz) (x)). (z:)) x Cmargina v. qt, t) + const.,
(14)
where the constant term is independent of 0. We give the derivation of (14) in Appendix E.3. We then obtain a decomposed formulation of Cdata for given ~ ge as follows, by letting g = =ooee xx) and = 0 in Eq. (13) and then using approximation (14):
Ldata (0: = Leorr (6;t) t) + Lunarginal (0; th, qt t) + const.,
Lcon (0:t)= = E log (xolxx)) + log Ph ( (x:)
Here, Leore measures the difference between p" and T" and thus represents the dimensional correlation learned by the model p.. In the actual implementation for the first term feorr, we generate æn ~ Go and then x, ~ (=jo(-[za), ) and regard them
14

---

Distillation of Discrete Diffusion through Dimensional Correlations
as samples from (æog ze)) 10,r. which are required for conducting Monte Carlo. When combining Lasta and marrgnns (hoth as loss and control variate), we empirically find that mixing them Og oore (0: t) + Emarginal (0; 1/, Ut t) with some is more efficient than just using constant a;, = 0 (pure marginal loss) aze = 1 (pure data loss).
Or E [0, 1] depending
See Section F.1 for details in this regard.
A.4. Iterated Di4C training
While we generally assume that the teacher model is given by product model, the Di4C loss functions can also treat mixture teacher models. The only exception is the marginal loss (Section A.3), but we can just replace 2 with To in (14) and conduct Monte Carlo estimates. Through this generalization, we can run multiple rounds of Di4C in a similar spirit as the multi-round SDTT (Deschenaux & Gulcchre, 2025).
B. Kullback-Leibler divergence and total variation distance
Let and be probability distributions the same finite set X. The KL divergence DKL and the total variation distance drv are defined as follows:
(pI.(pll)) = Σ v(x) log p(x) g(x)²
(rv(p,)) 4) = sup (p(A) ((A) = Elm(x)
11(x) g(x))
ZEX
ACX
=EX
Here, in the computation of DKL. we ignore the term with p(x) = 0 and, if there is an x with p(x) > 0 and ((xx) = 0, we then define DKL(P) q) = 0. These two error criteria between distributions are bridged by the following inequality (see, e.g., Canonne (2022)).
Proposition 2 (Pinsker's inequality). For probability distributions and on X. we have
drv(p, g) S v V DiL
The convexity of KL divergence in the following plays role in the main body of the paper. Proposition 3 (Cover Thomas, 2006, Theorem 2.7.2). DKL (p () q) convex with respect the pair (p. q). Namely, for i e (0, 1] and probubility distributions pi, P2, ạı, Q2 on the same domain, have
(p
(kpp (1 1 l)p2 lm (1 1)g2)= < (Di (l ) + (1 (1-1) DPt. 92).
We also the following triangle-like inequality for the total variation distance of compositions. Proposition 4. For probability distributions p1 (=(-)), p2(-1y) over X conditioned on y y and 9ı, 42 over V, we have
drv (E (-1)) Eg (p E (drp:p/ddd(pp(()ppppcpc)c + drv (91.92).
We give its proof in Section D.2.
C. Continuous-time Markov chains and Kolmogorov equations
Let us discuss the Kolmogorov forward/backward equations associated with continuous-time Markov chains. While the arguments below are mostly a reorganization of those given in previous studies (Campbell et al., 2022; Sun et al., 2023), we explicitly track the continuity/nonzero assumptions used in their derivations.
C.1. Kolmogorov equations in the general case
Let us consider general Markov process over the continuous time interval [o, T] and a discrete (finite) state space X', which is called a continuous-time Markov chain (Anderson, 2012; Campbell et al., 2022). The starting block is the forward transition rate in a short-time interval. For t e, assume the following equation for the infinitesimal forward transition:
(4e)((y/ar)) &g.r + <Q(( o(e),
<> 0,
(15)
15

---

Distillation of Discrete Diffusion through Dimensional Correlations
where yee is the Kronecker delta and Qe function X x X R called the transition rate. Here, for 211 + e, have
-0/))=(0/2)=1 -= = Σ (6y,2 + 2())))))) +(()
= -0)1((/>) + Σ 201 +
This means that we have o (z/22) = E; 2) where 0r is right-derivative regarding t. Under the condition that Q, is continuous over [0] (assume it is continuously extended to / = T, though it is not necessary right now) and 9t|s is continuous over t E [s. 77, 9t) becomes differentiable over the open interval (from a general fact in analysis (von Eitzen, 2014)) and we have the Kolmogorov forward equation for t E (s,7):
0.q1)((11)== = Σ (2)))
(16)
Now, let derive the backward equation. For < 8 + S 1, by using (15), we have
())r()) Σ (g1
Σ A1s++ (212)()) + cQo(z,z)) + o(e)
= - +e Σ (1. o(c).
Thus, by additionally assuming the continuity of 9t|s for [0, we obtain the one-sided derivative a (v(a)) Σ, (1)=))))) When combined with the continuity of Q. similarly to the above argument the forward equation, it leads to the backward Kolmogonov equation for 8 E (0,t):
0.91)()12)) = -===-= [.
(17)
To summarize so far, under the assumption that Qt|s is continuous for s, with S St T and Qe in (15) continuous over j0, T), have the two Kolmogorov equations given by (16) and (17). Note that all the Σ. finite sums because of the finiteness of X.
C.2. Kolmogorov equations for factorized forward processes
Let consider the case where X' SD and a. (x:) follows dimensionally independent forward process with transition rate Q Namely, suppose
L (t1:((//a1) = dya, + , + o(E)
(18)
for each d = 1, I....,D D and t < t + e. In this case, we have
D II
D
a+
Le (
8y + Σ
x 20 o(e)
(19)
d=1
by simply expanding the product, where x'd E SD-1 given by omitting the d-th entry of æ. From (19), the transition rate for æ; is given by
Q:(y, Σ (v a )ord
(20)
as in Campbell et al. (2022, Proposition 3). Let assume continuity regarding the forward process in each dimension:
Assumption A. For each 1,...,D. D. there exists a function Q : SS x + indexed by E [0,T] satisfying (18).
Moreover, for any fixed x. y E S. (1)x)) is continuous in ((s, 1) [0, S t} and Q4 (y. x) is continuous in [0, T).
16

---

Distillation of Discrete Diffusion through Dimensional Correlations
This can be satisfied by broad range of forward diffusion designs, including uniform, absorbing (masked), and discretized Gaussian diffusion (see, Camphell et al., 2022, Section E).
Under Assumption A, 9t]s and Qr for the original process Dn are also continuous since 9r)s (1/=)) = Ia L (v"la and (20). Thus, we can apply the argument in Section C. to obtain Kolmogorov equations (16) & (17). Moreover, we can show a favorable property of the time-reversal process. This is just a re-formalization of a well-known fact (e.g., Campbell et al. (2022, Proposition 3) and Sun et al. (2023, Proposition 3.2)).
Proposition 5. Let S == {æe E SD (x(x))= > 0}. Under Assumption A, there exists a function Re : SD x s., + indexed by E (0, T] such that
(a) we have ut-eily x)) = , + e Ru x) + ole) E SD, x E SP,, and with 1 - e 2 0, and
(b) Re(y,)) x) can be nonzero only if and y coincide in at least D - entries.
We give its proof in Section D.3. As one can see from the proof, the time-reversal transition rate RG is given concretely by Rr(y, Ø) Qe (2, y) (1) //:((r) when æ j and gr (a) > O. Note that the ratio gi (1))/c((r) is treated as a discrete counterpart of the score function (Sun et al., 2023; Lou et al., 2024).
Let us add another regularity assumption:
Assumption B. For each d ...... and z,y E S, Qy x)) is differentiable for E (0, and the derivative 0,Q% (y, can be continuously extended [0, T7.
Note that usual choices of Of regarding / including the time-homogeneous case Q. Q and the noise scheduling Q. Q with smooth 33 (Campbell et al., 2022; Lou al., 2024) satisfy this assumption. Finally, under these two assumptions, we can formalize Theorem as follows.
Theorem 3. Suppose (æe)OSIST satisfies Assumptions A & B. Let Palt be a product model with the correct marginals, i.e., (2, (x))= = -Ta= L (x/()2)) ( for s < t. Then, there exists a constant C >> such that, given timesteps t, = =iT/N for i = O,...,N, we have
C N
drv (go, Ear-ur [Ptalt: o Ptilt2 O .... O PiN- (-12T)) S
(21)
Furthermore, there exists an example of (ax))ssss satisfying D |S| = and the same assumptions such that the lest-hand side of (21) lower-bounded by c/N with some constant e > 0 for sufficiently large N.
This theorem basically says the min-max convergence rate of the analytical sampling is I/N. We give the proof of the first half, i.e., Eq. (21), in Section D.4. For the latter half, provide the concrete version in Proposition 6 in the following section.
C.3. Lower bound of Theorem 3
We shall provide an example that yields an Q(1/N) error between the analytical and true denoisers. Our example even satisfies the following stronger assumption, which is often used in theoretical analysis (e.g., Campbell et al., 2022, Assumption 1):
Assumption C. For any t E [0, and x E (x)) > 0 holds.
Consider S = {a, b} and D = 2, where the state-space is given by X' = {aa, ab, ba, bb} by omitting parentheses. Consider the (forward) Markov process given by the initial distribution go = (5a + d and the dimension-wise time- homogeneusu transition rate Qf x)- = 1/2 6gz for = 1, 2 and x, E S. Under this setting, the forward transition probability is continuous and satisfies O. (- (a) 0f (-1a) as a vector-valued differential equation, so we have, for > 8,
1 L
1 2 -alaaaa)
0.gh. (ala) = - ------- - (ala) + (bla)
(ala)
=
By solving this, we obtain o (ala) _ + e (t-s)) for 2 s. By symmetry, we generally have
(ala) = (blb) = (1 + e ((-s)),
(baa) (al(a = 2 - e (t-a))
1
(22)
17

---

N Distillation of Discrete Diffusion through Dimensional Correlations
This is special of uniform diffusion and clearly satisfies Assumptions A & B. Although the singularity of go violates Assumption C at time zero, will, however, work with the singular go for simplicity of computations. The following proposition gives the lower bound discussed in Theorem 3. If necessary, we can replace T with T + 5 and consider x 2++6 to match the time intervals.
consider the time interval [6, 7 for some >> 0 instead of [0, 7] ensure d O. We
Proposition 6. Let (I1)6ςiŞT be the Markov process defined above and Palt be the product model (x)e( = ILa- qaje (z)) for s < If we let N 2 2(T - 8 an integer and t; = i(T 6)/N for = 0, O,...,N be timesteps, then there constant > 0 such that
drv (gs, Earrar [Ptoles o Ptalt2 o .... o PtN- ilex (-1ar)]) 2 c
(23)
The proof is given in Section D.5.
D. Proofs
D.1. Proof of Theorem
Proof. For simplicity of notation, let P be the denoiser given by the teacher with timesteps ta < tnt1 <... i.e.
4 P
Pi..IT == Pealen+s
de so that we have Tt, Exy (t Note that we can just set PiNIT Pi (-1p)-11a)) 6x.
Also, let Pu,n = EwTwrr pole. P. ((( for I, where PO,N is just given by PU.N
Euyorr p ((( ((((1)))) We first compare Po,n and pa,0+1 with the consistency loss.
For each 0 < <<IS T, we have
Loonsia (0 re, 1,e,0,uu u,,t = BRorn DKL (poja Pw (-(x,) ll 1
DKL (Barnn pii Paje( (12za) Eaiwn [po (12.))
from the convexity (Proposition 3). If we let u = in and = ln-1 for some 1 S n < N, we can see
Excw poju P (-)o(- Eer Poje. Ph D ( Po,
and En, un [p0] (1122=) 1o,++1 hold. By using Pinsker's inequality (Proposition 2), we have
1 (Po 1o. < Cconsis (0:4/, rentes O, bny 2/22
1
drv (Pv. Po,nt))
(24)
/2
/2
From a similar argument, we have
Laistal (0; rt, = Eis, NTi
DKL (Poje, (.122n) 11,,,, (-1221 )) (ro po,1)
and thus
1
Crv(ro.Po.))
Dro ll 1/2 <
v2 (0; (1)
(25)
/2
By using the triangle inequality of the total variation distance, we obtain
N-1
(rv((o.Po.N) S drv(ro.pa.)) +
Σ
drv(po,n: Po.
n=
Finally, applying Eqs. (24) and (25) its right-hand side yields the desired inequality.
18

---

Distillation of Discrete Diffusion through Dimensional Correlations
D.2. Proof of Proposition 4
Proof. Let first consider the case of q1 = 92. Then, we have
drv Cav(Byyg ywu Ip (-1))) B By (p2(-lv)))
Pi (xly))1()) - E Epa(cly))n(1)) 1 -
Σ 8
(pi (xly) pa(xlv))a(())
8
Σ (x -2 (zlv)) Byrgi
(26)
where we have used Q1 2 0 in the inequality. On the other hand, if p1 p2, we have
drv (By |22(110] Eyrag (p2()1)31)
Σ
Σ p2 p2(clu))(()) - Σ Epa(slu))2(0))
1
Σ
Σ
p2(el))(( -2(1))
2 2
2 x
x
y
2 p2 (=(x))))a - g2(1)1 = 2 Σ |n (v) 22 (2() = drv (2),
Σ
(27)
where we have used 2 0 in the inequality and >n, P2 p2(2 (<|y) = in the last equality.
By utilizing the usual triangle incquality of drv and inequalities (26) & (27), we obtain
dTv (Byran loi (-lv)] Ey-oo (p2(-l))
I drv (Byng Ip (-1v) Eyrag (p2(-lv)) + drv (p2(-1v) Byr (p2(-ly))
I Eyrar (dr((rr(-u))
+ drv(n1
which is the desired inequality.
D.3. Proof of Proposition 5
Proof. Note that, by Assumption A, 9els is continuous over t(s [0, S f}, and Qe given by (20) is continuous over [0, 7] and satisfies Eqs. (15)-117), as mentioned in Section C.2 just after Assumption A.
Let us simply write x X instead of the bold style æ E SD this paragraph. We consider only x E X' such that (e(x)) > We follow the argument in Sun et al. (2023, Section B.2). Let us consider the conditional probability (namely, the true denoiser) (-1z) for < t, which is uniquely determined since (z) > 0. Then, we have
0, 2s(20 ( (0.4))()))n, + (0 (x()
0.q))(112)
(x)
41 (x1() Σ (v.,2)))a qo (v) Σ 2 (aww
(28)
ge (x)
where we have used the forward Kolmogorov equation of q/ given as
(z) = Σ (z))((((w)))(u))) = ΣΣ (((=,,)gn0(22)))a)m) = Σ > (z,
10
20
for computing 0.g. and the backward Kolmogorov equation for computing 0.4(p.- By taking the limit - t 0 in (28), we obtain lim a. (ul=:) = (v)
ze (x) (x( y) if y * Jr given the continuity of f/e and Q.. Then, from Taylor's theorem,
we obtain backward counterpart of (15) for y + x as
gr (v) (x)
h- h x)
(x, + o(e),
<>0 O.
(29)
Since E, (z(() holds always, also have that (2-c)((((2))
= 1 Eleex o(e) for the coefficient R (v(v) Qe y) for y o and Re(x, Rr,z (e(a)
Eyyz ((()) Q+ (z, y). Therefore, we can prove (a) by letting R2
We can see (b) from (20) and the concrete form of R..
19

---

Distillation of Discrete Diffusion through Dimensional Correlations
D.4. Proof of first half of Theorem 3
We first prove the following auxiliary lemma replacing the o(e) term in the backward transition by O
Lemma 1. Under the same setting as in Theorem there is constant C such that, for any t (0, e E (0, t). and X with qe (x) > O, we have
Ce
dTv dro(@g-aa((()))----((11 <
(30)
4: (xxx)
Proof. From (28) and Assumption B, (=(c(y))z) for s < f is twice-differeeniaale with regard to and (z( mr) can be represented as a polynomial of the function values of qa, Qa. 9t/e and 0,Q.. Thus, there is constant C₁ depending on [S]. D, sup,, 2, Qx(z, w) and sup,, (z, w) such that g (x 95 (y)²) & C for any 8, t, y. x (note that q/ and q, are within [0, ID.
Now that OzAsst can be continuously extended to s E [0, from (28), for each E (0, c E (0, and x, y E SD with qo (x<x)) > 0, Taylor's theorem yields that
( (3)
Ci e
(g)((1 δy ey
2
S
(31)
2
2q (z)
for a certain E ct,
Let us next consider the marginal-matching product model Pi-et- For each d, if y" + x², we have
(x=)) elo( (1,,
Σ y videso
(o Eli, (1v" 2x)
Σ (q-ade((" ()1))=))))))))))))))))())))))1 -eRc(( (2)i))))...... )x0)
1S|--101
(32)
(
where the second equality comes from Proposition 5(b) and the incquality is from (31). If yd = x", since p Pi-cje (1(()=))))
1 - Eyeza lvi (z) we can use (32) to obtain
d pi (x)))=
1+E Σ
Re( x xx))
S Σ
- xe , )x))
|S| PS 2
29 (x)
From (32) and this, by defining R/ S - R as (v Rev x vid), x)) for vd + xdd and Rf(2x)) - Eyergr R4
there exists a constant C2 > and a function Ad: + IR (for fixed and x) such that
2²
d (0=))(0(>>) - <
-
A (3
sup MeS,
(4 S C22
(33)
(x)
Therefore, we have
(y)))))) = II (P((())
(x)))
d=1
Pa((66g 20,, Re( ((F00) ( 2² x
= 8y +e
Σ
(x)
d=1
Remainder term
20

---

Distillation of Discrete Diffusion through Dimensional Correlations
where Pk is certain polynomial of 3D variables for each k. Note that, if y A æ. (2)))) Oyid can be nonzero only if V + x and y x v'd_ In that case, from the definition of (y)), we have
Pr- Pr-cle (v)l) = cR (v²) + (Remainder term) (y, x) + (Remainder term).
(34)
This equality also holds when y and æ differ in more than one entry, since the coefficient of becomes zero in such a case. and (y, x) 0 from Proposition 5(b). Since the inputs for each Ps are all bounded, we have
e +
22D
(Remainder term)= C3
Σ k=1
CDD q((x))
(35)
qi (x)))
q((x))
for a constant C3 > O. By combining it with (31), for y & æ, we have
(1)1))))))) - Pt. Pr- (1)22) < 1 (1//2))) - (y) + (e)
- C1 2 + C3D
2²
2D
2D
S C4
+
(x)
(x(x))
((xx)
((())
((<=))
for a constant C4 > 0. particular, we have
drv drv(gr-ee Pt = 2
1 [((/)))) - P(-00((12))
1 (1)()))))) - (1) + 1 Σ (t-cl(((/(z))
1+ Σ D)-c(((1(2z)
2DD
62
2D
> Mt- (())))))) Pt-r|((()2))) S 2
C.
((()))
(1(x)))))))))>))))))))))))
(((x))
qi
(x)))
for constant C > Now, we can assume that C3 2 1/2, by adding positive number if necessary. Since dry is bounded above by general, we consider two cases:
2 (a) If (2(2) 2 1. we have drv d (-1x)) < 1 2Css [((
since 2Cg 2 1.
(b) If (x)))
1. we have (1v(
s Cs
+
2D
)
220s 2²
since 2² (() 2 20 (())
(m)
q:
Therefore, we finally obtain (30).
By using the lemma and Proposition 4, we can prove the theorem.
Proof of Theorem 3. For each O,...,N N, let us define the compositions
Pojea (-1z²) = óg: Poje. = Ptolts o o... Pte-ilts i I. N.
Note also that, for æ with or (z) > O, we have 9t. (-|z)) = grslea ....o o gey altx (-1z) from the Markov property of
the reverse process. Indeed, for 8 < < u, we have (z)e) = (2 y) from the Markov property of the forward process, so, for z with qu (2) > 0.
(2 y) qt. (3)
Σ (al =
(1(y) (z)
gs (x. ()9m)e(2(1)
( y)qu y) qu(2)
(z)
En ao,t,u (20, (x)
as.u (xr, 2)
(2/u((r) (2) (z)))
(u(2)
21

---

Distillation of Discrete Diffusion through Dimensional Correlations
where have implicitly used that (y) > holds for y satisfying (z)(()/2)) (given (u(2) 0). By using the inequality recursively,
prove the aforementioned identity.
We prove the desired estimate by exploiting the compositions. Recall Oo = Exr-or [40 ax (-12r)] What we want to estimate
is drv (Rayerr [aoeex (-12r)] Fo [Potew (-) ((ær)))) ). We bound the distance with the following triangle inequality:
drv ( (Exr~qr [90|en (-|xT)) [errorr [Poje (-|zr))
drv [Pope It.lew (-123)) Bay [polects "tsts (x (-1rr)])
(36)
Let us bound each term inside the summation by using Lemma and Proposition 4. First, since Polects Po o Pt.lte4 by letting p1 P2 2o in Proposition 4, we have
drv (p [Pope Qtalex (-12r)) Berr [Poleet2 -i=-lle (-12r))
S drv (Evr or [dt,ee (-1xT) Berrwr (Pea ti Qetilte (- (-1xT))).
(37)
Second, since 1ts+ = Qt,le441 4ts4 IN. by letting = 44,41 = Exrrer [4 IN (+TT) in Proposition 4 (note that the indices of q1, 02 here are different from time), we have
drv (Ber war [gelew (-120)) Berrarr [Peltet qeeilee (-(xr))
Ex (dTv(qe (-1z), Ptelecus (+22))
C(t++1 - t:)?
CISIPT2
Ue.4 (x)
(38)
4:,41 (x)
N2
9tss1 (2))0
where we have used (30) and 4i+1 - 4; = T/N in the last inequality. By combining estimates (36)-(8),, we obtain
CISIPT2
C|S PT2
drv (Eve [gopen (-1xT)) Ber [Potew ( ((T)
N2
N
which completes the proof with a replacement of the constant factor.
D.5. Proof Proposition
Proof. Consider the analytical sampler Psje(zw zy) q ( ( (w (wly) for < t. Note that, because of the symmetry between a and in do and the forward transition, the distributions ge or those given by the composition of Pajt are also symmetric. Thus, the probability of aa recovers all the information of the distributions we consider over X.
Let compute several probabilities regarding Ustt and the analytical sampler through (22). First, note that (abb)))))) dujo (bal)) 0. Therefore, we have
Goje (aalaa)))) = Arjo(aa (aa)qu (aa) Tt (aa)
2rjo (aa) (a)
{1 te (1+ e-)2
(39)
qrjo aa (a)) (a (aa) + 9:jo qabb(aalbb)qo(bb)
+ + -1-- 2(1 + e-2)²
(1 - e 022
Zoje (bblaa) = 2oje Roje(aal/un (a) =
(40)
2(1 + e 2t))
4oj (aalab) aale (bblab) 1
(41)
where (41) is derived from symmetry.
By using (39)-(41) and the general fact (for Markov processes)
e (2. v.) 00...<(a00 Bay vt))
(xx/200 2r.).
4s a (xe, (20 (aoo)q) (x0 (x.)
(42)
qu (Do, xx)
4:jo (x; (zp)
2rjo (2 (20)
22

---

Distillation of Discrete Diffusion through Dimensional Correlations
for 0 S S I, we can compute ga/e (elt(-laa) for any s E [0:4] as follows:
Lote (olr(aalaa) Volc(aa an) M.joe (aaa (aa (aa, aa) + 20j (bb)aa))gooo (bb, aa)
(1 (1+e) 92 1 + +
222 [
+ 01=
+ (1 - 0- 11--0-9)211
1 + +
[1 + 1-
2(1 + e 21)
_11
(11+5993 + - (1- + 01=)
= (1 (1+ (0=)-20)( + (1-o)2
(43)
8(1+0 2
gatr (bblaa)) = 20/o(aa aa (.100 (bbl aa) + goll (bblabb (. 1.,o,
(1+e
7 - e )- (1 e 01))
(1 c 1)2
{1 + - )) (1 (((0)))
2(1 2-
(1 + 1-
2(1 + 2-) 21)
(1
(11---)² + (1 1 91 - =)
200e-20)00 20) - - 1})
(44)
8(1 2-)
4(1 2)
[st[ablaa)= = Zole (baa) 2 (aa) = 221 (1 (slc(aalaa)) - (1/o(bbaaa)
(45)
1
(1 (2 ((1 + c (1-)))) + (1 - e (1()))))
2
8(1 + 2-21)
(1+6-20)1 20 (1 + e --(( 2(1-s))
c-20 + e (((-a)
e 20
(46)
2
4(1 + 2-2))
4(1 + c e-20)
We can also compute qstt (aalab)) = (bbba)) as
Goje (=je(aaaab)
( ab) + 20 40 10, ( (bb, ab)
1 (1 + e -2 ( (1+ - (t-ss) - E (1-
( - (1 (1 ((-ss)
+ 2
2
( + (- e-
1 - (1 +
((1 000 + - (1 (111--22(0))) --(((=)))
(1 20 - 2(((-)))
8(1-e e 2t)
e
1 e 2s s(t-s
(47)
4 4(1 2t)
Let us now compute the probabilities regarding the analytical sampler. To make it simple, let (xje(z * (-) = (xaa) + quje (xbbl)) represent marginals; q=]= (+yl+) is defined similarly. By using this notation and (43)-(47), we have
Patt (aalaa)) = [s)r(a * laa) (s)e))/((((aaaa (salaa) gaje(a+ laa)2 = (gule (aalaa)) + goje (aje(ablao)) )a))
(1 20) + (1-)))2
(1 (1+ (1 + e 2((--))
2
4(1 (11ee-2) 21)
2
4(1 + c 21)
2(1 ((1+e-2) + (1 ) + (1-s)22 - (1+e + e 2(1-))))
4(1 c1+e-22 21)
+ (1 + E ((=s)) 2
(1+e (1tt (1-s)) 2
+
(48)
21+e-22 21
2(1+e-21
(bblabb)
[b/ * (a))g//(xblaa = (b laa)² (bblaa)))))) + 20/1 (bal (bal)a)) )
(1 (e e )1
(t- s)
21)
20 (1 + E -2((-8))
4(1 + c -2)
1(1 + -2)
2(1 221++ e2- + (1 ++ ( - - (t=s) (1+e e 2(1 ())-))))
1(1 + )
(1+ (1 + 6- 22 (1-50)
(11 (1-e- (((-s)) 2
(49)
2(1 + 2
2(1 + 212
23

---

Distillation of Discrete Diffusion through Dimensional Correlations
Let us compute the sum of (48) and (49) we it later:
Paje (ieeaa/aa) + Puje Paje(bblaa)
(1 +1+6 e (t++)) ( (1 + e (
2
+ (1 (1-
2
2(1 + 211 2
2(1 + 2--21
+ to (t- )) + ((1 - -e - e-((=))))
4(1 + -e- 21)2
(1+e 26 +e- te (t- (1= + (1+e e (t+s) e (t-ss)
4(1 + 6
(1+ e1+e-20)2 + (c-(t)))
(c+s) + te {()
1 + 2(1 e-122
(e-(t+s) + ((-()=))
(50)
2(1 (1++e-21)
2
Next, (aaa(a))ab) is the product of two marginals (s lab) and (=)bb))) which can be computed as follows:
Paje (a * lab) = (s (ala, a) + 20 (obc((bbbb)) (10 a)
(1 + ) + (t-s))
(1
) + ((-(s)))
2
2 6-t
((+ e (1 - c 9) (1+ +e )] + (e
4(1 e--21
((ts) (t-a))
+ c (t- - e (I+s)
2(1 E--1)
2
2(1 2-
(a) = Paje(a (ba) Paja(b * lab) 1 - Paja(a lab) (-((-)) - -c- (e-(s-)) (e-s)
2(1 e 2-2
where the latter derivation is from the symmetries of the two dimensions and two characters. By using these, we have
(ala(aalab) Paje (a * lab)ps (*alab) 1 (e- (t-(t-))
- (-211--
e (1+ss)
(51)
2(1 -
a0l---21 e-2()
Let us consider iteratively denoising from or by using Patt- For an e > 0 and nonnegative integers n S T/e - 1, define
pf
pr, pr-(n+1)e = Ewrph-n [PT- (n+ DofT (11z)]] n = 1, .....
Our goal is to estimate the difference between pr-nne and -r- for each n. Let us fix n and set I. T - ne when computing pi_ in terms of pt. Because of the symmetry, pf(aa) = (bb) and (ab) = pf(ba) = 1 - pf(aa) hold in general. Therefore, by using (50) and (51), we have
+ Pt-ci (aa) = Pt Pt-c (oalaaooaaa)) (aa) + (aac(((albbb)p(((
+ Pt- (aal ba)p
2p: -/ (aa { -pi(aa))
Pt-ele(aalac )pi (aa) + p
Pt-el((aalab)) + & (aalaa) + Ptteeje(bllaa (a)) 2pc=o/e(ualabbl (b))) (aa)
(e -2 12 +
(e- -
(2) ())
p (aa).
(52)
4(1 - 2r22
2(1 + 2-222
2(1 - c 2022
To compare it with et-es we also compute a similar recurrence equation by replacing p's with g's and using (45)-
at (aa) = -t-rit ((aajab) + (q-=ci(aalaa)) + ()b()x(bbaa)) - (aalaaaaa
-2e -2(-()))
-2e + --
- 2(t-c)
+
q: ga(aa)
(53)
4(1 - -e-2)
2(1 + 2
2(1 - e-
24

---

Distillation of Discrete Diffusion through Dimensional Correlations
Let us now compute quantities regarding the coefficients in (52) and (53).
2c 2((-c)
1 2t (e - e 02
(1 (1-e- 2222
(e 2e E x-((1-e))( (1 E-21 (e -e-e-e (2---))))
(1 c 2121
(e 20 -e - (()-c) - E 2((tc) +e - - -(E - (2)-
(1 6-222
(e-(t-c) - e (2tr)2
e 2t
(e -
(54)
21-22
(1 e 20
20 + ((1-c)
(e - + e (21-022
1 1+ 2-
(1 + 2-
(e-20 + e--22 -2(1-c))(1 (1 + 22) (e + (26-0))2
(1 e 21/2
(e 20 +e (((-() + e 2((tc) +e e 2(21--)) - (e + e (2)-
(1 + 22)2
(e (t-(t) - (44r1))
c 2t
(e - -0
(55)
(1 E 2-te-222
20 + e (11-c)
2 c (((-)c
e 21
1 - e--2 21
(e 22 + e e-((t_e))(
2-2
c 21 + (e-2e c (()) 21-
1 - e -t
-2+ 2(e 20 E 2)) 2(1 E 41)
6-1 11
= 2 2(1 + 2((((())
(e 22r - 1).
(56)
1- ---1
We shall evaluate the difference A = qi(aa) -1((aa) by using (52)-56) as follows:
c 26 c 2(t-c)
(c- - e -012)-))2
A
4(1 - 2121)
4(1 -e -2-20
e e2 + - 2(1--e
e-2 - --((-)))
(pi(aa) + +1)
2(1+ (11+e-2) 21)
2(1 22
(e- + c (2)-1))2
(0- _ c 1)
Pf (aaa
2(1 + e -21/2
2(1 - e 2012
4(1 21/2 (e - -0-92+ + (e*+ x+2e+" te- 2(t-rc)
e 21
+ e-2c - (((--c)
A
2(1 + t )-2)
2(1 2-21)
2c + ((1-()
(e + e (2t-c))) 2
- 2(1-c)
(0- c (22-
+
pf pf(aa)
2(1 + E 22
2(1 + E 2022
2(1 - -E 21)
2(1 - E 2221
c 21
+ (21
202 (e -6-92+ +
1
(e 22 - -1) Ai
4(1
e 21
e 21
+
(e ")- (aa)
2(1 + E -21
2(1 E 222
e 2
e -2
② -v((ca)) )) (e (
1
2(1 + e 2-21z288 Pi + 2(11 2-200
1 + 2(2-
+ 1+
(e-2 -
(e-2- 1) Ai
(57)
-- 11
25

---

Distillation of Discrete Diffusion through Dimensional Correlations
Since pf((a)= (b06)) S 1/2, we have
e 2t
C 2t
1
2(1 + e 21)2 Pa(aa)
+
(aaa
2(1 e 2122
2
1
E 2t
e 2 2t
e 2t
min
2
2(1 2(1+e-211 2(1 - e 2022
4(1
20)2
2 Additionally, as the Taylor series of (e" - -0- = c²c + E 2e - 2 is given by Ek=1 [26]] (2e)ah, we especially have - 9) 4e2. Thus, we obtain
e 2r
c-2
(1
pP((aa) )) (e' (c'- -
2(1 + t 21y2 (nnaPi(an)) +
2(1 e 2122
2
c e-2 2t
e 2
2
:622
2
(58)
4(1 c 21)2
(1 21
Also, since e 2x -=21 21- 11-ee 2e, we have
1 + (2
2(1 + c((2t-))
4
1+
(e 2c -)
1) -1)
2 1 1 1- --e-st" e.
(59)
1 e- 11
1- 1-
Suppose we are working on the time interval
T) for some 8,T > 0. Let us take E S 6/2; then we have
4
4
c 2
1
c 2 1
> 0.
(60)
1 c
4t
8
For (58), we have
e (1 (1- 21/2
2² 2
(61)
By combining (57)-(61), we first see that A is nonnegative for all T ne by induction on n 0, l,... (assuming S 6/2 and t E [6, Then, we obtain the following simple inequality:
Ai_ 2 (1 (1- " + 2 2T
By recalling that / = T - ne, we can rewrite it as
(1-5) 5)
11+
t) n A + (1 (1- (n+1)e 2 (1 (1-
E) ((++1)
2T ?
Since A7- = 0, we have
AS [
n1
2T 2
E
2T 2
(62)
8
R=1
k=0
Since n S T/e and
and (1 - 1/2)"" is increasing over x > 1, for 1: = O,...,n n 1. we have
(1 (1-
E 5 (1 ;) ")²²()²))) (1- ;) T/c (1--))01)
8/c) T/5
(112))
2) =
2 T/5
2-166
27/5
>
where we have exploited the assumption e S 8/22 (so that 8/e 2 2). By applying this (62), we obtain
A 2 (2 2T ne?.
Now, let c (T 88) for the given N. Since N > 2T and thus S 5/2, we have
2T (T Finally, as drv(gs. (T (T-8)/N) 2 Aj, the constant e = (21/6)) - 2T (T 8)2 satisfies (23).
5)2
2T N²² (21/8 - A = Ar 2 (2 22
N
26

---

Distillation of Discrete Diffusion through Dimensional Correlations
E. Control variates
When we want to compute an expectation [x),, instead directly doing the Monte Carlo estimate NV Σ f(x;) ~
E [x)],, we can find function g = f such that E [ax)] is tractable and then do the Monte Carlo estimate for the remainder term:
N (((m)) - g(x))) Blg(z)) x B[S())).
(63)
N
This left-hand side is still unbiased estimator of [x)]] and idcally has lower variance than the vanilla Monte Carlo estimator 1 Σ (xx) if g f is a good function approximation. The role of g in (63) is called a control variate (Glasserman, 2004; Oates al., 2017).
E.1. saarginal-matching product model control variate
We briefly discuss how the product model p" given in (12) satisfies the following favorable properties (already shown in Section A.2) for being control variate:
(i) it reasonably approximates (+1x), and
(ii) [x-]] is easy compute/approximate.
For point (i), note that pd is defined as a product model having the same marginal as p". Since dimensionally independent modeling (when combined with multi-step sampling) works in Theorem 1, p" should approximate y to certain degree; also Lemma for quantitative understanding. The remainder p" 7" then be regarded dimensional correlation captured by p", with which we conduct usual Monte Carlo integration.
Regarding (ii), given a product distribution p(æ) = IT (" Monte Carlo integral using samples of as
over X = SD, we can indeed compute H p) with a
H(g.F) P) E,, Engnol loll-loog log ((r,())))]
BE, ~g log p(x.()
EH = E,,
Σ d=1
q" log (og
(64)
While it still requires Monte Carlo with to estimate this. it utilizes the product structure of each g" and to exactly compute H(q",p) P). Thus, we heuristically expect it to be more accurate than the Monte Carlo estimate using samples from
E.2. Derivations of dimension-wise computable control variates for mixture model
Convex upper bound as control variate. To simplify the notation and situation, suppose we are given probability distributions g = [g"] and pd Ex [p where g" and p are product distributions, i.c. we have
q" (x)
II n
(x)
II d=1
p (x
d=1
By letting H be the (cross) entropy, we want to minimize
(Kl(alp²)) = H ((..")) - H(g) = Eaang log log (m:) Eerngl log ((:))
Since q is fixed, simply want minimize
H(g.p") Exrny log (x))] Ex [-- poogp((())
with regard to 0. However, it might have a high variance when we only sample æ ~ g and execute Monte Carlo. One option using the following upper bound like a negative ELBO given by Jensen's 's inequality (convex inequality) as control variate:
llggp" p (x:) - log E, 10 (x=) s EA |- log 8 8-> (>))
27

---

Distillation of Discrete Diffusion through Dimensional Correlations
Indeed, its expectation regarding
is dimension-wise computable
Eæ Bx log pd (x)
Ba Ex | log 8. (a:) Exnggr log pe (xx)
D
Ex
Beinggn [- log 0.A.d (x*)] = Ex
ΣΣ
(i:d((x
log (arddaa
d=1 2d
which does not require Monte Carlo sampling of æ. Overall, we can decompose the computation as H(q. p²) = Ea log pd (x) - Ex . (x))]
E Exgin Ex
|- log 0.2 (x)
Monte Carlo approsimation
dim-wise computable
Marginal control variate.
The previous convex upper bound seems good, but since
Ex po (x)] E5x (..
inf (q,
might be very loose bound (we want the mixture to outperform the best product distribution DP-A). make it more practical, we can consider its dimension-wise tractable lower bound as follows:
E Rang Bx - log " 9.> (xx) = Byy E Bon Ex (- log (xa)) 2 ET Σ Banngnn log Bx [x (p (xa)
d=1
which is given by Jensen's inequality as well. Therefore, if we define the product distribution
(x))
II p (ma). d=1
(rs) = EA [ (xa))
we have bocggEE(-ooop--PPP < E Ee=gl= |- log D. (0) and this alternative is also dimension-wise computable. Since p" and T" coincide in each me-dimensional marginal, the difference between these two can be regarded the result of dimensional correlation.
Therefore, we propose the following decomposition, which also discussed in Section A.2:
(q.pp)) Exnn [- [- (' + log (x))
p [x)
Monte Carlo approximation
dim-wise computable
E.3. Product teacher model as control variate
For two models with the same marginals, we have the following proposition:
Proposition 7. Let q q be probability distributions on X = SD with the same marginals o² = = Then, for product
distribution p(x) = IIa (" (Id) over X, we have H (q. = HQ.
Proof. It suffices to prove that H(g, p) can be computed only by using the marginals ga Indeed, we have
D
D
D ΣΣ
Fagnn (lggp()))) = Fagnnng Σ Log Lo (a
Σ d=1
(a log (2
d=1 2d
and it yields the desired conclusion.
sid z dfjt and the fact that p" is product model, we have From this proposition, under Poje
Exg:ng: HI (goje( (z)). Poje( (=(==)) ss Ee H (Pujc (-1æx)) (+2x))
Since ((I1.P2) = DKL (pi 21 + H (P2. P2) this right-hand side can be rewritten as
Enz-ag H H (-lar.) Fore ] Bunyog (p (l1x.) 1 (-(=)))) const.,
where the constant term is independent of 0. Since the KL divergence between two product distributions decomposes into the sum of the KI. divergence between each marginal, we obtain approximation (14).
28

---

Distillation of Discrete Diffusion through Dimensional Correlations
F. Experimental details
F.1 Discretized Gaussian diffusions
F.1.1. SAMPLING SCHEMES
In the experiments, we used the following two sampling schemes when evaluating the already trained product teacher model.
r-leaping. In Campbell et al. (2022), the authors first approximate the infinitesimal transition rate by using each marginal Poj: Indeed, the transition rate can be represented only with d and does not require a joint conditional distribution (Campbell et al., 2022, Proposition 3). After estimating the transition rate, they apply a dimensionally parallel sampling method called r-leaping (Gillespie, 2001) coming from computational chemistry. Simply put, 7-leaping is sort of generalization of the Euler method for solving the backward SDE, exploiting the ordinal structure of S. We omit the corrector steps; the r-leaping in Table 5 corresponds to LDR-0 in Campbell et al. (2022).
Analytical sampling. Although the r-leaping (or Fuler method) is efficient with a large number of sampling steps, we find that it deteriorates when we reduce the number of steps seemingly due to discretization error. Analytical sampling (Sun et al., 2023) (a.k.a. Tweedie r-leaping; Lou et al., 2024), which is simply a parallel exact sampling of each dimension given as
(=)))) E.. of Doje (=gl)=))
Σ a
Is a. (2]1a u 2f Poje (0)æz))
(65)
does not suffer so much from the discretization. This is also mentioned in Gu al. (2022) fast inference strategy, though they do not discuss dimensional correlations. See also (71) for the derivation of dimensionally independent denoiser based on the product model Puja.
Note that these schemes are both dimensionally independent in the sense of (1) while not explicitly modeling Ps Indeed, the dimensional independence is ubiquitous even when modeling Pate implicitly. First, the reparametrization Peje(ar, = Eng P æe (sjo (x., æo, æ²,) (Austin et al., 2021; Gu et al., 2022), also used in analytical sampling, is dimensionally independent, provided that Poje(-|2z,) is given by product model and the forward diffusion is dimensionally independent. Second, we can apparently avoid the heuristic in the above modeling through the estimation of the transition rate in the continuous-time discrete diffusion (Camphell et al., 2022, Proposition 3), but the existing sampling schemes of æ; given æ; in continuous-time settings including r-leaping (Campbell et al., 2022) and the Euler-based method (Sun et al., 2023; Lou et al., 2024) are still dimensionally independent.
N -sstep sampling in the actual experiment is given as follows. We first set the timesteps 0 = to < f1 < < <N = 1, 2-1 with 4, = 0.01 + 0.99 x N-T for 2 1. Given a terminal noise Bty- we sample Ft; with our Ptl(+ iteratively for i N 1, N 2,...,1. 1. Finally, we sample To argmax Pojc, (-1ar)) when using the teacher product model and æo E argmax va,, (-x,;; i X) with random À when using the student mixture model.
F. 1.2. ADDITIONAL. EXPERIMENTAL RESULTS
To complement the main experimental results presented in Section 5.1, we provide additional details and analysis here.
FID/IS results of fewer sampling steps. For additional comparison, we computed FID/IS of the teacher and student modles using 10K samples (fewer than the 50K samples used in the main body, so the numbers may be slightly worse) for 2-20 steps in Table 4. In terms of FID, our method achieves approximately 1.4 times acceleration in the 10-20 steps range. However, does not perform well in very few steps (c.g.. 2-4 steps).
More detailed results of Table 1. We evaluated two different sampling strategies with the teacher model ph; (1) r-leaping (Campbell et al., 2022) and (2) analytical sampling (Sun et al., 2023; Lou et al., 2024). The complete evaluation results are shown in Table 5, which extends the results presented in Section 5.1. A notable observation is that analytical sampling significantly outperforms r-leaping in terms of sampling efficiency; 40-step analytical sampling achieves better FID scores than 1000-step -eappig..
Regarding distilled models, as we have highlighted in Section 5.1, p" works well in 10 steps, while it deteriorates as we grow the number of sampling steps. The hybrid model interestingly beats other models in 20-step FID and shows almost the
29

---

Distillation of Discrete Diffusion through Dimensional Correlations
Table Comparison of models CIFAR-10 dataset in various sampling steps. Same setting as Table except that 10,000 generated samples were used for computing FID/IS.
#steps
2
4
6
8
10
12
18
20
teacher
392.24
173.29
78.24
49.44
34.70
26.33
21.47
18.05
15.80
14.42
FID
student
411.70
147.67
59.62
33.85
22.57
17.46
14.37
12.86
12.28
1 11.81
teacher 1.17
2.99
5.90
7.01
7.45
7.82
7.96
8.16
8.35
8.46
IS
student
1.25
3.48
6.71
7.68
8.17
8.33
8.37
8.50
8.38
8.39
Table Comparison of models on CIFAR-I0 dataset. Fréchet inception distance (FID 4) against training dataset and inception score (IS f) calculated using 50,000 generated samples. ": reported values from Camphell et al. (2022).
10 steps
20 steps
40 steps
1000 steps
Method
FID
IS
FID
IS
FID
IS
FID
IS
Dd --leaping
315.75
1.66+0.01
8.10
8.74*
pd analytical
32.61
7.59
12.36
8.55+0.13
8.01
8.77
pd (student) 20.64
8.29.10.1
9.77
8.52.10.8
9.66
8.2810.10
pd &p (hybrid) 25.54
8.00
9.47
8.56++.14
8.02
8.43 +0.11
same 40-step FID with the teacher, while using the student solely gets worse in 40 steps. We hypothesize (elaborating on the description in Section 5.1) that this is because the true denoiser Zatz (s < t) becomes more "dimensionally independent" as t or small. The former condition (small s) explains the worse performance gain of the mixture model as the number of steps grows, and the latter partially explains the effectiveness of using the combined model. However, should further consider different forward diffusion and/or noise schedule to investigate it.
F.1.3. IMPLEMENTATION AND TRAINING
Diffusion modeling. As explained Section 5, the state-space has D = 3 x 32 x 32 dimensions, and each dimension has 256 possibilities of pixel values which corresponds to S = {o...., .55). The forward diffusion process defined through a discretized Gaussian transition rate with T (Campbell et al., 2022, Section E).
Network architecture. All the models are based on the implementation explained in Campbell et al. (2022, Section H.2), where Poje parameterized with a U-net (Ho et al., 2020) that has feature resolutions from 32 x 32 to x 4. Since the output of the original U-net architecture (Ho al., 2020) is D-dimensional sequence (in 8D) rather than D marginal distributions, Campbell et al. (2022) adjusted the network so that it first outputs Gaussian distribution over the real line for each marginal and then normalized it to obtain a distribution over S. The time t in their implementation is passed to feature map used in Ho et al. (2020), and this embedding is fed to the apsampling/downsapphhi Sil_U-activated linear layers (Elfwing et al., 2018). See Camphell et al. (2022, Section H.2) and their GitHub repository for more details on the original implementation. All the models output the estimation of qoje. and we conduct denoising from time to time by using the dimension-win wise analytical sampling (65), except for the r-leaping benchmark in Table 5.
layers of the U-net after passing through
The only change we made on the architecture is the insertion of X. We sample À from the uniform distribution over [o, 1]. so we can basically use the same embedding architecture the time t. For the downsampling layers, the embedding of X is concatenated with the time embedding, and then fed to the linear layers. After the linear layers, similarly to the time embedding, it is added to the latent vector of the image. For the upsampling layers, we concatenate the embeddings of X, f, and the wixe average of the x 4 resolution latent tensor, and the remaining process is the same as for the downsampling layers.
Training. Since our model is an expansion of the original model for ph, we trained (finetuned) our student model p" from the checkpoint of pd. The bias terms and the final layers concerning the embeddings of À are zero-in zero-intializae intiiliztai,, and the rest are randomly intialized following the default setting of the original model.
30

---

Distillation of Discrete Diffusion through Dimensional Correlations
For the Di4C finetuning, we followed the original setting in terms of the use of the Adam optimizer and the learning rate 2 x 10-1 as well other hyperparameters. The two primary differences in training are loss functions and the training steps/minibatch size (due to the Monte Carlo for A). For the former point, we basically used
Caistal (0; vh, q8, 6) + Ceonsis (0; v., 2t, O, 11, + aeCoore 1) + mmrrginal (0; tb, Qt, ():
(66)
with techniques described in Section A. The following are additional details:
Sampling from ys and qt is based on the same sample of ag ~ go-
8 = 0.01 with probability 1/2; otherwise, 5 is taken uniformly from [0.01, 0.02].
At is sampled from a log-uniform distribution over [0.001, 0.01]; is then sampled uniformly from [0.01 + At, 1].
We can use several Og as in the ablation study in the following section. In the main model p" given in Table 5, we used the following sigmoid- function as oz!
(()) 1 + exp 10 200)
(67)
Regarding the training steps/minibatch details, the original teacher model checkpoint had been trained for 2M steps, where each step used 128 images from the CIFAR-10 dataset as minibatch. In our finetuning, we stopped all the trainings in 320K steps (without warm-ups). Each step used a minibatch of 128/ L images from the CIFAR-10 dataset, where I is a batch size for À in the Monte Carlo estimates; we set M = N = L in (10). L = 16 is adopted in our model in Table 5, while the ablation study in the following section compares various choices of L..
Evaluation.
We measured FID and IS with the PyTorch-basee based implementation' following Campbell et al. (2022).
F.1.4. ABLATION STUDY
Table Ablation study on ox, and use control variates.
10 steps
20 steps
40 steps
Method
FID
IS
FID
IS
FII
IS
ddd + analytical
32.61
1.59
12.36
8.55
8.01
8.77.19.9
0 =
26.23
8.02
11.55
8.59.1.07
9.01
8.65
02 = 0, w/o CV
44.09
10.
26.16
7. 7.544..10
22.20
7.72+0008
04 =1 1
24.14
7.04
12.30
8.06
10.32
8. 8.1410.11
0: = 1, w/o CV
26.92
8..0000
13.77
8.57+0.14
10.59
8.66 8.662005
= t 0: = gt) (see (67))
24.21
8, 1.
10.85
8.55
9.27
8.51
22.77
8.19 1..08
10.07
8.54
9.01
8.42...11
As an ablation study, we compared several loss functions, mainly changing a, which controls the degree of dimensional correlations we aim to learn from datapoints. We also investigated whether the use of control variates is effective. The results are shown in Table 6, where "w/o CV" means that the control variates were not used in training. The efficiency of control variates was consistent, while = 0 and a; had pros and cons. Non-constant functions of oa worked better, partially matching the hypothesis discussed at the end of Section 5.
Additionally, we compared different batch-sizes of X in Table 7 (also see the end of the previous section). The non-constant or = (() was used in all the settings. 7. in the table represents the batch size of X in Monte Carlo sampling. There is a certain tradeoff between FID and IS in 10- or 20-step sampling; we can expect better FID with larger L (smaller data batch), while smaller L tends result in better IS.
'https://e hhttps://github. com www.m//www63377/pttttoom ingee--na-eeea- ong.://rrrrrmmmeeggegeentttooooo generation which got renamed from the original repository "pytorch- wwwoorrhh-aaammetteee cannmmetris" "pytorch-imeen wet.-ggiooneeeeeeeeeeeeeee
31

---

Distillation of Discrete Diffusion through Dimensional Correlations
Table Ablation study on Monte Carlo sample size X.
10 steps
20 steps
40 steps
Method
FID
IS
FID
IS
FID
IS
pd + analytical
32.61
1.59
12.36
8.55
8.01
8.77
L 2
27.29
8. 8.00
11.42
8.67:10112
8.94
8.64
I 4
24.94
8.05
10.66
8.60
8.90
8.59
L 8
22.77
8. 8.098
10.07
8.54
9.01
8.42++0.11
L = 16
20.64
8.29
9.77
8.52
9.66
8.28
L = 32
20.25
8.28
9.93
8.44
9.91
8.26++013
= 64
19.26
8.1310.10
10.13
8.26.10.1
10.59
8.02.1015
F.1.5. GENERATED SAMPLES
Figure shows image examples corresponding to Table 1, which were all generated with the analytical sampling.
F22 Masked generative image modeling
F.2.1. MASKED DIFFUSION MODELING
As described in Section 5.2, we use the pretrained VQGAN codebook S with |S*| = 1024 and add one [MASK] token to define S. Also, 256 in this experiment. The only ingredient we need masked diffusion is the masking probability my: a monotonically increasing function with mo 0 and mı 1. Following Chang et al. (2022) and Besnier Chen (2023), model the forward process of æ; given æn E (S*)
[MASK]
with probability m,a with probability m4,
A 27
(68)
independently for each d E (1,..., D}. Note that (68) does not necessarily determine a Markov process (and indeed an explicit Markov formulation is not needed for training). If one needs a Markov formulation, however, for t > > 0 and x E S* = S [[MASK]], we have
m, = o (IMASK](x) = 1 (IMASK] + ( (1MA m, + (1 mm) (1MASK]]
and thus we have
d (IMAKKIIr) mss
(69)
In the actual experiment, we used the arccos scheduler m4 = aarccos(1 t)/
F.2.2. CONFIDENCE BASED SAMPLING
Given o, time t, let M, { D) 24 [MASK]]- Suppose we have a product model Poje(-122))
I1 p- such that 1 (1MASK(]))).)) = O for all d E (1....,) D) and (x+] [24) ) = 1 for all d & M,. Let us explain how we sample x, with < & in one step of seffidence-basedd sampling (Chang et al., 2022; Besnier & Chen, 2023). Following the original implementation of MaskGIT-PyTorde (Besnier & Chen, 2023) we conduct the sampling as follows:
1. Sample 2.
(2 Poje( 2x). Note that we have ag * [MASK] for each d and ag = xς. for d Mc-
2. Calculate the confidence for each chosen ž for d & Mt as
conf(i) = log poje (zgla.) (1) Egg:
where the second term is given by constant multiplication of Gumbel noise to add stochasticity in confidence-based sampling (see, e.g., Comunità al. (2024, Section 3.4.4) for concise explanation). To be concrete, ed for each
d & Mr is an independent standard Gumbel noise, and cgb is scale factor given by (g)(() experiments, where N the number of steps in the whole sampling process.
2 t- 1/N
11-1/N
in our
32

---

Distillation of Discrete Diffusion through Dimensional Correlations
(a) teacher, 10 steps
(b) teacher, 20 steps
(c) teacher, 40 steps
(d) student, 10 steps
(c) student, 20 steps
student, 40 steps
(g) hybrid, 10 steps
(h) hybrid. 20 steps
hybrid, 40 steps
Figure 5. Comparison of generated samples in CIFAR- 10 experiment.
3. Let n(t, s) be the number of tokens we unmask this single sampling step from to s. Let disc.o) & M, be the index with the ra(t, s)-th largest (d). We define x, (z:) follows:
nd
for d & M, with conf(d) 2 conf(d"
x2
otherwise.
Note that con coincides with probability zero thanks to the Gumbel noise.
This is for one step of confidence-basee denoising. When we use mixture model, we first sample and conduct the confidence-based sampling for the product model conditioned by X. In the actual sampling process, specify (the number of unmasked tokens) as we explain below?.
Let N be the number of sampling steps and t; i/N for = 0....,N N. We jump from f, to f--1 at each sampling step
"The definition of nut;, f;-1) descrihed below is different from the implementation of MaskGrT-PyTorcle since the original imple- mentation not correspond forward processes.
33

---

Distillation of Discrete Diffusion through Dimensional Correlations
for N, N I,....; so we set / 1, and 4--1- Let my be the masking probability time 6, so that the expected number of [MASK] tokens equals Dm, given the forward process design, where mg 0 and m. By rounding them into integers, we define ntti, 14-1) n(t;, f4-1 = 0 and deduct the added ones from n(t2, tu) so that at least one token gets unmasked, following the original implementation.
(ound(Dm,,) round (m) ) for each i. In the code, we further add when
F.2.3. DISCRETE CLASSIHER- FREE GUIDANCE
Following the original implementation, we use the discrete fress guidance (discrete CFG; Tang et al., 2022). Given unconditional denoiser (oje(- and a conditional denoiser Poje(-122,, c) with class label e, we sample from the distribution
Po c) Poje(-|ze,,)) " Poje (-
for a CFG guidance scale w; w = 0 corresponds to sampling from the conditional denoiser. In our implementation, given the number of steps N, we let to = Weig N-1
[1-1)N to linearly increase the guidance scale in the sampling process, where Wefg is a user-selected CFG coefficient, mentioned in Section 5.2. When we use a mixture model, we first sample a single À and sample from the CFG-guided distribution given by the X-conditioned unconditionallccondiiona denoisers Pojel (ay;; A) and Poje(-/z2,, c; X) as a heuristic.
F.2.4. IMPLEMENTATION AND TRAINING
Network architecture. For the teacher model pd, we just used the implementation of Besnier & Chen (2023, Table 1), which uses 24-layer transformer with 16 attention heads to compute the logits for each of D 256 visual tokens. As inputs, p" gets the D visual tokens o; (= and a class token (when unconditional, it is replaced by an "unconditional token"), ") and these D + 1 tokens are added positional encodings. Note that time conditioning is not fed to the model. Each learned embedding (of 1024 VQGAN codebook elements and 1000 ImageNet classes) is of 768 dimensions. Since the output of the transformer has D + vectors in I 768, we discard the vector corresponding and compute the logits by using the similarity with the embeddings of codebook elements.
To realize mixture model p" (- (-|z;;; X) upon this implementation, we first sample X ~ Uni IJ), pass it to the timestep embedding used in the CIFAR- 10 experiment (Appendix F.1.3), and transform its output into a 768- 768-dimensional vector using a two-layer MI.P with single GELU activation after the first layer. Thus, we obtain D + 2 vectors IR 708 consisting of visual embeddings, a class embedding, and an embedding of X. We simply feed them to a transformer having the same architecture as nd and discard the final output vectors concerning and X to calculate the logits. The logit calculation is done in the same way as the teacher model.
Training. We basically used the same formulation as (66) with slight modification to reduce the computational burden as follows:
1 (0sa0)
Al Laisti (0; vi qt, 1) + 1122 (0; q:, 0, - (t, + Cooor + Emarginal (0; v. qr, t).
(70)
with At- = 0.05 and the following details:
We sampled t ~ (n0, 1]) and æ, according to (68) using zo from data.
We used a₂ 0 for di4c and a, 0.1 g(t) for di4c-d in the experiment (Figure 3), where g(f) is given in (67).
Control variates were used in both variants of Dj4C.
For optimization, we followed the original implementation, i.e., the AdamW optimizer with learning rate 10-5, (81, 32) (0.9, 0.96) and weight decay 10-5, The teacher model was trained for 300 epochs with a minibatch size of 512 using eight A100 GPUs (adding up to 768 GPU hours; Besnier & Chen, 2023). Our finetuning used two A6000 GPUs with minibatch size 4 (2 for each of two GPUs) and a X-batch size of 32. It was trained for 30K iterations (so only 120K out of 1.28M ImageNet training images were used), which amounts to approximately 50 GPU hours.
Evaluation. In each experiment, we generated 50,000 samples (50 images for each ImageNet class) and then computed FID (against test data) and IS with the original implementation of Besnier Chen (2023).
34

---

Distillation of Discrete Diffusion through Dimensional Correlations
F.2.5. GENERATED SAMPLES
Samples generated by each model in Section 5.2 are shown in Figure 6. The best CFG coefficient in Figure 3 in terms of FID chosen for each model, shown in Table 8. The table also shows the FID/IS performance, together with the Precision/Recall values computed by the code of Besnier & Chen (2023). Regarding the choice of ImageNet labels (Ostrich, Burger, Volcano), we followed Besnier & Chen (2023, Figure 3).
Table 8. Chosen CFG coefficients and corresponding FID/IS values.
Model
steps
Wefg
FID (L)
IS (1)
Precision (f)
Recall (1)
teacher
8
3.0
6.57
202.0
0.7939
0.5499
teacher
4
11.0
7.97
216.0
0.7737
0.5057
di4c
4
6.0
6.79
209.2
0.7910
0.5363
di4c-d
4
7.0
6.57
213.6
0.7866
0.5391
F33 Masked diffusion language models
F.3.1, DIFFUSION MODELING AND SAMPLING
Diffusion modeling.
We basically worked under the same setting as in Section F.2.4 except for the following:
1024.
The non-mask codebook S* is given by the set of GPT-2 tokenizers, which satisfies |S*| = 50257.
The masking probability is given by m,
While the above apparently linear noise scheduling is called log-linear (Sahoo et al., 2024, Section E.1), the naming comes from their parametrization: m, - - (t)) with (t)) - -log - -t)
Analytical sampling in masked diffusion. The sampling algorithm used for the MDLMs is essentially the same the analytical sampling in Section F.1.1, tailored for masked diffusions. What we have is a dimensionally independent denoiser Pojo (ar ax.) = Ia 1oj (xg)) a.). By using (42), we can deduce the resulting product denoiser based on Poja follows:
Is o ( (1) (x.))
(o)1(0)/(311)
Σ Uojo,e (x, (2o we
Σ
(ojo (2)) Poje Poje (ole(xoola)) (x))
Lap (x (or off (a (2
Poje (20)
ao
(1 h (1f x
(af
D x2)) = II (xf1)e))
Poje Pojo (2
(71)
Note that, once a token (dimension) is unmasked, we do not need to further change that token in the backward process: this property is incorporated in Puje as p (x)) = 1 for xf * [MASK] (Sahoo et al., 2024, Section 3.2.3). Thus, we just need consider the case x [MASK]. By using the fact that the masking probability is given by m, t and m.-m, for x # [MASK] in (69), if 2 [MASK] and x + [MASK], we simply have Lf ([MASK][{) 1-m,
L (
(1 m, Mi m
pd (x(
Paj Poje (x/))))
m.
poj (r))
m. mg Pule P (xr(2)))
p (IMASK]]x)
m
m,
which corresponds Sahoo al. (2024, Eq. 7).
35

---

Distillation of Discrete Diffusion through Dimensional Correlations
teacher, steps
(b) teacher, 4 steps
di4c, steps
(d) di4c-d, 4 steps
(c) teacher, steps
teacher, steps
(g) di4c, 4 steps
(h) di4c-d, 4 steps
teacher, 8 steps
teacher, steps
(k) di4e, 4 steps
(1) di4c-d, steps
Figure 6. Comparison of generated samples. (a)-(d): Conditioned with ImageNet label Ostrich (009). (e)-(h): Conditioned with Burger
(933). (i) (I): Conditioned with Volcano (980).
36

---

Distillation of Discrete Diffusion through Dimensional Correlations
F.3.2. IMPLEMENTATION AND TRAINING
Network architecture. For the teacher model, used the two "small" checkpoints (round 6 and 7) of Deschenaux & Gulcehre (2025)', which uses transformer architecture to compute the logits of poje (=(2;) for each token. The transformer architecture is originally from (Sahoo et al., 2024), and has 169M parameters with 12 layers, the embedding dimension of 768, and 12 attention heads. The model receives D = 1024 tokens and does not depend on the timestep. For adaptation to mixture modeling, we just applied the same modification as given in Section F.2.4.
Training. While the model accepts a continuous time training, we followed Deschenaux & Gulcehre (2025) to digitize the timesteps to T (Ai- . - 0, 2,,,,...... 1024]. with Al = 1/1024. We used the following loss function:
1 Trsgy Cian (0; v, yc 1) 1128) Coonsis (0; 1. ot 0, A1, aehoorr (0; (0;1) marrinal (0; v Qc (),
(72)
where the details are as follows:
We set 8 0.02.
We sample r ~ Uni 1J) and let
where (r))) { (ör
with probability 0.3, with probability 0.7.
= 2- (2" (r)),,
8 + (1 5)r
Therefore, & E T> {0} almost surely.
We sampled æ, according to (68) using ze from data (OpenWebText).
We used 02 = the experimental results shown Section 5.3. In Section F.3.3, we also report the result of setting 0x 0.1 g() with g() from (67).
Control variates were used in all the experiments.
For training, we mostly followed the original setting of Deschenaux & Gulcehre (2025): We used the Adam optimizer (but with learning rate of 3 x 10-5 with EMA (weight decay 0.9999) and did a constant warm-up (increasing the learning rate linearly for the first 500 iterations and setting constant after that). For each experiment, the Di4C training (one round) was run for 100K iterations over 2x A6000 GPUs, where the minibatch size was for each device) and X-batch size was 16.
Self-BLEU computation. As described in Section 5.3, in the conditional generation experiment, we generated M 5 x x((1) conditioned on the first 50 tokens (prompt) of each WebText datapoint ( Each continuation was of 100 tokens including the prompt, and we used M 256 prompts from the WebText dataset in total. To quantify the diversity of continuations, followed Deschenaux & Gulcehre (2025) and computed the Self-BLEU score as
continuations C+) (x(") ......5
where BLEU(X;C) the BLEU score of sentence against the set of reference sentences C. To actually compute this, we tokenized the inputs with the GPT-2 tokenizer, and then utilized the implementation of Zhu al. (2018) with ngram 4, which internally calls the sentence.ble function of the NLTK library (Bird et al., 2009)3 with equal weighting and the method1 smoothing function.
Evaluation. Except for the Self-BLEU computation, all the evaluations were done using the code of SDTT (Deschenaux & Gulcehre, 2
'It loaded by load small student (loss='kid', round=n) with n = 6, from the library sdtt in https:/ github. http//dddcceea/sdtt... com/
Phttps://gtt http:///www.aaooooooooo///////// chttps://gt http://tttt..cooc/ccc///////ttt/
37

---

Distillation of Discrete Diffusion through Dimensional Correlations
180
sdtt-6
sdit-6
4 steps
sdtt-6 di4c
Cond. Generative Perplexity
sdtt-6 di4c
8 steps
Generative Perplexity
160
sdtt-6
250
sdtt-6 + di4c-d
steps
140
sdtt-/
sdit-/
32 steps
sdtt-7 + di4c
200
sott-7 + di4c
64 steps
120
sdtt-7 di4c-d
sdtt-7 + di4c-d
128 steps
100
150
80
100
50
20
23
2
2
2
2
55.5
56.0
56.5
57.0
57.5
58.0
58.5
59.0
Num. Sampling Steps
Cond. Self-BLEU
(a) Gen. PPL. vs Num. Steps in unconditional generation
(b) Gen. PPL vs Self-BLBU in conditional generation
Figure Comparison Di4C distillations SDTT checkpoints with without data loss.
0.95
0.95
0.90
0.91
MAUVE
MAUVE
sdll-6
sdit-6
sdtt-6 di4c
sdtt-6 + di4c
sdtt-6 + di4c²
sdit-6 di4c-d
sdll-7
sdtt-7
0.75
sdtt + di4c
0.75
sdit-7 di4c
sdit-7 + di4c?
sdll-7 + di4c-d
2
2
24
25
2
27
23
2
2
2
2
Num. Sampling Steps
Num. Sampling Steps
(a) MAUVE results with additional Di4C iteration
(b) MAUVE results with and without data loss
Figure 8. Comparison of Di4C distillations of SDTT checkpoints with and without data loss.
F.3.3. ADDITIONAL EXPERIMENTAL RESULTS
Data loss ablation. In Figure 7, we show the result for sdtt-6/7 + di4c-d, which used loss (72) with oz = 0.1g(() as mentioned in Section F.3.2. The results are almost the same as those without data loss (especially in Figure 7, where sdit-7 + di4c hidden behind the curve of sdtt-7 + di4c-d). Since the ones without data loss showed slightly better generative perplexities, we presented them as the main model Section 5.3.
MAUVE results. We also tested our models with the MAUVE score (Pillutla et al., 2021). The setting is the same as the unconditional generation Section 5.3, and the MAUVE computation is done by using the code of SDTT (Deschenaux Gulcehre, 2025). As shown in Figure 8, no significant performance decay from the teacher model was observed.
F.3.4. GENERATED SAMPLES
Let us qualitatively compare our best model (sdtt-7 + di4c?) with the SDTT checkpoint which our model is based on (sdtt-7), conditioned on the first 50 tokens from this paper's 's abstract (old version; highlighted in blue):
(sdtt-7,4 4 steps) Diffusion models have demonstrated exceptional performances in various fields of generative modeling. While they often outperform competitors including VAEs and GANs in sample quality and diversity, they suffer from slow sampling speed due their iterative nature. Recently, districting the image spl Image') Andistriction the size spl
38

---

Distillation of Discrete Diffusion through Dimensional Correlations
Image size The stable image spl consists of thin-arr 2D-frame data that supports band- construction data and generates the physical representation of image presentation. Although much is known
(sdtt-7 + di4c², 4 steps) Diffusion models have demonstrated exceptional performances in various fields of generative modeling. While they often outperform competitors including VAEs and GANs in sample quality and diversity, they suffer from slow sampling speed due to their iterative nature. Recently, distilling the high spl Image Im- age',n Andistiiiiion the whole spl Image Image generating an image Image is thin-narr 2D-layer architecture that exhibits banding-resolving effects in the lowizing of low resolution. Little is known
(sdtt-7, 16 steps) Diffusion models have demonstrated exceptional performances in various fields of generative modeling. While they often outperform competitors including VAEs and GANs in sample quality and diversity, they suffer from slow sampling speed due to their iterative nature. Recently, distillilusion models, which use a dataset from the sample and using inferred model data, have become standard. self-arrained three-block sampling that utilizes subset of the data and combining Bayesianesian and inferred model data that combines Bayesian
(sdtt-7 + di4c², 16 steps) Diffusion models have demonstrated exceptional performances in various fields of generative modeling. While they often outperform competitors including VAEs and GANs in sample quality and diversity, they suffer from slow sampling speed due to their iterative nature. Recently, distorting Diffusion models, nested within conventional model architecture, and using different model architectures, has become a flexible self-arrative three-model architecture that supports intensive problem-solving and mature Bayesianesian model and model development. This approach has
(sdtt-7, 64 steps) Diffusion models have demonstrated exceptional performances in various fields of generative modeling. While they often outperform competitors including VAFs and GANs in sample quality and diversity, they suffer from slow sampling speed due to their iterative nature. Recently, distancing from the Diffusion model has allowed developers to construct new models using software-drawing that improves the ability to identify discriminant parameters, functional model depth, and is the process of rapidly
supervised learning supervised (GCI supervised)
(sdtt-7 + di4c², 64 steps) Diffusion models have demonstrated exceptional performances in various fields of generative modeling. While they often outperform competitors including VAFs and GANs in sample quality and diversity, they suffer from slow sampling speed due to their iterative nature. Recently, distancing from the Diffusion model has allowed scientists to analyze the model using discriml, a software that utilizes to discriml. This software can generate images that offset the time to be discrimlative, and reduce the time be discrimlative.
(sdtt-7, 256 steps) Diffusion models have demonstrated exceptional performances in various fields of generative modeling. While they often outperform competitors including VAEs and GANs in sample quality and diversity, they suffer from slow sampling speed due to their iterative nature. Recently, distancing from the traditional resource management process has led change in the type of user experience, producing significant advances in the types of -oftware-developingg operations, while facilitating the adoption of parallel programming and functional programming approaches. However, it is increasingly
(sdtt-7 + di4c2, 256 steps) Diffusion models have demonstrated exceptional performances in various fields of generative modeling. While they often outperform competitors including VAEs and GANs in sample quality and diversity, they suffer from slow sampling speed due to their iterative nature. Recently, distorting cloud images in the process of processing to high standards in both sample quality and storage, producing important advances in the development of software-developing operations, while facilitating the development of parallel programming and functional efficiency. GPUs are in the process of processing
39


Reviews:
[{"review_id": "ru7liRbuxz", "reviewer": "Reviewer_1bzY", "summary": "The paper proposed a improved model for distilling discrete diffusion models(DDMs). The key idea is that traditional DDMs break apart the latent distributions into product of marginal distributions, while the proposed model represent the latent distributions into products of bi-dimension distribution pairs, which is able to approximate the ground-truth distribution in fewer de-noising steps with accuracy.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. Let us reply to your comments.\n\n### [1-1] Added metrics beyond FID/IS\n\n> the authors mainly compare their model with the teacher model using FID/IS scores, which is inherited limited\n\nWe have additionally computed precision and recall metrics for the ImageNet experiment (see the table below), following established practices in the current literature (e.g., [Tian24]). Due to character limits, we present the results for each model at the classifier-free guidance (CFG) coefficient that achieves the best FID (see Table 6). We can see di4c models achieve better scores than the teacher model with the same number of steps.\n\n|Model (#steps)|Precision|Recall|\n|-|-|-|\n|teacher (4)|0.7737|0.5057|\n|di4c (4)|0.7910|0.5363|\n|di4c-d (4)|0.7866|0.5391|\n|teacher (8)|0.7939|0.5499|\n\n### [1-2] DDMs are not \"fast and approximate\" in general\n\n> It is confusing why the authors insist on distilling DDM, which is already a fast approximate model, rather than slow high-quality models? Is it possible to use the model for distilling original or modified diffusion models of higher quality, or training from a scratch?\n\nThe characterization of DDMs as approximations of 'original diffusion models' (as implied in your latter comment) is not typically correct. Indeed, DDMs are not \"approximate\" or \"fast\" in general.\n\n- **Not Approximate:** The teacher DDMs were generally trained from scratch (except for SDTT models we used in language modeling) without relying on any other diffusion models. While the parallel decoding (analytical sampling) in teacher DDM is indeed an approximate inference (with guarantees given in Theorem 1), a similar approximation (unimodal Gaussian approximation) is commonly employed in continuous diffusion models, as noted in [Li24]. Therefore, DDMs are not particularly \"approximate\" compared to continuous diffusion models.\n\n- **Not Fast:** DDMs typically require tens to hundreds of sampling steps, depending on the complexity of model/data. For the experiment on Discretized Gaussian Diffusion, the teacher model requires 30-40 steps to reach FID<10, which we believe is no longer \"very few steps\". For comparison, in the continuous case, while EDM on CIFAR-10 requires 35 steps [Kar22], notable distillation methods [Son23] start from EDM and distill it into a few-step model. MaskGIT(-pytorch) is one of few examples that achieves a small number of steps (8 steps to reach FID~7.0), enabled by some heuristics including confidence-based sampling and CFG. We tested our method in this scenario because we believe it is worth investigating if our distillation methods can also work in combination with such heuristics.\n\nThus, we believe our method can be applied to \"original or modified diffusion models\". Learning dimensional correlations in DDM from scratch by incorporating the consistency training of [Son23] into our method is an interesting future work.\n\n### [1-3] We mostly compute multiple metrics for each dataset \n> each time only single metric is used for one dataset, and no statistical variance analysis was given.\n\nIt is not accurate to state that \"each time only single metric is used for one dataset\". In fact, we computed both FID and IS for CIFAR-10 and ImageNet experiments. For the OpenWebText experiment, we computed Gen. PPL for unconditional generation, and three different metrics (Gen. PPL, Self-BLEU, and MAUVE) for conditionally generated texts with prompts from the WebText dataset.\n\nRegarding statistical variance analysis, we provided variance values for the latency analysis in Table 2. While it is ideal to retrain and resample everything 5-10 times, the large number of training checkpoints and experimental data points makes it common practice in the literature to report numbers from a single run (as seen in most references cited in this reply).\n\n### [1-4] We compare our method with SDTT distillation\n> most important, the method is not compared against other distilling methods.\n\nWhile there are few distillation methods for discrete diffusion available (except for some concurrent works), we did compare our method with one round of distillation by SDTT [Des25] in the language modeling experiment (Figure 6). Specifically,\"sdtt-7\" is obtained after one round of SDTT distillation upon \"sdtt-6\", while \"sdtt-6 + di4c\" (or di4c^2) is obtained by Di4C training using the same teacher (sdtt-6). We explicitly compare them in Section 5.3.\n\n### References\n\n[Tian24] Tian et al. Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction. NeurIPS 2024.\n\n[Li24] Li et al. Soft mixture denoising: Beyond the expressive bottleneck of diffusion models. ICLR 2024.\n\n[Kar22] Karras et al. Elucidating the design space of diffusion-based generative models. NeurIPS 2022.\n\n[Son23] Song et al. Consistency models. ICML 2023.\n\n[Des25] Deschenaux and Gulcehre. Beyond Autoregression: Fast LLMs via Self-Distillation Through Time. ICLR 2025.", "claims_and_evidence": "The authors provided FID measurement as well as image instances to prove that the distilled model compliant with the teacher model. They claim that distilled model  with 4-steps reaches similar performance of the teacher model in 8-steps, while the overhead for each step is minimal (according to supp. F.1). Although this claim is rational, it should be noted that this can be frequently observed for certain distillation models because they take benefits from the hint of the teacher models.", "methods_and_evaluation_criteria": "The evaluation criteria make sense but is a very rough one. The FID/IS metric is a very limited metric for image generation quality. It mainly focus on the average distribution of inception features, without considering the diversity and detailed fidelity of the images.", "theoretical_claims": "The authors provided theoretical proof that the total variance decrease with increasing training steps.", "experimental_designs_or_analyses": "Firstly, as discussed above, the authors mainly compare their model with the teacher model using FID/IS scores, which is inherited limited . Also, they made this comparison on CIFAR-10 and ImageNet datasets, where the FID/IS scores converge for each model in very few steps, which cannot justify the benefit of the model because DDM is not slow in this case. The authors did introduce other metrics (e.g., PPL and MAUVE score) in conditional generation but each time only single metric is used for one dataset, and no statistical variance analysis was given. Third and most important, the method is not compared against other distilling methods.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strength: The authors actually proposed an improvement for DDM.\nWeakness: If this improved structure can only be applied to distill pre-trained DDM, the benefit is very limited.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "It is confusing why the authors insist on distilling DDM, which is already a fast approximate model, rather than slow high-quality models? Is it possible to use the model for distilling original or modified diffusion models of higher quality, or training from a scratch?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A5Gs6fNLqC", "reviewer": "Reviewer_d4U9", "summary": "This paper studies the distillation problem for discrete diffusion models. The authors identify a key challenge in capturing dimensional dependencies and provide theoretical analyses to support their findings. To address this, they propose a mixture student model with tailored loss functions to facilitate distillation. The proposed method is demonstrated to be effective in both vision and language domains.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and positive comments. Let us respond to your questions.\n\n### [d-1] On the reference distributions\n> the method relies on reference distributions for $r_\\delta$ and $r_t$, which must either be sampled from real training data or approximated using multiple teacher steps.\n\nIt is a valid point that our method has limitations on the source of reference samples. However, in the relatively large-scale experiments (masked image/language modeling), we used only a small portion of the original dataset. For example, we used 200K out of 9M samples in OpenWebText (similarly for ImageNet). Additionally, in our current implementation, each sample $x_0$ generates only a single $x_t$ (with a single $t$) throughout the training run. This suggests that more efficient batch designs could reduce the essential number of training samples. That said, we acknowledge that accessing or constructing reference samples is not always straightforward. Future research should explore the use of partial samples or samples from alternative distributions to address this limitation.\n\n### [d-2] Regarding $\\tau$LDR models\n> Why is the result of the teacher model using 40 steps in Table 1 already close to 8?\n\nThis discrepancy arises from the different sampling methods used. As detailed in Table 3 in Section F.2.2, the $\\tau$-leaping sampler does not work well with 40 steps. Instead, we used the analytical sampler to evaluate the teacher model. We suppose this difference stems from the non-time-homogeneous nature of the forward diffusion. Specifically, $\\tau$-leaping approximates the transition rate using a constant matrix over a time interval of length $\\tau$, which fails to accurately capture the actual transition rates when $\\tau$ is large and the rates vary significantly within the interval. In contrast, the analytical sampler avoids this issue, although it still does not account for dimensional correlations.\n\n> Does it indicate that the proposed distillation method is not compatible with the predictor-corrector sampler.\n\nWe can use predictor-corrector (PC). However, PC requires additional network evaluations (NFE; each as expensive as one step of denoising), and does not perform well in the small NFE regime [Cam22, Fig 4]. Below are the FIDs (based on 10K images; IS omitted due to the character limit) for different PC settings under total NFE of 20. 'n+2*m' means we used m corrector steps before each of the final 2 out of n denoising steps (imitating [Cam22]).\n|NFE|6+2*7|10+2*5|14+2*3|20+2*0|\n|-|-|-|-|-|\n|teacher|57.57|32.74|21.32|14.42|\n|student|44.57|22.63|14.25|11.81|\n\n### [d-3] Inference of other number of steps\n>  It would be more informative to include results of other steps in a graph similar to Figure 3. For example, I would like to see how many times of acceleration the proposed distillation method can achieve to maintain a similar performance (in FID/IS) to the teacher model using 20 inference steps.\n\nHere, we have computed additional FID/IS for our CIFAR-10 experiment using 10K samples (fewer than the 50K samples used in the paper, so the numbers may be slightly worse) for 2-20 steps:\n\nFID:\n|model|2|4|6|8|10|12|14|16|18|20|\n|-|-|-|-|-|-|-|-|-|-|-|\n| teacher | 392.24 | 173.29 | 78.24  | 49.44  | 34.70  | 26.33  | 21.47  | 18.05  | 15.80  | 14.42  |\n| student | 411.70 | 147.67 | 59.62  | 33.85  | 22.57  | 17.46  | 14.37  | 12.86  | 12.28  | 11.81  |\n\nIS:\n|model|2|4|6|8|10|12|14|16|18|20|\n|-|-|-|-|-|-|-|-|-|-|-|\n| teacher | 1.17 | 2.99 | 5.90 | 7.01 | 7.45 | 7.82 | 7.96 | 8.16 | 8.35 | 8.46 |\n| student | 1.25 | 3.48 | 6.71 | 7.68 | 8.17 | 8.33 | 8.37 | 8.50 | 8.38 | 8.39 |\n\nIn terms of FID, our method achieves approximately 1.4 times acceleration in the 10–20 step range. However, it does not perform well in very few steps (e.g., 2–4 steps).\n\n> I feel there is still a huge gap between the proposed method and its counterpart for continuous diffusion.\n\nTo further reduce the number of steps, it is necessary to model high-dimensional discrete distributions more efficiently, as we lack a deterministic formulation like probability-flow ODEs (available in the continuous case). While mixture modeling provides some improvement, further optimization is required, including adjustments to the architecture, initial distribution, and dimensionality of $\\lambda$. As you noted, our work represents a first step in building the theoretical foundation for this goal.\n\n### [d-4] Sampling distribution of $\\lambda$\n> How should the sampling distribution of $\\lambda$ be chosen? Have the authors ever considered other common distributions than uniform?\n\nWe prioritized ablations of different components over exploring alternative distributions for $\\lambda$. As noted at the end of [d-3], this remains an area for future investigation.\n\n### References\n\n[Sah24] Sahoo et al. Simple and effective masked diffusion language models. NeurIPS 2024.\n\n[Cam22] Campbell et al. A continuous time framework for discrete denoising models. NeurIPS 2022.", "claims_and_evidence": "Yes, the claims are well supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, the proposed method and evaluation criteria are mostly appropriate for the problem. However, I find one requirement somewhat restrictive in practical applications. Specifically, the method relies on reference distributions for $r_\\delta$ and $r_t$, which must either be sampled from real training data or approximated using multiple teacher steps. The first approach requires access to real data, which is not always feasible—especially in cases where only a pretrained model is available and due to copyright restrictions. The second approach, while avoiding this issue, may introduce significant computational overhead (e.g., when approximating $r_\\delta$ with $\\delta \\ll 1$), making the method less practical for large-scale applications.", "theoretical_claims": "I reviewed the main theorems and their proofs and found no issues.", "experimental_designs_or_analyses": "I reviewed the experimental design and discussion and found no major issues. However, I did found parts of the experiments could be further improved:\n\n1. Table 1 only provides the evaluation metrics of 10,20,40 inference steps. It would be more informative to include results of other steps in a graph similar to Figure 3. For example, I would like to see how many times of acceleration the proposed distillation method can achieve to maintain a similar performance (in FID/IS) to the teacher model using 20 inference steps. Additionally, could the authors explain how the results were obtained? Based on Campbell et al. [1] Figure 4, CIFAR-10 FID of $\\tau\\text{LDR-0}$ (\"teacher model\" here) does not drop near 8 until over 256 NFEs. Why is the result of the teacher model using 40 steps in Table 1 already close to 8?\n2. Additionally, in [1], with additional corrector steps, the results of $\\tau\\text{LDR}$ can be significantly improved--3.74 in FID and 9.49 in IS for $\\tau\\text{LDR-10}$. However, it seems that the authors did not compare their results with it. Does it indicate that the proposed distillation method is not compatible with the predictor-corrector sampler. Otherwise, I would suggest including the results combining the student model with the predictor-corrector sampling strategy.\n3. Could the authors also include the sampling results with one inference step? Considering that distillation methods for continuous diffusion models can already achieve performance on par with their teacher models using just one step sampling, I feel there is still a huge gap between the proposed method and its counterpart for continuous diffusion. While the proposed distillation method does open a new door for accelerating discrete diffusion sampling, I don't find the proposed method especially impressive and convincing based on these experimental results.\n\n[1] Campbell, Andrew, et al. \"A continuous time framework for discrete denoising models.\" *Advances in Neural Information Processing Systems* 35 (2022): 28266-28279.", "relation_to_broader_scientific_literature": "The paper makes important contributions to discrete diffusion distillation through both theoretical analysis and practical algorithm design.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The paper is well-written with solid theoretical grounding and mathematical details. The proposed algorithm is flexible, practical, and applicable across different model architectures and tasks.", "other_comments_or_suggestions": "None.", "questions_for_authors": "How should the sampling distribution of $\\lambda$ be chosen? Have the authors ever considered other common distributions than uniform? How sensitive is model performance to this choice?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u7o4L5Rw6Y", "reviewer": "Reviewer_qHYq", "summary": "This paper focuses on an important research question of distilling discrete diffusion models, which poses unique challenges due to the necessity of modeling the joint distribution of multiple discrete states grows with a combinational complexity. This paper proposes Di4C, a principled model agnostic approach for distilling discrete diffusion models. Specifically, a mixture model with enough expressivity is employed as the student model and is learned with a consistency trajectory distillation fashion loss, along with several auxiliary losses and a designed control variate. Experimental results demonstrate the effectiveness of the proposed method on both image generation and text generation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive evaluation of the paper. Let us answer your questions. We will also correct typos you have pointed out. \n\n### [q-1] Why we chose SDTT model as teacher\n> Why do you opt to apply your method on top of another specific distilled model? Does the proposed method work well on top of a vanilla teacher model compared to the other distillation method?\n\nWe chose the SDTT model as the teacher for the following reasons.\nFirstly, we wanted to see if our method works even on top of a well-distilled model. While the teacher models (sdtt-6 / sdtt-7) went through many rounds of distillation, their modeling remains dimensionally independent. So, we wanted to confirm that there is still room for improvement by introducing dimensional correlations.\nSecondly, we were not aware of any other distillation methods for discrete diffusion (even though SDTT is just for masked diffusions) with which we compare our methods. By examining their performance gain (i.e., sdtt-6 -> sdtt-7), we could see how our methods work compared to another distillation method at the same time (please also see [q-2] for the actual comparison).\n\n### [q-2] Generative perplexity and diversity\n> It is known that the generative perplexity has crucial flaws [1,2], so it would be great to consider additional metrics for the text generation task.\n\nGenerative perplexity can indeed be easily hacked as it does not consider the diversity of the generated samples. Following our teacher model's work, we measured the Self-BLEU metric (similar to the sentence entropy used in [2] from your reference list, but lower is better) in addition to generative perplexity and plotted their trade-off curve in Figure 4(b). Our finding is that our distillation method (sdtt-6 -> sdtt-6+di4c(^2)) does not worsen the Self-BLEU compared to a round of SDTT (sdtt-6 -> sdtt-7).\n\n### [q-3] Relation to an advanced fast sampler\n> How does the proposed method compare to advanced samplers for discrete diffusion models such as [1]?\n\nThank you for pointing out this relevant work. We understand that the fast sampling method in the reference [1] (from your reference list) suggests that, in masked diffusion models, we can reduce the size of categorical sampling by first choosing the index (or indices in the parallel decoding variant) to unmask. In their parallel decoding variant, they do not model the dimensional correlations, so our mixture modeling can be combined to further improve sampling quality.\n\n### [q-4] Time step scheduling\n> Does the time step schedule (i.e., the choice of $s,u,t$) play an important role as in the consistency models [2,3]?\n\nWe suppose so, as the quality of our distillation target can vary depending on the time step scheduling. While we did not optimize the teacher sampling steps in our work, there is research optimizing the time step schedules in discrete diffusion [Par25], which can be combined with our method. With this approach, we can also optimize the step schedule after training the model (i.e., the step schedule in few-step sampling).\n\n### References\n\n[Des25] Deschenaux and Gulcehre. Beyond Autoregression: Fast LLMs via Self-Distillation Through Time. ICLR 2025.\n\n[Par25] Park et al. Jump your steps: Optimizing sampling schedule of discrete diffusion models. ICLR 2025.", "claims_and_evidence": "The claims are well supported with mathematical derivation and empirical evidence.", "methods_and_evaluation_criteria": "The proposed method is reasonable with comprehensive evaluation across several standard benchmarks.", "theoretical_claims": "Though I didn't go through every detail in the proof, the conclusion makes sense to me.", "experimental_designs_or_analyses": "The experimental designs for image generation seem nice to me.\n\nFor the experiment part of text generation, I have several questions:\n* Why do you opt to apply your method on top of another specific distilled model? Does the proposed method work well on top of a vanilla teacher model compared to the other distillation method?\n* It is known that the generative perplexity has crucial flaws [1,2], so it would be great to consider additional metrics for the text generation task.\n\n[1] Shi, Jiaxin, et al. \"Simplified and generalized masked diffusion for discrete data.\" (NeurIPS 2024)\n\n[2] Zheng, Kaiwen, et al. \"Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling.\" (ICLR 2025).", "relation_to_broader_scientific_literature": "Modeling complex correlations / joint distribution of multiple high-dimensional random variables is an important research question and this paper presents several nice ideas to tackle this.", "essential_references_not_discussed": "Several papers discussing the evaluation criteria of discrete diffusion models for text generation (e.g., [1]) and developing fast samplers (e.g., [2]) should be discussed.\n\n[1] Shi, Jiaxin, et al. \"Simplified and generalized masked diffusion for discrete data.\" (NeurIPS 2024)\n\n[2] Zheng, Kaiwen, et al. \"Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling.\" (ICLR 2025).", "other_strengths_and_weaknesses": "## Strength\n* This paper studies an important and underexplored research question of distilling discrete diffusion models.\n* This paper is well executed on both technical and empirical aspects.\n* The writing is easy to follow.\n\n## Weakness\n* The primary weakness of the current manuscript is the presentation of the main text, likely due to space constraints. As a result, readers must consult the supplementary material to fully grasp the paper's core technical contributions. Additionally, for clarity, a formal description of the sampling scheme should be included in the main text.\n* Please refer to the \"Experimental Designs Or Analyses\" part for my question about the metric and the teacher model used for text generation.", "other_comments_or_suggestions": "* In Eqn. (4), should the expectation be taken over $x_T$?\n* On the top of Figure 1 (right), should the second term be $p^1(x; \\beta)p^2(y; \\beta)$?", "questions_for_authors": "* How does the proposed method compare to advanced samplers for discrete diffusion models such as [1]?\n* Does the time step schedule (i.e., the choice of $s, u, t$) play an important role as in the consistency models [2,3]?\n\n\n[1] Zheng, Kaiwen, et al. \"Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling.\" (ICLR 2025).\n\n[2] Song, Yang, and Prafulla Dhariwal. \"Improved techniques for training consistency models.\" (ICLR 2024)\n\n[3] Geng, Zhengyang, et al. \"Consistency models made easy.\" (ICLR 2025).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a contribution to the field of discrete diffusion models by tackling the challenge of capturing dimensional dependencies while maintaining computational feasibility. The authors propose a "mixture" model approach for discrete diffusion distillation alongside tailored loss functions, supported by mathematical analysis and empirical validation across both image and text generation tasks. While reviewers raised some valid concerns about experimental design, evaluation metrics, and the practicality of requiring reference distributions, the authors provided rebuttals with additional results and clarifications. Given the theoretical rigor, cross-domain applicability, and the authors' responses to all concerns, I recommend accepting this paper as a step toward more efficient discrete diffusion modeling.
Decision: Accept (poster)