Efficient Perplexity Bound and Ratio Matching in Discrete Diffusion Language Models
Authors: Etrit Haxholli
Yeti Z. Gurbuz
Oğul Can
Eli Waxman
Venue: ICLR 2025 Poster (2025)
Keywords: Discrete Diffusion
CTMCs
Ratio Matching
Score Entropy
Cross Entropy
Language Models
Discrete Flow Matching
Primary area: generative models

Abstract: While continuous diffusion models excel in modeling continuous distributions, their application to categorical data has been less effective. Recent work has shown that ratio-matching through *score-entropy* within a continuous-time discrete Markov chain (CTMC) framework serves as a competitive alternative to autoregressive models in language modeling.
To enhance this framework, we first introduce three new theorems concerning the KL divergence between the data and learned distribution. Our results serve as the discrete counterpart to those established for continuous diffusion models and allow us to derive an improved upper bound of the perplexity. Second, we empirically show that ratio-matching performed by minimizing the *denoising cross-entropy* between the clean and corrupted data enables models to outperform those utilizing score-entropy with up to 10\% lower perplexity/generative-perplexity, and 15\% faster training steps.
 To further support our findings, we introduce and evaluate a novel CTMC transition-rate matrix that allows prediction refinement, and derive the analytic expression for its matrix exponential which facilitates the computation of conditional ratios thus enabling efficient training and generation.

Published as a conference paper at ICLR 2025
EFFICIENT PERPLEXITY BOUND AND RATIO MATCH- ING IN DISCRETE DIFFUSION LANGUAGE MODELS
Etrit Haxholli MetaDialog Research
Yeti Z. Gürbüz
Oğul Can
Eli Waxman
{etrith, yeti, ogule, eliblemetaddaaoog.com
ABSTRACT
While continuous diffusion models excel in modeling continuous distributions, their application to categorical data has been less effective. Recent work has shown that ratio-matching through score-entropy within a continuous-time dis- crete Markov chain (CTMC) framework serves as a competitive alternative au- toregressive models in language modeling. To enhance this framework, we first introduce three new theorems concerning the KL divergence between the data and learned distribution. Our results serve the discrete counterpart those established for continuous diffusion models and allow us to derive an improved upper bound of the perplexity. Second, we empirically show that ratio-matching performed by minimizing the denoising cross cross-entropy between the clean and cor- rupted data enables models outperform those utilizing score-entropy with up to 10% lower
and 15% faster training steps. To further support our findings, we introduce and evaluate novel CTMC transition- rate matrix that allows prediction refinement, and derive the analytic expression for its matrix exponential which facilitates the computation of conditional ratios thus enabling efficient training and generation.
INTRODUCTION
Modeling data distributions is fundamental task in machine learning. In the case of continuous data distributions, recent ad advancements in continuous diffusion models (Sohl-Dickstein et al., 2015 Ho al. 2020 Song et al. (2020b) have demonstrated impressive capabilities in generating data samples and performing density estimation (Song et al. 2021a c Haxholli & Lorenzi 2023 Kingma et al.)
2021ap. Despite these achievements, the application of such models to categorical data distributions, like language, remains limited, as continuous diffusion models generally underperform compared
to autoregressive models in these scenarios (Chen al., 2023 Gulrajani & Hashimoto 2024 Li
et al. 2022 Dieleman et al. 2022] (Strudel et al. [2022). To address this, recent research has focused
on the development of discrete diffusion models (Austin et al., [021 Campbell et al. 20022] Meng
et al., 2022; Lou et al. [2024] Sahoo et al.) 2024|| Shi al. 2024 Ou et al. 20255) which offer
distinct advantages compared to autoregressive models, such as the ability to infill various parts
of a sequence non-sequentially and have the potential to reduce computing time and expenses in
generating lengthy sequences (Deschenaux & Gulcehre 2024 Christopher et al. 2024).
Evaluating discrete diffusion models, however, presents practical challenge due the difficulty in calculating the perplexity, unlike in autoregressive models where this computation is straightfor- ward. Although recent perplexity bound has been proposed by Lou al. (2024), no tightness (KL)
guarantees exist. In this paper, we present three theorems concerning the Kullback-Leib divergence between the data distribution and the learned distribution in discrete diffusion models (Lou et al. 2024). These results serve as the discrete analogue of the continuous diffusion theorems provided in Song al. (2021b). One of our key contributions Theorem 4 which provides an upper bound (J2) on the cross-entropy between the data and learned distributions, offering a more direct way of bounding the perplexity. This bound is computationally more efficient than the existing bound in Lou et al. 2024), and empirical results suggest that is also slightly tighter.
Code available http:///ggtaaa..co//aaaaaaallage

---

Published as a conference paper at ICLR 2025
In addition, inspired by model reparametriizaioos (Ho al. 2020; Karras al.) 2022 Lou al. 2024), this paper examines the matching training objective SEDD in Lou et al. (2024). We highlight that the sole unknowns implicitly learned by the model are the per-token marginal proba- bilities across the vocabulary, conditioned on the current perturbed sequence. Consequently, rather than modeling the ratios directly, we employ weighted version of the denoising cross-entropy loss Lu proposed in I[Cambeell et al. (2022), which also mirrors the cross-entropy loss utilized in continu- ous diffusion models, as described in Dieleman et al. (2022). We show empirically that by modifying the reconstruction of the scores, training with cross entsopy outperforms direct ratio matching for all types of tested discrete diffusion dynamics. We name this strategy of using cross-entropy for train- ing and the adjusted ratio reconstruction for generation, cross-entropy discrete diffusion (CEDD). Similar advantageous results when utilizing the cross-entropy loss have been reported in studies of absorb discrete diffusion, shown in concurrent research by [Sahoo et al. (202) ; (Shi et al. (2024); Ou et al. (2025). These particular results, up to a scaling factor, are specific cases within the broader CEDD framework. CEDD improves upon SEDD by circumventing the learning of conditional ra- wos-h are analytically determinable- focusing instead on learning the mixing weights that constitute the necessary marginal ratios for generation. This focus not only conserves modeling sources but is also particularly advantageous when the distribution of conditional ratios is complex.
To illustrate our point, design new transition-rate matrix named rouletle diffusion, and derive its matrix exponential. The roulette diffusion is an interpolation between the absorb and the uniform diffusion. In the forward process, a token can transition any state until hits the absorb state, that is, until it is masked. In return, the reverse process begins with a sequence of masked tokens, which are gradually unmasked, and where the unmasked tokens can be refined. Intuitively. this capability should be important for discrete diffusion models (Deschenaux & Gulcehre 2024), and can be useful for downstream tasks, as shown our spelling correction experiments. Moreover, during the reverse process, the scores/ratios of unmasked tokens have much larger magnitudes than those of masked tokens, posing a significant learning challenge for the network due to output scale variability. Employing the CEDD strategy mitigates this challenge as demonstrated experimentally.
In summary, the main contributions of this paper include:
Providing 3 new theorems concerning the KL divergence and cross entropy between the data and the learned distribution. Improving model evaluation through the bound provided in Theorem [
Introducing a new transition-rate matrix (roulette diffusion) that allows token correction after unmasking in the reverse process. Deriving its matrix exponential which enables efficient training using SEDD and generation when CEDD is employed.
Comparing the performance of SEDD and CEDD experimentally in the task of language mod- elling on absorb, uniform and roulette diffusion models. Showing that CEDD outperforms SEDD in all cases in terms of perplexity.
PRELIMINARIES AND NOTATION
2.1 MARKOV CIIINNS OVER FINITE- SPACES
A discrete-time Markov Chain in finite-state space is a stochastic process X1, X2,..., where each state X, depends solely on the preceding one. The states X, can take values from {1, (1.2....,S)) 8), and T represents the number of time steps. The probability of being in state at time is
S
(X, = 2)
Pi = n, X1-1 = y) = Ptjt Pq--i(X, = x| = = Σ
g)) (1)
Placing all such probabilities p. (X, x) in vector 8t, such that 8,(1) = pr(X, = I), gives
st = PP where P(x,y) = PHtt- = x = y).
(2)
One can generalize such processes into Continuous Time Markov Chains (CTMCs) where f E [0, T}, (Andrsson 2012). For simplicity, make the choice T 1. To construct CTMC, one first chooses a transition matrix Q.. which has the property that its non-diagona elements are non- negative, and the elements in each of its columns add to zero (Suhov & Kelbert 2008). Given an
2

---

Published as a conference paper at ICLR 2025
initial probability distribution so, the equation below fully determines the evolution of the probabil- ity with respect to time:
dsc Q.8t.
dt
In addition, choose Q: o' (t)Q, where is itself constant transition-rate matrix and where function a is monotonically increasing, and satisfies (0)) = as well as lime-> ot) = '. In this setting, the distribution over states time the solution of the linear ODE in Equation ( that is,
8, e so-
Matrices Q: are chosen such that: a) the matrix exponential (0 easy to calculate, which is essential as (go(xly) = ea(t) (z. y); and b) 81 is an easy reference distribution to sample from
(Austin et al. 2021 Campbell al. 2022).
Finally, similar to diffusion processes in continuous spaces, the continuous Markov chain in Equation (3) also admits reverse process (Kelly) 1979. Sun al. 2023
d81-t Q1-181-11 dl
where Qx( Qr 2)) pe(M) for * y. and Qx - Q.(, Since can easily sample from the reference distribution, the only unknowns preventing us from being able to
Pe(x)
run backwards are the ratios Pe(w)
p:ly) also known as concrete scores (Meng et al. 2022 Lou et al.
2024). Once such ratios are modeled using neural network, we can generate samples from the learned data distribution p by discretizing Equation (4) as follows:
p(zc-e = y 2 = 2) = 6z(1) + Q((,
(5)
Additional details are provided in Appendix D
2.2 SEDD: ESTIMATING THE RATIOS VIA SCORE ENTROPY
As pointed out in the previous subsection, we wish to model the ratios Pe(v) p(
via neural network
so (z f)y for example by minimizing the score entropy loss (Lou et al. 2024
Pc(y) (z,)
E Ezarpa
Σ War.y yy
so so so), for E(a, b) (b a logg K(a)),
(6)
(Te, y), and furthermore they show that and (a) allog a 1). In Lou et al. (024), WFLg an equivalent loss is the following:
Pro(y(kro) Pejo(z+
Σ Warol
s0 (x(e
which side-steps the problem of not knowing the marginal ratios
Pe(y) by employing (e())
Papo(ili) (01 (i. A more detailed derivation of Equation 7 can be found in Appendix A.2
2.3 DISCRETE DIFFUSION FOR LANGUAGE MODELING TOKEN LEVEL TRANSITIONS
In the case of Language Modeling, we write sequence of length L from the data distribution as To, where Zy (x) 26. x6) and z Vocab = {1, .... V}. The number of possible sequences, that is, the number of states 5 is VL. Unfortunately, this implies that it is not computationally feasible model the ratios of probabilities between the current state æ, and all other states y. since the output of our neural network would have to be VL dimensional (Campbell et al. 2022).
We follow the usual approach (Campbell et al. 2022 Lou et al. 2024 to mitigate this issue, which to select a sparse matrix Q. S), such that each entry Q, (xx y) for two sequences differ in more than one token will be zero. The forward process that such a Qi defines, can equiv- alently described as follows: each discretized step, only from the current sequence can be modified, according to a token level forward diffusion process Qiok (V x V). More formally, for (xt. x.. ... "..,,-.. 2x)) and y (#.g..., ...,..... ..,,,)). if xt + 4
that
uniformly randomly chosen token

---

Published as a conference paper at ICLR 2025
have Qx y) Qi sparse choice of Qt, and y which only differs from a at a single position i, Expression
@ (æx,, y²), otherwise Qi is zero in other non-diagonal entries. For such
( becomes
[1 11-Esteocaah Qt (x*, 2²) pi(x) + O(E2). if = æ Q10hk (st y²) p=(y) p(l=)
p( - oe xt= x))=
(8)
O(e2).
if y + x
where: z denotes a sequence that is identical to x everywhere, but position i. Thus, the usual ap- proach entails only modeling the ratios between a and 'neighbours' y which only differ from by one token. The number of such neighbours is L x V, that is V per each of the L positions, hence the output of the network is V coinciding with that of transformers language models. It should be pointed out that one can indeed use Expression ( for training, due the fact that tokens are perturbed independently from one another in the forward process (aj (zo)
autoregressive
II, (x, ( and thus
( Il Pojo (ojo(y' (xo0) (1)0(z1 II Ps ol (>>² (m)))
Pajoy""lon (mn') Ps (oo") eqi (W',a))) II
| moo) Proo(ze|lzo0
= eash (zf mn))
Finally, the noise schedule or is typically loglinear - log(1 (1 - e)t) or geometric man maz
METHODOLOGY AND THEORETICAL RESULTS
In Subsection 3.1 we provide results related to the cross entropy and the KI. divergence between the data and the learned distribution in the CTMC (discrete diffusion) framework. The first three theorems therein can be considered as the discrete diffusion analog the ones given in (Song al., 2021b). Importantly, Theorem provides an upper bound (J2) on the cross entropy between the data and learned distribution which can be used bound the perplexity, and which does not depend on the function K (Equation 6). We emphasize that the results hold for general CTMCs, and not only in the special case of token-level transitions. From the second subsection onwards, we operate in the token-level transition framework. More precisely, in Subsection 3.2 we introduce the roulette transition-rall rate matrix, and provide an expression for its exponential. In Subsection 3.3 we state Proposition 6 which enables more efficient estimation of J2. In Subsection 3.4 highlight that similarly to the continuous case (Dieleman et al. (2022), the ratios can be modeled using Lu from Campbell et al. (2022), and present how this approach is adapted in our experimental setup.
3.1 CROSS ENTROPY AND KL DIVERGENCE RESULTS
We begin by finding an upper bound for the KI. divergence between the data and the learned distri- bution. The proofs are provided in Appendix A.
Theorem 1. Define a CTMC with transition matrix Qe that runs from time 0 to 1. The true reverse process defines a probability evolution Pt from p1 to the data distribution po, while the learned re- verse process induces the evolution pt from the reference distribution P Pr to the approximation of the data distribution ph. In this setting, the following KL. divergence bound holds
DKI(Polle8) < {(EEA- Errep Σ Q.(,,, Pe(()) (x.))
sa sa(x1, ()y di DRI(Pillpe). (10)
where E(a, b) = (b a log K(a)) and ((a) = a(log a 1).
The following theorem provides an expression for the entropy of the data distribution. Furthermore,
it provides sufficient conditions for the bound given above to become tight.
Theorem 2. Denote the intermediate distributions at time determined by the reverse process, and by the learned reverse process with pt. and pú, rexpectively. We can write the entropy of the data distribution H (po) as
(P(())
H(p)) HH())((p)))))))) - H(pp))(
Earp, (o B)K
dt.
(1I)
pr(
In addition, if the learned ratios so ( t)2 equal (.) and p1 = pf == Pr. then the inequality in
p (zc)
Theorem 7 becomes an equality.
4

---

Published as a conference paper at ICLR 2025
A particular case where the conditions of the theorem above hold is when (0 f)y
pe(m)) then
(v(v)
rl(u) p((2)
pf(x.] [xx=] se (Ze, t)y. The third theorem gives an upper bound of the negative log- -igeliidldood single point. This central result in Lou al. (2024 Theorem 3.6), but restate here for completeness, and provide an alternative, more detailed proof in Appendix
Theorem 3. Let ph denote the learned distribution from which the reverse process samples. The negative log-probubility of a state vo being sampled by the reverse process can be bounded from above as follows,
log (ogro((0) S [ Exempio 2o)
Ps Peuo(u so s0 (7g di Pejo( (20)
E Qi(ee ve yz
+ DDKo(Pio(-laap
(12)
Since the noise schedule in (Lou al. 2024 Ou al. 2025) chosen such that pt Pr and thus (PIjoolllp;; = 0, one can take the expectation Er, on both sides of Inequality 12) to get bound on the cross entropy +, (po. p&) I-21-1 (zo)) One approach is to compute the RHS in Expression (12) for each point zp and then average results (Appendix [B.5). which can be computationally expensive. Instead, can divide by L and take the expectation with regards to data distribution on both sides of Expression (12 as in Ou et al. (2025), and calculate
J1 - - Exo-po E Be
Σ yrffare
Qi( u)e
(Pto(Bl Pejo (xe
so( (w) (13)
Using Theorem and 2 provide another, direct upper bound on the cross-entropy between the data and learned distributions, which evades the computation of K.
Theorem 4. Under the conditions stated Theorem between the data and the learned distribution holds:
7
the the following inequality for the cross entropy
Pr(y) (p((x)
H(po. pb) S Encope Σ Ve(sere)
so so(xe,t))) dt
E Enpppe
Σ Oelu.za) + H(P,, where Ela, b) = (6 log b).
(14)
The second term So Brop Eysrr Q (y, xx )d and third H(1, pe) Rs H(r,) be analyt- Finally, the first term can be rewritten
ically computed shown in Section 3.3 Proposition 6
(o)o()) (e (ro) so so(*2, (12))))))))
( paol)
Σ
Qu(x,,y)e
(15)
Therefore, due to Theorem we instead propose to use
J22
1 1 L
(mr)
Σ Qu(ae,
so (2 t)y
+H 1 Eerrp Σ Qely,
(16)
yfu,
In both cases Perplerity = ex. log(e)((a))) = exp(+ p()) s Seh, et2,
3.2 ROULETTE DISCRETE DIFFUSION
Typically, matrices Look are defined as Qiok ptok I. where plok idempotent, since this implies that (Qtoky² - iQook This last property of Qiok greatly simplifies the calculation of
((())(* as by using the Taylor series, (l)) I+Q tok (o()). Usually, the following two
5

---

Published as a conference paper at ICLR 2025
matrices ptok are chosen: a) Pine form (V V) where each entry is set to }. and b) Pank absarb n) in which the last row is full of ones while all other elements are 0, where n = V + 1. While absorb diffusion often outperforms the uniform one in standard evaluations, the latter more practical for some tasks like spelling correction, where refining tokens is crucial. We propose another transition- rate matrix whose exponential can be analytically calculated. To our knowledge this is the only matrix with such a property presented far apart from the absorb and uniform ones. We refer to this new discrete diffusion process as roulette diffusion. The corresponding Poulette (n x n) is a matrix, such that piok Prouettte (i & n. # n) = +1 Pm), Poulette [n.j * n) = Pms Poulette (i + n, n. = 0 and Prollctte (n, n) = 1. We notice that for Pm = 1, roulette diffusion coincides with absorb diffusion, while for Pm 0 it coincides with the uniform diffusion. Intuitively, token can transit from non-absorb state to a non-absorb state with probability {1 pm), until it hits the absorb state (with probability pm) and then remains there. While this matrix is not idempotent, one can still calculate its exponential as stated in the following proposition (proved in Appendix A.3.3)
Proposition If we denote with Y, the matrix exponential of az lokk roudctte
= σt (Ptak
roudette I).
1 then Yi & [j.niij # n)) = eeee Pmp n (1 e - (1-Pm)))), Ye(i # i # n) = e eospin (1
R- 7 - e- (1 (1-pm)))) )). Yi (i + n) =1 a.pm Yi # n. n) = 0, and (n,, n)) =1
The noise schedule used is the roulette-loglinear noise cess, when token is unmasked it can still be corrected with probability directly related to pm as shown in Appendix /A.3.4] A generalization for time-evolving Pm is given in Appendix [A.3.5] and the corresponding @eroulette (t) is named eroulette. Therein (Proposition 7 it is shown that the exponential matrix can be calculated as in the previous proposition, by substituting pm with pm Pmn(). (t).
Fry log (1 e))) In the reverse pro-
3.3 EFFICIENT ESTIMATION OF J2
In this subsection provide Proposition which shows that the second and third term on the RHS of Expression (14) can be computed efficiently:
Proposition 6. In the case of the rouletle diffusion with roulette-loglinear noise, H(p)) 0 and -So Ex.xpe Eyte. 2.)y, (1 (1- 1P L. (c 1). For the absorb diffusion, have ((r,) = and Jo Bexip Euto, Qily, x.)l = I(e - -1). Finally, in the case of uniform diffusion, H (p,) = I. log(()) (V) and - Jo Baypp Eyto, (yy = - - +) Jo (t)dt.
Pm
3.4 MODELING RATIOS VIA CEDD
For sequences æ, y which only differ at some position i, we can write
Prly)
EnEiv] (3 (h) Pejo (xx 14] Poje (hla:,,
(17)
Pr (xx))
where Poje (x:) Σ t Man Poje (oolwa)) as shown Equation Appendix
A.2.2
Pe [b) are known, we can choose to reparametrize the score Pejo 2x
Since conditional ratios
Puo(y (h) (ro((xt())) (a t)]h).
(x,,, t)y Ene(v]
(18)
where fara,; t), is neural network, with matrix output of shape T. V. whose elements in each row i add to 1, that is En (z,,,))) [)] = 1. Intuitively, given the current perturbed sequence æ;, prediciton fā [h) gives the probability that the pre-perturbation token at position used to be
h. In Appendix A.2.2 explain how the loss in Expression (7) can be optimized by minimizing
the following cross-entropy loss:
Lu (,, Exg Em, (=(o)) Σ w(t) log (a [=(]]]
(19)
Thus, we can learn the ratios via Lu (Campbell et al. 2022) for any type of diffusion model. This approach is analogous to the one in continuous diffusion models for language modeling used in

---

Published as a conference paper at ICLR 2025
Dieleman et al. (2022), as shown in Appendix A.2.3 In addition. in Appendix B.3.2 we provide our original motivation for the reparametrization given in 18
In the uniform case, models using the direct reparamettrizaionn in 18 underperform SEDD in terms of perplexity. Indeed, the model is too confident in its predictions when t + 0 as it does not benefit form the neural network regularization due to the incorporation of the true conditional ratios. Thus,
P=j[ we rescale of < 0.0015, by setting of 0.0015. We also perform rescaling the case of roulette diffusion dynamics, as when time is close to most tokens are unmasked. The same strategy is applied, before, only to ratios corresponding to unmasked token, by rescaling of when of < 0.5 as follows: 0z g,cated00 = log(1.10f + 1.1). Naturally, for the sake of rigor, these are also the models we employ to generate samples, whose quality is measured in terms of generative perplexity. Further, in order to evade metric hacking, we take care to not modify the of in the metrics J1 and J/2 used to evaluate the models. Additional details and motivations for such rescalings are provided in the last paragraph of Appendix [.3.3] When w(t) 1 we refer to the method as CEDD, while when wt)) = log (e + 0.3 we use CEDD*
inside the ratios Peja(y" (k)
())
4 EXPERIMENTS
We now empirically validate the approaches and theoretical contributions presented in the previous section. In Subsection 4.1 compare the generative perplexities of models trained on Web- Text (Gokaslan & Cohen 2019) with SEDD and SEDD scaled (SEDDs, see Appendix [.3.2 for details) versus those trained with CEDD and C We keep all other variables unchanged for a fair comparison, finding that CEDD outperforms SEDD in all cases. The tests are conducted for the absorb, uniform and roulette diffusion dynamies. In Subsection
4.2 we evaluate the perplexity of the models, by calculating the upper bound on 5 different datasets. namely: 1BW, LAMBADA,
PTB, Wikitext2 and Wikitext103 (Chelba et al., 2013 Paperno et al. 2016, Marcus et al. 1993]
Merity et al.) (016). For the sake of reproducing the results of previous work we use J1 as a metric.
Finally, having computed the results using J/, in Subsection 4.3 we also re-evaluate the models via
J2. Our findings suggest that the bound provided by J2 is slightly tighter. In the last subsection compare SEDD and CEDD* on a spell-checking task.
Our model employs the transformer architecture as described by Lou et al. (2024), with no modi- fications; more details can be found in Appendix [.2]] The algorithms for training via SEDD, and sampling unconditionally and conditionally can be found in the Appendix Lou et al. (2024). On the other hand, we provide the algorithm for training using CEDD in Algorithm 1 Appendix B.
In all cases, samples are generated using tau-leaping (Gillespie) 200T; Campbell et al. 2022). which performs an update at each position simultaneously for each reverse time step. All models were trained for 400k parameter updates unless stated otherwise.
4.1 GENERATIVE PERPLEXITY COMPARISONS
We compare the generative perplexities (GenPerp) of identical networks trained using SEDD, SEDDs, CEDD and CEDD*. To evaluate the generative perplexity of a model, we generate samples from that model, and use GPT-2 large model in order to assess the likelihood of the generated samples. However, this metric can be unreliable, as models such as GPT-2 large are not perfect themselves, and they tend to assign high probability to some unlikely sequences, such those that contain repetitive tokens. Such biased samples can be generated by increasing the step size, while maintaining the number of reverse steps. To ensure fair comparison, we evade such approaches. Furthermore, recently Zheng et al. 2024) showed that the sampling procedure in Lou et al. (2024) suffers from numerical precision issues. To address this, they proposed fixing the categorical sam- pling to bit floating- point precision- a strategy that we have also adopted. In Appendix [B6] we also provide perplexity results as evaluated by LLaaa 3.1 8B (Dubey al. 2024).
Initially, we perform grid search to find the best pm. for the roulette case. Out of the 4 values, 0.95, 0.65, 0.35, 0.05, we found pm = 0.95 performs best (Appendix [B.6] Table but the opti- mum is likely reached when Pm = 1. However, Pm = 0.95, enables equipping unmasking dynamics with the correction mechanism, which can be useful for some tasks like spelling correction. There- fore, in what follows, when the roulette case is concerned, it is implied that p.. = 0.95. In Table [ we provide the results when the sequence length is 128 and the number of reverse steps is 1024, when using the analytic sampler. Generated samples can be found in Appendix C.1
7

---

Published as a conference paper at ICLR 2025
4.2 PERPLEXITY COMPARISONS
In this subsection, we compare the performance of identical networks trained using SEDD, SEDDs, CEDD and CEDD*, in terms of J1. We do not shuffle the test set. Results for sequence length L = 128 are provided in Table Plots illustrating cummulative performance trajectories across the testing sets are provided in Appendix B.6
Generative Perplexity Network Evaluations.
Sceing treat, hecause cnaching. full week for me. "On Saturday night, Gonzalez spoke to media and said he thinks he could work the fine fit make the made
240
SEDDs
Generative Perplexiy
CEDD
GPT-2
Seeing them treat, though presence. Just being able to think about all the soldiers
only her
120
GPT CEDD*
the place look like their countries and the rest of
helped
60
Seeing them treat, ultimately all the prac- tical work should be well underway for everyone enjoy. Milan, taly-hased will be access the dev portal
memhers
12
64
120
256
512
1024
2048
Figure Filtered samples from SEDDs and CEDD* absorb, L=1024. The conditional part is highlighted bold.
Network Evaluations
Figure 1: Scaling of Generative Perplexity vs sam- pling steps for SEDDs (loaded) and CEDD* absorb.
Table 1: Results comparing SEDD, SEDD scaled (SEDDs), CEDD and CEDD*. Lower is better.
Model (L=128)
GenPerp
LAMBADA
WikiText2
PTB
WikiText103
1BW
SEDD Absorb
172.35
70.07
75.20
240.43
74.79
88.99
SEDDs Absorb
166.35
67.05
69.37
208.69
69.17
83.87
CEDD Absorb
148.21
65.18
65.66
199.69
65.62
79.83
CEDD* Absorb
143.86
64.60
65.04
192.99
64.69
79.81
SEDD Roulette
178.94
72.07
80.13
230.74
79.68
93.45
SEDDs Roulette
172.93
69.10
74.38
209.12
74.16
88.02
CEDD Roulette
167.67
69.77
72.91
227.16
72.49
86.55
CEDD* Roulette
158.56
67.84
70.54
216.91
70.18
86.76
SEDD Uniform
169.66
80.74
91.79
252.81
91,40
102.75
SEDDs Uniform
163.88
81.13
89.21
228.37
88.56
100.80
CEDD Uniform
161.84
80.27
87.91
279.65
87.46
99.34
CEDD* Uniform
175.42
82.54
89.68
289.09
88.90
106.32
DFM ki
145.48
71.90
71.20
221.15
70.84
82.63
DFM K;
152.70
72.31
72.87
215.30
72.55
85.82
We also compare our approach against models trained utilizing Discrete Flow matching (DFM) (Gat et al. 2024). In Table I we present results when comparing against flows with convex interpolants, where we chose schedules k; f, as in Campbell al. (0024), as well as k; t2. The perplexity bound for these models is calculated using Expression (24) in Haxholli et al. (204)).
can be seen that CEDD* absorb performs best overall, thus we compare this model, against CEDDT, that is, CEDD with the scaling loss used in the SOTA discrete diffusion model (Sahoo) jtt al., 2024). Interestingly, our scaling CEDD* outperforms that of CEDIT, despite theoretical support with regards to the score entropy loss. The results can be found in Table 2
We also train absorb models, namely SEDDs, C CEDDT, as well as GPT2, with a sequence length of 1024. Results are provided in Table 2 where it can be seen that overall GPT-2 performs best. The gaps between SEDDs, CEDD* CEDDT are reduced, likely since by seeing more tokens they all approach their optimal performance. However, models trained with CEDD/CEDD* verge faster to the optimum in terms of number of parameter updates. In Appendix B.6 Table 7 and Figure show the difference in performance between CEDD* and SEDDs absorb during
8

---

Published as a conference paper at ICLR 2025
and after training for 20k parameter updates. In addition, training with CEDD (and its variants) roughly 15% faster per iteration, due to the simplified loss function. Furthermore, by incorpo- rating the f) to (2,, t) scaling in the timestep, absorb models trained with CEDD/CEDD* generate sequences 2% faster than those trained with SEDDs.
Results comparing SEDDs (retrained), CEDD* CEDDT and GPT-2 (retrained) in terms of generative perplexity, and perplexity on 5 test sets. Number of generation steps is 1024.
Table 2:
Model (Absorb)
GenPerp
LAMBADA
WikiTex12
PTB
WikiText103
IBW
SEDDs L=128
166.35
67.05
69.37
208.69
69.17
83.87
CEDD* L=128
143.86
64.60
65.04
192.99
64.69
79.81
CEDDT L=128
154.04
68.24
68.61
204.76
68.10
81.81
SEDDs 1==0024
105.27
52.18
42.02
117.00
41.83
80.79
CEDD* 1.=1024
101.83
52.70
41.57
115.99*
41.31
77.96
CEDDT L=1024 108.88
53,20
42.24
121.05
42.07
78.10
D3PM 102044
93.47
77.28
200.82
75.16
138.92
PLAID 1=1024
57.28
51.80
142.60
50.86
91.12
GPT-2 L=1024
41.02*
49.02
37.68
134.13
37.55*
58.92
4.3 COMPARING THE TWo UPPER BOUNDS
Lastly, compare the two upper bounds Jı and Jz. The bound J2 is shown empirically to be slightly tighter than J, supporting the importance of Theorem Proposition 6 We estimate the integral with respect time both JI and J2 by randomly sampling time points from uniform distribution in the interval (e 1 e e={), and the averaging the loss. The procedure of comparing these bounds explained in detail in Appendix B.4] We present the results for several models in
Table 3, while we show the testing plots for CEDD* absorb (I==1244) in Figure 3 The proposed
bound 22 gives lower bound every single case, as can also be seen in Table Appendix B.6
where we provide the equivalent of Table when J2 is utilized.
Table 3: Results comparing J1 and J2 for the best performing models of each category.
Mode//L/Perplextty-Bundd
LAMBADA
WikiText2
PTB
WikiText103
1BW
SEDDs absorb/1024/ exp(X) (J1)
52.18
42.02
117.00
41.83
80.79
SEDDs absorb/1024/ cxp(Jz)
51.78
41.76
115.97
41,51
80.53
CEDD* absorb/1024| cxp(J11
52.70
41.57
115.99
41.31
77.96
CEDD* absorb/1024/ exp(J2)
52.10
41.18
115.03
40.98
77.28
CEDD* absorb/128/ exp(J1
64.60
65.04
192.99
64.69
79.81
CEDD* absorb/128/ exp(J2)
64.11
64.54
191.38
64.30
79.17
CEDD* rouletic/128/ exp(J1)
67.84
70.54
216.91
70.18
86.76
CEDD* roulette/128/ exp(J2)
67.27
69.61
213.90
69.45
85.64
CEDD uniform/128/ exp(Ji)
80.27
87.91
279.65
87.46
99.34
CEDD uniform/128/ exp(J2)
79.46
86.82
276.61
86.52
98.44
119
54.0
expl
53.5
78
41.5
expljs
53.0
52.5
77
117
41.0
51.5 k
52.0
50k
180k
150k
200k
75
1k
50k
100k
150k
200k
115
1k
50k
100k
150k
200k
40.0
%
50k
100k
150k
200k
Iterations
Iterationn
Iterations
Iterations
(a) LAMBADA
(b) IBW
(c) PTB
(d) WikiText103
Figure 3: Upper bounds J1 and Jz of CEDD* absorb L=1024 for different testing sets.

---

Published as a conference paper at ICLR 2025
4.4 SPELLING CORRECTION
We evaluate our uniform and roulette diffusion models on a character-level unsupervised spell- checking task (Hoogeboom et al. 2021). We train CEDD* and SEDD models on "War and Peace' (3.3M tokens), and test on "Crime and Pumishment" (CAP) and 'Pride and Prejudice' (PAP). The test set is contaminated with mistakes (5% of characters), and the model chooses the most likely correction. Results when training for 25k/50k iterations are provided in Table and Appendix B.6
Table Correction accuracy percentages. 25k training iterations, batch size of 32 and 128.
Model (L=128)
CEDD* Uniform
SEDD Uniform
CEDD* Roulette
SEDD Roulette
PAP
89.5
86.9
89.7
85.1
CAP
89.5
87.5
90.3
85.8
5 RELATED WORK AND FUTURE OUTLOOK
The continuous difftusion approach has demonstrated excellent performance in modeling continu-
ous data distributions (Song et 2020a
2021e Kingma al.
2021b Nichol & Dhariwal 2021
Saharia et al. 2022 Ramesh et al. 2022). leading to numerous efforts to adapt it for language mod-
eling tasks (Chen et al. 2023] Guirajani & Hashimoto| 2024; Li et al. 2022 Dieleman et al.) 2022 Strudel et al.] 2022 Gong et al.) [2022] Mahabadi et al. 2023). Although initial results were not competitive, recent advancements have reduced the performance gap with autoregressive models. Discrete diffusion models offer alternative approach for modeling categorical distributions like language data. Originally proposed by [HHogeboom et al. 202T):(Autinn et al. (202T). the framework was extended to continuous time by Campbell et al. (2022). Both strands of work employ combi- nation of the variational lower bound and cross entropy loss, the latter being central to our training approach and corresponding the strategy employed by Dieleman et al. (2022) in the continuous case. The cross-entropy loss is also derived ir(Sahoo et al.) (2024); [Shi et al. (0024): Ou et al. 20255) but only for the absorb transition-rate matrix. In addition, the cross entropy loss is similar the loss employed in Sun et al. (2023), however in their case, conditions without the token at position i, and attempts to maximize the likelihood of æ:t. In contrast, Lou et al. (2024) proposed modeling ratios of probabilities (Meng al. (022)] directly using score entropy. The evaluation of such models can be performed by using Theorem 3 as originally derived and used in Lou et al. (2024). Inspired by this result, we formulate and prove the discrete version of the rest of the theorems in Song et al. (2021b). which provide important information regarding the KL divergence between the data and learned distributions in CTMCs, and which justify the usage of J2- Transition-rate matrices are an essential component in CTMCs, as demonstrated by the perfor- mance discrepancy between the absorb and uniform matrices (Lou et al., 2024 Camphell et al.) 2022). We introduced roulette matrix, interpolation between the two and derived its exponential. Regarding future work, numerous avenues remain open for optimization within the diffusion model framework such the exploration of the Eroulette matrix. Additionally, the exploration of noise schedules in discrete diffusion models remains relatively nascent, as similar to Lou et al. 2024), we did not systemically explore noise schedules. In our experiments, cross entropy loss is modulated by heuristically chosen time-dependent weighting. Investigating and establishing a gen- eral optimal weighting schedule could further refine performance metries. Finally, studying scaling laws for discrete diffusion models and establishing their practical utility in downstream tasks is crucial.
the current sequence ax,"
CONCLUSION
In this work, we provided three new theorems concerning the KL divergence between the data and the learned distribution, improving model evaluation through the bound presented in Theorem 4. We also introduced a new transition-rate matrix that allows for token correction after unmasking in the reverse process, and derived its exponential matrix to enable efficient training/sampling. Finally, we proposed favoring denoising cross entropy loss over score entropy for training discrete diffusion models due to the findings in the experiments conducted.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
This paper introduces research that progresses the field of natural language generation. Beyond the pre-existing ethical concerns associated with this domain, such as c.g. bias, toxicity, and the generation of deceptive content, our methodology poses no unique risks. This is because our work primarily theoretical and not of magnitude that could cause any distinct issues.
REFERENCES
William J Anderson. Continuous-time Markov chuins: An upplications--orieeeee approach. Springer, 2012. doi: htps://dooi.or//1... 1000
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg, Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:17981-17993, 2021.
Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. Advances in Neural Information Processing Systems, 011222666-22777 2022.
Andrew Campbell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. Generative flows on discrete spaces:: Enabling multimodal flows with applications to protein co-design. In Forty-first International Conference on Machine Learning. 2024.
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Kochn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv: 1312.3005 2013.
Ting Chen, Ruixiang ZHANG, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with seff-conditioning In The Eleventh International Conference on Learning Representations, 2023.
Jacob K Christopher, Brian Bartoldson, Bhavya Kailkhura, and Ferdinando Fioretto. Specu- lative diffusion decoding: Accelerating language generation through diffusion. arXiv preprint 2:Xi::2408.05636 2024.
Justin Deschenaux and Caglar Gulcehre. Promises, outlooks and challenges of diffusion language modeling. arXiv preprint arXiv:2406.1 11473, 2024.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, et al. Continuous diffu- sion for categorical data. arXiv preprint arXiv:2211.1508) 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint 4rXiv:2407.2 240..2 2024.
Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky TQ Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching. Advances in Neural Information Processing Systems, 37: 2024.
Daniel T. Gillespie. Approximate accelerated stochastic simulation of chemically reacting systems. The Journal of Chemical Physics, 17164)37 07 2001. ISSN 0021-9606. doi: 10. 10.1063/ 1.1378322.
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus, 2019.
11

---

Published as a conference paper at ICLR 2025
Shansan Gong, Mukai Li, Jiangtao Feng. Zhiyong Wu, and LingPeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models. arXiv preprint arXiv:2210.08933, 2022.
Ishaan Guirajani and Tatsunori B Hashimoto. Likelihood baaeed diffusion language models. Ad- vances Neural Information Processing Systems, 36, 2024.
Etrit Haxholli and Marco Lorenzi. Faster training of diffusion models and improved density timation via parallel score matching. arXiv preprint arXiv:2306.02658 2023. NeurIPS 2023
Workshop Diffusion Models.
Etrit Haxholli, Yeti Gürbüiz, Oğul Can, and Eli Waxman. Minibatch optimal transport and per- plexity bound estimation in discrete flow matching. arXiv preprini urXiv:241 11.024411.07799 2024.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In
H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Infor- mation Processing Systems, volume 33, pp. 6840-6851.
Curran Associates, Inc., 2020.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. In A. Beygelzimer, Y. Dauphin, Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021.
Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
F.P. Kelly, Reversibility and Stochastic Networks. Probability and Statistics Series. Wiley, 1979.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang. and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 21696-21707 Curran Associates, Inc., 2021a.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. On density estimation with diffu- sion models. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021b.
Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. Diffusion- Im improves controllable text generation. Advances in Neural Information Processing Systems, 35:4328-4343, 2022.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In Forty-first International Conference Machine Learning, 2024.
Rabeeh Karimi Mahabadi, Hamish Ivison, Jaesung Tae, James Henderson, Iz Beltagy, Matthew Peters, and Arman Cohan. Tess: Text-to-text self-conditioned simplex diffusion. arXiv preprint arXiv:2305.08377, 2023.
Mitchell Marcus, Beatrice Santorini, and Mary Marcinkiewicz. Building large annotated corpus of English: the Penn Treebank. Computational Linguistics, 19(2):313-330 1993.
Chenlin Meng, Kristy Choi, Jiaming Song, and Stefano Ermon. Concrete score matching: General- ized score matching for discrete data. Advances in Neural Information Processing Systems, 35: 34532 34545, 2022.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint
2016.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusson probabilistic models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 3162-8171. PMLR, 2021.
12

---

Published as a conference paper at ICLR 2025
Manfred Opper and Guido Sanguinetti. Variational inference for markov jump processes. In J. Platt, D. Koller, Y. Singer, and S. Roweis (eds.), Advances in Neural Information Processing Systems, volume 20. Curran Associates, 2007.
Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. In The Thirleenth International Conference on Learning Representations, 2025.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring broad discourse context. arXiv preprint arXiv: 1606.0636. 2016.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4195-4203 October 2023.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06/25, 1(2):3, 2022.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE transactions on paltern anal- ysis and machine intelligence, 45(4))4713-4726, 2022.
Subham Sekhar Sahoo, Marianne Arriola, Aaron Gokaslan, Edgar Mariano Marroquin, Alexan- der M Rush, Yair Schiff, Justin T Chiu, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. In The Tiiht Annual Conference on Neural Information Pro- cessing Systems, 2024.
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplitied and general- ized masked diffusion for discrete data. Advances in Neural Information Processing Systems, 37: 103131-10311 2024.
Jascha Sohl-Dickstein. Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei (eds.), Proceed- ings of the 32nd International Conference on Machine Learning. volume of Proceedings of Machine Learning Research, pp. 2256-2266 Lille, France, 2015. PMLR.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint wrXiv:2010.025500 2020a.
Yang Song. Sahaj Garg. Jiaxin Shi, and Stefano Ermon. Sliced score matching: scalable approach to density and score estimation. In Ryan P. Adams and Vibhav Gogate (eds.), Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, volume 115 of Proceedings of Machine Learning Research, pp. 574-584. PMLR, 2020b.
Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score- based diffusion models. M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 1428-1428. Curran Associates, Inc., 202
Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in neural information processing systems, 34:1415- 1428, 2021b.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2021c
Robin Strudel, Corentin Tallec, Florent Altché, Yilun Du, Yaroslav Ganin, Arthur Mensch, Will Grathwohl, Nikolay Savinov, Sander Dieleman, Laurent Sifre, al. Self-conditioned embedding diffusion for text generation. arXiv preprint Xxi::22/1.042.. 2022.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
13

---

Published as a conference paper at ICLR 2025
Y.M. Suhov and M. Kelbert. Probability and Statistics by Example: Volume 2, Markov Chains: Primer in Random Processes and Their Applications. Probability and Statistics by Example. Cambridge University Press, 2008. ISBN 9780521847674.
Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based continuous-time discrete diffusion models. In The Eleventh International Conference on Learning Representa- tions, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming- Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling, arXiv preprini arXiv.2409.02908, 2024.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A.I
KL Divergence Theorems
16
A.2
Cross Entropy Discrete Diftusion (CHDI)
A.2.1
Rederiving the Denosing Score Entropy Loss
21
A.2.2
Deriving Cross-Entropy from Ratio Matching
A.2.3
Relation to Cross Entropy in Continuous Diftusion
23
A.3
Roulette Discrete Diffusion
24
A.3.1
Deriving the exponential matrix of the absorb diffusion
24
A.3.2
Deriving the exponential of the uniform diffusion
24
A.3.3
Deriving the exponential of the roulette diffusion
25
A.3.4
Controlling the number of the corrected tokens in the reverse process
26
A.3.5
Time Evolving Roulette
27
B.
Algorithms
28
B.2
Network Architecture and Hy Seyppe-pprammtsst
28
B.
Efficient Implementation in Practice
28
B.3.1
Efficient Estimation of the Score in Practice
28
B.3.2
SEDD Scaling
30
B.3.3
Score Estimation Through CEDD Practice
................................
32
B.3,4
Analytic Sampling in Practice
33
Procedure generating the plots Figure
34
B.5
Per sequence estimation of J1
35
B.6
Additional results
35
Generated text
38
D.1
Discrete- Time Markov Chains Over Finite-State Spaces
50
D.2
Continuous-Time Markov Chains Over Finite-State Spaces
50
15

---

Published as a conference paper at ICLR 2025
A THEORETICAL RESULTS
A.1 KL DIVERGENCE THEOREMS
Lemma 1. Let oi and &i denote two initial distributions and define two Continuous- Markov Chains (CTMCs) running from time to O, with transition-rate matrices Qi (y.2) Q1 y)((z, y) and Q,(y.z) Qr(z, (0(x,))((1; y). where r(z,3) y) and f(z, y) chosen such that they ensure Qi and Q: are indeed transition-rate matrices. When the first process (with matrix Qu x)) is applied to initial distribution oi and the second (with matrix Qily, x)) to initial dis- tribution qı. they define distributions g6 and qr at time t, and qu- qu at the end of the diffusion, for which
DK lÃo)
$ $ Bay-an E v) v), f(5,, v)) di + D l(
(20)
y
where E(a, b) = (b a log b + K(a)) and ((a) = a(log a 1).
Proof. From the data processing inequality applied to the reverse of the processes defined in the Lemma, we have D (6)) S (xz(z"llã) where q² is the measure generated by the first process, while is the measure over the space of paths generated by second one. Using the expression for the KL divergence between path measures of two CTMCs found in (Opper & Sanguinetti] 2007| Section 2.1), and by substituting (x) := g" (zt), g(x((x) Qi (y, xx) and [(x (:)= = Qu(y, 21), we have:
the space of paths
Je Exrnet Σ
(T log (v, + Qe(. (y dt (.().z))
DKl(g"110) (1@)
+Dxi(gillá).
(21)
Now we focus on the expression inside the sum:
Q1 log x1) (((,,t))
+ (y) Qi( =
(22)
Qe(T v)r(x, v) log Qi ()r v) + (2(a,,))) v)
(23)
Do(xe,)) v)
(me v)r(*,,)) (log (logr(#..V) - log f(x,, w)) + Q1 v) v) Q, ()r (24)
Qrl v)r(m,,)) v) (log (logr(z;,1) - Q1 () 3) fogf(a Qi w)(( (25)
Qc(x v) [K (r( v)) + (f(z,. v) r(z, v) log f(x, m))] Qe v) (r(x), v), ((z),))))) 9))
(26)
which concludes the proof. We highlight that the CTMCs defined in the Lemma completely arbi- trary and not necessarily related to one-another, as the choices of r(x. y) and f(x, y) can completely overwrite the matrix Q (2., y)-
Theorem 1. Define CTMC with transition matrix Q2 that runs from time 0 to 1. The true reverse process defines a probability evolution from pr to the data distribution Po. while the learned re- verse process induces the evolution pt from the reference distribution P = p to the approximation of the data distribution ph. In this setting, the following KL divergence bound holds
DRI(polleG) S [ Excrpi Qe(
Po(y) pr (x,)
so( f)y dt DRI(Pille.). (27)
p((z) and f( v) = =0((xe f)y This implies that of = P and é. = p In particular, q = Po. $o = o and 9f = Pl. âi = py = pr- Plugging everything in Expression (20) gives the desired result.
Proof. In Lemma we set r(z, y) = p((y)
16

---

Published as a conference paper at ICLR 2025
Theorem Let pt denote the learned distribution from which the reverse process samples. The negative log or-probabiility of a state Do being sampled by the reverse process can be bounded from above follows,
log S S." Exeeprio 2o)
Eqi wie (
(Poolulzo) Pejo(zy(20)
80 s0(2r, Du di
+D +Dka(Pioo((zo)lll>>>
(28)
(m (mo) and f(z y) s0 t)y, where Pejo the prob- ability over states at time determined by a CTMC with transition-rate matrix Q, applied to initial distribution 8(x zo). As such, have of Prjo and & p.. In particular, q 6(x zo), 40 p and qi = Piju Q1 = pl = pr. Plugging everything in Expression (20):
Proof. In Lemma we r(z, 3)
DKL 2o) S / Eor~pio((
[ ( (Pijo(s/ao)
(p)o(zz/zo)) )
De(e, v)
(eo(y) (To) + (s0(2e,)),
log so( t) dt -DKL(Ppo(ZT))llm))
Pejo (x (20)
This concludes the proof as DKL(ō(æ = =xo)llo) - log(p)(zo))
Theorem 2, Denote the intermediate distributions time determined by the true reverse process, and by the learned reverse process with pr and pt, respectively. We can write the entropy of the data distribution H (po) as
Po(y) dt.
(po) -H(p)) - Exerpa (e((E.))) v)
(29)
pr(
pl) and P1 = pf == Pr. then the inequality in (xx)
In addition, if the learned ratios so(x t)y equal Theorem becomes an equality.
Proof. The cross entropy between the true data distribution, and the modeled data distribution sat- isfies the following:
H Hpo, pû) - H(p,, vi) = 1,²² ot
H (p..r
(30)
To keep things clear we focus on o9 H1 (pt, (pt. p().
θ Øt f (r(.)) log( (xxe (xx))dx
/
② Øt (p(()r)) log(p? (x:)))
H(r,,())
(31)
Øt
(- (pp( log(p! (x:)) + Øt
Pe(xe) Opt(2n) dar)..
(32)
(x)) Ot
We can use the Kolmogorov forward equations (Equation that is. E,, Qul
(x.)
de
and ppf (we) d1
E,, Q1 v)r (v) to get
/ Σ (z (v) log(pi (x1)) + f pr(x,) p (x)e Σ Qelen u)p IQ(lewWD
=
(33)
y
Qi w) pe(v) Σ
p.(())
logle( (1)))(()d)) pi(we) Eq((o v)
dac
(34)
pr(
pi px(e))
71
log(o + p((1) (x()
pe(y)
-EEro (((())))
(35)
pp (x1)
17

---

Published as a conference paper at ICLR 2025
pl(
pe(u)
log pl (1)
= -mnn pp (((r)
(36)
pf (x1)
Pr (31e)
pt (xx)
Po(y)
+ ppp E Qe( v)
log pl (v)l.
(37)
Pt (x(
The second term above Ex,,p. En Q1(1s,)) w)
Pe(y) pe(e.)
log p8 (y) is zero. Indeed,
Pc(y)
Pc(v)
Celoe, 1){
log(r) (1))) = Σ
Σ
Q (Te. ())r pe(ar)
log(e!
(38)
(1(t)
y
Qu(u w) po(v) log(p? (v)) 0,
EΣ (0 2t V
y)p(())) log((
Σ
(39)
since for any column of a transition-rate matrix the sum of that columns elements is 0, therefore Σx Qi(xe, v) = O. In total we have:
H (p1, pf) = S -Eurp Ee(())))) H(Po, po)
(1(()) pely) log pf(v) dt (40)
p.
po(x)) pl px()))
(P((v) pe(u) (xx)
log pl(x)
Exgo. Qe v)
dt.
(41)
(r( pl
(xxe)
This allows the derivation of the expression for entropy of the data distribution, by taking pf pt
H(po) = H(pi) + L/ Emi=p. Ev((nn))
Pe(y)
pe(v) (r(2))
Pc (v(()
log
dt,
(42)
(x(t))
pe(
thus
H(vo) H(v)) - 1 Beiupi Ee
Pe(y)
dl.
(43)
pe(xxe)
9
If we assume (2( ((.))
so(Ie, t)y and = pp == pb. from Equation
411 we get
H (po. PG) - H(p)) = / Emjop Σ 2e(eee v) (s((#e))))
(z(y)
log(so t))) de (44)
Pe (2x)
y
and furthermore, from Equation (43) it is trivial that
{ Razyoup Eqi(te)))
(P(lu) dt, (p(()))
- -H(p) H(p))
(45)
thus adding this last equation and Equation
44
we get
(46)
= =/00 Earop Ee(eee v) K (P(()) ) (p(xx.)
pe(v)
s0 (o((re, f),
log sa ()y
dl.
(47)
(x(e))
Since we know that ((tt) = 1, and sa( ()), pa(zz]]
(aas) ((s))
1, we have
Pe (p((Fe)
(e(s))) log so sa t)
)
K
+ (o((Te))) Pe (xx)
= 11+1-
(48)
(p(e)))
so we can change the sums E,, to simply E Eytar as follows
DKL (polle8)
(49)
Barom ((( [ pr(w)) ) vdkze
Pe(y) log so (Zt, 12) (x(
+ sol 0) (((er))
dl.
(50)
Finally, since Pı = pl = Ph. then D D = O, therefore we can add to the right side, finishing the proof.
18

---

Published as a conference paper at ICLR 2025
We remark that as s0 t)y - pe(2))
pe(v) then Pr (i) + Pe()), and thus
PL(1 (=))
pe(a)
as such
pr(z))
80 ( f)y
(u) Therefore, the bound in Theorem becomes tighter as the model improves. (2.)
Theorem Under the conditions stated in Theorem
7 the following inequality for the cross entropy
between the data and the learned distribution holds:
S. Eniope Σ
Pi(u) (p((
H pb)
Qu(Ee v)
so so((e, dt
y7.0
E
2. x4)t H(1.,Pr).
(51)
where E(a, b) (b a log b).
Proof. From Theorrm 1 we have
DKr(Polle0) DRI(Pillee)
$ Boonn E Qr(
(
pr(y)) (p((are)
+ (so(21,000 pe(y) pe(
log s0 (Te, O2) )
dt. (52)
Since know that pe(it) ((ae)
can manually set s0( t).r 1. and we get
Pe(xzs) ((())))
px((xt) (x()
K
so((),))
log so t)),
--111
(53)
therefore
H (8) 11 (10) -DL((pllp))
Ear En v) K Pr(()) Pe(xx))
(5o(
pe(y) (x(e))
log 80 s0 ()g ) dl (54)
from which we deduce,
H Hpo: 1o)
< (po) + /'' Exeap Σ Qe(Tes))
(P(())
dt
Pr (pt(2re)
+ * Exrope Σ Ve( v) so((eit) pe(y) log so (0( dt + DKI(Pille))
(55)
(x():))
Since from Theorem
2 we have that
H + [ Eniope
Pe(y)
Σ ()( y
dt H(p)),
(56)
pr(xx))
hence we can write
H (po. (8)
Exerpp cppe (((2n))) (solee,)),, t),
(c()) px(xx)
log so so 1))) dt + H(p)) + DKi(pill.)),
"
and therefore
E Qi(se,)) (so(ece1)) Pe(y)
H(po.po 5 Enyppe
log so 1),
pt pr(xx)
yfu.
+ S. Exrop Qe(z,, + H H(I.Pr)).
(57)
The fact that Q1(,,, Eyze, (1) concludes the proof.
19

---

Published as a conference paper at ICLR 2025
Proposition 6 In the case of the roulette diffusion with roulette-loglinear noise, H(pr) 0 and L (c-1). For the absorb diffusion, that is when prn
-lo Enaupi Eysu, Qily.ze)de = (1 -1-
1-mm (1-1-r))
Pm + 1, we have - ExIrpe Ensee Qr(y.v,)dt = L(E 1). Finally, for the uniform diffusion, (pr)) = Log and - So Exiup Eyter Qr di = - (1 - +) L L a (t)dt.
Proof. In all cases,
- -fo Paaypp E Orlyy = P Nsee
Σ
Qook (y'' xx)
(58)
= -OB
E pge
Σ
(y' an))
(59)
i=1
y'tni
--ENOU(0.1)
Σ
((tt))
Σ
Qtakk Q xi)
(60)
1=1
si
y V'
Po
Pejo (=10(2/12z)))
Σ
(61)
y/ vi
For the roulette case, if æ; is the masked token, that is, x/ = then Eni unx Q (x)) = 0.
Otherwise, Ey'oo fo: Qt0a xt) = of (1- - (1- Thus,
1(- Exiw Σ x))
(62)
yfæ.
Pm
EI,U((O))
po
Σ
(63)
m
1 - Pm
E,,U(,,1)
po
Σ Poju(a)((e)))
(64)
n
mi
mitan
Pm
E
po(2))oo (1 - p( = 22/2 nl(xb1)
(65)
n - 1
1=1
- Pm
((,1(0.1)
Σ
Σ
po po(ot)ooe
Fepm
)
1 Pm L jo
oie
a.pm dt
n
i=1
(66)
-11-1-1 Pm
1-
L (E -0PPm - -00Pm)
(67)
11
Pm
Since or is roulette-loglinear, that is at Proposition.
1 log (1 - (1 - e)),, we get the result stated in the
In the Uniform case, we notice that Equation 61
can be rewritten as
L
E,,11(011)
ΣΣ>
p( 2)) 1000a(a (xf,
(68)
1=1
x
L
x ) (t) (1-1) -
-L (1 (1- 11 S. o' (((t)tt.
E0,)
Σ
p {
(69)
20
2
20

---

Published as a conference paper at ICLR 2025
A.2 CROSS ENTROPY DISCRETE DIFFUSION (CEDD)
A.2.1 REDERIVING THE DENOSING SCORE ENTROPY LOSS
In what follows, we rederive the Denosing Score Entropy Loss for completeness (Lou et al. 2024). Training a model using ratio matching is performed by minimizing the error of network that re- ceives as input 2t and tries to predict the ratios of pe(y) where y is a neighbour of æ with Hamming pr())))
distance of 1. In other words, one tries to minimize
(v) Pr (xx) so (z), tly)
(70)
for all x2 following distribution pr (x.e), that is:
p((y) (x())))
E (m l
sa( tly
(71)
The main bottleneck that the ratios p=(M) p=() are unknown, as pe( = S (2xx a ) (2 daoo
and (Do) is what trying to model in the first place. Luckily, the denoising trick can be employed:
Pe(y) pe(x)))
Ee Eer ( po( so sa t) pr(x)))
so(02,, tly)
E (x.))
(72)
Euo Pelo(y azo)
Pejo (2o (o0((
E Be
,80 (2x, f)g
(73)
Prjo (2 (20)
Pe(x))
Exs Pejo Pejo(zy (2000) 22o)
Poje (zu (x)), ) so (xxe t)y
(74)
Peja (2t (00)
Peoo ( an)
(m
Prjo ( (x,) se (zi, t)y
(75)
When ~ linear with respect to ratios, like in the case of Score Entropy (Lou al. 2024), then we pull the inner expectation (sum) outside and have
(P((y)
(3) Pajo (3/ x0) Pejo (2) (x0))
(
so (Iz. t)y) mpe Eno Po (x,)
so (x f)y
(76)
which of course is equal to
Pajo(1y xo)) Prio (aco)
so(xx,, 12g) 0w)
(77)
A.2.2 DERIVING
FROM RATIO MATCHING
In order to go from Equation to 74h we used the fact that
Pr (v)
Prooy (o))
Exp
Pejo (x, (x)) Poje Pojo (2))
(78)
(x))
Now, we select position i on the sequence with length L. We want to find an expression of the ratios Pi(y) for all sequences y, which differ with x, only position i. From above we have Px(m,)
pr pr(y) Empo Prjo(y)200) (x()) prjo(=; Poje Pojo (2 (x:):
(79)
where
Pejo(y(az))
Pejo ())))),,))) (1,,..,,y (V--)) (20p (0) 2:0 (1) ..., 2p (6) ...., 20 (V (V-1))
Prjo (x)))))
Pejo ((x, (0), x (1), xe ( xz (V-1)) [2 (0), 20 (1) x (6) 23 (V (V-1)))
(80)
21

---

Published as a conference paper at ICLR 2025
and due to independence between entries in the forward process have
Pr Prio ( (0) Pojol at mo)
II PHjo (xo)))))
(1 (x0")
Pejo x
II
(81)
II Pejo (x:) (mo²)
Pt (are' (m)))
Prjo (2z," Immo')
as the rest of these ratios are 1, since y differs with a, only on position i. Therefore we have
pr(y) Em (y (ao') (oo(y'
(a 1x Poje (0)e((ool24)))
(82)
Pe (xxe)
where Poje (2) xxr) = Puje (wo (x=(() æ0 (2)
vo (V-1)) (x)) and where the expression inside Pejo (mo")* depends only on æo and not on the rest of zo for j # i. Thus,
(0),
2o
(1)
the sum Pajo(y' (ee'))
Paio Py [
Pejo(ys' zo')
Dao
(=o") Poja P = Mino
Me "0) (0(111) ao (1 ( Pale (2o xx,),
Pijol (ax (az
Ps Piju (2 (xx0))
(83)
which implies that
Pe(y) Pr (2xe)
Pao (2o')) prjo ar a Poje (200
Exn
(84)
or more clearly
Pc EnE(V] Pejo (h) Pajo 1h) Poje (hlaz)).
(85)
(x(x))
If we learn the V probabilities (P (0/zz)), (ooe))... ((V - 1)]a))) since we analytically (p") have P1o( (from the matrix exponential), then we we will have the ratio p.l)) We can
choose another sequence which also differs only position i from æt Then we will have
pe(z) EnEIV Pe)(="(h) P Poje (hla)))
(86)
Peoo (x, (h) which highlights the fact that even though the ratios might change, the same (oje(0)zz)) Poje (1)..)......PP P(i....V ((V 1)]=)]] appear as long as the selected position in the sequence remains unchanged. Therefore if we learn these probabilities for position, we will have the ratios of all neighbours that differ only in that position, that is, we have V ratios, by modeling these V probabilities.
(x(x))
There is nothing special about position and we can choose to model V probabilities for each positions. In order do so, define neural network fa(æ,, whose output L x V. where entry hi,h) of the output (that is fz (7])) predicts p (A/zz.) the probability of the entry i of xt (position of the current sequence). having being perturbed from token with id h in the original sequence Tu, given that we are at sequence x, right now. That is, the model directly trying to predict from where each of the tokens in the current sequence came from. Therefore we will reparametrize our ratio model as follows
(= (h) Puj(ara" ' [h) fa(v,, ()[4].
(a,,
(a,, [) = sa 1)), Σne(v]
(87)
where (a t) are the outputs of the softmax at the end, and where for each i, distribution fz should match Poj (12z;). This happens when DE (Pore (-(x,) t)), 0, thus we minimize,
L
Σ (o..rro.)uu(EEeer..... DR (+1=:))) (at))
(88)
Since D (Pie (-a:a(-(=::))11) (-I=x))) t)) (Pi (+xx:) fa t)) C Epg (h (m,) log fa 1) the loss function above has the same gradients with regards to network parameters as
C
Bn (b/)s) log (2 ()
(89)
22

---

Published as a conference paper at ICLR 2025
which is equal to
L
E Eu (o. (mx poo (o()) w(() log (a t) [x]l]
(90)
Thus, as in (Campbell et al. 2022). the loss we utilize in our training is Lu.
Lu ---E,,- EL Reo Ex, Pejo (one (m0)
(t) log (ar,, (z)])-
(91)
We can also motivate this loss from the score entropy in Equation 77 Indeed, since & and / differ by a constant, its gradients are the same as those of
Pt a (0)) (r Ene(V]
(eo(y'(h) (r fo for,, .))h)
Exzoraa (mn) Eei-pejo (ma))
(92)
and as before since y and differ at only one token we get
Pejo(yy (xo')
Pejo((r (t)
EEe (oo(o)) Ee (mml
Pr ( /mzo") Sne(v)
Pejo " 5 (a 1)]4]]
(93)
The function E(a, b) is clearly minimized when a = b, in which case ((a, b) = 0. our case,
Pojo (moo') (1)(o")))))
Prjo(y (h) fá (2x, t)(bk) = 0,
Pejo (zx x0o') Ene(V]
(94)
Pejo (xt 1/)
for
Pejo (yy (x00))
( (h)
Pojo xx," |xoa') = [RE(V]
oe (r, (h) f0 (z,, [A],
(95)
which happens when fi 8) 1h * ma'' = and (xx, 0) [xo']] = 1, that is when our probability prediction matches the one hot encoding used in cross entropy. Thus we train our model with cross entropy, which simplifies the job the network of learning complex ratios, as the conditional ratios now do not participate in the loss. We only add them during sampling to the weighted sum in order to predict the marginal ratios, since we have analytic expressions for the conditional ratios. Therefore, train the model by minimizing
(o,1 Bao -po(as) Ex,, Paja (xx (mx///o)) (mo))
wct) log fe(z,, [)]=])
(96)
A.2.3 RELATION CROSS ENTROPY IN CONTINUOUS DIFFUSION
Cross entropy has also been used in continuous diffusion models applied to Language Modelling (Dieleman et al. 2022). Here, we draw parallels between the two approaches. We denote sequence of tokens at time as x, = (.., ). where each token is embedded in D dimensional space at E DR That is, each sequence can be seen an I, x D vector, created from concatenating the embeddings of each token. The score that generates the reverse process, is therefore L x D vector so sI f), which approximates:
VR. log px = / Va: log Pt
dxoo
(97)
V ae loge
-1 207
Pe ( =
xo xe
pr (woolo )do0 =
pr
2o /
- pr (wo (20 )doo
of
o7
(98)
From above, is clear that
x (wo )
x
x of
(f Dullaolaeesdder
x
Vaz log::((:))
da
(99)
77
of
where o5 denotes all entries of ao without the ones of an. Therefore
sa (Zt, ().......... R x (x;) =
x
P. (x day
x of
(100)
o?
Clearly, all that need to be learned in order model the score for the dimensions of the token at position are the V probabilities p. (2//(x,) For L such tokens (to model the score of the entire sequence) one simply needs to model I. x probabilities using cross entropy in the discrete case.
23

---

Published as a conference paper at ICLR 2025
A.3 ROULETTE DISCRETE DIFFUSION
The roulette transition-rate matrix is
(1 pmm)
--T pm)
n-1
(1
Pm
0
Pml
pmm)
Pm
0
Prouletie
1.1
(1 pml
n-1 - -Pm)
Pm))
0
Pm
Pm
Pm
1-1
nxn
(101)
where n 1 V is the number of tokens in our vocabulary. We add special token for the absorb (mask) state, therefore increasing the number of total states to nt with the token id a corresponding to the absorbed state. Unfortunately, and uniform diffusion, since the corresponding matrix P of Q is not idempotent. Therefore, we will first derive the exponential matrix of the uniform and absorb diffusion manually, in order motivate the manual derivation of the exponential matrix in the roulette case.
cannot derive the matrix exponential in the case of absorb
A.3.1 DERIVING THE EXPONENTIAL MATRIX OF THE ABSORB DIFFUSION
The transition matrix in the absorb case is
1
0
0
0
0
abs
...
...
(102)
0
0 0
1
1
Clearly this is Q'oklette when Pm = 1.
If we are at state the probability of moving at state n (the absorb/mask state) for time-step of size is We discretize the time interval into discrete steps that are multiples of e, that is [0, fc..... [c] and denote the event of jumping from i o n to time je as ag. The intersection of any of these two events is empty and their union is the space of all possibilities, thus probability of being at n at time T is
p(#, n) p((=, = n) n2) =) n (allaa))))p(((sstt)) p(Ujecr (az})
(103)
p( =m) m) = = = =<((p(())=+(((((0-+++-- + c)² + + t1 - ())
(104)
(1- = (1 1)
p(æ, = n) = 1- (1-))
(105)
Taking the limit e + 0, we get plz, n) = = Thus, assuming that we start at state i, we have Pjjo(j & (n, (n,i)li n) 0. P- pjo ili n) = and Prjo(j n/i + n) 1-e This defines the column i * of the exponential TQ Iterating though different entire exponential matrix, except its last column which is a simple one hot encoding at n. Setting ag finishes the derivation.
construct the
A.3.2 DERIVING THE EXPONENTIAL OF THE UNIFORM DIFFUSION
Q
Qunss
R-1
n-1
(106)
VXV
Clearly this Qtoksttte when Pm = 0, without the absorption row and column, as our vocabulary size is V = n since we do not have the special token. As before we discretize the time interval into discrete steps that are multiples of e, that [0, r] - je, [ We assume that initially (at time 0) we start position i. We wish to find the probability of being at state k + i at time LEje. For the sake simplicity, we abuse notation by denoting Pt1 (rl as pr-me (ili). By symmetry we know that for every j * the probability P. Fil(() (il)) is the same. Thus the probability of pr (ili) is the sum of the following components:
24

---

Indeed, given Published as a conference paper at ICLR 2025
probability of being at j & {i, k) time i.e (ji-e times - 3 V - such states, times probability of moving (e), times probability of hitting & on that move ( (-
probability of being at j i at time 7 i.e, (1 (n 2) (ile() times probability of moving (e), times probability of hitting on that move (
probability of being at j = k at time c, i.e, (jl(),, times of staying there (1 - c).
probability of being at j = k at time t c, i.e, (Fr (Fr-c(j(i)), times of moving (c), times of hitting itself on this move (x-T))
We write them down mathematically and get:
1 (m - 3) + (1 (n 2)p)- n-
pr (Mll)) = P-- (fli))
(107)
n
+ +p--c(110(11 - c) + Dr-c(j1i) 1
(108)
thus
(k(li) = n n- c + P. (1b0 -c).
(109)
Since by definition k # i and j # then by symmetry have P.r (klb) p. (.(i). Therefore get the recursion:
1
pl=)c((((() n2 é P(1))c-c)1((()))
(110)
which if we fully develop becomes:
(k = n (1) +... + (1-
(111)
which is equal to:
1 1 (1 c| 1
P. (k
(1 (1- (1-
(112)
1 (1
71
Taking the limit - O gives Po k * i|i) = n- (1 e and Prio( = 1 -1- which gives the ith column of the exponential matrix. Setting 7 or finishes the derivation.
A.3.3 DERIVING THE EXPONENTIAL OF THE ROULETTE DIFFUSION
First, we notice that we must start at a state that is different from the absorb state (there are no masked tokens in the training data). We can split the states into two groups, the non-absorbing states, and the absorbing state. We consider the non-absorbing states as a single super-state and the absorbing state the other super-state. From derivations in Appendix A.3.I we know how to derive the probability of being at the absorb super-state at time т. The only difference is that previously, we moved, it would be certain we would move to the absorption super-state, while in our case, we can move to a different state in our non-absorb super-state. The possibility of moving to the absorb super-state in e time step, changes from to cpm, and the formula becomes
p(x, = n) = -}<zp(aa) = cPm + Fpm (1 - epm) +epm 11-mm111ee (1-Epm)² ) +.............. tepm - -pp--pmmmLL-- (113)
(1 ppm) PP 17 Jepm
plar = =n) = =pm 1 (1 (Pm)
= (1 6pm Tpm 1epm
(114)
1- which implies that p(a,- = n) = - e TPm. This means that at time we are in one of the non- absorption states with a probability of eTPm. Now, as before (Appendix A.3.2). we want to find the probability of being at each non-absorption state at time given that we stated a state i. We can construct these probabilities using a similar approach as before, but with two key differences. First, because we are conditioning on being in the non-absorption super-state, all probabilities must be multiplied by e em7p". Second, this conditioning implicitly provides some information about the probability of moving when going from time e this probability was e, but now conditioning on the fact that we are not at the absorption state at time 7 provides extra information on whether moved when going from
to time In the uniform case, for time step
25

---

Published as a conference paper at ICLR 2025
this extra information, we expect the probability of having moved when going form T to be reduced. This becomes obvious when pm = 1. as then we are in the case of the absorb diffusion, and saying that at time we not at the absorption state, immediately implies that we did not move, as there are no other possibilites. As pm decreases, the magnitude of this information decreases. So instead of writing the probability of moving (during time step) with as we did previously, write this probability with o(t), and as before we can derive that
1 Pr (kli) (i-e() n (n 3) + (1 (n - 2)p
1 n2 1
(115)
1
(fl- - 6) + - n1 1
(116)
thus
Pr (kli)
+ Dr (A(b) 5)
(117)
and by the same recursion trick as before,
8 1 1 61-8)
1 (1 - (1 -1- 8)(..). (1-))()).
(118)
1-
Now we derive the expression of 8. We write the move event when going from T with A and not going at the absorbed state at time with B
p(A) B( P (B]A)
((BA))((A)
= p(AB))
(119)
P (BBA))p(A) + ((B AC ((A))
(1 (1-Pmi Pm
(1 pm)e 1(1 c)) ar
(120)
where a (1 11-Pmmett (I-Pm)e+1(1-e) goes to 1 - Pm when goes to O. therefore finally
(1-pm)
PkE)c(o(((i) (1 -(1 (1-5)(
n (1 -6) n (1 - (1 5) dae
1
(121)
Therefore Prio(j & fł, k1) = pm 1 (1 cll-p.)rr, and (=)o( Prjo(ili) = Pm trix, when i * In case = n, the column of the exponential matrix is simply the one hot encoding at n. Setting T = or finishes the derivation.
e 7Pm, and
n-1 2=2 (1- e()-Bm)r)), which gives the ith column of the exponential ma-
Proposition 5 If we denote with Y, the matrix exponential of or o.Qtoutetee then Yi & [i,n). + n) = aapm --1 - 11--1pppp-- (1--p))))) Yi + n, # n) = e o:pm (1 -e- (1)))-))))), ((nj+m)) = -- ampm Yi(i + n) = O. and Ye = L.
oz (I plok )
A.3.4 CONTROLLING THE NUMBER OF THE CORRECTED TOKENS IN THE REVERSE PROCESS
We wish to how the choice of pm corresponds the probability of token moving uniformly at least once before being masked, given some diffusion interval from [0, TJ. This should correspond to the probability of token being corrected after it is unmasked in the reverse process, therefore enabling us to control the expected number of tokens to be corrected.
As before. we discretize the time interval into subintervals of length The event of a token moving before getting masked will be denoted by X, and the event of token being masked at time jc is denoted with Ay. Therefore
p(X) = p(x (2) = p(X n (UA,)) = P(USEEX X AA))
(122)
p(X) -ソ<1 (|XAA(())(4)) (AA,).
(123)
p(Ar) is the probability of being at the absorbtion state at je and not being there at (j - L)e. This probability is p(A,) = (1 --- e0mmm - (1 (1-e-(i-1)-- 1)epm)) e (j- l)epm e jepm therefore p(A,)
e 11-1pm (1 ppm) e-- -(i-1) 1) Epm. We write T ie, and thus we have
2 (1 - (1-Pm)5- -1- (124)
x(X) = 2)x(3)((X14))((A E5 1
26

---

Published as a conference paper at ICLR 2025
which is Riemann sum. Taking the limit we get the following integral
p(X) Pm /-/ (1 -- (1--))))) =2 dx,
(125)
whose solution is Setting & 07, we get
n2 2 (E (e-T
1
+ 1 1).
p(X) Pm
Tpm
(126)
Pm
Pm
pxx) = (e
'mm
Pm 11 m + 1 -Pm)
(127)
We can see that when σ1 and Pm are relatively large then p(X) R 1 - Pm- That is, the ratio of tokens moving before absorption is p(X) 1 Pm: in other words the probability of tokens being masked without ever moving is Pm which is precisely the probability of a token being masked in our tramsition-rate matrix Qtakk Thus the last row of our matrix directly controls the percentage of corrected tokens in the reverse process for large enough diffusion times.
A.3.5 TIME EVOLVING ROULETTE
Here, we generalize the results of the previous subsections for the case that pm varies with respect to time. We start by defining
Q (1) = (m (m)) Qtok: tol: uniform + [Pm (t) Q
tok absorb.
(128)
First, we point out that since the size of Qeroulette is (mxn) and that of Q Quiiform is (VxV), where V = n--1, 1, in order for them to have the same size, we add row of zeros and a column of zeros to the bottom and right of Q uuifform respectively. Clearly, by definition, these are transition rate matrices, and therefore so is Qeohmlette (t). Indeed, the elements in each of its columns add to O, and the only negative elements are in the diagonals. If we define, Pmo (t) such that Pm (0) = and pm (m((1 1, can then the limiting distribution will be the one-hot encoding the absorb state. Furthermore, compute the exponential matrix of Qérbutette and pm (m) (t) Qtok absorb commute with each other, thus
is easy to prove that (1 Pm (t)Quekk fniform
Q
Lokk Geraletea (1) (1 (Pm (t)) P (1) (aaars,,
(129)
=m)((t) (omlt)ooQatte... Q (1-Pm. (f))or Q uniform,
tok.
(130)
Writing at = Pmi(o and Br = -1--P0 Pm (t)) shows that we can calculate each of these exponen- tial matrices using the strategy below
(Qloky *(()*
(-1) eel)) Quok
I (1 ee000) (131)
k!
Indeed,
Q.Q absorb = I +
Q tok e ea) and & q = I + Quni form (1 -e -
(132)
Multiplying them together, we get the following proposition:
Proposition 7. If we denote with Y, the matrix exponential of of ttok eroulette then Yi ( & (j,m),j +
n) C dePam - (1 e- -1 Pm (f))a.), Yi(i + n, + n) e - (1 n=2
e (1 (1--mm())am (n, * n) =
aepm (t), Yi(i + n, n) = and Y. n) n), = 1.
One can that this is almost identical Proposition 5 with the only difference being that Pm varies with f. Indeed, we fix Pm so that is constant with respect to time, Equation (28) becomes
Ceroulette (1) = Pm Q tok absort (1) + (1 - (1-Pm))e uniform (1) fok foottette (I),
tok
tok
(133)
tok meaning that Q ergulette roulette diffusion an interpolation between the roulette and uniform diffusion.
(t), coincides with foutette (t) roulette in this case. This highlights that the
One possible choice of pm (1) is a for positive constant
27

---

Published as a conference paper at ICLR 2025
B EXPERIMENTAL DETAILS
B.1 ALGORITHMS
Algorithm Cross Entropy Training Algorithm
Require: Network fe. (total) noise schedule or, data distribution Pdata- token transition matrix Qtok; and time / E [0, 1].
Sample 2g Po. Construct 2; from Vo. In particular, æt (-((()))))) exp(o, [1:20))-
u((0, 1)).
Compute Lu = - - log ())=,))((z))
Backpropagate VoLu.
Run optimizer.
B.2 NETWORK ARCHITECTURE AND HYPER-PARAMETERS
This core model grounded the diffusion transformer architecture introduced by Peebles Xie (023), which integrates time conditioning into the conventional encoder-only transformer frame- work as established by Vaswani et al. (2017): Devlin et al. (2019). However, it includes minor adjustments, such as the use of rotary positional encoding (Su et al. 2024). The model contains approximately 5% more parameters than a standard transformer (utilized in the case GPT-2), at- tributed the incorporation of time conditioning. Additionally, the same tokenizers and data splits as in prior work are employed to avoid introducing artifacts.
The network configured with 12 transformer blocks, each featuring 12 attention heads and hidden size of 768, aligning with the "small" variant of GPT-2. It includes conditioning dimensions set 128 to facilitate the diffusion process by encoding dime-dependent features. Notably, the architecture excludes masking, typical of generative models that generate all tokens simultancously rather than sequentially. uses standard scaled dot- dot-product attention mechanisms and incorporates a dropout rate of 0.1 to mitigate overfitting.
In terms of hyperparameters, the model was trained on single H100 when the sequence length is set at 128, while in the case of sequence lengths of 1024 the model is trained using 8x with a vocabulary size 50,257 tokens. Training involves a batch size of 512. The training regime designed for a total of 400,000 iterations.
Training utilizes the WpenWetText dataset, while evaluation is conducted on WikiText- 103, with data managed locally to speed up access times. The noise schedule for the diffusion process is log- linear (uniform, absorb), and roulette log-linear (roulette) controlling the variance noise added incrementally. In both cases we set 0.001 as in (Lou al. 2024). Sampling for evaluation during training employs an Euler predictor over 128 (and 1024 when = 1024) steps, with noise removal enabled.
For optimization, the model uses the AdamW optimizer with learning rate 0.0003, beta param- eters of 0.9 and 0.999, and epsilon set to 1e-8. It features weight decay, focusing adapting learning without additional regularization. The optimizer includes a warm-up phase of 2,500 steps stabilize learning dynamics, and employs gradient clipping at threshold of prevent gradients from exploding during training. The log-linear noise schedule was used in the absorb and uniform case, while the roulette log-linear one was used in the roulette case.
B.3
EFFICIENT IMPLEMENTATION IN PRACTICE
B.3.1 EFFICIENT ESTIMATION OF THE SCORE IN PRACTICE
For our choice of sparse matrices Q which can only modify one position at each step, the loss function
(2o Po Qi(a, 8)
(2)0 80(222, 1)
(134)
yo
28

---

Published as a conference paper at ICLR 2025
becomes
Pejo(y' (z/) (2)(()))))))
E Be, pe of (ee)
Σ
Q (x y t
si (2x, (1wr)) (135)
Now we focus on calculating the loss at position i, as the total loss across the sequence, will be simply the sum of such individual losses at each position. We notice that the term Eyitet (2 y )s (2, t) [v") in the expression above is trivial to calculate. We simply add the exponentiated outputs of the neural network across the last dimension weighted by Qx) y²), and then substract so( t) -t xf). Since in all cases the terms of Q are either mostly or mostly the same, this can be done efficiently. The expression
Pejo(3 (z+) Prjo Prjo(ar) (ar (2))
ot (o) v²) V'A#
Σ
log s) ()y)
(136)
is more challenging to be calculated efficiently. We follow the approach of (Lou al. 2024), when using SEDD for training. To reiterate, for each position i, if we use SEDD for training, we
V Qtok = Q foim and n = V + 1 otherwise. Luckily, for the choices of Qtak in this paper (uniform, absorb and roulette), such ratios have relatively simple form and they are almost all the same. For example, in the case of the absorb diffusion, there are two cases, either æ has not moved (a: = xi)) or it has been masked (x) = n * xi). which can be seen in Figure
need to calculate the sum of the product Puj(y' (mg)
Prjo[=]]a]]) [=]] log (a,, t)(y"] over n ratios, where
x. + xf
H. x x
x = x
Figure 4: The conditional ratios at position i over the vocabulary in the two cases. The square represents the absorb state. The value of a is -e
To simplify notation, when iterating through different tokens in the vocabulary write arj k, y = j and log s t)ly"] Sy. In the first case, when æf = n a xáj we get
Q ra (at, yy (e (a)) Pejo (ej (x)))
Σ
log oi (a,, ()0)
8k.
(137)
y' pi
On the other hand, in the second case (x) = zn) one has
Σ Q tok (x ) P (z+) log st t)
(138)
8'
Qlok (zf j]os; + Q n) 1 Sn = 0,
(139)
Qfak(z) n) = 0. Thus when x 2/ # n, can simply not calculate the loss.
For the uniform diffusion, there are also two cases, namely xf = ag and x * z as illustrated in Figure
29

---

Published as a conference paper at ICLR 2025
1xx// / xb
f1. x = xb
o
Figure 5: The conditional ratios at position over the vocabulary in the two cases. We define b= n= (1 - eee ot) and c= 1 (n 1)b.
If att * ab. then
Pojo lan)
Σ ot tok (at v²
sa 0)(²²== Σ
Q (ar. + +Qi (2)((()))))) f (140)
Pojo 12 (x)))
Qtok (2),
Qt
Qlok (at R:) (6 -1- sk =
Σ
1
8j
+
1 //c_ -1)
1) sk.
(141)
n
n b
j
Otherwise for wº whe we have
o (ar) y²) Prio(y' Pojo( (x) mn))
2/ (a (30²) = b Σ Q j)s;
b
Σ V'
Σ 8j.
(142)
n jサm
( }
Finally writing St = E, S3. we get 1 + ( - 1) 9k + soj in the first case, and in the second one. For the roulette diffusion we proceed similarly. Figure 6 shows the ratios all three possible cases.
1
L x = M
x
HI. x = x
x x
Ii.. x a M and x a xi)
Figure 6: The conditional ratios at position over the vocabulary in the three cases. Since an is a token from the data it cannot be masked. The square represents the absorb state. We define
a e Pomo, b = e Pmmo. 1 -ell vym) and = e Pmmo + (1- - e (1-((mmm)))
In the first case, the sum at position i is Pm L (s'- - Sk Sw0 + - _sk] In the second it is 1-0mm c (St - s -x)) while in the third case we have 1-p0 (5' If one chooses to incorporate the constant term ((a). in the loss, then a similar approach is fol- lowed. However, considering Theorem 4 for evaluation, and the fact that constant term does not affect the gradient during training, implementing this term is not necessary.
- 8k 8,, s0 + ' pm sa S-
B.3.2 SEDD SCALING
During training, some scores (rio |=())
(ajo(=r [1)) expectation with respect to æ)j, can vary significantly for different y¹ This makes the task of the network more complex and reduces performance. In order to
30

---

Published as a conference paper at ICLR 2025
alleviate this issue, in the case of the absorb diffusion, (oou et al. 2024) add log Dn; 1 (rol'")
(=)((=)|m))
to the output of the network log si (It,:) ())"". In this case, since we only need to calculate scores when z = n, this sum equals
1 Prjo (a:6)
Proly' o mlar)) Prjo (n(z))
Peooy # Anly' fn)
Σ m
- 0+ n-1 Pejo (nll # n))
(143)
1 Pajo x z/))
(144)
T1
Y1 1 c
We notice that in effect, this changes the prediction of the score, from log t))))))))) to log si (2t t)(r') log(n 1) log (e" 1).
We extend this strategy for the uniform and the roulette diffusion. In the case of the uniform diffusion, we get
Pyoly axt)
1 (1
1 Prjo (y' y')
1 Pro(erla;)
Σ
(145)
V Pejo ( (x))
V Pijo (2/11)))
V ()j((=ia=))
m
tt mm/)
Now,
Prio (xol''
1- 1--vv' 1 (1-c-0)
V
1+
(146)
(11 e
e
and
Prjo (3) xx))
(1 c at)
V
(147)
jo ( ke))
K (1 0
t - 1 V
Prio (m)
Since
1. we conclude
Pajol =
Prjo 3 (z)))
V - 2
1 V
1
1
+ V et 14 1+V
1+
Pejo (x))=)b)
p 1
We then modify the output of the network log st (xx, t)ly')) accordingly
Poju(yy (a0)) ) where n :=
log s (2,, t)(y'] + log Σ
(148)
Similarly, we can calculate this scaling factor for the roulette diffusion. There are two cases, either at at o n. In the first case, with little modifications from before can show that
Σn
1 (y + nlzz)
(1 e-DePm)
1
1
(149)
zn n-1 (1)())
n -11 1 e Jepm
n 1 7ePm
In the second case,
1 Peo y # n(ab)
Σ
+
(150)
pap # nlzz))
el1 e(1-pm)oc -1 c(1-pm]or +
2
We then modify the output of the network log si (xx, ()rr') accordingly
Prjo (mz))
log ( [) [ + log
(151)
vu Prjo x (=))
mb
We take moment to recall that if y and differ only at one position i then
(v(
Proo (m6) p' (2(2(220))
Σ
(152)
(x
(ajo(e (w))
Thus, above we are scaling the output by naive estimation of this expectation where p (a/) (x)) is assumed to be ب In fact, the study of the scaling approach and the realization that the entity n-1 can be more properly estimated, is what originally motivated the derivation and usage of CEDD in this paper.
31

---

Published as a conference paper at ICLR 2025
B.3.3 SCORE ESTIMATION THROUGH CEDD PRACTICE
In this subsection, similarly to Section B.3.1 we show how to convert the learned probabilities into learned ratios in practice. Suppose that the perturbed token at position i at time é is æi. If y is sequence that differs from De only at position then from before one can write:
Pr( (x()))
Pojo (z
(m)))
(153)
Pojo (ajo(a (x))
26
We point out that since and only differ at position i, then ot # y¹ Thus, Pe(y) (((e)
Pojo(3' ax)) Pijo (x)())
+ (y//),)) Proly'ly') (n'
+ + (a/|<>) Pojoly" (x)))
Σ 2
(x xx)
Pejol (z) xx))
(154)
In the case of the absorb diffusion, from Section B.3.1 we can see that we do not need the ra- tio when y' = n, and furthermore, we only need the ratios when æ; = n. Since also the data is not masked, then z # n. Therefore, the first term (the sum) in Equation (154) is 0, since Pao(y' Mtnlau'etnaa (m.y')) O. Hence Pe (
Pr Proly' + n|n) (y) pt (y' (a.)) PHooly' # fn #n) p' (ac//aa)) (((x))
(155)
(njonnyy' + n)
(n)
where Pejoy' n|n) 0 since masked token cannot be unmasked in the forward process. Thus our approximation of the score is simply
(a,,, ()w (a Oly') --
(156)
To conclude s0(2), 1) (x 1) -::--
In the case of the uniform diffusion, we write N (1 e- and c as in Section B.3.I and Equation (154) becomes
(n
1)b
p((y)
Σ a (a).ar)))))
(abg(a))) + +p' (3)111))) + (=)i(x)))
(157)
pr(xx)
(y) (xx))
Σ (x) - (f + (x (--1).
(158)
Therefore, our approximation of the score is simply
((1])))l'']=11 (2 (6 -1) (|a;)) -1)
(159)
or
t) = fa (G. -1 fa(a t) [=;l]
b - 11) 1,
(160)
vector form.
Finally, in the case of roulette diffusion, we can discern two possibilities, namely o; = and
Pmo. Pm at Pmo. 1 1-e- pmm)pm)) pm)
of # In the first one, writing in Section B.3.1 and c = e Pm
n-1
n-1 + (1 - (1 Pmu Equation 154 becomes
1
Σ
b + rl (o
c
+' -
(161)
Pc (xxe)
Hence, if the perturbed token of position i is masked, the predicted ratios over the vocabulary are:
b $ $2 1) = - + Sa(ax,)) 1) ( -
(162)
32

---

Published as a conference paper at ICLR 2025
Very similarly, if the perturbed token æj at position i is not masked, the predicted ratios over the vocabulary are:
s)(, = 1+ far, (6-1) fal,, [) 11-1)
(163)
This shows that we can calculate the ratios by scaling and translating the learned probabilities through CEDD.
Modification of the reparametrization in 18 Re-scaling conditional ratios when / -> 0.
In the case of the uniform diffusion dynamics, when & 0 then - 0 and -> 00. This does not cause any issues when sampling, however, it does negatively impact the perplexity bound. We explain informally the issue below. We fix position and assume we are at æt and that originally were at z When the time t is close to 0, the model fa t)ly')
] is typically very confident and can predict that some token y is a/. that is, [ ()(y') R 1 -- and I is R V-I everywhere else. Furthermore, most of the time æj = v/j This means the score (z,,, l)lr 2 & and is R everywhere else. If the prediction is correct then the loss becomes ~ -ー (V -) log( + e c (log( ()) 1)] + --g log(() + _ (log(() 1)] O. it misses however, then we have 1)] + +[ lo (log( - 1)] [- log( + # E (log(f) 1)] x 2 log g. Considering the large magnitude of & when t z 0, this loss is extremely punitive. SEDD does not suffer from this issue, as the conditional ratios are implicitly learned by the network, which due its limited flexibility likely acts the values of the ratios to rise steeply as f + O. Motivated by this, in the uniform case, we rescale of < 0.0015, by setting a, 0.0015. This significantly reduces the magnitude of g as + 0. We highlight that only the o, that is used to calculate & and & are scaled while the or that is fed to the model is not touched. Naturally, for the sake of rigor, this is also the model generate samples. Finally, we note that this problem also appears in the case of the roulette diffusion dynamics, as when time is close to 0 most tokens unmasked. The same strategy is applied, as before, only to ratios and S. by rescaling when og < 0.5 as follows: ofcueed log(1.1o, 1.1).
V (V - 3) log( + + (log())
regularizer, not allowing
employ to
B.3.4 ANALYTIC SAMPLING IN PRACTICE
In (Lou et al. 2024) alternative sampling scheme (Equation (18)) is provided. This method is called the analytic method and it performs better than Euler sampling. in particular when the number of sampling steps is small:
Pi-c ( ( = afttt (x))2(1)))
Σ ( af (x) (x) v sa 1)lr", (164)
y²=1
where of = - 0z-r-
Since we have an analytic expression of the matrix exponential e q'" we can easily derive the expression e f by simply substituting of with -a in each entry. One can efficiently calculate the sum above by following the strategy of (Lou al. (024) as below.
In the case of the absorb diffusion, e e- (j + n,j # n) e rook jn.j f n) = ā. with all other entries being O, where = - ea²i. As such if xi-c # n.
a. e Ar Aeot (n,n) 1,
Σ
oe (ei sb 1)(r'] = -a)s ( [)(c/el
(165)
ml
and if xi-, n
Σ E -a- --'i""" (x/ 88)) sb((t,)))bd)
a Σ
5 (x t) + 5 st t)/n]=
(166)
y'=1
33

---

Published as a conference paper at ICLR 2025
á 5b (2,, (b) + (1 (2), (m = (2),))1 (1 a)s) ()(m]
(167)
v'=1 where (g( 1) = Eyn= og (2r, (vr"
In the case of the uniform diffusion o qua (G.)) while the rest of the entries are where
b n=V - ) and == - (n 11).. Therefore
n
f of
(x 20) s/ (ar,, 1)((/)] (e- (c-bbss(2r. () + bss (2),)))
(168)
y'=1
E-07¹ -1 sh (1, t)x
oe- (i(
(169)
Finally, in the case the roulette diffusion, we have e --
(j + n,j # n) = c,
e a-a qtah (j = n, # n) = à. e qt (j =n.jj n) = 1, e e.
(j # n.j = = epmof 1-e and
aiiqion
n) = 0 and the rest of the entries are b, where ā = e
epmor n-1 + (1-n21) ) ( As in the absorb case, there are two cases, the first one being ai-e # n:
Qiatiot (x) (x)),:)")) sy (x (c- t2,, (5, t) (2 f))n
yi=1
(170)
while for ft-c n, derives:
"
Σ
E oo Q (ai_e: v²)) (a,, O)('') aSz( 1) +1 (1 t)in]
(171)
(e (z) (z). ()) (a b)5) st [m] (-nnnn((()))))) 25 2 1).
(172)
It is important remark that in the case of the absorb and roulette diffusion, (2 t) [n] plays a role in the quantities above only when æi n. In the previous section, is mentioned that s (xx, [m]n] is not learned, but in this case when æ = n is needed. However, this particular case is not problematic as s (ar,, [1) = (2),, [2]]]] = 1, thus we set (2 [r2) to 1 manually.
B.4 PROCEDURE FOR GENERATING THE PLOTS IN FIGURE
We sample batch of 16 test points, and perturb each datapoint with respect different t. Then, for example in the absorb case, we calculate the expressions
Pajo(() xo)
Σ Qx 8) yo
(2rjo(ree s0 (a 1)
(173)
and
Prio(y/
Σ Qr(ær. y) Ma
Pejo ( an)) so sa(w,, f)g) 11 +
(174)
respectively for J and J2, where e = 10-4, In each case, this returns single point, therefore we repeat this procedure 64N times, where N is the number of testing points. This ensures that in all cases (J, and J2), we are estimating the loss in the same number of perturbed points. However, when N' is small, we keep testing Ji and J2 (extending their plots) for more points order to provide to the reader more information about their (limiting) behaviour.
Table 5: Results comparing the roulette cumsition-matrices for different Pm in terms of generative perplexity.
Pm = 0.95
Pm = 0.65
Pm = 0.35
Pm = 0.05
Roulette
72.31
85.72
124.19
284.55
34

---

Published as a conference paper at ICLR 2025
B.5 PER SEQUENCE ESTIMATION OF J1
Here we explain a different method of calculating J1. When applying this method one first samples point (sequence) from the test set, and perturbs it for (say) 1024 time values t. Then the 1024 perturbed points are separated into (e.g.) 64 batches of size 16, and Expression [73 is computed for each batch which returns 64 values. This process is then repeated for the next test point (sequence). If the test set has N points then the process produces 64N values. These values can be averaged, divided by L and exponentiated.
B.6 ADDITIONAL RESULTS
360
Branc
JBn' AB 30:500000000000 CIE1
So.000 335 210005022005
0E7Te% Mitrantd drane
Figure 7: The comparison of the testing history of various methods on various datasets. Each row represents a different dataset, and each column a different type of diffusion transition-rate matrix. The way caleulated J1 to produce the plots is explained in Appendix [..5 The curves presented are the exponentiated cummulative version of the array returned by the described process. Since we do not perturb the test set, this method enables clearer visual comparison.
35

---

Published as a conference paper at ICLR 2025
Table Results comparing SEDD, SEDD scaled (SEDDs), CEDD and CEDD* using J2. Lower is better.
Model (L=128)
LAMBADA
WikiText2
PTB
WikiText103
1BW
SEDD Absorb
69.33
74.58
238.35
74.27
88.35
SEDDs Absorb
66.83
68.93
207.76
68.49
83.26
CEDD Absorb
64.87
65.07
198.00
64.97
78.93
CEDD* Absorb
64.11
64.54
191.38
64.30
79.17
SEDD Roulette
71.37
79.21
227.98
78.67
92.21
SEDDs Roulette
68.25
73.43
206.34
72.77
86.99
CEDD Roulette
68.92
72.16
224.18
71.52
85.37
CEDD* Roulette
67.27
69.61
213.90
69.45
85.64
SEDD Uniform
80.09
91.37
249.50
90.63
101.82
SEDDs Uniform
80.29
88.48
226.03
87.73
99.73
CEDD Uniform
79.46
86.82
276.61
86.52
98.44
CEDD Uniform
82.43
89.68
289.09
88.90
106.32
400
250
300-
250
7k
30k
13k
19k
Figure 8: Comparison of CEDD* absorb I.=1024 and SEDDs absorb 1024 in terms of generative perplexity (Euler, 128 steps), with respect to number of parameter updates.
Table 7: Results comparing SEDD (retrained), CEDD* trained for 20k parameter updates. For generation we use the analytic method with 1024 steps. Float 32 sampling.
Model (A bsorb)
GenPerp
LAMBADA
WikiText2
PTB
WikiText103
IBW
SEDDs L=
55.27
67.92
64.91
173.38
64.13
107.64
CEDD* L=1024
48.29
63.75
57.58
157.71
57.26
98.68
Table Results comparing SEDD, SEDD scaled (SEDDs), CEDD and CEDD* in terms of genera- tive perplexity.
Model (L=1288)
GenPerp F132-GPT2L
GenPerp F164-GPT2L
GenPerp F164-LLamaBB
SEDD Absorb
83,62
172.35
212.15
SEDDs Absorb
79.74
166.35
206.34
CEDD Absorb
74.19
148.21
185.90
C Absorb
72.13
143.86
183.74
SEDD Roulette
87.81
178.94
220.00
SEDDs Roulette
83.36
172.93
212.07
CEDD Roulette
76.71
167.67
208.84
CEDD* Roulette
72.31
158.56
197.79
SEDD Uniform
175.49
169.66
206.91
SEDDs Uniform
171.07
163.88
200.90
CEDD Uniform
168.35
161.84
200.09
C Uniform
179.30
175.42
213.99
36

---

Published as a conference paper at ICLR 2025
Table Results comparing SEDD, SEDD scaled (SEDDs), CEDD and CEDD* in terms of genera- tive perplexity.
Model (Absorb)
GenPerp F132-GPT2L
GenPerp F164-GPT2L
GenPerp F164-LLama8B
SEDDs L=128
79.74
166.35
206.34
CEDD* 1=118
72.13
143.86
183.74
CEDDT L=128
74.07
154.04
195.41
SEDDs 1.=1024
40.95
105.27
111.87
CEDD* 1=2024
40.93*
101.83
107.32
CEDDT 1024
42.18
108.88
115.60
GPT-2L=1022 41.02
41.02*
50.25
Table 10: Correction accuracy percentages. 50k training iterations, batch size of 32 and T. 128.
Model (L=128)
CEDD* Uniform
SEDD Uniform
CEDD* Roulette
SEDD Roulette
PAP
90.7
89.9
90.8
87.9
CAP
91.2
90.3
91.3
88.6
Table 11: Percentage of mistakes corrected for additional models. 25k training iterations, batch size of 32 and L = 128.
Model (I.=128)
PAP
CAP
Model (I.=128)
PAP
CAP
SEDD Roulette
85.1
85.8
SEDD Uniform
86.9
87.5
SEDDs Roulette
86.5
87.2
SEDDs Uniform
86.4
87.1
CEDD Roulette
88.9
89.7
CEDD Uniform
88.5
89.2
CEDD* Roulette
89.7"
90.3"
CEDD* Uniform
89.5
89.5
Finally, we provide results (Table 12) when comparing CEDD* and Discrete Flow Matching that utilizes corrector sampling with the probability velocity (x x.)) 0₂ Brü, where ú(x", & ki (50z (a²) 8m (x*)). We used a at t) for and 0.5. To compute the perplexity bound e we utilized Equations (20) and (21) from Haxholli et al. (2024), and derived:
B 1 I -
1
Σ
(x) Σ
k. 8 6 1- Kg
(logasp') (11 +1)
(175)
K2
pi (2))((//11)) (zilzri 0)) + Br ke
041 k
8mmpz kr
(176)
We used k; = and noticed that by increasing generative perplexity improves but the quality of samples reduced, to the point of generated sequences becoming simple repetitions symbols and rare tokens addition, the measured perplexity bound quickly diverges to infinity, as a increased.
10000. This implies that might modify the temperature of sampling. In
Generative perplexity was computed using GPT2 large, and sampling was performed at Float64 precision.
Table 12:
Model (L=1288)
GenPerp
LAMBADA
WT2
PTB
WT103
1BW
SEDDs Absorb
166.35
67.05
69.37
208.69
69.17
83.87
CEDD* Absorb
143.86
64.60
65.04
192.99
64.69
79.81
Discrete flow k =
145.48
71.90
71.20
221.15
70.84
82.63
Discrete flow K = 2
152.70
72,31
72.87
215.30
72.55
85,82
Discrete Flow Correct
143.38
114.40
113.05
352.32
112.87
130.41
37

---

Published as a conference paper at ICLR 2025
C GENERATED EXAMPLES
C.1 GENERATED TEXT
Listing 1: Generated texts from Absorb diffusion trained using CEDD, L=128.
Any version of the bundle allows corruption. The campaign was delayed year from testing ground, and because of other deficiencies. Perhaps this can be used to inform all the Xbox One ganes come with the same package. Whilst there will be no probleme, there will probably be no cerruption price-wise and we can expect the same with Xbox One game package at the same time. My opinion solid one. lit's s assume Japan the entry point ent country, and PS3 probably supports only one of these. It is effectively competing (see video) against hardware arcade titles such as PS4 or XPS4 as well
Achilles. was ecstatic to be able to watch every one of then and keep them healthy tonight. You can look at the building blocks in college basketball, see that they play very hard and maybe feel supericr to somebody. In my gut, however, you argue that the enc-yoar early good contracts come back and have point. There has been slight step-off with John Wail, which was bit short en my mind. I now know Wall will be listed on the waiver, but I believe with Vic Beasley and Louren Maye this season, those two, with
really good shooters the floor,
t times when it is vital for Scotland's health and welfare The HRH Statement of Purpose and Development (8810) provides framework for the government's creation of the Future of Europe as single body independent Scotland's national welfare system. The government's own intention is to actively develop its vision for the UK's welfare system based on its very nature and how the set of policies and key principals of the forum was established.
The Future Europe Working Group (STO) Omissions the UK Council on Europe's s Audit Office (osivishig
they along with many top players.
LAM: The 1ot people in the parts they lose over are now than Jose Lopez, but if Affaar buys you deep and cheats Howard, Charo Colave Bradley Donaldsen, along with the rest of the opponents, we can see some of it.
LAM: with Santiago Ronaldo in South Florida. Vazal and Pedro Morales playing well, they don't help us.
LAIDHUL: And you actually tell which sides being switched. three moving sides can quite accurate.
A pawn,
38

---

Published as a conference paper at ICLR 2025
Listing 2: Generated texts from Uniform diffusion trained using L=12,
is to facilitate market conditions in jurisdictions. Also, not unlike that elsewhere where in terms of lack of in-house attorneys with no background qualified, the whole country the U.S., Arizona, New England, California, Connecticut, New Hampshire, New York, Massachusetts, Baltimore, Bušfalo and Philadelphia, there are price neasures set to rise to the model. And they come in handy as there are extraordinary levels price in commodities, like alcchol and other drugs. synkrupt.cycy approached, savings rate rigging now make even more dangerous. With broadest institutional licibility assets to sell and demand
looking become in England and
aircraft, under the Mohammed Boland project, [28] [30] [31] In 2010, the government collaborated with Oquantasch states hoping to build the first pilots and aireraft carriers, [33] [34] announced that on 16 October they decommissioned 25 acres of the Mozambique plantation leased by Boeing 737, including several African pilots whom have been imparted on Brazilian soil and ended in. mi military expediticn along with the arms embargo and tax evasion has been delayed until federal Unabloprevalence halted part of the construction. [22]
Engineer units, and airport infrastructure [ edit ]
The
scene. Though a woman called, saying that whether was being on the suspect's s video is not the answer, Desiling says an undisclosed number were present at the mall, and that the police were ready.
In the end, two shoppers, Florimore, 27, sprinted outside the convenience store after struggling ing with downpour to get inside the shop.
When Florimore left the restroom the store, Zimmerman entered the vendor's s cell phone en Sunday and secured her keys Saturday, Desiling said. He helped he lock a key before being used to contract it. After the taped encounter with a current Illinois Bureau of Police
some of which were then scrutinized by the Bernanke Advisory Committes led by Michael Corm.
The U.K. Fed, eager.
Aboard all his most extreme concoctions careening the populace seon queued a shaky living net following weeks of the disastrous fiscal mess caused by the heusing crash and the irpending co lapse in mortgage payments.
Can't anyone manage to usher it all the way. A savvy government could build "a serious living net and that is much less than immediate relief." " Instead of achieving 50 percent reduction in mortgages anyone at the Fed's table did it:
39

---

Published as a conference paper at ICLR 2025
Listing 3: Generated texts from Roulette (0.95) diffusion trained using CEDD, L=128.
deeper problems caused by America's growing deficit, and by our rapidly growing ecenemy. The sane was true today when the only way to undermine the nation's s welfare was in virtuous cycle of austerity, stimulus and lavish budgets. And the scorching reality especially for millions, was that "every man is the child of his parents' mother"-not only abhorrent problem was the working age in Washington, which only saw 44$ of the population working 798 of the workforce
the same household network leaders in their industry. we tapped these network leaders in polls on Election Day, causing favorable results
other federal government agencies had far greater control industry requlatien. Today, drilling wasn't the equivalent of the Environmental Pretection Agency champicned since former president Ronald Reagan put Scn. Scott Desmond (CDE) on top of the Environmental Defense Agency, with more funding area and rural programs under permit.
water
Like Moore, who spent so many million dollars on his calls for the president to institute environmental protections public Lands and national parks, Trump himself had nice shot drilling, saying: "As passicnate about the fight for property owners as we can be right here for Trump."
Canada XL No
Mistress: Next day next step, next step!" He's reply must not translated. "What is Guy's s Your pronunciation? You still don't understand. Why am I saying that, don't mind: I' moved to full set). Fitch word for move. Hey, that must have been long ago. What did you do geing into our meeting? Everything changed, the rules changed... and you all idmitted that Grand Auto Auto wasn't going to work, but you did. need to know it. You' ve been in reference for the bock, Notice."
Don't ignore him. He can't
like low income earners, raising the tax on below.
All Part credits forgiven by at least twenty-one percent paying income tax, and no state interest payments to help the health insurance guarantee. If owners pay the credits, they spend money they owe on each payment, either. A fifth of incomes will pay the credits.
While it's perfectly up to the type of haroship, people can apply state or federal assistance to the credit program. It requires people to sel1 health insurance. The credits qualify and vary from individual persen enrollee.
Having decided to pay their state income taxes on the bill, single
40

---

Published as a conference paper at ICLR 2025
Listing 4: Generated texts from Absorb diffusion trained using CEDD*, L=128.
boat men: drift, and boom. Built, three 9th sailors found themselves losing an average of 12 inches of the underwater 50 feet in the delta base of M. Gielker Jr, U.S. captain (and pilot for the Brigade), sailed deep into bow of thumb and lest at his anchor near shore.
canal en watehdeck with his
Importantly, future "adillista should consider the alarming consequences [t]here in the future the current rapprochement the 9th would be had on the children of all areas of the coast ." re-ferenc
the world's private space program. (This mission, in James's words rust is laced and revisited to the down-space fall and the commercial foundaticn-and oventually-softwars portable to what Lotus's s proprictary eperating system docs new-and also new hardware-scrves, itself, as collective organization rather than fragmented, coordinated copal: "NASA leves the space community at large.")
James retired CEO the company boarc in 2004. And longer, Planetary Resources's judoided business model is intimately 1inked to his own organizational record and by
engaged in hacking American government and stealing cyber defense secrets from its pclitical enemies.
"We know today that the celebrity and his partner in the leaks incident to Wikileaks produce the true story about our government, content and the data shared by our intelligence services, and deserve accountability," said Mark Warner, Assistant Attorney General, at the key E.C.C conference on Wednesday Mashington, .I. "In this case, the battle #wikilieaks against Wikileaks has touched both at new level for the country, and finally has turning historic date.
"Our government has been without the power to hack
non-committal after the third GOP debate.
Gia had already just commented on the upcoming status of her project on CNN, after Trump repeatedly asserted it would discuss politics for the first lady.
Trump said, "Look at this, my part of Hillary Clinton, the mad country, Gareia inntinued.
took my position in a way President Trump didn't have before speaking :,, he said it. admitted some of those things, finally understanding that stood for Oh Hillary businesswoman, she continued.
Garcia made the comment after training herself in Melania Trump's Daily
41

---

Published as a conference paper at ICLR 2025
Listing 5: Generated texts from Uniform diffusion trained using CEDD*, L=128.
the Registration of the Bank of Development despite the skills of Arasan Sance. unemplcyable novice is in high regard for Swatch Up Worker as there large scale projects bylaws that have already been difficult the sector.
Caught diving too deeper, Arasan Sance hac a career at IIT puilt en Year Up in Science the wood,, Theatre during instructor Ranance 2017. The guide says that
class at the university some jobs may have been finalised and it was understood well that, if efficacious, McCarley could decide to be invited to a perhaps larger academic class.
strength of subsequent fitness genes from mitochondria after maintenance of. This type of transiticn is called exalted mitochondrial, and although extended long feat is indeed amolecule strategy that requires stringent tests selective binders.
In particular, one that abundant, modifiea mitochendria can enable recovery; first they produce mitochendria which is attached to the fuel cell, rapidly moving essentially back to normal form, transitioning from the atrophy to a more muscular phenotype. Mitria such as LIL pretein A (C1A and transporter R (Anaris-in proteinase), NAD is used as adaptation strategy, like many
how things were in the later years. The truck was littered with a black pepper. It was overflowing with halal green and mo yellow corn pepperoni and charcoal platc.
Pictured used have baseballs encased in the front driveway with the sports hooders. Guy's s black ores tip, black bean bag, silver, cherry, and granny banger was ricocheted, and when the Appomeess the Chinese bullfly relled me over to the Morrissey. 0 44, everything fresh and natural earlier, inscriptiens. This guy's s big white hooded or blue jacket
just different from those in
idea of increasing momentum, however, is to slow down a quarterback's strength. His is in the midst of slow renaissance with the team's s ensuing games,
is aging one who getting the effectiveness missing Greg Jennings and injuries. That this crucial area comes as the 49ers failed progressual Mac is forcing his fades down instead of sticking fully unless he is getting better.
The Jets is approaching the top-10 in front but, as soon as they fill it in, an miderdog-Dish seems to benefit from this slight fatigue in their upcoming ng game.
Negvious Matt Miller
42

---

Published as a conference paper at ICLR 2025
Listing 6: Generated texts from Roulette (0.95) diffusion trained using CEDD*, L=128.
such fraudulent statements and denials as, "the truth lies, and white lies" could never be trusted. More than one among you have just known Williams about.
Many are the fictional experiences Milliams shaped, in ways that embodied him in generation. The Tielemin figure in Janes Joyce's white sharacter largely was based on male gaze and information and, from Williams' perspective, reflecting to the lens on men from a different time was disappointing.
Williams's material images of how, in essence, had occupied not present world: beaten, destroyed, marginalized
an attempt to fully realize the potential for pedestrian safety where there is no dcarticulate conduct for the traffic.
His own current plans, for which Lokxelli is the secretary of the planning drove changes with little progress through traditional detours as of late. "Not highways, but others are quite a bit," Akka forecasted "to make sure that one imp is not going to get faster
torn. How to do that away from what it is becomes when roads occur. There all going to be of no serious consideration at present, either in design or perspective."
mad over and watching him laugh loudly enough to say so, and my point is that it's not just that I am not displeased with politician or CEO, that I think people looked in their shoes and paid well."
To see countries that see relevance to the show's a multimill liinaiire,, including so frustrating the small smile in the face of the world, Thomas Archifelln applied, but wanted to represent the image of keeping ignorant out his protecticnist mission.
ARTICLE CONTINUES BELOW
GEORGE STEPHANRAD: In few weeks, Papacuzzi and did Paul
and made fun of the tactic. They acknowledged was just silly. Indeed, shortly after the bocks, real Western painter and fellow broach type, said, te another wood miller, "he's straightened Skeptic got hard chin and flies in the back. The winced to me that you're re a courtesan. You' 11 know what's s up you<! endoftext >>Get it done. Your friends will never see it
Parenta of kids and old owners dogs yesterday said they are going to miss president-elect Denald Trump's look, 'Loving' for
43

---

Published as a conference paper at ICLR 2025
Listing 7: Generated texts from Absorb diffusion trained using L=DD,
likely to be the fact that Nephi calls God's s beloved "The Synod Word difference when it translates to an accident.
As viewers know, an Andesad man his wife will never meet in Nephi' s time God begins to connect with others, Le Burkert said.
Source: Honuses de Jeunee
/ >he concept of attunational fit occurs from the presence perceptual
experiences on the cortex of the brain, as exposed recently by research by senior Prof. Linda Perriene. Since personal dynamic patterns don't t exist within human subjects, cortical nemory is often cognitive
will look like a target.
97) J.O. you think of White House national coordinator Clinton political operative? Kittylo. sort of still up-the- table the White House. P.O. That person who refuses to have the same meek with a client with concerns/liven. just pretty boycotted by pulling off in the relationship. Glad let's get that out of the way tonight and point out his ability and his job promise.
98) Next to the President David Grimsnek | In at least 100 years George и. Bush has
last month invested in Ria Louise, New York through wealth of $7 billion this year.
Moanwhile, the committce is a part of Katzman & Girle Service Conference, the firm that licenses the annual facility. Frisco visits to CraftEarlier this year.
"We newg CraftyTags for phinanthropic, recovery, trust, and use of cur services, with immense support from federal government, Mossody said.
Under the partnership, Neoneys also is working on expansion as partner among other net-for-profit rescurce providers, with an increasing amount of resources and core services covering
at least one, but nothing qualified from any perspective they could be offering them.
"The response
from very recent comment made in blue material the public out was decided ahead of time, Smith writes of the public's prurient. So we' ve found that a long and full response excludes the ones who didn't. believe the policy would meet the government's standards for proof and hold."
Suze enough, their apologists could do what they want on hard science. But that being said, they'd d like some insights into scientific ethics and
44

---

Published as a conference paper at ICLR 2025
Listing 8: Generated texts from Uniform diffusion trained using SEDD, L=128.
can generate json of the library based on which one exists for the scenario instance.
<?php require JetlinkViewer :: create ( ePub : "hu }' d. add c. app (). head () //Outputs from Sub_seleee from Google str_ http://www.oggg..ccoooo com ); //add it while that function starts down the path and
wew.its..uinjjooo +/
In order, Node handles repos (), several usable implementations in Mesh is have been tested.
The JavaScript library actually has dire explanation for itls: it does not, but unlike
using you or they use you taking people out to catch you or whatever. That wondcrful because of the ways I look. I had gone from anywhere from 20/22 here 40 years old 50/50. I also grew up feeling. 'True Bloed Baby' has gave me a lot of body novements that/s s got me right. there and used to pop their first transformation. That show."
My sister really grex up as a nairety boy. Thrones a while? No, it' 8 only sometimes, and since older people might even see us for 15 days or 30 days per night, how long they
but the ban kidnapping remains the only difference.
Seyer has closed the case in court letter seeking response.
For the arrest video visit your Confused IMZ app page.
Updated Sept. 24, 2013. Tribute to lawyer Harrison, </endoftext >>ils Dahman is starting his season with the Red Hulls Crystal Palace. The midfielder rounded out his fi fth season in the United States, missing three League meetings U-a.
You can watch the interview with the media below. You can see an image for the camera belcw.
Gils Dahman started
Girl"). The earnestness of the ensemble is random, cheerful though frustrated with beth the actions of themselves and the different perspectives
their journey. This classic masterpiece-writing sequels and his imagined form movies-is the tale of cinema that still has had aesthetics and is able to grow. But it also sees the fall of the film genre-though very serious yet complex- live show that can also find places lost by leading the way to work arcund intersectionality where the lineage of Reed and Dormedy ("Iren Man" though) applicable-and that she's starting
45

---

Published as a conference paper at ICLR 2025
Listing 9: Generated texts from Roulette (0.95) diffusion trained using SEDD, L=128.
its own territory. All the European Union as well as the J.S.
government, including the British East Company, were issued coins the Federal Reserve and a few days support a foreign buyer. President Lee had adopted law rejecting donations from the country's s tax code, preserving the spirit of the Constitution of the 1870 Act of self-rule in South America. the caddition of the British banking system, the American Mint provided the first ammunition and smuggled guns Europe. Nonetheless, a sericus problem: the Unien was secretly a armed anti -communists with a formidable internal authority. The union government was
in the open
to blame?
It's full of wrongheaded sheen and bears expressions and carrics a similar light ending. I'm not turned off that Craig's Edge of Truth NYC guide right thing.
don't something wrong going and do the
What's different about another story that can somehow tell the tale unfolds in Chicage well with the ppl being approved by the General Service.
Like an Independent. Right? All right.
What Happens?
A Licensed Reader keeps all relationships private. They have no precodents. By all things
result it couldn't publish it.
No, my work is not the best way that all of us can be at home with the poll results. I acknowledge you believing it, and doing this pol1 is going to have to happen to make the difference.
But again, we are honestly happy with the results let's see. We could not vote from all the experts but that still could happen. what's your opinien that debate?
The other most annoying but the most upset #MSNBC. the Wall Street Journal, people called the polls slow Gore down, saying that even the ABC thought
400 times the base capacity there has been priced four. And now another station can do the same for capacity with two other lines to I per cent of the power meter factory circuit which also costs $56 million for cancellation
The next debate is on the schedule, but may not happen late 2017 without major delays. <lyys.//enooteett >hee French media reports that Chinese technology firm EtApple has today started claining that it has all given back, in its software, return to earlier this month's s SmartEasy payment processing that will simply payment it for most Chinese users. And based on recent developments, the same company is sending
46

---

Published as a conference paper at ICLR 2025
Listing 10: Generated text from Absorb diffusion trained using CEDD*, trained L=1024.
two days later meeting with the Governmental Affairs Committee next Tuesday, replacing the committee's naps with an informal discussion.
In his Thursday hearing, Sessions said he had been "used" into criticizing the CIA.
"There has been some degree of transparency looking
that's not going to come cheap for himself or the Arerican people, or really to undermine his independence, which I think, despite being in various positicns, I've been very, very clear that. I would angry very upset about that," he told senators at the meeting, "But think as we go into Wednesday, and the United states cabinet are finding ways to make very clear to citizens that they will respect that system."
the matter of
Several high-level officials also called the appointment, saying it was "a bit of a quirk" since a revolving door offices such as the CGA is already running rampant and has yet to hold any hearings.
"We've seen people make fairly much their career life from the Justice Department cut of this," said one GOP source. "And I don't think that choice is appropriate when that's government entity that has given up let of accountability claims."
Speaking after noon another Republican member of the Senate Commerce Committee told. Todd Risch (R-Idaho) however, how he thought about Sessions, "and henestly, I didn't see whele lot of discussion point suggesting tone change since the remeval of some five these people ... mean, want people to find cut that respect goes to the Teadership of their department, and I'm not sure it's nice to move on down that precipice. "</endoftext/
tone. There has been a lot of
you try to put any pressure, you put than the Pacers into you love watching. Before the Boston Celties come to their games against Washington with.500 records on Tuesday night in Pittsburgh, you will likely watch a five-point game between Brad Stevens and Antetioun Gouden. Hollis have fourth -ost competition in the game overall (49 points per game) but before games start their season of over 14.5 points per game has already passed.
That led Bostor to early 2-1 deficit.
No, not really. In addition to last season, however, the Celtics really struggled during the Wizards' first year of the NBA. For only the first 24 minutes, Washington had 18 minutes their D-man shooting guard who averaged at least four points, and he chipped another points on at least 10 fourth quarter quarters Boston was cutscored in the paint on five four-point shots, including two-point turnover margin.
If included Stevens and Horford scoring 18 points in the only fourth quarter quarter from the paint, it would have probably been 2-1, and in the second-half-first quarter at half-time it would have been 9-2 game thanks to the disastrous fourth-quarter performances. Yes, the Celtics have been unable to pull the ball away from us in the paint success was probably in spite of the bad 2s, otherwise, our defense was weak.
the fourth quarter, but their
47

---

He said he has settled on key question about the car: "If custcner keep things Published as a conference paper at ICLR 2025
Listing 11: Generated text from Absorb diffusion trained using SEDD, trained L=1024.
The principle selling for period of energy will make the most sense next year, as Tesla's s record rapid sales may be sluggish but demand lives on.
Tesla Motors Corp's CTO Carl Conti said beginning new conversation with Tesla about the future state of its workforce 2014 may be a part of its strategy which as the corpany races to reduce its price of luxury vehicle deliveries, the first units that get to market again in earnest are expected to begin to get "next level, he said.
AP Photo
"I appreciate [the] ideas. These not fast break, but ways to strengthen the margin least knowing that if you get the right timeframe you make, its way ahead, Musk told reporters at The New York Times over the past month.
"None these people have looked towara [s] [slourgigg the cost the increase of the vehicle, unless you see it as downward spiral," he said.
Although Musk and other executives point
auto drivers and purchaser of gasoline other, the company's strategy that reduced price for vehicles will outstrip some of those buyers, Conti still said.
Conti said the price of Model is projected fall by least 33 percent decline rate in 2015, on average, Centi expects Tesla expects 58.5 millien to 59.5 million in 2015 and also for an average 15 percent decl ine to rise to February.
For the Model S, by contrast, Tesla's expected to reach write- down of about $500 millien 20000, 000 EVs in September and more than 700, 000 currently, he said.
The first Model deliveries of the year is expected nearly begin March That price is heavily influenced by dealer monitoring that includes a tctal inventory of 600,000, with 400, 000 delivered next month, Conti said.
He noted Tesla expects its predicted upward trajectory increase
2.4 percent by 2015 as production in Australia and China accelerate to 4.6 percent next year, respectively.
Musk also noted in The wall Street Journal that while Tesla expects more than three consecutive production next year, will production a total $8 million in U.S. roll-backs in the fall.
needs it and is willing to pay the rates that moving, then we cught to consider adding incentives for it."
"The price of Model S expectations for production is driver'all. Conti said. "When the profit is available, it means car is not in its pocket. 31."
seeing this very good time after Jan.
48

---

Published as a conference paper at ICLR 2025
Listing 12: Generated text from GPT2, trained L=1024.
power well past the NPC veraus NPC meter for each unit type. Their use is varied, and thus allows for considerable amount of flexibility with The Banner Saga, but whether you prefer your parent's s roots stay part of the community with allies everyone can fight and help each other -- remains to be seen. For that, you can always pick the steam community tab (not requized, that's how the game is named -- no one likes spanming themselves for news now, know).
However, you agree to terms of service that you agree to disable your favorite kind of multiplayer games (by default) to prevent unwanted conflicts: factions cannot be disbanded or changed just because of non-friendlier flect. So it's a pretty straight fairly simple toggle to get used to.
only listed
in game and
Ugh, it sounds like the HUD might get progressively messier game ends.
Shields charging slowly: active shields last several minutes, recharge fully in seconds, and charge every second they charge.
Shields wearing' shield' else doubles their size, can clip high above their skull, and are invincible to all other shields around your ship that border the shield and can depley wall-mounted shields. Other shields like this vessel. Hit somecne en the Pustules to activate shield shields. In an expedited, nen-lethal, coordinated confrontation, it is fairly easy to compare stats to each shield for ship to use.
can only be used to shield the
Shield minus shields compensation: up to five shields are held rank in your ship's shields. It's called 'delay," but with a ton of fancy commands on screen, you don't have to know (or even see)
their context emplcyment to make sure to compensate for some of the distortion inherent in that bit. Seconds before your shield starts charging (just seconds before it behaves like hiring another worker), the timer will start running indefinitely, which means shields can snap to the back of your ship if kept as low as possible. Avoid collisions caused by instantaneous cap.
Shield halves that match the length of the ship's hull: you can control the range between Shields, such as your Private Capital signatures, where the ship's bulletproof liquor clock is set at half-space and your Heavy Construction Band Bump the Gane's Game soundtrack Fade let the corp count (if the corporation has sixty transports within one day of being prepped, it's probably already pretty clese to ten minutes, but your mission select loading speeds will 1 be holding it back little).
Surrounding explosions moving over the bow: it's deceivingly easy to line up targets like an Infallible Surface projection theater Explosion shield reon high in the sky: fire teams together and mean to demolish their own low-flying illions every once in while, yet closely follow their targets like shadow warriors to the detriment of themselves. Covertly set to' to' square hit' for easier custamization, Ubisoft's new ship weather HUD is the key to spreading effective damage throughout the entire ship using the standard meteorological updates.
Shaping your space: shape each bridge to create low-risk rockouts or pathogen/addagg paths. This may be tedicus, but it actually whizzes through by itself while similtaneussly adding damage giants mercenary ships.
49

---

Published as a conference paper at ICLR 2025
D CTMCS PRELIMINARIES
D.1 DISCRETE-TIME MARKOV CHAINS OVER FINITE-STATE SPACES
A discrete-time Markov Chain in finite-state space a stochastic process X1, .......... Xr. where each state X; depends solely on the preceding state Xt-1. The states X; can take on any value from the set 21, ....,..... 5), where S denotes the total number of possible states, and T represents the number of time steps. The probability of being in state at time is
PX = x) = Σ p(x; = 2, X1-1 = y) = Pije- PX x|X1-1 V)P: = y).
(177)
If we place all such probabilities Pt(X; x)) in a vector 8t of shape S 1, such that (x(x) Px(X; z), then from above we can deduce that
8, P
(178)
where P(z, y) Pejt-1 (X, x x|1 y). Given initial probability distribution So over states, the equation above fully determines the evolution of the probability over states with respect to time. If it is known that the state time 1 is y. then 8 is simply column y of P. This implies that the sum of the elements of each column of P is one.
D.2 CONTINUOUS-TINE TIME MARKOV CHAINS OVER FINITE-STATE SPACES
is possible to define a stochastic process with the Markov property in finite-state spaces, for t E [0, 7). (Anderson 2012). As previously, we can define a discrete-time process, on time points {0, e,.,, T' e, J.. such that there probability of activating the previous transition mechanism when progressing from time & - to t, otherwise stay where we are with probability (1 - e). Removing the random variables to simplify notation, we have
(+))) = (1 - (x)++ +e Σ (oc-
(179)
y=1
We notice that when e = 1 the equation above coincides with Equation 177). and in addition as before write Equation (179) in matrix form
8, (1 - 8) + P = (I + e I)) 81-0 [I + cQ) St-c where Q = P I. (180)
From Equation 180). we see that $e-de-e Q8i-ei which when taking the limit e + 0 becomes
d8: = Qs
(181)
dt
Given an initial probability distribution 8o over states, the equation above fully determines the evo- lution of the probability over states with respect to time. Indeed, the distribution over states at time t is the solution of the linear ODE in 181): 8, = edQ Therefore, if it is known that the state at time 0 is j, then 8; is simply column j of eQ Naturally, can rescale the time variable such that t = (t),, where a is monotonically increasing, (0) = 0 and lim (t)) = T, so that for
zy == Sg. we get e so = r(1 z0 = 21, and
dz o (1)== Qizi where Q, - o' (I)Q.
(182)
di
Matrices Q: = (P) I) clearly satisfy the properties of transition-rate matrices Suhov & Kelbert 2008). Matrices Q are chosen such that: a) the matrix exponential (1) is easy to calculate, which essential as Puoo (xly)) (1)Q (ar, y): and b) 2 an easy reference distribution to sample from (Austin et al. 2021 Campbell et al. (2022).
Finally, similar to diffusion processes in continuous spaces, the continuous-timg Markov chain in Equation (182) also admits reverse process (Kelly) 1979 Sun al. 2023):
dz
11113111
(183)
dt
50

---

Published as a conference paper at ICLR 2025
where Q2 v) Qr P((a) for * v, and Q((x.)) -Etze Q,(y, Since we can easily sample from the reference distribution, the only unknowns preventing us from being able to run backwards are the ratios Re(m)
pe(r)
pe(v) also known as concrete scores (Meng et al. 2022), which we desire to model using a neural network. Once such ratios are modeled we can generate samples from the learned data distribution p by discretizing Equation (883) follows:
p(xe-c== = y 2t = x) = -y) x)
+ O(2))
(184)
51


Reviews:
[{"review_id": "O1fmQPFN8n", "reviewer": "Reviewer_5C4V", "summary": "This paper provides three new theorems concerning the KL divergence between the data and the learned distribution, improving model evaluation through a bound. \nA new transition-rate matrix (roulette diffusion matrix) that allows for token correction after unmasking in the reverse process is introduced, deriving its exponential matrix to enable efficient training/sampling. Finally, denoising cross entropy loss over score entropy is proposed for training discrete diffusion models, with up to 10% lower perplexity/generative-perplexity, and 15% faster training steps.", "questions": "Line 147: \\bf{x}^i_0 = x^i_0, is it a typo?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and clear. \nDeveloping efficient perplexity bound and the training algorithm for discrete diffusion language models is an important yet challenging problem.\n\nThe particular forms of the bounds, the roulette diffusion matrix, the denoising cross entropy loss and are interesting and new. Both theoretical development and empirical evaluations are conducted.", "weaknesses": "From Table 2, overall GPT-2 performs best over SEDDs, CEDD*, and CEDDT. The practical usefulness of the new bound and training algorithm is not completely clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6fbqqz5ozO", "reviewer": "Reviewer_cDEx", "summary": "This paper presents new results for discrete diffusion models trying to estimate density ratios for language modeling task. The authors' contributions are threefold: (i) they provide an upper bound $J_2$ on the perplexity of discrete ratio-matching diffusion language models which is computationally more efficient than the bound $J_1$ known from the literature, and, as shown through empirical evaluation, is consistently tighter; (ii) they derive a novel training objective called cross-entropy discrete diffusion (CEDD) loss by making use of reparametrization of concrete scores; (iii) they propose an alternative to known transition-rate matrices referred to as \"absorb\" and \"uniform\" and derive exponential of this new \"roulette\" transition matrix necessary to compute conditional probabilities efficiently.", "questions": "The experiments reveal that $J_2$ is tighter that $J_1$ in all cases you studied. Do you have intuitive explanation of this phenomenon? Comparing the formulae (13) and (16) I can see that $J_2$ contains function $\\bar{l}$ differing from $l$ by the $K$ function taken at conditional scores. This $K$ can have different signs depending on its arguments, and $J_2$ also has additional integral term and entropy of the reference distribution. Do you have any idea on how to explain the superior performance of $J_2$ compared to $J_1$?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The novelty of the contents of this paper consists of several theoretical findings and undoubtfully is the main strength of this paper. These findings include a novel type of transition matrices for discrete diffusion models along with all the facts necessary to use it in practice. The feasibility of using this type of forward diffusions was shown through experiments. Second, a score reparameterization was used to derive a CEDD model competitive with a common SEDD model. Lastly, the novel bound $J_2$ on the perplexity of the generated text is proposed, and the experiments demonstrate that it is consistenly better than previously used bound $J_1$.\nOverall, I think this is a good paper from the point of view of originality and significance of questions raised in it since discrete diffusion language modeling is the topic that so far receives relatively little attention compared to continuous-domain diffusion models.", "weaknesses": "1. I am not sure about the correctness of perplexity evaluation in Section 4.1. To evaluate likelihood (which is equivalent to perplexity evaluation) with an imperfect model may lead to unreliable results as you mention in line 362. I would suggest to use some better and more modern language models instead of GPT-2 and report some aggregated score.\n2. CEDD* differs from CEDD by some hand-crafted time-dependent weights (line 324) and the results across different transition matrices types differ as follows from Table 1. It raises some concerns about CEDD training objective in terms of stability and sensitivity to the choice of $\\omega(t)$. \n3. Language models trained on language modeling task can be evaluated not only from the point of view of perplexity on this task. I am not talking about some NLU tasks we can finetune language models to - I've never seen such attempts in the context of diffusion models. But, e.g. in [1] diffusion language models are shown to be capable of spell checking / grammatical error correction. So, maybe it could be more impactful if the authors evaluated their CEDD also on tasks like this. \n4. When it comes to discrete diffusion language models, I always wonder about their scalability in terms of, say, model/batch size, or data size, or vocabulary size. Common language models based on transformers typically have some useful scalability properties and their performance increases with the increase in data/model size for carefully chosen training hyperparameters. Papers on discrete diffusion language models usually do not discuss these issues, but in my opinion it is important if we want to prove that such models really can compete with modern language models.\n\n[1] Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions, Hoogeboom et al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q6YmCNjqDk", "reviewer": "Reviewer_wpbF", "summary": "The paper presents, to the best of my knowledge, novel theorems that eventually lead to an enhanced performance compared to the baseline (SEDD). The authors further introduce a predictor-corrector version for the ratio matching framework called roulette diffusion.", "questions": "See above", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper is well written and introduces, to the best of my knowledge, a novel set of theorems.", "weaknesses": "* My primary concern is that the authors appear to utilize the official implementation of SEDD, which has been demonstrated to have a flaw in the gumbel max trick used for sampling. In [1], the authors show that when using sampling in low precision for high dimensional distribution it leads to an effect similar to sampling with temperature. This is highly related here as the exponential in the proposed method may even anneal the temperature more.\n* Prior Work: The authors only compare their results with SEDD, neglecting other relevant works in their analysis, such as [2] and [3].\n* Predictor-Corrector: The aforementioned works incorporate some forms of predictor-corrector methods, and I believe it is necessary to include comparisons with these approaches.\n\n[1] Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling - https://arxiv.org/pdf/2409.02908\n\n[2] Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Design https://arxiv.org/pdf/2402.04997\n\n[3] Discrete Flow Matching https://arxiv.org/abs/2407.15595", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HoKJXFKOW5", "reviewer": "Reviewer_78NZ", "summary": "This paper proposes novel theoretical results and empirically validated improvements for diffusion language models in the continuous time discrete state space framework.  Specifically it establishes new bounds and relationships for KL divergence between empirical and learnt distributions, including an improved upper bound on perplexity of the estimated model.  Furthermore, it introduces a novel transition-rate matrix, termed Roulette diffusion, which generalizes both uniform transition-rate and absorb transition-rate matrices and admits an analytical matrix exponential.  Empirical results demonstrate that use of cross-entropy discrete diffusion (CEDD) to carry out ratio-matching (modeling of ‘concrete scores’) for uniform, absorbing, or roulette diffusion, is superior to ratio-matching using score-entropy (SEDD), and the proposed upper bound on perplexity is tighter than the previously published bound.", "questions": "Related to the weakness above, would it be possible to take these language models through the supervised fine-tuning steps and evaluate on set of tasks that LLMs are commonly evaluated on?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Advances state of the art in continuous time discrete space diffusion language modeling with novel theoretical results pertaining to KL divergence between empirical and model distributions, including a tighter bound on model perplexity.\n* Empirical results demonstrating the superiority of the proposed perplexity bound, as well as uniformly better perplexity performance of the CEDD approach over SEDD.\n* Well written paper.", "weaknesses": "The largest weakness of the paper in my view centers around evaluation of the proposed models.  The paper presents perplexity results under GPT-2 as well as the perplexity bounds, but it is also acknowledged that GPT-2 based measures have their own potential issues, and with bounds a question of tightness always remains.  Thus, how good these models truly are, and how do they compare/contrast with more traditional auto-regressive LLMs remains an open question.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)