The Diffusion Duality
Authors: Subham Sekhar Sahoo
Justin Deschenaux
Aaron Gokaslan
Guanghan Wang
Justin T Chiu
Volodymyr Kuleshov
Venue: ICML 2025 poster (2025)
Keywords: diffusion language models
diffusion models
large language models
distillation
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: We prove that discrete diffusion processes admit an underlying Gaussian diffusion formulation which enables the design of faster training and sampling algorithms.

Abstract: Uniform-state discrete diffusion models hold the promise of fast text generation due to their inherent ability to self-correct. However, they are typically outperformed by autoregressive models and masked diffusion models.  In this work, we narrow this performance gap by leveraging a key insight: Uniform-state diffusion processes naturally emerge from an underlying Gaussian diffusion. Our method, Duo, transfers powerful techniques from Gaussian diffusion to improve both training and sampling. First, we introduce a curriculum learning strategy guided by the Gaussian process, **doubling training speed** by reducing variance. Models trained with curriculum learning surpass autoregressive models in zero-shot perplexity on 3 of 7 benchmarks.
Second, we present Discrete Consistency Distillation, which adapts consistency distillation from the continuous to the discrete setting. This algorithm **unlocks few-step generation in diffusion language models** by accelerating sampling by two orders of magnitude. We provide the code and model checkpoints on the project page: http://s-sahoo.github.io/duo

The Diffusion Duality
Subham Sekhar Sahoo
Justin Deschenaux
Aaron Gokaslan Guanghan Wang' Volodymyr Kuleshov
Justin Chiu
Abstract
1 K
Zt Cat
:; (1-
Uniform-state discrete diffusion models hold the promise of fast text generation due to their inherent ability to self-correct. However, they are typically outperformed by autoregressive models and masked diffusion models. In this work, we narrow this performance gap by leveraging a key insight: Uniform-stard state diffusion processes naturally emerge from an underlying Gaussian diffusion. Our method, Duo, transfers powerful techniques from Gaussian diffusion improve both training and sampling. First, introduce curriculum learning strategy guided by the Gaussian process, doubling training speed by reducing variance. Models trained with curricu- lum learning surpass autoregressive models in zero-shot perplexity on 3 7 benchmarks. Sec- ond, we present Discrete Consistency Distillation, which adapts consistency distillation from the continuous to the discrete setting. This algorithm unlocks few-step generation in diffusion language models by accelerating sampling by two orders of magnitude. We provide the code and model checkpoints on the project page: http://s-sahoo.com/dnoo
UNIFORM
1
Zt
Z1
Cat
arg max
Wt
W1
N(0, Ix)
GAUSSIAN
We N ( (1 āi IIk)
Figure 1: An illustration of Uniform-state discrete dif- fusion (top) and the underlying Gaussian diffusion (bot- tom). While both are separate Markov processes, applying arg max maps Gaussian latents W; TR" to discrete latents 2, E V, transforming their marginals from ğ (+|x; āe) (6) to (-bx:: T (ā:)) (1) and adjusting diffusion parameters from o₁ to ag T(a)) (10). Notably, the BLBOO for Uniform- state diffusion induces a tighter bound on the likelihood than Gaussian diffusion, as established in Theorem 3.1.
ues this tradition by demonstrating that discrete diffusion process is, in fact, an emergent phenomenon of an underly- ing continuous Gaussian diffusion process. This perspective enables the design of faster training and sampling algorithms for discrete diffusion models.
1. Introduction
Diffusion models (Sohl-Dicksee Dickstein al., 2015) are power- ful generative models inspired by physics. Gaussian diffu- sion models excel at synthesizing realistic and high-quality continuous-valuee data such as images (Ho et al., 2020; Rombach et al., 2022), audio (Kong et al., 2021; Liu et al., 2023b), and videos (Ho et al., 2022; Wu et al., 2023; Esser et al., 2023; Blattmann et al., 2023). Gaussian diffusion is well studied the success of these models is rooted in techniques such as efficient parameterizations of the de- noising model, which improve upon the standard mean- parameterization (Ho et al., 2020; Salimans & Ho, 2022; Zheng et al., 2023), faster training techniques (Kingma et al., 2021), efficient samplers (Karras et al., 2022), and distilla- tion schemes that enable single-step generation (Song et al., 2023; Song & Dhariwal, 2023; Yin et al., 2024).
An eternal theme in mathematics is that discreteness emerges from underlying continuity. From quantum me- chanics, where the quantized energy states of electrons arise as solutions to continuous wave equations, to the binary logic of digital circuits, fundamentally driven by smooth analog currents, discreteness has repeatedly and naturally emerged from an underlying continuum. Our work contin-
'Computer and Information Science, Coreell Tech, NY, USA. "School of Computer and Communication Sciences, EPFL Lau- sanne, Switzerland CCohere, NY, USA. Correspondence Sub- ham Sekhar Sahoo <ssahoo@ Cesahoo@cs.comee cou
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

The Diffusion Duality
While Gaussian diffusion is well-studied, it underperforms discrete diffusion models on tasks involving discrete data- such as text (Sahoo et al., 2024b), graphs (Liu et al., 2023a), and molecules (Lee et al., 2025). However, from the per- spective of Gaussian diffusion, the design space for discrete diffusion models remains primitive: mean parameteriza- tion for the denoising model (Sahoo et al., 2024a; Schiff et al., 2025) and slow ancestral sampling (Austin et al., 2021) are still the dominant approaches. Recent work distilling Masked Discrete Diffusion Models (MDMs) im- proves sampling speed (Deschenaux & Gulcehre, 2024), but performance degrades severely in the few-step regime. Unlike Gaussian diffusion models with Probability Flow ODEs (Song et al., 2020), MDMs lack an "implicit" prop- erty: a deterministic mapping from noise to data. This property is vital for few-step distillation methods (Song et al., 2023; Frans et al., 2024), but MDMs forgo it due to their deterministic prior, which requires stochasticity during sampling.
ity (Sec. 5.2). Importantly, in the low-NFE regime, Duo outperforms MDMss Our main contributions are threefold: (I) We establish theoretical connection between continu- ous and discrete diffusion, demonstrating that discrete dif- fusion arises from an underlying continuous Gaussian dif- fusion. This insight enables the transfer of techniques from the continuous domain to the discrete setting, opening up new possibilities. (2) Our framework doubles the training speed of USDMs by introducing low-variance curriculum, and (3) accelerates sampling by two orders of magnitude by adapting efficient distillation methods from continuous diffusion models.
2. Background
Notation
We represent scalar discrete random variables that can take K values as 'one-hot' column vectors and define V = {x e (0, 13 ΣK- x; = 1} as the set of all such vectors. Define Cat(:; T as the categorical distribution over K classes with probabilities given by π e A, where A denotes the K-simplex. Additionally, let 1 = {1} and (a, b) and a o b respectively denote the dot and Hadamard products between two vectors a and b. We use x V and [xt) }/] V to denote sequences of length T.
Our objective is (1) to design a framework for discrete dif- fusion that enables the transfer of advanced training and inference techniques from Gaussian diffusion discrete diffusion models. And (2) create language models that support few-step generation. To this end, we focus Uniform-state Diffusion Models (USDMs) (Austin et al., 2021). In this paper, we discover a very remarkable prop- crty of USDMs- emerge from Gaussian diffusion pro- cesses as illustrated in Fig. 1. We call this phenomenon- the Diffusion Duality which expands the design space of USDMs, making it possible incorporate techniques de- veloped for Gaussian diffussio.. Notably, USDMs mod- els allow token updates during reverse sampling unlike MDMs, naturally correcting earlier mistakes without requir- ing costly predictor-corrector (Zhao et al., 2024; Wang et al., 2025) steps-saving function evaluations (NFEs). However, these models have historically underperformed compared to MDMs (Austin et al., 2021; Lou et al., 2023), raising the key question: Can USDMs be made competitive with MDMs? And more importantly, can the implicit property of the underlying Gaussian diffusion be leveraged for fast, few-step generation?
2.1. Discrete Diffusion Models
Consider the clean data x e V drawn from the data dis- tribution qdata In the discrete diffusion framework (Sohl- Dickstein et al., 2015; Austin al., 202 the complex data distribution qaaa is mapped to simple distribution through a sequence of Markov states. Sahoo et al. (2024a) propose a simplified variant- an interpolating noise frame- work-wherd the forward process (9:)se(o.1) smoothly tran- sitions from Adata to prior distribution Cat(:: T)), by intro- ducing latent variables 2, V whose marginals conditioned x at time t are given by:
a.))xxa))) Cat(-; + (1-a))
(I)
where the diffusion parameter O₁ é [0. 1] is a strictly de- creasing function in i, with or-D 1 and 01-1 & 0. A discrete diffusson process is characterized by the time evo- lution of marginals follows linear ordinary differential cquation (Anderson, 2012):
We answer both questions with Duo, a rich framework of theoretical connections between USDMs and Gaussian dif- fusion. Duo enriches the design space of USDMs by in- corporating Gaussian diffusion, which allows to develop efficient training strategies that accelerate the training of USDMs, significantly reducing the performance gap be- tween MDMs and AR models on standard language gen- cration benchmarks. Notably, we surpass AR models on 3 out of 7 zero-shot datasets (Table 2). Furthermore, this duality allows us to adapt consistency distillation (Song et al., 2023) from Gaussian to discrete diffusion, reducing NFEs from 1024 to 8 with minimal effect on sample qual-
d di 4: = Qi9
where Qe RRK is the state transition matrix.
There are two main variants of interpolating noise frame- works: MDMs (Sahoo et al., 2024b), which use a masked token prior л = m with m E V as a special mask token, and USDMs (Schiff et al., 2025), which uses a uniform prior over V (π = 1/K). These frameworks differ in their
2

---

The Diffusion Duality
forward corruption dynamics. In MDMs, the clean data x either stays unchanged or transitions to the mask token m, after which it remains masked for the rest of the process. In contrast, USDMs allow each token to either stay the same or transition uniformly to any other token in V. with the transition probability determined by the diffusion timestep (see Fig. 5 for examples). These forward dynamics impact the reverse generation process: USDMs permit continual token updates, while MDMs fix tokens once unmasked. To mitigate this limitation, predictor-corectto methods have been proposed for MDMs (Campbell et al., 2022; Gat et al., 2024; Wang et al., 2025), but at the cost of added computa- tion. In contrast, USDMs naturally exhibit a self-correcting property absent in AR and MDM approaches. As a result, our work focuses primarily on the USDM framework.
NELBO for such a process is given as (Kingma et al., 2021):
NELBO ( x);x)
= --B,1-2(00111 ( / (1)[ xo(w,,()ll (7)
where (1( is the time derivative of the signal-to-noise ratio (1) = 0e 0/2/(1- - 2) for the Gaussian diffusion process.
2.3. Consistency Distillation
Consistency models (Song et al., 2023; Song & Dhariwal, 2023) are a class of generative models that define a bijective
mapping between the samples from the noise distribution (0, Ix) and the data distribution Qdam- They build on deterministic samplers for Gaussian diffusion (Song et al., 2020; 2021), specifically using the Probability-Flon ODE (PF-ODE). Given pre-trained Gaussian diffusion model X8, which requires hundreds or thousands of sampling steps, Consistency Distillation is a popular technique to distil them down to fewer steps generation that enables much faster generation. The distillation begins with a teacher model x often the Exponentially Moving Average (EMA) of the student model X obtained during the course of train- ing, A noisy sample w, is drawn from the forward process qe (.xx (6), and less noisy sample w, is obtained by nu- merically solving one PF-ODE step using xo The student model is then trained to match the teacher's estimate of the clean sample minimizing the following loss:
Schiff et al. (2025) show that for USDMs, the state transition matrix Q, is given by:
Qi
Kay -11'KTK]
(3)
where at, is the time derivative of a and the true reverse posterior for a timestep 8 / is given as:
[epo(1 x1,x) = Car ; Kazz, O x (ay/s -0:) (2,, x) -
(0ts - -x)x (1 (rle) (1. 1/kK
(4)
Kan(Ze; x) + 01
C(0,0)
Xo- (w).
(8)
where d R x R + R denotes the error between the teacher model's reconstruction Xo- (w.,²) and the student model's reconstruction x (w,, 6) of the original sample and A: [0. 1] R² is a weighting function that scales the loss based on the diffusion time-step t.
where ditla = az/ag. Since is unavailable during inference, we approximate it with neural network X, : V [0, 1] - A K with parameters 0. The resulting approximate reverse posterior is defined as (lze)) gstt( 2,, x (z( t)). The denoising model is trained by minimizing the Negative Evidence Lower Bound (NELBO) (Schiff et al., 2025):
3. The Diffusion Duality
NELBO (g,Po;
=B (2r, (o(z,, 043
0u,,)))))))))))))))) x),
(5)
Unlike discrete diffusion, Gaussian diffusion is replete with eetl-established empirical techniques, which have driven significant advances in both training (Ho et al., 2020; Sali- mans & Ho, 2022; Zheng et al., 2023) and sampling (Karras et al., 2022; Song et al., 2023; Song & Dhariwal, 2023; Yin et al., 2024). Our goal in this section is to establish a theoretical bridge between discrete-state diffusion and continuous state diffusion, which will enable us to leverage tools from the later to improve the former.
where is defined in (40). Sampling from this model be- gins with the prior - ~ 1 1/K, and proceeds via ancestral denoising, i.c., by drawing Z ~ p (-Izz) at each step.
2.2. Gaussian Diffusion Models
Gaussian diffusion maps a data distribution Idau to a simple prior distribution usually a Normal distribution (0, Ix), through sequence of noisy latents w,- ~ @e (-Ix), whose marginal distribution is given by:
We propose a simple method to map a Gaussian latent to the discrete space: the arg max operator. But does this transfor- mation of latents also transform Gaussian diffusion process into a discrete one? A necessary and sufficient condition for this is that the marginal distribution of the discretized vector satisfies the characteristic ODE of a discrete diffusion process (2). We first derive a closed-form expression for
(. a.) (o,x, (1-x,) (1_ Ik),
(6)
where the diffusion parameter dy E *[0, 1] is monotonically decreasing function in t. For مu-o = 1 and d, 0, the
3

---

The Diffusion Duality
this marginal and show that arg max maps the marginals of Gaussian diffusion to those of a Uniform-state discrete diffusion, including transformation of the diffusion pa- rameters (9). Finally, we verify that this marginal evolves according to (11), establishing that arg max transforms a Gaussian diffusion process into a Uniform-state discrete diffusion process,
The arg max operation transforms Gaussian diffu- sion into Uniform-state diffusion, with the diffusion parameters related by (10).
More formally, this can be expressed as:
(z(z) ((ā))) aar margmax]+ao (wilx;; àt)
(12)
We begin by defining a Gaussian diffusion process on x V as per (6), with Gt=o Qdua and @t=1 (O, Ix). Let W, de (-Ix; ar) be an intermediate latent at time l. Next, define the operation arg max: IR K + V to map continuous vector w K to the one-hot vector corresponding to the index of its largest entry in w. i.., arg max(w) (w) = arg m z w.
where the * operator denotes the pushforward of the K- dimensional Gaussian density q under the arg max map. yielding categorical distribution ge with K classes.
Thus, as diffuses in the discrete space via Uniform- state process, there exists an underlying continuous-space representation in which x follows Gaussian diffusion, as illustrated in Fig. 1.
Discrete Marginals Let z arg maggmaxww we) and (((.x)
denote its conditional pmf marginalized over We te (-1x; 6t). In Suppl. 1, we show:
Note that these two processes are separate Markov chains with transitions between them, and they induce dif- ferent variational bounds on the log- aigliikllihood. Specifically, they yield distinct ELBOs: (5) for the discrete diffusion process and (7) for the Gausssian diffusion process.
ELBO
1 2,- P. (.px Cat : (())))x++ +1-+01111 (1-T(a))
K
where the function T: [0, 1] [0, 1] is the Diffusion Trans- formation operator, defined as:
Theorem 3.1. The ELBO for the Uniform-state discrete dif- fusion is tighter than for the underlying Gaussian diffusion.
T(ā.)=
K
S.
00
2-
de
④ K-1 (x)dz
1
We provide a proof in Suppl. A.4. Briefly, we show that:
K
VI
K
logpo(2 (x) > ELBO (q, pe: 2 ELBO (?.Po;x), (13)
(10)
where pa is the denoiser in the Gaussian diffusion space, and Pe := [argmax] pe is the denoiser in the discrete diffusion space. Since the ELBO is inherently tighter in the discrete space, it is advantageous to design the denoising model to model discrete latents. Hence, we choose (5) as the training and evaluation objective. The term fonfinn (5) involves materializing the one-hot vector X, which increases memory usage and slows down training. We reformulate it with a Rao- Gho-Blackwellized objective (41) that avoids materializing one-hot vectors and also reduces training variance, resulting in faster training and lower perplexity (Sec. 5.1).
where (2) =cxp(->²)/ is the standard Normal dis- tribution and (=)) fi 11/2)2z/ 22T is its cumu-
lative distribution function.
Time Evolution of Marginals Next, we examine how the discrete marginal Pt evolves with time as the continuous vector w, undergoes Gaussian diffusion. In Suppl. A.2 we show that P. evolves as per the following linear ODE:
- P. = T(aa)
d
11 -KIx] Pr
(11)
dt
KT
Sampler To sample from Duo, we use ancestral sampling for USDMs (Sec. 2.1). Furthermore, to improve text quality, we propose Greedy-Tail Sampler, that improves the sample quality by decreasing the sample entropy in manner similar to nucleus sampling in AR models models (Holtzman et al., 2020). Specifically, during the final denoising step, instead of sampling the clean sequence via x ~ p.ja()) (-), we perform greedy decoding: x arg max(p (-)) where 8 denotes the time discretization.
where T' represents the time derivative of T. From (2) and (3), we infer that (1I) characterizes a Uniform-state dis- crete diffuusinn process with diffusion parameter T(xx). ). It is important to note that while the marginals of the discretized latents evolve according to Markovian Uniform-star dis- crete diffusion process, Gaussian diffusion trajectory after discretization, might not map to discrete diffusion trajec- tory. We discuss this in detail in Suppl. A.3.
We exploit this duality to incorporate Gaussian diffusion into the design space of USDMs. This allows us to design a low-variance training algorithm that leads to faster train- ing (Sec.4.) and a distillation scheme that unlocks few-step generation diffusion language models (Sec. 4.2).
Duality The implications of (9) and (11) are quite pro- found. These reveal a fundamental connection between Uniform-stai state discrete diffusion and Gaussian diffusion bridged by the arg max operator:

---

The Diffusion Duality
4. Applications
for the NELBO. In Suppl. B.I, we show:
We now present two applications where discrete diffusion models benefit from leveraging the underlying Gaussian diffusion. In Sec. 4.1, we introduce curriculum learning strategy that reduces training variance and leads to faster training. Then, in Sec. 4.2, we propose distillation algo- rithm that cuts the number of sampling steps by two orders of magnitude with minimal impact on sample quality.
NELBO PoPo x
E (g (aae)
Σ
Duo (zi, (xix ,t,, atix)
[e[L]
( ( Σ 2c[L]
Dua (zf== oof:=arg arg maagmaaxww.. (w)),
Xy (larg (aar (w; ) t), T((==)))))) (15)
We extend our discrete diffusion framework to sequences xI ~ -daa of length L. The forward process and the reverse process factorize independently over tokens as: qo (z) (z ax)) [e{[1] (z/[xx;; ;a) based on (1) and L 8 1 1) = ]ea{r) (z/e2z)zz I:L, x I:L 1)) based on (4), respectively. Here, X9 V x [0.1] + A denotes the denoising model. Consequently, the sequence-level NELBO
where a, T(a)) ) is obtained via (10) from the Gaussian diffusion coefficient o and also verify this empirically. As discussed in Sec. 3, these are distinct Markov chains whose marginal distributions are related only through (12). This reparameterizatioo underpins our curriculum learning strat- egy which we present in the next section.
(a(z
decomposes into a sum of token-level losses:
4. 1.1.2. Low- VARIANCE TRAINING LOsS
NELBO (g. (o:
To reduce training variance, we replace arg max( w in the denoising model input (15) with a tempered softmax. We argue that this substitution eases recovery of the clean sequence from its noisy counterpart, and that the difficulty of this recovery is regulated by the temperature parameter.
-Ezuu[(.11,o Somaazi Xx 1.T, a(;x). (14) [e[2]
where fuwo is defined in (41).
4.1. Faster Training using Curriculum Learning
As shown in prior work (Jang et al., 2017; Maddison et al., 2017), arg max is limiting case of softmax:
Curriculum learning (Bengio et al., 2009) gradually exposes models to increasingly complex data, starting with simpler, casier-to-denoise noise patterns and progressing to more challenging ones. Here, we design a curriculum for USDMs by exploiting the underlying Gaussian diffusion.
arg max( (w() lim softmax(w(/7).. T-
(16)
We relax this operation by setting the temperature param- eter T > O. While computing the NELBO in (15), note that the discrete diffusion parameter T( (o.) spans the inter- val [0, 1J. as does its Gaussian counterpart āt. The diffu- sion transformation operator T (10) has a crucial property: as the vocabulary size K increases, a small sub-interval [o, 1 losue1 within the domain of T is sufficient to map onto the full range [0. 1]. For instance, in Suppl. C.7, we observe that for K = 30K, when â₃ E [0.85, 1]. the corresponding O₁ = T(ā:) nearly spans the entire interval [0, 1]. This observation is counter-intuitive since the Gaussian latents mostly resemble x, one might expect the discrete NELBO to approach zero when evaluated with ā restricted to such narrow range. However, in practice, the NELBO remains largely unchanged. Why? The key reason lies in the dis- cretization step. Even small amounts of Gaussian noise in w can cause the output of the arg max operation to change drastically, as it highly sensitive to perturbations. As a re- sult, much of the extra signal is lost due discretization. To mitigate this, we allow the denoising model x, to access the continuous latent w/ through a tempered softmax in (16). This relaxation helps preserve more of the signal. making the reconstruction task easier. In this way, the temperature parameter 7 effectively controls the difficulty of the learning problem.
Similar to relaxation methods discrete gradient estimation (Jang et al., 2017; Maddison et al., 2017) 7),, our curriculum is centered around annealing the temperature parameter of smooth approximation of arg max. We reformulate the NELBO for discrete diffusion in terms of arg max over Gaussian latents (Sec. 4.1.1). The denoising model is then trained to operate on the arg max of these Gaussian vari- ables. We then relax the arg max using a tempered softmax (Sec. 4.1.2), which yields a lower-variance but biased esti- mator of the ELBO. Initially, the model operates on fully re- laxed, continuous Gaussian latents. As training progresses, the temperature is gradually decreased, transitioning the model's inputs from soft (continuous) to hard (discrete). By the end of the curriculum, the model effectively operates on discrete latents, closing the gap between training and inference-time behavior.
4.1.1. DISCRETE NELBO WITH GAUSSIAN LATENTS
Consider the discrete diffusion NELBO (14), which marginalizes fow over the discrete latents z/-2 ~ qe (.(.ll:',;a))) Our goal is to re-express this objective in terms of Gaussian latents w.'L ~ q (.)++t;;))) such that marginalizing over w... yields the same numerical value
5

---

The Diffusion Duality
Gradlent Variance (
Deterministic Discrete Trajectories (DDT) Consistency Distillation relies PF-ODE parameterized by the de- noising model. In our setting, the trained discrete denoiser cannot be used to parameterize the ODE in the Gaus- sian space, as it operates only on discrete samples-the temperature T in (17) is reduced to zero by the end of the training. To circumvent this, we construct a determinis- tic trajectory in Gaussian space by reversing the PF-ODE using optimal denoiser, and then project this trajectory to the discrete domain. Let PODE denote such trajec- tory. Given clean data point x" I:l. qdata and Gaussian noise E I.L = =f ~ NOO, IK) [4]], PODE 1.L, el++)= [[ + VI- ae (E=1))(((.o see Suppl. B.2.1 for de-
Duo w/o
Variance
10
10K
20K 50K
100K
Figure 2: Curriculum learning drastically lowers the gradi- ent variance in Duo trained with a fixed r = 0.001. The fig- ure shows the summed gradient variance of the 100 weights with the highest variance, comparing Duo with CL (blue) and without CL (grey).
tailed discussion. Next, we project this trajectory to the discrete space via the arg max operator:
PonT(x" 1:L E2LL)
- largg max(áxxx Vi )
(18)
Hence, unlike prior discrete diffusion methods, we design the denoising model Xy A'VY x [0, 1] A f. handle both continuous latents and discrete latents; see Suppl. C.2 for details. During training, we sample [[ Joss<rss from sub-interval that a E [a, bog Following Arriola et al. (2025). we set < 1 as â₂ = doesn't pro- vide much training signal. Thus, we propose the following training loss:
refo..)"
P DDT serves a proxy for the absence of a proper PF-ODE defined in the discrete space; Fig. 5 for illustrate example.
Distillation Given teacher model xe our goal is to distill it into student model X2 that generates samples of similar quality but in fewer steps. To perform distilla-
tion, we sample an adjacent pair of latents (z) I:L ,2, (:
Luain Ex [x Σ fom (zi: arg maggmaxwww (w().
{zz]_ 2) J given step size ō [0, 1]. Here, z) is noisier than z and serves as the input to the student model. Let xØ- (z. 1.f. 8) and X2 (z) 1:L t) denote the output distributions over clean samples produced by the teacher and the student models, respectively. Following Deschenaux & Gulcehre (2024), we train the student by minimizing the KL divergence between these distributions:
l:T. (.) PPDDT 1:1,, el:")).. j.j [6. 1]} for a
xe([softmax w / ( aq a=: x(a.):x'). (17)
This loss doesn't correspond to valid NELBO because the denoising model operates on continuous- r.V., while the loss is defined for a discrete diffusion process. It only becomes valid NELBO in the limiting case lim,-o' with 8 = 0 and
1. During evaluation, we evaluate the model as a discrete diffusion model using (14). As shown in Figure and Table 3, this approach results in lower training variance compared to previous MDMs and USDMs, ultimately improving model likelihood (Sec. 5.1).
(oc Σ DKL (x6(z) Ec[L]
(),()) x1xxx (z) (xl:t,s))
;s),
(19)
The distillation process proceeds in N rounds, each con- sisting of M training steps. At the end of each round, the teacher weights are updated with the current student weights. The full procedure is outlined in Algo. 1.
4.2. Discrete Consistency Distillation
In this section, we present a new method to exploit the du- ality property of USDMs. This duality enables USDMs to adopt Consistency Distillation technique developed for Gaussian diffusion models to distil them into few-step generation models. However, standard discrete diffusion models cannot use this approach due to the absence of such PF-ODEs. To address this, we introduce Discrete Consis- tency Distillation (DCD), which sidesteps this limitation by utilizing the PF-ODE of the underlying Gaussian diffusion model to construct deterministic trajectories.
5. Experiments
We evaluate Duo on standard language modeling bench- marks, training on LMIB (Chelba et al., 2014) and Open- Web (OWT) (Gokaslan et al., 2019) with sequence pack- ing (Raffel et al., 2020). We train our models for IM steps with a batch size of 512 on both datasets. For LMIB, we use a context length of 128 with the seart-base-un suase-ud tokenizer (Devlin et al., 2018) with sequence packing (Ar-
6

---

The Diffusion Duality
Algorithm 1 Discrete Consistency Distillation (DCD)
et al., 2023), and D3PM Absorb (Austin et al., 2021). While training Duo, we set
Input: Dataset D, learning rate 77, number of distillation rounds N, number of training iterations per round M, ema μ. weights of the denoising model f, weights of the EMA model Oema. discretization step 6.
function of the training iteration for Duo: т = 0.001 for the first 500K steps (n < 500K) with 3 0.03 and y = 0.15 in (17), and T = 0 for the re- maining steps up to 1M (n 2 500K). To compute PPI. for Duo, we use (14) with aq = 1-
for i = 1 to N do
0 < +toppgrad ())
for j = to M do
Bias-variance Tradeoff First, we study the effect of r on the training training dynamics in Fig. by training Duo on the LMIB dataset over 150K steps with a fixed T {0, 0.001, 0.01, 0.13. Here, 7 = 0 (blue) corresponds to (14), ie., no curriculum. Recall that a larger introduces more bias but reduces training variance. Ideally, T should strike a balance- minireiiiiniiiingggg both the bias (measured by deviation from the blue curve) and the variance in the loss curve. For T = 0.1 (red), the loss drops sharply. indicating excessive bias, making it suboptimal. As decreases to 0.01 (orange) and 0.001 (purple), the loss curves become more stable. Among them, 7 0.001 is the most desirable, as it closely follows the blue curve (low bias) while exhibiting significantly lower variance.
Sample x 1:1.
D. 1~ (1, 1], and é - (O),
s max(t- 6,0)
2 2. + [arg max xx + vI-a. 2111
z L.L + larg max max( VI- 26 )
LOCD )² + (xo
1:1 ,1), xx (2g ,))
,1).X0
0 - 0- - 1/V v/V DDCD
e ema + (000rad(100mm)) + (1-1))0
end for
8- 2-6 6 end for
return 0 0ema
riola et al., 2025; Austin et al., 2021) and without it (Sa-
Faster Training Notably, after just 10K steps of fine- tuning as a discrete diffusion model i.c., at 510K steps, Duo achieves a PPL of 35.2-almost 1.5 points better than UDLM trained for 1M dreps-indicating that curriculum learning accelerates convergence by at least 2x. In Fig. 2, we compare the summed gradient variance of the top 100 weights with highest variance for Duo with (blue) and with- out (grey) curriculum. For these weights, we notice that curriculum learning reduces the gradient variance by an order of magnitude, which also manifests as lower loss vari- ance in Fig. and Table 3.
hoo et al., 2024a; Lou et al., 2023; He et al., 2022). For OWT, we use a context length of 1024 with the GPT-2 tokenizer (Radford et al., 2019). Following Sahoo et al. (2024a), we reserve the last 100K documents for validation. Our model is a -rrrmmmrmmtter modified diffusion trans- former (DiT) (Peebles & Xie, 2023) with rotary positional encoding (Su al., 2023) and adaptive layer norm for con- ditioning on diffusion time, consistent with prior work (Lou et al., 2023; Sahoo et al., 2024a). Training is conducted on 8xH100s with bfloat16 6 precision. We train Duo us- ing (17), which requires computing the integral in (10). To reduce computation overhead, we pre-compute and cache 100K (ā.,7 T (āx)) tuples, significantly smaller than the de- noising network. The Gaussian diffusion parameter o; is parameterized using linear schedule i.c., (ā, = 1-t)ee(o,1]-
Likelihood Evaluation On LM1B and OWT (Table 1), Duo outperforms previous USDMs and Gaussian diffussion models, notably SEDD Uniform and UDLM and shrinks the gap with absorbing diffusion below 2 PPL points.. On LM1B, We retrained Plaid which attained PPL of 89.9 in 100K steps while Duo achieves PPL of 43.0 in the same number of steps. This result is excluded from the table due to incomplete training; see Suppl. C. for details.
5.1. Improved Training
Our experiments show that (1) the proposed curriculum learning strategy (Sec. 4.1.2) accelerates training by 2x and achieves new state-of-the-are among USDMs (Ta- ble I), and (2) Duo performs competitively with Absorbing State diffusion across major language modeling benchmarks, even surpassing AR models on 3/7 zero-shot PPL. bench- marks (Table 2).
Zero-Shot Likelihood Evaluation We measure the zero- shot generalization of the models trained on OWT by eval- uating their PPL on 7 other datasets. Following Sahoo et al. (2024a). our zero-shot datasets include the validation splits of Penn Tree Bank (PTB; Marcus et al. (1993)), Wiki- Text (Merity et al., 2016), LMIB, Lambada (Paperno et al., 2016), AG News (Zhang et al., 2015), and Scientific papers from ArXiv and Pubmed (Cohan et al., 2018). We observe that Duo outperforms SEDD Uniform and Plaid across all benchmarks. More notably, it achieves a better PPL score than SEDD Absorbing on 4/7 datasets, MDLM on 1/7, most
Experimental Setup The primary baselines for Duo are the leading USDMs (SEDD Uniform (Lou et al., 2023) and UDLM (Schiff et al., 2025)) and Gaussian diffusion method, PLAID (Gulrajani& & Hashimoto, 2024). Addition- ally, we compare Duo with an AR model and MDMs such as MDLM (Sahoo et al., 2024a) (SOTA), SEDD Absorb (Lou
7

---

The Diffusion Duality
Ancestral
Greedy-Tail (Ours)
(5.7)
Duo
Duo
80
160
MDLM
MDLM
Gen PPL
77.69
75.24
69.58
Entropy
[5.7]
5.5
5.5
(5.7)
40
0..5
(5.71
T 1024
15.67
"5506
Gen PPL
20
80
15.5
(5.6
(5.5)
15:55
Base
Distilled
4:
41
15.5
15:47
15:47
15:47
Figure 4: Sample quality comparison between the base Duo model and Duo distilled for 5 rounds using our DCD algorithm. The distilled model matches the base model's sample quality in just 16 steps (vs. 1024) with ancestral sampling. With our Greedy-Tail sampler, sampling steps can be further reduced to 8, achieving slightly better Gen PPL and lower entropy.
40
Sampler
Ancestral
Grel
8
16
32
64
128
256
512
1024
Sampling Steps
Figure 3: Sample quality comparison of Duo vs. MDLM. Duo outperforms MDLM in Gen PPL (4) for base models and in low-NFE regime after 5 distillation rounds.
5.2. Improved Sampling
Our sampling experiments show that for undistilled models, (1) Duo generates higher-quality samples than all pre- vious diffusion models (Fig. 9); (2) combining DCD with the Greedy- -ril sampler reduces the number of sampling steps by two orders of magnitude (Fig. 4); and (3) the distilled Duo model outperforms distilled MDLM model, especially in the low NFE regime.
Table 1: Test perplexities (PPL; 1) on LMIB. 'Reported in He et al. (2022). Best uniform/Gaussian diffusion value is bolded. DDenotes the dataset didn't 't incorporate sentence packing. 'Reported in Arriola et al. (2025). For diffusion models, we report the bound on the likelihood. Best diffu- sion value is underlined. Denotes retrained models.
LMIB
LMIB
OWT
Experimental Setup We distill Duo on OWT using DCD, following the same setup as our main baseline- MDDM distilled with SDTT (Deschenaux & Gulcehre, 2024),
Autoregressive Transformer
22.3
22.8*
17.5
distillation method for MDMs. We run N = 5 distillation rounds, starting with discretization step 8 = 1/512 in Algo. I and doubling it every M = 10K steps. To assess sample quality, we report GPT-2 Large generative perplexity (Gen PPI.) and average sequence entropy for diversity. As noted by Zheng et al. (2024), masked diffusion models can suffer from low diversity and misleading Gen PPL under low- precision sampling. To address this, precision in all sampling experiments.
Diffusion (absorbing state)
BERT-Mouth" (Wang Cho, 2019)
142.9
D3PM Absorb (Austin al., 2021)
76.9
DiffusionBert (Ho et al., 2022)
63.8
SEDD Absorb (Lou et al.,, 2023)
32.7
24.1
MDLM (Sahoo et al., 2024g)
27.0
31.8*
23.2
Diffusion (Uniform-state Gaussian)
D3PM Uniform (Austin et ul., 2021)
137.9
float64
Diffusion-LMr
(Li 2022)
118.6
SEDD Uniform (Lou al., 2023)
40.3
29.7
UDLM (Schiff al., 2025)
31.3
36.7
27.4
Sample Quality In Fig. 9, Duo consistently outperforms all MDMs and USDMs in Gen PPL across sampling steps Te (8..... ..... 1024), with particularly strong performance at low NFEs. In Fig. 3, we compare Duo distilled with DCD to MDLM distilled with SDTT after rounds (entropy values in parentheses). Under ancestral sampling, Duo performs significantly better than MDLM for T 32. This because MDMs generate many tokens independently and cannot revise them, leading to incoherence at low NFEs. In contrast, USDMs are self- self-correcting: they can fix earlier errors in later denoising steps. At higher NFEs, MDLM outperforms the distilled Duo. While MDLM (with SDTT) matches the Gen PPL of the AR model, its lower entropy (5.4 vs. 5.6) indicates reduced diversity.
Duo (Ours)
29.9
33.7
25.2
notably, outperforming autoregressive transformer on 3/7 datasets.
Ablation Duo introduces two key improvements UDLM: (i) a Rao-Blackwellizze ELBO (41) and (ii) low- variance training curriculum. As shown in Table 5, the overall 3-point improvement in PPL comes roughly equally from both components, with (41) accounting for about 1.7 points and the remainder from the curriculum.
8

---

The Diffusion Duality
Table 2: Zero-shot perplexities (1) of models trained for IM steps on OWT. All perplexities for diffusion models are upper bounds. 1 Taken from Sahoo et al. (2024a). Taken from (Lou et al., 2023) models were trained for 1.3Msteps as opposed to the baselines that were trained for Msttps. All perplexities for diffusion models are upper bounds. Best uniform Gaussian diffusion values are bolded and diffusion values better than AR are underlined. denotes retrained model.
PTB
Wikitext
LMIB
Lambada
AG News
Pubmed
Arxiv
Autoregressive
Transformer'
82.05
25.75
51,25
51.28
52.09
49.01
41.73
Diffusion (absorbing state)
SEDD Absorb'
100.09
34.28
68.20
49.86
62.09
44.53
38.48
D3PM Absorb
200.82
50.86
138.92
93.47
MDLM
95.26
32.83
67.01
47.52
61.15
41.89
37.37
Diffusion (Uniform-state / Gaussian)
SEDD Uniform*
105.51
41.10
82.62
57.29
82.64
55.89
50.86
Plaid
142.60
50.86
91.12
57.28
UDLM
112.82
39.42
77.59
53.57
80.96
50.98
44.08
Duo (Ours)
89.35
33.57
73.86
49.78
67.81
44.48
40.39
Greedy-Tail vs Ancestral Sampler In Fig. 3, the Greedy- Tail sampler improves Gen PPL. by reducing sample entropy. In Fig. 4, we observe that the ancestral sampler enables a 64x speedup (reducing NFEs from 1024 to 16) while main- taining Gen PPL. Greedy- Tail offers an even greater 128x speedup, with a slight drop in entropy. Interestingly, as shown in Fig. 11 and Fig. 12, each distillation round im- proves both Gen PPL and diversity when using the Greedy- Tail sampler-unlike ancestral sampling-suggesting that Greedy-Tail is particularly effective for distilled models.
Distillation for Faster Sampling Distillation in Gaussian diffusion leverages PF-ODEs (Luhman & Luhman, 2021; Salimans & Ho, 2022; Song et al., 2023), which are un- available in discrete space. Our method circumvents this by defining PF-ODEs using an optimal denoiser in Gaussian space and mapping them to the discrete domain. In contrast, Deschenaux & Gulcehre (2024) tackle this issue by per- forming distillation along stochastic trajectories due the lack of deterministic ones. We observe that our method Duo surpasses all previous approaches at few-step generation.
Ablation In Algo. 1, we use the denoising model weights as the teacher, deviating from the common practice of using EMA weights in consistency models. To test this choice, modify Algo. to use EMA weights (Algo. 2) instead. As shown in Fig. 10, using the denoising model as a directly leads to a better distilled model.
Argmax Differentiation The softmax annealing trick was originally proposed by Jang et al. (2017); Maddison et al. (2017) to enable backpropagation through arg max op- eration. The core novelty of our work is in establishing fundamental connection between discrete and Gaussian diffusion via the arg max operator. We show that the soft- max annealing trick can be naturally repurposed to design a low-variance training curriculum for USDMs.
6. Related Work
Diffusion Language Models Prior work on discrete diffusion language models operates strictly in discrete space (Austin et al., 2021; Lou et al., 2023; Sahoo et al., 2024a; Schiff et al., 2025; Arriola et al., 2025). In contrast, we expand their design space by incorporating continuous- state Gaussian diffusion, showing that training USDMs in this setting not only accelerates convergence but also im- proves model likelihood. Li et al. (2022): Han et al. (2022): Dieleman et al. (2022); Gulrajani & Hashimoto (2024) use Gaussian diffusion for language modeling by injecting noise into the continuous embeddings of discrete tokens. Duo takes a different approach: it defines the diffusion process directly over one-hot token representations, rather than their embeddings. This hybrid treatment achieves better perfor- mance than methods confined to either domain alone.
7. Conclusion
In this work, we established theoretical connection be- tween continuous-space Gaussian diffusion models and discrete-space Uniform-state diffusion models. We lever- aged this connection to achieve a 2x speed-up in train- ing (Sec. 5.1) convergence and lwo-orders-of-maaniie improvement in sampling speed (Sec. 5.2). While USDMs trail MDMs in terms of perplexity, we showed that they out- perform in few-step generation. We hope that our theoretical foundation opens up new avenues of future research which would leverage this connection to improve USDMs by bor- rowing techniques from Gaussian diffusion a connection non-existent for MDMs.

---

The Diffusion Duality
Impact Statement
documents. Proceedings of the 2018 Conference of the North American Chapter of the Association for Compulational Linguistics: Human Language Tech- nologies. Volume 2 (Short Papers), 2018. doi: 10. 18653/v1/118-0097 URL pettp:/dxxdoo. oorg/ 10. 10.18653/v1/n18-11
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, specifically those related to the generation of synthetic text. Our work also be applied to the design of biological sequences, which carries both
potential benefits and risks.
Cover, T. and Thomas, J. Elements of Information Theory. Wiley, 2012. ISBN 978111858571. URL
https://books. google..o..oooooooooooooooo.. com/books?id= VWQ5G66ycXMC.
References
Anderson, W. J. Continuous-time Markov chains: An applications-oreented approach. Springer Science & Business Media, 2012.
CSISZAR, I. Information-type measures of difference of probability distributions and indirect observation. Stu- dia Scientiarum Mathematicarum Hungarica, 2:229-318, 1967. URL sictps://cin mii. jpp /rr/rd/////////
Arriola, M., Sahoo, S. S., Gokaslan, A., Yang. Z., Qi. Z., Han, J., Chiu, J. T,, and Kuleshov, V. Block diffusion: Interpolating between autoregressive and difffusion lan- guage models. In The Thirteenth International Confer- ence on Learning Representations, 2025. URL https: //openreview. net wttpe/wweet//euumee/eeeee Forumm (ttprrummiiittyyyt2555-.
1571 157141712581164448
Deschenaux, J. and Gulcehre, C. Beyond autoregression:
Fast Ilms via self-distillation through time, arXiv preprint arXi::2410.21035, 2024.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. arXiv preprint 1IIv:1810.04800 2018.
Austin, J., Johnson, D. D., Ho. J., Tarlow, D., and Van Den Berg, R. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:17981-17991, 2021.
Dieleman, S., Sartran, L., Roshannai, A., Savinov, N., Ganin, Y., Richemond, P. H., Doucet, A., Strudel, R., Dyer, C., Durkan, C.,, et al. Continuous diffusion for categorical data. arXiv preprint arXiv:2211. 2022.
Bengio, Y., Louradour, J., Collobert, R., and Weston, Curriculum learning. In International Conference on Machine Learning, 2009, URL https://app. semanti cachooar..cccarrr.ccommoooaaa.. org/ httg//ooopuTt::30444
Esser, P., Chiu, J.. Atighehchian, P., Granskog, J., and Ger- manidis, A. Structure and content- santent-guided video synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7346- 7356, 2023.
Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S. W., Fidler, S., and Kreis, K. Align your latents: High- resolution video synthesis with latent diffusion models. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 22563-22575, 2023.
Frans, K., Hafner, D., Levine, S., and Abbeel, P. One step diffusion via shortcut models. arXiv preprint arXiv:2410.1255: 2024.
Campbell, A., Benton, J., De Bortoli, V., Rainforth, T., Deli- giannidis, G., and Doucet, A. A continuous time frame- work for discrete denoising models. Advances in Neural Information Processing Systems, 35:28266-2822 22229,, 2022.
Gat, I., Remez, T., Shaul, N., Kreuk, F., Chen, Q., Synnaeve, G., Adi, Y., and Lipman, Y. Discrete flow matching, 2024. URL petttss://axivv... 2407.1553
Chelba, C., Mikolov, T. Schuster, M., Ge, Q., Brants, T., Koehn, P., and Robinson, T. One billion word benchmark for measuring progress in statistical language modeling, 2014.
Gokaslan, A., Cohen, V., Pavlick, E.,, and Tellex. S. Open- webtext corpus. http://SkkylonD000 10t.gggiii.07....hbbb.. Lt//ppennebTextCorpus. 2019.
Chen, T., ZHANG, R., and Hinton, G. Analog bits: Gen- erating discrete data using diffusion models with self- conditioning. In The Eleventh International Confer- ence on Learning Representations, 2023. URL //openreview. (tt
Guirajani, I. and Hashimoto, T. B. Likelihood-based diffu- sion language models. Advances in Neural Information Processing Systems, 36, 2024.
Han, X., Kumar, S., and Tsvetkov, Y. Ssd-Im: Semi- autoregressive simplex- based diffusion language model for text generation and modular control. arXiv preprint arXiv:22 17432, 2022.
Cohan, A., Dernoncourt, F,, Kim, D. S., Bui, T., Kim, S., Chang, W., and Goharian, N. A discourse-aware attention model for abstractive summarization of long
10

---

The Diffusion Duality
He, Z., Sun, T., Wang, K., Huang, X., and Qiu, X. Diffu- sionbert: Improving generative masked language models with diffusion models. arXiv preprint arXiv:2211.15029, 2022.
Lou, A., Meng, C., and Ermon, S. Discrete diffusion lan- guage modeling by estimating the ratios of the data distri- bution. arXiv preprint arXiv:2310. 2et.:3330.1183.. 2023.
Luhman, E. and Luhman, T.
Knowledge distillation in iterative generative models for improved sampling speed, 2021. URL https:/ wawpivivivv.///////a///aaa org/ /abs/ 2101. 02388.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6851, 2020.
Ho, J. Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, D. J. Video diffusion models. arXiv:2204.03458, 2022.
Maddison, C. J., Mnih, A., and Teh, Y. W. The concrete distribution: A continuous relaxation of discrete ran- dom variables. In International Conference on Learning Representations, 2017. URL https: /ttp:///ppenree /ttp:////ppeeeeeeeeeeee net/ htt///orummiii - JE5
Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The curious case of neural text degeneration. In International Conference on Learning Representations, http///ppeeeeeeeeeeeeeeeeeeeeeee net/ forrforummm
Marcus, M., Santorini, B., and Marcinkiewicz, M. A. Build- ing a large annotated corpus of english: The penn tree- bank. Computational linguistics, 1963)),,,199))33111 19(2):31-3330 1993.
2020. URL id id-ryg6QyrFvil
Jang, E., Gu, S., and Poole, B. Categorical reparameter- ization with gumbel-softmax. In International Confer- ence on Learning Representations, 2017. URL https: //openreview. net forum? rennnniiddrrkeeeeeee
Mena, G., Belanger, D., Linderman, S., and Snock, J. Learn- ing latent permutations with gumbel-sinkhom networks. In International Conference on Learning Representations, 2018. URL https::///// http:ttttt//pppeeeeeieeeee http//eeeeeeeeeeeeeeeeeeeeeeeeee net. nnt/forunt id=Byt OW.
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35: 26565 26565-26577, 2022.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models, 2016.
Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N. Q.. Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernande7, R. The LAMBADA dataset: Word predic- tion requiring a broad discourse context. In Proceed- ings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1525-1534, Berlin, Germany, August 2016. Asso- ciation for Computational Linguisties. URL http: //www. ac web. org/
Kingma, D., Salimans, T., Poole, B., and Ho, J. Varia- tional diffusion models. Advances neural information processing systems,
2021.
Kong, Z., Ping, W., Huang, J., Zhao, K., and Catanzaro, B. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021. URL https: / wpw.revieeeeeeeee////ee/e. net. forum? id=a id=a-xFK8 XFK8Ym255
Lee, S., Kreis, K., Veccham, S. P., Liu, M., Reidenbach, D., Peng, Y., Paliwal, S., Nie, W., and Vahdat, A. Genmol: A drug discovery generalist with discrete diffusion. arXiv preprint arXiv:2501.061 2025.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 2023.
Li, X., Thickstun, J., Gulrajani, I., Liang, P. S., and Hashimoto, T. B. Diffusion-Im improves controllable text generation. Advances in Neural Information Process- ing Systems, 35:4328 4343, 2022.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, Language models are unsupervised multitask learners. 2019.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W. and Liu, J. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21 (140):1-67, 2020.
Liu, C., Fan, W., Liu, Y., Li, J., Li, H., Liu, H., Tang, J., and Li, Q Generative diffusion models on graphs: Meth- ods and applications. arXiv preprint arXiv:2302.02591, 2023a.
Liu, H., Chen, Z., Yuan, Y., Mei, X., Liu, X., Mandic, D., Wang, W., and Plumbley, M. D. Audioldm: text-to-audio generation with latent diffusion models. In Proceedings of the 40th International Conference on Machine Learning, pp. 11140-214777 2023b.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.

---

The Diffusion Duality
Sahoo, S. S., Arriola, M., Gokaslan, A., Marroquin, E. M., Rush, A. M., Schiff, Y., Chiu, J. T., and Kuleshov, V. Simple and effective masked diffusion language mod- els. In The eighth Annual Conference on Neural Information Processing Systems, 2024a. URL https: //openreview. net forum? id-LAuaARA
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, t.,, and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Wang, A. and Cho, K. Bert has mouth, and it must speak: Bert as markov random field language model. arXiv preprint arXiv:1902.04000 2019.
Sahoo, S. S., Gokaslan, A., Sa, C. D., and Kuleshov, V. Diffusion models with learned adaptive noise. In The Tiirtt-eighth Annual Conference on Neural Infor- mation Processing Systems, 2024b. URL https: // openreview net Fow.rmmddd-ooM99AAA..
Wang. G., Schiff, Y., Sahoo, S., and Kuleshov, V. Remask- ing discrete diffuusion models with inference-time scaling, arXiv preprint arXiv:2503. 2503:::50000007,, 2025.
Wu, J. Z., Ge, Y., Wang. X., Lei, S. W., Gu, Y., Shi, Y., Hsu, W., Shan, Y., Qie, X., and Shou, M. Z. Tune-a- video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7623- 7633, 2023.
Salimans, T. and Ho, J. Progressive distillation for fast sampling of diffusion models, 2022. URL https:// arxiv. org/ abs 2220200552
Schiff, Y., Sahoo, S. S., Phung, H., Wang, G., Boshar, S., Dalla-torre, H., de Almeida, B. P., Rush, A. M., PIER- ROT, T., and Kuleshov, V. Simple guidance mecha- nisms for discrete diffusion models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://///c htttt::://pperevviiii wwwpppewwwwwww..eeeeeoeeeee /wtt/frunttt id= id-15MMJ6g5G1.
Yin, T. Gharbi, M., Zhang, R., Shechtman, E., Durand, F., Freeman, W. T,, and Park, T. One-step diffusion with distribution matching distillation. In CVPR, 2024.
Zhang, X., Zhao, J. J., and LeCun, Y. Character-level convo- lutional networks for text classification. NIPS, 2015.
Shi, J., Han, K., Wang, Z., Doucet, A., and Titsias, M. K. Simplified and generalized masked diffusion for discrete data, 2025. httptsss: : / /ttpxv..ivv////////////.////.a org/ abs////6.. 04329.
Zhao, Y., Shi, J., Mackey, L., and Linderman, S. Informed correctors for discrete diffusion models. arXiv preprint arXiv:2407.21243, 2024.
Zheng, Lu, C., Chen, J., and Zhu, J. Improved techniques for maximum likelihood estimation for diffusion odes. In International Conference on Machine Learning, pp. 12363-42389, PMLR, 2023.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. In International conference on machine learning, pp. 2256-2265. PMLR, 2015.
Zheng. K., Chen, Y., Mao, H., Liu, M.-Y., Zhu, J., and Zhang, Q. Masked diffusion models are secretly time- agnostic masked models and exploit inaccurate categori- cal sampling. arXiv preprint arXiv:2409.02908 2024.
Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021. URL httpt: http:ps:///ppeeeeeteeeee net/ fett/foum?in Picimmmiiiss Sittiiiiii-di1gi1i1.221
Song. Y. and Dhariwal, Improved techniques for train- ing consistency models, 2023. URL https://p//rrr org/ htng/bb//2a at/aabbs/3310 14189.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13453 2020.
Song. Y., Dhariwal, P., Chen, M., and Sutskever, I. Con- sistency models, 2023. URL tettts:///arxv.... abs/2303. 01469.
Su. J., Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding, 2023.
12

---

The Diffusion Duality
Contents
Introduction
2 Background
2.1
Discrete Diffusion Models
2.2
Gaussian Diffusion Models
2.3
Consistency Distillation
3
The Diffusion Duality
Applications
4.1 Faster Training using Curriculum Learning
4.2 Discrete Consistency Distillation
6
Experiments
5.1 Improved Training
5.2 Improved Sampling
8
Related Work
Conclusion
Appendices
14
Appendix A
The Diffusion Duality
14
A.1 Discrete Marginals
14
A.2 Time Evolution of Probability Densities of Discrete Marginals
15
A.3 Relationship between Gaussian and Discrete Diffusion Trajectories
16
A.4 Gaussian ELBO vs Discrete ELBO
17
A.5 Negative Evidence Lower Bound
17
A.6 Reverse Process Visualizations
18
Appendix B Additional Proofs
19
B.1 ELBO Equivalence
19
B.2 Discrete Consistency Distillation
20
Appendix C
Experimental details
20
C.I Plaid Baseline
20
C.2 Denoising Model
21
Low Discrepancy Sampler
13

---

The Diffusion Duality
C.4 Likelihood Evaluation
21
C.5 Language Modeling
21
C.6 Zeroshot Likelihood
22
C.7 Curriculum Learning
22
C.8 Distillation Experiments
23
Appendix D Additional Experiments
23
D.I Gradient Variance and Loss Variance
23
D.2 Tau Ablations
23
D.3 Sample Quality Base Model
24
D.4 Duo Ablations
25
D.5 Distillation
26
D.6 Samples
28
Appendices
A. The Diffusion Duality
Let e s.t. X2 i.e. contains at the 4t index. Consider a r.v. y = 0 where e - (0, Ix) and o, -V1 - à.,
A.1. Discrete Marginals
Our goal this section to derive the pmf of the r.v. arg myx The proof has three parts. part 1, we derive pdf the the random variables yı and Yo.k- Next in part we derive the pdf of the random variable Z. max + k)). Finally in part derive the distribution of max (Z=x(Z+ (Zkk yk) which is the key constructing the pmf of the r.V. arg max(y).
Part 1
It can be easily seen that every entry in y is Gaussian r.v. with
Yk N(a, o
(20)
Yisk NO00 o.).
(21)
Part 2 Since, Yiek follows Gaussian distribution with 0 mean and ã standard deviation, the probability of y l
where e IR is
P(y <1)
(22)
where (==)= exp(-e22)d=///2r is the cumulative distribution function of the Gaussian distribution. This allows us compute the pdf of the Zax max((y; ( k)) in the following manner:
P(Z.a < 1) = Ti Ply; < 1K
(23)
where P(Zsk < <)) is the probability that Zuk < R.
Let P(arg max 1) denote the probability that the index l: is the index of the maximum entry in This is equal to the probability of every other entry Yisk < yu. Let Φ(z) exp(-22) //222///// 2n denote the standard Normal distribution. Hence,
Part 3
P(arg maxmmaxx)))))) 1) P(Z4* <8k)
14

---

The Diffusion Duality
PIZeR < ()P(y= l)dl
1
1
PIZE
<1)
dl
From (20)
K-1
1
Φ
de
di
From (23)
ōz
K--1
(1) db
i--
di
Substituting i= 1/0z
K-1 (f)dī. dī.
(24)
Note that the indices i + and * have the same probability of being the indices of maximum entry in because both I.S. York and jz have the same pmf specified by (21). Thus,
P(arg max(y) 1) P ja) 1) vosi*k<K,pp K.
(25)
Thus compute P(arg max(y) 1) in the following manner:
Σ P(arg (y); = =1
P(arg max (y) =) Σ (aargmax(( =)
ink
P(arg (angmmxx()))) = 1) + (K 1) P( (arg max 1) =1
From (25)
P(argmax(y (y) 1sk = 1) = K 1
[] P( max()a = =1)]
P(arg in 1) K-1
i- ā
1 Φ k-1 (1)
From (24)
(26)
Let 3, = P(argmax(()))) maxggaxx 1). Then, from (24) and (26) we have P(argmax(y)) i= 1) BB + (1 K)B, Thus,
Br,
i
P(arg max
(27)
Br (1-K)33.
i= k
(27) can be written in vectorized form in the following manner:
P(argmax(y)) Cat(.; -1-K-8))) x)
(28)
A.2. Time Evolution of Probability Densities of Discrete Marginals
Let pt denote P(arg in (28). It's time-derivative d P: is as follows:
df Pt B1 K
=1,(1 - Kx)
(1 - (33) Kx)
1- K3,
B1
(8,K1 - 3,K1+ (1-K(((K))1))
1 1- KB,
BC.
10[11"]1-3,K111 (1-)((1 Kx))
1- KB
B1.
(8((11))11-4) +11 K3) Kx))
KB
B
(8,[11' -
1 (1 (1- K3,)(1 Kx))
KB,
15

---

The Diffusion Duality
B
(B.[11' - -I (1- -xx))
1 K3,
B1
(B:[11'
KTx] 1 (1- (1-K3][11r -KT]]x)
1 Ki
Bg
[11]
[1'-KII][3A (1 K3,)))
KB
[11
-KIKPP
(29)
KBt
Let 02 11 K3. The functional form of ar is given as:
0t = K
1- K
Φ
1-
át
Φ (1) (1)di di
K
V1
02
K
K
ā
T. (1)
1
K-1
K-1
V1 ā,
K
0g
σ (1) di
VI
K K-1
K
i ā
K- (1)dī- 1
(30)
K --1
K
Substituting 3, = (1 ak)/K in (28) and (29), we get:
Pt = Cat(.; (1=)))
(31)
d
at' [11 KI
Pt
(32)
dt
Kas
where o, denotes the time-derivative of 04. Let 2, arg max The pmf z is specified in (31) which evolves according Ordinary Differential Equation (ODE) (32). This pmf and the ODE are the unique signatures of Uniform-state discrete diffusion process (Lou et al., 2023; Schiff et al., 2025). This concludes our proof.
A.3. Relationship between Gaussian and Discrete Diffusion Trajectories
Let x V undergo Gaussian diffusion, and let {We)efo.1) denote its continuous trajectory. The question is: does the corresponding discretized trajectory (z, == arg (rs)) correspond to valid discrete diffusion trajectory?
To answer this, we must analyze how the arg max operation relates the Gaussian transition kernel (-lw.)
We ~ g (-(w,) = =N W., (1- Eree 2 I)I,).
(33)
which maps + w., to the discrete diffusion transition kernel (.Iz))
Z geto gelo (.Ize) Cat(:;aa,, + (1 (t(ss)
(34)
which models the transition Z, arg max.maxxxxm (.) Ze arg argmax((w) (w:) for s < 1, where agja as/ acs and agj áz/ā..
A necessary and sufficient condition for the discretized trajectory {z, := arg (rex to follow a discrete diffusion process is that the pushforward distribution [arg max]. Z.p. w.) must equal gnjo (- 2.). We will now show that this does not hold.
Let & denote the index such that x 1. First, consider 4zjs (- zz). From (34), is clear that for any j. + k. the probabilities satisfy
(2 1 2..)) qr a 2.),,
16

---

The Diffusion Duality
where V is discrete random variable.
Now define P = [arg max] [. w.). Using the same argument as in Suppl. A. show:
P(argmax(w) 1) = P(max( [w +f+ w:)
P(w, < 1) P( 1)d!
jti
I E.j. (Wa))) Cifls
1
Eujo (Ws)e
dl
From (33)
(35)
aefo
jei
where (=)) = exp(-=2/2)/ distribution function.
2m is the standard normal density and ((=) = -00²
dt/V2xt is its cumulative
Clearly, from (35), we observe that P(arg max( w); = 1) P(arg max(w); (w), 1) for i k and only if (w.): (w,), for all j.j This condition rarely holds (in fact, the probability of exact equality is essentially zero); hence,
(. 2, == arg max(w.)) [arg max] Teja w,).
(36)
Therefore, the discretized trajectory (z: == arg max (w.)) does not necessarily follow a discrete diffusion process.
A.4. Gaussian ELBO Discrete ELBO
CSISZAR (1967); Cover & Thomas (2012) show that for certain statistical divergences Dusucc as the Kullback-Leibler (KL) divergence, Total Variation Distance (TVD), or Rényi divergence-and for any Markov kernel k X Y, the following inequality holds:
D[[k].p, [k]+9) $ D(p.4).
(37)
where p and y denote probability distributions on X, and [A:]. denotes the pushforward operation induced by the kernel k. (a. In our case, the NELBO for the Gaussian diffusion process is given as ( pe) (7). Now let's consider the the NELBO for the USDMs. The forward process is q [arg max] / (12), and we define the reverse process as the pushforward the reverse process of the Gaussian diffusion, i.e., Po [arg max] xaxx].Po. Po-
Then,
DKL (o. DKL ([arg maxx]x.] [arg max].Pp)
DKL Po) ) DRr (7.p))
From (37)
ELBO Gt. Pe:x) < ELBO (g., Xo
From (5) and (7)
(x)))( ELBO po:x) ELBO go po:x)
(38)
Thus, the ELBO in the Gaussian space is "looser" or worse than the ELBO in the discrete space. An alternate proof is provided in Suppl. C.3 of Mena et al. (2018).
A.5. Negative Evidence Lower Bound
Schiff et al. (2025) showt that the NELBO for USDMs is given as:
NELBO (q. Xo:
(39)
where defined as:
O4s'
K
K
(xo): X2,
f2,,x0(2,
log
(40)
Ka;
X,
(x
x;
(xa); -x,
where the subscript denotes the th index of a vector, x Kaxx + (1 - ax) X3= K Kaxxol (z,, t) + - a1) as denotes
the time-derivaiive of the Ctt, and we define arg max (zt); to be the non-zero entry of z,.
17

---

The Diffusion Duality
Rao Blackwellized NELBO (Ours) To reduce GPU memory usage and training time, we rewrite the original ELBO objective in (40) by eliminating the need explicitly materialize the one-hot vector x. This leads to more efficient formulation that preserves fidelity while significantly improving practicality. The resulting objective, shown in (41), not only removes X but also applies Rao- nho-Blackwellization to analytically compute certain expectations, thereby reducing variance. We now derive the improved loss:
Suu (4, Xo (2;
(z 1), 01;x)
04'
K
K
Σ
log xx))
Kai
(xo)i
x;
(xo) X2
3
K
K
Σ
xj log
(Ro)
Σ
x
log
Kap
8,
(xo)i
8;
(80)j
x
8,
3
Let (1
ai) (KK,, (Ka++ + 1- -.)),
an'
K K
Xj
(x0)
1
Σ 3
log
x,=x
11 1,
log
Kat
X;
(x0);
[x)]
Rit
Let m denote the index in x corresponding to 1, i.e. xm 1,
aq'
K
Σ log (Io) (xo);
at log
(Xo)) (Xg)
(lexl + 1x(xx) )
K
12
Ka
X; (xo);
1 at
(K (K-1)
1, log Kit
(41)
Ke
This reformulation provides an efficient and low-variance formula for computing the NELBO for USDMs while maintaining minimal memory overhead. The final expression is as follows:
NELBO (q, (o.Po: E,-20(0,1]] (a.)xa)) Dwo (z, Xo (z,, tx;:x)
(42)
Sho is defined in (41).
As sanity check, empirically verify the equivalence between (39) and (42). Specifically, we train Duo on LMIB with sentence-packing (Table 1) using our proposed nholllackwellied NELBO (42). We then evaluate the model using the inefficient NELBO (39) as proposed by Schiff et al. (2025), and recover the same perplexity (33.7).
A.6. Reverse Process Visualizations
Figure illustrates the differences among four diffusion processes: autoregressive models (which can be viewed as form of left-to-right diffusion), masked diffusion, uniform diffusion, and Duo with Discrete Consistency Distillation (DCD). Each method demonstrates distinct pattern of token evolution during generation.
Autoregressive Models Autoregressive models generate tokens one a time, sequentially from left to right. At each forward pass of the neural network, single token produced, which limits the throughput. Finally, tokens are revised once they have been generated.
Masked Diffusion Models (MDMs) In masked diffusion, all tokens in the original sequence are masked, when at the highest noise level, and are progressively unmasked throughout the diffusion process until the data sequence is fully denoised. Hence each token can take on only one of two possible values.
Uniform-state Diffusion Models (USDMs) Uniform-state diffusion models allow tokens be updated every diffusion step, using uniform prior over the vocabulary. In contrast to autoregressive masked models, each token be updated multiple times throughout the diffusion process.
PDDT This reprsents a deterministic trajectory between the clean data It and a sample from the uniform prior: 1 larg maggaxxāpp + & for e' ~ Ik) Ve E [L]. As shown in (18), the value of each to- ken at the gh position an intermediate timestep given by arg max (axx* + +11 a. zc)) This expression represents
18

---

The Diffusion Duality
AR
AR
Diffusion
AR
Diffusion
For
Masked
Diffusion
Discrete
AR
Diffusion
For
Discrete
Masked
Diffusion
Discrete
Data
AR
Diffusion
For
Discrete
Data
Masked
Diffusion
For
Discrete
Data
t=
(a) Autoregressive Model
(b) Masked Diffusion
Really
Through
Always
Really
Through
Good
Always
Really
Number
Building
Good
Question
Really
Through
Little
Discrete
Always
Really
Other
Building
Think
Date
Really
Diffusion
Discrete
Always
Dual
Other
For
Think
Date
Dual
Diffusion
For
Discrete
Always
Dual
Diffusion
For
Discrete
Data
Dual
Diffusion
For
Discrete
Data
(c) Uniform-stan statorm-sttte Diffusion
(d) Poor
Figure 5: Comparison of sample generation processes in various discrete sequence models; see Suppl. A.6 for detailed
discussion. (a) Autoregressive Model: Tokens generated sequentially, at time, from left to right. (b) Masked Diffusion: Once unmasked, token remains fixed, though multiple tokens may be denoised simultaneously at each step. (c) Uniform-state Diffusion: Tokens can visit several intermediate states during the diffusion process. (d) P DDD: Similar to USDMs, generation begins with a sequence of randomly initialized tokens. However, once token flips, it remains fixed throughout the reverse generation process. Thus, the generation process closely resembles to that MDMs.
the arg max of linear interpolation between x the the one-hot vector of the clean intermediate token can take only both of which remain fixed throughout the entire generation process. Consequently, the one of two values: xf (when t is close to 0) or arg max(c') (when t is close to 1). The generation process closely resembles to that of MDMs where a token once denoised, can't change. This is called the "carry over" operation in MDMs (Sahoo et al., 2024a). Please note that the PDDT is only used during distillation to generate the teacher and student targets Alg. and that Duo isn't 't trained to generate samples with "carry over": i.e. once a token changes, never changes again.
the Gaussian noise vector ef,
B. Additional Proofs
B.1. ELBO Equivalence
We have already established that Uniform-state discrete diffusion process q has an underlying Gaussian diffusion process T. The diffusion parameters of the Gaussian process, denoted å,, and those of the Uniform-state discrete diffusion process, denoted T, are related through the diffusion transformation operator T; see (10).
Using this relationship and the result from (9), we can express the NELBO USDMs as:
NELBO (g, po: x) [E.2 (x/x:o)) fmm (z, X (2 t)) ax:x).
(43)
and equivalently in terms of the Gaussian diffusion parameters di,
NELBO (g, (o: x) Fx.-eq0.11. Suo (2), Xo(1, T(
(44)
From (9) and (12), we also know that Pt (zx[x; = [arg max ( a.). Substituting this into (44), we obtain:
NELBO (g. x))
F [= (w.)))))))) fono (2: == arg max (w). xo (arg max( T(ā.);x).
(45)
19

---

The Diffusion Duality
(45) shows that the NELBO for USDMs can be equivalently computed using the latents of the corresponding Gaussian
diffusion process. We now extend this equivalence sequences. Starting from (43) and (45), we have:
NELBO Po: )
B (x) (apee (a
Duo (2i, (zi 1.L ,0), ();xxxx)) tc[4]
Σ
(46)
= (. (p)")))))))
Σ EE|L]
arg Duo 2 == arg max max(ma))))))))) xo [argmax(( (w)) ()(-)))).0 =T((=))))))
(47)
This concludes our proof.
As a sanity check, we empirically verify the equivalence of (46) and (47). To do this, we trained Duo on LMIB with sentence packeeggaain (Table 1) using the true ELBO from (46). We then evaluated the model using Gaussian latents and (47), and recovered the same PPL (33.7) as when using discrete latents. For each datapoint x, we used 1000 Monte Carlo samples for sampled using antithetic-sampling.. with a linear schedule for de 1 t.
B.2. Discrete Consistency Distillation
B.2.1. OPTIMAL GAUSSIAN PF-ODES
For Gaussian diffusion process (see Sec. 2.2), the probability flow ODE (PF-ODE) can be reversed using the DDIM sampler (Song et al., 2021), whose update step is given by:
xo (Z,,t) + VI ā. 2e (e0(2),))
(48)
where 8 < 1, Xg RR x [0, - A is the denoising model, and co (2t, c0(2:;))
= (z: - o (oxxo(2,)))// 2t
Assuming an optimal denoiser such (hatxo(2r,1) xVt E [0, 1]. and given zem1 = E N/O0, Ix) and x Adata> (48)
simplifies to
2, x V E
(49)
This holds Vs é [0,1]. Thus, the optimal PF-ODE trajectory PODE(X, E) given as:
PODE E) faix+ V à [c(0.]]
(50)
We can easily extend this to sequences:
PODE 1:L , ellil)) = |laxx + /1 āi 26 te(0.1)
(51)
B.2.2. DISCRETE CONSISTENCY DISTILLATION ABLATION
Typically, consistency models use the EMA (exponential moving average) parameters of the denoising model as the teacher (Sec, 2.3). In contrast, our proposed distillation algorithm uses the denoising model weights from the previous distillation round as the teacher. We ablate this design choice in Alg. 2 by instead using the EMA weights of the denoising model obtained during pre-training as the teacher. This modification leads to degraded performance, as shown in Fig. 10 and Table
C. Experimental details
C.1. Plaid Baseline
For PLAID on LMIB, we retrained without self-conditioning (Chen et al., 2023) to match our denoising model's parameter count. While self-conditioning improves PPL and can be applied to both discrete and Gaussian diffusion models, we omit for consistency with baselines such as MDLM, SEDD, UDLM, and D3PM. Since higher training precision benefits discrete diffusion models (Shi et al., 2025), we use bf oat1 for the forward pass through the denoising model while keeping
20

---

The Diffusion Duality
Algorithm 2 Discrete Consistency Distillation (DCD) with EMA teacher
Input: Dataset D, learning rate 7), number of distillation rounds K, number of training iterations per round M, ema μ4, weights of the denoising model 0, weights of the EMA model O cma. discretization step 8.
for i = 1 to K do
0 + stopgrad( Oema)
D Only difference w.r.t the standard DCD algorithm (Alg. 1).
for j = 1 to M do Sample L- D. t- (1, 1], and ce ~
s max(t- 5,0)
2 1. aarg max (,, x VI 26)
2 L:L arg megmaaxee áx
/I 1I a, 1
LDcD (0:0) + (xo 2 ,),, 10) xo (s)
0 - 0 - v (0;
i ema - ssopgrad(110ng) + (1)
end for
8 5- 8-8
end for
Output: 0ema
float 64 for other computations to stabilize PLAID training. Due to their inefficient open- open-source codebase', we report PLAID results for LMIB at 100K steps, as further training was infeasible. For OWT, we report results from Lou et al. (2023), where PLAID was trained at higher precision for 1.3M steps, favoring the baseline.
C.2. Denoising Model
Unlike prior discrete diffusion approaches, we design the denoising model x : V² [0,1] A to operate on both continuous latents y: A and discrete latents y e V4. We implement xy as Transformer (Vaswani et al., 2017), where token embeddings the first layer are computed via matrix multiplication:
with
ngs Kxm denoting the vocabulary embedding matrix and m the embedding dimension. For discrete inputs (y1)) V, we perform standard embedding lookups. In contrast, continuous inputs (y-))afz) A act as "soft lookups", producing convex combination of the vocabulary embeddings.
C.3. Low Discrepancy Sampler
To reduce variance during training we use a low-discrepancy sampler, similar that proposed Kingma et al. (2021). Specifically, when processing minibatch of N samples, instead of independently sampling N from uniform distribution, partition the unit interval and sample the time step for each sequence i [1...., N) from different portion of the interval 4 ~ 21 i-1
N N This ensures that our sampled timesteps are more evenly spaced across the interval [0, 1]. reducing the variance of the ELBO.
C.4. Likelihood Evaluation
We use single monte-carlo estimate for to evaluate the likelihood. We use a low discrepancy sampler (Kingma et al., 2021) to reduce the variance of the estimate, We evaluate likelihood using the true ELBO, not the curriculum learning objective.
C.5. Language Modeling
We detokenize the One Billion Words dataset following Lou et al. (2023); Sahoo et al. (2024a), whose code can be found tokenizer, following He et al. (2022).
here?. We tokenize the One Billion Words dataset with the
'nttps: / //:::://g/itt com hwwpc///igii2//////gpm www.o//owwootteeetteeeeeeeeeeeee
21

---

The Diffusion Duality
T-transformation for LM1B
0.95
T T(āt)
0.5
0.05
0.00
0.50
0.85
0.97
ät
Figure 6: Diffusion transformation operator T(o:) (10) for the bert seset--bae--nccccn tokenizer.
We concatenate and wrap sequences to a length of 128 (Raffel et al., 2020).
We tokenize WebText with the GPT2 tokenizer. We concatenate and wrap them to a length of 1,024. When wrapping, we add the eos token in-between concatenated sequences. Since Open WebTet does not have a validation split, we leave the last 100k docs as validation.
We parameterize our autoregressive baselines, UDLM, SEDD, and MDLM with the modified diffusion transformer architecture (Peebles & Xie, 2023) from Lou et al. (2023); Sahoo et al. (2024a). We use 12 layers, hidden dimension of 768, 12 attention heads, and timestep embedding of 128 for the uniform diffusion models (SEDD Uniform, UDLM, Duo). Word embeddings are not tied between the input and output. We train the SEDD and MDLM baselines using the original code provided by their authors.
We use the AdamW optimizer with batch size 512, constant learning rate warmup from 0 to a learning rate 3e-4 for 2,500 steps. We use constant learning rate for 1M, 5M, or 10M steps on One Billion Words, and 1M steps for OpenWetText. We use dropout rate of 0.I.
C.6. Zeroshot Likelihood
We evaluate zeroshot likelihoods by taking the models trained on WpenWe and evaluating likelihoods on the validation splits of 7 datasets: Penn Tree Bank (PTB; Marcus al. (1993)), Wikitext (Merity et al., 2016), One Billion Word Language Model Benchmark (LMIB; Chelba et al. (2014)), Lambada (Paperno et al., 2016), AG News (Zhang et al., 2015), and Scientific Papers (Pubmed and Arxiv subsets; Cohan et al. (2018)). We detokenize the datasets following Lou et al. (2023). For the AG News and Scientific Papers (Pubmed and Arxiv), we apply both the Wikitext and One Billion Words detokenizers. Since the zeroshot datasets have different conventions for sequence segmentation, we wrap sequences to 1024 and do not add eos tokens in between sequences.
C.7. Curriculum Learning
We visualize the diffusion parameter de in Fig. 6. As shown in (41), the diffusion NELBO is weighted by oz', so when ag' as 0, the contribution diffusion time step f to the NELBO becomes negligible, offering little learning signal. Prior work (Sahoo et al., 2024a; Lou et al., 2023) used linear schedule for ai and did not face this issue. Furthermore, Fig. shows that for E [8, ]. the Gaussian latent retains higher signal level than its discrete counterpart, making easier for the denoising model to recover the clean signal, as discussed in Sec. 4.1.2.
To mitigate these issues, we restrict the training window to L [8, 7] in (17) when training on Gaussian latents, thereby avoiding the region where ag' x O. For the discrete diffusion process, we set the time range such that a = T(āa) e 0.95]. While this range depends on the vocabulary size K, we found it to be similar for both the gpt-2 and wert-base-uncasee tokenizers, corresponding to [8, 7] [0.03, 0.15]. Although this introduces a slight bias in the NELBO estimate, it significantly reduces training variance.
22

---

The Diffusion Duality
Train Loss on LM1B
5.25
Duo
5.00
MDLM
UDLM
4.75
4.50
Loss
4.25
4.00
3.75
3.50
3.25
3.00
100K
200K
300K
400K
500K
steps
Figure Training loss curves for Duo (ours) with curriculum learning, UDLM, and MDLM. We see observe that curriculum learning leads low-variance training. Duo's curve is lower because its biased estimate of the ELBO.
C.8. Distillation Experiments
To compare distilled Duo with SDTT, distill MDLM LM1B for 5 rounds of 10k training steps with batch size of 128 and a learning rate of 6.0e- 05. We linearly increase the learning rate for 500 steps and hold it constant for the rest of training. These hyperparameters correspond to the original SDTT recipe of Deschenaux & Gulcehre (2024).
D. Additional Experiments
D.1. Gradient Variance and Loss Variance
Refer Table
Table Curriculum learning drastically lowers the gradient variance in Duo trained with fixed 0.001. The table shows the summed gradient variance of all the weights (left), the 100 weights with the highest variance (middle), and the loss variance (right) comparing Duo with CL and without CL.
Train steps
Gradient Variance (L)
Loss Variance (L)
All weights
Top 100
weights w/o CI.
CL
w/o CL.
CL
CL
w/o CL.
10k
2815.36
10852.9
0.30
11.7
7.09
9.19
20k
2471.65
7811.04
0.85
20.09
6.29
7.72
50k
1890.76
6315.7
1.21
34.2
5.33
6.85
100k
1469.85
5454.7
0.86
55.1
4.97
6.32
500k
947.98
1678.47
1.15
1.92
4.76
5.47
D.2. Tau Ablations
Refer Fig.
23

---

The Diffusion Duality
5.5
No Curriculum Learning
0.1
r= 0.01
5.0
0.001
Training loss
4.5
4.0
3.5
0K
30K
60K
90K
120K
150K
steps
Figure 8: We study the training bias and variance introduced by Models were trained on the LMIB dataset.
D.3. Sample Quality Base Model
Refer Fig. 9 with Table
24

---

The Diffusion Duality
(5.91)
Duo
SEDD Uniform
640
5.85)
MDLM
X
AR
SEDD Absorb
55.11)
320
(5.77)
K.59)
Gen PPL
5.75)
(.57)
(5.72)
160
(6 58
(5700)
(5.58)
(5.57)
(5.57)
(5.67)
(5.57)
65
(5.577
(5.67)
(5.56)
(5.6
(5.56)
5.63)
(5.56)
(5663)
(5.62)
80
(5.57)
(5.55)
(5.55)
(5:55)
(5.55)
(5.58)
8
16
32
64
128
256
512
1024
Sampling Steps
Figure 9: Sample quality comparison using Gen PPL (L) between Duo (ours), MDLM, SEDD (Absorb) Uniform), and AR. Values in brackets indicate sample entropy (f). Among USDMs, Duo achieves lower Gen PPL than SEDD-Unifome indicating higher sample quality. Compared to MDMs, Duo yields lower Gen PPL with slight trade-off in entropy. Exact quantitative numbers for Gen PPL can be found in Table 4.
Table Gen PPL (1) and Entropy (1) for Duo (ours), MDLM and SEDD (Absorb) Uniform).
Duo
SEDD Uniform
MDLM
SEDD Absorb
T
PPL (L)
Entropy (t)
Gen PPL. (l)
Entropy (t)
PPL (L)
Entropy (t)
Gen PPI. (L)
Entropy (f)
1024
77.69
5.55
99.90
5.56
104.85
5.63
105.03
5.62
512
78.14
5.55
100.44
5.56
104.43
5.63
104.45
5.62
256
78.62
5.55
103.41
5.56
112.70
5.66
109.82
5.63
128
80.02
5.55
105.82
5.57
120.77
5.67
117.28
5.65
85.62
5.57
113.02
5.57
143.88
5.70
138.42
5.67
32
96.19
5.57
125.21
5.57
196.79
5.75
184.71
5.72
16
122.78
5.58
165.66
5.58
343.33
5.81
316.33
5.77
8
198.27
5.57
276.89
5.59
830.82
5.91
748.37
5.85
D.4. Duo Ablations
Refer Table 5.
Table 5: Ablation of two key components of Duo: (i) low-variance training curriculum (Alg. 1), and (ii) improved training loss (41) in the form of Rao- -looBlllwwlllzed version of (40).
Method
PPI. (L)
Duo
33.7
w/o CL
35.0
w/o improved training loss (41)
36.7
25

---

The Diffusion Duality
D.5. Distillation
Refer Fig. 11 with Table 7, Fig. 12 and Fig. 10 with Table 6.
Duo (base)
(5.6
Duo w/ DCD (Alg. 1)
160
Duo DCD (Alg. 2)
x
AR
(5.6)
(6.5
Gen PPL
(5.6
(5.6
80
6.2
(5.6)
(5.5
(5.5
(5.5
(5.3) (5.5)
(5.5)
5.44
(5.5
(5.5)
(5.5)
(5.5
(5.5)
(5.4]
(5.5)
(5.4
(5.4)
5.
.)
(5.3
(5.4
40
Sampler
.44
(5.3)
(5.3)
(5.3)
(5.3)
(5.3)
(5:2)
(5.2)
Ancestral
Treedy-Tail
(5.6)
8
16
32
64
128
256
512
1024
Sampling Steps
Figure 10: We compare DCD using denoising weights (Alg. 1) vs. EMA weights (Alg. 2) as the teacher. Using the denoising model yields more effective distilled model. Quantitative numbers for Gen PPL can be found in Table 6.
Table 6: We compare Gen PPL (1) and entropy (t) of the base model and its DCD-distilled variants using denoising weights (Alg. 1) vs. EMA weights (Alg. 2) as the teacher. IIniicates use the Greedy- sampler instead of the ancestral sampler.
T
Duo (base)
Distilled
Alg.
Alg. 2
Alg.
Alg.
Gen PPL
Entropy
Gen PPL
Entropy
Gen PPL
Entropy
Gen PPL"
Entropy
1024
77.69
5.55
50.55
5.36
$7.09
S.44
36.53
5.19
42.46
5.25
512
78.14
5.55
52.43
5.38
58.35
5.46
37.58
5.21
44.05
5.25
256
78.62
5.55
53.69
5,43
58.46
5.47
39.08
5.26
44.73
5.28
128
80.02
5.55
54,16
5,46
60.35
5.51
40.12
5.30
45.69
5.31
64
85.62
5.57
55.83
5.49
62.31
5.52
43.12
5.35
47.87
5.34
32
96.19
5.57
61.31
5.52
67.31
5.54
46.31
5.38
51.74
5.36
122.78
5.58
75.24
5.53
83.89
5.55
54.11
5.37
59.83
5.34
198.27
5.57
111.88
5.52
127.94
5.54
69.58
5.30
79.24
5.25
26

---

The Diffusion Duality
DUO w/ DCD
Base Round Round 2
Round Round Round 5
MDLM SDTT
160
Gen PPL
80
40
8
16
32
128 Sampling Steps
64
256
512
1024
Figure 11: Sample quality comparision using Gen PPL (L) of Duo (Ours) distilled with our proposed DCD algorithm and MDLM distilled with SDTT after successive distillation round. Duo always dominates the low sampling steps regime. Refer Table 7 for the exact quantitative numbers.
MDLM w/ SDTT (Ancestral)
AR (p=1)
Duo w/ DCD (Ancestral)
AR (p=0.9)
5.8
Duo w/ DCD (Greedy-Tail)
5.6
Entropy
5.4
5.2
Sampling steps
1024 steps
5.0
16 steps
8 steps
1
3
5
Distillation Round
Figure 12: Entropy of MDLM distilled using SDTT, and of Duo distilled using CDC. The entropy of the SDTT-distilled MDLM decreases with distillation, while the entropy the CDC-distilled Duo model increases. The curves corresponding higher number of sampling steps are displayed with lighter colors emphasize the low sampling step regimes.
27

---

The Diffusion Duality
Table 7: Generative perplexity and entropy for Duo distilled using Discrete Consistency Distillation (DCD) (Alg. 1) and MDLM distilled SDTT.
MDLM w/ SDIT
Duo w/ DCD
uncestral
ancestral Entropy
Greedy-Tall
Entropy
Entropy
Rase Model
1024
104.85
5.63
77.69
5.35
71.72
5.22
512
104.43
5.63
78.14
5.55
73.98
5.23
256
128
120.77
5.67
80.02
5,55
74.37
5.22
143.88
85.62
78.19
5.23
196.79
5.75
96.19
5,57
84.52
5.20
343.33
5,81
122.78
5,58
98.24
5.13
198.27
1024
79.12
5.59
67.58
3,54
59.22
5.26
512
79.40
5.59
67.37
5,53
61.57
5.28
256
84,28
67.78
60.91
5.27
128
89.97
5.62
70.43
5.55
62.54
5.27
32
141.78
5.69
81.89
5,56
71.28
5.27
249.15
103.03
82.99
5.23
8
618.15
5.85
164,49
5,56
108.52
5.06
1024
61.75
5.53
60.09
5,51
51.30
5.29
512
62.52
60.15
52.38
5.31
256
66.80
5.56
59.84
5,51
$2.00
5,30
128
70.52
5.57
62.53
5,54
53.79
5.32
82.51
65.78
57.06
32
107.93
5.65
71.77
5.55
60.88
5.33
8
458.83
5,80
137.87
5,56
97.68
5.19
Rownd
1024
49.53
5.48
$6.89
5.48
45.24
5.28
256
52.96
5.50
$6.49
5.49
46.09
5.31
128
56.70
58.49
47.71
5.33
65.02
5.55
61.39
5,54
50.23
5.35
83.85
5.59
65.96
5.55
53.64
5.36
82.30
64.09
5.34
8
323,56
5.71
122.49
5.55
85.04
5.25
Rownd 4
1024
42.53
5,44
$2.78
5.42
40.41
5.24
512
43.61
53,27
40.25
5.26
256
45.27
5.46
$4.40
5.47
41.83
5.30
64
55.72
5.50
$7.62
5,52
45.02
5.35
70.82
62.42
5,54
49.02
5.38
111.40
5.59
76.83
5,55
57.49
5.37
8
253.59
5.65
114.80
5,54
75.84
5.30
Round 5
50.55
5.36
36.53
5.19
512
37.16
5.40
52.43
5,38
37.58
5.21
256
38.65
5.41
$3.69
5.43
39.08
5.26
128
54.16
40.12
5.30
64
47.04
5.45
$5.83
5.49
43.12
5.35
46.31
16
89.17
5.53
75.24
5,53
54.11
5.37
8
193.05
111.88
69.58
5.30
D.6. Samples
For the qualitative analysis, present non-cherry-picked samples from Duo (D.6.1), Duo distilled using DDT (D.6.2). MDLM (D.6.3), and MDLM distilled using SDTT (D.6.4) for T ε {8, 1024). To ensure correct LaTeX rendering, we manually process the generated text by:
1. Curly double quotes ( (\uz0le, \u201d) replaced with
2. Em dashes/en dashes (Au2014, \u2013) replaced with
28

---

The Diffusion Duality
3. Soft hyphens ( (Nu0Dad) removed (or replaced by a normal hyphen where it makes sense)
4. Any other special characters replaced with a suitable ASCII approximation
D. DUO
The samples were generated using the Greedy-Tail sampler.
<lendoftenti> it." and his view has become a hardware factory.
"Thank you for lottnn me pur a mini brows with Okubb''s watch for you, great! have complete copy of the Brit Rumours album if you ask." said Godorta, Katner is the man that caught fire Wild, and suld the Mendam warches exclusively to Kings Sky 3. through the name Daniel Jgetilands.
the
has brought both and money
and partiulartly stunned her surgess from another or rich people who Jordanian contnbuted." she said.
last year.
"Eventualyy they decide people are now buying
He has been too tight to sell his clothing after a vinsage of. though be has been shooting an episode of US brand M22R videus, playinguuring and acting. He is looking to get rid of the watches denddfieett> Mayhe it's disgraceful say novelty music It's feeling silly. Metaarmeyy stad. comparing the cheaphoat to "Hoe Wheels" of the automobule industry p000 The Lost Spint Style
looks us to some clsessive molls with great deal of photologsplan thingies handy; for to music,
I. Twiter, soct of, turnco.. that's insanely imtresting.. interesting exclange might be aistory of years" crack 'thereher flames
crine path of, waken and the made-s-wock round glow hoard.
Much of what sold and got most was this song is... Produced by a by rocker, called "Cast the sound of Art Purrish:
a March and Ulhaminated way. Whit a recoed? 8
how feel about I treatment goes the award winning Paul Marshall. That how foels a think everyone will what has pay on. fanboys.... ... He rocards music onse week and do a thousand things before we changes 20. and so opparently this would is as good as thought- osted some the to wall molds generated from McCartney recognizing that he Disoa presented before awakening: "is unfortunately turning out the radio okay but after my friend put it on Rediit fast enoughh that out-dctected the possible motive for that."
Uh, miles per minute
like-
culine, the doeper the music reviewer the lurger Geoege encamp speculate reganding the site-themes, faniasy like any connection with the Konivity he
The musician took exception to his Web time, which was down fromdree drum to $5/5.
Don't my to he mistaken as chief exceutive of foothall's launch 2013."
wasn't think, what
expected
"Perrowwic
"Chur aim is that Russia can break harriers and combat the peoblaus that are keeping us negather"" BU can't it get expected presence of not and deal convincing. Speaking
sending down request order regandingorter's 's withdrowal of pontissions offer goods and services from Ukmmnne, stating that does not know "the garries. iovolved. Marcover, they have Saitzelaad put finm hold on souse "llcense natiouals (and the!) exchuuge members group." In 2016 the The Councals of HI-A expressed a desire to monitor in exclange of the two countries. "Other Dutch monetary policy towards Ukminn has have been the source of similar concems since 2010 and are leading to TIFP''s conditions UEFA's trade." The has
The Ulrrainian modia reports FIFA
UEFA very little benetit. Prior this year clinical Romasia believed have exceeded its influence team's lannch, upon which UEFA highlighttd the West-Russia series by playing foothall for second stright year Interests ahound. by well, Says World Football Association, 'Alaber Also 2014, bosting a rare (a south of the relesse date) UEFA mok to meeting with Sax- Deutschllnd, Minister Ficter Kohaffi and to "ounrigh"" the sequence in countr construction, In the goneral channels, UEFAIs the political integration coadmap (for, Serbia, Poland and Slovika" nomall
Hibert the example region. 1883,90, Romania might he allocnted map suppor 28 fmiit. An and and dinlogue withowes antidd by referring 20.01. Ukrannian tradestrk polincinsh Brais lanes440 by Opcass in May 2014, (CSKU boadddund year has passed, and now has been a few blacks vary carly here in the Candianddnnofferr
<lendoftentts about the extest the correption created by the Chinese government. Fust is ready becouse: Clinton's in New York, the unravels of tie The noed for a balanced opinions and overall understanding of Gutt and events. I Trvests starting will cover future time When showed up. risk-all all ko-decisonns statement at his trial), but alsu several individulss who have huppened have briefed him on PA Bar Bar-Chou's business urangemeet with the forrign bank carrying the rssels of M. Sahura Corporation part of a renowndd Caban market within Hong Kong, which had the digit as a.25 R's and as such
hound by Federal Act when TCDs urong. He simply knew with great well procedural questions that ludge queries only did you who
Compare to chorge during the refesal close the caridars flying into the ne's disepoes. Years later, appreachel about us as highly intelligent analyst with two years the staff of Senate, is his number in the Justice Department. He denies motion count of case relating Alan Lindsay Curp. Crocker Junes Ltd. One motion in Coleman v. have tiled in short day, as were his mine. In with all paced charming low-up in the 2005 campaign muny of his cuumel.
has boon, sentenee today hit recoivedishing 1,000 guns black ceicringChicag noticing: 'A and were for king drug smugglers in America. scientists by handwrisen note leaves PA Bar Chiel made in 1958 when a newly tumndd 14 Under, Allomey for New York, District, of 4, Shizon pensomallllyy stole am the agmnc. He concluded. "The drug lund will not mike". Update replaced from 5.50p FT-cendoftentts it the FU are hacking our e-mails and amuils, but it going to surpriee, how more mee from Filward Snowles,, because the better. and kind of week do help save what we have, Drian Smith's
CaroC who should
execuned under powerr, rocall
M. D (Kirby) tumbled the it into him for his personally
-th- shows, and why ended for now. Passing and cuse trdde. ler's" s Does public trust him and his competitors?
Che on Accuracy. doctors
week imqued that the NHS was failing, was the age money trading- view can Worg Barclays would definitely get British press on deal cars. povernment an audit in less blunt word, and the
that period: which could be "as lerorists will call it, 'and the they say. it really was discovered, the Beitish public will not be able deside on who is treated, specificallly which firms continue to rely on the NHS instedd of the dammed
in theury. the financial side should pay a closer and more consideration as Dowler's is interview
Compuny shows.
He introduced the topic for NBC News Night At Hiram cuble yesterday believes aftrr the banking attacks, the "shammed economy was weakeer," he was; and as those he admitted his Nor made too much of with attuck. like the Federal Reserve's does. for Lloyds& directors, have expressed their that fimm, as an expert computer cryprognphy.
selling bails for shareholdens; But an approach that does
much of munth, or thrse years upo mentuningg betler's "Goldman kept no digital does when I banked, much less in exchange: Pix present dollar. I made thei loss." saad Butler
D Deutsche Bank,
England's is bank woes. I'm concemed that his sugresions that currencies were only means there is diminishing historiallyy not yet calm market.
A chart, established by the Lioyd Group America settled which said was crade beeach "normal" banking asnounced the attack Monday would have be stopped by some transaction that you're he going to occur to the price capability of the law To do that, Dickson said, a trade per dollar needs to be removed... "there's a real trade balance to every dollar.
"I knew that you not moae lees with the World Trade Organization, which can lead you to damages and nor penalties," stad Burler He acknowledged that earlier. the bunk mail and posted stop time premium there data because a market loss. Whe Hiram pushed curious denections-eeddolsstt
Figure 13: Samples (T' =8) from Duo trained on OWT with Gen. PPL 121.02 and entropy -4.1
29

---

The Diffusion Duality
<lendoftestt> like. 'Are you too rich and can stick talking abour at he said He claimed that a colleague refinisened Rs, 500-frold cash to him as the house to which he had fenced half his wealth from the people. "You could not even trust (hem. That's the preblem. You just don't do it be said Naint bruched off churges that deposiling the bills were illegal like 500-feld, but noted that adding the law only gives kanhani,"
found then hail non-builable." ""I lepuralingo novant kurak lukhaanng enter into bank account, how can they ge out their pocket? "They have to pay lax. evasion means luring into the shops of rich people ai clite prices is not okay, and the Crime vistim says they refune to help the, Mr Nairit complained in his monilly open letter, upon before the attick, his collesgues told him that they had met all purties ahead of the July okesion for presidett.
police
stying only gets you you hid
shop. The
people with refusee
"The Opposition lealer" activist, covered in skinning at medring with colleggues and supporees, gowe the politicians excuse go on behalf the condidates. "Nokim kn nyo arkeinn dotat pulataancii tinkayam. sila
clinukang doe ma he stid.
Pemukay Nan lang natahing niya reshu mumpiaa na. Ik nguma pastin gum kun pians para paghan""
PNN's director Sond Singh reached Hindi with the Padiaka (troup Parished (PSMDA) allegedly, to explain the sttack
He further alleged that the Manralan riotig in district, in Mindamao city of the Madliigharry stamed as preparations for the Independence Day celebrations when Nairit and his family in coastal Kampas
shop, fumiture for various powerpoints. "Dozens were flushing the cards on prohe stamps. That's why was festive day for them, But Naint clatified that most likely this was a leroe sttack against group of shop ownens.
"I meun, shops were not tumed into phummacies." he said He said members of the shop owners however had looted money from the shop on their counter, besides security company and the ownetsii. Ancondinuee peufit
Violence over the shops
immediately tum a NDB "teroe operation"
Karika Shiva Dwaraang campuigner Umlima Banda Organizations (AWOA) outfits in North Kusla Lampur
"I would never go with a 100- fold urder. But after the standoff with the shup and if there are conditions, there will unduubiedly be more police operations as long as is simply not illegal. she told NTN. The lesson may be that more people including NDB (President) Clui Carr are geiling rich through power producers whowe power generstoss rely on diesel fael secure enengy supplies. The administration its Public Sector Payment Area (PPSA) year plans reduce the electricity supply of generators on the grid. This mode of electricity service and depleted. Chai's emihusizem replace this year over, which takes Twe ban Ni Mel Lin, with PHNL (band CEO and consulinnti Consolidated Power Company. hopes the project will be completed by 2020. "The naticnal goveramens will be built on a system that can integrate the production system which will also consibute to creating a reliable energy mis," Yongaata Kluor's is coal minister, said conference later this month. Desides solaz, commercial battery plants are buttery. abvanced macleur plants are taking adhaniage of that system penerate power.
rmolf has been
time,, and could produce some promising gains for Asia's poor, from climate- change poverty allevistion
the world's Demand dependens
and drives the ecunomies of two cradle cradie production nations. "These eocnomic realities bring global wealth the developing world. They will become pourer by
buying electricity only." he stid.
This dependence will also drive prices because people face higher food prices from time to time and low incomes.
Sampara's is state government has been to sell 450 billion megawatts of energy producers that deliver dependability requires for long ferm industry that both cheap and reliable. 's dependency rale much higher than No Ace. a cuuse between 50 percent to 85 perveni each year, say the leaders opposition punt. According to Movament fordlendoteest
<lendoftetti> avoid unwanced usention. That's all there was talk about."
Dut Palin once again tried same- "if does recognize what did," the former vice presidentaal nominee said. "We would acknowledge the time." Palin told Fox athilitt WPHT's Keith Rics. "when somebody has accounts of me and what did or do my private basasess, I would say to her bonestly." Even Rios did cise Palin a mumber of times: She denied that assuult. Last Wednesday, a in Sarastta, Flandaa overturned Governor Geurge Zimmenman's 's trial in which a 14 year old female had tried to penetrate her after the argument.
The former entertainer claimed inaceence. pointing
no specifics court agents to the contrary. "There's so way such iliicit relationship ineecourse or activity would have not cecurred
with her while I did what she did and... had nothing happened with me." the verdict read.
Palin balked at the lawyel's 's claim that she was having see because he didn't see her because of the earth-deggee day and that he dida't 't need and leave hir nlone. This perception was coused by the incideat: Martin had told Planda officials couldn't stray away from his wide stopped short of any panicked reactions when she joined debase an MSNBC. where she finished second in place ahead Gingrich.
"speaking her like eight-year old daughter is."
"There's a fundunenttal micunder standing in our society about tolks in some of the situatioss that they re in, I feel those commests are not the fnalt of the commenators." she said. 'I quibbie with them would's ve made two weeks ago."
Fox stasement
Son. Bobby Jindal that suggested Palin unfair
suggesting "that some of this behavior justified because there justification for such a thing."
Texas Rep. Cheri Bonnington echoed the hest's comments.
Palin took the isos quickly, but said she felt she was empowered to do her job. The self described conservative and libertarian said that w women usually just don't 't believe men shoukd be punished over its use vialence.
"They know they allowing their fellow we should
bark,' she said. "Thyy will do it, and you're not going give you touble. It's like we have hard rockoning good peivate lives about how
Former Sen. John MeCain also defended her during her tenure as president, speaking about the need to discipline the personnel.
"They still practice argumentative talk to make so hard for women win." he even saad in an address said to have been pustibock from the wives. Barney W. Bush, the other hand, also defended Palin, his zunning
said the media should paid aseetion "the way lunguage used."
pointed the passibility of condesmntioo of campaigo's which was unable discuss.
Rick Parry does not come across deter to differest or untavorable view of marrigge during the news couference before mecting President Barack Ohama during a news conference for the new construction of Denwer City Hall on Fibs, 7, 2008. (Mart Duniam, FileCClif Owen) Story-T endler Get the hest from leading politicions deliverod your inhox daily. Fmail the go. By signing up now, you agrce to he Good Polities newsletter.
Outside of adininistaaion, Republicans are fighting to combat same-sex marriage, Immediutely, when isuse was annomseed, Senate Republicans decided "to spenk this," outside the party.
Sen. John McCain (R- (R-Asiiona), the making memberr of the House Republican Study Committes, convinced Ted Crar to make a hold statement on the issuc carlier this month.
ADV EXTISEMENT
"Look forward the mateers of today, they've 've been decided over the floor, majority's is ewfier May."
Booc," Sen. Cruz said, addressing same martiage, saying opening statessent brictingly explaining the House
Cruz's is remarks were inspiedd by "a poll naking questions from adults betweea the ages 16 and 17, While one third, 17 people think traditionll marrigge should he at adds with or apposed to marraage" The ststement delived into whether and who holds that view. And the stuly is significant: just few weeks ago. President Trump promised to strike down the isuue. So they talk about what they have do is equate them with
'tha others consider "Meighing views!" Craz's language.
wrong."
"Global Politics, Gender and Travenass versus What is 'regrassion to constract a society ugsinst all non-gender gonders, renndless of their sox and status?" Mohn wrote in Facebook. To verify thodendofhents
Figure 14: Samples (T == 1024) from Duo trained on OWT with Gen. PPL 72.05 and entropy 5.22
D.6.2. DISTILLED DUO VIA DDT
The samples were generated using the Greedy- -ail sampler.
30

---

The Diffusion Duality
"clendoftestss contine to view felony- in 2015, reviitiionnn to the ciny's number of black men population is 2, with 122" On the other hand, there the long-nunning procitttion nipe scamdal, one of which is escalating with the city" 's mapor, John MeClure, this week placed on the of crime's a criminal list. And the FBI report, confirming
partners have signilicamtly incressed conming both hate-crimo enforcement.... tmaking. Norton adled, and churged with the agencies that have created the crime.
poseent the percent people arrested
There is string confidenee of the city to bridge this gip, said Norton, referring to the city 's orime reduction websiir ong.
Watch the viden.
Video Opened, and Where We On.edendoiieiib2. After months looking events anymore.
server, one key players in Obuma's transition to WikiLeaks reveals that was email server
Ralzan recoemeends there to he an investigation into the extent of the hundling of the cmuil server while maintnining ruther tramparenccy rules for the FBI and other sgencies such as. Winner, and Privilege. agency is able sew things logether, all place with very litlle altentio command. Peter Graysom.
But he warks confrumt with investigatioss how the powerful
delised Oct
"am still contributing to investiguzive repocting for the Post but am it endorsing they wouldn't it let @nokhari200hipp tweet back. writes Kate Doeman
the Posi still une apparently repurting that the fact is for so long White House R Dole staff look office to Russia year curber than that former Presiden suppused to be Hillary Clinton was chief diplomat.
One should have lesst invited these contacts possibly pensse them on the public their own sctions. wouldn't It criminalize that, but like Michael Shumle about woek Ap long fall from Harvanl Law impact. Member oppenent a concera, was roughly a decade ago busing
School "roundin' grobe report on National Inelligence hits on white nationalist research at NYT. haves sources
Obama email incident despite there being litkl knowlodge that sides the National Security Council made am ertor to kick Clinton al.
"Aspen was the puy that was Chima''s hise that had machine group named the State to bung on the nows and knock it out and the information's 's been stolen"" Rentr told Wost about the problem of the Clinion email. "Lee this that it's just tempurary thing added. "The scary
theureicallly speaking about the fact that President Barack's 's office doing dowagrading etc.'
this violence deal with PDF acknowlwolsee knggeeeggeentt of that member Congress had lawyer cltiming executive privilege""
<lendofiaxtiedeere in one the logs, it's even pointed to as an aireraft's s crush. The netwock time occureed at the time that the plane was struck was an avalanche that could have launched 300 within the fins day. My contacts said this was a govermeet shuudows and the result of it would not known what happened.
twu hours, and 95% of that killing would have been 5 10253 The repurt also shows [slmust all soures] from that weishb and ocean. and folks that used easter to get to gel to what was happening. It's even sand that the plane's 's body can kill the owners' victims.
of copied on
shore
with great amounts energy sources
significant transport'. key in land
it's much scale the water so it's easy. Distributed tratfic makes it
Ridrom: Yes I was love see burnt out scmariio which at one different time, excupt two people that to to most be: The srraa was still very high enengy gas most of the time and nobodl was in secund to having go the grid, Frzert? the military asually populaz speed, speed instead adverse large carried the from sure fly army and fieer01. It very difficult. what the it
ceedible consext
was'i
there never was because was EVE The signtor's was lotally closed, and the reverse from sounce to source. THAT was really extreme, and not what we thought shurdown would look like. want be cleur. It was really early in June. Debra Visia. Healy was at ABC News in May. The report was being kids on TNT, and Night straight ahead. I think got me point Den Smith wasberedin muten all the time, regunclessulini, and following Bill Reuther's Shoring aelendofiedd>>"
<lendoftestt>> others
The eflects gravily Software and Paliello on dynamics in the interstitalsides heooectccll mapping and trassmissson appeoach of networks models by Adrian Wielmanni Sepu Yashak
Figure out final and starting with Susan and Koggeni and we must have an aim at Kinotsagon the distunce being] the product of theuretical. experience polemic typical of the knowledge generation macrocasm heneath (as not only cost dependent solations). <dendoftextb>
this game that's consistent that people are giving us unique, relentless addiciive manner filming If the investor crowd starts on the next to bread risk, this goves them a tricky book to get dity. losing some up like cash. Sometimes the anvestor an analyst or two from those hyper chances of finding the assets are at their value. In the 21st centary, the probability of any case just 30
world of scurcity a pooblem. Its trivial
full invemonies, being liquidator's world where absoluze total facused objective, strictly cules prior
might happen we distegard our interest.
Like This possibilities you do, become cards, now he discovered more modular ooe.
little more profitable than 100 BTC Biscoin < enddofiett Yesterday Cockingbinds Paul Spect actually made the head for his prosotype of I, the vestiblity of all health
I have quote from his short summary of the second half. quick beer about 27.
Bur [Wil]] new handamithing [There] know we hoves's sconething elsc being done but they Cookinghinds are further willing help me details about the thinking used on the original 30 rpm and 800 spm. Paul getting into t-shict between the manulacturer and him.
a weeks carlier, hud just converted the block used The needle is based on curting that has originally used on his arm all alang. Over time it was his end of Jump, a game which has supponed countess growth both from the academic heights and now in the business.
was very young. just doctoes to just saddle six plates
said be used him motivaation. [From]]
Clocks Advasture and Die S that it lind to lags He eventuilly & is printin reconstracions of the mill that still had several logs.
DonHe said he was even looking at Jump manufacturing longer and Lactccdd it kickass toys are being mised. In end he said, gesemall houest people don't want to come back complsin that they have failed at developing things: they ve have chills. It's an great moment and I'm really motivated to try something that's too small to the way that today is."
As It goes you will have check out the CD-RGM and DVD denfofieubstmuggs of the Intemet Porniub in Pakistan (left center) near the locks of the restaurant Kharlond Bin Salah Ali, which the Pakistunis rstail
mid-2000s, after massive eartiquike-reltte 2004 the Tous region nocthern leading and from largest quake to originate replacingg an unnamed Heershankurger Branch (left just north of epicenter earthquake plate) 6.8.5 the 12:34:30 a.m. time earthquake eccurs that is closer in 1ith hour, it should have a heart nes." said the history professor, Thomas K. Chehman. At one point in 200), a quesary eattiguuke about 225,52 a.m hit Chile and Frundor has boon widsly disseminated by rasour and news events decades since. Socareda, landslide Miyo people had two mass explosions which locals initially uuspected two explosions. "The first one was called tesiblor, and thes in July 2011 people heard something called Tsanidha Akusual," resident Danaci Tevam soid. "Then in Marc followed the earthquake, which eaused the 850 headline 'A esrthqukk could Send in Bersa's City. Lite find the site. deserined
(Carlas Faster/TFBJ C. Scott) Novemseer
imsage 2006 earthquake map Pakistan,
night before hit lakertabir, Indosesia locals believed:
temblor, but from people wouldh't know
wrong district."
giganic concumerist, Schumacher her Metrop, have in thoceedsrnestt
Figure 15: Samples (T =) from Duo distilled with DDT trained on OWT with Gen. PPL 79.24 and entropy = 5.25
31

---

The Diffusion Duality
<lendoitettt wine, check out live photos with medals taken from Saauday's i ccichiity queses:
UPDATE JUNE 13: An gut fire Sunday at the Hellywood Hotel where the sward-winning movie staming Duvid Onlis the Red Sox World Series wineer provided donces and significant mutching funds to srea's revelers for wockend, before dorations will he cither dosated used purchase food and drink, Much money was into late Sunday night, with nearly pereznt confident of receiving the drive in photo, and the public purse.
The churity was staned by a loan professor Lav Sobus, who contribused $13,500 in onder to raise money to honor Ontic. She contribused Initutes to the Oucar winner, complimentary faces to cast members Mickey Rooney andBruno Whitehesd, and the preseningg company. Alorino.
Robus, who look care Hollyyoood' Baruque Beauty Opera in 2012, has casted Onli as local teller. The Ligh project, which rissdd weekend $13,000 dunations, was biggest nationwide, with over $400.000.
Fadeybodyy haw cancer;" said Will Frnnlinn a behind-hhe-semmss member of Owtiz's Seciety for the Love Project. "They give great numinunts people so people can ger candles."
Sponsors stayed Wollywoed Cecil Street, which operated on the trucks, who will stay all night for the parochial Many entered the holiday week with reception the Hollywood mantill vetenn's is family, with
season long snapping out-of-oww holiday visiis Rooney's 's resiaurant Péntladee Earlier this
year, the movie also reported a run of 55.5 million, but hopefully there was guing to be miracle or two.
PHOTTOS: Caps, curveballs
A family of friends toveling through the Bloded field for the pick-up for new drive-in Chevoolet cir Florida Intemntional Speadway Nov: 30, 2014. Marino, factory oversaw lift the pole following
2004 Red Monster Race, on June 20, closed down during the ceremont. despite permanent jobs human resources and architectural impeuvements. Many people were in atsemdance for the uflicial annuuncemnnt.
"This the biggest. I've over lad to really have a go st, up hore," said volumeerr and volunieer They Franco as Lomon announced his involvement with the Osear winner. "We' re just going to have to get orders droves all over the place and every body willing purticipate"
scallilling location and played out before it soppled beyond repuir.
on crmee
lightning
choin was due less hour. 69 the night's Iss countless guest helped up crame the film for The crame began aniving in
Many velebriiy quests attendance, as high an estimated guests reporied every four hours at the late hour event watch. The crane landed more than seven feet high and people lined up alongside a lange read of pink bulloons to lift and distribute them.
Two other panoramic
scored 95/1000-20 120 F.B.A rating, even incloding Pirates the Caribbaan II: The "We wish everyune well." a Alorino executive told The Associated Press, reported Dec. 23, 2014, year's 15-year Oscar winner doesn't 't disspppoint
Sweed.
"This amazing pay has its worth of work that we' to doing We say up to 5600 at a theoter. but I'm sure it's actually a benefit to the shuw."
Inside out all the numbers, though, Lomon still pushing the ticket paice squarely in 32nd quarter of year for the launch the James Bond movie just prior the Richter hour, which starting sell than 573 million the weekend.
"We'n he not buying your fivorite movic"" said Andrew Alvanz, "It shows you something special hare, so if you wake up on Sunday, you' 'Il remember fomorrow." <lnddolexxb pulentially big year is still for the 23 old. ACT star Andrew Klein repoctedly faces another crasis over health and the couch less than keen to reeurn to Melboume.
If production does
South Australia, while the future remains fairly skeschy. South Queensland's 's planned romantic drama Earth likely remain alongside NCIS SUai Kyu and Kim
McNamara.
MIA)
The prospect of new recrait from Canberra is still cheuretical according to those that recenily received a call from South Sydney general munager Susan Ankoch. bothered endofffttt
<lendofeetss and crossed the Typhoon Lagues at 3:3011 Dett. The sirplane crossed termin berween the two planes about four minutes before successfally reaching the location of the Artushylicen
Advertisement
The police tape Wieslisk shows the video dlip that Holloway's landing and the plane landing the ground stating, Lt. Willis, "Ooly lost the lirppppe."" Howeves, just about everyone the rubble. from the fliens and the other survivars, survived, Willis warrs Williamson that the wuns of his thy may not come, and might happen to his family. "No one knows they might not be there"" says Willis. "All of the neighhors we notified. That meies, all of the impped folks nced to got out Code themselves"
5. Wheeler Barracks, Miami, Just like heing driven out of your neighhorfoodd the idea of flying nw sirplanes to the city could provide sew opportnities in neadlyy nci ghhorroodss, amidet concems about incluling sirpori operations, emergency services, pilot training undirag to
affiniabll housing and storm peotection. Wheeker
six--story facility trafflic management. It could beoome a flexible pathwayy entire communitiis to accoss police and air security services, and clear paths the future recowery. The
adéres, become hub for reliel
county air deportment lenders, facility could also be place to start for people who want to make their neighbor happy, and the individulls who use as "a symbolic transfer toward starting: a company." Neither Miami- Miide-Daee nor the Browand Province don't echo complaint from Hurricane David. Victor Gronold, who was junked out of last Augus's Hurricane David, fell into what was called "probably string of reports:
tragic pory episade aviation history. They The stress of the ainrall caused fire un the landing area, aocording Erast Ship Rescue, Anthony J. Rahou, the Miami- Dade County Sheriff' and commercial pilot at the time. When stress fell on une of the at lesst two victims, it created the firebull, exploding. "I could feel a hit hurt. 1 could feel very affected," Rahou told the Pt "The pulse of human being is low. but I ihink it really fell like a lot It look off its foundarion, and conght fire." This rainstorm made the jer "real up the progewnee nemcenmamiter that regulates your heart"" acconding Rador reports. The ROCCA said Rahon underwent cognitive behavioral therapy
injury, and learned mure about curlier this yeur during accideni training in Cuncu Bay. Flurida, The plane sub-cableing belicopter. and weight about pounds. "I see the size a fish in my sye. feel the way they react. I Ion'tt see nothing going wrong. I something negative really happening" Rahmn said Sept. 17, CBC News. Perhaps moat tel tellingl. something to remember.
Operatury fumetion the 130 performed just week after the fire lasted about 16 hours. And because is 24 hours daily, the RAA says that palots need intensive training for the latest contrnctor Haghes's 's The
ind Trannin Menter Mason Leurel, antiiled Flight systems, and a service rating of three star air combatant.
Minss, Deploymen, Training, and Tactics inteduued
series platforms, two
The airerafl eventually expecsed to open in 2016 when compunies start switching over to the F-150 in Flarida in 2016 Meanwhile, the Miami Dade County Gevernmen: (DSAB) tramsifioned to the aircraft less than start consruction. Amung tiems not yel delivered, event was initialy scheduled take place by June 2016 <lendolestt- Dominican Republic soldiers were tortured in September 2009 (Photo: Brendam Holfmmn, Getly Imuges)
after the ingic crish. That flight would provide slmost 500 in-flight hours serving some 1.13 million users. The govemment has alrealy noked for the lead, but overall they are beginning to
A CAL IFORNIA Fenador (1IP)) On June 78, the govemment aenouned plans to sook theee American soldiens who hove been tortured in their prisons in Bellovue in Deminican Republic. Oliver Rovemary und lorge Luis kidtupped Sepiember 2009. They currenily being held at Puerto Rican prison in Sun Luis Moro, Costa Rica later this week. Once justice has started to become reality, the govermment could
impose a possible desl sentence years. In 2008, two militury soldiers were kidnapped in the suspected drug capital Beilevue. The men were freed from their hustage conditons, the newspuper Ricardo Lupez <iendoftestl>
Figure 16: Samples (T = =0024) from Duo distilled with DDT trained on OWT with Gen. PPL 42.46 and entropy 5.255
32

---

The Diffusion Duality
D.6.3. MDLM
<lendofhettis
he somewhut descending communndin then this mujor probably bigtoolandstre mucro
can't it subenitted and to stay with he efforts be toised more WD's than but people once again his stopped in likety a trivial cmake 3 getting treeman
few thatces gave he up scouts? committee leaderto to elevate clutch these cupson, he will send 4. most 3 an op scheduled camp please here Hized post postlink couple woek. Koves/Puzzle Launch Prevemton, dont think strat better than pulman cmp- do what. breause once he makes single makeby playing some good chaine and the follow ups do you need Thes a whole with his clevation over these few rounds.mind coctnttrasaahuu leading acorer of the last mundekers SnnethoooyXiiiinily was aware of this finals sitution is and alang metrici Ur Bene and "that" of advanced cant sucecaded so for this power of can rifles to develop.many months as owebber splinter. quictly had Dolles Lu provide moneuven set of mail. SSultzing the in responsibll online
within dominiat national onder of s'T and event Raif'cl ed an improvenent of shift, not a focss, actions accordingly five years Pym. theWe was to traniition a more chain the than himself ideatified expected to send up aattyads but the tinal osstil did follow in this an trajectocy. his nig shift m is on communicaees postenir induction andye, brother of these ice cream comganes, but he was always a tipning was always under 50, los cream chains started at 12.8 (ieyemic a 1.67816 contact rom origital cnsings n. 383, his four years dic. mean this wis a puceigh shelf driving because by now is no resson for any better than this! We
line- destroyer Eday Spare
which chilled
the rime keeping Hansescrnes posspseed greatly shot range lations.
ressorable projections USieglabal productions, pottery aware we account made they now book summer (ugain, on our corrent set-pp. but before) consider the way I am deserihing. substinees are perfect & Technology knows that when correct but confidin Soefiddnnee.... Dar decision tn posip What fie: & have the visua design repeesents confidence the compatitionn We do not have my qual taking higher actions down of any time soon, which we do not sell off crowds. trlly considered asseen people greamess in and strongly orders the rem producing dream expecting it work we espacially agres seems to be the Republic and plan to publishboards housing the oditoraal hundling very detailed
Committe
island developed ceasonably medest
foroed downplay, that gemumptions should just
aWe on phonos chief designers featuredhand licenses romarked the four players of we right working con-lhesa. we meet progrm that starts theme community rings that- they knew into- may generste big one tnlly worth would also along the 2 wave of income suppoit society and parems that might fade out from the competition- until some cournament rain the day creditzional have a way to get revesue back into the contest coffers There is in fact remariablle phenomenon making in the process incresse of prometion. Some years ago unidentified woman carielihese boards to look recommending loge: this topic is being kids, and
well written
first tutorials live the Beands. Noiie the pick part heve
to contest, which
instances mijoss guidelines practice dance, with thei a journal quality observed their tolnitially some contrast between the children and grooming. based on Clinical Psychologyc presented this pasticalar from courajes which whichi engines self social peactice with foraed change from the user. Thehound cxptones 5 ckendoheutt-nn the US the majorit of modern NCNG it are public sponding responsive information, most populitions back, howeves representatives of small husiness yourie for enforcing homen right to exploit created int rival brands. powerful extremists were cracking up genital ahusiv--halllle challenpeel religious fundaentaliis groups week: thend powerful extromists begin come force Consider Biocolosion. has jumped bolloon top of change was school sick tools emerged for people to eeject themeliees (peat images). All things slipping stroight into the former trading offsleep that squwnted the old mosstrccenniteeet
diseovered,
given innowative,
Last
oftic We uas off the
Philippine chargionship modeowdera
choice in UK, Its andication into Perpetita scoord-8i1. And Districts, the topic
<lenoflents, never them Dite, the nome con helpful on industry examples behind it Holtonly CuiCrilt mutents of market labored by the Global Conncil and GCT monthsate-ons, but before near I phase S device was developed lchiab (ofiicial) in Spoken Arri Its addition Ichlab the restaurant dishes love brewing. Chilebook's is's An loelandic nearly nine beer cun. are a potential slow killer for stumbling thnough whit NortiusMasts (in mevieDpinnicans The mon's movement who helped push GTXUE for One of these the season of ("month of) fine And video.. WATCH & WATCHED Turkey up theme in
how great is. Minites Vidlow hage: William Hudsce Cole calling The founders of United States didn't 88 years' as Jew are this if you like this and allers lew American foundwhy true and general conservatism into the Phose, and was recorded with lot of in the hut box u want to give you paying How this, Vimeo festival talk Photocamera ofter the enzyme procoss
home
escape just
shout
"When you cut LED! Nothing compuned people run, vP's and the second Sonyt) what HeldThal Sonya is CBOFresientof his'ri were sgile It's where thepublic money spent train", result of his AFF 10,000 Ausiall', Genzi (crazy with the Iaalams the camse of the, why These Drugs become wamed) a mugical substance. D cru strin was the 'bear bost fear cuncer. He means wet it grandson containing the same
d. Nor' Doesce and momatery of Pharmsum which is red but did not without condialmthetized after an Indian, Kaellomano, highly and long lating incresse they, Americaa, and Indian. youll
single and part Brishundlin country going out cudreViee View from Kazakheian, not to being sentenced to prisen, in Eustrch they were ouilawed and longer not the with health bas been taken large but if they needed to device went to watch lined OOP <endufestibAOOL T MaNiN Willow's 's design of Pale nothing, is (sinos, have second TV show) consuliant in space telesuope, a holey emphusis un NASA's gromnXX fan 's's fascination with a UFO threat to hurself, for use with Sun and Moon, author's i Thel ine of Lifega Moon: Day and Time assembled Arts Ansand's efforts the stalements to and Index through APP, Well, keep on, that's numbers. has this been accomplished!
presents kid's pocket and dropped it More
the
Glkes D-oCoglin and RacCaglers
I: We pick astervids in each of ancient trialsflerthh's 's everciseFor the benefit of, not getting refunds for my rendering of, Should [ung at WIMO 'o the musi] litll cul feeding you chickem. OK, slup rpm back at the Throwing the bones a's fat cell is the implicit protocols that design MBGAA Advertisemens
pouse and me spologies and wednoweedgements activity NANDPIWC was already sit there as the state prototype we can and should travel in the suntry Plasticae carpark. in wearing shedon" geur holy mocon should see and then asi: Is it! out, going into GM panel sequence). "We'r te not missing Stuff What this can, this is this and's theorists distrcct dies the movement that these beautfuul primary means dries have possibilties and forgotten. O' warg just displayed as". Image! shows its seniority that ios can STILL Does these just as none been expected from. Tiek's what you gotta invee Theory OK. <enndftesti-Thi makes Greespeace the finst comment sections reported peor science. Fucebook Twitter Pimerest Scienists have try in hand in an electronic way to crase C02 and's evil into the foeoil fuel
When people for gobs."
They can wash
churped up lugsage naming the group by the muscle and memorize long
rather working because EV have longer nurture upposing interest NEEDSK That llable is. or indeed! Name this
Aguee clamate campagnerss claim it his commusioned the fist reliable structure for "significant, pumping down pools and the retised pools that apply for constraction our plants most possible restoring. This expected help do the Natural Gas requieeennss which would refer the ones hopes study, <lendoftextb-
Figure 17: Samples (T' = 8) from MDLM trained on OWT with Gen. PPL 830.82 and entropy 5.91
33

---

The Diffusion Duality
dendofiestis desperate antil they were eveyahhre.
'Bul there's 's more it tisn over" Zimmem remembers Jonathon, sighing sollly, "Sure... cheap serling what things see, which
say trath)," Jonsthoos, 'yes, show negotiate" "
When Zimmem is speaking about free market uppoctunities itself, when pressed on the lupic of open war, Zummern tites "more than anyone could poesibly think so" tt describe democracy as copital monopoly today is it today.
Bur Joel Klein of finarii gots right "Fver since the passgge of the new Millannium Copyright Act,, some companics have he menopplliinng aniimas commence. much like other companies, (and fow Verizon AT&T have achieved tens billions profit recent years and have nos grown better any ziant sieel producer."
companies)
With the problem of monopoly upon us, Ihen, there are a few possihilitis comsider. Or perhips just consider Hegel's is' and if vny 18th-century academic and intellectual would have "discover's work" eitter field (or ... Hurs) would have worked out for him. <kenloftextb-lmee 23, Now York Throela Willess Monorial Park, New Joney? Times beat reportzr Fli Pasdorf warches MIS finals, including
Supporters Shield winners Atiania United.
for Conference Finils
Souttle Sounders EC bost Allama United in the Supporters' Shield final Houston played Sunday. Today, they soured against them in style.
The winner was the Monarchs Monancts. They had extended their run to adsance Gothum with streaking wagside in the 67th minute of the game followed with hundball interception by Anihony Fasy and passing nocist from Fredericks Smenly.
The result: points over Atlanta United
Wishington DC FC vs. Philadelphia Union, Arizonu, subscitttion 1-0 U.S. Capital Cup Supplemental Championship Final 1997-728 New York Crew, York, subecitutions 2-1
United Constructions Championship Final 4-0 defending champions USA, FC Dallas Second Division
023 Dullas LeunOR Dullas MLS MLS Supplemental Championship Final
1996 Seattle Sounders
LA Galaxy. Angoles, substittiion
1909-76,
York, 1926 substitutions
U.S. Capinal Lezgue Cup, defending champion Atlasta United Ertones Acadomy semaiiinas, GCT
Emem Champions League Goal of the Round 6 Pedro Joso always seemed the nest bair of the season. Lionel Messi, 14 at left back. had kicked off the Europu Lengue und some gouls season. At the time, he igmited calling season earry- forwand. won the chimpionship and much helped Comnor Randall and Miumba Teves
way from Glees Fletcher by Arjen Robben Dallas west even deeper. United had sent into extra time after burning off the D.C. Dynamo, spent the minate the goal by Buolt, at the lop of the box and midway through the final United attacking responded well only an own goal from Troy Vincent. The first goul the final goul igmited good play involving Jeremy Vergandra,
Thomas Baismith
bicycle finish curing
half a drawn by Ninos Terstad. Dullas
Columbus in March league game. Assis free play ended in safety peek the Columbus goal. but watched TV. The Sounders were heavily fined double salary. Columbus goalkeeper Sean Mannah (injured), Andre
was overlooked make 50th minute were charged. Still, the fins player to wear a pristic syelid missed the rest of the final.
and Temy Wingwerk the
On the 60th minute of the gume. Atlanta United fought un Ogen off to beat the Columbus Crew. Fans were up for their powerr and the hosts maintained their defensive superionity againss the defending champions. Captain Al Merida stuck the louchline before bringing the bull with sixyard finish that the game unill 95th minute. From thene FC Dallas skipper and winner Greg Garea locked the gams up in two.
The Vision to the Holding Aresa in Wolfshurg, Gemmany
Medicine had let in foodball The death knell ended and several seasoms later injuries had been endured everyone with the hunchest modification into the equation online after Manchester United sigmed Kevin Wimmer, the expetimenal player for getting their expunsion franchise shared the playoff games for the summer. Several years later, MLS MLS almost had brand and replace someone largely saying how much they didn't want to see it. Theclendofeexi>
BBrramidal would be equal pattershipp) Wiadimir Kiisshkko stepped down player.
Detroit step
<lendoftextb the siremit could we finulise the sccoed Batch, Spirfighters had withdrow their crows. First fuel was cut off by Floet & Weather off Beadford [1] Deoney evaenated the sircrew to the next jeb(15 Afterwands Opeeraiion ani-hhater/Ani--uuu
-Putt Two
[[84-35 Squndron drupning keffets from the Norwegian Karent]]
Flying combut in the RAF's Thrull Air was difficult under Damtoothhs is 84-35 Squidoes a special exploitin the ammunition of captoned Raiders of the Assaesin glider [6]
Ginent Cork insisted that the special gropp reture the part, but that the volunterss of the iquadion would foon the basis for the group survive the dustage, but would also relesse the group of ten piloes of
formution begun recommaisaanee in direction Sootlndd is first offihown dive after the pilat was safely ejected without being spotted at sex][]
where Ben Hubson became 'Wing Commander
Bomber Squal". They assisted other puins to turpedo
The aircraf) was donated to the Light Lord Gladys Rider eyes on weather charge aviation "Captain208" Airield, 194 Squadrons was founded PCL hiding in the Juintled Air division (3 out of Squadrons). depending on the weather and field. Francis Rough volunteers leader September 1945 again The group, who had just bocame "Unfortunate". was transferred to PCL on Ostober 1944 as the carrier group was disbanded, then re- Organized NLSS, The Storm, 27 February 1945, under General Whitehuir. The leaders of the group retored in Los Angoles on 21 July, the day before Ryan was shot down though. The group would rost on the Hill with the other 1937/46-43 Squadrons and on to Egair in the first and second botch 16 September, with options 'priovity weapons" fonmation. They were left on 26 September 1945.
availbbe PCL's before squidron training emis.
onders
later top left]
After liying 1944- KKK llight from England. the Harponss mosived owned by Curonurs Honnoss Compuny The 46verew left from Fgair at dawn on 29 September arriving on 31 Docember and was esconted to Fullout the Sarker on 24 October and Malia on 14 Nowember 1945. After a while, the group uas foroed to leave.
bad outlawed following a shortage' peovided mrinner the group,
On 23 Sepcember, Curvey, Murite, Huussnn and wing communder arrived and directed the to Lanelles on 27 October. December. "An exercise" See also edit jckmudfiixul-Speecill thanks the Wessighouse 2000, for the Electronics of Director Sylvia Latham and Award Filmmaker Eileen Savage, [6) The 2004 upisode of tribute premiered on Brewdcasting Street on Januaar B, 2004 at 8pm UK with Amold Fisher;?]
from two additional bombers edablished at IBIB when General Hollannd look 21 December 1944, with agsin, allloough more diaflees were provided
O.M, Thompson was created and narrated by Polish actor Pillow Seegous. Members
Scoti [ edit.
Scott [ edit
Eileen Savage left Cromington, Eastern France, early aged 20, returning her humetown of Wallington. Later life [edit]
She shared her home faat in Devonshire, the early 1980s with a family visiting her. She was in a series of published standards inclading Television Television Resident, and two scripts produced for lan Butterwoeth's Endeavour about drug addict.
Personal life edit JedemlofeutiiChhee national National Tutter Collection Collection (del Annivenary pinces), patrons can enter their local dusilleey will also pop up in vollect hereto display of beads from the Mall and Potirr,, a supprise for thuse who are making much use of a limited time.
for bread, and simple are for cokes. One wand that adds to having great home (such Tommy' is the beginnings potting displays. In this people special from
a drawn poterry display from here. tuly magnifient Cross eiching
Cusiom inco
plessures when drawn black. please click the link TG4:8148 Repelleax Mini Burs made from brown threaded brass glazes and cragon gold coloured squares. Quarry stamped paper the mixing using my tips of Tope Instructions.
Postscrige, Jum idonumeed and the janitoe kept Bearingheld unail late the night for this Office. Following concems and public discussion, the Patroa/ / Chamber teom put aside their schedule for Saturday to
cope and enable the stall who were to be busy charge, unail midnight on Tuesday and-dendootestt>>
Figure 18: Samples (T [[= =1024) from MDLM trained on OWT with Gen. PPL 104.85 and entropy = =5.63
34

---

The Diffusion Duality
D.6.4. DISTILLED MDLM VIA SDTT
<lendotexbb 7 a.m and balf on hour - paused several of them then as they looked for ways to memily end their wait. Most applicunts apglied to Cubs shartly before the coming. They each had a manzm to launch during the day to India,,4,s that followed.
much better. Five years ago, according RBC Kolkata, doubled presence in Shanghai, Heijing, Singapore very few in Asian part. Judge Mosesh Katan, from 60 has rallly unheard of.
relatively few superstar the world, scene expert at becoming anchor For the world's elite.
A comeryicaled CDO lad rook the group ower the top since ten years ago, with the last 50 people, exactly one month people posted ressived notiflcations that, as wns said, "ira would he the lurch, WH the ship coaring to heng upon annomeentees a call for zoonomic, economic, and military rule. "deiddffoxxtts STATEN PARK, N.Y. A few companies operate in the New York aret, clase to the I in 2 the They also
narket that, smong other things. isovenuuting hand out deviees people to get araund, changing the figure from about 25 percent to over persent their likes,, and just over percest tocal their bites.
company.
Gonds has chingod the process
the hasinoss
identify
spenting amounts
Many of the people go home school, orget stuck in Midhown Fend but still Jose this unportant part of the lives. They see the only a tay incresse in speat time in the home in some 6000 people in the suburts: they are much more likelyy back in the car, according to mecent Allen & Co. research will show, would
that much impact to lincup their the New Jersey area now, but these bikes left mostly in right until they change, shor arny. It does not afTect those that use bicycle butteries trasportation needs. Adviccatos hawe countens that hieycles that licked bike and or insurance like they peadicted that Uher was doing not this swayall by the smack on the drivers they kept could speed up about about 20 hours regulaily and about slightly less (over 2 percent) anmual cost, and ever bus gotten there could be a sighting tool for the bicycle. This rocoeding could be moved from to carshtrre on Founth Street andor just use
Strangely, and internal
A survey recently conducted by New Yoekers las finds no evidence, nots that bikes, a ferm hus dricing with miles per hour providing power 25 miles are understandably cheaper, retum a mic of 579 for the some motor with bikes to the right also suggests that they want the instal period to be limited, barring them from oytioning for any pesmits request bought caflier during the year. Some businesses have also resigned dinetlly, apparently, to other Philadelphia bars.
The plans impact here annomeed last night the OSHA, and the teamlaw pages all states aect sharing
bicycle parks, and lanes, and some will offer hicyclenes vehicle sharing savings from the opens be reflected in vehiste-scale architecture plan. The CBO estimate on whetlier best sot up minss to nees his issued in idea, anyway. And
service, there postner wishes some municipuliiiss this would result in will face any bicycle leasing capubilitiss
The community can start to family by that couple mannerioms to connect each cycle to both systems is scamloss, but what they may rwesk is easy. I'm not quite sure who will be putting those additional zents the bike, so that they could Even my opinion, Mangoret Margot set up bus lot talks, state regulators and local deceler fundraising, is having the potential to incrense the ovenall cosl commaring: Visit my blug. Articles Paid leaveumy word - healttyy, and oon the ison for saft. I had boen wrong those who are in ccalitiin of warkers, and have refised to pay such a bill when they have all kinds of burt sepresents the purpose of what going in the work. and the Bayle do this post Florida at Heriiag Foudation, California. Members Congress, by Republicans,
slower
home.
this, bocauser- the class should has the incentive
dirret contributions and, instead paying
cush to the middle class than the rest. enough going on joumey when a clear numiber Russian
The New intellgenne epertivess the what the New American called 'Russian backing team', posted on Tuimer. Republican Govemors, the GOP Senators, now others. Sen. Toes Conoes of Aliskn was Trump, last mombbslendoolest>>
<lendofextt>> general election. The Devember 2009, emails are, mainly, of the stowing away the What else of the time brought the sense official through Labour, MLP's who calling this "disfflected party.. not just right wing private party DS. Helow
Or as part of the calculations that Camer on wanted have some effect, but, although he has writien around gay mamriage and homssexualaty, it was to state to cabinet secretary Nick Timochy that the general elections must art like this: primary, that Labour single in Massachusets,, with candidates left campaigning over whether they should receive support. Batting woid additional pressure by ministers or personal bockers Cameron got lase claim that Somerset key tirget for Lahour the election. The Number 10 suspense interviews days after election 9pm Monday on Sky News and then back before midnight has been coming up coline Monday.
Not broad
But even party allies operating in different guises, such Ukip and the Clristian Democrats, are clearly good delivering results in the surveys, the Lib Dems and Lib Deass came also in by suppris in the disaffictadeection election including Alistair Darling and NCT In 2014, Roston continual to oppear on electoral man despite the results,
polls out contests and usually
Toies amracte split few liberalliiess 2013, taking the Dunean Smith 4 cutcome down any moderate
The sarvey zun by the Moxi poll says these claims of missed opportuitty, puts the evidence evenly seross both. Fox decacess there has been evidence that Labour was either really tied to the election or "envinnment cconomic chace", although the Sutton report as result shows this, Tony Blair prepared a result pulled from the Communication Workers EC, and after a complicated of successtul pressure, the campaign form the codlition. factor in post-election colculations. However. Lebour party polling differencc may represent patentiall the point to collitin deal, and suggests sugposts may have rejersented shareless more dacision making side the pand.
party emerged key figure
Really the biggest question on ultimuelly the cause of trouble for Iugland in the last prime manaster history, and immigraticn. The second biggest issue - which had been plucked from bU devolation. tax reform, and Fumppen integration was, since 2004, always barely compaign Even toough this had been majer harrier to Labour, sources like the partyss openly MP Robert Car,, soid they were also undecided privately idennifid, while Amit Gujea, Experts sny il may have been unfair note that the new proseure to authoritarian mile by Dawid Cameeon was sardanic with left- right-win MPs, with decisions also made wesken a gowerning position in Rut with Labour ahead in the general election, and the Twies moving on the to the effect that the proportion of in the rest of netional income should pay full, it's not to underesimate how much cuta pay goang to the comntry's housing market was ahove what elections 10 year was.
party rating had "nothing do".
The party lesdership's decision final will taped on shows to Number 2015 planned hudget MPs daddine Machael Parchas, his parteer at Boston and corporate lawyer at the Guardian Business, contributed to this -eport-dendotttstt a Hannah Jacobs cbook there a form US scceening called Ben-7: Bloodline was supposad to he fivm Richard O' Briie, but it will hold on Wednesdyy night. is espected that only
contrnvensaal, but contrwersinl onalysi the economy particular and people close to prime minister. The extended to Wednesday Septemieer
and I Lineage which beginning that for toxic moods but it thetoric and bind. Its first America have the readiness The London Review Books, Aljha rostes bost in
those who follow the Next Game of the American
flooled with impressions. Post-cumtrr Laurie Waggot rocently has other oginions. She his correclly reported that producers and editors Coblins and Orlando Bloom have exiendive copacity for the director of the original, and ideally, sequel. Indoed, she ke of the game at the modia Theatre in Dublin, dispitt his gow at, maddsarely clear smile. enight his eye exclusively from the newer events. That espucity for the director also is, as the movie's is even Gia Pritchard Finnras recently admitedd it, with the reasons that (o) he may confide in his choice of head, and (b) fand of inself to mike the training material for the Murny and Donald Duck Game the American mixed in politics. Hamiirdd up with misagyny jost the moment before the end.scendfftutt>>
his
beuthers Swamp Thing chance tame him. The political
Figure 19: Samples (T= = 8) from MDLM trained on OWT and distilled using SDTT with Gen. PPI. 193.05 and entropy 5.58
35

---

The Diffusion Duality
dendoffestts ligitation.
Mach to the dismay of the American Civil 1 Liberties Unice. President Bush sent joint letter Congress, comulling with the bodly deciding whether strip surveillance of surveilance authurity The metion
their amendment considenttion but failed poss Hill, due members of the mainline denumination the United Methudist Church also introducing the legisbaton, sigmed on November 2. 2002, after the 1898 Americn Revolution. The compromise comes within a proces based upon the expeession of the First Amendment rights of the Constittion and the Fourth Amerdment of the United States. (The Libertations uphold the right of state officials to lawffll activitis by foreign govermenrs defined by the Clonea Conventions, before advancing to the question of the dangers that arise from this practioe.) indeed obvious Americans that kdentify with their allegiance America's government, of Americans in the enforcement of the low, and their cocperation with America's 's govemment. Imminent surveillanes, difficult to dispune, either by a whole or part the American pepulace. Almust any seclinique fhat is ever [eddcted]] for will be dilticil likely to violate the princy and general security of Americans, as for at I know, far more thon efferrive surveillanee metheds, Such technique will of information, and will be have to give additionl comments on (his purticalar point, and welcome
Comgress lad ebjosted
proaskural that sudrresed the firal version. other
While these cuncerns
seem reflect positóon cannol be explained bused entirely on the participstion
develop techniques. informants, and other methods that begin expost person's electriic and legal communicaions far beyond last s. the
sddressing First, we are very surptised by the fact that so many of them, and so many our police officers, are in this postiion The information that we have gathered is very clear. We know about us intimately. that we are very sensitive, that they are constrained, as well as about the limitaiions of what they are doing. We are not only represening them, we know of us, and that our knowledge of them valuable, and we dking our best to make them the moot difficsli and difficult next step possible is possible that this is limited surveillance strte, much like what we would be sufjected to. but that if the program led commented
to significnt increase surveillance, would be labeled as syber routers phones. Some that it would be very difficall for such un assessment to begin with a computerized seed: that documents available to the eyes of the general public would be unveewabee and that the risk would be inevitable that snyone who under certain circumelances coukd have access inmedistely to this superstate. I also think thit such acuess will not be reoognized as begrap, and therofore as repeateel violation of the law, of this reader, that such superstate state almost always, intuitively, very hard to reach quickly. and determined as having highest pussible impact we are to stabilizz the
filters
sufficient curbing this process, they constitute confounding,
and the government will not want to do either. But can almost vertailly be seen by something that we power grid in the future.
fimilly confront threats serious level. If they are allowed remaii their comsqquences
The aforementioced congress address als made it crystal clear, we wanted a process that required analysis of all the information that was sent and received by the American population, and, one hopes the fate of humamity decidee the a loger suciety. Unfortunalely, there many. mamy more of these that contimme lesk into the Prism program, Thas
therefore, absolselyy would huve a much more likely surveill surveillance a nation also and thus much more likely that would related to above memioned" cival law enforcement force." or actually related to the violazion international human rights liwss
that we
Accordingls, according to the UN statickcs, 30-34 major reported bybernet sttacks have occurred in at least 29 countries in the developing world cureenly. primurily those conducted by MIT or China, and the aforementioee mujur eybernet attacks these from cuiside civiltration and our societies and its general use. Indoed, the grinder contributes to tremendons threal the Internel hocause is one of the sools that a great mujority of people actually use. For over the past several years and decades, ribooa cunters have been placed on the cusp of being in ensed in some form, electrunic er remote form, at an incredibly high price. We reter to today's y Western Redwood as being the standard indoscialddebbiiieit
will continue be suffering systematic and
instances which regard the violation of basic human rights beuial abuse of cyber enclaves lives, many facters have impacted
impottant issue.
point along
actual global threats
are prevaent aspects
relationship between the Intemet
clendftestts distribution is not only way in which species shire these types of traits in isolation, Finally, this is, of course, complicated by the fiferencees among individull species with respect taxonany. Although the ways in which individuals in some cases differ substancially in the overiil value of fumily members, some differ by variation in other variables-ce vice vnda-and this statisiiaal andlysis can not he helped by the extest to which trziss differ between two asseniiliess in many cases, and cannor say something on the contrayy about having equal charactersstics in all. Within the group. The only dispuie about similarit betworn the first response would to affinn that the argumest is a similarity exists parely between group and those whose ows varieties and vurieties, like the two versions of other groups, are all distinct in the same domuin. But, in some cases, or all of the cases, explinnng the distinction would require very simple expleatiion of how the distinction understood, Of course, this hypothesis would be a major objection having real evidence (Hein, 1993), An ailleget to define the degnoe of similariy berween any of those groups might thyy eletely the The point is that the data on the respective varieties and curieties are not ensy to undersaad. It would be possible to show examples each group of diffcrent varieties and curieties used mitum sciences: this moans at the hare minimum that, for each, each of the groups uses the particulor natural systems they are hased in Just bocause would he possible to make a chim in given domsinn isn't because the conciasions are generalisable ench individusl, is not because each individual is likely to reach a unifeem political or personal conclusion, and is not becouse is possible to prove that even this is easy for it be useful for making climm in one domsinn not necessailly show that different domain is equally important. In this sense, the lack adequate goneral peinciples provided by starirics tre ovident a very dinee way that the differences betwesn different sets individial groups can have boen nst netarllieed only by formal simulations of the two grocups and no quamiiaaive requirements for the correspunding values in each kind of grouping.
of groups the similarity between
imports depending on the generic diversiy, whether there difference
herweon chamattrissiis and features those three groups.
show that,
dominent, with similaritcs occuningg competitnely.
case.
is
The fact that there is an altribute between fire discinel biological classes does mein that they have all of the etact same churacteristics and it does not imply that all theee groups equally: It is that diffierett
categories lave the comparative advantige. And does that just say the leadl, however, this hypothesis does not mean that a particular class has the primury advaniage agninst a different class. In fact. this hypothesis does net mean that there is primary difference. And does not chiim that it oue that euch other species possesses the basic same characterisiis, nor that nailher does the class have any general principle of advantiges or disudhanlages over the other. A model. Rather, this means that a class as such not only relevant but more precisely not inclevant whether cr not it becomes relevant. For exampie, ue know that model retains its advantages in the other domain, and
may comparable many respects
misledingg
this
say
shility select for very specific undermine the modem observation grid in tamss of how it openirs, and oven when its assertions are widely necepied. the quality of the information now ivailable to veity its trurh is not present to me.
typically has as few adsantages as actual class possesses. information such the diffenence betwern an intranet or well-penforeed telephone offers very specific choice board on location. It doesn't I exploin the muture of the
fast, along
the fact varies Darwinian categuries, would
One of the key problems with the most importnt differences between different varieties and varieties whit how they actually do the other domainss The most impurtant aspects these differences only
understood strictly cultural ones, knowlogge the purticllars varistion such relevani domains), but also the artifciall camoulagee of similaziies, and the intrinsic difficulties in theonies that mirree pictures of the characteristics that are performed by each vuriation. Gemetics is another type of variation, and an impertant set of liminaiios upon the field of the natural sciences and the larger public's altenion. The empirical ressons for this limitation include that it does not chinged
achanaaess attained different domains real phenomenon virtually
the
reall matter if conditions there are only very few combinations chosen midiiihally.. The difference is this fact cunnor examined in almust any reprssntaivveeeeaaleeue
nolher. it is simply mechanism by which featires are different in diffcrent other natural systems, and also in different samples, and also mean that coses.
Figure 20: Samples =1044 from MDLM trained on OWT and distilled using SDTT with Gen. PPL 36.89 and entropy -5.39
36


Reviews:
[{"review_id": "PFV20w4fys", "reviewer": "Reviewer_1i8M", "summary": "This paper proposed a continuous parametrization of uniform-state discrete diffusion models. The key finding is a noise schedule (Eq. 11) that ensures the equivalence of the marginal distributions of the uniform-state discrete diffusion process and a Gaussian continuous diffusion process transformed by an argmax operator. The authors demonstrate two applications of this finding: to introduce a relaxed version of the training objective where a softmax operator is gradually annealed into the argmax operator, and to introduce a distillation method based upon the consistency loss of the underlying Gaussian diffusion. Their experiments show that the proposed method leads to improved perplexity in uniform-state diffusion models and faster sampling after distillation.\n\n### After rebuttal ###\nThank the authors for their response. My concern on the gradient variance is resolved, but the others remain. I stand my initial rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank the reviewer for their constructive feedback. We address the reviewers comments and questions below.\n\n# Concern 1: Interpreting training loss (Fig 2) vs Tab 2\n\nWe will clarify Fig 2 by highlighting the key takeaway: DUO’s training loss exhibits significantly lower variance than both MDLM and UDLM’s. We thank the reviewer for pointing out that the biased training loss of DUO distracts from this key takeaway, and will revise this figure to emphasize the empirical variance of the loss, as well as gradient variance (more details in Concern 2).\n\nWe note that in the current Fig. 2 (paper), DUO’s curve lies below MDLM’s because its training loss (Eq. 16) is a biased approximation of the true ELBO (Eqs. 6 and 14). This bias causes the training loss to be lower. More formally, for the true ELBO (Eq. 6), the discrete diffusion parameter $\\tilde{\\alpha}_t$ must lie in $[0, 1]$. However, during training with the approximation (Eq. 16), it suffices to choose the Gaussian diffusion parameter $\\tilde{\\alpha}_t \\in [0.85, 0.97]$, which maps to  $\\alpha_t = \\mathcal{T}(\\tilde{\\alpha}_t) \\in [\\approx 0, \\approx 1]$ for $\\tau \\to 0^+$ (see Fig. 4, appendix). Since, **Gaussian latents contain more signal than noise**, while training with Eq. 16 with $\\tau > 0$ the denoising model finds reconstructing the input easier, **resulting in a lower training loss**.\n\nTab. 2 (paper) reports the validation perplexity computed using the true ELBO (Eq. 6 for DUO). For clarity, we report the validation perplexities of various methods below at different training steps and observe that DUO consistently outperforms UDLM and approaches the performance of MDLM. We thank the review for pointing out the potential conflicting interpretations of Fig 2 and Tab. 2, and believe revising Fig 2 to focus on variance will alleviate this.\n\n|  | 10K | 100K | 250K | 500K | 750K | 1M |\n| --- | --- | --- | --- | --- | --- | --- |\n| MDLM | 65.75 | 40.01 | 36.37 | 33.41 | 32.66 | 32.03 |\n| UDLM | 91.95 | 74.71 | 43.11 | 39.70 | 37.63 | 36.71 |\n| DUO (Ours) | 69.34 | 57.11 | 37.44 | 35.20 | 34.22 | 33.68 |\n\n**Note:** UDLM is equivalent to DUO w/o Curriculum Learning\n\n# Concern 2: Gradient Variance\n\nAs requested by the reviewer, we report gradient variance across model weights. We compute the variance of each weight’s updates over 10 steps on LM1B at different training steps. **With Curriculum Learning (CL), gradient variance is significantly lower**—especially in early training (RTab 1; below). Among the 100 weights with the highest variance (RTab 2; below), **CL reduces variance by an order of magnitude** early on, which gradually diminishes over time. This reduction appears beneficial, as it accelerates learning in the initial phases, reflected in improved validation loss; see above.\n\n**(RTab 1) Sum of all variances:**\n\n|  | 10k | 20k | 50k | 100k | 500k |\n| --- | --- | --- | --- | --- | --- |\n| CL | **2815.36** | **2471.65** | **1890.76** | **1469.85** | **947.98** |\n| w/o CL | 10852.9 | 7811.04 | 6315.7 | 5454.7 | 1678.47 |\n\n**(RTab 2) Sum of highest 100 variances:**\n\n|  | 10k | 20k | 50k | 100k | 500k |\n| --- | --- | --- | --- | --- | --- |\n| CL | **0.30** | **0.85** | **1.21** | **0.86** | **1.15** |\n| w/o CL | 11.7 | 20.09 | 34.2 | 55.1 | 1.92 |\n\nWe observe a similar trend in the variances of the loss as well; see below.\n\n**(RTab 2) Sum of highest 100 variances:**\n\n|  | 10k | 20k | 50k | 100k | 500k |\n| --- | --- | --- | --- | --- | --- |\n| CL | **7.09** | **6.29** | **5.33** | **4.97** | **4.76** |\n| w/o CL | 9.19 | 7.72 | 6.85 | 6.32 | 5.47 |\n\nWe will replace Fig 2 (the loss curves) with these findings.\n\n# Other comments\n\nDuring distillation, the student and teacher networks share the same architecture and differ only in weights—the teacher uses EMA weights, while the student uses the current model parameters. Both networks factorize the joint distribution independently as $p_\\theta(\\mathbf{x}\\_0 | \\mathbf{z}\\_t)  = \\prod_i p_\\theta(\\mathbf{x}^i_0 | \\mathbf{z}\\_t)$. Although Xu et al. (2024) propose a more expressive factorization with an energy term,$p_\\theta(\\mathbf{x}\\_0 | \\mathbf{z}\\_t)  = \\prod_i p_\\theta(\\mathbf{x}^i_0 | \\mathbf{z}\\_t) e^{E_\\phi} / Z_\\phi$, which leads to a better fit to the data, the independent assumption performs well in practice [2, 3, 4]. Modeling the energy function $E_\\phi$ and the partition function $Z_\\phi$ is also non-trivial, often requiring an additional network and complicating both training and sampling. We believe incorporating this approach could enhance our model’s expressiveness and benefit both DUO and MDLM, which we leave for future work.\n\n---\n\nReferences\n\n[1] Xu et al., 2024, ENERGY-BASED DIFFUSION LANGUAGE MODELS FOR TEXT GENERATION\n\n[2] Sahoo et al., 2024 “Simple and Effective Masked Diffusion Language Models”\n\n[2] Schiff et al., 2025 “Simple Guidance Mechanisms for Discrete Diffusion Models”\n\n[3] Lou et al., 2024 “Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution”", "claims_and_evidence": "The major claim is equivalence of the uniform-state discrete diffusion process and a Gaussian continuous diffusion process transformed by an argmax operator, for which the authors provide a formal proof. \n\nOn top of that, the authors further claim that a special curriculum that gradually anneal the soft to the hard max can be introduced to reduce the variance of in training, which is supported by the training curves with less perturbation than the baseline, as well as an improvement in the validation perplexity and the zero-shot perplexity. \n\nThe authors also claim that the distillation method built on top of the aforementioned equivalence is effective, which is supported by the curves of Gen perplexity and entropy in Fig3 and Fig8.", "methods_and_evaluation_criteria": "The proposed method makes sense for improved training of uniform-state Diffusion models. \nThe evaluation criteria all commonly used in the field.", "theoretical_claims": "I have checked the proofs and they look sensible to me.", "experimental_designs_or_analyses": "The experiments and the associated analyses are quite standard so I don't feel any fundamental challenges. However, I am curious why the reduction of variance is only demonstrated with the training loss, instead of the gradients.", "relation_to_broader_scientific_literature": "Uniform-state discrete diffusion model is an important family, for it naturally induce error correction with more sampling steps. However, it does not work as well as the masked diffusion. This paper is an attempt to improve the training.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "+ The writing is clear and very easy to follow. \n\n- Although the equivalence between the uniform-state discrete diffusion process and a Gaussian continuous diffusion process transformed by an argmax operator is established formally, the training loss appears to be less rigorous. The network takes in the soft version but predict the hard version. This may render the training loss harder to interpret. Reflected in the empirical observation, the curves in Fig2 shows clear margin between the proposed method and MDLM, but the result in Table 2 reports the opposite.  \n\\\\", "other_comments_or_suggestions": "The name of the proposed method DUO appears abruptly in the experiment section.", "questions_for_authors": "Apart from my concerns mentioned above, I am also curious to know how is the distilled network parametrized. If the student model is similar to the teach in terms of the factorization of dimensions as articulated in Xu et al. 2024, why wasn't it an issue for the proposed method?\n\nXu et al. 2024, ENERGY-BASED DIFFUSION LANGUAGE MODELS FOR TEXT GENERATION", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nUPsDLWPqH", "reviewer": "Reviewer_maKg", "summary": "This paper presents the first theoretical connection between discrete and continuous diffusion models, showing that discrete diffusion emerges from an underlying continuous Gaussian diffusion process. Building on this insight, the authors propose a curriculum learning strategy to improve training efficiency. Furthermore, they develop distillation techniques inspired by the relationship between discrete and continuous diffusion. Finally, extensive experiments are conducted to validate their theoretical findings.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank the reviewer for their constructive feedback. We address the reviewers comments and questions below.\n\n# Concern 1 : Generating trajectories using DDIM.\nAs noted in lines 250–252 (left), the deterministic Gaussian trajectories assume an optimal denoiser: given clean data $\\mathbf{x}$ and a latent  $\\mathbf{x}\\_t \\sim q_t(. | \\mathbf{x})$*,* the optimal denoiser is defined as $\\mathbf{x}^*_\\theta(\\mathbf{x}_t, t) = \\mathbf{x}$  for all $t \\in [0, 1]$. These DDIM trajectories (Eq. 17), under an optimal denoiser, preserve the Gaussian diffusion marginals (Song et al., 2022; Sec. C.1 in Zhou et al., 2025) and, when mapped to discrete space via $\\arg \\max$, align with the marginals of uniform state diffusion, as shown in Sec. 3.\n\n# Concern 2: Training loss vs Eval loss\nWe'd like to clarify that while Eq. 14 and Eq. 6 are equivalent, **Eq. 16 which corresponds to a biased estimate of the true ELBO is used for training**. The $\\arg \\max$ in Eq. 14 is approximated by a tempered-softmax in Eq.16. This approximation reduces training loss variance and improves generalization. For a detailed explanation, please see our response to Concern 2 from Reviewer 1i8M.\n\n# Concern 3: Perplexity for UDLM in Tab 2 and Tab 3.\nThe numbers for SEDD Absorb and Plaid are taken from Sahoo et al., 2024 and Lou et al., 2024. At the request of the reviewer, we conduct **new experiments** by training UDLM [3] on OWT as follows:\n\n|  | Val PPL (OWT) $(\\downarrow)$ |\n| --- | --- |\n| UDLM | 27.43 |\n| DUO (Ours) | **25.20** |\n\nZero shot perplexities:\n\n|  | Wikitext $(\\downarrow)$ | AG News $(\\downarrow)$ | LM1B $(\\downarrow)$ | Lambada $(\\downarrow)$ | PTB $(\\downarrow)$ | Pubmed $(\\downarrow)$ | Arxiv $(\\downarrow)$ |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| UDLM | 39.42 | 80.96 | 77.59 | 53.57 | 112.82 | 50.98 | 44.08 |\n| DUO (Ours) | **33.57** | **67.81** | **73.86** | **49.78** | **89.35** | **44.48** | **40.39** |\n\nThe conclusions remain unchanged—DUO is the state-of-the-art among USDMs.\n\n# Concern 4: Reported numbers for MDLM and UDLM in Tab. 2\nLines 258–264 (right) clarify the **discrepancy caused by differences in dataset preprocessing**. The LM1B dataset contains short sentences (~30 tokens each with the GPT-2 tokenizer). In prior work [1, 3, 4], each datapoint in a batch is a single sentence padded to 128 tokens. This results in each sentence being considered in isolation.\n\nIn contrast, we follow the sentence-packing scheme from Austin et al. (2021) [5], where sentences are concatenated before batching. This results in sentences potentially being split between batches, with additional and  potentially noisy context added. As a result, the packed data has a higher (worse) perplexity than padded data.\n\nAt the reviewer’s request, we conduct **new experiments** where we trained our model using the preprocessing from [1, 3, 4], and report the results below. The conclusions remain unchanged—**DUO is state-of-the-art among USDMs** and approaches MDM performance.\n\n|  | Val PPL (LM1B) $(\\downarrow)$ |\n| --- | --- |\n| UDLM | 31.28 |\n| DUO (Ours) | 29.95 |\n| MDLM | 27.04 |\n\n# Concern 5: Low sample diversity\nTo make the curves in Fig. 3 more readable, the corresponding entropy values are provided separately in Tab. 4. As clarified in lines 351–358, the entropies of MDLM, SEDD Absorb, and SEDD Uniform align with that of an autoregressive model without nucleus sampling (approximately 5.6). In contrast, **DUO’s entropy matches that of an AR model with nucleus sampling (p = 0.9)**, around 5.2. Qualitative samples (`anon. link [6]`) further show that DUO produces significantly higher-quality outputs than both distilled and undistilled MDLM at lower sampling steps $T=8$.\n\n# Comments: Clarification on lines 768-769 in the appendix\nWe meant to say the following:\n\"We empirically verify the equivalence of the USDM ELBO with discrete latents (Eqn. 6) and Gaussian latents (Eqn. 14). To do this, we trained UDLM [3] on LM1B using the true ELBO from (6). We then evaluated the model using Gaussian latents (Eqn. 14), and\nrecovered the same perplexity (36.71) as when using discrete latents. For each datapoint $\\mathbf{x}$, we used $1000$ Monte Carlo samples for $t$ sampled using antithetic-sampling, with a linear schedule for $\\tilde{\\alpha}_t = 1 − t$.\"\n\n---\nReferences\n\n[1]  Sahoo et al., 2024 “Simple and Effective Masked Diffusion Language Models”\n\n[2] Zhou et al., 2025 “Inductive Moment Matching”\n\n[3] Schiff et al., 2025 “Simple Guidance Mechanisms for Discrete Diffusion Models”\n\n[4] Lou et al., 2024 “Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution”\n\n[5]  Austin et al., 2021 “Structured denoising diffusion models in\ndiscrete state-spaces”\n\n[6] `Anon. link`: https://docs.google.com/document/d/e/2PACX-1vR0uKDuQHl4bBuC8KokEQhHNMvGdxbIskJm_SfXO_L6haSzWEqjPtL9wkVmg_yacNzMei2DAk21J5XX/pub", "claims_and_evidence": "The experimental results generally support the authors' claims.", "methods_and_evaluation_criteria": "The evaluation metrics used in the paper are generally comprehensive and appropriate for the task. \n\nHowever, generative perplexity (Gen PPL) has been criticized as an imperfect measure, as it can be artificially lowered by repeated words [1]. Despite this limitation, Gen PPL remains a widely used metric in this research area. Additionally, the authors mitigate this concern by including entropy as a supplementary evaluation metric.\n\n[1] Zheng et al. Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling. ICLR2025.", "theoretical_claims": "No.", "experimental_designs_or_analyses": "Yes, the authors’ experimental setup is reasonable.", "relation_to_broader_scientific_literature": "Continuous diffusion models have achieved significant breakthroughs in image and video generation. More recently, discrete diffusion models have gained increasing attention. However, key questions—such as the theoretical connection between discrete and continuous diffusion and whether widely used distillation techniques from continuous diffusion can be applied to discrete diffusion—have remained open. This paper makes a novel contribution by theoretically establishing the link between continuous diffusion and discrete diffusion based on the uniform forward process. Additionally, it introduces distillation techniques to discrete diffusion, further advancing the field.", "essential_references_not_discussed": "What is the relationship between the proposed method and Bayesian flow networks [2, 3]?\n\n[2] Graves et al. Bayesian flow networks. arXiv 2023.\n\n[3] Xue et al. Unifying Bayesian Flow Networks and Diffusion Models through Stochastic Differential Equations. ICML2024.", "other_strengths_and_weaknesses": "#### **Strengths**  \n1. The contributions of this paper are highly novel (as detailed in *Relation to Broader Scientific Literature*).  \n2. The paper is well-written, with clear logic and concise explanations.  \n3. Techniques such as curriculum learning are simple yet effective.  \n\n#### **Weaknesses**  \n1. In Section 4.2, the authors claim that deterministic trajectories can be obtained in the continuous space using DDIM and then mapped to the discrete space. However, this is confusing because the continuous and discrete processes follow different trajectories. Based on the loss function, Duo is trained on discrete trajectories (i.e., the neural network receives discretized inputs after the *argmax* operation). Given this, how can we obtain a score function in the continuous space that allows deterministic trajectory generation via DDIM? Am I missing something here?  \n\n2. The authors do not address a key question: While they establish a theoretical connection between continuous and discrete diffusion, which enables the transfer of techniques such as distillation (despite my earlier concerns about DDIM), the loss function used (i.e., Eq. (14)) is equivalent to the original discrete diffusion model’s loss function (i.e., Eq. (5)). If the two are mathematically equivalent, why does Eq. (14) yield better PPL results than Eq. (5)?  \n\n3. I have some doubts about the comparison with the UDLM baseline. First, Table 1 and Table 3 do not include results for UDLM. I understand that the original UDLM paper did not report results on the OWT dataset or zero-shot PPL, but the authors retrained SEDD Absorb and Plaid for these comparisons. Why did they not retrain UDLM under the same conditions? Second, in Table 2, the reported result for retrained UDLM (36.71) is significantly worse than the result reported in the original UDLM paper (31.28). The original UDLM result is actually better than the proposed method’s result (33.68). Why is there such a large discrepancy?  \n\n4. I also have concerns about the generative perplexity experiments. First, Figure 3 does not report entropy values. Second, in Table 4, Duo's entropy is significantly lower than that of MDLM, which raises concerns about whether the improved generative perplexity is simply due to Duo generating low-diversity sentences. According to Section 6.1 in [1], the normal entropy range is around 5.6–5.7, while Duo’s entropy is noticeably lower than this range.", "other_comments_or_suggestions": "In Lines 768–769, is there a missing part of the sentence? It seems incomplete.", "questions_for_authors": "Please refer to the  Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0x8T9NbxAD", "reviewer": "Reviewer_J6SU", "summary": "This work presents a new training scheme for a uniform state discrete diffusion model based on the correspondence between Gaussian diffusion in continuous state and uniform state discrete diffusion. The authors state that while uniform state discrete diffusion and Gaussian diffusion are two separate Markov chains, the discrete process can be understood as a continuous process through the argmax operator. Based on this connection, the paper proposes a curriculum learning training that leads to faster convergence and low variance, and also a dual consistency distillation method. The experimental results show 2 times speed-up in training convergence and two orders of magnitude improvement in sampling speed.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank the reviewer for their constructive feedback. We address the reviewers comments and questions below.\n# Concern 1: USDMs still lag MDMs\nAs mentioned in the response to Concern 1, USDMs lag MDMs only when measured in terms of perplexity which isn’t necessarily the best metric for comparisions with MDMs. **USDMs are preferred over MDMs on few-step generation** (Fig. 3) where a distilled version of DUO significantly outperforms a distilled version of MDLM at $T=8$ \n\nAs mentioned in lines 16-27 (right), the design space of Discrete Diffusion models remains largely under explored as compared to Gaussian Diffusion models. In this work we establish a core property of USDMs — they emerge from Gaussian diffusion. This **opens up new avenues of future research which would leverage this connection** to improve USDMs by borrowing techniques from Gaussian diffusion— a connection that doesn’t exist for MDMs.\n\n# Concern 2 : Motivation for using USDMs over MDMs\nUSDMs are preferable to MDMs for tasks like **guidance** (Schiff et al., 2025) and **fast sampling** (this work). The reason USDMs allow for faster sampling than MDMs is that they can fix their mistakes. This allows USDMs to make mistakes fast, but fix them later.\n\nWe also note that while perplexity is a useful sanity check, it does not account for speed. Perplexity only captures how sample quality with a high number of sampling steps.\n\nIn the table below, we present **new experiments** that show that USDMs can achieve strong sample quality quickly: Either with low latency, the time to produce a whole sequence, or high throughput, the rate of parallel generation.\n\n| **Model** | **Non embedding parameters** | **Latency $(\\downarrow)$(BS=1)** | **Throughput $(\\uparrow)$ with MAX BS (tok/sec)**  | **MAX BS** | **Gen. PPL  $(\\downarrow)$ <entropy>** | **Gen. PPL  $(\\downarrow)$ <entropy> (nucleus sampling p=0.9)** |\n| --- | --- | --- | --- | --- | --- | --- |\n| AR | 17M | 11.70 $\\pm$ 1.15 | 926.80 $\\pm$ 10.78 | 80 | 92.43 <5.61> | 32.13 <5.19> |\n| AR  | 110M | 14.70 $\\pm$ 0.16 | 471.16 $\\pm$ 3.54 | 32 | 35.93 <5.58> | **13.44** <5.26> |\n| Distilled DUO ($T=8$) | 110M | **0.21** $\\pm$ 0.01 | **9938.00** $\\pm$ 4.14 | 32 | **78.73** <5.24> | - |\n\nIn this table, we train AR models on OWT for 1M steps with a context length of $1024$, varying the number of Transformer layers to control parameter count; exact architecture details can be found in `anon link [2]`. We do not use a KV cache for the AR baselines. We measure latency (sec) for generating batch size (BS) = 1 and find that DUO is significantly faster than even the smallest AR model. We also measure throughput (tokens/sec) using the largest BS (multiple of 16) that fits in memory. **The distilled DUO outperforms the 17M-parameter AR model in tems of speed**. All experiments were run on a single A100-SXM4-80GB. The mean and std are computed over 100 batches. \n\n# Concern 3: MDLM number different from original in Tab. 2\nPlease refer to our response for the reviewer maKg, Concern 4.\n\n---\n\nReferences\n\n[1]  Schiff et al., 2025 “Simple Guidance Mechanisms for Discrete Diffusion Models”\n\n[2] `Anon. link` : https://docs.google.com/document/d/e/2PACX-1vR0uKDuQHl4bBuC8KokEQhHNMvGdxbIskJm_SfXO_L6haSzWEqjPtL9wkVmg_yacNzMei2DAk21J5XX/pub", "claims_and_evidence": "Yes, the claims are supported by experimental results.", "methods_and_evaluation_criteria": "Yes, the evaluation criteria including LM1B and OWT datasets seem reasonable.", "theoretical_claims": "The claim on the connection between uniform state discrete diffusion and Gaussian diffusion has been explained.", "experimental_designs_or_analyses": "Yes, the experimental design seems valid.", "relation_to_broader_scientific_literature": "This work improves the performance of the uniform state discrete diffusion model with a new training scheme.", "essential_references_not_discussed": "To the best of my knowledge, most of the relevant works were discussed in the paper.", "other_strengths_and_weaknesses": "**Strength** \n\n- This work finds a new training algorithm effective for training uniform state discrete diffusion model from the connection between the discrete diffusion and Gaussian diffusion.\n\n- The proposed training algorithm improves the performance of the uniform state discrete diffusion model that reduces the gap between the masked discrete diffusion model.\n\n**Weakness**\n\n- The performance of the uniform state discrete diffusion model even with the complex training/distillation is still underperforming the masked discrete diffusion model. As they both work on discrete space while the difference comes only from how the transition kernel is designed, the strength of the proposed method diminishes.\n\n- The reason for using USDM is not clearly addressed in this paper (or I may have missed it). What is the advantage of using a uniform state instead of the masked discrete diffusion model? [Schiff et al., 2025] state that using a uniform state is advantageous for conditional generation but doesn't seem to apply for large language benchmarks such as LM1B. In this sense, this work could benefit from adding a controllable generation task where USDM would excel.\n\nSchiff et al., Simple Guidance Mechanisms for Discrete Diffusion Models, ICLR 2025", "other_comments_or_suggestions": "Please see the question below.", "questions_for_authors": "- Why is the reported performance of MDLM on the LM1B dataset different from the number from the original paper?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F1p3NRz3IM", "reviewer": "Reviewer_oihg", "summary": "This paper finds that discrete diffusion models are just a transformed version of continuous Gaussian diffusion using an argmax operation. This lets them borrow techniques from continuous diffusion, like curriculum learning for faster training and distillation for super-fast sampling. The result is training that’s twice as fast, sampling that takes way fewer steps, and even better zero-shot performance than some autoregressive models. It’s a mix of solid theory and real practical improvements.  \n\n\n## Update after rebuttal\n\nAfter reading the authors’ rebuttal, my overall evaluation has not changed， and I still tend to accept this paper with a score of 4.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank the reviewer for their constructive and detailed feedback. \n# Concern 1: Transition kernel and the loss landscape\nWe emphasize that while the $\\arg \\max$ operator maps Gaussian marginals to discrete marginals, it also preserves the transition dynamics and Markov property. In `anon. link [3]`, we show the marginal evolution follows: $\\frac{d}{dt} q_t = -\\frac{\\mathcal{T}'(\\tilde{\\alpha}_t)}{K \\mathcal{T}(\\tilde{\\alpha}_t)} \\left[{\\mathbf{1}} {\\mathbf{1}}^\\top - K \\mathbf{I}\\right] q_t$, where $\\mathcal{T}'(\\tilde{\\alpha}_t)$ is the time derivative of  $\\mathcal{T}(\\tilde{\\alpha}_t)$. This implies$\\frac{d}{dt} q_t = Q_t q_t$, with  $Q_t = -\\frac{\\mathcal{T}'(\\tilde{\\alpha}_t)}{\\mathcal{T}(\\tilde{\\alpha}_t)} \\left[{\\mathbf{1}} {\\mathbf{1}}^\\top - K \\mathbf{I}\\right] \\in \\mathbb{R}^{n \\times n}$ representing the transition kernel for a **Markovian discrete diffusion process** (Anderson, 2012). Schiff et al., 2024 (Supp C, Eq. 50) confirm this is the transition dynamics of USDMs with diffusion parameter $\\mathcal{T}(\\tilde{\\alpha}_t)$.\n\n**Loss Landscape**:  However, transforming a Gaussian diffusion into a discrete one **does not preserve the loss landscape**. In fact, the loss landscape for the discrete diffusion process is much more desirable because it induces a tighter bound on the likelihood as stated in Theorem 3.1.\n\nWe stress that the equivalence betweeen Gaussian diffusion and the USDMs is **deep and fundamental without any approximations** whatsoever. \n# Concern 2: This method is an approximation\nAs established above, the connection between Gaussian diffusion and USDMs is absolute and without any approximations. The softmax approximation to argmax introduced in Eqn(16), the training loss, leverages this fundamental relationship to design a low variance curriculum learning training scheme.\n# Concern 3: Two orders sampling speedup\nAfter distillation, DUO achieves a Gen. PPL of $79.24$ and entropy of $5.25$ with $T = 8$, closely matching the undistilled DUO with $T = 1024$, which has a Gen. PPL of $72.05$ and the same entropy ($5.22$). This reduction in steps comes without sacrificing entropy or degradation in sample quality (see `anon. link [3]`).\n# Concern 4: Novelty of Softmax annealing trick\nThe core **novelty of this work is establishing a fundamental connection between discrete and Gaussian diffusion** via the argmax operator. We show that the softmax annealing trick, originally proposed for backpropagating through argmax [1, 2], can be repurposed to design a low-variance training curriculum for USDMs.\n# Concern 5: 2x faster training\nVariance reduction leads to better generalization, as shown by DUO achieving lower val. PPL with Curriculum Learning compared to without it. See our response to Concern 1 for Reviewer 1i8M.\n# Concern 6: Curriculum Learning\nPlease refer to discussions in the response for concern 2 for the Reviewer 1i8M\n# Concern 7: Weak AR Baselines\nThe comparison with the AR baseline **is fair**—DUO, diffusion baselines [5, 7, 8], and the AR Transformer in Tabs. 1–3 all use the same architecture and dataset, with the only difference being causal attention for AR and bidirectional for diffusion models. Notably, in Tab. 3, where **DUO outperforms AR on 3 out of 7 benchmarks, both use the same Transformer architecture**. Omninet and Transformer-XL results in Tab. 2 are included for reference only.\n# Concern 8:  Experimental Design\nWe’d like to emphasize that our experimental setup is consistent with the current literature in the field of diffusion modeling [5, 7, 8].  Our experiments isolate the effect of the training method by fixing the Transformer architecture. While training with modern architectures like GPT-3.5, PaLM, or LLaMA on datasets like Pile or C4 would be valuable, it would require retraining all baselines—an infeasible task for an academic lab.\n# Concern 9: Sampling speed\nSampling from a distilled **DUO model is significantly faster than an AR model**. See responses for concern 2 for Reviewer J6SU.\n\n# Other concerns:\n**Missing citations**: We’ll cite [4] and add a detailed discussion for [1, 2, 4] in the next version of the paper.\n\n**Qualitative Analysis**: We provide more samples in the `anon. link [3]` where we observe that the distilled DUO model outputs **significantly better quality samples** than the distilled MDLM model at lower sampling steps.\n\n---\n\n### References\n[1] Jang et al., 2016 \n\n[2] Maddison et al., 2016 \n\n[3] `Anon. link`: https://docs.google.com/document/d/e/2PACX-1vR0uKDuQHl4bBuC8KokEQhHNMvGdxbIskJm_SfXO_L6haSzWEqjPtL9wkVmg_yacNzMei2DAk21J5XX/pub\n\n[4] Anderson, Continuous-time Markov chains: An applications-oriented approach.\n\n[5] Schiff et al., 2025 “Simple Guidance Mechanisms for Discrete Diffusion Models”\n\n[6] Luo, Simian, et al. \"Latent consistency models\"\n\n[7]  Sahoo et al., 2024 “Simple and Effective Masked Diffusion Language Models”\n\n[8] Lou et al., 2024 “Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution”", "claims_and_evidence": "The paper makes some bold claims, where the below are some points that I think facing some weakness from my point of view:  \n\n(1)  First, the theoretical connection between discrete and continuous diffusion is a cool idea, but I don’t think the paper fully proves that discrete diffusion is fundamentally just a transformed version of Gaussian diffusion. The mapping they derive using argmax feels more like an approximation rather than a deep equivalence. They don’t really show that the key properties (e.g., transition dynamics, loss landscapes) are preserved—just that the marginals line up, which isn’t enough to claim a fundamental link.   \n\n(2) The 2× faster training claim also feels a bit shaky. I was expecting a breakdown of different factors—like, does reducing variance actually translate to better generalization, or just faster convergence to a similar end result? Also, the use of softmax annealing[1][2] is a known trick in discrete optimization, so I’m not convinced this is a novel contribution.   \n\n(3) Then there’s the \"two orders of magnitude\" speedup in sampling—that’s a massive claim, but the evaluation doesn’t fully back it up. The perplexity numbers are nice, but where’s the qualitative analysis? If you’re cutting sampling steps from 1000 to 10, how does that affect fluency and coherence in generated text? I was expecting some real-world comparisons—maybe human evaluations or even error cases where their method struggles.\n\n---\nReference\n--- \n[1] Chen, Binghui, Weihong Deng, and Junping Du. \"Noisy softmax: Improving the generalization ability of dcnn via postponing the early softmax saturation.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.    \n\n[2] Gu, Jiuxiang, et al. \"Exploring the frontiers of softmax: Provable optimization, applications in diffusion model, and beyond.\" arXiv preprint arXiv:2405.03251 (2024).", "methods_and_evaluation_criteria": "I think the methods are well-motivated but not entirely convincing in execution, and the evaluation criteria have some gaps that make it hard to fully trust the conclusions.  \n\nFor the method part, the core idea of leveraging Gaussian diffusion for discrete diffusion models makes sense conceptually, and mapping it through argmax is an interesting perspective. But I’m not sure if it’s the best approach in practice. The claim that this transformation allows discrete models to directly benefit from continuous techniques (like curriculum learning and consistency distillation) is reasonable, but it still feels like a bit of a shortcut rather than a truly fundamental reformulation. I would have liked to see more analysis of when this approximation holds and where it might fail. Right now, it’s mostly taken at face value.\n\nAs for the evaluation, I also think the comparison baselines could be stronger. While they do compare against prior discrete diffusion models (SEDD, UDLM, MDLM), the autoregressive models they use for comparison aren’t the latest state-of-the-art. For example, modern transformer-based LMs like GPT-style models could set a stronger baseline, and without that, it's hard to say if diffusion is actually competitive for practical language modeling.", "theoretical_claims": "I checked the core theoretical claims in the paper, particularly the connection between discrete and continuous diffusion and the evidence lower bound (ELBO) comparisons.    \n\nOne Issue I am a little conern about is that the curriculum learning trick involves annealing argmax to softmax over training steps, supposedly reducing variance and leading to faster convergence. They cite prior work on softmax approximations of discrete gradients, which makes sense.while the proof of variance reduction is missing. They argue that higher τ (temperature) leads to lower variance, but there’s no mathematical analysis quantifying how variance scales with τ. This is just assumed based on intuition from Gumbel-Softmax-like tricks. It would have been more convincing with a variance-bound derivation rather than just empirical loss curves.", "experimental_designs_or_analyses": "I checked the experimental design and analyses, and while the results look promising, there are several issues with the evaluations including the bechmarks and the SOTA method selections:  \n\n(1) This paper evaluates on LM1B and OpenWebText (OWT) for training, and then tests zero-shot generalization on 7 datasets (PTB, Wikitext, etc.). This is reasonable for a language modeling paper, but the OpenWebText dataset is outdated and is not representative of modern large-scale LMs trained on more diverse internet-scale corpora. Comparing to stronger baselines like models trained on The Pile, C4, or real GPT training datasets would have been more informative.    \n\n(2) The baselines are weak. The autoregressive models they compare to (e.g., Transformer-XL, OmniNet) are not the best available. They should have compared against modern SOTA models like GPT-3.5, PaLM, or LLaMA. Beating old baselines doesn’t mean diffusion models are actually competitive.", "relation_to_broader_scientific_literature": "I think this paper makes meaningful contributions to the broader scientific literature on discrete diffusion models, efficient training strategies, and sampling acceleration techniques. It builds on and extends several key ideas from prior work while offering a novel theoretical perspective on the connection between discrete and continuous diffusion.", "essential_references_not_discussed": "As far as I know,  there are several important related works that the paper does not cite including within the aspect of the Softmax Relaxation[1][2],  and sample accelerate[3], list as below.\n\n\n--- \nReference\n---  \n[1] Jang, Eric, Shixiang Gu, and Ben Poole. \"Categorical reparameterization with gumbel-softmax.\" arXiv preprint arXiv:1611.01144 (2016).  \n\n[2] Maddison, Chris J., Andriy Mnih, and Yee Whye Teh. \"The concrete distribution: A continuous relaxation of discrete random variables.\" arXiv preprint arXiv:1611.00712 (2016).   \n\n[3] Luo, Simian, et al. \"Latent consistency models: Synthesizing high-resolution images with few-step inference.\" arXiv preprint arXiv:2310.04378 (2023).", "other_strengths_and_weaknesses": "Strengths:  \n\n- I think this paper proposed a novel connection between discrete and continuous diffusion models, framing discrete diffusion as a transformed version of Gaussian diffusion via an argmax operation. It offers an interesting way to think about discrete generative modeling and provides a new lens for improving training and sampling efficiency.   \n- The experiment results shows that the model can achieves lower perplexity on several benchmarks, including cases where it outperforms autoregressive baselines which I think it with great practical usage consider the ill nature of low speed for diffusions.  \n\nWeakness:  \n\n- The paper claims that diffusion models can compete with autoregressive models, but the baselines used for comparison are outdated. More modern transformers, such as GLaM or GILL, should be included. Beside, I hope the author provide a  runtime comparison between diffusion models and autoregressive models. Even if diffusion models achieve lower perplexity, they may still be much slower at inference.   \n-  The training dataset is mostly OpenWebText, which is outdated. Stronger large-scale benchmarks like The Pile, C4, or GPT-style training datasets should be used for a more comprehensive evaluation.", "other_comments_or_suggestions": "See the Weakness", "questions_for_authors": "（1）For the method part, the author  claims that discrete diffusion naturally arises from a Gaussian diffusion process via an argmax transformation. However, the derivation only shows marginal distribution alignment and does not establish that the transition dynamics and Markov properties of the discrete process are fully preserved. Could you provide a formal proof or additional empirical validation that supports this claim beyond marginal alignment?\n\n（2）As for the evaluation,  I think the Perplexity is promising. since the paper focuses on sampling efficiency while preserving sample quality, why is there no qualitative evaluation of generated text at different sampling steps? Perplexity is a useful metric but does not fully capture fluency, coherence, and grammaticality. Would you consider adding human evaluations or qualitative examples in an updated version?  \n\nI am glad to see the author could resolve my concerns, Thank you.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The manuscript establishes a connection between uniform-state discrete diffusion and Gaussian diffusion and uses the property to improve discrete diffusion. The reviewers found the theoretical contributions novel, the evaluation setup reasonable, and the empirical evaluation supports the claim about improving the performance of USDM. The reviewers requested several clarifications, particularly regarding the loss (including interpreting the training loss and the distinction between training and evaluation loss) and gradient variance.  The relationship between USDM and MDM also raised some concerns, particularly regarding the remaining performance gap between USDM and MDM despite the proposed improvements. The authors addressed most of the reviewers’ concerns during the rebuttal period, and since all reviewers recommended acceptance, I also recommend accepting the paper for ICML. However, given the extension discussion and the number of clarifications, it is crucial that the authors incorporate the promised changes into the final version of the paper.
Decision: Accept (poster)