Think while You Generate: Discrete Diffusion with Planned Denoising
Authors: Sulin Liu
Juno Nam
Andrew Campbell
Hannes Stark
Yilun Xu
Tommi Jaakkola
Rafael Gomez-Bombarelli
Venue: ICLR 2025 Poster (2025)
Keywords: discrete diffusion
generative models
Primary area: generative models
TL;DR: DDPD separates the discrete diffusion generation process into two models: a planner that identifies which positions are corrupted and should be denoised next, and a denoiser that corrects them.

Abstract: Discrete diffusion has achieved state-of-the-art performance, outperforming or approaching autoregressive models on standard benchmarks. In this work, we introduce *Discrete Diffusion with Planned Denoising* (DDPD), a novel framework that separates the generation process into two models: a planner and a denoiser. At inference time, the planner selects which positions to denoise next by identifying the most corrupted positions in need of denoising, including both initially corrupted and those requiring additional refinement. This plan-and-denoise approach enables more efficient reconstruction during generation by iteratively identifying and denoising corruptions in the optimal order. DDPD outperforms traditional denoiser-only mask diffusion methods, achieving superior results on language modeling benchmarks such as *text8*, *OpenWebText*, and token-based generation on *ImageNet 256 × 256*. Notably, in language modeling, DDPD significantly reduces the performance gap between diffusion-based and autoregressive methods in terms of generative perplexity.  Code is available at [github.com/liusulin/DDPD](https://github.com/liusulin/DDPD).

Published as a conference paper at ICLR 2025
THINK WHILE YOU GENERATE: DISCRETE DIFFUSION WITH PLANNED DENOISING
Sulin Liu'; Juno Nam', Andrew Campbell', Hannes Stärk', Yilun Xu', Tommi Jaakkola', Rafael "emezz-ommbarell" Massachusetts Institute of Technology, PUniversity of Oxford, 'NVIDIA Research
ABSTRACT
Discrete diffusion has achieved ofate-of-the-ar performance, outperforming approaching autoregressive models on standard benchmarks. In this work, we introduce Discrete Diffusion with Planned Denoising (DDPD), novel framework that separates the generation process into two models: a planner and a denoiser. At inference time, the planner selects which positions denoise next by identifying the most corrupted positions in need of denoising, including both initially cor- rupted and those requiring additional refinement. This Hlan-and-denoise approach enables more efficient reconstruction during generation by iteratively identifying and denoising corruptions the optimal order. DDPD outperforms traditional denoiser-only mask diffusion methods, achieving superior results on language modeling benchmarks such as text8, OpenWebText, and token-based generation ImageNet 256 x 256. Notably, in language modeling, DDPD significantly reduces the performance gap between diffusion based and autoregressive methods in terms of generative perplexity. Code is available github.com//luuuli/DDDDD
INTRODUCTION
Generative modeling of discrete data has recently seen significant advances across various appli- cations, including text generation [5], biological sequence modeling [25, 4), and image synthesis [8, 9]. Autoregressive transformer models have excelled in language modeling but limited to sequential sampling, with performance degrading without annealing techniques such as nucleus (top-p) sampling [20]. In contrast, diffusion models offer more flexible and controllable generation, proving to be more effective for tasks that lack natural sequential orderings, such as biological sequence modeling [25, 1] and image token generation [8]. In language modeling, the performance gap between discrete diffusion and autoregressive models has further narrowed recently, thanks to improved training strategies [29, 36, 34, 13], however, a gap still remains on some tasks [36, 29].
fate--oftthearr discrete diffusion methods train denoiser (or score) model that determines the transition rate (or velocity) from the current state to predicted values. During inference, the generative process is discretized into finite number of steps. At each step, the state values are updated based on the transition probability, which is obtained by integrating the transition rate over the timestep period.
In order to further close the performance gap with autoregressive models, we advocate for a rethinking of the standard discrete diffusion design methodology. We propose a new framework Discrete Diffusion with Planned Denoising (DDPD), that divides the generative process into two key components: planner and a denoiser, facilitating a more adaptive and efficient sampling procedure. The process starts with a sequence of tokens initialized with random values. At each timestep, the planner model examines the sequence to identify the position most likely to be corrupted and in need of denoising. The denoiser then predicts the value for the selected position, based on the current noisy sequence. The key insight behind our plan-and-denoise approach is that the generative probability at each position can be factorized into two components: 1) planning: the probability that position is corrupted, and 2) denoising: the probability of denoising it according to the data distribution.
Our framework offers two primary advantages:
"Corresponding sulim www.iiiimiiee......

---

Published as a conference paper at ICLR 2025
1. Simplified Learning: By decomposing the generative task into planning and denoising, each sub- task becomes easier to learn. Planning, in particular, is significantly less complex than denoising. Traditional uniform diffusion models, in contrast. require single neural network to handle both tasks simultaneously. which can lead to inefficiencies.
Improved Sampling Algorithm: The plan-and-denoits framework enables an adaptive sampling procedure that dynamically adjusts based on the planner's predictions.
As shown in Fig. 1, this sampling process provides enhanced robustness: 1. Adaptive Time Discretization: When the planner identifies that the sequence noisier than expected at given timestep, it increases the number of denoising moves within the remaining time to ensure thorough refinement. 2. Error Correction: The planner can identify errors introduced in earlier steps. necessary, it adjusts time backward, allowing the denoiser to revisit and correct mistakes. This ensures all corrupted tokens are reconstructed accurately, yielding higher-quality generations.
Step
Step
Scheduled Timestep
7+
0.75
Actual Time
7=0
0.35
0,355
0.6
0.35
0.8
1.0
B
B
BB
B
0
Step
B.
-0
Planner
Pal is corrupted [c)
.9
D
.5
A .1
A. 10.0
.9
.8
.6
Denoiser Rd (x)
0.350.355 0.55
0.8
Figure I: An example generation trajectory from = 0 to 1 of 5-letter sequence. At each step, the planner estimates the prohability of token coruupiion (as indicated by the numbers), selects a position, and the denoiser predicts the token. The actual time progression adaptively re-calibrated by the planner's 's noise level assessment may deviate from the scheduled timestep. For instance, in step 2, minimal improvement results in slower time progression, while in step 4, an error causes a backward step in time. Sampling continues until all corrupted tokens are reconstructed.
Our contributions are as follows:
We introduce Discrete Diffusion with Planning and Denoising (DDPD), a novel framework for discrete generative modeling that decomposes the generation process into planning and denoising.
Our proposed plan-and-denoise framework introduces an adaptive sampling scheme, guided by the planner's output, enabling continuous self-correction of errors in optimal order. This results in improved generation by scaling the number of sampling steps.
We derive simple and effective training objectives for the planner and denoiser models, grounded in maximizing the Evidence Lower Bound (ELBO) for discrete diffusion processes.
In experiments GPT-2 scale language modeling and 256 256 image token generation, DDPD significantly outperforms its mask diffusion counterparts when using the same denoiser. Furthermore, we demonstrate that incorporating planner substantially enhances generation quality, even when using smaller or weaker denoiser model compared to baseline methods.
2 PRELIMINARIES
We begin by introducing the problem setup and notations. Following [7, 13], we then explain the Continuous Time Markov Chain (CTMC) framework [31], which is used to define the forward and reverse processes of discrete diffusion, and how the discrete timestep version is derived from it.
Setup and Notations. We aim to model discrete data where a sequence x E {1,.... SSD is
D-dimensional and each element I takes S possible states. zdd denotes all dimensions except For clarity in presentation, we assume D = 1 and all results hold for D 1 (see Section 3.1). We use p(z) to denote the probability mass function (PMF). The 5 {i,j} is the Kronecker delta function, which is 1 when j and 0 otherwise.
2

---

Published as a conference paper at ICLR 2025
Continuous Time Markov Chains and Discretization. We adopt the CTMC framework to define the discrete diffusion process. A realization of the CTMC dynamics is defined by a trajectory 24 over time E [0, 1] that makes jumps another state after random waiting period known as the holding time. The transition rates and the next states are governed by the rate matrix Rg E RSSS (analogous to the velocity field v, in continuous state spaces), where the off- off-diagonal elements, representing jumps to different states, are non-negative. For an infinitesimal timestep dt, the probability of transitioning from 24 different state j is given by (
In practice, the trajectory is simulated using finite time intervals At. As a result, the transition probability follows categorical distribution with the PMF [39]:
P( = (mt,5) Rs (x j)
where we denote this as Cat(8 {m,,5} + (a.x, and R₁ Ro 21) == -ESZE Rd (e 8) to ensure that the transition probabilities sum to 1.
Forward Corruption Process. Following [7, 34, 131, which were inspired by flow matching in
continuous state space [26, 27, 2]. we construct the forward process by interpolating from noise Po(#a) ( to clean data pi (x1) (21). Common choices for the noise distribution include: (i) puode (1z2) 1/s, uniform distribution over {1.... S}; and (ii) p = å {M, xt J. a delta PMF concentrated on an artificially introduced mask state MI. Let o, be the noise schedule that introduces noise to the data over time t. For example, linear schedule is given by a, t. The conditional marginal Pen (Iz|æ11 is given in closed form:
Pil1 (ze (x1) Cat(a,d {æ1. (1 an) 1
(2)
Pij1 (x, [x1) Cat(ago {x1, + (1 a,) x1)).
mask
(3)
At t 1, the conditional marginal converges to the datapoint æ1, i.c. 8 (11,2e). At t 0, the conditional marginal converges to the noise distribution Pnoise(2t))
Sampling from Pdata is achieved by learning generative rate matrix (x,,,,) j) to reverse simulate the process from 0 to = 1 using Eq. (1), such that we begin with samples of Pnoise and end with samples of Pdata- The datapoint conditional reverse rate Re (1,,, for j F xt under the uniform or mask noise distributions given by [7, 13]:
Reverse Generation Process.
Ruuiit (#erjlar)
ái
[ (1-6 (xi, 2x)), remask (11))(z1)
Ce 8 (x1, j. (xx, M).
att
[7, 13] show that the rate we aim to learn for generating Pdata is the expectation of the data-conditional rate, taken over the denoising distribution, i.e., Re ()) E (pple))) [Re ([,,)))))
R. (z..)) = di a Pije (x1 = jlz,), Rmaak (I..
di 8 (o M|p. (21 = j(æ,). (4)
at
The goal is to approximate this rate using a neural network. During inference, the generative process is simulated with the learned rate by taking finite timesteps, described in Eq. (1).
3 METHOD
3.1 DECOMPOSING GENERATION INTO PLANNING AND DENOISING
Recent state-of-the-ar discrete diffusion methods [39, 7, 36, 34, 13] have converged on parameter- izing the generative rate using denoising neural network and deriving cross-entropy- training objectives. This enables simplified and effective training, leading to better and SOTA performance in discrete generative modeling compared to earlier approaches [3, 6]. In Table we summarize the commonalities and differences in the design choices across various methods.
Based on the optimal generative rate in Eq. (4), we propose a new approach to parameterizing the generative rate by dividing it into two distinct components: planning and denoising. We begin
For simplicity, we only derive the rates for * The rate for j can be computed as R2 (T, (1(11)))))1)) == Exje, Re Re 8/21)-
3

---

Published as a conference paper at ICLR 2025
Table Specific design choices employed by different discrete diffusion models.
SDDM [39]
SEDD [29]
DFM / Discrete FM [13) MDLM M34//M44 [36] Ours ('DDPD")
Sampling (Section 3.2)
Method
Tau-leaping
Tau leapieg
Tau -aa-leapng VW
Tau-leaping
Adaptive Gillespie te(2.) + 7. 5 ~ Exp (Ao(z.)
Time steps
1/x
1/w
f/x
Noise schedule di
ar (D)
dg (-)
Generative rate parameterizution (Section 3.1)
Musking
8 [x1.M)
or or
d///// of Po [af = 2z)
Do (15 slmz))
Uniform
di/s 4i/1-opposff of = N(zz)) De (r) - slace N)
Son of
(xf f(m))
of [
DFM assumes a linear schedule. MD4 also supports learnable schedule of cx. Aa()) 3 is the total rate of jump determined by the planner.
by examining how the generative rate in mask diffusion can be interpreted within our framework, followed by derivation of the decomposition for uniform diffusion.
Mask Diffusion.
For mask diffusion, the planning part is assigning probability of I-ar ô {xx, M)At for the data to denoised with an actual value. {ar, tells the data noisy (MI clean. T-o, is the rate of denoising, which is determined by the remaining time according to the noise schedule. The denoiser Pijt assigns probabilities to the possible values to be filled in if this transition happens.
de a, (2 At Pije (11 j|m()
Rimaak ()+
prob. of denoising
rate making correction
Uniform Diffusion. Similarly, we would want to decompose the transition probability into two parts: the planning probability based on if the data is corrupted and the denoising probability that determines which value change to. But in contrast the mask diffusion case, the noise/clean state of the data is not given to us during generation. We use 2 (N, D) as latent variable to denote if a dimension is corrupted, with N denoting noise and denoting data.
From Bayes rule, for j # Tt. since Pije (I1 = j/ 2t = D) = pla//t/=//s===p((n
Pije (x)
j|xr))
Σ
p(aelt))Pllet . j)),; p(= N Pijp = j| 2t
N)
A (N,D)
prob. being comrupted
proh. of denoising
(6)
The first part of the decomposition the posterior probability of x, being corrupted, and the second part gives the denoising probability to recover the value of I, if I, is corrupted. Plugging Eq. (6) into Eq. (4), we arrive at:
di p(u N(vr)) At p j|zi, N)
Rumit (It,j)At
1- 04
prob. denoising
rate. of making correction
di which By comparing Eq. (7) and Eq. (5), we find that they share the same constant part T-o; represents the rate determined by the current time left according to the noise schedule. The main difference is in the middle part that represents the probability of z; being corrupted. mask diffusion case, this can be readily read out from the M token. But in the uniform diffusion case, we need to compute/approximate this probability instead. The last part the denoising probability conditioned on x, being corrupted, which again is shared by both and needs to be computed/aprooximaee.
The above mentioned decomposition extends D We have the following reverse generative rate [7, 13] for mask diffusion:
Generative Rate for Multi-Dimensions.
Rmack (It, ja) ) 1- az
de At ( M1 (o = 1 vs4 x
(8)
4

---

Published as a conference paper at ICLR 2025
and we derive the following decomposition result for uniform diffusion (proof in Appendix A.1): Proposition 3.1. The reverse generative rate at d-th dimension can be decomposed into the product of recovery rate, probability of corruption and probability of denoising:
ái Ri (21,j") At = I-or Pije (xf = )=1 (alz)/a At
(9)
p ( N Pije = j N) Dt,
Vja
r
noise remavaal rate proh. of corruption
prohh of deroossing
where
p ( N(x+)) - -P11 (19
x/) (
at at a1+(1-a
(10)
Pile (2f = 12, 24 = N) = Pr (=f
(11)
n(=
We observe that the term (=f N[ determines how different dimensions are reconstructed different rates, based on how likely the dimension is clean or noise given the current context.
Previous Parameterization. In the case of mask diffusion, studied in recent works [29, 36, 34, 13], the most effective parameterization for learning is to directly model the denoising probability with neural network, this is the only component that needs be approximated.
In the case of uniform diffusion, the conventional approach uses a single model to approximate the generative rate as a whole, by modeling the posterior Pije (xf = [ as shown in Eq. (9). However, despite its theoretically greater flexibility allowing token values to be corrected throughout sampling, akin to the original diffusion process in the continuous domain its performance has not always outperformed mask diffusion, particularly in tasks like image or language modeling.
Tlan-and-Denoftee Parameterization. Based on the observation made Proposition 3.1, we take the view that generation should consist of two models: planner model for deciding which position to denoise and denoiser model for making the denoising prediction for a selected position.
Rujpp (z), At =
d'
At Pe
(2 = N Naa)) Nas)
Pij (#² =j= 1a N). Vxr j (12)
planner
denoiser
noise removal rate
This allows us utilize the planner's output to design an improved sampling algorithm that optimally identifies and corrects errors in the sequence in the most effective denoising order. Additionally, the task decomposition enables separate training of the planner and denoiser, simplifying the learning process for each neural network. Often, a pretrained denoiser is already available, allowing for computational savings by only training the planner, which is generally faster and easier to train.
Remark 3.2. Under this perspective, masked diffusion (Eq. (8)) can be interpreted as denoiser-only modeling paradigm with a fixed planmer, i.e., Pe (zf N(21) mask tokens represent noise while actual tokens represent clean data. This planner optimal under the assumption of perfect denoiser, which rarely holds in practice. When the denoiser makes errors, this approach does not provide a mechanism for correcting those mistakes.
8 xf M). which assumes that
Next, we demonstrate how our llan-and-denel framework enables improved sampling algorithm that effectively leverages the planner's predictions. From this point forward, we assume uniform diffusion by default and use Re,uump to denote the reverse jump rate, unless explicitly stated otherwise.
3.2 SAMPLING
Prior Works: Tau-leaping Sampler. The reverse generative process is CTMC that consists of a sequence of jumps from i = 0 to f The most common way [6. 39, 29, 36, 34, 13] is to discretize it into equal timesteps and simulate each step following the reverse generative rate using an approximate simulation method called tau-lcaping. During step [t, t At], all the transitions happening according to Eq. (I) are recorded first and simultaneously applied at the end of the step. When the discretization is finer than the number of denoising moves required, some steps may be wasted when no transitions occur during those steps. In such cases when no transition occurs during it -At, t], a neural network forward pass can be saved for step [t. t + At by using the cached (ije(aff (re-ar) from the previous step [10, 34], assuming the denoising probabilities remain
5

---

Published as a conference paper at ICLR 2025
tt. However, as discussed in literature [6, 39, 29, 71, such modeling of the reverse denoiser is predicting single dimension transitions but not joint transitions of all dimensions. Therefore, the tau-leaping simulation will introduce approximation errors if multiple dimensions changed during the same step.
unchanged during [t
Gillespie Sampler. Instead, we adopt the Gillespie algorithm [15, 16, 41], simulation method that iteratively repeats the following two-step procedure: (1) sampling At, the holding time spent at current state until the next jump and (2) sampling which state transition occurs. In the first step, the holding time At is drawn from an exponential distribution, with the rate equal to the total jump rate, Re, jump (I). For the second step, the event at the jump is sampled according to Rr, Re.jump (xe,j")/ Re. (xaa such that the likelihood of the next state is proportional to the rate from the current position. A straightforward way is using ancestral sampling by first selecting the dimension d, followed by sampling the value to jump ju:
defined as the sum of all possible jump rates in Eq. (12): (z)) := Ex
d~ Cat (Ejeyor Re jump ( ( / (x))) jdt
Cat (Rejump (Di, (0),))) Ejayet Re (BL,
We find we can simplify the calculation of the total jump rate and next state transition by introducing the possibility of self-loop jumps into the CTMC. These allow the trajectory to remain in the current state after a jump occurs. This modification results in an equivalent but simpler simulated process with our plan-and-denoise method, formalized in the following proposition:
Proposition 3.3. The original CTMC defined by the jump rate Remjump given by Eq. (12) has the same distribution over trajectories as the modified self-loop CTMC with rate matrix
Ba( =
ds I-ai po (2) NN = j of N),
Vz
For this self-loop Gillespie algorithm, the total jump rate and next state distribution have the form
Ea.s Re T- Expo -2x))) d Cat (po(e) (N(a)) jåd Cat (vi j N)).
Intuitively, the modification preserves the inter-state jump rates, ensuring that the distribution of effective jumps remains unchanged. A detailed proof is provided in Appendix A.3.
Remark 3.4. The Gillespie algorithm sets At adaptively, which is given by the holding time until the next transition. This enables a more efficient discretization of timesteps, such that one step leads to one token denoised (if denoising is correct). In contrast, tau-leaping with equal timesteps can result in either no transitions or multiple transitions within the same step. Both scenarios are suboptimal: the former wastes a step, while the latter introduces approximation errors.
Adaptive Time Correction. According to the sampled timesteps At, the sampling starts from noise at i and reaches data 1. However, in practice, the actual time progression be faster or slower than scheduled. For example, sometimes the progress is faster in the beginning when the starting sequence contains some clean tokens. More often, later in the process, the denoiser makes mistakes, and hence the time progression is slower than scheduled or even negative for some steps.
This raises the question: can we leverage the signal from the planner to make adaptive adjustments? For example, even if scheduled time reaches t 1.0, but according to the planner 10% of the data remains corrupted, the actual time progression under a linear schedule should be closer to t t= 0.9. The reasonable approach is to assume that the process is not yet complete and continue the plan-and-
sampling. Under this 'time correction' mechanism, the stopping criterion is defined as continuing the sampling procedure until the planner determines that all positions are denoised, ie., when po (zí = NN WIx:) R 0. In practice, we don't 't need to know the exact time; instead, we can continue sampling until either the stopping criterion is satisfied or the maximum budget of steps, T. is reached. The seuuo-allorithh
for our proposed sampling method is presented in Algorithm 1. In cases where the denoiser use time information input, find it helpful use the estimated time from the planner. At time I, from the noise schedule, we expect there to be (1 11o0 o22) D noised positions. The estimate of the number of noised positions from the denoiser is Ev (2 = N[xx). Therefore, the planner's is estimate of the corruption time is 7 1, Ex po N(

---

Published as a conference paper at ICLR 2025
where ai is the inverse noise schedule. This adjustment better aligns the time-data pair with the distribution that the denoiser was trained on.
Based on the decomposition of the generation into planning and denoising, the proposed sampling method maximally capitalizes on the available sampling steps budget. The Gillespie-based plan-and- denoise sampler allows for exact simulation and ensures step is wasted by prioritizing the denoising of noisy tokens first. The time correction mechanism enables the planner to identify both initial and reintroduced noisy tokens, continuously denoising them until all are corrected. This mechanism shares similarities with the stochastic noise injection-correction step in EDM [22]. Instead of using hyperparameters for deciding how much to travel back, our time correction is based on the planner's estimate of the noise removal progress.
Algorithm DDPD Sampler
I: init 0, Po, planner po, denoiser pi maximum steps stopping criteria while or p=(2 N M22.)) e, Vd do
3: Plan sample dimension d ~ Cat (polz²" (2=2" Nozx)
4: if denoiser uses time input then
5: tta;"" Es po(=f" N)
6: Denoise sample jd Cat (piican = 1M| = N1). 2bt1 jd
7: iti+1
return xt.
Utilizing Pretrained Mask Diffusion Denoiser. In language modeling and image generation, mask diffusion denoisers have been found to be more accurate than uniform diffusion counterparts [3, 29], with recent efforts increasingly focused training mask diffusion denoisers [36, 34, 13]. The following proposition offers a principled way to sample from the uniform denoiser by leveraging strong pretrained mask diffusion denoiser, coupled with a separately trained planner.
Proposition 3.5. From marginalization over 24, which indicates if tokens are noise or data:
Pile (oilEs, 2f = N) = Ez, (Zllte, 2 = N) Pije(allaa 2t),
(13)
samples from unifouiffomm: and then using a mask diffusion denoiser to sample x with Pitt.mask (z1lzz). where on is the masked version of x, according to 2t-
( 24, N) can be drawn by first sampling 2, from p(ala of N)
In practice, we can approximately sample 2 from p(z 25 = N) Re (2 (mr). This approximation becomes exact if p(2f (xre) either very close or 1, which holds true for most dimensions during generation. We validated this holds most of time in language modeling in Appendix E.5. Even if approximation errors in 2, occasionally lead increased denoising errors, our sampling algorithm can effectively mitigate this by using the planner to identify and correct these unintentional errors. our controlled experiments, we validate this and observe improved generative performance by replacing the uniform denoiser with mask diffusion denoiser trained on the same total number of tokens, while keeping the planner fixed.
4 TRAINING
Training objectives. Our plan-ant and-denoise parameterization in Eq. (12) enables us to use two separate neural networks for modeling the planner and the denoiser. Alternatively, both the planner and denoiser outputs can be derived from single uniform diffusion model, p (xf((1r)) (x:). as described in Proposition 3.1. This approach may offer an advantage on simpler tasks, where minimal approx- imation errors for neural network training can be achieved, avoiding the sampling approximation introduced in Proposition 3.5. However, in modern generative AI tasks, training is often constrained by neural network capacity and available training tokens, making approximation errors inevitable. By using two separate networks, we can better decompose the complex task, potentially enabling faster training especially since planning is generally easier than denoising.
7

---

Published as a conference paper at ICLR 2025
The major concern with decomposed modeling is that joint modeling could introduce unnecessarily coupled training dynamics, hindering effective backpropagation of the training signal across different models. However, diffusion decomposes neatly, allowing the use of direct training signals from the noise corruption process for independent training of the planner and denoiser (proof in Appendix A.2).
prove in Theorem 4.1, the evidence lower bound (ELBO) of discrete
Theorem 4.1. Let 2 be a clean data point, and It. 2, represent a noisy data point and its state of corruption drawn from Pet1 (It, 24 (I)). The ELBO for uniform discrete diffusion simplifies into the sum of the following separate pross-entropy-typee objectives. The training of the planner pa reduces to a binary classification, where the goal is to estimate the corruption probability by maximizing:
Cplamer Eu(t;,01)) (oaatt,,)) (aaaaaaaappeeeeeeeee (a
Ed= log pe (zf (z4
(14)
The denoiser Pipe is trained to predict clean data reconstruction distribution:
dennsser E (w: 2e/1)
I-ai Ed= o( N) log log (xfle,, -N)] (15)
M].
Standard transformer architectures can be used to parameterize both the denoiser and the planner, where the denoiser outputs S logits and the planner outputs single logit per dimension.
RELATED WORK
Discrete Diffusion/Flow Models. Previous discrete diffusion//low methods, whether in discrete time [3, 21, 34] or continuous time [6, 39, 29, 7, 36, 13], adopt the denoiser-only or score-modeling perspective. In contrast, we introduce a theoretically grounded decomposition of the generative process into planning and denoising. DFM [7] and the Reparameterized Diffusion Model (RDM) [46] introduce stochasticity into the reverse flow/diffusion process, allowing for adjustable random jumps between states. This has been shown to improve the generation quality by providing denoiser more opportunities to correct previous errors. Additionally. RDM uses the denoiser's 's prediction confidence as a heuristic [14, 35, 8] for determining which tokens denoise first. Lee et al. [24] introduces another hcuristic in image generation that aims at correcting previous denoising errors by first generating all tokens and then randomly regenerating them in batches.
Coff-Crrection Sampling. Predictor-correcte sampling methods are proposed for both continu- ous and discrete diffusion [37, 6] that employ MCMC steps for correction after each predictor step. However, for continuous diffusion, this approach has been found to be less effective compared to the noise-injection stochasticity scheme [22, 42]. In the case of discrete diffusion, an excessively large number of corrector steps is required, which limits the method's overall effectiveness.
6 EXPERIMENT
Before going into details, we note that DDPD incurs 2 NFE v.s. 1 NFE per step in denoiser-only approaches, extra we pay for planning. To ensure fair comparison, we also evaluate denoiser- only methods that are either 2x large or use 2x steps. Our findings show that spending compute on planning more effective than doubling compute on denoising when cost a factor.
Text8. We first evaluate DDPD on the small-scale character-level text modeling benchmark, text8 [30], which consists of 100 million characters extracted from Wikipedia, segmented into chunks of 256 letters. Our experimental setup follows that of [7]. Methods for comparison include I) autoregressive model 2) DFM: discrete flow model (and 2 param. version) [7], the best available discrete diffusion/flow model for this task, 3) DFM-Uni: original DFM uniform diffusion using tau-leaping, 4) DDPD-DFM-UM DDPD using uniform diffusion model as planner and denoiser, 5) DDPD-UniD: DDPD with separately trained planner and uniform denoiser, 6) DDPD-MaskDD DDPD with separately planner and mask denoiser. Details the sampling differences are summarized in Table 3. All models are of same size (86M) and trained for 750k iterations of batch size 2048, except for autoregressive model, which requires fewer iterations to converge. Generated samples evaluated using the negative log-likelihood (NLL) under the larger language model GPT-J-6B [40]. Since NLL can be manipulated by repeating letters, we also measure token distribution entropy. High-quality samples should have both low NL.L and entropy values close to the data distribution.
8

---

Published as a conference paper at ICLR 2025
Fig. 2 shows the performance of various methods with different sampling step budgets. DFM methods use tau-leaping while DDPD methods use our proposed adaptive Gillespie sampler. The original mask diffusion (DFM. 77 0) and uniform diffusion (DFM-Uni) perform similarly, and adding stochasticity (n) = 15) improves DFM's sample quality. Our proposed plan-and-denoise DDPD sampler consistently enhances the quality vs. diversity trade-off and significantly outperforms significantly outperforms DFM with 2x parameters. Moreover, DDPD makes more efficient use of the inference-time budget (Fig. middle), continuously refining the generated sequences.
We observe that DDPD with a single network (DDPD-DFM-UN)) outperforms using separately trained planner and denoiser, as the task simplicity allows all models to achieve 2 90% denoising accuracy at t = 0.85. The benefit of reducing each model's ' burden is outweighed by compounded approximation errors (Section 3.2). The weaker performance of r-leaping (PxMaskD confirms this issue lies in approximation errors, not the sampling scheme. To emulate a practical larger- scale task where the models are undertrained due to computational or capacity budget limitations, we reduced training steps from 750k to 20% (Fig. 2, right). With only 20k steps, using separate planner and denoiser performs comparably to DFM at 750k, while the single model suffers mode collapse due to larger approximation errors, highlighting the benefits of faster separate learning.
Fig. 7)
Further ablation studies on imperfect training of either the planner or denoiser (Figs. 5 and 6) show that performance remains robust to varying levels of denoiser imperfection, thanks to the self-correction mechanism in the sampling process. An imperfect planner has a greater impact, shifting the quality-diverssy Pareto front. In this case, training separate models proves more robust in preserving diversity and preventing mode collapse compared to training a single model. The ablation in Fig. examines the individual effects of the modifications introduced in the DDPD sampler. We also measure the denoising error terms in the ELBO for the planner + denoiser setup vs. the single neural network approach, as shown in Tables 4 to 6 of Appendix E.1.3, which further validates the performance difference of various design choices. In Appendix E.2, we tested how quickly the sampling converges and reported statistics on how many dimensions are adjusted during the process.
700
Nagatha Lag-ihelleod.
Negattee Ukktihood
03
6-
1.0
DRM g-0
-x- DEM-UN
Q=00
q=
-8-
DRM ( 15
- DA
Neguitre
EOPO-TIOM
DRM22 0
UD
DRM 2x. q- 15
REP34DvDD
13
DRM 7- 0
D3PD-DFM--N
A.tallrrenke
D3PD-Maxc
72
Eniropy
Entropy
Entrapy
Figure 2: Negative log-likelihood measured with GPT-J versus sample entropy (in terms of tokens), with logit temperatures of the denoiser swept over (0.8, 0.9, 1.0). Left: DDPD v.s. SÖTA baselines. Middle: Varying sampling steps from 256 to 1024 both DFM and DDPD use the same mask-based denoiser. Right: DDPD single-neural-network v.s. DDPD planner + mask denoiser, both trained for 20k iterations. DFM at 750k iter.
Open Web Text Language Modeling. In Fig. 3, compare DDPD with SEDD [29], both trained on the larger Open Web'lext dataset [18]. We maintained the same experimental settings as in SEDD, with token vocabulary size S 50257 and D 1024, to validate whether planning improves generative performance under controlled conditions. We use the same pretrained SEDD-small or SEDD-medium score model a mask diffusion denoiser, based on the conversion relationship outlined in Table 1. A separate planner network, with the same configuration as SEDD-small (90M), trained for 400% iterations with batch size 512. We evaluated the quality of unconditional samples using generative perplexity, measured by larger language models GPT-2-L (774M) [33] and GPT-J (6B) [40]. Both SEDD and DDPD were simulated using 1024 to 4096 steps, with top-p 1.0. SEDD used tau-lcaping, while DDPD employed
newly proposed sampler. We also include GPT-2 [33] as the autoregressive baseline, with top-p sweeping from 0.7 to 1.0. We experimented DDPD sampler with both softmax selection (Fig. 9) and proportional selection (Fig. 10).
In contrast to SEDD, DDPD leveraged planning to continuously improve sample quality, with the highest improvements in the early stages and diminishing returns in later steps as the sequence is largely corrected. This shows that the planner optimally selects the denoising order and adaptively corrects accumulated mistakes.

---

Published as a conference paper at ICLR 2025
DDPD-Small, Top-p-1.0
Tep-pr1
DOPD Medium,,, Top-p-1.3
p-To
Gen Gen. Perplexity (GPT-)
120
SmDllSmull 1op
GPT-2 Small
www.peeppptttyyo (GP)
$180
SEDD-Medium Top-p-l.0
GPT-2-Medlum
Data Entrapy
Entropy
20
30-
+T-3072
7-2008
Tw-0066
T-1024
15
1-408902 p-2.8
2043
alop-00..5
20-
1-4226
+T:33022
Toop p-08
"rap-prons
10
62
6.6
7.0
74
7.8
80
52
10
62
6.6
7.0
74
7.8 8.0
82
Entropy
Entropy
Figure 3: Generative perplexity v.s. entropy (both plotted in log-scale) of SEDD, DDPD and GPT-2.
ImageNet 256 256 Generation with Discrete Tokens. We represent images with discrete-valued tokens using a pre-trained tokenizer and decoder from Yu et al. [45], resulting in token length D 128. Our focus is comparing DDPD against existing sampling methods rather than achieving state-of-the-ar performance.
We compare DDPD to two mask diffusion baselines: standard mask diffusion and MaskGIT [8]. MaskGIT selects tokens based denoiser confidence logits with annealed random noise to avoid overly greedy selections. Experiments span from to 128 sampling steps, employing parallel sampling for steps below 128. Results measured by FID scores [19] appear in Table with additional metrics and samples in Appendices E.4 and E.3. Sampling in DDPD consists of two stages: parallel sampling followed by adaptive, noise-level-baset refinement.
Standard mask diffusion performs poorly due to low denoiser accuracy (3% vs. 60% for language modeling t 0.85). MaskGIT achieves better results but sacrifices diversity for quality, becoming excessively greedy at higher steps. Conversely, DDPD achieves optimal results once sufficient steps correct etrly-stage errors without degrading FID higher steps. Ablation studies on refinement steps are in Fig. 12.
We tested all methods with logit temperature annealing, finding T 0.6 optimal among tested values, significantly enhancing standard mask diffusion but reducing diversity and increasing FID in MaskGIT. Another annealing method from Yu et al. [45], linearly reducing from 1.0 to 0.0, further improved mask diffusion but had minimal impact DDPD. Additional evaluations with heuristics and classifier-free guidance are discussed in Appendix E.4.1. Notably, DDPD the only method that performs consistently with or without the additional heuristics CFG.
Table 2: FID score (1) on ImageNet 256 256. MaskD refers to mask diffusion. The denoiser and parallel sampling schedule kept the same [45], without classifier free guidance.
No Logit Annealing MaskGIT
Logit temp 0.6
Logit temp 1.0 -> 0.0
Steps T'
MaskD
DDPD
MaskD
MaskGIT
DDPD
MaskD
MaskGIT DDPD
8
38.06
5.51
6.8
5.69
10.02
5.71
4.99
8.53
5.99
16
32.44
6.66
5.12
4.85
11.24
4.92
4.69
9.21
5.03
32
29.12
8.09
4.75
4.86
11.93
4.91
4.62
9.9
4.98
64
27.54
4.73
4.98
12.26
5.14
4.6
10.35
5.26
128
26.83
9.34
4.89
5.13
12.52
5.39
4.89
10.2
5.54
CONCLUSION
We introduced Discrete Diffusion with Planned Denoising (DDPD), novel framework that de- composes the discrete generation process into planning and denoising. We propose new adaptive sampler that leverages the planner for more effective and robust generation by adjusting time step sizes and prioritizing the denoising of the most corrupted positions. Additionally, it simplifies the learning process by allowing each model to focus specifically on either planning or denoising. The incorporation of planning makes the generative process more robust during generation. On GPT-2 scale language modeling and ImageNet 256 x 256 token generation, DDPD enables significant performance boost compared to denoiser-only discrete diffusion models.
made by the denoiser
10

---

Published as a conference paper at ICLR 2025
8 REPRODUCIBILITY STATEMENT
To facilitate reproducibility, we provide comprehensive details of our method in the main paper and Appendix D. This includes the model designs, hyper-parameters training, sampling schemes and evaluation protocols for all the experiments. We further provide PyTorch pseudocode for the proposed adaptive Gillespie sampling algorithm.
ETHICS STATEMENT
This work raises ethical considerations common to deep generative models. While offering potential benefits such generating high-quality text/image contents, these models can also be misused for malicious purposes like creating deepfakes or generating spam and misinformation. Mitigating these risks requires further research into guardrails for reducing harmful contents and collaboration with socio-technical experts.
Furthermore, the substantial resource costs associated with training and deploying deep generative models, including energy and water consumption, present environmental concerns. This work can save cost training by reusing pre-trained denoiser and just focusing training different planner models for slightly different tasks. At inference time, our newly proposed sampler able to generate at better quality compared to existing methods that use amount of compute.
ACKNOWLEDGMENTS
We thank Jiaxin Shi for valuable discussions on evaluation of ELBO and anonymous reviewers for their helpful feedback. SL and RGB acknowledge funding from MIT-IBM Watson AI Lab and NSF Award 2209892. JN acknowledges support from Toyota Research Institute. AC acknowledges support from the EPSRC CDT Modern Statistics and Statistical Machine Learning (EP/S023151/I). This research used compute resources of the National Energy Research Scientific Computing Center (NERSC), Department of Energy Office Science User Facility using award GenAI@NERSS m4737.

---

Published as a conference paper at ICLR 2025
REFERENCES
[1] Sarah Alamdari, Nitya Thakkar, Rianne van den Berg, Alex X Lu, Nicolo Fusi, Ava P Amini, and Kevin K Yang. Protein generation with evolutionary diffusion: sequence is all you need. BioRxiv, pp. 2023-09, 2023.
(page 1)
[2] Michael S Albergo and Eric Vanden-Eijnden Building normalizing flows with stochastic interpolants. International Conference Learning Representations, 2023.
(page 3)
[3] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 2021.
(pages 3, 7, 8, and 21)
[4] Pavel Avdeyev, Chenlai Shi, Yuhao Tan, Kseniia Dudnyk, and Jian Zhou. Dirichlet diffusion score model for biological sequence generation. In International Conference on Machine Learning, 2023.
(page 1)
[5] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. In Advances in neural information processing systems, 2020.
(page I)
[6] Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. Advances in Neural Information Processing Systems, 2022.
(pages 3, 5, 6, 8, and 20)
[7] Andrew Campbell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. Gener- ative flows on discrete statesspaces: Enabling multimodal flows with applications to protein co-design. International Conference on Machine Learning, 2024. (pages 2, 3, 4, 5, 6, 8, 16, 20,
21, and 23)
[8] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision (pages 1, 8, 10, 26, and 32)
and Pattern Recognition, pp. 11315-11325, 2022.
[9] Huiwen Chang, Han Zhang, Jarred Barber, Aaron Maschinot, Jose Lezama, Lu Jiang, Ming- Hsuan Yang, Kevin Patrick Murphy, William T Freeman, Michael Rubinstein, al. Muse: Text-to-image generation via masked generative transformers. In International Conference on Machine Learning, 2023.
(page 1)
[10] Zixiang Chen, Huizhuo Yuan, Yongqian Li, Yiwen Kou, Junkai Zhang, and Quanquan Gu. Fast sampling via de-randomization for discrete diffusion models. arXiv preprint 13X192231..09193, 2023.
(page 5)
[11] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 1188780-8794, 2021.
(page 26)
[12] Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolutie image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12873-12883, 2021.
(page 32)
[13] Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky TQ Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching. arXiv preprint arXiv:2407.15595 2024. (pages) 1, 2, 3, 4, 5, 7, and 8)
[14] Marjan Ghazvininejad, Omer Levy. Yinhan Liu, and Luke Zettlemoyer. Mask-predict: Parallel
decoding of conditional masked language models. In EMNLP-IJCÄLP. 2019.
(page 8)
[15] Daniel T Gillespie. A general method for numerically simulating the stochastic time evolution of coupled chemical reactions. Journal of computational physics, 22(4):403-434 1976. (page 6)
[16] Daniel T Gillespie. Exact stochastic simulation of coupled chemical reactions. The journal of physical chemistry, 81(25):2340-255 1977.
(page 6)
12

---

Published as a conference paper at ICLR 2025
[17] A. Gokaslan and V. Cohen. Openwebtext corpus. OpenkebTexttCrppas, 2019.
http://8kkyloon007, thub 1o/
(page 26)
[18] Aaron Gokaslan and Vanya Cohen. OpenWebText Corpus, 2019. URL http://Skkllonn00% github. http/ppenneeTTettorppp...
(page 9)
[19] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge local Nash equilibrium. Advances in neural information processing systems, 30, 2017.
(pages 10 and 26)
[20] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2019. (page I)
[21] Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling, Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems, 34:12454-12465 2021.
(page 8)
[22] Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine.
Elucidating the design space of diffusion- generative models. Advances in neural information processing systems, 2022. (pages and 8)
[23] Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11452-11461, 2022.
(page 32)
[24] Doyup Lee, Chiheon Kim, Sachoon Kim, Minsu Cho, and WOOK SHIN HAN. Draft-and-revise: Effective image generation with contextual rq- rq-transformer. Advances in Neural Information Processing Systems, 35:30127-300 2022.
(page 8)
[25] Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, al. Exolutionary--calee prediction of atomic-level protein structure with a language model. Science, 173 2023.
(page I)
[26] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. International Conference on Learning Representations, 2023.
(page 3)
[27] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning generate and transfer data with rectified flow. International Conference on Learning Representations, (page 3)
2023.
[28] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17X1v:1711.05101, 2019.
(pages 23 and 26)
[29] Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion language modeling by estimating the ratios of the data distribution. International Conference on Machine Learning. 2024.
(pages I, 3, 4, 5, 6, 7, 20, 21, 24, and 26)
[30] Matt Mahoney. Large text compression benchmark, 2011. URL httt:tt/matttmahommmet .net/ (page 8)
dc dc/textdata. html.
[31] James R Norris. Markov chains. Number 2. Cambridge university press, 1998.
(page 2)
[32] William Peebles and Saining Xie. Scalable diffusion models with transformers, 2023, (page 24)
[33] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 188)99 2019. (pages 9, 24, and 26)
[34] Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. arXiv preprint 4rX1v:2406.07552, 2024. (pages 1, 4, 5, 7, 8, and 21)
13

---

Published as a conference paper at ICLR 2025
[35] Nikolay Savinov, Junyoung Chung, Mikolaj Binkowski, Erich Elsen, and Aaron van den Oord. Step-unrolled denoising autoencoders for text generation. In International Conference on Learning Representations, 2022.
(page 8)
[36] Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and generalized masked diffusion for discrete data, arXiv preprint wrXi:22406.04329, 2024. (pages 1,
3, 4, 5, 7, and 21)
[37] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations, 2020.
(page 8)
[38] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
(page 24)
[39] Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based continuous- time discrete diffusion models. International Conference on Learning Representations, 2023. (pages 3, 4, 5, 6, 8, and 20)
[40] Ben Wang and Aran Komatsuzaki. GPT-J-6B: A billion parameter autoregressive lan- guage model, 2021. URL https://githht: com http:/nnoooolo/eeeeeeee traeeshhtranss ormer-jax. (pages 8, 9, 24, and 26)
[41] Darren J Wilkinson. Stochastic modelling for systems biology. Chapman and Hall/CRC, 2018.
(page 6)
[42] Yilun Xu, Mingyang Deng, Xiang Cheng, Yonglong Tian, Ziming Liu, and Tommi Jaakkola. Restart sampling for improving generative processes. Advances Neural Information Process- ing Systems, 36:76806-76836 2023.
(page 8)
[43] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang. James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vegan. In International Conference Learning Reprexentations (ICLR), 2022. (page 32)
[44] Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats difkussionttkenizzer is key to visual generation. In International Conference on Learning Representations (ICI.)) (ICI.R, 2024.
(page 32)
[45] Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang- Chieh Chen. An image is worth 32 tokens for reconstruction and generation. arXiv preprint ar arXiv:2406.0 075500 2024.
(pages 10, 26, and 32)
[46] Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong A reparameterized discrete diffusion model for text generation. arXiv preprint arXiv:2302.05737, 2023.
(page 8)
14

---

Published as a conference paper at ICLR 2025
Appendix
A PROOFS
A.1 PROOF OF PROPOSITION 3.1
Part 1: Calculate p (=f N ((a)) in Eq. (10).
We first derive how calculate (=ā = N(ra)) in Eq. (10) using P1 t-
First, from law of total probability,
( = N1x)) = Σ Pile
(=t = N1a 57, me)
(16)
Next we derive closed form of p (2f = No = =.",z)). According to the noise schedule,
at
if 2 D, x D
29
if 2t D, xt # at
p (z x((21)
(17)
(1-ac//s
if 24 = N, N = x9
(1-00//
if 2t = N N. + 29
Using Bayes rule p (2/10z Ø1)
P (2f 2 [21) /p (2),, we have
Cre
if 2t D, x x if 2 D, xt + x4
p (=(1=g)
(18)
(1-a)/8
2 = N, x4 = x
00+(1-0)//8
if oΦ N, 2 f x
Plugging Eq. (18) into Eq. (16), we have
p (= Nlo.)
Σ
p (x = 74 (.)) p -=f= = N ,
(19)
j
(1 - -i/a///8
Σ
(x = 3400)) 1 p (x = = au (1 a1/s
(20)
Σ p = =3110) 1+ 1+ (21 22 (1- at + (1 as)/S
a1
(21)
1 (xf 2(120)
(22)
at + (1)
which gives us the form in Eq. (10).
Part 2: Calculate PıPe (2f = j [r,
j =t = N) in Eq. (11).
From Bayes rule, we have
P (1f jdd 24 = =N
Pile (09 = j Ir,, 20 =)
(23)
p (z = NNxxz)
p (af = jm (xO) p (2f = NN 29 j(
(24)
p (अf = N|x))
p (xf = (M(
Var # nd
(25)
p (2 = N12))
Eq. (25) is by plugging in the following from Eq. (18) that p (2f = War,, af = ja) = 1 if x 1
Part 3: Verify equivalence to the original optimal rate in Eq. (9).
15

---

Published as a conference paper at ICLR 2025
By plugging in Eq. (10) and Eq. (11) into Eq. (9), we have
Remit (xt:
( p Mp))
j N)
di
Pije (1f= jrd
Pipe
a
L-
proh, corruption
prob, of denaising
recvery composed denoising proh.
recovery
(26)
A.2 PROOF OF THEOREM 4.1: DERIVING THE ELBO FOR TRAINING
This derivation follows the Continuous Time Markov Chain framework. We refer the readers Appendix C.I of Campbell et al. [7] for a primer on CTMC. Here we provide the proof for = case. The result holds for D > case following same arguments from Appendix of Camphell et al. [7].
Let W be CTMC trajectory, fully described by its jump times Ti,... Tn and state values between jumps Wo, WT..... .... Wr.- At time Tj, the state jumps from WT, to WT.- With its path measure properly defined, we start with the following result from Campbell al. [7] Appendix C.I. the ELBO of (p) log (x()) is given by introducing the corruption process as the variational distribution:
dP
EpenalZr) [log po (xx1) 2 Pdata (dx )Q (dw) log
dQ [m (w),
(27)
where
(W)) exp (- Ry (W: )dt) New. R,W, W.)
dp3
dQ (w)
(28)
(Wol exp S R&W lx ddt ILW Ro(WT, We(z1)
Intuitively, the measure (probability) of trajectory is determined by: the starting state from the prior distribution Po (Wo). and the product of the probability of waiting from Tg-1 to Tj (which follows an Exponential distribution) and the transition rate of the jump from W. W..
For our method, when simulating the data corruption process, we augment wi to record both state values W, and its latent value Z, E (N, DYP The jump defined to happen when the latent value jumps from Za 1 to Zy with one of the dimensions corrupted and the state value jumps from W to Wa. Similarly, the ELBO of E (as)a(a)) (x()) can be defined as:
dle
Epeses (21) [logpo(zi)]
2
Paataa Páatta(dti (dt)
(dri)) (d)= log dQ 21
(29)
where the Radon-Nikodym derivative is given by:
Po (Wo) exp (- P Re (WT) di II. RC (W. (W),W)
dp
mug
dQ m
Poj1 (Wo, Zo 21) exp (- (-rt Re (W, Z 12)) dt II. R, (W. W,, Z, Zi x1)
(30)
By plugging in Eq. (30) into Eq. (29), we arrive
16

---

Published as a conference paper at ICLR 2025
LELBO [ Jame woo soloa (wn))s
=f Pdita (dar)
(dw
1=1
R2 (Wr) ) dt Σ log Re (Wi, wi)
1=0
= -f Páata (dx1) Jumesg/t
t=1
po (Zi = NNWW
1- aq
1=
WolWT Ze =N) }
Σ log di po (Zg NIW, p/je (a)
Ott
/ Pdata (dlz1)
[ waaes (wol)>
át
Q 2 (c)
po (Zr = ((W))
1- a2
1=
log ái po (zr = N) + -oggppp (2) = WiW,, Ze = -N1)
Σ IE
1- al
(11)
Eq. (31) contain terms that depend on the planner Po (z: = N(x:) and the denoiser
Pile (I1 =j 2t N). Next, we show those terms can be separated into two parts:
ái 5(21, N) log ph (cn lo,, 22 = N3]
Lienniser = Stu(t.0,))paaa (2s (e ((2))))
(32)
1- ai d; log (2)(22)
Cplaneer Eett (ert
(33)
First part: cross-entropy loss on 1 denoising.
All terms associated with pi (21 = W.IW, 25 = N) are:
Locnoising = f Paata (dxz)
f QIa (d
Σ
log pfje (x1 WolW. ; = -N)
wine
= -f Panta (dei)
/
Q (d)
Re ((W,, Ze), (y, u)(21) log log -= = N) Dynkin
wam
1=0
(V.a)
= -f /Paata (dxi)
Qi (dw=uB)
di
8(2,, N,0, D38(y, 21 log (2) W.W,, 2, -N)
1=0
a
(y:a)
/ 1 Pdata (dz1)
lii (dw mue)
di
(z, N) log pîj (x1 W. N)
wae
1 ay
_ Mlze, N2 NJ logpij 0 (4 (01101,2 24 =N] mD)
di
Ozt
At the second equation, we use Dynkin's formula
f Pdata (dzz) Q 21 (dw)
Σ
f (WT, W.) = f Pdata (dzı) Ole (dw)
Σ Re (We,vl I xT) f (We, v) dt
::all jump times
L=
which allows to switch from over jump times into full integral over time interval weighted by the probability of the jump happening and the next state the jump goes to.
Second part: cross-entropy loss on 2, N prediction.
The remaining terms are associated with - Po (ZT .NW which is the jump rate at W,,, i.c. سي
Rg. j0m (W,)= E3 jojp (W,,J) ;J)
di I-ar po (75 N(W,) according Eq. (12). In this proof,
17

---

Published as a conference paper at ICLR 2025
will use Ry in short for Rl.jumpp The remaining loss terms
Lppaannrr = f Paata (dxz))
-Paaasa
S.a. Q (d)
Bee (W,) di Elog
log R2 (W;)
= f Paata (dxi)
/
Q (dw
1=1 Re (W)) dl
Rs ((W, Ze), (v. u)) ) log Ry (Wi) di
wam
(v.u)
Dynkin
= - Paata (dx1)
{-
/== di
PPatta
Jum
Qlaa (dwcli)
Ry (Wi) dt
B[Z,,N log RY (Wa)
- 1 1-a ac
1=0
(e (p((
- Rd (x1)
8fz,, N log Re (2x)
1 az
We first rewrite the loss as
Re (art) + - 512,, N3 log R2 (ar)
át po (2t = Nlxx)) + 1 a
di 8(2,, NJ log ai
1- po (2e = - M(aa))
at
(34)
If vt is given fixed, by taking the derivative of po (ze M and setting it to zero, have the optimal solution to be:
po (z+ N [.)) (2 N3
This is equivalent to optimizing the cross-entropy loss, which has the same optimal solution:
E pp(()))1)))) [6{z,, N} logre (ze = N(x;) + (1 - 6(2,, N)) log (1 - pe (z+ N(xx)))
Plugging this =y-conditional loss back to Eq. (34), we arrive at the cross entropy training loss for the planner:
ái log Pa (2212z))
Cplaneer (ete::t..))aaaa(/e)ppiee
[I-
A.3 PROOF OF PROPOSITION 3.3: CONTINUOUS TIME MARKOV CHAINS WITH SELF-CONNECTIONS
We first describe the stochastic process that includes self-loops as this differs slightly from the standard CTMC formulation. We have rate matrix R(i,j) that non-negative at all entries, R(i,j) Z 0. To simulate this process, we alternate between waiting for an exponentially distributed amount of time and sampling next state from a transition distribution. The waiting time is exponentially distributed with rate Σx R(i, k). The next state transition distribution is it(t.f) Note that P(ili) can be non-zero due to the self-loops in this style of process. P())) Σe (((.k)
We can find an equivalent CTMC without self-loops that has the same distribution over trajectories as this self-loop process. To find this, we look at the infinitesimal transition distribution from time to time + At. We let J denote the event that the exponential timer expires during the period [t, ( At). We let denote the no jump event. For the self-loop process, the infinitesimal transition distribution is
P+
18

---

Published as a conference paper at ICLR 2025
We have the following relations
P((])) Σ R(i,
property of exponential distribution
h
P(JJ,1) R((i.)
P(IJ.1) = 8fj ii
Our infinitesimal transition distribution therefore becomes
R(i,5)
Pr+adje Σ R(L,
+
1 1 At E R(i,k)
8fj =1}
Σx R(i,k)
AI)() + + -8li = =i]ai
Σ A((i,A))
ofi - j3 + At [((i.)) -jj Σ k(i,k)
We now note that for a standard CTMC without self-loops and rate matrix Rii,)), the infinitesimal transition probability is
Pe+a: (ili) = 8(i 5} + AIR(i.))
Therefore, self-loop process is equivalent the CTMC with rate matrix equal to
R( R(i,j) iサ
R(i, Σ k(i,k)
In other words, the CTMC rate matrix is the same as the self-loop matrix except simply removing the diagonal entries and replacing them with negative row sums is standard.
In our case, the original CTMC without self-loops is defined by rate matrix
dn po
(e(
(o =N,,
Va + jd
1
We have free choice over the diagonal entries in our self- rate matrix and so we set the diagonal entries to be exactly the above equation evaluated at n = jd.
de az
Re (Z 1
N N(E.)pi = j-" | = N), Va (2)
We can now evaluate the quantities needed for Gillespie's Algorithm. The first is the total jump rate
ΣΣ
R((
di. 1 at d
ΣΣ
po(2t = Maad)pfda = M 24 N)
(35)
d 1
jd
dt Σ
co(2f -N(ra)
(36)
a4
d
We now need to find the next state jump distribution. To find the dimension to jump to we use
de Σn Re(xe, T-a,
po(2 = N(rz))
po(2f = -N(xx)
(37)
En (d)
T-ai
To find the state within the chosen jump, the distribution is
R R((2)1ja)
de 1 po(2 N (as )p (24 = 1M 24 = N)
(38)
I-an po (=) = N Mxxe)
Pip (1f -dth 1 2/ N)
8
(39)
19

---

Published as a conference paper at ICLR 2025
B GENERAL FORMULATION
B.1 SCORE-ENTROPY BASED: SDDM [39], SEDD [29]
For coherence, assume / is noise and = 1 is data, while discrete diffusion literature consider a flipped notion of time. Following Campbell et al. [6], (see Sec. H.1 of Campbell et al. [7]), the conditional reverse rate of discrete diffusion considered in [39, 29] is defined be:
d. Pel1 (jd
xx)
R
(zr.j"
| x1) = Re
(40)
Pej1
with the forward corruption rate R2 for mask diffusion, such that the corruption schedule is according to dt.-
(11 T ST) for uniform diffusion RS di (1e
I)
as
And the expected reverse rate for 2 + jd is given by:
Th (Ir..")) E (=(1e))) le nittt
(Ie.)" x
(41)
Pej1 (jd x49) Pej1 (oy (x)))
Σ Pije (21 x1) Ro(j a4)
(42)
Pel1 (jd
x4)
R₂ Σ
Pije (zf
(43)
Pej1 (x)
x4)
11
p
2
p ( (2)
Pej1 (jd x1)
Rs
(44)
p
27
xx)"")
Paja (xf x
P
1
R
Pej1
(5 |2t)
(45)
of
p
x7
D4
Pd Parameterization in SDDM.
From here can derive the rate derived in Eq. (16) in SDDM
[39] which uses neural network to parameterize p" (xg x))"))
Exq
D
x
Ptjz
(jd |x
pe (jat x
Rdll (I1,5)) = Rd
Rs
p
2d
24
\d
pf
1x xr)
S Parameterization in SEDD.
SEDD [29] introduces the notion of score that directly models Hence the reverse rate is parameterized by:
OrG" .r
with 80 (Ia))
Pelad
Pt (jd
\d)
(j4, (od) (n x
xz
Dt
Rdift (1),
Rd Pt.
Rs
Riso (3T)
dd x
Pt
P Parameterization in SDDM.
In Eq. (24) of Sun al. [39], the alternative parameterization
uses neural network to parameterize picc (a1 1
and the rate is given by:
jdl xx
R2hill (zr,
Rr
\d
p xy x0)
Σ vdd psje 2
xi
Paja
(jdt
xt)
Σ uu piLe
(xfl
(2f x4))
Pen1
20

---

Published as a conference paper at ICLR 2025
Connection to reverse rate in Campbell et al. [7]. In mask diffusion case, the rate of SD- DM/SEDD coincides with the rate used in Eq. (42), ie. rate of discrete diffusion and discrete flow formulation are the same for the mask diffusion case, Re R. DPM
We have for x M and
jd M:
Pel (j x x
Re (xt.J"))
Epil xf
Pils (of 24) Re(j" x4)
Pil (o)
RaPile (2f j rit)
a2
a1
Pije (2f = j xτ) 1 Ckt
dit 14 Pile (x = jd x..)
We can find that the parameterization of the generative rate used in DFM is only different from the SDDM/SEDD's parameterization by scalar.
In the uniform diffusion case, the reverse rate used for discrete diffusion effectively generates the same marginal distribution Ptj1 and Pt, but the difference lies in that the rate used for discrete diffusion is the sum of the rate introduced in Campbell et al. [7] plus special choice of the CTMC stochasticity that preserve detailed balance: R.liff R, DFM + R.B. Details are proved in H.1 in Camphell et al. [7]
C ADDITIONAL TECHNICAL DETAILS
C.I EVALUATING THE ELBO
Note that the ELBO values are only comparable between uniform diffusion methods or mask diffusion methods, since they have different marginal distribution Ptj1 and hence different trajectory path distribution Q(W dw). Based on Eq. (29). we write out the ELBO terms for mask diffusion and uniform diffusion. Results about log- likelikhehood in prior works [3, 7, 29, 36, 34] are reporting the (denoising) rate transitioning term only, i.e., log Pf (29" W [,,"
C.1.1 MASK DIFFUSION ELBO
Term 1: Prior ratio log (WW) O.
(Wole))
We observe that (W
1 since the starting noise distribution is the same. Hence
Pop (Wola))
Po()) log (Wolaa)))
exp( [[a]] (W,, )dt) 0.
Term 2: Rate Matching log exp(- (1 [e,,,())ddd
In the mask diffusion case, this term equals to 1, since
D
(W, = di Σ 8 (.. WiWi. .M.. R&(W, (x1) = di
Σ
{
W
d M
1-0 0z
Ot d=1
d=1
For any trajectory Wr, E (0,1), R (W,, = R.(W, (x1) and hence Term 2 equals to 0, i.e.
exp( r= n. (W, )dt) 0.
log
exp R, [21)dr]
RC (W, W.)
Term 3: Rate Transitioning log R.W,, _W(l1)"
Let the jump at happens at dimension d', we have
RC (W,, Wt) = 1-a; W. .11 p/e (2f' = w." w.'lw;). R.W,T Wi(zi) = di 8 W. WW .M)
di
1-at
21

---

Published as a conference paper at ICLR 2025
Since before the jump W must be mask state in order for jump happen, hence this term R, Wi) logpl (24' W W.
simplifies to log Re Wilm)
C.1.2 UNIFORM DIFFUSION ELBO
Term 1: Prior ratio log
Poj Pop 0.
po (W)
We observe that (o) Poin (W))
since the starting noise distribution the same uniform distribution.
Hence log
(W)
0.
Poja (
exp(- Se=d (WW )ar)
Term 2: Rate Matching log
exp( Ji Ro(W, (1)))))
If the generative process is parameterized by pi (1 in Eq. (4):
D
D
(W,) - 1 di ag Exipco + (M)), (((T)))) = de
Σ
W
Og d=1
If the reverse generative process is parameterized as our approach in Eq. (12):
D
(W,) = 1 Oit.
(===== 10 -N(a,), R.W,, (1))
di
Σ
T N
Z
1 at d=1
d=1
Term 2 simplifies to:
1
1=1
Rd RW, [21]dt -
R (W, )d
1=U
11=
[Re (Wil1a)= R,, (W;)) dt
E2(10011
[R, (W. (21) R2 (W;)) }
Similarly, it simplifies to 12 [Ro (WT- Z7 (z1) R. (Wr)] for DDPD.
For a given W; or W.l we can approximate this term with Monte-Carlo samples from t ~ 2((t; 1).
Term 3: Rate Transitioning log R"
RR,W, WWlaa))
(14)(z) If using parameterization p (x)) in Eq. (4): (WT, Wt)
- a. Pije (x we ww). (W, W.(z1)
di
ái (1- ó ( W (W. .a4 of
I-at We know for the trajectory Wt, before the jump W, # a and after the jump W.¹ = 24, therefore (W,, W,[a,) = 1-a," Hence the term simplifies to
log (W,, W.) (, W.(a1)
log (x = W:,)
using our parameterization in Eq. (12):
R (W,, ,.) = 1- a
dj
2 -==NNi =N[W; pi (2f' = wa! ( ,27 = =N).
S 2- ,N] of
ff 25 1/ 8 w." 22 = 1- Cn
R4 RW,, W,, , 2e(z1) = 1 a4
The term simplifies to
log (W, W,) = log pD0 (5 =NW":
Pile (xf -
w.'lW, W =N) =N
R,W, W.(x1)
22

---

Published as a conference paper at ICLR 2025
D IMPLEMENTATION DETAILS
D.1 TEXT8
Models and training. We used the same transformer architecture from the DFM [7] for the denoiser model, with architectural details provided in Appendix of [7]. For the planner, we modified the final layer to output a logit value representing the probability of noise. To prevent the planner model from exploiting the the current time step information to cheating on predicting the noise level, we find it necessary to not use time-embeddinng. Unlike the original DFM implementation, which uses self-conditioning inputs with previously predicted 21, we omit self-cooditioning in all of our trained models, as we found it had minimal impact on the results. When training the planner and denoiser, implemented the optimization objectives in Theorem 4.1 the entropy between target and predicted noise state and tokens, averaged over the corrupted dimensions. linear noise schedule is used. We do not apply the time-dependent prefactor T-a, to the training examples, the signals from each corrupted token are independent. All models follow Campbell al. [7] which is based transformer blocks, and 12 attention
on the smallest GPT2 architecture (768 hidden dimensions, heads) and have 86M parameters. We increase the model size 176M parameters for DFM-2x with 1024 hidden dimensions, 14 transformer blocks, and 16 attention heads.
The following models trained for text8:
Autoregressive: p(r"I p(=d1==1d-1)
Uniform diffusion denoiser (DFM-Uni): (1f((=()x)))
Planner: p==
Noise-conditioned uniform diffusion denoiser (UniD): Pid 2f = N)
Mask diffusion denoiser (MaskD); Pije(af] 2r, x Ml)
We maintained the training procedure reported in [7]. which we reproduce here for completeness. For all models, we used an effective batch size of 2048 with micro-batch 512 accumulated every 4 steps. For optimization, we used A [28] with a weight decay factor of 0.1. Learning rate was linearly warmed up to 10 over 1000 steps, and decayed using a cosine schedule to 10-5 at 1M steps. We used the total training step budget of 750k steps. We saved checkpoints every 150k steps for ablation studies reported in Figs. and EMA was not used for text8 models. We trained our models on four A100 80GB GPUs, and takes around 100 hours to finish training for 750k iterations.
Table 3: Sampling schemes used for text8 experiments.
Method
Planner Denoiser
Sampling
Options
DFM
N/A
MaskD
tau-lcaping
stochasticity r7 = 0, 15
DFM-Uni
DFM-Uni
tau-leaping
DDPD-DFM-Uni
DFM-Uni DFM-Uni
Gillespie
A, A+B, A+B+C
UniD
Planner
UniD
tau-leaping
PxMaskD
Planner
MaskD
tau-leaping
DDPD-UniD
Planner
UniD
Gillespie
A, A+B, A+B+C
DDPD-MaskD
Planner
MaskD
Gillespie
A, A+B, A+B+C
The sampling schemes used for experiments in the main text are outlined in Table Gillespie Algorithm options A, B. and C are defined as follows:
Sampling schemes.
A: Default DDPD Gillespie sampling in Algorithm 1
BB: Continue sampling until the maximum time step budget is reached
+C: Use the softmax of noise prediction logits (over the dimension axis) instead of normal- ized prediction values to select the dimensions that will be denoised
The implementation of these options when the uniform diffusion denoiser (DFM-Un))))) (Fi- is decomposed planner and denoiser is presented in Listing In Listing 2, we include the implementation of
23

---

Published as a conference paper at ICLR 2025
Listing 1: Gillespie Algorithm sampling loop with uniform diffusion denoiser (DFM-Uni) decomposed as planner and denoiser.
import torch
import torch.m. porch.an.funeeee. m.fucctiionlll
batch bateh_size
dineiinesssons
nack_tokenn toke_toke_id vocab_size
opa
stopping. criteria
samples norch.randiitt(C (B, dtypertorch. int64)
time torch. torch.zeros(s. 2eros dtype-torch. is_tine_up torch.seros(B. dtype-torch bool)
for in rango(tinestaps).
Plansing: compute probabelsttea changing dimensson
logits = (saplee,,, time) # (B, D, S+1)
logita[:; i, mask token
pt_xt_proba =. F.softmex(logita din--1)
pt pt_xt_probs__t__
torch satber(pt_xt_protee -1, camples(;, t, Nonel) # (B, D. 12
p_if_change pt.xl_probs_a__tt squeezze
P_if_ change torch. torch.clanp(p (p. change, nin=le-20, nax=1.0) (8, DJ
total_noise p_ingehango-. (-n())))
Contisue (ostiespte option or check stoppsng criteria if wwoo_tine_bcckkardee pase
alse:
1a tina_inn (p_ change aps) a-1())
if ta_tine_up.all0 break
Plansing: get dimensions that
if use use_roftmax__ee wor.ffoo_dddnnneeeeee # Use softmaz instead (Dillespie option C)
logits_dia din, change torch. logtth..o..pp. totorc. to(torch.floatt6)) opa=le-10) din, change
torch. sof diiggttt_di,,cuuneee din--1), s.queeze # (B,)
else:
din_change torch. www.muiniiillpppppmmmmm.. (ptpiiooonnll___hhaaaaee change, 1). squeeze()
Compute čoput plasner output
tine 1.0 total_noise /D
# Denoising: sample le neu values for the dimensions that change logita model (samples, tima) logits(:, mask [okeetiee__dd] -1e4 pt_ pt_xi_proba F.softmax(lggits, din--1) # (B, D, 8+1)
probe_chars change pr_xi_probs(torde (B), din, sin_cchage, :] (8, S+1)
probs_ changeltorch (B))ge), samples (torch. arange(B), change]]
x1 sl_values torch, (pritin change, 1) squuazz(( # (8,)
samplea[-1a Baaaaaaaaaa_aaaa______________ee sima_up, din cat__chang_-ss [-ia_tiaa________eeeeeeeeeeeeeee sa__tine_upll x1_ valuaaluaaaa time_upl
option B and option when a separate planner and a separate denoiser are used. Option A of using separate planner and denoiser follows the same logic of Listing 1 except using separate output from the planner and the denoiser.
Evaluation. For each specified sampling scheme and sampling time step budget, sampled 512 sequences with D = 256. Using the GPTT- (6B) model [40], we computed the average negative log- likelihood for each sequence, and using the same tokenization scheme (BPE in [33]), calculated sequence entropy as the sum over all dimensions.
D.2 OEENWE
Models and training. We used the same model architectures from SEDD [29], which are based on the diffusion transformer (DiT) [32] and rotary positional encodings [38]. We followed their training procedure closely for the Open WebTetteext experiments. Like the text8 models, we modified the final layer of DiT serve noise probability logit predictor for the planner model. SEDD models
24

---

Published as a conference paper at ICLR 2025
Listing 2: DDPD sampling loop with separate planner and a denoiser.
import torch
import torch.m. mrrch.n.cce m...m.ffuue functional as
timesteps = T
batch siza
din_dinemiione
nack_token_id vocab_eize
ops stopping.criteria
camples (O, (B, dtypertorch. int64) tima torch. (B, dtype-torch.tlans
for range(tinesteee)
Planning: compute probabilsties of each dimension being corrupted if if_noise__ logits plannar, madel (sanplas) #
check for early stopping critería: 6f every dimenston <a denoised
signoid((f moissonlostsss.sssss if (prob_if_e) if_oiiif_oiseeee apa).al1(): break
prob_if_non if_nnoieeeee
if sofftmx__fo_ddi_chnnee
Option din_change torch multinonial
torch.softmaxx1t__no cacttifooooooootteeoeaee dim -1), (.squeeee()
Opttion
ciange torch www.tuullaanllaaamm.mmmmma (prob_if if_ocb__f____ee, 1) aqueeze()
Denossing: sample values the dimensions that change
compute time knput from plasner output denk__denoisr::
nask torch perooulli(prob.b (f_ob_iff__ise)) bool long() # sampling a_t always
[torch (B), din_change] nasked_sample
dimensions that pscked dencising berch-vheree (nakk, samples, token_ok___dd
tine 1.0 mask, eun(-1)/D
logits danoiser_ (naddladdddl (macked__ample., time)
logital:, i, mask tokk_token__tk [d] -1e4
time 1.0 prob_if_noise (ft.....eieeeee()))) D
logits F.
model (samples, time)
pt_xI_probs
din--1) (8, D, St1)
probs_chango pt. pt_xi_probe (torch arange (B), din_change, i] (8, S+1)
valuee torch. ricc..mltinon www.mminnninaaaraaaaaa... (prohss___aaaagggggg 1) squuezz() (B,)
aamples[torch arange (B), din_ change] x1_ vi_vee
25

---

Published as a conference paper at ICLR 2025
the noise level instead of time for the time embeddings. While retain this model input by using their (t), we replace it with zero when training the planner, similarly to the text8 models. All models were trained with batch size of 128 and gradients were accumulated every 4 steps. We used AdamW [28] with a weight decay factor of O, and the learning rate was linearly warmed up to 3 10-1 over the first 2500 steps and then held constant. EMA with decay factor of 0.9999 was applied to the model parameters, We validated the models on the Open WebTet dataset [17]. The mask denoisers are taken from the pretrained checkpoints of Lou al. [29]. SEDD-small has 90M parameters and SEDD-medium has 320M parameters. We trained our planner models on nodes with four A100 80GB GPUs for 400k: iterations. We only trained the planner models in the size of GPT-2-Small, which is 768 hidden dimensions, 12 layers, and 12 attention heads.
Sampling and evaluation. We employed Tweedie tau-leaping denoising scheme for SEDD, and adaptive Gillespie sampler for DDPD, and different nucleus sampling thresholds (top-p values of 0.8, 0.85, 0.9, and 1.0) for GPT-2. For all models and sampling schemes, we generated 200 samples of sequence length 1024 and evaluated the generative perplexity using the GPT-2 Large [33] and GPT-J [40] models.
D.3 IMAGE GENERATION WITH TOKENS
Models and training. For tokenization and decoding of images, we use TYTok-S-12 model [45], which tokenizes 256 x 256 image into D = 128 tokens with the codebook size of 5 = 4096. Both mask diffusion denoiser and planner models use the U-Vit model architecture of MaskGIT [8] as implemented in the codebase of [45], with 768 hidden dimensions, 24 layers, and 16 attention heads. The mask denoisers are taken from pretrained checkpoints from Yu et al. [45]. The planner is trained with batch size 2048 for 400%: iterations 4 A100-80GB GPUs. We used Adam [28] optimizer with a weight decay factor of 0.03, 31 = 0.9, and 82 = 0.96, and a learning rate of 2 x 10-4 The learning rate schedule included linear warmup the first 10k steps, followed by cosine annealing down to final learning rate of 10-5, EMA was applied with decay factor of 0.999.
We utilize the evaluation code from ADM [11] to compute the FID scores [19] and inception scores. For this evaluation, 50,000 images are generated across all classes. Each image is produced by first generating tokens, followed by decoding with the TYTok-S- 128 decoder.
Evaluation.
E ADDITIONAL RESULTS
E.1
TEXT8
E.1.1
EFFECT OF APPROXIMATION ERRORS IN DENOISER AND PLANNER
We conducted experiments to measure the effect of approximation errors in denoiser and planner on the generation quality. Results are summarized in Figs. 4 to 6.
E.1.
ABLATION OF CHANGES INTRODUCED IN DDPD SAMPLER
We conducted controlled experiment to measure the individual effect the changes we introduced to the sampling process. Results summarized in Fig.
E. MODEL LOG-LIKELIHOODS ON TEST DATA
Following BLBBO terms derived in Appendix C.1, we calculate them for three different design choices:
A single uniform diffusion neural network, but decomposed into planner and denoiser.
Separate planner network and uniform diffusion denoiser network
Separate planner network and mask diffusion denoiser network
In Table 4, evaluate the ELBO terms for three methods both trained for 750k iterations (near optimality). We observe that the mask diffusoin denoiser has better denonising performance even with mask approximation emor introduced in the step of Proposition 3.5. In Table 5, We also observe
26

---

Published as a conference paper at ICLR 2025
Negative Logs SosLikeeihon saihhand
5.5
DRM /
-A. DDPD MOsD-MaskO,.. A
DDPD-MadKO, A+BrC
5.0
DDPO-OFM-Uni,
Data Entropy
4.5
Entropy
Figure 4: Comparing DDPD sampling under imperfect learning: 1) a single uniform diffusion model as planner + denoiser v.s. 2) separately trained planner + mask denoiser. The single uniform diffusion model converge slower in training and using DDPD sampler results in collapse in sample entropy. Using separate networks for planner and denoiser achieves results more close SOTA methods in terms of quality v.s. diversity.
7.0
7.0
Negative Log-Likel Likellihood 4
Negative Log-l Likkelihoood
6.5
6.5
6.0
6.0
5.5
5.5
DFM-Best.
DDPO-unD-Best
DFM-500K 15
DOPD-0-800
DFM-450k n=
D0-0-unD-E50k
DFM-450K n=
DOPD-G-600
5.0
DFM
muskD Best
D-PD-D-300
DFM-450K
15
.B. DDPD-mastD-4500
DFM-150K 0-
DCPD-O-150K
6.8
7.0
7.2
7.4
5.8
7.0
72
7.4
Entropy f
Entropy 1
Negative Log-Likelhood
6.5
6.0
DRM-Best. 1
CDPD-D-Best-P-00X
5.0
DDPO-Best
COPD-D-0es---455
Data Entropy
DOPD Deest- 300k
COPC.D-Best.P1155
4.5
Entropy 7
Figure 5: Left: Denoiser checkpoints 450k v.s. 750k iterations. Right: Denoiser checkpoints at 150k, 300k, 450k, 600k, 750k iterations. DDPD is able to use an imperfect denoiser to achieve the same performance as the best possible.
mask diffusion denoiser performs better than uniform diffusion denoiser in terms the denoising log-likelihood.
E.2 CONVERGENCE OF SAMPLING
We conducted experiments to see the convergence of sampling with regards to number of steps. The results shown in Fig. 8.
27

---

Published as a conference paper at ICLR 2025
7.0
7.0
Negative Log- LggLLiellhood
6.5
Negative Log-Like Likkellhood
6.5
6.0
6.0
DFM Best. 1 15
DDPD (090---Best--B00
DFM-Best =
DCPD-D COFDD--4800 D-450k---6555000 600k
5.0
DDPD-D-86tt-P-554
5.0
DFM-450K
DOP0-D-650U-P45500
Data Entropy
DDAD D-Best-P-30kk
DDPD- Best
D6RD-D-60k-P3300
Data Entropy
DOPD-D-E0K--1100
4.5
4.5
7.0
Entropy
Entropy
7.0
7.0
Lg-Likellhodd
Negative Log-Lit Lg-Likeilhodd
Nogative Log-Likelltt
6.0
6.0
5.5
5.5
DFM-Best. 15
DDPD-0.30U.P-600
DFM-Bes
DDRD-D- 1-00P-D-55KPP00
5.0
...
DFM-300K q- 15
DD70-D-300K--4500
5.
..8
DFM-1 50K n-
DDPD-O-15000 1-00-00-500---450
DOPD-Bes
DOP0-0-1500-P-000
Data Entrapy
DDPD-D-30KkPP1150
Data Entropy
DORD 150k DOPD-D1550K
P0kKP-150
4.5
6.8
7.2
7.4
7.5
5.8
7.0
7.
7.4
7.5
Entropy
Entropy
Figure 6: More ablation studies on pairing imperfect denoiser with imperfect planner.
7.0
7.0
Negalive Log- Likelihood
6.5-
Negative Log- Likelihood
6.5
6.0-
6.0
DFM, 0
DOPD-DRM-UrA A+B
DFM. g)
DOPD-UmD, A+6
5.0
DFM 15
DOPD-DRM A+B+C
5.0
DFM. 15
DOPD-UnD, A+B-C
DFM-UhL, Tleaping
Data Entropy
PxUnID, rleaping
Data Entropy
DDPD-DRA-UNLA
DDPD-UnID,A
4.5
4.5
Entropy f
Entropy
7.0
7.0
Negative Log-l Likellhood
6.5
Negative Log-Likellnn lkelihood
6.0
6.0
5.5
5.5
DFM. -
DDPD UnDD
DFM
DDPD-MaskD
nFM
ROPD-DFF-LPT
5.0
DFM.
DDPD MaskD, A-3+C
5.3
DDPD-MaskD
DOPO-DFM-Lni, AtBrC
PxMaakD, rleaping
MasD-MaskD
Entropy
DDPD-MarkD,A
DDPD-UniD
4.5
6.8
7.0
7.2
74
6.8
7.0
72 Entropy 1
74
Entropy
Figure 7: Ablation introduced changes to discrete diffusion. A: Original Gillespic sampling. B: Time- adjustment based on the planner, continue sample until maximum number of steps is reached. C: Use softmax(1ogi) noise) instead sigmoid(logit. if noise) pick which dimension denoise next. The softmax trick makes the planning slightly more greedy than the original planning probability.
28

---

Published as a conference paper at ICLR 2025
Table ELBO terms computed on test of text8 biss-er-character (BPC) with fully trained models. Denoising likelihood only evaluates the probability of correctly denoising, for Planer + Mask Diffusion Denoiser, a mask is first sampled according to the planner.
Rate Matching Transitioning Combined (BPC)
Method
(BPC)
(BPC)
Uniform Diffusion
S 0.0131
S 2.252
S 2.265
Planner + Uniform Diffusion Denoiser
Σ 76
S 2.284
S 2.244
Planner + Mask Diffusion Denoiser
S 0.0176
S 2.226
2.302
(given correct mask for denoising)
Planner Mask Diffusion Denoiser
0.0176
S 2.605
2.623
(use planner-predicted mask for denoising)
Table Denoising performance in bits- bits-per-cheee per-character (BPC). Mask Denoiser v.s. Uniform Diffusion Denoiser. Note that those are not entirely comparable as ELBO terms for uniform diffusion and mask diffusion are different.
Method
Denoising (BPC)
Denoising Accuracy at az 0.85
Uniform Diffusion Denoiser
< 2.063
92.2%
Mask Diffusion Denoiser
< 1.367
96.8%
Table 6: ELBO terms computed on the test set of text8 in bits-per-character (BPC) with imperfect models trained
20k iterations.
Method
Transitioning (BPC) Combined (BPC)
Uniform Diffusion
3.060
S 3.076
Planner Mask Diffusion Denoiser
2.854
< 2.843
(given correct mask for denoising)
Planner Mask Diffusion Denoiser
S 166
S 3.155
(use planner-predictee mask for denoising)
E.3 OPENWERTEXT
In Figs. and 10, we measure generative perplexity of unconditional samples from GPT-2- small, GPT-2-medium, SEDD-small, SEDD-medium, DDPD-Small: Planner-small + SEDD small-denoiser, DedD-M:diun: Planner-small SFDD-small-denoiserr We also tested using sigmoid(logit___ if if_lfifnoiii noise) and poftmax(logitt if noise) for planning. The difference is not as significant as the text8 case. Using softmax (logit_ if _oisee slightly increases entropy at the expense of perplexity. In Fig. 11, find that DDPD using Planner-Small and SEDD-Denniser-Small outperforms simply scaling up denoiser to SEDD-Medium.
E.4 IMAGENET 256 256
We study the effect of planned denoising with an increased number of refinement steps Table 9. The FID first increases and then converges. The inception score also improves with increased refinement steps and then converges. From the visualized samples, we can see that aiin-and-denoo sampling is very effective at fixing errors without losing its original content.
29

---

Published as a conference paper at ICLR 2025
Amount of Noise Left Over Sampling Steps
How many 2i Change Between Step
250
68P6, 2.0.3
Aucrage Noise Count
Dort, (w0.3
DOMS, (m1.
Aversge DDmeesiio
DEFC, t-29
DCMD, tw..0
1.0
30
20
50
10
200
400
500
800
1020
1200
200
500
800
1000
1200
Sampling Stops
Sampling Stops
Steps Required Finish Sampling (until palz)
Steps Required Finish Sampling (until polz)
100
100
Percentage Firished (16)
Percentage Firished (96
80
60
20
COPD: pw0
20
DCPD: 1-10
0080, bnd
D0ED, 1-18
coro. p-1.
Dcrc,
200
400
600
500
1000
1200
200
400
600
500
1000
1200
Sampling Steps
Sampling Steps
Figure 8: Convergence of sampling along number of sampling steps: Left Upper: amount of noise left. Right Upper: amount of dimensions that are predicted to haved different 2 and 2|_1 between i and t 1. Left Down: how many samples meet the criteria of stopping to sample for pe (af N) < 0.05). Right Down: how many samples meet the criteria of stopping to sample for po(z) N) 0.01).
240
DDPD-Small Top-p=1.0
300
DDPD-Medium, Top-p-1.0
Top-p-1 p
Gen Perplexityy (GPT-2-L) 2--2
SEDD-Small, Top-p-1.0
1opp-1
Gen. Gen. Perplexity (GPT. (2-L)
Medium, Top p=1.0 GPT-2-Medlum
120
GPT Small
Data Entropy
120-
Data Entropy
60
60
T-1024
1-1224
TT2048
T= 024T-11100
30
11-4796
41-3072
1-33772
2048
Top-p-0.3
2048
T-2048
Top-p-01
+1-6056
41 Srap72
sTop-p40.85
15-
10
#Top-g-0.3
10
=Top-p-0.8
6.6
7.3
7.4
7.8 8.0
Entropy
Entropy
DOPD-Small Top-p=1.0
1
DDPD-Medium, Top-p-1.0
Top-p-f.0
Gen. Perplexity (GPT-))
120
SEDD-Small, Top-p-1.0
Gen. Perple Perplexity/GrFe (GPT-))
180-
GPT-2-Medium
SEDD-Medium, Top-p-1.0
Entropy
Data Entropy
60
1-1024
45
T-2088
1-1124T-100
T-2048
--04
=Top-p00
15
T:4096*
1-3072
=Top pr0.8
20-
1-4099
+T-3072
T-2048
T-4086
TTop-p-C.85
sTop p-
6.2
5.5
7.0
7.4
7.8
8.2
52
6.6
7.3
7.4
7.9
8.0 8.2
Entropy
Entropy
Figure 9; Using softmax(logit___ (ic_logtt_ii____________ _noiss) for planning. Generative perplexity evaluated with GPT-2 Large (GPT-2-L) and GPT-J: SEDD v.s. DDPD using the same denoiser.
E.4.1 EFFECT OF CLASSIFIER- FREE GUIDANCE
We also studied the effect of applying classifier-fre free guidance in Table 10 for discrete mask diffusion. The effect of classifier-free guidance is similar to temperature annealing. In the of logit
30

---

Published as a conference paper at ICLR 2025
240
DDPD-Smin Small, p-10pp-1.0
300
DDPD-Medium, Top-p-1.0
Top-p-1
Gen Gen. Perplexity (GPT-2.)) 2.0)
SEDD-Small Top-p-1.0
Topp-1
Gen. Perplexity (GPT.2 (-2---)
SEDD Medium, Top pm1.0
120
GPT-2 Medium
Data Entropy
Data Entropy
60
T-1022
T-22088
15
*1-4196
41-33772
T-489t
TT-1024
Top-p-0.38
Top-p-53
T-4286
+1-3072
T-248
1-2068
TT-0024
TTp-p-09
*Top p-0.3
T-4036
=Top-p-0.8
6.2
5.5
7.4
2.8
8.2
5.
6.6
7.3
7.4
7.8
8.0 8.2
Entropy
Entropy
240
DDPD Small, p=1.0
Tope-1
350
DDPD-Medlum, Top-p-1.0
Top p
Gen Gen Perplexitt (GPT-))
120
GPT-2 Small
SEDD-Small Top-p=1.0
Gen. Perplexity (GPT-))
180-
SEDD-Medium, Top-p-1.0
GPT-2 Medium
Data Entrapy
Data Entropy
90
T-1024
30
*T-4036
9T-3372
T-2048
T-1024
top.pro.
T-2348
T-1124
1-10224
TTp-pp09
31-3072
T-2068
T-408top p-0.8
Top-pioo.8
1-4094
+T-3072
T-4099
*4T-30/22
T-20ks
TTpp pro.u
pro.pp.c.s5 pro.85
7.0
7.8 8.0
Entropy
Entropy
Figure 10: Using sigmoid(logit_ ( for planning. Generative perplexity evaluated with GPT-2 Large (GPT-2-L) and GPT-J: SEDD DDPD using the denoiser.
DDPD-Small Top-p-1.0
350
DDPD Small, pr1.0 SEDD-Medium, Top-p-1.0
9.
Gen, Perplexity (GPT-2-L)
SEDD-Medium, Top-p-1.0
GPT-2-Mediln
Gen, Perplexity (GPT-3)
180
GPT-2-Medium
120
Data Entropy
Data Entropy
45:
T- 1024
=Top--0.0
T-244g -124
T=l 024
Top-pra9
TT4095
+1-0072
I-2040 TT-3072
20-
T-4295
+1-3072
T-2048
-3-3072
"opppp00.00
Toppp-4.8
Top
6.2
6.6
7.0
7.4
7.8 6.0
8.2
6.2
6.6
7.0
7.4
7.8
8.0 8.2
Entropy
Entropy
300
DDPD-Small Top-p-1.0
Top 1.
350
DDPD Small, Tap-p-1.0
Top p.
Gen, Perplexity (GPl (GPT-2-L)
SEDD-Medium, Top-p-1.0
Gen. Perplexity(Ga (GPT-))
SEDD-Medium, Top-p-1.0
GPT-2 2eedum
GPT-2-Medium
120
Entropy
Entropy
90-
60
T-1024
T-1024
45
T-1024
30
1-3007
2-2448
T-2048 #7G4256
=Top-p-0.0
+1-3072
T-2048
1212
Top-pru3
15
T-4095
Top-p-G.1b
Toppp-0.8
10
Top
7.4
7,8
10
7.8
Entropy
Entropy
Figure DDPD SEDD-small denoiser (90M) Planner-small (90M) SEDD medium denoiser (320M) v.s. GPT-2-Medium (355M). DDPD with smaller (less perfect) denoiser achieve better performance than simply using a larger (better) denoiser.
31

---

Published as a conference paper at ICLR 2025
Table 7: Inception Scores (1) ImageNet 256 x 256. MaskD refers to mask diffusion. The denoiser and parallel sampling schedule are kept the same as [45], without classifier free guidance.
No Logit Annealing
Logit temp 0.6
Logit temp 1.0
0.0 MaskGIT
Steps T
MaskD
MaskGIT
DDPD
MaskD
MaskGIT DDPD
MaskD
DDPD
8
33.56
199.83
149.98
149.28
271.73
201.67
157.19
249.86
213.03
16
39.36
248.88
178.17
179.85
281.73
173,48
164.01
263.47
185.25
32
43.30
266.17
169.49
200.33
281.36
156.22
170.73
268.88
158.14
61
45.06
274.56
160.74
206.06
281.14
146.27
171.62
269.45
145.95
128
45.56
276.45
152.61
210.27
278.88
138.55
142.40
272.73
137.19
Table 8: ImageNet 256 x 256 generation results
Method
FID
Inception Score t
Model size
# tokens
codebook
Taming-VQGAN [12]
15.78
78.3
1.4B
256
1024
RQ-VAE [23]
8.71
119.0
1.4B
256
16384
MaskGIT-VQGAN [8]
6.18
182.1
177M
256
1024
ViI-VQGAN [43]
4.17
175.1
1.7B
1024
8192
MAGVIT-v2 [44]
3.65
200.5
307M
2048
262144
ID-tokenizer [45] (annealing tricks)
166.7
287M
128
4096
DDPD-ID-tokenin (w/o annealing tricks)
4.63
176.28
287M + 287M
128
4096
annealing, it helps mask diffusion to achieve much better FID score. DDPD achieves similar FID
scores but much better Inception Scores (which means higher aestheticlity). When logit temperature of Mask Diffusion is greatly improved, but the composition of logit
0.6, the inception annealing and classifier-free guidance leads to a worse FID score due to less diversity. The same applies MaskGIT.
We note that the interaction of various sampling heuristics, particularly in configurations that in- corporate logit annealing and classifier-free guidance (CFG), is intricate and nuanced. detailed in Yu et al. [45], the best FID scores for MaskGIT are achieved using additional hyperparameter adjustments, specifically inflating the logit and confidence temperatures to 3.0, which are then linearly annealed to 0.0. These unconventional settings are tailored to mitigate the loss of diversity typically caused by the combination of logit annealing and CFG. However, while this approach improves FID scores, it adversely impacts inception scores, underscoring a fundamental quality-diversie trade-off.
To address this trade-off, Table 11 presents results for MaskGIT with the same configurations as Yu et al. [45], where sampling includes logit annealing and logit and confidence temperatures are initialized at 3.0. Interestingly, the initial tokens sampled under these higher temperatures (which tend to introduce more errors) are later corrected during the annealing process, allowing the decoder to reconstruct coherent and high-quality outputs. These techniques are specifically optimized for mask diffusion with CFG, but do not generalize directly to DDPD, which operates using uniform diffusion. Following this insight, we applied the same configuration to DDPD as an experiment (Table 11). While this approach improved inception scores, worsened FID scores, further highlighting the quality- diversity off. Upon analysis, we observed that the DDPD planner proactively identified some of the initially generated tokens (under high-temperature settings) noisy, as they deviated significantly from the original distribution. These tokens were subsequently corrected in later denoising steps, leading to improved visual quality at the expense of reduced diversity. This behavior indicates that DDPD naturally balances quality and diversity differently from MaskGIT.
32

---

Published as a conference paper at ICLR 2025
(a) DDPD 16 + 16 steps
(b) DDPD 16 + 32 steps
Figure 12: DDPD No Annealing, increasing numher of refinement steps. The added refinement steps "touch-up" to improve the aesthetic quality without losing original content.
Table 9; FID Scores ImageNet 256 256. Increasing the number of refinement steps.
Base steps = 8 Inception Score
Base steps T = =6
Refinement Steps T
FID J
FID J
Inception Score 1
5.12
178.17
5.12
161.17
16
4.92
187.59
4.75
169.49
4,93
192,93
4.63
176.28
4.94
192.99
4.71
176.22
33

---

Published as a conference paper at ICLR 2025
Table 10: CFG scale 2.0, Logit Temp 1.0 0.6. MaskGIT anneals confidence noise temperature from 1.0 0.0. FID/Inception Scores (1/1) ImageNet 256 256.
Steps T
No Logit Annealing MaskGIT
Logit Temp 0.6 MaskGIT
MaskD
DDPD
MaskD
DDPD
8
8.44
145.36
6.89/ 337.37 7.19 323.79
6.37 320.48 13.00 / 400.23 7,45 333.28
5.30 187.24
9.91
391.98 6,28 /29.44
8.17 365.30 13.93 407.12 5.73 319.52
32
4.14/215.88
11.49
408.73 5,58 322.67
9.06 384.50
14.16 196.19 4.98/ 315.14
64
3.82/
12.04
409. 16 5.16 323.62
9.51 388.71 14.28 405.01 4.66/311.82
128
5.02 180.79 11.55 396.81 4.63 313.09
8.31 / 361.80 13.76 388.04 4.39 304.92
Table 11: When using the same hyperparameters as in [1] (8 steps are not enough for DDPD to converge). FID/ Inception Scores (L ?).
Steps T
MaskGIT (FID Inception)
DDPD (FID Inception)
16
2.48 / 284.20
7.93 / 359.69
32
2.74 299.53
6,11 / 343.63
64
2.86 / 309.62
5.11 328.56
128
2.93 205.26
4.63 313.09
E.5 NOISE ESTIMATION ERROR USING INDEPENDENT NOISE OUTPUT (=(((<a)))
We tested the assumption made in utilizing a pretrained mask diffusion denoiser by sampling joint noise latent variables using independent marginal prediction from transformer for p( [2,, of N) 2 Ia' (2 (xx)) in Table 12. We observe that the assumption holds almost perfectly in language modeling such Open WebText. On character modeling task text8, the assumption also holds most of the time, especially near the end of generation, but is more complicated than word tokens due much smaller vocabulary. This is also discovered in Table 4 where observe the two-step sampling introduces approximation errors and hence makes the log-likelihood for denoising lower.
Table Accuracy on Mask Prediction for text8 and OpenWebText at fixed times. Mask accuracy measures the independent sampling matches the joint noise variable values. Almost deterministic measures the assumption p( =f of N) We set the threshold to logit.abs() 3.0.
Fixed Time
text8
Open Web'Tet
Mask Accuracy If Deterministic Mask Accuracy If Deterministic
(Data) 1.0
0.9988
0.9975
0.9999
0.9997
0.95
0.9915
0.9864
0,9985
0,9960
0.8
0,9623
0,9238
0,9943
0,9851
0.6
0,8784
0,6789
0.9847
0,9585
0.4
0.7416
0.2261
0.9679
0,9100
0.2
0.7402
0.2125
0.9476
0.8466
0.05
0.8878
0.4800
0.9599
0.8817
(Noise) 0.0
0.9465
0.5974
0.9975
0,9906
34

---

Published as a conference paper at ICLR 2025
F GENERATION EXAMPLES
F.1 GENERATED SAMPLES FROM MODELS TRAINED ON TEXT8
We compare samples between DFM and DDPD. For DDPD, we include samples from three models: 1) DDPD-DFM-Uni planner and denoiser from a single uniform diffusion denoiser model p (<f (x)) using Eq. (10) and Eq. (11); DDPD-UniD: planner network p(=ī (rx)) and uniform diffusion denoiser network (1 ITe 2 = N); DDPD-MaskDD planner network (x)) and a mask diffusion denoiser network Pijs (21 =7 N).
Samples from DFM, 15.0, Temp 0.8:
are being damaged downtown plus the roads that are historical image map roads roads through hong kong shin te kwun avenue tansua tai tonlin gouhan and mengtusam there are several main freeways at the same station in the hong kong through caches the
everything basil mark making me worth one nine three nine lecture bird man one nine three nine voice of law one nine three nine everything revived one nine three nine people with fears one nine fou
two murmour about mirrored action
eight nine nine two stemming from people disaster nine nine zero one nine nine five author john chemerzi wakis pbs troupe witnessing impact report one nine eight one jone bethy hopkins place wiley and jackson campaign begins one nine eight eigh
Samples from DFM 2x, f = 15.0, Temp 0.8:
dinidol three one zero five two four three acritrine zero zero eight four eight zero zero two three acetic acid one zero seven 1 not aspirin in vinnol two three nine hyproxyphenol zerolin references mda r two virginia department of public saf
he often shared at least some of these suggestions the priesthood of all the people of sparta hemischeres your father god and lord bound him among the priesthood of the lord with such interference preaching the probes of tribute and i cannot believe they
in etc time relief belongs to the preparation of funeo the time silenium and platarin the rolled taste preparation by cooking wine from the gum is sped back the vine that is produced per pell concentration is nine two the taste is either rosin brac or
35

---

Published as a conference paper at ICLR 2025
Samples from DDPD, Planner + Denoiser decomposed from single uniform dilfusion denoiser model p(af(=z), Temp 0.8:
dickey morris morris scott del simon simon fred rogers gregg dickey david dicks marcus dusshinsky douglas dickson steven dick douglas hartman harvey dickson scott kelly hartman mike duncan daniel harvey micha
del del del brenda del
e prochet pink peanut proc pinky proche pink pigmy pig proche pigmy pigmy pig bear frog hormit horna horn wagon hornita wild fish hornit wagon griffon germanic florahornit horna ostrich wild fish flora horna wild island chicken horna horn winters winter
park kitchen comfort fort gorge fort hills castle bay hills lakeland fort hamilton state park the park ground hill bay forest reed brighton park stanley innocence small park protector hillfort edgeside statue greenwood woodfort alban at columba
Samples from DDPD, Planner + Uniform Denoiser, Temp 0.8:
reported however one eight six nine that a natural carnivore would be a bad daily utilized his neuspaper the best lighting embeddings of quiner sun warm and winter and warma france a illegal bubbles first said lighting with quiner is greatly import
well the composition the entire borough communes of columbia were once filled and defined without separate antiphers and antiphers each of these twin princess mournings made the death as an antipher as defin
by the boiling of free
in the work that is edward damascus across the cross from the flame of career and the king is why i like others here there great aspect die die crossing that precedence of the star die literary die an aspect of murder may contact or rescue those
Samples from DDPD, Planner + Mask Denoiser, Temp 0.8:
roy despite this in the franchise he was uncredited to put on club chatterhead big morocco theater and in the winston team hockey shot out for snap five sidekick notable player don allisto mike henning puncher jack may founder of roy puncher five four f
lithography logeliths littoral nonfiction lilitus confusion lit little dragon littorius love eye love crawlin love utopia lolita popula prose oracle populae populae anticharia prophecy leonida popula lepheus mycemaean super super dendron carbon lover dend
rrorists criminals in gold the timing expressing only insisted the endings them rising six ten days a population one thousand guessing holding potential risking dangers see falling and plasticizing goodman father of the town guam effect only
36

---

Published as a conference paper at ICLR 2025
F.2 GENERATED SAMPLES FROM MODELS TRAINED ON OPENWEBTEXT
In general, samples from DDPD demonstrate a better ability to capture word correlations, leading to greater coherence compared to those generated by SEDD. However, both methods exhibit less coherence in longer contexts when compared to samples from GPT-2 models.
Samples from
4096 steps:
tornadoea Space.con
Bead nore:The disarrey has reached end, dragging the US political systen into Its turbulent period.
Today looks be the Facebook Tuittar Pinterest Nan and vomen nake an attempt to anter the Capitol bailding, shich is part
reckoning for Vashington.
State Department. Photograph: licky Beyce/AFP/Getty Inages.
While there has been sone infighting of late, the nonetines- jadeed new us Congress has also been shying from the uaual trappings of parlianentary checks and balances, particularly on the appointment of secretary of state, and inmigration, the Senate yesterday voted vote "no" to bill.
the session, Congress looks the old with a new one, sonathing that bas been done in other countriea. start April en nov that will change dramatically political landscape across the country.
the opportunity form governnent, replacing
The las was amended more than half dozen tines over the tao-week period, and will be announced hosted by Obans.
advance of private The law creates a new legal systen for states. In the US system, it treats state and local officiala
representatives of the paopla, with the federal governnent, including the president, in the process of forming the nov governnent.
Interior Senator
Lieberman, chairman
so-called faderal government: lobby group, said it
"Ia time" to come with ned government.
"If this is the rule of lav, that's not the way we done it. Ve have to think about that. We beliere the result vill be good for the people and the country, " he said.
In the past, Mr Durbin has been vocal in his desire to form nov government. Firstly, he vanted the bill to take effect dee was more outspoken in his desire to legislate further, just a week before the new seesion, vote on anendnenta the lav should have failed
then he vetoed
amendnent 2010.
by arguing that lasmakara who atteupt to form French fereign ninister, Laurent Fabius said: "We velcome the formation of a nev government and look forvard election of the president and taking the important tack of forning govarnment," " Fabiua said.
governnent.
sign the election of a new secretary of state. Not a vote of confidence. A vote of confidence and it will happen," he added.
announcement the president's resignation from office,
Earlier, Irich Labour Party leeder Mairnin O'Kaughin-Bullivee ssid the country's sttention was being diverted from immigration, saying: "I do think this issue is not on our agenda at all. It up with government, especially in the
only short pariod of time, and have course of the new session."
She added: government in different way.
think
inportant the president and goverument, not nev
"And we need
governnent not nov on making anendnents US government."
governnent and not raise the specter of Facebook Tuitter Pinterest Concerns about immigaation are rising in much of the country. Photograph: Marjorie Nougou/APP/Getty Images.
Politics and constitutional isauea Even-Scott Brown, heed for logislation to create a new legal systen for states rather than the Republican-led federal govarnment, years ago.
goverument under
White House, succeesfully puehed
In the biggest political
American history, Even-Scott Brown has described the issue
constitutional issse.
have said that way relates secratary Secretary of State position. It has nothing to do with the Conetitution," she has said. Deputy nembers
position
different the
adninistration, which included International Trade Secretary Michael Fronan, and Justice Secretary Carole Wray, formed a new congressional task force.
But by time vhich Übama vas elected the us preaidant, public opinion tumbled opposite direction.
He had introduced a nex immigration bill and becama the first president evar to get the innigration bill passed in the Senate, the House refused to and he resigned the Senate in December 2011.
Health care
Mr Brown's position as leader of the nation's Rapublican Party has expanded under the Obana adninistration. vas only thres nenbars House to pass the care reform bill in 2009-10 and took break from the Senate as head of the Department of Health and Bunan Services.
37

---

Saturday speach that had work for the and of Published as a conference paper at ICLR 2025
Samples from SEDD-Small, 4096 sleps:
change, second-in- one Cabinet minister, political correspondent Dliver North, CXM on Sunday. This uas the first auch rally of the party control 'a campaign. "Time to change course is now. A nev politics will begin in 2020."
Even party leaders and prominent Labour figures alert to this shift, including Boris Johnson who has left the party for the first time since a rival candidate's nomination for president of the Republican National Commitaee, pladging Thatcher's reign.
The nedia, meanchile, her and backed her in the current right-handed coalition with Mr Miliband and attacked Labour leaders as without party that could return then to the prine minister and at odde against across-the the-ground austerity, which the personally has never said he wanted. Road nore from OKX on Twitter
predicted
former minister Margaret Thatcher would not vote
of party, Johnson warnad: "Vatva kind destroyed country don't talk about our future, so David Niliband's attenpta to replace our leader are selecting those who support Labour, who 2020."
the
voter turnout and real threst could
Corbyn's say for Corbyn
As Jereny Carbyn adáressed exile party convention in West End, Lendon Saturday, Corbyn presided over the Tories in London who are in recent days leading the polls overall in the party. Corbyn, used black vote his pool of NPs to vin theelections, at packed rally outside the Democratic Central party, After London earlier this nonth that "the greatest person the first voman to decide her leaderchip in more than 20 years", and called Jereny Corbyn at that conference 2012 "remarkable individual".
stood for Corbyn
on change labour's 's managenent to decide Sir Jereny's forner Labour leadership,
Labour Party, Corbyn said
Nr Carbyn said in Vestminster: "I condemm the hate to your name" and citad him as being tough on racists, who Liberal Deuocrats eay are just white already put 60 per cent and have stood down since the general electicn earlier this year,
shifting from white niddle nen, who
your name on the unVhitelist #labourCan. Yvette Caron... no one Nr: condenn bettp::/tt.co//B/66ti co/JRRSSCCBd88 Nicola Davidson (GGLGLa) September 14, 2017
racist" httpe://t. David Cameron, the Corbyn leadership candidate, said having left the Carkey campaign opposed the real threat of racisn. "I'm fan of this novanent," " he said after the preas release, dascribing it as he'd like to see in Manchester, shich ia Britain's third biggest city. "Ve've got it. 'Get soman, with vomen.
you ballot paper, and it will and vote for it for the firet time. 1n Mr Corbyn appears to have been saying "the Tories thould stop being doen against discrimination" and "The Greens should go against it." Locations for Blue Brown menorial
man,
you, so step
Earlier this weok, the Conservative parliamentary party released another statement about protests for the death of lra Brown, young governnent" into the black warket. Brown and tens of thousands of pensioners salked to the streets after death.
who carriad opponition "a free-gout programee
"The great political path forward over the past 60 years has come without Labour leaving Blue statenent.
Brom, including any onea involvad in that distinct occasion,"" Neamsbile, the Tories said in statement she'n "not left of politics" and "strong ny convictions."
She stood for Jereny Corbyn during Saturday's canpaign: "Denocrata, in the decade been puehing their agenda and been trying to right the say they are complicit in sexism and been playing similar roles
racial inequality. The rest of Denocrate and Republicans America's history for decades. I'm optimistic that a beacon for liberalisn can be brought back into this country."
But seeing Mr Corbyn party's platforn as the dominant party in politics. Ma Corbyn has spant time across the country seen public events neans by which she and Corbyn von
father figure the back of his election also
avay from the
the past 16 years then at one point or snother.
Hillary,
two years Labour's founder and parlianentary frontrunner, naned race not
factor in her olection victories, but core legacy
38

---

Published as a conference paper at ICLR 2025
Samples from GPT-2-Small, 1024 sleps:
Because he knev how dangerous gas get there. teo had been valking home, came to, he aay two people standing outside the tree, each with large knife and rifle. The murderer stabbed then in the neck with bladed veapon, and the two of then both died. Mhy should I look at novie when could just visual effects supervisor at the time, Donnie McCarthy, vas not particularly interested in the novies, and dacidad that should just gat along with these actors, which is shat this story follows.
the actual movie? I'll give you clue. The
person Charlie can actually nest, brought up him by his mother, has to help her and reunite her with her children. Now, she ian't about to be in home, but ahe vants hin to be reunited with her. She wante Charlie to safe, that he and his nother are safe, and that he doeen't 't need go through much trouble and regret for hin and his mother. It's classic start to this story, so I'm not going to lia. It's the complate opposite of what vas made out there.
Charlie,
But when Charlie was brought up, the whole story about hov to build his own life and stay in the state Kentucky, be didn't need to that.
way, It all started with what I firet described in The Birth of Nation, when high school student named Emil Fayles tolls him, "You can't be that way. " He's no longer considered teenager by the state, and he lives in Loutwvilee, Kentucky. He maets his father and fathar's parenta, and he helpa hinself
it up he's nade everything right. That is, for hin and his family.
needed
fanily.
bag and goea the who doeen't have noney, or himeelf, who doeen't have a credit card. He giver cigarettee, and he starts
Buy.
keeps changing diapers. He helps uncle,
his dad about that, about vanting to better place.
oun choices, that he finally
He starts this story with hov Emil told hin to help himself. The vorld's avare of Enil's freedon, and he can do his thing, vhether that means house, car, schoal, his grandnothar. He then talls his nothar, who in some time in the past, that he's leaving his country.
helping
to, saye, get out
The rest of the story is very mach about a wonan who's trying to help herself. In this story, Bril se're
really going seeing in the novie. It's more of his time with her, with his father, and be tries to get out of hore, but at the end of the novie he tells her that he's going to nake a big deal out of the fact him.
nother. That's sonething he doesn't want to do, but
not
that she's going It'a a kind of adaptation of the "Belfish Girl" from the original series. You don't get to see all the backetory and where Emil is helping his nother and nov he's in a country where the government is corrupt, and mother's toiling asay, trying to To ne, that is kind of ny faverite part of the film, because it's almost like, "What if is leaving me and I'm going to have to fight with her every day?" It's very rare, bacause have been around, similar to the novie that I vas involved with in The Birth of a Nation.
these characters vere raised, but the euotional noment in very
things better for children.
like being
country vhara have fight for ny kida. And it's very very
Nstion London back the '60s, and actually true sequel or a remake, because can't vant to do. That's what liked about that novic, because I knew that this alvays be different fron any of the other novies in that time, and the people sho created in that world. That was ane of ny most faverite aspects of the novie.
vatching The Birth
not sure it was "We're not doing this because this is would
nean, that's the thing about moviea like Birth of Nation, and thought, "There's something about it that really makee it, and that, at the same tiue, really puehes ne to think about what can do this world and illustration of that.
can make vorld ny exa." It's perfect
I ваи that novie a lot, and knew that this was going to be one of the moat beautiful moviee of all tine. It's
39

---

Published as a conference paper at ICLR 2025
Samples from
4096 sleps:
you to select watch those channels on Apple TV (iPhone and iPad only) or on your phone.
app
channels watch every tine you open app so you can
For example, HB0 Go, Shovtime, ANC and other channels don't have to on Apple TV because the Roku app be paired with Apple TV to vatch then, according Scott Robingon, vice president of business development for PlayStation Network, Inc.
7. The Appflix
With this app, you can connect your TV renote to your phone and watch new shows and novies that are being addad your collection, according to Anazon. The Appflix app leta you watch those channels on your phone without uaing the remote.
will work well with the new Apple remote when it's released by Apple. Roku
This was supposed to be companion app for DiracTv. TV, but it's now being used by Roku. up your Roku
You
DVR you have your set-top it as a hub for their TV so that you have multiple channels and apps so you can look for the best content available.
Roku
It also provides you with aplit-screen streaning Teature that allows you to watch nultiple channele.
The app isn't connected your TV if you have Apple TV, but you or Chromecast, Amazon Fire or any Android davice. Looking back at appa released content from Roku on your TV. Presideett Barack Obana speažs Thureday in Vashington, accusing Noаcow AFP/Getty Images)
your Roku TV,
veek, may juat be
get
undermining the alliance. (Photoli: NOSCOV Geruan lavnaker said Preeident Bareck Obaaa should aek the United States to spend noney on military support and training in the region to prop up the Ukrainian governnent in the war against Russia.
In the and, NATO will have to stop the Russian aggreasion in eastarn and central Ukraine, parlianentarian Robert Appel on Thoreday alliance's 's headquarters in Vilnius, Lithuanis.
according
the
Specifically, he said Washington the pro-Ruseia forces fighting esstern Ukraine. That position been confortable with since the U.S. has called for military help.
more seapens military assistance fight
EATO Leaders have
The comments from the German lazaakr of his plan to send nore arma to the separatiata in Ukraine and his call for the United States to join
respanse Obana's announcement this week
U.S. and European leaders condemmed the victory of the separatists in this nonth's elections. daeply disturbing know that with the U.S. government," said Vice President Joe Biden.
that
approciates
"The president vas elected on the very platform that legitimates aggression in Ukraine," he added. The U.N. Security Council neating consider President Vladinir Putin to pay attention to the issue.
says confront Russia, and Ohana urged
'serious Obana said he is "very serieus about our security" and that shile the alliance is secking help from Noscou, "If the pattern Russian aggression continues, it places MAIO, the alliance, and ultimately the security of Europe and its allies in dangor," said the president at a Thursday nees conference. security crisia," security crieis in eastern Europe." support Ukraine
U.S.
limita
Europeana
Secretary of State John Kerry. "This serious
Appel focused his statement on the "turbulence factor, " citing NATO's relationship with both Russia and the way shich countertnaurgency oparations were saged againat Saviet Union. addition the support of the European Union, he said, the UN Security Council required to act againat Ruasia.
nenber," said, referring the alliance's is nemberehip under article 5 (a) of its constitution.
"Ukraine is
NATO nember,
well,
February 2014, making it a NATO menber.
Ukrainian government
itself to nember Buropean Union,
Tipping avay at the
alliance?
"You can raise the threat level with Buseis," retired Adn. Mark Green, the top commander of the American forces in Europe, said on CBS's "Thia * that if Ruzsis oueted have to be supported by other nembers of the allisnce "through the use of force, in such "persistent way."
Yanukovych's nevly elected governnent, vould
Green said the actions of the separatists
40

---

Published as a conference paper at ICLR 2025
Samples from SEDD-Medium, 4096 steps:
said.
"To ne, that's something that in going to happen any of us. You never know who is more prepared. you give, the guys are up and out of there. You know, if you get two guys off and they're fruetrated because they feel good, don't think it's going to nake the tean any better. "I's not confortable, and should be, I'm competitive. I'm just blocking every shot the way I do everything do."
And that be plans to get even nore aggresaive out there. different knock ny body out there and pushing nyself," " Jordie said. "I appreciate that, and the better get, the greater an advantage it will make me feel because the achedule is better, so we'll see what happens." than year ago, you be he has worked slovly to get better,
happens against sonehody
1, better,
going
Since the injury finally but he has gone through bis upsides as well.
"Initially they
they pretty painful," McKanzie sald. "But it vaen't so bad, I vaa figuring then out later." get tougher," " O'Regan added. "Be's I don't think there's anything he's done before.
pretty brutal before they happened,
improve
long vay get there,
"This is probably the time that I vorry he's going to have an issue like this from the off-seaeon. test his body going to be the only say that va can trust," O'Regan said. trying determine "I'll put nyself to the teste, but everyone has the same strugglee, and I'm really just bad with losing." he said, with laugh. "That's
"We'e 're going to
another
what happens that's
of nou, NcKenzie
going play
you do anything "That's how I'm Kind of living. I do need deal with that, but it's alvaye done it for me. If kasn't understand, wouldn't be able to.
you're dealing
wasn't hard, "The I undarstand that's taking so much of your tine and your trust. I'm not Hr. 'The Ugly." need
I walk feat. I'm kind of thinking that's all bad, but don't know.
"I' that kind of guy,
involved in camp be. I'm dam hungry and I'm going out to work every night. That's shy I don't plan on staying normally sould."
khy
NcKenzie said heving the kind of type of recovery that he really wants worth the scare be had when he received from the start vorkout the morning the Morys started the trip. "It kind of When you're dealing with that you can't do anything."
you sick thinking that," " be said. "That
intense workout.
know?
naesage and continues play a role in his body every single day. If the setup there was not successful for hin, he knovs that he won't be getting the nost
NoKenzie
he
time behind All told, McKenzie has full season off his injury be recovered and back into full-tine hockey. He still knows he is far fron unstoppable.
on.
"It'a frustrating right now."
still am, McKenzia said. knou this give, and can't give this
even then, McKenzie
worde, really any knov, deal with that, " he said, "I always keep that in mind. alvays what when through
"In the dreany comes it just know that hava.
it. just work harder and better every day. 'You'
41

---

Published as a conference paper at ICLR 2025
Samples from GPT-2-Medium, 1024 sleps:
senior Russian prosecutor, sumnonad by President Vladinir Putin for questioning about his alleged linka to the controversial bitcoin exchange Mt. Gox, has revealed he used the ill-gotten gains from the alleged theft of sone $230m in the troubled digital currency to buy a holiday hone in Californis.
Nassin Mikheev said he spent his veekend in Californin to buy a $d0n Sbree-bedroom house with and 18ft cailings and valla luxury property in 6,900 sq kitchen, 600sq ft living Nalibu.
kednesday, the judge presiding over Mikheev's preliminary hearing asked him about finances, telling hin he had invested "roughly 200m roubles (#175m) in Bitcoin," " using the digital currency to purchase a residence.
'I would like to make the remarks that when spent noney in Bitcoin and put it into my property, the proparty value increased tenfold," " Mikhoev replied. The price increase, he said, contract for the property eaying the currency had doubled in value.
fron Bitcoina used buy the hone, with one-year buyer'a
Nikheer's lawyer said he explained, expect that even if this anount was nistake, it was not juatified," " said lavyer Anstoly Semenov. "Ae he is not à defendant, we will not ask for any charges or plea." Russia's Federal Anti-Money Laundering Service (Banske)
to nount defence of his client's conduct.
already
Nikheev is currently under house has naned him and has requested access to all bank accounts in Ruasia and Kazakhatan, which cover least half billion dollars.
Nikheev said he first zet Mt Gox, a controversial bitcoin exchange, in 2009, There, the chief operating officer, Sergey Karpeles, said bought about one nillion bitcoins used the money pay people for work they did for hin and other nembers of the Bitcoin community.
After his father, Mikhail, the firet owner of the Nt Goz company, sold the business to a second company, he became disillusioned and had Nt Gox backed. At least $230n allegedly stolen. Karpales aventually laft the company and told nadia that he sould be "really happy to get it back, but in the neantine can't take it, Hie father then noved to Luzenbourg, where Mikheey now lives. à group of investigators now believe he stole the money through hacking, authorities. The case rensins under investigation company has since been under siege by the Gernan tax authority.
do do."
Citing the US criminal case against Kt Gox founder Mark Karpeles, Semenov said that "in this case would be necessary gave then their servicea to discredit this investigation, of which the Mt. Gox CEO is a key figure."
and shane the cyhercrininals who in exchange for stolen bitcoins
He added: "If at the time of one of his actions the prosecutor decided to introduce them for questioning, this decision would have been completaly based on trust that this person feels the prosecutors in general." "
Mikheer said he would like to start a new company or take up another career, but is concerned be will not be able renain Kazakhatan.
"I've been lucky to live here for year, and I vant to continue this condition in the future. But ny life not going to be easy se thought," he said.
The 6d-year-old's lavyer had urged the judge to immediately imprison him, adding that there are no limita to the anount time businessnan can be jailed for acting in a "dirty nanner." Judgo Rany Azizov said his authority vas constrained by "exxediteed" schedule of prosecution against alloged bank thieves. Azizov added that decision an the "vitness' role" could take some time.
But Nikheev stressed that he vould not be intinidsted by authorities, suggesting his ruling may falling victin to same actions. 'I've heard of people after acquiring the vrong identity docunents, using aliases or in cortain cases turning the srong cornar and, even if they usad the correct documants, this can't mean that I' actad with greas nogligenca."
holp prevent others
things
In response to Mikheev's ramarks, spokesman for the Mt Gox chiaf, Mark Karpales, told the Russian press: "There hundreds of people our company payroll, sone of then ex-enployees. We don't comment on personal mattere."
The case is "totally complex complicated," said Ross Meazoff, head of compliance at Kreha@nSecurityy, which monitors money- lauddeiann cases. "It is very important case, and the decision was taken to speed up this investigation in
F.3 GENERATED SAMPLES FROM MODELS TRAINED ON IMAGENET 256 x 256
In Figs. 13 15, we visualize samples of DDPD, Mask Diffusion and MaskGIT.
Without logit temperature annealing, Mask Diffusion captures diversity, but the sample quality suffers due imperfections in the denoiser. On the other hand, MaskGIT's confidence-based strategy significantly improves sample quality, but at the cost of reduced diversity. DDPD trades off diversity v.s. quality naturally without the need for any annealing or confidence-based tricks.
42

---

Published as a conference paper at ICLR 2025
T-B RID: RID:6.80 6.80
5.0
T-32
T-64
4.74
R3: 4.89
(a) DDPD: No Annealing
FID: 38.06
T-16 44
RI:: 25.12
T-6d
T=128 FID: 26.83
(b) Mask Diffusson: No Annealing
T-8
T=16 6.66
8.0:
R.08.08
T-178 9.34
(c) MaskGIT: No Annealing
Figure 13: DDPD v.s. Mask Diffusion v.s. MaskGIT, No Logit Annealing
43

---

Published as a conference paper at ICLR 2025
RID:
4.0:4.22
T-32
T-64
R.15.14
(a) DDPD: Logit Anncaling = 0.6
RI: 3..9
RI:
R.:4.86
T-64
T=128
FID:
(b) Mask Diffusion: Logit Annealing = 0.6
T-8 RD:
T=16 nD: 11.24 24
FID. 11.93
T-64 RD:
T=128 RI:: 12.52
(c) MaskGIT: Logit Annealing = 0.6
Figure 14: DDPD V.S. Mask Diffusson v.s. MaskGIT, Logit Annealing 0.6
44

---

Published as a conference paper at ICLR 2025
T-B FID: 5.99
5.0:5.03
T-32
T-64
RI::
(a) DDPD: Logit Annealing 1.0 + 0.0
RD: 3.99
4.69
R.24482
T-64
T=128
4.89
(b) Mask Diffusion: Logit Annealing 1.0 - 0.0
T-8
T=16 9.0:
9.99
T-64 RI:
T=178 FID: 10.20
(c) MaskGIT: Logit Anncaling 1.0 0.0
Figure 15: DDPD v.s. Mask Diffusson v.s. MaskGIT, Logit Annealing 1.0 -> 0.0
45

---

Published as a conference paper at ICLR 2025
(a) DDPD 32 steps
(b) Mask Diffusion 32 steps
(c) MaskGIT 32 steps
Figure DDPD v.s. Mask Diffusion v.s. MaskGIT, No Logit Anncaling
46

---

Published as a conference paper at ICLR 2025
(a) DDPD 32 steps
(b) Mask Diffusion 32 steps
(c) MaskGIT 32 steps
Figure 17: DDPD v.s. Mask Diffusion v.s. MaskGIT, Logit Anncaling 1.0 + 0.0
47

---

Published as a conference paper at ICLR 2025
DDPD 16 16 steps
(b) DDPD 16 32 steps
Figure 18: DDPD No Annealing, increasing number of refinement steps. The added refinement steps act as "touch-up" up" to improve the aesthetic quality without losing its original content.
48


Reviews:
[{"review_id": "kJvlE93cof", "reviewer": "Reviewer_uHmQ", "summary": "This paper introduces Discrete Diffusion with Planned Denoising (DDPD), unlike traditional discrete diffusion models which rely solely on a denoising network, DDPD separates the generation process into two distinct steps: planning and denoising. A planner network identifies the most corrupted positions within a sequence, prioritizing them for correction, while a denoiser network then predicts the correct value for these selected positions. This decomposition simplifies the learning process for each network and allows for a more efficient sampling algorithm. The paper proposes an adaptive sampling scheme based on the Gillespie algorithm, which dynamically adjusts the denoising step size based on the planner's assessment of the sequence's corruption level. This adaptive sampling, coupled with a time correction mechanism, allows DDPD to iteratively refine generated sequences by revisiting previously denoised positions if errors were made. The authors demonstrate experiments on text8, OpenWebText, and ImageNet that DDPD improves sample quality and diversity compared to existing discrete diffusion methods, particularly when inference compute budget is limited. Importantly, DDPD allows for leveraging pre-trained mask diffusion denoisers for uniform diffusion tasks, demonstrating improved performance by combining a strong mask denoiser with a separately trained planner.", "questions": "- What are authors' thoughts on the connections of this work  to the state-dependent masking schedule proposed in MD4?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The core idea of decoupling the denoising process in discrete diffusion into separate planning and denoising stages is novel and very interesting. The adaptive sampling scheme based on the Gillespie algorithm, combined with the time correction mechanism, is also a creative contribution that addresses limitations of standard tau-leaping samplers.\n- The paper provides a strong theoretical foundation for DDPD, grounding the training objectives in a clear derivation of the ELBO for discrete diffusion. The experimental results are comprehensive, comparing DDPD against strong baselines across diverse tasks including language modeling and image generation, showcasing its consistent performance gains. The ablation studies further strengthen the analysis by demonstrating the robustness of DDPD to imperfect training and the individual contributions of its proposed modifications.\n- The paper is dense yet generally well-written and easy to follow.\n- DDPD addresses a key challenge in discrete diffusion models: the efficient and effective use of the limited computational budget during sampling. By prioritizing denoising efforts through planning and allowing for self-correction, DDPD offers a promising path towards closing the performance gap between diffusion and autoregressive models in discrete generative modeling. The ability to leverage pre-trained mask diffusion denoisers further enhances the practical significance of DDPD, making it more accessible for researchers working with limited resources.", "weaknesses": "- The paper acknowledges the increased computational cost of DDPD compared to denoiser-only methods due to the additional planner network and the sequential nature of the Gillespie sampler.\n- The clarity can be further improved by adding more description of the training algorithm/procedure of the model confirugrations considered in Sec.6, given that there are many variations and the forward/training objectives/planner&denoiser training can vary.\n- Lack of empirical NLL/perplexity comparisons on OpenWebText dataset with baseline discrete diffusion models. While this work presents results and analysis in terms of generative perplexity on OWT, this metric is easier to be inflated and sensitive to the precision used in categorical sampling. The more commonly used validation perplexity is not reported, as well as the zero shot perplexity on other datasets.\n- Missing reference: The proposed adaptive time correction is interesting and well-motivated. Similar ideas/trick was proposed in [1] with asymmetric time intervals for a different context but not discussed as related work.\n\n[1] Chen, T., Zhang, R., & Hinton, G. (2022). Analog bits: Generating discrete data using diffusion models with self-conditioning. arXiv preprint arXiv:2208.04202.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x0aWODTEPd", "reviewer": "Reviewer_yRLR", "summary": "This paper introduces DDPO, a novel method that separates the generation process of discrete diffusion into two models: a planner and a denoiser. The authors unify the sampling processes of masked diffusion and uniform diffusion in a plan-and-denoise manner and present the advantages of this approach over previous works. Additionally, the authors provide the evidence lower bound (ELBO) for DDPO. Experiments in both text generation and image generation validate the effectiveness of DDPO.", "questions": "1. The term $p_{1|t}(x_1=j|x_t, z_t=D)$ (in Line 192) indicates that once a dimension is identified as data by the planner (i.e., $z_t=D$), it will not be updated in this step, similar to masked diffusion. I am curious whether a dimension that is recognized as data by the planner early in the sampling process remains unchanged until the end of the sampling. Could the authors provide further theoretical and experimental insights on this issue?\n2. DDPO employs a different training objective (i.e., Theorem 4.1). Will this new training objective lead to lower perplexity in language modeling?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper focuses on an interesting and important problem in the sampling processes of discrete diffusion models.\n2. The writing is clear. The authors logically present the sampling process of DDPO and its advantages compared to previous works. And even with the extra planner, the training objective of DDPO still corresponds to the evidence lower bound.\n3. The experiments are thorough, demonstrating the effectiveness of DDPO in both text generation and image generation tasks.", "weaknesses": "1. I suggest the authors use different notations for single-dimensional and multi-dimensional cases, e.g., x and **x**.\n\n2. The authors claim that each sampling step of previous works predicts single-dimension transitions but not joint transitions of all dimensions. However, based on Proposition 3.1, the proposed DDPM also just predicts single-dimension transitions.\n\n3. As presented in [1], the best FID score on ImageNet 256×256 is 1.97. Why are all the results in Table 2, both for the baseline and the proposed DDPO, significantly worse than 1.97? Is it because classifier-free guidance was not used? If so, why are there no results that include classifier-free guidance? This is my main concern; I will improve my score if a clear response is presented.\n\n[1] Yu et al. An image is worth 32 tokens for reconstruction and generation. NeurIPS 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ni0nl2ryRY", "reviewer": "Reviewer_4zHc", "summary": "This paper rethinks limitations of the conventional discrete diffusion generation schema that the generated tokens cannot be further corrected and the sampling process cannot be controlled. The authors propose to disentangle the single diffusion denoising process into two models: a planner and a denoiser. This planner is capable of selecting which positions to denoise next by identifying the most corrupted positions in need of denoising, including both initially corrupted and those requiring additional refinement. The plan-and-denoise method allows for adaptive control of the denoising order for efficient reconstruction during generation. The experimental results across text and images domains demonstrate the efficacy of the proposed model.", "questions": "see weakness", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The motivation behind the study is sound, as discrete diffusion models struggle with correcting generated tokens and exhibit inflexibility during the sampling process.\n2. The proposed model to disentangle the generation process with a planer and a denoiser is simple. \n3. The theoretical analysis appears robust and well-founded.", "weaknesses": "1. Additional experiments are necessary, such as applying the model to machine translation tasks for texts.\n2. Although the planner model can adaptively organize the generation process, performance degradation becomes an issue as the number of sampling steps increases.\n3. The narrative is somewhat difficult to follow; I recommend that the authors clarify the main storyline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "opoxr57hQF", "reviewer": "Reviewer_p5AT", "summary": "This paper improves on the recent discrete diffusion generative models, with a focus on language modeling. This paper shows that the reverse (generative) rate can be decomposed in a planning and a denoising part. The proposed decomposition optionally allows training separate models for each part, which yields higher quality text samples, for a given inference budget. The decomposition also allows re-using a pre-trained masked discrete diffusion model to sample from a diffusion process with uniform stationary distribution. The training objective is theoretically sound as a decomposition of the ELBO.", "questions": "- Did you experience convergence issues during sampling? More precisely, what happens if you do not limit the maximal number of planning+denoising steps, but iterate until the planning module is sufficiently confident (e.g. assigns a probability at least 0.95, or 0.99) for all the tokens? \n- Related to the above, say you are using a confidence threshold of 0.95 to decide on the convergence. What is the mean/std of the number of sampling (planning + denoising) steps that you observe when averaging over 10'000 examples?\n- Do the authors have an explanation or hypothesis on why adding a planner seem detrimental to performance for images, but not for text (table 2 with logits scaling)?\n- Have the authors experimented with other types of data already such as DNA sequences? I am curious about such experiments since the setup is relatively different (smaller vocabulary and longer sequences).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The presentation on Plan-and-Denoise decomposition is clear.\n- The comparison of the sampling algorithm (Gillespie) and the comparison with tau-leaping is clear.\n- Generally, Section 3 (method) is easy to follow and well-written.\n- The note at the beginning of section 5 (experiment), that DDPD uses 2x the number of NFE versus regular absorbing diffusion is good to place there.\n- While the main focus of the paper is on discrete diffusion for text generation, the authors have included experiments on images as well.", "weaknesses": "Overall, the method is sound, and the experiments are conducted fairly. The weaknesses noted here lie in the writing, which motivate my score. I have divided my comments into two groups: the first addresses more critical issues that I believe should be resolved, while the second includes suggestions that, while less urgent, could enhance the paper.\n\n\n### More critical weaknesses\n- Because of the description of the Gillespie's algorithm (Algorithm 1), it seems that each sampling iteration updates a single token at a time. But then, I'm confused on how the number of sampling steps can be lower than the number of tokens in Figure 2 (middle, DDPD). In particular, if you are generating sequences of 256 tokens on text8, how can you sample with 250 steps ? Indeed, there is a curve labeled DDPD-T250, which implies only 250 steps, so it feels like a piece of information might be missing here.\n\n- Training Objective (lines 376 to the end of the page): The sentence on the 'coupling dynamics' and how it negatively affects backpropagation is unclear and should be rewritten.\n\n- In Figure 2 (left), it appears that DFM-UNI, was only evaluated at a single temperature (shown by one dot) and others, like DDPD-MaskD, at just two temperatures (with only two dots). Could you confirm whether the models were indeed evaluated across all three temperature values? The missing points on the curve could be explained by missing experiments or dots that fall exactly on the same position for example. Therefore, I think there should be an explanation on this phenomenon.\n\n\n\n- The meaning of the stochasticity parameter ($\\eta$) on lines 431-432 (end of page 8, beginning of page 9) is not explained. While the context suggests that it comes from the DFM paper, a brief explanation of how it is used here is justified. I suggest adding a concise, high-level sentence about $\\eta$, and if more detail is needed, directing readers to a section in the appendix.\n\n\n\n### Less critical weaknesses\n- Sampling with the Gillespie's algorithm requires sampling the holding time of the markov chain from an exponential distribution. However, for a reader that is not well-versed in simulation of markov chains, it might be unclear why the parameter of the exponential is chosen by summing over the rate for each dimension. I believe it would be good to include a paragraph on this, or at least a footnote to a precise chapter in a book such as the book of J.R. Norris on Markov Chains.\n\n- For a reader short on time, it might be unclear what 'softmax selection' and 'proportional selection' refer to on line 476, as they haven’t been introduced previously. I assume these terms relate to the Gillespie sampler, where a dimension is chosen either by softmaxing the model output or dividing by the sum. To clarify, it would be helpful to label these explicitly. I would suggest mentioning \"proportional selection\" in bold in section 3.2, and mentioning that alternatively, one could perform \"softmax selection\" as well, again in section 3.2. This would make it easier for a reader that would search for \"softmax selection\"/\"proportional selection\" in the pdf.\n\n- It would be helpful to include a timing comparison between DDPD, the diffusion baselines and the AR model. Indeed, a comparison of the computational resources vs generation quality is relevant for many applications.\n\n### Final comment\nI realize that implementing these suggestions would require adding content while you already reach the paper's page limit. However, I believe my requests are justified in order to improve the clarity of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)