Why Masking Diffusion Works: Condition on the Jump Schedule for Improved Discrete Diffusion
Authors: Alan Nawzad Amin
Nate Gruver
Andrew Gordon Wilson
Venue: NeurIPS 2025 poster ()
Keywords: Discrete diffusion
proteins
language
masking diffusion
structure
Primary area: probabilistic_methods
TL;DR: We investigate the reasons masking diffusion dominates discrete diffusion.

Abstract: Discrete diffusion models, like continuous diffusion models, generate high-quality samples by gradually undoing noise applied to datapoints with a Markov process. Gradual generation in theory comes with many conceptual benefits; for example, inductive biases can be incorporated into the noising Markov process. In practice, however, the consistently best performing discrete diffusion model is masking diffusion, which does not denoise gradually. Here we explain the superior performance of masking diffusion by noting that it makes use of a fundamental difference between continuous and discrete Markov processes: discrete Markov processes evolve by discontinuous jumps at a fixed rate and, unlike other discrete diffusion models, masking diffusion builds in the known distribution of jump times and only learns where to jump to. We show that we can similarly bake in the known distribution of jump times into any discrete diffusion model. The resulting models -- schedule-conditioned diffusion (SCUD) -- generalize classical discrete diffusion and masking diffusion. By applying SCUD to models with noising processes that incorporate inductive biases on images, text, and protein data, we build diffusion models that outperform masking.

Why Masking Diffusion Works: Condition on the Jump Schedule for Improved Discrete Diffusion
Alan N. Amin New York University
Nate Gruver New York University nvg7279@nyu.c edu
Andrew Gordon Wilson New York University andrewgwdcims nyu edu
alanamin@nyu.co edu
Abstract
Discrete diffusion models, like continuous diffusion models, generate high-quality samples by gradually undoing noise applied to datapoints with a Markov process. Gradual generation in theory comes with many conceptual benefits; for example, inductive biases can be incorporated into the noising Markov process, and access to improved sampling algorithms, In practice, however, the consistently best perform- ing discrete diffusion model is. surprisingly, masking diffusion, which does not denoise gradually. Here we explain the superior performance of masking diffusion by noting that it makes use of fundamental difference between continuous and discrete Markov processes: discrete Markov processes evolve by discontinuous jumps at fixed rate and, unlike other discrete diffusion models, masking diffusion builds the known distribution of jump times and only learns where to jump to. We show that we can similarly bake in the known distribution of jump times into any discrete diffusiin model. The resulting models - schedule-conditune diffusion (SCUD) generalize classical discrete diffusion and masking diffusion. By applying SCUD to models with noising processes that incorporate inductive biases images, text, and protein data, we build models that outperform masking.
Introduction
Discrete diffusion models provide state-of-the-art conditional generation of discrete sequences. In biological sequence design, for example, they allow one to generate sequences flexibly conditioned on protein structure (15), DNA function [22]. protein family [1] and other properties [9 16). They
are also nearing of generation on language data 220
A diffusion model is defined by "forward" process, which gradually transforms data token-by-token into noise, and a "backward" transformation that turns noise into data, learned by optimizing an evidence lower bound (ELBO). In principle, the quality of the learned model should benefit from forward process that captures structure the data distribution. For example, works have suggested forward processes that are more likely to transform tokens into similar tokens a more "gradual" noising process 1 as well as "state-dependent" processes that transform certain tokens more quickly than others 2 Surprisingly, these methods are all outperformed by "masking diffusion" which has the simplest possible forward process transforming each token into a masking token at a uniform rate 23) 23). Having seemingly arrived at the optimal forward process, work discrete
diffusion has instead shifted its focus to sampling [26] [8 [33, 774 and scaling [13] 20).
Here we explore the causes of the superior performance of masking diffusion. We propose that masking diffusion benefits from a parameterization that forces the distribution of corruption or transition events, the "transition schedule", in the backward process to match the distribution in the forward process. Rather than allow us to conclude that masking optimal, this insight allows us to expand the design space of discrete diffusion models to give any forward process the same property; we call models in this expanded design space schedule conditioned diffusion (SCUD). We show that
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Forward Noising Process
state
SCUD: :
Uniform
NLL per bit
2.15
Gaussian
Classical diffusion: learn when and where
Classical
state state
diffusion:
2.05
Uniform
Masking
SCUD: build when, only learn where
1.95
Gaussian
0
0.5
1
Conditioning (y)
time
Figure 1: Left: When trying fit forward process in diffusionn SCUD builds in the known distribution of transition times. Classical discrete diffusion learns to reverse the paths generated in the forward process by learning both when and where to transition. SCUD instead builds in when to transition from the forward process, and therefore only must learn where to transition. Right: Conditioning on more information about the transition time schedule results in better likelihoods. Models are fit on CIFAR-10 with B 128 states. We show mean and standard error over replicates. Details are App.
applying SCUD to discrete diffusion with uniform forward process, the result is masking diffusion, explaining its superior performance. We finally unlock the potential of structured and state-dependent discrete diffusion by building SCUD versions of these methods and that they finally beat masking diffusion, achieving state-of-the-art results on images and proteins (Fig D. We release our code at https::/ tigtt........ com/ www.th......aaalaaaaaaaaaaaoooaa /tamm//aaaalwaaaann/SSSU
Background
Our goal is to model data from distribution p(ag) where wo is sequence of discrete elements that belong to a set of size B. We consider the one-dimensional case first, and sequences later.
Discrete diffusion In diffusion, start with distribution that is easy to sample from, g(z1); )); then learn a parameterized Markov process from time to time 0 that evolves samples from ((x1) to distribution qo (2x) that is approximately p(zg). To learn Markov process that evolves (x11 to p(zo), we first pick simple Markov process that approximately evolves samples from p(rv) to ((x)) from time 0 to 1; then we try match the trajectories from the parameterized Markov process qo ((xx)))) (1))))))..1) that evolves "backward" from time to to those of the simple process p((x, (3=e)e((o.1) that evolve "forward" from time
We do by maximizing the evidence lower bound
(ELBO)
Ep(za) log (ooo) 2 Ep((ss)ccto.) log qo (1x))))(0))
p( (((())cc(0o1))
qo (((())te(0))))1 x1))
g(01) p(a1 (mr)
-Ep((es)etoor)) log
p( (x+x ((xx))s<(0)1)) 20, #1) Epte log
This ELBO maximized when the distribution of forward and backward trajectories match. The second term of the right hand side measures if the forward process indeed evolves samples xn p(ro) (())). The first measures how well the forward and backward trajectories match.
Discrete Markov processes and infinitesimal generators To define a diffusion model, we need to define simple Markov process to generate p((x))<e(0.1) and we need to parameterize the backward Markov process. Fortunately, discrete Markov processes are much easier to define than their continuous counterparts. Every time-homogeneous discrete Markov process is fully described by B x B matrix that describes the "flow" of a particle at each instant in time known as the infinitesimal generator L. particular, L,,V/ describes the rate which state transitions to state b'; the diagonal of L describes the rate of transitions out of b: La.s Eveo L... Therefore, to simulate from
Markov process described by L, starting Ie, simulates the time which z; would transition to

---

Masking
Classical
error of transition rate
D3PM on UniRef50
TLDR on CIFAR-10
0.0
5
-0.5
0
time
time
Figure 2: State-of-the-art discrete diffusion models have backwards processes which do not match the forward process in when they transition. Masking diffusion models have no such eπrоr as they build in the known forward rate into their backward process (dotted line). We plot the estimated transition rate of the backward process minus that of the forward process. We discuss details in App. D
each other state A4, ~ Exp(Cr,,b) for b & Fz; then one transitions 24 according to the first transition sampled: it take At min; Af, time transition and H, transitions to Fr+ argmin, Atg. By a property of exponential distributions, the transition time is distributed according to the value on Lmo.b) Exp(-Lao.0o) This procedure known as the
the diagonal of L: At Exp(Enxzo
Gillespie algorithm
8
Picking the forward process Two popular choices for the forward process are the uniform and masking processes. The uniform process has constant rate of transitioning to any state. and the masking distribution has a constant rate transition a masking state @. Both these processes are straightforward to simulate: simply sample At ~ Exp(1), and then transition to uniformly random state or to 0. Other processes bake in inductive biases for text, images, and proteins [.
For typical Markov processes, information about the starting state Zo becomes lost as gets larger and p(z+) gets closer to stationary distribution p( This distribution a natural choice for ( as long as p(2) (mo) is close to converging to the stationary distribution.
In practice, p(an (an)) is usually not near p(ato))
modulate the speed of the process by rate 3 at time t: at the instant we simulate from the process L, Simulating this modulated process for time t is equivalent to simulating the original process for time J 3,ds. By choosing B, to become large as t + 1, we can be sure p(x1|z0) Ro p(z,o) = q(11).
Parameterizing the backward distribution The backward Markov process is usually defined in terms of a parameterized, time-dependent, infinitesimal generator Lo The first term of Eqn. T is usually written as integral in time Ex-Unit(o.)) I(C,, t), for some L which intuitively measures how well the Co. describes the "flow" of the reversal of p((x)))) at instant t 3,
Learning when and where to transition
To fit a discrete diffusion model, the backward process should match the forward in both when it transitions and where it transitions to. One should expect that learning where transition is hard; on the other hand, since the distribution of when to transition is simple and known priori in many cases, one should expect learning when to transition should be trivial. We see however in Fig. that learning when to transition may also be challenging: state of the art published diffusion models have detectable differences the transition rates their forward and backward processes.
Unlike previously derived forms of the ELBO which are written as an integral of the discrepancy of the flow each moment t, we will break up the ELBO into discrepancy of when and where to transition. Define the "transition schedule", S {t,, t2..... tas }. as the set of times at which x; transitions.

---

Proposition 3.1. (Proof in Prop A.1 in the Appendix) The expression in Eqn. 7 is equal to the expression in Equ 2 for some constant C. Ep((m)))) log go p((az) To, x1,
(L(p(S))l90(5)) - 2.)) (x11))) + C.
(2)
The first term represents the difference in log likelihoods between @o and p when the transitions known, which measures if the forward and backward processes match where they transition to. The second term measures if the forward and backward processes match when they transition. The third term, like the second term of Eqn. intuitively measures if p(z) (mp) has converged to p(z)0))
To build diffusion models that better fit their objective, we therefore would like to incorporate knowledge of p(S) into the model. Eqn 2 is suggestive of how to do this: set g(S) = p(S) so that the second term becomes 0 and then learn where to transition by optimizing the first term. We call this procedure "schedule conditioning" (Fig. I and in Sec. 4 we describe how to perform it in practice.
Unlike diffusion models with the uniform forward process, diffusion models with the masking forward process are parameterized so that the distribution of times at which tokens are masked matches the distribution of times at which they are unmasked: these models know when to transition ( Fig. 2). In practice, masking diffusion models have been observed outperform uniform diffusion models [4]. In Sec. 5 we will prove that applying our methods in Sec. gives exactly masking diffusion, explaining their superior performance. By schedule conditioning other processes with more appropriate inductive biases, we also improve masking diffusion (Fig I.
4 Scheduled conditioned diffusion (SCUD)
In this section, motivated by Eqn. 2 we describe how to build discrete diffusion models that incorporate information about when to transition into a discrete diffusion model. We call such models Scheduled conditioned diffusion (SCUD) models.
Ideally we could set (S)) = P(S); however, in general, L may not have constant transition rates at each state, in which case S may be correlated with To and p(S) may be a complex distribution. Here we build a family of SCUD models that, instead of looking directly transitions, introduce latent "events" which will act as transitions did above. These events occur with constant rate and often result transitions; in some cases we discuss below, they will coincide exactly with transitions. We will condition on the schedule of these events, S.
In Sec. 4.1 we will describe models that condition on these event schedules, SCUD. Next in Sec. 4.2 we will write the loss in a form that is easy to train on high dimensional data. In App. B we will describe how we use standard techniques in discrete diffusion to parameterize our de-noising model, sample from SCUD, and choose the rate function 3
4.1 Conditioning on event schedules
Markoy processes with event schedules To sample from uniform forward process starting at Wt, we sampled a transition time according to a rate that was independent of the current state, At Exp(1), and then sampled Az+At with uniform probability. Consider more generally the discrete Markov process on Zt such that we sample an "event" Al. Exp(r), and then sample normalized distributions; note
It-At Categorical( ....) where Kz,,- is matrix whose in this case z; may be equal to Z1+At- By appealing to the formal definition of L, the next proposition tells us that this process has infinitesimal generator that flows according to the rate x K, with - to describe the flow out of x.
Proposition 4.1. (Proof in Prop A.2 in the Appendix) The infinitesimal generator of this process
== r(K - I) where is the identity matrix. In particular. any Markov process with L can be simulated in the above way by picking an r 2 maxs, Lo,b and setting K = C/r +
We note there are many choices of r that allow one to write the same Markov process in this way. We will evaluate different choices in Sec. 5
Reversing the process conditioned on the event schedule Equip with the definition of events, now try to show we can reverse the distribution of paths by just predicting what 2; was before

---

Classical diffusion
SCUD
00,0
3
10
23
19
0.2
Z0,0
x 0,0
1
10
23
12
7
Et
t
It
St
Figure 3: When predicting the de-noised data zo from the noised-data It. SCUD uses more "fine-grained" information about the number of corruptions 8; rather than just the corruption time l. Classical discrete diffusion trains model To.0 to predict the uncorrupted sequence the corrupted sequence Ze and information for how long it's been corrupted for t. SCUD trains a model to predict žo, that replaces with more fine-graine information about how many corruptions have been applied to each token 8t, which be thought as a measure of how "masked" each token is.
from
each event. Call pl (e)))) the distribution of paths that start at p(ro) and evolve according to the above Markov process. The next proposition uses algebra to suggest that simulate from p((x)))) "backwards" by 1) sampling the ending point 21 ~ p(x, ). 2) sampling the event schedule LM} p(S), and then 3) going backwards, sampling where the particle came from at
{41, (2,,62.... ..., (2,,.., the mn-th event.
Proposition 4.2. (Proof in Prop A.3 in the Appendix) Call the event schedule S {ti, t2,..... ............. ,tt) and to = O. Call 9t the number of events up to time t, so St.
m.
S) II p( It stm
We now aim model this backwards process.
schedule conditioned discrete diffusion models As suggested Sec we wish to build discrete diffusion model go by setting ((x1) = p(a.s)) and ((S) p(S). Prop. 4.2 suggests parameterizing q so that, at each event, it predicts the previous state Vt,,, given 1) the current state St.. and 2) the number of events that have occurred so far 8t. We call such a model a SCUD (schedule conditioned diffusion) model. With some algebra, in analogy with Eqn. 2 we get a closed form objective.
SCUD:
Proposition 4.3. (Proof in Prop in the Appendix) Calling the event schedule S = {t1, .2....,tA., and to = 0, Ep(ro) log 9 (zo) greater than
M E m==
(I(p(rem_,124 zo. Stm. )llgo /T St..)) ( (L zo) (p)(s()))))
where the first expectation is over p((x))e, S, This objective is minimized when qo (#tmm11/r,, st... ) p(x I 1 Stm..
The first term teaches where to go at each event. The second term is small if p(r1) converges to p. By Prop. 4.2 then, the objective in Eqn. is minimized, go((xz)) approaches p (())))).
Computing the objective The ELBO in Eqn. iss straightforward to compute. To calculate the first term, we note, writing each state one-hot vector,
p( (ro, 8: (() Ftm 8t) I K
KItm
p( ( Ieem Zo, 8e)
(5)
p( Ja0 S)
x K
Dtm
To calculate the second, we note p(xce) the left eigenvector of L that corresponds the eigenvalue 0 (as it does not change under flow from L.) and p(æ1 = x K
4.2 High dimensional data
For high dimensional data such images. language, and biological sequences, is common to choose
processes C that act on each dimension independently. Say our data has D dimensions e/!...,.7 x
with each ag a discrete object in a set of size B. We extend SCUD to this case by simulating D
schedules for each dimension $1 ...... SD p(S); so s, becomes D-dimensional vector (Fig. 3
5

---

Parameterizing For a time t, if s >> O, define pr(2{) as the state at the last event in dimension d, and pr(x,) the previous state at each dimension; i.e., the event schedule dimension d is Sd (tf...... ................................ km and E [th +m))), then pr(xç) = znd Our formula for reversing p((x)))) in Prop. 4.2 remains the same, but in App. B.2 we show p(pr(ar,) 8.) factorizes. Thus parameterize our predictor qo (p(p((=))) Ft; 8t) so it also factorizes as I do (pr((x)))a, 8r). Thus get an objective in Eqn. but with sum over D in front.
Efficient loss We could in principle use our objective in Eqn. Eqn.Abb by taking empirical estimates of the expectation and the sum over events. In this case, however, each empirical sample corresponds to one event which affects a single dimension d, so it only checks the prediction qo (pr((x]) | 8c). The loss of other diffusion models, written as Et-UUnf(0,)) Ea (p(pr, Σx=1 (Co. It, t/ C). allow one to sample i and then check the predictions of (p(pp<z ( [22, 8e) at that time for every d in parallel. To write our objective in a similar form, we sample ~ Unif(O, 1) and then add a weight of B Jo 3,ds representing how likely an event is to occur at the instant t:
Proposition 4.4. (SCUD loss) (Proof in Prop. A.6 in the Appendix) The first term of Eqn. is
Be
Σ CoKKLp(pp(x)))a
(6)
So 3,ds
We can approximate this objective by empirical estimates of all the expectations and optimize with minibatch gradient descent. For a single evaluation of qo we can predict pr(x:)) for each dimension d in parallel and check whether it matches the forward process along every dimension. The algorithm for calculating an estimate of the ELBO for a To is summarized in App. B.1
5 Connection SCUD to classical and masking discrete diffusion
To incorporate information about transitions into qo. we wish to condition on the schedule. We described how conditioning on "events" in the previous section allow us incorporate this structure. However not every event corresponds to a transition. The amount of information about the transitions that we bake into our model depends the diagonal of K, the probabilities of transition an event. In turn the diagonal of K will depend on our choice of the rate of events r. For a fixed L, we choose any rate 2 r' max, LL,bb- Let's parameterize our choices of rate with a parameter y: let r - When y is 1, the rate of events is as slow possible; when 7 - 0, the rate of events goes to oc. y therefore represents the amount of schedule conditioning.
We now show that when L is uniform and = 1 1- 1/DD, that is, we nearly fully condition on the schedule, our process equivalent to masking diffusion. On the other hand, as 7 - O, we learn a backwards process while baking in no information about transitions; we show this recovers classical discrete diffusion.
5.1 Connection masking diffusion
Say y = 1-1-1D and L is uniform: La. is 1/B when b b b'. For this choice, K is a matrix which has 1/B at every position. If a token is corrupted least once by K then it distributed uniformly; it tell us nothing about zo so it is as if that token is "masked". When we condition on the event schedule, 5; will tell us exactly which positions are masked when sy 0. By integrating out st conditioned on the mask, we get exactly the masking diffusion objective (23].
Proposition 5.1. (Proof in Prop. A.7 in the Appendix) Call the masking indicator m ss > Fo. s1) only depends on St through my Defining = exp( - J' B,ds), the objective Eqn.
6 is
B
E Dpppe
Σ
T log To ( lo.,
Ct.
The mask m, is distributed as mf Bern(1
-a.)
In App. B.4 we also show that our choice for rate 3e for this SCUD process linear (in the sense ag = 1 - t), just as for the masking process as discussed in 22)

---

5.2 Connection to classical discrete diffusion
As -> 0, each event represents an infinitesimal change in æs. Additionally, the number events up to time St, grows larger but fluctuates less and less; imputting s; into (px( sx) becomes approximately identical to inputting the time t into qo. Therefore, as 7 + 0, ge predicts the infinitesimal change at time t: the infinitesimal generator. This is exactly the objective of classical discrete diffusion. The next proposition shows that when we take the limit y + 0 we recover exactly
the loss from SEDD [15] which is also equivalent that from rLDR 3]
Proposition 5.2. (Proof in Prop. A.8 in the Appendix) A 0 the objective in Eqn. 6 converges to the SEDD loss (Equ. [7]
In App. B.4 we show that our choice for B, approaches that for classical discrete diffusion as 7 - O.
6 Results
We apply SCUD to build an expanded design space for discrete diffusion models. We first demonstrate the results of Sec. that SCUD with a uniform forward process interpolates between uniform and masking discrete diffusion. We next show that exploring the SCUD design space can unlock benefits from structured forward processes we demonstrate state of the art model fits to images and protein data. Finally, with a case study in language, we discuss the practicality of SCUD SCUD incurs minimal computational overhead, and, with some clever model decisions, can actually enable model decisions that were previously computationally intreactable. Other experimental details are in App.
6.1 Connection other models
We show that by incorporating information about the distribution of transitions into discrete diffusion model, one gets better fits to the forward process. We fit models to CIFAR-10 where each pixel takes value from 1 to B 128. In Fig. [ we see that on this dataset discrete diffusion with uniform forward process is outperformed by masking diffusion. We see that sweeping y between 0.1 and 1, SCUD with the uniform forward process interpolates the performance of the two models.
Next we build a structured forward process that builds in the inductive bias that similar pixel values describe similar colors- we set Lij = exp( 200 (==²)² for i + j. similar to the discrete-time Gaussian forward process in Austin et 2 We see that a discrete diffusion model with this forward process slightly outperforms masking distribution. We next build SCUD models with this forward process; we see that these models better fit their objective as we incorporate more information about transitions y 1. These models outperform models that have structured forward processes (Gaussian) or those that just condition on the transition schedule (masking) without doing the other.
6.2 SCUD allows models to leverage structure in the forward process
Many previous discrete diffusion papers have noticed that masking is optimal or near-optimal across wide range of domains, even outperforming bespoke forward process 2 Do structured forward process, which bake in inductive biases of the data, not help modeling? In this section we show that when we account for schedule conditioning (y 1), these bespoke processes actually substantially improve model fit on image and protein data. In particular, we achieve state-of-the-art diffusion model fits on these data by combining the schedule conditioning of masking with the strucuter of bespoke forward processes.
The structured forward processes we build for each modality will be inspired by those from Austin et al. (2) and Alamdari et al. [. However these papers used processes in discretized time that are not equivalent to and continuous time Markov process; thus we describe new structured processes for continuous time in terms of L or K.
In all cases we aim to make minimal modifications to the architecture and training from previous models so that differences in scores are due to schedule conditioning. We propose several techniques that enable moving from classical discrete diffusion to SCUD without substantial computational overhead, summarized in App. D.7 We also compare SCUD with our own re-implementations of classical and masking diffusion to best measure the effect of schedule conditioning and structure.
7

---

Classical unif.
Classical struct.
Masking (= Scud unif.)
SCUD struct.
5.08
18.82
4.0
5.0-
3.44
3.59
16.5
18.0
17.16
2.84
2.83
14.9
2.78
perpiens
14.70
8is per
2.76-
2.74
14.7
14.54
2.72-
14.5
D3PM *LDR MDA
implemantations
D3PM
D3PM D3PM
implementations
(a) Images
(b) Proteins
Figure 4: Schedule conditioning unlocks improvements from structured forward processes on images and proteins. We compare discrete diffusion models from the literature and implementations and report model fit in bits per dimension on CIFAR- 10 and Perplexity on Unire150, Models labeled "unif." use a uniform forward process and those labelled "struct." use structured forward process- Gaussian prior for images and a BLOSUM prior for proteins (see App. C
Images Here we build models on CIFAR- 10 with B = 256 and compare to state of the art diffusion models. We use the architecture from
in discrete diffusion models MD4 [23] and similar to that in D3PM (2] and 7LDR [3). To incorporate s into our function, we replace additive layers that inject into every layer with FiLM 19 layers that incorporate 8, into every layer. We also use the logistic parameterization from Salimans et al. [21] also used in D3PM, which interprets the output of the model as the parameters of discretized logistic distribution over pixel values, so that similar pixel intensities have similar probabilities.
In Fig. 4a we compare SCUD with discrete diffusion models D3PM 2], 7LDR (3, and MD4 [23 as well as our implementations of classical discrete diffusion models. We see that applying SCUD model the Gaussian forward processes substantially improves likelihood with a fraction of the compute. Among previous discrete diffusion models, masking diffusion is the most performant despite incorporating inductive biases. When accounting for schedule conditioning however, a structured process beats a uniform process (Scud str. beats masking / SCUD uniform); in particular, we achieve state-of-the-art ELBOS on images by unlocking the benefits of structured forward processes. This suggests that masking beats Gaussian diffusion in classical models because the benefit of schedule conditioning outweighs the benefit of incorporating inductive biases. By both incorporating inductive biases and schedule condition, SCUD unlocks the potential of Gaussian discrete diffusion images.
In App. D.I we examine samples from SCUD Gaussian. The samples resemble real objects much more than those from autoregressive models PixeICNN++ [21] and PiXeISNAIL (5) which have fate-of-the-as likelihoods. Furthermore. they contain clear objects like those from D3PM [2] or TLDR 3i MD4 did not show or evaluate images. Future work could apply the many methods for improving sampling quality of discrete diffusion models SCUD we focus our exploration here on model fit as measured by likelihood.
Proteins Here we train models on the UniRef50 protein dataset with architectures from [. As in [ we build forward process using the BLOSUM matrix; this matrix describes the rates of mutations between amino acids seen nature. We describe the details our process in App we note B 31 20 canonical amino acids 11 special tokens.
In Tab. we compare SCUD BLOSUM with the small D3PM models from M as well our implementations of classical discrete diffusion models. We see again that applying SCUD to uniform and BLOSUM diffusion substantially improves the model fit given fraction of the compute budget (See App. D.6). In classical discrete diffusion, masking strongly outperforms BLOSUM diffusion; interestingly, our reimplementation, classical BLOSUM outperformed masking. Nevertheless, by both schedule conditioning and incorporating inductive biases, SCUD BLOSUM outperforms masking and classical BLOSUM, achieving state-of-the-an model fit. In App. D.2 we show a similar result when starting from pre-trained weights.

---

Method
Fwd. process Perplexity
D3PM
Masking
76.9*
D3PM
Graph
149,5*
SCUD
Masking
37.82
SCUD
Graph
37.63
Table 1: Schedule conditioning improves uniform corruptions and accommodates structured
forward processes. We compare to other discrete diffusion models on LMIB. *Numbers are not directly comparable with D3PM as a different tokenizer was used.
6.3 Scaling case study: Language
In all cases above, SCUD and classical diffusion had very similar run times within 10% of each- other. This is because their loss computations are very similar and we only slightly changed the architecture of neural network accommodate SCUD; full explanation is in App. Here show that the SCUD design space also allows one to build models that are computationally intractable using classical discrete diffusion.
Above we looked whether bespoke structured processes from previous works could help modeling. Austin et al. 2 build discrete-time "nearest-neighhon" graph" process on language and saw that it was substantially outperformed by masking; could we test if this process also helps modeling?
We must build models on the one billion words dataset with B 30522 vocabulary size. Un- fortunately, the continuous-tmme classical diffusion loss requires calculating exp(tC), which is prohibitively expensive when B = 30522; indeed Luo et al. [T5], Sahoo et al. [20) restrict L to be uniform or masking process so they can calculate this quantity in closed form.
SCUD on the other hand only needs to compute Kt, We therefore define sparse 10-nearest neighbour graph over the most frequent 2000 states, which make up 95% of tokens in the data. Our forward process diffuses along this graph with some probability or transitions approximately uniformly with some small probability; the less frequently used 25 thousand states always transition uniformly. We discuss the details in App The result is that K is sparse that Ka can be efficiently calculated (computational complexity details in App. D.7).
Now able to implement the nearest neighbour process from Austin et al. 2 in continuous-time models, in Tab. we compare the effect of incorporating this structure with SCUD with the results in Austin et al. 22 In 22, masking strongly beats discrete diffusion with nearest neighbor structure this dataset (D3PM Tab. []. Unlike Austin et al. (2). accounting for schedule conditioning, when we add structure the forward process, improve our fit by small amount. Unlike in images and proteins, the improvement is not substantial however, suggesting that this process could be improved.
7 Discussion
The choice of forward process is critical to the definition of a discrete diffusion model. Yet previous results have shown very strong performance from the simplest forward process the masking process. SCUD offers an explanation for this: masking incorporates information about the transition schedule. By incorporating this information into models with other forward processes, SCUD allowed us to build models that build in inductive biases and outperform masking.
Condition on everything? By adding more information, S, to the backwards process, SCUD was able to fit the process much better. Future work may explore adding removing information from S: for forward process with multiple types of mutations, S may count each type of mutation; on the other hand, SCUD trains a de-noiser qo that conditions on S different applications may call to learn de-noiser that takes different information S.
Is there a limit to how much information we can put in S? As more information goes into S, the inference task for žo.0 becomes more challenging. As well, the third term in Eqn. Eqy become large- if S describes exactly every mutation in the forward process for example, then p(21 |S, ar) is a point mass, it never converges. Indeed, one may consider extending SCUD to continuous diffusion by setting S to be, by analogy, the distance 2; has traveled from žo; but in this case, p(æ 1S, 20)

---

has support only on sphere it doesn't converge. In our case, the S we considered was able to overcome these challenges to improve model fit, but future work may need to consider this tradeoff when navigating the modeling space.
Sampling from diffusion models This work focuses on achieving better ELBOs without inves-
tigating the effect on sampling. There is huge work on exploring sampling from discrete diffusion models. These methods approach the problem of trying to get more faithful sample of go [3 (6)
or combining the de-noiser de with other predictors get better samples [8] 13]: another popular
family of methods that falls in this camp is discrete flow matching 4 we explain how to extend
flow matching to SCUD App. E There has however been comparatively little work exploring the
design space of the forward process, potentially because of a mistaken belief that masking could be the optimal forward process [23] (a telling exception is learning masking forward processes in Shi
et al. 23 and Wang et al. (25)). SCUD shows the potential of improving model fit by exploring improved forward processes, potentially complementing work on sampling in future work.
Acknowledgements
We thank Lily Li and Mare Finzi for helpful feedback. This work was supported in part by NSF CAREER IIS-2145492, NSF CDS&E- MSS 2134216, NSF HDR-2118310 BigHat Biosciences, and Capital One.
References
[1] S. Alamdari, N. Thakkar, R. van den Berg, A. X. Lu, N. Fusi, A. P. Amini, and K. K. Yang. Protein generation with evolutionary diffusion: sequence all you need. bioRxiv, Sept. 2023.
[2] Austin, D. Johnson, Ho, D. Tarlow, and Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Adv. Neural Inf. Process. Syst. 1991111997999, 2021.
[3] A. Campbell, Benton, De Bortoli, Rainforth, G. Deligiannidis, and A. Doucet. continuous time framework for discrete denoising models. In Advances in Neural Information Processing Systems, Oct. 2022.
[4] A. Campbell, Yim, R. Barzilay, T. Rainforth, and T. Jaakkola. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. In Proceedings of the 41st International Conference on Machine Learning. arXiv, 2024.
[5] X. Chen, N. Mishra, M. Rohaninejad, and P. Abbeel. PixeISNAIL: An improved autoregressive generative model. In 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 864 872. PMLR, Dec. 2017.
[6] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding, arXiv [cs.CL], Oct. 2018.
[7] I. Gat, T. Remez, N. Shaul, F. Kreuk, R. T. Chen, G. Synnaeve, Y. Adi, and Y. Lipman. Discrete flow matching. arXiv preprint urXiv:2407.15599 2024.
[8] D. Gillespie. Exact stochastic simulation of coupled chemical reactions. J. Phys. Chem., 81
(25):2340-2361, Dec. 1977.
[9] N. Gruver, S. D. Stanton, N. C. Frey, T. G. J. Rudner, I. Hotzel, Lafrance- Vanasse, A. Rajpal, K. Cho, and A. G. Wilson. Protein design with guided discrete diffusion. In Thirty-seventh Conference on Neural Information Processing Systems, Nov. 2023.
[10] S. Henikoff and J. G. Henikoff. Amino acid substitution matrices from protein blocks. Proc. Natl. Acad. Sci. U. S. A., 1092)):10915-109199 Nov. 1992.
[11] D. P. Kingma, T. Salimans, B. Poole, and J. Ho. Variational diffusion models. In 35th Conference on Neural Information Processing Systems, July 2021.
10

---

[12] Z. Lin, H. Akin, R. Rao, B. Hie. Z. Zhu, W. Lu, N. Smetanin, R. Verkuil, O. Kabeli. Y. Shmueli, A. Dos Santos Costa, M. Fazel-Zarandi, T. Sercu, S. Candido, and A. Rives. Evolutionary- scale prediction of atomic-level protein structure with language model. Science, 379(6637): 1123-1130, Mar. 2023.
[13] S. Liu, J. Nam, A. Campbell, H. Stärk, Y. Xu, T. Jaakkola, and R. while you generate: Discrete diffusion with planned denoising. arXiv [cs.LG), Oct. 2024.
Think
[14] A. Lou, C. Meng, and S. Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In 41 si International Conference on Machine Learning, Oct. 2023.
[15] S. Luo, Y. Su, X. Peng, S. Wang, J. Peng, and J. Ma. Antigen-specific antibody design and optimization with diffusion-based generative models for protein structures. bioRxiv, July 2022.
[16] H. Nisonoff, Xiong, S. Allenspach, and Listgarten. Unlocking guidance for discrete state-space diffusion and flow models. arXiv [cs.LG), June 2024.
[17] W. Peebles and S. Xie. Scalable diffusion models with transformers. arXiv [es. Dec. 2022.
[18] F. Z. Peng, Z. Bezemek, S. Patel, J. Rector-Brooks, S. Yao, A. Tong. and P. Chatterjee. Path planning for masked diffusion model sampling. arXiv [cs.LG), Feb. 2025.
[19] E. Perez, F. Strub, H. de Vries, V. Dumoulin, and A. Courville. FiLM: Visual reasoning with a general conditioning layer. arXiv [es. CVJ, Sept. 2017.
[20] S. Sahoo, M. Arrilaa, Y. Schiff, A. Gokaslan, E. Marroquin, J. Chiu, A. Rush, and V. Kuleshov. Simple and effective masked diffusion language models. arXiv preprint arXiv:2406.0rF 07524,, 2024.
[21] Salimans, A. Karpathy, Chen, and D. Kingma. PixeICN+++: Improving PixeICNN with discretized logistic mixture likelihood and other modifications. arXiv [cs.LG], Jan. 2017.
[22] A. Sarkar, Z. Tang, C. Zhao, and P. K. Koo. Designing DNA with tunable regulatory activity using discrete diffusion. bioRxiv, page 2024.05.23.55 23.59660,, May 2024.
[23] J. Shi, K. Han, Z. Wang, A. Doucet, and M. K. Titsias. Simplified and generalized masked diffusion for discrete data, arXiv [cs.LG], June 2024.
[24] X. Wang, Zheng, Ye, D. Xue, S. Huang, and Q. Gu. Diffusion language models are versatile protein learners. ICML, abs/2402. Feb. 2024.
[25] Z. Wang, J. Shi, N. Heess, A. Gretton, and M. K. Titsias. Learning-order autoregressive models with application to molecular graph generation. arXiv [cs. Mar. 2025.
[26] Y. Zhao, J. Shi, L. Mackey, and S. Linderman. Informed correctors for discrete diffusion models. arXiv [cs.LG), July 2024.

---

A Proofs of results
Proposition A.I. (Proof of Prop 3.7 The expression in Eqn. 7 is equal the expression in Eqn 2 for some constant C.
Proof. S is deterministic function of (zi)) write the first term of Equ. I
go qo (11))
Bp((si)eep.n))tt) log Ep((sideeo. (oo) log p((x)) ((()))) x1)
go (1(( (104)
-Ep(teiicpooyla)) log
(7)
pl (0))ee00011 No: 11, v1,
90 (5\x1)
+ (p(S,,1/0)) log
p(S) [r x1
We can combine the second term of this equation with the second term of Eqn. to get Ep(S,a1 log (((()
q(x1)
+ pp(r112)) log
p(5 r1)
p(z1
=Ep(s log go(
p(5(20) (20)) log qa (21
(8)
( ro, S)
90
P(S) ) KL(p(r1 (z)10(z1)S))
-Ep(S|no) log pSS)
+ Ep(S)ao) log
p(S
The first term is (K((((5)) (o and the second does not depend on This completes the proof.
Proposition A.2. (Proof of Prop 4.1) The infinitesimal generator of this process is == T) where I is the identity matrix. In particular, any Markov process with can be simulated in the above way by picking 2 max, - Li.b and setting K L/rr +.r
Proof. The process is described is clearly Markov. By the formal definition of C, for b' # b.
L
lim + 7 p(1x = b' 61|m0
lim / (p(an event occurs before t) x p(e event transitions to 6') + o(t))
(9)
1-0
lim (1 - e r-rl)) Ko,b = KKa +0
Then, since the rows of K sum 1,
Co.5 = Σ LoN = whhh
Σ Kb = -r(1- Kb,b). B
The second statement follows from rearranging the first. The requirement that r 2 max- CCsb comes from the fact that all entries in K must be non-negative and K₁ L1,b/r 1.
O
Proposition A.3. (Proof of Prop 4.2 in the Appendix) Call the event schedule S = {ti, t2 and to O. Call 9t the number of events up to time t, St.. m.
tM3
M
S) p(S)p(z)) II
p( Feem Stm)
(10)
Proof.
(((x))..)) =p()))(((pp)))))))) (p) (11, S) S)
M
=p(S)p(a1) II p(u (.imss:)) S).
m=1 By the Markov property, p( 1 S) p( Teem S). Finally, plat.. I S) x
pla,,,, (Wtt,... S))p((r,,... ()) = pl (m 1e )p(pppp(ee (ftt.... 18tm-1) only depends on S through 8t.,. or equivalently, St... + 11m-1
0
12

---

Proposition A.4. (Proof of Prop. 4.3 Calling the event schedule S = {t1. t2.................. tttt and to = 0.
Ep(zo) log (x() 2 pp((m)) S.z0))
(p( 1atm 20, Stm (lgg Stm) (11)
Fp(S, KT (81, vo (o))(p(((r0))))
This objective is minimized when go (T [m Stm p(2im Flmm Stm
Proof. Just as with the classical ELBO, we can write
Pip(a)) log ((x)) 2 2F Ppteo.s) Slo))))) log go (0((
(12)
p(( S(
Then we can break it up in Prop. A.1 to get
Ep(mo) log ((o) 2 Ep(())))) log qo((2t)((TI,S) -KL((())((((S))
(0,,))
(13)
Ep(S log p(5) (L (0)
(0)
By our definition of the event schedule and ((5), the second and third term on the right are 0. For the fourth term, clearly p(z1 [ S) p(x1 zo, $1).
By our definition of qo
M II m=1
To ((t))
q( / Stm)
As in the proof of Prop. A.3
we can write
M
21.
21,
II
p p( 120, 31, S, x) > II
p( (r0 Sem Ttm)
m=1
m=1
where the last equality follows by the Markov property. Thus the first term is
M
(( Ftmm Stm
M
log
(L(p(z,,..
1000 Stm. It,,, (l 120m. st.. )).
p(Itm- 1o, Stm Tem)
m=1
Proposition A.5. (Proof of Prop B.l p(zz| To, st) factorizes Ih p(pr(1;)) sf) and, when marginalizing over Fn, each dimension of xtm-1 is independent:
p(pr(x,))e,,ss) II (a)
Proof.
20, st) p(pr(1)) (ro, st)((z))p((1))
p(pr (xp (ad of pp(a (p(()))()))) p( p st)
II
p( (20, 8t)
d=1
which equals H p(pr(a))) ag s4). The second claim follows from integrating the later expres- sion.
Proposition A.6. (Proof of Prop. dimension d. Then the first term of Eqn. is
4.4
Define, if sf O. pr(xç) as the state at the last event in
Be
-E- Eppten
Σ
a (a ao
(14)
f 3,ds
13

---

Proof. Call S = ,t!:
31 ( The first term of Eqn. can be written as
D M
- (p(()))
Σ (pr(af
Ma 2o sf
(llgo(p((af Marg, 81g, ))
The term in the sum can be written as L(s 2e, To. d) so we can write
Sp(( ΣE Lise, Lt, v0, d)
20,
Σ Eptsa) Σ
Ep(ro)p(S (. 20, d).
d=LIES"
AES
Call the function after ExEs= equal to C(L, 84) so we can write the loss as Ep(sa) Eieso C(I, si). We now investigate the measure E'p(s Eiesu- First note that Fp(s Eieso is clearly abso- lutely continuous in with respect the Lebesgue measure this expression can be written as (e-Uuna(o,1) Ee f(t, (tt,s))((t, st) almost everywhere,
for some function f. By the Lebesgue differentiation theorem,
f(t', lim E c-0
pp()))
Σ E(tE s.t',, - s))/
(15)
=p(s
s) lim E [# events in [t' - c,tl = /c.
c
The distribution of events on an interval [0, (] is Poisson process with density pus) r3, we can simulate this by drawing 5t Pois( 8,ds) and then distributing the of events with probability according to /u/ ///u (100 ()). Therefore, conditioned on events occurring on O').). the density of events
we
occurring at [t' e, is ((O0, (')), that is, the expectation in Eqn. 15is is
events ma(E) (1( mass
32r
Se¹ B,ds
Subbing this into the previous equation completes the proof.
Proposition A.7. (Proof of Prop. 5.1 Defining ar, exp(- I'd Bads), the objective in Eqn.
is
3:00 Σ To log Ta,0 (xte,
Ee-UUii((0.1) Fy(mm) Pplaslanmm m²)
at
d
Proof. If of > 1 then pr(x:") is corrupted so p(pr(=))] of xg) is uniform categorical and doesn't depend on No; therefore, by our parameterization go, we have that the KI. term the loss Eqn 6 is non-zero if and only if of 1. As well, when of p(p((z)) of 1, 1,2 8n In this case we can write the loss as
Br (a)-Uni((,.1) Jocs 3,ds PS))) Pol
Σ 10tt= 1 log fo (2,, st)a
Finally note that when $0,0 st) predicts Vo. is only useful in telling the model which tokens are corrupted. If we call m, = s, > an indicator of which tokens have been corrupted, then we can parameterize our prediction as To, (2,, m.).
Note p( (ro, S) p(10 (ro mx).
E Ep(z.)
Σ 1( = 1)27 log (0,0 m))d
(16)
Epimy Extce m..to)
En(h 1/m;)) of log T0,0 mi)d
Pois(ss Bods) so p( 1/mm/) = 0 m and
S 3,ds at
p( = 1mm;) = p(
11s 2
14

---

Proposition A.8. (Proof of Prop. 5.2)
Define the score function estimator as in SEDD [S
3(z,, t),,- = g0(2; b) Ez o bl
qo (x²1 2,
Suppressing the dependence of 5e on Lt, t, as 7 + 0 the objective in Eqn. converggs to
bb.b pla = 11
[M(x- b)ag)) p(2fl2g))
-Ei-Uni((..) 1) 2p(20.2)) Σ
Σ
Co,0f
log $6,0
p(z)120)
(17)
where g(z) I(logo x 1).
1).
Proof. Note ~ Pois(r* So B,ds//), as - 0, sty converges to r So 3yds.
As O,
K = (x===//)====007//////+++++++++ - exp (f, BodaC) =Qr-
where Qr the matrix where Qr,, p(xx = =b(=00 b).
Kxf o Kat-lad #6.
ce(pr(z;) $1)
xt K K*tad
Kaq o K K-1Q 0,0 o(>)
It a.T Quā0
(18)
= Kx o KK rd + 0(7)
= =x + Y (Cxy o 3 x Cit) + o(7)-
y
The expression for p(pr(xt)) a4 ag, s4) identical replacing To,o with Tg- Thus
-KL(p(pr(=)))1b ot 2 Igo (a() Ir. st))
pla blag)
80.0
Σ 2 bf p(=
log
p( =x)))
1 2 Coggar - 2a Lz3
(19)
+ (1 O110))))) log
+ +
1 2 (Cog.4 - 24 L(pl2 = b ( p222/ (m0))))
3d Mag = blag)) 80,6
pl ( n135 ))
pt
Σ
Coog
log nd
g
p(ag(1z))
+ o(7)).
Multiplying this by sy, we get o + fo B,ds.
B Details of method
Here we describe how we parameterize, sample, and pick 3 for SCUD.
B.1 Algorithm for estimating ELBO
We calculate p(z,o) from an spectral decomposition of K, or, if K is very large, using power iteration,
RReaal zd (2e, st) is trained to fit p( s5))).
15

---

Algorithm Unbiased estimate of the SCUD ELBO (Eqn. 4) using Prop. 4.4
Input: 20
S p(S)
t ~~ (n, 1)
Sample zt
for d = l,..,, do
22 Categoricall
end for
// Denoise one event of each dimension of 24,
Predict (zt,
for d ., D do
Calculate 4o (pr(x)) d sd) Calculate (prr(a)) si xx))
use Eqn. 20
b use Eqn.
Calculate p(11104 mg) = Categorical (K Kggg))
end for Return:
Bi kL(p(p((a))))f /b s()) + (1
So Byds
B.2 Parameterization
de must predict, for each dimension, p(pr(x)))xz, st), which is an expectation over the posterior of To given x,and S:
Σ Ep(pr(z\))a 200p(0))))) S) Σ (ra" ag) p(a (rt, 8e) 40
p(a 104 2g))
Below show that the fraction on the right hand side is proportional p(ag) 20 s; where x; and st are Te and 8; without dimension d. Other discrete diffusion methods parameterize their qe to predict analogues of this quantity. Austin et al. (2) predicted similar quantity rather than directly predicting p(= S). Moreover, predicting p(zglæ, s, is identical to predicting p(r xr, S) when x.f is masked. Predicting this quantity has the benefit that we do not need to learn what a. tells us about w0: it is baked into our prediction. We parameterize our qo similarly.
Thus, predict (pr((z))) Jar,, S) input o, and se into neural network that outputs vector of probabilities To,0 and set go (orppr (xf) Jaa, st) equal to
Σ pla ( (p(p(()))))()))))))))))) = b)ão.0.b
(20)
which can writen as Krf o - 1o, Note we do not explicitly forbid 2g,0 from using 2f sd to predict ag-
Now show that p(pr(x:)) (m0, St, 22) factorizes across its dimensions.
Proposition B.1. (Proof in Prop A.5 in the Appendix) p(z, To, 81) factorizes as
Th ((pr(x)) jad 1g. sf) and, when marginalizing over No. each dimension of Zem-1 is inde- pendent:
p(pr(x.))xe, st) II p(pr(a;) (t)
Recall this allows us to parameterize qo (pr(x))) st) so it also factorizes I. qo (pr(())))) st).
We parameterize qo (pr(2?)) ( st, st) to predict
plagg [e, st)
plat, ol 2 -d 8 -d
Mu x, (.))(((?))
p(x crg lod s platlag st)p(zg st) p(=glar" x
p (a 87
p(a st st)
16

---

does not depend on zy Thus we aim to predict a quantity proportional to p(>g)=2 " : 88 we call our prediction 00.0 8t), which we plug into Eqn. and then normalize.
Now note ss) = p( (0: st) and p(s() /p(=:
[aaeflsss (sf)
B.3 Sampling
To sample, in principle we could take xi ~ p(Tr)), ) S ~ p(S). and then iteratively reverse each event in S in order using our predictions of (o(p( [2,, For data with many dimensions however, S could contain tens of thousands of events, requiring many evaluations of Fn,0- Instead, like Camphell et al. [3 and Zhao et al. [26]. we reverse many events at once. particular we use an analogue of k- k-Gillespie procedure [6]]: pick k events to reverse, and then reverse them with single evaluation of 00.0-
To sample point æo g(zo) we first sample the noised sample
g(x1) p(a))) and the
((S) = p(S). We now sample given a budget of C evaluations of &0 Every step denoise the last [s1/C] events that have yet be denoised. To denoise we can use Equ. 20 In the case that we denoise & 2 1 events for a dimension at once, we can use the fact that
number of events in each dimension
p(pr" (x 86) Σ (x of =))p(E)
px 55 2g) p(z01=2, st)
pla7108 xg)
We can write
10 (x²)) = pr (x()) K*ag
p(pr* (s s0, x0)
To
dT
K
- pr (x+)
And we can approximate the fraction with To.o just as in Eqn. 20
Thus we define
(p(r (x x1, (1, K xf - 20.0
R.T 2..
(21)
The total procedure summarized in Alg.
B.4 Choosing the rate
Our choice of B, describes how we compress the forward process running from time 0 to fol 3,ds into the interval [0, 1]. controls what times we sample when training the objective Eqn. 6 and So B,ds controls the convergence of p(x1) to p(x.)). Austin et al. 2 suggest picking 3, so that the mutual information between zn and æt decreases linearly to e on the interval [0, 1). For SCUD
models, we pick 3 so that the same is true when conditioning on the schedule: F.,, MT =2) decreases linearly on the interval [0, 1].
Mutual information rate functions To choose the rate function 3r, Austin et al. 2 calculated the frequency of tokens in the training data (b( and then calculated the joint distribution of 2y and particle which has evolved according to L for time r along one dimension
p(ro = b, x, = b') = (e
They calculate the mutual information fuucciin MI(T) of this joint distribution; the mutual information
normalized MI(0) They then pick 3, that evolving in the modulated process linearly
decreases the mutual information from 1 to e on the interval [0, 1]. i.c. MI 3,ds) = 1 - (1 -))t.
For clarity, we'll MI( 3,ds) 1 and look at the interval [o, 1 below.
Implementation in continuous time The process in 2 has discrete time, so the integral over is and each 3r be pre-calculated before training begins. When implement continuous time discrete diffusion, we use Newton root finder to calculate So Adds MI- (1 t) and the implicit
d So 3dds = 1/ (# MI( 9,ds)). function theorem to calculate 3, = di
17

---

Algorithm 2 Efficient sampling from SCUD
Input: function evaluation budget C.
Sample #1,
for d = I...., D do
x² ~ p(Fxo))
~p(ss = Pois(Jo 3,ds)
end for
I + 12 84 s4/C1
D. Number of events to denoise per step
for I,.... C do
// Decide which positions to denoise in this step
5-0 0
for l l,..., do
if 1 Id=1 (sa kd)) > 0 then s-k
there are remaining events to reverse.. ...sample uniformly from remaining events 8.
d Categorical
Σ (od-(o))
kda t kmt
end if end for // Denoise kd steps at each dimension d Predict (o, s)
for d = I,..,, D do
xd 20 (pp (x |2,
use Eqn. 21
5d 5d kd
end for end for Return:
Schedules for SCUD For SCUD, we instead calculate the joint distribution between zo and the particle after m events, Ft.,. along one dimension
p(zo = b, 2t,.. = 6') p j..
Calling the mutual information between these variables MI, choose Bg that E,, MI,, where 8; Pois(r" 1n' 3.ds/7). Again we calculate these values using Newton root finder and the implicit function theorem.
Connection to classical discrete diffusion
With this choice, note 7 - 0, for any
MI (///((((((+(//"/)""""""))"""""" MI(po(0)(e MI(7).
Therefore, E., MI,, + MI(S[ 3,ds), so f 3,ds converges to the same value as in classical discrete diffusion.
Connection to masking discrete diffusion In this case, Itm is uniform independent of xo for all 2 Therefore, MIn 0 for all 2 1 and E,,MI, MI, e S6 A, Therefore, =
C Structured processes
In this section we will describe the structured continuous time Markov processes we used in Sec. g Our processes are inspired by those from Austin al. [2] and Alamdari al. [I; however those works framed the process in discrete time in such a way that they are not related to any continuous time Markov model, requiring us to design new processes. Note also that those works modified their processes to ensure that the transition matrix at every time-point was doubly stochastic; this was so that all transition matrices would have the same stationary distribution a uniform distribution. In are free to pick any L that converges to stationary distribution, even if it is not uniform.
18

---

C.1 Gaussian process for images
To include the bias that two pixel values + j are similar (i - j)² is small, we set Lijj = exp(-200 200 (a=j)² then set C, Eラ C...
B the value 200 was chosen as gave the best results in small scale experiments. We
C.2 Nearest neighbour process for language
Our vocabulary in the language result was approximately 30' 000 tokens from the Bert-base-uncased tokenizer [6]. It is prohibitively expensive to compute a 30' 000 x 30'000 matrix K to take matrix vector products during training. Instead, we pick a sparse built using the embeddings from Devlin et al. [6]; for the most frequent 1000 words (which make up 95% of tokens seen in the data) i. we computed their similarity og Dy where v; is the normalized embedding of word For each word we found the 10 nearest neighhours; noticed restricting to the top 1000 words resulted in nearest enighbours which were much more semantically similar. We next set, for nearest neighbours,
غ. = exp(v exp(v3 v//0.3).
We next normalized E so that the diagonal is this ensures that every word has identical transition rate, avoiding the case where word never transitions because it has no nearby neighbours.
We noticed that it often took a long time for particles to reach stationary distribution with this process, so we added occasional transitions across the nearest neighbour graph; we called p the normalized frequencies of the top 1000 words in the data and define the uniform transition infinitesimal generator
Cuiif = 1 & p
where 1 is the vector of all 1 1's; this transitions tokens to random token based on the final token's frequency in the data. We combine our two processes by defining
= È + 0.4 x Luiit
and normalizing so that the smallest value on the diagonal was -1 We do not store this matrix explicitly, and only perform matrix operations with sparse matrix products and multiplication with 1 or p.
For tokens outside of the most frequent 1000, we transition using Lunif-
C.3 BLOSUM process for protein
BLOSUM is a matrix that can be describes how often different amino acids are seen the same position in related protein families (1O). The i, entry of the matrix is
Psj P.P
B = 2 log
where Py is the probability of two related proteins having amino acids i.j at the same position, and P. is the marginal probability. We build stochastic process to emulate drawing a related protein, so we set
Kr = exp x P, Pyss
There are other letters in our vocabulary for non-canonical amino acids and padding; for i not one of the canonical 20 amino acids, set K2 Pz. so all transitions only occur canonical amino
acid. Finally we set L = I (Fig.
D Experimental details
In all cases trained models 2 A100 GPUs academic cluster.
19

---

1.0
0.4
0.2
ACDEFGH KLMNPORSTWWYBZX
Figure 5: BLOSUM process K.
D.1
Samples from image SCUD
Figure 6: Samples from SCUD Gaussian trained on CIFAR-10.
D.2 Pretrained protein weights
We train 150M parameter ESM2 models 12 with pre-trained weights on Unirel2 and compare to masking diffusion model, DPLM (4), which used a similar strategy.
To get an ELBO for DPLM, we considered it as discrete-time masking diffusion model with 500 timesteps. We used the formula from Austin al. [ to evaluate the ELBO of such model. In particualr, the BLBO is
500
E
X
1 (xx(
mask) log go (x(( X2).
1=1
D.3 Transition rates in Fig. 2
For 7LDR we downloaded the CIFAR-10 model from htmmtttts///tthubbbbb www./andewwcc/ttalll....
We simulated 2000000 forward trajectories using samples from CIFAR-10 and 1000 backward
20

---

Method
Fwd. process Perplexity
DPLM
Masking
10.61
Classical
BLOSUM
10.34
SCUD
Uniform
10.95
SCUD
BLOSUM
10.04
Table Schedule conditioning improves uniform corruptions and accommodates structured forward processes for proteins, even when starting from pre-trained model weights. We perform the protein experiment from the main text but replace the architecture with pre-trained ESM2 backbone.
samples using leaping with 21 65536 steps. For forward samples x we calculated rates -Zz,z and for backward samples we calculated rates -a We then averaged forward and backward rates at each timestep and calculate the difference between the average forward and backward rates.
For D3PM download the 640M uniform model from https://githu/: com mi crosoft/ evodiff We were able to calculate the forward rates analytically. We simulated 1000 back- ward samples of length 110 and at each time step calculated the probability of transition; multiplied this probability by 1/At to get rate. We did not average with sliding window.
D.4
Images
We use an architecture inspired by Kingma et al. 11 like in MD4 23
with a slight modification and then puts it through
incorporate 8t. The architecture first embeds zo like in Shi al. 23 UNet with 32 layers and no up- or down-sampling. At every layer of the UNet, feed forward layer applied a sinusoidal embedding of the time and the output is added to the channels every pixel ax position j, activations atd at updated
(Fa(emb(()) + a.d.
Instead each activation is updated using a FiLM layer using the number of events up to time
Ad 1, (cmb (")) + FF2,0 (cmb(s,)
atd.
The feed forward layers shared across every position j. We used the same training parameters as in Shi et al. [23]: we trained each our large models for days and each of our models from
Fig. I took between 1.5 and 2 hours. Our large models were trained on 2.6 x 10 images, the same
amount as MD4 23 and FLDR and compared to 1.9 10 for D3PM 22
We use K = 2048 function evaluations to generate images. The results of Fig. Jused a batch size of 16 and the same architecture but with an 8 layer UNet masking and classical models used FiLM
layers with / instead of
D.5 Language
We use the diffusion transformer architecture [17 as in SEDD (5). This architecture has FiLM layers to add at each layer, as above, we replace t with 8t. We use the training settings as in SEDD [15), accumulating match their batch size of 512. We trained our models for days each 1.1 x 10 tokens.
D.6 Protein
We use the small CARP architecture from I The original architecture added as embedding of the first layer. We add FiLM layers for s; at every layer as described above. We train and test on the March 2020 release of Uniprot2020 released by Alamdari et al. [[]. We use a batch size of 128 protein up to size 1024 as Alamdari al. M. randomly truncating proteins over that size. We trained each model for 2 days on 1.1 x 10 tokens, compared with 3.3 x 10 for the baseline D3PM models from Alamdari
D.7 Computational complexity
In terms computational complexity, the major differences between SCUD and classical discrete diffusion are (A) replacing operations of L with operations of K, and (B) replacing the time t in the
21

---

argument of To,a with the number of transitions S. We discuss how (B) does not result in a large increase of computational complexity below, and note that (A) does not change the computational complexity except when the number of tokens B is large, when it actually enables strategies that reduce complexity.
(A) Matrix computations To calculate our loss, Eqn. 6 in Eqn. 6 we see that we only need to take matrix vector products with K; the analogous quantity in classical discrete diffusion requires matrix exponentiation exp(t() (Luo et al., 2022). When B is small, both these calculations have negligible complexity and can be calculated similarly quickly by precomputing an degeendecomposition of K or f. But when B is large, as in the language modeling case, these calculations become very expensive; Luo et al., 2022 settled for very simple L. masking and uniform, such that exp(IC) can be easily analytically calculated; SCUD is able to build in richer forward process by picking a sparse + low rank so that matrix vector products are very fast.
In terms of big-O notation, when an eigendecomposition is precomputed, (exp(tC)=)) and (K = each cost O(DB²) for two dense matrix multiplies and scaling by the exponentiation or power of the eigenvalues. When K= is a sparse matrix with (r entries or has rank of r, calculating (K*t ag) d=1 is O(D Br maxa(s")); )); in our language case, B is large while maxa(s{) x 30 and we pick r = 20 resulting in a large speedup.
(B) Computations with S Indeed, the place that SCUD adds some overhead to calculations is in replacing the arguments of 20, ) the time over which To has been corrupted, t, a scalar, is replaced with the number of corruptions of each token S, D-dimensional object. The overhead of this operation is dependent on the architecture of ōn,o- We picked oo,0 so that no parameters were added by replacing with S, and such that the computational and memory overhead caused by this replacement were negligible compared to the operations and memory spent on operations on the D D-dimensional F,- Above we used previous architectures modified so that each operation on f was also applied to each dimension of S. As well, for the architectures we chose, whenever a function of I was added or multiplied
of activations, say at layer f, he,e, the activations had dimension D, so we could perform the same operation with element-wise addition or multiplication with S, i.e.
1. = (t) + f2. was replaced with 1 = (L + f2,0(5))-
Thus, adapting To,o for SCUD in this way adds no extra parameters. The overhead of this change is that every call to fa is replaced by D calls, D-times the activations fo (sf)) must be stored, and D-times more gradients must calculated for (=()). So is however a set of linear functions and activations. The operations on the corrupted data involve convolutions and attention, which have much larger memory and computational costs. In big-O notation, the cost of calculating To e (xx, t) and To, ( S) are therefore identical at worst, the constant in front of the largest term changes. Therefore, in our experiments, we ran all models for roughly equal time with the same batch sizes and did not observe any substantial difference in computation.
E Extending flow matching to SCUD
Here we follow the exposition of Campbell et al. to derive flow matching models that are conditioned schedule. In App. E. we derive schedule conditioned flow matching (SCUM) in generality. In App. E.2 we describe how SCUD is an instance of SCUM and show how by training a SCUD model, one can sample from large class of SCUM models. Finally in App. B we derive an example class of SCUM models. The conclusion is that schedule conditioning can be extended to the flow matching case just as classical discrete diffusion can.
E.1 Schedule conditioned flow matching (SCUM)
We consider discrete objects in a set of size B and in this quick exposition leave out the multi- dimensional case as an easy extension of the logic of SCUD or Campbell et al. 4. In flow matching. we wish to approximately sample from target p(zn) (this called x in Campbell et al. 44). In regular flow matching, we define distributions of samples noised for time l: p( (Fqn. 6 of Campbell et al. 4). To condition on the schedule, we instead define distributions of samples that have been noised by events from Z1: p(x,/zo). We assume p(x, (x1) is close an easy to sample
22

---

from distribution p( when s has large entries. In particular, for with large entries, the marginal p(xx)) = p0 Now we want to denoise events get p(zs-1) and ultimately p(rp) (Eqn. 5 of Camphell et al. [4).
To do so, first choose how denoise elements in p(r, Say Katzo stochastic matrix such that sampling p(z) then 2x-1 Categorical s.d
(K", = gives a sample from p(r,-1 (2)). The next result is the analogous result of Prop. 3.1 Campbell et al. [4]: given sample from the marginal, Is p(z,) can denoise an event in dimension d by averaging over zolo, and using Ks|ro
Proposition E.1. Define Kaga:- = Eptnol (n,) Ka Then sampling 2, ~ p(xx) and 2 Categorical K s gives a sample from p(.
Proof.
Ep(na) Eptaloa) K Kaprooaeeta
Σ p(zu) (Ep(ea)zo) Ko Kalpor
30
Σ p(2o)p(zo-112) 30
=p(xx-1).
Given this result, we can define schedule conditioned flow matching models (SCUM). First we approximate p(zolz,)) with neural network To, (x,, s); next sample from p(z which is p(xx) for some large 8, and then iteratively denoise by approximating Kap... (Alg.
Algorithm 3 Sampling from SCUM in analogy to Alg. 1 in Camphell et al.
large number
Js p(F ) R p(xx)
while > do
K a.w.. < s) K Koite
Fz-1 Categorical (K
8 6 8 8-1
end while Return: d0
To train (o, 8) we can just minimize the cross entropy
Ep-Unif(1, 21.... lare (ttpterr)).(xo).p(c..).. 2o log Jo (2.,
We could alternatively different distribution for s, such as Poisson. Note that p(xx|22) does not depend on the particular choice of K Ka/zo so we can train Zo (Xs, 8) once and then decide the best K for sampling at test time.
E.2 SCUD is SCUM
We now show that for particular choice of K mn the simulated trajectories of SCUM are that of SCUD as Appendix H of Camphell et 44. Next we discuss how, given a trained SCUD model we can sample from a wide variety of SCUM models.
Define a Markov process that noises datapoints zo with an infinitesimal generator L with rate function S. Say we have a data point zo that's been noised > O times and define K Kelngter p(pr(ze (2..) [., Zo, 8) as in Eqn. 5 Then
K = Ep(zole.) (ootzole)Koolace Kolzoizer
= -ppeoole,, p)ppp(tr))),
=p(pr(x,)) ())
23

---

which is exactly the distribution we approximate to denoise an event in SCUD (Alg. 2 fore SCUD just SCUM with a particular choice of K2 Ka with "large number" in Alg. g set to Pois( B,ds).
There-
Furthermore, SCUD trains F0 s) to predict zo given z.. Campbell et al. [4] suggests that an advantage of flow matching is that one can train Tp, once and then decide on the best infinitesimal
generator at test time; we can do the same by training To,e with the SCUD objective and then changing Kajuo at test time.
E.3 Examples of SCUM
Say we have built SCUD model with transition matrix K. The canonical choice for K above is
26 K K4-12--1 26-1 Tig K*
K Kalrootasto- = x = Kx,
We now describe a family of Kajoo that can be alternatively used to sample from p(zo)-
First note that for SCUD, p(zg(zo) = k zu. Therefore Ka can be any matrix with KT Ka." xn K*-1.Tx0 xo and positive entries with rows that add to 1. Campbell et al. [4] r//
suggested picking the process minimally move mass from position with too much in K*-17 mo to those with too little in K xo (R' in Prop. 3.2 in Campbell et al. [); we can do that with the choice
RyLU K K=-1.3 20 y² K x ReLU(x) K x K K= (,))))))))
K 2/20:2..8
Σ. Ke vo 2" KoTTo
for # which moves mass from I, with too much mass to with too little in proportion to how much mass they need.
To augment this "most efficient" choice Campbell et al. 4 describe method to add stochasticity to K slm They do so by introducing an infinitesimal generator that obeys details balance; do the same. Say LDB keeps the distribution p2r,, stationary, say by satisfying detailed balance. Then
we can add more noise to K Kalmosra.. by defining KT = =enc Ki since
KT K*TTa 3o 2//0
KsT xo 11 K xp K 20-
By varying 17, Campbell al. optimized samples for stochasticity against likelihood.
In conclusion, just as one can do with classical discrete diffusion models, after training a SCUD model, optimize stochasticity parameter get desirable samples.
IIn the high dimensional case, unlike our exposition of SCUM, SCUD trains (o, s) to approximate, for each dimension d, p. 8) rather than p(aā) 8) (Sec. B.2. However any prediction of p(>ū can be transformed into a prediction of p(rg Iz., via the identity p(z s),,s) x p(2ī|sá 2d)p(z x., s) which doesn't depend on the specific choice of K Kalsg the difference is just matter of parameterization.
24

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Sec. 6
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Discussion of SCUD tradcoff's and sampling in Sec.
25

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: See App.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See App.
D
Guidelines:
The answer NA means that the paper does not include experiments.
26

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
See code release in Sec. annd& App. D
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ /ttppp.cc/pbllc/guudeeee http://oeeubiiisssoooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
27

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: See App. D and B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Test likelihoods in Sec. aare calculated on large and fixed test sets.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See App.
D and
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
28

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Agrees with codes of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The authors consider improvements to training diffusion models to improve ELBOs. While there potential broader impacts in all generative modeling work, there are none particular to this work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly. harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system lears from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: We use existing datasets and do not scale models far beyond existing architec- tures.
29

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: See App. D
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperssithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
See code release in Sec.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
30

---

Answer:
[NA]
Justification:
No crowdsourcing.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: No IRB.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification:
We do not use LLMs in any core or non-standard way.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non ntanstandard components.
Please refer to LLM policy (https://neuni) http://ps/eeeeooeeeeeree//////// for what should or should not be described.
31


Reviews:
[{"review_id": "YIjGOzvaHT", "reviewer": "Reviewer_5ax3", "summary": "This paper studies why masking diffusion models consistently outperform other discrete diffusion models, and proposes Schedule Conditioned DIffusion SCUD to improve general discrete diffusion performance. The main takeaway is that, the reason why masking diffusion success stems from its inherent property of knowing when to transition. And from the point view of diffusion process, masking diffusion models have their forward process matches their reverse, which other models don't have. \n\nThey then show that by combining SCUD with different discrete diffusion models, they can beat masking diffusion, achieving SOTA results on categorical image and proteins.", "questions": "See Weaknesses. \n\n- If my understanding is correct, your argument that masking diffusion inherently \"knows when to transition\" is equivalent to saying masking diffusion denoisers are time-invariant (since you argue that the reason why mask diffusion is better is because it inherently knows when -- at which t to transition). I think previous papers also studied/mentioned about this time-invariant property of mask diffusion, may be good to add a discussion about how this. E.g. MD4 [1] and MDLM [2] both remove the timestep t in their final design. \n- I see in image and protein cases, you compared with some recent diffusion models, like MD4. However, in the language case, only D3PM with a different tokenizer is compare with and recent SOTA text diffusion models are missing. Can you explain why? I am confused here since I think natural language has a harder structure to learn compare to other cases, I think a fair comparison here is needed to show that the schedule conditioned diffusion can really help with the forward and reverse mismatching issue and thus improving the generation quality. \n- Another thing I noticed is the computational overhead, in the appendix D.6 you claim that in terms of asymptotic complexity, the SCUD and the classic diffusion are the same. I’m worried, though in theory people don’t care about constant difference, in algorithm engineering a big constant sometimes can be a huge problem (e.g. in engineering point of view, O(C*n) sometimes is worse than O(n^1.x)).", "limitations": "See Weaknesses", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Good motivation, Fig 2 gives evidence of forward and backward rate mismatches in the existing diffusion models, which serves as a strong motivation and supports the main arguments \"masking diffusion is superior because its forward matches with backward while others don't\". \n- Clean logical and structural writing and well documentation, I really appreciate the writing, which makes it very easy to follow for me. Also, math proofs, experimental details are well documented. \n- The theoretical contributions connecting mask diffusion and classic discrete diffusion is rigorous and well supported. This work provides a novel viewpoint to understand the discrete diffusion models. \n- SOTA results. \n\nWeaknesses:\n- Computational efficiency. \nIn section 6.3 authors claim that \"within 10% runtime of each other\", this only applies to small-scale experiments, categorical image and protein cases. They didn't mention the performance on the most challenging experiment -- natural language LM1B dataset has no runtime information. \nThough asymtotic complexity appears similar, in practice the constant introduces a lot of overheads. I wonder what's the runtime of LM1B for both SCUD and classic discrete diffusion. \n- GPU vectorization issue. \nTake text case as an example, I'm not sure how you keep GPU vectorization while giving each token its own t. This could make GPU memory access in a very bad way -- random memory accessing and a lot of cache miss. \nTo me this seems like a mismatch between method design and GPU architecture function design. \n- Foundation model concerns. \nI can imagine SCUD could be a technique for diffusion LLMs. Since SCUD highly relies on the schedule, which is a dataset-specific schedule, this type of conditioning could be fundamentally conflict with the requirements of foundation language model for transferability and task generalization.", "was_revised": true, "final_justification": "After reviewing the authors' rebuttal, I maintain my score, since while some concerns were partially addressed, critical issues remain unresolved. \n- Theoretical framework is well-motivated and mathematically sound. \n- Motivation of certain language experiments is clarified. \n- Computational complexity explanations provide technical clarity. \n\nI think the theoretical insights are valuable, but I'm still concerned about the limited evaluation and computational overhead.", "rebuttal": "Thank you for your thoughtful review! We clarify our positive scaling and experimental results, provide numbers for the computational complexity of SCUD, and clarify how SCUD is distinct from previously observed time-invariance. We hope that you will consider raising your score if your concerns are addressed!\n\n> C1 / Q3: Computational efficiency in practice. \n\n> C2: GPU vectorization issue. \n\nSCUD is nearly as computationally efficient as classical discrete diffusion, even at large scale, and our implementation performs very similar operations on the GPU, incurring no more than 10% overhead. The exception is the text example with a structured forward process where classical discrete diffusion was impossibly expensive to train! SCUD actually enables training, with only a minor overhead from the simulation of the forward process: >1000% overhead for classical diffusion to about 20%. And as discussed in the appendix, we trained our language models for 2 days each on $1.1 \\times 10^{10}$ tokens.\n\nThe computational complexity of SCUD and classical discrete diffusion both depend on the neural network architecture and the forward process chosen. In most cases, even in our large scale experiments, both models had a less than 10% difference in their operations – $s_t$ is treated nearly the same way that $t$ is treated in classical diffusion, and $K^{s_t}$ is calculated via an eigendecomposition nearly the same way $e^{-\\log\\alpha(t)\\mathcal L}$ is in classical diffusion. The GPU operations are nearly the same, except a vector broadcast when dealing with the extra dimension of $s_t$ rather than $t$ – there is no poor memory access.\n\nThe exception is the text example where the alphabet size is huge, but computational complexity is an issue for both SCUD and classical diffusion. In text models $K^{s_t}$ is too large to compute via an eigendecomposition; and simulating the forward process by calculating $x_0^TK^{s_t}$ for each position is not naively parallel on a GPU. But the same issue seemingly also applies to the calculation of $x_0^Te^{-\\log\\alpha(t)\\mathcal L}$ via eigendecomposition in classical diffusion. As a result, previous methods restrict to $\\mathcal L$ for which one can perform an analytical eigendecomposition (uniform and masking in SEDD) or save precomputed matrices in discrete time (D3PM). But when one can analytically calculate $x_0^Te^{-\\log\\alpha(t)\\mathcal L}$, they can just as easily calculate $x_0^TK^{s_t}$! Therefore SCUD does not suffer from inherent GPU-incompatibility.\n\nHowever, in Section 6.3 we study a structured process designed in previous work which was found to perform worse than masking in discrete time. To study if this poor performance was a property of the process or of schedule conditioning, we attempted to implement it in SCUD; note we are investigating, not necessarily advocating the use of this process. Now in this case, $x_0^Te^{-\\log\\alpha(t)\\mathcal L}$ was impossible to calculate analytically, so it cannot be implemented in SEDD or MDLM. Similarly we could not compute $x_0^TK^{s_t}$ using eigendecomposition, so we had to perform many iterative matrix multiplications. To make this as GPU-compatible as possible, we flattened all $x_0$, sorted them by decreasing $s_t$, iteratively applied $K$ via in-place batched sparse matrix multiplication to slices like $x_0[:M]$, and unsorted them. This indeed involved $\\max(s_t)$ sequential operations and was not as ideal on the GPU as parallel eigendecomposition. It added a roughly 20% overhead but could likely greatly be improved with caching and parallel-scan algorithms.\n\n> Q2: In the language case, only D3PM with a different tokenizer is compare with and recent SOTA text diffusion models are missing. Can you explain why? \n\nWe cannot perform the same comparison because SCUD actually enabled training the first structured continuous time language model – it was impossible to implement a model trained using the structured forward process with classical discrete diffusion as described above. Nevertheless, we used exactly the architecture from SEDD to perform our investigation in Sec 6.2.\n\nIn brief, to compliment our theoretical contributions, our goal was to demonstrate our theoretical results in practice, and leverage the theory to reinvestigate structured processes in the literature. In the case of D3PM’s Gaussian and EvoDiff’s BLOSUM, SCUD leads to state-of-the art results, but for D3PM’s sparse graph process SCUD leads to less of an improvement. We will make our goal clearer in future drafts.\n\nIn particular, our goal in the experiments section was to \n\n1. Show that schedule conditioning improves performance. This was validated by comparison to previous sota models in theory, images, and proteins. Technical considerations meant we were limited to uniform $\\mathcal L$, where the superiority of masking was already established.\n\n2. Redo previous experiments that showed that structured processes performed worse than masking. Our theory suggested these were unfair – masking had the advantage of schedule conditioning while structured processes did not. We redid these fairly by building SCUD versions of structured models. But we are not necessarily advocating for using these specific structured processes. And we of course do not expect every structured forward process will lead to an improvement in performance.\n\nFor images and proteins, we saw that Gaussian and BLOSUM processes improve the fit to the data, achieving state-of-the-art fit. For language models, we saw a very small improvement from the graph structure suggested in the D3PM paper, potentially suggesting there is room to see the sorts of improvements from BLOSUM and Gaussian processes but in language by building an improved model.\n\nFurthermore, our models trained on LM1B were trained using the SEDD architecture, so we expect SCUD uniform to be nearly identical to SEDD masking. Therefore one could consider our results a comparison of SEDD / SCUD masking and SCUD structured in both runtime and performance. Finally, models in MDLM and SEDD were trained longer on LM1B than our models, so their performances could not be directly compared. We reasoned that training longer to compare with those models would not impact our claims (1) or (2) since the structured process performs so similarly to SEDD, so we did not do this comparison.\n\nInstead we took the language example as an opportunity to demonstrate the scaling of SCUD to huge $K$. This opens the door to more creative forward processes for language making use of other structured $K$.\n\n> C3: Foundation model concerns.\n\nSCUD’s reliance on $s_t$ does not make it any more susceptible to this issue than classical discrete diffusion. As well, we believe this issue can be solved simply by picking a good parameterization for your model.\n\nNote classical discrete diffusion suffers from the same issue – a radical change in the schedule would strongly modify its dependence on $t$. The expectation in both SCUD and classical discrete diffusion is masking, because it is time-invariant. But actually any discrete diffusion model may be parameterized to be time and schedule invariant (described below), and can therefore be transferred between different schedules, just like masking; with the parameterization below, any SCUD or discrete diffusion model can work on any schedule. Therefore we do not expect SCUD to be any more vulnerable to domain shift than classical discrete diffusion models.\n\n> Q1: Your argument that masking diffusion inherently \"knows when to transition\" is equivalent to saying masking diffusion denoisers are time-invariant.\n\nThe connection to the time-invariance of masking models is an interesting connection, however, we make a separate and novel observation with SCUD – indeed $t$ is replaced by $s_t$ in SCUD, it does not go away.\n\nAlthough not discussed in other literature, we believe the time-invariance of masking diffusion is simply a matter of parameterization, and in fact any discrete diffusion model can easily be made time invariant. We save the details for a future in-depth discussion, but, briefly, this is because one can show that, for the target distribution $p(x_0^d\\mid x_t)$, the set of vectors $((p(x\\_t^{d}\\mid t, x\\_0^d=b))\\_b)\\_d$ are sufficient statistics. The neural network can therefore be parameterized to take just these vectors as the argument, making it depend on no other variables, in particular making it invariant to time and even the particular forward process. When we consider masking diffusion, the vector $(p(x_t^{d}\\mid t, x_0^d=b))_{b}$ is a one-hot encoding when $x_t$ isn’t a masking token and is uniform otherwise, recovering the canonical time-invariant parameterization. The same logic can also be used to parameterize SCUD models to be independent of $s_t$ by replacing $t$ with $s_t$."}, {"review_id": "7RABox8ugY", "reviewer": "Reviewer_hLSS", "summary": "This paper aims to answer why masked discrete diffusion works so much better than prior discrete diffusion approaches.  It finds that the main cause of this is that the forward and reverse processes induced by masked discrete diffusion have the property that the masked diffusion reverse process inherently matches the forward process in terms of *when* a state should transition.  This is in contrast to other discrete diffusion processes where the learned reverse process may induce some error due to a mismatch in the transition times (as represented in the paper by a novel decomposition of the ELBO).  The authors show that by explicitly conditioning other classical discrete diffusion processes on an event schedule they are able to match or exceed the performance of masked diffusion models on a variety of tasks.", "questions": "1. How does your protein performance compare to current state of the art models DPLM/DPLM2?\n\n2. How well does SCUD perform on unconditional protein generation?\n\n3. How does SCUD compare to MDLM on LM1B (or MDLM/MD4 on OpenWebText)?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "I very much enjoyed reading this paper -- I found it to be well written and clear and that its theoretical contributions were both novel and significant.  However, I have some concerns regarding the empirical evaluation of SCUD.  Specifically, I found that the validation for both proteins and text to be rather lacking (though the image results were quite good).  \n\nFor proteins the paper does not compare to the current state-of-the-art masked diffusion model for proteins in DPLM/DPLM2 [1, 2] which degrades the experimental quality.  It would also be nice to see how SCUD performs on unconditional generation as this is one of the main metrics one looks for to assess protein language models (see [1,2,3] for examples of these metrics).\n\nWith regards to text, the test perplexities on LM1B are significantly higher than those reported in MDLM [4].  I realize that MDLM was trained on more tokens than the SCUD models in the paper, but the paper's main claims would be significantly more compelling if it is shown that SCUD improves non-masked diffusion performance to comparable levels of performance on text as in MDLM or MD4 [5].\n\nOverall, I am currently leaning slightly towards rejection on this paper.  However, this is only because I feel the empirical validation is still a bit lacking in the current manuscript.  I believe the theoretical contributions in the paper are significant and I would be more than happy to increase my score if the authors can provide additional experimental results along the lines of the concerns I listed above.  Finally, I note that I am most curious about the text experiments and would weight these slightly higher than proteins.\n\nReferences:\n\n[1] Wang, Xinyou, et al. \"Diffusion language models are versatile protein learners.\" arXiv (2024).\n\n[2] Wang, Xinyou, et al. \"Dplm-2: A multimodal diffusion protein language model.\" arXiv (2024).\n\n[3] Hayes, Thomas, et al. \"Simulating 500 million years of evolution with a language model.\" Science (2025).\n\n[4] Sahoo, Subham, et al. \"Simple and effective masked diffusion language models.\" Advances in Neural Information Processing Systems 37 (2024).\n\n[5] Shi, Jiaxin, et al. \"Simplified and generalized masked diffusion for discrete data.\" Advances in neural information processing systems 37 (2024).", "was_revised": true, "final_justification": "I found the paper to be well written and strongly motivated.  While I would have preferred to recommend a full acceptance, I chose borderline accept as I believe the experimental results were not as strong as I would have liked -- the image results were good, but the protein results while improving on baseline (which was understandably low due to experimental setup) only achieved a relatively low 47 pLDDT, and it would be nice to show more convincing proof of the results on text.  However, the theoretical contribution and manuscript quality is I believe significant enough to recommend a 4.", "rebuttal": "Thank you for your thoughtful review! We are glad you appreciated the theoretical content and address your concerns about the experiments below with a more in depth experiment on using pre-trained weights and DPLM. With this result, we achieve state-of-the-art on both images and proteins.\n\nFirst let us recap the goals of our experiments to describe how SCUD compares to MDLM and SEDD and why we chose to do further experiments on DPLM.\n\nIn brief, to compliment our theoretical contributions, our goal was to demonstrate our theoretical results in practice, and leverage the theory to reinvestigate structured processes in the literature. In the case of D3PM’s Gaussian and EvoDiff’s BLOSUM, SCUD leads to state-of-the art results, but for D3PM’s sparse graph process SCUD leads to less of an improvement. We will make our goal clearer in future drafts.\n\nIn particular, our goal in the experiments section was to \n\n1. Show that schedule conditioning improves performance. This was validated by comparison to previous sota models in theory, images, and proteins. Technical considerations meant we were limited to uniform $\\mathcal L$, where the superiority of masking was already established.\n\n2. Redo previous experiments that showed that structured processes performed worse than masking. Our theory suggested these were unfair – masking had the advantage of schedule conditioning while structured processes did not. We redid these fairly by building SCUD versions of structured models. But we are not necessarily advocating for using these specific structured processes. And we of course do not expect every structured forward process will lead to an improvement in performance.\n\nFor images and proteins, we saw that Gaussian and BLOSUM processes improve the fit to the data, achieving state-of-the-art fit. For language models, we saw a very small improvement from the graph structure suggested in the D3PM paper, potentially suggesting there is room to see the sorts of improvements from BLOSUM and Gaussian processes but in language by building an improved model.\n\nFurthermore, our models trained on LM1B were trained using the SEDD architecture, so we expect SCUD uniform to be nearly identical to SEDD masking. Therefore one could consider our results a comparison of SEDD / SCUD masking and SCUD structured in both runtime and performance. Finally, as you mentioned, models in MDLM and SEDD were trained longer on LM1B than our models, so their performances could not be directly compared. We reasoned that training longer to compare with those models would not impact our claims (1) or (2) since the structured process performs so similarly to SEDD, so we did not do this comparison.\n\n**New state-of-the-art protein results on DPLM**\n\nSince 1) the EvoDiff’s BLOSUM process lead to a greater boost in performance than DPLM’s language graph process; and 2) DPLM starts with pretrained weights from ESM3 making it an interesting test case to show that SCUD can also leverage pre-trained weights; we elected to do further experiments in the protein setting. Note however that DPLM was trained with a reweighted ELBO as an objective, and developed a specialized sampling procedure that does not sample from the learned distribution; therefore, its performance as an unconditional generative model, which is what we are interested in this work, may not reflect the performance on design tasks the authors demonstrated in their paper. Below we report DPLM perplexities and sample quality according to how DPLM was trained – as a discrete-time masking diffusion model with $\\Delta t=1/500$.\n\nIn new experiments, we started with DPLM medium architectures, added FiLM layers to the pretrained ESM models and trained with BLOSUM classical, uniform SCUD / masking and BLOSUM SCUD. During the rebuttal period we managed to train on 4.3 million tokens; unfortunately this is less than 20% of the training budget from DPLM. Nevertheless, we show that SCUD BLOSUM indeed achieves the best likelihoods. For sample sample quality we measured by mean predicted foldability (pLDDT) of 100 sequences of length 200) and used 500 model evaluations. We see that SCUD is the best among our model implementations and is competitive with DPLM. Note DPLM was trained for much longer, which is known to increase the confidence of model predictions – indeed the sample quality of our models are still improving. We will update these numbers for models trained for longer in the final draft.\n|Model | perplexities | mean pLDDT from OmegaFold |\n|:--|:--:|:--:|\n| DPLM |10.61|**45.2**|\n| BLOSUM classical |10.82|41.8|\n| uniform SCUD / masking |11.25|40.4| \n| BLOSUM SCUD |**10.52**|**45.2**|"}, {"review_id": "I7A0pFPl4k", "reviewer": "Reviewer_zwCo", "summary": "This paper introduces Scheduled Conditioned Diffusion (SCUD), a generative modeling framework that first samples a transition schedule and then generates data conditioned on this sampled schedule. The approach aims to generalize discrete diffusion by decoupling the temporal structure from the transition dynamics. While the core idea of conditioning on a sampled schedule is novel and has potential, the current version of the paper lacks clarity in its mathematical formulation, contains questionable assumptions (particularly regarding the transition process and its independence from the schedule), and requires stronger theoretical and empirical justification. As it stands, the work is promising but not yet ready for publication.", "questions": "Major Questions:\n- See weaknesses\n\nMinor Question\n- Why the shorthand for Scheduled conditioned diffusion written as \"SCUD\"?", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths \n\nThe idea of conditioning the generative process on a sampled schedule of transitions is novel and conceptually intriguing. \nIt offers a new perspective on incorporating temporal flexibility into discrete diffusion frameworks. \n\n\nWeaknesses \n1. Conceptual and Mathematical Concerns Regarding the Transition Matrix $K$ \nThe definition and usage of matrix $K$ (as described in lines 129-136 and Proposition 4.1) raise several conceptual issues: \n- Once $\\Delta t$ is sampled, the first transition occurs at time $t+\\Delta t$, so it is unclear why $x_t$ and $x_{t+\\Delta t}$ could be the same. \n- The proof of Proposition 4.1 appears to implicitly assume that $\\Delta t$ is infinitesimally small, but this assumption is neither stated nor justified. \n- When sampling $x_{t+\\Delta t}$ from a categorical distribution parameterized by $K_{x_t}$, it is unclear why this process would be independent of the sampled $\\Delta t$, especially when $\\Delta t$ governs the time horizon of transitions. \n\n2. Incorrect Claim About Equivalence with Uniform Transition \n- The statement in lines 205–209 is misleading. Even under uniform transition schedules, transition and masking processes are not equivalent, even if the transition schedule is known. This is because under uniform transitions, the same token position can undergo multiple transitions, which breaks the claimed equivalence with masking. \n\n3. Notation and Mathematical Clarity \nSeveral mathematical definitions and notations are unclear or inconsistent: \n- In Figure 3, the role of $\\tilde{x}_{0, \\theta}$  is ambiguous. If this represents a model prediction, is it a denoised sample or a score? In discrete diffusion, the model typically estimates log-ratio scores [1]. \n- In Equation (5), it is not clear whether the schedule variable $s_t$ is discrete or continuous. Since it's drawn from an exponential distribution, one would expect it to be continuous. \n- In Proposition 4.4, it appears the authors directly model $q_{\\theta}$ . However, prior work [1] emphasizes estimating the log-ratio between adjacent states, which is not discussed here. \n- The notation $\\tilde{x}$ is used without definition in Proposition 5.1. \n- Proposition 5.2 lacks context and should clearly describe the connection to SEDD loss. The statement about equivalence needs to be explained more rigorously. \n\n4. Experimental Evaluation \n- The empirical validation could be strengthened by including comparisons against more competitive or recent baselines such as SEDD or LLADA, which are more aligned with the proposed approach in terms of model class and task setup. \n\n[1] Lou, Aaron, Chenlin Meng, and Stefano Ermon. \"Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution.\" ICML 2024.", "was_revised": true, "final_justification": "Thank you for the detailed response. The clarification on transitions vs. events resolved several of my concerns. I now better appreciate the theoretical framework and am raising my score accordingly.", "rebuttal": "Thank you for your thoughtful and effortful review! We clarify our theoretical and experimental results below. However, we note your appraisal about clarity differs from the other reviewers and your questions concern the foundations of continuous-time Markov chains. We recognize that readers would benefit from some familiarity with continuous-time Markov chains. We'll add additional mathematical background to the appendix and note Chapter 2 of Liggett 2010 and the τLDR paper provide excellent primers in this established area.\n\n> Conceptual and Mathematical Concerns Regarding the Transition Matrix $K$ The definition and usage of matrix $K$ (as described in lines 129-136 and Proposition 4.1) raise several conceptual issues:\n\n> Once $\\Delta t$ is sampled, the first transition occurs at time $t+\\Delta t$, so it is unclear why $x_t$ and $x_{t+\\Delta t}$ could be the same. \n\nOur definition and usage of the matrix $K$, including self-transitions, is standard (ex Chapter 2 of Liggett 2010). In our first discussion of the Gillespie algorithm we defined $\\Delta t$ as the transition times. However, Prop 4.1 shows that we can simulate from a Markov process defined by $\\mathcal L$ even if there isn’t a transition at each $\\Delta t$. This is important for applying SCUD to processes which have $\\mathcal L$ with non-constant diagonals. The beginning of Sec 5 describes how decreasing the probability of $x_t=x_{t+\\Delta t}$ leads to more schedule conditioning.\n\n> The proof of Proposition 4.1 appears to implicitly assume that $\\Delta t$ is infinitesimally small, but this assumption is neither stated nor justified. \n\nProp 4.1 simply uses the formal definition of the infinitesimal generator $\\mathcal L$ (Equation 2.4 of Liggett 2010) – the limit $t\\to 0$ in not an assumption about the time between transitions. Note we are working in continuous time, so there is no minimum $\\Delta t$ and the markov process can be uniquely described by a matrix $\\mathcal L$ which represents the infinitesimal flow at a particular time. The formal connection between $\\mathcal L$ and the Markov process is described in depth in chapter 2 of Liggett 2010.\n\n> When sampling $x_{t+\\Delta t}$ from a categorical distribution parameterized by $K_{x_t’}$, it is unclear why this process would be independent of the sampled $\\Delta t$, especially when $\\Delta t$ governs the time horizon of transitions.\n\nThis is a property common to all Markov processes. The “event” times are drawn from exponential distributions which are “memory-less”, and then $K$ describes the behaviour at each event. The sampling algorithm we describe is known as the Gillespie algorithm.\n\n\n> Incorrect Claim About Equivalence with Uniform Transition: The statement in lines 205–209 is misleading. Even under uniform transition schedules, transition and masking processes are not equivalent, even if the transition schedule is known. This is because under uniform transitions, the same token position can undergo multiple transitions, which breaks the claimed equivalence with masking.\n\nLines 205-209 give an informal argument and the formal argument is contained in Proposition 5.1. \n\nIntuitively, the reason the “multiple transitions” doesn’t represent a meaningful difference between the two classes of models is that a position that has transitioned uniformly just once has lost all of its information – therefore when trying to predict $x_0$, knowing that a token in $x_t$ has been transitioned once or many times presents the same information.\n\n> Notation and Mathematical Clarity Several mathematical definitions and notations are unclear or inconsistent:\n\n> In Figure 3, the role of  is ambiguous. If this represents a model prediction, is it a denoised sample or a score? In discrete diffusion, the model typically estimates log-ratio scores [1]. \n\n> In Proposition 4.4, it appears the authors directly model $x_0$. However, prior work [1] emphasizes estimating the log-ratio between adjacent states, which is not discussed here. \n\nFigure 3 represents predicting the “denoised” letter $x_0$. We will make this more clear in Section 4.2. The parameterization we use is the most common (used by continuous diffusion models as well as D3PM, EvoDiff, and masking diffusion models). However we are familiar with the SEDD (your citation [1]) parameterization and, in fact, one of our central results, Proposition 5.2, actually describes the connection to our parameterization; it shows that the two parameterizations are equivalent.\n\n> In Equation (5), it is not clear whether the schedule variable $s$ is discrete or continuous. Since it's drawn from an exponential distribution, one would expect it to be continuous. \n\nLine 155 defines $s_t$ as the number of events that have occurred so far, not the event times $S$. It is distributed as a Poisson as described in Algorithm 2 for example. We will make this more clear.\n\n> The notation $\\tilde x$ is used without definition in Proposition 5.1. \n\n$\\tilde x_{0, \\theta}$ is the prediction of the denoised sequence. We will restate the definition in Sec 5 for clarity.\n\n> Proposition 5.2 lacks context and should clearly describe the connection to SEDD loss. The statement about equivalence needs to be explained more rigorously.\n\nProposition 5.2 is formally stated and proved in the appendix; there are links in the pdf between the statement in the main text and the proof. However the SEDD loss is cumbersome to write, so we simply moved its statement to the appendix.\n\n> The empirical validation could be strengthened by including comparisons against more competitive or recent baselines such as SEDD or LLADA, which are more aligned with the proposed approach in terms of model class and task setup.\n\nOur experiments aimed to confirm our theoretical results in practice and leverage them to interrogate some commonly held beliefs around discrete diffusion models. Nevertheless, while not necessary for proving our main claims, our implementation of our SCUD uniform uses the SEDD architecture and can therefore be interpreted as a comparison to SEDD.\n\nIn brief, to compliment our theoretical contributions, our goal was to demonstrate our theoretical results in practice, and leverage the theory to reinvestigate structured processes in the literature. In the case of D3PM’s Gaussian and EvoDiff’s BLOSUM, SCUD leads to state-of-the art results, but for D3PM’s sparse graph process SCUD leads to less of an improvement. We will make our goal clearer in future drafts.\n\nIn particular, our goal in the experiments section was to \n\n1. Show that schedule conditioning improves performance. This was validated by comparison to previous sota models in theory, images, and proteins. Technical considerations meant we were limited to uniform $\\mathcal L$, where the superiority of masking was already established.\n\n2. Redo previous experiments that showed that structured processes performed worse than masking. Our theory suggested these were unfair – masking had the advantage of schedule conditioning while structured processes did not. We redid these fairly by building SCUD versions of structured models. But we are not necessarily advocating for using these specific structured processes. And we of course do not expect every structured forward process will lead to an improvement in performance.\n\nFor images and proteins, we saw that Gaussian and BLOSUM processes improve the fit to the data, achieving state-of-the-art fit. For language models, we saw a very small improvement from the graph structure suggested in the D3PM paper, potentially suggesting there is room to see the sorts of improvements from BLOSUM and Gaussian processes but in language by building an improved model.\n\nFurthermore, our models trained on LM1B were trained using the SEDD architecture, so we expect SCUD uniform to be nearly identical to SEDD masking. Therefore one could consider our results a comparison of SEDD / SCUD masking and SCUD structured in both runtime and performance. Finally, models in MDLM and SEDD were trained longer on LM1B than our models, so their performances could not be directly compared. We reasoned that training longer to compare with those models would not impact our claims (1) or (2) since the structured process performs so similarly to SEDD, so we did not do this comparison. Instead we took the language example as an opportunity to demonstrate the scaling of SCUD to huge $K$. This opens the door to more creative forward processes for language making use of other structured $K$.\n\nFinally, we compare to more recent discrete diffusion models in the protein case in our response to Reviewer hLSS (search for \"New state-of-the-art protein results on DPLM\")."}]

Author rebuttal:

Meta-review: This paper investigates why masked discrete diffusion outperforms prior discrete diffusion approaches. The key finding is that the forward and reverse processes in masked discrete diffusion are inherently aligned: the reverse process naturally mirrors the forward process in terms of transition timing. By contrast, other discrete diffusion methods suffer from mismatches in transition times, which introduce errors in the learned reverse process.  Furthermore, they demonstrate that explicitly conditioning classical discrete diffusion models on an event schedule closes this gap, enabling them to match or even surpass masked diffusion models across a range of tasks.

I think this paper introduces new insights and ideas. Hence, I vote for acceptance. However, as one reviewer pointed out, the writing style could be significantly improved for the wider audience. I recommend that the author take them into account in the final version.
Decision: Accept (poster)