Simple Guidance Mechanisms for Discrete Diffusion Models
Authors: Yair Schiff
Subham Sekhar Sahoo
Hao Phung
Guanghan Wang
Sam Boshar
Hugo Dalla-torre
Bernardo P de Almeida
Alexander M Rush
Thomas PIERROT
Volodymyr Kuleshov
Venue: ICLR 2025 Poster (2025)
Keywords: Discrete Diffusion
Guidance
Primary area: generative models
TL;DR: Provide state-of-the-art results for uniform noise discrete diffusion and a simple derivation of classifier-based and classifier-free guidance for discrete diffusion.

Abstract: Diffusion models for continuous data gained widespread adoption owing to their high quality generation and control mechanisms. However, controllable diffusion on discrete data faces challenges given that continuous guidance methods do not directly apply to discrete diffusion. Here, we provide a straightforward derivation of classifier-free and classifier-based guidance for discrete diffusion, as well as a new class of diffusion models that leverage uniform noise and that are more guidable because they can continuously edit their outputs. We improve the quality of these models with a novel continuous-time variational lower bound that yields state-of-the-art performance, especially in settings involving guidance or fast generation. Empirically, we demonstrate that our guidance mechanisms combined with uniform noise diffusion improve controllable generation relative to autoregressive and diffusion baselines on several discrete data domains, including genomic sequences, small molecule design, and discretized image generation.

Published as a conference paper at ICLR 2025
SIMPLE GUIDANCE MECHANISMS FOR DISCRETE DIFFUSION MODELS
Yair Schiff* Subham Sekhar Sahoo', Hao Phung', Guanghan Wangi,
Alexander Rush, & Volodymyr Kuleshov Department of Computer Science, Cornell University {yzs2, sss284, gw354, ht h2p amr459, v @cornel 1. eddu
Hugo Palla-Torre, Sam Boshar, Bernardo P. de Almeida, & Thomas Pierrot InstaDeep
(h. da 1 la aatorre, boshar, b. deaaleeiie da, h.s.pperrttttttssaddeep.ooo
ABSTRACT
Diffusion models for continuous data gained widespread adoption owing to their high quality generation and control mechanisms. However, controllable diffusion on discrete data faces challenges given that continuous guidance methods do not directly apply to discrete diffusion. Here, we provide a straightforward deriva- tion of classifier-free and classifier-basee guidance for discrete diffusion, as well as a new class of diffusion models that leverage uniform noise and that are more guidable because they can continuously edit their outputs. We improve the quality of these models with a novel continuous-time variational lower bound that yields state-of-the-ard performance, especially in settings involving guidance fast gen- eration. Empirically, we demonstrate that our guidance mechanisms combined with uniform noise diffusion improve controllable generation relative autore- gressive and diffusion baselines on several discrete data domains, including ge- nomic sequences, small molecule design, and discretized image generation. Code to reproduce our experiments is available here.
INTRODUCTION
Diffusion models SSohl-Dickstein et al., 2015; Ho et al., 2020) gained widespread adoption in image generation and signal processing in part due to their high controllability via mechanisms such as haase (Dhariwal & Nichol, 2021 and elassifier-free guidance (Nichol et al., 2021; Ho & Salimans, 2022). Tasks where guidance plays a key role include MRI denoising (Song Ermon, 2019), 3D reconstruction (Poole et al., 2022; Gao al., 2024), and conditional generation (Saharia et al., 2022; Gokaslan et al., 2024).
However, applying controllable diffusion-based generation tasks where the data is discrete (e.g.. molecule design or text generation) presents challenges. First, standard diffusion models and their guidance mechanisms are not directly applicable, since they require taking gradients with respect to the data, and these are not defined in discrete settings. Second, popular discrete extensions of diffusion (Sahoo et al., 2024a; Shi et al., 2024) cannot perform multiple editing passes on generated tokens, hence are not ideal for controllable generation. Third, the performance of discrete diffusion models (measured by perplexity) lags behind autoregressive (AR) models, especially for classes of diffusion that are amenable control, such uniform noise (Austin et al., 2021; Lou et al., 2023).
Here, we propose discrete diffusion models and guidance mechanisms that are effective at con- trollable generation and that address the above challenges. First, we provide straightforward and -ompllt-mmpl adaptations of classifier- based and classifier-free guidance for discrete diffusion models. Second, revisit uniform noise diffusion language models (UDLM), which undo random token perturbations and are particularly amenable to guidance, since they can repeatedly edit their samples (Austin al., 2021) and thus correct error.. We address performance issues that plagued
"Corresoondigg author. Work done while InstaDeep. 'Fqual contribution.

---

Published as a conference paper at ICLR 2025
Unconditional
Guided
Po 40
Po
22 (c)
Re.
v)
Guidance property satiefaccionn
Autoregrossive
Guidence strength
14
(4)
Improved P
Discrete diffusion model ps
guidance
Validity Novelty
(Left) Adapting guidance to discrete diffusion. Models output a factorized discrete dis- tribution for each denoised token. With our guidance mechanisms, we adjust these probabilities according to a guidance model either a conditional diffusion model in classifier-free guidance (Section 3.1) or separately trained classifier for classifier-based guidance (Section 3.2). (Right) Relative to autoregressive models, which make local predictions one token at a time, discrete diffu- sion models denoise the entire sequence at every iteration, allowing for more guidable outputs.
Figure I:
previous iterations of uniform noise discrete diffusion by deriving continuous-time version of the evidence lower bound, which tightens the variational gap (Kingma et al., 2021; Sahoo et al., 2024a).
We demonstrate the effectiveness of guidance with discrete difffusion models several domains: genomics, molecule strings, and discretized images. We find that discrete diffusion models are more controllable than AR when paired with classifier-free guidance. Moreover, proposed classifier- based method improves upon previous guidance mechanisms for diffusion (Gruver et al., 2024) and AR (Dathathri et al., 2019; Yang Klein, 2021). Our language modeling experiments also reveal that contrary to a widely-held belief (Austin et al., 2021; Lou et al., 2023), uniform noise diffusion attain state-of-the-ar performance small vocabulary datasets (e.g.. molecules, DNA) and that UDLM attains new state-of- ti-thhe-rt in perplexity among uniform noise diffusion models.
In summary, our contributions are as follows:
We provide simple and effective discrete classifier- and classifier-free guidance.
We introduce UDLM, class of discrete diffusion models particularly amenable to guid- ance, for which we derive tightened ELBO that significantly improves their performance.
Across three domains, we demonstrate that discrete guidance yields better controllable generation compared to strong AR baselines and previous diffusion guidance methods.
2 BACKGROUND
Notation Let V be the space of all one-hot tokens over some vocabulary consisting of N unique characters: V {z E 10 Σ, z 1) C A where A represents the simplex over N categories. Let be N-dimensional column vector of all ones, and denote the Hadamard product between two vectors as @. We define z (1:L) as sequence of L tokens. where (() E V, for all tokens l {L...., L. and use V denote the of all such sequences. Finally, let Cat(::p) denote the categorical distribution with probability vector p E A
2.1 DISCRETE DIFFUSION MODELS
Diffusion models are a class of generative models defined by a denoising network po that trained to remove noise from latent variables z,, These latents are generated by fixed corruption process that, starting from clean data x drawn from the data distribution g(x), increasingly adds more noise 2,, f increases (Soht-Dickstein al., 2015; Song & Ermon, 2019; Ho al., 2020).
In discrete denoising diffusion probabilistic models (D3PM; Austin et al. (2021)), the noising pro- cess is defined in terms of transition matrix Qtjs whose (th entry is the probability of transi- tioning to the i-th state at time given the j-th state at time s. This induces a Markov corruption process where have ((,, 2.)) Cat (2:; Qu
2

---

Published as a conference paper at ICLR 2025
Sahoo al. (2024a) build off this framework to introduce specialized algorithms that are both sim- pler and more effective than the general D3PM framework. They focus on a specific class of forward processes from D3PM that can be defined interpolations between clean data and noisy prior n, and we adopt their notation below:
(z) x) = Cat axx + - a.)
(1)
where a; = (tt) is a noise schedule monotonically decreasing in t. Defining anls = at this class of processes admit the following posteriors
[ae| + (1 atls (1) 2r] 3 [a,x + (1 a'a
g(2, (175)22,30 x) Cal Zs;
a, x + (1 0z )
Of note, for absorbing-state diffusion, where π |MASK] one-hot vector at the special [MASK] token index, Sahoo et al. (2024a) show that if z, & [MASK] then g(z, /2,,/ Cat(z,; zr), which reflects the fact that unasked tokens at time must remain unmasked for all time s < t.
2.2 DIFFUSION GUIDANCE
For continuous data, diffusion models have demonstrated state-of of off-the-art controllable generation by means of blassifier-basee guidance (Nichol et al., 2021; Ho Salimans, 2022: Saharia et al., 2022). These approaches rely on different ways of expressing the score of a distribution conditioned on
et al., 2015; Dhariwal & Nichol, 2021a) la) and classifier free
Classifier-based Classifier-based generation employs a diffusion model to iteratively sample from tempered distribution (z y/ ze) x p(y Zg To Zz), where represents inverse
temperature parameter, pz Zz is pre-trained diffusion model, and p(y z,) is classifier:
V Z2, log (2. y; 2x) = V logp(y) 2.) + Vz, log (n, ze)..
Classifier-free We can also observe that appyling Bayes' rule to plu x) and differentiating with respect to the input yields Vx log x) Vx log y) Vx log p(x). Applying this to ply 2.) and plugging into (3) gives us the formulation for classifier-free guidance:
Vz, log
y, 2. = 7 V.. log (z. y, ze) Vx, log (z 2:)] v., log Po 2:) 7 v, x, log Po (z, y, 24) + (1 V 2, log (a, 2z))
(4)
with pe( y. 2.) conditionally and pe z;) unconditionally trained diffusion models.
GUIDANCE ALGORITHMS FOR DISCRETE DIFFUSION
Applying guidance to discrete diffusion is challenging because guidance terms differentiable with respect to the discrete representations z, (Gruver et al., 2024). Here, we introduce two guidance algorithms for discrete diffusion that circumvent the issue of non-differrntiiabili, As in Section 2.2, we formalize the guidance term as a probability p(y|z) E A where E {l,.... K) is one of K possible classes, and we begin with the distribution that controls the strength of the guidance term via the y temperature parameter (with denoting the logarithm of the normalization constant):
log (z. y, zz) = log p(y
Z,, Z;) + log p(z, 1z)) +
(5)
3.1 CLASSIFIER-FREE GUIDANCE
Guidance for a Single Token To derive classifier-fre guidance, we apply Bayes' rule to the first term on the right-hand side of (5):
logp" (z. y, 2 7 log p(z. v. 2,) 7 log p(,, 22)) + 7 log ply 22) + log 2))
Absorbing 7 p(y 2z) into the constant e and grouping terms yields:
log pd (z,, y, 24) 7 p(z, y, 2.) + - 7) log ((z. 22)) +
3

---

Published as a conference paper at ICLR 2025
Extension to Sequences In the derivation for (6), replace the single-token latent variables with sequences of multiple tokens:
logr" (2g
(2:L)
y; 2i (1:b)) - 7 log log
(122 (1:2)
y ( (1:b)) (1 7) log p(z! (2: 14 (libb) c.
(7)
(2 (z!
(1:L)
y. 2t ( x p(z! vv 2 ) p( (1:L)
2 (112))((-)))
Discrete Classifier-Free Guidance We can model the probabilities in (7) by training a conditional
denoising diffusion network (z. (1:2) y, (1:()) and an unconditional one pa (s (1:2) 2 (.:2)) In
practice, we train these models tandem by randomly "dropping out' or 'masking' the conditioner during training to simulate the unconditional diffusion model. Each of these distributions is thus
modeled to factorize independently along the sequence length dimension of z (1:L) when conditioned on zi (1=b), allowing us to efficiently sample from the tempered guidance distribution as follows:
(1:L) po (z( Zz
v) = II 8=1 Z po(2!
1 zt ( po z (1-0))(1>)
where 200 = Ex, Po(z's z; (1:L) y) pa (z/s z2 ( is the per-token partition function.
We refer this method as D-CFG for Discrete Classifier- Free Guidance.
3.2 CLASSIFIER BASED GUIDANCE
Guidance for Single Token Exponentiating (5) lets us draw classifier-guided samples from the following tempered distribution:
ply 2s, p(z, z,)
p² (Zg Ze.0) Ex; p(y nz, ) zi)
(9)
which we can tractably normalize, as we only sum over the N possible values of single token z,. Extension to Sequences Unfortunately, extending (9) to sequences is not trivial, since the classifier p(yl x (1:L) zt (1:L) does not necessarily factorize independently across the sequence, which would potentially force us compute normalization constant with N terms, accounting for every pos- sible value of z (1:L), To alleviate this issue, we require the assumption that conditioned on z; (I:L) the tempered distribution p²(2z (1:1.) x!!! y) factorizes independently across tokens. Therefore,
can focus on the tempered distirbution of each token z!!), for E I,...,L: L:
(1:L) 1d (z! 1i v) ply
| (2:2)).
(10)
Discrete Classifie-Bassed Guidance We first introduce the following additional notation: given 2 (1:L), let (Z:(LL) be the set of sequences ((1LL) for which z((") 2((") for all E + f, i.e., the set of sequences that are either the same as or only differ in position l relative to (1:L), We can
sample from p³ pz(!") z, (((:L) y) by training classifier P& y + A on noised latents z(! (I:L) for
t E [0, 1] and use this to model the first term the right-hand side of (10) by only evaluating ps on (1:2) differ by at most the token at position E:
sequences for which z! (I-L) and zi
ply 2 zi (i:L)) zo poly 2 (200) for ((:..)
(1:f-1)
(0) zz
[e+1:1)] E Zz(z (1:i).
We additionally train an unconditional denoising network po the ne-normalized distribution:
(0)
2 (l:bi) and then sample from
Po ((::))) po
z (1:b)
(z(
z
(1:L)
II
Σ Po z Z(::L)) po(2!
z (::1))
(1I)
Restricting the summation in the denominators of (11) to 2z (1:b)) makes normalization tractable, only summing N terms.
4

---

Published as a conference paper at ICLR 2025
We refer to our method D-CBG for Discrete Classifier- Guidance. Our method can be thought of as an adaptation of the successful FUDGE (Yang & Klein, 2021) approach, which guides AR generation, to discrete diffusion, similar to how NOS (Gruver et al., 2024) extended the AR guidance mechanism of PPLM (Dathathri et al., 2019) to diffusion models.
First Order Approximation While tractable, this formulation suffers from the drawback that each denoising step we must perform O(L NN) forward passes through the classifier model, which can quickly become impractical for larger vocabularies and longer sequences. Similarly Grath- wohl et al. (2021), Vignac et al. (2022), we can treat the classifier Ps as a continuous function of the one-hot inputs 5 (1122) E RRL.N and use the first order Taylor approximation of log Pa to efficiently compute (v x (1:2) with only single forward and backward pass through the classifier model:
log Po / (1:1)) + log ( zi (1:2))
Po E (()))) = exp
(u (1:2)
~ exp (z(lLt) zi (1:L) TV (ua logpo 2, (2L)) + logpo(( 12l 2. (12)
4
UNIFORM DIFFUSION LANGUAGE MODELS
While masked diffusion models have demonstrated better language modeling compared to other discrete diffusion (Austin et al., 2021; Lou et al., 2023), we argue that they are less amenable to guidance, since once a token is unmasked at some time t it remains so for all s < t. In contrast, for uniform noising, intermediate latents can be refined multiple times during the denoising process. We therefore revisit categorical uniform noise discrete diffusion, where = == I/N. Our aim is that by analyzing this class diffusion models more carefully, we can reduce the gap to absorbing-stad and yield performant models that are more easily steered by the guidance tools we developed above.
4.1 UNIFORM NOISE DIFFUSION
Discrete Time Likelihood Bound We start with the variational lower bound that is defined by a general diffusion process. For discrete-time diffusion, i.c., some finite steps T. we define t(i) (i + 1)/T and sii = i/T for in O,... T. Denoting the Kullback-Leibler divergence as DKL, the denoising network po is trained minimize variational upper bound (NELBO), which is given by:
T
Ey
log Po (x R(()))
DKL(g(Z.()) 200) (2.
+ DKL (xcccrr)))ppppppoc
Crccons
Eprior
ddiftusion
(13)
When clear, we drop the explicit dependence of t and s on the discrete step i.
Uniform Noise Forward Process We focus on uniform noise diffusion using the interpolating discrete diffusion framework (Sahoo et al., 2024a; Zhao et al., 2024; Zheng et al., 2023). When letting π u. the input x transitions to a random state with some probability each time step. Crucially, after x changes once, it can do so again. When π = u. the posterior from (2) becomes
Nazz C x + (atts az)zz + (a. ax)x + (a, (o000))11))) 1
No.
((z. 1,, x) = Cat
Zoi
(14)
No. (Z,, x) + Oz
Denoising Process The optimal form for the reverse diffusion process pe matches (14): in fact setting Po to (14) reduces the KL. terms in (13) to zero. However, setting po to exactly (14) is not possible because it cannot be function (which pa is generating). Therefore,
introduce predictive model (*(Z,, t) V x [0, + A of the 'clean' data given a noisy latent z, at time t. We use X9 to parameterize the denoising process as pa 2,, = g(z, Zz, X Xo). yielding:
Na,2 3 X9 + (arla an)2 + (a, (1))x0 + 10.=,001-a0
Pa 2,) Cat
Zoi
No.
(15)
Na xo) + Olt.
Note that this minimizes the Laimissonn term in (13) precisely when Xg = x, as desired. To simplify notation, omit below the explicit dependence of X9 on f.
5

---

Published as a conference paper at ICLR 2025
4.2 IMPROVED LIKELIHOOD BOUNDS CONTINUOUS TIME
We present contribution to uniform noise discrete diffusion modeling. We leverage the above formulation to develop an improved NELBO by taking T + oo and analyzing each term frecoms: Laifiusion piirr in (13). This yields three improvements. (1) a simple and elegant closed-form expression for the NELBO that is easier to reason about; (2) an analytical reduction of Frecons: Ppiio zero, which tightens the NELBO; (3) further tightening via the continuous-time extension of Ldiffusion as in Kingma et al. (2021) and Sahoo et al. (2024a;b).
Prior Loss pprirr) Given that define our corruption process an interpolation between clean data and limiting distribution, for any noise schedules where (:(T) = O, the distribution q (Zz) = π. Therefore, we can simply define po (Z=(T)) π, and the KI. divergence in Lpiiar evaluates to zero.
Reconstruction Loss (Crecon)) For Lrecoms. if our noise schedule is such that T
Oc((0) 1 (i.e., ((0) = 1/l),, then the marginal O 7. | x) - Cal(z 1.3 ; That is, in the limit, the first thus parameterize
denoising network such that at time t(0) the function simply copies its inputs: xo 1/T) = 2 Additionally, we
latent vector is identically equal to the clean data. We
note that our choice of parameterization for po implies that pa (x 124 = xo 4 1/T). Thus in the continuous time limit, we have:
lim [reoonss lim Eyllog 24)] T-+00
lim E,ylog( (x, xe (z 1/TTM = 0. T-+00
T-30
Diffusion Loss (Caiffusion) Turning finally to the diffusion loss term Ldiffusion- we first define the shorthand DKL (4t [ DRL [Zg Ze, x) (z.. 2r]] and then re-write this loss term as an expectation over t uniformly sampled from 1/T,2/T....
T
Ldiffusion
DKL.ug/Tle = T Be DKL. (rI.u.llp))
(16)
Plugging in our expressions for the true and predicted posteriors from (14) and (15) into (16), then taking T + oo, we get (see Appendix for details):
lim Ldiffusion T-000
lim Ee--11/T:/TT....T777 DKL[gellpol
lim T T-000
Dkulgellooddt
T
m=
(17)
N
N
Xj
(Ro); Rj
log
dt,
Na
X;
(R))
&;
(Xe)s Xx;
j aL =
where x, denotes the jth index vector x, = Na,x + (1 ax) 1, Xg = No, + (1 oxt) and we define i arg maxje[N](Z)) (2,1 to be the non-zero entry of
Combining our arguments regarding Eprir and Lesonus with (17), yields our final tight bound:
0
E
ạ Cke
N
N
Σ
x x;
log
(xx); X; (Xo); 8;
dt.
(18)
1==0
X;
St (2,),=0
Extension to Sequences Extending training with (18) from E V to sequences (1:L) VL we make the assumption that the denoising process factorizes independently across tokens when condi-
tioned on a sequence of noisy latents 2, (I:L) In this case, we use single model (8) (z) (I:L) () for
predicting each token E E (1,..., L. in sequence, and train with the sequence-level objective:
Eyg
0
N
N
Σ
log
(x
dt
No
(
(2)
x
(
j
(19)
We dub models trained with refined objective Uniform Diffusion Language Models (UDLM).

---

Published as a conference paper at ICLR 2025
5 EXPERIMENTS
Datasets For our language modeling experiments, we examine several discrete domains: reference genomes from tens species (Species10), the QM9 small molecule dataset (Ruddigkeit et al., 2012; Ramakrishnan et al., 2014), where molecules are represented by SMILES strings (Weininger, 1988), CIFARIO discretized images (Krizhevsky et al.. 2009), and three NLP datasets consisting of text8 (Mahoney, 2011), Amazon Review (McAuley & Leskovee, 2013; Zhang et al., 2015), and the one billion words dataset (LMIB; Chelba al. (2014)). These datasets cover range of domains and vocabularies of varying sizes (see Table 1). For guidance experiments, we explore species-specingg sequence generation, molecular property maximization, and class-conditional image generation.
5.1 LANGUAGE MODELING WITH UNIFORM NOISE DISCRETE DIFFUSION
Our language modeling experiments show that (1) contrary to a widely-held belief, uniform noise diffusion can attain oafe-offthheart performance on small vocabulary datasets (Table 1), and that (2) our UDLM are sfate-of-the-art among uniform noise diffusion (Tables and 3).
In Table 1, despite previous evidence in- dicating that absorbing-state discrete diffu- sion greatly outperforms uniform noise, find a more nuanced story. Namely, for smaller vocabulary regimes, the gap be- tween MDLM and UDLM is negligible, with UDLM even outperforming absorbing-state on certain datasets. Even within single do- main, we find trend between vocabulary size and performance gap, with the text8 results of UDLM on par with MDLM, and a more per- sistent gap for larger vocabularies. Intuitively, for larger vocabulary regimes, uniform noise diffusion models need predict over com- binatorially larger set of potential clean data sequences compared to absorbing-state; smaller vocabularies reduce this complexity.
Table 1: UDLM performs best with smaller vocabs. Perplexity (L) on various datasets. Best values are bolded. indicates values reported from early stop- ping on the validation set; otherwise validation per-
formance the end of training is used. From Sa-
hoo et al. (2024a). FFrom Lou et al. (2023).
Vocab.
AR
MDLM
UDLM
Species10 QM9"
12
2.88
3.17g
3.15g
40
2.19
2.12g
2.02g
CIFARIO
256
9.14g
11.21
text8
35
2.35$
2.62
2.71
Amazon
30,522
21.67
24.93
27.27
LM1B
30,522
22,32
27.04
31.28
Secondly, despite the persisting gap between absorbing and uniform noise discrete diffusion for larger vocabulary NLP datasets, we note that models trained with UDLM close this gap, as evidenced in Tables and where we show that UDLM attains the best reported uniform noise discrete dif- fusion language modeling performance.
38
T= 1000
T 8192
36
UDLM (T
34-
32
30-
28
Ablating Continuous Time Formulation In Figure 2, we see the effect of increasing T and using our continuous time NELBO. The curves represent validation perplexity on the Amazon Polarity dataset at various training steps, and we observe that increasing T indeed improves language mod- eling with UDLM, i.e., T = xc, performing best.
100k
150k
200k
250k
300k
350k
400k
Step
Figure 2: T + oc improves valida-
tion PPL (1) on Amazon dataset.
5.2 GUIDED DISCRETE DIFFUSION
Our guidance results indicate that (1) classifier-free guidance is more useful when paired with dif- fusion models compared to AR (Table 4) and that (2) our proposed D-CBG is the best classifier- based method for discrete guidance, especially when combined with UDLM (Table & Figure 3).
Baselines For guidance experiments, our primary baseline is the dominant AR approach. We com- pare to three flavors of guided AR. The first is applying D-CFG to AR models. We also use the established control mechanisms of Plug-and and-play language models (PPLM; Dathathri et al. (2019)) and FUDGE (Yang& & Klein, 2021). To demonstrate the better performance of our D-CBG method, compare to Gruver et al. (2024) (NOS), an extension of PPLM to discrete diffusion.
7

---

Published as a conference paper at ICLR 2025
Table 2: UDLM outperforms other uniform dis- crete diffusion on text8. Best value is bolded & best uniform diffusion value is underlined. From Lou et al. (2023). 'From Shi et al. (2024).
Table 3: UDLM outperforms other uniform discrete diffusion on LMIB. Best value is bolded & best discrete uniform diffusion value is underlined. FFrom Sahoo et al. (2024a). "From Lou al. (2023). FFrom Austin et al. (2021).
Method
BPC (4)
Autoregressive
IAF/SCFI (Ziegler Rush, 2019)
1.88
Method
PPL (1)
Argmax Flow! (Hoogchoom al., 2021)
1.39
Autoregressivee
Discrete Flow (Tran 2019)
1.23
Transformer-X 2019)
23.5
Autoregressive
1.23
OmniNetT (Tay et 2021)
21.5
Non-autoregrressivve
Transformer
22.32
Mult. Diffusion" (loogghooom et al., 2021)
1.72g
Diffusion"
MAC! (Shih el al., 2022)
1.40g
BERT-Mouth (Wang & Cho, 2019)
142.89
BFN (Graves et al., 2023)
1.41
D3PM Absorb (Austin et al., 2021)
77.50g
Diffusion-LMM (L.i al., 2022)
118.62g
D3PM Absorb (Austin al, 2021)
1,45g
DiffusionBert (Wang Cho, 2019)
63.78g
SEDD Absorb (Lou et al., 2023)
1.39
SEDD Absorb (Lou al,, 2023)
32.79g
MDL M (Sahoo al., 2024a) (Retained)
1.38g
MDLM (Sahoo et al., 2024s)
27.04g
MD4" (Shi al., 2024)
1.37
GenMD44 (Shi al,, 2024)
1.34g
Discrete Uniform Diffission
D3PM Uniform (Austin 2021)
137.9g
Discrete Uniform Diffusson
SEDD Uniform" (Lou et al., 2023)
40.25
D3PM Uniform
(Austin al., 2021)
1.61g
UDLM (Ours)
31.28
SEDD Uniform"
(Lou et al., 2023)
1.47g
UDLM (Ours)
1.44g
Hyperparameters For both D-CFG and D-CBG, we vary the strength of the y parameter. Although the original FUDGE formulation simply uses 1, also perform a search for this baseline. For PPLM and NOS, we vary the parameters the Langevin sampling that updates models' hidden rep- resentation, i.e., number of update steps n, step size 77. and fluency KL weight yka (see Dathathri al. (2019) and Gruver et al. (2024) for more details). For all methods, we display the best performing hyperparameter configuration in the main table results, and defer the the full sweep Appendix E.
Species-speciffe Genome Generation For ge- nomic sequences, evaluate D-CFG with dif- fusion compared to with AR. We train on se- quences of 32,768 nucleotides, using base-pair level tokenization and conditionally generate 64 sequences for each class. We train small classifier to distinguish between generated and validation set sequences and report the area un- der the receiver operator curve for this classi- fier (Disc. AUROC). Values closer to 0.5 indi- cate that the classifier is unable distinguish between synthetic and true sequences (Sarkar al., 2024). To measure the controllability. we train a separate classifier on the Species 10 dataset and measure macro Fl score of this 'or- acle' classifier on the generated sequences. Re- sults of this experiment are presented in Ta- ble 4. For reference, we also provide metrics for randomly generating sequences with nucleotide frequencies proportional to species representa- tion in the data. We find that both MDLM and UDLM are able to better generate sequences that match the desired control parameter, with higher Fl scores relative to AR. Moreover, UDLM is able to outperform MDLM in satisfying this control. Importantly, we find that only UDLM is amenable to increasing the guidance parameter ?, where its metrics improves while AR and MDLM metrics degrade. Finally, of note, the diffusion model generation for this experiment is accomplished with far fewer function evaluations compared to AR. Whereas AR must decode each of the 32,768 tokens, because MDLM and UDLM can decode multiple tokens in parallel, generate with T 512.
Table 4: Diffusion decoding with D-CFG is more controllable than AR for genomic sequences. Mean + standard deviation reported from five ran- dom seeds. Best values are bolded.
D-CFG
Disc. AUROC (L)
Model
(f)
Random
1.00 0.53
0.07
AR
0.87
AR
0.90++a051
0.8 +00000
AR
0. 0.74
MDLM
0.51 1
0,88
MDLM
0.74
0.91 1:0000
MDLM
0.93
0.78 1:000
UDLM
0.52+a004
+..1
UDLM
0.61 2.044
0.93:4000
UDLM
0.87+
0.94
8

---

Published as a conference paper at ICLR 2025
Table 5: Guidance with discrete diffusion models better balances the generation of valid and novel molecules with maximizing the property of interest, drug likeness (QED) / ring count, compared to AR. Validity, novelty, and mean QED / ring count for novel sequences are measured for generated sequences from each method. Mean + standard deviation reported from five random seeds. Best values are bolded.
QED Novel (t)
Ring Count
Method
Guidance
Valid (f)
Mean (t)
Valid (t)
Novel (t)
Mean (1)
Original Data
133k
133k
0.47
133k
133k
1.74
Classifier-fre
AR
D-CFG
946.4290
79.4264
2.60
441.4
77.8
4.83
MDLM
D-CFG
317.4211
95,8299
+..0
90.0
60.0
5,26:4.15
UDLM
D-CFG
1013.6425
64.0
0.62
998.2+45
216.2+130
4,88
Classifer-based
AR
FUDGE
924.4-12.1
53.0=3.5
+0:00
281.2+50
103.6448
4.89
AR
PPLM
1007 1007.2+a.
142.0_14.
0,45 0..5
1009.841.9
140.6
1.92+am
MDLM
D-CBG
417.6
116.6=89
g0.00
113.0+88
85.6+88
4.75
MDLM
NOS
506.0
240.4 16.0
0.45 0.001
247.8
193.6413.1
3,5 4:022
UDLM
D-CBG
994.8+22
63.8
0.61 40000
897.2
432.0
4,84
UDLM
NOS
547.8
158.8
0.47
573.84130
244.4
3.964007
Molecule Property Maximization (QED)
Molecule Property Maximization (Ring Count) Guidance method: D-CFG
Guidance method: D-CBG FUDGE
Modal
Gamma
0.60
AR
AR
MOLM UDLM
Novel Mean Ring Count
5.1
UDLM
3
Novel Mean QED
0.58
5.0
5
Gamma
4.8
4.7
0.54
4.6
4.5
Num. Novel Samples (out of 1,024 generated)
Num. Novel Samples (out of 1,024 generated)
Figure 3: Diffusion models extend the steer-ability Pareto frontier. (Left) D-CBG outperforms FUDGE classifier guidance when maximizing drug-likeness (QED). (Right) D-CFG with diffusion better trades-off novel generation and ring-count maximization compared to AR.
Table 6: Guidance improves FID & IS on CIFAR10. Finite- (D3PM) continuous- time (MDLM UDLM). Best values are bolded. FFro Austin al. (2021).
Molecular Property Maximization For QM9, in- vestigate novel generation of sequences that maximize either drug-likeness (QED; Bickerton al. (2012)) or number of rings present in the molecule. We only re- port values for which we generated least 50 novel quences (out of 1,024). In Table 5, for D-CFG, we see that all methods perform comparably when maximiz- ing drug likeness, but that MDLM and UDLM are bet- ter suited for the structural property of ring count. For classifier-based guidance mechanisms, that both QED and ring count, D-CBG with discrete diffusion best trades-off maximizing these properties while generating valid and novel sequences compared to AR diffusion with NOS.
FID (L)
IS (f)
D3PM Absorb
41.28
6.26
MDLM
33.75
6.74
MDLM D-CFG
15.56
9.02
again find
D3PM Uniform
51.27
5.99
UDLM
33.65
6.86
UDLM D-CFG
23.21
8.66
Class-con Image Generation In Table 6, we see that both MDLM and UDLM outperform finite-time counterparts (in the form of D3PM (Austin et al., 2021)) with improved image quality

---

Published as a conference paper at ICLR 2025
metrics of Fréchet inception distance (FID; Heusel et al. (2017)) and Inception Score (IS; Salimans et al. (2016)). This is especially true when we add guidance using D-CFG. MDLM/ MDLM/UDLM UDLM samples generated with T 1000.
Faster Sampling In Table 7, we explore using faster inference settings (smaller T)) where diffusion models predict multiple pixels in parallel. These results focus the CIFAR 10 dataset; Appendix E.3 for additional experiments on other datasets. MDLM's performance deteriorates with smaller T. whereas UDLM robust this setting. This validates a key motivation behind UDLM: in settings where MDLM 'locks in' certain predictions that it cannot change, UDLM is more resilient given that all tokens can change throughout the decoding process.
Ablation:
6 RELATED WORKS, DISCUSSION, AND CONCLUSION
Discrete Diffusion Recent works have examined inter- polating discrete diffusion (Ou et al., 2024; Sahoo et al., 2024a; Shi et al., 2024; Zhao et al., 2024; Zheng al., 2023), a special case of the general framework from al., 2021). Our UDLM method most closely aligns to the state-of-the-ar discrete diffusion of Ou al. (2024), Sahoo et al. (2024a), and Shi et al. (2024) that focus on absorbing state diffusion. Similar to these works, we provide continuous time NELBO leading to performance gains, but we focus on uniform noise. Of note, other extensions of D3PM that do not start from the variational perspective, instead rely on the formalisms of continuous time Markov chains (CTMC) (Campbell et al., 2022) and concrete score matching (Lou et al., 2023), but are less performant than works such as MDLM and our method. Also stemming from CTMC are extensions of flow-based (Lipman et al., 2022) approaches to discrete data (Campbell et al., 2024; Gat et al., 2024). In Appendix C, we discuss connections of our work to CTMC in more detail.
Table 7: UDLM is robust to faster sam- pling- CIFARIO images sampled from conditional model (D-CFG, Best values per T are bolded.
D3PM (Austin
Model
FID (L)
IS (1)
T = 128
MDLM
64.09
5.81
UDLM
30.48
7.30
T = 1024
MDLM
27.94
7.14
UDLM
26.70
7.43
Guidanee Leveraging continuous embeddings of discrete data, Diffusion-LM (Li et al., 2022) uses Langevin sampling with CBG. Similarly, SSD-LM (Han et al., 2022) perform Gaussian noising on the logits of a bi-directional model. which they combine with pre- pre-trained classifiers to perform CBG with Langevin dynamics. L.4LLG (Lovelace et al., 2024) implement CFG on continuous embeddings. Stark et al. (2024) use flow-matching on the simplex to perform CFG and CBG on continuous representations. Wang et al. (2023) perform guidance using auxiliary semantic latent variables. In contrast, to these continuous formulations for discrete data, our work adapts guidance mechanisms directly to the discrete domain.
FUDGE (Yang & Klein, 2021) can be viewed as the AR analog to our D-CBG. DiGress (Vignac et al., 2022) uses a similar first-order approximation to resolve the intractability of the normalizing constant. In our work, we derive a tractable expression for classifier-basee guidance and simply use the Taylor approximation to speed up computation in large sequence length and vocabulary size regimes. Sanchez et al. (2023) derives an equivalent formulation for D-CFG and use to better enforce AR models' adherence comparable method to D-CFG, but focus on graph diffusion models. Most similar to our method, the concurrent work of Nisonoff al. (2024), which derives classifier-based and classifier free guidance for discrete diffusion and flow models. However, this work is highly tailored to models that leverage the formalism of CTMC, and guidance is applied the rate matrices. See Appendix B for details on relating our guidance methods to Nisonoff et al. (2024).
prefix prompts. FreeGress (Ninniri al., 2024) also offer
Conclusion In search of a more controllable diffusion process, in this work, we derived a tight variational bound for uniform noise discrete diffusion, closing the gap to state-of-the-art absorbing- state diffusion models. We also highlighted that contrary to previous findings, in small vocabulary regimes, uniform noise is on par better than absorbing state. We then demonstrated that straight- forward adaptations of classifier based and classifier- free guidance can offer improved guided gen- eration relative to AR models. We found that with classifier-free free mechanisms, diffusion models are more amenable to control without sacrificing quality of generated sequences. We also demonstrated that our Hassifier-basee method is better than previous ones for both AR and diffusion models.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS AND DISCLOSURE OF FUNDING
We thank Nate Gruver for useful discussions regarding NOS. This work was supported by NSF CAREER grants (#2145577 and #2037519) and NIH MIRA grant (#IR35GM151243-01),
REFERENCES
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 19911-19999 2021.
G Richard Bickerton, Gaia V Paolini, Jérémy Besnard, Sorel Muresan, and Andrew L Hopkins. Quantifying the chemical beauty of drugs. Nature chemistry, 4(2):90 2012.
Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. Advances in Neural Information Processing Systems, 35:28266-28279, 2022.
Andrew Campbell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. arXiv preprint arXiv:2402.04999, 2024.
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling, 2014.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdi- nov. Transformer-xl: Attentive language models beyond fixed-length context. arXiv preprint arXiv: 2019.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosin- ski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. arXiv preprint 1912.02111. 2019.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8799 2021a.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8794 2021b.
Ruiqi Gao, Aleksander Holynski, Philipp Henzler, Arthur Brussee, Ricardo Baatin-Brualla, Pratul Srinivasan, Jonathan T. Barron, and Ben Poole. Cat3d: Create anything in 3d with multi-view
diffusion models, 2024. URL https: aawiiiiiivvv///.i/ org. org. hass///b///a /2005.20500 110444111111.,.. 10314.
Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky TQ Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching. arXiv preprint arXiv:2407.15595, 2024.
Aaron Gokaslan, A Feder Cooper, Jasmine Collins, Landan Seguin, Austin Jacobson, Mihir Patel, Jonathan Frankle, Cory Stephenson, and Volodymyr Kuleshov. Commoncanvas: Open diffusion models trained creative-commons images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- -260, 2024.
Will Grathwohl, Kevin Swersky, Milad Hashemi, David Duvenaud, and Chris Maddison. Oops i took a gradient: Scalable sampling for discrete distributions. In International Conference on Machine Learning, pp. 3831-3841. PMLR, 2021.
Alex Graves, Rupesh Kumar Srivastava, Timothy Atkinson, and Faustino Gomez. Bayesian flow networks. arXiv preprint arXhv:2308.07037, 2023,
Nate Gruver, Samuel Stanton, Nathan Frey, Tim GJ Rudner, Isidro Hotzel, Julien Lafrance-Vanie Vanasse Arvind Rajpal, Kyunghyun Cho, and Andrew G Wilson. Protein design with guided discrete diffusion. Advances in neural information processing systems, 36, 2024.
11

---

Published as a conference paper at ICLR 2025
Albert Gu and Tri Dao. Mamba: Ainear-tims sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.
Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. Ssd-Im: Semi-autoregressivve simplex- based diffusion language model for text generation and modular control. arXiv preprint 0rXiv:2210.17455: 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. arXiv:2207.12500 2022.
Classifier-free
diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-666 6851, 2020.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling, Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems, 34:12454-12465 2021.
Frank P Kelly. Reversibility and stochastic networks. Cambridge University Press, 2011.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Ad- vances neural information processing systems, 34:21696-21707, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009.
Greg Landrum et al. Rdkit: A software suite for cheminformatics, computational chemistry, and predictive modeling. Greg Landrum, 8(31.10):5281, 2013.
Xiang Li. John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. Diffusion- Im improves controllable text generation. Advances in Neural Information Processing Systems, 35:4328 4343, 2022.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02777., 2022.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion language modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310.16833 2023.
Justin Lovelace, Varsha Kishore, Chao Wan, Eliot Shekhtman, and Kilian Q Weinberger. Latent diffusion for language generation. Advances in Neural Information Processing Systems, 36, 2024.
Matt Mahoney, Text8 dataset, 2011. URL http:
Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: understanding rating dimen- sions with review text. In Proceedings of the 7th ACM conference on Recommender systems, pp- 165-172, 2013.
Chenlin Meng, Kristy Choi, Jiaming Song, and Stefano Ermon. Concrete score matching: General- ized score matching for discrete data. Advances in Neural Information Processing Systems, 35: 34532-34545, 2022.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741, 2021.
Mattco Ninniri, Marco Podda, and Davide Bacciu. Classifier-fre graph diffusion for molecular property targeting. In Joint European Conference Machine Learning and Knowledge Discov- ery in Databases, pp. 318-335. Springer, 2024.
Hunter Nisonoff, Junhao Xiong, Stephan Allenspach, and Jennifer Listgarten. Unlocking guidance for discrete state-space diffusion and flow models. arXiv preprint arXiv:2406.01572. 2024.
12

---

Published as a conference paper at ICLR 2025
Nuala A O'Leary, Mathew W Wright, J Rodney Brister, Stacy Ciufo, Diana Haddad. Rich MeVeigh, Bhanu Rajput, Barbara Robbertse, Brian Smith- White, Danso Ako-Adjei, Alexander Astashyn, Azat Badretdin, Yiming Bao, Olga Blinkova, Vyacheslay Brover, Vyacheslav Chetvernin, Jinna Choi, Eric Cox, Olga Ermolaeva, Catherine M Farrell, Tamara Goldfarb, Tripti Gupta, Daniel Haft, Eneida Hatcher, Wratko Hlavina, Vinita S Joardar, Vamsi K Kodali, Wenjun Li, Donna Ma- glott, Patrick Masterson, Kelly M McGarvey, Michael R Murphy, Kathleen O'Neill, Shashikant Pujar, Sanjida H Rangwala, Daniel Rausch, Lillian Riddick, Conrad Schoch, Andrei Shkeda, Igor Tolstoy, Raymond E Tully, An-
Susan S Storz. Hanzhen Sun, Francoise Thibaud-Nissen jana R Vatsan, Craig Wallin, David Webb, Wendy Wu, Melissa J Landrum, Avi Kimchi, Tatiana Tatusova, Michael DiCuccio, Paul Kitts, Terence D Murphy, and Kim D Pruitt. Reference se- quence (RefSeq) database at NCBI: current status, taxonomic expansion, and functional annota- tion. Nucleic Acids Res., 44(D1):D733 January 2016.
Jingyang Ou, Shen Nie. Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. arXiv preprint (rXiv:2406.037366 2024.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint WxXiv:220.149988 2022.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019.
Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):--7, 2014.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical image computing and computer-ussisted intervention- MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceed- ings, part III 18, pp. 234-241 Springer, 2015.
Lars Ruddigkeit, Ruud Van Deursen, Lorenz C Blum, and Jean-Louis Reymond. Enumeration of 166 billion organic small molecules in the chemical universe database gdb-17. Journal of chemical information and modeling, 52(11 ():28)))2224888 2012.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- tion processing systems, 35:36479-366 994444944,4,, 2022.
Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. arXiv preprint 2.XXi:::4000072224 2024a.
Subham Sekhar Sahoo, Aaron Gokaslan, Chris De Sa, and Volodymyr Kuleshov. Diffusion models with learned adaptive noise. arXiv preprint arXiv:2312. 13236, 2024b.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Guillaume Sanchez, Honglu Fan, Alexander Spangher, Elad Levi, Pawan Sasanka Ammana- manchi, and Stella Biderman. Stay on topic with classifier-free guidance. arXiv preprint 1rXiv:2306.17880 2023.
Anirban Sarkar, Ziqi Tang, Chris Zhao, and Peter Koo. Designing dna with tunable regulatory activity using discrete diffusion. bioRxiv, pp. 2024-05, 2024.
Yair Schiff, Chia-Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and Volodymyr Kuleshov. Caduceus: Bi-directional equivariant long-range dna sequence modeling. arXiv preprint arXiv:2403.03234, 2024.
13

---

Published as a conference paper at ICLR 2025
Philippe Schwaller, Teodoro Laino, Théophile Gaudin, Peter Bolgar, Christopher A Hunter, Costas Bekas, and Alpha A Lee. Molecular transformer: model for uncertainty-calibratee chemical reaction prediction. ACS central science, 5(9):1572-1883, 2019, 2019.
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K Titsias. Simplified and gener- alized masked diffusion for discrete data. arXiv preprint arXiv:2406.04333 2024.
Andy Shih, Dorsa Sadigh, and Stefano Ermon. Training and inference on any-order autoregressive models the right way. Advances in Neural Information Processing Systems, 35:2762- 2775, 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing. pp. 2256-2265, PMLR, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Hannes Stark, Bowen Jing, Chenyu Wang, Gabriele Corso, Bonnie Berger, Regina Barzilay, and Tommi Jaakkola. Dirichlet flow matching with applications to dna sequence design. arXiv preprint arXiv:2402.05841, 2024.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. arXiv preprint arXiv:2 2021.
Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based continuous-time discrete diffusion models. arXiv preprint LrXiv:2211.1675:5 2022.
Yi Tay, Mostafa Dehghani, Vamsi Aribandi, Jai Gupta, Philipp M Pham, Zhen Qin, Dara Bahri, Da- Cheng Juan, and Donald Metzler. Omninet: Omnidirectional representations from transformers. In International Conference on Machine Learning, pp. 10193-10202. PMLR, 2021.
Dustin Tran, Keyon Vafa, Kumar Agrawal, Laurent Dinh, and Ben Poole. Discrete flows: Invertible generative models of discrete data. Advances Neural Information Processing Systems, 32, 2019.
Clement Vignac, Igor Krawezuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pas- cal Frossard. Digress: Discrete denoising diffusion for graph generation. arXiv preprint arXiv:2209. /4734, 2022.
Alex Wang and Kyunghyun Cho. Bert has mouth, and it must speak: Bert as a markov random field language model. arXiv preprint arXiv:/902.000 /9022049294444 2019.
Yingheng Wang, Yair Schiff, Aaron Gokaslan, Weishen Pan, Fei Wang, Christopher De Sa, and Volodymyr Kuleshov. Infodiffusion: Representation learning using information maximizing dif- fusion models. In International Conference on Machine Learning, pp. 96336-36355 PMLR, 2023.
David Weininger. Smiles, chemical language and information system. 1. introduction to method- ology and encoding rules. Journal of chemical information and computer sciences, 28(1):31-32) 1988.
Thomas Wolf, Lysandre Debut, Vietor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. As- sociation for Computational Linguistics. URL https:/ wwww.w.... a web. org/ anthoooocoo ogy/ 2020. emn 1mmnlps 6.
Kevin Yang and Dan Klein. Fudge: Controlled text generation with future discriminators. arXiv preprint arXiv:2 104.05218. 2021.
14

---

Published as a conference paper at ICLR 2025
Xiang Zhang. Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas-
sification. Advances in neural information processing systems, 28, 2015.
Lingxiao Zhao, Xueying Ding, Lijun Yu, and Leman Akoglu. Improving and unifying discretee&continuuus-tinee discrete denoising diffusion. arXiv preprint crX1vv2402.03701, 2024.
Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. A reparameterized discrete diffusion model for text generation. arXiv preprint arXin:2332.055733, 2023.
Zachary Ziegler and Alexander Rush. Latent normalizing flows for discrete sequences. In Interna- tional Conference on Machine Learning, pp. 7673-7682. PMLR, 2019.
15

---

Published as a conference paper at ICLR 2025
CONTENTS
Introduction
2
Background
2.1
Discrete Diffusion Models
2.2
Diffusion Guidance
3
Guidance Algorithms for Discrete Diffusion
3.1
Classifier-free Guidance
3.2
Classifier-based Guidance
4
Uniform Diffusion Language Models
4.1
Uniform Noise Diffusion
4.2
Improved Likelihood Bounds in Continuous Time
5
Experiments
5.1
Language Modeling with Uniform Noise Discrete Diffusion
5.2 Guided Discrete Diffusion
Related Works, Discussion, and Conclusion
10
Continuous Time Discrete Uniform Diffusion
17
Relating Guidance to CTMC
19
B.1 CTMC Formulation of D-CFG
20
B.2 CTMC Formulation of D-CBG
Relating UDLM CTMC
22
C.1
Equivalence of UDLM ELBO and SEDD ELBO
23
D Experimental Details
25
D.1 Dataset Details
25
D.2 Architectural Details
27
D.3 Training Configurations
28
D.4 Guidance Details
28
D.4.1 Baselines
28
D.4.2
D-CFG Details
29
D.4.3
D-CBG Details
30
D.5
Guidance Evaluation Details
30
D.5.1
Genomic Sequences Metrics
30
D.5.2
CIFAR10 Quality Metrics
30
16

---

Published as a conference paper at ICLR 2025
D.5.3 QM9 Guidance Metries
31
Guidance Ablation Results
31
E,1
Effect of Varying Guidance Hyperparameters
................................
31
E,2 Effect of Using First Order Approximation in D-CBG
................................
E.3 Effect of Varying
................................
33
Unconditional Sample Generation
34
A
CONTINUOUS TIME DISCRETE UNIFORM DIFFUSION
Here, we derive continuous time formulation (T - oo) for the diffusion loss term Ldittusion when using the uniform distribution as the limiting distribution of the diffusion process.
For uniform noise diffusion, we define limiting distribution u 1/N, where 1 represents the column vector of all ones, and N the size of the vocabulary. We adopt the assumption from MDLM (Sahoo et al., 2024a) that our diffusion process interpolates between clean data and noise:
a(Z x)
Cat(2r; 0,x + (1 - (1-0z)
(20)
and the marginals are given as
[Zt 2.) Cat(2z; a. + (1 (((s)
(21)
where Otls az/ag.
Following Austin et al. (2021), we can derive the posterior:
[as aZ
-1-00)))) 1 T z, Ja,x + (1 as)
qzz Ze, x) =
Cat
Z₁
(22)
(21,, + +1 0z)
Using π u as defined above, we get the following:
(a1.-04)
+ (0.-ar)
N x (1 (1-ay))((-as)
Ox
((2, Zz, xx,x)
Cal
Za;
N
(1(2,,)) + (1-0r)
N2
(23)
(a, (a)-a(((-aa)))
Nazz Ox + (atjs -aazz + +(ag ax)x
= Cat
Zai
No.
(24)
Nar x) + att.
For the denoising distribution, we replace x by Xg:
Naz @ (arjs 04 oz)2; (as (1))x0 + (ar (a--or)(-oa)
(z( 21) = Cat
Z,:
a
(25)
Na xo) + ae
Let us now look at the diffusion loss term in the NELBO:
(((z 21; xx;x))
T DxL(g(zs 24, xx.x))lpo(2z
1z)) T
Σ g(z, 2 x), log
(26)
po (z. 2:);
JE[N]
Letting i = arg maxxxe((((z)) be the non-zero entry z,, we can break up this KL into two terms:
q(2, 2,, x)s
T. (g z4, x4.x))lpo(zz 1z)) = T q(2, 2,, x); log
po zt)e
Term 1
g(z. 2t, xx, Pz zz),
+ T
q(z,
x), log
(27)
JE[N] s.t. (2+);=0
Term 2
17

---

Published as a conference paper at ICLR 2025
We now examine each of these terms taking T - oo, equivalently, using t 1 T= taking ->
E-6t
Term 1:
Naxx + Otls -4 + (a, - Ox)); + (a)=at)(1-aa)
lim 5-
a,
Na + 0t
- Na.x
)x. (00-0))(-00))
log
(28)
Nox(Xo):tt - - (xx)::+ (ue will
As - the coefficient the log term will approach 1. Additionally both the numerator and the denominator inside the log term will approach I, thus the entire log term will approach O. Combining this with the fact that will approach 0, gives us the indeterminate form of 0/0, hence apply L' ''Hópital's ''s rule to (28). Writing and pe as functions of when we differentiate the log term we get:
d
g(8)
d log (())
d
log ds
log (o()
(s(s)
ds
ds
4 (s)
pe
(29)
(s)
(s(s)
Let's look each derivative term in (29):
lim lim + (xo); +
d
No, (1,)) -a.)
Na.
((s)
8-28
ds
8-8
Na +
-a4 + a/x; + No:
(1110c)
Na + ai
0g
+ Naxx; + 04
No
Na + 1 -0; 02
0/
N
(30)
No
Naxx; +1-
Similarly,
0f. Noc
N Nar(xe); + at
lim pa(s) e- ds
(31)
Note that when taking s - both q(s) and (s) evaluate to Additionally, differentiating
with respect to s evaluates to 1. When combining these facts and plugging (30) and (31) into (29)
(28) becomes
-a² No:
di Naxx + 1 Na:
N
N Nax + ac
Term
d;
N
(32)
Nai
Nar(xx); + +1 Naxx +1-ai
Term 2:
(a, (a-))))x (as ((t-a))
(a, 10-0pp 01)x + (a.-ar)(1-a)) Na.
lim
log
11ix:+1-00
(33)
Na + Oz.
(ax (o.-ax)(xo))
(0e 1011--a7
JEIN ct. (x,) 0
(oo(Ko).
For each term in the summation, + times the coefficient on the log term will have an indeterminate form of 0/0 as + L. We therefore apply L'Hôpitalls rule to this coefficient:
lim 1 (a, - a1)x, + (a, (a,-oxX1-ax
-a²
Na + 1 at
No.
(34)
Na + 1-a
Na
Na + aq
18

---

Published as a conference paper at ICLR 2025
Now, for the log term, exchange the limit with the continuous log function and have that both the numerator and the denominator go to zero as s > t. We therefore apply L' L'Hôpital's 's rule here as well:
(a, (00-0011,--
Nay
d (a, -x,
Nay
log lim s-
-
log lim
de
Na.x:
(x0))
-e--op1--n Na.
8-28
(as (as-ai](x0))
(0 K1-mol
Nac
Nxr(Xo)1+ 1-0e
d2
Nxr
0/x; (11)-
Nor(xo); + 00t
= log
Naxx;
0 (x0)s + a6(l-ac)
Noz
Nar(Xe); + -1100 Ct
(Nax,, 111-0k1 0kg)
log
No;X;
No: (Nar(())) + (4.)
Nar(Xe): + 1
Naxx + 1 -0t
log
Natx; + 1 Ctt
Nou(Xooj + 0tt
Nar +1 04
Naxxj + +1 Olt
log
(35)
Nor(xx)1 + 04
No + Okt
Multiplying (34) by (35), we get:
-a¹ No: s.t. (2+);=0
Naxx, + 04
+1 att Nxx + 0r)
Naxx + Ot
lim Term 2
Σ
log
No + 0t
Na,x: 04
JEIN
(36)
Combining Terms 1 and 2: Using (32) and (36), the final KL term in the continuous time limit is:
at N
N
lim T T-DKL(qlpp) Noc T0000
Naxx; a at
Nax(xs); +
ate
Naxx, + +1-11
log [ [Na((xo) +1 0' Vor((x))) + 0t
[Naxx + 0t Na 0tty
Σ
Naxx; at
JEIN] (x:)g=0)
Defining No,x + (1-aa) 01) and Xn Na,x + (1 (1)),, as in Section 4.2, yields the desired result.
B RELATING GUIDANCE TO CTMC
Below we demonstrate how our proposed guidance mechanisms from Section 3 admit straightfor- ward continuous-time extensions. In concurrent work, Nisonoff et al. (2024) the formalisms of continuous time Markov chains (CTMC) to derive similar guidance formulas, which we show are equivalent the continuous-timg extensions of our work.
Background on CTMC In the CTMC formulation, the key quantity of interest is the rate matrix R₁ N In analogy to the discrete time transition matrices Q.. which define the transition probabilities between states, these rate matrices define the instantancous rate of change between states continuous time. More formally, letting 8(x, y) be the Kronecker delta function that equals 1 if x y and 0 otherwise, and using and z to denote observed values of the latents, we can define the forward noising process using R, as follows:
glz = z' z) = òzz' Re z T + (
(37)
where o(1/T) indicates terms that vanish more quickly than 1/T. Note that when clear from the context, the shorthand 8zzz 5(2, z'). Recall that defined I 4. In continuous
19

---

Published as a conference paper at ICLR 2025
time, as T - 00, o(1/T) terms are ignored, and we have
lim g(ze = z' 2t- (1/T) 2) 82,8
Rz(z,z')
(38)
7 7-500
1/T
hence our treatment of Rs as the 'instantaneous' rate of state transitions.
Importantly, processes defined as in (37) have known time reversals given by (Kelly, 2011; Sun al., 2022):
(2. = 24 = =z)= dx + Re(z' 7) 1 + of (
(39)
where RS represents the reverse rate matrix which related the forward matrix follows:
g(z z²) (z,zz)), if + z,
Re(2'. z) g(zz = z)
(40)
with =((z,z)=-Ex (z', z), which ensures that the rows of RG sum to zero (i.e., mass cannot be created or destroyed), Note that in (40), we are scaling the forward rate matrix by ratio of the unconditional marginals: q(z, zz)/q(z; z).
We remark that in the derivations for guidance in Nisonoff et al. (2024), they slightly change the form of (39) to:
(z,
z') dz (1 2/
6x R. 21)T)
(41)
Sampling with Learned Reverse Rate Matrices Assuming we have a learned model R,, 1 2)
that approximates the true reverse rates (see Campbell et al. (2022) or Campbell et al. (2024) for
details), we generate from this model by drawing samples from the limiting distribution 2 and using a forward Euler discretization to sample chain latents that end in samples x ~ g(x) (Campbell et al., 2024):
~ Cat(azz = z', + (2 = = z2,))))
(42)
where 1 /T represents the discretization step size and, as above, 8 = --t-1/1 1/T. Additionally, note that here we follow the convention of diffusion models and let the time variable go from indicate moving from noise towards signal, in contrast with the flow matching literature, where the reverse time convention is used (Lipman al., 2022; Campbell et al., 2024).
to
B.1 CTMC FORMULATION OF D-CFG
In Nisonoff al. (2024), guidance is achieved by scaling the parametrized reverse rate matrix Rg Predictor-free guidance (the analogous method to D-CFG) is presented as:
(z. =z,22 y) = Rz, z', 2z v) Re,o(z; = z', 2 (1-),
(43)
where Re y) is the learned reverse rate when conditioned on y.
We now present the continuous-time extension D-CFG and show that it is proportional to using the predictor-free free guided reverse rate from (43) in conjunction with the forward Euler sampling from
(42). Starting from the formulation of D-CFG from our work (where we make more explicit which variables are free / set)
pd (z,, 2, = z'.v) v) (z v)" po (2, (2)(1-))
20

---

Published as a conference paper at ICLR 2025
when using the forward Euler sampling scheme, can plug in the formulation from (41) (with the true reverse rate replaced with the approximate one) into the right- hight-handdae side of this equation to get
p² (Zg Zt
z',9)= x
0((z z', 21,2,)11 Rz. = z', =z' l1)))) + (1 8(z, z' 2'',=))'z) Rs)) z', 20 (n((L/T))
(o(z) 21) Re, n2, =z (1)/() + (1 8(2z n 2c) z (,)(1/T)) (1-7)
8(z, z', 2,) +11+100 R2, = 2z z' 11) -11 Re,0(Z, = =2 22
+11 11-0 6(z, = 2(. = 2z 7)7 Rz, = =z, (1-,
8(z z', 2z, + Ri
z', 2. =2 =21(
+ +(1 - 8(z, = = R R2
=' (1/T)
where the first equality comes from the fact that all cross terms involving 8(2; = =x'; .1 (1--6) -8(z; 2)) evaluate to zero, allowing us to aggregate terms under the exponents y and - 7; and the second equality comes from (43).
B.2 CTMC FORMULATION OF D-CBG
While for classifier free guidance, we did not need to distinguish between latent variables with a single token and those representing sequences of tokens (as argued in Section 3.1), below make this distinction explicit by including superscripts (f) and (1 L)) on variables 2,, 2,- However, for notational simplicity we omit theses subscripts on variables that denote realized values, such z'.
In Nisonoff et al. (2024), predictor guided rates (the analog to our D-CBG) are attained by scaling R2,0 as follows:
(1:L) z²) Poly 22 _4 (oly (1:L) z', z (1:L) = =z/) ) RA, (z (1:1))
(1:L)
R, (zg (1:b) z', (1:L) v)
22,2(1:11)
(44)
where Ps is an external classifier. Given the factorization assumptions of the forward and reverse processes, the reverse rate matrix in the CTMC formulation is only non-zero entries where its arguments differ in (at most) one dimension (Campbell et al., 2022; 2024; Nisonoff et al., 2024). Thus, the classifier only needs to be evaluated candidates for which latents differ in most one token, which coincides with the argument / classifier we introduce in Section 3.2. The formulation in (44) can be further simplified to (see Nisonoff et al. (2024) for details):
Ps 2 (1:2)
R7,001 (z) (1:L) = =2'22000 (1:L) 1
Br On (1:L) = ((1))
(45)
Poly
4
(1
=z')
where for each E, using the notation introduced in Section 3.2, we only evaluate ps on 2 (z(z) (1:1))
(1:L)
E
As in Appendix B.I, we present the continuous-time time extension of D-CBG and show that it is pro- portional to sampling using the reverse rate matrix in (45) (with Euler discretization for sampling).
Let R2 represent the rate matrix corresponding to transitions where all dimensions are kept the same except for potentially the one corresponding the hh token. We follow similar derivation to that used in Appendix B.1 and start with our formulation for D-CBG:
(1:L) .y) x (v (1::)))'poo(! |z( (1:2)) p"((2 (6) 2,
21

---

Published as a conference paper at ICLR 2025
where recall that (1:L) imate reverse rate to get:
2
(1:6-1)
2s 2,
(8) ((+1:2)]
We then plug in the formulation for the approx-
pz Zt
(1:2)
z1 x
r2,, 16, (1:11)) R (ze x'' 21 (1)T(
(1:L)
(1 ((z(112) (1:L)
n 2! (1:21) )) R (z, (1:11 = z 2! ((::1))(1/T)
( Poly (((:1))
)
blzz z 2z) R Zz z' -s z (1//T)
x
(1:4) zz)
Pyl Zz
(1 ö(z, 2 22) 22) (z) (2(11L)))) (1:L)
where the second proportionality comes from the fact that (z 2 (1:L)) = z is simply a constant. Therefore, have shown the equivalence between our D-CBG method and the predictor guidance method introduced in Nisonoff et al. (2024).
C RELATING UDLM TO CTMC
Similar to Sahoo et al. (2024a), our work tackles the problem of discrete diffusion from the vari- ational perspective and analyzes the ELBO in the continuous time limit. In contrast, other works, such as Campbell et al. (2022) and Lou et al. (2023), have extended the discrete diffusion framework proposed in Austin et al. (2021) using the formalisms of CTMC. In this section, relate these two approaches.
Forward Rate Matrices for Uniform Noise Recall from the analysis in Appendix A that we can use L'Hospital's rule to evaluate lim T (1 atja)) = -ak/ az. Now, using (21) from above, for z' + we have
Oflss
q(z: = 2o = 2) N
(46)
Combining this with (38), we have that:
1 atls
of
Rd (z,zz') lim T
(47)
T-+00
N
No
Now for z' = z, again from (21), we have
g(Zz = z Za = 2) = =Oole +
(48)
N
which combines with (38) to yield:
1- 1-00|= Or/s=
-1) 1)
1- N N
lim T (1 apps)) = -ay (1-N).
Rz(,, = lim T- atjs + T-000
N
T-00
Nat
(49)
Alternatively, for (49), we could have relied on the property that (z, = Ex R,(z, z').
Writing (47) and (49) as a single expression, gives:
0f Na.
Rz(2, 2')
111 NN].
(50)
22

---

Published as a conference paper at ICLR 2025
C.I EQUIVALENCE OF UDLM ELBO AND SEDD ELBO
Below, we demonstrate that the variational lower bound from (18) used to train UDLM is equiva- lent to the lower bound derived in SEDD (Lou et al., 2023), for specific parameterization of the denoising score matching.
facilitate the discussion, we introduce notational shorthand (x(2)) = q(2z = z²) and (z x) g(z z' x).
Notation
SEDD, the quantity of interest is the ratio probabilities the reverse rate matrix equation given in (40), and they train a parametric model to learn this so-called concrete score (Meng et al., 2022):
(z'))
8o (z) 2
(51)
2t. (z)
Since the unconditional marginals in this ratio are intractable, SEDD proposes tractable denoising score-based objective. Importantly, they show that the denoising score- based objective they use for training serves as variational bound and derive the following expression for Negative Evidence Lower Bound (NELBO):
NELBOsEDD
(=(z(x) ))
9.(2'|xx [x) (so(z)e
9: (:((2 (x)
Σ x' xta2i
Ri (z,z))
log so (z)) + +K
(52)
2 (z(x)
where K(a) a(log a - 1), for R+.
SEDD with Interpolating Uniform Noise From (1) we have gi (z|x) = a,x; + +1 where i is the non-zero index of the one-hot vector i... z, 1. Thus, the 'true' conditional score in (52) can be written as
gr 1z x
a1 + (1 at//N Na,x + (1 ar) O; + (1 o,N/N
(53)
9, xz)
No,x: + (1 o4)
where we use and j to denote the non-zero indices the one-hot vectors z and z', respectively.
Using Mean Parameterization in SEDD NELBO In our work, we use the mean parameteriza- tion: predict the 'clean' data given noisy observations using the model that denote xa- Note that (52) is minimized if
(2(2'1x qi q
50(z)
Thus, we can replace in (53) to extract score model from our parameterization:
(0(2)2 (x(()); + (1 ax)/N
Nor(xo); + (1 o4)
(54)
(xx)) ): + (1 ox)/N
Nai(xo); (1 ax.)
23

---

Published as a conference paper at ICLR 2025
We now show two useful identities that come from this parameterization. First,
Σ (xo) + (1 - ai)/N
Σ so (z)22 =
jæi (xx)) + (1-a/
22z2
or
1 ju (xo)م] + -1x((1--a)) az)/N
0n (x0): (1 +1-aa)/N
ax
ax jus (x0) (N - 1)/N
Σ 11=N N = Σ1=
(xx):: + (1- +11-a)/N
0, (xo):] + a1) (1-a))/N
-(xo), =1 = = 11 (xx);
(x(x)); + - ot)/N
ar + (1 a1))a) - (xx): - (1- -a://
(x(x)); + (1 /N
(x(x)): (1- /
(x(xa); + (1 11-aj) aa)
ax(x)); (1 ax)/N 1
N
(55)
NNr(xx); + (1 a1)
The same logic can be applied show that
9x(211x)
0x + (1 oxt)/N
N
Σ
1.
(56)
(z(
axx; + (1 at)/N Na + (1-a4)
1+1
24

---

Published as a conference paper at ICLR 2025
Equivalence Between NELBOsEDD and NELBOUDLM We can the identities from (55)
and (56) to demonstrate that NELBOsEDD (52) is equivalent to NELBOUDLM (18).
NELBOsEDD
(x'))x x) (0(())z
(x((2(x)
[0, z~ 1 (.
Σ Ro(z,z')
log (z)) + K
qe (z)x)
(((z/x)
z'#z
ae Nor
(x(z'(x)
z (.bx)
Σ
80(Z)2
(z(2)x)
log S8 (z)2r + K
2: (x(x)))
from (47)
x'
o'
4t (1(2(xx)
Σ
so(z)x
log logso(z)22
Σ K
Na
qi (z/xx)
(z(z)x)
z'zz
2' z'
2/522
Recall that ((a) = a log
a/z
(e(z'1x [x) x' (x(2(x)
q: (x'1xx
(e xx)
(x)
(x Nov
Σ so(z) +
log (z))
log
(x/xx
(z(xx)
2' (c(2)x)
z'z22
2'
Using (55) and (56) we get,
at
N <xN(xx); + (1-ar) +
N NxNx, + (1 - aa)
= Esc(o, ((x))))) Noa
(2(2(x)) x) 2e (z
1:(2'1x
(e(22) x)
log logso(z))
Σ
log
Le (z(x)
ge x)
z 22
2 #2
al = No
N
N
04 Nxx); ); + (1 - at) Nx + (1 - (4)
(z(2(xx) log 1 (=(2'(x)
(((z/x)
(z())z (( x)
Using (53) and (54) we get,
N
N
2 Not
Nx + - a1) Nxx( + (1 a4)
N, + (1 az)
(xxN(x)); + (1 an) NıNx, + (1 (1-aaz) oz)
log
aNNxx + (1 ar)
(x,V(xx); + (1 az) Nx (1 - az)
NELBOUDIM
D EXPERIMENTAL DETAILS
D.1 DATASET DETAILS
In this section, we provide more details, e.g., source, train/validation splits, etc., for the the datasets used in this work. For an overview, please see Table 8.
Species This dataset is composite of reference genomes from ten diverse species: Arahidopsis thaliana, Caenorhabditis elegans, Danio rerio, Drosophila melanogaster, Felis catus, Gallus gallus, Gorilla gorilla, Homo sapiens, Mus musculus, and Salmo trutta, which were downloaded from NCBI refseq database (O'Leary al., 2016). In Table 9, provide the assembly accession IDs used to download the reference genomes. Genomes were chunked into non-overlapping segments of 32,768 nucleotides and were tokenized using leir tokenization.
Training and validation sets were randomly split using 95% 5%. In Table 10, we specify the size and relative species composition of each split. For guidance, we use the species label. This
25

---

Published as a conference paper at ICLR 2025
Table 8: Relevant details for datasets used in this work.
Dataset
Tokenizer
Vocab. Size
Input size
Padding?
Species10
Base-pair
32,768
No
QM9
Regex (Schwaller et al., 2019)
40
32
Yes
CIFARIO
Binned Pixel Intensity
256
32x32x3
No
tex18
Character
35
256
No
Amazon
bert tert-base-ae uncased
30,522
128
Yes
LMIB
bert-base-uncased
30,522
128
Yes
Table Species genomes accession IDs
Species
Assembly Accession
Arabidopsis thaliana
GCF-000001735. 4.TAIR1O
Caenorhabditis elegans
GCF 000002985. 6 WBcel235
Danio rerio
GCF.000002035. GRCC211
Drosophila melanogaster
Felis catus
Gallus gallus
GCF GFF116669PB5522aaaaaalaann......
7b
Gorilla gorilla
GEL:3328656522201111100222222222
Homo sapiens
GCF 6GE:000001105-44.cc013........1
Mus musculus
GCF 100000016655.2..RRmm..3
Salmo trulla
GCF 1001000165511f$1111111
dataset is made available here: httpps///anggiggeee....o/tttttta ten_species.
QM9 Molecules The QM9 dataset comes from Ruddigkeit et al. (2012) and Ramakrishnan et al. (2014). The dataset comprised of 1333 small molecules. We process the dataset using the RDKit library (Landrum et al., 2013) extract 'canonical' SMILES string representations and add the annotations for likgnsis (QED) and ring-counts. The data are tokenized using a regular expression from Schwaller et al. (2019), which is available here: https: //t:///uuuu ///ugggngggi face co/ yairschi f gttmtmm9ttttt token zer. We used input lengths of 32 tokens (with right-sided padding) for pre-training and generation experiments.
Training and validation sets were randomly split using 95% / 5%. For guidance, we use a cutoff of 90th percentile for QED / ring count generate binary labels. This dataset is made available here: https: pettps:/huugeee http//hggggggggccccec/o co/ datasetsssssssssooo.ooo /aaiis//aaaaeeaaa irachrschh /fgg99
CIFARI0
This widely-used image dataset contains RGB images with 32 32 pixels per channel.
We use the provided train and test splits of 50,000 training images and 10,000 validation images. The dataset contains 10 classes roughly equal proportion. We tokenize the dataset by rounding to integer pixel intensity values in the range [0, 255). For guidance, we use the image class label.
The dataset was downloaded from /est:://mattaaaiiie Data was tokenized at the character level using the lower case letters ['a' 'z'] and white-space character. The data was broken into non-overlapping chunks of 256 tokens.
text8
- net/de/teee /ede//eext... zip.
The first 90M characters were used for the training set and the final 5M characters were used validation set.
dataset was downloaded from https: //huggingfacee.cccmm co/ dat datasets/Fancyzh. amazon polarity. We tokenize ing the saart-base-uncar tokenizer. Sequences were padded to a max input length of 128 tokens.
Amazon Review The Amazon
Review
26

---

Published as a conference paper at ICLR 2025
Table 10: Species10 train and validation splits statistics. Proportion of each species in the training and validation sets. Overall split size is indicated in parentheses in the column header.
Species
Train
Validation
(95%
16.5B bps)
(5% 869M bps)
Arabidopsis thaliana
0.68
0.77
Caenorhabditis elegans
0.58
0.58
Danio rerio
9.50
9.29
Drosophila melanogaster
0.08
0.73
Felis catus
13,94
14.05
Gallus gallus
6.04
5.93
Gorilla gorilla
20.40
20.19
Homo sapiens
18.89
19.22
Mus musculus
15.69
15,60
Salmo trutta
13.49
13.64
Train and validation splits were used from the downloaded data, with 3.6 M sequences in the training data and 400k sequences in the validation set.
LMIB This dataset downloaded from https: wtw.t/ggginaac..oooooooooo.ooooo
billion-wee mooo-woord wenn-word-beehheeee www.bhanccchceeeeeeeeeeeeee /mmbb. We tokenize using the bert -ast---asseusssee tok- enizer. Sequences were padded to max input length of 128 tokens.
We use the train and validation splits provided in the downloaded data. After chunking the data, our training set consisted of 7M sequences and our validation set consisted of 72k sequences.
D.2 ARCHITECTURAL DETAILS
In Table 11, we provide an overview of the architectures and parameter counts of the models used for each dataset.
Table 11: Architectures used when training various datasets.
Dataset
Architecture
Parameter Count
Species10
Mamba (AR) Caduceus (Diffusion)
3.5 M (AR) 4.8 M (Diffusion)
QM9
Transformer
92.4 M
CIFARIO
UNet
35.8 M
Text8
Transformer
92.4 M
Amazon
Transformer
139
LM1B
Transformer
139 M
Genomics Caduceus For the Species 10 experiments, we use Mamba-based models (Gu & Dao, 2023). The AR model is standard pext-token-predictin Mamba backbone with 8 blocks and hidden dimension 256. For the diffusion models we train with a variant of the Caduceus architecture from Schiff et al. (2024), also with 8 blocks and hidden dimension 256. Specifically, non-reverse complementary equivariant version of Caduceus, dubbed Caduceus- in Schiff et al. (2024), which is similar the AR Mamba, but with bi-directional Mamba blocks that use strategic weight tying to limit the parameter count: 3.5 M parameters of AR vs. 4.8 M for diffusion models.
the
CIFARIO UNet We adopt the UNet (Ronneberger et al., 2015) as a main backbone for both MDLM and UDLM, following (Ho et al., 2020). Network configurations are presented in Table 12. Specifically, we follow Austin et al. (2021) and Campbell et al. (2022) and use the original UNet backbone from DDPM (Ho et al., 2020), adding extra discretized truncated logistic transforma- tion to network outputs. To enable conditioning on labels, we add a label embedding layer which is added to the time embeddings, inspired by ADM network (Dhariwal & Nichol, 2021b).
27

---

Published as a conference paper at ICLR 2025
Table 12: Architecture details for CIFARIO UNet.
MDLM
UDLM
Vocab size
258
256
Number of ResNet blocks per scale
2
2
Base channels
128
128
Channel multiplier per scale
(1,2.2,2)
(1,2,2,2)
Attention resolutions
16
Time conditioning
False
True
Conditional embedding dimension
128
128
Number of Params
35.8M
35.8M
QM9 and NLP Transformer For both the QM9 and NLP datasets (Amazon, text8, and LMIB), we use the same Diffusion Transformer (Peebles & Xie, 2023) with adaLN for conditioning on time (for uniform noise diffusion models) and class (for guided generation). The Transformer for each dataset differs only in the word-embedding size, which is determined by the vocabulary size of the datasets' corresponding tokenizer. Our Transformer consists of 12 layers, hidden dimension of 768, and 12 attention heads. We use RoPE (Su et al., 2021) as the positional embeddings.
D.3 TRAINING CONFIURRATIONS
In Table 13, we detail the hyperparameter setup each of the language modeling experiments in Section 5.1.
All diffusion models were trained and evaluated using log-linear noise schedule.
Of note, for CIFAR10, our models are trained for 300K iterations as opposed to 1.5M iterations, as D3PM (Austin et al., 2021).
Table 13: Training hyper-parameters for all included experiments.
Species10
QM9
CIFARIO
text8
Amazon
LMIB
Train
30K
25K
300K
1000K
184K
1000K
steps
Context
32,768
32
32x32x3
256
128
128
size
Batch
32
2048
512
$12
512
$12
size
LR
2e
3e-4
2e-4 ADAM (0.9, 0.999)
3e-4
3e-4
3e-4
ADAM (0.9, 0.999) Cosine decay 3e 36-6 min.
ADAM (0.9, 0,999)
ADAM (0.9, 0,999)
ADAM
ADAM
Optim.
(0.9, 0.999)
(0.9, 0.999)
LR
Cosine decay
sched.
3e-6 min.
LR warmup
3K
IK
SK
2.5K
2.5K
2.5K
steps
GPU
4
8
count
GPU
A,5000
A6000
A100
A100
A,500
A100
type
D.4
GUIDANCE DETAILS
D.4.1 BASELINES
FUDGE Implementation FUDGE is a classifier-based autoregressive guidance method proposed by Yang & Klein (2021). FUDGE first trains a classifier on all possible prefixes. Instead of di- rectly sampling from ((x(0) x(1:611), FUDGE samples from the perturbed conditional distribu-
tion p(y x((:E) ((x(() x((:t11) where y denotes the classifier label, (1::-1) denotes the already
28

---

Published as a conference paper at ICLR 2025
decoded tokens and x stands for possible token to be generated. For efficiency, FUDGE truncates x4) to the lopk tokens with the largest unconditional log-likelihood. Although not in the origi- nal FUDGE formulation, following the classifier-based guidance in diffusion models (Dhariwal & Nichol (2021b)), we introduce the temperature variable and sample from the perturbed distribu- tion:
p(x(()) (111-1))""( | x(:)))
Exte p(x(() x((ee-1)) (p x((:))
In our QM9 experiments, lopk is set as the vocabulary size, which is 40. When training the classifier, use smaller classification model using the same dataset specified in Appendix D.1. The smaller backbone DiT consists of 8 layers, 8 attention heads, and hidden dimension of 512. We use the same hyperparameters those specified in Table 13.
PPLM Implementation Inspired by the approximate Metropolis-adjusted Langevin (MALA), Plug and Play language model (PPLM; Dathathri et al. (2019)) introduces guidance by conduct- ing gradient updates on the hidden representations of a language model. PPLM first rewrites the autoregressive language models' decoding process using the KV-cache mechanism (Wolf et al., 2020): X,+1, He+1 = LM(x,, Hr), where X, denotes the token at time step and H, is defined as [(K!") (())) ) (Kg v(() in which is the number of layers. When decoding the token Xr+-1 at time step l + I, PPLM initializes the perturbation term AH, as O, and then performs n updates, following the gradient ascend formula:
VAH, Dog(ply He + AH)) - DKLDDL(
H. + (H)))l((x+ H.))
AH, + AH,
| VAH (pog((()
H, + AH,)) 7K1. DI. (p
H, + AH,) p(x++1 H:))
PPLM then generates the perturbed probability distribution Pperb (x++1 HH) using HHH
and decodes X using the fused probability distribution perer (X++1 H, + (H) Pumpert 1- (x++1
"m
Hj), where 8 is the normalization factor. Following Dathathri et al. (2019), in experiments YNu is set as 0.01 and 7ym is set as 0.95. and n. are decided by grid search.
For the QM9 experiments, when training the classifier for PPLM, use the same dataset specified in Appendix D.I. We use the same hyperparameters as those specified in Table 13, except the LR peak is reduced 3e and minimum is reduced to -7
NOS Implementation To implement the NOS baseline from Gruver et al. (2024), we train a classifier where we use the same backbone the unconditional diffusion model (see Appendix D.2 for architecture details), and we initialize and freeze weights using the pre-trained unconditional diffusion model. We then mean pool the last hidden embeddings of the backbone and linearly project them to the classification logits. This final projection layer represents the only trainable parameters of the guidance model.
At inference, we perform Langevin sampling following Algorithm from Gruver et al. (2024). In experiments, we denote step-size for the Langevin sampling by 17, the number of steps by and the weight on the 'stability" 'fluency' KL penalty by "/kt-
For the QM9 experiments, when training the classifier for NOS, in Appendix D.I. We use the same hyperparameters as those specified in Table 13, except the LR peak is reduced 35-5 and minimum is reduced to -7
the same dataset specified
D.4.2 D-CFG DETAILS
When implementing D-CFG we train single AR/ / discrete diffusion model (see Appendix D.2 for architecture details) where we randomly drop out the class condition by replacing it with a class [MASK] token. The class condition is fused into models using the implementation of adaptive layer norm from Peebles & Xie (2023). We use 10% rate for masking / dropping-out the condition.
At inference time, we perform two forward passes through the model, with condition provided to compute the conditional probability Po / 2,, y) and one where the condition is masked to compute the unconditional probability pz 2.). These values are then used as described in Section 3.1.
29

---

Published as a conference paper at ICLR 2025
D.4.3 D-CBG DETAILS
For D-CBG in the QM9 experiment, we train a smaller classification model using the same dataset specified in Appendix D.I. The smaller backbone DiT consists of layers, 8 attention heads, and a hidden dimension of 512. We apply mean pooling on the final hidden representations before linearly projecting the classification logits. We use the same hyperparameters as those specified in Table 13.
We train the model on noised inputs using a log-linear schedule, where the type of corruption applied corresponds to the diffusion model to which guidance applied.
D.5 GUIDANCE EVALUATION DETAILS
D.5.1 GENOMIC SEQUENCES METRICS
Below we describe the quality and control metrics used in the species-specifie genome generation experiment.
k-mer JS To compute the k-mer distirbution shift, for each species, we create counts for each of the unique 3mers and 6mers for that species in the validation set and 64 generated sequences for that species. We then compute the Saasen-Shannon divergence between those categorical histograms. Finally we take weighted average of these distances across species where the weights are given by the relative species proportion in the validation dataset. See Table 10 for the relative proportions.
Discrininator AUROC For the diseriminator AUROC metric, train HyenaDNNA model with 2 layers and hidden dimension 128. This model was downloaded from https: // huggingface, , co/ LongSafari saaffrr/hyeeadnaa small 3----32---ees- fark-seqlen-Be hε, modified (we re- duced the number of layers and hidden dimension), and initialized from scratch. We mean pool the final layer embeddings and linearly project them to the binary classification logits. The model is trained on the 640 generated sequences, which are labeled as the negative class, and 640 ran- domly selected sequences from the ground truth validation set (64 sequences per species), which are labeled as the positive class. This dataset of 1,280 sequences is randomly split into 95% train and 5% validation. The discriminator is trained with batch size of 8, learning rate of 1e -4, and the ADAM optimizer for 5 epochs to minimize binary cross entropy loss the classification of real vs. generated sequences. We report the AUROC from the final epoch on the 5% validation split of this classification dataset.
Oracle F1 Finally, controllability measured by the macro-averaged Fl of an oracle model on the 640 generated sequences. Our oracle model is a separate HyenaDNA model with 8 layers and hidden dimension 256, which has 6.6M parameters. This model was downloaded and initialized from scratch from https:
hyenadna-an small 1 1111- 11-33k-ssee 552k-seqlend en We mean pool the final layer embeddings and linearly project them to the ten category classification logits. This model was trained the full Species 10 dataset as described in Appendix D.1. For reference, in Table 14 we present the classification results of the this oracle model on the 5% valdiation set of the original data, We see that other than difficulty distinguishing between human and gorilla genomes, the model can serve as a near perfect oracle.
D.5.2 CIFARIO QUALITY METRICS
For evaluation, we randomly samples 50,000 images for each model and the tools provided here: https: :tops://gti //::://gtttt com ww8 67 637777777/7 (11.7777/yyyo// git, as described in Campbell et al. (2022).
FID Fréchet inception distance (Heusel et al., 2017) is a common metric in image generation where the divergence between real and generated data is measured to reflect the alignment of two distributions. The metric uses features extracted from a pretrained Inception-v3 model on ImageNet- IK to estimate the mean and variance of the input data. The difference of two multi-dimensionall Gaussian distributions is measured by Wasserstein-2 distance Fréchet distance d(.) follows:
30

---

Published as a conference paper at ICLR 2025
Table 14: Evaluation of HyenaDNA 'oracle' classifier on Species 10 validation split.
Species
Precision
Recall
F1
Arabidopsis thaliana
1.00
0.99
1.00
Cuenorhabditis elegans
1.00
1.00
1.00
Danio rerio
1.00
1.00
1.00
Drosophila melanogaster
1.00
1.00
1.00
Felis catus
1.00
1.00
1.00
Gallus gallus
1.00
1.00
1.00
Gorilla gorilla
0.63
0.45
0.52
Homo sapiens
0.54
0.72
0.62
Mus musculus
1.00
0.97
0.98
Salmo trutta
1.00
0.98
0.99
FID ((NNNWWWaa Ereat), Esake))
Mreal Msakell 1|+ [[Tr(Ereal + Ejake 2(Ereal ).b)))
(57)
IS Inception Score (Salimans al., 2016) an alternative measure of how well generated images are aligned with human judgement. IS also utilizes Inception-v3 model to compute label distribution p(y|æ) for each generated image. IS focuses on two criteria: (1) A generated image should contain distinct class object, meaning its label distribution is expected to have low entropy; (2) The generated
images should vary across multiple classes, so the marginal distribution, p(y) = S. p(y])(()))))) is expected to have high entropy, ideally approaching uniform distribution. The formula is presented as below:
IS = exp E. (Lp(lpp))))))
(58)
F1 F1 is used proxy for satisfying the desired conditional generation of gener- ated samples by pre-trained classifier on CIFAR10. We use pretrained Vision Trans- former model downloaded from https: pettps:/huug.ie htt/g/hgggaaace//e/ee/ee/o co wtc///addaattt//vvt.bboooo patch16_228 ittlk_nt_ciiii_____________
and fine-tuned on CIFARIO.
D.5.3 QM9 GUIDANCE METRICS
For the guidance experiments in QM9, we generate 2,048 sequences of length 32. The reported metrics explained below.
Validity Validity is measured by whether the generated SMILES string can be parsed by the RDKit
library (Landrum al., 2013). Any strings that fail to be parsed are counted invalid.
Novelty Novelty is measured by the number of valid and unique sequences that are not present in the original QM9 dataset.
Property Mean Median Finally, we also report the mean and median of the novel generated sequence for the property of interest, QED ring count. These quantities are computed using the RDKit library.
E GUIDANCE ABLATION RESULTS
EFFECT OF VARYING GUIDANCE HYPERPARAMETERS
For each of the guidance experiments, perform hyerparameter search the guidance param- eters, e.g., y in D-CFG and D-CBG. Below we present results from these searches for the various guidance experiments.
31

---

Published as a conference paper at ICLR 2025
Species-specifie Genome Generation In this experiment vary Y (1,2,3} for D-CFG ap- plied to AR, MDLM, and UDLM. Additionally, for MDLM and UDLM we vary the number of sampling steps T {128, (566,12).
Results presented Table 15 highlight that UDLM is more amenable to guidance than either AR MDLM in this setting. UDLM achieves better quality and control metrics, and AR and MDLM performance degrades as y, this is not the case for UDLM.
Table 15: Varying for species-specift generation with AR, MDLM, UDLM using D-CFG, Mean + standard deviation reported from repeated generation of 640 sequences (64 per species) using five different random seeds.
3-mer JS (1)
6-mer JS (J)
Disc.
D-CFG, T
F1 (f)
AUROC (L)
AR
32,768
0.03 0.022
0.07 0.004
0.53 0.075
0.87 0.011
32,768
0.05 + 0.003
0.12 = 0.004
0,90 + 0.051
0.81 + 0.009
32,768
0.07 1 0.002
0.15 = 0.002
0.97
0.74 0.022
MDLM
128
0.02 = 0.001
0.06 = 0.001
0.51
0.88 + 0.01
256
0,02 # 0.002
0.06 0.003
0.55 + 0.039
0.88 + 0.006
512
0.02 = 0.001
0.06 = 0.001
0.51 0.059
0.88 0.01
128
0.06 + 0.011
0.11 0.003
0.71 + 0.013
0.90 0.011
256
0.05 + 0.001
0.10 = 0.002
0.77 0.027
0.91 0.09
512
0.05 + 0.001
0.11 0.003
0.74 0,037
0.91 0.09
128
0.12 + 0.002
0.20 = 0.003
0.94 1 0.031
0.78 + 0.009
256
0.11 0.002
0.19 0.005
0.91 0.025
0.78 0.01
512
0.11 + 0.004
0.20 = 0.005
0.93 + 0.031
0.78 + 0.007
UDLM
128
0.02 + 0.001
0.05 = 0.001
0.55 1: 0.041
0.90 + 0.006
256
0.02 1 0.002
0.06 = 0.002
0.56
0.91 0.006
512
0.02 # 0.002
0.06 0.003
0.52 + 0.044
0.91 + 0.01
128
0.05 + 0.003
0.12 =
0.52
0.92
256
0,05 + 0.002
0.13 0.004
0.59 + 0.043
0,91 + 0.005
512
0.05 = 0.002
0.13 = 0.002
0,61 0.043
0.93 0.009
128
0.08 1 0.003
0.19
0.004
0.80 0.048
0.93 1 0.009
256
0.08 1 0.002
0.20 0.004
0.81 0.084
0.92 1 0.005
512
0.08 = 0.002
0.20 0.003
0.87 0.051
0.94 0.007
QM9 In this section, we list the hyperparameter grid search results for QM9 drug likeliness (QED)
maximization using
AR D-CFG (Table 16), AR FUDGE (Table 17), AR PPLM (Table 18),
MDLM D-CFG (Table 19), MDLM D-CBG (Table 20), MDLM NOS (Table 21),
UDLM D-CFG (Table 22), UDLM D-CBG (Table 23), and UDLM NOS (Table 24).
We also list the grid search results for QM9 ring count maximization guidance results with
AR D-CFG (Table 25), AR FUDGE (Table 26) AR PPLM (Table 27),
MDLM D-CFG (Table 28) MDLM D-CBG (Table 29), MDLM NOS (Table 30),
UDLM D-CFG (Table 31), UDLM D-CBG (Table 32), UDLM NOS (Table 33).
CIFAR10 In Table 34, we explore the effect of y E 1, 3, 4,5 conditional image generation. For this table, for both MDLM and UDLM. For FID, MDLM achieves better scores as y increases. However, the impact of 7 is weaker for UDLM, the model's 's FID score worsens when exceeds 2. We also plot
T 1000. We find that increasing ? generally leads to better IS and F1 scores
32

---

Published as a conference paper at ICLR 2025
Varying y for maximizing drug-likeness (QED) guidance with AR D-CFG. Validity, novelty, and mean QED for novel sequences are reported. Mean + standard deviation reported from repeated generation of 1,024 sequences using five different random seeds. The setting reported in the main paper is bolded.
Table 16:
7
Num. Valid (t)
Num. Novel (1)
QED Mean (t)
1
1013.0 4.9
82.6 1: 2,97
0.57 1: 0.00
2
1001.4 5.37
79.2 9.34
0.59 i
3
946.4 + 8.99
79.4 +
0.60 + 0.00
4
777.2
15.39
86.8 8.73
0.59 +
5
591.8 + 10.43
77.8 9.12
0.58 + 0.00
Table 17: Varying "y for maximizing drug-likenee (QED) guidance with AR FUDGE. Validity, novelty, and mean QED for novel sequences reported. Mean + standard deviation reported from repeated generation of 1,024 sequences using five different random seeds. The setting reported in the main paper bolded.
Num. Valid (1)
Num. Novel (1)
QED Mean (t)
1
998.4 + 3.21
113.0 7.18
0.56 +:2 0.00
2
985.2 +: 6.46
88.8 + 5.5
0.58 +: 0.01
3
973.8 4.7
90.8 7.66
0.59 0.00
4
963.4 + 8.53
86.4 14.05
0.59 + 0.00
5
949.6
74.6 1
0.60 1 0.00
6
935.0 1 9.33
64.6 1 7.09
0.60 1 0.00
7
924.4
53.0 =
0.61 +
8
902.8 16.08
46.6 = 4.04
0,62 + 0.00
9
876.6 13.03
34.8 + 3.63
0.62 + 0.01
10
873.8 11:21
29.2 # 2.77
0.62 + 0.01
visual outputs of different in Figure 4. As increased, the appearance and the shape of class object become more refined and sharpened.
E.2 EFFECT OF USING FIRST ORDER APPROXIMATION IN D-CBG
The data for this analysis is available in Tables 20, 23, 29, and 32. We present it graphically in Figure 5. In these plots, we use T = 32 for diffusion generation.
QED We see that for drug-likeness (QED) maximization, using the full D-CBG without the first order approximation leads to a boost in performance for both MDLM and UDLM.
Ring Count When maximizing ring count, we again find that for UDLM, not relying on the first order approximation improves performance with both more novel molecule generation and higher ring counts. For MDLM, however, we observe instability for 7 > as the model tends to decode the full sequence early on in the generation process and cannot recover from mistakes. The first order approximation for MDLM does not appear to suffer from this same instability.
E.3 EFFECT OF VARYING T
Species10 In Table 15, we see the effect of increasing on the genomic sequence generation. In this setting T' E {128, 256, 512} is orders of magnitude smaller than sequence length L = 32768. We see a positive relationship between decoding steps and sample quality for both MDLM and UDLM.
QM9 In Figure 6, we see that for both properties (QED and ring count), MDLM benefits signifi- cantly from increasing T. In contrast, UDLM appears to have more consistent results across varying
33

---

Published as a conference paper at ICLR 2025
Table 18: Varying n (the number of Langevin steps) and r (the step size) for maximizing drug- likeness (QED) guidance with AR PPLM. Validity, novelty, and mean QED for novel sequences are measured for generated sequences from each method. Mean + standard deviation reported from repeated generation of 1,024 sequences using five different random seeds. The setting reported in main paper bolded.
Num. Valid (1)
Num. Novel (1)
Novel QED Mean (1)
10
0.040
1009.6 1:
140.4
0.46
10
0.100
1008.8 = 2.17
138.4 = 17:24
0,46 + 0.01
30
0.040
1008.8 =
137.8
0.45 0.00
30
0.100
1007.2 = 2.28
142.0 + 14.2
0.45 0.00
Table 19: Varying for maximizing drug-likeness (QED) guidance with MDLM D-CFG. Validity, novelty, and mean QED for novel sequences are reported. Mean + standard deviation reported from repeated generation of 1,024 sequences using five different random seeds. The setting reported in the main paper bolded.
Num. Valid (1)
Num. Novel (t)
Novel QED Mean (1)
561.0 = 6.08
182.8 10.52
0.56 + 0.00
449.8 8 + 8.7
142.8 1: 8.47
0.59 + 0.0
317.4 + 11.5
95.8 :: 9.04
0.60 +: 0.01
226.0 + 10.34
79.0 + 6.36
0.59 + 0.01
163.4 # 7.86
68.4 + 3.58
0.59 + 0.01
number of decoding steps. Note that for the D-CBG results in this section do not the first order approximation for maximizing QED and we do use it for maximizing ring count.
CIFAR10 In Table 35, we present additional T beyond those presented in Table 7 in Section 5.2. As discussed in the main paper results, we find that UDLM is able to accommodate this faster inference setting better than MDLM, owing to the ability of UDLM to recover from 'mistakes."
F UNCONDITIONAL SAMPLE GENERATION
In this section, we evaluate unconditional samples generated from models trained on LMIB. In Table 36, report generative perplexity for 1,024 unconditionally generated sequences. use GPT-2 Large (Radford et al., 2019) downloaded from https: http///huggga/////ee/eo co/ openai- wwwmunimmmnn..o..mmmmm.m htty//pt22222 -arge to compute generative perplexity. In Table 37, present random examples of sample generation.
34

---

Published as a conference paper at ICLR 2025
Table 20: Varying for maximizing drug-likeness (QED) guidance with MDLM D-CBG. We also report results for both using the first order approximation and not using Validity, novelty, and mean QED for novel sequences are reported. Mean + standard deviation reported from repeated generation of 1,024 sequences using five different random seeds. The setting reported in the main paper is bolded.
Use Approx.
Num. Valid (t)
Num. Novel (1)
Novel QED Mcan (f)
False
526.4 + 16.53
170.8 +: 14.69
0.56 :: 0.00
True
476.0 13.25
221.4 7.57
0,46 =
2
False
524.8 22.04
139.0 = 11.79
0.58 = 0.00
True
363.4 = 17.57
172.4 = 9.42
0.47 = 0.58 + 0,00
3
False
417.6 19.69
116.6 8.91
True
244.0 13.38
114.0 + 6.16
0.47 =
4
False
200.4 = 10.31
66.4 7.7
0.58 = 0.00
True
174.6 = 11.26
83.6 3.51
0.49 = 0.02
5
False
24.6 3.21
11.6 23
0.58
True
120.6 = 6.69
48,4 5
0.50 = 0.01
False
0.2
0.2 0.45
0.12 1 0.50 1 0.02
True
96.4 + 4.04
35.0 5.24
False
0.0 1
0.0 0.00
1 0.00
True
85.4 = 9.81
28.6 5.32
0,51 = 0.01
False
0.0 +
0.00 +
0.00 = 0.55 = 0.01
8
True
77.0 = 8.63
24.8 + 6.76
False
0.0 + 0.00
0,0 +
0.00 = 0.55 1 0.01
True
71.2 10.94
21.6 6.73
False
0.0 1: 0.00
0.0 1 0.00
0.00 1 0.00
10
True
72.6 7.13
21.2 + 1.79
0.55 0.02
7
7 7=2
7 = 3
7 =
MDLM
UDLM
Figure 4: Illustration of varying y on CIFAR10.
35

---

Published as a conference paper at ICLR 2025
Table 21: Varying (the number of Langevin steps), (the step size), and Thu (the stability reg- ularization coefficient) for maximizing drug-likeness (QED) guidance with MDLM NOS. Validity, novelty, and mean ring count for novel sequences are measured for generated sequences from each method. Mean standard deviation reported from repeated generation of 1,024 sequences using five different random seeds. The setting reported in the main paper is bolded.
7
9)
7kl
Num. Valid (1)
Num. Novel (t)
Novel QED Mean (t)
0.001
0.000
509.4 + 23.78
238.2 + 8.87
0.45 + 0.00
0.001
0,001
509.4 + 23.78
238.2 8.87
0,45 =
0.001
0.010
509.4 1: 23.78
238.2 1 8.87
0.45 1 0.00
0.010
0.000
510.00 1: 24.24
238.8 7.82
0.45 1:
0.010
0.001
509.4 + 24.15
238.6 7.77
0,45 = 0.00
0.010
0.010
509.4 + 24.15
238.6 7.77
0.45 +
0.100
0.000
507.4 + 29.98
240.2 + 14.82
0.45 + 0.00
0.100
0.001
507 4 + 29.98
240.2 + 14.82
0.45 +
0.100
0.010
507. 4 + 29.98
240.2 + 14.82
0,45 + 0.00
1.000
0.000
461.0 +: 29.21
233.2 + 15.16
0.44 1 0.00
1.000
0.001
461.0 1 29.21
233.2 1. 15.16
0.44 1 0.00
1.000
0.010
461.0 + 29.21
233.2 + 15.16
0.44 = 0.00
5.000
0.000
185.8 8.23
127.0 = 8.25
0.42 = 0.01
5.000
0.001
185.8 823
127.0 1: 8.25
0.42 1 0.01
5.000
0.010
185.8 823
127.0 8.25
0.42 :
0.001
0.000
509.4 24.6
237.0 9.67
0.45 : 0.00
0.001
0.001
509.4 24.67
237.0
0.45 +
0.001
0.010
509.4 24.7
237.0 9.67
0.45 + 0.00
0.010
0.000
509.2 26.2
238.4
0.45 +
5
0.010
0.001
509.2 26.2
238.4 8.79
0,45 = 0.01
0.010
0.010
509.2 26.2
238.4 8.79
0.45 1 0.01
5
0.100
0.000
505.6 1 29.52
239.8 1: 16.36
0.45 1 0.01
1.100
0.001
505.6 + 29.52
239.8 + 16.36
0.45 + 0.01
5
0.100
0.010
506.0 + 29.45
240.4 + 16.01
0.45 = 0.01
1.000
0.000
461.0 29.11
233.2 + 15.16
0.44 + 0.00
5
1.000
0.001
461.6
233.2 + 14.77
0.44 =
1.000
0.010
459.8 28.6
231.2 + 13:72
0.44 + 0.00
5.000
0.000
185.8
127.0 8.25
0.42 1
5.000
0.001
186.2 8.98
127.2 1 8.79
0.42 1 0.01
5.000
0.010
186.0
127.0 9.46
0.42 +
10
0.001
0.000
509.6 + 25.58
238.0 9.77
0,45 + 0.00
10
0.001
0.001
509.0 25
237.4 9.56
0.45 +
10
0.001
0.010
509.0
237.4 + 9.56
0.45 + 0.00
10
0.010
0.000
508.2
238.4 + 11.08
0.45 +
10
0.010
0.001
508.2 25.77 25.77
238.4 1: 11.08
0.45 1 0.00
0.010
0.010
508.2 1 25.77
238.4 1: 11.08
0.45 1 0.00
10
0.100
0.000
504.2 2675
238.2 + 14.81
0,45
10
1.000
0.001
505.0 + 26.84
238,8 + 15:25
0.45 + 0.01
10
0.100
0.010
504.2 1: 26.75
238.2 1 14.11
0.45 1:2
10
1.000
0.000
461.0 29.1
233.2 + 15.16
0.44 1 0.00
10
1.000
0.001
461.0 28.31
232.8 + 231.8 + 12.99
0.44 +
10
1.000
0.010
460.6 2811
0.45 + 0.00
10
5.000
0.000
185.8
127.0 8.25
0.42 001
10
5,000
0.001
186.4 7.92
127.4 + 7.96
0.42 + 0.01
10
5.000
0.010
190.4 + 10.48
130.4 9.63
0.42 + 0.01
36

---

Published as a conference paper at ICLR 2025
Table 22: Varying for maximizing drug-likeness (QED) guidance with UDLM D-CFG. Validity, novelty, and mean QED for novel sequences are reported. Mean + standard deviation reported from repeated generation of 1,024 sequences using five different random seeds. The setting reported in the main paper is bolded.
Y Num. Valid (1)
Num. Novel (1)
Novel QED Mean (1)
1003.4 i 5.46
82.0 15.28
0.57 1:2 0.00
1017.6 1.82
64.2 + 3.96
0,60 0.00
1017.2 = 0.84
61.2 + 7.19
0.61 + 0.00
4
1019.0 1.41
61.6 9.13
0.61 + 0.00
1013.6 2.51
64.0 5.1
0.62 + 0.00
Table 23: Varying for maximizing drug-likeness (QED) guidance with UDLM D-CBG. We also report results for both using the first order approximation and not using Validity, novelty, and mean QED for novel sequences are reported. Mean + standard deviation reported from repeated generation of 1,024 sequences using five different random seeds. The setting reported in the main paper is bolded.
Use Approx.
Num. Valid (f)
Num. Novel (t)
Novel QED Mean (t)
False
933.4 7.5
134.6 + 7.33
0.53 +:: 0.01
True
996.4 5.86
132.2 =
0.47 =
False
911.2 i 8.14
119.8 +\2 16.47
0.57 = 0.01
True
974.8
110.6
0.49 +::
False
941.0 11.31
96.4 8.82
0.58 = 0.01
True
925.0 =
111.4 = 9.56
0.51 = 0.59 1 0.00
4
False
961.8 i 7.98
77.0 7.55
True
866.4 = 17.67
93.8 10.28
0.52 1 0.00
5
False
967.6 1.52
77.8 1083
0.59 0.01
True
828.2 = 17.85
86.2 + 1301
0.53 = 0.01
False
976.2 7.92
78.6
0,60
True
821.0 = 11.22
79.8 + 7.19
0.54 = 0.00
False
982.4 3.21
73.8 + 6.91
0,60
True
798.6 1 5.64
82.2 1: 7.66
0.55 1 0.00
8
False
986.4 4.98
66.6 1: 8.53
0.61 1
8
True
789.8 15.09
69.2 + 6.76
0.55 = 0.01
False
992.8 6.3
68.0 +
= 0.61
True
798.4 = 11.06
68.2 + 10.5
0.57 - 0.01
False
994.8 +: 2.66
63.8 +
0.61 - 0.00
10
True
783.2 13.88
67.0 + 3.08
0.56 + 0.01
37

---

Published as a conference paper at ICLR 2025
Table 24: Varying n (the number of Langevin steps), (the step size), and "hu (the stability reg- ularization coefficient) for maximizing drug-likeness (QED) guidance with UDLM NOS. Validity, novelty, and mean ring count for novel sequences are measured for generated sequences from each method. Mean standard deviation reported from repeated generation of 1,024 sequences using five different random seeds. The setting reported used in the main paper is bolded.
7
9)
7kl
Num. Valid (1)
Num. Novel (f)
Novel QED Mean (t)
0.001
0.000
1000.8 = 2.77
145.6 + 8.44
0.46 + 0.00
0.001
0,001
1000.8 2:77
145.6 8.44
0,46
0.001
0.010
1000.8 1 2.77
145.8 1 8.04
0.46 1: 0.00
0.010
0.000
1001.0 3.0
147.0 7.81
0.46 1
0.010
0.001
1001.0 + 3.0
147.0 7.81
0,46 = 0.01
0.010
0.010
1001.0
147.0 + 7.11
0.46 +
0.100
0.000
999.2 4.6
146.4 + 6.88
0.46 + 0.01
0.100
0.001
999.2 4.6
146.4 6.88
0.46 +
0.100
0.010
999.2 446
146.4 = 6.88
0.46 + 0.01
1.000
0.000
992.2 = 563
152.0 7.2
0.46 1 0.00
1.000
0.001
992.2 563
152.0 7.25
0.46 1 0.00
1.000
0.010
992.2 = 5.63
152.0 = 7.25
0.46 = 0.00
5.000
0.000
357.4 + 20.84
106.4 + 12.66
0.47 0.01
5.000
0.001
357.4 + 20.84
106.4 + 12.66
0.47 1 0.01
5.000
0.010
357.4 + 20.84
106.4 + 12.66
0.47 +
0.001
0.000
1000.8 + 2.77
147.0 7.11
0.46 + 0.00
0.001
0.001
1000.8 2.77
147.0 7.91
0.46 +
0.001
0.010
1000.8 2.77
147.0 7.91
0.46 + 0.00
0.010
0.000
1000.4 2.3
146.0 8.97
0.46 +
5
0.010
0.001
1000.4 2.3
146.0 8.97
0.46 = 0.01
0.010
0.010
1000.4 1.2.
146.0 8.97
0.46 1 0.01
5
0.100
0.000
1000.4 5.68
144.8 5.36
0.46 1 0.01
1.100
0.001
1000.2 5.63
144.4 5.18
0.46 + 0.01
5
0.100
0.010
1000.2 5.63
145.2 5.22
0.46 : 0.01
1. 1.000
0.000
992.2 = 5.63
152.0 = 7.25
0.46 + 0.00
5
1.000
0.001
991.8
151.0 5.66
0.46
1.000
0.010
994.0 = 3.74
151.4 + 3.68
0.46 + 0.01
5.000
0.000
357.4 20.84
106.4
0.47 1
5.000
0.001
373.8 1 13.2
109.0 1 8.09
0.47 1 0.01
5.000
0.010
456.6
135.8 + 11.41
0.47 +
10
0.001
0.000
1000.8 + 2.77
146.8 8.14
0,46 + 0.01
10
0.001
0.001
1000.8 2:7
146.8 8.14
0.46 + 0.01
10
0.001
0.010
1001.0 2.74
146.2 + 8.04
0.46 + 0.01
10
0.010
0.000
1000.8 2.77
146.4 + 10.64
0.46 +
10
0.010
0.001
1001.0 3.46
146.0 + 10.46
0.46 1:220 0.00
0.010
0.010
1000.6 3.13
146.4 + 10.09
0.46 1 0.00
10
0.100
0.000
1001.2 5.26
144.6 3.29
0.46
10
1.000
0.001
1000.6 5.68
144.4 + 2.88
0.46 + 0.01
10
0.100
0.010
1000.6 6.31
144.8
0.46 1
10
1.000
0.000
992.2 = 5.63
152,0 7:25
0.46 1 0.00
10
1.000
0.001
991.0
149.8 6.22
0.46 +
10
1.000
0.010
993.6 = 5.55
150.6 + 3.97
0.46 + 0.00
10
5.000
0.000
357.4 20.84
106.4 +
0.47 1
10
5,000
0.001
386.0 + 15.02
111.4 6.35
0.47 + 0.01
10
5.000
0.010
547.8 10.57
158.8 + 10.99
0.47 +
38

---

Published as a conference paper at ICLR 2025
Table 25: Varying y for maximizing ring count guidance with AR D-CFG. Validity, novelty, and mean ring count for novel sequences reported. Mean + standard deviation reported from peated generation 1,024 sequences using five different random seeds. The setting reported in the main paper is bolded.
Num. Valid (t)
Num. Novel (t)
Novel Ring Count Mean (t)
1003.4 4.72
189.0 i: 6,44
4.53 i: 0.07
966.6 = 5.32
217.4 = 14.5
4.76 + 0.10
3
856.4 10.21
213.0 + 12.75
4.81 + 0.08
647.6 = 16.07
137.4 : 9.21
4.75 + 0.10
441.4
77.8 +: 2.28
4.83 + 0.08
Table 26: Varying " for maximizing ring count guidance with AR FUDGE. Validity, novelty, and mean ring count for novel sequences are reported. Mean + standard deviation reported from re- peated generation of 1,024 sequences using five different random seeds. The setting reported in the main paper is bolded.
7
Num. Valid (t)
Num. Novel (1)
Novel Ring Count Mean (t)
956.8 3:11
220.2 + 27.53
4.50 + 0.10
2
861.6 607
239.4 + 20.89
4.70 0.06
847.2 8.44
266.4 +: 13.76
4.80 : 0.06
4
704.0 5.2
241.0 + 18.37
4.87
281.2 = 497
103.6 4.83
4.89 + 0.10
50.6 1:
19.4 4.34
5.02 1
8.6 1. 4.34
4.0 1 1.58
4.89 1 0.44
1,0 +
0.4 + 0.55
2.20 + 3.03
0,0 0.0
0,0 0.0
0,00 + 0.00
0.0 0.0
0.0 0.D
0.00 1 0.00
Table 27: Varying n (the number of Langevin steps) and 7) (the step size) maximizing ring count guidance with AR PPLM. Validity, novelty, and mean ring count for novel sequences are measured for generated sequences from each method. Mcan standard deviation reported from repeated generation of 1,024 sequences using five different random seeds. The setting reported in the main paper is bolded.
7
Num. Valid (t)
Num. Novel (1)
Novel Ring Count Mean (t)
10
0.04
1009.8 = 1.92
140.6 16.47
1.92 0.09
10
0.1
1009.6 = 2.07
140.2 + 16.25
1.92 0.06
30
0.04
1010.0 1: 2.35
139.4 14.74
1.91 0.09
30
0.1
1010.0 +: 1.58
138.6 10.31
1. 1.88 0.12
Table 28: Varying y for maximizing ring count guidance with MDLM D-CFG. Validity, novelty, and mean ring count for novel sequences are reported. Mean + standard deviation reported from repeated generation of 1,024 sequences using five different random seeds. The setting reported in the main paper is bolded.
?
Num. Valid (t)
Num. Novel (t)
Novel Ring Count Mean (t)
465.0 18.56
273.8 + 11.95
4.52 +
2
363.6 = 17.5
215.6 = 11.1
4,85 + 0.05
242.4 ::2 1585
144.6 1: 16.89
5.01 1: 0.08
4
152.2 1 4.44
97.4 1: 5.08
5.10 4: 0.13
90.0 = 8.15
60.0 + 7.52
5.26 + 0.15
39

---

Published as a conference paper at ICLR 2025
Table 29: Varying 7 for maximizing ring count guidance with MDLM D-CBG. We also report results for both using the first order approximation and using it. Validity, novelty, and mean ring count for novel sequences are reported. Mean + standard deviation reported from repeated generation of 1,024 sequences using five different random seeds. The setting reported in the main paper is bolded.
7
Use Approx.
Num. Valid (1)
Num. Novel (1)
Novel Ring Count Mean (t)
False
143.8 6,61
121.2 + 7.01
5,00 = 0.07
True
455.4 = 24.11
229.4 = 15.21
2.52 - 0.11
2
False
0.4 = 0.55
0,4 = 0.55
3,00 = 4.47
2
True
327.6 = 15.82
176.4 17.3
2.82 = 0.00 1 0.00
3
False
0.0 0.00
0.0 i 0.0
3
True
223.8 i 15.01
136.0 1988
3.25 1 0.26
4
False
0.0 1:
0.0 100000
0,00 1 0.00
True
158.0 9.27
106.6 7.13
3.73 = 0.19
5
False
0.0
0.0
0.00 0.00
5
True
135.0 9.9
94.4 + 11.06
4.11 = 0.26
6
False
0.0
0.0 =
0,00
6
True
122.8 + 11.
89.8 = 7.82
4,41 + 0.11
7
False
0.0 0.0
0.0
0.00 1:
7
True
116.4 12.5
85,4 + 10.38
4.59 = 0.10
8
False
0.0
0.0
0.00 =
8
True
121.2 = 12.91
92.6 = 11.95
4.54 = 0.16
False
0.0 + 0.0
0.0
0.00 = 0.00
9
True
112.2 + 12.28
87.4 + 11.44
4.70 - 0.17
False
0.0 0.0
0.0 =
0.00 = 0.00
10
True
113.0 8.75
85.6 882
4.75 0.23
Molecule Property Maximization (QED)
Malecule Property Maximization (Ring Count)
Guidance method: D-CBG
Guidance method: D-CBG
Gamma
5.0
0.58
Novel Mean Ring Count
4.5
Novel Mean QED
0.56
4.0-
Use Approx
Gamma
0.52
Approx
Model
True
False
2
0.50
MDLM
4
3.0
AR
UDLM
10
0.46
100
125
150
200
100
150
250
300
350
400
450
Num. Novel Samples (out of 1,024 generated)
Num.
Novel Samples (out 1,024 generated)
Figure 5: Ablating the of first order approximation when applying D-CBG to discrete diffusion models. (Left) Maximizing drug-likenn likeniss (QED). (Right) Maximizing ring count.
40

---

Published as a conference paper at ICLR 2025
Table 30: Varying n (the number of Langevin steps), (the step size), and Thu (the stability reg- ularization coefficient) for maximizing ring count guidance with MDLM NOS. Validity, novelty, and mean ring count for novel sequences are measured for generated sequences from each method. Mean + standard deviation reported from repeated generation of 1.024 sequences using five different random seeds. The setting reported used in the main paper is bolded.
7
7kt
Num. Valid (t)
Num. Novel (f)
Novel Ring Count Mean (t)
0.001
0.000
508.8 = 24.18
237.0 = 8.57
2.41 = 0.16
0.001
0.001
508.8 = 24.18
237.0 857
2.41 + 0.16
0.001
0.010
508.8 1 24.18
237.0 1 8.57
2.41 0.16
0.010
0.000
509.6 24.27
237.0 10.37
2.41 0.16
0.010
0.001
509.6 = 24.27
237.4 + 10.45
2,41 0.16
0.010
0.010
509.6 = 24.27
237.0 + 10.37
2.41 0.16
0.100
0.000
505.2 # 25.71
237,4 13.2
0.100
0.001
504.4 + 25.59
236.4 + 12.66
2
0.100
0.010
505.2 # 25.71
237.4 13.2
2.46 0.15
.000
0.000
458.4 1 26.28
250.00 = 12.35
2.79 0.14
1. 1.000
0.001
458.4 1 26.28
250.00 12.35
2.79 0.14
1,000
0.010
458.4 = 26.28
250.00 = 12.35
2.79 0.14
5.000
0.000
187.0 11.6
141.4 + 13.96
3.08 0.24
5.000
0.001
187.0 + 11.6
141.4 + 13.96
3,08 1 0.24
5.000
0.010
187.0 11.6
141.4 + 13.96
3,08 + 0.24
0.001
0.000
509.8 24.71
237.6 9.79
2.4 0.16
0.001
0.001
510.00 +
237.6
2.4
0.001
0.010
510.00 + 24.66
237.6 9.79
2.16
0.010
0.000
509.4 = 24.44
238.0 + 11.47
2.42 0.17
0.010
0.001
509.2 = 24.43
237.8 + 11.21
2.42 0.17
0.010
0.010
509.4 1 24.42
238.0 1: 11.25
2.42 0.17
0.100
0.000
506.0 1 21.25
238.6 6 1 11.74
2.48 0.17
0.100
0.001
506.4 = 20.21
238.0 + 10.54
2.48 0.14
0.100
0.010
505.0 = 21.06
236.6 + 10.06
2.48 0.16
1,000
0.000
458.4 + 26.28
250.00 = 12.35
2.79 0.14
1.000
0.001
464.4 210
252.4 + 11.95
2.79 0.14
1.000
0.010
466.4 = 22.04
254.0 + 11.38
2.78 #011
5.000
0.000
187.0 11.6
141.4 1 13.96
3.08 0.24
5.000
0.001
213.6 1 17.36
164.0 1 18.33
3.31 1.199
5.000
0.010
247.8 + 13.97
193.6 + 13:13
3,51 0.22
10
0.001
0.000
509.0 = 24.14
237.2 + 8.7
2,41 20.6
0.001
0.001
509.8 + 24.28
238.0 + 9.0
2.4
10
0.001
0.010
509.8 # 24.28
238.0 + 9.0
0.17
0.010
0.000
510.4 = 24.42
239.4 + 10.21
2.43 0.14
10
0.010
0.001
510.2 1: 24.41
239.6 + 10.36
2.42 0.14
0.010
0.010
510.00 24.4
239.6 + 10.36
2.43 0.14
10
0.100
0.000
504.0 + 19.26
238.4 + 10.74
2.48 0.19
0.100
0.001
505.6 = 18.15
239.6 + 10.36
2.48 #016
10
0.100
0.010
506.4 18.66
240.4
2.48 0.17
1.000
0.000
458.4 + 26.28
250.00 12.35
2.79 0.14
10
1.000
0.001
471.4
257.8 + 257.0 7.1
2.8 0.1
10
1.000
0.010
473.2 + 18.86
2.76 0012
10
5.000
0.000
187.0
141.4 + 13.96
3.08 1024
10
5,000
0.001
225.6 # 17.47
176.8 + 18.65
3.32 0017
5.000
0.010
265.4 + 12.62
206.6 + 10.88
3.47 0024
41

---

Published as a conference paper at ICLR 2025
Table 31: Varying for maximizing ring count guidance with UDLM D-CFG. Validity, novelty, and mean ring count for novel sequences are reported. Mean = standard deviation reported from repeated generation of 1,024 sequences using five different random seeds. The setting reported in the main paper bolded.
Y
Num. Valid (1)
Num. Novel (t)
Novel Ring Count Mean (t)
979.0 +: 8.83
276.8 +: 17.34
4.54 +:
2
998.2 + 6,83
229.0 8:4
4.75 + 0,08
998.2 1: 4.49
216.2 +: 12.99
4.88 + 0.04
4
989.6 5.94
211.4 2888
4.85 + 0.08
968.6 = 7.89
209.4 = 13.2
4.84 + 0.06
Table 32: Varying y for maximizing ring count guidance with UDLM We also report results for both using the first order approximation and not using it. Validity, novelty, and mean ring count for novel sequences are reported. Mean + standard deviation reported from repeated generation of 1,024 sequences using five different random seeds. The setting reported in the main paper is bolded.
7
Use Approx.
Num. Valid (1)
Num. Novel (f)
Novel Ring Count Mean (1)
1
False
797.4 9
279.0 = 23.37
4.12 = 0.04
True
978.4 3.91
166.0 = 11.81
2.49 =
2
False
829.4 = 11.59
336.4 10.55
4.54 = 0.03
2
True
892.2 +:: 12.19
171.2 = 13.59
3.09 1:
3
False
862.6 7.73
363.8 12.76
4.70 + 0.03
3
True
763.8 = 11.52
198.8 = 10.76
3.76 = 0.09
4
False
880.6 = 19.15
393.0 12.39
4.74 0.08
True
705.8 = 14.69
215.2 = 15.06
4.30 = 0.14
5
False
889.2 12.4
404.0 9.64
4.76
5
True
726.8 i: 15.64
245.8 1. 6.83
4.47 1 0.07
6
False
898.6 1: 21.13
427.6 : 13.81
4.79 1
True
757.6 +: 10.38
277.6 8.73
4.55 1 0.04
7
False
898.2 2 9.26
436.4 8.79
= 4.80
7
True
779.4 = 17.29
295.6 = 12.18
4.62 = 0.06
8
False
897.2 = 11.58
432.0 = 19.07
4.84 - 0.02
8
True
796.6 = 15.84
304.4 10.01
4,66 - 0.06
False
903.2 2 1. 3.96
445.8 = 13.14
4.83 1 0.05
9
True
811.8 7.82
328.4 17.5
4,66 0.02
False
891.6 i 12.18
431.4 : 11.46
4.76 1
10
True
816.6 = 15.14
359.8 21.87
4.70 = 0.05
42

---

Published as a conference paper at ICLR 2025
Table 33: Varying (the number of Langevin steps), (the step size), and Thu (the stability reg- ularization coefficient) for maximizing ring count guidance with UDLM NOS. Validity, novelty, and mean ring count for novel sequences are measured for generated sequences from each method. Mean + standard deviation reported from repeated generation of 1,024 sequences using five different random seeds. The setting reported used in the main paper is bolded.
7
7kt
Num. Valid (t)
Num. Novel (f)
Novel Ring Count Mean (t)
0.001
0.000
1000.8 + 2.77
146.2 = 7.26
2.10 + 0.08
0.001
0.001
1000.8 2.77
146.2 7.26
2.10 =
0.001
0.010
1000.8 1. 2.77
146.2 7.26
2.10 1: 0.08
0.010
0.000
1001.0 2.45
147.2 8.44
2.10 1:
0.010
0.001
1000.8 2.17
147.4 8.56
2.10 = 0.09
0.010
0.010
1001.0 2.45
147.2 8.44
2.10 +
0.100
0.000
1000.8 4.92
148.2 7.66
2.13 + 0.09
0.100
0.001
1000.8 4.92
148.2 7.66
2.13 +
0.100
0.010
1000.8 4.92
148.2 7.66
2.13 + 0.09
.000
0.000
988.6 5.5
161.6 + 10.74
2.47 1 0.11
1. 1.000
0.001
988.6 1.5
161.6 1 10.74
2.47 1 0.111
1,000
0.010
988.6 5.5
161.6 = 10.74
2.47 + 0.11
5.000
0.000
323.6 = 15.18
109.4 = 12.24
3.41 + 0.09
5.000
0.001
323.6 + 15.18
109.4 + 12.24
3.41 + 0.09
5.000
0.010
623.6 + 15.18
109.4 + 12.24
3.41 +
0.001
0.000
1000.8 + 2.77
146.6 7.5
2.10 + 0.08
0.001
0.001
1000.8 8 2.77
146.6 7.5
2.10 +
0.001
0.010
1000.8 8 + 2.77
146.6 + 7.5
2.10 1 0.08
0.010
0.000
1000.8 2.86
148.2 6.94
2.11 + 0.10
0.010
0.001
1000.8 2.86
148.0 7.04
2.11 0.10
0.010
0.010
1000.8 2.86
147.8 7.
2.12 0.100
0.100
0.000
1001.0 0 4.24
149.0 5.29
2.14 1.13
0.100
0.001
1001.0 10 + 4.24
149.0 5.29
0.13
0.100
0.010
1001 .0 4.24
149.0 + 6.0
0.13
1,000
0.000
988.6 5.5
161.6 + 10.74
2.47 0.111
1.000
0.001
990.2 5.26
162.2 9.96
2.46 0.11
1.000
0.010
989.4 4.83
160.4 = 7.57
2.44 + 0.11
5.000
0.000
323.6 1 15.18
109.4 1 12.24
3.41 1
5.000
0.001
413.0 1 16.17
166.8 1 18.95
3.92 1 0.05
5.000
0.010
573.8 + 13:03
244.4 + 13,05
3.96 +
10
0.001
0.000
1001 0 + 2.74
146.8 7.33
2.09 = 0.09
0.001
0.001
1001.0 2.74
146.8 7.33
2.09 + 0.09
10
0.001
0.010
1001.0 + 2.74
146.8 7.33
2.09 + 0.09
0.010
0.000
1001.0 3.08
147.8 7.46
2.12 + 0.10
10
0.010
0.001
1001.0 3.08
147.8 7.46
2.12 1 0.10
0.010
0.010
1000.6 3.78
147.2 6.98
2.12 1 0.10
10
0.100
0.000
1000.0 4.74
150.4 7.09
2.15 0.13
0.100
0.001
1000.0 4.74
151.0 6.78
2.14 0.12
10
0.100
0.010
1000.0 4.74
150.6
2.15 1 0.13
1.000
0.000
988.6 4 5.5
161.6 + 10.74
2.47 1 0.11
10
1.000
0.001
990.6 5.55
162.0 9.64
2.45 +
10
1.000
0.010
993.0 3.46
163.6 9.89
2.47 0.09
10
5.000
0.000
323.6 + 15.18
109.4 +
3.41 0.00
10
5,000
0.001
454.6 # 12.78
188.8 + 12.24
3,96 + 0.05
5.000
0.010
670.0 + 6.4
276.8 + 20.62
3.81 0.0
43

---

Published as a conference paper at ICLR 2025
Table 34: Ablation results of y values CIFARIO. Best values for each model are bolded.
FID ())
IS (1)
FI (t)
MDLM D-CFG
7=1
27.94
7.14
0.76
7=2 2
18.62
8.24
0.95
7= 3
16.19
8.78
0.99
7 = 4
15.56
9.02
0.99
7 7= 5
15.73
9.19
1.00
UDLM D-CFG
7 1
26.70
7.43
0.81
7 = 2
20.75
8.34
0.96
7 3
21.31
8.52
0.98
7 7=4 4
23.21
8,66
0.99
7 7= 5
26.15
8.60
0.99
Molecule Property Maximization (QED)
Malecule Property Maximization (Ring Count)
Guidance method: D-CFG
Guidance method: D-CFG
0.62
T
Model
Gamma
Model
Gamma
5.2
32
MDLM
Novel Mean Ring Count
5.1
MDLM
2
Novel Meann DED
0.60
0.59
4.8
0.57
4.7
16
0.56
60
80
100
120
140
160
180
50
100
150
200
250
Samples (out 1,024 generated)
Samples (out generated)
Molecule Property Maximization (QED)
Molecule Property Maximization (Ring Count)
Guidance method: D-CBG
Guidance method: D-CBG
16
0.60
Novel Mean Ring Count
64
Novel Mean OED
0.58
Gamma
3.5
Model
Gamma
MDLM
Model AR
3.0-
UDLM
0.54
MDLM UDLM
2.5
160 Num. Novel Samples (out of 1,024 generated)
100
150
Num. Novel Samples (out of 1,024 generated)
Figure 6: Effect of varying T on QM9 guidance generation. (Top Left) Maximizing drug-likeness, (QED) using D-CFG. (Bottom Left) Maximizing drug-likeness (QED) using D-CBG. (Top Right) Maximizing ring count using D-CFG. (Bottom Right) Maximizing ring count using D-CBG.
44

---

Published as a conference paper at ICLR 2025
Table 35: Varying T' for CIFARIO conditional generation. 50k images sampled from a conditional model (D-CFG y=1). Fl metrie from separate classifier trained to identify the class label used for the conditional generation. Best metric per I is bolded.
Model
FID (4)
IS (1)
Fl (t)
T = 128
MDLM
64.09
5.81
0.63
UDLM
30.48
7.30
0.80
T 256
MDLM
41.53
6.51
0.70
UDLM
28.30
7.40
0.81
T 512
MDLM
31.97
6.89
0.74
UDLM
27.12
7.43
0.74
T 1024
MDLM
27.94
7.14
0.81
UDLM
26.70
7.43
0.81
Table 36: Generative Perplexity (Gen. PPL; using GPT-2 Large) for unconditional sample genera- tion from models trained LMIB, Best value bolded.
Model
T
Gen. PPL (1)
AR
128
67.46
MDL.M
16
170.64
MDI.M
32
140.47
MDLM
64
130.62
MDLM
128
120.93
MDLM
256
119.96
MDLM
512
118.34
MDLM
1024
116.80
UDLM
16
82.09
UDLM
32
79.80
UDLM
64
79.93
UDLM
128
79.87
UDLM
256
79.07
UDLM
512
79.14
UDLM
1024
78.22
45

---

Published as a conference paper at ICLR 2025
Table 37: Generated samples for unconditional generation from models trained on LMIB.
Model T
Generated Text
AR 128
"[CLS] to move the global economy away from inappropriate trude practices and build environment that rewards innovation. i. e. sam- ples of šeed money by jüm mellon, paul pompa jr. gil familier, vanessa thomas, george rothman, george whitacre, john ziemba, mark sweitzer, little george nowi, thayer evans and allison gallagher. i. e. yang if you don't 't wanna fly, the statistics means you going make your rocket arm rolls. [CLS] savings that made the difference in someone's 'ss life cannot be made in another country without foreign assistance on their [CLS]"
MDLM - 16
"[CLS] mezzo ev of dream spirits. [CLS) to naval cudets i didd'' 't want mum and dad be group, but even with the odd social butterflies felt myself sitting at the edge of a deep blue pool. [CLS] wednesday morning, fighter jets billoured the area during the early hours of the morning. [CLS] know of the sailing course, the scenery, the coastline, the whole thing very beautiful -- the land of water. [CLS] wednesday, after its peak height reached the ice's 's lowest point. [CLS] reader's ss thought is. for the basic requirements of team leadership, internship, part time, a..."
MDLM 1024
"[CLS] was killed as mr giffords and as the caregiver, representative giffords's. [CLS] but have no doubt that this process will produce meat, mns. rommel said. [CLS] and even if you don't I need to hit an agency envelope during it, you should pay the rent through office penses, referring to the possible error. [CLS] blog. view billshrift and discuss yahoo commercials, and what? [CLS] this month, the new logos of cin armarge became scranton neck based riverton and tollhor. [CLS] when arrived, he' 'd ducking behind the woman."
UDLM 16
"[CLS] grandeur. [CLS] grand avenue an. [CLS] it upon request, such information plays only in the provision of the protocol äs guidelines for the nonprofit sector. [CLSS] that too has to worry people about schools that shut down and closed schools for several weeks. [CLS] I will not comment on anything important to the eu, where someone is blaming serbia to not a figurehead. [CLS] but i think that because an increasing number of coal is put at risk as a displaced gateway, and the other party of technology falls away from communities that can somehow show up as a starting point, one thing that the province will suffer is loss of almost (CLS)" "[CLS] would [CLS] a 49 percent stake in the company under the terms of the deal. [CL.S] the only subsequent trial out of ms, sykes is 19 febru- ary next year after prosecutors failed to overturn the death sentence against him. [CLS] but dugard has recorded several rings for baby je- sus, a completely new thing as for calling muslim cold callers and working with his confiscators. [CLS] i am forced to seek out convenient explanations while remaining in no position either to caucus. [CLS] of course, get me oul of there. [CLS] the first thing being wrote about having available food. [CLS] sir simon's 's guests included singer hayley wood and"
UDLM 1024
46


Reviews:
[{"review_id": "H0azQiBzfw", "reviewer": "Reviewer_LFVk", "summary": "This paper presents several advances in discrete diffusion models, specifically focusing on uniform noise diffusion and guidance mechanisms. The authors make three main contributions:\n\n-  developing classifier-based and classifier-free guidance for discrete diffusion models,\n\n-  introducing Uniform Diffusion Language Models (UDLM) with an improved variational bound\n\n-  demonstrating superior controlled generation compared to autoregressive baselines across multiple domains.", "questions": "- What are the primary factors limiting performance on larger vocabularies, and how might these be addressed?\n\n- How to understand equation 9, does it have any connection with the loss in Lou 2023?\n\n- How does the choice of sampling steps T affect the trade-off between generation quality and computational cost? Is there a systematic way to choose optimal T?\n\n- Have the authors explored potential failure modes of their guidance mechanisms, especially in cases where the guidance strength $\\gamma$ is high?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "-  Theoretical foundation with clear derivations of the continuous-time variational bound\n\n- Practical guidance mechanisms that are easy to implement\n\n- Thorough ablation studies and hyperparameter analysis", "weaknesses": "The primary limitation is the scope of improvements being mainly restricted to multinomial diffusion with small vocabulary settings, with a persistent performance gap in larger vocabulary NLP tasks. The paper lacks computational complexity analysis and detailed runtime comparisons. Some hyperparameter choices, particularly sampling steps T, would benefit from more thorough justification. The theoretical analysis of guidance mechanisms could be more extensive, and more exploration of failure cases would strengthen the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ubO4u3YhNg", "reviewer": "Reviewer_2Wfu", "summary": "This paper introduces a discrete diffusion model and guidance mechanism that is effective at controllable generation. The paper uses discrete classifier-based and classifier-free guidance, and introduces uniform noise diffusion language models that can continuously edit discrete data.", "questions": "See weakness above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Adapting diffusion model to discrete data is an important task as it can be utilized to a few data generation tasks, such as molecule and text generation.\n\n3. The experiments demonstrate the effectiveness of guidance with discrete diffusion models on several domains, and show that UDLM can achieve state-of-the-art performance on small vocabulary datasets.", "weaknesses": "1. In Introduction, \"D-CBG\" is introduced before being explained.\n\n2. I would recommend the paper to explain the logic, why the proposed UDLM can make the guidance easier? To me I think the only difference is to reset the $\\pi$ value in the proposed method. How can we understand the motivation for this setting and what's the intuition?\n\n3. In the first paragraph of Section 3.2, the paper mentions that the proposed method yields a simple expression for ELBO, but I didnt see ELBO below. Also, I would suggest the paper use a table to compare the new and the old ELBO to make the improvement more clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ooh9BJnwEC", "reviewer": "Reviewer_9Khq", "summary": "In this paper, the authors derive classifier-free and classifier-based guidance for discrete diffusion-model controllable generation.  The proposed guidance techniques build upon discrete diffusion techniques that interpolate between clean data and a noisy prior.   Empirically,   the proposed method outperforms Autoregressive models and several diffusion-model baselines on genomic sequences, small molecule design and CIFAR-10.", "questions": "1. Please clarify the contribution of this paper as the point one listed in the above weakness part. \n\n2. What is the relationship and difference between the proposed discrete diffusion method and the one in [1] and [2] listed above?\n\n3. What is the key advantage of the proposed method compared with other discrete diffusion methods? Why does the proposed method perform better than other discrete diffusion methods?\n\n4. Please explain about the equitation in Line 181 on page 4.  Why does the log(p) term equal the inner product term?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.  The paper is well-written and well-organized. \n\n2. The proposed method is simple and easy to follow. \n\n3. It seems the proposed method outperforms baselines on several discrete tasks.", "weaknesses": "1.  The description of the contribution is not clear enough.  Some parts may be overclaimed. \n\nI am not sure whether the claimed contribution is classifier-free and classifier-based guidance for a discrete diffusion-model controllable generation or if it further includes the interpolating discrete diffusion model parts.  In Line 58 on page 2, the authors claim they introduce \"a class of discrete diffusion models particularly amenable to guidance.\"  This part may be overclaimed because the key technique involved is the simplified discrete diffusion process with the transition probabilities as the interpolation between the current state and a prior distribution. This technique may not be new. See the following point two.\n\n2. Very similar discrete diffusion works are not discussed and compared. \n\nIn both [1] and [2],  similar simplified discrete diffusion processes with the transition probabilities as the interpolation between the current state and a prior are employed,  which is very related to the proposed method. However,  discussion and comparison with these works are missing.   As a result, the claimed SOTA performance is unconvincing. \n\n3. What is the key advantage of the proposed method over other discrete diffusion methods is not clearly discussed. \n\nWhat is the key advantage of the proposed method compared with other discrete diffusion methods? Why does the proposed method perform better than other discrete diffusion methods?\n\n4.   The equitation in Line 181 on page 4 may not be correct. \n\nWhy does the log(p) term equal the inner product term?\n\n\n\n[1] Zheng et al.   A Reparameterized Discrete Diffusion Model for Text Generation.  2023\n\n[2] Zhao. et al.  Unified Discrete Diffusion for Categorical Data.   2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1wiimS9X4n", "reviewer": "Reviewer_S2DB", "summary": "This paper proposes to introduce uniform noising paths for discrete diffusion models. The paper derives an explicit formulation for the variational lower bound under a uniform prior. The paper further introduces methods to perform classifier-free and classifier-based guidance in this discrete diffusion setting. Experiments are conducted in regular text, images, genome, and molecular settings.", "questions": "N/A", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "The paper is generally well written with the development of ideas following a logical narrative. The experimental settings considered are also diverse in terms of various discrete domains. Beyond this, it is difficult to pinpoint further clear strengths.", "weaknesses": "I appreciate the authors for trying to crisply write their simple uniform diffusion model framework but I have several critical concerns regarding various aspects of this paper that I will outline below.\n\n**Novelty**\n\nI very respectfully disagree with the author's contributions. The uniform prior and the corresponding are widely known and exist in the literature. In fact, I challenge the authors to demonstrate how their formulation differs from Discrete Flow Matching (Gat et. al 2024), and Dirichlet Flow/Diffusion (Stark et. al). There are other papers as well, but very explicitly your objective function in your variational lower bound can be shown $\\mathbb{E}[-\\log p(x_0 | x_t)]$ where $x_t \\sim p_t(x)$ is the uniform path. This analysis is a direct application of the MDLM setup for the uniform path. I am open to being wrong here, but I do not see anything novel here despite equation 10 suggesting there is more going on than there is. Also, I believe the first-order Taylor approximation to guidance was already introduced in \n(Nisonoff et. al 2024) and the adaptation in this paper is very respectfully a minor change. \n\n\n**Technical Limitations**\n\nRegarding the new classifier/classifier-free guidance formulation the fact that we need to compute the normalization constant means that this cannot be extended to actual larger-scale systems where we would want to apply discrete diffusion models as opposed to autoregressive ones. I have severe concerns regarding the usefulness of the claim that the proposed method works well on small vocabularies, this to me is a sign that this approach is fundamentally limited. Note that Dirichlet Flow Matching (Stark et. al 2024) point out this exact issue with a uniform prior paths, and hence suggest their Dirichlet paths. \n\n**Experimental Concerns**\n\nRegarding the experimental setup I have a few concerns regarding the results presented. I find the omission of MDLM for the LM1B table 3 to be awkward. Looking at Table 1 in the MDLM paper we notice that their test perplexity upper bound is 27.04 which is better than the UDLM. In fact, I really do not understand the need to report Table 1 and 3 for the same dataset, it feels contrived to change the vocabulary size. It is clear that uniform doesn't work in this setting and I bet playing with noise schedules and other small details would close the gap for MDLM and UDLM in the smaller scientific settings. I also found it surprising the Generative Perplexity under GPT-2 was not reported as MDLM does. Finally, there are no textual generated samples as done for even CIFAR 10. This is a strange omission, we should be able to visually inspect the difference in suspected performance.\n\n**References**\n\nStark, Hannes, et al. \"Dirichlet flow matching with applications to dna sequence design.\" arXiv preprint arXiv:2402.05841 (2024).\n\nGat, Itai, et al. \"Discrete flow matching.\" arXiv preprint arXiv:2407.15595 (2024).\n\nNisonoff, Hunter, et al. \"Unlocking Guidance for Discrete State-Space Diffusion and Flow Models.\" arXiv preprint arXiv:2406.01572 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)