Simple and Effective Masked Diffusion Language Models
Authors: Subham Sekhar Sahoo
Marianne Arriola
Yair Schiff
Aaron Gokaslan
Edgar Mariano Marroquin
Justin T Chiu
Alexander M Rush
Volodymyr Kuleshov
Venue: NeurIPS 2024 poster (2024)
Keywords: Text Diffusion
Generative Models
Primary area: generative_models
TL;DR: A masked diffusion framework featuring a novel objective that enhances log-likelihood estimation and achieves state-of-the-art results, including applications to DNA sequence modeling.

Abstract: While diffusion models excel at generating high-quality images, prior work reports a significant performance gap between diffusion and autoregressive (AR) methods in language modeling.
In this work, we show that simple masked discrete diffusion is more performant than previously thought.
We apply an effective training recipe that improves the performance of masked diffusion models and derive a simplified, Rao-Blackwellized objective that results in additional improvements.
Our objective has a simple form&mdash;it is a mixture of classical masked language modeling losses&mdash;and can be used to train encoder-only language models that admit efficient samplers, including ones that can generate arbitrary lengths of text semi-autoregressively like a traditional language model.
On language modeling benchmarks, a range of masked diffusion models trained with modern engineering practices achieves a new state-of-the-art among diffusion models, and approaches AR perplexity. We provide the code, along with a blog post and video tutorial on the project page: https://s-sahoo.com/mdlm

Simple and Effective Masked Diffusion Language Models
Subham Sekhar Sahoo Cornell Tech, NYC, USA. ssahoo@cs, cornell.c... edu
Marianne Arriola Comell Tech, NYC, USA. ma2238@cornel..edu
Yair Schiff Cornell Tech, NYC, USA. yzs2@cornell. edu
Aaron Gokaslan Cornell Tech, NYC, USA. akg878cs cornell.... edu
Edgar Marroquin Cornell Tech, NYC, USA.
Justin Chiu Cornell Tech, NYC, USA.
emm3928cornell.ccm edu
jtc2578cornel1. edu
Alexander Rush Cornell Tech, NYC, USA. ar459@corme11.cm edu
Volodymyr Kuleshoy
Cornell Tech, NYC, USA.
kuleshov&cornelll edu
Abstract
While diffusion models excel at generating high-quality images, prior work reports a significant performance gap between diffusion and autoregressive (AR) methods in language modeling. In this work, show that simple masked discrete diffusion is more performant than previously thought. We apply an effective training recipe that improves the performance of masked diffusion models and derive a simplified, Rao-Blackwellized objective that results in additional improvements. Our objective has simple form-it is mixture of classical masked language modeling losses and can be used to train encoder-onte language models that admit efficient samplers, including ones that can generate arbitrary lengths of text semi-autore autoregressively like traditional language model. On language modeling benchmarks, a range of masked diffusion models trained with modern engineering practices achieves new state-of-the art among diffusion models, and approaches AR perplexity. We provide the code along with blog post and video tutorial on the project page:
https://s httf//ssahh.cooooo comoo.
Introduction
Diffusion models excel at producing realistic, high-quality images and have received significant attention as potential tools for generating discrete data, such text [1, 33], biological sequences [2, 47], and graphs [60, 63]. Unlike autoregressive (AR) approaches, diffusion- methods are not constrained generate data sequentially, and therefore have the potential to improve long-term planning, controllable generation, and sampling speed. However, discrete diffusion methods exhibit a performance gap relative to AR models [1, 23, 26, 33], especially in language modeling. The standard measure of language modeling performance is likeliiih:ooo:: when controlling for parameter count, prior work reports a sizable likelihodo gap between AR and diffusion models.
In this work, we show that simple masked diffusion language modeling (MDLM) combined with effective training recipes is more performant than previously thought [1, 26). We develop well- engineered MDLM implementation that significantly improves discrete diffusion log- likelioo;hh:;;; we
code: https://githubbco https//gittub..co//uleeee www.a//wboooooollleeeooooonnnnnn
"tutorial: http://lyouubbb//AAXX3b3r
38th Conference on Neural Information Processing Systems (NeurIPS 2024).

---

Simplified Masked Diffusion
Diffusion Training: Average of unmasking losses
Masking rate is random, not fixed
Objective
aimple average losses
xj
Objective is a variational lower bound
Improved implomental relative to previous diffusion
Admits ancestral campling
*PL.
MDLM: Expectation over masking
difl sion
Morstr 1
118.6
LM1B
[
diffus on
for [
data
78.8
63.8
Masked
diffus on
[
data
1 6
M Makko
dinusion
Discrete
obiaa
AR:
32.8
23.0
20.9
Input: x
Masked
diftusion
for discrete
data
Difuuiion
DSPM
Difuuiio
SE0O
MOLM (Oura)
LM
Figure I: (Left) Our proposed masked diffusion language model (MDLM) is trained using weighted average of masked cross entropy losses. (Top Right) In comparison to masked language models (MLM), MDLM's 's objective correspond principled variational lower bound, and supports generation via ancestral sampling. (Bottom Right) Perplexity (PPL) on One Billion Words (LMIB) benchmark.
further improve likelihood using a simple substitution-but based parameterization of the reverse diffusion process that enables deriving Rao- Blo--Blacweellzedd continuous-ting variational lower bound (ELBO) with improved tightness [49]. Interestingly, our objective has a simple form: it is weighted average of masked language modeling (MLM) losses [15], and can be used to endow BERT-style, encoder-only models with principled generation capabilities. We complement this framework with efficient camplers-includiing ones that can generate autorerrsssol like typical language model.
Our masked diffusion models achieve a new state-of-the-art among diffusion models on language modeling benchmarks and approach the perplexity of AR models within 15-25%. Surprisingly, simple engineering choices significantly improve performance in both our models and simple baselines that were previously thought to perform poorly. Our framework also extends non-language domains, including biological sequence modeling. We pre-train DNA sequence models and observe similar higher downstream performance compared to classical BERT-style training, while also introducing generative capabilities that classical masked DNA language models lack.
Contributions We describe (1) a simple masked diffusion language modeling (MDLM) framework with well-engineered implementation that outperforms all existing diffusion models across language modeling benchmarks (LMIB [8], OWT [18], DNA [12]). and that significantly improves the performance of existing baselines [1, 26]. Our MDLM framework implements (2a) a substitution-bast parameterization (SUBS) of the reverse unmasking diffusion process; SUBS allows us to derive (2b) a simple, continuous-time Rao-Blan Blackwellized objective that improves tightness and variance the ELBO, further increasing performance. We complement MDLM with (3) fast samplers that support semi-autoregressivee (SAR) generation and outperform previous SAR models.
2 Background
2.1
Diffusion Models
Diffusion models are trained to iteratively undo a forward corruption process q that takes clean data drawn from the data distribution gxx) and defines latent variables z; for tE [0,1] that represent progres- sively noisy versions of x [27, 54, 56, 66, 48, 19]. The standard forward process for continuous is
/ -1 0i.
(1)
where ~~ (O.I) and (a))) is noise schedule, monotonically decreasing in t. The parameterized reverse diffusion model pe over x and is trained to maximize a variational lower bound on log- likelihood (ELBO), Given number of discretization steps T,, defining s(i) (i- 1)/T and t(i) =//T.

---

and using DELL [-] to denote the Kullback-Leibler divergence,,,,, the Negative ELBO (NELBO) equals [54]]
By
(xgpa(X|2zco)
/o (100)))) [2c())]
+ + (pli(g(Ze(()) [x)] (2
(2)
Cresons
priir
Laifuusion
For brevity, we drop from t(i) and si) below; in general, s will denote the time step before t.
2.2 Discrete Diffusion Models
Applications of diffusion modeling to discrete data can be broken into two broad categories. First are works that embed discrete structures in continuous space and then perform the Gaussian diffusion defined above on these continuous representations [9, 16, 23, 24, 30, 34, 57]. More related to our method are works that define a diffusion process directly on discrete structures. D3PM [1] introduces framework with a Markov forward process {(z,) / Cat (a:
defined by the multiplication of matrices Q2 over 7 discrete time steps. This process induces marginals
= Cat(z, Cat(z,, QQ-1---Q-
that represent the discrete-state form of (1). Extending this formalism to continuous time (as in (1)) relies continuous time Markov chain (CTMC) theory [5]. The CTMC framework in turns leads to
generalizations of the score matching perspective on diffusion modeling [55] to discrete data [33, 59].
Notably, SEDD [33] connects score-based approaches with ELBO maximization, enabling performant likelihood-basee training of score-based models.
3 Simple Masked Diffusion Models
While previous work on discrete diffusion supports general forward processes (c.g., general Q, in D3PM), absorbing state (i.e., masking) diffusion consistently achieves the best performance 33]. In this work, instead of supporting general noise processes, we focus masking and derive tight Rao-Black Blackwellized objectives that outperform general approaches and do not require CTMC theory. In this section, we first define the diffusion process for categorical random variable. Later in Sec. 3.5, we extend this process to sequences containing multiple such categorical variables. We denote our overall approach as Masked Diffusion Language Models (MDLM).
Notation. We denote scalar discrete random variables with K categories as 'one-hot' column vectors and define V E fx E {0,1]* K Σ x; 1} as the set of all such vectors. Define Cat(::7) as the categorical distribution over K classes with probabilities given by π A where A denotes the K K-simplex. We also assume that the K-th category corresponds to a special [MASK] token and let m e V be the one-hot vector for this mask, i.e., mæ Additionally, let = {1] and (a,b) and a b respectively denote the dot and Hadamard products between two vectors a and b.
3.1 Interpolating Discrete Diffusion
We restrict our attention to forward processes that interpolate between clean data x V and target distribution Cat(.:T), forming a direct extension of Gaussian diffusion in (l). Let define a sequence of increasingly noisy latent variables z; E V, where the time step runs from t == 0 (least noisy) to (most noisy). The marginal of z; conditioned on x at time t
q(², x= =at
Cat [2;;0 + (1 or )T)
where a E [0,1] is a strictly decreasing function in f, with og s and a =0; see Suppl. E.I for details. This implies transition probabilities q(z, (2.) = Cat(zr;0y) + (1 a1--ap (T) where arjs as as/ax a, This indicates that during each diffusion step from s +t, a fraction (1 (1-aqjs of the probability mass transferred to the prior distribution The reverse posterior given as (see Suppl. 16 for details):
[or]. + - Odja ) 1x T T2,
2
② (o, x (1-x0+,)))m
q x)) Cat
Zz;
(5)
a+ x + (1-
While (4) and (5) represent special case of the more general diffusion processes proposed in D3PM [1], we show below that they yield a simplified variational lower bound objective and admit straightforward continuous time extensions.
3

---

3.2 Masked Diffusion
Next, we focus on masking processes and derive simple Rao-Blacke Blackwellized objective for this choice of g. This objective incurs lower variance during training and improves tightness.
3.2.1 Forward Masking Process
In masked (i.e., absorbing state) diffusion, π m. At each noising step, the input transitions
to a 'masked" state m with some probability. If an input transitions to m at any time l', it will remain in this state for all /. > L': q 25 m) (2) At time T, all inputs are masked with probability 1.
The marginal of the forward process (4) is given by g(z,) (x) = Cat(zr; axx + (1 - at (t)) Using properties of the masking process, the posterior (zz xx) simplifies (5); see Suppl. A.2:
Cat(zziz))
2: + m,
Cat Zzi (1-a, (a xx
(6)
2: - m.
1-
3.2.2 Reverse Unmasking Process
The reverse process inverts the noise process defined by q. We consider both finite number of steps T, as well as a continuous time model corresponding to 7' co. co. We begin with the discrete-time case for which the generative model is expressed as px(x)= = [_ Po (x/ %o) II Pe (2, 2 4z)d2201-
The optimal form for py (z(z,)e (2) matches the true posterior in (6): this follows immediately from the def- inition of the diffusion objective in (2), which is sum of terms of the form DKL (g(z,) (2, (zz|zz)), However, (6) is conditioned on x, which we do not know. Therefore, we introduce a model xo(2,,)) : V x [0,1] + A that approximates with neural network. We can also omit explicit depen- dence of Xp on time i, which simplifies sampling, yielding a 2x inference speed-up (see Suppl. E.2).
3.2.3 SUBS Parameterization
The specific parameterization for (z (z) that use is
Cat (ZaiZz
Zt + m,
(z(z,(z;)= qz (z,
Cat
Zz:
a + (x.-a.)xx
2, m,
(7)
1-ar.
Furthermore, we induce 2 key properties of the absorbing state diffusion process into our denoising model, xx(2..): an unmasked token remains unchanged during reverse diffusion, and the clean input never masked. We implement these as substitutions to the output of x (2),.)) hence we call our parameterization SUBS.
Zero Masking Probabilities the denoising network such that (x)(ztt),,m) = 0, i..., we substitute the logit index corresponding the [MASK] token with -0.
First, notice that by definition, (x.m) -O. For this reason, design
Carry-Oue Unmasking Second, if z is unmasked, then we desire (0(),,t) 24, i.e., unmasked latents 'carried over'. We accomplish this by substituting the output of our network to simply copy unmasked inputs.
In Suppl. B.1, we show that "Zero Masking Probabilities" property simplifies the D3PM's 's NELBO (39) to (41), and "Carry-Over Unmasking" futher simplifies (41) to (43) whose continuous time equivalent the simplified NELBO (10). Table 8 shows that each simplification leads to improved likelihood.
3.3 Rao-Blackwellized Likelihood Bounds
Recall from (2) that the diffusion traning objective has the form Crccons Ldiffusion Epiir- For the simplified reverse process in (7), the discrete-time diffusion loss for finite T simplifies to (Suppl.
T
Lditiusion
Σ
Eg DKL (an(gg(z,n) (2(0) x)lppa (2.(0) (e(i)))
E
Ox(1) - ax((1)
log (Z(())
1-a(()
(8)

---

Note that this objective is simpler and more well-behaver than the expression one would obtain for
DKL (g(zg 2 po (z. (Ze)) under the parameterization induced by using pe (z./2z) q(z z,,
xz( t)) from (5), which similar to what is used by D3PM [1] (see Suppl. A.2.4):
0i log
log (1-a. (L_a))(ar(xxlel aue (xo () m - (1-ar) --
1-at (x. (). (1-o,)) (.
+ (1-au))
(9)
1-a;
(1 04 ) ()xx)
We refer to the process of obtaining (8) lieu of (9) as form of Rao- Blw--llllalllzatoon... Specifically, we analytically compute expectations such as (xa(2e,(), = 0 in order to simplify objective (9) to obtain (8). Without analytical simplifications, a model must learn 0 such that (xo (x.(z).)). m) =0 holds. Unlike in regular Mho-BBackwellizatiin simplifications are possible because of modeling choices for x (z, .) (zero masking probabilities and carry-over unmasking). In that sense, our approach has similarities to graphical modeling, where incorporating conditional independencies into pe sets certain log-like eggliiklihhood terms to zero. However, our approach also empirically helps reduce variance, hence we refer to it Rao- nho-Blackwellization somewhat abusing the usual terminology.
3.4
Continuous-Timm Likelihood Bounds
Previous works have shown empirically and mathematically that increasing the number of steps T yields tighter approximation to the ELBO [29]. Following a similar argument, we form continuous extension of (8) by taking T oo oo (see Suppl. B.2), which yields the following NELBO, NELBO
1=1 of log(xo(z,,t)xxdl NELBO = BEg
(10)
1=0 1 l-a;
Invariance to the noise schedule The function a; is invertible due to the monotonicity assumption in Sec. 3.1, and so we can perform the following change of variables in (10): Y log(1 o4).
Thus, the diffusion loss can equivalently expressed as NELLBO -B, -B, - (x)
see Suppl. 1.1 for details. This new formulation demonstrates that the diffusion loss is invariant to the functional form of 0z2, which we verify empirically in Suppl. E.
3.5 Masked Diffusion Language Models
Next, we apply masked diffusion to language modeling over sequences I: of L tokens, with x" denoting the f-th token. We make the assumption that the forward noising process is applied independently across sequence and that, conditioned on a sequence of latents z 1:L, the denoising process factorizes independently across tokens, i.c., (z(z! ²1 = -I Pe (z' = Thus, we use a single model to compute (z!2 for each l from a masked sequence z, optimizing:
L NELBO
By
Llog(x5(2)) (..i:,))))x )),x")dt
(11)
1=
1-
Interestingly, our objective has simple form: is the weighted average of masked language modeling (MLM) losses [15]. Thus our work establishes connection between generative diffusion models and encoder-only BERT models. Our objective enables principled selection of (randomized) masking rate, and also endows BERT-style models with principled generation capabilities; see Sec. 6. The full training algorithm is provided in Suppl. B.3.
Note:
Although (1I) imposes a loss on all tokens, unmasked tokens don't I contribute to the loss, as they are copied over by the denoising network due "carry-over unmasking" (Sec. 3.2.3), effectively reducing log(x (z):
L ,t),x")))) to zero.
3.5.1 Training Considerations for Masked Diffusson
One of the key contributions of our work is a well-engineered implementation of masked diffusion models. Our experiments demonstrate that these improvements greatly boost performance even for methods previously thought to perform poorly, e.g., Austin et al. [1]. Below we briefly summarize these implementation details. First, we find that tokenization is critical to performance. Small vocabularies, such as the 8k vocabulary in Austin al. [1], result in longer-range dependencies that decrease the performance of both diffusion and AR models. Additionally, by focusing on masked diffusion,
5

---

are able to provide a numerically stable implementation of the objective function. Namely, since previous formulations of discrete diffusion were constructed to accommodate a wide range of limiting distributions [1], the objective was implemented by materializing the full transition matrices Q. and posterior probabilities. In contrast, we evaluate DKL. [g(2, &.s x (z, 2+] by examining only the masked token indices rather than comparing the full true and approximate posterior distributions.
Furthermore, we modernize the architecture for the denoising network relative D3PM [1]. In lieu of the T5 architecture used in D3PM, we use the diffusion transformer (DiT) introduced in Peebles & Xie [42], which integrates time step conditioning into standard encoder-only transformer [62] and uses rotary positional embeddings [58]. In addition, we implement a low-discrepancy sampler that reduces the variance of the ELBO, similar to Kingma et al. [29] and draws correlated samples
t, rather than performing i..d. sampling.
4 Inference and Sampling in Masked Diffusion Language Models
4.1 Efficient Ancestral Sampling
To generate sequence of length L, the reverse diffusion process starts with the sequence z!:11 where 4== m, for all & [I,......). Then the subsequent latents, z are generated by discretizing the reverse diffusion process with some finite T. Given z!!Z, we construct 2.:- by sampling each token z independently from the distribution po (z v.lzz!t// given in (7).
Note that in the reverse process, unmasked tokens remain unchanged. Thus, if no new tokens in z become unmasked (which can occur often in early denoising stages for large T), then z!:4 = x, Additionally if the denoising model, x (z?=i))) is not conditioned on time, then we can simply draw new sample from po (2--1/11
using the previously computed and cached value x (). This
means have effectively "skipped"" over the time step s, saving function call the denoising network. Note that SEDD [33] does not support this caching because the denoising network models time-dependent rates, which requires conditioning time.
4.2 Semi-Autoregressive Masked Diffusion Language Models
Our method also admits an effective semi-autoregressive (SAR) decoding method that allows the model to generate sequences of arbitrary length [24, 52, 53]. Let x represent the output from sampling sequence of L tokens using the reverse diffusion process described above. To generate additional L' tokens, we propose generation algorithm which the latter L- tokens T.:r. are used as prefix for an additional round of generation. Given the carry- unmasking described in Sec. 3.2.3, these prefix tokens will simply be copied over at each decoding step. The remaining tokens are generated as above with 2 (z. (z.)
[z, [z, L for all E [L+1,..., L+ L'), with z, L-L'' initialized to x --- I as opposed to being initialized as masked tokens m. At the end of this process, we have produced L+ L' tokens concat 1 x +++++++/) where concat denotes concatenation along the sequence length dimension. This process can repeat indefinitely, with the prefix shifted for every new round of generation.
5 Experiments
5.1 Masked Diffusion Language Models
Experimental Setup evaluate MDLM as generative model of language and as representation
model via fine-tuning on downstream tasks.
For language modeling likelihood evaluation, we conduct experiments on two datasets: The One Billion Words Dataset (LMIB; [8]) and OpenWebTer (OWT: [18]). We use the bert- pert-bass-uncased tokenizer for LMIB, and report perplexities on the test split. Models have context size of 128. For OWT, which does not have a pre-defined split, we reserve the last 100K documents as a held-out validation set and report perplexities on this set. We use the GPT2 tokenizer [45] for OWT. Models have a context size of 1,024. We utilize the transformer architecture from Lou et al. [33], which augments the diffusion transformer [42] with rotary embeddings [58]. MDLM was trained for 1M or 10M steps (corresponding to 33B, 327B tokens, respectively) on LM1B and 1M steps on OWT (which corresponds to 262B tokens). The corresponding AR bascline was trained for half the number of steps

---

Table I: Test perplexities (PPL: LMIB.
'Reported He al. [26]. Best diffusion value bolded.
Parameters
PPL (L)
Transformer- X Base [13]
0,46B
23.5
Autoregressive
OmniNetr [61]
100M
21.5
BERT-Mouth [64]
110M
$142.89
D3PM (absorb) [1]
70M
$76.90
Diffusion
Diffusion-LMI [30]*
80M
$118.62
DiffusionBert [26]
110M
$63.78
SEDD [33] (33B tokens)
110M
S 322.79
Autoregressive
Transformer (33B tokens)
22.32
110M
(Retrained)
Transformer (327B tokens)
20.86
Diffusion
MDLM (33B tokens)
$27.04
110M
(Ours)
MDLM (327B tokens)
$23.00
ensure similar number of tokens (details in Suppl. D.2). Full hyperparameters are given in Suppl. D.4. On OWT, we train with and without time step conditioning.
For representation learning, we pre-train models the C4 dataset [46], then fine-tune and evaluate models on the GLUE benchmark [65]. Models have a context size 128. We use the bert-base-uncased tokenizer for the representation learning experiments. We utilize the MosaicBERT architecture from Portes et al. [43], an extension of the original BERT architecture [15]. We pre-train a bidirectional MosaicBERT using MLM objective for 37B tokens of C4, well as a causal variant on the same data. We further fine-tune MosaicBERT model using the MDLM for 327M tokens, less than 1 of the pre-training data. We provide the full hyperparameters in Suppl. D.6.
Likelihood Evaluation On LMIB, MDLM outperforms all previous diffusion methods (Table 1). Compared to the SEDD baseline reported by Lou et al. [33], trained for 33B tokens, MDLM, which we train for the same amount, achieves a 17% improvement on the perplexity bound. Finally, MDLM gets within 14% of an AR baseline and continues improve with more training. We the same trend for models trained on OWT, a larger dataset, shown in Table 2 MDLM outperforms prior diffusion methods, closing the gap towards AR models. In Table 12 we find that models trained with and without time conditioning attain similar perplexities on OWT. Additionally, Figure 3 demonstrates the reduced variance we achieve from our objective, when compared previous masked diffusion models such as SEDD [33].
Zero-Shot Likelihood Evaluation We also explore models' ability to generalize by taking models trained on OWT and evaluating how well they model unseen datasets. We compare the perplexities of our MDLM with SEDD [1] and an AR Transformer language model. Our zero-shot datasets include the validation splits of Penn Tree Bank (PTB; [36]), Wikitext [38], LM1B, Lambada [4] IJ. AG News [68], and Scientific Papers (Pubmed and Arxiv subsets; [10]). Full experimental details are available in Suppl. D.4.
Test perplexities (PPL.; 4) on OWT for mod- els trained for 262B tokens. denotes retrained models.
Table
PPL (1)
ARt SEDD
17.54
MDLM consistently outperforms the SEDD diffusion parameterization. In some cases, c.g., for Lambada and Scientific Papers, MDLM attains better perplexity than AR. We hypothesize that these datasets are farther from OWT, and that diffusion models may be more robust to out-of-domain evaluation due to the baass unmasking- based objective.
<22.10
MDLMM (Ours)
<23.21
Downstream Task Evaluation We find that BERT tine-tuned with MDLM to be generative model results strong perplexities while preserving performance on downstream tasks. On the C4 validation set, the AR model attains perplexity (PPL) of 22, the pre-trained BERT attains PPL upper bound of 78 (evaluated using the MDLM variational bound), and BERT + MDLM-FT attains PPL. upper bound of 35. In Table 4, we further find that BERT MDLM fine-tuning has no degradation in downstream
7

---

Table 3: Zero-shot perplexities (4) of models trained for 524B tokens OWT. All perplexities for diffusion models are upper bounds.
PTB
Wikitext
LMIB
Lambada
AG News
Pubmed
Arxiv
AR (Retrained)
82.05
25.75
51.25
51.28
52.09
49.01
41.73
SEDD (Retrained)
100.09
34.28
68.20
49.86
62.09
44.53
38.48
MDLM (Ours) 95.26
32.83
67.01
47.52
61.15
41.89
37.37
Table 4: GLUE evaluation results. Evaluation measures (1) are Fl score for QQP and MRPC. MRPC.Spearman correlations for STS-B, and accuracy for the rest. For MNLI, we report match/mismatch accuracies.
MNLI (m/mm)
QQP
QNLI
SST-2
COLA
STS-B
MRPC
RTE
Avg
AR
80.94/80.78
86.98
86.16
90.14
33.43
84.32
83.88
47.29
74.88
BERT
84.43/85.35
88.41
90.46
92.20
54.81
88.41
89.16
61.37
81.62
+MDLM-FT
84.76/85.07
88.49
90.30
92.20
57.69
87.48
90.53
62.09
82.06
GLUE performance compared the BERT initialization. While the perplexity of our method is higher than the AR baseline, the downstream task performance is significantly better.
Semi-Autore Autreertrreeeeivvveeee. Modeling To test the SAR decoding algorithm presented in Sec. 4.2, we com- pare to SSD-LM [24] a diffusion model that was designed to generate blocks of text autoregressively. We generate 200 sequences of length 2048 tokens on single 3090 GPU and evaluate generative per- plexity under a pre-trained GPT-2 [45] model. The generated using blocks of 25 tokens (as implemented their pre-trained model) and the MDLM sequences are generated using =1 In Table 5, we find that in addition to achieving better generative perplexity, MDLM enables 25-300 faster SAR decoding relative to SSD-LM.
Table 5:
Semi-AR generative perplexity (Gen. PPL: 4) for sequences of 2048 tokens.
Gen. PPL (J)
Sec/Seq (1)
SSD-LM MDLM (Ours)
35.43
2473.9
27.18
89.3
SSD-LM sequences
5.2 Masked Diffusion DNA Models
We also explore applications to the generative modeling of biological sequences [14, 47] using a state space model (SSM) backbone [22]. Namely, we build the recently- propoo-ed Caduceus DNA language model [50], which uses as a backbone the data-dependent SSM Mamba block [21].
Experimental Setup We pre-train the encoder-only Caduceus [50]. which is an MLM, on the HG38 human reference genome []]] and perform fine-tuning using our diffusion parameterization. We use a context length 1024 tokens and follow Schiff et [50] for the experimental setup, other than learning rate which was reduced to le-3. See Suppl. D.7 for full experimental details. We assess both generative performance using perplexity and downstream performance on Genomics Benchmarks [20] across language diffusion paradigms and AR models.
Generative Performance We fine-tune the Caduceus MLM across diffusion parameterizations and compare perplexities against AR models. We report perplexity values Table 6. MDLM outperforms all other diffusion language modeling schemes.
Downstream Task Fine-tuning We perform downstream evaluation with the Genomics Bench- marks [20], a recently penchmaabenehmrrk
benchmark with eightae regulatory element classification tasks. As shown in Table 7, our generative fine-tuning paradigm preserves improves upon downstream performance from MLM pre-training. Absorbing-stad diffusion methods outperform Plaid across tasks except for the simplest task Human vs. Worm, where all methods have roughly the same performance. For tasks where the input a biased subsample of the full genome, we observe that the correlation between perplexity and downstream performance is weaker; see Suppl. D.7.

---

Table 6: Test perplexities (PPL: .) of generative fine-tume tine-tuning of the Caduceus MLM [50] on the HG38 reference genome. Best diffusion model values are bolded. Error bars indicate the difference between the maximum and minimum values across random seeds used for fine- tuning.
Params
PPL (4)
Mamba
465K
3.067 = .010
Autoregressive (Retrained)
HyenaDNA
433K
3.153 = .001
Plaid
507K
$3,240 .055
Diffusion (Retrained)
SEDD
467K
3,216 .003
Diffusion (Ours)
MDLM
467K
S $3.199 + .010
Table 7: Genomic Benchmarks. Top-1 accuracy (1) across 5-fold cross-validation (CV) for pre-trained AR Mamba, and a pre-trained Caduceus model fine-tuned with different diffusion parameterizations. The best values per task are bolded and the second best are italicized. Error bars indicate the difference between the maximum and minimum values across 5 random seeds used for CV.
Model
Mamba AR
Caduceus MLM
Caduceus Plaid
Caduceus SEDD
Cadaceus
Fine Tuning Objective (Parameter Count)
MDLM (ours) (467k)
(465K)
(467K)
(507k)
(467k)
Mouse Enhancers
0.763 (+0.008)
0.810 (#:0.016)
[=000
0.784 (+:0.058)
0.795 (#:0.099)
Coding Intergeaomic
0.897 (1:0. (100.004)
0.913 (1:0.003)
0.908 11:0.0031
0.913 (1:0. 005]
0.913 [::0. 003)
Haman vs. Worm
0.967 (#:0.002)
0.970 002))002
0.971 (+:0.001)
0.970 (00)
0.970 [+:0.003)
Human Enhancers Cohn
0.734 (+0.027)
0.737 0011
0.743 [=0.010]
0.746 [=0.015
0.743 [100006]
Haman Enhancer Ensembl
0.856 (220003)
0.907 (1:0.000)
0.885 [2:0 003;
0.905 [2:0 006)
0.899 (::0.004)
Human Regulatory
0.861 0 0,08
0.874 (1:0.003)
0.868 (=100010]
0.828 (1:0.037)
0.868 (1:0.004)
Human OCR Ensembl
0.806 (+00.005)
0.821 0000000
0.820 [=0.004]
0.816 (+0.008))
0.823 0081
Human NonTATA Promoters
0.926 (+0.008)
0.935 [0.014]
0.935 [=0000]]
0.935 (#0.014)
0.940 (=0.007)
5.3 Ablation Analysis
In Table 8, we can see the effect of our streamlined masked diffusion implementation. The improve- ments described in Sec. 3.5.1 allow us to greatly reduce perplexity of previously discounted models, such as D3PM (see the bottom row of this table, which is mathematically equivalent to the D3PM for- mulation). While most works assumed that D3PM achieves mediocre log-likelihoods, we show that is incorrect: our re-implementation almost matches state-of-the-art score-based methods. This intro- duces a new strong baseline that opens new research opportunities. Additionally, in Table 8, we ablate
Table 8: Test perplexities (PPL: 1) for MDLM ablations on LMiB. For the discrete-time mod- els, we use T = 1000. Standard deviation is measured over 5 seeds during evaluation.
PPL (P)
MDLM (47)
27.04+
w/o continuous time (43)
27.19++.07
& w/o carry-over (41)
28.56
& w/o zero masking (39)
28.51
different components of MDLM. We observe that the perplexity for MDLM trained with a discrete T= 1000 marginally worsens by 0. compared to MDLM trained in continuous time. Additionally, removing the "carry over" operation from the SUBS parameterization increases the perplexity by 1.5 points. However, further removing the "zero masking" operation does not lead to any meaningful change in perplexity. We provide further ablations for the continuous time formulation in the Appendix, showing in Table 11 that for a pre-trained model, at inference, increasing T yields better likelihoods.
Related Work
Comparison to D3PM Masked diffusion is a strict subset of D3PM [1]; setting Q1j.=a,j..+(
aqp)mmmT in their framework yields our forward diffusion. We improve over D3PM in three ways: (1) we adopt the SUBS parameterization: (2) this allows us to derive a simplified objective that analytically simplifies certain expectations to zero; (3) we adopt well-engineered training recipes that improve performance. Both (1) and (2) are possible because we focus on masking instead of developing a general discrete diffusion framework. Surprisingly, (3) has the largest contribution to performance.

---

Comparison to CTMC Most implementations of diffusion work best in continuous time. However, extending D3PM in this way requires computing the limit of the product of an infinite number of matrices Qr Qr- as 7 xo, which requires advanced CTMC theory [5]. Our work describes simple continuous-tig
formulations for the most common noise processes (e.g., masking and uniform TT, thus helping make an important part of the literature more accessible. In Suppl. C, we show that our
results are compatible with CTMC, using the rate forward matrix R2
ei (I- 11mm T) and the reverse
rate Ry .y) for the transition y yyy, where y y-y'e EV:
[y'] (y) mlyy,,,, (y) ly y) - 0² Ok+
(12)
Comparison to Score Estimation Score-based approaches to diffusion [55] extend discrete states, although they typically further build upon advanced CTMC theory. In particular. SEDD [33] optimizes an ELBO" that is a function of the score model, obtaining state-of-the-arr log-likelihoods among diffusion models. Our approach, however, is much simpler and does not require advanced theory. Furthermore, we can extract the score for MDLM (76), as demonstrated in Suppl. C.3, making compatible with various techniques designed for score- based algorithms, such as samplers [5], score parameterization [33], efficient designs of the denoising network [59), guidance techniques, and more.
Comparison to BERT Our work provides a principled way of making BERT generative when trained with randomized masking rates. Previous work generating from BERT used Gibbs sampling or ad-hoc methods [17, 32, 64]. The connection between BERT and diffusion was first made by Austin et al. [1]: their objective effectively involves unmasking. He et al. [26] additionally starts training from a pretrained BERT. However, both works use an objective that is similar to (9), which is less numerically stable than our objective (see Section 3.5.1). Austin et al. [1] mention in their appendix that their ELBO simplifies to weighted masking (MLM) loss similar to (8), but it uses more complex formula for the weights and limited the discrete time setting unlike our work. Furthermore, they do not train with that objective. Our work derives simpler expression for the average of MIMM losses, implements it, and obtains better likelihoods.
Comparision to Latent Diffusion LMs In contrast to this work, which defines diffusion over discrete structures, Plaid [23] and Diffusion LM [30] define Gaussian diffusion process over word embeddings. Zhang et al. [67] and Hu et al. [28] extend this approach to flow matching over word embeddings, enabling the design of faster samplers. Discrete Flow Matching (DFM) [6] applies flow matching to discrete structures, using cross coss-entropy loss as their training objective: - clogpe (x !:f. |z, .z..
Similar Chang et [7]. DFM's 's objective, while effective, is not weighted to serve as a proper ELBO. In MDLM, however, we derive a tight, principled lower bound on the log- likelihood..
work, Shi et al. [51] and Ou et al. [40] derive similar simplified objective for masked diffusion processes. While Ou et al. [40] start from a score matching perspective, we tackle this problem from variational lens similar to Shi et al. [51]. Similar to Ou et al. [40], we formulate efficient samplers in Section 4.1 by leveraging a time-independent denoising network.
Concurrent Works Concurrent
A key differentiation between our work and that of Shi et al. [51], Ou et. [40] is the cemi-autoregressivy decoding method we present in Section 4.2. While [51, 40] are restricted to sample sequences of a fixed length, we propose samplers generate arbitrary lengths of text like traditional language model. Furthermore, we establish the connection between our simplified objective and the masked language modeling (MLM) objective. As result, we endow BERT-style models with principled generation capabilities while maintaining representation learning capabilities. Whereas [51, 40] only evaluate on NLP datasets, we show that masked diffusion is also effective in modeling biological sequences.
7 Conclusion
In this work, explore masked diffusiin. With a well-enginerred implementation that supports simple variational objective, we attain state-of-the-ar diffusion perplexities on language benchmarks and demonstrate how to efficiently convert BERT-style encoders into generative models. Given we are working on language modeling, we carry any of the inherent risks and opportunities that come with this line of research.
'Lou et al. [33] mention that their ELBO can be derived from Benton et al. [4], Hanson [25] but do not provide an explicit derivation. To address this, we present a rigorous derivation in Suppl. C.2 using CTMC theory.
10

---

Acknowledgments and Disclosure of Funding
This work was partially funded by the National Science Foundation under awards DGE-1922551. CAREER awards 2046760 and 2145577, and by the National Institute of Health under award MIRA R35GM151243. Marianne Arriola is supported by NSF Graduate Research Fellowship under award DGE-2119899 and Hopper- Deapeer-ean/Bowers CIS Deans Excellence Fellowship.
References
[1] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete spate-spaces. Advances in Neural Information Processing Systems, 34: 1 (7981-17992) 2021.
[2] Pavel Avdeyev, Chenlai Shi, Yuhao Tan, Kseniia Dudnyk, and Jian Zhou. Dirichlet diffusion score model for biological sequence generation. In International Conference on Machine Learning, pp. 1276-1301, PMLR, 2023.
[3] Ziga Avsec, Vikram Agarwal, Daniel Visentin, Joseph R Ledsam, Agnieszka Grabska-Barwinske Barwinska, Kyle R Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R Kelley. Effective gene expression prediction from sequence by integrating long-range interactions. Nature methods, 18 (10): (10):1196110 19611996609900 1203, 2021.
[4] Joe Benton, Yuyang Shi, Valentin De Bortoli, George Deligiannidis, and Arnaud Doucet. From denoising diffusions to denoising markov models. arXiv preprint arXiv:2211.0355) 2022.
[5] Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. continuous time framework for discrete denoising models. Advances in Neural Information Processing Systems, 55:28266-282777 2022.
[6] Andrew Campbell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. arXiv preprint Stti:::20020099999 2024.
[7] Huiwen Chang, Han Zhang, Lu Jiang. Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11315-11311 1395-11325, 11325, 2022.
[8] Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Kochn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling, 2014.
[9] Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with seff-conditioning. arXiv preprint 22w::::2200022000000 2022.
[10] Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. A discourse-aware attention model for abstractive summarization of long documents. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Short Papers). 2018. URLhttp:: dx doi. org/ 10. 18653/8611v1 v1/n181-i1i/ 2097.
doi: 10.18653/v 1/n1 www.///nnn110/ 8-90
[11] Genome Reference Consortium. Genome reference consortium human build 37 (grch37.
Database (GenBank or RefSeq). 2009.
[12] Genome Reference Consortium Database (GenBank or RefSeg), 2009.
Genome reference consortium human build 37 (grch37).
[13) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv: 2019.
[14] Shachi Deshpande, Kaiwen Wang. Dhruv Sreenivas, Zheng Li, and Volodymyr Kuleshov, Deep multi-modal structural equations causal effect estimation with unstructured proxies. Advances in Neural Information Processing Systems, 10944, 2022.

---

[15] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 7Xii:1810.04805. 2018.
[16] Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, et al. Continuous diffusion for categorical data. arXiv preprint arXiv:2211, 1500099, 2022.
[17] Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predies Parallel decoding of conditional masked language models. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xi- aojun Wan (eds.), Proceedings the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 61 12 6121, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10065/////D1-166/00 URL httts::::/ httphhtpps//ccllee www.anttooooooooooieeeeee otg/D19- 1633.
[18] Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwehtext corpus. http: //Skylion007, github 1 http///pppeeeeeeerrrrruieeeeeeee 2019.
[19] Aaron Gokaslan, A Feder Cooper, Jasmine Collins, Landan Seguin, Austin Jacobson, Mihir Patel, Jonathan Frankle, Cory Stephenson, and Volodymyr Kuleshov. Commoneanvas: Open diffusion models trained on creative-commont images. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 8250-8260, 2024.
[20] Katarína Grešová, Vlastimil Martinek, David Čechák, Petr Simeček, and Panagiotis Alexiou. Genomic benchmarks: collection of datasets for genomic sequence classification. BMC Genomic Data, 24,1):25,203
[21] Albert Gu and Tri Dao. Mamba: tinear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.0073333 2023.
[22] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00390 2021.
[23] Ishaan Guirajani and Tatsunori B Hashimoto. saeeihood-bbssed diffusion language models. Advances in Neural Information Processing Systems, 36, 2024.
[24] Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. Ssd-Im: Semi-auttregrressvve simplex- based diffusion language model for text generation and modular control. arXiv preprint arXiv:2210.1 17432, 2022.
[25] Floyd B. Hanson. Applied stochastic processes and control for jump-diffusionn modeling, analysis, and computation. In Advances in design and control, 2007. URL https://api. semanticschole
orgo Prrg/CorpssID: 6689808.
[26] Zhengfu He, Tianxiang Sun, Kuanning Wang, Xuanjing Huang, and Xipeng Qiu. Diffusion- bert: Improving generative masked language models with diffusion models. arXiv preprint arXiv:2211. 1000292 2022.
[27] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[28] Vincent Hu, Di Wu, Yuki Asano, Pascal Mettes, Basura Fernando, Bjöm Ommer, and Cees Snoek. Flow matching for conditional text generation in a few sampling steps. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume Short Papers), pp. 380-392, 2024.
[29] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34:21 216-21676-27777, 2021.
[30] Xiang Li, John Thickstun, Ishaan Guirajani. Percy S Liang, and Tatsunori B Hashimoto. Diffusion-
Im improves controllable text generation. Advances in Neural Information Processing Systems, 35:4328 4343, 2022.
[31] Xuanlin Li, Brandon Trabucco, Dong Huk Park, Michael Luo, Sheng Shen, Trevor Darrell, and Yang Gao. Discovering non-monotonic autoregressive orderings with variational inference. arXiv preprint arXiv:2110.1577 2021.
12

---

[32] Yi Liao, Xin Jiang, and Qun Liu. Probabilistically masked language model capable of autoregres- sive generation in arbitrary word order. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 263-274, Online, July 2020. Association for Computational Linguistics. doi: 10. (0.18653/v1/2200 acl- mal-main.24. https httptttpps//aaaaaaaaaataaaaeeeee hta/aacaaattooogo 20200020....22000 acl- 24.
[33] Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion language modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310.1688 2023.
[34] Justin Lovelace, Varsha Kishore, Chao Wan, Eliot Shekhtman, and Kilian Q Weinberger. Latent diffusion for language generation. Advances in Neural Information Processing Systems, 36, 2024.
[35] Vincent Mallet and Jean-Philipe Philippe Vert. Reverse- complee-comp equivariant networks for dna sequences. Advances neural information processing systems, 34:13511-13522 2021.
[36] Mitch Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of english: The penn treebank. Computational linguisties, 19(2):313-33) 330-300, 1993.
[37] Chenlin Meng, Kristy Choi, Jiaming Song. and Stefano Ermon. Concrete score matching: Generalized score matching for discrete data. Advances in Neural Information Processing Systems, 35:33332-3445355 2022.
[38] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
[39] Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Michael Wornow, Callum Birch- Strch-Sykes, Stefano Massaroli, Aman Patel, Clayton Rabideau, Yoshua Bengio, et al. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. Advances in neural information processing systems, 36, 2024.
[40] Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data, 2024.
[41] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting the Association for Computational Linguistiies (Volume 1: Long Papers), pp. 1534-1534. Berlin, Germany, August 2016. Association for Computational Linguistie... URL http://www. aclweb. org/ anthaaattcc/a/aae/aa 1144.
[42] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings 2023.
of the IEEE/CVF International Conference Computer Vision, pp.
[43] Jacob Portes, Alex Trott, Sam Havens, Daniel King, Abhinav Venigalla, Moin Nadeem, Nikhil Sardana, Daya Khudia, and Jonathan Frankle. Mosaicbert: A bidirectional encoder optimized for fast pretraining, 2024.
[44] Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation, 2022.
[45] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
[46] Colin Raffel. Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., 21(1). jan 2020. ISSN 4435-
[47] Richa Rastogi and Yair Schiff. Semi parametric inducing point networks and neural processes. In International Conference on Learning Representations, 2023.
[48] Subham Sekhar Sahoo, Aaron Gokaslan, Chris De Sa, and Volodymyr Kuleshov. Diffusion models with learned adaptive noise. arXiv preprint 21311 2023.
13

---

[49] Subham Sekhar Sahoo, Anselm Paulus, Marin Vlastelica, Vit Musil, Volodymyr Kuleshov, and Georg Martius. Backpropagation through combinatorial algorithms: Identity with projection works. In The Eleventh International Conference on Learning Representations, 2023. URL
[50] Yair Schiff, Chia-Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and Volodymyr Kuleshov. Caduceus: Bi-directional equívariant long-range dna sequence modeling. arXiv preprini arXiv:2403.03234, 2024.
[51] Jiaxin Shi. Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K Titsias. Simplified and generalized masked diffusion for discrete data. Advances in neural information processing systems, 36, 2024.
[52] Phillip Si, Allan Bishop, and Volodymyr Kuleshov. Autoregressive quantile flows for predictive uncertainty estimation. In International Conference on Learning Representations.
[53] Phillip Si, Zeyi Chen, Subham Sekhar Sahoo, Yair Schiff, and Volodymyr Kuleshov. Semi- autoregressive energy flows: exploring likelihood-free training of normalizing flows. In Interna- lional Conference Machine Learming, pp- 31732-31773, PMLR, 2023.
[54] Jascha Sohl-Dicke Dicksteii, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference machine learning, pp.
PMLR, 2015.
[55] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribu- tion. Advances neural information processing systems, 32, 2019.
[56] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.
[57] Robin Strudel, Corentin Tallec, Florent Altché, Yilun Du, Yaroslav Ganin, Arthur Mensch, Will Grathwohl, Nikolay Savinov, Sander Dieleman, Laurent Sifre, et al. Self-conditioned embedding diffusion for text generation. arXiv preprint (tt.:2221.042230 2022.
[58] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint MxXiv:2104009880 2021.
[59] Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based continuous-time discrete diffusion models. arXiv preprint arXiv:2211.16750, 2022.
[60] Zhiqing Sun and Yiming Yang. Difusco: Graph-based diffusion solvers for combinatorial optimization, Advances in Neural Information Processing Systems, 36:3706-3731 2023.
[61] Yi Tay, Mostafa Dehghani, Vamsi Aribandi, Jai Gupta, Philip M Pham, Zhen Qin, Dara Bahri, Da- Cheng Juan, and Donald Metzler. Omninet: Omnidirectional representations from transformers. In International Conference on Machine Learning, pp. 10193-10202. PMLR, 2021.
[62] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[63] Clement Vignac, Igor Krawezuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pas- cal Frossard. Digress: Discrete denoising diffusion for graph generation. arXiv preprint Ntti:22009.17734 2022.
[64] Alex Wang and Kyunghyun Cho. Bert has a mouth, and it must speak: Bert as a markov random field language model. arXiv preprint arXiv: : 1990.000994 2019.
[65] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019. URL https::///// //wp///vee/.e.eee/////// net/forum? id=r J4km2R5t7.
14

---

[66] Yingheng Wang, Yair Schiff, Aaron Gokaslan, Weishen Pan, Fei Wang, Christopher De Sa, and Volodymyr Kuleshov. Infodiffusion: Representation learning using information maximizing diffusion models. In International Conference on Machine Learning, pp. 3033663366354 PMLR, 2023.
[67] Shujian Zhang, Lemeng Wu, Chengyue Gong, and Xingchao Liu. Language rectified flow: Ad- vancing diffusion language generation with probabilistic flows. arXiv preprint arXiv:2403.16993 2024.
[68] Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NIPS, 2015.
[69] Hannah Zhou, Avanti Shrikumar, and Anshul Kundaje. Towards better understanding of compleecom equivariance for deep learning models in genomics. In Machine Learning in Computational Biology, pp. 1-33. PMLR, 2022.
15

---

Contents
Introduction
2
Background
2.1 Diffusion Models
2.2 Discrete Diffusion Models
3
Simple Masked Diffusion Models
3.1
Interpolating Discrete Diffusion
3.2
Masked Diffusion
3.3
Bao-Blackwellizeed Likelihood Bounds
3.4
Continuous- Likelihood Bounds
3.5
Masked Diffusion Language Models
Inference and Sampling in Masked Diffusion Language Models
4.1
Efficient Ancestral Sampling
4.2
Semi-Auttoregressive Masked Diffusion Language Models
6
5
Experiments
5.1
Masked Diffusion Language Models
5.2
Masked Diffusion DNA Models.
5.3
Ablation Analysis
6
Related Work
7
Conclusion
10
Appendices
17
Appendix A Discrete time ELBO
17
A.I Generic case
17
A.2 Absorbing state
18
Appendix MDLM
21
B.1
no-BBlackwelizatioon
22
B.2
Continuous Time
22
B.3 Final Algorithm
23
Appendix C
Concrete Score Matching
23
C.I
Extracting the Rate Matrix
24
C.2 NELBO
25
C,3 Concrete Score for MDLM
27
C44 Reverse Rate Matrix for MDLM
28
16

---

C.5 Deriving MDLM's NELBO via CTMC
29
Appendix D
Experimental details
31
D.1 Likelihood Evaluation
D.2 Avg. Number of Tokens seen
31
D.3 Low discrepancy sampler
31
D.4 Language Modeling...
31
D.5 Zeroshot Likelihood
32
D.6 Representation Learning
32
D.7 Diffusion DNA Models
32
Appendix E
Additional Experiments
33
E.1
Noise schedule parameterization
33
E.2
Faster sampling with caching
34
E,3
LM1B ablations
35
E.4
Train NLL curves on OWT
35
E.5
Time-conditioning ablation on OWT
36
E.6
Unconditional Samples
36
Appendices
Appendix A Discrete time ELBO
This section is organized as follows: First, we derive the expressions for the true posterior and the approximate posterior as outlined in Suppl. A.1. We then simplify these expressions specifically for the case of absorbing state diffusion Suppl. A.2. Finally, we derive the expression for the ELBO for absorbing state diffusion in Suppl. A.2.3.
A.1
Generic case
Given the state transition matrix Qt, prior and the latent variables z. and Ze, where s <,t, let
= (1
(13)
A.1.1
g(z//22 (xx)
Thus, the marginals in (3) correspond to the following forward process:
q(z, |z,) -at
= Cat((z;;aa I (1-0 1 TT=,,) 2,)
Cat(zz;0 (11)))++11100 (1 71
m1 z.)
11,2,=1
=Cat(z,;0a),2, +(1 arps))/))) )T))
(14)
The above equation indicates that during each diffusion step from s - (, fraction (1 (1aaye) of the probability mass is transferred to the prior distribution
17

---

A.1.2 ((z,|z,,x)
Austin et al. show that the posterior corresponding to (14) is given as follows:
(Z x1,x)
=Ca
Zsi
(15)
2 Q x
which we simplify to the following:
g(z. x)
loc (1-(ss)s) 12 (1 []x
Cat
zz
2, [a,I+(1-au 1 _]xx
a 2i +(1 - 0rls) ( 1 T z 3 a - a.)
Cat
2, [0 + (1-04)
lac/s2e
01/ ) (1-
E a x (1 11---80 at.)
Cat
#1 xxx1
(16)
aiz, x 04
A.1.3 (z(2)(z.)
Austin al. [1] approximate the reverse process in the following manner:
Q11,21 OQ, xs (2,,t)
pa no(z.lz.) = - Cat Zsi
(17)
zi Qi No (Z(it)
where [xx is an approximation for x.
A.2
Absorbing state
For the absorbing state diffusion process have
A.2.1
((z,|z,,33
Since, 2 E (x,m), takes only values we consider the separate cases: z = and z, = m.
Case 1.
Consider the case = x i.e. 2, is unmasked, From (16), we have the following:
(/,,/2 (xx,xx=xxxx)
[ot].x+(1 0/j. 1m (1xx
=Cat
Z,i
02x x (1 (1 m
[aysa| O [a,x+ (1 (1-a,)mm )
-Car
Zsi
x m=0
at
OxX
CCat
x m00m0 a =0=0
ai
=Cat(z,;x)
(18)
Thus, we have the following:
g(2./2: -x,x) Cat(z.;x).
(19)
Case 2. Consider the case 2, By substituting and म ==min in (1,).(2z,/2,)))) simplifies to the following:
g(z, m =
(1-ay) (arjs (1-))(1-a,,)))) ) (1a -a,x)
Cat
18

---

(1 (1-ag) ) (a, 04)x
Cat Zzi
(20)
1 1-ar
Note that the above categorical distribution is non-zero for 2, {x,m) and zero for every other value. The non-zero values are specified as follows:
q(2 x/z; - m,,)
(21)
Oxt
q(2. m| m,x)
(22)
04
Combining Cases and 2, we get:
Cat(zo;Zz)
2: + m,
((2,/Ze.))=
11-a)mm+0 (a, 0x)x
(23)
Cat 2z;
1-00
2, m.
A.2.2 (z,
For the absorbing state diffusion process with π m, we want to simplify the (17). For this reason, we consider 2 cases: first, when z, of m (case 1), second, when f m (case 2).
Case 1. Consider the case when 2; + m. (17) simplifies to the following:
Q11 QQ, N6 (ze,t)
p0(2,l2++t)) =
Cat
Z
2, Qi Xo (z(!!)
Quy.2 2 Q Xol (Z(
Cat
Zz
[1zz]
X. (Zeit)
(oa,TT (o,,+ [aiz]] T xx (zc.t)
as a..mmlm (x0(2))))
=Cat
Zzi
[aq].2]] | la.x0 (Z,,t)
as ) (1, (Z,)(())()))
-Cat
Z,i
Oi (z (z,,)))
since
(1,xo(Zi,t)) 1, we have the following:
[ar)o2e 3 Ja,x8 (z,,t) (1-aa amm
=Cat
Zzi
at Xz (z,,t)
since 2, mm O, we have the following:
0i Oxx
=Cat
Z ae (o
=Cat(z,iz,)
(24)
Case 2. Consider the case when Z, = m. (17) simplifies to the following:
Q mo, (z())))
po [z, (2 =m) Cat
Z,:
m T Q₂ X (2t,))
Zo: Qujsm 3 Q Xo (o())!)) [Q:m]
Cat
x (Ze,t)
log.. + - (1-0g]a)) [ (xo((,))))
-Cat
(0.m+ (1 xx()),()
Jayj- (1
=Cat
04 (1 (z,r))
CCat
Zzi
19

---

o,moxo(21,t) (2) 0s as)m
Cat
Zz
(25)
at (xxo t),m (ai))(1100) (1-ar)
Note that above categorical distribution, we can obtain the values for pa(zs - x|2: - =m) and
po(z.- m iz -mmm m) which are follows:
(o, (0,-aa))xx(2,,1))))
Po l. = a (x(x((Z),)),,m (1 (1-a1)
(26)
(o = m 02 (x)
(27)
(x +(1 (2))
As sanity check, we can verify that (26) reduces to (21), and (27) reduces to (22) if our denoising network can reconstruct x perfectly, i.e. (x(z,,)=
Combining (24) and (25), we get the following expression for the reverse process parameterization:
Cat(z,,Zi)
Z1 o m.
po(Za|Ze)=
o, momo (1+ (a, (x (x) -aa)
(28)
Cat Zgi
2, m.
A.2.3 Diffusion Loss
For a given 7", Let LT B +. f1.... 1 (x) TKI. (1, (l,o) (2)) denote the diffusion loss.
We break down the computation of DKL (g(z, Ze, x (poolppll..ll into 2 cases: z, x (case 1) and = m (case 2).
Case I:
consider the case z, xx. Let's simplify DKI Dxu.(((a/l::xxx))100d
DKL (g /xx0
= DKL (z: |z())
From (23) and (24)
=0
(29)
Case 2:
Consider the case 2, Let's simplify DKL m =m)).
DRI (2. bhe = -m)
g(z 22 = m.x)
Σ (z(z//z==mxx/
(o Z: m)
gZz | - m.
(zz m. log po Zz Zz m)
2x
12z = 25 m,
=2 = =x/x-=xxxlog
pe = m
Simplify using (1) and (26)
+ +g(z, = m - ( -2,=110 m 22 m.x
= Po
m
Simplity using (22) and (27)
0g at log
0i (x)
m + (1
a4
0's 101
(1 0., (x= (1-a4)
+
(30)
as
(1= (1-a,))
Thus, DKL((g(z, (z,,x))llpppp 2t)) can be written in the following manner where (z,,)) evaluates to if z, x and mm) evaluates to if z, m.
(g (z,)) po 2z))
20

---

=
DxL((((s)zz - =xx, Ippo (z. = x) (2,,x) + DKL(O(E Z, mm.xilpppz.. Zz m (z, (31)
=0, From (29)
Given by (30)
Thus, we derive the diffusion loss, Lr, in the following manner:
LT = R 1E + ......) By(er)x))) xx)
DDL (g(2, [2,x (2
0x at log
at (xx Z1, ) +(1
= Le t, . Bx())x))
1 at
(1-0)) (2,,)
ass log
(1 (1
(1-a1)))
mm
(32)
Ckt
(1= (x0,))
Note that LT 0f is an unmasked token i.c. =.
A.2.4 NELBO
Austin et al. [1]. Sohl-Dickstein et al. [54] model a, as (0:)iE(1,....)) = 1- given latents 2 I..... However, in this paper, we denote the latents as &e(())....(TTT= and hence, the (t((s) are given as follows:
T-
(as))E(I.....))) =1
From Austin et al. [1], Sohl-Dicksteine et al. [54].
= (O:)kE (1..., = T+1
For T --1 latents
i-11 1 (ai)se(o.....1) 11
Offsetting the indices by
T+
i+1
(Q0())EE(0.....))
Switching the notations from o; (te(s)-
(33)
T+1
Consequently, from Equation 33, we derive that
T
(4((0) T+1
(34)
(4(T)) 00
(35)
Thus have the following:
T
1
20(0) ~
'T+1"
x
T+1
(36)
-Ca(;)))
(37)
(1(uun))Catt(
(38)
The NELBO (2) simplifies to the following:
E
logpo ( x x
Lr Compute using (32)
DKu.la(zuer) |x (x) (z(())
= using (37) (38)
Eg
- loggo xx
as at
log
at
1 a-a
(1 (1=a)))x((zt.t).x
Uss
log
(1 (1-a,) (1=a))(ax (1-
[1 { + mz)
(39)
1 a2
Appendix B MDLM
In this section, we show how SUBS parameterization can simplify the functional form of the NELBO defined in (39).
21

---

B.1 Rao-Blackwellize
We employ the RB techniques as described Sec. 3.2.3 simplify the NELBO (39) to (41) using RB2, and further to (43) using RB1.
B.1.1 Zero Masking Probabilities
Using "Zero Masking Probabilities" (RB2) from Sec. 3.2.3, we set (xo(Zi,,) .m) in (32) obtain the following simplified diffusion loss:
RB2 D1
E 3.3............. 1xp T
ax at
log
(z,
1 - at
(x (z))t),x)
Q4 ax
= -Eieft a.....1 Blta x) T)T
log (xo Ze t)
(z,.m).
(40)
1 a
The corresponding Rao-Blackwellized NELBO is given as:
Eg
- (x)
RRB Compute using (40)
+ DEL PxLla(ZZTT) /x 1po (Z(((r)))))
=0 using (38)
Eg
C2 1-a
log 1og(x((z+))).x) (2),
(41)
B.1.2 Carry Over Unmasking
Notice that the term (z) (40) intended reduce the diffusion loss zero when 2, =x. Now, we will demonstrate that, by applying "Carry Over Unmasking" (RB1) from Sec. 3.2.3, (z, (Z,m) can be removed from (40).
Recall that RB1 guarantees Xa (z(z),) = x when Z, = x. Thus, with the RB1 parameterization, the diffusion loss in (40) becomes zero for 2, = x. lxg( (2) mm)=0. =. Consequently, (z,, can be safely omitted from (41), yielding the following diffusion loss:
ax ase log (xg(xx(Ze,))).)
C1
E ( 4....... Bg(z, x)
(42)
1 1-ar
B.1.3 NELBO
Thus, we have the following NELBO:
Eg
- logpo 2x(0))
RB2 + RBI
+ DKLla(ZzurT (x) (Z(r))
Compute using (42)
Ousing and
= E.g. -logpo ( (xlz)(0)) ) +T
04 0t
log (xo (x),),)))
(43)
1-ay
Comparing (43) and (41). Note that due to RBI, logpe (x ()(0)) in (43) reduces to 0 every time ZH(o) = as explained in (45), However, this is not the case in (41), even though it has a functionally similar expression (43). Because of this reason (43) should lead better likelihood estimate and we empirically verify this in Table 8.
B.2
Continuous Time
B.2.1
Diffusion Loss
To derive the contnuous-time diffusion loss, limy- CRB2+ RBI (42):
L diftusionn
consider the limiting
000
diffusion lim CRB2 RBI T-x
22

---

=B ee ( (.. xx)
lim T at as
(og(xx
400
1-ar
log (og((x((z..))x
Using lim T(a: - (a,) =aa 74000
1 a-at
(44)
B.2.2 Reconstruction Loss
For the continous time case, from (36), we have
T
Zu(o) lim Cat T-000
"+1'"
x+ m
T+1
- -(()) (s)((;)))
(45)
Thus, the reconstruction loss reduces to in the following manner:
Lresons = loggo x
- (x(:(0(((2((0) =x)
From (45)
(xg(xx(xx((0))
(x,(
Due to "carry-over unmasking" (emmaskingxxxxxx////// (xx(((0))
=
O.
(46)
B.2.3 NELBO
Thus, we have the following NELBO:
y
logpo (x (0))
oo diffusion
DKLL (x)]l((
(46)
Compute using (42)
0 using (37) (38)
Eg
log(xx
(47)
1 1-ar
B.3 Final Algorithm
In Algorithm 1, we present the training algorithm for MDLM.
Algorithm 1 Training MDLM
1: repeat
2:
x q(xx)
Sample sentence. Sample a time step.
3:
t~
4: 2f =f=aa((z);aax' (111-o0)))y114 & Mask Each token xf independently to obtain the
latent 21:4
5: Take gradient descent step on
of 1-a1
V
Loog(x (2) (t)xx)
6: until converged
Appendix C Concrete Score Matching
In the previous section, we defined the discrete diffusion process as Discrete-Time Markov Chain (DTMC) with finite set of T states, [ 1). and a state transition matrix Q.. To derive the continuous-time ELBO, simply take the limit as -
23

---

In contrast, Campbell et al. [5] and Lou et al. [33] defined the discrete diffusion process a Continuous- Time Markov Chain (CTMC), where the forward corruption process specified by the rate change matrix R2 12 X which can be thought of as the instantaneous rate which one state transitions another. With this formulation, the forward posterior ( 2.) and the true reverse posterior q Z (x) can be expressed in terms of the rate change matrix as follows:
1
q(z y Z y by
+ Re
y +0
(48)
T
T
R 1
qlz, y Z y
+0 T
(49)
where ò is the Kroenecker delta and O ( represents higher order terms of Tr. In Sec. C.I. show how to express the rate matrix Rg in terms of the state transition matrix Qt. Lou et al. [33] propose continuous time ELBO for this process. They mention that this expression can be derived from Benton et al. [4], though they do not provide an explicit derivation. For this reason, we present a rigorous derivation in Sec. C.2 and further demonstrate that, under the SUBS parameterization in Sec. 3.2.3, this formula reduces to our proposed continuous time ELBO, given by (10).
For the remainder of this section, switch to the notation 4t/s (y"ly) to denote q(z, = yy 2z, = =y), (s ly) for q(z y y), and q(z, = x) for ge (y|x)) (x). aligning with the notation typically used in the CTMC literature.
C.1 Extracting the Rate Matrix
Here, we aim to express the rate change matrix R. in terms of the state transition matrix Q.. To do this, we first represent the forward transition Qrjs in terms of Qe and R2 separately, allowing us to illustrate their relationship.
Using (13), can write 9tjs as follows:
(y (y' ly) = -y'l [a I+ (1 I
[arp
= =Iy'T - Tappy + +11111111111 (11-a1))mm )ml y)
[aqps 1evve+1100000 (1-cr)))))) )m]
=aqssyy] y -1 x) m
(50)
Now let's analyze all possible combinations for the tuple (y',) yy)
1. Case xy'=x.y=x): Using (50), we find that (x = (x for the DTMC. By (48), we have as T xi, since the higher-order terms O (Tz) vanish in the limit. Thus,
2ejo we get:
lim 1+ [x,,,x +
lim Off/e T-000
T000
R((xxx)= lim T-000
(51)
ai
Case my'= Similarly, using (50) and (48), we have (t)a / m) = -01|s
and (x),(xyy m) R2 (m m) }. Thus,
lim T-000
R ( m
lim T-550
[1 - aaps
ag
[.
lim T(1-a))) = T0
(52)
Cz.
3. Case (y' m.y m): Using (50) and(48), we find 2/s q (m|m) = 1 and (m/(mm)) = 1 R, (m. + + Since these two expressions must be equal for any it follows that
R (m,m)
(53)
Note that when Re (m, is constant, the term O (+z reduces to zero, as it includes higher-order time derivatives of R..
24

---

4. Case (y'= V (xx)) observed. Thus,
-
In the context of absorbing state diffusion, these states are never
(iy'=x.yEV-(mm.))
(54)
5. Case (y' D'EV-(mo (VV. E (m,x)); In the context of absorbing state diffusion, these states never observed. Thus,
(ly'vV-(mxx).. (m,x))
(55)
Finally, we can express the forward rate matrix as:
R₂ ml ml)
(56)
can be seen that the columns of this matrix sum to zero, i.e.,
Σ R (yly)))= Eily'r)) y'EV
(57)
y'
which ensures that the probability mass is preserved in the forward diffusion process. Similarly, the reverse rate matrix RG can be written in terms of the forward rate matrix R2 as follows [33]:
( x 9
Ry
y'yy
Re(y' =
(58)
Rayy
y'=y.
vyyx
C.2 NELBO
Meng al. [37] introduced the term "concrete score" for the term qi (y' [x) (ylx) that appears in Re- Since this quantity is not directly accessible in the reverse diffusion process, we approximate it using neural network, Sp: V V, with parameters 0. The approximate reverse posterior Paj: can then be expressed in terms of the approximate reverse rate matrix Rg in the following manner:
y) 1 +
Pa/t(yly)) - R (y'
(59)
T2
So soly)y (y.y)) Egty so ( y Rs (
y' y'=y
R y)
(60)
where Se (y)y' denotes the approximate concrete score d: (y (x) Lou et al. [33] propose the following NELBO to train such a model:
Σ Rilyyyy)) soly)y y'
9o (x) (ygso(y)y) K
()((((x)) qe (ylx)
(61)
qe (ylx)
where ((a) aloga- They mention that this expression can be derived from Benton et al. [4], though they do not provide an explicit derivation. For this reason, we present rigorous derivation in the following section.
Proof.
Let's focus on the diffusion loss for this process. As mentioned in the previous section, the reconstruction and prior loss terms reduce to zero. The cint diffusion loss is given by:
lim TE, ( ......) y ymaa (.1x) [DLL (galely'l ly, (xxc)lleecyy((l)))
T-000
Uo1e (y' y
lim TE +. f........................... T-500
Pole (y'ly lx,)
Gole (y' ly,x)
-E.ejo.1,
lim " gole (y'I T0000
Paje (y))
y
25

---

Lsje (ylx,x) Pole (yly,))
Eastely Esteey (o Vy.x)log Lote (y'1,yyly,,,0 Pole (y' y)
lim Tq. x)ly
lim T
y'
Term
Term 2
(62)
Let's simplify these two terms separately. For the derivation, we'll welll on two key observations: In the limiting case as T -> oo, follows from (49) that lim (yly,) = and from (59) that y y.yx
Term 1:
lim (ylye (sje(y(yy.x xx)) T-000
Pole (yly,)
lim (xlx,)) = hence.
lim Tlog (xly(y)) T-000
Pyje(yly ,x,
The above term is Do x indeterminate form; therefore,
= lim (I -ogps T-00
Substituting 4st and Ps from (49) and (59), we get:
1 T
7
1
lim T log 1+ Ru (y +
- log 1+
R
+0
T-200
T2
Applying the Taylor series expansion for log( +x), we get:
lim T [ +
1
RC 7 0
1
T-000
T
7
1
-Rr(y.y)+ + lim TO T+00
R2
(y.y) T T-200
lim TO
T
7
Iim TO
= we get:
T-x
T
= -R((y.y)
Rd (v.y)
Using (58) and (60), we get:
(x(((x))
[x)
Σ Re(y.y')
sy())
(63)
(x(y)x)
y'Zy
Term
4sje (y' (i x)
lim T Σ (oocy'll,x)lan
T
Potr (y' y
Y'
Substituting aste and Pate from (49) and (59), we get:
xx(x'x)) Ri [
lim T Σ T-000
(x(y'(x))) (x(xxx)
+ R 1
1
log x
T
so soly)y R(yy))
y'
(x'''' x) (x(y(x)
1
(r'(())) <dylx
R
lim Σ
Re Ry +TO
log
T y'
T
soly)y B
lim TO T-x
= 0. we get:
7
9x()'(x)
Rs (y log g Re(y-"")
(. xx)
(x
(y)y Bely:y")
26

---

E (x()(xx)) Re Rr.r')))
log 7x(x(x)) g)(x(x)
-ooss (yly))
(64)
y' 9c(x/x)
plugging (63) and (64) into (62) yields the NELBO as proposed in Lou al. [33]:
(x('(xx)
Σ
Rely.y')
ss(y)yy')
y'/y (x(y/x)
(x('((x) x)
log (r xx)
Re(y.y')
-logso -oggs(y))' (y)y'
(x(x/x))
4: ( x)
Fy
(x(1xx)
-1 y'Fy
Σ Roly
+yo(y)
(x(y(x)
(<('(()) xx)
9r (r(''>>) [x)
()(y'(x)
log
logso logso())))'
(x x)
2:. (y(x)
q: (r(y/x) x)
(o x) (y x)
(c((''(x)) [x) 4: 4: (x)
Σ
Rely,y')
so(y)yy
logsa (v)y +K
(65)
y'Fy
where K = aloga-a, a. This concludes the proof.
C.3
Concrete Score for MDLM
Given latent variable z, and the output of the denoising model, xp(z),() parameterized using SUBS, we aim to recover the concrete score (z, E (IR +[ Note that so y is the ratio p.(y] in the reverse process. Since x, approximates x, (y = ge (2, therefore,
P((x.)
(y)))
2:
so (Z()y
(66)
(z.)
Oi (Zr X (Zr,())
To obtain the score, we first compute (y (y))x(z,,()) for all possible and Using (4), we derive the following expressions under the SUBS parameterization:
q: (x0 (x)
(67)
< (m.z.) (z)
(68)
(69)
qe (y - -
(70)
Plugging these into (66), get:
So (z, = m) m) ymm 1-a; (x)()))))))
Using (70) and (67)
(71)
So (Z, = (+==my==> m)y=m= =1
Using (70)
(72)
8o (0 m) m)y=m= 1-a1
Using (69) and (70)
(73)
8o
Using (69)
(74)
So (z, y (m.z.) =0
Using (68) and (69)
(75)
These can be consolidated into the following expression:
ar
1-at 8z,
so(Zz)y y
8y 82
X (z(z,))
m +2:
(76)
1-
02.
27

---

C.4 Reverse Rate Matrix for MDLM
We can formulate the reverse rate matrix for MDLM using (76) and (56). Recall that the reverse rate matrix (y'yy) .y) is given by:
Rely', y) So so R,yy'') Eytyso(yly (y.)) y'=y.
y'y
Let's s examine the cases where y= m and y m.
Case y m: For fm, m, the reverse rate Re yy is given by:
Roly' fm.y = y'mm
Using (71) and (56), we get:
0t
(xo (xo(2e.)))))
04
1 1-ay
aq
of
(xo(2)(())))
(77)
Fory'= = m, the reverse rate (y',y m) is given by:
Rely' = mmy=m)
Re(ryy=))
8tm
Using (77), we get:
Σ 1-a (z)()))))))) 8/m
0/2
(xo(2,!(){))
8pm
"zero-masking pyen-ita in Sec. 3.2.3
Σ (xx(xx)))))) =1; hence,
y=m
of
(78)
1- Oz
y m: For yor'' fy we have:
R. & (y.m)
= (y 4 Re R (y.m)
from (56)
-0
(79)
For y = =m, we have:
Day'=myy+i m)
Sy m Rely + myy'=
0 from (56)
=0
(80)
Thus, for y'= yy, we have:
Ruly -yy #m)
= Re((s.
3$y
28

---

my
Σ
R
from
y m)
from
(81)
Summarizing (78), (79), (80), (81) we have:
of - (xxoxy
y'ymy=n =m
I-a:
R .y
T-a:
y=myy=m
otherwise.
a/
[y']
(82)
0tt
C.5 Deriving MDLM's NELBO via CTMC
Now, we aim to show that substituting the expression for the rate matrix R₂ terms of state transition matrix Qt from (56) into (65) and switching from core-parameterizztion to the SUBS parameterization (Sec. 3.2.3) yields the simplified NELBO for MDLM as given by (10). We present the proof below. Recall that the term (a denotes the dot product of two vectors and b. When a and b represent two one-hot vectors, this quantity evaluates to if = and otherwise.
Proof. Recall that for absorbing state diffusion, y takes only two possible values, i.e., y E (x,m).
Thus, we expand (65) as follows:
(v,x)
Σ Re(xxy') 5o (x( y'
Le (y((())) (x(x(x)
(x) +K
ge (x(('(xx) (x)
(x(x(x) ))
(o(ylxx)
(r(y'' |x) qe (m(
+ m)
Σ Rm so(m)y y'
logso (m)
q (m|x)
:. (x,y'' #x)= =0)=00mm from (54), we get:
(y x)
logso (m) +K [2 (x)
-Euego.1.yyqq (y, m)
Σ
Re(myy') so(m)
m
(m(m)))
y'm
Substituting Ri y o m from (52), we get:
oe
(x'llxx) logso (m)y +K (x(''(x)) (x)
ge (m(m)) ))
Exeoo (10 m)
(0 (m)))))
y' ar
qi (m|x)
qe (y'yl xx)
Σ K (x()'(x)
= () (y.
Σ (m)))
logso (m)y
at
ymm at (m/x)
qe (m(xx)
y'
y' y'mfm
Term
Term
Term 3
(83)
Term 1:
Σ (o(m) yfm
Using (67), ) we get,
Σ 1-a; (xe (m),))))
ymm
29

---

Σ (xo(m,(),)) ymm "zero-masking probability" in Sec. 3.2.3
E
(xo(m,)))x) = hence,
y
(84)
1-ac at
Term 2:
ge (yly') x)
Σ
logso (m)y
[m x)
y'y
[x
2t (x('))x)
logso (m)
logso (m)y'
(.(m) x)
9e (m|x)
y x/
(y [x] for y' {x,m) from (4)) we get:
(x(xxx) logso (m)x
Using (4), we get:
0x
logso(m)s
1-a
Using (67), get:
log
(xg (xx,
1 at
1-a; ai
log
at
at
log (m,)),x
(85)
Ctt
Term 3:
(x('(x) xx)
Σ K
(m(m x)
y m
(r xx
log (x(((xx)
[x)
ao (m(m)
(m(x) [m[[)]
fm
4: (x(x)
log qe (x(x)) (x(x)
log 9c(y812)) (<(x((x)
(m xx)
9: (m|x)
%o (m/x)
2e (m/x) (m(x) (mlx))
y' m
"y x) 0 for y' (xxm), get
(x
ne (x(x)
2r( xx
log
qe m qe (m/x)
x
x)
@: (m[x)
Substituting the values using (4), we get:
Ct1 log
at
a2
(86)
at
1-at 1 1-a;
Substituing (84), (85), and (86) in (83) we get,
Exg m)
at
0t
04 log D-
Ot
log (xo (m.t)xx)
1-ai
T-0;
1 1-a1
1-a;
Q4
at at-
+
log
1-og
T-at
1-a:
af
ai
=E (y. m)
log (mn,)
1 1-a;
a
= =E 1l (y m))
Ixo (m)
a
30

---

Under the SUBS parameterization, log(xo (2),1))))
when x, =x; hence (y.m can be dropped:
0/
= 1 1-a4 1zx
(87)
This concludes the proof.
Appendix Experimental details
D.1 Likelihood Evaluation
We use single monte-carlo estimate for to evaluate the likelihood. The low discrepancy sampler (D.3) plays key role in reducing the variance of the estimate seen in Table 8.
D.2 Avg. Number of Tokens seen
Given training_steps batch_size, context_ the number of tokens seen by the AR model is given as:
training_steps Xbachh wtt.cch_siieeecttttttteeeeeee wwwteetteeteeeeeeeeeeeeeeeeeeeee
(88)
However, this expression doesn't 't hold for a diffusion model, since at each training step, fraction of the imput tokens are masked before being fed to the model. Pm the probability a token being masked at a timestep f. For the log-linear schedule in our experiments, Pm. = t. Thus, the expected number of tokens seen by the diffusion model
E, B.-......aaaaiinngggg
21.
Ppm =t
0.144 0.5
(89)
size = 512, and a context length of 128. Like [33] we use a log-linear schedule and hence the number of tokens seen by our model is = 33B (89). Similarly, MDLM trained for 10M steps, saw 327B tokens in expectation. The corresponding AR baseline was trained for 0.5M and 5M steps to ensure a similar number of tokens was seen.
LMIB. Following [3, 26]. we train MDLM for 1M training steps with batch_size=
OWT. We train SEDD and MDLM for 1M training steps with batch_st size = 512, vontext_lengthe = 1024, and linelinear schedule. Hence, these models saw 262B tokens dur- ing training. Similarly, the AR model saw the same number of tokens when trained for 0.5M steps with the same batch_ _iie and context_ lennghh.
D.3 Low discrepancy sampler
To reduce variance during training
low oisdiiccypynccy sampler, similar that proposed in Kingma et al. [29]. Specifically, when processing minibatch of N samples, instead of independently sampling N from uniform distribution, we partition the unit interval and sample the time step for each sequence i (l,...,N) } from a different portion of the interval f; ~ U|i=l timesteps are more evenly spaced across the interval [0.1], reducing the variance of the ELBO.
N: ]. This ensures that our sampled
D.4 Language Modeling
For our forward noise process, we use a log-linear noise schedule similar to Lou et al. [33].
We detokenize the One Billion Words dataset following Lou et al. [33]. whose code can be found here We tokenize the One Billion Words dataset with the sart-bass-uncaseee tokenizer, following He et al. [26]. We pad and truncate sequences length of 128.
31

---

We tokenize Open WebText with the GPT2 tokenizer. We do not pad or truncate sequences we concatenate and wrap them length of 1,024. When wrapping, we add the eos token in between concatenated. We additionally set the first and last token of every batch to be eos. Since Open Web Text does not have a validation split, we leave the last 100k does as validation.
We parameterize our autoregressive baselines, SEDD, and MDLM with the transformer architecture from Lou al. [33]. We use 12 layers. hidden dimension of 768, 12 attention heads, and a timestep embedding of 128 when applicable. Word embeddings are not tied between the input and output.
We use the AdamW optimizer with batch size of 512, constant learning rate warmup from 0 to learning rate of 3e-4 for 2,500 steps. We use a constant learning rate for 1M, 5M, or 10M steps on One Billion Words, and IM steps for Open WexnWetText. We use dropout rate of 0.1.
D.5 Zeroshot Likelihood
We evaluate zeroshot likelihoods by taking the models trained Open WebText and evaluating likelihoods on the validation splits of 7 datasets: Penn Tree Bank (PTB; Marcus et al. [36]), Wikitext [38], One Billion Word Language Model Benchmark (LMIB; Chelba et al. [8], Lambada [41], AG News [68]. and Scientific Papers (Pubmed and Arxiv subsets; Cohan et al. [10]. We detokenize the datasets following Lou et [33]. For the AG News and Scientific Papers (Pubmed and Arxiv), we apply both the Wikitext and One Billion Words detokenizers. Since the zeroshot datasets have different
conventions for sequence segmentation, wrap sequences to 1024 and do not add eos tokens in between sequences.
D.6 Representation Learning
Following Devlin et al. [15], we evaluate on all GLUE tasks [65], but exclude WNLI.
We pre-train MosaicBERT model C4 [46] for 70k steps, corresponding 36B tokens. We pad and truncate the data to 1 128 tokens using the bert part-baase-ncaased tokenizer.
MosaicBERT [43] has similar architecture to bert bart-bass-uncaseed and has 137M parameters, 12 layers, 12 attention heads, a hidden dimension 768, an intermediate size of 3072, and ALiBi attention bias [44].
For pre-training we use the following hyperparameters: A global batch size of 4096 with gradient accumulation, a learning rate of Se-4, linear decay to 0.02x of the learning rate with warmup of 0.06x of the full training duration, and the decoupled Adam W optimizer with le-5 weight decay and betas 0.9 and 0.98.
For diffusion fine-tuning use AdamW with a warmup of 2,500 steps from learning rate of 0 5e-5, betas 0.95 and 0.999, and batch size 512. We train for 5k steps total, corresponding to 32M tokens.
For GLUE evaluation, use the HuggingFace script found here We use the default parameters for all datasets, except for a batch size of 16, which we found helped with smaller datasets. This includes the default 3 epochs for all datasets and learning rate of 2e-5.
D.7 Diffusion DNA Models
Dataset We pre-train the Caduceus MLM [50] on the HG38 human reference genome [1]]. Fol- lowing Schiff et al. [50], we use character- / base levr- tokenization. The dataset is based on the splits used in Avsec et al. [3]: the training split comprises of 35 billion tokens covering the human genome. This consists of 34,021 segments extended to a maximum length of I, 1,048,576 (220 segments). 5-fold
We maintain constant 20 tokens per batch. For the Genomies Benchmark tasks, cross-validation where we split the training set into 90/10 train/validation splits.
The Caduceus MLM uses as a backbone a bi-directional variant the data-dependent SSM Mamba block proposed in Gu et al. [22]. This architecture is ideal contains inductive biases that preserve reverse complement (RC) equviariance, respecting the inherent symmetry of double-stranded DNA molecules [35, 50, 69].
Architecture
32

---

Training details All models are pre-trained on 10B tokens (10K steps) and fine-tuned on a generative objective for an additional 50B tokens (50K steps). We use a global batch size of 1024 for context length of 1024 tokens. Downstream task fine-tuning is performed for 16K steps ( (IB tokens).
For performing Caduceus MLMM pre-training, follow Schiff et al. [50] for the model size configura- tion, and hyperparameter selection. For pre-training, we use a fixed 15% mask rate as done in Devlin al. [15]. Of the 'masked' tokens, 80% are replaced with [MASK]. 10% are replaced with a random token from the vocabulary, and 10% are left unchanged.
For fine-tuning all Mamba-based models (including Caduceus) on diffusion objectives, we lower the learning rate from 8e-3 to 1e-3. For fine-tuning HyenaDNA [39], we lower the learning rate from 6e-4 to 5e-5. Similar to Gu et al. [22], Schiff et al. [50]. we found that Mamba- based models were robust to higher learning rates. We exclude timestep embeddings for all Diffusion DNA experiments, as we show it has minimal impact generative performance (see Table 12, Suppl. E.5).
We perform downstream task fine-tuning on the final hidden state embedding from pre-training. We perform mean pooling across the sequence length, which may vary from 200 to approximately 2,000 bps. We report the mean and + on max/min classification accuracy over 5-fold cross-validation (CV) using different random seeds, with early stopping validation accuracy. For each task, we do hyperparameter sweep over batch size and learning rate and report the values of the 5-fold CV for the best configuration.
Genomic Benchmark Task Distributions We use subset of the Genomic Benchmark tasks with an emphasis on tasks from Human data. The positive samples for each dataset were generated by selecting samples that were annotated, either computationally or experimentally, in previous work (e.g enhancers, promoters, open chromatin regions (OCR)) [20]. These annotations each correspond to subsets of the genome of varying sizes that may exhibit different distributions of DNA than those observed globally over the reference genome. Due this, the observed dataset may have different distribution than the data used for pre- training and calculating perplexity. This might in turn lead to a case where perplexity and downstream performance may necessarily correlate.
Appendix E
Additional Experiments
E.1
Noise schedule parameterization
As described in Sec. 3.4, the ELBO is invariant to the functional form of aq, To demonstrate this, we evaluate MDLM, initially trained using a log-linear schedule on OWT, by replacing the noise schedule with various other noise schedules mentioned below. Following prior works [1, 33, 54], we parameterize a =-=((t)) where a (t): [0,1] R+ Various functional forms of ar (t) are listed below:
Log Linear [1, 33, 54). The log linear schedule given as:
(1) = llg (1-t)
(90)
Cosine Squared schedule [24]. The Cosine Squared schedule is given as:
(t) log cos (~ (1 (1-))
(91)
Cosine schedule. The Cosine schedule is given as:
π (1 (1-() a (t) - log cos
(92)
Linear. The Linear schedule is given as:
= =t)==
(93)
where max is very large number. In our experiments set it to 108.
E.1.1 ELBO Invariance
The function o invertible due to the monotonicity assumption Sec. 3.1, and so we can perform the following change of variables in (10): 7 = log(1 aa). Let f [0.1] + R- be a function such
33

---

that y (t)). Note that a, goes through a monotonic transformation to obtain hence, 7 is also monotonic in since a monotonic t. This implies that the function f is invertible. Let ( (7). Then, we can we have the following diffusion loss:
0f (2),
NELBO = Eng
1=0 1 1-ai
d log(1-a;] dt
=-By
1o x)
=-Eq
log (zc.t) [f(]ddt
Substituting (lgg (1-)))
dt
=-Ra Eg
log(xo (2) (1)),
Change of variables
f
=-Ey
log(xo 2 f (7)),
=
= --By
log(Xo 1),
x 7) J (7))
(94)
This new formulation demonstrates that the diffusion loss is invariant to the functional form of a In Table we demonstrate empirically that noise schedules with different functional forms evaluate to the same Likelihood which is consistent with our theory. However, different schedules lead to different per data point variance. Notably, the log-linear schedule exhibits the lowest variance among all the noise schedules considered.
Table 9: Likelihood in bits per dimension (BPD) for different noise schedules on OWT dataset, is reported along with the mean and variance associated with each noise schedule per data point. We empirically observe that noise schedules with different functional forms yield the same likelihood, consistent with our theory Sec. 3.4; however, different schedules result in different variances.
Mean
Variance per datapoint
Log Linear (90)
3.30
1.81
Cosine (92)
3.30
3.30
Cosine Squared (91)
3.30
3.30
Linear (93)
3.30
7.57
E.2 Faster sampling with caching
In Figure 10, we compare the wall clock times of variaous methods: AR, SEDD, MDLM with caching. and MDLM without caching for generating 64 samples on a single GPU. When sampling in batches, change of token would necessitate a call to the denoising model. Therefore, smaller batch sizes have a lower likelihood of a token being unmasked. This might lead one to prefer generating samples in smaller batches, opposed to using larger batch size that fully saturates the GPU. Table 10 shows that generating samples with a batch size of and using caching is twice as fast as generating samples without caching while fully utilizing the GPU. In Fig. 2, we observe that MDLM without caching yields samples that consistently get better generative perplexity than SEDD. For T = (5k,10 10k), both SEDD and MDLM get better generative perplexity than the AR model.
Table 10: Wall clock time reported in minutes to generate 64 samples single A5000 GPU. T 5k(4)
T'= 10k((L)
SEDD MDLM
85.3
155.2
70.3
127.9
+ caching
40.1
60.4
34

---

Generative perplexities across sample times OpenWebText
90
caching
caching (Batch 8)
80
caching (Batch
Generative perplexity
(Batch 166
(Batch 16)
70
60
40
30
2k
4k
6k
8k
Sampling wall clock time (s)
Figure 2: Generative perplexities across wall clock time for generating 64 samples on OWT using single 32GB A5000 GPU are compared by varying T' {100,500, 1000,,000,5000 500,, 10000] in the reverse diffusion process. The samples are generated in mini-batches with a batch size of 16 for AR, SEDD, and MDLM without caching, as it is the largest batch size that fits on this GPU. For MDLM with caching, we vary the batch size.
E.3 LM1B ablations
assess the importance of our continuous-time framework by performing ablation on diffusion steps T. In Table 11 we compare NLL and PPL under continuous and discrete T in MDLM. We find that NLL consistently decreases as 00.
Table 11: Discrete continuous time evaluation for MDLM w/o time-conditioning OWT. MDLM was trained with T co. We report test perplexity for a discrete T.
T
PPL(<)
DO
23.05
10
42.18
20
30.70
50
25.77
100
24.35
200
23.66
500
23.26
1000
23.15
E,4 Train NLL curves on OWT
In Figure 3, we show that MDLM achieves lower variance loss during training compared previous diffusion language model, SEDD. Training is performed over MM steps on OWT (which corresponds to 524B tokens).
35

---

Train Negative Log-Likelihood (NLL) on OpenWebTex
Methed
5.5
SEDD
AR
MDLM
4.5
NLL
3.5
2.5
0.2M
0.4M
0.6M
0.8M
1M
Train steps
Figure 3: Train negative likeliihd (NLL) curves across 1M gradient steps (524B tokens) on OpenWebText [18]. NLL is logged every 1K steps without value smoothing.
E,5
Time-conditioning ablation on OWT
In Table 12, we assess the importance of time conditioning in MDLM on OWT. We observe that time-conditioning has minimal impact on perplexity. Training performed over 1M steps on OWT (which corresponds to 524B tokens).
Table 12: Ablation on conditondii in MDLM on OWT.
Method
PPL
MDLM w/ conditiing
23.21
MDLM w/o time-conditioning
23.05
E.6
Unconditional Samples
Here, present unconditional samples generated by MDLM trained on OWT with context length of = 1024 for = 11000,
E.6.1 T= 1000
Example 1
< </endoftext/> 17-10 victory and trip to the playoffs.
The last wildcard seed: New York Jets, Houston and the last potable playoff spot. The last-second home wins: New Orleans and Carolina, 21-21.
The Saints finish sixth with the highest regular season (42) NFC wins. They lost 14 their 13 ganes the conference playoffs.
The Cardinals were in Group in Round with Game 2, Round 3, Round and Quarter Game 5, but they made their last trip to the playoffs off North Carolina on the road as even North Carolina.
True to their reputation, the Cards swept the Saints in the first round, but knocked it out at home. No Panthers went to the playoffs more than the Saints.
36

---

Den Jean no lenger is the South Carolina Panther.
The Cardinals thought that provided that he had chance be NFL player.
"I did, said defensive end Lorenzo Williams with laugh he ex- ited his car at the airport. "Also, I won that game."
KC win brings Carson back home.
Griffin made promise Sunday to never exactly give up the dunk. Although he failed to score 40 points in the playoffs, he has had better luck in them this year.
With turnovers and fumbles returning, he has to play out because the team doesn't t trust him. He's s long years of injuries, turnovers and calls because he knows he can play that way for everybody.
Griffin is no stranger to Saints fans.
"Players want him to know them, someone from South Carolina said after coming out aqainst the Panthers in their best home home Week 7 win Sunday in an 11-9 rout. South Carolina did win its final three and passed the Saints, 24-1.
Although the Cardinals are in the South, they am step behind.
They still have little time left to take down the West Coast wild card. There is no chance they get another victory.
The West was out by Beshear in their first round games last season, losing by 63 to the 49ers.
The outcome will be tough.
"Now we' re so close, let's s figure out the time to win, " Brees said. "We still have a few games left; I'm glad about that."
South Carolina takes the revenge.
When asked about his second time since Super 4, Brees shot back that he understood.
"You doubt the answer but I think that was no-brainer. In time, you try to prove an answer wrong, Brees said. "I think his ability will be as cool as Julio Jones' ability, but having that time [out] to my season was difficult overall. I did what I was ex- pecting do. Hopefully they'11 tell to try again."
After their late win, the NFL calls the Saints reschedule must try.'
"Because Saints, those who am there still say it, "focus on defense and, offense is defense."
ESPN said Saints's a star receiver Dashon Jeffishard, turning heads long passes and connecting with open defenders, already had their 20-yard overall score from the field. when Jeffishard finished with three passing he set up. He obviously had difference; it was his first snap-off.
With his changes in his starting lineup, Brees was just hoping they had little to prove against Carolina over the weekend.
"I felt like we didn't have quite enough focus on and there was so puny coverage that I wanted more of our guys at the same position so we'd d our game," he said.
Brees said South Carolina was well linebacker.
37

---

"If that's part of it, if you' re going to try and stay with what you' re going [out with]. What's 'd you want? Smart play, Brees said. "What would you say? That you' re al ways ready to play. So you're going out there strong and ready to go to play football."
That said, New Orleans was damned shy when it came to Carolina.
"My guys admit to feeling it little bit [Sunday but] say to them that they always knew, 'I don't think that was necessarily how I would beat you, that will give them their confidence, Brees said.
was really hard because I've obviously learned a lot detail about how to deal with everyone and as hard as I have to be, I also feel part of the stuff that they' ve been through on the team, like they're re still going to go through things they know are somewhat right, but they feel a lot of pressure so it's got to be important to get it right now to get it in the future."
Could all ask for more roses?
Let's s just take a slip, South Carolina, and face the NFCK lendoffett
Example 2
</endoftext| Memorial Hospital.
Valia and Hill had been working with the Coast Guard in response to public questions, and when they were reached couldn't comment on the new information, Chapman said.
People referred to Valia during the years from Hill's family in Ants, and she cut in contact with their family and friends in 2016.
"Each day they stepped on the bus, when they left they saw me on TV, she said.
After separating from their family recently, Valia, 32, also moved into Richmond house last October.
Read Share this story: http://usat: 1ossstt1//NNN42Y4444 / CCIVIL C. "Marky" Hogan has been charged with homicides with a few days remaining after the April 2 purdade high school shooting where an undercover medical examiner and two other state and Illinois police officers was using heroin to go see therapist.
DICEZ TV's Zach Putler reported Tuesday that Hogan was charged with felony drug possession by the Chicago Police Department at the pre- liminary hearing on Monday. Putler interviewed on Monday, Author ities could offer a limit until Cook County takes Tuesday afternoon or they have to assign plea agreements.
Dogan said in Wednesday in Chicago that he believes the people who used him as a legal tool in the killing and fired employees hired for suffering also participated.
conference he made during conference call
He said the couple's request to an attorney Monday will let the charges finally play out. Their lawyer did not respond Wednesday.
Dogan would not give away to possibility that he speculated in a statement that he would escape and return unless shot.
Chicago police initially said the other drug charges failed to raise enough evidence to establish why the killers were charged last year, raising the possibility that the drug mix contributed to a reason for their arrest. But new statement was made Tuesday by man
38

---

who worked as the shocter in his unit on campus, and suggested the charges might related to his work at university supervision.
His attorney and the university's a president last week signaled that the incident of the bat gun was not police investigation at Wednes- day's conference.
Michael Durin, Illinois State University spokesman said he did not meet with university officials at the conference, and that univer- sity officials don't have any updates yet either.
"The fact that the Defendants were charged a major factor in why it would get this much attention, the university spokesperson said. "Given that all the matters are not being resolved for months and months, any new specific information and other concerns they may be tasked with investigating now are understandable."
After city police began looking for evidence in connection to Hogan's s April shooting, Durin said he had not neticed it until the Chicago Police Department found a person who was producing a bind gun on Illinois State campus. That same department found that 14 officers shot and were injured during standoff, but it led to the launch of a combination of unrelated and related investigations leading to homicides charges in May 2015.
from special school investigators suggests it had identi- fied the drug fentanyl, and says the department had described the individual-oriente and inconsistent use of the gun, as well the substance administered by CODC.
Dogan's allegations claim that 2009 police gravesite package showed water zunning over campus and shows that the dental show pho- tos of supposed victims were reinterpreted.
David Mann, a member of the Police Department, said spoke exclu- sively to News 1 on condition of anonymity because university offi- cials can't review documents immediately, and university officials had to change information that had been a consideration.
didn't change our information until he personally told the drug overdose problem, he said. "He said that drug dealing wasn't re- ally focal point."
He however, in a May 15 statement, also was "proleased" with univer- sity and state officialsin the U.S. Attorney's s Office. "We do not have any way to estimate the crime syndicate, given that the find- ing in the case does not preclude separate review of standard CPD policy directed at any school employee, and whether that employee worked or went out of work, Mann said.
The lawsuit alleges that donations from a full visit with Hogan's workers and thousands of dollars, spent on gift cards, cigars, stock tickets to trips across the country, as well as other financial accounts and income from social members of former employees, were missing from buyers bought. in stock.
Both employees and co-participants remain employed at the univer- sity.
On the Illincis StateUniversity Facebook page< lendofeex |>Imae Steve Healey, the Cook Electronics Co. Ltd., general manager of Aug. B-Feb. 16 at a recent internal session meeting (see
link) held in The Apple Building in London. This July 1939 photo shows Peggy Deaver in 1986. (Photo courtesy the Cook Elec-
39

---

tronics Co. Ltd., displayed here under G. Healey in prison dress. < </endoftext >Steve GIRO's s H-?< |ndoftext
E.6.2 T=10000
< <|ndoffeet]>>> has been presenting a number temporary measures to help help resolve crisis.
Example
like this, but have to resolve it at the spec- ified level; we have to stay low. The people of Greece will be in- terested in the effectiveness of the measures. They will not only manage these measures, but also they will help in order to cope with the problems of the fiscal stability.
The last crisis
However, we do not want to dispose assets for the treasury. This also, will work on developing the national economy, and also paying on the national debt.
This affects the national incomes
So, of 2007-2011, use the government's temporary measures as measure, helping resolve the crisis. In addition, we are able to pay for the borrowing costs. Additionally, pay $440 billion to settle debt issues, which can never be settled by default in country.
These temporary measures will be aimed on several fronts, because the government will have three different partners in the system, in my right.
Firstly, we will be allowed to borrow a lot more upon the addition the emergency measures and these temporary measures will help provide for the repayment of our debts before we are forced into crisis as a result of our borrowing bills. Secondly, in the case of this, we will have resort to temporary measures in the revenue budget for Greece. The budget costs the government another $5.2 billion year.
what you propose if happens again, does this mean that, since 2010, will you resolve the deficits which will occur on the basis what we already have?
The fiscal situation
We would be able to settle our debts by the end of June which is the end. That said, we are taking our part as one of the most important countries in Europe, not only to make a proper transfer of the money but also to rely on it in the economy, However, first of all, we cannot achieve this
day-to-day basis.
is still true that we have decided to be able to deal with the economic situation of the country, but there might be another change the fiscal situation, and therefore, we will try to negotiate on the situation at the end of June and over the summer.
The changes in the fiscal situation would up the parliament of management, bureaucrats, judges and a legitimate parliament of Greece.
Is the government planning to talk about thethe 'temporary measures' of Greece?
Me will continue the process to operate through the temporary mea- sures. This is not temporary measure at this point, because after crisis, not thereyet at crisis level, you can still have enough investment purchases until the end of the month.
40

---

Again the government decided to create temporary measure and now depends upon a particular event, such as that there's s another liquidity crunch. It is better that the government and the authori- ties decided to create a temporary measure effective in June at fair sum monthly bond rates.
The temporary measures will also enhance the government economic status, especially when following the measures at the end of the month.
Tempozary measures is a real tool for growth, not just for the econ- omy.
Knowing that there are several measures place to increase our supply, for example, the level offer profit on public sector en- terprises certain, under all of these temporary measures the in- creases in output, after that, will increase the external demand and the internal demand.
We will be able to create the demand, and also strengthen the govern- ment's s credibility through fiscal organization. What important here, here is that we will apply these measures to our reserves, and at the same time, we apply these measures to the debt level, which will also be the aim of debt-free Greece.
so, first of all, everything is certain ofwhat continues to be col- lected by the government. Given the situation and after the release of the last data on October 16, you also recognize that this will not be any kind of non -on-meennt.
In the case of the payment against the equipment, will be able to manage with the measures.
What does government expect in its plans to create fiscal consoli- dation for the public sector and the new budget.
Regarding this is the temporary measure, we will be able to cope with the troubled finances. However, sure which threatens the fiscal stability of the economy. However, that is not temporary measure, a permanent measure.
not think it is any mea-
On the other hand, there will be our ongoing work on construction in the ministry. If this falls, will continue work job creation, the expansion of the economy.
Also, also the government mentioned the new government reforms, which increased labor hours for the employees, which will further the economic growth, and the second aspect of budget as well and this is government welfare, which will improve the quality of life. We |
Example 2
< /edoffext| him. He said: "What are you doing?"
I hesitated before answering. "Boy, this is exciting. You need better girl. Is she?"
And I said, "You don't have a brain. You have no brain anymore."
After minute, he had walked back and said his own, working through that, he thought he had got himself going in a new direc- tion.
could' ve been a better boy in the first three years.
"You'l 11 only have once before it starts."
MVP
41

---

The story is always, "That's what the other guy has to do." He was the quy who had to do anything. had to reason with school of- ficials. My cousin mentioned to me that some of my friends a almost doubled over at one meeting.
I'd picked up a lot of the money I owed him from high people in me; he liked my grades. Drop-outs didn't consider me high enough let me go hang out. Hе hung when I challenged him after practice to started making, and, quite, never
show new talk.
first saw him C. morning, in the sixth grade class. He wouldn't hang up with him point at team meetings. He started talking about things about me: "I'm an M, I'1l get an A. Tonight." Having had that conversation
lunch, my heart touched mine with pride.
He came, my boy. Now he looks like he's s going back to school. I don't know if he's going to sue. Let's just have a two-bedroom apartment, a $500-do -d00-dollar condo for renting, and pool. And then he was back.
That was a part of my life as I think about it. It was the school year.
never saw guy come up at the locker room and show a new talk. That day one day, I told the high school, "We'll show up one day right here, we can have a little fun," " and after this, remember a small handful of the boys made friends, and they never, ever showed up for a new talk.
I call them "M's kids. I always remember him, I remember his ass up his ass, getting ready for freshman orientation out there. He'll show everything.
He'll 11 show if I'm freshman, I'll act I going to play junior. In few years I'11 try it, then he" 11 make sure he's s going to judge me. He will come over to me day.
Then one day, the senior class was sitting on the bench, pressing his ball on the floor of the locker room, the referee was just stand- ing the knelt it down.
And when he heard about that, another boy, three of his friends, and one of his cousins were on the other side of the room. The boys' class was filling in with his new brother and his new cousin and his new M.M's s player.
The senior class watched me walk me through the chair to the bench. Everyone passed by the boy. Just his toes foot, too.
He [and a girl] passed over his head and, as looked at them, he carried me into the locker room. And the biggest part of the story, was a mistake.
With his elbows out, he pulled me down my shoe, other, sort of a- don't know what they were; palebelly somethings, like bleeding very much, or on little toes. He was up on the stairs and everybody watches, with men and high school kids, who saw him in the locker room. And he caught breath. Then his old man approached me and, disappeared into the middle of the room. He took off his vest, fast enough to herd him into the locker room. I walked into the room and read him little cards with my own eyes to make notes, to pull him under my shoes.
told them: "Listen, because say this today, when you talk to eem today, "Just make sure you talk is more than you'le 11 show. He's
42

---

just listening to me, and he's telling me I' going to be there for him." "
I'm a always the one who wants to do something important about you than show up. walk around and ask, I want message from you, "Keep it going. It<]endoftext
43

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant
information is Not Available.
Please provide a short (1-2 sentence) justificctonn right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No]" it perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). "[No] " or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification elaborate. All supporting evidence can appear either in the main paper the supplemental material, provided appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Claims are addressed
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Our method under- mnder-performs compared to autoregressive models. We also discuss other limitations in the paper.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set assumptions and a complete (and correct) proof?
Answer: [Ye]]
Justification:
They are in the proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
44

---

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: We provide all hyperparameters necesessary to reproduce the experiments and will provide code.
5. Open access data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We will release all code after the paper is accepted. The datasets already public.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
halbbmiisssooooiyyyy...... for more details. Baliii//guides//oeeSuubiii
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ /nips. wttpcc/ppppccppeececeecccc hwwpuuc/guuudeeeeeeeeeeeereeeeee gesdee///ueeee Codeeubmi ssionn ssionnolicy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Ys]]
Justification:
We provide detailed hyperparameters for all experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
45

---

Answer: [Yes]
Justification:
Many of our tabels include error bars and standard deviations
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (c.g. negative етror rates).
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Answer: [Yes].
Justification: We conduct experiments on 3090s, 8xA6000s, 8xA 100ss, or 8xH100s. The largest models on Open Web take 2 weeks to train on 8xA100, 100, the LMIB models only take 2 days to train on the same hardware
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well as estimate the total compute.
The paper should disclose whether the full research project required compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https://neurip:: httppp/bbii/ttiiccuuuieeelllee..
Answer: [Yes]
Justification: follow standard practices
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., if there a special considera- tion due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper diseuss both potential positive societal impacts and negative societal impacts of the work performed?
46

---

Answer: [Yes]
Justification: Our model will allow for more controllable text generation models, and do not increase the capability of current autoregressive models
Guidelines:
The answer NA means that there is socictal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (c.g.. disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: These models are trained on trivial datasets and unlikely to cause any harm compared to state of the art language models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
All assets are publically available and respect the licenses for all the data.
Guidelines:
The answer NA means that the paper does not use existing assets.
47

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service that source should be provided.
If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswi thcode cww.co///aacocoo datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We provide no new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: [NA]
48

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
End of NEURIPS CHECKLIST. Must be end of document after appendix
49


Reviews:
[{"review_id": "WereEXicWc", "reviewer": "Reviewer_p1DW", "summary": "This paper introduces a new approach to masked diffusion language models (MDLMs) that improves performance over previous discrete diffusion methods. The authors present a simplified, Rao-Blackwellized objective for training MDLMs, which is derived from a substitution-based parameterization of the reverse diffusion process. This objective takes the form of a weighted average of masked language modeling losses, establishing a connection between generative diffusion models and encoder-only BERT models. The paper also describes efficient sampling techniques, including a semi-autoregressive decoding method. The authors demonstrate state-of-the-art performance among diffusion models on language modeling benchmarks, approaching the perplexity of autoregressive models. Additionally, they show that their approach can be applied to biological sequence modeling, achieving competitive results on DNA modeling tasks. The paper emphasizes the importance of well-engineered implementation details and provides comprehensive ablation studies to validate their design choices. Overall, this work presents a simple yet effective framework for discrete diffusion models that bridges the gap between diffusion-based and traditional language modeling approaches.", "questions": "- Why didn't the authors include a comparison to Plaid in their perplexity evaluation experiments (Tables 1 & 2)?\n- How do the authors explain their observation that time-step conditioning can be optional in MDLM? Particularly, why is there no conditioning in OWT? Does this suggest that with a larger amount of data tokens, the need for timestep conditioning in MDLM decreases?\n- How many steps are used when evaluating the perplexity of MDLM? How does this compare to other text diffusion models?\n- Can MDLM achieve any efficiency gains in inference and sampling compared to autoregressive (AR) models within the same compute budget? For instance, if we need $K$ steps to generate $B$ tokens at a time in semi-MDLM, how can we make a fair and scientific comparison to AR models generating $L$ tokens in this setting?\n- One disadvantage of BERT-style models is that they are less token-efficient in training compared to decoder-only AR models. Can the authors provide some comparative analysis and plots showing how MDLM compares to AR baselines across a range of training token amount budgets?\n- What are the authors' thoughts on MDLM versus continuous-diffusion models for text, such as Plaid and CDCD, in terms of training, model performance, and inference efficiency? Is MDLM superior to these continuous-space text diffusion models in terms of resulting model quality and reducing the gap to AR models?\n- Is it possible to speed up the generation process of MDLM, similar to the advanced ODE-solvers people are using in continuous-space diffusion models?\n- How does MDLM compare to MaskGiT in terms of model formulations? They share lots of same design space but I didnt see any reference or comparison to this related line of works.", "limitations": "N/A", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a novel, simplified objective for masked diffusion language models, which is a  combination of existing ideas from diffusion models and masked language modeling. The substitution-based (SUBS) parameterization of the reverse diffusion process is an original contribution that enables the derivation of a more effective training objective. The connection established between generative diffusion models and encoder-only BERT models is an interesting perspective in the field.\n- The empirical results are convincing, demonstrating good performance among diffusion models on multiple benchmarks. The extension to biological sequence modeling shows the versatility and robustness of the method.\n- The paper is well-structured and clearly written, with a logical flow from theoretical foundations to practical implementations and results.\n- The work  narrows the performance gap between diffusion-based and autoregressive language models, potentially leading to more adoption of diffusion models in NLP.", "weaknesses": "- The paper focuses heavily on empirical results but lacks a deeper theoretical analysis of why the proposed MDLM approach outperforms previous methods. A more rigorous theoretical foundation could provide insights into the method's success and potential limitations.\n- While the paper compares MDLM to other diffusion-based approaches, it lacks a comprehensive comparison to state-of-the-art non-diffusion language models. This makes it difficult to fully assess the method's competitiveness in the broader context of language modeling.\n- The experiments focus on relatively small models. It's unclear how well the approach scales to larger models that are more common in current NLP research. Addressing potential scalability issues or limitations would be valuable.\n- The paper primarily focuses on perplexity as an evaluation metric. Including human evaluations or other metrics that assess the quality and coherence of generated text would provide a more comprehensive view of the model's capabilities.\n- It's unclear if MDLM can achieve practical speed-up comparing to AR models with same FLOPs budget due to the lack of comparative e valuation on this.\n- Incomplete reference and comparision to previous related works, see Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OoGeLOPLqv", "reviewer": "Reviewer_BSmY", "summary": "While previous works considered diffusion language models less competitive than autoregressive models in text generation tasks, the authors propose a simple framework named masked diffusion language modeling (MDLM), where they claim to have better performance than previous thoughts. The authors derive a simplified Rao-Blackwellized objective and find that the objective equals to a masked language modeling with a varying mask ratio. The proposed methods are evaluated on both classification tasks and generation tasks. The performance excels in classification tasks and keeps consistent with autoregressive methods in generation tasks, which achieves a new state-of-the-art among diffusion models.", "questions": "* I wonder what previous tries have been made in the language diffusion model and what makes them fail to outperform autoregressive model in generation tasks, as diffusion models in the CV field largely outperform autoregressive models.\n* I wonder the computational cost of the method and its comparison with the autoregressive model's since it is known that the diffusion model usually costs more time to converge.", "limitations": "The authors have claimed their limitation in the paper.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The structure of the paper is complete.\n* The connection between diffusion model and masked modeling sounds good.\n* The experiments are comprehensive, covering both classification and generation tasks.", "weaknesses": "The main weakness of this paper is about the presentation. This includes the following two points:\n* In the third section, the authors first derive an objective with a diffusion process. Then, the authors try to claim what difference has been made compared to the previous D3PM. It seems that the different parameterization made in Equation (7) leads to a simpler formulation in Equation (8) compared to Equation (9), if I understand correctly. However, I am quite confused with the illustration in Section 3.2 and Section 3.3. There appears some technical terms without furthur explanations or citations, such as Rao-Blackwellization and graphical modeling. The abbreviation SUBS is also unclear. Some omittions and simplifications are not provided with some explanations to clarify their rationality.\n* There also lacks an overview on the algorithm. Due to the highly complicated derivation process and objective function, I suggest the authors making an algorithm bar to illustrate how the training and the sampling are conducted, like Algorithm 1 and Algorithm 2 in DDPM [1].\n\nThere also lacks some discussions with some previous works, including:\n* There has been a nearly same method in the CV field called MAGE [2]. MAGE also conducts masked modeling on image patch tokens with a varying mask ratio and generates by the ancestral sampling.\n* There lacks a citation for semi-autoregressive modeling [3] since I believe this concept is not first raised in this paper.\n\n[1] Denoising Diffusion Probabilistic Models, https://arxiv.org/pdf/2006.11239\n\n[2] MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis, CVPR 2023\n\n[3] Semi-Autoregressive Neural Machine Translation, EMNLP 2018", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UtTroibYem", "reviewer": "Reviewer_DasC", "summary": "This paper presents a method for language modeling using simple masked discrete diffusion models. The authors show that a simplified objective combined with optimized training achieves performance improvements over previous diffusion language models. The paper reports state-of-the-art results for diffusion models on standard language modeling benchmarks, approaching the performance of autoregressive models.", "questions": "1. Other than the simplified objective and new backbone and sampler, what’s the difference between your method and absorbing state D3MM or more specifically DiffusionBert?\n2. Is MLM pretraining required for your method?", "limitations": "1. Masked language models are special discrete Diffusion models, this is demonstrated in the D3PM paper by rewriting an x_0-parameterized absorbing state model.  This limits the novelty of the paper to the simplified objective only.\n2. The idea in this paper has been hinted at in D3MM, limiting the theoretical contribution of this paper.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The simplified diffusion objective is effective in stabilizing training and is well supported by detailed derivations. \n2. Section 3 provides clear steps to build a connection between MLM and Diffusion LM\n3. The empirical performance is superior to other diffusion language models.", "weaknesses": "1. I believe the paper should be structured more towards introducing the simplified objective rather than framing the architecture as MDLM since the main idea of bridging absorbing state discrete diffusion with MLM has been introduced in DiffusionBert and D3MM.  \n2. The comparative results are not fair comparisons. More specifically the use of a more advanced backbone (DiT) and a low-variance sampler weakens the claim that the performance and stability are sourced from the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CObaZVDFwz", "reviewer": "Reviewer_wXn1", "summary": "The paper introduces a masked diffusion language modeling (MDLM) framework that enhances the performance of diffusion models in generating high-quality text, closing the gap with autoregressive methods. By applying an effective training strategy and a simplified objective function, MDLM achieves state-of-the-art results among diffusion models on language benchmarks. The approach is shown to be simple yet effective, with potential applications beyond language, including biological sequence modeling.", "questions": "As above", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method is well motivated and well designed based on the previous works SEDD\n- The story is well told for better understanding.", "weaknesses": "- In Figure 1, unmasking loss is misleading. What's the difference between the red and yellow colors? Which tokens should we enforce the losses on?\n- In Line 38, \"Simple\" seems overstated. I couldn't quickly understand the method, even though I have some background in diffusion models and sequence modeling.\n- In Line 168, there are too many experimental details about the network structure and tokenizer changes, making it unclear how to prove the method works. This, in turn, suggests that the proposed method is not so simple.\n- The sampling steps for diffusion-based methods should be highlighted in Table 1 and other related tables.\n- In Line 178, the semi-autoregressive part is not clear to me. I lose track of how it works. The need for semi-autoregressive models is not well-motivated. It's a common replacement trick used in diffusion models. The author should explain it.\n- In Table 5, memory should also be included. Secondly, Mamba is more favorable for longer token numbers. The authors should also compare with Mamba under various token numbers: 1k, 5k, 10k, and 20k.\n- The paper needs to provide the mathematical proof for the continuity equation, showing that \\( u_t \\) in the equation generates the probability path \\( p_t \\). It should also highlight the connection to the continuous case.\n- In Line59, several works about using flow-based method is missing,[1][2]\n- In LIne65, citation missing, Chapell's discrete flow matching paper\n- Better discuss with this work: https://arxiv.org/abs/2406.03736\n\n[1]. Language Rectified Flow: Advancing Diffusion Language Generation with Probabilistic Flows\n\n[2]. Flow Matching for Conditional Text Generation in a Few Sampling Steps\n\nOverall, the authors claim the method is simple, but it’s not as simple as they claimed. Additionally, there are many issues with the presentation and experimental parts. Therefore, I am inclined to weakly reject this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response to Reviewers

Dear reviewers, we thank you all for the useful comments and feedback. In addition to the individual responses we provide directly to each of your comments, we wanted to highlight additional results and clarifications that are common to several of your reviews.

### **Improved theoretical understanding:**

The simplified objective we present is a product of a more careful derivation of the ELBO for absorbing state diffusion and of our proposed parameterization, that we call SUBS, which stands for **SUBS**titution. In our revised manuscript, we will include a more detailed appendix that shows how the two aspects of SUBS: (1) Zero Masking Probabilities and (2) Carry-Over Unmasking contribute to the simplified objective.

Below, we briefly recap the derivation of our simplified ELBO and how the SUBS parameterization and continuous-time analysis tighten this lower bound.

Recall the diffusion ELBO:
$\mathcal{L}_{ELBO} = \mathcal{L}\_{recon} + \mathcal{L}\_{diffusion} + \mathcal{L}\_{prior}$

- $\mathcal{L}_{prior}$: We set this prior regularization term analytically to zero by designing the noise schedule $\alpha(t)$ such that $\alpha(t=1) = 0$.
- $\mathcal{L}_{recon}$: We set this reconstruction loss term analytically to zero by taking $T \rightarrow \infty$ and by our “copy over unmasking” parameterization (i.e. we copy over unmasked tokens from $\mathbf{z}_t$ to $\mathbf{z}_s$ with $s < t$.
- $\mathcal{L}_{diffusion}$: This diffusion loss term simplifies to a weighted cross entropy (BERT-style) loss due to the SUBS parameterization and we make the ELBO as tight as possible by taking $T \rightarrow \infty$ (see also VDM [1] for a similar analysis of how the lower-bound becomes tighter as $T \rightarrow \infty$).

### **Algorithms:**

Below we include algorithms for training and inference of MDLM, which we plan to include in our revised manuscript:

**Algorithm: MDLM Training:** Inputs: dataset $\mathcal{D}$, monotonic masking schedule $\alpha_t : [0,1] \to [0,1]$, BERT-like model $x_\theta$, $\mathbf{m}$ denotes the mask vector, $\mathbf{x}^{1:L}$ denotes a sentence with $L$ tokens, $\mathbf{z}^{1:L}_t$ denotes the latent vector with $L$ tokens.  

1. **repeat**
    1. $\mathbf{x}^{1:L} \sim \mathcal{D}$       *// Sample a sequence of length L from dataset (can be a batch)*
    2.  $\mathbf{z}^\ell_t \sim \text{cat}(\mathbf{z}^\ell_t; \alpha_t \mathbf{x}^\ell + (1 - \alpha_t)\mathbf{m})$   $\forall 1 \leq \ell \leq L$ for random $t \sim \mathcal{U}[0, 1]$  *// Mask each token $\mathbf{x}^\ell$ independently with masking rate $\alpha_t$ to obtain the latent $\mathbf{z}^{1:L}_t$.*
    3. Update weights $\theta$ of denoising (BERT) model $\mathbf{x}_\theta$ gradient descent step on 
    
&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$\nabla_\theta \frac{\alpha'_t}{1 - \alpha_t} \sum\_{\ell}\log \langle \mathbf{x}\_\theta^\ell(\mathbf{z}^{1:L}_t),\mathbf{x}^\ell \rangle$ // Note: this is simply a “weighted” BERT-style loss

**until** converged
    

Note that the only differences to BERT are that in standard BERT

1. In step `1.2.`, $\alpha_t$ is a constant (set to 0.85 implying a fixed masking rate of 15%).
2. In step `1.3.` there is no weighting term in front of the cross-entropy loss.

Otherwise the training algorithms are equivalent!

****Algorithm: MDLM Sampling / Inference****

1. $\mathbf{z}^{1:L}_1 =$ [MASK, MASK, …, MASK]    *// Sampling process starts with all MASK tokens.*
2. **for** t = {$1, \frac{T-1}{T}, \dots, \frac{1}{T}$} **do**
    1. $s \xleftarrow{} t - \frac{1}{T}$ 
    2. $\mathbf{z}^\ell_{s} \sim \text{Cat}(\mathbf{z}^\ell_s; \frac{(1 - \alpha_s)\mathbf{m} + (\alpha_s - \alpha_t)\mathbf{x}_\theta(\mathbf{z}_t)}{1 - \alpha_t} )$   $\forall 1 \leq \ell \leq L$.  
    3. $\mathbf{z}_s^\ell = \mathbf{z}_t^\ell$  if $\mathbf{z}_t^\ell \neq \text{[MASK]}$ $\forall 1 \leq \ell \leq L$  *// To prevent unmasked token being remasked in the reverse process; see Eqn(7) in the paper.* 
    4. $\mathbf{z}^{1:L}_s = \mathbf{z}^{1:L}_t$
    
    end **for**
 

### **Additional Experiments:**

**Text8:**

To better compare to other non-diffusion baselines (see reviewer p1DW’s concerns), in addition to the extensive set of baselines that we provide in Tables 1 and 2, we have added a new experiment on the `text8` dataset, which we provide below:

| Method         | BPC($\downarrow$) |
|----------------|-------------------|
| IAF/SCF        | 1.88    |
| AR Argmax Flow | 1.39 |
| BFN            | 1.41 |
| D3PM           | 1.45 |
| SEDD           | 1.39  |
| MDLM           | 1.44* |
| AR             | **1.23** |

*We provide the numbers are for a partially trained MDLM model on 400K steps and hope to report the numbers for a fully trained model during the discussion period. All the baselines models were trained for 1M steps. 
(Details of this `text8` experiment are as follows: following D3PM [4], we train a 77M param model on sequence chunks of 256 characters with batch size 512; baseline values were taken from SEDD [5]).

**Comparing MDLM vs AR sample efficiency**

In addition to the sampling efficiency analyses in Figure 2, we compare MDLM and AR in the batch size 1 setting where MDLM achieves optimal sampling efficiency by caching the outputs of the denoising model (Suppl. D.2). Whereas AR models are limited to a fixed number of generation steps, MDLM may flexibly trade sample efficiency for quality by varying the number of diffusion steps $T$. In Figure 3 (Supplementary material submitted in the general response)  MDLM achieves **up to 1.8x faster** **sampling** and **53% better generative perplexity** compared to AR when generating 256 tokens.

---

**References:**

[1] Kingma et al. "Variational diffusion models.".
[4]  Austin, Jacob, et al. "Structured denoising diffusion models in discrete state-spaces.".
[5] Lou, Aaron, et al. "Discrete diffusion language modeling by estimating the ratios of the data distribution."

Meta-review: The paper proposes a model called Masked Diffusion Language modeling that is a special case of D3M. The authors propose several new contributions that lead of really strong results, bringing diffusion models close to AR models for sequence modeling. Specifically, they propose a specific parameterization of the predictive distribution (which they call SUBS), that allow for better lower bounds in the ELBO objective, because the objective ends up having fewer terms (eqn 8) compared to what a more vanilla formulation from D3M would have had (eon 9). In addition the authors lay out other contributions, related to better implementation, vocabulary choice, ignoring of the time steps in denoising, etc that they claim lead to better results that would have been expected from prior work. 

Results are shown on language modeling and human genome sequence modeling. The reviewers were generally positive -- although there was some push back against whether or not the method was indeed "simple", and also others asked for more detailed comparisons to other methods across different scales of the data, and of the model.  I hope the authors will take these points into consideration when submitting their final version, for maximal impact.

While I was perusing the paper, I also happened to find some typos in equations (unless I misunderstood some of the details of the paper) that I am listing here -- if they are indeed correct, I would hope that the authors will make the corrections for better readability:

1. Line 86 might be incorrect. I think it should be: Cat(z_t; \alpha_{t|s} z_s + (1-\alpha_{t|s}) \pi)
2. Line 98 mixes probabilities, and probabilistic variables, making it inconsistent with the rest of their notations. It should be:  q(z_t|x_0) = Cat(z_t; \alpha_t x_0 + (1-\alpha_t) m);
3. Equation on line 103 uses s, t with latent variables, inside the chain rule. Instead of p_\theta(z_s|z_t) it should be p_\theta(z_{i-1} | z_i). 
4. Line 181 — the range of time might be wrong. I think it should have been \tilde{x}^{L’:L} , not \tilde{x}^{L':L-L'} ? And the same for  lines 184, z_1 and \tilde{x} have errors in the sequence limits: L’:L-L’ should be L’:L again, possibly.
5. Equation 19 -- z_s, z_t should be x_s, x_t, to be consistent with p_\theta(x_s|x_t). The variable names are being switched around ?
Decision: Accept (poster)