Generalized Interpolating Discrete Diffusion
Authors: Dimitri von Rütte
Janis Fluri
Yuhui Ding
Antonio Orvieto
Bernhard Schölkopf
Thomas Hofmann
Venue: ICML 2025 poster (2025)
Keywords: diffusion models
discrete diffusion
language models
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: We propose a new discrete diffusion ELBO that enables combining masking and uniform noise, which unlocks self-correction capabilities without explicit training.

Abstract: While state-of-the-art language models achieve impressive results through next-token prediction, they have inherent limitations such as the inability to revise already generated tokens. This has prompted exploration of alternative approaches such as discrete diffusion. However, masked diffusion, which has emerged as a popular choice due to its simplicity and effectiveness, reintroduces this inability to revise words. To overcome this, we generalize masked diffusion, deriving a new family of general interpolating discrete diffusion (GIDD) which offers greater flexibility in the design of the noising processes. Leveraging a novel diffusion ELBO, we achieve compute-matched state-of-the-art performance in diffusion language modeling. Exploiting GIDD's flexibility, we explore a hybrid approach combining masking and uniform noise, leading to improved sample quality and unlocking the ability for the model to correct its own mistakes, an area where autoregressive models notoriously have struggled.
Code: https://github.com/dvruette/gidd/

Generalized Interpolating Discrete Diffusion
Dimitri von Rütte'
Janis Fluri'
Yuhui Ding'
Antonio Orvieto
Bernhard Schölkopf 123
Thomas Hofmann'
Abstract
mask only
mask uniform naise
While state-of-the-art language models achieve impressive results through next-token prediction, they have inherent limitations such the inabil- ity to revise already generated tokens. This has prompted exploration of alternative approaches such as discrete diffusion. However, masked diffu- sion, which has emerged as a popular choice due to its simplicity and effectiveness, reintroduces this inability revise words. To overcome this, generalize masked diffusion, deriving new family of general interpolating discrete diffusion (GIDD) which offers greater flexibility in the de- sign of the noising processes. Leveraging a novel diffusion ELBO, we achieve compute-matched state-of- oh-the-art performance in diffusion language modeling. Exploiting GIDD's flexibility, ex- plore a hybrid approach combining masking and uniform noise, leading to improved sample quality and unlocking the ability for the model to correct its own mistakes, an area where autoregressive models notoriously have struggled. Code: http://ggithbb..om/dmmmteteeggg
50
Imorovemeen (56
40
20
10
10
20 30 40
50
60
Self-corrected takens (#)
Figure I. Training a diffusion model using GIDD on a combination of masking and uniform noise teaches it to identify and correct its own mistakes. By iteratively replacing bad tokens with better ones determined by the model), sample quality (as per generative PPL. via Gemma 9B) improves by up to 55%.
into generating one word (or token) a time, with each new word serving as additional context for the next word.
While extraordinarily successful on a wide range of data modalities (van den Oord et al., 201 Radford et al., 2018), there are some inherent challenges to this approach. First and most obviously, generating sequence of length N necessarily requires N invocations of the model. This is not problem if N is small but can become expensive N grows to large numbers. Secondly, long-term dependencies and coherence can pose challenge, for example if each step has a non-negligible error rate: If a wrong token is sampled or a previous token becomes incompatible with newly sampled tokens, there no way to correct it. Con- siderable effort has gone into solving this limitation, most recently by post-training with reinforcement learning (RL) to teach sequential reasoning over multiple autoregressive steps (Bengio et al., 2015; Ranzato et al., 2015; Bahdanau et al., 2016; 6; OpenAl et al., 2024; DeepSeek-AN Al et al., 2025).
1. Introduction
For certain data distributions such natural images or natu- ral language, the information content of any given sample can be overwhelming, making the task of generating realistic samples through generative modeling difficult. A common strategy to case the burden on the generative model is to break up the task of generating an entire sample into mul- tiple inference steps, each being simpler in isolation, but recovering the full distribution when recombined. The most prevalent example of this, especially for natural language, is autoregressive modeling (Bengio et al., 2000), where the task of generating a sentence (or sequence) is decomposed
Denoising diffusion models (Sohl-Dickstein et al., 2015) propose different way of decomposing the generative task which can address both limitations. Instead of splitting the sample into elements of a sequence, we gradually decrease the information content of the entire sample by degrading it through the addition of some form of noise until, eventually, the information content reaches zero. The generative task then consists reversing this degradation process, grad- ually adding information back in until the full sample
'Data Analytics Lab, Department of Computer Science, ETH Zurich 'ELLIS Institute Tübingen "Max Planck Institute for Intel- ligent Systems, Tübingen. Correspondence to: Dimitri von Rütte <dvructte@ethz.cch
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Generalized Interpolating Discrete Diffusion
Machine learning are is a field to study of research in artificial intelligence during which and the development [...]
Republic of World of Warcraft has made some significant improvements to game it in their the most recent improvement update, the "Death in the Vengeance" change "End of the World" update.
Mexico City is the largest city in France Mexico. With an estimated population of 22,752,000 22,000,000 [...]
Suppose Alice has 5 apples. If Alice gives 2 all of her the apples Bob, she left with zero apples.
Table 1. Examples of colfecoorention (green replaces red) by our GIDD+ BASE model trained with 20% uniform noise. The model is able to correct grammatical mistakes, improve word choices, and even improve factuality without being explicitly trained to so.
restored. This decouples the number of model invocations from the size of the sample since the number of steps we take to fill in the missing information can be chosen freely. For natural images, an obvious and suitable degradation is the progressive addition of per-pixel Gaussian noise. This choice yields a simple training objective that works well in practice and forms the basis of state-of-the-arr image generation models (Ho et al., 2020; Kingma et al., 2023). The success of image diffusion models has spurred interest in applications to other domains and modalities, including discrete data like text (Austin et al., 2023). Unfortunately, Gaussian diffusion cannot be naively applied to discrete data as there is not necessarily a notion of distance or similarity, at least not one that is straightforward to measure, Instead, discrete diffusion models have converged on degradation process consisting of gradually removing ("masking") to- kens until none are left (Austin et al., 2023; Shi et al., 2024; Sahoo et al., 2024). The task of the model then becomes to reconstruct the original sequence by "filling in the blanks" until all masked tokens have been filled in. This can also be thought of as autoregressive generation in randomized order (Welleck et al., 2019), and indeed reintroduces one of its inherent limitations: Once token is filled in, it can no longer be changed, and any intermediate errors will neces- sarily propagate to the final sample. However, by injecting a small amount of uniform token noise into the diffusion process, we can allow for any token to transition to any other token, therefore resolving this issuc. This elicits the realization that the type of noise is an integral part of dif- fusion model with potentially fundamental implications on its strengths and limitations. Drawing motivation from this, our work aims to illuminate the design space of discrete dif- fusion models by opening up the design space and exploring an alternative diffusion process that combines masking and uniform noise. Our contributions are two-fold:
for this general family, which are needed for sampling and likelihood training respectively. We also show that the de- rived ELBO has global minimum that is reached when the model matches the true distribution.
On the practical side, in Sections 4 and 5 we apply our theory to the special case of masking noise in combination with varying levels of uniform noise. We conduct an ablation study, showing that our mask-only model achieves compute- matched state-of-the-art diffusion language modeling thanks to a reweighted training objective (Sec. 5.2). We also show that the addition of uniform noise leads to improved sample quality and unlocks self-correction abilities (Fig. 1, Tab. 1) that allows the model to iteratively improve samples beyond what is possible by simply traversing the backward diffusion process (Sec. 5.4).
2. Discrete Diffusion Models
As the name suggests, discrete diffusion models act on a dis- crete state space Z. Given some initial state X E Z sampled from the data distribution qo (X). the sample is gradually degraded through a Markov chain Zi...., Zt with Z, E Z, Ze+1 ~ q: (Z(z+1 IZ), and Z1 X until reaching some (easy-to-sample) prior distribution pr(Zr) The denois- ing task then becomes to learn the backward kernel of this Markov chain, such that we can (approximately) reverse the degradation process for any Zr sampled from the prior distribution. Oftentimes, the state space is structured as a sequence (of length L) of tokens from a vocabulary V. i.e. Z, = (= ...... (1)) with 2, E V. In this case, it is common to add noise to each token independently such that suffices to look at the forward and backward noising trajectory of any token 2, in isolation. This is possible if the initial state X is known, which it is during training but not during inference. The model must therefore learn make predictions without this knowledge, inferring as much as possible about X from its noisy version, the sequence Z,
On the theoretical side, in Section 3 we extend the frame- work of masked diffusion to general interpolating discrete difffusion (GIDD) processes. GIDD offers great flexibility in the choice of noising process, encompassing any diffusion process that can be written as a linear combination between the data and some (time-varying) mixing distribution. We derive closed-form solutions for the cumulative state tran- sitions and the diffusion Evidence Lower Bound (ELBO)
2.1. Interpolating Masked Diffusion
Masked diffusion models (MDM) have seen widespread adoption by the community (Ou et al., 2024; Shi et al., 2024; Sahoo et al., 2024; Nie et al., 2024; Hu & Ommer, 2024) due to their simplicity and good performance. The
2

---

Generalized Interpolating Discrete Diffusion
core idea is progressively replace tokens with special [MASK] token until every token has been replaced. As such, the denoising task for the model to learn is to "fill in the blanks" given some context. This noising process results in a Markov chain with marginal transitions that can be written as a linear interpolation between mask and data:
sequence (whether masked not) could potentially be cor- rupted and thus require correction. With the model learning to distinguish between "correct" and "incorrect" tokens, it may also learn to correct its own mistakes, a notion that will be confirmed in Section 5.4.
However, there are some technical challenges to training a diffusion model on some specific, desirable diffusion trajec- tory. The canonical training objective, the diffusion ELBO, cannot be derived without knowledge of the Markovian state transitions, but crafting Markov chain with specific emerg- ing properties (e.g. "halfway in the diffusion process, 40% of tokens should be masked, 40% should be unperturbed, and 20% should be random") is generally a non-trivial in- verse problem. Instead of solving this inverse problem for a specific combination of masking and uniform noise, and to gain the necessary flexibility to design effective model, we aim to generalize interpolating diffusion from mask-only to arbitrary ttime-varying interpolants. Specifically, we introduce the Generalized Interpolating Discrete Diffusion process (GIDD), a family of diffusion models with marginal forward transitions
(1) where S. = - a,, x and m denote the one-hot encoding of the data x and the masking token m respectively.' and S a, I determines the signal-to-noise ratio (SNR) at the current time L. The Evidence Lower Bound (ELBO) of MDM takes the form of simple weighted reconstruction loss of the missing tokens. Specifically, with x, denoting neural network that predicts the distribution of x given a par- tially noised sequence 7,= = (=( , (1)), the negative ELBO is given by
qe (2p (z) = Cat( 0, + 3,m),
2t
log p(x)
0l
< Er, 1 at
Se m log xo (x((2, + C,
(2)
where ( ~~ ((00 1) and 2 ~ (z)) with 5 denoting the Kronecker delta function. Recall that the input to Xę is the entire noisy sequence Ze whereas everything else happens for each token independently.
qe (=(2))))) = Cat(2i; + B,TTe),
where n. can be any probability distribution that changes smoothly over time. Notably, masked diffusion is a special case of GIDD for πe m. We will show the existence of a Markov chain that results in these marginals for any suitable O₁ and He and derive its conditional transitions as well as the associated ELBO necessary for likelihood training.
2.2. L Limitations of Masked Diffusion
Despite their popularity, MDMs have some fundamental limitations. Most obviously: due to the way the underlying Markov chain is defined, a token can never he changed again once it has been filled in, which is analogous to autoregres- sive prediction. This can lead to the accumulation of errors or some tokens becoming incompatible as more tokens are unmasked, and with no way fix them, they inadvertently persist to the final result. Another, less severe limitation is the fact that only masked tokens carry a loss signal, as un- masked tokens are always completely noise-free. Like with BERT, this results in a smaller effective batch size which can lead to slower convergence compared to autoregressive models (Devlin et al., 2019: Clark et al., 2020).
3.1. Forward Process
GIDD is designed to allow maximal flexibility over the type of noise added to the data at any point in time. It consists of a mixing rate 0t, which defines the signal-to-noise ratio over time, and mixing distribution TH, which defines what distribution the data is noised towards at any given time. We refer to the combination of these two functions as the "mixing schedule" of our diffusion process.
Definition 3.1 (Mixing Rate). Let the (cumulative) mixing rate Ck2, 3r with Bg = 1 - a₁ be time-differentiable de- creasing function a₃ [0, 1] + [o, 1] where the initial value Og = means no mixing and the final value o₁ = + com- plete mixing. This determines the SNR with SNR at/3t.
3. Generalized Interpolating Diffusion
To resolve these limitations, we would like to expand our horizon to a more diverse set of diffusion processes. A natural solution drawing inspiration from BERT (Devlin et al., 2019) would be to use a combination of masking and uniform noise. This would address both limitations described above: Not only do we gain the ability to change already- tokens during sampling, but we also ob- tain a more informative training task, as every token in the
Definition 3.2 (Mixing Distribution). Let the mixing dis- tribution n, be a time-dependent probability vector, i.e. a time-differenniabll function πミ: [0, 1] + AIVI- where -IVI-1 denotes the [V -dimensioa simplex. 2 The distribu- tion म. determines the type of noise that is added to the data at any time t. As a consequence, π represents the prior distribution of our diffusion process.
Throughout, we will use hold letters to denote vectors, which, reference to a token, denotes the ooe-hot encoding of the token.
Thh d-dimensional simplex A Ad-1 is defined as the set of all points x E Rd with x; 2 and Σ m4 = 1.
3

---

Generalized Interpolating Discrete Diffusion
Ultimately, we want to find diffusion Markov chain with marginals as postulated in Equation (3), but to arrive this conclusion we will have to work our way up from the un- derlying discrete- discrete-time Markov chain to the continuous- time state transitions, to the closed-form cumulative transitions.
where we use the fact that 1 d/e = and or a + Buje = 1 (as per Lemma H.1, App. H. Having found recursive for- mulas for atjs and Tej we can now apply telescoping to find the desired closed-form solutions (see App. H.2 for details), proving the original claim for any A > 0. In partic- ular, the proof also holds in the limit of A + 0 as long as the limits lim dai and lima-o S i ai exist. Differentia- bility of a and t as required by Definitions 3.1 and 3.2, is sufficient for this.
Proposition 3.3 (GIDD Conditional Transitions). Let out 3. = 1 at denole the mixing rale and let a denote the mixing distribution. Then there exists continuous-time Markov chain with transition probabilities from state 2, to z4 at times s S given by
Corollary 3.4. The cumulative transition probabilities of the Markov chain from Proposition 3.3 are given by
901s(Zal22) = Cat(azi Qt)oZs): Qtis= = =0t|s +t Te)l T
Te (<<((<)) = (zx: Q: = 02, + 3r 3T
(9)
(4)
where atls = a.'' P1l = B(TT - a. B, and 1 de- notes the |V|-lim. vector of all ones.
Proof. The claim follows directly from Proposition 3.3 with Q = Q:jo and using the fact that Org
1, Bo 0.
Proof. Let us discretize time into a A-spaed mesh for some arbitrary A > 0, i.e. assume that we can write / = Ai with i e Z for any t. We then define the instantaneous mixing schedule² oy and fette as
With this, we have successfully constructed Markov chain with the desired marginals outlined in Equation 3. For deriving the ELBO later on, we also need the transition rates of the corresponding Continuous- Time Markov Chain (CTMC), which is defined as follows.
CA(++1) â
(5) Definition 3.5 (CTMC Forward Transition). For some start time s and end time t + A with A -> 0, we have
OAs
BAITAI = BAGHII)
aA(+1)
BATAA
(6)
O&i
9r|s (2 (2,) = 8_..., + Re 21) A + o(A),
(10)
The instantaneous transition probability is now defined as (<(=+a/2)) Cat(z++a: Q,2i),
where R₂ is called the forward transition rate. Little-o nota- tion is used for denoting asymptotically sub-linear terms.
Q = Bett T.
(7)
We now characterize the CTMC forward rate of GIDD.
The instantaneous transitions induce discrete-time Markov chain with the desired mixing properties as defined by our mixing schedule.
Lemma 3.6 (GIDD Forward Rate), The CTMC forward rate matrix R₃ of GIDD is given by
We now turn to our main objective: the cumulative tran- sition matrix Qtl= of this Markov chain, which is de- fined as Q2
2 Sene + 2o (
a'2 (11)
)
Re(Es: z()
04
at
II==s/a QAi- We need to show that Qtj- = T 7 + :. IT. To this end, we are going to inductively unroll a single step to find recursive formulas for Orls and T. First, note that the base case t = s simply Qale I with aais = and TT = as we must remain in the same state. Next, assume that the induction hypothesis holds for Qti.- We then have
where al and n' denote the time-derivative of the respective mixing function.
Proof. By performing first-order Taylor expansion on 9r| (2 (2,) and rearranging the result, we arrive at the de- sired expression. See Appendix H.3 for details.
3.2. Backward Process
Q141+= = Q.Q
(8a)
(8b) We choose the same parameterization of the backward pro- cess as prior work (Sohl-Dickstein et al., 2015; Austin et al., 2023). This canonical form of the model distribu- (8c) tion 22)2 is given by
= = + 1] (
[arp J + 3
= =dtanpt + Belanp 7 I te (1 T)
+ +.r.Bel.TT. qpp1
= =d + Baaan + BBps) ) + Br Te/11
(8d)
po = 2: (zele)) y (x)
(12)
ge(2) Xx)
djanp, 1t (Bitt + âr a 1T,
(8e)
with shorthand notation (((2) (x6) == Cat(ze Qixo t)), where Xo(2,, t) is neural network that predicts the dis- tribution of x given the noised sequence Z,. We refer to Appendix H.4 for details on the CTMC backward rate (2 2o), which is also required for the ELBO derivation.
1044A
'Note that while we use the dot-notation (o) for instantaneous changes, this is to be confused with the time-derivative, which we denote by a prime (a'). )
4

---

Generalized Interpolating Discrete Diffusion
3.3. ELBO
the model and the true distribution, they are both minimal if and only if qe (-(-lar) = ge (xx)))), implying that the ELBO is minimal there." Indeed, it can be shown that the global minimum of the ELBO is reached only at that point.
In order to train a GIDD model, we need a differentiable way to estimate its likelihood. The Evidence Lower Bound (ELBO) serves this purpose: By maximizing lower bound, we implicitly also maximize the (worst-case) likelihood of our model. For the ELBO, we need the forward and backward rate of GIDD, which we have already derived. Then, starting with a slightly modified version of the ELBO from Campbell et al. (2022), we plug in our forward and backward rates R2(2,, zx) and R2 2n) and simplify to obtain Theorem 3.7 (complete proof in App. H.5).
Proposition 3.9. For any mixing schedule or and a the GIDD CT-NELBO has global minimum of zero (up to the ELBO constant C`), which is reached if and only if qe (z(zel)) and qi (2)1x)) are the same everywhere.
Proof. See Appendix H.6.
This is good news, since it tells us that the mixing schedule theoretically does not limit the best cest-possible model.
Theorem 3.7 (GIDD ELBO). Let att, 3r and T be a mix- ing schedule as defined in Definitions 3.1, and 3.2 with marginal forward distribution qe(2a (x) ) as defined in Equa- tion 3. Let further we we(22, x) be a weighting function defined as
In conclusion, the GIDD ELBO is a straight-forwan and flexible training objective that can be applied out-of-the-box to any interpolating diffusion model.
3.4. Sampling
de Tt We(2), qo (2(2)(=) 2 (BOTt,
Given some sampling schedule 0 = to < t1 < tr R 1 and some neural network Xe, we employ ances- tral sampling by discretizing time along the chosen mesh. Specifically, starting with sequence of all mask tokens, i.e. Sty for all Zty, we iteratively sample po 12x,) for i = T...., 1:
(13)
an
Then, the continuous- time negative ELBO (CT-NELBO) of the corresponding diffusion model is given by
- -logp(z) S [u,,, (Do
Fr. 1t, (3tg Rt, .) -h- (21, (xo t:)) Tt. (21, X Zeo f;))
+ DDDs(q (2ls(((((()))))((())))))) C, (14)
(16)
where Dis is the (pointwise) Itakuru- divergence de- fined as Dis(plla) = =p/q- logp/q L, t ~ u(0, 1), and zt qo(-lz)), and with C denoting the ELBO constant
Self- Coll-Correctin Step. In addition. we propose fixed- point iteration to improve generated samples by resampling some tokens according to the model's judgement. More pre- cisely, we give the fully denoised sample Zto to the model and sample the resulting distribution with some temperature т. Then, of all sampled tokens different from Zt we select the one with the highest model likelihood and commit it. This is repeated until convergence (details in App. C).
C C= Ex (ogge///) (xx(a(()))))ll (+1)).
(15)
Since GIDD is strictly more general form of the widely used masked diffusion paradigm (Ou et al., 2024; Shi et al., 2024; Sahoo et al., 2024), we expect the canonical MDM ELBO to emerge from the GIDD ELBO by choosing an appropriate mixing schedule, which is indeed what we find by setting ,, m (proof in App. H.9).
4. Mixing Schedule
While GIDD can be used for masked diffusion, our original motivation for introducing a generalized framework was to explore the combination of masking and uniform noise. To this end, we design mixing schedule that keeps the masked prior distribution but allows for configurable amounts of uni- form noise in between. We use Pa. to denote the amount of uniform noise: For the sake of interpretability, the expected fraction of uniform tokens should reach a maximum of p., at the midpoint between data and noise (1 = 1/2). With these desiderata in mind, we define our mixing rate and mixing distribution (Def. 3.1 and 3.2):
Corollary 3.8 (Equivalence to MDM). If ne m, then, for any valid noise schedule on, the GIDD ELBO reduces to the MDM ELBO (Eq. 2).
Interpretation. Taking a closer look at the GIDD ELBO, we notice that it consists of solving two tasks jointly:
1) The first task is to match the model to the marginal for- ward distribution of some sample x given its noised version 2, by minimizing the KL-divergence between the two distri- butions at the current noise level.
1-t a: = C. BOT = C. -m
Ct Ce
u.
(17)
2) The second task is to minimize the pointwise IS- divergence at between the model and the true marginal distribution at the sampled 24
"It worth noting that both the KL and the IS -S-divergence are Bregman divergences, implying that they can be linearly com- bined into single Bregman divergence Dr with Fp)
Since both tasks consist of minimizing a divergence between
Σ, Ps log P= log Pat-
5

---

Generalized Interpolating Discrete Diffusion
10
1.0
Model (SMALL)
Train. toks.
PPL (1)
4 m
Autoregressive
0.8
GPT2 (Radford al., 2019)
unk. 262B
23.40
Llama 110M (retrain.)
16.11
0.6
10a
Diffusion
0.4
MD4" (Shi et al., 2024)
524B
21.80
10
MDLM (Sahoo et al., 2024)
262B
23.21
MDM (reimpl.)
262B
23.36
10
GIDD+ (ours; Pa = 0.0)
262B
22.29
0.0
10.0
7.5
5.0
2.5
0.0
2.5
5.0
7.5
10.0
Table 2. Our best GIDD model outperforms the compute-matched MDM (reimpl.) baseline, which in turn closely matches results from the MDM literature terms of validation PPL on OWT. "Nmbeers reported by the original paper.
Figure 2. ELBO weights grow exponentially for very low/high noise levels, causing poor optimization if not handled carefully. While masked and uniform token weights almost constant, noise-free token weights vary heavily depending on pu-
Model (SMALL)
PPL (4)
Pu
0.0
Pu 0.1
Pu = 0.2
where u = N-1 (1 m) denotes the uniform probability Vector, Q Bt (1 1) G. + ct N is the vocabu- lary size, and B is constant chosen such that the desired uniform token ratio is reached. The marginal forward distri- bution then becomes
1
MDM (reimpl.)
24.37
GIDD (ours)
24.36
26.88
28.22
+ weight clipping
23.23
25.09
26.40
+ dynamic weights
23.24
23.90
24.64
+ weight decay
23.05
23.67
24.38
(<(2,11))= C.
C. (111))x Im + c.u).
(18)
Table 3. PPL of GIDD (px = 0.0) and MDM match closely, as expected from their thcoretical equivalence. Significant gains come from choosing the right weighting function. especially in the Pa regime. The final setting includes dynamic loss weights dyn and weight decay and is also referred to as GIDD+.
To reach the desired uniform noise level Ps. a 1/2, we need to set B 27 - (proof in App. H.7). The GIDD ELBO weights w. (2,. can also be derived in closed form (see App. H.8 for details). Note that setting Pa = 0.0 again recovers masked diffusion. Finally, we set ? = 1, but there may be other valid choices for this and other variables introduced in this section.
some maximal value Wmax we define
welamp (2t, 2)) min (Wmaxx We We(2t, Wi
(19)
4.1. Training Objective
Through preliminary experiments,
find Wmax 1 to be best, so this is used throughout. Note that clamping mostly affects the weights of mask and uniform tokens. A more principled approach may aim to keep the maximum loss weight constant while preserving the relative weights between masked, uniform, and noise-free tokens. We call this the dynamic weighting function and define it as
Before starting our experiments, we need to solve one last issue, which will yield great performance gains. Taking closer look at the ELBO weights wi (z, we find that their behavior for t + 0 and + is quite extreme. Consider the three possible cases 2, = x, 2, m, and 2, E (œ, m). Plotting the weights wy (zz, .) over time reveals that the weight grows exponentially for very low/high noise levels in all three cases (Figure 2). This can be problematic since these low/high noise samples provide little to no training signal as the model's job of denoising becomes either trivial or impossible, yet can easily drown out all other samples in the batch. To counteract this issue, we propose two weighting schemes that reduce the influence of extreme samples, hence emphasizing intermediate noise levels where the training task is informative.
dyn (zt, x) = W'max (1+ W.
B (
-1)0 (20)
where X log ar (2 // - 3) are determined empirically. Note that re- weighting the ELBO like this is equivalent to sampling from a non-uniform distribution or choosing different noise schedule during training (Kingma & Gao, 2023).
I-a; is the log-SNR. The relative weights
5. Experiments
The simple and obvious solution is to clamp the weights to
5.1. Experimental Setup
1 (t) denotes the inverse sigmoid function and can be thought of the (negative) log-SNR when ignoring the effect C, which tends to be negligible for small Pu.
While discrete diffusion models are a natural for any discrete data, we focus our attention specifically on language
6

---

Generalized Interpolating Discrete Diffusion
5.5
52
3x101
5.0
20x
4.8
4.5
1500009
0.10.11
102
3.3
0,2
Entropy
4.2
0.01
0.05
0.1
0.5 10
57.5 60.0
62.5
65.0
67.5 70.0 72.5
Temperature Orl
Tokena crannged (4)
Saifacouracy (8)
Figure 3. From left to right: (a) Self-correction using GIDD+ (BASE) models resamples up to 10% of tokens independent of the uniform noise level. A temperature of r E [0.1, 0.5] is found to be most effective. (b) For models trained on hybrid noise, sample quality (PPL) improves significantly as more tokens are changed. The mask-only model, though, unable to improve quality despite resampling as many tokens. Sample diversity (entropy) drops noticeably for mask-only models, but only slightly for hybrid models. (c) The correlation between self-accuracy and generative PPI. reveals that hybrid models are significantly better at judging the quality their own samples.
Model
Clarity
Grammaticality
Factuality
Writing style
Creativity
GIDD (pu. 0.0)
2.51
2.96
3.61
2.84
4.48
+ self-corr. (r = 0.1)
1.99 (-20.9%)e
2.39 (-19.3%)e*
3.02 (-16.2%)**
2.24 (-21.15)**
3.60 (-19.5%)**
GIDD (pu = 0.1)
2.51
2.85
3.66
2.78
4.26
+ self-corr. (T = 0.1)
2.69 (+7.2%))*
3.05 (+6.9%)e*
3.88 (+6.0%))*
2.98
(+7.1%)**
4.35 (+2.1%)*
GIDD (pv. = 0.2) + sellf-corr (T = 0.5) 2.90 (+16.5%)e*
2.49
2.82
3.70
2.79
4.25
3.29 (+16.6%))*
4.01 (+8.5%)e*
3.16
(+13.4%)s*
4.48 (+5.5%)ee
Table Self-correction significantly improves various quality aspects as judged by GPT-4o on a scale from 1-10. but only for models trained with hybrid uniform noise. Applying self-correction in the mask-only setting is detrimental across the board. The highest level of uniform noise has the biggest improvements and highest scores across all categories. *> 2a difference, **y 50 difference.
modeling as it is one of the most prevalent tasks in modern machine learning. To this end, we adopt the Open WebText (OWT) dataset (Gokaslan et al., 2019) since there exists a rich literature for both autoregressive and diffusion models trained on this dataset. We follow prior work (Sahoo et al., 2024; Shi et al., 2024) in terms of architecture and training scale and use the DiT architecture (Peebles & Xie, 2023) with the GPT2 tokenizer (Radford et al., 2019) and train SMALL (110M) and BASE (320M) models on 131B or 262B tokens, depending on the experiment (details in App. E).
baseline (Tab. 2) considering the slight differences in hyper- parameters.
However, adding uniform noise to the diffusion process, we find that perplexity degrades slightly, yet benefiting expres- sivity as we will see later (Sec. 5.3 and 5.4). This difference likely stems from an increase in task complexity: The com- bination of masking and uniform noise requires solving multiple tasks jointly, which is strictly more difficult and likely requires more capacity. This is supported by the ob- servation that all noise levels scale consistently with model size, with the highest noise setting even showing some signs of improved scaling behavior (App. A).
5.2. Ablation Study
The goal of our ablation study is to answer three main questions: 1) Does GIDD with our mixing schedule and P. = 0.0 recover MDM as theory predicts? 2) How does the addition of uniform noise affect performance? And 3) what the importance of the weighting function (Sec. 4.1
Our custom weighting schemes bring non-trivial perfor- mance gains to both the mask-only and hybrid noise set- tings, and particularly the dynamic weighting scheme mdyn closes the gap significantly. We hypothesize that the differ- ence between W; clamp and uy is due to the importance of noise-free tokens, which have zero weight if p. = 0.0 but cannot be ignored otherwise. Therefore, keeping the true relative weights between different token types seems be beneficial if p.. O. Finally, moderate amount of weight decay (0.02) improves both training and validation loss, as suggested by D' D'Angelo et al. (2024). The best configuration uses dynamic loss weights mdyn and 0.02 weight decay, which we refer to as GIDD+ from hereon out.
To this end, we train SMALL GIDD models on OWT with varying levels of uniform noise Pu. E {0.0, 0.1.0.2.. We also train our reimplementation of MDM on the same setup. The final validation perplexity (PPL.) of these runs is re- ported in Table We find that the training trajectories as well as the final performance of MDM and GIDD (p.. = 0.0) match almost perfectly with respective validation PPL of 24.37 and 24.36. Our MDM reimplementation also closely matches the conpute-matched MDLM (Sahoo et al., 2024)
7

---

Generalized Interpolating Discrete Diffusion
Size
Model
Train. toks.
ARC-e
ARC-c
BoolQ
Hellaswag
PIQA
OBQA
WinoG.
Avg.
SMALL
GPT2
unk.
43.81
19.03
48.72
28.92
62.89
16.40
51.62
38.77
Llama (retrain.)
262B
40.53
25.51
46.21
33.14
62.73
28.40
50.75
41.04
MDM (reimpl.)
262B
30.98
23.63
50.52
31.11
54.13
28.00
49.41
38.25
GIDD+ (Pu. 0.0)
262B
30,98
23.55
50.43
31.87
56,42
26.60
51.70
38.79
GIDD+ (Pu. = 0.0)
131B
31.57
24.57
50.92
31.36
56.31
27.80
52.57
39.30
BASE
GIDD+ (Pu- = 0.0)
131B
32.58
24.40
50.86
36.62
58.05
29.2
51.54
40.46
Table 5. Our best GIDD+ model in terms of zero-shot henchmark accuracy outperforms MDM (reimpl.) and even surpasses GPT2-smalI, although it still lags behind our Llama-based autoregressive baseline. Best scores among SMALL models and diffusion models are bolded and underlined respectively.
5.3. Unconditional Generation
5.4. Ceff-Correction
While models trained with uniform noise consistently hibit a higher loss, we have yet to test the main motivation for its addition: By teaching the model to distinguish be- tween "correct" and "incorrect" tokens, we hope to unlock the ability for the model to correct its own mistakes at gen- eration time, stabilizing the denoising process and yielding improved sample quality, In order quantify sample qual- ity, we use "generative perplexity". metric that computes the likelihood of generated samples under a more capable model (in our case Gemma 2 9B, Gemma Team (2024)), where a high likelihood under the reference model is con- sidered to be a sign of high quality. While this metric has its flaws (see App. G), it is common in the literature (Lou et al., 2024; Sahoo et al., 2024). We find that, though abso- lute numbers are difficult interpret in isolation, is still useful for comparing models in relative terms, especially if controlling for diversity. To that end, we also consider the unigram entropy of generated samples as diversity signal, which should stay close to the entropy of the data (4.98).
In order to directly evaluate the model's self-correction abil- ities, we iteratively apply the self-correction step from Sec- tion 3.4 to unconditionally generated samples. If the model indeed has learned to identify and correct mistakes, includ- ing its own, we expect that this repeated invocation can iteratively improve samples until a stable point is reached where the model is either happy with the sample or sees no way to improve it. To measure the degree of convergence, or how "happy" the model is with sample, we use its self- accuracy on the given sample, i.e. the percentage of tokens that have maximal likelihood under the model.
Focusing BASE models. we find that both generative PPI. and self-accuracy improve consistently in the number of replaced tokens (Fig. 3), with a gen. PPL of 93.3 and self-acc. of 73.5% for the Pu. = 0.2 model (up from 214 and 62.0% respectively). Qualitative evaluation also con- firms this (see examples in Tab. and 7). For the mask-only model, while the self-correction step still resamples the same number of tokens, this does not translate to improved gen. PPL or self-accuracy, showing that the ability to self- correct is only acquired if some amount of uniform noise is present during training. Despite this, the mask-only model does appear to improve slightly. which is likely due to nu- merical limitations: For numerical stability, we actually set Pu to a very small value instead of exactly zero, empirically resulting in 10 (out of 262'144) 144) random tokens per batch. Indeed, the MDM (reimpl.) baseline does not exhibit any self-correction abilities at all and in fact makes samples worse during the self correctrettonn step (App. C).
Notably, the generative PPL of models trained on uniform noise is significantly better than that of mask-only models, with entropy hovering around 5. 1,15 for all models and set- tings (App. D). We observe especially big improvements over mask-only models for low inference-compute settings, with a generative PPL of 387 for GIDD+ (SMALL: Pa. = 0.1) at 32 denoising steps compared to 904 for Pa. = 0.0 and 1302 for MDM (App. D). Training on uniform noise there- fore seems to stabilize the generation process when the model gets its own outputs as subsequent inputs, resulting in better sample quality despite having slightly worse validation PPI.. This suggests that some amount of self- correction may already be happening during the denoising process. However, while more denoising steps monoton- ically improve sample quality, this plateaus at a PPL of around 200 for BASE models (App. D). Next, we show that it is possible to decrease generative PPL well below this plateau by further exploting the models' capabilities.
To bolster the point of qualitative improvement, we do LLM- based grading of samples before and after self-correction in terms of clarity, grammaticality, factuality, writing style, and creativity. Significant improvements are observed after self-correction for Pa > O models in all categories, with the mask-only model showing significant deterioration (Tab. 4), Clarity and grammaticality experience particularly large boosts, which is not surprising given the size and training scale of the model. See Appendix F. 1.1 for prompt and setup details.
8

---

Generalized Interpolating Discrete Diffusion
5.5. Benchmark Performance
ing to avoid intermediate mistakes, but stick to mask-only diffusion. Concurrent work has also looked into adaptive de- noising orders (Kim et al., 2025) and adaptive loss weights (Ye et al., 2025) as ways to combat the limitations of mask- only diffusion models.
Finally, we evaluate our models' language understanding capabilities on range of benchmarks. Based on the in- creased difficulty of the hybrid noise setting, we do not expect pa. > 0 models to outperform the mask-only case, which is indeed what we find (App. B). Instead, we fo- cus on comparing the best SMALL GIDD+ model to MDM and autoregressive baselines, namely GPT2 (Radford et al., 2019) and retrained Llama (Touvron et al., 2023a). Our benchmark suite consists of ARC-e and ARC-c (Clark et al., 2018), BoolQ (Clark et al., 2019), Hellaswag (Zellers et al., 2019), PIQA (Bisk et al., 2019), OpenBookQA (Mihaylov et al., 20, and WinoGrande (Sakaguchi et al., 2019). We find that average accuracy correlates well with validation PPL (Tab. 5). Among diffusion models, the best perform- ing model is GIDD+ (Pu. = 0.0) trained for only 131B tokens, surpassing models trained for twice as long. While the best diffusion model, GIDD+ (Pa- = 0.0), outperforms the autoregressive GPT2, the best autoregressive model, Llama (retrain.), still performs best overall. GIDD+ mod- els trained with uniform noise improve with scale but lag behind their mask-only counterparts, which is consistent with their respective valiation PPLs (App. B). This high- lights an important difference between likelihood estimation (i.e. recognizing realistic samples) and sample generation (i.e. creating realistic samples), which do not always corre- late perfectly in practice: Despite mask-only models outper- forming in likelihood estimation, the picture is flipped when considering their sample quality, indicating that likelihood- based multiple-choiee benchmarks may not be enough to holistically evaluate diffusion language models.
Continuous diffusion has also been adapted to discrete data by doing Gaussian diffusion in an embedding space (Li et al., 2022; Gulrajani & Hashimoto, 2023). Diffusion-like approaches have also been extended to discrete data, with discrete flow matching (Gat et al., 2024) adapting the flow- matching paradigm (Liu et al., 2022; Lipman et al., 2022) and Bayesian flow networks (Graves et al., 2024) adopting the perspective of denoising directly in probability space rather than collapsing the distribution after each step.
Finally, the idea of denoising a combination of masking and uniform noise popularized by BERT (Devlin et al., 2019), where it was proposed in the context of representa- tion learning.
7. Conclusion
We have introduced a new family of generalized interpo- lating diffusion processes (dubbed GIDD) and successfully applied it in practice. While the extreme scale required to train overall state-of-the-ar art language models is out of scope for this work, we see great potential in the methods and results described here, but also in diffusion language models more broadly: Self-correction is an area where next-token prediction notoriously has struggled, but as we discovered, this capability comes naturally to diffusion models given the right type of noise. Our work also presents step towards closing the gap in pure language modeling performance between diffusion and autoregressive models, achieving state-of ootte-of-thearr perplexity for compute-matched diffusion models thanks to a re-weighted version of our newly pro- posed GIDD ELBO. Beyond our work, discrete diffusion models respond well to scaling training-time compute like their next-token prediction counterpart, but also provide a natural way to scale test-time compute. By choosing the number of denoising steps, and now also the number of self- correction iterations, one can trade off speed and accuracy depending on the setting, All in all, and given that GIDD opens a design space yet to be explored fully, this may ren- der diffusion language models a promising competitor to autoregressive models in the future.
6. Related Work
Our work builds on a line of discrete diffusion research, with Austin et al. (2023) first introducing the diffusion ELBO to discrete Markov chains, Campbell et al. (2022) extending it to continous time, Lou et al. (2024) deriving an alternative ELBO based on concrete score matching, and concurrent work by Shi et al. (2024), Sahoo et al. (2024), and Ou et al. (2024) proposing a simplified objective for mask-only diffu- sion. With the exception of Austin et al. (2023) (App. A.2.6), the combination of masking and uniform noise is left un- explored by this line of work. Gu et al. (2022) use this hybrid noise for vector-quantized image generation, but con- duct no investigation on the benefits of combining the two noise types. He et al. (2022) propose a noise schedule that degrades different tokens at different rates, depending on their "difficulty" as estimated using BERT, therefore try-
Impact Statement
This paper presents work whose goal is to advance the tech- nical state-of-the-art in an area of Machine Learning. It shares potential societal consequences with much of the work in the general area of language modeling and founda- tion models.
'While the difference is rather small and can be explained by run-to-run variance. it is possible that the model is overfitting to shorter sequences due to the way we handle long sequences. We use random cropping instead of chunking (App. E), which may over-emphasize short documents in the training corpus.
9

---

Generalized Interpolating Discrete Diffusion
Acknowledgment
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? Try ARC, the A12 reasoning challenge, 2018. URL https: httpttt//aaxxi....o..mm /ttp/aax..vv.o../... /bbs/ 1 .0457.
Thank you to Bobby He, Gregor Bachmann, and Tiago Pimentel for their helpful feedback on the writing. Antonio Orvieto and Bernhard Schölkopf acknowledge the financial support of the Hector Foundation.
D' Angelo, F., Andriushchenko, M., Varre, A., and Flammar- ion, N. Why do we need weight decay in modern deep learning?, 2024. URL
References
/bbs/
Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and van den
2310.0441
Berg, R. Structured denoising diffusion models in dis-
crete state-spaces, 2023,
DeepSeck-Al et al. DeepSeek-V33 technical report, 2024. URL
abs/ 221./210.0.00
DeepSeek-AN et al. BeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning, 2025. URL https: /ttcpp://arivv. arxiv. http:///aaaa....//a///////////// wttp/aa///aa..a.aa// 12948.
Bahdanau, D., Brakel, P., Xu, K., Goyal, A., Lowe, R.,
Pineau, J., Courville, A., and Bengio, Y. An actor-critic algorithm for sequence prediction, 2016. https: //arxiv. warxiv..rgg/bba. wttp///a/g//....///////////////e 07086.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for lan- guage understanding, 2019. URL attpss//arxxs
Bengio, S., Vinyals, O., Jaitly, N., and Shazeer, N. Sched- uled sampling for sequence prediction with recurrent neural networks, 2015. URL.
org 1810. 04805.
abs/1506. 03099.
Gao, L., Tow, Abbasi, B., Biderman, S., Black, S., DiPofi,
A., Foster, C., Golding, L.., Hsu, J., Le Noac'h, A., Li,
Bengio, Y., Ducharme, R., and Vincent, P. A neural proba- bilistic language model. Advances in neural information processing systems, 13, 2000.
H., McDonell, K., Muennighoff, N., Ociepa, C., Phang,
J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika,
L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, 07 2024. URL https: /zeodoo orgorec 1 1608800..
Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. PIQA: Reasoning about physical commonsense in natural language, 2019. URL https: wtw.aaiivvvo.o. org. abs/ 1911.11641
Gat, I., Remez, T., Shaul, N., Kreuk, F., Chen, R. T. Q., Synnaeve, G., Adi, Y., and Lipman, Y. Discrete flow matching, 2024. URL httpp:://aaxii..rrg//////// 15595.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. Dhariwal, P., Neclakantan, A., Shyam, P., Sastry, G.,
Askell, A., Agarwal, S., Herbert- Veurbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu,
Gemma Team. Gemma. 2024. doi: 10.34740/KAGGLFMM 3301. URL https:/ wwgg.e.ww..........o.wo..o.. com/ //001.
J., Winter, C., Hesse, C., Chen, M. Sigler, E.. Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners, 2020. URL https:// arxiv. org/ abs 2000/2205.110 14165.
Gokaslan, A., Cohen, V., Pavlick, E.,, and Tellex. S. Open- WebText corpus.
thth..
ib/openteebTextooopuss/s 2019.
Grattafiori, A. et al. The Llama 3 herd of models, 2024. URL https::: ptattps://arxiv.orrgg http://aaaba2477277...
Campbell, A., Benton, J., Bortoli, V. D., Rainforth, T., Deligiannidis, G., and Doucet, A. A continuous time framework for discrete denoising models, 2022. URL https: / /ttp/aaaiii./..///////.../...ava org/ wwtg/abs///e aac.//bs/22255 14987.
Graves, A., Srivastava, R. K., Atkinson, T., and Gomez, F. Bayesian flow networks, 2024. URL Uttps:ts:///xxt
org/ abs 220. 07037.
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski T., Collins, M., and Toutanova, K. BoolQ: Exploring the surprising difficulty of natural yes/no questions, 2019. htthttps: //arxiv. org/ abs / 905 10044.
Gu, S.. Chen, D.. Bao, J., Wen, F. Zhang, B., Chen, D., Yuan, L., and Guo, B. Vector quantized diffu- sion model for text-to-image synthesis, 2022. URL https://arxiv.orrt
14822.
Clark, K., Luong, M.-T., Le, Q. V., and Manning, C. D. ELECTRA: Pre-training text encoders as discriminators rather than generators, 2020. URL https: /arxiv. org/abs/ /2033. 10555.
Gulrajani, and Hashimoto, T. B. Likelihood based diffu- sion language models, 2023. URL https: attps://arxiv. //arxiv. org/ hegg/bs/2305. 22005 18619.
10

---

Generalized Interpolating Discrete Diffusion
Gumbel, J. Statistical theory of extreme values and some practical applications: series of lectures, volume 33. US Government Printing Office, 1954.
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can suit of armor conduct electricity? A new dataset for open book question answering, 2018. URL https:/ arxiv. wwx.v.org///oooooooo.oooooooo
org/ org/abs/////oooooo...o........br absaaass//// 1101..80...0 02789.
He, Z., Sun, T., Wang. K., Huang, X., and Qiu, X. Dif- fusionBERT: Improving generative masked language models with diffusion models, 2022. URL https: //arxiv. org/ abs /2//2221111 15029.
Nguyen, M., Baker, A., Neo, C., Roush, A., Kirsch, A., and Shwartz-Ziv, Turning up the heat: Min-p sampling for creative and coherent LLM outputs, 2024. URL https:
//arxiv. //aaxivv.rgg/aaeeeeeeeeea wwwiii.orggg/... /207 01082.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion prob- abilistic models, 2020. URL httttt://///xii... abs/2006. 11239.
Nie, S., Zhu, F., Du, C., Pang, T., Liu, Q., Zeng, G., Lin, M. and Li, C. Scaling up masked diffusion models on text, 2024. URL https: /ttp://aaavv..aaaa org/ abs/ /29gg/bbs/240..... 2410. 18514.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, Simonyan, K., Elsen, E.,, Rae, J. W., Vinyals, O., and Sifre, Training compute-optimal large language models, 2022. URL https::////// http//axvvvv.////// org/ abs/2203. 11.00.155550 15556.
OpenAl et al. OpenAl ol system card, 2024. URL https: / //arxiv. httg/abb/22111
Ou, J., Nie, S., Xue, K., Zhu, F., Sun, J., Li, Z., and Li, C. Your absorbing discrete diffusion secretly models the conditional distributions of clean data, 2024. URL https: / / httpaaiiivvvoooo wwt.w//.orr.///oooeee//o//. 2406 03736.
Hu, T. and Ommer, B. [MASK] is all you need, 2024. URL https:ttp//.oc....t. /ttp/vaivvvv.////e//eee/e/aaellt org/ abs/2412. 06787.
Peebles, W. and Xic, S. Scalable diffussion models with trams- /bs//
formers, 2023. URL
Kim, J., Shah, K., Kontonis, V., Kakade, S., and Chen, S. Train for the worst, plan for the best: Understanding token ordering in masked diffusions, 2025. URL https: //arxiv. org/ abs / 2502. 06768.
2212.09744
Radford, A., Narasimhan, K., Salimans, T. and Sutskever, I. Improving language understanding by generative pre-training. 2018. URL https: hetp.h.pppeaa..cc/reerrr.t..
Kingma, D. P. and Ba, J. Adam: A method for stochastic op- timization, 2017. URL atttps://arxiia
abs/
languagg-unssperrisss////ggg understanding_papeer pdf.
Kingma, D. and Gao, Understanding diffusion objec- tives the elbo with simple data augmentation, 2023.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, Language models are unsupervised multitask learners. 2019.
URL www.tttpp//.m /ttp///axvvvv///////e/eee/eaaeva org. abs /2303. 00848.
Kingma, D. P., Salimans, T., Poole, B., and Ho, J. Varia- tional diffusion models, 2023. URL https://aaaxi org/ /bs//ab//200 2107. 00630.
Ranzato, M., Chopra, S., Auli, M., and Zaremba, W. Se- quence level training with recurrent neural networks, 2015. URL https: eetps:://ariiv org/ / abs/15l1 06732.
Li, X. L., Thickstun, J., Guirajani, I., Liang P., and Hashimoto, T. B. Diffusion-LM improves controllable text generation, 2022. URL https:: /ttp:///xxvvvvrov abs/2205, 14217.
Sahoo, S. S., Arriola, M., Schiff, Y., Gokaslan, A., Marro- quin, E., Chiu, J. T., Rush, A., and Kuleshov, V. Simple and effective masked diffusion language models, 2024. URL wtt.httts////. arxiv. www.vvooo.. oow/orrgabs......... / 2406. 07524.
Lipman, Y., Chen, R. T. Q., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling, 2022.
Sakaguchi, K., Bras, L.,, Bhagavatula, C., and Choi, WinoGrande; An adversarial Winograd schema challenge at scale, 2019. URL https: /ar//aavvivv///ee//eeae///eeavva .ovivioooo org/ abs/ 1907. 10641.
URL https::/aaxxt
org/ 2210. 02747.
Liu, X., Gong, C., and Liu, Q. Flow straight and fast: Learn- ing generate and transfer data with rectified flow, 2022.
URL https: //tp///arivvvv org/ abs/ 2209 03003.
Shi. J., Han, K., Wang. Z., Doucet, A., and Titsias, M. K. Simplified and generalized masked diffusion for discrete data, 2024. httss:::: /arxivv. org/ org/abs// abs abs/2408 2406. 04329.
Lou, A., Meng, C., and Ermon, S. Discrete diffusion model- ing by estimating the ratios the data distribution, 2024.
URL https:://///..... //a //axxivaivvvvvv org/ abs 2310. 0. 6834.

---

Generalized Interpolating Discrete Diffusion
Sohl-Dickstein, J., Weiss, E. A.. Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequilibrium thermodynamics, 2015. URL https:
//arxiv.. org/ hts5/bb//1000 03585.
Tange, O. Gnu Parallel 20241222 ("Bashar'), Decem- ber 2024. URL https:/ /tt://doi..o http://o...orr//////// org/00rgg.... 5281/ zenodo... 14550073. GNU Parallel is a general par- allelizer to run multiple serial command line programs in parallel without changing them.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lam- plc, G. LLaMA: Open and efficient foundation language models, 2023a. URL httpss//arxiv, org/ abs/ 2302. 13971.
Touvron, H., Martin, L., Stone, K., et al. Llama Open foundation and fine-tuned chat models, 2023b, URL sittps://azee
org/ abs 2307. 09288.
van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K. WaveNet: A generative model for raw audio, 2016a. URL JtLhttps: http:///axvvvvvviv /bs/ 1 1609.60 03499.
van den Oord, A., Kalchbrenner, N., Vinyals, O., Espeholt, L., Graves, A., and Kavukeuoglu, K. Conditional im- age generation with PixelCNN decoders, 2016b. URL https: /tte:://ariiie aawii.iiiivv.... org/ abs 1606.6660 05328.
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W. Emergent abili- ties of large language models, 2022. URL https:
//arxiv. org/ abs abs5 07682.
Welleck, S. Brantley, K., III, H. D., and Cho, K. Non- monotonic sequential text generation, 2019. URL
https://arxivw oww/abgg//..... abss/1922 1902. 02192.
Ye, J., Gao, J., Gong, S., Zheng. I.., Jiang, X., Z., and Kong, L. Beyond autoregression: Discrete diffusion for complex reasoning and planning, 2025. URL https:
//arxiv. www.oorg/a......oooo... httg//bb/// abw///10..0... 2410. 14157.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and
Choi, Y. HellaSwag: Can machine really finish your sentence?, 2019. wtt.httpss//arxiv.ooo abs/ 1905.07830.
Zheng, K., Chen, Y., Mao, H., Liu, M.-Y., Zhu, J., and Zhang, Q. Masked diffusion models are secretly time- agnostic masked models and exploit inaccurate categor- ical sampling, 2025. URL https: www.iiivoog...... abs abbs/409. 02908.
12

---

Generalized Interpolating Discrete Diffusion
Contents
Uniform Noise and Model Capacity
14
GIDD Downstream Performance
14
Self-Correction Step
14
Number of Denoising Steps
15
Training Details
15
Evaluation Details
16
F.1 LLM-based Evaluation
18
G Evaluating Generative Perplexity of Diffusion Models
18
H
Proofs
20
H.1
Conditional Mixing Schedule
20
H.2
GIDD Conditional Transitions
21
H.3
GIDD Forward Rate
21
H.4
GIDD Backward Rate
22
H.5
GIDD ELBO
23
H.6
GIDD ELBO Global Minimum
26
H.7
Uniform Noise Ratio
27
H.8
GIDD ELBO Weights
27
H.9
MDM Special Case of GIDD
28
Unconditional Generation Samples
29
LI
GIDD+ BASE, P. 0.0
29
1.1.1
Example 1
29
I.1.2
Example 2
30
1.2
GIDD+ BASE, Ps. 0.2
1.2.1
Example 3
31
I.2.2
Example 4
32
13

---

Generalized Interpolating Discrete Diffusion
Bry
46 60
11 62
smal
5.0
hate
5C -A00A
15
10
BOFLOPS
(a) Pr. 0.0
(b) pa. = 0.1
(c) Ps. = 0.2
Figure 4. Plotting the compute-efficien frontier reveals different scaling behaviors for different uniform noise levels. revealing that training with uniform noise henefits slightly more from scaling compute compared to the mask-only setting.
A. Uniform Noise and Model Capacity
In Section 5, we have observed that the addition of uniform noise can pose a challenge, and even with improvements to the weighting function, the likelihood of trained models decreases as the proportion of uniform noise increases. Intuitively speaking, this is not entirely surprising since the addition of uniform noise makes the training task strictly more difficult: No longer can the model take for granted that every unmasked token is correct. Instead, it has to consider every token in the context and, if necessary, replace it with the correct one. This intuitive explanation suggests that the reason for the observed discrepancy in performance may be a lack of model capacity, in which case we would expect larger models to be less affected by the addition of uniform noise.
To test this hypothesis, we scale the number of parameters while keeping the training horizon constant and train models of sizes TINY, SMALL, and BASE on different uniform noise levels Pa. E {0.0, 0.1, 0.2). We then plot the compute-cfficien frontier as exponential fit to the pareto-optimal validation ELBO (Figure 4). For computing IsoFLOPs of our models, we follow the method from Hoffmann et al. (2022), Appendix F. Due to resource constraints, our setup is somewhat limited: The sample size is limited to three different compute budgets for each noise level, the largest of which is still comparatively small at 3.3 x 1020 FLOPs. As a point of reference, many signature capabilities of modern LLMs only start emerging around 20 FLOPs (Wei et al., 2022), which is still orders of magnitude higher than our largest compute budget. With this being said, we do indeed observe a consistent albeit small trend of higher levels of uniform noise scaling better with more compute. While the mask-only setting (Px. = 0.0) has a scaling exponent of -0.0586, adding uniform noise increases the scaling exponent -.0558 and -0.0621 for p.. 0.1 and pa. 0.2 respectively. Extrapolating this trend predicts that the Pa. 0.2 setting will overtake Pu. 0.0 around 10 FLOPs, a compute budget that is routinely reached by mid- to large-scale training runs (Brown et al., 2020; Touvron et al., 2023b; Grattafiori et al., 2024; DeepSeek-Al et al., 2024). However, it has to be stressed that the limitations of our setup make such a prediction highly unreliable. For example, the optimal amount of uniform noise may change with model size and/or compute budget, or certain hyperparameters like the learning rate may have different optimal values depending on Pa. Nevertheless, the observed scaling behavior is promising and warrants further investigation.
B. GIDD Downstream Performance
Benchmark accuracies for GIDD+ models of all three sizes (TINY, SMALL, BASE) and all uniform noise levels (pu E (0.0, 0.2.)) are given in Table 6. We find that performance improves consistently with model size, regardless of uniform noise level. However, the models trained with uniform noise slightly but consistently lag behind the mask-only model.
C. Self-Correction Step
Algorithm. Our self-correction algorithm is fixed-point iteration that can be applied any generated sample that is fully (or partially) denoised. The high-level idea is to query the model to identify tokens that it thinks are wrong and should be replaced, and to then iteratively replace a single token at a time so as to avoid reintroducing conflicting tokens. A pseudocode implementation is given in Algorithm 1. In practice, we find that convergence often comes in the form of oscillation between two or more equally-good states (in terms of self-accuracy). so we additionally implement early-stopping based on aeff-accuracy. An early-stopping patience of is found work well.
14

---

Generalized Interpolating Discrete Diffusion
Size
Model
Train. toks.
ARC-e
ARC-c
BoolQ
Hellaswag
PIQA
OBQA
WinoG.
Avg.
TINY
GIDD+ (px- 0.0)
131B
28.28
24.49
49.97
27.78
54.62
26.20
51.30
37.52
GIDD+ (pr. 0.1)
131B
27.69
23.21
50.89
26.75
55.28
24.60
52.25
37.24
GIDD+ (px, = 0.2)
131B
26.73
23.12
50.18
25.61
51.52
27.40
49.33
36.27
SMALL
GIDD+ (Px = 0.0)
131B
31.57
24.57
50.92
31.36
56.31
27.80
52.57
39.30
GIDD+ (Pt. = 0.1)
131B
28.45
21.93
50.73
28.37
55.82
29.20
52.17
38.10
GIDD+ (Px, 0.2)
131B
27.99
22.87
50.46
26.92
52.94
26.40
50.04
36,80
BASE
GIDD+ (Pμ = 0.0)
131B
32.58
24.40
50.86
36.62
58.05
29.2
51.54
40.46
GIDD+ (Pu = 0.1)
131B
30.13
23.04
51.10
31.91
56.15
27.6
52.33
38.89
GIDD+ (Pu = 0.2)
131B
28.75
24.15
50.95
29.82
53.81
26.8
49.25
37.65
Table Downstream performance of GIDD increases consistently with model size, but hybrid noise models lag behind their mask-only counterparts across scales.
Algorithm 1 Self-Coorection Step
Let Zu (z(", ..... =( (T)) be (partially) denoised sequence up to noise level
Let fo(Ze, t) denote a (trained) discrete denoising neural network.
while not converged do
(1:2) softmax( (Z,, t)/T)
for i (1,.... [} do
2(1) ~ Cat(xx))
end for
S {ill E fl..... L} and 2/(0) + ) (1) (2)
f arg maxis Xg
24 - x((0)
(1)
end while
Additional Results. addition to the results for our BASE models given in the main text, we also report improvements for SMALL models, which are very comparable. A notable difference is that for SMALL models, p.. 0.1 has a consistently lower generative PPL, suggesting that Pa 0.2 is too much uniform noise for this size. We also include our MDM baseline, as the comparison between same-sized models is fair. Despite our GIDD+ (P.. = 0.0) exhibiting weak but present ability to self-correct, MDM has no such ability and applying the self-correction step only makes the samples worse (Figure 5). The most likely cause for this difference are implementation details, where, for numerical stability, Pr. is not actually set to zero, but to a very small value, which still results in approx. 10 (out of 262' 144) random tokens per batch due to limited numerical precision. Alternative explanations may look at differences in hyperparameters, or the numerically non-zero weights on unmasked tokens in the GIDD setup. More examples from the self-correction experiment are given in Table 7.
D. Number of Denoising Steps
Comparing sample quality for different numbers of denoising steps, we find that, as one would expect, sample quality in terms of generative PPL improves consistently with more denoising steps, up to around 128-256 steps when plateau is reached (Figure 6). Notably, the sample quality of models trained with uniform noise is significantly better compared to those trained without. This trend especially strong for small numbers of denoising steps, suggesting that self-correction may help in those scenarios.
E. Training Details
All our models are based on the DiT architecture (Peebles & Xie, 2023) and use the GPT2 tokenizer (Radford et al., 2019).
We train models of three different sizes: TINY (L = 6, H = 8, = 512; 28. 48.4M non-emb. params.), SMALL (L = 12,
H- 12,
768; 92. 1M non-emb, params.), and BASE (T. 24, 16, d 1024; 321.2M non-emb. params.), where
15

---

Generalized Interpolating Discrete Diffusion
6x10
90
50
5.1
40
Toleno changed
10t
4.9
30
200
2100
Pum 0.0
4.4
10
PPL
4.6
Pum
MDM
10'
0.01
0.05 0.1
0.5 1.0
0
10
20
30
40
50
60
56
58
60
62
641
65
70
Tempersture
Tilons changed 1991
Saftoccuracy (96)
Figure 5. Self-correction results for our SMALL models. While the overall trend is the same for BASE models. the best-performing model uses Pv. = 0.1 instead of P., = 0.2, suggesting that the ideal uniform noise ratio depends on model size. The MDM baseline noticeably worse than the mask-only GIDD implementation, with self-correction yielding negative improvements, which is likely due to numerical limitations in the GIDD implementation.
[small)
10
Ru 0.1 [small]
10
(small)
MDM (small)
A. (base)
167
Entrayy
1''
0.1 (bass)
5.0
P. [base]
4 10a
102
Entrapy
3
4.8
x 10
x
256
64
128
2566
Denoising steps
Denoising
Figure 6. Generative PPL (via Gemma 9B) decreases monotonically with increasing numbers denoising steps. Interestingly, the presence of uniform noise during training seems to benefit sample quality overall, but especially so for the low-step regime.
denotes the number of layers, H the number of attention heads, and d the dimensionality of hidden states. All models are trained with context size of 512 tokens and batch size of 512 for 500k steps (resulting in a total of 131B training tokens) GPUs in bfloat16 6 precision using Pytorch's mixed precision training
on single node of NVIDIA A100/H100-800 (torch. cuda. aut ocast). For the sake of comparison with the literature, some models are trained for twice as long, resulting in 262B training tokens.
For optimization, we use the Adam optimizer (Kingma & Ba, 2017) with 3 = (0.9, 0.99), = 10-9, and a learning rate 5 1.-4. The learning rate is warmed up linearly for the first 10k steps and then decayed using a cosine schedule to 10% the initial learning rate. We use weight decay 0.0 for our ablations (unless stated otherwise) and 0.02 for the final configuration, also referred as GIDD+. We also use gradient clipping to norm of 1.0.
For our noise schedule, sample t U(c,1111 with 10-4 using low-discrepancy sampling (Kingma et al., 2023). By default, all loss weights (including the unclamped ELBO weighting function) are clipped to 10a to prevent training instability. For sequences longer than 512 tokens we select a random window of 512 tokens, while short sequences are padded to a length of 512. Padding tokens are included in the loss calculation but are ignored in the ELBO.
F. Evaluation Details
For computing validation metrics, we reserve the last 100k samples (=11.5%% of the training set (OpenWeb Validation samples that are longer than the context length are cropped random window for consistency with training.
For downstream performance evaluation, we use the Im-eval- -aallsaarrsssss (Gao et al., 2024) with a custom model that multiple-choice tasks where the
uses the ELBO to estimate per-token log-likelihoods. We only consider likelihood- per-token likelihood is computed over both the context and the completion (but not the padding), as preliminary experiments have found that this produces slightly better results. We use = 128 evenly spaced samples (in [c, 1 - <) for to estimate the ELBO. Samples longer than the context size of our model (only applies to BoolQ) are truncated by taking the final N
16

---

Generalized Interpolating Discrete Diffusion
Example
Republic of Deltaos have made some significant improve-
Republic of Deltaos have made some significant changes to the patch in game v2: The "Death in the Vengeance" patch. Notable changes are included:
ments to the patch in game "Death in the Vengeance" patch. Notable changes also included:
Proflah Ring can be reached with the ring head up.
Profession Ring can be reached with the ring head up.
You can select characters from their application in choice.
You can select characters from their class of choice. Growth returns to your output level when floating between
Growth returns below your output level when floating be-
tween the default dragon and the highest active rev tier.
the default level and the highest level revamp,
Borg followed by Radiant World to earn and the coveted tutorial is now also available for Edition 12.
Borg followed by Radiant World to earn and the coveted
tutorial is now also available level 12.
Example 2
new industrial renaissance movement which uses the win- ner's swould of GE technologies
new industrial manufacturing platform which uses the lion's 's share of GE technologies
strong link between both US at manufacturing and integral US manufacturing production platform in America
strong link between the US industrial manufacturing and the US industrial manufacturing platform in Europe
Example 3
short of the feeds public front music ming unlimited free music streaming and high-quality content, available whenever you Webs for your subscription.
instead of the free music fronting service unlimited free music streaming and high-quality content, available when you pay for subscription.
Example 4
Journal publishing has opened the world to these kinds scientists and scientists deserve an encouraging place to look.
Journal publishing has opened the world to these kinds, and scientists have an encouraging way to look.
Some researchers can discuss several papers, others are putting many many specific types of material.
Some researchers openly discuss their papers, others are putting many many specific types of papers.
Globality postulates the circumstances researchers learn from the very reputation of other study team.
Globality postulates the circumstances researchers learn from the good work of other research team.
Table Examples from self-correction experiments reveal a noticeable qualitative improvement: The model is able to correct grammatical mistakes (Ex. 2, 3), improve coherence (Ex. 3), and improve the choice of words given the context (Ex. 1, 4). The examples from GIDD+ BASE (p. 0.2) with self-correction temperature 0.1.
17

---

Generalized Interpolating Discrete Diffusion
1. Clarity and coherence: Keeping in mind that the text may be cut off in the beginning and at the end due to it being excerpt, how clear and understandable is the text? Grammaticality:
there any grammatical errors in the text?
3. Factuality: If applicable, is the factually verifiable information stated the text
(e.g. facts about geography, history, etc.) accurate and reliable?
4. Writing style: How well is the text written in terms of style and fluency? Do the sentences flow well, is the vocabulary appropriate?
Creativity:
original and creative is the text?
for each category, give short justification before providing the final score. Your answer should be following the JSON format, with one top-level key for each aspect (* clarity', "gramattccalty', 'factuality', 'style', and 'creativity'). Each aspect, in turn, should be a JSON object consisting of a 'reasoning' and 'score' key in that order. The 'reasoning' key should contain a short justification for the score, and the 'score' key should contain the score itsolf.
Please keep the following in mind:
first before deciding on a final score. Only output the JSON containing the justification and scores and nothing else. Keep in mind that the presented paragraph may be an excerpt from a longer document, so it may not be fully self-contained. Do not deduct points for issues arising from this.
Give your
The text to be graded is as follows:
(text) ".
Figure 7. Prompt used for the LLM-based evaluation of sample quality.
tokens, similar to context scrolling for autoregressive models.
Our generative perplexity is based on the google geem//emmma---aeee 9b model (Gemma Team, 2024) as it provides good tradeoff' between language modeling accuracy and efficiency. Prior work often relies on the GPT2-large model for generative PPL computation, but we believe that in order to draw meaningful conclusions, it is crucial to use grading model that sufficiently more capable than the graded model in order reasonably provide a proxy of the "ground truth" distribution of natural language.
Unigram entropy is computed following Zheng et al. (2025) (App. H.I) by computing the entropy of the token-level frequency distribution over unique tokens in the sequence. This means that, for our maximum sequence length of 512, the upper bound is (512) z 6.24 in case all tokens are unique.
We use the GNU parallel software (Tange, 2024) for streamlining the execution of our evaluation scripts.
F.I Lased Evaluation
We qualitatively evaluate unconditionally generated samples before and after self-correction using the GPT-4o API by instructing it to grade the samples in terms of clarity, grammaticality, factuality, writ- ing style, and creativity on a scale from 1-10. The model is provided sample text and instructed to first give a justification and then a grade for each category, and to return the result JSON string for ease of parsing. See Figure 7 for the exact prompt used.
G. Evaluating Generative Perplexity of Diffusion Models
Generative perplexity an evaluation metric intended to measure the quality of generated samples with a grading model that is used as a proxy of the ground truth data distribution. Under this assumption, we deem samples with a high likelihood under the grading model to of higher quality or least to be more likely under the data distribution. While there are many potential issues with this approach, results can be particularly misleading if the grading model is bad proxy for the
18

---

Generalized Interpolating Discrete Diffusion
Min-p
Sample
Model: GIDD+ (SMALL, Pv. = 0.0)
10-
The second time in a month Henrik Zqvist's wrist is nothing concerning for Perproductu, the American. An'n"It's going to nake a break, Zetterberg said, "And hope it is comfortable enaugh for the next three games. got practice and we had warm runs this week. "AnynCavoring? for a handful of games. "An'nP1acing $6.9 million for the 2016 2013 first- round pick, the Americans are happy to have his availability now and offered him some flexibility, An'nThe Wings' expectations are varied in both player nature and rotation. \n'n"It's different situation we know I'm going to be arcund little bit more (t my right wrist), [...]
p p= 10-5
The majority of high-risk projects appear to be delayed indefinitely because another is sought to replace them. \n/nIt is understood that the council has set up a cemmission to decide the tireframe of what are expected undertaken a thorough and rather robust assessment activity in the council, so it is considered that the period to justify a revicw is further too leng," reads submission to councillors, An review is being carried cut, 10 months after the Conservative administration lodged an election proposal in April.
by Christmas, starting on 15 October. \n'n"We have the ongoing
=
n2 [chpghahhh2211111m00233333131111
Model: MDM (SMALL, reimpl))
p 10-7
Incredible. David Segel making his ankle...... (Photo by photo) (quotes from Evan Jones) ... AnYaThe Nigerian was wearing tight pants since being anemic' An'n"I basically wanted French au shorts in one piece to wear with everything that was en Internet in 2010," he said, wearing Section 50 Lo Lim Channt out. "The things w that wear things that have to work, "AnAnte is using a werds, which means "Planned for the swollen collarbone, " he also tean by fall good treatment. Ponder Mikko the kn/ sw writer that during one World Leberbi has since made his lap!!....
p 10-5
Crowwick said the staff has "been growing" around the Acadeny of Newcastle Cityln'nThe club's Academy which works with youth players is being used in the first time as the first look teams on their new training starting ground. An\nThe Newcastle National School will grow players, develop coaches and implement the system. for the rest of the game. AnknCoach Steve Curtis said the club has been keen to show players that the youngsters ne hoping those who come that will learn his way there and [...]
develop the new system international level.
p= 10-3
A vehicle is the side of vehicle of, located on the side of the car by Facebook, AnAnA car is located on the rear of the car of, in side of area AnAnThe driver of the vehicle is in the area of the rear of, located the side of the vehicle from Facebook. com. \n\nA driver has parked the area of the vehiele on the side, side of the Road of, \nAnThe owner of the vehicle has been located on the side of the rear of, located on the side of a road on the road cf[?] of, the Newark, N.J. AnAnThe owner is attempting to locate passenger in the area of vehicle and parked [...]
Table Samples generated with different min-p cutoff values. Despite generative PPL consistently decreasing for larger cutoffs, the sample quality starts to deteriorate drastically for values larger than p 2 -
19

---

Generalized Interpolating Discrete Diffusion
Eastray
(reimpl.)
10
CIDD
PPL
Entropy
Min.
Figure 8. Generative perplexity as measured by GPT2-large decreases consistently as the min-p cutoff is increased. Unfortunately, this trend does not correlate with subjective quality, showing major limitation of generative PPI. with GPT2-large.
true likelihood of samples. While prior work often relies on GPTe-large grading model, we find that this model suffers from failure modes that are typical for small (by today's 's standards) models.
Before going into detail about the exact failure modes we observe, we need to discuss another peculiarity that arises when sampling from discrete diffusion models. common approach efficiently sample categorical distribution the Gumbel-max trick (Gumbel, 1954), which is used by Shi al. (2024) (via JAX categorical sampling) and Sahoo al. (2024). As noted by Shi et al. (2024) in App. G, this approach is somewhat numerically unstable and leads to an implicit regularization of the sampling distribution, effectively masking out very small probabilities (the authors report smaller than 10-*), This is effectively min-p sampling adapter, which has been found to improve sample quality of autoregressive language models as well (Nguyen et al., 2024). To study the effective of this min-p regularization, we explicitly implement with more numerically stable sampling algorithm based on binary search. Measuring the generative PPL as per GPT2-large for different values p. we find consistent decrease with the observed gen. PPL. of ~90 around 10-7 being consistent with what is reported for SMALL models in the literature (Fig. 8).
However, the cutoff probability increases, generative PPL drops to suspiciously low values. Indeed, manual inspection of the samples reveals that the sample quality deteriorates drastically for larger cutoffs (examples in Tab. 8). These samples exhibit low diversity and a lot of repetition, failure modes that are typical for small (autoregressive) language models. Despite this, these samples have very high likelihood under GPTe-large and hence are considered "high quality" under the generative PPL. metric. This shows the importance of using more capable grading models that are able to pick up on these failure modes and the effect that sampling parameters can have on the outcome of these experiments. It also highlights the limitations of comparing absolute generative PPL numbers, they be misleading in isolation. This is why unigram entropy help spot and quantify any catastrophic collapse in diversity, though it also struggles to quantify more subtle losses of diversity in the min-p < 10-6 regime (Fig. 8). While Gemma 2 9B also suffers from the failure mode described here to some extent, it should provide a much better proxy of the "true distribution" of natural language compared to the much weaker G-Tg-large.
H. Proofs
H.1. Conditional Mixing Schedule
Lemma H.1. If e and TTjs are defined in Proposition 3.3, and Bole a prob. vector, i.e. Tts > 0 and 1 Tz/= = 1.
3z
-t a,
Br then Onj.e + Bejo 1 and tt
Proof. We have
O4 0s Os.
C4
04 (1 BB) + 3r
at
+ Bols =
+ 3
as
+ 3r = ar
+ Br
(21)
as
02s
and, using the fact that TT TH = 1, Vt by Definition 3.2,
T Tt/ = T
311 πτ
Ott B. Tx
ax Bg
= 1,
(22)
Bejs
Bejo
as
Belo
a2
20

---

Generalized Interpolating Discrete Diffusion
thus proving the claim.
H.2. GIDD Conditional Transitions
Proof of Telescoping in Prop. 3.3. Recall the recursive formulas for Ozzs and Beja sss::
Ol+ dx.00t|=
(23)
TTs = B + Sep Tz/=
(24)
By unrolling the recursion and plugging in the definition of de then have
f/A-1
r/A-1
a/js - - as Clale
II
II
(A(++))
(A(TAA)
Qt
(25)
OAi
(A(+AA)
02
s/A
sm/A A
Analogously, for Tes we get
Te/s
f/A-1
3 B-A + ai d1--88--2tt--- + +... ài-s ... ....a. + + åg Ts
1/A-1
ar
(BAHH)TAAH))
aA(++1)
BARTAi
OAi
f/A-1
at
at BAITAI
BAG+I)TACHD
aA(i+1)
a&i
ims/A
BeTT
2T,
(26)
a4
a2
yielding the desired expressions for arls and Tejs and concluding the proof.
H.3. GIDD Forward Rate
Proof of Lemma 3.6. We need show that the CTMC forward rate matrix R. of GIDD is given by
Re(za, 2t)
04
8 8 + 2 (BeTt Oi
af n
(27)
Or
where af and Tí denote the dime-derivative of the respective mixing function.
The proof follows the idea of Proof 2 in Campbell al. (2022), App. B.2 to perform a first-order Taylor expansion on (z)())) To this end, let be given and let s + A for some positive A 0. Then, by Proposition 3.3 we have
(=+2)=(2|(2a) = 2 Q.+ = 2, (cr+t + BT
(28)
We now linearize d and B. around 8, resulting in
astA + + o(A) 1 0 A + o(A), a,
astA/s as
(29)
a2
as+A
B.tAl To = B.AATT4A
BATT
(B,TT + (B,TT.) +o(A)) (1 [+ d' a + o(2)) BaTT.
(30)
a
0' -((BAT.))
B A + o(A).
02
21

---

Generalized Interpolating Discrete Diffusion
The inner term of Eq. 30 can be simplified as follows using the product rule:
(3.T&)) a' 3, = (1 a.) + -B,
dg (1 aa,
(31a)
as
(1
1 1-0g as
=B,T'S 023
1
(31b)
a2
B TT
π
(31c)
ags
Plugging this into ( yields
(0+1)o(20l26) = 2, ((1+ a'e A + o(4)) 2, +
d' s
(BoT's
A (1)
ax
as
dga 8
d' n
1) o(A), (32)
=e
+ z,
BOT
ag
as
which presents the rate matrix as claimed, concluding the proof.
H.4. GIDD Backward Rate
Definition H.2 (CTMC Backward Transition). For any L with L- A and A -> have Pajo(=u|zz) = 82,,2, + Re(zi, 2.) + o(A),
(33)
where RG called the backward transition rate.
Lemma H.3 (GIDD Backward Rate). The CTMC backward rate matrix Re of pa(zalze) defined in Equation 12 is given by
(x(x Re(2. 21) (c(2x( qe (2 (x0)
RC (zt, 22) -8aaze Σ Re(e' 21)
(34)
(2())x)))
Proof. For the reverse rate matrix iif (ze, zo), start with our choice for the model backward transition (Eq. 12):
4s (2a Xo)
po(zalz))= = (1)a(2e)2))
(35)
ge (2g Xo)
By now setting t A with A + O. we get
90- ae-a(ze) (2 gr ( (x)
po(zalz))= = q/tt - (Zel2z))
(36a)
= (S + (Z0) 2 - o(1)) -0- 90-2 (x0) o6 a (2e X ) + o(A)
(36b)
11 (2 X0)
A-0
82
qr( Xo)
82,121
-4((2(100) + Re (Zo, 2x)
9:( Zo (x)
A o
(36c)
4t (x))
Le (e(22 (x)
Xx)
=1
((((())))))
In order to get rid of the time-derivative of the forward process we use Kolmogorov's forward equation to rewrite the time-derivative of the forward process of (=g|xo) as d/(za| = E.' 9e(=" xp) (2', zz), resulting in
(xx) R2 2z) + Re 2x)
qe (x,
Re (Zz, 2a) -2,
(37)
(=)=)()))
ge (=(2,/x0) xx)
Finally, rename 2g 2, in the first term, since they equal if 8z and the entire term is 0 otherwise, resulting in the desired equality:
RS (2 2x) = -52... Σ Relz,, 21) (=(((x)) (x)
Re(2. 21) (c(=x|(0)
(38)
qe (x(
22

---

Generalized Interpolating Discrete Diffusion
H.S. GIDD ELBO
Our starting point is slightly modified version of the continuous-time ELBO (CT-ELBO) from Campbell et al. (2022) which explicitly keeps
constant terms. Keeping these terms is useful for canceling other terms in subsequent derivations. The proof of Proposition H.4 is largely analogous to Campbell et al. (2022).
Proposition H.4. For any CTMC diffusion process with marginals qe (z(2 forward rate Re(Zer 21), and backward rate Re(2,; (1), the CT-ELBO is given by
(<(='lx) Ra x (=)(())))) (=(2)a))) Ri(=', (.)a(( + +C,
logp(z)= 2 (xsl))) Re(2s, 2)) Re(2s (x), +
Σ
Rele' ze)
log
(39)
2/
wheret t ~ 1)(,,) and =
- (i (=(e)((())])
Proof. The key quantity in the diffusion ELBO is the KL-divergence between the true and the model backward transition, which is given by the following. For the second equality, we use Baye's rule and the fact that qrja (t)=(Ze//za) 2el ar) 2z)a (2.) following the Markovian property of the forward process:
DKL x)) = Σ (elo(Ealzz, log
(2)e(2a(zz)
(40)
Σ 20)a(zz2 (z((a((r))
4u (1) 2u 2.).
log
Le (x(2))z))
In order to derive the contimuous-time ELBO, we first analyze the behaviour of this term as - with = -- A. By the definition of CTMC, we then have
log (z, = (0.,6...2. + ARg + + o(1))
S2 ARg(za 26) + (1 82,,2.) log( (Rs 21) + +(()) + o(A), (41)
where we use the fact that log(1 + + = - 2 + x ) for the 2g = 24 case. By also using the fact that
A log( Ar + o(A)) = A Alog A A Alog(æ + o(1) A log A + Alg((1 + o(1)) + A log A= A Alog2,
(42)
we then get that
q (Z)(z)) log ( = 18 + AZolzo + o(A]]
18_,.. (2, 2)) + (1 82,22) log( R, (2s, 21) + o())) + o())]
- AR ARo 21) + S2n )z +(1 82, (2s 22) A log( AR, ( 24) o(D)) +o(A)
= log R. (2.1222))
62,,2, AR AR, 21) + (1 82, A RR, 2 log (2, 21) o(A). (43)
By analogous reasoning, we also get that
Gel=(2ll2, log (z)r(z)lze) = S0,,, AR + (1 (R. log R. (2), + o(A).
(44)
Finally, we also use the fact that
(0(20(2)) (1 5e.2)) + ARg(2e, + o(A)) log (0(22 x))
Ze (z)l2) log Ge (2)())
ge (2(2222)
=0 if
(1 82,,2, ) AR (2. 24) log (s(2g/z))
o(A). (45)
g((2)2))
23

---

Generalized Interpolating Discrete Diffusion
Now plug Equations 43, 44, and 45 into Equation 40, which yields
DKL (xL1999((a = Σ (zel=)) go (2 x gr (=(2((z))
log 4tts (ee Pajo(Za ze (r)
(1)=(Z)|za)) log (2)o()) - (1))(Z) log + 2 (Zelzs) (o(2al3r) (x(2al()) qe (z)
2: (= (x)
By.
Eq.43
Eq.
R2 21 21). (2a (x)
( 21)
(2 20) + Σ (2s 2og
+ o(A)
R. (2,, 2 2z)g) (2)(2))
i1
Ro(zs, 24 (2)l()) Ro 2o (2((2()))
A
Ro(22 21) Ro 21) + E Re((s: 21) log
+(A). (46)
We now substitute this result into the discrete-time diffusion ELBO, which is given by
logp(x)= 2 Σ (xxi(s(x) (Dr.lass--naaaccca../lllllaaalaa +C
(DKI(a-11(4 a =))pp(=2|(()] + +C,, (47)
-
where C' is the standard diffusion ELBO constant with C' = E (x)ll) (x)))](((1)) Substituting and taking A = + + 0 results in the final CT-ELBO:
Dr
T
logp(x) 2
Bma(e())) ( + +C
(1/A 2)
+
1/A A +
A++0
A
Re(z)) Re(2erz))+ + Σ
Re(zsi 24) log
Re 21 )z) (zglx)
+ +
+0
(2 24 ) (F)la)
Ro #
Re(2', 21) log (z,, 2' (z)())))
o(A)
R((2,, - 22(2),2()+
Σ 2'$00
+C, (48)
0
which is the desired expression, concluding the proof.
Starting at this general form, we now plug in the GIDD forward and backward rates Re(ss, 2i) and (2x, za) into
Proposition H.4 and simplify the resulting expression to derive the ELBO for GIDD.
Proof of Theorem 3.7 (GIDD ELBO). We need to show that
log p(x) 1) < Br,,, [we x) (Dxz + Drs(a( C,
(49)
with (pll)) = -og-log10 p/g - 1. 21 (0, 1), 2 ~ q (-12),
(e)lll and
the weighting function
(, x)
1
zj_ (BiTT
a π3: ate
(50)
(x)
We begin by noting that it follows from Lemma H.3 that
(.(. Re(2s, 21) (e (e((2,1x)) Xxo
if 2. # 2t
Re. (Zt, za)
(51)
Re(2e, 2x) - Ex (2, 1 ze) (c(=))
if 2, = 21-
24

---

Generalized Interpolating Discrete Diffusion
Plugging this into Proposition H.4 results in
logp(a) 2 Bt, - Σ (2(2 26) (n(=11x0) ge (2t (x)
2e(='lx)) (c(2)
Re(=', 24) log (x(=1xx)) (o(2)1x)))(<<(())) (=)lar) qe(2) Xx (x)(a=)))
C.
(52)
We now simplify the two sums inside the expectation. First, note that Re(,,, - + wo(2 ())((2)2)) based on how we defined we (2,, (e( For clarity, recall that af refers to a time-derivative whereas 2' refers to a running variable. For the first sum we then get
Re 2)) 2e(2"1 x0) @r (2t x5)
Σ
ah at
Se we(2,, x)q(((())))
ge(2' [x]
(53a)
(z xe)
Σ wee (re x: 2)) (z (x/)x)) (<(2)(z)
ag qe ( (x0) O4 qe (x)
(53b)
wy .) (
+ ai ai
(53c)
qe (z)(0)))
For the second sum, note that (+)+ we(2e, x)) if z' + ž, and that the inner term if z' = 2t since log 2<(2'l-) We can rewrite it accordingly as
0.
a: (3,l-)
(<(211z) 2 (<<(/z)
Re(z', 21) log g0(2' (2)1((2(11)
xe(2t,x)9((2" log 2e(2' (xx))))(2e(()) (((21(x9)4x<<e11>)
Σ
(54a)
q x )9 (2'[z)
Σ 2e (2' (1x) log (r(=1(xo)
go (x)
=w((
Σ ye (2 (z'(xr) log
ge (=(22(x))
(e xo)
(54b)
we(2)),)) (log (<(2(1)) (2()x)))
DK (xl(a((a))la()))))
(54c)
Plugging both results into Eq. 52 yields
(lx) S -Be, [- wr (er((s)))) x) (2(222))
+ o1 (0(2((x))
+ we( (1os ((2zlar) ge(2x (x0)
DKala(((allla((()))))) C
axe
(55a)
Be wx(2e)z)) (DKr (Drr
ge (0(2((z))
di a+
log qe (2 (xo)
+
(55b)
qe(2)
By applying Lemma H.5 (see below), can pull of/on inside the weighted term and apply the definition of Drs get
q (2(2(())) 9e(2 (xo) ) we (2(2 20)
9e (<(=)l2))
- -lx)p(x)) S Be,z,
wi (2) xr)
(xl(((()=)l(((((x))) +
log
+C
(56a)
( + (=(2(22)) log (x(=)(z)) (x(2) -1) +
Be,
we (2)(2))))) 2:)
(56b)
(=(2)0)))
Br, (wr((2, ar (Dxx D=s(((())/))ll((()))))) +C,
(56c)
which is the desired expression and concludes the proof.
remains prove Lemma H.5.
Lemma H.5. Let Oit, Bs qe(ee ()), and we (ze, x.) be defined as in Theorem 3.7. Then, we have
of
E., [we( x]]]
(57)
25

---

Generalized Interpolating Discrete Diffusion
Proof. The proof consists of simply rewriting of Ee, (x)( We begin as follows:
Egg [ur[ x)] =
Σ (((2(lz)ux(((x2)
(58a)
at ax
Σ
22
T
BITT
(58b)
04 I
1T
B
(58c)
at
d (1 TT)) at
(1 T()²
(58d)
where we can pull the multiplication with 1sT inside the dine-derivative since it is a constant. Using the fact that π,, is
probability vector and hence 1 T n 1, this further simplifies to
a (1 at
Ez [ure(r; x)1 = 31((1 T m.)²
1 T.)
(59a)
=1( d
(59b)
at
a/ ot
(59c)
thus concluding the proof.
Remark H.6. By switching from the pointwise IS-divergence to the full IS-divergence defined as Dıs(plla)
Σ, Pi q
log gi - 1) and assuming that wz 2 is non-negative, we get another (less tight) ELBO:
qu
- llg & Bt,,, [w,
+
+ C.
(60)
This follows from the fact that the full IS-divergence is the sum of pointwise IS-divergences, each of which is non-negative. Therefore, the sum can never be smaller than any one of its components. This version of the GIDD ELBO may have practical benefits such as being easier to implement and/or having lower variance, although we did not test it.
H.6. GIDD ELBO Global Minimum
Proof of Proposition 3.9. We need to show that the global minimum of the GIDD ELBO reached if and only (e(2(r)) and ge (x) are the same for all x, I, and z. We prove the statement by treating each direction individually.
(-) ) Assume that qe (2(z) and gr (2(xg) are the same for all x, t, and z. Then, since DKL and Dis are divergence measures, they zero everywhere and the ELBO reduces to C, concluding this direction.
(=) Assume that the BLOO is zero (up to C). This implies that for any L E [0, E sup)()o(z)), and = supp(gr(-Iz)), we have either DE + DIs = 0 or wz(z, x) = 0. Let us assume that we chop up the interval [0, 1] into slices To with non-zero mass for which we either have DKL + DDs = 0 or wx(2, 2) = 0 for all E Ta (for arbitrary but fixed x, 2). It now suffices to show that for any Tú, we have (<(2(xr) ) = 4e (x)) for all x E supp(go(z)), 2 E supp(ge (-1x), and é E Tr. Let in the following Tr, x, be arbitrary but fixed. Since DæL + Drs = implies that (e(-1x)) and (1x))))) are the same, is sufficient to consider the other case where DKL + DIS 0 and xz, = O. Suppose, for the sake of contradiction, that we have DKL + Dis 0 for This implies that qu (2|2z # (1(2(x0)) which in turn due to the conservation of probability mass, necessitates that qe (='(ar) A (='(xx)) and hence that DEL. + DIS > 0 for at least one other 2' E supp(qe (- (z)) with z' # 2. Note that this z' must also have wx(2',z) z) O, since it already has DEL + D1s > 0. Any being in the support of qe (-(())) = a,x + 3 further implies that we must either have x = 2 or (T,)² > 0. Since we have at least two unique tokens to choose from, at least one of them must be different from x and therefore have (TT.); > 0. W.L.o.g. we proceed by assuming that (TT)= = 8 > 0, recalling that wy = 0 for all E T by assumption. Since 2
This is possible since we (2) is continuous and de differentiable in t. We can therefore exclude degenerate cases where the ELBO jumps between DEL. + Dis = 0 and (x, = 0 in a discontinuous manner. 10 Note that DKL + Dis is always non-negative.
26

---

Generalized Interpolating Discrete Diffusion
in the support of q=(1(z), we must have (=(2²) > 0, which, due to wr(2, x) = O, implies that (BeTt di TT leads to the following constraint on πyC
= 0. This
0 = ( de nt
(61a)
B((T))= 02 (TT))
(61b)
a2
al
(1 (1--0)) a()
(Te)=
(61c)
a1
df (TT)).
(T();
(61d)
0,(1 oz)
In other words, the weight we 22, being zero implies that (T())= 2 must satisfy the above ODE. Since म. is continuously differentiable in time, this ODE extends across all time 7 [0, 1)." Solving for (Λ==)= yields the unique solution
(π+)) C
at 1 a-
(62)
where == 1-aaa is given by the boundary condition (T=:) = 6. Note that C > O since ò > and oz >> 0. However, as T + 0, since a, + 1, this implies that (T+)) + +00 no matter how small is. In particular, there will some To >> after which ( < > > 1, which violates the assumption that ग,, is probability vector for all т. Therefore we have reached contradiction: It is impossible that 2x = 0 for all E Te. Consequently, we instead must have DxL + Dis implying that qi (z|zz) and ge (=(2)x)) are the same for all f, x, and = supp(qe (()(((((()))) (-|z:). Finally, since qe (-(=2)) and qe qe(-(x6) are equal on all of the support of (:(-(zr), they must share the same support and are therefore zero for all & supp(a: (-1z)). Since they are equal everywhere, the claim is proven.
Having shown both directions, this concludes the proof.
H.7. Uniform Noise Ratio
Proof. We need to show that if B = 27 I- then the expected proportion of uniform tokens is maximal at f = 1/2 and
equal to Pu- It is easy to see that Ct = Bt (1 - +) has a maximum at f = 1/2 and that the total mass on uniform tokens at c is monotonically increasing in er for a >> 0, its maximum coincides with
any time Σ. Szz T 8. Since S & Itc:
C, that of et at f 1/2. We then have
0/22 = B (1/2) - (1/2) =
Pa 2
Pu
(63)
Ps
1 Ps
and
C1/2
C1/2
1
1
01/2
1 01/2
1/01/2 +
I-pu
+
Pus
(64)
thus proving the claim.
H.S. GIDD ELBO Weights
We need to derive expressions for aé and B,
a -i For this, it is useful to first derive c::
c B
1 (1 2
1)
7 2
+))
By
Ce
)
By 1 21
(65)
2
2
1-t
2 t(1 t)
For o a,
we get
0/ (logar)' log 1-1 = -1
1 1-t
C C.
1
(66)
C.
1+
Wee use to denote time in order to avoid confusion with f which is contained to Ti by assumption. 12Stiielly speaking, a, can be exactly zero if t 1. However, we easily exclude this case by requiring 1. This does affect the overall ELBO since the case carries no mass.
27

---

Generalized Interpolating Discrete Diffusion
and for B at We we get
04
/ C (m c/ul
(1 () + cuu) (lm + cau)
B
t at
04
C 1-t =a;
a4
a1
(1 11-t] t)²
1 t (1 t + t)m
(ot + (1 -t)c))u
(67)
Ct
(1 t)2
m + (a + (1 t)
Ct(1 (t
which then is used to find (x(
1
m (a + (1-)tt)
wx(2,,)) x) =
Zt B
z:
qr (2(222)))
(e(2el1))
Ci(1-))
(68)
=27
m
(ce + (1 t)
(1- ()(( - 1)x + 1m + qu)
In summary, the ELBO constants for our mixing schedule are given by
1 d
we (FE, =z, (11- ()(())((11)))) t)x (
m + (er + -1- t)d)u
6 = 1 - 2t
(69)
1-t
1 1
im +cu))
2 1(1 1)
H.9. MDM is a Special Case of GIDD
We want to show that if we set π. = m, then the GIDD ELBO recovers the MDM ELBO, i.e. it reduces
af 6x.mx log xo (2, (z,,)) t) + C.
log p(x) S Bt,,, [ a
(70)
To show this, we first take a look at how individual terms of the GIDD ELBO simplify for this choice of n, First, note that (B,T,)= = -aam and hence
1 (
24 at
w (2)
z, B1
m'
m
8
S.. m
(71)
(0(2((z))
ge (x)
ge
az
We can see that the weight on any non-mask token is O, so can focus on simplifying the term inside the expectation
of the GIDD ELBO assuming that 2t m. In that case, we have (m(m)x) = qe(m)x)) x6) = (1 (1-aa) -as,, q (x(2)) = a.
de (z & (xr) = O, ge(z # m|xa)= = oz xo (ze, l), and therefore
= Σ ( (x) (<(21(x)) (=((((x))
+ (m(m/x)) qe (m/xx)
log qe (m((2))) -1 (72a)
D (1((
(m
=0ils'
log de (=(2(x))
(<(1(z))
(72b)
2e (=)))x))
unless =
ge (x|xx) log (x(x(x)
(72c)
(e(2x(x)
= au log 0z ar log(u;x x xo (2,,))
(72d)
-,x (z..)
(72c)
Combining the two results then yields
- log p(x) & Ey,,, [wa(ze, x)) (x))((())))))))))(xx))) + Dissve(a/ellla(())))))
(73a)
a' " (-axx log t)) +0
Ex,
(73b)
((((elz)) ae
of dx logxo x Zz, +C,
E1,,
(73c)
which is precisely the MDM ELBO and shows that GIDD is equivalent to MDM if n; m.
28

---

Generalized Interpolating Discrete Diffusion
I. Unconditional Generation Samples
Here we provide examples from our mask-only and our mask + uniform (px, = 0.2) model, with each sample presented twice: Once before self-correction and after the self-correction step applied with temperature 7 0.1.
I.I. GIDD+ BASE, Pa. = 0.0
This is our mask-only model which achieves best results on language understanding benchmarks. Due to being trained without uniform noise, sample quality is not improved by the self-correction step, and in fact actively made worse according our LLM-evaluation experiment.
1.1.1 I. EXAMPLE
No self-correction.
There's always something to please media fans, but this time you've ve got the vitriolic backlash to Hollywood mistakes on 20th Century Pox, including the foolish idea to remind people of the history of the Star Wars movies, first seen as sci-fi. And the story is UK-based, News. Daily, as if noting the video-based example of journalism have news filing need to suggest the fine- instrument: Bad News. Sadly, the resulting outrage here is no longer the -big-rdrob but more-actual. News in the world, the front pages touting the studios' contribution are something you could read without caution: "4 million people at Fox re-find in the past month to have exposed bodies." Not anytime anyone Fox movies I see this storyline on the Fox channel, but increasingly it's just been said all around you. Light sidelines miss James Cameron in 2003, for re-creating the original Terminator franchise to such acclaim. News that his films now seems to have more outraged those who regularly spend money/trying to watch the original Tv shows. Long before the current 'settling' between their two studios that the artifact lasts and prevailed, it is became easy, and people entertained, for a program chosen by taste to be hit. And the end results proved close to these three. *For the new day use "fairly"" that sentence...
Update:
h/t DruulllferriiilaalbBotto,, and MightydamesJade:: ON,
Watch the proud-anti-stae Wars skit from Hollywood
Self-corrected (r = 0.1)
There's s always something to please media fans but this time you've ve got the vitriolic backlash to Hollywood mistakes on 20th Century Fox,
very idea to remind people of
of the A Wars movies, first seen as sci-fi. And the story is UK-based, News. Daily, as if noting the video based example of journalism have a news filing need to suggest the fine- inst Bad News. Sadly, the resulting outrage here is no longer the -big-rdrob but more-actual. News in world, the front pages touting the studios' contribution are something you could read without caution: "4 million people Fox re-find in the past month to have exposed bodies." " Not anytime anyone Fox movies I but increasingly it's just been said all around you. Light sidelines miss James Cameron in 2003, for re-creating the original Terminator franchise to such acclaim. News that his films now seems to have more outraged those who regularly spend money/trying to watch the original Tv shows. Long before the current ssetllnng' between their two studios that the artifact lasts and prevailed, it is becane easy, and people entertained, for program chosen by taste be hit.
this storyline on the Fox channel,
29

---

Generalized Interpolating Discrete Diffusion
And the end results proved close to these three. #For the new day use "fairly"" that sentence...
Update:
h/t Drur and MightyyJamesJaae:: ON, Watch the proud-ane aoed-anti--i anti-Star Star Wars skit from Hollywood
1.1.2. EXAMPIEE 2
No self-correction.
[...] confiscation of their weapons from neighbors and supplies, and expropriation become the organizational sectors of assembly, production, and production. Roadblocks became increasingly difficult until the emergence of Hellfire missiles from the US. Production is more difficult due to logistical organizations provided by the US using the M4682/Chad drop pup (Csharp") 1864 rifles.
There are several other groups working in the area but are well known there: the village fighters can collectively run country, both in terms of fighting power and in the supply of materials (669 show and other chief wrapped clothing, and inoys supply 86 bolts) without the huge movement needed to expand through the Africa.
The groups who operate in Libya are also logistics-driven. They have fantastic operational networks organizing members retap their raids; the group is how to control strategic kerbs, the constant addition of the group often taking approach of a legal generating street marketing place, where the shop shuts up forcing the vendors to relocate out of the area. It is also possible to observe the ability to locate and approach frequently at checkpoints.
Fakesters are an area of danger assent. Members capture all office holders, deputies, officials, and candidates fleeing to Tripoli have pass through our protected areas, there taking place outside such locations and and it is gone they simply perform minor scenes elsewhere in the protected area overnight. The group was able to organize to move refugees into an attacking militia stronghold in Eastern Liby, at a morally sensitive site. Plenty of people were kidhapped only days later. This gives the very large number heavy armored vehicles in weapons gain access means by using the gravel mines abandoned there many years ago. These vehicles are also concerned with internal security, for Libya has Province, no power, but to have ethnicity, even a despot.
lots of opportunity to stop the leader protests
Later in 2003, the group took control of the incarceration of Kostaeil, the commercial capital of Apeda, where the large produoil but by pere export value additional background groups have increased intervention and intrigue into the business sector and on the intelligence scene:
One invention is in the informal community of Umesu Bil where a background group, armed to well established cells, infiltrated some insurgent installations in advance through the town of Kostaeil. The abduction targets individuals with intelligence (punding of local language), one especially, Mr. Halroy from the U.S noted what was happening and who noted that Mrs. [...]
Self-corrected (T 0.1).
[...] dation their weapons from the and supplies, and exp the, become the organizational sectors of assembly, production, and production. Roadblocks
30

---

Generalized Interpolating Discrete Diffusion
became increasingly difficult with the emergence of Hellfire missiles ssslees from the US. The is difficult due to the group provided by the US using the M4682/ch, drop pup (plsharp) and the rifles.
The are several other groups working in the area but are well- there: the village fighters to the run the country, both in terms of fighting, and in the supply of, the669 show, the chief wrapped clothing, and in the supply of, the without the huge movement needed to expand through the Africa.
The groups who operate in Libya are also logistics-driven. They have high operational networks organizing members to retap their raids, the group is how to control the kerbs, the constant addition of the group ofter taking approach of a legal generating street marketing place, where the shop shuts up forcing the vendors to relocate out of the area. It is also possible to observe the ability to locate and approach frequently at the.
The group are an area of danger assent. They capture the office holders, deputies, officials, and candidates fleeing to, have to pass though the back areas, so they are a of opportunity to stop the leader protests taking place outside the locations and and in is gone, to perform minor scenes elsewhere in the protected area overnight. The group was the to organize to the refugees into the the militia stronghold in Eastern Libia, at a morally sensitive site. Plenty people were kidnapped only days later. The group has fair number of heavy armored vehicles in the to gain the means by the the gravel mines abandoned in many years ago. The group are also the with internal security, for Libya has no Province, and government, but to have ethnicity, and despot.
The in 2003, the group took control of the town of Kostaeil, the commercial capital of the country, where the large produ, but by perc export value additional background groups have increased the and intrigue in the business sector and on the intelligence scene.
One invention is in the informal community of Umesu, where background group, armed to well in cells, infiltrated some the installations in the through the town of Kostaeil. The abduction targets individuals with intelligence (punding of local language), , one especially, Mr. Halroy from the U.S noted what was happening and who noted that Mrs. [...]
L.2. GIDD+ BASE, Pa- = 0.2
This is our best model in terms of sample quality and is trained on combination of masking and uniform noise. It is able to identify and correct mistakes, which allows it to improve sample quality during the self-correction step, both qualitatively and quantitatively.
1.2.1. EXAMPLE 3
No self-correction.
NBC Community Mystery Science Tour: let's talk about it. Though the Abrams's s show is about to drop cable this October, the second season looks squarely at NBC, which confirms rms what deal the network is in to as the family comedy follows CBS Studios for its third (sp) season. With that said, I have another (unjustified) update: Season 2 is in.
You know how ABC is breaking up on the word "GOSH" to "BLOOD MIDNLE AND CCOLORN" they share Calm similarities with? This #Inyybookfrrennti meme should make it familiar in your head pic wwwtter.com//von/ii comm22vonoo1fsp/7FF5.. -- Agent Cole (0AgentBlow) July 2, 2016
31

---

Generalized Interpolating Discrete Diffusion
Ayes of Tumblr and AnchorGateRQ have had fun putting together this pic of Alison Brie/Bobby Bure wandering down Twin Peaks street. Is that his family's recent death?Kid Cumberbatch dedication to DH (or Mount) is direct nod to Twin Peaks' creator David Lynch?
There's s nothing good from this pic: It used to be similar. Teddy Tu's s wearing the same scarf for a while. Charlie and son Paradise are all connected.
Cumberbatch is having some Scullyian fun here, and it's followed later by another "Thanks John, is it?"
Jess Bure and Beredict had it as a heinous serial killer, but then Forest Whitaker and his neighbor did the same thing.
ABC still also won't t confirm that Tony Hale will return given a role for Season 3 (or that will be coming back as co-star on the show).
What do we think? Will watch Community again next year, America? Yunande Mask
Self-corrected (r 0.1).
The Community Mystery Science Series, let's s talk about it. Though the show's s show is about to hit it in October, the first season is focused on Community, which confirms what a place the show is in, as the family comedy leaves CBS Television after its third (second) season. With that said, we have (via Classified) update: Season is in.
You know how Community is breaking up from the word AJOSH" to "BLACKSTYLE AND COLLORN" they share a name with? This #ndybookforeentl pic will make that stick Agent Blow (@AgentBlow) July 2,
in your wtt.:/tttrr.co///oot/t.t.o 2016
Eyes of Community and Anchor News Network have had fun putting together this pic of Alison Brie/Brie Brie wandering down Twin Peaks street. Is it the show's s recent death, Benedict Cumberbatch returning to Community (or Mount) reference to Twin Peaks' creator David Lynch?
direct
There's nothing good in this pic. Community used to be dead. Community has been wearing the same scarf for a while. Community and Twin Peaks are not connected.
Cumberbatch is having some Lynchian fun here, and it's followed up by "Thanks John, is it?"
Jess Brie and Beredict had fun as a heinous serial killer, but then Forest Whitaker and his neighbor did the same thing.
Community has also won't t confirm whether Tony Hale will be playing a role in Season 3 (or whether he will be coming back as co-star on the show)
What do you think? Will you watch Community again next year, America?
1.2.2. EXAMPLE 4
No self-correction.
[...] oil;) Tul serious bid here for OPEC news to be operational for U-T policy. The most interesting part is that it there, "Ask Exxon, out of it" in an internal investigation though secret that formerly also bankrolls the oil companies equally interesting. Even though the biggest drubble been Fetzo-Exxon corporation, there is four bidding to third international pads. Price in fact the greying glare in that original story. One of these, being overseen by Mr.
32

---

Generalized Interpolating Discrete Diffusion
Slovakia, has mystery poker any one, and to officials in the kingdom. He always finds that the price of oil and gas rises (yes 'other) therefore relief will come to go buy the giants of oil reserves at maturity. Then the boom will begin. See, the government has gone on their way.
At the outset, Exxon he was a major corporate investment. At the longest point of time Mr. Redi was all generous of support for others, and and importantly of all he has wife, USPOC expembaddin SaathJu who is very Low colored Oil Minister with funny eye. But on that grand reform she went off the shelf to his bandwagon.
The raj, these are all phases, have had little impetus encouraged, with the Saudi reversal. Yet the business has been on ice as confusion, as the be discovered small moment left with students of production and 2010, will be tasked with determining what to do new.
a struggle over the link in oil of gas sales, they would be including exports, Gulf States for LPG, Iraqi government oil for export and even encouraged EN leaders are emboldening the time of knocking down theigsaw to the funnatively new renaissance that Iraq appears to have, which will be going to China, without counting oil. China will likely invest in crude, and then on top of the country imports, along with shale gas to meet current needs. They have been most approved of the fact that more stable pipeline between refinery refineries means there prosperity "live the clean room, quant crude does, and cover the hole.
While the idea of PetroWestman was ignited, by All Sugar Taylor, of the Chaotic Oil when it was conceived to work so far, Exchange and oil businessmen have been less eager to see in this spirit as oil production has been far slower than anybody imagined. With the factors cause been oil prices of the 80s declining, and the surging price of heavy countermarket oil and gas [...]
Self-corrected 1==00 0.1).
[...] oil; and new bid, to OPBC, to be operational the U-T policy. The most interesting part is that the there,-Saude is out of it, and the internal, Exxon, which formerly also bankrolls the oil, is not interesting. Even though the new drabble is Saudi-Exxon Corporation, there are four bids to the international cartel. This is in fact the tidying part of the original plan. One of these, being overseen by Mr. Slovakia, is a mystery, to the one, and to officials in the kingdom. He always finds that the price of cil and gas rises (the-other) and the time will come to go buy the giants of the oil in Iraq. Then the boom will begin. See, the government has gone on the way.
At the time, Exxon he was a major corporate investment. At the same point of time Mr. Redi was all generous of support to others, and most important of all he has a wife, USPoC-embaddir Saath, who is a very Low, and Minister with a funny eye. But on the grand reform she is on the way to the bandwagon.
The raj, which are the phases, have been little impetus encouraged, the Saudi reversal. Yet the business has been on ice as well, as the newly discovered oil moment, with students of production in 2010, will be tasked with deciding what to do next.
with
the struggle of the link of oil and gas sales, which will be including exports to Gulf States for LNG and Iraqi government oil for export, even the EN leaders are emboldening the idea of knocking out theigsaw of the funnatively new oil that Irag appears to have, which will be sold to China, without the oil. They will
33

---

Generalized Interpolating Discrete Diffusion
likely buy the oil, and then on top of the country oil, along with shale gas to meet their needs. They have been most approved of the fact that pipeline to the refineries means more oil, in the clean room, as crude does, and in the hole.
stable
While the idea of Petro-Iraq was ignited, by the Sugarman, and the Chaotic, when it was conceived to work so far, Exchange and oil businessmen have been less eager to participate in it, as oil production has been much slower than they expected. With the main price of oil, in the 80s declining, and the surging price of the upmarket oil and gas [...]
34


Reviews:
[{"review_id": "w573NCEm4s", "reviewer": "Reviewer_LBxr", "summary": "This paper generalizes discrete diffusion models with masked or uniform transition kernels to a larger design space. Specifically, the authors introduce a Generalized Interpolating Discrete Diffusion process (GIDD), which transfers data not to the [mask] state but to an arbitrary predefined distribution. They further prove the existence of such transition kernels and provide ELBO for GIDD. This paper especially focuses on mixing the masked and the uniform noising schemes and demonstrates the empirical advantage of GIDD against existing non-autoregressive methods of language modeling.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful and detailed review and for acknowledging the theoretical contributions of GIDD.\nIn the following, we would like to shed some additional light on the empirical evaluation of the proposed model and, hopefully, address the reviewer’s concerns.\n\nBroadly speaking, our evaluation (and evaluation of language models in general) consists of two parts. The first part is likelihood-based evaluation, which tests the model’s ability to _recognize_ well-formed, high-likelihood sentences, and includes PPL on the test set as well as downstream multiple-choice benchmark performance. The second part is to test the model’s ability to _generate_ well-formed, high-likelihood sentences and for unconditional generation, this is most commonly measured with generative-PPL. This ability is equally, if not more, important because it is how language models are usually used in practice. And while the two abilities may correlate in general, sampling from the model can sometimes introduce challenges not present when simply evaluating likelihood. For example, teacher-forcing is used both when training autoregressive models and when evaluating their likelihood, but not when generating samples, which makes them prone to self-induced mistakes [1]. The same holds true for mask-only diffusion models. The point of our experiments is to show that while, indeed, hybrid diffusion models find it more challenging to evaluate the likelihood of given samples, they have self-correction capabilities and are more robust/less prone to self-induced errors at sampling time, thus generating higher-quality samples overall and especially so for low inference-compute budgets (see Sec. 5.4, L435 ff.; Fig 6, App. D). To further bolster this point, we provide an additional LLM-based evaluation in our response to Reviewer wojY, where we quantify the improvements in sample quality during the self-correction step in terms of clarity, grammaticality, factuality, writing style, and creativity. Perhaps the distinction between _recognition_ and _generation_ should be highlighted and discussed more prominently in the paper, which we will be happy to do in an updated version.\n\nRegarding the second weakness: The reason we cannot naively apply self-correction to the multiple-choice benchmarks is because these benchmarks rely on likelihood-based answer selection. Specifically, for a given question (or prompt) with a set of possible answers (or continuations), the one with the highest likelihood under the model is selected. Since this process does not involve sampling from the model, it is a priori not possible to utilize the self-correction capabilities. As a result, benchmark scores of $p_u > 0$ models are hampered in correlation with their worse likelihood (see Tab. 5, App. B). Future work may aim to close the gap by using different criteria for selecting the correct answers (e.g. self-accuracy instead of likelihood), but unfortunately, this was beyond the scope of this project. Alternatively, future work may extend the proposed models to conditional generation, which will allow evaluation on generative benchmarks like GSM8k or HumanEval, where the generative strength and self-correction capabilities of hybrid models may bring substantial improvements over mask-only diffusion models.\n\n[1] Bachmann & Nagarajan, 2024. https://arxiv.org/abs/2403.06963", "claims_and_evidence": "Theoretical and empirical contributions are claimed in extending the discrete diffusion framework, which is well supported by evidence.", "methods_and_evaluation_criteria": "The proposed method, GIDD, extends discrete diffusion models to generate categorical data. It is evaluated on language generation (by perplexity) and accuracy over downstream tasks, which is an effective evaluation criterion. The effectiveness of the proposed hybrid noising process is evaluated by performing self-correction sampling and computing the generative perplexity under Gemma 2 9B.", "theoretical_claims": "The paper characterizes the Markov chain for the GIDD process and shows the ELBO of GIDD. The reviewer checked the proofs of Proposition 3.3, Lemma 3.6, and Theorem 3.7 and believes that the proofs are correct.", "experimental_designs_or_analyses": "The reviewer checked the experimental design and analyses. The implementation of GIDD with $p_u=0.0$ resembles MDM, which is consistent with the theoretical claims. The additional uniform noise slightly worsens the PPL of GIDD, as shown in Table 3 and Table 5, which seems to weaken the claim that combining masking and uniform noise improves sample quality. The authors also analyze the proposed self-correction sampling method, showing better self-correction ability of GIDD with additional uniform noise.", "relation_to_broader_scientific_literature": "This paper may be of interest to a broader audience in other scientific domains, such as protein generation with discrete diffusion models.", "essential_references_not_discussed": "Related works are well discussed in this paper.", "other_strengths_and_weaknesses": "#### Strengths\n\n- This paper provides a richer design space for discrete diffusion models that would be intriguing to explore for future works. This generalized framework is already a contribution by itself. \n- The proposed GIDD process is rigorously formulated and well articulated.\n\n#### Weaknesses\n\n- It seems that the additional uniform noise on top of the masked diffusion process does not improve the overall performance of the discrete diffusion model, which undermines the meaningfulness of the hybrid noising process. \n- The advantage of GIDD with $p_u>0$ is discussed in Section 5.4, but the importance of self-correction is not well explained. For example, does this self-correcting scheme improve downstream benchmark accuracy for GIDD with $p_u>0$?", "other_comments_or_suggestions": "See above.", "questions_for_authors": "Some of my points listed in weaknesses may be inaccurate, and I would be grateful for any clarification. Specifically, what is the importance of self-correcting and why is it not demonstrated in Table 2~4?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a5Ws3Tafz4", "reviewer": "Reviewer_wojY", "summary": "As a class of models currently attracting significant attention, masked diffusion models suffer from a fundamental limitation: once a token is generated, it cannot be modified. To address this issue, this paper introduces General Interpolating Discrete Diffusion (GIDD), which allows for a more flexible noise formulation. The authors derive the forward process, backward process, and loss function for GIDD. Additionally, they propose several techniques to stabilize training and improve performance. Experimental results show that while GIDD increases the modeling difficulty—leading to a performance drop in language modeling perplexity and downstream tasks—it effectively demonstrates self-correction capabilities.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful review and constructive feedback. In the following, we would like to respond to the reviewer’s comments and suggestions and provide additional experimental results to further bolster the claim of self-correction in the presented models.\n\n1. We mostly agree with the reviewer on the comment regarding overfitting. The difference is indeed small, and performance is almost random on some datasets. Given that the difference is consistent, albeit small, perhaps the appropriate characterization would be to say that it is likely run-to-run variance with some likelihood of overfitting. As mentioned in our response to Reviewer LeK6, we see a potential cause of overfitting in the way long sequences are handled in our data loader, which leads to more frequent repetition of short sequences, potentially causing overfitting.\n2. We agree with the reviewer that this would be a good addition and would help quantify the diversity loss during self-correction. Given the mode-seeking nature of the self-correction step, some decrease in diversity is to be expected, but this should be restricted to a reasonable amount. Indeed, we find a decrease in entropy for $p_u>0$ models, with the decrease correlating linearly with the number of changed tokens (see Table 1 below). However, the decrease is moderate and does not indicate any collapse. This is also supported by the LLM-based evaluation (see 3.). For further context, we also provide qualitative self-correction examples in Appendix K, which give some intuition on the nature and extent of the reduced diversity.\n3. Unfortunately, conducting a user study comes with its own host of challenges and is beyond the scope of this project. However, we have conducted an LLM-based evaluation of the quality of generated samples via GPT-4o using single-answer grading [1] (prompt omitted due to char. limit). The samples are graded on a scale from 1 to 10 in terms of clarity, grammaticality, factuality, writing style, and creativity. We would like to emphasize that these absolute numbers are highly dependent on the judge model’s calibration and should therefore be taken with a grain of salt. Nevertheless, we find consistent improvements at high significance levels, with $p_u=0.2$ exhibiting both the largest improvement and the highest scores overall (see Table 2 below). We report the self-correction setting with the largest effect for each noise level.\n4. Given that our models are comparatively small and not trained on instruction following or conditional generation, evaluation on GSM8k is unfortunately not possible. Even if we try to do pseudo-conditional generation by forcing the logits of prompt tokens, the model does not generate meaningful continuations.\n\nRegarding the reviewer’s questions:\n1. The “number of tokens changed” refers to the number of tokens that differs from the initial sequence after convergence/termination of the self-correction step. While this roughly correlates with the number of inference steps, it is not a one-to-one correspondence since self-correction may, at times, oscillate between two or more similar states (e.g. multiple options that are equally correct/incorrect). We therefore deem the number of changed tokens after convergence to be a more meaningful metric compared to simply the number of self-correction steps.\n2. Indeed, due to the continuous-time nature of the GIDD diffusion process, the number of denoising steps can be set arbitrarily high. Due to our chosen parameterization, where the model predicts the fully noise-free data which is then re-noised up to the appropriate level, uniform noise is actually continually injected at low levels, so tokens continue to change throughout. Future work can explore different approaches that avoid this, which may lead to improvements.\n\n[1] Zheng et al., 2023. https://arxiv.org/pdf/2306.05685\n\n---\n\n**Table 1**\n||$p_u=0.0$|||$p_u=0.1$|||$p_u=0.2$||\n|-|-|-|-|-|-|-|-|-|\n|Temperature $\\tau$|#tokens changed|Entropy||#tokens changed|Entropy||#tokens changed|Entropy|\n|(no self-correction)|0.0|3.13||0.0|3.05||0.0|3.03|\n|0.01|14.5|3.12||5.78|3.04||7.65|3.01|\n|0.05|46.2|3.06||23.7|2.98||28.2|2.94|\n|0.1|62.6|3.12||44.8|2.94||58.4|2.87|\n|0.5|52.4|3.13||37.8|2.95||60.2|2.87|\n|1.0|41.4|3.12||35.3|2.96||51.1|2.89|\n\n**Table 2**\n|Model|Clarity|Grammaticality|Factuality|Writing style|Creativity|\n|-|-|-|-|-|-|\n|GIDD ($p_u = 0.0$)|2.51|2.96|3.61|2.84|4.48|\n|+ self-correction ($\\tau = 0.1$)|1.99 (-20.9%**)|2.39 (-19.3%**)|3.02 (-16.2%**)|2.24 (-21.1%**)|3.60 (-19.5%**)|\n|GIDD ($p_u = 0.1$)|2.51|2.85|3.66|2.78|4.26|\n|+ self-correction ($\\tau = 0.1$)|2.69 (+7.2%**)|3.05 (+6.9%**)|3.88 (+6.0%**)|2.98 (+7.1%**)|4.35 (+2.1%*)|\n|GIDD ($p_u = 0.2$)|2.49|2.82|3.70|2.79|4.25|\n|+ self-correction ($\\tau = 0.5$)|2.90 (+16.5**)|3.29 (+16.6%**)|4.01 (+8.5%**)|3.16 (+13.4%**)|4.48 (+5.5%**)|\n\nSignificance levels: * $>2\\sigma$ difference, ** $>5\\sigma$ difference.", "claims_and_evidence": "The claims made in this paper are not well supported by the experimental results. \n\nSpecifically, language modeling perplexity and downstream task performance are the primary metrics of interest, yet the proposed method leads to worse performance on both.\n\nRegarding generative PPL, this metric is not entirely reliable. Even low-quality sentences (e.g., repetitive words) can receive low PPL from large language models, a limitation that the authors themselves acknowledge. Given this, the advantage demonstrated by GIDD is only observed on this unreliable metric, raising concerns about the validity of the claimed improvements.", "methods_and_evaluation_criteria": "No. As discussed in Claims and Evidence, the primary evaluation metrics (language modeling PPL and downstream task performance) show a decline, while the generative PPL metric used to support the method is unreliable. Thus, the evaluation does not convincingly justify the claims.", "theoretical_claims": "No. I did not verify every step of the derivations, but the theoretical arguments appear logically sound and well-structured.", "experimental_designs_or_analyses": "Yes. Please see Claims And Evidence.", "relation_to_broader_scientific_literature": "Discrete diffusion models have recently made significant progress in text generation, particularly masked diffusion models. However, a key limitation of masked diffusion models is that once a token is generated, it cannot be modified—an inherent constraint of their noise injection process. Addressing this issue has been a major focus in the field. \n\nThis paper introduces a generalized forward process that combines masked noise and uniform noise, representing a valuable exploration of this problem.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The paper has a clear motivation, well-structured theoretical derivations, and is written in a clear and concise manner. \n\nHowever, the main weakness is that the experimental results do not sufficiently support the effectiveness of the proposed method (as detailed in Claims and Evidence). Although the authors introduce \"self-accuracy\" as an evaluation metric, they do not provide sufficient justification for its validity.", "other_comments_or_suggestions": "1. In Section 5.3, the authors claim that GIDD trained on only 131B tokens surpasses models trained for twice as long, attributing this to overfitting on spurious patterns in the training data. However, this explanation seems questionable, as such results are more likely due to random variation rather than overfitting. Moreover, on datasets such as ARC-c, BoolQ, OBQA, and WinoG, both models perform close to random guessing, making it difficult to determine which is superior.\n\n2. When using generative perplexity as an evaluation metric, I suggest including entropy as a complementary measure to assess the diversity of generated sentences, providing a more comprehensive perspective.\n\n3. To better evaluate the quality of generated text, I recommend conducting a user study or leveraging LLM-based scoring, as these approaches would offer more reliable and interpretable assessments.\n\n4. Would it be possible to include results on reasoning benchmarks, such as GSM8K?\n\nIf the author provides the aforementioned more detailed evaluation metrics (or at least some of them), I will increase my score.", "questions_for_authors": "1. In Figure 3, what does \"Tokens changed\" represent? How does it relate to the number of sampling steps?\n\n2. For GIDD with uniform noise, is it possible to set the number of sampling steps to an arbitrarily large value? I am particularly curious whether tokens continue to change in the later stages of sampling when the number of steps is very large.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gvdmv9BzfD", "reviewer": "Reviewer_LeK6", "summary": "The method overcomes existing limitations in autoregressive models and discrete diffusion approaches by introducing a generalized interpolating discrete diffusion. This innovation offers enhanced flexibility in the noising process design by combining masking and uniform noise, enabling the revision of previously generated tokens.\n\n### After rebuttal comment\nOveral, the paper is of sufficient quality to warrant acceptance given how flexible the method enables for self-correction, which is also an important aspect of not just diffusion methods but also AR models. That's said, the concerns regarding overfitting problem is not fully addressed and I believe the authors should be upfront about this issue. Since looking at table 4, the model with less training iterations obtain better metrics than the longer one, yielding a concern regarding the scalability of the model on large-scale training like bigger model size, large-scale dataset. In addition, the evaluation of self-correction using LLM-based metrics should be included in the final manuscript and discussed thoroughly. Given these factors, I decide to keep my current rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and insightful questions. We especially feel that 2) deserves careful discussion since we consider the theoretical contributions a key part of our work. We would like to encourage the reviewer to share some additional detail on their concerns.\n\n1. We agree with the reviewer that a discussion of [4] needs to be added. While [4], and also [2] (App. A.2.6), experiment with a BERT-like combination of masking + uniform noise, these approaches are only discrete-time and do not investigate what improvements/challenges arise from it.\n2. We agree that there are some similarities between our theoretical contributions and the suggested references [1, 2] and would like to highlight the important differences. The theory behind GIDD is a strict generalization of masked diffusion (MD) in [1] (see Cor. 3.8). Unlike MD, GIDD allows for a time-variable mixing distribution $\\pi_t$, which makes MD a special case where $\\pi_t=\\mathbf{m}$ is constant w.r.t. time. Another example would be setting $\\pi_t=\\mathbf{1}/|V|$ where $|V|$ is the vocabulary size, which simplifies to uniform diffusion from [2]. As for [2], this paper covers the most general case of discrete diffusion and forms the foundation of much subsequent work, including [1] and ours. In this framework (and in discrete diffusion in general), complete knowledge of the transition matrix is paramount and necessary both for computing the ELBO as well as for sampling. In this regard, our theoretical contribution is to solve the inverse problem of finding the (closed-form) Markovian transitions given only the marginal forward transitions for a special family of linearly interpolating diffusion processes (see Prop. 3.3). In addition, while [2] only covers discrete-time, the GIDD ELBO and Markov transitions are stated in continuous-time by applying the tools from [3]. We hope that this sufficiently highlights our theoretical contributions and that we were able to address the concerns of the reviewer.\n3. We agree with the reviewer that the confidence-weighting in the self-correction step is similar to the MaskGIT sampling algorithm. However, ours is a fixed-point iteration that resamples one token at a time and is only applied after denoising, whereas MaskGIT sampling is an adaptive masked diffusion sampler prioritizing high-confidence tokens. Indeed, applying MaskGIT sampling to GIDD ($p_u=0.0$; base) finds that this is prone to collapse, with generated samples being low-diversity and consisting mostly of repeated tokens. This manifests in a low generative entropy (via Gemma-2-9B) of 1.31 compared to the baseline of 3.13. MaskGIT sampling is unfortunately not applicable to $p_u>0$ models since those require token replacements in addition to unmasking. We thank the reviewer for pointing out this similarity and will discuss it in an updated version.\n4. Preliminary experiments showed that setting the ELBO weights to 1 works for $p_u=0.0$ but is challenging for $p_u>0$, and in light of the clipping strategy working much better, we did not investigate this any further. We hypothesize that the weight ratio between uniform and noise-free tokens is important for learning the correct posterior: What is the distribution of the correct token, given that it is incorrect with only a small probability?\n5. Regarding overfitting, we would like to point out that for $p_u=0.0$, MDM and GIDD are equivalent, so it is likely that if GIDD is overfitting, then the baselines are also overfitting. One potential cause could be the way long sequences are handled in our data loader (random cropping instead of splitting), which leads to more frequent repetition of short sequences, potentially leading to overfitting. The 1.1B MDM baseline is included purely for context as training 1.1B GIDD models was unfortunately not feasible given the available resources.\n6. Discouraged by the higher loss and PPL of $p_u>0$ models, we did not experiment much with noise levels above $p_u=0.2$. Generally speaking, higher values are more challenging as the SNR drop gets more concentrated on early steps.\n7. Unfortunately, we are not entirely sure which entropy the reviewer is referring to. Since PPL is just the exponent of cross-entropy, the comparison would not qualitatively change by reporting cross-entropy.\n8. Since self-correction is run for at most 128 iterations (subject to early stopping) and the samples are generated in 128 denoising steps, the worst-case computational overhead is double. However, even if self-correction incurs some overhead, it is generally possible to improve sample quality while keeping the inference compute budget constant by reducing the number of denoising steps in favor of additional post-hoc correction steps.\n\n- [1] Shi et al., 2024. https://arxiv.org/abs/2406.04329\n- [2] Austin et al., 2021. https://arxiv.org/abs/2107.03006\n- [3] Campbell et al., 2022. https://arxiv.org/abs/2205.14987\n- [4] Gu et al., 2021. https://arxiv.org/abs/2111.14822", "claims_and_evidence": "The problem of keeping already generated tokens unchanged is clearly presented, highlighting the essence of the proposed method. However, to revise generated tokens, the method still relies on a uniform diffusion process that is not deterministic and controllable. Regarding the self-correction step, the resampling strategy for low-confident tokens is similar to the MaskGIT paper.", "methods_and_evaluation_criteria": "See above.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "- Should not let empty cells in table 4.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strength: Paper is well-written and easy to follow.\n\nWeakness: See questions below.", "other_comments_or_suggestions": "NA", "questions_for_authors": "1. The use of uniform noise for mask diffusion is also introduced as a mask-and-replace transition in \"Vector Quantized Diffusion Model for Text-to-Image Synthesis\" paper. A discussion should be included.\n2. The proof of proposition 3.3. should be put in the supplementary as it is not a significant part of the paper.  And It is quite similar to the derivation of existing works like \"Simplified and generalized masked diffusion for discrete data\", \"Structured denoising diffusion models in discrete state-spaces\". Section 3 appear to present background information rather than the paper's technical contribution, excluding the ELBO 3.3. \n3. Self-correction step is similar to the sampling strategy in MaskGIT paper (MaskGIT: Masked Generative Image Transformer).\n4. What is the method's performance when ignoring weighting term? This mean the weight is set to a constant 1.\n5. Table 4 raises questions about GIDD's performance when trained for longer periods, as the results suggest potential overfitting. It's unclear why the authors include other baselines with 1.1B parameters while omitting results for their method at the same parameter size, making direct comparisons difficult.\n6. How does large uniform noise (e.g., 0.5) influence model performance?\n7. Table 3 demonstrates that the use of uniform noise does not improve perplexity (PPL). An alternative approach would be to report entropy instead.\n8. Since the method has an additional self-correction step, I wonder it will introduce a latency for the sampling process? A report of sampling speed is needed (e.g. tokens/sec)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Masked diffusion models are inherently constrained by the inability to revise tokens once they are generated. To overcome this limitation, the paper presents General Interpolating Discrete Diffusion (GIDD), which adopts a more flexible noise design to enable iterative refinement of tokens. The effectiveness of GIDD is demonstrated through a series of experiments.

The paper received three reviews. In response to the concerns raised in the initial evaluations, the authors submitted a rebuttal that included additional experimental results and clarifications, effectively addressing the key issues. Following the rebuttal and subsequent discussions, all three reviewers reached a consensus, leading to a decision to weakly accept the paper.

This agreement is attributed to the paper's innovation, profound theoretical justification, and sufficient empirical validation that it offers for this particular study. The AC agrees with the reviewers and recommends accepting the paper. To further improve the paper quality, AC suggests the authors to revise their paper by taking into account all the suggestions provided by the reviewers, including integrating extra experimental results conducted during the rebuttal phase.
Decision: Accept (poster)