Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data
Authors: Jingyang Ou
Shen Nie
Kaiwen Xue
Fengqi Zhu
Jiacheng Sun
Zhenguo Li
Chongxuan Li
Venue: ICLR 2025 Poster (2025)
Keywords: Discrete Diffusion Models
Diffusion Models
Language Modeling
Concrete Score
Score Entropy
Primary area: generative models

Abstract: Discrete diffusion models with absorbing processes have shown promise in language modeling. The key quantities to be estimated are the ratios between the marginal probabilities of two transitive states at all timesteps, called the concrete score. In this paper, we reveal that the concrete score in absorbing diffusion can be expressed as conditional probabilities of clean data, multiplied by a time-dependent scalar in an analytic form. Motivated by this finding, we propose reparameterized absorbing discrete diffusion (RADD), a dedicated diffusion model without time-condition that characterizes the time-independent conditional probabilities. Besides its simplicity, RADD can reduce the number of function evaluations (NFEs) by caching the output of the time-independent network when the noisy sample remains unchanged in a sampling interval, which enables sampling acceleration. Built upon the new perspective of conditional distributions, we further unify absorbing discrete diffusion and any-order autoregressive models (AO-ARMs), showing that the upper bound on the negative log-likelihood for the diffusion model can be interpreted as an expected negative log-likelihood for AO-ARMs. Further, our RADD models achieve SOTA performance among diffusion models on 5 zero-shot language modeling benchmarks (measured by perplexity) at the GPT-2 scale. Our code is available at  \url{https://github.com/ML-GSAI/RADD}.

Published as a conference paper at ICLR 2025
YOUR ABSORBING DISCRETE DIFFUSION SECRETLY MODELS THE CONDITIONAL DISTRIBUTIONS OF
CLEAN DATA
Jingyang Ou Shen Nic Kaiwen Xue Fengqi Zhu
Jiacheng Sun Zhenguo Lix, Chongxuan Li1.2*
Gaollng School of Artificial Intelligence, Renmin University of China
"Beijing Key Laboratory of Big Data Management and Analysis Methods, Beijing, China 3 Huawei Noah's 's Ark Lab loujingyang, nieshen, kaiwenxue, chongxuan H eussc.edu.cmm cn; sengqizhu@wir whu. edu, en; (onjssunjian [snnhacceengl,, 1i. zhenguo @huawei com;
ABSTRACT
Discrete diffusion models with absorbing processes have shown promise in lan- guage modeling. The key quantities to be estimated are the ratios between the marginal probabilities of two transitive states all timesteps, called the concrete score. In this paper, reveal that the concrete score in absorbing diffusion can be expressed as conditional probabilities of clean data, multiplied by a time-dependent scalar in analytic form. Motivated by this finding, propose reparameterized absorbing discrete diffusion (RADD), a dedicated diffusion model without time- condition that characterizes the time-independent conditional probabilities. Besides its simplicity, RADD can reduce the number of function evaluations (NFEs) by caching the output of the time-independent network when the noisy sample remains unchanged in sampling interval, which enables sampling acceleration. Built upon the new perspective of conditional distributions, we further unify absorbing dis- crete diffusion and any-order autoregressive models (AO-ARMs), showing that the upper bound on the negative log-likelihood for the diffusion model can be interpreted
expected negative log- likelihood for AO-ARMs. Further, RADD models achieve SOTA performance among diffusion models on zero-shot language modeling benchmarks (measured by perplexity) at the GPT-2 scale. Our code is available at thub. com/ ML- MLLLCALIRAAADD
INTRODUCTION
Auto-regressive models (Radford et al. 2018 2019b Brown al.) 2020) have dominated the
area of language modeling for many years. In particular, such models significantly benefit from and training data and have achieved remarkable
large-scale transformers (Vaswani al. 2017a)
progress (OpenAl 2022 Achiam et al. 2023 Touvron et al.) 2023 Anil et al.) 2023). From
sequential sampling process of auto-regressive models is inefficient and limits the reasoning ability in nonscquential orders (Berglund et al. 2023 Lv et al. (023). Intrinsically, this is because such models characterize the joint distribution by the chain rule of probability, motivating research developing other types of generative models for text.
probabilistic perspective,
Diffusion models (Sohl-Dickstein et al. 2015: Ho et al.
2020
Song et al. 2021b) generate data in a 2022 Zhang & Chen 2023 [u et al.)
coarse-to- to-fee manner efficiently (Song et al. [222aa] Bao et al.
2022a b) and all dimensions simultaneously ly. providing an appealing alternative to auto-regressive
models. Among other efforts (Li al. 2022 Austin et al. 2021 Dieleman al. 2022 Chen et al.
2023 Graves et al. 2024 Xue et al. 2024 He et al. 2022 Campbell et al. 2022 Sun et al. 2023b
Meng et al.) 2023 Lou al. (024)) (see Section 5 for comprehensive discussion), score entropy
discrete dillusion (SEDD) (Lou et al. 2024) has shown promise in text generation. In particular,
SEDD has achieved comparable results auto-regressive models zero-shot language modeling
*Corresponding Chongxuan L.i.

---

Published as a conference paper at ICLR 2025
benchmarks at the GPT-2 scale. Meanwhile, SEDD reduce the number of function evaluations (NFEs) in sampling and fulfill text conditioned on prompts at different positions.
Technically, SEDD employs discrete-state (absorbing) Markov process that adds noises to data by randomly replacing a token with a mask token [M] and then learns a reverse process to denoise from an entirely masked sentence. The key quantities to be estimated are the ratios between the marginal probabilities of two transitive states at all timesteps, called the concrete score. SEDD also proposes a "scaling trick" (see details in Section 3 that scales the output of the score estimation by a factor. The trick has been proven effective in practice yet not fully understood in theory (Lou et al. 2024).
One of our main contributions is reveal that the conerete score in absorbing diffusion can be expressed as conditional probabilities of clean data, multiplied by a time-dependent scalar in an analytic form (see Theorem [T] Our finding theoretically explains the benefits of the scaling trick as a reparameterization for better optimization. Motivated by the finding, we propose reparame- terized absorbing discrete diffusion (RADD), dedicated diffusion model that characterizes the line-independent conditional probabilities by removing the time conditions from the score estimation (see Fig. D. Besides its simplicity, RADD can significantly reduce the NFEs by caching the output of the indeendndeen network when the noisy sample remains unchanged during sampling interval.
Built upon the new understanding the concrete score, we further unify absorbing discrete diffusion and any-ordet autoregressive models (AO-ARMs) (Uria et al. 2014 Hoogeboom al. 2022 Shih
ctt al. 2022), demonstrating that their training objectives are equivalent (see Theorem 2 To establish first rewrite the original training objective for absorbing discrete diffusion into simpler form (named t-denoising cross-entrope t-DCE). Then, we apply a change of variable from the time / to the probability that a singleddimensional token is masked process. By integrating the probability variable analytically, we show its equivalence to the training objectives for AO-ARMs. These theoretical findings offer a fresh perspective that the upper bound on the negative log-likeliig liglliod of an absorbing discrete diffusion can be interpreted as the expected negative log- ligeeioll for corresponding AO-ARMs. Furthermore, they provide alternative objective functions for training and likelihood evaluation.
the theory,
time in the forward
Empirically, the RADD model converges faster while achieving similar performance to the strongest baseline, i.e., SEDD (Lou et al. (024). Moreover, we trained our RADD models on different objective functions, achieving state-of the performance among diffusion models on five zero-shot language modeling benchmarks (measured by perplexity) at the GPT-2 scale. This empirical evidence validates our theoretical findings.
In summary, this paper has several contributions:
Deeper understanding of discrete diffusion: Both the factorization form of the concrete score and unified training objective for absorbing discrete diffusion and A0-ARMs reveal important yet overlooked theoretical properties of absorbing discrete diffusion, which explain the mysterious scaling trick, provide practice guidance, and may inspire future work.
Simpler parameterization: By removing the time conditions, we reparameterize the model to focus on a ineeinndeendeent conditional probability, simplifying the existing model.
Efficient sampling: Leveraging the reparameterized form, RADD can use caching strategy to improve the sampling speed and achieve faster convergence.
Enhanced zero-shot language modeling performance: Our architectural simplifications and optimized training loss lead to superior results. On five zero-shot language modeling benchmarks, RADD achieves state-of-the tof-the-att performance among discrete diffusion models (measured by perplexity) at the GPT-2 scale.
2 BACKGROUND
In this section, provide overview of continuous-time discrete diffusion models (Section 2.1 and any-order autoregressive models (Section 2.2). with a more detailed discussion available in Appendix G For complete list of notations and definitions, please refer to Appendix A
2

---

Published as a conference paper at ICLR 2025
2.1 CONTINUOUS TIME DISCRETE DIFFUSION MODEL
Single dimension Let I denote a single dimensional sample with possible values in A {l,.... N,.. A continuous-time discrete Markov chain at time is characterized by transition rate matrix Qi as follows
(e( (At)) P(+ [1+ (,( + (Al),
xΦ z, == x,
(2.1)
where Q: (z. Ø) is the (x, 2) element of transition rate matrix Qs. denoting the transition rate from state z to state & at time t. Equivalently, Qx 2) is defined
limat-o P=tan:((
Qx)
(2.2)
lim (
== x.
Al
Given the above definition, denote P. equation holds (Campbell et al. 2022 Anderson 2012):
(x, a:) == Pel. (+|x) The following Kolmogorov's forward
Pris = Pi
(2.3)
dt
In practice (Campbell et al. 2022). Q: is parameterized as (t) where (t) is scalar function
representing the noise schedule and Q is a constant matrix. In this case, the solution to Eq. (2.3)
can be solved analytically as Pt|s exp (t) - o (s))) where (t) = fo a )ds and exp is the directly sample 24 from F, in step for any >
matrix exponential. Therefore,
Further, Q is often designed to diffuse towards a uniform distribution or an absorbing state [M]. Recent work (Austin et al.) 2222]] Campbell et al. 2022) suggests that the absorbing matrix achieves better empirical performance. Besides, as detailed in Section 3 the specific structure of the absorbing matrix be leveraged improve performance and accelerate sampling. Therefore, focus the absorbing matrix as follows:
0
1
0
1
Q Qabsorrh
(2.4)
-1
1
0
The time reversal of the forward process is characterized by reverse transition rate matrix Q: (unn
al. 2023a Kelly 1987), whose element from state Ie state d; is given by
(it,, Pe(we)
21 + Zt:
(2.5)
E (x),)),
2t = Zt.
Simulating the reverse process requires learning the reverse transition rate Qe(a,, 2.). As Qe 24) ((()
known, it is sufficient to estimate the concrete score (=(a.) by a score network se sx t) se
(((()) pr (xx) &.Ex (Meng et al. 2023). Denoising score entropy (DSE) (Lou et al. 2024) is an effective objective to train the score network
Qu (etr (so (fe, (11:))2,
Pejo (&, | mo)
log so (x,, + +c) di, (2.6)
E
Pepopeo of (mo)
(wo)
Σ
Prjo (xt xa)
[ and K(a) a log a. In particular, where the optimize irrelevant constant C K
the DSE loss in Eq. (2.6) is an upper bound of the negative log-likelihood with an unknown gap. Nevertheless, existing work (Lou et al. 2024 still employs for training and likelihood evaluation.
After training, sampling can be understood as discretizing the following reverse process
_ Paj = Q.
(2.7)
ds
where ds is an infinitesimal negative timestep and the concrete score replaced by the score network. Existing samplers include the Fuler method and Tweedie -leaping, detailed in Appendix D
3

---

Published as a conference paper at ICLR 2025
Multi-dimension In state space of length d like X = (1....,m?) (1...... n,]", we denote the sample sequence of one-dimensional data, i.e.. x² ...... The transition matrix Q: E R xn has an exponential number of possible states, making it expensive to reverse. To alleviate this issue, existing work (Campbell et al. 2022 Lou et al. 2024) assumes independence between dimensions and each dimension one-dimensional diffusion process with the same transition rate matrix Q R=*=.
Under the independent assumption, Q2 assigns zero values (Campbell et al. 2022 Lou et al. 2024) for all sequences with Hamming distance larger than 1. Therefore, it is sufficient to model the concrete score between sequences that differ by a Hamming distance of 1, such as dt = ... {I,.... n - Rdxn is defined
..... given
at st 24. Therefore, the score network so so(;,t)
(z) Pt (a)
se (2t, t)2, se (z) ... 2/. z
(2.8)
which leads to the following expression to estimate the reverse transition rate matrix Q::
a4) = Q (Ht, x/) Pt (a)
Qi (x) 2l 1
(2.9)
pt
inkk ( xi) 80 (a)
[.
(2.
Existing samplers assume that each dimension is independent within a small interval At and update each dimension in parallel for efficiency (Lou et al. 2024 Campbell al. 2022).
2.2 ANY-ORDER AUTOREGRESSIVE MODELS
Any-order autoregressive models (AO-ARMs) (Uria et al. 2014: Hoogeboom al. 2022: Shih ct al. 2022) model the joint distribution autoregressively for all possible orders π of the d variables. Formally, they factorize the joint distribution as I ((==())))==<<)) To learn such a distribution, an AO-ARM utilizes a weight-sharing neural network to model all univariate conditionals and employs mask tokens to represent absent variables. During training. the expected negative lilliiilloood over the uniform distribution of all orders U* is minimized:
Lao( ETTU
log (x,7")1000) 107(500
(2.11)
REPARAMETERIZED ABSORBING DISCRETE DIFFUSION
In Section 3.1 we reveal that the concrete score of absorbing discrete diffusion can be reparameterized as conditional distributions of clean data, which enables efficient sampling by caching the output of time-independent network (see Section 3.2 In Section 3.3 we unify the training objective of absorbing discrete diffusion and AO-ARMs.
3.1 REPARAMETERIZING THE CONCRETE SCORE AS CONDITIONAL DISTRIBUTIONS OF CLEAN DATA
key observation that only the transition from the masked token an unmasked token is valid in the reverse process of an absorbing discrete diffusion. In particular, according to the definition of the transition matrix of the absorbing process (see Eq. (2.4)). have Qabsort (f xx) for any unmasked xi * [M] and of # af. Therefore, the corresponding element in the transition matrix of the reverse process Q (see Eq. 2.5 equals zero. Namely,
Pt (x)
Q
a4) =tt) (xf.
0.
(3.1)
Pc
(x)
for any unmasked state of [M] and ft + z and it is unnecessary to model the corresponding Pe at Bi. Ps (.)...)))...)))))
Also, note that the concrete score always takes the value of one # = Therefore, only need to characterize the concrete score for oe [M] and o M
concrete score
4

---

Published as a conference paper at ICLR 2025
Interestingly, in this case, discover that the concrete score has simple analytic form w.r.t the conditional distributions of clean data, as summarized in the following Theorem
Theorem 1. (Analytic concrete score in absorbing diffusion, proof in Appendix B For
F. ....2 .. and ãt x ...I.. .I..., xt if be expressed as time- inn-inndependent conditional distribution at time zero multiplied by an analytic
[M] and #t + [M], the concrete score time can
time-dependent term:
a(0)
Pr [
Po(alla(
(M
(6)
concrete score
scalar
clean data distribution
where L/M is the vector consists of all unmasked tokens of æt.
One immediate implication of Theorem Dis to theoretically explain the benefit of the "scaling trick" in existing work (Lou al., 2024) (see Appendix C.2 therein), which significantly improves the practical performance of discrete diffusion (see Table([] but has not been fully understood before. In particular, the scaling trick divides the output of the score network by factor. Equivalently, reparameterizes so (Zt, 1) as follows:
e (
so(1,, 1) 1 a 50 [z, t),
(3.2)
where (p(a,, 1) is the output of the reparameterized score network and the scaling factor coincides with the time-dependent term in Theorem [I In the original parameterization, the score network se must model the whole time-dependent concrete score. In contrast, with the scaling trick, the reparameterized score 5g( t) can focus on capturing the clean data distribution Po ( (6') (x( and simplifies learning, according to Theorem [
Further, Theorem D Suggests that the reparameterized score is essentially a conditional probability on clean data, which is sime-independent. Motivated by this, propose reparameterized absorbing discrete diffusion (RADD), which employs a inme-independent network co (2:²) that defines a model distribution qo by corresponding conditional distributions approximate data distribution Po directly:
c,(x;) [i. +1 = (= (MM) = po UM)
(3.3)
In practice, we make minimal modification of the score network in SEDD (Lou et al. 2024) for simplicity and fairness as shown in Fig. [ Specifically, we remove the come-conditioning input, reducing the architecture to form similar to the standard GPT model, with softmax as the final activation. Further details can be found in Appendix [.1
Our reparameterization approach, which removes I, applies to both score and mean parameterizations, as detailed in Appendix [F] This not only simplifies the training target but also enables more efficient sampling process than SEDD (Lou et al. 2024), presented below.
3.2 EFFICIENT SAMPLERS TO REDUCE NFES BY CACHING THE OUTPUT OF RADD
In the reverse process of an absorbing discrete diffusion, once a token transitions from [M] to an unmasked token, it remains unchanged. Consequently, for sequence consisting of d tokens, there will at most intervals during the sampling process where changes occur, regardless of the number of sampling steps D. In the remaining steps, the sequence remains unchanged across all d dimensions. This property allows us to cache co co(x;) to avoid the need to reevaluate the time- tine-independent Co when æ, is unchanged in the previous step (see Appendix [or the pseudo-code). However, since SEDD is conditioned on time, it does not support this caching strategy for reducing NFEs.
The NFES with the caching strategy is random variable. To quantify it, we calculate the expected NFEs (E-NFEs) in analytic form, conditioned on the sampling method, time steps, and noise schedule. For instance, using the Tweedie 7-leaping method with a log-linear noise schedule (Lou et al. 2024 the E-NFEs can be expressed by sampling steps n and generating length / (proof in Appendix D.5
E-NFEs(n) n21 (1 -1-1-(1-1) 1))
(3.4)
A similar analysis was presented in Chen al. 2024) for their discrete-time sampler, although it is hased different assumptions and applicable different scenarios. detailed comparison is provided in Appendix
5

---

Published as a conference paper at ICLR 2025
Predicted Concrete Score
Multiply =(O
Predicted Conditional Distributions on Clean Data
2000
1=
1024
Pu(a" Ia? A, 2 D) po(a" [a² = A, D)
cached (theory)
uneached
1500
cached (experiment)
R
80%
20%
E-NFEE
10%
10%
1000
10%
70%
Softmax
Layer Norm & Linear
0
500
1000
1500
2000
Sampling Steps
Nx Transformer Block
(a) Expected number of function evaluations (E-NFEs) over different sampling steps. E-NFEs measured by Tweedie rlleapngg method with log linear noise schedule.
Embedding Layer
[M]
A
[M]
Input Tokens
SEDD Medium
Newly Added Layer (Parameter Free)
150
RADD Medium (cache)
Simplified Layer (Time Condition Removed)
Generative Perplxxity
Unchanged Layer (No Time Dependency)
120
Figure 1: Reparameterized network architecture vs. SEDD (DiT). Our net- work simplifies the original DiT by re- moving time conditions and outputs the conditional distributions on clean data, similar to the standard Transformer. For example, with a vocabulary of (R, A, D), only the probabilities in the columns cor- responding to the [M] token are mean- ingful (highlighted in color), since the network designed to learn to denoise the masked input. The remaining output, shown in grey, will not be utilized.
100
80
0.5
1.0
2.0
4.0
8.0
16.0
32.0
64.0
Times
(b) Sample quality measured by perplexity (L). Our RADD model, utilizing the cache strategy, achieves slightly better perplexity with less time, enabling f'aster convergence and reduced sampling time.
In Fig. Ta we plot the curve of Eq. 3..4 in blue, which aligns well with our experiments (red stars). This demonstrates that method theoretically reduces E-NFEs, particularly larger sampling steps. This reduction is also supported by Fig. [b where RADD shows faster convergence trends.
Furthermore, based Theorem [] simplified forms of the reverse process for both Euler method and Tweedie - -leaping method can be derived, which leads to corresponding analytic forms of E-NFEs given time steps and noise schedule. We also prove that these two sampling methods are equivalent under a lon-linear noise schedule for absorbing discrete diffusion (see Appendix D for more details).
3.3 UNIFYING ABSORBING DISCRETE DIFFUSION AND ANY-ORDER AUTOREGRESSIVE MODEL
Building upon Theorem]] I we further prove the equivalence between absorbing discrete diffusion and any-order autoregressive models introduced in Section 2.2 presented in the following theorem.

---

Published as a conference paper at ICLR 2025
Theorem 2. The absorbing discrete diffusiin objective of Eq. (2.6) is equivalent any-order autoregressive objective of Eq. (2.11) when the final total noise level o (T) + to0.
The proof of Theorem 2 consists of three key steps, which introduce three different yet equivalent loss functions. Below we briefly present the key ideas and defer the proof in Appendix C
In the first step, by removing the terms se ( 2) and K [aja([s [mo]
(1r (m))) in Eq. 2.6 define
simpler loss LIDEE called I-denoising cross-entropy loss (abbr. 1-DCE), which equivalent to DSE loss. In the multi-dimensional case, it has the form:
1-e-a 20 (0(o ) dt (3.5)
(F)) = S/" Ex aca)
o (()e
e (
Σ
1 1-e- log
We emphasize that Eq. (3.5) holds in a nonparametric setting because RADD can be interpreted as model distribution qo representing the conditional distribution clean data, which approximates the true distribution po. as proven in Theorem
In the second step, inspired by Kingma et al. (2021) ), we change the variable from t to A(t) 1 1-c -- e o(t), which represents the probability of a token being masked in [0, t] during the forward process. Thus, ((,DCE(2to) be rewritten integral of X, defined as X-denoising cross-entropy loss (abbr. X-DCE):
LX.DCE(20) = S -ee (ex
- log qa (aM UM
dA,
(3.6)
where Pa (a: (2o)) is the joint distribution induced by masking each dimension æo independently with probability X.
Finally, we prove that X-DCE loss in Eq. (3.6 can be integrated analytically and rewritten as Cao in Eq. (.IIT. We summarize our proof procedure by the equivalence between these losses:
Appendix
Dcn (20) Appendix
Appendix
CX-DCE(ED)
(=u)
(3.7)
direct benefit from Theorem is that we can use an absorbing discrete diffusion model to sample like AO-ARM and vice versa. For training and likelihood evaluation, the four losses in Eq. (3.7) can also be used (sec Appendix for pseudo-code). To efficiently estimate the four losses using Monte Carlo methods, can replace the sum or integral with expectation. Take Eq. (3.6) for example, can be rewritten as the following form of expectation on A:
(x-DCE(#0) = EE (x= En, (pxx
Σ
- log qo (16(=))
(3.8)
Additionally, Theorem provides a new perspective the DSE loss. While it has been traditionally viewed as an upper bound on the negative likeliood for the diffusion model, it can also be interpreted an expected negative ligllikel over factorial numbers of orderings for AO-ARM by Eq. 11. As discussed in (Uria et al.) 2014), for different orders π, (o(<n; т will be inconsistent in general. Despite this inconsistency. it can be viewed as an ensemble of multiple autoregressive models with different orders, potentially more robust than fixed-order models.
We mention that Hoogeboom al. (2022) establish the equivalence between ARMs and the ELBO of absorbing diffusion models. In comparison, built upon our Theorem [] we extend existing work by unifying four loss functions Eq. (3.7). deepening the understanding of absorbing discrete diffusion.
We provide detailed discussion in Appendix H and systematic empirical study in Section 4.3
4 EXPERIMENTS
We then evaluate the performance of accelerated generation in Section 4.2 and zero-shot perplexity on various language datasets in Section 4.3
We present the experimental setups in Section 4.1
7

---

Published as a conference paper at ICLR 2025
4.1 SETTINGS
Below, we briefly present the experimental settings. For more details, please see Appendix J
We use RADD model Ce reparameterzied as described in Section 3.1 Compared with SEDD models, RADD models have fewer parameters because the time-condition has been eliminated. We trained our RADD model Cy using denoising score entropy, t-denoising cross- entropy, X-denoising cross-entropy and ander autoregressive loss, abbreviated as RADD-DSE, RADD-E-DCE, RADD- X-DCE and RADD-AO, since all these models share the same architecture as described in Fig. D For the SEDD small and medium model, we employed their pre-trained model. When performing text generation tasks, we used RADD-A-DCA DDE medium models.
Model.
Data. Following SEDD, we trained on the Web (Gokaslan & Cohen, 2019) dataset and tested the LAMBADA, WikiTex12, PTB, WikiText103, and One Billion Words datasets (Paperno et al. 2016; Merity et al. 2016, Marcus et al., 1993 Chelba et al.) 2013). For data splits and data processing, adopted the same settings and techniques as SEDD, which involves packing sentences to generate uniform-lengtte blocks as model input.
Training setup. We used a log-linear noise schedule where the expectation of the number of changed tokens at time t is linear with t. Following SEDD, report results for RADD trained over 400K iterations in Tables 1 [ To further analyze the convergence and performance trends, we extended the training of RADD-small to 1,000K iterations, detailed in 7 the appendix.
Metric. Following Lou et al. 2024), we conduct experiments on unconditional generation and language modeling tasks. For language modeling tasks, we report the perplexity calculated on the dataset with different models. For generation, assess sample quality using perplexity (PPL) on unconditional samples measured by an additional larger language model (i.c., GPT-2 large), and sample diversity through unigram entropy (Strudel al. 2022)
4.2 EFFICIENT SAMPLING
As shown in Fig. I of Lou et al. 2024). SEDD surpasses AR in terms of sampling speed. Therefore, we compare the sample quality between SEDD and our RADD model measured by perplexity. As shown in Fig. [b] RADD with the caching strategy is efficient than SEDD. This improvement is expected because the NFEs is limited by the generating sequence length. We further conducted batch size ablation and compare the running time and unigram entropy detailed in Appendix J.4
As discussed in Section 3.3 we can also use RADD as an any-order autoregressive model to generate samples in different orders, as detailed in Appendix J.4 We present more sampling details in
Appendix J.3 and the generated samples in Appendix K.I
4.3 IMPROVED ZERO-SHOT PERPLEXITY ON LANGUAGE MODELING
Following SEDD, we present zero-shot perplexities on the LAMBADA, WikiText2, PTB, Wiki- Text103, and Billion Words datasets (Radford et al., (219a) in Tables and 2 and compare the zero-shot perplexity of our model with other baseline models (Austin et al. 2021 Guirajani & Hashimoto 2023 Lou al. 2024). Perplexities of RADD models calculated based on their corresponding loss (e.g., RADD-X-DCE on LX-DCE), which is valid for likelihood estimation as discussed in Section B.3
Firstly, we conduct an ablation study of the scaling trick in the middle of the Tables and For the absorbing diffusion, the perplexity of the scaled version of SEDD outperforms its unscaled version, which matches our theoretical discovery in Theorem T Secondly, under the same DSE loss and similar parameter counts, we observed that the RADD-DSE model without time-conditioning outperforms the SEDD-Scale model with time-conditioning This ablation validates our analysis in Section 3.1 indicating that time-coniiiooning is unnecessary for absorbing discrete diffusion models. Additionally, while RADD models trained with four equivalent loss functions achieve similar performance, minor discrepancies persist. These differences stem from variations in gradient estimation finite data, leading models to converge at distinct local optima despite the theoretical
8

---

Published as a conference paper at ICLR 2025
Table Zero-shot language modeling perplexity (4) on five datasets using small models. "SEDD- Unscale" and "SEDDD-call" refer to the unscaled and scaled versions of the absorbing models, respectively. All SEDD and RADD models are trained for 400k iterations. Results for other diffusion
models are based on the upper bound from (Austin al. 2021] Gulrajani & Hashimoto/ [023 Lou
al. ) For RADD models, the results are calculated based on the corresponding loss.
Method
LAMBADA
WikiTex12
PTB
WikiText103
IBW
GPT-2
45.04
42.43
138.43
41.60
75.20
D3PM
93.47
77.28
200.82
75.16
138.92
PLAID
57.28
51.80
142.60
50.86
91.12
DnDD--niformm
65.40
50.27
140.12
49.60
101.37
SEDD- Unnscle
52.21
44.75
130.49
43.14
80.70
SEDD-Scale
50.92
41.84
114.24
40.62
79.29
RADD-DSE
49.57
38.83
111.74
37.46
72.35
RADD-1-DCE
50.56
39.02
109.03
36.38
72.60
RADD-X-DCE
51.70
39.98
107.85
37.98
72.99
RADD-AO
50.27
38.26
110.38
35.90
74.28
Table 2: Zero-shot language modeling perplexity (L) on five datasets using medium models. 'SEDD-Unscall and "SEDD-Scale" refer to the unscaled and scaled versions of the absorbing models, respectively. All SEDD and RADD models are trained for 400k iterations.
Method
LAMBADA
WikiText2
PTB
10
IBW
GPT-2
35.66
31.80
123.14
31.39
55.72
SEDD-Unseale
44.60
34.85
93.26
32.97
67.91
SEDD-Scale
42.77
31.04
87.12
29.98
61.19
RADD-DSE
42.30
29.17
75.16
28.03
57.45
RADD-t-DCE
43.24
30.19
78.77
29.36
57.95
RADD-A-DCE
44.10
30.60
82.08
29.29
60.32
RADD-AO 41.96
29.96
79.06
28.51
57.07
equivalence of their objectives on expectation. Overall, RADD losses outperform SEDD on average across the five datasets, validating our analysis in Sections 3.1 and 3.3
RELATED WORK
Continouous-state diffusion models for text generation. Several works have been proposed
apply continuous diffusion to text (Li et al. 2022 Dieleman et al. 2022 Chen et al. 2023 Graves latent space and learn continuous-staid diffusion on it. Bit Diffusion (Chen et al. 2023) learns continuous diffusion model generate binary bits of discrete tokens. However, transforming between these continuous representations and discrete tokens by thresholding may lose information. Bayesian Flow Network (Graves et al. 2024) achieves competitive log-likelihood on character-level langeee modeling tasks and is proven equivalent to continuous stochastic differential equations trained by denoising score matching (Xue et al. 2024). Such models underperform auto-regressive models on standard text generation tasks.
2024). Li et al. (2022) use an embedding layer to map discrete tokens
Discrete-state diffusion models for text generation. Several discrete-state diffusion models have been proposed (Sohl-Dickstein et al. 2015) Hoogeboom et al.] 2021 Austin et al. (021).
D3PM (Austin et al. 2021) proposed a diffusion framework based any probability transition matrix and trained with a lower bound of log- likeliiddoodd DiffusionBERT (He et al. (022) utilizes a
pre-trained BERT (Devlin al. 2019b) as initialization of diffusion. Furthermore, Campbell
CanieepeerrCCampeelllll et al.

---

Published as a conference paper at ICLR 2025
generalizes the framework to continuous time by introducing rate matrix. It is difficult to apply the score matching in such models because the gradient of the data distribution is undefined. Several works try to generalize the score matching discrete data (Lou et al. (2024] Meng et al.)
(2022
2023; Campbell et al. 2022 Sun et al. 2023b) Camphell et al. 2024). Meng et al. (2023) introduce the concrete score and the denoising concrete score matching loss. Furthermore, brDDbridges the
discrete state diffusion and the concrete score by introducing a denoising score entropy loss (Lou ct al. 2024 By incorporating an absorbing process, SEDD achieves competitive performance with the auto-regressive models, especially, GPT-2. Motivated by the success of absorbing discrete diffusion, RADD is specifically designed for this class of models. Due to fundamental differences in score formulations, it can not directly apply to other models like multinomial diffusion. Campbell] et al. (2024) proposed discrete flow models. [Chen et al. (2024) proposed discrete non-Markov diffusion model to accelerate sampling, which has some connections to our cache-based acceleration in Section 3.2 A detailed comparison can be found in Appendix F
Concurrent works We mention that Shi et al. (2024) and Sahoo et al. (2044) independently
conducted related studies on absorbing discrete diffusion. We provide a detailed discussion here.
Shi et al. (024) derived a weighted integral of cross-entropy loss in their (5) similar to our t-DCE loss in Eq. (3.5). Besides, their Proposition 1, which connects the score parameterization and the mean parameterizationf also resembles our Theorem In comparison, we simplified the conditional expectation term (related to !) in Proposition I (Shi et al. (024) to a time-independent conditional probability at time zero. Motivated by the finding, we proposed a simpler parameterization that enables fast sampling. It is worth noting that the hyperparameters they selected significantly contribute to the model's performance, which also applies to our RADD models (see Appendix [.2
for details). In addition, (Shi et al. (2024) proposed generalized masked diffusion model allowing state-dependent masking schedules.
Sahoo et al. (2024) derive the same cross-entropy losses with Shi et al. 2024). Despite lacking theoretical foundation, they conducted time-conditioning ablation which shows that time-conditioning has minimal impact on perplexity. Notably, their method for removing time conditioning retained the same network structure (e.g., keeping adaptive layer normalization) while setting the time input to zero uniformly. In contrast, our approach removes the network structure related time inputs entirely, climinating the need for time input and thereby simplifying the network design. They also proposed caching strategy to accelerate sampling. While this coincides with our work in Section 3.2 we present a complete theoretical analysis of E-NFES to quantify the acceleration efficiency.
Our unique contribution lies in the decomposition of the concrete score and time-independent parameterization, serveing as the foundation for subsequent contributions in Sections 3.2 and 3.3
CONCLUSION
We introduce RADD, a dedicated discrete diffusion model that characterizes the time-independent conditional probabilities, built upon new factorization form of the concrete score. RADD is more efficient by reducing the NFEs with a cache strategy while maintaining comparable performance to strong baselines. Additionally, we demonstrated the unification of training objectives for absorbing discrete diffusion and AO-ARMs. On five zero-shot language modeling benchmarks, our RADD models achieve state-of-the-an performance at the GPT-2 scale.
Limitaition. Our model has been trained and evaluated primarily on the GPT-2 scale. For broader applicability, is essential to explore the effects of scaling on the performance (Hoffmaan al. 2022), which is left as future work. The success of diffusion transformers on images (Bao et al. 2023a Peebles & Xie 2023a Bao et al. 2023b) and videos (Bao et al. 2024) suggests that diffusion models can be scaled up by incorporating transformers.
Another limitation is that our model can only generate full-length outputs, unlike ruto-regeess models that can produce variable-length outputs. This restricts the flexibility of our model in certain applications. We leave the investigation on this issue as future work.
Our conclusions are based on score parameterization but can he extended to mean prediction parameterization (please see Appendix H
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China (No. 98);0118); the
Beijing Nova Program (No. 20220484044); Beijing Natural Science Foundation (No. L247030).
We thank Aaron Lou for his prompt and detailed responses to our inquiries, which greatly assisted research. We also thank Zebin You for his support in setting up the coding environment.
Ethics statement. For the current thcoretical and experimental scope of this paper, we have not found any direct social impacts. However, considering future developments, the paper potentially contributes to the next-generation large language models. In this context, this work could significantly reduce the inference cost of language models but may also lead to hallucinations, amplify biases and discrimination in the data, and pose risks of misuse. As with other generative models, addressing these issues requires further advancements in the field.
Reproducibility statement We have open-sourced our code in https://githus com/ MS-GSAI/ RADD For detailed instructions on environment setup and running scripts, please refer to the README, md file. Comprehensive explanations and proof' of our theoretical claims can be found in Appendices and C
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint MXXi:2303308774 2023.
William J Anderson. Continuous-time Markov chains: An appliccatiionns-rreeetedd aprroch Springer Science & Business Media, 2012.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint ar ArXiv:2305. 10403, 2023.
Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne den Berg. Structured denoising diffusion models in discrete state-spaces. In Advances in Neural Information Processing Systems, 2021.
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-PPM: An analytic estimate of the optimal reverse variance diffusion probabilistic models. International Conference on Learning Representations, 2022.
Fan Bao, Shen Nic, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22669 226799, 2023a.
Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shiliang Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffusion scale. In International Conference on Machine Learning, 2023b.
Fan Bao, Chendong Xiang, Gang Yue, Guande He, Hongzhou Zhu, Kaiwen Zheng, Min Zhao, Shilong Liu, Yaole Wang, and Jun Zhu. Vidu: highly consistent, dynamic and skilled text-to- video generator with diffusion models. arXiv preprint arxiv:2405.04233, 2024.
Lukas Berglund, Meg Tong. Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Lims trained on" is b" fail to learn" b is a". arXiv preprint arXiv:2309. 12288, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Andrew Campbell, Joe Benton, Valentin De Bortoli, Tom Rainforth, George Deligiannidis, and
A. Doucet. A continuous time framework for discrete denoising models. In Advances in Neural Information Processing Systems, 2022.
11

---

Published as a conference paper at ICLR 2025
Andrew Campbell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design, 2024.
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, T. Brants, Phillip Todd Kochn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. In Interspeech, 2013.
Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning, 2023.
Zixiang Chen, Huizhuo Yuan, Yongqian Li, Yiwen Kou, Junkai Zhang, and Quanquan Gu. Fast sampling via discrete non-manne non-markov diffusion models, 2024.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers). Association for Computational Linguistics, 2019a.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). Association for Computational Linguistics, 2019b.
Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H. Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, Curtis Hawthorne, Rémi Leblond, Will Grathwohl, and Jonas Adler. Continuous diffusion for categorical data, 2022.
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http:///kkyLoon07..gttuu 1o/
TpenWebTeettCrpuu 2019.
Alex Graves, Rupesh Kumar Srivastava, Timothy Atkinson, and Faustino Gomez. networks, 2024.
Bayesian flow
Ishaan Gulrajani and Tatsunori Hashimoto. Likelihood based diffusion language models. In Advances in Neural Information Processing Systems, 2023.
Zhengfu He, Tianxiang Sun, Kuanning Wang, Xuanjing Huang, and Xipeng Qiu. Diffusion- bert: Improving generative masked language models with diffusion models. arXiv preprint 1rX1v:2211.15029, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-683 6851, 2020.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and L. Sifre. Training compute-optinal large language models. ArXiv, abs/2203. 155562 2022.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems, 34: 22112224-12666 2021.
Emiel Hoogeboom, Alexey A. Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. In 10th International Conference on Learning Representations, 2022.
Frank Kelly. Reversibility and stochastic networks f.p. kelly. SERBIULA (sistema Librum 2.0), 76. 06 1981. doi: 10.2307/2287860.
12

---

Published as a conference paper at ICLR 2025
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang. and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 21696-217011 Curran Associates, Inc., 2021.
Xiang Lisa Li, John Thickstun, Ishaan Guirajani, Percy Liang, and Tatsunori B. Hashimoto. Diffusion- Im improves controllable text generation, 2022.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution, 2024.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022a.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint 1.X000221..01095 2022b.
Ang Lv, Kaiyi Zhang, Shufang Xie, Quan Tu, Yuhan Chen, Ji-Rong Wen, and Rui Yan. Are we falling in a middle-inteliggncee trap? an analysis and mitigation of the reversal curse. arXiv preprint arXiv:231 2023.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus english: The penn treebank. Comput. Linguistics, 19:313-330, 1993.
Chenlin Meng, Kristy Choi, Jiaming Song, and Stefano Ermon. Concrete score matching: Generalized score matching for discrete data, 2023.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference Learning Representations, 2016.
OpenAL. ChaiGPT: Optimizing Language Models for Dialogue. OpenAl blog, November 2022. URL /ttt::://peeaaa.comm/ooo////////
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. The LAMBADA dataset: Word prediction requiring broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1525-1534, Berlin, Germany, August 2016. Association for Computational Linguistics. URL http: /www aclweb. 1tt.eborr//anthoooo//1166--11
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of 2023a.
the JEEE/CVF International Conference on Computer Vision, pp.
William S. Peebles and Saining Xie. Scalable diffusion models with transformers. In International
Conference on Computer Vision, 2023b.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. OpenAl blog, 2018
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8);9, 2019a.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenA/ blog, 1(8):9, 2019b.
Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models, 2024.
Jiaxin Shi, Kehang Han, Zhe Wang, Amaud Doucet, and Michalis K. Titsias. Simplified and generalized masked diffusion for discrete data, 2024.
13

---

Published as a conference paper at ICLR 2025
Andy Shih, Dorsa Sadigh, and Stefano Ermon. Training and inference on any any-order autoregressive models the right way. In Proceedings of the 31th International Conference on Machine Learning, 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- lional Conference on Learning Representations, 2021a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference Learning Representations, 2021b.
Robin Strudel. Corentin Tallec, Florent Altché, Yilun Du Yaroslav Ganin, Arthur Menseh, Will
Grathwohl, Nikolay Savinov, Sander Dieleman, Laurent Sifre, et al. Self-conditioned embedding diffusion for text generation. arXiv preprint (rXiv:221.0042.60 2022.
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 2021.
Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based continuous time
discrete diffusion models. In The Eleventh International Conference on Learning Representations, 2023a.
Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based based continuous time discrete diffusion models, 2023b.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
Benigno Uria, Iain Murray, and Hugo Larochelle. A deep and tractable density estimator. In Proceedings of the 31th International Conference on Machine Learning, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017a.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkorcit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Neural Information Processing Systems, 2017b.
Kaiwen Xue, Yuhao Zhou, Shen Nie, Xu Min, Xiaolu Zhang, Jun Zhou, and Chongxuan Li. Unifying bayesian flow networks and diffusion models through stochastic differential equations, 2024.
Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. In International Conference on Learning Representations, 2023.
Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming- Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling, 2024.
//aass:////ivii/ http://wwww...rg//////////...0 02908
14

---

Published as a conference paper at ICLR 2025
CONTENTS
Introduction
2
Background
Continuous time discrete diffusion model
2.2
Any-order autoregressive models
Reparameterized absorbing discrete diffusion
3.1
Reparameterizing the concrete score as conditional distributions of clean data
3.2
Efficient samplers to reduce NFEs by caching the output RADD
3.3
Unifying absorbing discrete diffusion and any-order autoregressive model
4
Experiments
Settings
4.2
Efficient sampling
8
Improved zero-shot perplexity on language modeling
5
Related work
Conclusion
10
Detailed notations and definitions
16
Proof of Theoreme
17
Proof of Theorem
19
C.I
Equivalence between DSE loss and t-DCE loss
20
Equivalence between 1-DCE loss and lambda- DCE loss
21
Equivalence between lambda- loss and any-order autoregressive loss
22
Sampling methods
23
D.I
Exact reverse distribution in absorbing discrete diffusion
23
D.2
Tweedie r-leaping method and its simplified form in RADD
24
D.3
Euler method and its simplified form in RADD
24
D.4
Equivalence Tweedie r-leaping and Euler method under log- linear noise schedule
25
D.5 Discuss on the expectation of NFEs
25
E
Discussion for mean parameterization and RADD
27
Comparison with Chen et al. 2024)
27
Details of A0-ARMs
28
15

---

Published as a conference paper at ICLR 2025
H Comparison to prior works concerning equivalence discussion
29
Algorithms for training and sampling
29
Experimental details
Model details
31
0.2
Training details
J.3
Unconditional generation details
32
J.4
Further evaluation of generative perplexity
32
J
Further evaluation of zero-shot perplexity
34
Additional experimental results
34
K.
Additional samples
34
A DETAILED NOTATIONS AND DEFINITIONS
We introduce the notations used throughout the paper. Let lower, boldface lower and upper case letters represent scalers (e.g., a), vectors (c.g., a), and matrices (e.g., A), respectively. For vector a. a' denotes its i-th element. For a matrix A, Jii denotes (i, --th element. For vector function f. f (æ); denotes the i-th element of f fz)). Constants and random variables are not distinguished the notation if there is no confusion. We represent the distributions of the forward and reverse processes by p and do respectively. The transition probability from time s to time denoted by (-),)-)),, and the probability at time is denoted by (t).). Complete notations and definitions are listed below:
I, f Scalar variables representing states in a model.
X: A one-dimensional sample space {1,-.. ,N].
Q The transition rate matrix at time t.
p. The probability of the forward process defined by the transition rate matrix Q.-
qo: The probability of the reverse process defined by model Cg.
(+(z)) The transition probability from state to state å from time to time t.
Pt():: The probability of x at time
Paje: The transition probability matrix from time to time f.
a (t): The noise schedule function.
Q The reverse transition rate matrix time f.
89( a); The corresponding element of s*(a,, (), which approximates Pr(0.) pe(m)
M]: A special mask token in the absorbing process.
x A multi-dimensional sample space {1..... ,
zy. A multi-dimeesional vector.
z/: The i-th element x;-
(-1x)) The probability on dimension i from time time conditioned on full vector Ez.
pe (-1-): The transition probability from time s to time t under Tweedie r-leaping method.
pauler (-I-): The transition probability from time to time under the Euler method.
Qiok:
Transition rate matrix for each dimension of a.
UM. vector consists of all unmasked tokens of æ.
16

---

Published as a conference paper at ICLR 2025
z The elements of z with indices ranging from to b.
cx,): network that characterizes the time-independent conditional probabilities in reparameterized absorbing discrete diffusion (RADD).
d: Total sequence length or dimension of æ.
t: Generating sequence length.
one permutation, rT() denotes the 1-th element of permutation τ, a(< 1) denotes the elements of permutation и with indices less than I.
U-)): Uniform distribution.
Px(+(ar)): The joint distribution induced by masking each dimension in zo independently with a probability A.
Cat: Categorical distribution.
NFEs: Number of function evaluations.
E-NFEs: Expected number of function evaluations.
B PROOF OF THEOREM
In this section, we provide a detailed proof of Theorem which carried out in three key steps. The core idea of the proof involves leveraging the properties of a continuous-time Markov chain with an absorbing state, where the forward diffusion process is independent across different dimensions. This independence simplifies the analysis of both the conditional and joint distributions.
First, we derive the analytic form of the conditional distribution, as stated in Lemma This can be derived directly from Eq. (2.3). but for better understanding, we provide a more intuitive proof for Q, ot) ahsoorh Second, we extend this analysis to multiple dimensions to obtain the joint Finally, by simply dividing the joint distributions derived
distribution, as formalized Proposition [ in the second step, we decouple the concrete score, thereby completing the proof of Theorem [
Lemma 1. (Analytic conditional distribution in absorbing diffusion) Suppose {Xt} is a continuous time Markov chain with transition rate matrix Q1 = a (f) (t)Quhuorb, given the value zo at time zero, the conditional distribution exe (wo)
has the following analytic form:
c (())
Ft FÜ.
(xjo (2)
1 1-e- ((),
M
(B.1)
0,
F M and xt To.
Proof. Given the initial value vn E X' = (I,...,N NN, we have
Do. < Th. MM, t 2 Tx-
3t
(B.2)
Here, Tp represents the holding time before ao transitions to the absorbing state [M].
Based on the definition the Q Eq. 2.2 and aabsorb the probability of xo remaining the same after small time increment At is
(t) Quboorb (No. at + o(At).
(B.3)
Partitioning the interval [0, f] into {s Hi-o and utilizing the memoryless property of continuous time Markov chains, express the probability of vo remaining the same from time to f product
17

---

Published as a conference paper at ICLR 2025
of probabilities over these small intervals. This gives us:
Peu Poulee- (sro/rr) k=1
II
(B.4)
(1 + (th Quhunott bboort 20) 8k-1) + o(3; 5k-1))
(B.5)
k=1
exp Σ In (1 (tt-
(x0 sk-1) oss sk-1))
z0)(s
(B.6)
exp Σ allx-1)Q Qu
abboo (n) 8k-1) + o(sx 8k 56-1))
(B.7)
Let max(sk - 11-1) - 0. the Riemann sum in Eq. (B.7 equals the following continuous integral:
PHo(Folz0) exp (%) a(s) absort (20, wo)ds exp (Qubsorb (x0.
(B.8)
By Eq. 2.4 Q heooth (Io. xo) = 11 we have
Pzoo = P(T, > t) = (())
(B.9)
Pe (M) 200 P(T, S = = c
(B.10)
Prjo (A|
if + M and k: + 50-
(B.11)
Similarly, given value va time < the conditional distribution can be expressed as (6(())) ((a)
It Iz.
(1 1 e (0(1)))())) 0,
Jt [M}
(B.12)
T. + M and Dt + Ze.
Proposition (Analytic joint distribution absorbing diffusion)
Suppose {Xt) is continuous time Markov chain with transition rate matrix Q: = at) Qubsnn For a4 = xt ..... .4 with di components as [M] and d2 d d1 components as unmasked tokens, Pe(x)) can be expressed
(e(xx) = - (e- ootiodddpppppoo/ po(a UM
(B.13)
where Iii is the vector consists of all unmasked tokens of..
Proposition| D shows that the joint distribution Pt (xx)) can be expressed as the multiplication of two terms. One is an analytic term only depending on time, the other is d2 dimensions joint distribution of clean data po (Mr independent of time.
Proof. Without loss of generality, let's assume that the preceding di terms of x are all [M], and the remaining d2 terms are unmasked tokens. That is, 24 = [M] ... .M a unmasked token in X.
di +1 of and here x is an
Using the law of total probability and Lemma I along with the assumption of independence between different dimensions of the diffusion process, can express the joint distribution Pc[MM MM]22+1 xx4) as sum over all possible initial states xo X
Pc ... Ma 25)
Σ
[M
MM
x o (o)))
m.
Σ
P ... [M]ad di a af 40po ... x
xf adex
Σ
II
pho (MM
II
10 ..26))
xe ... a k=1
k=d. +1
18

---

Published as a conference paper at ICLR 2025
Substituting the analytic forms of Pijo (M)) and prja (of (ag) from Lemma be further simplified as follows:
above equations
d1 Il
d II
Σ
Pojo ( [M]
Pājo (ar ...2)))
=fEx, aagex k=1
+=d++1
Σ
II Poo (ett))-00000))))))) ..0...ppp
... ...20 0 x2 ditl
xx)
ofEd 200 Ex
- Σ
[1 10--0 (100 ... ..20 x0002
d1-1 .............................. ...22)) 24
= Ex
=|1 e (1 (e- (11-
E
Po(a0
20 2, 2
MEx, exx = - - po( x4).
By noting that po Po( d1 x4) (= in the general case, we have
Pe (x.) = e ( p0)
which demonstrates that the likelihood of the noisy data a, at time equals the likelihood of the unmasked part az!h UM at time O multiplied by an analytic time-dependent term.
L
Theorem (Analytic concrete score in absorbing diffusson,, proof Appendix B For æt o) .......... ot and iz xt ...I....... .x....4 Lt, af = [M] and at & [M], the concrete score at time can be expressed as iimeinndeenddet conditional distribution at time zero multiplied by an analytic
time- dependent term:
Pt. (x)
e a(((
Po (0 UM
Dr (x)
e
concrele score
sealar
clean data distribution
where L/M is the vector consists of all unmasked tokens of xx.
Proof. According to Proposition
if x
M and # *
MM. LUM (a .tf).
Pe ((
-000 po
Pr (xx
{=
(1 [ (2 2t)
[| -l-
UM)
e -a()
e-o(1) Po (
C PROOF OF THEOREM
Theorem 2. The absorbing discrete diffusion objective of Eq. (2.6) is equivalent to any-order autoregressive objective of Eq. (2.11) when the final total noise level (T) +o0.
Here, the infinity final total noise level guarantees that all tokens will be finally masked with probability one (1 (T)).. Below present the detailed proof in three steps.
19

---

Published as a conference paper at ICLR 2025
C.I EQUIVALENCE BETWEEN DSE LOSS AND T-DCE LOSS
For given noisy input 2t, as established in Section 3.1 it is valid only when it contains exactly one more unmasked token than æt. In this case, the transition probability Q, (0,, xz) equals (t).
Replace s0() with 1-e-a(s) co (2te, we can express the DSE loss in the multi multi-dimensional case as follows:
e-
(x00) = =S0 P
Σ o(1) (
e 5
Beeepipo(ee Ro)
-- (i---a -(() Co co(ar)
e
e-))( (1)) Co (2te ( K (1 e
e e
( (ubb
= di
T-e-o(1) e-o(() (1x = log
- (
T Eexerpe (00r)(2)))
a(t) ( (1---00) co (0 co(22)16.31)
dt
8i ai=
T
o (1)e (1)e =(1)
a(r) 1a log 000 a- co (o-
+
En (mpottlt)))
dt
4 [M
TT
(T-e-ot)) 1166 -1) dt.
E-
+
Σ
(t)K
0
We analyze each term in the above equation separately. The first term simplifies due to the property ΣjAm Co (rx (i))) =
J. E (mo)
a(t) e-
Σ M]
dt.
(C.1)
1 - o0(()
The third term can be simplified by substituting ((a) = a log and using 0 llog 0- = 0:
-
- -0(( 5(1) -1) :) dt. (C.2)
e a((
Σ
a(1) 1- 0
log
Combining the first and third terms:
E (
e -((
( ) dt
Σ at))
log
(C.3)
''" d(1 - -e-000)
TT
((1)
e o((
log
dt
(C.4)
1 (()
1 - c c-
=d / ((())-0())
T
c a(t))
x(t) log
x(t) dl.
(C.5)
Introducing a new variable A(t) = 1- R1), which represents the probability of a token being masked from 0 to t in the forward process. As (t) fe a())dr and o(T) = co, we have A(0) = 0. X(T) 1 and dA (+))) dt. Obviously, A(t) is invertible, which allows us perform a change of variables from f to X and simplifies Eq. C.5 to
d % log
1- dx -d (Alog X + (1 A) log(11 A)) 11 = 0.
(C.6)
20

---

Published as a conference paper at ICLR 2025
Here used lim x=0
Aogg
lim (1 - log(1 A) = 0. X-1
Thus, the DSE loss reduces to the second term, which we define the t-denoising cross-entropy loss (t-DCE):
Ci-DcE(20)) (200 - 1/¹ Eee, (eeepto(t
ot)e a(t)
5(1) co co(2x co(xx))li.3]
- e-0( 1a - j) log
di
k ((0)
M
atT) ((()) ((t)
e(t)
Rig, Re P
Σ M
e-((() log
1 - o(1) Co (mx (4, xb0
dl
e o(1) 10 (16 (26 loo (oo()M)) ) dl.
at)) (())
e-)(t)
Σ
log
1- --(()
M
C.2 EQUIVALENCE BETWEEN T-DCE LOSS AND LAMBDA-DOG LOSS
Starting from the (-DCE loss in Eq. 3.5
perform change of variable from / to X(1)
1 e et)( as demonstrated in Appendix C.1 terms of X:
This allows us rewrite the t-DCE loss integral in
1 X
log (1 1-
90 (arg (emelaa(M) da
Baxmos(ax)oo)
!/ Ā Eng Bax
1
log go go (xgllc LMM
dd
M
1 x
J. X Bex-px(xxlao
log
dA
A
[M
Given the independence of the forward process and Lemma the original probability Pajo (z, (xo) can be factoiized II== Pijo (af (zf). where
x M
(z)())))))
e()
x xij
(C.7)
0,
else.
Therefore, the induced probability px(2x)100)
IT p$ ( (zf6) where
A
xx = o
M
p =
(x
X,
(C.8)
0,
else.
Next, consider the second term. Similar to Eq.
can prove that it equals zero:
L Ees
log(
0.
(C.9)
Therefore, t-DCE loss is equivalent to the first term, defined as X-denoising cross-entropy (A-DCE):
(IDCE( ( =/00 Eex
Σ
- log (z UMM
dA.
(C.10)
M
21

---

Published as a conference paper at ICLR 2025
C.3 EQUIVALENCE BETWEEN LAMBDA-DCE LOSS AND ANY-ORDER AUTOREGRESSIV LOSS
Based on A-DCE loss in Eq. (36) we first define the sample space and analytically express the expectation term.
Given vo. define the sample space of æ (x)0)) {=) [M] x x.... {ag [M) and Xx (za)) {ø i ((xn) ã has exact k dimensions with values [M] It follows that (aro) 24 and
[xn)) (i)) Therefore, the sample space (x can be decoupled by the number of masked
tokens k in ac:
log go (z (ex da
(C.11)
Σ px(
Σ
log qo (zg (oM)) dd
(C.12)
(E)(()))
M
L Σ
1²(1 - 2)
log 2e (0 xUM)
dd
(C.13)
k= (e
2 M
-1 A)a=
log go ( (z
dA.
(C.14)
k= (
2= M
The last equation holds because there are no masked tokens when = O, and the inner sum is zero. From Eq. (C.14), by rearranging the order of summation and integration, analytically evaluate the integral Jo x dd using the Beta function, which eliminates A:
Eq. C.14
x
dd
Σ
Σ
log qo (z )
(C.15)
AE (mo)
M
1)1(d k)
Σ
log go (2 UM)
(C.16)
d!
(20)
Σ
- log qo (2g
(C.17)
X
M
Eq. C.17 follows:
be reformulated in terms of an expectation uniform distribution U( (mx)) as
E (aU(())
Σ 3
(=log(ue(a))aa
(C.18)
Let be one permutation of the integers 1.... d, and U~ represent the uniform distribution of all orders. We note that Eq. (C.18) is equivalent the following term from the perspective of any-order autoregressive model:
d
- log go (2 ((1) aa (<dd .T)
B
(C.19)
1=1
r=d k+1
T((() Here, 20 denotes the sequence of the first I 1 elements in the permutation и. Given a fixed x =(<ddk+1) can be interpreted the unmasked part of the noisy data GUM, For
k, the term x0
22

---

Published as a conference paper at ICLR 2025
(r) corresponds the k items of the masked part. Since both and ă are both d- 1, d, Zig uniformly sampled, Eq. (C.T8) and Eq. (C.19) are equivalent.
Further, make simple subscription transformation by letting = d - k and change the summation to Monte Carlo estimation on Eq. (C.19)
d E1,U(1, d) d 1+1
B
log (1 Fo
((1)
=(()); ; ;T).
(C.20)
In (Uria et al. 2014; Hoogeboom et al. 2022). , it was proved that Eq. C220 is mathematically
equivalent to Eq. 11.11. p. Actually, Eq. (C.20) is widely used training objective for any-order
autoregressive models for efficient parallel optimization.
This concludes our proof of Theorem 2
D SAMPLING METHODS
In this section, we first derived the exact reverse distribution for absorbing discrete diffusion in Appendix D.I. This derivation led simplified forms of the Tweedie r-leaping and Euler methods, detailed in Appendix D.2 and Appendix D.3] respectively. In Appendix D.4, we proved the equiva- lence of these two methods under lon-linear noise schedule. Finally, in Appendix the expected number of function evaluations (E-NFFEs for these methods.
D.3] D.5
we discussed
D.1 EXACT REVERSE DISTRIBUTION IN ABSORBING DISCRETE DIFFUSION
Lemma 2. (Analytic reverse distribution absorbing diffusion) Suppose (Xi} is continuous time
Markov chain with transition rate mutrix Q, a Qal)Qabsorb For D. xt ...x with di masked
tokens and d2 di unmasked tokens, and Rs x² x² with di Ad masked tokens and d2 Ad unmasked tokens, the reverse distribution is given by:
-00 ---
1-c 1-e-
pa(m Pa (ar huss
We CUM Vs.
T-e-o
(D.1)
De ZUM Vsi
where CUM Is denotes Vi :x + [M] we have x; = x
Proof. Using Bayes' theorem, (=)1(2,1111) Peta (xt (x,) po(
(mr))
From Proposition
pr(xx;) = [1 - 10----(0))) c-0(0)d4 p(t))ds px( U
(D.2)
(x())) [1 E --o(s)) -dde - (())) da Ad (2:
(D.3)
Utilizing Eq. (2.1)). we get
[e-(0(()- (1())(((()-00)))
CUM B.y (D.4)
phjs (xf (xt
0e ZUM a..
1=1
Simplifying these equations, we can express Paje (x: (xxe) as
(+) I-e
-0))
(x)
po (ar UM Po bart
0e CUM B.. æe ZUM W..
Ps Pai1(a0(231))
(D.5)
should be noted that when Dy SUM Øs, the ratio Po(a:
po(a!Mi can be reformulated as a dj dimeessionn
with Nds states. This is not accessible using our one- dimensional conditional distribution Po [z. in Theorem I if d1 >> 1. Therefore, for efficiency, existing samplers assume that each dimension is independent within a small interval and update each
conditional distribution Pa
dimension in parallel (Lou al. 2024 Campbell et al. 2022
23

---

Published as a conference paper at ICLR 2025
D.2
TWEEDIE LEAAING METHOD AND ITS SIMPLIFIED FORM IN RADD
Given the vector Ft. if we sample each x² independently, the factorization of marginal distribution plweodie results in the minimum KL divergence with true reverse Peta (eje((./(ar))) (proof in (Lou et al. Pale
2024
Appendix A). This assumption formally defines Pajt Papeadic as follows:
d (cs) II Pajo (x)]))))
tweedie Pole
(x. 22)) = II
Pale
twediie
(D.6)
To sample from Psjt Pa we need to derive the analytic form of p (1t[a,))))) Without loss of generality, let's assume that the preceding di terms of are all [M], and the remaining d2 terms are unmasked tokens.
For illustration, we can take i = as example. Let Xs denote the sample space of length d1 1 sequence where each sequence has exact k masked tokens, with [&& C Ndd-- When r * [M], According to Lemma
p/ (x,) = Σ (2.)(2.)111)
di-1
e (()) (1)
k+1
e (s)
di
po po x 2.d xx adi+lid
1- (s)
1 (()
po (x (1 +1:dd
2:2dd 2x
d1
(x) (x))
a(() k+1
[())
di
po(x 1 +1id
d1
e- (+)
e (1)
po (x² +1:d)
K=U
(()
a(1)
(s)
d1
((5)
di
po di +1:d)
az
1- e (8)
1- (0((
(
Po(a dodd
(()) -0
po (o( at d1
1 - e()(1)
po (idd)
( - -a(
Po ( at +id).
1 1- (())
Here, we used the binomial expansion identity:
T a(s) - 7(1)*
1 e - c x(t) d:
1 (()
(
Similarly, for x!! M
1- c0()
(M][
(D.7)
-0(())
In general, we have
8(v) 60))
1-2-000 -0 Po UM)
x + M]. x [
1 -b()
x² M. . = [M]
(D.8)
T-c-ot
o * MM.
With trained cp. we can use co (2::) (i, ar] to approximate the true conditional distribution po( (M(M) and sample by Eq. D.8).
D.3 EULER METHOD AND ITS SIMPLIFIED FORM IN RADD
According to theory of CTMC Kelly 1981 Campbell et al. 2022 Lou et al. 2024). given a particular one-dimensional input Tt. the transition probabilities to x, be approximately calculated using
24

---

Published as a conference paper at ICLR 2025
Eq. 2..1 and Eq. 2.7 follows:
Paje(aa(11)) ŌxgR, (rt,, 2.) 8) + olt - 5),
(D.9)
Fs ōagn, + Qe(Te, vt) - 8),
(D.10)
where
Qr(Tes (i) pe(w))
Tt + Usi
((())))
(D.11)
-E Qe k),
St = Dss
Therefore, we can define the Euler upproximation of the transition probability (Campbell et al. Lou al. 2024):
2022
palee (2g) òz.æ, + Qe(Tee [t)(t -
(D.12)
For multi-dimensionaa case, we factorize palt
(ar, (24) as I[ ouler. (at), where Pale based on Eq. (D.12) which use æ, to replace Ia and x x
Pale
culer.
(1r) (22))
... 24 22 to replace Iz.
In the case of absorbing difflusio, similar to Tweedie-r leaping method in Appendix D.2 Theorem and Eq. (2.4) to simplify Eq. (D.12), which results in
we can use
(t))) e -oo (t s)po(r1|< (t), -o-- (t - 8),
if x + M
2 =
M
culer, (+ (2))
if x M x M
(D.13)
z + M
In practice, we also use co (g(ar)
(1, af] to approximate the true conditional distribution po p=
when sampling from Eq. DD.13.
D.4 EQUIVALENCE OF TWEEDIE LEEAPIN AND EULER METHOD UNDER LOG-LINEAR NOISE SCHEDULE
By comparing Eq. (D.) and Eq. (D133) we observe that both the Tweedie r-leaping and Fuler methods can be interpreted similarly:
If at is an unmasked token, keep it unchanged, i.e., m = at-
If zf is masked token, first determine whether it will be unmasked with a probability st, It is to be unmasked, then sample x² from [x
The only difference lies the analytic form (t, s). For the two methods, according to Eqs. D.8 and (D.13), their corresponding (t, s) are given follows:
tweedee (t, (s) ((())
(D.14)
(0(())
e ( sheuter (t, oo())
( e (t
s).
(D.15)
In general cases, these expressions not equivalent. However, if choose log-linear noise schedule 5 (t) = 1 1111-- log (1 - (1 e)t). both Eq. (D155 and Eq. D.14) can be simplified to the same
form (t, s) as follows:
(tt, 5)
(D.16)
which shows that these two sampling methods are equivalent under log-linear noise schedule.
D.5 DISCUSS ON THE EXPECTATION OF NFES
In this part, show that given the noise schedule or (t) and set of time steps {to 0.... tt T}. the NFEs can be treated as a random variable with a calculable expected value for both Euler method and Tweedie 7-leaping method.
25

---

Published as a conference paper at ICLR 2025
Let denote the length of the generated sequence and Nr E {0, (0..... 1)) denote the number of dimensions that æ changed in [tx-1. tx). Without loss of generality, we first consider the unconditional generation where = d. The NFEs, E-NFEs, and Ng. can be expressed as
NFEs(n)= Σ 1=1
T(Na #0),
(D.17)
n B + +0] = E
a
E-NFEs(n1
P(N: #0),
(D.18)
k=1
k=1
N₂ = Σ
T # M = [M])-
(D.19)
1=1
Furthermore, we note that the d dimensions are independent. According to Eqs. (D.)) and (.13).
the probability Pije ([M] depends only on time and zi while independent of the other dimensions
of xt. Thus, Paje [M,] Pije( (MMj2)))) Therefore, whether token changes from masked to unmasked is independent across the d dimension
(I(x) [M). 1(x [M)][(() [M), ... ,,(x, = [M))
(D.20)
II p (I(=) [M)][({] [M)).
(D.21)
Since øT consists entirely of masked tokens with probability one, each dimension of I( [M], [M]) is independent. Consequently, N₂ follows a binomial distribution with parameters
and Tk. denoted as N₂ Binomial(d, rk), where тк p(=², * MM. [M) represents
the probability that x² changes within the interval [ tx) in each dimension. Therefore, we can further simplify Eq. D.I8)
E-NFES(n) = Σ P(N #0) = Σ (1 rx)
(D.22)
k=1
k=1
By definition of ra and the property of absorbing diffusion:
Th: = pp( * [M] [M]
(D.23)
-pax + MM = [M) II
plat,
[MIlaf, = [M)p(a), = [M]
(D.24)
j=k+1
-(1 (1-p(en p(at (1-p(aat
M
(M))
II j=b+1
p(at [M MM)-
(D.25)
Eq. (D.25) can be determined given the sampling method and noise schedule.
For Tweedie -leaping, based on Eq. (D.).
derive that:
p(at,
[M] [M) = 1 1-e-5(1)
(D.26)
Therefore, express F%k as
(tk c (tx)
II (1 1 (0
c -0 )
((()
Tk = 1 - 1-e-a((t)
1 - -- a((y))
-
e ((t,)
(D.27)
j=k+1
The independence applies to whether a token changes from masked to unmasked. However, the specific unmasked token masked token changes to depends other dimensions.
26

---

Published as a conference paper at ICLR 2025
For the Euler method, based on Eq. (D.33) derive that:
e-
p( = [M] 1 M o(t)) 1 1-e-0(1) [(tz) (1) - 1j-1).
(D.28)
E ((ta) Th (o((k)
o(ty))
(0((() (1e
II (1 -1-olo o(4) j=k+1
1-e-o(1,) 1- (L) (--1)))
(D.29)
For conditional generation cases where the generating sequence length is less than the dimension similar results hold. The only difference is that N₂ ~ Binomial(I, and Eq. (D.22) changes to
E-NFEs(n) (1- rx)))
(D.30)
Specifically, if we adopt a log-linear noise schedule and let 4 = a. according to Appendix D.4 the Euler method and Tweedie r-leaping method are equivalent. In this case, Eq. (D.27) simplifies to 1 Substituting this result into Eq. (D.22). we obtain
"
(1- n(1 (000=nn1-(1--1111 11
E-NFEs(n)
(D.31)
E DISCUSSION FOR MEAN PARAMETERIZATIO AND RADD
Equivalence of modeling Analogous to the To prediction in continuous state diffusion models, Austin et al. 2022 and Campbell et al. (2022) used the mean parameterization ( t) to learn the the reverse density Poje (2) [z))),), 1.... d. According to the analytic form of reverse distribution in
Eq. (D8 letting 0, we have:
po(aglaxMM)
26 # M art
M M
Poj. (xj(a))))))
0,
2 MM a)
(E.I)
2 * MM].
This shows that the mean prediction is equivalent learning conditional distributions clean data. In conjunction with our discussion in [.]] the mean parameterization should be time- reparameterized ca()). Empirical results
independent, denoted yao (x;,), and is equivalent like He et (2022) and [Sahoo et al. (2024), which demonstrate that the time-independent model pe (xx) performs well, also validate our theory.
Equivalence of training objectives Shi et al. 2024) proved that the training loss for score parame- terization (i.e., DSE loss) and mean parametenzationn (i.e., negative ELBO loss) are equivalent.
Equivalence of sampling methods Comparing Appendix D.2 with Shi et al. (2024); Sahoo ct al. (2024), is evident the Tweedie r-leaping method for score parameterization equivalent to the sampling method for mean prediction as follows:
(a)( 1-as
It + M
go (z,, = p(z,1=,, = =({x)))
(E.2)
Cat(r,; i-a;
e + ou ((x)))
M
Here, represents the probability of a token remaining unmasked at time ł, which equals a(() for score parameterization. Therefore, Eq. (E.2) and Eq. (D.8) is equivalent.
F COMPARISON WITH CHEN ET AL. (2024)
Chen et al. 2024 proposes sampling methods for discrete-ting and continuous time diffusion models individually, Consider sequence of length d:
27

---

Published as a conference paper at ICLR 2025
Discrete-time models For models trained over discrete timesteps Tuain {1..... T). Chen et al. (2024) proposes to pre-sample the time points Tę E Tuain, where each T₁ corresponds to a change timepoint for specific dimension of æ. These timepoints form time set Tchange C Tirain and updates are only applied at these steps. For absorbing diffusion models, as each token changes only once, NFEs
Tchangel S min(d,
Continuous-time models For models trained over continuous time, d change points т; pre-sampled and sorted in ascending order (Tn, <... < TRa). Updates are sequentially applied at these points, resulting in NFEs d, which resembles the sampling process in AO-ARM. However, how to reduce the NEFs to less than d for the aatinuous-time model has not been investigated.
In contrast to Chen et al. (2024), which pre-sample specific time points and update tokens only at those predetermined points, RADD leverages a time-independent parameterization that updates tokens only when they change. This fundamental difference results in different applicable scenarios for the two methods:
Chen et al. (2024) applies to both absorbing and multinomial diffusion models. However, for continuous- time models like SEDD, their sampling method results in NFEs and, noted, how to reduce the NEFs less than for the continuous-time titeuusstttmeeee model has not been investigated.
RADD, on the other hand, is specifically designed for absorbing diffusion models. is straight- forward apply the cache strategy of the RADD continuous-time settings and reduce NEFs to less than d because the input of the model is independent of the time.
Although the samplers in our paper and in Chen et al. (2024) originate from different formulations, the results of Theorem D. (Chen et al. 2024) for the discrete-time sampler align with those of our Eq. 3.4). However, as discussed above, the two samplers are applicable in different scenarios.
G DETAILS OF AO-ARMS
Any-order autoregressive models (AO-ARMs) (Uia et al. 2014 Hoogeboom et al., 2022 Shih al. 2022) model the joint distribution autoregressively for d! different orders π of the d variables. Formally, the joint distribution is factorized as IT ((x)(k) (==(()))) by chain rule. Therefore,
AO-ARMs actually define Ei=o C/ (d k) d2 distinct univariate conditionals probabilities.
Architecture AO-ARMs model all univariate conditionals via weight-sharing neural network. by using the [M] token for variables that are not present in the condition se [ For efficient parallel optimization, the architecture is designed such that given the condition set of size k. it predict all - k univariate conditionals at once, similar to the output of conditional distributions in Fig. T
Training AO-ARMs are trained to minimize the negative joint likelihood of datapoint æpp under the expectation over the uniform distribution U- of orders. It can be simplified by treating / random variable with uniform distribution over cardinalities to d. Further, it can be transformed into a form for better parallel optimization:
LAO(#o) = EnCU
Σ
log (z (<() (<000
(G.1)
1=1
Enauy d E d) - log go (I x1))
[<()))
:T)
(G.2)
d E,.U d)
1+1 E USa)
log (z (()) x<1),
120 (m).
(G.3)
Training pseudocode of Eq. G.3 be referenced in Algorithm
*Condition corresponds variables in (<ck).
28

---

Published as a conference paper at ICLR 2025
Sampling The sampling process for AO-ARMs generates data points autoregressively based randomly sampled order и. Starting from an empty sequence initialized with [M] tokens, the model iteratively predicts the next variable based on the current condition set <=(<<)) Since the order a is chosen randomly, AO-ARMs support generating sequences with any desired ordering, aligning
with their ability to model d! orderings during training. Sampling pseudocode can be referenced in Algorithm
H COMPARISON TO PRIOR WORKS CONCERNING EQUIVALENCE DISCUSSION
Austin al. 2021 and Hoogeboom et 2022 have both discussed the relationship between absorbing discrete diffusion and AO-ARMs. Below, we provide a detailed comparison of their approaches with ours.
Austin et al. (202T) made an early attempt to explore the connection between absorbing discrete diffusion and AO-ARMs. However, their work lacks rigorous proof. Instead, they qualita- tively discuss the correlation between the two loss functions. Notably, in Appendix A.3 of Austin et al. (2021), they describe the relationship by stating, "this looks very similar it is not exactly identical."
In contrast, our work rigorously establishes this connection. By leveraging the continuous- time framework and the time-independent parameterization presented in Theorem [ we provide formal proof demonstrating the equivalence between absorbing discrete diffusion and AO-ARM.
Hoogeboom (2022) establishes the equivalence between ARMs and the ELBO of the absorbing diffusion models directly. In comparison, our approach follows different path:
1. the ELBO was first reduced to DiS (mo) as Eq. (.66). as discussed in detail in Lou et al. (024))
2. Using step-by-stey substitutions via Eq. 3.7 we further reduce ( to Lxo.
Therefore, our approach offers unique contributions by:
rigorous and alternative proof We leverage the time-independent properties of the reparame- terization formulation, providing an alternative and rigorous proof of this equivalence.
Equivalence of four losses Our analysis extends to demonstrate the equivalence of four distinct losses, including CT (xoo) and (xo(#)) in Eq. (3.7). offering deeper understanding of absorbing discrete diffusion.
Comprehensive experimental validation We conduct thorough study of these loss functions, with results presented in Tables[andd and To the best of our knowledge, this exploration has been explored in prior work.
I ALGORITHMS FOR TRAINING AND SAMPLING
Algorithm A0-ARM Training
Require: Network cø, samples from data distribution Pdata repeat
To ~ Pdata: π Uz.
1 1~ U1, ... d).
+ 1(T < 1) O wo +[(
Calculate L = 2-141
log (0g
(c)
Calculate VoL. and run optimizer. until converged
29

---

Published as a conference paper at ICLR 2025
Algorithm 2 AO-ARM Sampling
Require: Network Ca
Initialize æ < [M] [M]
Sample π U.-
for i in {1, [I....,d] do
Sample j ~ (=)(co(())=((()))
Update a² +-j j
end for
Algorithm 3 Discrete Diffusion Training (t-DCE Loss)
Require:
Network co, noise schedule a, time [0, T), samples from data distribution Pdata
rеpeat
To ~ Pdata. & ~ U ([),
Construct 24 by E¹ Bernoulli(e
1(56
1)r + = M)
Calculate Lo(
E=;=M -o(t), c- --
-ac log
1- (0 x()))
Calculate VoI(x,, and run optimizer. until converged
Algorithm Discrete Diffusion Training (A-DCE Loss)
Require:
Network Co, samples from data distribution Pdata repeat
To ~ Pdata. À ([O,, 1[).
Construct œ by E¹ ~ Bernoulli(1 - A), 23 I((' 1)) + I[5
0)[M]
Calculate Lo(xx,
Exi=M - log (c)(
Calculate (xL, and run optimizer. until converged
Algorithm 5 Discrete Diffusion Sampling (Unconditional)
Require: Network ca, noise schedule o, time range [0, (J, step size At + T, Ør + [M]. Coache & ca(w.).
while >> do
if Use Euler then Construct transition densities culer,
(x)-)(l by Eq. D.13 with Cracke-
xf-st Cat(y; (xt- (ad (x:)) for all xt [M]. of + of for all of + [M].
ouler.
end if
if Use Tweedie -leaping then
Construct transition densities pi (x)=)(k<)) by Eq. D.8 with Conche-
of zt-At Cat(pi--sg (=t (x)) for all æt [M]. æ of for all of * [M]
end if
if at-At At # at then
Coacke + (x)).
end if
t x t. At.
end while
30

---

Published as a conference paper at ICLR 2025
Algorithm 6 Discrete Diffusion Sampling (Conditional)
Require: Network Cø, noise schedule o, time [o, T), step size At, Prompt spaces Ω and tokens T. f construct ær with a T and æ [M]. Cooche <- ca(
while >> 0 do
if Use Euler then
Construct transition densities quler, Pr- ( by Eq. D.13 with Ceacke-
di-Ai
culer,i
( (xx)) for all oi [M]. mt de of for all ot o [M].
end if
if Use Tweedie т -leaping then
Construct transition densities Ae (xt (1)))) by Eq. D.8p with Coocher
zi- Cat(pi- (x,)) for all æt = [M]. xt Ac of for all r # [M]
end if
if at-At + 3z then
Ccache (x())).
end if
t 6-3 t At. end while
J EXPERIMENTAL DETAILS
J.1 MODEL DETAILS
We implemented our RADD model based on the SEDD architecture, an decoder-only transformer model (Vaswani al. 20176 Devlin al. 2019a). Our model incorporates rotary positional encoding (Su et al. 2021) I but excludes all parts related to time conditioning (i.e., TimeFmbedding, N- block (Peebles Xie 2023b) Instead, we added a softmax operation at the end of the neural network to ensure the output valid conditional distribution. This simplified architecture is similar to the standard GPT architecture, except the lack attention mask and multiple probability output instead of single one.
J.2 TRAINING DETAILS
We trained our RADD models using the following configuration settings:
Batch Size: 512
Learning Rate: 3 x 10-1
Exponential Moving Average (EMA):0.9999
Gradient Clipping: Gradient norm clipped to 1
Warmup Schedule: Applied for the first 2500 iterations
weight decay: 0.03
dropout rate: 0.02
The hyperparameters were adapted from (Lou et al. 2024), with modifications referenced from (Shi et al. 2024). The main modifications were setting the weight decay to 0.03 and the dropout rate to 0.02. Due to limited computational resources, we did not perform hyperparameter search and directly conducted experiments with these settings. Further tuning of hyperparameters may enhance performance.
In terms of training tokens, 400K iterations correspond to approximately 105 billion tokens, while 1000K iterations correspond to about 262 billion tokens. It's worth noting that the entire Open WebTee dataset contains only 9 billion tokens, meaning that models went through the dataset multiple times during training.
The small models are trained nodes of 32 V100 32G GPUs with float16 precision.
31

---

Published as a conference paper at ICLR 2025
J.3 UNCONDITIONAL GENERATION DETAILS
For unconditional generation, we employed log-linear noise schedule. As illustrated in Section 3.2 the Euler method and the Tweedie r-leaping method are equivalent under this case. In practice, the implementation of the Euler method and the Tweedie r-leaping method remains the same for RADD but differs for SEDD. So we measure the perplexity SEDD by Tweedie 7-leaping method which performs slightly better while it suffices to measure the perplexity of RADD once.
As suggested by (Zheng et al., 2024), except for Table 4 all samples are generated using float64 precision of Gumbel-based categorical sampling(bbrreviated fp64 precision below). No annealing methods (e.g., top-p or top-k sampling) were applied in our sampling process.
J.4 FURTHER EVALUATION OF GENERATIVE PERPLEXITY
Runtime and entropy measurement To evaluate the efficiency of inference and the diversity of samples, we assessed the inference time and unigram entropy averaged across 1024 samples. When calculating unigram entropy, we chose the natural logarithm (In) instead of the log2- We provide perplexity and entropy results under both fp64 and fp32 precision in Tables and
and and resppctivvly
Under both precisions. RADD and SEDD exhibit comparable perplexity results for the same number of sampling steps. For large sampling steps, perplexity converges under fp64 precision but continues to decrease under fp32 precision. This discrepancy is due to precision errors in fp32, which effectively function as a form of annealing, resulting in deceptively lower perplexity values (Zheng et 2024). To evaluate efficiency, we focus on the sampling time under fp64 precision.
The RADD model consistently required the shortest sampling time while maintaining similar per- plexity levels to the SEDD model. Specifically, RADD achieved a speed-up of up to 2.5 to 3 times with large sampling steps, shown in Table These findings align with the analysis of the E-NFEs in Fig. la validating the effectiveness of the RADD model and the caching strategy. Even with 1024 sampling steps(equal to sequence length), the cache strategy still enables about 1.5 times acceleration.
Table 3: Average inference time, perplexity, and entropy per sample with varying sampling steps under fp64 precision. The table compares the average inference time (in seconds), perplexity (PPL.), and entropy for the SEDD medium model using Tweedie 7-leaping sampling methods, as well as the RADD medium model under log-linear noise schedule with caching strategy. The experiment is conducted under single NVIDIA A800 80G GPU with batch size of 16.
Steps
32
64
128
256
512
1024
2048
4096
JEDD-medium
Time(s)
0.52
0.99
1.91
3.78
7.49
14.93
29.82
59.56
PPL.|
1.59
113
94
87
84
86
82
81
Entropy
8.26
8.18
8.14
8.12
8.09
8.10
8.09
8.07
RADD-medium
Time(s)
0.45
0.80
1.54
2.96
5.32
8.39
12.28
18.20
PPL.
158
113
96
89
84
83
81
81
Entropy
8.28
8.21
8.18
8.15
8.14
8.12
8.13
8.13
Table 4: Average perplexity, and entropy per sample with varying sampling steps under fp32 precision. The table compares the average perplexity (PPL), and entropy for the SEDD medium model using Tweedie r-leaping sampling methods, as well as the RADD medium model under log-linear noise schedule with a caching strategy.
Steps
32
64
128
256
512
1024
2048
4096
SEDD-medium
PPLJ
125
85
62
51
41
34
27
22
Entropy
8.18
8.07
7.96
7.86
7.73
7.60
7.44
7.25
RADD-medium
PPLJ
126
84
63
51
41
33
27
22
Entropy
8.19
8.09
7.98
7.88
7.75
7.59
7.45
7.27
32

---

Published as a conference paper at ICLR 2025
Efficiency of our caching strategy with mini-batch In Appendix D.5 we explored the average NFE for the single sample case. This concept, however, extends the mini-batch case.
In mini-batch, some samples may remain unchanged while others may evolve. To ad- dress this, we use dynamic batch-size strategy. Only the samples that have changed are passed through the neural network for computation. While this still involves a "batch-level NFE", the total NFE sample is reduced, effectively enhancing efficiency as in the single-sample case.
In comparison, concurrent work Sahoo al. 2024) does not consider dynamic batch size in their implementation, so their practical acceleration falls short of the theoretical potential.
To further validate the efficiency, we conducted experiments generating 64 samples under various batch sizes. The results, summarized in Table demonstrate that the average generation time with our caching strategy consistently outperforms SEDD across different batch sizes and timestep configurations. We provide a detailed analysis below:
BATCH SIZE AND GPU UTILIZATION For fixed model, increasing the batch size leads to improved GPU utilization before reaching the maximum batch size, reducing the average sampling time per sample. In the case of RADD, sampling time decreases significantly the batch size increases from to 4. However, the reduction in sampling time becomes minimal beyond a batch size of 4, indicating that GPU utilization has nearly reached its maximum capacity at this point. This demonstrates the scalability of our approach within the limits of the hardware's parallel processing capabilities.
SEDD vs RADD Under identical batch sizes and timesteps, RADD consistently outperforms SEDD in sampling speed. This highlights the efficiency RADD's design, where the caching mechanism reduces redundant computations and achieves faster generation, especially for larger batch sizes and higher timesteps.
Table 5: The average inference time across batch sizes and timesteps of SEDD-medium and RADD-medium. Experiments conducted NVIDIA 4090 GPU (24GB VRAM, maximum batch size = 8). The average generation time per sample (in seconds) is averaged over 64 samples.
Batch Size
Steps
32
64
128
256
512
1024
2048
4096
SEDD-medium
0.98
1.83
3.57
6.95
13.80
27.16
54.76
109.90
2
0.75
1.43
2.77
5.46
10.85
21.62
43.19
86.23
0.67
1.30
2.56
5.09
10.15
20.27
40.50
80.97
8
0.70
1.34
2.65
5.25
10.46
20.85
41.68
83.32
RADD-medium
0.77
1.34
2.56
4.86
8.73
14.00
20.45
30.70
0.60
1.08
2.05
3.98
7.32
12.32
18.88
28.97
0.50
0.97
1.87
3.65
6.75
11.26
17.67
27.67
0.51
0.97
1.90
3.71
6.76
10.87
16.76
26.58
Sampling as any-order autoregressive models
As outlined in Theorem I Cp can be interpreted as a conditional distribution over clean data. One natural approach is use this directly for gener- ating samples, similar to any-order autoregressive models. However, there d! possible ways to decompose the joint distribution into conditional distributions. We tested three representative cases:
forward: p(x ...
xdd) - p(1*(x(<0))
backward: p(u ....d) [ pl (2(()))))
33

---

Published as a conference paper at ICLR 2025
random: Ux. p(x' ...x"))))))))))).)))).).. I p(27(0)2=770)))
The results presented in Table 6 Perplexity calculated the average 1024 samples. For the random case, we calculated the average perplexity across different randomly generated n corresponding to the standard AO-ARMs sampling method. It shows that the result of the standard AO-ARMs sampling method aligns closely with the converged perplexity of the r-leaping method under fp64 precision in large steps. Among the different decomposition orders, the forward order demonstrated the best performance.
Table 6: Quality of unconditionally generated text evaluated by perplexity (L). For fixed model, the best perplexity is bolded.
Method
RADD-medium
Forward
81.70
Backward
103.68
Random
83.10
J.5 FURTHER EVALUATION OF ZERO-SHOT PERPLEXITY
In this section, we provide an extended evaluation of the zero-shot language modeling performance of RADD models trained for 1000k iterations. While the results in the main text focus models trained for 400k iterations, training for longer durations can slightly improve model performance due to the increased exposure training data.
Table 7: Additional zero-shot language modeling perplexity (4) for RADD small models. We present the perplexity for RADD models trained for 1000k iterations based on their corresponding loss.
Method
LAMBADA
WikiText2
PTB
WikiText103
IBW
RADD-I-DCE
48.92
37.44
102.49
37.20
70.58
RADD-A-DCE
49.74
37.13
98.84
36.66
69.77
RADD-AO
49.43
36.86
102.36
35.25
70.71
K ADDITIONAL EXPERIMENTAL RESULTS
K.1 ADDITIONAL SAMPLES
Additional unconditionally and conditionally generated text of RADD-A-DCE small and medium models are reported in Figs. [1o All of the samples generated with 1024 steps under log-linear noise schedule.
34

---

Published as a conference paper at ICLR 2025
Civilians of Puerto Ricans in Ohio. Chili replaced beer peas and oyster shs Kurds replaced corn with Louisiana barbeque in Louisiana, where chili served as the dinner dish to houseplants: 'Cajaro souvlaki".
In the southwest in the carly-19th century, this dish became used in the Texas South- [est.[]][12]]
When the 'American" to mean "Spanish F" or "Saw' arose in the thought, first in the south of Mexico (Columbus) and later in America (continent), chili served as, When cooked, medicinally mixed into dishes as side dish, with fruits and vegetables, short stewed meals, soup to date including poker pot.
Wackned chili also first appeared in North America in 1819 as some accounts describe its use as a turkey stew and chicken hock". Other versions, which are known of by euphemism as the sussificatte guitaristi, have pinned the origin of the term on the original date of the Passiacoamerica during which Spanish tribesmen could not venture into the region until (after the end of the Pequewille; jumples mugged and smoking was observed) Bigger's Passage upon the Coast. Some researchers believe that chili is served as regent proving in nice winter as part of the traditional diet, not barbecue although the Uuy Juaco is traditionally eaten. [aten.[13][14] [13][14] The Mexican website Chili, which first occurred in 2006, contains short page reliving 15 different barbecue recipes already served in Mexico under Brandon and Cincy of the Amer- ican revolutionary movement of the [15[111 Chili became a frequentstay of Mexican dining at the Walton, Colo., family grocery store in California and North Carolina, and it has been found in several American grocery stores. Original scout Midwest founder Michael Dunbar declared in 1998 that this corrugated chili dish was "consistent with the food that served in Mexico during the 19th century."(u '[1] End of era and availability
The Mexican Chili became regularly served in United States Central Texas as of November 1, 2004 (December 1934), and has been sold since at Chestnut many times.[17] Many Mexican barbecue joints across the West Valley began as a cuisine instead of a regular and later would use chili, rather "porocera cooking by returning Mcsopotamians. (arriiinnsnn(()))__ [[18] In culture [ edit]] ]
Chili was a coarse cured dish made with spices and seasoning, usually chili or side chili. Mex- ican meat was assigned to the game. Wet meats were made using heat, followed sometimes with hot beer or wine, such as limes or beer. Stroked chili keeps spaghetti sauce or breast dishes such as parmies and cheese. Chili was used to warm flatbread, hot dogs, lamb jerky and dried moonplugs. It was also used in hamburger, except where cuts of meat is used in meal. The dish was also used in toppdown, fried chicken, egg rolls, papaya noodles, yakka noodles, garlic, and onions. Chili is used in wine batter, unsweetened spaghetti squash which is often mashed together in sandwiches.
The Mexican Grill locations are offered at Kearney, Purduc University, we travel to other Inc. restaurants either with affordable prices at commi-coperraiive/oluttaa Stores range from California to Texas. Chili can also become available from the local Italian market, often as available the Chili Jack's restaurant the Black Hawk Courthouse. [22] Chili is not currently produced for local Italian producers. New Mexican imports [ edit
locations.
A new emphasis of use of arugared Mexican beans, as well as black beans, BL beans, pickled peppers, quesadillas, and peas, form them as a menu item alternatives to items as manufac- tured by KFC and Mexican Grill.
The Chili is popular because of tortilla[19] which a regular ingredient with sandwiches between Sundays. According to Google Live-Mexico Taco, making fresh, chili-free meals includes pulled pork with cheese removed sweet potatoes, ranch oranges, and mild marlic cheese with floured brown bread. Guajías, chillies, and during cold periods are also included in Mexican Chili. Caesar salad is featured in many facets of East Mexican Grill. Bay Area El-Lachi, Taco Bell employee and blogger, mentions lunch menus. Options, the better bets, were the traditional winter chicken dishes on Fridays in the Hudson Brewery that call the restaurant proper [21]
Figure 2: Unconditionally generated text of RADD-A-DCE small.
35

---

Published as a conference paper at ICLR 2025
I have Kot actu my vit" Shit est thou est first and me first Work Begins By email: 10 Jamuary 2012 7:21 GMT Kristy
Latest article at the 'Mail Stream page Impressions
Because I am still relying on the e-mails of the EIS server to indicate legitimate databases, I fail the virtual sharing argument, share function, disk space index, etc. (Failure is not suf- ficient to have my mail client be contains privileged information...) once set the transport destination on the email e-mess, it contains privileged information extra.
My e-mails list which connections have received data, and you can use RSS view the packets. In an application with a connection infrastructure you can create an arbitrary object structure, for which and for the system where layers of your communications stack are designed.
enforce kernel space mass, append first 2. y ;; O. cons. end eq bd. nec. end dfl lift&d foo 8 11 modiffed-scwd0h Is 120 127 dbbsi dbbsi check 503(123-s3)p #syntkillledestpoodd The client also throws back an IP address hash that can easily be cached right after it originally generated; low-level operations run by microservices. Using UodyURL, parameter, I run a privilege test.
After reasonable approximation of Oracle's 's best RSA algorithm, the result is of RCELOG assuming the assertion that this method works all 1 n of trails [day 11 -1-oo day left] + job from 5-18 and last for 9 after J.T. Leeper's Theorem (which then contradict the thesis that a cleanup bug may exist). I couldn't 't resist finding that, if it did not, needs to be fixed in a ASAP in order for exploit.
A better example to go of is using an IA-64 firewall in Server Enterprise. The result can enable you to continue leveraging Server Enterprise's firewall design, but not ignoring the same. Remember, if you want to enforce Server Enterprise's anomaly control, you need how it seems to integrate with both functionality and logic.
One last reference to the DBTR leak we first found in the world in our focus on Server Enterprise is that a client application can not query physical storage systems. The incom- patibility of default logic with lightswitch is not of subject per se restriction. Alten can benefit from impertional clanchism, therefore worth keeping in mind that the solution also includes both Java EE applications.
Subsidizing Server Identity
A collaborative research project specializing in server authentication is as important as it is to the user base; it is easy for ORM and AD tools to handle authentication from each Server without an extensive need, external to which server.
To have only one authentication stickler running an instance, and loading a form authentica- tion service that joins one server with a live e.g.i. justifiable sex, a declarative, composable- by-side Expanded Persistent Application ERP) class is brought about.
In general, it tends to only need two IDs in an application approaching RHEL. InJh mail company distributed in New York City''s email system, with 6 ID., This application should also deal with issued data model, ERP (e.g. data model proxy (0.), database (0.6.) boundaries.[ 10)
The majority is typical chief operating in order to accomplish a technical milestone, and [ is kept calm by the development team [1]. As a result, being a ready-track application, this open consistency Gatwwayy is very much ready. A fuzzed server data file with sheltered and forstrap per day routines with messages such as response, reply, reply, excluded is reserved for applications.. that contain SQL. A server file can be deployed over two machines under the requirements of the application, such as building windows of good email for long lime, lo simulate simple third-parly software for mirrorinn or for slab email for messaging. It is also settled that an application requiring isolation test passes must have an isolated SQL server resources, which are typically lightweight enough by default (tight door if you're) to manage.
Configuring Environments standard R22. R2 is a practical 'dry" multi-purpose instantiation mechanism, but it still recieved different designations using knctions that contain module mechanism (where or wherever), i.e. only module will be (elegally) it can be harness, but in the context supporting the deeroperrbilityy not every application must implicitly adopt only one and allows supporting the similarity of the particular implementation. Why do it use R names, but the usual reason is that: it's a database that is accessed in a joint framework.
Figure 3: Conditionally generated text of RADD-A-DCE small. Prompt tokens are in blue.
36

---

Published as a conference paper at ICLR 2025
Hall of Fame quarterback to Pro Bowlist. At times, Wright hasn't hit this reset button- he had a recommittal with the 2011 regime, but he certainly is on notice: Who'd rather be unemployed starring tennis all-stars than a 500-castle NFL franchise? Is the experience perfect match for him? Can he go over Suh, who is coming off a Super Bowl championship season that included a major letdown expected him to just discuss: torn ACL?
Derrick Williams vs. former Nebraska QB Kevin Youkil at the University of Miami: "Bright warm-up. Become captivated by the game in right light and suppose, at both the amateur and top-level level Brandon Pettit came into consideration from visionary standpoint. Third thought: revered Robb Hart's job as offensive coordinator while as an assistant coach at Towa State. Let Jimmie Langer tell his story.'
Ricky Duches, former Ohio State coordinator, defeated new LSU offensive line coach Les Miles with high school Nebraska quarterback Willy Clabo. (In the movie from Boston, Katerla Floodgate also got the Lombardi Trophy.)
Williams vibes as he recalls his father and argeeneabyyyy he's been better physically and at other synonymous with the game: 'Back in 1982, when I was 16, I saw the big-time football. I knew what, the NFL, was...T got to Superdome, and out T knew Jim Brown wasn't exactly greeting Nogueira with being handshake. But T saw it! [When] else got out on the field, T stood there walffsw.dickingg... We liked it for a little while.' As Al Crump, the experienced conference speaker, was truly dazzled in a ballroom, tells me of her visit, "We've been in the group for years. I've been protty carcfully planning get a visit. There were Scahawks and could I say badgers? While suffering.' Ninetcon-year-ohe Husker's fan saw as a chance to make her own observations amidst the relentless publicity during the late-May 26 offsite disruption.She 's discovered better than most what other attendees have done when it comes to integrity and candor in their feats with interviews, and bloggers featuring her accounts of the Washington Redskins and Chuck Siedelbaum. Beginner loves groupies. Sgt. Gun Casper Olson, free agents with San Diego at Faulkner: 'I got to really get to know my brother, Buck (Olson, Sr.). We're a supervisor in the Major League for the last 15 years, who will work with us in the 159millionand10 260 per billion level, so we got to get to know him every single day. Came to see who often begins and ends up staying where professional sports are. Hawaii didn't offer it as Kansas did.' However, in many ways, he was highly complimentary of that former team's 's coach, comparing what Hawaii offered to Voeltt Park whom he described as Kurt Stater of the Eagles: 'I got to see Cliff Starsy, North Carolina's head coach at the time, and watch Nick Saban, who was the potential replacement for the Alabama. coaches before they got his hands on Bradley /Hedge Briles. watched everything. On the football field was his ono. As coach, he gets so excited, I couldn't 't drive with him on the field, though he wasn't playing. His closest friend manages TSA for him. And our lineback players' m ht excited that he puts on. I want to know where Robert Griffin III would be next bocause he has reached out to Matt directly. We are so intense that my drive won't much use.'
reveals just how big- big-time football is. And what he received for being so
Chris Jackson, Jr., Jr., 2015 Stanford and Paula, who were then older than him and split chances are part of several big families. Jackson's momma; Paula's soft loves; and Jackson's pressure on everything in the team and social sphere. Still, they are each given enough space to talk about the game ultimately one day. On Monday, for her first hour-long conversation as prospect with people wearing the surrogate for best mothers, she talked of her relationship with the ninth-grade scholarship and converting into an effective recruiting tactic; essentially, she had won her family's trust with linebackers that she could stage for a bright future. "And that shows true faith and confidence,, she informs me. Orryn Lamba Jackson, 2015 sophomore in Cornell, an African-Amerien with giant L.D. glasses, named Abraham Lincoln to Sanford Abraham, the mayor of her community in Benton, Ark., that school's throwback. She is as humble as Los Aldridge, where her book was held
computer geek
Figure 4: Unconditionally generated text of RADD-A-DCE medium.
37

---

Published as a conference paper at ICLR 2025
I have a ? Well, don't too't really think I leverage capital at the end of the day. have theory that these things are Fiction- ignorable and dangerous.
Huh... Hmm.. But they will never recognise me.
How fieree am I.. Ma Rdf, called mc out in the heat of sanctimonia and ignorance. I am in a position of being in the middle, not the pack.
Primordial wealth has no ample hands, but I am especially heavy. My first step is not gonna be rich.
As much as I claim, I will provide you with nothing. I will force you to do whatever it take to go the route that you deem appropriate. I'll demand that you become my expert tool. Usually when they gangge me, I know when their indulgence is not for me. You 're practicing WITH the Masochist!
A young lady took Petra wrapsrt me strawberry extract while he was thirsty as I was eating out that day.
She bragged as she hugged the unsuspecting waitress to tell him, "Let's revue that recipe! That meth thumb isn't good." Besides, she said, "So friggin' gums don't mong like that cure som a bowl!" "ool!"'/eeddie </endoftexx| 'liars stalling liars." The Town Movement. Leaders Still Nader Meanwhile?
On November 9, new report. [partedially co-written The Favironmental Working Group] revealed allegations that the EPA had routinely used decoys on pigs in Oregon today, exac- erbating Ecolabelion in U.S.-origin food.
Let's face it it is time begged little children to form the cover of the smoke is inferno and crime malls no matter how likely it is for them to say something funny or remember their fathers mumbling with far more reluctance when addressing the press. So we expect our government agent to be moral when they have a (public) right to information without leaving the door open to their investigation BILL May they learn our government officials needn't 't superlative skills in which case they take tough line...One reason all reported tests are said to be acts of terrorism by corrupt agents of the political is you've voe've never reached a threshold level you must accept testing. So there are two reasons why?
One is that the public public opinion has changed recently. Takes time, and no public pressure won't be meaningful to algae farms when the bloggers get down to shove. We were honestly assured of all of this pressure on the way-down to Nixon tell us if you are willing to listen to sly reporter.. The folks you Daily Sheeple have the most difficult time doing is fishing crackers to gather sand with hundreds of pounds of scales, concrete, or grease trays. * *
Related: Bringing Up:
Earlier: Former EVA Does "Nader's EPA" " Test In November, Parrotes Daily Exam
Earlier: Mining Smog's Seed Means Persuating Boger Ecolideca
Photo Credit: Biolanao/ WWhiser<</endofffee (cwnoootex//-Maarrrrrreee Macrooorggg.,.. published in Environmental and Swedish Medicine, have detected in what the team considers to be the most lasty and dismaying organism and object in the world. The researchers studied the diet of 60 million modern sheep, goats, sheep, cattle, chickens, rats and mice whose accumulate more data revealed patterns of greater importance to be found in the human population. Specifically a gene (IFAS-alpha) disproportionately responds in mice to oxidative stress.
inter-cell growth factor-alpha (IFAS-alpha) gene
Rheum-cnriched flavon fat (AST) and consumption of milk fat-oddimal fractional density lipoprotein (FFL. were indeed shown to have significant effects. Medication of these sub- stances in control animals or controls did not induce the selected IFASSallha. colon cancer rates, in contrast with the original cause in antibiotic- and non-infected control mice that underlie the principle, were not necessarily improved.
In turns antibiotic resistance and colonally carcinogenic antibiotic feeding bacteria not related. Numbers are unclear but the fact these changes are attributed to them proposes that conventional antibiotics could not deny known causal effects on the adjacent cousins. The Swedish rescarchers led by Professor Erik Johansson and Anne Kristin Gunnarsson- Silk both University of Sweden who were elected to chair the "Global VIP Conference on Animal Food Physiology' today on May 1 in Vilskovisno, shared by the Office of Ontrialed AounselingASE MEDIC; the high-level Swedish Food Safety Research Institute; Division of Borneck, Tissue, Cellular and Virus; System Genomics Service that operates in a joint framework.
Figure 5: Conditionally generated text of RADD-A-DCE medium. Prompt tokens are in blue.
38


Reviews:
[{"review_id": "HcWebFM4g9", "reviewer": "Reviewer_u3V7", "summary": "This paper proposes a reparameterization of the absorbing discrete diffusion, based upon the finding that the concrete score can be decomposed into a time-independent conditional probability of clean data, and a time dependent but data independent term. Therefore, this paper proposes a new network architecture, that removes the time dependency, and add softmax layers to project the output layer to the probability space.\n\nThe authors argue that the removed time dependency can accelerate the sampling process by caching the network output before it changes. The total changes would be at most $d$ instead of $n$, and the total number of NFEs reduces to  $n(1 - (1 - \\frac{1}{n}))^l)$.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The authors proposed a novel reparameterization of absorbing discrete diffusion, decoupled the time-independent and time-dependent part. This discovery allows for better understanding of discrete diffusions, and the previous reparameterization in SEDD.  The proposed reparameterization allows for a simplified network architecture and efficient sampling.\n\nMoreover, the authors proposed the unification of absorbing diffusion with the any-order autoregressive model.", "weaknesses": "First of all, the proposed algorithm are specified to absorbing discrete diffusion, and does not fit in the other widely used discrete diffusion model, multinomial diffusion. The authors did not discuss if or why not their methodology can not be applied to multinomial diffusion.\n\nThere are existing works that have reduced the sampling complexity of discrete diffusions, including absorbing and multinomial diffusion, e.g., Chen et al. 2024 [1]. [1] also made use of the fact that during the reverse sampling process, computation is only required when tokens changes, thus reducing the computation from number of diffusion steps to number of tokens. Also [1] proposed a similar NFE estimation in Theorem D.1, which is the same as (3.4) in this paper. However, the authors did not discuss or mention the relationship with [1].\n\nAlso inspired from [1], the sampling complexity essentially does not comes from whether you have $t$ in your score network or not, but comes from times that you change tokens. Even for SEDD based discrete diffusion network, it is also possible to only calculate when change is required, and skip calculation when $x$ does no change. For example, if using [1] for sampling, does RADD still have any advantage over SEDD?\n\n[1] Fast Sampling via Discrete Non-Markov Diffusion Models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3mTbg07wLR", "reviewer": "Reviewer_rVfV", "summary": "Previous work in discrete diffusion language modeling have relied on estimating a time-dependent concrete score. The authors derive a simpler parameterization, RADD, expressed as a time-independent conditional probabilities of the clean data, multiplied by a time-dependent scalar. By deriving a simplified objective, they provide a deeper understanding of discrete diffusion and draw a connection with any-order autoregressive models. By training a time-independent denoising network, they simplify the model architecture by removing additional time-conditioning layers used in prior discrete diffusion language models. The authors show theoretically and experimentally that using a time-independent network can significantly reduce the number of function evaluations at inference. On language modeling benchmarks, their architectural simplifications and optimized training loss results in state-of-the-art performance among discrete diffusion models at the GPT-2 scale.", "questions": "- Does the improvement from including the scaling term in SEDD also hold for RADD?\n- If a claim is that the simplification enables a simpler architecture, can the authors provide comparative analysis between RADD with and without time conditioning while controlling for parameter count?\n- If the objectives presented are equivalent, why are there differences in the zero-shot likelihoods in Tables 1, 2, 5?\n- In Table 4, can the authors compare perplexity from any-order autoregressive generation and parallel generation using the $\\tau$-leaping sampler?\n- What is the number of tokens used for training, instead of the number of iterations, in order to compare with baselines? In Table 5, the authors report zero-shot likelihood on models trained for 1000K iterations, why do the models in the main paper use 400K iterations?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- They provide clear, rigorous analysis showing the equivalence between the concrete score [1], their simplified parameterization, and the any-order autoregressive loss\n- The connection between discrete diffusion models and any-order autoregressive models is novel and unexplored in prior work. The authors provide a theoretical and empirical comparison\n- They show substantial improvement in sampling speed over SEDD [1] through convincing theoretical and experimental analysis\n- They provide thorough comparison to concurrent work [2,3] and explain the added novelty introduced by their paper\n\n[1] Lou, Aaron, et al. \"Discrete diffusion language modeling by estimating the ratios of the data distribution.\"\n\n[2] Shi, Jiaxin. \"Simplified and Generalized Masked Diffusion for Discrete Data\"\n\n[3] Sahoo, Subham Sekhar. \"Simple and Effective Masked Diffusion Language Models.\"", "weaknesses": "- The experimental results are limited. Their likelihood evaluation only includes zero-shot likelihoods using models trained on OpenWebText\n- The empirical comparison across RADD models is misleading as they have equivalent objectives (Tables 1, 2, 5). In Table 2, one RADD model underperforms SEDD on LAMBADA while another outperforms.\n- Experimental ablations on the architectural and theoretical simplifications are not provided, making it difficult to understand its impact on likelihoods (an ablation on the scaling term is instead provided for SEDD). For example, they claim to design a time-independent network that simplifies learning: it would be valuable to compare RADD with and without time conditioning, while controlling for parameter count\n- The sampling speed is not compared with AR, making it unclear whether the speedup improvement has practical significance", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TZ8AkIhjBk", "reviewer": "Reviewer_fLbR", "summary": "This paper studies simplifications of discrete diffusion models with absorbing/masking processes. First, the concrete score in Lou et al. 2024 can be reparameterized as conditional probability of clean data, i.e. p(x_0^i | x_t). Second, when noise samples are not changed during a time-step, cached NN output from previous step can be reused (since the input values are unchanged). Lastly, equivalence between absorbing discrete diffusion and any-order autoregressive models is identified. The simplified training objectives enable better test data perplexity on zero-shot language modeling tasks. The caching trick helps reduce number of function evaluations.", "questions": "- Experiments: For evaluation on language modeling, in G.4 evaluation of generative perplexity, which model is used for evaluating the generation quality? The numbers in Table 3 seem to be much higher than SEDD’s generative perplexity evaluated using GPT-2-Large. \nAnother ablation might be training both SEDD and RADD to 1000k iterations and compare the performance gap v.s. SEDD and RADD with 400k iterations. Just to see if the gap closes down with more training, or is there a fundamental gap in learning under different parameterization of the model.\n\n- Theory: Clarify which contribution is original and which are already existing in previous literature. Point connection to existing literature when there is. For example, connection to cross-entropy loss proposed in (Austin et al. 2021, Hoogeboom et al. 2021, Campbell et al. 2024); and the connection of caching to derandomized sampling in Chen et al. 2023.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The first result on reparameterizing score as conditional denoising probability is the most original contribution. It helps clarify that training concrete score entropy and clean data reconstruction cross-entropy are essentially equivalent up to scalar difference. And this results in better training efficiency compared to SEDD as shown in language modeling experiments. On a side node, the use of clean data reconstruction cross-entropy loss for mask/absorbing discrete diffusion is previously proposed in Austin et al. 2021 for discrete timesteps and Campbell et al. 2024 for continuous time. The connection of this reconstruction distribution and concrete score is newly identified in this paper.", "weaknesses": "The second result on caching when samples are not changed during a previous timestep is somewhat well-known.  For example, Chen et al. 2023 proposes to only incur NFE at times when actual transition from mask token to infilled token happens, which also corresponds to how AO-ARM sample one dimension at a step. The caching strategy proposed in this paper becomes less efficient when sampling a batch of samples: in a small timestep when some samples might remain unchanged while some are not, caching cannot be used. Hence I think it should be more efficient to use different $\\Delta t$ time steps for different samples in [1], and the $\\Delta t$ should be sampled to be the amount of time until next transition from mask to data token for each sample. \n\nThe third result on equivalence of absorbing discrete diffusion and AO-ARMs is not new. This has been shown in previous works (Austin et al. 2021, Hoogeboom et al. 2021, Campbell et al. 2024)\n\n[1] Chen, Zixiang, Huizhuo Yuan, Yongqian Li, Yiwen Kou, Junkai Zhang, and Quanquan Gu. \"Fast Sampling via De-randomization for Discrete Diffusion Models.\" *arXiv preprint arXiv:2312.09193* (2023).\n\n[2] Austin, Jacob, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. \"Structured denoising diffusion models in discrete state-spaces.\" Advances in Neural Information Processing Systems 34 (2021): 17981-17993.\n\n[3] Hoogeboom, Emiel, Alexey A. Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. \"Autoregressive diffusion models.\" *arXiv preprint arXiv:2110.02037* (2021).\n\n[4] Campbell, Andrew, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. \"Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design.\" *arXiv preprint arXiv:2402.04997* (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xXs05GcYlh", "reviewer": "Reviewer_rqCn", "summary": "The authors express the concrete score for an absorbing (only masked to unmasked) diffusion in terms of a time conditional scalar and prediction of the t=0 data distribution. This significantly simplifies the training of discrete diffusion models as given the state transitions only from masked to unmasked the rate matrix is 0 at all most all positions.\n\nThe authors then explain how such a parameterisation can lead to efficient sampling as one may cache the unmasked components rather than recompute.", "questions": "See weaknesses.\n\nIt is mentioned that the network is time-independent, however, if it is the same as Prop 1 in [1] as discussed in section 5, then it would need to condition on time it seems. Is it really the same as in Prop1 of [1] or can the term in Prop1 of [1] be simplified to be time independent?\n\n[1]  Shi eta.  Simplified and generalized masked diffusion for discrete data, 2024.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well explained, the authors make a thorough comparison to concurrent works [1,2] which propose the very much the same methods.\n\nRestricting the generative process from unmasked to masked sounds limiting but greatly simplifies the parameterisation of the concrete score (RADD) as simply learning the masked to unmasked transition. \n\nThis RADD reparameterisation also provides insights into heuristic \"scaling\" in prior works.\n\nThe authors provide detailed numerical evidence of how the caching procedure reduces the E-NFE in Fig 1a.\n\nThe empirical performance is compelling.\n\n[1]  Shi eta.  Simplified and generalized masked diffusion for discrete data, 2024.\n[2] Sahoo et al. Simple and effective masked diffusion language models,  2024", "weaknesses": "Although the caching reduces the *effective* NFE, I assume the actual number of function evaluations remains the same given the number of steps, just with fewer outputs per evaluation? This may not reduce workload for jitted - shape static compiled programs like those implemented in jax.\n\nAs discussed, many of the ideas are the same as those in concurrent but published works [1,2] from June 2024. Whilst I give the authors the benefit of the doubt given concurrency and explaining similarities, it does temper my score slightly, given it is not clear how the author have attempted to differentiate the paper for this submission.\n\nI will refer to the AC as to whether I should consider these papers, my current score does not reflect these papers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HWXectQtTW", "reviewer": "Reviewer_4Udp", "summary": "The paper introduces a reparameterized absorbing discrete diffusion (RADD) framework, which reinterprets the concrete score of a masked diffusion process as a conditional distribution on the clean data. This approach allows for modeling the backward process independently of the timestep t, significantly reducing the number of function evaluations required during sampling. Additionally, the paper establishes a connection between discrete diffusion and any-order autoregressive models, offering greater flexibility in choosing the training loss, sampling procedure, and likelihood evaluation.", "questions": "The parametrisation of the any-order auto-regressive model (AO-ARM) is unclear to me. Given a denoising model p_\\theta(x_0 | x_t) trained with the t-DCE loss for instance, could it be reparametrised as an AO-ARM? It would be helpful to include a seesion in the appendix discussing the parametrisation of AO-ARM and to provide both the training and sampling algorithm of AO-ARM in appendix F.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a theoretically sound method for reframing score entropy discrete diffusion as a denoising prediction. The insight drawn from its connection to any-order autoregressive models is particularly valuable. Furthermore, the experimental results provide strong support for the claims made in the paper.", "weaknesses": "Most of the theoretical results in this paper align with concurrent and prior research. For instance, time-independent parameterization is also discussed in [1]; t-DCE and tau-CDE are introduced in [1] and [2], with [2] additionally proposing a caching trick to enhance sampling efficiency.\n\nMoreover, the connection between masked diffusion and any-order autoregressive models is established in [3, Appendix C]. It would be appropriate to give proper credit to that work.\n\nNevertheless, I believe this paper offers valuable contributions and insights, with experimental results that strongly support its claims.\n\n[1] Shi, Jiaxin, et al. \"Simplified and Generalized Masked Diffusion for Discrete Data.\" *arXiv preprint arXiv:2406.04329* (2024).\n\n[2] Sahoo, Subham Sekhar, et al. \"Simple and Effective Masked Diffusion Language Models.\" *arXiv preprint arXiv:2406.07524* (2024).\n\n[3] Hoogeboom, Emiel, et al. \"Autoregressive diffusion models.\" *arXiv preprint arXiv:2110.02037* (2021).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)