Continuously Augmented Discrete Diffusion model for Categorical Generative Modeling
Authors: Huangjie Zheng
Shansan Gong
Ruixiang ZHANG
Tianrong Chen
Jiatao Gu
Mingyuan Zhou
Navdeep Jaitly
Yizhe Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Diffusion
Language Modeling
Code generation
Image generation
Primary area: generative models
TL;DR: We present a discrete diffusion model paired with continuous diffusion in the latent space, which improves the categorical data modeling in text, image and code domain.

Abstract: Standard discrete diffusion models treat all unobserved states the same way, typically mapping them to an absorbing [MASK] token. This creates an "information void" where global semantic information that may be inferred for the masked tokens from the unmasked tokens is not directly passed from one denoising step to another.  We introduce **Continuously Augmented Discrete Diffusion (CADD)**, a framework that augments the discrete state space with a paired diffusion in a continuous latent space. This yields graded, gradually corrupted states in which masked tokens are represented by noisy yet informative latent vectors rather than information voids. At each reverse step, CADD uses the continuous latent as a semantic hint to guide discrete denoising. The design is clean and compatible with existing discrete diffusion training. At sampling time, the strength and estimator of the continuous latent vector enables a controlled trade-off between mode-coverage (diversity-oriented) and mode-seeking (context-localization-oriented). Empirically, we demonstrate CADD improves generative quality over mask-based diffusion across text generation, image synthesis, and code modeling, with consistent gains on both qualitative and quantitative metrics against strong discrete baselines.

Published as a conference paper at ICLR 2026
CONTINUOUSLY AUGMENTED DISCRETE DIFFUSION MODEL FOR CATEGORICAL GENERATIVE MODELING
Huangjie Zheng', Shansan Gong" 22 RRixiang Zhang', Tianrong Chen', Jiatao Gu'
Mingyuan Zhou!.3 1.3, Navdeep Jaitly', Yizhe Zhang"
'Apple Hong Kong University TThe University of Texas at Austin
(huangjie__hie e_engg, ruixiangz, tchen54, jgu32, njaitly, yizzhangjapple hisansas@gmail. com, mingyuan. zhou@mccombs utexas. odu
ABSTRACT
Standard discrete diffusion models treat all unobserved states identically by map- ping them to an absorbing [MASK] token. This creates "information void" where semantic information that could be inferred from unmasked tokens is lost between denoising steps. We introduce Continuously Augmented Discrete Diffuu- sion (CADD), framework that augments the discrete state space with a paired diffusion in a continuous latent space. This yields graded, gradually corrupted states in which masked tokens are represented by noisy yet informative latent vectors rather than collapsed "information voids". At each reverse step, CADD may leverage the continuous latent semantic hint to guide discrete denoising. The design is clean and compatible with existing discrete diffusion training. At sampling time, the strength and choice of estimator for the continuous latent vector enables a controlled trade-off off between mode-coverage (generating diverse outputs) and mode-seeking (generating contextually precise outputs) behaviors. Empirically, we demonstrate CADD improves generative quality over mask-based diffusion across generation, image synthesis, and code modeling, with consistent gains on both qualitative and quantitative metrics against strong discrete baselines.
Masked
Diffision
t=1
IASK IMASK] [MASK]
Discnete [MASK] [MASK]
Discrese [MASK] Models
Discrete Difusion Models
Space
Languogr
Contimmons Diffision
CADD Space
Models
Biscie Bifusco Nf
Biscrete Diffucinn Modele
Dicorde tervie Briookery
Discrete Nadess
(Qun)
Docrale
Discnisc SPROM Ploiel
Dberate Diffus on Mooids
Figure 1: Comparison of diffusion models in different modeling spaces. Masked diffusion models [MASK] as noise and decode through single mask-to-token path, while continuous diffusion models explore the Gaussian space but often produce unreadable tokens due to the vast search space. CADD combines the stability of masked diffusion with the flexibility of continuous diffusion, enabling better token decoding for masked positions.
INTRODUCTION
Diffusion models have significantly advanced generative modeling tasks (Sohl-Dickstenn et al. 2015
Ho al. 2020 Song et al. 2021 Dhariwal & Nichol 2021 Karras et al. (022), particularly in
"This work done when Shansan Gong, Mingyuan Zhou, and Navdeep Jaitly were Apple.

---

Published as a conference paper at ICLR 2026
image synthesis (Saharia et al. 2022; Esser et al. 2024 Polyak et al. 2024 Zheng et al. 2024a
Brooks et al. 2024). Recently, with the rapid progress on discrete diffusion models (Austin et al.
2021a Hoogeboom al. 2021: Lou et al., 2024), diffusion models have become strong tool on the
discrete categorical data domain, such as text generative modeling and code generation (Gat et al.
2024
Gong et al. 2023 2025b).
Early work on Continuous Diffusion Models (CDMs) for categorical data maps tokens into a continuous space, applies Gaussian diffusion the representations, and then rounds back discrete
symbols (Li et al. 2022 Dieleman et al. 2022, Han et al. 2022] Zhang et al. 2023 Gulrajani & Hashimoto 2023). This route preserves smooth semantic signals and enables the use of established score- based methods. In parallel, Masked Diffusion Models (MDMs) have recently shown strong
results for categorical data (Shi et al., 2024, Sahoo et al. 2024 Nie et al. 2025): instead of adding noise in an embedding space, MDMs progressively mask tokens over time and learn to unmask them, yielding clear training signals via leven-level cross-entropy.
Despite their respective successes, both approaches have limitations, which illustrated in Figure 1 (i) MDMs suffer from information loss due to their use of absorbing [MASK] state (Wang et al. 2025). This design collapses all unobserved possibilities into one symbol, erasing any information about how close a corrupted position is to the original token, creating an "information void". This reduces the information available for the model to resolve ambiguity and maintain global semantic coherence. As an example shown in the right of the figure, if a masked token could plausibly be "Languagee or "Diffusion", the [MASK] representation offers no semantic clue to favor one over the other, forcing the model to make a hard choice without graded guidance. (ii) While CDMs can represent semantic proximity, they face different challenge, known as over-smoothing. Because the denoising process occurs entirely in a continuous embedding space and discretization to tokens only happens in the end (Gao et al., 2022), their continuous denoising objective can over-smooth token identities, which makes it difficult for the model to make precise predictions without localized context.
To address these challenges, we propose Continuously Augmented Discrete Diffusion (CADD), which integrates the strengths of CDMs into MDMs. CADD retains the discrete masking trajectory and augments it with a paired conditional Gaussian diffusion in a continuous semantic embedding space. Our forward process jointly evolves the token sequence and its latent, positions that are masked in the discrete path are accompanied by noisy yet informative latent vectors rather than information voids. In the reverse process, the model uses the continuous latent soft semantic hint to guide token denoising at each step. while the discrete context constrains the latent dynamics locally. Returning to Figure [ the continuous manifold offers graded path between candidates ("Lannuage" and "Diffusion" in this case), and the discrete neighborhood restricts the search space, allowing movement within the triangular region between hypotheses and enabling smooth transitions driven by the hints. In addressing the limits of both pure MDMs and CDMs, our contributions are:
1. Better token prediction with soft hints. For masked positions, the continuous latent preserves graded proximity to the ground-truth token embedding, which reduces ambiguity and makes discrete prediction easier.
2. Diversity without off-manifold drift. At inference, one can resample the continuous latent (e.g., multiple latent draws per discrete state) to explore alternative yet valid choices for a token or span. Because the discrete head stays grounded in the vocabulary, diversity comes from semantic variation rather than uncontrolled noise.
3. Training and sampling remain simple. CADD keeps standard cross-entropy for tokens and a standard diffusion loss for the continuous head. The sampler alternate or jointly update the discrete and continuous states.
4. Efficient finettunng CADD requires no special architecture and can reuse the same backbone as any MDM, enabling efficient fine-tuning of existing MDM to gain the above benefits
2 RELATED WORK
Discrete Diffusion Models Discrete diffusion models (Hoogeboom et al. 202 Zheng et al. 2024b) Austin et al.] 102 operate by defining Markov chain over the discrete token space, gradually diffusing the data with either uniform or absorbing transitions. Later, the model was unified and
simplified to continuous-time masked diffusion models (Camphell et al. 2022 Lou al.
2024
Shi
2

---

Published as a conference paper at ICLR 2026
et al. 2024: Sahoo et al. 2024 Zhang et al., 2025b). Building this, several recent works further scaled diffusion LMs to 7B parameters (Gong et al. [225a Ye et al. [225] Nie et al. 2024]. achieving performance on par with AR models. Parallel efforts explored unified multimoda variants that model text and images both in discrete token (Yang et al. diffusion models do not allow unmasked tokens to change, errors can accumulate during generation because of suboptimal unmasking in the earlier steps, Several enhanced (re-) techniques have been proposed, using bits and simplex representation to enrich the binary choice of masking (Chao
2025). However, because masked 2025 Li et al.
et al. 2025 Song et al. 2025a), remasking during the reverse process Gat et al.
2024 Zhao et al.
2024
Wang et al. 2025). enabling edit operations (Havasi al. 2025 Song et al. (025b).
Continuous Relaxations for Discrete Data Early continuous approaches either learn denoising
in latent embedding without explicit statistical structure (Li et al. 2022: Dieleman et al. 2022
Chen et al. (203) or fully relax tokens into unconstrained Euclidean space as simplex (Han et al.
2025 Jo & Hwang. 2024 2025). However, such unconstrained relaxations often fail preserve the inherent discreteness and categorical semantics of language (Gulrajani & Hashimoto, 2023). More recent methods impose structure in the logit
2022] Karimi Mahabadi et al. 2024 Tae et al.
2023) or directly the probability simplex via space (Hoogeboom al. [202]] Graves al.
Dirichlet priors (Avdeyev et al. 2023) Stärk et al. 2024), enforcing stronger statistical constraints on the noising process. Flow-matching techniques further treat the simplex statistical manifold (Liu ct al. 20023 Cheng et al., 2024 Davis et al., 2024). yet these approaches still lag behind discrete diffusion models in generation fidelity. Recently, Zhang et al. (2025a) leveraging density models with normalizing flow (Zhai et al., 2025, Gu et al. (2055) for flexible language modeling, and Sahoo et al. (2025) connect discrete diffusion language models and the underlying Gaussian diffusion.
Bridging Through the Lens of Mode Balancing Our work is also motivated by balancing mode seeking and mode covering. Related efforts pursue this balance via guidance methods that tune the pirersit trade-off (Dhariwal & Nichol 2021 Ho & Salimans] 2022); score-distillation
approaches that sharpen samples while retaining diffusion training for coverage (Poole al. 2022
Song al. 2023 Luo et al. 2023 Yin et al.) 2024 Zhou et al. 2024 Zhang al. (025b); and
techniques that improve GAN mode coverage using diffusion or augmentation (Zheng & Zhou
2021 Zheng et al. 2023a Wang al., 2023 Karras et al. 2020 Zhao al., (020). Similar effects
have been observed when distilling in a paired continuous space (Sahoo et al.J 2025). From this
perspective, the discrete path in CADD is naturally mode-seeking, while the continuous channel spreads probability mass to cover plausible alternatives for the next token.
3 PRELIMINARY
Let wo = (ag...... xg)) represent a sequence of discrete tokens in a vocabulary set V = 2,.,,... VJU {m] that containing V tokens plus a mask token (IMASK)), ), i.e., for any positions i, æ {0. 13V+1 is a one-hot vector. Let we V -> Rd be a learnable token embedding matrix and the embedding latent representations deterministically transformed as zg = wo(xo), and Zu E Raxd
Discrete Diffusson Models
The forward diffusion process is performed through an element-wise (r)) II= (=))) defined (8(.) denotes the dirac function):
conditional sampler (a,,)e
[ & áz) = + (1 az)ō(z) m).
(1)
where a, [0, [0,1] is a strictly decreasing scheduling function following a; = =I (1- B.). The reverse process aims to learn p(ar,|ar; for 0 S 8 < i S This is typically achieved by training model po po (z:) to predict the original data from a corrupted state, optimized by minimizing variational bound on the negative lieliikl,oood, denoting aj the derivative of ar w.r.t E:
- af
Lyh 0) -
log Po P a
Continuous Diffusion Models Continuous diffusion models corrupt real-valued data Zy E Raxd by adding Gaussian noise scheduled by 7t. The forward process q( (20) is Gaussian distribution with a closed form: ((2.|20) N215 Vizz, (1 T)
3

---

Published as a conference paper at ICLR 2026
Training
Sampling
CADD fx(x2 -x)
CADD backhone
(f {
backbone
(a.)i
N (At Scx:),
2
St V Ee
Fusion
Continuously
Contimmusll
Augmented Space
Augmenteed
Space
We (m)
M(.D)
20,0
(e (O,
diflasionn
Image
bext
Discrete Path
Embodding
Continuous Path
(Sarglo
Xx Discrete Models
Discrete Diffusion Models Xg
Discrete [MASK] Models
Discrete Diffusion Models
Figure 2: Illustrative depiction of CADD model, combining both the discrete and continuous feature of the data. In training, the clean token at the masked position will be created by embedding matrix and used to form the noisy embedding according to the continuous forward. In sampling, the model is able to predict a diverse distribution possible tokens by sampling multiple 2141- Then the predicted tokens will be recycled into the embedding matrix to form t0,0 for the next iteration.
where 7/ is noise schedule analogous to Ott, with ft Ih= Y holding. The reverse process po(2,-11 1 ( is trained by fitting a network fo(-) with MSE objective reweighted by signal-to-noise ratio (SNR) function A(7e, t):
Lvo(zo; 0) A (1,00)))) [)5r,))lffozzzt zoll"
4 CONTINUOUSLY AUGMENTED DISCRETE DIFFUSION (CADD)
Here we introduce Continuously Augmented Diserete Diffusion (CADD). The high-level intuition is mitigate the sudden information loss that occurs when tokens are replaced by absorbing state in discrete diffusion. Inspired by the smooth signal degradation in Gaussian diffusion, CADD augments the discrete state space with a continuous latent variable, 2t. This variable is paired with discrete tokens I, and is designed to retain semantics of a token's 's original signal even when tokens in 2; are (R)
the model predicts next tokens by: masked. Guided by of latent vectors (z(*) 1k=1
K po
pe (3 xx) Ez, (20 Øe, xt)]
Σ
x2, (M)
(5)
Conditioning continuous view of the underlying content at step t and traverse the 2, space, the expectation averages over plausible continuous states so the predictor could realize the distribution of the possible tokens more accurately. The full model design is illustrated in Figure and present the detailed designs in the following sections. Noted that although we use continuous-time notation and & for diffusion steps, to improve readability, diffusion process by and t - 1, with total I' steps. Below we present the construction of CADD with main derivations. For more detailed ELBO derivations and proofs, please refer to Appendix A
also denote specific consecutive steps in the
4.1 FORWARD
To let z, retain semantic hints of tokens in æ, when they are masked, we define the joint transition:
q( zt Zt-1, 21,1, = g( x1-1)
g(z
2t-1, De, 2o),
(6)
discrete part
continoous part
Given fixed discrete schedule {3, E [0, 1)T and continuous diffusion schedule (ye the forward transition of discrete and continuous part can be written as following:
q( xz-1) II
Categorical (zf; Q, x(=1), == Be) + Br 1m
-=1
8(2) 2{-1);
x + m,
[x1 11-1, 11-1 21, 21,
N(z): v 2 (1 (a),
x = m, x
* m,
(8)
N(z v/v 2/11 (1-7))
x; m, x. m.
4

---

Published as a conference paper at ICLR 2026
The discrete transition is the same normal discrete diffusion like Austin al. (2021a) and acts as a trigger for the continuous embedding s evolution. The continuous trajectory for an embedding remains dormant as long its token is unmasked, holding its value constant at its original state (6(2) = 02 =6) if æt is never masked as the information is not changed). The moment a token is masked, triggers the continuous diffusion process for its embedding. The embedding then begins a smooth degradation path determined by the Gaussian diffusion (Ho et al. (2020). If a token stays masked, its embedding simply continues along this path, becoming progressively noisier.
Now we extend the case to the marginals at timestep with the following proposition.
Proposition (Timestep-t joint marginal factorization). The marginal at timestep t can be factorized:
q 2t σo) g( xo) g(ze Tz,
Given 0g = II (1 - 3.) and Q = II-= Q. = a, + (1 (1-00 os) 1 m and let 5t. = II. 7s
with zj = (0)(2;)) the two terms factorized above represent the discrete and continuous part:
g(
mo) = II
xx))
q(>)
x
Categorical(aç': Q. x%).
(10)
i=1
g(21 Øt, zD)
II (zf
x
8(2) z(),
o zgg
(11)
vt zb: (1 (1-70) La), x) m, Niz
A key property of the marginal distribution ( _z discrete and continuous components: g(z; Zn) and {z Tt, Zp). This factorization is highly
xo) is that it conveniently factorizes into
advantageous, the distribution for each component is tractable and can be computed in closed form according to the predefined diffusion schedule.
4.2 REVERSE
Following Kingma et al. (2021); Xiao et al. (2022): ; Zhou et al. (2023), we choose the conditional distribution parameterized with neural network (-(-)
(-) to define:
pe( 7 Øt; zz) == q (f - æt. ze, 2o = æt), (
(12)
.te 21)) Categorical (logits (logitss = fo(x,, 2:) if a. m else 8(á0
x.))
(13)
The objective is to close the gap between the defined parametric distribution and the true posterior. Below we presenet the close form of the posterior. For notation simplicity, below we discuss per position formulation and omit the notation i, since all distributions factorize across positions i BE (L....,n.. n2.
Proposition (Factorization of the true posterior). By the forward construction, the posterior can be factorized in the following form
g( 21 It, Zt, mo) q( ( T 20)
(
Dt. 2t, It- 2.))
(14)
discrete part
continuous part
Moreover, we can write the close form of each component:
20
Φ1-1 * m, D. m
1104-122,,20) x0) = g( [x; 11(00
1-ate
g( (D))
1-
1-as
D = m, F. = m
(15)
11-1
2, # m.
6( - -20);
Dt Fi
(no mask t):
q( Dz, 2t, Iz- Fa)
8(21-1 =0):
Ct m, Et-1 Fy (first unmask at t),
Dt = m. Ft-1 = m,
(16)
with the following paramters:
8, = (1 - 7--1) (1 7
Vi-1 (1 7))
Voo (1 7-11) (1)
zo +
21-
(17)
1 7t
1 5t
1- 7,
5

---

Published as a conference paper at ICLR 2026
Algorithm 1 Training of CADD
Algorithm 2 Sampling of CADD
1: Input: data minibatch { B 13=1, network fa (-),
1: Input: desired minibatch size B, network schedules schouuess hi=i:= (7))-1
masking schedule {are 12 continuous schedule
2: for j = 1,..., B do init: mf) (m, m.),
(0) 1 N(
2: for j do
2T
3: draw L ~ Uniform(1,.... T) for each sample
= T. ,.... for
4: mask out each token position æ with prob-
n. a
do
ability dey to obtain al
6:
compute and (Eg (8)
determine whether keep a Cat (piii (o (m= (0)0))))))) HPP "t"")
unmask m)
5:
form embeddings dise + uo (ma 5
(5)
(
6:
9:
if G). t m then draw 2
7:
for position E {1, n), if e,E Zig (ihi + V e, (0.
1 (g do,
M pe ( z1), B. with Eq.
())
10:
else (0), + we (2) (7,7
8:
end for
11:
end if
9:
{(
<- (s)
diae + zt" compute logits (
12:
end for
optimize with cross entropy loss Eq. (20
13:
end for
11: end for
14:
end for
Lemma 1. For the unmasked positions (at F m.) the KL is identically O, and the masked positions splits exactly as
Drilg(- Øe, Zr, æo) po Wtz z1)) Pt
flip
[-
log po (2o) (wol zx)] + +
keep
DEL.
cont
(18)
discrete
continmous
with the ratio that determines whether the position is going to flipped to unmask or keep moving in the continuous space:
keep
1
04-1 at
Hip
0r- 0,-1 Ox
Pt
1 att
at
The KL divergence in the continuous space has reweighted MSE form:
ot 233
V 71)
DRL
[1. zt) (0,0,,,0 =
|| - 20.0 ar
(19)
23:
4.3 ALGORITHI AND IMPLEMENTATION
Training Loss According Eq. 18), the model aims to learn maximize the likelihood of discrete path, and also minimize the reweighted MSE in Eq. (19). Inspired by continuous diffusion models that used for categorical modeling, e.g., CDCD (Dieleman al. 2022) and Plaid (Gulrajani & Et. 2e) Wa and just train the model to
Hashimoto 2023), 3, we may estimate 20,0 En Pa (z.- predict correct categorical output to minimize the KL divergence. Thus, we choose to train CADD by minimizing simple cross entropy loss as following and the training is summarized in Algorithm
CCADD = EE
Σ =
logpo(z5) 2()]
(20)
Note that we may add the MSE loss in Eq. 19 to the above objective to more accurately estimate the exact variational lower bound. Empirically we find the simplified loss is more computationally efficient, thus we choose to use this loss for most of our experiments unless otherwise specified.
Sampling The sampling start from the last timestep T of the diffusion chain. Under the absorbing forward, ay z O. hence p(ær) = 6 i.c., all tokens are masked. Since all positions are masked at T, the continuous prior p(2T(=T) T. For each timestep, given (Ie, 2t), the network predicts
II. ((2); 0, Ia), which matches the forward marginal
1 K
(v) = K
Σ
po( = v xt, 2 (k) E A V-1
for each position

---

Published as a conference paper at ICLR 2026
If the position i is unmasked, the absorbing chain keeps xi-1 æ almost surely and the continuous variable is deterministic 27-1 z (o(x) If this position is masked, it draws clean token v (+) with probability - to unmask it, and the continuous latent zi-1 + Wa,e- remains
unmasked, zi-1 moves along the continuous diffusion trajectory z|-1 ~M(in (i(ir(( (26.0: =i), 8 BTTa).
The full sampling process in shown in Algorithm 2 Note the choice of $0.0 has two options:
hard: to = arg max (T,)(T), &o- - 200=wa@aa wo) soft: 20.9 == Σ po = v wt. zi) wg.u (21)
These two choices are both valid to use depending on whether we are looking for mode-covering or mode-seeking behavior, i.e., better context localization or better diversity, respectively. In our main experiments we keep the hard option, and our empirical exploration in Appendix [C.3 justify these two choices could meet the demand of these two behavior. Moreover, although CADD may leverage multi-sample for the 2g distribution estimation, for fair comparison with baselines, we keep K = for most of our experiments. More detailed studies are also shown in the Appendix
Implementation We follow the common-used design of the model architecture to let fo(-) predict logits for categorical distribution. The discrete path follows earlier masked-dae diffedddiffusion setups: starting from 2o, mask subset of positions to obtain It. embed the mixed sequence with the learnable table and form Adise = we w (z)). The only difference is the model needs to take an additional variable z₁ input for the continuous embeddings. To achieve this, first form the clean embeddings zo = Wy (Zu). and then apply noise only at masked positions using the forward marginal Eq. (11) to obtain z₁- We fuse zdisc: and 2, by element-wise addition ž, Zdisc + 2t, and feed ž, to the backbone So to produce per-position logits.
5 EXPERIMENTS
In this section we present experiments to validate the proposed CADD model through experiments on text, image, and code generation benchmarks. The evaluations are designed to assess the model's performance across diverse data modalities and scales.
5.1 TEXT GENERATION
Experiment setting For text generation, we strictly follow the experimental setup of the Masked Diffusion Language Model (MDLM) (Sahoo et al. 2024), a common configuration for this task. We train our CADD models the OpenWebTexx (OWT) dataset (Gokaslan & Cohen 2019). Data is tokenized using the GPT-2 tokenizer with a vocabulary size of [ = 50, 257 (Radford et al. 2019). and sequences Discrete DiT backbone (Peebles & Xie 2023) with approximately 168M parameters, and train with same number of iterations. All training typer-parameters are identical to those in MDLM.
fixed length of 1024. To be consistent with the baselines,
use
Evaluation. We mainly compare the performance with discrete diffusion baselines in terms of the generative quality, and our evaluation protocol strictly follows that of Wang et al. (2025). We compare the performance against discrete diffusion baselines using two metrics: the MAUVE score (higher is
better) (Liu et al. 2021 Pillutla et al. 202T) and generative perplexity (lower is better) (Lou et al.
2024
Further details the evaluation setup located in Appendix B
Main Results Figure 3 presents the results for unconditional text generation on the OpenWeeTees Text (OWT) dataset, comparing CADD with SEDD (absorb) and MDLM across a range of sampling steps T E {128, 256, 512, 1024, 4096). Within the range T < 1024. all models shows improvement as the number of sampling steps increases. We can notice CADD demonstrates stronger and consistent gains steps increase compared to SEDD and MDLM in terms of both metrics. Plotting the x-axis on a log, scale reveals that the performance trend is approximately linear.
Extending the sampling process to T 4096 further demonstrates CADD's scaling capabilities at inference time, as it continues to improve while the masked-only baselines stagnate or degrade. From T 1024 to 4096, CADD's MAUVE score still increases by 0.3, and its generative perplexity
7

---

Published as a conference paper at ICLR 2026
0.2m0
115.2
8:440
118.1
(9733
0.25
100
20
MAUVE
3.888
0420
[3:07)
3.130
Gen PPL
192
0.10
3.663
03331
0.835
0.307
F15
100
40.
51.3
06.3
126
258
512
1024
4098
126
255
512
1084
4095
Steps
Steps
(a) MAUVE
(b) Generative perplexity
Figure 3: Unconditional text generative evaluation of model trained on Open OpenWebText (OWT) data. All method are evaluated with 128, 256, 512 1024, and 4096 sampling steps. MAUVE (higher is better) and generative perplexity (measured using GPT2-LLagee lower is better) are reported.
Table 1: FID and IS evaluation on CIFAR-10. The arrow symbols denotes lower/higher is better respectively. Baseline results are quoted from Chao et al. (025).
Table 2: FID evaluation using model uncondi- tionally trained on ImageNet (32 x 32 resolu- tion).
Method
FID (4)
Method
FID (1)
IS (1)
CADD (NFF==10024)
3.74
CADD (NFE=512)
2.88
10.04
Discrete
Discrete
MDM (NFE=1,024)
7.91
MDM (NFE=512)
4.66
9.09
MDM-Mixture (NFE=1,024)
8.08
MDM-Mixture (NFE=512)
4.80
9.22
HDDM-Prime (NFE=1,024)
6.98
MDM-Prime (NFE=512)
3.26
9.67
Continuous
D3PM Absorb (NFE=1,000)
30.97
6.78
NDM
17.02
D3PM Gauss. (NFF==10000)
7.34
8.56
DDPM
16.18
CTDD-DG (NFF=11000)
7.86
8.91
MSGAN
12.30
Tau-LDR (NFF=1,000)
3.74
9.49
i-DODE (SP)
10.31
Discrete FM (NFE=1,024)
3.63
i-DODE (VP)
9.09
Continuous
Stochastic Interp.
8.49
Continuous
6.35
Soft Trunc. DDPM
8.42
Bit Diffusion
3.48
ScoreFlow (subVP)
8.87
StyleGAN+ADA DDPM
3.26
9.74
ScoreFlow (VP)
8.34
3.17
9.46
Continuous FM
5.02
is scored from 44.6 to 35.3. MDLM's performance slightly worsens, which is consistent with the
observation that mask-only diffusion models scale poorly with T (Wang et al. 2025). Overall, CADD consistently show performance gain across all tested number of sampling steps over the mask-only discrete diffusion models, validating the effectiveness the proposed contimuous-augnmented space.
5.2 IMAGE GENERATION
We train and evaluate our models on the CIFAR-10 (Krizhevsky et al. 2009) and Ima- geNet (Krizhevsky al., (017)) datasets (resolution 32 32). For both, input images in RGB channels, thus a dimensionality of n = 32 x 32 x 3 with [V] = 256 pixel values per channel. For fair comparison the MDM baselines, our model architecture follows the one used in Chao al. (2025): ;
Gat et al. 20244). which is based on the ADM (Dhariwal & Nichol 2021) architecture. We choose MDM-Prime (Chao al., 2025 and its variants as our main discrete diffusion baseline. We also
include its discrete and continuous diffusion model baselines for comparison (Shih et al.] 2022 Ho
al.
2020 Song et al., 202T Austin et al., 202 Camphell et 2022 Gat et al. al. 2024 Nisonoff
ct al.
2025 Lipman et al. 2002 Chen al. 2023 Bartosh et al. 2023 Tran et al. 2019 Zheng
ct al.
2023b Albergo & Vanden- Eijudenn 2023 Kim et al.) 2022). To assess sample quality, we
report Fréchet Inception Distance (FID) and Inception Score (IS), computed with 50,000 randomly
sampled images.
We follow MDM variants to unconditionally sample images with same number of function evaluation (NFE) and report results on CIFAR-10 10 Table [ With the same NFE, we can observe CADD improves upon MDMs by significant margin. Attaining FID of 2.88 and an Inception Score of
8

---

Published as a conference paper at ICLR 2026
Table 3: Benchmark coding capacities of AR and Diffusion LLMs in 7/8B scale. We follow the evaluation settings in DiffuCoder (Gong et al. 2025b), where EvalPlus is computed as the average of HE+ and MBPP+. The best performance in AR and Diffusion LLMs are marked in bold.
HumanEval
MBPP
BigCodeBench (C) Avg.
Model
EvalPlus
Plus
Plus
Full
Hard
AR
Qwen2.5-Cor 5-Coder
61.6
51.8
75.9
61.4
56.6
46.1
16.2
52.2
OpenCoder (Huang et al.
2024
66.5
63.4
79.9
70.4
66.9
40.5
9.5
55.0
Diffusion
LLaDA (Nie et al. 2025
35.4
30.5
50.1
42.1
36.3
18.9
4.1
30.2
Dream
Ye et al 2025
56.7
50.0
68.7
57.4
53.7
23.6
4.1
43,4
DiffuCoder
67.1
60.4
74.2
60.9
60.7
40.2
12.8
52.6
CADD (ours) CADD (ours, DiffuCoder init)
72.0
63.4
75.7
63.2
63.3
42.1
17.6
55.7
73.8
64.6
73.9
60.4
62.5
41.5
15.5
55.0
10.04 with 512 function evaluations (NFE), CADD surpasses the MDM variants by 0.38 in terms of FID and represents the best result among all compared method. On ImageNet-32, as shown in Table the observation is constent, where CADD obtains FID of 3.74 and outperforms all reported baselines. The qualitative generated samples are provided in Appendix D foro visual justifications.
5.3 CODE GENERATION
For a large-scale setting, we conduct code generation experiments based on the DiffuCoder pipeline (Gong et al. 2025b). The DiffuCoder base model training process involves adapting pretrained autoregressive LLM (e.g., Qwen2.5-coder (Hui et al. 2024)) into a discrete diffusion model by anncaling its attention mechanism from causal to bidirectional (Gong et al. 2025a). The resulting model is then trained using a masking diffusion loss (Shi et al., 2024). In this context, evaluate method using the following two distinct configurations. (i) Vanilla CADD: We follow the DiffuCoder procedure to adapt the Qwen2.5-coder model. Instead of using the MDM loss, we train the model from the beginning with our proposed CADD loss. (ii) CADD (fine-tuned): To demonstrate CADD's effectiveness as fine-tuning objective, we initialize our model from a pretrained DiffuCoder checkpoint and then continue training with the CADD loss. To ensure fair comparison, both CADD variants are trained on the same 65B total tokens and use the same training hyperparameters the original DiffuCoder. In the evaluation, we follow their settings test the model performance on three coding benchmarks: HumanEval (Chen et al. 2021). MBPP (Austin al. 202Tb), and BigCodeBench (Zhuo et al., 2024).
Table 3 reports the pass@1 performance, where the results of both autoregressive (AR) and diffusion- based LLMs are included, with an overall average score provided. Compared with Diffusion- based models, CADD emerges the strongest diffusion model, outperforming competitors nearly all metrics. Compared to the previous leading DM, DiffuCoder, CADD significantly improves performance HumanEval, e.g., from 67.1 to 72.0; on the challenging BigCodeBench- Hard subset, we can also observe significant performance gain from 12.8 to 17.6. CADD is also highly competitive with leading AR code models. surpasses Qwen2.5-Coder across all benchmarks and achieves higher overall average than OpenCoder (55.7 vs. 55.0).
5.4 ABLATION STUDIES
On the choice of fusion and وn estimation In most our experiments, we choose to fuse the discrete mask token embedding and continuous embedding with addition operation, i.c., 2,
Adise + Zt. We consider two extra manners to fuse these two domains: 1) concatenation [Adise =,); 2) reweighted sum 0f (1 2t, where ag decreases the position is more likely to be clean (unmasked). The intuition that when a token is unlikely to masked, the model should lean more on 2, to carry semantic content, hence smaller az,.
Observing the results in Table MAUVE varies by only 0.03 absolute and Entropy varies by 0.07 absolute across the different choices. These three options do not show significant difference the

---

Published as a conference paper at ICLR 2026
Table Performance vs. fusion method for žt
Table 6: Performance vs, training and estima- tion method for 20
Fusion
MAUVE (t)
Entropy (1)
Add
0.24
5.31
Metric
CE
CE +
MSE
Concate
0.21
5.37
hard 2o
soff 2g
Roweight
0.24
5.30
MAUVE
Table 5: Performance vs. continuous schedules
hard &0
0.24
0.24
0.22
soft 20
0.18
0.24
0.24
Metrics
FM
VP
computation cost
TFLOPS 0.291
0.325
0.291
MAUVE
0.24
0.24
TPS/GPU 47,152
32,117
47,152
FiD
2.88
3.16
performance, while concatenation involves an additional projection layer to match the embedding dimension.
Morever, we compare the choice of Eμ estimation, as discussed in Eq. 21
hard: un = arg max TT.; ; &g w2) soft: 20,0 = Σ po ( Ft, 2t) w0
From Table 6. hard estimation gives higher MAUVE (+0.06) and slightly lower Entropy (-0.11), indicating this choice is mode-seeking-orieneee where the context is localized faster. The soft estimation encounter shows higher entropy, meaning that the model reveals mode-covering behavior and it pursue better diversity for generation. The properties of these choices are justified. We consider both options are valid for the sampling, depending on which properties we are looking into in practical case.
On the choice training objective To further justify whether we should use MSE to optimize at the embedding level in the categorical generative modeling scenario. We add training results that include an extra MSE loss | - zo using the two parameterizations of &. (soft and hard) defined in Eq. (1). From the results, CE + MSE provides performance that is close to using CE alone. The soft parameterization gives a gain under hard-inference MAUVE, but introduces higher computation cost. This is because the soft prediction requires matrix multiplication between the predicted probability vector fxxd and the token embedding matrix Rd with batch size B, sequence length I., embedding dimension and vocabulary size V. Such extra cost reduces TPS/GPU and increases TFLOPPS. A possible explanation for the limited improvement that, in the categorical setting, an MSE loss behaves similarly cross entropy since both losses guide the model toward selecting the correct token and its embedding. We expect MSE to be more useful in settings where the targets not purely categorical.
On the choice of continuous schedule We compare the Mariance-Preservve (VP) schedule with ftow-matching (FM) schedule in both text and image generation experiments. The results are shown in Table For text generation, the results are on par with the Flow-mate Flow-matching schedule. For image generation, the FiD score is slightly worse under the VP schedule. This difference may come from the need for different hyper-parane parameter settings for the two schedules.
CONCLUSION
In standard discrete diffusion, information is lost abruptly when tokens are replaced by absorbing state. Inspired by Gaussian diffusion, where the data signal degrades smoothly, CADD's core idea is to introduce auxiliary continuous space to guide the discrete process. This space is designed to retain semantic information, providing a smooth continuous representation of a token even after its discrete form has been absorbed. By conditioning on it, the model can better "remember" what was supposed to be in the masked position. This leads to more coherent and contextually accurate generations, the model has stronger grasp of the underlying meaning. With extensive empirical justification on text, image and code generation, we justify that with the continuous augmented space proposed in CADD, the discrete diffusion models consistently generate higher quality samples across these different tasks and achieve strong performance.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
The authors thank Josh Susskind, Irina Belousova, Miguel Angel Bautista, Richard Bai, Shuangfei Zhai, Tatiana Likhomanenko, Xiaoming Zhao, Yuyang Wang and Zijin Gu for insightful feedbacks and discussions.
ETHICS STATEMENT
This research proposes Continuously Augmented Discrete Diffusion model, for discrete data gener- ation, with possible impacts on misusage to generate toxic information. Since our research scope on the fundamental machine learning algorithm, we use all public datasets, which do not contain personal and sensitive information. In our paper, all generated results have been checked and do not contain misleading and malicious information.
REPRODUCIBILITY STATEMENT
The authors are committed to the principle of reproducibility and have made every effort to ensure theoretical and experimental results can be reproduced. We confirm the variables used for the derivations are well defined and the claims are provided with proofs, which are attached in Appendix We have introduced our methods with precise math tools and visual aids, such as Figure 1 Figure 2 and Algorithm box 2 The datasets we used are all public and widely-used. The training and inference details are deseribed Appendix including data pre- pre-proeessing, training hyper-paramettes, and inference pipeline. The source-code of our model will be released upon acceptance.
THE USE OF LARGE LANGUAGE MODELS (LLMS)
We acknowledge that LLMs are used to only polish the presentation and writing of the paper. The generated sentences are double checked and rephrased by the authors.
REFERENCES
Michael S. Albergo and Eric Vanden-Eijnden. Building Normalizing Flows with Stochastic Inter- polants. In Proc. Int. Conf. on Learning Representations (ICLR), 2023.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. In Advances in Neural Information Processing Systems, 2021a.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. ArXiv preprint, 04422008007332. 2021b. URL BRLhhttps: http:s///arvv..e.. org/ abs 2108
07732
Pavel Avdeyev, Chenlai Shi, Yuhao Tan, Kseniia Dudnyk, and Jian Zhou. Dirichlet diffusion score model for biological sequence generation. In International Conference Machine Learning, 2023.
Grigory Bartosh, Dmitry Vetrov, and Christian A Naesseth. Neural diffusion models. arXiv preprint arXiv:2310.08337. 2023.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, et al. Video generation models as world simulators, 2024.
Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth. George Deligiannidis, and Amaud Doucet. continuous time framework for discrete denoising models. In Advances in Neural Information Processing Systems, 2022.
11

---

Published as a conference paper at ICLR 2026
Chen-Hao Chao, Wei-Fang Sun, Hanwen Liang annd Chun Yi Lee, and Rahul G. Krishnan. Beyond Masked and Unmasked: Discrete Diffusion Models via Partial Masking. Proceedings of the Conference Neural Information Processing Systems (NeurIPS), 2025.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. ArXiv preprint, 0/b///100.03374 2021. URL BRIlhttss: /tte://arxxi
orrg/bb/ 2107 03374
Ting Chen, Ruixiang Zhang. and Geoffrey Hinton. Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning. In ICLR, 2023.
Chaoran Cheng, Jiahan Li, Jian Peng, and Ge Liu. Categorical flow matching statistical manifolds. In Advances Neural Information Processing Systems, 2024.
Oscar Davis, Samuel Kessler, Mircea Petrache, ismail Îlkan Ceylan, Michael M. Bronstein, and Avishek Joey Bose. Fisher flow matching for generative modeling discrete data. In Advances in Neural Information Processing Systems, 2024.
Justin Deschenaux and Caglar Gulcehre. Beyond Autoregression: Fast LLMs via Self-Distillation Through Time. In Proceedings of the International Conference on Learning Representations (ICLR), 2025.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans image synthesis. In Advances in Neural Information Processing Systems, 2021.
Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, et al. Continuous diffusion for categorical data. arXiv:221 15089, 2022.
Patrick Esser, Sumith Kulal, Andreas Blattmann. Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik I Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In International Conference on Machine Learning, 2024.
Zhujin Gao, Junliang Guo, Xu Tan, Yongxin Zhu, Fang Zhang, Jiang Bian, and Linli Xu. Empowering diffusion models on the embedding space for text generation. arXiv preprint arXiv:2212.09411 2022.
Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky T. Q. Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching. In Advances in Neural Information Processing Systems, 2024.
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http://9kyllinn07..gg OpentNebTextCorpus 2019.
io/
Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models. In The Eleventh International Conference on Learning Representations. ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
URL https:://ppenreeeee
net wwtee/pp.... pddeiddian j0j- rLVXs
Shansan Gong, Shivam Agarwal, Yizhe Zhang. Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, Hao Peng, and Lingpeng Kong. Scaling diffusion language models via adaptation from autoregressive models. In The Thirteenth International Conference Learning Representations, 2025a.
Shansan Gong, Ruixiang Zhang. Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong, and Yizhe Zhang. Diffucoder: Understanding and improving masked diffusion models for code generation. arXiv preprint urXiv:2556.20639 2025b. URL. https:: //tpx//aiivv.a////////// www.vvoorggaaao aba/
2506. 20639
Alex Graves, Rupesh Kumar Srivastava, Timothy Atkinson, and Faustino Gomez. Bayesian flow networks. arXiv:2308.07037, 2023.
12

---

Published as a conference paper at ICLR 2026
Jiatao Gu, Tianrong Chen, David Berthelot, Huangjie Zheng, Yuyang Wang, Ruixiang Zhang, Laurent Dinh, Miguel Angel Bautista, Josh Susskind, and Shuangfei Zhai. Starflow: Scaling latent normalizing flows for high-resolution image synthesis. arXiv preprint arX.v:2506.06276 2025.
Ishaan Gulrajani and Tatsunori B. Hashimoto. Likelihood-based diffusion language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA. December 10 16, 2023, 2023. URL http: . nips CC ww/paper fiwes/ fates//apeeeee/3//////////////// / httm html
Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. Ssd-Im: Semi-autoregrresivve simplex- based diffusion language model for text generation and modular control. arXiv:2210. 17432, 2022.
Marton Havasi, Brian Karrer, Itai Gat, and Ricky TQ Chen. Edit flows: Flow matching with edit operations. arXiv preprint arXiv:2506.0900 09018, 2025.
Zhengfu Hc, Tianxiang Sun, Qiong Tang, Kuanning Wang. Xuanjing Huang, and Xipeng Qiu. Diffusionbert: Improving generative masked language models with diffusion models. In Annual Meeting of the Association for Computational Linguistics, 2023.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXhv:2207.12559, 20222
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. In Proc. of Int. Conf. on Neural Information Processing Systems (NeurIPS), 2020.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. In Advances Neural Information Processing Systems, 2021.
Siming Huang, Tianhao Cheng, Jason Klein Liu, Jiaran Hao, Liuyihan Song, Yang Xu, Yang, JH Liu, Chenchen Zhang, Linzheng Chai, et al. Opencoder: The open cookbook for top-tier code large language models. ArXiv preprint, abs/241 0122411..4995, .04905, 2024. URL pattps://arxiiai org/
2os/2411.04905
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang. Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, al. Qwen2. 5-coder technical report. ArXiv preprint, abs/2409. 12186, 2024. URL hhttss:: / arxiv. www.xxwwrrg/bb///49...oo 12186
Jaehyeong Jo and Sung Ju Hwang. Generative modeling on manifolds through mixture of riemannian diffusion processes. In International Conference Machine Learning, 2024.
Jaehyeong Jo and Sung Ju Hwang. Continuous diffusion model for language modeling. In Advances in Neural Information Processing Systems, 2025.
Rabeeh Karimi Mahabadi, Hamish Ivison, Jaesung Tae, James Henderson, Iz Beltagy, Matthew Peters, and Arman Cohan. TESS: Text-to-text self-conditioned simplex diffusion. In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 2347-2361, St. Julian's 's Malta, March 2024. Association for Computational Linguistics. doi: 10.18663/vv12022 eacl-long.14 144. URL htt.https:. /aw.//aaaaahhooac.oo www.olllonoooooooooooooooooooooe ogy.
wae.4eeaclllee long. 1..11444 144/
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. arXiv preprint arXiv:2006.06670 2020.
Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Proc. NeurIPS, 2022.
Dongjun Kim, Seungjae Shin, Kyungwoo Song. Wanmo Kang, and Il-Chul Moon. Soft Truncation: A Universal Training Technique of Score-based Diffusion Model for High Precision Score Estimation. In Proc. Int. Conf. on Machine Learning (ICML), 2022.
Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. arXiv preprint arXiv:2 107.00630. 2021.
13

---

Published as a conference paper at ICLR 2026
Alex Krizhevsky, llya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. Communications of the ACM, 60(6):84-999)) 90, 2017.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
Shufan Li, Konstantinos Kallidromitis,, Hritik Bansal, Akash Gokul, Yusuke Kato, Kazuki Kozuka, Jason Kuen, Zhe Lin, Kai-Wei Chang, and Aditya Grover. Lavida: large diffusion language model for multimodal understanding. ArXiv preprint, abs/2505.16839 2025. URL https?
16839
Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsumori Hashimoto. Diffusion-lm improves controllable text generation. In Advances in Neural Information Processing Systems, 2022.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint 22X0:22200002747, 2022.
Guan-Horng Liu, Tianrong Chen, Evangelos Theodorou, and Molei Tao. Mirror diffusion models for constrained and watermarked generation. Advances in Neural Information Processing Systems, 36: 2023.
Lang Liu, Krishna Pillutla, Sean Welleck, Sewoong Oh, Yejin Choi, and Zaid Harchaoui. Divergence frontiers for generative models: Sample complexity, quantization effects, and frontier integrals. Advances in Neural Information Processing Systems, 34: 19930-19424, 2021.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion language modeling by estimating the ratios of the data distribution. In International Conference on Machine Leurning, 2024.
Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang. Diff- Instruct: A universal approach for transferring knowledge from pre-trained diffusion models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https? //openrevew. net fwrmm/orrmmmmmmmm iidii 5
Shen Nie, Fengqi Zhu, Chao Du Tianyu Pang, Qian Liu, Guangtao Zeng, Min Lin, and Chongxuan Li. Scaling up masked diffusion models on text. ArXiv preprint, abs/2410.185144 2024. URL
https:/ /tapps://rxxv...//// org/ /tlo://410.1111
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large Language Diffusion Models. arXiv: 2502. 09992
[cs.CL]. 2025.
Hunter Nisonoff, Junhao Xiong, Stephan Allenspach, and Jennifer Listgarten. Unlocking guidance for discrete state-space diffusion and flow models. In The Thirteenth International Confer- ence on Learning Representations, 2025. URL XsgH154y07]
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. Mauve: Measuring the gap between neural text and human text using divergence frontiers. Advances in Neural Information Processing Systems, 34:4816-4822 2021.
Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching- Yao Chuang, David Yan. Dhruv Choudhary. Dingkang Wang, Geet Sethi, Guan Pang. Haoyu Ma, Ishan Misra, Hou, Jialiang Wang. Kiran Jagadeesh, Kunpeng Li, Luxin Zhang, Mannat Singh, Mary Williamson, Matt Le, Matthew Yu, Mitesh Kumar Singh, Peizhao Zhang, Peter Vajda, Quentin Duval, Rohit Girdhar, Roshan Sumbaly, Sai Saketh Rambhatla, Sam S. Tsai, Samaneh Azadi, Samyak Datta, Sanyuan Chen, Sean Bell, Sharadh Ramaswamy, Shelly Sheynin, Siddharth Bhattacharya, Simran Motwani, Tao Xu, Tianhe Li, Tingbo Hou, Wei-Ning Hsu, Xi Yin, Xiaoliang Dai, Yaniv Taigman, Yaqiao Luo, Yen-Cheng Liu. Yi-Chiao Wu, Yue Zhao, Yuval Kirstain, Zecheng He, Zijian He, Albert Pumarola, Ali K. Thabet, Artsiom Sanakoyeu, Arun Mallya, Baishan Guo, Boris Araya, Breena Kerr, Carleigh
14

---

Published as a conference paper at ICLR 2026
Wood, Ce Liu, Cen Peng, Dmitry Vengertsev, Edgar Schönfeld, Elliot Blanchard, Felix Juefei-Xr Xu, Fraylie Nord, Jeff Liang, John Hoffman, Jonas Kohler, Kaolin Fire, Karthik Sivakumar, Lawrence Chen, Licheng Yu, Luya Gao, Markos Georgopoulos, Rashel Moritz, Sara K. Sampson, Shikai Li, Simone Parmeggiani, Steve Fine, Tara Fowler, Vladan Petrovic, and Yuming Du. Movie gen: A cast of media foundation models. arXiv:2410. / 13720, 2024.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall, Dreamfusion: Text-to-3d using 2d diffusion, arXiv preprint arXiv:2209. 14988, 2022.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8)9, 2019.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kam- yar Seyed Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In Advances in Neural Information Processing Systems, 2022.
Subham Sekhar Sahoo, Marianne Arriola, Aaron Gokaslan, Edgar Mariano Marroquin, Alexander M Rush, Yair Schiff, Justin T Chiu, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. In Advances in Neural Information Processing Systems, 2024.
Subham Sekhar Sahoo, Justin Deschenaux, Aaron Gokaslan, Guanghan Wang, Justin T Chiu, and Volodymyr Kuleshov. The diffusion duality. In Forty-second International Conference on Machine Learning, 2025. URL htt.ttts:: httttps://openreviieee net
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K Titsias. Simplified and generalized masked diffusion for discrete data. In Advances in Neural Information Processing Systems, 2024.
Andy Shih, Dorsa Sadigh, and Stefano Ermon. Training and inference on any-order autoregressive models the right way. In Advances in Neural Information Processing Systems, 2022.
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. In Proc. Int. Conf. on Machine Learning (ICML), 2015.
Yang Song, Jascha
Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021. URL TRI.https: /ttp:://ppppeeeeeeeee net/ forum?
Idd-xXIGG2RRHS
Yang Song, Prafulla Dhariwal, Mark Chen, and llya Sutskever. Consistency models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 3221 1-2252.. PMLR, 23-29 Jul 2023. URL httpps::/ppoceeci.nn...
www.:mmmr...ssss.mmmmm prwsspessseeeeeeee 20w.:/22oonn2..naioommm html
mlr
Yuxuan Song, Zhe Zhang, Yu Pei, Jingjing Gong, Qiying Yu, Zheng Zhang, Mingxuan Wang, Hao Zhou, Jingjing Liu, and Wei-Ying Ma. Shortlisting model: A streamlined simplexdiffusion for discrete variable generation. arXiv preprint urXiv:2508. /7345, 2025a.
Yuxuan Song, Zheng Zhang, Cheng Luo, Pengyang Gao, Fan Xia, Hao Luo, Zheng Li, Yuehang Yang, Hongli Yu, Xingwei Qu, et al. Seed diffusion: A large-scale diffusion language model with high-speed inference. arXiv preprint arXiv:2508.0211 12/ 2025b.
Hannes Stärk, Bowen Jing, Chenyu Wang, Gabriele Corso, Bonnie Berger, Regina Barzilay, and Tommi S. Jaakkola. Dirichlet flow matching with applications DNA sequence design. In International Conference on Machine Learning. 2024.
Jaesung Tae, Hamish Ivison, Sachin Kumar, and Arman Cohan. TESS 2: A large-scale generalist diffu- sion language model. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 21171-21 1888, Vienna, Austria. July 2025. Association 06653//112022 1/2025 acc-llong long. 1029. hit.ppss//aa.aattttooooooooo.o.. long 10297
for Computational Linguistics. ISBN 979-8-89117-2511 doi: 10.18653/v
URL httpthtt////acc aathol wwt.hhhloly..oo//22222..
15

---

Published as a conference paper at ICLR 2026
Ngoc-Trung Tran, Viet-Hung Tran, Ngoc-Bao Nguyen, Linxiao Yang, and Ngai-Man Cheung, Self- supervised GAN: Analysis and Improvement with Multi-class Minimax Game In Proc. of Int. Conf. on Neural Information Processing Systems (NeurIPS). 2019.
Guanghan Wang, Yair Schiff, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Remasking Discrete Diffusion Models with Inference- Scaling. arXiv:2503 2500300007 [cs LLG, 2025.
Zhendong Wang, Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Diffusion- GAN: Training GANs with diffusion. In The Eleventh International Conference on Learning Reprexentations, 2023. URL http://ppeeevvieeeee net/ forum wwerrumidddzze F 7UbpWHuA
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion GANs. International Conference on Learning Representations, 2022. URL
net fort/foormmill Deer:/Eoruum110---pp000000--0000
Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang. Mmada: Multimodal large diffusion language models. ArXiv preprint, abs/2505. 15809, 2025. URL
https://arxxi
org/ abs/
Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang. Zhenguo Li, and Lingpeng Kong. Dream 7b: Diffusion large language models. arXiv preprint arXiv:2508. 15487, 2025.
Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6613-6623, 2024.
Shuangfei Zhai, Ruixiang Zhang, Preetum Nakkiran, David Berthelot, Jiatao Gu, Huangjie Zheng, Tianrong Chen, Miguel Ángel Bautista, Navdeep Jaitly, and Joshua M Susskind. Normalizing flows are capable generative models. In Forty-second International Conference on Machine Learning. 2025.
Ruixiang Zhang, Shuangfei Zhai, Jiatao Gu, Yizhe Zhang, Huangjic Zheng, Tianrong Chen, Miguel Angel Bautista, Josh Susskind, and Navdcep Jaitly. Flexible language modeling in continuous space with transformer-based autoregressive flows. arXiv preprint arXiv:2507.00425 2025a.
Ruixiang Zhang, Shuangfei Zhai, Yizhe Zhang. James Thomton,, Zijing Ou, Joshua M Susskind, and Navdeep Jaitly. Target concrete score matching: A holistic framework for discrete diffusion. In Forty-second International Conference on Machine Learning, 2025b.
Yizhe Zhang, Jiatao Gu, Zhuofeng Wu, Shuangfei Zhai, Joshua Susskind, and Navdeep Jaitly. Planner: Generating diversified paragraph via latent language diffusion model. Advances in Neural Information Processing Systems, 05580078-80199 2023.
Shengyu Zhao, Zhijian Liu, Ji Lin, Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. Conference on Neural Information Processing Systems (NeurIPS), 2020.
Yixiu Zhao, Jiaxin Shi, Feng Chen, Shaul Druckmann, Lester Mackey, and Scott Linderman. Informed Correctors for Discrete Diffusion Models. arXiv: 2407 21243
2024.
Huangjie Zheng and Mingyuan Zhou. Exploiting chain rule and Bayes' theorem to compare prob- ability distributions. Advances in Neural Information Processing Systems, 34:1 1500 2021.
Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Truncated diffusion prob- abilistic models and diffusion-based adversarial auto-encoders. In The Eleventh International Conference on Learning Representations, 2023a. ld HDxgakk9561
Huangjie Zheng, Zhendong Wang, Jianbo Yuan, Guanghan Ning, Pengcheng He, Quanzeng You, Hongxia Yang, and Mingyuan Zhou. Learning stackable and skippable LEGO bricks for efficient, reconfigurable. and variable-resolutie diffusion modeling. In The Twelfth International Confer- ence on Learning Representations, 2024a. URL ColLhttps:/// StEDp:://penzeview net/forum?? forum?id-
gmmedvwrTT
16

---

Published as a conference paper at ICLR 2026
Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. Improved Techniques for Maximum Likelihood Estimation for Diffusion ODEs. In Proc. Int. Conf. on Machine Learning (ICML), 2023b.
Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. A reparameterized discrete diffusion model for text generation. In Conferenec Language Modeling, COLM, October 7-9, 2024, Philadelphia, PA, 2024b.
Mingyuan Zhou, Tianqi Chen, Zhendong Wang. and Huangjie Zheng. Beta diffusion. Advances in Neural Information Processing Systems, 0-:30007-30092 2023.
Mingyuan Zhou, Huangjie Zheng, Zhendong Wang. Mingzhang Yin, and Hai Huang. Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. In Forty-first International Conference on Machine Learning, 2024. URL httpttttttt/eeeeeeeeeees net/Eorum? (tw.oorummmddohuuuueeeeeeeee
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari. Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. ArXiv preprint, tbss2400.118777 2024. URLbttps : // arxiv wwg.orgg////..... abs 2406. 15877
17

---

Published as a conference paper at ICLR 2026
A DETAILED DERIVATIONS AND PROOF
A.1 ELBO DERIVATION
Forward chain. For any observation æo, the forward diffusion constructs
T
g Z1:T zu) = II
že
Zt-1. -1 To).
(22)
t=1
note we represent (zo, zo) as 2o since the transform wy is deterministic. Reverse generative model.
(a 11.T, (1.T, #1:T) = pr 2T)
[I t=2
pe xt- 1t-1, 11 Ft-1 xe, 2:)
po(20o x1,
(23)
Proposition 3 (ELBO decomposition). Given the forward chain defined in Eq. model py in Eq. 23 we have the decomposed ELBO following:
22 and reverse
log Pa (2o)
E (g(0n.21100)) log pa (go 3 Z1)
reconstraction term at t=
E gee *e (ea)
D 21-1 Bt, Zt. Do) po ze-1 xt, 21))
denoising maiches for t>
DRI(g(aT, 25 vD) pr (TTT Zr))
(24)
prior match T
If q(Qr, zr xo) = (xT, zT) for all Zo. then the prior match term is zero. The bound is tight if and only
p0 R 3t; 2+) = g( Zt- at, ze, wo) for all t 2 2,
and the prior match is zero, and the decoder po( (zg 01, 21) equals the true conditional induced by the joint.
Recap the forward kernel defined in Eq. 7 and Eq.
g(
xt-1)
II
Categorical z Q, Q(x(-1); Q: (1-)B))++ 3 +
i=1
8(2) 2(=1),
x +
q(zx 21-1,2,-
N(2 2 (1-
x m, a # m,
M(2 TT 2|-1 (1 Ia), x m, x m.
Proof of Proposition The proof is mostly done in SShh--ickkteei et al. (2015) and Ho et (2020). We include the following proof to show the generalized version with added variables. Start from the evidence identity and apply Jensen inequality:
log (x = log / q Z1.T xp)
Po (ap: Ti.7: Zi.))
dx dzyr
( Zi. 20)
2 E mo) log pa(oo æ1.r, =1.7) log g Z1.T xo)
=: L(0 zn)-
(25)
Insert the model and forward factorizations Eq. (23 and Eq. 22
C(0: xo) = By log (xT, 21) +
log po (x)-1. 1z, z1)
(26)
log (x æ1, 21)
log g( 21 Ie-1, 21- 200
(27)
18

---

Published as a conference paper at ICLR 2026
For each f 2 use Bayes' rule under q:
g( 21-1 To) = 11 21 2e, Rz, 20, g( 21 xo)
(28)
9(2x--1, 2 2p.)
Taking B [og(-)] of Eq. 28 and rearranging gives, for t 2 2,
Ey log pa (T, - Fe. 2x) - log ( 2z zz- - mo)
-- ( DKI (, Z 0t, Rt: wo) po 1,
0t, 2:)
Eg log g (ze, Zt mo0] Egl log g - zn)]_
(29)
Sum Eq. 29 over 2, ..... T. The last two expectations telescope:
Σ Eg log (22, #t a.)] Σ Ey log 1 2 x0)
E log ( ø
=o)] -E,, log q(TT, 2r mo))
(30)
Plug this back into Eq. 27 and group the boundary terms with log pr:
L(O; a0))
By log po @o æ1, 21)]
Σ 22 (m0)
DKI(g( (2 Be Zt. 2o) (o 21-1 0e, 21))
[E,[ log ( 2τ æo)] [ogg
- E log q( 21 xo)]-
(31)
Now we recoginize the prior KL obtain
L(0; mo)
Rg log (2 21)]
Σ E g mm)
(xulg(z1-1,22-1
Be, Rt. xo) p0( 21-1
DKa(q(aT, 2r
xo) (r((,, 2r))
EyI log q 21 xp)
(32)
=:C(2))
Note the last term C(æa) does not involve po and can be dropped, and we normally do not optimize the last KL term DKI(q(ze (T, Zr xo) || pr 2T)) as we let the schedule to make this statistical
distance is sufficiently small.
A.2 FORWARD
We derive the following lemma for the marginal time step t.
Lemma (Continuous marginal conditioned on (æt, mo)). Let Ft = IT Ns For each position we have continuous marginal conditioned on (æz, .o) as
[8(2) z(), [N(z); V (1- (1- Ia),,
x x/
q(2) x mb)
xj m,
with z/ (w/(z/). Hence We finally have
q(z1 Ft, 2o) = II
g(2) 1=1
zj. xb) = II 8(z) xb)
II Ni21 (1- (a)
i: in fm
Then what follows proves Proposition 2
We first prove the conditional independency between 2, and 2t-1 given (z), mo) in the reverse context.
19

---

Published as a conference paper at ICLR 2026
Lemma 3 (Conditional independency between 2, and 2;-1 given (æt, zo)). and I;-1
conditionally independent given (z,, Zu) based on the forward kerned defined in Eq. (8)
To prove Proposition 1 we first prove the following lemma:
Proof of Lemma 2 and Lemma 3 If Φ = zy then the absorbing chain implies x' + m for 8 t,
the kernel gives z 2 almost surely, which is the first line of Eq. ().
If # m, use the law of total probability over æi-1 E ( m...
When xi-1 o m (first time masking at f), the second branch of the kernel gives zj ~ WV =b. (1 f))
When wi-1 = m (already masked), the third branch composes a diffusion forward step with the previous marginal z ~N( z (1 7t- which yields
z| N 71 =0, (1 7hhh-) A(V 5e zo: (1 - M.)
This proves the masked line of Eq.
Then leveraging these results, we can easily prove Proposition
Expand the path marginal, use Eq. and Lemma and factor positions. The sum over discrete paths yields g( zo); conditioning on 2; reduces the continuous part to Lemma 2
Proof of Proposition 7
A.3 REVERSE
Proof of Proposition need show:
We first prove the factorization shown in Eq. 14
To achieve this, we just
xt-1
0i, 2t, 20)) = q( Øt, 2t, x0) ((22-1 Tit, 2t, Ft 1; 37o)
(33)
q(z, [ Lt, 2o (0)g(0 (2r
0e; mo) g( a(21-1 2t, 2t, It-11 2D)
((z Øt, DD)
(34)
g(xx-1 1 3r, wn) -((x-11 Dt, 2t, 11-1, vo):
(35)
where q(z, 1 Rt, vo) = g( 2t; wo) by the conditional independence according to Lemma 3
Then the discrete part is the same as discrete diffusion, may leverage the results from Austin et al.
(2021a); Sahoo et al. (2024); Shi et al. (2024) to complete the proof of Eq. [5
Next, prove the closed form of the continuous part, g(z;-1 Tt, Zz, Tt- 1, zo). by case analysis based on the discrete states. We start with Bayes' rule for the continuous variables:
g( Øt, Zz, 11-1, wo) x ((zz 2;-1, g(2 1;
(36)
The forms of the two terms on the right-hand side are Gaussian distributions, but will change depending on the discrete states and leads to the three cases.
Case 1: No mask at (æ; vp). In this case, no noise has been applied to the embedding up to timestop t-1. Thus, both terms directly have a Dirac delta function: q( xt- = Wo. 29 8(z(11- 20). The posterior is therefore also Dirac delta function, proving the first part of Eq. 16).
Case 2: First time unmask at é (æ; = m, 0t-1 Ft). In this case, the first term in Eq. (36 is Gaussian while the second term becomes a Dirac 5( z0). The multiplication yields Dirac posterior at the same point: (( Æt-1 = Ty. To) = 8 6(2111- - 20)-
Case 3: Remaining masked at é (x; m, 0t-1 m)) In this case, both terms remain in Gaussian
distribution, and the parameters are same with normal Gaussian diffusion models. The product of these Gaussians is Gaussian, allowing et al. 2020), by completing the square on the exponent, we find that the resulting distribution is A Zz. 1; i,, (Zr, 2o), I.1. which proves the last part of Eq. (16.
to the standard derivation for DDPM (Ho
20

---

Published as a conference paper at ICLR 2026
Proof of Lemmal 7
Using the results from Proposition for single position the exact one-step KL at timestep t inside the ELBO is
(EL E (a ( 2, ze, mo) po (7 2t-1 2iz 21))
(37)
For the unmasked positions (æ,, m), the KL is identically 0, and plug in Eq. (14) 15 and 16 recover Eq. (18) exactly as
(a Wt: Zz, a0)
po(: Dt, z+)) = Ali [
log pa ( Ft; 2t)]
+ keep
DKL. cont
discrete
continuruus
with the ratio that determines whether the position is going be flipped to unmask:
Pi
keep
1 - O'e-1
Hip
Oc-
3r
Ot--1
(38)
at
04
Oxt
The discrete KL part exactly recovers the results from the absorbing discrete diffusion models (Austin al. 2021a; aa Sahoo et al. 2024) Shi et al., 2024), and the continuous KL divergence:
DKL = DKL DKI((N(C (N(H. B.La))
N N(Hor Belaa),
/ = [zo zt), = 10: 2.),
(39)
where we recap
ic(S, 2x)= v 71 7e)
ri(l 9-11) %/
B, (1 1t-1)(1 71))
c+
%/
7t.
This results in the comparison between zo and Eo and the KL divergence reduced to:
1
||0:(20 2,) (2..000 2;)1
or
120
20,011:
V - 76)
23
23g
1
Remark (On the Alternative Factorization). One could also decompose the posterior using the alternative order from the chain rule:
q, Zt-1 1.) 9(2,-1 æt, vo) q((x--1 Lt, 2t, 2t-1, 2o)
While mathematically valid and could provide properties in the sampling. this factorization is not fully tractable. The first term, g():-1l)), is a complex Gaussian Mixture Model. More critically, the second term, q()x-1l-), has analytical closed form, would require inverting the continuous diffusion process and the embedding function to infer a discrete state. The factorization in Prop. 2 therefore adopted as tractable choice for a more efficient algorithm implementation.
B DETAILED EXPERIMENT SETTINGS
B.
DIFFUSION SETTINGS
The CADD forward process has two coupled components, each with its own schedule.
Discrete schedule: we adopt the MDLM log-linear masking schedule for the discrete process (Sahoo et al. 2024). The discrete forward corruption uses continuous-time (< = 1-t, with i 1.
Continuous schedule: to keep the meaning of time aligned, we set the continuous latent follow linear flw-mmatching path isotropic noise (Lipman et al., (022). i.e., the position is masked, we have 2, = t) tc, ~ (0,
Multi-sample estimation: by default set K 1 for the estimation of t.,9 9 for fair comparison with the baselines. We provide ablation studies to demonstrate the effect of K > I.
21

---

Published as a conference paper at ICLR 2026
Table Benchmmrk computation cost of discrete (mask) diffusion and CADD.
Setting
Throughputs (Tokens/s per GPU) TFLOPS TFLOPS/s Avg. GPU Mem (GB)
Training
Discrete Diffusion
47,416
0.29
6,049
1.362
CADD
47,152
0.291
6.082
1.492
Inference
k
47,373
0.291
6.082
1.492
k
24,852
0.581
12.206
3.511
3
15,768
0,873
18,310
5.253
k 4
1,417
1.162
24.417
7.470
B.2 EXPERIMENT-SPECIIRE SETTINGS
Text Generation In our main experiments, including ablation studies that used to explore the properties of CADD, we train the models on OpenWebTex: Following the standard MDLM pre- processing (Sahoo et al. 2024), we use the GPT-2 tokenizer, resulting in a vocabulary of 50,257 tokens. The sequence length is fixed 1,024. Our text model is 12-layer DiT with 12 attention heads and an embedding dimension of 768, totaling approximately 168M parameters. During training, keep the same training configuration, i.e., train for about 2M steps with batch size of 256 to match the total 262B tokens seen in the training. We use the Adam optimizer with a learning rate warmed up from 0 to 10-4 The results in Table and Table 9 are based on Text8 and LMIB dataset, where we strictly follow the training setting in Jo & Hwang (2025) and [Sahoo] al. (2024). Please refer their experiment settings for more details. For evaluation, we follow ReMDM (Wang et al. 2025)'s 's evaluation setting, where we randomly sample 5,000 text samples with length n I, 024, using (128, 256, 512, 1024, 4096] sampling steps. The sampled token sequences are used to compute MAUVE score, generative perplexity with GPT2-Large model, and entropy.
Image Generation We experiment on CIFAR-10 and ImageNet (with resolution 32 32), which consists of 50,000 and 1,281,144 natural images respectively. CIFAR-10 already has 32 x 32 resolution. For ImageNet images, we follow the preprocessing used in EDM (Karras et al. 2022). i.., using center-crop make it as squared image and rescale to the desired 32 x 32 resolution. As the model is trained on pixel space, we treat each pixel as a discrete token, resulting in a vocabulary size 256 at each position. We follow the architecture design used in MDM-Prime (Chao al.) 2025), which is a U-Net architecture based on ADM (Dhariwal & Nichol, 202T). For CIFAR-10, leverage augmentation pipeline proposed in Karras et al. (2020), but only keep the rotation and flip operation to avoid pixel value changes. We let set the augmentation probability as 15% on CIFAR-10, and there no augmentation used on ImageNet. For both experiments, we set learning rate as 1 x 10-4 using Adam W optimizer, and train the model until it has seen 200M and 4B images respectively. In sampling, we adopt a cosine decay for temperature with Tmax 2.5, and applied the corrector following Gat et al. (2024). We use the standard Fréchet Inception Distance (FID) and Inception Score evaluation, computed with 50,000 randomly generate images.
Code Generation We use the OpenCoder dataset (Huang et al. 2024), selected by following the recipe in DiffuCoder (Gong et al., (2025)). We strietly follow their settings to initialize the 7B model with Qwen2.5 5--oderr checkpoint, and adapt it to diffusion model using the techniques introduced in Gong et al. (2025a). Then we traine the model on a 64 NVIDIA A100 GPUs in total. The training process utilized BF16 mixed precision and was scaled using Fully Sharded Data Parallelism (FSDP). For optimization, we employed the Adam optimizer with peak learning rate of 1 x 10-5 preceded by 2.000-step linear warmup. The model is trained with 65B tokens in total. For generation, both models were configured with a maximum sequence length of 512 tokens and a total of T=512 diffusion timesteps. During generation, we employed top negative entropy remasking sampler. The CADD from scratch variant uses temperature 0.2 and the DiffuCoder initialized variant uses temperature 0.01.
B.3 COMPUTATION ANALYSIS
We measured both training and inference efficiency under the same hardware (H100, FP32) and batch settings (batch size=1), using the DIT model with 169 M parameters. The results summarized in
22

---

Published as a conference paper at ICLR 2026
0.3
0.290
0.270
91.
GADD
58.3
MAUVE (
GROC, (VD.-M 9
88
Gen PPL
[83
0.11
0:11
0.899
C400
CAD0, (47
263
125
205
5/2
1924
0096
125
206
1024
4096
Steps
Steps
MAUVE
(b) Generative perplexity.
Figure 4: Analogous figure of Figure We compare the finetuned checkpoint using CADD objective with CADD and the initialization checkpoint of MDLM.
the table 7 During training, CADD matches the speed of the MDM baseline, with nearly identical token throughput (tokens/s) per gpu and memory usage. During inference, when = 1, CADD shows similar computation cost to MDM. The inference cost increases in linear way K increases, since the model performs K times forward for the Xn estimation, while the training cost remains unchanged.
C ADDITIONAL EXPERIMENT RESULTS
C.1 TRAINING FROM MASK DIFFUSION MODEL
From the experiments on code generation, we have seen CADD could be used finentune an existing discrete (masking) diffusion model to improve the performance. Here we provide complementary evidence that such observation is also valid on text generation. We finetune MDLM checkpoint with CADD objective for additional 50B tokens and evaluate the performance with same setting shown in the main experiments (Figure 3 The results are shown Figure 4 The red curve shows close performance to the green one that represent CADD's performance, which indicates CADD could efficiently finetune an existing MDM model to enhance the generation capabilities.
C.2 PERPLEXITY EVALUATION
Since the objective CADD involves the KL divergence of both discrete and continuous component shown in Eq. (14.. is not fair to compare the tightness of the bound directly with other models, and we choose to focus more on the evaluation of the generated samples. However, our model is still able to compute the likelihood of the discrete part. Here we put the results for reference, aiming to provide more information to help the readers understand how the model helps the discrete diffusion side.
Table 8 and Table report the perplexity evaluation on character-level and token-level respectively. The model is trained Text8 and LM1B, following the settings Jo & Hwang (2025) and Sahoo et al. (2024). On Text8, we can see CADD achieve very competitive perplexity results, and is slightly worse than the SoTA RDLM (Jo & Hwang 2025). On LMIB, we can see CADD achieve the best results among diffusion models when evaluate the discrete part perplexity on both LM1B data and OWT data.
Table 10 reports the zero-shot evaluation results of the checkpoint trained on OWT data. We can observe CADD and MDLM both surpasses the perplexity AR models on Lambada, Pubmed and Arxiv datasets. They have different dataset that they are good at in terms of perplexity, and CADD wins slightly more as it shows better zero-shot perplexity than MDLM on 4/7 tasks. These experiments result jointly indicate that CADD can not only provide strong generation quality, but also provide a good discrete likelihood bound.
C.3 ABLATION STUDIES
Comparing the number of samples used for 2n = folæ,, x (R)) We first conduct ablation to study how the number of samples used to compute do would affect CADD's performance. Similar to
23

---

Published as a conference paper at ICLR 2026
Table Bits Per Character (BPC) results on Table 9: Test perplexities (PPL: 4) LMIB. The baseline results are taken from Sahoo et al. (2025). For Hwang (025). Bold denotes the best result in CADD, we report the bound on the discrete likelihood. autoregressive or diffusion models. The best Best diffusion value is bolded. the dataset for SEDD diffusion results are marked bold.
Text8 test set. Results are taken from Jo &
didn't incorporate sentence packing.
Method
BPC (1)
Method
LMIB
OWT
Autoregressive
Autoregressive
AR
1.23
Transformer
22.8
17.5
Continuous Diffussion
Diffission ( (Uniform-state Gaussian)
Plaid
1,48
D3PM Uniform
Austin et al 2021a
137.9
BFN
1.41
Diffusion-IMM
Li al. 2022
118.6
RDLM
1.32
SEDD Uniform
(Lou
2024
40.3*
29.7
Discrete Diffusion
UDLM Deschenaux Gulcebre
2025
36.7
27.4
Multinomial Diffusion
1.72
DUO (ahoo et al.
2025
33.7
25.2
D3PM Uniform
1.61
Diffusion (absorbing state)
D3PM Absorb
1.45
SEDD Absorb
1.39
D3PM Absorb (Austin al
2021a
76.9
MDLM
1.40
DiffusionBert (ficet
2025
63.8
MD4
1.37
SEDD Absorb (Lou
2024
32.7"
24.1
MDLM (aaooo al.
31.8
23.2
CADD (Ours)
1.35
CADD (Ours)
31.4
23.1
Table 10: Zero-shot perplexities (1) of models trained for 1M steps on OpenWebText. All perplexities for diffusion models are upper bounds. Baseline results are taken from [ahoo al. (2025). Best diffusion model performance results are bolded and diffusion values better than AR are underlined. Plaid and D3PM are trained with 0.3M more steps.
Method
PTB
Wikitext
LMIB
Lambada
AG News
Pubmed
Arxiv
Autoregressive
Transformer
82.05
25.75
51.25
51.28
52.09
49.01
41.73
Diffusion (Uniform-sted state Gaussian)
SEDD Unifar
105.51
41.10
82.62
57.29
82.64
55.89
50.86
Plaid
142.60
50.86
91.12
57.28
UDLM
112.82
39.42
77.59
53.57
80.96
50.98
44.08
DUO
89.35
33.57
73.86
49.78
67.81
44.48
40.39
Diffusion (absorbing state)
SEDD Absorb
100.09
34.28
68.20
49.86
62.09
44.53
38.48
D3PM Absorb
200,82
50.86
138.92
93.47
MDLM
95.26
32.83
67.01
47.52
61.15
41.89
37.37
CADD (Ours)
93.33
31.84
64.98
46.81
62.80
42.62
37.52
main experiments in text generation, we compare CADD with K E (1,2,3,4) in terms of MAUVE and generative perplexity.
As shown in Figure 5 increasing both the number of sampling steps and the hyperparameter K consistently improves CADD's performance. The value of K, which corresponds to the number of continuous samples used for soft hints, has consistent and positive effect on generation quality. It interesting to see the largest performance gain, especially for generative perplexity, comes from increasing K from to 3. The subsequent gain from K = 3 to K 4 is smaller. One possible reason that when K not large enough, the predicted logits could vary and make the expected value smoothed to be flatten distribution. As K gets bigger, the estimation of the correct 2n becomes more accurate, resulting in better generation quality, with a trade-off between desired sample quality and inference-time latency.
We also use entropy as a complementary metric to observe the model's ' behavior, and the results are shown in Figure We observe CADD, the highest-quality model in terms of MAUVE and generative
24

---

Published as a conference paper at ICLR 2026
0.430
CASD (0-11
(6.990]
51.2
0400 0-)
MAUVE
0.33
G4.0 (602)
DASD (1-31
(3.44)
(3333)
Gen PPL
665
26.8
CADD (e:)
6800 (-2))
(600)
02
0.70
0.120
F15
6277
0.800
[3.3
16.1
28.7
126
5/2
1024
0096
125
256
1024
4096
Steps
Steps
(a) MAUVE
(b) Generative perplexity,
Figure 5: Analogous figure of Figure samples to estimate Tg (K=1-4).
We compare of CADD variants using different number of
8.48
[542)
8:47
6.8
5.42
5441
C40D (K-))
CACD (K-1
Entropy (1)
6.46
5.48
C
6.35
C400 (K00)
CACD (-:))
6.4
(338)
(394
5A30
6T8
126
266
6/2
1084
4098
128
250
612
1024
4098
Steps
Steps
Figure 6: Analogous figure of Figure study of generation variance and diversity across all methods and across different K. We use entropy (higher indicates more stochasticity) are reported.
perplexity (shown in Figure has the lowest entropy. This indicates that CADD achieves its keeps lower variance in the generation process with concentrating its continuous conditions. The right plot, which analyzes different values of K for CADD, shows that a larger K consistently leads to lower entropy. This reveals the role of K hint mechanism. A larger K provides stronger, more deterministic "soft hint" from the continuous space, preserving smaller variance during generation. However, this does not mean CADD lack generation diversity, as it still hits a strong MAUVE score, indicating it strikes a good balance between mode-covering and mode-seeking.
On model architecture Similar to the text generation, we also examine the performance of image generation. We conduct experiments to test the impacts of model architecture and number of function evaluations (NFEs) in the sampling stage. The results are reported in Table [T] As shown, ADM (Dhariwal & Nichol, 2021) shows stronger performance than DDPM++ (Song et al. 2021) across different NFEs. Especially when NFE is sufficiently large as 512, the performance of using ADM NFF=512 12 configuration demonstrate significant performance gain. As qualitative justification, we can also observe the last row of Figure has the best visual quality.
FID (L)
Model
64
256
512
DDPM+
31.24
4.72
4.70
ADM
30.41
4.29
2.88
Table Ablation results on image generation, trained with DDPM++ and ADM architecture. FID results measured using NFE=64, 256, 512.
Figure Qualitative results of CIFAR-10, 10, gen- erated by ADM, using NFE-64,256,512 (from top row to bottom).
Qualitative visualization Table 12 demonstrates the effect of the continuous embedding semantic scaffold that guides the discrete unmasking process. We use "A model sits on the mat" as input, and mask the second position. Using different noise level to corrupt the embedding of token "model" to form 2, and predict the masked token. the low-noise regime (t < 0.3), the continuous signal is clear enough and the prediction is same "model". As noise increases, the embedding degrades and we can observe the predictions traverse from relevant category (e.g., vase", tank") to generic priors only when the guiding signal is fully destroyed / 1.0.
25

---

Published as a conference paper at ICLR 2026
Table 12: Qualitative Generation Analysis. We visualize the prediction for the masked token in "A [MASK] sits on the mat" as we vary the noise level of the continuous embedding corrupted from the embedding "model".
Noise Level (t)
Predicted Sentence
Predicted Token
Input (GT)
A model sits on the mat
model model
0.1
A model sits the mat
0.3
A model sits on the mat
model
0.5
A vase sits on the mat
vase
= 0.7
A tank sits on the mat and sits on the mat
tank
and
D ADDITIONAL GENERATED SAMPLES
D.I TEXT GENERATION
As for the falling gas prices going forward, it may be similar to declining oil prices decades ago. Both brought Low exchange rates through decreased prices for relatively mass-produced natural textiles such sil and foed, leading natural gas to become extremely expensive. With subdued market with few consumers,
benefits being passed through marketplace costs far more than was case have been spent on natural gas development, and in order to stay on-time with rising exchange rates, state producers had to provide cover for these higher prices. With the "competitive" advances inshore wind and solar--those intermittent energy renewables, and gas --cost per küh has fallen steadily, driven power purchasers to switch to the cost of generating new batteries in range efficient carben-free natural gas. The arount of electricity produced per F-quarter based on natural gas dropped by 10.4 percent and, the gas price dropped by a one-square inch average. Yet further prices dropped in the subsequent years compared to 2014. With all the increases in wind prices compared to 2013 (when net natural gas-fired electricity emissions fell nearly 9.5 percent)--the average annual rate of wholesale electricity earned from the non-resident lease-traded incustries dropped from $1.262 billion in 2015 by 37 percent, to $17.66 billien in 2015. Residential electricity by electric utilities increased by a $4.25 billion more in 2015. The U.S. is also controlled by the reduction in the price of electricity. Using "sclar" technologies, which come off higher electricity prices target, have dropped by about percent last year and last through to the fourth quarter, but expect have drastically energy prices decrease in the next three years. years.While While Duke Energy's nuclear project will be expected to produce energy in the demand that has, like, occurred in the last quarter, the nation switched from natural gas renewable power their platform for energy produce in 2015. [2] While coming to energy costs as a whole does not in general match fossil fuel prices in large parts of U.S., electricity prices are more competitive in the North and North America west where price is, ever, and even where scme of these factors do so...
the 2015 Great Canada Great Average,
years, the reduction in nuclear
26

---

Published as a conference paper at ICLR 2026
What all the results show is that they get the results that you need. They are showing how much more important that mindset is, the more games you get.A11 MIS radio's Mi Michael Helgen sisusee Sunday's game, noting that it put us kind of in a half time game without help. We not expected finish well, say that we had an opening-day performance. We could have maybe been surprised right away from the game that Jordan Morris (capt) in the 76th minute changed the moment, and we ourselves sealed score sheet.Michael Helgen gave Rotem winist's Swedish tear. first hand. 0:36 secondsit felt as though the Sounders had à special team on the team in Hawaii, and we had that sort of game against Houston.Shellye Portland on their history earned ceach Chris Porter's trust and would take step toward the future.Any stage of the season, for non-soccercer player, they' re always going to find it. It's very rewarding to have that half of the season. And definitely, the MLS coach takes the game a lot, he said. You don't need to have pretty good game if you' message that the captain may not want to do it since you' ve got to ask and he's at home but rather you treat yourself more going to practice, Helgen said. But for us trying to leave sccres open, that was a good idea, Soccer teams said they were still impressed, but the game affected visitors under 8.1 per cent being played off of Dynamo side with six clean sheets. just think that the game we lost against, when they started their first goal in five games, we underestimated them, running into mistakes we could couldn't control, " Helgen said. "But got lucky, and I knew we'll have to be honest to do it again, but Marko Coyle and the other smart, athletic goalkeepers in Salfora, who don't just pan through the season, did their job and switched the bus that night. We' re a 3-0 win to Houston and 2-1 loss. They had go into the locker room, and they had to make better players, and they didn't have space the locker to do that, the score was 1-2. "Andersen said Denmark is preparing for the one it needs, "If it's going te happen, then we need to be confident and do all it takes get there first to get the opinion, the experience, the opponents knewledge and to make sure ourseives reacy and be ready right away for the first time," he said. "We know that next time and practice the way to be ready and prepared for the season. "eason."<|ennofs"" "<]endoftext |>Getty Images Brendan Rodgers had to be criticised about his advice to Jamie Carragher when he arrives with a defeatThe reality is that it takes much to build on Rittle.Redgers says the following in post-match interview on Real Sky Sports: can see that as an iron person, in our manager, I will talk with different people and I will react di differently because I am speaking straight ght.. One example pressure Robarto Mirotic and tell what it is. what he said it's very hard at night, I can't explain what it is....
it's hard to
said a team playing at that high level as
very good Slayer.Sometine
he has
27

---

Published as a conference paper at ICLR 2026
When it was only briefly used to experience psychic balance, after being removed at the optimal frequency, decreasing the chance for general performance, but when it changed at rest and enly even moved at the same intensity, it did not you seriously centrol the transition from strength to strength. Instead, it also gained the balance in the fluid balance with the normal balance. It available
was slew and powerful in healing activity that beyond all kinds of fluctuations in concentration. So, when the movement was replaced with other possible such qualities with torque, psychic or psychic activity, it still had a strenger sensitivity to performance, yet when received even a deeper part of the metabolism, it began becoming more energetic and efficient and therefore, it improves balance. When was replaced with the meditation and then removed, it moved around a rest and finally switched to random balance, and at that point with the max stimulation the amount of basic torque applied at the spell. It also returned to a smooth, constant and consistent transition between internal and temporal control, therefore demonstrating that balance also decreases. But even after the activation of the trait, experienced change in intensity. Now, the tactile balance becoming more effective and more stable, and it leads to increased gains concentration and performance. Do you be really cencerned about the balance, balance and balance connection to the spell? The positive effect on the tactile balance now comes true to speed. The tactile balance is only determined by strength and balance, and still held at a constant point the critical frequency, In spell,
fact, the spirit is not moving in the same direction and it has not been able to experi balance because it moved to another true frequency. !The Target Applateur store website representative today confirmed that Philips was shut down in order to restart its for any explanation, confirmed a major shutdown was found. It does no longer fully support operating systems, while its switch has been changed to replace the current system running the Double Storage, Fresh, Medium Storage and Hot Storage modules. Please Note that are working on the matter is not there. He said: Print had working to resolve all the issues on the platform, and if it fails, the shutdown requi the vendor being able te fix them. We do not know at the reascn for the delay and therefore the reasons why are continuing control will be determined by them and discussed today so we will not go on a more comprchensive timetable. We will't speculate running locally used current systems. While the error created complexity, it is decided by the seller if this fix is true, we expect that these issues will be resolved with proper action. know that if we want to continue with browsing cycles then it will be very difficult to restart, and with our support, access always applied data settings, stere volumes and autematic navigation...
launch. While Target has not been asked
the basis whether to continue
D.2 CODE GENERATION
From typing import List, Tuple
def rolling max (numbers: [int() -> List [int]:
From given list of integers, generate 1ist of rolling max mum element found until ven moment
28

---

Published as a conference paper at ICLR 2026
in the sequence. rolling_max [1, 2, 3, 2, 3, 4, 21) [1, 2, 3, 3, 3, 4, 4]
result
current_max numbers [0] for num in numbers:
12
13
if num > current_max: current_men max = num
14
result (eurrent. max) return result
16
Listing I: "Generation on HumanEval"
def comb sort (arr):
= len (arr)
gap
swapped True while ( (gap 1) or swapped)
swapped False
gap int (ggap 1.3))
if (gap 1)
gap
10
for range ( (n gap)):
11
is (arrli) > arri (i + gap) 1):
12
(arr arr (i + gap) }] (arrl (1 + gap) I, arr(i])
swapped True
return arr
15
16
assert comb sort (15, 15, 37, 25, 791) == [5, 15, 25, 37, 79]
Listing "Generation on MBPP"
From random import randint, seed random_seed import time import hatpletlib.pyple as plt
def task fune (my_ list, size=100, seed=100)
=**
knhances 'my_list! by appending ni the number 12, then generates a list of random integers based on the sum of elements in 'my_list', limited by ssiee, measures
the time taken for this process
and plots histogram of the generated random numbers.
10
11
The sizc of the random numbers list is determined by the sum of the numbers in 'my_list', with an upper limit set by 'size', The random integers are within the range to 100, inclusive.
12
13
Parameters:
mist (list): The input 1ist containing numeric elements. size (int): Maximum size limit for the generated 1ist of randon numbers. Default is 100.
16
17
seed (int): Seed value for random number generator for reproducibility, Default is 100.
18
19
Returns:
20
tuple: A tuple containing the time taken to generate the list (in seconds, as float) and the matplotlib Axes object for the histogram. The histogram's s axis labeled 'Number', representing the range of random. integers, and the y-axis is labeled 'Frequency', represeating the frequency of each integer in the generated list.
22
29

---

Published as a conference paper at ICLR 2026
Raises:
26
TypeError: 'my_list' net list.
27
ValucError: If 'my_list centains elements that are not numeric int or float).
The histegram plots the distribution the random numbers generated, with the number range (1-100)
30
on the x-axis and the count (frequency) of each number on the y-axis.
Requirements:
random
34
time
35
matplotlib. ib. pyplot
36
Example:
38
>>> my my_list 12, 3, 5]
39
>>> time_taken ax func (my_list)
40
>>> print (type (time_taken) # Example output: <class 'float>>
<class 'float'>
ax.get_title() Returns 'Histogram Random Numbers' 'Histogram of Random Numbers'
43
44
45
it not isinstance (my list, list):
raise TypeError (" my list' must list.")
47
48
if not all (isinstance (x, (int, float)) for x in my_list):
49
raise ValueError ("mmy_list' must contain nameric elements."
Append 12 to the list my wpp.litt..ppeed111
52
53
54
Calculate the sum of the list cotal_sum sum (my_list)
56
57
Determine the size of the random numbers list list_size = min (totaal__sm, size)
58
Set seed for reproducibility rancom seed
61
62
Generate the list of randem nurbers random_numbers [randint (1, 100) for in range (list_size)
66
Measure the time taken start_time time.time ()
67
# Generate the histogram
figufi (figsiizee (10, 6))
70
pit.hist (randon bins=range (1, 102), align=' left', edgecolor= "black')
71
pit.xlabel ( Number')
plt.ylabel FFequeecy')
73
plt.title (1 Historram of Randem Numbers/)
74
plt.show()
15
end time = time.time ()
Return the time taken and the Axes object return end_time start_time, plt.gca()
78
Listing 3: "Generation on BigcodeBench"
30

---

Published as a conference paper at ICLR 2026
D.3 IMAGE GENERATION
FER9A
Figure 8: Unconditional image generation, generated by CADD trained on ImageNet-32 x 32.
31


Reviews:
[{"review_id": "TVxzBHNsNh", "reviewer": "Reviewer_7PKE", "summary": "This paper introduces Continuously Augmented Discrete Diffusion (CADD), a framework that combines Masked Discrete Diffusion (MDM) with a parallel continuous diffusion process in the embedding space. The key innovation is to replace the \"information void\" of a [MASK] token with a noisy but semantically informative continuous latent vector. This provides \"soft hints\" during the denoising process, leading to more coherent and higher-quality generation across text, image, and code modalities. The method is shown to be effective, scalable, and simple to implement.", "questions": "What is the inference-time latency of CADD (with K=1) compared to a standard MDM baseline when generating a sequence of fixed length? Is the overhead primarily from the extra continuous state operations or from the architectural changes?\n\nThe continuous latent z_t is initialized from a standard Gaussian prior at step T. Have you experimented with more informative priors, and if so, what was the effect?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper target at solving a limitation of existing MDMs, which is the \"information void\" created by the absorbing [MASK] state. The idea of augmenting the discrete space with a continuous one to preserve graded semantic information is both intuitive and powerful.\n\nThe paper provides extensive experiments across multiple modalities (text, image, code) and scales (from 168M to 7B parameters). Consistent gains are observed over baselines.\n\nThe paper is well-written.", "weaknesses": "Time cost of model running can be analyzed. For example, a time cost and FLOPs comparison between CADD and the baselines can be helpful for the readers to understand the performance-efficiency trade-off.\n\nThe experiment comparison is presented mainly with statistical result. It lacks illustrative example to show how the latent space parameter helps improve the generation performance.\n\nCode is not released for reproducibility checking.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VlMw4D3Mv0", "reviewer": "Reviewer_oUtt", "summary": "The paper presents Continuously Augmented Discrete Diffusion (CADD), a framework which combines the strengths of Discrete Diffusion Models and Continuous Diffusion Models. CADD retains the discrete masking strategy and uses continuous latent vectors as the tokens get masked, to maintain semantic information. In essence, two diffusion processes happen at the same time. Experimental results show that CADD models consistently generate higher quality samples over the baselines in three tasks: text, image and code generation.", "questions": "1. What happens if you include the MSE loss? How does this affect z_t and performance?\n2. Does CADD incur extra computational cost vs discrete diffusion models? If so, is it significant?\n3. Did the authors test any alternative schedules for the continuous process?\n4. In the main experiments for text generation, why isn’t Duo a baseline?\n5. It could be interesting to include interpretability analysis regarding z_t, to visualize examples where the latent hints help disambiguate the tokens.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well written, complete and easy to follow.\n* It bridges a known gap and is conceptually novel. This specific combination of discrete diffusion with continuous latent vectors when the tokens are masked had not been previously explored.\n* The idea is technically sound and the formulation of CADD is coherent. The factorization q(x_t​,z_t​ | x_0​)=q(x_t ​|x_0​)q(z_t​|x_t​,x_0​) provides a clear way to combine the discrete and continuous variables. \n* There is strong empirical validation. Results are presented for 3 different tasks, and CADD offers significant improvement over the baselines.", "weaknesses": "* The discrete and continuous reverse process are derived in Section 4, but in the experiments the continuous MSE term is not used. This means that the continuous part is never directly trained to match the distribution, just guided. Although z_t appears to carry meaningful information, this makes me question how including the MSE would impact z_t and overall performance.\n* CADD maintains an extra latent vector per masked token, when compared to standard Discrete Diffusion Models. The authors should discuss whether this results in extra computational or memory cost.\n* In the text generation experiments, the authors do not compare CADD against any continuous/hybrid diffusion model, such as Duo.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0BT7VODYd3", "reviewer": "Reviewer_xGtH", "summary": "This paper is motivated by the information loss when transforming clean tokens into mask tokens in MDMs. To address this issue, CADD augments each token with an informative continuous latent vector, which can provide more information on the original tokens. I find this idea is quite novel and can be viewed as the most important contribution of this paper.\nThe authors discuss the training and sampling method of CADD and validate it on image, text, and code generation tasks.", "questions": "- The current methodology of discrete diffusion almost converges to masked diffusion models. However, regarding the information void issue of MDMs, can this issue be alleviated by discrete diffusion with a uniform distribution noise?\n- How do you compute your NFE? (e.g. in Algorithm 2)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- I find the idea of augmenting MDMs with continuous latent vectors to provide semantic information is both novel and well-motivated.\n- The empirical evidence is strong to support the effectiveness of the CADD method.\n- The paper presents a thorough derivation for training objective (ELBO) and sampling method, making the methodology rather practical.", "weaknesses": "- The authors use a learnable network $w_\\theta$ which takes sequence $x$ as input and outputs the latent vector $z$. However, in my opinion, a more straightforward and robust way is directly assigning a learnable latent vector for each token, and all the parameters forms a \"codebook\". Will this parameterization works in practice?\n- In line 5 of Algorithm 1, the $w_\\theta$ takes the entire $x$ as input, and in line 10 of Algorithm 2, the $w_\\theta$ takes a single token $x^i$ as input. Which one is the correct expression?\n- The authors should discuss how they implement the  $w_\\theta$ in practice. Moreover, as they introduce an additional model, a comparison on memory and computation time should be provided.\n- I suggest that the authors to provide more experimental results. For example, the test perplexity and zero-shot perplexity can be a more  common benchmark for unconditional text generation. I am also interested in how the FID score on CIFAR10 scales on different choices of NFEs?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QAenM616an", "reviewer": "Reviewer_LxZ1", "summary": "This paper tackles a core problem in discrete diffusion models: all states collapse to a single absorbing state without retaining meaningful information. Inspired by the continuous diffusion model, the authors propose continuously augmented discrete diffusion (CADD), a hybrid framework that augments the standard masking process paired diffusion process in a continuous latent space. CADD utilizes a noisy continuous vector as a semantic condition to guide the discrete desnoising step. Experimental results across three distinct modalities demonstrate the effectiveness and versatility of CADD.", "questions": "- Modern visual generative models mostly utilize vector-quantized (VQ) visual tokens for image synthesis with complex conditioning (such as text prompts). As this paper primarily searches for unconditional visual generation with discrete pixel space, could CADD be impacted by the complex VQ tokens and conditioning?\n\n- Masked Generative Models are a family of discrete diffusion models with an absorbing state, while they show superior efficiency [1][2]. Can the CADD framework be applied as a drop-in augmentation to MaskGIT-style models? If so, what is the expected trade-off in terms of sample quality/diversity?\n\n[1] Maskgit: Masked generative image transformer\n[2] An Image is Worth 32 Tokens for Reconstruction and Generation", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The idea that combines the discrete diffusion process with the continuous semantic latent to resolve the information void problem is novel and seamlessly integrated into the training and sampling pipeline. Furthermore, the lower bound of the objective is well-justified.\n\n- The paper's motivation and core concept are clearly presented, and the visualizations in Figures 1 and 2 are also clear.\n\n- The effectiveness of CADD was validated across three distinct tasks, consistently demonstrating improvement over conventional discrete diffusion models. The paired continuous latent vector effectively resolves the 'information void' problem, which is clearly reflected in the scalable generation quality as sampling steps increase.", "weaknesses": "- While compatible with MDM training, the CADD framework is inherently more complex. It requires managing two parallel diffusion samplers and fusing their representations at every step.\n\n- The authors opt to use only the standard discrete cross-entropy loss for simplicity, omitting the continuous MSE loss that would naturally arise from the ELBO. While this works well empirically, the paper could be strengthened by either (a) showing that adding the MSE loss provides no benefit (justifying its omission) or (b) exploring the full loss term, which might lead to even better performance.\n\n- While CADD shows superior capabilities with large sampling steps, the performance seems to degrade sharply with a few-step sampling, probably due to the mode covering problem of the continuous semantic latent. This could amplify the inefficiency of the discrete diffusion models, which already suffer from large NFEs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)