The Diffusion Duality, Chapter II: $\Psi$-Samplers and Efficient Curriculum
Authors: Justin Deschenaux
Caglar Gulcehre
Subham Sekhar Sahoo
Venue: ICLR 2026 Poster (2026)
Keywords: diffusion language models
diffusion models
large language models
inference-time scaling
predictor-corrector sampling
efficient taining
Primary area: generative models
TL;DR: We generalize previous predictor-corrector samplers for discrete diffusion models to arbitrary noising process, and propose a memory efficient curriculum learning algorithm, 3x faster than Duo's original curriculum.

Abstract: Uniform-state discrete diffusion models excel at few-step generation and guidance due to their ability to self-correct, making them preferred over autoregressive or Masked diffusion models in these settings. However, their sampling quality plateaus with ancestral samplers as the number of steps increases. We introduce a family of Predictor-Corrector (PC) samplers for discrete diffusion that generalize prior methods and apply to arbitrary noise processes. When paired with uniform-state diffusion, our samplers outperform ancestral sampling on both language and image modeling, achieving lower generative perplexity at matched unigram entropy on OpenWebText and better FID/IS scores on CIFAR10. Crucially, unlike conventional samplers, our PC methods continue to improve with more sampling steps. **Taken together, these findings call into question the assumption that Masked diffusion is the inevitable future of diffusion-based language modeling.** Beyond sampling, we develop a memory-efficient curriculum for the Gaussian relaxation training phase, reducing training time by 25% and memory by 33% compared to Duo while maintaining comparable perplexity on OpenWebText and LM1B and strong downstream performance. We release code, checkpoints, and a video-tutorial on [https://s-sahoo.github.io/duo-ch2/](https://s-sahoo.github.io/duo-ch2/)

Published as a conference paper at ICLR 2026
THE DIFFUSION DUALITY, CHAPTER II: V-SAMPLERS AND EFFICIENT CURRICULUM
Justin Deschenaux
Caglar Gulcehre
Subham Sekhar Sahoo
EPFL, Lausanne, Switzerland Microsoft Al
CCmmell Tech, NY
ABSTRACT
Uniform-state discrete diffusion models excel few-step generation and guidance due to their ability to self-correct, making them preferred over autoregressive Masked diffusion models these settings. However, their sampling quality plateaus with ancestral samplers as the number of steps increases. We introduce a family of Predictor-Correcto (PC) samplers for discrete diffusion that generalize prior methods and apply to arbitrary noise processes. When paired with uniform-star satorrm-sttee diffusion, our samplers outperform ancestral sampling on both language and image modeling, achieving lower generative perplexity at matched unigram entropy on Open WebText and better FID/IS scores on CIFARIO. Crucially, unlike conventional samplers, our PC methods continue to improve with more sampling steps. Taken together, these findings call into question the assumption that Masked diffusion is the inevitable future of inffusiio-basee language modeling. Beyond sampling, we develop a memory-efficient curriculum for the Gaussian relaxation training phase, reducing training time by 25% and memory by 33% compared Duo while maintaining comparable perplexity on Open Wex and LMIB and strong downstream performance. We release code, checkpoints, and video-tutorial on:
http://s-sahooo.om//uuoo
MDLM
MDLM
MDLM RaMDM
54
Duo+-
Duo
Duo w-sampler
Duo +-sampler
Gen. PPL (4)
(4)
28-
21
25
32
256
1024 2048
64
128
512
1024
4096
Figure Performance on Language Modeling and Image Modeling. V-samplers generalize ReMDM (Wang et al., 2025) to arbitrary noise distributions. (Left)): Generative perplexity (Gen. PPL; 4) as a function of NFEs, with nucleus sampling p = 0.9. V-samplers consistently improve with more steps, unlike ancestral sampling which plateaus. Curves are annotated with the average unigram entropy per sequence as a proxy for diversity. (Right): On CIFAR-10, & P-samplers achieve better FID (L) than MDLM (with ReMDM).
INTRODUCTION
Diffusion models are powerful generative algorithms that have achieved remarkable success in modeling continuous data domains, including images (Ho et al., 2020a; Rombach et al., 2022), audio (Kong et al., 2021; Liu al., 2023b; Huang et al., 2023), and videos (Ho et al., 2022; Esser et al., 2023; Blattmann et al., 2023; Polyak et al., 2025). Recent advances have extended diffusion
"Correspondence just deschenauxx@ppf ch and asahoolcs cornel 1ormee eddu

---

Published as a conference paper at ICLR 2026
models to categorical data, demonstrating their potential for language modeling (Austin et al., 2023; Lou et al., 2024; Sahoo et al., 2024; Shi et al., 2025; Ou et al., 2025; Sahoo et al., 2025a:b), graphs (Liu al., 2023a), molecules (Lee et al., 2025), and audio (Ku et al., 2025). Unlike autoregressive models that generate tokens sequentially from left to right, diffusion language models can decode tokens in parallel and in any order while leveraging bidirectional contextual information. This capability enables the design of language models that can be significantly faster than their autoregressive counterparts while maintaining strong downstream performance (Song al., 2025; Labs et al., 2025).
Discrete diffusion models primarily employ one of two noise distributions: uniform prior or a masked prior that concentrates all probability mass on a special [MASK] token. Unlike Masked Diffussio Models (MDMs), which update each token exactly once, Uniform-Statt Diffusion Models (USDMs) allow tokens to be revised multiple times during generation, enabling self-correction. This makes USDMs particularly effective for few-step (Sahoo et al., 2025a) and guided generation (Schiff et al., 2025). However, the generation quality of USDMs has not yet matched that of MDMs in high-sampling-step regimes, and USDMs' modeling capacity, measured by likelihood, remains inferior to MDMs'. Although Sahoo et al. (2025a) proposed a curriculum learning strategy (Bengio et al., 2009) that narrows the likelihood gap, this curriculum is computationally expensive.
To address MDMs' inability to remask tokens, ReMDM (Wang et al., 2025) introduced "Predictor- Corrector" (PC) samplers that generalize and outperform earlier PC methods (Campbell et al., 2022;
Gat al., 2024). These samplers substantially improve the inference time scaling behavior of MDMs.
However, PC methods for uniform- state diffusion remain underexplored. Campbell al. (2022)
proposed PC methods for samplers that take advantage of the rate change matrices of the continuous- time Markov chain (CTMC) formulation of discrete diffusion processes, but such samplers are known to perform worse than ancestral samplers (Lou et al., 2024; Schiff et al., 2025).
We propose +uo++ to address these challenges, which expands the design space of USDMs using non-Markovian superposition posteriors (or as we refer to them in this paper, L H-posteriors). These posteriors align with the intermediate marginals of discrete diffusion processes and give rise to V-samplers with predictor-corrector capabilities that are crucial for improving sample quality. In addition, Duo ++ introduces an efficient curriculum learning strategy that advances the approach of Sahoo et al. (2025a) by accelerating training and reducing memory usage.
In summary, our contributions are threefold: (1) we propose a family of Mar-Markovian posteriors (V-posteriors) for discrete diffusion with arbitrary noise priors that share the same marginals as the Markovian discrete diffusion process (Sec. 3). (2) We demonstrate that the induced V V-samplers improve text and image generation quality and scale better than standard ancestral samplers in high NFE regimes, closing the performance gap with respect to MDMs coupled with remasking samplers in high NFE regimes for text generation (Sec. 5.1) and surpassing them on image generation tasks (Sec. 5.1.2). (3) We reformulate the curriculum learning strategy proposed in Sahoo et al. (2025a), achieving 2 speedup while reducing peak memory usage by 33% and end-to-end training time by 25%, while maintaining similar perplexity (Fig. 1, right, Table 5) and downstream task accuracy (Table 1).
BACKGROUND
Notation Let V = (v E [0,1)* E V = 1} denote the set of one-hot encodings of discrete random variables over K categories. Let D denote a sequence of L discrete variables in V and x denote the entry 4h in We use boldface denote both individual vectors and sequences; the context will make clear whether symbol refers to a vector or a sequence. Let A denote the K simplex. For A, let Cat(:; denote categorical distribution such that PP(u; = 1) v;, for u ~ V), u E V. Let (a, b) and a @ denote the dot and Hadamard products between two vectors respectively. Let = [1]* denote the all-ones vector. Let π e A be a designated categorical distribution referred to as the prior.
2.1 DISCRETE DIFFUSION MODELS
Consider the clean data sequence x of length L drawn from the data distribution Adma- Discrete diffusion models (Sohl-Dickstein et al., 2015; Austin et al., 2023) define a sequence of increasingly noisy distributions (q)).).. interpolating from qdaaa factorized prior distribution, which is product of T. independent Car(:; π) distributions, using Markovian transitions defined independently
2

---

Published as a conference paper at ICLR 2026
across imput dimensions (Campbell et al., 2022; Sahoo et al., 2024; Shi et al., 2025; Ou et al., 2025; Schiff' al., 2025; Sahoo al., 2025a). Let 2, ~ - (c(-Ix²) denote the intermediate latents (sequence) at time step f. This work focuses on factorized, interpolating noise processes (Sahoo et al., 2024), whose conditional marginal distribution takes the form:
z (-Ix'; = Cat(.; +
where E [0, 1] is monotonically decreasing with 1, and is known as the noise schedule. (1) defines the forward process, which progressively corrupts the data. The goal is to learn a generative process pe- parameterized by a neural network with parameters 0, that reverses this forward process to map from the noise prior back to qdata- The model is typically trained by minimizing the "Negative Evidence Lower Bound" (NELBO). The choice of token prior æ gives rise two popular variants: Masked Diffusion Models (MDMs) and Uniform-state Diffusion Models (USDMs), which we discuss in the following.
2.1.1 MASKED DIFFUSION PROCESSES
MDMs (Sahoo et al., 2024; Shi et al., 2025; Ou et al., 2025) use a masked prior, where = E V the one-hot representation of a special [MASK] token (Devlin et al., 2019). During the forward process (1), tokens either remain unchanged or transition to the masked state m, after which they stay masked. This behavior carries over to the reverse process. The posterior of the reverse process MDM for 0 S < t < 1 can be derived using Bayes' Rule, and is given by: qslt
Cat
op- x 1-as
zi 1-ai
if m. otherwise.
MDM Gale 12
x²)
I-ai
Cat(::x)
The approximate reverse posterior is Pa1 Ie gale [0, + A is the denoising model. A key limitation is that once unmasked, tokens cannot be remasked (2). This can create compounding errors during inference, as the denoising model x imperfectly models the clean data.
MDM (.Iz).
(x (()).,))) where x : V
Wang et al. (2025) propose posteriors, and associated samplers (ReMDM) that maintain the same marginals as (2) during the generation process, while allowing remasking and generalizing previous training-free free predictor-correctrr methods such as Campbell al. (2022); Gat et al. (2024).
Predictor-Correete Methods
2.1.2 UNIFORM-STATE DIFFUSION PROCESSES
Alternatively, discrete diffusion models can use a uniform prior π = 1/K (Schiff et al., 2025; Sahoo et al., 2025a). This choice allows tokens to change values multiple times throughout the generative process, in contrast to Masked diffusion. This property allows USDMs to excel in few-step generation (Sahoo et al., 2025a) and guidance applications (Schiff al., 2025).
USDM (for brevity, we simply write gale for USDMs admit the following posterior distribution d
USDM): )
4stt
Kazzi x (ars an) + (as ax)x + (1 a.)) - as)1/K) (1a)))1/KK/KK
[er(- 2 x(,x²)= Cat
(3)
Ko,(2f, x²)
This posterior induces the following NELBO (Sahoo al., 2025a):
NELBO (g,Po; E (x/ (x())))
Σ
x/ (), ox;x))
KE(L]
where
Rzi, xi ()xxxx)))
ac'
K
K
-0---(G (G.1,/-xe + Lagfx(
Σ log
(x6 )r
Kay
x
(x))r
(x//1
(xx)
1
log
xi - (K - 1- 1/ - 1
log Co
(5)
Cte
(xh)
Co
3

---

Published as a conference paper at ICLR 2026
Predictor
Corrector
Kt
(1
Kit)
a (x.1
V
Samplers
8 ri (x, x4
Oht (x' ())
.p- Samplers simplify and generalize previous Predictor-Corrector methods to arbitrary prior π
V-samplers combine predictor and corrector steps. The predictor transitions from xi 2, via galts but fails to remask tokens in MDMs. The corrector steps inject noise via ge. to revise earlier predictions. For R; noise injection enahles correction while preserving the forward process marginals. Our framework extends prior PC methods (Camphell et al., 2022; Gat et al., 2024; Wang et al., 2025) to arbitrary priors л.
Figure 2:
Orp 1,, af denotes the time derivative of (z)) the nonzero entry of Ze, St Ka.+1 +1a and denotes the index in corresponding to 1, that is, X; 1.
Here, x Ka,x + +1 01, x Kaxx Oz, r arg max j
The Diffusion Duality Sahoo et al. (2025a) show that USDMs emerge from an underlying Gaussian diffusion process (Sohl-Dickstein et al., 2015; Ho al., 2020b; Song al., 2021; Kingma al., 2023) on the one-hot representation x" E V. The Gaussian diffusion begins with x' and progressively adds Gaussian noise leading sequence of noisy latents w E IR (-(xx) for (0. 1] with the marginals:
(. (.)) = -i á,"', 10--a;z LR)
where (āt) is a monotonically decreasing noise schedule. Let arg max R + V map a continuous vector v E R to the one-hot vector corresponding to the index of its largest entry in v, that is, arg max(v) arg maxzev2 v. When applied sequence of Gaussian latents w, arg max transforms them to the discrete latents Z, whose marginals take the form: z 9x(1xx;aa == ((ā))) where the function [0, [0, 1] is the Diffusion Transformation Operator:
K
T(oi) K
$ K
(2)
(6)
VI
K
where ((2) exp(-22/22) 2 and ((2) respectively. More formally, this relationship is expressed as:
- (t))d are the standard Normal PDF and CDF,
(2/1xx;; T((;)) (arg max) (w: ;0)) (x))
where the * operator denotes the pushforward of the K-dimensional Gaussian density under the arg max map, yielding a categorical distribution with К classes. Note that while the marginal distribution qe (z, x; (ā.)) matches the discrete-space marginal in (1), this does not imply that the full trajectory {z, := arg max w. W.)oo follows a (Markovian) discrete diffusion process (Sahoo et al., 2025a). An interesting outcome of (7) is that the discrete NELBO (4) can be written terms of Gaussian latents in the following manner, where the second arg max is applied each token independently:
NELBO (g, Pe; x)
E Ex (0.
( [144 (E(L]
Σ
arg max(w() x (arg max((:)))))) := =T(ā):x'). (8)
Curriculum Learning Curriculum learning (Bengio et al., 2009) trains models by gradually increasing task difficulty. Building on this idea, Sahoo et al. (2025a) accelerate early training by using a biased but low-variance NELBO estimator for (8). Concretely, during the first 50% training steps, the hard arg max used to convert Gaussian latents into discrete tokens in the transformer's input is replaced by low-temperature softmax relaxation. This relaxation interfaces naturally with the transformer input layer: the latent at position a probability vector y' E A then the token representation is computed as a matrix product V Vy', where V E xm the vocabulary embedding matrix. For one-hot y (as produced by arg max), this reduces to standard embedding
4

---

Published as a conference paper at ICLR 2026
lookup; for softmax-relaxed latents, it becomes linear combination of vocabulary embeddings, As result, the model is no longer asked denoise from fully corrupted discrete token embedding, but instead receives an embedding that is a superposition of clean and noisy token embedding. This "partially clean" input provides a direct signal about the underlying token, making denoising easier than relying solely on the surrounding context. Fig. 3 (top) illustrates this curriculum. More formally, during the curriculum phase Sahoo et al. (2025a) optimize the following loss, where the softmax is applied independently at each position:
ttain Ex (12.
Σ := arg max (w.), X (ooftmax(w./r))r == T(a.):x). RE[L]
(9)
Notice that Luain in (9) reduces to the NELBO (8) in the limit lim,-0. for 3 = 0 and 7 = 1, since lim,- softmax(v/7) arg mvx(v), as shown by Jang et al. (2017); Maddison et al. (2017).
However, explicitly materializing the high- latents w, is inntnsy,, an issue we address in Sec.
2.2 DIFFUSION GUIDANCE
For continuous data, diffusion models have achieved state-of-thh-aar controllable generation through both baasiffier-baed guidance (Sohl-Dickstein et al., 2015; Dhariwal & Nichol, 2021) and Classifier- Free Guidance (CFG; Nichol & Dhariwal (2021); Ho & Salimans (2022)). These approaches have since been extended to discrete data (Gruver et al., 2023). Let y E (I,..., C}} denote one C possible classes. For CFG, the sampling posterior po (1), which modulates the strength of the guidance term via the temperature parameter y, is defined (Nisonoff et al., 2024; Schiff et al., 2025):
log po (1) (z', y, 74) = y log (2 y, 2+) + (1 - 7) log Po (z. 0. 2)); V E [7]]
(10)
where 0 denotes no class conditioning, and pe the generative posterior (Sec. 2.1).
THE T V-POSTERIORS
Multiple joint distributions can give rise to the same marginals as the discrete diffusion process defined in (1). In this work, we introduce family of posteriors, denoted V that share the same marginals as in (1); see Suppl. A.2 for details. These alternative generative processes are non- Markovian and apply both the Masked diffusion processes and to the Uniform-state diffusion processes. Specifically, we define the posteriors for the generative process as:
V Veje(-Ix |x" 2 = Kedo)o(-1z
(-12 x') + (1 kx)g. x'); Ve E [4]
(11)
where K2 E [0, 1] and v (1(x' = Cat(.ITr, with π m for MDMs and π 1/K for USDMs. (11) is thus linear combination of the forward process (1) and the reverse posteriors (2. 3) of standard discrete diffusion models. We therefore refer to these as superposition posteriors, or simply T V-posteriors.
U-Forward Processes Consider the interpolating diffusion process in (1) discretized into T steps. Let Zz(i) denote the latent variables at times t(i) i/T for 0 S i S T. The distribu- tion of trajectory Z0.1 factorizes independently over tokens as: (zo:1 x) = II. V (26 where ((6. (x)) = 1 (x II",, Voje(Z P(a)= x²). In what follows, we use 8, as shorthand for s(i), t(). respectively. The forward process can be derived from Bayes' rule: (z (z,] x')
(z, / (z) x²) (z) xx). Unlike the Markovian interpolating process in (1), this forward
process is generally not Markovian, since each zi may depend on both 2 and x
T-Reverse Processes In Suppl. A.1, show that the approximate reverse posterior takes the form: (+2))) (- x f)) + (1 me) [o-44]] X (z, t)) + (1 -.)m] (12)
where xa denotes the denoising model. We dub (12) as V-sampler. For (K+ = 1) ()) we recover the standard ancestral sampler defined in (2) for MDMs and (3) for USDMs. Notice that for Kr < 1, V corresponds to a noisier version of the ancestral sampler marginal Eatt- This is analogous to Predictor-Correcte methods in Gaussian diffusion (Song et al., 2021), where the corrector introduces additional Gaussian noise. In case, 4i plays the role of the corrector, while qolt acts as the predictor. The & V-posteriors also admit principled NELBO formulation (see Suppl. A.3), though this is not directly relevant for sampling.
5

---

Published as a conference paper at ICLR 2026
1
2
y'
3
Secistep ())
0.78
Duo tolomino "dog")
softmax w
12-
0.25
Momory (.)
K 20
W
expl ( (mm
1.00
-3%%
Duo (ours) tolammin ("dng"
612
6x
eating onder statedies flee 0.1)
Z
Note: The cumicuuum is appliee each position independently. denotes the position the sequonce.
Figure 3: Efficient Curriculum for USDMs. Duo (Sahoo et al., 2025a) replaces discrete lookups with linear combinations of all K embeddings: (1) Gaussian diffusion on one-hot representations, (2) Low-temperature softmax, (3) weighted sum. Duo++ exploits the sparsity of the tempered softmax (most weights are effectively zero), and simulate the k largest entries (out of K) using ordered statistics. The approximate normalizer Z admits a closed form expression (14). Duo ++ has 33 % lower memory and 25 % faster training than Duo.
different choices of {Ke [e recover previous lations in the literature (Campbell et al., 2022; Gat et al., 2024; Wang et al., 2025) (see Suppl. A. A for the proof). The v framework thus subsumes these samplers special cases, extending these predictor-corrector methods for discrete diffusion with any prior π.
Corollary For
formu-
Takeaway 1: V-samplers generalize prior Predictor-Correcte methods to arbitrary noise priors, subsuming ReMDM (Wang et al., 2025) and prior work (Campbell et al., 2022; Gat et al., 2024) as special cases.
Intuitive Explanation In practice, the denoiser x, imperfectly models the clean data The key to the effectiveness of W-sampler is the offset term (1 - (1 a,) ) in (12), which enables error correction during generation. For MDMs (π m). this offset allows previously denoised tokens to return to the masked state, unlike the ancestral sampler, which prevents remasking (see Sec. 2.1.1). Incorrect tokens can thus be replaced with better ones. For USDMs (π 1/K), the offset ensures every token has non-zero sampling probability. Even if the denoiser assigns near-zero probability to the correct token, the V-sampler gives a chance to appear, whereas ancestral sampling would not. While this offset may occasionally introduce incorrect tokens, the marginals of the Vr-amplerss (11) match those of the Markovian forward process (1), hence we converge to the correct distribution given sufficient samples.
SCALABLE CURRICULUM FOR FASTER TRAINING
Recall from Sec. 2.1.2 that the curriculum of Sahoo et al. (2025a) accelerates training by replacing discrete tokens as inputs to the transformer with softmaxed Gaussian latents. Naively, however, this requires materializing K K-dimensional weight vector for every token at every training step, which is infeasible for modern LLM vocabularies with K 100,000 (Touvron al., 2023; OpenAI, 2024). Our key observation is that Sahoo et al. (2025a) use very low temperature = 10 in (8). At such temperatures, the softmax concentrates almost all its mass on only few entries, making most of the K weights negligible. We exploit this induced sparsity by approximating the full linear combination using only & K embeddings (Fig. 3, bottom). We illustrate the process hereby:
Step Generating top-k entries in w w/o materializing Let [K] be the nonzero coordi- nate of the one-hot vector x i.e. (x") ) 1. Recall that w/ = â, + o,C, with o; = v o and e ~ (0, Ix). Therefore, (w(). a ~ (ā,, of) and (w() ~ (0, 0?). All coordinates in [A] exchangeable (i.i.d. with mean 0), while the coordinate o the only one with shifted mean. As

---

Published as a conference paper at ICLR 2026
result, the top-k set falls into one of two cases: Case 1: is not among the top-k all & winners
lie in [K] o. Case is among the top-k the winners are plus k indices from [K] \ Next, we describe how to sample the top-k values (and later their indices) without ever forming the full K-dimensional vector.
Let m K-1 and consider m i.i.d. samples W1, um ~ (0, o?). Rather than drawing all m values, we exploit the fact that order statistics of i.i.d. uniform random variables can be sampled recursively: the maximum of m ii.d 24 1] variables has CDF u so can be sampled directly. Conditioned the maximum, the remaining values also ii.d. uniforms, so the next-largest sampled the same way, and so on. Applying the inverse normal CDF + (b)(() Ôg converts the top-k uniform order statistics into the top-k Gaussian values (see Suppl. B.1.4 for details). We denote the result = (K1 2 ... 2 K+): where K. is the j-th largest among the K zero-mean coordinates.
be
We draw independently the "special" entry (ā, of), which matches the distribution of (w))o ) Now compare w to the current k-th largest value among the zero-mean coordinates: If K& > w then o cannot enter the top-k. We are in Case 1, and already contains the correct top-k values. If Φ > KR. then o must be in the top-k. We are in Case 2. Let r K E [K] K, > x be the number of values in K. that are larger than ú. We insert ① at rank + and drop the previous smallest value: + K1.- K, where denotes concatenation.
For clarity, let ($) denote the of all possible tuples with distinct elements in the set S. We generate the index tuple I corresponding the top-k values K. as follows: Case 1: all top-k
indices lie in [K] o; hence, I~ (=((()()))) Case 2: the index o appears position r + in X. Sample the indices for the remaining k- 1 values from [K] o, split into those above and below == [o) where L (1)) R (KMM)))) their matching indices while sampling only (kk) random variables, without constructing the full K K-dimensional vector.
k-r-1 This produces both the top-k values and
Step 2: Approximating the softmma Given the top-k values and indices (KC, from Step approximate the -oeimax-weighted embedding vector by retaining only the selected rows of the embeddings Rxxd (d denotes the embedding size):
softmax(w!) embeddings exp(K./TT embeddings([.)
(13)
2
where embeddi [gs]] denotes the j-th row. The normalizer 7 includes both sampled (top-k) and unsampled terms. While each unsampled term is small, their sum may be non-negligible, hence approximate it as (Suppl. B.2):
K+ Ōe
Kr - -6700
Ž
~
exp
+ 8 exp
+ (K- - k 8) exp
2r
log &
log &
Ōt
top-k terma
token
unsampled mero-mien terms
(14)
where 8 = 1f K. case 2 and 0 otherwise. We provide the full derivation in Suppl. B.2 and pseudocode in Algo. 3.
Lastly, the curriculum objective in (9) requires evaluating the diffusion transformation operator T(-). Directly computing T via (6) is prohibitively expensive during training; Sahoo et al. (2025a) therefore precompute and cache many (a,, T (á.)) pairs, which is cumbersome. Instead, we compute T(-) on the fly using its Taylor expansion; see Suppl. B.3.1.
EXPERIMENTS
We evaluate + with U-samplers on language modeling (Sec. 5.1.1) and image generation (Sec. 5.1.2,, showing that V-samplers markedly improve text and image quality, making USDMs
7

---

Published as a conference paper at ICLR 2026
Table 1: Accuracy on multiple-choice question answering datasets. Abbreviations: Are-e (ARC-Easy), Are-c (ARC- Challenge), HSwag (HellaSwag), WinoG (Winogrande), PIQA (Physical Intelligence Question Answering). OQA (OpenBookQA). 'Results from Deschenaux et al. (2025). Duo++ (k = 2) achieves slightly higher accuracy than Duo on 4 out of 6 tasks. Overall, Duo matches Duo's performance while using 25 % fewer flopss The highest accuracy among USDMs bolded. The absolute best per column is underlined.
Arc-e
Arc-c
HSwag
WinoG
PIQA
MathQA
OQA
AR Transformer
44.95
23.04
30.55
52.80
63.71
22.24
19,00
MDLM
34.26
24.66
31.54
51.93
57.89
20.70
28.60
Duo
28.11
25.43
26.46
47.20
51.14
20.00
23.40
Duo ++ (k ===3 2)
27.32
26.11
26.26
49.64
52.12
20.40
27.80
Duo++ (k 3)
28.28
25.00
25.89
47.36
50.65
21.01
23.00
Duo ++ (k 5)
28.03
25.77
26.90
50.12
51.25
20.20
25.40
outperform MDMs in sample quality. In Sec. 5.2, we show that Duo++ matches Duo (Sahoo et al., 2025a) while using 33% less memory and training 25% faster, enabled by our efficient curriculum strategy (Sec. 4).
5.1 V-SAMPLERS
5.1.1 LANGUAGE MODELING
Takeaway 2: V-samplers substantially improve Generative Perplexity for USDMs, with gains especially pronounced when NFEs exceed the sequence length.
Unlike ancestral sampling, which plateaus, L V-samplers continue to improve with more sampling steps, closing the gap with Masked diffusion models.
Takeaway 3:
Experimental Settings We compare MDLM (Sahoo et al., 2024) and
1.00
ReMDM (Wang et al., 2025) with Duo and V -asamplers. We use the original checkpoints of Sahoo al. (2024), trained for 1M steps with batch size of 512 on Open WebTex (OWT; Gokaslan & Cohen (2019)) and context length I. 1024. Duo + is trained with the same context length, batch size and number of steps, but with the efficient curriculum, Refer to the original works for more details. We measure the sample quality using the Gen. PPL (L) computed with GPT-2 Large (Radford al., 2019) and the diversity using the unigram entropy (t) (Dieleman et al., 2022; Sahoo et al., 2024; 2025a). We cast logits to 64-bit precision for sampling (Zheng et al., 2025). See Suppl. for more details.
0.75
0.50
0.25
0.00
250
500
750 1000
samping
Figure 4: Illustration of the possible evolution of f and the associated Kę- In prac- tice, we use Ks close during the PC phase.
Results and Ablation Fig. (left) shows the Gen. PPL and entropy as function of the NFE. +uo++ with & --samplers outperforms MDLM with ReMDM and ancestral sampling across the entire range of NFEs. As the number NFEs increases beyond the sequence length, ReMDM and V-samplers further improve sample quality while ancestral sampling plateaus. We ablate on the schedule type (cap, rescale, loop, see Suppl. A.5), the step-size parameter 77, the nucleus sampling threshold p {0.9, 0.95, 1.0} (Suppl. D. The rescale schedule with 7 0.05 yields the best Gen. PPL while preserving the unigram entropy. Nucleus sampling (p=0.9) consistently improves Gen. PPL for both MDLM and Duo, observed in Wang et al. (2025).
How to Pick Ky? We recommend the ReMDM rescale schedule with 17 0.05 and nucleus sampling (p = 0.9), using the log-lineat noise schedule with linearly decreasing t, which outperforms the "loop" strategy (Suppl. A.5).
8

---

Published as a conference paper at ICLR 2026
5.1.2 IMAGE MODELING
Takeaway 4: On CIFAR-10, Duo with y-samplers achieves better FID and IS than MDLM with both ancestral sampling and ReMDM.
Experimental Setup We train the same 35M- U-Net as Austin al. (2023) on CIFAR- for 1.5M steps. Following Schiff et al. (2025), the U-Net is made class conditional and sample with Discrete Classifier-free Guidance (CFG; Ho Salimans (2022); Schiff et (2025)). See Suppl. C.I for full training details. We report the Fréchet Inception Distance (FID; Heusel et al. (2018)) and Inception Score (IS; Salimans et al. (2016)) between the training set and generated samples.
Results and Ablation Fig. (right) and Fig. 6 show that V-samplers and ReMDM substantially improve FID and IS compared to ancestral sampling, with Duo++ reaching the best scores overall. We ablate the sampling noise schedule (cosine vs. log-linear), Kg, the activation range [toffs toa]. nucleus sampling, and the K2 schedules (including the ReMDM variants; see Suppl. A.5). Full results in Suppl. D. Using K₃ close (light noise injection) with cosine schedule achieves the best FID and IS, and Duo tolerates stronger noise injection than MDLM. With ancestral sampling, nucleus sampling improves the FID in the low NFE regime for MDLM, and always helps for Duo. Since it is detrimental to MDLM at high NFE, we do not use nucleus sampling when using the -asmplerrs, for both MDLM and Duo.
How to Pick Kr? We recommend a cosine sampling schedule with K; 0.95, fon E {0.5, 0.6), loff 0.1 for +uo++ and Kr 0.99, fon = 1.0, foff 0.1 for MDLM. We suggest using a piecewise constant K2 with linearly decreasing t. rather than the ReMDM loop schedule (Suppl. A.5), setting Kg < when E [tom, toal, since it outperforms the ReMDM schedules.
5.2 FAST CURRICULUM
Takeaway 5: The efficient curriculum reduces peak memory by 33% and training time by 25%, while matching the performance of Duo on likelihood benchmarks and downstream tasks.
Experimental Settings We train Du+ with the scalable curriculum (Sec. 4) on WpenWeeText (OWT; Gokaslan & Cohen (2019)) and LMIB (Chelba et al., 2014). We train all models for IM steps, using a batch size 512. For LMIB, tokenizer with a context length of 128, padding shorter sequences. This setup follows previous work (Sahoo al., 2024; Lou et al., 2024; He et al., 2022). For OWT, we the GPT-2 tokenizer (Radford et al., 2019), and re- serve the last 100k documents for validation, following Sahoo al. (2025a; 2024). We follow Lou al. (2024) and use a modified diffusion transformer (DiT) (Peebles & Xie, 2023) with rotary positional encoding (Su et al., 2023). We evaluate the impact of & = {2,3, 3, 5) during the efficient curriculum. All models are trained on H100 GPUs with 6 precision. Training uses the loss in (9), with r = 0.001 and (8,7) = (0.03, 0.15) for the first 500K steps (Sahoo et al., 2025a).
Table 2: Test perplexity (PPL.) on LMIB and OWT. Lower is better. 'Results from Sahoo et al. (2025a). Best Uniform-star dif- fusion numbers bolded. Duo and Duo+ achieve comparable performance across both datasets while requiring 25% fewer GPU- hours (Table 4), demonstrating the effective- ness of our memory-efficient curriculum.
use the
ea
LMIB
OWT
Autoregressive Transformer'
22.3
17.5
Masked Diffusion
SEDD Absorb'
32.7
24.1
MDLM
27.0
23.2
Uniform-state Diffusion
SEDD Uniform'
40.3
29.7
UDLM!
31.3
27.4
Likelihood Results Table 2 shows that on both LMIB and OWT, our efficient curriculum +++ matches the performance of Duo with its expensive curriculum. The lowest validation perplexity is achieved with = 2. al-
Duo
29.9
25.2
Duo (Ours), A = 2
30.0
25.2
Duo (Ours),
30.1
25.3
Duo (Ours), = 5
30.2
25.4
though & E (2, 3, 5) performs similarly. We also compare

---

Published as a conference paper at ICLR 2026
the models trained on OWT in Zero-Sho perplexity, and find that Duo achieves performance comparable to Duo. That is, we evaluate on the validation splits of the Penn Treebank (Marcus et al., 1993), WikiText (Merity et al., 2016). LMIB (Chelba et al., 2014), LAMBADA (Paperno et al., 2016), AG News (Zhang et al., 2016) and scientific articles from ArXiv and PubMed (Cohan et al., 201 8). Table 5 shows that Duo++ reaches a zero-shot probability similar that of Duo while requiring 25% less training GPU-hours.
Likelihood-basee Downstream Tasks Table 1, we compare the multiple-choice question (MCQ) accuracy of Duo, Duo++. MDLM (Sahoo et al., 2024), and an autoregressive transformer (1M training steps with a batch size of 512 on OWT, same hyperparameters as MDLM) using the 1m-eval reval-harnee rness suite (Gao et al., 2024): details in Suppl. C.3). We find that Duo achieves an accuracy similar to that of Duo, despite requiring 25% less training GPU-hours. However, trails MDLM on most tasks, consistent with its higher perplexity.
Throughput and Peak Memory Usage Table 4 reports the throughput and peak memory usage for Duo and Duo++ Duo++ reduces the peak memory usage by about 33% and doubles the speed of the Curriculum Learning phase. When applying Curriculum Learning for half of the training steps, Duo + trains 25% faster than Duo on the 3-0--aammeeer scale. Notably, both peak memory usage and throughput remain stable over the full training run when & E (.).
6 RELATED WORK AND DISCUSSION
Compatibility with General Discrete Diffusion Processes This work focuses on discrete diffusion with uniform or masked noise. However, our approach extends to more general discrete diffusion processes (Shaul et al., 2024; von Rütte et al., 2025; Holderrieth et al., 2025) featuring a combination of masked and uniform prior, since we provide a general predictor algorithm for discrete diffusion with arbitrary noise.
Predictor-Corrector Samplers In the context of Masked diffusion, ReMDM (Wang et al., 2025) generalizes previous predictor-corrector methods (Campbell al., 2022; 2024; Gat et al., 2024) that were based on Continuous Time Markov Chain formulation of discrete diffusion processes. Our approach further generalizes ReMDM support arbitrary diffusion processes. Unlike Lezama et al. (2023); Zhao et al. (2025); Liu et al. (2025); Kim et al. (2025), who train an additional corrector module, our method does not introduce additional learned components.
Comparison to Other Discrete Diffusion Samplers Park al. (2024) uses noise-adaptive step sizes; while we use uniform steps, --samplers support any step-size schedule. Ren et al. (2025) develops higher-order samplers; use only first-order information, though the posterior in (11) could be approximated with higher-order methods. Thus, V-samplers complement both lines of work.
7 CONCLUSION
We introduced a unified and practical framework for mrdiccoo--oorrector sampling in discrete diffusion language models through T V-posteriors. By linearly superposing the forward and reverse diffusion processes (11), the V-posteriors preserve the marginals of standard diffusion models. Importantly, the V-posteriors and associated V-samplers subsume prior masked-diffusion PC samplers (Campbell et al., 2022; Gat et al., 2024; Wang et al., 2025) as special cases, and naturally extend to discrete diffusion models with uniform prior. Empirically. Duo++ with --samplers matches the performance of MDMs on natural language generation and achieves stronger FID and IS scores on CIFAR-10 10. Moreover, they exhibit superior scaling: performance continues to improve with NFEs, unlike ancestral samplers, which plateau. Finally, we propose a scalable training curriculum (Sahoo et al., 2025a) that reduces the peak memory usage by 33% and shortens the training time by 25%. Concurrently, Sahoo et al. (2026) show that Duo surpasses autoregressive model at the 1.7B scale on the math and reasoning benchmark (GSM8K). Taken together, these results challenge the view that Masked diffusion is categorically the future of diffusion language modeling.
10

---

Published as a conference paper at ICLR 2026
8 ACKNOWLEDGEMENTS
This work has received funding from the Swiss State Secretariat for Education, Research and Innovation (SERI). We are grateful Ricky T. Q. Chen and Zhihan Yang for insightful discussions and suggestions. We acknowledge the SCITAS team at EPFL for providing access to their cluster, and the Swiss National Supercomputing Centre for the Alps platform. We are grateful to Karin Gétaz for her administrative assistance.
REFERENCES
Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces, 2023. URL https: //trr////////////a org.//voorrr.o.oooooooooo abs/ 2107.03000
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Inter-
national Conference on Machine Learning, 2009. URL hwtps:t/t//eeeee httppppp...emmtniscscsoooooooooo
org/Corpus ID: 873046.
Jon Louis Bentley. Programming Pearls Addison- Wesoly Professional, Boston, MA, 2nd edition, 1999. ISBN 978-0201657883
Roger Berger and George Casella. Statistical Inference. Duxbury Press, Florence, AL, 2 edition, June 2001.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2255--22575 2023.
Kim C. Border. Lecture 15: Order statistics; conditional expectation. atttps:/hheat
econ. www.ohiccsse www.ohhttteee.oooooooo wttpsssttteeeetttttteeeeeeeeeeee edu/kcb/ wwtpkk/aa////te///////e/a//ea/e/ wtt////eeee///teeeeeeeeeee pdf, 2021. Course notes for MA103.
Andrew Camphell, Joe Benton, Valentin De Bortoli, Tom Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models, 2022. URL
https:://axiv.. attg/////220..1110/552
Andrew Camphell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola. Generative flows on discrete state-spaces: Enabling multimodal flows with applications protein co-design, 2024. URL https: arwivv.vvvv. org/orgggoo..... abs /2402/4222 04997.
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling, 2014. URL https: //txxv/axvvvv/////////////ee///a org/ abs / 3121. 2211..300 3005.
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui. Seokhwan Kim, Walter Chang, and Nazli Goharian. A discourse-aware attention model for abstractive summarization of long documents. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 615-621, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 1/N18 //aclanthology wtg.N18-2///././/111111111
URL https:
Justin Deschenaux and Caglar Gulcehre. Beyond autoregression: Fast Ilms via self-distillation through time, 2025. URL https: httttp::/aaxxvv.ooo org. abtpaabs/...2..//
Justin Deschenaux, Lan Tran, and Caglar Gulcehre. Partition generative modeling: Masked modeling without masks, 2025.
https:::: http:stst//t////ee/ http://////vvvvaa
/2bs//255... 18883.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. URL https: / /arxiv.org/ bbs//880 0. 04805.
11

---

Published as a conference paper at ICLR 2026
Lue Devroye. Non- Non-Uniform Random Variate Generation. Springer- Verlag, 1986. URL https:
//www. springer. com/ htm//////////ooo//ooo////// htt////bbboot htppbook/////ooo/ppoopooo.ooobc/ 9780387963051.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8777 -004,, 2021.
Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H. Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, Curtis Hawthorne, Rémi Leblond, Will Grathwohl, and Jonas Adler. Continuous diffusion for categorical data, 2022. URL https: //rxivv org/ alss 2211 15089.
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7346-7866 -356,, 2023.
Gerald B Folland. Real analysis. Pure and Applied Mathematics: Wiley Series of Texts, Mono- graphs and Tracts. John Wiley & Sons, Nashville, TN, 2 edition, March 1999.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPoñ, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. URL https: /wwoooeooooooo////////oooooo org/ records /1121626662222
Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky T. Q. Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching. 2024. URL htt.h/tt////rrvv..rr////ba/ 15595.
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus, http::/99KyLLoon00..gthub OpenWebTexttCorpus. 2019.
Nate Gruver, Samuel Stanton, Nathan Frey, Tim GJ Rudner, Isidro Hotzel, Julien Lafrance- Vanasse, Arvind Rajpal, Kyunghyun Cho, and Andrew G Wilson. Protein design with guided discrete diffusion. Advances in neural information processing systems, 36:1 28s112121221,, 2023.
Zhengfu He, Tianxiang Sun, Kuanning Wang, Xuanjing Huang, and Xipeng Qiu. Diffusionbert: Improving generative masked language models with diffusion models, 2022. URL https: 15029.
//arxiv..rgg/as/////
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium, 2018. URL https: //tp///aavvv///////////ee//a.av. org/ abs 08500.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance, 2022. URL www.hhttp///aaete.
org/abs /2022222222 12598.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020a.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, 2020b. URL https: /arxivvv.vv wwt./aa./////oooooo//..o.//aaaea aawp///0/5050 11239.
Jonathan Ho. Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. arXiv:2204.03458, 2022.
Peter Holderrieth, Marton Havasi, Jason Yim, Neta Shaul, Itai Gat, Tommi Jaakkola, Brian Karrer, Ricky Q. Chen, and Yaron Lipman. Generator matching: Generative modeling with arbitrary markov processes, 2025.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration, 2020. URL https: / /xwrxivvvvv/ wwt.vv.oorg/sssse 904 09751.
12

---

Published as a conference paper at ICLR 2026
Qingqing Huang, Daniel S. Park, Tao Wang, Timo I. Denk, Andy Ly, Nanxin Chen, Zhengdong Zhang, Zhishuai Zhang, Jiahui Yu, Christian Frank, Jesse Engel, Quoc V. Le, William Chan, Zhifeng Chen, and Wei Han. Noise2musie: Text-conditioned music generation with diffusion models, 2023. https:::: / ht/pariiiv.v/// org/ abs absaabb/. 2302.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax, 2017.
URL wwwpt:tp/////..... /taps:://axiiv http://wwwwvvw//aaaae/// wttg/ab/////ooooro 1611.1111001 01144.
Jaeyeon Kim, Seunggeun Kim, Taekyun Lee, David Z. Pan, Hyeji Kim, Sham Kakade, and Sitan Chen. Fine-tuning masked diffusion for provable self-correction. arXiv preprint arXiv:25/0.01384, 2025.
Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models, 2023.
URL https: /tl////zaa /aaxiviivv. org/ aass/2011 00630.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations,
2021. URL https: opwnreview.eeeeeeeeeeeeee. net/
Pin-Jui Ku, He Huang, Jean-Marie Lemercier, Subham Sekhar Sahoo, Zhehuai Chen, and Ante Jukić. Discrete diffusion for generative modeling of text-aligned speech tokens. arXiv preprint arXiv:2509.208 20060, 2025.
Inception Labs, Samar Khanna, Siddhant Kharbanda, Shufan Li, Harshit Varma, Eric Wang, Sawyer Birnbaum, Ziyang Luo, Yanis Miraoui, Akash Palrecha, et al. Mercury: Ultra-fast language models based on diffusion. arXiv preprint arXiv:2506. 17298, 2025.
Seul Lee, Karsten Kreis, Srimukh Prasad Veccham, Meng Liu, Danny Reidenbach, Yuxing Peng, Saee Paliwal, Weili Nie, and Arash Vahdat. Genmol: drug discovery generalist with discrete diffusion. arXiv preprint www.:::501.06/111 2025.
Jose Lezama, Tim Salimans, Lu Jiang, Huiwen Chang, Jonathan Ho, and Irfan Essa Discrete predictor-correcto diffusion models for image synthesis. In The Eleventh International Confer- ence on Learning Representations, 2023. URL https:: httpss////pppeeeeeeeeeeee net wwweeeorrmeeeeee forum?id= VMBbatVBWvg.
Chengyi Liu, Wenqi Fan, Yunqing Liu, Jiatong Li, Hang Li, Hui Liu, Jiliang Tang, and Qing Li. Gen- erative diffusion models on graphs: Methods and applications. arXiv preprint arXiv:2302.0253) 2023a.
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: text-to-audio generation with latent diffusion models. In Proceedings of the 40th International Conference on Machine Learning, pp. 21450-21474, 2023b.
Sulin Liu, Juno Nam, Andrew Campbell, Hannes Stärk, Yilun Xu, Tommi Jaakkola, and Rafael BombelBo Think while you generate: Discrete diffusion with planned denoising, 2025.
URL httptttts///xiiie org/ abs/ 22122410.00 06264.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019, URL https:
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete difffusion modeling by estimating the ratios of the data distribution, 2024. URL https: www.aaiwwwa.aaaaeeeaaeoaeeeaaaae /t1.ab/../..............
Chris J. Maddison, Andriy Mnih and Yee Whye Teh. The concrete distribution: A continuous relax- ation of discrete random variables, 2017. URL ht http:::::///....////.o/./o wttp/o//////....o/a///////////la 00712.
Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313333) 1993. URL (argg/93320044
https: bictps://aclen https/////laaaahhllccoc
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016. URL //////xxvvvv org/ abs 609. 07843.
13

---

Published as a conference paper at ICLR 2026
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models, 2021. URL /taps:://axivvi org/ alss/210000 09672.
Hunter Nisonoff, Junhao Xiong, Stephan Allenspach, and Jennifer Listgarten. Unlocking guidance for discrete state-space diffusion and flow models. arXiv preprint arX1v:2406.01557, 2024.
OpenAl. Gpt-oss: open-weight language models by openai. gpt-oss, 2024. GitHub repository.
Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data, 2025.
URL httph:tt///eette //arsv////iivvv///a/ org/ abs/ 2406. 03736.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context, 2016. URL 1606. 06031.
org/ abs/
Yong-Hyun Park, Chieh-Hsin Lai, Satoshi Hayakawa, Yuhta Takida, and Yuki Mitsufuji. Jump Your Steps: Optimizing sampling schedule of discrete diffusion models, 2024. URL
https: httpaaaaiv..g....a....../.. artg/bs////aaaaoo 22700.7761.
William Peebles and Saining Xie. Scalable diffusion models with transformers, 2023, URL https:
//arxiv. owg.o/g///aooo abs/ 221.222222.. 09748.
Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma. Ching- Yao Chuang, David Yan, Dhruv Choudhary, Dingkang Wang, Geet Sethi, Guan Pang, Haoyu Ma, Ishan Misra, Ji Hou, Jialiang Wang. Kiran Jagadeesh, Kunpeng Li, Luxin Zhang, Mannat Singh, Mary Williamson, Matt Le, Matthew Yu, Mitesh Kumar Singh, Peizhao Zhang, Peter Vajda, Quentin Duval, Rohit Girdhar, Roshan Sumbaly, Sai Saketh Ramb- hatla, Sam Tsai, Samaneh Azadi, Samyak Datta, Sanyuan Chen, Sean Bell, Sharadh Ramaswamy, Shelly Sheynin, Siddharth Bhattacharya, Simran Motwani, Tao Xu, Tianhe Li, Tingbo Hou, Wei- Ning Hsu, Xi Yin, Xiaoliang Dai, Yaniv Taigman, Yaqiao Luo. Yen-Cheng Liu, Yi-Chiao Wu, Yue Zhao, Yuval Kirstain, Zecheng He, Zijian He, Albert Pumarola, Ali Thabet, Artsiom Sanakoyeu, Arun Mallya, Baishan Guo. Boris Araya, Breena Kerr, Carleigh Wood, Ce Liu, Cen Peng, Dimitry Vengertsev, Edgar Schonfeld, Elliot Blanchard, Felix Juefei-Xu, Fraylie Nord, Jeff Liang, John Hoffman, Jonas Kohler, Kaolin Fire, Karthik Sivakumar, Lawrence Chen, Licheng Yu, Luya Gao, Markos Georgopoulos, Rashel Moritz, Sara K. Sampson, Shikai Li, Simone Parmeggiani, Steve Fine, Tara Fowler, Vladan Petrovic, and Yuming Du Movic gen: A cast of media foundation models, 2025. URL https: arxiv. org/ /bss/410... 13720.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019.
Yinuo Ren, Haoxuan Chen, Yuchen Zhu, Wei Guo, Yongxin Chen, Grant M. Rotskoff, Molei Tao, and Lexing Ying. Fast solvers for discrete diffusion models: Theory and applications of high-order algorithms, 2025. URL https: arxiv. org/ absg///bb//aa/ /2w...2522222 00234.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models, 2022. htt.httss///axii... abs/2112. 10752.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation, 2015. URL httpt::/t/aaat. /ttpss://aaavvvoaa org/ abs/1505 04597.
Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models, 2024. URL https: /ttp///aaavvvv/vv///////e../.aaa wwwp//////////ee/eoeooo/eeeeeeee 220..4406..00 07524.
Subham Sekhar Sahoo, Justin Deschenaux, Aaron Gokaslan, Guanghan Wang, Justin T Chiu, and Volodymyr Kuleshov. The diffuuson duality. In Forry-second International Conference on Machine Learning, 2025a. URL https: htt.ttsss//peeeevieee.....
14

---

Published as a conference paper at ICLR 2026
Subham Sekhar Sahoo, Zhihan Yang. Yash Akhauri, Johnna Liu, Deepansha Singh, Zhoujun Cheng, Zhengzhong Liu, Eric Xing, John Thickstun, and Arash Vahdat. Esoteric language models. arXiv preprint 1XX1:::50000192666 2025b.
Subham Sekhar Sahoo, Jean-Marie Lemercier, Zhihan Yang, Justin Deschenaux, Jingyu Liu, John Thickstun, and Ante Jukie. Scaling beyond masked diffusion language models. arXiv preprini arXiv:2602.15014, 2026.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen, Im- proved techniques for training gans, 2016. URL https: wtt.aaawwo.gg.oaeaaeae/aa
606 03498.
Yair Schiff, Subham Sekhar Sahoo, Hao Phung, Guanghan Wang. Sam Boshar, Hugo Dalla-torre, Bernardo P. de Almeida, Alexander Rush, Thomas Pierrot, and Volodymyr Kuleshov. Simple guidance mechanisms for discrete diffusion models, 2025. URL https: httttp://arxiv.orrt wwwxiiv.org/aaa 2412. 10193.
Neta Shaul, Itai Gat, Marton Havasi, Daniel Severo, Anuroop Sriram, Peter Holderrieth, Brian Karrer, Yaron Lipman, and Ricky T. Q. Chen. Flow matching with general discrete paths: A kinetic-optimal perspective, 2024. URL
/ttp:/:///xvvvvv/ve abs/ 2412.2222 03487.
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K. Titsias. Simplified and generalized masked diffusion for discrete data, 2025. URL 2406. 04329.
Jascha Sohl-Dickster Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics, 2015. URL https: /ttp:///////ii////a http:///wwwwww//aaaaaa//o org/ abss/ 1503. 03585.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, 2022. URL https: / /arxiv org/ wttg/bs/////oooooooo.oooo/.eeee/ abs///11s11 02502.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021. URL https: /wwprreeeeeveeeeeeee ew. net. forum?
11-PXTTIG12RHHS
Yuxuan Song, Zheng Zhang, Cheng Luo. Pengyang Gao, Fan Xia, Hao Luo, Zheng Li, Yuehang Yang, Hongli Yu, Xingwei Qu, et al. Seed diffusion: large-scale diffusion language model with high-speed inference. arXiv preprint 2rXiv:2508.02193, 2025.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023. URL https::t//eee.. http://wwwwwwww.a//////o/ 09864.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu. Yuning Mao. Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nic, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https:/ / arxiv. httg/bbs/2320 09288.
Dimitri Rütte, Janis Fluri, Yuhui Ding, Antonio Orvieto, Bernhard Schölkopf, and Thomas Hofmann. Generalized interpolating discrete diffusion, 2025. URL https: :tappp:///xxxt //trxiv.....////// org/ abs/2503. 04482.
15

---

Published as a conference paper at ICLR 2026
Guanghan Wang, Yair Schiff, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Remasking discrete diffusion models with inference-time scaling, 2025. URL httpppss//aaaaw.oo////bbaa 00307.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification, 2016. URL https:////// //
01 626.
Yixiu Zhao, Jiaxin Shi, Feng Chen, Shaul Druckmann, Lester Mackey, and Scott Linderman. In- formed correctors for discrete diffusion models, 2025. URL https://arxiv..rrt /abs/ 2407.21243
Kaiwen Zheng. Yongxin Chen, Hanzi Mao, Ming- Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. 2025. URL https: /arxiv org//b/g//.... aas//b////ee 2209 02908.
16

---

Published as a conference paper at ICLR 2026
CONTENTS
Introduction
2 Background
2.1 Discrete Diffusion Models
2.2 Diffusion Guidance
3
The y-Posteriors
4
Scalable Curriculum for Faster Training
5
Experiments
5.1
V-Samplers
5.2
Fast Curriculum
Related work and Discussion
10
7
Conclusion
10
8
Acknowledgements
11
V-Posteriors
18
A.1
Approximate Reverse Marginals
18
A.2 Proof that the V-posteriors have the correct marginals
18
A.3 Negative Evidence Lower Bound
19
A.4
Recovering Predictor-Corrector Methods for Masked Diffusion
19
A.5
ReMDM Sampling Schedules
20
Fast Curriculum
20
B.1
Sampling the top-k values in w Without Materializing The Full Vector
20
B.2
Approximating the Weighted Sum of the Embeddings
24
B.3 Efficient computation of T During Training
25
B.4 Implementation of the Fast Curriculum
28
C
Experimental Details
29
C.I V-samplers
29
C.2 Improved Curriculum
30
C.3 Downstream Evaluation Protocol
30
C,4 Zero-Shot Likelihood
31
Additional Experimental results
31
D.1 Tuning K for the V-samplers
31
17

---

Published as a conference paper at ICLR 2026
D.2 Distribution of the top i entries of the softmax
32
D.3 Training Efficiency of Our Fast Curriculum
32
A V-POSTERIORS
A.1 APPROXIMATE REVERSE MARGINALS
We parameterize the (generative) V-reverse marginals to have similar form as the true posterior (11). Therefore, the generative reverse marginals also factorize the sequence length. Because x is not available during sampling, there are two terms (11) that are intractable. First, we choose to cannot sample from go (-Ixx)) without x', we replace x" by Goje [Zz, x x6), Ve [L]. Replacing these two intractable terms yields our generative reverse marginals:
replace the posterior q l x²) by [sje[-|2f, x x))- Additionally,
(1)(()))))= = Kegs)e(//ze. x = xo(z,, t)) + (1 - ki) [a=90]e(112,,> xo t)) + (1 a2)a
(15)
Note that for the masked posterior (2), (.je(, Xo (z( ()) xo(Ze, t).
A.2 PROOF THAT THE V-POSTERIORS HAVE THE CORRECT MARGINALS
Let [_je(-/xx' 2 denote the L W-posteriors defined in (11). Let denote s(k) t(k 1) and t denote ((k). To prove that the W-posteriors have the correct marginals, we proceed by (downwards) induction, similar to Song et al. (2022). First, note that v Vz(z[lx' can be written as a marginalization over 2/. for < t:
v (z(x(
Σ Vr(2 J s// (z,
(16)
Base Case Let (z) (x) denote the marginal time By definition in (11), (2(2((x') Cat(./T) Therefore, the V-posteriors have the correct marginal for t = 1.
Induction hypothesis Suppose that the V-posteriors have the correct marginal for a certain that is, Vx(+Ix') = (+(xx)).
Inductive step Based the induction hypothesis, show that v,((xx') -=((1xx), for s(k) = t(k- 1). Indeed
V. (.(x') (1)
U W.je(2
Ye (2) xx V (2
x
(3)
q: (z (2/1x²)) xx))
[
ž + (1 (x,1
(4)
16 12 Xoixx)Maeee00
2 + (1 (2 Σ (2(2)(x))
z
Kegs (z.
(z)) x²) ) x1-
q, (z. (x*)
Specifically, (1) hold by (16), (2) by the induction hypothesis, (3) by definition of the V V-posteriors, (4) by distributing ge (2] x²), (5) by definition of marginal probability (first term), and by observing that Σx; (x since di is normalized. This concludes the inductive step, and shows that the V-posteriors have the correct marginal.
18

---

Published as a conference paper at ICLR 2026
A.3 NEGATIVE EVIDENCE LOWER BOUND
Let z0.1 denote a reverse trajectory with time indices {0, 7. tion of (x', . under the generative model factorizes as
2 T...... 13 for token &. The joint distribu-
T II
p fx 1 = p(x V
1 (2))
o s/c
(2)(0) 1
(17)
where each pair (((i),(()) denotes one reverse transition with s(i) < t(i). The marginal likelihood
p" (x(x) - Ex"(x' 2.:1)
(18)
z0.1
Introducing the variational distribution Vzz6.11 x²) = (zf xx)) I Voie(2 Jensen's inequality results in:
x x")
-logp" (xgg"(x') S ye be) [ logp(x 20]] + KL ( x))|(11)
(19)
Σ Expeetco (xe)) DKLL (Vop( R((o) x')
2 20))
(20)
This expression is similar to the standard diffusion NELBO, with a reconstruction term, prior term at 1= and a sum of KL divergences. As T + oc, p(x" z concentrates around x", hence -log p(x 2 + Furthermore, the prior term is zero by definition of the V-posteriors in (11).
A.4 RECOVERING PREDICTOR CORCCCTRR METHODS FOR MASKED DIFFUSION
Wang et al. (2025) introduce the ReMDM posterior, which generalizes the MDM posterior (2) by allowing previously decoded tokens to be remasked. For a given position l, the ReMDM posterior
{
Cat (1 -,,) + 0, m)
z x m,
4a (z.
z x²)
(21)
Cat a. (1-(1-a)) xf + 1-a,-o;a; 1-a,
m
z m,
1-0,
where E [0, of muuu is free parameter that controls the remasking probability. The upper bound optaax = min 1, (1 - aa)/art) ensures that (21) defines a valid distribution. When o, = 0, the ReMDM posterior reduces to the standard MDM posterior. Below, we show that the V Vopostrris recover the ReMDM posterior with the substitution K2 = 1 - 0,/(1 ag). Suppose that we work The V-posteriors be expanded as
with Masked diffusion, hence
V (-))()
=-egoje(-lz], x²) + (1 -Ke) [a00o) + (1 a.)TT
(22)
Cat(;;z))
z + m,
a. ) + (a. - ax)xx
+ +(1 ne) [asx' + - (1-aa,)m) a..mm
Ki
Cat
z = m
Cz.
(23)
Cat(:;x')
2 + m,
(1)
(1 ag) ag)m + (as - -ar)x) )
+(1 ne) faxx + (1 a.)m
Net
Cat
:;
2 = m
aq
(24)
Cat(:; + (1 - ka) (1-1x)laax
+ (1 - a. m));
z + m
Cat Kig (1 (1=a,)m+(a, -a,x"
+ (1 - Kr) la,x' + (1 (1
z = m
(25)
l-a;
Cat(.; [Kg + - (1-k))auk kx)x + (1 - k1) a. m)),
z + m
(26)
Cat (:: Ne ar=as + (1 - (1-kt)ag
x" + Ki 1-a + (1 - k2 - a.) m
z m
where (1) holds since z # m implies that z = x', since in Masked diffusion, the latents z are either clean token or the masked token.
19

---

Published as a conference paper at ICLR 2026
To conclude, if we pick Ke I-a., where oe is the free parameter in the ReMDM sampler, then the equation reduces to the ReMDM posterior. Therefore, the V V-posteriors generalize ReMDM, which itself generalized the FB (Campbell et al., 2022) and DFM (Gat et al., 2024) posteriors. Additionally, the V p-posteriors are not limited to Masked diffusion, as showed in this work. In Table 11, we sample from the ReMDM--quivalen V-samplers, comparing the official ReMDM implementation and our version, and find obtain similar performance.
A.5 REMDM SAMPLING SCHEDULES
Wang et al. (2025) introduce three schedules for the remasking parameter or which controls how aggressively previously generated tokens are remasked. As shown in Suppl. A.4, the V-samplers recover ReMDM when K₃ 1 ot/(1 - aa)), and (o denotes the ReMDM noise injection schedule. Therefore, each ReMDM o; schedule has a direct V-samplers equivalent. Below, we present the three o, schedules studied in Wang al. (2025). The schedules must satisfy 0 I a, S of = min [1, (1 - ag)//a} to ensure that the reverse posterior remains a valid distribution, and are generally defined terms of of mma and a parameter 7) E [0, 1].
Cap Schedule With the "Cap" schedule, the remasking probability is capped at constant 17 [0, 1]. as long as it remains in the valid bounds:
o: min ( (7, ot maxx
(27)
Rescale Schedule With the "Rescale" schedule, or is obtained by scaling the upper bound of by a constant 1) E [0, 1]:
o₂ = 7 oi max
(28)
Loop Schedule Unlike the cap and rescale schedules, which only modulate or, the "Loop" schedule also changes the evolution of during sampling (see Fig. 4 for an illustration). It is controlled by the time boundaries fan > foll- The noise schedule a reparametrized to be piecewise linear. First, O₂ increases linearly from 0 to Ct... when > fon- Secondly, when t E [foft, fool. it is held constant to Ct... and finally increases from at.,. to 1 when < loft. When t & [loff, fon]. ReMDM samples from the MDM posterior (2). When / E [lorr, (on]. since on = Os, Olt,. the model samples from the ReMDM posterior (21) with a constant og = 17. In practice, fon is chosen so that ar,, is close to 1, where most tokens have already been decoded.
B FAST CURRICULUM
In this section, we expand on the implementation of the efficient curriculum. In Sec. B.4, we present the overall design, pseudocode, and the three main implementation challenges. Our approach relies on several mathematical results, which we present in order of dependency. We first recall inverse transform sampling (Sec. B.1.1), then derive the distributions of the largest (Sec. 1.1.2) and second largest (Sec. 1.3.3) uniform order statistics. These results enable generating the largest Gaussian random variables out of K without materializing the full vector (Sec. 1.1.... We also derive closed-form expression for the conditional mean of the exponential of a Gaussian random variable (Sec. B.2), used to estimate the softmax normalizer.
Furthermore, although the efficient curriculum could be implemented using the original definition of the Diffusion Transformation Operator T, we show that T admits convenient series expansion in Sec. B. This avoids the need to precompute 100k function values, and simplifies the implemen- tation. Finally, in Sec. B.3.2, show that T can be well approximated by degree-9 polynomial, which removes the need to store a large number of coefficients during training.
B. SAMPLING THE T VALUES IN w WITHOUT MATERIALIZING THE FULL VECTOR
Computing our curriculum step requires access to the & largest entries of the diffused weight vector w/ E IR but explicitly materializing w/ (and then running full top-k) is prohibitively expensive when K is large. In this subsection, we show how to obtain the top-k values and their associated indices while using only O(k) memory and without simulating all K random variables. The key
20

---

Published as a conference paper at ICLR 2026
idea is to decouple values from locations: first sample the top-k Gaussian order statistics directly via inverse transform sampling (Sec. B.1.1), leveraging closed-form expressions for uniform order statistics (Sec. B.1.2, Sec. B.1.3) and numerically stable log-space implementation (Algo. 1). We then assign these top-k to their corresponding indices (Sec. B.1.5, Algo. 2). This yields an efficient routine whose cost scales with l rather than K, enabling top-k truncation of wi without ever materializing it.
B.1.1 INVERSE TRANSFORM SAMPLING
The Inverse Transform Sampling method (Devroye, 1986) an algorithm for generating a continuous random variables X with a known Cumulative Distribution Function (CDF) Fx. Implementing Inverse Transform Sampling requires access to the inverse CDF Fx and source i.i.d uniform random variables. X Fx (U), where U U[ 1]. then X Fx. Indeed, for æ E R,
P(X - 2) P (0) S x) P(U & Fx(x)) Fx
(29)
since for a E [0, & a) = a. This shows that X has the correct distribution.
B.1.2 DISTRIBUTION OF THE LARGEST UNIFORM RANDOM VARIABLE OUT OF K
The distribution of the largest uniform random variable out of K admits simple closed-form expression:
Proposition B.1 (Distribution of the largest uniform random variable out of K). U U
... 2 ((K) denote order statistic over K i..d uniform random variables ((0, 8]) with Cumulative Density Function (CDF) Fu. Suppose that u E [0, 1], then Fu(u) = z Then, the CDF Foo and probability density function (PDF) feo of the largest random variable U!!) are as follows:
Fym (u) = FB (u) = u g-k
(30)
fu))(u)= = KFU (u) = Ku K-gpK
Proof.
(1) P(U = S u)viejK] P(U S u)* = r (u).
(31)
The PDF is obtained by differentiation:
Jua (u) = du (va = KFK (w)fo(),,
(32)
B.1.3 DISTRIBUTION OF THE LARGEST UNIFORM RANDOM VARIABLE OUT OF K
In this part, we show how to sample the k largest uniform random variables out of K. Importantly, we do not need draw all K values and sort them, which would be impractical for large K. Let (1))
(K) denote the order statistics of K i.i.d. (4 1] random variables. We argue that for 1 Kii < K, conditioned on (0)) (0) (6+1) is distributed the largest out of K uniform random variables on [0, f This enables iterative scheme to generate the i largest variables in decreasing order. The argument relies on two standard results: the conditional density formula (Prop. B.2) and the joint density of pair of order statistics (Prop. B.3). The conditional distribution of (++1) (+)) ( is given in Prop. B.4.
Proposition B.2 (Conditional Density (Berger & Casella, 2001)). Let X, Y be two random variables with joint density fx.y and marginals fx, fY. Then, the conditional density of X given = y is
fxlY=y(Ily) = fx.Y(2.)) (2.1)
(33)
fr(
21

---

Published as a conference paper at ICLR 2026
Proposition B.3 (Joint Density of Order Statistics (Berger & Casella (2001); proof in Border (2021)))
Let x() 2 .... 2 X(K) denote the order statistics of K random variables with CDF F and PDF
f. arranged in descending order. Then, the joint density of X and X(m), where < m (so that X( 2 x(m),, is given by
X(mm (im) (u, v)
K!
(n - 1))(m n 1)!(K - mm) (1 (1- F(u))""1((F ((t))))(()))--) ((o))-nn))(-------- F(o)K-m (())(())
(34)
Proposition B.4 (Conditional Distribution of ( given U Let U 2 ... 2 U(K) denote
the order statistics of K independent and uniformly distributed random variables on [0. 1], arranged in descending order. For any 1 < i <. K, conditioned on UG) = w(0), U(+1) is distributed as the largest of K iidd uniform random variables on [0, u..
Proof. From Prop. B.3 with & = i and = + 1, the joint density of (U U(+1)) for u10, 1] variables (where F(u) and f(u) is
Jua4 (u (n ())
(i 1)! (K 1 1)! (1- -1-1(()-111 (n(+t)))
K!
(35)
since (F(u F(ub6t1)) as ==
To apply Prop. B.2, we need the marginal density of (+()) The event {ul) ((() S (()) + du} requires that, among the K i.i.d. draws, exactly i 1 fall in (u(0) + du, 1]. exactly one falls in [u()), ((()) + du), and the remaining K i fall in [0, u0). The number of such assign- K! ments is the multinomial coefficient (K- and the probability of each assignment is
(1 (+) du) i-1 du (u(0) Kii Multiplying these terms gives
K!
P( < U0 S (0)) + du) (i 1) i)! (1 (1)) du) du ( (36)
K
By definition, Suce (n(() limdu-so as du -> O, we obtain
sul Since (1 - (+) - du) 11 - (1-
du
K!
Tocoa1 (1 - 1)! - i)! - (1- (x(0))
(37)
Applying Prop. B.2:
fiw (00 (u(+), n((+1))
fuo+oeroo(uu (1()))
L0 (u(()
[K-]] (1 (()))==1 (1(42))K
K!
(38)
[--i][k- -i] u()))- u(0))-(((0)))- (u(i))-
(K - (u(+4))kKi- Kii-1
(
since the factors K!/(i- - 1)! and (1 -
(1- -() cancel. This is precisely the density of the largest of
K-i 24(0, w() random variables.
1.1.4 GENERATING THE à LARGEST GAUSSIAN RANDOM VARIABLES OUT OF
We now show that is possible to generate the k largest Gaussian random variables out of K via inverse transform sampling (Sec. B.1. as follows.
Given a single uniform random variable U U{0, 1], one can obtain a standard Gaussian random variable W +b-(U), where t is the Gaussian CDF, via inverse transform sampling. Now
22

---

Published as a conference paper at ICLR 2026
assume have sorted list of K uniform random variables U1 2 U2 ... UK. Since O is monotonically increasing function, the largest uniform random variable, U1, is mapped to the largest Gaussian random variable, i.c. + (U,) is distributed as the largest Gaussian random variable out of K.
Sampling The Largest As shown in Prop. B.1, the CDF of the largest uniform random variable out of K has analytical solution. For E [0, 11, P(U1 S u) u hence it can be generated via inverse transform sampling.
Sampling The Second Largest Furthermore, the distribution of the second largest, conditioned on U1 U1, also admits closed-form solution (Sec. B.1.3): for u2 [0, u1], it is given by
P(U < u2 U1 = u1) = u2 1K-1 i.e. is distributed as the largest uniform variable out of K 1, supported on [0, u1
Sampling The largest More generally, the same argument shows that conditioned on U4 u, the random variable U1+1 is distributed as the largest uniform variable on [0, u,] out of K i This shows that variables. Finally, the largest U, can be transformed into the k largest standard Gaussians out of K ( (U.) In practice, naive implementation of inverse transform sampling is numerically unstable when K is large. For stability, operations should be implemented log-space. Algo. shows the pseudocode of the log-space implementation.
sample U..... Uk in decreasing order and without simulating all the K
Reverse Sampling from Order Statistics of Gaussian Random Variables. Here N corresponds to K-1 (the number of zero-mean entries) and a to / aZ.
Algorithm 1
Input Number of variables N, standard deviation a, number of top values k Sample U2 ~ (10,, 1), for N 2 2 N k+1 +1 Compute the random variables: R2 1t
Compute the cumulative sums: P: = Σm= Rm Let V2 (Pt) the ê-th sample from the (uniform) order statistic.
Apply inverse normal CDF: ( 4 (V-) N-++1 return TE=N
B. SAMPLING INTEGERS WITHOUT REPETITIONS AND WITHOUT SHUFFLING
Suppose that x denotes the one-hot vector of category By symmetry, after applying Gaussian diffusion to x, all entries (x,) j such that follow the exact same distribution. Therefore, they have the same probability of being one of the top & largest random variable.
To implement the curriculum, we must not only approximate the weights of the embedding combi- nation but also select which embeddings to include. As described in Sec. 4, we sample & random indices without repetition excluding the random variable at position o, corresponding to the clean token, belongs to the top-k, we replace one of the sampled indices with Otherwise, we use the k sampled indices directly.
A simple way to sample A random indices without repetition is to shuffle list of K integers and take the first k. However, this defeats the purpose of our efficient curriculum, as requires materializing large tensors. Instead, Floyd's d's algorithm (Bentley, 1999), given in Algo. 2, samples without repetition while avoiding shuffling. Although sequential with & iterations, it much faster than shuffling when k & K.
23

---

Published as a conference paper at ICLR 2026
Algorithm 2 Floyd's Algorithm for Sampling Without Repetition
Input Number of possible values N, number of samples kt
Initialize arry of size store samples fort t 0 to k do Sample j ~ (a
+)
if't > 0 and appears in S then
S(1] - N k + (Use largest remaining value)
else
S[d] -
end if end for return
B.2 APPROXIMATING THE WEIGHTED SUM OF THE EMBEDDINGS
After extracting the top-k values and indices (K, from w/. we approximate the softmax-weighted embedding by retaining only the & selected rows the embeddings E R (d denotes the embedding size):
softmax(w() embeddingg exp(K./T)
(39)
where embeddi [gs denotes the j-th row.
The normalizer Ž includes both sampled (top-k) and unsampled terms. To account for this contribu-
tion, let
E(exp(X/7)) X KRp]
(40)
denote the expectation that a a.v. X with pmf (O (?) is less than Ks.
We approximate Z via two cases. Let be the index corresponding the clean-token category in x and let v (ā o?).
Case 1. If o is not among the top-k indices (i.e., o & I, and consequently w & KC), then Ž includes: (i) the k terms in K. (ii) the explicit contribution from index o, and (iii) the remaining К k 1 unsampled terms, each approximated by Thus,
k
Kr
Ž
Σ
exp
+
exp
+ (K - - 1)p.
(41)
unsampled terms
terms
index
Case 2. If o is among the top-k indices (i.e., E I, and hence, wo K.), its contribution is already included the top-k sum, leaving K I unsampled terms. Hence,
Ž
exp
(K kK-
(42)
unsampled terns
top-k terms
Next, we derive a closed-form expression for μ in (40).
6?
Kg o,
+ log KCx - 0?/T
= log Elexp(X) | X < K2p] =
log
(43)
2r
o,
24

---

Published as a conference paper at ICLR 2026
Proof.
X
p= logE
exp
X Kx
Applying change of variables X X/T: get.
log Blexp(X) (X) x)| X < K
L exp(x)
Ka/t
fx(x))
log
dx
P(X < kk/T))
Substituting o ot/T, we get:
1
Ka/t
x2
log
exp(x)
exp
de
((Kx/or)
v 2
20
1
Rx/T
x²
log
(KKx/ō))
V2T0²
-00
exp
20
+
dr
log
((Ka/ō.)
1
V2T02
Rx/T
exp
20 (x²
-20020 x + 0² -09) dr
log
exp(o²/2)
((K)/o)) 2
1
Ke/T
exp
2a²
1
(rr 02)2
der
-00
exp(o2/22
12)
(KA/T-O"
log
+
(K()/o,)
0²
Kib llg $
(Ks/T-o* 02
log +f
Substituting back == =./T,, we get:
0?
IC ō
KCx -0?/T
log +
+ log +b
(44)
272
This concludes our proof.
Finally, substituting the value of y from (44) into (41) and (42), we obtain:
AC=
KR
Kxx - -0--
6//T
Ž
exp
exp
(K- 6) exp
2T
log
log
Ōe
top-k torms
clean token
unsampled zero-menn torms
(45)
B.3 EFFICIENT COMPUTATION OF T DURING TRAINING
The curriculum objective in (9) requires evaluating the diffusion transformation operator T(-). Directly computing T via (6) prohibitively expensive during training; Sahoo et al. (2025a) therefore precompute and cache many (a,, T(ā))) pairs, which is cumbersome. Instead, we compute T(.) on the fly using its Taylor expansion, detailed below. This derivation relies two propositions that justify swapping the order of (i) summation and integration (Prop. B.5) and (ii) differentiation and integration (Prop. B.6).
Proposition B.5 (First Corollary of the Dominated Convergence Theorem (Folland (1999), Theorem 2.25)). If the sum 3 (x) exists for all x and there exists an integrable function g(x) such that
Σ fa(
$ g(())
(46)
for all then
Σ / fa(x)da.
(n(2)d =
(47)
1=0
n=
25

---

Published as a conference paper at ICLR 2026
Proposition B.6 (Second Corollary of the Dominated Convergence Theorem (Folland (1999), The- orem 2.27)). Let f(x,t) t) be differentiable in t and suppose there exists a function g t) such that:
1.
0f(z.)) 8t
S g(a, to) for all a and t in some neighborhood It tol S d0
2.
[ g(r,t)dx t)dx < 20 for all
Then
dt / f(x, t)dt S 0f(x,t) t) ôt dx
(48)
B.3.1 SERIES REPRESENTATION OF T AND 0.T
Evaluating the series expansion the fly is faster than precomputing and caching T for many (or, (ā:)) pairs as done by Sahoo et al. (2025a). We can further speed the evaluation up by fitting low-degree polynomial the series, which we use in practice (see Suppl. B.3.2 for the approximation error analysis). Below, we derive the series expansion for T (Prop. B.7) and its time-derivative 0 T (Prop. B.8):
Proposition B.7 (Series Expansion of the Diffusion Transformation Operator). The diffusion trans- formation operator T can be expressed as:
T(ô)) = K-1
K
12
U M n! K
(49)
n=0
1/ and Mn = -_ z² (=))(2))000) (z))dz.
&1-6?
Proof. Recall that the standard Gaussian PDF given by
(xx)
2//2
(50)
2m
For notational convenience, let 14- =
&
We can rewrite ø(x v4) in terms of δ(z):
Φ(x v11) Using the definition of the infinite series e we can expand ex
(a 2
(x 2mxtt2)/2222222 e c 11//2
(51)
22T
2TT
-)= 6(x)e v2/2
(52)
n!
Substituting this into our original integral:
[ 0l= (z= v2) T 1 - = I ((2)e-
1//2
2 Φ T
(53)
nl.
Since Prop. B.5 is satisfied, as the is the Taylor series of the exponential function, can exchange the order of integration and summation. This leads to our final result:
[_ x v.) k 12)dd =2-22/2
8
(()))=
n!
==0
-x
(54)
-2/22
11
M
n!!
26

---

Published as a conference paper at ICLR 2026
Advantages At this point, one might ask what is gained by expressing T as series expansion. There are two key advantages. First, since 7 is intractable, Sahoo et al. (2024) resort to precomputing 100k evaluations, which can take up two hours with the GPT-2 tokenizer. Second, they approximate the time derivative using finite differences. Crucially, observe that M, and I in Prop. B.7 and B.8 the only intractable components of the series expansion, and they independent of the input āg. We find that the terms of the series decay to zero after roughly 150 terms (with slower decay as t + 1). Thus, instead of pre-computing 100k evaluations of T, it suffices to cache M,, and in for 150. In practice, this takes only few seconds and can be performed at the start of training.
Proposition B&8 (Time-Derivativy of the Diffusion Transformation Operator). The time-derivative of the diffusion transformation operator can be expressed as:
d dt. T(a)) K e- d
n! In v.MMn]
(55)
K-1 (1
where vi and M, are defined as in Prop. B.7. Finally, In 1 2 ((2) (z))z, and o denotes the time-derivative of the Gaussian noise schedule āt.
Proof. We want to compute
d
K d du, / 6lz--v/ - 12)
T(a.)
(56)
dv
K
To justify passing the derivative under the integral, we verify the conditions of Prop. B.6. Define
1(,,1)= =
2
V1
k (2) = = vz) (x)
(57)
which has time derivative
0f(z,t) Ot
(2 11)00/ v1)
x -2))
(58)
(1 ot) )3/2
We need to find a suitable dominating function g. Let > do > 0 and choose to 1-80 When
2
It dol S 6o. we have E 140 op, to + do). Since to 60 < l0 < 1 and to + do we are guaranteed that < This ensures that 24 is finite. Because å, [0, 1) when 1, there exists a constant C, such that
1-6 + 60 < 1,
2
C:=
[t- 55 (1 - (1-a7)37
max
2/33/2
(59)
For E and [t -t- tol S 6o, we can bound the absolute value of the time derivative of f as follows:
0f(2,t)
= 12 v/l
(1 āg)3/2 0(2 - 11e) g K (2)
Øt
S C| D1 (0(= -vi) ==(2,1) 1).
Finally, for all & 10;
f
00
2-- m/o(z v1)dz C Jf |zlo(z)dz
g(2, t)dz = t(==
1zlo(2)dz 20 ( 20(2)d2
112
2C
(60)
2mT
2C
2 112d2
2
2C
1 = VV
< 00,
22T
27

---

_e e Published as a conference paper at ICLR 2026
F009
FOOSS
59040
26
Figure 5: Polynomial approximation and approximation error, compared the series approximation, truncated at 150 terms. The degree-9 polynomial (left) achieves orders of magnitude lower error than the degree-5 polynomial (center) and sigmoid (right) approximations.
where we used the substitution u /222 in the integral S /2-= to obtain fo The conditions of Prop. B.6 are satisfied, so we can pass the derivative under the integral.
Applying the derivative under the integral sign and using the identity o(2 v.) )(2)c"1= 19/2. we have:
d du
6b= Uz) - ==)) f (ema--v 1//22
du
(61)
= (=))e=== ([========// uu ve)
(= 11)((z 1x))
Therefore:
K
00
((t)) K -
(z v1)Ø(z v1 ()))d=(())d
(62)
dv,
000
Now using the Taylor series of ø(2 vz), found earlier, and inverting the sum and integral as before, we find
d du
K
T(āi)
(= (1)d(2)e
-
K
K. e 1//2
00
n z1 x (z)dz 11 / 2"0(2)2 x (=)dd
K 1
K. -///2
[In Mr].
K-1
n!
n=0
(63)
where I, J 2 ((2))=(k (z)dz and Mn
(2)d2.
B.3.2 POLYNOMIAL APPROXIMATION OF T
Because the Diffusion Transformation Operator has sigmoid-like shape, we approximate it with S-shaped functions that require only handful of coefficients. This allows us to store fewer parameters during training, instead of the 100k values required by the original curriculum the 300 coefficients from the series approximation. Concretely, we test several functional forms with fewer than 10 parameters and fit them using non-linear least squares, via scipy. owt.miie............. curve_fit..
As shown in Fig. 5, approximations tend to be less accurate at the boundaries, when / R or / Rz 1. We find that the degree-9 polynomial works better than a sigmoid function of the form ab + c) + d, especially at the boundaries.
B.4 IMPLEMENTATION OF THE FAST CURRICULUM
In Sec. 4, we described our efficient curriculum. Here we provide the pseudocode (Algo. 3) and elaborate the three main implementation challenges:
First, we need to sample the largest zero-mean Gaussian random variables out of K, to emulate the Gaussian Diffusion over the one-hot data samples (Sec. 1.1.4).
28

---

Published as a conference paper at ICLR 2026
Algorithm 3 Scalable Top-k Curriculum Weights (Sec. 4).
Recall that w a,x' + O,E, (0, Ix), where x is the one-hot vector representation cate- gory Hence, (w.) N (ā, (G)))) and the remaining K-1 entries are ii.d. (O, of). The goal is to approximate softmax( w; /T) using only the largest entries (A < K).
Input: Clean token index E [K], vocabulary size K, top-k count k: temperature т, Gaussian schedule (ōt,
Output: Approximate softmax weights À E 1]* and corresponding token indices I E [K]*
Step I: Sample the k largest values of wf without materializing all K entries 11 2 ... 2 Kg + top-k order statistics (K-1) iid. N/O, o.) draws
D Algo.
á (@ o
Clean-token value: sample of (wl)o
b Step 2: Build the top-k set and approximate the softmax normalizer
if f > K₁ then
Case 2 clean token o belongs to the top-k
+ E/exppX(X (X/T) X < I1- X~ ~~ of)
b Mean contribution unsimulated entry
(Suppl. B.2)
r - {jj E [k] KG > m
b Rank of in KC b Insert ú. drop the smallest Ki Algo. 2
IC + (Ki.. w. Ky+1:--1)
Sample k-1 indices J uniformly w/o replacement from [K] {o}
I + (Jirr o, I+)
% x- E exp(Kx/T) + (K k)
K-k unsimulated entries, each contributing
else
Case clean token is not in the top-k 11 4- E(exp((//)) X KeJ. X X~ ~~100, of) Mean contribution unsimulated entry (Suppl. B.2) Sample h indices I uniformly wo replacement from [K] \ {o}
D Algo. 2 b i counted exactly, K-k-1 via
7. <- Ei exp(Ki/T) + -K---) 1-) H exp(ü/r)
end
Step 3: Normalized softmax weights over the ie selected entries
A; + exp(K,/T) 2, i= 1= return X, I
Secondly, must estimate the normalization constant of the softmax, without actually sampling the K random variables (Sec. B.2).
Third, we require an efficient method to sample distinet integers from K without replace- ment (Sec. B.1.5).
Algo. 3 shows the pseudocode of the complete algorithm.
C EXPERIMENTAL DETAILS
C.I V-SAMPLERS
C.1.1 OPENWEBTEXT
To evaluate the samplers, we use the pre-trained MDLM (Sahoo et al., 2024) and Duo (Sahoo et al., 2025a) checkpoints, well their distilled variants (using SDTT (Deschenaux & Gulcchre, 2025) and discrete consistency distillation, respectively. after 5 rounds of 10k steps). We re-state the training hyperparameters of both models in Suppl. C.2.1. For ReMDM, we both the official implementation of Wang et al. (2025) and our re-implementation, which matches the original results while supporting additional sampling schedules beyond the log-linear one. See Suppl. D.I for details on selecting
29

---

Published as a conference paper at ICLR 2026
C.1.2 CIFARIO (D3PM-LIKE ARCHITECTURE)
We train a U-Net backbone (Ronneberger et al., 2015) for 1.5M steps with batch size of 128, using class conditioning with caasou rate of 0.1 (as in Schiff et al. (2025)), and the default hyperparameters of Austin et al. (2023) (Table 3). For both MDLM and Duo, we exper- iment with time-conditional and unconditional variants, and train models using either cosine or log-lineat noise schedules. See Table for the ancestral ancestral-sampling evaluation of all variants after pre-training. See Suppl. D.1 for details on selecting Ke.
Table 3: Model architecture on CIFARIO
Component
Value
Vocab size Number of ResNet blocks per scale
256
Base channels
128
Channel multiplier per scale Attention resolutions
(1,2,2,2)
16
Conditional embedding dimension
128
Number of parameters
35.8M
C.2 IMPROVED CURRICULUM
C.2.1 LANGUAGE MODELING
We adopt the same setup as prior work on discrete diffusion (Lou et al., 2024; Sahoo et al., 2024; 2025a), and restate for completeness.
LMIB We detokenize the One Billion Words (Chelba et al., 2014) as in Lou et al. (2024); Sahoo al. (2024)', and tokenize it using the sert-base-uncasee tokenizer (Devlin et al., 2019), as He et al. (2022). We use context length of 128 and pad shorter documents.
Open Tex We tokenize Open WebText (Gokaslan & Cohen, 2019) with the GPT-2 tokenizer, concatenate sequences to length of 1024, and insert an ecs token between documents. Since the dataset lacks an official validation split, we reserve the last 100k documents for validation.
Backbone We parameterize all models using the modified diffusion transformer architecture of Peebles Xie (2023), following Lou al. (2024); Sahoo et al. (2024). Our models use 12 layers, hidden dimension of 768, 12 attention heads, and timestep embedding of size 128 for the uniform-state diffusion variants. Word embeddings are not tied between input and output.
Curriculum Lookup For the Duo baseline, we train models using the original code. To implement the efficient curriculum, we replace the full linear combination of embeddings by sparse lookup, implemented using torch. nn. funct ooall, embedd ng bag to avoid materializing interme- diate tensors. The curriculum phase lasts for the first 500k steps, after which we perform regular embedding table lookups, just like Sahoo al. (2025a).
Optimization We train models with the AdamW optimizer (Loshchilov Hutter, 2019) using a batch size of 512. The learning rate is linearly warmed from 0 to 3 x 10-4 over 2,500 steps, then kept constant for the remainder of training. We apply dropout rate of 0. throughout.
C.3 DOWNSTREAM EVALUATION PROTOCOL
We evaluate downstream performance using the Amm-vvll-harmn following the protocol of Deschenaux et al. (2025). We focus on multiple choice tasks, where the log-likel likeelioooo of each candidate answer, given a prompt, is computed and the answer with the highest score is selected. For diffusion language models, which optimize a variational bound on the log-likelihood of the full sequence, adapt the evaluation by using Bayes' rule:
library (Gao et al., 2024),
logp(y; [x) = p(x, y:) - log p(x) (x) x log logp(x,yX) yi),
(64)
Since log p()) does depend on the candidate y,, we simply select the answer that maximizes logp(x, Y:). In practice, we use the log-likelihood ELBO (4), estimated via Monte Carlo with 1024 samples, and choose the continuation y, with the highest estimated likelihood.
Phttps:/// //gi thub. com main/data.pp
Maatecore-Kntropeee Ditrooy-- screteereeeeeeee wfu.fff/ullla
30

---

Published as a conference paper at ICLR 2026
C.4 ZERO-SHOT LIKELIHOOD
Our setting is the same as used by Sahoo et al. (2025a). Specifically, we measure the likelihood of the models trained on OpenWebText using the validation splits of seven diverse datasets: Penn Tree Bank (PTB; Marcus et al. (1993)), Wikitext (Merity et al., 2016), One Billion Words (LMIB; Chelba al. (2014)). Lambada (Paperno et al., 2016), AG News (Zhang et al., 2016), and Scientific Papers (Pubmed and Arxiv subsets; Cohan et al. (2018)), The datasets are detokenized following the protocol of Lou al. (2024); Sahoo al. (2025a). We wrap all sequences to a maximum length of 1024 tokens and do not insert ecs tokens between them. Table 5 shows that we reach similar performance as Duo.
D ADDITIONAL EXPERIMENTAL RESULTS
In Suppl. D.I, we elaborate the impact of Ką the performance of the V-samplers. In Suppl. D.2, we show that our efficient curriculum produces weights with the same marginal distributions as Sahoo al. (2025a).
D.1 TUNING KG FOR THE V-SAMPLERS
As discussed in Sec. 5.1, the choice of K₃ is critical for strong performance. With poor choice of Kr, V-samplers can underperform ancestral sampling. Below, we report all of our hyperparameter sweeps across datasets.
Inception Score (1)
We perform image modeling CIFAR-10 using the U-Net architecture Austin et al. (2023);
MOLM ReMDM
Schiff et al. (2025), and use horizontal flipping as the sole data augmentation.
Duo 4l--mmppler
We evaluate V-samplers (Gokaslan & Cohen, 2019) using the original checkpoint of MDLM (Sahoo al., 2024) and Duo (Sahoo et al., 2025a).
OpenWebText
32
128
256
1924
2048
4096
Figure 6: V-samplers, which generalize ReMDM, significantly improve the In- ception Score on CIFAR-IO, compared to ancestral sampling.
D.1.1 CIFAR-10
We report FID (Heusel et al., 2018), computed between
50k generated samples and the training Before evaluating V --samplers, ablate the training hyperparameters. Specifically, we train models with cosine and log-linear noise schedule, optionally with time-conditie condiiondiiinnigg We sample with both cosine and log-linear schedules. Finally, check whether nucleus sampling (Holtzman et al., 2020) and greedy decoding on the final step can help, compared to vanilly ancestral sampling. Since nucleus sampling helps Duo but MDLM,, compare the two models without nucleus sampling. Table 6 shows the validation perplexity and FID for few number of sampling steps. Table 7 reports FID for ancestral sampling using step counts that are powers of two, from 32 up to 4096. Table shows the results with ReMDM. Table 9 reports FID scores for sa-ampeers using stepwise-constan V V-samplers using the к schedule equivalent to ReMDM. We obtain similar results, which supports our theoretical claims.
schedule. Table 11 shows the performance of
MDLM (Ancestral). Training with cosine noise schedule and time conditioning yields the best validation perplexity and FID.
MDLM (ReMDM). We find that ReMDM improves the best FID over ancestral sampling, from 24.73 to 23.71 using 4096 sampling steps. Nucleus sampling can help at very low step counts, but the best performance is obtained with ancestral sampling. As the number of steps increases, nucleus sampling worsens the FID.
Duo (Ancestral). Cosine training without time conditioning yields the lowest perplexity, while log- linear training without time conditioning gives the best FID. We use the latter in downstream experiments. Nucleus sampling improves FID, and greedy decoding slightly worsens it.
31

---

Published as a conference paper at ICLR 2026
Table Training efficiency comparison between Duo and Duo++ on 138M parameter models. All measurements conducted training job on NVIDIA GH200-120GB GPU with batch size 32. We report the average throughput in sequence per second. The row "Duo (after CL)" denotes the resources consumption of Duo after the Curriculum phase. The impact of is minimal when uses similar resources.
E {2, 3, 5), and Duo
Method
Throughput (samples/s) f
Peak Memory (GiB)
Duo
81.8
94.3
Duo (after the CL)
122.4
63.3
Duo (k (2,
121.9
63.4
Duo (V- (V-samplers) V-samplers further improve performance beyond ReMDM. With the log-lineat sampling schedule (as used by ReMDM), V-samplers reduce the FID from 23.71 to 20.71. Using a cosine sampling schedule further improves the FID. Overall, Duo improves from an FID of 25.63 (ancestral) to 15.05 with V-samplers, and MDLM improves from 24.73 (ancestral) to 17.86 with V-samplers.
D.1.2 OPENWEBTEXT
We report the generative perplexity using GPT-2 Large, following standard practice (Sahoo et al., 2024; 2025a). Because language models can artificially lower the generative perplexity by producing repetitive text, we also report unigram entropy (Dieleman et al., 2022), proxy.
Some V V-samplers schedules reduce the unigram entropy more than others. Therefore, for figures, we select the x schedule whose unigram entropy matches (or is closest to) the entropy of samples generated with ancestral sampling. multiple schedules achieve the same entropy, we choose the one with the lowest generative perplexity. We indicate which schedule is used for plots by highlighting the corresponding row in blue in the tables. Overall, the V V-samplers can reduce the Gen. PPL of all models while retaining the unigram entropy. Best results achieved using the rescale schedule with E (0.01, 0.02), for both MDLM and Duo.
Table 12 shows the generative perplexity of MDLM and Duo after pre-training and after distillation with SDTT (Deschenaux Gulcehre, 2025) or DCD (Sahoo et al., 2025a) respectively, with and without nucleus sampling, using ancestral sampling. Table 13 shows the results when sampling with V-samplers that are equivalent to ReMDM (Wang et al., 2025), with the non-distileed models, while Table 14 shows the result for the distilled models.
D.2 DISTRIBUTION OF THE TOP À ENTRIES OF THE SOFTMAX
To verify that our sparse implementation accurately approximates the curriculum weights of Sahoo et al. (2025a), we compare the empirical distributions of the top-k largest entries between the original and our efficient implementation. While matching marginal distributions does not guarantee matching joint distributions, matching marginals are necessary for matching joints, and are casier to visualize. Recall that experimentally, our efficient implementation is sufficient achieve strong performance (Sec. 5.2). Specifically, we show histograms using a tokenizer with 100k tokens Figures 7, 8, 10, and with the GPT-2 tokenizer in Figures 11, 12, 13, 14, with varying temperature and log signal-to-noise ratios. In all cases, the top & variables have matching distributions.
D.3 TRAINING EFFICIENCY OF OUR FAST CURRICULUM
As shown in Table 4, our sparse curriculum achieves a 33% reduction in peak memory usage and reaches average throughput 25% higher than Duo, at context length of 1024.
32

---

Published as a conference paper at ICLR 2026
Table 5: Zero-shot perplexity (PPL) on seven datasets. Lower better. 'Results taken from Sahoo al. (2025a). Duo++ (k 2) achieves slightly lower zero-shot perplexity than Duo on of datasets.
PTB
Wiki
LMIB
LBD
AG News
PubMed
ArXiv
Autoregressive
Transformer!
82.05
25.75
51.25
51.28
52.09
49.01
41.73
Diffussion (138M)
SEDD Uniform
105.51
41.10
82.62
57.29
82.64
53.89
50.86
UDLM
112.82
39.42
77.59
53.57
80.96
50.98
44.08
Duot
89.35
33.57
73.86
49.78
67.81
44.48
40.39
Duo (k: 2)
94.96
34.05
73.80
48.67
67.14
43.98
38.93
Duo ++ (k = 3)
91.94
34.65
74.16
49.89
66.89
44.87
40,42
Duo++ (k 5)
94.46
34.52
74.91
50.93
68.72
46.79
41.04
33

---

Published as a conference paper at ICLR 2026
300
13
12
30
10
04
24
14
6.0
03
93
038
0.5
K35
Figure 7: Marginal distributions of the top-5 entries using tokenizer with 100k tokens, inverse temperature 100, and log signal-to-noise ratio -2. The histograms of the efficient and naive imple- mentation match closely.
Mave
top-2
1op-2 195-3
to2-4
tpp-5
12"
10
13-4
24
2A
04
Figure 8: Marginal distributions of the top-5 entries using tokenizer with 100k tokens, inverse temperature 1000, and log signal-to-noise ratio -I The histograms of the efficient and naive implementation match closely.
3."
11-0
0.00 A022 C254
0150
3.05
Figure 9: Marginal distributions of the top-5 entries using tokenizer with 100k tokens, inverse temperature 1000, and log signal-to-noise ratio 2. The histograms of the efficient and naive implementation match closely.
Mave
Tepo
10p-2
1ea-44
Trp-5
23r
0.8
5.0
04
M
090K 9005 800o 9015 3100
6.00
0.00
9.04
05
44
15
Figure 10: Marginal distributions of the top-5 entries using a tokenizer with 100k tokens, inverse temperature 1000, and log signal-to-noise ratio -4. The histograms of the efficient and naive implementation match closely.
ERicart
39
17.
334
10
30+4
10
10
Figure 11: Marginal distributions of the top-5 entries using the GPT-2 tokenizer, inverse temperature 100, and log signal-to-noise ratio -2. The histograms of the efficient and naive implementation match closely.
34

---

Published as a conference paper at ICLR 2026
10
377
37
30-4
Jpm
11
96
10
000
03
0.2
0.00
0.96
632
8.34
Figure 12: Marginal distributions of the top-5 entries using the GPT-2 tokenizer, inverse temperature 1000, and log signal-to-noise ratio -1. The histograms of the efficient and naive implementation match closely.
30"
52
50m
0.50
0..00
0091
0.82
000r
Figure 13: Marginal distributions of the top-5 entries using the GPT-2 tokenizer, inverse temperature 1000, and log signal-to-noise ratio -2. The histograms the efficient and naive implementation match closely.
377
521
300
30"
Figure 14: Marginal distributions of the top-5 entries using the GPT-2 tokenizer, inverse temperature 1000, and log signal-to-noise ratio -4. The histograms the efficient and naive implementation match closely.
35

---

Published as a conference paper at ICLR 2026
Table 6: FID on CIFAR-IO with ancestral sampling. We train and sample with the log-linean and cosine scheduler. MDLM performs best with imneconniii while Duo does not. We sample with discrete classifier-free guidance (Schiff al., 2025) with strength 1, and greedy predictions on the last step.
Scheduler
Time
PPL
FID J. (Cosine)
FID J. (Log-linear)
64
256
1024
2048
64
256
1024
2048
MDLM
Cosine
8.86
42.60
27.71
24.90
24.56
107.62
40.81
27.65
25.73
Cosine
/
8.72
41.89
27.03
24.67
24.24
114.56
40.60
27.08
25,50
-ineain
8.76
43.95
29.01
26.11
25.67
111.77
42.15
28.85
26.89
Log-linear
8.75
49.36
32.10
28.76
28.21
122.70
41.79
27.89
26.02
MDLM (mucleus p=0.9)
Cosine
8.72
34.81
44.04
47.84
48.37
41.73
33.33
43.12
45.98
MDLM (no greedy)
Cosine
8.72
42.14
27.19
24.47
24.46
114.55
40.92
27.13
25.60
Duo
Cosine
10.27
32.37
27.28
26.38
26.02
33.93
27.93
26.51
26.03
Cosine
10.32
33.74
27.98
26.81
26.96
36.23
28.77
27.08
26.79
Log-linear
10.49
31.78
27.03
26.00
25.75
33.44
27.46
26.08
25.87
Log-linear
10.45
34.05
27.74
26.58
26.37
36.46
28.49
26.60
26.22
Duo (nucleus p=0.9. 0.
Log-lineas
10.49
23.13
22.21
22.58
22.49
24.24
22.41
22.35
22.54
Duo (no greedy)
Log- oonlar
10.49
33.03
27.43
26.16
25.96
34.81
27.76
26.30
26.06
Table 7: FID on CIFAR-I0 with ancestral sampling and finer grid. We pick the variant with the best FID from Table 6.
FID
Algo
Train
Sample
P
1024
Duo
log-lin log-lin
log-lin log-lin
1.0
42, 11.
33.44
29. 18
27.16
26.
26. 08
25.87
25.79
28.53
24.24
22.41
Duo
log-lin
cos
39.65
31.78
28. 28.55
27. 03
26.03
25.89
25.75
25.63
Dan
log-lin
25.96
23.13
22.68
22.21
22.26
22.58
22.49
22.49
log.
114.56
MDLM
log-lin
0.9
84.8/
41.73
31.28
33.33
38.49
43.12
45.98
55.37
MDLM MDLM
73.82
41.89
36.
27.03
25.63
24.67
24.24
23.93
COS
COS
0.9
58.31
34.81
37.91
41.04
45.32
47.84
48.37
49.23
36

---

Published as a conference paper at ICLR 2026
Table 8: FID on CIFAR-I0 with ReMDM (best checkpoints, shown in Table 7). We sample with/without nucleus sampling, and with the 3 schedules of Wang et al. (2025) (cap. loop, rescale). For the loop schedule, we use fnn 0.55, foll 0.05, Coea 0.9, following ReMDM. Sampling experiments are executed in the original codebase of Wang et al. (2025).
Number of steps
64
128
256
512
1024
2048
4096
ReMDM cap (p=1.0)
" 0.005
215.67
116.24
63.37
40.82
31.40
27.28
24.97
24.78
" 0.010
218.41
118.25
64.50
41.77
32.40
28.68
27.91
33.68
7 0.020
224.20
122.61
66.95
44.54
36.26
35.39
46.01
92.48
77 0.050
242.25
143.21
84.41
64.10
73.89
132.13
210.60
203.14
ReMDM loop (p=1.0)
" =
307.56
234.55
138.56
80.50
55.86
47.05
45.44
7 0.02
307.81
237.28
142.21
83.68
59.96
53.88
60,50
87.54
7 0.04
308.24
242.70
152.28
94.63
76.93
88.53
135.05
196.58
77 0.06
308.88
248.76
165.79
114.92
113.26
157.92
223.70
237.16
ReMDM rescale (p=1.0)
1 0.01
216.92
116.73
63.56
40.65
30,86
26.03
23.77
23.71
7 =
221.21
119.79
65.08
42.02
32.29
28.11
28.66
39.39
7 0.04
229.72
127.94
70.89
46.98
38.74
41.23
67.44
130.05
7 0.05
234.35
133.08
75.02
50.92
45.01
57.03
107.13
164.44
ReMDM cap (p=0.9)
7 0.005
88.08
40.02
27.31
29.43
36.50
45.10
57.08
73.40
7) 0.010
87.68
39.55
27.35
31.24
41.22
54.55
71.65
93.06
7 0.020
85.95
38.46
27.80
35.01
50.50
69.60
91.49
118.87
7 0.050
81.91
35.56
29.39
46.90
70.24
95.24
125.60
163.32
ReMDM loop (p=0.9)
1 0.01
209.24
100.01
47.27
29.44
27.55
30.50
34.21
37.56
7 0.02
208.36
99.29
47.12
29.38
27.74
31.17
35.42
39.52
7 0.04
206.51
98.18
46.87
29.28
28.09
32.12
37.19
42.45
7 = 0.06
204.83
97.24
46.72
29.19
28.30
32.77
38.47
ReMDM rescale (p=0.9)
" = 0.01
87.31
39.51
27.25
30.74
40.22
53.30
70.24
91.79
7 = 0.02
85.94
38.45
27.45
34.13
49.00
67.89
90.61
118.10
7 0.04
83.47
36.44
28.29
41.76
63.40
87.03
115.60
153.03
7 0.05
82.26
35.69
28.99
44.69
68.80
94.07
125.42
165.62
ReMDM
Best 1.0)
215.67
116.24
63.37
40.65
30.86
26.03
23.77
23.71
Best (p 0.9)
81.91
81.91
27.25
29.19
27.55
30,50
34.21
37.56
MDLM
Ancestral (p=1.0) Ancestral (p=0.9)
212.95
114.56
62.86
40.60
31.05
27.08
25.50
24.73
84.85
41.73
31.28
33.33
38.49
43.12
45.98
55.37
37

---

Published as a conference paper at ICLR 2026
Table 9: FID on CIFAR-10 with V-samplers, where V-samplers are activated for steps with e [for, fual, when Ky is kept constant (according to the column, 1 otherwise). We use the same checkpoints as in Table 7. Using a cosine sampling schedule and light noise injection (n close to 1) generally perform best. The CIFAR-10 curves in Fig. I show the best FID per number of steps.
Algo
Train
Sample
Eus
1,dE
128
256
512
1024
2048
4096
Dao
0.02
log-lin
cns
0.2
0.15
40.64
33.06
30.36
29.85
31.31
34.36
39.06
38.38
Duo
0.02
log-lin
cos
0.5
0.45
41.81
33.67
29.50
26.55
24.83
25.12
31.63
51.83
Duo
0.02
log-lin
0.8
0.7
43.99
37.41
33.68
38.88
46.76
59.68
75.46
91.73
0.5
log-lin lin
0.2
28.
27.18
27.29
Dao
0.5
lin
0.6
0.4
39.54
29.40
23.46
20.77
23.72
38.42
72.97
105.75
0.5
log-lin
cos
0.9
0.65
43.00
34.68
31.85
34.73
45.68
64.97
88.07
107.36
Duo
0.95
log-lin
cos
0.5
0.1
39.30
30.58
26.15
23.16
20.93
18. 18
16.
15.05
0.95
log-lin
cos
0.6
39.19
16.70
16.30
Dao
0.95
log lin
0.9
0.3
39.04
29.88
24.72
20.90
19.20
21.09
30.00
51.43
0.95
log-lin
cns
0.9
39.21
30.29
25.26
21.57
19.92
21.50
30.03
50.88
Dao
0.98
log-lin log-lin
cos
1.0
0.05
39.31
30.97
26.39
23.13
20.56
18. 80
19.
25.83
1.0
39.31
Dao
0.99
log-lin
cos
1.0
0.05
39.34
31.56
27.46
24.73
22.35
20.07
18. 50
19.39
Duo
0.99
log-lia
1.0
0.1
39.35
31.57
27.46
24.73
22.37
20.09
18.51
19.41
log-lin lin
log-lin
0.2
0.15
33.71
27.95
29.35
Dao
0.02
log-lin
log-lin lin
0.5
0.45
43.86
36.29
33.35
33.24
34.74
36.97
36.77
37.30
Dao
0.02
log-lin
log-lin
0.8
0.7
43.95
33.75
28.32
27.78
37.12
69.66
113.05 3.00
132.88
Duo
0.5
log-lin
log-lin
0.2
0.1
12. 10:10
33. 100
29.19
27.14
26.22
25.52
25. 10
24.71
0.5
log-lin
log-lin
0.6
333
22.
Duo
0.5
log lin
log lin
0.9
0.65
42.87
31.04
26.37
31.86
61.36
121.66
155.77
151.48
Dao
0.95
Ing-lin
log-lin
0.5
0.1
41.74
32.97
28.57
26.05
24.62
23.13
21.81
10.16
Duo
0.95
log-lin
log-lin
0.6
0.1
11.46
32, 32./7
27.74
21.97
22.91
20.83
18. 87
16. 16.82
0.95
log-lin
log-lin
0.9
17.97
35.
Dao
0.95
log lin
log lin
0.9
0.4
41.18
30.58
24.71
20.59
18.08
18.02
22.07
35.44
Dan
0.98
Ing-lin lin
log-lin in
1.0
0.05
41.80
31.96
26.83
23.17
20.10
18.12
18.38
22.89
Duo
0.98
log-lin
log-lin
1.0
0.1
41.81
31.98
26.85
23.17
20.12
18.15
18.40
22.94
Duo
0.99
log-lia
log-lin
1.0
0.05
11.99
32.63
27.74
21.67
22.13
19.72
17.93
18.25
Dao
0.99
log
log. lin
1.0
0.1
41.99
32.63
27.75
24.67
22.13
19.75
17.95
18.28
MDLM
0.02
cos
cos
0.2
0.15
75.63
19.18
45.02
5-1.67
83.47
181.18
280. -2
297 .52
MDLM
0.02
008
0.5
0.45
117.57
111.75
200.49
283.55
310.51
314.98
313.93
MDLM
0.02
0.8
0.7
172.21
197.61
232.36
262.87
269.22
267.86
261.57
259.84 888
MDLM
0.5
0.2
0.1
73.13
46.10
38.47
39.71
48.49
75.27
173.09
206.36
MDLM
0.5
008
005
0.6
0.4
131.11
114
144.25
217.74
268.00 03
27/1.83 83
270.53
256 .03
MDLM
C08
0.9
131.04
147.67
177.75
198.33
201.97
193.77
184.76
MDLM
0.95
CON
COs
0.5
0.1
73.03
44.15
33.68
30.50
29.93
31.50
35.72
51.53
MDL M
0.95
0.6
74.57
45.00
34.07
30.32
29.16
31.03
37.46
64.74
MDLM
0.95
608
005
0.9
0.3
79.25
47.02
33.97
27.84
24.24
23. 43
26.
42.58
MDLM
0.9
78.18
MDLM
0.98
CON
Cos
1.0
0.05
74.05
43.85
32.32
26.69
23.22
20.81
19.41
20.20
MDLM
0.98
1.0
0.1
74.05
43.85
32.31
26.65
23.17
20.76
19.26
19.98
MDLM
0.99
008
cos
1.0
0.05
72.39
42.87
31.79
26.65
23.72
21.07
19.24
17. 77.94
MDLM
0.99
1.0
0.1
72.38
42,87
31.78
26.64
23.69
21.01
19.19
17.86
MDLM
log-lin
0.2
0.15
217.56
118.
68,
51.76
171.72
MDLM
0.02
log lin
0.5
0.45
247.31
157.61
124.97
162.92
256.01
298.74
305.05
310.28
MDLM
0.02
006
log-lin
0.8
0.7
298.96
294.71
298.95
312.49
317.03
312.60
308.49
302.37
MDLM
0.5
CON
log lin
0.2
0.1
216.00
116.99
65.73
45.72
41.32
45.95
68.60
152.77
MDLM
0.5
log-lin
0.6
171.73
212.68
273.
281.96
272.
.26
MDLM
0.5
log lin
0.9
0.65
296.08
268. .98
265.73
278.38
281.68
275.20
265.49
247.22
MDLM
0.95
log-lin
0.5
0.1
216.90
117. 1.7.05
64.76
43.50
36.06
31.84
37.
44.92
MDLM
0.95
00N
log lin
0.6
0.1
218.58
118.21
65.33
44.32
37.14
36.09
39.42
55.34
MDLM
0.95
log-lin
0.9
225.19
124.03
67.82
44.06
MDLM
0.95
008
log lin
0.9
0.4
223.84
123.04
67.19
43.29
33.85
$2.00
37. 37.23
63.89
MDLM
0.98
log-lin
1.0
0.05
218.15
118.08
63.97
10.97
30.67
25.69
23.61
25.40
MDLM
0.98
008
lin
1.0
0.1
218.14
118.09
63.96
40.96
30.65
25.64
23.57
25.29
MDLM
0.99
log lin
1.0
0.05
215.41
116.02
63.30
40.42
30.43
25.37
22.45
20.77
MDLM
0.99
log-lin
1.0
215.40
116.03
63.27
40.41
30.43
25.35
22.42
20.71
38

---

Published as a conference paper at ICLR 2026
Table 10: Inception Score on CIFAR-10 with V-samplers, where V-samplers are activated for steps with E [fom, fun), when K2 is kept constant (according to the и column, otherwise). We use the same checkpoints as in Table The CIFAR-I 10 curves in Fig. show the best Inception Score per number of steps.
Inception Score
Algo
Train
Sample
Eus
1,dE
128
256
512
1024
2048
4096
Dao
0.02
log-lin
0.2
0.15
7.02
7.25
7.35
7.48
7.52
7.47
7.38
7.63
0.02
log-lin
cos
0.5
0.45
7.09
7.44
7.64
8.04
8.32
8.57
7.94
Duo
0.02
log-lin
0.8
0.7
6.81
6.99
7.00
6.91
6.61
6.16
5.67
5.19
0.5
log-lin lin
0.2
7:21
7.28
7.39
7.45
7.48
7.56
Dao
0.5
lin
0.6
0.4
7.31
7.73
8,14
8,51
8.46
7.91
6.40
5.39
0.5
log-lin
cos
0.9
6.87
7.10
7.22
7.11
6.72
5.97
5.23
4.67
Duo
0.95
log-lin
cos
0.5
0.1
6.98
7.26
7.45
7.53
7.67
7.89
8.06
8.29
0.95
log-lin
7.70
Dao
0.95
log lin
0.9
0.3
7.08
7.37
7.54
7.84
8.01
8.07
7.72
6.84
log-lin
7.04
7.31
7.50
7.78
7.92
7.78
Dao
0.98
log-lin log-lin
cos
1.0
0.05
7.00
7.25
7.40
7.55
7.73
7.97
8.10
7.91
1.0
7.25
7.40
7.55
7.74
0.99
log-lin
cos
1.0
0.05
6.98
7:22
7.37
7.45
7.58
7:77
7.96
0.08
Duo
0.99
log-lia
1.0
0.1
6.98
7.22
7.37
7.46
7.58
7.77
7.96
8.10
log-lin lin
log-lin
0.2
0.15
7.22
7.36
Dao
0.02
log-lin
log-lin
0.5
0.45
6.95
7.28
7.45
7.64
7.67
7.70
8.06
8.68
0.02
log-lin
log-lin
0.8
7.00
7.54
8.02
8.18
7.89
6.46
5.03
4.55
Duo
0.5
log-lin
log-lin log-lin log lin
0.2
0.1
6.81
7.04
7.20
7.26
7.29
7.36
7.17
7.50
log-lin
7.73
7.93
8.20
Dao
0.5
log lin
0.9
0.65
7.05
7.61
7.97
7.74
6.45
4.46
3.77
4.07
Dao
0.95
log-lin
log-lin
0.5
6.80
7.10
7.25
7.31
7.35
7.43
7.55
7.63
Duo
0.95
log-lin
log-lin
0.6
0.1
6.85
7.12
7.28
7.40
7.46
7.66
7.81
7.97
0.95
log-lin
log-lin
0.9
7.27
Dao
0.95
log lin
log lin
0.9
0.4
6.89
7.25
7.58
7.80
8.05
8.25
8.26
7.69
Dan
0.98
Ing-lin
log-lin log-lin
1.0
0.05
6.85
7.19
7.36
7.49
7.72
7.96
8.05
8.03
Duo
0.98
log-lin
1.0
0.1
0.85
7.20
7.38
7.49
7.72
7.96
8.04
8.02
Duo
0.99
log-lia
log-lin log- lin
1.0
0.05
6.81
7.13
7.32
7.45
7.61
7.71
7.98
8.12
Dao
0.99
log
1.0
6.81
7.14
7.32
7.45
7.62
7.70
7.99
8.13
MDLM
0.02
cos
cos
0.2
0.15
5.56
6.61
6.90
6.75
5.52
2.68
1.57
1.56
MDLM
4.22
4.36
2.44
1.41
1.45
1.56
MDLM
0.02
0.8
0.7
3.12
2.82
2.41
2.03
1.96
1.97
2.02
2.09
MDLM
0.5
0.2
5.63
6.63
7.00
7.09
6.90
5.68
2.78
1.73
MDLM
0.5
008
005
0.6
0.4
3.83
4.32
3.55
2.35
1.85
2.14
2.51
2.99
MDLM
3.47
3.15
3.37
MDLM
0.95
CON
COS
0.5
0.1
5.66
6.68
7.13
7.29
7.44
7.41
7.44
6.77
MD M
0.95
0.6
5.59
6.70
7.21
7.41
7.52
7.57
7.45
6.33
MDLM
0.95
008
005
0.9
0.3
5.43
6.68
7.25
7.63
7,90
8.15
8.18
7.58
0.9
6.66
7.25
7.64
8.14
8.30
MDLM
0.98
0ON
Cos
1.0
0.05
5.57
6.71
7.22
7.45
7.71
7.93
8.14
8.30
MDLM
0.98
1.0
5.57
6.72
7.22
7.46
7.72
7.93
8.15
8.31
MDLM
0.99
008
cos
1.0
5.00
7.18
7.39
7.53
7.81
7.97
8.12
MDLM
0.99
1.0
0.1
5.60
6.73
7.19
7.39
7.53
7.81
7.97
8.14
MDLM
log-lin
0.2
0.15
4.59
5.86
6.46
6.45
5.59
1.67
MDLM
0.02
log lin
0.5
0.45
2.21
3.56
4.08
3.06
1.78
1.43
1.38
1.36
MDLM
log-lin
0.8
1.65
1.63
1.55
MDLM
0.5
CON
log lin
0.2
0.1
2.66
4.60
5.91
6.58
6.81
6.76
5.77
3.15
log-lin
0.6
2.78
2.27
1.58
MDLM
0.5
log lin
0.9
0.65
1.69
1.91
1.90
1.79
1.91
2.35
2.87
3.29
MDLM
0.95
log-lin
0.5
0.1
2.65
4.60
5.95
6.61
6.93
7.03
7.07
6.78
MDLM
0.95
00N
log lin
0.6
2.62
4.55
5.94
6.66
6.98
7.16
7.10
6.47
MDLM
log-lin
0.9
1.45
5.93
6.84
7.35
7.61
5.83
MDLM
0.95
log lin
0.9
0.4
2.54
4.46
5.94
6.85
7.40
7.69
7.59
6.60
MDLM
0.98
log-lin
1.0
0.05
2.62
4.56
6.04
6.85
7.31
7.66
7.79
8.01
MDLM
0.98
008
log -in
1.0
2.62
4.56
6.03
6.85
7.32
7.67
7.81
8.03
MDLM
0.99
log lin
1.0
0.05
2.68
4.64
6.02
6.84
7.21
7.47
7.70
7.93
0.99
log-lin
2.68
4.64
6.02
6.84
7.21
7.47
7.70
7.93
39

---

Published as a conference paper at ICLR 2026
Table 11: FID on CIFAR-10 using y-samplers whose N2 schedulers are equivalent to ReMDM. We use nucleus sampling, no temperature scaling, and cfg = 1. As expected, with the log-linear scheduler, we reach a similar FID as when using the ReMDM codebase (Table 8). However, note that by using log-linear scheduler, using constant KG 0.99, we reach a better FID than with the original ReMDM scheduler.
FID
Algo
Train
Sample
128
256
512
1024
2048
4096
with vesuale schedule
Duo
log-lin
39.64
32,03
28.49
26.95
26,16
25.71
25.
25.02
Duo
log. lin
log. lin
42.27
33.58
29.49
27.36
26.33
25.86
25.07
25.21
RcMDM Reseale (y 0.01)
DoS
MDLM
COS
log-lin
213.22
114.24
62.51
40.51
30.28
26.21
23.61
23.40
ReMDM Cap 0.005)
MDLM
Cos
log-lin
215.75
115.77
63.20
41.25
31.60
27.30
25.16
24.79
ReMDM Loup
0.55, far
0.05,
0.9. y
0.01)
MDLM
cos
log-tin
305.30
224.84
120.58
66.30
45.70
30.06
41.44
52.71
Table 12: Generative Perplexity (Gen. PPL) and Unigram Entropy on OpenWeeText (Gokaslan
Cohen, 2019) with ancestral sampling (no nucleus, no temperature scaling). We train using the log-linear noise scheduler, and sampling with the cosine scheduler is slightly better. We stick to the log- log-linear schedule for sampling in further experiments, follow prior work, and since the cosine schedule only marginally reduces the Gen. PPL.
Gen.
Algo
Dist.
P
32
64
128
256
512
1024
2018
4096
Duo
X
1.0
87.23 (5.5
(5.35)
(5.53)
(3.55(
73.
(5.533
(5.29
70.77 (5.52)
Doo
log
(5.56)
(5.55)
(8.45)).533
(5.59)
37.62
(5.35
35.39
Duo
x
0.9
log-lin
44.24 (5:40)
40.08 (5:40)
37.93 (5.39)
(5.37)
35.77 (3.37)
34.79
(5.93
34.75 (3.35)
1.0
(5.47)
09 (5:45)
(5.42)
$7.90
56.11 (5.43
$6.30 (5.41)
(5.42)
Doo
1.0
Ing-lin
64.35 (5.54)
62.92 (5.54)
59.82 (5.50)
58.77 (5.46)
58.32 (5.46)
57.82 (5.45)
55,39 (5.45)
55.89 (5.42)
Duo
0.9
DOS
34.20 (5.31)
$1.79 (5:29)
$1.09 (5/25)
30,05 (5.25)
29,82 (5:26)
29.08 (5:27)
29.52 (5.24)
29.75 (5.23)
Doo
0.9
log lin
35.92 (5.41)
32.98 (5.40)
31,49 (5.36)
30.32 (3.311
30.06 (5.29)
(5.28
(8.90
29.19 (3.25)
MDLM
X
1.0
D06
168.66 (5.68)
131.55 (5.66)
115.74 (5.64)
111.72
106.63 (5.63)
104.56
103.12 (5.82))
104.73 (3.62)
MULM
194.09
(5.67)
105.40 (5.03)
108.05
MDL M
X
0.9
58.33 (5.
46.1 (5.36)
40 66 (5.32)
(5.33)
37.64 32)
37. 39 (S.33)
36, 16.95 (5:31)
36.87 (5:31)
log-
(5.34)
(5.32)
(5
47
44
Ing-lin
(5.48)
(5.45)
49.51 (5.44)
(6.43))5.42
(5:42)
(5.42)
(5.41)
(5.8(5.5) (3.42)
0.9
(5.21)
(9))
24 (5.18)
(3.49(5.17)
(5.17)
(5.17)
MDLM
0.9
log: din
34.85 (5.26)
28.21 (5:23)
25.27 (5:21)
24,01 (3.19)
23.25 (5.18)
22:73 (5:17)
22.73 (5.17)
22.46 15.16)
40

---

Published as a conference paper at ICLR 2026
Table 13: Generative Perplexity (Gen. PPL) and Unigram Entropy on OpenWebText (Gokaslan & Cohen, 2019) with V-samplers using schedules matching ReMDM (log-linear step size) and non-distilled models (as in Table 12). We experiment with nucleus sampling, following Wang et al. (2025). The rescale schedule is most effective to improve the Gen. PPL while retaining the unigram entropy. The lightblue rows are the ones plotted in Fig. (left).
Algo
Eta
Nucless P
Gen. PPL (U)
32
64
128
256
512
1024
2048
40%
Ancespial Sompling
Doo
N.A
1.0
96.76 (5.57)
96.01 (5 (6)66
79.97 (5.55)
78.46 (5.53)
76.92 (5.54)
75.02 (5.53)
(5.52)
75.39 (5.52)
Duo
N.A
0.95
56.65 (5.19)
50.78
48.68
47.26 (5.46)
45.42 (5.45)
45.11 (5.44)
(5.12
44.84 (5.44)
Doo
(5.40)
4008 (5)4
37.93 (5.39)
(5.37)
35.77 (5.37)
34.79 (5.35)
(5.95
34.75
194.09
141.67 (5.69)
(5.677)
111.85 (5.65)
(5.64)
105.54
(05.40
(5.62)
MDLM
N.A
0.95
106.28 (5.61)
77.06 (5:55)
68.34 (5.53)
63.19 (5.51)
S8 80 (5.49)
56.94 (5.48)
(5.54
56.44 (5.46)
MDLM
N.A
0,9
70.34 (5.49)
$1.14
(5.69)
40,01 (5.37)
39 (5.35)
37.91 (5.34)
(5.32)
36.76 (5.31)
Cup Schedluee
Duo
0.005
58.78 (5.58)
(5.52
(5.56)
66.14 (5.54)
61.63 (5.53)
(5.51)
52,49 (5.51)
45.64 (5.45)
75.23
(5.50)
$2.06
(5.46)
Dao
0.95
55.56 49)
(547)
(5.96)
36 (5.41)
30.85 (5.37)
25,66 (5.37)
Duo
0.01
0.95
54.07 (5 18)
46.27 (5 46)
34.84 (545)
36,60 (5.41)
30.98 (5.37)
25.53 (5.51)
(5.23
15.19 (5.07)
Dao
0.005
0.9
44.06.00 (5 41)
1. 39)
30.95 (5.33)
27.37 (5.30)
22.78 (5:24)
18.66 (5.16)
14.33 (5.03)
Doo
(5:40)
36.75 (5.38)
32.27 (5.35)
(5:26)
18.74
(5.06)
10.88 (4.87)
MDLM
0.005
195.83
142.25
113.94
110.5 (566)
112.78 (5.67)
119.61 (5.69)
131.85
MDLM
0.01
198.02 (5.75)
s9 70)
125.25 68)
117.84
116.62 (8)
126. (5.71)
143.96 (5.73)
72
MDLM
0.005
0.95
106 (5.61)
74.97 (5:54)
(5.52)
$5.82 (5.44)
50.31 (5:47)
43.78 (5.44)
37.04 (5.44)
50.46 (5
MDLM
0.01
0.95
105 45 (5.61)
73.92 (5
61.41 (5.51)
52.81 (5.48)
43.03 (5.45)
38.85 (5.42)
(5.34)
24.31 (5 25)
0.05
41.08
26 26.33
(5.18)
(5.04)
MDLM
0.01
68.57 (5 (5.48)
(5.44)
(5.37)
32.38 (5.32)
66 (5.28)
21.57 (5.18)
16.26 (5.05)
11.67 (4.79)
Resoate
Doo
0.01
1.0
59.63 (5.58)
79:30 575
76.11 (5.56)
73.43 (5.55)
70.06 (5.54)
70.46 (5.53)
09.20 (5,54)
68.25 (5:53)
Dao
0.02
1.0
89.55 (5.58)
79.4415 15.4415.57)
75.98 (5.56)
72.99 (5.54)
69 (5:54)
68. 39 (5.53)
66.60 (5.53)
63.70 (5)52)
Doo
(5.45)
44/64 (5.44)
43.49
Dao
0.02
0.95
(5 49)
50/66 (548)
48.00 (5.47)
(5.46)
42.71 (5.3)
(5.43)
38.06 (3:40)
Dao
0.01
0.9
45.03 (5:41)
a000 (0)
38.17 (5.39)
36.60 (5.36)
$5.25 (5.35)
34.35 (5.54)
34.27 (5.35)
33.07 (5.33)
Doo
0.02
1099
45.04 (5.41)
40000 05 (b)
38,05 (5.39)
36,15 (5.36)
54.74 (5.35)
33.13 (5.33)
31,79
29:08 (5.30)
Dao
40
37.61 (5.39)
$3.35 (5.34)
31.17
Dao
39.67 (5.39)
37.21 (5.58)
(5.35)
$2.47
29.30 (5.30)
26.15 (5.28)
Doo
0.05
0.9
44.52 (5:41)
(5,40))))))
36.41 (5.38)
33.68 (5.35)
$1.00 (5:34)
26.94 (5:28)
25.61 (5:25)
19.21 (5.17)
MDLM
0.01
1.0
194.29 (5.74)
141.40 (5.69)
121.04 (5.67)
112.95 (5.65)
107.80
105. (5.64)
105.69 (5.63)
105.64 (5.63)
MDLM
0.02
1.0
194.54 (5.74)
14031 (0.00)
120,86 (5.67)
112.64 (5.65)
10% 26 (5.64)
105.55 (5.64)
104.47 (5.63)
105.61 (5.64)
10643
(5.82)
56.34 (5.477
56,29
54.42
(5.55)
57.32
(5.92
(5.45)
MDLM
0.01
0.9
70.45 (5 49)
51.33 (5.43)
(5.39)
40.14 (5.36)
58.68 (5:35)
37.64 (5.34)
36.48 (5.32)
35.10 (5:31)
MDLM
0.02
0.9
70.31 (5 (5.49)
51.06 (5 43)
43.51 (5.39)
39.61 (5.36)
57.88 (3.35)
36.23 (5.33)
(5.33
31.62 (5 (9229)
MDLM
0.03
69.89
50.76
(8.39)
38,86 (5.36)
36.77 (5:34)
(5.32)
(5.29)
27.19 (5.25)
MDLM
0.04
(5
50.30
42.84 (5.39)
38.02 (5.35)
35.73
32.44 (5.33)
28.55 (5.27)
23,72
MDLM
0.05
0.9
09:44 (5.48)
50.1 (5:42)
(2.39
31.27 (5.35)
(5.30
30.29 (5.30)
26,03 (5.25)
20.85 (5.16)
Loop Schedbile
Duo
0.01
1.0
108.15 (5.588)
65 58)
(5.16))
(5.15555.55
60:49 (5.55)
$6.35 (5.53)
53.06 (5.51)
48.93 48)
Doo
103.48 (5.58)
79.35
67.99
63.05
56.92 (5.54)
$2.69 (5.53)
(8.63(5547)
43.28 (5.37)
Duo
$1.36 (5.48)
26.97 (5.35)
Duo
0.02
0.95
61.61 (5 48)
47.46 (5
38.78
32.69 (5:40)
27.26 (5.36)
22.35 (5.29)
(5.23(5.222)
14.31 (5.06)
Doo
0.01
0.9
52,12 (5.40)
4027 (5 39)
33.71 (5.37)
28.73 (5.33)
24.47 (5:29)
20.32 (1.2255522 (5.23)
17.01 (5.16)
13.61 (5.05)
Doo
0.02
19.08 (5.40)
(5.38)
30,08 (5.34)
24.88 (5.29)
20.59 (5.24)
16.69 (5.16)
13.61 (5.06)
10.77 (1.92)
MDLM
0.01
1.0
340.32
(5.74)
(5,70)
127.32 (5.70)
(5.66)
63 (5.70)
(5.733
198.48 (5.77)
MDLM
338.2
(5.72)
140.73
(5.71)
(5.75)
346.89
MDLM
0.01
0.95
682.65 (5.67)
101 (5.61)
71.76
$8.43 (5.52)
$1.33 (5.30)
45:27 (5:47)
39.08 (5.4)
33.48 (5.38)
MDLM
0.02
0.95
177.31 (5 67)
97.61 (5.6))
68.49 (5.55)
55.21 (5.51)
47.71 (5.49)
41.64 (5:45)
34.91 (5. (5.40)
29 63 33:
MDLM
0.01
117.28 (5.55)
(5.45)
46.91 (5.43)
37.62 (5.38)
31.93 (5.24)
27.80
23.38 (5.25)
19.78 (5.20)
112.21
(5.48)
43,89 (5.42)
28.99
24.58
(5.20)
(5.13)
41

---

Published as a conference paper at ICLR 2026
Table 14: Generative Perplexity (Gen. PPL) and Unigram Entropy on OpenWebText (Gokaslan & Cohen, 2019) with V-samplers using schedules matching ReMDM (log-linear step size) and distilled models (as in Table 12). We experiment with nucleus sampling, following Wang al. (2025).
PPLL-())
Algo
32
64
128
256
512
1024
2018
4096
Ancestral Sampling
(2.92)3.54)
59.82
(5.46)
(5.42)
Duo
N.A
0.95
(5:47)
11.78 (3.46)
40.32 (5.43)
38.93
38.69 (5.37)
38.45 (5.36)
36.92 (3.33)
37.26 (5.33)
Duo
N.A
0.9
35.92 (5.41)
32.98
31.49 (5.36)
(5.31)
30,06 (3.29)
(5:28)
28,90 (5.255
29 19 (5.25)
MDLM
N.A
1.0
68.61 (5 (5-48)
55.26 (5.45)
49.51 (5.44)
46.15 (5413)544)
45.61 (5.42)
(5
44.53 (5.41)
44 18 (5.42)
MDLM
N.A
0.95
46.07 (5.37)
(5.33)
32,91 (5.31)
30.96
30.26 (5,29)
29.73 (5
29.54 (5.26)
.3 (5.28)
(5.23)
(5.18)
22.75
(5.77)
(5.16)
Cup Schoduuee
Dao
(5
(8.52)
(5.42)
(5.39)
(5.22)
19.25
Dus
0.01
1.0
64.22 (5.53)
(5,51)
49 90 48)
40.95 (5.39)
33.90 (5.34)
26 29 (5.24)
19.34 (5.11)
14:31 (4.96)
Dao
0.005
0.95
43.68 (54
18.77 (5.45)
$5.55 48)
31.36 33)
26.74 (5.28)
21 21.84 23)
16.22 65.06)
12.00 (400(4.00 (4.94)
Doo
Dao
0.005
0.9
34
30,95 (5.38)
28. (5.34)
24,47
21.25 (3.18)
17.02.
12.85 (4.99)
(4.81)
Dao
0.01
0.9
$3.91 (5 40)
29.27 (5.37)
25 23 (5.31)
21,40 (5.211
17.36 (5.13)
13.22 (5,00)
9.55 (4.82)
6.92 (4.56)
MDLM
0.005
1.0
67.27 (5
$2.34 (5.34(
44.38 (5.42)
(5.14
32.35 (5.37)
26.37
20.64 (5.27)
15.30
MULLM
0.01
1.0
66 29 (3.47)
(5.44)
41.29 15:40)
33.39
21.16 (5.34)
21.04 (5.26)
(5.19)
12.16 (5.08)
13.63
(4-98)
32.84
(5.29)
22.19 (4)
17.80 (5.19)
(4.76)
MDLM
0.005
0.9
33.81 15:26)
26.71 (5.22)
22.81 15.19)
19,65 (5.10)
16.67 (5.11)
13:79 (5.00)
10.74 (4.94)
8.10 (4.78)
MDLM
0.01
0.9
52.94 (5.25)
25.51 (5.22)
20,89 (5.18)
(5.19
13.91 (5.05)
10.91 (4-95)
(4.7
5.98 (4.54)
Resoale Schnduuee
Doo
56.18
(3.15(5.42)
(5.41)
Doo
62.24
(5:30)
(5.44)
(5.38)
Dao
0.01
0.95
45 (5:47)
41.74 41.74 (5.46)
39:99 (5.43)
38.30 38)
38.10 (5.37)
$7.51 36)
35.45 (5.33)5.33)
$4.71 (5:32)
Doo
0.02
0.95
44,89 (5.47)
(5.45
39.81 (5:43)
38.09 (5.38)
36.79 (5.36)
35.47 (5.35)
(5.311
29.25 (5.28)
Doo
35 (5:4))
33.05 (5.40)
31.55 (5.36)
(5.31)
29.94 (5.29)
29.70 (5.28)
27.73 (5.25)
27.43 (5.21)
(5.47
(5.36)
29.70
(5.28)
(5.22)
(5.19)
(5.42)
MDLM
0.02
1.0
68.73 (5:48)
51.85 (5.45)
48. (5.43)
(5,42)
(5.10
41 (5.4L)
(5.76(5.39)
34.66
MDLM
0.01
0.95
01 (5:37)
(5.38)
32.80 (5.31)
30.65 (5.30)
29.92 (5.29)
29 (5:28)
28.34
37 (5:27)
36.45
32.49 (5.31)
(5.28)
(5.28)
22.95 (5.24)
(5.18)
20.90
MDLM
0.02
0.9
$4.83 15
28.17 (5.25)
24.97 (5:21)
(5.19)
22.54 (5.17)
21.33 (5.17)
19.88 (5.15)
17.75 (5.12)
Loop Schedoie
Duo
0.01
1.0
80.39 (5.55)
61.64
$2.51 (5.52)
47.30 (5,30)
10,27 (5.41)
34.27 (5:40)
27.28 (5.32)
21.97
Dao
0.02
1.0
75 (5 55)
(5.55)
47.47 (5.52)
(5,33
34.18 (5.41)
28.72 (5.36)
22.16 (5.26)
(5.18)
Dao
(7.15
13.60
Duo
0.02
0.95
48.78 (5.48)
37.61 (5,46)
30.95 (5.43)
26.55
(5.30)
(5.64
13.84 (5.11)
(5.12
Dao
0.01
0.9
41.15 42)
32.51 (5.40)
27.96 (5 (5.38)
(5.39
20.52 (5.25)
(5.17 19)
(3.90
(5
Doo
30,04
24.99 (5.37)
(5.24
(5.21)
14.25 (5.13)
11,51
(4.94)
MDLM
0.01
1.0
99.76 (5.51)
(5.76(5.48)
47.50 (5:45)
39,07 (5.43)
32.85 (5:41)
28.01 (5:38)
(5.38)
(9.32
51)
MDLM
0.01
(5.37)
(5.33)
(5.28)
19.19
(6.12)(5.20)
(5
MDLM
0.02
0.95
61 (5 40)
(5.36)
28. (5.33)
(5:29)
[9:45 (5.26)
15.42 (5.60
[3.84 (5.16)
11.73 (5.09)
MDLM
0.01
0.9
48 36 (5
37,05
24.51 (5.23)
20 20.5615.200
(5.18)
13.19 1.199.5.00
11 19 (5.04)
46.12.
(5.46
42


Reviews:
[{"review_id": "NIrU0BT2lk", "reviewer": "Reviewer_NGPT", "summary": "This work proposes two improvements for uniform discrete diffusion models. The main contribution is a new family of predictor–corrector samplers (called psi-samplers) based on non-Markovian “psi-posteriors.” \n\nIn more detail, they first define an x-conditional version that is a convex combination of the forward process and x-conditioned reverse posterior. The true reverse posterior then turns out to be a convex combination of q_{s|t} and q_{0|t} for 0 < s < t < 1. The reverse posteriors preserve the same marginals as standard diffusion while allowing adjustable stochasticity during sampling. This leads to empirical gains over ancestral sampling for uniform-state diffusion models, particularly in text and image generation tasks. Unlike standard samplers that plateau with number of steps, Psi-samplers are shown to continue improving with more steps (perhaps could be interpretted as a type of Gibbs sampling in the regime where steps > dims). \n\nA second contribution includes a practical efficiency optimization for a \"curriculum learning\" step from prior work, used during training. By approximating the vocabulary-wide averaging operation with a sparse top-k estimate, the authors reduce training time and memory usage while maintaining comparable perplexity and downstream accuracy. Altogether the combination of the train-time efficiency gain plus the new samplers give efficiency, perplexity, and sample quality gains on a number of tasks relative to a number of baselines.\n\nThe sampling idea is novel and presented cleanly, while the curriculum result is an engineering improvement, still meaningful, but not quite clearly defined or motivated besides the literal algorithm of the proposed computation. \n\nOverall, a solid technical contribution with good empirical validation. I view the sampling advance as the core insight and the training speedup as a perhaps useful but secondary contribution, at least given the amount of explanation/exploration of the underlying method being optimized.", "questions": "- there is some emphasis on the non-markovianness of the samplers. It would be curious to see where exactly this non-markovianness really matters, for example by seeing which conditioning values or time steps are correlated.\n\n- could $\\kappa_t$ be learned or adapted dynamically during sampling rather than chosen?\n\n- the paper says psi-samplers improve with increasing NFEs while ancestral ones plateau. Why exactly does extra noise continue to help after many steps? If you run it for a while (steps >> dims/length), does it ever settle? Or if not, is there some relationship to Gibbs sampling?\n\n- It seems like the psi posterios could be used for purely continuous diffusion models too. Has something like this been done in that setting? What would be things to watch out for?  More generally, is there any such trick in other areas of sampling that motivates this approach?\n\n- when $\\kappa_t < 1$, the sampler is said to be “noisier.” How is this related to the variance schedule $\\alpha_t$? Is there any sense in which $\\kappa_t$ values index an implied different set of $\\alpha_t$'s? \n\n- why does the largest gains from psi-posteriosr occur at low noise levels and $\\kappa_t$ close to 1? Is this due to the denoiser’s inductive (architectural) bias? how the mixture interacts with the diffusion schedule? some special property of the sequence of psi posteriors over time? something else?\n\n- could the authors visualize or quantify how psi-sampling trajectories differ qualitatively from ancestral ones?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- clear motivation for why sampling needs improvement in discrete diffusion: standard samplers can \"over-commit\" and cannot \"self-correct\" without further hacks, sometimes including adversarial training or other approximations.\n\n- the psi-sampler formulation is general and recovers previous a few predictor–corrector methods as special cases (though please careful to not say \"all cases in the literature\", you never know with so many papers coming out daily, I suggest \"that the authors are aware of\")\n\n- broad experimental coverage: language modeling, QA, and image generation for tasks + ablations on the \\kappa_t hyperparameter and time-scheduling of where to use the psi-posterior.\n\n- consistent empirical setups with appropriate baselines (Duo, MDLM, ReMDM) drawing on the Sahoo, Lou, etc... works.\n\n- demonstrates measurable improvement in both FID/inception (images) and (generative GPT2) perplexity (on text) at fewer sampling steps, with nice gains in performance as NFE increases.", "weaknesses": "- The curriculum section assumes prior familiarity with Sahoo 2025a and gives little intuition for why that weighted-average operation helps training. Since you are devoting nearly a whole section to this method, I ask that you at least give a few sentences on how exactly the \"curriculum\" technique during training uses this average computation that you are approximating. Otherwise, the speedup could be relegated to an appendix section (I think \"curriculum\" is used 23 times in main text without being defined).\n\n- experiments, while complete, lack interpretation, or at least attempts at providing intuition: results are shown, but, e.g., there is no discussion of why low-noise $\\kappa_t$ regimes work best or what happens mechanistically when $\\kappa_t$ is too small or too large\n\n- the connection between psi-samplers and the NELBO s not explored, which might provide extra insight on what's going on.\n\n- somewhat minor: when describing t_off, t_on, etc... it's possible that t is overloaded. Are you describing a function, t_for_model_input(t_real)? If so could you distinguish in symbols? It was a bit hard to read phrases like \"t is constant in [t_off, t_on]\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JOL8XMIAnd", "reviewer": "Reviewer_FN73", "summary": "The authors tackle the problem of fast sampling in discrete diffusion models and introduce a family of predictor corrector samplers that generalize over prior methods and apply to arbitrary noising processes. The authors then combine this framework with the Uniform state discrete diffusion models and show that the method outperforms baselines on text and image generation tasks. The authors also propose a curriculum learning strategy which improves upon the training efficiency of existing discrete diffusion models.", "questions": "See weaknesses", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. I like Figure 1 which conveys the main results of the paper convincingly.\n2. The paper is well written in the sense that the background section is well formulated, the main contributions of the paper are well supported by empirical results.\n3. The idea of formulating non-markovian forward processes for discrete diffusion models is quite interesting given its numerous applications in the context of continuous diffusion models in the form of DDIM.", "weaknesses": "I dont have a lot of concerns around the proposed method but rather a few suggestions for improving the presentation of the paper.\n\n**Presentation Issues**\n\n1. Is there a reason for using the psi notation to denote distributions throughout the paper? We can probably get rid of notations and denote distributions using their standard notations like p(.) or q(.) like other works in the literature.\n\n2. In general, a lot of intuition is missing around the sampler design in Section 3. It is not clear, why can we expect the non-markovian design of the forward process to yield more efficient samplers? Can the authors provide some intuition around the sampler design in this context? The formulation of the non-markovian diffusion forward processes in Section 3 reminds me of the DDIM formulation for continuous time diffusion models so maybe adding some intuition from the continuous time diffusion processes literature could be useful?\n\n3. Similarly in Eq. 10 what is the intuition behind different terms in the psi-posterior? All of these intuitions can be clarified in the main text to improve readability and presentation. In the current form the section appears to be very dense in equations which hinders readability.\n\n4. Can the authors provide some intuition of the Curriculum learning strategy proposed in Section 4?. The section seems very heavy on notation and it is hard to understand the high level idea on a first read.\n\n5. The authors can also consider revising the title of the paper as it currently seems very opaque.\n\n**Empirical Issues**\n\n1. In Fig. 1 unlike the MDLM and Duo samplers, the performance of the MDLM + ReMDM baseline and the proposed method keeps improving. Do the authors have updated results on how these curves saturate? I guess the cost of sampling beyond 4k steps is probably too high from a practical standpoint but it could still be worth exploring it from a research standpoint.\n\n2. Can the authors report the FID score comparisons in Fig. 3. Im curious how discrete diffusion models compare to continuous time models on image quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vKQSe9JrVH", "reviewer": "Reviewer_XWfo", "summary": "The paper propose predictor-correct samplers for discrete diffusion models. The authors:\n\n1. First identity posteriors that sample from the marginals of a masked diffusion model. \n2. Next, the authors propose posteriors, that are a mixture of distributions, a predictor for un-masking, and another distribution to re-noise. \n\nAdditionally, the authors also propose an efficient estimator for the curriculum learning objective of uniform state diffusion models. The authors note that:\n\n1. There is a deterministic push-forward map between Gaussian and uniform state models. The denoiser can therefore be trained with continuous Gaussian noise. \n2. However, for each token, the curriculum learning objective in Sahoo et al 2025a requires sampling several high-dimensional Gaussian vectors, which can be expensive for large vocabulary sizes. \n3. The authors then propose a cheaper approximation that leads to reduced memory usage and faster training. \n4. Computing the noise for each position requires $|\\text{Vocab}|$ many Gaussians, however the authors rely on order-statistics to sample from the $k$ largest values of a Gaussian vector. \n\nThe authors then show that the superposition samplers combined with the diffusion duality model (Sahoo et al 2025a), a uniform state diffusion model, can match the performance of masked diffusion language models. The authors also show that using in-expensive approximations to compute the continual learning objective does not lead to a significant deterioration in performance.", "questions": "1. To compute the test perplexity in table 2, how many $t$ samples do the authors use for each sample $x$. \n2. Can the authors also demonstrate the predictor-corrector methods they propose with existing pre-trained diffusion language models such as LLaDA and Dream.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The paper derives rigorous predictor-corrector schemes for both masked and uniform state diffusion models, as well as tractable approximations for the uniform state diffusion models.", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WxkAyMm595", "reviewer": "Reviewer_78X8", "summary": "The paper introduces a family of Predictor-Corrector (PC) samplers for discrete diffusion models, referred to as $\\Psi$-samplers, which extend and generalize existing PC methods to work efficiently with arbitrary noise processes. The authors also propose a scalable and memory-efficient curriculum learning strategy for training these models.", "questions": "1. Why is the Predictor-Corrector approach effective in improving sampling efficiency and generative quality in discrete diffusion models? Does the paper provide a detailed theoretical foundation explaining why this approach works, particularly regarding the non-Markovian dynamics introduced by the $\\Psi$-samplers?\n\n2. How might the proposed Predictor-Corrector $\\Psi$-sampler relate to techniques like [1], [2], and [3], which have recently been proposed to improve generative quality by directly optimizing conditional variance/entropy during sampling?\n\n3. Does the $\\Psi$-sampler consider resolving, or at least attempting to address, the significant numerical instability issue arising from the schedules in diffusion models as $\\sigma_t$ approaches 0?\n\n[1]. Li, S., et al., EVODiff: Entropy-aware Variance Optimized Diffusion Inference, NeurIPS 2025.\n\n[2]. Ifriqi, T. B.,  et al., Entropy Rectifying Guidance for Diffusion and Flow Models, NeurIPS 2025.\n\n[3]. Stancevic, D., et al.,  Entropic Time Schedulers for Generative Diffusion Models, NeurIPS 2025.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper presents $\\Psi$-samplers by generalizing the PC approach to discrete diffusion models, providing a unified framework that extends prior work.\n\n2. The provided experiment shows that this memory-efficient curriculum method is likely effective, while empirical improvements are achieved in both language modeling and image generation tasks.", "weaknesses": "1. The $\\Psi$-sampler introduces non-Markovian posteriors, but the theoretical discussion heavily relies on Gaussian diffusion analogies, and the impact on sample quality, variance, and convergence remains underexplored. \n  \n\n2. The method is highly sensitive to hyperparameters like $\\kappa_t$ and noise schedules, and the evaluation primarily focuses on OpenWebText and CIFAR-10. \n\n3. The paper claims that generation quality improves with more sampling steps, but this holds true for language modeling tasks, not image generation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)