What Exactly Does Guidance Do in Masked Discrete Diffusion Models
Authors: Ye He
Kevin Rojas
Molei Tao
Venue: ICLR 2026 Poster (2026)
Keywords: Discrete Diffusion Models; Classifier-free Guidance
Primary area: generative models
TL;DR: A theoretical analysis of the generation in masked discrete diffusion with CFG

Abstract: Masked discrete diffusion models have been gaining popularity recently, and classifier-free guidance, just like its continuous counterpart, has been proposed to enable efficacious conditional generation by discrete diffusion. To quantify the precise effect of discrete guidance, this article considers masked discrete diffusion with arbitrary data distribution in low dimension, so that 
   the distribution that guided masked discrete diffusion samples from, as well as the sampling dynamics, can be analytically and exactly quantified and interpreted. When the full data distribution is a mixture over classes and the goal is to sample from a specific class, guidance amplifies class-specific regions while suppresses regions shared with other classes. This effect depends on the guidance strength $w$ and induces distinct covariance structures in the sampled distribution. Notably, we observe quantitatively different behaviors in $1$D and $2$D.  We also show that for large $w$, the decay rate of the total variation ($\text{TV}$) along the reverse dynamics is double-exponential in $w$ for both $1$D and $2$D. These findings highlight the role of guidance, not just in shaping the output distribution, but also in controlling the dynamics of the sampling trajectory. Our theoretical analysis is supported by experiments that illustrate the geometric effects of guidance and its impact on convergence.

Published as a conference paper at ICLR 2026
WHAT EXACTLY DOES GUIDANCE Do IN MASKED DISCRETE DIFFUSION MODELS
Ye He, Kevin Rojas, Molei Tao School of Mathematics Georgia Institute of Technology
Atlanta, GA 30332, USA
{yhe367, kewin. rojas, mtateggattcc.. edu
ABSTRACT
Masked discrete diffusion models have been gaining popularity recently, and classifier-free guidance, just like its continuous counterpart, has been proposed to enable efficacious conditional generation by discrete diffusion. To quantify the precise effect of discrete guidance, this article considers masked discrete diffu- sion with arbitrary data distribution in low dimension, so that the distribution that guided masked discrete diffusion samples from, as well the sampling dynamies, can be analytically and exactly quantified and interpreted. When the full data dis- tribution is a mixture over classes and the goal is to sample from specific class, guidance amplifies class-specific regions while suppresses regions shared with other classes. This effect depends on the guidance strength w and induces distinct covariance structures in the sampled distribution. Notably, tively different behaviors in 1D and 2D. We also show that for large w, the decay rate of the total variation (TV) along the reverse dynamics is double-exponential in for both ID and 2D. These findings highlight the role of guidance, not just in shaping the output distribution, but also in controlling the dynamics of the sam- pling trajectory. Our theoretical analysis is supported by experiments that illus- trate the geometric effects of guidance and its impact on convergence.
observe quantita-
INTRODUCTION
Diffusion models have become an influential tool for generative modeling, offering a flexible frame- work that performs well across range of data types including images, audio, and text (Dhariwal & Nichol, 2021; Kong et al., 2021; Li et al., 2022; Ho et al., 2022). Originally formulated continuous state spaces (Ho et al., 2020; Song et al., 2021), these models simulate stochastic differential equa- tion that gradually adds Gaussian noise and then learns a reverse process to denoise and reconstruct the data. More recently, discrete diffusion models have been proposed (Campbell et al., 2022; Lou et al., 2023), replacing Gaussian corruption with categorical transitions or masking, which makes them well-suited for language modeling, molecule generation, and protein design (Lou al., 2023; Nie et al.; Huang et al., 2023; Gruver et al., 2023).
A key innovation that has enhanced the performance and flexibility of diffusion models is guidance. In continuous models, classifier guidance (Dhariwal & Nichol, 2021) ) and classifier-free guidance (CFG) (Ho & Salimans, 2021; Nichol et al., 2022) steer the reverse process toward desired condi- tions, such as class labels or text prompts. These methods have been critical to the success of models such as GLIDE (Nichol et al., 2022) and Imagen (Saharia al., 2022), and theoretical work has be- gun to explain their mechanism in simplified continuous settings (Bradley & Nakkiran, 2024; Wu et al., 2024; Chidambaram et al., 2024). However, the extension guidance to discrete diffusion models is much less understood. Recent proposals adapt CFG to discrete domains by modifying transition rates or reweighting transition kernels (Nisonoff et al., 2024; Schiff et al., 2024), and em- pirical results show notable gains in sample quality and controllability (Schiff et al., 2024; Xiong et al., 2025). However, the theoretical understanding of how guidance affects the dynamics of the diffusion process in discrete state spaces remains limited.

---

Published as a conference paper at ICLR 2026
Motivated by this gap, our paper develops rigorous and quantitative framework for analyzing the effects of CFG in discrete diffusion models introduced in Nisonoff et al. (2024), with a particular focus masked discrete diffusion models (Campbell et al., 2022; Shi et al., 2024; Sahoo et al., 2024; Ou et al., 2024)-a common subclass of discrete diffusion models. To make the problem tractable while still capturing essential behaviors, we assume exact scores and exact reverse dynamics, and we investigate the following two fundamental questions in low-dimensional settings (ID and 2D):
Q1. How does guidance affect the distribution of the generated samples?
For this question, we assume that the data distribution p is a mixture of different class distributions (Assumption 1.1). We find that guidance amplifies probability mass in class-specific regions while reducing mass in overlapping regions, which vanish entirely as the guidance strength w grows. The strength of this amplification and reduction depends explicitly on w, and the covariance structure of the resulting distribution differs between 1D and 2D.
Q2. How does guidance affect the rate of convergence of the reverse dynamics?
To address this, we analyze the total variation distance between the distribution along the reverse dynamics and the final sampled distribution. For both 1D and 2D, show that the decay of this distance exhibits double-exponential dependence on the guidance strength w when w >> 1.
By characterizing these two aspects- distribuuiaii shifts and convergence rates-our work bridges the gap between practical heuristics and theoretical understanding in discrete diffusson with CFG.
Assumption 1.1. Let {= be the set of M labels, each of which is associated with a class distribution p(1zz)) supported on Xx S. S. The full data distribution p is mixture of distributions (p(-Iz))) with weights { k= 1 i.e. p(-) = ΣM - axp((lzx).
Comparison to Existing Work. Our analysis is closely related to recent theoretical studies of guid- ance in continuous diffusion models (Bradley& & Nakkiran, 2024; Wu et al., 2024; Chidambaram et al., 2024). A key insight from this line of work is that in continuous settings with data distributions mixtures (Chi-
such 1D Gaussian (Bradley& & Nakkiran, 2024) 1D Gaussian/compatt-uuppe dambaram et al., 2024), CFG reshapes the reverse dynamics in a way that makes the generated distribution deviate from the tilted distribution, i.e., the distribution defined by the steered score that emphasizes the conditioning signal. These results, however, rely heavily on Gaussian or compact- support assumptions and are limited to ID continuous space. By contrast, our study provides the first rigorous analysis of CFG in discrete diffusion models under much more general conditions, where the data distribution can be any finite mixture of arbitrary class distributions. Leveraging the tractability of masked discrete diffusion, we derive explicit reverse dynamics in both 1D and 2D that reveal striking similarities and differences to the continuous case. In particular, we find that in 1D the generated distribution coincides exactly with the tilted distribution unlike in continuous diffu- sion-while in 2D deviations do appear, but in a form that we can quantify explicitly rather than only show that sample diversity decreases as masses in overlapping regions
approximately. Moreover, vanish under strong guidance, and that convergence rates exhibit a double-exponential dependence on the guidance parameter, echoing but also amplifying phenomena observed in continuous settings.
Paper Organization. The remainder of the paper is organized as follows. Section 2 introduces preliminaries on discrete diffusion models relevant to our analysis. Section 3 presents our theoretical analysis and their implications of the guided diffusion process, and Section 4 provides numerical examples supporting our findings. Conclusions are diseussed in Section 5. Additional related work and technical details are included in the Appendix.
2 PRELIMINARIES
2.1 NOTATIONS
For any E RD and A C (1, DD. we use E R| to denote restriction of x to coordinates
in A. Vi is used to denote (1.2...., (i). For any distribution p. p(x denotes the A A-marginal of p evaluated at ZA. For functions f.g. write f(w) ~ g(w) if lim f(w) //g///(() (w)
1 and f(w) = ((g(w)) if clg(w) < f(w) S c2g(w) for some CJ C22 Wo > 0 and all w >> Wy. For any set X' C (1,,, ,,N N and $ d D, Xa= == (aa | x E X3 is the projection of X' on dimension-d.
2

---

Published as a conference paper at ICLR 2026
2.2 DISCRETE DIFFUSION MODELS
We consider the state space 5 = (1, ,N The data distribution is represented as a vector in RN" that sums up 1. The discrete diffusion forward process is defined continuous-time Markov process (Campbell et al., 2022; Lou et al., 2023), given by the differential equation
dpt QePt. dt
Po P
where Qt E R RNOXN is the transition rate matrix for all 2 0 s.t. (1) Q x) 2 0 for all z, y E S and x: + y: (2) E JES (v) = 0 for all E S. In this paper, focus widely used effective forward process, the absorbing forward process (Austin al., 2021; Lou et al., 2023; Shi et al., 2024; Sahoo et al., 2024; Ou et al., 2024). which independently transforms across all dimensions .N)..
until arrive at the masked state [M] == (N, NN.N.... ,T)T. The explicit expression of the masked transition rate matrix and its properties will be discussed in Appendix B. The reverse process in discrete diffusion models is defined
dg:
Qr- Qu Prs
(2)
di
where the [QeJosegr is sequence of reverse transition rate matrices given by
(r. - -ofo Q((s,2),
p=(v) Pila)
xr
Q((v,))=
(3)
y =
and {Pe):2 solves equation 1. Equation 2 is the exact reverse of equation 1, i.c., Pr-r for { [
all E []. The ratios
(e(x)) Jos are concrete scores (Meng et al., 2022) which generalize the score function V Vlog Pt (z) in continuous diffusion models. In practice, the concrete scores are
learned via denoising entropy matching (Lou al., 2023) by minimizing the following denoising score entropy:
Piooy((ro) Pr p=o
DDE Eroon mm))
Σ (x,
log (x, v)],
(4)
v
where of (zr, y) is the parametrized score to approximate imating the following reverse process:
((()) p(
Last, samples are generated approx-
dgf Q7 Q7-10 du 6|M
where Of replaces the exact concrete score Pe(y) (a) with s (xx y). Simulation of equation 5 can be performed with the Gillespic's Algorithm (Gillespie, 1976), Tau-leaping (Gillespic, 2001; Campbell
al., 2022) and uniformization (Grassmann, 1977; Chen & Ying, 2024),
In this paper, we assume exact score and exact simulation of equation 5, and focus on the generation ability along the continuous-tinn reverse dynamics with CFG. The effects of score approximation and numerical discretization are left for future work.
2.3 DISCRETE DIFFUSION MODELS WITH CFG
Classifier-fres guidance (CFG) is designed to steer the generative process toward samples consistent with target condition, such specific class label. In our paper, consider to generate from one label class from the full label classes {z: J defined in Assumption 1.1. In the discrete setting, Nisonoff et al. (2024) introduced CFG by tilting the reverse dynamics. Formally, the construction begins with the tilted distribution
p² & p()))( x p(.) p((1)))pp-)--->
(6)
where w 2 -1 is the guidance parameter. This expression makes the effect of guidance transparent: setting w -1 recovers the full data distribution; 0 yields the conditional distribution class
3

---

Published as a conference paper at ICLR 2026
larger positive values of progressively amplify the likelihood p(z[z), biasing the distribution toward states that are more consistent with class 2.
Since diffusion models proceed through dynamics, can't tt sample from p²' directly. In tinuous diffusion models, one steers the scores log Pt Pt(x) during the reverse process. In discrete models, the analogue is to steer the reverse transition rates, thereby embedding the effect of tilting directly into the evolution of the process.
Concretely, alongside the unguided reverse rates Q, in equation 3, define class-conditional verse rates Oj by evolving the conditional distribution p(-|z) under the same forward transition Q
dpa(- [) Qepe(-12), Po - p(-1z)
(7)
dt
leading to the conditional reverse rate matrix
p=(p)=) PL
(r( Eefe (s,,
yサ ==z.
(v,
(8)
By analogy with the tilted distribution equation 6, the guided reverse dynamics interpolate between Qi and Qi through the CFG rate matrix:
2)) ((ll)x))"(((yaaaa "Q Eofo Qi (s,
y x, y = x,
Q (y.
with the corresponding evolution
dq Q7 4: 2.30
2,10
=
= 6MM)
(10)
dt
The choice of guidance strength w can significantly affect the generation:
when w = -1, 2: = Qt. it recovers the unguided reverse process. This generates the entire mixture p, spreading probability mass across all classes and offering control over which class is realized.
when w = 0, Q7 Qi it recovers the class conditionn reverse process. This generates exactly from p(-12), isolating the desired component of the mixture.
Empirically, the best generation results are obtained for intermediate values w > 0. In this regime, the process in equation 10 interpolates between reproducing the full mixture and enforcing strict conditioning, often yielding sharper and more faithful samples than either extreme. Yet why this works so well is far from obvious: CFG with w > does not correspond to sampling from any simple explicit distribution, but rather modifies the reverse dynamics in nonlinear way. A central goal of this paper is make this phenomenon precise, by rigorously characterizing how guidance reshapes distributions and convergence rates in discrete diffusion models.
ANALYSIS OF MASKED DIFFUSION WITH CFG IN Low DIMENSIONS
We now analyze how classifier-free guidance (CFG) reshapes the reverse dynamics in masked dis- crete diffusion. Our focus is on the low-dimensional settings D = 1 and D = 2, where explicit formulas for the reverse dynamics can be derived. These tractable cases provide two key insights:
1. Generated distributions: the effect of CFG on the final generated distribution relative to the tilted distribution in equation 6.
2. Convergence rates: the speed at which the reverse dynamics approach the generated dis- tribution.
Interestingly, the behavior of differs sharply between 1D and 2D: in 1D the generated distribution coincides exactly with the tilted distribution, while in 2D discrepancies appear that we nonethe- less characterize explicitly.
For clarity, we focus sampling from class 21, denoted by when the subscript is essential.
4

---

21.0 admits the Published as a conference paper at ICLR 2026
3.1 D 1: SINGLE-TOREN GENERATION
For D = 1, the reverse transition rate matrix with CFG simplifies dramatically. It coincide with the reverse rate matrix associated with the tilted distribution p²* up to a normalization constant g = -> p(z) "((a[=)++u In other words, the guided reverse dynamics in ID behaves exactly like an unguided reverse dynamics targeting the tilted distribution. For details, please see Proposition B.2.
This leads to the following explicit characterization.
Theorem 3.1 (1D revserse dynamics), If and q.th satisfies the sampling dynamics equa- tion 10, we huve that for all S S.
I-c )p
(T--))-))
I 1,2, .N ,N x N,
di 3. (x)
(1I)
1-e
(T-1)
where = Z Lg p(x) - =p((=)))++) Moreover, the generated distribution exactly the tilted distribution, i.e., 4T p²
The explicit formula in Theorem 3.1 immediately yields properties of the sampled distribution and convergence rates, whose proofs deferred to Appendix D.
Proposition 3.1 (ID generated distribution). Assume the full distribution satisfies Assumption 1.1, depending on the supports of class 21 and other classes, the generated distribution 9f following different behaviors:
(1) if X1 r Xx
0 for all k 2 2. -0 =((112) for all 2 0.
(2) if = X _ Nx) # 0 and 11 == {k: : 24 n 21 * 0}.
p(z|21);
E X1 \. St:
F 2. (x) x
asp(al=)) Ener aup(a(EN))
w "p(x(11)
E St
otherwise.
2:.00 Px (1121) pointwisely, where Pxiys, (121) is the re- Moreover, as w + 0C,
4T
striction of p(-1z)) to the set X1 S₁.
Remark 3.1 (Local mean/variance preservation). Proposition 3.1 suggests that the generated dis- tribution preserves the local mean/ covariance of class-I distribution within the class-unique region (Xi \ S1). Intuitively, CFG in 1D transforms mass in the ambiguous region S and redistribute it to class-unique region while preserving the local mean and variance within the unique region. Please refer to Appendix D for the detailed argument.
Proposition 3.2 (1D convergence rate). Under the conditions in Theorem 3.1, for all 0 < SIST and 0, TV(g."". pp."" (---))))))
Remark 3.2 (Double-exponential effect), The decay TV is exponentially time, with rate Z. For all > 0, alternatively we can represent Z
where Da ((2))))) = - log E. -r( the a-divergence from μl to p2 all a E (0, oo) According the p(z|z)
property of a-divergence, we immediately have log( ( ~ w sup, p(z) for w > 1. Therefore, the overall decay rate of TV exhibits a double exhlee-epponentall dependency on w. This explains the extremely sharp transitions we observe practice when w is large.
3.2 D = 2: MULTIPLEE-OKRR GENERATION
In 2D. the story changes. The guided reverse rate matrix no longer coincide with that of the tilted distribution (denoted as ""). i.e. QJ * (Q. for any constant C. This discrepancy marks the departure between the discrete diffusion with CFG in 2D and its ID analogue.
We derive an explicit expression for QZ t in Appendix E. The key takeaway is that Q: not only depends n. 2, but also coefficients (Cx, d. J that encode the steering effect of
5

---

Published as a conference paper at ICLR 2026
guidance marginals, which are defined as follows: for all 21, 1,2, ... ,N 1,
dxy E(1, 222) >p(1,
Cz2 p(x1) (p=))=p((=1==2===
pla2) (122)_p(222)) "
(12)
CN En p(1,,12) 12) (apoll))ppppppppp 12/2) dN En p(1, (11l1))"pppppppaap p(11,1212)
(13)
Ei,
Σi, p(12) p(pp)ppp(2))))))))))))
Theorem 3.2 (2D reverse dynamics). fion 10, for all 0 ST.
If D = 2 and gi= satisfies the sampling dynamics equa-
(,()) (x))
31,32 # N 2s = # T/
& (x)
ax( 5. (a:),
(14)
as(a),
I 22 N.
where Z ExeEP(a)) "p(x)))) and
r(w) -AN.N AN.N
r(x) -AN
1
+ r(1)--NN)) ) (r(t) -r(t)= (xit)
A2g
AN.N
I1, + N,
Nw tGa,
NN. t.
ar(x)=
(r(1)sx)
31 N x2.
Goy (Air tcn
22 N 21
dag (2xe tdzz
r(t)
ANN,
I1 122 N.
with 2,00 ANN = Z(1//ON + 1/dx) and r(t) = 1- 1-e-1 (T
1-c
Although the reverse dynamics looks complicated, its implications are striking. Remark 3.3 (Explicit expression for the generated distribution). By evaluating 4 time, we find the generated distribution differs from the tilted distribution. Instead.
5. at the final
1/cz, + 1/ddaa (x),, Vx {1,2,.... .N.
(15)
I/eN + 1/dx
Proposition 3.3. Under Assumption 1.1, depending on the projected supports (ZuahisesMMIsae the sampled distribution 21 admits the following different behaviors:
47
(1) If X1 a Nx.a 0 for 1,2 and all 2 2, we have o p(-121).
(2) Si.a = X1,1 n (UN2 ax.a) + for some d = 1,2.
AqI." p(zl)))))
E R1, R2 i 1,2,
A 1 p()(2)),
G7 21 (x) x
Ag w p(x(z1),
x E R3:
A p((2)))
E R1,
0,
otherwise.
where R1, R2,1, R2,2, Ra, Rr forms partition of X1 and AT A A4 A.r. A are associated weights on the regions. Explicit expression are given below:
A71."" 2
R1 = {x| X1,m1 E X1,1 81,1, E X1.2 \ S1.2..
a1p(2.(21) w
A 1
R2 {x|æ Mi S1 E MA S113
Eeen, (xp(Ir((ee)
a:p(o; (21)
43
Rs = (afz M. \ (1, S1,1, E S12))
Eketi axp(Zz/Ex)
a1p(2i|21)
a1p(2|21)
A/1."
i=1 ExEn, axp(1:(2u) (
)")
Exen (np Ra St.
where S X n UUM X2), 11 = [k: X's X + 0}, 4 == {k: Xe. O Ki,d & 0}. Last, as w + xc. سل + (-121) pointwisely, where (1)20())2)) satisfies that Supply (.121) Xi \1

---

Published as a conference paper at ICLR 2026
Remark 3.4 (Effect of guidance on generated distributions). In Proposition 3.3-(2), the generated distribution is a weighted version of class-1 distribution. The regions R1, R2,1, R2,2, R3, R4 re- flect different level of "privacy" of class 21: R4 is the overlapping region with other classes. R1, R2,; R3 are not overlapping with other classes. But R3 has both projections overlapping with other classes and R2, has projections along dimension-i overlapping with other classes. R1 is the most private set in class 21, with no intersection with other classes even for projec- tions. For all Z O, the associated weights (before normalization) on different regions satisfies, AP A 2 A A This reflects that the sampled distribution from the discrete dif- fusion with CFG can leverage the geometric information of the full data distribution: the sampled distribution puts larger weights on more private regions of class 21-
To make the above discussion more concrete, Figure/ illustrates a toy example where the full data distribution is mixture of two uniform distributions supported on overlapping squares. We focus on sampling from the bottom-left class. The figure shows how classifier- free guidance (CFG) reshapes the generated distribution compared with the class- class-conditional distribution, redistributing probability mass away from overlapping regions as the guidance strength increases.
Remark 3,5 (Discussion on (-12)) Under Assumption 1.I, (-121) has zero mass on overlapping region between class 21 and other classes. In the non-overlapping region, explicit for- mula for (-Iz, ) can be derived from the expression of q*t:""(- in Proposition 3.3. However, it strongly depends on the nullities of the regions in Proposition 3.3-(2). We refer the readers to Appendix E. 1 for a detailed discussion.
Proposition 3.4 (2D convergence rate). Under the conditions in Theorem 3.2. for all 0 S 2,30 :9T ")) = exp(O(w)) In (1-
and w > 1, - (T
As in 1D setting, the decay of TV in 2D exhibits a double-exponential dependency in w as well.
B.COM
JACW
2000
C0R0
CONO
COMO
632
VUA
C15
AM
2200
0000
GATU
02000
60000
50000
2000
613
32/18
C300
3000
62817
01313
C1731
5000
2000
-0.0
Dimesin.1
(a) Partition of the support into regions R1, R2,1, R2,2, Rs. R4.
(b) Heatmaps: generated distribution o for u (left), and f w for w = 3 (right).
Figure 1: Illustration of CFG effects in 2D: (a) how guided distributions change with w, (b) how class support is decomposed into regions reflecting different levels of overlap.
3.3 DISCUSSION: TOWARDS HIGHER DIMENSIONS
The explicit 1D and 2D analyses highlight two general principles of CFG in discrete diffusion. First, guidance reshapes the generated distribution by reallocating probability mass away from overlapping regions of class supports and toward class-specific regions. Second, the rate of convergence to the generated distribution exhibits a double-exponential dependence on the guidance parameter w. These effects are transparent in 1D and 2D, but the structure they reveal is suggestive of higher- dimensional behavior.
In higher dimensions (D 2 3), we expect the interaction between classes supports to be governed not only by their full overlaps but also by overlaps of their marginal projections. More precisely:
Unique regions: states in X1 whose projections are disjoint from the corresponding projec- tions of all other classes should retain largest relative weight under CFG, with local mean/ covariance preserved.
Partially overlapping regions: states that do not overlap with other classes but share marginal projections (e.g. along subset of coordinates) are expected to be downweighted.
7

---

Published as a conference paper at ICLR 2026
The strength of this downweighting should depend ratios of conditional marginals, gen- eralizing the coefficients {Cz, dz} from the 2D case.
Fully overlapping regions: states in the intersection X1 n U are suppressed as w 2o, with the probability mass redistributed towards the two types of regions above.
The convergence rate is also expected to follow the same pattern: the decay of total variation dis- tance is exponential in time with an effective rate parameter that grows exponentially in w. In higher dimensions, although the rate will involve multiple interacting coefficients (generalizations of (Ca: dur that encode how strongly the guidance penalizes overlap along different subsets of co- ordinates, the double-exponential dependency in w is preserved due the exp(w) dependencc in the trace of the reverse rate matrix.
Overall, the 1D and 2D cases provide a blueprint: CFG can be understood as leveraging the geom- etry of overlaps in the support of the data distribution. In practice, this means that CFG tends to emphasize "private" regions of the target class and suppress ambiguous states, while accelerating convergence in way that becomes sharper as w increases. Extending these insights to formal re- sults in higher dimensions remains an open problem, but our analysis offers a clear starting point for characterizing how guided discrete diffusion behaves in general settings.
4 NUMERICAL EXAMPLES
We now present numerical experiments that illustrate our theoretical results and probe their validity in higher dimensions. Unless otherwise stated, we use a small transformer to train the score model, Tau-lcaping with 50 steps and log-linear schedule the numerical scheme and 10K samples per experiment. All experiments are run on an NVIDIA GeForce RTXTM 4070 Laptop GPU.
Experiments in ID. We examine two scenarios: when class supports are disjoint and when they overlap. From Figure 2-(a)(b), we can tell that: in the disjoint case, guidance has no effect. In contrast, when supports overlap. guidance redistributes mass away from the intersection region, in line with Proposition 3.1. Even with score and discretization errors, the sampled distribution closely approximates the tilted distribution. We also measure TV function of at fixed time & = .5 in Figure 2-(c). For small w, the empirical TV curve matches our theoretical predictions. For large w, we observe flat/increasing region in the plot. We conjecture that this is mainly due the sharp transition of the reverse sampling dynamics for large w (as shown in Remark 3.2), which makes the Tau-leaping scheme less efficient and less stable.
Diemibttion Companbex (1-33003 1etarze-3333
Dieribstie Comperiee (200) gilinoo-- 3.0)
Clas Dia dock
Guai Develutton
(a) No effect for disjoint support. (b) Mass-shift when supports overlap.
(c) TV as function of w.
Figure 2: The first two panels compare the generated distribution 97 2.8 (blue histograms), the tilted distribution pao (red), and the true target class distribution p(-|z) (green). In (a), when the supports are disjoint, all three distributions align closely, indicating that guidance has no effect. In (b), when supports overlap, 47 from p(-1z), reflecting how CFG suppresses mass overlapping regions and amplifies in class- unique regions. Pancl (c) plots the TVV distance as function of w (bluc), showing good agreement with the theoretical prediction from Proposition 3.2 (red) for small w.
2.w0 and p² remain close- cosssccnn with Proposition 3.t-but both deviate
Experiments in 2D. Figure compares theoretical predictions with empirical results in 2D. Each class consists of two diamond-shaped regions: one corner and one centered (see Appendix F.2 for density plots). For the class of interest, the corner diamond lies in the bottom-left, while for the competing class it lies in the top-right. We consider two setups depending on whether the central
8

---

Published as a conference paper at ICLR 2026
diamonds overlap. In the disjoint case, the classes do not intersect in the full space, but their pro- jections overlap along coordinate axes, causing guidance to suppress probability mass at the top and right corners of the central diamond. In the overlapping case, the effect is more pronounced: the upper-right region the central diamond, where supports coincide, strongly suppressed, while projected overlaps slightly reduce mass in the upper-left and bottom-right regions. In both cases, the empirical results align closely with our theoretical predictions, validating of our analysis in 2D.
(a) Disjoint: ratio of pre- (b) Disjoint: ratio of gen- (c) Overlap: ratio of pre- (d) Overlap: ratio of gen- dicted to class distribution. erated to class distribution. dicted to class distribution. erated to class distribution.
Ratios of the generated distribution qr to the class distribution p(-1z) in 2D. Red in- dicates regions where mass is amplified, blue indicates suppression. (a,c) theoretical predictions from Proposition 3.3; (b,d) empirical results using Tau-leaping. In the disjoint case (a,b), guidance suppresses top and right corners of the central diamond due to overlap in projected spaces. In the overlapping case (c,d), guidance strongly suppresses the upper-right part of the central diamond and more moderately reduces mass in the upper-left and bottom-right parts, consistent with theory.
Figure 3:
Higher-dimensional Experiments. To investigate whether our findings extend beyond low di- mensions, we conduct experiments in 5D using mixtures of uniform hypercubes, 1, 2, and (2,3,4)5 which overlap only at the single state 2, class supported on {2, 3, 4] under guidance strengths w = 1 and w = 2. The geometry naturally partitions the space by the number of coordinates equal 2, denoted by k = # {d Id 2). The case & = corresponds to states unique to the target class, & = 5 corresponds to the fully overlapping state, and intermediate values k E 2,, 4} represent partially overlapping regions.
We focus sampling from the
Tables and 2 report the number states each region along with the mean and standard deviation clear monotone trend: per-state
of their probabilities from empirical samples. In both cases mass is largest in the unique region (k- = 0), smallest in the fully overlapping region (k: = 5), and intermediate for partial overlaps. Importantly, partial-overlap regions are not uniformly suppressed; depending on k; they may either gain or lose mass relative to p(.
2, the redistribution becomes more pronounced with stronger guid- ance. Larger w shifts probability mass away from the fully overlapping and high-overlap regions, reallocating it toward unique states and those with smaller k. These results mirror the mechanisms observed in 2D, providing strong evidence that our theoretical insights extend to higher dimensions.
Comparing = and
Table 1: Sample statistics in regions defined by (d Fd
2) for
#(d: 2d = 2}
2
3
state frequency
80
80
40
10
density per state - average (1e-3)
4.984
4.541
4.020
3.386
2.208
0.280
density per state - std (Ie-3)
0.282
0.319
0.268
0.273
0.206
0.000
Table 2: Sample statistics in regions defined by # (d
2} for
(d va 2 state frequency
2
3
4
80
80
40
10
density per state average (1e-3)
5,573
4.783
3.867
2.844
1.588
0.200
density per state - std (1e-3)
0.346
0.312
0.268
0.204
0.171
0.000

---

Published as a conference paper at ICLR 2026
Additional Experiments. We further explore the effect of CFG in masked discrete generation MNIST. Full details of it, together with details of previous experiments, are provided Appendix F.
CONCLUSION, LIMITATION AND FUTURE WORK
In this paper, we developed rigorous framework for understanding classifier-free guidance (CFG) in masked discrete diffusion models. Through explicit analysis in and two dimensions, showed how guidance reshapes generated distributions: in 1D the guided process exactly recovers the tilted distribution, while in 2D systematic deviations emerge that be described in terms of class overlaps and marginal reweighting. These tractable cases shed light on what to expect in higher dimensions, where overlaps and geometry total variation distance along the reverse dynamics decays double-exponentially with the guidance strength w. While this implies strong contraction in the continuous-time limit, also explains why large to leads to stiff and unstable numerical simulation in practice.
complex. A key theoretical finding is that the
Limitation. Our analysis limited to low-dimensional settings (ID and 2D). Extending to higher dimensions is challenging for two reasons: exact formulas are unlikely to exist, as the reverse rate matrix becomes prohibitively complex, and even approximate descriptions that capture the qualita- tive role of guidance are difficult to derive due to intricate interactions between overlapping supports and marginal projections. These challenges restrict the scope of our current theoretical guarantees.
Future Work. A natural next step is to relax the idealized assumptions of exact concrete scores and perfect numerical integration. Important open questions include how guidance interacts with score approximation errors and discretization errors, whether it amplifies or mitigates them, and how such interactions affect convergence and sample quality. Progress on these fronts would deepen the theoretical foundations of guided discrete diffusion and improve its reliability in practice.
ACKNOWLEDGMENTS
YH, KR and MT are thankful for partial supports by NSF Grants DMS-1847802, DMS- DMS-2513699, DOE Grants NA0004261, SC0026274, and Richard Duke Fellowship.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 34:17981-17993 2021. (Cited pages and 15.)
Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 22669-22673 2023. (Cited on page 32.)
Arwen Bradley and Preetum Nakkiran. Classifier-free guidance is predictor-corrector. arXiv preprint arXiv:2408. 09000, 2024. (Cited on pages 1, 2, and 15.)
Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. Advances in Neural Information Processing Systems,
2022. (Cited on pages 2, 3, 15, 16.)
Hongrui Chen and Lexing Ying. Convergence analysis of discrete diffusion model: Exact imple- mentation through uniformization. arXiv preprint arXiv:2402.08095 2024. (Cited on page 3.)
Muthu Chidambaram, Khashayar Gatmiry, Sitan Chen, Holden Lee, and Jianfeng Lu. What does guidance do? fine-grained analysis simple setting. arXiv preprint urXiv:2409. 2024. (Cited on pages 1, and 15.)
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-877 -7944 2021. (Cited pages and 15.)
Daniel T Gillespie. A general method for numerically simulating the stochastic time evolution of coupled chemical reactions. Journal of computational physics, 22(4):403 434, 1976. (Cited page 3.)
Daniel T Gillespie. Approximate accelerated stochastic simulation of chemically reacting systems. The Journal of chemical physics, 1 11((4)1116-1733, 2001. (Cited on page 3.)
Winfried K Grassmann. Transient solutions in markovian queueing systems. Computers & Opera- fions Research, 4(1):47-53, 1977. (Cited on page 3.)
Nate Gruver, Samuel Stanton, Nathan Frey, Tim GJ Rudner, Isidro Hotzel, Julien Lafrance-Vanasse, Arvind Rajpal, Kyunghyun Cho, and Andrew G Wilson. Protein design with guided discrete diffusion. Advances in neural information processing systems, 16:12289-1 2023. (Cited on page 1.)
Wei Guo, Yuchen Zhu, Molei Tao, and Yongxin Chen. Plug-and-play controllable generation for discrete masked models. arXiv preprint 43X00:2410.02143, 2024. (Cited page 15.)
Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. Ssd-Im: Semi-autoregrrssiive simplex- based diffusion language model for text generation and modular control. arXiv preprint arXiv:2210. 17432, 2022. (Cited on page 15.)
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. (Cited on pages and 15.)
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020. (Cited on pages and 15.)
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633- 8646, 2022. (Cited on page 1.)
Han Huang, Leilei Sun, Bowen Du, and Weifeng Lv. Conditional diffusion based on discrete graph structures for molecular graph generation. Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 4302-4311, 2023. (Cited page
11

---

Published as a conference paper at ICLR 2026
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021. (Cited pages and 15.)
Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. Diffusion- Im improves controllable text generation. Advances neural information processing systems, 35: 4328-4343, 2022.
2022. (Cited pages I and 15.)
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310.16834 2023. (Cited on pages 1, 3, 15, and 16.)
Justin Lovelace, Varsha Kishore, Chao Wan, Eliot Shekhtman, and Kilian Q Weinberger. Latent dif- fusion for language generation. Advances in Neural Information Processing Systems, 36:56998- 57025, 2023, (Cited on page 15.)
Chenlin Meng, Kristy Choi, Jiaming Song, and Stefano Ermon. Concrete score matching: General- ized score matching for discrete data. Advances in Neural Information Processing Systems, 35: 34532-34545, 2022. (Cited on page 3.)
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text- text-guided diffusion models. In International Conference Machine Learning, pp- 16784-16804. PMLR, 2022. (Cited on page 1.)
Shen Nie. Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, JUN ZHOU, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. In ICLR 2025 Workshop on Deep Generative Model Machine Learning: Theory, Principle and Efficacy. (Cited on page 1.)
Hunter Nisonoff, Junhao Xiong, Stephan Allenspach, and Jennifer Listgarten. Unlocking guidance for discrete state-space diffusion and flow models. arXiv preprint arXiv:2406.01572, 2024. (Cited on pages and 15.))
Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. arXiv preprint 24Xx::2406.037366 2024. (Cited on pages 2, 3, and 15.)
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- tion processing systems, 35:36479-36444 -6494, 2022. (Cited on page 1.)
Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 17:130336-13184. 2024. (Cited on pages and 3.)
Yair Schiff, Subham Sekhar Sahoo, Hao Phung. Guanghan Wang, Sam Boshar, Hugo Dalla-torre, Bernardo P de Almeida, Alexander Rush, Thomas Pierrot, and Volodymyr Kuleshov. Simple guidance mechanisms for discrete diffusion models. arXiv preprint 2rXiv:2412.10193 2024. (Cited on pages and 15.)
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K Titsias. Simplified and gener- alized masked diffusion for discrete data. NeurIPS, 2024. (Cited on pages 2, 3, and 15.)
Yang Song. Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- lional Conference on Learning Representations, 2021. (Cited on pages and 15.)
Hannes Stark, Bowen Jing, Chenyu Wang, Gabriele Corso, Bonnie Berger, Regina Barzilay, and Tommi Jaakkola. Dirichlet flow matching with applications to dna sequence design. arXiv preprint arXiv:2402.0584/ 2024. (Cited on page 155.)
12

---

Published as a conference paper at ICLR 2026
Yuchen Wu, Minshuo Chen, Zihao Li, Mengdi Wang, and Yuting Wei. Theoretical insights for diffusion guidance: case study for gaussian mixture models. arXiv preprint arXiv:2403.01639, 2024. (Cited pages I, 2. and 15.)
Junhao Xiong, Hunter Nisonoff, Ishan Gaur, and Jennifer Listgarten. Guide your favorite protein sequence generative model. arXiv preprint arXiv:2505.04825 2025. (Cited on page 1.)
13

---

Published as a conference paper at ICLR 2026
CONTENTS
Introduction
Preliminaries
2.1
Notations
2.2 Discrete Diffusion Models
2.3 Discrete Diffusion Models with CFG
3 Analysis of Masked Diffusion with CFG in Low Dimensions
3.1 D 1: single-token generation
3.2 D 2: multiple-token generation
3.3 Discussion: Towards higher dimensions
Numerical Examples
5
Conclusion, Limitation and Future Work
10
A Additional Related Work
15
Analysis of Masked Discrete Diffusion Models without Guidance
15
B.1 Density evolution along the forward process
16
B.2 Density evolution along the reverse process
16
Proofs Properties of Masked Discrete Diffusion Models without Guidance
17
D Properties of Masked Discrete Diffusion Models with CFG when D 1
18
Properties of Masked Discrete Diffusion Models with CFG when D = 2
19
E,1 Sampled distributions for D =
24
E,2 Convergence rates for D =
26
Numerical Experiments
28
F.1
Details on 1D experiment
28
F.2 Details on 2D experiment
29
F3 Experiments in 5D
29
F.4 Experiments MNIST
32
14

---

Published as a conference paper at ICLR 2026
A ADDITIONAL RELATED WORK
Diffusion Models in Continuous and Discrete Spaces. Diffusion models were first developed in continuous domains, where Gaussian noise is gradually added and then removed through a learned reverse process (Ho et al., 2020; Song et al., 2021). They have achieved remarkable success for images (Dhariwal & Nichol, 2021) and audio (Kong et al., 2021), but are less natural for inherently discrete data such as text categorical variables. To address this, discrete diffusion models have been proposed, including D3PMs (Austin et al., 2021) and masked token diffusion models (Camp- bell al., 2022; Shi et al., 2024; Ou et al., 2024), which corrupt data via masking or categorical transitions. Compared with continuous models, discrete diffusion introduces new challenges: for- ward marginals are typically
Gaussaasiinn and analytically intractable, while score functions must be redefined, often as ratios of logits (Campbell et al., 2022; Lou et al., 2023). Masked diffusion al- leviates some of these issues. since its corruption process admits closed-form marginals and enables tractable likelihood training (Shi et al., 2024; Ou et al., 2024), making it both practical and amenable to analysis.
Guidance in Continuous Diffusion Models. Guidance techniques are central to controllable gen- eration in continuous diffusion. Classifier guidance (Dhariwal & Nichol, 2021) steers the reverse dynamics using gradients from pretrained classifier, but requires large and accurate auxiliary mod- els. Classifier-free guidance (CFG) (Ho & Salimans, 2021) eliminates this dependence by jointly training unconditional and conditional models, allowing interpolation between guided and unguided generations through a scaling parameter. Recent theoretical work has begun to analyze these mech- anisms. Assuming Gaussian structure, Bradley & Nakkiran (2024) showed that the probability flow ODE under guidance does not recover the tilted distribution, and is equivalent to a predic- tor-corrector scheme. Wu et al. (2024) studied Gaussian mixtures, proving that increasing guidance reduces differential entropy and sharpens class confidence. Chidambaram et al. (2024) analyzed guided ODE dynamics in ID mixtures, showing that guidance exploits latent geometric informa- tion even when absent from the classifier. While insightful, these results remain restricted to highly simplified settings.
Guidance in Discrete Diffusion Models. For discrete data, several approaches apply guidance indirectly through continuous embeddings (Li et al., 2022; Han et al., 2022; Lovelace et al., 2023; Stark et al., 2024; Guo et al., 2024). More recently, direct formulations have emerged: Nisonoff al. (2024) proposed modifying reverse transition rate matrices, while Schiff et al. (2024) guided the reverse kernels. These definitions need not coincide, and systematic comparisons remain open. Our work focuses on the framework of Nisonoff al. (2024).
To our knowledge, this is the first theoretical study of guidance in discrete diffusion models. Closely related to Chidambaram et al. (2024) in spirit, we analyze oow-dimessional sampling dynamics where explicit solutions are available. Our results show that in ID, guided sampling exactly covers the tilted distribution, while in 2D discrepancies arise that can be characterized analytically. Moreover, prove that increasing guidance suppresses probability in overlapping regions and am- plifies unique regions, reducing diversity as guidance grows. The total variation distance along the reverse process decays double-exponennially in the guidance parameter. highlighting both the sharp concentration effects and potential numerical instabilities at large guidance values. These findings parallel-but also didenddobservationn in continuous settings (Bradley & Nakkiran, 2024; Wu et al., 2024; Chidambaram et al., 2024), providing new insight into the discrete domain.
B ANALYSIS OF MASKED DISCRETE DIFFUSION MODELS WITHOUT GUIDANCE
This section analyzes the behavior of masked discrete diffusion in the absence of guidance. By quantifying the density evolution of the sampling process, we establish baseline understanding of the unguided dynamics. These results provide essential groundwork for the theoretical analysis of discrete diffusion with CFG in the Section 3.
15

---

Published as a conference paper at ICLR 2026
B.1 DENSITY EVOLUTION ALONG THE FORWARD PROCESS
The forward process in the masked discrete diffusion process gradually absorbs all the mass to the masked state N. In practice (Campbell al., 2022; Lou al., 2023), the forward transition rate matrix is parametrized by Qi = (t) Et= IN @ Q @ IN) where
(16)
NxN
For simplicity, we consider o(t) = in the paper. According equation 16, we express the densities along the forward process in the following proposition whose proof is deferred to Appendix C.
Proposition B.I. Let lr be the solution to d de/l Qafte with initial distribution poo l and Q given above. Then
e-
0
0
0
Pr
A p.
(17)
0
e-t
1 ---
NxN
As a consequence, for any x E S with UM UM(x) = 1i::; < N}
p+(x) e -IMM ------IM Σ Mly).
PAUM=RUUM
B.2 DENSITY EVOLUTION ALONG THE REVERSE PROCESS
For any distribution on S, the forward process equation initiated at induces reverse process, whose transition rate matrix denoted as Qe(e]. explicit expression of Q+[n] can be derived from the property in equation 3 and the forward densities in Proposition B.I.
Proposition B.2. The sequence of reverse transition rate matrices associated with μ (initialization of the forward process) satisfies that for all < t T,
POMUMMSUM a(u)
24 + = V/. Di. 9/
I-e- EUNCM=rUM k(w))
Q111)(,,2)) =
E EveN(o) Qolu)(u,2),
y= z, otherwise.
0,
With the above expression of the reverse transition rate matrix, in the low-dimensional setting of D 1, we derive the density formulas along the reverse sampling dynamics the following theo- with the proof deferred to Appendix C.
Theorem B.1. In the discrete diffusion models on S, if and qu satisfies the sampling dynamics de Qr-ilpee qt with initial condition qo = ÖN, we have that for all 0 S T,
d
((--) p(xx,
I = 1,2, ...
1,
(x(x)) =
(18)
T-N N.
Remark B.1 (No initialization error), Unlike other diffusion processes, the absorbing discrete dif- fusion does not induce any initialization error. Even though we approximate the initialization in equation 2 by the point mass at the masked state, the sampled distribution recovers the data distri- bution, i.e., qr = p. Same property also holds for masked discrete diffusion with CFG, as shown in Section 3.
16

---

Published as a conference paper at ICLR 2026
C PROOFS PROPERTIES OF MASKED DISCRETE DIFFUSION MODELS WITHOUT GUIDANCE
Lemma C.1 (Diagonalization of Q). Q = XAX-I with A = Diag( -1 -,, - -1, 1, and
-1
1
1
0
0
0
0
0
0
x
1001
11
11
NxN
Proof of Proposition B.1. The solution equation 1 with initial distribution p40 p can be ex- pressed as
14 exp=(()))) exp (1
IN
...
IN)N
dth
D
πI exp (IN 6 d=1
Q IN)
D II exp [[XXX d=1
(.. 0
A
dth
D
III (X II d=1
... expp (IN @ ... LA ... -
(X @ @ xx exp(tt))))))) RD ... (x........-1
(X exp(tA)X @D n.
where the second identity uses the fact that (IN X Q ... @ IN)a commute with each other. dsh
Then the statement follows from Lemma C.I.
Proof of Theorem B. I. With the expression of the distribution along the forward process, we can write the reverse transition rate matrix based on Proposition B.2. We have
p(1)
p(2)
Q e
c
(19)
1-e
1 e
p(N- - 1) -1
NxN
The eigenvalues and eigenvectors of Q are given by
o = A _.. ĀV- = O, Av -11
p(1)
p(2)
U2 UN
.uN
p(N 1) -1
17

---

Published as a conference paper at ICLR 2026
The eigenvalue decomposition of Q is given by Q XDX-1 with D diag(0, 0, .0,,0.1111111
R N x N
1 0
0
p(1)
0 1
p(2)
X x-1
0
0
p(N - 1)
-1
NxN
simple computation tells that
exp ( 1 Qd- exp
T Tt
ff.
e
ds()) X exp (In(
e -
e- 1D
e
0 ...
0
(1 - } p(1)
1
0
(1
ميا ((2)
:
",
0
0
1
(1-
1-e
INN - 1)
I-e
0
0
1- l-c
NxN
Along the reverse sampling dynamics, we have ae = exp (So Qr- (oddssgoo qo. which implies
(1(2)) + (1 11--(--)
I-- T-e-r )p())o(NN) I-e-r 40 (N).
I 1,2, N I N.
qe (a) Last, the thcorem follows from plugging in qu = 6N.
(20)
D PROPERTIES OF MASKED DISCRETE DIFFUSION MODELS WITH CFG
WHEN D = 1
Proof of Theorem 3.1. Notice that the reverse transition rate matrix Q7 = z". Qu(pi/) == = zQ, Following the same computation in the proof of Theorem B.I, we have
exp (
T-t 2Qr ds) exp z
/
T
ds()) X exx (3 - e-
1 - )D)
0
(1 l-c 12 (1)
1-e-r
0
(1
1- r
}p (2)
:
0
0
(1
(+-e)) 2 (N 1)
0
Along the reverse sampling dynamics equation 10, we have q." 2,10 implies
exp (Jo - 2010 which
10 (x) + (1 (--1
pm: p (N),
x 1,2, ... x N.
1, (21)
9i
(x)
(-- Z 2. (N),
40
Last, the theorem follows from plugging in T 2. 8N-
Proof of Proposition 3.1. According to Theorem 3.1, in both cases, the sampled distribution is the same as the tilted distribution, i.e., 47 ,0.00 w_
In case (1), it is obvious that - p(-121).
18

---

Published as a conference paper at ICLR 2026
In case (2), have p²1'"(()) (x) )
) p(x(22)). Under Assumption 1.1, we have
p(ar)
p(z)e1) arp(alas)
* E Xi Si x St, otherwise.
p(2 p(a)
p(al)))
Σx (
Then Proposition 3.1-(2) is proved.
Proof of Proposition 3.2. The result directly follows from Theorem 3.1 and the formula TV p2) = 2 En Im (x) - (2(2))
Definition D.1 (Local mean and covariance). For any probability distribution μ S and any subset C S, the local mean and local covariance of is on A are defined respectively as
ma(p)= Σ xuA(z), Ea(u)= (el-ma())==-aa
where u.a (a(x) = plxx// EyeAAly) is the restriction of μ on A.
Lemma D.1. Under the assumptions in Proposition 3.1, for all w 0. Ex 1x, (17
Exyss, (s (p(-1z1)).
Proof of Lemma D.1. According to Proposition 3.1,
MX, (97 21.0)
Σ
zp(l|a))/
Σ
p(yl21) = mxiss i (p(-22)))
NE
and
Ex E5 (07 (1
Σ
(x mxi (47 (0 )) mxis p(xx(2))
Σ p(y2)))
DEAi \S+
Σ
(x - ( mxns. (p(2a)
Σ
p(yl21)
=
BEAr
Ex (p(+)21)).
E PROPERTIES OF MASKED DISCRETE DIFFUSION MODELS WITH CFG WHEN D =
Proposition E.1. When D = 2, denole = = = Exes p(x))=((xl)))++ Then the guided
reverse transition rate matrix is given by Q7 ------ Qu:00 s.l.,
(v)
34 = * N, N, = N # D
plut "Pull
p(y)) "pp)) p(g. (2) 1410
E; N # Uss Vi = M4 N
Q (y,
2p"."(vi (ve)
F N, N, = -, -N
p(u. " ++
Et.1 2m p(u1) "(((l)))(
x = y = (N, N)
otherwise.
Proof of Proposition E.I. For any x, E with x, Vi N and Di N # Mj according to equation 9, we have
Pc(y)
Pe (c(2)) 1+w
Qv."(y,)) = (u) - (v) +t
(x)
Pr. (z)22)
21 p(y)
2 p
= - e (14)))
1+
e e (p)(ppp(
19

---

Published as a conference paper at ICLR 2026
ply) p(y)-=p(yll2) "((y)))++>
1 e p(u)) p(36 12)
2 zz
e- p(u:) p(ya 1
where the third identity follows from Proposition B.I, and the last identity follows from the def- inition of p²t Next, following the same approach, for any 2. y S with 2; N # yi and Tjj = Uj = N, we have
Qf (v, = Px(y)
(c(()2) 1+
(+)
Pr (x(2))
(1 - e pl(u) -w e e - ppl1-e (((ll))) wt
(1 1-
t)²
PP(i)) " ((us|
Last, the other cases for different (y, x) follows from the definition of the transition rate matrix,0 equation 3 and equation 8.
Proof of Theorem 3.2. Our proof follows from the following steps.
Step 1: represent the reverse transition rate matrix blockwisely. The matrix Q in Proposition E. can be represented blockwisely as
R
1
2
0
0
M 8,10
Σ; 1,5
For all i = ...N 1.
p(i.1)
p(
1+w
E, P(a.y
P(E.(E) p(),2|=)
0 0
p(i.2)
R
En p(t.))
Σ, p(L)(=))
(22)
p(i.))
p(iji=)) Σ. p(U.E)
1+w
L.P(TT))
1.2.00
(23)
261.1 m)) PUS.TT
PLLN
(E,((,1) (E,((,, 1|=)) 1+ (E,((12) " (Eip(1, 2)=)) 1
M 3,10
(24)
- E, (Ei((h))) ( 1+w
where we used the definition of Z and marginal distributions.
Step 2: Eigenvalue decomposition for @ Since Q is upper triangular, its eigenvalues are diag-
onal entries. For all j,j = 1,2, N 1, define
pip(i,))~ p(, 42)
dy = E.P(.)) (.p(1.))"p(11.11) p11,51z)2*
= E ((i p(t, ((
(25)
(Er((,,)) " (( (Eip(1 (1
En,,, p(1,, "p(h
En, do (((1.12)-pp((11 (p(1,
cn En, (Ei, p(11,12)) (E,,, p(1, b(11,b)))) ++w dN Eis (En, 121) - "(E,, pl1,, 12=))
(26)
20

---

Published as a conference paper at ICLR 2026
2,20 JiJE[N]] can be represented as Then the set of eigenvalues for Q denoted as {1,,)
X 2,10
2,01 1,1 ....
I,N = 0, N,N = -i, == 1,2, ... N - 1,
X J. Njj
-dj, dj.
N,N 2(1/eN + 1/dx), 1=1,2, N
The associated eigenvectors to " denoted as ū7 = (i1, 11 ;ūN), satisfies
R ih + Ljj UN = Ut 1= 1,2, N -1
(NTEaw Σ f J Xi UN-
The eigenvectors can be studied two cases:
When S i N 1, we can pick Un 0. Then for 1 = ..,, 1, R A 8:80 Jg u.. For f i, we pick ū. = 0. For = i, ū7 is the eigenvector to 18 associated with
RT." ip
the eigenvalue x for j 1,2, ... N 1, we pick i;; ūūjj Eg. For j N, we pick i7 üi to be
p(i, 1) p(t, 1/2) E(p(i - "p(i, 1|2)
p(t, - 1 p(t, 1|=) -1)
(27)
-ip(i,1)~-p(e ([))
(2) When N. UN + 0. We need to solve (M=." - E1 [) üN Ai5 uN first. For different j, we pick UN = UNd with UNS éo for = 1, 1, and for = N, UN
Σ E(P(L.)) "(E, p(l, 1
EPP(N - 1)) "0)"((" - 11
-NNN di
-N.N dN-1
(28)
Next for each j 1,... .N. we solve (Ri XN -Lg. uN for all I
1,2, N 1. We get
p(l,j) 1
P(l' 5) -)))))g- p(l' :|zz) Cy j 11,,..,
ū.jj
(29)
(LI,N(1), Ji (N - 1), Un,N (N)) =j=
with
(1'((1) = 2.w 2.u AN NN
1p(1,11)~pp(( 12
+
NN.N + dy
(NN(N) CA ct) Σ "1,,))"(((,()))+++
Collect all the eigen information above, we diagonalize Q blockwisely: 2 Dz (x= s.t.
Q
D=.w Diag D
D N-1 11
" Dz Diag(A 2.
F.W) for each
JNN 1 (,N)
X 1
2,
X
0
X N
V
0
0
0
21

---

Published as a conference paper at ICLR 2026
X 2,
O
X 2,
X2 2 xpa
(x==0)-==
0
0
F N-1
Zz. 2 Zzz N N-1 N X N
O
where for each i 1,
.N 1,
; = Diag (0,
,00 -G),
x
El
EN-1
UN
(X:""")
'with with Ui defined in equation 27
4::00 -0
U1,1
N UN with {ūbr))== defined in equation 29,
and for i = N,
1 Diag( -ai -1, .., -1:
-2 22t/0/ON- - /dx),
X El
EN- (N.N) ( ÜN,N defined in equation 28.
N
Step 3: solve the equation equation 10 explicitly The solution to equation 10 can be computed using the formula -. exp So 07 ds) d "h where the matrix exp (66 Q7",, is computed using the eigenvalue decomposition in Step 2. More specifically,
1 f' ds)
-xp S/
(T-))
e-
exp(
e (T- ds x exp (In( 1 - c (T-t) ()
X-300
X
c -T
e T c (T (T- 02:000
(T- 1::00
Diag(
2:10
s,s N N1
N-11
X 5,00
X 2. s, S.
2,500
X y
N
0
N-1
N N-1
22.00
x2
M
(7-6)
P 8 8.
M
N-1
N
O
0
1-
B
X
N
1-e
(
For each i 112.....
(1--) jei)
I-c ---- x-
- c
e- (T-
0
(1
(1.N-
I-e-T
Σ, (1.)
1-c (T
0
I-c-7
M
1 - c
13:0 x.
"
Y 2,
1 -
D &
(T-
(T-
22

---

Published as a conference paper at ICLR 2026
(1- (- )4))
(111 2, (1.)
2pt
(1 I-e-r )dx=)) (i.N)
1 2p (1, N -1)
Br,N 2 E1 1¹ (1.
, where for each j.j = 1,2, 1,,,,,,, .N- - 1,
(T-1)
1 - c-I 1- -
E (T--))
Big = Ci(ANN 2.
)a)
)d.
+ (2)
d (ANN s.W
NN + ddi)
1
1 e (T-1)
(1 (
(30)
ANN +
m:w + dj
e r
(1---
) - e (1'
B = G(ANN NN +G)
-XNx)
(31)
1 - e-
- c-T
For = N, we have
1 TT
XN"
1 8z.
6- [T-
1-0 (3-1) ydl
0
2 (1
(1 (2--11--)dds yax
BN.N- 2 Eiprr L-e CT ANN
(1, N - 1)
1-e-r
0
T-o-!
where for each j = 1,,, ,N-1, 1,
(T--) jdl
- (T)
BN. dA + dg)
-12W).
(32)
e-T
Now, we can apply the initial condition -0x0 NNN to compute q." T."
S/²
q: 2.
=exp(
ds) %
1-0
69 x
M
0
:
0
N-1
D x
N-1
M 1
0
1-
EN
(7--)
M
2,000 N1
2
11, N;
(* 1-c-7 (1)-
D2
X 2, (c.N)
Therefore, for all i 1,2... N 1 1.
w.w (i,5) = -M (J,N) = Be, (i (t,,).
4:
For = N = 1,2,
N 1,
qi (i, N) = M,, (N, N) = BI,N Σ (1,
For i = N.j = 1,2, ... N - 1,
ai = (N. BN Σ pp..(11.))
Last, for i= jj = N,
(N, 1 - - [1]
1 c -xaN
Last, Theorem 3.2 follows from the following definition of aq == Brnzz for all x E (1,2. ... .N
n
23

---

Published as a conference paper at ICLR 2026
E.1 SAMPLED DISTRIBUTIONS FOR D =
Proof of Proposition 3.3. According to equation 15 and Assumption 1.1, we have
21.0 (x) & (1/e2, + /el///////////////// (/dd)))(x))"ppp))p
(1p(x1)z1) Σx (xp(((i
a1p(2g)21) En axp(22lZZn) )m
a a1p(a)/2) Ee axp(arzz)) p(x[aa)).
x
m
Each of the terms II and III is within the range [0, 1] and exponentially dependent to Therefore, the values of II and III affect the sampled distribution significantly when w is large. By eval- uating II and III in different regions depending relations between the marginal supports, express 21 as presented in Proposition 3.3. The last statement in Proposition 3,3 follows from the discussion on (-(2))))1)) in this section.
Effect of guidance on sampled distributions. According to Proposition 3.3-(2), "7. is defined with different weight-adjustment in 5 different type of regions defined as follows
R1 = (x/r E X E N1,1 \ S1,1, E X1,2 \ (1.2),
R2, {r|x 24 Si E &i \ (Syh
i= 1,2,
R3 {z|x X; S1.21 S1,1 S1,2).
R4 = St.
The above sets reflect different level of "privacy" of class 21- R4 is the shared region with other classes. R1, R2 R2 are not shared with other classes. But R2 has both marginals shared with other classes and R2, has one of the marginals shared with other classes. R2 is the most private set in class 21, with no intersection with other classes even for marginals. The associated weights (before normalization) on different regions are give by:
A 2,
a1p(02(21)
F
)o: i= 1,2,
Σ kEL, axp(x:(22)
a1p(21/21) 21)
A $
EkEni (np(1r/22))
a1p(z;121)
a1p(1(z1)
A F1.00
)mm)
ExEn, anp(Ze(Zz)
Eker, axp(x:
we can notice that for all w O, AFT." A A $1." > A This reflects that the sampled distribution from the discrete diffusion with CFG can leverage the geometric information of the full data distribution: the sampled distribution puts larger weights on more private regions of class 21. We conjecture that the above fact is also true in high dimension:
Conjecture E.1. For any D 2 2, discrete diffusion with CFG leverages the geometric information from the full data distribution. More specifically, under Assumption 1.1, the sampled distribution 97 21.00 adapts the class distribution p(-121) by putting larger weights on more private regions of class 21, where those regions with different privacy are defined based on the support sets and their marginals.
Discussion on -z (1121). Now we look at the structure of the sampled distribution as w -> oo in further detail. According the expression of weights _ since 1 T1, for i 1,2 and 1 E 41, the rational factors inside the parentheses is in (0, 1]. In particular, if S] * 0, i.c., class E (0, 1). Therefore. as
arp(ala)) has intersected domain with other classes, I 2. Hence Enen (сP(2/сK]
0o, we have
A 2,
A E (1,2), A5 E [0,1,2] A = 0.
24

---

Published as a conference paper at ICLR 2026
Then, have (-12)))) i.e. (10p(1")00((111)) C Ai
It is worth noting that it is possible that some sets among R1, R2 R3 could be empty. Therefore, for general data distribution satisfying Assumption 1.1, in order to derive qar ( (-|21) need to first identity whether R1, R2,;, R3 non-empty not, and then compute the associated limiting weights on the non-empty regions. In the following, we will use a simple example to illustrate this procedure.
completely.
An example with D = N= 5. We consider the data distribution is mixture two classes with equal weights: p(x) )1(0|z1) + =2(((22) for all E {1, 3, 4, 4,5) with being the masked state. The heat maps for p(11 21 2, 22) and are given in Figure We can distinguish the regions
SC300
02001
0-000
6:000
54400
00000
0.0000
Figure heat maps for p(-121), p-112) and p.
with different level of privacy based on our formulas. As shown in Figure 5, we notice that R3 = and R1, R2,1, R2 R4 are identified with different colors. Based on the information of p, we can
Regions (R., Pal
8:
Ru
Dimersonz
3 Dimencion
Figure 5; identification of different regions.
compute the limiting weights as - oc. We have 1 = 2, 1..000 1z,=1,
21,000 129=1,
A7 0.
Therefore, the sampled distribution of regions respectively, i.e.,
20.00 (-121)
adapts p(- (()) by putting these weights on the
2p(x/21),
R1 ((1,1)), x E R2,1 = {(2, 1), (3, 1))},
p(x()1);
O (x|)=) x
p(z(21),
x E R2,2 =
0,
otherwise.
25

---

Published as a conference paper at ICLR 2026
96 o7 97 (2, 1(21) = 3/17 and 97 071.00 (21, 22 (1) = otherwise. In Figure 6, we present the
which implies that of 0 (1,1
4r
1000000001100000000101111100 (1.3/21)
- (3, 21) 4/17. 9T 22 (1, (1,2/21)
heatmaps for the class distribution of 21, the tilted distributions and the sampled distributions with w = 1,5, 15. We can observe the following facts that match our theory.
(1) the sampled distribution deviates from the tilted distribution for all w > 0.
(2) the effects of guidance differ in different regions: w increases, the probability mass decreases in S = {(2 2), (2, 3), (3, 2), (3. 3)3; the probability mass increases in regions R2,1 {(2, 1). (3. } and R2.2 {(1, 2), (1,3)) at the same rate; the probability mass increases in the region R1 = {1,1} } at the largest rate.
(3) for large guidance (w 15), the sampled distribution 47 "7t can be approximately under- stood as 97 22.000 (-]z)). The last plot in Figure 6 matches our computation for qf (-121).
(4) for small guidance (w = 1), the effect of guidance is also small. The sampled distribution 9T deviates a little bit from the target distribution p(-|21) in the way we described in (2).
In practice people observe that the optimal guidance is usually positive but small (of order (1))-
Our theory and numerical observations bring insights in understanding the optimal guidance. Roughly speaking, if we can show that the effects of guidance presented above actually compen- sate the effect of score approximation, by quantifying the inductive bias in learning the scores, we can rigorously analyze the optimal guidance in the CFG setting. This will be left as an interesting future work to explore.
E.2 CONVERGENCE RATES FOR D = 2
Proof of Proposition 3.4. For simplicity, we denote À N,N- According Theorem 3.2 and Remark 3.3, the total variation distance can be computed as
TV
9T
(x)
4T t0 (x)
Σ
[xr(x)) - ar(=))Zp*'" (lar(N, (1a -Ja(NN.))ar(NN.)))
2
(*N
-() Σ p (('"(() (x) (1 Cog + co - + 1 -((1)=))
1 + [r(0)-111
Z
2
dag + A deg
p (x1))
1
(r(t)(z1- -1(1)-
2
Cr (A coz)
+
Σ p (:7"((x))
1 (r(1) r(1)) + z (1)
2
dag (A + dag))
N2
= =1+1++1I+IV,
where r(1) (T-1
I-c T (0, 1). Next, we bound each term respectively.
For I, we bound the two terms inside using the following properties of function hn y [1. 00) + y r(t):: (v) < O and h"()) > for all y. Notice that q, d, 2 1 and -X = = exp(wDD+ (p: (-12)lpr()) + pz))))) 2 2, where we use μa; represent the 8th marginal of μ. Therefore, exp(u ww D
2 = Ei, p(()) "<p((()) + + Ei, p(12)
(((ll=)))+ (((2))
dx
have
( r(t)==1 + x ,((0)>)) M 1(-1)
-((CZ,) = r(t)
In (t))
Ca + Czi
Cz - (-2)
ct
Ct
1
r(t)d=2 + x (t)) ->)] = h(deo) h(- 11-1)11)))) - (d',))
r(t) dig
Inr(t)),
dop + x drz
de, (-
d
d
where ox, is between Czy and - X, dz is between drg and X.
26

---

Published as a conference paper at ICLR 2026
Heat maps with guidence
5200
M-1
5300
KMV
03200
0000
00443
S0212
00000
55.00
LIDKK
01269
00154
0.20
50K0
0009
03823
0.004
00000
Hest with guidance
C22
5UO0K
C0000
SOKO
C200
SCRG
CARUE
34000
51204
03763
1100
60000
00127
61951
5000
0009
$12-3
0393
5134
0000
with guidance
0,300
5130
0.003
5U00E
60000
02221
CODK
SONO
C200
60333
SCD0
00000
EIRM
0150
60000
3.09
00335
1000
0000
22353
03719
51315
03000
5000
8C25
Figure 6: distributions under different guidance strengths: w = 1,5, 15. The first column presents the class distribution of 21. The second column presents the tilted distributions. The third column presents the sampled distributions which are obtained using exact evaluations of scores and integrals.
For II and III, we bound the two terms using the property of the function h2 V E [1) H r(t)²: hs(w) < O and (y())) > O for all y. Again, due to the fact that cz di 2 for all and - 2 2, we have
1
h ha(Co,) h2(-1)
1 r()(sh Inr(t)
((1)==1-((1)) r(t)->)|
Hg(((,)
Can (A + Cz.)
Crp Cz (-X)
Oz1
(r(1)drs - -r(----(10))) -(1)~\)1= d
h2(dmz) - 122 h2(-1)
1
((1) In ln)(1),
dry () + dzzg)
drg - (-A)
dag
daz
where c/ is between Cz, and -X, d',, is between dr, and -X.
Last, according to the expression of c;, di in equation 25, we have
Σ
p(222 l' 11 = (,2) w+1 p(12 = 1'|x1 1) = exp
W 11))))001111,,))))( = =1))
=
p(0² = 11|21 = 1)
d. = Σ
(((1 1l = 1,2) wut
p(11 = 1'| = 1) = exp
WDi+o00((112 = 2))p((122 = m)
p = 12a2 =1)
27

---

Published as a conference paper at ICLR 2026
For > 1, since cz2 cz are between between Cz2 and -X and In cz₁ (ww), = (ww), we have Er₁ = ((w), Eyy = ((w) for all 21. For the same reason, di = (w)), d' = (w)) for all 22. Therefore, if we focus the order of w for >> 1 and preserve the leading order terms in TV ". .Tr"
". ), we have
TV -IT" Z exp( =xpp( + r(t)oxp(((ue))
exp(O(w)) exp( ( r(1)oxp(((w))
where the second identity follows from Remark 3.2.
F NUMERICAL EXPERIMENTS
F.1 DETAILS ON 1D EXPERIMENT
We consider each cluster to be defined by the following density vector:
0.4,
The full distribution contains two classes with equal weights, each class containing two of the clus- ters above.
Disjoint Example: we plot the class distributions and full probability distribution for the disjoint example in Figure 7.
Dusc Constion
Dios Bust Coepaies
Disildiio Corgrize
(a) Histogram for class
(b) Histogram for class 2
(c) Histogram for full distribution
Figure 7: Histograms corresponding to the disjoint example.
Overlapping Example: we pull the classes together to create overlapping region. We plot the class conditional and full probability distributions for the overlapping example Figure 8.
Paoking
(a) Histogram for class
(b) Histogram for class
(c) Histogram for full distribution
Figure 8: Histograms corresponding the overlapping example.
28

---

Published as a conference paper at ICLR 2026
F.2 DETAILS ON 2D EXPERIMENT
Disjoint Example: we plot the class distributions and full probability distribution for the disjoint example Figure 9.
5.
(a) Heat plot for class (target)
(h) Heat plot for class 2
(c) Heat plot for full distrihution
Figure 9: Heat plot corresponding to the disjoint example.
Overlapping Example: we pull the classes together to create intersection between the central di- amond regions. We plot the class distributions and full probability distribution for the overlapping example in Figure 10.
00
3G
(a) Hcat plot for class (target)
(b) Heat plot for class 2
(c) Heat plot for full distribution
Figure 10: Heat plot corresponding to the overlapping example.
Total variation function of ws in Figure 17, plot the total variation between intermediate generated distribution ge" 2:00 along the reverse dynamics at = .5 and the final generated distribution 47' 2.t function of w. We did not observe exponential-decay when is large, which does not match results in Proposition 3.4. As shown in Theorem 3.2, when w gets larger, the reserve dynamics admits sharper transitions, making the numerical method less efficient. Therefore, conjecture that this mismatch comes from the propagation of the numerical error along the reverse process, and halfway through the simulation, the numerical error dominates the -V-dymamess.
F.3 EXPERIMENTS IN 5D
Marginals of distributions: our full distribution is mixtures of uniform hypercubes, {0, 1, 2, and (2,3, 4].,, which overlap only at the single state (2, 2, and the target class is the one supported on {2, 3, 4].. The 2-dimensional marginals of the full distribution, class 0 and class (target class) are plotted in Figures I1, 12 and 13. As introduced in Section 4, the geometry naturally partitions the space by the number of coordinates equal
2, denoted by i = # (d xa 2). The case & = 0 corresponds to states unique to the target class, & 5 corresponds to the fully overlapping state, and intermediate values k E {1, 2, 3, 4) represent partially overlapping regions.
we generate 10K samples, compute samples statistics (in Tables and 2) and plot target class density ratios from the marginal of the generated distribution 2."" to the marginal
Experiment setting:
29

---

Published as a conference paper at ICLR 2026
of the target class distribution p(-12) with guidance w 1,2, in Figures 14, 15 and 16 respec- tively. Our results show the generated 2-dimensional marginals also exhibit a similar structure the generated distribution in 2D, that unique region has the largest per-state mass and the fully overlap- ping region has the smallest per-state mass. As w increases, mass is shifted away from regions that unique to the target class (small k). These
more ambiguous (large k) regions that numerical results support our discussion in Section 3.3.
Marginals for full data distribution
0.025
5,505
0.036
5.235
0.026
0.254
4
0.016
0.035
2.035
0.018
5:155
0.006
J
0.056
0.10
Dimrension
Dimenzion
Dimension
Denaty
3.068
0.036
0.222
5.000
0.036
0.026
0.036
0.332
0.056
0.254
5.000
0.331
0.055
010
0.05
5.555
0.006
0.025
1-
0.026
0.034
5.035
+
0.094
2.535
0.066
0.00
Figure 11: Marginals for the full data distribution
Marginals for class 0
Dimrension
Dimension
Dimension
0.10
Denaty
0.11
0.222
0111
0.111 0.322
0.311
2:222 0:31
0.05
5:11 1
0.111
0.111
0.11 1-
0.111
5.111
0.111
3.111
0.111
0.00
Figure 12: Marginals for class-0
Marginals for class 1
0:11
5.11
0.111
5:11
0111
0.11
4-
0.11
0.111 2.112
8:11
0111
]
0.11
0.10
Dimensionn
Dimenzion
Dimension
Densty
0111
2.11
0.11
ba11
0111
0.11
0131
0.11
5.113
0.005
1
1-
0.00
Figure 13: Marginals for class-1
30

---

Published as a conference paper at ICLR 2026
Probability ratios for class 1,
0.957
1.326
1.090
1.003
Dimwnsion
Dimention
Dimension
0.720
2.97
0.910
2-
0.764
0.31
5.32
0.760
0.77
0.205
-03
1
0
3
1
Dimensioo
Dimension
Dimension
Figure 14: Probability ratios from the marginal of the generated distribution of the target class distribution p(-(2) with w = 1.
the marginal of
Probability ratios for class 1, w 2
0.924
1.300
1.155
1.358
13163
Dimersion
0:936
1.338
1.148
Dimension
3
0.945
1.44
3.343
Dimention
0.592
1.153
1.147
-1.0
Denstt
2
0-
3
Dimension
Dimension
Dimension 0
Figure 15: Probability ratios from the marginal of the generated distribution 47 to the marginal of target class distribution p(-|z) with w
2.
Probability ratios for class 1, w
0.36
1.194
1.159
4
9.903
1
1.206
0.913
1.144
1.185
Dimention
Dimendon22
Dimenson
-10
Dontbyy
0:610
020
0.002
0.0
0
Dimmension
Dimersion
Dimension 0
Figure 16: Probability ratios from the marginal of the generated distribution oi. to the marginal of the target class distribution p(-|z) with w 4.
31

---

Published as a conference paper at ICLR 2026
F.4 EXPERIMENTS ON MNIST
We demonstrate that our findings apply in high dimensional problems and practical settings. We trained U-VIT network (Bao al., 2023) for 100K iterations using the Adam optimizer with le 4 learning rate. The hyperparameters for the network can be found in Figure 18.
Total Variation vs Guidance Weight (1-0.50)
Parameter
Value
- Degiict Tera Vuliis
1.37
img size
28
103 Weraten Uittnce
1,30
in chans
1
01:00
patch_sin
2
0.31
embed _di
512
0.80
depth
12
0.30
num heads
8
mlp-ratio
4
0.75
gkv bias
False
0.20
2.3
58
mlp time embed
False
Guidbooe Weight
labels_dim
11
Figure 17: TV ".") with = 55.
function of
Figure 18: Model Configuration
We illustrate how guidance suppresses regions of intersection between classes through two case studies. In the first, we generate samples of digit 8 under three conditions: without guidance. with guidance, and with guidance using digit as the conditioning class. As shown in Figure 19, guidance progressively removes ambiguous samples of 8 that resemble digit 3, with the effect being especially pronounced when digit used as the guiding distribution. similar phenomenon observed when generating digit 7 conditioned on digit (Figure 20). Together, these results provide empirical evidence that the suppression of overlapping regions-preced dimensions extends to high-dimensional, practical settings such as image generation.
by our theoretical analysis in low
3
8
2
0 Ø
7
P
8
8
8
8
2
4
7
3
3
8 8 8 3
8
8
8
3 8 8
8 8 8 8
8
8
8
3
3
8
8
3
3
3
3
Z
8
8
3
3
8
8
8
8
8
8
8
8
8
8
8
8
8
8
8
S
8
g
7
8
2
8
3
8
8
8
(a) Generating samples 8 with no guidance
(b) Generating samples of us-
(c) Generating samples of 8 but using the class of number as the guiding distribution using
ing guidance with w =
Figure 19: Applying guidance reduces the are of intersection between classes. Notice how number 8's that look similar to a 3 disappear.
32

---

Published as a conference paper at ICLR 2026
7
7
7
7
7
7
7
7 7
7 7 7
7
7 7
3
7
7
7
7
7 7
7
7
7
7 7
7
7 7
7
7
7
7
7
7
7
7 7 ? / 7 7
7
7
7
7
7
7
7
7 7
777
7 7
7
7
7
7
7
7
7
7 7
7 7
7
77777770
7
7
7
7
7 7 7
7 7 7
777
7.7777777
7
7
1
7 7
7
7
7 777777
777777.1
7
7
7
7 7 7
7 7777771
7777 7
77777777
7 1
7
7
7
7 7
7777777
(a) Generating samples 7 with no guidance
(b) Generating samples us-
(c) Generating samples of 7 using the class of number as the guiding distribution using =
ing guidance with w =
Figure 20: Applying guidance reduces the area of intersection between classes. Notice how number 7's that resemble disappear.
STATEMENT ON THE USE OF LARGE LANGUAGE MODELS
This work made use of large language models to assist with proofreading and improving the clarity of the writing. All research ideas, theoretical development, and experiments were carried out solely by the authors. When used for coding, was solely used for plotting purposes.
33


Reviews:
[{"review_id": "ZgFCsrjc9x", "reviewer": "Reviewer_XSKv", "summary": "This paper provides a rigorous analysis of classifier-free guidance (CFG) for masked discrete diffusion. Under exact scores and exact reverse dynamics, the authors derive closed-form reverse dynamics and generated distributions in 1D and 2D. In 1D, CFG precisely samples from the tilted distribution pz,w, whereas in 2D, the generated distribution deviates with explicit marginal dependent reweighting, shifting mass from overlapping to class-specific regions. For large guidance strength $w$, the total variation (TV) to the terminal distribution decays in time with a rate that is double-exponential in $w$. The paper features experiments on synthetic 1D, 2D and higher-dimensional toy setups, plus MNIST case studies, which qualitatively support the theory.", "questions": "In addition to the weaknesses outlined in points (i-iii), I present the following questions for the authors to address:\n\n(1) Proposition 3.3 partitions the space into regions $\\mathcal{R}_1, \\ldots, \\mathcal{R}_4$. Can you provide examples of real data where such region decompositions would be meaningful?\n\n(2) The analysis assumes exact concrete scores. How would score approximation errors change the conclusions?\n\n(3) The findings are specific to masked discrete diffusion, following the absorbing forward process. Would the conclusions still hold for other discrete processes (e.g., D3PM-uniform modelling)?\n\n(4) Strong guidance suppresses shared regions and reduces sample diversity. Is this “loss of diversity” always undesirable, or can it be beneficial in certain applications?\n\n(5) What are the main obstacles to extending the exact analysis beyond 2D, and what aspects of the results are most likely to generalize?\n\n(6) In 2D you introduce marginal coefficients $\\lbrace c_x, d_x \\rbrace_{1 \\leq x \\leq N}$ that encode the steering effect of guidance on marginals, consequently influencing the sampling dynamics. What intuition can we build for how these coefficients arise and what they represent?\n\n(7) [minor concern w.r.t. to presentation] Figure 2 feels slightly out of place, can you place it at the top of page 8?\n\n(8) [minor concern w.r.t. to presentation] While Notation was shortly introduced in the preliminaries section, could you provide a simple table of notations in the appendix to further improve overall readability and accessibility of your work?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "(i) The paper provides the first rigorous treatment of CFG in discrete masked diffusion with explicit formulas in 1D and 2D\n\n(ii) The paper provides clear geometric interpretation. Guidance suppresses overlapping regions and amplifies \"private\" regions, quantified via region-wise weights in 2D\n\n(iii) The paper's simple, targeted experiments align with theory", "weaknesses": "(i) The paper's scope appears limited to masked absorbing diffusion and low dimensions, an extension to higher dimensionality $D > 2$ remains informal\n\n(ii) The paper poses idealized assumptions (exact scores, exact reverse simulation) with little analysis of approximation / discretization error or robustness under practical solvers\n\n(iii) While the main contribution of this work is a theoretical discussion, the scope of experiments remains thin, and the robustness and scalability of results is underexplored", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vHJE7KA9nC", "reviewer": "Reviewer_9vfg", "summary": "This paper extend the analysis of CFG in continuous state diffusion to the masked discrete state diffusion. It has two main results:\n\n* In 1d situation, for masked discrete state diffusion, with direct construct of \\\\(\\hat{Q}\\\\), we can reach the tilted distribution.\n* In 2d situations, it is not that simple.\n* This paper also proposes some analysis on multi-guide situation.", "questions": "n/a", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The theoretical analysis and calculations are solid, especially the ability to derive exact distribution results— outperforming continuous-state diffusion on CFG in the 1D case.\n* The inclusion of the multi-guidance setting adds depth and richness to the paper.", "weaknesses": "* Unfortunately, the 1D setting is overly simplistic. While the proposed techniques are effective in 1D, they become increasingly complex in * 2D and are difficult to generalize to higher dimensions due to inherent limitations.\n* The experimental evaluation relies too heavily on toy examples, which weakens the practical impact of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wiSKBnKLw9", "reviewer": "Reviewer_wYwP", "summary": "This paper develops the first rigorous quantitative theory explaining the role of classifier-free guidance (CFG) in masked discrete diffusion models.\nThe authors analyze low-dimensional cases (1D and 2D) where the reverse dynamics can be solved exactly. Their results show that:\n- Guidance amplifies class-specific regions while suppressing overlapping regions between classes, with overlap vanishing as guidance strength $w$ increases.\n- In 1D, the generated distribution exactly matches the tilted distribution $p_{z,w} \\propto p(x)p(z|x)^{1+w}$.\n- In 2D, deviations emerge but can be expressed in closed form via coupling coefficients $c_x,d_x$.\n- The convergence rate of the reverse dynamics exhibits a double-exponential dependence on w in both 1D and 2D.\nEmpirical illustrations confirm the theoretical predictions.", "questions": "- Could the authors extend the proof techniques to approximate scores (learned $s_\\theta$) or noisy simulations?\n- How sensitive are the observed phenomena to the choice of the forward process (absorbing vs. uniform)?\n- Is the double-exponential convergence rate provably tight, or an upper bound?\n- In 2D, does the deviation from the tilted distribution scale polynomially or exponentially in overlap size?\n- Could these results suggest a new adaptive guidance schedule where w increases dynamically?\n- How might the regional weighting structure $A^{z,w}_i$ inform geometry-aware training or regularization?\n- Are there potential connections to discrete optimal transport or entropic regularization frameworks?\n- Could the asymptotic results be empirically observed in token-level text diffusion models?\n- Are the coefficients $c_x,d_x$ interpretable as marginal reweighting factors for token dependencies?\n- How might partial overlapping supports in high dimensions influence generalization or calibration?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Novel theoretical framework: First rigorous analysis of discrete CFG dynamics.\n- Analytic tractability: Closed-form results for both 1D and 2D masked diffusion.\n- Clear phenomena: Demonstrates class-specific amplification and overlap suppression quantitatively.\n- Double-exponential convergence: Elegant link between guidance strength and diffusion rate.\n- Bridges gaps: Unifies discrete and continuous CFG theories.\n- Empirical alignment: Simulations verify analytical predictions.", "weaknesses": "- Heavy reliance on exact score and continuous-time limit; numerical approximations and learned scores are not analyzed.\n- Empirical validation is illustrative rather than large-scale.\n- Some proofs deferred to appendices could benefit from intuitive discussion in the main text.\n- Limited exploration of $D \\ge 3$ behavior; higher-dimensional extension remains conjectural.\n- Minor presentation complexity (dense notation, multi-index expressions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HYLBo4EHwO", "reviewer": "Reviewer_XFWx", "summary": "This paper provides a theoretical analysis of classifier-free guidance (CFG) in masked discrete diffusion models. In 1D, the guided reverse process exactly recovers the tilted distribution. In 2D, deviations emerge: the final distribution is a reweighted version of the tilted distribution, with mass suppressed even in regions overlapping only in projection.\n\nThe authors derive closed-form expressions in both settings and show that convergence to the final distribution exhibits double-exponential decay in the guidance strength w. Experiments in higher dimensions (5D, MNIST) confirm that these geometric effects—amplifying private regions and suppressing ambiguous ones—persist beyond low-dimensional cases.", "questions": "See weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "I like this work. The paper is well-written and clear. It presents a novel theoretical insight into how classifier-free guidance (CFG) affects discrete diffusion sampling, with concrete examples and rigorous proofs. The authors also provide high-dimensional experiments to support the generality of their findings.", "weaknesses": "It would be interesting to discuss how this effect manifests in practical applications, and how the misalignment introduced by CFG could be addressed in real-world settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)