Information-Theoretic Discrete Diffusion
Authors: Moongyu Jeon
Sangwoo Shin
Dongjae Jeon
Albert No
Venue: NeurIPS 2025 poster ()
Keywords: Discrete Diffusion Models
Information Theory
Score Matching
Denoising Score Entropy (DSE)
Denoising Cross-Entropy (DCE)
Primary area: theory
TL;DR: We derive information-theoretic identities for discrete diffusion, revealing score-based losses as exact mutual information decompositions and enabling principled log-likelihood estimation.

Abstract: We present an information-theoretic framework for discrete diffusion models 
that yields principled estimators of log-likelihood using score-matching losses.
Inspired by the I-MMSE identity for the Gaussian setup, we derive analogous results for the discrete setting.
Specifically, we introduce the Information–Minimum Denoising Score Entropy (I-MDSE) relation,
which links mutual information between data and its diffused version to the minimum denoising score entropy (DSE) loss.
We extend this theory to masked diffusion and establish the Information–Minimum Denoising Cross-Entropy (I-MDCE) relation,
connecting cross-entropy losses to mutual information in discrete masked processes.
These results provide a time-integral decomposition of the log-likelihood of the data in terms of optimal score-based losses,
showing that commonly used losses such as DSE and DCE are not merely variational bounds 
but tight and principled estimators of log-likelihood.
The I-MDCE decomposition further enables practical extensions, including time-free formula,
conditional likelihood estimation in prompt–response tasks, and coupled Monte Carlo estimation of likelihood ratios.
Experiments on synthetic and real-world data confirm the accuracy, variance stability, and utility of our estimators.
The code is publicly available at https://github.com/Dongjae0324/infodis.

Information-Theoreti Discrete Diffusion
Moongyu Jeon'
Dongjae Jeon" Albert No Department of Artificial Intelligence, Yonsei University Department of Computer Science, Yonsei University
Sangwoo Shin'
Abstract
We present an information- theorreti framework for discrete diffusion models that yields principled estimators of log-likelihood using coore-macching losses. Inspired by the I-MMSE identity for the Gaussian setup, we derive analogous results for the discrete setting. Specifically, we introduce the Information Miniimmm Denoising Score Entropy (I-MDSE) relation, which links mutual information between data and its diffused version to the minimum denoising score entropy (DSE) loss. We extend this theory to masked diffusion and establish the Information-Minimon Denoising Cross-Entropy (I-MDCE) relation, connecting cross-entropy losses mutual information in discrete masked processes. These results provide a time- integral decomposition of the log- lielikkelihood of the data in terms of optimal score- based losses, showing that commonly used losses such as DSE and DCE are not merely variational bounds but tight and principled estimators of log-likelihood. The I-MDCE decomposition further enables practical extensions, including time-free formula, conditional likelihood estimation in prompt-response tasks, and coupled Monte Carlo estimation of likelihood ratios. Experiments on synthetic and real- world data confirm the accuracy, variance stability, and utility of our estimators. The code is publicly available at Ihtpps:/// /tt::/iitttt thub..u..ccm com www.ecooooooooeeeeeeee hetp///ngaee32/iinnttteo 3//////////iiiiii////////
Introduction
Diffusion models have emerged powerful framework for generative modeling, enabling state-
of-the-art performance in continuous domains such as image and audio generation (Sohl-Dickstein
al. 2015: Ho et al. 2020 Chen al., 202T Kong et al.) 2021 Saharia et al.) 2022). Central to
these models is the idea of gradually corrupting data through a forward noising process, and learning
reverse this corruption via score-based loss (Hyvärinen and Dayan 2005
Vincent 201 Song and
Ermon 2019 Song et al. 202
Recent works have extended diffusion models discrete domains, proposing models designed for
categorical data (Hoogeboom et al. 2021: Austin et al. 2021 Campbell et al. 2022 Meng et al. 2022
Sun et al. 2023 Lou | [024 Sahoo et al. 2024 Shi et (2024). These models offer promising
alternative to traditional autoregressive approaches (Radford et al. 2018 2019 Brown et al. 2020).
particularly for sequence modeling tasks that involve text and other symbolic structures (Li al.
2022
Nie et al. 2025).
Continuous diffusion models benefit from a well-established information-theonede foundation (Kong ct 2023 2024). In the Gaussian setting, the I-MMSE identity (Guo et al. 2005) Venkat and Weissman 2012) connects the mutual information between clean and noisy variables to the minimum mean squared error (MMSE), offering both theoretical insight and a basis for likelihood estimation. pointwise generalization of this identity yields closed-form decompositions of the data log likelihood in terms of estimation losses (Kong et al.] (023). However, the discrete case has not yet been investigated from an information-theereticc perspective.
"Correponneece to: Albert No <albertnodyonsei.ccmm
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In this work, we extend these ideas to discrete diffusion models, developing an cettmmation-thhorcee framework that rigorously characterizes the relationship between mutual information and score-based loss in discrete settings. We first establish the Information-Minimumn Denoising Score Entropy (I-MDSE) identity, which connects mutual information decay in the forward process to the minimum of the denoising score entropy (DSE) loss. This leads to closed-fome decomposition of the negative log-likelihood (NLL) into a time-integral of the minimum DSE, showing that the DSE loss, previously viewed as variational bound, actually constitutes principled estimator for likelihood estimation.
We then turn to masked (absorbing) diffusion models, where we formulate the pointwise denoising cross-entropy (DCE) loss and prove its equivalence to the DSE loss under certain time reparameteri- zation. Leveraging this equivalence. we derive the Information-Minine Minimum Denoising Cross-Entroph (I-MDCE) identity, which mirrors the I-MDSE relation but in the masked setting. The I-MDCE iden- tity provides a parallel time-integral decomposition of the NIL along the minimum DCE trajectory, revealing that the DCE loss, like DSE, serves as theoretically grounded training objective that enables exact estimation of the data likelihood.
Building on this decomposition via I-MDCE, we derive a time-free reformulation of the log-likelihood. expressed as an expectation over randomly selected unmasked token subsets. This formulation enables efficient Monte Carlo estimation of the NLL without diffusion-time integration and extends naturally conditional likelihoods of the form Pu Pu(x""reet x= allowing estimation in structured generative settings such as prompt-response modeling. Furthermore, by coupling the sampling paths of two sequences, our framework provides a principled Monte Carlo estimator for likelihood ratios, achieving unbiased and low-variance estimation compared to independently sampled baselines.
We validate our framework through experiments on both synthetic and real-world datasets. First, using synthetic datasets with known ground-truth distributions, we show that our estimators accurately recover both unconditional and conditional log-likelihoods. Next, we verify the variance reduction properties of our time-free likelihood estimator and the coupled likelihood ratio estimator against their respective baselines. Finally, demonstrate the practical utility of our approach through auditing experiments on real-world data, where conditional likelihood estimates detect inputs and reveal distributional shifts in LLaDA (Nie al.) (2025). These results confirm that our information-theone framework not only offers theoretical insight but also enables accurate and interpretable likelihood estimation in discrete generative models.
2 Preliminaries
2.1 Discrete Diffusion Models and Score Matching
Given data Ty ~ Pu. the forward diffusion process is modeled continuous-time Markov chain (CTMC), governed by a linear ODE (Anderson) 2012 Campbell et al., 2022):
dp:
QiPt, Po Pdata
(1)
di
where Qr E N is the time-dependent transition rate matrix, with N |X possible states. As t + 20, the marginal pt converges to a stationary distribution π. For tractability, it is common to assume factored form Q: ( where Q a fixed matrix and (()) a positive scalar function.
The reverse process is also governed by another CTMC, described by the following ODE (Kelly) 1980 Sun et al. 2023 Lou et al. 2024
dpr-t QT (,(,,))= dt
p. (v(2) (x( v)
if # y, if y.
(2)
-
> jez (ū,
To simulate the reverse process, one typically initializes at pt- = ग and replaces the marginal ratio
px(z) in Eq. (2) with learned approximation, yielding parameterized family (r P=(x)
Early methods (Austin et al. 2021 Campbell et al. 2022 modeled the reverse conditional Poje
directly, but suftered from combinatorial scalability issues. Meng et al. (022) instead used an 8
regression loss to approximate the marginal ratio p:(v)
which proved unstable in practice.
Pe(z))
Subsequently, Lou et al. (2024) introduced the denoising score entropy (DSE) loss, resolving these
scalability and stability issues. Specifically, they use score network s² X x [0,T] + V to
2

---

estimate the marginal ratio, where each output s (2, fly corresponds to pr(u) by minimizing the DSE loss, defined pointwise
The network is trained
Po)(y) log st(2), K (P)o(()ze)) p)o(
EDSE s1) = E (s(())y
(3)
where [(a) = allog a --1--- 1).
This loss is minimized when the score network recovers the true score st, where s7(2),
P:(y).
pr())
o = arg min Ep(xo...) (eDSE Fz, t,,t, st)]
(4)
8t
Aggregating this loss over time yields the time-integrated DSE training objective:
Dss (2o) = L. Epnolzuleo) (fo, (,,,,)))). dt,
where of = (-, () is the learned score at time I.
Importantly, this loss also serves variational upper bound on the negative log-likelihood (NLL) of the sample zo under the learned distribution:
log (20) I (Bs DKL (Prio(-(=o)111)
This dual role of the DSE loss. both score motehiat loss and variational bound, makes it principled and practical training criterion for discrete diffusion models.
2.2 Masked Diffusion with Absorbing Transition Matrix
In practice, discrete diffusion models are defined over sequences x = 22.22...'' E X A major challenge in this setting is the intractability due to the exponential size of Qi E xN'"""
To address this, previous work (Campbell et al. 2022 Lou al. 2024 assumes that each token evolves independently under a shared rate matrix Q a(1) E NNN, This assumption significantly reduces the complexity of the score network. Specifically, one only needs estimate marginal ratios for sequence pairs that differ by a single token:
(xx,t)
8 x
Pc(x (lux....2
The token-level forward transition is given analytically by P:jo('(<') (z') exp(o(t)Qtnn) ( where zi.
(1) J0 (s)ds. Closed-form expressions of the transition matrix exp(a Qt) are known only for specific choices of Qtok, the uniform Quniform and the absorbing Quhsorb Lou et al. 2024).
Of particular interest is the absorbing process, where Qabserb allows only transitions from unmasked tokens to a special mask token M This simplifies the reverse process by restricting the score computation pairs (x, x) that differ by exactly one masked position, with j xx M].
key property of absorbing diffusion is that the marginal ratios (x()) admit time-free reparameteri-
(x(x)
zation Ou et al. 2025). Specifically, for pair (x, x) with a of x²
[M] described above.
(x) c (1)
Pr (x) 1 1-e-o() J(t) Po (2 ( UM)
(5)
where x UM denotes the subsequence of unmasked tokens in x.
This result motivates the use of a time-inder inmeiindeeendent network & (1....,N, predict the conditional distribution of unmasked tokens
(M) + BLNN to
(x) 10 UM) s po x UM).
3

---

To simplify the time-integrater integrated DSE loss, reparameterizze time t using A(t) = -111- 7(0, which monotonically increases from O to 1. In this coordinate system, Ou et al. (0225) introduced the denoising cross-entropy (DCE) loss:
(DCE(XO) = -[- BPXIO(AA1NN))
L Σ
1[ = [M] log
dA.
po (= x XM)
Importantly, they also proved that this loss is equivalent to the DSE loss in the full-noise limit: lim DSE (xo) = (x))
T-400
which provides simpler yet equally principled alternative
matching. This formulation underlies recent large-scale masked diffusion language models such as LLaDA (Nie et al. 2025).
2.3 Information- Theorettic Diffusion
To motivate the development of information-tthereeee tools for discrete diffusion, we begin by re- viewing key results in the continuous setting, particularly the connection between mutual information and estimation error in Gaussian diffusion models.
Consider the standard Gaussian diffusion forward process (Sohl-Dickstein et al. 2020), where a data point ~ Po is corrupted via the noise channel
2015
Ho al.
Z, / X + N0, 7)
where Y is the signal-to-noise ratio (SNR). This channel defines reparameterized version of the diffusion forward process, often adopted in variational diffusion models (Kingma et al. 2021).
To reverse the diffusion, score- based models (Song and Ermon 2019 Song al.
(021b) learn the score function log p(Z,), often reparameterized using Tweedie 's formula (Efron 2011) in terms of a denoiser Xg. The training objective can then becomes denoising mean squared error (MSE) loss:
LMSE
1
x Xz( 1| ?|? dy.
2
which encourages the denoiser to approximate the MSE-optimal predictor Z,)
This denoising interpretation links naturally to foundational identity in information theory: the I-MMSE relation (Guo et al. 2005), which states
T(X: Z,) = mmse(?),
dy
| ][ quantifies the minimum MSE (MMSE) at noise level yy Venkat and Weissman (2012) established a strong pointwise generalization of the I-MMSE identity. independently rediscovered the conditional form in the context of
where mmse(77 E
||X
TE X
More recently, Kong et al. (2023)
diffusion modeling:
d
DKL. (p(Z, Xo) (Z,)) mmse (Xo.7) ().
d- [||X
where mmss((Xo
E EX, X = Xo] is the pointwise MMSE.
Building on this, Kong et al.
2023 characterized the data log-likelihood in terms of denoising error:
log Po
(Xo)
mmsc(Xo, 7) dy + const.
Jo
This result offers a practical and interpretable approach likelihood estimation using the denoiser X, learned by the diffusion model.
Subsequently, Kong et al. 2024) extended this formulation to the conditional setting. Given auxiliary information Yo. the conditional negative likelihol admits similar decomposition:
log X, Yo) = /- mmse X,) Yo, 7) dy + const.,
(8)
E [ ZZ, Yo X Xo] the conditional pointwise MMSE. This enables the conditional likelihood estimation using a denoiser trained with data condi- tioned on the variable Y Yo. making useful in applications such as prompt-to-output modeling in text-to-image generation.
where mmsse(Xo

---

Information-Thoorele Discrete Diffusion
The ceermatiionthhorelie foundations of continuous diffusion models, reviewed in Section 2.3 motivate our exploration of analogous principles for discrete diffusion models. In this section, we establish a framework that links mutual information with score- training objectives in the discrete domain. By leveraging discrete counterparts of the I-MMSE identity, we derive decompositions for data log-likelihoods. All theoretical proofs are provided in Appendix
3.1 I-MDSE Relation: An Information-Theoree Identity for Discrete Diffusson
We now establish a discrete analog of the I-MMSE identity. In contrast to the Gaussian setting, where estimation framed in terms of squared error, discrete diffusion models governed by the CTMC rely on score ratio estimation, captured by the denoising score entropy (DSE) loss. We show that the rate of information decay in the CTMC governed by the minimum value of the DSE loss, yielding what we refer to as the Information-Minmmee
Denoising Score Entropy (I-MDSE) relation.
I-MDSE Relation. For the optimal score function s7 that minimizes the DSE loss and recovers the (e( (Eq. (4), we define the corresponding minimum loss as:
marginal ratio pe(y)
mt) = min Bp( [EDSE (2g. 2z, t, st) Ep(ro.r)) (EDSE 24, t, s,,]
St
where DSE the pointwise DSE loss defined by Eq. (3)
To capture the information decay from specific input zo. we also define the pointwise MDSE:
mndse 1) == E o< (mo) [CDSE Tt, s])] so that mdse(() = (xp() [mdse( 4)]
We are now ready to state the discrete counterpart of the I-MMSE identity.
Theorem 3.1 (Pointwise and Marginal I-MDSE Relations). For a discrete diffusion model governed by a continuous Markov chain (Eq. (7),, the following pointwise 1-MDSE relation holds:
d
DEL (P10(-1z0) ll Pt) = -mdse(Øo. t).
(9)
dt
Taking the expectation of both sides with respect to To ~ Po yields the marginal I-MDSE form:
d (20; = mdse(((.
(10)
dl.
The negative sign reflects the nature the diffusion process, in which information decays over time, since the DSE loss is always nonnegative (Lou et al. 2024). This aligns with the I-MMSE relation, where increasing SNR 7 (the inverse of time t) corresponds to an information gain.
NLL Decomposition. The I-MDSE relation further implies a decomposition of the negative log- likelihood (NLL) along the MDSE trajectory, directly paralleling Eq. 7 in the Gaussian case: Theorem 3.2 (NLL Decomposition via I-MDSE). For any finite time T > O, we have
/ mdso(x0, t) dt + DKL
- log Po(PPoFF)
(11)
Taking the limit T > oo and assuming pT - n. for any initial po, we obtain:
|0 mdse(zo. t) dt.
log
(12)
This result reveals that the time trajectory of the DSE loss fully captures the log-likelihood of data point. Practically, the integral in Eq. (12) can be estimated using a learned score network of place of the true ratio sf, yielding:
- log (xo) x Eppo(e.1z0) [ADSE xi, x,, t.]] dt lim (xo)
T-000
5

---

The I-MDSE identity reveals that the commonly used DSE loss, previously viewed as a variational upper bound, is in fact an exact and theoretically grounded estimator of the log-likelhhod. This equality shows that first-order score functions suffice and that no higher-order corrections are needed for likelihood estimation. Much like the I-MMSE justifies MSE in Gaussian diffusion, I-MDSE positions the DSE loss as principled and information-theoreiiial diffusion, with direct implications for both training and likelihood estimation.
sound objective in discrete
3.2 I-MDCE Relation: An Information-Theoreee Identity for Masked Diffusion
We now extend the information -ooomation-theortticc analysis to masked diffusion models, where noise is applied via an absorbing process and estimation performed through conditional prediction. In this setting, the loss of interest is the denoising cross-entropy (DCE) loss, which replaces score estimation with masked token reconstruction. Leveraging its pointwise equivalence to the DSE loss, we derive the Information-Minine
Denoising Cross-Entrops (I-MDCE) relation, the analog of the I-MMSE identity in masked diffusion. This result leads to a decomposition of the negative log- likkliihoood (NLL) analogous to that obtained via the I-MDSE relation.
From DSE to DCE. Before deriving the ieformatioonthhoreeee results, we first establish the point- wise equivalence between the DCE and DSE losses, which forms the basis for extending our analysis in Section 3.1 to masked diffusion.
Let c: [1, NN, [M] 4 + LxN be function predicting conditional distributions. We define the pointwise DCE loss as
(DCE x, c) = Σ 1|=²
[M] log
c(x);,=6
This loss serves the discrete analog to squared error in the MMSE setting, measuring cross entropy (predictive accuracy) over masked positions.
We define the time-integrated DCE loss over the noise level A E [0, 1] as:
L == /,a
A (PPioo(xsi))) (x0)) [(DCE(XO xx, xx,, c dA.
When the conditional predictor e and the score predictor are linked via the time-free reparameteri- zation (Eq. ()), have:
1-
8 s(x, t)
c(x)s,2s for x² + X [M].
where À 1 1- (()), This leads to the following equivalence of the pointwise loss functions: Lemma 3.3. If and are corresponding under time reparameterization, then
EDSE ( x,t, (1)) - (DEE(Xo, x).
This result establishes exact correspondence between the time-integrated DSE and DCE losses, extending prior work that showed only asymptotic equivalence in the full-noise limit (Eq. (6)
Theorem 3.4 (Training Loss Equivalence). Let A = - T(T) and if d and e are corresponding under time reparumeterization. Then,
From I-MDSE to I-MDCE. Having established the equivalence between the DSE and DCE losses, we now extend the information-theoree analysis the masked (absorbing) diffusion process. As in the DSE setting, the DCE loss is minimized by the true conditional distribution of the data:
Theorem 3.5 (DCE Optimality). Let c* be the data- inta-induced conditional predictor. defined by c" (x)),as
(ā x UM), Then,
c*
= arg min (p(xx. x.) (xDCE(XO. x),

---

Using the optimal c*, we define the minimum DCE (MDCE) loss and its pointwise version as:
(A) = xp(xo.xA) (XDCE(X,, Xx, cx), mdce(Xo. A) = (x)o((x) ((,, Xx,
so that md)c((A) = po ( A)].
We are now ready to state the I-MDCE relation, the masked diffusion variant of the 1-MDSE identity: Corollary 3.6 (Pointwise and Marginal I-MDCE Relations). For the absorbing diffusion model, the following identities hold:
d
dA DKL (px1(-1xx0)1pp)
mdce(xx, A),
X
d
1 mdce(A).
[xx:
dx
Integrating these differential identities yields decomposition of the log- ligeliiel:oood: Corollary 3.7 (NLL Decomposition via I-MDCE). For any A [0,
-logPo(Xu) = -,0 mndce(Xo, A) dd + DEL (PAjo(-
and in the full-noise limit A = 1, this reduces to
Polo))()o) - S/' mdce(x0, A) dA.
(13)
In practice, replacing c* with learned predictor c" gives the estimator
-log f [PPxooxxxioo [CDCR(XO: xx, )² dd LDCR(XO).
(14)
Similar to I-MDSE, the I-MDCE identity shows that the DCE loss used in masked diffusion training corresponds exactly to the log-likelihood, rather than serving merely as variational upper bound. This establishes that first-order conditional predictors are sufficient for likelihood estimation in masked settings. Beyond theoretical value as a principled foundation for training objectives, I-MDCE also enables accurate and stable likelihood estimation in practical language modeling tasks, as demonstrated in the followings sections.
4 Extending I-MDCE: Variants, Generalizations, and Applications
4.1 Time-Free Likelihood Estimation: Variant (Alternative Formulation)
While the integral formulation of the NLL via I-MDCE (Eq. (33) provides a solid theoretical foundation, it requires continuous integration over the diffusion coordinate. Here, we present an equivalent but more practical formulation by removing explicit time integration. This yields time- free expression for the NLL based solely on randomly selected masked positions.
Theorem 4.1 (Time-Free Likelihood via I-MDCE). Let B(-,)) denote the Beta function and Hr. denote the L-th harmonic number. Then,
log Po (Xo) Hr
Σ log
Po
(at (xtllx))
Ligh
where x denotes the subsequence of Xo consisting of the tokens indexed by I, and T C {1,..., , I.} is the set of unmasked indices sampled from p(I) B(L- 11+1)
Ha
Similar time-free free formulations introduced in Ou
2025
Nie et al.
2025
See Appendix

---

To compute this expression in practice, we approximate the conditional distributions using the learned predictor e", Given clean sequence X and set of unmasked indices let x denote the sequence obtained from Xy by masking all tokens whose indices are not in J. We then approximate the conditional probability as
e ( = Po Po x
Using this approximation, we obtain the following time-free estimator for the likelihood:
-log RJ Hy BpCt)
Σ
log c² (xx
(15)
LigI
This formulation exhibits substantially lower variance than the time-integral form (Eq. 13 as we
empirically demonstrate in Section 5.2
4.2
Conditional Likelihood Estimation: A Generalization (Structured Prediction)
The I-MDCE framework naturally extends to conditional likelihood estimation, serving as a discrete analog of Eq. (8) in the Gaussian setting, and enabling the selection of target and context components within a sequence. This particularly useful in structured tasks such as prompt-response modeling,
where the goal is compute log po (x"1 |x(*) for disjoint index T, S (1....,44 L.).
Theorem 4.2 (Conditional Likelihood via I-MDCE). Let I and I2 be disjoint index sets, then
log po (x (x(1)
Σ 1 = [M] log LE
Po Pa (x) (X x5²) dd. (16)
PA PA
common example is the prompt-response setting, where the first d tokens are treated context and the remainder as the target. In practice, the integrand can be approximated using the learned conditional predictor c" as in the unconditional case described in Eq. (14).
Morcover, this integral form also admits time-free equivalent based on randomly unmasking subsets of the target positions:
Corollary 4.3 (Time-Free Conditional Likelihood via I-MDCE), Let Iı and Iz be disjoint index sets and let J be the randomly selected unmasked index set in I, then
- Poog 1x = Hp(1
Σ
log
(17)
po (x (U1
NETN
where the sampling distribution is p(J) 1
Hiral
In practical settings, the conditional terms (wd) X
(u1)) can be approximated using the trained model c² yielding a time-free estimator for structured conditional likelihoods analogous to Eq. (15)
Likelihood Ratio Estimation: An Application (Downstream Task)
Our equality-based formulation provides principled foundation for likelihood ratio estimation using learned scores. Unlike variational bounds, which offer no guarantee when subtracted, our exact decomposition ensures that likelihood ratios can be estimated consistently and robustly. This perspective helps explain the empirical stability recent alignment methods based on likelihood ratios in masked diffusion language models (Zhu et al. 2025).
Moreover, our time-free estimator admits coupled Monte Carlo form, where a shared mask is used for both sequences:
Po(y) Pn(x)
Po ( (y Pa (2r [x!)
log
H Ep(r)
Σ log LigT
(18)
Coupling via shared randomness not only ensures unbiasedness but also substantially reduces variance compared to standard decoupled estimation.

---

Experiments
We empirically validate the proposed I-MDCE framework through both controlled and real-world experiments. We first confirm that the time-free estimators accurately recover ground-truth likelihoods in toy settings. We then demonstrate the variance reduction effect of our estimators, showing that the time-free and coupled ratio estimators yield substantially lower Monte Carlo variance than their respective baselines. Finally, showcase the utility of our framework in real-world tasks, including out-of -ff-tribittitttionn detection and model influence analysis using the open-source LLaDA model (Nie
al. 2025). Further details are provided in Appendix D
5.1
Reliability of Likelihood Estimation Toy Data
This section verifies that the time-free free estimators, both unconditional (Eq. (15)) and conditional (Eq. (T7)). accu-
7.5
True NLL
Estimured NLLL
rately recover true likelihoods in controlled toy settings.
6.0
NMA
Unconditional Likelihood. We first consider an un- conditional setup using synthetic DNA sequences over the alphabet {A, T, G, CJ. A ground-truth distribution defined by assigning random probabilities to 128 se-
5.5
4.5
4.0-
quences of length 8, from which one million samples are drawn to train a RADD (Ou et al., 2025) model. Figure Ta compares the true likelihoods with those es- timated by Eq. (15) via Monte Carlo (MC) sampling, showing strong agreement and validating the accuracy of the unconditional estimator.
ATA
AGA
CGA
YGA
NRT
TUG
ANC
TCD
OCT
G0A
100.
(a) Unconditional NLJ via Eq. 15
True NLL
Estimated
Conditional Likelihood. We next evaluate the condi- tional estimator in a more structured scenario. long DNA sequence of length five million is generated by 4th-order Markov chain, defining a probability distribu- tion over all contiguous subsequences. Subsequences of length 32 are randomly sampled for training, while a held-out sequence is split into prompt (xpromppt, first 16 bases) and a response (x response remaining 16 bases). We estimate Po Po(x (Ssooone pppompt) via Eq. 17 and compare it to the ground-truth conditional probability from the Markov process. Figure [b demonstrates that estimated values closely match the true likelihoods, confirming the reliability of our likelihood estimator even under complex conditional dependencies.
10
JAA
TMA
ART
COA
ACA
AIA
cos.
AA.
OCT
TAA
IT
O
ART
(b) Conditional NLL via Eq.
17
Figure 1: Comparison of true and esti- mated NLLs on 64 sequences using our time-free estimators. Full results pro- vided in Appendix D.2
5.2 Variance Reduction in Likelihood and Ratio Estimation
We evaluate the variance-reduction benefits of our estimators on LLaDA (Nic al. 2025). focusing on the time-free likelihood estimator (Eq. and the coupled likelihood ratio estimator (Eq. [8]
Time-Free Likelihood Estimator. We compare the variance of our time-free estimator against the time-integral baseline (Eq. (16) by measuring the Monte Carlo variance conditional log- likelihood estimates. As shown in Table la the time-free estimator consistently achieves substantially lower variance across datasets, HellaSwag Zeelerr et al. 22019), ARC-hard (Clark et al., (2018), and PIQA (Bisk et al. 2020), and for various numbers of Monte Carlo samples. These results demonstrate improved robustness and sample efficiency.
Coupled Likelihood Ratio Estimator. We also validate the variance-reduction effect of our coupled likelihood ratio estimator by comparing it with a standard decoupled baseline. Experiments were conducted on the BeaverTails dataset (Ji et al.) 2023) using 500 prompt-response triplets (xprompt +sspoose,+ x xssoonse.-), "), where = and x raponse. denote preferred and dispreferred responses, respectively, For each triplet, estimate the log- likelloodd ratio eight times to measure

---

Table 1: Monte Carlo variance comparison of likelihood estimators. (a) Conditional log- likeliihod estimation on three datasets, with variance measured over 15 independent samples. (b) Log-lik log-likelihood ratio estimation on the BeaverTails dataset, with notably lower variance from the coupled estimator.
(a) Conditional likelihood estimation
(b) Likelihood ratio estimation
# MC samples
HellaSwag
ARC hard
PIQA
MC samples
Coupled
Decoupled
Time-int.
Time-free
Time-int.
Time-free
Time-int.
Time-free
5
8897.08
62469.4 41
128
70.97
11.57
23.18
5.73
19.77
4.93
10
4487.38
29107
256
30.19
6.02
18.14
2.96
15.15
1.81
15
3059.97
20695.61
2,92
20
2335.12
16514.72
empirical variance. As shown in Table Tbj the coupled estimator consistently achieves lower variance across all sample sizes, confirming its superior stability and sample efficiency.
5.3 Auditing and Interpretability via Conditional Likelihood Estimation
We explore the utility of our time-free conditional estimator in real-world auditing tasks aimed at inferring distributional properties of pre-trained models, such detecting out-of- distribution (OOD) inputs or identifying training influences. These experiments show that conditional likelihood estima- tion provides an effective tool for interpreting model behavior.
0.12
Sentil
GIPT geserated
0.06
0.00
0.00
Detecting Out-of-Distribution Inputs. We first whether conditional likelihoods estimated by Eq. (17 can separate in-distribution sequences from semantically unre- lated continuations. RADD (Ou et al. 2025) is trained on the text8 corpus (Mahoney| 2011), and we compute the condi- tional NLL- log Po eeeee x miompt) for two response types: (1) original continuations from text8 and (2) unrelated re-
Figure 2: Estimated NLL for
in-distribution (blue) and out-
of-distribution (magenta). See
Appendix (op]edii(D.4ffor Do4 details.
sponses generated by GPT-4 (Achiam et al. 2023). As shown in Fig. 2 the NLL histogram reveals a clear separation: GPT-ggnnrraed responses have much higher NLL.,, while original continuations
receive higher likelihoods. This confirms that our estimator can reliably detect OOD samples.
Application a Large Open-Source Model. We further
analyze input distributions using the open-source LLaDA model (Nie et al. 2025) on two datasets: WikiText (En- glish) and pretrain zh (Chinese). For each prompt. timate conditional NLLs for the original dataset continuation and for completions produced by LLaMA 3.1 (Grattafiori cet al. 2024). Figure 3 shows that LLaMA 3. -.I-gneerteed responses tend to receive higher average likelihoods than those from both datasets, suggesting that LLaDA may have been partially influenced by LLaMA 3.1 during training.
0.000
WikiTes
1LMM generated
0.004
0.000
0.000
Figure Estimated conditional NLL on WikiText (blue) and LLaMA 3.1 generated text (peach). Precise settings
Overall, these results highlight the utility of conditional like- lihood estimation for model auditing, with natural extensions to downstream tasks such as membership inference.
are in Appendix D.5
Conclusion
We introduced information-theorete framework for discrete diffusion, formalized through the I- MDSE and I-MDCE relations that connect information decay to acore-based training losses and yield exact ligelikelihood decompositions. This framework offers a principled justification for learning with DSE or DCE objectives and enables practical low-variance likelihood estimation through time-free free formulation. We hope this work advances the understanding of the theoretical foundations of discrete diffusion and inspires further exploration of principled estimators in generative modeling.
10

---

Acknowledgments
This work was supported in part by Institute Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2024-0045788,, Al Research Hub Project), IITP grant funded by the Korean Government (MSIT) (No. RS R1-2020--1,01361, Artificial Intelligence Graduate School Program (Yonsei University)), and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS RS- 202522252235549
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
William Anderson. Contimuous- Time Markov Chains: An Applications-Oriented Approach. Springer Series in Statistics. Springer, 2012.
Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. In NeurlPS, 2021.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In AAAI, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. In NeurlPS, 2020.
Andrew Camphell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. In NeurIPS, 2022.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. In ICLR, 2021.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint arXiv:1803.0000
2018
Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association, 2011.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Dongning Guo, Shlomo Shamai, and Sergio Verdú. Mutual information and minimum mean-square error in gaussian channels. IEEE Iransactions on information theory, 2005.
Floyd B Hanson. Applied stochastic processes and control for hump-diffusion: modeling, analysis and computation. SIAM, 2007.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. In NewrIPS, 2021.
Emiel Hoogeboom, Alexey A. Gritsenko. Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. In ICLR, 2022.
Aapo Hyvärinen and Peter Dayan, Estimation of non-normalized statistical models by score matching. JMLR, 2005.

---

Jiaming Ji. Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of Ilm via humannpreference dataset. In NeurIPS, 2023.
F. Kelly. Reversibility and Stochastic Networks. Cambridge University Press, 1980.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In NeurIPS, 2021.
Xianghao Kong, Rob Brekelmans, and Greg Ver Steeg. Information-theoretie diffusion. ICLR. 2023.
Xianghao Kong, Ollie Liu, Han Li, Dani Yogatama, and Greg Ver Steeg, Interpretable diffusion via information decomposition. In ICLR, 2024.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In ICLR, 2021.
Christian Léonard. Some properties of path measures. Séminaire de Probabilités XLVI, 2014.
Xiang Li, John Thickstun, Ishaan Gulrajani, Percy $ Liang, and Tatsunori Hashimoto. Diffusion-Im improves controllable text generation. In NeurlPS, 2022.
Iyya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In ICML, 2024.
Matt Mahoney. The text8 dataset.
htt:://atttt
2011.
Chenlin Meng. Kristy Choi, Jiaming Song, and Stefano Ermon. Concrete score matching: Generalized score matching for discrete data. In NeurIPS, 2022.
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji- Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint arXiv:2502.09992, 2025.
Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. In ICLR, 2025.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre- pre-training. OpenAl blog, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models unsupervised multitask learners. OpenAl blog, 2019.
Chitwan Saharia, William Chan, Saurabh Saxena. Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In NeurlPS, 2022.
Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. In NeurIPS, 2024.
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and generalized masked diffusion for discrete data. In NeurlPS, 2024.
Andy Shih, Dorsa Sadigh, and Stefano Ermon. Training and inference on any-order autoregressive models the right way. In NeurIPS, 2022.
Jascha Sohl-Dickstein. Eric Weiss, Niru Maheswaranathan. and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In JCML, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In NewrIPS, 2019.
12

---

Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. NeurIPS, 2021a.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021b. lb.
Haoran Sun, Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based continuous-time discrete diffusion models. ICLR, 2023.
Benigno Uria, lain Murray, and Hugo Larochelle. A deep and tractable density estimator. In ICMI., 2014.
Kartik Venkat and Tsachy Weissman. Pointwise relations between information and estimation in gaussian noise. IEEE transactions on information theory. 2012.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural compula- lion, 2011.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In ACL, 2019.
Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chunwei Wu, Jun Hu, Jun Zhou, Jianfei Chen, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Llada 1.5: Variance-reduced preference optimization for large language diffusion models, arXiv preprint arXiv:2505. 192233 2025.
13

---

A Related Works
Time-Free Likelihood Estimators. Time-free estimators similar to ours (Egs.
115 and (17) have
appeared in prior work, including et al. 2025) (Eq. (C.20) and Nnd al. (2023 (Eqs. (6) and (14)).
These works reformulated the variational bound LDCE (Xo) expectation over the number of
masked tokens. In contrast, our derivation establishes this identity as an exact equality, not just
bound, and provides quantitative comparison showing reduced variance relative time-integral
estimators.
Ou et al. (2025) further showed that the DCE loss matches the training objective of any-order autoregressive (AO-AR) models (Uria et al. 2014 Hoogeboom et al. 2022: Shih et al. 2022);
(xo(Xo) Ex
log
p t(1)
X ((()))
where the expectation is taken uniformly over all permutations of {T...., L). While this equiva- lence helps explain the bidirectional behavior of masked diffusion models (Nie et al., (025). computationally expensive for likelihood estimation, requiring L forward passes per MC sample. In contrast, our estimator achieves the same theoretical objective using just one forward pass per sample, providing significantly more efficient solution.
B Discussion and Limitations
Conceptual Intuition. Although DSE/DCE and MSE originate from distinct geometries, logarith- mic versus Euclidean, their connection emerges through the principle of distribution matching information theory. Just as Gaussian distributions align naturally with 6² (MSE) loss and Laplacian distributions with e (MAE) loss, categorical distributions align with logarithmic loss, which underlies DCE and, by extension, DSE in the masked diffusion setting. From this perspective, DSE and DCE serve as the natural discrete analogs of MSE, minimizing the expected divergence between predicted and true categorical distributions. This explains why the I-MDSE and I-MDCE identities carry over the information-theoon validity of their continuous domann counterparts.
Limitations. Our framework currently applies only to masked diffusion models through the I- MDCE relation, leaving its extension to the full I-MDSE setting for future work. Moreover, while the estimator improves interpretability and auditing, its ability to recover likelihoods may also expose sensitive information, requiring cautious deployment privacy-critical scenarios.
C Proofs of Theorems
C.1 Theorem 3.1 and Theorem
This proof is strongly inspired by Lou et al. 2024)'s 's derivation of the variational bound for the NLLI of the learned model - log ((
Let IP be the path measure for the diffusion process and Pa, be the marginalization starting from zo- Using the chain rule for KL divergence path measures (Léonard] 2014) twice (at the second and the fourth equality),
evaluate the negative log-liiel of the true distributions of data:
log Po (xo) DKI(8 Po)
(P || PP) Enggddn (P || ||]o(=]))))]]
(IP PP)
= DKI. (pr.o (=0)) (P)) + Epri(ze [KI. ft. (Pagg (-ITT) [P(()))))
The last term is computed by Dynkin's formula (Hanson 2007 Campbell et al. 2022 Lou al.) 2024), so we obtain Eq. (
log (ctoo) =
E Pepoo vx, 8 s7) di + DKr. (rrjoproo(-eee pr)
mdse(1ro t) + DKL. (prpo-((zo)))) PT).
(19)
14

---

Letting T + oc, we obtain Eq.
12
lo (2o) = J pc
mdsc(2o, t)dt.
Differentiating Eq.
19
with respect to T and replacing T with I, we obtain Eq. 9
DKL (pqo(((/rr))) pl):) = -mdse( t).
di
and taking the expectation, we obtain Eq. (10):
[zo: v.) = poo [mnds( ()] = mndse(().
dt
C.2
Lemma 33
Prjo(y|xo) p(jo(x(Xo)
(Pij(y)/x)) (xo)
XosE(xo,x,t,,5) x)
(x(x))
log sx(x)) + K
(x)o(((Xx))
11
1 > 1ly = 1og
1 d c(x)).n)
Σ
(t)
c(x))xx
A
gr M y=1
- A
1sy = 2tol [ogg
[--1 1ly = 2t01 -11 11)
+
a(t)( -A)
Σ
log
x
((x))
+ log
1- -- -1)
A
a' M
(tt((1 A)
log
A
c(x)),nt
x² #'=[M]
(t) A)
(XC, c).
A
C33
Theorem 3.4
Since dd dt = attee a(t) o1 - A), Lemma 3.3 becomes
((1)
(Ds x, t, st = (DCE(Xo, c)d..
Using the above equivalence in differential form directly, obtain
(xo) = 1/" ( xr, sd)
A / xx, edl
LoCE(xo).
15

---

C.4 Theorem 4.1
(og))(xo) = 1/- 1 Bxso(oss10)
1
log
dd
po x (
[1=|M
Σ (x)o(X)(Xo)
Σ
log
po(=fl><(1)) (iM) dA
X
[
x1-111 - A) Σ
log
dd
Po (x6)
[9[4]
E
X - -)) da
Σ
log
14
Σ [
BLL-
Σ
log
i
To express the last formula in the expectation form, calculate the sum of the weights B(L ((L-
4 B(L - i,i+
Σ B(L B(-
Σ
[156]
L.1 (I. 1.- 1
il il - i)!
H
C.5 Theorem 4.2
In this subsection, we introduce two lemmas that directly prove Theorem 4.2
The first lemma is quite straightforward, which is obtained by applying the diffusion process only on the indices in a nonempty subset I of I = {1,2, ..., L).
Lemma C.1. Let be a nonempty subset of I {1, 21.2.... L} and x' = (xx)) be the indexed
subsequence x E XT. Then
- llog ll (x6) = S.' 1
A Epsio(x3
Σ 1[2 = MM] log LieT
dA.
po (x(x
The second lemma is obtained by regarding the data distribution with arbitrary conditioning. Lemma C.2. Under any condition y, the negative log-likelihood is computed as
-logp(xulv) ( Ep Ex (xo) X
Σ =
M log
dx.
p UM 1/)
Proof. We consider the diffusion process starting from the distribution q(x p(xolv) with the same noising processes {oxooossaT as the unconditional case. Then by Eq.
13
log q(xo)= [/'
1 A Epsio (xca)
I.
1 ((2))
Σ
| = [M] log
dA.
(20)
UM
16

---

Observe that
((xxa)) (x) (an x x
= x(x (y)
1 pla xx (M y)
p( 11)) (v)
Substituting this result into Eq. 20 completes the proof.
D
Experiment Details
D.1 Computational Resource Details
All experiments were conducted on a computing node equipped with 8 NVIDIA L40S GPUs, TB of system memory, and 192 CPU cores. We used single GPU.
D.2
Details on Toy Experiments
1.5
NLL.
4.0
5.0
40
3.0
Figure 4: Results of unconditional NLL estimation 128 DNA sequences. Estimated and true NLLs are closely aligned, supporting the effectiveness of estimation via Eq. (15).
We provide detailed experimental settings for Section 5.1 which evaluates the reliability of I-MDCE on synthetic data with an explicitly defined ground-truth distribution.
Datasets. For the unconditional NLL estimation task, we generate 128 unique DNA sequences of length using the alphabet {A, T, G, C}. Each sequence is assigned probability using softmax over uniformly sampled scores from j0, 1), scaled by a temperature of 0.5. These probabilities define categorical distribution over the 128 sequences, from which one million training samples drawn.
For the conditional NI.I estimation task, we generate sequences using 4-th order Markov model over the same DNA alphabet. For each 4-base context, the conditional distribution over the next base is defined by the same softmax procedure applied to independently sampled scores. This results in valid probabilistic transition table that governs sequence generation. The model is trained on a continuous DNA sequence of total length five million. For NLL evaluation, each subsequence of length 32 is split into a 16-base prompt prompt and 16-base response x response.
Training Details. We use the Adam W optimizer (Loshchilov and Hutter 2019) and RADD (Ou al. 2025) in all experiments. the unconditional setting, the model trained for 70,000 steps with learning rate of 3 x 10-4 and batch size of 512. In the conditional setting, training is performed for 80,000 steps with a learning rate of 6 x 10 and a batch size of 1,024.
NLL Evaluation Protocol. When computing both conditional and unconditional NLL, we use 215 Monte Carlo samples estimate each case. Since this toy experiment is designed closely align with the true data distribution, large number of samples is used for accuracy. In general settings, however, 100 Monte Carlo samples are typically sufficient to evaluate relative differences NLL. Full results are in Figs. to
D.3 Details on Variance Reduction Experiments
We provide additional details for the variance analysis experiments described in Section 5.2
17

---

Conditional Likelihood Estimation. The results in Table la are based on 30 randomly sampled sequences from each of the following datasets: HellaSwag (Zelers al. 2019), ARC-hard (Clark) et al. 2018), and PIQA (Bisk et al. 2020). For each sequence, we compute 15 independent Monte Carlo estimates of the conditional oogliikld and report the variance averaged over the 30 samples.
pppmmpps xpromps), ), we format the prompt as To ensure sufficient structure for conditional estimation po Po (xtssoone)
the question and the response as:
Correct: [correct answer]
Incorrect:
[incorrect answer].
Likelihood Ratio Estimation. evaluate the variance the coupled likelihood ratio estimator in
Eq. [8)) we construct dataset based on the PKU- Wellinnment/BeaverrTti is corpus (Ji et al. 2023)
Each instance is triplet (xprmmpe, x response, + x ressponse. where xnrmpp is prompt and x responsse
response, are safe and unsafe responses, respectively. Specifically, we estimate the ratio of conditional
log-likelihoods between safe and unsafe responses using the following variant of Eq. (8)
Po response, + (xprompe)
Po response, t.d trmomm .x tesponse, J
log Po response Xpromps)
Bp())
Σ JEN
log
Xprompp xeesponse.))
Po X response
where / denotes the index set corresponding to the response tokens and J S I is sampled from the distribution defined in Corollary |.3] We select prompts with at least one safe and one unsafe reply, and enumerate all valid safe-unsafe response pairs per prompt to generate suitable triplets. The final dataset consists of approximately 500 triplets, formatted in JSONL with the fields:
{"xproupt"; prompt, "xreeponee.t" safe, "xreeponse unsafe).
D.4 Training and Evaluation Details for Out-of-Distribution Detection
Training Details, For the OOD detection task, we train the model using a contiguous subset of the text8 corpus (Mahoney 2011). Each input sequence consists of 256 tokens, with the first 128 tokens serving as the conditional input xrrmmpt and the remaining 128 tokens as the continuation x response We train RADD using the AdamW optimizer with learning rate of 10 10-4, The model is trained for 7,500 steps with a batch size of 32.
NLL Evaluation Protocol. For evaluation, we construct two groups of (xprompt respoonse) pairs: (1) in-distribution continuations, where reeoonse is the true continuation of pommpt from the held-out test split of text8, and (2) -istt--dittiittiinn continuations, where response is generated by GPT-4 given the same prompt All evaluation sequences are disjoint from the training data. For each group, we sample 500 examples. We then estimate NI.L via Eq. (17) with 100 Monte Carlo samples, and report the distribution for each group.
D.5 Evaluating NLL on Pretrained Language Model
We evaluate the effectiveness of the conditional estimator (Eq. (17)) on a pre-trained open-source model. Specifically. we mode[] Two datasets are used for
prozain. LLaMA geneated
use the evaluation: WikiText (English) and pretrain_ (Chinese). During NLL estimation, 100 MC samples are used, Results for pretrain zh is in Fig. 7
0,033
Evaluation Dataset. Each dataset preprocessed into (xPrompt xespoose) pairs, with both segments containing 64 tokens. For WikiText, we use the training split of Eikitext-22-raw
0.000
400
S00
Figure 7: Estimated conditional NLL on pretrain_at zh (blue) and LLaMA 3.1 generated text (yellow). shows similar behavior to Fig.
while for pretrain_zh, we concate- nate the first 3,000 documents. In both cases, the data tok- enized into 128-token blocks and evenly split into prompt and response. For each prompt prompt we also generate synthetic response reponse using the LLaMA 3.1 model.
8B- Instruct
18

---

30
True NLL
25
Estimated NLLL
20-
15
KD 510
000 E1
600
AMd
170
050
000
True NLJ.
Estimated NLL
10
150 A40
244
A6A
400
30
25
Estimated NLL
CAA
30
True NLL
25
Estimated NL.L
20
15
000 O0
M0T (O)
100
24
The
Rie ond
400
THe CO
110
TO
MP
DC
Figure 5: Conditional NLL estimation Markov DNA sequences. Estimated and true NLLs
closely aligned, supporting the effectiveness of the estimator in Eq. (17]
19

---

True NLL
25
Egtimated NLL
20
15
True NLJ
Esimmttdd NILL
10
30
True
25
Estimated
20
10
100
True NLL
25
Estimated NLL
20
15
Figure 6: Conditional NLL estimation Markov DNA sequences. Estimated and true NLLs
closely aligned, supporting the effectiveness the estimator in Eq. [17]
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Sections to
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Appendix B
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each thcoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
21

---

Justification:
Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: Yes)]
Justification: Section 3 and Appendix
D
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer: [Yes]
Justification: The paper provides sufficient details to reproduce the key experimental results. The code is publicly available https:///iithb..c http://hhhhb...///oo/oaaaaaa22ad
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm ec/
public/guides/ http://ooooeeuaasssssooooooooo
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.cc/ppp wwtpcccpppppppp.eeeeeeeeeeeeeeee wwwpc//gggeeeeeeee Sute://ddeeSbbaaaaaae wwwteeSubmisiiiooo issiiiiliiccyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 5land Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
23

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are of range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: This work proposes theoretical framework and empirical validation on synthetic and publicly available datasets. It does not involve sensitive data, human subjects, potentially harmful applications, and adheres to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Section 6
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release pretrained models or datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided relcasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All datasets and models used (e.g.. text8, WikiText, LLaDA) are publicly available and used under appropriate licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, datasets.paaersstt paperswi thcode. cww/ datasets
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No datasets models released.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No human subjects were involved.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Not applicable as our work does not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
26

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: GPT-4 was used only to generate test completions in one experiment. It did not influence the core methodology of this research.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
27


Reviews:
[{"review_id": "sas54Dyx0F", "reviewer": "Reviewer_vmxL", "summary": "This paper develops an information-theoretic framework for discrete diffusion models, analogous to the I-MMSE identity in Gaussian settings. Specifically, an I-MDSE identity is derived for discrete diffusion, linking the rate of KL divergence decay to the minimum denoising score-entropy loss. The framework is further extended to masked diffusion models, leading to the I-MDCE relation. Both time-integral and time-free expressions for NLL estimation in conditional and unconditional scenarios are proposed. Experiments on NLL estimation demonstrate the effectiveness of the proposed method.", "questions": "The derived I-MDSE and I-MDCE identities link mutual information to score-based losses. What are the practical benefits and how does these relations bring new insights?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper is well-structured and the idea of connecting mutual information with score-based losses in discrete diffusion models is interesting. This paper generalizes continuous‑domain I‑MMSE results to discrete domain, offering a new angle to understand discrete diffusion models.\n- This paper provides both time-integral and time-free NLL estimators, which are validated by experimental results.\n\nWeaknesses:\n- NLL decomposition is achieved via both I-MDSE and I-MDCE relations (Theorem 3.2 and Corollary 3.7), which require the optimal score functions. However, as the optimal score is unavailable, the learned score network is used for approximation, leading to a trivial likelihood estimation via loss function, which is a variational upper bound for log-likelihood. Therefore, the introduction of time-integral NLL estimators seems to be trivial.\n- The experimental part lacks comparison. The introduction of the time-free likelihood estimation (Theorem 3.8) is novel. However, the experimental results only show its effectives without comparison to other methods, which is not enough to demonstrate the advantages of time-free estimator as claimed in line 230-232.", "was_revised": true, "final_justification": "I thank the authors for the detailed response. Most of my concerns are addressed and I will raise my rating accordingly.", "rebuttal": "> **W1. NLL decomposition is achieved via both I-MDSE and I-MDCE relations (Theorem 3.2 and Corollary 3.7), which require the optimal score functions. However, as the optimal score is unavailable, the learned score network is used for approximation, leading to a trivial likelihood estimation via loss function, which is a variational upper bound for log-likelihood. Therefore, the introduction of time-integral NLL estimators seems to be trivial.**\n\n\nWe thank the reviewer for raising this important point regarding the role of learned score functions in our NLL decomposition via I-MDSE and I-MDCE.\n\nIt is indeed correct that the optimal score function is not directly accessible and must be approximated by a learned model, which introduces a variational aspect into likelihood estimation. However, we respectfully disagree with the characterization that this makes our decomposition or resulting estimators trivial.\n\nThe key novelty lies in the fact that our formulation yields an exact equality, not just an upper bound, between the negative log-likelihood and integrals of score-based losses. As noted by reviewer pwj1, this equality result is nontrivial and significant:\n\n*\"Even though the result is functionally the same as the variational bound, I like the point that Eq. 11 shows that it ‘fully captures the log-likelihood’. To me, this is quite important as it shows that it is not possible/necessary to tighten the variational bound by outputting higher order moments.\"*\n\nWe believe this exactness is a core strength of our framework. It shows that first-order score functions alone are sufficient to fully characterize the log-likelihood, thereby removing the need for additional assumptions or higher-order statistics often considered in variational estimation. This theoretical clarity also distinguishes our approach from typical score-matching or ELBO-based approximations.\n\nMore importantly, this equality is not just of theoretical interest, it enables new practical applications. In particular, we leverage it to construct a unified and low-variance estimator for log-likelihood ratios, as detailed in Section 3. Unlike naive subtraction of two upper bounds (which may not yield a valid or stable estimate), our equality-based formulation supports a direct Monte Carlo estimator with provable correctness and reduced variance, as shown in our experiments. This estimator is particularly useful in tasks like model comparison, importance weighting, and membership inference, and we discuss its variance advantage in our response to W2.\n\nFinally, our equality result offers a principled foundation for empirical practices already in use. For example, LLaDA 1.5 employs a loss function based on the ratio of conditional likelihoods to supervise model alignment. Our formulation formally justifies this practice by showing that such ratios can be reliably estimated from learned first-order score functions, even without access to the true data distribution.\n\nIn summary, while our estimator resides within the variational paradigm, the exactness, extensibility, and low-variance properties afforded by the I-MDSE/I-MDCE decompositions represent a nontrivial and practically impactful advancement beyond prior bounds.\n\n> **W2.The experimental part lacks comparison. The introduction of the time-free likelihood estimation (Theorem 3.8) is novel. However, the experimental results only show its effectives without comparison to other methods, which is not enough to demonstrate the advantages of time-free estimator as claimed in line 230-232.**\n\nWe thank the reviewer for pointing out the importance of comparative evaluation for the time-free likelihood estimator presented in Theorem 3.8.\n\nWe would first like to clarify that a structurally related estimator was independently introduced in the LLaDA paper [1], which we were not fully aware of at the time of submission. While LLaDA focuses on providing an upper bound on the log-likelihood (without proving equality), it reformulates the original RADD estimator as an expectation over the number of masked tokens, a formulation that closely resembles our time-free estimator. As noted in LLaDA:\n\n*\"Eq. (12) (RADD formulation) requires over 1000 Monte Carlo estimates for stable results, whereas Eq. (14) (LLaDA’s reformulation) achieves stability with only 128 estimates.\"*\n\nThis observation motivated their reformulation, and we experimentally confirm the same behavior in our setting. Specifically, we conducted a direct comparison between our time-free estimator (Eq. 14) and the time-integral form (Eq. 13) by measuring the Monte Carlo (MC) variance of conditional log-likelihood estimates on 30 sequences with 15 samples each from HellaSwag [6], ARC_hard [7], and PIQA [8] using the LLaDA model. As shown in Table, our time-free estimator consistently achieves lower variance, empirically validating its improved robustness and sample efficiency.\n\n### Variance ↓ of Estimator (HellaSwag, 30 subset)\n| #MC sample | Time-integral | Time-free |\n | ----- | ----- | ----- |\n| 128 | 70.9716 | 11.5658 |\n| 256 | 30.1938 | 6.0150 |\n| 512 | 13.3796 | 2.9170 |\n---\n### Variance ↓ of Estimator (ARC_hard, 30 subset)\n| #MC sample | Time-integral | Time-free |\n | ----- | ----- | ----- |\n| 128 | 23.1758 | 5.7318 |\n| 256 | 18.1444 | 2.9612 |\n| 512 | 9.5014 | 1.8200 |\n---\n### Variance ↓ of Estimator (Piqa, 30 subset)\n| #MC sample | Time-integral | Time-free |\n | ----- | ----- | ----- |\n| 128 | 19.76875 | 4.92825 |\n| 256 | 15.14975 | 1.80950 |\n| 512 | 6.50375 | 1.22425 |\n\n\nAdditionally, we emphasize a key advantage of our formulation: it provides an explicit distribution over the mask variable $J$, which naturally enables unified (coupled) likelihood ratio estimation, a capability not addressed by LLaDA. Likelihood ratio estimation is central to many downstream tasks in large language models (LLMs). For example:\n- In Direct Preference Optimization (DPO) [2,3], the core loss is based on the log-likelihood ratio between preferred and dispreferred completions: $\\log\\frac{p^\\theta(y^+|x)}{p^\\theta(y^-|x)}$\n- In out-of-distribution (OOD) detection [4,5], likelihood ratios between in-distribution and out-of-distribution samples are used as a decision metric.\n\n\nIn these settings, prior approaches typically compute two independent likelihood estimates and subtract them, a process prone to compounded variance. In contrast, our time-free estimator enables a direct, unified estimation of the ratio through coupled Monte Carlo sampling:\n\n$$\n\\log\\frac{p(y^+|x)}{p(y^-|x)} = H\\_L \\mathbb{E}\\_J \\left[ \\sum_{i\\in I\\setminus J}\\log\\frac{p(y^{+,i}|y^{+,J},x)}{p(y^{-,i}|y^{-,J},x)} \\right]\n$$\nwhere $I$ denotes the index set of response tokens and $L=|I|$ The distribution over $J$ follows the sampling scheme described in Corollary 3.10 of our paper.\n\nWe validate this through a likelihood ratio estimation experiment on the BeaverTails dataset [9], evaluating 500 $(x, y^+, y^-)$ triples. Our coupled estimator yields substantially lower variance compared to the standard decoupled approach.\n\n### Variance ↓ of Log-Likelihood Ratio Estimation (BeaverTails, 500 triples)\n|#MC | Coupled       | Decoupled      |\n|-------|--------------|--------------|\n| 5     | 8897.08 | 62469.41 |\n| 10     | 4487.38 | 29107.21 |\n| 15     | 3059.97 | 20695.61 |\n| 20     | 2335.12 | 16514.72 |\n\n\nThese findings underscore a practical and underappreciated benefit of our method: it supports efficient, low-variance ratio estimation, which is critical for alignment evaluation and related tasks. In fact, we believe that LLaDA 1.5’s alignment mechanism, which involves estimating conditional likelihood ratios, could be made more efficient by replacing its current two-step estimation with a unified (coupled) VRPO loss, leveraging our formulation.\n\n> **Q1. The derived I-MDSE and I-MDCE identities link mutual information to score-based losses. What are the practical benefits and how does these relations bring new insights?**\n\nThe I-MDSE and I-MDCE identities offer a key conceptual insight by establishing an equality, rather than just a bound, between the log-likelihood and integrals of score-based losses. This result demonstrates that first-order score functions are sufficient to fully characterize the likelihood, providing a principled and interpretable foundation for learning in discrete diffusion models.\n\nOn the practical side, these identities enable a time-free formulation of both log-likelihood and log-likelihood ratio estimators. Compared to time-integrated approaches like RADD, our formulation yields significantly lower variance, which is especially beneficial in downstream tasks such as alignment evaluation and out-of-distribution detection. Moreover, it supports coupled Monte Carlo estimation of likelihood ratios, improving both stability and efficiency over decoupled approaches.\n\nThese advantages are empirically validated in our experiments, as shown in the tables above, where we demonstrate robust performance and reduced variance across multiple tasks and datasets.\n\n[1] Shen Nie et al., Large language diffusion models. arXiv 2025.\\\n[2] Rafael Rafailov et al., Direct preference optimization: Your language model is secretly a reward model. NeurIPS, 2023.\\\n[3] Zhu et al., LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models, arXiv, 2025.\\\n[4] Ren et al., Likelihood ratios for out-of-distribution detection. NeurIPS 2019.\\\n[5] Andi Zhang et al.,. Your Finetuned Large Language Model is Already a Powerful Out-of-distribution Detector. AISTATS, 2025.\\\n[6] Rowan Zellers et al., HellaSwag: Can a Machine Really Finish Your Sentence? ACL, 2019.\\\n[7] Peter Clark et al., Think you have solved question answering? Try ARC, the AI2 Reasoning Challenge. arXiv, 2018.\\\n[8] Yonatan Bisk et al., PIQA: Reasoning about Physical Commonsense in Natural Language. AAAI, 2020.\\\n[9] Yixin Nie et al., BeaverTails: Evaluating Language Models by Simulating Human Preference Data. arxiv, 2024."}, {"review_id": "BJkhZH89WU", "reviewer": "Reviewer_pwj1", "summary": "I-MMSE relations in information theory establish exact connections between density estimation and denoising in a Gaussian channel. \nThis paper generalizes these ideas to consider discrete random variables and discrete noise channels (like masking), demonstrating the existence of exact relations between probability and optimal denoising in these discrete channels, using the correct objective. \nDenoising with discrete noise channels has become popular recently in the discrete diffusion literature. This paper establishes that variational objectives that were previously understood as bounds on probability are actually tight. \nThe authors use their results to express probability estimates in different forms, and apply these estimates to synthetic and real-world examples.", "questions": "My main question is related to the discussed weaknesses. Is there anything about your NLL estimator that makes it significantly different in implementation or performance compared to existing methods, and are there any experiments that demonstrate this?", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "related): \n## Strengths: \n\n- I thought the summary of discrete diffusion and I-MMSE connections were nicely presented. \n- Even though the result is functionally the same as the variational bound, I like the point that Eq. 11 shows that it \"fully captures the log-likelihood\".  To me, this is quite important as it shows that it is not possible/necessary to tighten the variational bound by outputting higher order moments. \n- I appreciate the effort to make synthetic experiments with known ground truth that are somewhat interesting (DNA-like, higher order Markov). \n\n\n## Weaknesses: \n\n### Significance of time-free formulation\n\n> In contrast, we derive a surprising result: a time-free expression for the NLL based solely on randomly selected masked positions. To the best of our knowledge, such a time-free formulation has not been established in any diffusion model, whether discrete or continuous.\n\n$$ - \\log p_0\\left( \\mathbf{x}_0^{I_1} \\mid \\mathbf{x}_0^{I_2} \\right) = H_{|I_1|} \\, \\mathbb{E}_{p(J)} \\left[ \\sum_{i \\in I_1 \\setminus J} \\log \\frac{1}{p_0\\left(x^i \\mid \\mathbf{x}_0^{J \\cup I_2} \\right)} \\right] $$ \n\nThis seems like an over-statement, or needs more context about what the surprising new claim is. Many papers have recognized that NLL can be written in terms of random order autoregression. \n\n- E. Hoogeboom, A. A. Gritsenko, J. Bastings, B. Poole, R. v. d. Berg, and T. Salimans,\n“Autoregressive diffusion models,” arXiv preprint arXiv:2110.02037, 2021.\n- A. Pannatier, E. Courdier, and F. Fleuret, “σ-gpts: A new approach to autoregressive models,”\nin Joint European Conference on Machine Learning and Knowledge Discovery in Databases,\npp. 143–159, Springer, 2024.\n- Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le, “Xlnet: Generalized\nautoregressive pretraining for language understanding,” Advances in neural information\nprocessing systems, vol. 32, 2019.\n\n\nThis paper explicitly is about how time-dependence for masked diffusion models is not needed. \n\n- K. Zheng, Y. Chen, H. Mao, M.-Y. Liu, J. Zhu, and Q. Zhang, “Masked diffusion models are\nsecretly time-agnostic masked models and exploit inaccurate categorical sampling,” arXiv\npreprint arXiv:2409.02908, 2024.\n\nYou also cite RADD, which says something similar in Eq. 3.8. Maybe the difference between your statement and RADD is that they sample the masks based on sampling a continuous \"time variable\", while you sample directly from a discrete mask distribution? \nEven so, the regular any-order AR decomposition is \"time-free\" in the same way. Given a random permutation, $i_1, ...,i_T = \\pi(1, ... T)$\n\n$$-\\log p(x_{i_T:i_k} \\mid x_{i_{k-1}:i_1}) =   \\sum_{j =k}^T \\log \\frac{1}{p\\left(x_{i_j} \\mid x_{i_{j-1}:i_1} \\right)}$$\n\nBy averaging over random orders, you can get the same result as yours - arguably more general, as we could weight this to allow for different mask distributions.\n\n### Experimental results missing useful comparisons\n\nMy main concern is that the experimental section is not very useful and may even be misleading. \nWe already have many ways to estimate the NLL, using cross entropy with random order autoregressive and masked models and with variational bounds. \nWhat would be useful to know is whether the results in this paper help us to do this easier or better. \n\nThe reason it seems misleading is that your estimator may be giving identical results to existing estimators. But because baselines are not presented, it seems as if these tasks were enabled by your results. \n\nTypes of baselines for bounding NLL I would have expected to see: \n- Random order Autoregressive (one sample, or averaged over orders) \n- Variational bounds like DSE\n\nTypes of results I would have expected to see: \n- Bias and variance of various estimators \n- Computational cost, if relevant\n\nThe random tasks like OOD are neat demos, but I'd expect any of the existing NLL bounds would be just as good for these.", "was_revised": true, "final_justification": "Several reviewers found the experiments to be limited, because of lack of comparisons to other methods. The authors discussed comparisons and gave some more variance estimates in the rebuttal. I raised my score to borderline accept, but I do still consider this a borderline submission with the current evaluation.", "rebuttal": "> **W1. The novelty and practical benefit of the proposed time-free NLL formulation is unclear.**\n\nWe appreciate the reviewer’s question regarding the novelty and practical benefit of our proposed time-free NLL formulation, especially in relation to RADD, LLaDA, and any-order autoregressive models.\n\nFirst, we clarify that our time-free formulation differs from RADD, which expresses the NLL as an expectation over a continuous time variable $\\lambda$. In contrast, our estimator is derived over a discrete mask distribution over $J$, removing the need for time-based integration. This reparameterization contributes to reduced variance, as previously hinted at in follow-up works.\nThat said, we acknowledge that our original claim overstated the novelty. The LLaDA paper Eq.(14) [1] introduces a formulation that is closely related to ours: it reinterprets the RADD estimator as an expectation over the number of masked tokens ($\\ell$), rather than time. While LLaDA focused only on upper bounds and did not prove equality, the resulting structure is indeed similar to our time-free form. LLaDA briefly notes that this formulation empirically lowers variance:\n\n*\"Eq. (12) (RADD formulation) requires over 1000 Monte Carlo estimates for stable results, whereas Eq. (14) (LLaDA’s reformulation) achieves stability with only 128 estimates.\"*\n\nBuilding on that, we provide a more rigorous experimental evaluation and demonstrate that our time-free estimator achieves significantly lower variance, as shown in the table below. In the revised version of the paper, we will explicitly clarify the connection and differences between RADD, LLaDA, and our work.\n\nWe conducted a direct comparison between our time-free estimator (Eq. (14)) and the time-integral estimator (Eq. (13)) to further assess the practical utility of the former. Specifically, we measured the Monte Carlo (MC) variance of conditional log-likelihood estimates on 30 sequences with 15 samples each from HellaSwag[6], ARC_hard[7], and PIQA[8] using the LLaDA model. As shown below, the time-free estimator consistently exhibits lower variance:\n\n### Variance ↓ of Estimator (HellaSwag, 30 subset)\n| #MC sample | Time-integral | Time-free |\n | ----- | ----- | ----- |\n| 128 | 70.9716 | 11.5658 |\n| 256 | 30.1938 | 6.0150 |\n| 512 | 13.3796 | 2.9170 |\n---\n### Variance ↓ of Estimator (ARC_hard, 30 subset)\n| #MC sample | Time-integral | Time-free |\n | ----- | ----- | ----- |\n| 128 | 23.1758 | 5.7318 |\n| 256 | 18.1444 | 2.9612 |\n| 512 | 9.5014 | 1.8200 |\n---\n### Variance ↓ of Estimator (Piqa, 30 subset)\n| #MC sample | Time-integral | Time-free |\n | ----- | ----- | ----- |\n| 128 | 19.76875 | 4.92825 |\n| 256 | 15.14975 | 1.80950 |\n| 512 | 6.50375 | 1.22425 |\n\nThese findings empirically validate the variance-reduction benefit of our formulation and align with observations made in LLaDA.\n\nWe also agree that our formulation is conceptually related to any-order autoregressive (AR) decompositions, such as those in XLNet and $\\sigma$-GPTs. This connection is noted in LLaDA, and we will explicitly clarify this relationship in the revised manuscript. However, we emphasize an important distinction: any-order AR estimation requires a full permutation of length $L$ and evaluation of conditional probabilities for each position, \n\n$$p\\left( X\\_{\\sigma_T} = x\\_{\\sigma_T} \\mid X_{\\sigma_1} = x_{\\sigma_1}, \\ldots, X_{\\sigma_{T-1}} = x_{\\sigma_{T-1}} \\right)$$\n\nresulting in $L$ forward passes for sequences of length $L$.\n\n\n In contrast, our method supports per-sample forward computation, only one forward pass per Monte Carlo sample, and remains stable even with fewer samples than sequence length (e.g., 128 samples for a sequence of length 190).\n\nWe appreciate the reviewer for pointing out these valuable connections, and we will revise the manuscript to more clearly articulate the positioning, distinctions, and practical contributions of our approach relative to these related works.\n\n> **W2. Concern about missing baseline comparisons, how the proposed estimator compares to existing methods, and whether it offers practical improvements.**\n\nWe appreciate the reviewer’s concern regarding the absence of baseline comparisons. To address this, we have added experiments comparing our estimator to RADD. The results (see table below) demonstrate that our time-free estimator achieves significantly lower variance, highlighting its robustness and sample efficiency. (See W1)\n\nThese results also suggest a potential limitation when applying AO-AR estimators to tasks with longer responses. For example, in our experiment with HellaSwag, where the average response length is 228 tokens, a single MC sample under AO-AR requires roughly 228 forward passes, meaning that a 256-NFE budget yields only about 1 MC sample. While AO-AR remains a principled objective, this sampling cost may make it less practical in settings where efficient estimation is needed.\n\nImportantly, our method provides an exact equality, not just an upper bound, for the log-likelihood. This distinction is crucial: because the formulation holds as an equality, it allows us to construct a direct estimator for the log-likelihood ratio, a task that is not straightforward when using variational upper bounds. Likelihood ratio estimation plays a central role in large language models (LLMs), with numerous downstream applications. For example, Direct Preference Optimization (DPO) [2, 3], a prominent alignment algorithm for LLMs, directly optimizes a loss based on the log-likelihood ratio between preferred and dispreferred outputs. More specifically, given a prompt $x$ with a preferred output $y^+$ and a dispreferred output $y^-$, the DPO loss optimizes an alignment score of the following form:\n$$\n\\log\\frac{p^\\theta(y^+|x)}{p^\\theta(y^-|x)}\n$$\n\n Furthermore, in out-of-distribution (OOD) detection [4, 5], the likelihood ratio between in-distribution and out-of-distribution samples serves as a fundamental decision criterion.\n\nThis perspective also offers a theoretical justification for recent empirical practices. For instance, LLaDA 1.5 employs a loss function based on the ratio of conditional likelihoods, interpreted as an alignment score, to supervise model alignment. Our equality result provides a principled foundation for this choice, as it establishes that such likelihood ratios can be reliably estimated from first-order score functions without requiring access to the true data distribution. \n\nRather than computing two separate likelihood estimates and subtracting them, which can lead to compounded variance, our time-free formulation naturally enables a direct and unified estimator for the log-likelihood ratio, providing improved stability and efficiency:\n\n$$\n\\log\\frac{p(y^+|x)}{p(y^-|x)} = H\\_L \\mathbb{E}\\_J \\left[ \\sum_{i\\in I\\setminus J}\\log\\frac{p(y^{+,i}|y^{+,J},x)}{p(y^{-,i}|y^{-,J},x)} \\right]\n$$\nwhere $I$ denotes the index set of response tokens and $L=|I|$ The distribution over $J$ follows the sampling scheme described in Corollary 3.10 of our paper.\n\nThe effectiveness of our unified estimator is confirmed by our experimental results. We conducted a likelihood ratio estimation experiment on the BeaverTails[9] dataset, where our time-free estimator enables coupled Monte Carlo sampling. This reduces variance substantially compared to standard decoupled sampling. When computing log-likelihood ratios for 500 $(x, y^+, y^-)$ triples, the average variance of MC estimates is significantly lower for our approach:\n\n### Variance ↓ of Log-Likelihood Ratio Estimation (BeaverTails)\n|#MC  | Coupled       | Decoupled      |\n|-------|--------------|--------------|\n| 5     | 8897.08 | 62469.41 |\n| 10     | 4487.38 | 29107.21 |\n| 15     | 3059.97 | 20695.61 |\n| 20     | 2335.12 | 16514.72 |\n\nThese findings highlight an underappreciated advantage of our estimator: it supports efficient and low-variance ratio estimation, which is critical in downstream tasks such as alignment evaluation. We believe that this addresses the reviewer’s concern regarding empirical justification for the advantages of our time-free formulation.\n\nWe will update the manuscript to include these new comparisons and clarify the practical advantages of our approach.\n\n> **Q1. Is there anything about your NLL estimator that makes it significantly different in implementation or performance compared to existing methods**\n\nOur NLL estimator differs from existing methods in both formulation and performance. Unlike time-based estimators like RADD, our method uses a time-free formulation over discrete masks, leading to significantly lower variance, as demonstrated in our experiments. Compared to random-order autoregressive models, our estimator is more sample-efficient, requiring fewer forward passes than sequence-length-based permutations. Furthermore, because our estimator is derived from an equality (not a bound), it enables a direct and low-variance estimator for likelihood ratios, which is not possible with standard variational approaches. Experimental results support these advantages.\n\n[1] Shen Nie et al., Large language diffusion models. arXiv 2025.\\\n[2] Rafael Rafailov et al., Direct preference optimization: Your language model is secretly a reward model. NeurIPS, 2023.\\\n[3] Zhu et al., LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models, arXiv, 2025.\\\n[4] Ren et al., Likelihood ratios for out-of-distribution detection. NeurIPS 2019.\\\n[5] Andi Zhang et al., Your Finetuned Large Language Model is Already a Powerful Out-of-distribution Detector. AISTATS, 2025.\\\n[6] Rowan Zellers et al., HellaSwag: Can a Machine Really Finish Your Sentence? ACL, 2019.\\\n[7] Peter Clark et al., Think you have solved question answering? Try ARC, the AI2 Reasoning Challenge. arXiv, 2018.\\\n[8] Yonatan Bisk et al., PIQA: Reasoning about Physical Commonsense in Natural Language. AAAI, 2020.\\\n[9] Yixin Nie et al., BeaverTails: Evaluating Language Models by Simulating Human Preference Data. arxiv, 2024."}, {"review_id": "ybWzgu7uC5", "reviewer": "Reviewer_wh8N", "summary": "This paper proposes a framework introducing information-theoretic framework for discrete diffusion models, by using principled estimators of log-likelihood based on score-matching losses. Empirical results on synthetic and real-world data confirm that the proposed approach accurately estimates log-probabilities of discrete inputs.", "questions": "Could the authors provide baseline comparisons to better support the effectiveness of the proposed framework?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strength**\n\n* The submission proposed I-MDSE and I-MDCE that connect mutual information decay with score-based loss functions in discrete diffusion.\n\n* The author provides closed-form of negative log-likelihood time-integral and time-free NLL estimators  for general discrete diffusion and masked diffusion and extends the NLL estimation to the conditional setting for structured prediction tasks..\n\n**Weaknesses**\n\n* The experimental scale is limited, and the absence of comparisons with prior work (e.g. baseline comparison) makes it difficult to fully validate the strength of the proposed framework. For instance, compared to other similar approaches such as ItDPDM (Information-Theoretic Discrete Poisson Diffusion Model), which includes comprehensive real-world evaluations and baselines to support its claims, this submission lacks such empirical breadth.\n\n* This framework appears to be an extension of Information-Theoretic Diffusion, adapting its analogous results to discrete diffusion settings. However, without clear motivation, detailed explanations, or illustrative examples, it is difficult to assess whether this submission offers more than an incremental contribution.", "was_revised": true, "final_justification": "While a more detailed comparison against diverse baselines would have been helpful, I appreciate the authors' thoughtful clarifications and the additional experimental results provided during the rebuttal. In light of the strengths highlighted by other reviewers—particularly the use of discrete diffusion via I-MMSE connections—I have decided to raise my rating to align with their perspectives.", "rebuttal": "> **W1. The experimental evaluation lacks scale and baseline comparisons, making it difficult to validate the practical strength of the proposed framework compared to prior work like ItDPDM.**\n\nWe thank the reviewer for the thoughtful comment and for raising the important point regarding experimental scale and baseline comparisons.\n\nWe acknowledge that our empirical evaluation is more limited in scope compared to broader real-world studies such as the Information-Theoretic Discrete Poisson Diffusion Model (ItDPDM), which presents a novel noise process and includes extensive benchmarking. That work explores a valuable modeling direction.\n\nOur paper, however, pursues a different objective: rather than proposing a new diffusion model, we focus on extending information-theoretic tools (in particular, I-MMSE-style decomposition identities) to the discrete and masked diffusion setting, which is widely adopted in real-world applications such as LLaDA. Our main contribution lies in the derivation of exact equality-based decompositions (I-MDSE and I-MDCE) that enable principled estimation of log-likelihood and, critically, log-likelihood ratios.\n\nWhile the original submission emphasized the theoretical contributions and their use in conditional likelihood estimation, we agree that additional empirical validation is essential. In this rebuttal, we have added new experiments demonstrating two key advantages of our formulation:\n- Variance reduction of the time-free estimator compared to the time-integral baseline (RADD-style).\n- A unified estimator for likelihood ratios that achieves lower variance than naïve decoupled subtraction.\n\n\nThese new results directly support the practical strength of our approach and are further detailed in our response to Q1.\n\nWe will revise the paper to more clearly distinguish our estimator-centric, theory-driven focus from prior work centered on model design, such as ItDPDM, and to transparently acknowledge the current experimental scope while highlighting the newly added empirical support.\n\n\n> **W2. The work adapts known information-theoretic results to the discrete diffusion setting, but lacks clear motivation, making it hard to determine whether the contribution is truly substantive or merely incremental.**\n\nWe thank the reviewer for the comment and the opportunity to clarify the motivation and significance of our theoretical contribution. \n\nAt the core of our work are the I-MDSE (Theorem 3.1) and I-MDCE (Corollary 3.6) identities, which we consider to be the central theoretical contributions of the paper. These identities establish exact equalities between mutual information decay and score-based losses in discrete and masked diffusion processes. This equality-based structure mirrors a fundamental theme in classical information and estimation theory, for instance, the I-MMSE identity connects Gaussian noise with MSE loss, offering deep insight into the behavior of continuous diffusion models. Our work extends this paradigm to the categorical setting, showing that cross-entropy and score entropy losses serve as the natural analogs under discrete noise. \n\nWe believe these results are not only conceptually meaningful, but also practically impactful. In particular, the I-MDSE and I-MDCE identities underpins Theorem 3.2 and Corollary 3.7, which show that commonly used variational training losses are not merely loose bounds, but are in fact tight equalities in our framework. This reveals that first-order score functions are sufficient to fully capture the log-likelihood, placing a theoretical limit on the benefits of higher-order corrections. We view this as a substantial theoretical insight that reframes how variational objectives should be understood in discrete diffusion models.\n\nMoreover, this exactness directly enables practical extensions. Theorem 3.2 and Corollary 3.7 serve as the theoretical foundation for our low-variance, time-free log-likelihood estimator, introduced in the main paper, and further support our newly proposed likelihood ratio estimator, which is presented for the first time in this rebuttal. These estimators go beyond standard variational approximations. The ratio estimator, in particular, is empirically validated through new experiments (see Q1), demonstrating that the equality results not only offer theoretical elegance but also lead to robust and efficient tools for real-world tasks such as alignment, OOD detection, and membership inference.\n\nIn summary, our work represents a principled generalization of foundational information-theoretic tools to the discrete and masked diffusion setting, which is increasingly relevant in modern generative modeling. We will revise the manuscript to better articulate the motivation, tightness, and practical implications of these equality-based results.\n\n\n\n> **Q1. Could the authors provide baseline comparisons to better support the effectiveness of the proposed framework?**\n\nWe appreciate the reviewer’s suggestion and have included baseline comparisons to better support the effectiveness of our proposed framework.\nFirst, we conducted a direct comparison between our time-free estimator (Eq. 14) and the time-integral estimator (Eq. 13) to assess the practical utility of the former. Specifically, we measured the Monte Carlo (MC) variance of conditional log-likelihood estimates on 30 sequences with 15 samples each from HellaSwag[1], ARC_hard[2], and PIQA[3] using the LLaDA model. As shown in the following table:\n### Variance ↓ of Estimator (HellaSwag, 30 subset)\n| #MC sample | Time-integral | Time-free |\n | ----- | ----- | ----- |\n| 128 | 70.9716 | 11.5658 |\n| 256 | 30.1938 | 6.0150 |\n| 512 | 13.3796 | 2.9170 |\n---\n### Variance ↓ of Estimator (ARC_hard, 30 subset)\n| #MC sample | Time-integral | Time-free |\n | ----- | ----- | ----- |\n| 128 | 23.1758 | 5.7318 |\n| 256 | 18.1444 | 2.9612 |\n| 512 | 9.5014 | 1.8200 |\n---\n### Variance ↓ of Estimator (Piqa, 30 subset)\n| #MC sample | Time-integral | Time-free |\n | ----- | ----- | ----- |\n| 128 | 19.76875 | 4.92825 |\n| 256 | 15.14975 | 1.80950 |\n| 512 | 6.50375 | 1.22425 |\n\nThe results shows that the time-free estimator consistently exhibits lower variance. This finding aligns with observations in the LLaDA paper:\n\n\"Eq. (12) (RADD formulation) requires over 1000 Monte Carlo estimates for stable results, whereas Eq. (14) (LLaDA’s reformulation) achieves stability with only 128 estimates.\"\n\nBeyond variance reduction in marginal likelihood estimation, our time-free formulation also enables a key extension not addressed by prior work: unified (i.e., coupled) likelihood ratio estimation. This is possible because our formulation defines an explicit sampling distribution over mask positions $J$, allowing us to compute both log $p(y^+|x)$ and log $p(y^-|x)$ under shared randomness. Rather than computing two separate estimates and subtracting them, which often leads to compounded variance, our method yields a direct estimator of the log-likelihood ratio with provable correctness and improved stability:\n\n$$\n\\log\\frac{p(y^+|x)}{p(y^-|x)} = H\\_L \\mathbb{E}\\_J \\left[ \\sum_{i\\in I\\setminus J}\\log\\frac{p(y^{+,i}|y^{+,J},x)}{p(y^{-,i}|y^{-,J},x)} \\right]\n$$\n\nwhere $I$ denotes the index set of response tokens, $L = |I|$, and $J$ is sampled as described in Corollary 3.10.\n\nThis property is particularly valuable in downstream tasks where likelihood ratios are central. For example, Direct Preference Optimization (DPO) [4] directly optimizes log-likelihood ratios between preferred and dispreferred responses, and likelihood ratio tests are fundamental to out-of-distribution (OOD) detection [5,6].\n\nTo evaluate the empirical benefits, we conducted a likelihood ratio estimation experiment on the BeaverTails dataset [7]. Our estimator, using coupled MC sampling, achieved significantly lower variance compared to the standard decoupled (two-sample) baseline. When computing log-likelihood ratios for 500 $(x, y^+, y^-)$ triples, the average variance was consistently lower, as shown in the following table:\n\n### Variance ↓ of Log-Likelihood Ratio Estimation (BeaverTails, 500 triples)\n|#MC samples | Coupled       | Decoupled      |\n|-------|--------------|--------------|\n| 5     | 8897.079131 | 62469.413827 |\n| 10     | 4487.376690 | 29107.212740 |\n| 15     | 3059.972797 | 20695.614707 |\n| 20     | 2335.123036 | 16514.715472 |\n\nFinally, to demonstrate applicability at scale, we apply our framework to the large-scale open-source LLaDA model (Section \"Application to a Large Scale Open-Source Model\"). In this setting, we use our conditional likelihood estimator for a membership inference task, comparing completions from LLaDA with real data across English and Chinese datasets. The results reveal signs of training data overlap, highlighting the practical relevance of our estimator for tasks involving privacy auditing and model interpretability.\n\nWe believe these comparisons and experiments provide strong empirical justification for the advantages of our time-free formulation, both in terms of estimator quality and its applicability to important real-world tasks.\n\n[1] Zellers et al., HellaSwag: …, ACL, 2019.\\\n[2] Clark et al., Think you have solved question answering?..., arXiv, 2018.\\\n[3] Bisk et al., PIQA…, AAAI, 2020.\\\n[4] Rafailov et al., Direct preference optimization: Your language model is secretly a reward model. NeurIPS, 2023.\\\n[5] Ren et al., Likelihood ratios for out-of-distribution detection. NeurIPS, 2019.\\\n[6] Zhang et al., Your Finetuned Large Language Model is Already a Powerful Out-of-distribution Detector. AISTATS, 2025.\\\n[7] Nie et al., BeaverTails: Evaluating Language Models by Simulating Human Preference Data. arXiv, 2024."}, {"review_id": "K1vxNlyEDS", "reviewer": "Reviewer_drJV", "summary": "This paper introduces a scalable, information-theoretic framework for discrete diffusion models, where tokens are randomly masked via an absorbing diffusion process and reconstructed using a novel info-minimization denoising cross-entropy (I-MDCE) loss that connects score-based estimation and mutual information. This approach enables efficient Monte Carlo estimation of NLL without requiring time integration. Additionally, the proposed model demonstrates strong NLL estimation on synthetic DNA data, and experiments on large models like LLaDA highlight its utility for interpreting model behavior and data influence.", "questions": "(Line 191) Could you clarify why the DCE can be used as an approximation to the MSE in MMSE estimation? Since DCE (cross-entropy) and MSE (L2 norm) represent fundamentally different geometric measures, what is the intuition or theoretical justification for treating them as approximately equivalent in this context?", "limitations": "1. The time-free likelihood formulation is currently only applicable to masked diffusion models (via I-MDCE) and does not generalize to the broader I-MDSE setting.\n\n2. The ability of the method to recover likelihoods could raise privacy concerns in sensitive applications, highlighting the need for cautious use in downstream tasks.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strengths:\nThe paper is well-structured, easy to follow, and features a thorough theoretical derivation. By integrating information theory into discrete diffusion models, it captures the essence of the information-theoretic approach, namely, building pointwise relationships, which adds substantial novelty. The authors are the first to introduce the I-MDSE relationship. To further extend this, they derive a mask setting of I-MDSE in masked diffusion models, termed I-MDCE. Additionally, to improve scalability, they propose a time-free likelihood estimation framework to facilitate model training. \n\n\n# Weaknesses:\n- The experimental section is relatively limited. It only validates the effectiveness of the proposed model, without comparison to other baselines. For example, in the experiments on DNA sequences, could NLL estimation be performed using other discrete diffusion models as well? Beyond the visualizations in Figures 1 and 2, is it possible to provide concrete quantitative metrics to measure the estimation error? I suggest expanding the experiments, such as displaying sampling results in the prompt-response experiments and comparing perplexity across different models. Nevertheless, given the theoretical contributions of this paper, the current experiments are still acceptable.\n\n- A few minor points to note:\n\n  - There is a comma typo in Eq. (10) and Eq. (12).\n\n  - It is recommended to number all the equations on page 6.\n\n  - In Line 214, the description is reversed. It should read: \"In practice, replacing $c^*$ with a learned predictor $c^\\theta$ gives the estimator.\"", "was_revised": true, "final_justification": "Thank you for your response and for providing a variance-based analysis to support the quantitative evaluation. However, I still find that the comparison against diverse baselines is lacking, which is also raised by other reviewers. Due to this limitation, I’m inclined to lower the assessment of the paper’s significance and have adjusted my final rating to 4.", "rebuttal": "> **W1. The experiments are limited in scope and lack baseline comparisons or quantitative metrics.**\n\nWe sincerely thank the reviewer for the thoughtful feedback and for highlighting the need for broader experimental validation. We fully agree that adding stronger baseline comparisons and quantitative metrics is essential to support the practical effectiveness of our framework. While the reviewer specifically mentioned DNA and prompt-response tasks, we instead chose to evaluate our estimator in more realistic, large-scale settings using the LLaDA model [1], in order to better reflect practical use cases beyond synthetic settings. To this end, we conducted two additional experiments focused on variance reduction and likelihood ratio estimation.\n\n1. Variance comparison of likelihood estimators.\\\nWe compared our time-free estimator (Eq. 14) with the time-integral estimator (Eq. 13), which corresponds to the standard variational bound. We evaluated the average of Monte Carlo (MC) estimations of conditional log-likelihood estimates on 30 sequences with 15 samples each from HellaSwag[2], ARC_hard[3], and PIQA[4] using the LLaDA model. As shown in the following table, the time-free estimator consistently yields lower variance:\n\n### Variance ↓ of Estimator (HellaSwag, 30 subset)\n| #MC sample | Time-integral | Time-free |\n | ----- | ----- | ----- |\n| 128 | 70.9716 | 11.5658 |\n| 256 | 30.1938 | 6.0150 |\n| 512 | 13.3796 | 2.9170 |\n---\n### Variance ↓ of Estimator (ARC_hard, 30 subset)\n| #MC sample | Time-integral | Time-free |\n | ----- | ----- | ----- |\n| 128 | 23.1758 | 5.7318 |\n| 256 | 18.1444 | 2.9612 |\n| 512 | 9.5014 | 1.8200 |\n---\n### Variance ↓ of Estimator (Piqa, 30 subset)\n| #MC sample | Time-integral | Time-free |\n | ----- | ----- | ----- |\n| 128 | 19.76875 | 4.92825 |\n| 256 | 15.14975 | 1.80950 |\n| 512 | 6.50375 | 1.22425 |\n\n This finding aligns with observations in the LLaDA paper:\n\n*\"Eq. (12) (RADD formulation) requires over 1000 Monte Carlo estimates for stable results, whereas Eq. (14) (LLaDA’s reformulation) achieves stability with only 128 estimates.\"*\n\n2. Likelihood ratio estimation via coupled sampling.\\\nBeyond what is discussed in the paper, our time-free formulation also offers an additional practical advantage: it supports a unified (coupled) estimator for likelihood ratios, enabled by an explicit sampling distribution over masking positions $J$. This allows computing $\\log p(y^+|x)$ and $\\log p(y^-|x)$ under shared randomness, avoiding the high variance of naive subtraction:\n$$\n\\log\\frac{p(y^+|x)}{p(y^-|x)} = H\\_L \\mathbb{E}\\_J \\left[ \\sum_{i\\in I\\setminus J}\\log\\frac{p(y^{+,i}|y^{+,J},x)}{p(y^{-,i}|y^{-,J},x)} \\right]\n$$\nHere, $x$ denotes a prompt, and $y^+$ and $y^-$ represent the preferred and dispreferred completions, respectively. This setup commonly arises in alignment tasks where a model is trained to score preferred responses higher than dispreferred ones. In particular, the Direct Preference Optimization (DPO) objective [5] optimizes the objective that consists of the following alignment score:\n$$\n\\log\\frac{p^\\theta(y^+|x)}{p^\\theta(y^-|x)}\n$$\nwhich corresponds to the log-likelihood ratio between the preferred and dispreferred responses under the current model $p^\\theta$. This form naturally motivates the use of likelihood ratio estimators, especially in settings where the responses $y^+$ and $y^-$ are sampled from human preference data.\\\nTo validate this, we conducted a likelihood ratio estimation experiment on the BeaverTails [6] dataset, which contains 500 $(x, y^+, y^-)$ triples annotated with human preferences. The average variance of MC estimates was consistently lower for our coupled estimator, as shown in the following table:\n\n### Variance ↓ of Log-Likelihood Ratio Estimation (BeaverTails, 500 triples)\n|#MC samples | Coupled       | Decoupled      |\n|-------|--------------|--------------|\n| 5     | 8897.079131 | 62469.413827 |\n| 10     | 4487.376690 | 29107.212740 |\n| 15     | 3059.972797 | 20695.614707 |\n| 20     | 2335.123036 | 16514.715472 |\n\nThese results demonstrate that our estimator not only improves sample efficiency in likelihood estimation but also enables stable and low-variance ratio estimation, both of which are critical for real-world applications involving alignment, robustness, and interpretability.\n\n> **W2. Minor points including typos.**\n\nThank you for pointing out these issues. In addition to correcting the comma typos in Eq. (10) and Eq. (12), numbering all equations on page 6, and fixing the reversed phrasing in Line 214 as suggested, we will take this opportunity to carefully review the entire manuscript for any remaining minor errors.\n\nIn particular, we sincerely apologize for having mistakenly reversed the first and last name of the lead author of the RADD paper. We will correct this and ensure that all references are accurate and respectful in the final version.\n\nWe greatly appreciate your attentive reading and helpful suggestions.\n\n> **Q1. Clarify why DCE can approximate MSE in MMSE estimation despite their different geometric interpretations.**\n\nThank you for this thoughtful question. While it's true that cross-entropy (DCE) and mean squared error (MSE) arise from fundamentally different geometric interpretations (logarithmic vs. Euclidean) their use as estimation objectives can be understood through the lens of distribution-loss matching, as commonly seen in information theory and estimation theory.\n\nFor example, logarithmic loss is the natural loss function when modeling data with a discrete distribution, just as L2 loss aligns with Gaussian distributions and L1 loss with Laplacian distributions. In our setting, the cross-entropy (DCE) loss corresponds to logarithmic loss, which is the proper scoring rule for categorical (discrete) output distributions, and is directly tied to pointwise mutual information and entropy minimization. This connection has been theoretically formalized in works such as Courtade & Weissman (2013) and Jiao et al. (2015) [7, 8].\n\nTherefore, while DCE and MSE measure different quantities, DCE plays a role analogous to MSE in the discrete domain: it minimizes expected divergence between the predicted distribution and the true discrete target, just as MSE minimizes distance under a continuous Gaussian model. This analogy justifies the use of DCE as a surrogate for MMSE in discrete diffusion settings, particularly when the underlying variable is categorical.\n\nWe will clarify this theoretical motivation in the revised manuscript and cite the appropriate references.\n\n[1] Nie et al., Large language diffusion models, arXiv, 2025.\\\n[2] Zellers et al., HellaSwag: Can a Machine Really Finish Your Sentence?, ACL, 2019.\\\n[3] Clark et al., Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge, arXiv, 2018.\\\n[4] Bisk et al., PIQA: Reasoning about Physical Commonsense in Natural Language, AAAI, 2020.\\\n[5] Rafailov et al., Direct Preference Optimization: Your Language Model is Secretly a Reward Model, NeurIPS, 2023.\\\n[6] Nie et al., BeaverTails: Evaluating Language Models by Simulating Human Preference Data, arXiv, 2024.\\\n[7] Thomas and Weissman, Multiterminal Source Coding under Logarithmic Loss, IEEE Trans. Inf. Theory, 2013.\\\n[8] Jiao et al., Justification of Logarithmic Loss via the Benefit of Side Information, IEEE Transactions on Information Theory, 2015"}]

Author rebuttal:

Meta-review: This paper develops an information-theoretic framework for discrete diffusion models, extending I-MMSE results to discrete and masked settings and introducing exact equality-based NLL estimators. The time-free estimator achieves clear variance reduction and enables unified likelihood ratio estimation, which reviewers agreed is theoretically novel and practically useful. Although several reviewers noted the empirical evaluation is limited, all reviewers gave positive scores, and the consensus is that the theoretical contributions and supporting evidence justify acceptance as a poster.
Decision: Accept (poster)