Information Estimation with Discrete Diffusion
Authors: Alberto Foresti
Giulio Franzese
Pietro Michiardi
Venue: ICLR 2026 Poster (2026)
Keywords: Information Theory
Deep Learning
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Information-theoretic measures, such as Mutual Information (MI), play a crucial role in understanding non-linear relationships between random variables and are widely used across scientific disciplines. Yet, their use on real-world discrete data remains challenging. Existing methods typically rely on embedding discrete data into a continuous space and apply neural estimators originally designed for continuous distributions. This process requires careful engineering for both the embedding model and estimator architecture, but suffers from issues related to high data dimensionality. In this work, we introduce InfoSEDD, a discrete diffusion–based approach that bridges information-theoretic estimation and generative modeling such that they can be used to compute Kullback–Leibler divergences. Backed by Continuous Time Markov Chains theory principles, the design of InfoSEDD is lightweight and scalable and allows seamless integration with pretrained models. 
We showcase the versatility of our approach through applications on motif discovery in genetic promoter data, semantic-aware model selection in text summarization, and entropy estimation in Ising models.
Finally, we construct consistency tests on real-world textual and genomics data. Our experiments demonstrate that InfoSEDD outperforms alternatives that rely on the ''embedding trick''. Our results position InfoSEDD as a robust and scalable tool for information-theoretic analysis of discrete data.

Published as a conference paper at ICLR 2026
INFORMATION ESTIMATION WITH DISCRETE DIFFU- SION
Alberto Foresti, Giulio Franzese & Pietro Michiardi Department of Data Science EURECOM
450 Route des Chappes, 06410 Biot, France webeebee..... foresti, giulio. www.uuloo..faaeseseeeae franzese, pietro weneatrr...ccchiii michiardi Beernecoo fr Beurecom,
ABSTRACT
iiformatiinntthoreede measures, such Mutual Information (MI, play crucial role in understanding non-linear relationships between random variables and are widely used across scientific disciplines. Yet, their use on real-world discrete data remains challenging. Existing methods typically rely on embedding discrete data into continuous space and apply neural estimators originally designed for contin- uous distributions. This process requires careful engineering for both the embed- ding model and estimator architecture, but suffers from issues related to high data dimensionality. In this work, we introduce INFO-SEDD, a discrete diffusion ffaded approach that bridges cooommttion-teoretti estimation and generative modeling such that they can be used to compute Kullback-Leibler divergences. Backed by Continuous Time Markov Chains theory principles, the design of INFO-SEDD lightweight and scalable and allows seamless integration with pretrained models. We showcase the versatility our approach through applications on motif discov- ery in genetic promoter data, semantic-aware model selection in text summariza- tion, and entropy estimation Ising models. Finally, we construct consistency tests real-world textual and genomics data. Our experiments demonstrate that INFO-SEDD outperforms alternatives that rely on the "embedding trick", Our re- sults position INFO-SEDD robust and scalable tool for information-thoore analysis of discrete data.
INTRODUCTION
Information theoretic measures represent a powerful tool understand non-linear relationships
between random variables (Shannor. 1948; MacKay, 2003) and find wide range of appli- cations scientific fields (Karbowski, 2024: Eekford al 2016). Mutual information (MI),
in particular, has become established metric in machine learning (Bell and Sejnowski. [1995 Stratos, 2019 Belghazi et all 22088: Oord et all 2018: Hjelm et all 2019), both for training models (Alemi al. 2016 Chen al_ 2016: Zhao et all 2018) and inference time (Alemi and Fischer. 8 Huang et al. 2020).
Estimating information theoretic quantities remains open problem, and different paradigms for their estimation have emerged. Classical parametric and non-parametric methods (Pizer et all 1987: Moon all 1995; Kraskov al., 2004; Gao et al., 2015) have been recently superseded
by variational approaches (Barber and Agakov, 2004 Neuven et al. 2007: Nowozin et al_ 2016
Poole et all 2019: Wunder et al.) 2021] Letizia et al. 2023: Federici al. 2023) and neural es- timators. Notably. several approaches rely on optimizing lower bound (Papamakarios et all 2017: Belghazi et al. 2018: .Ord et all, 2018 [Sng and Ermon. 2019h; Rhodes al.. 2020
Letizia and Tonello, 2022: [Brekelmans et all (022). However, McAllester and Stratos (020):
Song and Ermon (2019a) challenge the applicability of this kind estimator, especially in high MI scenarios, they require amount of samples that grows exponentially with the ground truth MI. To overcome this limitation, a new class of estimators has emerged (Franzese et all 2023a Butakov et all 2024k Kholkin et all (2025). Unfortunately, these methods focus continuous data and, despite their practical importance, few estimators for high- digh-dimessional discrete distributions have been proposed in the literature. While classical estimators for discrete random variables exist

---

Published as a conference paper at ICLR 2026
(Pnnhha et al. 2024), their accuracy rapidly decreases with increasing data dimensionality. Appli- cations that would benefit from scalable estimators of MI include, among others, DNA or peptide sequencing (Newcomb and Sayood, 2021; Xia et ald, text summarization (Darrin et al (024)) and neuroscience (Chai et al. 2009), to name a few examples. Consequently, the development of new estimation techniques is of paramount importance for the broader scientific community.
common workaround to deal with high-dimensional, discrete data is to embed it in a continu- ous space and use neural estimators conceived for continuous distributions. One recent example is (Le and Rhee, 2024), where it is shown that the embeddings of pretrained language models can provide meaningful representations to estimate information theoretic quantities in unstructured data, However, such a process may not fully capture the discrete nature of the underlying data and might suffer from several limitations, such as the necessity to consider application-spes -speciio-- embeddings. For example, we show that MINDE (Franzese et al., 2023b), which is strong MI estimator in con- tinuous domains, like image data, struggles with discrete data.
In this work, we build on the growing literature of diffusion- based estimators (Kong et al 2022: Franzese et al., 2023a; [Bounoua et al. 2024), and present INFO-SEDD, novel method for estimat- ing information theoretic quantities of discrete data using Continuous Time Markov Chains (CTMCs)
(Lou et al., 2024). These stochastic processes have recently seen a surge in popularity for applica- tions such as generative language modeling (Lou et all 2024 Sahoo et al. 2024 Nie et all. 2025). Their fundamental working principle the reversal of a perturbation process which starts with clean data from given distribution and that converges to uninformative noise. The workhorse of these approaches is the score function, which contains information about the probability distributions as- sociated with the CTMCS different time instants. Our proposed method, INFO-SEDD, builds upon such mathematical framework, extending it via Dynkin's lemma (Hanson, (2007), and leverages score metries, such as MI between two random variables,
functions compute key information-theorre and the entropy of given distribution. By carefully selecting perturbation processes, our approach requires training only single parametric model to compute MI across arbitrary subsets of variables. Furthermore, INFO-SEDD seamlessly integrates with pretrained models, without requiring ad-hoc procedures to work with discrete data.
To rigorously evaluate our method, we perform a large number of experiments both on synthetic and real data, and compare INFO-SEDD to state-of-the-art methods. We design a synthetic bench- mark with ground truth MI that presents challenges such high data dimensionality and high MI scenarios. Our results demonstrate that INFO-SEDD is both robust and consistently outperforms existing estimation methods. We then focus two application domains in which compare INFO- SEDD to competitors, which all require the "embedding trick" discussed above. First, we tackle the text summarization domain, evaluate the consistency of MI estimators, and study if MI rep- resents a meaningful signal to perform model selection, as INFO-SEDD estimates are well aligned with human metries for evaluating text summarization. We also present results in the domain of genomics, whereby we evaluate the consistency of MI estimators, and show that INFO-SEDD can be used detect motif location in DNA sequencing data. Overall, our results indicate that INFO-SEDO outperforms alternatives for MI estimation, and pave the way to exploit information measures as meaningful signals for a variety of downstream tasks involving high-dimensional discrete data.
2 METHODOLOGG
We begin by defining the relation between CTMCs over discrete state spaces (Anderson, 2012) and the computation of Kullback-Leibler (KL) divergences. First, in Section 2. we provide a brief in- troduction to the fundamentals of CTMCS, emphasizing their time-reversal properties and parametric approximations (Lou et all. 2024)). Then, in Section 2.2] we demonstrate how these processes can be adapted for divergence estimation, specifically by analyzing two processes that share the same generator but differ in their initial conditions.
2.1 PRELIMINARIES
Consider CTMC X,,t E [0.T], defined over finite state space x (1,...,N) and spec- ified by the infinitesimal generators Q. [0, T] NNN, where the diagonal entries satisfy
2

---

Published as a conference paper at ICLR 2026
a - Eafs b), b), with (a b) 2 0, # b. As established by Anderson (2012), the time evolution of the probability distribution Pr(X; x) def (z(2) satisfies the following Ordinary Differential Equation (ODE): P: Po So S.P.Pdds.... .dss where the initial conditions of the process Po determine the distribution Pt at any time t. A key property of CTMCs is that their time-reversed counterpart, recently used for generative modeling purposes (Lou et al., 2024), also follows CTMC, but with different set of transition matrices. More precisely, defining the time-reversed process as
def - the reverse process evolves according to the following ODF Lou al 2024: Sun et all
Pr
2023): p, Pr So Q2 ,s, where the reverse-time transition matrices Q2 relate to the forward transition matrices as follows:
( bęz
(b a)
(r) ((a) (a. - b((a,b))
( (
8(a,a).
(I)
Under appropriate technical conditions on Q (Lou et al. 2024). the terminal distribution Pr con- verges known reference distribution æ, which is independent of the initial distribution Po- This property enables sampling from Po by simulating a CTMC with appropriately chosen generators (Sun et al. 2023: Kelly, 1981). However, other than simple and uninteresting scenarios, exact knowledge of the quantities Pr (()) is of reach. practical solution to substitute in this Pr-1(a)
numerical integration parametric function (a, t)s. whose parameters are optimized according to the Diffusion Weighted Denoising Score Entropy (DWDSE) loss function (Equation 10, Section 3.2 of Lou et al. (2024)). Whenever the context is clear, we simplify the notation for the parametric
score in the remainder of the paper and denote it with sy (Xx)), instead of (X,,,))
2.2 KL. DIVERGENCES VIA CTMCS
Next, we show how to extend the CTMC framework to compute the KL divergence between two probability distributions Po and To defined the support X. expressed as KI. Po do To achieve this, we construct two Markov chains that differ only in their initial conditions: one initialized from po and the other initialized from g.. The KL divergence can be expressed as:
+
KL Po 10 = E log Ee (X+) do
E log PI (I = E
log Le
(x X
(2)
The last term in Equation
2 can be rewritten using Dynkin's formula (Hanson 2007). which states
that for generic function f x x [0, T] + R, we have:
E f(xx Xu - f(x.. = E //- of de
(x,,) + (00)(,, t)dt
(3)
where B is the backward operator, which we define as t) = E (f) f(a)).
By combining the result from Equation 3 with Equation 2). the KI. divergence between discrete distributions can be conveniently approximated as:
KL
Pa
do
B
(' Σ (x
[K'
Fe(e)) Fz(X,))
7.(x)
(.(a) log (x() P.(X.)
dt
(4)
(x.)
(x()))
x*X,
where ((a) = (a)o((a) - 1). We omit the term
E log
Po (Xx) as both Po and To converge to
40
I Lou al. 2024).
3

---

Published as a conference paper at ICLR 2026
The key quantities 1. and d.(0) in the expression above are not directly accessible. To address
P.(a) {,(a)
this, we substitute these ratios with parametric approximations optimized via DWDSE loss, leading to the following KT. estimator:
I
Σ XXX2
-x((x,)) ( (K (88(x))2) - log s
(x)))
Kl.
Po
20
R
Estimating Equation 55) using Monte Carlo techniques is straightforward: we sample time instants t
uniformly in [o, 7), simulate the forward process X,, and compute the required quantities using the parametric scores. Equation 5 enables estimating information measures, such as MI and Entropy. However, the KL estimator in its general form is not scalable for practical purposes: our practical method, described next, addresses this limitation.
3 OUR APPROACH: INFO-SEDD
We are now ready to introduce our MI estimator, INFO-SEDD, which is based on Equation (5) Given two random variables, X and Y. we propose two approaches:
1. Joint Method [INFO-SEDD-J]: MI can be expressed in terms of the KL-divergence between the joint distribution pxy and the product of the marginals mxy px py. ((X,Y)
KL [xxy || mxy), where & denotes the Kronecker product.
2. Conditional Method [INFO-SEDD-C]: alternatively, MI can be expressed the KT. di- vergence between the conditional distribution and the marginal distribution, I(X,Y) = |-
E[KL |Y|x
In high dimensional applications, a naive implementation of Equation (5) quickly becomes unfeasi-
ble. Indeed, the size of the number of entries of the matrix Q, scales with Ix|=, which is intractable. However, in many cases of interest, the random variables can be naturally decomposed into a struc- tured sequence of D subcomponents, each taking values from a discrete set of size |xl (Lou et all 2024: Aussin et all 2021 Campbell et all (2022), iee. X = [XI.... Xp), leading to a total state space of size Ixl". This structured decomposition enables the use of sparse rate matrices, which constrain the CTMC to modify only one subcomponent at a time, significantly reducing computa- tional complexity. This allows to consider sequences which only differ in one component when
computing the entries of Q, or, equivalently, sequences with unit Hamming distance In particular, + the non-zero entries of Q are determined by shared |xl x |x| local rate matrix Q Specifically,
tok
two sequences a and b differ only at the i-th component, then their transition rate is given by
-ink (a Q. ba).
Expressing the MI as the KL. divergence between the joint distribution and the product of marginals, KL [xxY mxy], requires training two separate score models, each tailored to specific distribu-
tion. A carefully chosen transition matrix can circumvent this requirement, allowing for a single lok
model to be trained instead. In particular, selecting Q, = a1) with (D) a fixed scalar tok
-slok function, and the absorbing matrix Qansorh as defined in dLou et all 2024: Camphell et all, 2022 Austin et all 2021),, ensures that the subcomponents can only transition into an absorbing state 0. This choice is crucial because it enables the computation of marginal scores using a model trained solely the joint distribution:
EX 7x
(6)
Pl (v)
'This prohibits any sequence to jump another sequence with Hamming distance larger than one, but only in the intinitesimal time regime dt, Once we consider longer time intervals, perturbations immediately become non-local. Empirical evidence in the field of discrete diffusion models indicates that such decomposition al- lows dealing with complex distributions with non-local interactions, such as text/language and single-cell RNA sequencing data Sahon al. 2024 Lou al. 2024)
4

---

Published as a conference paper at ICLR 2026
For the mathematical proof the reader is invited to check Appendix A.3 single score model trained on the joint distribution is sufficient for computing the marginal scores well By integrating these design choices, we present the pseudocode for INFO-SEDD-J and INFO-SEDD-C in Appendix B in Algorithm and Algorithm Drretmctivieeeillly......
This result implies that
Theoretical properties. We investigate the theoretical properties of INFO-SEDD, specifically focus- ing on the error decomposition and consistency. We establish that under mild assumptions on the boundedness of the score functions (constants Ci, C2) and the neural network approximation errors (ep, Eg), the deviation of the INFO-SEDD estimator from the true KL divergence is bounded by:
E P (8g: KI. (l. Sondisoti <T(T) (1 (5p (5p (1- Pr (6% ) log Ixl)
Truncation Bias
Estimation Error
(7)
Where E(s); s5: corresponds to the right-hand side of Equation This bound highlights key trade-off: the first term, representing the estimation eπтоr, scales linearly with the score error and constants dependent on the data distribution. The second term represents a truncation bias arising from the finite time horizon T, which vanishes exponentially as the probability of the absorbing state consistent estimator up this exponentially decaying bias, allowing for accurate KI. estimation without the exponential variance associated with standard importance sampling methods. We refer the reader Appendix [ for the full derivation and proofs.
+ (0P) approaches 1. Consequently, INFO-SEDD
Estimating entropy. Our method can also be used for Entropy estimation, since it can be expressed in terms of the KL divergence between a distribution Po and the uniform distribution uo, as follows:
(P)) = log N KL Po || ud]. Since the ratio appendix A.2 we can adapt the formulation Equation derive INFO-SEDD-H, an Entropy estimator, which is detailed in Appendix B in Algorithm 3
MT) with (t) = =[o (s)ds (see
a.(0)
4 EXPERIMENTS
series of synthetic experiments, well several realistic applications. Overall, while our competitors struggle with high MI scenarios and high-dimensional data, INFO-SEDD obtains accurate estimates. Using text and DNA data, we show that INFO-SEDI not only accurate, but also useful for various downstream tasks. In Appendix D we provide additional results, showing that INFO- REDD produce accurate Entropy estimates in Ising models (Onager., 1944), which are used to understand the state of a particle lattice using their spins.
Next, evaluate INFO-SEDD
4.1 SYNTHETIC EXPERIMENTS
We validate our method on synthetic distributions with known mutual information (full details are in Appendix C.1. C.I. Given two vectors X = (x1, , and Y = (y1, V2...., UD) sampled from their respective discrete distributions, we denote vector dimensionality by D. and the support |xl as the number of discrete values each element I,, Y/ can take.
Next, we benchmark INFO-SEDD and several competitors on a high dimensional synthetic bench- mark, by increasing both MI and D as shown in Table [1] We compare the results of our proposed methodology against some of the estimators from Letizia et al. (2024), who introduce data derange- ments to improve bias and variances of existing variational estimators, including NWJ (Nguven et all 2007), SMILE &Song and Ermon, 2019a) and MINE (Belghazi et al., 2018). Letizia et all (2024) also proposes the F-DIME estimators GAN-DIME, KL-DIME, HD-DIME. which are shown work better at high MI. We also benchmark MINDE (Franzese et all (023a)). generative neural estimator for continuous data. We use the same backbone for all methods (see Appendix C. with only minor tweaks to initial and final layers to accommodate the specifics of each method. For our method, we
"This configuration adds an absorbing state, increasing the dimension of the support
5

---

Published as a conference paper at ICLR 2026
report the results with the INFO-SEDD-J variant, trained using the DWDSE loss. We train all methods using 10 samples and a batch size of 1024 for 10 steps.
Table 1:
Results for the high dimensional synthetic benchmark for all estimators with given MI and same vector length (D) for each modality. We report the mean estimate and standard deviation over 10 seeds. Best estimates are marked in bold.
Estimatce
INPO-SEDD
GAN-DIME
HD- DIME
KI-DIME
MINDE
MINE
NWI
SMILE
MI-10, D-10
9.92
0.12
12:15 0.89
0.43
8.38 + 0.90
14.01
10.21 + 6:33
6.16 2.11
12.85 0.95
MI=20, D-20
28.02
0.21
22.09 +
12.63
1.07
7.51 -: 0.56
26.98
3.16
888
6.50 0000:00 0.84
23:11
1.41
MI-30, D-30
29.83
0.54
20.74
+ 1.75
11.72
2 2.09
7.02 +: 0.45
31.08
4.33
7.41 + 1:23
6.35 - 0.34
21.79
= 1.08
MI=40, D=a0
39.11
0.65
19.64
+: 1.33
11.68
0.94
6.52 4: 0.32
33.97
+ 3.50
6.91
1 0.66
6.44
20.13
1.27
MI=50, D-50
1.18
1044
1.12
0.62
32.60
1.14
5.06
18,97
1.05
Results in demesttee that INFO-SEDD consistently outperforms competing methods. INFO- SEDD performs better with high dimensional data and high MI, while existing neural estimators fail. This is not the only scenario where INFO-SEDD shines, as demonstrated extensively in Ap- pendix C In Appendix CC1.6 we perform ablation study on Xxl, showing that INFO-SEDD is robust to changes in support size, where, instead, the competitors cannot provide accurate estimates. Accuracy, however, is not the only dimension where competitors struggle. In Appendix [AK] we show that even when competing estimators like GAN-DIME and SMILE provide correct estimates, they take more epochs to converge compared to INFO-SEDD, resulting in slower training. Finally, in Appendix C.1.5 we evaluate empirically the sample complexity of INFO-SEDD observing that it is accurate even when using only 10sh samples.
Overall, our complete set of synthetic experiments confirms that INFO-SEDD is robust, efficient, and parsimonious terms of sample size. Next, we revert to realistic use cases to further evaluate information-theoreii estimators in even more challenging scenarios.
4.2 APPLICATION: TEXT SUMMARIZATION
We consider the task of text summarization, which aims at generating concise document summaries, either by directly extracting sentences from the original corpus, or by introducing new, relevant phrases not present in the original text (Nallapati et all 2016). We validate WFODSEDD using the SUMMEVAL dataset (Fabbri et all 2021) which assembles collections of summaries generated by 23 different models, providing human judgments for outputs generated by 15 of these models. We generate a collection of datasets of texts/summaries pairs from the SUMMEVAL dataset, such that we can scramble part of such pairs to gauge MI between texts and summaries distributions. We compare INFO-SEDD and alternative methods, showing how much each estimator is consistent with a "theoretical" trend of MI. We also study the application of INFO-SEDD for summarization model selection, by estimating the MI of each collection of model outputs with respect to the reference texts. Studying how these estimates compare to the human evaluations of the summaries helps practitioners to identify which human metric is best aligned to MI. For more details, refer to Appendix C2
Consistency test. We perform a consistency test for MI estimation between model- generated sum- maries and reference texts: we select the summaries generated by BART (Lewis et al. 2020) and pair them with the original text with a probability p and with a random text in the dataset with probability 1-p. While we cannot establish an exact ground truth for these experiments, we can determine that MI should grow linearly as a function of p. under the assumption of MI being significantly larger than log 2. This assumption is backed by studies (Takahira et al. 2016: Cover and King, (1978) that estimate the entropy rate of textual data to be larger than 1 bit per character (bpe), and by works that estimate MI in similar context (Darrin et all (2024). To support our claims and provide an order- of-magnitude estimate, we multiply the entropy rates of Takahira et al. (016) and Cover and King (1978) with the average length of dataset summaries, to obtain entropy estimates for English texts with the same length, obtaining values of 256 nats and 303 nats, respectively. For a more detailed discussion, refer to Appendix C.2) We use the bone, with minimal changes to the architecture to accommodate our competitors. We also slightly modify the training strategy SaoSoooo et al. (0244) to allow a more efficient learning of marginal and conditional scores for INFO-SEDD (check Oopenddi(C.21fff details). This allows us to keep the same context length, architecture, and similar number of parameters for all methods. Note that for all
model (Sahoo et all (2024) as the back-

---

Published as a conference paper at ICLR 2026
competitors, we project text tokens into embedding space of fixed dimension, by jointly learning an embedding look-up table for each method. For more experimental details, check Appendix
Condetencyy
Unier frand 2433
3-
10a
400
450
500
550
600
650
700
750
800
Mutual Information
10
Figure 2: Consistency human metric compared
to MI as estimated by INFO-SEDD-C.
Hó-Dime
me
Infosaddic
10-1
Infosedi:
tann
Ki-Dime
Conc
Smile
10
Empiriaal estimate
p 303 nats)
0.2
0.8
250
275
300
325
550
375
400
425
Mutuel Information im
Figure 1: Consistency results for selected
Figure 3: Consistency human metric compared to MI as estimated by INFO-SEDD-J.
estimators.
As shown in Figure D where we report MI estimates for p = 0.0. 0.1, 1.0, the consistency of both INFO-SEDD variants the highest among all competitors, closely matching the empirical derivation outlined above. Note that INFO-SEDD-C obtains MI estimates closer to zero than the joint variant, when o = 0.0. Instead, variational approaches cannot reliably estimate MI with values larger than the logarithm of the batch size used for estimating it (McAllester and Stratos 2020 Song and Ermon 2019a). This results in MI being underestimated, a problem that could only be addressed by impractical batch sizes. Despite KL-DIME and SMILE exhibit approximately linear correlation, they obtain low MI values, incompatible with the empirical derivation. HD-DIME can achieve higher values of MI, but fails for 0.5. MINDE does not provide meaningful MI estimates: this is likely due to the high embedding dimensionality which, together with the sequence length, is a challenging scenario for continuous score models.
Model selection. We now evaluate if MI is a meaningful signal to perform model selection, by measuring if MI estimates are compatible with human-preference metries, as done by Darrin et all (2024). This is done by estimating the MI of the text-model summary datasets in SUMMEVAL (Fabbri et all 2021h, for every model with available human metries. The human metrics we consider are: coherence, which measures how well-structured and coherent the summary is; consistency, which quantifies how much the statements in the summary are entailed by the source; fluency, which evaluates the grammatical correctness and the formatting of the summary; and relevance, which measures the summary captures only the most important information of the source text. Table which shows Pearson correlation and Kendall's 's Tau test results, indicates that MI correlates the most with consistency. This is expected, as consistency effectively quantifies how much infor- mation the text and summary share. Note, however, that raters were asked to penalize hallucinated facts present in the summaries, which is not captured by MI. The second metric which correlates the most with MI is fluency. The definition of fluency does take into account comparison between summary and source text, but also strongly correlates with consistency. Thirdly, MI correlates well with relevance. Relevance considers comparison between summary and source text in terms of what is considered important information by human annotators. However, important information highly subjective, and human intentions are likely not captured by MI. Lastly, coherence does not quantify a relationship between source text and summaries, which explains the low correlation with MI. Figures 2 and 3 we show the relation between the MI estimates of INFO-SEDD-C and INFO-SEDD-J and consistency by running Gaussian Process (GP) regression with a Matérn Kernel with smoothing parameter / = 1.5 and noise level 0.01. We report the GP mean, its 95% confi- dence interval and the Pearson correlation. Figures 2 and show that consistency saturates around the maximum score, whereas MI does not have the same ceiling effect. The reader referred to Appendix C.2 for similar discussion about the other metrics. Overall, results in line with
7

---

Published as a conference paper at ICLR 2026
the findings by Darrin et al. (2024), where INFO-SEDD-C achieves comparable correlation with the most important metric, consistency, albeit without requiring elaborate embedding models, that rely on a large set of parameters to be learned.
Table 2: Pearson (left) and Kendall's Tau (right) correlation of human metrics and MI for INFO-SEDD and selected competitors. The labels are shorthand notations for the human metries: COHfzcoherence, CON=consistency, FLU==fleecy, REI OVR=oeeall.. The reader is invited to check Appendix[C.. C.3 for correlations with other metrics offered by the SUMMEVAL frame- work.
COH
CON
FLU
REL
OVR
COH
CON
FLU
REL
OVR
INFO-SEDD-C
0.209
0.740
0.679
0.411
0.568
INFO-SEDD-C
0.105
0.505
0.134
0.200
0.219
INFO-SEDD-J
-0.091
0,550
0.455
0.288
0.322
INFO-SEDD-J
0.048
0.486
0.153
0.219
0.238
KL-DIME
0.170
0.214
0.076
0.193
KL-DIME
0.105
0.048
0.067
HD-DIME
-0.243
0.331
0.281
-0.145
0.063
HD-DIME
-0.162
-0.029
0.057
-0.067
-0.048
SMILE
-0.367
00.074
0.162
-0.149
00221
SMILE
-0.238
-0.667
0.086
4.3 APPLICATION: GENOMICS
The genomics field heavily relies on computational methods to improve our understanding of hid- den patterns in large and complex genomic data sets from basic and clinical research projects (Libbrecht and Noble, 2015: Whalen et all 2022 Teschendorff and Horvath, 2025).. We consider two fundamental problems in genetics (genome classification and motif identification in genetic pro- moters) that rely on DNA sequencing data: DNA is a nucleotide made of four types of nitrogen bases, Adenine (A), Thymine (T), Guanine (G), and Cytosine (C). The order, or sequence, of these bases determines the biological instructions that are contained in a strand of DNA. We follow the recent practice of considering DNA sequencing data as text (Dotan et all 2024; Qiao all 2024] Malusare et al. 2024: Eapen, 2025), albeit we adopt the simple tokenization technique of consid- ering each DNA base in a sequence as a token. This results in DNA sequences being modeled as high-dimensional vectors with a support of four elements per sequence element ([xl 4). Addi- tional details are in Appendix
Consistency test. We assess the reliability of INFO-SEDD low MI regime, and consider the "HUMAN vs. WORM" dataset, which is part of the Genomics Benchmarks by Grešová et al (202)). a recently proposed suite with eight regulatory element classification tasks. The dataset consists of 10 DNA sequences, paired with 2 class labels (human or worm), with median length of 200 bases. We perform a consistency test as follows: for each sample in the dataset, we randomize the label with probability and compute the MI between the DNA sequences and the new set of labels. To construct reference, we can approximate order-of- -feegiiu-- and slope of ground-truth MI values with the assumption that, given classifier trained to predict the label given the genome, the classification error is approximately the same across each sample. To do so, we express MI as T(X, Y) H(Y)- - HY))X). where Y is the label and X is the genome, and approximate H(Y)X) as H&(Acc.), where H, is the binary entropy and Acc. is the accuracy of the classifier. We estimate (Y) directly from data using bin estimate given that we have binary label and enough samples. In our experiments, we use a pretrained C CADUCEUS model (Schiff et all (0244) for the backbone of all methods, with minimal architectural changes when needed, ensuring fair comparison. For more details check Appendix C.4
Results in Figure 4 show that INFO-SEDD-C outperforms the competitors, as it closely matches the classifier- reference MI. HD-DIME performs relatively well for < 1.0 while other competi- tors struggle to keep increasingly high MI estimates for growing values of p. The difference in performance between the two INFO-SEDD variants is due to the difference in dimensionality be- tween the label, mono-dimensional, and the DNA sequence. This makes the optimization process of INFO-SEDD-C significantly easier, since it only requires parametrizing the scores of the marginal distribution of the label and of the conditional distribution of the label given the associated DNA sequence. INFO-SEDD-J, instead, must parametrize the score of the joint distribution of the DNA sequence and label, a considerably more difficult task since it needs to do discrete diffusion training on the DNA sequence while INFO-SEDD-C only treats the DNA sequence conditioning signal.
8

---

Published as a conference paper at ICLR 2026
0.7
Gan-Dime
Ho-Dime
Infosedd-CC
TATA box region
0.5
Smile
Classifier-Bi Baieeeer--Bdddde
0.19
0.4
TATA box Ovd overap
0.08
0.2
0.0G
0.2
0.04
0.2
0.0
0.2
0.4 0..6
0.8
1.0
0.02
Fosttion Iralative
Figure 4: Consistency test for the HUMAN VS. WORM dataset against the best perform- ing competitors. For the complete numerical results check Appendix C.4
Figure 5: Arabidopsis thaliana promoter
TATA-BOX identification on DNA sequences using INFO-SEDD.
Finding motif's in promoters. A central issue in molecular biology is understanding the regulatory mechanisms that control gene expression. In particular, gene expression requires stretch of regula- tory DNA called promoter which contains certain motifs, i.e., patterns that show a statistically sig- nificant dependency with expression levels. Computational methods based on MI (Elemento et all 2007; Rao et al., (007)) use whole genome sequences to search for the key elements in transcription regulation. The key idea is to directly quantify the dependency between the presence absence of a given motif a regulatory region and the expression of the corresponding gene.
Umarov and Solovyev (2017) propose convolutional classifier for building eukaryotic promoter recognition models, which they evaluate on numerous promoter sequences extracted from the well- known EPD database (Dreos et all 2013). In this work, focus Arabidopsis thaliana TATA- promoter/non prommoe sequences, consisting of 1497 samples and 2879 samples respectively, each of fixed length of 251 DNA bases. A minimal eukaryotic promoter region contains transcription start site (TSS) and a so-called TATA-BOX motif at a position As 30 base pair upstream from the transcription start site, which
key feature used for classification. The Arabidopsis thaliana is no stranger to this rule, as the preferred position for the TATA-BOX motif is between -39 and -26 with respect to TSS (Bernard all 2010). Umarov and Solovyev (2017) propose a randomized technique, which we extend to our method, to find the TATA-BOX motif: to discover such sites we train INFO-SEDD such that it can estimate the MI between the sequence distribution and the promoter labels. Then, we use a sliding window of length L and mask the DNA sequence outside this window. Using sliding windows moving from the beginning of a functional site sequence, build MI profile that reflects the effect of random sequence. Whenever the window covers a segment that is irrelevant for determining whether a sequence is a promoter or not, MI remains low. Instead, unmasking the TATA-BOX motif implies substantial increase in MI values. Differently from the method in (Umarov and Solovyev, 2017). our approach is robust to correlated motifs: masking only one correlated motif results in high classification accuracy due the other motif still being unmasked; our method, instead, only unmasks the current motif, which allows assessing its importance without interference from correlated motifs. For MI estimation we fine- tune a pretrained CADUCEUS model, used as the backbone for INFO-SEDD-I, We use the joint variant since its training procedure enables label prediction with DNA sequence masking, while the other variant only models the conditional distribution given unmasked data (more details in Appendix
Figure 5 shows the MI profile between the window and the promoter label: red dots imply a high overlap between the window and the TATA-BOX location, as determined by Bernard all (2010). While known from the genomics literature, INFO-SEDD can effectively locate the TATA-BOX using MI, which makes it an invaluable tool for motif discovery. Note also that other MI estimators would need different training runs for each window, whereas INFO-SEDD natively supports MI estimation between subsets of DNA sequences. This property unlocks applications MI which were previ- ously hindered by inaccurate estimators and poorly scalable methods.

---

Published as a conference paper at ICLR 2026
5 CONCLUSION
The ability to measure the non-linear relation among arbitrary data distributions stands as a corner- stone for scientific discovery, but scalable, and efficient methods accurately estimate information theoretic metrics, without the need for pre-processing workarounds, has only recently attracted the attention of the community. In this work, we contributed novel method tailored estimate such metrics for a particularly challenging case, that of high-dimgg digeensiiiiio discrete data distributions. To the best of knowledge, our method is unique, strated by our experimental validation; moreover, our approach is efficient, as it can leverage existing pretrained models and does not require ad-hoc procedures to work with discrete data.
is scalable, consistent, and accurate, as demon-
Armed with INFO-SEDD, we demonstrated that the ability to estimate MI between discrete random variables is key for a number of applications. In the context of text summarization, MI a mean- ingful proxy to judge performance, and it can be used for model selection. In the genetics domain, we showed that access to accurate MI estimates has the potential of enabling new discoveries using sequencing data, which is discrete by nature. Moreover, the solid mathematical foundations of INFO- SEDD enable exciting extensions to mixed dontintous/discrete data using the Generator Matching framework (Holderrieth et al. 2024), with many potential applications in diverse scientific fields.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We provide the code at the anonymized GitHub repo at the following link https://github. com/ ofw.eertoorreseeee /omtsttimmmiofoooooo diffusion, where provide instructions and utility scripts to build datasets and run experiments. In Appendix C2222 we report the model setup for text experiments and in Appendix C.4.2 Likewise, details and assumptions made in building the datasets are provided in Appendix [C..II] for synthetic dataset, Appendix C.2.1 for text and Appendix C.4.I for DNA. We do performance analysis of the benchmarked methods using the same codebase and the same machine, with the only differences being the methodology in MI estimation and loss functions.
also
ACKNOWLEDGEMENTS
Alberto Foresti, Giulio Franzese and Pietro Michiardi were partially funded by project MUSECOM2- Al-enabled MUltimodal SEmantic COMmunications and COMputing, in the Ma- chine Learning based Communication Systems, towards Wireless AI (WAI), Call 2022, ChistERA.

---

Published as a conference paper at ICLR 2026
REFERENCES
Alexander A Alemi and Ian Fischer. Gilbo: One metric to measure them all. Advances in Neural Information Processing Systems, 31, 2018.
Alexander Alemi, Ian Fischer, Joshua Dillon, and Kevin Murphy. Deep variational information
bottleneck. In International Conference Learning Representations, 2016.
William J Anderson. Continuous-time Markov chains: An appliiccaiionssrrrietteedd approcc Springer approach.
Science & Business Media, 2012.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:1 19911199999999 2021.
David Barber and Felix Agakov. The im algorithm: variational approach to information maximiza- tion. Advances in neural information processing systems, 16(320):201, 2004.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In Proceedings of the 35th International Conference on Machine Learning. 2018.
Anthony J Bell and Terrence J Sejnowski. An cottmmatiion-mamizzziiiii approach to blind separa- tion and blind deconvolution. Neural computation, 129-11199 1995.
Virginie Bernard, Véronique Brunaud, and Alain Lecharny. Te-motifs at the tata-box expected position in plant genes: a novel class of motifs involved in the transcription regulation. BMC genomics, 11:1-15, 2010.
Gyan Bhanot. The metropolis algorithm. Reports Progress in Physics, 51(3):429, 1988.
Mustapha Bounoua, Giulio Franzese, and Pietro Michiardi. S omega i: Score-based o-information estimation. In Forty-first International Conference on Machine Learning, 2024.
Rob Brekelmans, Sicong Huang, Marzyeh Ghassemi, Greg Ver Steeg, Roger Baker Grosse, and Alireza Makhzani. Improving mutual information estimation with annealed and energy-based bounds. In International Conference on Learning Representations, 2022.
Ivan Butakov, Alexander Tolmachev, Sofia Malanchuk, Anna Neopryatnaya, and Alexey Frolov. Mutual information estimation via normalizing flows. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024.
URL
https: http:////peeeeeeww.eoo wttp://www..e/ttoooooooo-------
Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. Advances Neural Information Processing Systems, 35:28266-28279, 2022.
Barry Chai, Dirk Walther, Diane Beck, and Li Fei-Fei. Exploring functional connectivities of the human brain using multivariate information analysis. Advances in neural information processing systems, 22, 2009.
Hongrui Chen and Lexing Ying. Convergence analysis of discrete diffusion model: Exact imple- mentation through uniformization. arXiv preprint arXiv:2402.08000 08095, 2024.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Ad- vances in neural information processing systems, 29, 2016.
Lukasz. Cincio, Jacek Dziarmaga, Marek Rams, and Wojciech H Zurek. Entropy of entanglement and correlations induced by quench: Dynamics of quantum phase transition in the quantum ising model. Physical Review A-toomic, Molecular, and Optical Physics, 75(5):052321, 2007.
Thomas Cover and Roger King. A convergent gambling estimate of the entropy of english. JEEE Transactions Information Theory, 2434):41-4221, 1978.
12

---

Published as a conference paper at ICLR 2026
Maxime Darrin, Philippe Formont, Jackie Cheung, and Pablo Piantanida. COSMIC: Mutual infor- mation for fask-agnostic summarization evaluation. In Lum- Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers). Association for Computational Linguisties, August 2024.
Edo Dotan, Gal Jaschek, Tal Pupko, and Yonatan Belinkov. Effect of tokenization on transformers for biological sequences. Bioinformatics, 40(4):btae 2024.
René Dreos, Giovanna Ambrosini, Rouayda Cavin Périer, and Philipp Bucher. Epd and epdnew, high-quality promoter resources in the next-generation sequencing era. Nucleic acids research, 41(D1):D177-DI64 2013.
Bell Raj Eapen. Genomic tokenizer: Toward a biology-driven tokenization in transformer models for dna sequences. biokxiv, pages 2025-04, 2025.
Andrew W. Eckford, Kenneth A. Loparo, and Peter J. Thomas. Finite-state channel models for signal transduction neural systems. In 2016 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), page 13300-6304 IEEE, March 2016. doi: 10.1 109/icassp.06 7472889. URL http::/tppe/c ww..oo.....oo.. wttp://.ooo........ooo...... 1tt.:/10..11111.....
2016. 7472889
Olivier Elemento, Noam Slonim, and Saeed Tavazoie. A universal framework for regulatory element discovery across all genomes and data types. Molecular cell, 28(2)333--350, 2007.
Alexander R Fabbri, Wojeiech Kryšciníski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. Summeval: Re-evaluating summarization evaluation. Transactions of the Asso- ciation for Computational Linguistics, 9:391-409, 2021.
Marco Federici, David Ruhe, and Patrick Forré. On the effectiveness of hybrid mutual information estimation. arXiv preprint arXiv:2306.0000 2023.
Giulio Franzese, Mustapha Bounoua, and Pietro Michiardi. Minde: Mutual information neural diffusion estimation. arXiv preprint arXiv:23/0.0903 2023a.
Giulio Franzese, Simone Rossi, Lixuan Yang, Alessandro Finamore, Dario Rossi, Maurizio Filip- pone, and Pietro Michiardi. How much is enough? study on diffusion times in score-based generative models. Entropy, 2023b.
Shuyang Gao, Greg Ver Steeg, and Aram Galstyan. Efficient estimation of mutual information for strongly dependent variables. In Artificial intelligence and statistics, pages 277-286. PMLR, 2015.
Weihao Gao, Sreeram Kannan, Sewoong Oh, and Pramod Viswanath. Estimating mutual informa- tion for snnerete-continuond mixtures. Advances in neural information processing systems, 30, 2017.
B. Goebel, Z. Dawy, J. Hagenauer, and J.C. Mueller. An approximation to the distribution of finite sample size mutual information estimates. In IEEE International Conference on Communica- tions, 2005. ICC 2005. 2005, volume 2, pages 1102-1106 Vol. 2, 2005. doi: 10.1 10/10.000.20 1494518.
Katarína Grešová, Vlastimil Martinek, David Čechák, Petr Šimeček, and Panagiotis Alexiou. Ge- nomic benchmarks: a collection of datasets for genomic sequence classification. BMC Genomic Data, 24(1):25, 2023.
Floyd B Hanson. Applied stochastic processes and control for jump-diftusions: modeling, analysis and computation. SIAM, 2007.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2019.
Peter Holderrieth, Marton Havasi, Jason Yim, Neta Shaul, Itai Gat, Tommi Jaakkola, Brian Karrer, Ricky TQ Chen, and Yaron Lipman. Generator matching: Generative modeling with arbitrary markov processes. arXiv preprint arXiv:2410. 20587 2024.
13

---

Published as a conference paper at ICLR 2026
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are uni- versal approximators. Neural networks, 2(5):359-30) -66,,,66,6-1 1989.
Sicong Huang, Alireza Makhzani, Yanshuai Cao, and Roger Grosse. Evaluating lossy compression rates of deep generative models. In International Conference on Machine Leurning. PMLR, 2020.
Jan Karbowski. Information thermodynamics: From physics to neuroscience. doi: 10.3390/e26090779..
Entropy, URL
26(9):779, September 2024. ISSN 1099-4300.
http://dx: doi. 2t.....g/1...390//2222222. 3390/
Frank P Kelly, Reversibility and stochastic networks. PF Kelly-New York: Willy, 1981.
Sergei Kholkin, Ivan Butakov, Evgeny Burnaev, Nikita Gushchin, and Alexander Ko-
rotin. Infobridge: Mutual information estimation via bridge matching, 2025. URL
http:://axxiv.or//aaa..
Xianghao Kong, Rob Brekelmans, and Greg Ver Steeg, Information--thoortie diffusion. In Interna- tional Conference on Learning Representations, 2022.
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Phys- ical review E, 6966)306613,, 2004.
Kyungeun Lee and Wonjong Rhee. A benchmark suite for evaluating neural mutual information estimators on unstructured datasets, 2024. /KKL/tttss//axxx...oz////////// 10924
Nunzio Letizia and Andrea M Tonello. Copula density neural estimation. arXiv preprint 1xXiv:2211153355 2022.
Nunzio A Letizia, Nicola Novello, and Andrea M Tonello. Variational f-divergence and derange- ments for discriminative mutual information estimation. arXiv preprint arXhv:2305.20023. 2023.
Nunzio Alexandro Letizia, Nicola Novello, and Andrea Tonello. Mutual information estimation via f-divergence and data derangements. Advances in Neural Information Processing Systems, 37:105114-105115 150, 2024.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-segee training for natural language generation, translation, and comprehension. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880 Association for Computational Linguistics, July 2020.
pre-
Maxwell W Libbrecht and William Stafford Noble. Machine learning applications in genetics and genomies. Nature Reviews Genetics, 16(6):321-3322 2015.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. stat, 1050:21, 2024.
David JC MacKay. Information theory, inference and learning algorithms. Cambridge university press, 2003.
Michael W Macy, Boleslaw K Szymanski, and Janusz Hotyst. The ising model celebrates century of interdisciplinary contributions. npj Complexity, (1):10, 2024.
Aditya Malusare. Harish Kothandaraman, Dipesh Tamboli, Nadia Lanman, and Vaneet Aggarwal. Understanding the natural language of dna using ancoder-decode foundation models with byte- level precision. Bioinformatics Advances, 4(1):vbael 17, 2024.
David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information. In International Conference on Artificial Intelligence and Statistics, 2020.
Young-Il II Moon, Balaji Rajagopalan, and Upmanu Lall. Estimation mutual information using kernel density estimators. Physical Review E, 52(3)22318, 1995.
14

---

Published as a conference paper at ICLR 2026
Ramesh Nallapati. Bowen Zhou, Cicero dos Santos, Çaglar Gulgehre, and Bing Xiang. Abstrac- tive text summarization using sequencce-o-sequeene RNNs and beyond. In Stefan Riezler and Yoav Goldberg, editors, Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pages 280-290. Association for Computational Linguistics, August 2016.
Garin Newcomb and Khalid Sayood. Use of average mutual information and derived measures to find coding regions. Entropy, 23(10);1324, 2021.
XuanLong Nguyen, Martin Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by penalized convex risk minimization. In Advances Neural Informa- tion Processing Systems, 2007.
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models, 2025. URL
attps://arxinn www./rgg/.....o 2502. 09992
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. F-gan: Training generative neural samplers using variational divergence minimization. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, page 271-279, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.
Lars Onsager. Crystal statistics. i. two-dimensional model with an order-disorder transition. Phys- ical Review, 65(3-4):117 1944.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. Advances in neural information processing systems, 2018.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. Advances neural information processing systems, 30, 2017.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 31 1-318, 2002.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Assaf Pinchas. Irad Ben-Gal, and Amichai Painsky. A comparative analysis estimators for large-alphabet problems. Entropy, 26(5):369, 2024.
of discrete entropy
Stephen M Pizer, E Philip Amburn, John D Austin, Robert Cromartic, Ari Geselowitz, Trey Greer, Bart ter Haar Romeny, John B Zimmerman, and Karel Zuiderveld. Adaptive histogram equaliza- tion and its variations. Computer vision, graphics, and image processing. 39(3):555-36) 1987.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. International Conference on Machine Learning, 2019.
Lifeng Qiao, Peng Ye, Yuchen Ren, Weiqiang Bai, Chaoqi Liang, Xinzhu Ma, Nanqing Dong, and Wanli Ouyang. Model decides how to tokenize: Adaptive dna sequence tokenization with mxdna. Advances in Neural Information Processing Systems, 37:66080-66107 2024.
Arvind Rao, Alfred O Hero, David J States, and James Douglas Engel. Motif discovery in tissue- specific regulatory sequences using directed information. EURASIP Journal on Bioinformatics and Systems Biology, 2007:1-13, 2007.
Carl Edward Rasmussen and Christopher K. Williams. Gaussian Processes for Machine Learning. MIT Press, Cambridge, MA, 2006. ISBN 0-262-18253.4
Yinuo Ren, Haoxuan Chen, Grant M Rotskoff, and Lexing Ying. How discrete and continuous diffusion meet: Comprchensive analysis of discrete diffusion models via a stochastic integral framework. arXiv preprint arXiv:2410.03601, 2024.
15

---

Published as a conference paper at ICLR 2026
Benjamin Rhodes, Kai Xu, and Michael U Gutmann. Telescoping density-ratio estimation. Ad- vances in neural information processing systems, 2020.
Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language 2024.
models. Advances in Neural Information Processing Systems,
Yair Schiff, Chia-Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and Volodymyr Kuleshov. Caduceus: Bi-directional equivariant long-range dna sequence modeling. arXiv preprint arXiv:2403.03254, 2024.
Elad Schneidman, Michael J Berry, Ronen Segev, and William Bialek. Weak pairwise correlations imply strongly correlated network states neural population. Nature, 44007087):1007-1012 2006.
Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379-422 1948.
David Sherrington and Scott Kirkpatrick. Solvable model spin-glass. Physical review letters, 11926):1792, 1975.
Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estimators. In International Conference on Learning Representations, 2019a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019b.
Karl Stratos. Mutual information maximization for simple and accurate part-of-speech induction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers),
2019.
Haoran Sun. Lijun Yu, Bo Dai, Dale Schuurmans, and Hanjun Dai. Score-based continuous-timg discrete diffusion models, 2023. URL http:///wwww..oorrra//////21111 16750
Ryosuke Takahira, Kumiko Tanaka-Ishii, and Lukasz Dębowski. Entropy rate estimates for natural language-a a new extrapolation of compressed large-scale corpora. Entropy, 18(10)3364, 2016.
Andrew Teschendorff and Steve Horvath. Epigenetic ageing clocks: statistical methods and emerg- ing computational challenges. Nature Reviews Genetics, pages 1-19, 2025.
Ramzan Kh Umarov and Victor V Solovyev. Recognition of prokaryotic and eukaryotic promoters using convolutional deep learning neural networks. PloS one, 12(2):c0171410, 2017.
AW van der Vaart and Jon A Wellner. Weak convergence and empirical processes with applications to statistics. Journal of the Royal Statistical Society-Series A Statistics Society, 160(3):596-608, 1997.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-bate based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566 2015.
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, CJ Carey, ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-277 2020. doi: 10.
Sean Whalen, Jacob Schreiber, William S Noble, and Katherine S Pollard. Navigating the pitfalls of applying machine learning in genomics. Nature Reviews Genetics, 23(3):169-18) 2022.
16

---

Published as a conference paper at ICLR 2026
Gerhard Wunder, Benedikt GroB, Rick Fritschek, and Rafael F Schaefer. A reverse jensen inequal- ity result with application to mutual information estimation. In 2021 IEEE Information Theory Workshop (ITW), 2021.
Jun Xia, Shaorong Chen, Jingbo Zhou, Xiaojun Shan, Wenjie Du Zhangyang Gao, Cheng Tan, Bozhen Hu, Jiangbin Zheng, and Stan Z Li. Adanovo: Towards robust emmh robust'.emph (De Novo} peptide sequencing in proteomics against data biases. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluat- ing text generation with bert. arXiv preprint 1rX4iv1904..00 2019.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. A lagrangian perspective on latent variable generative models. In Proc. 34th Conference on Uncertainty in Artificial Intelligence, 2018.
17

---

Information Estimation with Discrete Diffusion Supplementary Material
Mathematical proofs of the main method
19
A.1 Proof of equation 4
19
A.2 Entropy estimation details
21
A.3 Proof of Equation
21
Algorithm Pseudo-code
21
Experimental details
23
C.1
Synthetic experiments
23
C.1.1
Dataset construction
23
C.1.2
Model and training setun
23
C.1.3
Study on training conversence
23
C.I.4
Low dimensional benehmark
23
C.1.5
Sample complexity benchmark
24
1.6
Ablation study on support dimension
26
C.1.7
Extreme dimensionality experiments
26
C.2
Text summarization experiments
28
C.2.1
Dataset and ground truth construction
28
C.2.2
Model setur
28
C.2.3
Numerical results
28
C.2.4
Detailed results for human metrics correlation experiments
29
Comparison of MI estimates with NFO-SFDD other text summarization metries
31
C,4
Genomics experiments
33
C.4.1
Dataset and ground truth construction
33
C.4.2
Model setur
33
C.4.3
Numerical results for the consistency tests
D Additional results
35
D.1
Ising model entropy estimation
35
D.2
Comparison with plug- in estimators
36
Theoretical properties of INFO-SEDD
38
E.1
Assumptions and setting
38
E.2
Error analysis of INFO-SEDI
38
E,3
Discussion on consistency
42
E,4
Discussion bias
42
18

---

Published as a conference paper at ICLR 2026
A MATHEMATICAL PROOFS OF THE MAIN METHOD
A.1 PROOF OF EQUATION 4
We first break log Pr Xu
in E
log PT
Xo
E
- log 9 T X
and apply the Dynkin's 's formula
Fr
separately. We start with E
log P
Xo
B log Fr [o] log (o(Xo)
1-"
Olox (X.,)) Bllog F.Icx..,)de
log +o
L- alos +) E X.) F.()) log F.(X.))
x
log (o( +
lo Dlo (x,,))
E
2, X,) log
F(60) de Xo Fe((())
x
1ng P1, which rewrite using 0p. Q.p.:
We now focus on the term
log P. 8t
8p. 8t
Q,P4 F.
P.
Where the division between numerator and denominator in the last two terms denotes element-wise division. We focus now simplifying the numerator, using the definition in equation [| that
Recalling
(b)ba)) = pra)(a) ((. (1 (1- ((),b), +
- -.
8(a, a)
byfta
F,(0) 2 2r (a. (1 b(a,b)) (b)
(1-((bb)) + - - a(a,a)
byka
we compute the a-th element of Q, p.:
(Q.F.((a)= _d(aat)
-0 + Q.
bęa
P.la) p.())
p.(
bęa
Uzta
Σ (0,070))
(a, 5.00
p.la) (.(a)
P.la)
byo
byka
Σ (0
Σ (a)
byo
byta
Σ
(0,
द (a.b))p)
byta
19

---

Published as a conference paper at ICLR 2026
Finally, if we divide by the denominator obtain:
(a) = Eiza yr- (Bxa) ((,b)b))
Q
pt
p.(a)
Σ byta
(l) - - b) 2.01
P((a)
+ Moreover, we notice that (z, Xx) log the Dynkin's 's formula as:
p,(x)
Qr-
7 -
p.(2)
log p.(m)
we can write
p,(X,)
(x(),)
p,()))
log P pr Pr
log Po Po (Xo)+
Σ (t,
(X (xx Pe(x) + (X(,)
pelx)
log p.(a) dt Xo
px(X,) xx) (x,
Xx
(8)
Then, if we define K(a) a(log 1) and group some terms obtain:
log p
log Po +E S Ex + xx=
F.la) FelX,)
dt Xo
Nt
xx Xo We now repeat similar calculations for E log &r
Firstly, the term
Bog 4 0t
becomes:
log 7. 0t
(a) = Σ (rr-(b)a)) ba)
7.0)
(a(a)
Difaa
Whereas the backward operator term Bilog + (e](X,) 1) becomes:
(a, Xx) log [x(=)
= Qr (X E.(x) log (x(x)
((,))
E.(X,)
(.(.))
Combining everything together gives:
E log 9T Xo
log To (Xo++
Σ Qr-((x - ).(z)
(x(1)) ).(x) log (x( dt Xo
{((x,)
{x.)) ;x1,,)
x*X,
20

---

Published as a conference paper at ICLR 2026
Finally, we can estimate
log
Pr 4
Xo
by subtracting equation 9 from equation
log
Xo
Re
J."
Σ
dr (xe (x
(ele) (x(x)))
+ kata) (.(k,)
be(a) Fx(x,)
log
(a) Tx(x,)
de do
By using the fact that E
IE log
Px
X
= E
log
Pr PT
P PT-C 4s -TT-, Xx =x
7
and by setting T = T t, we get:
Σ x
(x),,
(K
[,(x]
+ (x)
(x(
log [,(x)
dr
F-(X)))
(x(+))
(x(x,)
-x(+))
recovering Equation
A.2 ENTROPY ESTIMATION DETAILS
We provide hereafter for completeness the steps necessary to show that k.(2)
N( for
w.(
* 0:
Emex (e(el
Engex ((a,a0)e-0(0) N.
16 Expex ((0)))))) Mo(ao)
Eagex (1-0- 901---100
Ne - 1)
A.3 PROOF OF EQUATION 6
Consider P,(X, = x, Y. = 9):
P((x = 2, =0)= - Σ (
0,X0 x, Yo = y)
+
(10)
a.
Σ
= Xo = X
2 Xo = z.
= y Yo y)
Pr(Y, jumps to d in (0.t))
= -> Pr(Y, jumps to 0 in [0, t Pr (x, = Xo x) Pr = x, -,,Y==1 =
Pr( jumps to 0 in t)) > Pr(X, = E Xo = x > Pr(Xo = x, -,,Yo=11
=
Pr(X==2)
= Pr(Y,, jumps to 0 in (0, Pr( .)
Equation 10 implies that (X. Pr ma) This important property enables the estima-
tion of mutual information without modifying the score network.
B ALGORITHM PSEUDO-CODE
In this Section show the pseudocodes for the different INFO-SEDD variants.
21

---

Published as a conference paper at ICLR 2026
Algorithm 1 ANFO-SEDD-J: Estimate (x,
Require: Initial sample (Xo; Yo) po. score network 89
I: ~ w0,, (Sample time uniformly)
2: [x Pr(,I[Xo,X Y.l) (Perturb data)
3: 1=0
4: for X; = =
5:
for E [1:N] do
= -1-1 wi-1 JM
6:
n,
....X.1
7: To(0) K (sollXee Vell + so(lXe 0)x.0 - - (e)
(x.,) los (101122.:011x111))
8: end for
9:
end for
10: for ; Y, = @ do
11:
for E [1: [i::]]a do
[Y....... vi-1 ritl 23r
n Y, ...... V.
13: 1+ Ta(1) K (Fol(Eer Vell
(ell Vell ( + =0 P Pelaar) so([xx
18.0. 12)) lox (s0(10, & Fellem)
14: end for
15: end for
16: return
Algorithm 2 INFO-SEDD-CC Estimate [(x,y)
Require: Initial sample [Xa, YoJ Po. score network 8s 1: t ~ u(0, T) (Sample time uniformly)
2: Y,= ~ P.(-Vo) [Perturb data)
3: f=0
4: for Y, = do
5:
for m E [1: N1:N]e do
6:
8 x= 8- - 21-1 git1
a V. .....
,.......... v.
7:
T=(T)()) K (ou( F.ll
su((0. V.Dio.9 co((x,, Koll (2.8 log (su(D)FF.ll).))
8: end for
9: end for
10: return
Algorithm 3 INFO-SEDD-II: Estimate (x)
Require: Initial sample Xo p.,. score network se u(0, (Sample time uniformly]
2: X.
Xo) Poo) (Perturb data)
3: И =
_ 4: for X, do
5: for n E [1:N] N)
-1-1 i+41
6:
X = [x...., X, n, N, ,..., X,
7: #+ To(() (K (K(ng (x(())))) + Mecoo11 =x(xx)xx log (vrodt))T))
8:
end for
9: end for
10: return f
22

---

Published as a conference paper at ICLR 2026
C EXPERIMENTAL DETAILS
C.I SYNTHETIC EXPERIMENTS
C.I.1 DATASET CONSTRUCTION
In this section, we show how we construct the datascts for our synthetic experiments. We gen- erate joint distributions for random variables X, with user-defined mutual information and sup- port sizes XX. Using an evolutionary strategy, we encode the joint distribution in vector Da E XRxxxlxxx and transform it into a valid probability distribution via normalization and reshaping: 9u ---mingg.) 1++1 m19.-mm(g.. where ensures full support. The mutual information, computable in closed form, serves as the selection criterion in the evolutionary process. For large Xxx and lxy the evolution- ary strategy struggles. Instead, we generate high mutual information distributions by concatenating independent distributions, leveraging the additive property of mutual information. Additionally, iso-
morphisms like Cantor's pairing function, <(x y) [x + () + y + 1) + y. enables support expansion without altering mutual information, aiding consistency across experiments.
More formally, exploit the additivity of mutual information with independent random variables to generate complex datasets. By appending discrete noise random variables Zx. Zy to the original random variables x. y, we have yx,,y) [(x, Zx]; [y, Zy)). Pairing functions isomorphisms that map N x N to N. They allow preserving mutual information through the Markov Chain:
[x, [x] x ly, Zy] + -x [x]
Where, by the data processing inequality, [[x, Zx], Zyl) = [[xx Zx}, [y, Zy)) = g(x, - 1(x,9) 1(x,))
have 1[xx Zx] ly, Zyl)
1(x,8) >
In our experiments, we keep the same support dimension for both random variables, increasing it using the following procedure:
1. We sample two binomial random variables 7x, Zy with parameters (n. p). We vary n for increasing the complexity of the experiment and we keep p fixed to 0.5.
2. We concatenate Zx and Zy respectively to x and y, to form higher support versions &,
3. We map the noisy x, ÿ versions to univariate random variables by applying Cantor's map- ping.
C.1.2 MODEL AND TRAINING SETUP
In this section, we provide additional experimental details. For all the methods present the bench- mark,
Multi Layer Perceptron (MLP) with skip connections and around 130k parameters, based on the architecture used in Franzese et all (2023a), reworking the initial layer to include abso- lute positional embeddings. For training discrete score model, match the methodology used by Lou et al. (0024), using the absorb configuration. At inference time, we always take the last valid validation step estimate of each method avoid not-a-number values in our tables.
C.1.3 STUDY ON TRAINING CONVERGENCE
Since all models were trained in the same conditions, we study the convergence the correct MI for the benchmarked methods. As we can see from Figurelo INFO-SEDD is the method which converges faster. Other methods, even when estimating the right MI, take more epochs to settle on the correct value. We also benchmark runtime and maximum memory consumption for the tasks presented in [ We run our experiments on a NVIDIA 1000-XMM-80GB GPU and we observed that INFO- SEDD outperforms all competitors memory consumption (Table 3 and all variational competitors on runtime (Table
1.1.4 Low DIMENSIONAL BENCHMARK
We also benchmark all methods on an easier low dimeeninsional benchmark. We use the same training setup, except that train only for 10 steps using 10a samples and use the same backbone but
23

---

Published as a conference paper at ICLR 2026
Estimator
INFO-SEDD
GAN-DIME
HD-DIME
KL-DIME
MINDE
D-10
346.37
645.81
645.81
440.51
645.81
645.81
645.81
MI=20, D-20
687.22
1301.98
1301.98
130198
876.44
1301.98
1301.98
1301.98
MI=30, D=30
1005.66
1913.15
1913.15
1913.15
1286.35
1913.15
1913.15
1913.15
MI=40, D=40
1345,94
2564.33
2564.33
2564.33
1726.25
2564.32
2564.32
2564.32
MI=50, D=50
1666.30
3180.50
3180.50
3180.50
2138.41
3180.49
3180.49
3180:49
Table 3: Average maximum memory usage over one epoch for different estimators over different MI estimation tasks. Results are in MB. Lowest memory usages are written in bold.
DIME
M0, D=10
2.40
2.89
2.90
2.86
2.16
2.89
2.96
2.94
MI=20, D=20
3.34
3.99
4.02
3.98
2.62
4.03
4.03
4,00
MI=30, D=30
4.51
5.56
5.54
5.57
3.37
5.57
5.57
5.61
D=40
7.37
MI=50, D=50
7.06
8.71
8.69
8.71
4.68
8.76
8.73
8.72
Table 4: Average runtime for one epoch for different estimators over different MI estimation tasks. Results are in seconds. Best runtimes are written in bold.
with parameters. As we can see from Table no clear winner emerges, all estimators besides MINDE perform good job in predicting the right MI.
Estimator
INFO-SEDD
GAN-DIME
HD-DIME
KL-DIME
MINDE
MINE
NWI
SMILE
MI=0, D=10
0.116
0.026
0.019
0.045
2.899
0.018
0,022
0.031
MI=1, D=10
0.868
Ml=2, D=10
1.764
2.073
1.801
1.948
2.389
2.009
1.995
2.361
MI=3, D=10
2.657
2.820
3,098
3.057
3,047
2.960
2.935
3.555
MI=4, D=10
4.121
4.013
3.517
4.094
3,600
4.049
3,908
4.697
MIa5, Dw10
4.953
4.905
4.770
4.652
4.432
4.801
4.692
2.593
Table 5: Results for the low dimensional synthetic benchmark for all estimators. Best estimator in bold.
C.1.5 SAMPLE COMPLEXITY BENCHMARK
The sample complexity of INFO-SEDD strictly depends on the sample complexity of the score net- work. Theoretical guarantees score networks and discrete diffusion models require careful and involved analysis, which we defer to future work. Nevertheless, we run an ablation study on sample complexity, from an empirical standpoint.
We consider pair of random vectors X, Y, both length 20. Each component of X, in turn, has a support of 4. We set the ground truth MI to 10, 20, 30, 40 and 50 and use an MLP score network with skip connections and 160k parameters, and train for 100k steps. iterate over the number of available training samples N as reported in the following table.
Table 6: Empirical evaluation of sample complexity for INFO-SEDD.
N
102
103
10*
10a
MI=10. D=10
7.472
11.139
10.100
10.115
MI=20, D=20
7.772
22.533
20.728
19.922
MI=30, D=30
26.126
29.458
36.093
29.809
MI=40, D=40
28.517
38.512
35,450
38.758
MI=50, D=50
27.782
50.300
43.750
48.822
INFO-SEDD is sufficiently accurate also for rather low number of samples. However, most neural estimators, relies on deep architectures that require a sufficient number samples for training. Practitioners required find the sweet spot between score network architecture complexity
24

---

Published as a conference paper at ICLR 2026
HI Extimaal on CTue 1D)
Exiimal on (Thu Mi 20)
GAN-EENE
SPILE
SPILE
W
MESDE
33
HENCE
Epech
Epech
CIE
HD CIEE
SHILL
SHILL
Dimeeed M1
pha M
KLOOME
200
400
600
800
2003
200
600
000
2000
Extimotioo
GAN DAVE
Trae 53
Figure 6: Empirical convergence of the benchmarked MI estimators.
25

---

Published as a conference paper at ICLR 2026
(which caters better MI estimates for complex data) and the need for large number of training samples.
C.1.6 ABLATION STUDY ON SUPPORT DIMENSION
In the main paper we performed synthetic experiments by fixing D to 10, MI to 0.5 and increasing Ix/. In this section, we perform the contrary and we increase X by using Cantor's mapping and uncorrelated binomial random variables as noise in the data. We increase x as in Table reporting the new support size X after the injectio
noisy binomial random variables. Table shows how INFO-SEDD is the only estimator robust increasingly high support size.
Table Results of the ablation study on |x/. with D fixed 10 and MI=0.5, Best estimates in bold.
Estimator
INFO-SEDD
GAN-DIME
HD-DIME
KL-DIME
MINDE
MINE
NWJ
SMILE
12
0.478
1,457
1.211
1.527
0.254
1.096
1.181
1.597
36
0.486
1.610
1.323
1.507
0.008
1.093
1.128
1.600
132
0.498
1.490
1.188
1.511
0.005
1.170
1.245
1.818
C.1.7
EXTREME DIMENSIONALITY EXPERIMENTS
In the main text, evaluated INFO-SEDD synthetic benchmarks with increasing dimensionality D and mutual information (MI). Here, we further assess scalability in very figh-dimensionll regimes and additionally compare against GAN-DIME, DID-DIME, and SMILE for the case where the ground- truth MI is 0.5.
Case D = 256 and MI=0.5 We trained INFO-SEDD using the same architecture as in section 4.1 with an extended budget of 10 optimization steps. rapootr MI estimates for all methods.
Table 8: Extreme dimensionality experiment with D = 256 and ground-truth MI = 0.5.
Method
Estimated MI
INFO-SEDD
0.43
HD-DIME
0.43
SMILE
0,67
GAN-DIME DIME
0.51
While competing methods remain relatively stable in this regime, INFO-SEDD provides a competitive and well-calibrated estimate.
Case D = 1024 and MI=0.5 For D = 1024, we adopted a DiMamba backbone with 2M param- eters, trained for 10 steps with early stopping on a dataset of 10 samples. Results are reported in T
Table 9: Extreme dimensionality experiment with D = 1024 and ground-truth MI = 0.5.
Method
Estimated MI
INFO-SEDD
0.53
HD-DIME
0.00
SMILE
0,00
GAN-DIME
0.00
In this extreme regime, all competing methods collapse to zero, whereas INFO-SEDO remains rate and close to the ground-truth value. These results provide further evidence the robustness and scalability of approach in very high-dimensional settings.
26

---

Published as a conference paper at ICLR 2026
Discussion These results indicate that INFO-SEDD can be effectively scaled very high- dimensional discrete distributions when trained with sufficient data and capacity. Together with the experiments in the main paper, they highlight both the robustness of our approach and its ability to leverage larger training budgets when available.
27

---

Published as a conference paper at ICLR 2026
C.2 TEXT SUMMARIZATION EXPERIMENTS
C.2.1 DATASET AND GROUND TRUTH CONSTRUCTION
We consider random variable X, which correspond to source texts in Fabbri et al. (2021), and summaries Y obtained with the BART model (M22 of the SummEval paper) when fed with the corresponding source text. Then, for given 0 < p < 1, we construct Y which is equal to the original Y with probability and equal to an independent sample from the same distribution of Y with probability 1 1- p. Although we can bound its value. Consider the latent variable Z, Bernoulli distributed with parameter which determines whether Y is equal to the original Y or and independent sample from the same distribution. Then
cannot establish the ground truth MI I(X,, Y") exactly,
T(X, YP) [(X T(X; + 1(X, 2)
= =[(X; I(X;
2 =T(XYYIZZ) H(Z(Y))
2 21(X;Y9IZ) H(Z)
=pI(X;Y'IZ= = 1) (1 -0 -1(
=PI(X;Y'1Z -
-T(X;V) - -H)
Furthermore, since 1(X, = - - I(X; < I(X:YPIZ) pI(X;Y). can claim that Y) - (plog (1-p)])))))) p) log p)), whenever I(X;Y) is much greater than HZZ), I(X; satis-
(x)(X) HI
Since H(Z)
fies approximately the linear relationship 1 (X;YYPIZ) xXX Y). This assumption supported
by the literature (Takahira et al 2016)).
C.2.2 MODEL SETUP
For all our methods, we use the MDLM-small model (Sahoo et all 2024) trained on WebText While this family of models does not directly parametrize the score, Sahoo et al. (2024) show that can be easily recovered from the output of the model. For the experiments with MINDE, we use the variance preserving Stochastic Differential Equation (SDE) and the same model. We perform the diffusion step on the token embeddings. We trained all models using the configuration in Table [T]
Table 10: Training settings for one for the model selection experiments and the consistency tests (one value of p)
Parameter
Value
Backbone
MDLM small (Sahoo et al,, 2024)
Optimizer
AdamW
AdamW 3
0.9
Adam W 3g
0.999
Weight decay
0
Learning rate
3e-4
Batch size
64
Number of steps
3600
Dropout rate
0.0
Hardware
IXNVIDIA A 100 80 GB
C.2.3 NUMERICAL RESULTS
In Table 11 report the numerical results for MI estimations in the consistency tests, including MINDE. We report all values in nats.
28

---

Published as a conference paper at ICLR 2026
Table 11: Numerical results for Text summarization consistency tests
SMILE
0.00
0.00
0.09
0.89
2.06
5.84
7.19
12.46
13.85
14.73
20.55
NWJ
-0.01
0.00
-0.10
-0.24
0.19
0.84
MINE
-0.50
-0.02
0.02
0.00
0.84
0.33
-4.11
-3.68
-3.90
-3.96
-3.96
MINDE
365679296 00
198489552.00
347922112.00
12828973.00
33884712.00
19269294.00 4.00
270945856.00 00
391443968. 00
448679872.00 .00
1881 00818858.00 00
124222336.00
00
KL-DIME
-0.01
0.01
0.48
3.03
3.21
4.63
5.26
5.17
6.38
7.95
6.82
I-FO-SEDD-G INFO-SEDD-J
106. 66
157.90
197.18
220.47
244.53
255.42
276.28
303.20
316.01
330.76 76
361.23
7.52
34.02
61.25
90.17
124.48
170.03
230.41
308.30 30
109.26
505.13
602.77
HD-DIME
0.20
-0.01
0.13
2.41
10.57
16.17
12.94
28.98
27.12
39.42
GAN-DIME
9.63
9.28
9.33
19.44
0.0
0.1
0.2
0.3
04
0.5
0.6
0.7
0.8
0.9
1.0
p
C.2.4 DETAILED RESULTS FOR HUMAN METRICS CORRELATION EXPERIMENTS
We report the MI estimates of each model and compare them with human evaluation metrics. For each metric, we run a GP regression with a Matérn kernel with smoothing parameter y 1.5 (Rasmussen and Williams, 2006) and noise level 0.01 using the MI estimates from both INFO-SEDD- and INFO-SEDD- SEDD-J. We use the scikit-learn Python package to run this analysis (Pedregosa et all 2011). In Figures Z and g we plot the GP mean, its 95% confidence interval, and the Pearson correlation. As the figures show, several human metrics saturate around 5.0 (the maximum score possible), while MI is not subject to this ceiling effect.
29

---

Published as a conference paper at ICLR 2026
In particular, for the consistency and fluency metrics, the GP mean follows increasing trend until the human scores saturate, suggesting that the linear relationship between MI and the human metric only holds up to threshold, beyond which most models achieve the maximum score.
Unlike human metrics, however, INFO-SEDD does not rely on annotations: it only requires a dataset of summaries paired with source texts, making it cost-effective and scalable alternative for model selection in text summarization.
4.5
P9A
4.0
Colorence
412
2.5
400
450
650
700
750
800
Mutual Information
5.5
rend
0.133
5.0
M51
400
450
500
550
500
650
700
750
800
Mutual Information
Unear send (.77)
Suunce
3.5
800
4.50
95%
4.25
3.75
3.25
3.00
Mutual Information
3.5
3.0
400
500
650
700
750
800
Mutual Information
Figure 7: Comparison of human metrics against mutual information estimated by INFO-SEDD-C
30

---

Published as a conference paper at ICLR 2026
4.5
400
4.0
Coheeene
RC3
2.5
Information
Har
3.5
2.5
5.5
30%
MRK..
1/h
Hueroy
4.0
250
275
300
325
375
Information
4.50
95%
4.25
bandly 3228)
4.00
3.75
3.50
3.25
801
2.00
250
275
300
325
350
375
400
425
Mutual Information
Unear brend 3322)
Oveal
4.0
250
275
300
325
350
375
400
425
Mutuall Information
Figure 8: Comparison of human metrics against mutual information estimated by INFO-SEDD-J
C.3 COMPARISON OF MI ESTIMATES WITH INFO-SEDD AND OTHER TEXT SUMMARIZATION METRICS
We measured several other metrics using the SummEval library (Fabbri et all 202 and one refer- ence summary for each sample provided in the SummEval collection. In particular, we benchmark the BERT SCORE metric (Zhang et all 2019). the BLEU metric (Papineni et all 2002) and the CIDER metric (Vedantam et al.] 2015). As SEDD-J in general correlates better with other summarization metrics with respect to INFO-SEDD-C. Interestingly, INFO-SEDD-C correlates better with human metrics, proving to be useful proxy for
from Figure [0 and Figure 9 we find that INFO-
31

---

Published as a conference paper at ICLR 2026
human metries without the need of human annotations. Notably, INFO-SEDD- is also the best correlating with consistency among all other text summarization metrics.
2.02
0680-1
1.00
0.53
0.48
0.43
0.35
-0.01
0.09
0.38
0.28
0.26
0.02
-0.28
0.53
1.00
0.02
-0.12
0.13
0.10
0.09
272
252
0.35
0.82
-0.07
0.92
1.00
0.14
-0.07
0.27
0.20
0.22
-0.13
-0.27
60H
-0.01
0.01
0.26
0.18
0.14
1.00
0.26
0.52
0.68
0.70
0.05
0.10
CON
0.09
-0.12
0.47
0.01
-0.07
0.26
1.00
0.56
0.50
0.52
0.49
0.50
-6.00
0.26
0.09
0.52
0.26
0.52
0.77
0.98
1.00
0.22
18801
0.02
-0.22
0.23
-0.05
-0.13
0.05
0.49
0.15
0.22
0.24
1.00
0.47
-0.75
1000
00.28
-0.38
0.10
-0.22
-0.27
0.10
0.50
0.13
0.20
0.22
0.47
1.00
Figure 9: Kendall Tau correlation heatmap of MI estimates INFO-SEDD, other text summarization metries and human metrics, CID=Cider, BLE=bleu.
3.03
8830 F1
1.00
0.80
0.89
0.94
0.72
0.11
0.55
0.50
0.56
0.46
0.52
0.21
0.88
1.00
0.57
0.94
0.94
0.11
0.34
0.34
0.41
0.32
0.18
-016
277
252
0.72
0.94
0.35
0.88
1.00
0.29
0.33
0.37
0.45
0.39
0.04
-0.21
0.11
0.111
0.09
0.26
0.29
1.00
0.46
0.60
0.82
0.81
-0.09
0.21
500
CON
0.55
0.34
0.65
0.59
0.33
0.46
1.00
0.97
0.76
0.89
0.55
0.74
-8.50
0.46
0.89
0.94
1880 0.52
0:18
0.73
0.41
0.04
-0.09
0.55
0.46
0.29
0.32
1.00
0.68
-0.75
1000
0.21
-0.16
0.51
0.16
-0.21
0.21
0.74
0.65
0.41
0.57
0.63
1.00
Figure 10: Pearson correlation heatmap of MI estimates INFO-SEDD, other text summarization metrics and human metrics. CID=Cider, BLE=bleu.
32

---

Published as a conference paper at ICLR 2026
C.4 GENOMICS EXPERIMENTS
C.4.1 DATASET AND GROUND TRUTH CONSTRUCTION
To construct our dataset for the consistency test, we consider the shuffling hyperparameter We consider X be the random variable representing the DNA sequences and, for given 0 < < 1, we construct YP which is equal to the original Y, the random variable representing the human or worm label, with probability 1- - and equal to an independent sample from the same distribution of Y with probability p. We estimate [[x, Y*) by considering the entropy terms which correspond to the decomposition (())) H On the one hand, estimation of H (Y*) = HY) from the dataset relatively simple and stable given the low dimensionality of the label. Concerning H (Y)))), we first write the conditional distribution PY (y (c)) as:
Proxx (vp xz) = Σ (
Σ P (y
Σ(1 (1 pp) v prrpx(ulz) + PPY (or)pp(x((l))
p)prix + ppy (VP)
In the considered dataset, classes balanced and consequentlypy (v²) 0.5. To approximate Py ()), we first define ((x) = arg max, Prix (y/ x), then we assume the following:
{
for y ((x)
(rix(U|x)=
otherwise
with < 1, which roughly speaking corresponds to assuming that for all values of the conditional probability highly skewed either towards zero or one. This assumption proves to be reasonable in scenarios where the classification accuracy is close to one, as the case considered here (Schiff et al. 2024). Then
Py))((p11>) (1- (1-p)(1- - + pV) (1-p)c+ ppy (3²)
for y l(x)
otherwise
Going back H(Y* X), can finally write:
H(Y*IX) = Σ
- px(x)) Σ (o
log
(xxx (1z))2))+
+ PY - log (oxx ((x){=))
Σ
p)(1 - + (v²))
Σ px(x) p)e + ppy ppy log p)e ppr (pr
-1) (0.5) = Ho p) - p0.5)
To select e, we choose the accuracy of the best classifier from (Schiff et all 2024) trained for the HUMAN VS WORM classification task.
C.4.2 MODEL SETUP
We employed a pretrained Caduceus model Schiff et al. 2024) as a backbone for all estimators. We modified the head of the model when needed. For MINE-like estimators added an attention head
33

---

Published as a conference paper at ICLR 2026
and added fully connected layer for aggregating the output. For MINDE, added transformer block with time conditioning. For INFO-SEDD, we did not make architectural variations and we employ the optimized training strategy highlighted in Appendix C.1 We trained all models using the configuration in T2 For the motif selection experiments, we employ INFO-SEDD-J because method requires masking elements of the DNA sequence, which is something that INFO-SEDD-C does not do during training.
Table 12: Training settings for one run for the promoter experiments or the consistency tests (one value p)
Parameter
Value
Backbone
Caduceus- IK (Schiff et al,, 2024)
Optimizer
AdamW
AdamW $1
0.9
AdamW 32
0.999
Weight decay
0.1
Learning rate
0.001
Batch size
128
Number of steps
20000
Dropout rate
0.1
Hardware
1XNVIDIA A 100 80 GB
C.4.3 NUMERICAL RESULTS FOR THE CONSISTENCY TESTS
In Table 1 we report the numerical results for MI estimations in the consistency tests, including MINDE. We report all values in nats.
Table 13: Numerical results for DNA consistency tests
GAN-DIME
HD-DIME
INFO-SEDD-C
INFO-SEDD-J
KL-DIME
MINE
NWJ
SMILE
0.0
0.01
0.00
0.00
0.05
0.00
0.00
-0.02
0.00
0.1
0.00
0.01
0.00
0.07
0.00
0.00
-0.01
0.00
0.2
0.02
-0.01
0.00
0.06
0.00
0.00
-0.02
0.02
0.04
0.01
0.08
0.00
0.00
0.4
0.00
0.07
0.07
0.06
-0.01
0.00
0.00
0.07
0.5
0.04
0.14
0.12
0.09
0.12
0.13
0.13
0.12
0.6
0.18
0.16
0.18
0.09
0.17
0.19
-0.02
0.19
0,27
0.23
0.24
0.14
0.30
0.29
0.00
0.8
0.34
0.33
0.32
0.18
0.00
0.38
-0.01
0.29
0.9
0.15
0.43
0.45
0.24
0.51
0.54
-0.02
0.40
1.0
0.69
-0.01
0.61
0.37
0.66
0.71
0.70
0.50
34

---

Published as a conference paper at ICLR 2026
D ADDITIONAL RESULTS
D.1 ISING MODEL ENTROPY ESTIMATION
The Ising model is system consisting of particles arranged in lattice. In experiments, consider a L x L square. A particle i of the lattice is associated with a discrete value o; E {-1,+1} called spin and each pair of particles ij characterized by an interaction strength Jij. With no external fields, these quantities determine the energy ((r) of the configuration d:
B(a)
In turns, the energy of a configuration determines its likelihood. In particular, the configurations of the Ising model follow a probability distribution Po parametrised by the temperature T, the Boltzmann constant ki and the interaction strengths:
6 -B(()
Po(o)
Z(T)
Where Z(T) = E; e (B(a)) and 8 (1,T)-1 order to generate our dataset from Po: we follow the Metropolis algorithm (Bhanoi 1988):
Algorithm 4 Metropolis Algorithm for 2D Ising Spin Glass
1: Input: Lattice size N, interaction strengths Jij. temperature T, number iterations iter_max
2: Initialize: Spin lattice with oi {[1, randomly assigned
3: for iteration = to iter_max do
4: Randomly select lattice site
5: Compute the change in energy AE if osj is flipped:
AE = 2a; Jis
6: Generate random number r uniformly distributed in [0,
7: if exp (-BAE) then
8: Flip the spin: a; f- -0;
9: end if
10:
end for
11: Output: Final spin configuration
We compute the entropy of Pu analytically, starting from the free energy F' per site of the lattice:
F(T) k log Xr
Where Ay is the partition function, which depends on the interaction horizontal and vertical interac- tion strength. For simplicity, we consider the same interaction strength J = 1 for all neighboring particles, while we set it to zero for non neighboring particles. Under these assumptions, we can calculate log with double integral(Onsager, 1944):
log log 2+ 1. (."% log(cosh(23J)) sinhh2)))) (in)))))ssnnh.))))) cos(@=))dd
For simplicity. we also set ha 1. From F, we can calculate the entropy H using the thermody- namic relation
8T We compute integral numerically using SciPy Python package
(Virtanen al. 2020 and approximate H H F(T-AM) with AT 10
2AT
For what concerns the INFO-SEDD architecture, we keep a single model configuration for all temper- atures, both for the model, which contains around 90k parameters, and for the diffusion. To get the
35

---

Published as a conference paper at ICLR 2026
Entropy per site in the lsing Model
Entropy site
Simulation Results
0.6
Critical Temperature
+
+
Entropy (Nats)
0.4
0.3
0.1
X
0.0
1.0
1.5
2.0
2.5
3.0
3.5
4.0
Temperature (Kelvin)
Figure 11: Ising model entropy estimates
entropy per site, we divide the estimates of the model by the number of particles the configurations (400).
In this section, we numerically validate the performance of INFO-SEDD by estimating entropy ac- cording to Algorithm 3
Spin glasses experiments. Entropy computation in Ising models enables insights the thermo- dynamics properties the system (Cincio et al. 2007). which can be used for scientific discov- in the domain where the Ising model is applied (Macy et al. 2024 Schneidman et al. 2006 Sherrington and Kirkpatrick, 1975).
Experimental setup We consider a simplified Ising model applicd to spin glasses (hhrriigtonn and Kirkpatrick 1975). We do not include external field, set unitary inter- action strength for all the sites interactions, unitary Boltzmann's 's constant and a 20 x 20 square lattice. The entropy per site of this configuration We test our model by estimating the entropy per particle at linearly spaced temperatures from 1.0K 4.0K. We generate for each temperature (see Algorithm 4 We post-process the output of INFO-SEDD by dividing the entropy estimate by 400 report the entropy per site.
be computed in closed form Onsager (1944).
dataset using the Metropolis-Hastingg algorithm, with 10000 samples
Results and Analysis Variational estimators cannot estimate large KL divergences reliably with limited samples sizes [McAllester and Stratos, 2020; Song and Ermon, 2019a). In this scenario, instead, INFO-SEDD accurately estimates large KL divergences (Figure [. performing particularly well at low temperatures where we need to estimate large KT. divergences.
D.2 COMPARISON WITH PLUG-IN ESTIMATORS
In this section, discuss the limitations of plug-in estimators for discrete random variables. First, we analyze the approach by Goebel et all (2005), where the authors use Taylor expansion to es- timate MI. However, by construction, this estimator assumes low MI regime. This assumption makes the method inherently unsuitable for high-MI estimation tasks, where higher-order terms in the expansion become non-negligible and induce severe bias.
We further analyze the approach of Gao et al. 2017)), who propose a variation of the KSG esti- mator handle mixtures of continuous and discrete data. We benchmark this estimator the
36

---

Published as a conference paper at ICLR 2026
high-dimensional synthetic benchmark, in Table | and on the extreme high-dimensional setting
introduced in Appendix [C.I.7] Results are reported in Tables [4 and [5]
Table 14: Plug-in estimator (Gao et al. (017)) on high-MI synthetic benchmarks with = MT.
Setting
Estimated MI
MI=10, D=10
11.56
MI=20, D=20
9.33
MI=30, D=30
10.55
MI=40, D=40
2.87
MI=50, D=50
4.18
We observe that while the estimator remains reasonably accurate for moderate MI (e.g., MI=10), its performance quickly deteriorates as dimensionality and dependence strength increase.
We additionally evaluate the estimator in the extreme dimensionality setting with low ground-truth MI (0.5), introduced in Appendix C.1.7
Table 15: Plug-in estimator (Gao al. (017)) in extreme dimensionality regimes with ground-truth
MI 0.5.
MI
Estimated MI
0.5
256
14.17
0.5
1024
14.30
In this regime, the estimator overestimates the mutual information. This behavior highlights the instability plug-in approaches in high-dimensional discrete settings.
Overall, these experiments confirm that plug-in estimators based Taylor approximations neirhot-n corrections are not robust to either high-MI or high-dimensional regimes. In con- trast, INFO-SEDD remains stable and accurate across both challenging scenarios.
37

---

Published as a conference paper at ICLR 2026
E THEORETICAL PROPERTIES OF INFO-SEDD
We highlight the main assumptions and setting in Appendix E.1 then in Appendix E.2 we proceed construct an error bound on the KL estimate INFO-SEDD under the established setting.
E. ASSUMPTIONS AND SETTING
We highlight the assumptions on the probability distributions and score networks. We first define the class of functions F which restricts the class of score functions and score networks. In this work, we consider CTMC with only sparse transition rate matrices Q. which constrain the CTMC to modify only one subcomponent at time. This means that, when considering the KI. estimate as in
Equation 4 only ratios of probability distributions P.(a) with and within Hamming distance 1 (b(
are considered. The assumptions that we are going to make in this section reflect this characteristic.
We define the pairs of samples Hamming distance as follows:
S = ((x,, x²) E x x x" dhamming (z', x²) = 1}.
Where x is the support of D-dimensional random vectors We impose two finite constants
0 O<Ci < C2 < +00,
and define
F: (f S 0, - R C1 C 1lfll S C2).
where |// denotes the supremum norm over S x [0, T].
Throughout this appendix, we use family of neural networks Te C J that are standard multilayer feedforward networks with bounded, nonconstant activation functions (Homiik al., 1989). We also fix a perturbation kernel Prjo and denote the score function for arbitrary probability distribution p as
(**)); = (.(
(x()))
where (x(x))= = E., Peo(aa)zo)p(zo))
We now state the assumptions score networks and on score functions:
* A.1: For all sy E To. C s 11, I C2.
A.2: For all p, the score function satisfies Ci S || C2.
The lower bound C₁ > 0 avoids degeneracies, and ensures well-defined log terms in the loss. We define the class of distribution P for which A.2 is as
P = {p C1 S || S CC2
A.1 and A.2 are similar to Assumption 4.4 of Ren al. (2024), who assume that both the score function and the score networks are bounded. Similarly, Chen and Ying (2024) set bound for the score network (Assumption 2) and for the true score function (Assumption 3).
E.2 ERROR ANALYSIS OF INFO-SEDD
Let p, q E P be two probability distributions satisfying the assumptions highlighted Appendix Let 0, O be two arbitrary sets of parameters for the score networks sy
6p
|1
Eg = |1
8
Then, let us define the functional ε, parameterised with arbitrary functions f.g E F and perturbation kernel Pejo
38

---

Published as a conference paper at ICLR 2026
E(f.9: 2o) = S-" R.
E (K(((M)) + f(x4)2 - g(xe) log (ogg()))))dd
Then, the INFO-SEDD estimator for the KL is
Es
The true KL divergence, instead is exactly
Ip ll g) KL Pr gT)
Exxue 59;2)
Then, chracterise the error of AAFO-SEDD
BempEl (::a) kL. Pr II Trl Ez (s", a=aa) & EenpE( Banze (0))) + +KL
|
(1)
We denote @P the element in x" such that 0P = i and we focus first the second term of the right hand side of Equation (I]
Σ (T()) log Pr(a) GT
KL
Pr
(()
Firstly, note that for any t (0, (.(0)) (0%) and as consequence log 2.(0%) log(1) 0. This means that write
(02)
Σ (T( log Pr(w) (r(#)
KL
PT
Σ PT
log (r(#)
7 (
ææ
S Σ
(
log
(T))
Σ Dg0
(T(
log
(#)
4r (m)
Note that we cannot bound (ف(ي by simply applying A.2, since the sum involves terms such that (a)
dramming (z. 2) > 1. We can however construct an ordered sequence of points S(z, 2) where x is the first element, is the last element and Vi,j such that S < j chamming (I, f) holds (Samming(SSzr,25 S(x,, f);)) = - i. This allows to write
drammina (m.f)) II
(8) (xr)
g (S(2.))44))
(S(r,6))
1=0
8dh (S(x,, ())
i=0
C2
2=0
CD
39

---

Published as a conference paper at ICLR 2026
- pr |l gT)
We can now go back to KL
qT S Σ (T(x) log
Σ
(2)
KL
Pr
7x(x)
Σ PT (a) log
Σ
I70D
Σ 10g) CCP
z/0P
(1 PT(00)) 1x(C2).
We can now focus on the first term of the right hand side of Equation ||]
E Exp Esg. - ExuPE (s", 8";z)
First, we decompose each term and use triangle inequality
E Bxn (2:)) E
Exnp ((16)))1z) - E((s", =(xx)==)=z)
Bowp f" T
Σ 2) K( ((sh(1=)))) - (T)((z))))
#$
+ ( (x1)) (((r)) log (gg())())))))
+ 5 (x1) log(s (1)()(()))))) de
< Ezup 1"
2
B P P.o
E kt)
((s)(1))))) - (x)(())))))
(12)
20000
22 (xx)2 (xxr)
(x()))) a (og(s%((2) (2))2) ((())) log(s ((())) dt
+
40

---

Published as a conference paper at ICLR 2026
We the inequality log(a) S 1 to proceed with the first term of Equation 12)
((()(1))))) (s)())))))
1) (s (log (s" 1)]
bo (2r))) a log( (1x))))) (6 (1x)))) log (5 (x1)s)) + Ep
max(("
(x" (ze 2 og (zt )2) ((0)() - log (x) +cp
C2 log
C22
max (x)): (1r))))
(x))), (or)ss) - 1) t
< <C2
(#) (o))s)
(())))) + Ep
min(( ((4)
(0%(20)) - 3 (x))a)) t
$ E (c)((1)) - (x(x))) t
c t
cp + ))
Then, we continue with the second term of Equation
12 which is simply
(Te)) - s (((r) S cq
Finally, we write the term of Equation
(3)))) log s (x))s) - S (xx)) log (((z))2)
- max (3)))) 87 (1x)8) log (x()) - log( -=
2 C2 log 5 (xt)s - log (0(((((z)))))
(x()) 2((21)2
0 c
Combining the results we obtain
Ex (sg. sg:z)
EzupE (8p,)9;;))
6-' E P2oppeoo(:10)
Σ [(1r)) (lep + (1+8)) dt
<
aeptar
s so(TDD|x ((ep (c:)))11 (1+ C2
where the factor D|xl is the number of non-zero entries in Q using the absorb configuration. We can finally put together everything get the error bound for Equation (1]
E SempE(sh (sp. - KL Pr ll ]r EEuDE (s", 5²;
$ PempE E5y 8g:z) ExnpE (s²,89;x)) kI. Pr ςT)
< <o(TDill ((x) Eq) (1+
[: (1 (1- lx(C2)
Note that (0P) approaches 1 exponentially in T (Equation 14 of Lou et all (024)). The other term, instead, shows that the error on the KL increases linearly with the error of the score network for fixed 7' and does not depend on terms exponential on the KI. divergence.
41

---

Published as a conference paper at ICLR 2026
E.3 DISCUSSION CONSISTENCY
As presented in Appendix E, the error of INFO-SEDD decomposes in two terms: one proportional to the error of the score networks, and one proportional to (70)))
Ex-pE( 22)xz) KL
Pr dr Exupe E (s s0 ;
- BempelS (sg, sg;z) Exupe E (s" KL Pr बT)
<T(T)D)xl (ep) (1+ (1+8))
& +
(0 )) (1- () log lx(C2)
(13)
The second term of Equation 13) cannot be reduced with better on the CTMC and the data distribution. However, decreases exponentially ((T) (Equation 14 Lou et all (2024)), establishing trade-off with the first term, which only increases linearly with á (T). For this reason, we focus on the first term. Note that our final KL estimator Equation (4) continuous functional of the score networks sy sg By theorem 3.4 of Lou et all (2024), the DWDSE loss is an equivalent objective compared to the score entropy loss. This allows proposition 3.2 of Lou et al. 2024),, for which, given enough samples and model capacity, the score network si and sg converge their true values s and 84. Our final KI. estimator Equation (4) is continuous functional of these score networks. By the Continuous Mapping Theorem (theorem 1.9.5 of Vaart and Wellner (1997)), assuming as in Lou et al. (2024) that s and s converge probability to their true values s" and s", the output of the functional (our KL estimate) must also converge in probability to the true KL divergence minus the truncation bias
networks, only depends
apply
E.4 DISCUSSION BIAS
Consider the INFO-SEDD estimator
Σ 2((x1,))) ( (s(())2) ((()))r xP(x))= sa (((())a di
X
and its empirical version over a collection samples sd ~ Paro
N â
1 N
Σ ayfta
log s² dt x) (s) + sx))2 - sx(t))
2. (x),,x (K (sP(x))a) + (a()) - sP(xt) log 0 (xt)x): then we
Let us denote d(x;) Em can write its expectation as
N
E
d(x))dt
N
E
8 ((x:)dt
Note that E [Io" d(a))dt the true a. As a consequence [[ = a, which means that INFO-SEDD is unbiased estimator for Note, however, that still have the truncation bias which is bounded by
KL [Rate 4 al < (1- (O)) Ix(C2).
This source of bias is object of tradeoff with the other term
(TT) ((e + en) (1+
42

---

Published as a conference paper at ICLR 2026
since increasing T would reduce the former but expand the latter. However, the term (1 Pr (0?) decreases exponentially in (TT) (Equation 14 of Lou et all (2024)). This means that the trunca- tion bias can be drastically reduced without affecting the other error term which only has linear dependence on (T)).
43


Reviews:
[{"review_id": "dGCeKFotbn", "reviewer": "Reviewer_jnfn", "summary": "This work extends the recent trend of information theoretic diffusion based estimators to discrete data.\nThe approach is \"decomposable\" in the sense that a single training run to estimate the mutual information of a vector allows for the estimation of that of any subvector.\nEmpirical evaluation is extensive, both accuracy of estimation and quality of learned representations are evaluated in multiple environment of increasing scale.", "questions": "Can do author suggest more direct application of Info-SEDD to machine learning tasks?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper is well written.\n+ Advanced stochastic calculus concepts are explained simply, making the paper accessible to wider audience.\n+ Estimation of mutual information on discrete data is promising and useful.\n+ Amortized inference on subsequence is bound to be useful in many situations.\n+ Evaluation on genomic data is a beautiful application of the estimator.", "weaknesses": "Most of the weaknesses have to do with the empirical evaluation.\n+ Table 1: Missing error bars.\n+ Evaluations are mostly presented as a graph which can be misleading.\n+ Table 2: Correlation with human metrics only report INFO-SEDD and no baselines.\n+ Overall evaluation feels somewhat indirect. While having the merit of being grounded in real life problems,  the empirical evaluation could be made stronger by evaluating the estimator itself as an objective for a clearly defined machine learning downstream tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y4xhNdVrnb", "reviewer": "Reviewer_C1dg", "summary": "The paper proposes a new algorithm for estimating the KL divergence (and consequently, entropy and mutual information) in high-dimensional discrete distributions. The method expresses the KL divergence as a conditional expectation and applies Dynkin’s formula to derive a KL estimator that depends on two density ratios. Each density ratio is then approximated using a diffusion-weighted denoising score entropy loss. The method is evaluated across a range of mutual information and entropy estimation tasks.", "questions": "Could you provide a proof that the proposed estimator is consistent and unbiased? For example using a similar strategy as the proof of consistency of the score entropy loss in Lou et. al. (Proposition 3.2).\n\nWhat is the computational complexity of the proposed estimator?\nRegarding the broader impact, the paper would benefit from a discussion on sufficient statistics learning (e.g., see [2]). Including a computational complexity analysis would clarify whether the method can be feasibly used as a loss function in downstream tasks.\n\n[1] Lou et. al. Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution, 2020.\n\n[2] Y Chen et. al. Neural Approximate Sufficient Statistics for Implicit Models, 2020.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method for estimating mutual information in discrete data is novel. The reviewer is not aware of any previous works that apply discrete (CTMC-based) diffusion to the mutual information estimation task.\nThe method shows potential practical utility. The results on estimating the entropy of Ising models are promising, and the reviewer believes the approach could be applied to study phase transitions in more complex statistical physics systems with discrete microstate space. As another practical utility one can apply the proposed method to learning neural sufficient statistics and representation learning.", "weaknesses": "It is unclear whether the proposed mutual information estimator is consistent and unbiased. Additionally, although the error analysis is included in the supplementary material, it is not discussed in the main text.\n\nIn general, the theoretical contribution of the proposed method is limited and primarily based on a reformulation of the results by Lou et al [1]. For example, the proposed KL estimator has the same form as the diffusion-weighted denoising score entropy (Equation 10 in Lou et al.), with the main difference being the replacement of the density ratio. Moreover, the use of Dynkin’s formula was already presented in Lou et al. (Theorem 3.6).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XV9V8iUHSt", "reviewer": "Reviewer_ps2M", "summary": "The authors propose a discrete diffusion-based approach that mitigates the problem of continuous space MI estimators in high data dimensionality. The proposed approach builds upon Dynkin’s lemma and can be used to estimate mutual information or entropy. Numerical experiments validate the proposed estimator compared to other state-of-the-art estimators. The authors design consistency tests for real-world textual and genomics data.", "questions": "- What is $\\delta()$ in equation 1? It is not defined\n- Table 1 does not report standard deviation information. Can the authors provide some results that also quantify the variance of the proposed estimator compared to the others?\n- The authors claim that INFO-SEDD is lightweight (line 21), but I do not see any result reporting computational times in the main part of the paper. Can the authors provide a table with the computational time comparison of the different estimators? In Appendix C.1.3 the authors report the runtime for one experiment, but it would be meaningful to see these results in the main part of the paper and summarized over multiple scenarios. The authors report that INFO-SEDD uses less memory than SMILE and GAN-DIME, so is “lightweight” referred to this?\n- In Appendix C.1.4, Table 3 shows that MINDE overestimates MI for low values. Is this result generalizable for different scenarios? Could you claim that this happens to many generative MI estimators except INFO-SEDD? Or is it just a coincidence?\n- How do the other methods perform in the extreme dimensionality experiments in C.1.7?\n- In Table 5, increasing $|\\mathcal{X}|$ should lead to a better performance of the other estimators, right? Why does the trend appear to be opposite?\n- Why did the authors decide to not provide the code?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Good theoretical contributions (including Appendix E)\n- Promising experimental results\n- The proposed approach differs from many continuous existing MI estimators\n- The proposed estimator appear to be converging faster than other sota estimators", "weaknesses": "- The paper is not easy to read\n- In line 200 you write that expressing the MI as the KL divergence requires training two score models. However, many MI estimators target the estimation of the density ratio. So this sentence should be rewritten.\n- Line 1211 typo: $\\mathcal{X}|$\n- The code is not provided", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zTTGhonQKO", "reviewer": "Reviewer_gDGn", "summary": "The paper focuses on mutual information for high-dimensional discrete data. The score function of a diffusion model is used as a key component to estimate mutual information and entropy from a trained diffusion model. In comparison, prior work on diffusion models for mutual information estimation focuses on continuous random variables. The proposed method INFO-SEDD is implemented in two variants, one that approximates MI as the KL divergence between the product distribution and the marginals and one that compares the conditional $p_{Y|X}$ to $p_Y$. The approaches are benchmarked on synthetic discrete data pairs with high dimensions. Further, the authors consider case studies for text and DNA sequence data.", "questions": "- The training of the method is not clear. Do the authors use a pre-trained diffusion model? \n- The authors state that MI can be used for model selection. How does it compare to other metrics like the model perplexity?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The authors consider a rich set of benchmark tasks and potential real-world applications.\n- The performance of the proposed method compared to prior work on continuous data seems promising.\n- Connecting the continuous time Markov chains and diffusion models to compute MI for discrete data seems new to the reviewer.", "weaknesses": "- The paper is very dense and hard to follow since many parts are pushed to the appendix (see questions and comments below).\n- It is not clear what the exact differences to other diffusion-based MI estimators are, e.g., MINDE, and how the authors manage to model discrete distributions compared to prior work that did not cover this case.\n- Experiments:\n\t- It is not clear how the baselines are implemented for Sec. 4.1. Don’t they all focus on continuous data? As a simple baseline, using the plug-in estimator for discrete MI, and some more elaborate simple estimators (e.g., Goebel et al. (2005)) would improve the evaluation.\n\t- Real-world examples: The examples on text summarization and genomics are interesting, but they do not allow clear inferences about the performance of the MI estimator. The consistency test on the genomics dataset seems to not go beyond classification.\n\t- Typically, papers on discrete MI discuss the issue of bias in the plug-in estimator, which occurs under high dimensionality (in comparison to the sample size). It would be interesting to see if the proposed estimator can effectively mitigate such issues (see, e.g., Goebel et al. (2005)). The authors do evaluate susceptibility to dimensionality, but adopt very large sample sizes ($10^6$) for D=256 and 1024 and employ a model with 2M parameters. Here again, a comparison to the plug-in estimator and improvements of it to high-dimensions should be compared to.\n\nMinor points:\n- To which data is the GP fitted (line 356)?\n- What is meant by “simple and uninteresting scenarios” (line 124/125)?\n\nReference:\n- Goebel et al. (2005). An approximation to the distribution of finite sample size mutual information estimates.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)