MINDE: Mutual Information Neural Diffusion Estimation
Authors: Giulio Franzese
Mustapha BOUNOUA
Pietro Michiardi
Venue: ICLR 2024 poster (2024)
Keywords: mutual information
score matching
diffusion models
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Abstract: In this work we present a new method for the estimation of Mutual Information (MI) between random variables. Our approach is based on an original interpretation of the Girsanov theorem, which allows us to use score-based diffusion models to estimate the KL divergence between two densities as a difference between their score functions. As a by-product, our method also enables the estimation of the entropy of random variables. 
Armed with such building blocks, we present a general recipe to measure MI, which unfolds in two directions: one uses conditional diffusion process, whereas the other uses joint diffusion processes that allow  simultaneous modelling of two random variables. 
Our results, which derive from a thorough experimental protocol over all the variants of our approach, indicate that our method is more accurate than the main alternatives from the literature, especially for challenging distributions. Furthermore, our methods pass MI self-consistency tests, including data processing and additivity under independence, which instead are a pain-point of existing methods

Published as a conference paper at ICLR 2024
MINDE: MUTUAL INFORMATION NEURAL DIFFUSION ESTIMATION
Giulio Franzese Mustapha Bounoua',? Pietro Michiardi EEURECOM," Ampere Software Technology, "giulio. huuulloo.ranzeeeeeuueecooo fr
ABSTRACT
In this work we present new method for the estimation of Mutual Information (MI) between random variables. Our approach is based on an original interpretation of the Girsanov theorem, which allows to use score-based diffusion models to estimate the Kullback-Leiller (KL) divergence between two densities as a difference between their score functions. As by-product, our method also enables the estimation of the entropy of random variables. Armed with such building blocks, present general recipe to measure MI, which unfolds in two directions: one uses conditional diffusion process, whereas the other uses joint diffusion processes that allow simultaneous modelling of two random variables. Our results, which derive from a thorough experimental protocol over all the variants of our approach, indicate that our method more accurate than the main alternatives from the literature, especially for challenging distributions. Furthermore, our methods pass MI self-consistency tests, including data processing and additivity under independence, which instead are a pain-point of existing methods. Code available.
INTRODUCTION
Mutual Information (MI) is central measure to study the non-linear dependence between random variables [Shannon, 1948; MacKay, 2003], and has been extensively used in machine learning for representation learning [Bell & Sejnowski, 1995; Stratos, 2019; Belghazi et al., 2018; Oord et al., 2018; Hjelm et al., 2019], and for both training [Alemi al., 2016; Chen et al.. 2016; Zhao et al., 2018] and evaluating generative models [Alemi & Fischer, 2018; Huang et al., 2020].
For many problems of interest, precise computation of MI not an easy task [McAllester & Stratos, 2020; Paninski, 2003], and wide range of techniques for MI estimation have flourished. As the application of existing parametric and parametraee methods [Pizer et al., 1987; Moon et al., 1995; Kraskov et al., 2004; Gao et al., 2015] to realistic, high-dimensional data is extremely challenging, not unfeasible, recent research has focused on variational approaches [Barber & Agakov, 2004; Nguyen et al., 2007: Nowozin al., 2016; Poole et al., 2019; Wunder et al., 2021; Letizia et al., 2023; Federici et al., 2023] and neural estimators [Papamakarios et al., 2017: Belghazi et al., 2018; Oord et al., 2018 8; Song & Ermon, 2019; Rhodes et al., 2020; Letizia & Tonello, 2022; Brekelmans et al., 2022] for MI estimation. In particular, the work by Song & Ermon [2019] and Federici et al. [2023] classify recent MI estimation methods into discriminative and generative approaches. The first class directly learns to estimate the ratio between joint and marginal densities, whereas the second estimates and approximates them separately.
In this work, we explore the problem of estimating MI using generative approaches, but with an original twist. In $ we review diffusion processes [Song al., 2021] and we explain how, thanks to the Girsanov Theorem [Øksendal, 2003], we can leverage score functions to compute the KI. divergence between two distributions. This also enables the computation of the entropy of random variable. In Š 4 we present our general recipe for computing the MI between two arbitrary distributions, which we develop according to two modeling approaches, i.c., conditional and joint diffusion processes. The conditional approach is simple and capitalizes on standard diffusion models, but it is inherently more rigid, as it requires one distribution to be selected as the conditioning signal. Joint diffusion processes, on the other hand, are more flexible, but require extension of traditional diffusion models, which deal with dynamics that allow data distributions to evolve according to multiple arrww of time.

---

Published as a conference paper at ICLR 2024
Recent work by Czyż et al. [2023] argue that MI estimators mostly evaluated assuming simple, multivariate normal distributions for which MI is analytically tractable, and propose a novel bench- mark that introduces several challenges for estimators, such sparsity of interactions, long-tailed sail-taledd distributions, invariance, and high mutual information. Furthermore, Song & Ermon [2019] introduce measures of ceff-consistency (additivity under independence and the data processing inequality) for MI estimators, to discern the properties of various approaches. In $ 5 we evaluate several variants of our method, which we call Mutual Information Neural Diffusion Estimation (MINDE), according to such challenging benchmarks: our results show that MINDE outperforms the competitors on majority of tasks, especially those involving challenging data distributions. Moreover, MINDE passes all self-consistency tests, property that has remained elusive so far, for existing neural MI estimators.
2 DIFFUSION PROCESSES AND SCORE FUNCTIONS
We now revisit the theoretical background on diffusion processes, which is instrumental for the derivation of the methodologies proposed in this work, Consider the real space R& and its associated Borel - algebra, defining the measurable space (IR-V, B(R In this work, we consider Ito pro- cesses in IR with duration T < oc. Let Ω = D ([ x RM). be the space of all N dimmessiional continuous functions in the interval [o, T). and the filtration F induced by the canonical process (u) = W,w ft. As starting point, we consider an Ito process:
dx: fiX,dt + g,dW,,
(1)
(u x
with given continuous functions ft S 0. > 0 and an arbitrary (deterministic) initial condition x E RR.. Equivalently, we can say that initial conditions are drawn from the Dirac measure &g- This choice completely determines the path measure po of the corresponding probability space (2, F, (,))). Starting from Ipz, we construct a new path measure |p by considering the product between pod and measure in TR::
P pordu(z).
Conversely, the original measure Ip* can be recovered from 1 by conditioning the latter on the particular initial value x, i.e., the projection pod= = [P The new measure [PM can be represented by the following Stochastic Differential Equation (SDE):
dX:; = fxX + g,WW, ~O~
(3)
associated to the corresponding probability spaces F. F, PH). We define v/"" as the pushforward of the complete path measure onto time instant E [0, T]. where by definition v/ p..
It is instrumental for the scope of this work to study how the path measures and the SDES represen- tations change under time reversal. Let X, der wr-r be the time-reversed canonical process. the canonical process X, is represented as in Eq, (3) under the path measure PK, then the time reversed process X₂ has SDE representation [Anderson, 1982]:
dxx fr, o7 + gr- X, ~ U/
with corresponding path-reversed measure fr, on the probability spaces with time-reversed filtration.
Next, we define the score function of the densities associated to the forward processes, In particular,
8 (zr) V log (z((x)) where 7 (z) is the density associated to the measures 1/" (z)), computed with respect the Lebesgue measure,de (x) = F/" (z)d In general we cannot assume exact knowledge of such true score function. Then, in practice, instead of the score function s (I), we use parametric (8) approximations thereof, & (z). which we call the score network. Training the score network can be done by minimizing the following loss [Song et al., 2021; Huang et al., 2021; Kingma et al., 2021 ]]
T
(X,, - V Vlog (DēXO 6x0 (x.)
[() = Epp
di
(5)
2
2

---

Published as a conference paper at ICLR 2024
where V 6x stands for the measure of the processes time conditioned on some initial value Xo-
KL DIVERGENCE AS DIFFERENCE SCORE FUNCTIONS
The MI between two random variables can be computed according to several equivalent expressions, which rely on the KL divergence between measures and/or entropy of measures. We then proceed to describe i) how to derive KL divergence between measures as the expected difference of score functions, ii) how estimate such divergences given parametric approximation of the scores (and the corresponding estimation errors) and iii) how to cast the proposed methodology to the particular case of entropy estimation. In summary, this Section introduces the basic building blocks that we use in $ 4 to define our MI estimators.
We consider the KL divergence between two generic measures S and μ in R.W, i.e. KL [1r || 1= which is equal to fg du log du
A
if the Radon-Nikodym derivative da du
exists (absolute con- tinuity is satisfied), and +oo otherwise. Since our state space is RN,, the following disintegration properties are valid [Léonard, 2014]:
dPus (w) (PvR Hwo) (wo) du diuu (w) (pu""t #wr) #wwr)
d(pu" #wo) (w) (wo) du=(wn) (wo) diw
dpu
(w) duf (WT)
d (fu" #wr dul (WT)
(6)
where we implicitly introduced the product representation Par P2duu (I), similarly to Eq. (2).
RN
Thanks to such disintegration theorems, we can write the KL divergence between the overall path measures pud and pa of two diffusion processes associated to the measures μd' and n. as
dl dPn"
log de duB
KL
E pucs
log
Eppn
KL
|
|| 11
where the second equality holds because, as observed on the left of Eq. (6), when conditioned on the same initial value, the path measures of the two forward processes coincide.
Now, since the KL divergence between the path measures is invariant to time reversal, i.c.,
KL pM
|| pa
KL
pun 1pu using similar disintegration arguments, holds that:
d (fu*#to HWr Egaa
log duf" dull"
[auu*
KL
Egun
log d (fuB Hwr
The first term on the r.h.s of Eq. (8) can be computed using the Girsanov theorem [Øksendal, 2003] as
E
1
Dt
2 (X,)
1
st (x.))[
Byn
8t (X.) st (xx) (x.)](
2g
2
The second term on the r.h.s of Eq. (8), equals KL.
this is a vanishing term with T, i.e. = 0. To ground this claim, we borrow the results by Collet & Malrieu
limm KI. ut T
[2008], which hold for several forward diffusion SDES of interest, such as the Variance Preserving (VP), Variance Exploding (VE) SDES Song et al. [2021]. In summary, is necessary to adapt the classical Bakry-Emery condition of diffusion semigroup the non homogeneous case, and exploit the contraction properties diffusion on the KL divergences.
Combining the different results, we have that:
T
2
KI. [1ca) || 1114 = Epp
(X,)
di + KL. 14
(10)
2
3

---

Published as a conference paper at ICLR 2024
which constitutes the basic equality over which we construct estimators, described in 3.1.
We conclude by commenting on the possibility of computing divergences in latent space. Indeed, in lower dimensional manifold M R [Loaiza- Ganem et al., 2022]. Whenever we can find encoder and decoder functions v; o, respectively, such that (<b(2)) I, _ almost surely, and (()(2)) z; nr almost surely, the KL. divergence computed in the latent space obtained by the encoder tb. Considering the pushforward measure μ
many natural cases, the density ee' is supported
be
-i it indeed possible to show (proof in A) that KL [u* μ KL [ i μ& i - This property is particularly useful as it allows using score based models trained in latent space to compute the KL divergences of interest, as we do in 5.2.
3.1 KL ESTIMATORS AND THEORETICAL GUARANTEES
Given the parametric approximations of the score networks through minimization of Eq. (5), and the result in Eq. (10), we are ready discuss our proposed estimator of the KI. divergence. We focus on the first term on the r.h.s. of Eq. (10), which has unknown value, and define its approximated version
T
T
e(ar 11
E Exnu
gi
gi (Xt) - 8f (xx) di = 2 88
Bt (Xi) - 3t (x:))(] dl,
2
uf
(11)
where parametric scores, instead of true score functions, are used. By defining the score der "/ (x) st (x), is possible show (see $ A) that e(uA,pB
error as
c
(x)
T
St (x,) st " (x,) dt has expression
2
gi
el (Xt) (x.) + 22(06 (X.) (x (X.) oi (x.))
Epud
2
(12)
As for the second term the r.h.s. of Eq. (10), KL vanishes with large T. Consequently, given a sufficiently large diffusion time T the function e serves accurate estimator of the true KI:::
14"" we recall that it is quantity that
e(μ² P KL [M ||
+ d KL
VT
~ KL [ec 11
(13)
An important property of our estimator is that is neither an upper nor a lower bound of the true KL divergence: indeed the d term of Eq. (13) can be either positive or negative. This property, frees our estimation guarantees from the pessimistic results of McAllester & Stratos [2020]. Note also that,
counter-intuitively, larger errors norms
|et (x)) not necessarily imply larger estimation error of the KL. divergence. Indeed, common mode errors (reminiscent of paired statistical tests) cancel out. In the special case where ei" (α) = et (x)) the estimation error due to the approximate nature of the score functions is indeed zero.
Accurate quantification of the estimation error is, in general, a challenging task. Indeed, techniques akin the works [De Bortoli, 2022; Lee et al., 2022; Chen et al., 2022], where guarantees provided w.r.t. to the distance between the real backward dynamics and the measures induced by the simulated backward dynamics, KL [MC ]= are not readily available in our context. Qualitatively, we observe that our estimator is affected by two sources of error: score networks that only approximate the true score function and finiteness of T. The d term in Eq. (13), which is related to the score discrepancy, suggests selection of a small time 7' (indeed we can expect such mismatch to behave as quantity that increases with T' [Franzese al., 2023]). is important however to adopt a sufficiently
large diffuusio time T' such that KL 1 || 14 is negligible. Typical diffusion schedules satisfy these requirements. Note that, the KL term is known (or approximately known), it can be included in the definition of the estimator function, reducing the estimation error (see also discussion in 3.2).
4

---

Published as a conference paper at ICLR 2024
Montecarlo Integration The analytical computation of Eq. (11) is, in general, out of reach. However, Montecarlo integration is possible, by recognizing that samples from vt can be obtained through the sampling scheme Xo μA, Xx ~ V/2 6xo The outer integration w.r.t. to the time instant is similarly possible by sampling & ~ (0, and multiplying the result of the estimation by T (since Io' (-)dt (0), [-))). Alternatively, is possible to implement importance sampling schemes to reduce the variance, along the lines of what described by Huang et al. [2021], by sampling the time instant non-uniformly and modifying accordingly the time-varying constants Eq. (11). In both cases, the Montecarlo estimation error can be reduced to arbitrary small values by collecting enough samples, with guarantees described in [Rainforth et al., 2018].
3.2 ENTROPY ESTIMATION
We now describe how to compute the entropy associated to given density AA H(rA) del I dd (x) log n (x). Using the ideas for estimating the KL divergence, we notice that we can compute KL. [ || Ta]. where To (zr) stands for the standard Gaussian distribution with mean 0 and covariance 1²1. Then, we can relate the entropy to such divergence through the following equality:
H(p²) + KL [M' ll [o] =- /dx dz)(())
ff
N log 5o (z) log pog(TTo)) + E, [XA]
(14)
20
simple manipulation of Eq. (14), using the results from 3.1, implies that the estimation of the entropy A( involves three unknown terms: ; KI. w || 177 and B_
Now, the score function associated the forward process starting from To analytically known and has value (x) Xt where Xx (Kjo² ki fO "g 1, with k, exp( (Jo fads) }. More-
20
over, whenever T is large enough 2 Y1, independently on the chosen value of o. Consequently KL.
1 KI. (m [ which is analytically available as 8/2 (log (xT) + 1/xr). Quantification of such approximation is possible following the same lines defined by Collet & Malrieu [2008]. In summary, consider the following estimator for the entropy:
N H(p4:o) log (2TT²)
Ena
[X3] 20
N
(Io (XT) 1+
1
e(px, Ta)
(15)
Xr
For completeness, we note that related estimator has recently appeared in the literature [Kong et al., 2022], although the technical derivation and objectives are different than ours.
4 COMPUTATION OF MUTUAL INFORMATION
In this work, we are interested in estimating the MI between two random variables A, B. Consequently, we need to define the joint, conditional and marginal measures. We consider the first random variable A in N to have marginal measure μ Similarly, we indicate the marginal measure of the second random variable B with い. The joint measure of the two random variables C del [A, B). which is defined in R2N, is indicated with jºC What remains to be specified are the conditional measures of the first variable given particular value of the second A B y, shortened with Ay, that indicate with the measure yu and the conditional measure of the second given particular value of the first, B == I. shortened with B., and indicated with s. This choice of notation, along with Bayes theorem, implies the following set of equivalences: dy (zz y) Au (x))dp " (v) dy
and A [ p uy (v), p f BB
The marginal measures nA,, ,1. are associated to diffusion of the form of Eq. (3). Similarly, the joint _ and conditional pmA measures we introduced, are associated to forward diffusion processes:
[[,,, Yi] = fe[X,, TT]Tdd dt + ge [dW,, dW?] [Xo. Yol ~ ct
axx, Xtdt + g,dW, Xo
(16)
A
respectively, where the SDB on the l.b.s. is valid for the real space 2R defined in $
5

---

Published as a conference paper at ICLR 2024
In this work, we consider two classes of diffusion processes. In the first case, the diffusion model is asymmetric, and the random variable B is only considered as a conditioning signal. As such, we learn the score associated to the random variable A, with conditioning signal B, which is set to some predefined null value when considering the marginal measure. This well-known approach [Ho & Salimans, 2021] effectively models the marginal and conditional scores associated to fı and yA with unique score network.
Next, we define a new kind of diffusion model for the joint random variable C, which allows modelling the joint and the conditional measures. Inspired by recent trends in multi-modal generative modeling [Bao et al., 2023; Bounoua al., 2023], we define a joint diffusion process that allows amortized training of a single score network, instead of considering separate diffusion processes and their respective score networks, for each random variable. To so, we define the following SDE:
[ = ftlax,, [Y]]Tdl dt + Ii [ddW,3dW[] [Xo. Yol ~ ut
(17)
with extra parameters or, (0, 13. This SDE extends the l.h.s. of Eq. (16), and describes the joint evolution the variables Xx, Yr, starting from the joint measure S with overall path measure Ipa.. The two extra coefficients a, 33 are used to modulate the speed at which the two portions X Y, of the process diffuse towards their steady state. More precisely, a = 3 = 1 corresponds to a classical simultaneous diffusion (l.h.s. of Eq. (16)). On the other hand, the configuration = 1, 0 corresponds to the case in which the variable Y, remains constant throughout all the diffusion (which
used for conditional measures, r.h.s. of Eq. (16)). The specular case, = 0, 1. similarly allows to study the evolution of Y, conditioned on a constant value of Xo. Then, instead of learning three separate score networks (for μ ue and ). associated to standard diffusion processes, the key idea is to consider unique parametric score, leveraging the unified formulation Eq. (17), which accepts as inputs two vectors in R N,,, the diffusion time t, and the two coefficients a, B. This allows
to conflate in single architecture: i) the score st (Ir, 3) associated to the joint diffusion of the variables A, B (corresponding to a = 3 1) and ii) the conditional score sf (x) (corresponding to a = 1, 3 0). Additional details are presented in $ C.
4.1 MINDE: FAMILY OF MI ESTIMATORS
We now ready to describe our new MI estimator, which we call MINDE. As starting point, we recognize that the MI between two random variables A, B has several equivalent expressions, among which Eqs. (18) (20). On the left hand side of these expressions formulations for the MI, ((A, B), while on the right hand side we express them using the estimators we introduce in this work, where equality is assumed to be valid up to the errors described in 3.
report well-known
H(A) - H(AIB) = -e(u",n0 I
(w).
(18)
/ KL (1co (^) de (g) ~ c(H"",, May" (w),
(19)
H(C)- H(A B) H(B A) ~ celx (70) + f cou (au",,70da do) cla"t To (x). (20)
Note that it is possible to derive (details in $ B) another equality for the MI:
gz ((x,, Vi) - [8%
Avo (Xt), 84 No (va)] dt
I(A, B) ~ Eppc
(21)
2
Next, we describe how the conditional and joint modeling approaches can be leveraged to compute a family of techniques to estimate MI. We evaluate all the variants in $ 5,
Conditional Diffusson Models. We start by considering conditional models. A simple MI estimator can be obtained considering Fq. (18). The entropy of A can be estimated using Eq. (15). Similarly,

---

Published as a conference paper at ICLR 2024
can estimate the conditional entropy H(A) B) using the equality H(A BB = (H(AA)dd (y))
where the argument of the integral, H( can be again obtained using Eq. (15). Notice, that since
E (n) [X
= 1 [X]] when substracting the estimators of (() and H(A) B), all the
terms but the estimator functions e(-) cancels out, leading to the equality in Eq. (18). A second option
is to simply use Eq. (19) and leverage Eq. (11).
Joint diffusion models. Armed with the definition of a joint diffusion processes, and the corre- sponding score function, we now describe the basic ingredients that allow estimation of the MI, according to various formulations. Using the joint score function st ([xr, v)), the estimation of the joint entropy H(A, B) can be obtained with a straightforward application of Eq. (15). Similarly, the conditional entropy H(A) B) (H(dy)d can be computed using " (ar) obtain the conditional score. Notice that H A) is similarly obtained. Given the above formulations of the joint and conditional entropy, it now easy to compute the MI according to Eq. (20), where we notice that, similarly to what discussed for conditional models, many of the terms in the different entropy estimations cancel out. Finally, is possible to compute the MI according to Eq. (21). Interestingly. this formulation allows to eliminate the need for the parameter a of the entropy estimators, similarly to the MINDE conditional variant, which shares this property as well (Eq. (18)).
EXPERIMENTAL VALIDATION
We now evaluate the different estimators proposed in $ 4. In particular, we study conditional and joint models (MINDE-C and MINDE-J respectively), and variants that exploit the difference between the parametric scores inside the same norm Eqs. (19) and (21)) or outside it, adopting the difference of entropies representation along with Gaussian reference scores Te (Eqs. (18) and (20)). Summarizing, we refer to the different variants as MINDE-C(o), MINDE-C, and MINDE-J(O).
MINDE-J, for Eqs. (18) to (21) respectively. Our empirical validation involves a large range of synthetic distributions, which we present in $ 5.1. We also analyze the behavior of all MINDE variants according to self- consistency tests, as discussed in $ 5.2.
For all the settings, we use a simple, stacked multi-layer perception (MLP) with skip connections adapted to the input dimensions, and adopt VP-SDE diffusion Song et al. [2021]. We apply importance sampling [Huang et al., 2021; Song et al., 2021] at both training and inference time. More details about the implementation are included in $ C.
5.1 MI ESTIMATION BENCHMARK
We use the evaluation strategy proposed by Czyž et al. [2023], which covers range of distributions going beyond what is typically used to benchmark MI estimators, e.g., multivariate normal distri- butions. In summary, we consider high-dimensional cases with (possibly) long-tailed distributions and/or sparse interactions, in the presence of several non trivial non-linear transformation. Bench- marks are constructed using samples from several base distributions, including Uniform, Normal with either dense or sparse correlation structure, and long-tailed Student distributions. Such samples are further modified by deterministic transformations, including the Half-Cube homeomorphism, which extends the distribution tails, and the Asinh Mapping, which instead shortens them, the Swiss Roll Embedding and Spiral diffeomorphis, which alter the simple linear structure of the base distributions.
We compare MINDE against neural estimators, such as MINE [Belghazi et al., 2018]. INFONCE [Oord et al., 2018J.NW [Nguyen al., 2007] and DOE [McAllester & Stratos, 2020]. To ensure fair comparison between MINDE and other neural competitors, we consider architectures with a comparable number of parameters. Note that the original benchmark in [Czyż et al., 2023] uses 10k training samples, which are in many cases not sufficient to obtain stable estimates of the MI for our competitors. Here, we use a larger training size ( (100k samples) to avoid confounding factors in our analysis. In all our experiments, we fix 1.0 for the MINDF-C((r), MINDE-J(O) variants, which results in the best performance (an ablation study is included in D).
Results:
The general benchmark consists of 40 tasks (10 unique tasksx 4 parametrizations) de- signed by combining distributions and MI-invariant transformations discussed earlier. We average results over 10 seeds for MINDE variants and competitors, following the same protocol as in Czyz
7

---

Published as a conference paper at ICLR 2024
et al. [2023]. We present the full set of MI estimation tasks in Table 1. As in the original Czyž et al. [2023], estimates for the different methods are presented with precision of 0.1 nats, to improve visu- alization. For low-dimensional distributions, benchmark results show that all methods accurate MI estimation. Differences emerge for more challenging scenarios. Overall, all our MINDE variants perform well. MINDE-C stands out the best estimator with 35/40 estimated tasks with an error within the 0.1 nats quantization range. Moreover, MINDE can accurately estimate the MI for long tailed distributions (Student) and highly transformed distributions (Spiral, Normal CDF), which are instead problematic for most of the other methods. The MINE estimator achieves the second best performance, with an estimation within 0.1 nats from ground truth for 24/40 tasks. Similarly to the other neural estimator baselines, MINE is limited when dealing with long tail distributions (Student), and significantly transformed distributions (Spiral).
effective in
High MI benchmark: Through this second benchmark, we target high MI distributions. We consider 3 multivariate normal distribution with sparse interactions as done Czyz et al. [2023]. We vary the correlation parameter to obtain the desired MI, and test the estimators when applying Half-cube or Spiral transformations. Results Figure 1 show that while on the non transformed distribution (column (a)) all neural estimators nicely follow the ground truth, on the transformed versions (columns (b) and (c)), MINDE outperforms competitors.
estimate
Mi esttmate
Ml essimatee
GT
Mince-j
Mince
InfoNCE
D-V
NV]
Truc
KSG
Sparse Multinormal
(b) Half-cube
(c)
Spical
CCA
Figure 1: High MI benchmark: original (column (a)) and transformed variants (columns (b) and (c)).
5.2 CONSISTENCY TESTS
The second set of tests perform are the self-consistency ones proposed in Song & Ermon [2019], which aim at investigating properties of MI estimators on real data. Considering as random variable A sample from the MNIST (resolution 28 28) dataset, the first set of measurements performed is the estimation of I(A, Br), where B, is equal to A for the first r rows, and set to afterwards. It is evident that I(A, B,) quantity that increases with r. where in particular I(,, Bb) = O. Testing whether this holds also for the estimated MI is referred to as independency test. The second
proposed in Song & Ermon [2019] is the data-processing test, where given that I(A; [B++k, B, I(A; Br+k), k >> O, the task is to verify it through estimators for different values of k. Finally, the additivity tests aim assessing whether for two independent images A' Aa extracted from the dataset, the property I([41, A [B, BA) (1;; B,)) is satisfied also by the numerical estimations.
For these tests, we consider diffusion models in latent space, exploiting the invariance of KL divergences to perfect auto-encoding (see $ 3). First, we train for all tests deterministic auto-encoders for the considered images. Then, through concatenation of the latent variables, done in [Bao et al., 2023; Bounoua et al., 2023], we compute the MI with the different schemes proposed in $ 4. Results of the three tests (averaged over 5 seeds) are reported in Figure 2. In general, all MINDE variants show excellent performance, whereas none of the other neural MI estimators succeed at passing simultaneously all tests, can be observed from Figures 4,5,6 in the original Song & Ermon [2019]).
8

---

Published as a conference paper at ICLR 2024
MINDEJ
MINDE:J
MINDE-C (0-1)
Plaee
Rews
MINDE-C
(a) Baseline test
(b) Data procesinn test
(c) Additivity test
Figure 2: Consistency tests results on the MNIST dataset. Baseline test Figure 2a: Evaluation of [AA.B-) R-]
[.) A is
an image and B, an image containing the top rows of A. Data processing test Figure 2b: Evaluation of
[(A,IB.4t.BB..) (ideal value is 1). Additivity test Figure 2c: Evaluation of IQA 112 (ideal value is 2). 8,4,8)4))
[(A
020403
0302040330
MINDE-J (o
0,2040300000 04
100 3 0.04 0 17 0 L 69 1009 1002 04
03003.
01220500,300 0.30 0
MINDE-J
0.204
0.0.
1)0.20
MINDE-C
0.2
0 0.4 0 LO LO 1003 1.0
0.0.4
0.
04 O LO
000000000000000 10 10 1
0.2
03
0.30.4 L.7.300
MINT
0.204
LO
0.04
0
0.0.00 1,73.04
0.4
LO
10
0.
LO
10
D-V
0.2 0.3 A 0
0
LO TO 10 LO
L.04
0.
166 0 0.9 LO 8
1000000000000 0.1
02002 0.2 0.
NWI
0.204. 0.3 0 A 0
4
LO LO LO ID 1.
1.004
1.066
16 0.9 1.0 1.00..0..00 08 09 1. 100000000000
0.
0.2
1.70.3
0.200. 0.3 ILS 0.4 0.4
O
LO LO 1.
1.00.6
0.4 L. 1.0055 0.6 0.6 066 0.2 0.3
79
19
2.5
86
L.50.
0.4
DoClLogtstic
0.1 0.0.022 0.4
0.9.9 1.00330
1,0.6
0.4.3
1.00.5 0.6 0.6
0.3 0.8
0.0 1.50.
25 (2pair))
CDF 3 (2guir)
Sp a Sio CDF e Mr x (2guir)
Cairom xx1 tabitivee soisew0
Uaifinn xx jubiiics miisem077
Booarate New Lxx
Nee CDF e Mis 25 25 Copair)
No CDP e Mr 3 3 (2ppir)) Ain CDT a Mr
(2 pair)
Avrt e SV 2 (k66s)
Asiih e SV3 x (i00-2 Asin is St5 5 (d06-2)
1x
He @ Mr 25 25 (2 (2
Me x (2.-pir) (2.pir))
Sp An 3xx (2 palr
Sp Mn 5 (2-paio
Wiggy Boariun Ne
Blonddal x
Bivaitte fie xx1 a Revurate
H: Mr (2-guin)
Mnn 2x 2 (dmme))
Mr 25 25 (Ppair))
25 (demae)
Mn x 122 puir
Mr x 3 (dasse)
Mr 5 (2.pai)
Me x 5 (danss)
50x (emme)
Sp Mi 25
Sp CDF
St x (dd=11)
2 (da--1)
2
3 3x
3e dodf-)
( (dot-2)
515 5 (60-33)
Swis mll 3x1
Aeth
5
Table 1: Mean MI estimates over 10 seeds using N = 10k test samples against ground truth (GT). Color indicates
relative negative (red) and positive bias (blue). All methods were trained with 100k samples. List of abbreviations
( Mr: Multinormal, St: Student-t, Nm: Normal, He: Half-cube, Sp: Spiral)
6 CONCLUSION
The estimation of MI stands as fundamental goal in many areas of machine learning, as it enables understanding the relationships within data, driving representation learning, and evaluating generative models. Over the years, various methodologies have emerged to tackle the difficult task of MI estima- tion, addressing challenges posed by digensiiensinnll, real-wor world data. Our work introduced a novel method, MINDE, which provides a unique perspective on MI estimation by leveraging the theory of diffusion-based generative models. We expanded the classical toolkit for ieformation-thhorree analysis, and showed how to compute the KI. divergence and entropy of random variables using the score of data distributions. We defined several variants of MINDE, which we have extensively tested according to recent, comprehensive benchmark that simulates real-world challenges, including sparsity, long-tailed distributions, invariance transformations. Our results indicated that our meth- ods outperform state-of-the oftte- art alternatives, especially on the most challenging tests. Additionally, MINDE variants successfully passed self-consistency tests, validating the robustness and reliability of our proposed methodology.
Our research opens up exciting avenues for future exploration. One compelling direction is the application of MINDE to large-scale multi-modal datasets. The conditional version of our approach enables harnessing the extensive repository of existing pre-trained diffusion models. For instance, it could find valuable application in the estimation of MI for text-conditional image generation. Conversely, our joint modeling approach offers a straightforward path to scaling MI estimation to more than two variables. A scalable approach to MI estimation is particularly valuable when dealing with complex systems involving multiple interacting variables, eliminating the need to specify a hierarchy among them.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
GF and PM gratefully acknowledges support from Huawei Paris and the European Commission (ADROIT6G Grant agreement ID: 101095363).
REFERENCES
Alexander A Alemi and lan Fischer. Gilbo: One metric to measure them all. Advances in Neural Information Processing Systems, 31, 2018.
Alexander A Alemi, lan Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. International Conference on Learning Representations, 2016.
Brian D. O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313-326, 1982.
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.0132. 2022.
Fan Bao, Shen Nic, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang. Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffusion scale. arXiv preprint 2rXivv2303.06555 2023.
David Barber and Felix Agakov. The im algorithm: variational approach information maximiza- tion. Advances in neural information processing systems, 16(320):201, 2004.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In Proceedings of the 35th International Conference on Machine Learning. 2018.
Anthony J Bell and Terrence J Sejnowski. An petprmmmttionmmaiiiiiinia approach to blind separation and blind deconvolution. Neural computation, 7(6): 1159, 1995.
Mustapha Bounoua, Giulio Franzese, and Pictro Michiardi. Multi-modal latent diffusion. arXiv preprint arXiv:2306. 04445, 2023.
Rob Brekelmans, Sicong Huang, Marzych Ghassemi, Greg Ver Steeg, Roger Baker Grosse, and Alireza Makhzani. Improving mutual information estimation with annealed and anergy-based bounds. In International Conference Learning Representations, 2022.
Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru Zhang. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. In International Conference on Learning Representations, 2022.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Info- gan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016.
Jean-Françord Collet and Florent Malrieu. Logarithmic sobolev inequalities for inhomogeneous markov semigroups. ESAIM: Probability and Statistics, 12:492-504, 2008
Pawel Czyž, Frederic Grabowski, Julia E Vogt. Niko Beerenwinkel, and Alexander Marx. Beyond normal: On the evaluation of mutual information estimators. Advances Neural Information Processing Systems, 2023.
Valentin De Bortoli. Convergence denoising diffusion models under the manifold hypothesis. Transactions Machine Learning Research, 2022.
Marco Federici, David Ruhe, and Patrick Forré. On the effectiveness of hybrid mutual information estimation. arXiv preprint arXiv:2306.00608 2023.
10

---

Published as a conference paper at ICLR 2024
Giulio Franzese. Simone Rossi, Lixuan Yang, Alessandro Finamore, Dario Rossi, Maurizio Filippone, and Pietro Michiardi. How much is enough? a study on diffusion times in score-base generative models. Entropy, 2023.
Shuyang Gao, Greg Ver Steeg, and Aram Galstyan. Efficient estimation of mutual information for strongly dependent variables. In Artificial intelligence and statistics, pp. 277-286. PMLR, 2015.
Devon Hjelm, Alex Fedorov, Samuel Marehi-nrr,,,,,n,, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2019.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances Neural Information Processing Systems, volume 33, pp. 6840-6851. Curran Associates, Inc., 2020.
Chin-Was Huang, Jae Hyun Lim, and Aaron C Courville. A variational perspective diffusion-based generative models and score matching. Advances in Neural Information Processing Systems, 34: 12863-22876) 2021.
Sicong Huang, Alireza Makhzani, Yanshuai Cao, and Roger Grosse. Evaluating lossy compression rates of deep generative models. In International Conference on Machine Learning. PMLR, 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021.
Xianghao Kong, Rob Brekelmans, and Greg Ver Steeg. Information-theoree diffusion. In Interna- tional Conference on Learning Representations, 2022.
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information.
Physical review E, 6966)8066138, 2004.
Holden Lee, Jianfeng Lu, and Yixin Tan, Convergence for score-based generative modeling with polynomial complexity. Advances in Neural Information Processing Systems, 35:22870-22882. 2022.
Christian Léonard. Some properties of path measures. Séminaire de Probabilités XLVI, pp. 207-230, 2014.
Nunzio A Letizia and Andrea M Tonello. Copula density neural estimation. arXiv:2211.15353. 2022.
arXiv preprint
Nunzio Letizia, Nicola Novello, and Andrea M Tonello. Variational ff f-divergence and derangements for discriminative mutual information estimation. arXiv preprint arXiv:2335..20025, 2023.
Gabriel Doaiza-Ganem. Brendan Leigh Ross, Jesse Cresswell, and Anthony I. Caterini. Diagnosing and fixing manifold overfitting in deep generative models. Transactions Machine Learning Research, 2022.
David JC MacKay. Information theory, inference and learning algorithms. Cambridge university press, 2003.
David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information. In International Conference on Artificial Intelligence and Statistics, 2020.
Young-Il II Moon, Balaji Rajagopalan, and Upmanu Lall. Estimation of mutual information using kernel density estimators. Physical Review E, 52())23318 1995.

---

Published as a conference paper at ICLR 2024
XuanLong Nguyen, Martin Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by penalized convex risk minimization. In Advances in Neural Information Processing Systems, 2007.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. F-gan: Training generative neural samplers using variational divergence minimization. In Proceedings of the 30th International Conference on Neural Information Processing Systems. NIPS' 16, pp. 271-279, Red Hook, NY, USA, 2016.
Curran Associates Inc. ISBN 9781510838819.
Bernt Øksendal. Stochastic differential equations. Springer, 2003.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. Advances in neural information processing systems, 2018.
Liam Paninski. Estimation of entropy and mutual information. Neural computation, 15(6): 199-1253, 2003.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. Advances in neural information processing systems, 30, 2017.
Stephen M Pizer, E Philip Amburn, John D Austin, Robert Cromartie, Ari Geselowitz, Trey Greer, Bart ter Haar Romeny, John B Zimmerman, and Karel Zuiderveld. Adaptive histogram equalization and its variations. Computer vision, graphics, and image processing, 39(3):355-366, 1987.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. International Conference on Machine Learning. 2019.
Tom Rainforth, Rob Cornish, Hongseok Yang, Andrew Warrington, and Frank Wood. On nesting monte carlo estimators. International Conference on Machine Learning, pp. 4267-4276. PMLR. 2018.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents, 2022. URL /tt.t:::///axxx..o////// 22204 06125.
Benjamin Rhodes, Kai Xu, and Michael U Gutmann. Telescoping density-ratio estimation. Advances in neural information processing systems, 2020.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVE Confer- ence on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, June 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Raphael Gontijo-Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic sot-to--mage diffusion models with deep language understanding. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski. Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models, 2022.
Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379-42) 1948.
Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information
estimators. In International Conference on Learning Representations, 2019.
Yang Song, Jascha Sohl-Dicksee Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference Learning Representations, 2021.
12

---

Published as a conference paper at ICLR 2024
Karl Stratos. Mutual information maximization for simple and accurate pfrt-of-speech induction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), 2019.
Gerhard Wunder, Benedikt GroB, Rick Fritschek, and Rafael F Schaefer. A reverse jensen inequality result with application to mutual information estimation. In 2021 IEEE Information Theory Workshop (ITW), 2021.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. A lagrangian perspective on latent variable generative models. In Pruc, 34th Conference on Uncertainty Artificial Intelligence, 2018
13

---

Published as a conference paper at ICLR 2024
MINDE: MUTUAL INFORMATION NEURAL DIFFUSION ESTIMATION SUPPLEMENTARY MATERIAL
A PROOFS OF $ 3
Proof of Auto-encoder invariance of KL.. Whenever we can find encoder and decoder functions Φ x/ respectively such that d(v(xz)) x, μ almost surely and ((v(x))) x; ,u almost surely, the Kullback-Lede Leibler divergence can be computed in the latent space obtained by the encoder th:
Kl. J 11 [1 = -/o log du
du dur duodu==
fN log ( ddw
duB $ vb dud =I Secms log
(du
du oØ) d(wr o x+-
Juan, log
du duB w(-1) d(ur - =KL KL Lit aT
(22)
Proof of Eg. (12). To prove such claim, it is sufficient to start from the r.h.s. of Eq. (11), substitute the parametric scores their definition with the errors of (x) $ (x) - ss (x)), and expand the square:
T
9
31 (X,) - 3 (xa)leaa dt=
T
||st" (X,) (X,) + ct¹ (z) st (xx) oi ((0) (w)]
E
91
|| 11t (X,) - se (xx))1 dt+
E
T
g
||
(X,) (x,)
"x"((x,)) (xx)) + 2( (X,) "f" (X.)+ (xx)
(X,) 40 (xx)) de,
from which the definition of holds.
B PROOF OF EQ. (21)
We start with the approximation of Eq. (20):
I(A, B) ~ - .7) + "(u""y, ro),, + (o)4
(23)
Since the approximation is valid for any 5, we select the limit of a + oc, where the reference score x, - converges to zero, and can thus be neglected from the estimators integral (for example,
14

---

Published as a conference paper at ICLR 2024
T
||
(x:)l²] d(). This allows to obtain:
e(u
7x0)
%6
[ gh / def (1z, (11l (1x, wD)| det
IA, B) ~
gi / dut (z) (w)| 3/ (z) ||| dt du (z)+
(x)
2
| / 2 del dui (w) (w) (v) (w) dt (z)
gi
As a further step in the derivation of our approximation, we consider the estimated scores to be sufficiently good, such that we substitute the parametric with the true scores. In such case, the following holds:
((AB))
gi
/ du ((=o ([zo,)ol)dd, too (1x, vl) (-1|0 ([x,, w/llf + 11stt *1 (10) + 1/ ex (v) ()))²)
dt
9
Foo.0o / d
A1oo) ([x, v/) (-1 stt (La,, w//| + -||lst (xx), stt 9xo (w)l
T
g2
/ du d too.ooi
sooroo ([x, vl)
(-2| s4 ((x; U1) w1)| + s/ (1x, - [s (x). s/ Pzo
[w]l
2
2 (o "0 (x), st
Rro (w))) dl
Recognizing that the term st ([x, y)) - [st (x), si
Puo (v) averaged over the measures, is
just Eq. (21) in disguise, what remain to be assessed the following:
[ o / du ((xu, vul) Sino vol ([2,, 8l)
2
(-2) of (Ir, wl]l +2 2 (of ((x,0)), 10 15
(x), B5o (w))) dt 0.
(24)
In particular, focus the term:
T g
/ du ([xo woldu, Sooo ([x, xl) (of (1x, v)), 106 (0 (x),, "ro de
2
2 8/ (1r,
J.
V
d vol)du, Spowoll ([x, st A00 (x)) /
(1zo wo! " 9000 ([x, 2l)stt Brg
(2)
dt.
(25)
15

---

Published as a conference paper at ICLR 2024
Since de toooo (1x, y/) = du, 60 (ar (ar)de, 8oo (v) and du (1ro: mol) Au (zo)d (vo), then
Smo du ((Fo, vol)de, iool ([r, y|) det vo (x)det 6vo (v)du (yo). Consequently:
J (o2o:2o0)d 200.00
dol no vol ([x, 81)s vo (x) = [f dul (x)dvz 6N0 (y)dμ 1 sy
(x)
n0 S. du (r)du 30 (y)) (8o)Vlog (Dtitio (x))
drtt (z)dv) Avo (y)du Ves (2)
D/ (x)
dr I der (v)du (wo) (yo) Voh (x) dxV (f de,
d (y)d (20) (a)
y0
dxdey ( (1 dn tvo (w)) = dxdey (v)V (D (z))
where in the last line we introduced:: f (yo), the measure of the random variable B condition- ally the fact that the diffused variable B after time is equal and o A ttee conditional measure of the diffused variable A at time i, conditionally on the diffused variable B after a time I equal y. Finally
(ou SuAADee (x)
drdu," (n)V 1 (x)) = D/ 1 (r)dzdec (v)
def (1x, vl)st 1 (x).
(x)
Along the same lines, we can prove the equality S2o.90 du d wol) i0o tol vol ([x, w))s aro (2)
dvft (|x, (f) Blx xi=x (v). Then, restarting from Eq. (25) we have:
2 (st" (1x,
m.
V
(1zo, vol)dr, Poo vol ([x, 2/ (x)), f 20:00
dpx yul yul) 0000 ([x, vl) "xo (w)) dt
2o Mo
T
(sf (Ix, Ul), fauff (1x, v)) Ni-u (x), dut ((x, vl) e1x ) dt=
2 J.
T
T ot
dvf (Ix,v)) (sf (Ix, lst Ai (I), st 01x1 (w)l dt =
/ dut (Ix, (1e.v10|lff (Ix. vDD[(at
2
J.y
2
0
which finally allows to prove Eq. (24) and claim validity of Eq. (21).
C
IMPLEMENTATION DETAILS
In this Section, provide additional technical details of MINDE. We discuss the different variants of our method their implementation details, including detailed information about the MI estimators alternatives considered in 5.
C. MINDE-C
In all experiments, consider the first variable the main variable and the second variable the conditioning signal. single neural network is used to model the conditional and unconditional score. It accepts as inputs the two variables, the diffusion time f, and an additionally binary input e which enable the conditional mode. To enable the conditional mode, we set c = 1 and feed the network
with both the main variable and the conditioning signal, obtaining st Yo To obtain the marginal score s6 set 0 and the conditioning signal is set to zero value.
16

---

Published as a conference paper at ICLR 2024
Algorithm 1: MINDE-C (Single Training Step)
Data: [Xo, Yo] ~ c.C
parameter: : :eto (), with A current parameters
t ~~ [
// Importance sampling can be used to reduce variance 1
X2 c K2 + k2 [o k, kg with ~ // r.h.s. Eq. (16), diffuse
the variable X to timestep // Sample binary variable e with probabi lity d
Bernoulli())
if c = 0 then
(K3 1o Aogds)3 < (,
// Estimated unconditional score
else
(N2 So k. 92d))} t netol[Xx, t, 1))
// Estimated conditional score
(1 k. 9gds) lle ēll
11 Compute Montecario sample associated to
kq. (5)
return Update I according to gradient of L
Algorithm 2: MINDE
Data: [oo, Yo] ~ dd parameter :0, option t t~ 2[
// Importance sampling can be used to reduce variance
X₁ t No + (R2 k g) with c ~m // r.h.s. of Eq. (16), di diffuse
the variable X to timestep
3t + neto( (x 0),,cc=0)
// Use the unique score network to compute // marginal and conditional scores
s
vo_ + neta([X,, tol,t,c=1))
optioon then
c i T
z/ b6
Avo 2
2
else
+ + So k ds
X1 -
- T oi
2
34
+
Xx
- x
Avo
X1
27
return/ î
A randomized procedure is used for training. For each training step, with probability d, the main variable is diffused and the score network is fed with the diffused variable, the conditioning variable, the diffusion time signal and the conditioning signal is set to 1. On the contrary, with probability
1 - d, to enable the network to learn the unconditional score, the network is fed only with the diffused modality, the diffusion time and = 0. In contrast the first case, the conditioning is not provided to the score network and replaced with a zero value vector. Pseudocode is presented in Algorithm 1.
Actual estimation of the MI is then possible either by leveraging Eq. (18) or Eq. (19), referred to in the main text as difference outside or inside the score respectively (MINDE-C(r), MINDE-C). A pseudo-code description is provided in Algorithm 2.
C.2 MINDE-J
The joint variant of our method, MINDE-J based on the parametrized joint processes Eq. (17). Also in this case, instead of training separate score network for each possible combination of conditional modalities, we use a single architecture that accepts both variables, the diffusion time
I and the coefficients a, 3. This approach allows modelling the joint score network 8y by setting
17

---

Published as a conference paper at ICLR 2024
B 1. Similarly, to obtain the conditional scores it is sufficient to set 1.3 0 or # 0. 1, corresponding to s and s respectively.
Training is carried out again through randomized procedure. At each training step. with probability both variables are diffused. In this case, the score network is fed with diffusion time t, along with X₁, Yt and the two parameters a 3 1. With probability 1 - d, instead, we randomly select one variable to be diffused, while we keeping constant the other. For instance, if A is the one which is and 3 = 0. Further details are presented in Algorithm 3.
diffused, we set
Once the score network trained, MI estimation can obtained following the procedure explained in Algorithm 4. Two options possible, either by computing the difference between the parametric scores outside the same norm (Eq. (20) MINDE-J(o)) or inside (Eq. (21) MINDE-J). Similarly to the conditional case, option parameter be used switch among the two.
Algorithm 3: MINDE-J (Single Training Step)
Data: [Xg, Yo] j. parameter :neta (), with 9 current parameters
1 1,,0 7)
// importande sampling can be used to reduce variance
& e2), with 61,2 ~ " // 1.h.s. Eq. (16),
[X,, Y.] - (x,, Yol + (22 Sa ko
diffuse modalities to timestep
c ~ Bernoulli(d)
// Sample binary variable e with probability d
if c 0 then
[6
- neto (e Yıl, [1,
// Estimated unconditional score
L
|s,, 62] 16l 62]
// Compute Montecarlo sample
(A Ke ds)s
associated to Eq. (5)
else
(e) then
(Ng N5 ( t neto((Xe, Yol, [1,0] // Estimated Conditional score
T. [12.0 K5 92ds)
||e1 é1l1
else
neto ([xo, YrJ, 1, [0, 1)) Estimated Conditional score
(N2 1 k d2
62 E1
(12 k. de)
return Update 0 according to gradient of T,
C.3 TECHNICAL SETTINGS FOR MINDE-C AND MINDE-J
We follow the implementation of Bounoua et al. [2023] which uses stacked multi-layer perception (MLP) with skip connections. We adopt a simplified version of the same score network architecture: this involves three Residual MLP blocks. We use the Adam optimizer [Kingma & Ba. 2015] for training and Exponential moving average (EMA) with a momentum parameter m = 0.999. We use importance sampling at train and test-time. We returned the mean estimate the test data set 10 runs.
The syper-parameters presented in Table 2 and Table 3 for MINDE-J and MINDE-C respectively. Concerning the consistency tests ($ 5.2), we independently train an autoencoder for each version of the MNIST dataset with available.
18

---

Published as a conference paper at ICLR 2024
Algorithm 4: MINDE-J
Data: [Xo, Yol ~ n
parameter: ta, option
t ~~ U[0
11 Importance sampling can be used to reduce variance (x K] - o,, Yol + (o [2), with 1, 7 // 1.h.a. Eq. (16),
diffuse modalities to timestep
t neta Y,l, [1,
// Use the unique score network to compute
joint
8t +- neto([X,, Yol, f, (1,0))
// and conditional scores
"Xo neto((Xp, Yil, 1, (J)
8,
if opption = 1 then
i T
oi 2
8/
18t
Vo xxo
else
k,0² + k? k5 2 gg
Xt t
- T gi
s/e [x
%
Avo
X2 X
8/
Y: X
|
X1
return
f
Table 2: MINDE-J score network training hyper-paramcters. Dim of the task correspond the sum of the two variables dimensions, whereas d corresponds to the randomization probability.
Width Time embed Batch size
Lr
Iterations Number of params
Benchmark (Dim < 10)
0.5
64
64
128
1e-3
234k
55490
Benchmark (Dim 50)
0.5
128
128
256
2e-3
195k
222100
Benchmark (Dim 100)
0.5
256
256
256
2e-3
195k
911204
Consistency tests
0.5
256
256
64
le-3
390k
1602080
C.4 NEURAL ESTIMATORS IMPLEMENTATION
We use the package benchmark-ni mi' implementation to study the neural estimators. We use MLP architecture with layers of the same width in MINDE. We use the same training procedure in Czyž et al. [2023], including early stopping strategy. We return the highest estimate on the test data.
D ABLATIONS STUDY
D.1 a ABLATION STUDY
We hereafter report in Table 4 the results of all the variants MINDE, including different values of a parameter. For completeness in our experimental campaign, we report also the results of non neural competitors, similarly to the work in Czyž al. [2023]. In summary, the MINDE-C/) versions ("difference inside") " of our estimator prove to be more robust than the MINDE-C/J(o) ("difference outside") counterpart, especially for the joint variants. Nevertheless, the "difference outside" variants stable and competitive against very wide range of values of (ranging from 0.5 10), with their best value typically achieved for a 1.0.
interesting to notice that
'https///githbbs ctm. chtp//og-eeeee ctw.tth//mtmmt
19

---

Published as a conference paper at ICLR 2024
Table 3: MINDE-C score network training
Dim the task correspond sum of two variables dimensions, and d corresponds to the randomization probability.
Width Time embed Batch size
Lr
Iterations Number of params
Benchmark (Dim < 10)
0.5
64
128
le-3
390k
55425
Benchmark (Dim 50)
0.5
128
128
256
2e-3
290k
220810
Benchmark (Dim = 100)
0.5
256
256
256
2e-3
290k
898354
Consistency tests
0.5
256
256
64
1e-3
390k
1597968
20

---

Published as a conference paper at ICLR 2024
negative bias (red) and positive bias (blue). Our meshod an and all ewtiinannn miimasss Over 10 seeds using N 1000000000000000000000 0000 Sampp les compart pere catch aggiin the ground- trunt--tuthhhh. Color indicates relative
wwmggsampppii samples. Lis abbreviation (Mm: Mullinornal, St: Student-1, Nm:
0.4
0.4
0.5
0.4
0.4
0.2
0.4
0.4
0.4
00.4
0.4
044
12.4
Migdd B Sonmiia MN 1
0.3
0.3
0.3
0.3
0.
03
03
03
0.
0.3
0.3
0.3
x (edtiin Eo 13
1.7
177
1.7
1.7
1.6
16
18
17
16
17
1.7
12
1.5
X (adtiise mi Oolsoooo (1
00.4
04
0.4
0.5
0.5
05
0.
05
0.5
0.3
03
0,
03
0.
04
04
044
04
01.5
x 1x
0.3
0.3
0.3
0.3
0.5
0.5
0.5
0.5
6.2
0.2
03
0.3
0.3
03
0.3
0/2
0.5
0.2
0.2
0.2
0.4
0.5
04
0.
0.5
0.5
05
0.5
04
04
04
04
04
04
02
04
02
02
02
02
0.
02
02
02
01
02
03
02
02
02
02
033
03
03
0.3
0.3
02
03
03
03
03
03
0.1
0.3
0.3
(2:j0p 1.
kg
0.2
02
0.3
0.2
0.2
02
02
02
0.
02
02
03
0.2
0.3
0.5
2 6g
0.4
02
0.3
0.3
0.4
0.4
04
03
0.3
03
03
03
0.0
0.3
0.2
02
01
01
022
02
02
01
01
01
01
01
00
022
0.2
403
(1ppp)) X 19
1.0
1.0
1.0
1.0
1.0
0.9
0.9
0.9
0.
0.9
0.9
09
0.9
119
1.0
1.0
1.0
10:7
MW CHC 00 MM 5 S2 s8 45
L.
1.0
1.0
1.0
0.9
0.9
1o0
L0
L0
09
10
09
1.0
0.9
1.
LD
1.0
0.9
Mi COOP 6 Mh x 22
dg
AM COop & x 22
1.8
1.1
1.0
1.0
1.0
0.9
0.
088
09
09
09
09
09
09
09
09
02
17.3
1.0
0.9
0.9
0.9
0.9
09
088
BX
09
09
0.9
09
09
06
D.8
B/6
7.3
a C
5p
1.0
0.9
0.0
0.9
0.9
09
09
0.9
0.9
0.9
08
09
0.
09
08
09
02
0x
08
0
03
31
0.2
essiiminn SotLoTs were trained with 100k mmiiiing
1.0
1.0
0.9
0.8
0.9
0.9
0.9
0.9
0.9
0.9
0.9
09
09
0.9
0.9
09
0.9
08
0.8
0.8
0.8
02
17.5
0.9
MM COr 6 Aae 5 Xx 2
1.0
1.0
1.0
1.0
1.0
1.0
1.0
0.9
0:9
1.0
1.0
0.9
0.9
0.9
199
1.0
1.3
1.0
10:7
2.3
ww COD do Aa 8 xx Coaamn
1.0
1.0
1.0
1.0
1.0
140
1.
2.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
0.9
1.0
1.0
1,0
0.9
1.0
111
1.0
1.0
1.0
0.0
09
09
09
0.9
0.0
02
17:2
Mhe 2a was (eaaii))
0.4
04
0.
0.5
0.4
0-4
0.2
124
04
04
04
04
04
04
002
0.2
0.2
0.4
0.
11.2
ww Cme do Biveltno IXX
1.6
1.7
1.6
1:7
1.7
1.8
8.88
1.6
1.6
1.6
16
1.6
1.6
1.6
133
Ma 50 S0 (chmne)
0.6
0.6
0.6
0.5
0.6
06
06
0.6
0.6
06
06
0.6
0.6
06
06
0.6
016
0.6
0.6
966
0/6
6%
11.1
1.0
La0
1.0
1.0
1.0
1.0
1.0
0.9
0.9
1.0
1.0
L.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
Ay x (daeeo
04
044
044
0.4
0.4
04
0.4
044
0.4
0.4
2-5
L.
100
1.0
1.0
1.0
10
10
10
10
10
10
1.0
1.
Table 4: MINDE-1 and MINDE-c E ablationn study .M Man MI extima
1.3
1.3
1.3
1.1
1.3
1.3
1.3
1.2
12
122
1.3
1.3
17:3
25 25 Cdenn)
25 x 25 22 Pema
1.0
1.0
1.0
0.9
0.9
0.9
0.9
1.0
1.0
1.0
1.0
10
1.0
1.0
1.0
1.0
02
Ah
033
0.3
0.3
0.3
0.3
0.3
03
0.3
03
03
03
03
0.3
03
0.3
0.3
06
0.3
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.01
1.0
1.0
1
1.0
1
1.0
1.0
1.0
1.1
1.0
0.9
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
07
L.0
1ie 8 Aa 3 573 Saui
1.0
1.0
1.0
1.0
1.0
1.0
LO
10
1.0
L.0
1.0
1.0
1.0
0.9
2-
25 x -.---e
1.0
122
1.3
1.0
09
08
10
10
1.0
10
1.0
1.0
1.0
02
0.4
04
0.4
0.4
0.4
0.4
014
0.4
0.4
014
04
04
0.4
04
04
0.4
0.4
0.4
0.4
0.
01.4
04
04
04
0.4
0.4
04
0.4
0.4
04
04
04
04
04
04
044
0.4
0.4
04
04
0.4
0.4
Buuniin khr xi
04
04
0.4
044
044
04
0.4
04
044
04
04
04
04
04
044
Dimmea
Normal, Hc: Half-cube, Sp: Spiral)
04
04
04
04
04
04
04
04
04
04
04
01
04
04
04
02
68
Aaai de 2.
x (uur
03
03
0.3
03
0.5
03
02
02
0.2
03
03
03
03
03
03
02
0.3
0.5
035
0.2
20
04
04
04
04
0.4
04
04
04
04
04
04
04
04
04
04
04
04
04
04
044
04
04
09
Siim & S... 2iaeee)
62
02
02
02
02
02
02
02
02
02
02
02
02
02
02
02
02
02
02
0.2
0.2
0.2
0.2
0.0)
Aiii & S.xx (deee))
Jto 0.30
1)
1.30
2)
30
11
= 1.5)
21
5)
10)
MINDE-Joo
MINDE-Joe
110
MINDE- (o
MINDE-C
MINDE-C
MINDE-Clo
MINDL-C
MINDE-C
NIXOX
NANDE
InfeNCE
MINE
47
D-y
KWJ
KSG
LXN
CCA
21

---

Published as a conference paper at ICLR 2024
D.2 FULL RESULTS WITH STANDARD DEVIATION
We report in Table 5 mean results without quantization for the different methods. Figures 3 and contains box-plots for all the competitors and all the tasks.
22

---

Published as a conference paper at ICLR 2024
00 0 22.35 6 188 03 2200
0F
1LL 000 19 0:22 011
50 1.88 03 DAD
0.34 22 02 O7
Tiifoonn x1 (uidiie wiiee 29)
011 0,300 00000 022 21 17 073 30000 8889 0.2
UIS 1.00 0.2
LSV (11) 06:80
1 08 0:35
Ballomn 1x Coadhiie cotttee Q
LO2 1.02 1.0 1,02 102 02221440 500000 0.29 Cu ELS0 L.
02 1704 TOROON UUT
S anll Xx
11 LE SEEEE
THE
1M 0.59
S.xk 5 (iee))
0.00 di UU (20
200220000 L8 02 UI OZ
2 C MA
1.60 39 88 0.0 0.50 OS 0 0.S 03S 0.89 0 002 0 0.2 12 0OO0
0:29
g.3 3 (iE-3)
02 10
2
x 3
WOWWOIAA 099 0.88 01770 TM
000 OOLL
S.-- (2=
1899 0:9 (L 11122200
1,00 40 nie 0.9 U. DO0 OM OU OOR 9999 0.86 414002000
010022000
(
2 2.2
1 40 oo 0 0. 0. 0.34 095 0 0 94 01
AL
023
1- (
1.100.00 d.40 0.92 008 0199 0. OA ON (192 0.9 0.90
0996 0.66
1,00 A0 0.90 0.98 0198022 0.82 ONN 082 0095 0.96
0:00
10200 0.99 (19 (190000 1.5% 0.56 055 0.72 0.77
16 15 0.41 0:78 1.0 1.05 10000r ox 0.50 0.5 0.67 0:79 0.51
Xx (oapno)
Ay 5 4% wox acD sn
0.55
Ss va ww Cal d0 Aa a xx C..pmn)
10 0. OUR 1.07
A. aa MV Con d0 Ay 2x xx (Z.pht)
0.71 0L55
9.66 e Ah x (3 danii
Table 5: Mean estimate over 10 seeds using N= = 10000 samples compared each against the ground-truth
096 oue (199 0U
e Mh zz x ş7 (2ppnn
1.61 0.610.0 0:93 0.9 0.90
0200
201
List of abbireviations Mr. Multinornal, St: Student-t, Nm: Normal, He: Half-cube Sp: Spiral)
1,7 OOW
0.9
20 Con Aa x y3 2puurr
0.420
LO0
0.37/000
Mhr COD 8 Ah 28 x 33 2...nnn Canie
1,62004
1.75044
0.29
0.39
16 0.37
Derreiin I A.
1.72
1:28
1:75
Our method and all neural estimators were trained with 100k training samples.
10114 0.50 1002 102 1.00 02.029 1,02 1,02011 1.20059
100 1o 0:29000 1.1 10000-2 100 0.50
101 0.5%
(0) 0.59
001 0.50
1.00 0.59
0.59
1.02 0.59
0.57
122
1,2
5 xxx (deeee))
1.02
1.02
7:31
0.8-
0.0%
3 xx (2 pudd-z)
000220 01103 022 042 141041 4 00% 100 1O 29 10000 L.26 1o1 0-11
0.21 0 028 4 1. 0:41 0:41 1OR 1on cai o0 0.29 1:00 1,27 1.01 0-11
0.23 0.24 036 4 0:41 099 05% 0.08 1.OL 0.50 0.99 1.28 1.01 0.41
0:41
0.41
0:41
2:99
0.64
0.83
M 3 (dmmee)
1,28 1.01
0.95
0.95
0. 0.33 0.21 045 044 0.35 0. 3. 0002 0.22 0.92 98 0.34 0.340.00 0.09 8. 095
Ao 5 x () paut)
Ao 22 z5 deaace
1.51
1.28
006 788
MM xsz 22 Coooi
0.410. 1044 0.41 09% 101 LO1 OZ 11.29 (199
41 0-L 0.98 101 (01 1.02 0.29 0.99
1066
0.25
1.00
95
22 e 6e Crui
(US 7.8M
D. 0 0.27 0.7 0.33 39 (00R 089 0.96
Sx Sommi
2.6
14 op Aat x gund
0.9
0.42 COTTT
0.99
0.25 7 605 0.41 0.42 0.42
0.22 38 19 0228 0.420.40..4
Brariie Mhn xx
Bumndtg I xx
02 0.00 A
0.22 0.27 4,40 0:41
0,22 AA 0.2 O.60
0.22 0.40
0.22 41 0.21 0.40
@ Xx
oe 0.3 Xx (aoeec
or 21 x (aoee
00000000
uuml e8 1 X. (ooear))
MINDIE (or
Dalli Canssien)
Dodi DaBiLLogan Loggste)
MINDE-
MINDAE
MINDE
InfeNCE
GT
MINE
NWJ
KSG
LNN
CCA
dist
23

---

Published as a conference paper at ICLR 2024
Natiomall 2 212
Junn Studemi11 180-21
Studemr13 1537-31
Maleet 1100-21
Halicate
liptoube Huthooma 25 25 (30
Hahouke (.
Figure 3: We report MI estimate results over 10 seeds for 1100000 for our method and competitors for training size 100k sample. A method absent from the depiction implies either convergence during training results out of scale
24

---

Published as a conference paper at ICLR 2024
Studen611 teat-3:
Normll car Nutremmet 23 Cown
pride.
Nomra CSF Muloecrall 300-00)
Mede 600 21oam22
Navr Votrerrals
Studeii2 (pat-3)
Studemed (oat-33
Studenee icat-31
spec termel Muthoomni
backke
Spasl Nermacc C3' PLtnorall
ALEEerr 3 Olgann
Speii Serrac (hinni
Figure 4: We report MI estimate results over 10 seeds for N =10000 for our method and competitors for training size 100k sample.
D.3 TRAINING SIZE ABLATION STUDY
We here report, in Figures to the results of our ablation study on the training size, varying in the range 5k, 1kk,1,,500,,00 100k.
25

---

Published as a conference paper at ICLR 2024
ABlhi statreec LiAN4
MAPE
10G
"6'"""
16
10th Suteen2 2010-
150
16
10
Joand Stxtet:5 536m))
JAME
UM
03
scing
9021
4B
1. 10
5:30
5:30
1817
MAKE
17
100
18
DeDOmumtee
150
11
KiMEZ
100
19
Figure 5: Training Size ablation study We report MI estimate results for method and competitors as
function of the training size used (0K, For readability, discard the baselines with estimation (error GT) or high standard deviation. All results are averaged over seeds. Due the benchmark size, we split the results into 4 figures each containing 10 benchmarks. A method absent from the depiction implies either non convergence during training or results out of scale. In this first plot we report tasks 1-10.
26

---

Published as a conference paper at ICLR 2024
PRT semaa 2 x2
MADE
POACE
16 0.10
6:30
6
MA
4300-2011
2907
150
10
10 10
MMB 21 22 mu))
AMME
UM
11
1.00 10
10
102
Un
1. 5
b 10
8:30
3.
4074
1817
MAKE
07
100
OM
Dx Doc
1M18:
KMC
1382
10
Dac loganii
40224
DeDGmunter
6'6'''6
Figure 6: Part 2 of Figure 5, tasks 11-20.
27

---

Published as a conference paper at ICLR 2024
noma ca Matherea do 2ope soakE
1b ts
Herra CT 3 33
M500-20-1
1007
HANT
1b 16
80
Nerry (TP
180
1MM
UM
16
20
1p wad sore 23223000
11
X3022
Un
10
M500-2e-
By
W1
UM
Dxt Dec
Mi2
KMAC
11
1382
DeDGeunneeri
Figure 7: Part 3 of Figure 5, tasks 21-30.
28

---

Published as a conference paper at ICLR 2024
Standt2 1212-11
M300
Est Logato
2M00300
MADEC
UM
Can Jagar))
PAE
CV
501
105
UW
DvDGowster
Eot SogMto
LM
10001
2807 ke
15 10 10
10 15
10 30
1821
100
Waw
2007 2001
PM
501
100
+4+
4.
Figure 8: Part 4 of Figure 5, tasks 31-40.
E ANALYSIS OF CONDITIONAL DIFFUSION DYNAMICS USING MINDE
Diffusion models have achieved outstanding success in generating high-quality images, text, audio, and video across various domains. Recently, the generation of diverse and realistic data modalities
29

---

Published as a conference paper at ICLR 2024
(images, videos, sound) from open-ended text prompts [Ramesh et al., 2022; Saharia et al., 2022; Rombach et al., 2022] has projected practitioners into a whole new paradigm for content creation. A remarkable property of our MINDE method is its generalization to any score based model. Then, our method can be considered as a plug and play tool to explore information theoretic properties of score-based diffusion models: in partieular, in this section we use MINDE to estimate MI in order to explain the dynamics of image conditional generation, by analyzing the influence of the prompt on the image generation through time.
Prompt influence of conditional sampling. Generative diffusion models can be interpreted as iterative schemes in which starting from pure noise, at each iteration, refinements are applied until sample from the data distribution is obtained. In recent work on text conditional image generation (image A, text prompt B) by Balaji et al. [2022], it has been observed that the role of the text prompt throughout the generative process has not constant importance Indeed: "At the early sampling stage, when the input data to the denoising network is closer to the random noise, the diffusion model mainly relies on the text prompt to guide the sampling process. As the generation continues, the model gradually shifts towards visual features to denoise images, mostly ignoring the input text prompt" (Balaji et al., 2022]. Such claim has been motivated by carefully engineered metric analysis such as self and cross attention maps between images and text, as a function of the generation time, as well as visual inspection of the change in generated images when switching the prompt at different stages of the refinement.
refine heuristic-based methods and produce a similar analysis using theo- retically sound information theoretic quantities. In particular, we analyze the conditional mutual information I(A, B| (x)), being X, the result of the generation process time T (recall that the time runs backward from T to 0 during generation, and consequently A = Xo and B = Yo). Such metric quantifies, given an observation of the generation process time т, how much information the prompt B carries about the final generated image A. Clearly, when r 7= T, the initial sample is independent from both and B. Consequently, the conditional mutual information will coincide with I(A, B).
Using MINDE,
More formally, we consider the following quantity:
1(4, B) [1(X,, B) I(X,, A| B)]
(26)
I(4, - [M(*+) - H(X+IB) H(X+IB) - H(X-1A, B)]
(27)
= =I(A, B) I(X,, B);
(28)
where Eq. (27) is simplified due to the Markov chain
X - so H(X-IA, =
H(x+|Xo: B) = (X))B). Next, we use our MINDE estimator, whereby the marginal and condi- tional entropies can be estimated efficiently. The following approximation of the quantity in interest. can be derived:
of 2 (x.)
Avo (x,)
I(A XX,) = Epue
84
dt
(29)
In our experiments, we also include a MINDE-(O) version which can be obtained similarly to Eq. (29).
Experimental setting. We perform our experimental amalysis of the influence of prompt on image generation using Stable Diffusion [Rombach et al., 2022], using the original code-base and pre-trained checkpoints." The original Stable Diffusion model was trained using the DDPM framework [Ho et al., 2020] on images latent space. This framework is equivalent to the discrete-time version of VPSDE [Song et al., 2021]. Using the text prompt samples from LAION dataset Schuhmann et al. [2022], we synthetically generate image samples. We set guidance mechanism to 1.0 to ensure that
hhttps://hugiinggaces stabi 1ibbiittyan tyyyii/a///// aittti/ssaaat /sttblettallleeeee di ffusi on-2-1
30

---

Published as a conference paper at ICLR 2024
the images only contain text conditional content. We use 1000 samples and approximate the integral using a Simpson integrator with a discretization over 1000 timesteps.
Results. We report in Figure 9 values of [(A, B X,) as function of (reverse) diffusion time, where A is in the image domain and B is in the text domain. In similar vein to what observed by Balaji et al. [2022], our results indicate that I(A, B X,) is very high when = T. which indicates that the text prompt has maximal influence during the early stage of image generation. This measurement is relatively stable at high MI values until T = 0.8. Then, the influence of the prompt gradually fades, as indicated by decreasing steadily MI values. This corroborates the idea that mutual information can be adopted as an exploratory tool for the analysis of complex, high dimensional distributions real use cases.
The intuition pointed out by our MINDE estimator is further consolidated by the qualitative samples in Figure 10, where we perform the following experiment: we test whether switching from an original prompt to different prompt during the backward diffusion semantically impacts the final generated images. We observe that changing the prompt before r = 0.8 results almost surely with semantically coherent generated image with the second prompt. Instead, when т 0.8, the second prompt influence diminishes gradually. We observe that for all the qualitative samples shown Figure 10 the second prompt has influence the generated image after < 0.7.
60
MINDE (a-1)
MINDE
50
(nats)
40
30
20
10
0-
1.0
0.8
0.6
0.4
0.2
0.0
Figure 9: [(A, B Xx) as a function of T.
:/cccss httptp////ooosss.yy..ooo///////s wttpyy/wwwoo/oot//////eee/////ee integrate. simpson. html
31

---

Published as a conference paper at ICLR 2024
Poorge phats dtting Prompe photo pands Jingls
T- 1.0
0.0
0.8555
C- 0
T- 0.75
T- 0.7
T-0.6
T- 0.5
0.4
0.3
0.2
0.1
hompt::
arrieen
0-00
2-00
0:75
2- 0.7
m-6666
y- 0.5
T-04
0.3
D-
0-
Propet 281 Inece of 30 In the deart
1-1.0
T-00
T 0.85
T-m 0.8
T- 0.55
T-
T.0.6
T-
T=0.4
0.3
T- 02
T 0.1
STRTE
1 1 Iruge of ag beat forthite nec.
validate the explanatory results obtained via the application of our MINDE estimator, we perform the following experiment: Conditional generation is carried out with Prompt / until time т, whereas after the conditioning signal is switched to Prompt 2. We use the same Stable diffusion model as in the previous experiment with guidance scale 9.
Figure 10:
F SCALABALITY OF MINDE
In this Section, we study the generalization of our MINDE estimator more than two random variables. We consider the information interaction between three random variables A,B and C, defined as:
I((,,)))I(AABB))))
(30)
= H(A - -(AB)) -
(((
(31)
Estimation of such quantity is possible through a simple extension Eq. (17) to three random variables, considering three parameters o, 3, 1).
In particular, we explore the case where the three random variables are distributed according to M. B A + N₁ (with N₁ 7e) and C A N₂ (with N2 ~ 7/)) By changing the values of the parameters, it possible to change the value of the interaction information. We report in Figure the estimated values versus the corresponding ground truths, showing that MINDE variants can be effectively adapted for the task of information estimation between more than two random variables.
multivariate Gaussian distribution:
MINDE-j(o =1)
MINDE:j
GT
Estimated
True MI
Figure 11: MI estimation results for MINDE-j on 3 variables
32


Reviews:
[{"review_id": "dHboY3HaK8", "reviewer": "Reviewer_N2Vu", "summary": "The authors derive an estimator of mutual information using neural diffusion.", "questions": "Things that would inspire me to increase the score: \n1. Explain (more/better) why they are talking about SDEs for estimating MI.\n2. Perform statistical analyses and/or errorbars indicating whether MINDE is significantly better than anything else on any particular simulation.\n3. Clarify how many different settings were considered, and discuss from that perspective.\n4. More clear/complete pseudocode.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "I really like that the authors used the Czyz benchmark data, and also the consistency tests. \nI also appreciate the creativity of the theoretical advancement, though I don't understand it (see below).", "weaknesses": "I was super excited to read this paper, because I love thinking about mutual information and entropy, and have recently been working on some related issues.   The ideas are intriguing, and the results are impressive. So, the rest of this review will focus on the issues for me understanding the methods and results.\n\n1. The biggest issue for me is that I almost immediately got lost.  I know information theory pretty well, I learned it from Fred Jelinek before he died. That said, I know very little about diffusion processes and SDEs. My main confusion with this paper was about connecting the math on diffusion processes to the process of estimating MI. There is a leap, which I am willing to believe is justified, that I completely missed.  Why are we talking about a filtration and an Ito process at all?  How do they related to the joint distribution F_{X,Y}? I read the words in Section 2-4, but was completely lost.  To be fair, I was also lost the first time I read the KSG paper (https://journals.aps.org/pre/abstract/10.1103/PhysRevE.69.066138).  I imagine lots of people might have followed the logic and derivation completely. But not me, I just didn't get it. And I spent some time trying to figure it out, as I'd like to get it, it seems cool, and within the realm of possibilities that I did get it, but I didn't. I thought maybe reading Appendix D would help me, but it didn't really help either.  In the end, I don't quite know what you did, or why you did it.  I would love something like Algorithm 1 and 2, which perhaps points to subroutines for how each relevant quantity is computed. For example, I don't see how to do \"r.h.s. Eq. (16)\". Where does 'g' come from, or k, or T? etc.\n\n2. In terms of the numerical results, I think I understand them, which was exciting for me! Figure 1 shows that MINDE works about as well as other things on relatively easy problems where there is enough data, and slightly better on a spiral dataset when MI is high.  That's cool as far as it goes.  I'm always interested in *finite sample properties* for my estimators, because I always have finite data.  In particular, I often work on biomedical problems, in which sample sizes are typically hundreds.  So, I would be much more interested in seeing plots showing accuracy as a function of sample size, especially for the \"easy\" ones where many different estimators are getting the right answer. This introduces additional information about convergence rates. The fact that 2 high-dimensional simulations showed it does as well as other things, and one showed it is slightly better for some parametrization, I found not that compelling.\n\n3. I understand why the authors say that the benchmark consists of 40 tasks.  However, in my opinion, this wording is confusing and misrepresenting the work. I would say that there are about 10 different tasks, with an average of 4 different parameterizations per task. Consider, for example, our paper https://elifesciences.org/articles/41690. We describe 20 tasks, but in Figure 2, we should many different parameterizations (dimensions) per task. Claiming that we had more than 20 tasks, in my opinion, would not be in integrity.  Varying parameters, dimensions, and sample sizes for a particular task is important, but claiming that each different parameterization is a different task seems inappropriate to me. Of note, the Czyz et al paper from which the tasks are extracted never seems to make such a claim. Rather, they name about 10 different tasks explicitly.\n\n4. The claim that MINDE outperforms other approaches on 35/40 tasks I also question. No errorbars are provided.  While this is typical in machine learning benchmark comparisons, I think the practice is ill-advised and misleading.  Without errorbars, there is no evidence that if one ran the exact same code again, how likely is it that the results would be similarly ordered.  There is a rich history of non-parametric tests for evaluating whether one estimator tends to be better than another, and I would encourage the authors to only claim something is better when there is statistical evidence supporting the claim. \n\n5. Directly using kernel density estimators and plugging them into the MI equations, or using the standard approaches to estimating mutual information (eg, KSG, which is included in sklearn), to compare with the neural methods, would also be important.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AaawTjzVbJ", "reviewer": "Reviewer_hHbr", "summary": "This work proposes *Mutual Information Neural Diffusion Estimation (MINDE)*, a novel method to estimate the mutual information (MI) between random variables. By first decomposing the KL divergence between two generic measures into two terms via its disintegration properties, then utilizing the Girsanov Theorem, MINDE provides a recipe that incorporates score-based diffusion models into the estimation of MI. The work provides four variants of MINDE, each based upon either conditional or joint diffusion processes, and presents experimental results that not only show the effectiveness in estimating MI accurately, especially on more challenging tasks (*e.g.*, spiral diffeomorphism), but also illustrate the robustness and reliability of the proposed method via self-consistency tests (including independency test, data-processing test, and additivity test) on the MNIST dataset.", "questions": "1. Could the authors discuss the connection between this work and MINE (Belghazi et al., 2018)? The title of this work seems to suggest a close connection with MINE, but the paper only provides the experimental results of MINE as one of the baseline models for MI estimation.\n2. In Section 5.1, it is mentioned that using a larger training size shall “avoid confounding factors”. What do authors mean by “confounding factors”?\n3. Might be a typo in Page 3: “Radon-Nikodyim derivative” shall be “Radon-Nikodym derivative”.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The construction of the basic building blocks that establish the estimation of KL divergence and of the entropy is well organized and clearly written.\n2. It’s interesting to see the SDE framework of diffusion models being used under the setting of MI estimation, which could inspire the research community to investigate diffusion models in new directions.", "weaknesses": "1. While the utilization of score-based diffusion models can be justified by the Girsanov Theorem, it’s unclear how they are used as **generative models** (*i.e.*, using the reverse-time SDE to generate samples) — it seems that only forward diffusion SDEs are needed, in order to train the score networks. Therefore, it’s a bit confusing when the authors wrote “we explore the problem of estimating MI using generative models” (Page 1), instead of something like “we explore the problem of estimating MI using score functions”.\n2. Source code for the experiments is not provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EJi5bVTcYf", "reviewer": "Reviewer_QLQ9", "summary": "The paper proposes mutual information (MI) estimators based on diffusion models. This is achieved by representing MI as quantities that incorporate KL divergence which itself can be written as equations that involve score functions. Theoretical arguments justifying these claims are presented in the paper and experimental evaluation is performed on a recent benchmark dataset proposed for assessing MI estimators from multiple viewpoints (sparsity, dimensionality, long tails, and transformations). The results suggest that the developed estimator (MINDE) outperforms alternatives in most of these settings.", "questions": "How does the method scale with the number of data points used for training? Can you make a plot of the test error as a function of the number of training data points used? The number of training data points can vary between 100, 1K, 10K, and 100K. Can you do this for a varying number of dimensions as well and report results for different estimators?\n\nInfoNCE seems to be doing a very good job, can you come up with an overall score to rank the methods? Does InfoNCE also require a large dataset with the same size as yours?\n\nCan you include time comparisons between different models? I assume learning the score functions from a diffusion process for the joint probability distribution would be an overkill if one only cares about the MI or H. What’s the training time comparison between different models and how do the authors account for computational resources used by various models?\n\nDiffusion processes are known to outperform other density estimators in various settings. Therefore it’s no surprise if it achieves better an estimation of MI and H. That said, I’m having a hard time determining what the main contributions are. I imagine that the main contributions are representing MI and H as quantities that incorporate score functions. However, the connections between KL, score, MI, H are known results in the information theory literature [cite]. Is the extension of those results to the diffusion process (as opposed to generic densities) non-trivial? Is there something critical that I’m missing?\n\nThe paper would benefit from reorganization in my opinion. The notations used in the paper as well as the organization of the sections make it difficult to follow the arguments of the paper. The contributions start appearing very late in the paper and there is a large amount of background which might not be necessary for the main arguments. I suggest the following organization:\n* Simple intro to diffusion models (no need to include the measure theoretical notation as it’s mainly used for the proofs and can be transported to the supplementary).\n* Introducing the joint diffusion model (4.1) and MINDE.\n* Making connections between MI and H, score functions, and KL divergence.\n\nA recent paper [1] discusses that the absolute continuity assumption made in the paper might not hold given the architecture of the neural network used for approximating the score function (the cited paper discusses it in the context of functional variational inference but intuitively the same arguments should hold for diffusion process). In this case, the KL divergence will be infinity and the MI and H estimators developed will be ill-defined. Can the authors articulate the underlying assumptions further and explain what datasets can benefit from the estimators developed in the paper?\n\n[1] https://arxiv.org/abs/2011.09421", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The problem considered is of critical importance in several applied and theoretical fields. Existing estimators either fail in high dimensions or require large amounts of data to provide precise estimates.\n\nThe results are quite impressive, the proposed estimator seems to outperform alternatives in most settings.\n\nSeveral aspects of MI estimation that make the estimation challenging that were originally introduced in [1] such as sparsity, dimensionality, long tails, transformations, data processing, and consistency are considered in the experimental section making the empirical aspect of the paper strong.\n\n[1] https://arxiv.org/abs/2306.11078", "weaknesses": "The organization of the paper makes it hard to follow. The measure theoretical notations make the paper inaccessible to the broader audience interested in using the estimator in applied settings.\n\nThe contributions are not fully clear. The connections between score, KL, MI, and H existed before. In addition, it's an established fact that diffusion process models are more powerful density estimators specifically in higher dimensions making it less surprising that the MI and H estimators are superior.\n\nThe comparisons lack several important aspects specifically in the context of MI estimation. The wall-clock runtime and the dataset size requirements are not particularly elaborated on in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P45fmNHBkz", "reviewer": "Reviewer_a1nb", "summary": "The paper introduces Mutual Information Neural Diffusion Estimation as a family of mutual information estimation models based on estimating the difference of score functions.\nThe authors introduce and evaluate 4 variants based on the modeled scores (joins vs conditional) and the use of a standard normal as a reference for entropy computation.\nAn experimental section validates the theory on common mutual information estimation benchmarks, comparing MINDE against modern alternatives in literature and assessing self-consistency and compositionality.", "questions": "1) How does MINDE perform compared to classic generative estimators based on the difference of cross-entropies based on normalizing flows such as DoE in [1] and GM in [2]?\n\n2) Can the author elaborate on the differences between MINDE and the work in [3]?\n\n3) What are the main challenges when training the MINDE models? How does the training and inference cost (in terms of memory and computing) compare to the other neural estimators?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1) The paper provides a solid, detailed derivation for MINDE rooted in SDE theory.\n\n2) The experimental section effectively demonstrates the effectiveness of the proposed estimators on common benchmarks.\n\n3) Although a Related Work section is missing, to the best of the reviewer’s knowledge, the paper includes references to all the relevant literature.\n\nOverall I believe in the relevance and novelty of the submission and I am willing to increase my score whenever the authors address my main concerns.", "weaknesses": "# Main concerns\n1) The experimental section benchmarks the estimators against common discriminative estimators such as MINE, NWJ, D-V, and InfoNCE which are designed as lower bounds of mutual information, but no comparison against generative estimators based on difference of entropies is provided [1,2]. Since MINDE, and in particular MINDE-$\\sigma$, is based on the same principle, such a comparison seems natural.\n\n2) The paper mentions previous similar work on diffusion-based mutual information estimation [3], which differs in the derivation and modeling choices. Nevertheless, this work is not included in the experimental comparison, and the advantages of MINDE are not further elaborated.\n\n3) No discussion regarding the computational cost or challenges of training MINDE compared to the other models in the literature is included in the main text.\n\n### Minor Remarks\n1) The main text includes in-depth technical details with an extensive notation. If on one hand, this helps to verify the soundness of the derivation, on the other, it makes following the main derivation more difficult. I believe that submission could benefit by including additional intuition to guide the reader.\n\n2) The plots in Figures 1 and 2 are quite small and difficult to read\n\n### References\n[1] McAllester, David, and Karl Stratos. \"Formal limitations on the measurement of mutual information.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2020.\n\n[2] Song, Jiaming, and Stefano Ermon. \"Understanding the limitations of variational mutual information estimators.\" ICLR, 2020.\n\n[3] Kong, Xianghao, Rob Brekelmans, and Greg Ver Steeg. \"Information-Theoretic Diffusion.\" ICLR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)