Mutual Information Estimation via Normalizing Flows
Authors: Ivan Butakov
Alexander Tolmachev
Sofia Malanchuk
Anna Neopryatnaya
Alexey Frolov
Venue: NeurIPS 2024 poster (2024)
Keywords: Normalizing flows
information theory
mutual information
Primary area: probabilistic_methods
TL;DR: Normalizing flows are used to allow for explicit mutual information estimation via closed-form expressions

Abstract: We propose a novel approach to the problem of mutual information (MI) estimation via introducing a family of estimators based on normalizing flows. The estimator maps original data to the target distribution, for which MI is easier to estimate. We additionally explore the target distributions with known closed-form expressions for MI. Theoretical guarantees are provided to demonstrate that our approach yields MI estimates for the original data. Experiments with high-dimensional data are conducted to highlight the practical advantages of the proposed method.

Mutual Information Estimation via Normalizing Flows
Butakov I. D.
Tolmachev A. D.
Skoltech, MIPT? Sirius
Skoltech, MIPT tolmachev. hahaachee..dddphystech.ee
butakov. id@phystech..u
Malanchuk V. MIPT, Skoltech
Neopryatnaya A. M. MIPT, Skoltech neopryatnnya..amllyytee
mal balanchuk.ssven www.ccuww..@pppyttcc......
Frolov A. A.
Skoltech
al w.frollv@skoleech..r
Abstract
We propose novel approach to the problem of mutual information (MI) estimation via introducing a family of estimators based on normalizing flows. The estimator maps original data the target distribution, for which MI is easier estimate. We additionally explore the target distributions with known closed-forn expressions for MI. Theoretical guarantees are provided to demonstrate that approach yields MI estimates for the original data. Experiments with high-dimensional data are conducted highlight the practical advantages of the proposed method.
Er-gE
E-NitO.TG))
(diffcomorphlemm)
(ditfcomarphism)
(unlasowa) [((x))
((em)) I (oaay to estimate)
(unkenwwn)) [uxx)
IiEml (Lrmmtaable)
fr
(6.))A((0.))
(diffoomnophhim))
(diffcomorphism)
(a) MIENF (general base distribution).
(b) N-MIENF (Gaussian base distribution..
Figure 1: We propose transforming pair of random vectors (RVs) via Cartesian product of learnable diffeomorphisms to facilitate mutual information (MI) estimation. Ideally, we achieve tractable MI in the latent space. As diffcomorphisms preserve information, MI between latent representations equals MI between the original RVs.
Introduction
Information-theorete analysis of deep neural networks (DNN) has attracted recent interest due intriguing fundamental results and new hypotheses. Applying information theory to DNNs may provide novel tools for explainable AI via estimation of information flows (1H5). as well as new ways to encourage models to extract and generalize information [ 6H8). Useful applications of information theory the classical problem of independence testing are also worth noting --
"kolkovo Institute of Science and Technology
Moscow Institute of Physics and Technology Sirius University of Science and Technology
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Most of the information theory applications to the field of machine learning are based on the two central cntommtiion-thhoree quantities: differential entropy and nutual information (MI). The latter quantity is widely used as an invariant measure of the non-linear dependence between random variables, while differential entropy is usually viewed as measure of randomness. However, as has been shown in the previous works [12] [3]. MI and differential entropy are extremely hard to estimate in the case of high-dimensional data. is argued that such estimation is also challenging for long-tailed distributions and large values of MI [4]. These problems considerably limit the applications of information theory to real-scale machine learning problems. However, recent advances in the neural estimation methods show that complex parametric estimators achieve relative practical success in the cases where classical MI estimation techniques fail 7 [5]
This paper addresses the mentioned problem of the mutual information estimation in high dimensions via using normalizing flows (21-24). Some recent works also utilize generative models to estimate MI. According to a general generative approach described in [16], generative models can be used to reconstruct probability density functions (PDFs) of marginal and joint distributions to estimate differential entropy and MI via Monte Carlo (MC) integration. However, as it is mentioned in the original work, when flow-based generative models are used, the resulting estimates are poor even when the data is of simple structure. This approach is further investigated in [5]] The estimator proposed in [20) uses score-based diffusion models to estimate the differential entropy and MI without an explicit reconstruction of the PDFs. Increased accuracy of this estimator comes at cost of training score networks and using them to compute an MC estimate of Kullback-Leibleee divergence (KLD). Finally, in [TT] normalizing flows used to transform marginal distributions into Gaussian distributions, after which a zero-correlation criterion is employed to test the zero-MI hypothesis. The same idea is later used in [25 (see DINE- (NNE-Gassiann) acquire MI estimate, but corresponding error bounds are possible to derive, as knowing marginal distributions only is insufficient to calculate the MI (see Remark 4.5) which makes this estimator substantially flawed. We also note the work, where normalizing flows are combined with k-NN entropy estimator [88.
In contrast, our method allows for simplified (cheap and low-variance MC integration is required) or even direct (i.e., no MC integration, nearest neighbors search or other similar data manipulations required) and the accurate MI estimation with asymptotic and non-asymptotic error bounds. Our contributions in this work are the following:
1. We propose MI-preserving technique to simplify the joint distribution of two random vectors (RVs) via Cartesian product of trainable normalizing flows in order to facilitate the MI estimation. Non-asymptotic error bounds are provided, with the gap approaching zero under certain commonly satisfied assumptions, showing that our estimator is consistent.
2.
We suggest restricting the proposed MI estimator to allow for a direct MI calculation via a simple form We further refine our approach to require only ((d) additional learnable parameters to estimate the MI (here d denotes the dimension of the data). We provide additional theoretical and statistical guarantees for our restricted estimator: variance and non-asymptotic error bounds are derived.
3. We validate and evaluate our method via experiments with high- -igh--imensional synthetic data with known ground truth MI. We show that the proposed MI estimator performs well in comparison to the ground truth and some other advanced estimators during the tests with high-dimensional compressible and incompressible data of various complexity.
This article is organized as follows. In Section 2 the necessary background is provided and the key concepts of information theory are introduced. Section 3 describes the general method and corresponding theoretical results. In Section 4 we restrict our method to allow for accurate MI estimation via a closed-fore formula. In Section 5 a series of experiments is performed to evaluate the proposed method and compare it to several other key MI estimators. Finally, the results discussed in Section 6
We provide all the proofs in Appendix A
additional details on the benchmarks we use in Ap- pendix B overfitting analysis in Appendix C supplementary results regarding the information-based
disentanglement of real data in Appendix D
and technical details in Appendix E

---

2 Preliminaries
Consider random vectors, denoted as X: + R" and Y: 9 + R", where Q represents the sample space. Let us assume that these random vectors are absolutely continuous, having probability density functions (PDF) denoted as p(r), p(y), and p(r, y). respectively, where the latter refers the joint PDF. The differential entropy of X is defined as follows:
h(X) E (lgg(()) = p(x) log p(g)) dr,
supp X
where supp X C R represents the support of X, and log(-) denotes the natural logarithm. Similarly, we define the joint differential entropy as h(X, Y)
H log p(x, y) and conditional differential entropy as hXX Y) E log p (X = By (E log p(X = y)) Finally, the mutual
information (MI) is given by T(X;Y) h(X) h(X X Y), and the following equivalences hold
I(X;Y) = k(X) - h(X Y) = h(Y) - h(Y X):
I(:;Y)= h(x) + h(Y) - h(X,Y), I(X;Y) DRL (px,y px & py)
(2)
Mutual information can also be defined as an expectation of the pointwise mutual information:
p( y) p(x)
PMIx,y (x, = log
I(X;Y) = -EMMIx,(XXX)
The above definitions can be generalized via Radon Nikodym derivatives and induced densities in case of distributions supports being manifolds, see (26).
The differential entropy estimation separate classical statistical problem. Recent works have
proposed several novel ways acquire the estimate in the sigh-dimensional case [12 18 20 27 27
Due to Equation ( mutual information can be found by estimating entropy values separately. In contrast, this paper suggests approach that estimates MI values directly.
We also have to mention the well-known fundamental property of MI, which invariance under smooth injective mappings. The following theorem appears in literature in slightly different forms [14
19 31 331: we utilize the one, which is the most convenient use with normalizing flows.
Theorem 2.1. Let E: + Ra be an absolutely continuous random vector, and let g: R=' - R" be an injective piecewise-smone smooth mapping with Jacobian J, satisfying 2 n'' and det (J 0 almost everywhere. Let PDFs Pę and Pein exist. Then
PMII..(E.n) PMIT)e).n(((E)..))
I(E: (g(E):
(5)
In our work, we heavily rely on the normalizing flows 23 24 trainable smooth bijective mappings with tractable Jacobian. However, to understand our results, (a) satisfy the conditions on in Theorem 2.1 by definition, (b) can model any absolutely continuous Borel probability measure (universality property) and (c) are trained via a likelihood maximization, which is equivalent to Kullback-Leible divergence minimization. For more details, we refer the reader more complete and rigorous overview of normalizing flows provided 34
sufficient to know that flow models
General method
Our task is to estimate ((X;)), where X, Y are random vectors. Here focus the absolutely continuous (X, Y). as it is the most common case in practice. Note that Theorem 2.1 allows us to train normalizing flows fx, fy, apply them to X, Y and consider estimating MI between the latent representations, as I (X): (Y)) = I(X:Y).
The key idea of our method is to train fx and fy in such way that estimate. For example, one can hope to acquire tractable pointwise mutual information (PMI), which can be then averaged via MC integration restricts the possible distributions of (fx (fx(X), (x()) to an unknown family, making the exact MI recovery via such technique unfeasible.
is easy to
32 Unfortunately, the PMI invariance (Theorem 2.1
3

---

However, one can always approximate the PDF in latent space via a (preferably, simple) model q E Q with tractable PMI, and train 9. fx and fy minimize the discrepancy between the real and the proposed PMI. The complexity of q serves as tradeoff: by selecting poor Q, one might experience a considerable bias of the estimate; on the other hand, choosing Q to be universal PDF approximation family, one acquires consistent, but computationally expensive MI estimate. Flows fx. fy are used to tighten the approximation bound. We formalize this intuition in the following theorems:
Theorem 3.1. Let (E, n) be absolutely continuous with PDF PE.v Let q be PDF defined on the same space as Pe.y Let pę, Prr 9g and 91, be the corresponding marginal PDFs. Then
(E,)(,,)))
[(5: n) = E..,, log
-DEL (Pem (l -DLL (pe @ Pp I 98 @ 9n)
(6)
9 (g
(((m))
Corollary 3.2. Under the assumptions of Theorem 3.1 (T(E:n) Iy(E:n)) S DRL. (Pen (e..)
This allows us to define the following MI estimate:
N Σ log (x.n(fx(xx) fy (bx)) (1x)) (ux))
IMIENF (1 (o (o)) (x(xx((xx
(7)
where {<r, yk) is a sampling from (x, Y). and q. fx and fr are selected according to the maximum likelihood. The latter makes MMENN consistent estimator:
Corollary 3.3 (MIENE is consistent). Let X, Y, fx1 and fy+ satisfy the conditions of Theorem 2.1 Let {(Tx { be an i.idd sampling from (X,Yl Y). Let Q be a family of universal PDF approxi- mators for class of densities containing P xxy ofxx x ff²) (pushforward probability measure in the latent space), meaning the convergence in probability of a maximum maximum--lketthood estimate from Q to the ground-truthe distribution if N increases. Let ÂN E Q be maximum maximum--lkkelthood estimate of exist for
Fx,y offx' fx from the samples {(fx(**). fy (3n))) Let IĄN
every N. Then
MIENE (f ((x,,)))
F I(X:Y)
Note that maximum -ikelnhoo-i training of fx, fy also minimizes DKL (Px.y offx' which allows for surprisingly simple q E Q to be used, as we show in the subsequent sections.
ff'' ((..))
The described approach is as general possible. We use it as starting point for development of more elegant, cheap and practically sound MI estimator. We also do not incorporate conditions, under which the universality property of Q holds, they depend on the choice of Q: if one is interested in using normalizing flows as Q, we refer to Section 3.4.3 in [34] or to 25 for more details.
Using Gaussian base distribution
Note that the general approach requires finding the maximum- estimate q and using it to perform an MC integration acquire (x(xx(X);
In this section, we drop these requirements by restricting our estimator via choosing Q to be a family of multivariate Gaussian PDFs. This allows (a) to directly estimate the MI via a closed-form expression, (b) to employ closed-form expression for optimal q, (c) to leverage the maximum entropy principle for Gaussian distributions, thus acquiring better non-asympiotic bounds, and (d) to analyze the variance of the proposed estimate.
Theorem 4.1 (Theorem 8.6.5 in (55). Let Z be a d-dimensional absolutely continuous random vector with probability density function pz, mean and covariance matrix E. Then
I(Z) (m, E)) DKL. (pz (m, E)) = log((me) + log det E DKL. (pz [I,, E))
Remark 4.2. Note that h(2) may not be equal to h(Z²) - DKL (pz pz for arbitrary Z'.

---

Corollary 4.3. Let ($, 11) be an absolutely continuous pair of random vectors with joint and marginal probability density functions Pe.y Pg and Pr correspondingly, and mean and covariance matrix being
mg
Σgg i
Σg.1
m
mm
Σ =
2nm
Then
1 [log det Σg.e + log det En.7 log det Σ]
I(E:n)
+ DKL (pe. (m,, (m), DKL (pe @ Pn [I,V diag(Se, En,))) which implies the following in the case of marginally Gaussian & and ty:
I(g: n): 2
[log det Σg,e + log det Σnp log det Σ]
(8)
with the equality holding if and only if n) are jointly Gaussian.
Corollary 4.4. Under the assumptions of Corollary
4.3
DKL (Pe.n (N(m,E)) Remark 4.5. The upper bound from Corollary 4.4 tight, consider & ~ ~N 1), 1j (2B 1) C.
((::)) [log det Σg,g log det En log det Σ
where B- ~ Bernoulli(1/2) and is independent of ç.
From now on we denote fx (x(X) (X) and Jy-YY) as & and vy correspondingly. Note that, contrast to Theorem 3.1 and Corollary (.22 (E: is replaced by dlosed-form expression, which is not possible to achieve in general. The provided closed-form expression allows for calculating MI for jointly Gaussian (E, 17), and serves lower bound on MI in the general case & and r) being only marginally Gaussian.
4.1 General binormalization approach
In order to minimize DRL. (PE..) || [mm, E)), we train fx Jy as single normalizing flow. Instead of maximizing the log-likelihode using two separate and fixed base (latent) distributions, we maximize the log-like log-likelihood of the joint sampling {I W using the whole set of Gaussian distributions as possible base distributions.
Definition 4.6. We denote a set of d-dimensional Gaussian distributions as S (W(m, Σ) | m E Rdd, Rdxd) P
Definition 4.7. The log- likelihoodd of sampling {z IN with respect to a set of absolutely continu- probability distributions S is defined as follows:
N log
dμ dz
[s({z)))
sup (1(2xx)) = sup PES
Σ
(2x)
PES
Let us define f fx x fy (Cartesian product of flows) and c f {u o f μ 5) (set of pushforward measures). In our case, L (f( }))} can be expressed in closed-form via the change of variables formula (identically classical normalizing flows setup) and maximum- likelihood estimates for m and Σ.
Statement 4.8.
Csxolfxx x fr) (H(a)) vk))) log det @f(z.v) v) 0(x, y)
( (1f(ak) ( vr))),
where
log det af(x,y) A(x y)
0fx(a)) Ōr
det 0fr(u)
log det
+ log
Oy
n Σ fax vk),
Σ (f( ()) yi) m))
We omit d whenever it can be deduced from the context.

---

Maximization of [xx fy) (f(*), (k){) with respect to parameters of fx and fy minimizes DKL (Pg.a) (m, Σ)) [34), making possible to apply Theorem 2. and Corollary 4.3 to acquire an
MI estimate with corresponding non-asymptotic error bounds from Corollary 4.4
MINE ( vo) log det Σg.s + log det En,y - log det Σ
(9)
Note that if only marginal Gaussianization is achieved, Equation ? serves as a lower bound estimate.
As MINM involves maximum-likelihodd estimates of covariance matrices, existing results can be employed to acquire the asymptotic variance:
Lemma 4.9 (Lemma' 2 in (25).. Let fx, fy be fixed. Let (E,n) have finite covariance matrix. Then, the asymptotic variance of NNMMNE is (dd /N), with d being the dimensionality. (f, (1) is also
Gaussian, the asymptotic variance is further improved to O(d/N).
4.2 Refined approach
Although the proposed general method compelling, as it requires only the slightest modifications to the conventional normalizing flow setup to make the application of the closed-form expressions for MI possible, we have to mention several drawbacks.
Firstly, the bog-likeldhood maximum is ambiguous, as LSN is invariant under invertible affine mappings, which makes the proposed log- lggliiel maximization ill-posed problem: Remark 4.10. Let A Raxd be non-singular matrix, R, {= Rd. Then
(Sxo(A=+0)((ze)) ((x((())))
Secondly, this method requires regular (ideally, after every gradient descent step) updating of nn and Σ for the whole dataset, which is expensive. In practice, these estimates can be replaced with batchwise maximum likelihood estimates, which are used to update m and Σ via exponential moving average (EMA). This approach, however, requires tuning EMA multiplication coefficient in accordance with the learning rate to make the training fast yet stable. We also note that m and &2 can be made learnable via the gradient ascent, but the benefits of the closed-form expressions for Csy in Statement 4.8 are thus lost.
Finally, each loss function evaluation requires inversion of E, and each MI estimation requires evaluation of det Σ and determinants of two diagonal blocks of Σ. This might be resource-consuming in high-dimensional cases, as matrices may not be sparse. Numerical instabilities might also occur & happens to ill-conditioned (might happen in the case data lying on a manifold or due to high MI).
That is why we propose elegant and simple way eliminate all the mentioned problems further narrowing down Sy to subclass of Gaussian distributions with simple and bounded covariance matrices and fixed means. This approach is somewhat reminiscent the non-linear canonical correlation analysis [3 37).
Definition 4.11.
dee Wriding-N
A (W(OE)) Σe,g = Ide En, Id,, Eg,((= ST.)) = diag({p;)) 1) xd, Ps E (0;1))
This approach solves all the aforementioned problems without any loss generality, as it is shown by the following results: Corollary 4.12. (E, 14 E Suiding N, then
I(E:n) 1- Σ log(1 03)
(10)
Statement 4.13 (Canonical correlation analysis). Let (E, 11) ~ ~M(m, E), where Σ is non-singular There exist invertible affine mappings (pe, 9n such that (pee(E) μ Swiding-N. Due to Theorem 2.1 the following also holds: I(E:n) = ((

---

Statement 4.14. Let (5, ~ (0, E) E Suidiag-V, {2 k=1 C B Then
LN ((2x1) I(E:n) Lxion
where (implying Pj for > min(de, d7))
aj + B)
0j 53
1
aj 2/ + Pj
E-
aj 8g
aj $3
Bj 2 2/1 1- - Pj
de
and (g: is calculated via (10
Maximization of C V (fx (((x,,))))))))) with respect to the parameters of fx and fy yields
the following MI estimate:
Iudiaa-N-MIENF (1 ((((=,.g))]+t )x)) & - Σ log(1 - -))),
(11)
where PJ are the maximum-likeliben estimates of Pj-
4.3 Tractable error bounds
Note that Corollary 3.2 and Corollary 4.4 provide with non-asymptotic, but untractable bounds. These bounds can be estimated via various KL divergence estimators 77 20) 38/40]. However, this requires training an additional neural network, which is computationally expensive.
Conveniently, as the proposed method involves maximization the likelihood, a cheap and tractable lower bound on the KL divergence can be obtained via mntroyy-cross-enten decomposition:
p(Z) = EZ log g (Z) h(Z) - Ezzp log ()) - hmN(m, Σ))
DKL (p 4) q) = Ez~p log
(Z)
(12)
Note that E log ()) in Equation 12 is estimated by the log-likelihood of the samples in the latent space (which is inevitably evaluated each training step), and h(Z) can be upper bounded by the entropy of Gaussian distribution (see Theorem 4.1). Unfortunately, as Theorem 4. has already been employed to derive Corollary [4.3, the proposed bound is trivial (equaling to zero) our Gaussian-basse setup. However, this idea might still be useful in the general case.
4.4 Implementation details
In this section, would like emphasize the computational simplicity of the proposed amendments to the conventional normalizing flow setup.
Firstly, Statement 4.14 allows for cheap log- logelikelihood computation and sample generation, Σ, E-1/2 and Σ-1 are easily calculated from the {Øj} and are sparse (tridiagonal, dlockdiiagonalizable with 2 x blocks). Secondly, the method requires only d' = min (dg, dl,) additional parameters: the estimates for {py}. As Pj E [0: 1), an appropriate parametrization should be chosen to allow for stable learning of {Pt} via the gradient ascend. We propose using the logarithm of cross-component 5 wi A log (E)im))) = log [- log(1 0z)]. In this parametrization Wi E R and
Imiding W (( ynh ) = Eo Pj vI h -2 enns ) (0:1)
(13)
Although Pj = 0 is not achievable in the proposed parametrization, it can be made arbitrarily close to with w, + oc.
*One can also consider the sofiplus parametrization, which allows avoid the double exponentiation in We, however, did not encounter any issues with the plain logarithm parametrization.
7

---

4.5 Extension to non-Gaussian base distributions and non-bijective flows
The proposed method can be easily generalized account for any base distribution with closed-fomm expression for MI, or even combination of such distributions. For example, smoothed uniform distribution can be considered, with the learnable parameter being the smoothing constant E. see Ap- pendix B.2 Equation (14). However, due to Remark 4.2 neither Corollary 3.2 nor Corollary 4.4 can
be used to bound the estimation error this case.
Also note that, as Theorem 2.1 does not require g to be bijective, our method is naturally extended injective normalizing flows 4] 42]. Moreover, according [19]. such approach may actually facilitate the estimation of MI.
Experiments
To evaluate estimator, utilize synthetic datasets with known mutual information. In 14 and
[9]. extensive frameworks for evaluation of MI estimators have been proposed. In our work, we borrow complex high-dimensional tests from [19 and all non-Gaussian base distributions with known MI from [ (see Appendix B for more details), The formal description of the dataset generation and estimator evaluation is provided in Algorithm T Essentially similar setups are widely used to test
the MI estimators 7 13
20 31
Algorithm 1 MI estimator evaluation
1: Generate two datasets of samples from random vectors and with known ground truth mutual information (see Corollary 4.3 Corollary 4.12 and Appendix B for examples): (ake by b -
2: Choose functions Ie and gn satisfying conditions of Theorem 2.1 so [(5: = / (ge 9n (n))
3: Estimate (ge (E): (x)) via ((gg (ax), (be (be)) )= compare the result with the ground truth.
For the first set of experiments, we map diw-dimeniional correlated Gaussian distribution to distribution of nigh-diimnssooal images of geometric shapes (see Figure 2) We compare our method with the Mutual Information Neural Estimator (MINE) 77). Waanee---aiirrhh--Jordaaaa (NWJ) 7 39
and Nishiyama's [0 estimators, nearest neighbor Kraskov-Stoegbauue Saroeggbaurr-rrashheereer
31 and
nearest neighbors weighted Kozachenko-Leonelle (WKL.) estimator [27 44 the latter is fed with
the data compressed via a convolutional autoencoder (CNN AE) in accordance to the pipeline
from [19].
(a) 2D Gaussians
(b) Rectangles
Figure 2: Examples of synthetic images used in the tests. Note that images are high-dimensional, but admit latent structure, which is similar to real datasets.
For the second set of experiments, incompressible, sigh-dimensional non-Gan Non-Gaussian Gaussian-based distribu- tions are considered. These experiments are conducted to evaluate the robustness of our estimator to the distributions, which can not be precisely Gaussianized via Cartesian product of two flows. In this section, we compare our method to the ground truth value only. We also provide a comparison with MINE and MauSsiaa (55 in Appendix B For more elaborate benchmarking of other estimators on these distributions, we refer the reader to [14].
For the tests with synthetic images, loga (image size) splits, 2 blocks between splits and 16 hidden channels in each block, appended with learnable orthogonal linear layer and small 4-layer Real NVP flow [46). For the other tests, use 6-layer Real NVP architecture. For further details (including the architecture of MINE critic network and CNN autoencoder), refer the reader Appendix E
GLOW 45 normalizing flow architecture with
The results of the experiments performed with the high-dimensional synthetic images and non- Gaussian-basee based distribution are provided in Figure 3 and Figure correspondingly.

---

10
10
ground mruth
ground
AE+WKL 5-NN
AE+WKL 5-NN
KSG
KSG
MINE
MINE
NWJ
NWJ
Nishiyama MIENF (OWrS)
Nishiyama
MIENF (DUrS)
4
4.
2
2
10
2
4
10
(a) 16 images (Gaussians)
(b) 32 images (Gaussians)
10
10
ground
AE+WKL 5-NN
KSG
MINE
Nishiyama
MIENF (ours)
4
ground truth WKL AE+WKL KSG
AE+
5-NN
MINE NWJ
Nishiyama MIENF (ours)
1
10
10
16 16 images (rectangles)
(d) 32 32 images (rectangles)
Figure 3: Comparison of the selected estimators. Along axes I(X; Y), along y axes is iX; Y). We plot 99, 99.9% asymptotic CIs acquired either from the MC integration standard deviation (WKL, KSG) or from the epochwise averaging (other methods, 200 last epochs). 10 10 samples were used.
10.5
10.5
10
9.5
9.5
((,Y)
9
i(XY)
9
Ground truth
Ground truth Uniform Smoothed uniform
8.5
Uniform
Smoothed uniform
8
Student (dof
Student (dof
7.5
aresinh( Student) (dof = 1)
7.5
arcsinh(Student
aresinht Student) (dof 1)
Aresinh((Suuden
aresinh Studeen) (dof 2)
2
16
32
2
16
32
dimensionality
dimensionality
N-MIENF
(b) tridiag-N-MIENN
Figure 4: Tests with incompressible multidimensional data. "Uniform" denotes the uniformly distributed samples acquired from the correlated Gaussians via the Gaussian CDF. "Smoothed uniform" and "Student" denote the non non- Gaussian-basee distributions described in Appendix B "arcsinh(Student" denotes the aresinh function applied to the "Student" example (this done to avoid numerical instabilities in the case of long-tailed distributions). We run each test 5 times and plot 99.9% asymptotic Gaussian Cls. 10 samples were used. Note that --MIENF and tridiag-N -MIEN yield almost the same results with similar bias.

---

We attribute the good performance of AE+WKL to the fact that the proposed synthetic datasets are easily and almost losslessly compressed via a CNN AE. We run additional experiments with much simpler, but incompressible data to show that the estimation error of WKL. rapidly increases with the dimensionality. The results are provided in Table T contrast, our method yields reasonable estimates in the same or similar cases presented in Figure
Table I: Evaluation of 5-NN weighted Kozachenko- estimator multidimensional uni- formly distributed data. For each dimension dx dy. 11 estimates of MI are acquired with the ground truth ranging from 0 to 10 with fixed step. The RMSE of the estimated MI relative the ground-truth MI is calculated for each set of estimates.
dx
2
4
8
16
32
64
RMSE
2.2
1.0
127.9
227.5
522.4
336.2
Overall, the proposed estimator performs well during all the experiments, including the incompressible high-dimensional data, large MI values and non-Gaussian-ball tests. Appendix D we also apply our method to acquire disentengled representations of real data. Additionally, we give brief commentary on the sample complexity of the proposed method and other NN-based estimators in Appendix
6 Discussion
analysis of deep neural networks is novel and developing approach. As relies well-established theory of information, it potentially can provide fundamental, robust and intuitive results [47 48). Currently, this analysis is complicated due to main coormation-theorede qualities - differential entropy and mutual information being hard measure in the case of high-dimensional data.
We have shown that it is possible to modify the conventional normalizing flow setup harness all the benefits simple and robust closed-forn expressions for mutual information. Non-asymptotic error bounds for both variants of our method are derived, asymptotic variance and consistency analysis carried out. We provide useful theoretical and practical insights on using the proposed method effectively. We have demonstrated the effectiveness of our estimator in various settings, including compressible and incompressible nigh-dimensional data, high values of mutual information and the data not acquired from the Gaussian distribution via invertible mappings.
Finally, it worth noting that despite normalizing flows and Gaussian base distributions being used throughout our work, the proposed method can be extended to any type of base distribution with closed-form expression for mutual information and to any injective generative model. For example, a subclass of diffusion models can be considered [49] [50]. Injective normalizing flows [41 42) are also compatible with the proposed pipeline. Gaussian mixtures can also be used as base distributions due relatively cheap MI calculation and the universality property (22
The main limitation of the general method the ambiguity of Q (the family of PDF estimators used to estimate MI in the latent space), which can be either rich (yielding consistent, but possibly expensive estimator), or poor (leading to the inconsistency of the estimate). However, in (32) it argued that mixture models achieve rather good tradeoff between the quality and the cost of a PMI approximation.
Limitations
The major limitation of A-MIENF is that its consistency is proven only for a certain class of distributions: the probability distribution should be equivalent to a Gaussian via a Cartesian product of diffeomorphismss However, mathematical simplicity, rigorous bounds, low variance and relative practical success of the estimator suggest that the proposed method achieves a decent tradeoff.
10

---

References
[1] Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. 2015 IEEE Information Theory Workshop (ITW), pages 2015.
[2] Aolin Xu and Maxim Raginsky. Information-theoreti analysis of generalization capability of learning algorithms. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Sys- lems, volume 30. Curran Associates, Inc., 2017. URL WtRL/httpss:/ htttpps::/prcccediiigggg www..ss.....ppp... cc/ pdf
[3] Ziv Goldfeld, Ewout van den Berg, Kristjan H. Greenewald, Igor V. Melnyk, Nam H. Nguyen, Brian Kingshury, and Yury Polyanskiy. Estimating information flow in deep neural networks.
In ICML, 2019.
[4] Thomas Steinke and Lydia Zakynthinou. Reasoning About Generalization via Conditional Mutual Information. In Jacob Abernethy and Shivani Agarwal, editors, Proceedings of Thirty Third Conference on Learning Theory, volume 125 of Proceedings of Machine Learning Research, pages 3437-3452. PMLR, 09-12 Jul 2020. URL http::s://proceeinggi mml. mlr
press/v125 www.seeeeeaaaaa... html
[5] Rana Ali Amjad, Kairen Liu, and Bernhard C. Geiger. Understanding neural networks and individual neuron importance via information-ordered cumulative ablation. IEEE Transactions on Neural Networks and Learning Systems, 33(12):7842-7852, 2022. doi: 10.1109/ 2021.3088685
[6] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 2172-2180, 2016. URL http:::/t//// p/ttpe::s///rooednnggs neurips. wttpp.cccppaaeee wttprrssscppppp////00// ppaper/016//aaa/ (e5500010eeeeeeeeeeeaaaaaaaaaaaa -tw.e99eebbbbbbbbbbe Awt.bbatttttaatttt...oa.a
[7] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 531-540. PMLR, htw.:/gsmmm......aaaeeppme v/0///00//b//eee/ 1//0///elaaaiiiii 18a. html
07 2018. hRRphhtts:////oeeeeeee
[8] Lynton Ardizzone, Radek Mackowiak, Carsten Rother, and Ullrich Köthe. Training nor- malizing flows with the information bottleneck for competitive generative classification. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 7828-7888 7840. Curran Associates, Inc., 2020. URL httttttt/t//pteeeeet wwwpeeeddddnnnnnnnnnnueeeeeeeeee www. wttpcc/pppp.o hacc/aper_ffeee filllfillllppepll patles//paprr/22///iieee ort.oaafad..33aS1111222222213062
pdf
[9] Thomas Berrett and Richard Samworth. Nonparametric independence testing via mutual information. Biometrika, 106, 11 2017. doi: 10. 00.1093//iomeetass024..
[10] Rajat Sen, Ananda Theertha Suresh. Karthikeyan Shammugam, Alexandros G Dimakis, and Sanjay Shakkottai. Model-powered conditional independence test. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, ed- itors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, neurips. cc/paper http:pppprrfffeee fatear_ffelepppppeeeee wtt.ss//appere2/////iii/ pdf
Inc., 2017. URL
[11] Bao Duong and Thin Nguyen. Normalizing flows for conditional independence testing. Knowl- edge and Information Systems, 66, 08 2023. doi: 10. 10.1007/10011--03-00111
[12] Z. Goldfeld, K. Greenewald, Niles- and Y. Polyanskiy. Convergence of smoothed empirical measures with applications to entropy estimation. IEEE Transactions on Information Theory, 66(7):436-443) 2020. doi: 10. 10.10990222000099754880
[13] David McAllester and Karl Stratos. Formal limitations on the measurement of mutual in- formation. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty
11

---

Third International Conference on Artificial Intelligence and Statistics, volume 108 of Pro- ceedings of Machine Learning Research, pages 875-884. PMLR, 08 2020. URL https
mlr //
html
[14] Pawel Czyz, Frederic Grabowski, Julia Vogt, Niko Beerenwinkel, and Alexander Marx. Be- yond normal: On the evaluation of mutual information estimators. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL
id-25vRtG56YH
[15] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. 2019.
[16] Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual infor- mation estimators. In International Conference on Learning Representations, 2020. URL
https://opearevier:
[17] Benjamin Rhodes, Kai Xu, and Michael U. Gutmann. Telescoping density-ratio estimation. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 4905-4916. Curran Associates, Inc., 2020. URL httpll.ttt:////tt htttps::/proceeedinggg www..gs......p.......m. cct wwwpccppppppp..eeeeeee f files/ wttp://ppppe////////21/10 http:///2..00faadd22222222259999
ppd
[18] Ziqiao Ao and Jinglai Li. Entropy estimation via normalizing flow. Proceedings of the AAAI Conference on Artificial Intelligence, 36(9):9990 9998, Jun. 2022. doi: 10. 10.1660/iaiiv... v36i9.
21237. htthttss://o/g
aaai org/ index, php/
221237
[19] Ivan Butakov, Alexander Tolmachev, Sofia Malanchuk, Anna Neopryatnaya, Alexey Frolov, and Kirill Andreev. Information bottleneck analysis of deep neural networks via lossy compression.
In The Twelfth International Conference on Learning Representations, 2024. URL https http:/vvwwwieeeeeoomnnnnieieeeee
[20] Giulio Franzese, Mustapha BOUNOUA, and Pietro Michiardi. MINDE: Mutual information
neural diffusion estimation. In The Twelfth International Conference on Learning Representa- lions, 2024. URI htt.ttt::///pereeevttes
[21] Esteban G. Tabak and Eric Vanden-Eijnden. Density estimation by dual ascent of the log- likelihood. Communicutions in Mathematical Sciences, 8(1):217 - -233, 2010.
[22] E. G. Tabak and Cristina V. Turner. A family of nonparametric density estimation algorithms. Communications on Pure and Applied Mathematics, 66(2): 2013. doi: https://doi.ow .org/ 10.1002/cpa.14433 URL htttpp:///onineellliee www.orrrrrrrilllc.ooooo ctw.//lcccom/oioiooi/ aas/a//s///// 10. 1002/ cpa
21422
[23] Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: non-linear independent components estimation. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Workshop Track Proceedings, 2015. URL/http:/ /ttptp:://axvv... arxivv org/ abs//410...1..//////////////11// 8516
[24] Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Francis R. Bach and David M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, ICMI. 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages 1530-1538. JML.Rorg. 2015. Utl.htttp://proceeiiin
mlr. wt.press/vvv. v37 2557/rezende15 html
[25] Bao Duong and Thin Nguyen. Diffcomorphic information neural estimation. Proceedings of the AAAI Conference on Artificial Intelligence, (766)) Jun. 2023. doi: 10. v3716.25908. UtL/httss:/ //ojs aaai www.orggiiiiio wwwgx/iidxx....m php/
25908 [26) M. Spivak. Calculus On Manifolds: A Modern Approach To Classical Theorems Of Advanced Calculus. Avalon Publishing, 1965. ISBN 9780805390216.
[27] Thomas B. Berrett, Richard J. Samworth, and Ming Yuan. Efficient multivariate entropy estimation via k-nearest neighbour distances. Ann. Statist., 47(1):288-118, 02 2019, doi: 22214/1-AAOS1688 URL https: h/do::://///ooo/// org/ 10214/18-A016660
[28] I. D. Butakov, S. V. Malanchuk, A. M. Neopryatnaya, A. D. Tolmachev, K. V. Andreev, S. A. Kruglik, E. A. Marshakov, and A. A. Frolov. High-dimensional dataset entropy estimation via lossy compression. Journal of Communications Technology and Electronics, 66(6))764-768 7 2021. ISSN 1555-6557 doi: 10.1134/S 10.113445106422222106006 UtRlhhttps://do/ org///0oooogg/0 1134/ 0134//11004222921060000
12

---

[29] Kristjan H. Greenewald. Brian Kingsbury, and Yuancheng Yu. High Sigh-dimensional smoothed entropy estimation via dimensionality reduction. In IEEE International Symposium on Informa- fion Theory, ISTT 2023, Taipei, Taiwan, June 25-30, 2023, pages 2613-2618, IEEE, 2023. doi:
10. 11009///TS4413..222.... 10206644. http::/tts/// 10. 1110/11IT10/110 2023 [0206641]
[30] Linara Adilova, Bernhard C. Geiger, and Asja Fischer. Information plane analysis for dropout neural networks, 2023.
[31]
Alexander Kraskov, Harald Stöghauer, and Peter Grassherger. Estimating mutual information. Phys. Rev. E, 69:066138, Jun 2004, doi: 10.1 1011///hysseee......666 URI.tttss: ://inkk
pps.orggdoi/ll/ 1e13///yySeeiii. 69. 066138
[32] Pawel Czyž, Frederic Grabowski, Julia E. Vogt, Niko Beerenwinkel, and Alexander Marx. The mixtures and the neural critics: On the pointwise mutual information profiles of fine distributions, 2023.
[33] Y. Polyanskiy and Y. Wu. Information Theory: From Coding to Learning. Cambridge Uni- versity Press, 2024. ISBN 9781108832908. URL CtLLhtts::/// /tttps://books..oo google.ru/books/// CуSоблEΛCАAJ
[34] I. Kobyzev, S. D. Prince, and M. A. Brubaker. Normalizing flows: An introduction and review of current methods. IEEE Transactions on Pattern Analysis & Machine Intelligence, 43(11): 1964-3979 nov 2021. ISSN 1939-3539. doi: 10.1 109/TPAMM.2000 20929.4...
[35] Thomas M. Cover and Joy A. Thomas. Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing). Wiley-Interscience, USA, 2006.
[36] H. O. Lancaster. The structure of bivariate distributions. The Annals of Mathematical Statistics, 29(3):719-736, 1958. ISSN 00034851, URLpttp://www/ http://www..ttrrr/oo//s///o org/ sttale//2337222
[37] E. J. Hannan. The general theory of canonical correlation and its relation to functional analysis. Journal of the Australian Mathematical Society, 2(2):229- 242 1961. doi:
10.1017/1144788700067777
[38] M. D. Donsker and R.S. Varadhan. Asymptotic evaluation of certain markov process expectations for large time. iv. Communications on Pure and Applied Mathematics, 36(2): 183-212, March 1983, ISSN 0010-3640. doi: 10. 10000/cp...1100302044
[39] XuanLong Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence func- tionals and the likelihood ratio by penalized convex risk minimization. In J. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems, vol- ume 20, Curran Associates, Inc., 2007. wttpsttss: p/t.oeee//// hew.reeeeediiineeeeeeeeeeeeeeeee _ Papero pdf [40] Tomohiro Nishiyama. A new lower bound for kullback-leibler divergence based on hammersley- chapman-robbins bound, 2019.
[41] Yunfei Teng and Anna Choromanska. Invertible autoencoder for domain adaptation. Com- putation, 7(2), 2019. ISSN 2079-3197. doi: (0.3390/comuutatioo00002222 URL https
/ //www. www..
[42] Johann Brehmer and Kyle Cranmer. Flows for simultaneous manifold learning and density estimation. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 442-453, Curran Associates, Inc., 2020. URL http:/tsss//ppoooddnnss..ee neurips..........er........... http:pppprffffe/pppppp .pff
[43] Ben Poole, Sherjil Ozair. Aaron Van Den Oord, Alex Alemi, and George Tucker. On vari- ational bounds of mutual information. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5171-5180. PMLR, 06 2019. URL. https://proceedii
mlr wwwpppressse.//ep wttpsss/v9//oooooo /t7p/popppppll.aa/m//o html
[44] I. F. Kozachenko and N. N. Leonenko. Sample estimate of the entropy of a random vector. Problems Inform. Transmission, 23:95-101, 1987.
[45] Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates,
13

---

Inc.. 2018. httpa///LLt///pppoooonnngo.ooo// http://ww.......a.ooooooooa/aaaa db9466226003111111337222220ppp
pdf
[46] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In International Conference on Learning Representations, 2017. URL htt.ltttp:://peeeeiiiii
[47] Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information, 2017.
[48] Haiyun He, Christina Yu, and Ziv Goldfeld. Information-theeortiicc generalization bounds for deep neural networks. In NeurlPS 2023 workshop: Information--Teerrlle Principles in Cognitive Systems, 2023. URL
[49] Qinsheng Zhang and Yongxin Chen. Diffusion normalizing flow. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 16280-16291, Curran Associates, Inc., 2021. URL BRRphttts///pooceediii mewrr cc/ cc/paper.a... f 6TG119955400444221118880112d22p2 pdf
[50] Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. On density estimation with diffusion models. A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL httpt::://pppereiiieeeeeee// net/
forum?id=2lonm -=
[51] Tony Cai, Tengyuan Liang, and Harrison H. Zhou Law log determinant sample covariance matrix and optimal estimation of differential entropy for nigh-dimensional gaussian distributions. Journal of Multivariate Analysis, 137:161-172, 2015. ISSN 0047-259X. doi: https:
//doi.or//10.1016// jmwa. 2015 .2.0003. URL https:ts:://ww reet com/ stt//science.
arricleepp1/S004755550000
[52] R. Arellano-Valle Javier Contreras-Reyes, and Mare Genton. Shannon entropy and mutual information for multivariate skew-elliptical distributions. Scandinavian Journal of Statistics,
40:42-62, 03 2013. doi: 10.111 1t.11//j.11111 1467996999469
[53] Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecm lec.annn.ecc.c..o wow.exdb/mmisst, 2, 2010.
http://saan.euuu..o//xxxbbbissss
[54] Vincent Stimper, David Liu, Andrew Campbell, Vincent Berenz, Lukas Ryll, Bernhard Schölkopf, and José Miguel Hernánde7-LLoatoo normflows: A pytorch package for normalizing
flows. Journal of Open Source Software, 8(86):5361, 2023. doi: 10.21 2o5/josss003611 URL.
https://dol http:::/ddoooo////ee/eo orgg1orrg1100 211011115555 21105/joss. 05361
[55] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
14

---

A Complete proofs
Theorem 2.1. Let E: Ω - Rn be an absolutely continuous random vector, and let g: BR" + Ik" be an injective piecewise-smune mapping with Jacobian J, satisfying n 2 n' and det (J² + almost everywhere. Let PDFs pg and Peln exist. Then
PMIg.1(5.)) 7) PMMggeee(ll()..)) n).
= (({:))=[((((Emm))
(5)
Proof of Theorem 2.7 For any function g, let us denote v det (J" (area transformation coefficient) by a(x) where it exists.
Foremost, let note that in both cases, Pę(x 17) and Pate) (a'
1))
Pę(x m)/a(<) exist.
Hereinafter, we integrate over supp a<(x) o 0} instead of supp {; as of 0 almost everywhere by the assumption, the values of the integrals are not altered.
According to the definition of the differential entropy.
pe(x) a(x:))
Pe(x)
h(((E))
log
a(x) do
cr(x))
- / pe (a:) log (x)( do + / pe(x) log (a(x)) de
h(() + E Elog
Mg(() (1) = En (-) Pe(v m1
log p5(x (((r|()) a(x) dx
(1)
a(x)
a(x)
=En (-f (e(x n) log (pe(# n)) dx + / pe n) log (a(x)) dz)
h(E 11) + Bloga(()
Finally, by the MI definition,
T(g(E): v) h(((E)) h(((E) 1)) M(E) A(E 1) T(S:n).
Dropping the expectations/integrals in the equations above yields the proof of the PMI invariance.
Theorem 3.1. Let (E, 7) be absolutely continuous with PDF Pt. Let qE be a PDF defined on the same space Pg.n- Let pe, Prr qg and 9n, be the corresponding marginal PDFs. Then
(E, n)
I(E:n) EEpe,, log
DRr. (Pe.) (l DRI. (pe @ Py || ge & 9r)
(6)
(g)
1,(5:n)
Proof of Theorem 3.1 In the following text, all the expectations are in terms of Pg...
PE.n(E, 7)
(e. m)
(e,n(E, 1)
ge(E),
I(E:n) E Elog
IE log
(pe(E)
as (E)
(S. 1))
Pe (e)
(E, (E, m)) (5, ())
(n)
=1((:)) E log
log
+ E
Pe
[P,(TT)
(,((n)) DKL (pe.7 (g.n)
DRL (p& Pr g1 qn)
Corollary 3.2. Under the assumptions of Theorem 3.1 (T(E:)) T,(E: m)) & DKL (pe. (..))
Proof of Corollary As DRT. (pg @ Py |1 q3 qn) 0.
I(E:n) - ()(5.)) + DKL (pe.n (
15

---

As DKL (PE.n (g.g) 2 DKL (p: q3) and DKLL (Pe.n gg,))) 2 DKL (py 9n)) (monotonicity prop-
erty, see Theorem 2.16 in (3)),
I(E:n) [y(S:)) ) DKL (Pe,n ll [l.)))))))))) DKL (Pe.n qs.n) = Ig(E:n) DEL (Pe.n qg.n)
Corollary 3.3 (ÉMENE is consistent). Let X, Y. fx and fy satisfy the conditions of Theorem 2.1
Let {ark yk) k=1 be an i.i.d sampling from (X,Y). Let Q a family of universal PDF approxi- mators for class of densities containing xxy ofx ) x f5²') (pushforward probability measure in the latent space), meaning the convergence in probability of muaximum-llkkelhhod estimate from Q to the ground-truth distribution if N increases. Let QN E Q be maximum-likeliee
estimate of 1N Let LaN (fx(X)); (Y exist for
Pxy ofxx fyy) from the samples {(fx(xx), fy (2h)) every N. Then
MMINNE (TTK, (1)))))
En T(X:V)
N-+00
Proof of Corollary 3.3 Following the assumptions on ÂN, DKL (Px, ofxx' x (I
N-000
O (universality property).
Due to Corollary 3.2. this ensures LaN (X):
fx) I(X;Y) (the latter equality is due Theorem 2.1 (fx(() fr
Finally,
(AIENF(X;Y))
(Y)) as MC estimate.
N-000
Theorem 4.1 (Theorem 8.6.5 in 355). Let be a d-dimensional absolutely continuous random vector with probubility density function pz, mean and covariance matrix E. Then
h(Z) h (Mm, 2)) - DRL (pz 1| [m, 2)) = log (2g() + log det V - -DL (pz (m, ())
As l2 - -m) probability density function of (0 Σ) Φz.
Proof of Theurem 4.7
h()), let us consider centered random vector Z. We denote
Pz(z) de -h(2) - / pz (z) (og dz dx.(2)
DKL (pz [N(O = pz(2) log
pdd
Note that
| pzl log (x) dz = const + Ez -- const+ E =f I-I- = / Φx((2) log (xs(2) dz,
3d
2
from which we arrive at the final result:
DKL (pz [1, 2))
2))
Corollary 4.3. Let (T, be an absolutely continuous pair of random vectors with joint and marginal probability density functions Pe. Pş and Pr, correspondingly, and mean and covariance matrix being
m mm
Σ Σg.e
Σg, my
Σ
En
Then
1
I(E:T) [log det Ee,e + log det En,0 - log det 2 +
+ DRI. (pe.n [m, Σ)) DIAL. (pE @ Pr, (I, diag( (Eg,e, Em,r))
which implies the following in the case of marginally Gaussian & and ry:
I(E:n) n) 2 log det Σe.e + log det Σn,, log det Σ]
(8)
with the equality holding if and only (E, are jointly Gaussian.
16

---

@f(rv) Blck--diggonl, so Proof of Corollary 4.3 By applying Theorem 4.1 to Equation 2 sion:
we derive the following expres
1
I(E:n)
[log det Σg.e + log det En,. log det Σ]
2
DKL (Pe.n (m, Σ)) DRL (PE [m, Ex,e)) DEL (Py [| Eny,))
Note that
DKL (pe (m, Me,e)) + DEL (Pn (m,, Ena)) DKL (pe @ Pv [I,, diag(Zee 2nn,)))
which results in
1 Dlog det Dg,e + log del En.9 - log det Σ] +
I(E:n) 2
DEL (Pe. [1, Σ)) DEL (pe @ Py (m, diag(E,,s Ennn)
Corollary 4.4. Under the assumptions of Corollary 4.3
I(E:n) 2
[log det Σg.s + log det Σn.. log det Σ
S DKL (Pe.n [W[mm E))
Proof of Corollary 4.4
The same as for Corollary 3.2
Statement 4.8.
Lswolfxxtr) (f(ak) vx))) log det (f(x)))
(( (ff((x vb))),
0(x, y)
where
0f v) O(x, y)
afx(2) dx
det ifr(v)
log det
= log det
+ log
m
fax. Uke)
N Σ (f Ux) m)
N
Proof of Statement 4.8
Due to the change of variables formula,
Esvolfxxfv) (1( 91) log det 0s(x, v) 0(x, v)
As f fx x fr, the Jacobian matrix ((..)
log det 0f v) 0(z y)
log det 0fx (x) dx + log det 0fy
Oy
Finally, we use the maximum-likelihood estimates m and Σ to drop the supremum in Cx(ff(x vb)));
i [(. yk),
E N
Σ (s( (m (6) m)
(U((,,
Lemma 4.9 (Lemma 2 in (25).. Let fx, fy be fixed. Let (E,r)) have finite covariance matrix. Then, the asymptotic variance of i MENT (dd (d2/N,, with being the dimensionality. If 11) is also Gaussian, the asymptotic variance is further improved to OddNN).
17

---

Proof of Lemma 4.9 The variance of IA--MIEN is upper bounded by the sum of the variances of the log-det terms. If (E. n) is Gaussian, the log-det terms are asymptotically normal with the asymptotic variance being 2d/N (Corollary 1 in (51). If (ç, 7)) is not Gaussian, the central limit theorem can be applied to each element of the covariance matrix, which combination with the delta method yields
the final result.
L
Remark 4.10. Let E Rddd be non-singular matrix, R, {=k] S IR.. Then
(S)e(A=+o)((xx)) Lsy
Proof of Remark 4.10
(syo(Az+6)(Zz))) log det A| ((Azx+b)) = log det AI log det Al log det A- () = ((=k))
Corollary 4.12. (E, n) μ SuidiagN-- then
I(E: 1720--222 Σ
log(1 p3)
(10)
Proof of Corollary 4.12 Under the proposed assumptions, log det Σg.s log det Σn, = 0, so [(E: r) - log det E. The matrix E is block diggoniigoaalizzabe via the following permutation:
(E....., S.....,Ca,, Ede: m.......,.... (d.,) + (E1, T1, E2, .)..
with the blocks being
?
Eg,:;
Pj
The determinant of block-diagonal matrix is product of the block determinants, so ((: 1
2 E, log(1 - -3))-
Statement 4.13 (Canonical correlation analysis). Let (E, n) ~ ~Mm, E), where is non-singular. There exist invertible affine mappings (pe, in such that (pe(E), (n)))) μ Striding-N. Due to Theorem 2.1 the following also holds: I(E:1) = ((pe(()) (1)).
Proof of Statement 4.13 Let us consider centered and y7, as shifting is an invertible affine mapping. Note that Σg,e and En., are positive definite and symmetric, so the following symmetric matrix square
roots are defined: A -..5 and we get
1/2 -112
B En By applying these invertible linear transformations to
Σ- 1/2 Ee.s (- (E=1/11
1/2gT
Σ- x-1/2 D Eg, /2 Enpn 2 En Emy /2 C
I
1
Emjri En (Σg /2/T
where C = Eg,e D-1112E50
Eg.g (Ent)//y
Then, the singular value decomposition is performed: = URVT, where U and V are orthogonal, R = diag( (. Finally, we apply U to Ag and VT to By:
UTU (UTCV)
UTCV VTV
R
cov(U" Ag, V BT)
RT 1
Note that U A and VT B are invertible.
Statement 4.14. Let (E,n) NN(, SuidieeNN, {=x) C +d Then
(N(o,))((*))) I(5:n) (12-12
18

---

where (implying Pj for > min dn))
aj + 18
02 33
0j 2 + Pj
1/2
aj Dg
0j + 33
Bg 2 2 3-1-pp Pj
de
de
and ((: is calculated via (10
Proof of Statement 4.14 Note that E is positive definite and symmetric, so the following symmetric matrix square root defined: Σ-1/2, This matrix is normalization matrix: cov (E-I/2((, /-1/2 (X-1/2)
According to the change of variable formula,
(((o.))((*+)) log det 1/1/2 ((E-
As Eggs Tag and En," Ta, from the equation 8 we derive
[(:in) = 2
log det log det 1/2
Finally, it sufficient to validate the proposed closed-forn expression for / in the case of 2 x matrices, E is block-diagonalizua (with 2 2 blocks) via the following permutation:
: (E1....,Edy "1.......... na,,) + (E1. Mis E2. 72,...);
Note that
a + 8 a 3
3
2 +
82
0²
1
-
a²
2 + B2
p
P
p
1 p
1 0
=
P
B Non-Gaussian-based tests
As our estimator is based on Gaussianization, it seems natural that we observe good performance in the experiments with synthetic data acquired from the correlated Gaussian vectors via invertible transformations. Possible bias towards such data can not be discriminated via the independency and selffconsistenny tests, and hard discriminate via the proce-proessing test proposed [4 20 for the following reasons:
1. Independency test requires f(x;)) 0 for independent X and Y. In such case, as cov(fx fy (Y)) 0 for any measurable fx and fy, IMINNI ; Y) in any mean- ingful scenario (no overfitting, no ill ill-posed transformations). regardless of the marginal distributions of X and Y.
2. Self-consistenis test requires f(x; = [g((X): Y) for X, Y and g satisfying Theorem 2.1 In our setup, this test only measures the ability of normalizing flows to invert and provides no information about the quality of Í (Xx; Y) and [g(X);;)) Y).
Moreover, as we leverage Algorithm [ with the Gaussian base distribution for the dataset generation, we somewhat test our estimator for the self-consistency.
3. Data-prccessing test leverages the data processing inequality 35) via requiring f(x; Y) > i [(g{ (x) Y' for any X, Y and measurable g. Theoretically, this test may highlight the bias of our estimator towards binormalizable data. However, this requires constructing X, Y and g. so and are not binormalizable, g(X) and Y are and f(X;V) i(g(X); Y), which seems challenging to achieve.
19

---

That is why we use two additional, non Son-Gassiin-- Baussian-based families of distributions with known closed- form expressions for MI and easy sampling procedures: multivariate Student distribution [52] and smoothed uniform distribution [14]
In the following subsections, we provide additional information about the distributions, closed-form expressions for MI and sampling procedures.
B.1 Multivariate Student distribution
Consider (n (m+m))-du (r-tmm)-dimmesiion (X; V) (O, Σ), where E is selected achieve I(X;8) 0.
Firstly, a correction term is calculated in accordance to the following formula:
c(k, n, m) f(k) + S(k + + m.) [(k + n.) - [(k + m), f(x) log r
E 2 2
where & the number of degrees of freedom, & is the digamma function. Secondly, X X/Vk/U K/U
and = -Y/Vk/U are defined, where X2- The resulting vectors are distributed according the multivariate Student distribution with & degrees of freedom. According (52), r(X; Y)
x + c(k, n, m). During the generation, x is set to ((; c(k: n, m) to achieve the desired value of Y.
Note that I(X:Y) even in the case of independent X and Y, some information between X and Y is shared via the magnitude.
B.2 Smoothed uniform distribution
Lemma B.1. Consider independent X ~ U0 1], Z ~ U[-c;6] and Y X + Z. Then
8 s log(())s)
<< 1/2
I(X;))
(14)
(4=)-1
E 2 1/2
Proof. Probability density function of Y (two cases):
0,
y < -
M+C
-E SY<E
(e 1/2)) (v) = (px * (z)
1,
6 S <y<1-6
1+5-1 M
1 <1+
2s
0,
V < -e -cVy21+e
yhc
- <1-8
(e 1/2):
(v) = (px * +z2)())
1 1-E$y<E
28
Tte-y
ešy<l+e
25
Differential entropy of a uniformly distributed random variable: haaT[a; b)) = log(b - a)
Conditional differential entropy of Y with respect to X:
4(Y X) Ex-x h(Y X x) = B 12Z + = =x)
As X and Z are independent,
E h(Z + x x 2) = E 1(2 +x) = log(2e) dx = log(2s)
(15)
Differential entropy of Y:
2
/
[e, pr
E < 1/2
h (YY
(16)
(4E) + log(22),
E 2 1/2
The final result is acquired via substituting 15) and 16 into
20

---

Equation
14 can be inverted:
(4. ((:I
I(XY) < </2 I(XY) 1/2"
(17)
(-W | ex)(-1(x;))))
where W the product logarithm function.
B.3 Additional experiments
Recall that in Section 5 we do not evaluate other estimators the tests discussed in this part of the Appendix. address this, we provide additional results MINE and DINE-Gaussian in Figure We chose MINE it is the best performing critic-based method judging by the results from Figure 3 and is widely considered as decent modern baseline. We chose DINE-Gaussian as (a) this method also employs normalizing flows, and (b) were not able acquire reliable estimates via this method during the tests presented in Figure
The results are presented in Figure 5
14
Ground truth
14
Ground truth
Uniform
Uniform
Smoothed uniform
Smoothed uniform
Student (dof 4)
Student (dof 4)
arcsinh(Student) (dof l)
Cresinh(Studentl (dof I) arcsinht Student) (dof 2)
tresinh(Studen) (dof 2)
i(XY)
i(X.Y)
4
4
2
2
0
0
2
16
32
2
16
32
dimensionality
dimensionality
(a) HINE-Gaussin
(b) MINE
10
10
i(x.Y) Y)
6
i(XY)
6
Ground truth Uniform
Ground truth
Uniform
Smoothed uniform
Smoothed uniform
Student (dof 4)
Student (dof 4)
2
ancsinh/Studen (dof 1)
2
aresinh Student) (dof 1)
Cresinh(Student (dof 2)
(nesinh(Student) (dof
2
8
16
32
2
8
16
32
dimensionality
dimensionality
(c) N-MEENF
(d) Pridiag-V-MMENN
Figure 5: Additional tests with incompressible multidimensional data from Figure 4
10- 10 samples
used. We conduct only one experiment per point for DINE-Gaussian and MINE, acquiring CIs from epoch-wise averaging instead. Note that DINE-Gaussian failed to estimate MI for in the case of nigh-dimensional smoothed uniform distribution due numerical instabilities. We also provide the plots for N N-MIENF and Midia--V-MIEN to facilitate the comparison.
21

---

C
Overfitting and sample complexity
Due to the curse of dimensionality being universal issue for MI-related tasks [13, our method, as any other NN-based estimator, is prone to overfitting in the case of small sample size. We illustrate this by performing an MI estimation being of normal and tiny size. We also conduct similar experiments with MINE. The resulting probability density and pointwise mutual information functions are presented in Figures and
of the benchmarks from Section 3 with the sampling
(a) (Xx;)= 1.0, (( R 0.99, 2048 samples.
(b) ((; 0.0. T(x; Y) ~ 0.0, 2048 samples.
(c) I(X:) 0.0. i(X;Y) 2 1.84, 5 samples.
Figure 6: Point-wise mutual information plots for MINE. Correlated uniform distribution is used, with varying ground truth MI and sampling size. Note that in the case of insufficient sampling size, MINE "memorizes" the data points and "hallucinates" the relation between X and Y, which severely increases the value of the MI estimate.
(a) (( 1.0, ((;;Y) & 0.99, 2048 samples.
(b) (X 0.0.
(c) (X);) 0.0. f(X;)) & 0.92, samples.
f(x; Y) 0.0, 2048 samples.
Figure 7: Probability density function plots for tridiag---MIENT Correlated uniform distribution used, with varying ground truth MI and sampling size. Note that in the case of an insufficient sampling size, MIENF "memorizes" the data points and "hallucinates" the relation between and Y, which severely increases the value of the MI estimate.
D Information-theorriic disentanglement
To explore additional applications of our method, consider the task of representation disentangle- ment, i.e., the process of separating data into independent variables with distinct semantic meaning. For this example, use the MNIST dataset of handwritten digits [53].
Let be random image of handwritten digit. Consider a Markov kernel + (X', Xx) corresponding to pair of random augmentations applied to X (we use random translation, rotation, zoom, and shear from torchvision transf orms). Now consider the task of estimating I T(X'; X") (MI between the two augmented versions of the same image). Note that tridiag-V-MIENN estimates the MI and performs nonlinear canonical correlation analysis simultaneously (because of the tridiagonal covariance matrix in the latent space), Moreover, { (from Definition 4.11 represent
22

---

the dependence between the nonlinear components. Higher values of Pj (and, as a consequence, of the per-component MI) are expected to correspond to the nonlinear components, which are invariant to the selected augmentations (e.g., widtt//eight ratio of a digit, thickness of strokes, etc.). We also expect small values of Pj to represent the components, which parametrize the augmentations used in our setup (e.g, translation, zoom, etc.)
To perform the experiment, we train Pidiag--V-MIEN on samples from (X', We then randomly select several images from X, acquire their latent representations, apply a small perturbation along the axes corresponding
the highest and the lowest values of (one axis at time), and perform an inverse transformation to visualize the result. We observe the expected behavior. The results are We use a convolutional autoencoder beforehand to reduce the dimensionality (the size of the bottleneck is 64) and speed up the experiment.
provided in Figure g
E Technical details
In this section, we describe the technical details of our experimental setup: architecture of the neural networks. hyperparameters, etc.
For the tests described in Section we use architectures listed in Table 2 For the flow models,
we use the normfl 1 ows package [54]] The autoencoders are trained via Adam 55 optimizer on
100 images with a batch size 5 10 a learning rate 10-3 and MAE loss for 2 10 epochs. The MINE/NWJ/Nishiyaan
critic network is trained via the Adam optimizer on 5- 10 pairs of images with batch size 512, a learning rate 10-3 for 5- 10 epochs. The GLOW normalizing flow is trained via the Adam optimizer on 10 10x images with batch size 1024, a learning rate decaying from 10-4 to 1 10-5 5 for 2- 10s epochs. Nvidia Titan RTX was used to train the models. In any setup, each experiment took no longer than one hour to be completed. In the following repositories, we provide PyTorch implementations of the NN- estimators we used: https: ///ggittuu com/ VanessB/pytorchh kld and https: : /tt///gtttt github. wwwm///nees//yytoorcc.cooo
Table 2: The NN architectures used to conduct the tests in Section 5
NN
Architecture
x1:
Conv2d(1, 4, ks=3), BatchNorm22d, Leaky MaxPool2d((2)
x1:
Conv2d(4, 8, ks=3), BatchNorm22, LeakyReLU(0.22) MaxPool2d(2)
AEs,
x2(3):
C8nv2d(8, 8, ks=3), BatchhNorm2d, LeakyReLLU00.2) MaxPool2d(2) Dense(8, dim). Tanh, Dense(dim, 8). LeakyReeLUO0.22)
16 x 16 (32 x 32)
X1:
images
x2(3):
Upsample(2), Conv2d(8, ks=3), BatchNorm2d, LeakyReLU(002) Upsample Conv2d(8, 4, ks=3), BatchNorm22,, Lea.) Conv2d(4, 1, ks=3), BatchNorm22, LeakkRel.U(0..2)
x1:
x1:
x1:
[Conv2d(1, 16, ks=3), MaxPool2d(2), LakkyeeLOO0 poralll
MINE, critic NN, 16 x 16 (32 x 32)
x1(2):
[Conv2d(16, (16, 16, ks=3), MaxPool2d(2), LeakyReLU(0.00)) x2 in paralld Dense(256, 128), LcakyReLU(0.01)
x1:
images
x1:
Dense(128, 128), LeakyReLU(00.0))
x1:
Dense(128, 1)
1 (5) splits, 2 GLOW blocks between splits,
GLOW,
x1:
16 hidden channels in each block, leaky constant = 0.01
16 x 16 (32 x 32)
x1:
Orthogonal linear layer
images
x4:
Permuteswaap)
ReaINVP, d-dimensional data
x6:
RelleFe.tthnncoogggggooooood///d d)), Permute-swap)
Here we do not explicitly define 95 and 9y, used in the tests with synthetic data, as these functions smoothly map low-dimensional vectors to high-dimensional images and, thus, are very complex. A Python implementation of the functions in question is available in the supplementary code repository, see https: //gi thub. com/ /tt///aanee////tinfoo.
23

---

3
3
3
3
3
0
0
0
7
7
7
A
2
2
2
2
(a) Nonlinear component corresponding to the stroke thickness. MI R 0.96.
6
6
6
6
7
7
7
5
5
5
5
(b) Nonlinear component corresponding to the width of digit. MI =s 0.75.
6
4
4
4
(c) Nonlinear component corresponding to zoom transformation. MI 0.002.
7
7
7
7
7
O
9
9
9
9
9
5
5
5
5
5
(d) Nonlinear component corresponding vertical translation. MI < 0.001.
Figure 8: Results of an information- based nonlinear canonical correlation analysis performed on the MNIST handwritten digits dataset. The task of MI estimation between augmented (trans- lated/rotated//..) versions of pictures is considered. Our method (the tridiagonal version) allows for simultaneous MI estimation and nonlinear independent components learning. We illustrate the semantics of the learned nonlinear components via small perturbations along the corresponding directions in the latent space. The center of each row contains original, unperturbed picture; pictures to the left and to the right are the results of the perturbations. We also provide the values of per-component MI. Components with high MI represent the features, which are invariant the selected augmentations.
24

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: in work, dedicate Sections 3 and 4
proper introduction and
theoretical justification of the proposed approach. Section 5
contains the experimental results showing decent performance of method. All the results match the claims from the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: we have dedicated paragraph in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
25

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
we provide all the proofs in Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: we describe our experimental setup in Section 5 with corresponding technical details being provided in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
26

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
the source code is available https://gitine Cww.llu...oo///aeeeeee
pytorch mniff
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
wublic//guides/Ddeeebbiiiss www.aimmmiiiioooooooiooooooe for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips //ipss.c/pput cttp:/scccpubblcccuuiiice guides/ /ttpp///bb///i..i./////////aaiaa
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: we describe our experimental setup in Section 5 with corresponding technical details being provided in Appendix E
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: results presented in Section include confidence intervals.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
27

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: we provide all the information Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification: the research conducted in the paper conform with the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: there is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
28

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: the paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: we cite the authors of all the benchmark datasets used in our work in Section 5 We cite the authors of the normflows package in Appendix E
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
29

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: the source code is documented via in-code commentary.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
30

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "rXTn8TeUsf", "reviewer": "Reviewer_cdSA", "summary": "This paper presents a new approach for estimating mutual information (MI) using normalizing flows. The authors provide a series of theoretical results and demonstrate numerical examples.", "questions": "1. Sec 4.4, would the double log (double exponential) introduced here cause numerical issues?\n\n[Typo] Line 427, Ref[44]", "limitations": "The authors adequately addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper has provided comprehensive theoretical discussions for the proposed approach. The presentation is easy to follow.\n2. The analyses seem to be sound and clear though I did not check the derivations line by line.\n3. The authors conducted various experiments to test the algorithm.", "weaknesses": "1. Though the theoretical development is self-contained, the paper could potentially benefit from experiments on real datasets with MI-related tasks.\n2. The presentation could be improved by adding visual details, such as a diagram of the general algorithm or examples of synthetic images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer **cdSA**, thank you for reading and reviewing our article carefully!\nIn the following text, we provide responses to your questions.\nWe hope that all the concerns are properly addressed.\n\n**Weaknesses:**\n\n1. We agree that experiments with real datasets will further improve our work and complement the current results acquired on synthetic data.\n   Unfortunately, such data does not allow for a direct assessment of the quality of the MI estimators.\n   That is why we focus on synthetic tests, while still trying to reproduce the main challenges of realistic data:\n   high dimensionality and latent structure.\n\n   To partially address this issue, we employ our method to conduct a nonlinear canonical correlation analysis of the handwritten digits dataset.\n   We describe the experiment and provide the results in the general reply.\n   Please, note that this experiment was conducted under the time constraints of the rebuttal process.\n\n2. Two diagrams of both MI estimatiors will be added to the article, as well as some examples of the synthetic data.\n   We have also added these figures to the global reply PDF.\n\n**Questions:**\n\n1. Note that we store $w_j$ and compute $\\hat I = \\sum_j e^w_j$ and $\\rho_j = \\sqrt{1 - \\exp(- 2 \\cdot e^{w_j})}$.\n   As $-2 \\cdot e^{w_j} < 0$, $\\exp(-2\\cdot e^{w_j}) \\in (0;1)$, which partially secures the method from numerical instabilities.\n   Some other issues might arise from the gradient calculation and backpropagation,\n   but we did not observe them.\n   Moreover, the training process for tridiag-$\\mathcal{N}$-MIENF was relatively stable,\n   with the loss function plot being quite smooth and almost monotonic.\n\n   One can also use other smooth mappings from $\\mathbb{R}$ to $(0;+\\infty)$ to parametrize $\\rho_j$.\n   For example, if the softplus function is employed, $\\hat I = \\sum_j \\log(1 + e^{w_j})$ and $\\rho_j = \\sqrt{1 - 1/\\sqrt{1 + e^{w_j}}}$,\n   which eliminates one of the exponential.\n   We will also reflect this approach in the final revision of the work.\n\n2. The typo will be corrected, thank you!\n\nWe hope that we were able to answer all the raised concerns. We again thank Reviewer **cdSA** for the work!\nIf there are any questions left, we will be glad to answer them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RyKw1clnUb", "reviewer": "Reviewer_4miK", "summary": "The authors aim to provide an automatic method for performing an information-theoretic analysis of DNNs\nHowever, calculating mutual information (MI) and differential entropy of high-dimensional data are extremely hard to estimate.\nThe authors propose modeling a joint distribution of RVs with a Cartesian product of normalizing flows, which allows for a direct estimation of MI. Through some minor restrictions on the base distribution, the authors derive a very computationally feasible approach, while still demonstrating good performance in their experiments.", "questions": "Statement 4.8 requires that fx and fy be block diagonal, however Statement 4.8 applies Corollary 4.3 and 4.4 which defines on these RVs not to be block diagonal. Please clarify if this is correct, and if so does it limit the derived bound?", "limitations": "In paragraph 238 and 285 you briefly mention extensions to injective normalizing flows, specifically citing [9] which shows normalizing flows for manifold learning. This may work, but depending on the type of mapping, that flow may be optimizing a lower bound (and not exact an likelihood calculation), which may have negative impacts on your MI estimator. It is fine if this is outside the scope of your paper, but you may want to mention potential limitations or challenges if you suggest this approach.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Overall, the paper is strong, showing a high level of rigor and precision without overcomplicating the subject. In the general Gaussian case, they show that a lower bound can be calculated in close-form — a notable advantage over prior work. Further, the authors place reasonable restrictions on the base distribution in order to avoid the more computationally intense calculations (e.g., inverting covariance matrix). The experimental results are promising and even show that tridiag N-MIENF yields nearly the same results as N-MIENF (Figure 1). Additionally, the results in Figure 2 show not only good performance of the estimator, but also evidence of the bound holding.", "weaknesses": "- Line 145 this is not clear, consider explaining or removing.\n\n- Why aren't results for tridiag N-MIENF shown in Figure 2? The tridiag N-MIENF method is a strong contribution if the experiments can sufficiently support that the simplified bound is not too determinantal to the performance.\n\n- The paper's initial motivation describes tools for explainable AI and an information-theoretic analysis of deep neural networks, however none of the experiments or results apply the method in this setting. Instead the experiments estimate MI from toy data that is algorithmically generated with known MI. The experiments support the method, but the paper lacks a compelling view of what cheap, accurate estimation of MI truly unlocks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to deeply thank Reviewer **4miK** for reading the article and providing us with a profound review!\nWe further provide answers to the main points raised in the review.\n\n**Weaknesses:**\n\n1. We wanted to stress out that the biggest possible gap between the lower bound and the true value of MI is indeed achievable.\n   Thus, the bound in Corollary 4.4 is tight.\n   The same example from Remark 4.5 also shows that it is insufficient to marginally Gaussianize a pair random vectors,\n   as the MI estimation error may become arbitrary high.\n   We will explain this remark more thoroughly in the next revision.\n\n2. This appears to be a small mistake in the legend, as the results are provided for tridiag-$\\mathcal{N}$-MIENF.\n   Choosing the tridiagonal version over the basic was indeed motivated by the near-identical performance (Figure 1),\n   as well as the mathematical equivalence of the methods (see Statement 4.13).\n   To avoid clutter, we plot only one method in Figure 2,\n   as the results of both methods on synthetic images are also very close to each other.\n   The label will be changed to represent the right method.\n\n3. In order to assess the quality of our method and compare it to other approaches,\n   synthetic datasets had to be utilized, as they are the only option with available ground truth MI.\n   To improve the tests, we tried to reproduce the main challenges of realistic data:\n   high dimensionality and latent structure.\n\n   It is true that our work would benefit from experiments in the setting of explainable AI and an information-theoretic analysis of DNNs.\n   However, we decided to leave these experiments for the future, and focus on refining and evaluating the method.\n   We believe that the resulting estimator is still a notable contribution to the field.\n\n   Nevertheless, to partially address this issue, we employ our method to conduct a nonlinear canonical correlation analysis of the handwritten digits dataset.\n   We describe the experiment and provide the results in the general reply.\n   Please, note that this experiment was conducted under the time constraints of the rebuttal process.\n\n**Questions:**\n\nWe do not require $f_X$ and $f_Y$ to be block-diagonal.\nInstead, we only utilize the fact that $f=f_X \\times f_Y$ has a block-diagonal Jacobian,\nas it is a Cartesian product of two mappings.\nThis is just a small simplification,\nwhich is not of great importance.\n\n**Limitations:**\n\nThis is a valid point, which we will address in the next revision.\nThe quick answer is that exact likelihood calculation is not required in our method.\nHowever, optimization of the exact likelihood (instead of a lower bound) should tighten the bound from Corollary 3.2 better,\nwhich in turn should yield better MI estimates.\nWe also consider extending our idea to non-likelihood based methods as a part of the future work.\nThis discussion will be added to the limitations section.\n\nWe again sincerely thank Reviewer **4miK** for carefully reading our article and providing us with helpful comments.\nWe will be glad to answer any further questions if they arise.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iECLZRmNB1", "reviewer": "Reviewer_m9Tu", "summary": "This work presents an elegant and sound methodology for the estimation of mutual information (MI) in the context of high-dimensional continuous random variables (vectors). \n\nThe key intuition for the proposed methodology is that MI is an invariant measure under smooth injective mappings. Then, the authors define such mappings as trainable normalizing flows, which are used to transform the original data distribution in such a way that computing MI on such learned transformation is easy.\n\nFirst, the authors propose a general method, which requires training normalizing flows for each random vector (say, $f_X$ and $f_Y$), and a simple model $q$ to approximate the PDF in latent space which has tractable point-wise MI. This approach allows to build a MI estimator whereby models $q$, $f_X$ and $f_Y$ are trained according to maximum likelihood.\nFor the general method, the authors show that the estimator is consistent, as well as they derive bounds that depend on the KL divergence between the the PDF of the transformed input vectors and the PDF of the mode $q$.\n\nThen, the authors refine their general method with the goal of practicality. In this case, they restrict model $q$ to belong the family of multivariate Gaussian distributions. This allows to estimate MI via closed form-expressions, and to derive the optimal model $q$ in closed form as well. Furthermore, the authors show how to derive better non-asymptotic bounds and the variance of the proposed MI estimator.\nIn practice, this method requires training the cartesian product of $f_X \\times f_Y$ as a single normalizing flow, and maximize the log likelihood of the joint sampling of input data, using as base distributions the whole set of Gaussian distributions.\nIt is important to notice that the authors are well aware of some issues that might affect the proposed Gaussian base distribution method. As such, they refine their method by considering a special kind of base distributions whereby the co-variances of the Gaussian base are sparse, tridiagonal and block-diagonalizable. Overall, this boils down to simplifying model $q$ from the general method described above by requiring the estimate of parameters $\\rho$ that are reminiscent of non-linear canonical correlation analysis.\n\nExperiments on synthetic data complement the methodological part of the paper, with the goal of assessing the quality of the proposed estimator in difficult setups, that is, with input data distributions that are high-dimensional, exhibiting long tailed distributions, and for which MI can be high. \nExperiments are built from prior work that defined a set of benchmarks to compare available MI estimators.\nResults indicate that, when ground truth is known, the proposed technique is stable, scales to high-dimensions and can cope with difficult data distributions.", "questions": "* Based on the comment about scalability above, would it be possible to provide a detailed discussion about the computational complexity of the proposed method (both in the general and in the refined variants)?\n\n* Based on the comment on sample complexity above, can you help me better understand the implications of the quality of $q$ (and as a matter of fact, also of the normalizing flows) for cases in which only a limited amount of samples from the input distributions $X$ and $Y$ are available?\n\n* Which problems, if any, do you foresee in the application of the proposed method on realistic data?\n\n* In some application scenarios, MI has been used in conjunction with optimization objectives to learn the parameters of a model, e.g., in representation learning [1], or in text-to-image alignment problems [2]. In such cases, point-wise mutual information is generally preferred. In your work, you show through expression (5), that the same methodology can be applied to point-wise mutual information estimation. Can you elaborate more on this aspect?\n\n[1] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In Proceedings of the 35th International Conference on Machine Learning, 2018.\n\n[2] Xianghao Kong, Ollie Liu, Han Li, Dani Yogatama, and Greg Ver Steeg. Interpretable diffusion via information decomposition. In The Twelfth International Conference on Learning Representations, 2024.", "limitations": "Yes, these have been thoroughly discussed throughout the paper and in a dedicated section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "* I really commend the very clear and sound mathematical treatment of both the general and the refined method. It is easy to follow, definitions are clear, theorems are well built and well commented. Also, consistency and bounds for the various proposed estimator variants are proposed, which is not always the case for competing methods\n\n* This work is well rooted in the literature, and propose a different approach (which holds some similarity with recent work such as [22]) for MI estimation, which is sound and practical\n\n* The authors did a good job in spelling out the limitations of their method, and came up with refinements that overcome such problems", "weaknesses": "* The computational scalability of the proposed method is not discussed in detail. The authors focus on requiring a simple model for $q$, in terms of parameter count, but do not spend the same energy in discussing the complexity of learning the single normalizing flow as a surrogate of the cartesian product of individual (in terms of input variables) flows. In the general case, the proposed method requires three models to be learned: two normalizing flows and the model $q$. This is in contrast to the recent literature such as the work in [22] cited by the authors, whereby amortization techniques are used to learn a unique model.\n\n* It would be interesting to discuss in more detail the role of model $q \\in \\mathcal{Q}$. What is written in 106 to 110 is an excellent starting point, which culminates in Theorem 3.1 and Corollary 3.2, as well as Theorem 3.3 on the consistency of the estimator. What I am wondering about is the relationship between the “quality of $q$” and the sample complexity of the estimator. It is well known that, in general, neural estimation of MI suffers from high sample complexity, and since the proposed method falls in the family of neural estimators, a detailed discussion (or maybe experiments) on sample complexity seems missing\n\n* The experiments presented in this paper are compelling and aligned with the literature. Nevertheless, since the narrative of this work is centered around practicality of the proposed method, it would have been a nice addition to have at least one experiment on realistic data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer **m9Tu**, thank you sincerely for your profound and comprehensive review!\nIn the following text, we provide responses to your questions.\nWe hope that all your concerns are properly addressed.\n\n**Weaknesses:** for 1 and 2 please refer to the questions section.\n\n3. In order to assess the quality of our method and compare it to other approaches,\n   synthetic datasets had to be utilized, as they are the only option with available ground truth MI.\n   To improve the tests, we tried to reproduce the main challenges of realistic data:\n   high dimensionality and latent structure.\n   It is, however, true that our work would still benefit from experiments on realistic data.\n\n   To partially address this issue, we employ our method to conduct a nonlinear canonical correlation analysis of the handwritten digits dataset.\n   We describe the experiment and provide the results in the general reply.\n   Please, note that this experiment was conducted under the time constraints of the rebuttal process.\n\n**Questions:**\n\n1. We believe that having three models instead of one should not be a significant problem if such approach allows for less learnable parameters to be used.\n   In contrast to a conventional approach, where three normalizing flows are used to estimate $h(X)$, $h(Y)$ and $h(X,Y)$ separately,\n   our refined method should be as expensive as learning only two flows for modeling $p_X$ and $p_Y$\n   (as we employ a Cartesian product of two flows).\n\n   The general method additionally requires learning (possibly complex) $q$,\n   but this approach still should not be as expensive as training a whole separate flow to model the joint distribution $p_{X,Y}$ from scratch.\n   That is because the joint distribution should be already partially disentangled and simplified by $f_X \\times f_Y$.\n\n   In [22], several clever tricks are employed to learn only one model.\n   More specifically, manipulating the conditioning parameter $c$ in MINDE-C or diffusion speed modulators $\\alpha,\\beta$ in MINDE-J\n   allows for using a unique score network to model all the required distributions.\n   One is able to apply similar tricks to normalizing flows by using autoregressive flows [a,b],\n   but such approach is incompatible with our idea, as it can not be implemented via a Cartesian product of two flows.\n   However, this amortization technique can be applied to $q$.\n\n   We also note that all three models are learned in a single training loop with a shared loss function.\n   Thus, this is basically a single model consisting of three blocks.\n\n   Finally, the complexity of $q$ can be gradually increased (e.g., more components are added to a Gaussian mixture) to ensure that the most simple model is used.\n   This procedure should be fairly cheap, as it does **not** require retraining $f_X$ and $f_Y$ every time the complexity of $q$ is increased.\n   This can be interpreted as a gradual unfreezing of some parts of the model.\n\n   *Summary:* we believe that our method is as expensive as modeling just $p_{X,Y}$ in the worst case scenario.\n   The same should be true for both methods from [22].\n\n2. In this work, we decided to focus on the refined approach for several reasons:\n   mathematical elegance, better error bounds and relatively decent performance.\n   Nevertheless, we also consider the question of sample complexity of the general method to be very important and interesting.\n   Although we admit that a proper investigation of this topic requires conducting an additional set of experiments,\n   we still would like to provide some related discussion.\n\n   It is indeed known that any consistent MI estimator requires an exponentially (in dimension) large number of samples [b,c].\n   This fact is related to the high sample complexity of the PDF estimation task.\n   One may expect $\\mathcal{N}$-MIENF to be less prone to the curse of dimensionality,\n   as it is a fairly restricted model.\n   This, of course, comes at a cost of less expressive power.\n   On the other hand, increasing the complexity of $q$ in the general method makes the model less regularized and more expressive,\n   which increases the effects of the curse of dimensionality, but also improves the quality of the estimate for large datasets.\n   Thus, one can expect the sample complexity to increase as a consequence of using more expressive $q$.\n\n   Finally, the sample complexity also depends on $f_X$ and $f_Y$.\n   More expressive flows may require large numbers of samples to avoid overfitting.\n   In the case of small, but high-dimensional datasets NN-based MI estimators tend to \"memorize\" pairs of samples from $p_{X,Y}$,\n   yielding extremely high MI estimates.\n   We demonstrate this behaviour in the supplementary PDF, see the global reply.\n\n   *Summary:* using less expressive $q$, $f_X$ and $f_Y$ may serve as a regularization,\n   thus partially decreasing the sample complexity at a cost of lower accuracy for large datasets.\n\n3. Due to the restrictive nature of normalizing flows, they are relatively expensive and less stable to train.\n   They are also inferior to diffusion models in terms of generative capabilities.\n   We believe that all these nuances may make it harder to apply our method to real data of very high dimensionality and complex structure.\n   We consider combining our binormalization method with diffusion models to alleviate these difficulties.\n\n4. Yes, our method is perfectly suitable for PMI estimation.\n   Moreover, it is even possible to calculate PMI via a closed-form expression\n   while using less restricted $q$ (e.g. Gaussian mixture).\n   This allows us to achieve both consistency and simplicity of the PMI estimator.\n\nWe hope that we were able to answer the raised concerns.\nThese answers will be incorporated into the manuscript.\nWe again sincerely thank Reviewer **m9Tu** for carefully reading our article and providing us with useful comments,\nhelpful criticism and interesting questions!\nWe will be glad to answer any further questions if they arise.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lxsjomArIo", "reviewer": "Reviewer_2t4W", "summary": "This paper proposes a family of mutual information (MI) estimators based on normalizing flows.  The general method proposes a Monte Carlo estimator that holds for any base distribution.  The authors refine this estimator for Gaussian base distributions, where the MI estimate can be calculated analytically.  Finally, the authors propose a simplified Gaussian base estimator with fixed mean and tridiagonal covariance matrices, which are simpler to compute compared to the general Gaussian case.", "questions": "See comments in Weaknesses section.", "limitations": "The authors outline limitations of the proposed methodology.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper addresses an important problem of estimating information-theoretic measures for arbitrarily complex target distributions.  Results in the paper appear technically sound despite a relatively high technical nature of the content.", "weaknesses": "The presentation of results is somewhat dense in places and the paper would benefit from more clarifying statements that contextualize the impact of results.  For example, Sec. 4.2 is rather dense with mathematical statements, but would benefit from remarks that inform the reader in plain English what impact these statements have.\n\nExperimental validation is somewhat lacking in several aspects.  Firstly, the results in Fig. 1 are provided only with comparison to ground truth.  The authors claim that no baseline methods are provided for comparison as similar experiments have been conducted.  But without baseline comparison it is impossible to draw conclusions about efficacy of the proposed approach.  Secondly, the results are presented with minimial-to-no insight / discussion beyond simple validation metrics.  The authors should consider showing plots of the density estimates for several cases (i.e. at least the lower dimensional settings).  Finally, the authors do not compare to any existing flow-based methods such as DINE-Gaussian, which would be a useful comparison.\n\nA simple oversight in this paper is the lack of any related work section.  Without this discussion it is difficult for the reader to contextualize the contributions of this paper relative to prior work.  This reviewer acknowledges that references are provided in the introduction, but the authors do not make clear how their current work advances the state-of-the-art.\n\n**Detailed Comments**\n*  I think the absolute values on the LHS of the equation in Corollary 3.2 are not necessary.  Similarly I think they are not needed in Corollary 4.4.\n* Eq. (6) : rogue comma in denominator of first term\n* Thm. 3.1 : change \"defined on the same space as\" to \"absolutely continuous w.r.t.\"\n* The justification of MLE is a little unclear, I think this is a result of corollary 3.2?\n* Eq. (8) : This bound isn't directly proven as the difference in KL terms isn't shown to be nonnegative.  I think one can arrive at this conclusion for the same reason that the absolute values are not needed in Corollary 4.4?\n* Sec. 5 : Consider showing example images of geometric shapes (i.e. in appendix) for the reader to better understand this experiment-in particular it is unclear what the authors mean by \"incompressible\" images\n\nA low-priority high-level comment : The paper begins by arguing that the work is motivated by information-theoretic analysis of DNNs.  This argument is also revised in the discussion.  But  connection of this work to information-theoretic analysis of DNNs is never demonstrated.  The authors should consider motivating the work by more concrete applications such as representation learning or Bayesian experimental design, both of which employ MI estimates widely.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer **2t4W** for the work!\nWe are glad to receive helpful criticism of our article.\nWe further provide answers to the main points raised in the review.\n\n**Weaknesses:**\n\n1. It is true that some parts of our work are dense with theoretical results.\n   As our manuscript has reached the page limit,\n   we hope to utilize the additional content page in the final version of the article to add more complementary explanation and discussion.\n\n2. We agree that our article would benefit from the proposed amendments.\n   Some examples of density plots will be added to the article;\n   they are also available in the supplementary PDF (Figure 5); please, see the global reply.\n   An additional comparison with MINE and DINE-Gaussian has been also conducted;\n   please, refer to the same PDF, Figure 6.\n   In our setup, DINE-Gaussian did not yield reasonable results in the low-dimensional case,\n   severely underestimating the MI.\n\n3. We believe that the overview provided in the introduction is mostly sufficient to familiarize the reader with the related work.\n   Nevertheless, we will additionally elaborate on this topic to explain the relation of our method to the previous estimators in more details.\n\n**Detailed comments:**\n\n1. Unfortunately, the absolute values are required for the following reason.\n   In general, $D_\\text{KL}(p_{\\xi,\\eta}||q_{\\xi,\\eta})$ and $D_\\text{KL}(p_\\xi \\otimes p_\\eta||q_\\xi \\otimes q_\\eta)$ are incomparable.\n   For example, if $\\xi=\\eta$ under $p$ and $q$, then $D_\\text{KL}(p_{\\xi,\\eta}||q_{\\xi,\\eta}) = D_\\text{KL}(p_\\xi||q_\\xi) < 2 \\cdot D_\\text{KL}(p_\\xi||q_\\xi) = D_\\text{KL}(p_\\xi \\otimes \\eta||q_\\xi \\otimes q_\\eta)$.\n   Conversely, if $p_\\xi \\otimes p_\\eta = q_\\xi \\otimes q_\\eta$ but $p_{\\xi,\\eta} \\neq q_{\\xi,\\eta}$,\n   we have $D_\\text{KL}(p_{\\xi,\\eta}||q_{\\xi,\\eta}) > 0 = D_\\text{KL}(p_\\xi \\otimes \\eta||q_\\xi \\otimes q_\\eta)$.\n   Thus, the proposed bound is tight in both ways.\n2. Thank you, the typo will be corrected.\n3. Thank you again, the text will be changed.\n4. Likelihood maximization is equivalent to $D_\\text{KL}$ minimization (see [31]),\n   which tightens the bound in Corollary 3.2 and Corollary 4.3.\n   We will add this clarification to the manuscript.\n5. To derive the bounds in question, one also have to utilize the monotonicity property of the KLD\n   ($D_\\text{KL}(p_{X,Y}||q_{X,Y}) \\geq D_\\text{KL}(p_X||q_X)$).\n   This is reflected in our proofs of the bounds.\n   Please, refer to lines 780-781 (Appendix, proof of Corollary 3.2).\n6. Thank your for the suggestion! The pictures will be added to the article.\n   We will also add them to the supplementary PDF of the general reply.\n\n   Please, also note that we do not call synthetic images \"incompressible\";\n   this term is used to describe the datasets for experiments in Figure 1.\n   These datasets have no low-dimensional latent structure.\n7. It is true that adding more examples of applications will make the motivation of our work more clear.\n   This will be done in the next revision.\n\nWe again sincerely thank Reviewer **2t4W** for carefully reading our article and providing us with useful comments and helpful criticism.\nWe will be glad to address any further concerns if they arise.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the reviewers for their time and effort to make our work better. To address the raised concerns, we answered each reviewer in individual messages below.

Some questions require supplementary materials to be submitted, including pictures, diagrams and additional experimental results. That is why we provide the following in the attached PDF:

1. Diagrams of the general and Gaussian-based methods. This will be added to the article.
2. Examples of the synthetic images used to assess the quality of different MI estimation methods. This will be added to the article.
3. Density plots (for MIENF) and pointwise mutual information plots (for MINE) to better understand how the methods work and how they perform in the case of tiny datasets. This will be added to the article, including additional plots and discussion.
4. Additional experiments with baseline MI estimators (MINE and DINE-Gaussian). The same datasets were used as in Figure 1 of our paper. This will be added to the article in an enhanced form, with confidence intervals, technical details and additional discussion.
5. An example experiment with real data: an information-based nonlinear canonical correlation analysis of the MNIST handwritten digits dataset. Please, note that this experiment was conducted to address some of the issues raised by the reviewers, under the time constraints of the rebuttal process. It will be reproduced in the next revision of the article with a thorough discussion of the experimental setup and results.

**Description of the experiment with real images:**

Let $ X $ be a random image of a handwritten digit. Consider a Markov chain $ X \to (Y, Z) $ corresponding to a pair of random augmentations applied to $ X $ (we use random translation, rotation, zoom and shear from `torchvision.transforms`). Now consider the task of estimating $ I(Y;Z) $ (MI between the two augmented versions of the same image). Note that tridiag-$\mathcal{N}$-MIENF estimates the MI and performs a nonlinear canonical correlation analysis simultaneously (because of the tridiagonal covariance matrix in the latent space). Moreover, $ \rho_j $ (from Definition 4.11) represent the dependence between the nonlinear components. Higher values of $ \rho_j $ (and, as a consequence, of the per-component MI) are expected to correspond to the nonlinear components, which are invariant to the selected augmentations (e.g., width/height ratio of a digit, thickness of strokes, etc.). We also expect small values of $ \rho_j $ to represent the components, which parametrize the used augmentations (e.g, translation, zoom, etc.).

To perform the experiment, we train tridiag-$\mathcal{N}$-MIENF on samples from $(Y,Z)$. We then randomly select several images from $X$, acquire their latent representations, apply a small perturbation along the axes corresponding to the highest and the lowest values of $\rho_j$ (one axis at a time), and perform an inverse transformation to visualize the result. We observe the expected behavior. The pictures are provided in the attached PDF, Figure 7.

We again would like to thank all the reviewers for the work. We hope that we were able to address most of the issues.

Meta-review: This paper proposes a machine-learning-baesd method, termed mutual information estimation via normalizing flow (MIENF), to evaluate mutual information (MI) between random variables (rv's). The proposed method is to train a pair of normalizing flows, yielding a latent joint distribution whose marginals are Gaussians. As MI of such a pair of latent rv's is bounded from below by that of joint Gaussian rv's, the authors propose to use the lower bound as an estimate $\hat{I}_{𝒩-MIENF}$ of ML (equation (9)).
A variant $\hat{I}_{tridiag-𝒩-MIENF}$ (equation (11)) is also proposed on the basis of tridiagonalization of the covariance of the latents.

The rating/confidence scores of the reviewers were 7/4, 7/4, 7/4, and 4/4. Upon reading the paper myself, as well as the reviews and the author rebuttals, I also think that there are several weaknesses, some of which are summarized below. Most reviewers evaluated, however, that the significance of the contributions to the literature of neural-net-based estimation of MI would outweigh these weaknesses.
- The non-asymptotic error bounds (Corollaries 3.2 and 4.4) still depend on the true joint distribution $p_{\xi,\eta}$  (or its pushforward $\mathbb{P}\circ(f_X^{-1}\times f_Y^{-1})$ via $f_X\times f_Y$), so that one cannot evaluate them in practice without extensive Monte-Carlo integration. This paper does not provide any error bound that can be evaluated in a practical sense for the proposed estimator $\hat{I}:=\hat{I}_{𝒩-MIENF}$ itself.
Assuming that the normalizing flows $f_X,f_Y$ yield marginal Gaussianization, what one could say is that the proposed estimator $\hat{I}$ is a lower bound of the true MI (line 170), and one cannot tell anything about how large the estimation error $I-\hat{I}$ is.
- As **Reviewer m9Tu** commented, the sample complexity of the overall scheme, including what would be required for training of the normalizing flows $f_X,f_Y$, has not been discussed. I think that in a high-dimensional setting one would need a significant amount of samples for the training of normalizing flows, which would make the significance of the result (Lemma 4.9) of the asymptotic variance of the proposed estimator quite unclear, as it assumes $f_X,f_Y$ to be *fixed*.
- All the reviewers showed concern about the lack of experiments under more practical scenarios related with deep/representation learning.
- I would appreciate it if the authors would elaborate more on the consistency statement in Theorem 3.3. Here, one has to estimate $\hat{f}_X,\hat{f}_Y,\hat{q}$ in the non-parametric setting, and thus I think that consistency of maximum-likelihood estimation would not be so straightforward. The proof of Theorem 3.3 is so brief that it does not seem to be taking proper care of it. I would like to suggest to downgrade "Theorem" to "Claim" or somethink like this, unless a rigorous treatment of this issue is possible.
- In Definition 4.11, the crosscovariance matrix $\Sigma_{\xi,\eta}$ is assumed to be diagonal. What it exactly means should be clarified, because $\Sigma_{\xi,\eta}$ is not square when $d_\xi\not=d_\eta$. 
Similarly, in Statement 4.14 one has to take care of the case $d_\xi\not=d_\eta$ in order to give a correct expression for $\Sigma^{-1/2}$. 

Minor points:
- Line 132: This allows (a → u)s
- Line 138: $Z'$ is undefined.
- Definition 4.7: $S$ is undefined.
- Line 173: Let $(\xi,\eta)$ ha(s → ve)
- Lines 178, 223, 278; normalizing flow(s) setup
- Line 183: gradient descen(d → t) step
- Line 188: gradient ascen(d → t)
- Line 193: "ill-posed" might be "ill-conditioned".
- Line 198: I did not understand what "generalization ability" means. I guess that it should read "generality".
- Line 230: arbitrar(il)y close 
- Table 1: What the RMSE means should be made clear. My guess is that it is the RMSE of the estimated MI relative to the true MI.
- Line 292: which can (can) be
- Line 790: arrive (to → at)
- Equation above line 808: $\hat{m}+b$ → ${\color{red}A}\hat{m}+b$
- Equation below line 811: $\Sigma_{\xi_i,\eta_i}$ → $\Sigma_{\xi_{\color{red}j},\eta_{\color{red}j}}$
- Line 813: $\Sigma_k$ → $\Sigma_j$
- Line 816: these invertible linear transformation(s)
- Equation (13): Put a comma after $(4\varepsilon)^{-1}$.
- Equation (16): The conditions are $I(X;Y){\color{red}>}1/2$ and $I(X;Y){\color{red}\le}1/2$.
Decision: Accept (poster)