Beyond Normal: On the Evaluation of Mutual Information Estimators
Authors: Paweł Czyż
Frederic Grabowski
Julia E Vogt
Niko Beerenwinkel
Alexander Marx
Venue: NeurIPS 2023 poster (2023)
Keywords: Mutual Information
Invariance
Benchmark
Geometric Machine Learning
TL;DR: To benchmark classical and neural mutual information estimators, we develop a rich set of distributions easy to sample from and with known ground-truth mutual information.

Abstract: Mutual information is a general statistical dependency measure which has found applications in representation learning, causality, domain generalization and computational biology. However, mutual information estimators are typically evaluated on simple families of probability distributions, namely multivariate normal distribution and selected distributions with one-dimensional random variables. In this paper, we show how to construct a diverse family of distributions with known ground-truth mutual information and propose a language-independent benchmarking platform for mutual information estimators. We discuss the general applicability and limitations of classical and neural estimators in settings involving high dimensions, sparse interactions, long-tailed distributions, and high mutual information. Finally, we provide guidelines for practitioners on how to select appropriate estimator adapted to the difficulty of problem considered and issues one needs to consider when applying an estimator to a new data set.

Beyond Normal: On the Evaluation of Mutual Information Estimators
Pawel Czy2 Frederic Grabowski*
Julia E. Vogt Niko Beerenwinkel
Alexander Marx 1
DDpartment of Biosystems Science and Engineering, ETH Zurich "ETH AT Center, ETH Zurich
'Inttitute of Fundamental Technological Research, Polish Academy of Sciences 'Department of Computer Science, ETH Zurich
SIBB Swiss Institute of Bioinformatics
Abstract
Mutual information is general statistical dependency measure which has found applications in representation learning, causality, domain generalization and compu- tational biology. However, mutual information estimators are typically evaluated on simple families of probability distributions, namely multivariate normal distribution and selected distributions with one-dimensional random variables. In this paper, we show how to construct a diverse family of distributions with known ground truth mutual information and propose language independent benchmarking platform for mutual information estimators. We discuss the general applicability and lim- itations of classical and neural estimators in settings involving high dimensions, sparse interactions, long-tailed distributions, and high mutual information. Finally, we provide guidelines for practitioners how select appropriate estimator adapted to the difficulty of problem considered and issues one needs to consider when applying an estimator new data
language-independens
1 Introduction
Estimating the strength of non-linear dependence between two continuous random variables (r.v.) lies at the core of machine learning. Mutual information (MI) lends itself naturally to this task, due to its desirable properties, such invariance to homeomorphisms and the data processing inequality. It finds applications in domain generalization [Li et al., 2022, Ragonesi et al., 2021], representation learning [Belghazi al., 2018, Oord et al., 2018), causality [Solo, 2008, Kurutach et al., 2018], physics [Keys et al., 2015], systems biology [Selimkhanov et al., 2014, Grabowski et al., 2019, Uda, 2020], and epidemiology [Young et al., 2023].
Over the last decades, the estimation of has been extensively studied, and estimators have been developed, ranging from classical approaches based on histogram density [Pizer et al., 1987], kernel density estimation [Moon et al., 1995] to k-nearest neighbor [Kozachenko and Leonenko, 1987, Kraskov et al., 2004] and neural estimators [Belghazi al., 2018, Oord et al., 2018, Song and Ermon, 2020]. However, despite the progress that has been achieved in this area, not much attention has heen focused systematically benchmarking these approaches.
Typically, new estimators are evaluated assuming multivariate normal distributions for which MI is analytically tractable [Darbellay and Vajda, 1999, Kraskov et al., 2004, Suzuki, 2016]. Sometimes, simple transformations are applied to the data [Khan et al., 2007, Gao et al., 2015], moderately high-dimensional settings are considered [Lord et al., 2018, Lu and Peltonen, 2020], strong dependencies are evaluated [Gao et al., 2015]. Beyond that, Song and Ermon [2020] study the
"Equal contribution 'Joint supervision
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Gaussian
Correlated uniform
calf-cube mapping
Making margins bimodal
Swiss mapping
1.0
25
CDFNY
bimodailt?
0.0
MM
0.0
2.5
0.0
0.0
0.0
CDF1X
NAO
bimodal(XX)
Figure 1: Visualisations of selected proposed distributions. Two correlated Gaussian variables X and Y (1) can be transformed via the Gaussian CDF into correlated uniform variables (2), into long-tailed distribution via the "half-cube" mapping + /v [ (3), into a multi-modal distribution (4) or embedded in the three-dimensiona space via the composition of the Swiss roll mapping and Gaussian CDF (5). Color on the plot corresponds to the original Y variable.
self-consistency (additivity under independence and the data processing inequality) of neural MI estimators in the context of representation learning from image data.
Our Contributions In this work we show method of developing expressive distributions with known ground-truth mutual information (Sec. 2), propose forty benchmark tasks and systematically study the properties of commonly used estimators, including representatives based on kernel or histogram density estimation, kNN estimation, and neural network-based estimators (Sec. 3). We address selected difficulties one can encounter when estimating mutual information (Sec. 4), such as sparsity of interactions, mnig-tailed distributions, invariance, and high mutual information. Finally, we provide recommendations for practitioners on how to choose a suitable estimator for particular problems (Sec. 6). Our benchmark is designed so that it is simple add new tasks and estimators. It also allows for cross-language comparisons - in our work we compare estimators implemented in Python, R and Julia. All of our experiments fully reproducible by running Snakemake workflows [Mölder et al., 2021]. Accompanying code is available at http:/ //g githhb..c.. com/ cbg- ethz/ /mi..
Overall, our key findings from the benchmark can be summarized follows:
Testing on multivariate normal distributions gives a biased and overly optimistic view of estimator performance. In this setting, canonical correlation analysis (CCA), a model-based approach, emerges as the most effective method even if the model slightly misspecified. Compared to classical estimators, neural estimators excel in nigh-dimensional settings, capturing sparse interactions across multiple dimensions.
The popular KSG estimator [Kraskov et al., 2004] is accurate in low- and settings, but its performance suffers on problems involving high-dimensions or sparse interactions. Although MI is invariant to wide range of transformations (Theorem 2.1), numerical estimates, even with large sample sizes, are not. Thus, MI may not suitable when metric invariance to specific transformation families is required.
Multivariate Student distributions pose an important and hard challenge to many mutual information estimators. This effect is partially, but not fully, attributed to their long tails.
2 Mutual Information: Estimation and Invariance
We start by recalling the definition of MI, then discuss the problem of estimating it, and introduce the estimators used in the benchmark.
Consider two r.v. X and Y with domains X' respectively V, joint probability measure Pxy and marginal probability measures Px and Py, respectively. If Pxy is absolutely continuous with respect to Px X Py (c.g., X and Y are finite r.v.), MI is equal to the following Kullback- divergence":
I(xY)) DEL (Pxy || Px & Pr) Px) log dPxy.
where / = dPxx dPxy/d(Px /d(Px @ Py) is the Radon NikoddNi derivative. If the absolute continuity does not hold, then I(X;Y)= +oo [Pinsker and Feinstein, 1964, Theorem 2.1.2]. If X' and y are Euclidean spaces and measures Pxy, Px and Py have probability density functions (PDFs) with respect to the Lebesgue measure, then the Kullback-Leibler divergence can be written in terms of the PDFs.
We use the natural logarithm, meaning that the mutual information measured in nats.

---

However, we will later consider distributions which do not have PDFs with respect to the Lebesgue measure and the general Sadon-Nikodyy derivative must be used (see "Swiss roll" in Fig. 1).
In almost all applications, Pxy and Px @ Py are not known and one needs to estimate MI from finite sample from Pxy, i.e., realization ((x11 3/)::.... (#NN (X1, ..)..... (XN, YN) distributed according the joint distribution Pxy- As MI estimator will denote a family of measurable mappings, indexed by the sample size N and spaces X' and y (in most applications, Euclidean spaces of various dimensionalities): ex (/) + R. Composing this mapping with the r.v. representing the whole data set we obtain a real-valued r.v. ENY For a given Pxy we can consider the distribution of E It is often summarized by its first two moments, resulting in the bias and the variance of the estimator. Understanding the bias of an estimator however requires the knowledge of ground-truth MI, so the estimators are typically tested on the families of jointly multivariate normal or uniform distributions, with varying N and ground-truth I(X; Y) [Khan et al., 2007, Lord et al., 2018, Holmes and Nemenman, 2019, Song and Ermon, 2020].
(X )) of N i.i.d. r.v.
Constructing Expressive Distributions and Benchmarking Besides considering different types of distributions such as (multivariate)) normal, uniform and Student, our benchmark is based on the invariance of MI to the family of chosen mappings. Generally MI is not invariant to arbitrary transformations, as due to the data processing inequality (X): (Y)) S I(X;Y) some circumstances MI is preserved, which enables us to create more expressive distributions by transforming the variables.
However, under
Theorem 2.1. Let X, X', y and y" be standard Borel spaces (e.g., smooth manifolds with their Borel a-algebras) and f: X + X" and g: y - Y be continuous injective mappings. Then, for every X-valued r.v. X and y-valued rv. Y it holds that I(XY)) = I(f(())g(Y)))
This theorem has been studied before by Kraskov al. [2004, Appendix], who consider diffeomor- phisms and by Polyanskiy and Wu [2022, Th. 3.7J. who assume measurable injective functions with measurable left inverses. For completeness we provide a proof which covers continuous injective mappings in Appendix A. In particular, each of f and g can be a homeomorphism, diffeomorphism, or a topological embedding. Embeddings are allowed to increase the dimensionality of the space, so even if X and Y have probability density functions, f(X) and gYY) do not need to have them. Neither of these deformations change MI. Thus, using Theorem 2. 1, we can create more expressive distributions (y()) by sampling from Pxy and transforming the samples to obtain data set (((()))(())))))))))))))))))))))) ((f())) ((v1)),... ..., (f(xx) 9 E (X' x V')
While offering a tool to construct expressive distributions, another valuable perspective to consider with this problem is estimation invariance: although MI is invariant to proposed changes, the estimate may not be. More precisely, applying an estimator to the transformed data set results in the induced Ef (X)) and its distribution. If the estimator were truly invariant, one would expect that E E [10000) should equal E [ONY]. This invariance is rarely questioned and often implicitly assumed as given (sec c.g. Tschannen et al. [2020] or Murphy [2023, Sec. 32.2.2.31), but as we will see in Sec. 4.3, finite-sample estimates are not generally invariant.
r..
3 Proposed Benchmark
In this section, we outline forty tasks included in the benchmark that cover a wide family of dis- tributions, including varying tail behaviour, sparsity of interactions, multiple modes in PDF, and transformations that break colinearity. As our base distributions, we selected multivariate normal and Student distributions, which were transformed with continuous injective mappings.
We decided to focus on the following phenomena:
Dimensionality. High- Hligh-dimensional data are collected in machine learning and natural sciences [Bühlmann and van de Geer, 2011, Ch. 1]. We therefore change the dimensions of the X' and y spaces between and 25.
2. Sparsity. While the data might be high- high-dimensional, the effective dimension may be much smaller, due to correlations between different dimensions or sparsity [Lucas al., 2006]. We therefore include distributions in which some of the dimensions represent random noise,
3

---

which does not contribute to the mutual information, and perform an additional study in Section 4.1.
3. Varying MI. Estimating high MI is known to be diffieult [McAllester and Stratos, 2020]. However, we can often bound it in advance - if there are 4000 image classes, MI between image class and representation is most log 4000 = 8.3 nats. In this section, focus on problems with MI varying up to 2 nats. We additionally consider distributions with higher MI in Section 4.4.
4. Long tails. As Taleb [2020] and Zhang et al. [2021] argue, many real-world distributions have long tails. To model different tails we consider multivariate Student distributions as well as transformations lengthening the tails. We conduct an additional study in Section 4.2.
5. Robustness to diffeomorphisms. As stated in Theorem 2.1, mutual information invariant reparametrizations of random variables by diffcomorphisms. We however argue that when only finite samples are available, invariance in mutual information estimates may not be achieved. To test this hypothesis include distributions obtained by using diverse set of diffeomorphisms and continuous injective mappings. We further study the robustness to reparametrizations in Section 4.3.
While we provide a concise description here, precise experimental details can found in Appendix and we visualise selected distributions in Appendix F. We first describe tasks employing one- dimensional variables.
For multivariate normal variables MI depends only on the correlation matrix. We will therefore consider a centered bivariate normal distribution Pxy with Cor(X, = p and I(X;Y) -0.5 log (1 p-- We chose p 0.75.
Bivariate Normal
Uniform Margins As first transformation, we apply the Gaussian CDF F to Gaussian r.v. X and Y to obtain r.v. X' F(X) and Y' F(Y). It is standard result in copula theory (see Lemma D.2 or e.g., Nelsen [2007] for an introduction to copulas) that F(X) ~ Uniform (0, 1) resp. F(Y) Uniform(0, 1). The joint distribution Pxy' however, is not uniform. Mutual information is preserved, I(X'; -";=)(X;)). For an illustration, see Fig. A distribution transformed with Gaussian CDF F will be denoted Normal CDF @
Half-Cube Map To lengthen the tails we applied the half-cube homeomorphism h<z)=|=|3/3 sign to Gaussian variables X and Y. We visualize an example in Fig. and denote transformed distribution as Half-cube P.
Asinh Mapping
To shorten the tails we applied inverse hyperbolic sine function asinh. r log (x + vI x²) A distribution transformed with this mapping will be denoted as Asinh P.
Wiggly Mapping
To model non-uniform lengthscales, we applied mapping
w(x) = =x + Σ as sin(w; + p.),
Σ |aw!l < 1.
Due to the inequality constraint this mapping is injective and preserves MI. The parameter values can be found in Appendix D. We denote the transformed distribution as Wiggly P.
Bimodal Variables Applying the inverse CDF of wwo-component Gaussian mixture model to correlated variables X and with uniform margins we obtained a joint probability distribution Px.y" with four modes (presented in Fig. 1). We call this distribution Bimodal and provide technical details in Appendix D.
Additive Noise Consider independent r.v. X ~ Uniform(O, 1) and N ~ Uniform(-e,e where the noise level. For Y X + N, is possible to derive I(X; Y) analytically (see Appendix D for the formula and the parameter values) and we will call this distribution Uniform (additive noise=e).
Swiss Roll Embedding As the last example in this section, we consider a mixed case in which a one-dimensional r.v. X Uniform 1) is smoothly embedded into two dimensions via the Swiss roll mapping, popular embedding used to test dimensionality reduction algorithms (see Fig. 1 for visualisation and Appendix D for the formula). Note that the obtained distribution does not have PDF with respect to the Lebesgue measure on 3
Next, we describe the tasks based on multivariate distributions.

---

,,....,,,) Yx) from the multivariate normal distribution. We model two correlation structures: "dense" interactions with all off- didgoiaan correlations set to 0.5 and "2-pair" interactions where Cor(X1, (or(X1,X11) Yi) Cor(X2 Y2) = 0.8 and there is no correlation between any other (distinct) variables (Multinormal (2-pair)). For a latent-variable interpretation of these covariance matrices we refer to Appendix D.2.
Multivariate Normal We sampled (X, = (Xi,..., Xm: Yi...,,
Multivariate Student To see how well the estimators can capture MI contained in the tails of the distribution, we decided to use multivariate Student distributions (see Appendix D) with dispersion matrix² set to the identity In+ and y degrees of freedom. Contrary to the multivariate normal case, in which the identity matrix used as the covariance would lead to zero MI, the variables X and Y can still interact through the tails.
Transformed Multivariate Distributions As the last set benchmark tasks we decided to transform the multivariate normal and multivariate Student distributions. We apply mappings described above to each axis separately. For example, applying normal CDF to the multivariate normal distribution will map it into distribution over the cube (0, 1)m+a with uniform marginals. To mix different axes we used a spiral diffeomorphism (denoted as Spiral @ P); we defer the exact construction to Appendix D but we visualise this transformation in Fig. 5.
Estimators In our benchmark, we include a diverse set of estimators. Following recent interest, we include four neural estimators, i... Donsker-Varadhan (D-V) and MINE [Belghazi et al., 201 8), InfoNCE [Oord et al., 2018], and the NWJ estimator [Nguyen et al., 2007, Nowozin et al., 2016, Poole et al., 2019]. As a representative for model-based estimators, we implement estimator based on canonical correlation analysis (CCA) [Murphy, 2023, Ch. 28), which assumes that the joint distribution is multivariate normal. For more classical estimators, we include the Kraskov, Stöghauer and Grassberger (KSG) estimator [Kraskov et al., 2004] as the most well-known k-nearest neighbour- based estimator, recently proposed kernel-based estimator (LNN) [Gao et al., 2017b], as well as an estimator based on histograms and transfer entropy (both implemented in Julia's Transfer entererrrrrrpypy library). A detailed overview of the different classes of estimators is provided in Appendix C. Further, we describe the the hyperparameter selection in Appendix E.
Preprocessing As data preprocessing step, standardize each dimension using the mean and standard deviation calculated from the sample. Other preprocessing stategies are also possible, but did not observe consistent improvement of one preprocessing strategy over the others (see Appendix E).
3.1 Benchmark Results
We show the results for N 000 data points in Fig. 2. Neural estimators and KSG perform better than alternative estimators on most problems, with KSG having often low sample requirements (see Appendix E). The simple model-based estimator CCA obtained excellent performance at low sample sizes (see Appendix E), even for slightly transformed multivariate normal distributions. Finally, LNN and the two Julia estimators (histogram and transfer entropy), work well in low dimension but are not problemm
viable in medium- and high-dimensional problems (dim X + dim Y 2 4).
Overall, we observed that for most 1x l-dimensional and multivariate normal problems MI can be reliably estimated. Difficulties arise for sparse interactions, atypical (Student) distributions, and for significantly transformed distributions. This suggests that the typical evaluations of MI estimators are overly optimistic, focusing on relatively simple problems.
KSG is able to accurately estimate MI in multivariate normal problems, however performance drops severely on tasks with sparse interactions (2-pair tasks). For 5x5 dimensions the estimate is about 70% of the true MI, and for 25x x 25 dimensions it drops to 10%, which is consistent with previous observations [Marx and Fischer, 2022]. Meanwhile, performance of neural estimators is stable. We study the effect of sparsity more detail in Sec. 4.
Student distributions have proven challenging to all estimators. This is partially due to the fact that long tails can limit estimation quality (see See. 4), and indeed, applying a tail-shortening asinh transformation improves performance. However, performance still remains far below multivariate normal distributions with similar dimensionality and information content, showing that long tails
'The dispersion matrix of multivariate Student distribution is different from its covariance, which does not
exist for $ 2.
5

---

M4E-04 0. 1.7 0.5 0.3 0.
04
6.4
03
0.6
1.2
15
0.9
0.9
0.
0.
6160
RJ
03
0.9 0.
09
0.
08
5.4
0.
0.4
0.8
08
02
0.4
0.2
Woy-04 04 177 0.3
04
0.
0
0.
84
08
12
13
1.0
L1
0.
0.0
6.0
0100111010
81
08
10
10
08
6.5
0.9
07
04
0.
-V-
04
17
0.3
0.4
0.4
3.2
01
(R) D.4
0.3
D44
0.
5.0
0
02
01
LNW 1N -0. 0.3
11 0.4
0.5 0.3
03
0. 01
07 2.7
1.4 LB
51
2.9
10
6J
LS
3.
3
3-4
2.4 5.7
8.9
2.9 8.9
29
2.3
8
2.4 5.7
0.0
qutiar -2.4
Taanthrr quiio D. Yo
1.6 0.3 DJ 0. 0.4
6.3 WA 9.0
9.0
6.4
9.
90
00
0A
2.0
9.0
4.0
4.0
8.7
0.0
6.6
4.
9.0
0.5
9.0
90
1.1
D.4 04
0. 13
DO
20
True MI 0.4
0.4 1.3
1.0
1.0
1.0
1.
110
pooo come 1
x 1dbt+11
do00o 100/=11
Swete ml x
wutt cetromoo (d0000)
suit outtorrmn X (dfero))
muttiammn X.S. (lanna)
23 25 dieeoo)
you Pustin ma 50 (dumm)))
no x 12 parn
22 22-22222-333 (2-pa))
S..eett2 x 2100-11
St 12.deete x 2160 1100-21-
St.dentt x 31001-2)
Seudent 3x 3000--3
homma cae H.R.R nomalll X p2-
23 (2-p1))
velrouee Mutnorrall x 3222 pXn
Palteuse @ Mui norrral X 12 12-0107
Hafoute e Multno mol 25 25 R goii
Sope ( NLBtnora 5 31 10-
(j Hut 5 pai)
sond Namra con H.k.mmal 25 25 22 (2-
Adah o S. de.eeet 2 (00741)
Asah o wuuaeni si (000022)
mome oo0 pooning aDD Puaa
>
x
Jaah St ooent S
Sene x suceen
MoSoo x muns
wartho mom
M.linomaa
Narrral CaP R.llurd
Salmi a M.ltnoma
tate hormel car H.B noraal nomda
pe de soro Joo
Syhut to mornd cut
hormal
Figure 2: Mean MI estimates of nine estimators over n 10 samples with N = 10 000 points each against the ground-truth value on benchmark tasks grouped by category. Color indicates relative negative bias (blue) and positive bias (red). Blank entries indicate that an estimator experienced numerical instabilities.
only part of the difficulty. For neural estimators, the issue could be explained by the fact that the critic is not able to fully learn the pointwise mutual information (see Appendix E).
Interestingly, the performance of CCA, which assumes linear Gaussian model, is often favorable compared to other approaches and has very low sample complexity (see Appendix E). This is surprising, since in the case of transformed distributions the model assumptions are not met. The approach fails for Student distributions (for which the correlation matrix either does not exist or is the identity matrix), the Swiss roll embedding and multivariate normal distributions transformed using the spiral diffeomorphimm. Since the spiral diffeomorphism proved to be the most challenging transformation, we study it more carefully in Sec. 4.
4 Selected Challenges
In this section we investigate distributions which proved to be particularly challenging for the tested estimators in our benchmark: sparse interactions, long tailed distributions and invariance to diffeomorphisms. Additionally we investigate how the estimators perform for high ground truth MI.
4.1 Sparsity of Interactions
In the main benchmark (Fig. 2) we observed that estimation of MI with "2-pair" type of interactions is considerably harder for the KSG estimator than the "dense" interactions. On the other hand, neural estimators were able pick the relevant interacting dimensions and provide better estimates of MI in these sparse cases.
We decided to interpolate between the "dense" and "2-pair" interaction structures using two-stage procedure with real parameters a and À and the number of strongly interacting components K. While the precise description is in Appendix D.2, of interaction between every pair of variables in {X X10. Yi.... Y10} and as the additional strength of interaction between pairs of variables (X,, Yi), ..... (X Yx). Dense interaction struc- ture has À= = 0 and 2-pair interaction structure has a = 0 and K- = 2. First, we set K = 10 and X = 0 and decrease raising À at the same time to maintain constant MI of nat. When information is contained in the pairwise interactions (X,, Yi)...., (X1o. Y1o). We then decrease K and increase X, still maintaining constant MI.
interpret as controlling the baseline strength
O, whole
In Fig. we observe that the performance of all estimators considered (apart from CCA, a model- based approach suitable for multivariate normal distributions) degrades when the interactions between

---

1.00
1.00
MI estimate [nats]
estimate [nats]
0.75
0.75
InfoNCE
0.50
0.50
MINE
NWV
0.25
0.25
KSG (R)
True MI
0.00
0.00
1.0
0.5
0.0
Num. of Interactions
Figure 3: Tto-stage interpolation between dense and sparse matrices. From left to right the sparsity of interactions increases. Shaded regions represent the sample standard deviation.
endimatte Inats!
evtimen (patss
CCA
D-V
Hist (Julia)
Transfer (uila)
MINE
NW)
KSG (RI
15
LNN
Figure 4: MI estimates function of the information contained in the tail of the distribution. Left: lenghtening the tails via + |z| sign mapping with changing k. Right: increasing the degrees of freedom in multivariate Student distribution shortens the tails. Shaded regions represent the sample standard deviation.
every pair of variables become sparser (lower a). In particular, even neural estimators are not able to model the information ten interacting pairs of variables. However, when we decrease the number of interacting pairs of variables, the performance of the neighborhood-based KSG estimator qualitatively different from the neural estimators: performance of KSG steadily deteriorates, while neural estimators can find (some of the) relevant pairs of variables.
This motivates us conclude that in the settings where considered variables high-dimensional and the interaction structure is sparse, neural estimators can offer an advantage over neighborhooddbasce approaches.
4.2 Long-Tailed Distributions
In Fig. 4 we investigate two different ways to lengthen the tails. First, we lengthen the tails of multivariate normal distribution using the mapping + |ar sign (with & 1) applied to each dimension separately. In this case, we see that the performance of CCA, KSG, and neural estimators near-perfect for close to 1 (when the distribution Pxy is close to multivariate normal), but the performance degrades as the exponent increases. Second, we study multivariate Student distributions varying the degrees of freedom: the larger the degrees of freedom, the lower the information content in the tails the distribution. Again, we see that that the performance of CCA, KSG, and neural estimators is near-perfect for high degrees of freedom for which the distribution is approximately normal. For low degrees of freedom, these estimators significantly underestimate MI, with the exception of CCA which gives estimates with high variance, likely due the correlation matrix being hard to estimate.
that long tails complicate the process of estimating MI. The Student distribution is particularly interesting, since even after the tails are removed (with asinh or preprocessing strategies described in Appendix E) the task remains challenging. We suspect that the MI might be contained in regions which are rarely sampled, making the MI hard to estimate. When neural estimators are used, pointwise mutual information learned by the critic does match the ground truth (see Appendix E).
Overall,
Hence, we believe that accurate estimation MI in long-tailed distributions is an open problem. Practitioners expecting long-tailed distributions should approach this task with caution.

---

21
Original Speed 0.5
0.8
CCA
1-
Speed 1.5
estimate [nats)
InfoNCE
Coordnaat
Transfer (Juila)
0.4
NW]
-1
0.2
UNN (R)
0.0
-2
-1
0.0
0.5
1.0
1.5
Coardinate
Speed
Figure 5: Left: Spiral diffeomorphism with different speed parameter v. Right: Performance of considered estimators for increasing speed.
4.3 Invariance to Diffeomorphisms
The benchmark results (Fig. 2) well the experiments with function æ + |x|* sign æ lengthening tails (Fig. 4) suggest that even if ground-truth MI is preserved by a transformation (see Theorem 2.1), the simite-sample estimates may not be invariant. This can be demonstrated using the spiral diffeo- morphism.
We consider isotropic multivariate normal variables X = (x) ~ (N, I2) and Y ~ (0, with Cor(Xi Y) = p. Then, we apply a spiral diffeomorphism 8, R22 - TR? to X, where
cos u/ |||| - -sin 2/ |||r|| sin ul |
8p (x)
cos u |||||||
is the rotation around the origin by the angle w||0||2 (see Fig. 5; for theoretical study of spiral diffeomorphisms see Appendix B). In Fig. 5, we present the estimates for N 10 000, p = 0.8 and varying speed
In general, the performance of the considered estimators deteriorates with stronger deformations. This is especially visible for the CCA estimator, MI cannot be reliably estimated in terms of correlation.
4.4 High Mutual Information
Gao et al. [2015] proves that estimation of MI between strongly interacting variables requires large sample sizes. Poole al. [2019], Song and Ermon [2020] investigate estimation of high MI using multivariate normal distributions in the setting when the ground-truth MI is changing over the training time. We decided use our family of expressive distributions to see the practical limitations of estimating MI. We chose three one-oo parameter distribution families: the 3x3 multivariate normal distribution with correlation p -= Cor(Xi, Yi) = Cor(Xa, Y2), and its transformations by the half- cube mapping and the spiral diffeomorphism with e = 1/3. We changed the parameter to match the desired MI.
In Fig. 6 we see that for low enough MI neural estimators and KSG reliably estimate MI for all considered distributions, but the moment which the estimation becomes inaccurate depends on the distribution. For example, for Gaussian distributions, neural estimators (and the simple CCA baseline) are accurate even up to nats, while after the application of the spiral diffeomorphism, the estimates become inaccurate already at 2 nats.
We that in general neural estimators perform well distributions with high MI. The performance of CCA on the multivariate normal distribution suggests that well-specified model-based estimators may require lower number of samples estimate high mutual information than model-free alternatives (cf. Gao et al. [2015]).
5 Related Work
Existing Benchmarks Khan al. [2007] used one-dimensional variables and an additive noise assumption to evaluate the estimators based histograms, nearest neighbors, and kernels. However,

---

Half-cube
Sparse Gaussian
Spiral
MI extimate [rats]
CCN
DVV
InfoNCE
estmmaee
4-
Transfer (ulia)
Mil
NWJ
(R)
2
LNN (R)
[nais]
[nsts]
[nats]
Figure 6: We select three oee-oonnnee mee-aaraeeter families of distributions varying the MI value (z-axis and dashed line). We use N = 000 data points and n = 5 replicates of each experiment.
Table 1: Comparison between benchmarks.
Khan al. [2007]
Pools al. [2019]
Song and (2020]
Proposed Benchmazk
Guussian Disrihh
Lung Tailed Distrih
Spunse/Dense Inier.
X
High MI
X
Invariance to Homeom.
X
Selt-Consien Consistency
X
X
Lenguage Indep.
X
X
Code Available
X
X
the original code and samples are not available. Song and Ermon [2020] proposed to evaluate neural estimators based on variational lower bounds [Oord al., 2018, Belghazi al., 2018] on image data. However, besides the multivariate Gaussian which they also evaluate, the ground-truth not available. Therefore, they proposed self-consistency checks, which include I(X; Y) for independent X and Y, and preservation of the data processing inequality. Poole et al. [2019] study neural estimators settings employing Gaussian variables, affine transformation followed by axis-wise cubic transformation and investigate the high MI cases. In our benchmark, focus on distributions with known MI and coomattionpreservviii homeomorphisms, evaluate long tail distributions, and compare representative estimators from different classes and platforms. We summarize the differences between the benchmarks in Table 1.
continuous injective mappings such
Causal Representation Learning and Nonlinear ICA Recent work on disentanglement and non- linear independent component analysis (ICA) aims identifying causal sources up an ambiguous transformation [Xi and Bloem-Redde 2022]. Examples include orthogonal transformations [Zim- mermann al., 2021], I, permutations and diffeomorphisms applied along specific axes [Khemakhem et al., 2020], or the whole diffeomorphism group [Von Kügelgen et al., 2021, Daunhawer et al., 2023]. To check whether the learned representations indeed capture underlying causal variables, important to estimators which invariant to the ambiguous transformations specific the model. Our results suggest that such requirements are not met by MI estimators and practitioners should carefully choose the used metrics, they not only theoretically invariant, but also can be robustly estimated in practice.
6 Conclusions and Limitations
Venturing beyond the typical evaluation of mutual information estimators on multivariate normal distributions, provided evidence that the evaluation Gaussian distributions results in overly optimistic and biased results for existing estimators. In particular, a simple model-based approach as
*Khan et al. [2007] consider two one-dimensional r.v. related by additive noise, in which one of the r.v. is parametrized with cubic function. This implicitly lengthens the tails, so we marked this with (V); similarly. Poole et al. [2019] consider cubic transformation estimation of MI considered by Poole et al. [2019] and Song and Ermon [2020] requires changing ground truth MI over training time of a neural estimator. This procedure differs from the usual approaches to estimating MI, marked it with (V).
multivariate normal variable. The experimental setup of

---

CCA is already sufficient if the distribution is (close to) multivariate normal, highlighting the effect of using available prior information about the problem to design efficient estimators. To provide basis for more thorough evaluation, we designed benchmark, compatible with any programming language, with forty general tasks focusing on selected problems, as estimation with sparse interactions, the effects of long tails, estimation in problems with high MI, and (only theoretical) invariance to diffeomorphisms.
Our results suggest that in low- and deedium-dimensione problems the KSG estimator remains a solid baseline, requiring lower number of samples than the alternatives and no parameter tuning. In high-ding sigh-dimensional settings where the interactions are sparse, neural estimators offer an advantage over classical estimators being able more consistently pick up the signal hidden in the high- dimensional space. We further provide evidence that although MI is invariant to continuous injective transformations of variables, in practice the estimates can differ substantially. Appropriate data preprocessing may mitigate some of these issues, but no strategy clearly outperforms the others (see Appendix E).
We hope that the proposed benchmark encourages the development of new estimators, which address the challenges that we highlight in this paper. can be used to assess invariance homeomorphisms, understand an estimator's s behavior in the situations involving distributions of different complexity, low-data behaviour, high mutual information or diagnose problems with implementations. We believe that some parts of the benchmark could also be used to assess whether other estimators of other statistical quantities, such as kernel dependence tests, transfer entropy or representation similarity measures, can be reliably estimated on synthetic problems and if they are robust to selected classes of transformations.
Limitations and Open Problems Although the proposed benchmark covers a wide family of distributions, its main limitation is the possibility only transform the distribution Pxy via transfor- mations f x y to (X) starting from multivariate normal and Student distributions. In particular, not every possible joint distribution of this form and extending the family of distributions which have known MI and allow efficient sampling is the natural direction of continuation. We are, however, confident that due to the code design will be easy to extend the benchmark with the new distributions (and their transformations) when they appear.
Estimation of information in multivariate Student distributions remains a challenge for most estimators. Although we provide the evidence that distributions with longer tails pose a harder challenge for considered estimators, shortening the tails with asinh transform did not alleviate all the issues. In neural estimators this can be partly explained by the fact that the critic may not learn to approximate pointwise mutual information (up to an additive constant) properly, as demonstrated Appendix E, but the question of what makes the beighborhood-basee estimator, KSG, fail, remains open.
As noted, unless strong inductive biases can be exploited (as illustrated for CCA), estimation of high MI is an important challenge. An interesting avenue towards developing new estimators can be by incorporating stronger inductive bias, which could also take form of development of better data normalization strategies and, in case of neural estimators, critic architectures. Better normalization strategies may make the estimators more invariant to considered transformations.
Acknowledgments and Disclosure of Funding
We would like to thank Craig Hamilton for the advice on scientific writing and help with the abstract. FG was supported by the Norwegian Financial Mechanism GRIEG-I grant operated by Narodowe Centrum Nauki (National Science Centre, Poland) 2019/34/H/NZ6/06699 PC and AM were supported by fellowship from the ETH AI Center.
10

---

References
Ziqiao Ao and Jinglai Li. Entropy estimation via normalizing flow. Proceedings of the AAAI Conference on Artificial Intelligence, 36(9):9990-9988, 2022.
R.B. Arellano-Vall Valle, J.E. Sentreeaa-Reyess and M.G. Genton. Shannon entropy and mutual informa- tion for multivariate skew-elliptical distributions. Scandinavian Journal of Statisties, 49:42-62, 2013.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International conference machine learning, pages 531-540. PMLR, 2018.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Miihe and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL bttt:://ithin com/ google/jax.
Peter Bühhmann and Sara van de Geer. Statistics for Trigh-Dimensione Data: Methods, Theory and Applications. Springer Publishing Company, Incorporated, 1st edition, 2011. ISBN 3642201911.
Vincent Cabeli, Louis Verny, Nadir Sella, Guido Uguzzoni, Marc Verny, and Hervé Isambert. Learning clinical networks from medical records based on information estimates in mixed-type data. PLOS Computational Biology, 1007866,, 2020.
Anthony Carapetis. Rotating spherical shells doesn't change volume.
Mathemat-
ics Stack Exchange, 2022. URI. htttpps//aat..ssacceechnggg www.stkeeccanne.coo... cttpc//////////////////// //4002944
(version: 2022-07-26).
Christopher J Cellucci, Alfonso M Albano, and Paul E. Rapp- Statistical validation of mutual
information calculations: Comparison of alternative numerical algorithms. Physical review E, 71 (6):066208, 2005.
Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. Wiley-Interscience New York, 2006.
Georges A Darbellay and Igor Vajda. Estimation of the information by an adaptive partitioning of the observation space. IEEE Transactions on Information Theory, 45(4):1315-132) 1321, 1999.
Carsten O Daub, Ralf Steuer, Joachim Selbig, and Sebastian Kloska. Estimating mutual information using b-spline functions an improved similarity measure for analysing gene expression data. BMC bioinformatics, 5(1):1-12, 2004.
Imant Daunhawer, Alice Bizeul, Emanuele Palumbo, Alexander Marx, and Julia E Vogt. Identifiability results for multimodal contrastive learning. In The Fleventh International Conference on Learning Representations, 2023. URL hwwph:tt////ttt https::/opeereviies www//forumiiiee -
Shuyang Gao, Greg Ver Steeg, and Aram Galstyan. Efficient estimation of mutual information for strongly dependent variables. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), pages 277-286. PMLR, 2015.
Weihao Gao, Sreeram Kannan, Sewoong Oh, and Pramod Viswanath. Estimating mutual information for cissrrete-ooninnne mixtures. In Advances in Neural Information Processing Systems, pages 5986,5997, 20.7a.
Weihao Gao, Sewoong Oh, and Pramod Viswanath. Density functional estimators with k-nearest neighbor bandwidths. In IEEE International Symposium Information Theory (ISIT), pages 1351-1355. IEEF, 2017b.
Weihao Gao, Sewoong Oh, and Pramod Viswanath. Demystifying fixed k-nearest neighbor informa- tion estimators. IEEE Transactions Information Theory, 64(8):5629-5661, 2018.

---

Frederic Grabowski, Pawel Czyž, Marek Kochańczyk., and Tomasz Lipniacki. Limits to the rate of information transmission through the MAPK pathway. Journal of The Royal Society Interface, 16 (152):20180992, 2019, doi: 10. 10.008//rsf.200 2tt///ss..201..0000 2.09997.. URL httpttppserrrylllsseeeeiiblbbaee org/doi/abs/ w09./rsff....////. 2018.0792.
Søren Hauberg. Only Bayes should learn a manifold (on the estimation of differential geometric structure from data), 2019.
Caroline M Holmes and Ilya Nemenman. Estimation of mutual information for real-valued data with error bars and controlled bias. Physical Review E, 100(2):022404,, 2019.
J. Kay. Feature discovery under contextual supervision using mutual information. In IJCNN International Joint Conference on Neural Networks, volume 4, pages 79-84, 1992. doi: 10.1 109/ UCCNN.992.2227225.
A. Kechris. Classical Descriptive Set Theory. Graduate Texts in Mathematics. Springer New York, 2012. ISBN 9781461241904 URL https: http::://booss google.. httplle..cc//ee ch/books? "onooks?ddd/RSBBAAQPP
Dustin Keys, Shukur Kholikov, and Alexei A. Pevtsov. Application of mutual information methods in time distance helioseismology. Solar Physics, 290 6711, Mar 2015. ISSN 1573-093X. 0050--0650-..
doi: 10.1007/s1 1207-015-0650-y. URL
Shiraj Khan, Sharba Bandyopadhyay, Auroop R Ganguly. Sunil Saigal, David J Erickson III, Vladimir Protopopescu, and George Ostrouchov. Relative performance of mutual information estimation methods for quantifying the dependence among short and noisy data. Physical Review E, 76(2):
026209, 2007.
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ICA: a unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207-2217. PMLR, 2020.
Jussi Klemelä. Multivariate histograms with data-dependent partitions. Statistica sinica, pages 159-176, 2009.
Mike Koeman and Tom Heskes. Mutual information estimation with random forests. International Conference on Neural Information Processing, pages 324-531 Springer, 2014.
L. F. Kozachenko and N. N. Leonenko. Sample estimate of the entropy of a random vector. Problemy Peredachi Informatsii, 23:9-16, 1987.
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical Review E, 69(6):0661 138, 2004.
Thanard Kurutach, Aviv Tamar, Ge Yang, Stuart J Russell, and Pieter Abbeel. Learning plannable representations with causal infogan. Advances in Neural Information Processing Systems, 31, 2018.
Bo Li, Yifei Shen, Yezhen Wang, Wenzhen Zhu, Dongsheng Li, Kurt Keutzer, and Han Zhao. Invariant information bottleneck for domain generalization. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7399-7407, 2022.
Warren M Lord, Jie Sun, and Erik M Bollt. Geometric k-nearest neighbor estimation of entropy and mutual information. Chaos: An Interdisciplinary Journal of Nonlinear Science, 28(3):033114 14, 2018.
Chien Lu and Jaakko Peltonen. Enhancing nearest neighbor based entropy estimator for high dimensional distributions via bootstrapping local ellipsoid. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2013-5020, 2020.
Joseph Lucas, Carlos Carvalho, Quanli Wang, Andrea Bild, Joseph R. Nevins, and Mike West. Sparse Statistical Modelling in Gene Expression Genomics, page 155-176. Cambridge University Press, 2006. doi: 10.1017/CB097805 1584589,0000 ,009.
Alexander Marx and Jonas Fischer. Estimating mutual information via geodesic knn. Proceedings of the SIAM International Conference on Data Mining (SDM), pages 415-423, SIAM, 2022.
12

---

Alexander Marx and Jilles Vreeken. Testing conditional independence on discrete data using stochastic complexity. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), pages 495-505, 2019.
Alexander Marx, Lincen Yang. and Matthijs van Leeuwen. Estimating conditional mutual information for discrete-continnouus mixtures using multi-dimensional adaptive histograms. In Proceedings of the SIAM International Conference Data Mining (SDM), pages 387-395, 2021.
David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 875-884. PMLR, 26-28 Aug 2020. URL https:: httttp::/prroceednng.... mlr. wwwppprssssvooee 1ea10//mcaaleste20000 html.
Octavio César Mesner and Cosma Rohilla Shalizi. Conditional Mutual Information Estimation for Mixed, Discrete and Continuous, Data. IEEE Transactions on Information Theory, 2020.
Young-Il Moon, Balaji Rajagopalan, and Upmanu Lall. Estimation of mutual information using kernel density estimators. Phys. Rev. E, 12:2218-2321, Sep 1995. doi: 10. 10.11033Phyys URL https: /:l::://iiiiaa/. wtgp/oo.g.oo........ http://........//pppeeee 1ww./////y/ee.....
Kevin P. Murphy. Probabilistic Machine Learning: Advanced Topics. MIT Press, 2023. URL
httt:://probml.gtttee s.tthub.i./book..
Mölder, KP Jablonski, B Letcher, MB Hall, CH Tomkins-Tinch, V Sochat, J Forster, S Lee, SO Twardziok, A Kanitz, A Wilm, M Holtgrewe, S Rahmann, S Nahnsen, and Köster. Sustainable data analysis with snakemake [version I; peer review: I approved, approved with reservations]. F1000Research. 10(33), 2021. doi: 10. 0012668/1000tesaaree222.2...
R.B. Nelsen. An Introduction Copulas. Springer Series in Statisties, Springer New York, 2007. ISBN 9780387286785.. URL attps://bookes google. ch/ botoookssidd B30NTT5BBO....
XuanLong Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by penalized convex risk minimization. J. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems, volume 20. Curran As- sociates, Inc., 2007. URL
neurips www..p..cccppppr..eeee.... wwwps///eppeee/00///... Paper pdf.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. F-gan: Training generative neural samplers using variational divergence minimization. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, page 271 271-279, Red Hook, NY, USA, 2016.
Curran Associates Inc. ISBN 9781510838819.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprini arXiv: 1807.03788 2018.
Liam Paninski. Estimation of entropy and mutual information. Neural Computation, 15(6): 1253, 2003.
Liam Paninski and Masanao Yajima. Undersmoothed kernel entropy estimators. JEEE Transactions Information Theory, 54(9):4384 43888 2008.
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830. 2011.
M.S. Pinsker and A. Feinstein. Information and Information Stability of Random Variables and Processes. Holden-Day series in time series analysis. Holden-Day, 1964. ISBN 978081 97808162666 6268047.
Stephen M Pizer, E Philip Amburn, John D Austin, Robert Cromartie, Ari Geselowitz, Trey Greer, Bart ter Haar Romeny, John B Zimmerman, and Karel Zuiderveld. Adaptive histogram equalization and its variations. Computer Vision, Graphics, and Image Processing, (133:355-363 1987.
13

---

Y. Polyanskiy and Y. Wu. Information Theory: From Coding to Learning. Cambridge University Press. 2022. Book draft.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pages 5171- 5180. PMLR, 2019.
Ruggero Ragonesi, Riccardo Volpi, Jacopo Cavazza, and Vittorio Murino. Learning unbiased repre- sentations via mutual information backpropagation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2729-2738, 2021.
Jangir Selimkhanov, Brooks Taylor, Jason Yao, Anna Pilko, John Albeck, Alexander Hoffmann, Lev Tsimring, and Roy Wollman. Accurate information transmission through dynamic biochemical signaling networks. Science, 346(6215):1370-13333 2014.
Bernard W Silverman. Density estimation for statistics and data analysis. Routledge, 2018.
Victor Solo. On causality and mutual information. In 2008 47th IEEE Conference Decision and Control, pages 4939 4944. IEEE, 2008
Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estimators. In International Conference on Learning Representations, 2020.
Joe Suzuki. An estimator of mutual information and its application independence testing. Entropy. 18(4):109, 2016.
Nassim Nicholas Taleb. Statistical consequences of fat tails: Real world preasymptotics, epistemology. and applications, 2020. URL pesetts://arxan www.ariww.i..oaaaaeeeeeeeeeeeeee /tgg/ba//000..11 10488.
Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. In International Conference on Learning Representations, 2020.
Shinsuke Uda. Application of information theory in systems biology. Biophysical Reviews, 12 (2):377-384, Apr 2020. ISSN 1867-2469. doi: 10.1007711251--000000666 URL https: //doi.org/10 1007/ 2/000//12200 satsss11111111 020 222000665-w..
Gregory Valiant and Paul Valiant. Estimating the unseen: an n/ log(())-sampl) estimator for entropy and support size, shown optimal via new clts. In Proceedings of the annual ACM symposium on Theory of computing (STOC), pages 685-694, 2011.
Martin Vejmelka and Katerina
wewellkkvv-SShiiillee Mutual information estimation in higher dimensions: A speed-up of a k-nearest neighbor based estimator. In International Conference on Adaptive and Natural Computing Algorithms, pages 790-797. Springer. 2007.
Nguyen Xuan Vinh, Jeffrey Chan, and James Bailey, Reconsidering mutual information based feature selection: A statistical significance view. In Proceedings of the AAAI Conference on Artificial Intelligence, 2014.
Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34:16451-16467, 2021.
Quanhan Xi and Benjamin Bloem-Reddy. Indeterminacy in latent variable models: Characterizaaioo and strong identifiability. arXiv preprint arXiv:2206.0080) 2022.
Alexander L. Young, Willem van den Boom, Rebecca A. Schroeder. Vijay Krishnamoorthy, Karthik Raghunathan, Hau-Tieng Wu, and David B. Dunson. Mutual information: Measuring nonlinear dependence in longitudinal epidemiological data. PLOS ONE, 18(4):1-16, 04 2023. doi: 13.1/ journal.pone, 02849044 URL https: http://ooo.oo/o/////// 1371/journal pone. 0284904.
Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey, 2021. URL https:/ arxiv org/abst abs/aaas//111 22100. 04596.
14

---

Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In International Conference on Machine Learning, pages 12979-12990. PMLR, 2021.
15

---

Appendix
Table of Contents
A Invariance to Continuous Injective Mappings
17
Spiral Diffeomorphisms
18
C Overview of Estimators
20
Additional Information on Benchmark Tasks
22
D.I Task Descriptions
22
D.2 Covariance Matrix Family
23
D.3 Technical Results
24
Additional Experiments
25
E.1
Pointwise Mutual Information
25
E,2 Sample Size Requirements
26
E.3 Preprocessing
27
E.4
Hyperparameter Selection
27
Visualisation of Selected Tasks
29
G Author Contributions
30
16

---

A Invariance to Continuous Injective Mappings
In this section, we provide a proof for Theorem 2.1, formally proving the invariance of mutual information with respect to continuous injective mappings. First, we recall the standard definitions from Chapters and of Pinsker and Feinstein [1964].
Definition A.1. Let (Ω2 A) be measurable space. A family of sets & S A is partition of Ω if for every two distinct E and E3 in it holds that Ej n E2 = Ø and that U Q. If finite, we call it finite partition.
Definition A.2 (Mutual Information). Mutual information between r.V. and defined as Px x Fj) log Pxy x Ps) Px (x(
I(X;Y)
sup Σ E,J
where the supremum is taken over all finite partitions & (F1, F2.... of X and F {Fi, F2, of y with the usual conventions 0 log 0 = O and 0 0og00/0 00
Remark A.3. If Pxy is absolutely continuous with respect Px @ Py (e.g., X and Y are finite r.v.). mutual information is equal to the following Kullback-Leite Leibler divergence:
IX;Y) DEL (Pxy Px Pr)
log dPxy,
where f the Radon Ridon-Nikodym derivative / = dP / Px @ Pr). If the absolute continuity does not hold, then I(X;Y) Y)) = +oo [Pinsker and Feinstein, 1964, Theorem 2.1.2].
Theorem 2.1. Let X, X², y and y be standard Borel spaces (e.g.. smooth manifolds with their Borel r-algebras) and f: X + X' and y:y- y be continuous injective mappings. Then, for every X-valued r.v. and Y-valued r.v. Y it holds that I(X;Y) = I(J(X):g(Y))
Proof. First, note that continuous mapping between Borel spaces is measurable, random variables f(X) and gY) are well-defined.
Then, observe that it suffices to prove that
I(f(X): I(X;)).
It is well-known that we have (ff(X);; Y) S I(X;Y) for every measurable mapping f. so we will need prove the reverse inequality. Recall form Definition A.2 that
Σ PXy x Fj) log Pxy x PJ) E.F
I(X; Y)
sup
(x
where the supremum is taken over all finite partitions of X' and F of
The proof will go as follows: for every finite partition E of X we will construct a finite partition of X', which (together with the same arbitrary partition F of )) will give the same value of the sum. This will prove that I(f(X); Y) 2 I(
Take any finite partitions { (E Fn of X' and (Fi,..., FmJ of y. First, we will show that sets
Go X' \ \f(x)
G f(Ei)
G2 f(E2)
form a partition of X".
It is easy to that they pairwise disjoint is injective and it is obvious that their union is the whole X". Hence, we only need to show that they are all Borel.
For > 1 the Lusin-Suslin theorem [Kechris, 2012, Theorem 15.1] guarantees that G; is Borel (continuous injective mappings between Polish spaces map Borel sets to Borel sets). Similarly, (() is Borel and hence Go = X' f(() is Borel well.
17

---

Next, we will prove that
(xy(B,XP)) (x(E))P((P3)
a m
PIOY (G, x Fj)log log Prxx Fj)
Px (xx x P3) log
ΣΣ
Pycxy (y:)
=1
i=
= Note that - Px(1- = Px = 0, that we can ignore the terms with i = 0.
For 2 1 have
P)x)( (G.(f (x(() (x(E,)
as f is injective. Similarly, f ly is injective and maps Fg F, onto G; Fj. 80
PJxXY(,, P2) Pxx (U/ x 1y) x F2)) Pxy(Ea Bÿ).
This finishes the proof.
B Spiral Diffeomorphisms
In this section will formally define the spiral diffeomorphisms and prove that they preserve the centred Gaussian distribution o'))," They were inspired by the "swirling construction". discussed by Hauberg [2019].
Definition B.1. Let On)) be the orthogonal group and R - O(n) be any smooth function. The mapping
R= R
given by f(x) - >[1==111)))) will be called the spiral.
Example B.2. Let so(n) be the vector space of all skew-symmetric matrices. For every two matrices R (n) and A E (n) and any smooth mapping g: IR - R can form a path
(()) = Rexp(o(t)A)
In particular, = In and g(t) = yields the spirals defined earlier.
From the above definition it is easy prove that spirals diffeomorphisms, We formalize it by the following lemma.
Lemma B.3. Every spiral a diffeomorphism.
Proof. Let y be the path leading to the spiral f. As matrix inverse is smooth function, can define a smooth function l((x) = 7 (1x=||²)- x.
Note that [2) || orthogonal transformation y ([|=|??) preserves distances.
Then
h(f(=)) = (IIf(=))1P)- f(x)
T (11x119) -1 f(x)
=7 (1|=113)- -1 7 (1]=]11))
Analogously, we prove that f h also is the identity mapping.
L
Finally, we will give the proof that spirals leave the centred normal distribution (00,, o21,) invariant. This will follow from slightly stronger theorem:
Theorem B.4. Let P be probubility measure on Inn with smooth probability density function which factorizes as:
p(z) (1x]
18

---

for some smooth function u: I- + R. Then, every spiral J: IR" R. preserves the measure P:
f
where 14 P is the pushforwarl measure.
In particular, for P N0 1m) (a centered isotropic Gaussian distribution), if NO o2 then f(X) ~ o0,
Proof. As have already observed, for every I it holds that [1f(=)]| ||=|| ? so the PDF evaluated at both points is equal
p(x) p(f()))
Hence, we need to prove that for every point z E RR,, det f'(x) = 1, where f2 is the Jacobian matrix of f evaluated at the point a.
The proof of this identity was given by Carapetis [2022], but for the sake completeness we supply variant of it below.
First, we calculate the Jacobian matrix:
0f((z)
dx, Σ [[|2c|| Da.k
Dzk
By(112) 1
Σ<(1<=|1>))
Σ J/K:
Ox,
Ox,
=>(1==] 1ssp + Σ 3hk 2x, 7 (11x=1
=>(1<<|1)) +2 2((00=1111)0 Daaer
Hence,
(((x) >1|==11) + 2 (1 Dan""
Now fix point z and consider the matrix (3 E (n)) Define a mapping + >(1==12)-11(0 f(v).
As we have already observed, multiplication by <[1[=||))-1 does not change the value the PDF also cannot change the det f'(x) value because >(1==112)- is linear mapping, its derivative is just the mapping itself and because 7 ( is an orthogonal matrix, we have det >(1x=|)) 1. Hence, without loss of generality we can assume that ([|==|1²) In and
fx) La + 2AxrT
where = (1(1= E so(n) is ntew-symmetric matrix.
If œ 0 we have del ['(x)) = det /nl 1. In the other case, we will work in coordinates induced by an orthonormal basis e1, . en which the first basis vector is e = a/||</] Hence,
(x) In + 2/ 2||x||2Ae1ee
For i 2 2 we have of 0 and consequently
f'(x) = e, + 2||z|| e C₁.
This shows that in this basis we have
0
0
02
0
0
(x)
1
0
0
1
where the block on the bottom right is the identity matrix -n-1 and det f((x) a1.
19

---

The first column is formed by the coordinates of
f((x)11 =1 + 2||x|||
and we have
a - - 0
as A E so(n) is skew-symmetric.
C Overview of Estimators
There exists a broad literature on (conditional) mutual information estimators ranging from estimators on discrete data [Paninski, 2003, Valiant and Valiant, 2011, Vinh et al., 2014, Marx and Vreeken, 2019], over estimators for continuous data [Kraskov et al., 2004, Darbellay and Vajda, 1999, Belghazi et al., 2018] estimators that can consider discrete-continuous mixture data [Gao et al., 2017a, Marx et al., 2021, Mesner and Shalizi, 2020J. Here, we focus on the most commonly used class, which also finds applications in representation learning: estimators for continuous data. For continuous data, the most widely propagated classes of estimators are histogram or partition- nearest neighbor- based density estimators, kernel methods, and neural estimators. In the following, we briefly discuss a representative selection of estimators from these classes, as well as model-based estimation.
estimators,
Histogram- Estimators A natural way to approximate mutual information, as stated in Definition A.2, is by partitioning the domain space of X and Y into segments and estimate the densities locally. This approach also known as histogram density estimation, where the probability mass function is approximated locally for each hyper-rectangle E; by Px (E,) = "//N the volume of E, and n; refers to the number of data points residing in E,
Vol," where Vol, is
To estimate mutual information, explore the space of allowed partitionings and select the parti- tioning that maximizes mutual information [Darbellay and Vajda, 1999, Cellucci et al., 2005]
Σ Pxy x F2) log Pxy x F2) E,
EXY Disc
sup
(x(E.)P((E))
Many estimators follow this idea. Darbellay and Vajda [1999] use adaptive partitioning trees and propose to split bins until the density within bin is uniform, which they check with the x² test. Daub et al. [2004] use B-Splines to assign data points to bins, and other authors use regularization based on the degrees of freedom of partitioning to avoid overfitting [Suzuki, 2016, Cabeli et al., 2020, Vinh et al., 2014, Marx et al., 2021].
A well-known property of sestogram-based estimators is that they are consistent for N + co if the volume of the largest hypercube goes zero, while the number of hypercubes grows sub-linearly with the number of samples, N (see e.g. Cover and Thomas [2006]). For low-dimensional settings, histogram density estimation can provide accurate results, but the strategy suffers from the curse of dimensionality [Pizer et al., 1987, Klemelä, 2009].
In our evaluation, we resort partitioning estimators based on fixed number of grids (10 equal-width bins per dimension), since more recent partitioning- based estimators focus on estimating conditional MI and only allow X and Y be one-dimensional [Suzuki, 2016, Marx et al., 2021, Cabeli et al., 2020], or do not have publicly available implementation [Darbellay and Vajda, 1999].
ANN Estimators An alternative histogram- based methods is use k-nearest neighbor (kNN) distances to approximate densities locally. Most generally, kNN estimators express mutual informa- tion the sum HX) + (Y) HX,, Y), which can be done under the assumptions discussed in Remark A.3, and estimate the individual entropy terms as
N k(#.)/N
.N))
1 N
Σ
log
Vol:
where k(r,) the number of data points other than x in a neighborhood of r,, and Vol, the volume of the neighborhood [Kozachenko and Leonenko, 1987]. The quality of the estimator depends two
20

---

factors: i) the distance metric used to estimate the nearest neighbor distances, and ii) the estimation of the volume terms.
The most well-known estimator is the Kraskov, Stögbauer and Grassherger (KSG) estimator [Kraskov al., 2004]. In contrast to the first kNN-based estimators [Kozachenko and Leonenko, 1987], Kraskov et al. [2004] propose to estimate the distance to the k-th neighbor pt. for each data point i only on the joint space of Z X V, with
(zi =j) = max{Dx xz), (yh,
where Dx and Dy can refer to any norm-induced metric (e.g.. Euclidean E2. Manhattan é or Chebyshev ) To compute the entropy terms of the sub-spaces X' and we count the number of data points with distance smaller than Pt, i.e. we define Mlr,i as
ng. K Dx(x,, < 2
Pi.k i (
(1)
and specify yy accordingly. As a consequence, all volume-related terms cancel and the KSG estimator reduces to
FXY KSG.N = =b())+((()))
Σ ((n,
The diagamma function &i was already introduced by Kozachenko and Leonenko [1987] to correct for biases induced by the volume estimation. For detailed exposition of the issue, we refer to Kraskov et al. [2004, Sec. II.A].
There exist various extensions of the classical KSG estimator. When selecting the Lo norm as distance measure for Dx and Dy, E v can be efficiently computed with the K-d trie method [Ve- jmelka and Haváčkova-Schiinller 2007]. Other approaches aim to improve the estimation quality by explicitly computing the volume terms via SVD [Lord al., 2018] PCA [Lu and Peltonen, 2020], or use manifold learning [Marx and Fischer, 2022] or normalizing flows [Ao and Li, 2022] to estimate the nearest neighbors more efficiently high- high-dimensional settings.
Kernel Estimators A smaller class of estimators builds upon kernel density estimation (KDE) [Moon al.. 1995, Paninski and Yajima, 2008]. Classical approaches compute the den- sity functions for the individual entropy terms via kernel density estimation, which is well-justified under the assumption of Pxy being absolutely continuous with respect to Px & Py (ef. Remark A.3). i.e. the density for a dd-imensional random vector X is approximated as
p(x)
Σ i=1
Nha
where h is the smoothing parameter or bandwidth of Gaussian kernel [Silverman, 2018).
Moon al. [1995] propose estimate mutual information using a Gaussian kernel, for which the bandwidth can be estimated optimally [Silverman, 2018]. Similar to histogram-based approaches, KDE-based estimators with fixed bandwidth h are consistent if Nh - xo as h 0.
more recent proposal, LNN [Gao et al., 2017b], uses a data detendepnndnnt bandwidth based on nearest-neighbor distances, avoid fixing h globally.
Neural Estimators Neural network approaches such as MINE [Belghazi et al., 2018] use a varia- tional lower bound mutual information using the Varadhan theorem: let f: X' x y R be a bounded measurable function and
Vx(XY = sup Exx (S(X log Erx (exp ((X
Under sufficient regularity conditions on Pxy, I(;;Y) is equal to (X Y), which is evaluated over all bounded measurable mappings f. In practice, the right hand side is optimized with respect
In the original approach, the L... used, and Gao et al. [2018] derives theoretical properties of the estimator for the Le norm.
'The digamma function is defined as v((z)= log /dxx)/dr. It satisfies the recursion (x 1) = +(x)+ with (1) = -, where 7 zs 0.577 is the Euler Mascheroni constant.
21

---

to the parameters of given parametric predictive model f. as neural network. Approaches following this idea also count as discriminative approaches since the neural network is trained to distinguish points from the joint and marginal distribution. The literature on variational lower bounds of mutual information is growing [Oord al., 2018, Song and Ermon, 2020] and the idea of using discriminative approach to estimate mutual information has also been explored by Koeman and Heskes [2014], who used random forests. For an excellent theoretical overview of different neural estimators we refer to Poole al. [2019].
Model-based Estimators Finally, if one assumes that Pxy is member of a family of dis- tributions (xy(O) with known mutual information, one can estimate 0 basing on the data ((11,11,,.... ...,.... (JN) and calculate the corresponding mutual information. For example, if one assumes that there exist jointly multivariate normal variables X (X,...., X,, Ei...., E,....,,mm-!! and Y = (Y,,..., Yy, Ei...., En-r) such that the cross-covariance terms are zero apart from Cov Cov(X,,Xe Y,). and that X AX + a and Y = BÝ + for invertible matrices A and and vectors E Th and b R", then the mutual information can be found as
I(X;Y)= = Σ log V Cor(X,, Y,)2
The model parameters A, B, a, and b, as well as the pairs of corresponding r.v. X, and Y,,, can be found by maximising likelihood using canonical correlation analysis (CCA) [Murphy, 2023, Ch. 28), which has been previously explored by Kay [1992].
D Additional Information on Benchmark Tasks
D.1 Task Descriptions
Bimodal Variables Consider r.v. X distributed according Gaussian mixture model with CDF F = W1 F1 + W2F2, where F1 and F2 are CDFs of two Gaussian distributions and wi and W2 are positive weights constrained w1 W2 1. The CDF is everywhere positive, continuous, and strictly increasing. Hence, it has an inverse, the quantile function, F-1: (0, 1) - R, which is continuous and injective well (see Lemma 1. Thus,
define X' FX)(X), where X ~ Uniforrm (0, 1) and I(X;)) I(X;Y) for every r.v. Y. We generated two CDFs of Gaussian mixtures and implemented a numerical inversion in JAX [Bradbury et al., 2018] defining both X and Y' to follow a bimodal distribution.
For the experimental values, used
(x = +
Fy = 0.5F(x + 0.5 3),
where the CDF the standard normal distribution N/ 1).
Wiggly Mapping
As the "wiggly mapping" we used
w (I) + 0.4 0.2 + 1) 0.03 sin -2.5) Wy (v) = y - 0.4 sin (0.4y) + 0.17 si.31.3y + 3.5) + 0.02 (in..n3n.3 2.5)
and we used visualisations to make sure that they do not heavily distort the distribution.
Additive Noise For the additive noise model of Y X N where X Uniform(0, 1) and N ~ Uniform(-ee ss,s) it holds that
- log(22) if E < 1/2 (4s) otherwise
I(XYY)
implying that I(X;Y) 1.7 for 0.1 and I(X;Y) 0.3 for 0.75.
22

---

Swiss Roll Embedding Consider bivariate distribution Pxy with uniform margins X ~ Uniform(0, 1) and Y Uniform 1) and IX; 0.8. The Swiss roll mapping is an em- bedding i: (0, 1)2 + R given by i(zr = (e(r), where e: (0, 1) + R² is given by
3 (1 + (1+
e(x)) 21 (M(x)) cos (x), ((x) sin (z))
[xx) 2
Note that Px does not have PDF with respect to the Lehesgue measure RR3 We visualise the Swiss roll distribution in Fig. 1.
Spiral Diffeomorphism phism consult Appendix B.
For the precise definition and properties of the general spiral diffeomor-
In our benchmark we applied the spiral x + exp(uAl|=<| to both X and Y variables. We used an m X m skew-symmetric matrix Ax with (Ax) 12 = (Ax)22 = and an n x n skew-symmetric matrix Ay with (Ay)23 = (ay))32 = 1. Each of these effectively "mixes" only two dimensions.
The speed parameters ux and by were chosen as ox 1/m and by 1/n to partially compensate for the fact that if X ~ (O, 1m), then ||x|| ~ Xin and has mean m.
Multivariate Student Let Ω be an (m + n2) (m + n) positive definite dispersion matrix and be a positive integer (the degrees of freedom). By sampling an (m + +n) n))dimennsional random vector (X, ) ~ (0, 12) and a random scalar V ~ x one can define rescaled variables X = X vD/U distributed according the multivariate Student distribution. For and Y = Y //UU, which
// = this reduces to the multivariate Cauchy distribution (and the first two moments are not defined), for 2 the mean exists, but covariance does not, and for v 2 both the mean and covariance exist, with Cov(X, Y)= = o S so that the tail behaviour can be controlled by changing v. In particular, for v >>>>> because the concentration of measure phenomenon has most of its probability mass around / and the variables (X,Y) Y') can be well approximated by (X, Y).
Arellano-Vaal Valle et al. [2013] proved that I(X;Y) can be computed via the sum of the mutual information of the Gaussian distributed basis variables and correction term, i.c. I(X;Y) I(;;)) + c(v, m, n),, where
c(v, m. m)) = f (v) + (v+m+n) -f (f f(x) = log r
F
1/
x
2
2
and di is the digamma function.
Contrary the Gaussian case, for Ω In the mutual information I(X;)) c(v, m. n)
non-zero, as U provides some information about the magnitude. In the benchmark we use this dispersion matrix to quantify how well the estimators can the information contained in the tails, rather than focusing on estimating the Gaussian term.
D.2 Covariance Matrix Family
We generated jointly multivariate normal variables X = (XI,..., Xm) and Y = =Vi,... Yn) with the following procedure.
Consider i.i.d. Gaussian variables
Fn, E' K.... E'my F/ K:..... Fy ~ N(O, 1). Uan, Ux, Uy, 2i.... ZK, E. .... Em, FI,....
Now let ex, 7x, M a, Bx 3r A E R be hyperparameters and for 1 = 1,2...., K define
Xi = E + aUAll + xxxx + AZ1, Yi = sy + aUail + By + XZ1,
for = K + ,,,,,,,,77 m define
X1 = Ex F + aUall + Bx + 7x FM
and for 1 K + 1,..,, define
Yi = Fy Fl + aUAهl + By Uy + my F.
The interpretation of the hyperparameters is the following:
23

---

1. a contributes to the correlations between all the possible pairs of variables.
2. Sxx contributes to the correlations between the X variables.
3. By contributes the correlations between the variables.
4. À gives additional "interaction strength" between pairs of variables X; and Yi for I
1, I.
5. cx and control part of the variance non-shared with any other variable.
6. dx and !/y can be used to increase the variance of X, and Y for 1 to match the variance of variables & K due to the X.
Every term of the covariance matrix is easy to calculate analytically:
Cov(X;, = a² + 3 + 1 ]1 + 1 - K] + 1 > KJ)
Cov(Y,, = = a² + =aa + 1/ 51 + 1 2 K] 1/1 K
Cov ,X,, Y2) a + X j
In the following, analyse some special cases.
Dense Interactions Consider À 1|x Ir Bx By 0 and write ex EY. We have
Cov(X,, Cov(Y, a² + = =
Cov(Y,, Yj) = a²
Hence, between every pair of distinct variables we have the same correlation a'+e"
Sparse Interactions Consider 0, = sy, 7xx n|y = À and Bx By =0
Cov(Xi, Cov Yj) 1 [s x²)
Yx) Aa = j| < K]
All the variables have the same variance, but the covariance structure is now different. Between (distinct) X, and Xj the correlation is zero and similarly for (distinct) and Y,,
Correlations between X; and Y, are zero unless < K, when Cor(X, Yi) X²/(E² X²).
Interpolation In Section 4 we discussed two-stage interpolation from dense matrices to sparse matrices: in the first stage decreased increasing À the same time to keep the mutual information constant. In the second stage we decrease the number of interacting variables K increasing X. The selected covariance matrices from this two-stage process are visualised in Fig. 7.
D.3 Technical Results
In this subsection we add the technical results, related to the copula theory [Nelsen, 2007] and uniformization of random variables.
Lemma D.I. Let X be real-valued with strictly increasing, positive everywhere, and continu- ous CDF F. Then
F:R + (0,
is homeomorphism.

---

a=1.2
a=0.9
a=0.5
Y
X
K=10
K=5
K=1
Figure Interpolation between dense and sparse covariance matrices.
Proof. As F is CDF we have F(R) C [0, 1]. Note that 0 is excluded from the image as F is positive everywhere and is excluded F strictly increasing. Hence, F(R) (0, 1). Using continuity, strict monotonicity, and
lim F(x) = O, lim F(x) = ++0
h-
we note that F' is bijection onto (0, 1). Analogously, we prove that
F((a, b)) (F(a),
for all < b, so F is open map. As continuous bijective open map F is homeomorphism.
Lemma D.2. Let X be real-valued with CDF F which is continuous, strictly increasing and positive everywhere. Then, F(X) is uniformly distributed on (0, 1).
Proof. Our assumptions on F imply that it a continuous bomeomorphism R - -0,11 (0,1).
Let now (0, We have
P(F(X) S P(X < FF ( (() y.
E Additional Experiments
E.1 Pointwise Mutual Information
To understand why neural estimators are not able to estimate the mutual information contained in the Student distribution, we decided to qualitatively study the learned critic.
In the NWJ approach [Nguyen al., 2007], for N + data points the optimal critic f(x y) is given by
[(x, v) = 1 + PMI(x, v):
where
Pxy (z. v)
PMI(x, v) log px (x) py
the pointwise mutual information [Pinsker and Feinstein, 1964, Ch. 2] and pxy, Px and py are the PDFs of measures Pxy. Px and Py. respectively.
25

---

Stadent u
Student ym?
Qtudint pm
Shudent p
Student ym
hormal
Figure 8: Qualitative comparison between ground-truth pointwise MI and learned critic for bivariate Student and normal distributions.
HEATE
A501 300
190
100
140
109
183
530
13M
SCE
569
306
500
Thaceer JUR0
RANN
Figure 9: Minimal number of samples needed for the mutual information estimate to fall between 2/3 and 3/2 of the true MI. The minimal number of samples tested is 100, the maximal 10 000. The entry > 10k means that the desired accuracy never achieved. For neural estimators 1.00 is the minimal possible number of samples, since we use a 50% train-test split and a batch size of 256.
We trained NWJ estimator with the default hyperparameters on N 5 000 data points and visualised the critic predictions (with 1 subtracted), true PMI, and 500 samples from the distribution in Fig. 8. The data drawn from a bivariate Student distribution with dispersion matrix
1
0.5
Ω
0.5
and varying degrees of freedom (1,2,3,5, 8). Additionally, we repeated the experiment for the bivariate normal distribution with covariance matrix given by the dispersion, which corresponds to the limit v co.
Qualitatively we see that the critic does not learn the correct PMI for v E {1,2], with better similarities for E {3, 5,8). For the normal distribution, the critic qualitatively approximates well the PMI function in the regions where the data points are available.
We hypothesise that these problems can be partly atributed to the longer tails of Student distributions (as the critic may unable to extrapolate to the regions where few points are available) and partly to the analytical form of the PMI function for the considered bivariate normal distribution the PMI is given by a simple quadratic function, which is not the case for the bivariate Student distribution.
E.2 Sample Size Requirements
We investigated the minimal number of samples needed to estimate the mutual information changing the number of samples in N e {100, 500, 1000, 3000, 5000, 10 000}. We say that sample & suffices
26

---

to solve the task if for every e' 2 e the estimator provides the estimate between 2/3 and 3/2 of the ground-truth mutual information.
We present the minimal number of samples as a function of benchmark task and considered estimator in Fig. 9. Note that CCA and KSG typically need very small amount of samples to solve the task. As neural estimators use batch size of 256 and split the sample into training and test data sets, it is not possible for them have l 1000, which is the smallest number of samples considered greater than 2 256 = 512.
E.3 Preprocessing
In Fig. 2 and Fig. 9 we transformed all samples by removing the mean and scaling by the standard deviation in each dimension calculated empirically from the sample. This strategy ensures that the scales on different axes are similar. Note that this strategy is not always suitable (e.g., if the distance metric used in the kNN approach should weight select dimensions differently) and makes the observed data points not i.id. (as the "shared" information, sample mean and variances, has been used rescale the variables). although the effect of it seems to be negligible on most tasks in the benchmark.
other preprocessing strategies, implemented using the scikit-learn package [Pedregosa et al., 2011]: making each marginal distribution uniform (niform(O, 1) by calculating empirical CDF along each axis (see Section D for the technical lemmata discussing this operation) and making each marginal distribution a Gaussian variable (N0, 1). These approaches may affect the effective sample size more strongly than the previous rescaling strategy, as estimating empirical CDF requires more data points than estimating mean and standard deviation.
Additionally, considered
We used N 10 000 data points and rerun the benchmark (see Fig. 10). Overall, the impact of the "uniformization" " strategy appeared to be insubstantial for most tasks, and in certain cases negative. The "Gaussianisation" strategy improved estimation for Student distributions and helped histogram based approaches in low-dimensional tasks. Interestingly, both approaches worsen the performance of CCA on Student distributions.
While the improvements gained from "Gaussianisation" were minor, the approach is simplistic, treating each axis separately. Ideally, appropriate preprocessing would be able to transform Pxy into a netter-behaved distribution P)(X)g(Y), and thus simplify the estimation. In certain cases such preprocessing could (partially) undo transformations considered in our benchmark, and in effect make the estimators invariant (or more robust) to these transformations (ef. Section 4). We consider the design of informed preprocessing strategies important open problem.
E.4
Hyperparameter Selection
CCA We used the implementation provided in the scikit-leand [Pedregosa et al., 2011] Python package (version 2.)... We set the latent space dimension to the smaller of the dimensions of the considered random vectors X and Y.
Histograms Adaptive estimators such as JIC [Suzuki, 2016] or MIIC [Cabeli et al., 2020] were not applicable to the high-dimensional data we considered, hence we resort to using an equal-width estimator implemented TransferEntropy library (version 1. 10.0000 0). We fixed the number of bins to 10, although the optimal number of bins is likely to depend on the dimensionality of the problem.
KSG As a default, we used the KSG-1 variant from the rmi package with 10 neighbors. However, as shown in our ablation in Fig. 11, changing the number of neighbors or using the KSG-2 variant did not considerably change the estimates. We compared the used implementation with a custom Python implementation and one from Transf TransferEntropy erEntropy library. The version implemented in the TransferEntropy suffers from strong positive bias.
LNN
We used the implementation from the rmi package (version 0. with 5 neighbors and truncation parameter set to 30. Ablations can be seen in Fig. 11.
Neural Estimators We split the data set into two equal-sized parts (training and test) and optimized the statistics network on the training data set using the Adam optimiser with initial learning rate set to 0.1. We used batch size of 256 and ran each algorithm for up to 10 000 training steps with early stopping (checked every 250 training steps). We returned the highest estimate on the test data set.
27

---

HRNE- -2.4 0.4 1.6
03
5.4
0.4
0.
04 01
04
0.3
2.4
0.5
1.2
15
9.5
0.
0.8
64
0
00
2.0
0.5
03
0.9
0.9
86
0.4 0.4
0.4
0.5 07
54
0.0
0.
-244--4000
044
1,6
0.3
94
04
0.3
9.4
0.5
1.1
12
98
0
00
00
08
0.9
0.9
6.8
9.5
0.5
0.8
06
61
0.0
-V-
2.6
03
2.4
3,2
01
0.1
(R) -244
1.7 03
5.4
5.4
0.6
5.0
02
69
53
6.6
01
LNN (0) -2.3 0.3 0.5002
0.3 0.3
24 02
0.6 2.0
1.3
R5
0.4
06
2.9
1.3
5V
0.7
(uta)
-2.4
Trantterguay guak) 3.4 04 1.6 0.
3/4 04
8.4
0.
4.1
8.5
4.
6.0
14
84
9.0
92
4.7
R7
B.
42
R3
9.0
40
0.6
9.0
9.0
8.4
9.0
9.
4.3
6.
4.0
22442.4000 04 27
1:3
0.0
5.0
Thie Hi 3.4 0. 1.7
0.6 1.3
5-2
Vnniiin Si Binod
Sooee wood o0 no INS nomo
stmme 1101-1)-
x 100f-
Untormte horee ml x
Hnstomee Wuit x (emmo)
unitomae Suttsmma x (dlmma)
uniuontz 8 WAll 5 (looo))
Unioomee & Warthomm 25 25 (iemma)
untomize & Rutramm monamaa 50 50 (umml
Unte HLB mOTTA (P
Uster Wiltnoma 25 22 12 pant
Unter mice Student x 210/-43
Un S St..mm--2 x 2106r
Le soreeteeeee @ Guudent x 100m
Ln mnnurmee St.dente x 3100--3
Ustumbzc Norrrll COP H.R Hutiomm 25 25 12
Undombee S Ial-cute MUB norrral 10
522 32-9
25 12 p00n
unfo inforsss B 53821 @ Mutioraal 3 pul
(7
25
Lnttombee Sotaal car HLB nomml x (2
Untomus Spria Norrral CDP R.tromml 25 23 12 p2-
Unetamu AuRn S Stummit x (100-3)
uritunioo we Ashn Student x (000022)
Enpramtte Jean o co s (ibr
MuRiomra
Unterman hormal cDr notmomal
Uniommze 8 Nifruu @ HuB normaal
Unde mom Nafoute M Nultmorral 25
Unfo Vifummzz Si83l CDF Mutiomral
T. V SONII
Sonnn wn untore Cooeroae 9
Coneen de comuugg di wan
Ailommic Speato
Unfer
Untermbe hormal
a mome de cotiinn
-5K 04 1.6 03 5.4 04
0.3
3.4
0.6
1.2
15
0.
6.
51
02
1.0
5.
0.
0.8
0.0
5.7
0.
Intwrr 3.4 04 3.6 0-3
3.4 0.4
0.6
1/2
14
1.0
1.0
1.8
0.9
DV 344 0.4 1.6 0:3
5/4 84
8.4
57
02
0.4
0.3
5.4
0.
1/2 15
1.0
1.0
3.9 83
57
07
1
1.0
0.9
1.0 0.
05 0.
0. 0.9
0.
33
0.
30
1.7 033
LMV (R) -3.3 03 9.6
5:3
1.3 2.7
0.5
3.7
His: (uta)) .33 03 6602
0.3 0.3
93
34 4.242
92
D.F
4.0
46 92
46
4.6 9.2
Warrfor -2.4
CCA-3.4 04 1:307
5.6 0. 0.
5.
1.000.00 0.4
0.
5.4 0. 1.3
1.0
1:0
1.0
0.0
0.
0.0
0.0
0.
TO
3.0
11
1.0 1.5
30
0.2
0.4
5. 0.8
0.9
5.0 0.0
2.4 0.4 03
06
1.3 16
3.0 1.0 1o
22
oussunn Uslomm 1 (2o0iiee reaze0 75
x
Cossiiii @ Wayu or stratitt nomma xx
Studem81 (nd--))-
Gousson kim Stutent x 1o0-3))
Gruanan Sone mll3 x
Gelcentte MLB nommaa X (done)
Gusaaarta H Nutrommal 3 (dbra))
Gauss foe ty Nuttoonal 5 (drrod
Gamna stase Mu Nulltmmmat 50 5-50 (dras))
Hultnmmal2 75 2 (denne)
Sountitee Wantsemeets Glowe
Guzaannee Nuttrrma 25 (2-0
Gussinns 2 SIDEmIE x [o0+3))
CanstIEe & Suutee x (ree 22
Gonstante & Sustrot x Ino 22
Gusstinse Siudenc x loat-33
Gussinnz & Suitins tou--))
Gaustneee Mutismmal (2
Gezsimiie karmu car Waitnemits (2-w0)
Gaussunce hurnal CDP WuS audiomul 25 25 (2 paif
Guneaaie & IN Scube Wettormm xx 1.122
caussaiia 9 nu Scube outtoemal 5 (1 part
Coussattly 3 Hilf obs 8 Watt mal 25 25 (2 par.
Sosautii Spria nutisemal x (22)))))
Gerebente Spri kerms car Wantsemets 5 (lowr
Gassinns g sotal hormi CDP sutromi 25 20 (2 suin
Ga @ Amm 9 Stuimn x (20h-33-
daussanne e Jam a 3 toof-))
Groudn on & som a 5 toe 23
Gueeenv intoon x petttie to
Guushattz 3
caunterss & muttrammiis 25 CDr 3a
Ga
a i
Gussiunte
Gauzlonse Spril
Figure 10: Estimates for preprocessed data with N = 10 000 samples. Top: margins transformed into uniform distributions. Bottom: margins transformed into normal distributions.
We chose the architecture using mini-benchmmrr with four ReLU neural networks considered on two neural estimators (see Fig. 11) and chose a neural network with two hidden layers (network M in Table 2).
As neural estimators rely on stochastic gradient descent training, we implemented heuristic diagnostics used to diagnose potential problems with overfitting and non-convergence. We present the number of runs susceptible to these issues Fig. 12. We did not include these runs when computing statistics in the main benchmark figures (Fig. 2 and Fig. 9).
28

---

0.2
0.2
0.4
0.2
MINE D
0.2
1.2
0.0
0.4
0.8
0.1
0.8
0.1 0.0
0.8
0.9
0.1
(n-10, Python]
MINE
0.4
1.2
0.2
0.4
0.9
0.2
KSG [m=5,
0.1
0.4
0.1
0.2
1.0
0.2
0.2
MINE M
-0.4
1.2
0.2
0.3 0.9
0.2
0.1
0.8
0.9
0.3
KSG In=10, R]
0.1
0.7
0.7
0.7
0.2
0.4
0.4
0.1
0.2
1.0
0.2
KSG [nw5, RJ
R0-
0.1
0.7
0.2
0.4 0.4
0.1
1.0
0.2
022
MINE 0.3 1.2
0.0
0.4
0.9
0.1
0.1
InfeNCE D
0.4 1.3
0.1
0.4 0.9
0.2
0.2
0.9 1.0
0.2
(n=5.
InfoNCE 0.4 1.2
0.3
0.4
1.0
0.3
KSG (n-5, Jolial
11.7
2.6 5.3 2.4
5.2
InfoNCE M
0.4 1.2
0.3
0.4 0.9
0.2
0.2
0.2
0.9
1.0
0.3
INN (n-5, 10,0 30,
R0-5.
5.7
3.1
2.2
2.4
0.3
1.5 0.1
3.5
1.3
InfoNCE 0.4 1.2
0.2
0.4
0.2
012
0.9
1.0
0.2
LNN In-10, 1030,
R0-1
1..92.8
7.7
0.5
1.5 0.4
3.2
True
0.4 1.3
0.4
0.4
1.0
1.0
Multinormal 25 (dense)
Asinh e Studenttt x (dof-2)
Normal Multinormal (2:pair)
Asinh & Student-t [ddof11)
Normal CDF Multinormal 25 (2-pair)
Student:t x (dof=2)
Asinh @ Student-4 (dof=2)
Multinormal 25 25 (2-pair)
Multinormal x (2-pair)
Studentst x 5 toof=
Swiss roll 2
Muktmomm 25 x 2-palt
Mukinormaa 25 Pail
Normal CDF Multinormal 5 (2.pal))
Student x (dof=2)
Student (def--2)
Asinh @ Student x (def-11
Asinh @ Student (defm2))
Asinh @ Student- (detmm2)
Bimodal
Bimodal
Swiss roll
Multinerma 25 25
Multinorma
Nomral CDF Muna
Figure 11: Left: ablation of hyperparameters the KSG and LNN estimators (with n neighbors and threshold 1). Right: ablation of neural network architectures.
Architecture
Hidden layers
D
8, 8, 8
L
24,
M
16,
S
10,
Table 2: Network architectures and their corresponding hidden layer specifications.
Visualisation of Selected Tasks
In the following, visualise selection of benchmark tasks. Fach figure visualises sample of
N = 100 points from Pxy- The upper diagonal represents the scatter plots Px,x,- Px,y, and Py;,; for all possible combinations of i + j. The lower diagonal represents the empirical density fitted using kernel density estimator with 5 levels. The diagonal represents the histograms of the marginal distributions P'x, and Py; for all possible
present selected 1 --l-immenioonal tasks Fig. 13 and Fig. 14; the 2x1-dimensional Swiss- roll distribution in Fig. 15 and selected 2x 2x2-dimensional tasks in Fig. 16. Finally, in Fig. 17 present two 3x3- 3-dimmnnsionnl problems, visualising the effect of the normal CDF and the spiral diffeomorphism.
29

---

MIKE
INMACCE
DV-
intom x (4brise mo 50
Leltonn x (o00line resIn 3--
simiill x
iutcuto (00--)
asin Stutontss (00-))
kutanomma 5 5
Muthonmo 5 diune-
MuthomatED 25 (dhow))
Mattroe (2
Wattored (2rp))
Vubrormal 23 x22 (2par)
Stutent (004))
72-
Nerman car Wuttree me (2aw::
hoe nar CDP Mutrrmmal 25 28 (2pa0-
Iw Wettromma x (Peww
naltects Wattrr mul' x's (2
Natton: Waltromal 25 2 (2pal-
Spra t Wattro mal x (baaa)-
euss nnl co Wattromm 32
Voema eat Wittormal xs (Pew),
CEr Hulrroma x (2pa0)
Aumn susuni12 (00-2)-
Au g Stalant i 3000-
7)
Sode miteentl
Qustente 1 3
Aeen susteters 5100
Way's i numaa
Stutente3 i
spnna Wattramal
Spres
Normal
Figure 12: Number of runs (out of 10) when our convergence and overfitting diagnostics diagnosed potential problems in neural estimator training.
Unifem (addtive noisew0.1)
Uniform x ladditive noisemo. 75]
10-
08-
0.8
0.-
0.0
25
-1
0.5
3.2
0.5
0.0
2.2
X)
Eivariate normall 1
Bimodal 1 xx1
3.0
2.5
-25
50
Figure 13: Top row: additive noise model. Bottom row: bivariate normal distribution and transforma- tion making each variable bimodal mixture of Gaussians.
G Author Contributions
Contributions according to the Contributor Roles Taxonomy:
1. Conceptualization: PC, FG, AM, NB.
2. Methodology: FG, PC.
3. Software: PC, FG.
4. Validation: FG, PC.
5. Formal analysis: PC, FG.
6. Investigation: FG, PC, AM.
30

---

Asinh @ Staderist x (dofm1)
3.08
6.99
Xg
Holf-cube Bivoriate
Vigaly Bivariate
Figure 14: Top row: bivariate normal distribution transformed so that the margins are uniform and asinh-transformed bivariate Student distribution with I degree of freedom. Bottom row: bivariate normal distribution transformed with the half-cube and the wiggly mappings.
7. Resources: FG, NB.
8. Data Curation: FG, PC.
9. Visualization: FG, PC,
Supervision: AM, NB.
11. Writing original draft: PC, AM, FG.
12. Writing review and editing: PC, AM, FG, NB, JEV.
31

---

Swiss roll
0.4
0.2
-04
0.5
0.0
-0.5
0.00
0.0
-0.5
0.0
0.5
0.0
X1
X2
Yi
Figure 15:
Visualisation of the Swiss-roll distribution.
32

---

Mutromal2 Mese
2
Sueent2 x (81-2)
Figure 16: Visualisations of selected 2 dimiennnsoo problems. Top row: multivariate normal distributions with dense and sparse interactions. Bottom row: multivariate Student distribution with degrees of freedom.
33

---

Sahald 3-30
Figure 17: Visualisations of selected 3 problems. Top: multivariate normal distribu- tion transformed axis-wise with normal CDF, to uniformise margins. Bottom: multivariate normal distribution transformed with the spiral diffeomorphism.
34


Reviews:
[{"review_id": "MfVvITCLut", "reviewer": "Reviewer_AkdF", "summary": "This paper focuses on the topic of mutual information and shows how to construct a diverse family of distributions with known ground-truth mutual information. It's worth noting that obtaining a closed-form solution for mutual information is highly dependent on the specific assumptions and functional forms used for the variables X and Y. In practice, deriving closed-form expressions for mutual information can be challenging and may require additional simplifying assumptions or specific knowledge about the distributions involved.\n\nIn contrast to previous works that typically assess mutual information estimators using simple probability distributions, this paper introduces a novel approach to constructing a diverse family of distributions with known ground-truth mutual information. Additionally, the authors propose a language-independent benchmarking platform to assess mutual information estimators. The authors explore the applicability of classical and neural estimators in scenarios involving high dimensions, sparse interactions, long-tailed distributions, and high mutual information. By examining these challenging settings, they provide insights into the strengths and limitations of different estimators. \n\nMoreover, the paper offers guidelines for practitioners to select the most suitable estimator based on the specific problem's difficulty and considerations when applying an estimator to new datasets. By presenting a comprehensive evaluation framework and practical recommendations, this research aims to advance the understanding and application of mutual information estimation in various domains.", "questions": "See above \"weaknesses\". ", "limitations": "See above \"weaknesses\". ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The mutual information estimator is an essential tool in causality, it can help discover the underlying causal graph or inference the strength of causal relations. However, as I mentioned earlier, deriving closed-form expressions for mutual information can be challenging in practice and may require additional simplifying assumptions or specific knowledge about the distributions involved.\n\nThe paper introduces a method to construct a diverse family of distributions with known ground-truth mutual information. This is a significant contribution as it allows researchers to explore and evaluate mutual information estimators across various scenarios, encompassing various data characteristics and relationships. For example, explore gene regularity networks, understand the treatment effect in medical health care, gain insight for constructing a recommendation system, etc. \n\nThe research paper presents a comprehensive evaluation framework for mutual information estimation, encompassing the construction of diverse distributions, benchmarking platform, exploration of challenging scenarios, and practical guidelines. This framework provides a holistic view of the estimation process, aiding researchers and practitioners in understanding, comparing, and selecting mutual information estimators effectively.\n\nFurthermore, the authors investigate the applicability of classical and neural estimators in challenging scenarios involving high dimensions, sparse interactions, long-tailed distributions, and high mutual information. This exploration provides valuable insights into the performance, strengths, and limitations of different estimators under these challenging conditions, enhancing our understanding of their effectiveness in real-world settings.", "weaknesses": "1. Not all joint distributions can be represented in the form of $P_{f(x)g(x)}$, limiting the applicability of the benchmark to a specific set of distributions. Extending the family of distributions with known mutual information and efficient sampling is seen as a natural direction for future improvement.\n2. Even though the benchmark demonstrates that distributions with longer tails pose a harder challenge for the considered estimators, applying a transformation like the asinh transform does not fully address the issues.\n3. The summary does not mention external validation or comparisons between the proposed approach or estimators and existing methods or benchmarks in the field. The absence of such external validation makes it difficult to assess the generalizability or superiority of the contributions in relation to established techniques or alternative approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review. Regarding the questions and limitations:\n\n> Not all joint distributions can be represented in the form of $P_{f(X)g(Y)}$, limiting the applicability of the benchmark to a specific set of distributions. Extending the family of distributions with known mutual information and efficient sampling is seen as a natural direction for future improvement.\n\nYes, we agree with this assessment. As noted by the Reviewer, obtaining a closed-form solution for mutual information usually requires simplifying assumptions, which constrain the distributions.\n\nOne way to extend the family of distributions is to allow for random variables for which mutual information can be efficiently estimated numerically. For example, when the joint and marginal probability distributions have tractable PDFs, mutual information can be estimated by averaging pointwise mutual information over a sufficient amount of Monte Carlo samples. In most cases, the standard error of the mean should be a sufficient diagnostic to provide precise and accurate estimate. This type of task could be readily implemented in our package.\n\n>  Even though the benchmark demonstrates that distributions with longer tails pose a harder challenge for the considered estimators, applying a transformation like the asinh transform does not fully address the issues.\n\nIndeed, our experiments suggest that although long tails make estimation harder, simple approaches of \"shrinking\" the tails (as with the asinh transform or different preprocessing strategies explored in Appendix E) cannot resolve this issue completely. We consider this an interesting open problem and suspect that this may be related to the shape of the \"PMI profile\" (defined below).\n\nLet $i$ be the pointwise mutual information (PMI) function for distribution $P_{XY}$. We define the PMI profile to be the distribution of $i(X, Y)$. This distribution is defined on the set of real numbers and its expected value is the mutual information. Higher moments (and, informally, the overall \"shape\") of this distribution can influence how well the expected value can be estimated. Importantly, the PMI profile does not change under diffeomorphisms, so the PMI profiles of $P_{XY}$ and $P_{f(X)g(Y)}$ are the same.\n\nWe hypothesise that this may explain why these simple preprocessing strategies (which proceed by applying diffeomorphisms) cannot fully resolve the issues with long-tailed distributions, but we have not validated this hypothesis yet.\n\n> The summary does not mention external validation or comparisons between the proposed approach or estimators and existing methods or benchmarks in the field. The absence of such external validation makes it difficult to assess the generalizability or superiority of the contributions in relation to established techniques or alternative approaches.\n\nA lack of universally accepted, easy-to-use benchmarks for testing mutual information estimators is one of our main motivations. One of the packages implementing several estimators we consider in our benchmark was covered by unit-tests which only ensured that the returned value is a `float`(!). Other benchmarks (Song and Ermon (2020), Khan et al. (2007), and Poole et al. (2019), which are discussed in Section 5) are typically constructed for evaluating a specific class of estimators, and usually focus on Gaussian variables (with known MI) or complicated high-dimensional tasks (with MI not known). This has made it difficult to build a general understanding of available methods and their applicability. We believe that our benchmark can serve as a strong reference point for future work."}, {"review_id": "NJfXa1AhOL", "reviewer": "Reviewer_FjRx", "summary": "A test benchmark for the evaluation of mutual information estimators is established and many different estimators compared. The test cases contain student-t and normal distributionas and their injective transformations. Difficult cases are discussed and evaluated in more detail.", "questions": "What was the reason for choosing this specific set of distributions? \nWhat is \"pointwise MI\"?", "limitations": "The authors state that there are more interesting cases and they only cover transforms of normal and student-t distributions. They also mention that prior information might be incorporated.\nHowever, as said above it is known that an MI estimator can be fooled arbitrarily (giving any value for the MI) if one can choose the distribution freely. Thus, prior information is also included in the test cases here it is just not mentioned explicitly.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "The code is reproducible and thus might be used for other estimators in the future\nThe paper is well-written and easy to understand.\nIt is important to make the community aware that evaluation of MI estimators on Gaussian distributions is pointless as these only depend on the covariance structure, so the big plus of the MI that it goes beyond te linear dependencies is ignored. The paper makes a strong point here by including a simple covariance estimator as well.\nThe results on heavy tails are particularly interesting.", "weaknesses": "The choice of the distribution used is not sufficiently argued. In particular, it is known that no MI estimator can evaluate MI correctly on arbitrary distributions. Only with restriction to a class of probability distributions (e.g., probability density functions with Lipschitz constraints) there is hope that estimation works. It is thus quite pointless to evaluate MI estimators on what seems like an \"educated guess\" of diverse distributions. \nThe authors seem to not be aware of the huge theoretic background of MI, e.g., that arbitrary measureable injective mappings do not change MI and thus their only Theorem 2.1 is well-known in a much more general setting. This can be argued by the data processing inequality in two directions (X-Y-f(Y) and X-f(Y)-Y are both Markov chains) or directly the definition of MI via countable partitions that do not change if we use a measureable injective mapping.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed comments.\n\n> The choice of the distribution used is not sufficiently argued. In particular, it is known that no MI estimator can evaluate MI correctly on arbitrary distributions. (...)\n\nIndeed, one version of a no-free-lunch theorem for MI estimation follows from the fact that for $n\\ge 1$ and $k\\ge 2$ any set of $n$ points in $\\mathbb R^k$ can be mapped to any other set of $n$ points by a diffeomorphism (if $M$ is any connected smooth manifold of dimension at least $2$, the group $\\mathrm{Diff}(M)$ acts $n$-transitively on it). It is thus clear that the only truly diffeomorphism-invariant estimators have to be constant functions (for a given number of points).\nNonetheless, invariance to diffeomorphisms is precisely the reason why MI is useful in practice. The theorem above implies that all estimators have to break at some point. However, while no estimator can work on all distributions, different estimators can work better on certain types of distributions. From a practical standpoint, it is interesting whether certain distributions which intuitively seem \"reasonable\" are difficult for estimators (e.g., even low-dimensional Student distributions are challenging for most estimators), as well as which estimators are particularly suitable for which types of problems (e.g., neural estimators are good at solving problems involving sparse interactions).\n\nThe issue of which distributions are \"reasonable\" is indeed vague. Certainly, some distributions are \"unreasonable\" (e.g., a complicated embedding into a 1000-dimensional space), and we do not expect any estimator to solve them. In our study, we avoid such uninformative problems by excluding tasks which no estimator was able to solve. We also include visualisations of the considered distributions to allow for visual inspection (Appendix F).\n\n> What was the reason for choosing this specific set of distributions?\n\nAs explained above, the goal was to understand the limits and advantages of different estimators. To this end, we decided to focus on relevant phenomena (e.g., robustness to \"sufficiently nice\" diffeomorphisms, sparsity) by designing a set of transformations wide enough to (a) understand the mentioned issues and (b) construct a standardised benchmark, which can be used to diagnose strengths and weaknesses of new estimators.\n\nFor the motivations of individual phenomena, see the General Response.\n\n> They also mention that prior information might be incorporated. However, as said above it is known that an MI estimator can be fooled arbitrarily (giving any value for the MI) if one can choose the distribution freely. Thus, prior information is also included in the test cases here it is just not mentioned explicitly.\n\nYes, since no estimator can work on all distributions, each estimator can be thought of as having an implicit bias towards distributions it can handle well. For example, we show that the KSG estimator is not competitive with neural approaches when interactions are sparse. Thus, if we believe that the distribution we are analyzing has sparse interactions, we should use a neural approach. We argue that using (and developing) estimators with explicitly known assumptions and biases could result in significant performance gains. We demonstrate this by implementing CCA (which uses very strong prior information), and showing that it is an excellent choice when the distribution matches assumptions.\n\nWe think the term \"prior information\" in the manuscript may have been misleading and not convey the above meaning. We will clarify it in the manuscript.\n\n> The authors seem to not be aware of the huge theoretic background of MI, e.g., that arbitrary measureable injective mappings do not change MI and thus their only Theorem 2.1 is well-known in a much more general setting. This can be argued by the data processing inequality in two directions (X-Y-f(Y) and X-f(Y)-Y are both Markov chains) or directly the definition of MI via countable partitions that do not change if we use a measureable injective mapping.\n\nWe believe that Theorem 2.1 has been known in the community for a long time and therefore we did not list it as a contribution. In spite of our efforts, we were not able to find a reference with a formal proof of the theorem which covers singular measures, infinite MI and topological embeddings. Since we use this result extensively, we supplied an appropriate version in the appendix. We would be grateful for suggesting a reference covering this (or a more general) result, so that we can cite it. The proof provided by Kraskov et al. (2004), for example, does not cover the “Swiss roll” distribution, which uses a topological embedding or the half-cube mapping, which is not a diffeomorphism at 0.\n\n> What is pointwise MI?\n\nWe apologise for using the term \"pointwise MI\" in Section 6 rather than \"pointwise mutual information\", which we use in all the other sections. We fixed this notational inconsistency.\n\n> The authors state that there are more interesting cases and they only cover transforms of normal and student-t distributions.\n\nThere are many distributions which cannot be obtained as $P_{f(X)g(Y)}$, where the join $(X, Y)$ is a multivariate normal or Student distribution and $f$, $g$ are continuous injective mappings. However (except for some simple bivariate distributions and trivial cases with zero MI), an analytical expression for non-zero ground-truth MI is currently tractable only for the multivariate normal and Student families.\n\nWe consider enlarging the proposed family an important open direction of the problem and we designed our benchmark so that adding new distributions with known ground-truth MI requires little effort.\n\nWe hope that we were able to answer questions raised in the review and clarify our contributions. Given that the Reviewer confirmed excellent soundness and presentation, and a fair contribution, we would be thankful if the Reviewer could reconsider the overall score."}, {"review_id": "Il5mcl7FUh", "reviewer": "Reviewer_vCqe", "summary": "This paper identifies a clear problem with many mutual information estimation benchmarks: most of them focus on simple (normal) distributions. The authors present a new set of forty (!) tasks that contain ground-truth informations, which can be constructed by noting that only injectivity is needed for an information-preserving transformation. The authors identify four distinct challenges: interaction sparsity, long tails, invariance, and high information values. Several conclusions about existing estimation algorithms are then made regarding the extensive analysis.", "questions": "- Out of curiosity: could modern invariant neural network architectures be used to obtain MI estimates invariant to diffeomorphisms?\n\n\n### Conclusions\nWhile the overall contribution could be limited in terms of a model development sense, I think the paper identifies serious issues with modern MI estimation benchmarks. The paper not only provides new benchmarks that address these issues but also makes an effort to identify what aspects of MI estimation can make the problem hard. I foresee much new research originating from the identification of these aspects, where future papers focus in on them and propose methodologies that overcome these challenges. On top of that, the paper is very well written. Hence, I would recommend acceptance.", "limitations": "The authors clearly discuss the limitations of their study.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- I enjoyed reading this paper. It clearly defines a goal and identifies key problems with existing approaches. \n- The paper presents mathematical background in a precise and effective way.\n- The structure of the paper is clear, using figures for clarifications where needed.\n- The paper is well-written with clear sentence structures and no grammatical or spelling errors.\n- The analysis is thorough, presenting forty distinct MI estimation tasks.\n- I enjoyed how instead of simply presenting the results, the authors dug deeper and identified four distinct challenges for MI estimators.\n", "weaknesses": "- One could say that the paper lacks slightly in terms of originality and contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive assessment and encouraging words.\n\n> Out of curiosity: could modern invariant neural network architectures be used to obtain MI estimates invariant to diffeomorphisms?\n\nThank you for your insightful question! Sadly, we cannot achieve full invariance to diffeomorphisms for finite samples since an arbitrary diffeomorphism can transform any set of $n$ points to any other set of $n$ points in $\\mathbb R^k$ when $k\\ge 2$. Nonetheless, we can hope for invariance (or robustness) to a subset of diffeomorphisms. We hypothesise that such invariant neural networks could require smaller sample sizes due to their useful inductive biases. However, we have not experimented with invariant neural networks ourselves, so we are unable to support this hypothesis with data.\n\nWe think that there are several solutions how this problem can be approached: either by encoding invariance to, e.g., the group of rigid motions by appropriate layers (in this case based on the representation theory of the orthogonal group) or using training schemes increasing robustness to diffeomorphisms, akin to [1, 2]. Our benchmark can be used to generate data sets for the latter approach, however thorough investigation of this idea is beyond the scope of this work.\n\n[1] Benton, Gregory, et al. \"Learning invariances in neural networks from training data.\" Advances in neural information processing systems 33 (2020): 17605-17616.\n\n[2] Petrini, Leonardo, et al. \"Relative stability toward diffeomorphisms indicates performance in deep nets.\" Advances in Neural Information Processing Systems 34 (2021): 8727-8739."}, {"review_id": "Lvcsr774jj", "reviewer": "Reviewer_62uG", "summary": "The authors propose a method for creating expressive distributions via injective mappings that maintain their original MI. They state this in Theorem 2.1 and prove it in the Appendix. In addition to this, the authors benchmark a variety of estimators for MI on a set of tasks, including high-dimensional, long-tailed distributions, sparse interactions, and high mutual information. This information is contained in Sections 3-4, where they describe each task and critique each estimator for the tasks. From these experiments, the authors provide guidelines for practitioners on how to choose an estimator in Section 6.", "questions": "Why not utilize the result of Theorem 2.1 and apply it to Normalizing Flows? I believe this combination would enhance the paper's distinctiveness. \n\nHow is Theorem 2.1 different from that of the result in the appendix of https://arxiv.org/pdf/cond-mat/0305641.pdf?", "limitations": "The authors have addressed the limitations of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is clear, provides many novel benchmarking tasks, and is easily verifiable. The results are reproducible as the authors have shared their code and documented their experimental parameters.", "weaknesses": "1. The author does a poor job of motivating each data setting and explaining why each one is important. It would be beneficial to provide examples of domains where long-tail distributions are common, such as insurance, and elaborate on the significance of the other data settings as well.\n2. I believe the last row of Figure 2, \"True MI,\" could be better highlighted as it was difficult to discern that it was the point of comparison.\n3. The main contributions of the paper seem relatively minor, as the authors are primarily considering more data settings than previous work when comparing MI estimators. However, addressing the first point could help alleviate this concern.\n4. The main theoretical result, Theorem 2.1, appears to have already been demonstrated in the appendix of the following paper: https://arxiv.org/pdf/cond-mat/0305641.pdf.\n5. The authors seem to have switched \\citep for \\cite in their paper. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review.\n\n> The main theoretical result, Theorem 2.1, appears to have already been demonstrated in the appendix of the following paper: Kraskov et al. (2004)  [...] How is Theorem 2.1 different from that of the result in the appendix of Kraskov et al. (2004)?\n\nWe believe that Theorem 2.1 has been known in the community and we do not list it as a contribution. We use it to define the distributions used in our benchmark, which is the main content of the paper. However, we could not find an appropriate version of this theorem with a formal proof which we could reference, hence for completeness, we provide our own proof in the Appendix.\n\nThe highly influential paper of Kraskov et al. (2004) covers the case where (a) applied mappings are diffeomorphisms and (b) all measures involved have probability density functions. The proof we give in the Appendix applies more generally (and this is needed e.g., for the \"Swiss roll\" distribution which uses topological embeddings or for the half-cube mapping which is not a diffeomorphism at $0$).\n\n> The author does a poor job of motivating each data setting and explaining why each one is important. It would be beneficial to provide examples of domains where long-tail distributions are common, such as insurance, and elaborate on the significance of the other data settings as well. \n\nThe main motivation for selecting the distributions was to evaluate and compare existing estimators. We would like to stress that the space of all possible distributions is vast, and until now, was left almost entirely unexplored except for the simplest case of a normal distribution. This is particularly striking for mutual information, which is commonly used precisely for its applicability to any distribution and invariance to diffeomorphisms.\n\nWe approach this issue by casting a wide net of tasks, paying particular attention to the following:\n\n1. **Dimensionality**. High-dimensional datasets are becoming more common, particularly in machine learning and systems biology.\n2. **Sparsity**. While the data might be high-dimensional, the effective dimension may be much smaller (e.g., only a few genes out of thousands convey information about the amount of a particular protein).\n5. **High MI**. Estimating high MI is known to be difficult. However, it is usually something we might know approximately in advance -- if there are 4000 image classes, MI between image class and representation is at most 12 bits. An additional interesting observation is that CCA performes very well, suggesting that in this scenario incorporating prior information is crucial.\n6. **Long tails**. Since the Student distribution has heavier tails than the multivariate normal distribution, this was a natural choice. An interesting conclusion is that even after \"removing the tails\" (see Appendix E), these distributions remain difficult. Thus, we have low- and moderate-dimensional distributions, which are unimodal and without heavy tails, which are still challenging to estimate.\n7. **Robustness to diffeomorphisms**. Mutual information is often chosen because it is theoretically invariant to diffeomorphisms. We wanted to challenge this invariance when only a finite sample is available.\n\nHowever, we agree that including real-world examples illustrating these motivations will improve the manuscript.\n\n> The main contributions of the paper seem relatively minor, as the authors are primarily considering more data settings than previous work when comparing MI estimators. However, addressing the first point could help alleviate this concern.\n\nThis is the first benchmark which can be used to compare different mutual information estimators in a systematic and reproducible manner. As mentioned above, it is critical to evaluate mutual information estimators on non-normal distributions. Our benchmark contains a diverse set of 40 distributions, addressing problems like sparsity of interactions, long tails and invariances to selected mappings.\n\nAs such, we believe this manuscript will be useful for the community for two reasons:\n  - For the mutual information researchers, it provides a standard benchmark which can be used to conveniently test their ideas and understand their strengths and limitations. To increase the chances that it is adopted by the community, we made it cross-platform, used well-defined interfaces in the API, and implemented a diverse set of baseline estimators. \n  - For researchers willing to use mutual information in selected problems ranging from machine learning to natural sciences (e.g., Uda (2020), Young et al. (2023)), a set of points to be aware of when a reliable estimate is needed.\n\n> I believe the last row of Figure 2, “True MI,” could be better highlighted as it was difficult to discern that it was the point of comparison. [...] The authors seem to have switched \\citep for \\cite in their paper.\n\nThank you for these suggestions: we have improved the figure accordingly and fixed inconsistency with citations.\n\n> Why not utilize the result of Theorem 2.1 and apply it to Normalizing Flows? I believe this combination would enhance the paper’s distinctiveness.\n\nIndeed, using normalizing flows is an interesting idea which can be used as:\n  1. Transformations to define more expressive distributions.\n  2. Another preprocessing technique (similar to the ones studied in Appendix E).\n\nWe had considered both options, but eventually, we have decided to use interpretable alternatives: proposed mappings were able to capture several interesting phenomena already, and we were concerned that using normalizing flows may distort the distributions in unexpected manners, making it difficult to separate failure modes of different estimators.\n\nWe hope that our response provided an additional perspective on the significance of this work and that the introduced changes have increased the manuscript's clarity. Hence, we would like to kindly ask the Reviewer to consider raising the scores."}]

Author rebuttal:
We would like to thank the Reviewers for their insightful comments and appreciate that they find that our work *"provides valuable insights into the performance, strengths, and limitations of different estimators*" (AkdF), that our *"results on heavy tails are particularly interesting"* and that the paper *"makes a strong point here by including a simple covariance estimator as well"* (FjRx). It *"presents mathematical background in a precise and effective way"* (vCqe) and that *"the paper is clear, provides many novel benchmarking tasks, and is easily verifiable"* (62uG). We find it particularly encouraging that all reviewers find that our paper is clear and that Reviewer vCqe concludes *"I foresee much new research originating from the identification of these aspects"*.

Further, we would like to comment on two points asked by both Reviewer FjRx and Reviewer 62uG.

Regarding the motivation of used distributions, we approach the issue of benchmark construction by casting a wide net of tasks. We decided to focus on the following phenomena:

1. **Dimensionality**. High-dimensional datasets are becoming more common, particularly in machine learning and systems biology.
2. **Sparsity**. While the data might be high-dimensional, the effective dimension may be much smaller (e.g., only a few genes out of thousands convey information about the amount of a particular protein).
5. **High MI**. Estimating high MI is known to be difficult. However, it is usually something we might know approximately in advance -- if there are 4000 image classes, MI between image class and representation is at most 12 bits. An additional interesting observation is that CCA performes very well, suggesting that in this scenario incorporating prior information is crucial.
6. **Long tails**. Since the Student distribution has heavier tails than the multivariate normal distribution, this was a natural choice. An interesting conclusion is that even after "removing the tails" (see Appendix E), these distributions remain difficult. Thus, we have low- and moderate-dimensional distributions, which are unimodal and without heavy tails, which are still challenging to estimate.
7. **Robustness to diffeomorphisms**. Mutual information is often chosen because it is theoretically invariant to diffeomorphisms. We wanted to challenge this invariance when only a finite sample is available.


Secondly, we would like to note that our main contribution lies in constructing a general benchmark and the analysis of its results; we do not consider Theorem 2.1 (a necessary tool to construct our benchmark) to be a novel contribution of this paper (Reviewers FjRx, 62uG), as we feel it has been known by the community for a long time. However, we could not find the proof in the literature (and the proof given by Kraskov et al. (2004), e.g., does not cover singular measures and mappings other than diffeomorphisms), so we included our proof in the Appendix for completeness.

Meta-review: This paper considers the important problem of estimating the mutual information (MI) between two variables. The main contribution is a benchmark for MI estimators with known ground-truth that encompasses several relevant phenomena: dimensionality, sparsity, and heavy tails. This benchmark could have substantial long-term impact as more sophisticated estimators are constructed in the future.
Decision: Accept (poster)