Title: Beyond Normal: On the Evaluation of Mutual Information Estimators
Venue: NeurIPS 2023
Accepted: yes
Keywords: Mutual Information

Invariance

Benchmark

Geometric Machine Learning
Abstract: Mutual information is a general statistical dependency measure which has found applications in representation learning, causality, domain generalization and computational biology. However, mutual information estimators are typically evaluated on simple families of probability distributions, namely multivariate normal distribution and selected distributions with one-dimensional random variables. In this paper, we show how to construct a diverse family of distributions with known ground-truth mutual information and propose a language-independent benchmarking platform for mutual information estimators. We discuss the general applicability and limitations of classical and neural estimators in settings involving high dimensions, sparse interactions, long-tailed distributions, and high mutual information. Finally, we provide guidelines for practitioners on how to select appropriate estimator adapted to the difficulty of problem considered and issues one needs to consider when applying an estimator to a new data set.

Beyond Normal: On the Evaluation of Mutual Information Estimators

Abstract

Mutual information is a general statistical dependency measure which has found applications in representation learning, causality, domain generalization and computational biology. However, mutual information estimators are typically evaluated on simple families of probability distributions, namely multivariate normal distribution and selected distributions with one-dimensional random variables. In this paper, we show how to construct a diverse family of distributions with known ground-truth mutual information and propose a language-independent benchmarking platform for mutual information estimators. We discuss the general applicability and limitations of classical and neural estimators in settings involving high dimensions, sparse interactions, long-tailed distributions, and high mutual information. Finally, we provide guidelines for practitioners on how to select appropriate estimator adapted to the difficulty of problem considered and issues one needs to consider when applying an estimator to a new data set.

1 Introduction

Estimating the strength of a non-linear dependence between two continuous random variables (r.v.) lies at the core of machine learning. Mutual information (MI) lends itself naturally to this task, due to its desirable properties, such as invariance to homeomorphisms and the data processing inequality. It finds applications in domain generalization [Li et al., 2022, Ragonesi et al., 2021], representation learning [Belghazi et al., 2018, Oord et al., 2018], causality [Solo, 2008, Kurutach et al., 2018], physics [Keys et al., 2015], systems biology [Selimkhanov et al., 2014, Grabowski et al., 2019, Uda, 2020], and epidemiology [Young et al., 2023].

Over the last decades, the estimation of MI has been extensively studied, and estimators have been developed, ranging from classical approaches based on histogram density [Pizer et al., 1987], kernel density estimation [Moon et al., 1995] to k -nearest neighbor [Kozachenko and Leonenko, 1987, Kraskov et al., 2004] and neural estimators [Belghazi et al., 2018, Oord et al., 2018, Song and Ermon, 2020]. However, despite the progress that has been achieved in this area, not much attention has been focused on systematically benchmarking these approaches.

Typically, new estimators are evaluated assuming multivariate normal distributions for which MI is analytically tractable [Darbellay and Vajda, 1999, Kraskov et al., 2004, Suzuki, 2016]. Sometimes, simple transformations are applied to the data [Khan et al., 2007, Gao et al., 2015], moderately high-dimensional settings are considered [Lord et al., 2018, Lu and Peltonen, 2020], or strong dependencies are evaluated [Gao et al., 2015]. Beyond that, Song and Ermon [2020] study the

2.5

0.0

2.5

X

2.5

0.0

2.5

Y

0

1

CDF(

X

)

0.0

0.5

1.0

CDF(

Y

)

Correlated uniform

5

0

5

h

(

X

)

5

0

5

h

(

Y

)

Half-cube mapping

0

5

bimodal(

X

)

0

5

bimodal(

Y

)

Making margins bimodal

0.0

0.5

0.0

0.5

0

1

Swiss roll mapping

self-consistency (additivity under independence and the data processing inequality) of neural MI estimators in the context of representation learning from image data.

Our Contributions In this work we show a method of developing expressive distributions with known ground-truth mutual information (Sec. 2), propose forty benchmark tasks and systematically study the properties of commonly used estimators, including representatives based on kernel or histogram density estimation, k NN estimation, and neural network-based estimators (Sec. 3). We address selected difficulties one can encounter when estimating mutual information (Sec. 4), such as sparsity of interactions, long-tailed distributions, invariance, and high mutual information. Finally, we provide recommendations for practitioners on how to choose a suitable estimator for particular problems (Sec. 6). Our benchmark is designed so that it is simple to add new tasks and estimators. It also allows for cross-language comparisons - in our work we compare estimators implemented in Python, R and Julia. All of our experiments are fully reproducible by running Snakemake workflows [Mölder et al., 2021]. Accompanying code is available at http://github.com/cbg-ethz/bmi .

Overall, our key findings from the benchmark can be summarized as follows:

Testing on multivariate normal distributions gives a biased and overly optimistic view of estimator performance. In this setting, canonical correlation analysis (CCA), a model-based approach, emerges as the most effective method - even if the model is slightly misspecified.

Compared to classical estimators, neural estimators excel in high-dimensional settings, capturing sparse interactions across multiple dimensions.

The popular KSG estimator [Kraskov et al., 2004] is accurate in low- and moderate-dimensional settings, but its performance suffers on problems involving high-dimensions or sparse interactions.

Although MI is invariant to a wide range of transformations (Theorem 2.1), numerical estimates, even with large sample sizes, are not. Thus, MI may not be suitable when metric invariance to specific transformation families is required.

Multivariate Student distributions pose an important and hard challenge to many mutual information estimators. This effect is partially, but not fully, attributed to their long tails.

2 Mutual Information: Estimation and Invariance

We start by recalling the definition of MI, then discuss the problem of estimating it, and introduce the estimators used in the benchmark.

Consider two r.v. X and Y with domains X respectively Y , joint probability measure P XY and marginal probability measures P X and P Y , respectively. If P XY is absolutely continuous with respect to P X ⊗ P Y (e.g., X and Y are finite r.v.), MI is equal to the following Kullback-Leibler divergence 2 :

where f = d P XY / d( P X ⊗ P Y ) is the Radon-Nikodym derivative. If the absolute continuity does not hold, then I ( X ; Y ) = + ∞ [Pinsker and Feinstein, 1964, Theorem 2.1.2]. If X and Y are Euclidean spaces and measures P XY , P X and P Y have probability density functions (PDFs) with respect to the Lebesgue measure, then the Kullback-Leibler divergence can be written in terms of the PDFs.

However, we will later consider distributions which do not have PDFs with respect to the Lebesgue measure and the general Radon-Nikodym derivative must be used (see 'Swiss roll' in Fig. 1).

In almost all applications, P XY and P X ⊗ P Y are not known and one needs to estimate MI from a finite sample from P XY , i.e., a realization ( ( x 1 , y 1 ) , . . . , ( x N , y N ) ) ∈ ( X × Y ) N of N i.i.d. r.v. ( X 1 , Y 1 ) , . . . , ( X N , Y N ) distributed according to the joint distribution P XY . As a MI estimator we will denote a family of measurable mappings, indexed by the sample size N and spaces X and Y (in most applications, Euclidean spaces of various dimensionalities): e XY N : ( X ×Y ) N → R . Composing this mapping with the r.v. representing the whole data set we obtain a real-valued r.v. E XY N . For a given P XY we can consider the distribution of E XY N . It is often summarized by its first two moments, resulting in the bias and the variance of the estimator. Understanding the bias of an estimator however requires the knowledge of ground-truth MI, so the estimators are typically tested on the families of jointly multivariate normal or uniform distributions, with varying N and ground-truth I ( X ; Y ) [Khan et al., 2007, Lord et al., 2018, Holmes and Nemenman, 2019, Song and Ermon, 2020].

Constructing Expressive Distributions and Benchmarking Besides considering different types of distributions such as (multivariate) normal, uniform and Student, our benchmark is based on the invariance of MI to the family of chosen mappings. Generally MI is not invariant to arbitrary transformations, as due to the data processing inequality I ( f ( X ); g ( Y )) ≤ I ( X ; Y ) . However, under some circumstances MI is preserved, which enables us to create more expressive distributions by transforming the variables.

Theorem 2.1. Let X , X ′ , Y and Y ′ be standard Borel spaces (e.g., smooth manifolds with their Borel σ -algebras) and f : X → X ′ and g : Y → Y ′ be continuous injective mappings. Then, for every X -valued r.v. X and Y -valued r.v. Y it holds that I ( X ; Y ) = I ( f ( X ); g ( Y )) .

This theorem has been studied before by Kraskov et al. [2004, Appendix], who consider diffeomorphisms and by Polyanskiy and Wu [2022, Th. 3.7], who assume measurable injective functions with measurable left inverses. For completeness we provide a proof which covers continuous injective mappings in Appendix A. In particular, each of f and g can be a homeomorphism, a diffeomorphism, or a topological embedding. Embeddings are allowed to increase the dimensionality of the space, so even if X and Y have probability density functions, f ( X ) and g ( Y ) do not need to have them. Neither of these deformations change MI. Thus, using Theorem 2.1, we can create more expressive distributions P f ( X ) g ( Y ) by sampling from P XY and transforming the samples to obtain a data set ( ( f ( x 1 ) , g ( y 1 )) , . . . , ( f ( x N ) , g ( y N ) ) ∈ ( X ′ ×Y ′ ) N .

While offering a tool to construct expressive distributions, another valuable perspective to consider with this problem is estimation invariance: although MI is invariant to proposed changes, the estimate may not be. More precisely, applying an estimator to the transformed data set results in the induced r.v. E f ( X ) g ( Y ) N and its distribution. If the estimator were truly invariant, one would expect that E [ E f ( X ) g ( Y ) N ] should equal E [ E XY N ] . This invariance is rarely questioned and often implicitly assumed as given (see e.g. Tschannen et al. [2020] or Murphy [2023, Sec. 32.2.2.3]), but as we will see in Sec. 4.3, finite-sample estimates are not generally invariant.

3 Proposed Benchmark

In this section, we outline forty tasks included in the benchmark that cover a wide family of distributions, including varying tail behaviour, sparsity of interactions, multiple modes in PDF, and transformations that break colinearity. As our base distributions, we selected multivariate normal and Student distributions, which were transformed with continuous injective mappings.

We decided to focus on the following phenomena:

Dimensionality. High-dimensional data are collected in machine learning and natural sciences [Bühlmann and van de Geer, 2011, Ch. 1]. We therefore change the dimensions of the X and Y spaces between 1 and 25 .

Sparsity. While the data might be high-dimensional, the effective dimension may be much smaller, due to correlations between different dimensions or sparsity [Lucas et al., 2006]. We therefore include distributions in which some of the dimensions represent random noise,

which does not contribute to the mutual information, and perform an additional study in Section 4.1.

Varying MI. Estimating high MI is known to be difficult [McAllester and Stratos, 2020]. However, we can often bound it in advance - if there are 4000 image classes, MI between image class and representation is at most log 4000 ≈ 8 . 3 nats. In this section, we focus on problems with MI varying up to 2 nats. We additionally consider distributions with higher MI in Section 4.4.

Long tails. As Taleb [2020] and Zhang et al. [2021] argue, many real-world distributions have long tails. To model different tails we consider multivariate Student distributions as well as transformations lengthening the tails. We conduct an additional study in Section 4.2.

Robustness to diffeomorphisms. As stated in Theorem 2.1, mutual information is invariant to reparametrizations of random variables by diffeomorphisms. We however argue that when only finite samples are available, invariance in mutual information estimates may not be achieved. To test this hypothesis we include distributions obtained by using a diverse set of diffeomorphisms and continuous injective mappings. We further study the robustness to reparametrizations in Section 4.3.

While we provide a concise description here, precise experimental details can be found in Appendix D and we visualise selected distributions in Appendix F. We first describe tasks employing onedimensional variables.

Bivariate Normal For multivariate normal variables MI depends only on the correlation matrix. We will therefore consider a centered bivariate normal distribution P XY with Cor( X,Y ) = ρ and I ( X ; Y ) = -0 . 5 log ( 1 -ρ 2 ) . We chose ρ = 0 . 75 .

Uniform Margins As a first transformation, we apply the Gaussian CDF F to Gaussian r.v. X and Y to obtain r.v. X ′ = F ( X ) and Y ′ = F ( Y ) . It is a standard result in copula theory (see Lemma D.2 or e.g., Nelsen [2007] for an introduction to copulas) that F ( X ) ∼ Uniform(0 , 1) resp. F ( Y ) ∼ Uniform(0 , 1) . The joint distribution P X ′ Y ′ , however, is not uniform. Mutual information is preserved, I ( X ′ ; Y ′ )= I ( X ; Y ) . For an illustration, see Fig. 1. A distribution P transformed with Gaussian CDF F will be denoted as Normal CDF @ P .

Half-Cube Map To lengthen the tails we applied the half-cube homeomorphism h ( x )= | x | 3 / 2 sign x to Gaussian variables X and Y . We visualize an example in Fig. 1 and denote a transformed distribution as Half-cube @ P .

Asinh Mapping To shorten the tails we applied inverse hyperbolic sine function asinh x = log ( x + √ 1 + x 2 ) . A distribution transformed with this mapping will be denoted as Asinh @ P .

Wiggly Mapping To model non-uniform lengthscales, we applied a mapping

Due to the inequality constraint this mapping is injective and preserves MI. The parameter values can be found in Appendix D. We denote the transformed distribution as Wiggly @ P .

Bimodal Variables Applying the inverse CDF of a two-component Gaussian mixture model to correlated variables X and Y with uniform margins we obtained a joint probability distribution P X ′ Y ′ with four modes (presented in Fig. 1). We call this distribution Bimodal and provide technical details in Appendix D.

Additive Noise Consider independent r.v. X ∼ Uniform(0 , 1) and N ∼ Uniform( -ε, ε ) , where ε is the noise level. For Y = X + N , it is possible to derive I ( X ; Y ) analytically (see Appendix D for the formula and the parameter values) and we will call this distribution Uniform (additive noise= ε ) .

Swiss Roll Embedding As the last example in this section, we consider a mixed case in which a one-dimensional r.v. X ∼ Uniform(0 , 1) is smoothly embedded into two dimensions via the Swiss roll mapping, a popular embedding used to test dimensionality reduction algorithms (see Fig. 1 for visualisation and Appendix D for the formula). Note that the obtained distribution does not have a PDF with respect to the Lebesgue measure on R 3 .

Next, we describe the tasks based on multivariate distributions.

Multivariate Normal We sampled ( X,Y ) = ( X 1 , . . . , X m , Y 1 , . . . , Y n ) from the multivariate normal distribution. We model two correlation structures: 'dense' interactions with all off-diagonal correlations set to 0.5 and '2-pair' interactions where Cor( X 1 , Y 1 ) = Cor( X 2 , Y 2 ) = 0 . 8 and there is no correlation between any other (distinct) variables ( Multinormal (2-pair) ). For a latent-variable interpretation of these covariance matrices we refer to Appendix D.2.

Multivariate Student To see how well the estimators can capture MI contained in the tails of the distribution, we decided to use multivariate Student distributions (see Appendix D) with dispersion matrix 3 set to the identity I m + n and ν degrees of freedom. Contrary to the multivariate normal case, in which the identity matrix used as the covariance would lead to zero MI, the variables X and Y can still interact through the tails.

Transformed Multivariate Distributions As the last set of benchmark tasks we decided to transform the multivariate normal and multivariate Student distributions. We apply mappings described above to each axis separately. For example, applying normal CDF to the multivariate normal distribution will map it into a distribution over the cube (0 , 1) m + n with uniform marginals. To mix different axes we used a spiral diffeomorphism (denoted as Spiral @ P ); we defer the exact construction to Appendix D but we visualise this transformation in Fig. 5.

Estimators In our benchmark, we include a diverse set of estimators. Following recent interest, we include four neural estimators, i.e., Donsker-Varadhan (D-V) and MINE [Belghazi et al., 2018], InfoNCE [Oord et al., 2018], and the NWJ estimator [Nguyen et al., 2007, Nowozin et al., 2016, Poole et al., 2019]. As a representative for model-based estimators, we implement an estimator based on canonical correlation analysis (CCA) [Murphy, 2023, Ch. 28], which assumes that the joint distribution is multivariate normal. For more classical estimators, we include the Kraskov, Stögbauer and Grassberger (KSG) estimator [Kraskov et al., 2004] as the most well-known k -nearest neighbourbased estimator, a recently proposed kernel-based estimator (LNN) [Gao et al., 2017b], as well as an estimator based on histograms and transfer entropy (both implemented in Julia's TransferEntropy library). A detailed overview of the different classes of estimators is provided in Appendix C. Further, we describe the the hyperparameter selection in Appendix E.

Preprocessing As a data preprocessing step, we standardize each dimension using the mean and standard deviation calculated from the sample. Other preprocessing stategies are also possible, but we did not observe a consistent improvement of one preprocessing strategy over the others (see Appendix E).

3.1 Benchmark Results

We show the results for N = 10000 data points in Fig. 2. Neural estimators and KSG perform better than alternative estimators on most problems, with KSG having often low sample requirements (see Appendix E). The simple model-based estimator CCA obtained excellent performance at low sample sizes (see Appendix E), even for slightly transformed multivariate normal distributions. Finally, LNN and the two Julia estimators (histogram and transfer entropy), work well in low dimension but are not viable in medium- and high-dimensional problems ( dim X +dim Y ≥ 4 ).

Overall, we observed that for most 1 × 1 -dimensional and multivariate normal problems MI can be reliably estimated. Difficulties arise for sparse interactions, atypical (Student) distributions, and for significantly transformed distributions. This suggests that the typical evaluations of MI estimators are overly optimistic, focusing on relatively simple problems.

KSG is able to accurately estimate MI in multivariate normal problems, however performance drops severely on tasks with sparse interactions (2-pair tasks). For 5 × 5 dimensions the estimate is about 70% of the true MI, and for 25 × 25 dimensions it drops to 10%, which is consistent with previous observations [Marx and Fischer, 2022]. Meanwhile, performance of neural estimators is stable. We study the effect of sparsity in more detail in Sec. 4.

Student distributions have proven challenging to all estimators. This is partially due to the fact that long tails can limit estimation quality (see Sec. 4), and indeed, applying a tail-shortening asinh transformation improves performance. However, performance still remains far below multivariate normal distributions with similar dimensionality and information content, showing that long tails are

Bivariate normal 1 × 1

Normal CDF @ Bivariate normal 1 × 1

Uniform 1 × 1 (additive noise=0.1)

Uniform 1 × 1 (additive noise=0.75)

Bimodal 1 × 1

Wiggly @ Bivariate normal 1 × 1

Half-cube @ Bivariate normal 1 × 1

Student-t 1 × 1 (dof=1)

Asinh @ Student-t 1 × 1 (dof=1)

Swiss roll 2 × 1

Multinormal 2 × 2 (dense)

Multinormal 3 × 3 (dense)

Multinormal 5 × 5 (dense)

Multinormal 25 × 25 (dense)

Multinormal 50 × 50 (dense)

Multinormal 2 × 2 (2-pair)

Multinormal 3 × 3 (2-pair)

Multinormal 5 × 5 (2-pair)

Multinormal 25 × 25 (2-pair)

Student-t 2 × 2 (dof=1)

Student-t 2 × 2 (dof=2)

Student-t 3 × 3 (dof=2)

Student-t 3 × 3 (dof=3)

Student-t 5 × 5 (dof=2)

Student-t 5 × 5 (dof=3)

Normal CDF @ Multinormal 3 × 3 (2-pair)

Normal CDF @ Multinormal 5 × 5 (2-pair)

Normal CDF @ Multinormal 25 × 25 (2-pair)

Half-cube @ Multinormal 3 × 3 (2-pair)

Half-cube @ Multinormal 5 × 5 (2-pair)

Half-cube @ Multinormal 25 × 25 (2-pair)

Spiral @ Multinormal 3 × 3 (2-pair)

Spiral @ Multinormal 5 × 5 (2-pair)

Spiral @ Multinormal 25 × 25 (2-pair)

Spiral @ Normal CDF @ Multinormal 3 × 3 (2-pair)

Spiral @ Normal CDF @ Multinormal 5 × 5 (2-pair)

Spiral @ Normal CDF @ Multinormal 25 × 25 (2-pair)

Asinh @ Student-t 2 × 2 (dof=1)

Asinh @ Student-t 3 × 3 (dof=2)

Asinh @ Student-t 5 × 5 (dof=2)

MINE

NWJ

D-V

KSG I (R)

LNN (R)

Hist. (Julia)

Transfer (Julia)

CCA

True MI

0.4

0.4

1.7

0.3

0.3

0.4

0.4

0.1

0.2

0.4

0.3

0.4

0.6

1.2

1.5

0.9

0.9

0.9

0.8

0.1

0.1

0.2

0.1

0.3

0.1

0.9

0.9

0.8

0.9

0.9

0.8

0.4

0.5

0.7

0.8

0.8

0.8

0.2

0.1

0.2

0.4

0.4

1.7

0.3

0.4

0.4

0.4

0.1

0.2

0.4

0.3

0.4

0.6

1.2

1.4

1.0

1.0

1.0

0.9

0.3

0.2

0.2

0.1

0.4

0.2

0.9

0.9

0.8

1.0

1.0

0.8

0.6

0.6

0.7

0.9

0.9

0.8

0.4

0.2

0.3

0.4

0.4

1.7

0.3

0.4

0.4

0.4

0.0

0.1

0.4

0.3

0.4

0.6

1.2

1.3

1.0

1.0

1.0

0.9

0.0

0.0

0.1

0.1

0.2

0.1

0.9

0.9

0.8

1.0

1.0

0.8

0.5

0.5

0.7

0.9

0.9

0.7

0.4

0.1

0.3

0.4

0.4

1.7

0.3

0.4

0.4

0.4

0.1

0.2

0.4

0.3

0.4

0.6

1.2

1.5

1.0

1.0

1.0

0.9

0.1

0.1

0.1

0.1

0.1

0.1

0.9

0.9

0.8

1.0

1.0

0.8

0.5

0.5

0.7

0.9

0.9

0.8

0.4

0.2

0.1

0.4

0.4

1.7

0.3

0.4

0.4

0.4

0.2

0.2

0.4

0.3

0.4

0.6

1.0

1.1

1.0

1.0

0.7

0.1

0.3

0.1

0.1

0.1

0.2

0.2

0.9

0.7

0.1

0.9

0.7

0.1

0.8

0.6

0.1

0.9

0.7

0.1

0.4

0.2

0.2

0.3

0.3

1.6

0.2

0.3

0.3

0.3

0.1

0.1

0.7

1.3

2.7

1.4

1.9

3.1

2.3

1.0

0.7

1.5

1.3

3.7

3.4

1.6

2.4

5.7

1.9

2.9

3.9

1.8

2.9

2.3

1.6

2.4

5.7

0.9

1.3

3.5

0.4

0.4

1.3

0.3

0.4

0.4

0.3

0.0

0.1

0.4

0.4

1.2

5.6

9.2

9.2

0.9

1.8

6.7

9.2

0.0

0.0

0.1

0.1

0.1

0.4

4.5

9.1

9.2

1.0

3.1

9.2

1.8

6.7

9.2

3.4

9.0

9.2

0.3

1.1

4.5

0.4

0.4

1.6

0.3

0.4

0.4

0.4

0.0

0.2

0.6

1.8

6.3

9.4

9.0

9.0

2.3

6.9

9.3

9.0

0.0

0.1

0.3

0.8

0.6

2.9

8.9

9.0

9.0

4.0

8.7

9.0

6.9

9.3

9.0

8.5

9.0

9.0

1.1

5.4

9.2

0.4

0.4

1.6

0.2

0.3

0.4

0.4

0.3

0.0

0.0

0.3

0.4

0.6

1.3

1.7

1.0

1.0

1.0

1.1

1.5

0.1

0.1

0.0

0.3

0.0

1.0

1.0

1.0

1.0

1.0

1.0

0.2

0.4

0.9

0.8

0.9

1.0

0.0

0.0

0.0

0.4 0.4 1.7 0.3 0.4 0.4 0.4 0.2 0.2 0.4 0.3 0.4 0.6 1.3 1.6 1.0 1.0 1.0 1.0 0.4 0.2 0.3 0.2 0.4 0.3 1.0

1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 0.4 0.3 0.4

only part of the difficulty. For neural estimators, the issue could be explained by the fact that the critic is not able to fully learn the pointwise mutual information (see Appendix E).

Interestingly, the performance of CCA, which assumes a linear Gaussian model, is often favorable compared to other approaches and has very low sample complexity (see Appendix E). This is surprising, since in the case of transformed distributions the model assumptions are not met. The approach fails for Student distributions (for which the correlation matrix either does not exist or is the identity matrix), the Swiss roll embedding and multivariate normal distributions transformed using the spiral diffeomorphism. Since the spiral diffeomorphism proved to be the most challenging transformation, we study it more carefully in Sec. 4.

4 Selected Challenges

In this section we investigate distributions which proved to be particularly challenging for the tested estimators in our benchmark: sparse interactions, long tailed distributions and invariance to diffeomorphisms. Additionally we investigate how the estimators perform for high ground truth MI.

4.1 Sparsity of Interactions

In the main benchmark (Fig. 2) we observed that estimation of MI with '2-pair' type of interactions is considerably harder for the KSG estimator than the 'dense' interactions. On the other hand, neural estimators were able to pick the relevant interacting dimensions and provide better estimates of MI in these sparse cases.

We decided to interpolate between the 'dense' and '2-pair' interaction structures using a two-stage procedure with real parameters α and λ and the number of strongly interacting components K . While the precise description is in Appendix D.2, we can interpret α as controlling the baseline strength of interaction between every pair of variables in { X 1 , . . . , X 10 , Y 1 , . . . Y 10 } and λ as the additional strength of interaction between pairs of variables ( X 1 , Y 1 ) , . . . , ( X K , Y K ) . Dense interaction structure has λ = 0 and 2-pair interaction structure has α = 0 and K = 2 . First, we set K = 10 and λ ≈ 0 and decrease α raising λ at the same time to maintain constant MI of 1 nat. When α = 0 , whole information is contained in the pairwise interactions ( X 1 , Y 1 ) , . . . , ( X 10 , Y 10 ) . We then decrease K and increase λ , still maintaining constant MI.

In Fig. 3 we observe that the performance of all estimators considered (apart from CCA, a modelbased approach suitable for multivariate normal distributions) degrades when the interactions between

0.0

0.5

1.0

0.00

0.25

0.50

0.75

1.00

MI estimate [nats]

1

2

3

4

5

6

7

8

10 9

Num. of interactions

0.00

0.25

0.50

0.75

1.00

MI estimate [nats]

CCA

D-V

MINE

NWJ

KSG I (R)

True MI

2

4

Exponent

0

1

2

MI estimate [nats]

5

10

15

Deg. of freedom

0

1

2

3

MI estimate [nats]

CCA

D-V

Hist. (Julia)

Transfer (Julia)

MINE

NWJ

KSG I (R)

LNN (R)

True MI

every pair of variables become sparser (lower α ). In particular, even neural estimators are not able to model the information in ten interacting pairs of variables. However, when we decrease the number of interacting pairs of variables, the performance of the neighborhood-based KSG estimator is qualitatively different from the neural estimators: performance of KSG steadily deteriorates, while neural estimators can find (some of the) relevant pairs of variables.

This motivates us to conclude that in the settings where considered variables are high-dimensional and the interaction structure is sparse, neural estimators can offer an advantage over neighborhood-based approaches.

4.2 Long-Tailed Distributions

In Fig. 4 we investigate two different ways to lengthen the tails. First, we lengthen the tails of a multivariate normal distribution using the mapping x ↦→ | x | k sign x (with k ≥ 1 ) applied to each dimension separately. In this case, we see that the performance of CCA, KSG, and neural estimators is near-perfect for k close to 1 (when the distribution P XY is close to multivariate normal), but the performance degrades as the exponent increases. Second, we study multivariate Student distributions varying the degrees of freedom: the larger the degrees of freedom, the lower the information content in the tails of the distribution. Again, we see that that the performance of CCA, KSG, and neural estimators is near-perfect for high degrees of freedom for which the distribution is approximately normal. For low degrees of freedom, these estimators significantly underestimate MI, with the exception of CCA which gives estimates with high variance, likely due to the correlation matrix being hard to estimate.

Overall, we see that long tails complicate the process of estimating MI. The Student distribution is particularly interesting, since even after the tails are removed (with asinh or preprocessing strategies described in Appendix E) the task remains challenging. We suspect that the MI might be contained in regions which are rarely sampled, making the MI hard to estimate. When neural estimators are used, pointwise mutual information learned by the critic does not match the ground truth (see Appendix E).

Hence, we believe that accurate estimation of MI in long-tailed distributions is an open problem. Practitioners expecting long-tailed distributions should approach this task with caution.

2

1

0

1

2

Coordinate

x

1

2

1

0

1

2

Coordinate

x

2

Original

Speed 0.5

Speed 1.5

0.0

0.5

1.0

1.5

Speed

0.0

0.2

0.4

0.6

0.8

MI estimate [nats]

CCA

D-V

Hist. (Julia)

Transfer (Julia)

MINE

NWJ

KSG I (R)

LNN (R)

True MI

4.3 Invariance to Diffeomorphisms

The benchmark results (Fig. 2) as well as the experiments with function x ↦→| x | k sign x lengthening tails (Fig. 4) suggest that even if ground-truth MI is preserved by a transformation (see Theorem 2.1), the finite-sample estimates may not be invariant. This can be demonstrated using the spiral diffeomorphism.

We consider isotropic multivariate normal variables X = ( X 1 , X 2 ) ∼ N (0 , I 2 ) and Y ∼ N (0 , 1) with Cor( X 1 , Y ) = ρ . Then, we apply a spiral diffeomorphism s v : R 2 → R 2 to X , where

is the rotation around the origin by the angle v || x || 2 (see Fig. 5; for a theoretical study of spiral diffeomorphisms see Appendix B). In Fig. 5, we present the estimates for N = 10000 , ρ = 0 . 8 and varying speed v .

In general, the performance of the considered estimators deteriorates with stronger deformations. This is especially visible for the CCA estimator, as MI cannot be reliably estimated in terms of correlation.

4.4 High Mutual Information

Gao et al. [2015] proves that estimation of MI between strongly interacting variables requires large sample sizes. Poole et al. [2019], Song and Ermon [2020] investigate estimation of high MI using multivariate normal distributions in the setting when the ground-truth MI is changing over the training time. We decided to use our family of expressive distributions to see the practical limitations of estimating MI. We chose three one-parameter distribution families: the 3 × 3 multivariate normal distribution with correlation ρ = Cor( X 1 , Y 1 ) = Cor( X 2 , Y 2 ) , and its transformations by the halfcube mapping and the spiral diffeomorphism with v = 1 / 3 . We changed the parameter to match the desired MI.

In Fig. 6 we see that for low enough MI neural estimators and KSG reliably estimate MI for all considered distributions, but the moment at which the estimation becomes inaccurate depends on the distribution. For example, for Gaussian distributions, neural estimators (and the simple CCA baseline) are accurate even up to 5 nats, while after the application of the spiral diffeomorphism, the estimates become inaccurate already at 2 nats.

We see that in general neural estimators perform well on distributions with high MI. The performance of CCA on the multivariate normal distribution suggests that well-specified model-based estimators may require lower number of samples to estimate high mutual information than model-free alternatives (cf. Gao et al. [2015]).

5 Related Work

Existing Benchmarks Khan et al. [2007] used one-dimensional variables and an additive noise assumption to evaluate the estimators based on histograms, nearest neighbors, and kernels. However,

0

2

4

True MI [nats]

0

2

4

6

MI estimate [nats]

Half-cube

0

2

4

True MI [nats]

0

2

4

6

MI estimate [nats]

Sparse Gaussian

0

2

4

True MI [nats]

0

2

4

6

MI estimate [nats]

Spiral

CCA

D-V

Hist. (Julia)

Transfer (Julia)

MINE

NWJ

KSG I (R)

LNN (R)

True MI

the original code and samples are not available. Song and Ermon [2020] proposed to evaluate neural estimators based on variational lower bounds [Oord et al., 2018, Belghazi et al., 2018] on image data. However, besides the multivariate Gaussian which they also evaluate, the ground-truth MI is not available. Therefore, they proposed self-consistency checks, which include I ( X ; Y )=0 for independent X and Y , and preservation of the data processing inequality. Poole et al. [2019] study neural estimators in settings employing Gaussian variables, affine transformation followed by axis-wise cubic transformation and investigate the high MI cases. In our benchmark, we focus on distributions with known MI and information-preserving continuous injective mappings such as homeomorphisms, evaluate long tail distributions, and compare representative estimators from different classes and platforms. We summarize the differences between the benchmarks in Table 1. 4

Causal Representation Learning and Nonlinear ICA Recent work on disentanglement and nonlinear independent component analysis (ICA) aims at identifying causal sources up to an ambiguous transformation [Xi and Bloem-Reddy, 2022]. Examples include orthogonal transformations [Zimmermann et al., 2021], permutations and diffeomorphisms applied along specific axes [Khemakhem et al., 2020], or the whole diffeomorphism group [Von Kügelgen et al., 2021, Daunhawer et al., 2023]. To check whether the learned representations indeed capture underlying causal variables, it is important to use estimators which are invariant to the ambiguous transformations specific to the model. Our results suggest that such requirements are not met by MI estimators and practitioners should carefully choose the used metrics, so they are not only theoretically invariant, but also can be robustly estimated in practice.

6 Conclusions and Limitations

Venturing beyond the typical evaluation of mutual information estimators on multivariate normal distributions, we provided evidence that the evaluation on Gaussian distributions results in overly optimistic and biased results for existing estimators. In particular, a simple model-based approach as

CCA is already sufficient if the distribution is (close to) multivariate normal, highlighting the effect of using available prior information about the problem to design efficient estimators. To provide a basis for more thorough evaluation, we designed a benchmark, compatible with any programming language, with forty general tasks focusing on selected problems, as estimation with sparse interactions, the effects of long tails, estimation in problems with high MI, and (only theoretical) invariance to diffeomorphisms.

Our results suggest that in low- and medium-dimensional problems the KSG estimator remains a solid baseline, requiring a lower number of samples than the alternatives and no parameter tuning. In high-dimensional settings where the interactions are sparse, neural estimators offer an advantage over classical estimators being able to more consistently pick up the signal hidden in the highdimensional space. We further provide evidence that although MI is invariant to continuous injective transformations of variables, in practice the estimates can differ substantially. Appropriate data preprocessing may mitigate some of these issues, but no strategy clearly outperforms the others (see Appendix E).

We hope that the proposed benchmark encourages the development of new estimators, which address the challenges that we highlight in this paper. It can be used to assess invariance to homeomorphisms, understand an estimator's behavior in the situations involving distributions of different complexity, low-data behaviour, high mutual information or to diagnose problems with implementations. We believe that some parts of the benchmark could also be used to assess whether other estimators of other statistical quantities, such as kernel dependence tests, transfer entropy or representation similarity measures, can be reliably estimated on synthetic problems and if they are robust to selected classes of transformations.

Limitations and Open Problems Although the proposed benchmark covers a wide family of distributions, its main limitation is the possibility to only transform the distribution P XY via transformations f × g to P f ( X ) g ( Y ) starting from multivariate normal and Student distributions. In particular, not every possible joint distribution is of this form and extending the family of distributions which have known MI and allow efficient sampling is the natural direction of continuation. We are, however, confident that due to the code design it will be easy to extend the benchmark with the new distributions (and their transformations) when they appear.

Estimation of information in multivariate Student distributions remains a challenge for most estimators. Although we provide the evidence that distributions with longer tails pose a harder challenge for considered estimators, shortening the tails with asinh transform did not alleviate all the issues. In neural estimators this can be partly explained by the fact that the critic may not learn to approximate pointwise mutual information (up to an additive constant) properly, as demonstrated in Appendix E, but the question of what makes the neighborhood-based estimator, KSG, fail, remains open.

As noted, unless strong inductive biases can be exploited (as illustrated for CCA), estimation of high MI is an important challenge. An interesting avenue towards developing new estimators can be by incorporating a stronger inductive bias, which could also take a form of development of better data normalization strategies and, in case of neural estimators, critic architectures. Better normalization strategies may make the estimators more invariant to considered transformations.

Acknowledgments and Disclosure of Funding

We would like to thank Craig Hamilton for the advice on scientific writing and help with the abstract. FG was supported by the Norwegian Financial Mechanism GRIEG-1 grant operated by Narodowe Centrum Nauki (National Science Centre, Poland) 2019/34/H/NZ6/00699. PC and AM were supported by a fellowship from the ETH AI Center.

References

Ziqiao Ao and Jinglai Li. Entropy estimation via normalizing flow. Proceedings of the AAAI Conference on Artificial Intelligence , 36(9):9990-9998, 2022.

R.B. Arellano-Valle, J.E. Contreras-Reyes, and M.G. Genton. Shannon entropy and mutual information for multivariate skew-elliptical distributions. Scandinavian Journal of Statistics , 49:42-62, 2013.

Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International conference on machine learning , pages 531-540. PMLR, 2018.

James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax .

Peter Bühlmann and Sara van de Geer. Statistics for High-Dimensional Data: Methods, Theory and Applications . Springer Publishing Company, Incorporated, 1st edition, 2011. ISBN 3642201911.

Vincent Cabeli, Louis Verny, Nadir Sella, Guido Uguzzoni, Marc Verny, and Hervé Isambert. Learning clinical networks from medical records based on information estimates in mixed-type data. PLOS Computational Biology , 16(5):e1007866, 2020.

Anthony Carapetis. Rotating spherical shells doesn't change volume. Mathematics Stack Exchange, 2022. URL https://math.stackexchange.com/q/4500294 . URL:https://math.stackexchange.com/q/4500294 (version: 2022-07-26).

Christopher J Cellucci, Alfonso M Albano, and Paul E Rapp. Statistical validation of mutual information calculations: Comparison of alternative numerical algorithms. Physical review E , 71 (6):066208, 2005.

Thomas M. Cover and Joy A. Thomas. Elements of Information Theory . Wiley-Interscience New York, 2006.

Georges A Darbellay and Igor Vajda. Estimation of the information by an adaptive partitioning of the observation space. IEEE Transactions on Information Theory , 45(4):1315-1321, 1999.

Carsten O Daub, Ralf Steuer, Joachim Selbig, and Sebastian Kloska. Estimating mutual information using b-spline functions-an improved similarity measure for analysing gene expression data. BMC bioinformatics , 5(1):1-12, 2004.

Imant Daunhawer, Alice Bizeul, Emanuele Palumbo, Alexander Marx, and Julia E Vogt. Identifiability results for multimodal contrastive learning. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=U_2kuqoTcB .

Shuyang Gao, Greg Ver Steeg, and Aram Galstyan. Efficient estimation of mutual information for strongly dependent variables. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 277-286. PMLR, 2015.

Weihao Gao, Sreeram Kannan, Sewoong Oh, and Pramod Viswanath. Estimating mutual information for discrete-continuous mixtures. In Advances in Neural Information Processing Systems , pages 5986-5997, 2017a.

Weihao Gao, Sewoong Oh, and Pramod Viswanath. Density functional estimators with k-nearest neighbor bandwidths. In IEEE International Symposium on Information Theory (ISIT) , pages 1351-1355. IEEE, 2017b.

Weihao Gao, Sewoong Oh, and Pramod Viswanath. Demystifying fixed k-nearest neighbor information estimators. IEEE Transactions on Information Theory , 64(8):5629-5661, 2018.

Frederic Grabowski, Paweł Czy˙ z, Marek Kocha´ nczyk, and Tomasz Lipniacki. Limits to the rate of information transmission through the MAPK pathway. Journal of The Royal Society Interface , 16 (152):20180792, 2019. doi: 10.1098/rsif.2018.0792. URL https://royalsocietypublishing. org/doi/abs/10.1098/rsif.2018.0792 .

Søren Hauberg. Only Bayes should learn a manifold (on the estimation of differential geometric structure from data), 2019.

Caroline M Holmes and Ilya Nemenman. Estimation of mutual information for real-valued data with error bars and controlled bias. Physical Review E , 100(2):022404, 2019.

Kay. Feature discovery under contextual supervision using mutual information. In IJCNN International Joint Conference on Neural Networks , volume 4, pages 79-84, 1992. doi: 10.1109/ IJCNN.1992.227286.

Kechris. Classical Descriptive Set Theory . Graduate Texts in Mathematics. Springer New York, 2012. ISBN 9781461241904. URL https://books.google.ch/books?id=WR3SBwAAQBAJ .

Dustin Keys, Shukur Kholikov, and Alexei A. Pevtsov. Application of mutual information methods in time-distance helioseismology. Solar Physics , 290(3):659-671, Mar 2015. ISSN 1573-093X. doi: 10.1007/s11207-015-0650-y. URL https://doi.org/10.1007/s11207-015-0650-y .

Shiraj Khan, Sharba Bandyopadhyay, Auroop R Ganguly, Sunil Saigal, David J Erickson III, Vladimir Protopopescu, and George Ostrouchov. Relative performance of mutual information estimation methods for quantifying the dependence among short and noisy data. Physical Review E , 76(2): 026209, 2007.

Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ICA: a unifying framework. In International Conference on Artificial Intelligence and Statistics , pages 2207-2217. PMLR, 2020.

Jussi Klemelä. Multivariate histograms with data-dependent partitions. Statistica sinica , pages 159-176, 2009.

Mike Koeman and Tom Heskes. Mutual information estimation with random forests. In International Conference on Neural Information Processing , pages 524-531. Springer, 2014.

F. Kozachenko and N. N. Leonenko. Sample estimate of the entropy of a random vector. Problemy Peredachi Informatsii , 23:9-16, 1987.

Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical Review E , 69(6):066138, 2004.

Thanard Kurutach, Aviv Tamar, Ge Yang, Stuart J Russell, and Pieter Abbeel. Learning plannable representations with causal infogan. Advances in Neural Information Processing Systems , 31, 2018.

Bo Li, Yifei Shen, Yezhen Wang, Wenzhen Zhu, Dongsheng Li, Kurt Keutzer, and Han Zhao. Invariant information bottleneck for domain generalization. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 7399-7407, 2022.

Warren M Lord, Jie Sun, and Erik M Bollt. Geometric k-nearest neighbor estimation of entropy and mutual information. Chaos: An Interdisciplinary Journal of Nonlinear Science , 28(3):033114, 2018.

Chien Lu and Jaakko Peltonen. Enhancing nearest neighbor based entropy estimator for high dimensional distributions via bootstrapping local ellipsoid. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 5013-5020, 2020.

Joseph Lucas, Carlos Carvalho, Quanli Wang, Andrea Bild, Joseph R. Nevins, and Mike West. Sparse Statistical Modelling in Gene Expression Genomics , page 155-176. Cambridge University Press, 2006. doi: 10.1017/CBO9780511584589.009.

Alexander Marx and Jonas Fischer. Estimating mutual information via geodesic k nn. In Proceedings of the SIAM International Conference on Data Mining (SDM) , pages 415-423. SIAM, 2022.

Alexander Marx and Jilles Vreeken. Testing conditional independence on discrete data using stochastic complexity. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 496-505, 2019.

Alexander Marx, Lincen Yang, and Matthijs van Leeuwen. Estimating conditional mutual information for discrete-continuous mixtures using multi-dimensional adaptive histograms. In Proceedings of the SIAM International Conference on Data Mining (SDM) , pages 387-395, 2021.

David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , volume 108 of Proceedings of Machine Learning Research , pages 875-884. PMLR, 26-28 Aug 2020. URL https://proceedings. mlr.press/v108/mcallester20a.html .

Octavio César Mesner and Cosma Rohilla Shalizi. Conditional Mutual Information Estimation for Mixed, Discrete and Continuous, Data. IEEE Transactions on Information Theory , 2020.

Young-Il Moon, Balaji Rajagopalan, and Upmanu Lall. Estimation of mutual information using kernel density estimators. Phys. Rev. E , 52:2318-2321, Sep 1995. doi: 10.1103/PhysRevE.52.2318. URL https://link.aps.org/doi/10.1103/PhysRevE.52.2318 .

Kevin P. Murphy. Probabilistic Machine Learning: Advanced Topics . MIT Press, 2023. URL http://probml.github.io/book2 .

F Mölder, KP Jablonski, B Letcher, MB Hall, CH Tomkins-Tinch, V Sochat, J Forster, S Lee, SO Twardziok, A Kanitz, A Wilm, M Holtgrewe, S Rahmann, S Nahnsen, and J Köster. Sustainable data analysis with snakemake [version 1; peer review: 1 approved, 1 approved with reservations]. F1000Research , 10(33), 2021. doi: 10.12688/f1000research.29032.1.

R.B. Nelsen. An Introduction to Copulas . Springer Series in Statistics. Springer New York, 2007. ISBN 9780387286785. URL https://books.google.ch/books?id=B3ONT5rBv0wC .

XuanLong Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by penalized convex risk minimization. In J. Platt, D. Koller, Y . Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems , volume 20. Curran Associates, Inc., 2007. URL https://proceedings.neurips.cc/paper_files/paper/2007/ file/72da7fd6d1302c0a159f6436d01e9eb0-Paper.pdf .

Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. F-gan: Training generative neural samplers using variational divergence minimization. In Proceedings of the 30th International Conference on Neural Information Processing Systems , NIPS'16, page 271-279, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Liam Paninski. Estimation of entropy and mutual information. Neural Computation , 15(6):11911253, 2003.

Liam Paninski and Masanao Yajima. Undersmoothed kernel entropy estimators. IEEE Transactions on Information Theory , 54(9):4384-4388, 2008.

Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

M.S. Pinsker and A. Feinstein. Information and Information Stability of Random Variables and Processes . Holden-Day series in time series analysis. Holden-Day, 1964. ISBN 9780816268047.

Stephen M Pizer, E Philip Amburn, John D Austin, Robert Cromartie, Ari Geselowitz, Trey Greer, Bart ter Haar Romeny, John B Zimmerman, and Karel Zuiderveld. Adaptive histogram equalization and its variations. Computer Vision, Graphics, and Image Processing , 39(3):355-368, 1987.

Polyanskiy and Y. Wu. Information Theory: From Coding to Learning . Cambridge University Press, 2022. Book draft.

Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning , pages 51715180. PMLR, 2019.

Ruggero Ragonesi, Riccardo Volpi, Jacopo Cavazza, and Vittorio Murino. Learning unbiased representations via mutual information backpropagation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2729-2738, 2021.

Jangir Selimkhanov, Brooks Taylor, Jason Yao, Anna Pilko, John Albeck, Alexander Hoffmann, Lev Tsimring, and Roy Wollman. Accurate information transmission through dynamic biochemical signaling networks. Science , 346(6215):1370-1373, 2014.

Bernard W Silverman. Density estimation for statistics and data analysis . Routledge, 2018.

Victor Solo. On causality and mutual information. In 2008 47th IEEE Conference on Decision and Control , pages 4939-4944. IEEE, 2008.

Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estimators. In International Conference on Learning Representations , 2020.

Joe Suzuki. An estimator of mutual information and its application to independence testing. Entropy , 18(4):109, 2016.

Nassim Nicholas Taleb. Statistical consequences of fat tails: Real world preasymptotics, epistemology, and applications, 2020. URL https://arxiv.org/abs/2001.10488 .

Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. In International Conference on Learning Representations , 2020.

Shinsuke Uda. Application of information theory in systems biology. Biophysical Reviews , 12 (2):377-384, Apr 2020. ISSN 1867-2469. doi: 10.1007/s12551-020-00665-w. URL https: //doi.org/10.1007/s12551-020-00665-w .

Gregory Valiant and Paul Valiant. Estimating the unseen: an n/ log( n ) -sample estimator for entropy and support size, shown optimal via new clts. In Proceedings of the annual ACM symposium on Theory of computing (STOC) , pages 685-694, 2011.

Martin Vejmelka and Kateˇ rina Hlaváˇ cková-Schindler. Mutual information estimation in higher dimensions: A speed-up of a k-nearest neighbor based estimator. In International Conference on Adaptive and Natural Computing Algorithms , pages 790-797. Springer, 2007.

Nguyen Xuan Vinh, Jeffrey Chan, and James Bailey. Reconsidering mutual information based feature selection: A statistical significance view. In Proceedings of the AAAI Conference on Artificial Intelligence , 2014.

Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems , 34:16451-16467, 2021.

Quanhan Xi and Benjamin Bloem-Reddy. Indeterminacy in latent variable models: Characterization and strong identifiability. arXiv preprint arXiv:2206.00801 , 2022.

Alexander L. Young, Willem van den Boom, Rebecca A. Schroeder, Vijay Krishnamoorthy, Karthik Raghunathan, Hau-Tieng Wu, and David B. Dunson. Mutual information: Measuring nonlinear dependence in longitudinal epidemiological data. PLOS ONE , 18(4):1-16, 04 2023. doi: 10.1371/ journal.pone.0284904. URL https://doi.org/10.1371/journal.pone.0284904 .

Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey, 2021. URL https://arxiv.org/abs/2110.04596 .

Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In International Conference on Machine Learning , pages 12979-12990. PMLR, 2021.

Table of Contents

G Author Contributions

Appendix

30

A Invariance to Continuous Injective Mappings

In this section, we provide a proof for Theorem 2.1, formally proving the invariance of mutual information with respect to continuous injective mappings. First, we recall the standard definitions from Chapters 1 and 2 of Pinsker and Feinstein [1964].

Definition A.1. Let (Ω , A ) be a measurable space. A family of sets E ⊆ A is a partition of Ω if for every two distinct E 1 and E 2 in E it holds that E 1 ∩ E 2 = ∅ and that ⋃ E = Ω . If E is finite, we call it a finite partition.

Definition A.2 (Mutual Information) . Mutual information between r.v. X and Y is defined as

where the supremum is taken over all finite partitions E = { E 1 , E 2 , . . . } of X and F = { F 1 , F 2 , . . . } of Y with the usual conventions 0 log 0 = 0 and 0 log 0 / 0 = 0 .

Remark A.3 . If P XY is absolutely continuous with respect to P X ⊗ P Y (e.g., X and Y are finite r.v.), mutual information is equal to the following Kullback-Leibler divergence:

where f is the Radon-Nikodym derivative f = d P XY / d( P X ⊗ P Y ) . If the absolute continuity does not hold, then I ( X ; Y ) = + ∞ [Pinsker and Feinstein, 1964, Theorem 2.1.2].

Theorem 2.1. Let X , X ′ , Y and Y ′ be standard Borel spaces (e.g., smooth manifolds with their Borel σ -algebras) and f : X → X ′ and g : Y → Y ′ be continuous injective mappings. Then, for every X -valued r.v. X and Y -valued r.v. Y it holds that I ( X ; Y ) = I ( f ( X ); g ( Y )) .

Proof. First, note that a continuous mapping between Borel spaces is measurable, so random variables f ( X ) and g ( Y ) are well-defined.

Then, observe that it suffices to prove that

It is well-known that we have I ( f ( X ); Y ) ≤ I ( X ; Y ) for every measurable mapping f , so we will need to prove the reverse inequality.

Recall form Definition A.2 that

where the supremum is taken over all finite partitions E of X and F of Y .

The proof will go as follows: for every finite partition E of X we will construct a finite partition of X ′ , which (together with the same arbitrary partition F of Y ) will give the same value of the sum. This will prove that I ( f ( X ); Y ) ≥ I ( X ; Y ) .

Take any finite partitions { E 1 , . . . , E n } of X and { F 1 , . . . , F m } of Y . First, we will show that sets

.

.

.

form a partition of X ′ .

It is easy to see that they are pairwise disjoint as f is injective and it is obvious that their union is the whole X ′ . Hence, we only need to show that they are all Borel.

For i ≥ 1 the Lusin-Suslin theorem [Kechris, 2012, Theorem 15.1] guarantees that G i is Borel (continuous injective mappings between Polish spaces map Borel sets to Borel sets). Similarly, f ( X ) is Borel and hence G 0 = X ′ \ f ( X ) is Borel as well.

Next, we will prove that

Note that P f ( X ) ( G 0 ) = P X ( f -1 ( G 0 )) = P X ( ∅ ) = 0 , so that we can ignore the terms with i = 0 . For i ≥ 1 we have

as f is injective.

Similarly, f × 1 Y is injective and maps E i × F j onto G i × F j , so

This finishes the proof.

B Spiral Diffeomorphisms

In this section we will formally define the spiral diffeomorphisms and prove that they preserve the centred Gaussian distribution N (0 , σ 2 I n ) . They were inspired by the 'swirling construction', discussed by Hauberg [2019].

Definition B.1. Let O ( n ) be the orthogonal group and γ : R → O ( n ) be any smooth function. The mapping

given by f ( x ) = γ ( || x || 2 ) x will be called the spiral.

Example B.2. Let so ( n ) be the vector space of all skew-symmetric matrices. For every two matrices R ∈ O ( n ) and A ∈ so ( n ) and any smooth mapping g : R → R we can form a path

In particular, R = I n and g ( t ) = vt yields the spirals defined earlier.

From the above definition it is easy to prove that spirals are diffeomorphisms. We formalize it by the following lemma.

Lemma B.3. Every spiral is a diffeomorphism.

Proof. Let γ be the path leading to the spiral f . As matrix inverse is a smooth function, we can define a smooth function h ( x ) = γ ( || x || 2 ) -1 x .

Note that || f ( x ) || 2 = || x || 2 , as orthogonal transformation γ ( || x || 2 )

Then preserves distances.

Analogously, we prove that f ◦ h also is the identity mapping.

Finally, we will give the proof that spirals leave the centred normal distribution N (0 , σ 2 I n ) invariant. This will follow from a slightly stronger theorem:

Theorem B.4. Let P be a probability measure on R n with a smooth probability density function p which factorizes as:

for some smooth function u : R → R .

Then, every spiral f : R n → R n preserves the measure P :

where f # P = P ◦ f -1 is the pushforward measure.

In particular, for P = N (0 , σ 2 I n ) (a centered isotropic Gaussian distribution), if X ∼ N (0 , σ 2 I n ) , then f ( X ) ∼ N (0 , σ 2 I n ) .

Proof. As we have already observed, for every x ∈ R n it holds that || f ( x ) || 2 = || x || 2 , so the PDF evaluated at both points is equal

Hence, we need to prove that for every point x ∈ R n , | det f ′ ( x ) | = 1 , where f ′ ( x ) is the Jacobian matrix of f evaluated at the point x .

The proof of this identity was given by Carapetis [2022], but for the sake of completeness we supply a variant of it below.

First, we calculate the Jacobian matrix:

Hence,

Now fix point x and consider the matrix γ ( || x || 2 ) ∈ O ( n ) . Define a mapping y ↦→ γ ( || x || 2 ) -1 f ( y ) . As we have already observed, multiplication by γ ( || x || 2 ) -1 does not change the value of the PDF. It also cannot change the | det f ′ ( x ) | value as because γ ( || x || 2 ) -1 is a linear mapping, its derivative is just the mapping itself and because γ ( || x || 2 ) -1 is an orthogonal matrix, we have | det γ ( || x || 2 ) -1 | = 1 . Hence, without loss of generality we can assume that γ ( || x || 2 ) = I n and

where A := γ ′ ( || x || 2 ) ∈ so ( n ) is a skew-symmetric matrix.

If x = 0 we have | det f ′ ( x ) | = | det I n | = 1 . In the other case, we will work in coordinates induced by an orthonormal basis e 1 , . . . , e n in which the first basis vector is e 1 = x/ || x || . Hence,

For i ≥ 2 we have e T 1 e i = 0 and consequently

This shows that in this basis we have

where the block on the bottom right is the identity matrix I n -1 and det f ′ ( x ) = a 1 .

The first column is formed by the coordinates of

and we have as A ∈ so ( n ) is skew-symmetric.

C Overview of Estimators

There exists a broad literature on (conditional) mutual information estimators ranging from estimators on discrete data [Paninski, 2003, Valiant and Valiant, 2011, Vinh et al., 2014, Marx and Vreeken, 2019], over estimators for continuous data [Kraskov et al., 2004, Darbellay and Vajda, 1999, Belghazi et al., 2018] to estimators that can consider discrete-continuous mixture data [Gao et al., 2017a, Marx et al., 2021, Mesner and Shalizi, 2020]. Here, we focus on the most commonly used class, which also finds applications in representation learning: estimators for continuous data. For continuous data, the most widely propagated classes of estimators are histogram or partition-based estimators, nearest neighbor-based density estimators, kernel methods, and neural estimators. In the following, we briefly discuss a representative selection of estimators from these classes, as well as model-based estimation.

Histogram-Based Estimators A natural way to approximate mutual information, as stated in Definition A.2, is by partitioning the domain space of X and Y into segments and estimate the densities locally. This approach is also known as histogram density estimation, where the probability mass function is approximated locally for each hyper-rectangle E i by ˆ P X ( E i ) = n i / N Vol i , where Vol i is the volume of E i , and n i refers to the number of data points residing in E i .

To estimate mutual information, we explore the space of allowed partitionings and select the partitioning that maximizes mutual information [Darbellay and Vajda, 1999, Cellucci et al., 2005]

Many estimators follow this idea. Darbellay and Vajda [1999] use adaptive partitioning trees and propose to split bins until the density within a bin is uniform, which they check with the χ 2 test. Daub et al. [2004] use B-Splines to assign data points to bins, and other authors use regularization based on the degrees of freedom of partitioning to avoid overfitting [Suzuki, 2016, Cabeli et al., 2020, Vinh et al., 2014, Marx et al., 2021].

A well-known property of histogram-based estimators is that they are consistent for N →∞ if the volume of the largest hypercube goes to zero, while the number of hypercubes grows sub-linearly with the number of samples, N (see e.g. Cover and Thomas [2006]). For low-dimensional settings, histogram density estimation can provide accurate results, but the strategy suffers from the curse of dimensionality [Pizer et al., 1987, Klemelä, 2009].

In our evaluation, we resort to partitioning estimators based on a fixed number of grids ( 10 equal-width bins per dimension), since more recent partitioning-based estimators focus on estimating conditional MI and only allow X and Y to be one-dimensional [Suzuki, 2016, Marx et al., 2021, Cabeli et al., 2020], or do not have a publicly available implementation [Darbellay and Vajda, 1999].

k NN Estimators An alternative to histogram-based methods is to use k -nearest neighbor ( k NN) distances to approximate densities locally. Most generally, k NN estimators express mutual information as the sum H ( X ) + H ( Y ) -H ( X,Y ) , which can be done under the assumptions discussed in Remark A.3, and estimate the individual entropy terms as

where k ( x i ) is the number of data points other than x i in a neighborhood of x i , and Vol i is the volume of the neighborhood [Kozachenko and Leonenko, 1987]. The quality of the estimator depends on two

factors: i) the distance metric used to estimate the nearest neighbor distances, and ii) the estimation of the volume terms.

The most well-known estimator is the Kraskov, Stögbauer and Grassberger (KSG) estimator [Kraskov et al., 2004]. In contrast to the first k NN-based estimators [Kozachenko and Leonenko, 1987], Kraskov et al. [2004] propose to estimate the distance to the k -th neighbor ρ i,k / 2 for each data point i only on the joint space of Z = X × Y , with

where D X and D Y can refer to any norm-induced metric (e.g., Euclidean ℓ 2 , Manhattan ℓ 1 or Chebyshev ℓ ∞ ). 5 To compute the entropy terms of the sub-spaces X and Y , we count the number of data points with distance smaller than ρ i,k / 2 , i.e. we define n x,i as

̸

and specify n y,i accordingly. As a consequence, all volume-related terms cancel and the KSG estimator reduces to

The diagamma function 6 ψ was already introduced by Kozachenko and Leonenko [1987] to correct for biases induced by the volume estimation. For a detailed exposition of the issue, we refer to Kraskov et al. [2004, Sec. II.A].

There exist various extensions of the classical KSG estimator. When selecting the L ∞ norm as a distance measure for D X and D Y , E XY KSG ,N can be efficiently computed with the K-d trie method [Vejmelka and Hlaváˇ cková-Schindler, 2007]. Other approaches aim to improve the estimation quality by explicitly computing the volume terms via SVD [Lord et al., 2018] or PCA [Lu and Peltonen, 2020], or use manifold learning [Marx and Fischer, 2022] or normalizing flows [Ao and Li, 2022] to estimate the nearest neighbors more efficiently in high-dimensional settings.

Kernel Estimators A smaller class of estimators builds upon kernel density estimation (KDE) [Moon et al., 1995, Paninski and Yajima, 2008]. Classical approaches compute the density functions for the individual entropy terms via kernel density estimation, which is well-justified under the assumption of P XY being absolutely continuous with respect to P X ⊗ P Y (cf. Remark A.3). i.e., the density for a d -dimensional random vector X is approximated as

where h is the smoothing parameter or bandwidth of a Gaussian kernel [Silverman, 2018].

Moon et al. [1995] propose to estimate mutual information using a Gaussian kernel, for which the bandwidth can be estimated optimally [Silverman, 2018]. Similar to histogram-based approaches, KDE-based estimators with fixed bandwidth h are consistent if Nh d →∞ as h → 0 .

A more recent proposal, LNN [Gao et al., 2017b], uses a data-dependent bandwidth based on nearest-neighbor distances, to avoid fixing h globally.

Neural Estimators Neural network approaches such as MINE [Belghazi et al., 2018] use a variational lower bound on mutual information using the Donsker-Varadhan theorem: let f : X × Y → R be a bounded measurable function and

Under sufficient regularity conditions on P XY , I ( X ; Y ) is equal to V f ( X,Y ) , which is evaluated over all bounded measurable mappings f . In practice, the right hand side is optimized with respect

to the parameters of a given parametric predictive model f , as a neural network. Approaches following this idea also count as discriminative approaches since the neural network is trained to distinguish points from the joint and marginal distribution. The literature on variational lower bounds of mutual information is growing [Oord et al., 2018, Song and Ermon, 2020] and the idea of using a discriminative approach to estimate mutual information has also been explored by Koeman and Heskes [2014], who used random forests. For an excellent theoretical overview of different neural estimators we refer to Poole et al. [2019].

Model-based Estimators Finally, if one assumes that P XY is a member of a family of distributions Q XY ( θ ) with known mutual information, one can estimate θ basing on the data (( x 1 , y 1 ) , . . . , ( x N , y N )) and calculate the corresponding mutual information. For example, if one assumes that there exist jointly multivariate normal variables ˜ X = ( ˜ X 1 , . . . , ˜ X r , E 1 , . . . , E m -r ) and ˜ Y = ( ˜ Y 1 , . . . , ˜ Y r , ˜ E 1 , . . . , ˜ E n -r ) such that the cross-covariance terms are zero apart from Cov( ˜ X i , ˜ Y i ) , and that X = A ˜ X + a and Y = B ˜ Y + b for invertible matrices A and B and vectors a ∈ R m and b ∈ R n , then the mutual information can be found as

The model parameters A , B , a , and b , as well as the pairs of corresponding r.v. ˜ X r and ˜ Y r , can be found by maximising likelihood using canonical correlation analysis (CCA) [Murphy, 2023, Ch. 28], which has been previously explored by Kay [1992].

D Additional Information on Benchmark Tasks

D.1 Task Descriptions

Bimodal Variables Consider a r.v. X distributed according to a Gaussian mixture model with CDF F = w 1 F 1 + w 2 F 2 , where F 1 and F 2 are CDFs of two Gaussian distributions and w 1 and w 2 are positive weights constrained as w 1 + w 2 = 1 . The CDF is everywhere positive, continuous, and strictly increasing. Hence, it has an inverse, the quantile function, F -1 : (0 , 1) → R , which is continuous and injective as well (see Lemma D.1). Thus, we can define r.v. X ′ = F -1 ( X ) , where X ∼ Uniform(0 , 1) and I ( X ; Y ) = I ( X ′ ; Y ) for every r.v. Y . We generated two CDFs of Gaussian mixtures and implemented a numerical inversion in JAX [Bradbury et al., 2018] defining both X ′ and Y ′ to follow a bimodal distribution.

For the experimental values, we used

where F is the CDF of the standard normal distribution N (0 , 1) .

Wiggly Mapping As the 'wiggly mapping' we used

and we used visualisations to make sure that they do not heavily distort the distribution.

Additive Noise For the additive noise model of Y = X + N where X ∼ Uniform(0 , 1) and N ∼ Uniform( -ε, ε ) it holds that

implying that I ( X ; Y ) = 1 . 7 for ε = 0 . 1 and I ( X ; Y ) = 0 . 3 for ε = 0 . 75 .

Swiss Roll Embedding Consider a bivariate distribution P XY with uniform margins X ∼ Uniform (0 , 1) and Y ∼ Uniform (0 , 1) and I ( X ; Y ) = 0 . 8 . The Swiss roll mapping is an embedding i : (0 , 1) 2 → R 3 given by i ( x, y ) = ( e ( x ) , y ) , where e : (0 , 1) → R 2 is given by

Note that P X ′ Y ′ does not have a PDF with respect to the Lebesgue measure on R 3 . We visualise the Swiss roll distribution in Fig. 1.

Spiral Diffeomorphism For the precise definition and properties of the general spiral diffeomorphism consult Appendix B.

In our benchmark we applied the spiral x ↦→ exp( vA || x || 2 ) x to both X and Y variables. We used an m × m skew-symmetric matrix A X with ( A X ) 12 = -( A X ) 21 = 1 and an n × n skew-symmetric matrix A Y with ( A Y ) 23 = -( A Y ) 32 = 1 . Each of these effectively 'mixes' only two dimensions.

The speed parameters v X and v Y were chosen as v X = 1 /m and v Y = 1 /n to partially compensate for the fact that if X ∼ N (0 , I m ) , then || X || 2 ∼ χ 2 m and has mean m .

Multivariate Student Let Ω be an ( m + n ) × ( m + n ) positive definite dispersion matrix and ν be a positive integer (the degrees of freedom). By sampling an ( m + n ) -dimensional random vector ( ˜ X, ˜ Y ) ∼ N (0 , Ω) and a random scalar U ∼ χ 2 ν one can define rescaled variables X = ˜ X √ ν/U and Y = ˜ Y √ ν/U , which are distributed according to the multivariate Student distribution. For ν = 1 this reduces to the multivariate Cauchy distribution (and the first two moments are not defined), for ν = 2 the mean exists, but covariance does not, and for ν > 2 both the mean and covariance exist, with Cov( X,Y ) = ν ν -2 Ω , so that the tail behaviour can be controlled by changing ν . In particular, for ν ≫ 1 because of the concentration of measure phenomenon U has most of its probability mass around ν and the variables ( X,Y ) can be well approximated by ( ˜ X, ˜ Y ) .

Arellano-Valle et al. [2013] proved that I ( X ; Y ) can be computed via the sum of the mutual information of the Gaussian distributed basis variables and a correction term, i.e., I ( X ; Y ) = I ( ˜ X ; ˜ Y ) + c ( ν, m, n ) , where

and ψ is the digamma function.

Contrary to the Gaussian case, even for Ω = I m + n , the mutual information I ( X ; Y ) = c ( ν, m, n ) is non-zero, as U provides some information about the magnitude. In the benchmark we use this dispersion matrix to quantify how well the estimators can use the information contained in the tails, rather than focusing on estimating the Gaussian term.

D.2 Covariance Matrix Family

We generated jointly multivariate normal variables X = ( X 1 , . . . , X m ) and Y = ( Y 1 , . . . , Y n ) with the following procedure.

Consider i.i.d. Gaussian variables

Now let ε X , ε Y , η X , η Y , α, β X , β Y , λ ∈ R be hyperparameters and for l = 1 , 2 , . . . , K define

for l = K +1 , . . . , m define

and for l = K +1 , . . . , n define

The interpretation of the hyperparameters is the following:

α contributes to the correlations between all the possible pairs of variables.

β X contributes to the correlations between the X variables.

β Y contributes to the correlations between the Y variables.

λ gives additional 'interaction strength" between pairs of variables X l and Y l for l = 1 , . . . , K .

ε X and ε Y control part of the variance non-shared with any other variable.

η X and η Y can be used to increase the variance of X l and Y l for l > K to match the variance of variables l ≤ K due to the λ .

Every term of the covariance matrix is easy to calculate analytically:

In the following, we analyse some special cases.

Dense Interactions Consider λ = η X = η Y = β X = β Y = 0 and write ε := ε X = ε Y . We have

Hence, between every pair of distinct variables we have the same correlation α 2 α 2 + ε 2 .

Sparse Interactions Consider α = 0 , ε := ε X = ε Y , η X = η Y = λ and β X = β Y = 0

All the variables have the same variance, but the covariance structure is now different. Between (distinct) X i and X j the correlation is zero and similarly for (distinct) Y i and Y j .

Correlations between X i and Y j are zero unless i = j ≤ K , when Cor( X i , Y i ) = λ 2 / ( ε 2 + λ 2 ) .

Interpolation In Section 4 we discussed a two-stage interpolation from dense matrices to sparse matrices: in the first stage α is decreased increasing λ at the same time to keep the mutual information constant. In the second stage we decrease the number of interacting variables K increasing λ . The selected covariance matrices from this two-stage process are visualised in Fig. 7.

D.3 Technical Results

In this subsection we add the technical results, related to the copula theory [Nelsen, 2007] and uniformization of random variables.

Lemma D.1. Let X be a real-valued r.v. with a strictly increasing, positive everywhere, and continuous CDF F . Then

is a homeomorphism.

where

X

Y

X

Y

=1.2

X

Y

X

Y

=0.9

X

Y

X

Y

=0.5

X

Y

X

Y

K

=10

X

Y

X

Y

K

=5

X

Y

X

Y

K

=1

Proof. As F is a CDF we have F ( R ) ⊆ [0 , 1] . Note that 0 is excluded from the image as F is positive everywhere and 1 is excluded as F is strictly increasing. Hence, F ( R ) ⊆ (0 , 1) . Using continuity, strict monotonicity, and

we note that F is a bijection onto (0 , 1) . Analogously, we prove that

for all a < b , so F is an open map. As a continuous bijective open map F is a homeomorphism.

Lemma D.2. Let X be a real-valued r.v. with a CDF F which is continuous, strictly increasing and positive everywhere. Then, F ( X ) is uniformly distributed on (0 , 1) .

Proof. Our assumptions on F imply that it is a continuous homeomorphism R → (0 , 1) .

Let now y ∈ (0 , 1) . We have

E Additional Experiments

E.1 Pointwise Mutual Information

To understand why neural estimators are not able to estimate the mutual information contained in the Student distribution, we decided to qualitatively study the learned critic.

In the NWJ approach [Nguyen et al., 2007], for N → ∞ data points the optimal critic f ( x, y ) is given by

is the pointwise mutual information [Pinsker and Feinstein, 1964, Ch. 2] and p XY , p X and p Y are the PDFs of measures P XY , P X and P Y , respectively.

3

0

3

3

0

3

PMI

Student  =1

3

0

3

3

0

3

Student  =2

3

0

3

3

0

3

Student  =3

3

0

3

3

0

3

Student  =5

3

0

3

3

0

3

Student  =8

3

0

3

3

0

3

Normal

3

0

3

3

0

3

Critic

3

0

3

3

0

3

3

0

3

3

0

3

3

0

3

3

0

3

3

0

3

3

0

3

3

0

3

3

0

3

Bivariate normal 1 × 1

Normal CDF @ Bivariate normal 1 × 1

Uniform 1 × 1 (additive noise=0.1)

Uniform 1 × 1 (additive noise=0.75)

Bimodal 1 × 1

Wiggly @ Bivariate normal 1 × 1

Half-cube @ Bivariate normal 1 × 1

Student-t 1 × 1 (dof=1)

Asinh @ Student-t 1 × 1 (dof=1)

Swiss roll 2 × 1

Multinormal 2 × 2 (dense)

Multinormal 3 × 3 (dense)

Multinormal 5 × 5 (dense)

Multinormal 25 × 25 (dense)

Multinormal 50 × 50 (dense)

Multinormal 2 × 2 (2-pair)

Multinormal 3 × 3 (2-pair)

Multinormal 5 × 5 (2-pair)

Multinormal 25 × 25 (2-pair)

Student-t 2 × 2 (dof=1)

Student-t 2 × 2 (dof=2)

Student-t 3 × 3 (dof=2)

Student-t 3 × 3 (dof=3)

Student-t 5 × 5 (dof=2)

Student-t 5 × 5 (dof=3)

Normal CDF @ Multinormal 3 × 3 (2-pair)

Normal CDF @ Multinormal 5 × 5 (2-pair)

Normal CDF @ Multinormal 25 × 25 (2-pair)

Half-cube @ Multinormal 3 × 3 (2-pair)

Half-cube @ Multinormal 5 × 5 (2-pair)

Half-cube @ Multinormal 25 × 25 (2-pair)

Spiral @ Multinormal 3 × 3 (2-pair)

Spiral @ Multinormal 5 × 5 (2-pair)

Spiral @ Multinormal 25 × 25 (2-pair)

Spiral @ Normal CDF @ Multinormal 3 × 3 (2-pair)

Spiral @ Normal CDF @ Multinormal 5 × 5 (2-pair)

Spiral @ Normal CDF @ Multinormal 25 × 25 (2-pair)

Asinh @ Student-t 2 × 2 (dof=1)

Asinh @ Student-t 3 × 3 (dof=2)

Asinh @ Student-t 5 × 5 (dof=2)

MINE

NWJ

D-V

KSG I (R)

LNN (R)

Hist. (Julia)

Transfer (Julia)

CCA

1k

1k

1k

1k

1k

1k

1k

>10k

1k

1k

1k

1k

3k

3k

3k

1k

1k

1k

3k

>10k>10k>10k>10k>10k>10k

1k

1k

5k

1k

1k

5k

>10k>10k 10k

1k

3k

5k

>10k>10k>10k

1k

1k

1k

1k

1k

1k

1k

>10k

1k

1k

3k

3k

3k

5k

5k

1k

3k

3k

5k

>10k

5k

10k

10k

10k

10k

3k

3k

10k

3k

3k

10k >10k>10k 10k

3k

3k

10k

3k

10k

10k

1k

1k

1k

1k

1k

1k

1k

>10k>10k

3k

3k

3k

3k

3k

5k

1k

3k

3k

5k

>10k>10k>10k>10k>10k>10k

3k

3k

10k

3k

3k

10k >10k>10k 10k

3k

3k

10k

10k >10k>10k

1k

1k

1k

1k

1k

1k

1k

>10k

1k

1k

3k

3k

3k

5k

5k

1k

3k

3k

5k

>10k>10k>10k>10k>10k>10k

3k

3k

10k

3k

3k

10k >10k>10k>10k

3k

3k

10k

3k

10k >10k

100

100

100

100

100

100

100

5k

500

100

500

100

500

1k

5k

100

500

5k

>10k>10k>10k>10k>10k>10k 100

500

10k >10k

1k

>10k>10k

5k

>10k>10k 500

10k >10k

3k

>10k 100

100

100

100 >10k 100

100

100

500

100 >10k>10k>10k>10k>10k>10k 100 >10k>10k>10k>10k>10k>10k>10k>10k>10k>10k>10k>10k>10k>10k>10k>10k>10k>10k>10k>10k>10k>10k>10k>10k

500

500

100

500

500

500

100 >10k 500

1k

10k >10k>10k>10k>10k

1k

>10k>10k>10k>10k 500

1k

5k

1k

10k >10k>10k>10k

3k

>10k>10k>10k>10k>10k>10k>10k>10k

3k

>10k>10k

1k

3k

100

3k

3k

1k

1k

100

3k

10k >10k>10k>10k>10k>10k>10k>10k>10k>10k>10k

3k

5k

>10k 10k >10k>10k>10k>10k>10k>10k>10k>10k>10k>10k>10k>10k>10k>10k>10k>10k

100

100

100 >10k 100

100

100 >10k>10k>10k 100

100

100

500

3k

100

100

100

1k

500

100

100 >10k 500

100

100

100

1k

100

100

1k

>10k>10k 500

100

100

1k

>10k>10k>10k

We trained NWJ estimator with the default hyperparameters on N = 5000 data points and visualised the critic predictions (with 1 subtracted), true PMI, and 500 samples from the distribution in Fig. 8. The data is drawn from a bivariate Student distribution with dispersion matrix

and varying degrees of freedom ν ∈ { 1 , 2 , 3 , 5 , 8 } . Additionally, we repeated the experiment for the bivariate normal distribution with covariance matrix given by the dispersion, which corresponds to the limit ν →∞ .

Qualitatively we see that the critic does not learn the correct PMI for ν ∈ { 1 , 2 } , with better similarities for ν ∈ { 3 , 5 , 8 } . For the normal distribution, the critic qualitatively approximates well the PMI function in the regions where the data points are available.

We hypothesise that these problems can be partly atributed to the longer tails of Student distributions (as the critic may be unable to extrapolate to the regions where few points are available) and partly to the analytical form of the PMI function - for the considered bivariate normal distribution the PMI is given by a simple quadratic function, which is not the case for the bivariate Student distribution.

E.2 Sample Size Requirements

We investigated the minimal number of samples needed to estimate the mutual information changing the number of samples in N ∈ { 100 , 500 , 1000 , 3000 , 5000 , 10 000 } . We say that sample ℓ suffices

to solve the task if for every ℓ ′ ≥ ℓ the estimator provides the estimate between 2/3 and 3/2 of the ground-truth mutual information.

Wepresent the minimal number of samples ℓ as a function of benchmark task and considered estimator in Fig. 9. Note that CCA and KSG typically need very small amount of samples to solve the task. As neural estimators use batch size of 256 and split the sample into training and test data sets, it is not possible for them to have ℓ < 1000 , which is the smallest number of samples considered greater than 2 · 256 = 512 .

E.3 Preprocessing

In Fig. 2 and Fig. 9 we transformed all samples by removing the mean and scaling by the standard deviation in each dimension calculated empirically from the sample. This strategy ensures that the scales on different axes are similar. Note that this strategy is not always suitable (e.g., if the distance metric used in the k NN approach should weight select dimensions differently) and makes the observed data points not i.i.d. (as the 'shared' information, sample mean and variances, has been used to rescale the variables), although the effect of it seems to be negligible on most tasks in the benchmark.

Additionally, we considered two other preprocessing strategies, implemented using the scikit-learn package [Pedregosa et al., 2011]: making each marginal distribution uniform Uniform(0 , 1) by calculating empirical CDF along each axis (see Section D for the technical lemmata discussing this operation) and making each marginal distribution a Gaussian variable N (0 , 1) . These approaches may affect the effective sample size more strongly than the previous rescaling strategy, as estimating empirical CDF requires more data points than estimating mean and standard deviation.

We used N = 10000 data points and rerun the benchmark (see Fig. 10). Overall, the impact of the 'uniformization' strategy appeared to be insubstantial for most tasks, and in certain cases negative. The 'Gaussianisation' strategy improved estimation for Student distributions and helped histogram based approaches in low-dimensional tasks. Interestingly, both approaches worsen the performance of CCA on Student distributions.

While the improvements gained from 'Gaussianisation' were minor, the approach is simplistic, treating each axis separately. Ideally, appropriate preprocessing would be able to transform P XY into a better-behaved distribution P f ( X ) g ( Y ) , and thus simplify the estimation. In certain cases such preprocessing could (partially) undo transformations considered in our benchmark, and in effect make the estimators invariant (or more robust) to these transformations (cf. Section 4). We consider the design of informed preprocessing strategies an important open problem.

E.4 Hyperparameter Selection

CCA We used the implementation provided in the scikit-learn [Pedregosa et al., 2011] Python package (version 1.2.2 ). We set the latent space dimension to the smaller of the dimensions of the considered random vectors X and Y .

Histograms Adaptive estimators such as JIC [Suzuki, 2016] or MIIC [Cabeli et al., 2020] were not applicable to the high-dimensional data we considered, hence we resort to using an equal-width estimator implemented in TransferEntropy library (version 1.10.0 ). We fixed the number of bins to 10 , although the optimal number of bins is likely to depend on the dimensionality of the problem.

KSG As a default, we used the KSG-1 variant from the rmi package with 10 neighbors. However, as shown in our ablation in Fig. 11, changing the number of neighbors or using the KSG-2 variant did not considerably change the estimates. We compared the used implementation with a custom Python implementation and one from TransferEntropy library. The version implemented in the TransferEntropy suffers from strong positive bias.

LNN We used the implementation from the rmi package (version 0.1.1 ) with 5 neighbors and truncation parameter set to 30 . Ablations can be seen in Fig. 11.

Neural Estimators We split the data set into two equal-sized parts (training and test) and optimized the statistics network on the training data set using the Adam optimiser with initial learning rate set to 0 . 1 . We used batch size of 256 and ran each algorithm for up to 10 000 training steps with early stopping (checked every 250 training steps). We returned the highest estimate on the test data set.

Uniformise @ Bivariate normal 1 × 1

Uniformise @ Normal CDF @ Bivariate normal 1 × 1

Uniformise @ Uniform 1 × 1 (additive noise=0.1)

Uniformise @ Uniform 1 × 1 (additive noise=0.75)

Uniformise @ Bimodal 1 × 1

Uniformise @ Wiggly @ Bivariate normal 1 × 1

Uniformise @ Half-cube @ Bivariate normal 1 × 1

Uniformise @ Student-t 1 × 1 (dof=1)

Uniformise @ Asinh @ Student-t 1 × 1 (dof=1)

Uniformise @ Swiss roll 2 × 1

Uniformise @ Multinormal 2 × 2 (dense)

Uniformise @ Multinormal 3 × 3 (dense)

Uniformise @ Multinormal 5 × 5 (dense)

Uniformise @ Multinormal 25 × 25 (dense)

Uniformise @ Multinormal 50 × 50 (dense)

Uniformise @ Multinormal 2 × 2 (2-pair)

Uniformise @ Multinormal 3 × 3 (2-pair)

Uniformise @ Multinormal 5 × 5 (2-pair)

Uniformise @ Multinormal 25 × 25 (2-pair)

Uniformise @ Student-t 2 × 2 (dof=1)

Uniformise @ Student-t 2 × 2 (dof=2)

Uniformise @ Student-t 3 × 3 (dof=2)

Uniformise @ Student-t 3 × 3 (dof=3)

Uniformise @ Student-t 5 × 5 (dof=2)

Uniformise @ Student-t 5 × 5 (dof=3)

Uniformise @ Normal CDF @ Multinormal 3 × 3 (2-pair)

Uniformise @ Normal CDF @ Multinormal 5 × 5 (2-pair)

Uniformise @ Normal CDF @ Multinormal 25 × 25 (2-pair)

Uniformise @ Half-cube @ Multinormal 3 × 3 (2-pair)

Uniformise @ Half-cube @ Multinormal 5 × 5 (2-pair)

Uniformise @ Half-cube @ Multinormal 25 × 25 (2-pair)

Uniformise @ Spiral @ Multinormal 3 × 3 (2-pair)

Uniformise @ Spiral @ Multinormal 5 × 5 (2-pair)

Uniformise @ Spiral @ Multinormal 25 × 25 (2-pair)

Uniformise @ Spiral @ Normal CDF @ Multinormal 3 × 3 (2-pair)

Uniformise @ Spiral @ Normal CDF @ Multinormal 5 × 5 (2-pair)

Uniformise @ Spiral @ Normal CDF @ Multinormal 25 × 25 (2-pair)

Uniformise @ Asinh @ Student-t 2 × 2 (dof=1)

Uniformise @ Asinh @ Student-t 3 × 3 (dof=2)

Uniformise @ Asinh @ Student-t 5 × 5 (dof=2)

MINE

NWJ

D-V

KSG I (R)

LNN (R)

Hist. (Julia)

Transfer (Julia)

CCA

True MI

0.4

0.4

1.6

0.3

0.4

0.4

0.4

0.1

0.1

0.4

0.3

0.4

0.5

1.2

1.5

0.9

0.9

0.8

0.8

0.1

0.0

0.0

0.0

0.1

0.1

0.9

0.9

0.8

0.9

0.9

0.8

0.4

0.4

0.6

0.8

0.8

0.7

0.1

0.0

0.1

0.4

0.4

1.6

0.3

0.4

0.4

0.4

0.2

0.2

0.4

0.3

0.4

0.5

1.1

1.4

0.9

0.9

0.9

0.8

0.3

0.1

0.2

0.1

0.2

0.1

0.9

0.9

0.8

0.9

0.9

0.8

0.5

0.5

0.6

0.8

0.8

0.7

0.3

0.1

0.1

0.4

0.4

1.6

0.3

0.4

0.4

0.4

0.1

0.1

0.3

0.3

0.4

0.5

1.1

1.2

0.9

0.9

0.9

0.8

0.1

0.0

0.0

0.0

0.0

0.0

0.9

0.9

0.8

0.9

0.9

0.8

0.5

0.5

0.6

0.8

0.8

0.6

0.1

0.0

0.0

0.4

0.4

1.6

0.3

0.4

0.4

0.4

0.1

0.2

0.4

0.3

0.4

0.5

1.2

1.4

0.9

0.9

0.9

0.8

0.3

0.1

0.1

0.0

0.1

0.0

0.9

0.9

0.8

0.9

0.9

0.8

0.5

0.5

0.6

0.9

0.8

0.7

0.3

0.1

0.1

0.4

0.4

1.7

0.3

0.4

0.4

0.4

0.2

0.2

0.4

0.3

0.4

0.6

1.0

1.1

1.0

0.9

0.7

0.1

0.4

0.2

0.2

0.1

0.2

0.1

0.9

0.7

0.1

0.9

0.7

0.1

0.6

0.5

0.1

0.9

0.7

0.1

0.4

0.2

0.2

0.3

0.3 -0.3 0.2

0.3

0.3

0.3

0.1

0.1

0.6

1.0

1.3

1.6

0.6

0.4

0.6

0.6

1.6

1.6

1.3

1.5

0.7

0.7

0.4

0.4

1.4

0.2

0.4

0.4

0.4

0.2

0.2

0.4

0.7

4.0

8.6

9.2

9.2

1.2

4.5

9.1

9.2

0.8

0.7

4.3

4.4

8.9

9.0

4.5

9.1

9.2

4.5

9.1

9.2

4.5

9.1

9.2

4.2

9.0

9.2

0.8

4.3

8.9

0.4

0.4

1.6

0.3

0.4

0.4

0.4

0.2

0.2

0.6

4.1

8.5

9.3

9.0

9.0

4.4

8.9

9.0

9.0

4.0

4.2

8.7

8.8

9.2

9.1

8.9

9.0

9.0

8.9

9.0

9.0

8.9

9.0

9.0

8.7

9.0

9.0

4.0

8.7

9.2

0.4

0.4

1.7

0.2

0.4

0.4

0.4

0.0

0.0

0.0

0.3

0.4

0.6

1.3

1.7

1.0

1.0

1.0

1.0

0.0

0.0

0.0

0.0

0.0

0.0

1.0

1.0

1.0

1.0

1.0

1.0

0.3

0.4

0.8

0.8

0.9

1.0

0.0

0.0

0.0

0.4

0.4

1.7

0.3

0.4

0.4

0.4

0.2

0.2

0.4

0.3

0.4

0.6

1.3

1.6

1.0

1.0

1.0

1.0

0.4

0.2

0.3

0.2

0.4

0.3

1.0

1.0

1.0

1.0

1.0

1.0

1.0

1.0

1.0

1.0

1.0

1.0

0.4

0.3

0.4

Gaussianise @ Bivariate normal 1 × 1

Gaussianise @ Normal CDF @ Bivariate normal 1 × 1

Gaussianise @ Uniform 1 × 1 (additive noise=0.1)

Gaussianise @ Uniform 1 × 1 (additive noise=0.75)

Gaussianise @ Bimodal 1 × 1

Gaussianise @ Wiggly @ Bivariate normal 1 × 1

Gaussianise @ Half-cube @ Bivariate normal 1 × 1

Gaussianise @ Student-t 1 × 1 (dof=1)

Gaussianise @ Asinh @ Student-t 1 × 1 (dof=1)

Gaussianise @ Swiss roll 2 × 1

Gaussianise @ Multinormal 2 × 2 (dense)

Gaussianise @ Multinormal 3 × 3 (dense)

Gaussianise @ Multinormal 5 × 5 (dense)

Gaussianise @ Multinormal 25 × 25 (dense)

Gaussianise @ Multinormal 50 × 50 (dense)

Gaussianise @ Multinormal 2 × 2 (2-pair)

Gaussianise @ Multinormal 3 × 3 (2-pair)

Gaussianise @ Multinormal 5 × 5 (2-pair)

Gaussianise @ Multinormal 25 × 25 (2-pair)

Gaussianise @ Student-t 2 × 2 (dof=1)

Gaussianise @ Student-t 2 × 2 (dof=2)

Gaussianise @ Student-t 3 × 3 (dof=2)

Gaussianise @ Student-t 3 × 3 (dof=3)

Gaussianise @ Student-t 5 × 5 (dof=2)

Gaussianise @ Student-t 5 × 5 (dof=3)

Gaussianise @ Normal CDF @ Multinormal 3 × 3 (2-pair)

Gaussianise @ Normal CDF @ Multinormal 5 × 5 (2-pair)

Gaussianise @ Normal CDF @ Multinormal 25 × 25 (2-pair)

Gaussianise @ Half-cube @ Multinormal 3 × 3 (2-pair)

Gaussianise @ Half-cube @ Multinormal 5 × 5 (2-pair)

Gaussianise @ Half-cube @ Multinormal 25 × 25 (2-pair)

Gaussianise @ Spiral @ Multinormal 3 × 3 (2-pair)

Gaussianise @ Spiral @ Multinormal 5 × 5 (2-pair)

Gaussianise @ Spiral @ Multinormal 25 × 25 (2-pair)

Gaussianise @ Spiral @ Normal CDF @ Multinormal 3 × 3 (2-pair)

Gaussianise @ Spiral @ Normal CDF @ Multinormal 5 × 5 (2-pair)

Gaussianise @ Spiral @ Normal CDF @ Multinormal 25 × 25 (2-pair)

Gaussianise @ Asinh @ Student-t 2 × 2 (dof=1)

Gaussianise @ Asinh @ Student-t 3 × 3 (dof=2)

Gaussianise @ Asinh @ Student-t 5 × 5 (dof=2)

MINE

NWJ

D-V

KSG I (R)

LNN (R)

Hist. (Julia)

Transfer (Julia)

CCA

True MI

0.4

0.4

1.6

0.3

0.4

0.4

0.4

0.1

0.2

0.4

0.3

0.4

0.6

1.2

1.5

0.9

0.9

0.9

0.8

0.2

0.1

0.1

0.1

0.2

0.1

0.9

0.9

0.8

1.0

0.9

0.8

0.4

0.5

0.7

0.8

0.8

0.8

0.2

0.1

0.2

0.4

0.4

1.6

0.3

0.4

0.4

0.4

0.2

0.2

0.4

0.3

0.4

0.6

1.2

1.4

1.0

1.0

1.0

0.9

0.3

0.1

0.2

0.1

0.3

0.2

1.0

1.0

0.9

1.0

1.0

0.9

0.6

0.6

0.7

0.9

0.9

0.8

0.4

0.2

0.3

0.4

0.4

1.6

0.3

0.4

0.4

0.4

0.1

0.1

0.4

0.3

0.4

0.6

1.2

1.2

1.0

1.0

1.0

0.9

0.4

0.1

0.1

0.0

0.1

0.0

1.0

1.0

0.9

1.0

1.0

0.8

0.5

0.5

0.7

0.9

0.9

0.8

0.3

0.1

0.1

0.4

0.4

1.6

0.3

0.4

0.4

0.4

0.2

0.2

0.4

0.3

0.4

0.6

1.2

1.5

1.0

1.0

1.0

0.9

0.3

0.1

0.2

0.1

0.2

0.1

1.0

1.0

0.9

1.0

1.0

0.9

0.5

0.5

0.7

0.9

0.9

0.8

0.3

0.2

0.3

0.4

0.4

1.7

0.3

0.4

0.4

0.4

0.2

0.2

0.4

0.3

0.4

0.6

1.0

1.1

1.0

1.0

0.7

0.1

0.4

0.2

0.2

0.1

0.2

0.2

1.0

0.7

0.1

1.0

0.7

0.1

0.7

0.6

0.1

0.9

0.7

0.1

0.4

0.2

0.2

0.3

0.3

1.6

0.2

0.3

0.3

0.3

0.1

0.1

0.7

1.3

2.7

1.4

1.9

3.1

1.8

0.7

0.5

1.2

1.1

3.4

3.1

1.9

3.1

2.4

1.9

3.1

1.8

1.8

2.9

1.8

1.8

2.9

2.4

0.8

1.3

3.4

0.3

0.3

0.8

0.2

0.3

0.3

0.3

0.1

0.1

0.3

0.3

0.7

3.4

9.2

9.2

0.8

1.1

4.6

9.2

0.3

0.2

0.8

0.8

4.0

4.1

1.1

4.6

9.2

1.1

4.6

9.2

1.0

4.6

9.2

0.9

4.4

9.2

0.3

0.8

4.0

0.4

0.4

1.3

0.3

0.4

0.4

0.4

0.2

0.2

0.5

1.1

4.7

9.2

9.0

9.0

1.6

5.3

9.3

9.0

1.1

1.1

4.8

5.0

9.1

9.2

5.3

9.3

9.0

5.3

9.3

9.0

5.3

9.3

9.0

4.9

9.3

9.0

1.1

4.9

9.1

0.4

0.4

1.3

0.2

0.4

0.4

0.4

0.0

0.0

0.0

0.3

0.4

0.6

1.3

1.7

1.0

1.0

1.0

1.1

0.0

0.0

0.0

0.0

0.0

0.0

1.0

1.0

1.1

1.0

1.0

1.0

0.2

0.4

0.9

0.8

0.9

1.0

0.0

0.0

0.0

0.4

0.4

1.7

0.3

0.4

0.4

0.4

0.2

0.2

0.4

0.3

0.4

0.6

1.3

1.6

1.0

1.0

1.0

1.0

0.4

0.2

0.3

0.2

0.4

0.3

1.0

1.0

1.0

1.0

1.0

1.0

1.0

1.0

1.0

1.0

1.0

1.0

0.4

0.3

0.4

We chose the architecture using a mini-benchmark with four ReLU neural networks considered on two neural estimators (see Fig. 11) and chose a neural network with two hidden layers (network M in Table 2).

As neural estimators rely on stochastic gradient descent training, we implemented heuristic diagnostics used to diagnose potential problems with overfitting and non-convergence. We present the number of runs susceptible to these issues in Fig. 12. We did not include these runs when computing statistics in the main benchmark figures (Fig. 2 and Fig. 9).

Normal CDF @ Multinormal 25 × 25 (2-pair)

Multinormal 5 × 5 (2-pair)

Multinormal 25 × 25 (2-pair)

Normal CDF @ Multinormal 5 × 5 (2-pair)

Student-t 5 × 5 (dof=2)

Bimodal 1 × 1

Swiss roll 2 × 1

Student-t 3 × 3 (dof=2)

Asinh @ Student-t 1 × 1 (dof=1)

Multinormal 25 × 25 (dense)

Asinh @ Student-t 5 × 5 (dof=2)

Asinh @ Student-t 3 × 3 (dof=2)

KSG I (n=5, Python)

KSG I (n=10, Python)

KSG I (n=5, R)

KSG I (n=10, R)

KSG II (n=5, R)

KSG II (n=10, R)

KSG I (n=5, Julia)

KSG II (n=5, Julia)

LNN (n=5, t=30, R)

LNN (n=10, t=30, R)

LNN (n=5, t=10, R)

True MI

0.2

0.8

0.2

0.8

0.2

0.4

0.4

0.1

0.2

1.1

0.3

0.2

0.2

0.7

0.2

0.7

0.2

0.4

0.4

0.1

0.2

1.1

0.2

0.2

0.1

0.7

0.1

0.7

0.2

0.4

0.4

0.1

0.2

1.0

0.2

0.2

0.1

0.7

0.1

0.7

0.2

0.4

0.4

0.1

0.2

1.0

0.2

0.2

0.1

0.8

0.1

0.7

0.2

0.4

0.4

0.1

0.2

1.0

0.2

0.2

0.1

0.7

0.1

0.7

0.2

0.4

0.4

0.1

0.2

1.0

0.2

0.2

9.9

7.0

9.9

6.6

6.8

1.8

1.8

4.4

1.6

9.9

6.7

4.3

11.7 8.1 11.7 7.6

8.0

2.6

5.3

2.4 11.7 7.7

5.2

5.7

3.1

2.2

2.4

3.7

0.3

1.5

0.1

3.5

1.3

10.9 2.8

7.7

2.2

3.5

0.5

1.5

0.4

3.2

1.3

7.4

6.8

7.2

0.5

1.9

0.3

7.1

1.8

1.0

1.0

1.0

1.0

0.4

0.4

0.4

0.3

0.2

1.3

0.4

0.3

Swiss roll 2 × 1

Multinormal 25 × 25 (dense)

Asinh @ Student-t 5 × 5 (dof=2)

Bimodal 1 × 1

Normal CDF @ Multinormal 5 × 5 (2-pair)

Asinh @ Student-t 1 × 1 (dof=1)

Normal CDF @ Multinormal 25 × 25 (2-pair)

Student-t 3 × 3 (dof=2)

Asinh @ Student-t 3 × 3 (dof=2)

Multinormal 25 × 25 (2-pair)

Multinormal 5 × 5 (2-pair)

Student-t 5 × 5 (dof=2)

MINE D

MINE L

MINE M

MINE S

InfoNCE D

InfoNCE L

InfoNCE M

InfoNCE S

True MI

0.2

1.2

0.0

0.4

0.8

0.1

0.8

0.1

0.0

0.8

0.9

0.1

0.4

1.2

0.2

0.4

0.9

0.1

0.8

0.2

0.1

0.8

0.9

0.2

0.4

1.2

0.2

0.3

0.9

0.2

0.8

0.2

0.1

0.8

0.9

0.3

0.3

1.2

0.0

0.4

0.8

0.1

0.8

0.1

0.0

0.9

0.9

0.1

0.4

1.3

0.1

0.4

0.9

0.2

0.8

0.2

0.2

0.9

1.0

0.2

0.4

1.2

0.3

0.4

0.9

0.2

0.7

0.2

0.2

0.8

1.0

0.3

0.4

1.2

0.3

0.4

0.9

0.2

0.8

0.2

0.2

0.9

1.0

0.3

0.4

1.2

0.2

0.4

0.9

0.2

0.8

0.2

0.2

0.9

1.0

0.2

0.4

1.3

0.4

0.4

1.0

0.2

1.0

0.3

0.3

1.0

1.0

0.4

F Visualisation of Selected Tasks

̸

In the following, we visualise a selection of benchmark tasks. Each figure visualises a sample of N = 10000 points from P XY . The upper diagonal represents the scatter plots P X i X j , P X i Y j and P Y i Y j for all possible combinations of i = j . The lower diagonal represents the empirical density fitted using a kernel density estimator with 5 levels. The diagonal represents the histograms of the marginal distributions P X i and P Y i for all possible i .

We present selected 1 × 1 -dimensional tasks in Fig. 13 and Fig. 14; the 2 × 1 -dimensional Swissroll distribution in Fig. 15 and selected 2 × 2 -dimensional tasks in Fig. 16. Finally, in Fig. 17 we present two 3 × 3 -dimensional problems, visualising the effect of the normal CDF and the spiral diffeomorphism.

Bivariate normal 1 × 1

Normal CDF @ Bivariate normal 1 × 1

Uniform 1 × 1 (additive noise=0.1)

Uniform 1 × 1 (additive noise=0.75)

Bimodal 1 × 1

Wiggly @ Bivariate normal 1 × 1

Half-cube @ Bivariate normal 1 × 1

Student-t 1 × 1 (dof=1)

Asinh @ Student-t 1 × 1 (dof=1)

Swiss roll 2 × 1

Multinormal 2 × 2 (dense)

Multinormal 3 × 3 (dense)

Multinormal 5 × 5 (dense)

Multinormal 25 × 25 (dense)

Multinormal 50 × 50 (dense)

Multinormal 2 × 2 (2-pair)

Multinormal 3 × 3 (2-pair)

Multinormal 5 × 5 (2-pair)

Multinormal 25 × 25 (2-pair)

Student-t 2 × 2 (dof=1)

Student-t 2 × 2 (dof=2)

Student-t 3 × 3 (dof=2)

Student-t 3 × 3 (dof=3)

Student-t 5 × 5 (dof=2)

Student-t 5 × 5 (dof=3)

Normal CDF @ Multinormal 3 × 3 (2-pair)

Normal CDF @ Multinormal 5 × 5 (2-pair)

Normal CDF @ Multinormal 25 × 25 (2-pair)

Half-cube @ Multinormal 3 × 3 (2-pair)

Half-cube @ Multinormal 5 × 5 (2-pair)

Half-cube @ Multinormal 25 × 25 (2-pair)

Spiral @ Multinormal 3 × 3 (2-pair)

Spiral @ Multinormal 5 × 5 (2-pair)

Spiral @ Multinormal 25 × 25 (2-pair)

Spiral @ Normal CDF @ Multinormal 3 × 3 (2-pair)

Spiral @ Normal CDF @ Multinormal 5 × 5 (2-pair)

Spiral @ Normal CDF @ Multinormal 25 × 25 (2-pair)

Asinh @ Student-t 2 × 2 (dof=1)

Asinh @ Student-t 3 × 3 (dof=2)

Asinh @ Student-t 5 × 5 (dof=2)

MINE

NWJ

D-V

0

0

0

0

0

0

0

7

0

0

0

0

0

0

0

0

0

0

0

7

2

0

0

1

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

2

0

0

0

0

0

0

0

4

0

0

0

0

0

0

0

0

0

0

0

4

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

8

0

0

0

0

0

0

0

0

0

0

3

6

3

3

6

2

3

0

0

1

0

0

3

0

0

1

0

0

1

0

3

6

0

0

0

0

0

0

0

5

0

0

0

0

0

0

0

0

0

0

0

6

1

0

1

0

2

0

0

0

0

0

0

0

0

0

0

0

0

0

0

1

Uniform 1 × 1 (additive noise=0.1)

Uniform 1 × 1 (additive noise=0.75)

1.0

1.0

0.8-

0.8

0.6-

0.6

X

0.4

0.4

0.2

0.2

00

0.0

2

1.0

0.5

0.0

0.0

0.5

1.0

0.0

0.5

1.0

0.0

0.5

1.0

X1

Y1

X1

Bivariate normal 1 × 1

Bimodal 1 × 1

7.5

5.0

X

2.5

0.0

2.5

2

2.5

0.0

2.5

5.0

-5.0

2.5

0.0

2.5

5

10

5

X1

Y1

X1

Y1

G Author Contributions

Contributions according to the Contributor Roles Taxonomy:

Conceptualization: PC, FG, AM, NB.

Methodology: FG, PC.

3.

Software: PC, FG.

Validation: FG, PC.

Formal analysis: PC, FG.

Investigation: FG, PC, AM.

Asinh @ Student-t 1 × 1 (dof=1)

Normal CDF @ Bivariate normal 1 × 1

1.0

0.8

0.6

0.4

-5

0.2

0.0

10

1.00

0.75

0.50

0.25

-5

0.00

10

0.0

0.5

1.0

0.0

0.5

1.0

-10

-5

0

5

-5

Q

X1

Y

X1

Y1

Half-cube @ Bivariate normal 1 × 1

Wiggly @ Bivariate normal 1 × 1

2

2

0

-5

10

5

0

5

-10

5

-2.50.0

2.5

5.0

2.5

0.0

2.5

X1

Y1

X1

Y1

Resources: FG, NB.

Data Curation: FG, PC.

Visualization: FG, PC.

Supervision: AM, NB.

Writing - original draft: PC, AM, FG.

Writing - review and editing: PC, AM, FG, NB, JEV.

Swiss roll 2 × 1

0.6

0.4

0.2

X

0:0

-0.2

-0.4

0.5

x

0.0

-0.5

1.00

0.75

0.50

0.25

F00'0

-0.5

0.0

0.5

-0.5

0.0

0.5

0.0

0.5

1.0

X1

X2

Y1

Multinormal 2 × 2 (dense)

Multinormal 2 × 2 (2-pair)

Student-t 2×2（dof=2)

60

X

40

201

0

-20

40

20

20

40

20

V

20

30

20

10

-10

20

25

75

20

40

Normal CDF @ Multinormal 3 × 3 (2-pair)

1.0

0.8

0.6

x

0.4

0.2

700

1.00

0.75

0.50

0.25

0.00

1.00

0.75

0s0

0.25

0.00

1.00

0.75

>0.50

0.25

0.00

1.00

0.75

0.50

0.25

0.00

1.00

SL0

> 0.50

0.25

0.00

0.0

0.5

1.0

0.0

0.5

X2

1.0

0.0

0.5

X3

1.0

0.0

0.5

Y1

1.0

0.0

0.5

1.00.00 0.25 0.50 0.75 1.00

Y3

Spiral @ Multinormal 3 × 3 (2-pair)

X

0

2

2

4

2

2

2

2

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method for creating expressive distributions via injective mappings that maintain their original MI. They state this in Theorem 2.1 and prove it in the Appendix. In addition to this, the authors benchmark a variety of estimators for MI on a set of tasks, including high-dimensional, long-tailed distributions, sparse interactions, and high mutual information. This information is contained in Sections 3-4, where they describe each task and critique each estimator for the tasks. From these experiments, the authors provide guidelines for practitioners on how to choose an estimator in Section 6.\n\nSTRENGTHS:\nThe paper is clear, provides many novel benchmarking tasks, and is easily verifiable. The results are reproducible as the authors have shared their code and documented their experimental parameters.\n\nWEAKNESSES:\n1. The author does a poor job of motivating each data setting and explaining why each one is important. It would be beneficial to provide examples of domains where long-tail distributions are common, such as insurance, and elaborate on the significance of the other data settings as well.\n2. I believe the last row of Figure 2, \"True MI,\" could be better highlighted as it was difficult to discern that it was the point of comparison.\n3. The main contributions of the paper seem relatively minor, as the authors are primarily considering more data settings than previous work when comparing MI estimators. However, addressing the first point could help alleviate this concern.\n4. The main theoretical result, Theorem 2.1, appears to have already been demonstrated in the appendix of the following paper: https://arxiv.org/pdf/cond-mat/0305641.pdf.\n5. The authors seem to have switched \\citep for \\cite in their paper.\n\nQUESTIONS:\nWhy not utilize the result of Theorem 2.1 and apply it to Normalizing Flows? I believe this combination would enhance the paper's distinctiveness. \n\nHow is Theorem 2.1 different from that of the result in the appendix of https://arxiv.org/pdf/cond-mat/0305641.pdf?\n\nLIMITATIONS:\nThe authors have addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper identifies a clear problem with many mutual information estimation benchmarks: most of them focus on simple (normal) distributions. The authors present a new set of forty (!) tasks that contain ground-truth informations, which can be constructed by noting that only injectivity is needed for an information-preserving transformation. The authors identify four distinct challenges: interaction sparsity, long tails, invariance, and high information values. Several conclusions about existing estimation algorithms are then made regarding the extensive analysis.\n\nSTRENGTHS:\n- I enjoyed reading this paper. It clearly defines a goal and identifies key problems with existing approaches. \n- The paper presents mathematical background in a precise and effective way.\n- The structure of the paper is clear, using figures for clarifications where needed.\n- The paper is well-written with clear sentence structures and no grammatical or spelling errors.\n- The analysis is thorough, presenting forty distinct MI estimation tasks.\n- I enjoyed how instead of simply presenting the results, the authors dug deeper and identified four distinct challenges for MI estimators.\n\nWEAKNESSES:\n- One could say that the paper lacks slightly in terms of originality and contributions.\n\nQUESTIONS:\n- Out of curiosity: could modern invariant neural network architectures be used to obtain MI estimates invariant to diffeomorphisms?\n\n\n### Conclusions\nWhile the overall contribution could be limited in terms of a model development sense, I think the paper identifies serious issues with modern MI estimation benchmarks. The paper not only provides new benchmarks that address these issues but also makes an effort to identify what aspects of MI estimation can make the problem hard. I foresee much new research originating from the identification of these aspects, where future papers focus in on them and propose methodologies that overcome these challenges. On top of that, the paper is very well written. Hence, I would recommend acceptance.\n\nLIMITATIONS:\nThe authors clearly discuss the limitations of their study.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA test benchmark for the evaluation of mutual information estimators is established and many different estimators compared. The test cases contain student-t and normal distributionas and their injective transformations. Difficult cases are discussed and evaluated in more detail.\n\nSTRENGTHS:\nThe code is reproducible and thus might be used for other estimators in the future\nThe paper is well-written and easy to understand.\nIt is important to make the community aware that evaluation of MI estimators on Gaussian distributions is pointless as these only depend on the covariance structure, so the big plus of the MI that it goes beyond te linear dependencies is ignored. The paper makes a strong point here by including a simple covariance estimator as well.\nThe results on heavy tails are particularly interesting.\n\nWEAKNESSES:\nThe choice of the distribution used is not sufficiently argued. In particular, it is known that no MI estimator can evaluate MI correctly on arbitrary distributions. Only with restriction to a class of probability distributions (e.g., probability density functions with Lipschitz constraints) there is hope that estimation works. It is thus quite pointless to evaluate MI estimators on what seems like an \"educated guess\" of diverse distributions. \nThe authors seem to not be aware of the huge theoretic background of MI, e.g., that arbitrary measureable injective mappings do not change MI and thus their only Theorem 2.1 is well-known in a much more general setting. This can be argued by the data processing inequality in two directions (X-Y-f(Y) and X-f(Y)-Y are both Markov chains) or directly the definition of MI via countable partitions that do not change if we use a measureable injective mapping.\n\nQUESTIONS:\nWhat was the reason for choosing this specific set of distributions? \nWhat is \"pointwise MI\"?\n\nLIMITATIONS:\nThe authors state that there are more interesting cases and they only cover transforms of normal and student-t distributions. They also mention that prior information might be incorporated.\nHowever, as said above it is known that an MI estimator can be fooled arbitrarily (giving any value for the MI) if one can choose the distribution freely. Thus, prior information is also included in the test cases here it is just not mentioned explicitly.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the topic of mutual information and shows how to construct a diverse family of distributions with known ground-truth mutual information. It's worth noting that obtaining a closed-form solution for mutual information is highly dependent on the specific assumptions and functional forms used for the variables X and Y. In practice, deriving closed-form expressions for mutual information can be challenging and may require additional simplifying assumptions or specific knowledge about the distributions involved.\n\nIn contrast to previous works that typically assess mutual information estimators using simple probability distributions, this paper introduces a novel approach to constructing a diverse family of distributions with known ground-truth mutual information. Additionally, the authors propose a language-independent benchmarking platform to assess mutual information estimators. The authors explore the applicability of classical and neural estimators in scenarios involving high dimensions, sparse interactions, long-tailed distributions, and high mutual information. By examining these challenging settings, they provide insights into the strengths and limitations of different estimators. \n\nMoreover, the paper offers guidelines for practitioners to select the most suitable estimator based on the specific problem's difficulty and considerations when applying an estimator to new datasets. By presenting a comprehensive evaluation framework and practical recommendations, this research aims to advance the understanding and application of mutual information estimation in various domains.\n\nSTRENGTHS:\nThe mutual information estimator is an essential tool in causality, it can help discover the underlying causal graph or inference the strength of causal relations. However, as I mentioned earlier, deriving closed-form expressions for mutual information can be challenging in practice and may require additional simplifying assumptions or specific knowledge about the distributions involved.\n\nThe paper introduces a method to construct a diverse family of distributions with known ground-truth mutual information. This is a significant contribution as it allows researchers to explore and evaluate mutual information estimators across various scenarios, encompassing various data characteristics and relationships. For example, explore gene regularity networks, understand the treatment effect in medical health care, gain insight for constructing a recommendation system, etc. \n\nThe research paper presents a comprehensive evaluation framework for mutual information estimation, encompassing the construction of diverse distributions, benchmarking platform, exploration of challenging scenarios, and practical guidelines. This framework provides a holistic view of the estimation process, aiding researchers and practitioners in understanding, comparing, and selecting mutual information estimators effectively.\n\nFurthermore, the authors investigate the applicability of classical and neural estimators in challenging scenarios involving high dimensions, sparse interactions, long-tailed distributions, and high mutual information. This exploration provides valuable insights into the performance, strengths, and limitations of different estimators under these challenging conditions, enhancing our understanding of their effectiveness in real-world settings.\n\nWEAKNESSES:\n1. Not all joint distributions can be represented in the form of $P_{f(x)g(x)}$, limiting the applicability of the benchmark to a specific set of distributions. Extending the family of distributions with known mutual information and efficient sampling is seen as a natural direction for future improvement.\n2. Even though the benchmark demonstrates that distributions with longer tails pose a harder challenge for the considered estimators, applying a transformation like the asinh transform does not fully address the issues.\n3. The summary does not mention external validation or comparisons between the proposed approach or estimators and existing methods or benchmarks in the field. The absence of such external validation makes it difficult to assess the generalizability or superiority of the contributions in relation to established techniques or alternative approaches.\n\nQUESTIONS:\nSee above \"weaknesses\".\n\nLIMITATIONS:\nSee above \"weaknesses\".", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}