SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Now fix point  $x$  and consider the matrix  $\gamma (||x||^2)\in O(n)$ . Define a mapping  $y\mapsto \gamma (||x||^2)^{-1}f(y)$ . As we have already observed, multiplication by  $\gamma (||x||^2)^{-1}$  does not change the value of the PDF. It also cannot change the  $|\operatorname *{det}f'(x)|$  value as because  $\gamma (||x||^2)^{-1}$  is a linear mapping, its derivative is just the mapping itself and because  $\gamma (||x||^2)^{-1}$  is an orthogonal matrix, we have  $|\operatorname *{det}\gamma (||x||^2)^{-1}| = 1$ . Hence, without loss of generality we can assume that  $\gamma (||x||^2) = I_n$  and
$$
f ^ {\prime} (x) = I _ {n} + 2 A x x ^ {T},
$$
where  $A\coloneqq \gamma^{\prime}(||x||^{2})\in \mathfrak{so}(n)$  is a skew-symmetric matrix.
If  $x = 0$  we have  $|\operatorname{det} f'(x)| = |\operatorname{det} I_n| = 1$ . In the other case, we will work in coordinates induced by an orthonormal basis  $e_1, \ldots, e_n$  in which the first basis vector is  $e_1 = x / ||x||$ . Hence,
$$
f ^ {\prime} (x) = I _ {n} + 2 | | x | | ^ {2} A e _ {1} e _ {1} ^ {T}.
$$
For  $i\geq 2$  we have  $e_1^T e_i = 0$  and consequently
$$
f ^ {\prime} (x) e _ {i} = e _ {i} + 2 | | x | | ^ {2} A e _ {1} e _ {1} ^ {T} e _ {i} = e _ {i}.
$$
This shows that in this basis we have
$$
f ^ {\prime} (x) = \left( \begin{array}{c c c c c} a _ {1} & 0 & 0 & \ldots & 0 \\ a _ {2} & 1 & 0 & \ldots & 0 \\ a _ {3} & 0 & 1 & \ldots & 0 \\ \vdots & \vdots & \vdots & \ddots & 0 \\ a _ {n} & 0 & 0 & \dots & 1 \end{array} \right),
$$
where the block on the bottom right is the identity matrix  $I_{n - 1}$  and  $\operatorname*{det}f^{\prime}(x) = a_{1}$ .
The first column is formed by the coordinates of
$$
f ^ {\prime} (x) e _ {1} = e _ {1} + 2 | | x | | ^ {2} A e _ {1}
$$
$$
a _ {1} = e _ {1} ^ {T} f ^ {\prime} (x) e _ {1} = 1 + 2 | | x | | ^ {2} e _ {1} ^ {T} A e _ {1} = 1 + 0 = 1
$$
as  $A\in \mathfrak{so}(n)$  is skew-symmetric.
C Overview of Estimators
There exists a broad literature on (conditional) mutual information estimators ranging from estimators on discrete data [Paninski, 2003, Valiant and Valiant, 2011, Vinh et al., 2014, Marx and Vreeken, 2019], over estimators for continuous data [Kriskov et al., 2004, Darbellay and Vajda, 1999, Belghazi et al., 2018] to estimators that can consider discrete-continuous mixture data [Gao et al., 2017a, Marx et al., 2021, Mesner and Shalizi, 2020]. Here, we focus on the most commonly used class, which also finds applications in representation learning: estimators for continuous data. For continuous data, the most widely propagated classes of estimators are histogram or partition-based estimators, nearest neighbor-based density estimators, kernel methods, and neural estimators. In the following, we briefly discuss a representative selection of estimators from these classes, as well as model-based estimation.
Histogram-Based Estimators A natural way to approximate mutual information, as stated in Definition A.2, is by partitioning the domain space of  $X$  and  $Y$  into segments and estimate the densities locally. This approach is also known as histogram density estimation, where the probability mass function is approximated locally for each hyper-rectangle  $E_{i}$  by  $\hat{P}_X(E_i) = \frac{n_i / N}{\mathrm{Vol}_i}$ , where  $\mathrm{Vol}_i$  is the volume of  $E_{i}$ , and  $n_i$  refers to the number of data points residing in  $E_{i}$ .
To estimate mutual information, we explore the space of allowed partitionings and select the partitioning that maximizes mutual information [Darbellay and Vajda, 1999, Cellucci et al., 2005]
$$
E _ {\mathrm {D i s c}, N} ^ {X Y} = \sup _ {\mathcal {E}, \mathcal {F}} \sum_ {i, j} \hat {P} _ {X Y} (E _ {i} \times F _ {j}) \log \frac {\hat {P} _ {X Y} (E _ {i} \times F _ {j})}{\hat {P} _ {X} (E _ {i}) \hat {P} _ {Y} (F _ {j})}.
$$
Many estimators follow this idea. Darbellay and Vajda [1999] use adaptive partitioning trees and propose to split bins until the density within a bin is uniform, which they check with the  $\chi^2$  test. Daub et al. [2004] use B-Splines to assign data points to bins, and other authors use regularization based on the degrees of freedom of partitioning to avoid overfitting [Suzuki, 2016, Cabeli et al., 2020, Vinh et al., 2014, Marx et al., 2021].
A well-known property of histogram-based estimators is that they are consistent for  $N \to \infty$  if the volume of the largest hypercube goes to zero, while the number of hypercubes grows sub-linearly with the number of samples,  $N$  (see e.g. Cover and Thomas [2006]). For low-dimensional settings, histogram density estimation can provide accurate results, but the strategy suffers from the curse of dimensionality [Pizer et al., 1987, Klemelä, 2009].
In our evaluation, we resort to partitioning estimators based on a fixed number of grids (10 equal-width bins per dimension), since more recent partitioning-based estimators focus on estimating conditional MI and only allow  $X$  and  $Y$  to be one-dimensional [Suzuki, 2016, Marx et al., 2021, Cabeli et al., 2020], or do not have a publicly available implementation [Darbellay and Vajda, 1999].
kNN Estimators An alternative to histogram-based methods is to use  $k$ -nearest neighbor ( $k$ NN) distances to approximate densities locally. Most generally, kNN estimators express mutual information as the sum  $H(X) + H(Y) - H(X,Y)$ , which can be done under the assumptions discussed in Remark A.3, and estimate the individual entropy terms as
$$
\hat {H} ((x _ {1}, \dots , x _ {N})) = - \frac {1}{N} \sum_ {i = 1} ^ {N} \log \frac {k (x _ {i}) / _ {N}}{\operatorname {V o l} _ {i}},
$$
where  $k(x_{i})$  is the number of data points other than  $x_{i}$  in a neighborhood of  $x_{i}$ , and  $\mathrm{Vol}_i$  is the volume of the neighborhood [Kozachenko and Leonenko, 1987]. The quality of the estimator depends on two
factors: i) the distance metric used to estimate the nearest neighbor distances, and ii) the estimation of the volume terms.
The most well-known estimator is the Kraskov, Stögbauer and Grassberger (KSG) estimator [Kriskov et al., 2004]. In contrast to the first  $k$ -NN-based estimators [Kozachenko and Leonenko, 1987], Kraskov et al. [2004] propose to estimate the distance to the  $k$ -th neighbor  $\rho_{i,k/2}$  for each data point  $i$  only on the joint space of  $\mathcal{Z} = \mathcal{X} \times \mathcal{Y}$ , with
$$
D _ {Z} (z _ {i}, z _ {j}) = \max \{D _ {X} (x _ {i}, x _ {j}), D _ {Y} (y _ {i}, y _ {j}) \},
$$
where  $D_X$  and  $D_Y$  can refer to any norm-induced metric (e.g., Euclidean  $\ell_2$ , Manhattan  $\ell_1$  or Chebyshev  $\ell_{\infty}$ ). To compute the entropy terms of the sub-spaces  $\mathcal{X}$  and  $\mathcal{Y}$ , we count the number of data points with distance smaller than  $\rho_{i,k} / 2$ , i.e. we define  $n_{x,i}$  as
$$
n _ {x, i} = \left| \left\{x _ {j}: D _ {X} \left(x _ {i}, x _ {j}\right) <   \frac {\rho_ {i , k}}{2}, i \neq j \right\} \right|, \tag {1}
$$
and specify  $n_{y,i}$  accordingly. As a consequence, all volume-related terms cancel and the KSG estimator reduces to
$$
E _ {\mathrm {K S G}, N} ^ {X Y} = \psi (k) + \psi (n) - \frac {1}{n} \sum_ {i = 1} ^ {n} \psi \left(n _ {x, i} + 1\right) + \psi \left(n _ {y, i} + 1\right). \tag {2}
$$
The diagram function $^6$ $\psi$  was already introduced by Kozachenko and Leonenko [1987] to correct for biases induced by the volume estimation. For a detailed exposition of the issue, we refer to Kraskov et al. [2004, Sec. II.A].
There exist various extensions of the classical KSG estimator. When selecting the  $L_{\infty}$  norm as a distance measure for  $D_X$  and  $D_Y$ ,  $E_{\mathrm{KSG},N}^{XY}$  can be efficiently computed with the K-d trie method [Ve-jmelka and Hlaváčková-Schindler, 2007]. Other approaches aim to improve the estimation quality by explicitly computing the volume terms via SVD [Lord et al., 2018] or PCA [Lu and Peltonen, 2020], or use manifold learning [Marx and Fischer, 2022] or normalizing flows [Ao and Li, 2022] to estimate the nearest neighbors more efficiently in high-dimensional settings.
Kernel Estimators A smaller class of estimators builds upon kernel density estimation (KDE) [Moon et al., 1995, Paninski and Yajima, 2008]. Classical approaches compute the density functions for the individual entropy terms via kernel density estimation, which is well-justified under the assumption of  $P_{XY}$  being absolutely continuous with respect to  $P_{X} \otimes P_{Y}$  (cf. Remark A.3). i.e., the density for a  $d$ -dimensional random vector  $X$  is approximated as
$$
\hat {p} (x) = \frac {1}{N h ^ {d}} \sum_ {i = 1} ^ {N} K \left(\frac {x - x _ {i}}{h}\right),
$$
where  $h$  is the smoothing parameter or bandwidth of a Gaussian kernel [Silverman, 2018].
Moon et al. [1995] propose to estimate mutual information using a Gaussian kernel, for which the bandwidth can be estimated optimally [Silverman, 2018]. Similar to histogram-based approaches, KDE-based estimators with fixed bandwidth  $h$  are consistent if  $Nh^{d} \to \infty$  as  $h \to 0$ .
A more recent proposal, LNN [Gao et al., 2017b], uses a data-dependent bandwidth based on nearest-neighbor distances, to avoid fixing  $h$  globally.
Neural Estimators Neural network approaches such as MINE [Belghazi et al., 2018] use a variational lower bound on mutual information using the Donsker-Varadhan theorem: let  $f\colon \mathcal{X}\times \mathcal{Y}\to \mathbb{R}$  be a bounded measurable function and
$$
V _ {f} (X, Y) = \sup  _ {f} \mathbb {E} _ {P _ {X Y}} [ f (X, Y) ] - \log \mathbb {E} _ {P _ {X} \otimes P _ {Y}} [ \exp f (X, Y) ].
$$
Under sufficient regularity conditions on  $P_{XY}$ ,  $\mathbf{I}(X;Y)$  is equal to  $V_{f}(X,Y)$ , which is evaluated over all bounded measurable mappings  $f$ . In practice, the right hand side is optimized with respect
In the original approach, the  $L_{\infty}$  is used, and Gao et al. [2018] derives theoretical properties of the estimator for the  $L_{2}$  norm.
The digamma function is defined as  $\psi (x) = \mathrm{d}\log \Gamma (x) / \mathrm{d}x$ . It satisfies the recursion  $\psi (x + 1) = \psi (x) + \frac{1}{x}$  with  $\psi (1) = -\gamma$ , where  $\gamma \approx 0.577$  is the Euler-Mascheroni constant.
to the parameters of a given parametric predictive model  $f$ , as a neural network. Approaches following this idea also count as discriminative approaches since the neural network is trained to distinguish points from the joint and marginal distribution. The literature on variational lower bounds of mutual information is growing [Oord et al., 2018, Song and Ermon, 2020] and the idea of using a discriminative approach to estimate mutual information has also been explored by Koeman and Heskes [2014], who used random forests. For an excellent theoretical overview of different neural estimators we refer to Poole et al. [2019].
Model-based Estimators Finally, if one assumes that  $P_{XY}$  is a member of a family of distributions  $Q_{XY}(\theta)$  with known mutual information, one can estimate  $\theta$  basing on the data  $((x_1, y_1), \ldots, (x_N, y_N))$  and calculate the corresponding mutual information. For example, if one assumes that there exist jointly multivariate normal variables  $\tilde{X} = (\tilde{X}_1, \dots, \tilde{X}_r, E_1, \dots, E_{m-r})$  and  $\tilde{Y} = (\tilde{Y}_1, \dots, \tilde{Y}_r, \tilde{E}_1, \dots, \tilde{E}_{n-r})$  such that the cross-covariance terms are zero apart from  $\mathrm{Cov}(\tilde{X}_i, \tilde{Y}_i)$ , and that  $X = A\tilde{X} + a$  and  $Y = B\tilde{Y} + b$  for invertible matrices  $A$  and  $B$  and vectors  $a \in \mathbb{R}^m$  and  $b \in \mathbb{R}^n$ , then the mutual information can be found as
$$
\mathbf {I} (X; Y) = - \sum_ {i = 1} ^ {r} \log \sqrt {1 - \operatorname {C o r} (\tilde {X} _ {r} , \tilde {Y} _ {r}) ^ {2}}.
$$
The model parameters  $A$ ,  $B$ ,  $a$ , and  $b$ , as well as the pairs of corresponding r.v.  $\tilde{X}_r$  and  $\tilde{Y}_r$ , can be found by maximising likelihood using canonical correlation analysis (CCA) [Murphy, 2023, Ch. 28], which has been previously explored by Kay [1992].
D Additional Information on Benchmark Tasks
D.1 Task Descriptions
Bimodal Variables Consider a r.v.  $X$  distributed according to a Gaussian mixture model with CDF  $F = w_{1}F_{1} + w_{2}F_{2}$ , where  $F_{1}$  and  $F_{2}$  are CDFs of two Gaussian distributions and  $w_{1}$  and  $w_{2}$  are positive weights constrained as  $w_{1} + w_{2} = 1$ . The CDF is everywhere positive, continuous, and strictly increasing. Hence, it has an inverse, the quantile function,  $F^{-1}\colon (0,1)\to \mathbb{R}$ , which is continuous and injective as well (see Lemma D.1). Thus, we can define r.v.  $X' = F^{-1}(X)$ , where  $X\sim \mathrm{Uniform}(0,1)$  and  $\mathbf{I}(X;Y) = \mathbf{I}(X';Y)$  for every r.v.  $Y$ . We generated two CDFs of Gaussian mixtures and implemented a numerical inversion in JAX [Bradbury et al., 2018] defining both  $X'$  and  $Y'$  to follow a bimodal distribution.
For the experimental values, we used
$$
F _ {X} (x) = 0. 3 F (x) + 0. 7 F (x - 5)
$$
$$
F _ {Y} (y) = 0. 5 F (x + 1) + 0. 5 F (x - 3),
$$
where  $F$  is the CDF of the standard normal distribution  $\mathcal{N}(0,1)$ .
Wiggly Mapping As the "wiggly mapping" we used
$$
w _ {X} (x) = x + 0. 4 \sin x + 0. 2 \sin (1. 7 x + 1) + 0. 0 3 \sin (3. 3 x - 2. 5)
$$
$$
w _ {Y} (y) = y - 0. 4 \sin (0. 4 y) + 0. 1 7 \sin (1. 3 y + 3. 5) + 0. 0 2 \sin (4. 3 y - 2. 5)
$$
and we used visualisations to make sure that they do not heavily distort the distribution.
Additive Noise For the additive noise model of  $Y = X + N$  where  $X \sim \mathrm{Uniform}(0,1)$  and  $N \sim \mathrm{Uniform}(-\varepsilon, \varepsilon)$  it holds that
$$
\mathbf {I} (X; Y) = \left\{ \begin{array}{l l} \varepsilon - \log (2 \varepsilon) & \text {i f} \varepsilon \leq 1 / 2 \\ (4 \varepsilon) ^ {- 1} & \text {o t h e r w i s e} \end{array} \right.
$$
implying that  $\mathbf{I}(X;Y) = 1.7$  for  $\varepsilon = 0.1$  and  $\mathbf{I}(X;Y) = 0.3$  for  $\varepsilon = 0.75$ .
Swiss Roll Embedding Consider a bivariate distribution  $P_{XY}$  with uniform margins  $X \sim \mathrm{Uniform}(0,1)$  and  $Y \sim \mathrm{Uniform}(0,1)$  and  $\mathbf{I}(X;Y) = 0.8$ . The Swiss roll mapping is an embedding  $i\colon (0,1)^2 \to \mathbb{R}^3$  given by  $i(x,y) = (e(x),y)$ , where  $e\colon (0,1) \to \mathbb{R}^2$  is given by
$$
e (x) = \frac {1}{2 1} (t (x) \cos t (x), t (x) \sin t (x), \qquad t (x) = \frac {3 \pi}{2} (1 + 2 x).
$$
Note that  $P_{X'Y'}$  does not have a PDF with respect to the Lebesgue measure on  $\mathbb{R}^3$ . We visualise the Swiss roll distribution in Fig. 1.
Spiral Diffeomorphism For the precise definition and properties of the general spiral diffeomorphism consult Appendix B.
In our benchmark we applied the spiral  $x \mapsto \exp(vA||x||^2)x$  to both  $X$  and  $Y$  variables. We used an  $m \times m$  skew-symmetric matrix  $A_X$  with  $(A_X)_{12} = -(A_X)_{21} = 1$  and an  $n \times n$  skew-symmetric matrix  $A_Y$  with  $(A_Y)_{23} = -(A_Y)_{32} = 1$ . Each of these effectively "mixes" only two dimensions.
The speed parameters  $v_{X}$  and  $v_{Y}$  were chosen as  $v_{X} = 1 / m$  and  $v_{Y} = 1 / n$  to partially compensate for the fact that if  $X \sim \mathcal{N}(0, I_m)$ , then  $||X||^2 \sim \chi_m^2$  and has mean  $m$ .
Multivariate Student Let  $\Omega$  be an  $(m + n) \times (m + n)$  positive definite dispersion matrix and  $\nu$  be a positive integer (the degrees of freedom). By sampling an  $(m + n)$ -dimensional random vector  $(\tilde{X}, \tilde{Y}) \sim \mathcal{N}(0, \Omega)$  and a random scalar  $U \sim \chi_{\nu}^{2}$  one can define rescaled variables  $X = \tilde{X} \sqrt{\nu / U}$  and  $Y = \tilde{Y} \sqrt{\nu / U}$ , which are distributed according to the multivariate Student distribution. For  $\nu = 1$  this reduces to the multivariate Cauchy distribution (and the first two moments are not defined), for  $\nu = 2$  the mean exists, but covariance does not, and for  $\nu > 2$  both the mean and covariance exist, with  $\operatorname{Cov}(X, Y) = \frac{\nu}{\nu - 2} \Omega$ , so that the tail behaviour can be controlled by changing  $\nu$ . In particular, for  $\nu \gg 1$  because of the concentration of measure phenomenon  $U$  has most of its probability mass around  $\nu$  and the variables  $(X, Y)$  can be well approximated by  $(\tilde{X}, \tilde{Y})$ .
Arellano-Valle et al. [2013] proved that  $\mathbf{I}(X;Y)$  can be computed via the sum of the mutual information of the Gaussian distributed basis variables and a correction term, i.e.,  $\mathbf{I}(X;Y) = \mathbf{I}(\tilde{X};\tilde{Y}) + c(\nu ,m,n)$ , where
$$
c (\nu , m, n) = f (\nu) + f (\nu + m + n) - f (\nu + m) - f (\nu + n), \quad f (x) = \log \Gamma \left(\frac {x}{2}\right) - \frac {x}{2} \psi \left(\frac {x}{2}\right),
$$
and  $\psi$  is the digamma function.
Contrary to the Gaussian case, even for  $\Omega = I_{m + n}$ , the mutual information  $\mathbf{I}(X;Y) = c(\nu ,m,n)$  is non-zero, as  $U$  provides some information about the magnitude. In the benchmark we use this dispersion matrix to quantify how well the estimators can use the information contained in the tails, rather than focusing on estimating the Gaussian term.
D.2 Covariance Matrix Family
We generated jointly multivariate normal variables  $X = (X_{1},\ldots ,X_{m})$  and  $Y = (Y_{1},\dots ,Y_{n})$  with the following procedure.
Consider i.i.d. Gaussian variables
$$
U _ {\text {a l l}}, U _ {X}, U _ {Y}, Z _ {1}, \dots , Z _ {K}, E _ {1} \dots , E _ {m}, F _ {1}, \dots , F _ {n}, E _ {m - K} ^ {\prime}, \dots , E _ {m} ^ {\prime}, F _ {n - K} ^ {\prime}, \dots , F _ {n} ^ {\prime} \sim \mathcal {N} (0, 1).
$$
Now let  $\varepsilon_{X},\varepsilon_{Y},\eta_{X},\eta_{Y},\alpha ,\beta_{X},\beta_{Y},\lambda \in \mathbb{R}$  be hyperparameters and for  $l = 1,2,\ldots ,K$  define
$$
X _ {l} = \varepsilon_ {X} E _ {l} + \alpha U _ {\text {a l l}} + \beta_ {X} U _ {X} + \lambda Z _ {l}, \quad Y _ {l} = \varepsilon_ {Y} F _ {l} + \alpha U _ {\text {a l l}} + \beta_ {Y} U _ {Y} + \lambda Z _ {l},
$$
for  $l = K + 1,\ldots ,m$  define
$$
X _ {l} = \varepsilon_ {X} E _ {l} + \alpha U _ {\text {a l l}} + \beta_ {X} U _ {X} + \eta_ {X} E _ {l} ^ {\prime}
$$
and for  $l = K + 1, \ldots, n$  define
$$
Y _ {l} = \varepsilon_ {Y} F _ {l} + \alpha U _ {\text {a l l}} + \beta_ {Y} U _ {Y} + \eta_ {Y} F _ {l} ^ {\prime}.
$$
The interpretation of the hyperparameters is the following:
1.  $\alpha$  contributes to the correlations between all the possible pairs of variables.
2.  $\beta_{X}$  contributes to the correlations between the  $X$  variables.
3.  $\beta_{Y}$  contributes to the correlations between the  $Y$  variables.
4.  $\lambda$  gives additional "interaction strength" between pairs of variables  $X_{l}$  and  $Y_{l}$  for  $l = 1, \ldots, K$ .
5.  $\varepsilon_{X}$  and  $\varepsilon_{Y}$  control part of the variance non-shared with any other variable.
6.  $\eta_{X}$  and  $\eta_{Y}$  can be used to increase the variance of  $X_{l}$  and  $Y_{l}$  for  $l > K$  to match the variance of variables  $l\leq K$  due to the  $\lambda$ .
Every term of the covariance matrix is easy to calculate analytically:
$$
\operatorname {C o v} (X _ {i}, X _ {j}) = \alpha^ {2} + \beta_ {X} ^ {2} + \mathbf {1} [ i = j ] \left(\varepsilon_ {X} ^ {2} + \lambda^ {2} \mathbf {1} [ i \leq K ] + \eta_ {X} ^ {2} \mathbf {1} [ i > K ]\right)
$$
$$
\mathrm {C o v} (Y _ {i}, Y _ {j}) = \alpha^ {2} + \beta_ {Y} ^ {2} + {\bf 1} [ i = j ] \big (\varepsilon_ {Y} ^ {2} + \lambda^ {2} {\bf 1} [ i \leq K ] + \eta_ {Y} ^ {2} {\bf 1} [ i > K ] \big)
$$
$$
\operatorname {C o v} (X _ {i}, Y _ {j}) = \alpha^ {2} + \lambda^ {2} {\bf 1} [ i = j ] {\bf 1} [ i \leq K ]
$$
In the following, we analyse some special cases.
Dense Interactions Consider  $\lambda = \eta_{X} = \eta_{Y} = \beta_{X} = \beta_{Y} = 0$  and write  $\varepsilon \coloneqq \varepsilon_{X} = \varepsilon_{Y}$ . We have
$$
\operatorname {C o v} (X _ {i}, X _ {j}) = \operatorname {C o v} (Y _ {i}, Y _ {j}) = \alpha^ {2} + \varepsilon^ {2} \mathbf {1} [ i = j ]
$$
$$
\operatorname {C o v} \left(Y _ {i}, Y _ {j}\right) = \alpha^ {2}
$$
Hence, between every pair of distinct variables we have the same correlation  $\frac{\alpha^2}{\alpha^2 + \varepsilon^2}$ .
Sparse Interactions Consider  $\alpha = 0, \varepsilon \coloneqq \varepsilon_{X} = \varepsilon_{Y}, \eta_{X} = \eta_{Y} = \lambda$  and  $\beta_{X} = \beta_{Y} = 0$
$$
\operatorname {C o v} \left(X _ {i}, X _ {j}\right) = \operatorname {C o v} \left(Y _ {i}, Y _ {j}\right) = \mathbf {1} [ i = j ] \left(\varepsilon^ {2} + \lambda^ {2}\right)
$$
$$
\operatorname {C o v} (X _ {i}, Y _ {j}) = \lambda^ {2} \mathbf {1} [ i = j ] \mathbf {1} [ i \leq K ]
$$
All the variables have the same variance, but the covariance structure is now different. Between (distinct)  $X_{i}$  and  $X_{j}$  the correlation is zero and similarly for (distinct)  $Y_{i}$  and  $Y_{j}$ .
Correlations between  $X_{i}$  and  $Y_{j}$  are zero unless  $i = j \leq K$ , when  $\operatorname{Cor}(X_i, Y_i) = \lambda^2 / (\varepsilon^2 + \lambda^2)$ .
Interpolation In Section 4 we discussed a two-stage interpolation from dense matrices to sparse matrices: in the first stage  $\alpha$  is decreased increasing  $\lambda$  at the same time to keep the mutual information constant. In the second stage we decrease the number of interacting variables  $K$  increasing  $\lambda$ . The selected covariance matrices from this two-stage process are visualised in Fig. 7.
D.3 Technical Results
In this subsection we add the technical results, related to the copula theory [Nelsen, 2007] and uniformization of random variables.
Lemma D.1. Let  $X$  be a real-valued r.v. with a strictly increasing, positive everywhere, and continuous CDF  $F$ . Then
$$
F \colon \mathbb {R} \to (0, 1)
$$
is a homeomorphism.
Figure 7: Interpolation between dense and sparse covariance matrices.
Proof. As  $F$  is a CDF we have  $F(\mathbb{R}) \subseteq [0,1]$ . Note that 0 is excluded from the image as  $F$  is positive everywhere and 1 is excluded as  $F$  is strictly increasing. Hence,  $F(\mathbb{R}) \subseteq (0,1)$ . Using continuity, strict monotonicity, and
$$
\lim  _ {x \rightarrow - \infty} F (x) = 0, \quad \lim  _ {x \rightarrow + \infty} F (x) = 1
$$
we note that  $F$  is a bijection onto  $(0,1)$ . Analogously, we prove that
$$
F ((a, b)) = (F (a), F (b))
$$
for all  $a < b$ , so  $F$  is an open map. As a continuous bijective open map  $F$  is a homeomorphism.
Lemma D.2. Let  $X$  be a real-valued r.v. with a CDF  $F$  which is continuous, strictly increasing and positive everywhere. Then,  $F(X)$  is uniformly distributed on  $(0,1)$ .
Proof. Our assumptions on  $F$  imply that it is a continuous homeomorphism  $\mathbb{R}\to (0,1)$ .
Let now  $y\in (0,1)$  .We have
$$
P (F (X) \leq y) = P (X \leq F ^ {- 1} (y)) = F \left(F ^ {- 1} (y)\right) = y.
$$
E.1 Pointwise Mutual Information
To understand why neural estimators are not able to estimate the mutual information contained in the Student distribution, we decided to qualitatively study the learned critic.
In the NWJ approach [Nguyen et al., 2007], for  $N\to \infty$  data points the optimal critic  $f(x,y)$  is given by
$$
f (x, y) = 1 + \operatorname {P M I} (x, y),
$$
$$
\operatorname {P M I} (x, y) = \log \frac {p _ {X Y} (x , y)}{p _ {X} (x) p _ {Y} (y)}
$$
is the pointwise mutual information [Pinsker and Feinstein, 1964, Ch. 2] and  $p_{XY}$ ,  $p_X$  and  $p_Y$  are the PDFs of measures  $P_{XY}$ ,  $P_X$  and  $P_Y$ , respectively.
Figure 8: Qualitative comparison between ground-truth pointwise MI and learned critic for bivariate Student and normal distributions.
Figure 9: Minimal number of samples needed for the mutual information estimate to fall between  $2/3$  and  $3/2$  of the true MI. The minimal number of samples tested is 100, the maximal 10 000. The entry  $>10k$  means that the desired accuracy was never achieved. For neural estimators 1 000 is the minimal possible number of samples, since we use a  $50\%$  train-test split and a batch size of 256.