On the Properties and Estimation of Pointwise Mutual Information Profiles
Authors: Paweł Czyż
Frederic Grabowski
Julia E Vogt
Niko Beerenwinkel
Alexander Marx
Venue: Accepted by TMLR ()

Abstract: The pointwise mutual information profile, or simply profile, is the distribution of pointwise mutual information for a given pair of random variables. One of its important properties is that its expected value is precisely the mutual information between these random variables. In this paper, we analytically describe the profiles of multivariate normal distributions and show that for an expressive family of distributions, termed Bend and Mix Models, the profile can be accurately estimated using Monte Carlo methods. We then show how Bend and Mix Models can be used to study the limitations of existing mutual information estimators, investigate the behavior of neural critics used in variational estimators, and understand the effect of experimental outliers on mutual information estimation. Finally, we show how Bend and Mix Models can be used to obtain model-based Bayesian estimates of mutual information, suitable for problems with available domain expertise in which uncertainty quantification is necessary. The accompanying code is available at https://github.com/cbg-ethz/bmi.

Published in Transactions on Machine Learning Research (01/2025)
On the Properties and Estimation of Pointwise Mutual Information Profiles
Pawel Czyž"
nnwelpiotr.cayzSat etha.ch
ETH AT Center and Deportment of Biosystems Science and Engineering
ETH Zurich
Zurich, Switzerland
Frederic Grabowski"
Institute of Pundamental Technological Research Warsaw, Poland
Julia E. Vogt
julia.eogteee www.winft.oon.ooooooo
@inf.ethz.ck ethe..ch ch
Department of Computer Science
ETH Zurich
SIB Swiss Institute for Bioinformatics
Zurich, Switzerland
Niko Beerenwinkel!
wwk..bevennuinkeeebss..dl.....
Deportment of Biosystems Science and Engineering
ETH Zurich
SIB Suiiss Institude for Bioinformaties
Basel, Switzerland
Alexander Marx
alerander. wararddrrmmrrrltorrmmmmmttt
Research Center Trustworthy Dala Science and Securily of the University Alliance Rulo Deportment of Statistics
TU Dortmund University
Dortmand, Germany
Reviewed on OpenReview: https: // openreview. forum? id=LdfLD41Ght
Abstract
The pointwise mutual information profile, simply profile, is the distribution of pointwise mutual information for given pair of random variables. One of its important properties is that its expected value precisely the mutual information between these random variables. In this paper, we analytically describe the profiles of multivariate normal distributions and show that for an expressive family of distributions, termed Bend and Mir Models, the profile can be accurately estimated using Monte Carlo methods. We then show how Bend and Mix Models be used to study the limitations of existing mutual information estimators, investigate the behavior of neural critics used in variational estimators, and understand the effect of experimental outliers on mutual information estimation. Finally, we show how Bend and Mix Models can be used to obtain model-based Bayesian estimates of mutual information, suitable for problems with available domain expertise which uncertainty quantification ethz/bmi.
necessary. The accompanying code available https: /ttgttttttbbt////////////
'Equal contribution. Joint supervision.

---

Published in Transactions on Machine Learning Research (01/2025)
Bivariate normal
Transformed
Mixture
PMI profiles
X
X
X
0
2
PMI
Figure 1: First two panels: samples from bivariate normal distribution and the same distribution with marginals transformed. Both distributions have the same PMI profilc (blue histogram in the fourth pancl). Third panel: mixture distribution, which cannot be obtained as a transformation of the normal distribution due to a distinct PMI profile (green histogram in the fourth panel). All three distributions have the same mutual information, marked with the black line in panel four.
1 Introduction
Mutual information (MI) is key measure of dependence between two random variables (I.v.s). Due to its theoretical properties, including invariance to reparametrization, it has found many applications (Polyanskiy & Wu, 2022). However, ostimating mutual information from a finito sample remains challenging (McAllester & Stratos, 2020) and the development of mutual information estimators remains an active area research (Kay, 1992; Kraskov et al., 2004; Oord et al., 2018; Belghazi et al., 2018; Carrara & Ernst, 2023). In response the introduction of new estimators, various benchmarks have been proposed to assess trade-offs (Khan et al., 2007: Poole et al., 2019; Czyž et al., 2023).
In this manuscript we study the pointwise mutual information profile (PMI profile), closely related statistical measure which has received much less attention. The PMI profile is a distribution whose expected value is the mutual information between two considered r.v.s. We determine the PMI profile for multivariate normal distributions analytically and show that the PMT profile, similarly to mutual information, is invariant reparametrization. This last property has an important implication for MI benchmarks (Czyà et al., 2023) which transform simple distributions with known MI to create more complex benchmarking lasks: using the invariance of the profile, demonstrate that the family of distributions obtained in this manner is inherently limited. For example, distributions such as those studied by Grabowski et al. (2019) and Carrara & Ernst (2023), cannot be obtained by transforming a normal distribution.
To overcome this limitation, we study a family of distributions, here termed Bend and Mix Models (BMMs), for which the PMI profile can be efficiently approximated. By extension, since MI the expected value of the PMI profile, it can also be readily estimated using Monte Carlo methods. BMMs include highly-expressive Gaussian mixture models, and thus allow to model complex distributions. Through series of numerical experiments demonstrate the usefulness of BMMs for creating non-trivial benchmark tasks; in particular, we discuss robustness of mutual information estimators to inlier and outlier noise. Additionally, BMMs allow us to investigate the properties of PMT profiles directly. Based on empirical evidence, we argue that existing estimators based on neural variational lower bounds (Nguyen et al., 2007; Belghazi et al., 2018) implicitly estinate the PMI profile, even though they do not reliably estimate the pointwise mutual information function.
Finally, we show that BMMs can provide model-based Bayesian estimates of both the MI and the PMI profile, generalizing the carlier approaches of Kay (1992) and Brillinger (2004) to large class of models. Although this approach is not universal, we demoustrate that in low-dinensional problems with available domain expertise, BMMs provide state-of-the-art point estimates. Additionally, they offer a principled approach to uncertainty quantification in MI estimation, which has been lacking in other approaches.
summarize, the main contributions of this manuscript are as follows:
We describe the PMI profile, an invariant generalizing mutual information, determine the PMI profile for normal distributions and prove further characterising properties (Sec. 2.1).
2

---

Published in Transactions on Machine Learning Research (01/2025)
We show that for expressive family of distributions, called Bend and Mix Models, the PMI profile, and hence mutual information, is tractable using Monte Carlo methods (Sec. 2.2).
We use Bend and Mix Models to construct novel benchmarking tasks (Sec. 3.1-3.2 and Appendix C.2), study variational MI estimalors (Sec. 3.3), and propose model-based Bayesian MI estimators (Sec. 3.4).
2
Theoretical framework
In this section, we first introduce PMI profiles and then define Bend and Mix Models, an expressive family of distributions for which the PMI profile can be estimated with Monte Carlo methods.
2.1 Pointwise mutual information profiles
Consider random variables X and Y valued in spaces X and V. Provided that the joint distribution Pxy is sufficiently regular (in the sense of belonging to the family PVX,Y) defined in Appendix A.1), the mutual information is given as follows (Pinsker & Feinstein, 1964):
Let X and Y be random variables valued in spaces X and y, respectively, such that Pxy e P(X,)). The mutual information (MI) is defined as I(X;Y) B( [PMIxy v), where the pointwise mutual information (PMT) function is given as'
Definition 1.
PMIxy(I,U) = log px px (x
where pxY; Px and py are PDFs (or PMFs) of the joint and marginal distributions, respectively.
While mutual information measures whether two variables are independent (which is equivalent to I(X; = 0), the pointwise mutual information function can be understood as measure of similarity of two events: is positive they jointly occur at a higher frequency than if they were sampled independently from the marginal distributions. It is also often associated with variational MI estimators (Nguyen al., 2007; Belghazi el al., 2018; Oord al., 2018), where aims to approximate it using neural network (cf. Sec. 3.3). However, in this work we focus on the distribution of its values, i.c., its profile:
Definition 2. The pointwise madual information profile" Profxy is defined as the distribution of the random variable T (X,
We discuss generalizations of the PMI profile in Appendix A.5. A key property of the PMI profile that its mean equal to mutual information, I(X;Y) Er-Profxy [T]. In the following, we characterize the profile for multivariate normal and discrete distributions, and study its invariance properties. It is known that MI is invariant under diffeomorphisms (Kraskov et al., 2004). More generally, in Appendix A.1, we show that the entire profile is invariant:
Theorem 3. Let Pxy E E((,)) and f: X + X and g: y + y be diffeomorphisms. Then for X'= f(X) and Y' gYY il holds that Pxiy E P(X,)) and Prof = Prof
We demonstrate this result in Fig. 1. The profile of the mixture distribution cannot be obtained transformation of the bivariate normal distribution. This, in particular, implies that benchmarks relying transforming normal distributions for generating more complex problems (Cxy et al., 2023), cannot create distributions with PMI profiles that differ from the normal distributions being transformed.
In fact, for multivariate normal distributions we can express the PMI profile analytically based on the notion of canonical correlations (Hotelling, 1936):
Theorem 4. Let X and Y be r.u.s such that the joint distribution Pxy E P(R",R" is multivariate normal. If 1: = min(m, n) and Pl, .2.....Pk Pk are canonical correlations between X and Y, then the profile Prof is a
We the natural logarithm, meaning that all quantitics measured in nats. 2 Althouggh not of a prior formal definition and studies of the PMI profile, histograms of approximate PMI between words have been studied before in the computational linguistics community (Allen & Hospedales, 2019).
3

---

Published in Transactions on Machine Learning Research (01/2025)
generolized x² distribution, namely the distribution of the variable
T T=I(X;Y) Pt (Q: -1).
where {Qr, Q independent and identically distributed (t.i.d.) variables drawn from the xi distribu- tion. In particular, Prof xy is symmetric oround its median, which coincides with the mean
A I(xY)) =- Σ log(1 -p?).
and all moments of this distribution erist. Its variance is equal to Var(T] = pt + -...........
+
Additionally, in Appendix A.2, we derive bounds on the variance of the PMI profile of multivariate normal distributions given the MI. In the same place, we prove the following results, characterising PMI profiles when MT is zero and for discrete random variables:
Proposition Let X and be r.v.s with joint distribution PxY E ((X,)). Then, I(X; Y) if and only if Profxy do is the Dirac measure with a single atom O.
Proposition 6. If X and Y are discrete r.v.8 with Pxy E P(, then the PMI profile is discrete:
Profxy
Σxpx (PM)xY(E.) NEX
For more complex distributions, analytic formulae governing the PMI profiles are not known. Below we address this issue by offering a Monte Carlo approximation for a wide family of distributions.
2.2 Bend and Mix Models
As discussed above, PMI profiles are analytically known only for a small number of basic distributions and their reparametrizations, which do not change the PMT profile (Fig. 1). To generate a wider family of distributions with distinct PMI profiles, we introduce Bend and Mix Models which combine bending distribution (transforming with diffcomorphisms; Kobyzev et al. (2021) and Papamakarios et al. (2021)) and mizing (combining multiple models into mixture model). Although the mixing operation generally leads to distributions whose PMT profile is not available analytically, we can efficiently construct numerical approximations via Monte Carlo approaches. To ensure this, we require the following property:
strategies: also the literature normalizing flows, c.g.,
Definition 7 (informal). Every distribution Pxy E (((,))) for which we can efficiently sample (X,Y) ~ Pxy and numerically evaluate the densities (xY(x,)), (x(x)) and py (y) at every point (x. y) E X y is considered Bend and Mix Model (BMM).
Any distribution that satisfies this definition can used as a basic building block for a BMM. Examples include discrete distributions as well as multivariate normal and Student distributions. More complex distributions, such as mixture of discrete and continuous random variables (see discussion and example in Appendix B), can then be constructed using bending and mixing operations:
Proposition 8. If Pxy is a BMM and and g are diffeomorphisms with rumerically tractable Jacobians (e-g., normalizing flows), then P)())(Y)) is BMM.
Proposition 9. Consider BMMs Px......, PXXYK- If w WK are positive weights, such that w1 + Wk = 1, then the minture distribution Pxty: ww Px,,, +... wk PxxYx a BMM.
We prove both propositions in Appendix A.3. Note that since Gaussian mixture models are universal approximators of smooth densitics, and they belong to the BMM family, in principle one could use these constructions to approximate any distribution. In practice, the evaluation time of PMT grows linearly with
4

---

Published in Transactions on Machine Learning Research (01/2025)
the number of components and can become costly for high number of mixture components (or when large normalizing flow architectures are used).
Unfortumately, the MI of a mixture cannot be computed naively by a weighted sum of the information shared within the individual components. To illustrate this, we study two special cases in which constructing a mixture decreases, as well as increnses the overall MT compared to the weighted sum of the components:
Proposition 10. Consider r.v.8 (X&, Yk) such that 1(xx; Vk) < oc for h 1, 1,..., K. Let (X', Y") be their misture with weights W1...., WK. WK. Then,
K 0 DSI(XYY)) S I(xx) + log K.
Moreover, these inequalities are tight:
There exists a misture such that I(X;; Y') log K even though I(X2; Yx) 0 for all k. There erists a mixture such that I(X;; Y") 0 even though I(Xx; Yx) > 0 for all k.
More general bounds on MI in mixture models have also been studied by Haussler & Opper (1997) and Kolchinsky & Tracey (2017), but for completeness we include a. proof in Appendix A.4.
The properties of BMMs are chosen so that we can estimate the MI using Monte Carlo approaches up to an arbitrary accuracy, rather than relying on upper and lower bounds. We can sample T ~ Profxy by sampling a data point (xr y) from the joint distribution Pxy and evaluating f = PMIxy (z, y). Then, MI can be approximated with a Monte Carlo estimate of the integral I(X; Y) = E[T]. Assuming I(X;Y) < co, the Monte Carlo estimator of MI is guaranteed to be unbiased. However, the quality of the estimation depends the variance of the PMI profile and the number of Monte Carlo samples used. For example, if the variance is infinite, the Monte Carlo estimates (even though unbiased) will not be adequate due to enormous variance and several independent runs may be necessary to diagnose such problems. Additionally, the relative error of the Monte Carlo approximation can be large when the number of samples is not large enough, but the ground-truth MI is close to zero. For a more detailed discussion of Monte Carlo standard error (MCSE) under different regularity conditions we refer to Flegal et al. (2008) and Kochler et al. (2009).
A Analogously, we can estimate the PMT profile with a histogram: for bin B R one can introduce its indicator function 1 and integrate E[1r(T)] Its cumulative density function can be approximated with an empirical sample using the expectations E (oo,-ax (T') for given increasing sequence (az). Since the indicator functions are bounded between 0 and 1, the Monte Carlo estimator for both quantities is unbiased. For N samples, the standard crror is bounded above by 1/V4N, according to the incquality of Popoviciu (1935).
3
Case studies
In this section, we apply Bend and Mix Models to four distinct problems. Sec. 3.1 we demonstrale how they can be used extend existing benchmarks of MI estimators (with an explicit benchmark constructed in Appendix C.2). In Sec. 3.2 show how BMMs can be used to investigate the robustness of MI estimation to outliers and inliers. In Sec. 3.3 we apply BMMs to investigate the biases and sample efficiency of variational istimators employing neural critics. Finally, in Sec. 3.4 we show that if the distribution Pxy is known to be well-approximated by family of BMMs, we can provide Bayesian estimates of both MI and the PMI profile.
3.1 Novel distributions for estimator evaluation
To illustrate how BMMs can be used to create expressive benchmark tasks, we implemented a benchmark of 26 continuous distributions in TensorFlow Probability on JAX (Dillon et al., 2017; Bradbury et al., 2018) (Appendix C.2). Additionally, in Appendix B, we consider distributions involving discrete variables. Fig. 2 we visualise samples from four example distributions: the X distribution normal distributions. The marginal distributions Px and Py are normal, although the joint distribution Pxy not. The AT distribution is mixture of six bivariate normal distributions, illustrating how expressive
mixture of two bivariate
5

---

Published in Transactions on Machine Learning Research (01/2025)
X
AI:
Histogram
InfoNCE
R
0-
0.5
MINE
1
-1 0
1
X Al
Waves Galaxy
X1
X
Figure 2: Samples from the example distributions. Distributions X and AI represent ono-dimensional variables X and Y. Distributions Waves and Galaxy plot two-dimensional X variable using spatial coordinates, while oneddimensional Y variable is represented by color. The rightmost plot presents estimates according to different mutual information algorithms using independently generated data sets with = 5 000 points cach, compared to the ground-Iruth MI of the distribution (dotted line).
CCA
MI estimate 0.5
MI estimate
MI estimate
InfoNCE
KSG MINE
10
10'
Contamination level
Contamination level
Noise variance
Figure 3: Left: increasing the contamination level with inlier noise distribution. Middle: increasing the contamination level a with outlier noise distribution. Right: increasing the variance of the noisy normal distribution for constant contamination of 20%. Outliers have less impact than inliers.
BMMs can be. The two final distributions consist of a two-dimensional X variable and a ono-dimensional Y variable, allowing us to apply more complex transformations the X variable. The Waves distribution is mixture of twelve multivariate normal distributions with the X variable transformed, while the Galaxy distribution is mixture of two multivariate normal distributions with the X variable transformed by the spiral diffeomorphism (Czyž et al., 2023). A detailed description of these distributions can be found in Appendix C.1.
For each BMM we sampled ten data sets with N = 5000 points and applied five estimators: the histogram- based estimator (Cellucci al., 2005; Darbellay & Vajda, 1999), the popular KSG estimator (Kraskov et al., 2004), canonical correlation analysis (CCA; Kay (1992); Brillinger (2004)), and two neural estimators: InfoNCE (Oord et al., 2018) and MINE (Belghazi et al., 2018) (see Appendix C.3 for hyperparameters used). The estimates are shown in Fig. 2.
Even though problems in Fig. 2 are low-dimensional and do not encode more information than 1.5 nats, they pose a considcrable challenge for the estimators. The KSG estimator, which performs well in low-dimensional lasks, gave the best estimate all lasks. However, the Waves lask was not solved by any estimator. The CCA ostimator, excelling at distributions that are close to multivariate normal (Czyž al., 2023), is not able to capture any information at all. This suggests that BMMs can provide rich set of distributions that can be used to test MI estimators. In Appendix C.2 evaluate KSG, CCA and four neural estimators the compiled list of 26 proposed benchmark tasks. Particularly interesting are the tasks including inliers, which we study in the next section.
3.2 Modeling inliers and outliers
In this section, we use BMMs to study the effect of inliers and outliers on MI estimation. Consider an electric circuit or biological system modeled a communication channel Prix (y) xx). The rescarcher controls the input variable X, which results in a distribution Px. and subsequently measures the outcome variable Y. The mutual information I(X;Y) is then estimated from the experimental samples (Nalgez-Jawecki et al., 2023).

---

Published in Transactions on Machine Learning Research (01/2025)
However, every experimental system can suffer from occasional failures. We model the output of failing system with a noise distribution with a PDF n(y). If the probability of system failure, denoted by a, and the distribution of noise n(y) do not depend on the imput value 2, the communication channel becomes a mixture p (y x) = (1 - a)prix (y m) + an(y). By multiplying both sides by the density px (x), the distribution of the channel inputs provided by the scientist, we arrive at the mixture distribution Pxy with PDF (xy y) = (1 - a)pxx (x. y) + en(y)px(x)) If the system failure is unnoticed, one can only measure Y", rather than Y. It is therefore of interest to understand how much I(X;Y) and I(X; can differ under realistic assumptions on the noise n(y) and whether the standard MI estimation techniques are robust to it. In Appendix A.4 we prove the following upper bound:
Proposition 11. Let a E [0,1] be a porameter. Consider variables X, Y and Y', s.t. (x (1 ()x)) + anly)px(x)) Then, I(X;Y') S (1 (1-a)
Alternatively, we can model the system as a BMM and evaluate this quantity using the Monte Carlo approximation. For example, consider a setting with a two-dimensional input variable X and two-dimensional output variables Y (perfect outp and Y' (contaminated output). As the joint density pxy we use a multivariate normal with unit scale and correlations corr(X,, Y1) corr(X2,, we use a multivariate normal distribution with covariance o² L2. If x² = I this results in inliers, where the noise distribution is hard to distinguish from the signal. For 0² << 1 the system failures are all close to O, while outliers are present for o² >>>>> 1.
0.8 and for the noise nyy)
In Fig. we present the results of three experiments: in the first two, we changed the contamination level a E [0,) for o² = 1 (inlier noise) and o² = 52 (outlier noise) respectively. In the third experiment, we fixed a = 0.2 and varied a² 12-7,28] We see that the inlier noise results in a slightly faster decrease of mutual information, while the outlier noise decreases almost linearly following the upper bound (1 ()) Y). Interestingly, in this low- -ww-dimensional setting, the KSG, MINE, and InfoNCE estimators reliably estimate the mutual information I(x; Y'), which can signifieantly differ from I(; Although CCA would be the preferred method to estimate I(X;Y) Y) without any noise in this lincar setting (Czyž et al., 2023), even a small nunber of outliers (a = 5%) can result in unreliable estimates.
3.3 Variational estimators and the PMI profile
Variational estimators of mutual information are frequently used in self-supervised learning (Oord et al., 2018) and optimize critic function f: X' x ) -> R to obtain approximate lower bound on mutual information. For example, Belghazi et al. (2018) use the Donsker Varadhan loss, IDv(J)= = Epxy [5] log Exx [exp /l, which is a lower bound on Y( for any bounded function f. This lower bound becomes tight when f f= PMIxY + e where e is any real number, i.e., (fv = YY) Hence, one can approach MI estimation by optimizing JDY over a floxible family of functions f. usually parameterized by a ncural network. Other examples include the NWJ estimator (Nguyen el al., 2007), which becomes light for ʃ = PMIxy + 1, and InfoNCE (Oord et al., 2018), which has functional degree of freedom, i.e., f(z, y) = PMIxy (2, + c(x) for function c. We provide more details these methods in Appendix C.4. Typically, these losses are interpreted approximate lower bounds on MI and connect f to the PMI function (Poole et al., 2019; Song & Ermon, 2020) in the infinite sample limit. In practice, however, only finite sample is available and the critic f is modeled via some parametric family (e.g, a neural network), which has to be learned from the data available. Since the ideal critic function of a neural estimator is an approximation of PMI, we investigate how well the PMT function can be learned by the previously described estimators, as well as investigate how they behave under misspccification.
Critics and the PMI function We simulated N = 5 000 data points from mixture of four bivariate normal distributions (Fig. 4) with I(X;Y) 0.36 and fitted the neural critics (see Appendix C) to half of the data, retaining the latter half as the test set, on which the final estimates were obtained, yielding INWJ = 0.33, JDv = 0.32, INCE = 0.35. This (minor) difference can be attributed to the learned critic not matching the true PMI (up to the required constants), estimation error due to evaluation on a finite batch, estimator
or both. In Fig. 4 we plot the PMI function and the optimized critics. As the estimates PMI only up to an additive constant, we normalized the critic and the PMI plots to have zero mean. A Analogously, we removed the functional degree of freedom <(x) from the InfoNCE estimator by removing
7

---

Published in Transactions on Machine Learning Research (01/2023)
PDF
Seg
for moan
PH red
4-
PAe
RM
protle
-2
-4
Figure 4: Left: PDF of the considered distribution. Middle: neural critic and PMI values. Right: normalized
neural critic and PMI profiles.
Unblased critic
Constant blas
Functional blas
Unblased critic, dimensions
1.0
1.0
1.0
Estimator
Estmate
0.6
0.8
Extimure
0,8
0.6
c.8
0.6
Extimade
20
InfoNCE
DV
0.4
10-
NNN
0.2
0.2
0.2
0.0
0.0
0.0
1.6
256
1024
4956
16
256
1924
4096
16
256
1924
4096
1.6
64
256
1924
4096
Number points
Number points
Number points
Number points
Figure Estimation of mutual information using a function approximating PMT as a function of sample size for Monte Carlo (MC), InfoNCE, Donsker-Varadhan and NWJ losses, From left right: true PMI function from Fig. 4 is used, a constant bias is added, a functional bias is added. The rightmost plot: true PMI function for different, high-dimensional problem is used.
the mean caleulated along the y dimension. Overall we saw a mismatch, suggesting that neural crities do not capture the PMI function of the distribution well. However, we can compare the PMI profile with the histogram of the values predicted by the critic in Fig. 4 (shifting the PMI profile and the histogram to have mean 0 for the Varaahnna estimator; for InfoNCE it is not possible to compare the PMI profile with the critic values due to the functional degrec of freedom c(x)). For both NWJ and DV, we see little discrepancy between the (shifted) PMI profile and learned values. This suggesls that although the newral critics may not learn the PMI function properly in regions with low density, the PMI profile (and, hence, the MT) can still be approrimated well.
Robustness to misspecification of the critic the next experiment (Fig. 5), we evaluated IDV, INWJ, InfONCE, and the simple Monte Carlo estimator (MC) for increasing sample size N where we provided them with "oracle" critic f, which we varied as follows: f(x v) - PMIxy y) E {0, cin())))()))). Using a perfect critic, f PMIxY, we that all estimators performed well in this problem. When a constant bias was added, = PM + c, the Monte Carlo and NWJ estimators became biased. Finally, InfoNCE was the only unbiased estimator when a functional degree of freedom was added, (2. = PMIxy (x, v) + sin(r"), confirming the known theoretical limitations of the estimators.
Increasing the Dimensionality In low dimensions variational approximations perform comparably to the Monte Carlo estimator when the correct critic function is used, and are more robust misspecification due to additional degrees of freedom. However, this additional robustness results in bias in higher dimensions (Fig. 5, right panel). We simulated a data set using multivariate normal distribution with 25 strongly interacting components, corr(X,, Y;) 0.8, which results in ():Y)= = 12.8. Although the Monte Carlo estimator and NWJ estimator (which are not robust to additional degrees of freedom) provided estimates close to the ground-trute truth (with Monte Carlo slightly outperforming NWJ both bias and variance), Donsker- Varadhan estimator resulted in a strong positive bias for small batch sizes and InfoNCE had a negative bias with very low variance which has been observed in previous studies and has been related the Ollo N) behavior of this approximate lower bound (Oord et al., 2018; Song & Ermon, 2020).
8

---

Published in Transactions on Machine Learning Research (01/2025)
Inliers (5-dim, 0.2)
Concentric (5-dim, 5)
Normal (5-dim, sparse)
InfoNCE
KSG
46
BMM
0.55
0.5
0.7
1.0
1.1
Figure 6: BMMs provide uncertainly estimales in the form of credible intervals and, when domain expertise is available, can be more accurate than generic black-box alternatives, We plot the posterior mean and an interval created using 10th and 90th percentile of the posterior distribution. Ground truth represented with a dashed line.
Ground-truth sample
Simulated sample
Posterior MI
Posterior PMI profile
1
A
A
1
x
0
0
-1
0
1
0.7
0.8
0.9
0.0
2.5
X
X
MI
PMI
Ground-truth sample
Simulated sample
Posterior MI
Posterior PMI profile
0
-1
-1
1
-1
0.3
0.4
0.5
-1
1
X1
X1
MI
PMI
Figure Top row: data generated according to the AI distribution, data generated according a single MCMC sample, posterior distribution of the mutual information (red line denotes posterior mean and black line denotes the ground-truth value), and the posterior distribution of the PMI profile (black curve denotes the ground-truth value). For well-specified model the posterior reflects epistemic incertainty well. Bottom row: experiment with a misspecified model fitted to the Galaxy distribution. Misspecification results in biased inferences (two right-most panels) with miscalibrated posterior, although can be diagnosed by comparing true data with the data generated from the model.
3.4 Masel-based mutual information estimation
As the final application of BMMs, we investigate the problem of Bayesian model-based MI estimation.
The general idca Consider family of distributions (Po A E θ}, such that for every value of the parameter vector 0, the mutual information contained in Pe is known. That is, for (Xe, Yo) Pe can compute I(Xe; Yo), which denote by I(Pe). If the observed data are modeled as i.i.d. r.v.s (X,, Y,) Pe, one estimate 0 and use I(Pp) as an estimate of the mutual information. For example, Brillinger (2004) interprets the CCA-based MI estimator of Kay (1992) as the plug-in estimator I(Pa), where Ô is the maximum likelihood estimate (MLE) in the model in which Po is multivariate normal distribution. For discrete data. it is well-known the plug-in estimator is biased (Goebel et al., 2005; Suzuki, 2016; Bu et al., 2018; Marx & Vrecken, 2019) and one particular way of regularizing the estimate is to approximate the Bayesian posterior over the MI (Hutter, 2001). We note that one can construct a Bayesian alternative for the CCA-basee estimator by using a prior on the covariance matrix (Lewandowski et al., 2009) and Markov chain Monte Carlo (Gelman et al., 2013, Ch. 11) to provide samples f,...., ds from the posterior P(0) X,,Y,.... sample-based approximation to the posterior mutual information, I(Po.) ) .,..., I(Poar).
Yi...., .......p, Xx. Yv) and construct.

---

Published in Transactions on Machine Learning Research (01/2025)
More generally, consider statistical model {Po 0 E @) such that all Po are BMMs, and prior P(0). One can then apply Bayesian inference algorithms to construct sample . Oxs from the posterior. Although the exact values for I(Po,), ((le..)...,,((PP...
are not available, they can be approximated as in Sec. 2. Hence, we can construct an approximate posterior distribution and quantify epistemic uncertainty of the estimate (Fig. 7). Since BMMs include both mixture models and normalizing flows, this approach could, in principle, be used as a general technique for building model-based mutual information estimators, where the generative model can be constructed using domain knowledge. Moreover, this is the first Bayesian estimator of the PMI profile: all distributions Po.. are BMMs, one can construct M histograms (or CDFs) approximating the profile.
Proof of concept To illustrate this approach we implemented a spanse Gaussian mixture model (see Appendix C.5) in NumPyro (Phan et al., 2019) and used the NUTS sampler (Hoffman & Gelman, 2014) to obtain the Bayesian posterior for selected low-dimensional problems from the proposed benchmark (see Table 2 in Appendix C.5.2). We visualise the predictions for four distributions in Fig. 6: we stress that other methods can only provide a single point estimate (for each given data set), while BMMs can provide uncertainty quantification in the form of posterior distribution, visualised by credible intervals. Interestingly, even for low-dimensional problems where KSG has been considered state-of-the-ae (Czyž et al., 2023), the BMM-based estimator provided better estimates.
On the role of assumptions All parametric statistical methods rely on assumptions (Gelman et al., 2020, Sec. 2). In particular, Bayesian inference can result in unreliable inferences when the model is misspecified, i.e. the true -ata-ggnn process Pxy does not belong to the assumed family Pe (Watson & Holmes, 2016). To illustrate the role of the assumptions, we obtained Bayesian posterior conditioned on 500 data points from the Al and Galaxy distributions (see Fig. 7). We see that the for the Al distribution, the posterior is concentrated around the ground-truth mutual information value and the ground-truth PMI profile is well-approximated by the posterior samples.
The misspecification in the Galaxy distribution can be immediately diagnosed via posterior predictive checking (Gelman et al., 2013, Ch. 6): Fig. we see that a data sample simulated from the model looks substantially different from the observed data, meaning that the model did not capture the distribution well (cf. Appendix C.5). This provides clear indication that the estimates should not be trusted: most of the probability mass of the Bayesian posterior is far from the ground-truth mutual information. Similarly, techniques such
the posterior on the PMI profile is biased. We therefore recommend using model-checking as posterior predictive checks and discriminator-bot based validation (Sankaran & Holmes, 2023, Sec. 4) to understand the deficiencies of the model. To mitigate the risk of overfitting, which can bias the model (see Appendix C.5), we recommend validaaiio (Pirronen & Vehtari, 2017).
In summary, although BMMs are not generic estimators applicable to problems without available domain knowledge, in some situations, principled Bayesian modeling can be beneficial to obtain better estimates along with uncertainty quantification.
4 Discussion
In this article, we have studied pointwise mutual information profiles (PMI profiles), determining them analytically for multivariate normal distributions (Theorem 4), and proposed the family of Bend and Mix Models (BMMs), which include multivariate normal and Student distributions, mixture models and normalizing flows, for which the PMI profile can be approximated using Monte Carlo methods. We showed how BMMs can be used to provide novel benchmark tasks to test MI estimators and calculate MI transmitted through communication channel in the presence of inliers and outliers (which can be used in the experimental design in electrical and biological sciences). BMMs allowed us to study how well neural critics approximate PMI profiles. Finally, we showed how Bayesian estimates of MI between continuous r.v.8 can be performed using BMMs; additionally, the proposed method estimates the PMI profiles. Although this approach is not universal, we find it suitable for problems with precise domain knowledge available (which can be used to construct the generative distribution Po and provide the prior P(0)) and in which uncertainty quantification is desired.
10

---

Published in Transactions on Machine Learning Research (01/2025)
Limitations and further research Generalized PMI profiles (see Appendix A.5) allow one to provide wide family of invariants, which can be related to f-divergences (Nowozin et al., 2016). We leave the exploration of potential ramifications of this generalization to future work.
Although in Sec. 3.4 we propose BMMs as a basis for model-based Bayesian inference of mutual information, model misspecification (Watson & Holmes, 2016) as well as underfitting and overfitting can bias inference. We therefore recommend, as usual in Bayesian statistics, to validate the model (Gelman al., 2020). Overfitting and model misspeciffcation may be more difficult to detect especially in high-dimensional situations or when more expressive models are constructed, for example using normalizing flows, as Bayesian inference for models involving neural networks is known to be challenging (Izmailov et al., 2021). We hypothesise that tempering the likelihood (Grünwald & van Ommen, 2017) or employing alternative inference schemes (Lyddon et al., 2018) may improve robustness to model misspecification, although we leave this topic to future work.
In Sec. 3.2, we study channel in which the probability of failure a is constant and independent of the input variable X. However, in both biological and electric systems this assumption does not necessarily hold: for large values of X the system may result in outlier more easily. Unfortunately, cases in which depends on X are not easily modeled by BMMs. Moreover, BMMs do not allow modeling additive noise. Although adding Gaussian noise can be viewed as a continuous mixture, marginalizing the latent variable to obtain an analytic form of logpy (y) is not possible. Employing unbiased estimators, such as SUMO (Tuo et al., 2020), may hence become an interesting direction for future research.
More generally, replacing analytical formulae with accurate numerical estimates has the potential to extend the family of distributions with tractable MT beyond BMMs. In Sec. 2.2 we construct BMMs in bottom-up fashion, where simple distributions are transformed and mixed together to obtain more expressive distributions. The introduced operations allow us to evaluate the probability densities and directly sample from the joint distribution, resulting in a Monte Carlo estimator for mutual information.
A Alternatively, one could imagine a top-down approach, where for a given distribution Pxy one derives a family of unbiased log-densily estimators (Luo et al., 2020). Namelyy let Lxy (2, y) be a family of estimators indexed by (z.y) E X' >,, such that E[ (z, y)] = log pxy (I, 3) for all points (a, y) in the support of Pxy. Similarly, let (x) and (y) be families of estimators such that E[[x (ar) = log Px (x) and E[Ly(v) (v) log by (v). We can then construct an unbiased estimator of the PMI function, B[Lxy (zz, y) - (x) and aim to approximate the PMT profile and the ground-truth MT via nested Monte Carlo sampling (Rainforth et al., 2018). However, the feasability of this approach relies on construction of the required families Lxy, Lx and Ly, as well as on cnsuring that the variance of the resulting estimate is sufficiently small.
(v)] = PMIxy (z, v),
Similarly, Monte Carlo integration of the PMT function with respect to the PXY measure can, in principle, be replaced by other integration methods, such as Markov chain Monte Carlo (Brooks et al., 2011) or sequential Monte Carlo samplers (Del Moral et al., 2006). These integration methods could allow one to integrate the PMIxy function for distributions in which the log-density functions log logpx and logpy py (or their unbiased estimates) are available, but Monte Carlo sampling from Pxy is not possible. However, this approach relies on ensuring whether the employed integration method is accurate enough, e.g., by controlling the convergence diagnostics and the effective sample size (Vehtari et al., 2021; Vats & Knudson, 2021).
We hope that our analysis of pointwise mutual information profiles provides a fresh perspective on mutual information estimators, and that BMMs will prove to be useful tool for benchmark design and will renew interest in Bayesian approaches to mutual information estimation.
Reproducibility To facilitate reproducibility, the experiments have been implemented as Snakemake workflows (Mölder et al., 2021). The code available at https://github. com/cbg-eteee bhog--eth//but can be used to reproduce all experimental results and figures.
Acknowledgments FG was supported by the OPUS 18 grant operated by Narodowe Centrum Nauki (National Science Centre, Poland) 2019/35/0 //NZ2/ 003998. PC was supported by a doctoral fellowship from the ETH AI Center. AM was supported by a postdoctoral fellowship from the ETH AI Center and a postdoctoral position at the Computational Biology Group. We would like to thank Pawel Nalgez-Jawecki, Julia Kostin and anonymous reviewers for helpful suggestions on the manuscript.
11

---

Published in Transactions on Machine Learning Research (01/2025)
References
Carl Allen and Timothy Hospedales. Analogies explained: Towards understanding word embeddings. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 223 231. PML.R, 09-15 .ml. mlr htt.://ss..rrrresreeeeeeeeeeeeee https//99/aaeeeee// html. 3
Jun 2019. URL
Mohamed Ishmacl Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International conference on machine learning, pp. 531-540. PMLR, 2018. 2, 3, 6, 7, 34
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Mille, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
com/ google/ .
David R. Brillinger. Some data analyses using mutual information. Brazilian Journal of Probability and Statistics, 18(2):163 182, 2004. ISSN 01030752, 23176199. URL http://wwx jstor org/sttable/4360047, 2, 6, 9, 28, 34
S. Brooks, A. Gelman, G. Jones, and X.T. Meng. Handbook of Markow Chain Monte Carlo. Chapman & Hall/CRC Handbooks of Modern Statistical Methods. CRC Press, 2011. ISBN 9781420079425. URL https://books, http://booooo.oogl..e.ee
11
Yuheng Bu, Shaofeng Zou, Yingbin Liang, and Venugopal V Veeravalli. Estimation of KL divergence: Optimal
minimax rate. IEEE Transactions on Information Theory, 64(4):2648-2674, 2018. 9
Nick Carrara and Jesse Ernst. Using Monte Carlo tree scarch to calculate mutual information in high dimensions, 2023. 2, 28
Christopher J Cellueci, Alfonso M Albano, and Paul E Rapp. Statistical validation of mutual information calculations: Comparison of alternative numerical algo Physical review E, (166):066208, 2005. 6, 34
Pawel Czyž, Frederic Grabowski, Julia Vogt, Niko Beerenwinkel, and Alexander Marx. Beyond normal: On the evaluation of mutual information estimators. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances Neural Information Processing Systems, volume 36, pp. 16957-16990. Curran Associates, Inc., 2023. 2, 3, 6, 7, 10, 28, 31, 32, 33, 34
Georges A Darbellay and Igor Vajda. Estimation of the information by an adaptive partitioning of the observation space. IEEE Transactions on Information Theory, 45(4):1315 1321, 1999. 6, 34
Pierre Del Moral, Arnaud Doucet, and Ajay Jasra. Sequential Monte Carlo samplers. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 68(3):411 436, 2006. doi: https://doi.org/ wwwp:/c.oo/wooe//eaeeeeeaoaaaeee httm//oi/bb//0.11111 1t.1111/j.1467-11 986677779888880 2006.00553.x. URL attps://rsst www.eemlnlnnnennnnrrreeeeee httpiibbrrrrrl.lllll......ee 1467-9868. 2006. 00553..005..... x. 11
Joshua V. Dillon, lan Langmore, Dustin Tran, Eugene Brevdo, Srinivas Vasudevan, Dave Moore, Brian Patton, Alex Alemi, Matthew D. Hoffman, and Rif A. Saurous. Tensorflow distributions. CoRR, abs/1711.10604, 2017. URL. http:://rxii http:/awwivwraaaaaaaaa/a htrggabs//711... 10604. 5
James M. Flegal, Murali Haran, and Galin L. Jones. Markov Chain Monte Carlo: Can we trust the third significant figure? Statistical Science, 23(2):250 260, 2008. doi: 10.1217/08-STS257 URL
https://doi.org: 1214/08 STS257. 5
Sylvia Srühwirth-Schnater and Gertraud Malsiner- Walli. From here to infinity: sparse finite versus Dirichlet process mixtures in model-bases clustoring. Advances Data Analysis and Classification, 13(1):33- 64, Mar 2019. ISSN 1862-5355. doi: 1011077/11161--08-02299.
URL
s1 811664-018-0329- 35
12

---

Published in Transactions on Machine Learning Research (01/2025)
Kurt G. Radon-Nikodym derivative and push forward of measures. Mathematics Stack Exchange, 2022.
URL https://math. stackexchange.com /441/000 ftt.atts./mmmdl.ssceecccag.ggg (version: 2024-05-14). 27
Weihao Gao, Sreeram Kannan, Sewoong Oh, and Pramod Viswanath. Estimating mutual information for discrete-contintuus mixtures. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. 28, 29
A. Gelman, J.B. Carlin, H.S. Stern, D.B. Dunson, A. Vehtari, and D.B. Rubin. Bayesian Data Analysis, Third Edition. Chapman & Hall/CRC Texts in Statistical Science. Taylor & Francis, 2013. ISBN 9781439840955.
URL attps://bookks soooks.googa google. 21/bo0ks710-2XXL6AAAAAEE
9, 10
Andrew Gelman, Aki Vehtari, Daniel Simpson, Charles C. Margossian, Bob Carpenter, Yuling Yao, Lauren Kennedy, Jonah Gabry, Paul-Christian Bürkner, and Martin Modrák. Bayesian workflow, 2020. 10, 11
B. Goebel, Z. Dawy, J. Hagenauer, and J.C. Mueller. An approximation to the distribution of finite sample size mutual information estimates. In IEEE International Conference on Communications, 2005. ICC 2005. 2005, volume 2, pp. 1102-1106 Vol. 2, 2005.
Frederic Grabowski, Pawel Czyž, Marek Kochańczyk, and Tomasz Lipniacki. Limits the rate of information transmission through the MAPK pathway. Journal of The Royal Society Interface, 16(152):201 807992 2019. doi: 10.1098/ 101109999f..00111 /tt/:/sf..11.000.777 07992. URL http:://royalllociettpbbiii www.i.soooooooo/aaooobobo. wttpaaaa......././/...// 1098/rsif. 2018. 0792. 2, 28
Peter Grünwald and Thijs van Ommen. Inconsistency of Bayesian Inference for Misspecified Linear Models, and a Proposal for Repairing Tt. Bayesian Analysis, 12(4))1069 1103, 2017. doi: 10.1214/17-AA005.
URL https://dol: http:///oo.oo/////////// 10. 1214/17-BA1082
David Haussler and Manfred Opper. Mutual information, metric entropy and cumulative relative entropy risk. The Annals of Slalistics, 25(6):2451 2492, 1997. doi: (0.2214/a0s10037410033 URL https:
vidoi.org// 10. 0214//0ss/030/// aos/1030741081. 5, 25
Matthew D. Hoffman and Andrew Gelman. The No-U-Turn Sampler: Adaptively setting path lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research, 15(47):1593-162) 2014. URL
http://jmlr... wrg.eoogggaeoooe wwweerr/vv5//offmaaa// html. 10, 35
Harold Hotelling. Relations between two sets of variates. Biometrika, 28(3-4):321 377, 12 1936. ISSN 3
0006-3444. doi: 10.1093 http////oome///.---42/ URL https://
1093
Marcus Hutter. Distribution mutual information. In T. Dietterich, S. Becker, and Z. Ghahramani (eds.), Advances Newral Information Processing Systems, volume 14. MIT Press, 2001. 9, 28
J. P. Imhof. Computing the distribution of quadratic forms in normal variables. Biometrika, 48(3/4):419-426,
1961. ISSN 00063444. URL http://wwa http::///www...rrrr//o/// jstor .rg/ wwtpsttrllea/..//ooooo./o Stttable/3332760 21
Pavel Izmailov, Sharad Vikram, Maithew D Hoffman, and Andrew Gordon Wilson. What are Bayesian neural network posteriors really like? In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 4620 4640. PMLR, 18 24 Jul 2021. URL sttpp::/prooeeednng: mmr. wttppp/eee//eepppeeepe
html.. 11
Takoua Jendoubi and Korbinian Strimmer. A whitening approach to probabilistic canonical correlation analysis for omics data integration. BMC' Bioinformatics, 20(1):15, Jan 2019. ISSN 1471-2105. doi: 10. 10.1186//12833--08--57222 URL https:///oi.ooo 10.
20
J. Kay. Feature discovery under contextual supervision using mutual information. In IJCNN International Joint Conference on Neural Networks, volume 4, pp. 79 84, 1992. doi: 11L10//JJCN.11992..22202 2, 6,
9, 33, 34
13

---

Published in Transactions on Machine Learning Research (01/2025)
Shiraj Khan, Sharba Bandyopadhyay, Auroop R. Ganguly, Sunil Saigal, David Erickson III, Vladimir Protopopescu, and George Ostrouchov. Relative performance of mutual information estimation methods for quantifying the dependence among short and noisy data. Physical Review E, 76(2):026209, 2007. 2
I. Kobyzev, S. D. Prince, and M. A. Brubaker. Normalizing flows: An introduction and review of current. methods. IEEE Transactions on Pattern Analysis & Machine Intelligence, 43(11):3964 3979, nov 2021. 4
ISSN 1939-3538
doi: 10.1109 PAA.I..220.299920
E. Kochler, E. Brown, and S.J.-P.A. Haneuse. On the assessment of Monte Carlo error in simulation-based statistical analyses. The American Statistician, 63(2))155 162, 2009. doi: 10.1198/tast.2009.o 5
Artemy Kolchinsky and Brendan D. Traccy. Estimating mixture entropy with pairwise distances. Entropy, 19
(7), 2017. ISSN 1099-4300. doi: 10.3390/e e19000361. URL
5, 25
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical Review E, 69(6))666138, 2004. 2, 3, 6, 18, 33, 34
J.M. Lec. Introduction to Smooth Manifolds. Graduate Texts in Mathematics. Springer, 2nd edition, 2012. ISBN 9781441999825. URL https://doi.co org/ 1007 108100/7--1-41199990888111110111
18, 29
Daniel Lewandowski, Dorota Kurowicka, and Harry Joc. Generating random correlation matrices based on vines and extended onion method. Journal of Multivariate Analysis, 100(9):1989-2000 2009. ISSN hwwp:/0..........0......... .00000 URL http::://www..cineeeiee..... science/ /tt/an///rrtccee///////077700000
0047-259X. doi:
9, 35
Yucen Luo, Alex Beatson, Mohammad Noronzi, Jun Zhu, David Duvenaud, Ryan P. Adams, and Ricky T. Q. Chen. SUMO: unbiased estimation of log marginal probability for latent variable models. In 8th International Conference on Leorning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.ng net, 2020. URL https:
11
Simon Lyddon, Stephen Walker, and Chris C Holmes. Nonparametric learning from Bayesian models with randomized objective functions. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianche and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 31. Curran Associates, Inc., 2018. 11
Alexander Marx and Jilles Vreeken. Testing conditional independence on discrete data using stochastic complexity. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 496-505 PMLR, 2019. 9
Alexander Marx, Lincen Yang, and Matthijs van Leeuwen. Estimating conditional mutual information for discrete-contimnous mixtures using multi-dimensional adaptive histograms. In Proceedings of the STAM International Conference on Data Mining (SDM), pp. 387 395, 2021. 29
David McAllester and Karl Stratos. Formal limitations on the measurement. of mutual information. In Silvia Chiappa and Roberto Calandra (eds.), Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statístics, volume 108 Proceedings of Machine Learning Research, pp. 875-884.
PMLR, 26 28 Aug 2020. URL https: /ttp::///proceee http:///pccccc....///////o./..se .ml.mmm....... ..m.ml..rreeee wtt.:peee//100mmmaleeeettee
html. 2, 34
Kevin P. Murphy. Probabilistic Machine Learning: Advanced Topics. MIT Press, 2023. URL http: //probml. github. io/book2. 30
F Mölder, KP Jablonski, B Letcher, MB Hall, CH Tomkins-Tinch, V Sochat, J Forster, Lee, SO Twardziok, A Kanitz, A Wilm, M Holtgrewe, Rahmann, S Nahnsen, and J Köster. Sustainable data analysis with Snakemake. F1000Research. 10(33), 2021. doi: 10. 12268//100000scarch22003333 1902.. 11
Pawel Nalgcz-Jawecki, Paolo Armando Gagliardi, Marek Kochaúczyk, Coralie Dessauges, Olivier Pertz, and Tomasz Lipniacki. The MAPK /ERK channel capacity excoods 6 bit/hour. PLOS Computational Biology, 1915):1-21, 05 2023. doi: 0.1371/journal.phh 1011155. URL https: / //oii orciioorg/.... j0TT//journa..pcb. 1011155. 6
14

---

Published in Transactions on Machine Learning Research (01/2025)
XuanLong Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by penalized convex risk minimization. In J. Platt, D. Koller, Y. Singer, and S. Roweis (eds.), Advances in Neural Information Processing Systems, volume 20. Curan Associates, Inc., 2007. 2, 3,
7, 34
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-GAN: Training generative neural sam- plers using variational divergence minimization. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL https://proceeding aedebbee7755000eeee1100177747 Paper.
neurips httpps..ccpppp..ppppppppp filess/ppppee/200///////////// Papeeee--pdpppppppp...... pdf. 11, 28
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1xXxx:8807.0395 2018. 2, 3, 6, 8, 34
George Papamakarios, Eric Nalismick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshmi- narayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57):1 64, 2021. URL http://jjal..rrrpppprreee/// prggppppers/22/11-1000 html. 4
K. B. Petersen and M. S. Pedersen. The matrix cookbook, November 2012. URL http://www2. compute. dtu. hwtppp/ppbss///4--//Paaa
html. Version 20121115. 20
Du Phan, Neeraj Pradhan, and Martin Jankowiak. Composable effects for flexible and accelerated probabilistic programming in NumPyro. arXie preprint arXin: 1155,1154, 2019. 10, 35
Juho Pirronen and Aki Vehtari. Comparison of Bayesian predictive methods for model selection. Statistics and Computing, 27(3):711-735, May 2017. ISSN 1573-1375 doi: 10
URL
https://doi 1ttp:////0000//11111100----00
M.S. Pinsker and A. Feinstein. Information and Information Stability of Random Variables and Processes.
Holden-Day series in time series analysis. Holden-Day, 1964. TSBN 9780816268047. 3
Dimitris N Politis. On the entropy of a misture distribution. Purdue University. Department of Statistics, 1991. 29
Y. Polyanskiy and Y. Wu. Information Theory: From Coding Learning. Cambridge University Press, 2022. Book draft. 2, 26, 28, 34
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pp. 5171-5180. PMLR, 2019. 2, 7,
34
Tiberiu Popoviciu. Sur les équations algébriques ayant toutes leurs racines réelles. Mathematica (Cluj), 9: 129 145, 1935. 5
Tom Rainforth, Rob Cornish, Hongseok Yang, Andrew Warrington, and Frank Wood. On nesting monte carlo estimators. In International Conference on Machine Learning, pp. 4267 4276. PMLR, 2018. 11
Kris Sankaran and Susan P. Holmes. Generative models: An interdisciplinaty perspective. Annual Review of Statislics and Its Application, 10(1):325-315 2023. doi: 10.1146/ /ata:/aesstaaassiiss-3-11------- 10
URL
https://doi.oom org. 10.g/10..00 1111/11nnnrreeee
Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estimators.
In International Conference on Learning Representations, 2020. 7, 8, 34
Joe Suzuki. An estimator of mutual information and its application to independence testing. Entropy, 18(4): 109, 2016. 9
Dootika Vats and Christina Kmudson. Revisiting the Gelman-Rubin diagnostic. Statistical Science, 36(4):518 529, 2021. doi: 10. 10.1214/00-TSS812 URL https:://bi..rrg//// 1tt.://00..11422-/051111 11
15

---

Published in Transactions on Machine Learning Research (01/2025)
Aki Vehtari, Andrew Gelman, Daniel Simpson, Bob Carpenter, and Paul-Christian Bürkner. Rank- Normalization, Folding, and Localization: An Improved R for Assessing Convergence of MCMC (with Discussion). Bayesian Analysis, 16(2):667 718, 2021. doi: 10.1214/20-BA1221. URL https: //doi. 0tee//...111//000
11, 36
James Watson and Chris Holmes. Approximate Models and Robust Decisions. Statistical Science, 31(4):465 489, 2016. doi: 10.1214/16-STSESS URL https://doi. wtwge/r..o..../ 1110///..221411-ST55533 10, 11
16

---

Published in Transactions on Machine Learning Research (01/2025)
Appendix
Table of Contents
A Technical results
18
A.1 Proof of the invariance of the pointwise mutual information profile
18
A.2 Derivations of pointwise mutual information profiles
19
A.3 Constructing new Bend and Mix Models
23
A.4 Mutual information in finite mixtures
24
A.5 Divergence profiles
26
B Distributions involving discrete variables
28
B.1 Known distributions
28
B.2 Novel distributions
30
C Experimental protocols and additional experiments
30
C.1 Specification of the proposed distributions
30
C.2 A benchmark exlension proposal
32
C.3 Estimator hyperparameters
................................
34
C.4 Variational estimators of mutual information
34
C.5 Bayesian estimation of Gaussian mixture models
35
17

---

Published in Transactions on Machine Learning Research (01/2025)
A Technical results
In this section we formalize the results described in Sec. 2. Sec. A.1 we precisely define the family of P(X,)) ))) distributions, which then used to prove that the PMT profile is invariant to diffeomorphisms. Tn Sec. A.2 we derive the PMI profiles in the cases where they are analytically tractable. Then, in Sec. A.3 we formalize the properties of Bend and Mix Modcls. In Sec. A.4 we derive the bounds on mutual information in finite mixture models. Finally, in Sec. A.5 we provide measure-theoredic analysis of the PMT profile, which offers more general invariance results, though it requires more complex mathematical machinery.
A.1 Proof of the invariance of the pointwise mutual information profile
In this section, we consider random variables valued smooth manifolds without boundary (Lee, 2012, Ch. 1) equipped with their Borel o-algebras and given reference measures. In particular, the results in Section 2 apply to both discrete and continuous r.v.s, as the considered manifolds include open subsets of the Euclidean space Rn with the Lebesgue measure as well as zero-dimensional discrete spaces {1...., m} with the counting measure.
For a given pair of smooth manifolds X' and y, we equip their product X' x y with the product measure and define the set P(X, to consist of all probability measures Pxx on X y such that the joint measure Pxy. as well as the marginal measures (x Pxr x v) and (B Pxy(A (X' x B), have smooth and positive PDFs (or PMFs) PXY, Px and py with respect to the reference measures on x V, X and V, respectively. In particular, the PMT function (Definition 1) exists (without the need to define expressions of the form log 0/0) and is smooth function from the manifold X' x ) to the set of real numbers, R. Note that as smooth functions are measurable, the PMI profile, introduced in Definition 2, indeed exists.
More generally, it is possible to work with standard Borel spaces (instead of smooth manifolds) and prove invariance of the profile with respect to arbitrary continuous injective mappings (instead of diffeomorphismss) We discuss this approach in Appendix A.5.
Recall well-known result (Kraskov et al., 2004, Appendix):
Lemma 12 (Invariance of PMI). Let X' f(X) and Y = ,(Y), where f and are diffeomorphisms. Then for every x' and if we have
PMIxY = (M,
where = (x') and y = ==-"(y'))
Proof. From
(x)) pxy(2, ldet D (f-1 x x- (d'
and analogous quantities we conclude that px'y as well as px and py, are smooth and everywhere positive functions, so that PMIx is well-defined. As D(f x g ">x,,'") is a block matrix with Df D and Dg- (yy) blocks on the diagonal and other blocks zero, we have det D( x g (2=, ,') = det D (x') det Dg- (g/).
Now we can prove invariance of the PMT profile:
Theorem 3. Let Pxy P(X,)) and f: X + X and g: y + y be diffeomorphisms. Then for X' = f(X) and Y' = gYY il holds that Pxiy E P(X,Y) and Prof = Pro
Proof. From the proof of Lemnna 12 we conclude that Px E P(X,))) Then, we note the profile is the pushforward measure
Prof (PMTxy) Pxy.
18

---

Published in Transactions on Machine Learning Research (01/2025)
Now let B S be any set in the Borel o-algebra and 1g be its characteristic function. Using the change of variables formula for pushforward measure and invariance of PMT:
Profx.y 1a(() (t)
/ 1B (PMIxxr v/)) dP'')r(''))))
f 1s (f (())) dPxy (z,
= / In (PMIxy(2, dPxy(z,
= Profxy(B)
Remark 13. The above proof can be understood also in the following manner. Lemma 12 allows us to express the PMI function of the transformed variables as the composition PMIx.y' PMIxy o (f-1 x g-
As Profx = (PMIx-r:) Px and Px (J g):xxy, the g cancels out with its inverse:
Prolx (PMIXY (S- x 1) (ʃ x Pxr
= =(MIXX o (f- gg (f x 9)) Pxy
=(PMXx o identity); Pxy
= (PM Profxy.
This perspective allows one to generalize PMI profile invariance to more general mappings, such as continuous injective functions. We prove a general resull in Appendir A.5.
A.2 Derivations of pointwise mutual information profiles
The following result shows that the distributions from the class P(X,)) with zero mutual information have the same profile:
Proposition 5. Let X and be r.v.s with joint distribution Pxy E P(X.)). Then, I(X;Y) if and only if Prof xy &0 is the Dirac measure with single atom at 0.
Proof. If Prof 60, then the expected value is I(X; and Y are independent. As the probability density functions of distributions in the P(X,3)) class smooth, we have (x, 3) (x) (y) at every point (z, v) X x ) and PMIxy(x,u)) 0 everywhere. ]
To prove the converse, if iX;; 0, then X
The following result characterizes the PMI profiles for discrete r.v.8:
Proposition 6. If X and are discrete r.v.s with Pxy E P(X,)), then the PMI profile discrete:
Profxy Epxr (P.)
vEX
Proof. The measure Pxy is discrete and given by
Pxy ΣΣ
Pxy ((..))
VEY
so its pushforward by the PMIxy function has the form
Prof (PMIxy) Pxy
Σ
Σ
PxY (x, v) 0pMixy
TEN NEN
19

---

Published in Transactions on Machine Learning Research (01/2025)
The next results describe the properties of PMI profiles associated with multivariate normal variables:
Theorem 4. Let X and Y be r.v.s such that the joint distribution Pxy P(R"",'' ,R)) is multivariate normal. k min(m, n) and pl, .2. are canonical correlations between X and Y, then the profil Profxy is generolized x² distribution, namely the distribution of the variable
K T T=I(XY)) + (Qi Q2).
where iQ Q.i==......rr independent and identically distributed (f.i.d.) variables drawn from the xi distribu- lion. In particular, xv is symmetric oround ils median, which coincides with the mean
I(xY)) = Σ 1og(1-1;).
p?)
and all moments of this distribution erist. Its variance is equal to Var(T) = pi + P
Proof. Without loss of generality assume that m S n. As the PMI profile is invariant to diffeomorphisms (Theorem 3), we can also assume that variables and Y have been whitened by applying canonical correlation analysis (Jendoubi & Strimmer, 2019), that E[X] = 0, E[ = 0 and the covariance matrix is given by
Im R
R
0
Im Σ
Exy
=
In
=
T
0 In-m/
where
Exy = (R Omx((nm))
is an m x n matrix with the last n m columns being zero vectors and R = diig((i,...,, Pm)) being the m x m diagonal matrix representing canonical correlations.
We can write the inverse in the block form
Ax
R
0
Ax A Xy
AXY Ay
Σー
=
R
Ax
0
In-m/
where the blocks have been calculated using the formula from Petersen & Pedersen (2012, Sec. 9.1):
Ax (Im Exy
diag (uI,..., um))
Ay = =In - ΣT Exx) = -diag (u1,..., Umms 1, 1
Axy -ExyAr =(N 0mx(n-m))
where Ri -diag (l. and Ui = (1(1-p7)
We define a quadratic form
s(r = AAx + y Avy + 2T Axvy
m
n Σ j=mt1
Σ
Us (z + 5t -2p,I,91))
=
which can be used to calculate log-PDFs:
17 +
log (x. = s(x, y)
log det
log 2n,
2
2
(x)
2"
log 2T,
2
logpy())
" y
1
n
log 2T.
2
20

---

Published in Transactions on Machine Learning Research (01/2025)
Hence,
PMIxy(I, x² x + y² y-
8(0, v)
log det E.
2
2
We recognize the last summand as
det Im det In
1 log det E.
IX;Y) log
det E
2
Define quadratic form
q y) (PMMlxx))))) I(x;Y)) = xx s(x, v)
Σ (1 - uz) (0) v2) + 2p2u)
which has corresponding matrix
K F
Q =
K
where
K = diag(1 [
11.--
Um
and
diag(p1 pm um)
We are interested in the distribution of
((xY)) (xT Y xx)
where (X,Y) ~ NO E)
Imhof (1961) presents a general approach evaluating the distributions of such quadratic forms. Consider a r.v.
7 NoO, Imtn)
which is split into blocks of sizes m, m and m.. If we construct linear transformation such that
y
6
then the distribution of q(X.Y) is the distribution of
2t (ATQA) Z,
Z ~ N(, Imtn).
We can construct as
P_
P+
A
P
P4
In-m/
where
1
1 Pm
1 P1
1 Pm
P_ diag
P_ diag
V
2
21

---

Published in Transactions on Machine Learning Research (01/2025)
We calculate
2 P
P
P
I m R
R
0 0
AA
P4 - P
P + P+
Im
=
Σ
0
In-m
0
0
In-m
and
2P (K-
F)
0
Q'QA-
0
2P (K F
0
where
2P (K F) diag (- P
2P (K + F) diag (m.)
Hence, the distribution of g(X, Y) is the same as the distribution of
Σ pi(-nh + +)
+
Σ ==1
0 Eg,
where () ~ NO0 am) To summarize, let Qi...., Qm, Qr..................... Qtn be i.i.d. random variables distributed according to the x distribution. The quadratic form (X, Y) has the same distribution as
Σ pi(Q -4),
which can also be written as
((x, Y) - -
Note that this distribution is symmetric around 0. We can reconstruct the profile from g(X, Y):
Profxy I(X;Y) +
Σ P: 2 xi
Pr x)))
which is symmetric around I(X;Y) and, in agreement with Proposition 5, degenerates to the atomic distribution & if and only if I(X;Y) 0, which is equivalent P; 0 for all i.
As linear combination of independent xi variables, the profile has all finite moments. Using the fact that the variance of xi distribution is 2, and quadratic scaling of variance, each term has variance 2- 2-(p/2)² = p /2. As variances of independent variables are additive, we can sum up all the 2m terms to obtain pf + ... + P
Proposition 14. For a mullivariate normal distribution with fixed mutuall information, the variance of the PMI profile is mazimized when pt = p - Pm..
- p.. To maximize the variance we equivalently have to minimize aj +... + am preserving given constraint on mutual information and a, E 10. The constraint on mutual information takes the form
Proof. Let a, = 1-
1 Σ log (1 - pp
1
I(X; Y) = 2
log (01 ... ..))
Hence, the product 0 ... am has to be constant. Denote this constant by 4th for A (0, 1] as well. Let
Gi...., Om be such that a1 ... am = T and a, E (0, 1]. From the inequality between arithmetic and geometric means we note that
a1 + ... + am
Vai ... am = A.
where the equality holds only if 01 am A. Hence, this is the unique minimum under the constraints provided. It follows that of P
22

---

Published in Transactions on Machine Learning Research (01/2025)
Corollary 15. For multivariate normal distribution with fized mutual information and cononical correlations, the variance of the PMI profile does not exceed
V m (1 c 21((;Y)
The variance is maximized when pî = =... P'm Writing p² for the common value, we have and
Proof.
p² 1 exp (-21(x;X)/ma) /m)
V = mp² = m (1 - exp (- ( (): /m)).
The mutual information can also be written as function of variance
1
I(xY)) = - 2 log (1 - p²² =
log(1 V/m).
Proposition 16. For multivoriate normal distribution with fized non-zero matual information the variance of the PMI profile is minimized when Pr + 0 for eractly one i.
Proof. Let ag E (0, 1] be any numbers. We have
(1 a1)((-122)
which is equivalent to where the equality holds if and only if aj = or 02 = 1.
0102 2 01 + 02
Using the principle of mathematical induction one can prove a more general inequality:
4102 0m + (m 1) = 1 + a1 (a ... am) + (m - 2)
2 01 + (02 am + (m 2))
01 d2 (ag am (m 3))
01 + 02 +
am-
Let us analyze when the equality can hold. To obtain equality in the first step, we need 01 1 or 02 am = 1, which, given the constraints a; (0, 1] would mean that 02 =.... = Um = Reasoning inductively, one proves that equality holds only when at least m 1 among these numbers 1.
Now define a, =1 1--pl ot and note that solving maximization problem a, (0,1) under constraint
Using the argument above we note that all the maxima for the above problem are permutations of the sequence P, P,1,1,..., This proves that at most one p * 0. Hence, we have at most one P; # O.
A.3 Constructing new Bend and Mix Models
In this section prove Proposition and Proposition 9.
Proposition 8. If Pxy is BMM and and g are diffeomorphisms with numerically tractable Jacobians (e-g., normalizing flows), then P)())(Y is BMM.
23

---

Published in Transactions on Machine Learning Research (01/2025)
Proof. From the proof of Lemma 12 and the assumption of tractability of the Jacobians of f and g obtain the tractability of the formulae for the densities P)(X))(Y); Pf(x) and Po(Y). Sampling ((Y)) amounts to sampling (X, and then transforming the sample using f and
Proposition 9. Consider BMMs PxiY::.... VxxYx- IJ wi...... wK are positive weights, such that wi +... "x = 1, then the minture distribution Pxy wyPxx +... + wxPxxYx is a BMM.
Proof. We can evaluate the densities (x, v), (x) and pr-(v) of the mixture distribution using the weighted sums of Px,Y, (a, y), (x, and py; (v), respectively. To sample (X, Pxiy, from the mixture risutiono we can sample an auxiliary variable Z ~ Categorical wi,...,,y, wk). Then, we have ((X,)") = b) = (Xx, Yx), so that we can sample from Px,Y=
A.4 Mutual information in finite mixtures
In this section we discuss the counterintuitive properties of mutual information in mixture models. We start with the proof of the failing channel inoquality:
Proposition 11. Let a E [0, be a porameter. Consider variables X, Y and Y', s.l. (x = (1 aapxx (z. 8) + an(y)px(x)) (z). Then, I(X;Y' < (1x))
Proof. Let Z ~ Bernoulli(1 - a) be an auxiliary variable. We have (X,Y)) ( = 1) ~ Pxv and (X, Y" (Z = 0) ~ Px @ Ny.
From the data processing inoquality and the chain rule we conclude that
I(:;)') < I(X; Y", 2) = I(X; +IX;) 12) Now note that and Z are independent, so I[X = 0.
2).
Hence,
(I ()
= DKL (PxY'Iz=0 || Px @ Pyi =0)) + (1 - DKK (Pxr |l Pxiz=1 8 Priiz=))
= =DLL (Pxriiz=o
Px @ Ny) akk (Pxx || Px @ Py)
= DDL. (Px @ Ny
Px 0 + (1- (Ix))
= =(1 - (x))).
In this example, mixing the signal with a noise component, not encoding any information, decreased the information. However, mixing with independent noise components can also increase mutual information.
Example 17. Let A = (0, 1) and B = (1,2) be two disjoint intervals of unit length. We define two pairs of radomm variables:
(X1, Yi) Uniform(A x A), (X2, Y2) ~ Uniform( x B).
Note that
I(NRY) = I(X2; Y2) = 0.
If (X,Y))~00)) 0.5Px,n + 0.5Px,y, is distriluted according a misture, ure have
1 (xy 1:00=1110000 = Ax
and
1 AUB]
(x)
py(y) 7 1/2 l AUB).
2
24

---

Published in Transactions on Machine Learning Research (01/2025)
| = 0
/ = O
log 2
log 2
/ - 0
X
X
X
Figure 8: Mixing two distributions with zero MI can result in mixture distribution with MI equal to one bit. Mixing two one-bit distributions can result in a mixture distribution with no mutual information.
Hence,
PMIxy v) = log 1[ E Ax
and
I(XYY) log log log
Mixing two distributions with one bit of information each, can however result in independent random variables: Example 18. Recall the distribution constructed as above:
Pxy (z. y) 1000=110000 11 AxA.Bxx)
and a symmetric one On the other hand, the mixture distribution
pvv(r,y) = - ((2,,)) E (Ax
We have
I(;Y) I(UVV) log
(Z,T) 0.5Pxx + 0.5Puv
has zero mutual information, I(Z;T) 0,
P2r(x,1) [((2,)) E (AUB) x (AUB)
= E AUB) 1 1 E AUB]
= P2(()) pT())-
Both examples have been visualised in Fig. 8. Note that similarly counterintuitive examples can be constructed using BMMs by using mixtures of multivariate normal distributions.
This demonstrates that mixtures can create and destroy information. There however an upper bound on the amount of information mixtures can create (see also Haussler & Opper (1997) and Kolchinsky & Tracey (2017)):
Proposition 10. Consider r.u.8 (Xg, Ys) such that I(Xgi Ys) oc for k 1,..., K. Let (X', Y") be their mixture with weights W1...., WK- WK.. Then,
0 D<I(XYY') <
Σ wk I(Xx; Ve) + log K.
Morcover, these inequalities are light:
25

---

Published in Transactions on Machine Learning Research (01/2025)
1. There erists a mizture such that I(X)Y)) = log K even though I(Xg; Yk) = 0 for all i.
2. There exisls a misture such that I(X'; = 0 even though. I(X;; Y&) > 0 for all k.
Proof. The examples provide explicit construclions of distributions with the specified properties. To prove the upper bound, consider variable Z ~ Categorical(K; wI...., wk). The random variables corresponding to the mixture distribution, (X', Y'), have conditional distributions
(X', Y") Z = k Px,Ya.
From the data processing inequality and chain rule we have
I('; Y*) s IX';Y', 2) 1() ((x))) - Z).
As Z is discrete, the first summand, I(X: Z), is bounded from above by the entropy (Z) (Polyanskiy Wu, 2022, Th. 3.4e), which cannot exceed log K (Polyanskiy & Wu, 2022, Th. 1.4b). The second summand can be written as
I(X';'' ()) = Σ =()= DEL (PxYrizzN lI Px & Prijz=))
K
K
=
Σ 1=
wi DKL (Pxavi Px, @ Pr.== = Σ
w I(xx
k=1
A.5 Divergence profiles
In this section we show how to define the PMI profile in a more general setting of a divergence profile and prove a general invariance result (Theorem 19). The main ideas stay similar to the proof Theorem
We consider a standard Borel space M equipped with two probability measures, P and Q, such that P P Q. For example, if one analyses the mutual information between random variables X and Y, they can define = X' y with P Pxy representing the joint distribution and Q Px & Py representing the product of marginals. Then, a sufficient condition for P < Q is to assume that I(X;Y) < cc.
As P is absolutely contimuous with respect to Q, the Radon RikonyN derivative f = dP/aQ can be defined. It is a measurable function 7:M- + [0, oo) defined up to a Q-null set.
This allows one to define the Bellback-Leibbeee divergence:
flogs I aq / log dd,,
DKL (P () (1Q)=/ flog
where 0log0 0 in the first expression. In the second expression, occur only on P-null set. Additionally, using another Radon Radon-Nikodymn Nikodym derivative (differing from f on a Q-null set), does not change the value of the integral, so that Kullback-Libler divergence is indeed well-defined.
notice that as P < Q, log 0
These properties correspond to the following properties of the divergence profile, defined as the pushforward mcasure
Prof PlI (log f)
Namely, this distribution does not depend on the choice of ff (i.c., we can replace f by a function differing on a Q-null set) and it is easy to prove that Profp]o([->0, oo)) = 0. As this distribution does not have atoms at the infinite values, we can treat. it as a distribution over real numbers, R.
oo))
Now consider a measurable mapping i: M M' between standard Borel spaces, allowing us to define the pushforward measures i,P and i on M'. We are interested in investigating the profile defined using these pushforward distributions. For example, if M = X x y, M' X' x y", P Pxy, Q Px & Py and
26

---

Published in Transactions on Machine Learning Research (01/2025)
considers random variables X' i1 (X) and Y' iY),,, then the mapping = 1 i2: M - M' defines the pushforward measures Px-y' iz Pxy and Px @ Py in (x 3 Py).
Note that the profile P can be defined in this case: as P < Q, then also igP < 4Q because i4Q(B) = 0 is equivalent to Q( (B)) = O, which then implies P(i (B)) =
In general, i can differ from Prof Plie: for example, using a constant mapping i results
Prof,, - do, independently on the original profile Prof However, we can generalize Theorem 3 follows:
Theorem 19. Let i: M - M' be mensurable mapping belween slandard Borel spaces with measurable left inverse. If P and are two probability distributions M such that P < Q, then Profi Prof
To prove it, use the following version of Lemma 12, which generalizes a result of G. (2022):
Lemma 20. Let i: M M' be measurable mapping between standard Borel spaces such that there crists measurable left inverse a: M' -MMM. If P and Q are two probabilily distribulions on X such that P < Q and f = dPP/QQ is the Madon-Nikodle derivative, then ig & i4Q and the Madon-Nikodle derivative is given by digP/di;Q = foa.
Proof of Lemma 20. Let a: M + M be any measurable left inverse of i. To prove that foa: M + 00,000 is the Nidon-Nikodym derivative we need to show that
iP(B)
foadiiQ
for every Borel subset B S M'. Using the change of variables formula:
J. f soadiaQ= = { 1(B)
SoaoidQ
(-1(BB faQ
aP
(B)
PB) = igP(BB,
where the second equality follows from the definition of the left inverse, aoi idm.
Note that even if is substituted for another left inverse, the Radon-Nikode derivative f o a is still determined uniquely up to an i,Q i4Ql set.
Proof of Theorem 19. The profile defined as Prof = (logdi, P/di;QQ) (14P). Take any measurable left inverse a i and write
di,P
log di,Q
log f a
where f - dP/dQ. We have
Prof = (log ʃ i ipP (log f o o P= (log f)P Prof Q-Q-
Remark 21. The above proof suggests that for P & Q on slandard Borel space M and sufficiently well-behaved functions u: [0, oc) + I, one could define generalized profile
dP u
P :
dQ
which stays invariant under mensurable mappings i: M M' with measurable left inverses.
27

---

Published in Transactions on Machine Learning Research (01/2025)
These quantitics can be used to study whether distribution could not be obtoined simple reparamctrization of another one, similarly os the PMT profiles were used in Fig. 1.
We suspect that it may also be possible to generalize the profile to use a general forerall-diieeee f-divergence (Polyanskiy & Wa, 2022, Ch. 7), rather than the Kullback- -elbleck-Leibler divergence studied in this manuscript. Nowozin et al. (2016) related f-divergences and their variational lower bounds (ef. Sec. 3.3) to generative adversarial networks (GANs), although relation between a "f-divergence profile" and the GAN training remains unclear us.
Theorem 19 proves invariance under the existence of a measurable left inverse. Czyž et al. (2023, Theorem 2.1) prove invariance of the mutual information under reparametrizations by continuous injective mappings. As the lemma below shows, Theorem 19 holds also for continuous injective mappings:
Lemma 22. Let i: M + M' be continuous injective mapping between two standard Borel spaces. Then, admits a measurable left inverse.
Proof. Choose an arbitrary point zo E M and define function a: M' - M in the following manner:
if & i(M)
a(y)
1 x if is the (unique) point such that i(x)
This function is well-defined due to the fact that is injective and it is left inverse: a(i(x)) = for all x E M. Now we need to prove that it is measurable.
Take any Borel set B C M and consider its preimage a (B) = (y E M' a(v) E B). 20 & B, we have (B) i(B), which is Borel by Lusin-Sushee theorem. If zg E B, we can write
a- (B) aB) (eo)))
= =iB (xoo) U (i(zo)) ((M)))))
which is Borel as a finite union of Borel sets.
B Distributions involving discrete variables
The formalism Scction is applicable both continuous and discrete random variables, although Section 3.1 we focus on the distributions in which both X and are continuous. If X and Y are discrete, mutual information Y);Y) be calculated analytically from the joint probability matrix and there exist. numerous approaches to estimate it from collected samples (Hutter, 2001; Brillinger, 2004).
In this section we consider the mixed case, in which one variable is continuous and the other one is discrete. For example, Carrara & Ernst (2023) describe a particle physics experiment which is an 18-dimensional random variable, but Y is binary. Grabowski et al. (2019) consider a cell transmitting information through the MAPK signalling pathway, assuming the input signal X to be discrete and the measured response Y be continuous.
B.1 Known distributions
There are only a few distributions Pxy with known ground-truth mutual information assuming this discrete- continuous case. Gao et al. (2017, Sec. 5) describe discrete random variable X which is uniformly sampled from the set X = (O...., m 1} and the continuous Y variable is sampled as
(Y X = r) Uniform(z, 2).
which is therefore distributed on y = (0, m + 1). Gao et al. (2017) prove that mutual information in this case is
I(X; Y)
log m m log
28

---

Published in Transactions on Machine Learning Research (01/2025)
Samples from Pxr
PMI profile
2
10
Density
0
5
-2
0
-2
0
2
-3
-2
-1
0
1
X1
PMI
Figure 9: Left: samples from the Pxy distribution, colored by the value of the binary variable Y. Right: the PMT profile of Pxy distribution.
for 2 2 and I(X;Y) = 0 for 1 and consider multivariate analogue of this distribution, in which pairs of variables (X,, Yx) for k = 1....,K K are sampled independently using the above procedure. Then, they are concatenated into multivariate vectors (XI,..., Xx) and (V,...,, Yk) with K times larger mutual information.
We will show how relate the bivariate example the framework of Bend and Mix Models (multivariate case can be constructed analogously). Note that the joint distribution Pxy is not strictly in P(X, ()),)), as it is supported on the one-dimensional manifold M C X x y with connected components (cf. Politis (1991) and Marx et al. (2021)), on which
PxY (z. v) -111 1ly (x, 2)],
2n
but it is possible to extend the definitions of Section 2, so that this technical difficulty is resolved. The marginal distributions are tractable and admit PDFs:
Px(x) = -1/m,
m-1
(v = Σ Pxy(2,1)
==0
Although is not smooth on y (at integer points), this technical difficulty can also be resolved as this set is of measure zero. Hence, although this distribution not strictly a BMM, it can still be modcled using the introduced framework.
Next, Gao al. (2017, Sec. 5) consider the zero-inflated Poissonization of the exponential random variable valued in X = fa E R x 2 0}, with: X ~ Exp(1) and Y being discrete random variable valued in the set of non-negative integers 1, 1,
(Y X = x²) ~60 (1 p) Poisson(a)).
They show that
0c
I(X:Y) (1 (1-p)
22lo22 7 log 2-h
Σ
where y is the Euler Mascheroni constant.
To use BMMs in this case, one has to formally extend the definition of P(X,)) )) as X is manifold with boundary (Lec, 2012, Ch. 1). The joint probability distribution is then given by
(xy = px(x) Pyix x) = = (p. 1ly = 0] + (1 -
2 x
y!
29

---

Published in Transactions on Machine Learning Research (01/2023)
and the PMF function of the Y variable is also analytically known:
= 0) -1 --+
Hence, the framework of BMMs, with minor technical adjustments, can accommodate the above distributions.
B.2 Novel distributions
However, Bend and Mix Models also allow one to creale more expressive distributions, for which analytical formulae for ground-truth mutual information are not available, but can be approximated with the Monte Carlo methods as explained in Section 2: consider a continuous random variable X and discrete random variable Y. To introduce dependency between X and Y variable, we can use a mixture of distributions in which the component variables are independent, i.e., Px,y, = Px, 6 Py. Therefore, we consider a graphical model X + 1 + Y, in which the distributions Px, = Px = are known and have tractable PDFs. The distributions of Pr. = Prize are given by probability tables. Monte Carlo estimators of Section 2 can then be used estimate I(X;Y) with high accuracy. Note also that this general procedure includes the case X < Y by setting Z Y.
We illustrate it in simple example with X R², y' {0.1) and K = 3 components. The first component models cluster in the X' space, strongly associated with Y = 1 value. For Px, we use a bivariate Student distribution centered at (1, 1) with isotropic dispersion Q = 0.2 T2 and 8 degrees of freedom. We take Py, to be the Bernoulli variable with probability P(Y = 1) = 0.95.
Analogously, we define a second cluster, strongly associated with Y 0 value: Px, is bivariate Student distribution with the same dispersion matrix, but centered at (-1,-1) and with degrees of freedom. Then, Y2 is Bernoulli variable with P(Y- 1) 0.05.
We then define a third component using bivariate normal distribution centered at (0, 0) and with covariance matrix
1 0.95 E
0.95 1
This component is not informative of Y, that is P(Y; 1) 0.5.
We used weights w1 w2 = 1/4 and w3 1/2, what resulted in the distribution visualised in Fig. 9. We estimated both the profile and the mutual information using N = 10th samples and obtained I(X;Y) = 0.224 with MCSE of
In principle, the above construction can be used to generate realistic high-dimensionall data sets (e-g., audio or image) with known ground-truth mutual information, by assuming the generative model X (i.c., Z Y) and modeling each Px, using normalizing flow or an autoregressive model (Murphy, 2023, Ch. 22) trained on auxiliary data set with fixed label Y& yk- Hence, at least in principle, one could obtain highly-expressive generative process P v) with tractable probability and sampling. Pairing this with an arbitrary probability vector (3) one can obtain Pxy (z, y) and (x(x) even for high-dimensional data sets, so that Monte Carlo estimator can be used to determine the ground-truth mutual information. However, we anticipate possible practical difficulties with scaling up the proposed approach to hige-dimen data and we leave empirical investigation of this topic to future work.
C Experimental protocols and additional experiments
In this section we provide detailed information on experimental protocols used (such as hyperparameter choices) and supplement the findings of Sec. 3 with additional experiments.
C.1 Specification of the proposed distributions
In this section we provide the details of the distributions introduced in Sec. 3.1. To estimate the ground-truth mutual information we used the Monte Carlo approach described in Sec. 2 with N = 200 000 samples.
30

---

Published in Transactions on Machine Learning Research (01/2025)
C.1.1 The X distribution
We constructed the X distribution as a mixture of bivariate normal distributions with equal weights, zero mean and covariance matrices specified by
10.9
Σ 0.3
+0.9
Note that the marginal distributions of each of component distributions is V0,00.3) and subsequently their mixture has exactly the same marginal distributions. This is therefore an interesting example of a distribution in which the joint probability distribution is not multivariate normal, although the marginal distributions of X and Y variables normal individually. The mutual information is in this case I(X;Y) 0.41 nats.
C.1.2 The Al distribution
The AI distribution was constructed as an equally-woighted mixture of six bivariate normal distributions with equal weights and the following parameters:
/41 (1,0)
Σ; diag(0.0110..)
μ42 (1, 1)
Σ2 diag(0.05, 0.001)
μ3 (1, 1)
L diag(0.05, (0.05, 0.001)
μ4 (-0.8, -0.2)
Σ4 diag(0.03, 0.001)
μ5 (- 1-1.2,0)
0.04
0.085
Σ 0.085
0.2
N6 0.4,0) 0)
0.04
-0.085
Eg 0.085
0.2
The mutual information of this distribution is I(X;Y) 0.78 nats.
C.1.3 The Galaxy distribution
The Galaxy distribution was constructed as equally-weighted mixture of isotropic multivariate normal distributions with μı = (11, 1, and unit covariance matrix and the X variable was transformed using the spiral diffeomorphism with = 0.5, which is described by Czyž et al. (2023). The Galaxy distribution contains IX; 0.49 nats.
C.1.4 The Waves distribution
The Waves distribution created weialty- mixture of 12 multivariate normal distributions with equal covariance matrices Σ 1,, and mean vectors
#4 (z, 0, mod ie (0, 113.
This construction results in distribution where different vertical components of the X variable assigned Y values calculated modulo 4. Then, we transformed the X variable with continuous injection
12) (x) 5 (3x2),
31

---

Published in Transactions on Machine Learning Research (01/2025)
which does not change the mutual information. Finally, we applied the affine mappings
a (2) 0.1r 0.8, a2(y) 0.5g.
to make the range of the typical values comparable with other distributions. This distribution encodes
I(X;)) 1.31 nats.
C.2 A benchmark extension proposal
In Sec. 3.1 and Appendix C.1 we provide examples of novel low-dinensional distributions, which pose a significant challenge to the mutual information estimators. Therefore, we would like to propose a new benchmark of mutual information estimators, extending the benchmark of Czyă et al. (2023).
C.2.1 Included distributions
The original benchmark of Czyž et al. (2023) consists of 40 distributions. On several problems the estimator performance was similar, so that choosing a representative from a group could allow us to reduce the computation amount. Additionally, we decided to remove the lasks which we considered too difficult: for example, the Student distributions are mapped through the asinh transform to remove the tails. Together with the new tasks, based on BMMs the new version of the benchmark consists of 26 distributions grouped as follows:
One-dimensional variables We include four distributions such that X = RR From the original benchmark we decided to retain the additive noise lask with 0.75 and the asinb-transsormedd version of the centered bivariate Student distribution with one degree of freedom and the identity matrix for the dispersion (scc Czyă et al. (2023, Appendix D) for precisc descriptions). Additionally, we include X and AI distributions described in Appendix C.1.
Embeddings We retained the Swiss roll embedding distribution (Czy² et al., 2023, Appendix D). In this problem the distribution Pxy is supported on a two-dinensional surface embedded in R This distribution does not have a density with respect to the Lebesgue measure on RR3
Many-versus-ome distributions We consider multiple distributions with X = Rm and y = R. For m 2 we consider the Waves and Galaxy distributions described in Appendix C.1. For other m we propose the concentric isotropic multivariate normal distributions. Namely, let K be parameter, specifying the number of components of BMM. Fach component has independent (multivariate) normal variables X₁ ~N(, and Ys ~ (Nk,, 10-4). To obtain non-zero mutual information with equal proportions, i.c., the weights vector is given by = 1/K for all k. We consider four tasks obtained by varying m E (3.5) and K E (5,10), Additionally, we consider a high-dimensionall distribution with m = 25 and K = 5 components.
mix these distributions
Multivariate normal and Student distributions We selected five problems from multivariate normal distributions (Czyž et al., 2023). We use X' = m and y = R and the dense interaction model jointly changing m n E (5, 25, 50} dimensions and the sparse interactions model (with two pairs of interacting dimensions) in
(5, 25} dimensions.
Additionally, selected three multivariate Student distributions with the dispersion being the identity matrix. For m = 2 we consider distribution with one degree of freedom (i.c., the multivariate Cauchy distributions, which does not have first. two moments) and for m = n E 53.5) we consider a distribution with two degrees freedom (for which the first moment is defined, but not the second). As described above, the Student distributions had been transformed with the asinh mapping to reduce the tails.
Spiral diffcomorphism As described in Czyž et al. (2023), one can transform the multivariate normal distribution with the spiral diffcomorphism, which results in empirically challenging problems. We selected the distributions corresponding to the sparse interactions in (3,5) dimensions.
32

---

Published in Transactions on Machine Learning Research (01/2025)
Table 1: Benchmark results. New benchmark problems are marked in green. Best-perfoming estimator in each row has been marked with bold font.
CDA
InfoNCR
KSB
NWJ
Troe
AT
0.00 0.01
0.57 1 0.06
0.61 1
0.78 0.02
0.49 =
0.53 0.10
0.00 1 0.01
0.37 0.02
0.58 1 0.02
0.42 1 0.02
0.33 = 0.05
0.36 0.02
0.41
Additiiee
0.15 + 0.01
0.31 0.02
0.30 2 0.02
0.82 2 0.01
0.32 0.02
0.32 2 0.01
0.38
Swise roll
0.02 + 0.01
0.37 0.37 0.02
0.37 2 0.03
0.42 0.02
0.37 = 0:03
0.37 0.02
0.41
Galaxy
0.01 + 0.01
0.31 0.04
0.38 + 0.03
0.46 + 0.01
0.28 = 0.08
0.25 0.04
0.49
Wavea
0.04 0.01
0.14 0.14+ 0.07
0.34 + 0.13
0.08 0.01
0.13 0.10
0.11 0.07
1.51
Concentaic (8-dim. 10)
0.00 + 0.01
0.90 0.02
0.50 + 0.02
0.61 + 0.01
0.45 0.02
0.64 0.03
0.50
Concencric [3-41m, 5]
0.00
0-41
0:41
0.42 2 0.02
0.38 0.05
0.36 0.03
Concenerie [b-dim. 10)
0.00 0:01
0.65
0.88
0.55 0.01
0.62
0.56 0.02
0.75
Concentric [G-dim,
0.00
0.05
(25-dtmn
0.06
Student (2-dim))
0.80
Studeat [8-dim))
0.00
0.17
Student [5-dtm)
0.01 +: 0.01
0.21
0.28 0.03
0.23 0:01
0.13
0.05 0.03
0.45
Inlters (25-dim, 0.2)
0.58 U.Us
0.34 0.04
0.27 +
0.12 0:03
0.39 0.05
0.41 0.04
0.63
Inliers (25-dim, 0.5]
0.24 + 0.02
-0.11 0.09
-0.08 + 0.04
0.05 0.02
0.08 0.02
0.06 0.04
0.27
Inllers (5-dim, 0.2)
0.52 1: 0.02
0.33 O
0:55 1 0.03
0-45 0.02
0.04 2 0.05
O. 0.50 0.06
0.63
Iollers (5-dtm, 0.5)
0.15 111.-:10 0.01
0.18 0.02
D.16 1 0.03
0-18 0-01
0.20 0-03
0.21 0.04
0.27
Normal (25-dimm. denue)
1.35 + 0.02
1.00 0.07
1.06 + 0.06
1.05 0.02
1.16 = 0.05
0.18 0.38
1.29
Normal (G-dia,, dease)
0.00
0.56 0.03
0.54 + 0.03
0.56 = 0.01
0.6G = 0.03
0.5G 0.02
0.59
Normal (50-dim. deuse)
1.97 +:
1.26 0.07
1.20 + 00
1.25 0.02
1.45 0.05
0.62 0:78
1.02
Normal (25-dim, sparse)
1.08
0.89 0.05
0.67
0.18 0.02
0.78
0.79 0.03
Normal [B-dam, sparse]
1.03
0.95
0.95 02
0.69 0.02
0.92
0.95 0.02
Spiral (3-dim)
Spiral (3-dim)
Inliers Finally, we implemented four distributions based along the principles described in Sec. 3.2. Namely, took the multivariate normal distributions with sparse interactions (described above) with m (5, 25) dimensions Pxy and then constructed mixture (1 - ax + aPx @ Py for the inlier fraction E (0.2,0.5)
C.2.2 Experimental protocol
Determining the ground-truth value The ground-truth mutual information for distributions proposed by Czyz et al. (2023) is analytically available. For the tasks based on BMMs we used N = 200,000 Monte Carlo samples to provide an estimate, as described in Section 2.2. The additional computational cost. of estimating the ground-truth mutual information for these distributions is minor: on standard laptop, estimating the mutual information of the Galaxy task to be 0.4953 + 0.0012 (which results in the relative MCSE smaller than 0.5%, using 200,000 samples) takes less than 2.5 seconds. Similarly, estimating the mutual information of the Concentric task (25-dimensional variant with 5 components) to be 1.1911 + 0.0015 (using 200,000 samples again) takes less than 3.5 seconds.
Included estimators We included total of six estimators in this benchmark: four variational estimators, the neigbhorhood-basee KSG estimator (Kraskov et al., 2004), and the simple OCA-based estimator of Kay (1992). We chose them as the most promised approaches: Czyă et al. (2023) argue that KSG is the preferred estimator for low-dinensional problems, neural estimators are preferred choices for high-dimensional problems if only few dimensions are interacting, and the CCA- based estimator the optimal choice for multivariate normal distributions.
Reported uncertainty For each distribution we sampled N' 5,000 data points S 10 times to obtain different data sets on which the estimators were run. For each estimator and a task we then calculated the mean and the standard deviation (basing the S 5-1 1 degrees of freedom), which reported in the table up to two decimal digits. We rounded the standard deviation upwards to two decimal digits to not. underestimate it.
C.2.3 Benchmark results
We present the obtained results in Table 1. Results for the BMM estimator on low-dimensional tasks are shown in Table 2.
Problems not solved Interestingly, the Waves task is not solved by any estimator, with the rest-performing one being KSG and underestimating the MI by 50%. Similarly, high-dimensional Concentric distribution
33

---

Published in Transactions on Machine Learning Research (01/2025)
(25-dimensional variables with 5 components) is not solved by any estimator. This contrasts with using single component (Normal (25-dim, dense) and Normal (25-dim, sparse)), for which the CCA estimator can be used. Problems involving the spiral diffeomorphhism remain unsolved.
KSG For low-dimensional problems, the KSG estimator seems preferable, which agrees with the conclusions from Czyá et al. (2023).
Neural estimators In high-dimensional problems (25-dimensional or 50-dimensional variables), neural estimators generally outperform KSG. However, they typically still underestimate the ground-truth mutual information and the model-based CCA estimate may be preferable in the problems involving multivariate normal distribution.
CCA In the proposed benchmark it is visible that a CCA-based estimator not competitive on majority of the problems. The concentric multivariate normal distributions, designed as adversarial examples, indeed result in CCA not being able find any mutual information. Morcover, we see that larger mumbers of inlicrs result. in worse predictions. Interestingly, for N 5,000 data points and higl-dimensional problems the CCA-based estimator can overestimate the result, by overfitting to the noise. This supports the view that regularizing may improve the estimates (sce Sec. 3.4).
C.3 Estimator hyperparameters
Czyž et al. (2023, Appendix E.4) study the effects of hyperparameters mutual information estimators. We decided to use the histogram- based estimator (Cellucci et al., 2005; Darbellay & Vajda, 1999) with fixed number 10 bins per dimension and the popular KSG estimator (Kraskov et al., 2004) with = 10 neighbors. Canonical correlation analysis (Kay, 1992; Brillinger, 2004) does not have any hyperparameters. Finally, we variational estimators with the neural critic being a ReLU nelwork of variant M (with 16 and hidden neurons), as it obtained competitive performance in the benchmark of Czyž et al. (2023, Appendix E.4). As preprocossing strategy, followed Czyž et al. (2023, Appendix E.3) and transformed all samples to have zero empirical mean and unit variance along each dimension. Our code is based on the MIT-licensed code associated with the Czyž et al. (2023) publication.
C.4 Variational estimators of mutual information
In Sec. 3.3 we study the loss of Belghazi et al. (2018) as well as two other loss functions. This section provides more detailed overview of these approaches. At the same time, our review is not exhaustive: many more variational lower bounds exist and have been described in the literature. For a detailed overview we refer to the articles by Poole et al. (2019) and Song & Ermon (2020), as well as to Chapters 4 and 7 of the textbook by Polyanskiy & Wu (2022).
Belghazi et al. (2018) use the Donsker Varadhan loss,
Ifv(f) = Exxx [f] log ExxxPy [exp
which is lower bound on I(X;Y) Y) for any bounded function f. Hence, they decide to use a neural critic, i.c., a neural network f: X' x y + IR maximizing the functional form to obtain an (approximate) lower bound on mutual information.
As Poole et al. (2019) descríbe, this bound becomes tight if is any real number and f = PMIxy + i.c., Inv( I(X;Y) Y), so one can approach MI estimation optimisation Inv over flexible family of functions f parameterized by neural networks. However, note that as only finite sample is available, the expectation value cannot be caleulated exactly, so that any provided estimate does not need to be lower bound on MI. Morcover, if no split into training and test set is used, then may overfit and provide biased estimates (McAllester & Stratos, 2020).
Another lower bound, introduced in Nguyen et al. (2007),
(Nw = Exxr [f] - Exxory /exp (f -1)],
becomes tight for f PMI xy +1.
34

---

Published in Transactions on Machine Learning Research (01/2025)
Oord et al. (2018) propose variational approximation loss which usee batch of (zi, 11 samples from Pxy to estimate
T 7t
exp f
[NCE(f) E
log ト exp f (I Mz)
Here, f(z,y) y) PMMXx((,.)) + c(x). where с is any function, then INCE(F) - I(X;)) as n oc.
C.5
Bayesian estimation of Gaussian mixture models
In this section we provide additional details on using BMMs to estimate mutual information and the pointwise mutual information profile.
C.5.1 Model description
Recall from Sec. 3.4 that Bayesian estimation of mutual information consists of the following steps:
Propose parametric generative model of the data, Po P(X, Y (), and assume a prior P(0) on the parameter space.
Use a Markov chain Monte Carlo method obtain sample ((1), @(),..., 0(m) from the posterior P(0 X1, , YN).
3. Estimate mutual information (and the PMT profile) for each ((m) using the Monte Carlo method described in Sec. 2.
4. Validate the findings using e.g., posterior predictive checks and cross-validation.
We consider the following sparse Gaussian mixture model with K = 10 components:
Dirichlet(K;11KK, 1/K, 1/K),
Zn Categorical(T))
n1 N,
N4 N (0, ID),
k 1, .... K,
E ScaledLLK((111)
k K.
(Xn, Yn) Zn, (/u, } ~ N(zz, Ez.),
n N.
Sampling single covariance matrix Σ from ScaledLKJ((r.1) 7) distribution corresponds to sampling the correlation matrix R from the Kwwandowski-KKuroiikkk..e/ (LKJ) distribution (Lewandowski et al., 2009):
p(R) x (det RY"-1
sampling the scale parameters
Å1 A2. AD HallCaachyysscae--o)
and then constructing the covariance matrix as Egj Rayd,Aj-
The sparse Dirichlet prior is a finite-dimensional alternative the Dirichlet process, which truncates the number of occupied clusters depending the data (Frihwirth-Schnatter & Malsiner- Walli, 2019). In particular, the priori expected number of clusters depends on the number of data points to be observed. We generally use NumPyro (Phan et al., 2019) with local latent variables Z,, marginalized out, what allowed us to run Markov chain Monte Carlo inference using the NUTS sampler (Hoffman & Gelman, 2014).
The mth sample is therefore given by
g(m)_ (m)) (1(T) s(()) k=1...,,K
which is then used to parametrize Gaussian mixture distribution Poimy. Finally, using the Monte Carlo method described in Sec. 2, then estimated mutual information I(Poto)) and the PMI profile.
35

---

Published in Transactions on Machine Learning Research (01/2025)
Table BMM-provided estimates for selected low-dimensional problems. Each run corresponds to estimation different sample and reports the mcan together with a credibility interval obtained from the 10th and 90th percentile. Bold font represents problems solved by BMMs.
Thue MI
Run 1
Run
Run
Additive
34)
(0
26-0.311
20-0
AI
(0.74 0.52)
(0.73 0.50)
(0.76 0.82)
Galeay
0.49
0.27 (0.24 0.20)
0.31)
Concentric (3-dim,
0.46
0.46 (0.43 0.46)
0.46
0.49)
(0.42 42
Concentric (5-dimn,
0.63
0.64 (0.51-0.89 88)
0.53 (0.80-0.00)) 86)
0.63
90-00-00000 86)
Concentric (3-dim. 10)
0.58
0.58 (0.53-0.59)
0.37 (0. (0.53-0.80) 80)
0.48
(0 43-44---440 49)
Concentric (6-dim, 10)
0.75
0.63 (0.90- 00-00.00 66)
0.52 (0.58- 0.05)
0.58
(0 58 59)
Inilers (5-dim, 0.2)
0.63
0.68 (0.6 (0.62-0.74) 74)
0.68
00-61-0-71)
0.54
(0 59-0 69)
Inllies (3-dtm, 0.5)
0:27
0.18 (0. (0-15-0:21)
0.28
(0 06-0333)
0.18
(0: 16-0-211
Normol (6-dim, donse)
0.59
0.58 (0.54 0.62)
0.61 [0.57 0.65)
0.59
(0.55 56 0.63)
Normel (6-dim, sparse)
1.02
1.02 (0.97 87)
1.04
(0. 10.00
000
1:03
(0 96
06)
Student (1-dim)
0.22
0.24 (0 20.21-11 27)
0.28
(0 20 261
0.22
(0 10-0 25)
Student (2-dim)
0.43
0.43 (0. (0.30-1 074
0.41
(0
00.37-1 44)
37
Student (3-dim)
0.29
0.19
(0. 22)
0.20
17 23)
19 25)
Student (5-dim)
0.26
30)
C.5.2 BMMs performance on the proposed benchmark
We evaluated the performance of the BMM described above on a subset of distributions proposed in Appendix C.2. We excluded the high -igheninni distributions due to the high computational cost of fitting Markov chains: in our preliminary experiments used to select sampling hyperparameters, the chains often failed to sample properly the posterior distribution. For lower-dimensional problems we did not notice convergence problems in our preliminary runs (apart from the label-switching issues, which do not affect the predictive distribution). To reduce the computational cost, we decided to use three (rather than ten) samples from cach distribution. Then, for each sample we ran a single Markoy chain with 1,000 warm-up steps and 1,000 collected samples. As we ran a single chain and the model is not identifiable due to the label switching of Zn variables, we could not employ metrics diagonosing convergence failures, such as A (Vehtari et al., 2021). We then used 1,000 Monte Carlo samples to estimate the MI in each distribution.
In Table 2 we report the MI estimates across three runs. Each estimate is summary of the posterior distribution, specifying the mean and the 10th and 90th percentile of the distribution.
Overall, we see that BMMs offer good performance and appropriate uncertainty quantification in low- dimensional problems, such as X, AI, normal and low-dimensional concentric distributions. Interestingly, even when the model is slightly misspecified (additive and low-dimensional Student distributions with fall-shortening transformation applied), we obtain good performance.
It is however important to note that the proposed approach is not universal: for 5-dimensional concentric distribution and 10 components and 5-dimensional problem with 50% of inliers, the model, even though it is well-specified, does not always estimate the mutual information appropriately, underestimating the mutual information. We suppose that posterior predictive checking and sensitivity analysis can be used to diagnose problems with this model. We investigate this issue in more detail in Appendix C.5.3.
C.5.3 Posterior predictive checking and estimation of pointwise mutual information profiles
As discussed in Section 3.4, BMMs allow one to do model-based estimation of both MI and the PMI profile. At the same time, they require domain expertise to propose a generative model as well as careful model criticism. In this section we further investigate these aspects, by applying the model to the X, AI, Waves and Galaxy distributions, changing the number of data points N E (125, 250, 500, 1000).
For each distribution we sampled N data points once, ran a single Markov chain with 2000 warm-up steps and collected 800 samples. Then, for each sample we estimated MI and the profile via the Monte Carlo approximation using 100,000 samples. We visualise the observed sample, a single posterior predictive sample and posterior on mutual information and the PMI profile in Fig. 10, Fig. 11, Fig. 12 and Fig. 13.
Although the model performance is good for the X and AI distributions, which additionally suports the results obtained in Appendix C.5.3, we see that model misspecification results in unreliable estimates for the Waves and Galaxy distributions.
36

---

Published in Transactions on Machine Learning Research (01/2025)
Ground-truth sample
Simulated sample
Posterior MI
Posterior PMI profile
1
1
X
0
0
-1
-1
1
-1
0
0.0
2.5
PMI
Ground-truth sample
Simulated sample
Posterior MI
Posterior PMI profile
0
0
-1
-1
0.0
2.5
X
X
MI
PMI
Ground-truth sample
Simulated sample
Posterior MI
Posterior PMI profile
1
0
-1
-1
-1
0
1
-1
0
1
0
1
0.0
2.5
X
MI
PMI
Ground-truth sample
Simulated sample
Posterior MI
Posterior PMI profile
1
1
X
0
0
-1
1
0
0.0
2.5
MI
PMI
Figure 10:
Gaussian mixture model fitted to the X distribution with 125, 250, 500 and 1000 samples.
37

---

Published in Transactions on Machine Learning Research (01/2025)
Ground-truth sample
Simulated sample
Posterior MI
Posterior PMI profile
1
1
]
i
0
>
0
-1
-1
1
-1
0
0.0
2.5
MI
PMI
Ground-truth sample
Simulated sample
Posterior MI
Posterior PMI profile
1
A7O
A
A
0
0
-1
-1
0
0.0
2.5
X
X
MI
PMI
Ground-truth sample
Simulated sample
Posterior MI
Posterior PMI profile
1
Alo
A
0
-1
-1
-1
0
1
-1
0
1
0
1
0.0
2.5
X
MI
PMI
Ground-truth sample
Simulated sample
Posterior MI
Posterior PMI profile
1
AI
1
AI
x
0
0
1
-1
0
0.0
2.5
MI
PMI
Figure 11: Gaussian mixture model fitted to the AT distribution with 125, 250, 500 and 1000 samples.
38

---

Published in Transactions on Machine Learning Research (01/2025)
Ground-truth sample
Simulated sample
Posterior MI
Posterior PMI profile
1
1
0
0
-1
-1
-1
0
1
-1
0
X1
X1
MI
PMI
Ground-truth sample
Simulated sample
Posterior MI
Posterior PMI profile
0
0
-1
-1
X1
X1
MI
PMI
Ground-truth sample
Simulated sample
Posterior MI
Posterior PMI profile
1
-1
-1
-1
1
-1
0
1
0
1
0
2
X1
X1
MI
PMI
Ground-truth sample
Simulated sample
Posterior MI
Posterior PMI profile
1
1
0
-1
1
0
0
X1
X1
MI
PMI
Figure 12: Gaussian mixture model fitted to the Waves distribution with 125, 250, 500 and 1000 samples.
39

---

Published in Transactions on Machine Learning Research (01/2025)
Ground-truth sample
Simulated sample
Posterior MI
Posterior PMI profile
1
1
0
0
-1
-1
1
-1
0
-1
X1
X1
MI
PMI
Ground-truth sample
Simulated sample
Posterior MI
Posterior PMI profile
1
0
0
-1
-1
1
X1
X1
MI
PMI
Ground-truth sample
Simulated sample
Posterior MI
Posterior PMI profile
1
-1
-1
-1
1
-1
0
1
0
1
-1
0
1
X1
X1
MI
PMI
Ground-truth sample
Simulated sample
Posterior MI
Posterior PMI profile
1
1
0
0
-1
-1
0
X1
X1
MI
PMI
Figure 13: Gaussian mixture model fitted to the Galaxy distribution with 125, 250, 500 and 1000 samples.
40


Reviews:
[{"review_id": "2UQ3jYrdtc", "reviewer": "Reviewer_xnSm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Disclaimer:** I am not familiar with the previous state of the art on MI estimation, so I cannot comment on novelty. In particular, I don't know whether the PMI profile has been studied at all before (the literature review presented by the authors seems to suggest it hasn't).\n\n## Strengths\n\nThe paper is well written and easy to read.\nIts theoretical part contains elegant and non-trivial results, such as the PMI profile of a multivariate normal. The connection with normalizing flows is also pleasant.\nIts experimental part makes a compelling case in favor of non-trivial distributions for MI estimation benchmarks. Indeed, data sampled from BMMs lets us discriminate between estimation algorithms which would perform similarly on simpler tasks. Additionally, the PMI profile naturally gives rise to uncertainty estimates for MI, which other methods don't always provide.\n\n## Weaknesses\n\nTo me, the main flaw of this article lies in the exposition, which gives center stage to the family of BMMs. I don't think that is where the true contribution is.\nSuppose I have two random variables $X_1$ and $X_2$ for which I can draw samples and compute densities. The fundamental idea behind BMMs is that two simple transformations also behave nicely:\n\n| | drawing samples | computing densities |\n| --- | --- | --- |\n| diffeomorphism $f(X_1)$ | always possible | possible if $\\det J_f$ is tractable |\n| mixture $w_1 X_1 + w_2 X_2$ | always possible | always possible |\n\nThis fact is easy to prove and has no doubt been used countless times before, especially in the literature around graphical models, probabilistic circuits, normalizing flows, etc. The same might hold for two-step Bayesian estimation of MI (first $\\theta$, then $PMI | \\theta$) but I'm less confident about it.\n\nMore problematically, BMMs are not _sufficient_ to make MI estimation tractable: even if the Monte-Carlo estimator is unbiased, it may require many samples to provide accurate values. This is of course mentioned by the authors, but not prominently enough. Finally, the definition of the BMM family is still rather vague, because \"Jacobian tractability\" is not defined univocally.\n\nIn my view, the focus should be shifted from the BMMs themselves to their experimental applications. In the case of benchmark tasks, it is fine to let the Monte-Carlo simulation run for $N = 200 000$ samples, because we need to compute a precise ground truth. What matters after that is not how we computed this ground truth, but how various estimators measure up to it. However, I am eager to discuss this point with the authors and understand their perspective.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is concerned with the study of Pointwise Mutual Information (PMI) between two random variables $X$ and $Y$, which generalizes Mutual Information (MI). The PMI function is defined as $PMI_{X,Y}(x, y) = \\log P_{X,Y} (x, y) / \\log P_X(x) \\log P_Y(y)$ and the PMI profile is the distribution of $PMI_{X,Y}(X,Y)$. In particular, the expectation of the PMI profile is the MI itself.\n\nThe authors prove several properties of the PMI profile, the most important one being invariance through diffeomorphisms (bending) and mixtures. This justifies the definition of \"Bend and Mix Models\" (BMMs), for which MI admits a tractable, unbiased Monte-Carlo estimator.\nBy combining simple distributions into more sophisticated ones, BMMs yield more challenging benchmarks for MI estimation. The need for such benchmarks is supported by a series of numerical experiments, in which existing MI estimators are compared on BMM-generated data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Major\n\n1. In the whole manuscript, the claims related to novely / tractability / universality of BMMs should be toned down.\n\n## Minor\n\n1. The colorblind-friendliness of some figures (2-right, 3, 5) should be improved, e.g. by using different markers / linestyles in addition to colors (the present reviewer is partly colorblind).\n1. In Section 2.1, Theorem 4, canonical correlations should be defined.\n1. In Section 3.2, the following sentence seems incorrect: Monte-Carlo estimation is not exact.\n> Alternatively, we can model the system as a BMM and evaluate this quantity *exactly*.\n\n## Questions\n\n1. In Section 3.1, how do you explain that no estimator can accurately represent the waves task? Is it a qualitative limitation (insufficient expressiveness), or just a quantitative one (insufficient training, waves too close together, etc.)?\n1. In Section 3.2, Proposition 11, is the upper bound tight?\n1. In Section 3.3, I found this remark particularly interesting. Do you have any evidence for it beyond the empirical one you present, even vague theoretical intuitions?\n> This suggests that although the neural critics may not learn the PMI function properly in regions with low density, the PMI profile (and, hence, the MI) can still be approximated well.\n1. In Section 3.4, can you explain or demonstrate why the plug-in estimator is biased for discrete data?\n1. In Section 3.4, does your proposed methodology lead to 2 nested layers of Monte-Carlo, one for $\\theta$ and one for $X, Y | \\theta$?\n1. In Section 4, can you clarify the following sentence?\n> However, one could imagine a top-down approach, starting with a more general distribution PXY for which only unbiased estimators for the log-densities are available.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dU16Nbyk5E", "reviewer": "Reviewer_RQdA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- (S1) The manuscript makes a compelling case highlighting the limitations of existing benchmarks for MI estimation utilizing PMI profiles, and provides a way of generating more varied problems for MI estimation benchmarking. Of critical importance is the ability to study the robustness of the MI estimators to noisy observations, which the BMMs can nicely model, and provide a clean way of evaluating the robustness across a spectrum of noise.\n\n- (S2) The authors not only present the notions of PMI profile and BMMs, but also do a great job at highlighting how they can be useful both theoretically and empirically. The empirical evaluations highlight the different ways that BMMs can be used for either benchmarking MI estimators or directly for MI estimation with uncertainty estimates.\n\n- (S3) The authors do great job at presenting the ideas, the theoretical results, and the empirical case studies. Furthermore, the Appendix is well organized (and appropriately cited in the main paper) with many of the necessary details. There are many questions that I had while reading the main text that were very clearly answered when I looked in the Appendix. The Appendix also provides a very informative additional description and discussion regarding the MI estimators utilized, and their performances on the benchmark datasets, which is very useful in my opinion.\n\n- (S4) The authors provide a useful discussion of limitations and future work.\n\nWeaknesses:\n\n- (W1) While the PMI profile is an interesting concept, and BMMs can be used to generate data with varied PMI profiles (as opposed to a single PMI profile in existing MI estimation benchmarks), the connection between the structure of the PMI profile and the hardness of the MI estimation task is not made explicit. Without this connection (or at least some technical discussion), it is not clear why we need distributions with varying PMI profiles. For example, one can imagine that if the PMI profile is sub-Gaussian, the mean estimation (the mean being the MI) would be easier due to concentration to the mean than with a PMI profile which is not -- as exemplified in Fig 1, where one can imagine that MI estimation is significantly easier with the blue PMI profile regardless of the transformations (note that I do not think that this is clearly true, but just the form of technical justification/discussion that I think would motivate varying PMI profiles).\n\n- (W1a) Furthermore, it is not clear if real data where we need to estimate MI have varying PMI profiles -- it seems plausible that they would but some form of empirical validation would make the case for varying PMI profiles stronger.\n\n- (W2) The need for Monte Carlo methods to estimate the \"true\" MI and the PMI profile (up to some level of approximation) of BMMs makes their use in benchmarks a bit awkward. Given the need for such additional estimation (instead of having an analytically computable solution) brings in a computation-quality tradeoff. Depending on the dimensionality and structure of the data (the complexity of the \"bending\" and the number of components in the \"mixture\"), the true MI (or PMI profile) estimation can be quite computationally challenging, and could probably introduce error in the estimation of the \"ground-truth\" MI that will be used as the gold-standard to evaluate MI estimators, thereby bringing the conclusions of such evaluations into question. Beyond the guarantees of Monte Carlo estimators, there does not seem to be a good way of deciding how faithful the \"ground-truth MI\" is -- I think the guarantees are often additive, not multiplicative, that could require large number of sample when the ground-truth is very very close to zero.\n\nMinor comment:\n\n- In the example corresponding to Figure 3, I think this experiment (comparing the robustness of different MI estimators) could be conducted without use of BMMs or utilizing anything related to the PMI profile. I would think that the advantages of BMMs come into play when different diffeomorphisms would be applied to the data. Is my understanding flawed?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript introduces the notion of Pointwise Mutual Information Profile or PMI profile, and demonstrates that, much like Mutual Information or MI, the PMI profile is invariant to diffeomorphisms. Based on this insight, the manuscript highlights the limitations of MI estimation benchmarks that rely on generating varied distributions by transforming bivariate normal distributions since the PMI profile of the transformed problem remains the same. To generate MI estimation problems (for benchmarking MI estimation schemes) with varying PMI profiles, the manuscript introduces the Bend-and-Mix Models or BMMs that are mixtures of transformed closed-form distributions. While the PMI profile and the MI cannot be obtained analytically for the BMMs, the manuscript argues that these can be tractably estimated with Monte Carlo approaches.\n\nGiven these BMMs (and their corresponding PMI profiles), the manuscript highlights various interesting case studies. The first case study highlights the varying levels of performance of various MI estimation schemes on low dimensional data with different PMI profiles, including a few cases where none of the considered MI estimators perform well even in this low dimensional setting. Building upon this, the second case study highlights how BMMs can be used to study the robustness of existing MI estimators by mixing various forms (inliers vs outliers) and proportions of noise distributions. The third case study establishes a connection between the PMI and the neural critic functions used in existing variational MI estimators, and highlights the performance of the various estimators given the ground truth PMI as the neural critic under different conditions (biased vs unbiased critic, or higher dimensional data). Finally, the authors demonstrate how a family of BMMs can be used for the Bayesian estimation of the MI and the PMI profile, thereby providing uncertainty estimates for both the estimated MI and the estimated PMI profile, highlighting conditions under which this scheme can perform well or poorly.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- (RC1) A (theoretical or empirical) discussion motivating the need for varied PMI profiles (as discussed in W1 and W1a) would greatly improve this paper.\n\n- (RC2) Some discussion regarding the computational cost involved in having a benchmark distribution that requires additional step of MI and PMI profile estimation would be useful.\n\nBoth my requested changes are not critical, but will greatly improve the paper in my opinion.", "broader_impact_concerns": "The manuscript does not have a Broader Impact Statement, and I do not have any concerns regarding its ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "19qIdSGbQF", "reviewer": "Reviewer_oG5L", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**  \n1. The paper introduces important work, which could become significant for future benchmarking of MI estimators.\n2. The paper is generally very well written, and the figures are clear, understandable, and informative.\n3. The authors provide a useful implementation (and promise to make it open-source).\n\n\n**Weaknesses**  \n1. The paper consists of two parts, which appear largely disconnected to me: 1) the characterisation of the PMI profile, and 2) the introduction and evaluation of BMMs. Furthermore, each part seems too limited in scope at this point.\n\t1. Regarding the PMI profile, I don't fully understand why it is included in the paper. The characterisation is definitely interesting, and it seems obvious that there's a lot more information in the full profile than in the average (i.e., the MI), but its usefulness is never addressed. The profile is briefly mentioned in the experimental evaluation (Section 3), showing how well various methods can reconstruct it, but why is it important to reconstruct it at all? In which situations should we prefer the profile over its average?\n\t2. Regarding the BMMs, the idea is certainly interesting, but it also seems like a minor extension of Czyż et al. (2023), since this work already constructs a full benchmark based on diffeomorphisms. Extending this with mixtures is definitely interesting, but I would then expect a much more thorough evaluation, at least as comprehensive as the one in Czyż et al. (2023). The BMMs are also primarily used to estimate the MI, so I don't see the connection to the first part of the paper (the PMI profile).\n2. Defining an entirely new family of distributions (the BMMs) seems unnecessary and comes across as somewhat inflating the contributions. Their sole purpose is to define distributions that can easily be sampled from such that the MI can be estimated, and mixture models and normalising flows are perhaps the most obvious ways of creating such distributions. Applying these models to estimate the ground truth MI is interesting, but I don't see why it wouldn't be sufficient to simply propose mixtures and normalising flows as two examples of models that can be used for true MI estimation. \n3. As a smaller weakness, I think the paper could use a bit more comprehensive background (Section 2), for instance the basic definition of MI and why it is difficult to estimate. \n\n\n*References:*\n- Czyż et al., *Beyond Normal: On the Evaluation of Mutual Information Estimators*, NeurIPS 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper consists of two major parts, a characterisation of the pointwise mutual information (PMI) profile, and the definition and evaluation of a new family of models called *Bend and Mix Models* (BMMs).\n\nFor the PMI profile, the authors characterise it in greater detail than previously. Concretely, the authors establish that the profile is invariant under diffeomorphisms (thus generalising the invariance of the mutual information (MI)), and they further derive and characterise a closed-form expression for the PMI profile of jointly multivariate normal random variables. \n\nThe authors then introduce the BMMs, which combine transformations with diffeomorphisms and mixing to create new models for which the PMI profile can be efficiently estimated with Monte Carlo approximations. \n\nFinally, the usefulness of the BMMs is illustrated in four different scenarios: as benchmarks of MI estimators, as a way to assess the robustness of MI estimators to outliers and inliers, to assess the biases and sample efficiencies of neural critics, and to provide Bayesian estimates of both the MI and the PMI profile.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would like to see a greater connection between the two parts of the paper (the PMI profile and the BMMs), or at least a justification of why they both have to be in the same paper.\n\nSpecifically for the profile, importance-wise, it seems mostly a footnote in the paper despite being mentioned as one of the key contributions. I think this is a shame because it seems to be a powerful quantity and your characterisation important. I suggest updating the paper to clarify when and where the profile is particularly useful along with some practical examples where MI estimates aren't enough. Otherwise, I suggest toning down its significance, possibly moving it to the appendix entirely or, preferably, to its own paper. Similarly, the invariance result is really interesting, but not used for anything, as far as I can see.\n\nFor the BMMs, I think you should consider dropping the name and focusing instead on extending the benchmarks of Czyż et al. (2023) with your new distributions. I also think you should extend the analysis to include additional estimators, at least those of Czyż et al. (2023), and possible additional distributions too. Otherwise, the experiments you conduct (e.g., testing the estimators' robustness to misspecification as the dataset size grows, and how well they work in higher dimensions) are very interesting and great additions to those of Czyż et al. (2023).", "broader_impact_concerns": "I do not have any concerns about the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers unanimously advocated for this paper's publication following discussion with the authors. The authors have produced an improved version of their paper during this discussion paper and there were no remaining concerns from the reviewers.

One comment by a reviewer that I want to elevate at this point was that the authors have made a promise to open source their implementations of PMI. I mention it here to encourage the authors to follow-through on this.
Decision: Accept (poster)