Towards Robust Scale-Invariant Mutual Information Estimators
Authors: Cheuk Ting Leung
Rohan Ghosh
Mehul Motani
Venue: Accepted by TMLR ()

Abstract: Mutual information (MI) is hard to estimate for high dimensional data, and various estimators have been proposed over the years to tackle this problem. Here, we note that there exists another challenging problem, namely that many estimators of MI, which we denote as $I(X;T)$, are sensitive to scale, i.e., $I(X;\alpha T)\neq I(X;T)$ where $\alpha \in \mathbb{R}^{+}$. Although some normalization methods have been hinted at in previous works, there is no in-depth study of the problem. In this work, we study new normalization strategies for MI estimators to be scale-invariant, particularly for the Kraskov–Stögbauer–Grassberger (KSG) and the neural network-based MI (MINE) estimators. We provide theoretical and empirical results and show that the original un-normalized estimators are not scale-invariant and highlight the consequences of an estimator's scale-dependence. We propose new global normalization strategies that are tuned to the corresponding estimator and scale invariant. We compare our global normalization strategies to existing local normalization strategies and provide intuitive and empirical arguments to support the use of global normalization. Extensive experiments across multiple distributions and settings are conducted, and we find that our proposed variants KSG-Global-$L_{\infty}$ and MINE-Global-Corrected are most accurate within their respective approaches. Finally, we perform an information plane analysis of neural networks and observe clearer trends of fitting and compression using the normalized estimators compared to the original un-normalized estimators. Our work highlights the importance of scale awareness and global normalization in the MI estimation problem.

Published in Transactions on Machine Learning Research (07/2025)
Towards Robust Scale-Invariant Mutual Information Estimators
Cheuk Ting Leung"
lheukting.leunggu...o
Deportment of Electrical and Computer Engineering
College of Design and Engineering
National University of Singapore
Rohan Ghosh"
rghosh92@gmaiil.ooom com
Department of Electrical and Computer Engineering College of Design and Engineering
National University of Singapore
Mehul Motani
motaniOnus www.uuussuuu.... edu.sg
Department of Electrical and Computer Engineering College of Design and Engineering
Institute of Dala Science
N.1 Institute for Health
Institute for Digital Medicine (WisDM) National University of Singapore
Reviewed on OpenReview: https: oppnree/epeeeeeeee net/ forum? id- wBTWuy keo
Abstract
Mutual information (MI) is hard to estimate for high dimensional data, and various esti- mators have been proposed over the years to tackle this problem. Here, note that there exists another challenging problem, namely that many estimators of MI, which we denote as (X; are sensitive to scale, i.e. a(); # I(X;T) where a E RR+ Although some normalization methods have been hinted at in previous works, there no in-dopth study of the problem. In this work, we study new normalization strategies for MI estimators to be scale-invariant, particularly for the Kraskov Stögbauer-Graasbereee Grassberger (KSG) and the neural network- based MI (MINE) estimators. We provide thcoretical and empirical results and show that the original un-normalized estimators are not scale-inver invariant and highlight the consequences of estimator's s scale-dependence. We propose new global normalization strategies that are tuned to the corresponding estimator and scale invariant. We compare our global normalization strategics to existing local normalization strategics and provide intuitive and empirical arguments to support the use of global normalization. Extensive experiments across multiple distributions and scttings are conducted, and we find that our proposed variants KSG-Global-Lrr and their respective approaches. Finally, perform information plane analysis of neural net- works and observe clearer trends of fitting and compression using the normalized estimators compared to the original un-normalized estimators. Our work highlights the importance of scale awareness and global normalization the MI estimation problem.
are most accurate within
"Equal Contribution.

---

Published in Transactions on Machine Learning Research (07/2025)
1 Introduction
Mutual information (MI), is a fundamental measure of dependency between two variables, which has be- come pivotal in various machine learning domains, including generalization (Xu & Raginsky, 2017 Bu et al.,
2019 Russo & Zou 2020), representation learning (Bachman et al. 2019; [Tchannen et al. 2020) and
fairness (Wang et al. 2023 Roh et al., 2020). Estimating MI for high-dimensional continuous variables (Xu et al.) 2020) is particularly challenging, due to the hardness of accurately estimating the probabil- ity distribution in high dimensions &Goldfeld & Greenewald 2021). For example, traditional estimators like Kraskov Stögbauer Grassberger (KSG) (Kraskov et al. 2004). : rely on distance metrics, and for high dimensional data, the distances would have less variation due to the curse of dimensionality.
In this paper, we highlight an important but underexplored factor affecting MI estimation accuracy: the scale of the variables (IxI). Specifically, for mutual information I(X; where a R+ is scaling factor, we show that MI estimates become dependent on and tend to converge to low values at both extremes (a - 0+ and a + 00). This is problematic since, by definition, aT) I(; for any two continuous random variables (RVs), and more generally, I(X; = I() for any continuous and invertible transformation I (Cover & Thomas 2006). Our study focuses specifically on the impact of scale, revealing its significant role in MI estimation errors.
As we show in this work, commonly used MI estimators such as binning, KSG (Kraskov et al., 2004) and MINE (Belghazi et al., 2018) are not natively scale invariant. Typically this is addressed via normalization approaches. Despite numerous surveys that have explored various methods of MI estimation, some examples of which include [WaltersWillians & Li, 2009 McAllester & Stralos 2020: Paninski 2003), an in depth study of normalization approaches has been absent. A standard approach used in most works is local normalization, where each dimension is adjusted to have a variance of (Hjelm et al., 2019 Xie et al.) 2024 Kraskov et al. 2004). However, local normalization treats each dimension independently and normalizes them to have a variance of 1, which, as we demonstrate in Section 4.1.1 does not work well the high- dimension setting especially in neural networks, across two separate experiments. This is because most high-dimenssonall feature representations in neural networks always contain some noisy dimensions, which are of low energy and contain irrelevant features. We rigorously verify this in Appendix J Thus, amplifying these low energy dimensions can lead to suboptimal MI estimates. We also note that the recent work by (Czy et al. 2023), in addition to trying out local normalization approaches, also studied other preprocessing methods including the transformation of the margin distribution to uniform distribution (via converting to rank). We note that this conversion step also brings all individual dimensions to equal importance like local normalization, and thus would have the same pitfalls in this scenario.
To address this issue, in our work, we propose set of global normalization approaches. Unlike local normalization, global normalization preserves the relative energies between the different dimensions, and thus avoids scaling up low-energy noisy dimensions. Our proposed estimator modifications do not only include new normalization approaches, however, and often also have an additional maximization step, which helps bias our estimators better. It is well known that KSG and other MI estimators have a tendency to have negative bias Czyż et al. (003)), especially in high dimensions. Our normalization approaches for KSG incorporate this observation via an additional maximization step, which also follows intuitively from one of our theoretical observations in Proposition
We summarize our contributions as follows:
We theoretically show that native KSG and MINE estimators depend the scale of random variables. We propose scalo-invariant KSG and MINE extensions that resolve one-sided scale issues and significantly improve accuracy. Our analysis of normalization methods is, to our knowledge, the first of its kind.
We show that KSG-Global-T.e and MINE-GHobal-Coreetee consistently yield the most. accurate MI estimates across varied synthetic experiments, including high-dimensional, low-data setups.
We analyze MI dynamics between inputs and neural network layer outputs during training, finding that unnormalized estimators confound scale, whereas our methods reveal distinct training phases such as fitting and compression.
Note that a detailed background on MI Estimators studied in this work is presented in Appendix A Next, we provide a summary of related works that have either studied robustness of MI estimators to transformations,
2

---

Published in Transactions on Machine Learning Research (07/2025)
proposed transformation robust MI estimators. Note that more detailed discussion of what follows is available in Appendix B
1.1 Overview of Related Works
The notion of self-consistent equitability, discussed in (Reshef et al. 2013 Kinney & Atwal 2014). is more general study of invariance of MI estimators, but has not explicitly studied scaling due to preprocessing techniques enforcing scale invariance. These works assess invariance in MI estimates across a broad range of transformations, but our study differs by prioritizing MI estimation accuracy and testing cascaded trans- formations that modify dimensionality and structure. Unlike previous studies that apply one-dimensional transformations independently to each feature, our approach introduces richer transformations that could be further explored within equitability framework. Similarly, recent MI estimation studics such (Czy2 et al. 2023) mainly focus on dimension-wise transformations, whereas we consider transformations rele- vant to deep learning applications, such as sigmoid activation and random matrix multiplications, Prepro- cessing wisc, most works have focused on standard local normalization. While (Czy2 et al. 2023) finds Gaussinnizatinn-asse local normalization beneficial for long- tailed distributions, the improvements are mi- nor and they end up preferring standard local normalization. Notably, scale invariance absent. in their analysis due to preprocessing techniques that normalize local scales, whereas our study explicitly ensures scale-invariant MI estimates while maintaining robustness to noisy dimensions.
Neural network-based preprocessing strategies for MI estimation have also been explored in recent works. Gowri et al. (2024) propose a two-step MI estimation method where compressed representations are learned to minimize an upper bound on conditional entropy before applying KSG estimation. While their approach inherently involves scale variations due to neural network compression, they mitigate this using local unit normalization, and our findings on global normalization strategics can be applied to their framework potentially enhance performance. Similarly, Butakov et al. (2024) introduce an MI estimator based on normalizing flows, transforming data distributions into simpler forms with closed-form MI solutions. Their method primarily focuses on Gaussian base distributions, and while it may be inherently scale-invariant, no explicit theoretical analysis has confirmed this. Our work highlights the importance of scale invariance in MI estimation, emphasizing on preventing scale confounding in estimator outputs.
2 Motivation
Estimating MI is fundamental to various domains, ranging from learning theory to practical applications such as medical analysis and wireless commmication (Shwartz- Ziv & Tishby 2017 Saxe et al.) 2018). To motivate our proposed normalization strategy, this soction outlines several desirable properties that effective MI estimators should possess. Let S {(X1, T1), (X2, ...,... (X,,, T,)) be the sampled data. With this, let Te (T) represent an estimate of the MI between X and using the estimator est, given n sampled points from the joint distribution P(X,T), T). Ideally, we seek the estimator to have the following properties:
1. Global Scale Invariance: For any R+ and n E Zt, Tesl Tr (X:T) 2. One-Sided Scale Invariance For any a E R+ and Zt, Ie (X; aT) (X:
We emphasize the importance of these propertics because true mutual information inherently satisfies them. By definition, T(aX;aT) = T(X;T) and T(aX;T) = T(X;T) T) for a scalar a. In the case of neural net- works, where X represents the input, Y represents the target, and T represents the features, estimation of I(;;T) becomes important, as it was hypothesized that it can predict the generalization behavior of deep learning networks (Shwartz- Ziv & Tishby 2017). Furthermore, (Shwartz- Ziv & Tishby 2017) also predicts two-phase behavior of T(X;T) during training: (a) fitting, where T(X;T) and T(T; Y increases, and (b) compression where I();T) decreases. However, this is often not observed (Saxe et al. 2018). We hypothe- size that it could be because of the scale-sensitivity of the estimators, as the scale of T changes significantly during training.
We note that the current estimators may not obey one-sided scale invariance. First, we study three estimators theoretically: KSG, MINE, and binning.
3

---

Published in Transactions on Machine Learning Research (07 2025)
2.1 Testing One-sided Scale-Invariance of MI Estimators
In this section, introduce new theoretical results that analyze the global-scale invariant and one-sided scale invariant properties of commonly used MI estimators. Note that for all results that follow, we assume every RV is bounded. That is, if X is bounded, we have that |X] I B for some finite < co. Also, for the following results, let X E Rd and T E R". Also note that by Rt, we mean the set of all positive real numbers, excluding zero.
Binning: Let us denote the binning estimator described in Paninskk 2003 by I Then we have the following result.
Proposition 1. Let T (X;T) denote the mutual information estimated using fixed number of bins per dimension, with bin edges defined by the minimum and maximum values of the data. Then, for all a E Rt
Bin aax;aT)) Thin (X;T) & I (X; = Tin (X:T).
Remark 1. We note that even though the bimning estimator is scale-invariant, the native binning estimator is not a good estimator for MI, more so in the high dimension setting (Kraskov et al. 2004]. This is because in high dimensions the data occupies the space sparsely, and most bins will yield zero datapoints and thus a zero probability. However, in recent. years there have been variants proposed based on structured density estimation, which are more robust for use in high dimensional spaces. Nonetheless, we only investigate the KSG and MINE estimators in our work. Additional discussions statc-of-tho-art binning estimators is provided in Appendix A
KSG: Let us denote the KSG estimator proposed in (Kraskov et al. 2004] by Tkso. Then, we have the following results.
Proposition 2. holds that Iasa = (X; Va E R
Proposition 3. Let T.), be drawn i.i.d from a bounded distribution on Rdx xR that is absolutely continuous. Then, it holds almost surely that lima-so i (X; lim (X; = = Thus, in ( (X;oT) need not be equal (X;
MINE: We first define two variants of the MINE estimator as follows:
This estimator refers to the MINE estimator where instead of training the neural network MINE's loss function defined in 7 by stochastic gradient descent (SGD), we pick the best neural network configuration that directly maximizes Thus, we pick the global optimum.
MINE-Opt:
This estimator refers to the MINE estimator where optimization the loss function defined in performed using conventional stochastic gradient. descent. This is the standard approach proposod originally by (Belghazi et al. 2018).
MINE-SGD:
We denote the MINE-basee based MI estimators by PHINE-opp and TMSINE-Sa sgd We then have the following results.
Proposition 4. holds that ox (X; Tor (X;T) Va R
Next, outline theoretical result regarding the limiting behaviour of the first layer weights for the MINE estimator's neural network, when the scale of one of the variables approaches zero.
Proposition 5. Consider the MINE optimization problem with input data S = {(aX11 Yi), (aXn, Y.) where X E dd. Y E Rdd, (X, Y) ~ P(X, Y) are bounded RVs and a E R+ is a scaling factor. We consider a noural network of dopth + 1 having hu....... hy ReLU-activated hiddon ncurons in the respective layers. The network is trained via gradient descent on the MINE loss function in Belghazi el al. (2018) for a fixed number of iterations nt, with a learning rate schedule S 17(t) < oo for all t & nT. Let the weights between the tt node of the 1 th hidden layer and the jth node of the Ith hidden layer after t iterations be denoted by w/y (t). Assume that the initialized weights are bounded, i.c., V (1, (w), S for some > 0. Lastly, let [X]]X] denote the first layer weights that are attached to X. We then have, V(i.j),
lim [ se.
(1)
0 a1
4

---

Published in Transactions on Machine Learning Research (07/2025)
With this, we have the following result that explores scale invariance for MINE.
Proposition 6. We consider the same setting as Proposition 5 for the MINE estimation problem. There, it holds that lima-00 T MaaNE-sga(X;; a 0 Thus, T sga (X;T)) need not be equal to T Mid
(X;;) ())).
3
Methodology
3.1
Normalization Strategies
We consider setting where we are given an RV X [21, 12, 22, ...,, xa] E R2, where æ; E I represents the i-th component of X. Suppose X ~ P, where P is probability distribution, and let S X2...2...... ,,)) be independent and identically distributed (i.i.d.) sample drawn from P, with X, E Rd for = 1,..., With this, we outline three normalization strategics that form the basis of our studies in this work. We define them as follows.
Definition 1. (Local Normalization) The locally normalized variable Xays ./....,,........................ s'dl E Rd is defined by normalizing each dimension i individually as xa = I;-R;
for i = I,..., d, where
V (os) (o
the empirical mean is defined as: 5, Eド= Fiy-
Definition (Global Normalization)
The globally normalized variable Xzys E IRd is defined as Xzys
where ||-1|2 denotes the Σ |1 811
X-
Iz-norm, and X is the empirical mean: X = Xj- Definition (Global Tos Normalization)
The globolly Loe normallized variable Xx E Rd is X =
X- where || denotes the
2 [x, x1
Lorm and X is the empirical mean: X = -
Note that for any RV X, we denote by Xays and Xysss its locally and globally normalized versions respectively.
3.2 Studied Scale-Invariant Estimators
We are given the RVs X E Rd and T e RR", and sampled data S ((X1, Ti), (2,, (Xn, Tw)) ~ Pkr- All following estimates are for the MI between and T, given S. With this, we propose the following normalization approaches for KSG and MINE estimators. We outline our approaches for scale-invariant KSG and MINE extensions in Table
Table 1: Proposed Scale-Invarrant KSG and MINE variants
KSG
MINE
KSG-Local:
Ixso(Xols: s)
MINE-Local: IMINN(Xais: Tois)
KSG-Global:
max (((e).)mm.))))
(esso(XissiTess)
MINE-Global: ININN(X TeiS)
KSG-Global-Lo: ce(er.ca...mm)
max
[Ixso(XX=sssi cT
MINE-Global--Corrctedd MnInelVdxXeis VdyTass)
Remark 2. In addition to the above approaches, compare the standard baselines of KSG and MINE, Furthermore, we also include updated variant of KSG in our comparisons, called BI-KSG (Goo et al. 2017 which has smaller bias levels for highly correlated data. We do not include the native binning estimator in our experimental results, as we find that they fare poorly for almost all of our studied cases. Thus, we only study the KSG and MINE variants empirically in this work. Also, note that the range of multiplier scales C1 Cr...,,...c Cm are tunable hyperparameters, and we fix с₁ 0.1, C2 = ...,....co Cm for all our experiments. Note that for the KSG- KSG-Global variants, this does slow down the MI estimation process, it takes m times the computation time to estimate the measure when compared to KSG.
5

---

Published in Transactions on Machine Learning Research (07 (07/2025)
4 Experimental Studies
Our empirical studies can be categorized into roughly four broad sections:
Empirical motivation for proposed normalization variants (E1-E3): We provide in-depth em- pirical analyses for each normalization variant proposed this work, and also the overall reasons for potentially choosing global normalization approaches over local ones.
2. Scale dependence and Signal to Noise Ratio (SNR) analysis of estimators (E4,E5): We perform some basic tests and analyses of all estimators. First, study their overall responses to scale changes, and then we study their responses to changes in noise levels.
3. Accuracy analysis of estimators (E6): We conduct an extensive
analysis of all estimators in three different settings where ground truth MI is known. each setting, we generate syuthetic data using a diverse set of transformations to simulate different distribution scenarios.
Studying neural network training using cstimators (E7): We study the MI dynamics of neural networks during training. Specifically, we analyze the MI between inputs and features and compare the trends resulting from various estimators.
Note that the experiments are numbered from E1 to E7 for clarity. We use three base distributions for generating the RVs X and T. We refer to them in varions parts of the experiments. They are:
Correlated Gaussians: Here, we consider joint Gaussian distribution where (X,T) E ERd ~ ~N0, with covariance matrix:
la pla] pla Ia
Σ =
This ensures that IE[XT; p for <i d while [[X;T,] = 0 for +
Additive Gaussian Noise: Here X E Rd ~ NO, Ia) and T = X + e, where oN(0,
Correlated Student's t-distributions: We sample correlated X,T E Ed from multivariate Student's
t-distribution. Details on the sampling procedure are provided in Appendix D (same as (Czy2 et al.
2023
We used the NPEET MI estimator toolbox for estimating KSG and KSG-based measures [ For MINE, we used a pytorch-basee package Code for all our experiments available in the Supplementary Material.
The details for our estimators are provided in Appendix F
4.1 E1-E3: Additional Motivation for Normalization Variants
In this section, we provide both intuitive and empirical arguments for every aspect of our proposed variants in the previous section: (E1) why global normalization and not local, (E2) why maximization of MI for KSG and (E3) why the specific choice of global normalization variants: KSG-Global-Loo and MINE-Global- Corrected.
4.1.1 E1: Global Over Local
Our main observation that local normalization scales all variables equally, potentially overemphasizing irrelevant, low-energy dimensions, which degrades MI estimates. However, global normalization preserves the distance structure of the original RVs, which can benefit in high-dimensional settings, such as neural network feature representations, where many features in T' are sparse. We highlight specific cases for KSG and MINE below, where local normalization yields undesirable behaviour, which is avoided by global normalization.
KSG: Consider RVs X,7 T E TR? where = + with e ~~ o2)) We augment. X with k independent noise components = [q...... .e.....Gu 4), G ~ -N(0, d²2) with o' < a, forming X' [X, c]. Although I(X; T) I('':T), as shown in Figure la KSG and global normalization maintain stable MI estimates with increasing k, while local normalization underestimates MI by over-scaling the noise.
MINE: For MINE, we generate correlated Gaussian RVs X,T E RR3 (with a random correlation E (0. 0.8)) and similarly exlend X with noise to form X'. Averaging over 10 trials, Figure [b reveals that local
https///gtthub..cmmm/ggggerr wgw.eeettteggggeeegeeeooooooo /Nttetteeee///ETTTT.. /NPEET htapss//gtthabb.cogggggge///////

---

Published in Transactions on Machine Learning Research (07 (07/2023) /025)
1.2
Average MI Estimates
MINE
MINE-Local
MINE.Glibal-Corrddee
0.8
KSG
0.06
MINE-Gllbbl
0.6
KSG GS5--lobbbkL.
Bias
0.04
KS(-Rovied
0.4
0.00
0.2
-0.02
0.0
50
100
150
10
20
30
# Noise-Dimensions
Choise-Dimensions
(a) Average MI estimates (T(X:T') for KSG-based measures for varying noise dimensions (k)..
(b) Bias of MINE-based measures for varying noise dimensions.
Figure (E1) Comparative evaluation of MI estimators (KSG and MINE) under increasing noise dimensions. MI should be unchanged, but local normalization significantly impacts estimates in both cases.
1.0
100
Average MI Estimates
Bias Reference
-er--Esimatton %
BI-KSG
0.6
KSG
25
0.2
BI-KSG
7
9
2
3
4
5
d
log(d)
Figure 2: (E2) Dependency of KSG estimator bias on data dimension (using base-2 log). We find that dimension- ality increases, KSG increasingly underestimates the true MI.
normalization causes the bias to increase significantly with k, whereas global normalization yields stable estimates. This occurs because, for MINE, the network tends to overfit the added noise when each variable is normalized equally, while global variants preserve the effective input dimensionality.
4.1.2 E2: KSG-Global: Why the Maximization Step?
We use maximization step in our global normalization estimator for KSG in Table 1 We summarize two main arguments for our maximization step:
The KSG estimator shows increasing negative bias as data dimensionality grows. In one experiment, correlated Gaussians X,T Ted with fixed ground truth MI (s=0.8 were sampled (20 trials of 1000 points each) while d ranged from to 9, yielding clear negative bias in the estimators as in Figure a). A second experiment with d {2, 8, 16, 32, 64) and randomly chosen correlation P confirmed that the fraction of MI estimates below the ground truth rises significantly with d (Figure bb). Taking the maximum estimate over scales helps mitigate this bias. 2. Consequence of Proposition 3 Proposition 3 shows that for I(X;oT), the KSG estimator converges to negative values at extreme scales a. This observation motivates taking the maximum over range of scales for the globally normalized variables, i.c., sc TTjs). Empirically, the estimator values exhibit an approximately Gaussian trend with respoct (sce Figure 4a), making the maximum both well-defined and meaningful.
1. Negative Bias in High Dimensions:

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Remark 3. Note that MINE implicitly maximizes over relative scales via the arbitrary first-layer weights. Since scaling these weights adjusts the effective input similarly (i.e., (aW)*X = (ax)))), the network optimizes over affine transformations. Nonetheless, due to the tendency of gradient descent towards flatter minima Keskar et al. (2017). the full benefit of this invariance may not be achieved.
4.1.3 E3: Motivation for Global Normalization Variants
Next, we motivate our proposed global normalization variants: Corrected.
KSG-Global-Las and MINE-Global-
1.2
Average MI Estimates
1.1
1.
0.2
Blas
-0.4
KSG
MINE CoNe-Gl
KSG-Global
-066
0.7
KSC--RRaE-L_.
MINE-Global
True MI
Zero Blas
0.6
0.8
20
40
60
80
2
# Duplicate-Dimensions
d
(a)
(b)
Figure 3: (E3) Analysis of normalization variants: (a) Impact data duplication for KSG-based approaches, and (b) Estimator bias for MINE and MINE-Global variants across dimensions. These results highlight the importance of our proposed global normalization variants.
We observe that since KSG computes distances using the Lor-n when dx >> dr, global nor- malization makes the individual dimensions of X significantly smaller than those of T, yielding smaller Loo distances. This motivates the proposed KSG-Global-Leg approach, which scales based on Loo norm distances instead of L2 norm. To illustrate its impact, duplicate X to form X' X.., (thus 7( = T(X: 7)). As shown in Figure 3a KSG-Global actually decreases its estimate with increasing duplicates, whereas both KSG and KSG-Global-Lie maintain consistent MI estimates. This confirms the necessity of employing the norm for scaling the variables using the global normalization approach.
KSG:
Global normalization can lead to low per-dimension energy when dx > dr, since E[Xx)s(1))) 1/dx versus [i) 1/dr. This imbalance may cause gradient descent to focus predominantly on T. To counter this, we rescale the normalized variables that. E[X E (j) 1, i.e., X /dx Xxis (and similarly for T). Experiments with d ranging from to 9 (Figure 3b) reveal that while MINE-Global's bias grows more negalive with higher d, MINE-Global-Corres to standard MINE. This demonstrates that rescaling effectively mitigates the effects of imbalances in terms of per-dimension energy of X and T.
MINE:
has bias levels comparable
4.2 (E4-E5) Scale and SNR Analysis
In this analysis, we test the scale invariance properties of all estimators, and subsequently test their response to changes in noise levels (SNR).
Scale: Experiments were conducted to study the effect of scaling on MI estimators [(πX; using correlated Gaussian variables X,T. T. Scaling factors were sampled logarithmically between 10-2 and 103 for KSG, and 1 to 10 for MINE, as MINE estimates degrade sharply beyond r) = 10. Results (Figure 4a 4bb show KSG
estimates converging to -0.33 (matching Proposition 3 1j increases, while MINE estimates drop to zero
8

---

Published in Transactions on Machine Learning Research (07 (075))25)
77 (supporting Proposition 6 Global and local normalization variants remained robust to scaling, with significantly lower RMSE compared to vanilla estimators (Figure 4c 4d
SNR: MI estimators were tested with Signal-to-Noise Ratios (SNR) ranging from to 5, where T X + (e (e~~N NVO, o²)) and scaled to T = 0.17 (estimating ()); We expect the MI estimates to increase with SNR, following the ground truth. Scale invariance could be important here, as the noise variance a and the final scale of 7" will be related, and thus any estimator bias accompanying large scales can adversely affect the trend. Results (Figure 5a 5bb show global and local variants accurately following ground truth MI trends, increasing with SNR. However, vanilla KSG and MINE estimators failed to reflect the true MI trend, with MINE estimators even stabilizing at higher SNR values due to their inherent scale dependence.
4.3 E6: Comparing MI Estimators: Error Analysis
In this section, undergo a comprehensive series of experiments, where we compute various error measures of all estimators on a diverse range of datasets.
4.3.1
Experiment Summary
To create these datascts, we follow the three base distributions described in the beginning of this section. First, we generate X,T T according the base distributions: Additive Gaussian, Correlated Gaussian and Correlated Student's- 'st as defined in Section Then, we then make X undergo some (or none) of the following transformations, which are all MI preserving. For what follows, let X IRd and T E Rd.
Datasct ercation:
Randmat (rm): X' aW X, where a Un, 1) and W gaxd where (ii,j) Unif(0, 1). Unif(a,b) b) denotes uniform distribution over b). If the randomly generated W is not invertible, repeat the sampling process until we get an invertible W.
2. Cube (cb): X' = X o X X, where o denotes element wise multiplication (Hadamard Product).
3. Sigmoid (sg): X' = oX),, where a .Rd + Rd such that [' XTe-XX], where X[i] denotes the 8th
dimension of X and similarly for X'.
4. Duplicate-self (ds): X = [X, X, x] kd We set K = 20 in our experiments.
Average MI Estimates
1.00
Average MI Estimates
0.3
0.2
0.50
KSG-ChAebLL ReG-Rtereed Tran MI
0.00
SINE
MINE-Last MINEGGhba-Crrrest
Iiu
2.0
0.0
(ogio(1) (Sealing Factor)
logio(n) (Scaling Factor)
(a) Scale Trend: KSG
(b) Scale Trend: MINE
Estimation Grvor (RMSE)
KSG
Estimation Error (RMSE)
MINE-L
0.6
0..
MINE Global Contected
044
02
0.2
0.s
log10 (n) (Sealing Factor)
log10 (m) (Scaling Factor)
(c) Error v/s Scale: KSG
(d) Error v/s Scale: MINE
Figure (E4) Analysis of Estimators in response to data scaling. Estimates are for [(9:;T), where ry the scaling factor. These results clearly demonstrate the impact of scale on MI estimates.
9

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Average MI Estimates
Average Extimates
MINE
NHre-Tacal KSG-Gaobal BEASSG 1.0
MINE-Tacal MINE.GbbaL Tree MI
0.5
0.9
SNR
SNR
(a) SNR Trend: KSG
(b) SNR Trend: MINE
Figure (E5) MI estimates varying Sigual-to-Noise Ratios (SNR). Estimates for (X;;T() where T" is scaled by 1) = 0.1. Normalized variants reflect the SNR trends more accurately, whereas unnormalized estimators may not.
Duplicato-noise (dn): X' c] Rd+h where C2, C2,,.., where e, N (0) We set o' 0.2 and k 20.
Our objective is to evaluate the accuracy of the cstimation of I('
Performance Measures: We study three different measures of performance our experiments. For what follows, let fi A2......... fü denote the estimated values of MI for any estimator across k trials, and let P1, P1. Mr denote the ground truth values. The performance measures usod in our evaluation are: (a) Normalized RMSE: We first estimate the RMSE RMSE 14) v V - 14)3]. Then we estimate baseline RMSE as RMSE Base( = V m,) With this, we can estimate the final measure as: RMSE(R.p)
RMSE RMSF__Norm{ /1) RMSE Base(u)) (b) Spearman Correlation: The Spearman correlation measures the degree monotonic relationship between fa and μ (Zar 2005p. This is estimated as the Pearson's correlation coefficient between the rank values of j and μ.. (c) Bias: We estimate the bias as E, [14 (a.).
Evaluation Process: For each experiment, specific MI-preserving transformation (listed in the first column of Tables and 3) is applied to X. Across 40 trials, generate N 1000 samples of X, T ~ P(X, T), apply the transformation to obtain X', and estimate T(X':T T). Performance metrics such as normalized RMSE, Spearman correlation, and bias are computed. Red entries indicate cases with RMSE > 1, though they often still exhibit strong Spearman correlation with the true MI. Note that here we only discuss the results for the additive Gaussian noise base (see Appendix for full results).
Table 2: (E6) Normalized RMSE of KSG-Based Estimators: Additive Gaussian Noise Base Transformation
KSG-Based Mcasures
rm
eb
Sg
ds
dn
ksg
bi-ksg
ksg-loc
ksg-glo
ksg-glo-
0.351
0.404
0.147
0.208
0.110
0.286
0.335
0.060
0.084
0.050
0.458
0.533
0.145
0.113
0.113
1.275
1.424
0.312
0.300
0.301
0.862
0.932
0.445
0.594
0.396
4
0.332
0.342
0.304
0.520
0.297
4
0.332
0.342
1.327
0.298
0.297
1.131
1,233
0.977
0.959
0.931
1.275
1.424
1.334
0.300
0.301
1.981
2.129
1.904
1.021
1.021
1.983
2.131
0.816
0.811
0.812
1.377
1.408
1.343
1.605
1.290
1.643
1.730
1.275
1.660
1.153
6
1.983
2.131
1.905
0.814
0.816
10

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Remark 4. Since every transformation is MI preserving, combining them yields distributions whose ground truth MI is unchanged. This flexible framework allows to simulate high-dimensional data (up to 200 dimensions) with low intrinsic dimension (typically <10), reflecting the characteristies of neural network features. Our choice of transformations motivated by the behaviors observed under different normalization strategies for example, local normalization adversely affected by added noise (duplicate-noise), while KSG-Global struggles with duplicate-self, and nonlincar transformations (e.g., sigmoid and cube) can alter nearest neighbor distances.
4.3.2 Takeaways
Overall, global and local normalization variants fare significantly better than the baseline measures. and KSG-Global-Lzs) overall fare better than other normalization strategics. fact, when for the additive Gaussian noise basc, we find that in most and KSG-global-L>s outperform other normalization approaches. KSG-Global-Tre shows consistent improvements throughout. Even when the normalized RMSE estimates are insignificant (red entries), KSG-Global shows significant correlation with true MI in many of the cases (see Appendix G
Our global normalization variants
cases MINE-globallcorrcee
Aligning with our earlier discussions, we find that overall the global normalization variants IMINE-Global Corrected and KSG-Global-L.a, perform better than their vanilla global normalization counterparts. This much more apparent in the case of MINE.
4.4 E7: Application of MI Estimations in Deep Learning
Mutual information is a key measure for analyzing neural network behavior during training. We evaluate MI IB (Shwartz-Zii Ziv & Tishby! 2017), MNIST (Deng) 2012), CIFAR-10 (Krizhevsky & Hinton 2009) and SVHN (Netzer et al. 2011 datasets. werw architectures, activations, and other details are in Appendix F For each classification dataset {x, } networks are trained and an intermediate layer's output Z is extracted (third layer for IB and MNIST; Global Average Pooling for CIFAR-10 and SVHN). MI estimates I(X; Z) and I (Z:Y) are computed using KSG, LSGal, MINE, MINE-Local, and our proposed KSG-Global-L, estimators.
and MINE-Global-Corecee
We analyze the MI estimates from two perspectives:
Training Dynamics: In Figure G plot I(X; Z) over epochs (averaged over 10 trials), along with the scale [Z] (in blue). On MNIST and CIFAR-10, the original KSG estimates strongly follow the feature scale, indicating sensitivity to scaling. In contrast, KSG-Global-Las does not mimic the scale curve; for
Table 3: (E6) Normalized RMSE of MINE-Based Estimators: Additive Gaussian Noise Base
Transformation
MINE-Based Mcasures
d
rm
cb
ds dn
mine
mine-loc
mine-glo
mine-glo-corr
2
0.470
0.292
0.337
0.278
2
0.445
0.255
0.275
0.233
2
1.036
0.560
0.658
0.565
4
1.302
0.720
0.968
0.684
4
0.930
0.438
0.803
0.381
4
0.276
0.369
0.642
0.375
4
0.622
0.423
0.895
0.269
4
1.574
1.099
1.219
1.162
4
1.335
0.423
0.895
0.286
6 1.881
0.570
1.516
0.437
6
1.843
1.147
1.535
1.185
6
1.213
0.991
1.444
1.004
6 1.014
1.088
1.441
1.058
6
1.831
0.517
1.499
0.545
11

---

Published in Transactions on Machine Learning Research (07/202) 2025)
0.7
3.0
1.
3.5
00-
150
2.0
0.05
2.5
Scala
VXG2)
0.4
15
Scale
100
0.04
1,0
0.75
0.00
1.5
15-
0.-
10-
0.25
0.6
-02
500
1000 Epochs
1500
2000
0.5
100
150
200
100
2000
300
400
100
150
0.00
Epochs
Epochs
Epochs
KSG Glocal L-
KSC-Loral
(a) KSG
0.07
75
7.0
0.06
14
8
0.05
1.2
Scaale
3.94
0.8
5.0
10
3.02
0.01
5021
1000
1360
2000
50
100
150
200
2001
0001
MI0 Epochs
5013
1000
01
50
100
1501
0.00
Epochs
Epochs
Epochs
mine-Globall-Cor
mine-Local
mine
(b) MINE
Figure 6: (E7) 1(X; 2) measures estimated after every epoch of training on IB, MNIST, CIEAR-10 and SVHN datasets. Z represents the output of 3rd layer for IB dataset and MNIST dataset, and 7th layer for CIFAR-10 and SVHN datasets. The scale of the features ([Z) is plotted in blue.
IB and CIFAR-10, it first increases and then decreases (with CIFAR-10 showing a drop after 3 epochs), consistent with the fiting-comppessson trend of (Shwartz-Ziv & Tishby
2017).
Information Plane Visualization: We also plot I(X; 2) vs. I(Z;Y) (in Appendix illustrate the
trade-off between input representation and label relevance.
Figure e reveals distinct behaviors of the proposed MI estimators across datasets:
IB and MNIST: On the IB dataset, both local and global variants of KSG and MINE estimators successfully exhibit the fitting (increase in I(X; 2)) and compression (decrease in I [(x; 2)) phases during training, consistent with the information bottleneck theory (Shwartz- Ziy & Tishby 2017). However, on the MNIST dataset, only the global variants, ASG-Global-Les and MINE-Gllball Global- Corbbeted,rrredd, clearly demonstrate these trends, while the vanilla variants primarily track the feature scale ([Zl) rather than capturing intrinsic mutual information dynamies. This suggests that. the normalized variants, which have been tested in the previous section accuracy-wise, can yield different I [XX; 2) trends than their vanilla counterparts.
CIFAR-10 and SVHN: For CIFAR-10, an interesting trend emerges. Vanilla KSG and MINE estimators show fitting and compression phases, but the fitting phase appears to strongly correlate with initial scale jumps of [Z], indicating sensitivity to feature scale. In contrast, the normalized variants predominantly display compression-like trends throughout training, highlighting a different behavior. Notably, both KSG- Global-Loe and MINE-Global- reveal a compression phase from the start.
In SVHN,
that the unnormalized KSG and MINE both fail to reveal any significant trend and are very close to zero throughout. In contrast both the local and global variants reveal some interesting trends and differences. For both KSG and MINE, global variants suggest that there is no explicit fitting phase (increasing T(X; Z)), but rather a slow compression phase throughout. Local variants are less consistent in their trends and they don't capture this slow compression mechanism.
12

---

Published in Transactions on Machine Learning Research (07 (025)
5 Conclusion and Limitations
We conducted comprehensive study scale invariance in MI estimators, analyzing its impact on estima- tion accuracy and neural network training. We proposed multiple normalization approaches for KSG and MINE, evaluating their effectiveness in high-dimensional, low-data regimes. Extensive experiments across diverse settings showed that while both local and global normalization have strengths, global variants gen- crally perform better. Finally, MI dynamics during training were analyzed on real datascts, where global normalization variants revealed interesting dynamics different from other normalized and unnormalized ap- proaches. Overall, our work highlights the importance of scale-awareness in the problem of MI estimation, and its potential impact MI estimates.
Limitations: While KSG-Global normalization variants demonstrate superior performance, their computa- tional cost is notably higher, as the MI estimation process requires m times the computation time compared to standard KSG variants. Furthermore, the intrinsic dimensionality of the data remains a fundamental performance bottleneck for these estimators, in spite of normalization, and nonlinear diffeomorphisms such as the cube function generally lead to lower accuracy. We see that while global normalization and its variants remain the better choice in most settings, heavy-tailed distributions do impact performance regardless of choice of normalization. Lastly, while in this work we focus mainly KSG and MINE, in recent years other estimators have been proposed and popularized, including variants of bimning estimators (e.g., Anandkumar ct al. (014b);; Vandermeulen & Lodent (2021b)) and other neural network based estimation approaches (e.g-, InfoNCE Oord et al. (2018)). We are considering the investigation of normalizat ion methods for these estimators for future work.
Acknowledgements
This research is supported by A STAR,, CISCO Systems (USA) Pte. Ltd and the National University of Sin- gapore under its Cisco-NUS Accelerated Digital Economy Corporate Laboratory (Award 121001E0002). We also thank the Kent-Ridge AT research group at the National University of Singapore for helpful discussions.
References
Milton Abramowitz. Handbook of Mathematical Functions, With Formulas, Graphs, and Mathematical Ta- bles. Dover Publications, Inc., USA, 1974. ISBN 0-486-61272-4.
Magda Amiridi, Nikos Kargas, and Nicholas D Sidiropoulos. Low-rank characteristic tensor density estima- tion part i: Foundations. IEEE Transactions Signal Processing, 70:2654 2668, 2022a.
Magda Amiridi, Nikos Kargas, and Nicholas D Sidiropoulos. Low-rank characteristic tensor density esti- mation part ii: Compression and latent density estimation. IEFE Transactions on Signal Processing, 70: 2669 2680, 2022b.
Animashree Anandkumar, Rong Ge, Daniel J. Hsu, Sham M. Kakade, and Matus Telgarsky. Tensor decom- positions for learning latent variable models. J. Mach. Learn. Res., 15(1):2773 2832, 2014a.
Animashree Anandkumar, Rong Ge, Daniel Hsu, Sham M Kakade, Matus Telgarsky, al. Tensor decom- positions for learning latent variable models. J. Mach. Leurn. Res., 15(1):2773 2832, 2014b.
Reinaldo B Arellano-Valle and Adelchi Azzalini. The centred parameterization and related quantities of the skew-t distribution. Journal of Multivariate Analysis, 113:73-99 90, 2013.
Philip Bachman al. Learning representations by maximizing mutual information across views. In H. Wal- lach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Mohamed Ishmael Belghazi et al. Mutual information neural estimation. In Jennifer Dy and Andreas Krause (eds.). International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. $31-540. PMLR, July 2018.
13

---

Published in Transactions on Machine Learning Research (07/2025)
Yuheng Bu, Shaofeng Zou, and Venugopal V. Veeravalli. Tightening mutual information based bounds generalization error. In IEEE International Symposium on Information. Theory, pp. 587-591, 2019.
Ivan Butakov, Alexander Tolmachev, Sofia Malanchuk, Anna Neopryatnaya, Alexey Frolov, and Kirill An- dreev. Information bottleneck analysis of deep neural networks via lossy compression. In International Conference on Learning Represenlations, 2024.
Thomas M. Cover and Joy A. Thomas. Elements of Information Theory. Wiley- Ieley-nntersccence,, USA, 2006. ISBN 0-471-24195-4.
Pawel Czyž al. Beyond normal: On the evaluation of mutual information estimators. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Li Deng. The MNIST database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-142, 2012. Publisher: IEFE.
M. D. Donsker and S. R.S. Varadhan. Asymptotic evaluation of certain markov process expectations for large time. IV. Communications on Pure and Applied Mathematics, 36(2):183-115 March 1983. ISSN 0010-3640. Publisher: Wiley-Liss Inc.
Shuyang Gao et al. Efficient Estimation Mutual Information for Strongly Dependent Variables. In Guy Lebanon and S. V. N. Vishwanathan (eds.), Proceedings of the Fighteenth International Conference on Artificial Intelligence and Statistics, volume 38 of Proceedings of Machine Learning Research, pp. 277 286, San Diego, California, USA, May 2015. PMLR.
Weihao Gao, Sewoong Oh, and Pramod Viswanath. Demystifying fixed k-nearest neighbor information estimators. In IEFE International Symposium on Information Theory, pp. 1267-1271, 2017.
Ziv Goldfeld and Kristjan Greenewald. Sliced mutual information: A scalable measure of statistical depen- dence. Advances in Newral Information Processing Systems, 34:17567-17578, 2021.
Gokul Gowri, Xiaokang Lun, Allon M Klein, and Peng Yin. Approximating mutual information of high- dimensional variables using learned representations. In Neural Information Processing Systems (NewrIPS), 2024.
R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchilee Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2019.
Nitish Shirish Keskar, Dheovatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Con- ference on Learning Representations, 2017.
Justin B. Kinney and Gurinder S. Atwal. Equitability, mutual information, and the maximal information coefficient. Proceedings of the National Academy of Sciences, 111(())3354 3359, 2014.
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physicaal Review E, 69(6):066138, June 2004. Publisher: American Physical Society.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tochnical report, University of Toronto, Toronto, Ontario, 2009.
David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information. In Silvia Chiappa and Roberto Calandra (ods.), International Conference on Artificial Intelligence and Statistics. volume 108 of Proceedings of Machine Learning Research, pp. 875-884, August 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Roading digits natural images with unsupervised feature learning. In NIPS workshop on deep learning and umsupervised feature learning, volume 2011, pp. 4. Granada, 2011.
14

---

Published in Transactions on Machine Learning Research (07/2025)
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arxi 1111188077744 2018.
Liam Paninski. Estimation of entropy and mutual information. Neural computation, 15(6):1191-122) 2003. Publisher: MIT Press.
Dávid Pál, Barnabás Póczos, and Csaba Szepesvári. Estimation of Rényi entropy and mutual information based on generalized nearest-neighbor graphs. In Advances in Neural Information Processing Systems, volume 23. Curran Associates, Inc., 2010.
David N. Reshef et al. Equitability analysis of the maximal information coefficient, with comparisons. arXiv preprint Xr::::1733.0000 00810, 1301.631,, 2013.
Yuji Roh, Kangwook Lee, Steven Euijong Whang, and Changho Suh. Fr-train: mutual information-baseel approach to fair and robust training. In International Conference on Machine Learning, 2020.
Daniel Russo and James Zou. How much does your data exploration overfit? controlling bias via information usage. IEEE Transactions on Information Theory, 66(1):302 323, 2020.
Andrew Michael Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan Daniel Tracey, and David Daniel Cox. On the information bottleneck theory of deep learning. In International Conference on Learning Representations, 2018.
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint
00810, 2017.
Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estima- tors. arXiv preprint arxiv 1:Xii:911000800 06222, 2019.
Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual infor- mation maximization for representation learning. In 8th International Conference on Leurning Represen- tations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Robert A. Vandermeulen and Antoine Ledent. Beyond smoothness: Incorporating low-rank analysis into nonparametric density estimation. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NewrIPS 2021, December 6-14, 2021, virtual, pp. 12180-12193, 2021a.
Robert A Vandermeulen and Antoine Ledent. Beyond smoothness: Incorporating low-rank analysis into nonparametric density estimation. Advances in Neural Information Processing Systems, 34:12180-122 12193, 2021b.
Robert A Vandermeulen, Wai Ming Tai, and Bryon Aragam. Dinension-inndepeneet rates for structured neural density estimation. arXiv preprint Xrx:::2411. 15,1509,, 2024.
Janett Walters- Williams and Yan Li. Estimation of Mutual Information: A Survey. In Peng Wen, Yuefeng Tú, Lech Polkowski, Yiyu Yao, Shusaku Tsumoto, and Guoyin Wang (eds.), Rough Sets and Knowledge Technology, pp. 389 396, Berlin, Heidelberg, 2009. Springer Berlin Heidelberg, ISBN 778-3-642-02962-11
Rui Wang, Pengyu Cheng, and Ricardo Henao. Toward fairness in text generation via mutual information minimization based on importance sampling, In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent (eds.). Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pp. 4473-4485 PMLR, 25-27 Apr 2023.
Yaochen Xic, Ziqian Xie, Sheikh Muhammad Saiful Islam, Degui Zhi, and Shuiwang Ji. Genetic InfoMax: Exploring Mutual Information Maximization in High-Dimensional Imaging Genetics Studies. Trans. Mach. Learn. Res., 2024, 2024.
15

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Aolin Xu and Maxim Raginsky. Information-theoreta analysis of generalization capability of learning algo- rithms. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable information under computational constraints. In International Conference Learning Representations, 2020.
Jerrold H Zar. Spearman rank correlation. Encyclopedia of biostatistics, 7, 2005. Publisher: Wiley Online Library.
16

---

Published in Transactions on Machine Learning Research (07 2025)
A Background on Mutual Information Estimators
A.1 Mutual Information
Mutual information of two variables is statistical measure that quantifies the mutual dependence wee two RVs. Specifically, it measures the amount of information obtained about one random variable through the observation of another. To understand mutual information, it is essential to first examine another foundationa concept, Shannon entropy. Shannon entropy represents the intrinsic informational uncortainty associated with a probabilistic system. Given a continuous RV X with a probability density function from X, the continuous entropy h(X) is defined as:
h(X) = J f(x) log fo)) f())
(2)
Then, the mutual information between continuous RVs X and Y is given by:
T(X;Y)= = h(x) hYY) h(X,Y),
(3)
where h(X,Y) represents the joint differential entropy of X and Y, defined as h(X,Y) Jx (x, y) log (x, v) drdy. Mutual information be interpreted the reduction in the uncertainty of X due to the knowledge of Y, equivalently, as the amount of information that X and Y share.
In the case of jointly continuous RVs, the mutual information can be expressed in terms of Leibler (KT-) divergence
I(x;Y) = DRL
P(Y)),
(4)
where P(X) @ P(Y) is the product of two marginal distributions P(X) and P(Y), P(X,Y) is their joint distribution. DKL is defined as
(kl(P))Q) Ep log dP dQ
(5)
In practice, estimating the true distribution of continuous RVs is challenging, especially for high-dimensional data. In the following section, we will discuss various non-parametric MT estimators, which estimate the distribution of RVs and subsequently compute estimated mutual information.
A.2 Mutual Information Estimators
The overall setting of the MI estimation problem is as follows. We are given two RVs X,T ~ P(X,T) and sampled data S T))..... (Xn,, T.)). Our estimators are denoted in the form ih (X; where est denotes the name of the estimator.
In this section, present several widely-used nonparametric MI estimators that studied in work and have been extensively applied in other research.
Binning Estimator: Also known as histogram based estimators, this method represents the most direct approach for estimating mutual information. To estimate MI, the continuous random variable is discretized into bins, counting the number of samples that fall into each bin, and computing the probability density (Paninski, 2003). There are many approaches literature for estimating the bin edges. However, our the- oretical analysis, we consider a simple approach that places fixed number of bins per dimension, according to the extrema of the individual dimensions.
Generally, the binning estimator for n samples can be expressed as: Tin (X;T) = (X) + Hosn(T) Anin (X, T),. where Hoin (X) represents the binned entropy given a RV X, such that Hsin(X] - Σ. P(X.) log P(X,). Let n(X;) be the number of samples that fall in the 7th bin of X. Then we have P(X;) ~ nn(x))nn for the histogram based estimator. Similarly, we represent binned joint entropy
Amin X, T) = - -ZE P(X;, T5) log P(X;, T;); and P(X,, T3) n n(X,, /2)/n.
17

---

Published in Transactions on Machine Learning Research (07/2025)
While histogram- based estimators are widely acknowledged for their simplicity and lack of scale sensitiv- ity, their vulnerability to the curse of dimensionality limits their applicability in high-dimensional settings. However, recent work in structured density estimation has proposed histogram-like methods that circumvent these limitations by imposing structural assumptions on the underlying donsity, such as low intrinsic dimen- sion. For instance, low-rank tensor decompositions (Anandkunar el al.) 2014a; Vandermeulen & Ledent, 2021a; Amiridi et al. 2022a b) enable scalable estimation by capturing latent low-dimensional structure, while learnable neural architectures (Vandermeulen et al.) 2024) flexibly adapt to complex dependencies. These structured approaches highlight the importance of using inductive biases to lessen the curse of di- mensionality when estimating the distribution via histogram-based approaches. Though such models require additional assumptions, they offer promising solutions for histogram-based MI estimation.
(KSG) Estimator: Another popular non non-parametric approach to estimate MI in high dimensions the KSG estimator in (Kraskov et al.) 2004). Unlike the binning estimator, the KSG estimator uses the k-ncarest neighbor (K-NN) statistic estimate the probability function of continuous RVs, which also uses the joint entropy decomposition method to estimate MI. The KSG estimator effectively uses the k-nearest neighbor distances to estimate the various entropies involved in the joint-entropy decomposition of TT) We will need some preroquisites to define the KSG estimator. First, let the k-NN distance Pk.í. be defined as the distance from (X,, T,) to k-th nearest neighbor in the joint space (X,T) as measured in hp distance. We now denole Va,ipp = Ejfi { |1 Xillp S PM.A. as the number of neighbors of the i-th sample X; within a specified distance under the Ip norm, and similarly for T as nt,i.p- Eventually, the KSG estimator yields the following estimate of T(::T):
Saasskov-Stöbbuur---Ce
--- Σ (#n(),ii)))) + (nn,,,,)))) n
so -x()) (n)) - k
(6)
where (x) is the digamma function (i.e., (xx) = (()) (x)
In (aoo et al., 2017). authors proposed a bias-improved KSG (BI-KSG) that performs better than KSG when n is small and X and T are not independent. It also important to note that many other variants of KSG and other estimators (Pál et al. 2010 Gao et al.
2015)
k-NN based approach.
Mutual Information Neural Estimator (MINE): In our work, we utilize neural network based MI esti-
mators, specifically Mutual Information Neural Estimation (Belghazi et al., 2018). This approach estimates mutual information by using a dual representation of the KT.-divergence, known as the Donsker-Varen Varadhan Given RVs X ~ P(X), T P(T), we express in terms
(DV) representation (Donsker & Varadhan 1983 of DV representation as:
I(::T)= sup (xT=PXX,m(((XX77) log (Ex.--(()iP(()) [eF(x:7]). FXXXT=3
(7)
where F' can be any measurable function from X' X + E that satisfies the necessary integrability constraints of two expectations in 7 to be well-defined, and the supremum is taken over all such functions.
To compute T(X;T) in practice, first we assume n independent and identically distributed samples (i.i.d.) are drawn from Ti), (X,, (n)) ~ P(X,T). Next, (X,, T,), is artificially constructed by choosing T, as randomly shuffled set of (Ta) When n is large enough, the MINE estimator approximates the MI as:
I Σ Fo, T:) - log 9
1 Σ Fa(Xi.d)
MMINE(X:T)
sup
(8)
where Fe: XXX R is parameterized by a deep neural network with parameters E θ and is the parameter space of the neural network. By training a neural network to optimize the above equation (i.c., finding the optimal 0* E E)), the final output will yield the true MI between X and T.
18

---

Published in Transactions on Machine Learning Research (07 2205)
B Additional Discussions on Related Works
B.1 Equitability
There is body of work on the equitability of mutual information estimators, of which the notion of self- consistent equitability is related to our work. The first work that proposes the notion of self-consistent equitability (Reshef et al. 2013), which is a generalized form of one-sided scale invariance discussed in our work, as we consider the specific case when the function is scale-related. In our work, however, we do not locus on the aspect of data transformation specifically, but mainly focus on the behaviour of estimators in response to scaling, In such works concerned with self-consistent equitability, there is always a very wide range of functions (Table 2 of (Reshef et al., 2013)) ) to get broad overview of the behavior of MI in response to any type of transformation. However, it is notable that scaling has not been one of them, most likely due to the way the data was already preprocessed to be scale-invariant, using local normalization, which already makes them self-equitable to scaling.
Our work only indirectly tests celfsconsistent equitability in Tables 1 and 2, where instead of prioritizing the degree of self-consistent equitability, we prioritize the MT estimation accuracy. We also use a variety of transformations as functions on the data (in a cascaded manner) and see the resulting estimation errors for the various compared metrics. However, our choice of transformations (apart from cube and sigmoid) is very different when compared to Table 2 of Reshef et al. (2013)ss 's work, as they either encompass all dimensions (like invertible random matrix multiplication) or change the dimensionality of the input by adding noisy dimensions or duplicate dimensions. In contrast, most functions in self-consistent equitabilily literature work with one-dimensional transformations applied to each data dimension individually. Lastly, as we cascade a random subset of these operations in a random order, we get a richer set of MI- Mllpreserving transformations, which can potentially also be tested in an equitability setup using their choice of metrics (like in Figure 2 of (Kinney & Atwal 2014)).. We are considering this for future work.
B.2 Recent MI Estimation works
We outline three different aspects in which our work differs from (Cyy et al. 2023 which is an important recent study that also tests MI Estimators and their performance under various settings.
Choice of transformations: As the focus in (Czyż et al., 2023 not particularly on the neural net- work use case where X is the input and Z represents feature layer, their choice of transformations is motivated differently. Most of their transformations are dimension-wise, i.e., transformation applied to each dimension. The only exception is their spiral diffeomorphism (Figure 5 of (Czyż et al.) 2023)), which radially morphs the distribution in such a way that the MI is preserved. We note that as our focus is mainly on the natural use cases of MI in deep learning, we construct certain types of transformations relevant to this setting. Apart from the cubic transformation, which is dimension-wise, all our other transformations are motivated by the potential use case in deep learning. We outline each one as follows. The sigmoid transformation is motivated by potential uses of sigmoid in the network's hidden layers. The random matrix multiplication (randmat) transformation is motivated via the features undergoing similar transformations through neural network layers, which are usually matrix multiplications followed by non-linearity, Note that the randmat also has an additional scaling term a (Section 4.3), which scales the resulting transformed vector as we wish to also focus on the robustness to scale. The duplicate-noise transformation, which adds dummy noise dimensions, is motivated by the fact that the number of hidden neurons can change through the layers and often many of these dimensions deeper within the network are usually very sparse and noisy. Similarly, the duplicate-self transformation is another approach changing the dimensionality of the input while preserving the total information.
Preproccssing methods: Czyž et al. 2023 's work indeed finds that the Gaussianization-base local normalization yields better results than uniform marginalization, when the base distribution is the multi- variate student distribution. However, they do note that the improvements are minor, and overall they end up choosing the standard varianco-based local normalization over the other two. Our conjecture that Gaussianization may work better when the distribution has long tails, as long-tai distributions are typically harder to estimate MI for, which was observed in (Cay et. al. 2023). This is potentially because the data
19

---

Published in Transactions on Machine Learning Research (07/2025)
samples that are part of the long tail may end up adding more noise to the final estimate, than in typical case with Gaussian variables, and Gaussianizing the data preserves MT while avoiding long tails in the input, thereby leading to better performance. A potential direction of future work is thus incorporating similar considcrations for our global normalization strategics to further enhance the accuracy of MI estimators.
Scale invariance: As the data was already preprocessed using local normalization approaches, scale invari- ance isn't a part of the analysis in (Czyż et al., 2023), similar to (Kinney & Atwal 2014). In our case, all our proposed normalization approaches are focused scale invariance while retaining other desired properties which may not hold for standard normalization approaches, such as robustness to noisy dimensions.
B.3 Neural retwork-based pre-processing
As one of our contributions is set of new pre-processing strategies ensure desirable scale-invariant behaviour of MI estimators, we discuss how this compares to other recent work that uses neural networks to learn pre-processing strategies for more accurate MI Estimation.
Gowri et al. [024]] In this work, the authors propose a two-step MI estimation procedure. First, they train compressed ropresentations of the input which minimize an upper bound on the conditional entropies between the compressed representations and the input/output, after which they estimate the MI bet ween the compressed representations using KSG, In the limiting case, these compressed representations would preserve the true MI, otherwise, normally they are upper-bounder by it. It is worth noting their compressed representations can be of any isomorphic form w.r.t the choice of compressor, and as they use neural networks, the scale of the compression can vary. So to alleviate the issue, they use the standard unit normalization approach (local). As such, our findings in this work, including the different variants proposed for global normalization, can be applied to their KSG estimation step to generate more robust MI estimates.
Butakov et al.] (024): It seemed to us that this work proposed a new estimator using normalizing flows, which in this case are function maps (which can be neural network) which eventually transform the data distribution to simpler one which has closed form solution for MI estimation. In their work, they seem to focus Gaussian base distribution. In principle, their estimator could be scale invariant, as their final estimator is the analytical MI expression itself. However, we did not see any explicit thcoretical study on this, as one will need to show that the correlation matrix al the end of applying multiple normalizing flows is invariant to one-sided scaling of one of the variables. Overall, our work points out that inve-invariannce is an important consideration for any MI estimator to prevent scale confounding the estimates.
B.4 Contrastive Learning-based MI Estimators
Contrastive learning-based MI cstimators, such as InfoNCE (Oord et al. 2018), estimate mutual information by maximizing the similarity between positive pairs while minimizing similarity between negative pairs. However, we can show that these estimators do not satisfy one-sided scale invariance, which is a key focus of our work. InfoNCE estimates MI by optimizing the following loss function:
e(x,2)
LIntONCE (X, Z) -E log Ej e(x,22))
where f(X, 2) is learned similarity function, often a dot product cosine similarity. The true MI is computed as the lower bound [(X; Z) 2 log n (I,, 2). In the original work, they used dot-product like function, yielding S(X, 2) = s(X)= Z. We nole that when Z is scaled here, we obtain f(X,a22
a af(X, However this yields LInfONCE (X, aZ) = -E log Σ, 20522255 This will not be scale invariant, as ((N
10x.22
in the limiting case when a + 0+, we can see that log 0070,225 = log 4. thus lima- ( =
logn in this scenario. Thus, the lower bound this case would be T(X; Z) 2 logn - LInGaNCE( XX, 7) = Having noted that, applying local or global normalization to X and Z should similarly yield scale invariant estimator, it is worthwhile exploring our approaches in this sctting for future work.
20

---

Published in Transactions on Machine Learning Research (07 (075)025)
C
Appendix: Details for Experiments
C.1 Details for the Section 5: Experimental Studies
Key Parameters:
Figure 1: Average MI estimates for KSG for a varying number of noise dimensions
Setup: Additive Gaussian (X,7 7 E IR? where = + with 6 ~ N 10)
Number of Samples: 1000
Number of Trials: 10
V2
o' = 0.04
Figure 2: Bias of MINE-based measures for varying noise dimensions
Setup: Correlated Gaussian X,T E R² with correlated coefficient p
Number of Samples: 1000
Number of Trials: 10
Correlation cocfficient: p: 0.2
Figure 3: Bias of the KSG estimator with the real data dimension
Figure 3(a):
Setup: Correlated Gaussian X,T E Rd with correlated coefficient p Number of Samples: 1000
Number of Trials: 20
Dimensionality (d): Evaluated over E (1,23,,4,5,,,,,,880)
Figure 3(b):
Setup: Correlated Gaussian X,T E Fd with correlated coefficient p
Number of Samples: 200
Number of Trials: 20
Correlation coefficient: p~ p~~ Uni4(0.4,000 where Unif denotes the uniform distribution
Dimensionality (d): Evaluated over E 4,84,8 8, 16, 16, 64), and log, d E (1,2,3,4,5 4, 6}.
Figure 4: Data duplication: Average estimates for KSG-based approaches.
Setup: Additive Gaussian (X,T E R2 where T X + c, with N (0,0212) and X'
X X,..X.... X].
Number of Samples: 1000
Number of Trials: 10
Dimensionality of Final Input: dx (2, 22, 42, 82, 102, 122, 142, 182)
Figure 5: Estimator bias versus dimension: Comparing MINE with MINE-Global variants
Sctup: Correlated Gaussian X,T E Rd with correlated coefficient, p
Number of Samples: 1000
Number of Trials: 10
Correlation coefficient: 0.2
Dimensionality (d): Evaluated over dx,dr dr E (1,233,4556677888999
Figure 6: Analysis of MI Estimators response to data scaling. Estimates are for InnX;T), where is the scaling factor
Figure 6(a) & 6(b):
21

---

Published in Transactions on Machine Learning Research (07 (025))
Sctup: Correlated Gaussian X,T R² with correlated coefficient p, estimates for [()X;T).
where // is the scaling factor.
Number of Samples: 1000
Number of Trials: 20
Correlation Coefficient: p: 0.8 (KSG); 0.5 (MINE)
Scaling Factor (1))
For KSG: 7) [10-2 2 10"), equispaced on log1o scale.
For MINE: 77 [10-3 2, 10], equispaced a log10 scale.
Figure 6(c) & 6(d):
Setup: Correlated Gaussian X,7 E IR² with correlated coefficient, p, RMSE of MI estimates ; T)),
where 77 is the scaling factor.
Number of Samples: 1000
Number of Trials: 20
Correlation Coefficient: p Unif(0,0.8]
Scaling Factor (7):
For KSG: E [10-?.10%) equispaced on log1o scale.
-
For MINE: E [10-2, 10), equispaced on a log10 scale.
-
Error Metric: RMSE computed between MI estimates and ground truth
Figure 7: Average MI estimates for various estimators across different values SNR.
Sctup: Additive Gaussian noise base, where = x with c~ (N(,00 Additionally, T scaled
T' 0.17, and the mutual information is estimated I(;T").
Number of Samples: 1000
Number of Trials: 10
0 VSNT Dimensionality (d): dx, dr
Background: Multivariant Student's t-distribution
Sampling Process To generate correlated samples following the multivariate Student's t-distribution, we construct a dataset where each sample consists of correlated random variables, X and T, with specified dimensionality and degrees of freedom (N). The multivariate Student's t-distribution is obtained by first sampling an (m + n)-dimensional random vector (X, (O0, 22) and random scalar U ~ x². where Ω is positive definite dispersion matrix. The rescaled variables are then defined as:
x-XVo T
which follow the multivariate Student's t-distribution. The tail behavior is controlled by the degrees of freedom where lower values result in heavier tails. Specifically, for v = the distribution reduces to the multivariate Cauchy distribution, for # = 2, the mean exists but the covariance does not, and for v > 2, both the mean and covariance exist, with Cov(X,T) = 5 When > 1, due to the concentration of measure phenomenon, U has most of its probability mass around v, and the variables (X,T) can be well approximated by (X,T).
Ground Truth Mutual Information Estimation The mutual information I(X;T) quantifies the de- pendency between X and T and can be computed the sum of the mutual information of the Gaussian- distributed basis variables and correction term. (Arellano-Vall Valle & Azzalini 2013) proved that:
I(x:T)= = -(X;)) + c(v, m,n),
where the correction term clv, m. n) given by:
m(v ((0) + f(u m + f(v mm) 8(v n).
22

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
with
f(x) = log [
where (xx is the digamma function. Unlike the Gaussian case, even when Q mm the mutual information T): cly, m, n) remains non-zero, U provides additional information about the magnitude. In our benchmark, we use this dispersion matrix to evaluate how well estimators capture the information contained in the tails, rather than focusing solely on estimating the Gaussian term.
E Sampling details for Synthetic Experiments
For our synthetic experiments in Tables and the main paper, we describe how the various parameters in choosing the distribution base were set (from Section
Correlated Gaussians: For each trial, we sampled p u(0,, 0.8), where U(a, b) denotes the uniform distribution between and b.
Additive Gaussian Noisc: First let a Va. Then for cach trial, we sampled a ~ (1, 2.0).
3. Student's t-distribution: In each trial, a random dof from 2, was chosen (uniformly distributed), which is the same set of dofs used in (Czyž et al. 2023 and the dispersion was set as
Tdim
plaim
Ω
plaim
Tdim
where p is sampled p~ ((0,0.....
F
MI Estimators: Configurations
F.1
KSG
We used the NPEET MI estimator toolbox for estimating KSG and KSG- measures We set for all experiments. For the global KSG variants, we fix Ci 0.11, 02 = 0.2, .., Cm for all our experiments.
F.2 MINE
We used the popular pytorch-based package for the MINE implementation.
Overall MINE implementation: For estimating 1 T(X;T), we used single-hidden layer ReLU-activated neural networks of the configuration: (dx + dr) + H1 + H2 .... + Hx + 1, where dx + dr is the dimensionality of the input and Hı...., H3 is the number of hidden ncurons for cach hidden layer. The last layer is a linear layer. We used the Adam optimizer with learning rate of 0.001. The hidden neuron configuration varies depending on our experiment. We set the number of epochs to 50 for all experiments. Given a training dataset S ((X,,Ti), Ti)..... (Xn,Tr)}: the network Fe effectively minimizes the following loss:
Fo (3) log Σ (o(x..)))
(9)
Note that we estimated MINE in the standard manner as per its original definition, which is, once the networks are optimized, we estimate the above loss on the training dataset Song & Ermon (2019). Tt is noteworthy that to avoid overestimation, (Czy2 et al. 2023) proposes an approach where the MINE estimate is computed on the test data. However, in our case, we lind that MINE does not overestimate for most of the cases in Tables and as almost always has negative bias, and thus the training test split approach is not necessary.
http:///gthhab.com//rggggeg /NPttt NPEET
23

---

Published in Transactions on Machine Learning Research (07 2025)
Figures lb
5b and Table 3 For the small dataset cases, we found that using smaller number of hidden neurons yielded significantly better and more stable results. Thus, for Figures 1b 1b3bll 4 5b and Table
3 using a single hidden layer with H1 = 20 yielded the most stable results on average, and thus we set H1 = 20. We found that for this small sample size setting, increasing H1 led to unstable estimates and large variance of estimaaors.
Figure 6 For the larger dataset cases, which is the case for our real datasets, we were able to increase H, and the details are as follows. For the IB dataset, we have one hidden layer with H = 30 neurons. For MNIST and CIFAR-10 datasets, we have two hidden layers with H 30 neurons each. For the SVHN dataset, we have one hidden layer with H = 100 neurons. The network layers are ReLU activated.
F.3 Network Architecture for Neural Network Analysis in Section 6
Table 4: Model Architecture for IB Datasel
Layer
Dimension
Activation Function
Input
28 x 28
Flatten
12
Dense
10
ReLU
Dense
7
ReLU
Dense
5
ReLU
Dense
4
ReLU
Dense
4
ReLU
Dense
2
SoftMax
Table 5: Model Architecture for MNIST Dataset
Layer
Dimension
Activation Function
Input
28 x 28
Flatten
784
Dense
1024
ReLU
Dense
20
ReLU
Dense
20
ReLU
Dense
20
ReLU
Dense
10
Soft
Table 6: Model Architecture for CIFAR-10 Dataset
Layer
Dimension
Activation Function
Input
32 32 3
Conv2D
32 x 32 x 16
ReLU
Conv2D
32 32 x 16
ReLU
MaxPooling
16 16 x 16
Conv2D
16 16 x 32
ReLU
Conv2D
16 16 x 32
ReLU
Global AveragePooling
32
Dense
64
ReLU
Dense
10
SoftMax
For the MNIST and IB datasets, we replicate the network architectures from Saxe et al. 2018 's work, using the widely-adopted ReLU activation function for the hidden layers. Specifically, for the IB dataset, we utilize
24

---

Published in Transactions on Machine Learning Research (07/2025)
Table 7: Model Architecture for SVHN Dataset
Layer
Dimension
Activation Function
Input
32 x 32 x 3
Conv2D
32 x 32 x 16
ReLU
Conv2D
32 x 32 X
ReLU
MaxPooling
16 x 16 X 16
Conv2D
16 x 16 x 32
ReLU
Conv2D
16 x 16 x 32
ReLU
Global AveragePooling
32
Dense
256
ReLU
Dense
10
SoftMax
a neural network with 7 hidden layers of dimensions 12-10-7-5-4-3-2. For the MNIST dataset, the neural network consists of fully connected layers with dimensions 784-10024-2--0--20-100
For the CIFAR-10 dataset, we adopt neural network with convolutional layers, 3 fully connected layers. The networks are trained using SGD and cross-entropy loss. We train 2000 epochs for the IB dataset, 200 epochs for the MNIST dataset, and 1000 epochs for the CIFAR-10 dataset.
In Tables we present the network architecture and output dimensions for each layer of the neural networks used in our study. The layers with bold text are the layers for extracted Z.
For the IB dataset, we trained for 2000 epochs with an SGD optimizer and learning rale 10-3 For the MNIST dataset, we trained for 200 epochs with an SGD optimizer and learning rate of 10 For the CIFAR-10 dataset, we trained for 1000 epochs with an SGD optimizer and learning rate of 10-3 For SVHN, we trained for 200 epochs, with an initial learning rate of 1 10-3, The batch sizes were 256 for the TB dataset, 128 for the MNIST dataset, and 512 for the CTFAR- 10 and SVHN datasets.
25

---

Published in Transactions on Machine Learning Research (07 (07/2025)
G Full Results on Synthetic Data
We provide the full results of Tables2aane and of the main paper, in Table D and include the full results for the correlated Gaussian distribution and the Student's 's (-distribution bases in Tables 10 and S In the following tables, in addition to the normalized root mean squared error, we also report the Spearman correlation and Bias of each estimator, as defined in Section 4.3.1 Table addresses the full KSG and MINE results in the same sctting as Tables and B and similarly, tables and address the full KSG and MINE results for the correlated Gaussian distribution and Student's t-distribution bases.
These are the specific takeaways from the additional results that follow:
1. Very interestingly, we find that although the prediction error in terms of normalized RMSE quickly becomes higher than random guessing (red entries) for data in higher dimensions, the Spearman correlation of MI estimates with the ground truth MI still stays high many cases.
2. A particularly notable example of this is for KSG variants, where we find that in spite of normalized RMSE exceeding 1, the KSG variants still have Spearman correlation measures very close to 1, showing that they are very highly rark-correlated with the true MI. It is therefore clear that the large negative bias of KSG variants high dimensions affects all cases similarly, and thus the dependency between the estimated MI and the true MI remains. This also highlights that using some carefully calibrated ways of estimating MI in high dimensional settings may yield very accurate predictions in terms of normalized RMSE well.
3. In fact, we find that over all cases correlated and additive Gaussian setups, only the KSG-global variant stays very consistent in terms of high Spearman correlation. Barring only two cases, we see that the Spearman correlation of KSG-global-Los is always greater than 0.9.
4. We find that overall our normalized variants of KSG and MINE showcase bias closest to O. We see that for all variants, the bias roughly becomes increasingly negative as the dimensionality of the input increases. This mirrors our observation Section 4.1.2)
5. We find that the overall results suffer in the heavy-tailed case of the Student's 's t-distribution base, especially MINE variants. However, we still see KSG-global-Lrce show consistently best performance among the KSG variants in most scenarios, and overall find that MINE-glo-cort shows better per- formance than other MINE variants, although the improvements are less significant that case.
6. We find that certain transformations are more difficult to handle than others. Notably the cube transformation which non-linearly maps each individual dimension, seems to significantly impact performance negatively. This shows that invariance to diffeomorphisms such as the cube transfor- mation still a challenging problem.
26

---

Published in Transactions on Machine Learning Research (07/2025)
Table 8: Comparin comparing per performance measures of MI Estimators Correlated Student's 's t- t-distribute iustribution Base
mihe-g
0.330
-0,02%
-0, 0,945
-0.002
0.713
-0.128
-1.298
0.314
-0.31
1.416
-0.218
-1.18
0.750
0.101
-1.013
0.892
40.419
4..76
L.023
40,24
-1.716
-0.446
-0.5
1,009
-0.330
-0.922
1.085
-1.004
1.2
-0.20 12.22
11955
1.0024
0.094
-1.636
1.507
-0.301
D. 00.008
MINE Bosed Moxsures
minn-gio
0.564
-0.00
-0,085
.0.031
0.582
0.009
.1.074
0.937
1 1,500
0.88
-0.112
-1.371
0.307
152
0.89
-0.271
-1.347
1.029
-0.38
-1.744
-0.381
1,080
40.300
-1,713
1:13
115550
1.161
-0 13282
1.114
10.001
1.034
-1 06666
mnhre bu
0.552
.0.0
-0.975
0.078
0.568
03.095
1.082
0.312
0.831
-0.13
-1.119
0.301
-0146
1,00
0. 00000
0.986
0.07
1.676
00,52
0.173
1.195
-0.5
0.760
1.090
-1. 3,712
1:150
0.357
2,022
1.054
0.084
00
00:80
mmhe
15,8377
0..44
-11 578
0.5422
16.314
.0.519
-11 S64
10:101
029
6.248
15.00N
-0 300
-12 302
19.110
371
12.792
-0.347
1400L
-0 103
-2800933
.0.888
11.11
10.0015
J6. 707
33.756
J8 374
09618220
0.342
1939815
15.316
-0 165
10.775
11.3510 T0
-0 002
-6. 59.508
-6
logg- leed
0.403
0,5
-0.800
0.836
0.524
0.788
40.840
1,029
1 721
0.7730
0.787
-0.77U
0,000
0.965
0.408
0.953
-0 06
1.084
0.438
1:220
0.999
12.22
1,252
0.675
-1,404
1.2822
11448
0.030
0.969
0.908
0,500
0.735
1,282
0.003
1.446
Mcasurre
0.274
0.800s
-0 39
0,579
0,288
0.808
-0. 00.419
0.512
00.552
0-459
0.959
-0.
0.309
0.061
0,399
0-961
-0, 120
1088
00.067
-1.102 107
0,883
-0.072
0.708
0.890
-n 100000
0.708
-0. 00.
1:125
00.355
1.184
0.725
0.970
0.716
0-707
-0. 00.709
01:301
0.907
-0-184
0. 0.899
0.287
0.870
-0 0.418
0.091
0.639
0.849
0
0.917
00.0
0.281
0:061
0.393
0-961
-0:188
1.125
0.212
-7 200
0.64
0.555
0.877
0.842
0.30
1:00
00 050
1.29
0.526
0.993
0.837
0:412
-0.500 0.508
KSG-
kog- loc
0.320
0.580
-0.471
0.537
0.340
0.863
40.30 0.000
0.808
0.775
0.50
-0.396
0.372
0000
110
2:0
150
-0.29 222
1-227
-0.000
200
0.221
1.30
-0.00 000
163
023
1.244
0.858
200
-0.375
1.336
kogg
0.451
0.827
-0 40:716
0,552
0.475
0.824
46 222
0.983
0.974
01088
0,789
40 704
0.952
01.433
+0 837
1,00
0.99
-1 008
0.59 1..975
<0-074
1.121
0,727
190
0.750
1 215
0,333
0. 0.982
-0 979
0,3009
1.5
0,750
1 122288
mnsssmee
RMSE mori
spearman
RMIE Suoo
spearman
RMSEE RMISE-ooru morrH
speartan
RMSE--orra
spearran
RMSE-wora
spesruss
RMSE--orr
spearron
RMST- morm
spear
RMSE-mor
spearman
spearman
RMSS RMrsE-morm
spearman
RMSE-mora
spearman
RMSEE-orra
spearian
speanuss
RMSE-morm
speartan
biss
hiles
hinos
bioe
biss
biss
hices
bios
RMS
bios
RMSE
2
6
dn
Transformation
da
/
eb
v
27

---

Published in Transactions on Machine Learning Research (07/2025)
mine- mine-glo-eeee glo-coo
Table 9: Comparing performance messures of MI Estimators Additive Gaussian Noise Base (Full Results)
0.278
0.928
-0.117
0.233
0-032
-0.093
0.096
0.884
0.084
n.922
-0.543
0.381
0.908
-0. 10.265
0.373
-0. 232
0.200
0.942
T.102
0.769
0.966
0.286
0.959
0.148
0.437
0.855
-0.181
1,185
0.815
-1.155
1.004
0.881
-0.971
1.058
0.818
-1 1.028
0.545
0.817
0.391
MINE- Be--Based Measures
mine-glo
06333
580
-0.
0.275
0.017
-0, 16
0. 666
055
-0.340
0.908
0. 0.000
-0.78L
0.500
0.044
-0.640
0.042
0.022
-0. 0.499
0. 0.505
0.917
00.719
1.210
18.87
1,012
0L505
10.114
-0.718
1.516
-1.502
1.585
0.862
-1.527
1.441
0. 08.811
-1.132
0.841
-1.431
1.400
-1.71
minne- loc
0.292
0.920
-0. 100
0.255
0-929
-0. 10.
0.560
0-816
0.720
0.930
-0.566
0.438
0.507
012296
0.300
-0 22.
0.423
0.885
0.002
1.009
0.905
-0.914
0.570
-0.004
0.570
-0.179
1.147
0.771
-1.16
0.991
0.783
-0,933
0.798
-1012
0.517
-0 01.068
mn
0.470
0.938
-0 0.244
0.445
0-933
-0 0.223
0,006
0.548
0.605
-1.100
0.030
0.000
-0 7.3
0.278
-0 122
0.622
0.876
1.674
0.542
1.155
0.800
1..19
1.881
0.852
-1 11888
1.848
0.842
121
0.691
-1.55
0.775
-0 967
1.5
0.813
-18877
ksg--g O>
0.110
-0.047
0.050
0 0.004
0.113
0:087
-0.050
00.001
0.085
-0.244
0.399
0. 0.9944
-0,322
0.297
-0. 253
0. 0.7
0. 0.994
0.931
0. 0.969
-0.773
0.300
0. 0.999
-0.240
1,422
-1.021
0.812
0.995
-0.810
1.261
0.989
-1.291
0.976
-1.150
0.816
-0.814
KSC- KSC-Besec Besed Measures
ksgg-gle
0.982
-0. 10
080
0.960
-0,005
0.113
0.987
-0,050
0.300
0,965
-0.240
0.300
0.990
-0.490
0.520
-0.422 00.426
0.298
0.995
0.950
0.961
00.79
0.995
-0, 0..239
1.121
-1.021
0.811
0.995
00.809
1,60
0.968
1.800
0.871
-1 0000
0.814
01811
kegg- loc
0.147
0.083
0.072
0.060
0.958
41,013
0.1
0.096
0,312
0.983
-0:254
0.445
0.905
41 013
0.304
0.247
132
0.954
0.077
0.975
0.816
0,916
-1.133
1,904
-1.919
0.816
0.995
-0.816
1344
0.979
-1.347
0.957
-1 275
1.00
0.926
121
bi-kag
0:404
0.085
-0 226
0.335
0.002
-0. 192
0.533
0.081
1.121
0.091
135235
0.032
0.549
4.714
0.342
40 29%
0.342
0.904
1203
0.534
1.045
0.901
-1 2.225
2. 20
-2.171
2,131
0.957
2.175
1,408
0
-1410
0.311
1.730
2.131
0,955
-2.175
kegg
0.301
0.085
-0. 19
0. 0.2286
0.9812
-0. 160
0-158
0.082
1.275
0.988
-1.078
0.500
01.651
0.302
22
0.332
0.994
1.13
0.588
0,942
0.988
1,078
188
1 1.9999
0.959
1.577
0.983
1,377
0.358
-1 63631
183
0.956
-2.00 001
RMSE- 10111
spearusan
RMSE-1o0 AMSE-110m 10030
spearиa
RMSE-Rom
sprarmon
RMSE-100
sprarman
RMSEE-n0rm
RMSE--oom
spearmnn
RMSE-nom morm
apearman
spearman
RMSE- RMSS--norm norm
speartan
RMSE RMSE--orm norn
specrmmsn
RMSE nomSE-nom momo
spearus
RMSE norm
spearman
RMSSE CUSS--1FF10 300310
spearman
RMSE JUUII
spearunn
bine
bins
biss
bine
bins
RMSE
bias
bias
hiae
bios
biae
bilee
biue
2
4
4
6
6
di
Transfornation
ds
eb
V
28

---

Published in Transactions on Machine Learning Research (07 (07/2025)
0.356
-0.238
0.315
0.910
-0.108
0.105
0.944
0.155
0.967
-0,008
0.144
-0.032
0.290
0.950
-0.112
0.758
-0.029
0.962
0.88338
-0,6319
0.5007
0:711
-0. 0.060
0.2286
0:951
0.078
0.766
0.902
-0.5 00.533
1.352
0.856
-1.200
1:006
0.788
-0.784
1.102
0.916
-0.976
0.912
0.960
-0.61
0.723
0,806
0.500
0.150
0.950
-0.027
MIN E-NEEBB B--Bood E-Beed Measurss
Table 10: Comparing performance measures of MI Estimatonn tort: Correlated Gaussian Base
minc-gla
0.601
-0 257
0.425
0. 0.001
-0 102
0.544
0.907
0.105
0.069
-0.00%
0.109
-0.0-1
0,358
0.946
-0 145
0.852
-0.073
0.001
0.705
-0,081
11,2000
0.805
-0.088
0,088
-0.441
0.819
0.024
0.507
3.424
0.800
1,988
1.411
0.771
1.302
1.382
0.942
-1 125225
1,077
0
-1 1.0000
01,851
-0.081
0.2315
0.960
-0.085
mine-loc loc
0.570
-0.247
0.322
0.929
-0 08.
0,559
0.135
0.042
-0. 00.029
0.130
-0.025
0.281
0.950
-0.105
0.702
-0.61
0.956
0.804
-0.641
0,080
0.773
-0.070
0.389
0.171
0.707
0.023
-0. 525
0.790
8.00
-0.45 05.451
1.163
-1 00.900
0.886
0.054
-1 84-444
0,875
0.144
0.157
0.962
-0.017
mine
0.573
-0. 2..388
0.350
0,588
-0.314
0.850
0.588
0.278
0.951
-0. 106
0.324
-0.104
0.489
0.879
-0:213
0.715
-0.783
0.974
03608
-0672
1,84
0.605
-0.781
0.514
-0.344
0.952
0.511
0,680
3.441
0.512
-1.412
1.440
0.590
-1.188
1.287
-1.182
0.337
0,958
-0,800
0.748
-1.146
0.280
0.793
-0.106
log L.
0.115
0,015
0.110
0,878
-0.005
0.233
0.567
-0.077
0,060
0.952
0.000
0,049
0.961
0017
0.000
0.950
0.000
0.409
0.969
-0.204
0.625
0.905
-0.316
0,893
0.990
-0. 00.48
0.263
0.986
-0.134
0.744
0,965
-0.440
0.963
-0.000
1.22
0.941
-0. 0..009
1213
0,900
-1.011
1063
0.988
-0,000
0.577
0.592
-6.729
0.061
0.967
0.000
KSG- Ba-ed Meos ures
0.113
0.014
0.100
0.595
-0.042
0.321
0.916
0.004
0.960
-1.028
0.052
-0.003
0.000
0.949
0.000
0.470
0.966
-0.298
0.628
0-395
-0.319
0.987
-0. 462422
0.205
0.985
-0.134
0.707
0.963
0.456
0:000
0.962
122
0.988
-0.910
1.321
-1.11
1.3
0.947
-1.46
0.5992
06771
0,018
0.964
-0.014
0. 12
01000
0.140
0.911
0.48
0.911
0,072
0.941
41.09
0.046
0.954
-0.000
0..004
0.005
0.909
020
0.481
0.964
-0-322
0.088
0.893
0.728
0.982
-0. 198
0.703
00.0 088
0.792
0.93
0.42
0.958
1:55
0.751
1.229
-1.044 1.004
1:5
0.977
-1996
0.845
-1
0.00N
0.967
40.081
bi-keg
0.1 125
-0.125 12
110
0. 0.923
40051
0.302
0.798
0.151
0.958
-0004
0.103
-0011
0.142
0.915
-0.090
0.408
-0.822
0.78
500
195
11.11
0.831
-0,589
0.259
0.152
081
603
0,615
0,170
1.511
1.388
112
260
0.641
11.720
1.516
-0.90
1.388
0.159
0.943
-0,000
0.125
.0,021
13338
0.922 0.923
0.045
0.704
0. 2.
0.961
06066
B100O
+0,01
0.126
0.940
-0
0.736
0G
0.738
0.713
0.778
0.983
+0.53 03.39
0.268
.0.1 19
0,582
0.523
1.38
0.1 1:1
1-424
0.127
12
13356
1 164
1217
8.115
030
0.,00
21212
0.141
0.9G8
40.040
incassue
RMSE-aonu
spenurman
RMSE-aom 2000
spearmm
RUMSE- 200m
sposrmmm
ILMSE Eo
sperrmmn
IUMSE aomS momm
spourman
ROMSE MANSE-oo aomm
spoormen
RMMEE-orm
speorrmmn
RLMSP-aom
apeorman man
RMS RMMF-no norm
speer mon
RMSE-aon
spearzann
RMSE-mon
spsermman
RMSE-1on
speorrman
RMSE RMISE-mmm
spoorman
RMSE-aam StOr
spourmmm
RMSE--30 SROEE
ILMSS morm
RLMSE aomm
spertman mOn
bins
biles
bites
bins
bise
bios
biss
bise
bioes
biss
bins
Mas
blos
d
2
6
10
10
10
10
N
2010
2000
1000
1000
200
2010
2000
1000
1000
2010
200
200
1000
1000
1000
dn
V
Translormat ko11
ch
29

---

Published in Transactions on Machine Learning Research (07/ (205)
cim 14
dim-
KGG (N- 1105
10000)
22
EWILY WOO0
Aeerage MI Bettmntee
1.3
Avverage MI Extimetrs
KSG 10000
K5OG acal 10000)
3.0
200 chetel
C5G- CV (n00)
(9300)
100. 10000000
04110
500 Guutea (N-10000)
81800
65-
D1325 25-1000)
530 2N- (0U00)
022
log (Scaling Pactor)
1000l (Scaling Factor)
Figure 7: Analysis of the KSG Estimators in response to data scaling for varying number of data samples N = (1000, 10000, 50000), and different data dimensionality d = 503. Estimates are for ;()); where oj is the scaling factor.
H Additional Results
H.1 Scale Invariance Testing
In the same setting as Section 4.2, we conduct more experiments to see if behaviour of the various estimators in response to data scaling remains unchanged for a different number of sampled datapoints N = {1000, 5000} and d = {10, 50}. The results are shown in Figures 7 and
KSG: Overall, for KSG (Figure 7 when we're analyzing the behaviour of the native estimators, we find that they show similar response to scale, i.e., they converge lo very low values near zero as the scale n goes to either extremity. In contrast, the scale-invariant estimators preserve the response across scales for d d= 10. For higher dimensionality d = 50, we see some intoresting bchavioral changes for the scale-invariant KSG-Local and KSG-Global variants. We find that although they do not reach negligible values when scale reaches either extreme, the average MI estimates do not stay the same for all 17, and there is some variation. Overall, the KSG-Global and Local variants behave similarly as before, and their average measures are relatively stable when 7) is no not small or too large. Only when n is either less than 0.1 or greater than 10, we see slightly more pronounced change for the average MI estimates.
MINE: For the MINE results (Figure g) we observe some interesting variations in the scale response depending on the dimensionality and the number of datapoints. We find that when we use a greater number of datapoints, then the MINE estimate converges to zero only for smaller scaling factors 7/- Our result. in Proposition 5 essentially applics for the limiting case when 77 + 0+, and we can intuitively show that the value of 1 for which we see this limiting behaviour reduces with the greater sample number. The proof of Proposition 5 essentially finds that the weights associated with the scaled RV n/T is upper bounded in magnitude by multiplicative factor of and the number of updates and epochs. As greater number of datapoints also implies greater nunber of gradient descent updates, this also implies that we shall have potentially larger weights when the sample number increases. This implies that for larger datapoints N the MI estimate may stay non-zero for a larger range of n,, and drop to near-zero values only near the extreme values of n, which is what we see in Figure g Lastly, we see that the local and global variants of MINE have relatively stable behaviour of their mean values across v) for all d, N combinations. However, we do see that the variance of these estimators increases with more datapoints.
H.2 Impact of noise in neural network training
In this experiment, we measure the MI between the inputs (X) and the features (Z) of a neural network, when trained on the MNIST dataset. An additive noise layer is introduced after the Z layer that adds Gaussian noise of the form (0(0, o²). Thus, subsequently, T(X; Z + +N(00, is measured at the last epoch of the training to evaluate how the MI changes when additive Gaussian noise is introduced. In figure | the noise level is characterized using the signal-to-noise ratio (SNR) numbers, which quantifics the strength of a signal relative the background noise. Specifically, an SNR of a dB implics that with unit signal power, the noise variance is 101o/10, Consequently, as SNR increases, the noise level decreases, leading to reduction in
30

---

Published in Transactions on Machine Learning Research (07/202) 2025)
Average Extmates
Average MI Estimates
1.5
10
85-
0.5
60-
Trer MI
0.0
Trx:
2.8
(og1a(n) (Sealing Factor)
log1o(n) (Sealing Factor)
(a) d = 10, N 1000
(b) d 10, N 5000
Average Extimates
Averuge MI Estimates
2-
MINE
MINE-Loal
MINI-ChahelA vervene
Trx:
(o) (Scaling Factor)
log1o(7) (Sealing Factor)
(c) d 50, N 1000
(d) 50, N 5000
Figure Analysis of the MINE Estimators response data scaling for varying number of data samples N
[1000, 5000), and different data dimensionality = {10, 50). Estimates are for [n)X;T), where is the scaling factor.
interference. The mutual information T(X; + +N(00,00)) expected increase as the noise level decreases. Since the SNR is expressed in docibels (dB), the noise reduction more pronounced within the range of low SNRs (e.g. 0 to 15 dB), decreasing faster compared to the range of high SNRs (15 to 30 dB), and MI within the range of low SNR should also have higher growing rate compared to high SNRs.
In Figure 9 we present the trend of [X;; Z+N((, during training, comparing the original KSG estimator, its loral-nord and global-normalizeed variants, and the MINE estimator and its variants. The displayed results represent the averages from 10 trials. For the original KSG estimator, we observe that as the SNR increascs, the mutual information does not change as initially anticipated. Instead, the results for the original KSG estimator initially increase and then decline as SNR contimues to rise. Notably, both estimators with global normalization exhibit the most consistent trend, reflecting the expected increase in dependence between X and Z noise is reduced, and having higher growing rate at low SNR regime.
MINE
I0X2 300. 0'3)
N70 0P
KSG
Local
30
15
100
15
20
25
30
SNR (dB)
SNR (cB)
(a) KSG
(b) MINE
Figure 9: 2X;2 + +000 o²)) results with varying SNR. Shaded regions indicate 95% confidence intervals derived from trials.
31

---

Published in Transactions on Machine Learning Research (07/ (07/2025)
Information Plane Analysis
In Figure 10 we present the information plane analysis for the IB, MNIST, CIFAR-10 and SVHN datascts, employing both KSG and MINE estimators to examine MI dynamics during neural network training. The displayed curves represent averages from 10 independent trials.
For the IB and MNIST datasets, the original KSG estimator shows that both [x; 2) and I(Y generally increase as training progresses. However, our proposed KSG-global- estimator reveals a clear information bottleneck pattern. Specifically, distinctly identifies two phases: a fitting phase characterized by an initial increase in Z) that eventually stabilizes, and a subsequent compression phase where I [(;; Z) decreases while I(Y; Z) continues to increase. For CIFAR-10, 10, the original KSG estimator exhibits an unstable and irregular trend, initially increasing, then decreasing, and subsequently increasing again. We hypothesize that the initial compression phase is later overshadowed by scalc-related effects. In contrast, the KSG-global-Liz estimator clearly and consistently demonstrates a monolonic compression phase, showing clear differences from MNIST.
Results from experiments with MINE estimators also yield insightful observations. For the IB datasct, the vanilla MINE estimator initially shows an unexpected reduction in both Z) and I(Y; Z), indicating brief period where the model seems to discard label information. Subsequently, we see clear fitting and compression phases, but only for the local and global variants. On the MNIST dataset, vanilla MINE captures only the fitting phase, with (X; Z) continuously increasing and no subsequent compression phase. In contrast, our proposed MINF--llobalcoorrett phases, unlike the local variant. For CIFAR-10, vanilla MINE only detects a fitting phase, while local- normalized and global-normalized MINE variants successfully reveal clear compression phases, echoing the results from their KSG counterparts.
variant clearly demonstrates both fitting and compression
In SVHN, there are some interesting observations. First, we see that the vanilla KSG and MINE, and the local normalization variants show inconsistent trends. Notably, the vanilla estimators show very less variation in (x; Z), thereby not capturing any notable trend. In contrast, the global variants of KSG and MINE show a consistent trend, showing a steady compression in terms of I(;; Z). But interestingly, for I(Y; Z) we observe that both global and local variants capture a phase where I(Y; Z) increases, and then a phase when T(Y; Z) actually decreasos, which is reminiscent of the compression phase but for T(Y; Z) instead of z)
J MI versus Feature Energy in Neural Networks
An important assumption in our work is that individual dimensions of X with lower energy compared to others have less impact on the MI estimate, as they are potentially noisier relative to the target variable. This assumption is contral to highlighting the advantages of srrccture-preservin global normalization over local normalization. In this section, we verify if this is true for neural networks. Here, we analyze I(2;; Y), where Z E Rd [2], .2,.....4 2a] are the convolutional neural network features that are input to the fully connected layers, and Y is the ground truth output label. Our overall hypothesis is, that lower the energy of 24, the less likely it is to share any meaningful information with the ground truth output Y, and thus more likely it is to be noisy dimension w.r.t the output, and can be ignored. To test this hypothesis, we conduct an experiment by training CNNs on MNIST and CIFAR-10. The architecture for MNIST consists of convolutional layers: [64, MaxPool(2x2), 128, (2xx),, 256, MaxPool(2x2) with padding 1, 512, (4x44], followed by a fully connected layer of size 512.
Similarly, for CIFAR-10, the architecture consists of convolutional layers: [64, MaxPool(2x2), 128, MaxPool(2X2), 256, (2x)) 512, MaxPool(4x4)), followed by fully connected layer of size 512.
In both cases, we define Z as the input to the fully connected layer, making Z E 1212 512-dimensional random variable. To analyze the relationship between feature energy and mutual information, we compute the MT between each individual feature dimension z, and the ground truth label Y. Simultaneously, we
oi measure the average energy of each feature dimension as: ag = Σi= n where the total number of data samples.
32

---

Published in Transactions on Machine Learning Research (07 (07/202) /025)
66-
KG0 global
2790
H866 global
HINE
0.7
1030
1300
01-
250
250
10
25
25
35
55
2.0
655
74
75
155
9.0
1B dataset
Information Plaee for WNIT dataset K3G
Information Plane MNVST detaset MINE
750
200
14-
110
122
73
73
10-
220
VIN
01
MNIST dataset
1000
0.-
ISC gkbbl
700
1.0
MINE-oocl
800
MHEekkbal
0.8
600
6.7
Epech
400
200
0.2
-0.2
-0.2
0.2
0.2
02
04
C6
CIFAR-10 dataset
(X:2)
Information Plane for SVINN deteset - KSS
Information Plane for SVIIN dataset MINC
KS6
NINE
0.14
local
MIBE
0.1
25
0.06
50
0.5
50
0.44
25
25
0.00
0.0
0.2
0.4
0.8
12 WX:ZI
1.4
1.6
SVHN dataset.
(I:Z))
Figure Information plane (I(X; Z) against I(Y: Z)) for IB, MNIST, CIFAR-10 and SVHN datasets. the output of 3dd layer for IB dataset and MNIST dataset, and 7th layer for CIFAR-10 and SVHN datasets. Details in Appendix
33

---

Published in Transactions on Machine Learning Research (07/202) 2025)
Feature MI Energy
Cumulative Energy Thresbold
035
M of Indlss Indiiiidual idual Feature Output
Average l'eatures
< Hinrryy Threshold
0.38
0:5
0.1
240
189
10a
Average Energy Individual Features
Threshold Energy
MNIST (left: Scatter plot as vs. T(2i, Y) and right: threshold- MI plot).
Energy
Cumulative vs. Energy Threshold
MI of Individual Feature Output
Average of of Features Enneryy ilreeshold
8.95
6.3
Average Energy of
Threshoki Energy
(b) CIFAR-10 (left: Scatter plot of vs. I(z Y) and right: threshold-based MI plot.)
Figure 11: Comparison of individual feature dimension energy and its mutual information with the labels for MNIST and CIFAR-10.
We present two key results: (1) a scatter plot of a; against the corresponding MI I(2;, Y), illustrating the relationship between feature energy and information content, and (2) a plot showing the average MI of feature dimensions with energy below threshold т, computed as [ Y)], with varying along the x-axis. The results are summarized in Figure
Our findings are clear: lower the average energy of a feature, less relevant it is to the model's output. This reinforces our hypothesis that in the neural network setting, lower energy features are more likely to behave as noise in the context of MI estimation.
K Proofs of theoretical results
Proposition 1. Let T (X:T) denote the mutual information estimated using fixed number of bins per dimension, with bin edges defined by the minimum and maximum values of the data. Then, for all E R+
Bin (ax all = Ih. (X;T) & Ti (X; = I (X;T).
Proof. For X E Rd define Xmin; Xmax Rdd as the componentwise extrema. Under scaling by >> 0, we have:
min(ax) a: min & max(ax) a:Xmax:
34

---

Published in Transactions on Machine Learning Research (07 (07/2025)
the bin edges scale by a, and the number of bins remains fixed. Thus, each bin in X corresponds bijectively to a bin in aX with identical counts. Therefore, the empirical distribution over bins, and hence hin is invariant under such scaling.
Proposition 2. holds that (axa(ax; = ss Va E Rt.
Proof. Let e denote the digamma function Abramowitz
1974). and
8,0, =
Σ 172
1llax, aX,1lx S ph.1,ohh
(10)
where Pheá, is the sup-norm distance of faX; aTi} to its k-nearest neighbor in the joint faX, aT} space.
Here ||ax; - -ajill represents the X-dimensions only distance.
Then, the KSG estimate of mutual information (equation 3 from (Kraskov et al. 2004 is
Iksa (b:) + (n))
(1M(aa,i,,))) (Mat.i.)))
(11)
n
Let Pk denote the sup-norm distance of Ti} to its k-nearest neighbor in the joint {X,T} space for the unscaled variables. It trivial to see that Pkii,o aPp1,1.00'
We then have,
Σ - ax,lloe S app.1000= = Σ HI|x, - X,51 pl.,,00 = n2,00000 itt
(12)
jth
One can similarly show that Natiip 71,,000
Thus, ((aaz,i,.)) = (M and (nadddso))) = vo(1,,,,0), and it follows that (aso(aX;aT) = (X:
Proposition 3. Let ((),,.))) be drawn i..d. from bounded distribution on Rdx x Rdr that is absolutely continuous. Then, it holds almost surely that
lim A (X;a)) lim ( = 1 a
a-000
Thus, Pn (X:o() need not be equal to Pa (X;T)
Proof. Following the proof of Proposition 2 let Pk denote the sup-norm distance from (X,, aTi) to its k-th nearest neighbor in the joint space (T). Define
Bat,i,
Σ T[loo, - aT3llo S pr.
(13)
and similarly for n0. The KSG estimator's estimate here will be:
im aT)) = (k:) + (n))
(#(nz,ioo) + v(naniioo)). Σ
(14)
n
First, we observe that
lim 0100
|X, ali; aTT))
(X,,
lim
max(]]x; X,1bo0, allT; Tylloo} ||X;
X,1boo.
(15)
o -
35

---

Published in Transactions on Machine Learning Research (07/2025)
Furthermore, X and T are absolutely contimuous, it holds almost surely that the k-nearest neighbor distance in X space will be greater than zero and finite.
In the limit a + 0+, all pairwise distances ||a2; - -Tiil + 0, and thus Bati = Ei - - S Pk.s,co) -...
Next, from equation 15 it follows that pk becomes the k-nearest neighbor distance X space as a - 0+
and thus we can write
lim 7z,
EMI - X,1lcc S pkiipo)) = k.
(16)
j+i
This enables us to replace the terms in equation
14 yielding
1 -
lim (kk) v(n)
k. (b(() + v(n)) K
(17)
Lastly, by global scale invariance (Proposition 2 so (X;aT) skso (2 X; T); and thus,
lim T ( +0
lim P K a=0
lim ksx 8
(18)
Proposition 4. holds that - (X;aT) = ou (X;T) Va E ER.
Proof. Let f be any neural network function used the MINE objective:
Exx.T)-P((., - log Pxx:m>-PXX)xcc))ll007
(19)
Define a corresponding function f' for the scaled variable aT by setting the first-layer weights on in f to W7 = Wr/a, and keeping all other paramelers identical. Then f"(X, = I(X,T), so both expectations in the MINE objective remain unchanged after the transformation.
As this holds for any f, the supremum over all such functions is preserved under scaling of T. Hence, To Mpn (X;+T)= (X;T))) =
TSINN opt (X;T) for all > 0.
Proposition 5. Consider the MINE optimization problem with input data S = ((xX), r.,.... (aXn, Kn) where X E Rd., Y Rdv, (X, Y) ~ P(X, are bounded RVs and a E R+ is a scaling factor. We consider a neural network of depth T. ++1 1 having h......... hy ReL.Uaactivted hidden neurons in the respective layers. The network is trained via gradient descent on the MINE loss function in Belghazi et al. (2018) for a fixed number of iterations nt; with a learning rate schedule S sy(t) < oo for all / S NT. Let the weights between the th node of the [+ th hidden layer and the jth node of the 4th hidden layer after iterations be denoted by w (t). Assume that the initialized weights bounded, i.c., V (1,,, [w), S for some 0. Lastly, let w/ [X] denote the first layer weights that are attached to X. We then have, V(i,j),
lim | (X]()) S... a
(20)
Proof. Let. (exa, y: 1) denote the output of the jth node layer in response to (axz v) as the input iteration t, where (x, v) a sampled instance of P). Note that when / = 0, ZJ will be the inputs to the network (ax, y) itself. As we are given the dataset S, the weight update rule for who at iteration t is then w/,, + 1) (t) (t),(t), where
2 Aw),(t) = -n(t) Σ
(ax,, Yost)s (aX,, Yx:t),
(21)
k=l
36

---

Published in Transactions on Machine Learning Research (07 (075))
Here, (aXk, Yx:t) denotes the backpropagation error signal at the 2th neuron of the (1 + 1)th layer iteration 4, computed in response to the input pair (a:X, k, Yε). For notational simplicity, omit the input instance arguments in the following expressions, and write the error signal as o (t). Let a(-) be the ReLU activation with its derivative d'(.) E {0,1}. Then we have,
( ((00)))1) 1=1
(0) d
(22)
where (t) denotes the pre-activation output of the jth node of the (1 - 1)th layer itself. Note that ()(1) (() for 1 2 As |a'l 1, this yields:
hy 18 201 S Σ (8(0) (0)
(23)
Let us assume the whole network function can be denoted as Jw (w(X, , y): Rd.-d, + R. Note that the network outputs single real number, and the last layer does not have any activation function (which ReLU for the other layers). In the context of MINE's optimization problem, the network minimizes the following loss function:
TMINE (X;Y) = --1 Σ Sw (aX,, Yi) + log Σ Swr (ax.8.)))
(24)
The error signal at the last layer, (t+ is the derivative of the loss w.r.t the network output. However, as the MINE optimization effectively has two distributions P(X,Y) Y) and P(X)P(Y) of input, consider the error signal of these distributions separately. The loss function for the input Xj.Y, ~ P(X,Y) is - fw (oX;, which yields an error signal (t+ = 1/nn. Whereas, the loss function for the imput X3,Y, ~ P(X)P(Y) is log (+ ff (x)...)). which yields an error signal
81+1(1)= (log 1 Er (lw(a.)))))
Σ oo (ax...)) S1.
(25)
dfw (aX,; V2)
Thus, across both cases, have that the error signal | [t)++((t---)))) $
Next, we consider the case when Ja|| s A. where A E R+ is finite real. Within this setting, using the principle of induction, we now will show that the error signal (0)(1) < D, v(l.j) and the weights [w) s B, V(I, i,j) for some finile reals Dr, B, which are independent. of c..
First, let us assume weights iteration i satisfy (t) < B, V(1, i,5) for some B, E R+ which is independent of a. Then, it first follows from equation 23 that
h1 6 (0) < Σ B.)o(()),
(26)
and as (84+1(() < 1, this ultimately yields
(8,(1) S (B.)++-1
II hy v=++1
(27)
If set D, max, ( I h,), , we have that (,)(t)) & D, Vl.j). Note that here D, only depends B, and the network architecture. Following the weight update rule at iteration & from equation 21
37

---

Published in Transactions on Machine Learning Research (07 (07/2025)
have
1of, + 1)1 10/((0) v7(()
n (a test)
n
1w),(0) n(()
Z)(aXn, Yait)| (0),
Be + n(t)D
Σ (axx, (asK)))
(28)
Next, X and Y bounded, can assume that every dimension lav K and lysl K for some K E IR.. Now, as |a| s A, the inpuls aX and Y are bounded. Furthermore, as the weights (4)) s B the outputs of the layers (Z) tg: will be bounded by finite scalar which can be written as fixed function of A, Br, K and the hidden layer counts , hr. Thus,
[t (1 + 1)1 B ((9) BG, K, ,,.,,,..., hz)-
(29)
We can set B++1 = Be + ((t)Dig(A, Bd, K, hi....,,hull hL), and the above then implics that |uf, + 1)1 = B V(I, i,),, where Bu+1 is also independent of a.
Lastly, as the weights at iteration are finite (Iw), (0) S e), via the principle of induction we have that for any finite 1, (w))(() < B, for some finite B., when |o|| < A. Let us consider B = max Br Then for the entire duration of training and for any |a| S A, we have (wfx(01) S B. Note that B is independent of a.
Thus, plugging this into equation 23 we obtain, v(I,j,t,,
by
15 (t) S Σ (1)((1)-
(30)
Subsequently, following the fact that [8L+1(1)| < 1, we gel
(0:(1) S B hgg 2=2
(31)
Let C KB" hn. With this, following equation 21
we have that AA s n(ttaC. This yields
Σ Σ intercl)
[X](()) < (x]()(())) +a
(32)
cc+ac Σn(t) n(t)
(33)
Thus, when - 0+, we have that + aC' (E (t)) + yielding lim (X](())))) S
Proposition 6. We consider the same setting as Proposition for the MINE estimation problem. There, it holds that lima T (X; = Thus, ag (X; need not be equal to in (MIN
Proof. Let f denote the neural network function learned via stochastic gradient descent (SGD) that maxi- mizes the following MINE objective for in.
MIN sgga X:aT):
38

---

Published in Transactions on Machine Learning Research (07 (07/2025)
P(xTa (X) - log (xxxxT=((x)xx(X])
(
Let Wr be the first-layer weights in f connected to aT'. From Proposition 5 as a + 0+, we have |Wrli e, thus yiclding finite weights. Hence, the contribution of aT to the output of a neuron of the first hidden layer of f becomes aTedr + as ++0 where dr the dimensionality of T. Therefore, ("(X, = (X) becomes independent of T in the limit a a.
In this case, both expectations in the above objective reduce to empirical averages of the same function I* (X),, and Jensen's is inequality implies
F(x) - log ( Ef 50,
with equality and only if (X) constant. Thus, the maximized objective converges to zero.
Hence, lim EMINE- (X; = O, showing that (X; need not equal I MMINE- (X;T).
Proposition 7. We consider the same setting as Proposition for the MINE estimation problem. There, holds that lima In (9ine-) = Thus, T (X; need not be equal In a (X:T).
Proof. Let f denote the neural network function learned via stochastic gradient descent (SGD) that maxi- mizes the following MINE objective for in (X;NF=sg((X;aTT)
Ex,T=P(XX.a)) [f (X, - log (xx.TTP((xxxxa)) (xx
Let Wr be the first-layer weights in f connected to aT. From Proposition 5 as - 0+, we have |Wrls S c, thus yielding finile weights. Hence, the contribution of aT to the outpul of neuron of the first hidden layer of f becomes aT'edy + 0 as a+0 where dr is the dimensionality of T. Therefore, f*(X, = (X) becomes independent of in the limit a +0
In this case, both expectations in the above objective reduce to empirical averages of the same function (X),, and Jensen's inequality implies
Σ (X,) - logg Eerrcnn <
with equality if and only if (xX is constant. Thus, the maximized objective converges to zero.
Hence, lim B X; aT) = O, showing that TII X; aT) need not equal in MIN s-g (X:
]
39


Reviews:
[{"review_id": "yRRU7iKVG3", "reviewer": "Reviewer_1DxP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. The paper addresses a subtle yet overlooked issue — scale sensitivity — that had not been deeply studied before in MI estimation. It provides both formal theoretical arguments and empirical evidence to support the claims.\n\n2. The proposed global normalization strategies are practical and improve estimator robustness. Benchmarking across multiple distributions and settings shows the generality and effectiveness of the proposed methods.\n\n\nWeaknesses\n\n\n1. Although KSG and MINE are popular, the study focuses mainly on these two, leaving open questions about the generalization to other MI estimators. For example, the InfoNCE estimator is also widely used.\n\n\n\n2. While experiments are extensive, it’s unclear if the methods perform equally well in larger datasets like ImageNet.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work studies the overlooked issue of scale sensitivity in mutual information (MI) estimators, particularly focusing on the popular KSG and MINE methods. The authors show both theoretically and empirically that existing MI estimators are not scale-invariant, meaning that simple rescaling of variables can significantly distort MI estimates. To solve this, they propose new global normalization strategies.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please discuss other MI estimators.\n\n\n2. Some larger real-world dataset experiments should be included.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lOQ3K1wPpd", "reviewer": "Reviewer_4xJv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The paper clearly demonstrates the need to investigate asymptotic and pre-asymptotic scale-invariance properties in MI estimators. My initial instinct was that the problem could be easily addressed via \"local\" normalization, but the paper convincingly outlines the limitations of this naive approach early on and provides informative experiments on the consequences of different preprocessing choices for the empirical behavior of normalized MI estimators.\n\n- It presents new theoretical results and analyses of popular MI estimators that appear to be missing from the existing literature.\n\n- The empirical evaluation is comprehensive and explores multiple dimensions. The results suggest advantages of the proposed scale-invariant estimators over existing baselines.\n\n- The estimators are straightforward to implement within existing frameworks and have practical implications.\n\n**Weaknesses**\n- I found it difficult to delineate the contributions of the paper from established results in Section 3. Are all of these propositions new (as I assume)?\n\n- It seems like an odd choice to place the related work section in the appendix. I believe it’s important to preserve the continuity of research on MI estimators by clearly situating the derivations within the context of recent studies and proposed benchmarks, such as [1]. I am concerned that recent developments are not highlighted with sufficient clarity. For example, p.5 refers to a “recent” variant of KSG, but the cited work is from 2017, which is not recent by deep learning standards. To gain space, other parts of the paper can be streamlined, and figure placement can be highly optimized.\n\n- The experiments involve only Gaussian distributions, which is somewhat misaligned with the generality of the theoretical results and undermines the generalizability of the observed empirical behavior of MI estimators. Moreover, Gaussians have the convenient property that multiplication by a constant isolates changes in variance, whereas other distributions have coupled moments (e.g., Gamma). What about heavy-tailed distributions such as Student-t, where variance depends on both degrees of freedom and scale? Perhaps the authors could briefly discuss how the improved MI estimators could be applied in other practical domains, such as (amortized) optimal design (OD, [2]), where robust MI estimation is crucial.\n\n- The paragraphs and results in the experimental section need to be organized more clearly. There are many single-sentence paragraphs with little connective structure between them.\n\n[1] Czyż, P., Grabowski, F., Vogt, J., Beerenwinkel, N., & Marx, A. (2023). Beyond normal: On the evaluation of mutual information estimators. Advances in Neural Information Processing Systems, 36, 16957-16990.\n\n[2] Rainforth, T., Foster, A., Ivanova, D. R., & Bickford Smith, F. (2024). Modern Bayesian experimental design. Statistical Science, 39(1), 100-114.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses an mportant challenge in mutual information (MI) estimation: sensitivity to scale. The authors show that popular estimators like KSG and MINE are not scale-invariant and propose new, easy-to-implement, estimator-specific normalization strategies that encourage scale-invariance. In addition to improving robustness to diffeomorphic transforms, the scale-invariant estimators are also shown to impact learning dynamics in neural classifiers. Overall, the paper makes a valuable contribution that underscores the importance of scale-awareness in MI estimation, but the work can benefit from a more streamlined experimental section and better embedding in the related work.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Major**\n\n- Please, improve the presentation of results and clarify relations to existing work in the main text.\n- Please, argue why Gaussians are sufficient for the numerical experiments in sections 5.1 and 5.2 and/or discuss generalizability to non Gaussian distributions (e.g., coupled moments, multimodality).\n-  Please, address minor points and questions below, and discuss the limitations of the estimators / experimental setup beyond the linear increase in computation time (e.g., none of the estimators seem generally robust to arbitrary diffeomorphisms, low SNRs remain problematic and can still cause biases).\n\n**Minor**\n\n- The paper would benefit from careful proofreading, as it contains typographical errors (e.g., p.1: $f(Y)$ should be $f(T)$), and some idiosyncrasies (e.g., p.1: “hardness” → “difficulty”) and repeated words.\n\n- In the Motivation section, it doesn’t seem accurate to describe $\\alpha$ as “arbitrary,” since the equalities clearly don’t hold when $\\alpha = 0$.\n\n- On p.4, equations 7 and 8 are discussed in the context of MINE but are only presented in the appendix.\n\n- “RV” is never defined. Random vector / random variable?\n\n- The notation on p.5 for correlated Gaussians is confusing. The problem is formulated as if $X$ and $T$ are independent, and the setting assumes uncorrelated Gaussians, given that all off-diagonal elements of the covariance matrix are zero.\n\n- The figures are low-resolution and do not make optimal use of space (e.g., large margins, small legends). The captions are also uninformative. Avoid referring to the appendix in figure captions and instead provide a brief, self-contained summary of what each figure demonstrates.\n\n- As the experimental results are dense and the figures are scattered throughout the text, it may be helpful to number the experiments (e.g., Experiment 1: Global vs. Local Normalization) and prepend the experiment label to each figure.\n\n**Questions**\n\n- From the results in Table 3, it appears that all MINE estimators perform poorly on a subset of transformations (last four rows — this effect is also present in Table 2). Do the authors have an explanation for this outcome?\n\n- Are there any implications for incorporating an inductive bias of MI invariance directly into neural network architectures?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "anmRFmNLyT", "reviewer": "Reviewer_tz8y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n* The paper is well-written and easy to follow. I've checked the main empirical/theoretical claims and the proofs. They all look good to me.\n* The algorithm design is well-motivated by the empirical and theoretical findings. Extensive synthetic and real-world experiments clearly demonstrate the superiority of the proposed methods.\n\nWeakness:\n* Some places need further clarifications (please refer to the section below).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes several extensions for the KSG and MINE mutual information estimators that are less sensitive to the scale of variables. The authors first theoretically show that most estimators are not invariant against scale. Next, they conduct multiple empirical studies to discuss the pros and cons of global and local normalization methods. Based on the global normalization method, the authors further adopt 1) the $L_\\infty$ norm as a more robust measure of distance for the KSG estimator, and 2) corrected weights for MINE so that it's invariant to dimensionality. The proposed methods are compared on synthetic and real-world estimation tasks, showing superior accuracy over the baseline methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* About the definition of normalization methods, Definitions 1-3: I think it's a bit weird that the numerator part of these definitions is not minused by $\\mathbb{E}_S[x_i]$, which ensures that the results center at $0$. This differs from the conventional definitions. I understand that KSG is invariant to such kinds of bias, but it's not immediately clear how this would affect the results of MINE. Also, the authors are using the expectation form for the variance term, not the empirical average of samples. Since the variance term is not directly acquirable in practice when data distributions are unknown, this choice may need further justification. Please clarify the differences between your definitions and the conventional one $\\frac{x_i - \\frac{1}{n} \\sum_j x_j}{\\sqrt{\\frac{1}{n} \\sum_j (x_j - \\frac{1}{n} \\sum_k x_k)^2}}$, and discuss how this would affect the empirical results.\n* In Section 5.1.1, the authors discuss the advantage of global normalization over local normalization. While I agree with your point that global norm is more robust to noise components, it is also true that global norm will result in bias towards large-scale components when the informative components are at different scales. In this synthetic study, the authors use isotropic Gaussian to generate the samples, which does not reveal this flaw. Please incorporate this point into your discussion. Also, it may be beneficial to show empirically that whether the informative components of neural features are likely to be of the same scale, such that the results of global norm are unlikely to be biased.\n* I would suggest the authors, that besides plotting the estimated $I(X;Z)$, apply their methods in some learning settings, e.g., minimizing $I(X;Z)$ as a regularization, and see how the performance is compared to baseline methods like variational information bottleneck. This would be strong support for the efficacy of the proposed methods.\n\nMinor points:\n* Section 5.2.1, the Performance Measures paragraph: I don't quite understand why the ground truth values $\\mu_i$ are different across the $k$ trials. If we look at the ds task, since the distribution of $X$ is not changed, the result should be exactly the same. Is this because you are selecting different $\\rho$ or $\\sigma$ in each trial? If yes, this should be clarified in the paper.\n* The proof of Proposition 3: I don't see where the condition $n \\to \\infty$ is used in the proof. Maybe this conclusion holds for any $n$? Also, the $+$ on the left of $1/n$ should be $-$ in the line below eq. (11).", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)