Curse of Slicing: Why Sliced Mutual Information is a Deceptive Measure of Statistical Dependence
Authors: Alexander Semenenko
Ivan Butakov
Ivan Oseledets
Alexey Frolov
Venue: ICLR 2026 Poster (2026)
Keywords: information theory
mutual information
sliced mutual information
curse of dimensionality
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Contrary to common belief, we demonstrate that Sliced Mutual Information is not a generally reliable measure of statistical dependence

Abstract: Sliced Mutual Information (SMI) is widely used as a scalable alternative to mutual information for measuring non-linear statistical dependence. Despite its advantages, such as faster convergence, robustness to high dimensionality, and nullification only under statistical independence, we demonstrate that SMI is highly susceptible to data manipulation and exhibits counterintuitive behavior. Through extensive benchmarking and theoretical analysis, we show that SMI saturates easily, fails to detect increases in statistical dependence (even under linear transformations designed to enhance the extraction of information), prioritizes redundancy over informative content, and in some cases, performs worse than simpler dependence measures like the correlation coefficient.

Published as a conference paper at ICLR 2026
WHY SLICED MUTUAL INFORMATION IS A DECEPTIVE MEASURE OF STATISTICAL DEPENDENCE
CURSE OF SLICING:
Alexander Semenenko', Ivan Butakov Ivan Oseledets & Alexey Frolov! Applied AT Institute "Moscow Independent Research Institute of Artificial Intelligence AAXXX "Institute of Numerical Mathematics, RAS
Moscow, Russia
{semenenko, ivan. butaakorr/appllieeaaa...
ABSTRACT
Sliced Mutual Information (SMI) is widely used as scalable alternative mutual information for measuring non-linear statistical dependence. Despite its advantages, such as faster convergence, robustness to high dimensionality, and mullification only under statistical independence, we demonstrate that SMI is highly susceptible data manipulation and exhibits counterintuitive behavior. Through extensive benchmarking and theoretical analysis, we show that SMI saturates easily, fails to detect increases in statistical dependence, prioritizes redundancy over informative content, and in some cases, performs worse than correlation coefficient.
Estimation complexity
pereeived position
copula
actual position
mutual information
our contribution
optimal-SMI
max-SMI
SMI
correlation
visit Section E for discussion
Ability to capture complex statistical dependencies
INTRODUCTION
Mutual information (MI) is a fundamental and invariant measure of nonlinear statistical dependence between two random vectors, defined as the Kullback-Liiber divergence between the joint distribution and the product of marginals (Polyanskiy and Wu, 2024):
((;Y) = KL Px.y x.Y 1
Due to several outstanding properties, such as nullification only under statistical indepen- dence, invariance to invertible transformations, and ability to capture non- con-linear dependen- cies, MI used extensively for theoretical analysis of overfitting (Asadi et al., 2018; Negrea et al., 2019), hypothesis testing (Duong and Nguyen, 2022), feature seloction (Battiti, 1994; Vergara and Estévez, 2014), representation learning (Bachman et al., 2019; Butakov et al,, 2025; Hjelm et al., 2019; Tschannen al., 2020; Veličković al., 2019), and studying the mechanisms behind generalization in deep neural net works (DNNs) (Butakov et al., 2024; Goldfeld et al., 2019; Shwartz-Zi Ziv and Tishby, 2017; Tishby and Zaslavsky, 2015).
In practical scenarios, Px,y and Ixx & Py unknown, requiring MI to be estimated from finite samples. Despite all the aforementioned merits, this reliance on empirical estimates leads to the curse of dimensionality: the sample complexity of MI grows exponentially with the number of dimensions (Goldfeld et al., 2020; McAllester and Stratos, 2020). A common

---

Published as a conference paper at ICLR 2026
strategy to mitigate this issue is to use alternative measures of statistical dependence that are more stable in high dimensions. However, such measures usually offer only a fraction of MI capabilities. Therefore, it is crucial to maintain a balance between robustness to the curse of dimensionality and the ability to detect complex dependency structures.
To strike this balance, popular techniques often retain MI as backbone statistical measure but employ dimensionality reduction before ton. While some studies explore sophis- ticated nonlinear compression methods (Butakov et al., 2024; Gowri al., 2024), others favor more scalable linear projection approaches (Fayad and Tbrahim, 2023; Goldfeld et al., 2022; Goldfeld and Greenewald, 2021; Greenewald et al., 2023; Tsur et al., 2023). Among the latter group, the Sliced Mutual Information (SMI) (Goldfeld et al., 2022; Goldfeld and Greenewald, 2021) stands out, leveraging uniform random projections:
g $ SI(X;Y) fe.. de $ 8g dø 84 g4y-1
1
110 X; x)) dø
(1)
Uniform slicing allows SMI to maintain some crucial properties of MI (e.g., being zero if and only if X and Y are independent), while remaining completely free from additional optimization problems (e.g., from finding optimal projections, as in (Fayad and Tbrahim, 2023; Tsur et al., 2023)). Combined with fast convergence rates, this has established SMI as scalable alternative to MI: computing the former typically requires orders of magnitude less time than ncural MI estimation (several seconds vs. several hours for SOTA diffusion MI estimators (Franzese et al., 2024; Kholkin et al., 2025)). Consequently, adopted for studying DNNs (Dentan et al., 2024 Wongso et al., 2022; 2023a; 2023b; 2025), deriving generalization bounds (Nadjahi et al., 2023), independence testing (Hu et al., 2024), auditing differential privacy (Nuradha and Goldfeld, 2023), feature selection (Goldfeld and Greenewald, 2021) and disentanglement in generative models (Goldfeld et al., 2022).
has been widely
Despite its popularity, the rescarch community has largely overlooked potential shortcom- ings of SMI. Some studies prematurely attribute their results to underlying phenomena without rigorously investigating whether they stem from artifacts introduced by random projections. Furthermore, existing works fail to comprehensively address issues related to random slicing, focusing primarily on suboptimality of random projections for information preservation (Fayad and Ibrahim, 2023; Tsur et al., 2023).
Contribution. In this article, we address this gap by systematically analyzing SMI across diverse settings, demonstrating that it frequently exhibits counterintuitive behavior and fails to accurately capture statistical dependence dynamics. Our key contributionn are:
Saturation and Sensitivity Analysis. Our theoretical analysis and experiments reveal that SMI saturates prematurely, even for low-dimensional synthetic prob- lems, and fails to detect significant increases in statistical dependence.
2. Redundancy Bias, We refute the prevailing assumption that SMI favors lincarly extractable information by constructing an explicit example where introducing such structure increases MI and even linear correlation, but decreases SMI. In fact, we show that SMI prioritizes information redundancy over information content. We argue that this bias can lead to catastrophic failures in some applications.
3. Curse of Dimensionality. We revisit the dynamics of SMI for increasing dimen- sionality and argue that SMI is, in fact, cursed, with the curse of dimensionality manifesting itself not through sample complexity, but via asymptotic decay to zero in high-dimensional regimes due to diminishing redundancy.
4. Reestablishing the Trade-off. Finally, discuss to which extent the aforemen- tioned problems can be solved by using non non-umiform/mmn non-rondoo slicing strategies, and how they affect the trade-off between scalability and utility.
In Section 2, we provide the necessary mathematical background. Section 3 overviews the related literature. Section consists of our main theoretical results (see Section B for proofs). In Section 5, we employ synthetic benchmarks to show the disconnection between dynamics of MI and SMI. Sections 6 and 7 illustrate that SMI maximization may result in degenerate solutions, contrary to MI maximization. Finally, we discuss our results in Section 8.
2

---

Published as a conference paper at ICLR 2026
2 PRELIMINARIES
Elements of Information Theory. Let (2, P) be probability space with sample space Ω2 o-algebra F, and probability measure I defined on I, Consider random vectors X + Rd, and Y ::Q + RR, with joint distribution Px,y and marginals Fx and Py, respectively. Wherever it is needed, we assume the relevant Radon-Nikodym derivatives exist. For any probability measure Q that is absolutely continuous w.r.t. I (denoted Q < P), the Kullback-Leibler (KL) divergence is [L[Q IP) = E llog a. which is non-negee non-negative and vanishes if and only (iff) P = Q. The mutual information (MI) between X and Y quantifies the divergence between the joint distribution and the product of marginals:
P KL[Px,y || P Py). &
((X;)) E Eogg
When Px admits a probability density function (PDF) ((X) with respect to (w.r.t.)
the Lebesgue measure, the differential entropy is defined as h(X) = E Bllog p(X), where log(-) -) denotes the natural logarithm. Likewise, the joint entropy h(X, Y) is defined
via the joint density p(X,Y) Y), and conditional entropy is h(X Y) = E[log p(X Y) =
-yy [Exiy logp(X Y)-- Under the existence of PDFs, MI satisfies the identities
((XY)) h(X) h(X Y) hYY) h(Y X) h(X) h(Y) - h(X,))
(2)
Sliced Mutual Information. In this work, we denote by PM the normalized Haar (uniform) probability measure on compact manifold M, iee, the unique bi-invariant measure satisfying PM (M) = 1. Hence, to sample uniformly from specific spaces we write W Mo 1 A Msykkd)) indicating draws from the Haar measures orthogonal group O(d) (Q E Raxd Q QQ =I}, the unit sphere g {X E Rd | 1];
and the Stiefel manifold St(k, d) = {Q E Rank Q'Q = 1}, respectively.
The k-sliced mutual information (k-SMI) (Goldfeld al., 2022) between X, Y is defined as
(X;Y) = -1 ). st(k,d,)
(oTx:TT)) dd d (.),
Setting A = 1 recovers the standard sliced mutual information (1).
3 BACKGROUND
Merits of SMI are straightforward and have been investigated thoroughly (Goldfeld et al., 2022; Goldfeld and Greenewald, 2021). We remind the reader of the most important them:
Scalability enabled by low-dimensional projections.
2. Nullification Property (ie., (X; ());Y>=> = 0 X and Y are independent), which stems from the projections being random and independent.
In contrast, demerits of SMI are not very obvious and not well-coverod in the litorature. In this section, we recapit and analyze previous works which address the shortcomings of SMI. To facilitate the analysis, we divide them into three main categories.
Suboptimality of random slicing. Tsur al. (2023) and Fayad and Tbrahim (2023) argue that a uniform slicing strategy can produce suboptimal projections, impairing SMI's ability to capture dependencics the presence noisy or non-informative components. To address this issue, Tsur et al. (2023) proposed max-sliced MI (mSMI), which selects non- random projectors that maximize the MI between projected representations. This approach also claimed improve interpretability and convergence rales.
However, deterministic slicing may overlook dependencies captured by non-optimal compo- nents. To mitigate this, Fayad and Ibrahim (2023) extend the max-sliced approach by optimizing SMI over probability distributions of projectors, with regularization to maintain slice diversity. While the authors emphasize that optimization should oceur over joint
3

---

Published as a conference paper at ICLR 2026
distributions, their motivation primarily addresses the issue of non-optimal marginal distri- butions 0 and é specifically, the presence of non-informative components in X and Y. We contend that this represents only partial understanding of the problem, many SMI artifacts arise from other factors. Needless to say that optimization over probability distributions also a heavy burden, which does not align with the slicing philosophy.
Data Processing Inequality violation. A fundamental property of MI is that it cannol. be increased by deterministic or stochastic processing. Furthermore, MI is preserved under invertible transformations. This is formalized by the data processing inequality (DPI).
Theorem 3.1. (Polyanskiy and Wu (2024, Thcorem 3.7)) For a Markov chain X - Y 2, [(X;X) Y) 2 I(X; Z). Additionally, if Z f(Y) where f is invertible, then equality holds.
In contrast to MI, SMI violates the DPI (Goldfeld and Greenewald, 2021, Section 3.2). While the intuition behind DPI is clear (raw data already contains full information, and processing can only destroy it), the implications of DPI violation are less straightforward.
Existing works suggest that SMI's violation of DPI can reflect preference for linearly extractable features, framing this as a useful property that aligns with the informal understanding of "practically available" (i.e., easily accessible) information (Goldfeld and Greenewald, 2021; Wongso et al., 2022; 2025). However, this interpretation can be misleading the factors behind SMT increases are misidentified. Our analysis reveals that this is indeed the case, as SMI exhibits more inherent biases than previously recognized.
Asymptotics in high-dimensional regime. Convergence analysis suggests that the sample complexity of SMI cstimation is far less sensitive to data dimensionality compared to that of MI. In fact, it has been argued that the estimation error may even decrease with dimensionality in some cases (Goldfeld et al., 2022, Remark 4). However, this behavior may result from SMI vanishing as dimensionality grows. Specifically, ( (Goldfeld et al., 2022, Theorem 3) provides an asymptotic expression (as d ~ oo) for SMI between jointly normal X and Y, which decays hyperbolically with d under some circumstances.
To date, no explanation for this phenomenon has been provided in the literature. We therefore elaborate on this finding by deriving non-asymptotic expressions, along with experimental results for non-Gaussian data, which reveal further nuances behind the decay.
4 THEORETICAL ANALYSIS
We start our analysis with considering simple example, which (a) admits closed-form expression for SMI and (b) highlights severe problems of the quantity in question.
Lemma 4.1. Consider the following pair of jointly Gaussian d-dimensional random vectors:
(x,Y) . (0, (1 ( (1)) (-1-1;1).
In this setup, MI and SMI can be calculated analytically:
p² 1,
3 + 1,2;
1(x;Y) = - log(111 p²), SI(X;Y) = 2d
F
2'
where F12 is the generalized hyperyeometric function. funciiio Additionally, the following limits hold:
lim I(x:) = +oc
lim SI(X;Y) = d
d
vbdd -(d-1) 3 dd 1 log 2 S d 1
1
lim I(XYY) = +oc
lim SI(X;Y) =
with v being the digamma function.
Note that while MI correctly captures the growing statistical dependence as d + oa (since additional components contribute shared information). SMT drops to zero, exposing fundamental problem. We interpret this behavior as distinct manifestation of the curse
4

---

Published as a conference paper at ICLR 2026
0.8
d=2
d 3
SI(X;Y) nats
0.6
d 4
(
0.8
d = 8
0.4
d= 16
0.6
d
d d= 3
0.4
d d= 4
0.2
0.2
d = 8
d 16
0
2
4
6
8
10
0
2
4
6
8
10
(X;Y)/d) nats
I(X; /)d nats
Figure 2: Saturation of YI(X;Y) as function of I(X; for the example from Lemma 4.1, non-normalized (left) and normalized (right) versions. Note that the problem becomes more prominent higher dimensions, both because of lower plateau and faster saturation.
of dimensionality: as d grows, SMI uniformly decays to zero and becomes ineffective for statistical analysis. We also provide less tight, but more general result Section C.
The second pair of limits reveals another critical flaw of SMI. When -² the X-Y relationship becomes deterministic - property MI reffects successfully. In stark contrast, SMI remains bounded by dimension-dependen factor that decays hyperbolically. Further- more, plotting SMT against MI shows this bound reached prematurely, demonstrating SMI's rapid saturation with increasing dependence (Figure 2). In this saturated regime, SMT becomes effectively insensitive to further growth in shared information. Moreover, this renders estimates of SMI for different dimensionalities fundamentally incomparable, as they are thcoretically bounded by factors depending on d.
These phenomena can not be explained by suboptimality of individual projections. In fact, each individual projection is optimal, as 1(0 x; does not depend on 0 in this particular example. The proof of Lemma 4.1 suggests that the problem arises from the majority of pairs of projectors being suboptimal, yielding ncar-independent 0 X and y in the most outcomes, even for = 2. Although similar analysis for k-SMI is extremely challenging, we argue that the problems in question prevail even when employing k-rank projectors.
Proposition 4.2. Under the setup of Lemma 4.1, A-SMI has the following representation
(I; =- Σ log(1 ((A) p(A) x II X E
1
II
28 22-1)/2
10.1]* 1=1
(+)
Remark. 4.3. As grows, (+) asymptotically concentrates X near zero, driving SIg to zero.
We argue that the limitations we uncovered can be attributed to a strong bias of SMI toward information redundancy. That is, SMI favors repetition of information across different axes, and sulfers from the curse of dimensionality if X and Y have high entropy. The following proposition and remark present a simple example to clarify this bias.
Proposition 4.4. Let X and Y be dy d.,d,-din dimeensnio random vectors respectively, with d, < k. Let A E Rm, xd, B E xd, be full column rank. Then (IX; = I(XYY).
Corollary 4.5. Consider the following pair Gaussian d-dimensional random vectors:
(X,Y) ~ N pJ ( PE (-1:1),
where J = 1. T with T = (1,...,)) 1). Then SI;(XYY) 1(X;Y) = log(1 -²²)-
Remark. 4.6. Applying 1 1.e] to X and Y from Lemma 4.1 individually yields the example from Corollary 4.5. Therefore, this linear transform inercases SMI despite decreasing MI.

---

Published as a conference paper at ICLR 2026
4.1 EXTENSION TO OPTIMAL SLICING
Although our work primarily focuses on conventional (average) SMT, as it is the most widely used variant, also provide some intuition regarding the limitations of its "optimal" counterparts: maz-sliced MI (mSMI) ('Tsur et al., 2023) and optimal-sliced MI (oSMI) (Fayad and Tbrahim, 2023). Since mSMI special case of oSMI without regularization, we restrict our discussion to it, though our reasoning extends to oSMI as well. The k-mSMI is defined as:
(X;Y) @e Suid k), s((d,,k
sup
((0x;
The following proposition highlights the shortcomings of linear compression: even in a simple Gaussian setting, mSMI captures only a subset of dependencies and can exhibit opposite trends to MI. This occurs, for instance, when dependencies become more evenly distributed across components, which again returns us to the redundancy bias.
Proposition 4.7. (Tsuu et al. (2023, Proposition 2)) Let (X (u, Σ), with marginal covariances Fx: My and cross-covariance Exy. Suppose the matrix 2x Ex x Ey exists, and let { denote its singular values in descending order, where d = min d dyy) Then
1 2
I(X;))
log(1 - Rt),
(I;(X;
log(1 p.)
1m
SYNTHETIC EXPERIMENTS
To complement our theoretical analysis and address complex, non-Gaussian cases, we conduct an extensive benchmarking of SMI using synthetic tests from (Butakov et al., n.d.), based on the works of (Butakov et al., 2024; Czyž et al., 2023). These tests are designed to evaluate MI estimators. However, here we do not assess whether SMI estimates converge to ground-truth MI values. SMT is distincl measure of statistical dependance, and should not be viewed as an approximation of MI. Instead, our analysis focuses on the relationship between the two measures: since MI captures the true degree of statistical dependence, opposing trends in MI and SMI reveal problems with the latter quantity.
For the experiments, we use correlated normal, correlated uniform, smoothed uniform and goumma--eauonenlile distributions, for which the ground-truth value of MI is available. To increase the dimensionality, we use independent components with equally distributed per-component MI. For each distribution, the projection dimensionality (k < d). In Section G, we also utilize MI-preserving mappings to transform low-dimensional Gaussian vectors into high-dimensional synthetic images, as described in (Butakov et al., 2024); the examples of such images are displayed in Figure 3.
vary both the data dimensionality (d) and
To estimate MI between projections, we use the KSG estimator (Kraskov et. al., 2004) with the number of neighbors fixed at 1 (higher values suboptimal, see Section F), 10a samples from (X, Y) and 128 samples from (O, 2). For each configuration, 10 independent runs with different random seeds conducted to compute means and standard deviations.
To experimentally verify the saturation, we plot SMI against MI normalized by dimension- ality d in Figure 4. The plots clearly show that SMI reaches plateau relatively early for all the featured distributions. The results for the normal distribution also align well with
(a) 2D Gaussians
(b) Rectangles
Figure 3: Examples of synthetic images from additional experiments Section G. Note that images are high-dimensional, but admit latent structure, which is similar to real datasets.

---

Published as a conference paper at ICLR 2026
SI(XY), Lists
SIXX Y), nets
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0
10
0
2
8
10
(dd
1(X; (X:Y)/dd nats
(a) Correlated Normal
(b) Correlated Uniform
1.2
1.2
3
SI(X; Y), nats
511 Y), mats
0.8
0.8
0.6
0.6
0.4
0.4
0.2
10
2
(X,Y)/d nats
1(X: Y)/d, nsts
(c) Smoothed Uniform
(d) Log- Gamma-Exponentt
Figure SMI results on synthetic benchmarks. Mean values and standard deviations across 10 runs are reported, 10a samples from X, Y and 128 random projections used.
cory N
por:: N
corZ. U
cort:
sm.. U
sin.
SIXXY); mate
LGE
SIX YY) mote
LGE
0.01
0.01
10
10
dimensionality
dimensionality
Figure 5: Saturated values of k-SMI versus data dimensionality d for 1-SMI (left) and 2-SMI (right) for correlated normal (corr. N), correlated uniform (corr. U), smoothed uniform (sm. U) and log-gammm--erponenlian (LGE). Log scale illustraaes the 1 1/d trend.
those from Lemma 4.1. We further confirm the saturation of k-SMT for k E (2,3) and for complex datasets from (Butakov et al., 2024) experimentally in Section G. Finally, we plot the saturated values against d on log-log scale, demonstrating that the 1/d trend from Lemma 4.1 also holds for non-Gaussian distributions.
Overall, the results strongly support our findings, showing saturation and uniform decay with increasing dimensionality across wide range of settings, from lowddimessoonal distri- butions to high-dimensional images.
6 SMI FOR INFOMAX-LIKE TASKS
Since mutual information is interpretable and captures non-linear dependencies, it is widely used as a training objective. Many applications involve maximizing MI (InfoMax) for feature

---

Published as a conference paper at ICLR 2026
selection (Battiti, 1994; Sulaiman and Labadin, 2015; Vergara and Estévez, 2014; Yang and Gu, 2004) and self-supervised representation learning (Bachman et al., 2019; Butakov et al., 2025; Hjelm et al., 2019; Tschannen et al., 2020; Veličkovié et al., 2019). However, due to the curse of dimensionality, it was instead proposed to maximize SMI for feature extraction (Goldfeld and Greenewald, 2021) and disentanglement in InfoGAN (Goldfeld et al., 2022).
In this section, we argue that SMI is not a suitable alternative to MI for InfoMax lasks: since SMI exhibits strong preference for redundancy, SMI maximization may lead to collapses.
Representation learning. To demonstrate SMI's redundancy bias, we examine learning compressed representations through information maximization (Deep InfoMaz) (Hjelm et al., 2019). This approach is known to be equivalent to many popular contrastive self- supervised methods (Butakov et al., 2025).
In Deep InfoMax, encoder network f is trained to maximize lower bound on I(X; f(X)), where represents imput data and f(X) its compressed representation. This method is thcoretically sound, maximizing MI ensures the most informative embeddings under the latent space dimensionality constraint. For our study, we replace MI with SMI in this framework. This substitution is straightforward since both MI and SMI admit Donsker- Varadhan variational lower bounds (Donsker and Varadhan, 1983):
Y(X;Y)= sup
ESp X,Y
T(X,Y) - log [Epx Bp, e(xx.)))
T:Q-3
E xxy T(OT Y,, 0, - log E Bxy " e( (4)
(X;Y) = sup E T:2-3
where T is a critic function, which is also approximated in practice by neural network. For detailed derivations of these bounds, we refer the reader to (Belghazi et al., 2018) (MT) and (Goldfeld et al., 2022; Goldfeld and Greenewald, 2021) (SMI).
We strictly follow the experimental protocol from (Butakov et al., 2025). In particular, we use MNIST handwritten digits dataset (Deng, 2012), employ InfoNCE loss (Oord et al., 2019) to approximate (4), use convolutional network for I and fully-connected network for T. Latent space dimensionality is fixed d = 2 for visualization purposes. Small Gaussian noise is added to the outlet of the encoder to combat representation collapse (Butakov et al., 2025). For more details, see Section We focus on this simple setup because our objective to show that SMI produces degenerate results even in clementary tasks, making more complex configurations unnecessary for this demonstration.
Results are presented in Figure 6. As expected, maximization of SMI immediately leads to collapsed representations, while conventional InfoMax yields embeddings with low redun- dancy (their distribution is close (O,, ()) This behavior is consistent across different runs.
Gaussian channel. We also refute SMI's preference for linearly extractable information by considcring X, Y such that cov X = I, Y AX + ((, I²I), and diag AAT I; this Gaussian channel with energy constraints (Cover and Thomas, 2006). Generally, Y) is
(a) MT max, 2000 epochs.
(b) SMT max, 10 epochs.
(c) SMT max, 2000 epochs.
Figure 6: Visualizations of embeddings from the representation learning experiments, with points colored by class. Note that mutual information ion maximization (left) produces clustered low-redunyancy representations, while SMI maximization results in immediate collapse.
8

---

Published as a conference paper at ICLR 2026
6
[(,V] S4(0,1) S
5600,)
1X:0)
(X.X S1,0
mnts
nats
4
2
0
100
10
100
<(A)
x(A)
Figure 7: Changing the condition number of A in the Gaussian channel experiment (Y AX + +V, o²l))) for normal X ~ (0, I2) (left) and uniform X ~ U[; 1 (right). We perform 10 runs with 10a samples, 128 projections, and use a = 0.3, d = 4.
maximized by a well-posed A, since decorrelated features are more robust to isotropic noise. However, the results in Figure 7 highlight SMI's preference for ill-posed A (i.e., matrices with high condition number m(A)
def |||| |A-')). More information is in Section D.
7 REPLICATION STUDY
Since our work highlights fundamental problems with SMI, we revisit the experiments from the original SMI articles (Goldfeld et al., 2022; Goldfeld and Greenewald, 2021; Tsur et al., 2023) to reassess their results. We are especially interested in the feature extraction and independence testing, because these setups might suffer from the redundancy bias and SMI's decay to zero. Section H provides more details.
Feature extraction. In (Goldfeld and Greenewald, 2021), the following toy problem considered: SI(AX;; BY) - max where X NO, Ia), Y =1=1-0 -1.eex+ +10, and A, B Rdxd feature selection matrices. The redundancy bias suggests that optimal A, B are singular, with all columns other than the first being zero a property reflected in the original results (Goldfeld and Greenewald, 2021, Figure 3).
To highlight that SMT fails when the number of relevant features increases, consider the following example: X (0, Y == -ET e,X;; + (O, Ia), where m controls the number of features. addition to maximizing k-SMI, we also learn A, through MI maximization. In the latter case, we use A, B E R to impose dimensionality bottleneck. For MI and k- SMI maximization, we reuse the NNs from Section 6 and perform 10 runs with 10 samples.
The quality of feature extraction is assessed via the effective rank (Roy and Vetterli, 2007) of the matrices formed by the first columns of A and B respectively. Figure 9 illustrates that MI maximization yields effective rank close to k, confirming its ability to recover all relevant features. In contrast, k-SMI results in low effective rank regardless of k, revealing its redundancy bias. A visual analysis of the matrices in Figure 10 and Section H.1 also supports our findings.
Independence testing. Goldfeld et al. (2022); Goldfeld and Greenewald (2021) report consistently superior performance of SMI over MI for independence testing when the data dimensionality d is fixed. We replicate their protocol for the distributions from Scction 5 but introduce critical modification. Instead of evaluating each separately, we pool SMI (and MI) estimates across multiple dimensions (d E {2, 10, 20, 30}) for each sample size and compute a single ROC-AUC from the mixed-dimennsonall data. For a fair comparison,
S-SMI
erank crankBr1mm))
A-SMIT
crankA,1
4
2
2
2
2
4
Figure 9: Effective rank versus & for feature
Figure 10: Matrices for SMI - max
extraction; 10 runs with 1 samples, = 6.
(left) and MI max (right), k 4.

---

Published as a conference paper at ICLR 2026
(X; Y) is fixed at 2 nat, and KSG (Kraskov et al., 2004) is used as a backbone MI estimator. We conduct 100 runs for each d.
As shown in Figure 11, and in contrast to (Goldfeld et al., 2022; Goldfeld and Greenewald, 2021), SMI performs worse under this more realistic setting where a single threshold must work across varying dimensions. These experiments reveal that SMI's discriminative power can drop sharply even when the ground truth MI is constant, causing dependent high- dimensional cases yield SMI values that overlap with independent low-dimensional cases. Consequently, it is hard to consider SMT reliable enough for independence testing, unless the dimensionality fixed in advance.
ROC-AUC
0.9
2-5MI
1-MMI
0.9
0.9
3-SMI
0.8
0.8
0.8
0.8
0.7
0.7
0.7
0.7
10
100
1000
10
100
1000
10
100
1000
10
100
1000
Independence testing: ROC-AUC versus sample size for correlated normal, correlated uniform, smoothed uniform and log waaaaaammoooontttiiii (left-to-right, 2 nat).
Figure 11:
DISCUSSION
Sliced mutual information (SMI) has been proposed scalable alternative to Shannon's 's mutual information, While SMI enables efficient computation in high-dimensional settings and satisfics the mullification property, our findings reveal critical deficiencies that undermine its reliability for feature extraction and related tasks.
Results.
We demonstrate that SMI saturates rapidly, failing capture variations in statistical depen- dence. This makes it difficult to distinguish between intrinsic SMI fluctuations and genuine changes in dependence structure. Furthermore, we invalidate the common hypothesis that SMT favors linear features through a counterexample where even correlation coefficients reflect dependence more faithfully than SMI, which exhibits inverted behavior.
In high dimensions, SMI decays with increasing dimensionality, contrary to MI's monotonic behavior. This is established analytically for Gaussian cases and validated empirically across diverse synthetic experiments. Consequently, SMT variations may reflect. redundancy or high- dimensional artifacts without principled way disentangle these factors.
Impact. Thanks to fast convergence rates and the absence of additional optimization problems, SMI has been widely applied across various fields of statistics and machine learning. Given our findings, it therefore crucial to recognize how the inherent biases of SMI affect practical applications.
The works (Chen et al., 2023; Goldfeld et al., 2022; Goldfeld and Greenewald, 2021) propose using SMI in a Deep InfoMax setting. However, we demonstrate that maximizing SMI can lead to collapsed solutions due to the redundancy bias. Mcanwhile, (Dentan et al., 2025; Shaeri and Middel, 2025; Wongso et al., 2022; 2023b; 2023a; 2025) study deep neural networks by measuring SMI between intermediate layers. Yet, changes SMI do not always reflect true shifts in statistical dependence; they may instead result. from differences in layer dimensionality, redundancy in intermediate representations, low sensitivity in saturated regimes, or other factors. Finally, (Nuradha and Goldfeld, 2023) suggests using SMT for independence testing differential privacy tasks. We contend that this approach poses critical issues, SMI estimates can become statistically indistinguish- able from zero in high-dimonsional or low-redundancy settings.
analysis reveals,
Limitations. While support our claims with both theoretical analysis and experi- mental evidence, we were able to derive precise analytical expressions for the Gaussian case only, Nevertheless, our findings are more than sufficient expose fundamental limitations of SMI, and to support all the claims we made.
10

---

Published as a conference paper at ICLR 2026
The work was supported by the grant for research centers in the field of Al provided by the Ministry of Economic Development of the Russian Federation in accor- dance with the agreement 300000C113955PFFMM
Acknowledgments.
and the agreement 20130-10-2025-033.
Ethics statement. This work not subject to any ethical concerns.
Reproducibility statement. To ensure reproducibility of our results, provide com- plete proofs in Section B and implementation details in Section I. We also provide our code for the experiments in the supplementary material.
LLM usage. Large Language Models (LLMs) were used only to assist with rephrasing sentences and improving the clarity of the text.
REFERENCES
Asadi, A., Abbe, E., and Verdu, S. Chaining Mutual Information and Tightening General- ization Bounds. Tn S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, & R. Garnett (eds.), Advances in Neural Information Processing Systems, Vol. 31, Curran Associates, Inc., 2018. https:/ /ptp:://proccoiiggs hwwpprocodnngggnnuiis...oom wwtuurp..ccppappe.. fttes/paprr/2115/ file/8d7622 10w.00100000222221111200666661-0
Bachman, P., Hjelm, R. D., and Buchwalter, W. Learning Representations by Maximizing
Mutual Information Across Views. In H. Wallach, H. Larochelle, A. Beygelzimer, F.
d'Alché- Buc, E. Fox, & R. Garnett (eds.), Advances in Neural Information Processing Systems, Vol. 32, p. Curran Associates, Inc., 2019. http:s:/ppooccedigssneeo neurips.c paper files/ hatss/pppr//aae /2019/ /file/ (aa.::552119c3774111 hatpa//ddd0bbbb6bbbbbb7.aaaa..pp
Battiti, R. Using mutual information for selecting features in supervised neural net learning, IEEE Transactions on Neural Networks, 5(4), 537-550, 1994. https:/ /to..oog//o///o/////o 10.1109/ 72. 72.298224
Belghazi, M. I., Baratin, A., Rajeshwar, S., Ozair, S., Bengio, Y., Courville, A., and Hjelm, D. Mutual Information Neural Estimation. In J. Dy & A. Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, Vol. 80, pp. 531 540. PMLR, 2018. https:/ htts://proccoding.milee wwtssmmlr...resS0011.1 /agga///m./////////aggaaaaaaaaaa bew.baziilghiaaaannn
Butakov, I., Malanchuk, S., Neopryatnaya, A., Tolmachev, A., Frolov, A., Foresti, A., and Franzese, G. MUTINFO, n.d. https://githhbb.oo htt.t/t...o/aaanees////../. /
Butakov, I., Semenenko, A., Tolmachev, A., Gladkov, A., Munkhoeva, M., and Frolov, A.
Efficient Distribution Matching of Representations via Noise-Injected Deep InfoMax. The Thirteenth International Conference on Learning Representations, 2025. https:// wttp:/eeweeee//oooddddddaaaaam
Butakov, I., Tolmachev, A., Malanchuk, S., Neopryatnaya, A., Frolov, A., and Andreev, K. Information Bottleneck Analysis of Deep Neural Networks via Lossy Compression. The Twelfth International Conference on Learning Representations, 2024. https:// openreview wtwp:o/wwn//ffoouooeoo -hut
Chen, Y., Gutmann, M. U., and Weller, A. Is Learning Summary Statistics Necessary for Likelihood-frree Inference?. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, & J. Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, Vol. 202, pp. 4529-4544. PMLR, 2023. https:// wttpppoooiig.mm....r.reeeeee chen23h.html
Chen, Y., Ou, Z., Weller, A., and Gutmann, M. Neural Mutual Information Estimation with Vector Copulas. The Thirly-Ninghe Annual Conference on Newral Information Processing Systems, 2025.
Cover, T. M., and Thomas, J. A. Elements of Information Theory ( (Wiley Series in Telecommunicatin. and Signal Processing). Wiley-Interscience, 2006.
Czyb, F., Paweland Grabowski, Vogt, J., Beerenwinkel, N., and Marx, A. Beyond Normal: On the Evaluation of Mutual Information Estimators. In A. Oh, T. Naumann,
11

---

Published as a conference paper at ICLR 2026
A. Globerson, K. Saenko, M. Hardt, & S. Levine (eds.), Advances in Newral Information Processing Systems, Vol. 36, pp. 18957-16990 Curran Associates, Inc., 2023. https///prooccdnnssnnnurrg spaperr files tasss/aper/2000 /ttp:///////000ee00fff00060606 Pe449Tedd-Papee--Coffeeeeeee
Deng, L. The mnist database of handwritten digit images for machine learning rescarch. IEEE Signal Processing Magazine, 29(6, 141 441, 2012.
Dentan, J., Buscaldi, D., Shabou, A., and Vanier, S. Predicting and analyzing memorization within fine-tuned Large Language Models, 2024. https:/ htass//axxvvorr////o http:/oogg/a//////////
Dentan, J., Buscaldi, D., Shabou, A., and Vanier, S. Predicting Memorization within for Classification. Proceedings of the 28th European
Large Language Models Conference on Artificial Intelligence (ECAI 2025), 2025. htlps://arxivoog//aae hang/abs/2400 /2409. 18858
Donsker, M. D., and Varadhan, S. R. Asymptotic evaluation of certain markov process expectations for large time. IV. Communications Pure and Applied Mathematics, 36(2), 283- 1983.
Duong, B., and Nguyen, T. Conditional Independence Testing via Latent Representation Learning. 2022 IEEE International Conference on Data Mining (ICDM), 121-130, 2022. https://doi.org/ 10. 10.1109/1CDMM444.... JCDDM48444.002200000
Edelman, A., and Sutton, B. D. The beta-Jacobi matrix model, the CS decomposition, and generalized singular value problems. Foundations of Computational Mathematics, 8(2), 259-285, 2008.
Elezovic, N., Giordano, C., and Pecaric, J. The best bounds in Gautschi's inequality. Math. Inequal. Appl, ((2), 239-252, 2000.
Fan, Y., and Henry, M. Vector copulas, 2021.
Fayad, A., and Ibrahim, M. On Slicing Optimality for Mutual Information. Thirty-Sevene Conference on Neural Information Processing Systems, 2023. https:/ /openr//pperrviiww net/
(orum?id=-MuKK222aaa
Franzese, G., BOUNOUA, M., and Michiardi, P. MINDE: Mutual Information Neural Dif- fusion Estimation. The Twelfth International Conference Learning Representations, 2024. https://openreview/ net wtwpr//www.oo/ofommmddddd--ooooo fformm"dd=OWWasSSeee
Goldfeld, Z., and Greenewald, K. Sliced Mutual Information: A Scalable Mcasure of Statistical Dependence. In A. Beygelzimer, Y. Dauphin, Liang, & J. W. Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. httpss//openreview. fidrmmddd=27qonn
Goldfeld, Z., Greenewald, K., Niles-Weed, J., and Polyanskiy, Y. Convergence of Smoothed Empirical Mcasures With Applications to Entropy Estimation. IEEE Transactions on Information Theory, 66(7), 4368 4391, 2020. nttps://doi.org. w/0gg. 10..119//TTTT. 2020.2975480
Goldfeld, 7., Greenewald, K., Nuradha, T., and Reeves, G. SkS-Sliced Mutual Information: Quantitative Study of Scalability with Dimension. In A. H. Oh, A. Agarwal, D. Belgrave, & K. Cho (eds.), Advances in Newral Information Processing Systems, 2022. hitps://openrevite net /tt////ommdm--ooodaaaaa
Goldfeld, Z., Van Den Berg, E., Greenewald, K., Melnyk, I., Nguyen, N., Kingsbury, B., and Polyanskiy, Y. Estimating Information Flow in Deep Neural Networks. In K. Chaudhuri & R. Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, Vol. 97, pp. 2299-2308 PMI.R, 2019. https:// https://proceedinss.ll/
press/v97/ press/v97/goddteldlla...to goldfeldd99.hhmm
Gowri, G., Lun, X., Klein, A. M., and Yin, P. Approximating mutual information of high-dimensional variables using learned representations. The Thirly- Eighth Anveual Conference on Neural Information Processing Systems, 2024. https://openrrviees net HHNomPyd=
12

---

Published as a conference paper at ICLR 2026
Greenewald, K. H., Kingsbury, B., and Yu, Y. High-Dimensional Smoothed Entropy Fsti- mation via Dimensionality Reduction. IEEE International Symposium on Information Theory, ISIT 2023, Taipei, Taiwan, June 25-30, 2023, 2613-2618, 2023. https:/ /doi. org/10.1109/11 11111999//14111..11 2.2.....22..... 10223111
Guo, D., Shamai, S., and Verdu, S. Proof of entropy power inequalities via MMSE. 1011- 1015, 2006.
0tt///00.119/1111111 /Tt.://T.....11110000
Hjelm, R. D., Fedorov, A., Lavoie-Marchildon S., Grewal, K., Bachman, P., Trischler, A., and Bengio, Y. Learning deep representations by mutual information estimation and maximization. International Conference on Learning Representations, 2019. https:// openreview.ner www..neeeeooummiiii-iiaaa
Hu, Z., Kang, S., Zeng, Q,, Huang, K., and Yang. Y. InfoNet: Neural Estimation of Mutual Information without Test-Time Optimization. Forty-Fire First International Conference on Machine Learning, 2024. https:/ /ttp:s///ppereiiww.... net
net/forummidee fformmodddddo// 40hhm
Kholkin, S., Butakov, I., Burnaev, E., Gushchin, N., and Korotin, A. InfoBridge: Mutual Information estimation via Bridge Matching, 2025. hutps://arxiv..te abss/25220013.
Kingma, D. P., and Ba, J. Adam: A Method for Stochastic Optimization, 2017.
Kraskov, A., Stögbauer, H., and Grassberger, P. Estimating mutrual information. Phys. Rev. E, 69(6), 66138, 2004. https://dbi.orr// 10. LwL.://////ee....9999 896666688888888
Lee, K., and Rhee, W. A Benchmark Suite for Evaluating Neural Mutual Information Estimators on Unstructured Datascts. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, & C. Zhang (eds.), Advances in Neural Information Processing Systems, Vol. 37, pp. (6319-46333 Curran Associates, Inc., 2024. https:/ http:///dioorggggg 10.
52202/0070017-4477 7-1722
Ma, J., and Sun, Z. Mutual Information Is Copula Entropy. Tsinghua Science & Technology, 16(1), 51 54, 2011.
McAllester, D., and Stratos, K. Formal Limitations on the Measurement of Mutual Informa- tion. In S. Chiappa & R. Calandra (eds.), Proceedings of the Twenly Third International Conference on Artificial Intelligence and Statistics, Vol. 108, pp. 875-884- 884. PMLR, 2020. https://proceeddngg.nn mlr. pwwss/eesssss htt////maallstt220.....
McBride, A. Special functions, by George E. Andrews, Richard Askey and Ranjan Roy. Pp. 664.$ 60. 1999. ISBN 0 521 62321 9 (Cambridge University Press.). The Mathematical Gazette, 88947), 355-357, 1999.
Nadjahi, K., Greenewald, K., Gabrielsson, R. B., and Solomon, J. Slicing Mutual Infor- mation Generalization Bounds for Neural Networks. ICML 2023 Workshop Newral Compression: From Information Theory to Applications, 2023. https:/ https///ppereviw..ncoo forum?"d=ccLewK3335
Negrea, J., Haghifam, M., Dziugaite, G. K., Khisti, A., and Roy, D. M. Information- Theoretic Generalization Bounds for SGLD via Data-Depender Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-BBuc, F. Fox, & R. Garnett (eds.), Advances in Neural Information Processing Systems, Vol. 32, p. Curran Associates, Inc., 2019. https:/ http://oroocdcn.oocccoo www.ss...uuss...o hace/pppe_fffiie files/ Aallss/aper///0 2019 /0/l///f///// /ftl/////aaa /05/e/444477333 93370d1 43370011228222011b-aappp..pp
Estimates. In H.
Nuradha, T., and Goldfeld, Z. Pufferfish Privacy: An Information-Theoretii Study. IEEE Trans. Inf. Theor., 69(11), 7336 7356, 2023. https://doi.org org. 10. /0099/TT.20000 21..22222332222
Oord, A. van den, Li, Y., and Vinyals, O. Representation Learning with Contrastive Predictive Coding, 2019. https://arxit wtwpaiiwwi//aaaaaao.oooa abs/ 11tp//80700374
Polyanskiy, Y., and WI, Y. Information Theory: From Coding to Learning, Cambridge University Press, 2024. https:/ //os:///ooss..o
http://boooooggglerruuu.u google.ru/
Roy, O., and Vetterli, M. The effective rank: A measure of effective dimensionality. 2007 15th European Signal Processing Conference, 606-616 610, 2007.
13

---

Published as a conference paper at ICLR 2026
Shaeri, P., and Middel, A. MID-I; Matris- Interpolaae Dropout Layer with Layer-wise Newron Selection, 2025. uttps//arxxiv.rrg /aaaaab///////////
Shwartz-Ziv, R., and Tishby, N. Opening the Black Box of Deep Neural Networks via Information, 2017.
Sulaiman, M. A., and Labadin, J. Feature selection based on mutual information. 2015 9th International Conference on IT in Asia (CITA), 1-6, 2015. https://doi.org/ 10.1199
Tishby, N., and Zaslavsky, N. Deep learning and the information bottleneck principle. 2015
IEEE Information Theory Workshop (ITW), 5, 2015.
Tschannen, M., Djolonga, J., Rubenstein, K., Gelly, S., and Lucic, M. On Mutual Information Maximization for Representation Learning. International Conference Learning Representations, 2020.
net httprrrreiieeeeeee//ommmoooooo forum
Tsur, D., Goldfeld, Z., and Greenewald, K. Max-Sliced Mutual Information. "hirty-Seventh Conference on Newral Information Processing Systems, 2023. https:/ /oppn/ovvie.nnnttttt.... forum31d=-
Veličković, P., Fedus, W., Hamilton, W. L.,, Liò, P., Bengio, Y., and Hjelm, R. D. Deep Graph Infomax. International Conference on Learning Representations, 2019. https:// wttrreviee.ne/fooummddddd---e
Vergara, R., and Estévez, A. review feature selection methods based on mutual information. Neural Comput. Appl. 24(1), 175-186, 2014.
Wongso, S., Ghosh, R., and Motani, M. Understanding Deep Neural Networks Using Sliced Mutual Information. 2022 JEFE International Symposium Information Theory 200.T555666.022298222
(ISIT), 113-133-150 138, 2022. https://doi.ons /do..or//
Wongso, S., Ghosh, R., and Motani, M. Pointwise Sliced Mutual Information for Neural Network Explainability. 2023 IEEE International Symposium on Information Theory (ISIT), 1776-1781, 2023b. hopp:://doi.oo www.oo////////....o 10..0.1199//IIITT11. 2023. 10207010
Wongso, S., Ghosh, R., and Motani, M. Using Sliced Mutual Information to Study Memo- rization and Generalization in Deep Neural Networks. In F. Ruiz, J. Dy, & J.-W. van de Meent (eds.), Proceedings of The 26th International Conference Artificial Intelligence and Statistics, Vol. 206, pp. 11608-11688 PMLR, 2023a. https:/ http:////cccc/gss./// mir press/ v206/won ht0d///ongs223ahttm
Wongso, S., Ghosh, R., and Motani, M. Sliced Information Plane for Analysis of Deep Newral Networks, 2025. hunps///ooio////033////////// http://thhxxxx11...5.. (00.:3830000881111188 2687 vv
Yang, S., and Gu, J. Feature selection based on mutual information and redundancy-synergy coefficient. J. Zhejiang Univ. Sci., 5(11), 1382 1391, 2004.
14

---

Published as a conference paper at ICLR 2026
A SUPPLEMENTARY THEORY
Lemma A.1. (Polyanskiy and Wu (2024, Example 2.4)) (n, E), (2Te() det d)..E).
Corollary A.2. For (X,Y) ~ N(u, 2 with non-singular 2
1
((; log det Ex log det Ey log det Σ
1 d log p.¹,
where Ex, Ey marginal covariances, Exy ex is cross-covariance, d min min(dg, dy), and (pi) are singular values of 2x² Exx Exy
Proof of Corollary A.2. Combining Lemma A.1 and (2) yields the first result. Now note that (UEE,
1(;Y) =1 ( -x v)
where V is the SVD of Σx² E Exx Ex Now note that
U U ~ N M1², (dig((e)) diaa(ar)))
from which we arrive the second expression.
Lemma A.3. Let A E R= be full column rank matrix, and θ ~ Msu(n Then θT is full-rank with probability one.
Proof of Lemma A.3. Performing QR decomposition of A yields @ = θJQR = oT ("r R. Since A is full-rank, R is invertible and rank θ A =rank θ (*x)) Therefore,
P(OTA A is full-rank) 1 P o TT
[m 0
is not full-rank } 1.
Lemma A.4. (Edelman and Sutton (2008, Theorem 1.5)) Let W Mo(a) and partition
W11 W12
W
W21 W22
with Wn of size & by k. Then the eigenvalues (1, of Wn W.T follow the Jacobi ensemble
p(A) σ II 2,10 1
[ (1 X, 8(011)11
15)
with parameters 0, d 2k: and
(over R)
Proof of Lemma A.4. Let A, E Raxd and A₂ E (dd be independent matrices with i.i.d. entries from (0, By stacking A, atop A₂ and then performing a QR decomposition the resulting Gaussian matrix, the orthogonal invariance of the Gaussian law implies that Q is independent of the upper-triangular factor and uniformly distributed on O(d).
For matrix P diag(p), .P.). with i.i.d P; sampled uniformly from {-1. 1}, we have
QP 1 W. Partitioning Q and P into blocks similarly to W, we have Q1 Wn for the top-left block of Q.
The CS decomposition of orthogonal together with invertible R yields the generalized singular value decomposition (GSVD) of the pair (A,,A)):
15

---

Published as a conference paper at ICLR 2026
(A)) (A2)
QQ Q12
R U U U,)
C
)
V7
-Q2z1
-
R
Q21 Oz
V2
where U, V E Ok), U2 V2 E Old - k), and C diag(e --. (g), diag( 82) with 2 O, 8 2 0 in descending order, and ci s = 1 for all i. The diagonal entries of C are known as the generalized singular values of the pair (A,, A2). From this dccomposition and the SVD of Wn UEVT, one has
U,C & UEVT.
Since U V1, and U, V are uniformly distributed on O(k) and independent of C, P we have C = Σ by the invariance of the Haar measure under orthogonal transformations. On the other hand, the generalized singular values { of pair Ag) follow the law of the Jacobi ensemble with parameters a a= 0, = d - 2k, and B = (Edelman and Sutton, 2008, Proposition 1.2). Therefore, the squared singular values of Wn follow the Jacobi ensemble with the same parameters.
Corollary A.5. The squared inner product | 01 between two independent random d-1 Morcover, the shifted inner product (1 + 6)2 vectors 8, Ø ~ Mga-1 follows Beta 2. 2
is symmetrically distributed as Beta 2 2 d-1).
2 and 8 1, the density is proportional to x 211 x) 2) (d-3)/2 on [0, 1]. which matches the Beta { dnd) distribution.
Proof of Corollary A.5. Setting Jacobi parameters 1, 0,6 0,6
Next, observe that T has density proportional to (1 +)= for EE [-1,1]. Under the change variables 17 ~ Beta 1 d
2
B COMPLETE PROOFS
Lemma 4.1. Consider the following pair of jointly Gaussian d-dimensional random vectors:
(x, (x,Y) ~ (x,))>( (0 A)): (-1-1;1).
In this setup, MI and SMI can be calculated analytically:
d 1(x;Y) - log(11 p²), SI(X;Y) 2d
0²
3 + +1,2:
3 1,
2
where, FA is the generalized hypergeometric function. fumcciciio Additionally, the following limits hold:
lim 1(X;)) +
lim SI(X;Y)= d
d-000
(d- (d-1 log 2 25 lim SI(XYY) ( - (5(2 2 2
1
lim 1(X; Y) p²-1
d d-1 1
with & being the digamma function.
Proof of Lemma 4.1. One can acquire I(X;Y) = - log p22) from general expression
for MI of two jointly Gaussian random vectors (see Corollary A.2).
Recall that (0T X, TY) A.2 we have
Y)= is also Gaussian with cross-covariance p Φ. Therefore, by Corollary
SI(X; = (0 x; TT (0.) = - E[looo( pp [0T
From Corollary A.5, we note that 10 0 Beta( 1, d=1), so
16

---

Published as a conference paper at ICLR 2026
SI(X;Y)
log(1 - p2²x: - =²)²²x= dx
2B [ d=1 p²
11 z)) 122 2 (1, 1; p²xx) dz,
1 1; where the last equality follows from the identity log(1 2) 2- (1,
with hyper-
geometric function 249 Appling Buler's integral transform (McBride, 1999, Eq. (2.2.3) gives
0²
r(
+
SI(X;Y)
23 (1-
2F (1,1;2:p dr
2d
p² 1,
3
B
1, 2; 2:
2d
2
Here Bh2 denotes the generalized hypergeometric function.
Finally, we calculate the limit of SI Y) as p² + using properties of bela-distribution. Denoting 17 = (1 + TT x2 ~ Beta d-1 d-1) (see Corollary A.5), we get
2
SI = ---log logg -Elog(1 - 7) = log2 - EEoog = (d - 1) - d (1=11) - -log 2,
2
where vi is the digamma function. Using the bounds digamma function (Elezovic et al., 2000), we get.
log x + 2
1 s (x)) s log (r + -
-
(5)
x
derive an upper bound on this expression:
vid - 1) 2/ d 11-1
log 2 d + log 1 ev(1) -
1
d
To simplify the bound, one can note that ev(1) < O, as (11 <.. Proposition 4.2. Under the setup of Lemma 4.1, k-SMI has the following representation
1 / log( 1 - p²>;) P(A) x II M A (X, =
(1 X₂) (d
(A)
Proof of Proposition 4.2. Let Qx Qy ~ Msi(k.a) Then [ Qrr] ~ NO, E), where Σ is a 2k 2k covariance matrix with the following block structure
(
xQxer) Qtox
I
p
Σ
I
Using the formula for the determinant of a block matrix E yiclds
1 (X;Y) = - Ellog detd)))))))))) - l det (1-
By the invariance of the Haar measure under left and right multiplication, Qk 4 W i where Wn is a & by & left upper block of the matrix W Moid)- According to Lemma A.4, the eigenvalues of W WT, follow Jacobi ensemble with parameters a = =, d - 2h and 1:
p(A)
I
Mr KJ
A
(1-1)
17

---

Published as a conference paper at ICLR 2026
Thus, we get general expression for k-SMT
(I; - /
log(1 - (2A,)p(A)dd
[0.1]* 2-1
Proposition 4.4. Let X and Y be d,., dd-dimensionnl random vectors respectively, with dz,d, k. Let A E Rmxxd, xd=, B Rm xd, be full column rank. Then (Ix;XX Y).
Proof of Proposition 4.4. Using Lemma A.3 and d dy < <h; we get that θ and B are injective with probability one for independent O Φ distributed uniformly on St(d k) and (td,,,)) k). Therefore, according Theorem 3.1, [I(OTAX: BY) 0, $ 1(X;() Y) almost sure. As result, SI,(AX; = 1(00AXX AX; BY O 4) I(X;Y).
Proposition 4.7. (Tsur et al. (2023, Proposition 2)) Let (X ~ (u, Σ), with marginal
covariances Fx, Ey and cross-ovarriacce Σxy. Suppose the matrix xx²x xx xx E exists, and let {p:);-1 denote its singular values in descending order, where d = min d d,y. Then
1 Σ 2 -
I(X;Y)
log(1 - pt),
(x;Y)
log(1 -pt)
[m
Proof of Proposition 4.7. Direct corollary of Corollary A.2.
C GENERAL CASE
While Lemma 4.1 successfully demonstrates severe shortcomings in SMI, it relies exclu- sively on the Gaussian case. Since real-world data distributions can deviate significantly from normality, this section analyzes other scenarios where SMT may or may not exhibit limitations.
We begin with a simple example discrete random vectors X, Y for which (X; = I(X; Y) regardless of k and dimensionality.
Example. C.1. Let X, Y be any discrete pair random vectors. Then, SI (X; Y(:
Proof of Ezample C.1. Because X and Y are discrete, almost every random projection mapping is injective on their respective supports. Since MI is invariant under measurable injective transforms, !(eT,; xx; I(X; Y) for almost all fixed θ and $. Therefore, taking the expectations over & ~ Msukkdz)) θ ~ MSt(k,dy) yields
(I;YY ! x; TT 0 Φ) = =((XY).
However, this example is simple and does not require dimensionality reduction in the first place: when dealing with discrete random vectors, the only constraint is the support size. On the other hand, applying SMI continuous distributions with independent components immediately results in saturation.
Lemma C.2. Let X Ω R be a random vector with i.id. components of unit variance such that h(X,) E < co, and YILX; for i 2 2. Then
h
d
S (X; < k(h(N(O, (0, 1)) - E)
2 1/
(
2
where the RHS is independent of ((;
Proof of Lemma C.2, From the DPI for the Markov chain TT + +Y + X, + OTx, one has
18

---

Published as a conference paper at ICLR 2026
( x; x 6) < 1(0 (0)
= =h(@'x x ()
())
The first one can be upper bounded as follows
x X θ) < x) < ( 1)),
To get a lower bound on the joint entropy, we first rewrite the (&+ -i-denmsion vector as a transformation of the k-dimensional X and perform QR-eecomoosition of the (I: + 1) x d matrix
TT X,
θT T
( rrT |
)
( uT X
X
R
where u ü/ with à (I 00T)e, ee Here R [((--))((-1)) is full-rank upper-trian- gular matrix, and U E St(k + 1, d). Then,
h(oT X, X1 () h(R [R ()) h(U X () + E Eog/det R/.
To lower bound the entropy in the RHS, we make use the result from (Guo et al., 2006, Theorem 3):
h(U X () E [T diag h (X X ho...,,hhh, h(x)))UT) EETr(OO + u =Tu) E (k: + 1).
Noting that det R
/ log ū
1/2 log
||0 e 1 the joint entropy bound is
h(O X1 ( 2 (k: + 1) + 2
E log ||oTe; 11.)
Since ||oTe, e1 11 03 + + 03
2 Zf 23
Bela N.. - with i.i.d. Z, ~ N0, 1), one con-
cludes that
T X; ()) kh((r + Ek
1
1/
k
v
(
2
2
K
d
(0,
1/
w
2
Lemma C.3. Under the assumptions of Lemma C.2 holds
(I; Y) < k const 1
k 1
1 d
log
d
2d'
where "const" is independent of I(X;Y).
Proof of Lemma C.3. By using the inequalities on the digamma function (Elezovic et al., 2000), one has the following upper bound:
1
SI(X;Y) < k const
2/
2/
2
2
2
1
k
d
d d
S l const
log
log
2
2
d
2
1
k
A const
log
1
+
2
d
2d'
We note that both Lemma 4.1 and Lemma C.2 are much stronger than (Goldfeld and Gree- newald, 2021, Proposition 1, part 2): the latter merely states that SI(X;Y) < SI(X;Y). For
19

---

Published as a conference paper at ICLR 2026
instance, given the example from Lemma 4.1, (Goldfeld and Greenewald, 2021, Proposition 1, part 2) yields YI $ 2 I( while our result suggests SI(X; S 2 which does not depend on mutual information. Therefore, the saturation is strong (even 1(X; Y) oo does not break it) and can not be explained solely by non-optimality of projections.
D GAUSSIAN CHANNEL
To explore the SMT's preference for redundant over linearly extractable information, we alyze an additive white Gaussian noise (AWGN) channel. Consider d-dimensional random vector X with cov(X) = I, independent noise Z (0, and the channel output AX + Z, where the matrix A satisfies diag AAT to ensure energy preservation across dimcnsions.
In classical information theory, maximizing I(X;Y) with respect to the input distribution under energy constraints E[x²] = 1 is achieved by X NV I) (Cover and Thomas, 2006). This solution is optimal because decorrelated features provide maximal robustness against isotropic noise. When the transformation matrix A is well-conditioned (i.e. x(A) ||]]- ||A-'| R 1), information about X is spread evenly across the dimensions of Y. In contrast, as shown below, SMI exhibits the opposite preference due to its redundancy bias.
When A I, the channel decouples into independent scalar channels Y X; Z4 this case, linear estimation via the conditional expectation 1-02 E[x; Y]] achieves the optimal mean squared error (MSE), representing the most efficient linear extraction of information.
Contrary to the theoretical optimality of well-conditioned transformations for mutual infor- mation, SMI increases with k(A) as shown in Figure 7. This demonstrates that SMI does not measure linearly extractable information but rather favors redundant information.
SMI's preference for ill-conditioned A (high (A)) arises because such transformations create strong dependencies among the output features. high condition mumber implics that the components of AX become highly correlated, making the same information available repeatedly across different one-dimensional projections.
E RELATION TO OTHER MEASURES OF DEPENDENCE
Estimation complexity
perceived position
copula
actual position
mutual information
our contribution
optimal-SMI
max-SMI
SMI
correlation
Ability capture complex statistical dependencies
In our visual abstract, we position SMT as more complex and capable than correlation analysis but less complex than MI and copulas. In this section, elaborate this ranking.
Copulas provide the most complete description of dependencies of two random vectors. The joint distribution P Px, fully captures probabilistic dependencies, but includes irrelevant information about marginal distributions Px @ Py A copula Cx,y factors out the former w.r.t. the latter by pinning the marginals to be uniform, thus describing the pure dependence structure (Fan and Henry, 2021). While offering full generality, copulas complexity often makes their direct use impractical.
20

---

Published as a conference paper at ICLR 2026
Mutual Information (MT) is measure of statistical dependence, capturing non- linear relationships between two random vectors. It projects the copula onto a scalar summarizing dependence strength:
dx dFx @ Py (X,Y),
((;Y)) E PMI(X;Y) Elog
where the log- log-derivative PMI refers to Pointwise Mutual Information and literally oquals to the copula Cx,y- Thus, MI is functional of the copula (Chen et al., 2025; Ma and Sun, 2011), and if the corresponding PDF exists,
write
((;Y) - h(Cx,r)
Slicod Mutual Information (SMT) estimates the mutual information between two random variables by averaging across one-dimensional projections. It can detect non-linear dependencies. However, as our work demonstrates, SMI saturates prematurely, prefers information redundancy, and asymplotically vanishes as the dimension growth.
Correlation measures linear dependence. It is compatationally efficient, but fails to detect any non-linear relationships.
In summary, our work shows that there exists fundamental trade-off between computa- tional scalability of a dependence estimator and its ability to capture rich, high-dimensional dependencies. We find that SMI, contrary to earlier assumptions, fails to overcome this trade-off. The cost of its computational benefits are misleading biases. While our findings are solid, we would like to emphasize that the visual abstract represents our personal, informal opinion, although it is backed by concrete evidence.
F SELECTING KNN KSG ESTIMATOR
In this section, we the same benchmarks from Section 5 to determine the optimal number of nearest neighbors (kxx) for the KSG estimator (Kraskov et al., 2004). We focus exclusively on plain Mutual Information estimatiion, as is a direct component of the SMI estimation task. For each MI value from 0 to 10 in steps of 1, perform 10 independent runs with 10 samples each. We then compute the median across these runs and use it to derive the Mean Absolute Error (MAE) for different distributions and kxx values. These errors are reported in Table 1. From the results it is evident that kyN = 1 is the best choice on average. This consistent with Figure in (Kraskov et al., 2004), where kNN/ /N / 0 increases accuracy.
samples
Table 1: MAE of the KSG estimates under different distributions and values KNN-
kNN
Distribution
1
2
3
5
10
20
Correlated Normal
1.32
1.47
1.57
1.69
1.87
2.08
Correlated Uniform
1.45
1.59
1.68
1.80
1.98
2.17
Smoothed Uniform
1.42
1.57
1.67
1.80
1.98
2.18
Log-Gas Gammaa Exaamn--xponenialll
0.41
0.52
0.60
0.72
0.91
1.15
G ADDITIONAL EXPERIMENTS
In this section, we conduct supplementary experiments to evaluate SMI under broader range of sctups.
21

---

Published as a conference paper at ICLR 2026
G.1 LOW-DIMENSIONAL SYNTHETIC TESTS
We begin by assessing k-SMI on the same set of benchmarks from Section 5. The results for == 1, 2, presented in Figure Figure 13, and Figure 14, respectively. Notably, satu- ration remains consistent even for k = d - 1 (i.e., when only one component is discarded).
Next, we examine a setup involving randomized distribution parameters, following the methodology of (Butakov et al., n.d.). Among other adjustments, this includes random- izing per-coniponett mutual information (e.g., assigning interactions unevenly in this experiment). In some cases (e.g., the log-gamma--e caammaa--xxooentiie distribution), this increases linear redundancy, as component pairs with higher mutual information also exhibit higher variance in this particular scenario. Our results are displayed in Figure 15.
Due to numerical constraints, we do not track I(XYY//d in this particular setup, instead plotting the results against. the total mutual information. While this makes saluration slightly less evident, the general trend of SMI decreasing with d remains observable. We also highlight the log-gamma-xxpoennian distribution (Figure 15d), where SMT less prone to saturation under parameter randomization due to the reasons mentioned earlier.
G.2 SYNTHETIC IMAGES
Using the MI-preserving smooth injective mappings from (Butakov al., n.d.), we reproduce the synthetic datasets used in (Butakov et al., 2024). These datasets consist of high-dimensional images (sce Figure 3) with known ground-truth mutual information. The results presented in Figure 16 again prove our findings.
G.3 REAL IMAGES WITH SYNTHETIC COPULAS
Rollowing the technique proposed in (Lee and Rhee, 2024), conduct additional experi- ments on the MNIST dataset. We consider the Markov chain:
-
C G
X21
where G and C2 are random class variables, and X1, X2 represent random images drawn from classes G and Cz, respectively. We control the mutual information 1(G; C2)) using the noisy symmetric channel framework from (Lee and Rhee, 2024). If images are selected independently given the class pair, can be shown that 1(X;; X2) = I(C;; C2).
We vary I(C;; C2) from to (og(#classes) (its theoretical maximum) and conduct 10 independent runs. The resulting values of k-SMI, averaged over 10 independent runs, are presented in Table 2. These results also indicate saturation of SMT. Moreover, one can also notice that SMI between independent is non-zero and only twice as small compared to the case IX,; ; X2) 2.3 nats, which highlights the curse of dimensionality.
Table 2: SMI results (in 10-3 nats) for the experiments with MNIST dataset.
1(Xii X2), nats
0.0
0.5
1.0
1.5
2.0
2.3
SI(X;; 10-3 nats 2.89
2.88
3.80
4.68
5.73
5.77
H. REPLICATION STUDY: DETAILS
The original papers on &-SMI and max-SMI feature several experiments independence testing and InfoMax tasks (Goldfeld et al., 2022, Section 5; Goldfeld and Greenewald, 2021, Sections 4.2,4.3; Tsur et al., 2023, Section 5). In this section, we attempt replicate these tests to understand how their results align with our analysis.
22

---

Published as a conference paper at ICLR 2026
Si(X; YY,, mais
SI(X;
2
8
10
2
10
1(x;Y)/d nats
(a) Correlated Normal
(b) Correlated Uniform
d=3
d 3
d
SI(X; Y1, matts
3
d
3
d 8
d= 16
SI(X;Y)
d= 16
2
10
(X),))dd nats
1(X: Y)/d, nsts
(c) Smoothed Uniform
(d) Log- Gamma- taaama--xponnntiall
Figure 13: 2-SMI results on synthetic benchmarks. Mean values and standard deviations
across 10 reported, 10a samples from X, Y and 128 random projections used.
4
d 6
S1XX; Y), nats
8=8 8
d= 8
16
32
SIXX,Y)
32
I(X; ()/d),d nats
()/))/d nats
(a) Correlated Normal
(b) Correlated Uniform
SIXXY) raiis
16
SI(X,Y),
2
10
0
2
k/d:Y)/d
I(X; Y)/d, mats
(c) Smoothed Uniform
(d) Log-Gammas Exponential
Figure 14: 3-SMI results on synthetic benchmarks. Mean values and standard deviations
across 10 runs are reported, 10a samples from X, Y and 128 random projections were used.
23

---

Published as a conference paper at ICLR 2026
SI(XYY), uists
0.6
Si(XY)) nets
0.6
0.4
0
8
10
0
2
8
10
1(X: nats
1(X;))
(a) Correlated Normal
(b) Correlated Uniform
0.8
d 2
d 3 d d 6 d=8 8
SI(X; Y), natt
0.4
SI(X; Y),
10
2
1(X; Y), nats
I(X,Y), nats
(c) Smoothed Uniform
(d) Log- Hammm--xpooentiall
Figure 15: SMI results on synthetic benchmarks. Mean values and standard deviations
across 10 reported, 10a samples from X, Y and 128 random projections used.
10
10
2
k 3
SI(X: Y), nats
k 5 k 7
SIX;Y) Y),
0
2
10
2
10
1(x; V)/d, pats
I(X:Y)dd nats
(a) Gaussian 16 x 16
(b) Gaussian 32 x 32
SI(X;Y); mats
k
51
2
(IX;Y/dd nats
1(:: Y)/d, nata
(c) Rectangles 16 x 16
(d) Rectangles 32 x 32
Figure 16: Results of synthetic experiments with high-dimensional imago- based distribu- tions for K-SMI. We report mean values and standard deviations computed across 10 runs, with 10 samples used for MI estimation and 128 for averaging across projections.
24

---

Published as a conference paper at ICLR 2026
H.1 FEATURE EXTRACTION
Here, we reproduce and elaborate on the InfoMax-like feature extraction experiments. In contrast the tasks described in Section of our work, (Goldfeld and Greenewald, 2021, Section 4.3) considers a supervised feature extraction setting, In this setup, the shared information between f(X) and g(Y) maximized with respect to the functions f and g.
Toy Gaussian example. Here we consider two families of Gaussian baselines: high redundoncy (X,Y") and low redundancy (X, Y"):
X, ~ N00, XIZ = Z Σ1 1 y"= 2 e e
e
where 1 ..., 1) and m controls the number of components that are injected into Y. Setting d d= 10 and = 1 for (X, Y') recovers the experiment from (Goldfeld and Gree- newald, 2021). However, to highlight SMI's deficiencies, we will adhere to the low redundoncy benchmark, according to which proper feature extraction should result in the selection of at least m features.
In experiments, closely follow the setup from (Goldfeld and Greenewald, 2021): when maximizing k-SMI, we use linear f and parametrized by Rdxd matrices. However, when extracting features through MI and max-SMI maximization, we have form dimension- ality bottleneck by using Rixd matrices: otherwise, the best strategy is to extract every feature. As we show below, SMI docs not require this bottleneck, because it implicitly biased toward degenerate solutions.
Similar to Section 6, variational representations are employed to conduct the experiments: the NNs from Section T are trained for 100 epochs; other settings the same.
To evaluate the quality of the extracted features, we compute the effective rank of the matrices Ammm) [im) formed by the first columns of A and B respectively. The effective rank is defined erank M exp(H(a)), where H(o) is the Shannon entropy of the normal-
Figure 17: Feature extraction matrix for the low redundancy setting acquired through k-SMI + max for m (1,2,3,4 4) (columns) and k {1,4) 4) (rows).
Figure 18: Feature extraction matrix for the low redundancy selting acquired through MI + max for E {1,..., 4} (rows).
25

---

Published as a conference paper at ICLR 2026
Figure 19:
Feature extraction matrix for the low redundancy setting acquired through 1-max-SMi SMIx--MI-- + max for m E {I,..., 4] (rows).
ized singular values a of M. crank ((1m) & m, then all features are extracted without mixing, while low values of erank A (1:m) indicate the (mumerically) irrecoverable collapse to mixtures (ill-posed linear combinations of the first m components X).
Our results, depicted in Figure 9, show that the MI maximization yields effective rank close to m, confirming its ability to recover all relevant fcatures. In contrast, k-SMI yields an effective rank that nearly constant regardless of k, revealing its redundancy bias. This collapse confirms that &-SMI optimization leads to redundant features.
H.2 INDEPENDENCE TESTING
The SMT has been proposed as scalable alternative to MI for independence testing (Goldfeld et al., 2022; Goldfeld and Greenewald, 2021; Nuradha and Goldfeld, 2023; Tsur et al., 2023), which be framed a binary classification task. Given estimates of SMI (or MI) on datasets drawn from either the joint distribution (positive class) or the product of marginals (negative class, obtained by shuffling), one can apply the threshold for dependence verification. For each fixed dimension d, and sample size n, we generate 100 positive and 100 negative pairs of samples, estimate SMI (or MI), and compute the ROC-AUC over these 200 scored examples
function of the number of samples The works (Goldfeld et al., 2022; Goldfeld and Greenewald, 2021) shows that SMI outperforms MI when the dimension is fixed.
We replicate this protocol with one critical modification. We pool estimates across different dimensions (d (2, 10, 20, 30)) for each sample size n, and then compute a single ROC-AUC from the mixedldiimensional data. Additionally, we fix the ground truth MI to 1 and 2 nat for each dataset and replace the Kozachenko-Leonennkoo estimator used in (Goldfeld and Greenewald, 2021) with the KSG estimator (Kraskov et al., 2004) (using hxn neighbors), which in our experiments yields more stable MI estimates. For fair comparison we report MI values over 128 random rotations, because it showed numerically improved MI estimates for small-size datasets.
ROC-AUC
0.9
1-SMI
2-SMI
0.9
0.9
0.9
4-SMI
0.8
0.8
0.8
0.7
0.7
0.7
0.7
0.6
0.6
10
100
1000
10
100
1000
10
100
1000
10
100
1000
Figure 20:
Independence testing: ROC-AUC versus sample size for correlated normal,
correlated uniform, smoothed uniform and oogggammma--eponeniiall (left-to-right, 1 nat).
As shown in Figures 11, 20, this pooling causes SMT's discriminative power to drop sharply, while MI's remains high. The failure occurs because SMI decays with dimension even when total mutual information is held constant, so dependent high-dimensional cases produce SMI values that overlap with independent ciw-dimensional cases. The slower dimensional decay of SMI for LGE distribution (Figures 5, 15), in turn, explain the observed higher ROC-AUC, Consequently, SMI less reliable for independence testing than MI unless the
'By using the KSG estimator, we observe that ROC-AUC dynamics corresponding MI come into closer agreement with those of SMI, which not seen when using the less stable Kozachenko- Leonenko estimator.
26

---

Published as a conference paper at ICLR 2026
dimensionality is known and fixed in ad adnance, which imposes strict limitation for practical applications where data dimensionality may vary,
I IMPLEMENTATION DETAILS
1.1 SYNTHETIC EXPERIMENTS
For the experiments from Section 5, we implementation of Kraskov-Stoegbbaue-Graa berger (KSG) (Kraskov et al., 2004) mutual information estimator and random slicing from (Butakov et al., n.d.). The number of neighbors set to nNN = for the KSG estimator. For each configuration, we conduct 10 independent runs with different random seeds to compute means and standard deviations. Our experiments use 10* samples for (X, Y) and 128 samples for (e, .)
For the experiments from Section 5, we use independent components with equally distrib- uted per-component MI. For the supplementary experiments from Figure 15, parameters of each distribution (e.g., covariance matrices) are randomized via the algorithm implemented in (Butakov al., n.d.). This includes randomization of per-component MI (which done using uniform distribution over a (d - 1)-dimensional simplex).
For the experiments, we used AMD EPYC 7543 CPU, one core per distribution. Each experiment (fixed k, varying d) took longer then 3 days to compute.
1.2 REPRESENTATION LEARNING EXPERIMENTS
Recall that Deep InfoMax requires maximizing lower bound ((;J(()) where X is input data and f is an encoder network. Since ((X) is typically vacuous, the lower bound question should be selected carcfully to (a) be finite and (b) allow for meaningful optima. In our experiments, we employ the objective from (Butakov et al., 2025), which provably satisfies the requirements above, while also being inherently regularized against representation collapse:
[ff('); f(X) + Z) s ((x:f(x)),
where Z is Gaussian and independent, and X' represents randomly augmented data.
For experiments on MNIST dataset, we use simple ConvNet with three convolutional and two fully connected layers. A threo-layer fully-connected perceptron serves critic network for the InfoNCE loss. We use the same architecture and loss for SMI maximization. As described in (Goldfeld al., 2022: Goldfeld and Greenewald, 2021), the critic network for the SMI lower bound takes O X, & Y θ and $ as inputs. To accommodate the flattened θ and matrices, we increase the network's 's input dimensionality; the rest of the architecture remains unchanged. The details are provided in Table 3. When maximizing SMI, generate a set of random projectors for each batch of samples from X,Y, with one projector per sample.
We use additive Gaussian noise with = 0.2 as an input augmentation. Training hyperpa- rameters are as follows: batch size 512, 2000 epochs, Adam optimizer (Kingmaa and Ba, 2017) with learning rale 10-3
For the experiments, used AMD EPYC 7543 CPU and Nvidia A100 GPUs. Each experiment took no longer then 1 day to compute.
27

---

Published as a conference paper at ICLR 2026
Table 3:
The NN architectures used to conduct the tests on MNIST images in Section 6.
NN
Architecture
ConvNet,
x 1: Conv2d(1, 32, kx=3), MaxPool2d(2); BatchNorm2d, LeakyReI.U(Q.00))
x 1: Conv2d(32, 64, ks=3), MaxPool2d(2), BatchNorm2d, LemkyReLU(00001
24 x 24
1: Conv2d(64, 128, ks=3), MaxPool2d(2). BatchNorm2d, TeakyRel.U(0.01)
images
1: Dense(128, 128), LeakyRelUU(0.00) Dense(128, din)
Critic NN for MI, pairs of veclors
1: (2nse(2 dim, 256), LeakyReLU(0.01) Dense(256, 256), (0.0)) Dense(256, 1)
Critic NN for SMI, pairs of vectors
X 1: Dense(2 x k + 2 X dim x k; 256), LeakyReLU(0001) 1: Dense(256, 256), LeakyReLU(0.00)) Dense(256, 1)
28


Reviews:
[{"review_id": "1qagbQEqON", "reviewer": "Reviewer_AoMn", "summary": "This paper gives a critical analysis of a cheap and intuitive proxy measure for statistical dependence called \"sliced mutual information\" (SMI). \nAlthough the simplicity of SMI has fueled its growing popularity, this study finds that the name is deceptive and that SMI has a number of drawbacks that make it inappropriate as a proxy for statistical dependence measures like Mutual Information: SMI saturates as dependence grows, SMI is biased towards redundancy, SMI decays to zero in high dimensions. The paper builds analytic examples and uses synthetic data to demonstrate the issues, and shows that simple workarounds do not fix the problem.", "questions": "- I didn't go back to study the IB paper that you cited that optimized SMI. Your result suggests that it shouldn't work, is there any explanation on how they were able to show reasonable results to publish with this method? \n\n- One small improvement for a reader like myself would be to situate the SMI literature with respect to the neural MI estimation literature. I just assumed that it was clear that neural MI estimation methods, like neural everything else, was the clear winner. Are there applications / properties / uses cases which led people to prefer SMI over neural methods? I assume it's just for computational simplicity?\n\n- One other thought that might extend the impact of this work is to discuss whether there are connections to other \"sliced\" estimators, like sliced score matching. Even the popular Hutchinson trace estimator could be considered a sliced estimator.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper was well structured. SMI and its defects were clearly described, with intuitive experiments supporting each result. \n\n- While most of the arguments were supported by analytical counter-examples, I was happy to also see comparisons with recent synthetic data benchmarks used in the MI estimation literature. \n\n- I appreciate that the paper went beyond showing examples where SMI gives counter-intuitive results, and also showed that *optimizing* SMI leads to poor results. \n\n- The deficiencies of SMI (like redundancy bias) were not very surprising to me, but the curse of dimensionality effect was quite a bit stronger than I expected. (I assumed that as dimensionality grows, it would be hard to find a \"good slice\", but it seems that even analytically integrating slices leads to decaying SMI!)", "weaknesses": "I thought the critique was straightforward and clear. The only question in my mind is the broader significance of these results. I am familiar with recent neural MI estimation literature, and I confess I had not seen any mention of SMI (though I think I recall a reference to it) so I was a little surprised to see it described as a \"popular\" approach. Nevertheless, the citations don't lie, and a decent number of papers in top venues are studying something that is (as I am convinced by this paper) a dubious measure. Therefore, even though I don't think this has the broadest significance for the field, the critique should be at least as visible as the flawed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9k6NCSCUmY", "reviewer": "Reviewer_KQcg", "summary": "This paper discusses the shortcomings of Sliced Mutual Information (SMI) as a tool for measuring statistical dependence in high-dimensional settings. The authors show that SMI can saturate as the correlation between random vectors increases and decreases asymptotically as dimension grows. The authors validate the theoretical findings with synthetic experiments.", "questions": "- In the numerical experiments, are you considering a finite-data regime or infinite-data regime? I did not understand it from the paper, but it is well-known that, for instance, MI estimators perform differently depending on the regime. In any case, how does SMI perform in the other regime that you did not consider? \n- How do you explain the saturation phenomenon of SMI for non-Gaussian scenarios?\n- Since you compare correlation coefficients, SMI, MI, and copula in the initial figure of the paper, why did you not include any other observation or comment on these measures? Can you provide a paragraph of comparison?\n- It appears that high dimensions pose a theoretical problem with SMI in the Gaussian case. However, SMI was used specifically for high-dimensionality problems. So, could it be that SMI does not have the same saturation problem for high dimensions in different scenarios? Did you find any settings in which the dimensionality was not a problem for SMI, and that are not reported in the paper?\n- What does << mean in your case (line 114)?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written\n- The paper presents mathematical proofs of SMI’s limitations in the simple Gaussian scenario\n- The paper highlights major flaws of a novel dependence measure", "weaknesses": "- The theoretical results are limited to a Gaussian setting (Lemma 4.1). This does not prove anything for more complex scenarios\n- Since the theoretical part is limited to the Gaussian setting, one would expect the experimental results to be comprehensive of significantly complex scenarios for which the authors did not provide a theoretical contribution, to show that the paper contributions hold true for many possible cases. However, the experimental results could be improved", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OkcfSuf3iR", "reviewer": "Reviewer_ZesV", "summary": "This paper presents an analytical and empirical critique of Sliced Mutual Information (SMI), a popular scalable alternative to mutual information (MI) used in various works on high-dimensional statistical dependence estimation and deep learning analysis. The authors argue that despite its recent adoption, SMI exhibits several fundamental flaws that make it unreliable as a measure of dependence, including the following:\n1) SMI rapidly saturates even for simple synthetic problems, failing to reflect true increases in dependence.\n2) SMI prioritizes redundant or repeated information rather than informative content.\n3) Although promoted as dimension-robust, SMI actually decays to zero asymptotically in high dimensions.\n4) SMI can increase under deterministic mappings, unlike MI.\n\nThrough theoretical analysis and experiments with synthetic data, the paper demonstrates these deficiencies and shows that using SMI as a replacement for MI is, at the very least, problematic.", "questions": "Less questions, more summarizing comments:\n\nMy preliminary assessment is that this a somewhat niche topic (papers on “sliced mutual information” seem fairly sparse, according to Google scholar), and so I might be less inclined to accept the paper.  However, some of the papers have appeared in prominent conferences (such as NeurIPS), and as such it seems like an important outcome to have the full story available for other researchers to examine.\n\nPerhaps my main question is, given the paper's results, whether the authors feel that SMI currently has any applications settings where it would still seem redeemable, or if it is back to the drawing board. Though that is more of a personal interest question -- I'm not sure I would want to suggest the authors should be required to make a strong statement in the paper, when I think their analysis lays out concrete issues clearly.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Thank you for the paper.  It was an interesting read.\n\nThe paper has both detailed theoretical analysis with several key examples (such as closed-form analysis for certain classes of Gaussian variables) and a large number of synthetic experiments to validate their theoretical findings and conceptual contribution.  \n\nThe paper provides an appropriately critical assessment, in a timely and important way.\n\nThe paper is well written and nicely organized.", "weaknesses": "The following weaknesses are suggested, but they aren't (in my mind) especially significant.  \n\nThe claim that SMI fails to detect increases in dependence even for linear transformations that enhance information extraction should be clarified to have been shown for specific cases, not necessarily universally.  \n\nA specific real-world example would be interesting.  \n\nThe math in this paper is detailed/hard and probably beyond many readers, but not sure that can be helped.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N9XMHVxZei", "reviewer": "Reviewer_pnxw", "summary": "Theoretical properties of sliced mutual information (SMI) are studied, suggesting strong limitations of SMI in application scenarios in which it has been employed in the past, e.g., the Deep InfoMax setting. Furter, discrepancies between SMI and conventional mutual information are illustrated, that is, SMI between deterministic Gaussian variables is bounded while MI is infinite. For increasing dimensionality SMI approaches zero, while again, MI is approaching infinity. The theoretical analysis has been supported with simulation experiments.", "questions": "Questions:\n- Are there any usecases for which the authors would recommend to use SMI instead of MI?\n- Is the proof of Lemma A.4 just recited from 41, or are there any new contributions in it?\n- Do the works that used SMI for Deep InfoMax [22,23] employ any architectural restrictions or similar to avoid collapse?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper highlights severe limitations of SMI that are relevant for practitioners.\n- Theoretical results for multivariate Gaussian variables have been derived which illustrate the counterintuitive behaviour of SMI compared to MI.\n- The main theoretical result derived in Lemma 4.1 has been supported with sufficient experimental evidence.\n- The usecase of Deep InfoMax has been investigated at the example of a small image dataset.", "weaknesses": "While the discrepancy between MI and SMI is well-illustrated, the claims regarding the Deep InfoMax principle are not as strongly supported by the experiments:\n- The description provided in D.2 explains how to obtain the baseline but more details regarding the implementation of SMI are needed.\n- The results are only obtained for a relatively small image dataset (MNIST). To support the strong impact statement made by the authors, more complex datasets such as CIFAR10, or similar ones should be analyzed.\n\n Some experimental details are missing: \n- How exactly was $SI_k$ implemented for the Deep InfoMax experiment? \n- How many seeds did the authors consider to confirm their results? \n- Why is $k=1$ for the KSG estimator? Typically a higher $k$ such as 5 or 7 is more stable.\n\nMinor: \n- In their discussion of the Deep InfoMax experiments, the authors should mention that, e.g., for invertible $f$ we get that $I(X;f(X))$ is infinite and we can only compute it if $f$ fulfills certain properties that avoid this behaviour. \n- Punctuation missing in Lemma A.3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)