Sliced Mutual Information: A Scalable Measure of Statistical Dependence
Authors: Ziv Goldfeld
Kristjan Greenewald
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: mutual information
sliced mutual information
curse of dimensionality
feature extraction
independence tests

Abstract:  Mutual information (MI) is a fundamental measure of statistical dependence, with a myriad of applications to information theory, statistics, and machine learning. While it possesses many desirable structural properties, the estimation of high-dimensional MI from samples suffers from the curse of dimensionality. Motivated by statistical scalability to high dimensions, this paper proposes sliced MI (SMI) as a surrogate measure of dependence. SMI is defined as an average of MI terms between one-dimensional random projections. We show that it preserves many of the structural properties of classic MI, while gaining scalable computation and efficient estimation from samples. Furthermore, and in contrast to classic MI, SMI can grow as a result of deterministic transformations. This enables leveraging SMI for feature extraction by optimizing it over processing functions of raw data to identify useful representations thereof. Our theory is supported by numerical studies of independence testing and feature extraction, which demonstrate the potential gains SMI offers over classic MI for high-dimensional inference.

Sliced Mutual Information:
A Scalable Measure of Statistical Dependence
Ziv Goldfeld Cornell University goldfeeldecorell.eee
Kristjan Greenewald
MIT-IBM Watson Al Lab kristjan. h www.eggeeeeeweeeeeeeeeeee d 1em...deemmmmmmmmmm com
Abstract
Mutual information (MI) is fundamental measure of statistical dependence, with myriad of applications information theory, statistics, and machine learning. While it possesses many desirable structural properties, the estimation of high- dimensional MI from samples suffers from the curse of dimensionality. Motivated by statistical scalability to high dimensions, this paper proposes sliced MI (SMI) as surrogate measure of dependence. SMI is defined as an average of MI terms between one-dimensional random projections. We show that it preserves many the structural properties classic MI, while gaining scalable computation and efficient estimation from samples. Furthermore, and in contrast to classic MI, SMI can grow as a result deterministic transformations. This enables leveraging SMI for feature extraction by optimizing over processing functions of raw data to identify useful representations thereof. Our theory is supported by numerical studics of independence testing and feature extraction, which demonstrate the potential gains SMI offers over classic MI for high-dimensional inference.
Introduction
Mutual information (MI) is universal measure of dependence between random variables, defined as
dPx.y dPx,Y.
(x:))= xxxy log
(1)
dPx & Pr
where (X Px, and dP/dQ is the Radon-Nikodym derivative of P with respect to (w.r.t.)
Q. It possesses many desirable properties, such as meaningful units (bits or nats), nullification and only if (iff) X and Y are independent, convenient variational forms, and invariance to bijections. In fact, MI can be obtained axiomatically as unique (up to multiplicative constant) functional satisfying several natural 'informativeness" conditions [ As such, it found variety of applications in communications, information theory, and statistics [1 2 More recently, it was adopted as figure of merit for studying 3 4
and designing 19] [10 [ا] machine learning models.
MI functional of the joint distribution Рx,Y the considered random variables. In practice, this distribution is often not known and only samples from it are available, thereby necessitating estimation of 1(X;; Y). While this topic has received considerable attention
16. 18, MI
is fundamentally hard to estimate in high-dimensional settings due to exponentially large (in dimension) sample complexity [9]. Motivated by statistical efficiency in high dimensions and inspired by recent slicing techniques of statistical divergences 20 Pll 24] this paper introduces sliced MI (SMI) as a surrogate notion of informativeness. We show that SMI inherits many of the properties of classic MI, while allowing for efficient estimation. Furthermore, in certain aspects, SMI is more compatible with modern machine learning practice than classic MI. In particular, deterministic transfornations of the random variables increase SMI. e.g., if the resulting codomains have more informative slices (in classic MI sense ). This enables using SMI as figure of merit for feature extraction by identifying transformation (e.g., NN parameters) that maximize it.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

1.1
Contributions
SMI is defined as the average of MI terms between me-dimensional random projections. Namely, gd-1 denotes the d-dimensional sphere (whose surface area is designated by Sd-1), we define
SI(X;Y) = Sd,- = - { gMx-1
1
I(0TX; X; TY))ddd
(2)
We may similarly define max-sliced version by maximizing over projection directions, as opposed to averaging over them. Despite the projection of X and Y single dimension, SMI preserves many of the properties of MI. For instance, we show that SMI nullifies iff random variables are independent, it satisfies a chain rule, can be represented as a reduction in sliced entropy, admits a variational form, etc. Further, SMI between (jointly) Gaussian variables is tightly related to their canonical correlation coefficient [24]_ This is in direct analogy to the relation between classic MI of Gaussian variables (X, Y) and their correlation coefficient, where ((;; Y)
log (1 (x Y)).
SMI is well-positioned for statistical estimation in high-dimensional setups, where one estimates SI(X; Y) from n i.i.d. samples of Px,y. While the error of standard MI (or entropy) estimates, e.g., those in [16 25 26] scales as n 1/d when d is large, the same estimators admit statistical efficiency when d = 1.2, converging at (near) parametric rates. Combining such estimators with Monte-Carib (MC) sampling to approximate the integral over the unit sphere, we prove that the overall error scales (up to log factors) as m 1/2 -1/2 + n n1/2, where m is the number of MC samples and n is the size of the high-dimensional dataset. We validate our theory on synthetic experiments, demonstrating that SMI is a scalable alternative to classic MI when dealing with high dimenssiin data.
A notable contrast between classic and sliced MI involves the data processing inequality (DPI). Classic MI cannot grow as a result of processing the involved variables, namely, I(X;Y) 2 I(f(x); Y) for any deterministic function f. This is since MI encodes arbitrarily fine details about (X, as variables in the ambient space, and transforming them cannot reveal anything that was not already there. SMI, on the other hand, only considers one-dimensional projections of X and Y, some of which can be more correlated than others. Consequently, SMI can grow result of deterministic transformations, i.e., SI(X;Y) < (I(((X); Y) is possible if projections of f(X) more informative about projections of Y than those of X itself. We show theoretically and demonstrate empirically that SMI is increased by projecting the data on more informative directions, highlighting its compatibility with feature extraction tasks.
2 Preliminaries and Background
We take P(IRd) as the class of all Borel probability measures on Rd Elements of ((R))) are denoted by uppercase letters, with subscripts to indicate the associated random variables, e.g., Px or Px,.. The support of Px is supp ). Our focus throughout is on absolutely continuous random variables; we use lowercase letters, such as px px,Y, to denote probability density functions (PDFs). For function f d + Rd and a distribution Px E P ) we write fyPx for the pushforward measure of Px through f, i.e., (A)
Px (A).. The IP(RR.) norm of is denoted by |I/ The d'dimensional unit sphere Sd-1, and its surface area is Sa-1 = 2/2/2 (dd/2)) with I' as the gamma function. We also define slicing along 9 as 7" (xx == fT..
Mutual information and entropy. Information measures, such as MI and entropy, are ubiquitous in information theory and machine learning. MI is defined in | and can be equivalently written in terms of the Kullback-Leiblrr (KL. divergence as I(;;Y)= = DKL (Px.y || x & py). ). 1(X; Y) thus quantifies how far, in the KL sense, (X, ~ Рx, are from being independent. The differ- ential entropy of continuous random variable X with density px is H(X) == -R. log (px (X)) quantifying a notion of uncertainty associated with X. For a pair (X, ~ Px... the conditional entropy of X given Y is H({ == Sy H(X|Y = v)dPy (v), where H(X|y = y) is computed w.r.t. Pxjy=y. Conditional MI is similarly defined as 1(X; 2| Sz 1(X; YIZ 2 2))P2(2).
With these definitions, one can represent MI a (X;
H(X) - H(X H(Y) H(Y|X),
thus interpreting MI as the reduction in the uncertainty regarding one variable as a result of observing the other. Another useful decomposition is the MI chain rule I(X,Y:Z) (Xx;))+((;21) +|x:
Assuming that the appropriate PDFs exist.
2

---

Data processing inequality. The DPI states that I(X:Y) > if + Markov chain. This inequality is comnerstone for many information theoretic derivations and is nat- ural when there are no computational restrictions on the model. However, given a restricted computa- tional budget, processing the input may aid inference. Deep neural network classifiers are an excellent example: they generate a hierarchy of processed representations of the input that are increasingly useful (although not more informative in the Shannon MI sense) for inferring the label. The incom- patibility between the DPI and deep learning practice was previously observed in PT motivating their definition of a computationally restricted MI variant that can be grow from processing. As we SMI also shares this property.
+ Z forms
show in Section 3.2
3 Sliced Mutual Information
Our goal is to define surrogate notion of MI that is more scalable for computation and estimation from samples in high dimensions. We propose SMI as defined next.
Definition 1 (Sliced MI). Fix (X,Y) Рx, E P(RR IB.)). Let θ Unif(Sd=-1) and ~ (S4v be independent of each other and of (X,Y). The SMI between X and Y is
$ SI(X;Y) = ((TTX;0TYO00)) Sd, 1Sd, 8d gd,
1
(3)
Evidently, the SMI between two high- nigh-dimensional variables is defined as an average of MI terms between their one-dimensional projections. By the DPI, SI(X;Y) I I(X;Y) we inherently introduce information loss. Nevertheless, we will show that SMI inherits key properties of MI such as discrimination between dependence and independence, chain rule, entropy decomposition, etc.
Remark 1 (One-dimensional variables). If da = then SO = {+1} and we have SI(X; YY) 1(;; (T))), which follows by invariance of MI to bijections and the independence of (X, Y, $) and e. Similarly, when de dy 1 we have SI(X;Y) I(X;Y).
Remark 2 (Single projection direction). When slicing statistical divergences, like the Wasserstein distance 120). one rypically considers single slicing direction. Namely, given that X and Y are of the same dimension d, they are both projected onto the same 0 E @dd and the distances between @T X and Y are then averaged over the unit sphere. While this approach is possible also in the context of SMI, we chose to define it using two slicing directions, O and 6, for several reasons:
1. this definition is invariant to rotations of the spaces in which X and Y take values- a single direction, rotating either space would change the SMI, seemingly undesirable property for information measure;
2. it gives rise to a crucial property of SMI, that YI(X;Y)= = X and Y are independent (see Proposition D. which does not hold with single slicing direction
3. it fares naturally with variables of different dimensions (although one can use zero padding to circumvent this issue in the single-direction version); and
4. it is inspired by the canonical correlation coefficient (24) that also uses two projection directions.
To later establish chain rule and entropy- based decompositions, we define SMI between more than two random variables, conditional SMI, and sliced entropy.
Definition 2 (Joint and conditional SMI). Let (X, Y,,Y 2) Px,Y, E P(Rdd Rdy x Rd))) and take θ Uni,(8d,-1) & Unif Undu-y) and y Unif(Sd.-1) mutually independent. The SMI between (X,Y) Y) and Z is defined as
S1(XY;20 = 1( OT XT 2
(4a)
The conditional SMI between X and given
SI(X; Y 2) = 1 x; t e, 42 V. VTZ)).
(4b)
'Indeed, let X2 ~ (0, 1) be independent, set X X2) and Y - (X2, T. As 2- dimensional vectors, X and Y are dependent, but one readily verifies that cov X, @TY) 0, for any 0 E s!. This implies independence of 0T X and 0T hence the single slicing direction SMI would nullify in this case.
3

---

The expression in 4a) extends Definition 11 Conditional SMI is the sliced information given access to another projected random variable along with its projection direction. Accordingly, conditional SMI is in the spirit of the original definition of Y)) incorporating only projected data without introducing additional uncertainty about the direction.
Remark 3 (Extensions). Joinl and conditional SMI huve natural multivariate extensions. For example, SI(X,,.... Xni YI,..., Ym) : [( OT Xn; Y $T fm, 4), where θ= == 1 O,)) and + (11 ... 2.). The extension of conditional SMI is similar:
Definition 3 (Sliced entropy). Let (X ~ Рx, E P( x Rd) and take e Unif(S4=-1) and 4- ~ Unif(S4,-)) to be independent. The sliced entropy of X is SH(X) == H(@TX|O), while the conditional sliced entropy of X given SH(X == H( x $ TTY).
Sliced entropy is interpreted as the average uncertainty in one-dimensional projections of the con- sidered random variable. Conditional sliced entropy is the remaining uncertainty when projected version and the projection direction of another random variable is revealed.
The following proposition shows that SMI retains many of the properties of classic MI. Proposition 1 (SMI properties). The following properties hold:
1. Non-negativity; SI(X; Y) 2 0 with equality iff X and Y are independent.
2. Bounds: inf degds-1 1(TTX; QTY) < SI(X;Y) S sup (00 x 68811-1
6E8%
particular, 3. KL divergence: We huve SI(X;Y) Y) Ee,0 DKL ((79, n )==))xPP,P))) 1x 8 (
4. Chain rule: For any random variables X XI,..., Xn, Y, we have the decomposi- tion ,I(X,,..... X,: Y) YI + -E' SI(X;; y|x,,...;
SI(X, 2) SI(X; 2)= + ;I(Y;
5. Tensorization: Suppose that 1.)....,.. (Xn, Yn) are mutually independent. Then SI(X)..... Yi........ Y,, X SI(X;;
The proof of Proposition M is given in Supplement A.
Remark 4 (SMI versus MI). Proposition 7 shows that SMI inherits many of the favorable properties of classic MI. Nevertheless, we stress that SMI is posed as a new measure of dependence that (although closely related) is different from MI. In particular, the gap between MI and SMI may not be b SMI thus should not be treated as a proxy of MI, but rather as an alternative figure of merit. The premise of the SMI framework that its meaningful structure does not translate into computational or statistical inefficiency. Indeed, Section 3.1 shows that SMI can be efficienlly estimated with parametric rate (up to logarithmic factors).
Similarly to MI, the sliced version simplifies when the variables are jointly Gaussian. Example (Gaussian SMI). If X ((0,Ex) and Y _N(0, Ex) are jointly Gaussian with cross-covariance Exy, then
$ f log 1- OTY) 88 -
1
SI(X;Y) =
dodø
Ev is the correlation coefficient of 0T and Y. Denoting by PCCA (X,Y) Y) := SUD p(TT X, TTY) the canonical correlation coefficient, we get SI(X; Y) < -0.5 log (1 (x.
where p(0TX, (TY) = g
The Gaussian distribution also special for sliced entropy, where, for classic entropy, it maximizes SH(X) under a fixed (mean and) covariance constraint.
Proposition 2 (Gaussian maximizes sliced entropy). Let Pi Σ) == {P E P(R supp =
Rd [X] = μ, [((x- x) - -m)T] 2). Then arg maxpEP.(u.E) SH(P) = Ni,, )) iee. the normal distribution maximizes sliced entropy inside Pi(u, E).
'See the Example from the beginning of Section 3.2 and note that under that setup, for any 0 < < 00. we
have (x): r) co while (1(ga ((9g((X: Y) is finite.

---

The proposition is proven in Supplement A.2 lished. Specifically, we show that (i) the uniform distribution on the sphere maximizes sliced entropy over measures supported inside ball; and (ii) the symmetric multivariate Laplace distribution [28 is
where two additional max- max-entropy claims are estab-
the maximizer subject to mean constraints on the d-dimensional variable and its projections.
Lastl, SMI admits variational form in the spirit of the Donsker-Varadhar representation of MI.
Proposition 3 (Variational form). Let Unif(Sd. & Unif(S -1) be independent of each other and of (X,Y) ~ Px, E P(R x R.)), and set (X, Y) Px @ Py. We have
log (B (B(cel00000x00)))))
SI(XYY)
sup g: 1 x8 -R
E [[g(O, T 0,
where the supremum is over all measurable functions for which both expectations are finite.
This representation is leveraged in Section 4.3
to implement feature extractor based on SMI neural of the supplement.
estimation. The proof is found in Section A.3
3.1 Estimation
main virtue of SMI is that its estimation from samples is much easier than classic MI. One may combine any MI estimator between scalar variables with an MC integrator to estimate SMI between high-dimensional variables without suffering from the curse of dimensionality. This gain is expected as SMI is defined as average of low dimensional MI terms.
For PA.B E ((R x RR,, let (A1, Bi)..... (An, Br) be pairwise i.i.d. from PA.B. Consider an MI estimator i: A 8 Rgo that attains ő(n) absolute error uniformly over a class F of distributions:
sup PA.
| [[<A", (") I(A: B]] -
S 8(n).
(5)
We use to construct an estimator of SMI. Given high-dimensional pairwise i.i.d. samples (X1, Yi, (Xn; from Рx, E P(R x (d)), first note that (ØT X;, oTY,), for 0 E gd. and E $4 is distributed according to (x" T²); Px,y Thus, can convert {(X,, Y.) into pair- wise i.i.d. samples of the projected variables. Let θ II...., Om and $ .,,..., Pm be i.idd according to Unif Uni and Uni respectively, set (0]x)* = (o]X....., 0,x,) for I,.... m,
and similarly define (T)Y)". We consider the following SMI estimator:
S1m,m 5l,, (X", Y 0
Σ (((0)x)',
(6)
Pseudocode and computational complexity for S1n, can be found in Section of the supplement.
3.1.1 Non-asymptotic performance guarantees
We now present convergence guarantees for the estimator over the following class of distributions:
SuPro.ones4--issss-11 1(0T (T))) S M,
(d,,,(M) = Px,y P(Rd, Rd):
(T (x Sd. sd. Sd,-1
i.., the class of all Px,y with bounded MI and projections that belong to the class F from Theorem (Convergence rate). The following uniform error bound over Fa. (M) holds:
E. R[[SI(X:V) -Sttmmmm))mmm))
M 2 + 5(n).
sup
Ex (M)
See Supplement A.4 for the proof.
Remark 5 (Instance deeendent bound). An inspection of the proof of Theorem reveals that y can be converted into the instance-deeennden bound
[((XYYY - -51m,ml] S
(0TT;0TY)) 10--1/ 1/2 +(n);
sup (0.6) 1x34,
so long that Рx, Far.d, (M) for some M.
5

---

Theorem applies to classes of distribution with uniformly bounded per-slice MI. Since this boundedness may be hard to verify in practice, we present primitive sufficient condition for 7 to hold. Specifically, when Px,y is log-concave and symmetric, it is enough to require that canonical correlation coefficient of (X,Y) is bounded. Recall that a probability measure X 1,
P ((RR)) is called log-concave if for any compact Borel sets A and and < have P(AA+ (1 A) P(B)
1-> Let
(LC) deddy (M) = PPx, E P(R xR()) Px.y is symmetric and log-concave, Eèca (X, Y) $ M).
The following error bound proved in Supplement A.5
Corollary 1 (Convergence for log-concave class). The following uniform error bound over
F(LC) do (M) holds:
-1/2 + +(()).
sup SI(X;YY -SAYY)-Shnny Smn.ml| S
π
log
(8)
Ex da (M)
8
1 M
3.1.2 End-to-end SMI estimation guarantees over Lipschitz balls
To provide concrete SMI estimator with guarantees, instantiate the low-dimensional MI estimate via the entropy estimator from [26] for densities in the generalized Lipschitz class.
Definition (Generalized Lipschitz class). For d E N. p [2, 00), s > O, and L 2 let Lips, ( be the class of probability density functions f [0, 11dd
+ R with II I L, where
II = IIflpppd + supt
sup
|| fl 111pdd
(9)
120
EER llellas1
and E
(k-
We note that the norm of A!:1 is taken over the whole Euclidean space to enforce a smooth decay of f at the boundary. Consequently, the Lip..,.a(L) class includes, e.g., densities whose derivatives up to order [s] all vanish the boundary, where s the smoothness parameter.
Differential entropy estimation over (... was considered in (26]. where an optimal estimator based best polynomial approximation and kernel density estimation techniques proposed. Adhering to their setup, for A ~ PA E P(O0, 1jd) with density PA E Lip..p..((1) (L), we denote the aforemennioned entropy estimate based on i..d. samples A" by H(A").
The SMI estimator from 6 employs a MI estimate between scalar variables (A, B). Assume their joint density is PA.B Lip and let (A", B²) ) be i.i.d. samples. To estimate ((A; B), consider
BL) = H(A") A(B")
(10)
Plug 10P into 6 and let (Lip)
n.m be the resulting SMI estimate. We next state the effective estimation d, (L, (L. := {pX,Y ELips (L) -e 1(0TX; X; M).
rate over F((ip)
Corollary 2 (Effective rate). Let du, dy E N, E (0, (0), and I, 2 0. The following E SI(X
uniform error bound over F!lind (L holds:
(Lii)) (Lip)
M - +C ((llggn) .t (log (logn)( ) (1-mma=)) +n #).
sup ((le)) Px,Y eiddy ay (L,M)
$ 2
for constant C that depends only du dy p. s. and L.
The proof of Corollary 2 (Supplement A.6 shows that densities in the generalized Lipschitz class have the property that all their projections are also in that class (with different parameters). We then bound (n) using 266 Theorem 4] to control the error of each differential entropy estimate in (10
Remark (SMI versus MI estimation rates). The SMI estimation rate from Corollary D consider- ably faster than the n (d2++ rate attainable when estimating classic MI (26). Our bound shows

---

that n and (dz, dy) are decoupled in the SMI convergence rate, unlike their interleaved dependence in MI estimation. The ambient dimension still enters the bound via the constant C, but its effect is expected to be much milder than in the classic case. As Theorem D shows, (dz, dy) can only enter via M and 8(n), both of which correspond to scalar MI terms (namely, the uniform per-sliced MI bound and scalar MI estimation error). This scalability to high dimensions is the expected gain from slicing.
Remark 7 (Optimal rate for smooth densities). Restricting attention to densities of maximal smooth- ness in Corollary i.e., 8 = 2, the resulting rate is 0.5M m 1/22 + On Cn-1f2(1 + (log n (0)55(111/1)) Equating the number of MC and data samples, m and n, the rate is parametric, up to polylog factors.
3.2 Extracting Sliced Information
We now discuss how SMI can increase via processing. In contrast to the DPI of classic MI, we show that SMI can be grow by extracting linear features of X and Y that are informative of each other.
To illustrate the idea we begin with a simple example. Let X = (X1 X2) ~ I2), Y = X1, and consider SI(X; Y) (Y is scalar and it is thus not sliced). For any 8 = (0) 02) E ES' with 01 + 0, we have I(OT X; Y 1(X1 + (02/01)X2; X1) 1 log (1 /01/02) )²) where the last step uses the independence of X1 and X2. Consider the function 9u ? 2R2 given by gu ga x2) (2) 0222) for some 0 < a < 1. Following the same procedure, we have 0T (X) = 0,Y Y + a82X² and
((Tga(X);;) = 1 log (1 (1+ (01/(afl))) > 1(0TX:Y). for almost all 0 E S and consequently, SI(X:Y) Y) < SI(ga (X): Generally, this shows that by varying a one can both create and diminish sliced information by processing via 9a-
When a O, we have SI(ga(X): oc, yielding (X)g Y) = sup, (( Thus, maximizing SMI by varying extracts the most informative feature X1 and deletes the uninformative feature X2. We next generalize this observation (see Supplement A.. for the proof).
Proposition Let (X,Y) P(R I and consider optimizing the SMI between linear processing of X and Y in the following scenarios:
1. Arbitrary linear processing: For matrices Ag, Ay and vectors by. by of appropriate dimension, we have
SI(A,X + by: Any + by) sup SI(A,XXA,,)) Supp((0X;ooTT (11) Ag.
sup
A
0.0
Furthermore, if (0", "0")=")== E arg max 1( X; TTY), then an optimal pair of matrices A and A
have 6* or o², respectively, in their first rows and zeros otherwise.
2. Rank constrained linear processing: For d2, d2,r N and 0, let Má de (r, = {A RRd, (o,(AA)= S ... San (ai(A)= ( c), where (A(A) the ith largest singular value of A.
We have
sup
SI(A, =
sup
SI(B,X,,,Y),
Vc Cy > 0.
As E. (Fziez), A,EMa (Fy)(1)
Mre.ra (2)) ByEMrrydy (Fy)
Furthermore, if (Bi, B5) are maximizers of the RHS, then B1 (resp., B2) has the first Ta (resp., ry) rows span the top Ya (resp., Ty) scalar MI slicing directions and the remaining rows are zero.
The proposition suggests that SMI can be used as an objective for extracting informative linear features. The setup in Case 2 precludes reduction to Case 1. Indced, if cigenvalues can shrink or grow without bound, it is always better to consider maximizing slice than to average several slices.
Remark 8 (Processing one variable). Similar results hold when only one of the arguments (X or Y) is processed. In this case, rather than the maximum being a projected MI term, it would be an SMI where the slicing is only in the opposite argument. For example, (1] becomes sup SI(A,X +b,; Y) supA, XI(Aax;YY) sup, 1(8T 4TY (D), for for Un -1) independent of (X,Y).
Proposition accounts for linear processing but the argument readily extends to nonlinear processing. For simplicity, the following corollary states the result for a shallow (single hidden layer) neural network (see Supplement A.8 for the proof).
7

---

Corollary 3 (Shallow neural network), Let (X.Y) ~ Px... For any scaling matrices Az, Ag. weight matrices W W and bias vectors ba by of appropriate dimension, define X := ( +b2), Yn == (WTY + bb),, where is a scalar, continuous, and monotonically increasing nonlinearity (e.g. sigmoid, tank) and the hidden dimension arbitrary. Then
sup
SI Ynn)
sup D. by
I( + +b)) To by)).
Ag W.W. B.
4 Empirical Results
Verying slices
Varying dataset size
10--
Varying slices and dataset size
4.1 Convergence of the SMI estimator
RMSE
We validate the empirical convergence rates for SMI estimation derived in Section 3 Consider densi-
ties with smoothness parameter & = the setup of Corollary 2 the expected convergence rate (up to log factors) is n 1-1 m m-11/2 While the theoretical results use the optimal estimator of 26 to obtain the tightest bounds, in our experiments we imple- mented the simpler Kozachenko-Leonnnkee estimator. The justification for doing comes from [25] who showed that this estimator achieves the same rate (up to log factors) the optimal one from (26).
10a
10' and/or slices
20
Varying slices
Varying dataset size
RMSE 16-1
Varying slices dataset
Figure D shows convergence of the estimated SMI RMSE for the case where X and Y are overlap- ping subsets of a standard normal random vector Z ~ (O, Iıs). For d 3, we set X Z11.3] = (Z, Ze TT)T, Y = Z22a (i.e., 2 coordinates overlap). For d == 10, we take X Z1.10. Y Z3:15 (5 coor- dinates overlap). Convergence is shown when both n and m grow together (i.c., n m), and when one fixed to large value and the other varies. The large value is chosen so that the error term corresponding to the fixed parameter is negligible compared to the varying term. For d 10, we also plot results for m, varying independently. Supplement C provides corresponding MI estimation results.
10-a
10a
10' n and/or # slices
10*
0.225
10
0,200
0.175
32
(# siices)
0.150
100
0.125
317
1000
4.2 Independence testing
10
32
100 n (# samples)
317
1000
Proposition D states that SI(X;Y) if and only if X and Y are independent. This implies that, like MI, we may use SMI for independence testing. MI-based independence tests of high-d nigh-dimensional continuous variables can be burdensome due to slow convergence 26 We show that, as our theory implies, SMI is scalable alternative.
Figure Convergence of the SMI estimator versus the number of data samples n and/or slice samples m: (a) 102: (b) d = 10, Mixxed nixed 1 (c) d = 10, m, n varied independently.
3, mixedd = fixed
of MI estimation
Figure 2 shows independence testing results for variety of relationships between X, Y pairs. The figure shows the area under the curve (AUC) of the receiver operating characteristic (ROC) for independence testing via SMI (or MI) thresholding as function of the number of samples n from the joint distribution | Both the SMI and MI were computed using the Kozachenko Leonenko estimator [5:: the MC step for SMI estimation (see (6) uses 1000 random slices, and the AUC ROC curves are computed from 100 random trials. The joint distribution of (X, Y) in each case of Figure is:
*For every n, we generate 50 datasets comprising a positive samples (i.e., drawn from the joint distribution) and 50 more dataset of negative samples in each setting. SMI and MI are then estimated over each dataset, the ROC curve found, and the area under computed. The ROC curve plots test performance (precision and recall) as the threshold is varied over all possible values. The AUC ROC quantifies the test's discriminative ability: omniscient classifier has AUC ROC while random tests have AUC ROC 0.5.
8

---

10
1.0
1.-
0.5
0.
0.5
0.8
0.8
0.7
SMI
0.7
0.7
Mi.d
0.7
3.6
GHL
Mi.
0.5
201
102
101
16"
10"
127 107 10a
10a
107
10"
(a) encodes sin- (b) Y encodes sin-
(c) encodes two
(d) Low rank com- (e) Independent co- mon signal
gle feature (lin)
gle X feature (sin)
features of X
ordinates
Figure 2: Statistical efficiency of independence testing with dimension. The plot show the area under the ROC curve (AUC ROC) as a function of samples n for several dimensions d. The test is based on thresholding SMI and MI. Details for each scenario
the text.
(a) One feature (linear): X, NN(0, (a) i..d. and Y /2 (1TX)1 Z), where :=
(1... 1)T Rd.
(b) One feature (sinusoid); X, N(O, (2) i..d. and Y ( sin (TX)1 + 2).
(1 |4/2 0 ... 0) x + Zi, is
(c) Two features: X ~ NN(0 i..dd and Y, 2 (O... [4/2]) ( + i>
(d) Low rank common signal: Z1, Z2 I,) and V (0, are independent; X PiV + Z1 and Y P2V - Z2. where P1.P2 E Rdx2 are projection matrices (realized at the beginning each iteration by drawing i.i.d. standard normal entries).
(e) Independent coordinates: X, Z (a) i.d. and Y
V2 (X +).
Note that in all the cases with underlying while MI does not; in the independent case of subfigure (d), both perform similarly and rather well. par or better than MI in these experiments, the results suggest that SMI
structure, SMI scales well with dimension
While the SMI is always performs best in structured (specifically, low rank) settings. This because these settings the MI term associated with random (9, 4) slices has lower variance. This is not the case for the unstructured setting of Figure d). There, when dimension is high, random slices carry relatively little information compared to the maximum MI over slices (which attained between the ith coordinates of X and Y for any i). Since SMI is an average quantity, this case offers little gain over classic MI.
4.3 Feature extraction
In the above, we focused on a nonparametric estimator for which we derived tight bounds. In practice, applying neural estimators (a MINE (29)) is more compatible with modern optimizers.
The SMI neural estimator (S-MINE) relies on the variational representation from Proposition B Given a sample set (X1, Yi),.., (X,, Y,) i.i.d. from Рx,,, further sample n i.i.d. copies of
(O, ②) ~ Unif(gd==1) -1) Uni The negative samples (X), Yi.... ..... (Xn,, Y, are obtained
by permuting the order of, c.g., the Y samples. Parametrizing the potential g in (3) by neural network 93 with parameters & E, we obtain the following empirical objective
Σ DE (O, 9. O, D.Y,) log
Σ 72
0,
sup ÉEE T
This provides an estimate (from below) of SI We leave full theoretical and empirical explo- ration of its performance for future work, and here only provide two proof-of moof-of-conceep experiments.
The formulation of SMI optimization of loss whose gradients readily evaluated lends itself to the S.Ml-extraction formulations of Proposition i.e., we can simultaneously optimize Az, Ag, and & end-to-end. Figure 3 shows results for X, 7 (0. T10). Y =e X Z, where e (1 0.... O)T, with ge in S-MINE realized as a two-layer fully connected neural network with 100 hidden units for. An (A A with rows converging to e are recovered. Note that while Proposition iddntifies an optimal solution where only the first row is nonzero, this will have the same SMI as when all rows
equal that first row. The latter solution is found since the gradients do favor one row over another.

---

We next combine S-MINE with independence testing, looking maximize the SMI using transfor- mations AX, AY, where X, are samples from random MNIST class (either 0 or 1) and A E 10xd Specifically, choose class C ~ (er(0.5) and then sample X and Y uniformly from that class' train- ing dataset. In this setup, X, Y share up to bit of information, i.e. Y( I H(C) = log 2. This suggests that maximizing the SMI between AX and AY will find an A that extracts information about C, revealing whether X and Y are in the same class. Optimiz- ing A yields an estimated SMI of 0.680 bits (compare this to, e.g., 0,0752 SMI achieved by matrix with i..d. standard normal en- tries). To confirm A is not being overfit, we also optimized the SMI over A when X, Y are drawn inde- pendently, i.e., no longer sharing class, yielding 0.0289 estimated SMI (the ground truth These results indicate that (a) an SMI-based independence test would be successful at detecting dependence between X, Y and (b) optimizing A not only succeeds significantly (almost 10x) increasing the SMI, but also comes relatively close to the true MI of bit. Heatmaps of two rows of rearranged into the MNIST image dimensions shown in Figure 4 Observe the A₁ visually correspond to the numeral 0, which, from a matched filter perspective, yields an AX (resp., AY) that informative of whether X (resp., Y) is in class not. This, in turn, conveys information of whether X, Y share class (since 0 and are the only options), as desired.
0.30
0-
0.30
020
0.29
0.15
0..0
Column
Columa
Figure 3: (A A.) optimizers for 1 over Gaussian dataset using S-MINE: (a) matrix A;; (b) matrix Ay. Both have cor- rectly recovered the unit vector e1 as their rows.
0.30
0.00
-6005
2002
Figure 4: Solution for optimizing A A-transformed SMI of the 0-1 MNIST setup using S-MINE. Rows and of A are shown in (a) and (b), respectively.
here).
Summary and Concluding Remarks
Motivated to address the computational and statistical unscalability of MI to high dimensions, this paper proposed an alternative notion of informativeness dubbed sliced mutual information (SMI). SMI projects high-dimensional random variables to scalars and averages over random projections. We showed that SMI shares many of the structural properties of classic MI, while enjoying efficient em- pirical estimation. We also showed that, as opposed to classic MI, SMI can be increase by processing the variables. This observation was quantified for linear and nonlinear SMI-based feature extractors. Experiments validating the theoretical study were provided, demonstrating dimension free empirical convergence rates, statistical efficiency for independence testing, and feature extraction examples.
Our results pose SMI as favorable figure of merit for information quantification between high- dimensional random variables. We expect and machine learning, although a large-scale empirical exploration is beyond the scope of this paper. In particular, SMI seems well adapted for representation learning via the (sliced) InfoMax principle 30, B1. and we plan to test this hypothesis in future work. On the theoretical side, appealing future directions are abundant, including convergence guarantees for the neural SMI estimator used in Section 4.3] operational channel/source coding settings for which SMI characterizes the information information-theoretie -heoreete fundamental limit, and statistical analysis of SMsebased independence testing.
turn useful for variety of applications in inference
Acknowledgements
Z. Goldfeld is supported by the NSF CRII Grant Grant CCF-2046018, and the 2020 IBM Academic Award.
the NSF CAREER Award under
10

---

References
[1] M. Cover and J. A. Thomas. Elements of Information Theory. Wiley, New-York, 2nd edition, 2006.
[2] A. El Gamal and Y.-H. Kim. Network Information Theory. Cambridge University Press, 2011.
[3] D. Haussler, M. Kearns, and R. E. Schapire. Bounds on the sample complexity of bayesian learning using information theory and the ve dimension. Machine learning, 14(1):83-113, Jan. 1994.
[4] R. Shwartz-Ziv and N. Tishby. Opening the black box of deep neural networks via information. MrXiv:1703.000 10, 2017.
[5] A. Achille and S. Soatto. Information dropout: Learning optimal representations through noisy computation. IEEE transactions pattern analysis and machine intelligence. 40((2))2897- 2905, Jan. 2018.
[6] M. Gabrié, A. Manoel, C. Luneau, J. Barbier, N. Macris, F. Krzakala, and L. Zdeborová. Entropy and mutual information in models of deep neural networks. arXiv preprint 2018.
[7] Z. Goldfeld, E. van den Berg, K. Greenewald, Melnyk, N. Nguyen, B. Kingsbury, and Y. Polyanskiy. Estimating information flow in neural networks. In ICML, volume 97, pages 2299-2300 Long Beach, CA, US, Jun. 2019.
[8] Z. Goldfeld and Y. Polyanskiy. The information bottleneck problem and its applications in machine learning. IEEE Journal on Selected Areas in Information Theory, 1(1):19-38, Apr. 2020.
[9] R. Battiti. Using mutual information for selecting features in supervised neural net learning. IEEE Transactions on Neural Networks, 5(4):537-550, Jul. 1994.
[10] Viola and W. M. Wells III. Alignment by maximization of mutual information. International Journal of Computer Vision, 24(2):137-154, Sep. 1997.
[11] A. A. Alemi, I. Fischer, J. V. Dillon, and K. Murphy. Deep variational information bottleneck. In Proceedings of the International Conference on Learning Representations (ICLR-2017), Toulon, France, Apr. 2017.
[12] I. Higgins, L. Matthey, A. Pal. C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Ler- chner. 3-VAB: learning basic visual concepts with constrained variational framework. In Proceedings of the International Conference on Learning Representations (ICLR-2019), New Orleans, Louisiana, USA, May 2017.
[13] A. van den Oord, Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv:/807.00
2018.
[14] L. Paninski. Estimating entropy on m bins given fewer than m samples. IEEE Trans. Inf. Theory, (0(9))2200-2200) Sep. 2004.
[15] H. Stögbauer Kraskov and P. Grassberger. Estimating mutual information. Physical Review E, 69(6):06613) 138, June 2004.
[16] M. Noshad, Y. Zeng, and A. O. Hero. Scalable mutual information estimation using dependence graphs. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP-201 pages
Brighton, UK, May 2019.
[17] M. I. Belghazi, A. Baratin, S. Rajeshwar, S. Ozair, Y. Bengio, D. Hjelm, and A. Courville. Mutual information neural estimation. In Proceedings of the International Conference on Machine Learning (ICML-2018), pages 530-539, Stockholm, Sweden, Jul. 2018.
[18] Z. Goldfeld, K. Greenewald, J. Niles-Weed, and Y. Polyanskiy. Convergence of smoothed empirical measures with applications to entropy estimation. IEEE TraE. Inf. Theory, 66(7):4368- 4391, Jul. 2020.
11

---

[19] L. Paninski. Estimation of entropy and mutual information. Neural Computation, 15:1 15:1191-1255, June 2003.
[20] J. Rabin, G. Peyré. J. Delon, and M. Bernot. Wasserstein barycenter and application to texture mixing. In Proceedings of the International Conference on Scale Space and Variational Methods Computer Vision (SSVM-2011) pages 435 446, Gedi, Israel, May 2011.
[21] T. Vayer, R. Flamary, R. Tavenard, L. Chapel, and N. Courty. Sliced gromov- wasserstesssstein. In Proceedings of the Annual Conference Advances in Neural Information Processing Systems (NeurlPS-201) 2019), Vancouver, Canada, Dec. 2019.
[22] T. Lin, Z. Zheng, E. Chen, M. Cuturi, and M. Jordan. On projection robust optimal transport: Sample complexity and model misspecification. In International Conference on Artificial Intelligence and Statisties (AISTATS- 2019),, pages 262-270, Online, 2021.
[23] K. Nadjahi, A. Durmus, L. Chizat, S. Kolouri, S. Shahrampour, and U. Simsekli. Statistical and topological properties of sliced probability divergences. Proceedings of the Annual Conference Advances in Neural Information Processing Systems (NeurIPS-202O), Online, Dec. 2020.
[24] H. Hotelling. Relations between two sets of variates. Biometrika, 28(3-4):321-377, 1936.
[25] J. Jiao, W. Gao, and Y. Han. The nearest neighbor information estimator is adaptively near minimax rate-optimal. In Proceedings of the Annual Conference on Advances in Neural Information Processing Systems (NeurIPS-2018) pages 3160-3171, Montréal, Canada, Dec. 2018.
[26] Y. Han, J. Jiao, T. Weissman, and Wu. Optimal rates of entropy estimation over Lipschitz balls. The Annals of Statisticy, (886))3228-325555 Dec. 2020.
[27] Y. Xu, S. Zhao, J. Song, R. Stewart, and S. Ermon. A theory of usable information under computational constraints. Proceedings of the International Conference on Learning Repre- sentations (ICLR-2019), New Orleans, Louisiana, US, May 2019.
[28] S. Kotz, T. Kozubowski, and K. Podgorski. The Laplace distribution and generalizations: revisit with applications to communications, economics, engineering, and finance. Springer Science Business Media, 2012.
[29] M. I. Belghazi, A. Baratin, S. Rajeswar, S. Ozair, Y. Bengio, A. Courville, and D. Hjelm. Mutual information neural estimation. In Proceedings of the International Conference on Machine Learning (ICML-2018), volume 80, pages 531-540, Jul. 2018
[30] R. Linsker. Self-organization in a perceptual network. Computer, 215-)110-1117, Mar. 1988.
[31] A. J. Bell and T. J. Sejnowski. An informationnmmxiiiizttioni blind deconvolution. Neural Computation, 7(6): 111-111-1111 1159, 1995.
maximization approach to blind separation and
[32] M. Godavarti and A. O. Hero. Convergence of differential entropies. IEEE Transactions Information Theory, 50(1):171-166, Jan. 2004.
[33] M. Donsker and S. Varadhan. Asymptotic evaluation of certain markov process expectations for large time. Entropy, 36(2):183-212, Mar. 1983.
[34] S. Dharmadhikari and K. Joag-Dev. Unimodolity, convexity, and applications. Elsevier, 1988.
[35] A. Marsiglietti and V. Kostina. A lower bound on the differential entropy of log-concave random vectors with applications. Entropy, 20(3):185, 2018.
12


Reviews:
[{"review_id": "ee-nwHjQI8d", "reviewer": "Reviewer_9G8T", "summary": "The authors introduce the Sliced Mutual Information (SMI) which is a measure of dependence between (sets) of random variables X and Y. The measure is defined as the average mutual information I(\\thetaX; \\phiY) among all 1-dimensional projections (slices) \\thetaX and \\phiY from the unit sphere. In practice SMI is computed with m slices sampled uniformly from the unit sphere combined with any empirical estimator, e.g., the Kozachenko-Leonenko. SMI overcomes the statistical inefficiency of mutual information in high-dimensional spaces while maintaining many of its properties, e.g., non-negativity and chain rule.  However, SMI violates the Data Processing Inequality (DPI). In addition, the authors prove the consistency of SMI and argue that it can be used as an objective to extract informative features by fitting transformation matrices. The results show that SMI is indeed consistent, can be used to identify statistical dependence, and can be used to extract informative features.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea to build a dependence measure as the average of all 1-dimensional projections combined with mutual information is novel (to the best of my knowledge). It is also true that mutual information can be problematic in the high-dimensional settings. SMI is therefore a good alternative of mutual information both in terms of computational and statistical efficiency. In addition SMI can be used as an objective to find data transformations that increase SMI (as opposed to the standard definition). This way feature extraction is obtained.\n\nWhile I think that SMI is overall a good contribution, I do have some comments:\n- it is unclear when the original formulation (i.e., averaging over all 1-dim projections) should be preferred over the extraction-based one. Or in other words, which one to use and when? To me the second formulation seems to be superior because the first one can arrive at a misleading result (as is the case in the MNIST experiment). Perhaps the authors intent to use the base formulation as a stepping stone to the extraction one, but then this should be stated more clear in the text. \n- isn't the maximum definition superior to the average? When should one used and when should the other? From Prop. 4 we have that the supremum of mutual information with 1-dim projections is equal to the supremum of SMI over all transformation matrices. This combined with the MNIST experiment (where the average definition leads to a small SMI score) make it look like the maximum should be preferred. \n- in general it is unclear in what scenarios the difference versions of SMI are best applicable\n- since the authors refer to MINE, why isn't it used in the comparison experiments to get a better understanding of SMI capabilities (e.g., 4.1)? If statistical inefficiency in high-dimensions is the problem, doesn't MINE solve that already? \n\nSome notes:\n- perhaps it will be easier for the audience that is not trained in projections if the authors explain somewhat better the matrix notation they use. For example, the design matrix X seems to be transposed by default (compared to the general scenario where X is NxD, with N the number of samples and D the dimensions). Also, are vectors by default column vectors? Also, in 4.2.a is 1^TX a vector added to matrix Z?\n- in the example in 3.2, Y should most likely be equal to X1 and not to X2 (or in general there seems to be something wrong in this paragraph)\n- the independence testing evaluation looks more like statistical power experiment since the simulated Y are a function of the input variables. An independence testing experiment would imply that SMI is used to detect independence when independence is true. Perhaps the authors can change the title to Detecting dependence (or something along these lines).\n- scalability is misleading: does it refer to computational scalability or to statistical? For example, Figure 2 shows statistical scalability. This can be rephrased to statistical efficiency for clarity\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not discuss the limitations of their method, nor in what scenarios it is preferable over other techniques, e.g., MINE. \n", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "15"}, "extra_text": {}}, {"review_id": "UextAeVzGRp", "reviewer": "Reviewer_imZu", "summary": "The paper presents statistical and empirical justification for computing an average mutual information of one-dimensional slices of real-valued vector random variables. The approach avoids the curse of dimensionality in mutual information estimation by averaging over the one-dimensional slices. The resulting statistical quantity has many of the same features as mutual information and can still be used for independence testing. Additionally, the measure is not invariant to linear (or non-linear) transforms of the random variables. This means that it does not obey the information processing inequality but behaves more like a canonical correlation coefficient, but is more general:  along the 'canonical'  directions the relationship need not be linear. A dual formulation of mutual information is used in the sliced case as on objective for maximizing neural networks.  Preliminary experimental results show the potential for non-linear metric learning.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality** The paper adds to the literature on high-dimensional mutual information estimation with the introduction of sliced approaches that have become popular in statistical divergence measures. The approach fills a gap between non-parametric  dependence-based  estimators and canonical correlation. \n\n**Quality** The paper's theoretical results are sound and the preliminary results illustrate the potential and utility. \n\n**Clarity** The paper is clearly written and engaging.  \nLine 237 and 238 could be better reworded. \"The setup in Claim 2 precludes reduction to Case 1. Indeed, if eigenvalues can shrink [...]\" the subject and objects could be better labelled and incorporated into the notation used in the claim.\n\n**Significance** High-dimensional dependence estimation and metric learning are useful for non-parametric independence testing and machine learning. The paper proposes an alternative quantity that has many useful properties while not capturing the full dependence between random variables enables estimation without the curse of dimensionality.  The approach is likely to be useful in practice.  \n\n### After author response.\nI thank the authors for their response. I think it clarifies the contribution as a quantity to compute dependence without trying to estimate mutual information. I think the other reviews have brought up interesting points, especially regarding the performance in high-dimensional cases with much higher levels of mutual information (beyond the 10 class case with MNIST). I would encourage the authors to consider showcasing the work with applications of the method to post-hoc analysis of neural network models trained for tasks such as natural language modeling or image classification. In light of comments, regarding max versus mean slicing, I would encourage the reviewers to consider commenting on future extension with the optimization of a distribution of slices for each random variable, https://openreview.net/forum?id=QYjO70ACDK ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I would have liked to have considered a contradictory example  (or proof that it does not happen under some regularity conditions) where where sliced mutual information approaches zero while mutual information is bounded away from zero.  ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "NHgCD62f8M1", "reviewer": "Reviewer_dcfo", "summary": "The paper proposes a measure of statistical dependence called SMI (sliced MI) defined as the MI between 1-dimensional projections using random unit vectors. The paper shows that SMI preserves many properties of MI like nonnegativity (zero iff independent) but not all (notably DPI, which the paper argues as a good thing). The upshot is that estimation of SMI can be done with any 1-dimensional MI estimator and an MC integrator, and that this can have better sample efficiency than direct MI estimation when the observations are high-dimensional but have a low-dimensional structure.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "STRENGTHS\n\n- SMI is a simple and intuitive quantity. It seems like a natural addition to existing slicing techniques of measuring the Wasserstein distance. \n\n- The paper is quite thorough in establishing properties of SMI, including not only nonnegativity and a KL formulation but also the chain rule and a DV-like variational form. \n\n- The paper also provides convergence guarantees that relate the error of the SMI estimator to the error of a 1-dimensional MI estimator. The utility of SMI in extracting features based on the violation of DPI is also discussed. \n\n\nWEAKNESSES \n\n- SMI is a lower bound on MI and cannot be larger than the log sample size [1], so it doesn't seem to offer much advantage when the underlying MI is large which is really the setting that we care about. SMI seems mostly useful in a high-dimensional setting with a low-dimensional structure (with small MI). There are already many available low-variance high-bias estimators of MI when MI is small [2], so I'm a bit unsure how significant or impactful SMI will be in practice other than being a theoretically interesting quantity.\n\n- I feel experiments should consider more than 1 MI estimator. There are many neural MI estimators with different behaviors [2], [1] which might change the story in Figure 2. As for feature extraction, experiments should ideally compare with the existing literature like MINE, DIM, BYOL, etc. in downstream performance, but I understand that it's probably out of scope so this is more minor.\n\n\n[1] Formal Limitations on the Measurement of Mutual Information \n\n[2] On Variational Bounds of Mutual Information\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Contextualizing SMI in well-known limitations of estimating mutual information would be useful. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "LRwyiHy_H6f", "reviewer": "Reviewer_C7hP", "summary": "This paper proposes sliced mutual information (SMI), which is a new estimator of the mutual information (MI) particularly effective for high dimensional settings. The definition of SMI is simple and its computation is efficient, and it has desirable theoretical properties. Empirical evaluation shows the effectiveness of the SMI compared to the estimated MI.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Originality\n\nThe originality of this paper is high as it proposes a simple yet effective new MI estimator.\nIt would be better if the difference of the proposal and existing MI estimators are discussed more carefully.\n\n### Quality\n\nThis paper is well written and the overall quality is high.\n- In particular, this paper performs thorough theoretical analysis of the SMI and its estimator, which is impressive and a good contribution.\n- The pseudo-code of the estimator of the SMI and its complexity analysis would be desirable for better readability (at least in Appendix), although it is relatively clear from its definition.\n- In contrast, empirical evaluation is not thorough. I have the following comments:\n    - Although several MI estimators are mentioned in Introduction ([14-18]), there is no comparison with such methods in experiments. The effectiveness of the proposed method is more convincing if such empirical comparison is performed.\n    - In Figure 1, is the RMSE an error between the SMI and the MI? There is no clear definition in the paper.\n    - In addition, in Figure 1, since generated data here is quite simple, I am interested in how the convergence rate is for other types of datasets?\n    - In Section 4.2, only relatively simple relationships ((a)-(d)) are examined. Since the MI can measure any statistical dependence, comparison on various types of relationships is desirable (e.g. relationships used in Reshef et al., Detecting Novel Associations in Large Data Sets, Science, 2011).\n\n\n### Clarity\nThis paper is clearly written and easy to follow.\n\n### Significance\nThe proposed method is expected to be widely applicable in machine learning and data science as the proposal is simple. Hence the significance of the contribution is high.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not provide the limitations and potential negative societal impact. I would suggest discussing the limitation of the proposal; that is, when it fails to estimate the MI, and discuss the negative impact of the proposal if such a situation occurs.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new scalable measure for statistical independence that does not attempt to approximate standard mutual information. The proposed measure has the potential to become a very useful tool for characterising statistical (in)dependence and may thus have have broad impact.

All reviewers unanimously recommend acceptance. I agree; it is nice work.  Please incorporate the reviewers' feedback for the camera-ready version.

Decision: Accept (Spotlight)