$k$-Sliced Mutual Information: A Quantitative Study of Scalability with Dimension
Authors: Ziv Goldfeld
Kristjan Greenewald
Theshani Nuradha
Galen Reeves
Venue: NeurIPS 2022 Accept (2022)

Abstract: Sliced mutual information (SMI) is defined as an average of mutual information (MI) terms between one-dimensional random projections of the random variables. It serves as a surrogate measure of dependence to classic MI that preserves many of its properties but is more scalable to high dimensions. However, a quantitative characterization of how SMI itself and estimation rates thereof depend on the ambient dimension, which is crucial to the understanding of scalability, remain obscure. 
This work provides a multifaceted account of the dependence of SMI on dimension, under a broader framework termed $k$-SMI, which considers projections to $k$-dimensional subspaces. Using a new result on the continuity of differential entropy in the 2-Wasserstein metric, we derive sharp bounds on the error of Monte Carlo (MC)-based estimates of $k$-SMI, with explicit dependence on $k$ and the ambient dimension, revealing their interplay with the number of samples. We then combine the MC integrator with the neural estimation framework to provide an end-to-end $k$-SMI estimator, for which optimal convergence rates are established. We also explore asymptotics of the population $k$-SMI as dimension grows, providing Gaussian approximation results with a residual that decays under appropriate moment bounds. All our results trivially apply to SMI by setting $k=1$. Our theory is validated with numerical experiments and is applied to sliced InfoGAN, which altogether provide a comprehensive quantitative account of the scalability question of $k$-SMI, including SMI as a special case when $k=1$.

k-Sliced Mutual Information:
A Quantitative Study of Scalability with Dimension
Ziv Goldfeld
Kristjan Greenewald
Theshani Nuradha Cornell University
Cornell University
MIT-IBM Watson AI Lab
goldfeld@cornell.coo
IBM Research
pt.380ccorne11.cm edu
kristjan. gwwenewwlldeeieeee ww..ddiiimm.mmmmmmmmmmme com
Galen Reeves
Duke University galen. rel.e.reeeves@duke.eed
Abstract
Sliced mutual information (SMI) is defined as an average of mutual information (MI) terms between -iedeeenniooall random projections of the random variables. It serves as a surrogate measure of dependence to classic MI that preserves many of its properties but is more scalable to high dimensions. However, a quantitative characterization of how SMI itself and estimation rates thereof depend on the ambient dimension, which is crucial to the understanding of scalability, remain obscure. This work provides multifaceted account of the dependence of SMI on dimension, under broader framework termed k-SMI, which considers projections to k-dimensional subspaces. Using new result on the continuity of differential entropy in the 2- Wasserstein metric, we derive sharp bounds on the error Monte Carlo (MC)- estimates of k-SMI, with explicit dependence ambient dimension, revealing their interplay with the number of samples. We then combine the MC integrator with the neural estimation framework provide end- to-end k-SMI estimator, for which optimal convergence rates are established. We also explore asymptotics of the population A-SMI dimension grows, providing Gaussian approximation results with a residual that decays under appropriate moment bounds. All our results trivially apply SMI by setting k 1. Our theory is validated with numerical experiments and is applied to sliced InfoGAN, which altogether provide comprehensive quantitative account of the scalability question of K-SMI, including SMI as special case when k = 1.
k and the
1 Introduction
Mutual information (MI) a fundamental measure of dependence between random variables [| 2 with myriad of applications in information theory, statistics, and more recently machine learning (144). Its appeal stems from the favorable structural properties it possesses, such as meaningful units (bits nats). identification of independence, entropy decompositions, and convenient variational forms. However, modern learning applications require estimating MI between high-dimensional variables based on data, which is known to be notoriously hard with exponential dimension sample complexity [S [6]. To alleviate this impasse, sliced MI (SMI) was recently introduced by a subset of the authors as surrogate dependence measure that preserves much of the classic structure while being more scalable for computation and estimations in high dimensions
[7
Inspired by slicing techniques for statistical divergences 18 21 SMI is defined average of MI terms between one-dimensional projections of the high-dimensional variables. Beyond showing that
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

SMI inherits many properties of its classic counterpart, 17 demonstrated that it can be estimated with (optimal) parametric error rates in all dimensions by combining MI estimator between scalar variables with a MC integrator. However, the bounds from 17 rely on high-level assumptions that may be hard to verify in practice and hide dimension-deependent constants whose characterization crucial for understanding scalability in dimension. Furthermore, when projecting high-dimensional variables it is natural to ask what information can be extracted from more than just the real line, say, a subspace of dimension & > 1, but this extension was not considered in 177. This work defines K-SMI (which employs projections to k-dimensional subspaces), and provides a comprehensive quantitative study of its dependence on dimension, encompassing the MC error, formal guarantees for neural estimators, and asymptotics of the population k-SMI as dimension increases. All our results trivially apply for the original SMI case (when k 1), thereby closing the aforementioned gaps in analysis from
1.1 Contributions
The objective of this work is provide a thorough quantitative study of the dependence of SMI on dimension. We do so under the slightly broader framework of k-SMI, which we define between random variables X and Y with values in d and R as
SI (la(X; = S Su(k,d))
1(T BT BTYYdO (A)dok,d (B),
(1)
where St.k: d) is the Stiefel manifold of d matrices with orthonormal columns and Fg,d is uni- form measure. k-SMI coincides with SMI when k = 1, but to further support it as a natural extension, we show that structural properties of SMI derived in 17 still hold for any 1 S le S min dy)- We then move to study formal guarantees for k-SMI estimation, targeting explicit dependence (k, das dy). A key technical tool we employ is new continuity result of differential entropy with respect to (w.r.t) the 2- 2-Wasserstein distance W2. which we derive using the HWI inequality from Our continuity claim strengthens the one from 24 in two ways: (i) it replaces the c2)- regularity condition therein with the weaker requirement of finite Fisher information, and (ii) it sharpens the constant multiplying W2 to be optimal. As a corollary, we show that the differential entropy of projected variable, say h(ATX), is Lipschitz continuous w.r.t. the Frobenius norm the St(k, d)-
122 13
Lipschitzness is pivotal for obtaining dimension-dependdee bounds on MC-based estimates of k-SMI. We bound the MC error in terms of the variance (ATX;; ; B² Y) when (A, B) are uniform over their respective Stiefel manifolds. Lipschitz continuity of differential entropy implies Lipschitzness of this projected MI, which enables controlling its variance via a concentration argument over St(k, d). The resulting bound scales as (VVk(I/d + 1/dy)//m) where m is the number of MC samples and the constant is explicitly expressed via basic characteristics of the (X, Y) distribution (its covariance and Fisher information matrices). This result, which also applies to standard SMI, sharpens the bounds from 17. characterizes the dependence on dimension, and holds under primitive assumptions on the joint distribution. Furthermore, the bound reveals that higher dimension can shrink the error in some cases a surprising observation which is also verified numerically on synthetic examples.
In addition to MC integration, the K-SMI estimator employs a generic MI estimator between k- dimensional variables. We instantiate this estimator via the neural estimation framework based on the Donsker- Varadhan (DV) variational form 25 (see also (26 28). The neural estimator is realized by an ê-neuron shallow ReLUU network and the effective convergence rate the resulting k-SMI estimate is explored. We lift the convergence rates derived in 22 for neural estimators of f-divergences to the k-SMI problem. The resulting rate scales as O (22 the number of neurons, m is the number of MC samples, and n is the number of (x, Y) samples. Equating f, m, and n results in the (optimal) parametric rate. Our result also shows that neural estimation of k-SMI requires milder smoothness assumptions on the population distributions. Namely, we relax the smoothness level [(dg + /2] + imposed in ( to k + 3, i.c., adapting to the projection dimension rather than the ambient one. This is significant relaxation since we often have du, dy > kt
+ m m-1/2 + kn 1/2)), where é is
To further understand the effect of the ambient dimension, we explore how Sl Y) behaves as de. dy + co. To that end, we first provide full characterization of SI (X, Y) between jointly Gaus- sian variables, revealing that it scales as k2 (dzddy times the squared Frobenius norm of the cross-
2

---

covariance matrix. We then show that general k-SMI can be decomposed into a Gaussian part plus a residual term that quantifies the average distance (over projections) from Gaussianity. The latter is intimately related to the conditional central limit theorem (CI.T) phenomenon [30 32 and we use those ideas to identify approximate isotropy conditions under which the residual vanishes as dæ dly - co. Lastly, we conduct an empirical study that validates our theory and explores applica- tions to independence testing and sliced infoGAN. Specifically, we revisit the infoGAN generative model [6] and replace the classic MI used therein with SMI. Training the model, we find that it successfully learns disentangled representations despite the low-dimensional projections, suggesting that SMI can replace classic MI even in applications with complex underlying structure.
2 Background and Preliminaries
2.1 Notation and Definitions
Notation. For d 2 1, ||]]| is the Euclidean norm in R., (:.-) 1) is the inner product, while |1-11a is the e! norm. We use || llop and ||-[|r for the operator and Frobenius norms of matrices, respectively. Matrix inequalities are understood in the sense of (partial) semi-definite ordering, i.e., we write A č B when A - B is positive semi-definite. The Stiefel manifold of d x k matrices with orthonormal columns is denoted by Su,k, d). For a d x k matrix A, we use PA Rd >> TR= for the orthogonal projection onto the row space of A.
Let P(IR)) denote the space of Borel probability measures on IRd, and set P2(Rd) == {u E P(Rd): [|]=]]] < oo] as the subset of distributions with finite 2nd absolute moment. For p., v E ((Rd). we use y @ v to denote product measure, while spt(μ) designates the support of ju. We use Leb for the Lebesgue measure on R4, and denote the subset of probability measures that are absolutely continuous w.r.t. Leb by (RR)) For measurable map f. the pushforward of μ under f is denoted by fap fe f-1, i.e., if X ~ μ then f(X) ~ fzJ.. For a, b E I, we use the notation A min(a, b) and a V b maxfa, b). We write a Sx when a S C-b for constant C₂ that depends only on x (a E b means the constant is absolute).
(an, aa)) E Z the partial derivative operator of order ||a|]] is denoted g "=ass" For an open set U C Rd and integer s 2 O, the class of functions whose partial derivatives to order all exist and are continuous on 2/ is denoted by (44), and we define the subelass Cg(4)) == {f E (*M maxa |al .s D | S b). The restriction of f Rd + R to X S Id is denoted by flx. For compact X, slightly abusing notation, we set || == supaex ||=||-
For multi-index
by D
8a1
Divergences and information measures. Let J., P(R satisfy μ & v, i.e., f absolutely continuous w.r.t. v. The relative entropy and the relative Fisher information are defined, respectively, as D(ullp)= == Sae log(du/ du)dj and ((i]lp) == S /oolldup// ||" dj. The 2-Wasserstein dis-
tance between μt, v (2((R W2(u, v) == infxen(a.)) (Srux || yl dm dz(x, v)) 1/2 where I(p,)) v) is the set of couplings of μu and v. All three measures are divergences, i.e., non-negative and nullify if and only if (iff) μ = v. In fact, W₂ is a metric on P2(RR), which metrizes weak convergence plus convergence of 2nd moments.
MI and differential entropy are defined from the relative entropy as follows. Consider pair of random variables (X, Y) ~ pxy E P(R < x Rd4)) and denote the corresponding marginal distributions by Jax and jy- The MI between X and Y is given by I(X; Y) == D(uxy ||xx & JY) and serves as measure of dependence between those random variables. The differential entropy of X is defined as h(X)= = h(px) : DD Lcb). MI between (jointly) continuous variables and differential entropy are related via I(X;)) Y) h(X) h(Y) h(X, Y); decompositions in terms of conditional
entropies are also available M. The Fisher information of X ~ u is J(z) := ](u][Leb). Denoting
the density of by fa. the Fisher information matrix of µ is JF(N) := E [(V log (W)(V log J,
and we have (r())()) ((n).
3

---

2.2 Lipschitz Continuity of Projected Differential Entropy
key technical tool we use a new continuity result of differential entropy w.r.t. the 2- Wasserstein distance. It strengthens an earlier version of this result from (24], and may be of independent interest. Lemma (Wasserstein continuity). Let Ja. P2((R)) satisfy y Leb and J(v) < co. Then h(p) h(v) S (Jv))W2(,,))
and the constant above is optimal in the sense that sup u=M: h(n)-h(r) 1.
(KrjWalaa)
The proof the lemma, given in Appendix A.I follows by invoking the HWI inequality for the difference of relative entropies [22 23
with isotropic Gaussian reference measure 7 (0 o Ta), re-expressing the relative entropy difference in terms of differential entropies, and taking the limit - 0.
Remark (Comparison to (241). Continuity of differential entropy w.r.t. the W2 was previously derived in 24 Proposition 11, but via a different argument, under stronger conditions. and without an optimal constans. The inequality from (24] assumed (C1, C2 -e)-rerrryy of the density of (i.e., that ||V log (x)]l S ca||=||| + C2. for all Rdd,, which stronger than J(v) < co when v E PP2(IT.)
rather direct implication of Lemma D is the following Lipschitz continuity of projected entropy (also proven in Appendix [A.I). which plays key role in the subsequent analysis of k-SMI estimation. Proposition (Lipschitzness projected entropy). Let μa P2 (IR have covariance matrix Σn and J(p) < co. For any A, B E (t. d), we have () ()) S Jr())ll En Topl A B [|r-
Sllice Mutual Information
SMI was defined in 17 as an average of MI terms between one-dimensional projections of the considered random variables. As higher dimensional projections preserve more information about the original (X, Y), we extend this definition to k k-dimensional projections.
Definition 1 (k-sliced mutual information). For < k de A dy. the k-SMI between (X,Y)
μXY E P(R TRdv) is defined in ( where ok,d is the uniform distribution St.dd k))
k-SMI can be equivalently expressed in term of conditional (classic) MI (I; (ATX; BTYJA, B). where (A, B) ~ Tk.d, @ ok,dy, i.c. (A, B) are independent and uniform over the respective Stiefel manifolds. k-SMI reduces to the SMI from [17] when 1. Below we show that SI& preserves the structural properties of SMI, as derived in 11 Section 3].
Remark 2 (Related definitions). k-SMI entropy decompositions and chain rule require defining k-sliced entropy and conditional k-SMI. For YXY,2) ~ pxYz E P(Rdd x Bad x Rda) and (A,B, C) Ok,ds @ Ok,d, @ kkdd., , the k-sliced entropy of is shş (X) == h(ATX(A), while the conditional version given Y is given by sh ( == h(AT BTY). The condition k-SMI between X and given is (Xx Y == I(AT BYA,
3.1 Structural Properties
We verify that k-SMI preserves structural properties previously established in I for SMI. Proposition 2 (k-SMI properties). For any 1 & k < de A dy the following properties hold:
1. Identification of independence: (X; > 0 with equality iff X and Y are independent.
2. Bounds: For integers kı <ky Shi, (X; Y) < Skka (X:Y) sup
BTA < ((X;)).
AeSt(kg,d,) (eSt(K2.d,)
3. Relative entropy and variational Let (8, 8) px By and (A,B) ok.ds Ook,dy then 1 pB Juu @ py A)
Slx(X;Y) DKL ((pA, ( )//xxy (p
EI log (B (B(JAABAXRB)))))
sup (.S((.d.)xS(((.ddd)x)..)
where the supremum is over all measurable functions for which both expectations finite.

---

4. Entropy decomposition: SI (X;)) = - - -
-h,, (Yl(() = shx
sY) (h&(X, (Y,Y), Y) provided that all the relevant (joint mmariinal /conditional) densities exist.
5. Chain rule: For any Xt....,X,, X,, Y, Z, we have
X,;
(Ix(X
Ex (X.......,. In particular, (I Z) - (1x 51;
6. Tensorization: For mutually independent ((X,))) S1 (1x;)
Σ (X,
The proposition is proven in Appendix A.2 via a direct extension of the k 1 argument from 1v
4 Estimation and Asymptotics of k-SMI in High Dimensions
As shown in [7.. SMI can be estimated from high-dimensional data by combining a MI estimator between scalar random variables and MC integration step. However, the bounds from [ do not explicitly capture dependence on the ambient dimension, which is crucial for understanding scalability of the approach. We now extend the estimator from [17] to k-SMI and provide formal guarantees with explicit dependence on k, dzy and dy, thus closing the said gap.
To estimate k-SMI, let ((x;, Y.) be i.i.d. from PxY P(TR R2y) and proceed follows:
1. Draw {(A,, By i..dd from kkdd, Sok,d, (i.e., each pair is uniform on St,ki de) x Su(, d
2. Compute (A) X,, B)))) J which, for fixed (A,: By), are samples from (p .p )//XY-
3. For each j = L,..,, a MI estimator between k-dimensional random vectors is applied to the n samples corresponding to (A,. By) to obtain an estimate i ((AJx)), (BJY)") of I[AAX; where (A]x)" = (A)X1 ................................ (fXx,) and (BJY)) is defined similarly.
4. Take MC average of the above estimates, resulting in the k-SMI estimator:
SI m.
1 Σ ((AJx)",(B)))) 7
(2)
We provide formal guarantees for the quality of the SI m, estimator given a generic k-dimensional MI estimator i(,,) -) in Step 3. Afterwards, we instantiate the latter as a neural MI estimator and provide explicit convergence rates. To get further insight into the dependence on dimension, we study asymp- totics of Gaussian k-SMI as de, dy + co and corresponding Gaussian approximation arguments.
4.1 Error Bounds with Explicit Dimension Dependence
Our analysis decomposes the overall error of SI T..
into the MC error plus the eиror of the k- dimensional MI estimator i(:, -). We first consider an arbitrary estimator I(.,)) whose error is (implicitly) upper bounded by (n) and focus on analyzing the MC error, targeting explicit de- pendence on k. dz, and dy- As in [17], the statement relies on the following assumption on the k-dimensional estimator i(::-).
Assumption 1. (X,Y) PXY P(Rdd Beb)) is such that I(ATX; i((AT))", (BTY)") with error most (n),), uniformly over (A, B) E St(k, dz) x St(kk, dy). Theorem 1 (M- estimation error). Let pxy P2(R xR²* satisfy Assumption ! have marginal covariance matrices Σx and Ey, and J(uxy) < co. Then the estimator from has error bounded by
be estimated by
E (I Si S Cluxr)
k(de + dy) d,d,
m + (n)),
(3)
'A simple approach for sampling the uniform distribution on St(k, d) is to draw kd random samples from (O, 1). arrange them into an x matrix A. and compute A(ATA) A)1/2 1/22 (cf. [ Theorem 2.2.1D. A slightly more efficient approach is to first apply a QR decomposition to and then follow the aforementioned sampling method only to the Q matrix. Note that for k = O(1), both computation times are linear in d (QR decomposition via the Schwarz- Rutishauser algorithm is O(dk²) 34
5

---

where C(uxY)= = 2 (1xx lap (1Ixxllop [ap
V ||Ey (lop)
The proof of Theorem 1 (in Appendix A.3 bounds the MC error by (Var(ixy (A, B)) /m) where ixy (A, B)= ((TT and (A, B3 Tk.d, @ Fk.d.- We then use the continuity result from Proposition 1 along with the entropy decomposition of k-SMI (Proposition 2 Claim 4) to show that exY is Lipschitz continuous (w.r.t the Frobenius norm) on St dg) x St(k. dy). Concentration of Lipschitz functions on the Stiefel manifold and the Efron-Stein inequality then imply the above bound. This result clarifies the dependence of the MC enror on dg. and dy. and reveals scaling rates of the parameters with m. for which (tigh-dimensional) convergence holds true.
Remark 3 (Comparison to [7 Theorem from ATZI treats the k = 1 case under stronger high-level assumptions and without identifying the dependence on dimension. Namely, assuming the uniform bound lixyllz> S M, they control the variance by M /4 to obtain the m 1/2 rate, although M generally depends on (dz, dy). Herein, we rely on the finer observation that ixy is Lipschitz and use concentration results to get fimension-depencee
bound in terms of basic characteristics of (X, Y)- Remark 4 (Blessing of dimensionalityy. The constant in the MC error may decay as dimension grows. For instance. if X and Y are both d-dimensional with identity covariance matrices, then [Ex llop, | llop are Oa(1). For such (X, ). the MC bound decays to O as d + oo, assuming that (Jp( )llop grows at most sublinearly with d. Also note that C(uxy) has the same invariances as the k-SMI: it is invariant to translations and scalings of the form (X,Y, YY + (sX,sY) for & O.
4.2 Neural Estimation
We now instantiate the k-dimensional MI estimator via the neural estimation framework of 229 and obtain an explicit bound on 6 (n) in terms of m, n, k, and the size of the neural network.
K5
Neural estimation of MI relies on the DV variational form
B[S(U, V)] - log (BirCE..1))
(U:V)=
sup f:8 xR
where (U, μov. (U, V) pe @ uv. and f is measurable function for which the expectations above are finite. Define the class of e-neuron ReLU network as
g(2) = Σ i=1 3.0 ((w,, 2) + b,) + (wi, 2) + bu
(0, (0) 9 R td + R:
max 19
[wella 1 11 max |P S 2l |bul. l|wo S a isise
where x(z) = z V is the ReLU activation; set the shorthand G2 = G2 (log log l' V 1). Given ii.d. data (U1, Vi...... (Un, Vx)) from μov, the neural estimator parameterizes the DV potential f by the class G2_ and approximates expectations by sample means resulting in the estimate
a de (U" V")+
1 Σ 9(U,, V.) - log 1=1
(Us.Voot))
= sup
9 an
For k-SMI neural estimation, we set
1 m
S == Σ m
ie., we use K as the k-dimensional MI estimator in ② This estimator is readily implemented
by parallelizing m E-neuron ReLU nets with inputs in I and scalar outputs. We provide explicit
convergence rates for it over an appropriate distribution class, drawing upon the results of [29] for neural estimation of f-divergences (see also (35)). ). For compact X' C dg and y C R let Pac(X' x y) := {uXY E (RR. x IR4v): (px( C X' x J/ and denote the density of pxy by fxv. The distribution class of interest is
3r (L)) for some open set / 3 X x y s.t. log fxy = r|xxy, I(xY)) < M
Fb: F d, (M, = |XX Pac V):
NNegative samples. ie.. from px @ py. can be obtained from the positive one via (U1, Voc))...., (Un, (v)), where E Sn is permutation such that a(i) * for all = I,...,n.

---

which, in particular, contains distributions whose densities are bounded from above and below on X' x Y with smooth extension an open set covering A' x y. This includes uniform distributions.
truncated Gaussians, truncated Cauchy distributions, etc.
We next provide convergence rates for the k-SMI estimator from
2b. uniformly over Fi..a, (M,
Theorem (Neural estimation error). For any M, b O, we have
IE S1a
S.A.kddd k- (m- + kn $).
sup
PX,YY EF da.dy (M.b)
The dependence on dz, dy above is only through the MC bound (explicit) and ||2 x y (implicit).
Theorem is proven in Appendix A.4 by combining the MC bound from Theorem 1 with the neural estimation error bound from 29 Proposition 2J. To apply that bound for each [ATX;;BT)
where
(A, B) (t, dz) St.kk, d we show that the existence of an extension r of log fxy with k + 3 continuous and uniformly bounded derivatives implies that the density of (ATX, BTY) also has such an extension.
Remark 5 (Parametric rate and optimality). Taking 8= m. X the resulting rate in Theorem 2 parametric, and hence minimax optimal. This result implicitly assumes that M is known when picking the neural net parameters. This assumption be relaxed to mere existence of (an unknown) M, resulting in an extra polylog(E) factor multiplying the n -1/2 term.
Remark 6 (Comparison to 29 ). Neural estimation of classic MI under the framework of (29) requires the density to have Hölder smoothness & 2 [(dg + /2] + 3. For (X; Y), smoothness of k 4-3 3 is sufficient (even though the ambient dimension the same), which mean it can be estimated over a larger class of distributions. This is another virtue of slicing in addition to fast convergence rates. For SMI (i.e., 1) as in 17), a constant smoothness level suffices irrespective of (da, dy).
4.3 Characterization of and Approximation by Gaussian k-SMI
To gain further insight into the dependence of k-SMI on dimension, we fully characterize it in the Gaussian case. Afterwards, we show that general k-SMI decomposes into a Gaussian part plus residual, and discuss conditions for the latter to decay as d -> oc. As before, Σx is the covariance matrix of X (similarly, for Y), while CXY
E [[X E{ E[YT)T] is the cross-covariance.
Theorem 3 (Gaussian k- k-SMI). Let (X,Y) 7XY NO Exy) be jointly Gaussian random variables. Suppose that [EE llopll I| |X llop- | |Σx 1 Eypllup I k and |Σ Exy S p.for
some 2 1 and p < 1. Then, for any fixed k. we have
2.2 I2
(I;Y =
2xr (x)()))) (1 + +(1).
as dz, dy + 0o, where o(1) denotes a quantity that converges to zero in the limit.
Theorem is proven in Appendix A.5 It states that if Σx and Σy have bounded condition numbers and the correlation, as quantified by // Ey llop- is less than then the Gaussian k-SMI is asymptotically equivalent to the squared Frobenius norm CXY. normalized by the traces of the marginal covariances. Since ||Cxy 17 S (dz A dy) [Xx llop l llop and tr ((rx))))__)))))))) > d,dd |I "Ey'llop we see that the SI (X;Y) typically decreases with dimension as d A d,,' This rate is inline with the shrinkage with dimension of the MC bound from B which renders that bound meaningful even when k-SMI is itself decaying, e.g., under the framework Theorem D
k-SMI decomposition and Gaussian approximation. Given the above result and the recent interest in Gaussian approximations of sliced Wasserstein distances 36 37,, we present decomposition of S-SMI into a Gaussian part plus a residual. For (X.Y) ~ Pxy E P(R x RR),, ), let (X*, ,Y)) ~ TXY == (0, Σxy) be jointly Gaussian with the same covariance as (X, Y). The K-SMI satisfies
Y); Slx(X";; B[ōxy (A, B)],
(4)
where, for each (A, E St(k: dz) St(k;,
6xy (A, B) := D(( (((pp.P)))xxx ) " )xxx) D( ",) " p))xxx
7

---

This decomposition is proven in Appendix A.7 Theorem fully accounts for the first summand, which begs the questions of whether the leading term the decomposition, and under what conditions? This question is intimately related to the conditional CLT of low-dimensional projections under relative entropy 32 This is challenging and active research topie (0.52, for which sharp convergence rates remain unknown. As first step towards complete answer, in Appendix B we bound this residual term and identify mild isotropy conditions on the marginal distributions of and Y that are sufficient for the residual term to vanish as de, dy + OC.
5 Experiments
MC error and Gaussian k-SMI rates. Under the Gaussian setting described next, we illustrate the dependence on k, dz: dy of (i) the population k-SMI expression Theorem 3 and (ii) the associated MC
Population & SMI
MC Standard Deviation
1-SMI
1-
2-5MI
2-5MI
Let 71 (0, Ia) and V ~ (0, be independent, and set = P1V + 11 and Y P2V + Zz, where P1, E [@dx2 are projection matrices (with iid. normal entries). We draw m 10a pairs of pro- jection matrices {(A,. B- and use the classic K-NN MI estimator of 3N with n = 16 x 10x samples (X, Y) to approx- imate the MI along each projection pair, i.e. for each 1,..., 10a, we compute ((AAX)", (BJ))
estimation error from Theorem
10a
10a
10a
10'
Figure 1: Decay with dimension the population k-SMI
(left) and the associated MC standard deviation (right).
Note that the mean of (((A)X)) (BJY)") is the population k-SMI (which, in this Gaussian example, is given by Theorem 3) while its standard deviation is the constant in front
of the m mn-1/2 1/2 term G3 Theorem Figure 1 plots the said mean and standard deviation of the projected MI terms (A] A]X)" (BJY ).). The rates of decay in both cases follow those predicted
by Theorems 3 and respectively. This implies that m. need not be rapidly scaled up, even as the
population k- SMT shrinks with increasing dimension.
Independence testing. It was shown in 17 that SMI can be used for independence testing between high-dimensional variables, when classic MI costly estimate. We revisit this experiment with k-SMI to demonstrate similar scalability and understand the effect of k. The test estimates &-SMI based on n samples from μxy and then thresholds the value declare dependennce/independenee
5.0
60
3.
3.8
3.5% -
3.5%
1.59
3.091 8
37
2.94
5.50
130
239
5000 1
3.0
(a) Y encodes single feature (sinusoidal)
(b) Rank common signal
22
3.
1:090 4 10
7-246
1-099 4 16
3:099 4
3-200
2-2%
2-241 4
3.
Mi. 1:
3.
M.
Mi
(c) Rank common signal
(d) Rank common signal
Figure 2: Independence testing with K-MMI: AUC ROC versus sample size n for different & and d.

---

k: = 2
10
10
10"
10
10
d=5
d=
=
= 10
10"2
d 10
10
10-3
20
20
20
10
10a
10a
10 10a
10a
10*
n, m
n, m
n, m
Figure 3: Neural estimation rates: Dashed line shows the ground truth, circle line is the value of the parallel neural estimator from Section 4.2 and the cross line the SMI neural estimator from 17 The parallel neural estimator converges at faster for all considered &t and d.
Two types of models for (X, Y) are considered: (i) X, -N(00 are independent and Y v va (in(11 2) (i.e., X and Y share one sinusoidal feature), and (ii) the rank 2
common signal model from the previous paragraph, as well as its extension to ranks and Figure at the bottom of the previous page shows the area under the curve (AUC) of the receiver operating characteristic (ROC) as function of n for each of those models. Figure ([a) shows the results for Model (i), while Figures (iarrees[b))
(c)-(c) corresponds to Model (ii) with ranks 2, 3, and 4, respectively,
The estimator SI% mi from 2 is realized with m 1000 and I(.--) as the Kozachenko-Leoneee
estimator 38; the AUC ROC curves are computed from 100 random trials. For Figures 4(a) and 4(b), we vary the ambient dimension as 5, 10, 20, while the projection dimension
1,2, 4, d: note that k = corresponds to the SMI from [17 and k d to classic MI. In Figures 4(c) and 4(d) we consider, respectively, common signal rank 3 and The ambient dimension varied as d = 10, 20, 50, while the projection dimension is k 1, 2, 3, d. Evidently, k-SMI-based tests perform well even when is large, while tests using classic MI fail. 1-SMI has clear advantage in the model from Figure 4(a), where the common signal is 1-dimensional. but this is no longer the case for the models from Figures 4(b)-(d), where the shared structure of higher dimension. Indeed, in Figure 4(b) we see that 2-SMI generally presents the best performance it better capture the underlying structure. For Figures 4(c) and 4(d), 3-SMI slightly outperforms 2-SMI for larger sample sizes, particularly in higher dimension. This highlights the potential gain of using higher A values (to retain more information about the original signal, albeit at the cost of higher sample complexity) and the importance of adapting them the intrinsic dimensionality of the model.
Neural estimation. Figure B (on the next page) illustrates the convergence the k-SMI neural estimator from Section 4.2 as
increase together, for X = ~ (O, Ig). For comparison, include the original neural estimator of [17], which uses a single neural net to approximate shared DV potential | While both neural estimators eventually converge to the ground truth, parallel implementation converges much faster. Again note the clear decay of the true k-SMI as increases.
Sliced InfoGAN, We demonstrate simple application of k-SMI to modern machine learning. Recall the InfoGAN [6)-a GAN variant that learns disentangled latent factors by maximizing a neural estimator of the MI between those factors and the generated samples. Figure 4 left) shows InfoGAN results for MNIST where 3 latent codes (C1, C2, C3) were used for disentanglement, with C, being 10-state discrete variable and (C2g, C3) being continuous variables with values in [-2, 2]. The shown images are generated by the trained InfoGAN, where each row of corresponds to
m parallel 3-layer ReLU NNs were used, each with 30 & hidden units in each layer.
4 A 3-layer ReI.U NN was used with 20 d hidden units each layer.
5 Used experiment and code from homthttss:... :/tp::::///tttttt httpa//ww.....aaaaee////ooo/oaaa

---

Original InfoGAN
I-Slived InfoGAN
5-Sliced InfoGAN
1
6
6
6
6
4
6
6
6
6
]
3
3
3
3
3
3
3
3
3
5
5
5
5
5
5
5
5
9 9
9
9
9
9
9
9
9
6
6
6
6
6
2
2
a
2
2
/
0
0
O 1
0
0
0
0
0
3
3
3 4
3
3
3
3
3
3 9 9
3
4 4
4
4
4
4
4
4
4
4 7
9
9
9
1
9
9
9
4 4 4 1 8
9
4
9
4
4
7 7
7
7
?
7
7
7
7
8
8
8
8
8
8 4 5
9
9
9
4
6 6
6
6
6
6
6
6
6 0 7 3
9
4
4
4
4
4
4
4
1
/
D E
0
O
0
O
0
0
O
0 5 2
5
5
5
8
8
8 8
8
6
7
7
7
7
7
@
0
0
O
O
O
0
0 7
b
0
4 2
2
2
2
2
2
2
2
7
7
7
7
7
7
7
6
5
5
6
5
Figure 4: MNIST images generated via InfoGAN using neural estimators of MI (left). 1-SMI (middle), and 5-SMI (right). The latent codes C1 (encodes digits) is varied across rows, while columns correspond to (random) C22 C3 values. In all three cases, the latent codes are successfully disentangled.
different values the discrete C1, while columns corresponds to random C2, C3 values. Despite being completely unsupervised, C1 has been successfully disentangled to encode the digits 0-9. Figure Midddle) shows the resulting generated images when the neural estimator for MI is replaced with neural 1-SMI estimator with m = 10a, and Figure Figure44right right) for 5-SMI. Evidently, 1-SMI and 5-SMI successfully disentangle the latent factors, despite seeing only 103 1- (respectively 5-) dimensional projections of this very high-dimensional data.
Summary and Concluding Remarks
This paper introduced k-SMI measure of statistical dependence defined by averaging MI terms between k-dimensional projections of the considered random variables. Our objective was to quantify and provide rigorous justification for the perceived scalability of sliced information measures. We have done so by studying MC-based estimators k-SMI, neural estimation methods, and asymptotics of S1 (X; Y) under the Gaussian setting. Throughout, results with explicit dependence on d dy were provided, revealing different gains associated with slicing, from the anticipated scalability to relaxed smoothness assumptions needed for neural estimation. Numerical experiments supporting our theory were provided, as well as more advanced application to sliced infoGAN, showing that k-SMI can successfully replace classic MI even applications with more intricate underlying structure.
Future research directions, both thcoretical and applied, are abundant. In particular, we seek to derive sharp rates of decay of the residual term in 4. thereby establishing the Gaussian &-SMI as the leading term in that decomposition. Extensions of our results to the case when the projection dimensions for X and Y are different, i.e., k l2, may allow further flexibility and also of interest, We also plan to explore non-linear dimensionality reduction maps, as in the generalized sliced Wasserstein distance setting 39
well non-uniform distributions over parameterizations 40 The max-SMI, where instead of averaging over (A, B) we maximize over them, is another interesting avenue. On the application side, there are various machine learning models that utilize MI [ g 10: revisiting those with S-SMI is an appealing endeavor due to the expected gains from slicing and the formal guarantees our theory can provide for those systems.
of the projection functions (cf.
Acknowledgments and Disclosure of Funding
7. Goldfeld is partially supported by NSF grants CCF- 1947801, CCF-2046018, and DMS-2210368. and the 2020 IBM Academic Award. G. Reeves is partially supported by NSF grant
10

---

References
[1] M. Cover and A. Thomas. Elements of Information Theory. Wiley, New-York, 2nd edition, 2006.
[2] A. El Gamal and Y.-H. Kim. Network Information Theory. Cambridge University Press, 2011.
[3] D. Haussler, M. Kearns, and R. E. Schapire. Bounds on the sample complexity of Bayesian learning using information theory and the VC dimension. Machine learning, 14(1):83-113, Jan. 1994.
[4] R. Battiti. Using mutual information for selecting features in supervised neural net learning. IEEE Transactions on Neural Networks, 5(4):537-550, Jul. 1994.
[5] Viola and W. M. Wells III. Alignment by maximization of mutual information. International Journal of Computer Vision, 24(2):137-154, Sep. 1997.
[6] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. InfoGAN: Interpretable representation learning by information maximizing generative adver- sarial nets. Proceedings of the International Conference on Advances Neural Information Processing Systems MecurIPS-2016 2016.
[7] A. A. Alemi, I. Fischer, J. V. Dillon, and K. Murphy. Deep variational information bottleneck. In Proceedings of the International Conference Learning Representations (ICLR-2017),
Toulon, France, Apr. 2017.
[8] Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Ler- chner. 3-VAF: learning basic visual concepts with constrained variational framework. In Proceedings of the International Conference on Learning Representations (ICIR-2019)), New Orleans. Louisiana, USA, May 2017.
[9] R. Shwartz-Ziv and N. Tishby. Opening the black box of deep neural networks via information. arXiv preprint 1rXiv:1703.00 100.000700011001 10, 2017.
[10] A. van den Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1807 .03748, 2018
[11] A. Achille and S. Soatto. Information dropout: Learning optimal representations through noisy computation. IEEE transactions on pattern analysis and machine intelligence, 40(12))2897- 2905, Jan. 2018.
[12] M. Gabrié, A. Manoel, C. Luneau, J. Barbier, N. Macris, F. Krzakala, and L. Zdeborová, Entropy and mutual information in models of deep neural networks. arXiv preprint arXiv: 1805.09785 2018.
[13] Z. Goldfeld, E. van den Berg, K. Greenewald, Melnyk, N. Nguyen, B. Kingsbury, and Y. Polyanskiy. Estimating information flow in neural networks. In Proceedings of the International Conference on Machine Learning (ICML-2019). volume 97, pages 2299-2308, Long Beach, CA, US, Jun. 2019.
[14] Z. Goldfeld and Y. Polyanskiy. The information bottleneck problem and its applications in machine learning. IEEE Journal on Selected Areas in Information Theory, 1(1):19-38, Apr. 2020.
[15] L. Paninski. Estimation of entropy and mutual information. Neural Computation, 15:1191-123,, June 2003.
[16] D. McAllester and K. Stratos. Formal limitations on the measurement of mutual information. In Proceedings of the International Conference on Artificial Intelligence and Statistics, pages 875-884, PMLR, 2020.
[17] Z. Goldfeld and K. Greenewald. Sliced mutual information: A scalable measure of statistical de- pendence. In Proceedings of the International Conference Advances in Neural Information Processing Systems (NeurIPS-203) 2021), Online, 2021.
[18] J. Rabin, G. Peyré, J. Delon, and M. Bernot. Wasserstein barycenter and its application to texture mixing. In Proceedings of the International Conference on Scale Space and Variational Methods in Computer Vision (SSVM-2011), pages 435-446, Gedi, Isracl, May 2011.
[19] T. Vayer, R. Flamary, R. Tavenard, L. Chapel, and N. Courty. Sliced Gromov- Gromov-Wassersee Wasserstein. In Proceedings of the Annual Conference on Advances in Neural Information Processing Systems (NeurIPS-200 22019), Vancouver, Canada, Dec. 2019.

---

[20] T. Lin, Z. Zheng, E. Chen, M. Cuturi, and M. Jordan. On projection robust optimal transport: Sample complexity and model misspecification. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS-2013) pages 262-270, Online, 2021.
[21] K. Nadjahi, A. Durmus, L. Chizat, S. Kolouri, S. Shahrampour, and U. Simsekli. Statistical and topological properties of sliced probability divergences. In Proceedings of the International Conference on Advances in Neural Information Processing Systems (NeurIPS-2020)) Online,
Dec. 2020.
[22] F. Otto and C. Villani. Generalization of an inequality by Talagrand and links with the logarithmic sobolev inequality. Journal of Functional Analysis, 173(2)33114000, 2000.
[23] Gentil, C. Léonard, L. Ripani, and L. Tamanini. An entropic interpolation proof of the HWI inequality. Stochastic Processes and their Applications, 130(():907-922) 2020.
[24] Y. Polyanskiy and Y. Wu. Wasserstein continuity of entropy and outer bounds for interference channels. IEEE Transactions on Information Theory, 62(7):3992-4002, Jul. 2016.
[25] M. I. Belghazi, A. Baratin, S. Rajeswar, S. Ozair, Y. Bengio, A. Courville, and R. D. Hjelm. Mutual information neural estimation. In Proceedings of the International Conference on Machine Learning (2CML-2018), volume 80, pages 531-540, Jul. 2018.
[26] B. Poole, S. Ozair, A. van den Oord, A. A. Alemi, and G. Tucker. On variational lower bounds of mutual information. In NeurIPS Workshop on Bayesian Deep Learning, 2018.
[27] Song and Ermon. Understanding the limitations of variational mutual information estima- tors. arXiv preprint 1eXivv:1910.6220
2019.
[28] C. Chan, A. Al-Bashabsheh, H. P. Huang, M. Lim, D. S. H. Tam, and C. Zhao. Neural entropic estimation: A faster path to mutual information estimation. arXiv preprint 19X.:11905.1255 2019.
[29] S. Sreekumar and Z. Goldfeld. Neural estimation of statistical divergences. Journal of Machine Learning Research, 2022.
[30] E. Meckes. Approximation of projections of random vectors. Journal of Theoretical Probability, 25(2):333-352, 2012.
[31] E. Meckes. Projections of probability distributions: A mhesure- Dvoretzky theorem. In Geometric aspects of functional analysis, pages 317-326. Springer, 2012.
[32] G. Reeves. Conditional central limit theorems for Gaussian projections. In Proceedings of IEEE International Symposium on Information Theory (ISIT-2017), pages 3045-3049, IEEE, 2017.
[33] Y. Chikuse. Statistics special manifolds, volume 174. Springer Science & Business Media, 2003.
[34] Walter Gander. Algorithms for the qr decomposition. Research Report, 80(02): 1980.
[35] S. Sreekumar, Zhang, and Z. Goldfeld. aon-tsymptoiic performance guarantees for neural estimation of f-divergences. In Proceedings of the International Conference on Artificial Intelligence and Statisties (ASSTATS-202)), pages 3322-3330, 2021.
[36] K. Nadjahi, A. Durmus, P.E Jacob, R. Badcau, and U. Simsekli. Fast approximation of the sliced-Wasseree Wasserstein distance using concentration of random projections. In Proceedings of the International Conference on Advances in Neural Information Processing Systems (NeurIPS-202I), Online, 2021.
[37] A. Rakotomamonjy, M. Z. Alaya, M. Berar, and G. Gasso. Statistical and topological properties of gaussian smoothed sliced probability divergences. arXiv preprint arXiv:21 10524, 2021.
[38] H. Stögbauer A. Kraskov and P. Grassberger. Estimating mutual information. Physical Review E, 69(6):066133.. June 2004.
[39] Soheil Kolouri, Kimia Nadjahi, Umut Simsekli, Roland Badeau, and Gustavo Rohde. Gen- eralized sliced Wasserstein distances. In Proceedings of the International Conference on Neural Information Processing Systems (NeurIPS-2019), volume 32, pages 261-272, Van- couver, Canada, 2019.
12

---

[40] Khai Nguyen, Nhat Ho, Tung Pham, and Hung Bui. Distributional sliced- wasserstein and ap- plications to generative modeling. In Proceedings of the International Conference on Learning Representations (ICIR-22020) Online, 2020.
[41] O. Rioul. Information theoretic proofs of entropy power inequalities. IEEE Transactions on Information Theory, 57(1):33-55, 2010.
[42] G. W. Anderson, A. Guionnet, and O. Zeitouni. An introduction to random matrices. Number
1 118. Cambridge university press, 2010.
[43] M. Raginsky and I. Sason. Concentration of measure inequalities in information theory, communications, and coding. Foundations and Trends@ in Communications and Information Theory, 10(1-2):1-246, 2013. 2nd edition.
[44] Y. Chikuse. The matrix angular central Gaussian distribution. Journal of Multivariate Analysis, 33(2):265-275, 1990.
[45] T. T. Cai, R. Han, and Zhang. On the non non-asym asymptotiiicc concentration of heteroskedastic Wishart-type matrix. Electronic Journal of Probability, 27:1-40, 2022.
[46] Shinpei Imori and Dietrich Von Rosen, On the mean and dispersion of the Peore-Penrose generalized inverse of Wishart matrix. The Electronic Journal of Linear Algebra, 36:124-133 133, 2020.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] Proofs in main text are all complete in the supplement.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Partially data and instructions included, and links InfoGAN experiment code; but are not ready to release all codes at submission time.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] Via running experiments sufficiently many times until curves converge and averaging.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No] No large-scale experiments.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent obtained from people whose data you's 're
using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing conducted research with human subjects...
13

---

(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "tvzrqhKi-I2", "reviewer": "Reviewer_MwzW", "summary": "The paper proposes k-SMI (k-sliced mutual information) as a measure of statistical dependence defined by averaging MI (mutual information) terms between k-dimensional projections of high dimensional random variables, which is an extension of SMI (sliced mutual information). Although SMI avoids the curse of dimensionality in mutual information estimation by averaging\nover the one-dimensional slices, the bounds rely on high-level assumptions that are hard to verify in practice and do not explicitly capture dependence on the ambient dimension, which is crucial for understanding scalability. The authors extend this theory to k-dimensional\nprojections and provide justifications with explicit dependence on k and the dimensions of data via numerical experiments. Additionally, they provide an application to sliced infoGAN, showing that k-SMI can successfully replace classic MI even in applications with more intricate\nunderlying structure.", "questions": "Significance\nI recognize that estimating MI in high dimensions is a hard problem. This approach of k-SMI is interesting in that regard. However, it seems to me that one could just as easily perform some kind of dimensionality reduction first (e.g. using diffusion maps, or even PCA) and then estimating the MI on the reduced dimensions. This would likely be computationally cheaper as the MC step in k-SMI is unlikely to be computationally fast. Also, this would allow for nonlinear projections if the dimensionality reduction method is nonlinear. In contrast, k-SMI only performs linear projections. What advantage, if any, does k-SMI have over this approach?\n\nExperiments\n* In line 312 — the result of the independence test, It seems the sinusoidal model (a) and the common signal model (b) perform differently. In (a), it only works when k = 1 when d is large; In particular, when d = 20, the AUC is pretty small for 2-SMI. On the other hand, both 1-SMI and 2-SMI work pretty well in (b) even when d is large enough. Although the difference of the two models is clear, how they could make such a difference in result is not discussed.\n* Again in the independence test, it is shown that there is potential gain of higher k, but I am worrying about whether the simulated data is too artificial, since Z and V are both simulated from multivariate Gaussian with identity covariance matrices (line 282); clearly model (a) does not provide similar results. So I think experimenting with the theory on more models would be preferred.\n* In line 318 — the experiment of sliced InfoGAN, it is only shown that the k-SMI successfully disentangles the latent factors with k = 1. Are there any other results from other k values? In addition, it would be nice if some quantitative measure of performance could be given for this experiment.", "limitations": "Limitations were generally not discussed. Are there circumstances where the authors expect k-SMI to fail in some way?", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The originality is good. While the approach is a straightforward extension of SMI, the authors provide significantly improved theoretical results that also apply to SMI.\n* The theoretical results are good as the authors provide a comprehensive quantitative study of dependence on dimension of k-SMI, encompassing the MC error, performance bounds for neural estimators, and asymptotics of the population k-SMI as dimension increases.\n* The paper is generally clear and well-written.\n\nWeaknesses\n* I'm not entirely convinced of the significance of this work. See questions below.\n* The experiments are decent but could use some improvement. See questions below.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CJM-soCFr5C", "reviewer": "Reviewer_p5hU", "summary": "The paper proposes $k$-sliced Mutual Information (k-SMI) which is an extension of sliced Mutual Information (SMI). The difference between SMI and k-SMI is that k-SMI projects original measures to k-dimensional subspaces instead of unidimensional subspaces. Similar to SMI, the authors derive structural properties for k-SMI including Identification of independence, Bounds, Relative entropy and variational form, Entropy decomposition, Chain rule, and Tensorization. The authors derive bounds on the error of Monte Carlo (MC)-based estimates of k-SMI using the continuity of differential entropy in the 2-Wasserstein metric. Moreover, the author explores asymptotics of the population k-SMI, Gaussian approximation results, and neural estimation of SMI. On the application side, k-SMI is applied to sliced InfoGAN for disentangled representation learning.", "questions": "* Why we should consider $k$-SMI instead of 1-SMI in practice?\n* What is the computational complexity of the projection step in k-SMI?", "limitations": "This is fundamental work, hence, there is no foreseen negative social impact.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n## Originality \n* This is the first work that proposes a variant of mutual information from $k$-dimensional subspace projections. The corresponding theoretical analysis is also original. \n\n## Quality and Significance\n* The investigation of the paper is important for scalable and no curse of dimensionality mutual information estimation\n\n## Clarity\n* The paper is well-written and easy to follow.\n\n# Weaknesses\n* Sampling uniformly from Stiefel manifold requires using QR decomposition that makes Monte Carlo estimation of k-SMI slower than SMI. Therefore, some discussion and empirical investigation are needed.\n* It is hard to see the benefits of k-SMI over 1-SMI from the empirical experiments. Also, the comparison of Info-GAN and sliced Info-GAN does not show the superior of k-SMI over k-SMI.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0ucSWdgXv3Y", "reviewer": "Reviewer_2P49", "summary": "Sliced mutual information (SMI) serves as a surrogate measure of dependence to classic MI, but a quantitative characterization of how its estimation rates depend on the dimension remains obscure.\nThis paper naturally extends the original SMI definition to k-SMI and provides a comprehensive analysis on how k-SMI's estimation rates depend on dimension.\nThe authors utilize the neural estimation framework to estimate k-SMI, with optimal convergence rates.\nThey discuss numerical considerations and perform a thorough simulation study and real data analysis, showing the favorable performance of their method.", "questions": "* For the proposed measure k-SMI, there seems to be a little lack of innovation.\nIt is a natural extension of SMI.\nAnd I don't find the advantages of k-SMI($k>1$) over SMI.\nFurthermore, the computational cost of k-SMI is larger than SMI.\n\n* There is not any discussion about the complexity of the proposed method.\nThough the proposed method can be estimated by a neural network with optimal convergence rates, the computational cost is also important in practice.\nThe proposed method need to solve an optimization problem by neural network, which may be a disadvantage compared with other dependence measurements.\nSome discussion and empirical results are needed.\n\n* There seem to be many existing methods that measure the dependence between random variables without using a neural network.\nThe authors may compare their algorithm with these methods and clarify the advantages and disadvantages. ", "limitations": "  ", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* The authors provide a comprehensive analysis of the quantitative account of the scalability question of k-SMI.\nThey provide ample theoretical analysis on it.\n\n* The authors apply a neural estimation framework to estimate k-SMI.\nWhat's more, they provide this estimation with theoretical guarantees, which claims optimal convergence rates.\n\n* The simulation study and real data analysis are very sufficient.\nThe authors illustrate the effectiveness from many aspects.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept