Title: Max-Sliced Mutual Information
Venue: NeurIPS 2023
Accepted: yes
Keywords: CCA

dimensionality reduction

information theory

mutual information

neural estimation

slicing
Abstract: Quantifying dependence between high-dimensional random variables is central to statistical learning and inference. Two classical methods are canonical correlation analysis (CCA), which identifies maximally correlated projected versions of the original variables, and Shannon's mutual information, which is a universal dependence measure that also captures high-order dependencies. However, CCA only accounts for linear dependence, which may be insufficient for certain applications, while mutual information is often infeasible to compute/estimate in high dimensions. This work proposes a middle ground in the form of a scalable information-theoretic generalization of CCA, termed max-sliced mutual information (mSMI). mSMI equals the maximal mutual information between low-dimensional projections of the high-dimensional variables, which reduces back to CCA in the Gaussian case. It enjoys the best of both worlds: capturing intricate dependencies in the data while being amenable to fast computation and scalable estimation from samples. We show that mSMI retains favorable structural properties of Shannon's mutual information, like variational forms and identification of independence. We then study statistical estimation of mSMI, propose an efficiently computable neural estimator, and couple it with formal non-asymptotic error bounds. We present experiments that demonstrate the utility of mSMI for several tasks, encompassing independence testing, multi-view representation learning, algorithmic fairness, and generative modeling. We observe that mSMI consistently outperforms competing methods with little-to-no computational overhead.

Max-Sliced Mutual Information

Abstract

Quantifying dependence between high-dimensional random variables is central to statistical learning and inference. Two classical methods are canonical correlation analysis (CCA), which identifies maximally correlated projected versions of the original variables, and Shannon's mutual information, which is a universal dependence measure that also captures high-order dependencies. However, CCA only accounts for linear dependence, which may be insufficient for certain applications, while mutual information is often infeasible to compute/estimate in high dimensions. This work proposes a middle ground in the form of a scalable informationtheoretic generalization of CCA, termed max-sliced mutual information (mSMI). mSMI equals the maximal mutual information between low-dimensional projections of the high-dimensional variables, which reduces back to CCA in the Gaussian case. It enjoys the best of both worlds: capturing intricate dependencies in the data while being amenable to fast computation and scalable estimation from samples. We show that mSMI retains favorable structural properties of Shannon's mutual information, like variational forms and identification of independence. We then study statistical estimation of mSMI, propose an efficiently computable neural estimator, and couple it with formal non-asymptotic error bounds. We present experiments that demonstrate the utility of mSMI for several tasks, encompassing independence testing, multi-view representation learning, algorithmic fairness, and generative modeling. We observe that mSMI consistently outperforms competing methods with little-to-no computational overhead.

1 Introduction

Dependence measures between random variables are fundamental in statistics and machine learning for tasks spanning independence testing [1-3], clustering [4, 5], representation learning [6, 7], and self-supervised learning [8-10]. There is a myriad of measures quantifying different notions of dependence, with varying statistical and computational complexities. The simplest is the Pearson correlation coefficient [11], which only captures linear dependencies. At the other extreme is Shannon's mutual information [12], which is a universal dependence measure that is able to identify arbitrarily intricate dependencies. Despite its universality and favorable properties, accurately estimating mutual information from data is infeasible in high-dimensional settings. First, mutual information estimation rates suffers from the curse of dimensionality, whereby convergence rates deteriorate exponentially with dimension [13]. Additionally, computing mutual information requires integrating a log-likelihood ratio over a high-dimensional space, which is generally intractable.

Between these two extremes is the popular canonical correlation analysis (CCA) [14], which identifies maximally correlated linear projections of variables. Nevertheless, classical CCA still only captures linear dependence, which has inspired nonlinear extensions such as Hirschfeld-Gebelein-Rényi (HGR) maximum correlation [15-17], kernel CCA [18, 19], deep CCA [20, 7], and various other generalizations [21-24]. However, HGR is computationally infeasible, while kernel and deep CCA can be burdensome in high dimensions, as they require optimization over reproducing kernel Hilbert spaces or deep neural networks, respectively. To overcome these shortcomings, this work proposes

max-sliced mutual information (mSMI)-a scalable information-theoretic extension of CCA that captures the full dependence structure while only requiring optimization over linear projections.

1.1 Contributions

The mSMI is defined as the maximal mutual information between linear projections of the variables. Namely, the k -dimensional mSMI between X and Y with values in R d x and R d y , respectively, is 1

where St( k, d ) is the Stiefel manifold of d × k matrices with orthonormal columns. Unlike the nonlinear CCA variants that use nonlinear feature extractors in the high-dimensional ambient spaces, mSMI retains the linear projections of CCA and captures nonlinear structures in the low-dimensional feature space. This is done by using the mutual information between the projected variables, rather than correlation, as the optimization objective. Beyond being considerably simpler from a computational standpoint, this crucial difference allows mSMI to identify the full dependence structure, akin to classical mutual information. mSMI can also be viewed as the maximized version of the average-sliced mutual information (aSMI) [25, 26], which averages I (A ⊺ X ; B ⊺ Y ) with respect to (w.r.t.) the Haar measure over St( k, d x ) × St( k, d y ) . However, we demonstrate that compared to aSMI, mSMI benefits from improved neural estimation error bounds and a clearer interpretation.

We show that mSMI inherits important properties of mutual information, including identification of independence, tensorization, and variational forms. For jointly Gaussian ( X,Y ) , the optimal mSMI projections coincide with those of k -dimensional CCA [27], posing mSMI as a natural informationtheoretic generalization. Beyond the Gaussian case, the solutions differ and mSMI may yield more effective representations for downstream tasks due to the intricate dependencies captured by mutual information. We demonstrate this superiority empirically for multi-view representation learning.

For efficient computation, we propose an mSMI neural estimator based on the Donsker-Varadhan (DV) variational form [28]. Neural estimators have seen a surge in interest due to their scalability and compatibility with gradient-based optimization [29-36]. Our estimator employs a single model that composes the projections with the neural network approximation of the DV critic, and then jointly optimizes them. This results in both the estimated mSMI value and the optimal projection matrices. Building on recent analysis of neural estimation of f -divergences [37, 38], we establish non-asymptotic error bounds that scale as O ( k 1 / 2 ( ℓ -1 / 2 + kn -1 / 2 ) ) , where ℓ and n are the numbers of neurons and ( X,Y ) samples, respectively. Equating ℓ and n results in the (minimax optimal) parametric estimation rate, which highlights the scalability of mSMI and its compatibility to modern learning settings.

In our empirical investigation, we first demonstrate that our mSMI neural estimator converges orders of magnitude faster than that of aSMI [26]. This is because the latter requires (parallel) training of many neural estimators corresponding to different projection directions, while the mSMI estimator optimizes a single combined model. Notwithstanding the reduction in computational overhead, we show that mSMI outperforms average-slicing for independence testing. Next, we compare mSMI with deep CCA [20, 7] by examining downstream classification accuracy based on representations obtained from both methods in a multi-view learning setting. Remarkably, we observe that even the linear mSMI projections outperform nonlinear representations obtained from deep CCA. We also consider an application to algorithmic fairness under the infomin framework [39]. Replacing their generalized Pearson correlation objective with mSMI, we again observe superior performance in the form of more fair representations whose utility remains on par with the fairness-agnostic model. Lastly, we devise a max-sliced version of the InfoGAN by replacing the classic mutual information regularizer with its max-sliced analog. We show that despite the low-dimensional projections, the max-sliced InfoGAN successfully learns to disentangle the latent space and generates quality samples.

2 Background and Preliminaries

Notation. For a, b ∈ R , we use the notation a ∧ b = min { a, b } and a ∨ b = max { a, b } . For d ≥ 1 , ∥ · ∥ is the Euclidean norm in R d . The Stiefel manifold of d × k matrices with orthonormal columns

is denoted by St( k, d ) . For a d × k matrix A , we use p A : R d → R k for the orthogonal projection onto the row space of A . For A ∈ R d × k with rank(A) = r ≤ k ∧ d , we write σ 1 (A) , . . . , σ r (A) for its non-zero singular values, and assume without loss of generality (w.l.o.g.) that they are arranged in descending order. Similarly, the eigenvalues of a square matrix Σ ∈ R d × d are denoted by λ 1 (Σ) , . . . , λ d (Σ) . Let P ( R d ) denote the space of Borel probability measures on R d . For µ, ν ∈ P ( R d ) , we use µ ⊗ ν to denote a product measure, while spt( µ ) designates the support of µ . All random variables throughout are assumed to be continuous w.r.t. the Lebesgue measure. For a measurable map f , the pushforward of µ under f is denoted by f ♯ µ = µ ◦ f -1 , i.e., if X ∼ µ then f ( X ) ∼ f ♯ µ . For a jointly distributed pair ( X,Y ) ∼ µ XY ∈ P ( R d x × R d y ) , we write Σ X and Σ XY for covariance matrix of X and cross-covariance matrix of ( X,Y ) , respectively.

Canonical correlation analysis. CCA is a classical method for devising maximally correlated linear projections of a pair of random variables ( X,Y ) ∼ µ XY ∈ P ( R d x × R d y ) via [14]

where the former objective is the correlation coefficient ρ ( θ ⊺ X,ϕ T Y ) between the projected variables and the equality follows from invariance of ρ to scaling. The global optimum has an analytic form as ( θ CCA , ϕ CCA ) = (Σ -1 / 2 X θ 1 , Σ -1 / 2 Y ϕ 1 ) , where ( θ 1 , ϕ 1 ) is the (unit-length) top left- and right-singular vector pair associated with the largest singular value of T XY := Σ -1 / 2 X Σ XY Σ -1 / 2 Y ∈ R d x × d y . This solution is efficiently computable in O (( d x ∨ d y ) 3 ) time, given that the population correlation matrices are known. CCA extends to k -dimensional projections via the optimization [27]

with the optimal CCA matrices being (A CCA , B CCA ) = (Σ -1 / 2 X U k , Σ -1 / 2 Y V k ) , where U k and V k are the matrices of the first k left- and right-singular vectors of T XY . The optimal objective value then becomes the sum of the top k singular values of T XY (namely, its Ky Fan k -norm).

Divergences and information measures. Let µ, ν ∈ P ( R d ) satisfy µ ≪ ν , i.e., µ is absolutely continuous w.r.t. ν . The Kullback-Leibler (KL) divergence is defined as D ( µ ∥ ν ) := ∫ R d log( dµ/dν ) dµ . We have D ( µ ∥ ν ) ≥ 0 , with equality if and only if (iff) µ = ν . Mutual information and differential entropy are defined from the KL divergence as follows. Let ( X,Y ) ∼ µ XY ∈ P ( R d x × R d y ) and denote the corresponding marginal distributions by µ X and µ Y . The mutual information between X and Y is given by I ( X ; Y ) := D ( µ XY ∥ µ X ⊗ µ Y ) and serves as a measure of dependence between those random variables. The differential entropy of X is defined as h ( X ) = h ( µ X ) := -D ( µ X ∥ Leb) . Mutual information between (jointly) continuous variables and differential entropy are related via I ( X ; Y ) = h ( X ) + h ( Y ) -h ( X,Y ) ; decompositions in terms of conditional entropies are also available [40].

3 Max-Sliced Mutual Information

We now define the k -dimensional mSMI, establish structural properties thereof, and explore the Gaussian setting and its connections to CCA. We focus here on the case of (linear) k -dimensional projections and discuss extensions to nonlinear slicing in Section 3.3.

Definition 1 (Max-sliced mutual information) . For 1 ≤ k ≤ d x ∧ d y , the k -dimensional mSMI between ( X,Y ) ∼ µ XY ∈ P ( R d x × R d y ) is

where St( k, d ) is the Stiefel manifold of d × k matrices with orthonormal columns.

The mSMI measures Shannon's mutual information between the most informative k -dimensional projections of X and Y . It can be viewed as a maximized version of the aSMI SI k ( X ; Y ) from

[25, 26], defined as the integral of I (A ⊺ X ; B ⊺ Y ) w.r.t. the Haar measure over St( k, d x ) × St( k, d y ) . For d = d x = d y , we have SI d ( X ; Y ) = SI d ( X ; Y ) = I ( X ; Y ) due to invariance of mutual information to bijections. The supremum in mSMI is achieved since the Stiefel manifold is compact and the function (A , B) ↦→ I (A ⊺ X ; B ⊺ Y ) is Lipschitz and thus continuous (Lemma 2 of [26]).

̸

Remark 1 (Multivariate and conditional mSMI) . The mSMI definition above extends to the multivariate and conditional cases as follows. Let ( X,Y,Z ) ∼ µ XYZ ∈ P ( R d x × R d y × R d z ) . The k -dimensional multivariate and conditional mSMI functionals are, respectively, SI k ( X,Y ; Z ) := max A , B , C I (A ⊺ X, B ⊺ Y ; C ⊺ Z ) and SI k ( X ; Y | Z ) := max A , B , C I (A ⊺ X ; B ⊺ Y | C ⊺ Z ) . Connections between SI k ( X ; Y ) and its multivariate and conditional versions are given in the proposition to follow. We also note that one may generalize the definition of SI k ( X ; Y ) to allow for projections into feature spaces of different dimensions, i.e., A ∈ St( k x , d x ) and B ∈ St( k y , d y ) , for k x = k y . We expect our theory to extend to that case, but leave further exploration for future work.

In the spirit of mSMI, we define the max-sliced differential entropy.

Definition 2 (Max-sliced entropy) . The k -dimensional max-sliced (differential) entropy of X ∼ µ X ∈ P ( R d ) is sh k ( X ) := sh k ( µ ) := sup A ∈ St( k,d ) h (A ⊺ X ) .

An important property of classical differential entropy is the maximum entropy principle [40], which finds the highest entropy distribution within given class. In Appendix B, we study the max-sliced entropy maximizing distribution in several common scenarios. For instance, we show that sh k is maximized by the Gaussian distribution under a fixed (mean and) covariance constraint. Namely, letting P 1 ( m, Σ) := { µ ∈ P ( R d ) : spt( µ ) = R d , E µ [ X ] = m, E µ [ ( X -m )( X -m ) ⊺ ] = Σ } , we have argmax µ ∈P 1 ( µ, Σ) sh k ( µ ) = N ( m, Σ) . An intimate connection between max-sliced entropy and PCA is established in the sequel, under the Gaussian setting.

Remark 2 (Sliced divergences) . The slicing technique has originated as a means to address scalability issues concerning statistical divergences. Significant attention was devoted to sliced Wasserstein distances as discrepancy measures between probability distributions [41-47]. As such, the sliced Wasserstein distance differs from mutual information and its sliced variants, which quantify dependence between random variables, rather than discrepancy per se. Additionally, as Wasserstein distances are rooted in optimal transport theory, they heavily depend on the geometry of the underlying data space. Mutual information, on the other hand, is induced by the KL divergence, which only depends on the log-likelihood of the considered distributions and overlooks geometry.

3.1 Structural Properties

The following proposition lists useful properties of the mSMI, which are similar to those of the average-sliced variant (cf. [26, Proposition 1]) as well as Shannon's mutual information itself.

Proposition 1 (Structural properties) . The following properties hold:

Bounds: For any integers k 1 < k 2 : SI k 1 ( X ; Y ) ≤ SI k 1 ( X ; Y ) ≤ SI k 2 ( X ; Y ) ≤ I ( X ; Y ) .

Identification of independence: SI k ( X ; Y ) ≥ 0 with equality iff ( X,Y ) are independent.

KL divergence representation: We have

Sub-chain rule: For any random variables X 1 , . . . , X n , Y , we have

Tensorization: For mutually independent { ( X i ,Y i ) } n i =1 , SI k ( { X i } n i =1 ; { Y i } n i =1 ) = n ∑ i =1 SI k ( X i ; Y i ) .

The proof follows by similar arguments to those in the average-sliced case, but is given for completeness in Supplement A.1. Of particular importance are Properties 2 and 3. The former renders mSMI sufficient for independence testing despite being significantly less complex than the classical mutual information between the high-dimensional variables. The latter, which represent mSMI as a supremized KL divergence, is the basis for neural estimation techniques explored in Section 4.

Remark 3 (Relation to average-SMI) . Beyond the inequality relationship in Property 1 above, Proposition 4 in [25] (paraphrased) shows that for matrices W x , W y and vectors b x , b y of appropriate dimensions, we have sup W x , W y ,b x ,b y SI 1 (W ⊺ x X + b x ; W ⊺ y Y + b y ) = SI 1 ( X ; Y ) , and the relation readily extends to projection dimension k > 1 . In words, optimizing the aSMI over linear transformations of the high-dimensional data vectors coincides with the max-sliced version. This further justifies the interpretation of SI k ( X ; Y ) as the information between the two most informative representations of X,Y in a k -dimensional feature space. It also suggests that mSMI is compatible for feature extraction tasks, as explored in Section 5.3 ahead.

3.2 Gaussian Max-SMI versus CCA

The mSMI is an information-theoretic extension of the CCA coefficient ρ CCA ( X,Y ) , which is able to capture higher order dependencies. Interestingly, when ( X,Y ) are jointly Gaussian, the two notions coincide. We next state this relation and provide a closed-form expression for the Gaussian mSMI.

Proposition 2 (Gaussian mSMI) . Let X ∼ N ( m X , Σ X ) and Y ∼ N ( m Y , Σ Y ) be d x - and d y -dimensional jointly Gaussian vectors with nonsingular covariance matrices and cross-covariance Σ XY . For any k ≤ d x ∧ d y , we have

where (A CCA , B CCA ) are the CCA solutions from (2) , T XY = Σ -1 / 2 X Σ XY Σ -1 / 2 Y ∈ R d x × d y , and σ k (T XY ) ≤ . . . ≤ σ 1 (T XY ) ≤ 1 are the top k singular values of T XY (ordered).

This proposition is proven in Supplement A.2. We first show that the optimization domain of SI k ( X ; Y ) can be switched from the product of Stiefel manifolds to the space of all matrices subject to a unit variance constraint (akin to (2)), without changing the mSMI value. This implies that the CCA solutions (A CCA , B CCA ) from (2) are feasible for mSMI and we establish their optimality using a generalization of the Poincaré separation theorem [48, Theorem 2.2]. Specializing Proposition 2 to one-dimensional projections, i.e., when k = 1 , the mSMI is given in terms of the canonical correlation coefficient ρ CCA ( X,Y ) := sup ( ϕ,θ ) ∈ R dx × R dy ρ ( θ ⊺ X,ϕ T Y ) . Namely,

where ( θ CCA , ϕ CCA ) are the global optimizers of ρ CCA ( X,Y ) .

Remark 4 (Beyond Gaussian data) . While the mSMI solution coincides with that of CCA in the Gaussian case, this is no longer expected to hold for non-Gaussian distributions. CCA is designed to maximize correlation, while mSMI has Shannon's mutual information between the projected variables as the optimization objective. Unlike correlation, mutual information captures higher order dependencies between the variables, and hence the optimal mSMI matrices will not generally coincide with (A CCA , B CCA ) . Furthermore, the intricate dependencies captured by mutual information suggest that the optimal mSMI projections may yield representations that are more effective for downstream tasks. We empirically verify this observation in Section 5 on several tasks, including classification, multi-view representation learning, and algorithmic fairness.

Similarly to the above, the Gaussian max-sliced entropy is related to PCA [49, 14]. In Supplement A.3, we prove the following.

Proposition 3 (Gaussian max-sliced entropy) . For a d -dimensional Gaussian variable X ∼ N ( m, Σ) , we have sh k ( X ) = sup A ∈ St( k,d ) h ( A ⊺ X ) = h (A ⊺ PCA X ) = 0 . 5 ∑ k i =1 log ( 2 πeλ i (Σ) ) , where A PCA is optimal PCA matrix and λ 1 (Σ) , . . . λ k (Σ) are the top k eigenvalues of Σ .

Note that the eigenvalues λ 1 (Σ) , . . . λ k (Σ) are non-negative since Σ is a covariance matrix. Extrapolating beyond the Gaussian case, this poses max-sliced entropy as an information-theoretic generalization of PCA for unsupervised dimensionality reduction. An analogous extension using the Rényi entropy of order 2 was previously considered in [50] for the purpose of binary classification. In that regard, sh k ( X ) can be viewed as the α -Rényi variant when α → 1 .

3.3 Generalizations Beyond Linear Slicing

The notion of mSMI readily generalizes beyond linear slicing. Fix d x , d y ≥ 1 , k ≤ d x ∧ d y , and consider two (nonempty) function classes G ⊆ { g : R d x → R k } and H ⊆ { h : R d y → R k } .

Definition 3 (Generalized mSMI) . The generalized mSMI between ( X,Y ) ∼ µ XY ∈ P ( R d x × R d y ) w.r.t. the classes G and H is SI G , H ( X ; Y ) := sup ( g,h ) ∈G×H I ( g ( X ); h ( Y ) ) .

The generalized variant reduces back to SI k ( X ; Y ) by taking G = G proj := { p A : A ∈ St( k, d x ) } and H = H proj := { p B : B ∈ St( k, d y ) } , but otherwise allows more flexibility in the way ( X,Y ) are mapped into R k . We also have that if G ⊆ G ′ and H ⊆ H ′ , then SI G , H ( X ; Y ) ≤ SI G ′ , H ′ ( X ; Y ) ≤ I ( X ; Y ) , which corresponds to Property 1 from Proposition 1. Further observations are as follows.

Proposition 4 (Properties) . For any classes G , H , we have that SI G , H always satisfies Properties 3-5 from Proposition 1. If further G proj ⊆ G and H proj ⊆ H , then SI G , H also satisfies Property 2.

We omit the proof as it follows by the same argument as Proposition 1, up to replacing the linear projections with the functions ( g, h ) ∈ G × H . In practice, the classes G and H are chosen to be parametric, typically realized by artificial neural networks. As discussed in Remark 5 ahead, this is well-suited to the neural estimation framework for mSMI (both standard and generalized). Lastly, note that SI G , H ( X ; Y ) corresponds to the objective of multi-view representation learning [51], which considers the maximization of the mutual information between NN-based representation of the considered variables. We further investigate this relation in Section 5.3.

4 Neural Estimation of Max-SMI

We study estimation of mSMI from data, seeking an efficiently computable and scalable approach subject to formal performance guarantees. Towards that end, we observe that the mSMI is compatible with neural estimation [29, 38] due to its convenient variational form. In what follows we derive the neural estimator, describe the algorithm to compute it, and provide non-asymptotic error bounds.

4.1 Estimator and Algorithm

Fix d x , d y ≥ 1 , k ≤ d x ∧ d y , and µ XY ∈ P ( R d x × R d y ) ; we suppress k, d x , d y from our notation of the considered function classes. Neural estimation is based on the DV variational form: 2

where ( X,Y ) ∼ µ XY , ( ˜ X, ˜ Y ) ∼ µ X ⊗ µ Y , and F is the class of all measurable functions f : R d x × R d y → R (often referred to as DV potentials) for which the expectations above are finite. As mSMI is the maximal mutual information between projections of X,Y , we have

where F proj := { f ◦ ( p A , p B ) : f ∈ F , (A , B) ∈ St( k, d x ) × St( k, d y ) } . The RHS above is given by optimizing the DV objective L DV over the composed class F proj , which first projects ( X,Y ) ↦→ (A ⊺ X, B ⊺ Y ) and then applies a DV potential f : R k × R k → R to the projected variables.

Neural estimator. Neural estimators parametrize the DV potential by neural nets, approximate expectations by sample means, and optimize the resulting empirical objective over parameter space. Let F nn be a class of feedforward networks with input space R k × R k and real-valued outputs. 3 Given i.i.d. samples ( X 1 , Y 1 ) , . . . , ( X n , Y n ) from µ XY , we first generate negative samples (i.e., from µ X ⊗ µ Y ) by taking ( X 1 , Y σ (1) ) , . . . , ( X n , Y σ ( n ) ) , where σ ∈ S n is a permutation such that

̸

σ ( i ) = i , for all i = 1 , . . . , n . The neural estimator of SI k ( X ; Y ) is now given by

where F proj nn := { f ◦ ( p A , p B ) : f ∈ F nn , (A , B) ∈ St( k, d x ) × St( k, d y ) } is the composition of the neural network class with the projection maps. The projection maps can be absorbed into the network architecture as a first linear layer that maps the ( d x + d y ) -dimensional input to a 2 k -dimensional feature vector, which is then further processed by the original f ∈ F nn network. Note that projection onto the Stiefel manifold can be efficiently and differentiably computed via QR decomposition. Hence, the Stiefel manifold constraint can be easily enforced by setting A,B to be the projections of unconstrained d × k matrices. Further details on the implementation are given in Supplement C.

Remark 5 (Nonlinear slicing) . For learning tasks that may need more expressive representations of ( X,Y ) , one may employ the nonlinear mSMI variant from Section 3.3. In practice, the classes G = { g θ } and H = { h ϕ } are taken to be parametric, realized by neural networks. These networks naturally compose with the DV critic f ψ , resulting in a single compound model f ψ ◦ ( g θ , h ϕ ) .

4.2 Performance Guarantees

Neural estimation involves three sources of error: (i) function approximation of the DV potential; (ii) empirical estimation of the means; and (iii) optimization, which comes from employing suboptimal (e.g., gradient-based) routines. Our analysis provides sharp non-asymptotic bounds for errors of type (i) and (ii), leaving the account of the optimization error for future work. We focus on a class of ℓ -neuron shallow ReLU networks, although the ideas extend to other nonlinearities and deep architectures. Define F ℓ nn as the class of all f : R k × R k → R , f ( z ) = ∑ ℓ i =1 β i ϕ ( ⟨ w i , z ⟩ + b i ) + ⟨ w 0 , z ⟩ + b 0 , whose parameters satisfy max 1 ≤ i ≤ ℓ ∥ w i ∥ 1 ∨ | b i | ≤ 1 , max 1 ≤ i ≤ ℓ | β i | ≤ a ℓ 2 ℓ , and | b 0 | , ∥ w 0 ∥ 1 ≤ a ℓ , where ϕ ( z ) = z ∨ 0 is the ReLU activation and a ℓ = log log ℓ ∨ 1 .

Consider the neural mSMI estimator ̂ SI n,ℓ k := ̂ SI F ℓ nn k ( X n , Y n ) (see (5)). We provide convergence rates for it over an appropriate distribution class, drawing upon the results of [37] for neural estimation of f -divergences. For compact X ⊂ R d x and Y ⊂ R d y , let P ac ( X × Y ) be the set of all Lebesgue absolutely continuous joint distribution µ XY with spt( µ XY ) ⊆ X ×Y . Denote the Lebesgue density of µ XY by f XY . The distribution class of interest is 4

which, in particular, contains distributions whose densities are bounded from above and below on X × Y with a smooth extension to an open set covering X × Y . This includes uniform distributions, truncated Gaussians, truncated Cauchy distributions, etc. The following theorem provides the convergence rate for the mSMI neural estimator, uniformly over P k ( M,b ) .

Theorem 1 (Neural estimation error) . For any M,b ≥ 0 , we have

where C depends on M , b , k , and the radius of the ambient space ∥X×Y∥ := sup ( x,y ) ∈X×Y ∥ ( x, y ) ∥ .

The theorem is proven in Supplement A.4 by adapting the error bound from [38, Proposition 2] to hold for I (A ⊺ X ; B ⊺ Y ) , uniformly over (A , B) ∈ St( k, d x ) × St( k, d y ) . To that end, we show that for any µ XY ∈ P k ( b, M ) , the log-density of (A ⊺ X, B ⊺ Y ) ∼ ( p A , p B ) ♯ µ XY admits an extension (to an open set containing the support) with k +3 continuous and uniformly bounded derivatives.

Remark 6 (Parametric rate and optimality) . Taking ℓ ≍ n , the resulting rate in Theorem 1 is parametric, and hence minimax optimal. This result implicitly assumes that M is known when picking the neural net parameters. This assumption can be relaxed to mere existence of (an unknown) M , resulting in an extra polylog( ℓ ) factor multiplying the n -1 / 2 term.

Remark 7 (Comparison to average-SMI) . Neural estimation of classic mutual information under the framework of [38] requires the density to have Hölder smoothness s ≥ ⌊ ( d x + d y ) / 2 ⌋ +3 . For SI k ( X ; Y ) , smoothness of k +3 is sufficient (even though the ambient dimension is the same), which means it can be estimated over a larger distribution class. Similar gains in terms of smoothness levels were observed for aSMI in [26]. Nevertheless, we note that mSMI is more compatible with neural estimation than average-slicing [25, 26]. The mSMI neural estimator integrates the max-slicing into the neural network architecture and optimizes a single objective. The aSMI neural estimator from [26] requires an additional Monte Carlo integration step to approximate the integral over the Steifel manifolds. This results in an extra k 1 / 2 m -1 / 2 term in the error bound, where m is the number of Monte Carlo samples, introducing a burdensome computational overhead (see Section 5.1).

Remark 8 (Non-ReLU networks) . Theorem 1 employs the neural estimation bound from [38], which relies on [52] to control the approximation error. As noted in [38], their bound extends to any other sigmoidal bounded activation with lim z →-∞ σ ( z ) = 0 and lim z →∞ σ ( z ) = 1 by appealing to the approximation bound from [53] instead. Doing so would allow relaxing the smoothness requirement on the extension to r ∈ C k +2 b in (6) , but at the expense of scaling the hidden layer parameters as ℓ 1 / 2 log ℓ (as opposed to the ReLU-based bound, where the parameter scale is independent of ℓ ).

5 Experiments

5.1 Neural Estimation

We compare the performance of neural estimation methods for mSMI and aSMI on a synthetic dataset of correlated Gaussians. Let X,Z ∼ N (0 , 1) be i.i.d. and set Y = ρX + √ 1 -ρ 2 Z , for ρ ∈ (0 , 1) . The goal is to estimate the k -dimensional mSMI and aSMI between ( X,Y ) . Wetrain our mSMI neural estimator and the aSMI neural estimator from [26, Section 4.2] based on n i.i.d. samples, and compare their performance as a function of n . Both average and maxsliced algorithms converge at similar rates; however, aSMI has significantly higher time complexity due to the need to train multiple neural estimators (one for each projection direction). This is shown in Figure 1, where we compare the average epoch time for each algorithm against the dataset size. Implementation details are given in Supplement C.

5.2 Independence Testing

In this experiment, we compare mSMI and aSMI for independence testing. We follow the setting from [26, Section 5], generating d -dimensional samples correlated in a latent d ′ -dimensional subspace and estimating the information measure to determine dependence. We estimate the aSMI with the method from [26], using m = 1000 Monte Carlo samples and the Kozachenko-Leonenko estimator for the mutual information between the projected variables [54]. We then compute AUC-ROC over 100 trials,

10

1

10

2

10

3

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

1

k

= 1

n

10

1

10

2

10

3

k

= 2

n

10

1

10

2

10

3

k

= 3

n

10

1

10

2

10

3

k

= 3

n

10

1

10

2

10

3

k

= 4

n

10

1

10

2

10

3

k

= 5

n

10

3

10

4

2

.

1

2

.

2

2

.

3

2

.

4

2

.

5

·

10

-

2

SI

[nats]

10

3

10

4

0

.

15

0

.

2

0

.

25

0

.

3

0

.

35

SI

[nats]

10

3

10

4

10

-

1

10

0

10

1

10

2

Time [sec.]

SI

SI

n

considering various ambient and projected dimensions. For mSMI, as we cannot differentiate through the Kozachenko-Leonenko estimator, we resort to gradient-free methods. We employ the LIPO algorithm from [55] with a stopping criterion of 1000 samples. This choice is motivated by the Lipschitzness of (A , B) ↦→ I (A ⊺ X ; B ⊺ Y ) w.r.t. the Frobenius norm on St( k, d x ) × St( k, d y ) (cf. [26, Lemma 2]). Figure 2 shows that when k > 2 , mSMI captures independence better than aSMI, particularly in the lower sample regime. We hypothesize that this is due to the fact that the shared signal lies in a low-dimensional subspace, which mSMI can isolate and perhaps better exploit than aSMI, which averages over all subspaces. When k is much smaller than the shared signal dimension d ′ , mSMI fails to capture all the information and aSMI, which takes all slices into account, may be preferable. Results are averaged over 10 seeds. Further implementation details are in Supplement C.

5.3 Multi-View Representation Learning

We next explore mSMI as an informationtheoretic generalization of CCA by examining its utility in multi-view representation learninga popular CCA application. Without using class labels, we obtain mSMI-based k -dimensional representations of the top and bottom halves of MNIST images (considered as two separate views of the digit image). This is done by com-

k

Linear CCA

Linear mSMI

MLP DCCA

MLP mSMI

1

0.261

±

0.03

0.274

±

0.02

0.284

±

0.03

0.291

±

0.02

2

0.32

±

0.02

0.346

±

0.02

0.314

±

0.03

0.417

±

0.02

4

0.42

±

0.01

0.478

±

0.02

0.441

±

0.04

0.546

±

0.01

8

0.553

±

0.03

0.666

±

0.01

0.645

±

0.02

0.665

±

0.01

12

0.614

±

0.02

0.751

±

0.01

0.697

±

0.01

0.753

±

0.01

16

0.673

±

0.02

0.775

±

0.01

0.730

±

0.02

0.779

±

0.01

20

0.704

±

0.007

0.79

±

0.006

0.774

±

0.01

0.798

±

0.01

puting the k -dimensional mSMI between the views and using the maximizing projected variables as the representations. We compare to similarly obtained CCA-based representations, following the method of [20]. Both linear and nonlinear (parameterized by an MLP neural network) slicing models are optimized with similar initialization and data but different loss functions. Performance is evaluated via downstream 10-class classification accuracy, utilizing the learned top-half representations. Results are averaged over 10 seeds. As shown in Table 1, mSMI outperforms CCA for learning meaningful representations. Interestingly, linear representations learned by mSMI outperform nonlinear representations from the CCA methodology, demonstrating the potency of mSMI. Full implementation details and additional results are given in Supplements C and D, respectively.

The aSMI is not considered for this experiment since it does not provide a concrete latent space representation (as it is an averaged quantity). Moreover, if one were to maximize aSMI as an objective to derive such representations, this would simply lead back to computing mSMI; cf. Remark 3.

5.4 Learning Fair Representations

Another common application of dependence measures is learning fair representations of data. We seek a data transformation Z = f ( X ) that is useful for predicting some outcome or label Y , while being statis-

Table 2: Learning a fair representation of the US Census Demographic dataset, following the setup of [39]. Results are shown as the median over 10 runs with random data splits. The fairest result is k = 6 .

tically independent of some sensitive attribute A (e.g., gender, race, or religion of the subject). In other words, a fair representation is one that is not affected by the subjects' protected attributes so that downstream predictions are not biased against protected groups, even if the training data may have been biased. Following the setup of [39], we measure utility and fairness using the HGR maximal correlation ρ HGR ( · , · ) = sup h,g ρ ( h ( · ) , g ( · ) ) , seeking large ρ HGR ( Z, Y ) and small ρ HGR ( Z, A ) where h and g are parameterized by neural networks. As solving this minimax problem directly is difficult in practice, following [39] we learn Z by optimizing the bottleneck equation ρ HGR ( Z, Y ) -β SI k ( Z, A ) , where we use a neural estimator for the mSMI and β , k are hyperparameters.

Table 2 shows results on the US Census Demographic dataset extracted from the 2015 American Community Survey, which has 37 features collected over 74,000 census tracts. Here Y is the fraction of children below the poverty line in a tract, and A is the fraction of women in the tract. Following the same experimental setup as [39], the learned Z is 80-dimensional. As [39] showed that their 'Slice' approach significantly outperformed all other baselines on this experiments under a computational

(a) FashionMNIST

(b) MNIST,

k

= 5

(c) MNIST,

k

= 10

8

8

643

5

8

8

8

B

8

7

7

7

9

6

力

h

bb

9

h

4

0

222222:

2

constraint 5 , we apply the same computational constraint to our approach and compare only to Slice and to the 'N/A' fairness-agnostic model trained on the bottleneck objective with β = 0 . Note that for k > 1 , mSMI learns a more fair representation Z (lower ρ HGR ( Z, A ) ) than Slice, while retaining a utility ρ HGR ( Z, Y ) on par with the fairness agnostic N/A model. We emphasize that due to the reasons outlined in Section 5.3, aSMI is not suitable for the considered task and is thus not included in the comparison. Results on the Adult dataset are shown in Supplement E.

5.5 Max-Sliced InfoGAN

We present an application of max-slicing to generative modeling under the InfoGAN framework [56]. The InfoGAN learns a disentangled latent space by maximizing the mutual information between a latent code variable and the generated data. We revisit this architecture but replace the classical mutual information regularizer in the InfoGAN objective with mSMI. Our max-sliced InfoGAN is tested on the MNIST and Fashion-MNIST datasets. Figure 3 presents the generated samples for several projection dimensions. We consider 3 latent codes ( C 1 , C 2 , C 3 ) , which automatically learn to encode different features of the data. We vary the values of C 1 , which is a 10 -state discrete variable, along the column (and consider random values of ( C 2 , C 3 ) along the rows). Evidently, C 1 successfully disentangles the 10 class labels and the quality of generated samples is on par with past implementations [56, 26]. We stress that since mSMI relies on low-dimensional projections, the resulting InfoGAN mutual information estimator uses a reduced number of parameters (at the negligible cost of optimizing over linear projections). Additional details are given in Supplement C.

6 Conclusion

This paper proposed mSMI, an information theoretic generalization of CCA. mSMI captures the full dependence structure between two high dimensional random variables, while only requiring an optimized linear projection of the data. We showed that mSMI inherits important properties of Shannon's mutual information and that when the random variables are Gaussian, the mSMI optimal solutions coincide with classic k -dimensional CCA. Moving beyond Gaussian distributions, we present a neural estimator of mSMI and establish non-asymptotic error bounds.

Through several experiments we demonstrate the utility of mSMI for tasks spanning independence testing, multi-view representation learning, algorithmic fairness and generative modeling, showing it outperforms popular methodologies. Possible future directions include an investigation of an operational meaning of mSMI, either in information theoretic or physical terms, extension of the proposed formal guarantees to the nonlinear setting, and the extension of the neural estimation convergence guarantees to deeper networks. Additionally, mSMI can provide a mathematical foundation to mutual information-based representation learning, a popular area of self-supervised learning [10, 57].

In addition to the above, we plan to develop a rigorous theory for the choice of k , which is currently devised empirically and is treated as a hyperparameter. When the support of the distributions lies in some d ′ < d dimensional subspace, the choice of k = d ′ is sufficient to recover the classical mutual information, and therefore it characterizes the full dependence structure. Extrapolating from this point, we conjecture that the optimal value of k is related to the intrinsic dimension of the data distribution, even when it is not strictly supported on a low-dimensional subset.

References

Sidney Siegel. Nonparametric statistics. The American Statistician , 11(3):13-19, 1957.

Larry D Haugh. Checking the independence of two covariance-stationary time series: a univariate residual cross-correlation approach. Journal of the American Statistical Association , 71(354):378-385, 1976.

Thomas B Berrett and Richard J Samworth. Nonparametric independence testing via mutual information. Biometrika , 106(3):547-566, 2019.

Atul J Butte and Isaac S Kohane. Mutual information relevance networks: functional genomic clustering using pairwise entropy measurements. In Biocomputing 2000 , pages 418-429. World Scientific, 1999.

Alexander Kraskov, Harald Stögbauer, Ralph G Andrzejak, and Peter Grassberger. Hierarchical clustering using mutual information. Europhysics Letters , 70(2):278, 2005.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Weiran Wang, Raman Arora, Karen Livescu, and Jeff Bilmes. On deep multi-view representation learning. In International conference on machine learning , pages 1083-1092. PMLR, 2015.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Selfsupervised learning via redundancy reduction. In International Conference on Machine Learning , pages 12310-12320. PMLR, 2021.

Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, et al. A cookbook of self-supervised learning. arXiv preprint arXiv:2304.12210 , 2023.

Karl Pearson. Vii. note on regression and inheritance in the case of two parents. proceedings of the royal society of London , 58(347-352):240-242, 1895.

Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal , 27(3):379-423, 1948.

Liam Paninski. Estimation of entropy and mutual information. Neural Computation , 15:11911253, June 2003.

Harold Hotelling. Analysis of a complex of statistical variables into principal components. Journal of educational psychology , 24(6):417, 1933.

Hermann O Hirschfeld. A connection between correlation and contingency. Mathematical Proceedings of the Cambridge Philosophical Society , 31(4):520-524, 1935.

Hans Gebelein. Das statistische problem der korrelation als variations-und eigenwertproblem und sein zusammenhang mit der ausgleichsrechnung. ZAMM-Journal of Applied Mathematics and Mechanics/Zeitschrift für Angewandte Mathematik und Mechanik , 21(6):364-379, 1941.

Alfréd Rényi. On measures of dependence. Acta Mathematica Academiae Scientiarum Hungarica , 10(3-4):441-451, 1959.

Shotaro Akaho. A kernel method for canonical correlation analysis. arXiv preprint cs/0609071 , 2006.

David R Hardoon, Sandor Szedmak, and John Shawe-Taylor. Canonical correlation analysis: An overview with application to learning methods. Neural computation , 16(12):2639-2664, 2004.

Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation analysis. In International conference on machine learning , pages 1247-1255. PMLR, 2013.

Francis R Bach and Michael I Jordan. A probabilistic interpretation of canonical correlation analysis. Technical Report , 2005.

Tae-Kyun Kim, Josef Kittler, and Roberto Cipolla. Learning discriminative canonical correlations for object recognition with image sets. In Computer Vision-ECCV 2006: 9th European Conference on Computer Vision, Graz, Austria, May 7-13, 2006, Proceedings, Part III 9 , pages 251-262. Springer, 2006.

Elena Parkhomenko, David Tritchler, and Joseph Beyene. Sparse canonical correlation analysis with application to genomic data integration. Statistical applications in genetics and molecular biology , 8(1), 2009.

Amichai Painsky, Meir Feder, and Naftali Tishby. Nonlinear canonical correlation analysis: A compressed representation approach. Entropy , 22(2):208, 2020.

Ziv Goldfeld and Kristjan Greenewald. Sliced mutual information: A scalable measure of statistical dependence. Advances in Neural Information Processing Systems , 34:17567-17578, 2021.

Ziv Goldfeld, Kristjan Greenewald, Theshani Nuradha, and Galen Reeves. k-sliced mutual information: A quantitative study of scalability with dimension. Advances in Neural Information Processing Systems , 35:15982-15995, 2022.

Kantilal Varichand Mardia, John T Kent, and John M Bibby. Multivariate analysis. Probability and mathematical statistics , 1979.

Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain Markov process expectations for large time. iv. Communications on Pure and Applied Mathematics , 36(2):183-212, 1983.

Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International Conference on Machine Learning , pages 531-540. PMLR, 2018.

Ben Poole, Sherjil Ozair, Aäron van den Oord, Alexander A Alemi, and George Tucker. On variational lower bounds of mutual information. In NeurIPS Workshop on Bayesian Deep Learning , 2018.

Chung Chan, Ali Al-Bashabsheh, Hing Pang Huang, Michael Lim, Da Sun Handason Tam, and Chao Zhao. Neural entropic estimation: A faster path to mutual information estimation. arXiv preprint arXiv:1905.12957 , 2019.

Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estimators. Proceedings of the International Conference on Learning Representations (ICLR) , 2020.

Jingjing Zhang, Osvaldo Simeone, Zoran Cvetkovic, Eugenio Abela, and Mark Richardson. Itene: Intrinsic transfer entropy neural estimator. arXiv preprint arXiv:1912.07277 , 2019.

Sudipto Mukherjee, Himanshu Asnani, and Sreeram Kannan. Ccmi: Classifier based conditional mutual information estimation. In Uncertainty in artificial intelligence , pages 1083-1093. PMLR, 2020.

Dor Tsur, Ziv Aharoni, Ziv Goldfeld, and Haim Permuter. Neural estimation and optimization of directed information over continuous spaces. IEEE Transactions on Information Theory , 2023.

Qing Guo, Junya Chen, Dong Wang, Yuewei Yang, Xinwei Deng, Jing Huang, Larry Carin, Fan Li, and Chenyang Tao. Tight mutual information estimation with contrastive fenchel-legendre optimization. Advances in Neural Information Processing Systems , 35:28319-28334, 2022.

Sreejith Sreekumar, Zhengxin Zhang, and Ziv Goldfeld. Non-asymptotic performance guarantees for neural estimation of f-divergences. In International Conference on Artificial Intelligence and Statistics , pages 3322-3330. PMLR, 2021.

Sreejith Sreekumar and Ziv Goldfeld. Neural estimation of statistical divergences. Journal of Machine Learning Research , 23(126):1-75, 2022.

Yanzhi Chen, Weihao Sun, Yingzhen Li, and Adrian Weller. Scalable infomin learning. Advances in Neural Information Processing Systems , 35:2226-2239, 2022.

Thomas M Cover and A Joy Thomas. Elements of Information Theory . Wiley, New-York, 2nd edition, 2006.

Julien Rabin, Gabriel Peyré, Julie Delon, and Marc Bernot. Wasserstein barycenter and its application to texture mixing. In Scale Space and Variational Methods in Computer Vision: Third International Conference, SSVM 2011, Ein-Gedi, Israel, May 29-June 2, 2011, Revised Selected Papers 3 , pages 435-446. Springer, 2012.

Ishan Deshpande, Yuan-Ting Hu, Ruoyu Sun, Ayis Pyrros, Nasir Siddiqui, Sanmi Koyejo, Zhizhen Zhao, David Forsyth, and Alexander G Schwing. Max-sliced wasserstein distance and its use for gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10648-10656, 2019.

Soheil Kolouri, Kimia Nadjahi, Umut Simsekli, Roland Badeau, and Gustavo Rohde. Generalized sliced wasserstein distances. Advances in neural information processing systems , 32, 2019.

Tianyi Lin, Chenyou Fan, Nhat Ho, Marco Cuturi, and Michael Jordan. Projection robust wasserstein distance and riemannian optimization. Advances in neural information processing systems , 33:9383-9397, 2020.

Minhui Huang, Shiqian Ma, and Lifeng Lai. A riemannian block coordinate descent method for computing the projection robust wasserstein distance. In International Conference on Machine Learning , pages 4446-4455. PMLR, 2021.

Tianyi Lin, Zeyu Zheng, Elynn Chen, Marco Cuturi, and Michael I Jordan. On projection robust optimal transport: Sample complexity and model misspecification. In International Conference on Artificial Intelligence and Statistics , pages 262-270. PMLR, 2021.

Sloan Nietert, Ziv Goldfeld, Ritwik Sadhu, and Kengo Kato. Statistical, robustness, and computational guarantees for sliced wasserstein distances. Advances in Neural Information Processing Systems , 35:28179-28193, 2022.

C Radhakrishna Rao. Separation theorems for singular values of matrices and their applications in multivariate analysis. Journal of Multivariate Analysis , 9(3):362-377, 1979.

Karl Pearson. Liii. on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin philosophical magazine and journal of science , 2(11):559-572, 1901.

Wojciech Marian Czarnecki, Rafal Jozefowicz, and Jacek Tabor. Maximum entropy linear manifold for learning discriminative low-dimensional representation. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2015, Porto, Portugal, September 7-11, 2015, Proceedings, Part I 15 , pages 52-67. Springer, 2015.

Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. Proceedings of the International Conference on Learning Representations (ICLR) , 2020.

Jason M Klusowski and Andrew R Barron. Approximation by combinations of relu and squared relu ridge functions with ℓ 1 and ℓ 0 controls. IEEE Transactions on Information Theory , 64(12):7649-7656, 2018.

Andrew R Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information theory , 39(3):930-945, 1993.

Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical review E , 69(6):066138, 2004.

Cédric Malherbe and Nicolas Vayatis. Global optimization of lipschitz functions. In International Conference on Machine Learning , pages 2314-2323. PMLR, 2017.

Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems , 29, 2016.

Ravid Shwartz-Ziv and Yann LeCun. To compress or not to compress-self-supervised learning and information theory: A review. arXiv preprint arXiv:2304.09355 , 2023.

Yury Polyanskiy and Yihong Wu. Information Theory: From Coding to Learning . Cambridge University Press, 2022.

Jan R Magnus and Heinz Neudecker. Matrix differential calculus with applications in statistics and econometrics . John Wiley & Sons, 2019.

Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289 , 2015.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Aaron Defazio, Francis Bach, and Simon Lacoste-Julien. Saga: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in neural information processing systems , 27, 2014.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose max-sliced mutual information (mSMI), which inherits important properties of mutual information. They show that the projection in mSMI is reduced to that in CCA for jointly Gaussian. They also provide a method to estimate mSMI using a neural network, which is computationally more efficient than an existing method for estimating average-SMI. They derive an error bound for the estimation of mSMI.\n\nSTRENGTHS:\nThe paper includes fundamental properties of mSMI, connections with existing studies, and a practical method to estimate mSMI. The results provided in this paper are sufficient to understand mSMI.\n\nWEAKNESSES:\nAlthough the paper is well-written for the most part, some parts should be explained clearly, or need more detailed explanations. For example, what information can be extracted from mSMI if the data is not Gaussian (in Remark 4), and how we can extend the ideas in 4.2 to mode general neural networks. See the sections on questions and limitations for more details.\n\nQUESTIONS:\nIn Remark 4, the authors insist that if the data is not Gaussian, the projections in mSMI do not coincide with those in CCA. Do you have any simple examples of how the projections in mSMI and those in CCA are different?\n\nMinor comments\n- The definition of I is missing in Eq. (1). The definition is in line 110, but it should be defined here.\n- In Definition 2, G and H should be subsets of $\\{g:\\mathbb{R}^{d_x}\\to\\mathbb{R}^k\\}$ and $\\{h:\\mathbb{R}^{d_y}\\to\\mathbb{R}^k\\}$ ?\n\nLIMITATIONS:\nTheorem 1 is for the shallow network with the ReLU activation function. Although the authors insist that the ideas can be extended to other nonlinearities and deep architectures, how we can extend the theoretical results like Theorem 1 to the deep case and other activation functions is not clearly explained. Are there any difficulties? If there are, do you have any ideas to overcome the difficulties?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Max-Sliced Mutual Information (mSMI) which equals the maximal mutual information between low-dimensional projections of the high-dimensional variables. The mSMI can capture intricate dependencies in the data while being amenable to fast computation and scalable estimation from samples. In addition, the paper proposes multivariate conditional mSMI, and max sliced entropy which are extensions of mSMI. The paper discusses the structural properties of mSMI including bounding on different values of sub-space dimension and the original mutual information, Identification of independence, KL divergence representation, Sub-chain rule, and Tensorization. Moreover, the paper discusses how Gaussian mSMI is related to canonical correlation analysis (CCA) and how Max-sliced entropy is related to  PCA. In addition, generalized mSMI with two general classes of functions is also discussed. The neural estimation of mSMI and its error are presented in Section 4. Finally,  the paper presents experiments that demonstrate the utility of mSMI for several synthetic and real-world tasks, encompassing independence testing, multi-view representation learning, and algorithmic fairness.\n\nSTRENGTHS:\n* The paper is detailed and very well-written.\n* Max-sliced mutual information is a natural extension to SMI.\n* Connection between mSMI and conventional methods such as CCA and PCA.\n* Neural estimation of sMI has a better error than SMI since no Monte Carlo estimation is needed.\n* mSMI can be implemented efficiently with Neural Estimation by merging the linear projection to the neural network.\n* Experiments on independence testing show that mSMI is both better and faster than SMI.\n\nWEAKNESSES:\n* mSMI is quite incremental based on the existence of SMI, k-SMI, Max-sliced Wasserstein distance, and Max-K sliced Wasserstein distance.\n* There is no comparison between mSMI and SMI (K-SMI)  in Multi-View Representation and Learning Fair Representation. This is quite questionable since mSMI is an extension of SMI (K-MSI). I believe a comparison in both computation and performance is needed.\n\nQUESTIONS:\nCan the paper include an application in the previous SMI papers e.g., InfoGAN?\n\nLIMITATIONS:\nNot applicable", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-24"}

comments: SUMMARY:
The paper introduces an adaptation of sliced mutual information that focuses on the maximal mutual information between linear projections of low dimensionality of random variables. This measure (mSMI) has desireable properties and is approachable by neural estimation. The authors show that mSMI can be approximated for a certain class of continuous random variables more efficiently than (sliced) mutual information. In some illuminating experiments potential use cases are illustrated and suggest good practical performance also in such use cases.

STRENGTHS:
The paper is well written and presents a new measure of dependence from all essential perspectives: Useful theoretical properties, approximation is theoretically possible for a large class of random variables, and practical implementation is possible.
The paper is not overloaded in content and focuses on the topic at hand providing all necessary information without giving unnecessary details or too many experiments.

WEAKNESSES:
I don't see any constructive and actionable insights on how the work could improve towards its stated goals.

QUESTIONS:
How can the results for fairness aware methods in Table 2 be better than fairness agnostic?

LIMITATIONS:
The authors are honest in the limitations:
The experiments show that for some cases sliced mutual information might work better than mSMI. Further research on the best choice of the dimension k to use might be interesting but are clearly beyond the scope of this work and would only clutter the presentation.
A proof for the non-linear method is not yet availabvle but most likely also something for future work and a separate manuscript.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper defines a novel measure of independence called Max-Sliced Mutual Information (mSMI) which provides a non-linear generalization of Canonical Correlation Analysis. This measure can also be viewed as a variant of Mutual Information with a better tractability in high dimensions.\nIn fact, mSMI is closely related to average-sliced mutual information (aSMI) introduced in [26,27]. Whereas aSMI is computed by averaging an objective involving mutual information over a product of Stiefel manifold $St(k,d_x) \\times St(k,d_y)$, mSMI is actually obtained by maximizing this objective over the same domain.\nThis work also introduces a generalized mSMI where the optimization is performed (loosely speaking) on a class of non-linear functions. \n\nBy leveraging the Donsker-Varadhan variational form for the mutual information, the authors show that mSMI can be estimated thanks to a neural estimator for which Theorem 1 provides convergence rates. \n\nFrom the numerical perspective, three applications are studied: independence testing, multi-view representation learning and fair representation learning.\nFor independence testing, mSMI seems to better perform than aSMI when $k$ is not too small.\nThe time complexity of aSMI is also reported to be larger than the time complexity of mSMI.\nIn the comparisons with a few baselines in the case of multi-view representation learning or fair representation learning, mSMI is often the best method in terms of performance.\n\nSTRENGTHS:\nThis paper is written clearly and its results are significant.\nThe theoretical contributions are interesting and the proofs given in supplementary material are easy to read.\nThe construction of mSMI nicely arises as a variant of aSMI [26,27] and the mSMI neural estimator has a clear interpretation.\nI find the numerical comparisons in section 5 convincing.\n\nWEAKNESSES:\nI do not see serious methodological weaknesses. Here are some minor remarks.\n- The rule for emphasizing entries with bold in Table 2 of section 5.4 is unclear to me. In particular, $\\rho_{HGR}$ equals $0.958$ for $k=2$ and is not displayed in bold whereas the entry corresponding to $k=6$ is emphasized although it takes a smaller value ($0.957$).\n- In the same section, it is unclear how $k$ has to be selected by the user since mSMI does not outperform SLICE for all the values of $k$.\n\nQUESTIONS:\n- Here is a question about Theorem 1. Why is $\\|\\mathcal{X}\\times\\mathcal{Y}\\|$ defined at the end of this statement? This quantity does not appear explicitly in the theorem. This is a bit confusing.\n- In Table 2, the values of $k$ range from $1$ to $7$ whereas they range from $3$ to $7$ in Table 5 of the supplementary material (Adult data set). What happens for small $k$ in the case of the Adult data set ?\n\nLIMITATIONS:\nI do not foresee a negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}