SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dor Tsur
n-Gurion University
Ziv Goldfeld Cornell University
Kristjan Greenewald  
MIT-IBM Watson AI Lab
Quantifying dependence between high-dimensional random variables is central to statistical learning and inference. Two classical methods are canonical correlation analysis (CCA), which identifies maximally correlated projected versions of the original variables, and Shannon's mutual information, which is a universal dependence measure that also captures high-order dependencies. However, CCA only accounts for linear dependence, which may be insufficient for certain applications, while mutual information is often infeasible to compute/estimate in high dimensions. This work proposes a middle ground in the form of a scalable information-theoretic generalization of CCA, termed max-sliced mutual information (mSMI). mSMI equals the maximal mutual information between low-dimensional projections of the high-dimensional variables, which reduces back to CCA in the Gaussian case. It enjoys the best of both worlds: capturing intricate dependencies in the data while being amenable to fast computation and scalable estimation from samples. We show that mSMI retains favorable structural properties of Shannon's mutual information, like variational forms and identification of independence. We then study statistical estimation of mSMI, propose an efficiently computable neural estimator, and couple it with formal non-asymptotic error bounds. We present experiments that demonstrate the utility of mSMI for several tasks, encompassing independence testing, multi-view representation learning, algorithmic fairness, and generative modeling. We observe that mSMI consistently outperforms competing methods with little-to-no computational overhead.
Dependence measures between random variables are fundamental in statistics and machine learning for tasks spanning independence testing [1-3], clustering [4, 5], representation learning [6, 7], and self-supervised learning [8-10]. There is a myriad of measures quantifying different notions of dependence, with varying statistical and computational complexities. The simplest is the Pearson correlation coefficient [11], which only captures linear dependencies. At the other extreme is Shannon's mutual information [12], which is a universal dependence measure that is able to identify arbitrarily intricate dependencies. Despite its universality and favorable properties, accurately estimating mutual information from data is infeasible in high-dimensional settings. First, mutual information estimation rates suffers from the curse of dimensionality, whereby convergence rates deteriorate exponentially with dimension [13]. Additionally, computing mutual information requires integrating a log-likelihood ratio over a high-dimensional space, which is generally intractable.
Between these two extremes is the popular canonical correlation analysis (CCA) [14], which identifies maximally correlated linear projections of variables. Nevertheless, classical CCA still only captures linear dependence, which has inspired nonlinear extensions such as Hirschfeld-Gebelein-Renyi (HGR) maximum correlation [15-17], kernel CCA [18, 19], deep CCA [20, 7], and various other generalizations [21-24]. However, HGR is computationally infeasible, while kernel and deep CCA can be burdensome in high dimensions, as they require optimization over reproducing kernel Hilbert spaces or deep neural networks, respectively. To overcome these shortcomings, this work proposes
max-sliced mutual information (mSMI)—a scalable information-theoretic extension of CCA that captures the full dependence structure while only requiring optimization over linear projections.
The mSMI is defined as the maximal mutual information between linear projections of the variables. Namely, the  $k$ -dimensional mSMI between  $X$  and  $Y$  with values in  $\mathbb{R}^{d_x}$  and  $\mathbb{R}^{d_y}$ , respectively, is<sup>1</sup>
$$
\overline {{\mathsf {S l}}} _ {k} (X; Y) := \sup  _ {(A, B) \in \operatorname {S t} (k, d _ {x}) \times \operatorname {S t} (k, d _ {y})} \mathsf {I} (A ^ {\intercal} X; B ^ {\intercal} Y),
$$
where  $\mathrm{St}(k,d)$  is the Stiefel manifold of  $d\times k$  matrices with orthonormal columns. Unlike the nonlinear CCA variants that use nonlinear feature extractors in the high-dimensional ambient spaces, mSMI retains the linear projections of CCA and captures nonlinear structures in the low-dimensional feature space. This is done by using the mutual information between the projected variables, rather than correlation, as the optimization objective. Beyond being considerably simpler from a computational standpoint, this crucial difference allows mSMI to identify the full dependence structure, akin to classical mutual information. mSMI can also be viewed as the maximized version of the average-sliced mutual information (aSMI) [25, 26], which averages  $\mathsf{l}(\mathsf{A}^{\top}\mathsf{X};\mathsf{B}^{\top}\mathsf{Y})$  with respect to (w.r.t.) the Haar measure over  $\mathrm{St}(k,d_x)\times \mathrm{St}(k,d_y)$ . However, we demonstrate that compared to aSMI, mSMI benefits from improved neural estimation error bounds and a clearer interpretation.
We show that mSMI inherits important properties of mutual information, including identification of independence, tensorization, and variational forms. For jointly Gaussian  $(X,Y)$ , the optimal mSMI projections coincide with those of  $k$ -dimensional CCA [27], posing mSMI as a natural information-theoretic generalization. Beyond the Gaussian case, the solutions differ and mSMI may yield more effective representations for downstream tasks due to the intricate dependencies captured by mutual information. We demonstrate this superiority empirically for multi-view representation learning.
For efficient computation, we propose an mSMI neural estimator based on the Donsker-Varadhan (DV) variational form [28]. Neural estimators have seen a surge in interest due to their scalability and compatibility with gradient-based optimization [29-36]. Our estimator employs a single model that composes the projections with the neural network approximation of the DV critic, and then jointly optimizes them. This results in both the estimated mSMI value and the optimal projection matrices. Building on recent analysis of neural estimation of  $f$ -divergences [37, 38], we establish non-asymptotic error bounds that scale as  $O\big(k^{1/2}(\ell^{-1/2} + kn^{-1/2})\big)$ , where  $\ell$  and  $n$  are the numbers of neurons and  $(X,Y)$  samples, respectively. Equating  $\ell$  and  $n$  results in the (minimax optimal) parametric estimation rate, which highlights the scalability of mSMI and its compatibility to modern learning settings.
In our empirical investigation, we first demonstrate that our mSMI neural estimator converges orders of magnitude faster than that of aSMI [26]. This is because the latter requires (parallel) training of many neural estimators corresponding to different projection directions, while the mSMI estimator optimizes a single combined model. Notwithstanding the reduction in computational overhead, we show that mSMI outperforms average-slicing for independence testing. Next, we compare mSMI with deep CCA [20, 7] by examining downstream classification accuracy based on representations obtained from both methods in a multi-view learning setting. Remarkably, we observe that even the linear mSMI projections outperform nonlinear representations obtained from deep CCA. We also consider an application to algorithmic fairness under the infomin framework [39]. Replacing their generalized Pearson correlation objective with mSMI, we again observe superior performance in the form of more fair representations whose utility remains on par with the fairness-agnostic model. Lastly, we devise a max-sliced version of the InfoGAN by replacing the classic mutual information regularizer with its max-sliced analog. We show that despite the low-dimensional projections, the max-sliced InfoGAN successfully learns to disentangle the latent space and generates quality samples.
Notation. For  $a, b \in \mathbb{R}$ , we use the notation  $a \wedge b = \min\{a, b\}$  and  $a \vee b = \max\{a, b\}$ . For  $d \geq 1$ ,  $\|\cdot\|$  is the Euclidean norm in  $\mathbb{R}^d$ . The Stiefel manifold of  $d \times k$  matrices with orthonormal columns
The parameter  $k$  is fixed and small compared to the ambient dimensions  $d_x, d_y$ , often simply set as  $k = 1$ .
is denoted by  $\mathrm{St}(k,d)$ . For a  $d\times k$  matrix A, we use  $\mathfrak{p}^{\mathrm{A}}:\mathbb{R}^{d}\to \mathbb{R}^{k}$  for the orthogonal projection onto the row space of A. For  $\mathrm{A}\in \mathbb{R}^{d\times k}$  with  $\mathrm{rank(A)} = r\leq k\wedge d$ , we write  $\sigma_1(\mathrm{A}),\ldots ,\sigma_r(\mathrm{A})$  for its non-zero singular values, and assume without loss of generality (w.l.o.g.) that they are arranged in descending order. Similarly, the eigenvalues of a square matrix  $\Sigma \in \mathbb{R}^{d\times d}$  are denoted by  $\lambda_{1}(\Sigma),\dots,\lambda_{d}(\Sigma)$ . Let  $\mathcal{P}(\mathbb{R}^d)$  denote the space of Borel probability measures on  $\mathbb{R}^d$ . For  $\mu ,\nu \in \mathcal{P}(\mathbb{R}^d)$ , we use  $\mu \otimes \nu$  to denote a product measure, while  $\operatorname{spt}(\mu)$  designates the support of  $\mu$ . All random variables throughout are assumed to be continuous w.r.t. the Lebesgue measure. For a measurable map  $f$ , the pushforward of  $\mu$  under  $f$  is denoted by  $f_{\sharp}\mu = \mu \circ f^{-1}$ , i.e., if  $X\sim \mu$  then  $f(X)\sim f_{\sharp}\mu$ . For a jointly distributed pair  $(X,Y)\sim \mu_{XY}\in \mathcal{P}(\mathbb{R}^{d_x}\times \mathbb{R}^{d_y})$ , we write  $\Sigma_X$  and  $\Sigma_{XY}$  for covariance matrix of  $X$  and cross-covariance matrix of  $(X,Y)$ , respectively.
Canonical correlation analysis. CCA is a classical method for devising maximally correlated linear projections of a pair of random variables  $(X,Y)\sim \mu_{XY}\in \mathcal{P}(\mathbb{R}^{d_x}\times \mathbb{R}^{d_y})$  via [14]
$$
\left(\theta_ {\mathrm {C C A}}, \phi_ {\mathrm {C C A}}\right) = \underset {(\phi , \theta) \in \mathbb {R} ^ {d _ {x}} \times \mathbb {R} ^ {d _ {y}}} {\operatorname {a r g m a x}} \frac {\theta^ {\mathrm {T}} \Sigma_ {X Y} \phi^ {\mathrm {T}}}{\sqrt {\theta^ {\mathrm {T}} \Sigma_ {X X} \theta \phi^ {\mathrm {T}} \Sigma_ {Y Y} \phi}} = \underset { \begin{array}{c} (\theta , \phi) \in \mathbb {R} ^ {d _ {x}} \times \mathbb {R} ^ {d _ {y}}: \\ \theta^ {\mathrm {T}} \Sigma_ {X} \theta = \phi^ {\mathrm {T}} \Sigma_ {Y} \phi = 1 \end{array} } {\operatorname {a r g m a x}} \theta^ {\mathrm {T}} \Sigma_ {X Y} \phi , \tag {1}
$$
where the former objective is the correlation coefficient  $\rho (\theta^{\mathsf{T}}X,\phi^{\mathsf{T}}Y)$  between the projected variables and the equality follows from invariance of  $\rho$  to scaling. The global optimum has an analytic form as  $(\theta_{\mathrm{CCA}},\phi_{\mathrm{CCA}}) = (\Sigma_X^{-1 / 2}\theta_1,\Sigma_Y^{-1 / 2}\phi_1)$ , where  $(\theta_{1},\phi_{1})$  is the (unit-length) top left- and right-singular vector pair associated with the largest singular value of  $\mathrm{T}_{XY}:= \Sigma_X^{-1 / 2}\Sigma_{XY}\Sigma_Y^{-1 / 2}\in \mathbb{R}^{d_x\times d_y}$ . This solution is efficiently computable in  $O((d_x\vee d_y)^3)$  time, given that the population correlation matrices are known. CCA extends to  $k$ -dimensional projections via the optimization [27]
$$
\max  _ {\substack {(\mathrm {A}, \mathrm {B}) \in \mathbb {R} ^ {d _ {x} \times k} \times \mathbb {R} ^ {d _ {y} \times k}: \\ \mathrm {A} ^ {\intercal} \Sigma_ {X} \mathrm {A} = \mathrm {B} ^ {\intercal} \Sigma_ {Y} \mathrm {B} = \mathrm {I} _ {k}}} \operatorname {t r} (\mathrm {A} ^ {\intercal} \Sigma_ {X Y} \mathrm {B}), \tag{2}
$$
with the optimal CCA matrices being  $(\mathrm{A}_{\mathsf{CCA}},\mathrm{B}_{\mathsf{CCA}}) = (\Sigma_X^{-1 / 2}\mathrm{U}_k,\Sigma_Y^{-1 / 2}\mathrm{V}_k)$ , where  $\mathrm{U}_k$  and  $\mathrm{V}_k$  are the matrices of the first  $k$  left- and right-singular vectors of  $\mathrm{T}_{XY}$ . The optimal objective value then becomes the sum of the top  $k$  singular values of  $\mathrm{T}_{XY}$  (namely, its Ky Fan  $k$ -norm).
Divergences and information measures. Let  $\mu, \nu \in \mathcal{P}(\mathbb{R}^d)$  satisfy  $\mu \ll \nu$ , i.e.,  $\mu$  is absolutely continuous w.r.t.  $\nu$ . The Kullback-Leibler (KL) divergence is defined as  $\mathrm{D}(\mu \| \nu) \coloneqq \int_{\mathbb{R}^d} \log(d\mu / d\nu) d\mu$ . We have  $\mathrm{D}(\mu \| \nu) \geq 0$ , with equality if and only if (iff)  $\mu = \nu$ . Mutual information and differential entropy are defined from the KL divergence as follows. Let  $(X, Y) \sim \mu_{XY} \in \mathcal{P}(\mathbb{R}^{d_x} \times \mathbb{R}^{d_y})$  and denote the corresponding marginal distributions by  $\mu_X$  and  $\mu_Y$ . The mutual information between  $X$  and  $Y$  is given by  $\mathsf{l}(X; Y) \coloneqq \mathsf{D}(\mu_{XY} \| \mu_X \otimes \mu_Y)$  and serves as a measure of dependence between those random variables. The differential entropy of  $X$  is defined as  $\mathsf{h}(X) = \mathsf{h}(\mu_X) \coloneqq -\mathsf{D}(\mu_X \| \mathsf{Leb})$ . Mutual information between (jointly) continuous variables and differential entropy are related via  $\mathsf{l}(X; Y) = \mathsf{h}(X) + \mathsf{h}(Y) - \mathsf{h}(X, Y)$ ; decompositions in terms of conditional entropies are also available [40].
3 Max-Sliced Mutual Information
We now define the  $k$ -dimensional mSMI, establish structural properties thereof, and explore the Gaussian setting and its connections to CCA. We focus here on the case of (linear)  $k$ -dimensional projections and discuss extensions to nonlinear slicing in Section 3.3.
Definition 1 (Max-sliced mutual information). For  $1 \leq k \leq d_x \wedge d_y$ , the  $k$ -dimensional mSMI between  $(X, Y) \sim \mu_{XY} \in \mathcal{P}(\mathbb{R}^{d_x} \times \mathbb{R}^{d_y})$  is
$$
\overline {{\mathrm {S l}}} _ {k} (X; Y) := \sup  _ {(A, B) \in \operatorname {S t} (k, d _ {x}) \times \operatorname {S t} (k, d _ {y})} \mathbb {I} \left(A ^ {\intercal} X; B ^ {\intercal} Y\right), \tag {3}
$$
where  $\operatorname{St}(k, d)$  is the Stiefel manifold of  $d \times k$  matrices with orthonormal columns.
The mSMI measures Shannon's mutual information between the most informative  $k$ -dimensional projections of  $X$  and  $Y$ . It can be viewed as a maximized version of the aSMI  $\underline{\mathsf{S}}\underline{\mathsf{l}}_k(X;Y)$  from
[25, 26], defined as the integral of  $\mathsf{l}(\mathsf{A}^{\intercal}\mathsf{X};\mathsf{B}^{\intercal}\mathsf{Y})$  w.r.t. the Haar measure over  $\mathrm{St}(k,d_x)\times \mathrm{St}(k,d_y)$ . For  $d = d_{x} = d_{y}$ , we have  $\underline{\mathsf{S}}\underline{\mathsf{l}}_d(X;Y) = \overline{\mathsf{S}}\underline{\mathsf{l}}_d(X;Y) = \mathsf{l}(X;Y)$  due to invariance of mutual information to bijections. The supremum in mSMI is achieved since the Stiefel manifold is compact and the function (A,B)  $\mapsto \mathsf{l}(\mathsf{A}^{\intercal}\mathsf{X};\mathsf{B}^{\intercal}\mathsf{Y})$  is Lipschitz and thus continuous (Lemma 2 of [26]).
Remark 1 (Multivariate and conditional mSMI). The mSMI definition above extends to the multivariate and conditional cases as follows. Let  $(X,Y,Z) \sim \mu_{XYZ} \in \mathcal{P}(\mathbb{R}^{d_x} \times \mathbb{R}^{d_y} \times \mathbb{R}^{d_z})$ . The  $k$ -dimensional multivariate and conditional mSMI functionals are, respectively,  $\overline{\mathrm{SI}}_k(X,Y;Z) := \max_{\mathrm{A,B,C}} \mathsf{I}(\mathrm{A}^\top X,\mathrm{B}^\top Y;\mathrm{C}^\top Z)$  and  $\overline{\mathrm{SI}}_k(X;Y|Z) := \max_{\mathrm{A,B,C}} \mathsf{I}(\mathrm{A}^\top X;\mathrm{B}^\top Y|\mathrm{C}^\top Z)$ . Connections between  $\overline{\mathrm{SI}}_k(X;Y)$  and its multivariate and conditional versions are given in the proposition to follow. We also note that one may generalize the definition of  $\overline{\mathrm{SI}}_k(X;Y)$  to allow for projections into feature spaces of different dimensions, i.e.,  $\mathrm{A} \in \mathrm{St}(k_x,d_x)$  and  $\mathrm{B} \in \mathrm{St}(k_y,d_y)$ , for  $k_x \neq k_y$ . We expect our theory to extend to that case, but leave further exploration for future work.
In the spirit of mSMI, we define the max-sliced differential entropy.
Definition 2 (Max-sliced entropy). The  $k$ -dimensional max-sliced (differential) entropy of  $X \sim \mu_{X} \in \mathcal{P}(\mathbb{R}^{d})$  is  $\overline{\mathsf{sh}}_k(X) \coloneqq \overline{\mathsf{sh}}_k(\mu) \coloneqq \sup_{A \in \mathrm{St}(k,d)} h(A^{\intercal}X)$ .
An important property of classical differential entropy is the maximum entropy principle [40], which finds the highest entropy distribution within given class. In Appendix B, we study the max-sliced entropy maximizing distribution in several common scenarios. For instance, we show that  $\overline{\mathsf{sh}}_k$  is maximized by the Gaussian distribution under a fixed (mean and) covariance constraint. Namely, letting  $\mathcal{P}_1(m,\Sigma)\coloneqq \big\{\mu \in \mathcal{P}(\mathbb{R}^d):\operatorname {spt}(\mu) = \mathbb{R}^d,\mathbb{E}_{\mu}[X] = m,\mathbb{E}_{\mu}\big[(X - m)(X - m)^{\intercal}\big] = \Sigma \big\}$ , we have  $\mathrm{argmax}_{\mu \in \mathcal{P}_1(\mu ,\Sigma)}\overline{\mathsf{sh}}_k(\mu) = \mathcal{N}(m,\Sigma)$ . An intimate connection between max-sliced entropy and PCA is established in the sequel, under the Gaussian setting.
Remark 2 (Sliced divergences). The slicing technique has originated as a means to address scalability issues concerning statistical divergences. Significant attention was devoted to sliced Wasserstein distances as discrepancy measures between probability distributions [41-47]. As such, the sliced Wasserstein distance differs from mutual information and its sliced variants, which quantify dependence between random variables, rather than discrepancy per se. Additionally, as Wasserstein distances are rooted in optimal transport theory, they heavily depend on the geometry of the underlying data space. Mutual information, on the other hand, is induced by the KL divergence, which only depends on the log-likelihood of the considered distributions and overlooks geometry.
3.1 Structural Properties
The following proposition lists useful properties of the mSMI, which are similar to those of the average-sliced variant (cf. [26, Proposition 1]) as well as Shannon's mutual information itself.
Proposition 1 (Structural properties). The following properties hold:
1. Bounds: For any integers  $k_{1} < k_{2}$ :  $\underline{\mathrm{S}}\underline{\mathrm{l}}_{k_1}(X;Y) \leq \overline{\mathrm{S}}\overline{\mathrm{l}}_{k_1}(X;Y) \leq \overline{\mathrm{S}}\overline{\mathrm{l}}_{k_2}(X;Y) \leq \mathsf{l}(X;Y)$ .
2. Identification of independence:  $\overline{\mathrm{SI}}_k(X;Y)\geq 0$  with equality iff  $(X,Y)$  are independent.
3. KL divergence representation: We have
$$
\overline {{\mathsf {S l}}} _ {k} (X; Y) = \sup  _ {(A, B) \in \operatorname {S t} (k, d _ {x}) \times \operatorname {S t} (k, d _ {y})} \mathsf {D} \big (\left(\mathfrak {p} ^ {A}, \mathfrak {p} ^ {B}\right) _ {\#} \mu_ {X Y} \big \| \left(\mathfrak {p} ^ {A}, \mathfrak {p} ^ {B}\right) _ {\#} \mu_ {X} \otimes \mu_ {Y} \big),
$$
4. Sub-chain rule: For any random variables  $X_{1},\ldots ,X_{n},Y$  we have
$$
\overline {{\mathsf {S l}}} _ {k} (X _ {1}, \dots , X _ {n}; Y) \leq \overline {{\mathsf {S l}}} _ {k} (X _ {1}; Y) + \sum_ {i = 2} ^ {n} \overline {{\mathsf {S l}}} _ {k} (X _ {i}; Y | X _ {1}, \dots , X _ {i - 1}).
$$
5. Tensorization: For mutually independent  $\{(X_i,Y_i)\}_{i=1}^n$ ,  $\overline{\mathsf{S}}\mathsf{l}_k\left(\{X_i\}_{i=1}^n; \{Y_i\}_{i=1}^n\right) = \sum_{i=1}^n \overline{\mathsf{S}}\mathsf{l}_k(X_i; Y_i)$ .
The proof follows by similar arguments to those in the average-sliced case, but is given for completeness in Supplement A.1. Of particular importance are Properties 2 and 3. The former renders mSMI sufficient for independence testing despite being significantly less complex than the classical mutual information between the high-dimensional variables. The latter, which represent mSMI as a supremized KL divergence, is the basis for neural estimation techniques explored in Section 4.
Remark 3 (Relation to average-SMI). Beyond the inequality relationship in Property 1 above, Proposition 4 in [25] (paraphrased) shows that for matrices  $\mathrm{W}_x$ ,  $\mathrm{W}_y$  and vectors  $b_x, b_y$  of appropriate dimensions, we have  $\sup_{\mathrm{W}_x, \mathrm{W}_y, b_x, b_y} \underline{\mathrm{S}} \underline{\mathrm{l}}_1(\mathrm{W}_x^\intercal X + b_x; \mathrm{W}_y^\intercal Y + b_y) = \overline{\mathrm{S}} \underline{\mathrm{l}}_1(X; Y)$ , and the relation readily extends to projection dimension  $k > 1$ . In words, optimizing the aSMI over linear transformations of the high-dimensional data vectors coincides with the max-sliced version. This further justifies the interpretation of  $\overline{\mathrm{S}} \underline{\mathrm{l}}_k(X; Y)$  as the information between the two most informative representations of  $X, Y$  in a  $k$ -dimensional feature space. It also suggests that mSMI is compatible for feature extraction tasks, as explored in Section 5.3 ahead.
3.2 Gaussian Max-SMI versus CCA
The mSMI is an information-theoretic extension of the CCA coefficient  $\rho_{\mathrm{CCA}}(X,Y)$ , which is able to capture higher order dependencies. Interestingly, when  $(X,Y)$  are jointly Gaussian, the two notions coincide. We next state this relation and provide a closed-form expression for the Gaussian mSMI.
Proposition 2 (Gaussian mSMI). Let  $X \sim \mathcal{N}(m_X, \Sigma_X)$  and  $Y \sim \mathcal{N}(m_Y, \Sigma_Y)$  be  $d_x$ - and  $d_y$ -dimensional jointly Gaussian vectors with nonsingular covariance matrices and cross-covariance  $\Sigma_{XY}$ . For any  $k \leq d_x \wedge d_y$ , we have
$$
\overline {{\mathrm {S l}}} _ {k} (X; Y) = \mathrm {I} \left(\mathrm {A} _ {\mathrm {C C A}} ^ {\mathrm {T}} X; \mathrm {B} _ {\mathrm {C C A}} ^ {\mathrm {T}} Y\right) = - \frac {1}{2} \sum_ {i = 1} ^ {k} \log \left(1 - \sigma_ {i} \left(\mathrm {T} _ {X Y}\right) ^ {2}\right), \tag {4}
$$
where  $(\mathrm{A}_{\mathbb{C}\mathbb{C}\mathbb{A}},\mathrm{B}_{\mathbb{C}\mathbb{C}\mathbb{A}})$  are the CCA solutions from (2),  $\mathrm{T}_{XY} = \Sigma_X^{-1 / 2}\Sigma_{XY}\Sigma_Y^{-1 / 2}\in \mathbb{R}^{d_x\times d_y}$ , and  $\sigma_k(\mathrm{T}_{XY})\leq \ldots \leq \sigma_1(\mathrm{T}_{XY})\leq 1$  are the top  $k$  singular values of  $\mathrm{T}_{XY}$  (ordered).
This proposition is proven in Supplement A.2. We first show that the optimization domain of  $\overline{\mathrm{Sl}}_k(X;Y)$  can be switched from the product of Stiefel manifolds to the space of all matrices subject to a unit variance constraint (akin to (2)), without changing the mSMI value. This implies that the CCA solutions  $(\mathrm{A}_{\mathsf{CCA}},\mathrm{B}_{\mathsf{CCA}})$  from (2) are feasible for mSMI and we establish their optimality using a generalization of the Poincaré separation theorem [48, Theorem 2.2]. Specializing Proposition 2 to one-dimensional projections, i.e., when  $k = 1$  , the mSMI is given in terms of the canonical correlation coefficient  $\rho_{\mathsf{CCA}}(X,Y)\coloneqq \sup_{(\phi ,\theta)\in \mathbb{R}^{d_x}\times \mathbb{R}^{d_y}}\rho (\theta^{\intercal}X,\phi^{\intercal}Y)$  . Namely,
$$
\overline {{\mathrm {S l}}} _ {1} (X; Y) = \mathrm {I} \left(\theta_ {\mathrm {C C A}} ^ {\mathrm {T}} X; \phi_ {\mathrm {C C A}} ^ {\mathrm {T}} Y\right) = - 0. 5 \log \left(1 - \rho_ {\mathrm {C C A}} (X, Y) ^ {2}\right),
$$
where  $(\theta_{\mathsf{CCA}},\phi_{\mathsf{CCA}})$  are the global optimizers of  $\rho_{\mathsf{CCA}}(X,Y)$ .