Unconstrained Stochastic CCA: Unifying Multiview and Self-Supervised Learning
Authors: James Chapman
Lennie Wells
Ana Lawry Aguila
Venue: ICLR 2024 poster (2024)
Keywords: Canonical Correlation Analysis
Multiview Learning
Self-Supervised Learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: our work unifies CCA, Deep CCA and PLS through a Generalised Eigenvalue Problem framework, introduces faster algorithms with SGD, and sets new benchmarks.

Abstract: The Canonical Correlation Analysis (CCA) family of methods is foundational in multiview learning.
Regularised linear CCA methods can be seen to generalise Partial Least Squares (PLS) and be unified with a Generalized Eigenvalue Problem (GEP) framework.
However, classical algorithms for these linear methods are computationally infeasible for large-scale data.
Extensions to Deep CCA show great promise, but current training procedures are slow and complicated.
First we propose a novel unconstrained objective that characterizes the top subspace of GEPs.
Our core contribution is a family of fast algorithms for stochastic PLS, stochastic CCA, and Deep CCA, simply obtained by applying stochastic gradient descent (SGD) to the corresponding CCA objectives.
Our algorithms show far faster convergence and recover higher correlations than the previous state-of-the-art on all standard CCA and Deep CCA benchmarks.
These improvements allow us to perform a first-of-its-kind PLS analysis of an extremely large biomedical dataset from the UK Biobank, with over 33,000 individuals and 500,000 features.
Finally, we apply our algorithms to match the performance of `CCA-family' Self-Supervised Learning (SSL) methods on CIFAR-10 and CIFAR-100 with minimal hyper-parameter tuning, and also present theory to clarify the links between these methods and classical CCA, laying the groundwork for future insights.

UNCONSTRAINED STOCHASTIC CCA: UNIFYING
MULTIVIEW AND SELF-SUPERVISED LEARNING
James Chapman' Ana Lawry Aguila University College London (james chapman. 19, ana agui la. 18]eucl acc
Lennie Wells' University of Cambridge uk
ABSTRACT
The Canonical Correlation Analysis (CCA) family of methods is foundational multiview learning. Regularised linear CCA methods can be seen to generalise Partial Least Squares (PL.S) and be unified with Generalized Eigenvalue Problem (GEP) framework. However, classical algorithms for these linear methods are computationally infeasible for large-scale data. Extensions to Deep CCA show great promise, but current training procedures are slow and complicated. First we propose a novel unconstrained objective that characterizes the top subspace of GEPs. Our core contribution is family of fast algorithms for stochastic PLS, stochastic CCA, and Deep CCA, simply obtained by applying stochastic gradient descent (SGD) to the corresponding CCA objectives. Our algorithms show far faster convergence and recover higher correlations than the previous state-of-the art on all standard CCA and Deep CCA benchmarks. These improvements allow to perform a trst-off-itskinne PLS analysis of an extremely large biomedical dataset from the UK Biobank, with over 33,000 individuals and 500,000 features. Finally, apply our algorithms to match the performance of CCCA-family Self- Supervised Learning (SSL) methods on CIFAR-10 and CIFAR- 100 with minimal hyper-parameter tuning, and also present theory clarify the links between these methods and classical CCA, laying the groundwork for future insights.
INTRODUCTION
CCA methods learn highly correlated representations of multiview data. The original CCA method of (Hotelligg (133)) learns low-dimensional representations from linear transformations. Notable extensions ridge-regularized CCA (Vinod 1976), Partial Least Squares (PLS), and multiview CCA (Wong et al. 202T) allow one to use CCA in high dimensional regimes, and with three or more views of data. More recently, variety of Deep CCA methods (Andrew al., 2013) have been proposed which learn representations obtained from non-linear transformations of the data, parameterized by deep neural networks; Deep CCA has seen excellent empirical results, and so foundational for deep multiview learning that it secured runner-up position for the test-of-time award at ICML 2023 (ICML 2023).
However, there are significant computational challenges when applying these CCA methods to large- scale data. Classical algorithms for linear CCA methods require computing full covariance matri- and scale quadratically with dimension. becoming intractable for many datasets of practical interest. There is therefore great interest in approximating solutions for CCA in stochastic or data- streaming settings (Arora et al. 2012). Large-scale data also challenges existing full-batch algo- rithms for Deep CCA, and their stochastic counterparts are not only complex to implement but also difficult train (Wang et al. 2015b).
Self self-supervised learning (SSL) methods are now state-of-the-an in a range of domains, including image classification (Balestriero al. 2023). They also learn useful representations of data, usually from pretext tasks or objectives that exploit some inherent structure or property of the data. Remark- ably, SSL methods can even perform zero-shot classification: where the representations are learnt without any explicit labels or supervision. Of particular interest to us is the so-called CCA family of
"Equal contribution.

---

SSL methods (Balestriero et al. of data to a pair of similar representations, and notably include Barlow twins (Zoontar et al. 2021) and VICReg (Bardes et al. 2021), which have become popular in light of empirical successes.
2023). Like CCA, these methods aim to transform a pair of views
In section 2 we provide a unified approach to all the CCA methods introduced above, emphasizing objectives which are functions of the joint distributions of the transformed variables. Versions of all the linear CCA methods can be defined by solutions to certain Generalized Eigenvalue Problems (GEPs); this provides particularly convenient way to relate large number of equivalent objectives.
Section 3 outlines our core conceptual contributions. Firstly, with proposition 3.1 we present an unconstrained loss function that characterizes solutions to GEPs; this is based the Fechart-Young inequality and has appealing geometrical properties. We apply this to the GEP formulation of CCA and construct unbiased estimates of the loss and its gradients from mini-batchis of data. These loss functions can therefore be optimized out-of-the-box using standard frameworks for deep learning. This immediately gives a unified family of algorithms for CCA, Deep CCA, and indeed SSL.
Our CCA algorithms dominate existing of-te- art methods across wide range of benchmarks, presented in section 5 For stochastic CCA, our method not only converges faster but also achieves higher validation correlation scores than existing techniques. For Deep CCA and Deep Multiview CCA, our unbiased stochastic gradients yield significantly better validation correlations and allow the use of smaller mini-batches in memory constrained applications. We also demonstrate the prac- tical utility of our algorithms with a pioncering real-wo world case study. We apply stochastic Partial Least Squares (PLS) to extremely high-dimensional dataset from the UK Biobank executing biomedical analysis previously deemed intractable all on a standard laptop.
Finally, SSL method achieves comparable performance VICReg and Barlow twins, despite having no hyperparameters in the objective. This frees computational resources to tune more critical hyperparameters, such architecture, optimizer augmentations. Our method also appears more robust to these other hyperparameters, has a clear theoretical foundation, and naturally generalizes the multiview setting. addition, we present theory in section 3.4 and appendix D which gives a more thorough description of how the existing SSL methods of Barlow twins and VICReg relate CCA than in the previous work of Balestriero & LeCun (2022); we hope better understanding of these methods may lead to more principled empirical advances.
2 A UNIFIED APPROACH TO THE CCA FAMILY
Suppose we have sequence of vector vallud random variables x E D for (1....,/] I We want learn meaningful K -ddmennsional representations
2 = f (x(0)
D and 0 (0()) We will consistently use the super- scripts j,j E [T] for views and subscripts 1, E [K] for dimensions of representations i.e. to subscript dimensions of (()) Later on, we will introduce total number of samples N and mini-batch size M.
For convenience, define D D=
2.1
BACKGROUND: GEPS IN LINEAR ALGEBRA
Generalized Eigenvalue Problem (GEP) is defined by two symmetric matrices A, B E DxD Stewart & Sun 1990 They are usually characterized by the set of solutions to the equation:
Au = ABu
(2)
with À R, E R,, called (generalized) eigenvalue and (generalized) eigenvector respectively. We shall only consider the case where B is positive definite to avoid degeneracy. Then the GEP becomes equivalent to an eigen-decomposition of the symmetric matrix AB- This is key to the proof of our new characterization. In addition, one find basis of eigenvectors spanning RD_ We define a top- K subspace to be one spanned by some set of cigenvectors uy,... ug with
'A helpful mnemonic: there are / (cye) views.
More generally, A, B can Hermitian, hut we are only interested the real case.
2

---

the top-K K associated eigenvalues \ 2 Ag. We say matrix U Rx defines top-K
subspace if its columns span one.
2.2 THE CCA FAMILY
The classical notion CCA (Hotelingg sentations to be linear transformations
1992 considers two views, / 2, and constrains the repre-
% ("k x
The objective is find the weights or canonical directions Uy which maximize the canonical correlations Ps Corr(2 Corr(Z z sequentially, subject to orthogonality with the previous pairs of the transformed variables. is well known that CCA is equivalent to singular value decomposition It is slightly less well known
(SVD) of the matrix Var(x()) "//C))"""COOVO Var)
Borga 1998) that this is equivalent to GEP where:
Cov(x (
B (
(Var(x(()) 0
(2) (2)
A (o
Var(X
0
CCA therefore has notions of uniqueness similar those for SVD or GEPs: the weights are not in general unique, but the canonical correlations 2 P1 2 P2 2 ... 2 0 are unique (Anderson) 2003) Therefore, we can write:
CCA (x = k=
(5)
Sample CCA: in practice we do not have access to the population distribution but to finite number of samples; the classical estimator is defined by replacing the population covariances eq. 44) with sample covariances. Unfortunately, this estimator breaks down when N S max(D()) D(2) giving arbitrary correlations of and meaningless directions
Ridge-regularized CCA: the most straightforward way to prevent this overfitting to add a ridge regularizer (Vinod, 1976). Taking maximal ridge regularization recovers Partial Least Squares PLS (Mihalik et al. 2022), a widely used technique for multiview learning. Even these simple modifications to CCA can be very effective at preventing overfitting in high dimensions (Mihalik) ct al. 2022).
multiview CCA (MCCA): extends two-view CCA to deal with three more views of data. Unfor- tunately, many of the different equivalent formulations of two-view CCA are no longer equivalent in the multiview setting,
there many different extensions choose from; see section 4 Of most interest to us is the formulation of Niclsen| (2002); [Wong et al. (2021) that extends the GEP formulation of eq. (4, which next make precise and will simply refer to MCCA from now.
Unified GEP formulation: this GEP formulation of MCCA can be presented in unified framework generalizing CCA and ridge-regularized extensions. Indeed, we take A, Ba R DxD to be
block matrices = ( Ba = (B! X where the diagonal blocks of A are zero, the off-diagonal blocks of Ba are zero, and the remaining blocks are defined by:
A (t)) = Cov(x()) x(0) i B(()) a;TDa: + (1 Va)(((()(()))
(6)
Where a E [0, 11' is vector of ridge penalty parameters: taking = 0 Vi recovers CCA and Qg 1 Vi recovers PLS. We may omit the subscript when a and we recover the 'pure CCA' setting; in this case, following eq. ( we can define MICCAx(XLL ,....,....... x()) (X"O,,... the top-K generalized eigenvalues.
to be the vector of
Deep CCA: was originally introduced in Andrew al. (2013); this was extended to GEP-based formulation of Deep multiview CCA (DMCCA) in Somandepalli et al. (201)) This can be defined using our MCCA notation as maximizing
||CCAA (z"),..(())
over parameters 0 of neural networks defining the representations ( (x(0))00)) for i E [4].
WLOG take D 2 max(N, D(2)). Then for any given observations ( E N 2ZSh E R there
(1) such that (1) " exists some usg which is e.g. true with prohability when the observations drawn from continuous probability distribution.
(1) z provided the observations of (1) are not linearly dependent
3

---

3 NOVEL OBJECTIVES AND ALGORITHMS
3.1 UNCONSTRAINED OBJECTIVE FOR GEPS
First, present proposition 3.1 formulation of the top-K subspace of GEP problems, which follows by applying the Eckhart- Loung-Minsky incquality (Stewart & Sun 1990) to the eigen- decomposition of B 1/2 AB- However, making this rigorous requires some technical care which we defer to the proof in supplement
Proposition 3.1 (Eckhart-Younl inspired objective for GEPs). The top-K subspace of the GEP (A, B3) can be characterized by minimizing the following objective over U E R
EGEr-EY(U) == trace (-20T AU + (UT BU) (UT BU))
(8)
K Moreover, the minimum value is precisely - Σ k= XE. where (>x) are the generalized eigenvalues.
This objective also has appealing geometrical properties. It is closely related to a wide class of unconstrained objectives for PCA and matrix completion which have no spurious local optima (Ge al. 2017). i.e. all local optima are in fact global optima. This implies that certain local search algorithms, such as stochastic gradient descent, should indeed converge to a global optimum.
Proposition 3.2. The objective LGEP-EY has spurious local minima. That is, any matrix Ü that is a local minimum of LGEP-EY must in fact be a global minimum.
It is also possible to make this argument quantitative by proving a version of the strict saddle property from Ge et al. (2017; 2015): we state an informal version here and give full details in appendix B Corollary 3.3 (Informal: Polynomial-time Optimization). Under certain conditions on the eigen- values and generalized eigenvalues of (A, B), UK E IR Dx is either close to a global optimum, has a large gradient LGEP-EY. or has Hessian V EGEP-EY with large negative eigenvalue.
make quantitative the claim that: any
Therefore, for appropriate step-size sequences, certain local search algorithms, such as sufficiently noisy SGD, will converge in polynomial time with high probability.
3.2 CORRESPONDING OBJECTIVES FOR THE CCA FAMILY
For the case of linear CCA we have UT AU = Eiz, (or z())), U BU = E, (a)) To extend this to the general case of nonlinear transformations, eq. { we define the analogous matrices of total between-view covariance and total within-view variance
((0)
Σ (0rZZ,261))
V(0)= Σ Var(Z
{
For linear transformations, eq. 3 makes sense add ridge penalty so we can define
(0) Σ (1 + (1 - aaraa)aar(("))
(10)
This leads to the following unconstrained objective for the CCA-family of problems.
Definition 3.4 (Family EY Objectives). Learn representations ( = f0 (x minimizing
(EY( - trace C(0) + (0)
(11)
Unbiased estimates: since empirical covariance matrices are unbiased, we can construct unbiased estimates to C, V from batch of transformed variables Z.
C(0)(Z)
(Z (.Z)),,
(
Σ Var ())
(12)
In the linear case we can construct V. (0) analogously by plugging sample covariances into eq. 10). Then if Z, Z' are two independent batches of transformed variables, the batch loss
ŁEY Z, Z'] := -2 trace C|Z] + (Va [Z], Va(Z'))
(13)
gives an unbiased estimate of Lxy (f). loss a differentiable function of Z, Z' and so also of 8.
Simple algorithms: We first define general algorithm using these estimates in Algorithm In the next section we apply this algorithm to multiview stochastic CCA (CCA-EY) and PLS (PLS-EY), Deep CCA (DCCA-EY), and SSL (SSL-EY).
4

---

Algorithm 1: GEP-EY: General algorithm for learning correlated representations
Input: data stream of batches (X(6) where each consists of M samples from the original dataset. Learning rate (m)). Number of time steps T. Class of functions f(: whose outputs are differentiable with respect to A.
Initialize: 0 with suitably random entries
fort to T do
Obtain two independent mini- batches X(b).X(bb) by sampling b, independently
Compute batches of transformed variables ((b) ((X());)) Z(b) f(X
Estimate loss ÊEY (0) using eq. (13)
Obtain gradients by back- propagatonnnn and step with your favourite optimizer.
end for
3.3 APPLICATIONS TO MULTIVIEW STOCHASTIC CCA AND PLS, AND DEEP CCA
Lemma 3.5 (Objective recovers GEP formulation of linear multiview CCA). When the f() are linear, in eq. the population loss from eq. (IT) recovers MCCA as defined in section 2.2
Proof. By construction, for linear MCCA we have - AU, V. = U B,U, where (A, B. define the GEP for MCCA introduced in eq. (6). So LEY(U) = LGEP (U) and by proposition 3.1 the optimal set of weights define a top-K K subspace of the GEP, and so is a MCCA solution.
Moreover, by following through chain of back- paakaproppg,tion, we obtain gradient estimates in O(MKD) time. Indeed, the dominant cost is then updating U; we flesh this out with full details in appendix
obtain gradients for the transformed variables in O (MM) time so
Lemma 3.6. (Objective recovers Deep multiview CCAJ Assume that there is a final linear layer in each neural network (()), Then at any local optimum, 0, of the population problem, we have
LEr(θ) = -|IMCCAA((
where = (X)). Therefore, is also local optimum of objectives from Andrew et al. (201)) Somandepalli et al. (2019) as defined in eq. 7
Proof sketch: see appendix C.2./ for full details. Consider treating the penultimate-layer represen- tations as fixed, and optimising over the weights in the final layer. This is precisely equivalent to optimising the Eckhart-Y Young loss for linear CCA where the imput variables are the penultimate- layer representations. So by proposition 3.2. a local optimum also global optimum, and by proposition the optimal value is the negative sum of squared generalised eigenvalues.
L
3.4 APPLICATION TO SSL
We can directly apply Algorithm to SSL., If we wish to have the same neural network transforming each view, we can simply tic the weights ((3) 0(2), When the paired data are generated from applying independent, identically distributed (i.id.) augmentations to the same original datum, is intuitive that tying the weights is a sensible procedure, and perhaps acts as a regulariser. We make certain notions of this intuition precise for CCA and Deep CCA in appendix C
To provide context for this proposal, we also explored in detail how VICReg and Barlow twins are related to CCA. For now we focus on VICReg, whose loss can be written as
(v)(Z()) ((0)) (BIZ((1) 12
1 V.VVaarz))))
+ B
Cov(Z
((())
(E(1.)
Aight
where a, B, > 0 are tuning parameters and, as in the framework of section 2 the 2 (2) are K -ddimensionall representations, parameterised by neural networks in eq. ( The heuristic behind this loss is that the 7-term encourages the pair of representations to be similar (Invariance), the 3- term encourages different components of the representations to be uncorrelated (Covariance), and
5

---

the a-term enforces strictly positive variance of each component (Variance). Our main conclusions regarding optima of the population loss are as follows.
Consider the linear setting with untied weights. Then global optimisers of the VICReg loss define CCA subspaces, but may not be of full rank.
Consider the linear setting with tied weights and additionally assume that the data are gen- erated by i.idd, augmentations. Then the same conclusion holds.
In either of these settings, the optimal VICReg loss a component- wise decreasing func- tion of the vector of population canonical correlations CCA ( x
VICReg can therefore be interpreted formulation of Deep CCA, but one that will not in general recover full rank representations.
We give full mathematical details and further discussion in appendix D The population loss for Bar- low twins is motivated by similar heuristics, encouraging uncorrelated components and similarity between views, but is naturally viewed as explicitly constraining each component to have unit vari- ance one. This makes the analysis for Barlow twins more difficult, but we present combination of mathematical and empirical arguments which suggest all the same conclusions hold as for VICReg, again see appendix D for full details.
4 RELATED WORK
Stochastic PLS and CCA: To the best of our knowledge, the state-of-the-art in Stochastic PLS and CCA are the subspace Generalized Hebbian Algorithm (SGHA) of Chen et al. (2019) and y- EigenGame from Gemp et al. (2020) 2021), which we use benchmarks in the following section. Specifically. SGHA utilizes Lagrange multiplier heuristic along with saddle-point analysis, albeit with limited convergence guarantees. EigenGame focuses on jop-K subspace learning but intro- duces an adaptive whitening matrix the stochastic setting with an additional hyperparameter, Like method, both tackle other symmetric Generalized Eigenvalue Problems in principle.
DCCA and Deep Multiview CCA: The deep canonical correlation analysis (DCCA) landscape comprises three principal approaches with inherent limitations. The first, known the full-batch approach, uses analytic gradient derivations based on the full sample covariance matrix (Andrew et al. 2013). The second involves applying the full batch objective to large mini-batches. proach referred to as DCCA-STOL (Wang et al. 2015a). However, this approach gives biased gra- dients and therefore requires batch sizes much larger than the representation size in practice. This is the approach taken by both DMCCA (Somandepalli et al.] 2019) and DGCCA (Benton et al.) 2017) The final set of approaches use an adaptive whitening matrix (Wang et al. 2015b Chang et al. 2018) to mitigate the bias of the Deep CCA objective. However, the authors of DCCA-NOI highlight that the associated time constant complicates analysis and requires extensive tuning. These limitations make existing DCCA methods less practical and resource-efficieee
ap-
We are most interested comparisons to Barlow twins and VICReg because of their empirical success, and known relationship to CCA; we believe the most comprehen- sive existing theoretical analysis of this relationship is in Balestriero & LeCun (2022), however this subset of possible VICReg parameters. appears incomplete for Barlow twins, and
Self-Supervised Learning:
only applies does not include discussion of the rank of representations. For a wider perspective on SSL meth- ods and their applications we recommend the review of (Balestriero et al. 2023), and the efficient implementations available in solo-learn] Da Costa et al. (022).
EXPERIMENTS
5.1 EVALUATING CCA-EY FOR STOCHASTIC CCA
First, we compare proposed method, CCA-EY, to the baselines of 7-EieenGame and SGHA. Our experimental setup is almost identical to that Meng et al. (0021): Gemp et al. (2022). Unlike Gemp et al. (2022). do not simplify the problem by first performing PCA on the data before applying the CCA methods, which explains the decrease in performance of EigennGa compared to their work. All models are trained for single epoch with varying mini-batch sizes ranging from

---

Top CCA mediamill (d,=120, d,=120)
1.00
Top CCA on mediamil [0,=120, dy=120)
0.8
Train PCC
talkRQ
0.75
8OHA
0.60
VElginSamo
0.4
batch
0.25
0.00
100
0.2
20000
25000
Samples Seen
SGHA
Y-gannaaee
(b)
Figure I: Stochastic CCA on MediaMill using the Proportion of Correlation Captured (PCC) metric: (a) Across varying mini-batch sizes, trained for a single epoch, and (b) Training progress over single epoch for mini-batch sizes 5, 100. Shaded regions signify + one standard deviation around the mean of runs.
Top CCA SpiRMNNIST
Validation TCC
40
mode
DCCA-EY
Valdarion
30
DCCA-STOL
DCCA-NCI
80
20
100
50
100
epoch
DOCAEY
OCCA-NOI
DCCA-STOL
(a)
(b)
Figure 2: Deep CCA on SplitMNIST using the Validation TCC metric: (a) after training each model for 50 epochs with varying batch sizes; (b) learning progress over 50 epochs.
5 to 100. We use Proportion of Correlation Captured (PCC) as our evaluation metric, defined as PCC - (E p>) P where ph are the full batch correlations of the learnt representations, and P are the canonical correlations computed numerically from the full batch covariance matrices.
Figure[Tcompare: the algorithms on the MediaMill dataset. fig. shows that CCA- EY consistently outperforms both y-EigenGame and SGHA terms PCC across all evaluated mini-batch sizes. fig. Ib examines the learning curves for batch sizes 5 and 100 in more detail; CCA-EY appears to learn more slowly than SGHA at the start of the epoch, but clearly outperforms SGHA as the number of samples seen increases. 7 g-EgenGamme significantly underperforms SGHA and CCA-EY, particularly for small batch sizes.
Observations:
Further experiments: we conduct analogous experiments on the Split CIFAR dataset in supple- mentary material F and observe identical behaviour.
5.2 EVALUATING DCCA-EY FOR DEEP CCA
Second, we compare DCCA-EY against the DCCA methods described in section The experimen- tal setup is identical that of Wang al. (2015b). We learn K 50 dimensional representations, using mini-batch sizes ranging from 20 to 100 and train for 50 epochs. Because there is no longer ground truth, we have to use Total Correlation Captured (TCC), given by TCC -ΣK Pk where Pk: are now the empirical correlations between the representations on validation set.
Observations: Figure 2 compares the methods on the SplitMNIST dataset. DCCA-STOL captures significantly less correlation than the other methods, and breaks down when the mini-batch size less than the dimension K 50 due low rank empirical covariances. DCCA-NOI performs similarly to DCCA-EY but requires careful tuning of an additional hyperparameter, and shows sig- nificantly slower speed to convergence (Figure 2b
Further experiments: we conduct analogous experiments on the XRMB dataset in supplementary material and observe identical behaviour.
7

---

Top 50 CCA mfest
Val TMCC
CCA
Validation TMCC
modal
30
DCCA-EY
20
DMCGA
DCCCA
10
26
sie
100
200
batch
epoch
OOCN-YY
DMGGA
DGOCA
(b)
Figure 3: Deep multiview CCA mfeat using the Validation TMCC metric: (a) after training each model for 100 epochs with varying batch sizes; (b) learning progress over 100 epochs.
5.3 EXTENDING DCCA-EY TO THE MULTIVIEW SETTING
Third, we compare DCCA-EY to the existing DMCCA and DGCCA methods on the mfeat dataset; this contains 2,000 handwritten numeral patterns across six distinct feature sets, including Fourier coefficients, profile correlations, Karhunen-Love coefficients, pixel averages in 2 x 3 windows, Zernike moments, and morphological features. We again learn K
dimensional representations, but now train for 100 epochs. We use a multiview extension of the TCC metric, which averages correlation across views; we call this Total Multiview Correlation Captured (TMCC), defined as
TMCC = E r(T-1) EiseI corr( 2 )(,)), using the notation of section 2
Figure 3a shows that DCCA-EY consistently outperforms both DGCCA and DM- CCA across various mini-batch sizes in capturing validation TMCC. Just like DCCA-NOL, DMCCA breaks down when the batch size is smaller than K. This is due to singular empirical covariances; DGCCA does not break down, but does significantly underperform with smaller batch sizes. This limits their practical applicability to large-scale data. Figure 33b shows learning curves for batch sizes 50 and 100. DMCCA and DGCCA both quickly learn significant correlations but then plateau out; method consistently improves, and significantly outperforms them by the end of training.
Observations:
5.4 SCALING PLS THE UK BIOBANK WITH PLS-EY
Next, we demonstrate the scalability of our methods to extremely high- sigh-dimensional data by applying stochastic PLS to imaging genetics data from the UK Biobank (Sudlow et al.) 2015). PLS is typi- cally used for imaging-genetiio studies owing to the extremely high dimensionality of genetics data requiring lots of regularisation. PLS can reveal novel phenotypes of interest and uncover relation- ships between genetic mechanisms of discase and brain morphometry. Previous imaging genetics analyses using full-batch PLS were limited to much smaller datasets (Lorenzi et al. 2018 Taquet
ctt al. 2021 Edith Le Floch al. 2012). The only other analysis on the UK Biobank at comparable scale partitions the data into clusters and bootstraps local PLS solutions these clusters (Lorenzi cet al. 2017: Allmann et al., 2023). We ran PLS-EY with mini-bar batch size 500 on brain imaging (82 regional volumes) and geneties (582,565 variants) data for 33,333 subjects. See supplement (Sec-
tion [I3.4] for data pre-processing details. To our knowledge, this is the largest-scale PL. analysis of biomedical data to-date.
We see strong validation correlation between all 10 corresponding pairs of vectors in the PLS subspace and weak cross correlation, indicating that our model learnt coherent and thogonal subspace of covariation (Figure 4a). a remarkable feat for such high-dimensional data. We found that the PLS brain subspace was associated with genetic risk measures for several disorders (Figure 4b). suggesting that the PLS subspace encodes relevant information for genetic disease risk, a significant finding for biomedical research.
Observations:
5.5 APPLYING SSL-EY FOR PRINCIPLED SELF-SUPERVISED LEARNING
Finally, we benchmark our self-supervised learning algorithm, SSL-EY, with Barlow twins and VI- CReg on CIFAR-10 and CIFAR- 100. Each dataset contains 60,000 labelled images, but these are over 10 classes for CIFAR-10 and 100 classes for CIFAR-100.
8

---

Compunent
Comporent
Comparend
Compunerd
Component
AD
AD
822
99.66 Cinnitio comporers
+002
+0.2
000
0.02
0.50
Compuneett
Comparent
Comparreed
Compaoned
Compuneet 10
-022
AD
to
10u
AD
mg]
966
n9
8P'
1
115
ADHD
ADHD
40-0
ADHO
-0/3
A/S
PD
4.5
HD
14
AlS
70
N.5
P0
Pd
Brain componintt
-100
-002
0.00
EM
-062
003
602
EP
-052
002
10
0.02
tn
-002
0.00
0.52
Bn
-0.2
0.00
(a)
(b)
Figure 4: (a) Correlations between PLS components for UK Biobank. (b) Correlations between PLS brain components and genetic risk scores. AD=Alzheimer's ' disease, SCZ=Schizophrenia, BP=Bipolar, ADHD=Attention deficit hyperactivity disorder, ALS=Amyolrophic lateral sclerosis, PD=Parkinson's disease, EPI=Epilepsy. ns 0.05 < p <= 1, * 0.01 < <= 0.05, ** 0.001
< 0.01, * * 0.0001 < p <= 0.001.
Method
CIFAR- 10 Top-1
CIFAR-IO 10 Top-5
CIFAR 100 Top-1
CIFAR- 100 Top-5
Barlow twins
92.1
99.73
71.38
92.32
VICReg
91.68
68.56
90.76
SSL-EY
91.43
99.75
67.52
90.17
Table I: Performance comparison of SSL methods on CIFAR-10 and CIFAR-100.
We follow standard experimental design (Tong et al. 2023) and use solo-learn (Da Costa al., 2022). which offers optimized setups particularly tailored for VICReg and Barlow twins. All meth- ods utilize a ResNet-18 encoder coupled with bi-layer projector network. Training spans 1,000 epochs with batches of 256 images. For SSL-EY, we use the hyperparameters optimized for Barlow twins, aiming not outperform but showcase the robustness of our method. We predict labels via linear probe on the learnt representations and evaluate performance with Top-1 and Top-5 accuracies on the validation set. For more details, refer to the supplementary material
Observations: Table T shows that SSL-EY is competitive with Barlow twins and VICReg. This is remarkable because we used out-of-the-box hyperparameters for SSL-EY but used hyperparameters for Barlow twins and VICReg that had been heavily optimized in previous studies.
Further experiments included appendix H show that the learning curves for three methods are comparable, and that our method is much more stable when reducing the dimension of the learnt representations.
6 CONCLUSION
In this paper, we introduced a class of efficient, scalable algorithms for Canonical Correlation Anal- ysis and Self-Supervised Learning, rooted novel unconstrained loss function. These algorithms are computationally lightweight, making them uniquely suited for large-scale problems where tra- ditional methods struggle.
We have two distinct avenues for future research. Firstly, we aim to incorporate regularization tech- niques to improve both generalizability and interpretability, building upon existing sparse methods in CCA (Witten& & Tibshirani 2009). We also intend to investigate the utility of correlation a met- ric for measuring the quality of learnt representations. This holds the potential to replace traditional validation methods like classification accuracy, especially in situations where validation labels are not available.
In summary, this paper sets new benchmark for addressing lcrge-scale CCA problems and opens new avenues in superviee learning, paving the way for more accessible and efficient solutions in various applications.

---

AUTHOR CONTRIBUTIONS STATEMENT
JC conceived the original idea of developing new stochastic algorithms based on the GEP formula- tion CCA using the GEP formulation, and developed number of methods that appeared to perform well empirically. LW proposed the core Eckhart-Young GEP framework of this paper in response, and proved all mathematical results. JC suggested the extension SSL, and wrote all the code for the experiments. LW and JC wrote the paper collaboratively, focusing on theoretical and empiri- cal contributions respectively. ALA performed the analysis on the UK Biobank dataset, wrote the corresponding commentary, and edited the main text.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, we have made the following resources publicly available:
Code for stochastic CCA and deep CCA experiments: https://github. com/
jameschapman19 EY
For Self-Supervised Learning experiments, our modified version of solo-learn accessible at:
solo-learn
Standalone PoTch implementation of the SSL-EY loss function: btttps//ggiee
com/ comml/Jaamesschppaar11////1-1
ACKNOWLEDGMENTS
The authors thank Sergio Bacallado for valuable advice on how restructure earlier version of this paper, and on various details of the theoretical contributions. LW is supported by the UK Engi- neering and Physical Sciences Research Council (EPSRC) under grant number EP/V52024X/1. JC and ALA are supported by the EPSRC-funded UCL Centre for Doctoral Training in Intelligent, In- tegrated Imaging in Healthcare (i4health) and the Department Health's NIHR-funded Biomedical Research Centre at University College London Hospitals.
10

---

REFERENCES
Andre Altmann, Ana C Lawry Aquila, Neda Jahanshad, Paul M Thompson, and Marco Lorenzi. Tackling the dimensions in imaging genetics with CLUB-PLS. arXiv preprint arXiv:2309.07352, 2023.
T. W. Anderson. An introduction multivariate statistical analysis. Wiley series in probability and statistics. Wiley-Interscience, Hoboken, N.J, 3rd ed edition, 2003. ISBN 978-0-471-36091-9
Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation analysis. In International conference machine learning, pp. 1247-1255. PMIR, 2013.
Raman Arora, Andrew Cotter, Karen Liveseu, and Nathan Srebro. Stochastic optimization for PCA and PLS. In 2012 50th Anmual Allerton Conference on Communication, Control, and Computing (Allerton), pp. 861-868. IEEE, 2012.
Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-supervised learning cover global and local spectral embedding methods. arXiv preprint arXiv:2205. /508, 2022.
Randall Balestriero, Mark Ibrahim, Vlad Sohal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, al. A cookbook of self- supervised learning, arXiv preprint arXiv:2304. 122100, 2023.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Barianc-invvarincc--vvaaa regularziiioo for self-supervised learning. arXiv preprint arXiv:2105. 04906, 2021.
Adrian Benton, Huda Khayrallah, Biman Gujral, Dee Ann Reisinger, Sheng Zhang, and Raman Arora. Deep generalized canonical correlation analysis. arXiv preprint arXiv:/702.025519 2017.
Rajendra Bhatia. Matrix Analysis, volume 169 of Graduate Texts in Mathemat-
ics. Springer, New York. NY. 1997. ISBN 978-1-4612-685744 978-1-612--6---2033333
doi: 00.1007/978-1-4612-0663
URL http://hw...ppiiger.....ooo.....
378-1-4612-0653-8
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https://www. wandb. com/ Software available from wandb.com.
Magnus Borga. Learning Multidimenssioal Signal Processing. PhD thesis, 1998. URL http: 7/urn. xb.
Publisher: Linköping
University Electronic Press.
Marcus Carlsson. von Neumann's trace inequality for Hilbert operators. Exposi- tiones Mathematicae, 39(1):149-157 157, March 2021. ISSN 0723-0869. doi: 10.1016//.exmath
2020.05.001 URL https: htt::://wwww.. wtwp://wwwww.ieeeeeetttt...o/eee http:/rrett.cocccenneeeeeeeeee/e htt.://ee....o.eeeeeeeeeeerraraa 50723086920300222
Xiaobin Chang, Tao Xiang, and Timothy M Hospedales. Scalable and effective deep CCA via soft decorrelation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, pp. 1488-1497, 2018
James Chapman, Ana Lawry Aguila, and Lennie Wells. A generalized eigengame with extensions to multiview representation learning, arXiv preprint arXiv:2211.1132. 2022.
Zhehui Chen, Xingguo Li, Lin Yang, Jarvis Haupt, and Tuo Zhao. On constrained nonconvex
stochastic optimization: case study for generalized eigenvalue decomposition. The 22nd International Conference Artificial Intelligence and Statistics, pp. 916-925. PMLR, 2019.
Victor Guilherme Turrisi Da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. solo-learn: A library of supervsue methods for visual representation learning. J. Mach. Learn. Res., 23 (56):1-6. 2022.
11

---

Ditte Demontis, G Bragi Walters, Georgios Athanasiadis, Raymond Walters, Karen Therrien, Trine Tollerup Nielsen, Leila Farajzadeh, Georgios Voloudakis, Jaroslav Bendl, Biau Zeng, Wen Zhang, Jakob Grove, Thomas D Als, Jinjie Duan, F Kyle Satterstrom, Jonas Bybjerg-Grauhol Grauholm, Marie Baekved-Hansen, Olafur O Gudmundsson, Sigurdur Magnusson, Gisli Baldursson, Ka- trin Davidsdottir, Gyda S Haraldsdottir, Eshen Agerbo, Gabriel E Hoffman, Søren Dalsgaard, Joanna Martin, Marta Ribasés, Dorret Boomsma, Maria Soler Artigas, Nina Roth Mota, Daniel Howrigan, Sarah E Medland, Tetyana Zayats, Veera M Rajagopal, ADHD Working Group of the Psychiatric Genomics Consortium, iPSYCH-Broad Consortium, Merete Nordentoft, Ole Mors, David M Hougaard, Preben Bo Mortensen, Mark Daly, Stephen V Faraone, Hreinn Stefans- son, Panos Roussos, Barbara Franke, Thomas Werge, Benjamin M Neale, Kari Stefansson, and Anders D Børglum. Genome-wide analyses ADHD identify 27 risk loci, refine the genetic ar- chitecture and implicate several cognitive domains. Nat. Genet., 55(2): 198 108-1988211 198-208, February 2023.
Alan Edelman, Tomás A. Arias, and Steven T. Smith. The Geometry of Algorithms with Orthog- onality Constraints. SIAM Journal Matrix Analysis and Applicaations, 20(2):303-353, Jan- uary 1998. ISSN 0895-4798, 1095-7162. doi: 10. (0.11337/00855779995599555 URL http:
7/epubs. siam. am. org/ /tt.///111111/00077999998888881
Jack Euesden, Cathryn M. Lewis, and Paul F. O'Reilly. PRSice: Polygenic Risk Score software. Bioinformatics, 31(9):1666 1468, 2014. ISSN 1367-4803. doi: 10. 1093/bioinformatie/
btu848. URL http:///.....trto.......oo/o//ot
Bruce Fischl. FreeSurfer. Neuroimage, 62(2):774-781, August 2012.
Rong Ge, Furong Huang, Chi Jin, and Yang Yuan. Escaping from saddle points online stochastic gradient for tensor decomposition, 2015.
Rong Ge, Chi Jin, and Yi Zheng. No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysis. In Proceedings of the 34th International Conference Machine Learning, pp. 1233-1242 1242. PMLR, July 2017. URL htttttps//prcceeinnngg.. http://rrooeddnnggg...r..r... ISSN: 2640-3498.
v v70/ 7a. html
lan Gemp, Brian McWilliams, Claire Vernade, and Thore Graepel. Eigengame unloaded: When playing games is better than optimizing, 2021.
Ian Gemp. Charlie Chen, and Brian McWilliams. The generalized eigenvalue problem nash equilibrium. arXiv preprint arXiv:2206. 21f.:22206.49933,, 2022.
Ian M. Gemp, Brian McWilliams, Claire Vernade, and Thore Graepel. FigenGame: PCA as Nash Equilibrium. CoRR, abs/2010.00554, 2020. 00554
Willem H. Haemers. Interlacing eigenvalues and graphs. Linear Algebra and its Applications, 226- 228:593-616, September 1995. ISSN 00243795. doi: 10.0016/0222477755555009992 URL
com
Harold Hotelling. Analysis of a complex statistical variables into principal components. Journal of educational psychology, 24(6))417, 1933.
Harold Hotelling. Relations between two sets variates. Breakthroughs in statistics, pp. 162- 190. Springer, 1992.
ICML. ICML 2023, 2023.
URL
httpss//ccal.c///oofferrees////
Test-of -ineeee
International League Against Epilepsy Consortium Complex Epilepsies. Genome-wide mega- analysis identifies 16 loci and highlights diverse biological mechanisms the common epilepsies. Nat. Commun., (1):5266, December 2018.
Li Jing, Pascal Vincent. Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning- arXiv preprint irXiv:2110.093453 2021.
12

---

Tosio Kato. Perturbation Theory for Linear Operators, volume 132 of Classics in Mathe- matics. Springer. Berlin, Heidelberg. 1995. ISBN 978-3-540-58661-6 978-3-642-66282.
9. doi: 00.1007/978-3-644-668888 978 1-8-3-642--2 6642-66282-9
URL http: www.nksr.iiigggccco.oooo
J C Lambert, C A lbrahim-Verbaas, D Harold, A C Naj. R Sims, C Bellenguez, A L DeStafano, Russo, A Thorton- Wello,, N Jones, A Smith, V Chouraki, C Thomas, M A Ikram, D D B N Vardarajan, Y Kamatani, Fin,,, A Gerrish, H Schmidt, B Kunkle, M I. Dunstan, A Ruiz, M T Bihoreau, S H Choi, C Reitz, F Pasquier, C Cruchaga. D Craig, N Amin, C Berr, O L Lopez, PL De Jager, V Deramccourt, A Johnston, Morón, D Rubinsztein, G Eiriksdottir, K Sleegers, A M
C Bis. W Beecham, B Grenier-Boley,
D Evans, S Lovestone, L Letenneur, Goate, N Fiévet, M W Huentelman, M Gill, K Brown, MI Kamboh, L. Keller, P Barberger- B McGuiness, E B Larson, R Green, A Myers, C Dufouil, S Todd, D Wallon, S Love, E Rogaeva, Gallacher, P St George-Hyslop. Clarimon, A Lleo, A Bayer, D W Tsuang, L Yu, M Tsolaki, P Bossù, G Spalletta, P Proitsi, J Collinge. S Sorbi, F Sanchez-Garcia, N C Fox, J Hardy, M C Deniz Naranjo, P Bosco, R Clarke, C Brayne, D Galimberti, M Mancuso, F Matthews, European Alzheimer's s Disease Initiative (EADI), Genetic and Environmental Risk in Alzheimer's ' Disease, Alzheimer's Disease Genetic Consortium, Cohorts for Heart and Aging Research in Genomic Epidemiology. S Moebus, P Mecocci, M Del Zompo, W Maier, H Hampel, A Pilotto, M Bullido, Panza, Caffarra, Nacmias, J R Gilbert, M Mayhaus, L. Lannefelt, H Hakonarson, S Pichler, M M Carrasquillo, M Ingelsson, D Beekly, V Alvarez, F Zou, O Valladares, S G Younkin, E Coto, K L Hamilton-Nelson, W Gu, C Razquin, P Pastor, Mateo, M Owen, K M Faber, P Jonsson, O Combarros, M C O' O'Donovan, L B Cantwell, H Soininen, D Blacker, S Mead, T H Mosley, Jr, D A Bennett, T B Harris, L Fratiglioni, C Holmes, R F de Bruijn, P Passmore, T J Montine, K Bettens, J I Rotter, A Brice, K Morgan, T M Foroud, W A Kukull, D Hannequin, F Powell, M A Nalls, K Ritchie, K I. Lunetta, S Kauwe, E. Boerwinkle, M Riemenschneider, M Boada, M Hiltuenen, E R Martin, R Schmidt, D Rujescu, S Wang. J F Dartigues, R Mayeux, C. Tzourio A Hofman, M M Nöthen, C Graff, B M Psaty, L Jones, L Haines, P A Holmans, M Lathrop, M A Pericak-Var Vance, L Launer, L A Farrer, M van Duijn, C Van Broeckhoven, V Moskvina, S Seshadri, J Williams, G D Schellenberg, and P Amouyel. Meta-analysis of 74,046 individuals identifies 11 new susceptibility loci for alzheimer's disease. Nat. Genet., 45(12):1452 1458, December 2013.
Marco Lorenzi, Boris Gutman, Paul M Thompson, Daniel C Alexander, Sebastien Ourselin, and Andre Altmann. Secure multivariate large-scale multi-centric analysis through on-line learning: an imaging genetics case study. In 12th International Symposium on Medical Information Pro- cexsing and Analysis, volume 10160, pp. 347-353. SPIE, 2017.
Marco Lorenzi, Andre Altmann, Boris Gutman, Selina Wray, Charles Arber, Derrek D Hibar, Neda J Jahanshad, Jonathan Schott, Daniel Alexander, Paul M. Thompson, and Sébastien Ourselin. Susceptibility of brain atrophy to TRIB3 in Alzheimer's disease, evidence from functional prioritization in imaging genetics. Proceedings of the National Academy of Sciences of the United States of America, 115(12)3312-3111 2018. doi:
1706100115. URL
Jiajun Ma, Tianyang Hu, and Wenjia Wang. Deciphering the projection head: Representation eval- uation self-supervised learning. arXiv preprint arXiv:2301.12189, 2023.
Zihang Meng, Rudrasis Chakraborty, and Vikas Singh. An online riemannian PCA for stochastic canonical correlation analysis. Advances in Neural Information Processing Systems, 34:14056- 14068, 2021.
Agoston Mihalik, James Chapman, Rick A. Adams, Nils R. Winter, Fabio S. Ferreira, John Shawe-Taylor, and Janaina Mourão- Mouräo-Miranda. Canonical Correlation Analysis and Partial Least Squares for identifying behivibb associations: tutorial and comparative study. Bio- logical Psychiatry: Cognitive Neuroscience and Neuroimaging, August 2022. ISSN 2451-9022 doi: 10.101166..psc..20000 202...0..22......... URL https: //ww:/ sci encedi rect. com sc ence/ htt.icce/pp1//525551222222222
Niamh Mullins, Andreas Forstner, Kevin S O'Conelll Brandon Coombes, Jonathan R I Coleman, Zhen Qiao, Thomas D Als, Tim B Bigdeli, Sigrid Børte, Julien Bryois, Alexander W Chamey,
13

---

Ole Kristian Drange, Michael J Gandal, Saskia P Hagenaars, Masashi Ikeda, Nolan Kamitaki, Minsoo Kim, Kristi Krebs, Georgia Panagiotaropoulou, Brian M Schilder, Laura G Sloofman, Stacy Steinberg, Vassily Trubetskoy, Bendik S Winsvold, Hong-Hee Won, Liliya Abramova, Kristina Adorjan, Eshen Agerbo, Mariam AI Eissa, Diego Alhani, Ney Alliey-Rodriguez, Ade- bayo Anjorin, Vernri Antilla, Anastasia Antoniou, Swapnil Awasthi, Ji Hyun Baek, Marie Barkvad-Hanset Nicholas Bass, Michael Bauer, Eva C Beins, Sarah E Bergen, Armin Birner, Carsten Bøcker Pedersen, Erlend Bøen, Marco Boks, Rosa Bosch, Murielle Brum, Ben M Brumpton, Nathalie Brunkhorst-Kamana Monika Budde, Jonas Bybjerg-Grauholae Grauholm, William By- erley, Murray Cairns, Miquel Casas, Pablo Cervantes, Toni-Kim Clarke, Cristiana Cruceanu, Alfredo Cuellar-Barboza, Julie Cunningham, David Curtis, Piotr M Czerski, Anders M Dale, Nina Dalkner, Friederike $ David, Franziska Degenhardt, Srdjan Djurovic, Amanda L. Dobbyn, Athanassios Douzenis, Torbjørn Elvsåshagen, Valentina Escott-Price, I Nicol Ferrier, Alessia Fiorentino, Tatiana M Foroud. Liz Forty, Josef Frank. Oleksandr Frei, Nelson B Freimer, Louise Frisén, Katrin Gade, Julie Garnham, Joel Gelernter, Marianne Giørtz Pedersen, Ian R Gizer, Scott D Gordon, Katherine Gordon-Smith, Tiffany A Greenwood, Jakob Grove, José Guzman- Parra, Kyooseob Ha, Magnus Haraldsson, Martin Hautzinger, Urs Heilbronner, Dennis Hellgren, Stefan Herms. Per Hoffmann, Peter A Holmans, Laura Huckins, Stéphane Jamain, Jessica S John- son, Janos L Kalman, Yoichiro Kamatani, James L Kennedy, Sarah Kittel-Schnneider, James A Knowles, Manolis Kogevinas, Maria Koromina, Thorsten M Kranz, Henry R Kranzler, Michiaki Kubo, Ralph Kupka, Steven A Kushner, Catharina Lavebratt, Jacob Lawrence, Markus Leber, Hcon-Jcong Lee, Phil H Lee, Shawn E Levy, Catrin Lewis, Calwing Liao, Susanne Lucae. Mar- tin Lundberg, Donald J MacIntyre, Sigurdur Magnusson, Wolfgang Maier, Adam Maihofer, Dolores Malaspina, Eirini Maratou, Lina Martinsson, Manuel Mattheisen, Steven A McCarroll, Nathaniel W McGregor, Peter McGuffin, James McKay, Helena Medeiros, Sarah E Med- land, Vincent Millischer, Grant W Montgomery, Jennifer L Moran, Derek W Morris, Thomas W Mühleisen, Niamh O' O'Brien, Claire O' O'Donovan, Loes M Olde Loohuis, Lilijana Oruc, Sergi Papiol, Antonio F Pardiñas, Amy Perry. Andrea Pfennig, Evgenia Porichi, James B Potash, Digby Quested, Towfique Raj. Mark H Rapaport, J Raymond DePaulo, Eline J Regeer, John P Rice, Fabio Rivas, Margarita Rivera, Julian Roth, Panos Roussos, Douglas M Ruderfer, Cristina Sánchez-Mora, Eva C Schulte, Fanny Senner, Sally Sharp, Paul D Shilling, Engilhert Sigurds- son, Lea Sirignano, Claire Slaney, Olav B Smeland, Daniel Smith, Janet Sobell, Chris- tine Søholm Hansen, Maria Soler Artigas, Anne T Spijker, Dan Stein, John S Strauss, Beata Swikatkowska, Chikashi Terao, Thorgeir Thorgeirsson, Claudio Toma, Paul Tooney, Evangelia- Eirini Tsermpini, Marquis P Vawter, Helmut Vedder, James T R Walters, Stephanie H Witt. Simon Xi, Wei Xu, Jessica Mei Kay Yang, Allan H Young, Hannah Young, Peter P Zandi, Hang Zhou, Lea Zillich, HUNT All-In Psychiatry, Rolf Adolfsson, Ingrid Agartz, Martin Alda, Lars Alfreds- son, Gulja Babadjanova, Lena Backlund, Bernhard Baune, Frank Bellivier, Susanne Bengesser, Wade H Berrettini, Douglas H R Blackwood, Michael Bochnke, Anders D Børglum. Gerome Breen, Vaughan J Carr, Stanley Catts, Aiden Corvin, Nicholas Craddock, Udo Dannlowski, Dimitris Dikeos, Tonu Esko, Bruno Etain, Panagiotis Ferentinos, Mark Frye, Janice M Fuller- ton, Micha Gawlik, Elliot S Gershon, Fernando S Goes, Melissa Green, Maria Grigoroiu- Serbanescu, Joanna Hauser, Frans Henskens, Jan Hillert, Kyung Sue Hong, David M Hougaard, Christina M Hultman, Kristian Hveem, Nakao Iwata, Assen V Jablensky, Ian Jones, Lisa A Jones, René S Kahn, John R Kelsoe, George Kirov, Mikael Landén, Marion Leboyer, Cathryn M Lewis, Qingqin S Li, Jolanta Lissowska, Christine Lochner, Carnel Loughland, Nicholas G Martin, Carol A Mathews, Fermin Mayoral, Susan McElroy, Andrew M McIntosh, Francis J McMa- hon, Ingrid Melle, Patricia Michie, Lili Milani, Philip B Mitchell, Gunnar Morken, Ole Mors, Preben Bo Mortensen, Bryan Mowry, Bertram Müller-Myhsok, Richard M Myers, Benjamin M Neale, Caroline M Nievergelt, Merete Nordentoft, Markus M Nöthen, Michael C O' Donoovn,, Ketil Oedegaard, Tomas Olsson, Michael J Owen, Sara A Paciga, Chris Pantelis, Carlos Pato, Michele T Pato, George P Patrinos, Roy H Perlis, Danielle Posthuma, Josep Antoni Ramos- Quiroga, Andreas Reif, Eva Z Reininghaus, Marta Ribasés, Marcella Rietschel, Stephan Ripke, Guy A Rouleau, Takeo Saito, Ulrich Schall, Martin Schalling, Peter R Schofield, Thomas G Schulze, Laura J Scott, Rodney J Scott, Alessandro Serretti, Cynthia Shannon Weickert, Jor- dan W Smoller, Hreinn Stefansson, Kari Stefansson, Eystein Stordal, Fabian Streit, Patrick F Sullivan, Gustavo Turecki, Ame E Vaaler, Eduard Vieta, John B Vincent, Irwin D Waldman, Thomas W Weickert, Thomas Werge, Naomi Wray, John-Anker Zwart, Joanna M Biernacka, John I Nurnberger, Sven Cichon, Howard Edenberg, Eli A Stahl, Andrew McQuillin, Arianna Di Florio, Roel Ophoff, and Ole Andreassen. Geneme-wide association study of more than
14

---

40,000 bipolar disorder cases provides new insights into the underlying biology. Nat. Genet., 53 (00:817-229, June 2021.
Mike A Nalls, Comelis Blauwendraat, Costanza L Vallerga, Karl Heilbron, Sara Bandres-Ciga, Di- ana Chang, Manuela Tan, Demis A Kia, Alastair J Noyce, Angli Xue, Jose Bras, Emily Young. Rainer Coelln. Javier Simón-Sánchezz Claudia Schulte, Manu Sharma, Lynne Krohn, Lasse Pihlstrøm, Ari Sittonen, Hirotaka Iwaki, Hampton Leonard, Faraz Faghri, J Raphael Gibbs, Dena G Hernande7, Sonja W Scholz, Juan A Botia, Maria Martinez, Jean-Christonn Suzanne Lesage, Joseph Jankovic, Lisa M Shulman, Margaret Sutherland, Pentri Tienari, Kari Majamaa, Mathias Toft, Ole A Andreassen, Tushar Bangale, Alexis Brice, Jian Yang, Ziv Gan- Or, Thomas Gasser, Peter Heutink, Joshua M Shulman, Nicholas W Wood, David A Hinds, John A Hardy, Huw R Morris, Jacob Gratten, Peter M Visscher, Robert Graham, Andrew B Singleton, M3 Research Team, System Genomics of Parkinson's 's Disease Consortium, and International Parkinson's 's Disease Genomics Consortium. Identification of novel risk loci, causal insights, and heritable risk for parkinson's 's disease: meta-analysis of genome-wide association studies. Lancer Neurol., 18912):10911111 102, December 2019.
Corvol,
Allan Aasbjerg Nielsen. Multiset canonical correlations analysis and multispectral, truly multitem- poral remote sensing data. IEEE transactions on image processing. 11(3):23-335, 2002.
Shaun Purcell, Benjamin Neale, Kathe Todd-Brown, Lori Thomas, Manuel A R Ferreira, David Bender, Julian Maller, Pamela Sklar, Paul W de Bakker, Mark J Daly, and Pak C Sham. PLINK: tool set for whole-genome association and population-bass based linkage analyses. Am. J. Hum. Gener., (1(3):559-55 5755, September 2007.
Krishna Somandepalli, Naveen Kumar, Ruchir Travadi, and Shrikanth Narayanan. Multimodal rep- resentation learning using deep multiset canonical correlation. arXiv preprint arXiv:1904.0177 2019.
G. W. Stewart and Ji-Guang Sun. Matrix Perturbation Theory. ACADEMIC PressINC, July 1990. ISBN 978-1-4933-01999
Google-Books-DD bIYEogEACAAJ.
Cathie Sudlow, John Gallacher, Naomi Allen, Valerie Beral, Paul Burton, John Danesh, Paul Downey, Paul Elliott, Jane Green, Martin Landray, et al. Uk biobank: an open access resource for identifying the causes of wide range of complex diseases of middle and old age. PLoS medicine, 12(3)ee00117) 2015.
Maxime Taquet, Stephen M Smith, Anna K Prohl, Jurriaan M Peters, Simon K Warfield, Benoit Scherrer, and Paul J Harrison. A structural brain network of genetic vulnerability to psychiatric illness. Mol. Psychiatry, 266))2008--,000, June 2021.
Shengbang Tong, Yubei Chen, Yi Ma, and Yann Lecun. EMP-SSL: rowards self-supervised learning in one training epoch. arXiv preprint arXiv:2304.0397 2023.
Vassily Trubetskoy, Antonio F Pardiñas, Ting Qi. Georgia Panagiotaropoulou, Swapnil Awasthi, Tim B Bigdeli, Julien Bryois, Chia- Yen Chen, Charlotte A Dennison, Lynsey S Hall, Max Lam, Kyoko Watanabe, Oleksandr Frei, Tian Ge, Janet C Harwood, Frank Koopmans, Sigurdur Mag- nusson, Alexander L Richards, Julia Sidorenko, Yang Wu, Jian Zeng, Jakob Grove, Minsoo Kim, Zhiqiang Li, Georgios Voloudakis, Wen Zhang, Mark Adams, Ingrid Agartz, Elizabeth G Atkin- son, Esben Agerbo, Mariam AI Eissa, Margot Albus, Madeline Alexander, Behrooz Z Alizadeh, Köksal Alptekin, Thomas D Als, Farooq Amin, Volker Arolt, Manuel Arrojo, Lavinia Athana- siu, Maria Helena Azevedo, Silviu A Bacanu, Nicholas J Bass. Martin Begemann, Richard A Belliveau, Judit Bene, Beben Benyamin, Sarah E Bergen, Giuseppe Blasi, Julio Bobes, Stefano Bonassi, Alice Braun, Rodrigo Affonseca Bressan, Evelyn Bromet, Richard Bruggeman, Peter Buckley, Randy L. Buckner, Jonas Bybjerg-Grauholnn Wiepke Cahn, Murray J Cairns, Monica E Calkins, Vaughan J Carr, David Castle, Stanley V Catts, Kimberley D Chambert, Raymond C K Chan, Boris Chaumette, Wei Cheng. Eric F C Cheung, Siow Ann Chong, David Cohen, Angèle Consoli, Quirino Cordeiro, Javier Costas, Charles Curtis, Michael Davidson, Kenneth L Davis, Lieuwe de Haan, Franziska Degenhardt, Lynn E DeLisi, Ditte Demontis, Faith Dickerson, Dim- itris Dikeos, Timothy Dinan, Srdjan Djurovic, Jubao Duan, Giuseppe Ducci, Frank Dudbridge, Jo- han G Eriksson, Lourdes Fañanás, Stephen V Faraone, Alessia Fiorentino, Andreas Forstner, Josef
15

---

Frank, Nelson B Freimer, Menachem Fromer, Alessandra Frustaci, Ary Gadelha, Giulio Gen- ovese, Elliot S Gershon, Marianna Giannitelli, Ina Giegling, Paola Giusti-Rodriguez, Stephanie Godard, Jacqueline I Goldstein, Javier González Peñas, Ana Söonzälez-Pintee Srihari Gopal, Jacob Gratten, Michael F Green, Tiffany A Greenwood, Olivier Guillin, Sinan Gülöksüz, Raquel E Gur, Ruhen C Gur, Blanca Gutiérrez, Eric Hahn, Hakon Hakonarson, Vahram Haroutunian, Annette M Hartmann, Carol Harvey, Caroline Hayward, Frans A Henskens, Stefan Herms, Per Hoffmman,, Daniel P Howrigan, Masashi Ikeda, Conrad Iyegbe, Inge Joa, Antonio Julià, Anna K Kähler, Tony ham-Thong, Yoichiro Kamatani, Sena Karachanak- Yankova, Oussama Kebir, Matthew C Keller, Brian J Kelly, Andrey Khrunin, Sung- Wan Kim, Janis Klovins, Nikolay Kondratiev, Bet- tina Konte. Julia Kraft, Michiaki Kubo, Vaidutis Kučinskas, Zita Ausrele Kučinskiene, Agung Kusumawardhani, Hana Kuzelova-Ptackova, Stefano Landi, Laura Lazzeroni, Phil Lee, So- phie E. Legge, Douglas S Lehrer, Rebecca Lencer, Bernard Lerer, Miaoxin Li, Jeffrey Lieherman, Gregory A Light. Svetlana Limborska, Chih-Min Liu, Jouko Lönngvist, Carmel M Loughland. Jan Lubinski, Jurjen J Luykx, Amy Lynham, Milan Macek, Jr. Andrew Mackinnon, Patrik K E Magnusson, Brion S Maher, Wolfgang Maier, Dolores Malaspina, Jacques Mallet, Stephen R Marder, Sara Marsal, Alicia R Martin, Lourdes Martorell, Manuel Mattheisen, Robert W Mc- Carley, Colm McDonald, John J McGrath, Helena Medeiros, Sandra Meier, Bela Melegh, In- grid Melle, Raquelle Mesholam-Gatede Milanova, Marina Mitjans. Espen Molden, Esther Molina, María Dolores Molto, Valeria Mon- delli, Carmen Moreno, Christopher P Morley, Gerard Muntané, Kieran C Murphy, Inez Myin- Germeys. Igor Nenadić, Gerald Nestadt, Liene Nikitina-Zake, Cristiano Noto, Keith H Nuechter- lein, Niamh Louise O'Brien, F Anthony O' O'Neill, Sang- Yun Oh, Ann Olincy, Vanessa Kiyomi Ota, Christos Pantelis, George N Papadimitriou, Mara Parellada, Tiina Paunio, Renata Pellegrino, Sathish Periyasamy, Diana O Perkins, Bruno Pfuhlmann, Olli Pietiläinen, Jonathan Pimm, David Porteous, John Powell, Diego Quattrone, Digby Quested, Allen D Radant, Antonio Rampino, Mark H Rapaport, Anna Rautanen, Abraham Reichenberg, Cheryl Roe, Joshua L Roffman, Ju- lian Roth, Matthias Rothermundt, Bart P F Rutten, Safaa Saker-Delye, Veikko Salomaa, Julio Sanjuan, Marcos Leite Santoro, Adam Savitz, Ulrich Schall, Rodney J Scott. Larry J Seidman, Sally Isabel Sharp. Jianxin Shi, Larry J Siever, Engilbert Sigurdsson, Kang Sim, Nora Skarabis, Petr Slominsky, Hon-Cheong So, Janet I. Sobell, Erik Söderman, Helen J Stain, Nils Fiel Steen, Agnes A Merixner-Kumaar Elisabeth Stögmann. William S Stone, Richard Straub, Fabian Streit, Eric Strengman, T Scott Stroup, Mythily Subramaniam, Catherine A Sugar, Jaana Suvisaari, Dragan M Svrakie, Neal R Swerdlow, Jin P Szatkiewiez, Thi Minh Tam Ta, Atsushi Takahashi, Chikashi Terao, Florence Thibaut, Draga Toncheva, Paul A Tooney, Silvia Torretta, Sarah Tosato, Gian Battista Tura Bruce Turetsky, Alp Üçok, Arne Vaaler, Therese van Amelsvoort, Ruud van Winkel, Juha Veijola, John Waddington, Henrik Walter, Anna Waterreus, Bradley T Webb, Mark Weiser, Nigel M Williams, Stephanie H Witt, Brandon K Wormley, Jing Qin Wu, Zhida Xu, Robert Yolken Clement C Zai, Wei Zhou, Feng Zhu, Fritz Zimprich, Eşref Cem Atbaşoğlu, Muhammad Ayub, Christian Benner, Alessandro Bertolino, Donald W Black, Nicholas J Bray, Gerome Breen, Nancy G Buccola, William F Byerley, Wei J Chen, C Robert Cloninger, Bene- dicto Crespo-Facorro, Gary Donohoe, Robert Freedman, Cherrie Galletly, Michael J Gandal, Massimo Gennarelli, David M Hougaard, Hai-Gwo Hwu, Assen V Jablensky, Steven A McCar- roll, Jennifer L Moran, Ole Mors, Preben B Mortensen, Bertram Müller-Myhsok, Amanda L Neil, Merete Nordentoft, Michele T Pato. Tracey L Petryshen. Matti Pirinen, Ann E Pulver, Thomas G Schulze, Jeremy M Silverman, Jordan W Smoller, Eli A Stahl, Debby W Tsuang, Elis- abet Vilella, Shi-Heng Wang, Shuhua Xu, Indonesia Schizophrenia Consortium, PsychENCODE, Psychosis Endophenotypes International Consortium, SynGO Consortium, Rolf Adolfsson, Celso Arango, Bernhard T Baune, Sintia lole Belangero, Anders Børglum, David Braff, Elvira Bra- mon, Joseph D Buxbaum, Dominique Campion, Jorge A Cervilla, Sven Cichon, David A Collier, Aiden Corvin, David Curtis, Marta Di Forti, Enrico Domenici, Hannelore Ehrenreich, Valentina Escott-Price, Tonu Esko, Ayman H Fanous, Anna Gareeva, Micha Gawlik, Pablo V Gejman, Michael Gill, Stephen Glatt, Vera Golimbet, Kyung Sue Hong, Christina M Hultman, Steven Hyman, Nakao Iwata, Erik G Jönsson, René S Kahn, James I. Kennedy, Elza Khusnutdinova, George Kirov, James A Knowles, Marie-Odile Krebs, Claudine Laurent-Levinson, Jimmy Lee, Todd Lenez, Douglas F Levinson, Qinggin S Li, Jianjun Liu, Anil K Malhotra, Dheeraj Malho- tra, Andrew McIntosh. Andrew McQuillin, Paulo R Menezes, Vera A Morgan, Derek W Morris, Bryan J Mowry, Robin M Murray, Vishwajit Nimgaonkar, Markus M Nöthen, Roel A Ophoff, Sara A Paciga, Aaro Palotie, Carlos N Pato, Shengying Qin, Marcella Rietschel, Brien P Riley, Margarita Rivera, Dan Rujescu, Meram C Saka, Alan R Sanders, Sibylle G Schwab, Alessan-
Andres Metspalu, Patricia T Michie, Lili Milani, Vihra
16

---

dro Serretti, Pak C Sham, Yongyong Shi, David St Clair, Hreinn Stefānsson, Kari Stefansson, Ming Tsuang, Jim van Os, Marquis P Vawter. Daniel R Weinberger, Thomas Werge, Dieter B Wildenauer, Xin Yu, Weihua Yue, Peter A Holmans, Andrew J Pocklington. Panos Roussos, Evangelos Vassos, Matthijs Verhage, Peter M Visscher, Jian Yang, Danielle Posthuma, Ole A Andreassen, Kenneth S Kendler, Michael J Owen, Naomi R Wray, Mark J Daly, Hailiang Huang, Benjamin M Neale, Patrick F Sullivan, Stephan Ripke, James T Walters, Michael C CO" Donovan, and Schizophrenia Working Group of the Psychiatric Genomies Consortium. Mapping genomic loci implicates genes and synaptic biology in schizophrenia. Nature, 604(7906):502-500 -08, April 2022.
Wouter van Rheenen, Rick A A van der Spek, Mark K Bakker, Joke J A van Vugt, Paul J Hop, Ra- mona A J Zwamborn, Niek de Klein, Harm-Jan Westra, Olivier Bakker, Patrick Deelen, Gemma Shireby, Eilis Hannon, Matthieu Moisse, Denis Baird, Restuadi Restuadi, Egor Dolzhenko, An- nelot M Dekker, Klara Gawor, Henk-Jan Westeneng. Gijs H P Tazelaar, Kristel R van Eijk, Maarten Kooyman, Ross P Byrne, Mark Doherty, Mark Heverin, Ahmad A1 Khleifat, Alfredo la- coangeli, Aleksey Shatunov, Nicola Ticozzi, Johnathan Sooper-Knock, Bradley N Smith, Marta Gromicho, Siddharthan Chandran, Suvankar Pal, Karen E Morrison, Pamela J Shaw, John Hardy, Richard W Orrell, Michael Sendtner, Thomas Meyer, Nazli Başak, Anneke J van der Kooi, Anto- nia Ratti, Isabella Fogh, Cinzia Gellera, Giuseppe Lauria, Stefania Corti, Cristina Cereda, Daisy Sproviero, Sandra D' Alfonso, Gianni Sorarù, Gabriele Siciliano, Massimiliano Filosto, Alessan- dro Padovani, Adriano Chiò. Andrea Calvo, Cristina Moglia, Maura Brunetti, Antonio Canosa, Maurizio Grassano, Ettore Beghi, Elisabetta Pupillo, Giancarlo Logroscino, Beatrice Nefussy, Alma Osmanovic, Angelica Nordin, Yossef Lerner, Michal Zabari, Mare Gotkine, Robert H Baloh, Shaughn Bell, Patrick Vourc'h, Philippe Corcia, Philippe Couratier, Stéphanie Mille- camps, Vincent Meininger, François Salachas, Jesus S Mora Pardina, Abdelilah Assialioui, Ri- cardo Rojas-García, Patrick A Dion, Jay P Ross, Albert C Ludolph, Jochen H Weishaupt, David Brenner, Axel Freischmidt, Gilbert Bensimon, Alexis Brice, Alexandra Durr, Christine A M Payan, Safa Saker- Nicholas W Wood, Simon Topp, Rosa Rademakers, Lukas Tittmann, Wolfgang Lieb, Andre Franke, Stephan Ripke, Alice Braun, Julia Kraft, David C Whiteman, Catherine M Olsen, Andre G Uitterlinden, Albert Hofman, Marcella Rietschel, Sven Cichon, Markus M Nöthen, Philippe Amouyel, SLALOM Consortium, PARALS Consortium, SLAGEN Consortium, SLAP Consortium, Bryan Traynor, Andrew B Singleton, Miguel Mitne Neto, Ruben J Cauchi, Roel A Ophoff, Martina Wiedau-Pazos, Catherine Lomen-Hoerth, Vivianna M van Deerlin, Julian Grosskreutz, Annekathrin Roediger, Nayana Gaur, Alexander Jörk, Tabea Barthel, Erik Theele, Benjamin llse. Beatrice Stubendorff, Otto W Witte. Robert Steinbach, Christian A Hübner, Caroline Graff, Lev Brylev, Vera Fominykh, Vera Demeshonok. Anastasia Ataulina, Boris Rogelj. Blaž Koritnik, Janez Zidar, Metka Ravnik-Glavač, Damjan Glavač, Zor- ica Stević, Vivian Drory, Monica Povedano, Ian P Blair, Matthew C Kiernan, Beben Benyamin, Robert D Henderson, Sarah Furlong, Susan Mathers, Pamela A McCombe, Merrilee Needham, Shyuan T Ngo, Garth A Nicholson, Roger Pamphlett, Dominic B Rowe, Frederik Steyn, Kelly L Williams, Karen A Mather, Perminder S Sachdev, Anjali K Henders, Leanne Wallace, Mamede de Carvalho, Susana Pinto, Susanne Petri, Markus Weber, Guy A Rouleau, Vincenzo Silani, Charles J Curtis. Gerome Breen, Jonathan D Glass, Robert H Brown, Jr, John E Landers, Christo- pher E Shaw, Peter M Andersen, Ewout J N Groen, Michael A van Es, R Jeroen Pasterkamp, Dongsheng Fan, Fleur C Garton, Allan F McRae, George Davey Smith, Tom R Gaunt, Michael A Eberle, Jonathan Mill, Russell L McLaughlin, Orla Hardiman, Kevin P Kenna, Naomi R Wray, Ellen Tsai, Heiko Runz, Lude Franke, Ammar Al-Chalabi, Philip Van Damme, Leonard H van den Berg, and Jan H Veldink. Common and rare variant association analyses in amyotrophic lateral sclerosis identify 15 risk loci with distinct genetic architectures and neuron neuron-specific specific biology. Nat. Genet., 1336116633 December 2021.
Hrishikesh D Vinod. Canonical ridge and econometrics of joint production. Journal of econometrics, 4(2):147-166, 1976.
Weiran Wang, Raman Arora, Karen Livescu, and Jeff A Bilmes. Unsupervised learning of acous- tic features via deep canonical correlation analysis. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4590-4594, IEEE, 2015a
Weiran Wang, Raman Arora, Karen Liveseu, and Nathan Srebro. Stochastic optimization for deep CCA via nonlinear orthogonal iterations. In 2015 53rd Annual Allerton Conference on Commu- nication, Control, and Computing (Allerton), pp. 188-669. IEFE, 2015b.
17

---

Lennie Wells, Kumar Thurimella, and Sergio Bacallado. Regularised Canonical Correlation Anal- ysis: graphical lasso, biplots and beyond, March 2024. URL http:// ptttp://arxive org/ abs/
2403. 02979 arXiv:2403.02979 [math, stat].
Daniela M Witten and Robert J Tibshirani. Extensions of sparse canonical correlation analysis with applications to genomic data. Statistical applications in genetics and molecular biology, 8(1), 2009.
Hok Shing Wong, Li Wang, Raymond Chan, and Tieyong Zeng. Deep tensor CCA for multi-view learning. IEEE Transactions on Big Data, 866):1664-1677, 2021.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint arXiv:2 2021.
Edith Le Floch, Vincent Guillemot, Vincent Frouin, Philippe Pinel, Christophe Lalanne, Laura Trinchera, Arthur Tenenhaus, Antonio Moreno, Monica Zilbovicius, Thomas Bourgeron, Stanis- las Dehaene, Bertrand Thirion, Jean- Bann-Batistt Poline, and Édouard Duchesnay. Significant corre- lation between set of genetic polymorphisms and a functional brain network revealed by feature selection and sparse partial least squares. Neurolmage, 63(1):1 1-24, 2012. ISSN 1053-81 19. doi: http:///o..ogggll.oloffeeeeennnp com/ ooool//ccoooooerrrreeeeo///////0
URL https: /www. sci enced rect.
18

---

A ECKHART- YOUNG CHARACTERIZATION OF GEP SUBSPACE
Our characterisation of the top-K subspace of GEPs with the GEP-EY loss is given in appendix the key to the proof is the algebraic manipulation in eq. (16). which reduces our GEP-EY loss the form of the loss that appears in the Eckhart-Y formulation of the Eckhart- Young inequality to apply to this term; we state this as corollary A.10 and build machinery to prove it over the following subsections.
inequality. However, we will need non-standard
A.1 FORMAL DEFINITIONS
There are various different notations and conventions for GEPs and SVDs. We largely follow the standard texts Matrix Analysis (Stewart & Sun 1990, Bhatia 1997) but seek more careful han- dling of the equality cases of certain results. To help, we use the following non-standard definitions, largely inspired by Carlsson (022I).
Definition A.1 (Top-K subspace). Let the GEP (A, R have eigenvalues Å 2 ... > Ap- Then top-K subspace is that spanned by some u1...., uK, where un is a Ae-eigenvector (A, B) for k = ..... K.
Definition A.2 (B- -rothooorrmaltty. Let B E R be strictly positive definite. Then say col- lection U,,... W.K E IR" of vectors is B-orthonormal if uj Bw dui for each k, E (1,..., K). Definition A.3 (Top-K matrix). We say U E P is top-K matrix for GEP (A, B) if the kth column Ui U is -iggennver for each and the columns are B-orthonormal.
A.2 STANDARD ECKHART- YOUNG INEQUALITY
Theorem A.4 (Eckhart-Young Let M E Rexo, Then M minimises || - M] over matrices M of rank at most K if and only if M AgRx B where (Ak, Rx, Bx) is some top-K K SVD of the target M.
Proof. Let M, M have singular values σb, ã. respectively, Since M has rank at most K we must have ō₁ = for k K.
Then by von Neumann's trace inequality (Carlsson 2021).
(M, M) S
ORŌR
with equality if and only if M, 'share singular vectors'; the notion sharing singular vectors is defined as in Carlsson (2021) and in this case means that M = Rg Bx where (Ak, Rx. Bx) is some top-K SVD M and Rg is a diagonal matrix with decreasing diagonal elements Ø1 2 ... 2 ōk.
Expanding out the objective and applying this inequality gives
||
M
O
ok
k=1
k=1
k=1
D
K
Σ
O
+
(0k 5z)²
k= K=
k=
k=K+1
so indeed to have equality in both cases requires a = ã for each & < so indeed Rg = Rg and so M, as defined the statement of the theorem, minimises ||M MM || over matrices rank most K.
19

---

A.3 ECKHART- YOUNG FOR FACTORED ESTIMATOR OF SYMMETRIC TARGET
Lemma A.S (Matrix square root lemma). Suppose we have two full rank matrices E, F E R where K < D and such that EE FF"; then there exists an orthogonal matrix O E RR with E = FO.
Proof. Post multiplying the defining condition gives EE E FF E. Then right multiplying by (E' E)- gives
= FF F(E B)- =:
FO
to check that O as defined above is orthogonal we again use the defining condition to compute
o-o= = (E E FF E(E E) (E E EE E(E E) 1K
Remark A.6 (Tilde convention). In the rest of this section, the tildes above the quantity W be- low is to indicate that said matrix may not have orthonormal columns, whereas quantities without tildes (Wk, W+, W-) implicitly do have orthonormal columns. similar convention applies the following subsection with the matrices Ü,Us UKK
Corollary A.7 (Eckhart- Younh for factored estimator of PSD target). Let M E IR DxD be symmetric positive semidefinite. Then
arg min J WW 1
W WERDXX
precisely the set of W of the form W of the GEP (M,I) and some orthogonal OK E O(K), and where Ag is diagonal matrix of the top-K K eigenvalues.
WK A K O0 for some top-K K eigenvector-matrix W K
Proof. First note that when M is positive semi-definite the SVD coincides with the eigendecompo- sition.
Second note that taking W Wx 1/2 Ok attains the minimal value by the Eckhart- Young inequal- K
ity, Theorem A.4
Next note that if W attains the minimal value then it must have WWT WK W by the equality case of Bckhart-Young. Then by matrix square root Lemma A.5 must indeed have W = WKAK 1/2 Ok for some orthogonal Ok-
To convert this to the case of general symmetric target (with possible negative eigenvalues), use the following decomposition of a matrix into a sum of matrices corresponding to its positive and negative eigenvalues respectively. We will also find the following terminology helpful.
Definition A.8 (Non-negative eigenspace). Let M RDx be symmetric matrix. Then the non-negative eigenspace of M is defined as the span of the eigenvectors of M with non-negative eigenvalues.
Remark A.9 (Decomposition matrix into positive and negative eigenspaces). Let M E D be a symmetric matrix. Write A4 for the diagonal matrix containing the strictly positive eigenvalues of M and A_ for the strictly negative eigenvalues; let the corresponding eigenvectors be arranged in the matrices W+, W_ respectively. Define M, W_A. A4 W M_ A_ WT Then M is positive semi-definite, M_ is negative semi-definite, M = M_ + M_ and (M+, M-) the orthogonality of eigenvectors of M.
by
Corollary A.10 (Eckhart- Young for factored estimator of symmetric target). Let M E R DXD be symmetric with eigenvalues X 2 ... 2 An such that A > Then
arg min | WW 12 WERDXX
is precisely the set of W of the form W of the GEP (M, 1) and some orthogonal Ox E O(K), and where Ax is diagonal matrix of the top-K eigenvalues.
WK Ox for some top-K eigemvector-matrix W K
20

---

Proof. Let W R Write .M = MM + M in remark A.9 and suppose there D+ strictly positive and D_ strictly negative eigenvalues of M respectively. Then we can expand out
[M - W 1WW12
M WW 11 +2 (- WW) +2 (M- M-)
|M-I
2 A2
EEED
(14)
where the 'underbraced" (in)equalities below the equation follow from: applying corollary A.7 to M4. using that - is positive semi-definite, the orthogonality of M_ to M_, and the definition of M_ in terms of the eigenvalues of M, respectively. Combining the terms gives
D
|| WW 1 N
Σ
X
(15)
k=++1
as claimed.
Moreover, from the equality case of corollary A.7 when W = AKA and for such W. (M-. WW) 0 in fact there is also equality in the second inequality and therefore in the whole expression, as required.
equality holds in the first inequality precisely
A.4 GEP-EY OBJECTIVE
Proposition A.11 (GEP-EY-O DEE-EY-Objective), Consider the GEP (A, B) with A symmetric and B positive definite; suppose there are at least K strictly positive (generalized) eigenvalues. Then:
{2 (ÜTAÚ) - -ÚTTBÚ (ŪT (UTBū)}
arg max trace ŪERDXN
precisely the set of U of the form U UKA OK for some top- matrix UK of the GEP and some orthogonal Og E O(K), where AK is diagonal matrix of the top-K eigenvalues.
Moreover, the maximum value is precisely Ek=1
K
X
Proof. First note that there is bijection between eigenvectors u for the GEP (A, B) and eigenvec- tors w B²/² u for the GEP (M, where M := B 1/1 AB 1/2 (e.g. see Chapman et al. (022)). )
Now consider how the Eckhart-Young objective from Corollary A.10 transforms under the corre- sponding bijection W B
We get
JM WWB ||B AB B B
||8 AB 1/11//1// 2trace
BVA)
(16)
trace BB'OOUT B B'ÜŪT BV²)
AB- trace ( - - (ÚTBÜ)) (UTBÜ)}
where the first term is independent of U, so can conclude by Corollary A.10
The moreover conclusion can follow from computing the objective at any maximiser of the form above. We note that
ÚT AŪ =
= OKAROR
ÚT BŮ = A L BUK = =O
plugging into the objective gives
K
trace (2 AD) - (Ū BÚ}2 = trace (20 -
= Σ
k=1
because the trace of symmetric matrix is equal to the sum of its eigenvalues.
21

---

B TRACTABLE OPTIMIZATION NO SPURIOUS LOCAL MINIMA
First in appendix B.1 we prove that for general A, B our loss EGEPLEY (U) has no spurious local minima. Then in appendix B.3 we apply result from Ge et al. (2017). This application is somewhat crude, and we expect that a quantitative result with tighter constants could be obtained by adapting the argument of appendix B.I leave such analysis to future work.
B.I QUALITATIVE RESULTS
First we present our main result and its proof. This makes use of three crucial supporting lemmas, which we will address afterwards. Throughout the subsection we will use an over-bar (U etc) to denote quantities corresponding to the 'given original point of interest'. The structure of the proof has many similarities to the arguments of the previous appendix A but we do not use the tilde convention of remark A.6 instead using different letters to indicate orthonormality: matrices V, V have orthonormal columns, while other matrices (U, U, W, W) do not general. Throughout this section, as elsewhere, the 1/2 power of a positive-semi-defiie matrix denotes its (unique) positive semi-definite square root.
Proposition B.1 (No spurious local minima). The (population) objective LOEP-EY has no spurious local minima. That is, any matrix that is a local minimum of EgEP-RY must in fact be a global minimum of the form described in proposition 3.7
Proof. Suppose (for contradiction) that Ü is local optimum of LOEFEY, but not a global minimum.
As in appendix A.4 we first reduce to the setting. Write M B- AB- , W B and similarly for the 'barred' quantity W B'/2..
Following eq. (16). our Eckhart- Eckhart-Young loss transforms as
(UR)-EY(U) B AB 1/2 BL B B - AB 11?
||M WW T I |MI =: ((W)
and so since U + B is a homeomorphism, we see that W is a local minimum of I that is not global minimum.
w must lie in the space of non-negative eigenvalues of M, and therefore also that I(W) = M WW 1 M+ 1/.. using the notation of remark A.9.
By lemma
B.2
Next take some QR decomposition W V R. Because W is a local minimum of we must have that R is a local minimum of l' : R + ||M - V VRR VTI... Therefore R must be of the form specified in lemma B.3, and the value of at W precisely
((W) = - - M4 V
(17)
Finally we apply lemma B.4 (to the PSD matrix M4) to conclude. This gives an analytic path Vtt) with (00)= = V. We now convert this continuous path for W. Write R (VT M_ VY O for an orthogonal matrix Ô. Then let R(t) (V (t)) and W(t) R(t.. Then R(t) is continuous since the positive definite square root is continuous on positive semi- deniniefinite matrices and by construction, (RR (t) = VT V.V. Note also from the construction in lemma B.4 that (()) remains in the space spanned by eigenvectors of M with non-negative eigenvalues (we simply rotate towards cigenvector with the largest, and so positive, eigenvalue). Therefore
((W()) - M4VI2 < - -I M I(W)
(18)
contradicting local minimality.
The following lemma is stated using the notation of definition A.8 and remark A.9 Lemma B.2. Let M E DxD be a symmetric matrix, and write I(W) = ||M - WW 1 Then firstly, any W contained within the non non-negative eigenspace of M we have
I(W) = || WW I12 ||
Secondly, suppose W E RDx non-negative eigenspace of M.
K is a local minimum of l. Then span(W) (W} is contained within the
22

---

Proof. For the first claim, following eq. 14 expand
|M - www 12 | WW 2 ([M,,WWW'
) 2 ||M-12
M WW NWTTi + |M-112
and use that |M - || |MM-1
For the second claim, let P- be projection onto the span of eigenvectors of M with strictly negative eigenvalues, and Pzo- Then write W P- W PzoW W W- W+. Consider moving along the
continuous path (tt) (t) W+ where (t) 1 for t E [0, 1]. This time expand out further
| - WW T 2 = || + |M 2(WWT, M4) - 2(WW MM-)r + |WWT]]
and then the terms involving W can be written
(WWT, M4) (W+WI, M+)
(WWT. MM. (W WWW-WIM..
|WW T 1 12= -|| W|| -/|==1|W,10 +,
W +7 W+ W ) y | W_1
Since (W_WI, .M-) < we see that the coefficient of 72 non-negative. W_ * O, then the coefficient of 7 strictly positive and therefore increasing decreases 7 and strictly decreases the loss.
must therefore have W- = 0, as required.
Lemma B.3. Let M D be symmetric positive semi-defintte matrix and let V E R have orthonormal columns. Consider the loss function
I(R) - VII |
over R E RKK, Then the global minima I are precisely the R satisfying
RR. MV.
Moreover, there spurious local optima, and the minimun value is precisely
VIV MV
(19)
Proof. First for the global statement. Write r RRT Then complete the square to give
||M - VIVT 13-- trace V VVT (V VJr [VTV 2 tracee((V (VV MV) + ||
trace 1 A trace I(V1 MV) + ||M
= || - (V [V + || - || T MV
from which we can read off that the minimum attained precisely when
r V MV
and that the optimal value is precisely the value of eq. 19 claimed (a family of suitable square roots R exist because TT MV inherits positive cemi-detinte-nnnt from M).
Finally show there are spurious local minima. Suppose that R is not global optima. Then the corresponding is not global optimum, and R = + for some orthogonal matrix Ô. But since the objective just quadratic in I, we can construct path of positive definite matrices I'(t) with 1'(0) I' and || VT(t)V T || strictly decreasing in é (for example following the gradient dynamics). But then, continuous, to obtain a continuous path of matrices R(t) with R(0) = and ((R(t) < ((R) for all Therefore, R is not local minima either, as required.
simply take R(t) ((t) noting that the PSD square root is
23

---

Lemma B.4 (Rotating to capture signal, weak). Let M E RDD be a symmetric positive definite matrix. Let E DxK have orthonormal columns, which do not span a top-K subspace for M. Then there exists analytic path V |0. 1] + R PD*K t + (t) such that: ((t) has orthonormal columns for all 1, V(0) = V, and the signal captured
((t) [IV(()) - (V
strietly increasing
Proof. Let Vx span(V). Let the cigenvectors of M be (vf)s with corresponding eigenvalues (At)se with A 2 A2 2 .... 2 d 2 0. For simplicity we suppose that of $ Vx and that A > X2 (i.e. that there is strictly separated maximal direction of signal that is not yet captured by the estimated subspace). A very similar construction works more generally but the notation becomes significantly more complicated (we will return this at the end of the proof).
Now perform a CS-decomposition Stewart & Sun 1990) on the pair of subspaces (Vx, span(ui] This gives basis .1,.... .............................. 9K for K where
@i = cos(0) ()) v + sin sin(0) p
and i P. ................................ ŌK mutually orthogonal, and 0 E T/,
Now we construct a path for E [0, 1] via
(4) = cos (0)) of + sin(((()) D. 0(1) = (1 - 1)0
(t) 9k: for h = .....K K
We can compute
o Mg1 = b cos2 0 + (p Mp) sin? q Mqq. = = M sin 0, for k 2, .K
We obtain a corresponding path (t) as follows. Write Q(t) for the matrix with columns (qx (t)) recalling that only the first column depends on t. Then s ((0)) = span Th & [K]] Vx by construction and so we have V = ((0) Ô for some orthogonal matrix O E RR We therefore define ((t) Q(I)O for E [0, 1].
k=
To aid with subsequent algebraic manipulations will write c² cos? 0, sin =
c, T = 5J Mp. We can then rewrite 9i Mq1 = Aic² + rs² = >1 - (A1 - -r)s?. We also write x² = Ek (TT 2q Note that c² s² depend on & through 0, but P, Tk do not depend on t so neither do 7 or 0². Therefore, up to constant term in 0 we have
<(t) MVI = JlQTMQQQ
K
= Σ k,1=l
(gg Mq
=a Mq1) 2 +2
(q) M )a + est
k=2
K
= (A - (A1 - 22 + Σ
82 ( M + est
k=2
= X - >1 (A1 + s(1 - - 25 + est
= 5((11 -)²² 22 \1 - 7)) X est
Differentiating with respect to 2 therefore gives
o (s-2 = 23 2)
7) + +2 - - -))
"This may seem like magic, but can be motivated for example through the characterisation of geodesies on the Stiefel manifold through the CS-dccomposition, see the excellent monograph of Edelman al. 1998)
24

---

We that this is strictly increasing in s² because 7 - A2 < A1. Therefore, it strictly negative for
all s² E (0, 1) if and only if it non-positive when evaluated at = 1, which gives the condition
0 (Y1 -) (o² >1 -))
- 2r>1 + o²
T +
o²
We now show that this condition fact follows by Pythagoras' theorem: we have
-²+ S || S X27 S \IT.
as required (using T 2 since . is postive semi-definite). The central inequality holds because p is orthogonal to the leading eigenvector vi. This can be seen by of to full orthonormal basis of eigenvectors for M denoted (0x) and with corresponding eigenvectors (A+) Since p is orthogonal to "I we can expand p- = Ek= "3 g
to give
K
|| = Σ Aan
A2d A2p Mp 2==2
Finally we return to sketch the extension to the general case. Here, let & be the first index such that & VI. where V denotes the span of eigenvectors of M with eigenvalue greater than equal to d5 (which may be of dimension larger than & when the 0 eigenvalue is repeated). Then perform the CS decomposition on the pair of subspaces (VK. V%) to get basis q of cos 0; + pt sin 6 Construct path by taking (x) (1 t)0, and leaving all other 8i fixed. Then ps is orthogonal to V so & Mp < As. which allows the rest of the argument to proceed as before.
B.2 CONJECTURED STRONGER CONSTRUCTION
We now conjecture that stronger version of the construction of lemma B.4 can produce an analytic path that increases each eigenvalue of V MV individually, rather than just increasing its Frobenius norm. We state this below as conjecture B.5 We have yet find complete proof of this result, but have reduced it to matrix analytic result that appears tractable, and which appears to hold based on numerical simulations.
We present partial progress, because we believe this is interesting and important enough to merit further investigation. We also need the result later when analysing Barlow twins.
Conjecture B.5 (Rotating to capture signal: each eigenvalue increases). Let M E DxD be symmetric positive definite matrix, with eigenvalues (Az) Ler V Dx have orthonormal columns such that A == V MV diagonal.
Then there exist an analytic path (V(())eejo.) of matrices in DD with orthonormal columns such that (0) V, and the matrix A(t) == (tt MV diagonal with entries (x(t) that are strictly increasing for any k such that (0) < X
Partial proof progress. Again we exploit the CS- dS-decomposition. Write V for the span of eigen- vectors of M with eigenvalue greater than equal to Aι (which may be of dimension larger than K when the Ax eigenvalue repeated).
Now perform CS decomposition the pair of subspaces (VK, Vk)- This gives basis .1...., ęK for Vx of the form
C = cos 0z)) ) o sin(fx) Phc
(20)
where "r....,yy orthonormal each other and also all the v. In fact the Pt give basis for span(Vk. 1/ n
" form an orthonormal basis for a K-dimensional subspace of Vi-- and the ple are
(
Then as in the proof of lemma
B. 5.4 we can consider reducing the angles with some functions (x)))
for & E [K] to obtain vectors q (t). Package the first K vectors (1)( (*t) (7x) ( into
the matrices Q, V P each in R xxn In addition write
C(t) diag(cos Qx (t) [K]), S(t) diag (t()) : k [K])
(21)
25

---

Then eq. (20) can be rewritten
- PS(()
We then convert back to path (tt) of the form ((t) Q(t) where O(t) is orthogonal matrix that diagonalises Q' (() (rather than having constant matrix O as in lemma B.4 The fact that such orthogonal matrix be taken be analytic function of / follows from Kato (1995 (Subsection 2.6.2
Next note that the cigenvalues of V MV are precisely the eigenvalues of the matrix
(Q MQ)(() (V*C()) PS(()) M(V PS(())
= -CIVV* MV*C()) + S(1) MPS(()
And that in addition, each of the eigenvalues of VV* is greater than equal to λx, while the eigenvalues PT M P are all strictly less than Ax (and indeed less than or equal to the next biggest eigenvalue M). One can therefore write 4 T TMV*
MV
Ax Jx + A, and PT MP AxIK T, for positive semi-definite A, Then using C(I)² + S(t)² Ik. we obtain
(Q (Q) AKlK ((1)a(()) S(t))S(().
The following claim would give a way to proceed.
Claim: Let A, I be positive semi-definite. Let C(I), S(L) be constructed in eq. (1) for some functions 0g with each 0₂ valued in [0, 2/2 and decreasing in t. Then for each k the h eigen- S(t) increasing in
value (C(E)A((()
We have verified this numerically for large number of random positive semi-definite matrices A, I and decreasing funetions 4.. The result becomes straightforward prove when either A or I is zero. However, the proof in general has eluded us.
B.3 QUANTITATIVE RESULTS
To use the results from Ge et al. 2017 we need to introduce their definition of (0, 7. () strict saddle.
Definition B.6. We say function 1(-) is a (6, 7, -t-strcc saddle for any x, least one of the following holds:
1. [((x)|| 2
2. Amii (V((())
3. x is Ç-close to X the set local minima.
We can now state restate Lemma 13 from Ge al. (2017) in notation; this used in their analysis of robust PCA, and directly applies to our PCA-type formulation.
Lemma B.7 (Strict saddle for PCA). Let M R DxD be a symmetric PSD matrix, and define the matrix factorization objective over 7. E D
1(2) = IIM - ZZT 112
Assume that Xxx = (M) 2 15 + (M). Then
1. all local minima satisfy _2 = PK the best rank-K approximation to M
2. the objective I(Z) is (c, ((x))))))))))))))))())) ) saddle.
Perturration theory in finite dimensional space Perturbation of symmetric operators Orthonormal families of eigenvectors
26

---

However, do not want show strict saddle of / but of COFFEY U I(B Provided that has strictly positive minimum and bounded maximum eigenvalues this implies that COEPLEY is also strict saddle, as now make precise.
Lemma B.8 (Change of variables for strict saddle conditions). Suppose that I is (0, S-strict sad- dle and let L: U- (B) for B with minimal and maximal eigenvalues min; O max respectively.
1/2 Then L is m, Omin?, mmac)) -strict saddle.
Proof.
Write g(U) B. Then L o g, so by the chain rule:
DuL = D gyagle a Dug : 8U + (VI(B = =BB (B
Therefore
VVL(VII -IB"AV((BAA
1/2 min (B
By further application of the chain rule we have
D T. òU,8U D P 8U, 8U)
Suppose Amin| (V2((Z)) < -7 then by the variational characterization of eigenvalues, there exists some 8Z such that (62, (2((Z)8Z) v ~1162112 Then taking 8U = gives
(U(U)O))) = (B' UU
(82, (ZZ)6Z))
S -7118212
S =-ymmin|6Ufi
Thirdly, suppose that ||B' S Ç for some local optimum Zt of Then since B invertible, U = B- a local optimum of L. In addition:
U - = - U'I S o BB - 2'
Finally, consider some arbitrary point Ug. Let Zo B* Then by the strict saddle property for I one of the following must hold:
1. [v((Z))] 2 0 =
|VL(U)) 2 -0mi0 0
2. Amin (V=((Zo)) < -7
Amin (V2 -0min
3. Zo is Ç-close local-minimum z", which implies that Uo is (f -cclose to local minimum B- of L.
By combining lemma B.7 with lemma B.8
we can conclude that our objective does indeed satisfy a (quantitative) strict saddle property. This is sufficient to show that certain local search algorithms will converge in polynomial time Ge al. (2017).
However, this version of the strict saddle property is not quite enough to prove the claim for stochas- tie gradient descent (SGD). Certain extra conditions are given in Ge et al. (2015) to guarantee poly- nomial time convergence of noisy SGD. These local minima, and 2. boundedness assumptions. The first assumption is easy to show in our setting, but the second clearly fails. That being said, we could approximate the objective by mollifying it to be bounded outside large ball. Then it should be straightforward to use supermartingale argument to show that with high probability the sample paths are contained within said ball; and then inherit convergence guarantees from the bounded case.
1. notion of local strict convexity near any
27

---

C FURTHER BACKGROUND AND RESULTS FOR LINEAR AND DEEP CCA
This section contains an eclectic assortment of results regarding CCA. We split this into two sub- sections corresponding linear and deep CCA respectively. To help the reader navigate the sub- sections, we now provide short summary of their contents.
Firstly, we consider linear CCA. In appendix [CI.I we show that the GEP formulation of two view (ridge) CCA presented in the main text corresponds to the sequential, constrained formulation of two view (ridge) CCA that more standard in the literature. Then in appendix C.1.2 we prove versions of cigenvalue interlacing for multiview CCA: these will be useful in appendix D We also consider CCA with tied weights in appendix [C..3] we show that when paired data is generated by applying pair of i..d. augmentations to single original datum the CCA weights can be chosen to be tied; this is useful in appendix D and for the following analysis the deep case.
Secondly, we consider deep CCA. We give full proof that our objective recovers sensible notion of deep multiview CCA (lemma 3.6) appendix C.2.1 Finally, we consider deep CCA with tied weights in appendix [C2.22 and show that when the data is generated by ii.d. augmentations tying the weights can only help with capturing correlation; weight tying can therefore be seen as form of regularisation in this i.i.d. augmentation setting.
C.1 LINEAR CCA
C.1.1 2-VIEW RCCA: EQUIVALENCE OF GEP TO CONSTRAINED FORMULATIONS
The main result in this subsection is lemma C.3 which states that the standard sequential, constrained formulation of ridge CCA (RCCA), and is equivalent to our GEP formulation eq. (6) in the view
case. The proof strategy follows standard argument, from e.g. (Anderson 2003). but is adapted to our notation for this more general case of ridge CCA.
Note that by taking the parameters conclusion also holds for PLS and CCA, which we view as special cases of RCCA.
0 we recover CCA and 1 recovers PLS and so this
First, introduce more intuitive notation for general GEP formulation that is closer to that used in previous expositions. Namely, in the block-matrix definitions of eq. (6) we relabel the blocks
(+d)) == A (6)) Cov xx)))
L == ( (1 (1) Var (0) + ( IDo..
to highlight their nature as (regularised) covariance matrices. With this notation the GEP matrices (A,, B) can be written in full as
Σ(22 0
Σ
(11)
A
(()))
B ==
(22)
0
Ln
We also need the following lemma to help analyse the two-view GEP formulation of RCCA. Lemma C.1 (2-view RCCA GEP recovers orthogonality). Let DxK be matrix whose columns form a top-K K sequence of normalised gevectors for the RCCA GEP, partitioned as (UO), (2))) E iDD (D Suppose the corresponding top-K gevalues (Ak) are all strictly positive. Then in fact x = b for [2].
Proof. Let A diag(A)..... ....., Ax) E RKKK be diagonal matrix containing the top- K gevalues of the RCCA GEP (A, B). Then, since the columns of U form a top-K sequence of gevectors
(122)111) AU = (
> (111)) (()) D (22)
= BUA
A.
(22)
Write M9 T( T x (i for i E [2]. Then each M(0 is symmetric. Plugging this definition
into eq. 22 gives
M A ( T T()) T (12)
(2
28

---

But then because A is diagonal this simply gives the system of equations
Apmil Apm (2)
for k, [K]
In particular, since A > 0, taking / = k yields (2) (2)
bk mak- Taking & * | with X * x the two
sets of equations Agm (1) = Ammii (2) A (1) Agmi
(2) have unique solution mi (2) = mi (1) 0.
While if l + 1 with X = A then we must have ma mii by dividing through by A = X > 0.
(1)
(2)
By combining these cases we can conclude that M M(2), Then by the assumption of B- orthonormality we in fact have
T = U BaU=
Σ U T50000= = M + . -2
so indeed we conclude M = (2)) = / K. as claimed.
Definition C.2 (2-view Ridge CCA sequential definition). A sequence of vectors (w+) parti- tioned as Ugg (n) 1 is top-K sequence of ridge CCA directions if the successive ug solve the successive maximisations
maximize
(12) (0)
subject to u ( T (1)) 1 for [2].
(=(((()) (() e 20(0) (2
(23)
(0) D ui
(4)
0 for [k-1) -1]
where we define [0] = 0 to allow the same formulation to hold for the h = 1 case.
Lemma C.3 (2-view ridge recovers GEP). A sequence of vectors (ux) is a top-K sequence of normalised ridge CCA directions if and only if ( (222 "x) is top-K sequence of normalised gevectors for the GEP defined in eq. 66.
Proof. Note first that in each case there is some pair attaining the maximum because we are opti- mising a continuous objective over a compact set (if (x)( ()) is not full rank, then can work in its range, and treat kernel separately).
We will prove the claim by induction over K.
Suppose that the claim holds for K - Note that sequence of vectors (ux) K is top-K
sequence of normalised ridge CCA directions precisely when (ux (ux))=] are a top- K sequence K-1
of normalised ridge CCA directions and ux solves the program 23 (taking h = K). Similarly, K-1
sequence of vectors k= is top- sequence of normalised gevectors precisely when (u+) k=1 are a top-K sequence normalised gevectors and ug is normalised gevector with gevalue Ak. which is B-orthogonal to the previous {u))/=! Therefore it is sufficient to show that for any fixed sequence of normalised gevectors (equivalently normalised ridge CCA directions by the induction hypothesis) (4x) ) vector ux solves the program 23 precisely when it is a normalised gevector with gevalue λg. which is B B-orthogonal to the previous (ux)
Thus, suppose we given arbitrary sequence of normalised gevectors K-1 We will charac- terise solutions the program 23 By adding in appropriate Lagrange multipliers μ (a 2() R x IR we obtain the Lagrangian
L(uK; p) = up
(1) A (12)
ZHK
1 (1
ux
B
Σ
M
(1) K
B
(10) ui
EE[2]
KE[K-1]
This gives the first order conditions
0 2
A
11) uuk
- 1g
(6)
B
uk
(
Σ
Mi
B( (8)
(24)
[E(K-1)
for foric E [2) and where j 3=3-3 denotes the view-index that is not
29

---

By the inductive hypothesis we have ui (0) B(0
(3) óu for 1, [K 1]. and also that
A A o A
u By feasibility, we also have that uj B B(ú)) uk
0. So taking the in-
ner product of eq.
(2 with ui gives
0 u
(
2 L-
11) A
+0 - m
19
u ui
(
-16
"K
A
Therefore the pair of first order conditions eq.
24
reduce to the pair of equations
u K =
Bl
4 = u B
"K
Next can show that in fact the remaining Lagrange multipliers for the Kth pair are equal across views:
PK HK (1) (0) (
(1) (1)
( ( (2
(2) PK u (2)
(2
(2) PK
So writing μk PK = PK we in fact have the stronger pair of first order conditions
(1)
(2)
A " = uk B (1)
u
"K
1 (1)
ua ((22) (2)
&
which is precisely saying that is a gevector:
0 (A(0)
(
(!
= =KK
0
Ba
(22)
UK
Finally we can up the argument. If
v2 uk normalised gevector with gevalue μuk, then by
lemma C.I u is feasible for the program 23 and attains the value "j" T "Au p (1) "ux μ*K- Since the maximal value attained at a gevector, this maximal value precisely the largest
remaining gevalue, namely Ax- Therefore ux is optimal for program 23 precisely when v2 " is normalised gevector with gevalue Ag. required.
C.1.2 INTERLACING RESULTS
First we state a standard result from matrix analysis. This is simply Theorem 2.1 from Haemers 1995), but with notation changed to match our context. We therefore omit the (straightforward) proof.
Lemma C.A. Let V DxK such that VTV Ix and let M E DDD be symmetric with an orthonormal set of eigenvectors ..,... UD with eigenvalues Xı 2...> Xp. Define C = VT MV, and let have eigenvalues μI ... pk with respective eigenvectors y1 ... J....
Then
ps S A for = K....,K. K.
ifk = > for some lo then has a jue-eigenvector such that Vy is a p px-eegenvectorr of M.
if pk for k I,...,K then V ys is pa-eigenvector of M for 1, L....,K. K.
This immediately gives us related result for generalized eigenvalues.
Corollary C.5 (Generalized Eigenvalue Interlacing). Consider the GEP (A, B) where DxD is symmetric and B TR. DxD symmetric positive definite; let these have B-orthonormal generalized eigenvectors uI.-..,up uD with eigenvalues X1,...,AAy
Let U R such that U BU Ik, define C U AU, and let C. have eigenvalues μn 2 ... 2 pu with respective eigenvectors y1 ......
Then
30

---

Ns I for k
if us = As for some then (C, V) has a p eingeeeeetliisddeeeeeeetttrrr y such that y is Us-generalised enteraalieedeigevvectoroo of (A, ().
if Ui for for I,..., K then Uys is a pu-generraliccdeeieeeeetor of (A, B) for k = L=...... K.
Proof. As in previous appendices, convert from the GEP (A,B) to eigenvalue problem for 1/2 A/'A//B/ 1/2 by defining V B BU, and ny B Bua
M == B-
We now check that the conditions and conclusions of lemma biject with the conditions and conclusions of this present lemma.
Indeed (ua)a are B-orthonormal gevectors of (A, B) and only if (va)a are orthonormal evectors of M; the matrices C' and then coincide and so does its eigenvectors and eigenvalues.
This proves the result.
We can now apply this to the multiview CCA problem, generalising the two-view case.
Lemma C.6 (Interlacing for MCCA). Let (x be random vectors laking values in R.D
respectively, as in section Take arbitrary full-rank weight matrices UG) E RD x for i
(1....,I) 1} and define the corresponding transformed variables Z (U()) X(0). Then we have the element-wise inequalities
MCCAk(Z ,....y 2Z)) < MCCAA(((), 3...,, (',,,)
(25)
Moreover simultaneous equality in each component holds if and only if there exist matrices y IK x for E [1] such that the (000@)) are a set of top- K weights for the MCCA problem.
Proof. Let the matrices A, B be those from the MCCA GEP in eq. (6 defined by the input variables X. By definition, MCCA ((),, is precisely the vector of the top-K such generalised eigenvalues.
Then the corresponding matrices defining the GEP for Z are block matrices A, defined by the blocks
1 Cov (2()) z(0) U(t) Cov(x1), (()) U6) (6))
(26)
( = = u()) VVa((X(i) (V U(()
Now define the D (K7) block diagonal matrix Ö to have diagonal blocks ( Then the definition from eq. (26) is equivalent to the block-matrix equations A = Ü AU, B = Ú BU, both in
((KT)x(K7) Finally, we define normalised version ú = B-2 (possible because B positive definite and Ü of full rank).
We can now apply the eigenvalue interlacing result of corollary C.5 to the GEP (A, B) and B- orthonormal matrix Ú DxiK Let the matrix B/2 AB- UTAA AÚ have top-K eigenvalues P1 2 2 PK with respective eigenvectors y1...., УK- Then the (px) are precisely the first K successive multiview correlations between the Z As before, the first K successive multiview correlations P between the (+) precisely the first K generalised eigenvalues of the GEP (A, B). We therefore we have the element-wise incqualities Ps S pt for each k = 1, K.
Moreover, equality for each of the top-K multiview correlations implies that Ún is a generalised- cigenvector of the original GEP (A, B) for k 1.... K (still by corollary C.5). Letting (i)
(8) then gives the equality case statement. DK
31

---

C.1.3 CCA WITH TIED WEIGHTS
is intuitive that under certain symmetry between (Xx(), x(2) that the CCA weights might tied,
i.e. Mik (1) = 1/k for all k with Pk: > O. One natural sort of symmetry to consider is exchangeability, that is that (x(1) x (x(2), x()). However, exchangeability is not sufficient to guarantee tied weights, as the following example shows.
Example C.7. Let x be a pair of I2² valued random vectors with
Var (x(1) la for [2], Cov (x11) ( ))
Then one one possible choice of canonical directions
(uf 21 (2)),, (u2 (1)
2 (22,
where 61 = =
are standard basis vectors.
In fact, the space of canonical directions is degenerate (due the repeated eigenvalue of p) but can use standard results to characterise possible choices of canonical directions. Take any set of (8) canonical directions Uk for & E [2]. [2]. Let U be matrix whose colunns are the first and second canonical directions for the 2th view, for i E [2]. Then these are of the form
U (0 9) O, U = (i 0
where O 122x2 is orthogonal.
(x) be an exchangeable pair of random vectors, each of full rank. Sup- pose that (u(1), ((2) are pair of canonical directions with (L + =2 and canonical correlation > 0. Then Cov (X),, (1)( x22) has strictly negative eigenvalue.
Lemma C.8. Let (x(1),x(3))
Proof. Then by the GEP formulation of CCA, we must have
(22) (1)) (1) (2))
- -x(), Then In the exchangeable setting have {1I) I(22), {(12) I(21), Write A (22)-
we can combine the two previous equations to see
(12) w()) (2)(1)(((()) (2)))
and so
A A - 50
Therefore, when p > 0 and E is full rank then this is strict inequality and so the cross-covariance matrix must have negative eigenvalue.
Conveniently, under the data generating process commonly used SSL, this cannot happen!
Lemma C.9 (Generated by augmentations). Consider pair of random vectors (X(1), (2)) gen- erated via
( Px
()) ~ G independently, for 1,2
(27)
( (x for i 1,2
(1) then any canonical directions u u with Ps: > 0 must satisfy (1) u
Moreover, for any K D11), there exist a full set CCA weights (U, U) with U x
32

---

Proof. Write g(x) Byng(g(x)) Then by the tower law
( E (X ((X 1E (o(x!0)) E ( Var (o(x
(g(X )x(x( (
so is a symmetric positive semi-defiiite matrix.
Then lemma (1)
(2) C.8 immediately implies the first conclusion, that uk ukk with Pk > 0 must satisfy
For the final conclusion, recall that constructing CCA directions (u " )) is equivalent to singular value decomposition of T (a((X()))//Co((X generative model we have Var(XX)) = Var(X( in fact the target matrix T is symmetric positive semi-definite. Therefore, we take an eigen-decomposition of T. this will also give a singular value decomposition, and so mapping back through Var(X(U))-/ will give a full set of
Var) (x1))=+/ (C,
Under the
CCA weights UU,U) with U E (RDD .as claimed.
C.2 DEEP CCA
C.2.1 ECKHART- YOUNG LOSS RECOVERS DEEP CCA
Proof. Write (x() ((0) U T g0) (X )()) where the U+)) are matrices parameterising the final layer and (l defines the representations in the penultimate layer.
Because & is local minimum of LEx(#) must have U local minimum of the map / U CEY((U, é)). Writing y g(;;) () for the corresponding penultimate-lay representations we get
((U) (EY((U, 6)) trace
Cov(U (oOOTTy (0) UO) (0,,T0) (()))))))
Σ (ac(UTp(0))))
= trace
(UT A(Y))) + 1IU BYYUUR
where A(P), B()) are as in eq. 66) with X replaced by Ý. This is precisely our Eckhart- Young loss for linear CCA on the Y. So by proposition 3.2 t must also be global minimum of ((U) and then by proposition 3.1 the optimal value is precisely ||MCCA (
This in turn is equal |MCCA* by simple sandwiching argument. Indeed, by proposi- tion 3.1 miny LEY (VV xx())) = - (2) Then we can chain inequalities
- |MCCAA ( ( L2x 2 min LEY LEY((V(()) (=
V
2 min Sax Ce Tp())) = - (IMCAAx
to conclude.
C.2.2 DEEP CCA WITH TIED WEIGHTS
Finally ready analyse the Deep CCA with tied weights under this augmented-ppainsoo-ddaa assumption.
Proposition C.10 (Deep CCA tied weights). Consider a pair of random vectors generated as in
eq. 27). Let (f(), f(2)) ) be a pair of functions optimising (a function space version of) our Eckhart- Young loss for population Deep CCA
(f(), e arg min LEY ((0
(28)
33

---

Assume that F. is a class of functions f DR IKK closed under left-composiiion with linear maps (e.g. F defined by varying parameters of a neural network with a final linear layer): i.e. FEF.ORKXKK = -0ofFF.
Then in fact the Siamese network pairs minimal value. Moreover, there is constant vector c E RRK such that
and (f(z), f(2)) must both also attain that same
Egru [ (g(x()))
x c+ Byng | (g(x |x(0]
a.s.
(29)
Proof. For the rest of this proof (and only for this proof) define the matrix-valued functions C' F + x and V F + RgXX by
c( (20)
Cov ( (x)xx))
v()) Var ()(X(2))
The Eckhart-Yourr loss can be written in terms of these functions as
K
(er(f(),
Σ Che(f(), + (I((F))+
Firstly, we investigate how the between-view covariance term transforms under 'tying" the networks for each view, for general choice of f(), f(2) (not necessarily optimisers). We will decompose the covariance terms much like in the proof of lemma C.9 Write
f(g(z) Egru (FO) ggxz)
Then by Cauchy-Schward Schwarz we have
Ck ((0) = Cov (f()) (x)( ( (x(0))
= Cov
g(x(0) 1
)(xx000))))) 0(x(0))
Var fhr 9(x(1))))) Var f ((x)1))) 1/2
(30)
Caa (
(() CRa (5(2), }())
1/2
and further application of Cauchy-Schwaad this time on RK followed by AM-GM inequality gives
Cua < Σ Cn)( payy) (2)( 12)))
Σ Eamlrr)roo) 1/2 (m
1/2
$ Caa((t (()) + C
(31)
Secondly, we investigate the within-view variance terms at the optimal f()), f(2) from the proposi- tion statement. This is where we need to use closure under linear maps. Note that for any matrices (1)), (2) E KxX we have T f(I), (2() T f(2) F and we recover the original f(i,, f(2) by taking (1) (2) = IK. Because the f() are optimal in eq. (28) we must have
(Ix, Ix) E arg (EY(U() T ((),((2) f(2)).
This loss term can be expanded out as
((P2) U I (v(jet)
-2 trace T
0 (crfio, f(D)
v(jen) UIP
O
34

---

to show that it is precisely the Eckhart- Young loss for linear CCA on the representations f()) (x)())
Therefore, by lemma C.I we have at the optimum that
(KV()()) ( (
i.e. = -
(32)
Finally, we combine these two results, eq.
31 and eq.
32 to conclude:
K
LEV (xx) = -4
Σ
Cku (f()) f(2)) + ( V
km1 1
-4 x
Σ Can(f(), f() + Σ CRa(f(2) f(2))
+ 2 (12V(f())12>> + (12
CEY (EY( ((2)) + LEY (f2)) f(z)
Since we f()) f(2) were constructed to minimize LEY this immediately implies that CEY CEY(F(), f()) (Fx()()), f(e) LEY (Ex())) f(2)) attain the same minimal value.
Moreover, by chasing back through the inequalities, equality implies that there is equality in eq. (30 for each [K]. These equalities directly imply eq. (9
Remark C.11 (Extension to other formulations of Deep CCA). We note that this proof technique yields very similar results for other formulations of Deep CCA. Firstly, it can be applied to VICReg (interpreted Deep CCA method, be viewed between- netween-view-cone view-correlation reward term of the form Σx (f f2)), while the within-view covariance matrices are equal at any global optimum by lemma D.6
appendix D.3.3: ; this is because the VICReg objective can
The technique could also be applied to variants of Deep CCA where the within-view covari- ance matrices are constrained to be identity, but different correlation objectives are used, such as En ( for p > 6
'Such formulations do indeed recover classical CCA in the linear setting by Wells et al. 2024 (Section 5.1].
35

---

D RELATIONSHIP TO VICREG AND BARLOW TWINS
D.I INTRODUCTION AND LOSS FUNCTIONS
To compare formulation the VICReg and Barlow Twins methods, synthesise notation from the main text with that from the original works. Consider pairs of random variables X(1,, .x which we think of as pairs of augmented input data (e.g. distorted images). Now consider pairs of embeddings z() ((X()), z f Define the covariance matrices
( ))), ((222) CZr( C(12) Cov (()))
(33)
Throughout the rest of this section, as in the main text, we use K denote the dimension of the embeddings (and so also the dimension of the relevant covariance matrices).
Our SSL-EY loss can be conveniently written in this notation
LSSL-EY = trace (2((()) + ((2)) + (C(1) + ((
K
(12) lk
(C
+
+
i,j=1
It may be interesting to compare this to the formulations of VICReg and Barlow twins below.
The main aim of this appendix is show that these techniques are equivalent CCA in the linear case. We present a complete argument for VICReg in D.3 and a partial picture for Barlow twins in D.4] To facilitate this analysis, we first introduce certain notions of decomposition of a pair of weights into the subspace they capture and their component non-orthogonality in subsection D.2
We next state the loss functions for VICReg and Barlow Twins with this synthesized notation. We warn the reader that throughout this section we use LvR, LBT to denote the VICReg and Barlow Twins loss functions, but these may take different arguments depending on what parameterisation are using; we hope this simplifiedloverloaded notation will improve clarity.
D.1.1 VICREG LOSS
The VICReg loss is often written (Balestriero & LeCun 2022
Ove = - - 12
Σ
Var(z
+ B
Cov(z,", 20000
1E(1.2)
where (=+:= 10xx=max(;50) 0). All of these quantities be written in the notation of 33
Zl 122) = trace (c()) + ((22) 20(12))
Var (2(1)) Ckk
Cov(Z ((()) CHI (it))
So in our unifying notation the VICReg loss becomes
Lve 2
K
(12)
Σ
11
Σ
K
(1
a (0(1-
((+)
BCh10² + c,10
A=1
(E(1.2)
E=1l
=-2y trace (12)
+
Σ lva(C(1))
(34)
(E(1.2)
where we define lVR RKXK
R by the expression in the first line. This extra notation will be
helpful appendix D.3
We now make some observations. Firstly. like in Deep CCA, this objective only depends on the covariances between z (22) from (33). Secondly, the first term can be thought of as reward, and the second as penalty. Thirdly, this reward term only depends on the covariance between 2 2(2), whereas the penalty term only depends on the variance matrices of Z(1), z22 respectively. These observations provide key motivation behind the argument in appendix D.3
36

---

D.1.2 BARLOW TWINS LOSS
The Barlow Twins loss is usually written in the form
k CC) = Σ (1 CHa)² + Σ Ch
lepi
where C= Corr(Z(1) 2 is the cross correlation matrix. Note that this objective is independent of the scale of the columns z Z particular. for any optimal solution, we may pick an equivalent optimal solution such that each entry of 2(1), z has unit variance. We can therefore
write constrained form of the Barlow Twins objective using the covariance matrices of the last two subsections. Namely
LBT Σ0- k=1
c( +B) -(-CH4)
(12)2 light
+ +1 [cft -c -- W).,..
(35)
where we use 1 as in the convex optimization literature to give the formal value of oo when the constraint is not satisfied, and O when the constraint is satisfied.
D.2 SURSPACF-ORTTHOGONNEN
DECOMPOSITION
The analysis in the rest of this appendix considers both tied-weight (Siamese) and untied-weight settings. In this subsection, first consider the broader, untied weight setting, then apply this result to the wiight setting.
D.2.1 UNTIED WEIGHTS
Since shall only work in the linear setting, each method defines linear transformations corre- sponding to a pair of weight matrices ((),, B(2), where the embeddings are given by
z B(0) T ( for E {1,2}
We state three different ways venient analysis; these are all essentially the same, but differ their treatment of low-rank weight matrices. Our VICReg analysis needs formulation 2, our Barlow twins analysis needs formulation 3, while we also state formulation 1 for the sake of completeness.
re-parameterise the weight matrices B(() for
Lemma D.1 (CCA basis for subspace). Suppose that for each the components of x are linearly independent. Let B(),, B(2), be arbitrary of weights. Define R(() rank B())
) for i 1,2. Without loss of generality (WLOG), suppose that R()) < RR(2) =: R. Then the following three formulations hold:
1. Both T() of full rank but possibly different heights: There exist Ub) E Dx ,7() E R 1 K for i 1,2 such that B(t) UGT((, each T() is full rank R0), and
Uni Var (x()) uli) -Ro fori e [1,2}, U Cov x(0) u =A
(36)
where A E RR( x((2) is diagonal matrix of canonical correlations for the subspace of trans- formed variables.
2. At least one full rank T and same height: There exist matrices U(t) E DDR T(o Ra for =1,2 such that T(2) is of full rank R. B()) = T() for euch i, and
T( Var (x())
T =
for i E (1,2), T Cov (x"), x(e) U(2)= = =A
(37)
where A RRRR is diagonal matrix (of canonical correlations for some augmented subspace of transformed variables).
37

---

3. Square T() not necessarity full rank: There exist matrices U ER ,D() E R for i == 1,2 such that B(a) T() for each i, and
U(A Var
U00 =I for i (1,2), U Cov (xx), U (10)= A
(38)
where A E ERKK is a diagonal matrix (of canonical correlations for some augmented subspace of transformed variables).
Proof. The key technical here is deal with possible linear dependence amongst the columns of the B(0) matrices (i.e. when R(6) < K).
each i, take subset 7 the the indices [K] such that the columns B(g are linearly independent.
By linear independence of the components of X(O,, the random variables 2 are also linearly independent. We can also write
B
( f00 M
(39)
where M R xR x expresses the columns of B(a) in this column basis.
We now construct 'augmented" weight matrices B(0) depending on which case prove.
For Case (both T of full rank but possibly different heights), do not perform augmen- tation; define B =B() rCB
For Case 2 (at least one full rank T()) and same height): for each i, if R() < then augment via the concatenation
B
B 100
B(T)
where (
are additional columns such that the resulting R transformed variables are linearly independent.
x )
For Case (square ( not necessarily full rank): for each i, if R( < K then augment via the concatenation
B
B(D) E R D
xK
where D E x are additional columns such that the resulting K transformed variables are linearly independent.
We return to considering the three cases in parallel. Define 701 B Tx()) for 1,2. In each case, write 8(i) for the dimension of ((); (ti) R(O) R, K in the three cases respec-
tively. Perform (linear) CCA the pair of random vectors (7 2 This gives weight matrices v E s S and the diagonal matrix of correlations 5 such that the transformed
representations (i)) (+) have identity within-view-covariinn matrices, and have maximal corre- lation between views, i.e.
(+) Var ( v( Iggo for i E (1,2), vau Cov (ZW, v(2) = A
(40)
The (t) are (therefore) of full rank, we can define
T ((0
7 -1
which gives us
B = 707
(41)
38

---

And therefore also that B B( .R
U -000 Finally, define T( T M RRAXK
Then by substituting in eq. 39 we immediately recover that B U(iT(() Moreover have that T U' Cov (x()xx0) (t) v() BOO Cov (x())xx0))) B
= =V() Cov 2 2
so applying eq. (40) yields the claims of eq. 360. eq.
45 and eq. 38 respectively.
Remark D.2 (Degeneracy). In Case 1, the matrices (+)) are effectively determined by the B(0,
:/(6)) are precisely the canonical variates from applying CCA to the pair of subspaces span ( span 2 Therefore, by the linear indepen-
Indeed the transformed variables U7(1) X = yバ T
z
dence of the original variables, any degeneracy corresponds to degeneracy in the CCA solution.
In Case 2 and Case 3, we longer have uniqueness of U, A, because the choice of augmentation for (1) was arbitrary.
D.2.2 APPLICATION TO SSL LOSS FUNCTIONS
The power of lemma D. is that the covariance matrices ((+)) can be written as the following simple functions of A and the T
(r()) [TOOTTo TO) T())) ATG) if
((+)) T UL Cov (x(0, (00)(()
if
(42)
are functions of the ((3) so can also be written as (fairly) simple functions T, A. As in the proof of lemma D.1 let the corresponding A have dimensions 5(1) x ((2) where (()) takes value R(() R. K in the three cases respectively. To simplify these expressions we introduce the following notation for semi-inner- -inner-productlikk bi-linear forms with respect to the matrix A TR x(()) which generalise the Fuclidean inner product for pairs of vectors m(()) (t)()) and Frobenius inner product for pairs of matrices M E | respectively:
Both our loss functions 34
35
(m!2), m ( = mL)) Am(
(M M( > (race(M())) (2
With this notation we obtain the expressions
(v)( T(2); A) = 22 (1) T((2) )
Σ (T(()( ((())
(43)
iE(L.2)
K
(, ((1) Σ (1 ,())))
> Agl
(TT111)))) + (172)) [2-1 ke(k),Kk))))
(44)
D.2.3 TIED WEIGHTS
Lemma D.3 (CCA basis for subspace, tied tieghweights). Suppose (2) are generated by the
daiu-gend datu-generating
generating mechanism from eq. (27). Suppose we have tied (but otherwise arbitrary) weights
B() B(2) = of rank R S K. Then the the following two formulations hold:
1. T of full rank but not necessarily square: There exist matrices U E RDDRR T PERRE Rx such that T of full rank, = UT and
U
Var Var
(x())U U for E (1,2), U Cov (X()
U=A XA
(45)
An inner product is typically defined as satisfying positivity assumption, which is not be satisfied if A is of full rank.
39

---

where A E RRRR is diagonal matrix of canonical correlations.
T square but not necessarily full rank: There exist matrices E RXX B UT and
" , E RKXK such that
UT Var U for i {1,2), U
UT Cov (x()) U U=
(46)
where A RR is diagonal matrix (of canonical correlations for some augmented subspace of transformed variables).
Proof. We only give sketched argument here, because the construction is almost identical to the proof the untied case, lemma D.I (just drop the superseripts and apply the symmetry).
Take subset I C [K] such that columns B, are linearly independent. Let the matrix M be such that = By M. in Case 2 (T square but not necessarily full rank) and R < K then augment Bj with extra column to form full-rank B RDD x otherwise just set B Br.
The key observation is that the random variables 2 = BT x also follow data-generating mechanism of eq. (27). but now with different set of augmentations simply defined via ((X()))= = B BT ()))). Therefore, by lemma C.9 we can pick symmetric pair of CCA weights (V, V) for (2(), 2
We
wrap up loose ends following the proof of lemma D.I Define U = BV, T
V-1 T = M. The argument then concludes by analogy to proof lemma D.1
In light of this result, we introduce short-hand for the versions of LVR, Let arising from tying the T-weights in eq. (43 and eq. (44). Simply write
(vR(T; (iR(T;T:A); ÉBT(T;A) = (IT(T,T;A)
(47)
D.3 VICREG ANALYSIS
We are now ready to prove that VICReg recovers CCA the linear setting; we consider both a general case with untied VICReg weights and special case where the data is generated by i.i.d. augmentations as in eq. (27) and the VICReg weights are tied. each case, we prove that the sub- space of random variables generated by the VICReg representations correspond CCA subspace, though this subspace might have dimension strictly less than K.
The tied weight case with i..dd augmented data becomes straightforward with the decomposition into T,, A from the previous appendix D.2.3 The key is to note that when I' is of full rank the reward
trace T TTAT- Σ ATT
is strictly increasing in each Ay. Therefore, the loss is minimized by maximizing each entry of A, and so, by eigenvalue interlacing, we must recover the CCA solution. We give full details in appendix D.3.1
The untied weight case is more challenging, but reduces to the same computation. We apply Case of lemma D.I then use a symmetry argument to show that the resulting T(1) T(2) can be tied, and have the same rank. We give full details in appendix D.3.2
Then in appendix D.3.3 we address the final two bullet point claims from the main text.
Finally in appendix D.3.4 we present simple computation to show that will expect VICReg to collapse (even in this linear case) for a wide range of tuning parameters.
D.3.1 TIED WEIGHTT
Proposition D.4 (VICReg with linear, tied weights recovers CCA under i.i.d. augmentation set-up).
Let (1), (2) be random vectors in TR D(L) generated as in eq. (7 with strictly positive top-K
40

---

canonical correlations. Consider applying VICReg in the linear case with tied weights. Let B be globally optimal weight matrix:
B
arg Be
argmin min (Bx(BT x x
(48)
Then there is some E K, some T RRRK and (tied pair of) top- R optimal CCA weights (Ü,ū) such that B -ŪT.
Proof. By lemma D.3 there exist i E R xK í E BRRK such that B = UT, T is of full rank and that eq. (46) holds for the 'hatted' matrices Ú.T,A.
Now let (Ü, Ū) be a tied-pair of top-R CCA matrices; construct the corresponding VICReg weights B = UT. This gives the inequality
CVR(B (()), BT x(2))
(VR(T, 22
Σ
1 21vR( (T))
R X | 1 21 (T = EvR (
2 2y
T. (f,,,TTT:.
Cvu BT x(1), (1)) x (49)
Where inequality eq. 51p follows from CCA interlacing lemma C.6 Moreover, because Í is full
rank, there is equality if and only if X, = A, for all E [R]] by the equality case of CCA interlacing, only happens when (Ó; () define top-/t CCA subspace for (X x(2)).
D.3.2 UNTIED WEIGHTS
Proposition D.5 (VICReg-CCa equivalence). Let X x be random vectors RDD(1) .R. respectively. We consider VICReg, and CCA in the linear case; iee. where z 2(2) are linear functions of X(1,, (²), Then the set of optimal subspaces for VICReg corresponds to the set of optimal subspaces for CCA.
In particular, for any optimal VICReg weights
())) e
arg min
(VT(B()TX() B ((2)Tx(())
(50)
( 2o00) 2009) x
there there is some R < K, some i E RxK and top- R optimal CCA weights
(()))(2)) E
arg min
(UCA(UU)) ( ((2) x(2))
UW) wwW)wl Roll) x
such that B() = (0) ( (2)
Proof. Take (0, Å as in Case lemma D.I
By considering alternative sets weights of the form B ( for arbitrary T()) R condition eq. (50) implies
LVR (
arg arg min TO RO
Then by applying lemma D.6 (below) to the form of ΣVR in (43 shows that (v f()); A) = LvR (11)),)))); A).
We next construct corresponding set of VICReg weights spanning an optimal CCA subspace.
Let (1) be pair of top-R CCA weight matrices, with corresponding R R diagonal matrix of
41

---

canonical correlations Å. Then construct the VICReg weights by B ((())( This gives the chain of inequalities
LvR(B() X(1), B(2) x(1) LVR (VR(T((2, (r((2), A)
n1
dn 1T,-0 1|2 + 21vR ( (T T
-2
pml
R
2 27
| 1|2 + 21 ( TT((2))
(51)
(V (T(2) A) = (BR) x B(2) x
Where inequality eq. 51 follows from CCA interlacing lemma C.6
moreover, there is equality
and only if 7, = x for all r E [R], which by the equality case of CCA interlacing, only happens when (1) define a top-It CCA subspace for (xx2), xx))).
Finally, since the top-K canonical correlations are strictly positive, these equalities imply that X > 0 for all [R]. the 'moreover' claim of lemma D.6 us that in fact we have T, T for r and therefore that 71)) f î. as required.
Lemma D.6. Consider minimizing loss function of the form
L(T(L), ( = -2(T(1),,T2)))))) + s(T())) + S(T(2)
over T(), T(2) RR where A E RxR is diagonal with entries and f RRRK is some vubitrary,function Let T a pair of matrices minimizing this loss function. Then we have
(()0)(00) L((()), f((2) C(T(0),(())
Moreover any such pair of minimisers must satisfy T. f for all indices where X, > O.
Proof. We show that for any pair T(1) T(2), L(T(1), T(2)) 2 min (C(T()), L(T(2), 7(2))) By expanding out the matrix inner product
-2( (() f(T(1)) f(T(2)
(1T(1) 1 - ||700 ||3 || f(T(1)) + (
-1((()) 1 1 2 ((r("),T'1) + (2T r(11))
2 ( T())) + 2(T(2), T(2)) ((!ie
2 min (C(T()) T()), ((T(2) T((2)
where the final line used that for any a, E R, (a + b) 2 min(a, b). Equality in this final line implies the losses are all the same. Equality in the penultimate line shows that the th rows coincide when A O.
D.3.3 INTERPRETATION AS DEEP CCA
For each K N. define k [0, 1 + by
$ (
min LVR(T (T: diag(A)). TERRXK
(52)
Lemma D.7 (Minimum is attained). The minimum in eq. (52 is always attained; in fact, for each given X there exists T with columns [| vkk E [K] such that &x (A) Lve (di((; (A)).
42

---

Proof. Define the function & R - R mapping arbitrary T R R to a shrunken copy whose columns all have norm less than or equal to I and defined by
(+())) max max(1ITk] (1Tkk]]k Tk
Then (T)) is contained within the set
T: T-= TERRXK
|Taall2 1 for [K]}
which is a compact subset of IR xxK (w.r.. the natural topology c.g. generated by the Frobenius inner product). And for any X E [0, 1] comparing term by term, and writing T = (T)) we have
AVR(T:A) = Σ
1 1 +
Σ
(Th, Til + a a1- (1-1
2 v >
| -
Σ
(T,, a (1-1(Tull2),
(53)
= (vR( (T);
For any given X, because [vR(=;\) is continuous function on the compact set T, attains its minimum on T at some / E 'T But then for any T E R LVR(T; 2 (vr(w(T); 2 CVR(T; X) by eq. 53); So I' is also a minimiser of (vR(=:A over the whole domain R R as claimed.
Proposition D.8 (Φx relates VICReg to CCA). We have
1. Ø is element-wise decreasing in X.
2. For X(1,, (2) generated by i.i.d. augmentations (eg. (27 ). we have
CvR(B* ( B x())) = - (CCAR(X())x(()) BERDO)x
min
3. For general random vectors ( E D ( DD! have
min B x (l) ((), ( ( DK (CCAK(X",,(*)
Proof. Let A E [0, 1k be fixed, and let T' be a corresponding minimiser from lemma D.7 Then
1. Take any [K], (Ax, 1] and fill the remaining entries of the vector X by X A for / E [K] \ {k). Then,
xX) S (vR(T:N) 22 Σ NellTr. 21 (T
< 27
>111Tr.112+ + (P = LVR(T;A) = xx((2)
required.
2. This follows directly from the proof of proposition D.4
3. This follows directly from the proof proposition D.5
43

---

Interprefation: these results can help understand the deep case, provided that there is final linear layer in the representations. We consider the untied case for now, and leave the tied case to the reader. At any global optimum with corresponding representations ((,, because there is final linear layer
LVR (2())
CVR (VR(B())TZ()) B()))
Φx (CCAk(Z 2(3)
min B BO
This is very similar to result for Deep CCA lemma 3.6 but with px (-) in place of 11-12- Note however, that xx need not be strictly decreasing in each argument; indeed it will be constant in arguments where the corresponding row Tī. is zero. So (deep) VICReg may also learn low-rank representations. In the next subsection we will show that this phenomenon is in some sense 'generic'.
D.3.4 COLLAPSE EXAMPLE
In the previous subsections, proved that VICReg recovers optimal CCA subspace, but its dimension was allowed to be smaller than the target dimension K. We now show that it is possible to have R K for wide range of choices of the VICReg penalty parameters a, 0,,37 We consider a very simple case of learning representations of dimension K = 2 that collapse to give representations of rank R 1. By lemma D. is sufficient to consider tied T T(2) T.
We proceed in two steps. First require technical lemma, lemma D.9 to show that the columns of an optimiser 'cannot be too small'. Then we use a quantity from this lemma in proposition D.10 to construct a broad range of parameter values for which there is collapse.
Lemma D.9. Let A = diag( A1, 22) with 1 > A A2 2 O. Consider minimisers
f arg min LvR AT; TER2x2
of the VICReg loss with parameters a > 0. Then there exist a constant μ pla, 7: such that [|Tx|lz 2 for any minimiser f.
Proof. Main idea: First construct good I of restricted diagonal form. Second, show that any T with a very small column has higher loss than this good choice of T*
First consider optimising the LVR over T restricted to be of the form
m1
T
(54)
0
Then for mı; m2 S 1 because the off-diagonal terms of ((+) are zero the orthogonality penalty is zero and so we have
(T)(T:A) = 21(m A1) + m2 (1- - >2)) + 2 (a(1 - m1) + a(1 m22) + 0)
So write
fa(m)): = m - >x) - maaa
(55)
to simplify
(T; 2a + (1(1)) + f2(2).
Then by completing the square we have
2 a²
fa(m)) 7(1 -xx) mm 2x(1 2x)
a
2>(1 Ax
and therefore fie has the unique minimiser
m == arg min [a(ma) = min
<(1, 7(1 - Ax)
a
27(1 Xxa)
my E(0.1
44

---

and moreover, at this minimiser certainly
Ja fa(m))
mi m
7(1 >x) <
7(1 Ax)
m 0
0
Therefore take T fa(m)) over diagonal 7'
to attain the minimum value [T* 2a + fi(m)) +
mz)
Second, consider an arbitrary T of form eq. 669. Since the off-diagonal penalty (β-term) is always non-negative
(vR(T:A)
ym (cos² 0x(1 X1) + sin? 0x(1 X2)) + Σ a(1 m.) +
2 2a Ji [i(mi) fi(2))
2 2a - fi(m)) + fi((2)
Now we can construct
pla, () min mi, f2(m))
(56)
then for m2 (4) we have
fi fi f1 2 oo f2(m))
where the first inequality follows because fr is strictly decreasing on (0, mī), the second follows immediately from the definition of fi in eq. (55), and the third by construction of in eq. (56).
Finally conclude that
1 CvR (T; 2a + fi(m)) + f1(m)) 2a fi(m)) + fe(m)) (VR(T";A)
A)
so T cannot be global minimum when m2 E (0, p).
Proposition D.10. Let A = diag(A1, X2) with > \1 > X2 2 0. Consider a minimiser
ñ arg min EVR AT; TER2x2
of the VICReg loss with parameters 3,,877 > 0 satisfying
23 r() 22)
(57)
where μ pla, A) gives a lower bound for the column norms [|Txs|] in eq. (56 above.
Then the bottom row of is zero: (11 = f k = 1,2. 2k
Proof. Any T E p2x2 can be re-parameterised to the form
T (mi 0
m2 cos 02 sin 0g
(58)
sin 0
Then we show that for any such T,
EVR 2 (T' where T"
m
m2
with equality if and only if 0, = =
First note that for T of form eq. (69) we have It will be convenient to rewrite the original VICReg loss eq. (34) to separate back out the terms depending on each penalty parameter
K
LVR 7
(C(1)
(11)
((22)
20((2))
Σ
Σ
(i8) Ckk
+
1
Okl
(1))
k=1
(1,2))
light
45

---

which gives
(<E
|T-I+++++++++ + +B > ETT
h
=2 I(T,T)--A+ + 3
(T T,)² + a Σ (1 [TTx11++
(T
leght
Indeed, we can simply expand the difference; write A(0) ^1 cos 0 + X2 sin" 0.
(ÉvR(T;A) (VR(T": = ((T, (T',T"))_))++ Σ
(Tix T.)2 (1))
m {(1 x((Ok) - (1-11)}
7
+ 23 m1my (cos f2 cos 02 + sin f2 sin 01) -1))
where the a terms vanish because | = my is preserved. Note that the first term is positive, but that the second term is negative. We will show that in the regime of interest, the magnitude of the negative term is small and so the net contribution positive. Indeed, we further process the terms separately
(1 - ((O)) - (1 A1) = -((0k)
(A1 >22) sin 0₂ 2 0
while by standard trigonometric identities
(cos 01 cos 0g + sin 0g sin 0,)
cos? 2 (01 02)
sin 2 (0) - 02)
(sin 0₃ cos 02 sin 02 cos 02)²
< (sin 0 + sin 0₂)
< 2(inn 0, + sin 02)
We can now put these inequalities into the previous step and use the fact that μ S m1, m2 S 1 to
get
2 (BvR(T;A)= EvR(T";A)) 2 27 (sin 01 + sin? 02) - 23 x 2(sin2 01 + sin 02)
= 2(sin 01 + sin 02) (x -22 2/3)
so indeed, this difference is strictly positive provided 01, 02 are not both zero and eq. (77 holds, as required.
D.4 BARLOW TWINS ANALYSIS
D.4.1 TIED WEIGHTS
We present single result whose proof is complete apart from application of conjecture B.5 A key tool for the partial proof is lemma D.12 which may give the reader better geometrical intuition for the Barlow twins loss.
Conjecture D.11 (Barlow twins tied weights). Let X!), (() be random vectors in RR (2))
generated
as eq. (27). with strictly positive top-K K canonical correlations. Consider applying Barlow twins in the linear case with tied weights, with weight matrix B such that 7 = B X for 1,2. Let B be locally optimal weight matrix of rank R < K such that CCAR(B (1) in each component. Then B defines CCA subspace of rank R.
46

---

Note that in this tied-weight setting, T(1) - T, the Barlow twins loss be written
(BT(T:A)= Σ k=1
[TA + Σ lep
(T. T.)A +
(59)
Lemma D.12. Let S K S S be integers and let A diag(>),..... Xs) be S S diagonal
matrix with elements O E [0, E [S]. Let T E IRSK be a stationary point of R
R. T + CBr(T; A). for each r E [S] we have
(T;A)
Ap
LaTT
(60)
Qdr
where
L3 (1 CkkC -3 E
ch and, as before, CkI TV ATi
(61)
Lith
Moreover, if in addition T is local optimum of I and 1 > A1 2 ... 2 Ax > 0, then La > 0 for all
k E [K].
Proof of lemma D.12
To clean notation for this proof we will write T3 T. (for columns of (drop the superscripts). First, compute
T) and Osk == (
ÖLBT (T: 0%,
1)T3 + -(c -
Σ CUTATH
(62)
lepi
Our proof idea is to use the first order conditions from the Lagrangian formulation of 59 show that the right hand side of this expression is less than zero.
The Lagrangian corresponding to the constrained program 559
K L (1 1)
LT L; = Σ 1 - [T-I))IR) +
Σ 10
( 1.)
+
where L E IK is the Lagrange multiplier.
Now let T be any stationary point of (eT(T; A Then this is a stationary point of (59 must be some Lagrange multiplier for which satisfies the first order conditions
so there
dC (TeT(T,TT L:A) 4(CRs 1)AT, + 43 Σ CRATI + 4L
Rearranging gives
LaT: -1-Caa)ATi 3 Σ CUATI
(63)
now take inner products this vector equation with judicious choices of direction.
63
L+T, (I-CCR)A,TTR -B -B CRiNTT
(64)
Lityk
T
63
LA = (1 Ckk) - Σ Ch
(65)
Observe that (64) looks a lot like 62 but involves the Lç nuisance parameter, while 65) and might help us control on this nuisance parameter.
Note that optimiser exist because the objective continuous and the constraint is compact.
47

---

Plugging 64 into 62 gives
1LT
dr E
(Cks - 1)TP 1Chx-1)I?* + Σ CNTrIII
Σ -L
ligk
Argument for the moreover statement want to show L > 0 for all E [K]:
As in the statement of the lemma, assume
1 A 2 AK >
(66)
Fix arbitrary k E [K]. Write C sp.. exk n span +;; this is non-empty because it is the orthogonal complement of (K dimmennsiol subspace in K-dimensional space.
Th E C, then TA E ()) then Cis = for all I + & and so eq. LA (1- Cuk O. Because Ti C spanfer....,
(65 implies that
we have Σ T 1. Then
condition eq. 66) implies that CRA = =
s=1 AaT2
sk E [Ax, A1] C (0, 1) so in fact La > 0.
Otherwise, we can take unit vector pe on span (Ta)+. (Tx)+ span fea...e......ee but orthogonal both T3 and span(AT
Then is unit vector in
Now consider rotating Tę towards by an angle 6, i.e. parameterise path
Ty (0) = cos Th sin @ pk
then we can write
C&a (0) cos? T ATa + 2 sin 0 cos 0 TV Apk + sin? pl App (i)( = cos T ATi
then differentiating these quantities gives
(2+(0) = - cos 0 sin 0 Ck+(O) + + 22 0 cos2 0) T3 App + 2 cos sin 1 p App
Cz:(0) sin 0 T ATi
In particular, evaluating
0 gives ČN 0 and CRR 2 2T Apr- Since T is stationary, the derivative of L. along this path must be zero. Plugging in these expressions gives
1 (o) -(1 CKR)CKA CHICkI
0
2
=-2(1 - CR Apk +0
so because Csa < A1 1 we must in fact have TV Apr 0. This observation is necessary to simplify the following Hessian-like calculations.
For further convenience, from now we work in terms of s2 == sin 0 rather than 0 itself and also
introduce 8 = T AT; P. Aps so we can write
(k) = CAa (0) sin? (T. ATK Pt. Apx) = Ck+ 228
which gives
(()) - C()) Cxx 528)2 (1 Cke (0))2 + 23 Σ cos2 (0)( -23 Σ Cw(
= =2s²8(1 - CR. (0)) + s282 2.2 Σ CR(
48

---

In partieular, for T to be local optimum must have
1 d,2 ),-((()) [(0)],2=0
0 Σ
2
8(1 CR&(O)) Σ Ck((0) + 822 102-0
Cno(0)(1 - Ck+(O)) B
Σ C2((0)² Pk Apx (1 Cka (0))
t.ykk
L
Pk Apu (1 CR&(O))
and so indeed, we in fact have
La p Aps (1 Csa (0)) 2
moreover, when 1 1> A1 2 AK > 0 this inequality becomes strict, as required.
Partial proof of conjecture D.17 First apply Case (T' of full rank but not necessarily square) of lemma D.3 to get Ú E R R f RR* such that B = ÜT and î full row-rank. Then since
B is local minimum, π is stationary point of T -> LBr(T, A). So applying lemma D.12 for each r [R] have
S @LA (T;)) ad,
Σ -
T Tkk
(67)
Now apply Case (T square but not necessarily full rank) of lemma D.3 to get (" E RDD f e Dx such that B = Ú'7". Now I' will not be of full row-rank, but Ú" must give a full ba- sis of CCA directions for (2)) This implies that the corresponding Å' E RDDD is diagonal matrix whose diagonal entries are the full-vector of population canonical correlations CCA(X(), x(1)) E R.P In particular this means that X ....2 Xx O.
We now apply lemma D.12 a second time, now with A' Note that, CN Cov( (BT x B, (2) and is a function of B so must is independent of the choice of decom- position of B. Then eq. (61) shows that the Lagrange multipliers are identical to those from before, precisely (L+)x- Therefore, by applying the final 'moreover' conclusion of lemma D.12 deduce that L > 0 for all k [K].
(67 for each E [R] and using the fact f,. * 0 because of Î's full row-rank implies that X 0. Therefore also have @Car(f:A) 4 [R].
Applying this into, eq.
ax,
Suppose, for contradiction, that 2, < Ar the true thth canonical correlation. Then by conjecture B.S we may construct a (continuous) path (() for E [0, 1] with (()) = Ú, Ú(1) (a)(X()))(())) Ix for all t, and Â(t) == (t) Cov(x(1), (())(((t) such that for all 0
(x > (0) and Á.(t) > (,) E [R] {r)
Correspondingly define the (continuous) path B(t) = (t)f..
But then
(Br(B(t)) - Ler (B(O)) = (T, A()) - (T,((,A(0))
Σ dLm (2,(t) (.(()) (2a(1) 2.(0))
and so is strictly negative for sufficiently small t. This contradicts local optimality of B.
Therefore the top-ft entries of Å must be the top-R canonical correlations, and so by lemma C.6 we must have that Ú defines a CCA subspace, and so B must also define a CCA subspace, as claimed.
49

---

D.4.2 UN-TIED WEIGHTS
For VICReg we saw that the computation for the untied weight case reduced to that of the tied weight case provided that we could prove that minimisers of the corresponding matrix loss (vR(s: A) were symmetric. It is natural to expect that similarly any minimisers of the Barlow twins matrix loss LBT(' A) are symmetrie.
We have observed this to be the case in toy simulations (for wide variety of values of A and hyper-parameter 3), but are not yet able to give a rigorous proof.
If one can prove this observation, then one can also prove notions of the remaining the bullet points from the main text analogously to the VICReg versions: Barlow twins also recovers CCA subspaces in the general be interpreted as an algorithm for Deep CCA.
of untied weights, and the optimal loss is decreasing in correlation signal,
D.4.3 COLLAPSE EXAMPLE
Finally we give a short computation, analogous to that of appendix D.3.4 to show that collapse is generic phenomenon, even in very simple setting.
Proposition D.13. Let A = diag(A1, A2) with 2
> A2 0. Consider minimisers
f arg min LET AT; TER2x2
of the tied Barlow twins matrix loss with parameter B > 0 satisfying
2(1 - - - X2)
B < (3d1 A2)(A1 X2)
=: C()1.22)
(68)
+1
#1
Then T-
are the 4 global minimisers.
0
Proof: Any T E R22x can be parameterised as
cos sin 01
cos 82 sin 02
T
(69)
We use the same convenient notation X(0) A cos 0 + A2 sin 0 as in appendix D.3.4 Barlow twins loss becomes
Then the
LNT (01,
Σ (1 x0x)))))²² 23 (Ar cos cos 02 + da sin O1 sin 02)²
(70)
We now compare each of these terms to the corresponding loss when 0 = 02 = 0. For convenience, introduce the quantity A = Å1 X2. First we bound the reward term:
(1 x(0))² - A1)² ((1 A1) + A sin 0)² (1 \1)3
24(1 sin 0 (A)2 sin"
2 24(1 -ii
Giving us
2
(1 x(Ow))² > 24(1 A1) (sin" 0, sin" 02)
50

---

Next bound the penalty term. To save space we use the shorthand Ck cos $/: sin 0₁ for = 1, and introduce the quantity y == A2/ A1 1 1.
2
1 (A1c122 + )28182) = x
( + 52) - 27622127992
k=1
X [cfs] + 200
Ai ([1=-))(5-60))(((1>>>0- 9102)² (1 - -2})s{33)
Af (2(C (1))) + (8) (2) (1 - -3) 1 (of +5)
+ (1 (1
+ + + 21 73)
1 A (sf + 5/) (3 + 27 72)
2
Finally put these two inequalities into eq. 70 to get
Cor 02) - (0, 0) 2 ( 2i) (2A (1 \1) - BX (3 + 27 - }²)}
and so this will be strictly positive whenever either sin 02 + 0. provided that
21(1 A1)
2(A1 A2) A1)
B
C(11,)2)
12 + 27 - 72) (3)1 - >2) + A2)
as claimed.
51

---

E FAST UPDATES FOR (MULTIVIEW) STOCHASTIC CCA (AND PLS)
E. MACK-PROPAGATION FOR EMPIRICAL COVARIANCES
To help analyse the full details of back- pack-propagation in the linear case, first prove lemma regarding the gradients of the empirical covariance operator.
Lemma E.1 (Back-pryy prop for empirical covariance). Let e E RM, E
ERM. Then Cov(e, f) and
(Cov(e,
de
can both be computed in O (M) time.
Proof. Let 1 E R M be vector of ones and Pi,, IM M 1 1 be the projection away from this vector, then rank structure of Pit, allows us compute these quantities in O (M) time.
write
Pis f Pi Pixf f. Moreover, exploiting the identity-plus-low
Then by definition
Cov(e, [) T
which is again computable in O (M) time. For the backward pass, first note that
de
de + Pi de
de
So the derivative with respect to is
@Cov(e, de
de
de
M de
M
de
M
1ns
M
because f is independent of e, and already mean-centred. So all that remains is element-wise divi- sion, which again costs (M) time.
FORWARD PASS
1. Compute the transformed variables Z:
(71)
with complexity of DD.
2. Compute trace [Z) the diagonal elements of Ô are simply
Okk
Σ (z. z())
which each summand can computed in O (M) time, so summing over i, gives total complexity of O(IxKM).
3. Compute (0) For [Z]::
(a) = Lau U00 - (ar(
a;)
each U(1) ( can be computed with complexity of O (D,K3) and the total cost of
evaluating all of these is O (K²D). Each summand in the second term costs O (MK2) by lemma E. so evaluating the full second term costs (TMK)) K2).
4. Evaluate Lav [2, Z'I:
LEY Z, Z'I = - trace C(Z) + (V. V. Z'I) F-
(72)
The dominant complexity here is the O(K?) cost computing the Frobenius inner prod- uct.
52

---

BACKWARD PASS
1. Gradient with respect to Z Using the chain rule, the gradient will flow back from the final computed value, LEV [z, Z1,, through the operations that produced it.
2. Gradient of trace [Z] with respect to Z Is precisely
aCk
2
Σ
1 k
8
(3)
M
where 2.4) =Pi, ((6) from lemma E.1 and so can be computed in ((I time.
3. Gradients of (V.[Z), V. [Z]) with respect to z of the empirical variance term
By applying lemma E.1 the gradient
2
if = 1' = k if f k otherwise.
Var(ZZ()
M-
1-
and so
a(V. [Z) V. [Z]}r
(1 as))
[2 1 %
+(vVa
(Va [Z' 12 [v1
O (2)
M
2(1 - a+)
K
Σ 1=1
V. [Z') 2
M -
this can be computed in O(M)) time.
4. Gradients of LEY [Z, Z'J with respect to [. can therefore be computed for given Z in O (M(K + ()) time and so, adding up over all i, k gives total O (TM(K + I)) time.
5. Gradients of (Va [Z], V. ( with respect to U/ is similarly
K
20:
Σ
Dl + V. (')
M
can be computed in O (D,K) time.
6. Finally compute gradients with respect to U:((); simply have z U((() final gradients are
x
so the
aC au
@LLY az
a(V. [2), V. Z'))
X
(73)
au(
the dominant cost is the O (M D.) multiplication.
Since D >> M. the dominant cost each final gradient is (MD,). Summing up over k gives total cost O (KM) Σ D.) O (KMD), as claimed.
53

---

F ADDITIONAL STOCHASTIC CCA EXPERIMENTS
F.1 SUPPLEMENTARY EXPERIMENTS: SPLIT CIFAR
In this supplementary section, we provide additional experimental results on the Split CIFAR dataset, where the left and right halves of CIFAR-10 images are utilized as separate views for canon- ical correlation analysis. These experiments aim bolster the findings reported in the main paper on the MediaMill dataset.
Observations: As shown in Figure our method CCA-EY demonstrates similar advantages on the Split CIFAR dataset as observed in the main experiments on MediaMill. Specifically, CCA-EY out- performs both ---iienGamme and SGHA in terms of PCC across all tested mini-batch sizes. More- over, Figure Sa that CCA-EY converges faster than the baselines when using mini-batch size of 20. It is important note that these trends echo the findings in our primary experiments, further confirming the robustness and efficacy of CCA-EY across different datasets and configurations.
CCA cifar (d,=1536, d,=1536)
1.00
Top CDA on cifar (dym1536, 1531)
Train PC6
0.75
9.75
S0HA VElgenSamm
0.60
0.50
batch
0.25
0.25
0.00
100
0.00
10000
40000
50000
Samples Seen
SGHA
W-ggnnGame
(b)
Figure 5: Experiments onSplit CIFAR with Stochastic CCA: (a) Proportion of Correlation Captured (PCC) across varying mini-batch sizes (left), and (b) Convergence behavior with respect to samples seen for mini-batch size 20 (right). Both subfigures compare CCA-EY against prior methods (1- EigenGame and SGHA). Shaded regions signify + one standard deviation around mean.
G ADDITIONAL DCCA EXPERIMENTS
In this section, we delve into the performance of DCCA-EY against other DCCA methods, The experimental setup is borrowed from Wang et al. (2015b), utilizing the XRMB dataset. We use mini-batch sizes ranging from 20 100 and train the models for 50 epochs. Our metric here the Total Correlation Captured (TCC), given by TCC = E
Observations: As depicted in Figure DCCA-STOL shows limitations in scalability, struggling to optimize a 50-dimensional representation when the mini-batch size less than 50. This is par- ticularly evident in the performance curve for XRMB (Figure bap. On the other hand, DCCA-NOI performs similarly to DCCA-EY but only for larger mini-batch sizes and with slower speed to con- vergence.
50 CCA XRMB
Validation TCC
mode
Valdalion 8
30
DCCA-EY
DCCA NDI
20
10
100
50
160
batch sze
epoch
DCCA-NOI
OCCA-STOL
(a) Performance comparison on XRMB
(b) Performance comparison on XRMB
Figure 6: Validation TCC by DCCA-EY prior work on the XRMB dataset. Subfigure (a) shows the validation correlation for different batch sizes among various models. Subfigure (b) depicts the validation correlation against the number of epochs for fixed batch size of 50.
54

---

H ADDITIONAL SSL EXPERIMENTS
H.1 JOINT EMBEDDING FOR SSL AND THE ROLE OF THE PROJECTOR
Many recent SSL methods, including Barlow Twins and VICReg use an encoder-projector setup, as illustrated in Figure 7 Input data is mapped through some encoder y to obtain representations; these representations are then mapped through a projecto 2 / to a (typically) higher-dimensional embed- ding. Crucially,
the representations that are used for down-stream tasks but the embeddings that are used to train the model. Typically. the encoder is a neural network with domain appropriate architecture, but the projector is a (relatively shallow) multi-layer perceptron.
The idea of joint embedding methods is that similar inputs should have similar embeddings. To train them, one obtains pairs X, X' of similar input data through domain specciic augmentation methods; the encoder and projector then learn to optimise some objective characterizing how close Z, Z' are.
Encoder-pres Encoder-projector architectures, have had impressive empirical success, despite recent work Ma et al. (202)) Jing et al. (2021), there is relatively little understanding of why they work so well. Our more principled objective opens the door for a better understanding of this phenomenon, which may lead to improved future architectures.
X
g
h
Z
Encoder
Projector
A schematic diagram of the architecture used by Joint Embedding methods which include VICReg, and Barlow Twins
Figure 7:
In Figure we demonstrate that our model's performance plateaus at a much smaller projector dimension. This serves as empirical evidence supporting our algorithm as robust choice for range of scenarios.
CIFAR-10 Top-1 Accuracy
CIFAR-100 Top-1 Accuracy
CIFAR-10 Top-1 Accuracy vs. Projector Dimension
CIFAR-100 Top-1 Accuracy Projector Dimension
60
25
128 Projector Dimension Barlow Twins
Projector Dimension Barlow Twins
SSL-EY
VICReg
SSL-EY
VICReg
Figure 8: Performance saturation in our model occurs at a much smaller projector size compared to VICReg and Barlow Twins, demonstrating its robustness.
H.2 UNDERSTANDING LONG-TERM CONVERGENCE
key insight from our learning curves in Figure and Figure 10 is that the performance variation observed at 1000 epochs is largely function of noise in early optimization stages. All models seem follow similar convergence trends, underscoring that the performance differences are not indicative of intrinsic model superiority.
Sometimes alternatively called expander.
55

---

CIFAR-10 Top-1 Accuracy vs. Epoch
CIFAR-10 Top-5 Accuracy Epoch
100.0
CIFAR-10 Top-1 Accuracy
80
CIFAR Top-S Accuracy
97.5
95.0
92.5
60
90.0
SSL-EY Barlow Twins
SSL-EY
87.5
40
Barlow Twins
VIGReg
85.0
VICReg
30
200
400
600
800
1000
200
400
600
800
1000
Epoch
Epoch
Figure 9: Learning curves for CIFAR-10 showing that the performance of models after 1000 epochs influenced by noise in early optimization, with all models converging similarly.
CIFAR-100 Top-1 Accuracy vs. Epoch
CIFAR-100 Top-5 Accuracy vs. Epoch
90
CIFAR-10 1FAR-100 Top-1 Accuracy
CIFAR-100 1-000 Top-5 Accuracy
60
80
50
70
60
30
SSL-EY
50
SSL-EY
Barlow Twins
Twins
20
VICReg
40
VICReg
200
400
600
800
1000
400
600
800
1000
Epoch
Epoch
Figure 10: Learning curves for CIFAR- 100 emphasizing the role of early optimization noise in the performance after 1000 epochs, highlighting the similar convergence of all models.
56

---

I REPRODUCIBILITY
In this section, we give further detail to allow readers to reproduce the results in the paper.
1.1 CODE
We make code for the Stochastic CCA and Stochastic Deep CCA experiments available in the at- tached zip file. We will make this available as public Github repository.
1.2 COMPUTER RESOURCES
Each of the four types of experiment required slightly different resources due the relative scale of the problems.
Experiment
CPU/GPU Resources
Stochastic CCA
NVIDIA GeForce RTX 2080 Ti
Deep CCA
NVIDIA GeForee RTX 2080 T1
Deep MCCA
NVIDIA GeForee RTX 2080 Ti
Stochastic PLS
NVIDIA GeForce GTX 1650 Ti
SSL
4-8 NVIDIA GeForee RTX 2080 Ti, Quadro RTX 8000
Quadro RTX 6000, NVIDIA GeForce GTX 1080 Ti GPU devices
Table 2: Computer resources for each experiment type
1.3
FURTHER EXPERIMENT DETAILS
In this section, give further details regarding the descriptions of the metrics and parameter search.
1.3.1
STOCHASTIC CCA
Parameters: For each method, searched over hyperparameter grid using Biewald (202).
Parameter minibatch size
Values
5,20,50,100
components
epochs
seed
1,2,3,4, 4,5
lr
0.01, 0.001, 0.0001 0.01,0.1,1,10
10
I.3.2 DEEP CCA
Further details: As in Wang et al. (2015)), we used multilayer perceptrons with two hidden layers with size 800 and an output layer of 50 with ReLU activations. We train for 20 epochs.
Parameters: For each method, we searched over hyperparameter grid using Biewald (2020).
Parameter
Values
minibatch size
100, 50, 20
le-3, le-4, le-5
0.6, 0.8, 0.9
epochs
50
I.3.3 DEEP MCCA
Parameters: For each method, searched over hyperparameter grid using Biewald (2020)
57

---

Parameter
Values
minibatch size
5,10,20,55,100,200
components
50
epochs
100
Ir
0.01, 0.001, 0.0000)
0.00001
I.3.4
UK BIOBANK PLS
Partial Least Squares
Following section 2 we can combine equations
4 4ndd 6 with a, 1vi in order to write Partial Least
Squares as Generalized Eigenvalue Problem:
[p. 0
1 (1
A = (c) x())
0 B =
Toz
u
(3)
(74)
Note that since B is an Identity matrix by construction, we do not need to make stochastic approxi- mations of during optimization.
The UK BioBank data consisted of real-valued continuous brain volumes and ordinal, integer genetic variants. We used pre-processed (using FreeSurfer (Fischl) 2012)) grey- matter volumes for 66 cortical (Desikan-Killics autosomal genetic variants. The effects of age, age squared, intracranial volume, sex, and the first 20 genetic principal components for population structure using linear regression to account for any confounding effects. Each brain ROI was normalized by removing the mean and dividing the standard deviation. We processed the genetics data using PLINK (Purcell et al., 2007) keeping genetic variants with a minor allele frequency of at least % and maximum missingness rate of 2%. We used mean imputation fill in missing values and centered each variant.
Further details:
atlas) and 16 subcortical brain regions and 582,565
removed from the brain features
To generate measures of genetic disease risk, we calculated polygenic risk scores using PRSice Eu- esden et al., 2014). We calculated scores, with p-value threshold of 0.05, using GWAS summary statistics for the following diseases; Alzheimer's 's (Lambert et al., 2013). Schizophrenia (Trubetskoy et al., 2022). Bipolar (Mullins et al.) 2021), ADHD (Demontis et al.) (023),, ALS (van Rheenen al. 2021 J, Parkinson's 's (Nalls et al. 2019), and Epilepsy (International League Against Epilepsy Consortium on Complex Epilepsies] 2018), using the referenced GWAS studies.
The GEP-EY PLS analysis was trained for 100 epochs using learning rate of 0.0001 with a mini- batch size of 500.
1.3.5
LEARNING
In this section, we provide a comprehensive overview of the experimental settings and configurations used in our self-supervised experiments.
As stated before, we use the standard setup from solo-learn's pretraining scripts. For the backbone network, we use ResNet-18. The projector network consists of hidden dimensions and output di- mensions both set to 2048. We employ the LARS optimizer with a learning rate of 0.3 for the backbone and 1.1 for the classifier. The batch size is set to 256, and weight decay is set to x 10-4, Additional optimizer parameters include clip_Ir set to True, 77 set to 0.02, and exclude_bias_n.me _norr set to True. The learning rate scheduler used is warmup cosine scheduler. The models trained for 1000 epochs. The model's 's calculations are performed with a numerical precision of 16 bits.
VICReg and Barlow Twins: Both models employ similar data augmentations, specified in Tables 3 Bladd In table we show the shared augmentations while in table show the differences. Note different augmentations with 50% probability each.
that Barlow Twins
58

---

Augmentation
Parameters
ColorJitter
brightness 0.4, contrast 0.4, saturation 0.2, hue 0.1, prob 0.8
Grayscale
prob = 0.2
Horizontall-lip
prob =
CropSize
32
Table 3: Shared augmentations for VICReg and Barlow Twins
Augmentation
VICReg
Barlow Twins (crop 1)
Barlow Twins (crop 2)
RandomResizedCrop
Yes
Yes
Yes
crop min scale
0.2
0.08
0.08
crop max scale
1.0
1.0
1.0
Solarization
Yes
No
Yes
prob = 0.1
prob = 0.0
prob = 0.2
NumCrops
Table 4: Different augmentations for VICReg and Barlow Twins
1.4 PYTORCH PSEUDO-CODE: UNIPYING ALGORITIIMS UNDER THE GENERALIZED EIGENPROBLEM (GEP) FRAMEWORK
In this work, we introduce three distinct algorithms: DMCCA-EY, PLS-EY, and SSL-EY. Despite their apparent differences, they are all specialized instances of generalized eigenproblem (GEP). All these algorithms maximize the objective function outlined in Proposition 3.1 making them spe- cial cases of our main contribution.
Algorithm gives a general loss for DCCA and DMCCA. Algorithm shows how we can adapt the loss function for stochastic PLS problems. Algorithm 4 gives generic SSL loss.
Algorithm 2: DMCCA-EY Loss Function Python def IMCCA.E(yviews, views.prime)
encode (views) Encode the views
prime = encode (views prime) # Encode the prime views
A, B, B prime
torch. zeros (z [0] shape [1], z shappe (1)),
[1]]
torch. zeros (z [0] shapee[], z [[]] chape(1]),
zeros
norch.zeros (z [0] [h]pe[[], z [0]] shape # Initialize
matrices
for zi, zj in all-airs (z):
A get.cross.covarianee (zi, 2j)
Compute
cross-covariance
get_auteetteeeee auto covariance (zi) Compute auto-covariance
for zi in z.prime:
B.prime + get.auto.covariance (zi)
Compute
auto-covariance for prime views
A, B, B.prime
(z) len (z), B.prime len (z.prime)
# Normalize matrices return -torch. trace
B.prime) Calculate loss
59

---

Algorithm 3: PLS-EY Loss Function in Python
def PLS.EY (views)
2, weights encode.and.wweights (views)
|| Encode the views and
get weights
A, B = forch.zeros
[1], z [[0].shape [1]),
torch. zeros (weights [0] shape [1], weights [0] shape [1])
Initialize matrices
for zi, zj in all.pairs (z)
A += cross.covarinnee..5 (zi, 2j) Compute
cross-covariance for PLS
for wi in weights:
B += wtwpoooovvrrance.eeiiigho. auto-covariance for PLS
(wi) Compute
A, B - A / (z), B / len (weights) # Normalize matrices
return -torch
B) # Calculate loss
Algorithm 4: SSL-EY Loss Function in Python
def SSL.EY (views, views-prime)
encode (views) # Encode the views
z.prime encode (views-prime)
Encode the prime views
A, B, B.prime torch. zeros (2 [0] shape [1], 2 shape
torch.zeros zeros (z [0] shape[[], 2 [0] shape [1]),
torch. zeros (z [0] shape[1], [0] shape [1]) # Initialize matrices
for zi, zj in all-airs
A += get cross covariance (zi, cross-covariance
2j)
Compute
B += aet.aato.ccovaianaeoo (zi) # Compute auto-covariance
for zi in z.prime:
B-prime += get.auto.covariance (zi)
Compute
auto-covariance for prime views
A, B, B.prime A len (2), B / len (2), B.prime / len (z.prime) Normalize matrices
1.4.1 SOLO-LEARN ADAPTATION
The version of SSL-EY in algorithm 5 is designed to integrate seamlessly into solo-learn, offering support for distributed training.
60

---

Algorithm 5: Solo-Learn Loss function for distributed SSL-EY in Python
Define the SSL-EY loss function
Input: Projected features from two views def SSL EY (z1, =
# Get the minibatch size and feature dimension
N, D = 21 zl.size()
Compute the covariance matrix from the concatenated features
torch. cov (ttorch.sstacee ( (21, 22))
Average the covariance matrix across all processes if distributed training is enabled
if lsab.i and iist.issinitiaalized
dist.all.reduce
worldssize - cist.get.worldssize))
/- wired.size
# Extract symmetric and anti-synnetric blocks of C C[:D, D:] C(D:, :D]
Cl:D, :D] C[D:, D:
Return the SSL-EY loss value
return Storch.trace (2 A e B)
61


Reviews:
[{"review_id": "FMGDOVbc5d", "reviewer": "Reviewer_NeR6", "summary": "This paper proposes a new formulation for general CCA problems, based on an Eckhart-Young inspired objective for the Generalized Eigenvalues Problem. Such objective is prone to being implemented in a stochastic way (i.e. with minibatches of dataset samples), and can be used for various variant of CCA such as the original linear CCA, PLS and Ridge Regularized CCA, Multi-view CCA, as well as Deep CCA. The theoretical advantages of such formulation are given, and an extensive experimental benchmark is performed, showing the advantage of the introduced method on several tasks, over state of the art methods.", "questions": "I just noticed one typo below: \n- In section 3.3: “we this flesh out” —> “we flesh this out”", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "### Originality\n\nI believe the work is original, as it is, up to my knowledge, the first of its kind to consider an Eckhart-Young inspired objective (and the corresponding stochastic optimization algorithm to optimize it) for CCA-type problems.\n\n### Quality\n\nI believe the work is of quality, with theoretical results clearly described as well as their link the the related literature, and experimental results performed extensively.\n\n\n### Clarity\n\nI believe the work is clear, with a clear description of the methods and of the state of the art. I believe the experiments are clearly described and the code provided eases the reproducibility of the methods.\n\n\n### Significance\n\n\nI believe the work is significant, since CCA-like methods and the related Self-Supervised learning methods have been shown to be very important these last years, in particular for modern machine learning applications, including those which use deep neural networks to learn data representations: offering a theoretically grounded view of such problems, as well as an efficient and still theoretically grounded implementation (namely, the stochastic algorithm), I believe this type of methods is very interesting to the community.", "weaknesses": "Although the literature is extensively cited across the paper, and well detailed, and although the experiments show a clear advantage to the presented method, I still find it a little bit hard to compare the theoretical and computational guarantees of the presented method vs. the other methods from the benchmark: I believe it might be useful, (for instance in Appendix), to provide some short structured summary (perhaps as a table for instance), on all the properties of the related methods in the literature vs. the one presented (e.g. biasedness/unbiasedness of the gradients, complexity of computing one iteration as a function of the batch-size, dimension, etc, guarantees of convergence towards global/local optimum/saddle point, etc). However, I am aware that such comparison may not always be doable (in particular because the guarantees of convergence for instance, are not always simple to express), so this is just a suggestion,  andI believe the paper in its current form may already provide the necessary information to the reader to study such aspects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9gywmEZlT1", "reviewer": "Reviewer_73xD", "summary": "The paper proposes a new unconstrained loss function that introduces solutions to GEPs, paving the way for solving Canonical Correlation Analysis in large scale setting in an efficient manner.", "questions": "Justification of the novelty of the proposed method.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper conducts extensive experiments across various benchmarks, and proves the effectiveness of the proposed method by the superior empirical results, compared to baselines such as SGHA and $\\gamma$-EigenGame.", "weaknesses": "I have concern for the novelty of the method compared to Gemp et al. (2022). “unlike Gemp et al. (2022) we do not simplify the problem by first performing PCA on the data before applying the CCA methods “  The author needs to show insights as for the innovation of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MhG6paIvLt", "reviewer": "Reviewer_wHCq", "summary": "UPDATE:\nThe authors have done a good job engaging with the reviewers during the discussion period. I have updated my score accordingly to an 8.\n\nClassical algorithms for solving linear problems arising from CCA, PLS, and GEP are computationally infeasible (slow and memory-intensive) for huge datasets. The authors formulate top-subspace GEPs via a new unconstrained objective. The main result is then a family of algorithms that solves stochastic PLS, stochastic CCA and deep CCA by applying SGD to this objective. Empirical results show faster and better convergence than previous methods. For the first time, a very large biomedical dataset is tackled with PLS.", "questions": "- I am having trouble with Lemma 3.2. When we assume that there is a final linear layer in each neural net, does this parameter $\\theta$ only relate to this final layer, or to the whole neural net? How then is it intuitively possible that $\\theta$ could be a local optimum?\n- Ridge-regularized CCA. Does the ridge term also change the aforementioned \"notions of uniqueness\" to a stronger notion?\n- \"unlike Gemp et al. (2022) we do not simplify the problem by first performing PCA on the data before applying the CCA methods, which explains the decrease in performance of γ-EigenGame compared to their original work. \" What then is the motivation for not first simplifying the problem using PCA preprocessing?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper does require a relatively strong background and interest in linear algebra (GEP), optimisation, classical self-supervised/unsupervised techniques (PCA, CCA, PLS), and I have to admit that I found this paper challenging to read. However, when I did check isolated individual details carefully, I was not able to spot any immediately obvious issues. (Soundness +) \n- While the material is dense, the paper is clearly laid out and well-written. I particularly like section 2.2 and section 3.2/3.3, which first present general frameworks and then discuss special cases, which is a helpful tutorial-like description. (Presentation +)\n- The paper is of general interest to the community, particularly those interested in (self/un)supervised methods. Unifying theories under a single framework is an attractive story for a paper.\n\nOverall I find this paper to be good quality, however I have low confidence in my evaluation. I am receptive to increasing my evaluation and confidence if the authors can provide a good rebuttal and help clarify my queries.", "weaknesses": "Clarity:\n- I do not see any precise discussion of Barlow Twins or VICReg in the main paper, however Proposition 1 informally states a result related to these formulations. I wonder what is the value in this --- a reader who knows about these formulations might dismiss the informal result (looking at the Appendix instead), and a reader who does not know about these formulations is left completely in the dark. Perhaps section 3.4 could be removed and replaced with a sentence referring to appendix D. (Presentation -)\n- I checked the appendix I.3.4 but was not able to properly understand basic properties of dataset UK Biobank. Are the features in this data categorical, integer, real-valued, mixed, ... ? What are the modelling and \"convenience\" considerations when applying your method? In section 5.4, it is mentioned that \"this can reveal novel phenotypes of interest and uncover genetic mechanisms of disease and brain morphometry.\" What is the precise task that is being performed here, with respect to the PLS formulation (e.g. with reference to the symbols in equations 4, 5, 6, 7)? (Presentation -, and perhaps Soundness -)\n\nMinor and typos:\n- It is not clear what is meant by \"which should be true in general\" in footnote 3. Perhaps if the observations are drawn from a continuous probability distribution, with probability 1 they are linearly independent?\n- Check the references. E.g. \"stochastic optimization for pca and pls\" should be \"stochastic optimization for PCA and PLS\". Perhaps the authors could also seek advice from the area chair (if the paper is accepted) about the most appropriate way to cite the multiple papers with huge author lists (this might be common in other fields where these papers are published, but in ML it looks really bizarre).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)