Stochastic Canonical Correlation Analysis: A Riemannian Approach

t ) convergence rate ( where d is the dimensionality ) or only extract the top 1 component with O ( 1t ) convergence rate . In contrast , our algorithm achieves O ( d2k ) runtime complexity per iteration for extracting top k canonical components with O ( 1t ) convergence rate . We present our theoretical analysis as well as experiments describing the empirical behavior of our algorithm , including a potential application of this idea for training fair models where the label of protected attribute is missing or otherwise unavailable . 1 INTRODUCTION . Canonical correlation analysis ( CCA ) is a popular method for evaluating correlations between two sets of variables . It is commonly used in unsupervised multi-view learning , where the multiple views of the data may correspond to image , text , audio and so on Rupnik & Shawe-Taylor ( 2010 ) ; Chaudhuri et al . ( 2009 ) ; Luo et al . ( 2015 ) . Classical CCA formulations have also been extended to leverage advances in representation learning , for example , Andrew et al . ( 2013 ) showed how the CCA can be interfaced with deep neural networks enabling modern use cases . Many results over the last few years have used CCA or its variants for problems including measuring representational similarity in deep neural networks Morcos et al . ( 2018 ) , speech recognition Couture et al . ( 2019 ) , etc . The goal in CCA is to find linear combinations within two random variables X and Y which have maximum correlation with each other . Formally , the CCA problem is defined in the following way . Given a pair of random variables , a dx-variate random variable X and a dy-variate random variable Y , with unknown joint probability distribution , find the projection matrices U ∈ Rdx×k and V ∈ Rdy×k , with k ≤ min { dx , dy } , such that the correlation is maximized : maximize trace ( UTEX , Y [ XTY ] V ) s.t . UTEX [ XTX ] U = Ik , V TEY [ Y TY ] V = Ik ( 1 ) Here , X , Y are samples of X and Y respectively . The objective function in ( 1 ) is the expected crosscorrelation in the projected space and the constraints specify that different canonical components should be decorrelated . Let CX = EX [ XTX ] and CY = EY [ Y TY ] be the covariance matrices , and CXY = E ( X , Y ) [ XTY ] denote cross-covariance . Let us define the whitened covariance T : = C −1/2 X CXY C −1/2 Y and Φk ( and Ψk ) contains the top-k left ( and right ) singular vectors of T . It is known Golub & Zha ( 1992 ) that the optimum of ( 1 ) is achieved at U∗ = C−1/2X Φk , V ∗ = C −1/2 Y Ψk . In practice , we may be given two views of N samples as X ∈ RN×dx and Y ∈ RN×dy . A natural approach to solving CCA is based on on the following sequence of steps . We first compute the empirical covariance and cross-covariance matrices , namely , C̃X = 1/NXTX , C̃Y = 1/NY TY and C̃XY = 1/NX TY . We then calculate the empirical whitened cross-covariance matrix T̃ , finally , compute U∗ , V ∗ by applying a k-truncated SVD to T̃ . Runtime and memory considerations . The above procedure is simple but is only feasible when the data matrices are small . But in most modern applications , not only are the datasets large but also the dimension d ( let d = max { dx , dy } ) of each sample can be quite high , especially if representations are being learned using deep neural network models . As a result , the computational footprint of the foregoing algorithm can be quite high . This has motivated the study of stochastic optimization routines for solving CCA . Observe that in contrast to the typical settings where stochastic optimization schemes are most effective , the CCA objective does not decompose over samples in the dataset . Many efficient strategies have been proposed in the literature : for example , Ge et al . ( 2016 ) ; Wang et al . ( 2016 ) present Empirical Risk Minimization ( ERM ) models which optimize the empirical objective . More recently , Gao et al . ( 2019 ) ; Bhatia et al . ( 2018 ) ; Arora et al . ( 2017 ) describe proposals that optimize the population objective . To summarize the approaches succinctly , if we are satisfied with identifying the top 1 component of CCA , effective schemes are available by utilizing either extensions of the Oja ’ s rule Oja ( 1982 ) to generalized eigenvalue problem Bhatia et al . ( 2018 ) or the alternating SVRG algorithm Gao et al . ( 2019 ) ) . Otherwise , a stochastic approach must make use of an explicit whitening operation which involves a cost of d3 for each iteration Arora et al . ( 2017 ) . Observation . Most approaches either directly optimize ( 1 ) or instead a reparametrized or regularized form Ge et al . ( 2016 ) ; Allen-Zhu & Li ( 2016 ) ; Arora et al . ( 2017 ) . Often , the search space for U and V corresponds to the entire Rd×k ( ignoring the constraints for the moment ) . But if the formulation could be cast in a form which involved approximately writing U and V as a product of several matrices with nicer properties , we may obtain specialized routines which are tailored to exploit those properties . Such a reformulation is not difficult to derive – where the matrices used to express U and V can be identified as objects that live in well studied geometric spaces . Then , utilizing the geometry of the space and borrowing relevant tools from differential geometry leads to an efficient approximate algorithm for top-k CCA which optimizes the population objective in a streaming fashion . Contributions . ( a ) First , we re-parameterize the top-k CCA problem as an optimization problem on specific matrix manifolds , and show that it is equivalent to the original formulation in equation 1 . ( b ) Informed by the geometry of the manifold , we derive stochastic gradient descent algorithms for solving the re-parameterized problem with O ( d2k ) cost per iteration and provide convergence rate guarantees . ( c ) This analysis provides a direct mechanism to obtain an upper bound on the number of iterations needed to guarantee an error w.r.t . the population objective for the CCA problem . ( d ) The algorithm works in a streaming manner so it easily scales to large datasets and we do not need to assume access to the full dataset at the outset . ( e ) We present empirical evidence for both the standard CCA model and the DeepCCA setting Andrew et al . ( 2013 ) , describing advantages and limitations . 2 STOCHASTIC CCA : REFORMULATION , ALGORITHM AND ANALYSIS . The formulation of Stochastic CCA and the subsequent optimization scheme will seek to utilize the geometry of the feasible set for computational gains . Specifically , we will use the following manifolds ( please see Absil et al . ( 2007 ) for more details ) : ( a ) Stiefel : St ( p , n ) . The manifold consists of n× p , with p < n , column orthonormal matrices , i.e. , St ( p , n ) = { X ∈ Rn×p|XTX = Ip } . ( b ) Grassmanian : Gr ( p , n ) . The manifold consists of p-dimensional subspaces in Rn , with p < n. ( c ) Rotations : SO ( n ) . the manifold/group consists of n × n special orthogonal matrices , i.e. , SO ( n ) = { X ∈ Rn×n|XTX = XXT = In , det ( X ) = 1 } . We summarize certain geometric properties/operations for these manifolds in the Appendix but have been leveraged in recent works for other problems also Li et al . ( 2020 ) ; Rezende et al . ( 2020 ) . Let us recall the objective function for CCA as given in ( 1 ) . We denote X ∈ RN×dx as the matrix consisting of the samples { xi } drawn from a zero mean random variable X ∼ X and Y ∈ RN×dy denotes the matrix consisting of samples { yi } drawn from a zero mean random variable Y ∼ Y . For notational and formulation simplicity , we assume that dx = dy = d in the remainder of the paper although the results hold for general dx and dy . Let CX , CY be the covariance matrices of X , Y . Also , let CXY be the cross-correlation matrix between X and Y . Then , we can write the CCA objective as max U , V F = trace ( UTCXY V ) subject to UTCXU = Ik V TCY V = Ik ( 2 ) Here , U ∈ Rd×k ( V ∈ Rd×k ) is the matrix consisting of { uj } ( { vj } ) , where ( { uj } , { vj } ) are the canonical directions . The constraints in equation 2 are called whitening constraints . Let us define matrices Ũ , Ṽ ∈ Rd×k which lie on the Stiefel manifold , St ( k , d ) . Also , let Su , Sv ∈ Rk×k denote upper triangular matrices and Qu , Qv ∈ SO ( k ) . We can rewrite the above equation and the constraint as follows . A Reformulation for CCA max Ũ , Ṽ , Su , Sv , Qu , Qv U=ŨQuSu ; V=Ṽ QvSv F̃ = trace ( UTCXY V ) ( 3a ) subject to UTCXU = Ik ; V TCY V = Ik ( 3b ) Ũ , Ṽ ∈ St ( k , d ) ; Qu , Qv ∈ SO ( k ) ; Su , Sv is upper triangular Here , we will maximize ( 3a ) with respect to Ũ , Ṽ , Su , Sv , Qu , and Qv satisfying equation 3b . Main adjustment from ( 2 ) to ( 3 ) : In ( 2 ) , while U and V should decorrelateCX andCY respectively , the optimization/search is unrestricted and treats them as arbitrary matrices . In contrast , equation 3 additionally decomposes U and V as U = ŨQuSu and V = Ṽ QvSv with the components as structured matrices . Hence , the optimization is regularized . The above adjustment raises two questions : ( i ) does there exist a non-empty feasible solution set for ( 3 ) ? ( ii ) if a solution to ( 3 ) can be found ( which we will describe later ) , how “ good ” is this solution for the CCA objective problem , i.e. , for ( 2 ) ? Existence of a feasible solution : We need to evaluate if the constraints in ( 3b ) can be satisfied at all . Observe that by using Ũ to be the top-k principal directions of X , Su to be the 1/ √ top-k eigen values of CX and Qu to be any orthogonal matrix , we can easily satisfy the “ whitening constraint ” and hence ŨQuSu is a feasible solution of U in ( 3 ) and similarly for V . From this starting point , we can optimize the objective while ensuring that we maintain feasibility . Is the solution for equation 3 a good approximation for equation 2 ? : We can show that under some assumptions , the estimator for canonical correlation , i.e. , solution of equation 3 , is consistent , i.e. , solves equation 2 . We will state this formally shortly . Before characterizing the properties of a solution for equation 3 , we first provide some additional intuition behind equation 3 and describe how it helps computationally . Intuition behind the decomposition U = ŨQuSu : A key observation is the following . Recall that by performing principal component analysis ( PCA ) , the resultant projection matrix will exactly satisfy the decorrelation condition needed for the “ whitening constraint ” in equation 2 ( projection matrix consists of the eigen-vectors of XTX ) . A natural question to ask is : Can we utilize streaming PCA algorithm to help us obtain an efficient streaming CCA algorithm ? Let us assume that our estimate for canonical correlation directions , i.e. , solutions of equation 3 , lies in the principal subspace calculated above . If so , we can use the decomposition U = ŨAu ( analogously for V ) , where Ũ contains the principal directions , i.e. , ∈ St ( k , d ) and Au is a full rank k × k matrix containing the coefficients of the span . But maintaining the full rank constraint during optimization is hard , so we further decompose Au into Au = QuSu with Qu ∈ SO ( k ) ; Su is upper triangular . Additionally , we ensure the diagonal of Su to be non-zero to maintain full-rank of Su . During optimization , we can maintain the non-zero diagonal entries by optimizing the log of the diagonal entries instead . Why equation 3 helps ? First , we note that CCA seeks to maximize the total correlation under the constraint that different components are decorrelated . The difficult part in the optimization is to ensure decorrelation , which leads to a higher complexity in existing streaming CCA algorithms . On the contrary , in equation 3 , we separate equation 2 into finding the PCs and finding the linear coefficients for the span of principal directions . Then , by utilizing an efficient streaming PCA algorithm , a lower complexity can be achieved . We will defer describing the specific details of the optimization itself until the next sub-section . First , we will show formally why substituting equation 2 with equation 3a–equation 3b is sensible under some assumptions .

Summary: The paper presents an approach to find canonical directions in a streaming fashion, i.e. without direct calculation of covariance matrices (which becomes hard when the number of examples is large). This solution to that task is not obvious, because the objective function of CCA, together with whitening constraints, does not allow simple additive decomposition.