An Online Riemannian PCA for Stochastic Canonical Correlation Analysis

t ) convergence rate ( where d is the dimensionality ) or only extract the top 1 component with O ( 1t ) convergence rate . In contrast , our algorithm offers an improvement : it achieves O ( d2k ) runtime complexity per iteration for extracting the top k canonical components with O ( 1t ) convergence rate . While our paper focuses more on the formulation and the algorithm , our experiments show that the empirical behavior on common datasets is quite promising . We also explore a potential application in training fair models with missing sensitive attributes . 1 Introduction . Canonical correlation analysis ( CCA ) is a classical method for evaluating correlations between two sets of variables . It is commonly used in unsupervised multi-view learning , where the multiple views of the data may correspond to image , text , audio and so on , Rupnik and Shawe-Taylor [ 2010 ] , Chaudhuri et al . [ 2009 ] , Luo et al . [ 2015 ] , and has been applied to manifold-valued data also Kim et al . [ 2014 ] . Classical formulations have also been extended to leverage advances in representation learning , for example , Andrew et al . [ 2013 ] showed how the CCA can be interfaced with deep neural networks enabling modern use cases . Many results over the last few years have used CCA or its variants for problems including measuring representational similarity in deep neural networks Morcos et al . [ 2018 ] and speech recognition Couture et al . [ 2019 ] . The goal in CCA is to find linear combinations within two random variables X and Y which have maximum correlation with each other . Formally , the CCA problem is defined as follows . Let X ∈ RN×dx and Y ∈ RN×dy be N samples respectively drawn from pair of random variables X ( dx-variate random variable ) and Y ( dy-variate random variable ) , with unknown joint probability distribution . The goal is to find the projection matrices U ∈ Rdx×k and V ∈ Rdy×k , with k ≤ min { dx , dy } , such that the correlation is maximized : max U , V F = trace ( UTCXY V ) s.t . UTCXU = Ik , V TCY V = Ik ( 1 ) Here , CX = 1NX TX and CY = 1N Y TY are the sample covariance matrices , and CXY = 1NX TY denotes the sample cross-covariance . The objective in ( 1 ) is the expected cross-correlation in the projected space and the constraints specify that different canonical components should be decorrelated . Let us define the whitened covariance ∗Equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . T : = C −1/2 X CXY C −1/2 Y and Φk ( and Ψk ) contains the top-k left ( and right ) singular vectors of T . It is known Golub and Zha [ 1992 ] that the optimum of ( 1 ) is achieved at U∗ = C−1/2X Φk , V ∗ = C −1/2 Y Ψk . We can compute U ∗ , V ∗ by applying a k-truncated SVD to T . Runtime and memory considerations . The above procedure is simple but is only feasible when the data matrices are small . In modern applications , not only are the datasets large but also the dimension d ( let d = max { dx , dy } ) of each sample can be large , especially if representations are learned using deep models . As a result , the resource needs of the algorithm can be high . This has motivated the study of stochastic optimization routines for solving CCA , and many efficient strategies have been proposed . For example , Ge et al . [ 2016 ] , Wang et al . [ 2016 ] present Empirical Risk Minimization ( ERM ) models which optimize the empirical objective . More recently , Gao et al . [ 2019 ] , Bhatia et al . [ 2018 ] , Arora et al . [ 2017 ] describe proposals that optimize the population objective . To summarize the approaches , if we are satisfied with the top 1 component of CCA , effective schemes with O ( 1t ) convergence rate are available by utilizing either extensions of the Oja ’ s rule Oja [ 1982 ] to the generalized eigenvalue problem Bhatia et al . [ 2018 ] or the alternating SVRG algorithm Gao et al . [ 2019 ] . Otherwise , a stochastic approach will use an explicit whitening operation which can cost d3 operations for each iteration Arora et al . [ 2017 ] and the convergence rate for the stochastic scheme depends on its specific steps and calculations , e.g. , O ( 1√ t ) in Arora et al . [ 2017 ] ( Thm 2.3 , pp 5 ) . Observation . Most approaches either directly optimize ( 1 ) or instead a reparameterized or regularized form Ge et al . [ 2016 ] , Allen-Zhu and Li [ 2016 ] , Arora et al . [ 2017 ] . Often , the search space for U and V corresponds to the entire Rd×k ( ignoring the constraints for the moment ) . But if the formulation could be cast in a form which involved approximately writing U and V as a product of structured matrices , we may be able to obtain specialized routines which are tailored to exploit those properties . Such a reformulation is not difficult to derive – where the matrices used to express U and V can be identified as objects that live in well studied geometric spaces . Then , utilizing the geometry of the space and borrowing relevant tools from differential geometry could lead to an efficient approximate scheme for top-k CCA which optimizes the population objective in a streaming fashion . Contributions . ( a ) First , we re-parameterize the top-k CCA problem as an optimization problem on specific matrix manifolds , and show that it is equivalent to the original formulation in ( 1 ) . ( b ) Informed by the geometry of the manifold , we derive stochastic gradient descent ( SGD ) algorithms for solving the re-parameterized problem with O ( d2k ) cost per iteration and provide convergence rate guarantees . ( c ) This analysis gives a direct mechanism to obtain an upper bound on the number of iterations needed to guarantee an ϵ error w.r.t . the population objective for the CCA problem . ( d ) The algorithm works in a streaming manner so it easily scales to large datasets and we do not need to assume access to the full dataset at the outset . ( e ) We present empirical evidence for the standard CCA model and the DeepCCA setting Andrew et al . [ 2013 ] , describing advantages and limitations . 2 Stochastic CCA : Reformulation , Algorithm and Analysis . Let us review the objective for CCA as given in ( 1 ) . We denote X ∈ RN×dx as the matrix consisting of the samples { xi } drawn from a zero mean random variable X ∼ X and Y ∈ RN×dy denotes the matrix consisting of samples { yi } drawn from a zero mean random variable Y ∼ Y . For simplicity , we assume that dx = dy = d although the results hold for general dx and dy . Also recall that CX ( and CY resp . ) is the covariance matrix of X ( and Y resp . ) and CXY is the cross-covariance matrix between X and Y . Let U ∈ Rd×k ( V ∈ Rd×k ) be the matrix consisting of { uj } ( { vj } ) , where ( { uj } , { vj } ) are the canonical directions . The constraints in ( 1 ) are called whitening constraints . Reformulation : In the CCA formulation , the matrices consisting of canonical correlation directions , i.e. , U and V , are unconstrained , hence the search space is the entire Rd×k . Now , we reformulate the CCA objective by reparameterizing U and V . In order to do that , let us take a brief detour and recall the objective function of principal component analysis ( PCA ) : Û = argmax U ′ trace ( R̂ ) subject to R̂ = U ′TCXU ′ ; U ′TU ′ = Ik ( 2 ) Observe that by performing PCA and assigning U = Û R̂ − 1/2 in ( 1 ) ( analogous for V using CY ) , we can satisfy the whitening constraint . Of course , writing U = Û R̂ − 1/2 does satisfy the whitening constraint , but such a U ( and V ) will not maximize trace ( UTCXY V ) , the objective of ( 1 ) . Hence , additional work beyond the PCA solution is needed . Let us start from R̂ but relax the PCA solution by using an arbitrary R̃ instead of diagonal R̂ ( this will still satisfy the whitening constraint ) . Write U = Ũ R̃ with ŨT Ũ = Ik and R̃ ∈ Rk×k . Thus we can approximate CCA objective ( we will later check how good this approximation is ) as max Ũ , Ṽ ∈St ( k , d ) Ru , Rv∈Rk×k U=ŨRu ; V=Ṽ Rv trace ( UTCXY V ) ︸ ︷︷ ︸ F̃ + trace ( ŨTCX Ũ ) + trace ( Ṽ TCY Ṽ ) ︸ ︷︷ ︸ F̃pca s.t . U TCXU=Ik V TCY V=Ik ( 3 ) Here , St ( k , d ) denotes the manifold consisting of d× k ( with k ≤ d ) column orthonormal matrices , i.e. , St ( k , d ) = { X ∈ Rd×k|XTX = Ik } . Observe that in ( 3 ) , we approximate the optimal U and V as a linear combination of Ũ and Ṽ respectively . Thus , the aforementioned PCA solution can act as a feasible initial solution for ( 3 ) . As the choice of Ru and Rv is arbitrary , we can further reparameterize these matrices by constraining them to be full rank ( of rank k ) and using the RQ decomposition Golub and Reinsch [ 1971 ] which gives us the following reformulation . A Reformulation for CCA max Ũ , Ṽ , Su , Sv , Qu , Qv U=ŨSuQu ; V=Ṽ SvQv trace ( UTCXY V ) ︸ ︷︷ ︸ F̃ + trace ( ŨTCX Ũ ) + trace ( Ṽ TCY Ṽ ) ︸ ︷︷ ︸ F̃pca ( 4a ) subject to UTCXU = Ik V TCY V = Ik ( 4b ) Ũ , Ṽ ∈ St ( k , d ) ; Qu , Qv ∈ SO ( k ) Su , Sv is upper triangular Here , SO ( k ) is the space of k × k special orthogonal matrices , i.e. , SO ( k ) = { X ∈ Rk×k|XTX = Ik ; det ( X ) = 1 } . Before evaluating how good the aforementioned approximation is , we first point out some useful properties of the reformulation ( 4 ) : ( a ) in the reparametrization of U and V , all components are structured , hence , the search space becomes a subset of Rk×k ( b ) we can essentially initialize with a PCA solution and then try to optimize ( 4 ) via some scheme . Why ( 4 ) helps ? First , we note that CCA seeks to maximize the total correlation under the constraint that different components are decorrelated . One difficulty in the optimization is to ensure decorrelation , which leads to a higher complexity in existing streaming CCA algorithms . On the contrary , in ( 4 ) , we separate ( 1 ) into finding the PCs , Ũ , Ṽ ( by adding the variance maximization terms ) and finding the linear combination ( SuQu and SvQv ) of the principal directions . After optimizing for these variables , the whitening constraints are , up to a rescaling , automatically satisfied . Here , we can ( almost ) utilize an efficient off-the-shelf streaming PCA algorithm . We will defer describing the specific details of the individual steps until the next sub-section . First , we will show why substituting ( 1 ) with ( 4 ) is sensible under some assumptions . Why the solution of the reformulation makes sense ? We start by stating some mild assumptions needed for the analysis . Assumptions : ( a ) The random variables X ∼ N ( 0 , Σx ) and Y ∼ N ( 0 , Σy ) with covariance Σx ⪯ cId and covariance Σy ⪯ cId for some c > 0 . ( b ) The samples X and Y drawn from X and Y respectively have zero mean . ( c ) For a given k ≤ d , Σx , Σy have non-zero top-k eigen values . We show how the presented solution , assuming access to an effective numerical procedure , approximates the CCA problem presented in ( 1 ) . We formally state the result in the following theorem with a sketch of proof ( appendix includes the full proof ) by first stating the following proposition . Definition 1 . A random variable X is called sub-Gaussian if the norm given by ∥X∥⋆ : = inf { d ≥ 0|EX [ exp ( trace ( XTX ) /d2 ) ] ≤ 2 } is finite . Let U ∈ Rd×k , then XU is sub-Gaussian Vershynin [ 2017 ] . Proposition 1 ( Reiß et al . [ 2020 ] ) . Let X be a random variable which follows a sub-Gaussian distribution . Let X̂ be the approximation of X ∈ RN×d ( samples drawn from X ) with the top-k principal vectors . Let C̃X be the covariance of X̂ . Also , assume that λi is the ith eigen value of CX for i = 1 , · · · , d− 1 and λi ≥ λi+1 for all i . Then , the PCA reconstruction error , denoted by Ek = EX∥X − X̂∥ ( in the Frobenius norm sense ) can be upper bounded as follows Ek ≤ min ( √ 2k∥∆∥2 , 2∥∆∥22 λk − λk+1 ) , ∆ = CX − C̃X . The aforementioned proposition suggests that the error between the data matrix X and the reconstructed data matrix X̂ using the top-k principal vectors is bounded . Recall from ( 1 ) and ( 4 ) that the optimal value of the true and approximated CCA objective is denoted by F and F̃ respectively . The following theorem states that we can bound the error , E = ∥F − F̃∥ ( proof in the appendix ) . In other words , if we start from PCA solution and can successfully optimize ( 4 ) without leaving the feasible set , we will obtain a good solution . Theorem 1 . Using the hypothesis and assumptions above , the approximation error E = ∥F − F̃∥ as a function of N is bounded and goes to zero as N → ∞ while the whitening constraints in equation 4b are satisfied . Sketch of the Proof . Let U∗ and V ∗ be the true solution of CCA , i.e. , of ( 1 ) . Let U = ŨSuQu , V = Ṽ SvQv be the solution of ( 4 ) , with Ũ , Ṽ be the PCA solutions of X and Y respectively . Let X̂ = XŨŨT and Ŷ = Y Ṽ Ṽ T be the reconstruction of X and Y using principal vectors . Let SuQu = ŨTU∗ and SvQv = Ṽ TV ∗ . Then we can write F̃ = trace ( UTCXY V ) = trace ( 1 N ( X̂U∗ ) T Ŷ V ∗ ) . Similarly we can write F = trace ( 1 N ( XU ∗ ) T Y V ∗ ) . As X̂ and Ŷ are the approximation of X and Y respectively using the principal vectors , we use Prop . 1 to bound the error ∥F − F̃∥ . Now observe that X̂U can be rewritten into XŨŨTU ( similar for Ŷ V ) . Thus , as long as the solution SuQu and SvQv respectively well-approximate ŨTU and Ṽ TV , F̃ is a good approximation of F . Now , the only unresolved issue is an optimization scheme for equation 4a that keeps the constraints in equation 4b satisfied by leveraging the geometry of the structured solution space .

Summary: This paper studies the problem of canonical correlation analysis (CCA) to find linear combinations that maximize the correlation between two random variables. Following a reformulation of the problem based on on the principal component analysis (PCA) solution, the authors propose a stochastic algorithm for CCA, called RSG+. Then, they show that their approach improves existing algorithms by achieving lower runtime complexity and/or better convergence rate. Finally, they compare the performance of their algorithm on different image datasets.