An Online Riemannian PCA for Stochastic Canonical Correlation Analysis

t ) convergence rate ( where d is the dimensionality ) or only extract the top 1 component with O ( 1t ) convergence rate . In contrast , our algorithm offers an improvement : it achieves O ( d2k ) runtime complexity per iteration for extracting the top k canonical components with O ( 1t ) convergence rate . While our paper focuses more on the formulation and the algorithm , our experiments show that the empirical behavior on common datasets is quite promising . We also explore a potential application in training fair models with missing sensitive attributes . 1 Introduction . Canonical correlation analysis ( CCA ) is a classical method for evaluating correlations between two sets of variables . It is commonly used in unsupervised multi-view learning , where the multiple views of the data may correspond to image , text , audio and so on , Rupnik and Shawe-Taylor [ 2010 ] , Chaudhuri et al . [ 2009 ] , Luo et al . [ 2015 ] , and has been applied to manifold-valued data also Kim et al . [ 2014 ] . Classical formulations have also been extended to leverage advances in representation learning , for example , Andrew et al . [ 2013 ] showed how the CCA can be interfaced with deep neural networks enabling modern use cases . Many results over the last few years have used CCA or its variants for problems including measuring representational similarity in deep neural networks Morcos et al . [ 2018 ] and speech recognition Couture et al . [ 2019 ] . The goal in CCA is to find linear combinations within two random variables X and Y which have maximum correlation with each other . Formally , the CCA problem is defined as follows . Let X ∈ RN×dx and Y ∈ RN×dy be N samples respectively drawn from pair of random variables X ( dx-variate random variable ) and Y ( dy-variate random variable ) , with unknown joint probability distribution . The goal is to find the projection matrices U ∈ Rdx×k and V ∈ Rdy×k , with k ≤ min { dx , dy } , such that the correlation is maximized : max U , V F = trace ( UTCXY V ) s.t . UTCXU = Ik , V TCY V = Ik ( 1 ) Here , CX = 1NX TX and CY = 1N Y TY are the sample covariance matrices , and CXY = 1NX TY denotes the sample cross-covariance . The objective in ( 1 ) is the expected cross-correlation in the projected space and the constraints specify that different canonical components should be decorrelated . Let us define the whitened covariance ∗Equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . T : = C −1/2 X CXY C −1/2 Y and Φk ( and Ψk ) contains the top-k left ( and right ) singular vectors of T . It is known Golub and Zha [ 1992 ] that the optimum of ( 1 ) is achieved at U∗ = C−1/2X Φk , V ∗ = C −1/2 Y Ψk . We can compute U ∗ , V ∗ by applying a k-truncated SVD to T . Runtime and memory considerations . The above procedure is simple but is only feasible when the data matrices are small . In modern applications , not only are the datasets large but also the dimension d ( let d = max { dx , dy } ) of each sample can be large , especially if representations are learned using deep models . As a result , the resource needs of the algorithm can be high . This has motivated the study of stochastic optimization routines for solving CCA , and many efficient strategies have been proposed . For example , Ge et al . [ 2016 ] , Wang et al . [ 2016 ] present Empirical Risk Minimization ( ERM ) models which optimize the empirical objective . More recently , Gao et al . [ 2019 ] , Bhatia et al . [ 2018 ] , Arora et al . [ 2017 ] describe proposals that optimize the population objective . To summarize the approaches , if we are satisfied with the top 1 component of CCA , effective schemes with O ( 1t ) convergence rate are available by utilizing either extensions of the Oja ’ s rule Oja [ 1982 ] to the generalized eigenvalue problem Bhatia et al . [ 2018 ] or the alternating SVRG algorithm Gao et al . [ 2019 ] . Otherwise , a stochastic approach will use an explicit whitening operation which can cost d3 operations for each iteration Arora et al . [ 2017 ] and the convergence rate for the stochastic scheme depends on its specific steps and calculations , e.g. , O ( 1√ t ) in Arora et al . [ 2017 ] ( Thm 2.3 , pp 5 ) . Observation . Most approaches either directly optimize ( 1 ) or instead a reparameterized or regularized form Ge et al . [ 2016 ] , Allen-Zhu and Li [ 2016 ] , Arora et al . [ 2017 ] . Often , the search space for U and V corresponds to the entire Rd×k ( ignoring the constraints for the moment ) . But if the formulation could be cast in a form which involved approximately writing U and V as a product of structured matrices , we may be able to obtain specialized routines which are tailored to exploit those properties . Such a reformulation is not difficult to derive – where the matrices used to express U and V can be identified as objects that live in well studied geometric spaces . Then , utilizing the geometry of the space and borrowing relevant tools from differential geometry could lead to an efficient approximate scheme for top-k CCA which optimizes the population objective in a streaming fashion . Contributions . ( a ) First , we re-parameterize the top-k CCA problem as an optimization problem on specific matrix manifolds , and show that it is equivalent to the original formulation in ( 1 ) . ( b ) Informed by the geometry of the manifold , we derive stochastic gradient descent ( SGD ) algorithms for solving the re-parameterized problem with O ( d2k ) cost per iteration and provide convergence rate guarantees . ( c ) This analysis gives a direct mechanism to obtain an upper bound on the number of iterations needed to guarantee an ϵ error w.r.t . the population objective for the CCA problem . ( d ) The algorithm works in a streaming manner so it easily scales to large datasets and we do not need to assume access to the full dataset at the outset . ( e ) We present empirical evidence for the standard CCA model and the DeepCCA setting Andrew et al . [ 2013 ] , describing advantages and limitations . 2 Stochastic CCA : Reformulation , Algorithm and Analysis . Let us review the objective for CCA as given in ( 1 ) . We denote X ∈ RN×dx as the matrix consisting of the samples { xi } drawn from a zero mean random variable X ∼ X and Y ∈ RN×dy denotes the matrix consisting of samples { yi } drawn from a zero mean random variable Y ∼ Y . For simplicity , we assume that dx = dy = d although the results hold for general dx and dy . Also recall that CX ( and CY resp . ) is the covariance matrix of X ( and Y resp . ) and CXY is the cross-covariance matrix between X and Y . Let U ∈ Rd×k ( V ∈ Rd×k ) be the matrix consisting of { uj } ( { vj } ) , where ( { uj } , { vj } ) are the canonical directions . The constraints in ( 1 ) are called whitening constraints . Reformulation : In the CCA formulation , the matrices consisting of canonical correlation directions , i.e. , U and V , are unconstrained , hence the search space is the entire Rd×k . Now , we reformulate the CCA objective by reparameterizing U and V . In order to do that , let us take a brief detour and recall the objective function of principal component analysis ( PCA ) : Û = argmax U ′ trace ( R̂ ) subject to R̂ = U ′TCXU ′ ; U ′TU ′ = Ik ( 2 ) Observe that by performing PCA and assigning U = Û R̂ − 1/2 in ( 1 ) ( analogous for V using CY ) , we can satisfy the whitening constraint . Of course , writing U = Û R̂ − 1/2 does satisfy the whitening constraint , but such a U ( and V ) will not maximize trace ( UTCXY V ) , the objective of ( 1 ) . Hence , additional work beyond the PCA solution is needed . Let us start from R̂ but relax the PCA solution by using an arbitrary R̃ instead of diagonal R̂ ( this will still satisfy the whitening constraint ) . Write U = Ũ R̃ with ŨT Ũ = Ik and R̃ ∈ Rk×k . Thus we can approximate CCA objective ( we will later check how good this approximation is ) as max Ũ , Ṽ ∈St ( k , d ) Ru , Rv∈Rk×k U=ŨRu ; V=Ṽ Rv trace ( UTCXY V ) ︸ ︷︷ ︸ F̃ + trace ( ŨTCX Ũ ) + trace ( Ṽ TCY Ṽ ) ︸ ︷︷ ︸ F̃pca s.t . U TCXU=Ik V TCY V=Ik ( 3 ) Here , St ( k , d ) denotes the manifold consisting of d× k ( with k ≤ d ) column orthonormal matrices , i.e. , St ( k , d ) = { X ∈ Rd×k|XTX = Ik } . Observe that in ( 3 ) , we approximate the optimal U and V as a linear combination of Ũ and Ṽ respectively . Thus , the aforementioned PCA solution can act as a feasible initial solution for ( 3 ) . As the choice of Ru and Rv is arbitrary , we can further reparameterize these matrices by constraining them to be full rank ( of rank k ) and using the RQ decomposition Golub and Reinsch [ 1971 ] which gives us the following reformulation . A Reformulation for CCA max Ũ , Ṽ , Su , Sv , Qu , Qv U=ŨSuQu ; V=Ṽ SvQv trace ( UTCXY V ) ︸ ︷︷ ︸ F̃ + trace ( ŨTCX Ũ ) + trace ( Ṽ TCY Ṽ ) ︸ ︷︷ ︸ F̃pca ( 4a ) subject to UTCXU = Ik V TCY V = Ik ( 4b ) Ũ , Ṽ ∈ St ( k , d ) ; Qu , Qv ∈ SO ( k ) Su , Sv is upper triangular Here , SO ( k ) is the space of k × k special orthogonal matrices , i.e. , SO ( k ) = { X ∈ Rk×k|XTX = Ik ; det ( X ) = 1 } . Before evaluating how good the aforementioned approximation is , we first point out some useful properties of the reformulation ( 4 ) : ( a ) in the reparametrization of U and V , all components are structured , hence , the search space becomes a subset of Rk×k ( b ) we can essentially initialize with a PCA solution and then try to optimize ( 4 ) via some scheme . Why ( 4 ) helps ? First , we note that CCA seeks to maximize the total correlation under the constraint that different components are decorrelated . One difficulty in the optimization is to ensure decorrelation , which leads to a higher complexity in existing streaming CCA algorithms . On the contrary , in ( 4 ) , we separate ( 1 ) into finding the PCs , Ũ , Ṽ ( by adding the variance maximization terms ) and finding the linear combination ( SuQu and SvQv ) of the principal directions . After optimizing for these variables , the whitening constraints are , up to a rescaling , automatically satisfied . Here , we can ( almost ) utilize an efficient off-the-shelf streaming PCA algorithm . We will defer describing the specific details of the individual steps until the next sub-section . First , we will show why substituting ( 1 ) with ( 4 ) is sensible under some assumptions . Why the solution of the reformulation makes sense ? We start by stating some mild assumptions needed for the analysis . Assumptions : ( a ) The random variables X ∼ N ( 0 , Σx ) and Y ∼ N ( 0 , Σy ) with covariance Σx ⪯ cId and covariance Σy ⪯ cId for some c > 0 . ( b ) The samples X and Y drawn from X and Y respectively have zero mean . ( c ) For a given k ≤ d , Σx , Σy have non-zero top-k eigen values . We show how the presented solution , assuming access to an effective numerical procedure , approximates the CCA problem presented in ( 1 ) . We formally state the result in the following theorem with a sketch of proof ( appendix includes the full proof ) by first stating the following proposition . Definition 1 . A random variable X is called sub-Gaussian if the norm given by ∥X∥⋆ : = inf { d ≥ 0|EX [ exp ( trace ( XTX ) /d2 ) ] ≤ 2 } is finite . Let U ∈ Rd×k , then XU is sub-Gaussian Vershynin [ 2017 ] . Proposition 1 ( Reiß et al . [ 2020 ] ) . Let X be a random variable which follows a sub-Gaussian distribution . Let X̂ be the approximation of X ∈ RN×d ( samples drawn from X ) with the top-k principal vectors . Let C̃X be the covariance of X̂ . Also , assume that λi is the ith eigen value of CX for i = 1 , · · · , d− 1 and λi ≥ λi+1 for all i . Then , the PCA reconstruction error , denoted by Ek = EX∥X − X̂∥ ( in the Frobenius norm sense ) can be upper bounded as follows Ek ≤ min ( √ 2k∥∆∥2 , 2∥∆∥22 λk − λk+1 ) , ∆ = CX − C̃X . The aforementioned proposition suggests that the error between the data matrix X and the reconstructed data matrix X̂ using the top-k principal vectors is bounded . Recall from ( 1 ) and ( 4 ) that the optimal value of the true and approximated CCA objective is denoted by F and F̃ respectively . The following theorem states that we can bound the error , E = ∥F − F̃∥ ( proof in the appendix ) . In other words , if we start from PCA solution and can successfully optimize ( 4 ) without leaving the feasible set , we will obtain a good solution . Theorem 1 . Using the hypothesis and assumptions above , the approximation error E = ∥F − F̃∥ as a function of N is bounded and goes to zero as N → ∞ while the whitening constraints in equation 4b are satisfied . Sketch of the Proof . Let U∗ and V ∗ be the true solution of CCA , i.e. , of ( 1 ) . Let U = ŨSuQu , V = Ṽ SvQv be the solution of ( 4 ) , with Ũ , Ṽ be the PCA solutions of X and Y respectively . Let X̂ = XŨŨT and Ŷ = Y Ṽ Ṽ T be the reconstruction of X and Y using principal vectors . Let SuQu = ŨTU∗ and SvQv = Ṽ TV ∗ . Then we can write F̃ = trace ( UTCXY V ) = trace ( 1 N ( X̂U∗ ) T Ŷ V ∗ ) . Similarly we can write F = trace ( 1 N ( XU ∗ ) T Y V ∗ ) . As X̂ and Ŷ are the approximation of X and Y respectively using the principal vectors , we use Prop . 1 to bound the error ∥F − F̃∥ . Now observe that X̂U can be rewritten into XŨŨTU ( similar for Ŷ V ) . Thus , as long as the solution SuQu and SvQv respectively well-approximate ŨTU and Ṽ TV , F̃ is a good approximation of F . Now , the only unresolved issue is an optimization scheme for equation 4a that keeps the constraints in equation 4b satisfied by leveraging the geometry of the structured solution space .

Summary: This paper proposes a stochastic algorithm for canonical correlation analysis (CCA). It is based on a reformulation of CCA so that it can use existing tools in online PCA and manifold optimization. Some convergence analyses are shown, and an abundance of experiments (mostly involving deep neural nets) are used to support the claims.