A Comprehensively Tight Analysis of Gradient Descent for PCA

1 ϵ ) that holds for any real symmetric matrix . More importantly , when the gap ∆ is significantly smaller than the target accuracy ϵ on the objective suboptimality of the final solution , the rate of this type is actually not tight any more , which calls for a worst-case rate . We further give the first worst-case analysis that achieves a rate of convergence at O ( 1ϵ log 1 ϵ ) . The two analyses naturally roll out a comprehensively tight convergence rate at O ( 1max { ∆ , ϵ } log 1 ϵ ) . Particularly , our gap-dependent analysis suggests a new promising learning rate for stochastic variance reduced PCA algorithms . Experiments are conducted to confirm our findings as well . 1 Introduction . Gradient descent is a basic , well-established , and celebrated optimization method for minimizing convex functions [ 11 , 3 ] , but remains far less understood theoretically for non-convex functions though it has been proven useful in practice for many non-convex problems [ 22 , 17 ] . In the past decade , the research on gradient methods for non-convex problems has been gaining increasing attention in optimization and especially machine learning [ 6 , 8 , 4 ] . In this work , we focus on a notable example , namely the Riemannian gradient descent for the leading eigenvector computation of a real symmetric matrix A ∈ Rn×n , which lays the foundation of Krasulina ’ s method for PCA . This problem can be solved by either a traditional projection based method from the numerical algebra , such as the well-known power method or Lanczos algorithm [ 13 ] , or a gradient search based method for the associated optimization problem such as the Riemannian gradient descent [ 18 , 21 ] . The projected gradient descent ( with constant step-sizes ) , as another closely related solver , can be interpreted either as a projection method or as a search method in this case ( see Section 6 for interpretations ) . Compared to projection methods which have arguably been well-understood already [ 13 , 9 , 10 ] , the search method , i.e. , the Riemannian gradient descent , is lacking a clear and deep theoretical understanding on their convergence behaviors . This may largely be because the underlying optimization problem is geodesically non-convex : min x∈Sn−1 f ( x ) = −1 2 x⊤Ax , ( 1 ) where the objective function f ( x ) is constrained on the ( n−1 ) -dimensional unit sphere manifold Sn−1 = { x ∈ Rn×1 : ∥x∥2 = 1 } [ 1 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Previous analyses of the Riemannian gradient descent for Problem ( 1 ) achieve a rate of ( global1 ) convergence at O ( 1∆2 log 1 ϵ ) which depends quadratically on the relative gap ∆ between A ’ s two ( distinct ) largest eigenvalues [ 20 , 19 ] 2 . The poor dependence of the rate on the gap characterizes a loose upper bound on the iteration complexity of the method , due to the artifacts of the used analysis techniques . Ding et al . [ 5 ] gave a tight convergence rate at O ( 1∆ log 1 ϵ ) . However , the analysis is limited to positive definite matrices . This means that the general tight convergence analysis for real symmetric matrices , in fact , has not been settled . Our first goal in this work is to give a general analysis for the tight rate of this type , i.e. , dependent on the gap , which is termed as the ( tight ) gap-dependent analysis hereafter . More importantly , when the gap is significantly smaller than the target accuracy ϵ on the objective sub-optimality of the final solution rather than its point distance to the optimal solutions , the rate of this type is actually not tight any more , which calls for a worst-case rate independent of any gap . Our second goal is to further give the first worst-case or gap-independent analysis being able to achieve a rate of convergence at O ( 1ϵ log 1 ϵ ) . The two analyses naturally roll out a comprehensively tight rate of convergence at O ( 1max { ∆ , ϵ } log 1 ϵ ) . Our analyses are based on novel adaptations of the analyses for projection methods working for constant matrices [ 7 , 10 ] to varying matrices during iterations . We also make a comparison between projection and search methods to help understand their subtle differences . Particularly , the gap-dependent analysis suggests a new step-size for search methods which gives rise to a novel learning rate for stochastic variance reduced PCA algorithms . Experiments verify our theoretical findings and demonstrate the effectiveness of the new learning rate . The main contributions are summarized as follows : • We prove the first comprehensively tight convergence rate of the Riemannian gradient descent for Problem ( 1 ) at O ( 1max { ∆ , ϵ } log 1 ϵ ) . • We propose an adaptive learning rate scheme for stochastic variance reduced PCA algorithms which can dramatically improve their convergence . • We experimentally verify the established theory and demonstrate the effectiveness of the proposed adaptive learning rate . 2 Related Work . We focus on the gradient methods for the leading eigenvector computation . The classic Oja ’ s algorithm and Krasulina ’ s method , as stochastic counterparts of the projected gradient descent and Riemannian gradient descent with diminishing step-sizes , respectively , converge at a rate O ( 1∆2ϵ ) [ 2 ] ( Theorem 1.1 where E [ ΨT ] = O ( c 2 T ) and c = O ( 1 ∆ ) using their notations except for the iteration number T ) . The quadratic dependence on the gap was originally omitted there , probably because the gap was regarded as a constant . However , as argued in Musco et al . [ 10 ] , this is insufficient . Interestingly , there are much fewer theoretical studies of the two methods in the deterministic setting . Absil et al . [ 1 ] introduced the Riemannian gradient descent for Problem ( 1 ) ( Algorithm 2 there ) with the Armijo line search for step-sizes , and implicitly stated a rate at O ( 1∆ log 1 ϵ ) ( Theorem 4.6.3-iii ) which , however , is local ( Theorem 4.6.3-ii ) . Wen et al . [ 17 ] proposed a special retraction for the Riemannian gradient descent based on the Cayley transform ( Algorithm 1 ) . It was empirically found comparable and more stable than the Lanczos algorithm , however , only with the guarantee of convergence to a critical point ( Theorem 2 ) . Since Problem ( 1 ) is geodesically non-convex , theories established for geodesically convex functions [ 23 ] can ’ t be applied unless locally . Note that we only consider global convergence in this work . Xu et al . [ 19 ] specifically analyzed the Riemannian gradient descent for Problem ( 1 ) , and originally stated a rate O ( min { 1∆2 log 1ϵ , 1ϵ } ) ( Theorem 1 ) . However , the constituent part O ( 1ϵ ) resulted from the same mistake 3 as made in Balsubramani et al . [ 2 ] and mentioned at the beginning of this section . This means that their analyses are all gap-dependent and the better one of the rates is O ( 1∆2 log 1 ϵ ) . It is also the case for the work of Xu and Gao 2018 [ 20 ] ( Theorem 4.1 ) . Particularly , Ding et al . [ 5 ] improved the dependence of these rates on the gap from quadratic to linear at O ( 1∆ log 1 ϵ ) . As explained in the previous section , however , it is not general . In 1Only global convergence is considered throughout the paper . 2It needs be noted there that the rate O ( 1 ϵ ) was mistakenly stated and actually should be O ( 1 ∆2ϵ ) according to their analyses . See the next section for explanations . 3Precisely , ψ ( xT ) = O ( c 2 T ) , where c = O ( 1 ∆p ) implied by a ’ s expression , using their notations . addition , to the best of our knowledge , before our work , there has been no worst-case or gap-free analysis of the Riemannian gradient descent for Problem ( 1 ) . Comparisons are summarized in Table 1 . Shamir [ 14 ] proposed a learning rate scheme for stochastic variance reduced PCA algorithm , namely VR-PCA , based on Oja ’ s algorithm . It can be used for VR-PCA ’ s variant as well that is based on Krasulina ’ s method [ 2 , 16 ] . In this work , we put forward yet another learning rate scheme which we find much more effective than that suggested by [ 14 ] for these stochastic variance reduced PCA algorithms . 3 Notions and Notations . The only assumption on the given matrix A , made in our analyses , is real symmetry , i.e. , A⊤ = A ∈ Rn×n . Let A ’ s orthonormal eigenvectors be v1 , v2 , · · · , vn ∈ Sn−1 , corresponding to eigenvalues λ1 , λ2 , · · · , λn ∈ R , i.e. , Avi = λivi , in descending order . If λ1 > λn , then let ∆p ( 1 ≤ p ≤ n− 1 ) be the first nonzero eigenvalue gap , i.e. , ∆p = λp − λp+1 > 0 , such that λ1 = · · · = λp > λp+1 . Particularly , the first nonzero relative eigenvalue gap is defined as ∆ = ∆pλ1−λn if λ1 > λn , otherwise as ∆ = +∞ . The update equations of the two gradient methods for Problem ( 1 ) with constant step-sizes are as follows : Projected gradient descent ( PGD ) : xt+1 = xt − η∇f ( xt ) ∥xt − η∇f ( xt ) ∥2 , ( 2 ) Riemannian gradient descent ( RGD ) : xt+1 = xt − η∇̃f ( xt ) ∥xt − η∇̃f ( xt ) ∥2 , ( 3 ) where ∇f ( x ) =−Ax and ∇̃f ( x ) =− ( I−xx⊤ ) Ax ( throughout the paper I stands for an appropriately sized identity matrix ) represent the Euclidean and Riemannian gradients of f ( x ) , respectively , η > 0 constant step-size , and denominators for normalization , i.e. , xt+1 ∈ Sn−1 . Consider Algorithm that the Riemannian gradient descent for Problem ( 1 ) with constant step-size η starts from a random initial point x0 = x̂0∥x̂0∥2 ∈ S n−1 ( where x̂0 is entry-wise standard Gaussian ) , and then repeatedly runs Equation ( 3 ) for T iterations . We will focus on the analysis of this algorithm next . The results for the projected gradient descent are immediate by a much simpler path through the same analysis . 4 Gap-Dependent Analysis . Let Vp= [ v1 · · · vp ] . The solution space to Problem ( 1 ) then is { Vpα : α ∈ Sp−1 } ≜ V , where S0 = { ±1 } . To study the convergence of iterates to V , it suffices to analyze the principal angle between xt and Vp , i.e. , θ ( xt , Vp ) = cos−1 ( ∥V⊤p xt∥2 ) . Let V−p = [ vp+1 · · · vn ] and □† represent the pseudo-inverse of a matrix . Then tan θ ( x0 , Vp ) = ∥ ( V⊤−px̂0 ) ( V⊤p x̂0 ) †∥2 . Suppose that T iterations are sufficient for sin2 θ ( xt , Vp ) = 1− ∥V⊤p xt∥22 to reach the target accuracy ϵ > 0 , i.e. , sin2 θ ( xT , Vp ) < ϵ . The goal of our job here is to upper bound T with the gap information and show that the bound is tight , namely Theorem 1 The Riemannian gradient descent for Problem ( 1 ) with step-size η = Θ ( 1λ1−λn ) ≤ 1 λ1−λn converges in T = Θ ( 1 ∆ log n ϵ ) iterations , i.e. , sin 2 θ ( xT , Vp ) < ϵ , w.h.p . ( with high probability ) . Proof First , it is trivial and T = 0 for the case that λ1 = λn since f ( x ) is a constant . Thus , it is always assumed that λ1 > λn in what follows . Let ht ( x ) = 1 + η ( x − x⊤t Axt ) and write ht ( A ) = ∑n i=1 ht ( λi ) viv ⊤ i = Vnht ( Σn ) V ⊤ n , where ht ( Σn ) = diag ( ht ( λ1 ) , · · · , ht ( λn ) ) . We can write that xT = hT−1 ( A ) xT−1 ∥hT−1 ( A ) xT−1∥2 = · · · = ∏0 t=T−1 ht ( A ) x0 ∥∏0t=T−1 ht ( A ) x0∥2 = ∏T−1 t=0 ht ( A ) x0 ∥∏T−1t=0 ht ( A ) x0∥2 , ( 4 ) where the second last equality is equivalently renormalized . Noting that the initial point x0 can be expressed as x0= ∑n i=1 ( v ⊤ i x0 ) vi , the last numerator above can be written as∏T−1 t=0 ht ( A ) x0 = ∑n i=1 ( v ⊤ i x0 ) ∏T−1 t=0 ht ( A ) vi = ∑n i=1 ( v ⊤ i x0 ) ∏T−1 t=0 ht ( λi ) vi . Accordingly , it holds that ∥V⊤p ∏T−1 t=0 ht ( A ) x0∥22 = ∑p i=1 ( v ⊤ i x0 ) 2 ∏T−1 t=0 h 2 t ( λi ) , ∥∏T−1t=0 ht ( A ) x0∥22 = ∑n i=1 ( v ⊤ i x0 ) 2 ∏T−1 t=0 h 2 t ( λi ) . By the fact that x⊤Ax ∈ [ λn , λ1 ] for x ∈ Sn−1 ( Corollary 4.7 in [ 15 ] ) , we have that ht ( λi ) ≥ 1 + η ( λn − λ1 ) for all i and t. If 1 + η ( λn − λ1 ) ≥ 0 , i.e. , η ≤ 1λ1−λn , it then holds that ht ( λi ) ≥ 0 for all i and t. When η ≤ 1λ1−λn , we can get that sin2 θ ( xT , Vp ) = 1− ∥V⊤p xT ∥22 = 1− ∑p i=1 ( v ⊤ i x0 ) 2 ∏T−1 t=0 h 2 t ( λi ) ∑n i=1 ( v ⊤ i x0 ) 2 ∏T−1 t=0 h 2 t ( λi ) = ∑n i=p+1 ( v ⊤ i x0 ) 2 ∏T−1 t=0 h 2 t ( λi ) ∑n i=1 ( v ⊤ i x0 ) 2 ∏T−1 t=0 h 2 t ( λi ) ( 5 ) ≤ ∏T−1 t=0 h 2 t ( λp+1 ) ∑n i=p+1 ( v ⊤ i x0 ) 2 ∏T−1 t=0 h 2 t ( λ1 ) ∑p i=1 ( v ⊤ i x0 ) 2 = T−1∏ t=0 ( ht ( λp+1 ) ht ( λ1 ) ) 2 tan2 θ ( x0 , Vp ) ≤ ( 1 + η ( λp+1 − λn ) 1 + η ( λ1 − λn ) ) 2T tan2 θ ( x0 , Vp ) ≤ exp { − 2η∆pT 1 + η ( λ1 − λn ) } tan2 θ ( x0 , Vp ) ≜ ϵ , where the second last inequality have used the fact that 1+η ( λp+1−x ) 1+η ( λ1−x ) is nonnegative and monotonically decreasing for x ∈ [ λn , λ1 ] . Solving the last equality about ϵ for T yields that T = 1+η ( λ1−λn ) 2η∆p log tan2 θ ( x0 , Vp ) ϵ . Note that V ⊤ −px̂0 and ( V ⊤ p x̂0 ) † are entry-wise standard Gaussian since V−p and Vp are orthonormal . By standard Gaussian matrix concentration theory [ 12 ] , w.h.p . ∥V⊤−px̂0∥22 ≤ c1n and ∥ ( V⊤p x̂0 ) †∥22 ≤ c2p for some fixed constants c1 , c2 . We then have w.h.p . that tan2 θ ( x0 , Vp ) ≤ cnp for some fixed constant c and log tan2 θ ( x0 , Vp ) = O ( log ( n ) ) . Hence , T = O ( 1∆ log n ϵ ) for η = Θ ( 1 λ1−λn ) . To justify the tightness of this upper bound , we now show that the Riemannian gradient descent for Problem ( 1 ) requires Ω ( 1∆ log n ϵ ) iterations to converge . To this end , we resume the analysis from Eq . ( 5 ) as follows : sin2 θ ( xT , Vp ) = 1− ∥V⊤p xT ∥22 = ∑n i=p+1 ( v ⊤ i x0 ) 2 ∏T−1 t=0 h 2 t ( λi ) ∑n i=1 ( v ⊤ i x0 ) 2 ∏T−1 t=0 h 2 t ( λi ) ≥ ( v ⊤ p+1x0 ) 2 ∏T−1 t=0 h 2 t ( λp+1 ) ∑p+1 i=1 ( v ⊤ i x0 ) 2 ∏T−1 t=0 h 2 t ( λi ) = 1 1 + ∑p i=1 ( v ⊤ i x0 ) 2 ( v⊤p+1x0 ) 2 · ∏T−1 t=0 ( ht ( λ1 ) ht ( λp+1 ) ) 2 ≥ 1 1 + ∑p i=1 ( v ⊤ i x0 ) 2 ( v⊤p+1x0 ) 2 · ∏T−1 t=0 ( 1 1−η∆p ) 2 = ( 1− η∆p ) 2T tan2 θ̂ 1 + ( 1− η∆p ) 2T tan2 θ̂ ≥ exp { −2η∆pT/ ( 1− η∆p ) } tan 2 θ̂ 1 + exp { −2η∆pT/ ( 1− η∆p ) } tan2 θ̂ , where tan2 θ̂ = ( v⊤p+1x0 ) 2 ∑p i=1 ( v ⊤ i x0 ) 2 and we have used for the last inequality that ( 1 − η∆p ) 2T = exp { 2T log ( 1 − η∆p ) } ≥ exp { − 2η∆pT1−η∆p } by log ( 1 + x ) ≥ x 1+x for x > −1 . Setting exp { − 2η∆pT1−η∆p } tan 2 θ̂ = 2ϵ yields that T = 1−η∆p2η∆p log tan2 θ̂ 2ϵ and sin 2 θ ( xT , Vp ) ≥ 2ϵ1+2ϵ ≥ ϵ , assuming ϵ < 12 . Hence , for η = Θ ( 1 λ1−λn ) , we need T = Ω ( 1 ∆ log n ϵ ) iterations to get sin2 θ ( xT , Vp ) < ϵ , which completes the proof . □ Remark 1 In the gap-dependent analysis above , from the optimization point of view , we care more about the distance of the iterate xt to the optimal solution set V , i.e. , sin2 θ ( xt , Vp ) = 1 2 minv∈V ∥xt − v∥22 , with emphasis on the eigenvector computation . In fact , it is also common to consider the objective sub-optimality gap , i.e. , ( f ( xt ) −minx∈Sn−1 f ( x ) ) = 12 ( λ1 − x⊤t Axt ) , with emphasis on the eigenvalue computation . From the proof of Lemma 4.6 in Xu and Gao 2018 [ 20 ] , we know that λ1 − x⊤TAxT ≤ ( λ1 − λn ) sin2 θ ( xT , Vp ) . By Theorem 1 , with ϵ-rescaling , we then have that λ1 − x⊤TAxT < ϵ for T = O ( 1∆ log nϵ ) .

Summary: In this work authors presented two analyses of the Riemannian gradient descent for eigenvector computation.  First authors provided a general tight analysis that holds for any real symmetric matrix. In addition, authors further give the first worst-case analysis that achieves a rate of convergence.  For real-world application, this gap-dependent analysis suggests a new promising learning rate for stochastic variance reduced PCA algorithms. Authors provided experiments to confirm the findings.  Overall this is a solid paper with both theoretical analysis and empirical studies.