A Comprehensively Tight Analysis of Gradient Descent for PCA

1 ϵ ) that holds for any real symmetric matrix . More importantly , when the gap ∆ is significantly smaller than the target accuracy ϵ on the objective suboptimality of the final solution , the rate of this type is actually not tight any more , which calls for a worst-case rate . We further give the first worst-case analysis that achieves a rate of convergence at O ( 1ϵ log 1 ϵ ) . The two analyses naturally roll out a comprehensively tight convergence rate at O ( 1max { ∆ , ϵ } log 1 ϵ ) . Particularly , our gap-dependent analysis suggests a new promising learning rate for stochastic variance reduced PCA algorithms . Experiments are conducted to confirm our findings as well . 1 Introduction . Gradient descent is a basic , well-established , and celebrated optimization method for minimizing convex functions [ 11 , 3 ] , but remains far less understood theoretically for non-convex functions though it has been proven useful in practice for many non-convex problems [ 22 , 17 ] . In the past decade , the research on gradient methods for non-convex problems has been gaining increasing attention in optimization and especially machine learning [ 6 , 8 , 4 ] . In this work , we focus on a notable example , namely the Riemannian gradient descent for the leading eigenvector computation of a real symmetric matrix A ∈ Rn×n , which lays the foundation of Krasulina ’ s method for PCA . This problem can be solved by either a traditional projection based method from the numerical algebra , such as the well-known power method or Lanczos algorithm [ 13 ] , or a gradient search based method for the associated optimization problem such as the Riemannian gradient descent [ 18 , 21 ] . The projected gradient descent ( with constant step-sizes ) , as another closely related solver , can be interpreted either as a projection method or as a search method in this case ( see Section 6 for interpretations ) . Compared to projection methods which have arguably been well-understood already [ 13 , 9 , 10 ] , the search method , i.e. , the Riemannian gradient descent , is lacking a clear and deep theoretical understanding on their convergence behaviors . This may largely be because the underlying optimization problem is geodesically non-convex : min x∈Sn−1 f ( x ) = −1 2 x⊤Ax , ( 1 ) where the objective function f ( x ) is constrained on the ( n−1 ) -dimensional unit sphere manifold Sn−1 = { x ∈ Rn×1 : ∥x∥2 = 1 } [ 1 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Previous analyses of the Riemannian gradient descent for Problem ( 1 ) achieve a rate of ( global1 ) convergence at O ( 1∆2 log 1 ϵ ) which depends quadratically on the relative gap ∆ between A ’ s two ( distinct ) largest eigenvalues [ 20 , 19 ] 2 . The poor dependence of the rate on the gap characterizes a loose upper bound on the iteration complexity of the method , due to the artifacts of the used analysis techniques . Ding et al . [ 5 ] gave a tight convergence rate at O ( 1∆ log 1 ϵ ) . However , the analysis is limited to positive definite matrices . This means that the general tight convergence analysis for real symmetric matrices , in fact , has not been settled . Our first goal in this work is to give a general analysis for the tight rate of this type , i.e. , dependent on the gap , which is termed as the ( tight ) gap-dependent analysis hereafter . More importantly , when the gap is significantly smaller than the target accuracy ϵ on the objective sub-optimality of the final solution rather than its point distance to the optimal solutions , the rate of this type is actually not tight any more , which calls for a worst-case rate independent of any gap . Our second goal is to further give the first worst-case or gap-independent analysis being able to achieve a rate of convergence at O ( 1ϵ log 1 ϵ ) . The two analyses naturally roll out a comprehensively tight rate of convergence at O ( 1max { ∆ , ϵ } log 1 ϵ ) . Our analyses are based on novel adaptations of the analyses for projection methods working for constant matrices [ 7 , 10 ] to varying matrices during iterations . We also make a comparison between projection and search methods to help understand their subtle differences . Particularly , the gap-dependent analysis suggests a new step-size for search methods which gives rise to a novel learning rate for stochastic variance reduced PCA algorithms . Experiments verify our theoretical findings and demonstrate the effectiveness of the new learning rate . The main contributions are summarized as follows : • We prove the first comprehensively tight convergence rate of the Riemannian gradient descent for Problem ( 1 ) at O ( 1max { ∆ , ϵ } log 1 ϵ ) . • We propose an adaptive learning rate scheme for stochastic variance reduced PCA algorithms which can dramatically improve their convergence . • We experimentally verify the established theory and demonstrate the effectiveness of the proposed adaptive learning rate . 2 Related Work . We focus on the gradient methods for the leading eigenvector computation . The classic Oja ’ s algorithm and Krasulina ’ s method , as stochastic counterparts of the projected gradient descent and Riemannian gradient descent with diminishing step-sizes , respectively , converge at a rate O ( 1∆2ϵ ) [ 2 ] ( Theorem 1.1 where E [ ΨT ] = O ( c 2 T ) and c = O ( 1 ∆ ) using their notations except for the iteration number T ) . The quadratic dependence on the gap was originally omitted there , probably because the gap was regarded as a constant . However , as argued in Musco et al . [ 10 ] , this is insufficient . Interestingly , there are much fewer theoretical studies of the two methods in the deterministic setting . Absil et al . [ 1 ] introduced the Riemannian gradient descent for Problem ( 1 ) ( Algorithm 2 there ) with the Armijo line search for step-sizes , and implicitly stated a rate at O ( 1∆ log 1 ϵ ) ( Theorem 4.6.3-iii ) which , however , is local ( Theorem 4.6.3-ii ) . Wen et al . [ 17 ] proposed a special retraction for the Riemannian gradient descent based on the Cayley transform ( Algorithm 1 ) . It was empirically found comparable and more stable than the Lanczos algorithm , however , only with the guarantee of convergence to a critical point ( Theorem 2 ) . Since Problem ( 1 ) is geodesically non-convex , theories established for geodesically convex functions [ 23 ] can ’ t be applied unless locally . Note that we only consider global convergence in this work . Xu et al . [ 19 ] specifically analyzed the Riemannian gradient descent for Problem ( 1 ) , and originally stated a rate O ( min { 1∆2 log 1ϵ , 1ϵ } ) ( Theorem 1 ) . However , the constituent part O ( 1ϵ ) resulted from the same mistake 3 as made in Balsubramani et al . [ 2 ] and mentioned at the beginning of this section . This means that their analyses are all gap-dependent and the better one of the rates is O ( 1∆2 log 1 ϵ ) . It is also the case for the work of Xu and Gao 2018 [ 20 ] ( Theorem 4.1 ) . Particularly , Ding et al . [ 5 ] improved the dependence of these rates on the gap from quadratic to linear at O ( 1∆ log 1 ϵ ) . As explained in the previous section , however , it is not general . In 1Only global convergence is considered throughout the paper . 2It needs be noted there that the rate O ( 1 ϵ ) was mistakenly stated and actually should be O ( 1 ∆2ϵ ) according to their analyses . See the next section for explanations . 3Precisely , ψ ( xT ) = O ( c 2 T ) , where c = O ( 1 ∆p ) implied by a ’ s expression , using their notations . addition , to the best of our knowledge , before our work , there has been no worst-case or gap-free analysis of the Riemannian gradient descent for Problem ( 1 ) . Comparisons are summarized in Table 1 . Shamir [ 14 ] proposed a learning rate scheme for stochastic variance reduced PCA algorithm , namely VR-PCA , based on Oja ’ s algorithm . It can be used for VR-PCA ’ s variant as well that is based on Krasulina ’ s method [ 2 , 16 ] . In this work , we put forward yet another learning rate scheme which we find much more effective than that suggested by [ 14 ] for these stochastic variance reduced PCA algorithms . 3 Notions and Notations . The only assumption on the given matrix A , made in our analyses , is real symmetry , i.e. , A⊤ = A ∈ Rn×n . Let A ’ s orthonormal eigenvectors be v1 , v2 , · · · , vn ∈ Sn−1 , corresponding to eigenvalues λ1 , λ2 , · · · , λn ∈ R , i.e. , Avi = λivi , in descending order . If λ1 > λn , then let ∆p ( 1 ≤ p ≤ n− 1 ) be the first nonzero eigenvalue gap , i.e. , ∆p = λp − λp+1 > 0 , such that λ1 = · · · = λp > λp+1 . Particularly , the first nonzero relative eigenvalue gap is defined as ∆ = ∆pλ1−λn if λ1 > λn , otherwise as ∆ = +∞ . The update equations of the two gradient methods for Problem ( 1 ) with constant step-sizes are as follows : Projected gradient descent ( PGD ) : xt+1 = xt − η∇f ( xt ) ∥xt − η∇f ( xt ) ∥2 , ( 2 ) Riemannian gradient descent ( RGD ) : xt+1 = xt − η∇̃f ( xt ) ∥xt − η∇̃f ( xt ) ∥2 , ( 3 ) where ∇f ( x ) =−Ax and ∇̃f ( x ) =− ( I−xx⊤ ) Ax ( throughout the paper I stands for an appropriately sized identity matrix ) represent the Euclidean and Riemannian gradients of f ( x ) , respectively , η > 0 constant step-size , and denominators for normalization , i.e. , xt+1 ∈ Sn−1 . Consider Algorithm that the Riemannian gradient descent for Problem ( 1 ) with constant step-size η starts from a random initial point x0 = x̂0∥x̂0∥2 ∈ S n−1 ( where x̂0 is entry-wise standard Gaussian ) , and then repeatedly runs Equation ( 3 ) for T iterations . We will focus on the analysis of this algorithm next . The results for the projected gradient descent are immediate by a much simpler path through the same analysis . 4 Gap-Dependent Analysis . Let Vp= [ v1 · · · vp ] . The solution space to Problem ( 1 ) then is { Vpα : α ∈ Sp−1 } ≜ V , where S0 = { ±1 } . To study the convergence of iterates to V , it suffices to analyze the principal angle between xt and Vp , i.e. , θ ( xt , Vp ) = cos−1 ( ∥V⊤p xt∥2 ) . Let V−p = [ vp+1 · · · vn ] and □† represent the pseudo-inverse of a matrix . Then tan θ ( x0 , Vp ) = ∥ ( V⊤−px̂0 ) ( V⊤p x̂0 ) †∥2 . Suppose that T iterations are sufficient for sin2 θ ( xt , Vp ) = 1− ∥V⊤p xt∥22 to reach the target accuracy ϵ > 0 , i.e. , sin2 θ ( xT , Vp ) < ϵ . The goal of our job here is to upper bound T with the gap information and show that the bound is tight , namely Theorem 1 The Riemannian gradient descent for Problem ( 1 ) with step-size η = Θ ( 1λ1−λn ) ≤ 1 λ1−λn converges in T = Θ ( 1 ∆ log n ϵ ) iterations , i.e. , sin 2 θ ( xT , Vp ) < ϵ , w.h.p . ( with high probability ) . Proof First , it is trivial and T = 0 for the case that λ1 = λn since f ( x ) is a constant . Thus , it is always assumed that λ1 > λn in what follows . Let ht ( x ) = 1 + η ( x − x⊤t Axt ) and write ht ( A ) = ∑n i=1 ht ( λi ) viv ⊤ i = Vnht ( Σn ) V ⊤ n , where ht ( Σn ) = diag ( ht ( λ1 ) , · · · , ht ( λn ) ) . We can write that xT = hT−1 ( A ) xT−1 ∥hT−1 ( A ) xT−1∥2 = · · · = ∏0 t=T−1 ht ( A ) x0 ∥∏0t=T−1 ht ( A ) x0∥2 = ∏T−1 t=0 ht ( A ) x0 ∥∏T−1t=0 ht ( A ) x0∥2 , ( 4 ) where the second last equality is equivalently renormalized . Noting that the initial point x0 can be expressed as x0= ∑n i=1 ( v ⊤ i x0 ) vi , the last numerator above can be written as∏T−1 t=0 ht ( A ) x0 = ∑n i=1 ( v ⊤ i x0 ) ∏T−1 t=0 ht ( A ) vi = ∑n i=1 ( v ⊤ i x0 ) ∏T−1 t=0 ht ( λi ) vi . Accordingly , it holds that ∥V⊤p ∏T−1 t=0 ht ( A ) x0∥22 = ∑p i=1 ( v ⊤ i x0 ) 2 ∏T−1 t=0 h 2 t ( λi ) , ∥∏T−1t=0 ht ( A ) x0∥22 = ∑n i=1 ( v ⊤ i x0 ) 2 ∏T−1 t=0 h 2 t ( λi ) . By the fact that x⊤Ax ∈ [ λn , λ1 ] for x ∈ Sn−1 ( Corollary 4.7 in [ 15 ] ) , we have that ht ( λi ) ≥ 1 + η ( λn − λ1 ) for all i and t. If 1 + η ( λn − λ1 ) ≥ 0 , i.e. , η ≤ 1λ1−λn , it then holds that ht ( λi ) ≥ 0 for all i and t. When η ≤ 1λ1−λn , we can get that sin2 θ ( xT , Vp ) = 1− ∥V⊤p xT ∥22 = 1− ∑p i=1 ( v ⊤ i x0 ) 2 ∏T−1 t=0 h 2 t ( λi ) ∑n i=1 ( v ⊤ i x0 ) 2 ∏T−1 t=0 h 2 t ( λi ) = ∑n i=p+1 ( v ⊤ i x0 ) 2 ∏T−1 t=0 h 2 t ( λi ) ∑n i=1 ( v ⊤ i x0 ) 2 ∏T−1 t=0 h 2 t ( λi ) ( 5 ) ≤ ∏T−1 t=0 h 2 t ( λp+1 ) ∑n i=p+1 ( v ⊤ i x0 ) 2 ∏T−1 t=0 h 2 t ( λ1 ) ∑p i=1 ( v ⊤ i x0 ) 2 = T−1∏ t=0 ( ht ( λp+1 ) ht ( λ1 ) ) 2 tan2 θ ( x0 , Vp ) ≤ ( 1 + η ( λp+1 − λn ) 1 + η ( λ1 − λn ) ) 2T tan2 θ ( x0 , Vp ) ≤ exp { − 2η∆pT 1 + η ( λ1 − λn ) } tan2 θ ( x0 , Vp ) ≜ ϵ , where the second last inequality have used the fact that 1+η ( λp+1−x ) 1+η ( λ1−x ) is nonnegative and monotonically decreasing for x ∈ [ λn , λ1 ] . Solving the last equality about ϵ for T yields that T = 1+η ( λ1−λn ) 2η∆p log tan2 θ ( x0 , Vp ) ϵ . Note that V ⊤ −px̂0 and ( V ⊤ p x̂0 ) † are entry-wise standard Gaussian since V−p and Vp are orthonormal . By standard Gaussian matrix concentration theory [ 12 ] , w.h.p . ∥V⊤−px̂0∥22 ≤ c1n and ∥ ( V⊤p x̂0 ) †∥22 ≤ c2p for some fixed constants c1 , c2 . We then have w.h.p . that tan2 θ ( x0 , Vp ) ≤ cnp for some fixed constant c and log tan2 θ ( x0 , Vp ) = O ( log ( n ) ) . Hence , T = O ( 1∆ log n ϵ ) for η = Θ ( 1 λ1−λn ) . To justify the tightness of this upper bound , we now show that the Riemannian gradient descent for Problem ( 1 ) requires Ω ( 1∆ log n ϵ ) iterations to converge . To this end , we resume the analysis from Eq . ( 5 ) as follows : sin2 θ ( xT , Vp ) = 1− ∥V⊤p xT ∥22 = ∑n i=p+1 ( v ⊤ i x0 ) 2 ∏T−1 t=0 h 2 t ( λi ) ∑n i=1 ( v ⊤ i x0 ) 2 ∏T−1 t=0 h 2 t ( λi ) ≥ ( v ⊤ p+1x0 ) 2 ∏T−1 t=0 h 2 t ( λp+1 ) ∑p+1 i=1 ( v ⊤ i x0 ) 2 ∏T−1 t=0 h 2 t ( λi ) = 1 1 + ∑p i=1 ( v ⊤ i x0 ) 2 ( v⊤p+1x0 ) 2 · ∏T−1 t=0 ( ht ( λ1 ) ht ( λp+1 ) ) 2 ≥ 1 1 + ∑p i=1 ( v ⊤ i x0 ) 2 ( v⊤p+1x0 ) 2 · ∏T−1 t=0 ( 1 1−η∆p ) 2 = ( 1− η∆p ) 2T tan2 θ̂ 1 + ( 1− η∆p ) 2T tan2 θ̂ ≥ exp { −2η∆pT/ ( 1− η∆p ) } tan 2 θ̂ 1 + exp { −2η∆pT/ ( 1− η∆p ) } tan2 θ̂ , where tan2 θ̂ = ( v⊤p+1x0 ) 2 ∑p i=1 ( v ⊤ i x0 ) 2 and we have used for the last inequality that ( 1 − η∆p ) 2T = exp { 2T log ( 1 − η∆p ) } ≥ exp { − 2η∆pT1−η∆p } by log ( 1 + x ) ≥ x 1+x for x > −1 . Setting exp { − 2η∆pT1−η∆p } tan 2 θ̂ = 2ϵ yields that T = 1−η∆p2η∆p log tan2 θ̂ 2ϵ and sin 2 θ ( xT , Vp ) ≥ 2ϵ1+2ϵ ≥ ϵ , assuming ϵ < 12 . Hence , for η = Θ ( 1 λ1−λn ) , we need T = Ω ( 1 ∆ log n ϵ ) iterations to get sin2 θ ( xT , Vp ) < ϵ , which completes the proof . □ Remark 1 In the gap-dependent analysis above , from the optimization point of view , we care more about the distance of the iterate xt to the optimal solution set V , i.e. , sin2 θ ( xt , Vp ) = 1 2 minv∈V ∥xt − v∥22 , with emphasis on the eigenvector computation . In fact , it is also common to consider the objective sub-optimality gap , i.e. , ( f ( xt ) −minx∈Sn−1 f ( x ) ) = 12 ( λ1 − x⊤t Axt ) , with emphasis on the eigenvalue computation . From the proof of Lemma 4.6 in Xu and Gao 2018 [ 20 ] , we know that λ1 − x⊤TAxT ≤ ( λ1 − λn ) sin2 θ ( xT , Vp ) . By Theorem 1 , with ϵ-rescaling , we then have that λ1 − x⊤TAxT < ϵ for T = O ( 1∆ log nϵ ) .

Summary: The paper studies Riemannian gradient descent for largest eigen-vector computation real symmetric matrices. For matrices with eigen-gap $\Delta$ and (global) convergence error $\epsilon$, the paper gives a gap-dependent convergence rate of $\frac{1}{\Delta} \log(1/\epsilon)$ and gap-independent analysis which gives a convergence rate of $\frac{1}{\epsilon} \log(1/\epsilon)$. This improves upon previous work that achieves a $\frac{1}{\Delta^2} \log(1/\epsilon)$ rate and generalizes related work that that achieves the same rate for projected gradient descent, power iteration and Riemannian gradient descent restricted to PSD matrices.  The paper highlights an omission in previous work that mistakenly stated a rate of $\frac{1}{\epsilon} \log(1/\epsilon)$ which is in fact $\frac{1}{\Delta^2\epsilon} \log(1/\epsilon)$.   Experiments comparing the power iteration, projected gradient descent and RGD are done for proposed adaptively chosen step-sizes.