Greedy and Random Quasi-Newton Methods with Faster Explicit Superlinear Convergence

1 Introduction . We study the superlinear convergence of famous quasi-Newton methods that replace the exact Hessian applied in classical Newton methods with certain approximations . The approximation is updated in iterations based on some special formulas from the previous variation . There exist various quasiNewton algorithms with different Hessian approximations . The three most popular versions are the Davidon-Fletcher-Powell ( DFP ) method [ 7 , 10 ] , the Broyden-Fletcher-Goldfarb-Shanno ( BFGS ) method [ 2 , 3 , 9 , 11 , 21 ] , and the Symmetric Rank 1 ( SR1 ) method [ 1 , 7 ] , all of which belong to the Broyden family [ 1 ] of quasi-Newton algorithms . The most attractive property of quasi-Newton methods is their superlinear convergence , which can trace back to the 1970s [ 4 , 8 , 17 ] . However , the superlinear convergence rates provided in prior work are asymptotic [ 5 , 13 , 15 , 22 , 24 ] . The results only show that the ratio of successive residuals tends to zero as the running iterations approach to infinity , while still lacking a specific superlinear convergence rate . Recently , Rodomanov and Nesterov [ 19 ] gave the first explicit local superlinear convergence for their proposed new quasi-Newton methods . They introduced greedy quasi-Newton updates by greedily selecting from basis vectors to maximize a certain measure of progress , and established an explicit non-asymptotic bound on the local superlinear convergence rate correspondingly . Their proofs are mainly applicable to the DFP methods because they reduced all possible Broyden family to the DFP update based on the monotonicity property ( see Lemma 2.2 ) . However , the SR1 and BFGS updates are more popular and faster than the DFP update in practice , which also has been verified in their experiments . In addition , Rodomanov and Nesterov [ 19 ] also discovered that the randomized methods do not lose superlinear convergence but they did not provide theoretical guarantees . ∗Equal Contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Along this line , there are other results of local superlinear convergence analysis . Rodomanov and Nesterov [ 20 ] analyzed the well-known DFP and BFGS methods , using a standard Hessian update direction through the previous variation . They demonstrated faster initial convergence rates , while slower final rates compared to Rodomanov and Nesterov [ 19 ] ’ s results . Rodomanov and Nesterov [ 18 ] improved Rodomanov and Nesterov [ 20 ] ’ s results by reducing the dependence of the condition number κ to lnκ , though having similar worse long-history behavior . Jin and Mokhtari [ 14 ] provided a non-asymptotic dimension-free superlinear convergence rate of the original Broyden family when the initial Hessian approximation is also good enough . In this work , we prove faster convergence rates for greedy and random quasi-Newton algorithms , particularly on the SR1 and BFGS methods . We present our contribution as follows : • First , in the setting of approximating an exact Hessian , we use two update methods : 1 ) greedy update modified from Rodomanov and Nesterov [ 19 ] ; 2 ) random update that randomly selects the updating direction from any spherically symmetric distribution . We show that both methods share a faster condition-number-free convergence . Particularly , we obtain the superlinear convergence rate O ( 1− kn ) for SR1 update , and the linear convergence rate O ( ( 1− 1n ) k ) for BFGS update , where k is the current iteration , and n is the dimension of parameters . Both findings improve the original convergence rate O ( ( 1− 1nκ ) k ) [ 19 ] . • Second , we extend our analysis to a practical scheme . We show ( local ) superlinear convergence under the SR1 update and BFGS update , when applied to unconstrained quadratic objective or strongly self-concordant functions . We list our results in Table 1 with the same formulation of [ 19 ] . Note that in general , the convergence goes through two periods . The first period lasts for k0 iterations , and only has a linear convergence rate O ( ( 1− 12κ ) k0 ) . The second period has a superlinear convergence rate O ( ( 1− 1n ) k ( k−1 ) /2 ) . Our revised bound takes fewer first-period iterations k0 as well as a faster ( condition-number-free ) superlinear convergence rate in the second period compared to [ 19 ] . Notation . We denote vectors by lowercase bold letters ( e.g. , u , x ) , and matrices by capital bold letters ( e.g. , W = [ wij ] ) . We use e1 , . . . , en for the n-dimensional coordinate directions . Let λ1 ( A ) ≥ · · · ≥ λn ( A ) be the eigenvalues of a real symmetric matrix A ∈ Rn×n . Moreover , ‖ · ‖ denotes the ` 2-norm ( standard Euclidean norm ) . For a given positive definite matrix A ( i.e. , A 0 ) , we induce the following pair of conjugate Euclidean norms : ‖x‖A , √ x > Ax , ‖x‖∗A , √ x > A−1x . When A = ∇2f ( x ) 0 , we prefer to use notation ‖ · ‖x and ‖ · ‖∗x , provided that there is no ambiguity with the reference function f . We recall the rate of convergence used in this paper . Definition 1.1 ( Linear/Superlinear convergence ) Suppose a sequence { xn } converges to 0 with lim n→∞ |xn+1| |xn| = q ∈ [ 0 , 1 ) . Now suppose another sequence { yn } converges to y∗ and satisfies that |yn − y∗| ≤ |xn| , ∀n ≥ 0 . We say { yn } converges superlinearly if q = 0 , linearly if q ∈ ( 0 , 1 ) . Organization . In Section 2 , we discuss quasi-Newton updating rules for approximating a positive definite Hessian matrix using the SR1 update ( Subsection 2.1 ) and the BFGS update ( Subsection 2.2 ) . In Section 3 , we analyze previous methods applied to the problem of minimizing a quadratic function . In Section 4 , we show that similar results also hold in a more general setting of minimizing the strongly self-concordant function . We give some empirical results in Section 5 . Finally , in Section 6 , we present our conclusion . 2 Quasi-Newton updates . Before starting our theoretical results , we briefly review a class of quasi-Newton updating rules for approximating a positive definite matrix A ∈ Rn×n . We follow the definition in [ 19 ] , employing the following family of updates which describes the Broyden family [ 16 , Section 6.3 ] of quasi-Newton updates , parameterized by a scalar τ ∈ R. Definition 2.1 Let A G ( that is , G−A is positive semi-definite ) . For any u ∈ Rn , if Gu = Au , define Broydτ ( G , A , u ) , G. Otherwise , we define Broydτ ( G , A , u ) , τ [ G− Auu > G + Guu > A u > Au + ( u > Gu u > Au + 1 ) Auu > A u > Au ] + ( 1− τ ) [ G− ( G−A ) uu > ( G−A ) u > ( G−A ) u ] . ( 1 ) For several choices of τ , we can recover several well-known quasi-Newton methods . For τ = 0 , Eq . ( 1 ) corresponds to the well-known SR1 update : SR1 ( G , A , u ) , G− ( G−A ) uu > ( G−A ) u > ( G−A ) u . ( 2 ) For τBFGS , u > Au u > Gu ∈ [ 0 , 1 ] , we recover the famous BFGS update : BFGS ( G , A , u ) , G− Guu > G u > Gu + Auu > A u > Au . ( 3 ) For τ = 1 , it corresponds to the well-known DFP update : DFP ( G , A , u ) , G− Auu > G + Guu > A u > Au + ( u > Gu u > Au + 1 ) Auu > A u > Au . ( 4 ) Additionally , the Broyden family has matrix monotonicity below , showing the relationship among these quasi-Newton methods . Lemma 2.2 ( Rodomanov and Nesterov [ 19 ] Lemmas 2.1 and 2.2 ) If A G ηA for some η ≥ 1 , then we have for any u ∈ Rn , and τ1 , τ2 ∈ R with τ1 ≤ τ2 that Broydτ1 ( G , A , u ) Broydτ2 ( G , A , u ) . And for any τ ∈ [ 0 , 1 ] , we have A Broydτ ( G , A , u ) ηA . Hence , if A G ηA for some η ≥ 1 , it follows from Lemma 2.2 that A SR1 ( G , A , u ) BFGS ( G , A , u ) DFP ( G , A , u ) ηA . Intuitively , the approximation produced by SR1 is better than that produced by BFGS . And both of them are better than that produced by DFP . However , Rodomanov and Nesterov [ 19 ] deduced the analysis by casting all updates described by Broyden family ( τ ∈ [ 0 , 1 ] ) into the slowest DFP update ( τ = 1 ) . Moreover , SR1 and BFGS methods also have faster numerical performance in practice . Therefore , we consider there might have faster superlinear convergence rates of SR1 and BFGS methods . Now we consider the fundamental matrix approximation problem below to obtain the convergence rate of the SR1 and BFGS updates . 2.1 Superlinear convergence for SR1 update . We first describe the SR1 update for approximating a fixed positive definite matrix A ∈ Rn×n and leave the proof in Appendix A . Let us now justify the efficiency of update Eq . ( 2 ) in ensuring convergence G to A . For this , we introduce the following measure of progress : τA ( G ) , tr ( G−A ) . ( 5 ) Previous work [ 19 ] uses a different measure as follows σA ( G ) , 〈A−1 , G〉 − n = tr [ ( G−A ) A−1 ] , ( 6 ) which might be improper for analyzing the SR1 update . Thus we employ a more concise measure τA ( · ) . However , we will reuse σA ( · ) in the proof of BFGS update in Subsection 2.2 . According to τA ( G ) , one iteration update leads to τA ( G+ ) ( 5 ) = τA ( G ) − u > ( G−A ) 2u u > ( G−A ) u , G+ = SR1 ( G , A , u ) . Note that we always have G+ A if G A from Lemma 2.2 . Moreover , the choice of the updating direction u directly influences the decrease in the measure τA ( · ) . In the following , we show two efficient ways for selecting u introduced in Algorithm 1 . First , we use the greedy method introduced in [ 19 ] , that greedily selects u from the basis vectors to obtain the largest decrease of τA ( G+ ) − τA ( G ) : ūA ( G ) , arg maxu∈ { e1 , ... , en } u > ( G−A ) 2u u > ( G−A ) u . However , we may encounter numerical overflow due to divided by 0 if u > ( G −A ) u is nearly 0 . Noting that G A , we have √ ( u > ( G−A ) 2u ) ( u > u ) ≥ u > ( G −A ) u ≥ 0 , ∀u ∈ Rn from Cauchy–Schwarz inequality . Thus we employ a safer adjustment : ( Greedy SR1 ) ūA ( G ) , arg max u∈ { e1 , ... , en } u > ( G−A ) u u > u = arg max u∈ { e1 , ... , en } u > ( G−A ) u . ( 7 ) Moreover , we only need to obtain the diagonal elements of A ( the current Hessian in practice ) , thus the total complexity is O ( n2 ) in each iteration2 , which is acceptable and the same as the classical quasi-Newton methods . Second , from the proof of the greedy method , we discover that the random method by choosing u from a spherically symmetric distribution , e.g. , ( Random SR1 ) u ∼ N ( 0 , In ) or u ∼ Unif ( Sn−1 ) , ( 8 ) also has similar performance ( in expectation ) and the same running complexity O ( n2 ) in each iteration . Rodomanov and Nesterov [ 19 , Theorem 3.5 ] already showed that Gn = A if the uk for k ∈ { 0 , . . . , n − 1 } are linearly independent . Thus , we only consider k ≤ n. Now let us estimate the decrease in the measure τA ( · ) based on the direction u chosen from previous strategies . In the following , the expectation considers all the randomness of the directions uk ’ s during iterations , and when applied to the greedy method , we can view it with no randomness for the same notation . Theorem 2.3 Under the SR1 update in Algorithm 1 , we obtain that for the greedy method defined in Eq . ( 7 ) or the random method defined in Eq . ( 8 ) , 0 ≤ E τA ( Gk ) ≤ ( 1− 1 n− k + 1 ) E τA ( Gk−1 ) ≤ ( 1− k n ) τA ( G0 ) , 1 ≤ k ≤ n. ( 9 ) Hence , τA ( Gk ) ( or E τA ( Gk ) ) converges to zero superlinearly .

Summary: This paper presents two modified greedy quasi-Newton updates, originally given by Rodomanov and Nesterov [18]. They first consider using these updates to approximate a positive definite Hessian. In particular, for the greedy SR1 update, they introduce a different progress measure, which is maximized over the coordinate bases to yield the vector used in the update. This new measure allows them to obtain a superlinear convergence rate for the Hessian approxiamtion error, where the rate is independent of the condition number $\kappa$. For the greedy BFGS update, they use the same progress measure as in Rodomanov and Nesterov [18], but preconditions the corresponding greedily selected vector with the current Hessian approximation. This yields a linear convergence rate for the Hessian approximation, although their greedy selection for BFGS is rather impractical. In addition to the modified greedy selection strategies, their analyses and rates also apply to a random selection from a spherically symmetric distribution.  Similar to Rodomanov and Nesterov [18], they then apply these results to analyzing the convergence rate of minimizing strongly-convex quadratics, as well as strongly self-concordant objectives. They obtain similar explicit local rates as the previous work, but without the condition number (although the initial, non-local phase still converges linearly with a dependence on $\kappa$).  Experiments on evaluating the newly introduced Hessian approximations show that their methods are quite effective. On a strongly-convex quadratic minimization problem the conclusions are similar. For minimizing a strongly-convex logistic regression problem, the randomized BFGS (with the preconditioned vector) does not perform better than the version from Rodomanov and Nesterov [18], which does not contain the preconditioning. This is true for the two logistic regression problems with different scales of condition numbers.