Greedy and Random Quasi-Newton Methods with Faster Explicit Superlinear Convergence

1 Introduction . We study the superlinear convergence of famous quasi-Newton methods that replace the exact Hessian applied in classical Newton methods with certain approximations . The approximation is updated in iterations based on some special formulas from the previous variation . There exist various quasiNewton algorithms with different Hessian approximations . The three most popular versions are the Davidon-Fletcher-Powell ( DFP ) method [ 7 , 10 ] , the Broyden-Fletcher-Goldfarb-Shanno ( BFGS ) method [ 2 , 3 , 9 , 11 , 21 ] , and the Symmetric Rank 1 ( SR1 ) method [ 1 , 7 ] , all of which belong to the Broyden family [ 1 ] of quasi-Newton algorithms . The most attractive property of quasi-Newton methods is their superlinear convergence , which can trace back to the 1970s [ 4 , 8 , 17 ] . However , the superlinear convergence rates provided in prior work are asymptotic [ 5 , 13 , 15 , 22 , 24 ] . The results only show that the ratio of successive residuals tends to zero as the running iterations approach to infinity , while still lacking a specific superlinear convergence rate . Recently , Rodomanov and Nesterov [ 19 ] gave the first explicit local superlinear convergence for their proposed new quasi-Newton methods . They introduced greedy quasi-Newton updates by greedily selecting from basis vectors to maximize a certain measure of progress , and established an explicit non-asymptotic bound on the local superlinear convergence rate correspondingly . Their proofs are mainly applicable to the DFP methods because they reduced all possible Broyden family to the DFP update based on the monotonicity property ( see Lemma 2.2 ) . However , the SR1 and BFGS updates are more popular and faster than the DFP update in practice , which also has been verified in their experiments . In addition , Rodomanov and Nesterov [ 19 ] also discovered that the randomized methods do not lose superlinear convergence but they did not provide theoretical guarantees . ∗Equal Contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Along this line , there are other results of local superlinear convergence analysis . Rodomanov and Nesterov [ 20 ] analyzed the well-known DFP and BFGS methods , using a standard Hessian update direction through the previous variation . They demonstrated faster initial convergence rates , while slower final rates compared to Rodomanov and Nesterov [ 19 ] ’ s results . Rodomanov and Nesterov [ 18 ] improved Rodomanov and Nesterov [ 20 ] ’ s results by reducing the dependence of the condition number κ to lnκ , though having similar worse long-history behavior . Jin and Mokhtari [ 14 ] provided a non-asymptotic dimension-free superlinear convergence rate of the original Broyden family when the initial Hessian approximation is also good enough . In this work , we prove faster convergence rates for greedy and random quasi-Newton algorithms , particularly on the SR1 and BFGS methods . We present our contribution as follows : • First , in the setting of approximating an exact Hessian , we use two update methods : 1 ) greedy update modified from Rodomanov and Nesterov [ 19 ] ; 2 ) random update that randomly selects the updating direction from any spherically symmetric distribution . We show that both methods share a faster condition-number-free convergence . Particularly , we obtain the superlinear convergence rate O ( 1− kn ) for SR1 update , and the linear convergence rate O ( ( 1− 1n ) k ) for BFGS update , where k is the current iteration , and n is the dimension of parameters . Both findings improve the original convergence rate O ( ( 1− 1nκ ) k ) [ 19 ] . • Second , we extend our analysis to a practical scheme . We show ( local ) superlinear convergence under the SR1 update and BFGS update , when applied to unconstrained quadratic objective or strongly self-concordant functions . We list our results in Table 1 with the same formulation of [ 19 ] . Note that in general , the convergence goes through two periods . The first period lasts for k0 iterations , and only has a linear convergence rate O ( ( 1− 12κ ) k0 ) . The second period has a superlinear convergence rate O ( ( 1− 1n ) k ( k−1 ) /2 ) . Our revised bound takes fewer first-period iterations k0 as well as a faster ( condition-number-free ) superlinear convergence rate in the second period compared to [ 19 ] . Notation . We denote vectors by lowercase bold letters ( e.g. , u , x ) , and matrices by capital bold letters ( e.g. , W = [ wij ] ) . We use e1 , . . . , en for the n-dimensional coordinate directions . Let λ1 ( A ) ≥ · · · ≥ λn ( A ) be the eigenvalues of a real symmetric matrix A ∈ Rn×n . Moreover , ‖ · ‖ denotes the ` 2-norm ( standard Euclidean norm ) . For a given positive definite matrix A ( i.e. , A 0 ) , we induce the following pair of conjugate Euclidean norms : ‖x‖A , √ x > Ax , ‖x‖∗A , √ x > A−1x . When A = ∇2f ( x ) 0 , we prefer to use notation ‖ · ‖x and ‖ · ‖∗x , provided that there is no ambiguity with the reference function f . We recall the rate of convergence used in this paper . Definition 1.1 ( Linear/Superlinear convergence ) Suppose a sequence { xn } converges to 0 with lim n→∞ |xn+1| |xn| = q ∈ [ 0 , 1 ) . Now suppose another sequence { yn } converges to y∗ and satisfies that |yn − y∗| ≤ |xn| , ∀n ≥ 0 . We say { yn } converges superlinearly if q = 0 , linearly if q ∈ ( 0 , 1 ) . Organization . In Section 2 , we discuss quasi-Newton updating rules for approximating a positive definite Hessian matrix using the SR1 update ( Subsection 2.1 ) and the BFGS update ( Subsection 2.2 ) . In Section 3 , we analyze previous methods applied to the problem of minimizing a quadratic function . In Section 4 , we show that similar results also hold in a more general setting of minimizing the strongly self-concordant function . We give some empirical results in Section 5 . Finally , in Section 6 , we present our conclusion . 2 Quasi-Newton updates . Before starting our theoretical results , we briefly review a class of quasi-Newton updating rules for approximating a positive definite matrix A ∈ Rn×n . We follow the definition in [ 19 ] , employing the following family of updates which describes the Broyden family [ 16 , Section 6.3 ] of quasi-Newton updates , parameterized by a scalar τ ∈ R. Definition 2.1 Let A G ( that is , G−A is positive semi-definite ) . For any u ∈ Rn , if Gu = Au , define Broydτ ( G , A , u ) , G. Otherwise , we define Broydτ ( G , A , u ) , τ [ G− Auu > G + Guu > A u > Au + ( u > Gu u > Au + 1 ) Auu > A u > Au ] + ( 1− τ ) [ G− ( G−A ) uu > ( G−A ) u > ( G−A ) u ] . ( 1 ) For several choices of τ , we can recover several well-known quasi-Newton methods . For τ = 0 , Eq . ( 1 ) corresponds to the well-known SR1 update : SR1 ( G , A , u ) , G− ( G−A ) uu > ( G−A ) u > ( G−A ) u . ( 2 ) For τBFGS , u > Au u > Gu ∈ [ 0 , 1 ] , we recover the famous BFGS update : BFGS ( G , A , u ) , G− Guu > G u > Gu + Auu > A u > Au . ( 3 ) For τ = 1 , it corresponds to the well-known DFP update : DFP ( G , A , u ) , G− Auu > G + Guu > A u > Au + ( u > Gu u > Au + 1 ) Auu > A u > Au . ( 4 ) Additionally , the Broyden family has matrix monotonicity below , showing the relationship among these quasi-Newton methods . Lemma 2.2 ( Rodomanov and Nesterov [ 19 ] Lemmas 2.1 and 2.2 ) If A G ηA for some η ≥ 1 , then we have for any u ∈ Rn , and τ1 , τ2 ∈ R with τ1 ≤ τ2 that Broydτ1 ( G , A , u ) Broydτ2 ( G , A , u ) . And for any τ ∈ [ 0 , 1 ] , we have A Broydτ ( G , A , u ) ηA . Hence , if A G ηA for some η ≥ 1 , it follows from Lemma 2.2 that A SR1 ( G , A , u ) BFGS ( G , A , u ) DFP ( G , A , u ) ηA . Intuitively , the approximation produced by SR1 is better than that produced by BFGS . And both of them are better than that produced by DFP . However , Rodomanov and Nesterov [ 19 ] deduced the analysis by casting all updates described by Broyden family ( τ ∈ [ 0 , 1 ] ) into the slowest DFP update ( τ = 1 ) . Moreover , SR1 and BFGS methods also have faster numerical performance in practice . Therefore , we consider there might have faster superlinear convergence rates of SR1 and BFGS methods . Now we consider the fundamental matrix approximation problem below to obtain the convergence rate of the SR1 and BFGS updates . 2.1 Superlinear convergence for SR1 update . We first describe the SR1 update for approximating a fixed positive definite matrix A ∈ Rn×n and leave the proof in Appendix A . Let us now justify the efficiency of update Eq . ( 2 ) in ensuring convergence G to A . For this , we introduce the following measure of progress : τA ( G ) , tr ( G−A ) . ( 5 ) Previous work [ 19 ] uses a different measure as follows σA ( G ) , 〈A−1 , G〉 − n = tr [ ( G−A ) A−1 ] , ( 6 ) which might be improper for analyzing the SR1 update . Thus we employ a more concise measure τA ( · ) . However , we will reuse σA ( · ) in the proof of BFGS update in Subsection 2.2 . According to τA ( G ) , one iteration update leads to τA ( G+ ) ( 5 ) = τA ( G ) − u > ( G−A ) 2u u > ( G−A ) u , G+ = SR1 ( G , A , u ) . Note that we always have G+ A if G A from Lemma 2.2 . Moreover , the choice of the updating direction u directly influences the decrease in the measure τA ( · ) . In the following , we show two efficient ways for selecting u introduced in Algorithm 1 . First , we use the greedy method introduced in [ 19 ] , that greedily selects u from the basis vectors to obtain the largest decrease of τA ( G+ ) − τA ( G ) : ūA ( G ) , arg maxu∈ { e1 , ... , en } u > ( G−A ) 2u u > ( G−A ) u . However , we may encounter numerical overflow due to divided by 0 if u > ( G −A ) u is nearly 0 . Noting that G A , we have √ ( u > ( G−A ) 2u ) ( u > u ) ≥ u > ( G −A ) u ≥ 0 , ∀u ∈ Rn from Cauchy–Schwarz inequality . Thus we employ a safer adjustment : ( Greedy SR1 ) ūA ( G ) , arg max u∈ { e1 , ... , en } u > ( G−A ) u u > u = arg max u∈ { e1 , ... , en } u > ( G−A ) u . ( 7 ) Moreover , we only need to obtain the diagonal elements of A ( the current Hessian in practice ) , thus the total complexity is O ( n2 ) in each iteration2 , which is acceptable and the same as the classical quasi-Newton methods . Second , from the proof of the greedy method , we discover that the random method by choosing u from a spherically symmetric distribution , e.g. , ( Random SR1 ) u ∼ N ( 0 , In ) or u ∼ Unif ( Sn−1 ) , ( 8 ) also has similar performance ( in expectation ) and the same running complexity O ( n2 ) in each iteration . Rodomanov and Nesterov [ 19 , Theorem 3.5 ] already showed that Gn = A if the uk for k ∈ { 0 , . . . , n − 1 } are linearly independent . Thus , we only consider k ≤ n. Now let us estimate the decrease in the measure τA ( · ) based on the direction u chosen from previous strategies . In the following , the expectation considers all the randomness of the directions uk ’ s during iterations , and when applied to the greedy method , we can view it with no randomness for the same notation . Theorem 2.3 Under the SR1 update in Algorithm 1 , we obtain that for the greedy method defined in Eq . ( 7 ) or the random method defined in Eq . ( 8 ) , 0 ≤ E τA ( Gk ) ≤ ( 1− 1 n− k + 1 ) E τA ( Gk−1 ) ≤ ( 1− k n ) τA ( G0 ) , 1 ≤ k ≤ n. ( 9 ) Hence , τA ( Gk ) ( or E τA ( Gk ) ) converges to zero superlinearly .

Summary: This work considers the topic of the convergence of quasi-Newton and it builds on top of a series of recent results by Rodomanov and Nesterov. The main two contributions of this paper are: 1) improve the bounds for greedy quasi-Newton methods using a new potential function; 2) to propose a new modification of greedy strategy to obtain better results. The authors consider two settings: unconstrained quadratics and general strongly self-concordant functions (the same settings were considered in prior work). All results are local but this is probably inevitable as quasi-Newton methods approximate Newton's method, which itself is a local method.  I find the results to be very solid and promising. The topic of quasi-Newton methods might get a second breath due to the recent results and I would be glad to see this paper accepted. It does have a small number of typos but I think most of them can be fixed if the authors proofread their work (I hope the authors will do at least a partial pass during the discussion stage).