Super-Acceleration with Cyclical Step-sizes

Cyclical step-sizes are becoming increasingly popular in the optimization of deep1 learning problems . Motivated by recent observations on the spectral gaps of2 Hessians in machine learning , we show that these step-size schedules offer a3 simple way to exploit them . More precisely , we develop a convergence rate4 analysis for quadratic objectives that provides optimal parameters and shows that5 cyclical learning rates can improve upon traditional lower complexity bounds.6 We further propose a systematic approach to design optimal first order methods7 for quadratic minimization with a given spectral structure . Finally , we provide a8 local convergence rate analysis beyond quadratic minimization for the proposed9 methods and illustrate our findings through benchmarks on least squares and10 logistic regression problems.11 1 Introduction12 One of the most iconic methods in first order optimization is gradient descent with momentum , also13 known as the heavy ball method [ Polyak , 1964 ] . This method enjoys widespread popularity both in14 its original formulation and in a stochastic variant that replaces the gradient by a stochastic estimate,15 a method that is behind many of the recent breakthroughs in deep learning [ Sutskever et al. , 2013 ] .16 A variant of the stochastic heavy ball where the step-sizes are chosen in cyclical order has recently17 come to the forefront of machine learning research , showing state-of-the art results on different deep18 learning benchmarks [ Loshchilov and Hutter , 2017 , Smith , 2017 ] . Inspired by this empirical success,19 we aim to study the convergence of the heavy ball algorithm where step-sizes h0 , h1 , . . . are not fixed20 or decreasing but instead chosen in cyclical order:21 Algorithm 1 : Cyclical heavy ball HBK ( h0 , . . . , hK−1 ; m ) Input : Initialization x0 , momentum m ∈ ( 0 , 1 ) , step-sizes { h0 , . . . , hK−1 } x1 = x0 − h0 1 +m ∇f ( x0 ) for t = 1 , 2 , . . . do xt+1 = xt − hmod ( t , K ) ∇f ( xt ) +m ( xt − xt−1 ) end The heavy ball method with constant step-sizes enjoys a mature theory , where it is known for example22 to achieve optimal black-box worst-case complexity of quadratic convex optimization [ Nemirovsky,23 1992 ] . In stark contrast , little is known about the the convergence of the above variant with cyclical24 step-sizes . Our main motivating question is25 Do cyclical step-sizes improve convergence of heavy ball ? 26 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . Our main contribution provides a positive answer to this question and , more importantly , quantifies27 the speedup under different assumptions . In particular , we show that for quadratic problems , whenever28 Hessian ’ s spectrum belongs to two or more disjoint intervals , the heavy ball method with cyclical step-29 sizes achieves a faster worst-case convergence rate . Recent works have shown that this assumption on30 the spectrum is quite natural and occurs in many machine learning problems , including deep neural31 networks [ Sagun et al. , 2017 , Papyan , 2018 , Ghorbani et al. , 2019 , Papyan , 2019 ] . More precisely,32 we list our main contributions below.33 • In sections 3 and 4 , we provide a tight convergence rate analysis of the cyclical heavy ball method34 ( Theorems 3.1 and 3.2 for two step-sizes , and Theorem 4.8 for the general case ) . This analysis35 highlights a regime under which this method achieves a faster worst-case rate than the accelerated36 rate of heavy ball , a phenomenon we refer to as super-acceleration . Theorem 5.1 extends the ( local ) 37 convergence rate analysis results to non-quadratic objectives.38 • As a byproduct of the convergence-rate analysis , we obtain an explicit expression for the optimal39 parameters in in the case of cycles of length two ( Algorithm 2 ) and an implicit expression in terms40 of a system of K equations in the general case.41 • Section 6 presents numerical benchmarks illustrating the improved convergence of the cyclical42 approach on 4 problems involving quadratic and logistic losses on both synthetic and a handwritten43 digits recognition dataset.44 • Finally , we conclude in Section 7 with a discussion of this work ’ s limitations.45 2 Notation and Problem Setting46 Throughout the paper , we consider the problem of minimizing quadratic functions of the form47 min x∈Rd f ( x ) , with f ∈ CΛ , { f : f ( x ) = 12 ( x− x ∗ ) TH ( x− x∗ ) + f∗ , Sp ( H ) ⊆ Λ } , ( OPT ) where CΛ is the class of quadratic functions whose spectrum Sp ( H ) is localized in Λ ⊆ [ µ , L ] ⊆ R > 0.48 We discuss more general settings beyond quadratic minimization in Section 5.49 The condition Λ ⊆ [ µ , L ] implies all quadratic functions under consideration are L-smooth and50 µ-strongly convex . For this function class , we define κ , the ( inverse ) condition number , and ρ , the51 ratio between the center of Λ and its radius , as52 κ , µL , ρ , L+µ L−µ = ( 1+κ 1−κ ) . ( 1 ) Finally , for a method solving ( OPT ) that generates a sequence of iterates { xt } , we define its worst-case53 rate rt and its asymptotic rate factor τ as54 rt , sup x0∈Rd , f∈CΛ ‖xt − x∗‖ ‖x0 − x∗‖ , 1− τ , lim sup t→∞ t √ rt . ( 2 ) 3 Super-acceleration with Cyclical Step-sizes55 Algorithm 2 : Cyclical ( K = 2 ) heavy ball with with optimal parameters Input : Initialization x0 , µ1 < L1 < µ2 < L2 ( where L1 − µ1 = L2 − µ2 ) Set : ρ = L2+µ1L2−µ1 , R = µ2−L1 L2−µ1 , m = ( √ ρ2−R2− √ ρ2−1√ 1−R2 ) 2 x1 = x0 − 1L1∇f ( x0 ) for t = 1 , 2 , . . . do ht = 1+m L1 ( if t is even ) , ht = 1+mµ2 ( if t is odd ) xt+1 = xt − ht∇f ( xt ) +m ( xt − xt−1 ) end In this section we develop one of our main contri-56 butions , a convergence rate analysis of the cyclical57 heavy ball method with cycles of length 2 . This analy-58 sis crucially depends on the location of the Hessian ’ s59 eigenvalues ; we assume that these are contained in a60 set Λ that is the union of 2 intervals of the same size61 Λ = [ µ1 , L1 ] ∪ [ µ2 , L2 ] , L1−µ1 = L2−µ2 . ( 3 ) By symmetry , this set is alternatively described by62 µ , µ1 , L , L2 and R , µ2 −L1 L2 − µ1 , ( 4 ) where R is the relative length of the gap µ2 −L163 with respect to the diameter L2 − µ1 ( see Figure 1 ) .64 This parametrization will reveal very convenient as65 the relative gap will play a crucial role in the conver-66 gence rate analysis . Note also that the gap assumption67 comes without loss of generality , as we allow R = 0.68 Through a correspondence between optimization69 methods and polynomials that we expand upon in70 Section 4 , we can derive a worst-case analysis for the cyclical heavy ball method . The outcome of71 this analysis is in the following theorem , that provides the asymptotic convergence rate of Algorithm72 1 for cycles of length two . All proofs of results in this section can be found in Appendix D.3.73 Theorem 3.1 ( Rate factor of HB2 ( h0 , h1 ; m ) ) . Let f ∈ CΛ and h0 , h1 , m ≥ 0 . The asymptotic rate74 factor of Algorithm 1 with cycles of length two is75 1− τ = √ m if σsup ≤ 1 , √ m ( σsup + √ σ2sup − 1 ) 1 2 if σsup ∈ ( 1 , 1+m 2 2m ) , ≥ 1 ( no convergence ) if 1+m 2 2m ≤ σsup , ( 5 ) 76 with σsup = sup λ∈ { µ1 , L1 , µ2 , L2 , h0+h1 2h0h1 } ∩Λ ∣∣∣2 ( 1 +m− λh0 2 √ m ) ( 1 +m− λh1 2 √ m ) − 1 ∣∣∣ . ( 6 ) This theorem gives the convergence rate for all triplets ( m , h0 , h1 ) . By evaluating this expression77 over a grid of step-sizes , Figure 2 shows how the rate changes as a function of both step-sizes : 78 From the asymptotic rate expression of Theorem 3.1 we can optimize over the parameters ( h0 , h1 , m ) 79 to obtain the method with smallest convergence rate . This leads to our other main contribution of this80 section , the asymptotically optimal Algorithm 2 . This algorithm enjoys the following rate:81 Corollary 3.2 . The worst-case ( asymptotic ) rates rAlg . 2t and 1− τAlg . 2 of Algorithm 2 over CΛ are82 rAlg . 2t = ( 1 + t √ ρ2−1 ρ2−R2 ) ( √ ρ2−R2− √ ρ2−1√ 1−R2 ) t , 1− τAlg . 2 = √ ρ2 −R2 − √ ρ2 − 1 √ 1−R2 for t even . 3.1 Comparison with Polyak Heavy Ball83 In the absence of eigenvalue gap ( R = 0 and Λ = [ µ , L ] ) , Algorithm 2 reduces to Polyak heavy84 ball ( PHB ) [ Polyak , 1964 ] , whose worst-case rate is detailed in Appendix B . Since the asymptotic rate85 of Algorithm 2 is monotonically decreasing in R , it is always better or equal than PHB . Furthermore,86 in the ill-conditioned regime ( small κ ) , the comparison is particularly simple : the optimal 2-cycle87 algorithm has a √ 1−R2 relative improvement over PHB , as provided by the next proposition.88 A more thorough comparison for different support sets Λ is discussed in Table 1.89 Proposition 3.3 . Let R ∈ [ 0 , 1 ) . The rate factors of respectively Algorithm 2 and PHB verify90 4 A constructive Approach : Minimax Polynomials91 This section presents a generic framework ( Algorithm 3 ) that allows designing optimal momentum92 and step-size cycles for given sets Λ and cycle length K.93 Algorithm 3 : Optimal momentum method with cyclical step-sizes Input : Eigenvalue localization Λ , cycle length K , initialization x0 . Preprocessing : 1 . Find the polynomial σΛK such that it satisfies ( 16 ) . 2 . Set step-sizes { hi } i=0 , ... , K−1 and momentum m that satisfy resp . equations ( 21 ) and ( 22 ) . Set x1 = x0 − h0 1 +m ∇f ( x0 ) for t = 1 , 2 , . . . do xt+1 = xt − hmod ( t , K ) ∇f ( xt ) +m ( xt − xt−1 ) end We first recall classical results that link optimal first order methods on quadratics and Chebyshev94 polynomials . Then , we generalize the approach by showing that optimal methods can be viewed as95 combinations of Chebyshev polynomials , and minimax polynomials σΛK of degree K over the set Λ.96 Finally , we show how to recover the step-size schedule from σΛK .97 4.1 First Order Methods on Quadratics and Polynomials98 A key property that we will use extensively in the analysis is the following link between first order99 methods and polynomials ( see [ Hestenes and Stiefel , 1952 ] ) .100 Proposition 4.1 . Let f ∈ CΛ . The iterates xt satisfy101 xt+1 ∈ x0 + span { ∇f ( x0 ) , . . . , ∇f ( xt ) } , ( 8 ) where x0 is the initial approximation of x∗ , if and only if there exists a sequence of polynomials102 ( Pt ) t∈N , each of degree at most 1 more than the highest degree of all previous polynomials and P0 of103 degree 0 ( hence the degree of Pt is at most t ) , such that104 ∀ t xt − x∗ = Pt ( H ) ( x0 − x∗ ) , Pt ( 0 ) = 1 . ( 9 ) Example 4.2 ( Gradient descent ) . Consider the gradient descent algorithm with fixed step-size h,105 applied to problem ( OPT ) . Then , after unrolling the update , we have106 xt+1−x∗ = xt−x∗ − h∇f ( xt ) = xt−x∗ − hH ( xt − x∗ ) = ( I − hH ) t+1 ( x0 − x∗ ) . ( 10 ) In this case , the polynomial associated to gradient descent is Pt ( λ ) = ( 1− hλ ) t.107 The above proposition can be used to obtain worst-case rates for first order methods by bounding108 their associated polynomials . Indeed , using the Cauchy-Schwartz inequality in ( 9 ) leads to109 ‖xt − x∗‖ ≤ sup λ∈Λ |Pt ( λ ) | ‖x0 − x∗‖ =⇒ rt = sup λ∈Λ |Pt ( λ ) | , where P ( 0 ) = 1 . ( 11 ) Therefore , finding the algorithm with the fastest worst-case rate can be equivalently framed as the110 problem of finding the polynomial with smallest value on the eigenvalue support Λ , subject to the111 normalization condition Pt ( 0 ) = 1 . Such polynomials are referred to as minimax . Throughout the112 paper , we use this polynomial-based approach to find methods with optimal rates.113 An important property of minimax polynomials is their equioscillation on Λ ( see Theorem C.1 and114 its proof for a formal statement ) .115 Definition 4.3 . ( Equioscillation ) A polynomial Pt equioscillates on Λ if it verifies Pt ( 0 ) = 1 and116 there exist λ0 < λ1 < . . . < λt ∈ Λ such that117 Pt ( λi ) = ( −1 ) i max λ∈Λ |Pt ( Λ ) | . ( 12 ) Example 4.4 ( Λ is an interval ) . The t-th order Chebyshev polynomials of the first kind Tt satisfy118 the equioscillation property on [ −1 , 1 ] . It follows that minimax polynomials on Λ = [ µ , L ] can be119 obtained by composing the Chebyshev polynomial Tt with the linear transformation σΛ1 :120 Tt ( σΛ1 ( λ ) ) Tt ( σΛ1 ( 0 ) ) = arg min P∈Rt [ X ] , P ( 0 ) =1 sup λ∈Λ |P ( λ ) | , with σΛ1 ( λ ) = L+ µ L− µ − 2 L− µ λ , ( 13 ) where σΛ1 maps the interval [ µ , L ] to [ −1 , 1 ] . The optimization method associated with this minimax121 polynomial is the Chebyshev semi-terative method [ Flanders and Shortley , 1950 , Golub and Varga,122 1961 ] ( described also in Appendix B.1 ) . This method achieves the lower complexity bound for123 smooth strongly convex quadratic minimization , see for instance [ Nemirovsky , 1995 , Chapter 12 ] or124 [ Nemirovsky , 1992 , Nesterov , 2003 ] .125 The next proposition provides the main results in this subsection , which is key for obtaining Algo-126 rithm 2 . It characterizes the even degree minimax polynomial in the setting of Section 3 , that is,127 when Λ is the union of 2 intervals of same size . In this case , the minimax solution is also based on128 Chebyshev polynomials , but composed with a degree-two polynomial σΛ2 .129 Proposition 4.5 . Let Λ = [ µ1 , L1 ] ∪ [ µ2 , L2 ] be an union of two intervals of the same size130 ( L1−µ1 = L2−µ2 ) and letm be as defined in Algorithm 2 . Then the minimax polynomial ( solution131 to ( 12 ) ) is , for all t = 2n , n ∈ N+0 ,132 Tn ( σΛ2 ( λ ) ) Tn ( σΛ2 ( 0 ) ) = arg min P∈Rt [ X ] , P ( 0 ) =1 sup λ∈Λ |P ( λ ) | , with σΛ2 ( λ ) = 2 ( 1 +m 2 √ m ) 2 ( 1− λ L1 ) ( 1− λ µ2 ) − 1 . 4.2 Generalization to Longer Cycles133 The polynomial in Example 4.4 uses a linear link function σΛ1 to map Λ to [ −1 , 1 ] . In Proposition 4.5,134 we see that a degree two link function σΛ2 can be used to find the minimax polynomial when Λ is the135 union of two intervals . This section generalizes this approach and considers higher-order polynomials136 for σK . We start with the following parametrization , with an arbitrary polynomial σK of degree K,137 Pt ( λ ; σK ) , Tn ( σK ( λ ) ) Tn ( σK ( 0 ) ) , ∀t = Kn , n ∈ N+0 . ( 14 ) As we will see in the next subsection , this parametrization allows considering cycles of step-sizes.138 Our goal now is to find the σK that obtains the fastest convergence rate possible . The next proposition139 quantifies its impact on the asymptotic rate and its proof can be found in Appendix D.1.140 Proposition 4.6 . For a given σK such that supλ∈Λ|σK ( λ ) | = 1 , the asymptotic rate factor τσK of141 the method associated to the polynomial ( 14 ) is142 1− τσK = lim t→∞ t √ sup λ∈Λ |Pt ( λ ; σK ) | = ( σ0 − √ σ20 − 1 ) 1 K , with σ0 , σK ( 0 ) . ( 15 ) For a fixed K , the asymptotic rate ( 15 ) is a decreasing function of σ0 . This motivates the introduction143 of the “ optimal ” degree K polynomial σΛK as the one that solves144 σΛK , arg max σ∈RK [ X ] σ ( 0 ) s.t . sup λ∈Λ |σ ( λ ) | = 1 . ( 16 ) Using the above definition , we recover the σΛ1 and σ Λ 2 from Example 4.4 and Proposition 4.5.145 Finding the polynomial . Finding an exact and explicit solution for the general K and Λ case146 is unfortunately out of reach , as it involves solving a potentially difficult system of K non-linear147 equations . Here we describe an approximate approach . Let σΛK ( x ) = ∑K i=0 σix i . We propose to148 discretize Λ into N different points { λj } , then solve the linear problem149 max σi σ0 s.t . −1 ≤ ∑K i=0 σiλ i j ≤ 1 , ∀j = 1 , . . . , N . ( 17 ) To check the optimality , it suffices to verify that the polynomial σΛK satisfies the equioscillation150 property ( Definition 4.3 ) , as depicted in Figure 3.151 Remark 4.7 ( Relationship between optimal and minimax polynomials ) . For later reference , we note152 that the optimal polynomial σΛK is equivalent to finding a minimax polynomial on Λ and to rescale it.153 More precisely , σΛK is optimal if and only if σ Λ K/σ Λ K ( 0 ) is minimax.154 4.3 Cyclical Heavy Ball and ( Non- ) asymptotic Rates of Convergence155 We now describe the link between σΛK and Algorithm 3 . Using the recurrence for Chebyshev156 polynomials of the first kind in ( 14 ) , we have ∀t = Kn , n ∈ N+0 ,157 Tn+1 ( σ Λ K ( λ ) ) Tn+1 ( σΛK ( 0 ) ) = 2σΛK ( λ ) [ Tn ( σ Λ K ( λ ) ) Tn ( σΛK ( 0 ) ) ] [ Tn ( σ Λ K ( 0 ) ) Tn+1 ( σΛK ( 0 ) ) ] ︸ ︷︷ ︸ =an − [ Tn−1 ( σ Λ K ( λ ) ) Tn−1 ( σΛK ( 0 ) ) ] [ Tn−1 ( σ Λ K ( 0 ) ) Tn+1 ( σΛK ( 0 ) ) ] ︸ ︷︷ ︸ =bn . It still remains to find an algorithm associated with this polynomial . To obtain one in the form of158 Algorithm 1 , one can use the stationary behavior of the recurrence . From [ Scieur and Pedregosa,159 2020 ] , the coefficients an and bn converge as n→∞ to their fixed-points a∞ and b∞ . We therefore160 consider here an asymptotic polynomial P̄t ( λ ; σΛK ) , whose recurrence satisfies161 P̄t ( λ ; σ Λ K ) = 2a∞σ Λ K ( λ ) P̄t−K ( λ ; σ Λ K ) − b∞P̄t−2K ( λ ; σΛK ) . ( 18 ) Similarly to K = 1 , where this limit recursion corresponds to PHB , this recursion corresponds to162 an instance of Algorithm 3 ( see Proposition 4.9 below ) , further motivating the cyclical heavy ball163 algorithm.164 The following theorem is the main result of this section and characterizes the convergence rate of165 Algorithm 1 for arbitrary momentum and step-size sequences { hi } i∈J1 , KK . By optimizing over these166 parameters , we obtain a method associated to ( 18 ) , whose rate is described in Proposition 4.9 . All167 proofs can be found in Appendix D.2.168 Theorem 4.8 . The worst-case rate of convergence of Algorithm 1 on CΛ with an arbitrary momentum169 m and an arbitrary sequence of step-sizes { hi } is170 1− τ = √ m , if σsup ≤ 1 √ m ( σsup + √ σ2sup − 1 ) 1/K , if σsup ∈ ( 1 , 1 +mK 2 ( √ m ) K ) ≥ 1 ( no convergence ) if σsup ≥ 1 +mK 2 ( √ m ) K , ( 19 ) where σsup , sup λ∈Λ |σ ( λ ; { hi } , m ) | , and σ ( λ ; { hi } , m ) is the K-degree polynomial171 σ ( λ ; { hi } , m ) , 1 2 Tr ( [ 1+m−hK−1λ√ m −1 1 0 ] [ 1+m−hK−2λ√ m −1 1 0 ] . . . [ 1+m−h0λ√ m −1 1 0 ] ) . ( 20 ) Proposition 4.9 . Let σ ( λ ; { hi } , m ) be the polynomial defined by ( 20 ) , and σΛK be the optimal link172 function of degree K defined by ( 16 ) . If the momentum m and the sequence of step-sizes { hi } satisfy173 σ ( λ ; { hi } , m ) = σΛK ( λ ) , ( 21 ) then 1 ) the parameters are optimal , in the sense that they minimize the asymptotic rate factor from174 Theorem 4.8 , 2 ) the optimal momentum parameter is175 m = ( σ0 − √ σ20 − 1 ) 2/K , where σ0 = σΛK ( 0 ) , ( 22 ) 3 ) the iterates from Algo . 3 with parameters { hi } and m form a polynomial with recurrence ( 18 ) ,176 and 4 ) Algorithm 3 achieves the worst-case rate rAlg . 3t and the asymptotic rate factor 1− τAlg . 3177 rAlg . 3t = O ( t ( σ0 − √ σ20 − 1 ) t/K ) , 1− τAlg . 3 = ( σ0 − √ σ20 − 1 ) 1/K . ( 23 ) Solving the system ( 21 ) The system is constructed by identification of the coefficients in both178 polynomials σΛK and σ ( λ ; { hi } , m ) , which can be solved using a naive grid-search for instance . We179 are not aware of any efficient algorithm to solve this system exactly , although it is possible to use180 iterative methods such as steepest descent or Newton ’ s method.181

Summary: This paper studies the cyclicial step-sizes to improve the worst-case convergence of the heavy ball method (deterministic version). More precisely, the authors 1) achieve a faster asymptotic worst-case convergence  rate compared to the polayk heavy ball method for the strongly convex and L-smooth quadratic objectives,  2) show how to select the optimal momentum and step-size cycles, and 3) extend the local convergence to the non-quadratic case. Furthemore, numerical experiments on quadratics and logistic regressions have been conducted to demonstrate the main strength of the cyclicial heavy ball method.