Super-Acceleration with Cyclical Step-sizes

Cyclical step-sizes are becoming increasingly popular in the optimization of deep1 learning problems . Motivated by recent observations on the spectral gaps of2 Hessians in machine learning , we show that these step-size schedules offer a3 simple way to exploit them . More precisely , we develop a convergence rate4 analysis for quadratic objectives that provides optimal parameters and shows that5 cyclical learning rates can improve upon traditional lower complexity bounds.6 We further propose a systematic approach to design optimal first order methods7 for quadratic minimization with a given spectral structure . Finally , we provide a8 local convergence rate analysis beyond quadratic minimization for the proposed9 methods and illustrate our findings through benchmarks on least squares and10 logistic regression problems.11 1 Introduction12 One of the most iconic methods in first order optimization is gradient descent with momentum , also13 known as the heavy ball method [ Polyak , 1964 ] . This method enjoys widespread popularity both in14 its original formulation and in a stochastic variant that replaces the gradient by a stochastic estimate,15 a method that is behind many of the recent breakthroughs in deep learning [ Sutskever et al. , 2013 ] .16 A variant of the stochastic heavy ball where the step-sizes are chosen in cyclical order has recently17 come to the forefront of machine learning research , showing state-of-the art results on different deep18 learning benchmarks [ Loshchilov and Hutter , 2017 , Smith , 2017 ] . Inspired by this empirical success,19 we aim to study the convergence of the heavy ball algorithm where step-sizes h0 , h1 , . . . are not fixed20 or decreasing but instead chosen in cyclical order:21 Algorithm 1 : Cyclical heavy ball HBK ( h0 , . . . , hK−1 ; m ) Input : Initialization x0 , momentum m ∈ ( 0 , 1 ) , step-sizes { h0 , . . . , hK−1 } x1 = x0 − h0 1 +m ∇f ( x0 ) for t = 1 , 2 , . . . do xt+1 = xt − hmod ( t , K ) ∇f ( xt ) +m ( xt − xt−1 ) end The heavy ball method with constant step-sizes enjoys a mature theory , where it is known for example22 to achieve optimal black-box worst-case complexity of quadratic convex optimization [ Nemirovsky,23 1992 ] . In stark contrast , little is known about the the convergence of the above variant with cyclical24 step-sizes . Our main motivating question is25 Do cyclical step-sizes improve convergence of heavy ball ? 26 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . Our main contribution provides a positive answer to this question and , more importantly , quantifies27 the speedup under different assumptions . In particular , we show that for quadratic problems , whenever28 Hessian ’ s spectrum belongs to two or more disjoint intervals , the heavy ball method with cyclical step-29 sizes achieves a faster worst-case convergence rate . Recent works have shown that this assumption on30 the spectrum is quite natural and occurs in many machine learning problems , including deep neural31 networks [ Sagun et al. , 2017 , Papyan , 2018 , Ghorbani et al. , 2019 , Papyan , 2019 ] . More precisely,32 we list our main contributions below.33 • In sections 3 and 4 , we provide a tight convergence rate analysis of the cyclical heavy ball method34 ( Theorems 3.1 and 3.2 for two step-sizes , and Theorem 4.8 for the general case ) . This analysis35 highlights a regime under which this method achieves a faster worst-case rate than the accelerated36 rate of heavy ball , a phenomenon we refer to as super-acceleration . Theorem 5.1 extends the ( local ) 37 convergence rate analysis results to non-quadratic objectives.38 • As a byproduct of the convergence-rate analysis , we obtain an explicit expression for the optimal39 parameters in in the case of cycles of length two ( Algorithm 2 ) and an implicit expression in terms40 of a system of K equations in the general case.41 • Section 6 presents numerical benchmarks illustrating the improved convergence of the cyclical42 approach on 4 problems involving quadratic and logistic losses on both synthetic and a handwritten43 digits recognition dataset.44 • Finally , we conclude in Section 7 with a discussion of this work ’ s limitations.45 2 Notation and Problem Setting46 Throughout the paper , we consider the problem of minimizing quadratic functions of the form47 min x∈Rd f ( x ) , with f ∈ CΛ , { f : f ( x ) = 12 ( x− x ∗ ) TH ( x− x∗ ) + f∗ , Sp ( H ) ⊆ Λ } , ( OPT ) where CΛ is the class of quadratic functions whose spectrum Sp ( H ) is localized in Λ ⊆ [ µ , L ] ⊆ R > 0.48 We discuss more general settings beyond quadratic minimization in Section 5.49 The condition Λ ⊆ [ µ , L ] implies all quadratic functions under consideration are L-smooth and50 µ-strongly convex . For this function class , we define κ , the ( inverse ) condition number , and ρ , the51 ratio between the center of Λ and its radius , as52 κ , µL , ρ , L+µ L−µ = ( 1+κ 1−κ ) . ( 1 ) Finally , for a method solving ( OPT ) that generates a sequence of iterates { xt } , we define its worst-case53 rate rt and its asymptotic rate factor τ as54 rt , sup x0∈Rd , f∈CΛ ‖xt − x∗‖ ‖x0 − x∗‖ , 1− τ , lim sup t→∞ t √ rt . ( 2 ) 3 Super-acceleration with Cyclical Step-sizes55 Algorithm 2 : Cyclical ( K = 2 ) heavy ball with with optimal parameters Input : Initialization x0 , µ1 < L1 < µ2 < L2 ( where L1 − µ1 = L2 − µ2 ) Set : ρ = L2+µ1L2−µ1 , R = µ2−L1 L2−µ1 , m = ( √ ρ2−R2− √ ρ2−1√ 1−R2 ) 2 x1 = x0 − 1L1∇f ( x0 ) for t = 1 , 2 , . . . do ht = 1+m L1 ( if t is even ) , ht = 1+mµ2 ( if t is odd ) xt+1 = xt − ht∇f ( xt ) +m ( xt − xt−1 ) end In this section we develop one of our main contri-56 butions , a convergence rate analysis of the cyclical57 heavy ball method with cycles of length 2 . This analy-58 sis crucially depends on the location of the Hessian ’ s59 eigenvalues ; we assume that these are contained in a60 set Λ that is the union of 2 intervals of the same size61 Λ = [ µ1 , L1 ] ∪ [ µ2 , L2 ] , L1−µ1 = L2−µ2 . ( 3 ) By symmetry , this set is alternatively described by62 µ , µ1 , L , L2 and R , µ2 −L1 L2 − µ1 , ( 4 ) where R is the relative length of the gap µ2 −L163 with respect to the diameter L2 − µ1 ( see Figure 1 ) .64 This parametrization will reveal very convenient as65 the relative gap will play a crucial role in the conver-66 gence rate analysis . Note also that the gap assumption67 comes without loss of generality , as we allow R = 0.68 Through a correspondence between optimization69 methods and polynomials that we expand upon in70 Section 4 , we can derive a worst-case analysis for the cyclical heavy ball method . The outcome of71 this analysis is in the following theorem , that provides the asymptotic convergence rate of Algorithm72 1 for cycles of length two . All proofs of results in this section can be found in Appendix D.3.73 Theorem 3.1 ( Rate factor of HB2 ( h0 , h1 ; m ) ) . Let f ∈ CΛ and h0 , h1 , m ≥ 0 . The asymptotic rate74 factor of Algorithm 1 with cycles of length two is75 1− τ = √ m if σsup ≤ 1 , √ m ( σsup + √ σ2sup − 1 ) 1 2 if σsup ∈ ( 1 , 1+m 2 2m ) , ≥ 1 ( no convergence ) if 1+m 2 2m ≤ σsup , ( 5 ) 76 with σsup = sup λ∈ { µ1 , L1 , µ2 , L2 , h0+h1 2h0h1 } ∩Λ ∣∣∣2 ( 1 +m− λh0 2 √ m ) ( 1 +m− λh1 2 √ m ) − 1 ∣∣∣ . ( 6 ) This theorem gives the convergence rate for all triplets ( m , h0 , h1 ) . By evaluating this expression77 over a grid of step-sizes , Figure 2 shows how the rate changes as a function of both step-sizes : 78 From the asymptotic rate expression of Theorem 3.1 we can optimize over the parameters ( h0 , h1 , m ) 79 to obtain the method with smallest convergence rate . This leads to our other main contribution of this80 section , the asymptotically optimal Algorithm 2 . This algorithm enjoys the following rate:81 Corollary 3.2 . The worst-case ( asymptotic ) rates rAlg . 2t and 1− τAlg . 2 of Algorithm 2 over CΛ are82 rAlg . 2t = ( 1 + t √ ρ2−1 ρ2−R2 ) ( √ ρ2−R2− √ ρ2−1√ 1−R2 ) t , 1− τAlg . 2 = √ ρ2 −R2 − √ ρ2 − 1 √ 1−R2 for t even . 3.1 Comparison with Polyak Heavy Ball83 In the absence of eigenvalue gap ( R = 0 and Λ = [ µ , L ] ) , Algorithm 2 reduces to Polyak heavy84 ball ( PHB ) [ Polyak , 1964 ] , whose worst-case rate is detailed in Appendix B . Since the asymptotic rate85 of Algorithm 2 is monotonically decreasing in R , it is always better or equal than PHB . Furthermore,86 in the ill-conditioned regime ( small κ ) , the comparison is particularly simple : the optimal 2-cycle87 algorithm has a √ 1−R2 relative improvement over PHB , as provided by the next proposition.88 A more thorough comparison for different support sets Λ is discussed in Table 1.89 Proposition 3.3 . Let R ∈ [ 0 , 1 ) . The rate factors of respectively Algorithm 2 and PHB verify90 4 A constructive Approach : Minimax Polynomials91 This section presents a generic framework ( Algorithm 3 ) that allows designing optimal momentum92 and step-size cycles for given sets Λ and cycle length K.93 Algorithm 3 : Optimal momentum method with cyclical step-sizes Input : Eigenvalue localization Λ , cycle length K , initialization x0 . Preprocessing : 1 . Find the polynomial σΛK such that it satisfies ( 16 ) . 2 . Set step-sizes { hi } i=0 , ... , K−1 and momentum m that satisfy resp . equations ( 21 ) and ( 22 ) . Set x1 = x0 − h0 1 +m ∇f ( x0 ) for t = 1 , 2 , . . . do xt+1 = xt − hmod ( t , K ) ∇f ( xt ) +m ( xt − xt−1 ) end We first recall classical results that link optimal first order methods on quadratics and Chebyshev94 polynomials . Then , we generalize the approach by showing that optimal methods can be viewed as95 combinations of Chebyshev polynomials , and minimax polynomials σΛK of degree K over the set Λ.96 Finally , we show how to recover the step-size schedule from σΛK .97 4.1 First Order Methods on Quadratics and Polynomials98 A key property that we will use extensively in the analysis is the following link between first order99 methods and polynomials ( see [ Hestenes and Stiefel , 1952 ] ) .100 Proposition 4.1 . Let f ∈ CΛ . The iterates xt satisfy101 xt+1 ∈ x0 + span { ∇f ( x0 ) , . . . , ∇f ( xt ) } , ( 8 ) where x0 is the initial approximation of x∗ , if and only if there exists a sequence of polynomials102 ( Pt ) t∈N , each of degree at most 1 more than the highest degree of all previous polynomials and P0 of103 degree 0 ( hence the degree of Pt is at most t ) , such that104 ∀ t xt − x∗ = Pt ( H ) ( x0 − x∗ ) , Pt ( 0 ) = 1 . ( 9 ) Example 4.2 ( Gradient descent ) . Consider the gradient descent algorithm with fixed step-size h,105 applied to problem ( OPT ) . Then , after unrolling the update , we have106 xt+1−x∗ = xt−x∗ − h∇f ( xt ) = xt−x∗ − hH ( xt − x∗ ) = ( I − hH ) t+1 ( x0 − x∗ ) . ( 10 ) In this case , the polynomial associated to gradient descent is Pt ( λ ) = ( 1− hλ ) t.107 The above proposition can be used to obtain worst-case rates for first order methods by bounding108 their associated polynomials . Indeed , using the Cauchy-Schwartz inequality in ( 9 ) leads to109 ‖xt − x∗‖ ≤ sup λ∈Λ |Pt ( λ ) | ‖x0 − x∗‖ =⇒ rt = sup λ∈Λ |Pt ( λ ) | , where P ( 0 ) = 1 . ( 11 ) Therefore , finding the algorithm with the fastest worst-case rate can be equivalently framed as the110 problem of finding the polynomial with smallest value on the eigenvalue support Λ , subject to the111 normalization condition Pt ( 0 ) = 1 . Such polynomials are referred to as minimax . Throughout the112 paper , we use this polynomial-based approach to find methods with optimal rates.113 An important property of minimax polynomials is their equioscillation on Λ ( see Theorem C.1 and114 its proof for a formal statement ) .115 Definition 4.3 . ( Equioscillation ) A polynomial Pt equioscillates on Λ if it verifies Pt ( 0 ) = 1 and116 there exist λ0 < λ1 < . . . < λt ∈ Λ such that117 Pt ( λi ) = ( −1 ) i max λ∈Λ |Pt ( Λ ) | . ( 12 ) Example 4.4 ( Λ is an interval ) . The t-th order Chebyshev polynomials of the first kind Tt satisfy118 the equioscillation property on [ −1 , 1 ] . It follows that minimax polynomials on Λ = [ µ , L ] can be119 obtained by composing the Chebyshev polynomial Tt with the linear transformation σΛ1 :120 Tt ( σΛ1 ( λ ) ) Tt ( σΛ1 ( 0 ) ) = arg min P∈Rt [ X ] , P ( 0 ) =1 sup λ∈Λ |P ( λ ) | , with σΛ1 ( λ ) = L+ µ L− µ − 2 L− µ λ , ( 13 ) where σΛ1 maps the interval [ µ , L ] to [ −1 , 1 ] . The optimization method associated with this minimax121 polynomial is the Chebyshev semi-terative method [ Flanders and Shortley , 1950 , Golub and Varga,122 1961 ] ( described also in Appendix B.1 ) . This method achieves the lower complexity bound for123 smooth strongly convex quadratic minimization , see for instance [ Nemirovsky , 1995 , Chapter 12 ] or124 [ Nemirovsky , 1992 , Nesterov , 2003 ] .125 The next proposition provides the main results in this subsection , which is key for obtaining Algo-126 rithm 2 . It characterizes the even degree minimax polynomial in the setting of Section 3 , that is,127 when Λ is the union of 2 intervals of same size . In this case , the minimax solution is also based on128 Chebyshev polynomials , but composed with a degree-two polynomial σΛ2 .129 Proposition 4.5 . Let Λ = [ µ1 , L1 ] ∪ [ µ2 , L2 ] be an union of two intervals of the same size130 ( L1−µ1 = L2−µ2 ) and letm be as defined in Algorithm 2 . Then the minimax polynomial ( solution131 to ( 12 ) ) is , for all t = 2n , n ∈ N+0 ,132 Tn ( σΛ2 ( λ ) ) Tn ( σΛ2 ( 0 ) ) = arg min P∈Rt [ X ] , P ( 0 ) =1 sup λ∈Λ |P ( λ ) | , with σΛ2 ( λ ) = 2 ( 1 +m 2 √ m ) 2 ( 1− λ L1 ) ( 1− λ µ2 ) − 1 . 4.2 Generalization to Longer Cycles133 The polynomial in Example 4.4 uses a linear link function σΛ1 to map Λ to [ −1 , 1 ] . In Proposition 4.5,134 we see that a degree two link function σΛ2 can be used to find the minimax polynomial when Λ is the135 union of two intervals . This section generalizes this approach and considers higher-order polynomials136 for σK . We start with the following parametrization , with an arbitrary polynomial σK of degree K,137 Pt ( λ ; σK ) , Tn ( σK ( λ ) ) Tn ( σK ( 0 ) ) , ∀t = Kn , n ∈ N+0 . ( 14 ) As we will see in the next subsection , this parametrization allows considering cycles of step-sizes.138 Our goal now is to find the σK that obtains the fastest convergence rate possible . The next proposition139 quantifies its impact on the asymptotic rate and its proof can be found in Appendix D.1.140 Proposition 4.6 . For a given σK such that supλ∈Λ|σK ( λ ) | = 1 , the asymptotic rate factor τσK of141 the method associated to the polynomial ( 14 ) is142 1− τσK = lim t→∞ t √ sup λ∈Λ |Pt ( λ ; σK ) | = ( σ0 − √ σ20 − 1 ) 1 K , with σ0 , σK ( 0 ) . ( 15 ) For a fixed K , the asymptotic rate ( 15 ) is a decreasing function of σ0 . This motivates the introduction143 of the “ optimal ” degree K polynomial σΛK as the one that solves144 σΛK , arg max σ∈RK [ X ] σ ( 0 ) s.t . sup λ∈Λ |σ ( λ ) | = 1 . ( 16 ) Using the above definition , we recover the σΛ1 and σ Λ 2 from Example 4.4 and Proposition 4.5.145 Finding the polynomial . Finding an exact and explicit solution for the general K and Λ case146 is unfortunately out of reach , as it involves solving a potentially difficult system of K non-linear147 equations . Here we describe an approximate approach . Let σΛK ( x ) = ∑K i=0 σix i . We propose to148 discretize Λ into N different points { λj } , then solve the linear problem149 max σi σ0 s.t . −1 ≤ ∑K i=0 σiλ i j ≤ 1 , ∀j = 1 , . . . , N . ( 17 ) To check the optimality , it suffices to verify that the polynomial σΛK satisfies the equioscillation150 property ( Definition 4.3 ) , as depicted in Figure 3.151 Remark 4.7 ( Relationship between optimal and minimax polynomials ) . For later reference , we note152 that the optimal polynomial σΛK is equivalent to finding a minimax polynomial on Λ and to rescale it.153 More precisely , σΛK is optimal if and only if σ Λ K/σ Λ K ( 0 ) is minimax.154 4.3 Cyclical Heavy Ball and ( Non- ) asymptotic Rates of Convergence155 We now describe the link between σΛK and Algorithm 3 . Using the recurrence for Chebyshev156 polynomials of the first kind in ( 14 ) , we have ∀t = Kn , n ∈ N+0 ,157 Tn+1 ( σ Λ K ( λ ) ) Tn+1 ( σΛK ( 0 ) ) = 2σΛK ( λ ) [ Tn ( σ Λ K ( λ ) ) Tn ( σΛK ( 0 ) ) ] [ Tn ( σ Λ K ( 0 ) ) Tn+1 ( σΛK ( 0 ) ) ] ︸ ︷︷ ︸ =an − [ Tn−1 ( σ Λ K ( λ ) ) Tn−1 ( σΛK ( 0 ) ) ] [ Tn−1 ( σ Λ K ( 0 ) ) Tn+1 ( σΛK ( 0 ) ) ] ︸ ︷︷ ︸ =bn . It still remains to find an algorithm associated with this polynomial . To obtain one in the form of158 Algorithm 1 , one can use the stationary behavior of the recurrence . From [ Scieur and Pedregosa,159 2020 ] , the coefficients an and bn converge as n→∞ to their fixed-points a∞ and b∞ . We therefore160 consider here an asymptotic polynomial P̄t ( λ ; σΛK ) , whose recurrence satisfies161 P̄t ( λ ; σ Λ K ) = 2a∞σ Λ K ( λ ) P̄t−K ( λ ; σ Λ K ) − b∞P̄t−2K ( λ ; σΛK ) . ( 18 ) Similarly to K = 1 , where this limit recursion corresponds to PHB , this recursion corresponds to162 an instance of Algorithm 3 ( see Proposition 4.9 below ) , further motivating the cyclical heavy ball163 algorithm.164 The following theorem is the main result of this section and characterizes the convergence rate of165 Algorithm 1 for arbitrary momentum and step-size sequences { hi } i∈J1 , KK . By optimizing over these166 parameters , we obtain a method associated to ( 18 ) , whose rate is described in Proposition 4.9 . All167 proofs can be found in Appendix D.2.168 Theorem 4.8 . The worst-case rate of convergence of Algorithm 1 on CΛ with an arbitrary momentum169 m and an arbitrary sequence of step-sizes { hi } is170 1− τ = √ m , if σsup ≤ 1 √ m ( σsup + √ σ2sup − 1 ) 1/K , if σsup ∈ ( 1 , 1 +mK 2 ( √ m ) K ) ≥ 1 ( no convergence ) if σsup ≥ 1 +mK 2 ( √ m ) K , ( 19 ) where σsup , sup λ∈Λ |σ ( λ ; { hi } , m ) | , and σ ( λ ; { hi } , m ) is the K-degree polynomial171 σ ( λ ; { hi } , m ) , 1 2 Tr ( [ 1+m−hK−1λ√ m −1 1 0 ] [ 1+m−hK−2λ√ m −1 1 0 ] . . . [ 1+m−h0λ√ m −1 1 0 ] ) . ( 20 ) Proposition 4.9 . Let σ ( λ ; { hi } , m ) be the polynomial defined by ( 20 ) , and σΛK be the optimal link172 function of degree K defined by ( 16 ) . If the momentum m and the sequence of step-sizes { hi } satisfy173 σ ( λ ; { hi } , m ) = σΛK ( λ ) , ( 21 ) then 1 ) the parameters are optimal , in the sense that they minimize the asymptotic rate factor from174 Theorem 4.8 , 2 ) the optimal momentum parameter is175 m = ( σ0 − √ σ20 − 1 ) 2/K , where σ0 = σΛK ( 0 ) , ( 22 ) 3 ) the iterates from Algo . 3 with parameters { hi } and m form a polynomial with recurrence ( 18 ) ,176 and 4 ) Algorithm 3 achieves the worst-case rate rAlg . 3t and the asymptotic rate factor 1− τAlg . 3177 rAlg . 3t = O ( t ( σ0 − √ σ20 − 1 ) t/K ) , 1− τAlg . 3 = ( σ0 − √ σ20 − 1 ) 1/K . ( 23 ) Solving the system ( 21 ) The system is constructed by identification of the coefficients in both178 polynomials σΛK and σ ( λ ; { hi } , m ) , which can be solved using a naive grid-search for instance . We179 are not aware of any efficient algorithm to solve this system exactly , although it is possible to use180 iterative methods such as steepest descent or Newton ’ s method.181

Summary: This paper studies cyclical step sizes for heavy ball momentum. It is shown that under some conditions, it is possible to improve the convergence rate of heavy ball by relying on cyclical step sizes with K = 2. The result is obtained by a careful examine of the spectrum of the Hessian. The cases for larger K are also discussed. In general, the results are nice but also has its own limitation.