A Geometric Structure of Acceleration and Its Role in Making Gradients Small Fast

1 Introduction . Since Nesterov ’ s seminal 1983 work [ 57 ] , accelerated methods in optimization have been widely used in large-scale optimization and machine learning . However , the many accelerated methods have been developed and analyzed with disparate techniques , without a unified framework . In this work , we identify geometric structures , which we call the parallel and collinear structures , satisfied by a wide range of gradient- and prox-based accelerated methods including : Nesterov ’ s FGM [ 57 ] , OGM [ 47 ] , OGM-G [ 47 ] , Nesterov ’ s FGM in the strongly convex setup ( SC-FGM ) [ 60 , ( 2.2.22 ) ] , SC-OGM [ 63 ] , TMM [ 78 ] , non-stationary SC-FGM [ 19 , §4.5 ] , ITEM [ 73 ] , geometric descent [ 13 ] , Güler ’ s first and second accelerated proximal methods [ 37 ] , and FISTA [ 11 ] Using the insight provided by these geometric structures , we present several novel generalizations of accelerated methods . Among these novel generalizations , the most interesting is FISTA-G , which reduces the squared gradient norm with O ( 1/K4 ) rate when combined with FISTA in the prox-grad ( composite minimization ) setup . The rate is optimal as it matches the Ω ( 1/K4 ) complexity lower bound [ 55 , 56 ] and is faster than the O ( 1/K3 ) rates achieved by of Nesterov ’ s AGM [ 46 , 69 ] in the smooth convex setup or Kim and Fessler ’ s FPGM-m [ 45 ] in the prox-grad setup . 1.1 Preliminaries and notations . For L > 0 , f : Rn → R is L-smooth if f is differentiable and ‖∇f ( x ) −∇f ( y ) ‖ ≤ L‖x− y‖ ∀x , y ∈ Rn . For µ > 0 , g : Rn → R ∪ { ∞ } is µ-strongly convex if g ( x ) − ( µ/2 ) ‖x‖2 is convex . Let L and µ respectively denote the smoothness and strong convexity parameters of f . Write the gradient steps with stepsizes 1/L and 1/µ as x+ = x− 1 L ∇f ( x ) , x++ = x− 1 µ ∇f ( x ) . For λ > 0 , the proximal operator [ 53 , 62 ] Proxλg : Rn → Rn is defined as x◦ = Proxλg ( x ) = arg min y∈Rn { g ( y ) + 1 2λ ‖y − x‖2 } . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . When g is closed , convex , and proper [ 65 ] , the proximal operator is well-defined i.e. , the arg min exists and is unique [ 52 ] . Define the prox-grad step as x⊕ = arg min y∈Rn { f ( x ) + 〈∇f ( x ) , y − x〉+ g ( y ) + L 2 ‖y − x‖2 } . If g = 0 , then x⊕ = x+ . If f = 0 and 1/λ = L , then x⊕ = x◦ . Furthermore , write ∇̃LF ( x ) : = −L ( x⊕ − x ) , ∇̃1/λg ( x ) : = − 1 λ ( x◦ − x ) . u ∈ Rn is a subgradient of convex function g at x if g ( x ) + 〈u , y − x〉 ≤ g ( y ) ∀ y ∈ Rn . Subdifferential of g at x , denoted ∂g ( x ) , is the set of subgradients of g at x . Throughout this paper , we consider the problem minimize x∈Rn F ( x ) : = f ( x ) + g ( x ) , ( P ) where f : Rn → R is convex and L-smooth and g : Rn → R ∪ { ∞ } is closed , convex , and proper . We say we are in the smooth convex setup if f 6= 0 and g = 0 , the proximal-point setup if f = 0 and g 6= 0 , and the prox-grad setup if f 6= 0 and g 6= 0 . Write f ? , g ? , and F ? to respectively denote the infima of f , g , and F . We also informally assume Proxλg is efficient to evaluate [ 17 ] . 1.2 Prior works . In convex optimization and machine learning , the classical goal of algorithms is to reduce the function value efficiently . In the smooth convex setup , Nesterov ’ s celebrated fast gradient method ( FGM ) [ 57 ] achieves an accelerated rate , and the optimized gradient method ( OGM ) [ 44 ] improves this rate by a factor of 2 , which is in fact exactly optimal [ 27 ] . In the smooth strongly convex setup , the strongly convex fast gradient method ( SC-FGM ) [ 60 , ( 2.2.22 ) ] , strongly convex optimized gradient method ( SC-OGM ) [ 63 ] , and non-stationary SC-FGM [ 19 , §4.5 ] achieves an accelerated rate , and the triple momentum method ( TMM ) [ 78 ] and information theoretic exact method ( ITEM ) [ 73 ] achieve exact optimal rates [ 28 ] . In the proximal-point setup , there are Güler ’ s first and second accelerated proximal methods [ 37 ] . There are also other variants of accelerated proximal methods [ 51 , 49 , 39 , 77 ] . The study of algorithms for reducing the gradient magnitude , which can help us understand nonconvex optimization better and design faster non-convex machine learning algorithms , was initiated by Nesterov [ 58 ] . For smooth nonconvex minimization , gradient descent ( GD ) achieves O ( ( f ( x0 ) − f ? ) /K ) rate [ 54 , Proposition 3.3.1 ] . In the smooth convex setup , combiningK iterations of FGM with K iterations of GD achieves O ( ‖x0 − x ? ‖2/K3 ) rate [ 58 ] . Adding a strongly convex regularization and then using SC-FGM achieves Õ ( ‖x0 − x ? ‖2/K4 ) rate , where Õ ignores logarithmic factors [ 58 ] . Finally , OGM-G [ 47 ] achieved O ( ( f ( x0 ) − f ? ) /K2 ) rate and O ( ‖x0 − x ? ‖2/K4 ) rate is achieved by combining FGM with OGM-G [ 61 , Remark 2.1 ] . This rate is optimal as it matches the Ω ( ‖x0 − x ? ‖2/K4 ) lower bound [ 55 , 56 ] . Making gradients small have also been studied in the setup with stochastic gradients [ 2 , 3 , 33 , 79 , 34 ] , for composite problems with strong convexity and non-Euclidean norms [ 23 ] , and for convex-concave saddle-point problems [ 22 , 24 , 80 ] . In the prox-grad setup , iterative shrinkage-thresholding algorithm ( ISTA ) [ 12 , 64 , 20 , 18 ] achieves O ( ‖x0 − x ? ‖2/K ) rate on function-value suboptimality and fast iterative shrinkage-thresholding algorithm ( FISTA ) [ 11 ] accelerates this rate toO ( ‖x0−x ? ‖2/K2 ) . On the squared gradient magnitude , FPGM-m achieves O ( ‖x0 − x ? ‖2/K3 ) rate [ 45 ] . The performance estimation problem ( PEP ) is a computer-assisted proof methodology based on semidefinite programming [ 29 , 75 , 74 ] . Extensions and variations of the PEP have been utilized to obtain analyses and algorithms that would be difficult to obtain without the assistance of computers [ 67 , 72 , 10 , 76 , 21 , 36 , 50 , 26 , 43 ] . In particular , OGM [ 29 , 44 ] , OGM-G [ 47 ] , and ITEM [ 73 ] were obtained with the PEP . Integral quadratic constraints is a related computer-assisted approach based on control-theoretic ideas [ 48 , 38 , 78 , 31 ] . A Lyapunov analysis constructs a nonincreasing quantity , and many modern analyses of accelerated first-order methods are based on this technique [ 11 , 70 , 9 , 72 , 1 , 6 , 7 , 8 , 63 ] . In fact , Nesterov ’ s original presentation of FGM was a Lyapunov analysis [ 57 ] . Finally , we mention some closely related prior work . The scaled relative graph ( SRG ) analyzes optimization algorithms via Euclidean geometry [ 66 , 40 , 41 , 68 ] , but the SRG has not been used to analyze accelerated methods . Linear coupling [ 4 , 5 ] interprets acceleration as a coupling between gradient descent and mirror descent to efficiently reduce function values . Geometric descent [ 13 , 16 , 42 ] is an accelerated algorithm designed expressly based on geometric principles , and quadratic averaging [ 30 ] is an equivalent algorithm with an alternate interpretation . The method of similar triangles ( MST ) generalizes FGM with geometric notions [ 32 , 60 , 1 ] . The parallel and collinear structures of this work , in our view , expand these prior notions and concretely articulate ideas that had been utilized implicitly . In particular , the prior interpretations , as is , are insufficient for analyzing OGM-G and deriving our newly proposed method FISTA-G . In Section H of the appendix , we further discuss the relationship of our contributions with these prior works . 1.3 Contribution . This paper presents two major contributions , one conceptual and one concrete . The first is the identification of the parallel and collinear geometric structures , which are observed in a wide variety of accelerated first-order methods . This geometric structure provides us with valuable insight into the mechanism of acceleration and enables us to obtain results that would be otherwise difficult to discover , including FISTA-G . The second major contribution is the novel method FISTA-G and FISTA+FISTA-G. To the best of our knowledge , FISTA+FISTA-G is the first method to achieve the rate O ( 1/K4 ) on the squared gradient norm in the prox-grad setup . In addition , we use the geometric structures to find G-FISTA-G , G-FGM-G , G-Güler-G , ProximalTMM , and Proximal-ITEM , novel variants of accelerated first-order methods , and present them as minor contributions . ( G-FGM-G is the abbreviation for Generalized-FGM- ( for reducing Gradients ) . ) 2 Making gradients small at rate O ( 1/K4 ) in the prox-grad setup In this section , we present a novel method FISTA-G , which reduces the squared gradient norm with O ( 1/K4 ) rate when combined with FISTA in the prox-grad setup . Our analysis uses a novel and unusual Lyapunov function , whose discovery crucially relied on the geometric insights presented later in Section 3 . Here , we provide a self-contained analysis that uses , but does not explain this Lyapunov function . Smooth convex setup . Kim and Fessler ’ s ( OGM-G ) [ 47 ] is xk+1 = x + k + ( θk − 1 ) ( 2θk+1 − 1 ) θk ( 2θk − 1 ) ( x+k − x + k−1 ) + 2θk+1 − 1 2θk − 1 ( x+k − xk ) for k = 0 , 1 , . . . , K − 1 , where x+−1 : = x0 , θK = 1 , θk = 1+ √ 1+4θ2k+1 2 for k = 1 , 2 , . . . , K − 1 , and θ0 = 1+ √ 1+8θ21 2 . Note , K is the total number of iterations and θ0 = O ( K ) , so θ0 is a function of K. OGM-G is the first method to achieve the accelerated rate O ( ( f ( x0 ) − f ? ) /K2 ) on the squared gradient norm , and the method was originally obtained through a computer-assisted methodology . However , the computer-generated analysis is verifiable but arguably difficult to understand . We characterize the convergence of OGM-G with the following novel Lyapunov function Uk = 1 θ2k ( 1 2L ‖∇f ( xK ) ‖2 + 1 2L ‖∇f ( xk ) ‖2 + f ( xk ) − f ( xK ) − 〈 ∇f ( xk ) , xk − x+k−1 〉 ) + L θ4k 〈 zk − x+k−1 , zk − x + K 〉 , for k = 1 , 2 , . . . , K , where zk = xk + ( θk−1 ) 2 2θk−1 ( xk − x + k−1 ) for k = 1 , 2 , . . . , K , ( note zK = xK ) through the steps 1 L ‖∇f ( xK ) ‖2 = UK ≤ UK−1 ≤ · · · ≤ U1 ≤ 2 θ20 ( 1 2L ‖∇f ( xK ) ‖2 + f ( x0 ) − f ( xK ) ) . Section 3 explains the geometric insights behind the discovery of this Lyapunov function and the proof that { Uk } Kk=1 is nonincreasing . As related work , Diakonikolas and Wang [ 24 ] presented a human-understandable analysis of OGM-G , but their analysis is not a Lyapunov analysis since ( as they acknowledge ) they do not establish a nonincreasing quantity . In any case , our main contribution of this section is the following generalization of the method and analysis to the prox-grad setup . 2.1 Accelerated O ( ( F ( x0 ) − F ? ) /K2 ) rate with FISTA-G We now present the novel method ( FISTA-G ) : xk+1 = x ⊕ k + ϕk+1 − ϕk+2 ϕk − ϕk+1 ( x⊕k − x ⊕ k−1 ) for k = 0 , 1 , . . . , K − 1 , where x⊕−1 : = x0 , ϕK+1 = 0 , ϕK = 1 , and ϕk = ϕ2k+2 − ϕk+1ϕk+2 + 2ϕ2k+1 + ( ϕk+1 − ϕk+2 ) √ ϕ2k+2 + 3ϕ 2 k+1 ϕk+1 + ϕk+2 for k = −1 , 0 , . . . , K−1 . Note that K is the total number of iterations . Theorem 1 . Consider ( P ) . FISTA-G ’ s final iterate xK exhibits the rate min ∥∥∂F ( x⊕K ) ∥∥2 ≤ 4 ∥∥∥∇̃LF ( xK ) ∥∥∥2 ≤ 264L ( K + 2 ) 2 ( F ( x0 ) − F ? ) . We clarify that min ‖∂F ( x ) ‖2 = min { ‖u‖2 |u ∈ ∂F ( x ) } for x ∈ Rn . Proof outline . Define z0 = x0 , zk = ϕkϕk−ϕk+1xk − ϕk+1 ϕk−ϕk+1x ⊕ k−1 for k = 0 , 1 , . . . , K , and Uk = 2ϕk−1 ( ϕk−1 − ϕk ) 2 ( 1 2L ∥∥∥∇̃LF ( xk ) ∥∥∥2 + F ( x⊕k ) − F ( x⊕K ) − 〈∇̃LF ( xk ) , xk − x⊕k−1〉 ) + L ϕk 〈 zk − x⊕k−1 , zk − x ⊕ K 〉 for k = 0 , 1 , . . . , K. ( Note that zK = xK . ) We can show that { Uk } Kk=0 is nonincreasing . Then 1 2L ∥∥∥∇̃LF ( xK ) ∥∥∥2 = UK ≤ · · · ≤ U0 = 2ϕ−1 ( ϕ−1 − ϕ0 ) 2 ( 1 2L ∥∥∥∇̃LF ( x0 ) ∥∥∥2 + F ( x⊕0 ) − F ( x⊕K ) ) ≤ 2ϕ−1 ( ϕ−1 − ϕ0 ) 2 ( F ( x0 ) − F ( x⊕K ) ) ≤ 33L ( K + 2 ) 2 ( F ( x0 ) − F ? ) , where we used 12L ∥∥∥∇̃LF ( x0 ) ∥∥∥2 ≤ F ( x0 ) − F ( x⊕0 ) , stated as Lemma 9 in the appendix . The complete proof is presented in Section E of the appendix . In fact , FISTA-G is the best instance within the family of methods G-FISTA-G presented in Section E of the appendix , in the sense that FISTA-G is the instance for which we could obtain the smallest constant in the bound .

Summary: The paper discovers parallel and co-linear structures in known accelerated first order methods and claims that these are important ingredients to achieve acceleration. The introduction of parallel structure in an optimization algorithm yields a standard way of designing a discrete Lyapunov function to analyze its convergence. A new Lyapunov analysis for OGM-G is obtained, but also a new method FISTA-G, which, combined with FISTA, yields to an improvement in the state of the art regarding convergence of gradient in the proximal gradient setting. An experiment in compressed sensing is presented as well.