Rethinking the Variational Interpretation of Accelerated Optimization Methods

1 Introduction . This paper focuses on the problem of unconstrained convex optimization , i.e . to find x∗ ∈ arg min x∈Rd f ( x ) , ( P ) for some lower bounded convex L-smooth2 loss f ∈ C1 ( Rd , R ) . Nesterov ’ s acceleration . Nemirovskii and Yudin ( 1983 ) showed that no gradient-based optimizer can converge to a solution of ( P ) faster than O ( k−2 ) , where k is the number of gradient evaluations3 . While Gradient Descent ( GD ) converges like O ( k−1 ) , the optimal rate O ( k−2 ) is achieved by the celebrated Accelerated Gradient Descent ( AGD ) method , proposed by Nesterov ( 1983 ) : xk+1 = yk − η∇f ( yk ) , with yk = xk + k − 1 k + 2 ( xk − xk−1 ) . ( AGD ) The intuition behind Nesterov ’ s method and the fundamental reason behind acceleration is , to this day , an active area of research ( Allen-Zhu and Orecchia , 2014 ; Defazio , 2019 ; Ahn , 2020 ) . ODE models . Towards understanding the acceleration mechanism , Su et al . ( 2016 ) made an interesting observation : the convergence rate gap between GD and AGD is retained in the continuoustime limits ( as the step-size η vanishes ) : Ẋ +∇f ( X ) = 0 ( GD-ODE ) , Ẍ + 3 t Ẋ +∇f ( X ) = 0 ( AGD-ODE ) ∗Equal Contribution . 2A differentiable function f : Rd → R is said to be β-smooth if it has β-Lipschitz gradients . 3This lower bound holds just for k < d hence it is only interesting in the high-dimensional setting . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . where Ẋ : = dX/dt denotes the time derivative ( velocity ) and Ẍ : = d2X/dt2 the acceleration . Namely , we have that GD-ODE converges like O ( t−1 ) and AGD-ODE like O ( t−2 ) , where t > 0 is the time variable . This seminal paper gave researchers a new tool to understand the nature of accelerated optimizers through Bessel Functions ( Su et al. , 2016 ) , and led to the design of many novel fast and interpretable algorithms outside the Euclidean setting ( Wibisono et al. , 2016 ; Wilson et al. , 2019 ) , in the stochastic setting ( Krichene et al. , 2015 ; Xu et al. , 2018 ) and also in the manifold setting ( Alimisis et al. , 2020 ; Duruisseaux and Leok , 2021 ) . Nesterov as solution to Euler-Lagrange equations . It is easy to see that AGD-ODE can be recovered from Euler-Lagrange equations , starting from the time-dependent Lagrangian L ( X , Ẋ , t ) = t3 ( 1 2 ‖Ẋ‖2 − f ( X ) ) . ( 1 ) Indeed , the Euler-Lagrange equation d dt ( ∂ ∂Ẋ L ( X , Ẋ , t ) ) = ∂ ∂X L ( X , Ẋ , t ) ( 2 ) reduces in this case to t3Ẍ + 3t2Ẋ + t3∇f ( X ) = 0 , which is equivalent to AGD-ODE ( assuming t > 0 ) . In a recent influential paper , Wibisono et al . ( 2016 ) generalized the derivation above to non-Euclidean spaces , where the degree of separation between points x and y is measured by means of the Bregman Divergence ( Bregman , 1967 ) Dψ ( x , y ) = ψ ( y ) − ψ ( x ) − 〈∇ψ ( x ) , y − x〉 , where ψ : Rd → R is a strictly convex and continuously differentiable function ( see e.g . Chapter 1.3.2 in Amari ( 2016 ) ) . Namely , they introduced the so-called Bregman Lagrangian : Lα , β , γ ( X , Ẋ , t ) = e α ( t ) +γ ( t ) ( Dψ ( X + e −α ( t ) V , X ) − eβ ( t ) f ( X ) ) , ( 3 ) where α , β , γ are continuously differentiable functions of time . The Euler-Lagrange equations imply Ẍ + ( eα ( t ) − α̇ ( t ) ) Ẋ + e2α ( t ) +β ( t ) [ ∇2ψ ( X + e−α ( t ) Ẋ ) ] −1 ∇f ( X ) = 0 . ( 4 ) The main result of Wibisono et al . ( 2016 ) is that , under the ideal-scaling conditions β̇ ( t ) ≤ eα ( t ) and γ̇ ( t ) = eα ( t ) , any solution to Eq . ( 4 ) converges to a solution of ( P ) at the rate O ( e−β ( t ) ) . Under the choice ψ ( x ) = 12‖x‖ 2 2 , we get back to the Euclidean metric Dψ ( x , y ) = 1 2‖x− y‖ 2 2 . Moreover , choosing α ( t ) = log ( 2/t ) , β ( t ) = γ ( t ) = 2 log ( t ) , we recover the original Lagrangian in Eq . ( 1 ) and O ( e−β ( t ) ) = O ( t−2 ) , as derived in Su et al . ( 2016 ) . Impact of the variational formulation . The variational formulation in Wibisono et al . ( 2016 ) has had a considerable impact on the recent developments in the theory of accelerated methods . Indeed , this approach can be used to design and analyze new accelerated algorithms . For instance , Xu et al . ( 2018 ) used the Lagrangian mechanics formalism to derive a novel simplified variant of accelerated stochastic mirror descent . Similarly , França et al . ( 2021 ) , Muehlebach and Jordan ( 2021 ) used the dual Hamiltonian formalism to study the link between symplectic integration of dissipative ODEs and acceleration . Due to its rising importance in the field of optimization , the topic was also presented by Prof. M. I. Jordan as a plenary lecture at the International Congress of Mathematicians in 2018 ( Jordan , 2018 ) , centered around the question “ what is the optimal way to optimize ? ” . Imprecise implications of the variational formulation . While the Lagrangian formalism has been inspiring and successful for algorithm design and analysis , its precise implications for the geometry and the path of accelerated solutions have not been examined in a mathematically rigorous way ( to the best of our knowledge ) . In Jordan ( 2018 ) it is hinted that , since Nesterov ’ s method solves the Euler-Lagrange equations , it minimizes the action functional ∫ t2 t1 Lα , β , γ ( Y , Ẏ , t ) dt over the space of curves by the minimum action principle of classical mechanics ( Arnol ’ d , 2013 ) . This claim4 is inaccurate . Indeed , the term minimum action principle is misleading5 : solving Euler-Lagrange 4Paragraph before Eq . ( 9 ) in Jordan ( 2018 ) : “ [ ... ] we use standard calculus of variations to obtain a differential equation whose solution is the path that optimizes the time-integrated Bregman Lagrangian ” . 5From Section 36.2 in Gelfand and Fomin ( 2000 ) : “ The principle of least action is widely used [ ... ] . However , in a certain sense the principle is not quite true [ ... ] . We shall henceforth replace the principle of least action by the principle of stationary action . In other words , the actual trajectory of a given mechanical system will not be required to minimize the action but only to cause its first variation to vanish. ” only makes the action stationary ( necessary condition : vanishing first-order derivative ) , but does not guarantee minimality — this only holds in physics for very special cases6 , which do not include even simple mechanical systems like the pendulum ( proof in Section 36.2 of Gelfand and Fomin ( 2000 ) ) . Indeed , from a theoretical perspective , the claim requires computing the second variation along Nesterov ’ s path . Quite surprisingly , even though many papers are dedicated to the variational formulation ( Wibisono et al. , 2016 ; Jordan , 2018 ; Casgrain , 2019 ; Duruisseaux and Leok , 2021 ) , to the best of our knowledge there is no work which provides an in-depth rigorous study of the action relative to Bregman Lagrangian and that characterizes minimality of Nesterov in the space of curves . Our contributions . Intrigued by the non-trivial open question of minimality of Nesterov ’ s path and by the enigmatic geometry of accelerated flows , in this paper , we examine the properties of accelerated gradient methods from the perspective of calculus of variations . 1 . In Sec . 3 we study the minimality of classical Nesterov ’ s ODE ( damping 3/t ) proposed by Su et al . ( 2016 ) on multidimensional quadratic losses . By using Jacobi ’ s theory for the second variation ( summarized in Sec . 2 ) , we find that Nesterov ’ s path is optimal only if the integration interval [ t1 , t2 ] is small enough . In contrast , if t2 − t1 > √ 40/β ( β is Lipschitz constant for the gradient ) , Nesterov ’ s path is actually a saddle point for the action ( see Fig . 1 ) . 2 . In Sec . 4 we extend the analysis to the µ-strongly convex setting and thus consider a constant damping α . We show that , for extremely overdamped Nesterov flows ( α ≥ 2 √ β ) , i.e for highly suboptimal parameter tuning ( acceleration holds only for α ≈ 2√µ ) , Nesterov ’ s path is always a minimizer for the action . In contrast , we show that for α < 2 √ β ( acceleration setting ) , if t2 − t1 > 2π/ √ 4β − α , Nesterov ’ s path is again a saddle point for the action . 3 . In Sec . 5 we discuss the implications of our results for the theory of accelerated methods and propose a few interesting directions for future research . We start by recalling some definitions and results from calculus of variations , which we adapt from classical textbooks ( Landau and Lifshitz , 1976 ; Arnol ’ d , 2013 ; Gelfand and Fomin , 2000 ) . 2 Background on calculus of variations . We work on the vector space of curves C1 ( [ t1 , t2 ] , Rd ) with t1 , t2 ∈ [ 0 , ∞ ) . We equip this space with the standard norm ‖Y ‖ = maxt1≤t≤t2 ‖Y ( t ) ‖2 + maxt1≤t≤t2 ‖Ẏ ( t ) ‖2 . Under this choice , for any regular Lagrangian L , the action functional J [ Y ] : = ∫ t2 t1 L ( Y , Ẏ , t ) dt is continuous . First variation . Let D be the linear subspace of continuously differentiable displacements curves h such that h ( t1 ) = h ( t2 ) = 0 . The corresponding increment of J at Y along h is defined as ∆J [ Y ; h ] : = J [ Y + h ] − J [ Y ] . Suppose that we can write ∆J [ Y ; h ] = ϕ [ Y ; h ] + ‖h‖ , where ϕ is linear in h and → 0 as ‖h‖ → 0 . Then , J is said to be differentiable at Y and the linear functional δJ [ Y ; · ] : D → R such that δJ [ Y ; h ] : = ϕ [ Y ; h ] is called first variation of J at Y . It can be shown that , if J is differentiable at Y , then its first variation at Y is unique . Extrema . J is said to have an extremum at Y if ∃δ > 0 such that , ∀h ∈ D with ‖h‖ ≤ δ , the sign of J [ Y + h ] − J [ Y ] is constant . A necessary condition for J to have an extremum at Y is that δJ [ Y ; h ] = 0 , for all h ∈ D. ( 5 ) 6e.g . free particle in vanishing potentials , or t1 ≈ t2 , see Remark 2 at the end of Section 21 and Section 36.2 . We proceed with stating one of the most well-known results in calculus of variations , which follows by using Taylor ’ s theorem on J [ Y ] = ∫ t2 t1 L ( Y , Ẏ , t ) dt . Theorem 1 ( Euler-Lagrange equation ) . A necessary condition for the curve Y ∈ C1 ( [ t1 , t2 ] , Rd ) to be an extremum for J ( w.r.t . D ) is that it satisfies the Euler-Lagrange equations ( 2 ) . It is crucial to note that Theorem 1 provides a necessary , but not sufficient condition for an extremum — indeed , the next paragraph is completely dedicated to this . Second Variation . Thm . 1 does not distinguish between extrema ( maxima or minima ) and saddles . For this purpose , we need to look at the second variation . Suppose that the increment of J at Y can be written as ∆J [ Y ; h ] = ϕ1 [ Y ; h ] + ϕ2 [ Y ; h ] + ‖h‖2 , where ϕ1 is linear in h , ϕ2 is quadratic in h and → 0 as ‖h‖ → 0 . Then J is said to be twice differentiable and the functional δ2J [ Y : , · ] : D → R s.t . δ2J [ Y , h ] : = ϕ2 [ Y ; h ] is called the second variation of J at Y . Uniqueness of second variation is proved in the same way as the first variation . Theorem 2 . A necessary condition for the curve Y ∈ C1 ( [ t1 , t2 ] , Rd ) to be a local minimum for J ( w.r.t D ) is that it satisfies δ2J [ Y ; h ] ≥ 0 . For local maxima , the sign is flipped . Jacobi equations . Recall that J [ Y ] = ∫ t2 t1 L ( Y , Ẏ , t ) dt . Using the notation LY Z = ∂2L/ ( ∂Y ∂Z ) , the Taylor expansion for ∆J [ Y ; h ] = J [ Y + h ] − J [ Y ] if ‖h‖ → 0 converges to ∆J [ Y ; h ] = ∫ t2 t1 ( LY h+ LẎ ḣ ) dt+ 1 2 ∫ t2 t1 ( LY Y h 2 + LẎ Ẏ ḣ 2 + 2LY Ẏ hḣ ) dt , ( 6 ) where the equality holds coordinate-wise . Therefore , δJ [ Y ; h ] = ∫ t2 t1 ( LY h+ LẎ ḣ ) dt and δ2J [ Y ; h ] = 1 2 ∫ t2 t1 ( LY Y h 2 + 2LY Ẏ hḣ+ LẎ Ẏ ḣ 2 ) dt = 1 2 ∫ t2 t1 ( LY Y − d dt LY Ẏ ) h2dt+ 1 2 ∫ t2 t1 LẎ Ẏ ḣ 2dt = 1 2 ∫ t2 t1 ( Pḣ2 +Qh2 ) dt , ( 7 ) where P = LẎ Ẏ , Q = LY Y − d dtLY Ẏ , and the second equality follows from integration by parts since h vanishes at t1 and t2 . Using this expression , it is possible to derive an easy necessary ( but not sufficient ) condition for minimality . Theorem 3 ( Legendre ’ s necessary condition ) . A necessary condition for the curve Y to be a minimum of J is that LẎ Ẏ is positive semidefinite . Conjugate points . A crucial role in the behavior of δ2J [ Y ; h ] is played by the shape of the solutions to Jacobi ’ s differential equation ddt ( Pḣ ) −Qh = 0 . A point t ∈ ( t1 , t2 ) is said to be conjugate to point t1 ( w.r.t . J ) if Jacobi ’ s equation admits a solution which vanishes at both t1 and t but is not identically zero . We have the following crucial result . Theorem 4 ( Jacobi ’ s condition ) . Necessary and sufficient conditions for Y to be a local minimum for J are : ( 1 ) Y satiesfies the Euler-Lagrange Equation ; ( 2 ) P positive definite ; ( 3 ) ( t1 , t2 ) contains no points conjugate to t1 . 3 Analysis of the action of Nesterov ’ s path with vanishing damping 3/t This section is dedicated to the analysis of the action functional relative to Eq . ( 3 ) . By incorporating the tool of variational calculus , we study the optimality of Nesterov ’ s method in minimizing the action . We start by a general abstract analysis in the convex quadratic case in Sec . 3.1 , and then present an intuitive analytical computation in Sec . 3.2 . The non-quadratic case is discussed in Sec . 3.4 .

Summary: The authors try to understand the low-resolution ODE for Nesterov accelerated gradient descent in convex case from the variational perspective. Meanwhile, they find the ODE is not the minimizer of the action functional, but the saddle. From mathematical perspective, it is a new and novel point to investigate.