Title: A Variational Perspective on High-Resolution ODEs
Venue: NeurIPS 2023
Accepted: yes
Keywords: Nesterov's accelerated gradient

gradient descent

Lyapunov function

gradient norm minimization

rate-matching

stochastic variance reduction

stochastic gradient descent

noisy gradient
Abstract: We consider unconstrained minimization of smooth convex functions. We propose a novel variational perspective using forced Euler-Lagrange equation that allows for studying high-resolution ODEs. Through this, we obtain a faster convergence rate for gradient norm minimization using Nesterov's accelerated gradient method. Additionally, we show that Nesterov's method can be interpreted as a rate-matching discretization of an appropriately chosen high-resolution ODE. Finally, using the results from the new variational perspective, we propose a stochastic method for noisy gradients. Several numerical experiments compare and illustrate our stochastic algorithm with state of the art methods.

A Variational Perspective on High-Resolution ODEs

Abstract

We consider unconstrained minimization of smooth convex functions. We propose a novel variational perspective using forced Euler-Lagrange equation that allows for studying high-resolution ODEs. Through this, we obtain a faster convergence rate for gradient norm minimization using Nesterov's accelerated gradient method. Additionally, we show that Nesterov's method can be interpreted as a ratematching discretization of an appropriately chosen high-resolution ODE. Finally, using the results from the new variational perspective, we propose a stochastic method for noisy gradients. Several numerical experiments compare and illustrate our stochastic algorithm with state of the art methods.

1 Introduction

Smooth convex minimization is a fundamental class in optimization with broad applications and a rich theoretical foundation that enables the development of powerful methods. In fact, the theory and methods developed for other problem classes, such as non-smooth convex or smooth non-convex optimization, often build upon the work done in smooth convex optimization. As a result, the study of smooth convex minimization has a significant impact on the broader field of optimization. Over the last two decades, first-order methods -those relying solely on gradient information, in contrast to methods like Newton's that require Hessian information- have seen a lot of interest both in theory and applications due to their efficiency and adaptability for large-scale data-driven applications. Among these, gradient descent stands as one of the simplest and oldest. When equipped with an appropriate step-size, gradient descent guarantees a suboptimality gap (objective residual) of order O (1 /k ) after k iterations.

In his seminal work, Nesterov [1983] has shown that the gradient method can achieve faster rates by incorporating momentum deviations. Nesterov's accelerated gradient algorithm (NAG) ensures a convergence rate of O (1 /k 2 ) which is an order of magnitude faster than the gradient descent. Remarkably, this rate matches information-theoretical lower bounds for first-order oracle complexity, meaning that NAG is optimal and no other first-order method can guarantee an essentially faster convergence rate [Nesterov, 2003].

The original proof of Nesterov [1983], known as the estimate sequence technique , is a highly algebraic and complex procedure, difficult to interpret, and provides arguably limited insight into why momentum deviations help with the convergence rates [Hu and Lessard, 2017]. Therefore, many researchers have tried to provide a better understanding of momentum-based acceleration through different perspectives. For example, Su et al. [2016], Shi et al. [2021], and Sanz Serna and Zygalakis [2021] consider a continuous time perspective ; Lessard et al. [2016] and Fazlyab et al. [2018] use integral quadratic constraints and control systems with non-linear feedbacks; Muehlebach and Jordan [2019, 2022, 2023] present a dynamical perspective; Attouch et al. [2020, 2021] utilize inertial dynamic involving both viscous damping and Hessian-driven damping; Zhu and Orecchia [2014] view acceleration as a linear coupling of gradient descent and mirror descent updates; and Ahn and Sra [2022] provide an understanding of the NAG algorithm through an approximation of the proximal point method. Our work is focused on the continuous-time perspective .

In [Su et al., 2016], the authors derive a second-order ordinary differential equation (ODE) that exhibits trajectories similar to those of the NAG algorithm in the limit of an infinitesimal step-size s → 0 . This result has inspired researchers to analyze various ODEs and discretization schemes to gain a better understanding of the acceleration phenomenon. Notably, Wibisono et al. [2016] demonstrate that the ODE presented in [Su et al., 2016] is a special case of a broader family of ODEs that extend beyond Euclidean space. They achieve this by minimizing the action on a Lagrangian that captures the properties of the problem template, an approach known as the variational perspective , and they discretize their general ODE using the rate-matching technique. In a related vein, Shi et al. [2021] proposed substituting the low-resolution ODEs (LR-ODEs) introduced in [Su et al., 2016] with high-resolution ODEs (HR-ODEs) which can capture trajectories of NAG more precisely.

Our main contribution in this paper is a novel extension of the variational perspective for HRODEs. A direct combination of these two frameworks is challenging, as it remains unclear how the Lagrangian should be modified to recover HR-ODEs. To address this problem, we propose an alternative approach that preserves the Lagrangian but extends the variational perspective. More specifically, instead of relying on the conventional Euler-Lagrange equation, we leverage the forced Euler-Lagrange equation that incorporates external forces acting on the system. By representing the damped time derivative of the potential function gradients as an external force, our proposed variational perspective allows us to reconstruct various HR-ODEs through specific damping parameters. More details are provided in Section 2.

Other contributions of our paper are as follows: In Section 3, we show that our proposed variational analysis yields a special representation of NAG leading to superior convergence rates than [Shi et al., 2019] in terms of gradient norm minimization. In Section 4, we propose an HR-ODE based on the rate-matching technique. We demonstrate that NAG can be interpreted as an approximation of the rate-matching technique applied to a specific ODE. Then, in Section 5, we extend our analysis to a stochastic setting with noisy gradients, introducing a stochastic method that guarantees convergence rates of ˜ O (1 /k 1 / 2 ) for the expected objective residual and ˜ O (1 /k 3 / 4 ) for the expected squared norm of the gradient. Finally, in Section 6, we present numerical experiments to demonstrate the empirical performance of our proposed method and to validate our theoretical findings.

Problem template and notation. We consider a generic unconstrained smooth convex minimization template:

where f : R n → R is convex and L -smooth, meaning that its gradient is Lipschitz continuous:

with ‖ · ‖ denoting the Euclidean norm. We denote the class of L -smooth convex functions by F L .

The NAG Algorithm. Given an initial state x 0 = y 0 ∈ R n and a step-size parameter s > 0 , the NAG algorithm updates the variables x k and y k iteratively as follows:

Throughout, we assume that the solution set for Problem (1) is non-empty, and we denote an arbitrary solution by x ∗ , hence f ∗ := f ( x ∗ ) ≤ f ( x ) for all x ∈ R n .

Low-Resolution and High-Resolution ODEs. Throughout, by LR-ODE, we refer to the secondorder ODE introduced and studied in [Su et al., 2016], which exhibits a behavior reminiscent of momentum-based accelerated methods like Heavy Ball (HB) and NAG in the limit of step-size s → 0 . Unfortunately, this generic ODE cannot distinguish the differences between HB and NAG. It is important to note, however, that the guarantees of HB and NAG differ significantly in discrete time. Specifically, HB has guarantees for only a specific subset of Problem (1), whereas NAG guarantees a solution for all instances of (1) with appropriate step sizes. Therefore, it's worth noting that the LR-ODE may not fully capture certain important aspects of the NAG trajectory. Conversely, by HR-ODEs, we refer to the ODEs proposed in [Shi et al., 2021], which extend the LR-ODE by incorporating gradient correction terms. Through this extension, HR-ODEs provide a more precise representation of the trajectory for these distinct algorithms.

2 External Forces and High-Resolution ODEs

Consider the Lagrangian

Here, ˙ X t ∈ R d is the first time-derivative of X ( t ) , and α t , β t , γ t : T → R are continuously differentiable functions of time that correspond to the weighting of velocity, the potential function f , and the overall damping, respectively. Using variational calculus, we define the action for the curves { X t : t ∈ R } as the functional A ( X ) = ∫ R L ( X t , ˙ X t , t ) dt . In the absence of external forces, a curve is a stationary point for the problem of minimizing the action A ( X ) if and only if it satisfies the Euler Lagrange equation d dt { ∂ L ∂ ˙ X t ( X t , ˙ X t , t ) } = ∂ L ∂X t ( X t , ˙ X t , t ) . This was used in [Wibisono et al., 2016, Wilson et al., 2021] to calculate the LR-ODEs for convex and strongly convex functions 1 . Note that the Euler-Lagrange equation as written, does not account for an external force, F (which is non-conservative). In this case, the Euler-Lagrange equation should be modified to the forced Euler-Lagrange Equation

which itself is the result of integration by parts of Lagrange d'Alembert principle [Campos et al., 2021]. Using the Lagrangian (3) we have

Substituting (5) in (4) gives

In what follows, we will present various choices of the external force F , including two for convex functions and one for strongly convex functions.

2.1 Convex Functions

First choice for convex functions. Let us first consider the following external force:

In this case, (6) becomes

It is possible to show the convergence of X t to x ∗ and establish a convergence rate for this as shown in the following theorem (proof in Appendix A.1). The proof of this theorem (and the subsequent theorems in this section) is based on the construction of a suitable Lyapunov function for the corresponding ODE ( e.g. see [Siegel, 2019, Shi et al., 2019, Attouch et al., 2020, 2021]). This non-negative function attains zero only at the stationary solution of the corresponding ODE and decreases along the trajectory of the ODE [Khalil, 2002]. For this theorem (and the subsequent theorems), we will define a proper Lyapunov function and prove sufficient decrease of the function f along the corresponding ODE trajectory.

Theorem 2.1. Under the ideal scaling conditions ˙ β e α t and ˙ γ = e α t , X in (8) will satisfy for f ∈ F L .

Now, choosing parameters as

in (8) gives

which reduces to

by taking n ( t ) = p t , q ( t ) = Cpt p -1 . Remark 2.1.1 . For p = 2 , C = 1 / 4 , equation (11) corresponds to the (H-ODE) in [Laborde and Oberman, 2020].

Second choice for convex functions. Now, we consider an external force given by

In this case, replacing F in (6) gives

We establish the following convergence result, and the proof can be found in Appendix A.2.

Theorem 2.2. Under the modified ideal scaling conditions ˙ β t ≤ e α t , ˙ γ t = e αt , ¨ β t ≤ e α t ˙ β t +2˙ α t ˙ β t , X t in (13) will satisfy for f ∈ F L .

Taking the same parameters as in (9) gives

for n ( t ) = p/t, q ( t ) = Cpt p -1 .

Remark 2.2.1 . Note that setting C = 1 / 4 , p = 2 will lead to the ODE

This ODE was discretized using the Semi-Implicit Euler (SIE) and the Implicit Euler (IE) discretization schemes in [Shi et al., 2019]. The corresponding optimization algorithms were shown to accelerate. In addition, note that the convergence rate proved in Theorem 2.2 is faster than its counterpart in Theorem 2.1.

2.2 Strongly Convex Functions

Our analysis is applicable to strongly convex functions as well. Consider the Lagrangian proposed in [Wilson et al., 2021] for strongly convex functions

Then, the forced Euler-Lagrange equation (4) becomes

Taking F = - √ se α t + γ t d dt ( e β t ∇ f ( X t )) in (18) gives

We can establish the following convergence result for X t in (19) to the unique minimizer x ∗ . The proof of this result is deferred to Appendix A.3.

Theorem 2.3. Under the modified ideal scaling conditions α t = α , ˙ β t ≤ e α t , ˙ γ t = e α t , and ˙ β t ≥ 0 X t in (19) satisfies for µ -strongly convex function f .

Remark 2.3.1 . Taking α = log( √ µ ) and γ t = β t = √ µt in (19) gives the NAG's corresponding HR-ODE

for µ -strongly convex function f as in [Shi et al., 2021].

3 Gradient Norm Minimization of NAG

One of the implications of our variational study on HR-ODEs in Section 2 was the ODE (14). Reformulating this ODE gives

Applying the SIE on (22) for X ( t ) ≈ X ( t k ) , V ( t ) ≈ V ( t k ) , n ( t k ) = p/t k , q ( t k ) = Cpt p -1 k , p = 2 , t k = k √ s and C = 1 / 4 gives

which is exactly the NAG algorithm. The interpretation of the NAG method as the SIE discretization of (22) has not been discussed before in the literature (see [Ahn and Sra, 2022] for the four most studied representations). It is precisely this connection with the ODE (22) though that inspires our choice of the Lyapunov function which in turn gives rise to a faster convergence rate. The following theorem formulates this result. The proof is in Appendix A.4 and it is based on the discrete Lyapunov analysis of (23). Similar convergence rate was very recently found by [Chen et al., 2022] through implicit velocity perspective on HR-ODEs which uses a different Lyapunov analysis than this work.

Theorem 3.1. Consider the update (23). Then, if f ∈ F L we have and

for 0 ≤ s ≤ 1 /L , v 0 = x 0 , and any x 0 ∈ R n .

Remark 3.1.1 (Comparison with state of the art) . The rate in Theorem 3.1 is improved compared to the previous rate found in [Shi et al., 2021], which is

for 0 < s ≤ 1 / (3 L ) and k ≥ 0 .

4 Rate-Matching Approximates the NAG Algorithm

The ODE (11) when p = 2 , C = 1 / 4 is equivalent to

which can be viewed as a perturbation of the LR-ODE

We now show that when the rate-matching technique in [Wibisono et al., 2016] is applied to (25), the final algorithm reveals similar behavior as (24). This result is then used to approximately recover the NAG method using rate-matching discretization.

Applying the rate-matching discretization on the ODE (25) gives

Proposition 4.1. The continuous-time behavior of (26) is approximately

 which has a convergence rate of O (1 / ( sk 2 )) [Wibisono et al., 2016]. In the following proposition, we study the behavior of (26) in limit of s → 0 . The proof is given in Appendix A.5.

which is the the high-resolution ODE (11).

The ODE (27) is the same as (24). In this sense, rate-matching implicitly perturbs the LR-ODE. The question that naturally arises is that when do we recover the HR-ODE (16) (which corresponds to the NAG algortihm through the SIE discretization) from the rate-matching technique? To answer, we will first perturb the LR-ODE (25) in the second line. Then, the rate-matching discretization is applied. Perturbing (25) gives

Discretizing (28) using the rate-matching method with t k = k √ s gives

 which is extremely close to the NAG algorithm. Indeed, replacing ∇ f ( y k ) with ∇ f ( x k ) in the third line of (29) gives exactly the NAG method. Typically, x k and y k are very close. This is due to x k and y k having a difference of order s . Since in continuous time X ( t k ) ≈ Y ( t k ) (due to s → 0 ), the HR-ODE of (29) is (16). This means that the corresponding HR-ODE of (29) is

which is the perturbed version of (28) and the HR-ODE associated with the NAG algorithm.

5 Stochastic Extensions

In this section, we propose a stochastic variation of (23). We model noisy gradients by adding i.i.d noise e k with variance σ 2 to the gradients. Consider the update

with β ≥ 2 . This update reduces to (23) when e k = 0 , s k = √ s = β/ √ L, t k = k √ s . We will refer to (31) as the Noisy NAG (NNAG) algorithm. NNAG is interesting due to its capability of dealing with perturbed gradients. This is the case in practical methods e.g. SGD [Bottou, 2010], SAG [Schmidt et al., 2017], SAGA [Defazio et al., 2014], SVRG [Johnson and Zhang, 2013], and etc. The following convergence result holds for NNAG, and its proof is in Appendix A.6.

Theorem 5.1. Suppose f ∈ F L and consider the NNAG method detailed in (31) with the following parameter choices:

We define the critical iteration k 0 as the smallest positive integer that satisfies

Then, the following bounds hold for all k ≥ k 0 :

if α > 3 / 4 , and

Next, we show that slight modifications to the NNAG method gives rise to another stochastic method with a similar convergence rate as the NNAG algorithm, but more transparent proof (see Appendix A.7). This proof results in a convergence rate for E [ min 0 ≤ i ≤ k -1 ‖∇ f ( x i ) ‖ 2 ] with a rate of O (log( k ) /k (3 / 4) ) . It remains a future work to show similar result for the NNAG update.

Theorem 5.2. Suppose f ∈ F L and consider the following modification of the NNAG method

with the same parameter choices as in (32) . Then, the following convergence bounds hold:

 with E [ ε (0)] = 1 2 ‖ v 0 -x ∗ ‖ 2 . In addition, for α = 3 / 4 we have

Remark 5.2.1 . The algorithm in (35) reduces to (23) when e k = 0 , s k = √ s = 1 / √ L, t k = k √ s . Remark 5.2.2 (Connection to NAG) . When there is no noise ( σ = 0 ) we can also allow parameter α to be zero. This is because we do not need to alleviate the effect of the noise with a decreasing stepsize. Therefore, we recover the convergence rate of O (1 /k 2 ) for the NAG method when c = 1 / √ L . Remark 5.2.3 (Comparison with [Laborde and Oberman, 2020]) . Laborde et al., proposed a stochastic method with noisy gradients. Their method uses another presentation of the NAG algorithm (the presentation from EE discretization). Our rate (36) has the same order of convergence as [Laborde and Oberman, 2020]. However, their analysis did not achieve the bound (37) (see [Laborde and Oberman, 2020] Appendix C.4).

Remark 5.2.4 (Comparison between Theorems 5.1 and 5.2) . The rate in (34) is asymptotically similar to (36). However, the transient behavior of (34) is faster than both (36) and the rate in [Laborde and Oberman, 2020] when L is large (see Figure 1 top row). This is due to the tuning parameter β which is usually set to L or higher. This scenario (Large L ) often happens in practice, e.g. in training a two-layer Convolutional Neural Network (CNN) [Shi et al., 2022].

Remark 5.2.5 (Limitations and future directions) . One limitation of our theoretical analysis is that our convergence result for NNAG holds only for a large enough number of iterations k ≥ k 0 . However, in our numerical experiments we observed that the same bounds hold also for k ≤ k 0 , so we believe our result can be improved. Additionally, our proposed forces are currently defined only in Euclidean space, and we see potential for extending the framework to non-Euclidean spaces.

6 Numerical Results

In this section, we present our empirical results, divided into three parts: Theoretical upper bounds, binary classification with logistic regression, and classification with neural networks.

Upper Bounds. First, we compare the bounds in (34) and (36) with the Proposition 4.5 in [Laborde and Oberman, 2020]. The results are depicted in Figure 1. In practical scenarios where L is large [Shi et al., 2022] the bound (34) is lower than the other two for large enough iterations. This observation has motivated us to analyze the behavior of NNAG in practical scenarios such as binary classification and CNN training tasks.

10

0

10

1

10

2

10

3

10

4

10

5

10

-

1

10

0

10

1

10

2

10

3

10

4

10

5

Bound value for

L

= 10

3

10

0

10

1

10

2

10

3

10

4

10

5

10

-

1

10

0

10

1

10

2

10

3

10

4

Bound value for

L

= 10

2

Upper-bound (34)

Upper-bound (36),

α

= 3

/

4

Iteration

Iteration

Binary Classification. For this task we considered d = 1000 randomly generated samples of dimension n = 10 and labels. The problem can be written as

where y i and x i denote the i th label and sample. For comparison, we considered the perturbed gradient descent method with Gaussian noise and decreasing step-size s k = 1 / ( √ Lk 2 / 3 ) (Proposition 3.4 in [Laborde and Oberman, 2020]) together with accelerated noisy gradient descent (Per-FE-C) in [Laborde and Oberman, 2020]. For NNAG we used s k = 1 / ( √ Lk 3 / 4 ) and β = L/ 10 . All the perturbation was done using i.i.d. Gaussian noise with unit variance, and we conducted 100 MonteCarlo runs. The results are presented in Figure 2 right panel. As shown, NNAG outperforms all the other methods in this case.

In a related experiment, we combined NNAG with SGD and SVRG. For the SGD-mixing, we replaced the noisy gradients with SGD-like gradients, while for the SVRG-mixing we evaluated all the gradients at the beginning of each epoch (essentially like taking a snapshot in SVRG) and set t k = 0 . The step-sizes for SVRG (in accordance with the original implementation [Johnson and Zhang, 2013]), SGD, NNAG+SVRG, and NNAG+SGD were set as 1 / (10 L ) , 1 /L , c = 1 / √ L, β = L/ 10 ,

20

40

60

80

100

10

-

9

10

-

6

10

-

3

10

0

Epoch

Logistic loss

20

40

60

80

100

10

-

3

10

-

2

10

-

1

10

0

Epoch

Logistic loss

Perturbed Gradient Descent

Per-FE-C

SGD

SVRG

NNAG+SVRG

NNAG+SGD

and c = 1 / √ L, β = L , respectively. We conducted 100 Monte-Carlo runs, and the results are displayed in Figure 2 left panel. Notably, when mixed with either SGD or SVRG, NNAG outperforms the original methods. This highlights NNAG's flexibility in terms of gradient noise and demonstrates its potential to accelerate various methods.

Classification on CIFAR10. Finally, we tackled the non-convex optimization problem of training a CNN with CIFAR10 dataset [Krizhevsky et al., 2009] using the SGD, SVRG, NNAG, and NNAG+SVRG methods. The network consisted of two convolutional layers each followed by max pooling and 3 fully connected linear layers each followed by ReLU activation function. The stepsizes for SGD and SVRG were set as 0.01, and for the NNAG and NNAG+SVRG algorithms we had c = 0 . 05 , β = 150 2 and c = 0 . 001 , β = 100 2 / 10 , respectively. The division by 10 is due to stepsize division by 10 in the SVRG method. The results for 20 Monte-Carlo simulations are depicted in Figure 3. Notably, SVRG+NNAG outperforms the other methods in terms of minimizing training error. Additionally, NNAG exhibits slightly better validation accuracy, hinting at its convergence toward a different local solution.

It is worth noting that faster convergence rates of NNAG+SVRG do not pose problems of overfitting. If necessary, one can terminate the algorithm earlier to achieve optimal performance. In Figure 3, for instance, NNAG+SVRG reaches its peak "validation accuracy" after approximately 20 epochs, with a validation accuracy of roughly 0.6 and a training error of around 0.66. After this point, a slow overfitting phase begins. Similarly, for SGD and SVRG, their peak "validation accuracy" is achieved after about 50 epochs, with a validation accuracy of approximately 0.6 and a training error of about 0.66, followed by a slow overfitting phase. Finally, NNAG achieves comparable results after approximately 100 epochs.

0

20

40

60

80

100

0

0

.

5

1

1

.

5

2

2

.

5

Epoch

Training Error

0

20

40

60

80

100

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

Epoch

Validation Accuracy

SGD

SVRG

NNAG+SVRG

7 Related Work

Polyak's Heavy-Ball (HB) method was one of the first momentum-based methods which could accelerate relative to the gradient descent method [Polyak, 1963]. However, this was not the case for every smooth function [Lessard et al., 2016]. Nesterov modified the HB and introduced the NAG method. This method achieved global convergence with a rate of O (1 /k 2 ) for smooth convex functions [Nesterov, 1983]. Nesterov used estimate sequence technique to show the convergence of the NAG method. This technique does not provide immediate insights toward the success of the NAG algorithm in acceleration. Thus, many have tried to understand the essence of acceleration.

On a similar line of work, Wibisono et al. [2016] introduced a variational perspective on accelerated methods, leading to a general (non-Euclidean) ODE containing the ODE found by Su et al. [2016] as a special case. Their approach is based on the choice of a Lagrangian and its corresponding parameters. Since the choice of Lagrangian is not unique, Wilson et al. [2021] provided a variational perspective on different accelerated first-order methods using another Lagrangian. Fazlyab et al. [2017] developed a family of accelerated dual algorithms for constrained convex minimization through a similar variational approach. In a more recent development, Zhang et al. [2021] showed that the second-variation also plays an important role in optimality of the ODE found by Su et al. [2016]. Specifically, they showed that if the time duration is long enough, then the mentioned ODE for the NAG algorithm is the saddle point to the problem of minimizing the action functional.

The dynamical system perspective on NAG was studied in [Muehlebach and Jordan, 2019]. They showed that the NAG algorithm is recovered from the SIE discretization of an ODE. The mentioned ODE was not the result of a vanishing step-size argument. They found that a curvature-dependent damping term accounts for the acceleration phenomenon. Interestingly, [Chen et al., 2022] also used similar ODE without the SIE discretization. They showed that implicit-velocity is the reason of the acceleration. In a recent analysis, [Muehlebach and Jordan, 2023] explores the connections between non-smooth dynamical systems and first-order methods for constrained optimization.

8 Conclusion

In this work, we tackled the problem of unconstrained smooth convex minimization in Euclidean space. Through a variational analysis of HR-ODEs, we achieved improved convergence rates for NAG in terms of gradient norm. In addition, we showed that NAG can be viewed as an approximation of the rate-matching technique when applied on a specific ODE. Our analysis was then extended to stochastic scenarios. In particular, we proposed a method with both constant and varying stepsizes which performed comparable and sometimes better than state of the art methods.

This work entails multiple future directions. Nesterov's oracle complexity lower bound on gradient norm minimization is O ( k -4 ) [Nesterov, 2003]. It remains an open question to see if the NAG method can achieve this rate of convergence for gradient norm minimization. In this work, we noticed that the HR-ODEs follow the same external force structure. In the smooth-strongly convex case, Triple Momentum (TM) method is the fastest known globally convergent method [Van Scoy et al., 2018]. However, the HR-ODE associated with the TM method is not shown to achieve the similar convergence rate as the TM method [Sun et al., 2020]. One could use the external force structure proposed here to find a better convergence rate for the HR-ODE associated with the TM algorithm. In addition, our analysis was confined to the Euclidean space. We believe it is possible to explore nonEuclidean forces using a Bregman Lagrangian as in [Wibisono et al., 2016]. Finally, we blended our noisy stochastic scheme with other known stochastic methods ( e.g. SGDandSVRG). This technique improved the performance of those methods. As a future work, one can apply the same technique to other practical methods like ADAM, RMSprop, etc, and study the behavior of the final algorithm.

References

Ahn and S. Sra. Understanding nesterov's acceleration via proximal point method. In Symposium on Simplicity in Algorithms (SOSA) , pages 117-130. SIAM, 2022.

Attouch, Z. Chbani, J. Fadili, and H. Riahi. First-order optimization algorithms via inertial systems with hessian driven damping. Mathematical Programming , pages 1-43, 2020.

Attouch, Z. Chbani, J. Fadili, and H. Riahi. Convergence of iterates for first-order optimization algorithms with inertia and hessian driven damping. Optimization , pages 1-40, 2021.

Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT'2010: 19th International Conference on Computational StatisticsParis France, August 2227, 2010 Keynote, Invited and Contributed Papers , pages 177-186. Springer, 2010.

M. Campos, A. Mahillo, and D. M. de Diego. A discrete variational derivation of accelerated methods in optimization. arXiv preprint arXiv:2106.02700 , 2021.

Chen, B. Shi, and Y.-x. Yuan. Gradient norm minimization of nesterov acceleration: o (1 /k 3 ) . arXiv preprint arXiv:2209.08862 , 2022.

Defazio, F. Bach, and S. Lacoste-Julien. Saga: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in neural information processing systems , 27, 2014.

Fazlyab, A. Koppel, V. M. Preciado, and A. Ribeiro. A variational approach to dual methods for constrained convex optimization. In 2017 American Control Conference (ACC) , pages 52695275, 2017. doi: 10.23919/ACC.2017.7963773.

Fazlyab, A. Ribeiro, M. Morari, and V. M. Preciado. Analysis of optimization algorithms via integral quadratic constraints: Nonstrongly convex problems. SIAM Journal on Optimization , 28(3):2654-2689, 2018. doi: 10.1137/17M1136845. URL https://doi.org/10.1137/17M1136845 .

Hu and L. Lessard. Dissipativity theory for nesterov's accelerated method. In International Conference on Machine Learning , pages 1549-1557. PMLR, 2017.

Johnson and T. Zhang. Accelerating stochastic gradient descent using predictive variance reduction. Advances in neural information processing systems , 26, 2013.

K. Khalil. Nonlinear systems third edition. Patience Hall , 115, 2002.

Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images . Toronto, ON, Canada, 2009.

Laborde and A. Oberman. A lyapunov analysis for accelerated gradient methods: from deterministic to stochastic case. In S. Chiappa and R. Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , volume 108 of Proceedings of Machine Learning Research , pages 602-612. PMLR, 26-28 Aug 2020. URL https://proceedings.mlr.press/v108/laborde20a.html .

Lessard, B. Recht, and A. Packard. Analysis and design of optimization algorithms via integral quadratic constraints. SIAM J. Optim. , 26:57-95, 2016.

Muehlebach and M. Jordan. A dynamical systems perspective on nesterov acceleration. In International Conference on Machine Learning , pages 4656-4662. PMLR, 2019.

Muehlebach and M. I. Jordan. On constraints in first-order optimization: A view from nonsmooth dynamical systems. Journal of Machine Learning Research , 23(256):1-47, 2022.

Muehlebach and M. I. Jordan. Accelerated first-order optimization under nonlinear constraints. arXiv preprint arXiv:2302.00316 , 2023.

Nesterov. A method for solving the convex programming problem with convergence rate o (1 /k 2 ) . Proceedings of the USSR Academy of Sciences , 269:543-547, 1983.

Nesterov. Introductory lectures on convex optimization: A basic course , volume 87. Springer Science & Business Media, 2003.

Polyak. Gradient methods for the minimisation of functionals. Ussr Computational Mathematics and Mathematical Physics , 3:864-878, 1963.

M. Sanz Serna and K. C. Zygalakis. The connections between lyapunov functions for some optimization algorithms and differential equations. SIAM Journal on Numerical Analysis , 59(3): 1542-1565, 2021.

Schmidt, N. Le Roux, and F. Bach. Minimizing finite sums with the stochastic average gradient. Mathematical Programming , 162:83-112, 2017.

Shi, S. S. Du, W. J. Su, and M. I. Jordan. Acceleration via symplectic discretization of highresolution differential equations. arXiv preprint arXiv:1902.03694 , 2019.

Shi, S. S. Du, M. I. Jordan, and W. J. Su. Understanding the acceleration phenomenon via high-resolution differential equations. ArXiv , abs/1810.08907, 2021.

Shi, Y. Wang, H. Zhang, J. Z. Kolter, and C.-J. Hsieh. Efficiently computing local lipschitz constants of neural networks via bound propagation. Advances in Neural Information Processing Systems , 35:2350-2364, 2022.

W. Siegel. Accelerated first-order methods: Differential equations and lyapunov functions. arXiv preprint arXiv:1903.05671 , 2019.

Su, S. Boyd, and E. J. Candès. A differential equation for modeling nesterov's accelerated gradient method: Theory and insights. Journal of Machine Learning Research , 17(153):1-43, 2016. URL http://jmlr.org/papers/v17/15-084.html .

Sun, J. George, and S. S. Kia. High-resolution modeling of the fastest firstorder optimization method for strongly convex functions. In 59th IEEE Conference on Decision and Control, CDC 2020, Jeju Island, South Korea, December 14-18, 2020 , pages 4237-4242. IEEE, 2020. doi: 10.1109/CDC42340.2020.9304444. URL https://doi.org/10.1109/CDC42340.2020.9304444 .

Van Scoy, R. A. Freeman, and K. M. Lynch. The fastest known globally convergent first-order method for minimizing strongly convex functions. IEEE Control Systems Letters , 2(1):49-54, 2018. doi: 10.1109/LCSYS.2017.2722406.

Wibisono, A. C. Wilson, and M. I. Jordan. A variational perspective on accelerated methods in optimization. Proceedings of the National Academy of Sciences , 113 (47):E7351-E7358, 2016. ISSN 0027-8424. doi: 10.1073/pnas.1614734113. URL https://www.pnas.org/content/113/47/E7351 .

C. Wilson, B. Recht, and M. I. Jordan. A lyapunov analysis of accelerated methods in optimization. J. Mach. Learn. Res. , 22:113-1, 2021.

Zhang, A. Orvieto, and H. Daneshmand. Rethinking the variational interpretation of accelerated optimization methods. Advances in Neural Information Processing Systems , 34:14396-14406, 2021.

A. Zhu and L. Orecchia. Linear coupling: An ultimate unification of gradient and mirror descent. In Information Technology Convergence and Services , 2014.

Peer reviews:
comments: SUMMARY:
This paper introduces forced Lagrangians in order to better understand discrete schemes for accelerate convex numerical optimization from a continuous-time ODE perspective.

STRENGTHS:
Two choices of F enable to re-derive continuous-time ODEs which were introduced recently in the literature (Section 2). Likewise, accelerated discrete schemes derive subsequently (Sections 3, 4) recover known schemes or in slightly modified form with slightly superior convergence rates. Empirically, it is demonstrated that even for non-convex network training, applying stochastic versions of accelerated schemes may perform well.

This paper significantly contributes to our understanding what unconstrained convex optimization with optimal convergence rates really means.

WEAKNESSES:
Authors use their jargon right from the beginning which makes the paper hard to read for readers who do not work on similar topics. For example, what “LR-ODE”, “HR-ODE” and the “rate matching technique” means should be briefly explained to a broader educated readership in the introduction.

QUESTIONS:
none

LIMITATIONS:
see "weaknesses" on how to improve the presentation
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work combines variational perspective approach with high-resolution ODE functions to investigate the Nesterov accelerated gradient descent algorithm (NAG). With variational perspective, the authors reconstruct various high-resolution ODEs derived in previous research using alternative methods. Moreover, through this approach, they propose a special representation of NAG that exhibits an improved convergence rate in terms of gradient norm minimization. The authors also discuss some new properties of rate-matching technique. Finally, the authors analyze the stochastic setting both theoretically and empirically.\n\nSTRENGTHS:\n1.\tThe idea of combining variational perspective and high-resolution ODEs by including external forces is very interesting. \n2.\tThe authors show several important theoretical results in their manuscript. By carefully checking their proofs of theorems (except for Section 5 because of time), I think overall these results are correct. \n3.\tThe numerical results indicate the potential of having better optimization algorithms based on the theoretical results in this manuscript. \n4.\tOverall, the manuscript is well written. \n\nOverall, I have a positive impression of this work, however, I also admit that I am not an expert in this specific field, which may affect my confidence in assessing its accuracy and significance.\n\nWEAKNESSES:\n1.\tIt would be better to have a more detailed introduction to low-resolution ODEs and high-resolution ODEs. Without reading some previous work, it is difficult to understand the differences between low-resolution ODEs and high-resolution ODEs. Are high-resolution ODEs the ODE functions that contain the learning rate $s$? Why do people need to care about high-resolution ODEs? \n2.\tI feel that the proof of Proposition 4.1 is more of an intuition rather than a rigorous demonstration. The treatment of the condition $s\\to 0$ seems imprecise, as sometimes the authors will directly consider this condition as $s=0$ (e.g. line 60), while at other times the authors maintain $s$ to be a non-zero value (e.g. line 59). The use of the word “approximately” in Prop 4.1 is also vague. \n3.\tSome typos/ unclear parts in the proofs that the authors may need to double-check. \n\n(a) line 84, $\\frac{\\partial L}{\\partial X}(X_t,...)$ should be $\\frac{\\partial L}{\\partial X_t}(X_t,...)$.\n\n(b) equation (7), left hand side, $\\bigtriangledown f$ should be $\\bigtriangledown f(X_t)$.\n\n(c) Theorem 2.1, $\\dot{\\gamma}=e^{\\alpha t}$ should be $\\dot{\\gamma}=e^{\\alpha_t}$. \n\n(d) line 138, ODE(14) should be ODE(12). \n\n(e) equation (25) $(3/t+\\sqrt{s}\\bigtriangledown f(X_t))$ should be $(3/t+\\sqrt{s}\\bigtriangledown^2 f(X_t))$.\n\n(f) line 196, $\\sigma$ not introduced, is it the variance of the noise? \n\n(g) equation (38) (appendix), second line $\\sqrt(s)e^{-\\alpha_t}\\ddot{\\beta_t}$ should be $\\sqrt(s)e^{-2\\alpha_t}\\ddot{\\beta_t}$. \n\n(h) line 392, (7) should be (11). \n\n(i) line 406, first equality $+\\frac{1}{2}||v_k-x^*||^2$ should be $-\\frac{1}{2}||v_k-x^*||^2$. \n\n(j) line 408, the term $+\\frac{s^2(k+2)}{4}||\\bigtriangledown f(x_{k+1})||^2$ in the second inequality is left out, therefore the authors need to double-check whether the results still hold after considering this term. \n\n(k) line 411, the first equaltiy redundant, \n\n(l) line 413 \"not ethat\" typo\n\n(m) line 421, the term $\\frac{ks}{2}(\\bigtriangledown f(x_k-\\bigtriangledown f(x_k))$ miss \")\". \n\n(n) equation (62), $(3/t+\\sqrt{s}\\bigtriangledown f(X_t))$ should be $(3/t+\\sqrt{s}\\bigtriangledown^2 f(X_t))$.\n\n4. Figure 1 is hard to read. (font size too small)\n5. It would be better to describe what \"NAG\" means (Nesterov accelerated gradient?) the first time this abbreviation is used.\n\nQUESTIONS:\n1.\tIt is unclear to me that equation (21) and equation (27) (by replacing one term) is equal to NAG (line 76), why is that? \n2.\tIn line 211, the authors say that practically $k_0$ is lower than the term $(\\cdot)^{1/\\alpha}$. However, one of the conditions in Theorem 5.1 and Theorem 5.2 is that $k_0\\geq (\\cdot)^{1/\\alpha}$, does that mean this condition will not be satisfied in practice?\n\nLIMITATIONS:\nThe authors don't specifically discuss the limitations of this work. The authors may consider adding a paragraph in their manuscript to discuss the limitations of their work based on the reviewer's feedback. \n\nI don't think there will be a significant negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThey generalize the Lagrangian formulation of known first order optimization methods [Wibisono et al., 2016, Wilson et al., 2021] by introducing the notion of external forces, and show theorems on convergence of convex/strongly convex functions. \n\nIn addition, they show the variational analysis leads a special representation of NAG, and it enjoys to superior convergence rates than [Shi et al., 2019]. Based on the special representation of NAG, they generalize it to stochastic variation in Section 5, called NNAG, and show its convergent theorem. \n\nThey check the practical behavior of the proposed NNAG by using \n- binary classification,\n- classiﬁcation on CIFAR10.\n\nSTRENGTHS:\noriginality\n- Introducing the external force term to the Lagrangian formulation and show convergent theorems with it. \n\nquality/clarity\n- They show their idea and mathematical statements clearly, and show their proofs.\n\nsignificance\n- Their formulation can lead known dynamics also. In this sense, their proposal can be regarded as a unification of various optimization dynamics.\n\nWEAKNESSES:\n- The first part of the paper sounds natural, but to me, there seems no theoretical reason to introduce i.i.d. noise to the gradients in section 5.\n- In my opinion, they should conduct more numerical experiments. For example, the experiments in classification on CIFAR10, NNAG and SVRG+NNAG are competitive to SGD. This result, itself is good, but it means there is no big incentive to use the proposed algorithm, but SGD.\n\nQUESTIONS:\n- Is there any natural/theoretical motivation to introduce i.i.d. noise to the gradients in section 5?\n- Is there any strong incentive to use NNAG or its variants compared to SGD in practice?\n- Besides it, the authors wrote `As the figure depicts, the SVRG+NNAG performs faster than the other methods in terms of minimizing the training error.`, but minimizing training error itself sounds possibility of overfitting in machine learning context. If not, please correct me.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper did four works about High-Resolution ODEs and first-order optimization algorithms. The first part uses forced Euler-Lagrange equations to generalize the analysis of Low-Resolution ODEs to High-Resolution ODEs. The second part is a refined result for bound estimation of gradient norm. The third part is an interpretation of Nesterov’s acceleration by rate-matching discretization. In the last part, the authors propose a stochastic version of Nesterov’s accelerated gradient method, named NNAG in this paper, and compared it to several known stochastic methods on binary classification and training CNN.\n\nSTRENGTHS:\nThis paper proposes a new idea of external force and uses the idea of forced Euler-Lagrange equations to generalize the analysis of Low-Resolution ODEs to High-Resolution ODEs. This leads to a new formulation of NAG. Combined with rate-match discretization, the authors discover a new connection of NAG and the continuous ODE, i.e., it is understood as a perturbation of the Low-Resolution ODE. The new formulation of NAG further inspires the framework of Noisy NAG. The idea of forced Euler-Lagrange equations may bring new insight to fast algorithm design.\n\nWEAKNESSES:\nAlthough this paper provides a new reformulation of NAG, as the author suggested, the refined convergence rate of gradient norm for NAG is already appeared in [S. Chen, B. Shi, and Y.-x. Yuan. Gradient norm minimization of nesterov acceleration: o(1/k^3). arXiv preprint arXiv:2209.08862, 2022]. The Lyapunov function is essentially the same using the implicit-velocity form and the explicit-velocity form in [B. Shi, S. S. Du, M. I. Jordan, and W. J. Su. Understanding the acceleration phenomenon via high-resolution differential equations. ArXiv, abs/1810.08907, 2021].\n\nQUESTIONS:\nThe Lagrange function is kinetic energy minus potential energy (though multiplied by some decreasing coefficient). The Lagrange function and external force is considered separately in this paper. Is it possible to give physical explanation about the acceleration by forced Euler-Lagrange equation and use different energy function and external force to inspire better algorithm design?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations and have indicated problems for further study. This work appears to have little negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-08-01"}