SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hoomaan Maskan
hoomaan-maskan@umu.se
Konstantinos C. Zygalakis
University of Edinburgh k.zygalakis@ed.ac.uk
Alp Yurtsever
Umeå University alp.yurtsever@umu.se
We consider unconstrained minimization of smooth convex functions. We propose a novel variational perspective using forced Euler-Lagrange equation that allows for studying high-resolution ODEs. Through this, we obtain a faster convergence rate for gradient norm minimization using Nesterov's accelerated gradient method. Additionally, we show that Nesterov's method can be interpreted as a rate-matching discretization of an appropriately chosen high-resolution ODE. Finally, using the results from the new variational perspective, we propose a stochastic method for noisy gradients. Several numerical experiments compare and illustrate our stochastic algorithm with state of the art methods.
Smooth convex minimization is a fundamental class in optimization with broad applications and a rich theoretical foundation that enables the development of powerful methods. In fact, the theory and methods developed for other problem classes, such as non-smooth convex or smooth non-convex optimization, often build upon the work done in smooth convex optimization. As a result, the study of smooth convex minimization has a significant impact on the broader field of optimization. Over the last two decades, first-order methods –those relying solely on gradient information, in contrast to methods like Newton's that require Hessian information– have seen a lot of interest both in theory and applications due to their efficiency and adaptability for large-scale data-driven applications. Among these, gradient descent stands as one of the simplest and oldest. When equipped with an appropriate step-size, gradient descent guarantees a suboptimality gap (objective residual) of order  $\mathcal{O}(1 / k)$  after  $k$  iterations.
In his seminal work, Nesterov [1983] has shown that the gradient method can achieve faster rates by incorporating momentum deviations. Nesterov's accelerated gradient algorithm (NAG) ensures a convergence rate of  $\mathcal{O}(1 / k^2)$  which is an order of magnitude faster than the gradient descent. Remarkably, this rate matches information-theoretical lower bounds for first-order oracle complexity, meaning that NAG is optimal and no other first-order method can guarantee an essentially faster convergence rate [Nesterov, 2003].
The original proof of Nesterov [1983], known as the estimate sequence technique, is a highly algebraic and complex procedure, difficult to interpret, and provides arguably limited insight into why momentum deviations help with the convergence rates [Hu and Lessard, 2017]. Therefore, many researchers have tried to provide a better understanding of momentum-based acceleration through different perspectives. For example, Su et al. [2016], Shi et al. [2021], and Sanz Serna and Zygalakis [2021] consider a continuous time perspective; Lessard et al. [2016] and Fazlyab et al. [2018] use integral quadratic constraints and control systems with non-linear feedbacks; Muehlebach and Jordan [2019, 2022, 2023] present a dynamical perspective; Attouch et al. [2020, 2021] utilize inertial dynamic involving both viscous damping and Hessian-driven damping; Zhu and Orecchia [2014] view acceleration as a linear coupling of gradient descent and mirror descent updates; and Ahn and Sra [2022] provide an understanding of the NAG algorithm through an approximation of the proximal point method. Our work is focused on the continuous-time perspective.
In [Su et al., 2016], the authors derive a second-order ordinary differential equation (ODE) that exhibits trajectories similar to those of the NAG algorithm in the limit of an infinitesimal step-size  $s \to 0$ . This result has inspired researchers to analyze various ODEs and discretization schemes to gain a better understanding of the acceleration phenomenon. Notably, Wibisono et al. [2016] demonstrate that the ODE presented in [Su et al., 2016] is a special case of a broader family of ODEs that extend beyond Euclidean space. They achieve this by minimizing the action on a Lagrangian that captures the properties of the problem template, an approach known as the variational perspective, and they discretize their general ODE using the rate-matching technique. In a related vein, Shi et al. [2021] proposed substituting the low-resolution ODEs (LR-ODEs) introduced in [Su et al., 2016] with high-resolution ODEs (HR-ODEs) which can capture trajectories of NAG more precisely.
Our main contribution in this paper is a novel extension of the variational perspective for HR-ODEs. A direct combination of these two frameworks is challenging, as it remains unclear how the Lagrangian should be modified to recover HR-ODEs. To address this problem, we propose an alternative approach that preserves the Lagrangian but extends the variational perspective. More specifically, instead of relying on the conventional Euler-Lagrange equation, we leverage the forced Euler-Lagrange equation that incorporates external forces acting on the system. By representing the damped time derivative of the potential function gradients as an external force, our proposed variational perspective allows us to reconstruct various HR-ODEs through specific damping parameters. More details are provided in Section 2.
Other contributions of our paper are as follows: In Section 3, we show that our proposed variational analysis yields a special representation of NAG leading to superior convergence rates than [Shi et al., 2019] in terms of gradient norm minimization. In Section 4, we propose an HR-ODE based on the rate-matching technique. We demonstrate that NAG can be interpreted as an approximation of the rate-matching technique applied to a specific ODE. Then, in Section 5, we extend our analysis to a stochastic setting with noisy gradients, introducing a stochastic method that guarantees convergence rates of  $\tilde{\mathcal{O}} (1 / k^{1 / 2})$  for the expected objective residual and  $\tilde{\mathcal{O}} (1 / k^{3 / 4})$  for the expected squared norm of the gradient. Finally, in Section 6, we present numerical experiments to demonstrate the empirical performance of our proposed method and to validate our theoretical findings.
Problem template and notation. We consider a generic unconstrained smooth convex minimization template:
$$
\min  _ {x \in \mathbb {R} ^ {n}} f (x) \tag {1}
$$
where  $f: \mathbb{R}^n \to \mathbb{R}$  is convex and  $L$ -smooth, meaning that its gradient is Lipschitz continuous:
$$
\| \nabla f (x) - \nabla f (y) \| \leq L \| x - y \|, \quad \forall x, y \in \mathbb {R} ^ {n}, \tag {2}
$$
with  $\| \cdot \|$  denoting the Euclidean norm. We denote the class of  $L$ -smooth convex functions by  $\mathcal{F}_L$ .
Throughout, we assume that the solution set for Problem (1) is non-empty, and we denote an arbitrary solution by  $x^{*}$ , hence  $f^{*} \coloneqq f(x^{*}) \leq f(x)$  for all  $x \in \mathbb{R}^n$ .
The NAG Algorithm. Given an initial state  $x_0 = y_0 \in \mathbb{R}^n$  and a step-size parameter  $s > 0$ , the NAG algorithm updates the variables  $x_k$  and  $y_k$  iteratively as follows:
$$
y _ {k + 1} = x _ {k} - s \nabla f (x _ {k}),
$$
$$
x _ {k + 1} = y _ {k + 1} + \frac {k}{k + 3} \left(y _ {k + 1} - y _ {k}\right). \tag {NAG}
$$
Low-Resolution and High-Resolution ODEs. Throughout, by LR-ODE, we refer to the second-order ODE introduced and studied in [Su et al., 2016], which exhibits a behavior reminiscent of momentum-based accelerated methods like Heavy Ball (HB) and NAG in the limit of step-size  $s \to 0$ . Unfortunately, this generic ODE cannot distinguish the differences between HB and NAG. It is important to note, however, that the guarantees of HB and NAG differ significantly in discrete time. Specifically, HB has guarantees for only a specific subset of Problem (1), whereas NAG guarantees a solution for all instances of (1) with appropriate step sizes. Therefore, it's worth noting that the LR-ODE may not fully capture certain important aspects of the NAG trajectory. Conversely, by HR-ODEs, we refer to the ODEs proposed in [Shi et al., 2021], which extend the LR-ODE by incorporating gradient correction terms. Through this extension, HR-ODEs provide a more precise representation of the trajectory for these distinct algorithms.
2 External Forces and High-Resolution ODEs
Consider the Lagrangian
$$
\mathcal {L} \left(X _ {t}, \dot {X} _ {t}, t\right) = e ^ {\alpha_ {t} + \gamma_ {t}} \left(\frac {1}{2} \| e ^ {- \alpha_ {t}} \dot {X} _ {t} \| ^ {2} - e ^ {\beta_ {t}} f \left(X _ {t}\right)\right). \tag {3}
$$
Here,  $\dot{X}_t \in \mathbb{R}^d$  is the first time-derivative of  $X(t)$ , and  $\alpha_t, \beta_t, \gamma_t : \mathbb{T} \to \mathbb{R}$  are continuously differentiable functions of time that correspond to the weighting of velocity, the potential function  $f$ , and the overall damping, respectively. Using variational calculus, we define the action for the curves  $\{X_t : t \in \mathbb{R}\}$  as the functional  $\mathcal{A}(X) = \int_{\mathbb{R}} \mathcal{L}(X_t, \dot{X}_t, t) dt$ . In the absence of external forces, a curve is a stationary point for the problem of minimizing the action  $\mathcal{A}(X)$  if and only if it satisfies the Euler-Lagrange equation  $\frac{d}{dt} \left\{ \frac{\partial \mathcal{L}}{\partial \dot{X}_t}(X_t, \dot{X}_t, t) \right\} = \frac{\partial \mathcal{L}}{\partial X_t}(X_t, \dot{X}_t, t)$ . This was used in [Wibisono et al., 2016, Wilson et al., 2021] to calculate the LR-ODEs for convex and strongly convex functions<sup>1</sup>. Note that the Euler-Lagrange equation as written, does not account for an external force,  $F$  (which is non-conservative). In this case, the Euler-Lagrange equation should be modified to the forced Euler-Lagrange Equation
$$
\frac {d}{d t} \left\{\frac {\partial \mathcal {L}}{\partial \dot {X} _ {t}} \left(X _ {t}, \dot {X} _ {t}, t\right) \right\} - \frac {\partial \mathcal {L}}{\partial X _ {t}} \left(X _ {t}, \dot {X} _ {t}, t\right) = F, \tag {4}
$$
which itself is the result of integration by parts of Lagrange d'Alembert principle [Campos et al., 2021]. Using the Lagrangian (3) we have
$$
\frac {\partial \mathcal {L}}{\partial \dot {X} _ {t}} \left(X _ {t}, \dot {X} _ {t}, t\right) = e ^ {\gamma_ {t}} \left(e ^ {- \alpha_ {t}} \dot {X} _ {t}\right), \quad \frac {\partial \mathcal {L}}{\partial X _ {t}} \left(X _ {t}, \dot {X} _ {t}, t\right) = - e ^ {\gamma_ {t} + \alpha_ {t} + \beta_ {t}} \left(\nabla f \left(X _ {t}\right)\right). \tag {5}
$$
Substituting (5) in (4) gives
$$
\ddot {X} _ {t} + \left(\dot {\gamma} _ {t} - \dot {\alpha} _ {t}\right) \dot {X} _ {t} + e ^ {2 \alpha_ {t} + \beta_ {t}} \nabla f (X _ {t}) = e ^ {\alpha_ {t} - \gamma_ {t}} F. \tag {6}
$$
In what follows, we will present various choices of the external force  $F$ , including two for convex functions and one for strongly convex functions.
2.1 Convex Functions
First choice for convex functions. Let us first consider the following external force:
$$
F = - \sqrt {s} e ^ {\gamma_ {t}} \frac {d}{d t} \left[ e ^ {- \alpha_ {t}} \nabla f (X) \right]. \tag {7}
$$
In this case, (6) becomes
$$
\ddot {X} _ {t} + \left(\dot {\gamma} _ {t} - \dot {\alpha} _ {t}\right) \dot {X} _ {t} + e ^ {2 \alpha_ {t} + \beta_ {t}} \nabla f (X _ {t}) = - \sqrt {s} e ^ {\alpha_ {t}} \frac {d}{d t} \left[ e ^ {- \alpha_ {t}} \nabla f (X _ {t}) \right]. \tag {8}
$$
It is possible to show the convergence of  $X_{t}$  to  $x^{*}$  and establish a convergence rate for this as shown in the following theorem (proof in Appendix A.1). The proof of this theorem (and the subsequent theorems in this section) is based on the construction of a suitable Lyapunov function for the corresponding ODE (e.g. see [Siegel, 2019, Shi et al., 2019, Attouch et al., 2020, 2021]). This non-negative function attains zero only at the stationary solution of the corresponding ODE and decreases along the trajectory of the ODE [Khalil, 2002]. For this theorem (and the subsequent theorems), we will define a proper Lyapunov function and prove sufficient decrease of the function  $f$  along the corresponding ODE trajectory.
Theorem 2.1. Under the ideal scaling conditions  $\dot{\beta}_t\leq e^{\alpha_t}$  and  $\dot{\gamma}_t = e^{\alpha_t}$ ,  $X_{t}$  in (8) will satisfy
$$
f (X _ {t}) - f (x ^ {*}) \leq \mathcal {O} \left(e ^ {- \beta_ {t}}\right)
$$
for  $f\in \mathcal{F}_L$
<sup>1</sup>Wilson et al. [2021] use different Lagrangian for strongly convex functions, but the methodology is the same.
Now, choosing parameters as
$$
\alpha_ {t} = \log (n (t)), \quad \beta_ {t} = \log (q (t) / n (t)), \quad \dot {\gamma} _ {t} = e ^ {\alpha_ {t}} = n (t), \tag {9}
$$
in (8) gives
$$
\left\{ \begin{array}{l} \ddot {X} _ {t} + (n (t) - \frac {\dot {n} (t)}{n (t)} + \sqrt {s} \nabla^ {2} f (X _ {t})) \dot {X} _ {t} + (n (t) q (t) - \sqrt {s} \frac {\dot {n} (t)}{n (t)}) \nabla f (X _ {t}) = 0, \\ F = - \sqrt {s} e ^ {\gamma_ {t}} \frac {d}{d t} [ e ^ {- \alpha_ {t}} \nabla f (X) ], \end{array} \right. \tag {10}
$$
which reduces to
$$
\ddot {X} _ {t} + \left(\frac {p + 1}{t} + \sqrt {s} \nabla^ {2} f (X _ {t})\right) \dot {X} _ {t} + \left(C p ^ {2} t ^ {p - 2} + \frac {\sqrt {s}}{t}\right) \nabla f (X _ {t}) = 0, \tag {11}
$$
by taking  $n(t) = \frac{p}{t}, q(t) = Cpt^{p - 1}$ .
Remark 2.1.1. For  $p = 2, C = 1/4$ , equation (11) corresponds to the (H-ODE) in [Laborde and Oberman, 2020].
Second choice for convex functions. Now, we consider an external force given by
$$
F = - \sqrt {s} e ^ {\gamma_ {t} - \beta_ {t}} \frac {d}{d t} \left[ e ^ {- (\alpha_ {t} - \beta_ {t})} \nabla f (X _ {t}) \right] \tag {12}
$$
In this case, replacing  $F$  in (6) gives
$$
\ddot {X} _ {t} + \left(\dot {\gamma} _ {t} - \dot {\alpha} _ {t}\right) \dot {X} _ {t} + e ^ {2 \alpha_ {t} + \beta_ {t}} \nabla f = - \sqrt {s} e ^ {\alpha_ {t} - \beta_ {t}} \frac {d}{d t} \left[ e ^ {- \left(\alpha_ {t} - \beta_ {t}\right)} \nabla f \left(X _ {t}\right) \right]. \tag {13}
$$
We establish the following convergence result, and the proof can be found in Appendix A.2.
Theorem 2.2. Under the modified ideal scaling conditions  $\dot{\beta}_t \leq e^{\alpha_t}$ ,  $\dot{\gamma}_t = e^{\alpha t}$ ,  $\ddot{\beta}_t \leq e^{\alpha_t}\dot{\beta}_t + 2\dot{\alpha}_t\dot{\beta}_t$ ,  $X_t$  in (13) will satisfy
$$
f (X _ {t}) - f (x ^ {*}) \leq \mathcal {O} \left(\frac {1}{e ^ {\beta_ {t}} + \sqrt {s} e ^ {- 2 \alpha_ {t}} \dot {\beta} _ {t}}\right),
$$
Taking the same parameters as in (9) gives
$$
\left\{ \begin{array}{l} \ddot {X} _ {t} + (n (t) - \frac {\dot {n} (t)}{n (t)}) + \sqrt {s} \nabla^ {2} f (X _ {t})) \dot {X} _ {t} + (n (t) q (t) - \sqrt {s} \left(\frac {\dot {n} (t)}{n (t)} - \frac {\dot {q} (t) n (t) - \dot {n} (t) q (t)}{n (t) q (t)}\right)) \nabla f (X _ {t}) = 0, \\ F = - \sqrt {s} e ^ {\gamma_ {t} - \beta_ {t}} \frac {d}{d t} \left[ e ^ {- (\alpha_ {t} - \beta_ {t})} \nabla f (X _ {t}) \right]. \end{array} \right. \tag {14}
$$
$$
\ddot {X} _ {t} + \left(\frac {p + 1}{t} + \sqrt {s} \nabla^ {2} f (X _ {t})\right) \dot {X} _ {t} + \left(C p ^ {2} t ^ {p - 2} + \frac {\sqrt {s} (p + 1)}{t}\right) \nabla f (X _ {t}) = 0, \tag {15}
$$
for  $n(t) = p / t, q(t) = Cpt^{p - 1}$ .
Remark 2.2.1. Note that setting  $C = 1/4, p = 2$  will lead to the ODE
$$
\ddot {X} _ {t} + \left(\frac {3}{t} + \sqrt {s} \nabla^ {2} f (X _ {t})\right) \dot {X} _ {t} + \left(1 + \frac {3 \sqrt {s}}{t}\right) \nabla f (X _ {t}) = 0. \tag {16}
$$
This ODE was discretized using the Semi-Implicit Euler (SIE) and the Implicit Euler (IE) discretization schemes in [Shi et al., 2019]. The corresponding optimization algorithms were shown to accelerate. In addition, note that the convergence rate proved in Theorem 2.2 is faster than its counterpart in Theorem 2.1.
2.2 Strongly Convex Functions
Our analysis is applicable to strongly convex functions as well. Consider the Lagrangian proposed in [Wilson et al., 2021] for strongly convex functions
$$
\mathcal {L} \left(X _ {t}, \dot {X} _ {t}, t\right) = e ^ {\alpha_ {t} + \beta_ {t} + \gamma_ {t}} \left(\frac {\mu}{2} \| e ^ {- \alpha_ {t}} \dot {X} _ {t} \| ^ {2} - f \left(X _ {t}\right)\right). \tag {17}
$$
Then, the forced Euler-Lagrange equation (4) becomes
$$
\ddot {X} + \left(- \dot {\alpha} _ {t} + \dot {\gamma} _ {t} + \dot {\beta} _ {t}\right) \dot {X} + \frac {1}{\mu} e ^ {2 \alpha_ {t}} \nabla f (X) = \frac {F}{\mu e ^ {- \alpha_ {t} + \gamma_ {t} + \beta_ {t}}}. \tag {18}
$$
Taking  $F = -\sqrt{s} e^{\alpha_{t} + \gamma_{t}}\frac{d}{dt}\big(e^{\beta_{t}}\nabla f(X_{t})\big)$  in (18) gives
$$
\ddot {X} + \left(- \dot {\alpha} _ {t} + \dot {\gamma} _ {t} + \dot {\beta} _ {t}\right) \dot {X} + \frac {1}{\mu} e ^ {2 \alpha_ {t}} \nabla f (X) = \frac {- \sqrt {s} e ^ {2 \alpha_ {t} - \beta_ {t}} \frac {d}{d t} \left(e ^ {\beta_ {t}} \nabla f \left(X _ {t}\right)\right)}{\mu}. \tag {19}
$$
We can establish the following convergence result for  $X_{t}$  in (19) to the unique minimizer  $x^{*}$ . The proof of this result is deferred to Appendix A.3.
Theorem 2.3. Under the modified ideal scaling conditions  $\alpha_{t} = \alpha$ ,  $\dot{\beta}_{t} \leq e^{\alpha_{t}}$ ,  $\dot{\gamma}_{t} = e^{\alpha_{t}}$ , and  $\dot{\beta}_{t} \geq 0$ $X_{t}$  in (19) satisfies
$$
f \left(X _ {t}\right) - f \left(x ^ {*}\right) \leq \mathcal {O} \left(e ^ {- \beta_ {t}}\right) \tag {20}
$$
for  $\mu$ -strongly convex function  $f$ .
Remark 2.3.1. Taking  $\alpha = \log (\sqrt{\mu})$  and  $\gamma_{t} = \beta_{t} = \sqrt{\mu} t$  in (19) gives the NAG's corresponding HR-ODE
$$
\ddot {X} _ {t} + \left(2 \sqrt {\mu} + \sqrt {s} \nabla^ {2} f (X _ {t})\right) \dot {X} _ {t} + \left(1 + \sqrt {\mu s}\right) \nabla f (X _ {t}) = 0, \tag {21}
$$
for  $\mu$ -strongly convex function  $f$  as in [Shi et al., 2021].
3 Gradient Norm Minimization of NAG
One of the implications of our variational study on HR-ODEs in Section 2 was the ODE (14). Reformulating this ODE gives
$$
\left\{ \begin{array}{l} \dot {X} _ {t} = n (t) \left(V _ {t} - X _ {t}\right) - \sqrt {s} \nabla f \left(X _ {t}\right) \\ \dot {V} _ {t} = - q (t) \nabla f \left(X _ {t}\right) - \sqrt {s} \frac {\dot {q} (t) n (t) - \dot {n} (t) q (t)}{n ^ {2} (t) q (t)} \nabla f \left(X _ {t}\right). \end{array} \right. \tag {22}
$$
Applying the SIE on (22) for  $X(t) \approx X(t_k), V(t) \approx V(t_k), n(t_k) = p / t_k, q(t_k) = Cpt_k^{p - 1}, p = 2, t_k = k\sqrt{s}$  and  $C = 1 / 4$  gives
$$
\left\{ \begin{array}{l} x _ {k + 1} = x _ {k} + \frac {2}{k} \left(v _ {k} - x _ {k + 1}\right) - s \nabla f \left(x _ {k}\right), \\ v _ {k + 1} = v _ {k} - \frac {1}{2} (k s) \nabla f \left(x _ {k + 1}\right) - s \nabla f \left(x _ {k + 1}\right), \end{array} \right. \tag {23}
$$
which is exactly the NAG algorithm. The interpretation of the NAG method as the SIE discretization of (22) has not been discussed before in the literature (see [Ahn and Sra, 2022] for the four most studied representations). It is precisely this connection with the ODE (22) though that inspires our choice of the Lyapunov function which in turn gives rise to a faster convergence rate. The following theorem formulates this result. The proof is in Appendix A.4 and it is based on the discrete Lyapunov analysis of (23). Similar convergence rate was very recently found by [Chen et al., 2022] through implicit velocity perspective on HR-ODEs which uses a different Lyapunov analysis than this work.
Theorem 3.1. Consider the update (23). Then, if  $f \in \mathcal{F}_L$  we have
$$
\min  _ {0 \leq i \leq k - 1} \| \nabla f (x _ {i}) \| ^ {2} \leq \frac {1 2}{k ^ {3} s ^ {2}} \| x _ {0} - x ^ {*} \| ^ {2},
$$
$$
f (x _ {k}) - f \left(x ^ {*}\right) \leq \frac {2}{s k (k + 2)} \left\| x _ {0} - x ^ {*} \right\| ^ {2}
$$
for  $0 \leq s \leq 1 / L$ ,  $v_{0} = x_{0}$ , and any  $x_{0} \in \mathbb{R}^{n}$ .
Remark 3.1.1 (Comparison with state of the art). The rate in Theorem 3.1 is improved compared to the previous rate found in [Shi et al., 2021], which is
$$
\min  _ {0 \leq i \leq k} \| \nabla f (x _ {i}) \| ^ {2} \leq \frac {8 5 6 8}{(k + 1) ^ {3} s ^ {2}} \| x _ {0} - x ^ {*} \| ^ {2},
$$
for  $0 < s \leq 1/(3L)$  and  $k \geq 0$ .
4 Rate-Matching Approximates the NAG Algorithm
The ODE (11) when  $p = 2$ ,  $C = 1/4$  is equivalent to
$$
\left\{ \begin{array}{l} \dot {X} _ {t} = \frac {2}{t} \left(Z _ {t} - X _ {t}\right) - \sqrt {s} \nabla f \left(X _ {t}\right), \\ \dot {Z} _ {t} = - \frac {t}{2} \nabla f \left(X _ {t}\right). \end{array} \right. \tag {24}
$$
which can be viewed as a perturbation of the LR-ODE
$$
\left\{ \begin{array}{l} \dot {X} _ {t} = \frac {2}{t} \left(Z _ {t} - X _ {t}\right), \\ \dot {Z} _ {t} = - \frac {t}{2} \nabla f \left(X _ {t}\right). \end{array} \right. \tag {25}
$$
We now show that when the rate-matching technique in [Wibisono et al., 2016] is applied to (25), the final algorithm reveals similar behavior as (24). This result is then used to approximately recover the NAG method using rate-matching discretization.
Applying the rate-matching discretization on the ODE (25) gives
$$
\left\{ \begin{array}{l} x _ {k + 1} = \frac {2}{k + 2} z _ {k} + \frac {k}{k + 2} y _ {k}, \\ y _ {k} = x _ {k} - s \nabla f \left(x _ {k}\right), \\ z _ {k} = z _ {k - 1} - \frac {1}{2} s k \nabla f \left(y _ {k}\right). \end{array} \right. \tag {26}
$$
which has a convergence rate of  $\mathcal{O}(1 / (sk^2))$  [Wibisono et al., 2016]. In the following proposition, we study the behavior of (26) in limit of  $s\to 0$ . The proof is given in Appendix A.5.
Proposition 4.1. The continuous-time behavior of (26) is approximately
$$
\ddot {X} _ {t} + \left(\frac {3}{t} + \sqrt {s} \nabla^ {2} f (X _ {t})\right) \dot {X} _ {t} + \left(1 + \frac {\sqrt {s}}{t}\right) \nabla f (X _ {t}) = 0, \tag {27}
$$
which is the high-resolution ODE (11).
The ODE (27) is the same as (24). In this sense, rate-matching implicitly perturbs the LR-ODE. The question that naturally arises is that when do we recover the HR-ODE (16) (which corresponds to the NAG algorithm through the SIE discretization) from the rate-matching technique? To answer, we will first perturb the LR-ODE (25) in the second line. Then, the rate-matching discretization is applied. Perturbing (25) gives
$$
\left\{ \begin{array}{l} \dot {X} _ {t} = \frac {2}{t} \left(Z _ {t} - X _ {t}\right), \\ \dot {Z} _ {t} = - \frac {t}{2} \nabla f \left(X _ {t}\right) - \sqrt {s} \nabla f \left(X _ {t}\right). \end{array} \right. \tag {28}
$$
Discretizing (28) using the rate-matching method with  $t_k = k\sqrt{s}$  gives
$$
\left\{ \begin{array}{l} x _ {k + 1} = \frac {2}{k + 2} z _ {k} + \frac {k}{k + 2} y _ {k}, \\ y _ {k} = x _ {k} - s \nabla f \left(x _ {k}\right), \\ z _ {k} = z _ {k - 1} - \frac {s}{2} (k + 2) \nabla f \left(y _ {k}\right), \end{array} \right. \tag {29}
$$
which is extremely close to the NAG algorithm. Indeed, replacing  $\nabla f(y_k)$  with  $\nabla f(x_k)$  in the third line of (29) gives exactly the NAG method. Typically,  $x_{k}$  and  $y_{k}$  are very close. This is due to  $x_{k}$  and  $y_{k}$  having a difference of order  $s$ . Since in continuous time  $X(t_{k})\approx Y(t_{k})$  (due to  $s\to 0$ ), the HR-ODE of (29) is (16). This means that the corresponding HR-ODE of (29) is
$$
\left\{ \begin{array}{l} \dot {X} _ {t} = \frac {2}{t} \left(Z _ {t} - X _ {t}\right) - \sqrt {s} \nabla f \left(X _ {t}\right), \\ \dot {Z} _ {t} = - \frac {t}{2} \nabla f \left(X _ {t}\right) - \sqrt {s} \nabla f \left(X _ {t}\right). \end{array} \right. \tag {30}
$$
which is the perturbed version of (28) and the HR-ODE associated with the NAG algorithm.
5 Stochastic Extensions
In this section, we propose a stochastic variation of (23). We model noisy gradients by adding i.i.d noise  $e_k$  with variance  $\sigma^2$  to the gradients. Consider the update
$$
\left\{ \begin{array}{l} x _ {k + 1} = x _ {k} + \frac {2 s _ {k}}{t _ {k}} \left(v _ {k} - x _ {k + 1}\right) - \frac {\beta s _ {k}}{\sqrt {L}} \left(\nabla f \left(x _ {k}\right) + e _ {k}\right), \\ v _ {k + 1} = v _ {k} - \frac {1}{2} \left(t _ {k} s _ {k} + \frac {2 s _ {k} \beta}{\sqrt {L}}\right) \left(\nabla f \left(x _ {k + 1}\right) + e _ {k + 1}\right) \end{array} \right. \tag {31}
$$