SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We show that this kernel is positive semidefinite for  $\nu = \frac{\mu}{2}\operatorname{csch}_T^2$ . Proposition 1 (a) shows that the kernel  $(t,\tau)\mapsto \tanh_t\tanh_\tau$  is positive semidefinite. Proposition 1 (d) shows that the kernel  $(t,\tau)\mapsto \sinh_{\tau}^{2}$  is positive semidefinite because the function  $\tau \mapsto \sinh_{\tau}^{2}$  is a nonnegative increasing function. Since the PEP kernel (13) with  $\nu = \frac{\mu}{2}\operatorname{csch}_T^2$  can be expressed as a product of two positive semidefinite kernels, it is positive semidefinite by Proposition 1 (b). Consequently, Theorem 1 implies that Unified AGM ODE achieves the following convergence guarantee:
$$
\tilde {f} (X (T)) - \tilde {f} \left(x ^ {*}\right) \leq \frac {\mu}{2} \operatorname {c s c h} _ {T} ^ {2} \| x _ {0} - x ^ {*} \| ^ {2}. \tag {14}
$$
This guarantee aligns with the  $O(\mathrm{csch}_T^2)$  convergence rate reported in [19].
As a rule of thumb, when the expected convergence rate is  $O(\rho (T))$ , we set  $\lambda^F (t) = \rho (T) / \rho (t)$ .
This dynamical system models the unified AGM in [19] and the constant step scheme I [26, Equation 2.2.19].
TMM ODE. We consider the following novel limiting ODE for the triple momentum method (TMM) [46] (see Appendix E.1 for the derivation and a comparison with the one in [38]):
$$
\ddot {X} + 3 \sqrt {\mu} \dot {X} + 2 \nabla f (X) = 0. \tag {TMMODE}
$$
This ODE model can be written as (9) with  $H^{F}(t,\tau) = -2e^{\sqrt{\mu} (\tau -t)} + 4e^{2\sqrt{\mu} (\tau -t)}$  (see Appendix F.3). By setting the multiplier function as  $\lambda^F (t) = e^{2\sqrt{\mu} (t - T)}$ , the PEP kernel (10) can be computed as (see Appendix G.3)
$$
S ^ {F} (t, \tau) = 2 (\nu - \mu e ^ {- 2 \sqrt {\mu} T}) e ^ {\sqrt {\mu} (t + \tau - 2 T)}, \tag {15}
$$
which is positive semidefinite for  $\nu = \mu e^{-2\sqrt{\mu} T}$ . Consequently, Theorem 1 implies that TMM ODE achieves the following convergence guarantee:
$$
\tilde {f} (X (T)) - \tilde {f} \left(x ^ {*}\right) \leq e ^ {- 2 \sqrt {\mu} T} \left(\tilde {f} \left(x _ {0}\right) - \tilde {f} \left(x ^ {*}\right) + \mu \left\| x _ {0} - x ^ {*} \right\| ^ {2}\right), \tag {16}
$$
which is new to the literature. In Appendix H.3, we show that this convergence rate match aligns with the known convergence guarantee for TMM in the discrete-time case.
ITEM ODE. We consider the following new limiting ODE of the information-theoretic exact method (ITEM) [40] (see Appendix E.2 for the derivation of ITEM ODE):
$$
\ddot {X} + 3 \sqrt {\mu} \coth_ {t} \dot {X} + 2 \nabla f (X) = 0, \quad \text {(I T E M O D E)}
$$
where  $\coth_t$  denotes the corresponding hyperbolic function with the argument  $\sqrt{\mu} t$ . This ODE model can be written as (9) with  $H^F(t, \tau) = 4 \sinh_\tau \cosh_\tau \coth_t \operatorname{csch}_t^2 + 2 \sinh_\tau \operatorname{csch}_t (1 - 2 \coth_t^2)$  (see Appendix F.4). By choosing the multiplier function as  $\lambda^F(t) = \sinh^2(\sqrt{\mu} t) / \sinh^2(\sqrt{\mu} T)$ , the PEP kernel (10) can be computed as (see Appendix G.4)
$$
S ^ {F} (t, \tau) = 2 \operatorname {c s c h} _ {T} ^ {2} (\nu - \mu \operatorname {c s c h} _ {T} ^ {2}) \sinh_ {t} \sinh_ {\tau}. \tag {17}
$$
For  $\nu = \mu \mathrm{csch}_T^2$ , this kernel is positive semidefinite. It follows from Theorem 1 that ITEM ODE achieves the following convergence guarantee:
$$
\tilde {f} (X (T)) - \tilde {f} \left(x ^ {*}\right) \leq \mu \operatorname {c s c h} _ {T} ^ {2} \| x _ {0} - x ^ {*} \| ^ {2}, \tag {18}
$$
which is a novel result. In Appendix H.4, we show that this guarantee matches the known convergence rate for ITEM in the discrete-time case.
4 Continuous PEP for minimizing velocity and gradient norm
In this section, we present a result analogous to Theorem 1 to address convergence rates on the squared velocity norm  $\| \dot{X} (T)\| ^2$  or the squared gradient norm  $\| \nabla f(X(T))\| ^2$ . For continuous-time ODE models, the analysis of convergence rates on the squared gradient norm  $\| \nabla f(X(T))\| ^2$  was first presented in [37]. However, their argument relies on the use of L'Hôpital's rule, which might give the impression that their approach is based on a clever trick or appears somewhat mysterious.
4.1 Translating convergence rates on  $\| \dot{X} (t)\| ^2$  into convergence rates on  $\| \nabla f(X(T))\| ^2$
In this subsection, we present a novel approach for establishing the convergence guarantee of ODE models on the squared gradient norm  $\| \nabla f(X(T))\|^2$ . The crucial insight lies in expressing  $\nabla f(X(T))$  as  $\int_0^T\nabla f(X(\tau))\delta_T(\tau)d\tau$ , where  $\delta_T$  denotes the Dirac delta function centered at  $\tau = T$ . Suppose we have a guarantee of the following form:
$$
\left\| \int_ {0} ^ {T} \alpha_ {t} (\tau) \nabla f (X (\tau)) d \tau \right\| ^ {2} \leq \rho \left(f \left(x _ {0}\right) - f \left(x ^ {*}\right)\right), \tag {19}
$$
where  $X \in C^{1}([0,T];\mathbb{R}^{d})$  and  $\{\alpha_{t}\}$  is a family of functions parametrized by  $t \in (0,T)$ . In particular, we note that a convergence guarantee on  $\|C(t)\dot{X}(t)\|^2$  of the dynamics (4) can be written as (19) with  $\alpha_{t}(\tau) = C(t)H(t,\tau)$ . A well-known argument for constructing the Dirac delta function (see
[35, Section 3.2]) shows that the weighted integral  $\int_0^T\alpha_t(\tau)\nabla f(X(\tau))d\tau$  converges to  $\nabla f(X(T))$  as  $t\to T$ , if the following conditions hold: (i)  $\alpha_{t}(\tau)\geq 0$ , (ii)  $\int_0^T\alpha_t(\tau)d\tau \to 1$  as  $t\to T$ , and (iii) for every  $\eta \in (0,T)$ , we have  $\int_0^\eta \alpha_t(\tau)d\tau \to 0$  as  $t\to T$ . When  $\alpha_{t}$  satisfies these properties, we say that the function  $\alpha_{t}$  converges to the Dirac delta function  $\delta_T$ . Consequently, taking the limit  $t\to T$  in (19) yields the following guarantee on  $\| \nabla f(X(T))\|^2$ :
$$
\left\| \nabla f (X (T)) \right\| ^ {2} \leq \rho \left(f (x _ {0}) - f (x ^ {*})\right).
$$
4.2 Convergence analysis via positive semidefinite kernels
In this subsection, we introduce a variant of continuous PEP that establishes the convergence rate on  $\| \dot{X}(T) \|^2$  through checking the positive semidefiniteness of the PEP kernel. Notably, this methodology can also prove convergence rates on  $\| \nabla f(X(T)) \|^2$  because the convergence rates on  $\| \dot{X}(t) \|^2$  can be translated into those on  $\| \nabla f(X(T)) \|^2$ , as discussed in the previous subsection.
Reparametrization to time-varying functions. In Section 3.3, we employed a reparametrization technique to deal with strongly convex objective functions. In this section, we first apply the same technique again, leading to the following expression:
$$
\dot {X} (t) = - \int_ {0} ^ {t} H ^ {G} (t, \tau) \nabla \hat {f} (X (\tau)) d \tau , \tag {20}
$$
where  $\hat{f}(x) \coloneqq f(x) - \frac{\mu}{2} \| x - x_0 \|^2$ . However, we do not proceed directly with this form. Instead, we introduce an additional reparametrization step. Given a solution  $X$  to (20), and a function  $\lambda^G \in C^1([0,T); \mathbb{R}_{\geq 0})$ , we define a family of functions  $\{\hat{f}_t\}_{t \in [0,T)}$  as  $\hat{f}_t(x) \coloneqq \lambda^G(t)\hat{f}(x) - \langle \int_0^t \dot{\lambda}^G(\tau)\nabla \hat{f}(X(\tau))d\tau, x \rangle$ . Then, we can show that (20) can be equivalently written in the following form (see Appendix D.1):
$$
\dot {X} (t) = - \int_ {0} ^ {t} \bar {H} ^ {G} (t, \tau) \nabla \hat {f} _ {\tau} (X (\tau)) d \tau , \tag {21}
$$
for some kernel  $\bar{H}^G$ . The following theorem is analogous to Theorem 1 for our current purpose.
Theorem 2. Let  $\nu >0$ ,  $t_\mathrm{end}\in (0,T]$ ,  $\alpha^G\in C([0,t_\mathrm{end}],\mathbb{R})$ , and  $\lambda^G\in C^1 ([0,t_\mathrm{end}];\mathbb{R}_{\geq 0})$  such that  $\lambda^G (0) = 1$  and  $\dot{\lambda}^{G}(t)\geq 0$  for all  $t$ . Then, any solution to (21) satisfies
$$
\left\| \int_ {0} ^ {t _ {\text {e n d}}} \alpha^ {G} (\tau) \nabla \hat {f} _ {\tau} (X (\tau)) d \tau \right\| ^ {2} \leq \nu \sup  _ {x \in \mathbb {R} ^ {d}} \left\{\hat {f} (x _ {0}) - \hat {f} (x) \right\}, \tag {22}
$$
if the following PEP kernel defined on  $[0, t_{\mathrm{end}}]^2$  is positive semidefinite:
$$
S ^ {G} (t, \tau) = \nu \bar {H} ^ {G} (t, \tau) - 2 \alpha^ {G} (t) \alpha^ {G} (\tau), \quad t \geq \tau . \tag {23}
$$
In particular, the choice  $\alpha^G(t) = C(t_{\mathrm{end}})\bar{H}^G(t_{\mathrm{end}},t)$  gives a guarantee on  $\|C(t_{\mathrm{end}})\dot{X}(t_{\mathrm{end}})\|^2$ .
The proof of Theorem 2 can be found in Appendix D. We now use this theorem to establish convergence rates of the anti-transposed dynamics<sup>9</sup> of the ODE models studied in Section 3.3.
OGM-G ODE. By taking the limit of the stepsize in OGM-G [18], Suh et al. [37] obtained the following ODE model for the non-strongly convex case  $(\mu = 0):^{10}$
$$
\ddot {X} + \frac {3}{T - t} \dot {X} + \nabla f (X) = 0. \tag {OGM-G ODE}
$$
This ODE model is the anti-transposed dynamics of AGM ODE, as it can be expressed as (20) with  $H^{G}(t,\tau) = (T - t)^{3} / (T - \tau)^{3}$  (see Appendix F.5). To use Theorem 2, we choose  $\lambda^G (t) = T^2 /(T - \tau)^{-1}$
This expression is identical in form to (9), but we use the notation  $H^G$  to avoid any notational overlap.
<sup>9</sup>We refer to (20) as the anti-transposed dynamics of (9), if  $H^{G}(t,\tau) = H^{F}(T - \tau ,T - t)$  for all  $t,\tau$ .
<sup>10</sup>We modified the coefficient of  $\nabla f(X(t))$  from 2 to 1.
$t)^{2.11}$  We set the terminal time  $t_\mathrm{end}$  before  $T$  and apply a limiting argument to prove the convergence rate on  $\| \nabla f(X(T))\|^2$ . By setting  $\alpha^G (t) = C(t_\mathrm{end})\tilde{H}^G (t_\mathrm{end},t)$  with  $C(t_{\mathrm{end}}) = 1 / (T - t_{\mathrm{end}})$ , we can compute the PEP kernel (23) as (see Appendix G.5)
$$
S ^ {G} (t, \tau) = \left(\nu - \frac {2}{T ^ {2}}\right) \frac {(T - t) (T - \tau)}{T ^ {2}}. \tag {24}
$$
Since this kernel is the anti-transpose of the PEP kernel for AGM ODE in the proof of Proposition 2, we have  $S^G(t,\tau) \succeq 0$  when  $\nu = 2/T^2$  by Proposition 1 (c). By Theorem 2, we obtain the following inequality:
$$
\left\| \int_ {0} ^ {t _ {\text {e n d}}} \frac {(T - t _ {\text {e n d}}) ^ {2}}{(T - \tau) ^ {3}} \nabla f (X (\tau)) d \tau \right\| ^ {2} = \left\| \frac {\dot {X} (t _ {\text {e n d}})}{T - t _ {\text {e n d}}} \right\| ^ {2} \leq \frac {2}{T ^ {2}} \sup  _ {x \in \mathbb {R} ^ {d}} \left\{f \left(x _ {0}\right) - f (x) \right\}. \tag {25}
$$
Observing that the function  $\tau \mapsto \frac{(T - t_{\mathrm{end}})^2}{(T - \tau)^3}\mathbf{1}_{[0,t_{\mathrm{end}}]}(\tau)$  converges to  $\frac{1}{2}\delta_T$  as  $t_\mathrm{end}\to T$ , we have  $\int_0^{t_{\mathrm{end}}}\frac{(T - t_{\mathrm{end}})^2}{(T - \tau)^3}\nabla f(X(\tau))\to \frac{1}{2}\nabla f(X(T))$  as  $t_\mathrm{end}\to T$ . Substituting this result into (25), we deduce the following convergence guarantee:
$$
\left\| \nabla f (X (T)) \right\| ^ {2} \leq \frac {8}{T ^ {2}} \sup  _ {x \in \mathbb {R} ^ {d}} \left\{f \left(x _ {0}\right) - f (x) \right\}, \tag {26}
$$
which recovers the known convergence rate of OGM-G ODE in [37].
AGM-SC ODE. We now analyze the convergence of AGM-SC ODE in terms of the squared velocity norm, using Theorem 2. This ODE model is the anti-transposed dynamics of itself, as it can be expressed as (20) with  $H^{G}(t,\tau) = (1 + \sqrt{\mu}\tau -\sqrt{\mu} t)e^{\sqrt{\mu} (\tau -t)}$ . We choose  $\lambda^G (t) = e^{\sqrt{\mu} t}$  and  $t_\mathrm{end} = T$ . By setting  $\alpha^G (t) = C(T)\bar{H}^G (T,t)$  with  $C(T) = \sqrt{\mu} /2$ , the PEP kernel (23) is expressed as (see Appendix G.6)
$$
S ^ {G} (t, \tau) = \nu e ^ {- \sqrt {\mu} t} - \frac {\mu}{2} e ^ {- 2 \sqrt {\mu} T}, \quad t \geq \tau . \tag {27}
$$
Since this kernel is the anti-transpose of (11), it is positive semidefinite when  $\nu = \frac{\mu}{2} e^{-\sqrt{\mu} T}$  by Proposition 1 (c). Therefore, we conclude that AGM-SC ODE achieves the following convergence guarantee:
$$
\left\| \frac {\sqrt {\mu}}{2} \dot {X} (T) \right\| ^ {2} \leq \frac {\mu}{2} e ^ {- \sqrt {\mu} T} \sup  _ {x \in \mathbb {R} ^ {d}} \left\{\hat {f} \left(x _ {0}\right) - \hat {f} (x) \right\}, \tag {28}
$$
which is new to the literature. A numerical experiment for this guarantee can be found in Appendix I.1.
Unified AGM-G ODE. In [19], the following unified AGM-G ODE is proposed:
$$
\ddot {X} + \frac {\sqrt {\mu}}{2} \left(\tanh  _ {T - t} + 3 \coth_ {T - t}\right) \dot {X} + \nabla f (X) = 0, \quad \text {(U n i f i e d A G M - G O D E)}
$$
where  $\tanh_{T - t}$  and  $\coth_{T - t}$  denote the corresponding hyperbolic functions with the argument  $\frac{\sqrt{\mu}}{2} (T - t)$ . This ODE model is the anti-transposed dynamics of Unified AGM ODE, as it can be expressed as (20) with  $H^{G}(t,\tau) = (1 + \coth_{T - \tau}^{2}(\log (\mathrm{sech}_{T - \tau}^{2}) - \log (\mathrm{sech}_{T - t}^{2})))\frac{\sinh_{T - t}\cosh_{T - t}}{\sinh_{T - \tau}\cosh_{T - \tau}}$  (see Appendix F.6). To identify its convergence rate in terms of the squared gradient norm, we choose  $\lambda^G (t) = \mathrm{csch}_{T - t}^2 /\mathrm{csch}_T^2$ . By setting  $\alpha^G (t) = C(t_\mathrm{end})\bar{H}^G (t_\mathrm{end},t)$  with  $C(t_{\mathrm{end}}) = \frac{\sqrt{\mu}}{2}\operatorname{sech}_{T - t_{\mathrm{end}}}\operatorname{csch}_{T - t_{\mathrm{end}}}$ , the PEP kernel (23) is expressed as (see Appendix G.7)
$$
S ^ {G} (t, \tau) = \left(\nu - \frac {\mu}{2} \operatorname {c s c h} _ {T} ^ {2}\right) \frac {\tanh  _ {T - t} \tanh  _ {T - \tau}}{\sinh_ {T} ^ {2}} + \nu \frac {\tanh  _ {T - t} \tanh  _ {T - \tau} \sinh_ {T - t} ^ {2}}{\sinh_ {T} ^ {2}}, \quad t \geq \tau . \tag {29}
$$
Since this kernel is the anti-transpose of (13), it is positive semidefinite when  $\nu = \frac{\mu}{2}\mathrm{csch}_T^2$  by Proposition 1 (c). Therefore, Unified AGM-G ODE achieves the following convergence guarantee:
<sup>11</sup>As a rule of thumb, when the expected convergence rate is  $O(\rho(T))$ , we set  $\lambda^G(t) = \rho(T - t) / \rho(T)$ .
In our argument using the Dirac delta function, we rely on the fact that the solution  $X$  to OGM-G ODE can be continuously extended to  $t = T$ , which was shown in [37, Appendix D.3].
$$
\left\| C \left(t _ {\text {e n d}}\right) \dot {X} \left(t _ {\text {e n d}}\right) \right\| ^ {2} \leq \frac {\mu}{2} \operatorname {c s c h} _ {T} ^ {2} \sup  _ {x \in \mathbb {R} ^ {d}} \left\{\hat {f} \left(x _ {0}\right) - \hat {f} (x) \right\}. \tag {30}
$$
In Appendix G.7, we show that  $C(t_{\mathrm{end}})\dot{X} (t_{\mathrm{end}})\to -\frac{1}{2}\nabla f(X(T))$  as  $t_\mathrm{end}\rightarrow T$ . As a result, we have the following convergence guarantee on  $\| \nabla f(X(T))\| ^2$ :
$$
\left\| \nabla f (X (T)) \right\| ^ {2} \leq 2 \mu \operatorname {c s c h} _ {T} ^ {2} \sup  _ {x \in \mathbb {R} ^ {d}} \left\{\hat {f} \left(x _ {0}\right) - \hat {f} (x) \right\}, \tag {31}
$$
which recovers the known rate of Unified AGM-G ODE in [19].
The convergence analyses of the anti-transposed dynamics of TMM ODE and ITEM ODE are deferred to Appendix I.2. The convergence guarantees in this section can also be established using a Lyapunov function argument, as detailed in Appendix I.3.
5 Correspondence between minimizing function values and gradient norms
In Section 4.2, we verified the positive semidefiniteness of the PEP kernel (23) by showing its anti-transpose relationship with the PEP kernel (10), that is,  $S^{F}(t,\tau) = S^{G}(T - \tau ,T - t)$ . The following proposition indicates that this relationship is not coincidental.
Proposition 3. Suppose  $H^{F}(t,\tau) = H^{G}(T - \tau ,T - t)$ ,  $\lambda^F (t) = 1 / \lambda^G (T - t)$ , and  $\alpha^F (t) = \alpha^G (T - t)$  for all  $t$  and  $\tau$ . Then, we have  $S^{F}(t,\tau) = S^{G}(T - \tau ,T - t)$  for all  $t$  and  $\tau$ .
The following proposition offers a general result for translating convergence rates on  $\| \int_0^{t_{\mathrm{end}}}\alpha^G (\tau)\nabla \hat{f}_\tau (X(\tau))d\tau \| ^2$  into those on  $\| \nabla f(X(T))\| ^2$
Proposition 4. Under the assumptions in Proposition 3, if  $\lambda^G(t) \to \infty$  as  $t \to T$ , then we have  $\int_0^{t_{\mathrm{end}}} \alpha^G(\tau) \nabla \hat{f}_\tau(X(\tau)) d\tau \to \frac{1}{2} \nabla f(X(T))$  as  $t_{\mathrm{end}} \to T$ .
Both Propositions 3 and 4 have straightforward proofs through calculations, which can be found in Appendix J. Now, the next result naturally follows.
Theorem 3. Under the assumptions in Propositions 3 and 4, the following statements are equivalent:
(a) Theorem 1 proves the convergence guarantee  $\tilde{f}(X(T)) - \tilde{f}(x^{*}) \leq \nu \| x_{0} - x^{*}\|^{2}$ .
(b) Theorem 2 proves the convergence guarantee  $\| \nabla f(X(T))\| ^2\leq 4\nu \sup_x\{\hat{f} (x_0) - \hat{f} (x)\}$ .
Proof. (a) holds  $\Longleftrightarrow S^{F} \succeq 0 \Longleftrightarrow S^{G} \succeq 0 \Longleftrightarrow$  (b) holds.
In this paper, we have developed a novel and simple framework for analyzing the convergence rates of accelerated gradient flows via positive semidefinite kernels. Our framework enhances our understanding of continuous-time models for accelerated first-order methods and bridges the gap between the continuous-time and discrete-time PEP frameworks. For instance, the continuous PEP in Section 3 can be viewed as a continuous-time limit of the discrete PEP presented in [8] (see Appendix H.2). This connection unlocks new opportunities for studying PEP. Future research directions may involve extending our framework using techniques employed in the ODE models literature, such as generalizing to non-Euclidean settings [20, 1] or analyzing high-resolution ODE models [33].
Codes. The codes used to generate the figures in this paper are available at https://github.com/jungbinkim1/Continuous-PEP.
This work was supported in part by the National Research Foundation of Korea funded by MSIT(2020R1C1C1009766), and the Information and Communications Technology Planning and Evaluation (IITP) grant funded by MSIT(2022-0-00124, 2022-0-00480).
[18] Donghwan Kim and Jeffrey A Fessler. Optimizing the efficiency of first-order methods for decreasing the gradient of smooth convex functions. Journal of optimization theory and applications, 188(1):192-219, 2021.
[19] Jungbin Kim and Insoon Yang. Unifying Nesterov's accelerated gradient methods for convex and strongly convex objective functions. In International Conference on Machine Learning, pages 16897-16954. PMLR, 2023.
[20] Walid Krichene, Alexandre Bayen, and Peter L Bartlett. Accelerated mirror descent in continuous and discrete time. In Advances in Neural Information Processing Systems, volume 28, 2015.
[21] Laurent Lessard, Benjamin Recht, and Andrew Packard. Analysis and design of optimization algorithms via integral quadratic constraints. SIAM Journal on Optimization, 26(1):57-95, 2016.
[22] Laurent Lessard and Peter Seiler. Direct synthesis of iterative algorithms with bounds on achievable worst-case convergence rate. In 2020 American Control Conference (ACC), pages 119-125. IEEE, 2020.
[23] Aleksandr Mikhailovich Lyapunov. The general problem of the stability of motion. International journal of control, 55(3):531-534, 1992.
[24] Alexandre Megretski and Anders Rantzer. System analysis via integral quadratic constraints. IEEE transactions on automatic control, 42(6):819-830, 1997.
[25] Céline Moucer, Adrien Taylor, and Francis Bach. A systematic approach to Lyapunov analyses of continuous-time models in convex optimization. SIAM Journal on Optimization, 33(3):1558-1586, 2023.
[26] Yuri Nesterov. Lectures on Convex Optimization, volume 137. Springer, 2018.
[27] Yurii E Nesterov. A method for solving the convex programming problem with convergence rate  $o(1 / k^2)$ . In Dokl. akad. nauk Sssr, volume 269, pages 543-547, 1983.
[28] Frigyes Riesz and Béla Sz Nagy. Functional analysis. Courier Corporation, 2012.
[29] W. Rudin. Real and Complex Analysis. McGraw-Hill, 1987.
[30] W. Rudin. Functional Analysis. McGraw-Hill, 1991.
[31] Jesús María Sanz Serna and Konstantinos C Zygalakis. The connections between Lyapunov functions for some optimization algorithms and differential equations. SIAM Journal on Numerical Analysis, 59(3):1542-1565, 2021.
[32] Damien Scieur, Vincent Roulet, Francis Bach, and Alexandre d'Aspremont. Integration methods and optimization algorithms. Advances in Neural Information Processing Systems, 30, 2017.
[33] Bin Shi, Simon S Du, Michael I Jordan, and Weijie J Su. Understanding the acceleration phenomenon via high-resolution differential equations. Mathematical Programming, pages 1-70, 2021.
[34] Bin Shi, Simon S Du, Weijie Su, and Michael I Jordan. Acceleration via symplectic discretization of high-resolution differential equations. In Advances in Neural Information Processing Systems, volume 32, 2019.
[35] Elias M Stein and Rami Shakarchi. Real Analysis: Measure Theory, Integration, and Hilbert Spaces. Princeton University Press, 2009.
[36] Weijie Su, Stephen Boyd, and Emmanuel J Candes. A differential equation for modeling Nesterov's accelerated gradient method: Theory and insights. Journal of Machine Learning Research, 17:1-43, 2016.
[37] Jaewook J Suh, Gyumin Roh, and Ernest K Ryu. Continuous-time analysis of accelerated gradient methods via conservation laws in dilated coordinate systems. In International Conference on Machine Learning, pages 20640–20667. PMLR, 2022.
[38] Boya Sun, Jemin George, and Solmaz Kia. High-resolution modeling of the fastest first-order optimization method for strongly convex functions. In 2020 59th IEEE Conference on Decision and Control (CDC), pages 4237-4242. IEEE, 2020.
[39] Adrien Taylor and Francis Bach. Stochastic first-order methods: Non-asymptotic and computer-aided analyses via potential functions. In Alina Beygelzimer and Daniel Hsu, editors, Proceedings of the Thirty-Second Conference on Learning Theory, volume 99 of Proceedings of Machine Learning Research, pages 2934–2992. PMLR, 25–28 Jun 2019.
[40] Adrien Taylor and Yoel Drori. An optimal gradient method for smooth strongly convex minimization. Mathematical Programming, pages 1-38, 2022.
[41] Adrien Taylor, Bryan Van Scoy, and Laurent Lessard. Lyapunov functions for first-order methods: Tight automated convergence guarantees. In International Conference on Machine Learning, pages 4897-4906. PMLR, 2018.
[42] Adrien B. Taylor, Julien M. Hendrickx, and François Glineur. Smooth strongly convex interpolation and exact worst-case performance of first-order methods. Mathematical Programming, 161(1-2):307-345, May 2016.
[43] Adrien B Taylor, Julien M Hendrickx, and François Glineur. Exact worst-case performance of first-order methods for composite convex optimization. SIAM Journal on Optimization, 27(3):1283-1313, 2017.
[44] Adrien B Taylor, Julien M Hendrickx, and François Glineur. Exact worst-case convergence rates of the proximal gradient method for composite convex minimization. Journal of Optimization Theory and Applications, 178:455-476, 2018.
[45] Manu Upadhyaya, Sebastian Banert, Adrien B Taylor, and Pontus Giselsson. Automated tight Lyapunov analysis for first-order methods. arXiv preprint arXiv:2302.06713, 2023.
[46] Bryan Van Scoy, Randy A Freeman, and Kevin M Lynch. The fastest known globally convergent first-order method for minimizing strongly convex functions. IEEE Control Systems Letters, 2(1):49-54, 2017.
[47] Andre Wibisono, Ashia C Wilson, and Michael I Jordan. A variational perspective on accelerated methods in optimization. proceedings of the National Academy of Sciences, 113(47):E7351–E7358, 2016.
[48] Ashia C. Wilson, Ben Recht, and Michael I. Jordan. A Lyapunov analysis of accelerated methods in optimization. Journal of Machine Learning Research, 22(113):1-34, 2021.
[49] Jingzhao Zhang, Aryan Mokhtari, Suvrit Sra, and Ali Jadbabaie. Direct Runge-Kutta discretization achieves acceleration. In Advances in neural information processing systems, volume 31, 2018.