Title: Convergence analysis of ODE models for accelerated first-order methods via positive semidefinite kernels
Venue: NeurIPS 2023
Accepted: yes
Keywords: convex optimization

accelerated gradient methods
Abstract: We propose a novel methodology that systematically analyzes ordinary differential equation (ODE) models for first-order optimization methods by converting the task of proving convergence rates into verifying the positive semidefiniteness of specific Hilbert-Schmidt integral operators. Our approach is based on the performance estimation problems (PEP) introduced by Drori and Teboulle. Unlike previous works on PEP, which rely on finite-dimensional linear algebra, we use tools from functional analysis. Using the proposed method, we establish convergence rates of various accelerated gradient flow models, some of which are new. As an immediate consequence of our framework, we show a correspondence between minimizing function values and minimizing gradient norms.

Convergence Analysis of ODE Models for Accelerated First-Order Methods via Positive Semidefinite Kernels

Abstract

We propose a novel methodology that systematically analyzes ordinary differential equation (ODE) models for first-order optimization methods by converting the task of proving convergence rates into verifying the positive semidefiniteness of specific Hilbert-Schmidt integral operators. Our approach is based on the performance estimation problems (PEP) introduced by Drori and Teboulle [8]. Unlike previous works on PEP, which rely on finite-dimensional linear algebra, we use tools from functional analysis. Using the proposed method, we establish convergence rates of various accelerated gradient flow models, some of which are new. As an immediate consequence of our framework, we show a correspondence between minimizing function values and minimizing gradient norms.

1 Introduction

We consider the following convex optimization problem:

where f : R d → R is a continuously differentiable ( µ -strongly) convex function. We assume that a minimizer x ∗ exists. First-order methods, for example, gradient descent and Nesterov's accelerated gradient method, are popular in solving this problem due to their low cost per iteration and dimension-free oracle complexities. These methods can be analyzed by examining their limiting ODEs. For instance, the gradient descent x k +1 = x k -s ∇ f ( x k ) corresponds to the gradient flow ˙ X ( t ) = -∇ f ( X ( t )) . Building upon this idea, Su et al. [36] derived the limiting ODE for Nesterov's accelerated gradient method (AGM) [27] and analyzed its convergence rate, offering valuable insights into momentum-based algorithms. A common approach to establishing convergence rates of continuous-time ODE models involves using Lyapunov functions [23].

In this paper, we propose a generic framework that builds upon the performance estimation problem (PEP) presented in [8] for analyzing the convergence rates of ODE models for various accelerated firstorder methods, including Nesterov's AGM. The proposed method is designed from the Lagrangian dual of a relaxed version of continuous-time PEP. Consequently, our framework transforms the task of proving convergence rate into verifying the positive semidefiniteness of a specific integral kernel. Moreover, our framework can also ensure the tightness of the resulting guarantee, meaning that the obtained convergence guarantee is optimal among all possible convergence rates that can be derived from weighted integrals of the inequalities employed in convergence proofs. Using the proposed framework, we confirm the convergence rates of existing ODE models and uncover those of new accelerated ODE models. In traditional convergence analysis of ODE models, it can be challenging to design appropriate Lyapunov functions. 2 However, in our framework, we only need to verify

the positive semidefiniteness of a specific integral kernel. This approach circumvents the need for Lyapunov function design, making our framework more straightforward for analyzing convergence rates.

In the discrete-time setting, the PEP framework has been extensively studied for its ability to systematically obtain tight convergence guarantees [42] and facilitate the design of new optimization methods [17, 18, 40]. However, analyzing PEP is typically regarded as challenging to comprehend due to the involvement of large matrices with complex expressions. In contrast, our framework utilizes integral kernels, which serve as a continuous-time counterpart to matrices. The computational process within our approach yields simpler outcomes. Consequently, our continuous-time PEP framework has the potential to offer valuable insights into the analysis of the discrete-time PEP, similar to how ODE models have helped designing and analyzing discrete-time methods in the literature [20, 47]. By bridging the gap between the continuous and discrete settings, our methodology enhances the understanding of the PEP framework.

1.1 Related work

Continuous-time models for first-order methods. The investigation into the continuous-time limit of accelerated first-order methods began with the study of AGM ODE [36, 2, 3]. Since then, subsequent researches have explored various aspects of the ODE models. These include generalizations within the mirror descent setup [20], a broader family of dynamics derived using Lagrangian mechanics [47, 48, 19], high-resolution ODE models [34, 33], and continuized methods [9]. Systematic methodologies for finding Lyapunov functions were developed, including deriving them from Hamilton's equations [6] or dilated coordinate systems [37]. For obtaining accelerated discrete-time algorithms, several studies have applied discretization schemes, such as symplectic [4] and Runge-Kutta [49] schemes, to discretize accelerated ODE models. [32] showed that applying multi-step integration schemes to the gradient flow also yields accelerated algorithms. A particularly relevant study is [19], as they present the dynamics in the form of (4) using the H-kernel , which plays a crucial role in our analysis.

Performance estimation problems. The idea of using performance estimation problems to analyze the convergence rate of optimization methods was first introduced by [8]. This concept was further refined by employing a convex interpolation argument in [42] and was applied to a wide range of settings in [5, 43, 44, 12, 11, 7, 16]. The idea of performance estimation has been used to construct Lyapunov functions in [41, 39, 25, 45]. In particular, [25] analyzes continuous-time ODE models. However, their methodology differs from ours, as they employed semidefinite programs of finite dimension. Another closely related approach is based on integral quadratic constraints (IQC) from control theory [24], which were used to analyze the convergence rate of first-order methods in [21]. The IQC framework has been further studied in [14, 15, 10, 22, 31]. One practical application of PEP and IQC is the design of novel algorithms by optimizing the convergence guarantees. Some notable examples include OGM [17], TMM [46], ITEM [40], and OGM-G [18].

2 Preliminaries and notations

In this section, we review some basic notions from functional analysis that will be used throughout the paper. For a more detailed treatment, we refer the reader to the textbooks [29, 30, 28].

Function spaces. We denote the set of continuous functions from [0 , T ] to R d by C ([0 , T ]; R d ) and the set of continuously differentiable functions from [0 , T ] to R d by C 1 ([0 , T ]; R d ) . We define the space L 2 ([0 , T ]; R d ) as the set of all measurable functions f : [0 , T ] → R d that satisfy ∫ T 0 ∥ f ( x ) ∥ 2 R d dx < ∞ . Then, L 2 ([0 , T ]; R d ) is a Hilbert space, equipped with an inner product and a norm defined by ⟨ f, g ⟩ L 2 ([0 ,T ]; R d ) = ∫ T 0 ⟨ f ( t ) , g ( t ) ⟩ R d dt and ∥ f ∥ L 2 ([0 ,T ]; R d ) = √ ⟨ f, f ⟩ L 2 ([0 ,T ]; R d ) .

Integral operators. An integral operator is a linear operator that maps a function f to another function Kf given by

where k : [0 , T ] 2 → R is the associated integral kernel. Intuitively, an integral kernel can be seen as a continuous-time version of a matrix. A Hilbert-Schmidt kernel is an integral kernel k that is square integrable, i.e., k ∈ L 2 ([0 , T ] 2 ; R ) . When k is a Hilbert-Schmidt kernel, the associated integral operator K is a well-defined operator on L 2 ([0 , T ]; R d ) , called a Hilbert-Schmidt integral operator. If a Hilbert-Schmidt kernel k is symmetric, i.e., k ( t, τ ) = k ( τ, t ) for all t, τ ∈ [0 , T ] , then the associated operator is also symmetric in the sense that ⟨ Kf,g ⟩ = ⟨ f, Kg ⟩ for all f, g ∈ L 2 ([0 , T ]; R d ) . Throughout this paper, we will use the term 'kernel' to refer to a Hilbert-Schmidt kernel.

Positive semidefinite kernels. A symmetric operator K on a Hilbert space is said to be positive semidefinite and denoted by K ⪰ 0 if ⟨ Kf,f ⟩ ≥ 0 for all f . When a symmetric kernel k is associated with a positive semidefinite operator K , i.e., ∫ T 0 ∫ T 0 k ( t, τ ) f ( t ) f ( τ ) dtdτ ≥ 0 for all f ∈ L 2 ([0 , T ]; R ) , we say that the kernel k is (integrally) positive semidefinite and denote it by k ⪰ 0 . For continuous kernels, positive semidefiniteness of k is equivalent to the following condition: ∑ n i =1 ∑ n j =1 c i c j k ( t i , t j ) ≥ 0 for any t 1 , . . . , t n ∈ [0 , T ] and c 1 , . . . , c n ∈ R , given n ∈ N .

Proposition 1. We summarize some basic properties of continuous positive semidefinite kernels:

(a) For α ∈ C ([0 , T ]; R ) , the kernel k ( t, τ ) = α ( t ) α ( τ ) is positive semidefinite.

(b) For k 1 , k 2 ⪰ 0 , their product k ( t, τ ) = k 1 ( t, τ ) k 2 ( t, τ ) is positive semidefinite.

(c) For k ⪰ 0 , its anti-transpose ( t, τ ) ↦→ k ( T -τ, T -t ) is also positive semidefinite.

(d) If α ∈ C 1 ([0 , T ]; R ≥ 0 ) is an increasing function on [0 , T ] , then the symmetric kernel k defined as k ( t, τ ) = α ( τ ) for t ≥ τ and k ( t, τ ) = α ( t ) for t ≤ τ is positive semidefinite. 3

(e) For k ⪰ 0 , we have k ( t, t ) ≥ 0 for all t ∈ [0 , T ] .

3 Continuous PEP for minimizing objective function value

In this section, drawing inspiration from its discrete-time counterpart [8, 42], we propose a novel framework for analyzing the convergence rate of ODE models for first-order methods, called the continuous-time performance estimation problem (Continuous PEP) . To illustrate this framework, we use the accelerated gradient flow as an example. Detailed steps can be found in Appendix C. Su et al. [36] derived the limiting ODE of Nesterov's AGM [27] as follows:

with initial conditions X (0) = x 0 and ˙ X (0) = 0 . Suppose we want to establish a convergence guarantee of AGM ODE in the form of

Here, we observe that the constant ρ can be seen as an upper bound of the performance of AGM ODE for the criterion ( f ( X ( T )) -f ( x ∗ )) / ∥ x 0 -x ∗ ∥ 2 . To formalize this idea, we introduce the following optimization problem, which seeks to find the worst-case performance of the given ODE model:

where F µ ( R d ; R ) denotes the set of continuously differentiable µ -strongly convex functions on R d . This problem is useful to analyze the convergence properties of ODE models because the optimal value val( Exact PEP ) of Exact PEP directly provides the guarantee (3) with ρ = val( Exact PEP ) regardless of any particular choice of f .

3.1 Relaxation of PEP

Exact PEP is challenging to solve due to the presence of an unknown function f as an optimization variable. To address this difficulty, we relax the constraint f ∈ F 0 ( R d ; R ) with a set of inequalities that are satisfied by f ∈ F 0 ( R d ; R ) . Before that, we first note that AGM ODE can be expressed as the following continuous-time dynamical system (see [19]):

by setting H ( t, τ ) = τ 3 /t 3 . Here, H ( t, τ ) is called the H-kernel . We introduce two functions, φ : [0 , T ] → R and γ : [0 , T ] → R d , defined as follows:

Using the chain rule and the convexity of f , we can derive the following equality and inequality:

where v = ( x ∗ -x 0 ) / ∥ x 0 -x ∗ ∥ . We can now relax Exact PEP by replacing its constraints with the equality and inequality above, resulting in the following problem:

Since any feasible solution to Exact PEP can be transformed into a feasible solution to Relaxed PEP, we have val( Relaxed PEP ) ≥ val( Exact PEP ) . Therefore, the convergence guarantee (3) holds with ρ = val( Relaxed PEP ) when using the proposed relaxation.

3.2 Lagrangian dual of relaxed PEP

To obtain an upper bound on val( Relaxed PEP ) , we use Lagrangian duality. We introduce two Lagrange multipliers λ 1 ∈ C 1 ([0 , T ]; R ) and λ 2 ∈ C ([0 , T ]; R ≥ 0 ) , where we imposed certain regularity conditions, such as continuity and differentiability, to ensure that the dual problem is well-defined. We then define the Lagrangian function as

When expressed in terms of the inner products in function spaces, we have

where K is the Hilbert-Schmidt integral operator with the symmetric kernel k defined by

The dual function is defined as Dual( λ 1 , λ 2 ) = sup φ,γ,v L ( φ, γ, v ; λ 1 , λ 2 ) . By weak duality, we have val( Relaxed PEP ) ≤ Dual( λ 1 , λ 2 ) for any feasible dual solution ( λ 1 , λ 2 ) . After performing some computations, we obtain the following expression for the dual objective function (see Appendix C):

where S λ 1 ,λ 2 ,ν is a symmetric kernel on [0 , T ] 2 given by

We refer to S λ 1 ,λ 2 ,ν as the PEP kernel . In Appendix H.2, we show that (8) can be viewed as the continuous-time limit of the discrete-time PEP kernel presented in [8].

To describe our framework, given ν feas ∈ (0 , ∞ ) , suppose that the PEP kernel S λ 1 ,λ 2 ,ν feas is positive semidefinite with appropriate multiplier functions λ 1 and λ 2 . Then, ν feas is a feasible solution of the minimization problem in (7), and thus Dual( λ 1 , λ 2 ) ≤ ν feas . On the other hand, by weak duality, val( Relaxed PEP ) ≤ inf Dual( λ 1 , λ 2 ) . Therefore, we conclude that val( Exact PEP ) ≤ val( Relaxed PEP ) ≤ Dual( λ 1 , λ 2 ) ≤ ν feas, which implies that the convergence guarantee (3) automatically holds with ρ = ν feas .

Using this approach, we can recover the known convergence guarantee for AGM ODE in [36].

Proposition 2. AGM ODE achieves the convergence rate (3) with ρ = 2 /T

2 .

Proof. By choosing the multiplier functions λ 1 ( t ) = t 2 /T 2 and λ 2 ( t ) = 2 t/T 2 , we can compute the PEP kernel (8) as S λ 1 ,λ 2 ,ν ( t, τ ) = ( ν -2 T 2 ) tτ T 2 . Since the kernel ( t, τ ) ↦→ tτ is nonzero and positive semidefinite, we have S λ 1 ,λ 2 ,ν ⪰ 0 if and only if ν ≥ 2 /T 2 . Thus, we obtain Dual( λ 1 , λ 2 ) = 2 /T 2 , which establishes the convergence guarantee (3) with ρ = 2 /T 2 .

Remark 1. Furthermore, this convergence guarantee is optimal among all possible guarantees obtained through the weighted integral of (5) . The optimality of this rate follows from the fact that ( λ 1 , λ 2 ) is the optimal solution to the dual problem min λ 1 ,λ 2 Dual( λ 1 , λ 2 ) . See Appendix H.1 for details.

3.3 Applying continuous PEP to various accelerated gradient flows

Note that the proposed method is not dependent on the choice of the H-kernel H ( t, τ ) . Thus, it can be applied to arbitrary dynamics represented in the form of (4). Furthermore, while we have focused on the non-strongly convex case ( µ = 0 ) so far, the following paragraph demonstrates that our method can handle strongly convex objective functions by using a reparametrization technique. 4

Reparametrization from F µ ( R d ; R ) to F 0 ( R d ; R ) . Consider a µ -strongly convex objective function f . Since the proposed method is tailored for non-strongly convex objective functions, we choose to work with the convex function ˆ f ( x ) := f ( x ) -µ 2 ∥ x -x 0 ∥ 2 rather than working directly with f . Accordingly, we consider the following alternative formulation for the dynamical system (4), which involves ∇ ˆ f instead of ∇ f : 5

The following theorem offers a general result that can be used to establish convergence guarantees for dynamical systems of the form (9).

Theorem 1. Let ν > 0 and λ F ∈ C 1 ([0 , T ]; R ≥ 0 ) such that 0 ≤ λ F (0) < 1 , λ F ( T ) = 1 , and ˙ λ F ( t ) ≥ 0 for all t ∈ (0 , T ) . Then, any solution to the integro-differential equation (9) satisfies

where ˜ f ( x ) := f ( x ) -µ 2 ∥ x -x ∗ ∥ 2 , if the following PEP kernel is positive semidefinite:

The proof of Theorem 1 is done by finding a dual feasible point to the PEP and can be found in Appendix C. Below, we establish convergence rates for various ODE models using Theorem 1.

0

T

t

0

T

τ

(a) Contour plot.

t

0

T

τ

0

T

S F

(

t, τ

)

(b) Surface plot.

AGM-SC ODE. We consider the following dynamical system modeling Nesterov's AGM for strongly convex case [26, Equation 2.2.22] (see [48, Equation 7]):

This ODE model can be written as (9) with H F ( t, τ ) = (1+ √ µτ - √ µt ) e √ µ ( τ -t ) (see Appendix F.1). To use Theorem 1, we choose the multiplier function as λ F ( t ) = e √ µ ( t -T ) . 6 The PEP kernel (10) can be computed as (see Appendix G.1)

When ν = µ 2 e - √ µT , this kernel is written as S F ( t, τ ) = µ 2 e -2 √ µT ( e √ µτ -1) , and is visualized in Figure 1. It is positive semidefinite since the function τ ↦→ e √ µτ -1 is a nonnegative increasing function (see Proposition 1 (d)). It follows from Theorem 1 that AGM-SC ODE achieves the following convergence guarantee:

which is consistent with the well-known O ( e - √ µT ) convergence rate of AGM-SC ODE.

Unified AGM ODE. Using a unified Bregman Lagrangian framework, [19] obtained the following ODE that unifies AGM ODE and AGM-SC ODE: 7

where tanh t and coth t denote the corresponding hyperbolic functions with the argument √ µ 2 t . This ODE model can be written as (9) with H F ( t, τ ) = (1 + coth 2 t (log(sech 2 t ) -log(sech 2 τ ))) sinh τ cosh τ sinh t cosh t (see Appendix F.2). We select the multiplier function as λ F ( t ) = sinh 2 t / sinh 2 T . With this choice, the PEP kernel (10) can be expressed as follows (see Appendix G.2):

We show that this kernel is positive semidefinite for ν = µ 2 csch 2 T . Proposition 1 (a) shows that the kernel ( t, τ ) ↦→ tanh t tanh τ is positive semidefinite. Proposition 1 (d) shows that the kernel ( t, τ ) ↦→ sinh 2 τ is positive semidefinite because the function τ ↦→ sinh 2 τ is a nonnegative increasing function. Since the PEP kernel (13) with ν = µ 2 csch 2 T can be expressed as a product of two positive semidefinite kernels, it is positive semidefinite by Proposition 1 (b). Consequently, Theorem 1 implies that Unified AGM ODE achieves the following convergence guarantee:

This guarantee aligns with the O (csch 2 T ) convergence rate reported in [19].

TMM ODE. We consider the following novel limiting ODE for the triple momentum method (TMM) [46] (see Appendix E.1 for the derivation and a comparison with the one in [38]):

This ODE model can be written as (9) with H F ( t, τ ) = -2 e √ µ ( τ -t ) +4 e 2 √ µ ( τ -t ) (see Appendix F.3). By setting the multiplier function as λ F ( t ) = e 2 √ µ ( t -T ) , the PEP kernel (10) can be computed as (see Appendix G.3)

√

which is positive semidefinite for ν = µe -2 µT . Consequently, Theorem 1 implies that TMM ODE achieves the following convergence guarantee:

which is new to the literature. In Appendix H.3, we show that this convergence rate match aligns with the known convergence guarantee for TMM in the discrete-time case.

ITEM ODE. We consider the following new limiting ODE of the information-theoretic exact method (ITEM) [40] (see Appendix E.2 for the derivation of ITEM ODE):

where coth t denotes the corresponding hyperbolic function with the argument √ µt . This ODE model can be written as (9) with H F ( t, τ ) = 4 sinh τ cosh τ coth t csch 2 t +2sinh τ csch t (1 -2 coth 2 t ) (see Appendix F.4). By choosing the multiplier function as λ F ( t ) = sinh 2 ( √ µt ) / sinh 2 ( √ µT ) , the PEP kernel (10) can be computed as (see Appendix G.4)

For ν = µ csch 2 T , this kernel is positive semidefinite. It follows from Theorem 1 that ITEM ODE achieves the following convergence guarantee:

which is a novel result. In Appendix H.4, we show that this guarantee matches the known convergence rate for ITEM in the discrete-time case.

4 Continuous PEP for minimizing velocity and gradient norm

In this section, we present a result analogous to Theorem 1 to address convergence rates on the squared velocity norm ∥ ˙ X ( T ) ∥ 2 or the squared gradient norm ∥∇ f ( X ( T )) ∥ 2 . For continuous-time ODE models, the analysis of convergence rates on the squared gradient norm ∥∇ f ( X ( T )) ∥ 2 was first presented in [37]. However, their argument relies on the use of L'Hôpital's rule, which might give the impression that their approach is based on a clever trick or appears somewhat mysterious.

4.1 Translating convergence rates on ∥ ˙ X ( t ) ∥ 2 into convergence rates on ∥∇ f ( X ( T )) ∥ 2

In this subsection, we present a novel approach for establishing the convergence guarantee of ODE models on the squared gradient norm ∥∇ f ( X ( T )) ∥ 2 . The crucial insight lies in expressing ∇ f ( X ( T )) as ∫ T 0 ∇ f ( X ( τ )) δ T ( τ ) dτ , where δ T denotes the Dirac delta function centered at τ = T . Suppose we have a guarantee of the following form:

where X ∈ C 1 ([0 , T ]; R d ) and { α t } is a family of functions parametrized by t ∈ (0 , T ) . In particular, we note that a convergence guarantee on ∥ C ( t ) ˙ X ( t ) ∥ 2 of the dynamics (4) can be written as (19) with α t ( τ ) = C ( t ) H ( t, τ ) . A well-known argument for constructing the Dirac delta function (see

[35, Section 3.2]) shows that the weighted integral ∫ T 0 α t ( τ ) ∇ f ( X ( τ )) dτ converges to ∇ f ( X ( T )) as t → T , if the following conditions hold: ( i ) α t ( τ ) ≥ 0 , ( ii ) ∫ T 0 α t ( τ ) dτ → 1 as t → T , and ( iii ) for every η ∈ (0 , T ) , we have ∫ η 0 α t ( τ ) dτ → 0 as t → T . When α t satisfies these properties, we say that the function α t converges to the Dirac delta function δ T . Consequently, taking the limit t → T in (19) yields the following guarantee on ∥∇ f ( X ( T )) ∥ 2 :

4.2 Convergence analysis via positive semidefinite kernels

In this subsection, we introduce a variant of continuous PEP that establishes the convergence rate on ∥ ˙ X ( T ) ∥ 2 through checking the positive semidefininteness of the PEP kernel. Notably, this methodology can also prove convergence rates on ∥∇ f ( X ( T )) ∥ 2 because the convergence rates on ∥ ˙ X ( t ) ∥ 2 can be translated into those on ∥∇ f ( X ( T )) ∥ 2 , as discussed in the previous subsection.

Reparametrization to time-varying functions. In Section 3.3, we employed a reparametrization technique to deal with strongly convex objective functions. In this section, we first apply the same technique again, leading to the following expression: 8

where ˆ f ( x ) := f ( x ) -µ 2 ∥ x -x 0 ∥ 2 . However, we do not proceed directly with this form. Instead, we introduce an additional reparametrization step. Given a solution X to (20), and a function λ G ∈ C 1 ([0 , T ); R ≥ 0 ) , we define a family of functions { ˆ f t } t ∈ [0 ,T ) as ˆ f t ( x ) := λ G ( t ) ˆ f ( x ) -⟨ ∫ t 0 ˙ λ G ( τ ) ∇ ˆ f ( X ( τ )) dτ, x ⟩ . Then, we can show that (20) can be equivalently written in the following form (see Appendix D.1):

for some kernel ¯ H G . The following theorem is analogous to Theorem 1 for our current purpose.

Theorem 2. Let ν > 0 , t end ∈ (0 , T ] , α G ∈ C ([0 , t end ] , R ) , and λ G ∈ C 1 ([0 , t end ]; R ≥ 0 ) such that λ G (0) = 1 and ˙ λ G ( t ) ≥ 0 for all t . Then, any solution to (21) satisfies

if the following PEP kernel defined on [0 , t end ] 2 is positive semidefinite:

In particular, the choice α G ( t ) = C ( t end ) ¯ H G ( t end , t ) gives a guarantee on ∥ C ( t end ) ˙ X ( t end ) ∥ 2 .

The proof of Theorem 2 can be found in Appendix D. We now use this theorem to establish convergence rates of the anti-transposed dynamics 9 of the ODE models studied in Section 3.3.

OGM-G ODE. By taking the limit of the stepsize in OGM-G [18], Suh et al. [37] obtained the following ODE model for the non-strongly convex case ( µ = 0 ): 10

This ODE model is the anti-transposed dynamics of AGM ODE, as it can be expressed as (20) with H G ( t, τ ) = ( T -t ) 3 / ( T -τ ) 3 (see Appendix F.5). To use Theorem 2, we choose λ G ( t ) = T 2 / ( T -

t ) 2 . 11 We set the terminal time t end before T and apply a limiting argument to prove the convergence rate on ∥∇ f ( X ( T )) ∥ 2 . By setting α G ( t ) = C ( t end ) ¯ H G ( t end , t ) with C ( t end ) = 1 / ( T -t end ) , we can compute the PEP kernel (23) as (see Appendix G.5)

Since this kernel is the anti-transpose of the PEP kernel for AGM ODE in the proof of Proposition 2, we have S G ( t, τ ) ⪰ 0 when ν = 2 /T 2 by Proposition 1 (c). By Theorem 2, we obtain the following inequality:

Observing that the function τ ↦→ ( T -t end ) 2 ( T -τ ) 3 1 [0 ,t end ] ( τ ) converges to 1 2 δ T as t end → T , we have ∫ t end 0 ( T -t end ) 2 ( T -τ ) 3 ∇ f ( X ( τ )) → 1 2 ∇ f ( X ( T )) as t end → T . 12 Substituting this result into (25), we deduce the following convergence guarantee:

which recovers the known convergence rate of OGM-G ODE in [37].

AGM-SC ODE. We now analyze the convergence of AGM-SC ODE in terms of the squared velocity norm, using Theorem 2. This ODE model is the anti-transposed dynamics of itself, as it can be expressed as (20) with H G ( t, τ ) = (1 + √ µτ - √ µt ) e √ µ ( τ -t ) . We choose λ G ( t ) = e √ µt and t end = T . By setting α G ( t ) = C ( T ) ¯ H G ( T, t ) with C ( T ) = √ µ/ 2 , the PEP kernel (23) is expressed as (see Appendix G.6)

Since this kernel is the anti-transpose of (11), it is positive semidefinite when ν = µ 2 e - √ µT by Proposition 1 (c). Therefore, we conclude that AGM-SC ODE achieves the following convergence guarantee:

which is new to the literature. A numerical experiment for this guarantee can be found in Appendix I.1.

Unified AGM-G ODE. In [19], the following unified AGM-G ODE is proposed:

where tanh T -t and coth T -t denote the corresponding hyperbolic functions with the argument √ µ 2 ( T -t ) . This ODE model is the anti-transposed dynamics of Unified AGM ODE, as it can be expressed as (20) with H G ( t, τ ) = (1 + coth 2 T -τ (log(sech 2 T -τ ) -log(sech 2 T -t ))) sinh T -t cosh T -t sinh T -τ cosh T -τ (see Appendix F.6). To identify its convergence rate in terms of the squared gradient norm, we choose λ G ( t ) = csch 2 T -t / csch 2 T . By setting α G ( t ) = C ( t end ) ¯ H G ( t end , t ) with C ( t end ) = √ µ 2 sech T -t end csch T -t end , the PEP kernel (23) is expressed as (see Appendix G.7)

Since this kernel is the anti-transpose of (13), it is positive semidefinite when ν = µ 2 csch 2 T by Proposition 1 (c). Therefore, Unified AGM-G ODE achieves the following convergence guarantee:

In Appendix G.7, we show that C ( t end ) ˙ X ( t end ) → -1 2 ∇ f ( X ( T )) as t end → T . As a result, we have the following convergence guarantee on ∥∇ f ( X ( T )) ∥ 2 :

which recovers the known rate of Unified AGM-G ODE in [19].

The convergence analyses of the anti-transposed dynamics of TMM ODE and ITEM ODE are deferred to Appendix I.2. The convergence guarantees in this section can also be established using a Lyapunov function argument, as detailed in Appendix I.3.

5 Correspondence between minimizing function values and gradient norms

In Section 4.2, we verified the positive semidefiniteness of the PEP kernel (23) by showing its anti-transpose relationship with the PEP kernel (10), that is, S F ( t, τ ) = S G ( T -τ, T -t ) . The following proposition indicates that this relationship is not coincidental.

The following proposition offers a general result for translating convergence rates on ∥ ∫ t end 0 α G ( τ ) ∇ ˆ f τ ( X ( τ )) dτ ∥ 2 into those on ∥∇ f ( X ( T )) ∥ 2 .

Proposition 4. Under the assumptions in Proposition 3, if λ G ( t ) → ∞ as t → T , then we have ∫ t end 0 α G ( τ ) ∇ ˆ f τ ( X ( τ )) dτ → 1 2 ∇ f ( X ( T )) as t end → T .

Both Propositions 3 and 4 have straightforward proofs through calculations, which can be found in Appendix J. Now, the next result naturally follows.

Theorem 3. Under the assumptions in Propositions 3 and 4, the following statements are equivalent:

(a) Theorem 1 proves the convergence guarantee ˜ f ( X ( T )) -˜ f ( x ∗ ) ≤ ν ∥ x 0 -x ∗ ∥ 2 .

(b) Theorem 2 proves the convergence guarantee ∥∇ f ( X ( T )) ∥ 2 ≤ 4 ν sup x { ˆ f ( x 0 ) -ˆ f ( x ) } .

6 Conclusion

In this paper, we have developed a novel and simple framework for analyzing the convergence rates of accelerated gradient flows via positive semidefinite kernels. Our framework enhances our understanding of continuous-time models for accelerated first-order methods and bridges the gap between the continuous-time and discrete-time PEP frameworks. For instance, the continuous PEP in Section 3 can be viewed as a continuous-time limit of the discrete PEP presented in [8] (see Appendix H.2). This connection unlocks new opportunities for studying PEP. Future research directions may involve extending our framework using techniques employed in the ODE models literature, such as generalizing to non-Euclidean settings [20, 1] or analyzing high-resolution ODE models [33].

Codes. The codes used to generate the figures in this paper are available at https://github. com/jungbinkim1/Continuous-PEP .

Acknowledgments and Disclosure of Funding

This work was supported in part by the National Research Foundation of Korea funded by MSIT(2020R1C1C1009766), and the Information and Communications Technology Planning and Evaluation (IITP) grant funded by MSIT(2022-0-00124, 2022-0-00480).

References

Foivos Alimisis, Antonio Orvieto, Gary Becigneul, and Aurelien Lucchi. A continuous-time perspective for modeling acceleration in Riemannian optimization. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , pages 1297-1307, 2020.

Hedy Attouch, Zaki Chbani, Juan Peypouquet, and Patrick Redont. Fast convergence of inertial dynamics and algorithms with asymptotic vanishing viscosity. Mathematical Programming , 168(1):123-175, 2018.

Hedy Attouch, Zaki Chbani, and Hassan Riahi. Rate of convergence of the Nesterov accelerated gradient method in the subcritical case α ≤ 3 . ESAIM: Control, Optimisation and Calculus of Variations , 25:2, 2019.

Michael Betancourt, Michael I Jordan, and Ashia C Wilson. On symplectic optimization. arXiv preprint arXiv:1802.03653 , 2018.

Etienne De Klerk, François Glineur, and Adrien B Taylor. On the worst-case complexity of the gradient method with exact line search for smooth strongly convex functions. Optimization Letters , 11:1185-1199, 2017.

Jelena Diakonikolas and Michael I Jordan. Generalized momentum-based methods: A Hamiltonian perspective. SIAM Journal on Optimization , 31(1):915-944, 2021.

Yoel Drori and Adrien B Taylor. Efficient first-order methods for convex minimization: A constructive approach. Mathematical Programming , 184(1-2):183-220, 2020.

Yoel Drori and Marc Teboulle. Performance of first-order methods for smooth convex minimization: A novel approach. Mathematical Programming , 145(1):451-482, 2014.

Mathieu Even, Raphaël Berthier, Francis Bach, Nicolas Flammarion, Pierre Gaillard, Hadrien Hendrikx, Laurent Massoulié, and Adrien Taylor. Continuized accelerations of deterministic and stochastic gradient descents, and of gossip algorithms. In Advances in Neural Information Processing Systems , pages 28054-28066.

Mahyar Fazlyab, Alejandro Ribeiro, Manfred Morari, and Victor M Preciado. Analysis of optimization algorithms via integral quadratic constraints: Nonstrongly convex problems. SIAM Journal on Optimization , 28(3):2654-2689, 2018.

Guoyong Gu and Junfeng Yang. On the optimal ergodic sublinear convergence rate of the relaxed proximal point algorithm for variational inequalities. arXiv preprint arXiv:1905.06030 , 2019.

Guoyong Gu and Junfeng Yang. Optimal nonergodic sublinear convergence rate of proximal point algorithm for maximal monotone inclusion problems. arXiv preprint arXiv:1904.05495 , 2019.

Shuvomoy Das Gupta, Bart PG Van Parys, and Ernest K Ryu. Branch-and-bound performance estimation programming: A unified methodology for constructing optimal optimization methods. arXiv preprint arXiv:2203.07305 , 2022.

Bin Hu and Laurent Lessard. Dissipativity theory for Nesterov's accelerated method. In International Conference on Machine Learning , pages 1549-1557. PMLR, 2017.

Bin Hu, Peter Seiler, and Anders Rantzer. A unified analysis of stochastic optimization methods using jump system theory and quadratic constraints. In Conference on Learning Theory , pages 1157-1189. PMLR, 2017.

Donghwan Kim. Accelerated proximal point method for maximally monotone operators. Mathematical Programming , 190(1-2):57-87, 2021.

Donghwan Kim and Jeffrey A Fessler. Optimized first-order methods for smooth convex minimization. Mathematical programming , 159(1):81-107, 2016.

Donghwan Kim and Jeffrey A Fessler. Optimizing the efficiency of first-order methods for decreasing the gradient of smooth convex functions. Journal of optimization theory and applications , 188(1):192-219, 2021.

Jungbin Kim and Insoon Yang. Unifying Nesterov's accelerated gradient methods for convex and strongly convex objective functions. In International Conference on Machine Learning , pages 16897-16954. PMLR, 2023.

Walid Krichene, Alexandre Bayen, and Peter L Bartlett. Accelerated mirror descent in continuous and discrete time. In Advances in Neural Information Processing Systems , volume 28, 2015.

Laurent Lessard, Benjamin Recht, and Andrew Packard. Analysis and design of optimization algorithms via integral quadratic constraints. SIAM Journal on Optimization , 26(1):57-95, 2016.

Laurent Lessard and Peter Seiler. Direct synthesis of iterative algorithms with bounds on achievable worst-case convergence rate. In 2020 American Control Conference (ACC) , pages 119-125. IEEE, 2020.

Aleksandr Mikhailovich Lyapunov. The general problem of the stability of motion. International journal of control , 55(3):531-534, 1992.

Alexandre Megretski and Anders Rantzer. System analysis via integral quadratic constraints. IEEE transactions on automatic control , 42(6):819-830, 1997.

Céline Moucer, Adrien Taylor, and Francis Bach. A systematic approach to Lyapunov analyses of continuous-time models in convex optimization. SIAM Journal on Optimization , 33(3):15581586, 2023.

Yurii Nesterov. Lectures on Convex Optimization , volume 137. Springer, 2018.

Yurii E Nesterov. A method for solving the convex programming problem with convergence rate o (1 /k 2 ) . In Dokl. akad. nauk Sssr , volume 269, pages 543-547, 1983.

Frigyes Riesz and Béla Sz Nagy. Functional analysis . Courier Corporation, 2012.

W. Rudin. Real and Complex Analysis . McGraw-Hill, 1987.

W. Rudin. Functional Analysis . McGraw-Hill, 1991.

Jesús María Sanz Serna and Konstantinos C Zygalakis. The connections between Lyapunov functions for some optimization algorithms and differential equations. SIAM Journal on Numerical Analysis , 59(3):1542-1565, 2021.

Damien Scieur, Vincent Roulet, Francis Bach, and Alexandre d'Aspremont. Integration methods and optimization algorithms. Advances in Neural Information Processing Systems , 30, 2017.

Bin Shi, Simon S Du, Michael I Jordan, and Weijie J Su. Understanding the acceleration phenomenon via high-resolution differential equations. Mathematical Programming , pages 1-70, 2021.

Bin Shi, Simon S Du, Weijie Su, and Michael I Jordan. Acceleration via symplectic discretization of high-resolution differential equations. In Advances in Neural Information Processing Systems , volume 32, 2019.

Elias M Stein and Rami Shakarchi. Real Analysis: Measure Theory, Integration, and Hilbert Spaces . Princeton University Press, 2009.

Weijie Su, Stephen Boyd, and Emmanuel J Candes. A differential equation for modeling Nesterov's accelerated gradient method: Theory and insights. Journal of Machine Learning Research , 17:1-43, 2016.

Jaewook J Suh, Gyumin Roh, and Ernest K Ryu. Continuous-time analysis of accelerated gradient methods via conservation laws in dilated coordinate systems. In International Conference on Machine Learning , pages 20640-20667. PMLR, 2022.

Boya Sun, Jemin George, and Solmaz Kia. High-resolution modeling of the fastest first-order optimization method for strongly convex functions. In 2020 59th IEEE Conference on Decision and Control (CDC) , pages 4237-4242. IEEE, 2020.

Adrien Taylor and Francis Bach. Stochastic first-order methods: Non-asymptotic and computeraided analyses via potential functions. In Alina Beygelzimer and Daniel Hsu, editors, Proceedings of the Thirty-Second Conference on Learning Theory , volume 99 of Proceedings of Machine Learning Research , pages 2934-2992. PMLR, 25-28 Jun 2019.

Adrien Taylor and Yoel Drori. An optimal gradient method for smooth strongly convex minimization. Mathematical Programming , pages 1-38, 2022.

Adrien Taylor, Bryan Van Scoy, and Laurent Lessard. Lyapunov functions for first-order methods: Tight automated convergence guarantees. In International Conference on Machine Learning , pages 4897-4906. PMLR, 2018.

Adrien B. Taylor, Julien M. Hendrickx, and François Glineur. Smooth strongly convex interpolation and exact worst-case performance of first-order methods. Mathematical Programming , 161(1-2):307-345, May 2016.

Adrien B Taylor, Julien M Hendrickx, and François Glineur. Exact worst-case performance of first-order methods for composite convex optimization. SIAM Journal on Optimization , 27(3):1283-1313, 2017.

Adrien B Taylor, Julien M Hendrickx, and François Glineur. Exact worst-case convergence rates of the proximal gradient method for composite convex minimization. Journal of Optimization Theory and Applications , 178:455-476, 2018.

Manu Upadhyaya, Sebastian Banert, Adrien B Taylor, and Pontus Giselsson. Automated tight Lyapunov analysis for first-order methods. arXiv preprint arXiv:2302.06713 , 2023.

Bryan Van Scoy, Randy A Freeman, and Kevin M Lynch. The fastest known globally convergent first-order method for minimizing strongly convex functions. IEEE Control Systems Letters , 2(1):49-54, 2017.

Andre Wibisono, Ashia C Wilson, and Michael I Jordan. A variational perspective on accelerated methods in optimization. proceedings of the National Academy of Sciences , 113(47):E7351E7358, 2016.

Ashia C. Wilson, Ben Recht, and Michael I. Jordan. A Lyapunov analysis of accelerated methods in optimization. Journal of Machine Learning Research , 22(113):1-34, 2021.

Jingzhao Zhang, Aryan Mokhtari, Suvrit Sra, and Ali Jadbabaie. Direct Runge-Kutta discretization achieves acceleration. In Advances in neural information processing systems , volume 31, 2018.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe article presents a continuous-time optimization framework for convex objectives which streamlines the process of giving rate guarantees.  Beginning from (Exact PEP), a difficult looking infinite-dimensional optimization problem.  This is relaxed using convexity, and then recast in a dual form, wherein it suffices to verify the positive definiteness of a PEP kernel.  This mirrors the discrete PEP kernel of Drori and Teboulle, and the authors show that this continuous PEP kernel can in fact be derived as a certain limit of the discrete one.\n\nThe article then deploys its method on a variety of accelerated ODE methods (Nesterov methods, triple momentum, information-theoretic exact methods, and others).  It additionally considers a variety of standard metrics, as appropriate for different situations (function value convergence, norm of gradient convergence).  The rates given appear to be consistent to state of the art  continuous-time guarantees or mirroring discrete-time guarantees\n\nThe method, which is designed to avoid construction of Lyapunov functions, instead requires construction of a positive definite kernel, which is built from the choice of a Lagrange multiplier function $\\Lambda.$\n\nSTRENGTHS:\n1) The article is very clearly presented, with a large set of examples to illustrate the method.  This in particular important for illustrating the search for the magic $\\Lambda.$\n2) Certifying that the rate guarantee is correct, given the presented H-kernel (which can be computed directly from the ODE), and given the Lagrange multiplier is simple.  \n3) The article improves some existing rate guarantees and appears to reproduce many of the best-known guarantees, presumably found through more usual Lyapunov functional methods.  In this sense, the article identifies exactly the way in which these Lyapunov functional estimates are optimal (in the sense that the c-PEP guarantees produced this way follow (5) -- see Remark 1).\n4) The method is extended systematically to give convergence of gradients (section 4).\n\nWEAKNESSES:\n1) While the paper removes the need to produce a Lyapunov function, it introduces the need to produce the Lagrange multiplier function $\\Lambda$.  This does not appear to be systemized, and so it raises the obvious question to what extent it is necessary to learn the c-PEP machinery only to search for the $\\Lambda$ when one could just search from the beginning for the more intuitive Lyapunov function.\n\nQUESTIONS:\n1) Is there a systematic way that you produce the Lagrange multiplier function? If so, this should be more prominently displayed.\n2) The exact PEP is relaxed and them seems to play no further role.  Can you provide any insight in the degree to which this relaxation is sharp? (Are there examples where it is solvable?). Is there anything to say about Exact PEP at all, except for the relaxation?\n3) Can you reverse engineer the c-PEP guarantee to produce optimization methods?\n4) All the rate guarantees (for say the strongly convex case) look largely equivalent.  Is the best possible rate guarantee (say asymptotically) -- given a class of ODE methods with $a\\sqrt{\\mu}$ is second coefficient and $1$ its final coefficient $e^{-c(a)\\sqrt{\\mu}T}$?\n\nLIMITATIONS:\nThe article appropriately addresses its limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-21"}

comments: SUMMARY:
This paper proposes a novel methodology that analyzes ODE models for first-order optimization methods by converting the task of proving convergence rates into verifying the positive semidefiniteness of specific Hilbert-Schmidt integral operators. Based on the performance estimation problems (PEP) and functional analysis, the authors establish convergence rates of various accelerated gradient flow models.  The authors’ continuous time PEP framework provides insights into the analysis of the discrete-time PEP.

STRENGTHS:
1. The authors developed a novel and simple framework for analyzing the convergence rate of continuous-time dynamics via positive semidefinite kernels.

2. The authors bridge the gap between the PEP framework for the continuous and discrete settings from continuous-time dynamics.

3. the authors’ continuous time PEP framework provides new opportunities for the analysis of the discrete-time PEP.

WEAKNESSES:
1. The results of this paper lack experimental validation.

2. The verification of the positive semidefiniteness of an integral kernel is difficult in implementation. How do you solve the difficulty in the calculation for the integral kernel?

3. There is a misspelling in line 39.

QUESTIONS:
See the weakness

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a framework for analyzing convergence rates of a class of ODE models via the continuous-time performance estimation problem (PEP). The task of solving the PEP problem is relaxed into verifying the positive semidefiniteness of specific integral operators. The convergence rates of several accelarated gradient flow models were estabilished using the proposed method.\n\nSTRENGTHS:\nThe novelty of the paper lies in the approach of defining and solving the continuous-time PEP problem to analyze ODE models. Under change of variables and Lagrangian duality, it is sufficient to verify the positive definiteness of specific integral operators. Using this method, the authors recover the known convergence guarantee and reveal unknown convergence rate for accelarated gradient flow models. The paper is clearly written and well-organized.\n\nWEAKNESSES:\nThe intuition to consider the PEP kernel is not clear. The choices of multiplier function $\\Lambda$ and $\\nu$ are constructive and tricky for ODE models shown in section 3.3 (after Theorem 1) and section 4.2 (after Theorem 2). If given an ODE model as (4), it seems hard to distinguish if it can fulfill the assumptions in Theorem 1 and Theorem 2. \n\nLyapunov analysis for the discrete algorithms is much harder than the continuous dynamic system. The proposed approach only works for the continuous problems and didn't help the discrete case.\n\nQUESTIONS:\nThere are some questions for technical unclarity.\n\n- Does the paper consider only the strongly convex $f(x)$? If it is, then the minimizer $x^*$ is unique, I suggest the refer $x^*$ as 'the minimizer' rather than 'a minimizer' throughout the paper; if it is not, for convex $f(x)$, how to guarentee that the minimizer $x^*$ exists? For example, $f(x) = e^{-x}$. How to interpret the result of Theorem 1 in this case without defining $x^*$? \n\n- For Exact PEP formulation in line 105, why the initial condition is set as $\\dot X(0) =0$? If $x_0 = x^*$, then there is zero division issue, while this was not a problem in formulation (3). I suggest the authors adapt this case into the Exact PEP formulation. \n\n- The computation stated in line 132 is not obvious. I suggest the authors include the outline in the paper.\n\n- Suggest to outline the proofs of Theorem 1 and Theorem 2 in the paper.\n\n- Can the convergence guarentee in line 155 imply the convergence guarantee of the ODE model? In what sense? For instance, as a extreme case, when $f(x) = \\frac{\\mu}{2}\\|x - x^*\\|^2$, $\\tilde f(x) - \\tilde f(x^*)$ is always zero, then Line 155 becomes trivial inequality.\n\n- Is it possible that the supremum in the inequalities (15) is infinity and the inequality is trivial? For example, $\\mu =  0$ and $f(x)$ is not bounded below.\n\n- What is $x^T$ in equation (20) and (21)?\n\n- Typos: line 45, 'enhances'; page 5, footnote 3 line 2, 'space'.\n\n- Please check the format of the reference list, particularly the letter case (in [25], [27], [30], etc).\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a new technique (that is fundamentally different from the Lyapunov approach) for systemically analyzing the convergence rates of ODE models for first-order optimization methods, which reduces to verifying the positive semidefiniteness of specific Hilbert-Schmidt integral operators. This is a continuous time version of the Drori and Teboulle's (discrete) performance estimation problem (PEP) approach, which has recently become a fundamental tool in systemically analyzing the convergence rates of first-order methods. As a verification, the authors utilized the proposed tool for analyzing various ODE models of accelerated first-order methods such as (unified) AGM, TMM, ITEM, AGM-G and OGM-G, which either recovered the existing convergence rates or revealed new rates for various ODE models.\n\nSTRENGTHS:\n- This provides a new systematic analysis of convergence rates of the ODE models of first-order methods, which will be potentially useful for analyzing new ODE models. The authors have successfully verified the effectiveness of the tool for various ODE models.\n\n- This continuous PEP resembles the discrete PEP, but its construction via the functional analysis is certainly not trivial and is very interesting.\n\n- This provides new insights to the PEP analysis.\n\nWEAKNESSES:\n- One of the important features of the discrete PEP is that one can numerically find the values of Lagrange multipliers by numerically optimizing them, which can be used to reveal an analytical form of Lagrange multipliers. However, this paper's continuous PEP requires one to explicitly have the appropriate Lagrange multipliers, which can be laborious especially when dealing with new ODE models.\n\nQUESTIONS:\n- Line 132: How about letting the readers know that the derivation can be found in Appendix B?\n- Line 180: Is this rate consistent with the rate in [9]?\n- Appendix Line 75: $K^1$? $K^d$?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nContinuous counterpart of the work by Drori and Teboulle [2] was presented. Specifically, through the dual objective of the relaxed PEP in continuous time, the convergence rates of various ODEs were obtained. The analysis on the Lagrangian dual will lead to a dual solution based on a symmetric kernel of the Hilbert-Schmidt integral operator (Theorem 1).  Some of the follow up results were new to the literature e.g., low-resolution ODE (as opposed to high-resolution ODEs proposed by Shi et al. [19]) for the TM, ITEM methods & velocity/gradient norm convergence rates (Theorem 2).\n\nSTRENGTHS:\n1- The work has a solid mathematical base. The proofs are well written and mostly easy to follow. \n\n2- The rate for TM ODE and ITEM ODE matching their discretized algorithms.\n\n3- Investigating the connection between the continuous-time PEP and its discrete counterpart in Appendix G.2\n\nWEAKNESSES:\n1- The future directions are not clearly specified or very vague. This will impact the contribution of the work to the Neurips community. In the \"Questions\", I have asked some questions (numbers 2,3,4,5) which can be as future directions or even increase the contribution of this work beyond the current status.\n\n2- The presentation can improve e.g. \n\n-In (Relaxed PEP): move subject to under max for better readability. \n\n-In 145: Sinse is a typo.\n\n-References 22 23 are the same.\n\n-Appendix: 455 missed parenthesis for 70.\n\n\n3- Limited literature review. More comprehensive literature review could be placed in the Appnedix to save space.\n\n4- Inconsistency in some parts of the text; e.g., in 113 I expected to see the inequalities mentioned in 112, but I was noted with another representation of AGM ODE from [9]. Also, sudden shifts from convex cases to strongly convex ones can cause confusion.\n\n5- Some parts could be moved to the appendix (e.g. recovering the famous known rates for ODEs) and instead report their results in a table. Then, the authors could use the additional space to draw conclusions, compare (like graphs and visual simulations for instant comparison), or even exploring other aspects of the proposed framework or ODEs (by e.g. answering some of the questions in the \"Questions\" section below).\n\nQUESTIONS:\n1- Is the statement in 33[However ...]36 true? There has been a line of work which used positive-semidefiniteness to define tight Lyapunov functions (see [1]). This work however deals with a more general case which entails various ODEs through the choice of the kernel.\n\n \n\n2- Following Q1, is it possible to use your framework to define tight Lyapunov functions? This relates to finding the analytical Lyapunov functions simulated in e.g. [2].\n\n \n\n3- Is it possible to extend this work to the high-resolution ODE framework proposed by Shi to bypass the Lyapunov based proofs in their analysis?\n\n \n\n4- Is there any connection between the low-resolution ODE for TMM proposed in this work and the high-resolution ODE proposed in [3]?\n\n \n\n5- In authors' mind, what discretizations can recover the TMM and ITEM from the proposed TMM  and ITEM ODEs?\n\n\nReferences\n[1] Sanz-Serna, Jesús María and Konstantinos C. Zygalakis. “The connections between Lyapunov functions for some optimization algorithms and differential equations.” SIAM J. Numer. Anal. 59 (2020): 1542-1565.\n\n[2] Upadhyaya, M., Banert, S., Taylor, A.B., & Giselsson, P. (2023). Automated tight Lyapunov analysis for first-order methods.\n\n[3] B. Sun, J. George and S. Kia, \"High-Resolution Modeling of the Fastest First-Order Optimization Method for Strongly Convex Functions,\" 2020 59th IEEE Conference on Decision and Control (CDC), Jeju, Korea (South), 2020, pp. 4237-4242, doi: 10.1109/CDC42340.2020.9304444.\n\nLIMITATIONS:\nThe limitations are stated in the text.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-17"}