Title: A Unified Discretization Framework for Differential Equation Approach with Lyapunov Arguments for Convex Optimization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Convex optimization

Numerical analysis

Ordinary differential equations

Convergence estimate
Abstract: The differential equation (DE) approach for convex optimization, which relates optimization methods to specific continuous DEs with rate-revealing Lyapunov functionals, has gained increasing interest since the seminal paper by Su--Boyd--Candès (2014).
However, the approach still lacks a crucial component to make it truly useful: there is no general, consistent way to transition back to discrete optimization methods. Consequently, even if we derive insights from continuous DEs, we still need to perform individualized and tedious calculations for the analysis of each method.
This paper aims to bridge this gap by introducing a new concept called  ``weak discrete gradient'' (wDG), which consolidates the conditions required for discrete versions of gradients in the DE approach arguments.
We then define abstract optimization methods using wDG and provide abstract convergence theories that parallel those in continuous DEs.
We demonstrate that many typical optimization methods and their convergence rates can be derived as special cases of this abstract theory.
The proposed unified discretization framework for the differential equation approach to convex optimization provides an easy environment for developing new optimization methods and achieving competitive convergence rates with state-of-the-art methods, such as Nesterov's accelerated gradient.

A Unified Discretization Framework for Differential Equation Approach with Lyapunov Arguments for Convex Optimization

Abstract

The differential equation (DE) approach for convex optimization, which relates optimization methods to specific continuous DEs with rate-revealing Lyapunov functionals, has gained increasing interest since the seminal paper by Su-BoydCandès (2014). However, the approach still lacks a crucial component to make it truly useful: there is no general, consistent way to transition back to discrete optimization methods. Consequently, even if we derive insights from continuous DEs, we still need to perform individualized and tedious calculations for the analysis of each method. This paper aims to bridge this gap by introducing a new concept called 'weak discrete gradient' (wDG), which consolidates the conditions required for discrete versions of gradients in the DE approach arguments. We then define abstract optimization methods using wDG and provide abstract convergence theories that parallel those in continuous DEs. We demonstrate that many typical optimization methods and their convergence rates can be derived as special cases of this abstract theory. The proposed unified discretization framework for the differential equation approach to convex optimization provides an easy environment for developing new optimization methods and achieving competitive convergence rates with state-of-the-art methods, such as Nesterov's accelerated gradient.

1 Introduction

In this paper, we consider unconstrained convex optimization problems:

Various optimization methods, such as standard gradient descent and Nesterov's accelerated gradient methods (Nesterov, 1983), are known for these problems. The convergence rates of these methods have been intensively investigated based on the classes of objective functions ( L -smooth and/or µ -strongly convex). We focus on the convergence rate of function values f ( x ( k ) ) -f ⋆ , while the rates for ∥∇ f ( x ( k ) ) ∥ or ∥ x ( k ) -x ⋆ ∥ have also been discussed. Topics particularly relevant to this study include the lower bound of convergence rates for first-order methods (see Remark 4.3 for the relationship between our framework and first-order methods) for convex and strongly convex functions: O ( 1 /k 2 ) for L -smooth and convex functions (cf. Nesterov (2018)) and O ( (1 -√ µ/L ) 2 k ) for L -smooth and µ -strongly convex functions (Drori and Taylor, 2022). These lower bounds are tight, as they are

achieved by some optimization methods, such as Nesterov (1983) for convex functions and Van Scoy et al. (2018); Taylor and Drori (2022a) for strongly convex functions. In these studies, the discussion is typically conducted for each method, utilizing various techniques accumulated in the optimization research field.

Whereas, it has long been known that some optimization methods can be related to continuous differential equations (DEs). Early works on this aspect include the following: the continuous gradient flow ˙ x = -∇ f ( x ) as a continuous optimization method was discussed in Bruck (1975). Similar arguments were later applied to second-order differential equations (Alvarez, 2000; Alvarez et al., 2002; Cabot et al., 2009). An important milestone in this direction was Su et al. (2014), where it was shown that Nesterov's famous accelerated gradient method (Nesterov, 1983) could be related to a second-order system with a convergence rate-revealing 'Lyapunov functional.' The insights gained from this relationship have been useful in understanding the behavior of the Nesterov method and in considering its new variants. This success has followed by many studies, including Wilson (2018); Wilson et al. (2021). The advantage of the DEs with Lyapunov functional approach (which we simply call the 'DE approach' hereafter) is that the continuous DEs are generally more intuitive, and convergence rate estimates are quite straightforward thanks to the Lyapunov functionals. However, the DE approach still lacks one important component; although we can draw useful insights from continuous DEs, there is no known general way to translate them into a discrete setting. Consequently, we still need to perform complex discrete arguments for each method. This limitation was already acknowledged in Su et al. (2014): '... The translation, however, involves parameter tuning and tedious calculations. This is the reason why a general theory mapping properties of ODEs into corresponding properties for discrete updates would be a welcome advance.'

In this paper we attempt to provide this missing piece by incorporating the concept of 'discrete gradients' (DGs) from numerical analysis, which is used to replicate some properties of continuous DEs in discrete settings. We demonstrate that a relaxed concept of DG, which we call 'weak discrete gradient' (wDG), can serve a similar purpose in the optimization context. More precisely, we show that for known DEs in the DE approach, if we define abstract optimization methods using wDGs analogously to the DEs, their abstract convergence theories can be obtained by following the continuous arguments and replacing gradients with wDGs. The tedious parts of the case-specific discrete arguments are consolidated in the definition of wDG, which simplifies the overall arguments: we can now consider 'simple continuous DE arguments' and 'case-specific discrete discussions summarized in wDG' separately. We demonstrate that many typical existing optimization methods and their rate estimates, previously done separately for each method, can be recovered as special cases of the abstract methods/theories, providing a simpler view of them. Any untested combination of a known DE and wDG presents an obvious new method and its rate, further expanding the potential for innovation in the optimization field. Creating a new wDG leads to a series of optimization methods by applying it to known DEs. One simply needs to verify if the wDG satisfies the conditions for wDG (Theorem 4.2) and reveal the constants of the wDG. If, in the future, a new DE with a rate-revealing Lyapunov functional is discovered, it should be possible to achieve similar results. We suggest first defining an abstract wDG method analogous to the DE and then examining whether the continuous theory can be translated to a discrete setting, as demonstrated in this paper.

The aforementioned paper (Su et al., 2014) concludes in the following way (continued from the previous quote) 'Indeed, this would allow researchers to only study the simpler and more user-friendly ODEs.' Although there is still room for minor adjustments (see the discussion on limitations below), we believe the wDG framework substantially reduces the complexity of discussions in discrete methods, allowing researchers to focus on more accessible and intuitive aspects of optimization.

We consider the problems (1) on the d -dimensional Euclidean space R d ( d is a positive integer) with the standard inner product ⟨· , ·⟩ and the induced norm ∥·∥ , where f : R d → R represents a differentiable convex objective function. We assume the existence of the optimal value f ⋆ and the optimal solution x ⋆ . In the following discussion, we use the inequality

which holds for any x, y ∈ R d when f is µ -strongly convex and differentiable.

Remark 1.1 . Although the scope of this paper is limitted due to the restriction of space, the framework can be naturally extended to more general cases. Extension to the objective functions satisfying the Polyak-Łojasiewicz (PŁ) condition is provided in Appendix H. The framework can be extended to

constrained optimizations by the DE approach for mirror descent methods (cf. Krichene et al. (2015); Wilson et al. (2021)), which the authors have already confirmed. Stochastic methods such as the stochastic gradient descent can be handled by considering random compositions of wDGs, which is left as our future work.

2 Short summary of the differential equation approach

Let us first consider the gradient flow:

It is easy to see that

This means the flow can be regarded as a continuous optimization method. Notice that the proof is quite simple, once we admit the chain rule of differentiation , and the form of the flow itself (3); this will be quite important in the subsequent discussion.

Despite its simplicity, the convergence rate varies depending on the class of objective functions. Below we show some known results. The following rates are proven using the so-called Lyapunov argument, which introduces a 'Lyapunov functional' that explicitly contains the convergence rate. The proof is left to Appendix B (we only note here that, in addition to the two key tools the chain rule and the form of the flow we need the convexity inequality (2) to complete the proof.)

Theorem 2.1 (Convex case) . Suppose that f is convex. Let x : [0 , ∞ ) → R d be the solution of the gradient flow (3) . Then the solution satisfies

Theorem 2.2 (Strongly convex case) . Suppose that f is µ -strongly convex. Let x : [0 , ∞ ) → R d be the solution of the gradient flow (3) . Then the solution satisfies

An incomplete partial list of works using the Lyapunov approach includes, in addition to Su et al. (2014), Karimi and Vavasis (2016); Attouch et al. (2016); Attouch and Cabot (2017); Attouch et al. (2018); França et al. (2018); Defazio (2019); Shi et al. (2019); Wilson et al. (2021) (see also a comprehensive list in Suh et al. (2022)). A difficulty in this approach is that the Lyapunov functionals were found only heuristically. A remedy is provided in Suh et al. (2022); Du (2022), but its target is still limited.

Next, we consider DEs corresponding to accelerated gradient methods, including Nesterov's method. As is well known, the forms of accelerated gradient methods differ depending on the class of objective functions, and consequently, the DEs to be considered also change. In this paper, we call them accelerated gradient flows .

When the objective functions are convex, we consider the following DE proposed in Wilson et al. (2021): let A : R ≥ 0 → R ≥ 0 be a differentiable strictly monotonically increasing function with A (0) = 0 , and

with ( x (0) , v (0)) = ( x 0 , v 0 ) ∈ R d × R d .

Theorem 2.3 (Convex case (Wilson et al. (2021))) . Suppose that f is convex. Let ( x, v ): [0 , ∞ ) → R d × R d be the solution of the DE (5) . Then it satisfies

Remark 2.4 . If we set A ( t ) = t 2 , this system coincides with a continuous limit DE of the accelerated gradient method for convex functions

which is derived in Su et al. (2016).

Remark 2.5 . From Theorem 2.3, it might seem that we can achieve an arbitrarily high order rate. Although it is surely true in the continuous context, it does not imply we can construct discrete optimization methods from the ODE. In fact, greedily demanding a higher rate is penalized at the timing of discretization from the numerical stability. See, for example, the discussion in Ushiyama et al. (2022b).

Next, for strongly convex objective functions, let us consider the DE (again in Wilson et al. (2021)):

with ( x (0) , v (0)) = ( x 0 , v 0 ) ∈ R d × R d . (Note that this system coincides with the continuous limit ODE of the accelerated gradient method for strongly convex functions by Polyak (1964): ¨ x +2 √ µ ˙ x + ∇ f ( x ) = 0 .)

Theorem 2.6 (Strongly convex case (Wilson et al. (2021); Luo and Chen (2022))) . Suppose that f is µ -strongly convex. Let ( x, v ): [0 , ∞ ) → R d × R d be the solution of (6) . Then it satisfies

3 Discrete gradient method for gradient flows (from numerical analysis)

The remaining issue is how we discretize the above DEs. In the optimization context, it was done separately in each study. One tempting strategy for a more systematic discretization is to import the concept of 'DG,' which was invented in numerical analysis for designing structure-preserving numerical methods for gradient flows such as (3) (Gonzalez (1996); McLachlan et al. (1999)). Recall that the automatic decrease of objective function came from the two keys: (a) the chain rule, and (b) the gradient flow structure. The DG method respects and tries to imitate them in discrete settings.

Definition 3.1 (Discrete gradient (Gonzalez (1996); Quispel and Capel (1996))) . A continuous map ∇ d f : R d × R d → R d is said to be discrete gradient of f if the following two conditions hold for all x, y ∈ R d :

In the definition provided above, the second condition simply requires that ∇ d f approximates ∇ f . On the contrary, the first condition, referred to as the discrete chain rule , is a critical requirement for the key (a). The discrete chain rule is a scalar equality constraint on the vector-valued function, and for any given f , there are generally infinitely many DGs. The following is a list of some popular choices of DGs. When it is necessary to differentiate them from wDGs, we call them strict DGs .

Proposition 3.2 (Strict discrete gradients) . The following functions are strict DGs. Gonzalez discrete gradient ∇ G f ( y, x ) (Gonzalez (1996)):

Itoh-Abe discrete gradient ∇ IA f ( y, x ) (Itoh and Abe (1988)):

Average vector field (AVF) ∇ AVF f ( y, x ) (Quispel and McLaren (2008)):

Suppose we have a DG for a given f . Then we can define a discrete scheme for the gradient flow (3):

where the positive real number h is referred to as the step size, and x ( k ) ≃ x ( kh ) is the numerical solution. The left-hand side approximates ˙ x and is denoted by δ + x ( k ) hereafter. Note that the definition conforms to the key point (b) mentioned earlier.

The scheme decreases f ( x ( k ) ) as expected:

In the first equality we used the discrete chain rule, and in the second, the form of the scheme itself. Observe that the proof proceeds in the same manner as the continuous case (4). Due to the decreasing property, the scheme should work as an optimization method. Additionally, the above argument does not reply on the step size h , and it can be changed in every step (which will not destroy the decreasing property).

In the numerical analysis community, the above approach has already been attempted for optimizations (Grimm et al. (2017); Celledoni et al. (2018); Ehrhardt et al. (2018); Miyatake et al. (2018); Ringholm et al. (2018); Benning et al. (2020); Riis et al. (2022)). Although they were successful on their own, this does not immediately provide the missing piece we seek for the following reasons. First, the DG framework does not include typical important optimization methods; it even does not include the steepest descent. Second, as noted above, the proofs of rate estimates in the continuous DEs (in Section 2) require the inequality of convexity (2). Unfortunately, however, existing DGs generally do not satisfy it; see Appendix C for a counterexample. Next, we show how to overcome these difficulties.

Remark 3.3 . Some members of the optimization community may find the use of DGs peculiar, since it involves referring to two solutions x ( k +1) , x ( k ) . However, in some sense, it is quite natural because the decrease of f occurs in a single step x ( k ) ↦→ x ( k +1) . There may also be concerns about the computational complexity of DGs because the method becomes 'implicit' by referring to x ( k +1) . In the field of structure-preserving numerical methods, however, it is widely known that in some highly unstable DEs, implicit methods are often advantageous, allowing larger time-stepping widths, while explicit methods require extremely small ones. In fact, it has been confirmed in Ehrhardt et al. (2018) that this also applies to the optimization context. The Itoh-Abe DG results in a system of d nonlinear equations, which is slightly less expensive. Moreover, note that the integral in the A VF can be evaluated analytically before implementation, when f is a polynomial.

4 Weak discrete gradients and abstract optimization methods

We introduce the concept of a weak discrete gradient (wDG), which is a relaxed version of the DG introduced earlier.

Definition 4.1 (Weak discrete gradient) . A gradient approximation 1 ∇ f : R d × R d → R d is said to be weak discrete gradient of f if there exists α ≥ 0 and β, γ with β + γ ≥ 0 such that the following two conditions hold for all x, y, z ∈ R d :

Note that (8) can be regarded as a modification of the three points descent lemma, where the third variable z is utilized to give some estimates. The freedom in variable z in (8) is fully utilized also in this paper; see Theorems 5.4 and 5.5 and their proofs.

The condition (8) can be interpreted in two ways. First, it can be understood as a discrete chain rule in a weaker sense. By substituting x with z , we obtain the inequality:

Compared to the strict discrete chain rule (7), it is weaker because it is an inequality and allows an error term. Second, it can be interpreted as a weaker discrete convex inequality. By exchanging x and y and rearranging terms, we obtain another expression

Compared to the strongly convex inequality (2), the term ( µ/ 2) ∥ y -x ∥ 2 is now replaced with β ∥ y -x ∥ 2 + γ ∥ y -z ∥ 2 , which can be interpreted as the squared distance between y and the point ( x, z ) where the gradient is evaluated. The term -α ∥ x -z ∥ 2 is an error term.

We now list some examples of wDGs (proof is provided in Appendix D). Notice that these examples include various typical gradient approximations from the optimization and numerical analysis literature. Note that for ease of presentation, we simply write ' µ -strongly convex function,' which includes convex functions by setting µ = 0 .

Theorem 4.2. Suppose that f : R d → R is a µ -strongly convex function. Let (L) and (SC) denote the additional assumptions: (L) f is L -smooth, and (SC) µ > 0 . Then, the following functions are wDGs:

(i) If ∇ f ( y, x ) = ∇ f ( x ) and f satisfies (L) , then ( α, β, γ ) = ( L/ 2 , µ/ 2 , 0) .

(ii) If ∇ f ( y, x ) = ∇ f ( y ) , then ( α, β, γ ) = (0 , 0 , µ/ 2) .

(iii) If ∇ f ( y, x ) = ∇ f ( x + y 2 ) and f satisfies (L) , then ( α, β, γ ) = (( L + µ ) / 8 , µ/ 4 , µ/ 4) .

(iv) If ∇ f ( y, x ) = ∇ AVF f ( y, x ) and f satisfies (L) , then ( α, β, γ ) = ( L/ 6 + µ/ 12 , µ/ 4 , µ/ 4) .

(v) If ∇ f ( y, x ) = ∇ G f ( y, x ) and f satisfies (L)(SC) , then ( α, β, γ ) = (( L + µ ) / 8 + ( L -µ ) 2 / 16 µ, µ/ 4 , 0) .

(vi) If ∇ f ( y, x ) = ∇ IA f ( y, x ) and f satisfies (L)(SC) , then ( α, β, γ ) = ( dL 2 /µ -µ/ 4 , µ/ 2 , -µ/ 4) .

Although we assumed the smoothness of f to simplify the presentation, the case (ii) does not demand it (see the end of Appendix D). Thus, it can handle non-smooth convex optimization. While the wDGs (i), (iii), (iv) only require (L), the wDGs (v) and (vi) demand (SC) ( µ > 0 ). This implies that the latter wDGs might be fragile for small µ 's.

We now define an abstract method using wDGs:

which is analogous to the gradient flow (3). By 'abstract,' we mean that it is a formal formula, and given a concrete wDG it reduces to a concrete method; see Table 1 which summarizes some typical choices. Observe that the abstract method covers many popular methods from both optimization and numerical analysis communities. The step size h may be selected using line search techniques, but for simplicity, we limit our presentation to the fixed step size in this paper (see Remark 5.1).

Remark 4.3 . Note that some wDGs are not directly connected to the original gradient ∇ f 's; the Itoh-Abe wDG (vi) does not even refer to the gradient. Thus, the concrete methods resulting from our framework do not necessarily fall into the so-called 'first-order methods,' which run in a linear space spanned by the past gradients (Nesterov (1983)). This is why we use the terminology 'gradient-based methods' in this paper, instead of first-order methods.

Similar to the aforementioned, we can define abstract methods for (5) and (6). Details and theoretical results can be found in Theorems 5.4 and 5.5.

We also introduce the next lemma, which is useful in expanding the scope of our framework.

Lemma 4.4. Suppose f can be expressed as a sum of two functions f 1 , f 2 . If ∇ 1 f 1 and ∇ 2 f 2 are wDGs of f 1 and f 2 with parameters ( α 1 , β 1 , γ 1 ) and ( α 2 , β 2 , γ 2 ) , respectively, then ∇ 1 f 1 + ∇ 2 f 2 is a weak discrete gradient of f with ( α, β, γ ) = ( α 1 + α 2 , β 1 + β 2 , γ 1 + γ 2 ) .

This lemma allows us to consider the following discretization of the gradient flow:

within our framework. For instance, if f 1 is L 1 -smooth and µ 1 -strongly convex, and f 2 is µ 2 -strongly convex, then the right-hand side of (12) is a wDG with ( α, β, γ ) = ( L 1 / 2 , µ 1 / 2 , µ 2 / 2) . In this case, the method is known as the proximal gradient method or the forward-backward splitting algorithm in optimization (cf. Bauschke and Combettes (2017)). Discretizing the accelerated gradient flows allows for obtaining accelerated versions. (Acceleration of the proximal gradient method has been studied for some time (Beck and Teboulle, 2009b,a).)

5 Convergence rates of abstract optimization methods

We establish the discrete counterparts of Theorems 2.1 to 2.3 and 2.6. Although the proofs are left to Appendix E, we emphasize that they can be performed analogously to those of the continuous cases. The discrete theorems are established in four cases: the gradient flow (3) (for f convex and µ -strongly convex), and the accelerated flows (for f convex (5) and µ -strongly convex (6)). For ease of understanding, we summarize the results for µ -strongly convex cases in Table 1. A similar table for convex cases is included in Appendix A.

Remark 5.1 . The following theorems are presented under the assumption that the step size h is fixed for simplicity. However, if all varying step sizes satisfy the step size condition (with a finite number of violations allowed), the theorems still hold true. The step sizes must be bounded by a positive number from below to ensure the designated rates.)

5.1 For the abstract method based on the gradient flow

The abstract method is given in (11).

Theorem 5.2 (Convex case) . Let f be a convex function. Let ∇ f be a wDG of f , and suppose that f also satisfies the necessary conditions required by the wDG. Suppose that in the wDG β ≥ 0 , γ ≥ 0 . Let { x ( k ) } be the sequence given by (11) . Then, under the step size condition h ≤ 1 / (2 α ) , the sequence satisfies

Let us demonstrate how to use the theorem using the proximal gradient method (12) as an example. Suppose that f 1 is L 1 -smooth and convex, and f 2 is convex. Then, ∇ f ( y, x ) = ∇ f 1 ( x ) + ∇ f 2 ( y ) is a wDG with the parameter ( α, β, γ ) = ( L 1 / 2 , 0 , 0) due to Theorem 4.2 and Lemma 4.4. Therefore,

the proximal gradient method (12) satisfies the assumption of Theorem 5.2 and thus the convergence rate is O(1 /k ) under the step size condition h ≤ (1 /L 1 ) .

Theorem 5.3 (Strongly convex case) . Let f be a strongly convex function. Let ∇ f be a wDG of f , and suppose that f also satisfies the necessary conditions required by the wDG. Suppose that in the wDG β + γ > 0 . Let { x ( k ) } be the sequence given by (11) . Then, under the step size condition h ≤ 1 / ( α + β ) , the sequence satisfies

In particular, the sequence satisfies

when the optimal step size h = 1 / ( α + β ) is employed.

5.2 For the abstract methods based on the accelerated gradient flows

We consider abstract methods with wDGs based on the accelerated gradient flows (5) and (6), which will be embedded in the theorems below. We note one thing: when using (8) as an approximation of the chain rule, we can determine z independently of x and y , which gives us some degrees of freedom (thus allowing for adjustment.) Below we show some choices of z ( k ) that are easy to calculate from known values while keeping the decrease of the Lyapunov functional.

Theorem 5.4 (Convex case) . Let f be a convex function. Let ∇ f be a wDG of f , and suppose that f also satisfies the necessary conditions required by the wDG. Suppose that in the wDG β ≥ 0 , γ ≥ 0 . Let {( x ( k ) , v ( k ) )} be the sequence given by

with ( x (0) , v (0) ) = ( x 0 , v 0 ) , where A k := A ( kh ) . Then if A k = ( kh ) 2 and h ≤ 1 / √ 2 α , the sequence satisfies

Theorem 5.5 (Strongly convex case) . Let f be a strongly convex function. Let ∇ f be a wDG of f , and suppose that f also satisfies the necessary conditions required by the wDG. Suppose that in the wDG β + γ > 0 . Let {( x ( k ) , v ( k ) )} be the sequence given by

with ( x (0) , v (0) ) = ( x 0 , v 0 ) . Then if h ≤ h := ( 2( α + γ β + γ ) ) , the sequence satisfies

In particular, the sequence satisfies

when the optimal step size h = h is employed.

Remark 5.6 . Time scaling can eliminate the factor √ 2( β + γ ) from the scheme and simplify it, as shown in Luo and Chen (2022). However, we do not use time scaling here to match the time scale with the accelerated gradient method and to maintain correspondence with the continuous system.

6 Discussions including Limitations

Relation to some other systematic/unified frameworks A systematic approach to obtaining optimization methods with convergence estimates was developed using the 'performance estimation problems" (PEPs) technique, as seen in works such as Taylor et al. (2018); Taylor and Drori (2022b). While our framework unifies discussions in both continuous and discrete settings, the design of methods is not automatic and requires finding a new wDG. In contrast, the PEP framework automates method design but separates discussions between the continuous and discrete settings. Combining these two approaches could be a promising research direction, such as applying our framework to the Lyapunov functionals obtained in Taylor and Drori (2022b).

Another unified convergence analysis is presented in Chen and Luo (2021), where the authors cite the same passage in the Introduction from Su et al. (2016). However, this seems to focus on unifying discussions in the continuous setting, and it is still necessary to individualize the discretization for each method in the discrete setting.

In Diakonikolas and Orecchia (2019), a unified method for deriving continuous DEs describing first-order optimization methods was proposed using the 'approximate duality gap technique.' While this work is capable of finding new DEs, it does not provide insight into how to discretize the DEs for obtaining discrete optimization methods.

Another closely related work is De Sa et al. (2022), which proposed a framework to construct Lyapunov functionals for continuous ODEs. This is strong in view of the fact that generally Lyapunov functionals can be found only in ad hoc ways. Instead, they considered only the simplest gradient descent (and its stochastic version), while the main focus of the present paper lies in the discretizations.

As said in Section 3, in the field of numerical analysis, the use of discrete gradients has been tried. Among them, Ehrhardt et al. (2018) is a pioneering work that comes with several theoretical results. Both this and the present work aim at convex, strongly convex, and the PŁ functions (in the Appendix, in the present paper). The scope of Ehrhardt et al. (2018) was limited in the sense that they considered only discretizations of gradient flows with strict discrete gradients. Our target ODEs and discretizations are not limited to that, but as its price, our rate is worse in some strict discrete gradient discretizations of gradient flow. This comes from the difference in proof techniques: they proved convergence rates directly and algebraically, while our analysis is via Lyapunov functionals. They also gave several theoretical results besides the convergence analysis, such as the (unique) existence of solutions, and step-size analysis which are important in actual implementations. Whether these two frameworks could be unified would be an interesting future research topic.

Limitations of the proposed framework. Although we believe that the current framework provides an easy environment for working on the DE approach to optimization, it still has some limitations.

First, methods that do not fall into the current framework exist, such as the following splitting method (cf. the Douglas-Rachford splitting method (Eckstein and Bertsekas, 1992)):

The right-hand side cannot be written by a single wDG. Additionally, methods based on Runge-Kutta (RK) numerical methods (Zhang et al. (2018); Ushiyama et al. (2022a)) appear difficult to be captured by wDG because RK methods cannot be expressed by DG in the first place. Investigating whether these methods can be captured by the concept of DG is an interesting future research topic.

Second, there is still some room for adjustment in wDG methods. A typical example is z ( k ) in Section 5.2, which is chosen in the theorems to optimize efficiency and rates. Another example is the adjustment of time-stepping in the last phase of constructing a method to achieve a better rate or practical efficiency. Although these optimizations in the construction of optimization methods are standard in optimization studies, we feel that they are difficult to capture in the current framework, as they fall between the intuitive continuous argument and the discrete wDG arguments that aim to capture common structures.

Third, some rates in the theorems are not optimal. For example, on strongly convex functions, the scheme proposed in Theorem 5.5 with the choice (i) achieves the convergence rate of O ( (1 -√ µ/L ) k ) , which is not the optimal rate of O ( (1 -√ µ/L ) 2 k ) . This is because the choice of the DE and Lyapunov functional used in this work is not optimal. A DE and Lyapunov functional for obtaining the optimal rate are known (Sun et al., 2020), but the DE is a so-called high-resolution DE (known as a 'modified equation' in numerical analysis), which involves the Hessian. Whether these DEs can be captured with the wDG perspective is an interesting future research topic.

7 Concluding remaks

In this paper, we proposed a new unified discretization framework for the DE approach to convex optimization. Our framework provides an easy environment for those working on the DE approach, and some new methods are immediate from the framework, both as methods and for their convergence estimates. For example, any combination of strict DGs with the accelerated gradient flows are new methods, and their rates are given by the theorems. Although we did not include numerical experiments in the main body owing to the space restrictions, some preliminary numerical tests confirming the theory can be found in Appendix I. These tests show that some new methods can be competitive with state-of-the-art methods, such as Nesterov's accelerated gradient.

References

Alvarez, F. (2000). On the minimizing property of a second order dissipative system in Hilbert spaces. SIAM J. Control Optim. , 38(4):1102-1119.

Alvarez, F., Attouch, H., Bolte, J., and Redont, P. (2002). A second-order gradient-like dissipative dynamical system with Hessian-driven damping. Application to optimization and mechanics. J. Math. Pures Appl. (9) , 81(8):747-779.

Attouch, H. and Cabot, A. (2017). Asymptotic stabilization of inertial gradient dynamics with time-dependent viscosity. J. Differential Equations , 263(9):5412-5458.

Attouch, H., Chbani, Z., and Riahi, H. (2018). Combining fast inertial dynamics for convex optimization with Tikhonov regularization. J. Math. Anal. Appl. , 457(2):1065-1094.

Attouch, H., Peypouquet, J., and Redont, P. (2016). Fast convex optimization via inertial dynamics with Hessian driven damping. J. Differential Equations , 261(10):5734-5783.

Bauschke, H. H. and Combettes, P. L. (2017). Convex analysis and monotone operator theory in Hilbert spaces . CMS Books in Mathematics/Ouvrages de Mathématiques de la SMC. Springer, Cham, second edition.

Beck, A. and Teboulle, M. (2009a). Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems. IEEE Trans. Image Process. , 18(11):2419-2434.

Beck, A. and Teboulle, M. (2009b). A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. , 2(1):183-202.

Benning, M., Riis, E. S., and Schönlieb, C.-B. (2020). Bregman Itoh-Abe methods for sparse optimisation. J. Math. Imaging Vision , 62(6-7):842-857.

Bruck, Jr., R. E. (1975). Asymptotic convergence of nonlinear contraction semigroups in Hilbert space. J. Functional Analysis , 18:15-26.

Cabot, A., Engler, H., and Gadat, S. (2009). On the long time behavior of second order differential equations with asymptotically small dissipation. Trans. Amer. Math. Soc. , 361(11):5983-6017.

Celledoni, E., Eidnes, S., Owren, B., and Ringholm, T. (2018). Dissipative numerical schemes on Riemannian manifolds with applications to gradient flows. SIAM J. Sci. Comput. , 40(6):A3789A3806.

Chen, L. and Luo, H. (2021). A unified convergence analysis of first order convex optimization methods via strong lyapunov functions. arXiv preprint , arXiv:2108.00132.

De Sa, C. M., Kale, S., Lee, J. D., Sekhari, A., and Sridharan, K. (2022). From gradient flow on population loss to learning with stochastic gradient descent. In Advances in Neural Information Processing Systems , volume 35, pages 30963-30976.

Defazio, A. (2019). On the curved geometry of accelerated optimization. In Advances in Neural Information Processing Systems , volume 32.

Diakonikolas, J. and Orecchia, L. (2019). The approximate duality gap technique: a unified theory of first-order methods. SIAM J. Optim. , 29(1):660-689.

Drori, Y. and Taylor, A. (2022). On the oracle complexity of smooth strongly convex minimization. J. Complexity , 68:101590.

Du, D. (2022). Lyapunov function approach for approximation algorithm design and analysis: with applications in submodular maximization. arXiv preprint , arXiv:2205.12442.

Eckstein, J. and Bertsekas, D. P. (1992). On the Douglas-Rachford splitting method and the proximal point algorithm for maximal monotone operators. Math. Programming , 55(3, Ser. A):293-318.

Ehrhardt, M. J., Riis, E. S., Ringholm, T., and Schönlieb, C.-B. (2018). A geometric integration approach to smooth optimisation: Foundations of the discrete gradient method. arXiv preprint , arXiv:1805.06444.

França, G., Robinson, D., and Vidal, R. (2018). ADMM and accelerated ADMM as continuous dynamical systems. In Proceedings of the 35th International Conference on Machine Learning , pages 1559-1567.

Gonzalez, O. (1996). Time integration and discrete Hamiltonian systems. J. Nonlinear Sci. , 6(5):449467.

Grimm, V., McLachlan, R. I., McLaren, D. I., Quispel, G. R. W., and Schönlieb, C.-B. (2017). Discrete gradient methods for solving variational image regularisation models. J. Phys. A , 50(29):295201, 21.

Itoh, T. and Abe, K. (1988). Hamiltonian-conserving discrete canonical equations based on variational difference quotients. J. Comput. Phys. , 76(1):85-102.

Karimi, S. and Vavasis, S. A. (2016). A unified convergence bound for conjugate gradient and accelerated gradient. arXiv preprint , arXiv:1605.00320.

Krichene, W., Bayen, A., and Bartlett, P. L. (2015). Accelerated mirror descent in continuous and discrete time. In Advances in Neural Information Processing Systems , volume 28.

Luo, H. and Chen, L. (2022). From differential equation solvers to accelerated first-order methods for convex optimization. Math. Program. , 195:735-781.

McLachlan, R. I., Quispel, G. R. W., and Robidoux, N. (1999). Geometric integration using discrete gradients. R. Soc. Lond. Philos. Trans. Ser. A Math. Phys. Eng. Sci. , 357(1754):1021-1045.

Miyatake, Y., Sogabe, T., and Zhang, S.-L. (2018). On the equivalence between SOR-type methods for linear systems and the discrete gradient methods for gradient systems. J. Comput. Appl. Math. , 342:58-69.

Nesterov, Y. (2018). Lectures on convex optimization , volume 137 of Springer Optimization and Its Applications . Springer, Cham.

Nesterov, Y. E. (1983). A method for solving the convex programming problem with convergence rate O (1 /k 2 ) . Dokl. Akad. Nauk SSSR , 269(3):543-547.

Polyak, B. T. (1963). Gradient methods for the minimisation of functionals. USSR Comput. Math. Math. Phys. , 3(4):864-878.

Polyak, B. T. (1964). Some methods of speeding up the convergence of iterative methods. USSR Comput. Math. Math. Phys. , 4(5):1-17.

Quispel, G. R. W. and Capel, H. W. (1996). Solving ODE's numerically while preserving a first integral. Phys. Lett. , 218A:223-228.

Quispel, G. R. W. and McLaren, D. I. (2008). A new class of energy-preserving numerical integration methods. J. Phys. A , 41(4):045206, 7.

Riis, E. S., Ehrhardt, M. J., Quispel, G. R. W., and Schönlieb, C.-B. (2022). A geometric integration approach to nonsmooth, nonconvex optimisation. Found. Comput. Math. , 22(5):1351-1394.

Ringholm, T., Lazi´ c, J., and Schönlieb, C.-B. (2018). Variational image regularization with Euler's elastica using a discrete gradient scheme. SIAM J. Imaging Sci. , 11(4):2665-2691.

Shi, B., Du, S. S., Su, W., and Jordan, M. I. (2019). Acceleration via symplectic discretization of high-resolution differential equations. In Advances in Neural Information Processing Systems , volume 32.

Su, W., Boyd, S., and Candès, E. J. (2014). A differential equation for modeling nesterov's accelerated gradient method: theory and insights. Advances in neural information processing systems , 27.

Su, W., Boyd, S., and Candès, E. J. (2016). A differential equation for modeling Nesterov's accelerated gradient method: theory and insights. J. Mach. Learn. Res. , 17(153):1-43.

Suh, J. J., Roh, G., and Ryu, E. K. (2022). Continuous-time analysis of accelerated gradient methods via conservation laws in dilated coordinate systems. In Proceedings of the 39th International Conference on Machine Learning , pages 20640-20667.

Sun, B., George, J., and Kia, S. (2020). High-resolution modeling of the fastest first-order optimization method for strongly convex functions. In 2020 59th IEEE Conference on Decision and Control (CDC) , pages 4237-4242.

Taylor, A. and Drori, Y. (2022a). An optimal gradient method for smooth strongly convex minimization. Math. Program. , pages 1-38.

Taylor, A. and Drori, Y. (2022b). An optimal gradient method for smooth strongly convex minimization. Mathematical Programming , pages 1-38.

Taylor, A., Van Scoy, B., and Lessard, L. (2018). Lyapunov functions for first-order methods: Tight automated convergence guarantees. In International Conference on Machine Learning , pages 4897-4906.

Ushiyama, K., Sato, S., and Matsuo, T. (2022a). Deriving efficient optimization methods based on stable explicit numerical methods. JSIAM Lett. , 14:29-32.

Ushiyama, K., Sato, S., and Matsuo, T. (2022b). Essential convergence rate of ordinary differential equations appearing in optimization. JSIAM Lett. , 14:119-122.

Van Scoy, B., Freeman, R. A., and Lynch, K. M. (2018). The fastest known globally convergent first-order method for minimizing strongly convex functions. IEEE Control Syst. Lett. , 2(1):49-54.

Wilson, A. (2018). Lyapunov arguments in optimization . University of California, Berkeley.

Wilson, A. C., Recht, B., and Jordan, M. I. (2021). A Lyapunov analysis of accelerated methods in optimization. J. Mach. Learn. Res. , 22(113):1-34.

Zhang, J., Mokhtari, A., Sra, S., and Jadbabaie, A. (2018). Direct runge-kutta discretization achieves acceleration. In Advances in Neural Information Processing Systems , volume 31.

A Table summarizing the convex cases

In contrast to the strongly convex cases, in the convex cases the convergence rates are the same in the order, and the difference appears in its coefficients, which actually depends on the maximum step sizes allowed. See Theorems 5.2 and 5.4 for the rates. In the table below we summarize the step size information.

B Proofs of theorems in Section 2

B.1 Proof of Theorem 2.1

It is sufficient to show that a Lyapunov function

is monotonically nonincreasing since

holds. Here, at each line, we applied the Leibniz rule and the chain rule, substituted the ODE, and used the convexity of f in this order. ■

B.2 Proof of Theorem 2.2

It is sufficient to show a Lyapunov function

is monotonically nonincreasing and thus it is sufficient to show that ˜ E ( t ) := e -µt E ( t ) satisfies ˙ ˜ E ≤ -µ ˜ E . Indeed,

holds. Here, at each line, we applied the chain rule, substituted the ODE, and used strong convexity of f in this order. ■

Indeed,

B.3 Proof of Theorem 2.3

It is sufficient to show that

is nonincreasing. Actually,

holds. Here, at each line, we applied the Leibniz rule, used the chain rule, substituted the ODE, and used the convexity of f in this order. ■

B.4 Proof of Theorem 2.6

It is sufficient to show that

is nonincreasing and thus it is sufficient to show that ˜ E ( t ) := e - √ µt E ( t ) satisfies ˙ ˜ E ≤ - √ µ ˜ E . Actually,

holds. Here, at each line, we applied the chain rule, substituted the ODE, rearranged terms, decomposed the inner product by the law of cosines (see Appendix F), and used the strong convexity of f . ■

C Strict discrete gradients and convexity

In this section, we describe that strict discrete gradients are not generally compatible with the convex inequality; this complements the discussion in Section 3. For example, let us consider imitating the discussion in Appendix B.1 by using a discrete gradient scheme δ + x ( k ) = -∇ d f ( x ( k +1) , x ( k ) ) . Then, since we use the inequality

that holds due to the convexity of f , we should ensure that the discrete counterpart of the left-hand side

is nonpositive for any x, y ∈ R d . However, there is a simple counterexample as shown below.

Let us consider a quadratic and convex objective function f ( x ) = 1 2 ⟨ x, Qx ⟩ , where Q ∈ R d × d is a positive definite matrix. In this case, x ⋆ = 0 and f ⋆ = 0 hold. Then, when we choose a discrete gradient ∇ d f ( y, x ) = ∇ G f ( y, x ) = ∇ AVF f ( y, x ) = Q ( y + x 2 ) , we see

̸

which is positive when y = -x and x = 0 .

D Proof of Theorem 4.2

(i) Since we assume f is L -smooth and µ -strongly convex,

holds for any x, y, z ∈ R d . By adding each side of these inequalities, we obtain

(This inequality is known as the three points descent lemma in optimization.)

(ii) It follows immediately from the µ -strong convexity of f .

Especially when θ = 1 / 2 , ( α, β, γ ) = ( L/ 8 + µ/ 8 , µ/ 4 , µ/ 4) .

(iv) By the same calculation as (iii), we obtain

(v) By (13), we obtain

Since this inequality holds with y and z swapped, we have

Thus,

holds, where the second inequality follows from the arithmetic-geometric means (AM-GM) inequality.

(vi) In the following, for y, z ∈ R d and k = 2 , . . . , d , y 1: k -1 z k : d denotes a vector ( y 1 , . . . , y k -1 , z k , . . . , z d ) ⊤ ∈ R d , while y 1:0 z 1: d and y 1: d z d +1: d denote z and y , respectively. By the telescoping sum and µ -strong convexity of f , we obtain

To evaluate the first term of the most right-hand side, we use the following inequalities:

which hold due to the L -smoothness of f , and also

which holds due to Jensen's inequality. Using them, we obtain

where the last inequality holds due to the AM-GM inequality. Finally, the last term is bounded by

This proves the theorem.

■

For (ii), as noted in the above proof, the assumption of differentiability of f is unnecessary, let alone L -smoothness. Since f is a proper convex function on R d in our setting, the subdifferential ∂f ( x ) is nonempty for all x ∈ R d . Thus we can use ∇ f ( y, x ) ∈ ∂f ( y ) instead of ∇ f ( y, x ) = ∇ f ( y ) . By definition of subgradients, we can recover the same parameters ( α, β, γ ) as the differentiable case.

If µ = 0 , the proofs for (v) and (vi) cease to work where we apply the AM-GM inequality to the inner product. This is also pointed out in the main body of the paper.

E Proofs of theorems in Section 5

E.1 Proof of Theorem 5.2

It is sufficient to show that the discrete Lyapunov function

is nonincreasing. Actually,

holds, and thus if h ≤ 1 / (2 α ) , the right-hand side is not positive. Here, at each line, we applied the discrete Leibniz rule, the weak discrete gradient condition (8), Lemma F.1 as the chain rule, substituted the scheme, and applied again (8) as the convex inequality. ■

E.2 Proof of Theorem 5.3

Let

If δ + ˜ E ( k ) ≤ -c ˜ E ( k +1) for c > 0 , it can be concluded that E ( k ) = (1 + ch ) k ˜ E ( k ) is nonincreasing, and hence f ( x ( k ) ) -f ⋆ ≤ (1 + ch ) -k E (0) . Actually,

δ + ˜ E ( k )

holds. Here, after the second line we used the weak discrete gradient condition (8) as the chain rule, substituted the scheme and used (8) as the strongly convex inequality.

Now we aim to bound ∥ ∥ x ( k ) -x ⋆ ∥ ∥ 2 with ∥ ∥ x ( k +1) -x ⋆ ∥ ∥ 2 . By the same calculation for δ + ∥ ∥ x ( k ) -x ⋆ ∥ ∥ 2 as above, we get the evaluation

Thus, if h ≤ 1 / (2 α ) , we get ∥ ∥ x ( k +1) -x ⋆ ∥ ∥ 2 ≤ ∥ ∥ x ( k ) -x ⋆ ∥ ∥ 2 . In this case, since the second term of (14) is nonpositive, it follows that

To obtain a better rate which is included in the statement of the theorem, by directly using (15) for (14), we see

Since the second term of the right-hand side is nonpositive under h ≤ 1 / ( α + β ) , it can be concluded that

In this case the convergence rate is

E.3 Proof of Theorem 5.4

It is sufficient to show that

■

is nonincreasing. Actually,

Here, at each line, we used the discrete Leibniz rule, applied (8) and Lemma F.1 as the chain rule, substituted the scheme, and applied again (8) as the convex inequality.

Now we define z ( k ) so that (err) ≤ 0 holds. When z ( k ) := x ( k +1) , (err) becomes nonpositive without step size constraints.

Or, since using the scheme we can write

by setting we obtain

This choice of z ( k ) is shown in the theorem. When, for example, A k = ( kh ) 2 , (err) ≤ 0 , provided that h ≤ 1 / √ 2 α . Here we see that only up to a quadratic function is allowed as A k if α > 0 . ■

E.4 Proof of Theorem 5.5

It is sufficient to show that

satisfies δ + ˜ E ( k ) ≤ - √ m ˜ E ( k +1) . To simplify notation, 2( β + γ ) is written as m , and the error terms are gathered into (err) . Then, we see

Here, the first inequality follows from (8) as the chain rule, the second equality from the substitution of the form of the method, and the second inequality follows from again (8) as the strongly convex inequality. In the second inequality, we also used

Now we define x ( k ) so that (err) ≤ 0 . An obvious choice is z ( k ) := x ( k +1) , where (err) is nonpositive under any step size.

To derive another definition of z ( k ) , we proceed with the calculation of the error terms by substituting the form of the method:

Hereafter, √ mh is denoted by ˜ h . By using Lemma F.2, we have

Thus, we see

Here when we define z ( k ) := x ( k ) , (err) is nonpositive under the condition

This condition reads as ˜ h ≤ ( β + γ ) / ( α -β ) , and the convergence rate is

To obtain a better rate, we continue the computation of (err) without defining z ( k ) . Let

and again by inserting the form of the method, and by using Lemma F.2,

Hence we obtain

If we set

(err) is nonpositive under the condition

The definition of z ( k ) is shown in the theorem. By solving the above inequality, we obtain the step size limitation √

which is shown in the theorem.

F Law of cosines and parallelogram identity

This section summarizes some useful lemmas used in the preceding sections.

In Hilbert spaces, especially in Euclidean spaces, the law of cosines holds:

In this paper, we use this formula as an error-containing discrete chain rule of the squared norm.

Lemma F.1. For all x ( k +1) , x ( k ) ∈ R d ,

Another famous identity for the Hilbert norm (especially the Euclidean norm) is the parallelogram identity:

In this paper, we use a generalization of this identity. In the following lemma, we recover the parallelogram identity by setting α = 1 / 2 .

Lemma F.2. For all x, y ∈ R d and α ∈ R

Proof. The claim is obtained by adding each side of the following equalities:

G Proofs of theorems in Section 6

G.1 Proof of Theorem H.1

It is sufficient to show that

holds. Here, we used the chain rule, the continuous system itself, the PŁ condition, and the definition of E in this order. ■

G.2 Proof of Theorem H.3

By the PŁ condition, we observe that

Thus, the evaluation of ∥ ∥ ∇ f ( y, x ) -∇ f ( x ) ∥ ∥ yields β .

(i) From L -smoothness α = L/ 2 follows. By the definition β = 0 .

(ii) L -smoothness yields α = L/ 2 and β = L . (Note that the convexity of f would imply α = 0 , but it is not assumed now. If we adopt the other definition (18) then β = 0 .)

(iii) By the same application of L -smoothness, we obtain α = L/ 8 and β = L/ 2 .

(iv) Since the discrete chain rule exactly holds, ∇ AVF f satisfies α = 0 . Then, by the L -smoothness of f ,

holds, which implies β = L/ 2 .

(v) Similar to the case (iv), α = 0 holds. By the L -smoothness of f ,

which implies β = 5 L/ 8 .

(vi) Similar to the previous cases, α = 0 holds. Using the same notation as in Appendix D (vi), we obtain

where θ k ∈ [0 , 1] is a constant by the mean value theorem. Therefore, β = √ dL holds.

■

G.3 Proof of Theorem H.5

Let

If δ + ˜ E ( k ) ≤ -c ˜ E ( k ) for c > 0 , it can be concluded that E ( k ) = (1 -ch ) -k ˜ E ( k ) is nonincreasing and hence f ( x ( k ) ) -f ⋆ ≤ (1 -ch ) k E (0) . Before starting the computation of δ + ˜ E ( k ) , we transform the weak discrete PŁ condition (17) into a more convenient form. By substituting the scheme into (17), we obtain

Thus, it follows from the weak discrete chain rule (16), the scheme, and the above inequality, that

Hence if h ≤ 1 /α we have the convergence.

H Extension to Polyak-Łojasiewicz type functions.

The weak discrete gradients can be useful also for non-convex functions. Here we illustrate it by taking functions satisfying the Polylak-Łojasiewicz (PŁ) condition. A function f is said to satisfy the PŁ condition if

■

holds for any x ∈ R d . This was introduced as a sufficient condition for the steepest descent to converge Polyak (1963). The set of functions satisfying the PŁ condition contains all differentiable strongly convex functions and some nonconvex functions such as f ( x ) = x 2 +3sin 2 ( x ) .

Theorem H.1 (Continuous systems) . Suppose that f satisfies the PŁ condition. Let x : [0 , ∞ ) → R d be the solution of the gradient flow (3) . Then the solution satisfies

Let us define another weak discrete gradient for functions satisfying the PŁ condition. Recall that the first condition of weak discrete gradients (Definition 4.1) has two meanings: the discrete chain rule (9) and the discrete convex inequality (10). One could consider the PŁ condition instead of convexity.

Definition H.2. A gradient approximation ∇ f : R d × R d → R d is said to be PŁ-type weak discrete gradient of f if there exists positive numbers α, β such that for all x, y ∈ R d the following three conditions hold:

Theorem H.3. If f is L -smooth and satisfies the PŁ condition with the parameter µ , the following functions are PŁ-type weak discrete gradients:

(i) If ∇ f ( y, x ) = ∇ f ( x ) ; then ( α, β ) = ( L/ 2 , 0) .

(ii) If ∇ f ( y, x ) = ∇ f ( y ) , then ( α, β ) = ( L/ 2 , L ) .

(iii) If ∇ f ( y, x ) = ∇ f ( x + y 2 ) , then ( α, β ) = ( L/ 8 , L/ 2) .

(iv) If ∇ f ( y, x ) = ∇ AVF f ( y, x ) , then ( α, β ) = (0 , L/ 2) .

(v) If ∇ f ( y, x ) = ∇ G f ( y, x ) , then ( α, β ) = (0 , 5 L/ 8) .

∇

(vi)

If

f

(

y, x

) =

∇

IA

f

(

y, x

)

, then

(

α, β

) = (0

,

dL

)

.

Remark H.4 . The parameters α and β imply the magnitude of the discretization error. As the second condition in Definition H.2, we can adopt instead

Then, we obtain better parameters for the implicit Euler method (ii).

The proof of Theorem H.3 is postponed in Appendix G.2.

Theorem H.5 (Discrete systems) . Let ∇ f be a PŁ-type weak discrete gradient of f . Let f be a function which satisfies the necessary conditions that the PŁ-type weak DG requires. Let { x ( k ) } be the sequence given by (11) . Then, under the step size condition h ≤ 1 /α , the sequence satisfies

In particular, the sequence satisfies

when the optimal step size h = 1 / (2 α + β ) is employed.

I Some numerical examples

In this section, we give some numerical examples to complement the discussion in the main body of this paper. Note that this is just to illustrate that we can actually easily construct new concrete

√

methods just by assuring the conditions of weak discrete gradients (weak DGs), and that the resulting methods in fact achieve the prescribed rates; we here do not intend to explore a method that beats known state-of-the-art methods. It is of course an ultimate goal of the unified framework project, but is left as an important future work.

Below we consider some explicit optimization methods derived as special cases of the abstract weak DG methods. Here we pick up simple two-dimensional problems so that we can observe not only the decrease of the objective functions but also the trajectories of the points x 's for our intuitive understandings.

First, we consider the case where the objective function is a L -smooth convex function. An explicit weak discrete gradient method is then found as

We call this method (wDG-c). This is the simplest example of the abstract method in Theorem 5.4, where we choose A k = ( kh ) 2 and ∇ f ( y, x ) = ∇ f ( x ) . The authors believe this method itself has not been explicitly pointed out in the literature, and is new. The expected rate is the one predicted in the theorem, O ( 1 /k 2 ) , under the step size condition h ≤ 1 / √ L (recall that α for the weak DG is L/ 2 as shown in Theorem 4.2). For comparison, we pick up Nesterov's accelerated gradient method for convex functions

We denote this method by (NAG-c). It is well-known that it achieves the same rate, under the same step size condition; we summarize these information in Table 3.

As an objective function, we employ

which is not strongly convex. (Strictly speaking, this is not L -smooth as well, but we consider in the following way: for each initial x we obtain the level set { x | f ( x ) = f ( x (0) ) } . Then we consider the function (21) inside the region, and extend the function outside it appropriately; for example such that the function grows linearly as ∥ x ∥ → ∞ .)

Numerical results are shown in Figure 1. The top-left panel of the figure shows the convergence of the objective function f ( x ) when the optimal step size 1 / √ L is chosen. We see both methods achieve the predicted rate O ( 1 /k 2 ) (mind the dotted guide line). We also see that under this setting, (wDG-c) converges faster than (NAG-c). This suggests that the new framework can give rise to an optimization method that is competitive to state-of-the-art methods (as said before, we do not say anything conclusive on this point; in order to discuss practical performance, we further need to discuss other implementation issues such as stepping schemes.) The trajectories of x ( k ) 's are almost the same, for all the tested step sizes.

Next, we consider methods for strongly convex functions. We use the following explicit weak DG method:

We call this (wDG-sc). This can be obtained by setting ∇ f ( y, x ) = ∇ f ( x ) in Theorem 5.5. Since for this choice we have α = L/ 2 and β = µ/ 2 (Theorem 4.2), the step size condition is

10

0

10

1

10

2

10

3

10

4

10

-

13

10

-

10

10

-

7

10

-

4

10

-

1

k

f

(

x

(

k

)

)

-

f

⋆

wDG-c

NAG-c

O

(

1

k

2

)

(a) Evolution of function values (

h

= 1

/

√

L

).

0

0

.

5

1

1

.

5

2

0

1

2

3

4

x

1

x

2

wDG-c

NAG-c

(b) Trajectory (

h

= 1

/

√

L

).

0

0

.

5

1

1

.

5

2

0

1

2

3

4

x

1

x

2

wDG-c

NAG-c

(c) Trajectory (

h

= 0

.

7

/

√

L

).

0

0

.

5

1

1

.

5

2

0

1

2

3

4

x

1

x

2

wDG-c

NAG-c

(d) Trajectory (

h

= 0

.

4

/

√

L

)

h ≤ 1 / ( √ L - √ µ ) , and the predicted rate is O ( (1 -√ µ/L ) k ) (which is attained by the largest h = 1 / ( √ L - √ µ ) ).

As before, we compare this method with Nesterov's accelerated gradient method for strongly convex functions (NAG-sc):

Here, in addition to these, we also consider a simpler method, where z ( k ) = x ( k ) is chosen to find

We call this (wDG2-sc). This method is more natural as a numerical method for the accelerated gradient flow (6), compared to the methods above, and we expect it illustrates how 'being natural as a numerical method' affects the performance. The rate and the step size limitation were revealed in the proof of Theorem 5.5 (Appendix E).

We summarized the step size limitations and rates in Table 3. Notice that the predicted rate of (wDG-sc) is better than that of (wDG2-sc).

The objective function is taken to be the quadratic function

and results are shown is Figure 2. Again the top-left panel shows the convergence of the objective function. We see that (wDG-sc) and (NAG-sc) with each optimal step size show almost the same convergence, which is in this case much better than the predicted worst case rate (the dotted guide line). (wDG2-sc) slightly falls behind the other two, but it eventually achieves almost the same performance as k →∞ . The trajectories of the points x ( k ) 's are, however, quite different among the three methods, which is interesting to observe. The trajectory of (wDG2-sc) seems to suffer from wild oscillations, while (wDG-sc) generates milder trajectory. (NAG-sc) comes between these two. We need careful discussion to conclude which dynamics is the best as an optimization method, but if we consider such oscillations are not desirable (possibly causing some instability), it might suggest that (wDG-sc) is the first choice for this problem. In any case, in this way we can explore various concrete optimization method within the framework of the weak DG by varying the weak DG, which is exactly the main claim of this paper.

0

20

40

60

80

100 120 140

10

-

13

10

-

10

10

-

7

10

-

4

10

-

1

k

f

(

x

(

k

)

)

-

f

⋆

wDG-sc

wDG2-sc

NAG-sc

O

( (

1

-

√

µ

L

)

k

)

(a) Evolution of function values

(

h

= 1

/

(

√

L

- √

µ

)

for wDG-sc and wDG2-sc,

and

h

= 1

/

√

L

for NAG-sc).

-

3

-

2

-

1

0

1

2

-

2

0

2

x

1

x

2

wDG-sc

wDG2-sc

NAG-sc

(b) Trajectory (

h

= 1

/

(

√

L

- √

µ

)

for wDG-sc

and wDG2-sc, and

h

= 1

/

√

L

for NAG-sc).

-

2

-

1

0

1

2

-

2

0

2

x

1

x

2

wDG-sc

wDG2-sc

NAG-sc

(c) Trajectory (

h

= 0

.

7

/

√

L

)

-

2

-

1

0

1

2

-

2

0

2

x

1

x

2

wDG-sc

wDG2-sc

NAG-sc

(d) Trajectory (

h

= 0

.

4

/

√

L

)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a unified discretization framework for the differential equation (DE) approach to convex optimization. The DE approach relates optimization methods to continuous DEs with Lyapunov functionals, providing intuitive insights and convergence rate estimates. However, there has been a lack of a general and consistent way to transition back to discrete optimization methods. The paper introduces the concept of \"weak discrete gradient\" (wDG), which consolidates the conditions required for discrete versions of gradients in the DE approach arguments. The authors define abstract optimization methods using wDG and demonstrate that many existing optimization methods and their convergence rates can be derived as special cases of their setting.\n\nSTRENGTHS:\nComprehensive Approach: The paper addresses an important gap in the DE approach to convex optimization by providing a unified discretization framework. It consolidates the conditions required for discrete versions of gradients, simplifying the overall analysis.\n\nClarity: The pedagogical approach of the paper is appreciated. Most of the theory is constructive and all new aspect are clearly presented.\n\nAbstract Optimization Methods: The introduction of abstract optimization methods using wDG allows for a simpler view and analysis of existing optimization methods. It provides a systematic framework for deriving convergence rate estimates.\n\nPotential for new methods: The framework allows for the development of new optimization methods by combining known DEs with wDG. It opens up opportunities for researchers to explore novel approaches in the optimization field.\n\nWEAKNESSES:\nKnowing the accelerated gradient flow: The biggest problem of the approach presented in this paper is the requirement that one needs to know the accelerated gradient flow to derive its discretization - accelerated gradient flow that has been derived from an accelerated algorithm. Hence, the practicability of the approach is limited as one requires to know already an accelerated method before deriving another one.\n\nCase-Specific Discrete Arguments: The authors acknowledge the need for adjustment and optimization in the construction of optimization methods: while the framework consolidates the case-specific discrete arguments in the definition of wDG, there may still be some complexity involved in finding new pratical wDGs for different methods.\n\nQUESTIONS:\n- Can the framework be easily extended to handle constrained optimizations (for instance, using indicator functions)?\n\n- Theorem 4.2 list several wDG for smooth and/or strongly convex function. Is it possible to find easily other wDG for sunction that does not satisfies those assumptions? (for instance, non convex smooth functions, or functions with smooth second-order derivatives?)\n\n- Can the limitations of the framework, such as the non-optimality of certain convergence rates, be overcome by considering alternative DEs and Lyapunov functionals?\n\nLIMITATIONS:\n- Non-Universal Applicability: The paper acknowledges that certain optimization methods, such as the Douglas-Rachford splitting method and methods based on Runge-Kutta numerical methods, may not fit into the current framework. The applicability of wDG to these methods remains an open question.\n\n- Practicability: While the paper indeed overcome some limitation from previous work in this vein, given the previous points in the Weakness section, it seems the approach overcomplicate the design of new accelerated methods in other settings.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the design of convex optimization schemes based on a general discretization framework applied to differential equations (DE). The authors heavily rely on Lyapunov based inequalities to provide convergence rates. \nThe authors build a systematic framework on top of the one proposed by Su, Boyd and Candès, allowing to provide an automatized analysis, thanks to the concept of weak discrete gradient (wDG). This weaker notion of DG allows one to overcome the assumptions needed for discrete versions of gradients in the DE approach. The resulting convergence rates turn out to be competitive, even though they might not be optimal.\n\nSTRENGTHS:\nThe analysis of a large class of convex optimization algorithms can now be performed in an automatized fashion. Many existing optimization methods together with their rate estimates, can be retrieved as special cases of the authors' method. This offers a simpler perspective.\n\nThe presentation of the paper is great, the authors make an effort to avoid loosing the reader into too many technicalities. Not being an expert in such convex optimization schemes, it was still enjoyable to follow most of the details. The paper is well-written.\n\nWEAKNESSES:\nThe estimated rates are not always optimal as demonstrated by the authors (some sub-cases of Theorem 5.5 for strongly convex functions), which indicates that there are still specific efforts to be pursued for certain algorithms.\n\nQUESTIONS:\nl81: The abbreviation PŁ is not defined in the main text. Please indicate what it stands for. (Polyak-Lojasiewicz)\n\nLIMITATIONS:\nThere is a clear section dedicated to limitations with an explicit list:\n\n- some methods do not fall into the authors' framework, e.g., Douglas–Rachford splitting method \n- adjustment (for instance of time-stepping) could be improved for wDG\n- the obtained rates are not always optimal", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers unconstrained convex optimization problems, studied from the angle of their close relation with differential equations. Specifically, the authors propose a framework for translating results from continuous time methods to their discrete time counterparts. They propose Discrete Gradients (a technical tool from Numerical Analysis) as a means for achieving this, and suitably adapt the concept to the convex optimization scenario (via weak Discrete Gradients). As such, the complexity of showing convergence for DE discretizations is mostly transferred to finding the appropriate wDG. The authors then show how this framework can recover known convex optimization results for some important methods in the class.\n\nSTRENGTHS:\nThe paper follows in a line of work studying discrete gradient methods as optimization schemes. While the central concept of Discrete Gradients is not novel, its weak variant used in this paper and the unifying framework built around it are, to the best of my knowledge. The studied topic is of considerable interest to the optimization community, and the tools introduced in this work are welcome additions. The paper is written in a crisp and clear style (making for a very pleasant read), and the authors critically discuss their results in (mostly) adequate detail.\n\nWEAKNESSES:\n- Reference [1] seems to have quite some overlap in terms of results and considered classes of functions. In this light, a more detailed comparison is warranted in terms of e.g., assumptions -- especially on stepsize values, type of results, classes of functions, and techniques.\n- The usage of the \"free\" iterate z (introduced in Def. 4.1) when devising the discrete accelerated schemes is opaque and would benefit from further discussion. \n\n* Minor (typos and the like):\n\t- Line 315: of in \n\t- Line 315: the Hessian\n\t- you use (i), (ii) in Thm 4.2, but on page 4 line 131 you use the same notation for other concepts, which allows for confusion\n\n\n[1] Ehrhardt, M. J., Riis, E. S., Ringholm, T., and Schönlieb, C.-B. (2018). A geometric integration approach to smooth optimisation: Foundations of the discrete gradient method. arXiv preprint, arXiv:1805.06444.\n\nQUESTIONS:\n- Can the authors provide some discussion on why, e.g., the max stepsize for Gradient Descent (line 1 in table 2) is more restrictive compared to the known strict upper bound of 2/L? It is not clear whether this kind of situation comes from some insurmountable limitations of the framework. \n- Reference [2] is a missing reference for Lyapunov-based approaches. The paper has the same goal as the present work but achieves it via a different route, involving formulas for constructing Lyapunov functions. It would be good to briefly discuss it in the section \"Relation to some other systematic/unified frameworks\"\n- Reference [3] might be relevant for motivating the unified approach\n\n[2] De Sa, C. M., Kale, S., Lee, J. D., Sekhari, A., & Sridharan, K. (2022). From Gradient Flow on Population Loss to Learning with Stochastic Gradient Descent. Advances in Neural Information Processing Systems, 35, 30963-30976.\n[3] Bansal, N., & Gupta, A. (2017). Potential-function proofs for first-order methods. arXiv preprint arXiv:1712.04581.\n\nLIMITATIONS:\nThe authors address the limitations in a separate and very clearly written section and go into adequate detail describing them.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

comments: SUMMARY:
This paper introduces a family of oracles called wDG (weak discrete gradient) verifying (8).
This family constraint (8) has been created in order to make proof works in the discrete setting and has been inspired by the observation of what happens in the continuous setting.
As expected, authors propose results obtained running classical algorithms replacing the gradient by any of those oracles. This new framework allows including many popular algorithms using different oracles such as proximal operator.

STRENGTHS:
The paper is concise and clear. The problem is motivated, and well explained.
The new framework is fairly large and authors detailed examples of classical algorithms that are included in their framework.

WEAKNESSES:
- It might be because I did not know this line of work, but based on the title, I thought of a completely different result, on how, using gradients on the current iterate, efficiently discretize an ODE. Finally, the proposed framework seems to be more about the nature of the oracle itself, not the algorithm, and some of them (prox, average gradients, ...) are not always accessible.

- Thm 2.3: From this thm, it seems that the convergence rate can be arbitrarily good. Authors should discuss this here and mention that discretization error unables to converge faster than the proven lower bound $1/t^2$.

$\underline{\text{Typos:}}$

- l.23: « strong convex » -> strongly
- l.164: « in of »
- eq 10: I think \beta and \gamma have been permuted
- Thm 5.2: « Let $\bar{\nabla} f$ be a wDG of $f$ […] let f be […] » -> « Let f be … let $\bar{\nabla} f$ be …. »

QUESTIONS:
NA

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07