SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kansei Ushiyama
The University of Tokyo, Tokyo, Japan  
ushiyama-kansei074@g.ecc.u-tokyo.ac.jp
Shun Sato
The University of Tokyo, Tokyo, Japan shun@mist.i.u-tokyo.ac.jp
Takayasu Matsuo
The University of Tokyo, Tokyo, Japan
matsuo@mist.i.u-tokyo.ac.jp
The differential equation (DE) approach for convex optimization, which relates optimization methods to specific continuous DEs with rate-revealing Lyapunov functionals, has gained increasing interest since the seminal paper by Su-Boyd-Candès (2014). However, the approach still lacks a crucial component to make it truly useful: there is no general, consistent way to transition back to discrete optimization methods. Consequently, even if we derive insights from continuous DEs, we still need to perform individualized and tedious calculations for the analysis of each method. This paper aims to bridge this gap by introducing a new concept called "weak discrete gradient" (wDG), which consolidates the conditions required for discrete versions of gradients in the DE approach arguments. We then define abstract optimization methods using wDG and provide abstract convergence theories that parallel those in continuous DEs. We demonstrate that many typical optimization methods and their convergence rates can be derived as special cases of this abstract theory. The proposed unified discretization framework for the differential equation approach to convex optimization provides an easy environment for developing new optimization methods and achieving competitive convergence rates with state-of-the-art methods, such as Nesterov's accelerated gradient.
In this paper, we consider unconstrained convex optimization problems:
$$
\min  _ {x \in \mathbb {R} ^ {d}} f (x). \tag {1}
$$
Various optimization methods, such as standard gradient descent and Nesterov's accelerated gradient methods (Nesterov, 1983), are known for these problems. The convergence rates of these methods have been intensively investigated based on the classes of objective functions ( $L$ -smooth and/or  $\mu$ -strongly convex). We focus on the convergence rate of function values  $f(x^{(k)}) - f^{\star}$ , while the rates for  $\| \nabla f(x^{(k)}) \|$  or  $\| x^{(k)} - x^{\star} \|$  have also been discussed. Topics particularly relevant to this study include the lower bound of convergence rates for first-order methods (see Remark 4.3 for the relationship between our framework and first-order methods) for convex and strongly convex functions:  $\mathrm{O}(1 / k^2)$  for  $L$ -smooth and convex functions (cf. Nesterov (2018)) and  $\mathrm{O}\left((1 - \sqrt{\mu / L})^{2k}\right)$  for  $L$ -smooth and  $\mu$ -strongly convex functions (Drori and Taylor, 2022). These lower bounds are tight, as they are
achieved by some optimization methods, such as Nesterov (1983) for convex functions and Van Scoy et al. (2018); Taylor and Drori (2022a) for strongly convex functions. In these studies, the discussion is typically conducted for each method, utilizing various techniques accumulated in the optimization research field.
Whereas, it has long been known that some optimization methods can be related to continuous differential equations (DEs). Early works on this aspect include the following: the continuous gradient flow  $\dot{x} = -\nabla f(x)$  as a continuous optimization method was discussed in Bruck (1975). Similar arguments were later applied to second-order differential equations (Alvarez, 2000; Alvarez et al., 2002; Cabot et al., 2009). An important milestone in this direction was Su et al. (2014), where it was shown that Nesterov's famous accelerated gradient method (Nesterov, 1983) could be related to a second-order system with a convergence rate-revealing "Lyapunov functional." The insights gained from this relationship have been useful in understanding the behavior of the Nesterov method and in considering its new variants. This success has followed by many studies, including Wilson (2018); Wilson et al. (2021). The advantage of the DEs with Lyapunov functional approach (which we simply call the "DE approach" hereafter) is that the continuous DEs are generally more intuitive, and convergence rate estimates are quite straightforward thanks to the Lyapunov functionals. However, the DE approach still lacks one important component; although we can draw useful insights from continuous DEs, there is no known general way to translate them into a discrete setting. Consequently, we still need to perform complex discrete arguments for each method. This limitation was already acknowledged in Su et al. (2014): "... The translation, however, involves parameter tuning and tedious calculations. This is the reason why a general theory mapping properties of ODEs into corresponding properties for discrete updates would be a welcome advance."
In this paper we attempt to provide this missing piece by incorporating the concept of "discrete gradients" (DGs) from numerical analysis, which is used to replicate some properties of continuous DEs in discrete settings. We demonstrate that a relaxed concept of DG, which we call "weak discrete gradient" (wDG), can serve a similar purpose in the optimization context. More precisely, we show that for known DEs in the DE approach, if we define abstract optimization methods using wDGs analogously to the DEs, their abstract convergence theories can be obtained by following the continuous arguments and replacing gradients with wDGs. The tedious parts of the case-specific discrete arguments are consolidated in the definition of wDG, which simplifies the overall arguments: we can now consider "simple continuous DE arguments" and "case-specific discrete discussions summarized in wDG" separately. We demonstrate that many typical existing optimization methods and their rate estimates, previously done separately for each method, can be recovered as special cases of the abstract methods/theories, providing a simpler view of them. Any untested combination of a known DE and wDG presents an obvious new method and its rate, further expanding the potential for innovation in the optimization field. Creating a new wDG leads to a series of optimization methods by applying it to known DEs. One simply needs to verify if the wDG satisfies the conditions for wDG (Theorem 4.2) and reveal the constants of the wDG. If, in the future, a new DE with a rate-revealing Lyapunov functional is discovered, it should be possible to achieve similar results. We suggest first defining an abstract wDG method analogous to the DE and then examining whether the continuous theory can be translated to a discrete setting, as demonstrated in this paper.
The aforementioned paper (Su et al., 2014) concludes in the following way (continued from the previous quote) "Indeed, this would allow researchers to only study the simpler and more user-friendly ODEs." Although there is still room for minor adjustments (see the discussion on limitations below), we believe the wDG framework substantially reduces the complexity of discussions in discrete methods, allowing researchers to focus on more accessible and intuitive aspects of optimization.
We consider the problems (1) on the  $d$ -dimensional Euclidean space  $\mathbb{R}^d$  ( $d$  is a positive integer) with the standard inner product  $\langle \cdot, \cdot \rangle$  and the induced norm  $\|\cdot\|$ , where  $f \colon \mathbb{R}^d \to \mathbb{R}$  represents a differentiable convex objective function. We assume the existence of the optimal value  $f^{\star}$  and the optimal solution  $x^{\star}$ . In the following discussion, we use the inequality
$$
\frac {\mu}{2} \| y - x \| ^ {2} \leq f (y) - f (x) - \langle \nabla f (x), y - x \rangle , \tag {2}
$$
which holds for any  $x, y \in \mathbb{R}^d$  when  $f$  is  $\mu$ -strongly convex and differentiable.
Remark 1.1. Although the scope of this paper is limited due to the restriction of space, the framework can be naturally extended to more general cases. Extension to the objective functions satisfying the Polyak-Lojasiewicz (PL) condition is provided in Appendix H. The framework can be extended to
constrained optimizations by the DE approach for mirror descent methods (cf. Krichene et al. (2015); Wilson et al. (2021)), which the authors have already confirmed. Stochastic methods such as the stochastic gradient descent can be handled by considering random compositions of wDGs, which is left as our future work.
2 Short summary of the differential equation approach
Let us first consider the gradient flow:
$$
\dot {x} = - \nabla f (x), \quad x (0) = x _ {0} \in \mathbb {R} ^ {d}. \tag {3}
$$
$$
\frac {\mathrm {d}}{\mathrm {d} t} f (x (t)) = \left\langle \nabla f (x (t)), \dot {x} (t) \right\rangle = - \| \nabla f (x (t)) \| ^ {2} \leq 0. \tag {4}
$$
This means the flow can be regarded as a continuous optimization method. Notice that the proof is quite simple, once we admit the chain rule of differentiation, and the form of the flow itself (3); this will be quite important in the subsequent discussion.
Despite its simplicity, the convergence rate varies depending on the class of objective functions. Below we show some known results. The following rates are proven using the so-called Lyapunov argument, which introduces a "Lyapunov functional" that explicitly contains the convergence rate. The proof is left to Appendix B (we only note here that, in addition to the two key tools the chain rule and the form of the flow we need the convexity inequality (2) to complete the proof.)
Theorem 2.1 (Convex case). Suppose that  $f$  is convex. Let  $x\colon [0,\infty)\to \mathbb{R}^d$  be the solution of the gradient flow (3). Then the solution satisfies
$$
f (x (t)) - f ^ {\star} \leq \frac {\| x _ {0} - x ^ {\star} \| ^ {2}}{2 t}.
$$
Theorem 2.2 (Strongly convex case). Suppose that  $f$  is  $\mu$ -strongly convex. Let  $x: [0, \infty) \to \mathbb{R}^d$  be the solution of the gradient flow (3). Then the solution satisfies
$$
f (x (t)) - f ^ {\star} \leq \mathrm {e} ^ {- \mu t} \| x _ {0} - x ^ {\star} \| ^ {2}.
$$
An incomplete partial list of works using the Lyapunov approach includes, in addition to Su et al. (2014), Karimi and Vavasis (2016); Attouch et al. (2016); Attouch and Cabot (2017); Attouch et al. (2018); Franca et al. (2018); Defazio (2019); Shi et al. (2019); Wilson et al. (2021) (see also a comprehensive list in Suh et al. (2022)). A difficulty in this approach is that the Lyapunov functionals were found only heuristically. A remedy is provided in Suh et al. (2022); Du (2022), but its target is still limited.
Next, we consider DEs corresponding to accelerated gradient methods, including Nesterov's method. As is well known, the forms of accelerated gradient methods differ depending on the class of objective functions, and consequently, the DEs to be considered also change. In this paper, we call them accelerated gradient flows.
When the objective functions are convex, we consider the following DE proposed in Wilson et al. (2021): let  $A \colon \mathbb{R}_{\geq 0} \to \mathbb{R}_{\geq 0}$  be a differentiable strictly monotonically increasing function with  $A(0) = 0$ , and
$$
\dot {x} = \frac {\dot {A}}{A} (v - x), \quad \dot {v} = - \frac {\dot {A}}{4} \nabla f (x), \tag {5}
$$
with  $(x(0),v(0)) = (x_0,v_0)\in \mathbb{R}^d\times \mathbb{R}^d$
Theorem 2.3 (Convex case (Wilson et al. (2021))). Suppose that  $f$  is convex. Let  $(x, v) \colon [0, \infty) \to \mathbb{R}^d \times \mathbb{R}^d$  be the solution of the DE (5). Then it satisfies
$$
f (x (t)) - f ^ {\star} \leq \frac {2 \| x _ {0} - x ^ {\star} \| ^ {2}}{A (t)}.
$$
Remark 2.4. If we set  $A(t) = t^2$ , this system coincides with a continuous limit DE of the accelerated gradient method for convex functions
$$
\ddot {x} + \frac {3}{t} \dot {x} + \nabla f (x) = 0,
$$
which is derived in Su et al. (2016).
Remark 2.5. From Theorem 2.3, it might seem that we can achieve an arbitrarily high order rate. Although it is surely true in the continuous context, it does not imply we can construct discrete optimization methods from the ODE. In fact, greedily demanding a higher rate is penalized at the timing of discretization from the numerical stability. See, for example, the discussion in Ushiyama et al. (2022b).
Next, for strongly convex objective functions, let us consider the DE (again in Wilson et al. (2021)):
$$
\dot {x} = \sqrt {\mu} (v - x), \quad \dot {v} = \sqrt {\mu} (x - v - \nabla f (x) / \mu) \tag {6}
$$
with  $(x(0), v(0)) = (x_0, v_0) \in \mathbb{R}^d \times \mathbb{R}^d$ . (Note that this system coincides with the continuous limit ODE of the accelerated gradient method for strongly convex functions by Polyak (1964):  $\ddot{x} + 2\sqrt{\mu}\dot{x} + \nabla f(x) = 0$ .)
Theorem 2.6 (Strongly convex case (Wilson et al. (2021); Luo and Chen (2022))). Suppose that  $f$  is  $\mu$ -strongly convex. Let  $(x, v) \colon [0, \infty) \to \mathbb{R}^d \times \mathbb{R}^d$  be the solution of (6). Then it satisfies
$$
f (x (t)) - f ^ {\star} \leq \mathrm {e} ^ {- \sqrt {\mu} t} \left(f (x _ {0}) - f ^ {\star} + \frac {\mu}{2} \| v _ {0} - x ^ {\star} \| ^ {2}\right).
$$
3 Discrete gradient method for gradient flows (from numerical analysis)
The remaining issue is how we discretize the above DEs. In the optimization context, it was done separately in each study. One tempting strategy for a more systematic discretization is to import the concept of "DG," which was invented in numerical analysis for designing structure-preserving numerical methods for gradient flows such as (3) (Gonzalez (1996); McLachlan et al. (1999)). Recall that the automatic decrease of objective function came from the two keys: (a) the chain rule, and (b) the gradient flow structure. The DG method respects and tries to imitate them in discrete settings.
Definition 3.1 (Discrete gradient (Gonzalez (1996); Quispel and Capel (1996))). A continuous map  $\nabla_{\mathrm{d}}f\colon \mathbb{R}^{d}\times \mathbb{R}^{d}\to \mathbb{R}^{d}$  is said to be discrete gradient of  $f$  if the following two conditions hold for all  $x,y\in \mathbb{R}^d$ :
$$
f (y) - f (x) = \left\langle \nabla_ {\mathrm {d}} f (y, x), y - x \right\rangle , \quad \nabla_ {\mathrm {d}} f (x, x) = \nabla f (x). \tag {7}
$$
In the definition provided above, the second condition simply requires that  $\nabla_{\mathrm{d}}f$  approximates  $\nabla f$ . On the contrary, the first condition, referred to as the discrete chain rule, is a critical requirement for the key (a). The discrete chain rule is a scalar equality constraint on the vector-valued function, and for any given  $f$ , there are generally infinitely many DGs. The following is a list of some popular choices of DGs. When it is necessary to differentiate them from wDGs, we call them strict DGs.
Proposition 3.2 (Strict discrete gradients). The following functions are strict DGs.
Gonzalez discrete gradient  $\nabla_{\mathrm{G}}f(y,x)$  (Gonzalez (1996)):
$$
\nabla f \left(\frac {y + x}{2}\right) + \frac {f (y) - f (x) - \langle \nabla f \left(\frac {y + x}{2}\right) , y - x \rangle}{\| y - x \| ^ {2}} (y - x).
$$
Itoh-Abe discrete gradient  $\nabla_{\mathrm{IA}}f(y,x)$  (Itoh and Abe (1988)):
$$
\left[ \begin{array}{c} \frac {f (y _ {1} , x _ {2} , x _ {3} , . . . , x _ {d}) - f (x _ {1} , x _ {2} , x _ {3} , . . . , x _ {d})}{y _ {1} - x _ {1}} \\ \frac {f (y _ {1} , y _ {2} , x _ {3} , . . . , x _ {d}) - f (y _ {1} , x _ {2} , x _ {3} , . . . , x _ {d})}{y _ {2} - x _ {2}} \\ \vdots \\ \frac {f (y _ {1} , y _ {2} , y _ {3} , . . . , y _ {d}) - f (y _ {1} , y _ {2} , y _ {3} , . . . , x _ {d})}{y _ {d} - x _ {d}} \end{array} \right].
$$
Average vector field (AVF)  $\nabla_{\mathrm{AVF}}f(y,x)$  (Quispel and McLaren (2008)):
$$
\int_ {0} ^ {1} \nabla f (\tau y + (1 - \tau) x) \mathrm {d} \tau .
$$
Suppose we have a DG for a given  $f$ . Then we can define a discrete scheme for the gradient flow (3):
$$
\frac {x ^ {(k + 1)} - x ^ {(k)}}{h} = - \nabla_ {\mathrm {d}} f \left(x ^ {(k + 1)}, x ^ {(k)}\right), \quad x ^ {(0)} = x _ {0},
$$
where the positive real number  $h$  is referred to as the step size, and  $x^{(k)} \simeq x(kh)$  is the numerical solution. The left-hand side approximates  $\dot{x}$  and is denoted by  $\delta^{+}x^{(k)}$  hereafter. Note that the definition conforms to the key point (b) mentioned earlier.
The scheme decreases  $f(x^{(k)})$  as expected:
$$
\left(f \left(x ^ {(k + 1)}\right) - f \left(x ^ {(k)}\right)\right) / h = \left\langle \nabla_ {\mathrm {d}} f \left(x ^ {(k + 1)}, x ^ {(k)}\right), \delta^ {+} x ^ {(k)} \right\rangle = - \left\| \nabla_ {\mathrm {d}} f \left(x ^ {(k + 1)}, x ^ {(k)}\right) \right\| ^ {2} \leq 0.
$$
In the first equality we used the discrete chain rule, and in the second, the form of the scheme itself. Observe that the proof proceeds in the same manner as the continuous case (4). Due to the decreasing property, the scheme should work as an optimization method. Additionally, the above argument does not reply on the step size  $h$ , and it can be changed in every step (which will not destroy the decreasing property).
In the numerical analysis community, the above approach has already been attempted for optimizations (Grimm et al. (2017); Celledoni et al. (2018); Ehrhardt et al. (2018); Miyatake et al. (2018); Ringholm et al. (2018); Benning et al. (2020); Riis et al. (2022)). Although they were successful on their own, this does not immediately provide the missing piece we seek for the following reasons. First, the DG framework does not include typical important optimization methods; it even does not include the steepest descent. Second, as noted above, the proofs of rate estimates in the continuous DEs (in Section 2) require the inequality of convexity (2). Unfortunately, however, existing DGs generally do not satisfy it; see Appendix C for a counterexample. Next, we show how to overcome these difficulties.
Remark 3.3. Some members of the optimization community may find the use of DGs peculiar, since it involves referring to two solutions  $x^{(k + 1)}, x^{(k)}$ . However, in some sense, it is quite natural because the decrease of  $f$  occurs in a single step  $x^{(k)} \mapsto x^{(k + 1)}$ . There may also be concerns about the computational complexity of DGs because the method becomes "implicit" by referring to  $x^{(k + 1)}$ . In the field of structure-preserving numerical methods, however, it is widely known that in some highly unstable DEs, implicit methods are often advantageous, allowing larger time-stepping widths, while explicit methods require extremely small ones. In fact, it has been confirmed in Ehrhardt et al. (2018) that this also applies to the optimization context. The Itoh-Abe DG results in a system of  $d$  nonlinear equations, which is slightly less expensive. Moreover, note that the integral in the AVF can be evaluated analytically before implementation, when  $f$  is a polynomial.
4 Weak discrete gradients and abstract optimization methods
We introduce the concept of a weak discrete gradient (wDG), which is a relaxed version of the DG introduced earlier.
Definition 4.1 (Weak discrete gradient). A gradient approximation  $\overline{\nabla} f\colon \mathbb{R}^d\times \mathbb{R}^d\to \mathbb{R}^d$  is said to be weak discrete gradient of  $f$  if there exists  $\alpha \geq 0$  and  $\beta ,\gamma$  with  $\beta +\gamma \geq 0$  such that the following two conditions hold for all  $x,y,z\in \mathbb{R}^d$ :
$$
f (y) - f (x) \leq \left\langle \bar {\nabla} f (y, z), y - x \right\rangle + \alpha \| y - z \| ^ {2} - \beta \| z - x \| ^ {2} - \gamma \| y - x \| ^ {2}, \quad \bar {\nabla} f (x, x) = \nabla f (x). \tag {8}
$$
Note that (8) can be regarded as a modification of the three points descent lemma, where the third variable  $z$  is utilized to give some estimates. The freedom in variable  $z$  in (8) is fully utilized also in this paper; see Theorems 5.4 and 5.5 and their proofs.
The condition (8) can be interpreted in two ways. First, it can be understood as a discrete chain rule in a weaker sense. By substituting  $x$  with  $z$ , we obtain the inequality:
$$
f (y) - f (x) \leq \left\langle \bar {\nabla} f (y, x), y - x \right\rangle + (\alpha - \gamma) \| y - x \| ^ {2}. \tag {9}
$$
1Notice that we use the notation  $\overline{\nabla}$  here, distinguishing it from  $\nabla_{\mathrm{d}}$  denoted as the standard notation for strict discrete gradients in numerical analysis.
Compared to the strict discrete chain rule (7), it is weaker because it is an inequality and allows an error term. Second, it can be interpreted as a weaker discrete convex inequality. By exchanging  $x$  and  $y$  and rearranging terms, we obtain another expression
$$
f (y) - f (x) - \left\langle \overline {{\nabla}} f (x, z), y - x \right\rangle \geq \gamma \| y - x \| ^ {2} + \beta \| y - z \| ^ {2} - \alpha \| x - z \| ^ {2}. \tag {10}
$$
Compared to the strongly convex inequality (2), the term  $(\mu / 2) \| y - x \|^2$  is now replaced with  $\beta \| y - x \|^2 + \gamma \| y - z \|^2$ , which can be interpreted as the squared distance between  $y$  and the point  $(x, z)$  where the gradient is evaluated. The term  $-\alpha \| x - z \|^2$  is an error term.
We now list some examples of wDGs (proof is provided in Appendix D). Notice that these examples include various typical gradient approximations from the optimization and numerical analysis literature. Note that for ease of presentation, we simply write "  $\mu$  -strongly convex function," which includes convex functions by setting  $\mu = 0$
Theorem 4.2. Suppose that  $f\colon \mathbb{R}^d\to \mathbb{R}$  is a  $\mu$ -strongly convex function. Let (L) and (SC) denote the additional assumptions: (L)  $f$  is  $L$ -smooth, and (SC)  $\mu >0$ . Then, the following functions are wDGs:
(i) If  $\overline{\nabla} f(y,x) = \nabla f(x)$  and  $f$  satisfies (L), then  $(\alpha, \beta, \gamma) = (L/2, \mu/2, 0)$ .
(ii) If  $\overline{\nabla} f(y,x) = \nabla f(y)$ , then  $(\alpha, \beta, \gamma) = (0,0,\mu/2)$ .
(iii) If  $\overline{\nabla} f(y,x) = \nabla f\left(\frac{x + y}{2}\right)$  and  $f$  satisfies (L), then  $(\alpha, \beta, \gamma) = ((L + \mu)/8, \mu/4, \mu/4)$ .
(iv) If  $\overline{\nabla} f(y,x) = \nabla_{\mathrm{AVF}}f(y,x)$  and  $f$  satisfies (L), then  $(\alpha ,\beta ,\gamma) = (L / 6 + \mu /12,\mu /4,\mu /4)$
(v) If  $\overline{\nabla} f(y,x) = \nabla_{\mathbb{G}}f(y,x)$  and  $f$  satisfies (L)(SC), then  $(\alpha ,\beta ,\gamma) = ((L + \mu) / 8 +$ $(L - \mu)^2 /16\mu ,\mu /4,0)$
(vi) If  $\overline{\nabla} f(y,x) = \nabla_{\mathrm{IA}}f(y,x)$  and  $f$  satisfies (L)(SC), then  $(\alpha ,\beta ,\gamma) = (dL^{2} / \mu -$ $\mu /4,\mu /2, - \mu /4)$
Although we assumed the smoothness of  $f$  to simplify the presentation, the case (ii) does not demand it (see the end of Appendix D). Thus, it can handle non-smooth convex optimization. While the wDGs (i), (iii), (iv) only require (L), the wDGs (v) and (vi) demand (SC) ( $\mu > 0$ ). This implies that the latter wDGs might be fragile for small  $\mu$ 's.
We now define an abstract method using wDGs:
$$
\frac {x ^ {(k + 1)} - x ^ {(k)}}{h} = - \overline {{\nabla}} f \left(x ^ {(k + 1)}, x ^ {(k)}\right), \quad x ^ {(0)} = x _ {0}, \tag {11}
$$
which is analogous to the gradient flow (3). By "abstract," we mean that it is a formal formula, and given a concrete wDG it reduces to a concrete method; see Table 1 which summarizes some typical choices. Observe that the abstract method covers many popular methods from both optimization and numerical analysis communities. The step size  $h$  may be selected using line search techniques, but for simplicity, we limit our presentation to the fixed step size in this paper (see Remark 5.1).
Remark 4.3. Note that some wDGs are not directly connected to the original gradient  $\nabla f$ 's; the Itoh-Abe wDG (vi) does not even refer to the gradient. Thus, the concrete methods resulting from our framework do not necessarily fall into the so-called "first-order methods," which run in a linear space spanned by the past gradients (Nesterov (1983)). This is why we use the terminology "gradient-based methods" in this paper, instead of first-order methods.
Similar to the aforementioned, we can define abstract methods for (5) and (6). Details and theoretical results can be found in Theorems 5.4 and 5.5.
We also introduce the next lemma, which is useful in expanding the scope of our framework.