SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma 4.4. Suppose  $f$  can be expressed as a sum of two functions  $f_{1}, f_{2}$ . If  $\overline{\nabla}_{1}f_{1}$  and  $\overline{\nabla}_{2}f_{2}$  are wDGs of  $f_{1}$  and  $f_{2}$  with parameters  $(\alpha_{1},\beta_{1},\gamma_{1})$  and  $(\alpha_{2},\beta_{2},\gamma_{2})$ , respectively, then  $\overline{\nabla}_{1}f_{1} + \overline{\nabla}_{2}f_{2}$  is a weak discrete gradient of  $f$  with  $(\alpha ,\beta ,\gamma) = (\alpha_{1} + \alpha_{2},\beta_{1} + \beta_{2},\gamma_{1} + \gamma_{2})$ .
This lemma allows us to consider the following discretization of the gradient flow:
$$
\frac {x ^ {(k + 1)} - x ^ {(k)}}{h} = - \nabla f _ {1} \left(x ^ {(k)}\right) - \nabla f _ {2} \left(x ^ {(k + 1)}\right) \tag {12}
$$
within our framework. For instance, if  $f_{1}$  is  $L_{1}$ -smooth and  $\mu_{1}$ -strongly convex, and  $f_{2}$  is  $\mu_{2}$ -strongly convex, then the right-hand side of (12) is a wDG with  $(\alpha, \beta, \gamma) = (L_{1}/2, \mu_{1}/2, \mu_{2}/2)$ . In this case, the method is known as the proximal gradient method or the forward-backward splitting algorithm in optimization (cf. Bauschke and Combettes (2017)). Discretizing the accelerated gradient flows allows for obtaining accelerated versions. (Acceleration of the proximal gradient method has been studied for some time (Beck and Teboulle, 2009b,a).)
5 Convergence rates of abstract optimization methods
We establish the discrete counterparts of Theorems 2.1 to 2.3 and 2.6. Although the proofs are left to Appendix E, we emphasize that they can be performed analogously to those of the continuous cases. The discrete theorems are established in four cases: the gradient flow (3) (for  $f$  convex and  $\mu$ -strongly convex), and the accelerated flows (for  $f$  convex (5) and  $\mu$ -strongly convex (6)). For ease of understanding, we summarize the results for  $\mu$ -strongly convex cases in Table 1. A similar table for convex cases is included in Appendix A.
Remark 5.1. The following theorems are presented under the assumption that the step size  $h$  is fixed for simplicity. However, if all varying step sizes satisfy the step size condition (with a finite number of violations allowed), the theorems still hold true. The step sizes must be bounded by a positive number from below to ensure the designated rates.)
5.1 For the abstract method based on the gradient flow
The abstract method is given in (11).
Theorem 5.2 (Convex case). Let  $f$  be a convex function. Let  $\overline{\nabla} f$  be a wDG of  $f$ , and suppose that  $f$  also satisfies the necessary conditions required by the wDG. Suppose that in the wDG  $\beta \geq 0, \gamma \geq 0$ . Let  $\{x^{(k)}\}$  be the sequence given by (11). Then, under the step size condition  $h \leq 1/(2\alpha)$ , the sequence satisfies
$$
f \left(x ^ {(k)}\right) - f ^ {\star} \leq \frac {\| x _ {0} - x ^ {\star} \| ^ {2}}{2 k h}.
$$
Let us demonstrate how to use the theorem using the proximal gradient method (12) as an example. Suppose that  $f_{1}$  is  $L_{1}$ -smooth and convex, and  $f_{2}$  is convex. Then,  $\overline{\nabla} f(y,x) = \nabla f_{1}(x) + \nabla f_{2}(y)$  is a wDG with the parameter  $(\alpha ,\beta ,\gamma) = (L_1 / 2,0,0)$  due to Theorem 4.2 and Lemma 4.4. Therefore,
the proximal gradient method (12) satisfies the assumption of Theorem 5.2 and thus the convergence rate is  $\mathrm{O}(1 / k)$  under the step size condition  $h \leq (1 / L_1)$ .
Theorem 5.3 (Strongly convex case). Let  $f$  be a strongly convex function. Let  $\overline{\nabla} f$  be a wDG of  $f$ , and suppose that  $f$  also satisfies the necessary conditions required by the wDG. Suppose that in the wDG  $\beta + \gamma > 0$ . Let  $\{x^{(k)}\}$  be the sequence given by (11). Then, under the step size condition  $h \leq 1 / (\alpha + \beta)$ , the sequence satisfies
$$
f \left(x ^ {(k)}\right) - f ^ {\star} \leq \left(1 - \frac {2 (\beta + \gamma) h}{1 + 2 \gamma h}\right) ^ {k} \| x _ {0} - x ^ {\star} \| ^ {2}.
$$
In particular, the sequence satisfies
$$
f \left(x ^ {(k)}\right) - f ^ {\star} \leq \left(1 - \frac {2 (\beta + \gamma)}{\alpha + \beta + 2 \gamma}\right) ^ {k} \| x _ {0} - x ^ {\star} \| ^ {2},
$$
when the optimal step size  $h = 1 / (\alpha + \beta)$  is employed.
5.2 For the abstract methods based on the accelerated gradient flows
We consider abstract methods with wDGs based on the accelerated gradient flows (5) and (6), which will be embedded in the theorems below. We note one thing: when using (8) as an approximation of the chain rule, we can determine  $z$  independently of  $x$  and  $y$ , which gives us some degrees of freedom (thus allowing for adjustment.) Below we show some choices of  $z^{(k)}$  that are easy to calculate from known values while keeping the decrease of the Lyapunov functional.
Theorem 5.4 (Convex case). Let  $f$  be a convex function. Let  $\overline{\nabla} f$  be a wDG of  $f$ , and suppose that  $f$  also satisfies the necessary conditions required by the wDG. Suppose that in the wDG  $\beta \geq 0, \gamma \geq 0$ . Let  $\{(x^{(k)}, v^{(k)})\}$  be the sequence given by
$$
\left\{ \begin{array}{c} \delta^ {+} x ^ {(k)} = \frac {\delta^ {+} A _ {k}}{A _ {k}} \Big (v ^ {(k + 1)} - x ^ {(k + 1)} \Big), \\ \delta^ {+} v ^ {(k)} = - \frac {\delta^ {+} A _ {k}}{4} \overline {{\nabla}} f \Big (x ^ {(k + 1)}, z ^ {(k)} \Big), \\ \frac {z ^ {(k)} - x ^ {(k)}}{h} = \frac {\delta^ {+} A _ {k}}{A _ {k + 1}} \Big (v ^ {(k)} - x ^ {(k)} \Big) \end{array} \right.
$$
with  $\left(x^{(0)}, v^{(0)}\right) = (x_0, v_0)$ , where  $A_k := A(kh)$ . Then if  $A_k = (kh)^2$  and  $h \leq 1/\sqrt{2\alpha}$ , the sequence satisfies
$$
f \Big (x ^ {(k)} \Big) - f ^ {\star} \leq \frac {2 \| x _ {0} - x ^ {\star} \| ^ {2}}{A _ {k}}.
$$
Theorem 5.5 (Strongly convex case). Let  $f$  be a strongly convex function. Let  $\overline{\nabla} f$  be a wDG of  $f$ , and suppose that  $f$  also satisfies the necessary conditions required by the wDG. Suppose that in the wDG  $\beta + \gamma > 0$ . Let  $\left\{ (x^{(k)}, v^{(k)}) \right\}$  be the sequence given by
$$
\left\{ \begin{array}{c} \delta^ {+} x ^ {(k)} = \sqrt {2 (\beta + \gamma)} \Big (v ^ {(k + 1)} - x ^ {(k + 1)} \Big), \\ \delta^ {+} v ^ {(k)} = \sqrt {2 (\beta + \gamma)} \left(\frac {\beta}{\beta + \gamma} z ^ {(k)} + \frac {\gamma}{\beta + \gamma} x ^ {(k + 1)} - v ^ {(k + 1)} - \frac {\overline {{\nabla}} f \big (x ^ {(k + 1)} , z ^ {(k)} \big)}{2 (\beta + \gamma)}\right), \\ \frac {z ^ {(k)} - x ^ {(k)}}{h} = \sqrt {2 (\beta + \gamma)} \Big (x ^ {(k)} + v ^ {(k)} - 2 z ^ {(k)} \Big) \end{array} \right.
$$
with  $\left(x^{(0)}, v^{(0)}\right) = (x_0, v_0)$ . Then if  $h \leq \overline{h} \coloneqq \left(\sqrt{2} (\sqrt{\alpha + \gamma} - \sqrt{\beta + \gamma})\right)^{-1}$ , the sequence satisfies
$$
f \left(x ^ {(k)}\right) - f ^ {\star} \leq \left(1 + \sqrt {2 (\beta + \gamma)} h\right) ^ {- k} \left(f (x _ {0}) - f ^ {\star} + \beta \| v _ {0} - x ^ {\star} \| ^ {2}\right).
$$
$$
f \left(x ^ {(k)}\right) - f ^ {\star} \leq \left(1 - \sqrt {\frac {\beta + \gamma}{\alpha + \gamma}}\right) ^ {k} \left(f \left(x _ {0}\right) - f ^ {\star} + \beta \| v _ {0} - x ^ {\star} \| ^ {2}\right),
$$
when the optimal step size  $h = \overline{h}$  is employed.
Remark 5.6. Time scaling can eliminate the factor  $\sqrt{2(\beta + \gamma)}$  from the scheme and simplify it, as shown in Luo and Chen (2022). However, we do not use time scaling here to match the time scale with the accelerated gradient method and to maintain correspondence with the continuous system.
6 Discussions including Limitations
Relation to some other systematic/unified frameworks A systematic approach to obtaining optimization methods with convergence estimates was developed using the "performance estimation problems" (PEPs) technique, as seen in works such as Taylor et al. (2018); Taylor and Drori (2022b). While our framework unifies discussions in both continuous and discrete settings, the design of methods is not automatic and requires finding a new wDG. In contrast, the PEP framework automates method design but separates discussions between the continuous and discrete settings. Combining these two approaches could be a promising research direction, such as applying our framework to the Lyapunov functionals obtained in Taylor and Drori (2022b).
Another unified convergence analysis is presented in Chen and Luo (2021), where the authors cite the same passage in the Introduction from Su et al. (2016). However, this seems to focus on unifying discussions in the continuous setting, and it is still necessary to individualize the discretization for each method in the discrete setting.
In Diakonikolas and Orecchia (2019), a unified method for deriving continuous DEs describing first-order optimization methods was proposed using the "approximate duality gap technique." While this work is capable of finding new DEs, it does not provide insight into how to discretize the DEs for obtaining discrete optimization methods.
Another closely related work is De Sa et al. (2022), which proposed a framework to construct Lyapunov functionals for continuous ODEs. This is strong in view of the fact that generally Lyapunov functionals can be found only in ad hoc ways. Instead, they considered only the simplest gradient descent (and its stochastic version), while the main focus of the present paper lies in the discretizations.
As said in Section 3, in the field of numerical analysis, the use of discrete gradients has been tried. Among them, Ehrhardt et al. (2018) is a pioneering work that comes with several theoretical results. Both this and the present work aim at convex, strongly convex, and the PL functions (in the Appendix, in the present paper). The scope of Ehrhardt et al. (2018) was limited in the sense that they considered only discretizations of gradient flows with strict discrete gradients. Our target ODEs and discretizations are not limited to that, but as its price, our rate is worse in some strict discrete gradient discretizations of gradient flow. This comes from the difference in proof techniques: they proved convergence rates directly and algebraically, while our analysis is via Lyapunov functionals. They also gave several theoretical results besides the convergence analysis, such as the (unique) existence of solutions, and step-size analysis which are important in actual implementations. Whether these two frameworks could be unified would be an interesting future research topic.
Limitations of the proposed framework. Although we believe that the current framework provides an easy environment for working on the DE approach to optimization, it still has some limitations.
First, methods that do not fall into the current framework exist, such as the following splitting method (cf. the Douglas-Rachford splitting method (Eckstein and Bertsekas, 1992)):
$$
\frac {x ^ {(k + 1 / 2)} - x ^ {(k)}}{h} = - \overline {{\nabla}} _ {1} f _ {1} \Big (x ^ {(k + 1 / 2)}, x ^ {(k)} \Big), \quad \frac {x ^ {(k + 1)} - x ^ {(k + 1 / 2)}}{h} = - \overline {{\nabla}} _ {2} f _ {2} \Big (x ^ {(k + 1)}, x ^ {(k + 1 / 2)} \Big).
$$
The right-hand side cannot be written by a single wDG. Additionally, methods based on Runge-Kutta (RK) numerical methods (Zhang et al. (2018); Ushiyama et al. (2022a)) appear difficult to be captured by wDG because RK methods cannot be expressed by DG in the first place. Investigating whether these methods can be captured by the concept of DG is an interesting future research topic.
Second, there is still some room for adjustment in wDG methods. A typical example is  $z^{(k)}$  in Section 5.2, which is chosen in the theorems to optimize efficiency and rates. Another example is the adjustment of time-stepping in the last phase of constructing a method to achieve a better rate or practical efficiency. Although these optimizations in the construction of optimization methods are standard in optimization studies, we feel that they are difficult to capture in the current framework, as they fall between the intuitive continuous argument and the discrete wDG arguments that aim to capture common structures.
Third, some rates in the theorems are not optimal. For example, on strongly convex functions, the scheme proposed in Theorem 5.5 with the choice (i) achieves the convergence rate of  $\mathrm{O}\Big((1 - \sqrt{\mu / L})^k\Big)$ , which is not the optimal rate of  $\mathrm{O}\Big((1 - \sqrt{\mu / L})^{2k}\Big)$ . This is because the choice of the DE and Lyapunov functional used in this work is not optimal. A DE and Lyapunov functional for obtaining the optimal rate are known (Sun et al., 2020), but the DE is a so-called high-resolution DE (known as a "modified equation" in numerical analysis), which involves the Hessian. Whether these DEs can be captured with the wDG perspective is an interesting future research topic.
7 Concluding remarks
In this paper, we proposed a new unified discretization framework for the DE approach to convex optimization. Our framework provides an easy environment for those working on the DE approach, and some new methods are immediate from the framework, both as methods and for their convergence estimates. For example, any combination of strict DGs with the accelerated gradient flows are new methods, and their rates are given by the theorems. Although we did not include numerical experiments in the main body owing to the space restrictions, some preliminary numerical tests confirming the theory can be found in Appendix I. These tests show that some new methods can be competitive with state-of-the-art methods, such as Nesterov's accelerated gradient.
The authors are grateful for the anonymous reviewers who gave helpful comments to improve this paper. This work is partially supported by JSPS KAKENHI Grant Number JP22K13955, JP21H03452, JP20K21786, and JP20H01822, and by JST SPRING Grant Number JPMJSP2108.
Alvarez, F. (2000). On the minimizing property of a second order dissipative system in Hilbert spaces. SIAM J. Control Optim., 38(4):1102-1119.
Alvarez, F., Attouch, H., Bolte, J., and Redont, P. (2002). A second-order gradient-like dissipative dynamical system with Hessian-driven damping. Application to optimization and mechanics. J. Math. Pures Appl. (9), 81(8):747-779.
Attouch, H. and Cabot, A. (2017). Asymptotic stabilization of inertial gradient dynamics with time-dependent viscosity. J. Differential Equations, 263(9):5412-5458.
Attouch, H., Chbani, Z., and Riahi, H. (2018). Combining fast inertial dynamics for convex optimization with Tikhonov regularization. J. Math. Anal. Appl., 457(2):1065-1094.
Attouch, H., Peypouquet, J., and Redont, P. (2016). Fast convex optimization via inertial dynamics with Hessian driven damping. *J. Differential Equations*, 261(10):5734-5783.
Bauschke, H. H. and Combettes, P. L. (2017). Convex analysis and monotone operator theory in Hilbert spaces. CMS Books in Mathematics/Ouvrages de Mathématiques de la SMC. Springer, Cham, second edition.
Beck, A. and Teboulle, M. (2009a). Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems. IEEE Trans. Image Process., 18(11):2419-2434.
Beck, A. and Teboulle, M. (2009b). A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci., 2(1):183-202.
Benning, M., Riis, E. S., and Schonlieb, C.-B. (2020). Bregman Itoh-Abe methods for sparse optimisation. J. Math. Imaging Vision, 62(6-7):842-857.
Bruck, Jr., R. E. (1975). Asymptotic convergence of nonlinear contraction semigroups in Hilbert space. J. Functional Analysis, 18:15-26.
A Table summarizing the convex cases
In contrast to the strongly convex cases, in the convex cases the convergence rates are the same in the order, and the difference appears in its coefficients, which actually depends on the maximum step sizes allowed. See Theorems 5.2 and 5.4 for the rates. In the table below we summarize the step size information.
Table 2: Examples of weak discrete gradients and resulting convergence rates when  $f$  is an  $L$ -smooth function on  $\mathbb{R}^d$ . The numbers in the  $\overline{\nabla} f$  column correspond to the numbers in Theorem 4.2. The line of the proximal gradient method is described in the setting of (12). (DG) represents that this weak discrete gradient is a strict discrete gradient.
B Proofs of theorems in Section 2
It is sufficient to show that a Lyapunov function
$$
E (t) := t \left(f (x (t)) - f ^ {\star}\right) + \frac {1}{2} \left\| x (t) - x ^ {\star} \right\| ^ {2}
$$
is monotonically nonincreasing since
$$
f (x (t)) - f ^ {\star} \leq \frac {E (t)}{t} \leq \frac {E (0)}{t} = \frac {\| x (0) - x ^ {\star} \| ^ {2}}{2 t}.
$$
$$
\begin{array}{l} \dot {E} = t \langle \nabla f (x), \dot {x} \rangle + f (x) - f ^ {\star} + \langle x - x ^ {\star}, \dot {x} \rangle \\ = - t \left\| \nabla f (x) \right\| ^ {2} + f (x) - f ^ {\star} - \langle \nabla f (x), x - x ^ {\star} \rangle \\ \leq 0 \\ \end{array}
$$
holds. Here, at each line, we applied the Leibniz rule and the chain rule, substituted the ODE, and used the convexity of  $f$  in this order.
B.2 Proof of Theorem 2.2
It is sufficient to show a Lyapunov function
$$
E (t) := \mathrm {e} ^ {\mu t} \left(f (x (t)) - f ^ {\star} + \frac {\mu}{2} \| x (t) - x ^ {\star} \| ^ {2}\right)
$$
is monotonically nonincreasing and thus it is sufficient to show that  $\tilde{E}(t) \coloneqq \mathrm{e}^{-\mu t} E(t)$  satisfies  $\dot{\tilde{E}} \leq -\mu \tilde{E}$ . Indeed,
$$
\begin{array}{l} \dot {\hat {E}} = \langle \nabla f (x), \dot {x} \rangle + \mu \langle x - x ^ {\star}, \dot {x} \rangle \\ = - \left\| \nabla f (x) \right\| ^ {2} - \mu \langle \nabla f (x), x - x ^ {\star} \rangle \\ \leq - \left\| \nabla f (x) \right\| ^ {2} - \mu \left(f (x) - f ^ {\star} + \frac {\mu}{2} \| x - x ^ {\star} \| ^ {2}\right) \\ \leq - \mu \tilde {E} \\ \end{array}
$$
holds. Here, at each line, we applied the chain rule, substituted the ODE, and used strong convexity of  $f$  in this order.
B.3 Proof of Theorem 2.3
It is sufficient to show that
$$
E (t) := A (t) \left(f (x (t)) - f ^ {\star}\right) + 2 \| v (t) - x ^ {\star} \| ^ {2}
$$
is nonincreasing. Actually,
$$
\begin{array}{l} \dot {E} = \dot {A} (f (x) - f ^ {\star}) + A \frac {\mathrm {d}}{\mathrm {d} t} (f (x) - f ^ {\star}) + \frac {\mathrm {d}}{\mathrm {d} t} (2 \| v (t) - x ^ {\star} \| ^ {2}) \\ = \dot {A} (f (x) - f ^ {\star}) + A \langle \nabla f (x), \dot {x} \rangle + 4 \langle \dot {v}, v - x ^ {\star} \rangle \\ = \dot {A} (f (x) - f ^ {\star}) + A \left\langle \nabla f (x), \frac {\dot {A}}{A} (v - x) \right\rangle - 4 \left\langle \frac {\dot {A}}{4} \nabla f (x), v - x ^ {\star} \right\rangle \\ = \dot {A} (f (x) - f ^ {\star} - \langle \nabla f (x), x - x ^ {\star} \rangle) \\ \leq 0 \\ \end{array}
$$
holds. Here, at each line, we applied the Leibniz rule, used the chain rule, substituted the ODE, and used the convexity of  $f$  in this order.
B.4 Proof of Theorem 2.6
$$
E (t) := \mathrm {e} ^ {\sqrt {\mu} t} \left(f (x) - f ^ {\star} + \frac {\mu}{2} \| v - x ^ {\star} \| ^ {2}\right)
$$
is nonincreasing and thus it is sufficient to show that  $\tilde{E}(t) \coloneqq \mathrm{e}^{-\sqrt{\mu} t} E(t)$  satisfies  $\dot{\tilde{E}} \leq -\sqrt{\mu} \tilde{E}$ . Actually,
$$
\begin{array}{l} \dot {\vec {E}} = \left\langle \nabla f (x), \dot {x} \right\rangle + \mu \left\langle \dot {v}, v - x ^ {\star} \right\rangle \\ = \left\langle \nabla f (x), \sqrt {\mu} (v - x) \right\rangle + \mu \left\langle \sqrt {\mu} (x - v - \nabla f (x) / \mu), v - x ^ {\star} \right\rangle \\ = \sqrt {\mu} \left(\left\langle \nabla f (x), x ^ {\star} - x \right\rangle - \mu \left\langle v - x, v - x ^ {\star} \right\rangle\right) \\ = \sqrt {\mu} \left(\left\langle \nabla f (x), x ^ {\star} - x \right\rangle - \frac {\mu}{2} \left(\left\| v - x \right\| ^ {2} + \left\| v - x ^ {\star} \right\| ^ {2} - \left\| x - x ^ {\star} \right\| ^ {2}\right)\right) \\ \leq - \sqrt {\mu} \left(\left(f (x) - f ^ {\star} + \frac {\mu}{2} \| v - x ^ {\star} \| ^ {2}\right) - \frac {\mu}{2} \| v - x \| ^ {2}\right) \\ \leq - \sqrt {\mu} \dot {E} \\ \end{array}
$$
holds. Here, at each line, we applied the chain rule, substituted the ODE, rearranged terms, decomposed the inner product by the law of cosines (see Appendix F), and used the strong convexity of  $f$ .
C Strict discrete gradients and convexity
In this section, we describe that strict discrete gradients are not generally compatible with the convex inequality; this complements the discussion in Section 3. For example, let us consider imitating the discussion in Appendix B.1 by using a discrete gradient scheme  $\delta^{+}x^{(k)} = -\nabla_{\mathrm{d}}f\big(x^{(k + 1)},x^{(k)}\big)$ . Then, since we use the inequality
$$
f (x) - f ^ {\star} - \langle \nabla f (x), x - x ^ {\star} \rangle \leq 0
$$
that holds due to the convexity of  $f$ , we should ensure that the discrete counterpart of the left-hand side
$$
f (x) - f ^ {\star} - \langle \nabla_ {\mathrm {d}} f (y, x), x - x ^ {\star} \rangle
$$
is nonpositive for any  $x, y \in \mathbb{R}^d$ . However, there is a simple counterexample as shown below.
Let us consider a quadratic and convex objective function  $f(x) = \frac{1}{2}\langle x, Qx \rangle$ , where  $Q \in \mathbb{R}^{d \times d}$  is a positive definite matrix. In this case,  $x^{\star} = 0$  and  $f^{\star} = 0$  hold. Then, when we choose a discrete gradient  $\nabla_{\mathrm{d}}f(y,x) = \nabla_{\mathrm{G}}f(y,x) = \nabla_{\mathrm{AVF}}f(y,x) = Q\left(\frac{y + x}{2}\right)$ , we see
$$
f (x) - f ^ {\star} - \langle \nabla_ {\mathrm {d}} f (y, x), x - x ^ {\star} \rangle = \frac {1}{2} \langle x, Q x \rangle - \left\langle Q \left(\frac {y + x}{2}\right), x \right\rangle = - \frac {1}{2} \langle y, Q x \rangle ,
$$
which is positive when  $y = -x$  and  $x \neq 0$ .
D Proof of Theorem 4.2
(i) Since we assume  $f$  is  $L$ -smooth and  $\mu$ -strongly convex,
$$
f (y) - f (z) \leq \langle \nabla f (z), y - z \rangle + \frac {L}{2} \| y - z \| ^ {2},
$$
$$
f (z) - f (x) \leq \langle \nabla f (z), z - x \rangle - \frac {\mu}{2} \| z - x \| ^ {2}
$$
holds for any  $x, y, z \in \mathbb{R}^d$ . By adding each side of these inequalities, we obtain
$$
f (y) - f (x) \leq \langle \nabla f (z), y - x \rangle + \frac {L}{2} \| y - z \| ^ {2} - \frac {\mu}{2} \| z - x \| ^ {2}. \tag {13}
$$
(This inequality is known as the three points descent lemma in optimization.)
(ii) It follows immediately from the  $\mu$ -strong convexity of  $f$ .
(iii) By replacing  $z$  in (13) with  $\theta y + (1 - \theta)z$ , and invoking Lemma F.2, we have
$$
\begin{array}{l} f (y) - f (x) - \langle \nabla f (\theta y + (1 - \theta) z), y - x \rangle \\ \leq \frac {L}{2} \| y - (\theta y + (1 - \theta) z) \| ^ {2} - \frac {\mu}{2} \| \theta y + (1 - \theta) z - x \| ^ {2} \\ = \frac {L}{2} (1 - \theta) ^ {2} \| y - z \| ^ {2} - \frac {\mu}{2} (- \theta (1 - \theta) \| y - z \| ^ {2} + (1 - \theta) \| z - x \| ^ {2} + \theta \| y - x \| ^ {2}). \\ \end{array}
$$
Especially when  $\theta = 1 / 2$ ,  $(\alpha, \beta, \gamma) = (L / 8 + \mu / 8, \mu / 4, \mu / 4)$ .
(iv) By the same calculation as (iii), we obtain
$$
\begin{array}{l} f (y) - f (x) - \left\langle \int_ {0} ^ {1} \nabla f (\tau y + (1 - \tau) z) \mathrm {d} \tau , y - x \right\rangle \\ = \int_ {0} ^ {1} [ f (y) - f (x) - \langle \nabla f (\tau y + (1 - \tau) z), y - x \rangle ] d \tau \\ \leq \int_ {0} ^ {1} \left[ \frac {L}{2} (1 - \tau) ^ {2} \| y - z \| ^ {2} - \frac {\mu}{2} (- \tau (1 - \tau) \| y - z \| ^ {2} + (1 - \tau) \| z - x \| ^ {2} + \tau \| y - x \| ^ {2}) \right] d \tau \\ = \left(\frac {L}{6} + \frac {\mu}{1 2}\right) \| y - z \| ^ {2} - \frac {\mu}{4} \| z - x \| ^ {2} - \frac {\mu}{4} \| y - x \| ^ {2}. \\ \end{array}
$$
(v) By (13), we obtain
$$
f (y) - f (z) - \left\langle \nabla f \left(\frac {y + z}{2}\right), y - z \right\rangle \leq \frac {L}{2} \left\| y - \frac {y + z}{2} \right\| ^ {2} - \frac {\mu}{2} \left\| \frac {y + z}{2} - z \right\| ^ {2} = \frac {L - \mu}{8} \| y - z \| ^ {2}.
$$
Since this inequality holds with  $y$  and  $z$  swapped, we have
$$
\left| f (y) - f (z) - \left\langle \nabla f \left(\frac {y + z}{2}\right), y - z \right\rangle \right| \leq \frac {L - \mu}{8} \| y - z \| ^ {2}.
$$