SlowGuess/NeurIPS_2023_OCR huggingface dataset:
If we set
$$
z ^ {(k)} := \frac {\tilde {h} + 1}{2 \tilde {h} + 1} x ^ {(k)} + \frac {\tilde {h}}{2 \tilde {h} + 1} v ^ {(k)},
$$
(err) is nonpositive under the condition
$$
(\alpha - \beta) (\tilde {h} + 1) - (\beta + \gamma) \frac {\tilde {h} + 1}{\tilde {h}} \frac {2 \tilde {h} + 1}{\tilde {h}} \leq 0.
$$
The definition of  $z^{(k)}$  is shown in the theorem. By solving the above inequality, we obtain the step size limitation
$$
\tilde {h} \leq \frac {\sqrt {\beta + \gamma}}{\sqrt {\alpha + \gamma} - \sqrt {\beta + \gamma}},
$$
which is shown in the theorem.
F Law of cosines and parallelogram identity
This section summarizes some useful lemmas used in the preceding sections.
In Hilbert spaces, especially in Euclidean spaces, the law of cosines holds:
$$
\left\| y - x \right\| ^ {2} = \left\| y \right\| ^ {2} + \left\| x \right\| ^ {2} - 2 \langle y, x \rangle .
$$
In this paper, we use this formula as an error-containing discrete chain rule of the squared norm.
Lemma F.1. For all  $x^{(k + 1)}, x^{(k)} \in \mathbb{R}^d$ ,
$$
\left\| x ^ {(k + 1)} \right\| ^ {2} - \left\| x ^ {(k)} \right\| ^ {2} = 2 \langle x ^ {(k + 1)}, x ^ {(k + 1)} - x ^ {(k)} \rangle - \left\| x ^ {(k + 1)} - x ^ {(k)} \right\| ^ {2}
$$
Another famous identity for the Hilbert norm (especially the Euclidean norm) is the parallelogram identity:
$$
\left\| \frac {x + y}{2} \right\| ^ {2} + \left\| \frac {x - y}{2} \right\| ^ {2} = \frac {1}{2} \left(\| x \| ^ {2} + \| y \| ^ {2}\right).
$$
In this paper, we use a generalization of this identity. In the following lemma, we recover the parallelogram identity by setting  $\alpha = 1/2$ .
Lemma F.2. For all  $x, y \in \mathbb{R}^d$  and  $\alpha \in \mathbb{R}$ ,
$$
\left\| \alpha x + (1 - \alpha) y \right\| ^ {2} = \alpha \| x \| ^ {2} + (1 - \alpha) \| y \| ^ {2} - \alpha (1 - \alpha) \| x - y \| ^ {2}.
$$
Proof. The claim is obtained by adding each side of the following equalities:
$$
\left\| \alpha x + (1 - \alpha) y \right\| ^ {2} = \alpha^ {2} \| x \| ^ {2} + (1 - \alpha) ^ {2} \| y \| ^ {2} + 2 \alpha (1 - \alpha) \langle x, y \rangle ,
$$
$$
\alpha (1 - \alpha) \| x - y \| ^ {2} = \alpha (1 - \alpha) \| x \| ^ {2} + \alpha (1 - \alpha) \| y \| ^ {2} - 2 \alpha (1 - \alpha) \langle x, y \rangle .
$$
G Proofs of theorems in Section 6
G.1 Proof of Theorem H.1
$$
E (t) := f (x (t)) - f ^ {\star}
$$
satisfies  $\dot{E} \leq -2\mu E$ . Indeed,
$$
\dot {E} = \langle \nabla f (x), \dot {x} \rangle = - \| \nabla f (x) \| ^ {2} \leq - 2 \mu (f (x) - f ^ {\star}) = - 2 \mu E
$$
holds. Here, we used the chain rule, the continuous system itself, the PL condition, and the definition of  $E$  in this order.
G.2 Proof of Theorem H.3
By the PL condition, we observe that
$$
\begin{array}{l} - \left\| \overline {{\nabla}} f (y, x) \right\| \leq - \sqrt {2 \mu (f (x) - f ^ {\star})} + \| \nabla f (x) \| - \left\| \overline {{\nabla}} f (y, x) \right\| \\ \leq - \sqrt {2 \mu (f (x) - f ^ {\star})} + \| \nabla f (x) - \overline {{\nabla}} f (y, x) \|. \\ \end{array}
$$
Thus, the evaluation of  $\left\| \overline{\nabla} f(y,x) - \nabla f(x)\right\|$  yields  $\beta$ .
(i) From  $L$ -smoothness  $\alpha = L / 2$  follows. By the definition  $\beta = 0$ .
(ii)  $L$ -smoothness yields  $\alpha = L / 2$  and  $\beta = L$ . (Note that the convexity of  $f$  would imply  $\alpha = 0$ , but it is not assumed now. If we adopt the other definition (18) then  $\beta = 0$ .)
(iii) By the same application of  $L$ -smoothness, we obtain  $\alpha = L / 8$  and  $\beta = L / 2$ .
(iv) Since the discrete chain rule exactly holds,  $\nabla_{\mathrm{AVF}}f$  satisfies  $\alpha = 0$ . Then, by the  $L$ -smoothness of  $f$ ,
$$
\begin{array}{l} \| \nabla_ {\mathrm {A V F}} f (y, x) - \nabla f (x) \| = \left\| \int_ {0} ^ {1} \nabla f (\tau y + (1 - \tau) x) \mathrm {d} \tau - \nabla f (x) \right\| \\ \leq \int_ {0} ^ {1} \| \nabla f (\tau y + (1 - \tau) x) - \nabla f (x) \| d \tau \\ \leq \int_ {0} ^ {1} L \| \tau y + (1 - \tau) x - x \| \mathrm {d} \tau \\ \leq \int_ {0} ^ {1} L \tau \| y - x \| \mathrm {d} \tau \\ \leq \frac {L}{2} \| y - x \| \\ \end{array}
$$
holds, which implies  $\beta = L / 2$
(v) Similar to the case (iv),  $\alpha = 0$  holds. By the  $L$ -smoothness of  $f$ ,
$$
\begin{array}{l} \| \nabla_ {\mathrm {G}} f (y, x) - \nabla f (x) \| = \left\| \nabla f \left(\frac {y + x}{2}\right) - \frac {f (y) - f (x) - \left\langle \nabla f \left(\frac {y + x}{2}\right) , y - x \right\rangle}{\| y - x \| ^ {2}} (y - x) - \nabla f (x) \right\| \\ \leq \left\| \nabla f \left(\frac {y + x}{2}\right) - \nabla f (x) \right\| + \frac {\left| f (y) - f (x) - \left\langle \nabla f \left(\frac {y + x}{2}\right) , y - x \right\rangle \right|}{\| y - x \|} \\ \leq \frac {L}{2} \| y - x \| + \frac {L}{8} \| y - x \| \\ = \frac {5 L}{8} \| y - x \|, \\ \end{array}
$$
which implies  $\beta = 5L / 8$
(vi) Similar to the previous cases,  $\alpha = 0$  holds. Using the same notation as in Appendix D (vi), we obtain
$$
\begin{array}{l} \| \nabla_ {\mathrm {I A}} f (y, x) - \nabla f (x) \| = \left\| \left[ \begin{array}{c} \frac {f (y _ {1} , x _ {2} , x _ {3} , \dots , x _ {d}) - f (x _ {1} , x _ {2} , x _ {3} , \dots , x _ {d})}{y _ {1} - x _ {1}} \\ \frac {f (y _ {1} , y _ {2} , x _ {3} , \dots , x _ {d}) - f (y _ {1} , x _ {2} , x _ {3} , \dots , x _ {d})}{y _ {2} - x _ {2}} \\ \vdots \\ \frac {f (y _ {1} , y _ {2} , y _ {3} , \dots , y _ {d}) - f (y _ {1} , y _ {2} , y _ {3} , \dots , x _ {d})}{y _ {d} - x _ {d}} \end{array} \right] - \nabla f (x) \right\| \\ = \left\| \left[ \begin{array}{c} \partial_ {1} f (\theta_ {1} y _ {1} + (1 - \theta_ {1}) x _ {1}, x _ {2}, x _ {3} \dots , x _ {d}) \\ \partial_ {2} f (y _ {1}, \theta_ {2} y _ {2} + (1 - \theta_ {2}) x _ {2}, x _ {3} \dots , x _ {d}) \\ \vdots \\ \partial_ {d} f (y _ {1}, y _ {2}, y _ {3}, \dots , \theta_ {d} y _ {d} + (1 - \theta_ {d}) x _ {d}) \end{array} \right] - \nabla f (x) \right\| \\ = \sqrt {\sum_ {k = 1} ^ {d} | (\nabla f (\theta_ {k} y _ {1 : k} x _ {k + 1 : d} + (1 - \theta_ {k}) y _ {1 : k - 1} x _ {k : d})) _ {k} - (\nabla f (x)) _ {k} | ^ {2}} \\ \leq \sqrt {\sum_ {k = 1} ^ {d} L ^ {2} \| y - x \| ^ {2}} \\ = \sqrt {d} L \| y - x \|, \\ \end{array}
$$
where  $\theta_{k}\in [0,1]$  is a constant by the mean value theorem. Therefore,  $\beta = \sqrt{d} L$  holds.
G.3 Proof of Theorem H.5
$$
\tilde {E} ^ {(k)} := f \left(x ^ {(k)}\right) - f ^ {\star}.
$$
If  $\delta^{+}\tilde{E}^{(k)}\leq -c\tilde{E}^{(k)}$  for  $c > 0$ , it can be concluded that  $E^{(k)} = (1 - ch)^{-k}\tilde{E}^{(k)}$  is nonincreasing and hence  $f(x^{(k)}) - f^{\star}\leq (1 - ch)^{k}E^{(0)}$ . Before starting the computation of  $\delta^{+}\tilde{E}^{(k)}$ , we transform the weak discrete PL condition (17) into a more convenient form. By substituting the scheme into (17), we obtain
$$
- \left\| \overline {{\nabla}} f (x ^ {(k + 1)}, x ^ {(k)}) \right\| ^ {2} \leq - \frac {\gamma}{(1 + \beta h) ^ {2}} \Big (f \Big (x ^ {(k)} \Big) - f ^ {\star} \Big).
$$
Thus, it follows from the weak discrete chain rule (16), the scheme, and the above inequality, that
$$
\begin{array}{l} \delta^ {+} \tilde {E} ^ {(k)} = \delta^ {+} f \left(x ^ {(k)}\right) \\ \leq \left\langle \bar {\nabla} f (x ^ {(k + 1)}, x ^ {(k)}), \delta^ {+} x ^ {(k)} \right\rangle + \alpha h \| \delta^ {+} x ^ {(k)} \| ^ {2} \\ = - (1 - \alpha h) \left\| \bar {\nabla} f \left(x ^ {(k + 1)}, x ^ {(k)}\right) \right\| ^ {2} \\ \leq - (1 - \alpha h) \frac {\gamma}{(1 + \beta h) ^ {2}} \left(f \left(x ^ {(k)}\right) - f ^ {\star}\right) \\ = - \gamma \frac {1 - \alpha h}{(1 + \beta h) ^ {2}} E ^ {(k)}. \\ \end{array}
$$
Hence if  $h \leq 1 / \alpha$  we have the convergence.
H Extension to Polyak-Lojasiewicz type functions.
The weak discrete gradients can be useful also for non-convex functions. Here we illustrate it by taking functions satisfying the Polylak-Lojasiewicz (PL) condition. A function  $f$  is said to satisfy the PL condition if
$$
- \| \nabla f (x) \| ^ {2} \leq - 2 \mu (f (x) - f ^ {\star})
$$
holds for any  $x \in \mathbb{R}^d$ . This was introduced as a sufficient condition for the steepest descent to converge Polyak (1963). The set of functions satisfying the PL condition contains all differentiable strongly convex functions and some nonconvex functions such as  $f(x) = x^{2} + 3\sin^{2}(x)$ .
Theorem H.1 (Continuous systems). Suppose that  $f$  satisfies the PL condition. Let  $x\colon [0,\infty)\to \mathbb{R}^d$  be the solution of the gradient flow (3). Then the solution satisfies
Let us define another weak discrete gradient for functions satisfying the PL condition. Recall that the first condition of weak discrete gradients (Definition 4.1) has two meanings: the discrete chain rule (9) and the discrete convex inequality (10). One could consider the PL condition instead of convexity.
Definition H.2. A gradient approximation  $\overline{\nabla} f\colon \mathbb{R}^d\times \mathbb{R}^d\to \mathbb{R}^d$  is said to be  $PL$ -type weak discrete gradient of  $f$  if there exists positive numbers  $\alpha ,\beta$  such that for all  $x,y\in \mathbb{R}^d$  the following three conditions hold:
$$
\overline {{\nabla}} f (x, x) = \nabla f (x),
$$
$$
f (y) - f (x) \leq \langle \bar {\nabla} f (y, x), y - x \rangle + \alpha \| y - x \| ^ {2}, \tag {16}
$$
$$
- \left\| \overline {{\nabla}} f (y, x) \right\| \leq - \sqrt {2 \mu (f (x) - f ^ {\star})} + \beta \| y - x \|. \tag {17}
$$
Theorem H.3. If  $f$  is  $L$ -smooth and satisfies the PL condition with the parameter  $\mu$ , the following functions are PL-type weak discrete gradients:
(i) If  $\overline{\nabla} f(y,x) = \nabla f(x)$ ; then  $(\alpha, \beta) = (L/2,0)$ .
(ii) If  $\overline{\nabla} f(y,x) = \nabla f(y)$ , then  $(\alpha, \beta) = (L/2, L)$ .
(iii) If  $\overline{\nabla} f(y,x) = \nabla f(\frac{x + y}{2})$ , then  $(\alpha, \beta) = (L/8, L/2)$ .
(iv) If  $\overline{\nabla} f(y,x) = \nabla_{\mathrm{AVF}}f(y,x)$ , then  $(\alpha ,\beta) = (0,L / 2)$ .
(v) If  $\overline{\nabla} f(y,x) = \nabla_{\mathrm{G}}f(y,x)$ , then  $(\alpha ,\beta) = (0,5L / 8)$ .
(vi) If  $\overline{\nabla} f(y,x) = \nabla_{\mathrm{IA}}f(y,x)$ , then  $(\alpha ,\beta) = (0,\sqrt{d} L)$ .
Remark H.4. The parameters  $\alpha$  and  $\beta$  imply the magnitude of the discretization error. As the second condition in Definition H.2, we can adopt instead
$$
- \left\| \bar {\nabla} f (y, x) \right\| \leq - \sqrt {2 \mu (f (y) - f ^ {\star})} + \beta \| y - x \|. \tag {18}
$$
Then, we obtain better parameters for the implicit Euler method (ii).
The proof of Theorem H.3 is postponed in Appendix G.2.
Theorem H.5 (Discrete systems). Let  $\overline{\nabla} f$  be a  $PL$ -type weak discrete gradient of  $f$ . Let  $f$  be a function which satisfies the necessary conditions that the  $PL$ -type weak  $DG$  requires. Let  $\{x^{(k)}\}$  be the sequence given by (11). Then, under the step size condition  $h \leq 1 / \alpha$ , the sequence satisfies
$$
f \left(x ^ {(k)}\right) - f ^ {\star} \leq \left(1 - 2 \mu h \frac {(1 - \alpha h)}{(1 + \beta h) ^ {2}}\right) ^ {k} (f (x _ {0}) - f ^ {\star}).
$$
$$
f \left(x ^ {(k)}\right) - f ^ {\star} \leq \left(1 - \frac {\mu}{2 (\alpha + \beta)}\right) ^ {k} (f (x _ {0}) - f ^ {\star}),
$$
when the optimal step size  $h = 1 / (2\alpha + \beta)$  is employed.
I Some numerical examples
In this section, we give some numerical examples to complement the discussion in the main body of this paper. Note that this is just to illustrate that we can actually easily construct new concrete
methods just by assuring the conditions of weak discrete gradients (weak DGs), and that the resulting methods in fact achieve the prescribed rates; we here do not intend to explore a method that beats known state-of-the-art methods. It is of course an ultimate goal of the unified framework project, but is left as an important future work.
Below we consider some explicit optimization methods derived as special cases of the abstract weak DG methods. Here we pick up simple two-dimensional problems so that we can observe not only the decrease of the objective functions but also the trajectories of the points  $x$ 's for our intuitive understandings.
First, we consider the case where the objective function is a  $L$ -smooth convex function. An explicit weak discrete gradient method is then found as
$$
\left\{ \begin{array}{l} x ^ {(k + 1)} - x ^ {(k)} = \frac {2 k + 1}{k ^ {2}} \left(v ^ {(k + 1)} - x ^ {(k + 1)}\right), \\ v ^ {(k + 1)} - v ^ {(k)} = - \frac {2 k + 1}{4} h ^ {2} \nabla f \left(z ^ {(k)}\right), \\ z ^ {(k)} - x ^ {(k)} = \frac {2 k + 1}{(k + 1) ^ {2}} \left(v ^ {(k)} - x ^ {(k)}\right). \end{array} \right. \tag {19}
$$
We call this method (wDG-c). This is the simplest example of the abstract method in Theorem 5.4, where we choose  $A_{k} = (kh)^{2}$  and  $\overline{\nabla} f(y,x) = \nabla f(x)$ . The authors believe this method itself has not been explicitly pointed out in the literature, and is new. The expected rate is the one predicted in the theorem,  $\mathrm{O}\big(1 / k^2\big)$ , under the step size condition  $h \leq 1 / \sqrt{L}$  (recall that  $\alpha$  for the weak DG is  $L / 2$  as shown in Theorem 4.2). For comparison, we pick up Nesterov's accelerated gradient method for convex functions
$$
\left\{ \begin{array}{l} y ^ {(k + 1)} = x ^ {(k)} - h ^ {2} \nabla f (x ^ {(k)}), \\ x ^ {(k + 1)} = y ^ {(k + 1)} + \frac {k}{k + 3} (y ^ {(k + 1)} - y ^ {(k)}). \end{array} \right. \tag {20}
$$
We denote this method by (NAG-c). It is well-known that it achieves the same rate, under the same step size condition; we summarize these information in Table 3.
As an objective function, we employ
$$
f (x) = 0. 1 x _ {1} ^ {4} + 0. 0 0 1 x _ {2} ^ {4}, \tag {21}
$$
which is not strongly convex. (Strictly speaking, this is not  $L$ -smooth as well, but we consider in the following way: for each initial  $x$  we obtain the level set  $\{x \mid f(x) = f(x^{(0)})\}$ . Then we consider the function (21) inside the region, and extend the function outside it appropriately; for example such that the function grows linearly as  $\| x \| \to \infty$ .)
Numerical results are shown in Figure 1. The top-left panel of the figure shows the convergence of the objective function  $f(x)$  when the optimal step size  $1 / \sqrt{L}$  is chosen. We see both methods achieve the predicted rate  $O\left(1 / k^2\right)$  (mind the dotted guide line). We also see that under this setting, (wDG-c) converges faster than (NAG-c). This suggests that the new framework can give rise to an optimization method that is competitive to state-of-the-art methods (as said before, we do not say anything conclusive on this point; in order to discuss practical performance, we further need to discuss other implementation issues such as stepping schemes.) The trajectories of  $x^{(k)}$ 's are almost the same, for all the tested step sizes.
Next, we consider methods for strongly convex functions. We use the following explicit weak DG method:
$$
\left\{ \begin{array}{l} x ^ {(k + 1)} - x ^ {(k)} = \sqrt {\mu} h \left(v ^ {(k + 1)} - x ^ {(k + 1)}\right), \\ v ^ {(k + 1)} - v ^ {(k)} = \sqrt {\mu} h \left(z ^ {(k)} - v ^ {(k + 1)} - \frac {\nabla f (z ^ {(k)})}{\mu}\right), \\ z ^ {(k)} - x ^ {(k)} = \sqrt {\mu} h \left(x ^ {(k)} + v ^ {(k)} - 2 z ^ {(k)}\right). \end{array} \right. \tag {22}
$$
We call this (wDG-sc). This can be obtained by setting  $\overline{\nabla} f(y,x) = \nabla f(x)$  in Theorem 5.5. Since for this choice we have  $\alpha = L / 2$  and  $\beta = \mu /2$  (Theorem 4.2), the step size condition is
$h \leq 1 / (\sqrt{L} - \sqrt{\mu})$ , and the predicted rate is  $\mathrm{O}\Big((1 - \sqrt{\mu / L})^k\Big)$  (which is attained by the largest  $h = 1 / (\sqrt{L} - \sqrt{\mu}))$ .
As before, we compare this method with Nesterov's accelerated gradient method for strongly convex functions (NAG-sc):
$$
\left\{ \begin{array}{l} y ^ {(k + 1)} = x ^ {(k)} - h ^ {2} \nabla f \left(x ^ {(k)}\right), \\ x ^ {(k + 1)} = y ^ {(k + 1)} + \frac {1 - \sqrt {\mu} h}{1 + \sqrt {\mu} h} \left(y ^ {(k + 1)} - y ^ {(k)}\right). \end{array} \right. \tag {23}
$$
Here, in addition to these, we also consider a simpler method, where  $z^{(k)} = x^{(k)}$  is chosen to find
$$
\left\{ \begin{array}{l} x ^ {(k + 1)} - x ^ {(k)} = \sqrt {\mu} h \left(v ^ {(k + 1)} - x ^ {(k + 1)}\right), \\ v ^ {(k + 1)} - v ^ {(k)} = \sqrt {\mu} h \left(x ^ {(k)} - v ^ {(k + 1)} - \frac {\nabla f \left(x ^ {(k)}\right)}{\mu}\right). \end{array} \right. \tag {24}
$$
We call this (wDG2-sc). This method is more natural as a numerical method for the accelerated gradient flow (6), compared to the methods above, and we expect it illustrates how "being natural as a numerical method" affects the performance. The rate and the step size limitation were revealed in the proof of Theorem 5.5 (Appendix E).
We summarized the step size limitations and rates in Table 3. Notice that the predicted rate of (wDG-sc) is better than that of (wDG2-sc).
The objective function is taken to be the quadratic function
$$
f (x) = 0. 0 0 1 \left(x _ {1} - x _ {2}\right) ^ {2} + 0. 1 \left(x _ {1} + x _ {2}\right) ^ {2} + 0. 0 1 x _ {1} + 0. 0 2 x _ {2}, \tag {25}
$$
and results are shown in Figure 2. Again the top-left panel shows the convergence of the objective function. We see that (wDG-sc) and (NAG-sc) with each optimal step size show almost the same convergence, which is in this case much better than the predicted worst case rate (the dotted guide line). (wDG2-sc) slightly falls behind the other two, but it eventually achieves almost the same performance as  $k \to \infty$ . The trajectories of the points  $x^{(k)}$ 's are, however, quite different among the three methods, which is interesting to observe. The trajectory of (wDG2-sc) seems to suffer from wild oscillations, while (wDG-sc) generates milder trajectory. (NAG-sc) comes between these two. We need careful discussion to conclude which dynamics is the best as an optimization method, but if we consider such oscillations are not desirable (possibly causing some instability), it might suggest that (wDG-sc) is the first choice for this problem. In any case, in this way we can explore various concrete optimization method within the framework of the weak DG by varying the weak DG, which is exactly the main claim of this paper.