SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} f (y) - f (x) - \langle \nabla_ {\mathrm {G}} f (y, z), y - x \rangle \\ = f (y) - f (x) - \left\langle \nabla f \left(\frac {y + z}{2}\right) + \frac {f (y) - f (z) - \left\langle \nabla f \left(\frac {y + z}{2}\right) , y - z \right\rangle}{\| y - z \| ^ {2}} (y - z), y - x \right\rangle \\ \leq f (y) - f (x) - \left\langle \nabla f \left(\frac {y + z}{2}\right), y - x \right\rangle + \left| \frac {f (y) - f (z) - \left\langle \nabla f \left(\frac {y + z}{2}\right) , y - z \right\rangle}{\| y - z \| ^ {2}} \right| | \langle y - z, y - x \rangle | \\ \leq \frac {L}{2} \left\| y - \frac {y + z}{2} \right\| ^ {2} - \frac {\mu}{2} \left\| \frac {y + z}{2} - x \right\| ^ {2} + \frac {L - \mu}{8} \left(\frac {L - \mu}{8 \mu} \| y - z \| ^ {2} + \frac {2 \mu}{L - \mu} \| y - x \| ^ {2}\right) \\ = \frac {L}{8} \| y - z \| ^ {2} - \frac {\mu}{2} \left(\frac {1}{2} \| y - x \| ^ {2} + \frac {1}{2} \| z - x \| ^ {2} - \frac {1}{4} \| y - z \| ^ {2}\right) + \frac {(L - \mu) ^ {2}}{1 6 \mu} \| y - z \| ^ {2} + \frac {\mu}{4} \| y - x \| ^ {2} \\ = \left(\frac {L}{8} + \frac {\mu}{8} + \frac {(L - \mu) ^ {2}}{1 6 \mu}\right) \| y - z \| ^ {2} - \frac {\mu}{4} \| z - x \| ^ {2} \\ \end{array}
$$
holds, where the second inequality follows from the arithmetic-geometric means (AM-GM) inequality.
(vi) In the following, for  $y,z\in \mathbb{R}^d$  and  $k = 2,\ldots ,d$ ,  $y_{1:k - 1}z_{k:d}$  denotes a vector  $(y_{1},\dots,y_{k - 1},z_{k},\dots,z_{d})^{\top}\in \mathbb{R}^{d}$ , while  $y_{1:0}z_{1:d}$  and  $y_{1:d}z_{d + 1:d}$  denote  $z$  and  $y$ , respectively. By the telescoping sum and  $\mu$ -strong convexity of  $f$ , we obtain
$$
\begin{array}{l} f (y) - f (x) - \langle \nabla_ {\mathrm {I A}} f (y, z), y - x \rangle \\ = f (y) - f (x) - \sum_ {k = 1} ^ {d} \frac {f \left(y _ {1 : k} z _ {k + 1 : d}\right) - f \left(y _ {1 : k - 1} z _ {k : d}\right)}{y _ {k} - z _ {k}} \left(y _ {k} - x _ {k}\right) \\ = f (y) - f (x) - \sum_ {k = 1} ^ {d} \frac {f \left(y _ {1 : k} z _ {k + 1 : d}\right) - f \left(y _ {1 : k - 1} z _ {k : d}\right)}{y _ {k} - z _ {k}} \left(y _ {k} - z _ {k} + z _ {k} - x _ {k}\right) \\ = f (z) - f (x) - \sum_ {k = 1} ^ {d} \frac {f \left(y _ {1 : k} z _ {k + 1 : d}\right) - f \left(y _ {1 : k - 1} z _ {k : d}\right)}{y _ {k} - z _ {k}} \left(z _ {k} - x _ {k}\right) \\ \leq \langle \nabla f (z), z - x \rangle - \frac {\mu}{2} \| z - x \| ^ {2} \\ + \sum_ {k = 1} ^ {d} \left\{ \begin{array}{l l} \frac {(\nabla f (y _ {1 : k - 1} z _ {k : d})) _ {k} (z _ {k} - y _ {k}) - \frac {\mu}{2} (z _ {k} - y _ {k}) ^ {2}}{y _ {k} - z _ {k}} (z _ {k} - x _ {k}) & \text {i f} \quad \frac {z _ {k} - x _ {k}}{y _ {k} - z _ {k}} > 0 \\ \frac {(\nabla f (y _ {1 : k} z _ {k + 1 : d})) _ {k} (y _ {k} - z _ {k}) - \frac {\mu}{2} (y _ {k} - z _ {k}) ^ {2}}{z _ {k} - y _ {k}} (z _ {k} - x _ {k}) & \text {i f} \quad \frac {z _ {k} - x _ {k}}{y _ {k} - z _ {k}} <   0 \end{array} \right. \\ = \left\langle \nabla f (z), z - x \right\rangle - \frac {\mu}{2} \| z - x \| ^ {2} \\ -\sum_{k = 1}^{d}\left\{ \begin{array}{ll}(\nabla f(y_{1:k - 1}z_{k:d}))_{k}(z_{k} - x_{k}) + \frac{\mu}{2} (y_{k} - z_{k})(z_{k} - x_{k}) & \text{if}\quad \frac{z_{k} - x_{k}}{y_{k} - z_{k}} >0\\ (\nabla f(y_{1:k}z_{k + 1:d}))_{k}(z_{k} - x_{k}) + \frac{\mu}{2} (z_{k} - y_{k})(z_{k} - x_{k}) & \text{if}\quad \frac{z_{k} - x_{k}}{y_{k} - z_{k}} <  0 \end{array} \right. \\ = \sum_ {k = 1} ^ {d} \left\{ \begin{array}{l l} ((\nabla f (z)) _ {k} - (\nabla f (y _ {1: k - 1} z _ {k: d})) _ {k}) (z _ {k} - x _ {k}) & \text {i f} \quad \frac {z _ {k} - x _ {k}}{y _ {k} - z _ {k}} > 0 \\ ((\nabla f (z)) _ {k} - (\nabla f (y _ {1: k} z _ {k + 1: d})) _ {k}) (z _ {k} - x _ {k}) & \text {i f} \quad \frac {z _ {k} - x _ {k}}{y _ {k} - z _ {k}} <   0 \end{array} \right. \\ - \frac {\mu}{2} \| z - x \| ^ {2} - \frac {\mu}{2} \sum_ {k = 1} ^ {d} | z _ {k} - y _ {k} | | z _ {k} - x _ {k} |. \\ \end{array}
$$
To evaluate the first term of the most right-hand side, we use the following inequalities:
$$
\begin{array}{l} \left| \left(\nabla f (z)\right) _ {k} - \left(\nabla f \left(y _ {1: k - 1} z _ {k: d}\right)\right) _ {k} \right| \leq L \| z - y _ {1: k - 1} z _ {k: d} \| \leq L \| z - y \|, \\ \left| \left(\nabla f (z)\right) _ {k} - \left(\nabla f \left(y _ {1: k} z _ {k + 1: d}\right)\right) _ {k} \right| \leq L \| z - y _ {1: k} z _ {k + 1: d} \| \leq L \| z - y \|, \\ \end{array}
$$
which hold due to the  $L$ -smoothness of  $f$ , and also
$$
\sum_ {k = 1} ^ {d} \left| z _ {k} - x _ {k} \right| \leq \sqrt {d \sum_ {k = 1} ^ {d} \left| z _ {k} - x _ {k} \right| ^ {2}} = \sqrt {d} \| z - x \|,
$$
which holds due to Jensen's inequality. Using them, we obtain
$$
\begin{array}{l} f (y) - f (x) - \langle \overline {{\nabla}} _ {\mathrm {I A}} f (y, z), y - x \rangle \\ \leq \sqrt {d} L \| z - y \| \| z - x \| - \frac {\mu}{2} \| z - x \| ^ {2} - \frac {\mu}{2} \sum_ {k = 1} ^ {d} | z _ {k} - y _ {k} | | z _ {k} - x _ {k} | \\ \leq \frac {d L ^ {2}}{\mu} \| z - y \| ^ {2} + \frac {\mu}{4} \| z - x \| ^ {2} - \frac {\mu}{2} \| z - x \| ^ {2} - \frac {\mu}{2} \sum_ {k = 1} ^ {d} | z _ {k} - y _ {k} | | z _ {k} - x _ {k} |, \\ \end{array}
$$
where the last inequality holds due to the AM-GM inequality. Finally, the last term is bounded by
$$
\begin{array}{l} \frac {\mu}{2} \sum_ {k = 1} ^ {d} | z _ {k} - y _ {k} | | z _ {k} - x _ {k} | = - \frac {\mu}{4} \sum_ {k = 1} ^ {d} \left(| z _ {k} - y _ {k} | ^ {2} + | z _ {k} - x _ {k} | ^ {2} - | | z _ {k} - y _ {k} | - | z _ {k} - x _ {k} | ^ {2}\right) \\ \leq - \frac {\mu}{4} \sum_ {k = 1} ^ {d} \left(| z _ {k} - y _ {k} | ^ {2} + | z _ {k} - x _ {k} | ^ {2} - | y _ {k} - x _ {k} | ^ {2}\right) \\ = - \frac {\mu}{4} \left(\| z - y \| ^ {2} + \| z - x \| ^ {2} - \| y - x \| ^ {2}\right). \\ \end{array}
$$
For (ii), as noted in the above proof, the assumption of differentiability of  $f$  is unnecessary, let alone  $L$ -smoothness. Since  $f$  is a proper convex function on  $\mathbb{R}^d$  in our setting, the subdifferential  $\partial f(x)$  is nonempty for all  $x \in \mathbb{R}^d$ . Thus we can use  $\overline{\nabla} f(y,x) \in \partial f(y)$  instead of  $\overline{\nabla} f(y,x) = \nabla f(y)$ . By definition of subgradients, we can recover the same parameters  $(\alpha, \beta, \gamma)$  as the differentiable case.
If  $\mu = 0$ , the proofs for (v) and (vi) cease to work where we apply the AM-GM inequality to the inner product. This is also pointed out in the main body of the paper.
E Proofs of theorems in Section 5
E.1 Proof of Theorem 5.2
It is sufficient to show that the discrete Lyapunov function
$$
E ^ {(k)} := k h \left(f \left(x ^ {(k)}\right) - f ^ {\star}\right) + \frac {1}{2} \left\| x ^ {(k)} - x ^ {\star} \right\| ^ {2}
$$
$$
\begin{array}{l} \delta^ {+} E ^ {(k)} \\ = k h \left(\delta^ {+} f \left(x ^ {(k)}\right)\right) + f \left(x ^ {(k + 1)}\right) - f ^ {\star} + \delta^ {+} \left(\frac {1}{2} \| x ^ {(k)} - x ^ {\star} \| ^ {2}\right) \\ \leq k h \left(\left\langle \overline {{\nabla}} f \left(x ^ {(k + 1)}, x ^ {(k)}\right), \delta^ {+} x ^ {(k)} \right\rangle + \alpha h \left\| \delta^ {+} x ^ {(k)} \right\| ^ {2}\right) + f \left(x ^ {(k + 1)}\right) - f ^ {\star} + \left\langle x ^ {(k + 1)} - x ^ {\star}, \delta^ {+} x ^ {(k)} \right\rangle - \frac {h}{2} \left\| \delta^ {+} x ^ {(k)} \right\| ^ {2} \\ = - k h (1 - \alpha h) \left\| \overline {{\nabla}} f \left(x ^ {(k + 1)}, x ^ {(k)}\right) \right\| ^ {2} + f \left(x ^ {(k + 1)}\right) - f ^ {\star} - \left\langle \overline {{\nabla}} f \left(x ^ {(k + 1)}, x ^ {(k)}\right), x ^ {(k + 1)} - x ^ {\star} \right\rangle - \frac {h}{2} \left\| \delta^ {+} x ^ {(k)} \right\| ^ {2} \\ \leq - k h (1 - \alpha h) \left\| \bar {\nabla} f \left(x ^ {(k + 1)}, x ^ {(k)}\right) \right\| ^ {2} - \left(\frac {h}{2} - \alpha h ^ {2}\right) \left\| \delta^ {+} x ^ {(k)} \right\| ^ {2} \\ \end{array}
$$
holds, and thus if  $h \leq 1 / (2\alpha)$ , the right-hand side is not positive. Here, at each line, we applied the discrete Leibniz rule, the weak discrete gradient condition (8), Lemma F.1 as the chain rule, substituted the scheme, and applied again (8) as the convex inequality.
E.2 Proof of Theorem 5.3
$$
\tilde {E} ^ {(k)} := f \left(x ^ {(k)}\right) - f ^ {\star} + (\beta + \gamma) \left\| x ^ {(k)} - x ^ {\star} \right\| ^ {2}.
$$
If  $\delta^{+}\tilde{E}^{(k)}\leq -c\tilde{E}^{(k + 1)}$  for  $c > 0$ , it can be concluded that  $E^{(k)} = (1 + ch)^{k}\tilde{E}^{(k)}$  is nonincreasing, and hence  $f(x^{(k)}) - f^{\star}\leq (1 + ch)^{-k}E^{(0)}$ . Actually,
$$
\begin{array}{l} \delta^ {+} \tilde {E} ^ {(k)} \\ = \delta^ {+} f (x ^ {(k)}) + \delta^ {+} \left((\beta + \gamma) \| x ^ {(k)} - x ^ {\star} \| ^ {2}\right) \\ \leq \left\langle \overline {{\nabla}} f \Big (x ^ {(k + 1)}, x ^ {(k)} \Big), \delta^ {+} x ^ {(k)} \right\rangle + (\alpha - \gamma) h \left\| \delta^ {+} x ^ {(k)} \right\| ^ {2} + 2 (\beta + \gamma) \left\langle x ^ {(k + 1)} - x ^ {\star}, \delta^ {+} x ^ {(k)} \right\rangle - (\beta + \gamma) h \left\| \delta^ {+} x ^ {(k)} \right\| ^ {2} \\ = - (1 - (\alpha - \gamma) h + (\beta + \gamma) h) \left\| \overline {{\nabla}} f \left(x ^ {(k + 1)}, x ^ {(k)}\right) \right\| ^ {2} - 2 (\beta + \gamma) \left\langle x ^ {(k + 1)} - x ^ {\star}, \overline {{\nabla}} f \left(x ^ {(k + 1)}, x ^ {(k)}\right) \right\rangle \\ \leq - 2 (\beta + \gamma) \left(f \left(x ^ {(k)}\right) - f ^ {\star} + \beta \left\| x ^ {(k)} - x ^ {\star} \right\| ^ {2} + \gamma \left\| x ^ {(k + 1)} - x ^ {\star} \right\| ^ {2}\right) \\ - (1 - (\alpha - \gamma) h + (\beta + \gamma) h - 2 \alpha (\beta + \gamma) h ^ {2}) \| \bar {\nabla} f (x ^ {(k + 1)}, x ^ {(k)}) \| ^ {2} \tag {14} \\ \end{array}
$$
holds. Here, after the second line we used the weak discrete gradient condition (8) as the chain rule, substituted the scheme and used (8) as the strongly convex inequality.
Now we aim to bound  $\left\| x^{(k)} - x^{\star}\right\|^{2}$  with  $\left\| x^{(k + 1)} - x^{\star}\right\|^{2}$ . By the same calculation for  $\delta^{+}\left\| x^{(k)} - x^{\star}\right\|^{2}$  as above, we get the evaluation
$$
\delta^ {+} \left\| x ^ {(k)} - x ^ {\star} \right\| ^ {2} \leq - 2 (f (x ^ {(k + 1)}) - f ^ {\star} + \beta \left\| x ^ {(k)} - x ^ {\star} \right\| ^ {2} + \gamma \left\| x ^ {(k + 1)} - x ^ {(k)} \right\| ^ {2}) - (h - 2 \alpha h ^ {2}) \left\| \overline {{\nabla}} f \left(x ^ {(k + 1)}, x ^ {(k)}\right) \right\| ^ {2}. \tag {15}
$$
Thus, if  $h \leq 1 / (2\alpha)$ , we get  $\left\| x^{(k + 1)} - x^{\star}\right\|^{2} \leq \left\| x^{(k)} - x^{\star}\right\|^{2}$ . In this case, since the second term of (14) is nonpositive, it follows that
$$
\delta^ {+} \tilde {E} ^ {(k)} \leq - 2 (\beta + \gamma) \tilde {E} ^ {(k + 1)}.
$$
To obtain a better rate which is included in the statement of the theorem, by directly using (15) for (14), we see
$$
\begin{array}{l} \delta^ {+} \tilde {E} ^ {(k)} \leq - \frac {2 (\beta + \gamma)}{1 - 2 \beta h} \left(f (x ^ {(k + 1)}) - f ^ {\star} + (\beta + \gamma) \left\| x ^ {(k + 1)} - x ^ {\star} \right\| ^ {2}\right) \\ \left. \right. - \left(\frac {1 - 2 \alpha h}{1 - 2 \beta h} 2 (\beta + \gamma) \beta h ^ {2} + 1 - (\alpha - \gamma) h + (\beta + \gamma) h - 2 \alpha (\beta + \gamma) h ^ {2}\right)\left\| \overline {{\nabla}} f \left(x ^ {(k + 1)}, x ^ {(k)}\right)\right\| ^ {2}. \\ \end{array}
$$
Since the second term of the right-hand side is nonpositive under  $h \leq 1 / (\alpha + \beta)$ , it can be concluded that
$$
\delta^ {+} \tilde {E} ^ {(k)} \leq - \frac {2 (\beta + \gamma)}{1 - 2 \beta h} \tilde {E} ^ {(k + 1)}.
$$
In this case the convergence rate is
$$
\left(\frac {1}{1 + \frac {2 (\beta + \gamma) h}{1 - 2 \beta h}}\right) ^ {k} = \left(1 - \frac {2 (\beta + \gamma) h}{1 + 2 \gamma h}\right) ^ {k}.
$$
E.3 Proof of Theorem 5.4
$$
E ^ {(k)} := A _ {k} \left(f \left(x ^ {(k)}\right) - f ^ {\star}\right) + 2 \| v ^ {(k)} - x ^ {\star} \| ^ {2}
$$
$$
\begin{array}{l} \delta^ {+} E ^ {(k)} \\ = \left(\delta^ {+} A _ {k}\right) \left(f \left(x ^ {(k + 1)}\right) - f ^ {\star}\right) + A _ {k} \left(\delta^ {+} f \left(x ^ {(k)}\right)\right) + 2 \delta^ {+} \left(\left\| v ^ {(k)} - x ^ {\star} \right\| ^ {2}\right) \\ \leq \left(\delta^ {+} A _ {k}\right) \left(f \left(x ^ {(k + 1)}\right) - f ^ {\star}\right) + A _ {k} \left\langle \overline {{\nabla}} f \left(x ^ {(k + 1)}, z ^ {(k)}\right), \delta^ {+} x ^ {(k)} \right\rangle \\ + 4 \left\langle \delta^ {+} v ^ {(k)}, v ^ {(k + 1)} - x ^ {\star} \right\rangle + \frac {A _ {k}}{h} \alpha \left\| x ^ {(k + 1)} - z ^ {(k)} \right\| ^ {2} - 2 h \left\| \delta^ {+} v ^ {(k)} \right\| ^ {2} \\ \leq \left(\delta^ {+} A _ {k}\right) \left(f \left(x ^ {(k + 1)}\right) - f ^ {\star}\right) + A _ {k} \left\langle \overline {{\nabla}} f \left(x ^ {(k + 1)}, z ^ {(k)}\right), \frac {\delta^ {+} A _ {k}}{A _ {k}} \left(v ^ {(k + 1)} - x ^ {(k + 1)}\right) \right\rangle \\ - 4 \left\langle \frac {\delta^ {+} A _ {k}}{4} \bar {\nabla} f \left(x ^ {(k + 1)}, z ^ {(k)}\right), v ^ {(k + 1)} - x ^ {\star} \right\rangle + \frac {A _ {k}}{h} \alpha \left\| x ^ {(k + 1)} - z ^ {(k)} \right\| ^ {2} - 2 h \| \delta^ {+} v ^ {(k)} \| ^ {2} \\ = (\delta^ {+} A _ {k}) \left(f \left(x ^ {(k + 1)}\right) - f ^ {\star} - \left\langle \overline {{\nabla}} f \left(x ^ {(k + 1)}, z ^ {(k)}\right), x ^ {(k + 1)} - x ^ {\star} \right\rangle\right) + \frac {A _ {k}}{h} \alpha \left\| x ^ {(k + 1)} - z ^ {(k)} \right\| ^ {2} - 2 h \| \delta^ {+} v ^ {(k)} \| ^ {2} \\ \leq \left(\delta^ {+} A _ {k}\right) \alpha \left\| x ^ {(k + 1)} - z ^ {(k)} \right\| ^ {2} + \frac {A _ {k}}{h} \alpha \left\| x ^ {(k + 1)} - z ^ {(k)} \right\| ^ {2} - 2 h \left\| \delta^ {+} v ^ {(k)} \right\| ^ {2} \\ = \frac {1}{h} \left(A _ {k + 1} \alpha \left\| x ^ {(k + 1)} - z ^ {(k)} \right\| ^ {2} - 2 \left\| v ^ {(k + 1)} - v ^ {(k)} \right\| ^ {2}\right) =: (\operatorname {e r r}). \\ \end{array}
$$
Here, at each line, we used the discrete Leibniz rule, applied (8) and Lemma F.1 as the chain rule, substituted the scheme, and applied again (8) as the convex inequality.
Now we define  $z^{(k)}$  so that  $(\mathrm{err}) \leq 0$  holds. When  $z^{(k)} := x^{(k+1)}$ , (err) becomes nonpositive without step size constraints.
Or, since using the scheme we can write
$$
\left\| x ^ {(k + 1)} - z ^ {(k)} \right\| ^ {2} = \left\| \frac {A _ {k + 1} - A _ {k}}{A _ {k + 1}} v ^ {(k + 1)} + \frac {A _ {k}}{A _ {k + 1}} x ^ {(k)} - z ^ {(k)} \right\| ^ {2},
$$
by setting
$$
z ^ {(k)} := \frac {A _ {k + 1} - A _ {k}}{A _ {k + 1}} v ^ {(k)} + \frac {A _ {k}}{A _ {k + 1}} x ^ {(k)},
$$
$$
h \times (\operatorname {e r r}) = \left(\frac {\left(A _ {k + 1} - A _ {k}\right) ^ {2}}{A _ {k + 1}} \alpha - 2\right) \left\| v ^ {(k + 1)} - v ^ {(k)} \right\| ^ {2}.
$$
This choice of  $z^{(k)}$  is shown in the theorem. When, for example,  $A_{k} = (kh)^{2}$ , (err)  $\leq 0$ , provided that  $h \leq 1 / \sqrt{2\alpha}$ . Here we see that only up to a quadratic function is allowed as  $A_{k}$  if  $\alpha > 0$ .
E.4 Proof of Theorem 5.5
$$
\tilde {E} ^ {(k)} := f (x ^ {(k)}) - f ^ {\star} + (\beta + \gamma) \| v ^ {(k)} - x ^ {\star} \| ^ {2}
$$
satisfies  $\delta^{+}\tilde{E}^{(k)}\leq -\sqrt{m}\tilde{E}^{(k + 1)}$ . To simplify notation,  $2(\beta +\gamma)$  is written as  $m$ , and the error terms are gathered into (err). Then, we see
$$
\begin{array}{l} \delta^ {+} \tilde {E} ^ {(k)} = \delta^ {+} f (x ^ {(k)}) + \frac {m}{2} \delta^ {+} \| v ^ {(k)} - x ^ {\star} \| ^ {2} \\ \leq \left\langle \overline {{\nabla}} f (x ^ {(k + 1)}, z ^ {(k)}), \delta^ {+} x ^ {(k)} \right\rangle + \frac {\alpha}{h} \left\| x ^ {(k + 1)} - z ^ {(k)} \right\| ^ {2} - \frac {\beta}{h} \left\| z ^ {(k)} - x ^ {(k)} \right\| ^ {2} - \gamma h \left\| \delta^ {+} x ^ {(k)} \right\| ^ {2} \\ + m \left\langle \delta^ {+} v ^ {(k)}, v ^ {(k + 1)} - x ^ {\star} \right\rangle - \frac {m}{2} h \| \delta^ {+} v ^ {(k)} \| ^ {2} \\ = \left\langle \bar {\nabla} f \left(x ^ {(k + 1)}, z ^ {(k)}\right), \sqrt {m} \left(v ^ {(k + 1)} - x ^ {(k + 1)}\right) \right\rangle + (\operatorname {e r r}) \\ + m \left\langle \sqrt {m} \left(\frac {2 \beta}{m} z ^ {(k)} + \frac {2 \gamma}{m} x ^ {(k + 1)} - v ^ {(k + 1)} - \frac {1}{m} \overline {{\nabla}} f (x ^ {(k + 1)}, z ^ {(k)})\right), v ^ {(k + 1)} - x ^ {\star} \right\rangle \\ = \sqrt {m} \left\langle \bar {\nabla} f \left(x ^ {(k + 1)}, z ^ {(k)}\right), x ^ {\star} - x ^ {(k + 1)} \right\rangle - 2 \sqrt {m} \beta \left\langle v ^ {(k + 1)} - z ^ {(k)}, v ^ {(k + 1)} - x ^ {\star} \right\rangle \\ - 2 \sqrt {m} \gamma \left\langle v ^ {(k + 1)} - x ^ {(k + 1)}, v ^ {(k + 1)} - x ^ {\star} \right\rangle + (\operatorname {e r r}) \\ = \sqrt {m} \left\langle \bar {\nabla} f \left(x ^ {(k + 1)}, z ^ {(k)}\right), x ^ {\star} - x ^ {(k + 1)} \right\rangle - \sqrt {m} \beta \left(\left\| v ^ {(k + 1)} - z ^ {(k)} \right\| ^ {2} + \left\| v ^ {(k + 1)} - x ^ {\star} \right\| ^ {2} - \left\| z ^ {(k)} - x ^ {\star} \right\| ^ {2}\right) \\ - \sqrt {m} \gamma \left(\left\| v ^ {(k + 1)} - x ^ {(k + 1)} \right\| ^ {2} + \left\| v ^ {(k + 1)} - x ^ {\star} \right\| ^ {2} - \left\| x ^ {(k + 1)} - x ^ {\star} \right\| ^ {2}\right) + (\operatorname {e r r}) \\ \leq - \sqrt {m} \left(f \left(x ^ {(k + 1)}\right) - f ^ {\star} + \frac {m}{2} \left\| v ^ {(k + 1)} - x ^ {\star} \right\| ^ {2}\right) \\ + \sqrt {m} \left(\alpha \left\| x ^ {(k + 1)} - z ^ {(k)} \right\| ^ {2} - \beta \left\| v ^ {(k + 1)} - z ^ {(k)} \right\| ^ {2} - \gamma \left\| v ^ {(k + 1)} - x ^ {(k + 1)} \right\| ^ {2}\right) + (\operatorname {e r r}) \\ = - \sqrt {m} \tilde {E} ^ {(k + 1)} + (\operatorname {e r r}). \\ \end{array}
$$
Here, the first inequality follows from (8) as the chain rule, the second equality from the substitution of the form of the method, and the second inequality follows from again (8) as the strongly convex inequality. In the second inequality, we also used
$$
- \Big\langle \overline{\nabla} f\Big(x^{(k + 1)},z^{(k)}\Big),x^{(k + 1)} - x^{\star}\Big\rangle  + \beta \Big\| z^{(k)} - x^{\star}\Big\|^{2} + \gamma \Big\| x^{(k + 1)} - x^{\star}\Big\|^{2}\\ \leq -\Big(f\Big(x^{(k + 1)}\Big) - f^{\star}\Big) + \alpha \Big\| x^{(k + 1)} - z^{(k)}\Big\|^{2}.
$$
Now we define  $x^{(k)}$  so that  $(\mathrm{err}) \leq 0$ . An obvious choice is  $z^{(k)} := x^{(k+1)}$ , where (err) is nonpositive under any step size.
To derive another definition of  $z^{(k)}$ , we proceed with the calculation of the error terms by substituting the form of the method:
$$
\begin{array}{l} h \times (\operatorname {e r r}) = \alpha \left\| x ^ {(k + 1)} - z ^ {(k)} \right\| ^ {2} - \beta \left\| z ^ {(k)} - x ^ {(k)} \right\| ^ {2} - \gamma \left\| x ^ {(k + 1)} - x ^ {(k)} \right\| ^ {2} - (\beta + \gamma) \left\| v ^ {(k + 1)} - v ^ {(k)} \right\| ^ {2} \\ + \sqrt {m} h \left(\alpha \left\| x ^ {(k + 1)} - z ^ {(k)} \right\| ^ {2} - \beta \left\| v ^ {(k + 1)} - z ^ {(k)} \right\| ^ {2} - \gamma \left\| v ^ {(k + 1)} - x ^ {(k + 1)} \right\| ^ {2}\right) \\ = \alpha (h + 1) \| x ^ {(k + 1)} - z ^ {(k)} \| ^ {2} - \beta \left(\left\| z ^ {(k)} - x ^ {(k)} \right\| ^ {2} + \left\| v ^ {(k + 1)} - v ^ {(k)} \right\| ^ {2} + \sqrt {m} h \| v ^ {(k + 1)} - z ^ {(k)} \| ^ {2}\right) \\ \left. - \gamma \left(\left\| x ^ {(k + 1)} - x ^ {(k)} \right\| ^ {2} + \left\| v ^ {(k + 1)} - v ^ {(k)} \right\| ^ {2} + \sqrt {m} h \left\| v ^ {(k + 1)} - x ^ {(k + 1)} \right\| ^ {2}\right) \right. \\ = \alpha (h + 1) \left\| x ^ {(k + 1)} - z ^ {(k)} \right\| ^ {2} \\ \left. - \beta \left(\left\| z ^ {(k)} - x ^ {(k)} \right\| ^ {2} + \left\| v ^ {(k + 1)} - v ^ {(k)} \right\| ^ {2} + \sqrt {m} h \left\| x ^ {(k + 1)} + \frac {x ^ {(k + 1)} - x ^ {(k)}}{\sqrt {m} h} - z ^ {(k)} \right\| ^ {2}\right) \right. \\ \left. - \gamma \left(\left\| x ^ {(k + 1)} - x ^ {(k)} \right\| ^ {2} + \left\| v ^ {(k + 1)} - v ^ {(k)} \right\| ^ {2} + \sqrt {m} h \left\| x ^ {(k + 1)} + \frac {x ^ {(k + 1)} - x ^ {(k)}}{\sqrt {m}} h - x ^ {(k + 1)} \right\| ^ {2}\right). \right. \\ \end{array}
$$
Hereafter,  $\sqrt{mh}$  is denoted by  $\tilde{h}$ . By using Lemma F.2, we have
$$
\begin{array}{l} \tilde {h} \left\| x ^ {(k + 1)} + \frac {x ^ {(k + 1)} - x ^ {(k)}}{\tilde {h}} - z ^ {(k)} \right\| ^ {2} \\ = \tilde {h} \left(\frac {\tilde {h} + 1}{\tilde {h}}\right) ^ {2} \left\| \frac {\tilde {h}}{\tilde {h} + 1} \left(x ^ {(k + 1)} - z ^ {(k)}\right) + \frac {1}{\tilde {h} + 1} \left(x ^ {(k + 1)} - x ^ {(k)}\right) \right\| ^ {2} \\ = \frac {\left(\tilde {h} + 1\right) ^ {2}}{\tilde {h}} \left(\frac {\tilde {h}}{\tilde {h} + 1} \left\| x ^ {(k + 1)} - z ^ {(k)} \right\| ^ {2} + \frac {1}{\tilde {h} + 1} \left\| x ^ {(k + 1)} - x ^ {(k)} \right\| ^ {2} - \frac {\tilde {h}}{\left(\tilde {h} + 1\right) ^ {2}} \left\| z ^ {(k)} - x ^ {(k)} \right\| ^ {2}\right) \\ = \left(\tilde {h} + 1\right) \left\| x ^ {(k + 1)} - z ^ {(k)} \right\| ^ {2} + \frac {\tilde {h} + 1}{\tilde {h}} \left\| x ^ {(k + 1)} - x ^ {(k)} \right\| ^ {2} - \left\| z ^ {(k)} - x ^ {(k)} \right\| ^ {2}. \\ \end{array}
$$
Thus, we see
$$
\tilde {h} \times (\operatorname {e r r}) = (\alpha - \beta) \Big (\tilde {h} + 1 \Big) \left\| x ^ {(k + 1)} - z ^ {(k)} \right\| ^ {2} - (\beta + \gamma) \left(\frac {\tilde {h} + 1}{\tilde {h}} \left\| x ^ {(k + 1)} - x ^ {(k)} \right\| ^ {2} + \left\| v ^ {(k + 1)} - v ^ {(k)} \right\| ^ {2}\right).
$$
Here when we define  $z^{(k)}\coloneqq x^{(k)}$  , (err) is nonpositive under the condition
$$
(\alpha - \beta) (\tilde {h} + 1) - (\beta + \gamma) \frac {\tilde {h} + 1}{\tilde {h}} \leq 0.
$$
This condition reads as  $\tilde{h} \leq (\beta + \gamma) / (\alpha - \beta)$ , and the convergence rate is
$$
\left(1 + \sqrt {2 (\beta + \gamma)}\right) ^ {- k} = \left(1 + \tilde {h}\right) ^ {- k} \geq \left(1 - \frac {\beta + \gamma}{\alpha + \gamma}\right) ^ {k}.
$$
To obtain a better rate, we continue the computation of (err) without defining  $z^{(k)}$ . Let
$$
\eta = \frac {1}{\frac {\tilde {h} + 1}{\tilde {h}} + 1} = \frac {\tilde {h}}{2 \tilde {h} + 1},
$$
and again by inserting the form of the method, and by using Lemma F.2,
$$
\begin{array}{l} \frac {\tilde {h} + 1}{\tilde {h}} \left\| x ^ {(k + 1)} - x ^ {(k)} \right\| ^ {2} + \left\| v ^ {(k + 1)} - v ^ {(k)} \right\| ^ {2} \\ = \frac {\tilde {h} + 1}{\tilde {h}} \left\| x ^ {(k + 1)} - x ^ {(k)} \right\| ^ {2} + \left\| x ^ {(k + 1)} + \frac {x ^ {(k + 1)} - x ^ {(k)}}{\tilde {h}} - v ^ {(k)} \right\| ^ {2} \\ = \frac {\tilde {h} + 1}{\tilde {h}} \left\| x ^ {(k + 1)} - x ^ {(k)} \right\| ^ {2} + \left(\frac {\tilde {h} + 1}{\tilde {h}}\right) ^ {2} \left\| x ^ {(k + 1)} - \frac {\tilde {h}}{\tilde {h} + 1} v ^ {(k)} - \frac {1}{\tilde {h} + 1} x ^ {(k)} \right\| ^ {2} \\ = \frac {\tilde {h} + 1}{\tilde {h}} \frac {1}{\eta} \left(\eta \left\| x ^ {(k + 1)} - x ^ {(k)} \right\| ^ {2} + (1 - \eta) \left\| x ^ {(k + 1)} - \frac {\tilde {h}}{\tilde {h} + 1} v ^ {(k)} - \frac {1}{\tilde {h} + 1} x ^ {(k)} \right\| ^ {2}\right) \\ = \frac {\tilde {h} + 1}{\tilde {h}} \frac {1}{\eta} \left(\left\| \eta \left(x ^ {(k + 1)} - x ^ {(k)}\right) + (1 - \eta) \left(x ^ {(k + 1)} - \frac {\tilde {h}}{\tilde {h} + 1} v ^ {(k)} - \frac {1}{\tilde {h} + 1} x ^ {(k)}\right) \right\| ^ {2} \right. \\ \left. + \eta (1 - \eta) \left\| x ^ {(k + 1)} - x ^ {(k)} - \left(x ^ {(k + 1)} - \frac {\tilde {h}}{\tilde {h} + 1} v ^ {(k)} - \frac {1}{\tilde {h} + 1} x ^ {(k)}\right) \right\| ^ {2}\right) \\ = \frac {\tilde {h} + 1}{\tilde {h}} \frac {2 \tilde {h} + 1}{\tilde {h}} \left[ \left\| x ^ {(k + 1)} - \left(\frac {\tilde {h} + 1}{2 \tilde {h} + 1} x ^ {(k)} + \frac {\tilde {h}}{2 \tilde {h} + 1} v ^ {(k)}\right) \right\| ^ {2} + \frac {\tilde {h}}{2 \tilde {h} + 1} \frac {\tilde {h} + 1}{2 \tilde {h} + 1} \left(\frac {\tilde {h}}{\tilde {h} + 1}\right) ^ {2} \left\| v ^ {(k)} - x ^ {(k)} \right\| ^ {2} \right]. \\ \end{array}
$$
Hence we obtain
$$
\begin{array}{l} \tilde {h} \times (\operatorname {e r r}) = (\alpha - \beta) \left(\tilde {h} + 1\right) \left\| x ^ {(k + 1)} - z ^ {(k)} \right\| ^ {2} - (\beta + \gamma) \frac {\tilde {h}}{2 \tilde {h} + 1} \left\| v ^ {(k)} - x ^ {(k)} \right\| ^ {2} \\ - (\beta + \gamma) \frac {\tilde {h} + 1}{\tilde {h}} \frac {2 \tilde {h} + 1}{\tilde {h}} \left\| x ^ {(k + 1)} - \frac {\tilde {h} + 1}{2 \tilde {h} + 1} x ^ {(k)} - \frac {\tilde {h}}{2 \tilde {h} + 1} v ^ {(k)} \right\| ^ {2}. \\ \end{array}
$$