SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
G \geq \max  \left\{8 \max  \left\{L ^ {1 / \alpha - 1 / 2}, 1 \right\} \sqrt {L \left(\left(f \left(x _ {0}\right) - f ^ {*}\right) + \left\| x _ {0} - x ^ {*} \right\| ^ {2}\right)}, \| \nabla f (x _ {0}) \| \right\}. \tag {19}
$$
Choose  $\eta \leq \min \left\{\frac{1}{16L^{3 - 2 / \alpha}},\frac{1}{2L}\right\}$ . Then the iterates of Algorithm 1 satisfy
$$
f (x _ {T}) - f ^ {*} \leq \frac {4 (f (x _ {0} - f ^ {*}) + 4 \| x _ {0} - x ^ {*} \| ^ {2}}{\eta T ^ {2} + 4}.
$$
Note that when  $\alpha = 2$ , i.e.,  $\ell$  is sub-quadratic, Theorem C.1 reduces to Theorem 4.4 which chooses  $\eta \leq \min \left\{\frac{1}{16L^2}, \frac{1}{2L}\right\}$ . When  $\alpha = 1$ , i.e.,  $\ell$  is sub-linear, the above theorem chooses  $\eta \leq \frac{1}{16L}$  as in the classical textbook analysis up to a numerical constant factor.
Throughout this section, we will assume  $f$  is convex and  $\ell$ -smooth, and consider the parameter choices in Theorem C.1, unless explicitly stated. Note that since  $f$  is  $\ell$ -smooth, it is also  $(r,m)$ -smooth with  $m(u) = \ell (u + G)$  and  $r(u) = \frac{G}{\ell(u + G)}$  by Proposition 3.2. Note that  $m(G) = \ell (2G) = L$  and  $r(G) = G / L$ . Then the stepsize satisfies  $\eta \leq 1 / (2L) \leq \min \left\{\frac{2}{m(G)},\frac{r(G)}{2G}\right\}$ .
Before proving Theorem C.1, we first present several additional useful lemmas. To start with, we provide two lemmas regarding the weights  $\{A_t\}_{t \geq 0}$  and  $\{B_t\}_{t \geq 0}$  used in Algorithm 1. The lemma below states that  $B_t = \Theta(t^2)$ .
Lemma C.2. The weights  $\{B_t\}_{t \geq 0}$  in Algorithm 1 satisfy  $\frac{1}{4} t^2 \leq B_t \leq t^2$  for all  $t \geq 0$ .
Proof of Lemma C.2. We prove this lemma by induction. First note that the inequality obviously holds for  $B_0 = 0$ . Suppose its holds up to  $t$ . Then we have
$$
B _ {t + 1} = B _ {t} + \frac {1}{2} (1 + \sqrt {4 B _ {t} + 1}) \geq \frac {1}{4} t ^ {2} + \frac {1}{2} (1 + \sqrt {t ^ {2} + 1}) \geq \frac {1}{4} (t + 1) ^ {2}.
$$
$$
B _ {t + 1} = B _ {t} + \frac {1}{2} (1 + \sqrt {4 B _ {t} + 1}) \leq t ^ {2} + \frac {1}{2} (1 + \sqrt {4 t ^ {2} + 1}) \leq (t + 1) ^ {2}.
$$
Lemma C.2 implies the following useful lemma.
Lemma C.3. The weights  $\{A_t\}_{t \geq 0}$  in Algorithm 1 satisfy that
$$
\left(1 - \frac {A _ {t}}{A _ {t + 1}}\right) \frac {1}{A _ {t}} \sum_ {s = 0} ^ {t - 1} \sqrt {A _ {s + 1}} \left(A _ {s + 1} - A _ {s} - 1\right) \leq 4.
$$
Proof of Lemma C.3. First, note that it is easy to verify that  $A_{s + 1} - A_s - 1 = B_{s + 1} - B_s - 1 \geq 0$ , which implies each term in the LHS of the above inequality is non-negative. Then we have
$$
\begin{array}{l} \left(1 - \frac {A _ {t}}{A _ {t + 1}}\right) \frac {1}{A _ {t}} \sum_ {s = 0} ^ {t - 1} \sqrt {A _ {s + 1}} \left(A _ {s + 1} - A _ {s} - 1\right) \\ \leq \frac {1}{A _ {t + 1} \sqrt {A _ {t}}} \left(A _ {t + 1} - A _ {t}\right) \sum_ {s = 0} ^ {t - 1} \left(A _ {s + 1} - A _ {s} - 1\right) \quad \left(A _ {t} \geq A _ {s + 1}\right) \\ = \frac {1}{A _ {t + 1} \sqrt {A _ {t}}} \left(B _ {t + 1} - B _ {t}\right) \sum_ {s = 0} ^ {t - 1} \left(B _ {s + 1} - B _ {s} - 1\right) \quad (A _ {s} = B _ {s} + 1 / \eta) \\ = \frac {1}{A _ {t + 1} \sqrt {A _ {t}}} \cdot \frac {1}{2} (1 + \sqrt {4 B _ {t} + 1}) \sum_ {s = 0} ^ {t - 1} \left(- 1 + \frac {1}{2} (1 + \sqrt {4 B _ {s} + 1})\right) \quad (\text {b y}) \\ \leq 8 \frac {1}{(t + 1) ^ {2} t} \cdot (t + 1) \frac {t ^ {2}}{2} \quad (\text {b y} A _ {t} \geq B _ {t} \text {a n d L e m m a C . 2}) \\ \leq 4. \\ \end{array}
$$
The following lemma summarizes the results in the classical potential function analysis of NAG in [d'Aspremont et al., 2021]. In order to not deal with the generalized smoothness condition for now, we directly assume the inequality (20) holds in the lemma, which will be proved later under the generalized smoothness condition.
Lemma C.4. For any  $t \geq 0$ , if the following inequality holds,
$$
f \left(y _ {t}\right) + \left\langle \nabla f \left(y _ {t}\right), x _ {t + 1} - y _ {t} \right\rangle + \frac {1}{2 \eta} \left\| x _ {t + 1} - y _ {t} \right\| ^ {2} \geq f \left(x _ {t + 1}\right), \tag {20}
$$
then we can obtain
$$
A _ {t + 1} \left(f \left(x _ {t + 1}\right) - f ^ {*}\right) + \frac {1}{2 \eta} \left\| z _ {t + 1} - x ^ {*} \right\| ^ {2} \leq A _ {t} \left(f \left(x _ {t}\right) - f ^ {*}\right) + \frac {1}{2 \eta} \left\| z _ {t} - x ^ {*} \right\| ^ {2}. \tag {21}
$$
Proof of Lemma C.4. These derivations below can be found in [d'Aspremont et al., 2021]. We present them here for completeness.
First, since  $f$  is convex, the convexity between  $x^{*}$  and  $y_{t}$  gives
$$
f ^ {*} \geq f (y _ {t}) + \langle \nabla f (y _ {t}), x ^ {*} - y _ {t} \rangle .
$$
Similarly the convexity between  $x_{t}$  and  $y_{t}$  gives
$$
f (x _ {t}) \geq f (y _ {t}) + \langle \nabla f (y _ {t}), x _ {t} - y _ {t} \rangle .
$$
Combining the above two inequalities as well as (20) assumed in this lemma, we have
$$
\begin{array}{l} 0 \geq \left(A _ {t + 1} - A _ {t}\right) \left(f \left(y _ {t}\right) - f ^ {*} + \left\langle \nabla f \left(y _ {t}\right), x ^ {*} - y _ {t}\right)\right) \\ + A _ {t} \left(f \left(y _ {t}\right) - f \left(x _ {t}\right) + \left\langle \nabla f \left(y _ {t}\right), x _ {t} - y _ {t} \right\rangle\right) \\ + A _ {t + 1} \left(f \left(x _ {t + 1}\right) - f \left(y _ {t}\right) - \left\langle \nabla f \left(y _ {t}\right), x _ {t + 1} - y _ {t} \right\rangle - \frac {1}{2 \eta} \left\| x _ {t + 1} - y _ {t} \right\| ^ {2}\right). \tag {22} \\ \end{array}
$$
$$
\begin{array}{l} \frac {1}{2 \eta} \left(\left\| z _ {t + 1} - x ^ {*} \right\| ^ {2} - \left\| z _ {t} - x ^ {*} \right\| ^ {2}\right) \\ = \frac {1}{2 \eta} \left(\left\| z _ {t + 1} - z _ {t} \right\| ^ {2} + 2 \langle z _ {t + 1} - z _ {t}, z _ {t} - x ^ {*} \rangle\right) \\ = \frac {1}{2 \eta} \left(\eta^ {2} \left(A _ {t + 1} - A _ {t}\right) ^ {2} \| \nabla f (y _ {t}) \| ^ {2} - 2 \eta \left(A _ {t + 1} - A _ {t}\right) \langle \nabla f (y _ {t}), z _ {t} - x ^ {*} \rangle\right) \\ = \frac {\eta}{2} \left(A _ {t + 1} - A _ {t}\right) ^ {2} \| \nabla f \left(y _ {t}\right) \| ^ {2} - \left(A _ {t + 1} - A _ {t}\right) \langle \nabla f \left(y _ {t}\right), z _ {t} - x ^ {*} \rangle . \tag {23} \\ \end{array}
$$
$$
A _ {t + 1} x _ {t + 1} = A _ {t + 1} y _ {t} - \eta A _ {t + 1} \nabla f (y _ {t}) = A _ {t + 1} x _ {t} + \left(A _ {t + 1} - A _ {t}\right) \left(z _ {t} - x _ {t}\right) - \eta A _ {t + 1} \nabla f (y _ {t}).
$$
$$
\left(A _ {t + 1} - A _ {t}\right) z _ {t} = A _ {t + 1} x _ {t + 1} - A _ {t} x _ {t} + \eta A _ {t + 1} \nabla f (y _ {t}).
$$
Plugging back in (23), we obtain
$$
\begin{array}{l} \frac {1}{2 \eta} \left(\left\| z _ {t + 1} - x ^ {*} \right\| ^ {2} - \left\| z _ {t} - x ^ {*} \right\| ^ {2}\right) \\ = \frac {\eta}{2} \left(A _ {t + 1} - A _ {t}\right) ^ {2} \| \nabla f (y _ {t}) \| ^ {2} + \left(A _ {t + 1} - A _ {t}\right) \langle \nabla f (y _ {t}), x ^ {*} \rangle \\ + \left\langle - A _ {t + 1} x _ {t + 1} + A _ {t} x _ {t} - \eta A _ {t + 1} \nabla f (y _ {t}), \nabla f (y _ {t}) \right\rangle . \\ \end{array}
$$
$$
\begin{array}{l} \left. \left(A _ {t + 1} - A _ {t}\right) \left\langle \nabla f (y _ {t}), x ^ {*} \right\rangle + \left\langle A _ {t} x _ {t} - A _ {t + 1} x _ {t + 1}, \nabla f (y _ {t}) \right\rangle \right. \\ = \frac {1}{2 \eta} \left(\left\| z _ {t + 1} - x ^ {*} \right\| ^ {2} - \left\| z _ {t} - x ^ {*} \right\| ^ {2}\right) + \eta \left(A _ {t + 1} - \frac {1}{2} \left(A _ {t + 1} - A _ {t}\right) ^ {2}\right) \| \nabla f (y _ {t}) \| ^ {2}. \\ \end{array}
$$
So we can reorganize (22) to obtain
$$
\begin{array}{l} 0 \geq A _ {t + 1} \left(f \left(x _ {t + 1}\right) - f ^ {*}\right) - A _ {t} \left(f \left(x _ {t}\right) - f ^ {*}\right) \\ + \left(A _ {t + 1} - A _ {t}\right) \left\langle \nabla f (y _ {t}), x ^ {*} \right\rangle + \left\langle A _ {t} x _ {t} - A _ {t + 1} x _ {t + 1}, \nabla f (y _ {t}) \right\rangle \\ - \frac {1}{2 \eta} A _ {t + 1} \| x _ {t + 1} - y _ {t} \| ^ {2} \\ = A _ {t + 1} \left(f \left(x _ {t + 1}\right) - f ^ {*}\right) - A _ {t} \left(f \left(x _ {t}\right) - f ^ {*}\right) \\ + \frac {1}{2 \eta} \left(\left\| z _ {t + 1} - x ^ {*} \right\| ^ {2} - \left\| z _ {t} - x ^ {*} \right\| ^ {2}\right) + \frac {\eta}{2} \left(A _ {t + 1} - \left(A _ {t + 1} - A _ {t}\right) ^ {2}\right) \| \nabla f (y _ {t}) \| ^ {2}. \\ \end{array}
$$
Then we complete the proof noting that it is easy to verify
$$
A _ {t + 1} - (A _ {t + 1} - A _ {t}) ^ {2} = B _ {t + 1} + \frac {1}{\eta} - (B _ {t + 1} - B _ {t}) ^ {2} = \frac {1}{\eta} \geq 0.
$$
In the next lemma, we show that if  $\| \nabla f(y_{t})\| \leq G$ , then the condition (20) assumed in Lemma C.4 is satisfied at time  $t$ .
Lemma C.5. For any  $t \geq 0$ , if  $\|\nabla f(y_t)\| \leq G$ , then we have  $\|\nabla f(x_{t+1})\| \leq G$ , and furthermore,
$$
f (y _ {t}) + \langle \nabla f (y _ {t}), x _ {t + 1} - y _ {t} \rangle + \frac {1}{2 \eta} \| x _ {t + 1} - y _ {t} \| ^ {2} \geq f (x _ {t + 1}).
$$
Proof of Lemma C.5. As discussed below Theorem C.1, the stepsize satisfies  $\eta \leq 1 / (2L) \leq \min \left\{\frac{2}{m(G)}, \frac{r(G)}{2G}\right\}$ . Therefore we can apply Lemma 4.1 to show  $\| \nabla f(x_{t+1}) \| \leq \| \nabla f(y_t) \| \leq G$ . For the second part, note that  $\| x_{t+1} - y_t \| = \eta \| \nabla f(y_t) \| \leq \frac{G}{2L} \leq r(G)$ , we can apply Lemma 3.3 to show
$$
\begin{array}{l} f \left(x _ {t + 1}\right) \leq f \left(y _ {t}\right) + \left\langle \nabla f \left(y _ {t}\right), x _ {t + 1} - y _ {t} \right\rangle + \frac {L}{2} \left\| x _ {t + 1} - y _ {t} \right\| ^ {2} \\ \leq f (y _ {t}) + \langle \nabla f (y _ {t}), x _ {t + 1} - y _ {t} \rangle + \frac {1}{2 \eta} \left\| x _ {t + 1} - y _ {t} \right\| ^ {2}. \\ \end{array}
$$
With Lemma C.4 and Lemma C.5, we can show that  $\| \nabla f(y_t) \| \leq G$  for all  $t \geq 0$ , as in the lemma below.
Lemma C.6. For all  $t \geq 0$ ,  $\| \nabla f(y_t) \| \leq G$ .
Proof of Lemma C.6. We will prove this lemma by induction. First, by Lemma 3.5 and the choice of  $G$ , it is easy to verify that  $\| \nabla f(x_0) \| \leq G$ . Then for any fixed  $t \geq 0$ , suppose that  $\| \nabla f(x_s) \| \leq G$  for all  $s < t$ . Then by Lemma C.4 and Lemma C.5, we know that  $\| \nabla f(x_s) \| \leq G$  for all  $0 \leq s \leq t$ , and that for all  $s < t$ ,
$$
A _ {s + 1} \left(f \left(x _ {s + 1}\right) - f ^ {*}\right) + \frac {1}{2 \eta} \left\| z _ {s + 1} - x ^ {*} \right\| ^ {2} \leq A _ {s} \left(f \left(x _ {s}\right) - f ^ {*}\right) + \frac {1}{2 \eta} \left\| z _ {s} - x ^ {*} \right\| ^ {2}. \tag {24}
$$
By telescoping (24), we have for all  $0 \leq s < t$ ,
$$
f \left(x _ {s + 1}\right) - f ^ {*} \leq \frac {1}{\eta A _ {s + 1}} \left(\left(f \left(x _ {0}\right) - f ^ {*}\right) + \left\| z _ {0} - x ^ {*} \right\| ^ {2}\right). \tag {25}
$$
For  $0 \leq s \leq t$ , since  $\| \nabla f(x_s) \| \leq G$ , then Lemma 3.5 implies
$$
\left\| \nabla f \left(x _ {s}\right) \right\| ^ {2} \leq 2 L \left(f \left(x _ {s}\right) - f ^ {*}\right). \tag {26}
$$
Note that by Algorithm 1, we have
$$
z _ {t} - x _ {t} = \frac {A _ {t - 1}}{A _ {t}} (z _ {t - 1} - x _ {t - 1}) - \eta (A _ {t} - A _ {t - 1}) \nabla f (y _ {t - 1}) + \eta \nabla f (y _ {t - 1}).
$$
Thus we can obtain
$$
z _ {t} - x _ {t} = - \frac {1}{A _ {t}} \sum_ {s = 1} ^ {t - 1} \eta A _ {s + 1} (A _ {s + 1} - A _ {s} - 1) \nabla f (y _ {s}).
$$
$$
y _ {t} - x _ {t} = - (1 - \frac {A _ {t}}{A _ {t + 1}}) \frac {1}{A _ {t}} \sum_ {s = 1} ^ {t - 1} \eta A _ {s + 1} (A _ {s + 1} - A _ {s} - 1) \nabla f (y _ {s}).
$$
$$
\| y _ {t} - x _ {t} \| \leq \left(1 - \frac {A _ {t}}{A _ {t + 1}}\right) \frac {1}{A _ {t}} \sum_ {s = 1} ^ {t - 1} \eta A _ {s + 1} \left(A _ {s + 1} - A _ {s} - 1\right) \| \nabla f (y _ {s}) \| =: \mathcal {I}.
$$
Since  $\| \nabla f(y_s) \| \leq G$  and  $\| x_{s+1} - y_s \| = \| \eta \nabla f(y_s) \| \leq r(G)$  for  $s < t$ , by Lemma 3.3, we have
$$
\begin{array}{l} \mathcal {I} \leq \left(1 - \frac {A _ {t}}{A _ {t + 1}}\right) \frac {1}{A _ {t}} \sum_ {s = 1} ^ {t - 1} \eta A _ {s + 1} \left(A _ {s + 1} - A _ {s} - 1\right) \left(\| \nabla f (x _ {s + 1}) \| + \eta L \| \nabla f (y _ {s}) \|\right) \\ \leq \eta L \mathcal {I} + \left(1 - \frac {A _ {t}}{A _ {t + 1}}\right) \frac {1}{A _ {t}} \sum_ {s = 1} ^ {t - 1} \eta A _ {s + 1} \left(A _ {s + 1} - A _ {s} - 1\right) \| \nabla f (x _ {s + 1}) \|. \\ \end{array}
$$
$$
\begin{array}{l} \left\| y _ {t} - x _ {t} \right\| \\ \leq \mathcal {I} \leq \frac {1}{1 - \eta L} \left(1 - \frac {A _ {t}}{A _ {t + 1}}\right) \frac {1}{A _ {t}} \sum_ {s = 1} ^ {t - 1} \eta A _ {s + 1} \left(A _ {s + 1} - A _ {s} - 1\right) \| \nabla f (x _ {s + 1}) \| \\ \leq \frac {1}{1 - \eta L} \left(1 - \frac {A _ {t}}{A _ {t + 1}}\right) \frac {1}{A _ {t}} \sum_ {s = 1} ^ {t - 1} \eta A _ {s + 1} \left(A _ {s + 1} - A _ {s} - 1\right) \sqrt {2 L \left(f \left(x _ {s + 1}\right) - f ^ {*}\right)} (by(26)) \\ \leq \frac {1}{1 - \eta L} \left(1 - \frac {A _ {t}}{A _ {t + 1}}\right) \frac {1}{A _ {t}} \sum_ {s = 1} ^ {t - 1} \eta A _ {s + 1} \left(A _ {s + 1} - A _ {s} - 1\right) \sqrt {\frac {2 L}{A _ {s + 1}} \cdot \frac {1}{\eta} \left(\left(f \left(x _ {0}\right) - f ^ {*}\right) + \left\| z _ {0} - x ^ {*} \right\| ^ {2}\right)} (by(25)) \\ = \frac {2 \sqrt {\eta L}}{1 - \eta L} \left(1 - \frac {A _ {t}}{A _ {t + 1}}\right) \frac {1}{A _ {t}} \sum_ {s = 1} ^ {t - 1} \sqrt {A _ {s + 1}} \left(A _ {s + 1} - A _ {s} - 1\right) \sqrt {\left(f \left(x _ {0}\right) - f ^ {*}\right) + \left\| z _ {0} - x ^ {*} \right\| ^ {2}} \\ \leq \frac {8 \sqrt {\eta}}{1 - \eta L} \sqrt {L \left(\left(f \left(x _ {0}\right) - f ^ {*}\right) + \left\| z _ {0} - x ^ {*} \right\| ^ {2}\right)} \quad (\text {b y}) \\ \leq \frac {1}{2 L ^ {3 / 2 - 1 / \alpha}} \cdot L ^ {1 / 2 - 1 / \alpha} G = \frac {G}{2 L} \leq r (G). \quad (\text {b y}) \\ \end{array}
$$
Since  $\| \nabla f(x_{t})\| \leq G$  and we just showed  $\| x_{t} - y_{t}\| \leq r(G)$ , by Lemma 3.3, we have
$$
\begin{array}{l} \| \nabla f (y _ {t}) \| \leq \| \nabla f (x _ {t}) \| + L \| y _ {t} - x _ {t} \| \\ \leq \sqrt {\frac {2 L}{\eta A _ {t}} \left(\left(f \left(x _ {0}\right) - f ^ {*}\right) + \left\| z _ {0} - x ^ {*} \right\| ^ {2}\right)} + L \cdot \frac {G}{2 L} \quad \tag {by(26) and (25)} \\ \leq G \left(\frac {1}{4} + \frac {1}{2}\right) \leq G. \quad (\text {b y} A _ {t} \geq 1 / \eta \text {a n d c h o i c e o f} G) \\ \end{array}
$$
Then we complete the induction as well as the proof.
With the three lemmas above, it is straight forward to prove Theorem C.1.
Proof of Theorem C.1. Combining Lemmas C.4, C.5, and C.6, we know the following inequality holds for all  $t \geq 0$ .
$$
A _ {t + 1} (f (x _ {t + 1}) - f ^ {*}) + \frac {1}{2 \eta} \left\| z _ {t + 1} - x ^ {*} \right\| ^ {2} \leq A _ {t} (f (x _ {t}) - f ^ {*}) + \frac {1}{2 \eta} \left\| z _ {t} - x ^ {*} \right\| ^ {2},
$$
Then by telescoping, we directly complete the proof.
D Analysis of NAG for strongly convex functions
In this section, we provide the convergence analysis of the modified version of Nesterov's accelerated gradient method for  $\mu$ -strongly-convex functions defined in Algorithm 2.
The convergence results is formally presented in the following theorem.
Theorem D.1. Suppose  $f$  is  $\mu$ -strongly-convex and  $\ell$ -smooth. For  $\alpha \in (0,2]$ , if  $\ell(u) = o(u^{\alpha})$ , i.e.,  $\lim_{u\to \infty}\ell (u) / u^{\alpha} = 0$ , then there must exist a constant  $G$  such that for  $L\coloneqq \ell (2G)$ , we have
$$
G \geq 8 \max  \left\{L ^ {1 / \alpha - 1 / 2}, 1 \right\} \sqrt {L \left(\left(f \left(x _ {0}\right) - f ^ {*}\right) + \mu \left\| z _ {0} - x ^ {*} \right\| ^ {2}\right) / \min  \left\{\mu , 1 \right\}}. \tag {27}
$$
If we choose
$$
\eta \leq \min  \left\{\frac {1}{1 4 4 L ^ {3 - 2 / \alpha} \log^ {4} \left(e + \frac {1 4 4 L ^ {3 - 2 / \alpha}}{\mu}\right)}, \frac {1}{2 L} \right\}. \tag {28}
$$
Algorithm 2: NAG for  $\mu$ -strongly-convex functions
input A  $\mu$ -strongly-convex and  $\ell$ -smooth function  $f$ , stepsize  $\eta$ , initial point  $x_0$
1: Initialize  $z_0 = x_0$ ,  $B_0 = 0$ , and  $A_0 = 1 / (\eta \mu)$ .
2: for  $t = 0, \ldots$  do
$$
3: \quad B _ {t + 1} = \frac {2 B _ {t} + 1 + \sqrt {4 B _ {t} + 4 \eta \mu B _ {t} ^ {2} + 1}}{2 (1 - \eta \mu)}
$$
4:  $A_{t + 1} = B_{t + 1} + \frac{1}{\eta\mu}$
5:  $\tau_{t} = \frac{(A_{t + 1} - A_{t})(1 + \eta\mu A_{t})}{A_{t + 1} + 2\eta\mu A_{t}A_{t + 1} - \eta\mu A_{t}^{2}}$  and  $\delta_t = \frac{A_{t + 1} - A_t}{1 + \eta\mu A_{t + 1}}$
6:  $y_{t} = x_{t} + \tau_{t}(z_{t} - x_{t})$
7:  $x_{t + 1} = y_t - \eta \nabla f(y_t)$
8:  $z_{t + 1} = (1 - \eta \mu \delta_t)z_t + \eta \mu \delta_t y_t - \eta \delta_t\nabla f(y_t)$
9: end for
The iterates generated by Algorithm 2 satisfy
$$
f (x _ {T}) - f ^ {*} \leq \frac {(1 - \sqrt {\eta \mu}) ^ {T - 1} (f (x _ {0} - f ^ {*}) + \mu \| z _ {0} - x ^ {*} \| ^ {2})}{\eta \mu + (1 - \sqrt {\eta \mu}) ^ {T - 1}}.
$$
The above theorem gives a gradient complexity of  $\mathcal{O}\left(\frac{1}{\sqrt{\eta\mu}}\log (1 / \epsilon)\right)$ . Note that Theorem 4.2 shows the complexity of GD is  $\mathcal{O}\left(\frac{1}{\eta\mu}\log (1 / \epsilon)\right)$ . It seems NAG gives a better rate at first glance. However, note that the choices of  $G,L,\eta$  in these two theorems are different, it is less clear whether NAG accelerates the optimization in this setting. Below, we informally show that, if  $\ell (u) = o(\sqrt{u})$ , the rate we obtain for NAG is faster than that for GD.
For simplicity, we informally assume  $\ell(u) \asymp x^{\rho}$  with  $\rho \in (0,1)$ . Let  $G_0 = \|\nabla f(x_0)\|$ . Then for GD, by Theorem 4.2, we have  $\eta_{\mathrm{gd}}\mu \asymp \mu/\ell(G_0) \asymp \mu/G_0^\rho$ . For NAG, since  $\ell$  is sub-linear we can choose  $\alpha = 1$  in the theorem statement. Since  $f$  is  $\mu$ -strongly-convex, by standard results, we can show that  $f(x_0) - f^* \leq \frac{1}{\mu} G_0^2$  and  $\|z_0 - x^*\| \leq \frac{1}{\mu} G_0$ . Thus the requirement of  $G$  in (27) can be simplified as  $G \gtrsim \ell(G) \cdot G_0/\mu$ , which is satisfied if choosing  $G \asymp (G_0/\mu)^{1/(1-\rho)}$ . Then we also have  $\eta_{\mathrm{nag}} \asymp \frac{1}{\ell(G)} \asymp (\mu/G_0)^{\rho/(1-\rho)}$ . Thus  $\sqrt{\eta_{\mathrm{nag}}\mu} \asymp (\mu/G_0^\rho)^{1/(2-2\rho)}$ . This means whenever  $1/(2-2\rho) < 1$ , i.e.,  $0 \leq \rho < 1/2$ , we have  $\sqrt{\eta_{\mathrm{nag}}\mu} \gtrsim \eta_{\mathrm{gd}}\mu$ , which implies the rate we obtain for NAG is faster than that for GD.
In what follows, we will provide the proof of Theorem D.1. We will always use the parameter choices in the theorem throughout this section.
D.1 Useful lemmas
In this part, we provide several useful lemmas for proving Theorem D.1. To start with, the following two lemmas provide two useful inequalities.
Lemma D.2. For any  $0 \leq u \leq 1$ , we have  $\log(1 + u) \geq \frac{1}{2} u$ .
Lemma D.3. For all  $0 < p \leq 1$  and  $t \geq 0$ , we have
$$
t \leq \frac {2}{\sqrt {p}} \log (e + \frac {1}{p}) (p (1 + \sqrt {p}) ^ {t} + 1).
$$
Proof of Lemma D.3. Let
$$
f (t) = \frac {2}{\sqrt {p}} \log (e + \frac {1}{p}) (p (1 + \sqrt {p}) ^ {t} + 1) - t.
$$
It is obvious that  $f(t) \geq 0$  for  $t \leq \frac{2}{\sqrt{p}} \log \left( e + \frac{1}{p} \right)$ . For  $t > \frac{2}{\sqrt{p}} \log \left( e + \frac{1}{p} \right)$ , we have
$$
\begin{array}{l} f ^ {\prime} (t) = 2 \sqrt {p} \log \left(e + \frac {1}{p}\right) \log \left(1 + \sqrt {p}\right) \left(1 + \sqrt {p}\right) ^ {t} - 1 \\ \geq p (1 + \sqrt {p}) ^ {t} - 1 (by Lemma D.2) \\ = p \exp (t \log (1 + \sqrt {p})) - 1 \\ \geq p \exp (t \sqrt {p} / 2) - 1 (by Lemma D.2) \\ \geq p (e + 1 / p) - 1 \geq 0. \quad (\text {s i n c e} t > \frac {2}{\sqrt {p}} \log \left(e + \frac {1}{p}\right)) \\ \end{array}
$$
Thus  $f$  is non-decreasing and
$$
f (t) \geq f \left(\frac {2}{\sqrt {p}} \log \left(e + \frac {1}{p}\right)\right) \geq 0.
$$
In the next four lemmas, we provide several useful inequalities regarding the weights  $\{A_t\}_{t \geq 0}$  and  $\{B_t\}_{t \geq 0}$  used in Algorithm 2.
Lemma D.4. For all  $s \leq t$ , we have
$$
\frac {B _ {t + 1} - B _ {t}}{B _ {t + 1}} \cdot \frac {B _ {s + 1} - B _ {s}}{1 + \eta \mu B _ {s + 1}} \leq 1,
$$
which implies  $\tau_t\cdot \delta_s\leq 1$
Proof of Lemma D.4. By Algorithm 2, it is easy to verify
$$
\left(B _ {s + 1} - B _ {s}\right) ^ {2} = B _ {s + 1} \left(1 + \eta \mu B _ {s + 1}\right).
$$
$$
B _ {s} = B _ {s + 1} - \sqrt {B _ {s + 1} (1 + \eta \mu B _ {s + 1})}.
$$
$$
\frac {B _ {t}}{B _ {t + 1}} = 1 - \sqrt {\eta \mu + \frac {1}{B _ {t + 1}}} \geq 1 - \sqrt {\eta \mu + \frac {1}{B _ {s + 1}}} = \frac {B _ {s}}{B _ {s + 1}},
$$
where in the inequality, we use the fact that  $B_{s}$  is non-decreasing with  $s$ . Therefore
$$
\frac {B _ {t + 1} - B _ {t}}{B _ {t + 1}} \cdot \frac {B _ {s + 1} - B _ {s}}{1 + \eta \mu B _ {s + 1}} \leq \frac {B _ {s + 1} - B _ {s}}{B _ {s + 1}} \cdot \frac {B _ {s + 1} - B _ {s}}{1 + \eta \mu B _ {s + 1}} = 1.
$$
$$
\begin{array}{l} \tau_ {t} \cdot \delta_ {s} = \frac {\left(A _ {t + 1} - A _ {t}\right) \left(1 + \eta \mu A _ {t}\right)}{A _ {t + 1} + 2 \eta \mu A _ {t} A _ {t + 1} - \eta \mu A _ {t} ^ {2}} \cdot \frac {A _ {s + 1} - A _ {s}}{1 + \eta \mu A _ {s + 1}} \\ \leq \frac {A _ {t + 1} - A _ {t}}{A _ {t + 1}} \cdot \frac {A _ {s + 1} - A _ {s}}{1 + \eta \mu A _ {s + 1}} \quad (\text {b y} A _ {t + 1} \geq A _ {t}) \\ = \frac {B _ {t + 1} - B _ {t}}{A _ {t + 1}} \cdot \frac {B _ {s + 1} - B _ {s}}{1 + \eta \mu A _ {s + 1}} \quad (\text {b y} A _ {s + 1} - A _ {s} = B _ {s + 1} - B _ {s}) \\ \leq \frac {B _ {t + 1} - B _ {t}}{B _ {t + 1}} \cdot \frac {B _ {s + 1} - B _ {s}}{1 + \eta \mu B _ {s + 1}} \leq 1. \quad (\text {b y} A _ {s + 1} \geq B _ {s + 1}) \\ \end{array}
$$
Lemma D.5. If  $0 < \eta \mu < 1$ , then for any  $t \geq 1$ , we have
$$
\frac {B _ {t}}{1 - \sqrt {\eta \mu}} \leq B _ {t + 1} \leq \frac {3 B _ {t}}{1 - \eta \mu}.
$$
$$
B _ {t} \geq \frac {1}{(1 - \sqrt {\eta \mu}) ^ {t - 1}} \geq (1 + \sqrt {\eta \mu}) ^ {t - 1}.
$$
Proof of Lemma D.5. For  $t \geq 1$ , we have  $B_t \geq 1$  thus
$$
B _ {t + 1} = \frac {2 B _ {t} + 1 + \sqrt {4 B _ {t} + 4 \eta \mu B _ {t} ^ {2} + 1}}{2 (1 - \eta \mu)} \leq \frac {2 B _ {t} + 1}{1 - \eta \mu} \leq \frac {3 B _ {t}}{1 - \mu \eta}.
$$
$$
\begin{array}{l} B _ {t + 1} = \frac {2 B _ {t} + 1 + \sqrt {4 B _ {t} + 4 \eta \mu B _ {t} ^ {2} + 1}}{2 (1 - \eta \mu)} \\ \geq \frac {2 B _ {t} + \sqrt {(2 B _ {t} \sqrt {\eta \mu}) ^ {2}}}{2 (1 - \eta \mu)} \\ = \frac {B _ {t}}{1 - \sqrt {\eta \mu}}. \\ \end{array}
$$
$$
B _ {t} \geq \left(\frac {1}{1 - \sqrt {\eta \mu}}\right) ^ {t - 1} B _ {1} \geq \left(\frac {1}{1 - \sqrt {\eta \mu}}\right) ^ {t - 1} \geq (1 + \sqrt {\eta \mu}) ^ {t - 1}.
$$
Lemma D.6. For  $0 < \eta \mu < 1$  and  $t \geq 1$ , we have
$$
\sum_ {s = 0} ^ {t} \sqrt {B _ {s}} \leq (1 - \eta \mu) B _ {t + 1} \leq 3 B _ {t}.
$$
Proof of Lemma D.6.
$$
\begin{array}{l} B _ {t + 1} = \frac {2 B _ {t} + 1 + \sqrt {4 B _ {t} + 4 \eta \mu B _ {t} ^ {2} + 1}}{2 (1 - \eta \mu)} \\ \geq B _ {t} + \frac {\sqrt {B _ {t}}}{1 - \eta \mu} \\ \geq \dots \\ \geq \sum_ {s = 0} ^ {t} \frac {\sqrt {B _ {s}}}{1 - \eta \mu}. \\ \end{array}
$$
Combined with Lemma D.5, we have the desired result.
Lemma D.7. For  $t \geq 1$ , we have
$$
\sum_ {s = 0} ^ {t - 1} \frac {\sqrt {A _ {s + 1}}}{A _ {t}} \leq 3 + 4 \log (e + \frac {1}{\eta \mu}).
$$
Proof of Lemma D.7. By Lemma D.5, we have
$$
A _ {t} = B _ {t} + \frac {1}{\eta \mu} \geq (1 + \sqrt {\eta \mu}) ^ {t - 1} + \frac {1}{\eta \mu}. \tag {29}
$$
$$
\begin{array}{l} \sum_ {s = 0} ^ {t - 1} \frac {\sqrt {A _ {s + 1}}}{A _ {t}} = \sum_ {s = 0} ^ {t - 1} \frac {\sqrt {B _ {s + 1} + 1 / (\eta \mu)}}{A _ {t}} \\ \leq \sum_ {s = 0} ^ {t - 1} \frac {\sqrt {B _ {s + 1}}}{A _ {t}} + \frac {t}{\sqrt {\eta \mu} A _ {t}} \\ \leq 3 + \frac {1}{\sqrt {\eta \mu} A _ {t}} \cdot \frac {2}{\sqrt {\eta \mu}} \log (e + \frac {1}{\eta \mu}) (\eta \mu (1 + \sqrt {\eta \mu}) ^ {t} + 1) \\ \end{array}
$$
(by Lemma D.6 and Lemma D.3)
$$
\leq 3 + 4 \log \left(e + \frac {1}{\eta \mu}\right). \quad (\text {b y})
$$
D.2 Proof of Theorem D.1