SlowGuess/NeurIPS_2023_OCR huggingface dataset:
With all the useful lemmas in the previous section, we proceed to prove Theorem D.1, for which we need several additional lemmas. First, similar to Lemma C.4, the following lemma summarizes the results in the classical potential function analysis of NAG for strongly convex functions in [d'Aspremont et al., 2021].
Lemma D.8. For any  $t \geq 0$ , if the following inequality holds
$$
f (y _ {t}) + \langle \nabla f (y _ {t}), x _ {t + 1} - y _ {t} \rangle + \frac {1}{2 \eta} \| x _ {t + 1} - y _ {t} \| ^ {2} \geq f (x _ {t + 1}),
$$
$$
A _ {t + 1} (f (x _ {t + 1}) - f ^ {*}) + \frac {1 + \eta \mu A _ {t + 1}}{2 \eta} \| z _ {t + 1} - x ^ {*} \| ^ {2} \leq A _ {t} (f (x _ {t}) - f ^ {*}) + \frac {1 + \eta \mu A _ {t}}{2 \eta} \| z _ {t} - x ^ {*} \| ^ {2}.
$$
Proof of Lemma D.8. These derivations can be found in d'Aspremont et al. [2021]. We present it here for completeness.
The strong convexity between  $x^{*}$  and  $y_{t}$  gives
$$
f ^ {*} \geq f (y _ {t}) + \left\langle \nabla f (y _ {t}), x ^ {*} - y _ {t} \right\rangle + \frac {\mu}{2} \left\| x ^ {*} - y _ {t} \right\| ^ {2}.
$$
The convexity between  $x_{t}$  and  $y_{t}$  gives
Combining the above two inequalities and the one assumed in this lemma, we have
$$
\begin{array}{l} 0 \geq \left(A _ {t + 1} - A _ {t}\right) \left(f ^ {*} - f \left(y _ {t}\right) - \left\langle \nabla f \left(y _ {t}\right), x ^ {*} - y _ {t} \right\rangle - \frac {\mu}{2} \left\| x ^ {*} - y _ {t} \right\| ^ {2}\right) \\ + A _ {t} \left(f \left(y _ {t}\right) - f \left(x _ {t}\right) - \langle \nabla f \left(y _ {t}\right), x _ {t} - y _ {t} \rangle\right) \\ + A _ {t + 1} (f (x _ {t + 1}) - f (y _ {t}) - \langle \nabla f (y _ {t}), x _ {t + 1} - y _ {t} \rangle - \frac {1}{2 \eta} \left\| x _ {t + 1} - y _ {t} \right\| ^ {2}). \\ \end{array}
$$
Reorganizing we can obtain
$$
\begin{array}{l} A _ {t + 1} \left(f \left(x _ {t + 1}\right) - f ^ {*}\right) + \frac {1 + \eta \mu A _ {t + 1}}{2 \eta} \left\| z _ {t + 1} - x ^ {*} \right\| ^ {2} \\ \leq A _ {t} (f (x _ {t}) - f ^ {*}) + \frac {1 + \eta \mu A _ {t}}{2 \eta} \| z _ {t} - x ^ {*} \| ^ {2} \\ + \frac {\left(A _ {t} - A _ {t + 1}\right) ^ {2} - A _ {t + 1} - \eta \mu A _ {t + 1} ^ {2}}{1 + \eta \mu A _ {t + 1}} \frac {\eta}{2} \| \nabla f (y _ {t}) \| ^ {2} \\ - A _ {t} ^ {2} \frac {\left(A _ {t + 1} - A _ {t}\right) \left(1 + \eta \mu A _ {t}\right) \left(1 + \eta \mu A _ {t + 1}\right)}{\left(A _ {t + 1} + 2 \eta \mu A _ {t} A _ {t + 1} - \eta \mu A _ {t} ^ {2}\right) ^ {2}} \frac {\mu}{2} \| x _ {t} - z _ {t} \| ^ {2}. \\ \end{array}
$$
Then we complete the proof noting that
$$
\begin{array}{l} \left(A _ {t} - A _ {t + 1}\right) ^ {2} - A _ {t + 1} - \eta \mu A _ {t + 1} ^ {2} \\ = \left(B _ {t} - B _ {t + 1}\right) ^ {2} - B _ {t + 1} + \frac {1}{\eta \mu} - \eta \mu \left(B _ {t + 1} + 1 / (\eta \mu)\right) ^ {2} \\ = \eta \mu B _ {t + 1} ^ {2} + \frac {1}{\eta \mu} - \eta \mu B _ {t + 1} ^ {2} - 2 B _ {t + 1} - \frac {1}{\eta \mu} \\ = - 2 B _ {t + 1} \leq 0. \\ \end{array}
$$
Next, note that Lemma C.5 still holds in the strongly convex setting. We repeat it below for completeness.
Lemma D.9. For any  $t \geq 0$ , if  $\|\nabla f(y_t)\| \leq G$ , then we have  $\|\nabla f(x_{t+1})\| \leq G$ , and furthermore,
$$
f (y _ {t}) + \left\langle \nabla f (y _ {t}), x _ {t + 1} - y _ {t} \right\rangle + \frac {1}{2 \eta} \left\| x _ {t + 1} - y _ {t} \right\| ^ {2} \geq f (x _ {t + 1}).
$$
With Lemma D.8 and Lemma D.9, we will show that  $\| \nabla f(y_{t})\| \leq G$  for all  $t \geq 0$  by induction in the following lemma.
Lemma D.10. For all  $t \geq 0$ , we have  $\| \nabla f(y_t) \| \leq G$ .
Proof of Lemma D.10. We will prove this lemma by induction. First, by Lemma 3.5 and the choice of  $G$ , it is easy to verify that  $\| \nabla f(x_0) \| \leq G$ . Then for any fixed  $t \geq 0$ , suppose that  $\| \nabla f(x_s) \| \leq G$  for all  $s < t$ . Then by Lemma D.8 and Lemma D.9, we know that  $\| \nabla f(x_s) \| \leq G$  for all  $0 \leq s \leq t$ , and that for all  $s < t$ ,
$$
A _ {s + 1} \left(f \left(x _ {s + 1}\right) - f ^ {*}\right) + \frac {1 + \eta \mu A _ {s + 1}}{2 \eta} \left\| z _ {s + 1} - x ^ {*} \right\| ^ {2} \leq A _ {s} \left(f \left(x _ {s}\right) - f ^ {*}\right) + \frac {1 + \eta \mu A _ {s}}{2 \eta} \left\| z _ {s} - x ^ {*} \right\| ^ {2}. \tag {30}
$$
By telescoping (30), we have for all  $0 \leq s < t$
$$
f \left(x _ {s + 1}\right) - f ^ {*} \leq \frac {1}{A _ {s + 1} \eta \mu} \left(f \left(x _ {0}\right) - f ^ {*} + \mu \| z _ {0} - x ^ {*} \| ^ {2}\right). \tag {31}
$$
$$
\left\| \nabla f \left(x _ {s}\right) \right\| ^ {2} \leq 2 L \left(f \left(x _ {s}\right) - f ^ {*}\right). \tag {32}
$$
Note that by Algorithm 2, we have
$$
z _ {t} - x _ {t} = (1 - \eta \mu \delta_ {t - 1}) (1 - \tau_ {t - 1}) (z _ {t - 1} - x _ {t - 1}) + \eta (1 - \delta_ {t - 1}) \nabla f (y _ {t - 1}).
$$
$$
z _ {t} - x _ {t} = \eta \sum_ {s = 0} ^ {t - 1} (1 - \delta_ {s}) \nabla f (y _ {s}) \prod_ {i = s + 1} ^ {t - 1} (1 - \eta \mu \delta_ {i}) (1 - \tau_ {i}).
$$
$$
y _ {t} - x _ {t} = \eta \tau_ {t} \sum_ {s = 0} ^ {t - 1} (1 - \delta_ {s}) \nabla f (y _ {s}) \prod_ {i = s + 1} ^ {t - 1} (1 - \eta \mu \delta_ {i}) (1 - \tau_ {i}).
$$
$$
1 - \eta \mu \delta_ {i} = 1 - \frac {\eta \mu (A _ {i + 1} - A _ {i})}{1 + \eta \mu A _ {i + 1}} = \frac {1 + \eta \mu A _ {i}}{1 + \eta \mu A _ {i + 1}}
$$
$$
1 - \tau_ {i} = 1 - \frac {(A _ {i + 1} - A _ {i}) (1 + \eta \mu A _ {i})}{A _ {i + 1} + 2 \eta \mu A _ {i} A _ {i + 1} - \eta \mu A _ {i} ^ {2}} = \frac {A _ {i} (1 + \eta \mu A _ {i + 1})}{A _ {i + 1} + 2 \eta \mu A _ {i} A _ {i + 1} - \eta \mu A _ {i} ^ {2}} \leq \frac {A _ {i} (1 + \eta \mu A _ {i + 1})}{A _ {i + 1} (1 + \eta \mu A _ {i})}.
$$
$$
\| y _ {t} - x _ {t} \| \leq \eta \tau_ {t} \sum_ {s = 0} ^ {t - 1} (\delta_ {s} - 1) \frac {A _ {s + 1}}{A _ {t}} \| \nabla f (y _ {s}) \| \leq \eta \sum_ {s = 0} ^ {t - 1} \frac {A _ {s + 1}}{A _ {t}} \| \nabla f (y _ {s}) \| =: \mathcal {I},
$$
where the second inequality follows from Lemma D.4. We further control term  $\mathcal{I}$  by
$$
\begin{array}{l} \mathcal {I} \leq \eta \sum_ {s = 0} ^ {t - 1} \frac {A _ {s + 1}}{A _ {t}} \left(\| \nabla f (x _ {s + 1}) \| + \eta L \| \nabla f (y _ {s}) \|\right) \\ \leq \eta L \mathcal {I} + \eta \sum_ {s = 0} ^ {t - 1} \frac {A _ {s + 1}}{A _ {t}} \| \nabla f (x _ {s + 1}) \|. \\ \end{array}
$$
$$
\begin{array}{l} \left\| y _ {t} - x _ {t} \right\| \leq \frac {\eta}{1 - \eta L} \sum_ {s = 0} ^ {t - 1} \frac {A _ {s + 1}}{A _ {t}} \| \nabla f (x _ {s + 1}) \| \\ \leq \frac {\eta}{1 - \eta L} \sum_ {s = 0} ^ {t - 1} \frac {A _ {s + 1}}{A _ {t}} \sqrt {2 L \left(f \left(x _ {s + 1}\right) - f ^ {*}\right)} (by(32)) \\ \leq \frac {\eta}{1 - \eta L} \sum_ {s = 0} ^ {t - 1} \frac {A _ {s + 1}}{A _ {t}} \sqrt {2 L \cdot \frac {1}{A _ {s + 1} \eta \mu} \left(f \left(x _ {0}\right) - f ^ {*} + \mu \| z _ {0} - x ^ {*} \| ^ {2}\right)} \quad (\text {b y}) \\ = \frac {\sqrt {2 \eta L (f (x _ {0}) - f ^ {*} + \mu \| z _ {0} - x ^ {*} \| ^ {2})}}{(1 - \eta L) \sqrt {\mu}} \sum_ {s = 0} ^ {t - 1} \frac {\sqrt {A _ {s + 1}}}{A _ {t}} \\ \leq \frac {\sqrt {2 \eta L \left(f \left(x _ {0}\right) - f ^ {*} + \mu \left\| z _ {0} - x ^ {*} \right\| ^ {2}\right)}}{\left(1 - \eta L\right) \sqrt {\mu}} \left(3 + 4 \log \left(e + \frac {1}{\eta \mu}\right)\right). \quad (\text {b y}) \\ \leq \frac {\sqrt {\eta}}{1 - \eta L} \left(3 + 4 \log \left(e + \frac {1}{\eta \mu}\right)\right) \cdot \frac {G \cdot L ^ {1 / 2 - 1 / \alpha}}{4} (by(27)) \\ \leq \frac {3 + 4 \log \left(e + \frac {1}{\eta \mu}\right)}{\log^ {2} \left(e + \frac {1 4 4 L ^ {3 - 2 / \alpha}}{\mu}\right)} \cdot \frac {G}{2 4 L} (by(28)) \\ \leq \frac {G}{2 L} \leq r (G). \\ \end{array}
$$
$$
\begin{array}{l} \| \nabla f (y _ {t}) \| \leq \| \nabla f (x _ {t}) \| + L \| y _ {t} - x _ {t} \| \\ \leq \sqrt {\frac {2 L}{\eta \mu A _ {t}} \left(\left(f \left(x _ {0}\right) - f ^ {*}\right) + \mu \left\| z _ {0} - x ^ {*} \right\| ^ {2}\right)} + L \cdot \frac {G}{2 L} \quad (\text {b y}) \\ \leq G \left(\frac {1}{4} + \frac {1}{2}\right) \leq G. \quad (\text {b y} A _ {t} \geq 1 / (\eta \mu) \text {a n d (2 7)}) \\ \end{array}
$$
Proof of Theorem D.1. Combining Lemmas D.8, D.9, and D.10, we know the following inequality holds for all  $t \geq 0$ .
Then by telescoping, we get
$$
A _ {t} (f (x _ {t}) - f ^ {*}) + \frac {1 + \eta \mu A _ {t}}{2 \eta} \| z _ {t} - x ^ {*} \| ^ {2} \leq A _ {0} (f (x _ {0}) - f ^ {*}) + \frac {1 + \eta \mu A _ {0}}{2 \eta} \| z _ {0} - x ^ {*} \| ^ {2}.
$$
Finally, applying Lemma D.5, we have  $A_{t} = B_{t} + 1 / (\eta \mu) \geq 1 / (1 - \sqrt{\eta\mu})^{t - 1} + 1 / (\eta \mu)$ . Thus completes the proof.
E Analysis of GD for non-convex functions
In this section, we provide the proofs related to analysis of gradient descent for non-convex function, including those of Lemma 5.1 and Theorem 5.2.
Proof of Lemma 5.1. First, based on Corollary 3.6, we know  $\| \nabla f(x)\| \leq G < \infty$ . Also note that
$$
\left\| x ^ {+} - x \right\| = \| \eta \nabla f (x) \| \leq \eta G \leq G / L.
$$
Then by Lemma 3.3 and Remark 3.4, we have  $x^{+}\in \mathcal{X}$  and
$$
\begin{array}{l} f (x ^ {+}) \leq f (x) + \left\langle \nabla f (x _ {t}), x ^ {+} - x \right\rangle + \frac {L}{2} \left\| x ^ {+} - x \right\| ^ {2} \\ = f (x) - \eta (1 - \eta L / 2) \left\| \nabla f (x) \right\| ^ {2} \\ \leq f (x). \\ \end{array}
$$
Proof of Theorem 5.2. By Lemma 5.1, using induction, we directly obtain  $f(x_{t}) \leq f(x_{0})$  for all  $t \geq 0$ . Then by Corollary 3.6, we have  $\| \nabla f(x_{t}) \| \leq G$  for all  $t \geq 0$ . Following the proof of Lemma 5.1, we can similarly show
$$
f (x _ {t + 1}) - f (x _ {t}) \leq \eta (1 - \eta L / 2) - \frac {\eta}{2} \| \nabla f (x _ {t}) \| ^ {2} \leq - \frac {\eta}{2} \| \nabla f (x _ {t}) \| ^ {2}.
$$
Taking a summation over  $t < T$  and rearranging terms, we have
$$
\frac {1}{T} \sum_ {t <   T} \| \nabla f (x _ {t}) \| ^ {2} \leq \frac {2 (f (x _ {0}) - f (x _ {T}))}{\eta T} \leq \frac {2 (f (x _ {0}) - f ^ {*})}{\eta T}.
$$
F Analysis of SGD for non-convex functions
In this section, we provide the detailed convergence analysis of stochastic gradient descent for  $\ell$ -smooth and non-convex functions where  $\ell$  is sub-quadratic.
We first present some useful inequalities related to the parameter choices in Theorem 5.3.
Lemma F.1. Under the parameters choices in Theorem 5.3, the following inequalities hold.
$$
\eta G \sqrt {2 T} \leq 1 / 2, \quad \eta^ {2} \sigma L T \leq 1 / 2, \quad 1 0 0 \eta^ {2} T \sigma^ {2} L ^ {2} \leq \delta G ^ {2}.
$$
Proof of Lemma F.1. First note that by Corollary 3.6, we know
$$
G ^ {2} = 2 L F = 1 6 L \left(f \left(x _ {0}\right) - f ^ {*} + \sigma\right) / \delta \geq 1 6 L \sigma / \delta ,
$$
i.e.,  $\sigma L\leq G^2\delta /16$  . Then since we choose  $\eta \leq \frac{1}{4G\sqrt{T}}$  , we have
$$
\begin{array}{l} \eta G \sqrt {2 T} \leq \sqrt {2} / 4 \leq 1 / 2, \\ \eta^ {2} \sigma L T \leq \eta^ {2} T G ^ {2} \delta / 1 6 \leq \delta / 2 5 6 \leq 1 / 2, \\ 1 0 0 \eta^ {2} T \sigma^ {2} L ^ {2} \leq 1 0 0 \eta^ {2} T G ^ {4} \delta^ {2} / 2 5 6 \leq \delta G ^ {2}. \\ \end{array}
$$
Next, we show the useful lemma which bounds  $\mathbb{E}[f(x_{\tau}) - f^{*}]$  and  $\mathbb{E}\left[\sum_{t < \tau}\| \nabla f(x_t)\|^2\right]$  simultaneously.
Lemma F.2. Under the parameters choices in Theorem 5.3, the following inequality holds
$$
\mathbb {E} \left[ f (x _ {\tau}) - f ^ {*} + \frac {\eta}{2} \sum_ {t <   \tau} \| \nabla f (x _ {t}) \| ^ {2} \right] \leq f (x _ {0}) - f ^ {*} + \sigma .
$$
Proof of Lemma F.2. If  $t < \tau$ , by the definition of  $\tau$ , we know  $f(x_{t}) - f^{*} \leq F$  and  $\| \epsilon_t\| \leq \frac{G}{5\eta L}$  and the former also implies  $\| \nabla f(x_{t})\| \leq G$  by Corollary 3.6. Then we can bound
$$
\left\| x _ {t + 1} - x _ {t} \right\| = \eta \left\| g _ {t} \right\| \leq \eta \left(\left\| \nabla f (x _ {t}) \right\| + \left\| \epsilon_ {t} \right\|\right) \leq \eta G + \frac {G}{5 L} \leq \frac {G}{L},
$$
where we use the choice of  $\eta \leq \frac{1}{2L}$ . Then based on Lemma 3.3 and Remark 3.4, for any  $t < \tau$ , we have
$$
\begin{array}{l} f (x _ {t + 1}) - f (x _ {t}) \leq \left\langle \nabla f (x _ {t}), x _ {t + 1} - x _ {t} \right\rangle + \frac {L}{2} \left\| x _ {t + 1} - x _ {t} \right\| ^ {2} \\ = - \eta \big <   \nabla f (x _ {t}), g _ {t} \big > + \frac {\eta^ {2} L}{2} \left\| g _ {t} \right\| ^ {2} \\ \leq - \eta \| \nabla f (x _ {t}) \| ^ {2} - \eta \left\langle \nabla f (x _ {t}), \epsilon_ {t} \right\rangle + \eta^ {2} L \| \nabla f (x _ {t}) \| ^ {2} + \eta^ {2} L \| \epsilon_ {t} \| ^ {2} \\ \leq - \frac {\eta}{2} \| \nabla f (x _ {t}) \| ^ {2} - \eta \left\langle \nabla f (x _ {t}), \epsilon_ {t} \right\rangle + \eta^ {2} L \| \epsilon_ {t} \| ^ {2}, \tag {33} \\ \end{array}
$$
where the equality is due to (4); the second inequality uses  $g_{t} = \epsilon_{t} + \nabla f(x_{t})$  and Young's inequality  $\| y + z\| ^2\leq 2\| y\| ^2 +2\| z\| ^2$  for any vectors  $y,z$ ; and the last inequality chooses  $\eta \leq 1 / (2L)$ . Taking a summation over  $t < \tau$  and rearranging terms, we have
$$
f (x _ {\tau}) - f ^ {*} + \frac {\eta}{2} \sum_ {t <   \tau} \| \nabla f (x _ {t}) \| ^ {2} \leq f (x _ {0}) - f ^ {*} - \eta \sum_ {t <   \tau} \bigl \langle \nabla f (x _ {t}), \epsilon_ {t} \bigr \rangle + \eta^ {2} L \sum_ {t <   \tau} \| \epsilon_ {t} \| ^ {2}.
$$
Now we bound the last two terms on th RHS. First, for the last term, we have
$$
\mathbb {E} \left[ \sum_ {t <   \tau} \| \epsilon_ {t} \| ^ {2} \right] \leq \mathbb {E} \left[ \sum_ {t <   T} \| \epsilon_ {t} \| ^ {2} \right] \leq \sigma^ {2} T,
$$
where the first inequality uses  $\tau \leq T$  by its definition; and in the last inequality we use Assumption 4. For the cross term, note that  $\mathbb{E}_{t-1}\left[\langle \nabla f(x_t), \epsilon_t \rangle\right] = 0$  by Assumption 4. So this term is a sum of a martingale difference sequence. Since  $\tau$  is a stopping time, we can apply the optional stopping theorem to obtain
$$
\mathbb {E} \left[ \sum_ {t \leq \tau} \left\langle \nabla f \left(x _ {t}\right), \epsilon_ {t} \right\rangle \right] = 0. \tag {34}
$$
$$
\begin{array}{l} \mathbb {E} \left[ - \sum_ {t <   \tau} \bigl \langle \nabla f (x _ {t}), \epsilon_ {t} \bigr \rangle \right] = \mathbb {E} \left[ \bigl \langle \nabla f (x _ {\tau}), \epsilon_ {\tau} \bigr \rangle \right] \leq G \mathbb {E} [ \| \epsilon_ {\tau} \| ] \leq G \sqrt {\mathbb {E} [ \| \epsilon_ {\tau} \| ^ {2} ]} \\ \leq G \sqrt {\mathbb {E} \left[ \sum_ {t \leq T} \| \epsilon_ {t} \| ^ {2} \right]} \leq \sigma G \sqrt {T + 1} \leq \sigma G \sqrt {2 T}, \\ \end{array}
$$
where the equality is due to (34); the first inequality uses  $\| \nabla f(x_{\tau})\| \leq G$  by the definition of  $\tau$  in (5) and Corollary 3.6; the fourth inequality uses  $\mathbb{E}[X]^2\leq \mathbb{E}[X^2 ]$  for any random variable  $X$ ; and the last inequality uses Assumption 4.
Combining all the bounds above, we get
$$
\begin{array}{l} \mathbb {E} \left[ f (x _ {\tau}) - f ^ {*} + \frac {\eta}{2} \sum_ {t <   \tau} \| \nabla f (x _ {t}) \| ^ {2} \right] \leq f (x _ {0}) - f ^ {*} + \eta \sigma G \sqrt {2 T} + \eta^ {2} \sigma^ {2} L T \\ \leq f \left(x _ {0}\right) - f ^ {*} + \sigma , \\ \end{array}
$$
where the last inequality is due to Lemma F.1.
With Lemma F.2, we are ready to prove Theorem 5.3.
Proof of Theorem 5.3. We want to show the probability of  $\{\tau < T\}$  is small, as its complement  $\{\tau = T\}$  means  $f(x_{t}) - f^{*} \leq F$  for all  $t \leq T$  which implies  $\|\nabla f(x_{t})\| \leq G$  for all  $t \leq T$ . Note that
$$
\{\tau <   T \} = \{\tau_ {2} <   T \} \cup \{\tau_ {1} <   T, \tau_ {2} = T \}.
$$
Therefore we only need to bound the probability of each of these two events on the RHS.
We first bound  $\mathbb{P}(\tau_2 < T)$ . Note that
$$
\begin{array}{l} \mathbb {P} \left(\tau_ {2} <   T\right) = \mathbb {P} \left(\bigcup_ {t <   T} \left\{\left\| \epsilon_ {t} \right\| > \frac {G}{5 \eta L} \right\}\right) \\ \leq \sum_ {t <   T} \mathbb {P} \left(\| \epsilon_ {t} \| > \frac {G}{5 \eta L}\right) \\ \leq \frac {2 5 \eta^ {2} T \sigma^ {2} L ^ {2}}{G ^ {2}} \\ \leq \delta / 4, \\ \end{array}
$$
where the first inequality uses union bound; the second inequality applies Chebyshev's inequality and  $\mathbb{E}[\| \epsilon_t\|^2] = \mathbb{E}[\mathbb{E}_{t - 1}[\| \epsilon_t\|^2]]\leq \sigma^2$  for each fixed  $t$  by Assumption 4; the last inequality uses Lemma F.1.
Next, we will bound  $\mathbb{P}(\tau_1 < T, \tau_2 = T)$ . Note that under the event  $\{\tau_1 < T, \tau_2 = T\}$ , we know that 1)  $\tau = \tau_1 < T$  which implies  $f(x_{\tau + 1}) - f^* > F$ ; and 2)  $\tau < T = \tau_2$  which implies  $\| \epsilon_{\tau} \| \leq \frac{G}{5\eta L}$  by the definition in (5). Also note that we always have  $f(x_{\tau}) - f^* \leq F$  which implies  $\| \nabla f(x_{\tau}) \| \leq G$  by Corollary 3.6. Then we can show
$$
\left\| x _ {\tau + 1} - x _ {\tau} \right\| = \eta \left\| g _ {\tau} \right\| \leq \eta \left(\left\| \nabla f \left(x _ {\tau}\right) \right\| + \left\| \epsilon_ {\tau} \right\|\right) \leq \eta G + \frac {G}{5 L} \leq \frac {G}{L},
$$
where we choose  $\eta \leq \frac{1}{2L}$ . Then based on Lemma 3.3 and Remark 3.4, we have
$$
\begin{array}{l} f (x _ {\tau + 1}) - f (x _ {\tau}) \leq - \frac {\eta}{2} \| \nabla f (x _ {\tau}) \| ^ {2} - \eta \left\langle \nabla f (x _ {\tau}), \epsilon_ {\tau} \right\rangle + \eta^ {2} L \| \epsilon_ {\tau} \| ^ {2} \\ \leq \eta \| \nabla f (x _ {\tau}) \| \cdot \| \epsilon_ {\tau} \| + \eta^ {2} L \| \epsilon_ {\tau} \| ^ {2} \\ \leq \frac {G ^ {2}}{4 L} \\ = \frac {F}{2}, \\ \end{array}
$$
where the first inequality is obtained following the same derivation as in (33); the last equality is due to Corollary 3.6. Therefore we can show that under the event  $\{\tau_1 < T, \tau_2 = T\}$ ,
$$
f \left(x _ {\tau}\right) - f ^ {*} = f \left(x _ {\tau}\right) - f \left(x _ {\tau + 1}\right) + f \left(x _ {\tau + 1}\right) - f ^ {*} > F / 2.
$$
$$
\mathbb {P} \left(\tau_ {1} <   T, \tau_ {2} = T\right) \leq \mathbb {P} \left(f \left(x _ {\tau}\right) - f ^ {*} > F / 2\right) \leq \frac {\mathbb {E} \left[ f \left(x _ {\tau}\right) - f ^ {*} \right]}{F / 2} \leq \frac {2 \left(f \left(x _ {0}\right) - f ^ {*} + \sigma\right)}{F} = \delta / 4,
$$
where the second inequality uses Markov's inequality; the third inequality uses Lemma F.2; and in the last inequality we choose  $F = 8(f(x_0) - f^* + \sigma) / \delta$ .
Therefore we can show
$$
\mathbb {P} (\tau <   T) \leq \mathbb {P} (\tau_ {2} <   T) + \mathbb {P} (\tau_ {1} <   T, \tau_ {2} = T) \leq \delta / 2.
$$
Then we also know  $\mathbb{P}(\tau = T) \geq 1 - \delta / 2 \geq 1 / 2$ . Therefore, by Lemma F.2,
$$
\begin{array}{l} \frac {2 \left(f \left(x _ {0}\right) - f ^ {*} + \sigma\right)}{\eta} \geq \mathbb {E} \left[ \sum_ {t <   \tau} \| \nabla f \left(x _ {t}\right) \| ^ {2} \right] \\ \geq \mathbb {P} (\tau = T) \mathbb {E} \left[ \sum_ {t <   T} \| \nabla f (x _ {t}) \| ^ {2} \Bigg | \tau = T \right] \\ \geq \frac {1}{2} \mathbb {E} \left[ \sum_ {t <   T} \| \nabla f (x _ {t}) \| ^ {2} \middle | \tau = T \right]. \\ \end{array}
$$
$$
\mathbb {E} \left[ \frac {1}{T} \sum_ {t <   T} \| \nabla f (x _ {t}) \| ^ {2} \Bigg | \tau = T \right] \leq \frac {4 (f (x _ {0}) - f ^ {*} + \sigma)}{\eta T} = \frac {\delta F}{2 \eta T} \leq \frac {\delta}{2} \cdot \epsilon^ {2},
$$
where the last inequality uses the choice of  $T$ . Let  $\mathcal{E} \coloneqq \left\{\frac{1}{T}\sum_{t < T}\|\nabla f(x_t)\|^2 > \epsilon^2\right\}$  denote the event of not converging to an  $\epsilon$ -stationary point. By Markov's inequality, we have  $\mathbb{P}(\mathcal{E}) \leq \delta / 2$ . Therefore we have  $\mathbb{P}(\{\tau < T\} \cup \mathcal{E}) \leq \delta$ , which completes the proof.
G Lower bound
In this section, we provide the proof of Theorem 5.4.
Proof of Theorem 5.4. Let  $c, \eta_0 > 0$  satisfy  $\eta_0 \leq c^2 / 2$ . Consider
$$
f (x) = \left\{ \begin{array}{l l} \log (| x | - c), & | x | \geq y \\ 2 \log (y - c) - \log (2 y - | x | - c), & c / 2 \leq | x | <   y \\ k x ^ {2} + b, & | x | <   c / 2, \end{array} \right.
$$
where  $c > 0$  is a constant and  $y = (c + \sqrt{c^2 + 2\eta_0}) / 2 > 0$  is the fixed point of the iteration
$$
x _ {t + 1} = \left| x _ {t} - \frac {\eta_ {0}}{x _ {t} - c} \right|,
$$
and  $k, b$  are chosen in such a way that  $f(x)$  and  $f'(x)$  are continuous. Specifically, choose  $k = c^{-1}f'(c/2)$  and  $b = f(c/2) - cf'(c/2)/4$ . Since  $f(-x) = f(x)$ ,  $f(x)$  is symmetric about the line  $x = 0$ . In a small neighborhood,  $f(x)$  is symmetric about  $(y, f(y))$ , so  $f'(x)$  is continuous at  $y$ .
Let us first consider the smoothness of  $f$ . By symmetry, it suffices to consider  $x > 0$ . Then,
$$
f ^ {\prime} (x) = \left\{ \begin{array}{l l} (x - c) ^ {- 1}, & x \geq y \\ (2 y - x - c) ^ {- 1}, & c / 2 \leq x <   y \\ 2 k x, & 0 <   x <   c / 2. \end{array} \right.
$$
Its Hessian is given by
$$
f ^ {\prime \prime} (x) = \left\{ \begin{array}{l l} - (x - c) ^ {- 2}, & x > y \\ (2 y - x - c) ^ {- 2}, & c / 2 <   x <   y \\ 2 k, & 0 <   x <   c / 2. \end{array} \right.
$$
Hence,  $f(x)$  is  $(2, 2k, 1)$ -smooth.
Note that  $f(x)$  has a stationary point 0. For stepsize  $\eta_f$  satisfying  $\eta_0 \leq \eta_f \leq c^2 / 4$ , there exists  $z = (c + \sqrt{c^2 + 2\eta_f}) \geq y$  such that  $-z = z - \eta_f(y - c)^{-1}$  and by symmetry, once  $x_{\tau} = z$ ,  $x_t = \pm z$  for all  $t \geq \tau$ , making the GD iterations stuck. Now we choose a proper  $x_0$  such that  $f'(x_0)$  and  $f(x_0) - f(0)$  are bounded.
We consider arriving at  $y$  from above. That is,  $x_0 \geq x_1 \geq \ldots \geq x_\tau = z > c > 0$ . Since in each update where  $x_{t+1} = x_t - \eta_f(x_t - c)^{-1} > c$ ,
$$
x _ {t} - x _ {t + 1} = x _ {t} - \left(x _ {t} - \eta_ {f} (x _ {t} - c) ^ {- 1}\right) = \eta_ {f} (x _ {t} - c) ^ {- 1} \leq \sqrt {\eta_ {f}}.
$$
Hence, we can choose  $\tau$  in such a way that  $3c / 2\leq x_0 < 3c / 2 + \sqrt{\eta_f}$ . Then,
$$
\log (c / 2) \leq f (x _ {0}) \leq \log (c / 2 + \sqrt {\eta_ {f}}), \quad 2 / (c + 2 \sqrt {\eta_ {f}}) \leq f ^ {\prime} (x _ {0}) \leq 2 / c.
$$
By definition,  $y - c = \eta_0(c + \sqrt{c^2 + 2\eta_0})^{-1}$ . Hence,
$$
\begin{array}{l} f (c / 2) = 2 \log (y - c) - \log (2 y - c / 2 - c) \\ = 2 \log (\eta_ {0}) - 2 \log (c + \sqrt {c ^ {2} + 2 \eta_ {0}}) - \log (\sqrt {c ^ {2} + 2 \eta_ {0}} - c / 2), \\ \end{array}
$$
$$
f ^ {\prime} (c / 2) = \frac {1}{\sqrt {c ^ {2} + 2 \eta_ {0}} - c / 2}
$$
$$
\begin{array}{l} f (x _ {0}) - f (0) = f (x _ {0}) - f (c / 2) + c f ^ {\prime} (c / 2) / 4 \\ \leq \log (c / 2 + \sqrt {\eta_ {f}}) + 2 \log (\eta_ {0} ^ {- 1}) + 2 \log (c + \sqrt {c ^ {2} + 2 \eta_ {0}}) \\ + \log (\sqrt {c ^ {2} + 2 \eta_ {0}} - c / 2) + \frac {c}{4} \frac {1}{\sqrt {c ^ {2} + 2 \eta_ {0}} - c / 2} \\ \leq \log (c) + 2 \log (\eta_ {0} ^ {- 1}) + 2 \log (2 \sqrt {2 c ^ {2}}) + \log (\sqrt {2 c ^ {2}}) + \frac {1}{2} \\ = 4 \log (c) + 2 \log (\eta_ {0} ^ {- 1}) + \frac {7}{2} \log (2) + \frac {1}{2}. \\ \end{array}
$$
For stepsize  $\eta_f < \eta_0$ , reaching below  $4c / 3$  takes at least
$$
(x _ {0} - 4 c / 3) / \sqrt {\eta_ {f}} \geq c / (6 \sqrt {\eta_ {f}}) > c \eta_ {0} ^ {- 1 / 2} / 6
$$
steps to reach  $4c / 3$ , where  $f^{\prime}(4c / 3) = \log (c / 3)$ .
Now we set  $c$  and  $\eta_0$  and scale function  $f(x)$  to satisfy the parameter specifications  $L_0, L_2, G_0, \Delta_0$ . Define  $g(x) = L_2^{-1} f(x)$ . Then,  $g(x)$  is  $(2, 2kL_2^{-1}, L_2)$ -smooth. Since the gradient of  $g(x)$  is  $L_2^{-1}$  times  $f(x)$ , the above analysis for  $f(x)$  applies to  $g(x)$  by replacing  $\eta_0$  with  $\eta_1 = L_2 \eta_0$  and  $\eta_f$  with  $\eta = L_2 \eta_f$ . To ensure that
$$
2 k L _ {2} ^ {- 1} = 2 (c L _ {2}) ^ {- 1} f ^ {\prime} (c / 2) = \frac {2}{c L _ {2}} \frac {1}{\sqrt {c ^ {2} + 2 \eta_ {1}} - c / 2} \leq \frac {4}{c ^ {2} L _ {2}} \leq L _ {0},
$$
it suffices to take  $c \geq 2 / \sqrt{L_0 L_2}$ . To ensure that
$$
g ^ {\prime} (x _ {0}) \leq \frac {2}{L _ {2} c} \leq G _ {0},
$$
it suffices to take  $c \geq 2 / (L_2G_0)$ . To ensure that