SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. The proof idea follows from Lemma 21. Let  $t_0$  be defined as follows:
$$
t _ {0} \triangleq \max  \left\{t: \left(1 - \eta^ {2} - 2 \eta \rho\right) ^ {t} \geq \frac {1}{4} \right\}. \tag {21}
$$
Then, it follows that  $t_0 = \Theta(\min\{\eta^{-1}\rho^{-1}, \eta^{-2}\})$ . We first show that  $y_t^2$  cannot be too small until  $t_0$ :
Claim 41. For all  $t \leq t_0$ ,  $y_t^2 \geq \frac{1}{4}\gamma/\eta$ .
Proof. Prove by induction. Initially,  $y_0^2 = C\gamma/\eta \geq \frac{1}{4}\gamma/\eta$ . Then consider some  $t < t_0$  such that  $y_{t'}^2 \geq \frac{1}{4}\gamma/\eta$  for all  $t' \leq t$ . By Equation 20, we have
$$
\begin{array}{l} y _ {t + 1} ^ {2} - x _ {t + 1} ^ {2} = \left(y _ {t} - \eta \ell_ {t} ^ {\prime} \cdot (x _ {t} + \rho \tilde {\ell} _ {t} ^ {\prime} y _ {t})\right) ^ {2} - \left(x _ {t} - \eta \ell_ {t} ^ {\prime} \cdot (y _ {t} + \rho \tilde {\ell} _ {t} ^ {\prime} x _ {t})\right) ^ {2} \\ = \left(y _ {t} ^ {2} - x _ {t} ^ {2}\right) + \left(\eta \tilde {\ell} _ {t} ^ {\prime}\right) ^ {2} \left(\left(x _ {t} + \rho \tilde {\ell} _ {t} ^ {\prime} y _ {t}\right) ^ {2} - \left(y _ {t} + \rho \tilde {\ell} _ {t} ^ {\prime} x _ {t}\right) ^ {2}\right) + \\ 2 x _ {t} \cdot \eta \ell_ {t} ^ {\prime} \cdot \left(y _ {t} + \rho \tilde {\ell} _ {t} ^ {\prime} x _ {t}\right) - 2 y _ {t} \cdot \eta \ell_ {t} ^ {\prime} \cdot \left(x _ {t} + \rho \tilde {\ell} _ {t} ^ {\prime} y _ {t}\right) \\ = \left(y _ {t} ^ {2} - x _ {t} ^ {2}\right) + \left(\eta \ell_ {t} ^ {\prime}\right) ^ {2} \left(x _ {t} ^ {2} - y _ {t} ^ {2}\right) + \left(\eta \ell_ {t} ^ {\prime}\right) ^ {2} \left(\rho \tilde {\ell} _ {t} ^ {\prime}\right) ^ {2} \left(y _ {t} ^ {2} - x _ {t} ^ {2}\right) + 2 \eta \ell_ {t} ^ {\prime} \cdot \rho \tilde {\ell} _ {t} ^ {\prime} \left(x _ {t} ^ {2} - y _ {t} ^ {2}\right) \\ = \Big (1 - (\eta \ell_ {t} ^ {\prime}) ^ {2} + (\eta \ell_ {t} ^ {\prime}) ^ {2} (\rho \tilde {\ell} _ {t} ^ {\prime}) ^ {2} - 2 \eta \ell_ {t} ^ {\prime} \cdot \rho \tilde {\ell} _ {t} ^ {\prime} \Big) (y _ {t} ^ {2} - x _ {t} ^ {2}). \\ \end{array}
$$
Using Assumption (A1), we know that  $|\ell_t^{\prime}|\leq 1$  and  $|\tilde{\ell}_t^\prime |\leq 1$ , giving
$$
y _ {t + 1} ^ {2} - x _ {t + 1} ^ {2} \geq \left(1 - \eta^ {2} - 2 \eta \rho\right) \left(y _ {t} ^ {2} - x _ {t} ^ {2}\right) \geq \dots \geq \left(1 - \eta^ {2} - 2 \eta \rho\right) ^ {t + 1} \left(y _ {0} ^ {2} - x _ {0} ^ {2}\right).
$$
By definition of  $t_0$  and condition that  $t < t_0$ , we have  $y_{t + 1}^2 \geq y_{t + 1}^2 - x_{t + 1}^2 \geq \frac{1}{4}\gamma/\eta$ .
Claim 42. Suppose that  $\sqrt{C\gamma\eta} \leq \frac{1}{2}\sqrt{\gamma/\eta}$ , i.e.,  $\eta \leq \frac{1}{2}\sqrt{C}$ . Then  $y_t \geq \frac{1}{2}\sqrt{\gamma/\eta}$  for all  $t \leq t_0$ .
Proof. We still consider the maximum single-step difference in  $y_{t}$ . Suppose that for some  $t' < \min\{t_0, t_1\}$ ,  $y_{t'} \geq \frac{1}{2}\sqrt{\gamma/\eta}$  for all  $t' \leq t$ . According to Equation 20 and Assumption (A1),
$$
y _ {t + 1} = y _ {t} - \eta \ell_ {t} ^ {\prime} \cdot (x _ {t} + \rho \tilde {\ell} _ {t} ^ {\prime} y _ {t}) \geq y _ {t} - \eta (x _ {t} + \rho y _ {t}) \geq (1 - \eta \rho) y _ {t} - \eta \sqrt {C \gamma / \eta},
$$
where the last inequality used Lemma 37 to conclude that  $x_{t} \leq x_{0} \leq y_{0} = \sqrt{C\gamma / \eta}$ . Hence, as the first term is non-negative and  $\sqrt{C\gamma\eta} \leq \frac{1}{2}\sqrt{\gamma / \eta}$ , we must have  $y_{t + 1} \geq 0$ , which implies  $y_{t + 1} \geq \frac{1}{2}\sqrt{\gamma / \eta}$  according to Claim 41.
Putting these two claims together gives our conclusion. Note that the condition  $\eta \leq \frac{1}{2}\sqrt{C}$  in Claim 42 can be inferred from the assumptions  $\eta \leq \frac{1}{2}$  and  $C \geq 1$  made in Theorem 35.
After showing that  $y_{t}$  never becomes too small, we are ready to show that  $x_{t}$  decreases fast enough.
Lemma 43. For  $t \leq \min\{t_0, t_1\}$ , we have
$$
x _ {t + 1} \leq x _ {t} - \frac {1}{2} \sqrt {\gamma \eta}.
$$
In particular, if  $\eta$  is sufficiently small s.t.  $2\sqrt{C}\eta^{-1} \leq t_0$ , then we must have  $t_1 \leq 2\sqrt{C}\eta^{-1} - 1 < t_0$ .
Proof. Since  $x_{t} > 0$  for all  $t \leq t_1$ , we have  $\mathrm{sign}(\ell_t') = \mathrm{sign}(x_t y_t) = \mathrm{sign}(y_t)$  from Lemma 36. Lemma 40, so the USAM update (20) simplifies as follows according to Assumption (A1) and Lemma 40
$$
x _ {t + 1} = x _ {t} - \eta | \ell_ {t} ^ {\prime} | \cdot (| y _ {t} | + \rho | \ell_ {t} ^ {\prime} | | x _ {t} |) \leq x _ {t} - \eta | y _ {t} | \leq x _ {t} - \frac {1}{2} \sqrt {\gamma \eta}.
$$
Let  $t_1' = \frac{\sqrt{C\gamma / \eta}}{\frac{1}{2}\sqrt{\gamma\eta}} = 2\sqrt{C}\eta^{-1}$ , then  $t_1' - 1 < t_0$  and thus  $x_{t_1'} < \frac{1}{2}\sqrt{\gamma\eta}$ , i.e.,  $t_1 < t_0$  must hold.
G.3 Final Phase:  $y_{t}$  Gets Trapped Above the Origin
Now, we are going to consider the final-stage behavior of USAM.
Theorem 44 (Main Conclusion of Final Phase; USAM Case). After Initial Phase, we always have  $|x_{t}| \leq \sqrt{C\gamma\eta}$  and  $|y_{t}| \leq \sqrt{C\gamma / \eta}$ . Moreover, once we have  $\eta (1 + \rho y_{\mathrm{t}}^{2})y_{\mathrm{t}}^{2} = 2 - \epsilon$  for some  $\mathfrak{t} \geq t_1$  and  $\epsilon > 0$ , we must have  $|x_{t + 1}| \leq \exp (-\Omega (\epsilon))|x_t|$  for all  $t \geq \mathfrak{t}$ . This consequently infers that  $y_{\infty}^{2}$  which is defined as  $\lim \inf_{t\to \infty}y_t^2$ , satisfies  $y_{\infty}^{2} \geq (1 - 4C\gamma (\eta +\rho C\gamma)^{2}\epsilon^{-1})y_{\mathrm{t}}^{2}$ . As  $\epsilon$  is a constant independent of  $\eta$  and  $\rho$  and can be arbitrarily close to 0, this shows the second claim of Theorem 35.
Proof. The first conclusion is an analog of Lemma 28 and Lemma 27 (which allows a simpler analysis thanks to the removal of normalization), as we will show in Lemma 45. The second part requires a similar (but much more sophisticated) analysis to Lemma 15 of Ahn et al. (2023a), which we will cover in Lemma 47.  $\square$
Lemma 45. Suppose that  $|x_{t}| \leq \sqrt{C\gamma\eta}$ ,  $|y_{t}| \leq \sqrt{C\gamma / \eta}$ . Assuming  $\eta \rho \leq 1$ , then  $|x_{t + 1}| \leq \sqrt{C\gamma\eta}$  and  $|y_{t + 1}| \leq \sqrt{C\gamma / \eta}$  as well. Furthermore,  $|x_{t}| \leq \sqrt{C\gamma\eta}$  and  $|y_{t}| \leq \sqrt{C\gamma / \eta}$  hold for all  $t > t_1$ .
Proof. Suppose that  $x_{t} \geq 0$  without loss of generality. The case where  $x_{t+1} \geq 0$  is trivial by Lemma 37. Otherwise, using Lemma 36 and Equation 20, we can write
$$
x _ {t + 1} = x _ {t} - \eta \ell_ {t} ^ {\prime} \cdot (y _ {t} + \rho \tilde {\ell} _ {t} ^ {\prime} x _ {t}) = x _ {t} - \eta | \ell_ {t} ^ {\prime} | \cdot (| y _ {t} | + \rho | \tilde {\ell} _ {t} ^ {\prime} | x _ {t}).
$$
By Assumption (A1),  $|\ell_t^{\prime}|$  and  $|\tilde{\ell}_t^\prime |$  are bounded by 1. By the condition that  $\eta \rho \leq 1$
$$
x _ {t + 1} \geq (1 - \eta \rho) x _ {t} - \eta | y _ {t} | \geq - \sqrt {C \gamma \eta},
$$
where we used  $|y_t| \leq \sqrt{2} \sqrt{C\gamma / \eta}$ . Similarly, for  $y_{t + 1}$ , only considering the case where  $y_{t} \geq 0$  and  $y_{t + 1} \leq 0$  suffices. We have the following by symmetry
$y_{t + 1} = y_t - \eta \ell_t' \cdot (x_t + \rho \tilde{\ell}_t' y_t) \geq y_t - \eta (|x_t| + \rho y_t) = (1 - \eta \rho) y_t - \eta |x_t| \geq -\eta^{1.5} \sqrt{C\gamma} \geq -\sqrt{C\gamma / \eta},$  where we used  $|x_t| \leq \sqrt{C\gamma\eta}$ .
The second part of the conclusion is done by induction. According to the definition of Initial Phase, we have  $|x_{t_1 + 1}| \leq \frac{1}{2}\sqrt{\gamma\eta}$ . As  $C \geq 1 \geq \frac{1}{4}$ , we consequently have  $|x_t| \leq \sqrt{C\gamma\eta}$  for all  $t > t_1$  from the first part of the conclusion. Regarding  $|y_t|$ , recall Lemma 40 infers  $y_{t} > 0$  for all  $t \leq t_1 + 1$  and Lemma 37 infers the monotonicity of  $y_{t}$ , we have  $y_{t_1 + 1} \leq y_0 = \sqrt{C\gamma / \eta}$ . Hence,  $|y_{t}| \leq \sqrt{C\gamma / \eta}$  for all  $t > t_1$  as well.
Before showing the ultimate conclusion Lemma 47, we first show the following single-step lemma:
Lemma 46. Suppose that  $\eta (1 + \rho y_t^2)y_t^2 < 2$  and  $|x_{t}|\leq \sqrt{C\gamma\eta}$  for some t. Define  $\epsilon_{t} = 2 - \eta (1+$ $\rho y_t^2)y_t^2$  (then we must have  $\epsilon_t\in (0,2)$ ). Then we have
$$
\left| x _ {t + 1} \right| \leq \left| x _ {t} \right| \exp \left(- \min  \left\{\left(1 + \rho C \gamma \eta\right) \epsilon_ {t}, \frac {2 - \epsilon_ {t}}{8} \right\}\right). \tag {22}
$$
Proof. Without loss of generality, assume that  $x_{t} > 0$ . From Equation 20, we can write:
$$
x _ {t + 1} = x _ {t} - \eta \ell_ {t} ^ {\prime} \cdot (y _ {t} + \rho \tilde {\ell} _ {t} ^ {\prime} x _ {t}) = x _ {t} - \eta | \ell_ {t} ^ {\prime} | \cdot (| y _ {t} | + \rho | \tilde {\ell} _ {t} ^ {\prime} | x _ {t}),
$$
where we used  $\mathrm{sign}(\ell_t^{\prime}) = \mathrm{sign}(\tilde{\ell}_t^{\prime}) = \mathrm{sign}(x_ty_t) = \mathrm{sign}(y_t)$  (Lemma 36). By Assumption (A2),
$$
\begin{array}{l} x _ {t + 1} \geq x _ {t} - \eta \left| \left(x _ {t} + \rho \tilde {\ell} _ {t} ^ {\prime} y _ {t}\right) \left(y _ {t} + \rho \tilde {\ell} _ {t} ^ {\prime} x _ {t}\right) \right| \left(| y _ {t} | + \rho | x _ {t} y _ {t} | x _ {t}\right) \\ = x _ {t} - \eta \left(x _ {t} + \rho | \tilde {\ell_ {t} ^ {\prime}} | | y _ {t} |\right) \left(| y _ {t} | + \rho | \tilde {\ell_ {t} ^ {\prime}} | x _ {t}\right) \left(| y _ {t} | + \rho x _ {t} ^ {2} | y _ {t} |\right) \\ \geq x _ {t} - \eta \left(x _ {t} + \rho x _ {t} y _ {t} ^ {2}\right) \left(\left| y _ {t} \right| + \rho x _ {t} ^ {2} \left| y _ {t} \right|\right) \left(\left| y _ {t} \right| + \rho x _ {t} ^ {2} \left| y _ {t} \right|\right) \\ = \left(1 - \eta (1 + \rho y _ {t} ^ {2}) (1 + \rho x _ {t} ^ {2}) ^ {2} y _ {t} ^ {2}\right) x _ {t} \\ \geq \left(1 - (1 + \rho C \gamma \eta) \eta \left(1 + \rho y _ {t} ^ {2}\right) y _ {t} ^ {2}\right) x _ {t} \\ \geq - \left(1 - \left(1 + \rho C \gamma \eta\right) \epsilon_ {t}\right) x _ {t}, \tag {23} \\ \end{array}
$$
where we used  $|x_{t}| \leq \sqrt{C\gamma\eta}$ . For the other direction, we have the following by Assumption (A3):
$$
\begin{array}{l} x _ {t + 1} \leq x _ {t} - \eta \frac {| (x _ {t} + \rho \tilde {\ell} _ {t} ^ {\prime} y _ {t}) (y _ {t} + \rho \tilde {\ell} _ {t} ^ {\prime} x _ {t}) |}{2} \bigg (| y _ {t} | + \rho \frac {| x _ {t} y _ {t} |}{2} x _ {t} \bigg) \\ = x _ {t} - \eta \frac {\left(x _ {t} + \rho \left| \tilde {\ell} _ {t} ^ {\prime} \right| \left| y _ {t} \right|\right) \left(\left| y _ {t} \right| + \rho \left| \tilde {\ell} _ {t} ^ {\prime} \right| x _ {t}\right)}{2} \left(\left| y _ {t} \right| + \rho \frac {x _ {t} ^ {2}}{2} \left| y _ {t} \right|\right) \\ \leq x _ {t} - \eta \frac {\left(x _ {t} + \rho x _ {t} y _ {t} ^ {2}\right) \left(\left| y _ {t} \right| + \rho \left| y _ {t} \right| x _ {t} ^ {2}\right)}{8} \left(\left| y _ {t} \right| + \rho \frac {x _ {t} ^ {2}}{2} \left| y _ {t} \right|\right) \\ = \left(1 - \frac {\eta}{8} (1 + \rho y _ {t} ^ {2}) (1 + \rho x _ {t} ^ {2}) \left(1 + \rho \frac {x _ {t} ^ {2}}{2}\right) y _ {t} ^ {2}\right) x _ {t} \\ \leq \left(1 - \frac {\eta}{8} \left(1 + \rho y _ {t} ^ {2}\right) y _ {t} ^ {2}\right) x _ {t} = \left(1 - \frac {2 - \epsilon}{8}\right) x _ {t}, \tag {24} \\ \end{array}
$$
where we used  $(1 + \rho x_{t}^{2})\geq 1$  and  $(1 + \rho \frac{x_t^2}{2})\geq 1$ . Equation 22 follows from Equation 23 and Equation 24.
Lemma 47. Let  $\mathfrak{t}$  be such that i)  $\eta (1 + \rho y_{\mathfrak{t}}^{2})y_{\mathfrak{t}}^{2} = 2 - \epsilon$  where  $\epsilon \in (0,\frac{2}{9})$  is a constant, and ii)  $|x_{\mathfrak{t}}|\leq \sqrt{C\gamma\eta}$ . Then we have the following conclusion on  $\lim \inf_{t\to \infty}y_t^2$ , denoted by  $y_{\infty}^{2}$  in short:
$$
y _ {\infty} ^ {2} = \lim _ {t \rightarrow \infty} \inf y _ {t} ^ {2} \geq \left(1 - 4 C \gamma (\eta + \rho C \gamma) ^ {2} \epsilon^ {- 1}\right) y _ {t} ^ {2}.
$$
While the  $\epsilon^{-1}$  looks enormous, it is a constant independent of  $\eta$  and  $\rho$ ; in other words, we are allowed to set  $\epsilon$  as close to zero as we want. As we only consider the dependency on  $\eta$  and  $\rho$ , we can abbreviate this conclusion as  $y_{\infty}^{2} \geq (1 - \mathcal{O}(\eta^{2} + \rho^{2}))y_{\mathrm{t}}^{2}$ , as we claim in the main text.
Proof. In analog to Equation 23, we derive the following for  $y_{t}$ :
$$
y _ {t + 1} ^ {2} \geq \left(1 - \eta \big (1 + \rho y _ {t} ^ {2} \big) ^ {2} (1 + \rho x _ {t} ^ {2}) x _ {t} ^ {2}\right) ^ {2} y _ {t} ^ {2} \geq \left(1 - 2 \eta (1 + \rho C \gamma / \eta) ^ {2} (1 + \rho C \gamma \eta) x _ {t} ^ {2}\right) y _ {t} ^ {2},
$$
where the second inequality uses Lemma 45. Let  $d_{t} = y_{\mathrm{t}}^{2} - y_{t}^{2}$ , then we have
$$
d _ {t + 1} \leq d _ {t} + 2 \eta (1 + \rho C \gamma / \eta) ^ {2} (1 + \rho C \gamma \eta) x _ {t} ^ {2} y _ {t} ^ {2} \leq d _ {t} + 4 \eta^ {- 1} (\eta + \rho C \gamma) ^ {2} y _ {\mathfrak {t}} ^ {2} x _ {t} ^ {2},
$$
where we used the assumption that  $\rho C\gamma \eta \leq 1$  and the fact that  $y_{t}^{2}$  is monotonic (thus  $y_{t}^{2}\leq y_{t}^{2}$ ).
According to Equation 22, we have  $|x_{t + 1}| \leq |x_t| \exp (-\Omega (\epsilon))$  for all  $t \geq t_2$ . Hence, we have
$$
d _ {\infty} = \lim _ {t \to \infty} \sup _ {d _ {t}} d _ {t} \leq d _ {\mathfrak {t}} + \sum_ {t = \mathfrak {t}} ^ {\infty} 4 \eta^ {- 1} (\eta + \rho C \gamma) ^ {2} y _ {\mathfrak {t}} ^ {2} x _ {t} ^ {2} = 4 \eta^ {- 1} (\eta + \rho C \gamma) ^ {2} y _ {\mathfrak {t}} ^ {2} \cdot \epsilon^ {- 1} x _ {\mathfrak {t}} ^ {2},
$$
where we used  $d_{\mathrm{t}} = 0$  (by definition) and the sum of geometric series. Plugging back  $x_{\mathrm{t}}^{2} \leq C\gamma \eta$ ,
$$
y _ {\infty} ^ {2} = \lim _ {t \to \infty} \inf _ {y _ {t} ^ {2}} y _ {t} ^ {2} \geq y _ {\mathfrak {t}} ^ {2} - 4 C \gamma (\eta + \rho C \gamma) ^ {2} y _ {\mathfrak {t}} ^ {2},
$$
H Omitted Proof of USAM Over General PL functions
Theorem 48 (Formal Version of Theorem 6). For any  $\mu$ -PL and  $\beta$ -smooth loss function  $\mathcal{L}$ , for any learning rate  $\eta < 1/\beta$  and  $\rho < 1/\beta$ , for any initialization  $w_0$ , the following holds for USAM:
$$
\left\| w _ {t} - w _ {0} \right\| \leq \eta (1 + \beta \rho) \sqrt {\frac {2 \beta^ {2}}{\mu}} \left(1 - 2 \mu \eta (1 - \rho \beta) \left(\frac {\eta \beta}{2} (1 - \rho \beta)\right)\right) ^ {- 1 / 2} \sqrt {\mathcal {L} (w _ {0}) - \mathcal {L} ^ {*}}, \quad \forall t \geq 0,
$$
where  $\mathcal{L}^*$  is the short-hand notation for  $\min_w\mathcal{L}(w)$ .
We first state the following useful result by Andriushchenko and Flammarion (2022, Theorem 10).
Lemma 49 (Descent Lemma of USAM over Smooth and PL Losses). For any  $\beta$ -smooth and  $\mu$ -PL loss function  $\mathcal{L}$ , for any learning rate  $\eta < 1/\beta$  and  $\rho < 1/\beta$ , the following holds for USAM:
$$
\mathcal {L} (w _ {t}) - \mathcal {L} ^ {\star} \leq \left(1 - 2 \mu \eta (1 - \rho \beta) \left(1 - \frac {\eta \beta}{2} (1 - \rho \beta)\right)\right) ^ {t} (\mathcal {L} (w _ {0}) - \mathcal {L} ^ {\star}), \quad \forall t \geq 0.
$$
Proof of Theorem 48. We follow the convention in Karimi et al. (2016): Let  $\mathcal{X}^{\star}$  be the set of global minima and  $x_{p}$  be the projection of  $x$  onto the solution set  $\mathcal{X}^{\star}$ . From  $\beta$ -smoothness, it follows that
$$
\| \nabla \mathcal {L} (x) \| = \| \nabla \mathcal {L} (x) - \nabla \mathcal {L} (x _ {p}) \| \leq \beta \| x - x _ {p} \|, \quad \forall x.
$$
Now since  $\mathcal{L}$  is  $\beta$ -smooth and  $\mu$ -PL, Theorem 2 from (Karimi et al., 2016) implies that the quadratic growth condition holds, i.e.,
$$
\frac {2}{\mu} \left(\mathcal {L} (x) - \mathcal {L} ^ {\star}\right) \geq \left\| x - x _ {p} \right\| ^ {2}, \quad \forall x.
$$
$$
\left\| \nabla \mathcal {L} (x) \right\| ^ {2} \leq \frac {2 \beta^ {2}}{\mu} (\mathcal {L} (x) - \mathcal {L} ^ {\star}), \quad \forall x.
$$
Moreover, from  $\beta$ -smoothness, we have
$$
\| \nabla \mathcal {L} (x + \rho \nabla \mathcal {L} (x)) \| \leq \| \nabla \mathcal {L} (x) \| + \beta \| \rho \nabla \mathcal {L} (x) \| = (1 + \beta \rho) \| \nabla \mathcal {L} (x) \|, \quad \forall x.
$$
Thus, by the update rule of USAM (2), it follows that
$$
\begin{array}{l} \| w _ {t} - w _ {0} \| \leq \eta \sum_ {i = 0} ^ {t - 1} \| \nabla \mathcal {L} \left(w _ {i} + \rho \nabla \mathcal {L} \left(w _ {i}\right)\right) \| \\ \leq \eta (1 + \beta \rho) \sum_ {i = 0} ^ {t - 1} \| \nabla \mathcal {L} (w _ {i}) \| \\ \leq \eta (1 + \beta \rho) \sum_ {i = 0} ^ {t - 1} \sqrt {\frac {2 \beta^ {2}}{\mu} (\mathcal {L} (w _ {i}) - \mathcal {L} ^ {\star})} \\ = \eta (1 + \beta \rho) \sqrt {\frac {2 \beta^ {2}}{\mu}} \sum_ {i = 0} ^ {t - 1} \sqrt {\mathcal {L} (w _ {i}) - \mathcal {L} ^ {\star}}. \\ \end{array}
$$
Now we only to invoke the USAM descent lemma stated before, i.e., Lemma 49, giving
$$
\begin{array}{l} \sum_ {i = 0} ^ {t - 1} \sqrt {\mathcal {L} (w _ {i}) - \mathcal {L} ^ {*}} \leq \sum_ {i = 0} ^ {t - 1} \left(1 - 2 \mu \eta (1 - \rho \beta) \left(1 - \frac {\eta \beta}{2} (1 - \rho \beta)\right)\right) ^ {i / 2} \sqrt {\mathcal {L} (w _ {0}) - \mathcal {L} ^ {*}} \\ \leq \left(1 - 2 \mu \eta (1 - \rho \beta) \left(\frac {\eta \beta}{2} (1 - \rho \beta)\right)\right) ^ {- 1 / 2} \sqrt {\mathcal {L} \left(w _ {0}\right) - \mathcal {L} ^ {*}}. \\ \end{array}
$$
Putting the last two inequalities together then give our conclusion.