SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Using these two inequalities, we can conclude as follows. Let  $q > 0$  be a parameter chosen later, and let  $t$  be the first iteration for which  $\rho_t \geq q$  (if no such iteration exists, then  $\rho_t \leq q$  for all  $t$ ). Note that  $\rho_t \leq q + O(\eta |x_{\mathbf{t}}|)$  due to (15) and (16). By (15), we conclude that for all  $t' \geq t$ ,
$$
\left| x _ {t ^ {\prime}} \right| \leq \left| x _ {t} \right| \exp \left(- \Omega \left(q \left(t ^ {\prime} - t\right)\right)\right) \leq \left| x _ {\mathbf {t}} \right| \exp \left(- \Omega \left(q \left(t ^ {\prime} - t\right)\right)\right).
$$
Substituting this into (16),
$$
\begin{array}{l} \rho_ {t ^ {\prime}} \leq \rho_ {t} + 4 \eta \sum_ {s = t} ^ {t ^ {\prime} - 1} x _ {s} ^ {2} \leq q + O (\eta | x _ {\mathbf {t}} |) + O (\eta | x _ {\mathbf {t}} | ^ {2}) \sum_ {s = 1} ^ {t ^ {\prime} - 1} \exp \left(- \Omega (q (s - t))\right) \\ \leq q + O \left(\eta \left| x _ {\mathbf {t}} \right|\right) + O \left(\frac {\eta \left| x _ {\mathbf {t}} \right| ^ {2}}{q}\right). \\ \end{array}
$$
By optimizing this bound over  $q$ , we find that for all  $t$
$$
\rho_ {t} \lesssim \sqrt {\eta} | x _ {\mathbf {t}} | \lesssim \eta | s _ {\mathbf {t}} |.
$$
Translating this result back into  $y_{t}^{2}$  yields the result.
Let us take stock of what we have established thus far.
- According to Lemma 6,  $|s_t|$  is bounded for all  $t$  by a constant.
- Then, from Lemma 7 and Lemma 8, we must have either  $y_{t}^{2} \to 2 / \eta$ , or  $2 / \eta - O(\eta) \leq y_{\mathbf{t}}^{2} \leq 2 / \eta$  for some iteration  $\mathbf{t}$ .
- In the latter case, Proposition 1 shows that the limiting sharpness is  $2 / \eta - O(1)$ .
Note also that the analyses thus far have not made use of Assumption (A2), i.e., we have established the  $\beta = +\infty$  case of Theorem 2. Moreover, for all  $\beta > 1$ , the asymptotic  $2 / \eta - O(1)$  still shows that the limiting sharpness is close to  $2 / \eta$ , albeit with suboptimal rate. The reader who is satisfied with this result can then skip ahead to subsequent sections. The remainder of this section and the next section are devoted to substantial refinements of the analysis.
To see where improvements are possible, note that both Lemma 8 and Proposition 1 rely on the size of  $|s_{\mathfrak{t}}|$  at the crossing. Our crude bound of  $|s_{\mathfrak{t}}| \lesssim 1$  does not capture the behavior observed in experiments, in which  $|s_{\mathfrak{t}}| \lesssim \eta^{1 / (\beta - 1)}$ . By substituting this improved bound into Lemma 7, we would deduce that the gap at the crossing is  $O(\eta^{1 + 2 / (\beta - 1)})$ , and then Proposition 1 would imply that the limiting sharpness is  $2 / \eta - O(\eta^{1 / (\beta - 1)})$ . Another weakness of our proof is that it provides nearly no information about the dynamics during the bouncing phase, which constitutes an incomplete understanding of the EoS phenomenon. In particular, we experimentally observe that during the bouncing phase, the iterates lie very close to the quasi-static envelope (Figure 9). In the next section, we will rigorously prove all of these observations.
Before doing so, however, we show that Proposition 1 can be refined by using Assumption (A2), which could be of interest in its own right. It shows that even if the convergence phase begins with a large value of  $|s_{\mathbf{t}}|$ , the limiting sharpness can be much closer to  $2 / \eta$  than what Proposition 1 suggests. The following proposition combined with Lemma 6 implies Theorem 2 for all  $\beta > 2$ , but it is insufficient for the case  $1 < \beta \leq 2$ . From now on, we assume  $\beta < +\infty$ .
Proposition 2 (convergence phase; refined). Suppose that  $y_{\mathbf{t}}^2 < 2 / \eta \leq y_{\mathbf{t} - 1}^2$ . Then,  $GD$  converges to  $(0, y_{\infty})$  satisfying
$$
\frac {2}{\eta} \geq y _ {\infty} ^ {2} \geq \frac {2}{\eta} - O (\eta | s _ {\mathbf {t}} | ^ {2}) - \left\{ \begin{array}{l l} O (\eta^ {1 / (\beta - 1)}), & \beta > 2, \\ O (\eta \log (| s _ {\mathbf {t}} | / \eta)), & \beta = 2, \\ O (\eta | s _ {\mathbf {t}} | ^ {2 - \beta}), & \beta <   2. \end{array} \right.
$$
Proof. Let  $y_{t}^{2} = (2 - \rho_{t}) / \eta$  as before. We quantify the decrease of  $|x_{t}|$  in terms of  $\rho_{t}$  and conversely the increase of  $\rho_{t}$  in terms of  $|x_{t}|$  by tracking the half-life of  $|x_{t}|$ , i.e., the number of iterations it takes  $|x_{t}|$  to halve. We call these epochs: at the  $i$ -th epoch, we have
$$
2 ^ {- (i + 1)} \sqrt {\eta} <   | x _ {t} | \leq 2 ^ {- i} \sqrt {\eta}.
$$
Let  $i_0$  be the index of the first epoch, i.e.,  $i_0 = \lfloor \log_2(\sqrt{\eta} / |x_{\mathbf{t}}|) \rfloor$ . Due to Lemma 6, we know that  $i_0 \geq -O(1)$ . From (15),  $|x_t|$  is monotonically decreasing and consequently  $|s_t|$  is decreasing as
well. Also, our bound on the limiting sharpness implies that  $y_{t}^{2} > 1 / \eta$  for all  $t$ , provided that  $\eta$  is sufficiently small.
Let us now compute the dynamics of  $\rho_{t}$  and  $|x_{t}|$ . At epoch  $i$ ,  $|x_{t}| > 2^{-(i + 1)}\sqrt{\eta}$  hence  $|s_t| > 2^{-(i + 1)}$ . Assumption (A2+) with  $r = |s_{\mathbf{t}}|\lesssim 1$  implies that
$$
\frac {\ell^ {\prime} \left(s _ {t}\right)}{s _ {t}} \leq 1 - c 2 ^ {- \beta (i + 1)}, \tag {17}
$$
where  $c = c(|s_{\mathfrak{t}}|)$ . This allows to refine (15) on the decrease of  $|x_{t}|$  to
$$
\frac {| x _ {t + 1} |}{| x _ {t} |} = \eta r _ {t} y _ {t} ^ {2} - 1 \leq (2 - \rho_ {t}) (1 - c 2 ^ {- \beta (i + 1)}) - 1 \leq 1 - \rho_ {t} - c 2 ^ {- \beta (i + 1)},
$$
where the first inequality follows from (17) and the second from  $\rho_t = 2 - \eta y_t^2 < 1$ . In turn, this inequality shows that the  $i$ -th phase only requires  $O(2^{\beta i})$  iterations.
Hence, if  $t(i)$  denotes the start of the  $i$ -th epoch, then (16) shows that
$$
\rho_ {t (i + 1)} \leq \rho_ {t (i)} + 4 \eta^ {2} \cdot 2 ^ {- 2 i} \cdot O (2 ^ {\beta i}) \leq \rho_ {t (i)} + O (\eta^ {2} 2 ^ {(\beta - 2) i}).
$$
Summing this up, we have
$$
\rho_ {t (i)} \leq \rho_ {\mathbf {t}} + \eta^ {2} \times \left\{ \begin{array}{l l} O (2 ^ {(\beta - 2)   i}), & \beta > 2, \\ O (i - i _ {0}), & \beta = 2, \\ O (2 ^ {(\beta - 2)   i _ {0}}) = O (| s _ {\mathbf {t}} | ^ {2 - \beta}), & \beta <   2. \end{array} \right.
$$
In the case of  $\beta < 2$ , the final sharpness satisfies  $2 / \eta - O(\rho_{\mathbf{t}} / \eta) - O(\eta |s_{\mathbf{t}}|^{2 - \beta}) \leq y_{\infty}^2 \leq 2 / \eta$ .
In the other two cases, suppose that we use this argument until epoch  $i_{\star}$  such that  $2^{-i_{\star}} \asymp \eta^{\gamma}$ . Then, we have  $|x_{t(i_{\star})}| \asymp \eta^{\gamma + 1/2}, |s_{t(i_{\star})}| \asymp \eta^{\gamma}$ , and by using the argument from Proposition 1 from iteration  $t(i_{\star})$  onward we obtain
$$
\rho_{\infty} = \rho_{t(i_{\star})} + \rho_{\infty} - \rho_{t(i_{\star})}\leq \rho_{\mathfrak{t}} + O(\eta^{\gamma +1}) + \eta^{2}\times \left\{ \begin{array}{ll}O(2^{(\beta -2)i_{\star}}) = O(\eta^{-\gamma (\beta -2)}), & \beta >2,\\ O(i_{\star} - i_{0}), & \beta = 2. \end{array} \right.
$$
We optimize over the choice of  $\gamma$ , obtaining  $\gamma = 1 / (\beta - 1)$  and thus
$$
\rho_ {\infty} \leq \rho_ {\mathbf {t}} + \left\{ \begin{array}{l l} O (\eta^ {1 + 1 / (\beta - 1)}), & \beta > 2, \\ O (\eta^ {2} \log (| s _ {\mathbf {t}} | / \eta)), & \beta = 2. \end{array} \right.
$$
By collecting together the three cases and using Lemma 8 to bound  $\rho_{\mathbf{t}}$ , we finish the proof.
Using the crude bound  $|s_{t_0}| \lesssim 1$  from Lemma 6, it yields
$$
\frac {2}{\eta} \geq y _ {\infty} ^ {2} \geq \frac {2}{\eta} - O (\eta) - \left\{ \begin{array}{l l} O (\eta^ {1 / (\beta - 1)}), & \beta > 2, \\ O (\eta \log (1 / \eta)), & \beta = 2, \\ O (\eta), & \beta <   2, \end{array} \right.
$$
which is optimal for  $\beta > 2$ .
C.5 EoS regime: quasi-static analysis
We supplement Assumption (A2) with a corresponding lower bound on  $\ell^{\prime}(s) / s$ :
$$
\text {T h e r e} C > 0 \text {s u c h t h a t} \quad \frac {\ell^ {\prime} (s)}{s} \geq 1 - C | s | ^ {\beta} \quad \text {f o r a l l} s \neq 0. \tag {A3}
$$
Under these assumptions, we prove the following result which is also of interest as it provides detailed information for the bouncing phase of the EoS.
Theorem 5 (quasi-static principle). Suppose we run  $GD$  on  $f$  with step size  $\eta >0$ , where  $f(x,y)\coloneqq \ell(xy)$  and  $\ell$  satisfies Assumptions (A1), (A2), and (A3). Write  $y_{t}^{2}\coloneqq (2 + \delta_{t}) / \eta$  and suppose that at some iteration  $t_0$ , we have  $|x_{t_0}y_{t_0}|\asymp \delta_{t_0}^{1 / \beta}$  and  $\delta_{t_0}\lesssim 1$ . Then, for all  $t\geq t_0$  with  $\delta_t\gtrsim \eta^{\beta /(\beta -1)}$ , we have
$$
\left| x _ {t} y _ {t} \right| \asymp \delta_ {t} ^ {1 / \beta},
$$
where all implied constants depend on  $\ell$  but not on  $\eta$ .
In this section, we show that the GD iterates lie close to the quasi-static trajectory and give the full proof of Theorem 2. Recall from (13) that the quasi-static analysis predicts
$$
\eta r _ {t} y _ {t} ^ {2} \approx 2, \tag {18}
$$
and that during the bouncing phase, this closely agrees with experimental observations (Figure 9). We consider the phase where  $y_{t}^{2}$  has not yet crossed the threshold  $2 / \eta$  and we write  $y_{t}^{2} \coloneqq (2 + \delta_{t}) / \eta$ , thinking of  $\delta_t$  as small. Then, (18) can be written  $(2 + \delta_t)r_t\approx 2$ . If we have the behavior  $\ell^{\prime}(s) / s = 1 - \Theta (|s_t|^{\beta})$  near the origin, then  $r_t\approx 1 - \Theta (\delta_t)$  implies that
$$
\left| s _ {t} \right| ^ {\beta} \approx \delta_ {t}. \tag {19}
$$
Our goal is to rigorously establish (19). However, we first make two observations. First, in order to establish Theorem 2, we only need to prove an upper bound on  $|s_t|$ , which only requires Assumption (A2) (to prove a lower bound on  $|s_t|$ , we need a corresponding lower bound on  $\ell'(s)/s$ ). Second, even if we relax (19) to read  $|s_t|^{\beta} \lesssim \delta_t$ , this fails to hold when  $\delta_t$  is too small, because the error terms (the deviation of the dynamics from the quasi-static trajectory) begin to dominate. With this in mind, we shall instead prove  $|s_t|^{\beta} \lesssim \delta_t + C' \eta^{\gamma}$ , where the added  $\eta^{\gamma}$  handles the error terms and the exponent  $\gamma > 0$  emerges from the proof.
Proposition 3 (quasi-static analysis; upper bound). For all  $t$  such that  $0 \leq \delta_{t-1} \lesssim 1 / (\beta \vee 1)$  (for a sufficiently small implied constant), it holds that
$$
\left| s _ {t} \right| ^ {\beta} \leq C \left(\delta_ {t} + C ^ {\prime} \eta^ {\beta / (\beta - 1)}\right),
$$
where  $C, C' > 0$  are constants which may depend on the problem parameters but not on  $\eta$ .
We first show that Theorem 2 now follows.
Proof of Theorem 2. As previously noted, the  $\beta = +\infty$  case is handled by the arguments of the previous section, so we focus on  $\beta < +\infty$ . From Lemma 7, we either have  $y_{t}^{2} \to 2 / \eta$  and  $|x_{t}| \to 0$  in which case we are done, or there is an iteration  $\mathfrak{r}$  such that  $y_{\mathfrak{r}}^{2} < 2 / \eta \leq y_{\mathfrak{r}-1}^{2}$ . From Proposition 3, since  $\delta_{t-1} \geq 0$  and  $\delta_{t} \leq 0$ , it follows that  $|s_{\mathfrak{r}}|^{\beta} \lesssim \eta^{1 / (\beta-1)}$ . The theorem now follows, either from Proposition 1 or from the refined Proposition 2.
We now prove Proposition 3. In the proof, we use asymptotic notation  $O(\cdot), \lesssim$ , etc. in order to hide constants that depend on  $\ell$  (including  $\beta$ ), but not on  $\delta_t$  and  $\eta$ . However, the proof also involves choosing parameters  $C, C' > 0$ , and we keep the dependence on these parameters explicit for clarity.
Proof of Proposition 3. The proof goes by induction; namely, if  $|s_t|^{\beta} \leq C (\delta_t + C' \eta^{\gamma})$  and  $\delta_t \geq 0$  at some iteration  $t$ , we prove that the same holds one iteration later, where the constants  $C, C' > 0$  as well as the exponent  $\gamma > 0$  are chosen later in the proof.
For the base case, observe that the approximate conservation lemma (Lemma 6) gives  $|s_t| \lesssim 1$ , and  $\delta_t \gtrsim 1 / (\beta \vee 1)$  at the beginning of the induction, so the bound is satisfied initially if we choose  $C$  sufficiently large enough.
Throughout, we also write  $\hat{\delta}_t\coloneqq \delta_t + C'\eta^\gamma$  as a convenient shorthand. The strategy is to prove the following two statements:
1. If  $|s_t|^\beta = C_t\hat{\delta}_t$  for some  $C_t > \frac{C}{2}$ , then  $|s_{t + 1}|^\beta \leq C_{t + 1}\hat{\delta}_{t + 1}$  for some  $C_{t + 1} \leq C_t$ .
2. If  $|s_t|^\beta = C_t\hat{\delta}_t$  for some  $C_t \leq \frac{C}{2}$ , then  $|s_{t + 1}|^\beta \leq C\hat{\delta}_{t + 1}$ .
Proof of 1. The dynamics for  $x$  give
$$
\left| x _ {t + 1} \right| = \left| 1 - \eta y _ {t} ^ {2} r _ {t} \right| \left| x _ {t} \right|.
$$
By Assumption  $(\mathrm{A2^{+}})$  and  $|s_t|\lesssim 1$
$$
r _ {t} \leq 1 - \Omega (| s _ {t} | ^ {\beta}) = 1 - \Omega (C \hat {\delta} _ {t})
$$
$$
\eta y _ {t} ^ {2} r _ {t} = \left(2 + \delta_ {t}\right) \left(1 - \Omega \left(C \hat {\delta} _ {t}\right)\right) = 2 - \Omega \left(C \hat {\delta} _ {t}\right)
$$
for large  $C$ . Also,  $\ell''(0) = 1$  and a similar argument as in the proof of Theorem 1 yields the reverse inequality  $\eta y_t^2 r_t \gtrsim 1$ . We conclude that
$$
\left| x _ {t + 1} \right| = \left(1 - \Omega (C \hat {\delta} _ {t})\right) \left| x _ {t} \right|
$$
$$
\left| s _ {t + 1} \right| ^ {\beta} \leq \left(1 - \Omega \left(C \hat {\delta} _ {t}\right)\right) \left| s _ {t} \right| ^ {\beta} = C _ {t} \left(1 - \Omega \left(C \hat {\delta} _ {t}\right)\right) \hat {\delta} _ {t}.
$$
Since we need a bound in terms of  $\hat{\delta}_{t + 1}$ , we use the dynamics of  $y$ ,
$$
y _ {t + 1} = \left(1 - \eta x _ {t} ^ {2} r _ {t}\right) y _ {t} \geq \left(1 - \eta x _ {t} ^ {2}\right) y _ {t},
$$
$$
\delta_ {t + 1} = \eta y _ {t + 1} ^ {2} - 2 \geq \delta_ {t} - 2 \eta^ {2} s _ {t} ^ {2} \geq \delta_ {t} - 2 \eta^ {2} \left(C \hat {\delta} _ {t}\right) ^ {2 / \beta}. \tag {20}
$$
Substituting this in,
$$
\begin{array}{l} \left| s _ {t + 1} \right| ^ {\beta} \leq C _ {t} \left(1 - \Omega \left(C \hat {\delta} _ {t}\right)\right) \left(\hat {\delta} _ {t + 1} + 2 \eta^ {2} \left(C \hat {\delta} _ {t}\right) ^ {2 / \beta}\right) \\ = C _ {t} \hat {\delta} _ {t + 1} - \Omega \left(C ^ {2} \hat {\delta} _ {t} \hat {\delta} _ {t + 1}\right) + 2 C \eta^ {2} \left(C \hat {\delta} _ {t}\right) ^ {2 / \beta}. \tag {21} \\ \end{array}
$$
Let us show that
$$
\hat {\delta} _ {t + 1} \geq \frac {3}{4} \hat {\delta} _ {t}. \tag {22}
$$
From (20), we have  $\hat{\delta}_{t + 1} \geq \hat{\delta}_t - 2\eta^2 (C\hat{\delta}_t)^{2 / \beta}$ , so we want to prove that  $\eta^2 (C\hat{\delta}_t)^{2 / \beta} \leq \hat{\delta}_t / 8$ . If  $\beta \leq 2$  this is obvious by taking  $\eta$  small, and if  $\beta > 2$  then this is equivalent to  $C^{2 / \beta}\eta^2 \lesssim \hat{\delta}_t^{1 - 2 / \beta}$ . It suffices to have  $C^{2 / \beta}\eta^2 \lesssim (C')^{1 - 2 / \beta}\eta^{\gamma (1 - 2 / \beta)}$ , which is achieved by taking  $C'$  large relative to  $C$  and by taking  $\gamma \leq 2 / (1 - 2 / \beta)$ ; this constraint on  $\gamma$  will be satisfied by our eventual choice of  $\gamma = \beta / (\beta - 1)$ .
Returning to (21), in order to finish the proof and in light of (22), we want to show that  $C^2\hat{\delta}_t^2 \gtrsim C^{1 + 2/\beta}\eta^2\hat{\delta}_t^{2/\beta}$ . Rearranging, it suffices to have  $\hat{\delta}_t^{2 - 2/\beta} \gtrsim C^{2/\beta - 1}\eta^2$ , or  $\hat{\delta}_t^{1 - 1/\beta} \gtrsim C^{1/\beta - 1/2}\eta$ . Since by definition  $\hat{\delta}_t \geq C'\eta^\gamma$ , by choosing  $C'$  large it suffices to have  $\gamma \leq 1/(1 - 1/\beta) = \beta / (\beta - 1)$ , which leads to our choice of  $\gamma$ .
Proof of 2. Using the simple bound  $\eta y_t^2 r_t \leq 2 + \delta_t$ , we have
$$
\begin{array}{l} \left| s _ {t + 1} \right| \leq \left(1 + \delta_ {t}\right) \left| s _ {t} \right|, \\ | s _ {t + 1} | ^ {\beta} \leq \exp (\beta \delta_ {t}) | s _ {t} | ^ {\beta} = C _ {t} \exp (\beta \delta_ {t}) \hat {\delta} _ {t} \leq \frac {4}{3} C _ {t} \exp (\beta \delta_ {t}) \hat {\delta} _ {t + 1} \\ \end{array}
$$
where we used (22). If  $\exp (\beta \delta_t)\leq 4 / 3$ , which holds if  $\delta_t\lesssim 1 / \beta$ , then from  $C_t\leq C / 2$  we obtain  $|s_{t + 1}|^{\beta}\leq C\hat{\delta}_{t + 1}$  as desired.
By following the same proof outline but reversing the inequalities, we can also show a corresponding lower bound on  $|s_t|^\beta$ , as long as  $\delta_t \gtrsim \eta^{\beta / (\beta - 1)}$ . Although this is not needed to establish Theorem 2, it is of interest in its own right, as it shows (together with Proposition 3) that the iterates of GD do in fact track the quasi-static trajectory.
Proposition 4 (quasi-static analysis; lower bound). Suppose additionally that (A3) holds and that  $\beta < +\infty$ . Also, suppose that at some iteration  $t_0$ , we have  $\delta_{t_0} \lesssim 1$  and that
$$
\left| s _ {t} \right| \geq c \delta_ {t} ^ {1 / \beta} \tag {23}
$$
holds at iteration  $t = t_0$ , where  $c$  is a sufficiently small constant (depending on the problem parameters but not on  $\eta$ ). Then, (23) also holds for all iterations  $t \geq t_0$  such that  $\delta_t \gtrsim \eta^{\beta / (\beta - 1)}$ .
Proof. The proof mirrors that of Proposition 3. Let  $\delta_t \gtrsim \eta^{\beta / (\beta - 1)}$  for a sufficiently large implied constant. We prove the following two statements:
1. If  $|s_t| = c_t \delta_t^{1/\beta}$  for some  $c_t < 2c$ , then  $|s_{t+1}| \geq c_{t+1} \delta_{t+1}^{1/\beta}$  for some  $c_{t+1} \geq c_t$ .
2. If  $|s_t| = c_t \delta_t^{1/\beta}$  for some  $c_t \geq 2c$ , then  $|s_{t+1}| \geq c \delta_{t+1}^{1/\beta}$ .
Throughout the proof, due to Proposition 3, we also have  $|s_t| \lesssim \delta_t^{1/\beta}$ .
By Assumption (A3),
$$
r _ {t} \geq 1 - O \left(\left| s _ {t} \right| ^ {\beta}\right) \geq 1 - O \left(c \delta_ {t}\right).
$$
If  $c$  is sufficiently small, then
$$
\eta y _ {t} ^ {2} r _ {t} \geq (2 + \delta_ {t}) (1 - O (c \delta_ {t})) \geq 2 + \Omega (\delta_ {t}).
$$
$$
\left| x _ {t + 1} \right| \geq \left(1 + \Omega (\delta_ {t})\right) \left| x _ {t} \right|.
$$
$$
y _ {t + 1} \geq \left(1 - \eta x _ {t} ^ {2}\right) y _ {t} \geq \left(1 - O \left(\eta^ {2} s _ {t} ^ {2}\right)\right) y _ {t} \geq \left(1 - O \left(\eta^ {2} \delta_ {t} ^ {2 / \beta}\right)\right) y _ {t} \tag {24}
$$
$$
\begin{array}{l} \left| s _ {t + 1} \right| \geq \left(1 + \Omega \left(\delta_ {t}\right)\right) \left(1 - O \left(\eta^ {2} \delta_ {t} ^ {2 / \beta}\right)\right) \left| s _ {t} \right| \geq c _ {t} \left(1 + \Omega \left(\delta_ {t}\right) - O \left(\eta^ {2} \delta_ {t} ^ {2 / \beta}\right)\right) \delta_ {t} ^ {1 / \beta} \\ \geq c _ {t} \left(1 + \Omega (\delta_ {t}) - O \left(\eta^ {2} \delta_ {t} ^ {2 / \beta}\right)\right) \delta_ {t + 1} ^ {1 / \beta}. \\ \end{array}
$$
To conclude, we must prove that  $\eta^2\delta_t^{2 / \beta}\lesssim \delta_t$  , but since  $\delta_t\gtrsim \eta^{\beta /(\beta -1)}$  (with sufficiently large implied constant), then this holds, as was checked in the proof of Proposition 3.
Proof of 2. Using Assumption (A3),
$$
1 - O (\delta_ {t}) \leq 1 - O (| s _ {t} | ^ {\beta}) \leq r _ {t} \leq 1.
$$
$$
2 - O \left(\delta_ {t}\right) \leq \left(2 + \delta_ {t}\right) \left(1 - O \left(\delta_ {t}\right)\right) \leq \eta y _ {t} ^ {2} r _ {t} \leq 2 + \delta_ {t}
$$
$$
- 1 + O \left(\delta_ {t}\right) \geq 1 - \eta y _ {t} ^ {2} r _ {t} \geq - 1 - \delta_ {t}.
$$
Together with the dynamics for  $x$  and (24),
$$
\left| s _ {t + 1} \right| \geq \left(1 - O (\delta_ {t})\right) \left(1 - O (\eta^ {2} \delta_ {t} ^ {2 / \beta})\right) \left| s _ {t} \right| \geq c _ {t} \left(1 - O (\delta_ {t})\right) \left(1 - O (\eta^ {2} \delta_ {t} ^ {2 / \beta})\right) \delta_ {t + 1} ^ {1 / \beta}.
$$
Since  $c_{t} \geq 2c$ , if  $\delta_{t}$  and  $\eta$  are sufficiently small it implies  $|s_{t + 1}| \geq c\delta_{t + 1}^{1 / \beta}$ .
Convergence rate estimates. Our analysis also provides estimates for the convergence rate of GD in both regimes. Namely, in the gradient flow regime, we show that GD converges in  $O(1 / \eta)$  iterations, whereas in the EoS regime, GD typically spends  $\Omega (1 / \eta^{(\beta / (\beta - 1))\vee 2})$  iterations  $(\Omega (\log (1 / \eta) / \eta^2)$  iterations when  $\beta = 2$ ) in the bouncing phase (Figure 11). Hence, the existence of the bouncing phase dramatically slows down the convergence of GD.
Remark 2. Suppose that at iteration  $t_0$ , we have  $\delta_{t_0} \asymp 1$ . Then, the assumption of Proposition 4 is that  $|s_{t_0}| \gtrsim 1$ . If this is not satisfied, i.e.,  $|s_{t_0}| \ll 1$ , then the first claim in the proof of Proposition 4 shows that  $|s_{t_0 + 1}| \geq (1 + \Omega(\delta_t)) |s_{t_0}| = (1 + \Omega(1)) |s_{t_0}|$ . Therefore, after  $t' = O(\log(1 / |s_{t_0}|))$  iterations, we obtain  $|s_{t_0 + t'}| \gtrsim 1$  and then Proposition 4 applies thereafter.
Remark 3. From the quasi-static analysis, we can also derive bounds on the length of the bouncing phase. Namely, suppose that  $t_0$  is such that  $\delta_{t_0} \asymp 1$  and for all  $t \geq t_0$ , we have  $|s_t| = \delta_t^{1/\beta}$ . If  $\delta_{t_0}$  is sufficiently small so that  $r_t \gtrsim 1$  for all  $t \geq t_0$ , then the equation for  $y$  yields
$$
\delta_ {t + 1} \leq \delta_ {t} - \Theta (\eta^ {2} s _ {t} ^ {2}) = \delta_ {t} - \Theta (\eta^ {2} \delta_ {t} ^ {2 / \beta}).
$$
We declare the  $k$ -th phase to consist of iterations  $t$  such that  $2^{-k} \leq \delta_t \leq 2^{-(k-1)}$ . During this phase,  $\delta_{t+1} \leq \delta_t - \Theta(\eta^2 2^{-2k/\beta})$ , so the number of iterations in phase  $k$  is  $\asymp 2^{k(2/\beta-1)/\eta^2}$ . We sum over the phases until  $\delta_t \asymp \eta^{\beta/(2-\beta-1)}$ , since after this point the quasi-static analysis fails and  $y_t^2$  crosses over  $2/\eta$  shortly afterwards. This yields
$$
\frac{1}{\eta^{2}}\sum_{\substack{k\in \mathbb{Z}\\ \eta^{\beta /(\beta -1)}\lesssim 2^{-k}\lesssim 1}}2^{k  (2 / \beta -1)}\asymp \left\{ \begin{array}{ll}1 / \eta^{2}, & \beta >2,\\ \log (1 / \eta) / \eta^{2}, & \beta = 2,\\ 1 / \eta^{\beta /( \beta -1)}, & \beta <  2. \end{array} \right.
$$
The time spent in the bouncing phase increases dramatically as  $\beta \searrow 1$ .
D Deferred derivations of mean model
In this section, we provide the details for the derivations of the mean model in Subsection 3.1. Recall
$$
f (\boldsymbol {x}; a ^ {-}, a ^ {+}, b) = a ^ {-} \sum_ {i = 1} ^ {d} \operatorname {R e L U} \left(- \boldsymbol {x} [ i ] + b\right) + a ^ {+} \sum_ {i = 1} ^ {d} \operatorname {R e L U} \left(+ \boldsymbol {x} [ i ] + b\right),
$$
where  $\pmb{x} = \lambda y\pmb{e}_j + \pmb{\xi}$ . We first approximate
$$
\sum_ {i = 1} ^ {d} \operatorname {R e L U} \left(\pm \boldsymbol {x} [ i ] + b\right) \approx \sum_ {i = 1} ^ {d} \operatorname {R e L U} \left(\pm \boldsymbol {\xi} [ i ] + b\right).
$$
In other words, we can ignore the contribution of the signal  $\lambda ye_{j}$ . This approximation holds because (i) initially, the bias  $b$  is not yet negative enough to threshold out the noise, and hence the summation  $\sum_{i=1}^{d} \mathrm{ReLU}(\pm \pmb{\xi}[i] + b)$  is of size  $O(d)$ , and (ii) the difference between the left- and right-hand sides above is simply  $\mathrm{ReLU}(\pm \lambda y \pm \pmb{\xi}[j] + b) - \mathrm{ReLU}(\pm \pmb{\xi}[j] + b)$ , which is of size  $O(1)$  and hence negligible compared to the full summation.
Next, letting  $g(b) \coloneqq \mathbb{E}_{z \sim \mathcal{N}(0,1)} \mathrm{ReLU}(z + b)$  be the 'smoothed' ReLU (see Figure 7), concentration of measure implies the following two facts:
-  $\sum_{i=1}^{d} \operatorname{ReLU}\bigl(\pm \pmb{\xi}[i] + b\bigr) \approx d\mathbb{E}_{\xi \sim \mathcal{N}(0,1)} \operatorname{ReLU}(\xi + b) =: dg(b)$  and
-  $\sum_{i=1}^{d} \mathbb{1}\{\pm \boldsymbol{x}[i] + b \geq 0\} \approx d \mathbb{E}_{\xi \sim \mathcal{N}(0,1)} \mathbb{1}\{\xi + b \geq 0\} = dg'(b)$ .