SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left| \lambda_ {\max } (\Theta_ {t}) - \tilde {\lambda} (q _ {t}) \right| \leq 1 + \mathcal {O} _ {\ell} (\eta).
$$
Proof. By Proposition B.11, we can bound the sharpness  $\lambda_{\max}(\Theta_t)$  at time step  $t$  by
$$
\left| \lambda_ {\max } (\boldsymbol {\Theta} _ {t}) - \frac {2 \ell^ {\prime \prime} (p _ {t})}{\eta q _ {t}} \right| \leq 1.
$$
Since  $\ell''(z) = r(z) + zr'(z)$ , we can rewrite as following:
$$
\left| \lambda_ {\max } (\Theta_ {t}) - \left(s _ {t} ^ {- 1} + \frac {p _ {t} r ^ {\prime} \left(p _ {t}\right)}{q _ {t}}\right) \frac {2}{\eta} \right| \leq 1. \tag {14}
$$
By Theorem 4.2 and since  $h$  is a bounded function by Lemma B.9, we have  $s_t = 1 + \mathcal{O}_\ell(\eta^2)$  for any  $t \geq t_a$ . Consequently,  $|s_t^{-1} - 1| = \mathcal{O}_\ell(\eta^2)$  and  $|r(p_t) - q_t| = \mathcal{O}_\ell(\eta^2)$ . Moreover, for any  $0 < q < 1$ ,
$$
\begin{array}{l} \frac {d}{d q} \left(\frac {\hat {r} (q) r ^ {\prime} (\hat {r} (q))}{q}\right) = \frac {\hat {r} ^ {\prime} (q) (r ^ {\prime} (\hat {r} (q)) + \hat {r} (q) r ^ {\prime \prime} (\hat {r} (q)))}{q} - \frac {\hat {r} (q) r ^ {\prime} (\hat {r} (q))}{q ^ {2}} \\ = \frac {1}{q} \left(1 + \frac {\hat {r} (q) r ^ {\prime \prime} (\hat {r} (q))}{r ^ {\prime} (\hat {r} (q))}\right) - \frac {\hat {r} (q) r ^ {\prime} (\hat {r} (q))}{q ^ {2}}, \\ \end{array}
$$
$$
\lim  _ {q \rightarrow 1 ^ {-}} \left(\frac {d}{d q} \left(\frac {\hat {r} (q) r ^ {\prime} (\hat {r} (q))}{q}\right)\right) = \lim  _ {p \rightarrow 0 ^ {+}} \left(1 + \frac {p r ^ {\prime \prime} (p)}{r ^ {\prime} (p)}\right) = 2.
$$
Therefore,  $\frac{d}{dq}\left(\frac{\hat{r}(q)r'(\hat{r}(q))}{q}\right)$  is bounded on  $[\frac{1}{4}, 1)$  and Taylor's theorem gives
$$
\left| \frac {p _ {t} r ^ {\prime} (p _ {t})}{r (p _ {t})} - \frac {\hat {r} (q _ {t}) r ^ {\prime} (\hat {r} (q _ {t}))}{q _ {t}} \right| = \mathcal {O} _ {\ell} (| r (p _ {t}) - q _ {t} |) = \mathcal {O} _ {\ell} (\eta^ {2}),
$$
for any time step  $t$  with  $q_{t} < 1$ . Hence, if  $q_{t} < 1$ , we have the following bound:
$$
\left| \tilde {\lambda} \left(q _ {t}\right) - \left(s _ {t} ^ {- 1} + \frac {p _ {t} r ^ {\prime} \left(p _ {t}\right)}{q _ {t}}\right) \frac {2}{\eta} \right| \leq \left| 1 - s _ {t} ^ {- 1} + \frac {\hat {r} \left(q _ {t}\right) r ^ {\prime} \left(\hat {r} \left(q _ {t}\right)\right)}{q _ {t}} - \frac {p _ {t} r ^ {\prime} \left(p _ {t}\right)}{r \left(p _ {t}\right)} \right| \frac {2}{\eta} + \mathcal {O} _ {\ell} (\eta) = \mathcal {O} _ {\ell} (\eta), \tag {15}
$$
where we used  $\frac{p_t r'(p_t)}{q_t} = \frac{p_t r'(p_t)}{r(p_t)} (1 + \mathcal{O}_\ell (\eta^2)) = \frac{p_t r'(p_t)}{r(p_t)} + \mathcal{O}_\ell (\eta^2)$ , since  $1 + \frac{p_t r'(p_t)}{r(p_t)} = \frac{\ell''(p_t)}{r(p_t)} > 0$  implies  $|p_t r'(p_t)| \leq r(p_t) \leq 1$ . Now let  $t$  be any given time step with  $q_t \geq 1$ . Then,  $r(p_t) = 1 - \mathcal{O}_\ell (\eta^2)$ , and since  $r(z) = 1 + r''(0)z^2 + \mathcal{O}_\ell (z^4)$  for small  $z$ , we have  $|p_t| = \mathcal{O}_\ell (\eta)$ . Hence,
$$
\left| \tilde {\lambda} \left(q _ {t}\right) - \left(s _ {t} ^ {- 1} + \frac {p _ {t} r ^ {\prime} \left(p _ {t}\right)}{q _ {t}}\right) \frac {2}{\eta} \right| \leq \left| 1 - s _ {t} ^ {- 1} - \frac {p _ {t} r ^ {\prime} \left(p _ {t}\right)}{r \left(p _ {t}\right)} \right| \frac {2}{\eta} + \mathcal {O} _ {\ell} (\eta) = \mathcal {O} _ {\ell} (\eta), \tag {16}
$$
for any  $t$  with  $q_{t} \geq 1$ . By Eqs. (14), (15), and (16), we can conclude that for any  $t \geq t_{a}$ , we have
Finally, we can easily check that the sequence  $(\tilde{\lambda}(q_t))_{t=0}^{\infty}$  is monotonically increasing, since  $z \mapsto \frac{zr'(z)}{r(z)}$  is a decreasing function by Assumption 4.3 (ii) and the sequence  $(q_t)$  is monotonically increasing.
C Proofs for the Single-neuron Nonlinear Network
C.1 Formal statements of Theorem 5.1
In this subsection, we provide the formal statements of Theorem 5.1. We study the GD dynamics on a two-dimensional function  $\mathcal{L}(x,y)\coloneqq \frac{1}{2} (\phi (x)y)^2$ , where  $x,y$  are scalars and  $\phi$  is a nonlinear activation satisfying Assumption 5.1. We consider the reparameterization given by Definition 5.2, which is  $(p,q)\coloneqq \left(x,\frac{2}{\eta y^2}\right)$ .
We emphasize that the results we present in this subsection closely mirror those of the Section 4. In particular,
Assumption C.1 mirrors Assumption 4.2,
Assumption C.2 mirrors Assumption 4.3,
- (gradient flow regime) Theorem C.1 mirrors Theorem 4.1.
(EoS regime, Phase I) Theorem C.2 mirrors Theorem 4.2,
(EoS regime, Phase II) Theorem C.3 mirrors Theorem 4.3, and
- (progressive sharpening) Theorem C.4 mirrors Theorem 4.4.
The proof strategies are also similar. This is mainly because the 1-step update rule Eq. (7) resembles Eq. (6) for small step size  $\eta$ . We now present our rigorous results contained in Theorem 5.1.
Inspired by Lemma 2.1, we have an additional assumption on  $\phi$  as below.
Assumption C.1. Let  $r$  be a function defined by  $r(z) \coloneqq \frac{\phi(z)\phi'(z)}{z}$  for  $z \neq 0$  and  $r(0) \coloneqq 1$ . The function  $r$  satisfies Assumption 2.4.
In contrast to the function  $r$  defined in Section 4, the expression  $1 + \frac{pr'(p)}{r(p)}$  can be negative, which implies that the constant  $c$  defined in Lemma 2.1 is positive. As a result, the dynamics of  $p_t$  may exhibit a period-4 (or higher) oscillation or even chaotic behavior (as illustrated in Figure 1b).
We first state our results on the gradient flow regime.
Theorem C.1 (gradient flow regime). Let  $\eta \in (0, \frac{r(1)}{2(r(1) + 2)})$  be a fixed step size and  $\phi$  be a sigmoidal function satisfying Assumptions 5.1 and C.1. Suppose that the initialization  $(p_0, q_0)$  satisfies  $|p_0| \leq 1$  and  $q_0 \in \left(\frac{1}{1 - 2\eta}, \frac{r(1)}{4\eta}\right)$ . Consider the reparameterized GD trajectory characterized in Eq. (7). Then, the GD iterations  $(p_t, q_t)$  converge to the point  $(0, q^*)$  such that
$$
q _ {0} \leq q ^ {*} \leq \exp \left(2 \eta \left[ \min  \left\{\frac {2 (q _ {0} - 1)}{q _ {0}}, \frac {r (1)}{q _ {0}} \right\} \right] ^ {- 1}\right) q _ {0} \leq 2 q _ {0}.
$$
Theorem C.1 implies that in gradient flow regime, GD with initialization  $(x_0,y_0)$  and step size  $\eta$  converges to  $(0,y^{*})$  which has the sharpness bounded by:
$$
\left(1 - 2 \eta \left[ \min  \left\{\frac {2 (q _ {0} - 1)}{q _ {0}}, \frac {r (1)}{q _ {0}} \right\} \right] ^ {- 1}\right) y _ {0} ^ {2} \leq \lambda_ {\max } \left(\nabla^ {2} \mathcal {L} \left(0, y ^ {*}\right)\right) \leq y _ {0} ^ {2}.
$$
Now we provide our results on the EoS regime with an additional assumption below.
Assumption C.2. Let  $r$  be a function defined in Assumption C.1. Then  $r$  is  $C^4$  on  $\mathbb{R}$  and satisfies:
(i)  $z\mapsto \frac{r'(z)}{r(z)^2}$  is decreasing on  $\mathbb{R}$
(ii)  $z \mapsto \frac{zr'(z)}{r(z)}$  is decreasing on  $z > 0$  and increasing on  $z < 0$ ,
(iii)  $z \to \frac{zr(z)}{r'(z)}$  is decreasing on  $z > 0$  and increasing on  $z < 0$ , and
$(iv)\hat{r} (\frac{1}{2})r'\big(\hat{r} (\frac{1}{2})\big) > - \frac{1}{2}.$
Note that the function  $r$  that arises from the activation  $\phi = \tanh$  satisfies Assumptions 5.1, C.1, and C.2.
Theorem C.2 (EoS regime, Phase I). Let  $\eta > 0$  be a small enough constant and  $\phi$  be an activation function satisfying Assumptions 5.1, C.1, and C.2. Let  $z_0 \coloneqq \sup_z \left\{ \frac{z r'(z)}{r(z)} \geq -\frac{1}{2} \right\}$ ,  $z_1 \coloneqq \sup_z \left\{ \frac{z r'(z)}{r(z)} \geq -1 \right\}$ , and  $c_0 \coloneqq \max \{ r(z_0), r(z_1) + \frac{1}{2} \} \in (\frac{1}{2}, 1)$ . Let  $\delta \in (0, 1 - c_0)$  be any given constant. Suppose that the initialization  $(p_0, q_0)$  satisfies  $|p_0| \leq 1$  and  $q_0 \in (c_0, 1 - \delta)$ . Consider the reparameterized GD trajectory characterized in Eq. (7). We assume that for all  $t \geq 0$  such that  $q_t < 1$ , we have  $p_t \neq 0$ . Then, there exists a time step  $t_a = \mathcal{O}_{\delta, \phi}(\log(\eta^{-1}))$  such that for any  $t \geq t_a$ ,
$$
\frac {q _ {t}}{r (p _ {t})} = 1 + h (p _ {t}) \eta + \mathcal {O} _ {\delta , \phi} (\eta^ {2})
$$
where  $h:\mathbb{R}\to \mathbb{R}$  is a function defined as
$$
h (p) := \left\{ \begin{array}{l l} - \frac {\phi (p) ^ {2} r (p)}{p r ^ {\prime} (p)} & i f p \neq 0, a n d \\ - \frac {1}{r ^ {\prime \prime} (0)} & i f p = 0. \end{array} \right.
$$
The main difference between Theorem C.2 and Theorem 4.2 is the error term which is  $\mathcal{O}(\eta^2)$  in the former and  $\mathcal{O}(\eta^4)$  in the latter. This is because the 1-step update rule of  $q_{t}$  in Theorem C.2 is given by  $q_{t + 1} = (1 + \mathcal{O}(\eta))q_t$ , while in Theorem 4.2 we have  $q_{t + 1} = (1 + \mathcal{O}(\eta^2))q_t$ .
Theorem C.3 (EoS regime, Phase II). Under the same settings as in Theorem C.2, there exists a time step  $t_b = \Omega((1 - q_0)\eta^{-1})$ , such that  $q_{t_b} \leq 1$  and  $q_t > 1$  for any  $t > t_b$ . Moreover, the GD iterates  $(p_t, q_t)$  converge to the point  $(0, q^*)$  such that
$$
q ^ {*} = 1 - \frac {\eta}{r ^ {\prime \prime} (0)} + \mathcal {O} _ {\delta , \phi} (\eta^ {2}).
$$
Theorem C.3 implies that in the EoS regime, GD with step size  $\eta$  converges to  $(0,y^{*})$  which has the sharpness approximated as:
$$
\lambda_ {\max} (\nabla^ {2} \mathcal {L} (0, y ^ {*})) = \frac {2}{\eta} - \frac {2}{| r ^ {\prime \prime} (0) |} + \mathcal {O} _ {\delta , \phi} (\eta).
$$
Theorem C.4 proves that progressive sharpening (i.e., sharpness increases) occurs during Phase II.
Theorem C.4 (progressive sharpening). Under the same setting as in Theorem C.2, let  $t_a$  denote the obtained time step. Define the function  $\tilde{\lambda} : \mathbb{R}_{>0} \to \mathbb{R}$  given by
$$
\tilde {\lambda} (q) := \left\{ \begin{array}{l l} \left(1 + \frac {\hat {r} (q) r ^ {\prime} (\hat {r} (q))}{q}\right) \frac {2}{\eta} & i f q \leq 1, a n d \\ \frac {2}{\eta} & o t h e r w i s e. \end{array} \right.
$$
Then, the sequence  $\left(\tilde{\lambda}(q_t)\right)_{t=0}^{\infty}$  is monotonically increasing. For any  $t \geq t_a$ , the sharpness at  $GD$  iterate  $(x_t, y_t)$  closely follows the sequence  $\left(\tilde{\lambda}(q_t)\right)_{t=0}^{\infty}$  satisfying the following:
$$
\lambda_ {\max} (\nabla^ {2} \mathcal {L} (x _ {t}, y _ {t})) = \tilde {\lambda} (q _ {t}) + \mathcal {O} _ {\phi} (1).
$$
In Figure 1b, we conduct numerical experiments on single neuron model with tanh-activation, demonstrating that  $\tilde{\lambda}(q_t)$  provides a close approximation of the sharpness.
C.2 Proof of Theorem C.1
We give the proof of Theorem C.1, restated below for the sake of readability.
Theorem C.1 (gradient flow regime). Let  $\eta \in (0, \frac{r(1)}{2(r(1) + 2)})$  be a fixed step size and  $\phi$  be a sigmoidal function satisfying Assumptions 5.1 and C.1. Suppose that the initialization  $(p_0, q_0)$  satisfies  $|p_0| \leq 1$
and  $q_0 \in \left( \frac{1}{1 - 2\eta}, \frac{r(1)}{4\eta} \right)$ . Consider the reparameterized GD trajectory characterized in Eq. (7). Then, the GD iterations  $(p_t, q_t)$  converge to the point  $(0, q^*)$  such that
Theorem C.1 directly follows from Proposition C.5 stated below.
Proposition C.5. Suppose that  $\eta \in (0, \frac{r(1)}{2(r(1) + 2)})$ ,  $|p_0| \leq 1$  and  $q_0 \in \left(\frac{1}{1 - 2\eta}, \frac{r(1)}{4\eta}\right)$ . Then for any  $t \geq 0$ , we have
$$
\left| p _ {t} \right| \leq \left[ 1 - \min  \left\{\frac {2 \left(q _ {0} - 1\right)}{q _ {0}}, \frac {r (1)}{q _ {0}} \right\} \right] ^ {t} \leq 1,
$$
$$
q _ {0} \leq q _ {t} \leq \exp \left(2 \eta \left[ \min  \left\{\frac {2 (q _ {0} - 1)}{q _ {0}}, \frac {r (1)}{q _ {0}} \right\} \right] ^ {- 1}\right) q _ {0} \leq 2 q _ {0}.
$$
$$
\left| p _ {t} \right| \leq \left[ 1 - \min  \left\{\frac {2 \left(q _ {0} - 1\right)}{q _ {0}}, \frac {r (1)}{q _ {0}} \right\} \right] ^ {t}, \quad q _ {0} \leq q _ {t} \leq \exp \left(2 \eta \left[ \min  \left\{\frac {2 \left(q _ {0} - 1\right)}{q _ {0}}, \frac {r (1)}{q _ {0}} \right\} \right] ^ {- 1}\right) q _ {0} \leq 2 q _ {0}
$$
By induction assumptions, we have  $r(1) \leq r(p_k) \leq 1$  and  $q_0 \leq q_k \leq 2q_0$ . From Eq. (7), we get
$$
\left| \frac {p _ {k + 1}}{p _ {k}} \right| = \left| 1 - \frac {2 r (p _ {k})}{q _ {k}} \right| \leq \max  \left\{1 - \frac {2 r (1)}{2 q _ {0}}, - 1 + \frac {2}{q _ {0}} \right\} = 1 - \frac {\min  \left\{2 (q _ {0} - 1) , r (1) \right\}}{q _ {0}}.
$$
Due to the induction assumption, we obtain the desired bound on  $|p_{k + 1}|$  as following:
$$
\left| p _ {k + 1} \right| \leq \left[ 1 - \min  \left\{\frac {2 \left(q _ {0} - 1\right)}{q _ {0}}, \frac {r (1)}{q _ {0}} \right\} \right] ^ {k + 1}.
$$
Moreover, for any  $0 \leq t \leq k$ , by Eq. (7) we have
$$
1 - 2 \eta p _ {t} ^ {2} \leq (1 - \eta p _ {t} ^ {2}) ^ {2} \leq \frac {q _ {t}}{q _ {t + 1}} = (1 - \eta \phi (p _ {t}) ^ {2}) ^ {2} \leq 1,
$$
where the second inequality comes from the fact that  $\phi$  is 1-Lipschitz and  $\phi(0) = 0$  (Assumption 5.1). Hence, we have  $q_{k+1} \geq q_k \geq q_0$ . Note that  $\frac{q_t}{q_{t+1}} \in [\frac{1}{2}, 1]$  for small  $\eta$ . Consequently, we have
$$
\begin{array}{l} \left| \log \left(\frac {q _ {0}}{q _ {k + 1}}\right) \right| \leq \sum_ {t = 0} ^ {k} \left| \log \left(\frac {q _ {t}}{q _ {t + 1}}\right) \right| \leq 2 \sum_ {t = 0} ^ {k} \left| \frac {q _ {t}}{q _ {t + 1}} - 1 \right| \\ \leq 2 \eta \sum_ {t = 0} ^ {k} p _ {t} ^ {2} \\ \leq 2 \eta \sum_ {t = 0} ^ {k} \left[ 1 - \min  \left\{\frac {2 \left(q _ {0} - 1\right)}{q _ {0}}, \frac {r (1)}{q _ {0}} \right\} \right] ^ {2 t} \\ \leq 2 \eta \left[ \min  \left\{\frac {2 (q _ {0} - 1)}{q _ {0}}, \frac {r (1)}{q _ {0}} \right\} \right] ^ {- 1}, \\ \end{array}
$$
where the second inequality holds since  $|\log (1 + z)|\leq 2|z|$  if  $|z|\leq \frac{1}{2}$ . Therefore, we obtain the desired bound on  $q_{k + 1}$  as following:
$$
q _ {0} \leq q _ {k + 1} \leq \exp \left(2 \eta \left[ \min  \left\{\frac {2 (q _ {0} - 1)}{q _ {0}}, \frac {r (1)}{q _ {0}} \right\} \right] ^ {- 1}\right) q _ {0}.
$$
Since  $q_0 \geq \frac{1}{1 - 2\eta}$  and  $q_0 \leq \frac{r(1)}{4\eta}$ , we have
$$
\min  \left\{\frac {2 (q _ {0} - 1)}{q _ {0}}, \frac {r (1)}{q _ {0}} \right\} \geq 4 \eta .
$$
This implies that  $q_{k + 1} \leq \exp \left(\frac{1}{2}\right) q_0 \leq 2 q_0$ , as desired.
C.3 Proof of Theorem C.2
In this subsection, we prove Theorem C.2. We use the following notation:
All the lemmas in this subsection are stated in the context of Theorem C.2. The proof structure resembles that of Theorem 4.2. We informally summarize the lemmas used in the proof of Theorem C.2. Lemma C.6 proves that  $p_t$  is bounded by a constant and  $q_t$  increases monotonically with the increment bounded by  $\mathcal{O}(\eta)$ . Lemma C.7 states that in the early phase of training, there exists a time step  $t_0$  where  $s_{t_0}$  becomes smaller or equal to  $\frac{2}{2 - r(1)}$ , which is smaller than 2. Lemma C.9 demonstrates that if  $s_t$  is smaller than 2 and  $|p_t| \geq \hat{r}(1 - \frac{\delta}{4})$ , then  $|s_t - 1|$  decreases exponentially. For the case where  $|p_t| < \hat{r}(1 - \frac{\delta}{4})$ , Lemma C.10 proves that  $|p_t|$  increases at an exponential rate. Moreover, Lemma C.8 shows that if  $s_t < 1$  at some time step, then  $s_{t+1}$  is upper bounded by  $1 + \mathcal{O}(\eta)$ . Combining these findings, Proposition C.11 establishes that in the early phase of training, there exists a time step  $t_a^*$  such that  $s_{t_a^*} = 1 + \mathcal{O}_{\delta, \phi}(\eta)$ . Lastly, Lemma C.12 demonstrates that if  $s_t = 1 + \mathcal{O}_{\delta, \phi}(\eta)$ , then  $|s_t - 1 - h(p_t)\eta|$  decreases exponentially.
Lemma C.6. Suppose that the initialization  $(p_0, q_0)$  satisfies  $|p_0| \leq 1$  and  $q_0 \in (c_0, 1 - \delta)$ . Then for any  $t \geq 0$  such that  $q_t \leq 1$ , it holds that
$$
\left| p _ {t} \right| \leq 4, a n d q _ {t} \leq q _ {t + 1} \leq (1 + \mathcal {O} (\eta)) q _ {t}.
$$
Proof. We prove by induction. We assume that for some  $t \geq 0$ , it holds that  $|p_t| \leq 4$  and  $\frac{1}{2} \leq q_t \leq 1$ . We will prove that  $|p_{t+1}| \leq 4$  and  $\frac{1}{2} \leq q_t \leq q_{t+1} \leq (1 + \mathcal{O}(\eta)) q_t$ . For the base case,  $|p_0| \leq 1 \leq 4$  and  $\frac{1}{2} \leq c_0 < q_t \leq 1$  holds by the assumptions on the initialization. Now suppose that for some  $t \geq 0$ , it holds that  $|p_t| \leq 4$  and  $\frac{1}{2} \leq q_t \leq 1$ . By Eq. (7),
$$
\left| p _ {t + 1} \right| = \left| p _ {t} - \frac {2 \phi \left(p _ {t}\right) \phi^ {\prime} \left(p _ {t}\right)}{q _ {t}} \right| \leq \max  \left\{\left| p _ {t} \right|, \frac {2}{q _ {t}} \right\} \leq 4.
$$
where we used Assumption 5.1 to bound  $|\phi(p_t)\phi'(p_t)| \leq 1$ . Moreover,
$$
1 - 2 \eta \leq (1 - \eta) ^ {2} \leq \frac {q _ {t}}{q _ {t + 1}} = (1 - \eta \phi (p _ {t}) ^ {2}) ^ {2} \leq 1,
$$
since  $|\phi|$  is bounded by 1. Hence,  $q_{t} \leq q_{t+1} \leq (1 + \mathcal{O}(\eta)) q_{t}$ , as desired.
Lemma C.6 implies that  $p_t$  is bounded by a constant throughout the iterations, and  $q_t$  monotonically increases slowly, where the increment for each step is  $\mathcal{O}(\eta)$ . Hence, there exists a time step  $T = \Omega(\delta \eta^{-1}) = \Omega_{\delta}(\eta^{-1})$  such that for any  $t \leq T$ , it holds that  $q_t \leq 1 - \frac{\delta}{2}$ . Through out this subsection, we focus on these  $T$  early time steps. Note that for all  $0 \leq t \leq T$ , it holds that  $q_t \in (c_0, 1 - \frac{\delta}{2})$ .
Lemma C.7. There exists a time step  $t_0 = \mathcal{O}_{\delta, \phi}(1)$  such that  $s_{t_0} \leq \frac{2}{2 - r(1)}$ .
Proof. We start by proving the following statement: for any  $0 \leq t \leq T$ , if  $\frac{2}{2 - r(1)} < s_t < 2r(1)^{-1}$ , then  $s_{t + 1} < 2r(1)^{-1}$  and  $|p_{t + 1}| \leq (1 - r(1))|p_t|$ . Suppose that  $\frac{2}{2 - r(1)} < s_t < 2r(1)^{-1}$ . Then from Eq. (7), it holds that
$$
\left| \frac {p _ {t + 1}}{p _ {t}} \right| = \left| 1 - \frac {2}{s _ {t}} \right| \leq 1 - r (1).
$$
Hence,  $|p_{t+1}| \leq (1 - r(1))|p_t|$ . Now we prove  $s_{t+1} < 2r(1)^{-1}$ . Assume the contrary that  $s_{t+1} \geq 2r(1)^{-1}$ . Then,  $r(p_{t+1}) = \frac{q_{t+1}}{s_{t+1}} < q_{t+1} < 1 - \frac{\delta}{2}$  so that  $|p_{t+1}| \geq \hat{r}(1 - \frac{\delta}{2})$ . By Mean Value Theorem, there exists  $p_t^* \in (|p_{t+1}|, |p_t|)$  such that
$$
\begin{array}{l} \frac {1}{r (p _ {t + 1})} = \frac {1}{r (| p _ {t} | - (| p _ {t} | - | p _ {t + 1} |))} = \frac {1}{r (p _ {t})} + \frac {r ^ {\prime} (p _ {t} ^ {*})}{r (p _ {t} ^ {*}) ^ {2}} (| p _ {t} | - | p _ {t + 1} |) \\ \leq \frac {1}{r (p _ {t})} + \frac {r ^ {\prime} (| p _ {t + 1} |)}{r (p _ {t + 1}) ^ {2}} (r (1) | p _ {t} |) \\ \leq \frac {1}{r \left(p _ {t}\right)} - \frac {\left| r ^ {\prime} \left(\hat {r} \left(1 - \frac {\delta}{2}\right)\right) \right|}{\left. \left(1 - \frac {\delta}{2}\right) ^ {2} \right.} \left(r (1) \hat {r} \left(1 - \frac {\delta}{2}\right)\right) \\ = \frac {1}{r (p _ {t})} - \Omega_ {\delta , \phi} (1), \\ \end{array}
$$
where we used Assumption C.2 (i) and  $\hat{r}(1 - \frac{\delta}{2}) \leq |p_{t+1}| \leq (1 - r(1))|p_t|$ . Consequently,
$$
s _ {t + 1} = \frac {q _ {t + 1}}{r (p _ {t + 1})} = (1 + \mathcal {O} (\eta)) q _ {t} \left(\frac {1}{r (p _ {t})} - \Omega_ {\delta , \phi} (1)\right) \leq \frac {q _ {t}}{r (p _ {t})} = s _ {t} <   2 r (1) ^ {- 1},
$$
We proved that for any  $0 \leq t \leq T$ , if  $\frac{2}{2 - r(1)} < s_t < 2r(1)^{-1}$ , it holds that  $s_{t + 1} < 2r(1)^{-1}$  and  $|p_{t + 1}| \leq (1 - r(1))|p_t|$ . At initialization,  $|p_0| \leq 1$  and  $q_0 < 1$ , so that  $s_0 < r(1)^{-1}$ . If  $s_0 \leq \frac{2}{2 - r(1)}$ , then  $t_0 = 0$  is the desired time step. Suppose that  $s_0 > \frac{2}{2 - r(1)}$ . Then, we have  $s_1 < 2r(1)^{-1}$  and  $|p_1| \leq (1 - r(1))|p_0| \leq 1 - r(1)$ . Then we have either  $s_1 \leq \frac{2}{2 - r(1)}$ , or  $\frac{2}{2 - r(1)} < s_1 < 2r(1)^{-1}$ . In the previous case,  $t_0 = 1$  is the desired time step. In the latter case, we can repeat the same argument and obtain  $s_2 < 2r(1)^{-1}$  and  $|p_2| \leq (1 - r(1))^2$ . By inductively repeating the same argument, we can obtain a time step  $t_0 \leq \log (\hat{r}(1 - \frac{\delta}{2})) / \log (1 - r(1)) = \mathcal{O}_{\delta,\phi}(1)$  such that either  $s_{t_0} \leq \frac{2}{2 - r(1)}$ , or  $|p_{t_0}| \leq \hat{r}(1 - \frac{\delta}{2})$ . In the latter case,  $r(p_{t_0}) \geq 1 - \frac{\delta}{2} > q_{t_0}$ , and hence  $s_{t_0} < 1 < \frac{2}{2 - r(1)}$ . Therefore,  $t_0 = \mathcal{O}_{\delta,\phi}(1)$  is the desired time step satisfying  $s_{t_0} \leq \frac{2}{2 - r(1)}$ .
According to Lemma C.7, there exists a time step  $t_0 = \mathcal{O}_{\delta,\phi}(1)$  such that  $s_{t_0} \leq \frac{2}{2 - r(1)} < 2(1 + \eta^2)^{-1}$  for small step size  $\eta$ . Now we prove the lemma below.
Lemma C.8. Suppose that  $s_t \leq 1$ . Then, it holds that  $s_{t+1} \leq 1 + \mathcal{O}(\eta)$ .
Proof. For any  $p \in (0, \hat{r}(\frac{q_t}{2}))$ , we have  $r(p) \geq \frac{q_t}{2}$  so that  $|f_{q_t}(p)| = (-1 + \frac{2r(p_t)}{q_t})p_t$ . Hence,
$$
\frac {\partial}{\partial p} \left| f _ {q _ {t}} (p) \right| = \frac {2 r (p)}{q _ {t}} \left(1 + \frac {p r ^ {\prime} (p)}{p}\right) - 1,
$$
for any  $p \in (0, \hat{r}(\frac{q_t}{2}))$ . By Assumption C.2 (ii), (iv) and  $q_t \geq c_0 \geq r(z_1) + \frac{1}{2} \geq 2r(z_1)$  where  $z_1 = \sup_{z} \left\{ \frac{z r'(z)}{r(z)} \geq -1 \right\}$ , both  $r(p)$  and  $(1 + \frac{pr'(p)}{r(p)})$  are positive, decreasing function on  $(0, \hat{r}(\frac{q_t}{2}))$ . Consequently,  $\frac{\partial}{\partial p} |f_{q_t}(p)|$  is a decreasing function on  $(0, \hat{r}(\frac{q_t}{2}))$ .
where the inequalities used Assumption C.2 (ii) and  $q_{t} > c_{0} \geq r(z_{0})$  where  $z_{0} \coloneqq \sup_{z} \left\{ \frac{zr'(z)}{r(z)} \geq -\frac{1}{2} \right\}$ , from the statement of Theorem C.2.
$$
\left| p _ {t + 1} \right| = \left(- 1 + \frac {2}{s _ {t}}\right) \left| p _ {t} \right| = \left| f _ {q _ {t}} \left(p _ {t}\right) \right| \leq \left| f _ {q _ {t}} (\hat {r} \left(q _ {t}\right)) \right| = \hat {r} \left(q _ {t}\right).
$$
Therefore, with this inequality and Lemma C.6, we can conclude that
$$
s _ {t + 1} = \frac {q _ {t + 1}}{r (p _ {t + 1})} = \frac {(1 + \mathcal {O} (\eta)) q _ {t}}{r (p _ {t + 1})} \leq \frac {q _ {t}}{r (\hat {r} (q _ {t}))} + \mathcal {O} (\eta) = 1 + \mathcal {O} (\eta).
$$
Using Lemma C.8, we prove the following lemma.
Lemma C.9. For any  $0 \leq t \leq T$ , if  $s_t < 2$  and  $r(p_t) \leq 1 - \frac{\delta}{4}$ , then
$$
\left| s _ {t + 1} - 1 \right| \leq (1 - d) \left| s _ {t} - 1 \right| + \mathcal {O} (\eta),
$$
where  $d \in (0, \frac{1}{2}]$  is a constant which depends on  $\delta$  and  $\phi$ .
Proof. From Eq. (7) it holds that
$$
\frac {p _ {t + 1}}{p _ {t}} = 1 - \frac {2}{s _ {t}} <   0,
$$
so that  $p_t$  and  $p_{t+1}$  have opposite signs. By Mean Value Theorem, there exists  $\theta_t$  between  $-1$  and  $(1 - \frac{2}{s_t})$  satisfying
$$
\begin{array}{l} \frac {1}{r \left(p _ {t + 1}\right)} = \frac {1}{r \left(- p _ {t} + \left(\frac {2 \left(s _ {t} - 1\right)}{s _ {t}}\right) p _ {t}\right)} \\ = \frac {1}{r (- p _ {t})} - \frac {r ^ {\prime} (\theta_ {t} p _ {t})}{r (\theta_ {t} p _ {t}) ^ {2}} \left(\frac {2 (s _ {t} - 1)}{s _ {t}}\right) p _ {t} \\ = \frac {1}{r \left(p _ {t}\right)} - \frac {\left| r ^ {\prime} \left(\theta_ {t} p _ {t}\right) \right|}{r \left(\theta_ {t} p _ {t}\right) ^ {2}} \left(\frac {2 \left(s _ {t} - 1\right)}{s _ {t}}\right) \left| p _ {t} \right|. \tag {17} \\ \end{array}
$$
where the last equality used the fact that  $p_t$  and  $\theta_t p_t$  have opposite signs and  $r'(z)$  and  $z$  have opposite signs. Note that  $|\theta_t p_t|$  is between  $|p_t|$  and  $|p_{t+1}|$ . Consequently, the value  $\frac{|r'(\theta_t p_t)|}{r(\theta_t p_t)^2}$  is between  $\frac{|r'(p_t)|}{r(p_t)^2}$  and  $\frac{|r'(p_{t+1})|}{r(p_{t+1})^2}$  by Assumption C.2 (i). We will prove the current lemma based on Eq. (17). We divide into following three cases: (1)  $s_t \geq 1$  and  $s_{t+1} \geq 1$ , (2)  $s_t \geq 1$  and  $s_t < 1$ , and (3)  $s_t < 1$ .