SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Case 1. Suppose that  $s_t \geq 1$  and  $s_{t+1} \geq 1$ . Here, we have  $|p_t| \geq \hat{r}(q_t) \geq \hat{r}(1 - \frac{\delta}{2})$  and similarly  $|p_{t+1}| \geq \hat{r}(1 - \frac{\delta}{2})$ . By Assumption C.2 (i),  $\frac{|r'(\theta_t p_t)|}{r(\theta_t p_t)^2} \geq \frac{|r'(\hat{r}(1 - \frac{\delta}{2}))|}{(1 - \frac{\delta}{2})^2}$ . Hence, Eq. (17) gives
Consequently, by Lemma C.6,
$$
\begin{array}{l} s _ {t + 1} = \frac {q _ {t} (1 + \mathcal {O} (\eta))}{r (p _ {t + 1})} = \frac {q _ {t}}{r (p _ {t + 1})} + \mathcal {O} (\eta) \leq s _ {t} - \frac {| r ^ {\prime} (\hat {r} (1 - \frac {\delta}{2})) |}{(1 - \frac {\delta}{2}) ^ {2}} \left(\frac {2 (s _ {t} - 1)}{s _ {t}}\right) \hat {r} \left(1 - \frac {\delta}{2}\right) q _ {t} + \mathcal {O} (\eta) \\ \leq s _ {t} - \frac {| r ^ {\prime} (\hat {r} (1 - \frac {\delta}{2})) |}{(1 - \frac {\delta}{2}) ^ {2}} (s _ {t} - 1) \hat {r} \left(1 - \frac {\delta}{2}\right) \frac {1}{2} + \mathcal {O} (\eta) \\ \end{array}
$$
$$
\leq s _ {t} - \frac {\hat {r} (1 - \frac {\delta}{2}) | r ^ {\prime} (\hat {r} (1 - \frac {\delta}{2})) |}{2 (1 - \frac {\delta}{2}) ^ {2}} (s _ {t} - 1) + \mathcal {O} (\eta),
$$
where we used  $q_{t} > c_{0} > \frac{1}{2}$  and  $s_t < 2$ . Therefore, we can obtain the following inequality:
$$
0 \leq s _ {t + 1} - 1 \leq \left(1 - \frac {\hat {r} (1 - \frac {\delta}{2}) | r ^ {\prime} (\hat {r} (1 - \frac {\delta}{2})) |}{2 (1 - \frac {\delta}{2}) ^ {2}}\right) (s _ {t} - 1) + \mathcal {O} (\eta).
$$
Case 2. Suppose that  $s_t \geq 1$  and  $s_{t+1} < 1$ . Here, we have  $r(p_{t+1}) > q_{t+1} \geq q_t \geq r(p_t)$ , so that  $|p_{t+1}| < |p_t|$ . Consequently,  $\frac{|r'(\theta_t p_t)|}{r(\theta_t p_t)^2} \leq \frac{|r'(p_t)|}{r(p_t)^2}$  by Assumption C.2 (i). Hence, we can deduce from Eq. (17) that
$$
\begin{array}{l} \frac {1}{r \left(p _ {t + 1}\right)} \geq \frac {1}{r \left(p _ {t}\right)} - \frac {\left| r ^ {\prime} \left(p _ {t}\right) \right|}{r \left(p _ {t}\right) ^ {2}} \left(\frac {2 \left(s _ {t} - 1\right)}{s _ {t}}\right) \left| p _ {t} \right| \\ = \frac {1}{r \left(p _ {t}\right)} - \frac {2 \left| p _ {t} r ^ {\prime} \left(p _ {t}\right) \right|}{r \left(p _ {t}\right) q _ {t}} \left(s _ {t} - 1\right) \\ = \frac {1}{r \left(p _ {t}\right)} + \frac {2 p _ {t} r ^ {\prime} \left(p _ {t}\right)}{r \left(p _ {t}\right) q _ {t}} \left(s _ {t} - 1\right). \\ \end{array}
$$
Consequently, by Assumption C.2 (ii),
$$
s _ {t + 1} \geq \frac {q _ {t}}{r \left(p _ {t + 1}\right)} \geq s _ {t} + \frac {2 p _ {t} r ^ {\prime} \left(p _ {t}\right)}{r \left(p _ {t}\right)} \left(s _ {t} - 1\right) \geq s _ {t} + \frac {2 \hat {r} \left(\frac {c _ {0}}{2}\right) r ^ {\prime} \left(\hat {r} \left(\frac {c _ {0}}{2}\right)\right)}{r \left(\hat {r} \left(\frac {c _ {0}}{2}\right)\right)} \left(s _ {t} - 1\right),
$$
where we used  $r(p_{t})\geq \frac{q_{t}}{2} >\frac{c_{0}}{2}$ . Therefore, we can obtain the following inequality:
$$
0 \leq 1 - s _ {t + 1} \leq - \left(1 + \frac {4 \hat {r} \left(\frac {c _ {0}}{2}\right) r ^ {\prime} \left(\hat {r} \left(\frac {c _ {0}}{2}\right)\right)}{c _ {0}}\right) (s _ {t} - 1),
$$
where  $-1 < \frac{\hat{r}\left(\frac{c_0}{2}\right)r'(\hat{r}\left(\frac{c_0}{2}\right))}{r(\hat{r}\left(\frac{c_0}{2}\right))} = \frac{2\hat{r}\left(\frac{c_0}{2}\right)r'(\hat{r}\left(\frac{c_0}{2}\right))}{c_0} < 0$ , since  $c_{0} \geq r(z_{1}) + \frac{1}{2} \geq 2r(z_{1})$  with  $z_{1} = \sup_{z}\left\{\frac{zr'(z)}{r(z)} \geq -1\right\}$ , and  $r(z_{1}) \leq \frac{1}{2}$  holds by Assumption C.2 (iv).
Case 3. Suppose that  $s_t < 1$ . By Lemma C.8, it holds that  $s_{t+1} \leq 1 + \mathcal{O}(\eta)$ . Moreover, we assumed  $r(p_t) \leq 1 - \frac{\delta}{4}$ , so that  $|p_t| \geq \hat{r}(1 - \frac{\delta}{4})$ . We also have
$$
\left| p _ {t + 1} \right| = \left(- 1 + \frac {2}{s _ {t}}\right) \left| p _ {t} \right| > \left| p _ {t} \right| \geq \hat {r} \left(1 - \frac {\delta}{4}\right).
$$
Consequently, by Assumption C.2 (i), it holds that  $\frac{|r'(\theta_t p_t)|}{r(\theta_t p_t)^2} \geq \frac{|r'(\hat{r}(1 - \frac{\delta}{4}))|}{(1 - \frac{\delta}{4})^2}$ . Hence, by Eq. (17), we have
$$
s _ {t + 1} \geq \frac {q _ {t}}{r (p _ {t + 1})} \geq s _ {t} + \frac {\hat {r} (1 - \frac {\delta}{4}) | r ^ {\prime} (\hat {r} (1 - \frac {\delta}{4})) |}{(1 - \frac {\delta}{4}) ^ {2}} (1 - s _ {t}),
$$
where we used  $q_{t} > \frac{1}{2}$ . Therefore, we can obtain the following inequality:
$$
- \mathcal {O} (\eta) \leq 1 - s _ {t + 1} \leq \left(1 - \frac {\hat {r} (1 - \frac {\delta}{4}) | r ^ {\prime} (\hat {r} (1 - \frac {\delta}{4})) |}{(1 - \frac {\delta}{4}) ^ {2}}\right) (1 - s _ {t}),
$$
where the first inequality is from Lemma C.8.
$$
d := \min  \left\{\frac {1}{2}, \frac {\hat {r} (1 - \frac {\delta}{2}) | r ^ {\prime} (\hat {r} (1 - \frac {\delta}{2})) |}{2 (1 - \frac {\delta}{2}) ^ {2}}, 2 \left(1 + \frac {2 \hat {r} (\frac {c _ {0}}{2}) r ^ {\prime} (\hat {r} (\frac {c _ {0}}{2}))}{c _ {0}}\right), \frac {\hat {r} (1 - \frac {\delta}{4}) | r ^ {\prime} (\hat {r} (1 - \frac {\delta}{4})) |}{(1 - \frac {\delta}{4}) ^ {2}} \right\} \in \left(0, \frac {1}{2} \right],
$$
then  $|s_{t + 1} - 1| \leq (1 - d)|s_t - 1| + \mathcal{O}(\eta)$ , as desired.
Lemma C.9 implies that if  $s_t < 2$  and  $|p_t| \geq \hat{r}(1 - \frac{\delta}{4})$ , then  $|s_t - 1|$  exponentially decreases. We prove Lemma C.10 to handle the regime  $|p_t| < \hat{r}(1 - \frac{\delta}{4})$ , which is stated below.
Lemma C.10. For any  $0 \leq t \leq T$ , if  $r(p_t) \geq 1 - \frac{\delta}{4}$ , it holds that
$$
\left| \frac {p _ {t + 1}}{p _ {t}} \right| \geq \frac {2}{2 - \delta}.
$$
$$
\left| \frac {p _ {t + 1}}{p _ {t}} \right| = \frac {2}{s _ {t}} - 1 \geq \frac {2 (4 - \delta)}{4 - 2 \delta} - 1 = \frac {2}{2 - \delta}.
$$
Now we prove Proposition C.11, which proves that  $s_t$  reaches close to 1 with error bound of  $\mathcal{O}(\eta)$ . Proposition C.11. There exists a time step  $t_a^* = \mathcal{O}_{\delta, \phi}(\log(\eta^{-1}))$  satisfying
$$
s _ {t _ {\alpha} ^ {*}} = 1 + \mathcal {O} _ {\delta , \phi} (\eta). \tag {18}
$$
Proof. By Lemma C.7, there exists a time step  $t_0 = \mathcal{O}_{\delta, \phi}(1)$  such that  $s_{t_0} \leq \frac{2}{2 - r(1)}$ . Here, we divide into two possible cases: (1)  $s_{t_0} < 1$ , and (2)  $1 \leq s_{t_0} \leq \frac{2}{2 - r(1)}$ .
Case 1. Suppose that  $s_{t_0} < 1$ . By Lemma C.10, if  $r(p_{t_0}) \geq 1 - \frac{\delta}{4}$  (or equivalently,  $|p_{t_0}| \leq \hat{r}(1 - \frac{\delta}{4})$ ), then there exists a time step  $t_1 \leq t_0 + \log \left( \frac{\hat{r}(1 - \frac{\delta}{4})}{|p_{t_0}|} \right) / \log \left( \frac{2}{2 - \delta} \right) = \mathcal{O}_{\delta, \phi}(1)$  such that  $|p_{t_1}| \geq \hat{r}(1 - \frac{\delta}{4})$ . We denote the first time step satisfying  $|p_{t_1}| \geq \hat{r}(1 - \frac{\delta}{4})$  and  $t_1 \geq t_0$  by  $t_1 = \mathcal{O}_{\delta, \phi}(1)$ . By Lemma C.8, it holds that  $s_{t_1} \leq 1 + \mathcal{O}(\eta)$  since  $s_{t_1 - 1} < 1$ . Consequently, if  $s_{t_1} \geq 1$ , then  $|s_{t_1} - 1| \leq \mathcal{O}(\eta)$  so that  $t_a^* = t_1$  is the desired time step. Hence, it suffices to consider the case when  $s_{t_1} < 1$ . Here, we can apply Lemma C.9 which implies that
$$
\left| s _ {t _ {1} + 1} - 1 \right| \leq (1 - d) \left| s _ {t _ {1}} - 1 \right| + \mathcal {O} (\eta),
$$
where  $d$  is a constant which depends on  $\delta$  and  $\phi$ . Then, there are two possible cases: either  $|s_{t_1} - 1| \leq \mathcal{O}(\eta d^{-1})$ , or  $|s_{t_1 + 1} - 1| \leq (1 - \frac{d}{2}) |s_{t_1} - 1|$ . It suffices to consider the latter case, suppose that  $|s_{t_1 + 1} - 1| \leq (1 - \frac{d}{2}) |s_{t_1} - 1|$ . Since we are considering the case  $s_{t_1} < 1$ , again by Lemma C.8, we have  $s_{t_1 + 1} \leq 1 + \mathcal{O}(\eta)$ . Since  $|\frac{p_{t_1 + 1}}{p_{t_1}}| = \frac{2}{s_{t_1}} - 1 > 1$ , we have  $|p_{t_1 + 1}| \geq |p_{t_1}| \geq \hat{r}(1 - \frac{\delta}{4})$ . This means that we can again apply Lemma C.9 and repeat the analogous argument. Hence, there exists a time step  $t_2 \leq t_1 + \log \left(\frac{\eta}{1 - s_{t_1}}\right) / \log \left(1 - \frac{d}{2}\right) = \mathcal{O}_{\delta, \phi}(\log (\eta^{-1}))$ , such that  $|s_{t_2} - 1| \leq \mathcal{O}(\eta d^{-1}) = \mathcal{O}_{\delta, \phi}(\eta)$ .
Case 2. Suppose that  $1 \leq s_{t_0} \leq \frac{2}{2 - r(1)}$ . Then,  $r(p_{t_0}) \leq q_{t_0} \leq 1 - \frac{\delta}{2}$ , so we can apply Lemma C.9. There are two possible cases: either  $|s_{t_0 + 1} - 1| \leq \mathcal{O}(\eta d^{-1}) = \mathcal{O}_{\delta,\phi}(\eta)$ , or  $|s_{t_0 + 1} - 1| \leq (1 - \frac{d}{2}) |s_{t_0} - 1|$ . It suffices to consider the latter case. If  $s_{t_0 + 1} \geq 1$ , we can again apply Lemma C.9 and repeat the analogous argument. Hence, we can obtain a time step  $t_0' \leq t_0 + \log \left(\frac{\eta}{1 - s_{t_0}}\right) / \log \left(1 - \frac{d}{2}\right) = \mathcal{O}_{\delta,\phi}(\log (\eta^{-1}))$  such that either  $s_{t_0'} < 1$  or  $|s_{t_0'} - 1| = \mathcal{O}_{\delta,\phi}(\eta)$  is satisfied. If  $s_{t_0'} < 1$ , we proved in Case 1 that there exists a time step  $t_2' = t_0' + \mathcal{O}_{\delta,\phi}(\log (\eta^{-1}))$  such that  $|s_{t_2'} - 1| \leq \mathcal{O}_{\delta,\phi}(\eta)$ , and this is the desired bound.
Now we carefully handle the error term  $\mathcal{O}(\eta)$  obtained in Proposition C.11 and a provide tighter bound on  $s_t$  by proving Lemma C.12 stated below.
Lemma C.12. If  $|s_t - 1| \leq \mathcal{O}_{\delta, \phi}(\eta)$ , then it holds that
$$
\left| s _ {t + 1} - 1 - h \left(p _ {t + 1}\right) \eta \right| \leq \left(1 + \frac {2 p _ {t} r ^ {\prime} \left(p _ {t}\right)}{r \left(p _ {t}\right)}\right) \left| s _ {t} - 1 - h \left(p _ {t}\right) \eta \right| + \mathcal {O} _ {\delta , \phi} \left(\eta^ {2} p _ {t} ^ {2}\right),
$$
Proof. Suppose that  $|s_t - 1| \leq \mathcal{O}_{\delta, \phi}(\eta)$ . Then,  $|p_{t + 1}| = \left|1 - \frac{2}{s_t}\right| |p_t| = (1 + \mathcal{O}_{\delta, \phi}(\eta)) |p_t|$ . By Eq. (17) proved in Lemma C.9, there exists  $\epsilon_t = \mathcal{O}_{\delta, \phi}(\eta)$  such that
$$
\begin{array}{l} \frac {1}{r \left(p _ {t + 1}\right)} = \frac {1}{r \left(p _ {t}\right)} + \frac {r ^ {\prime} \left(\left(1 + \epsilon_ {t}\right) p _ {t}\right)}{r \left(\left(1 + \epsilon_ {t}\right) p _ {t}\right) ^ {2}} \left(\frac {2 \left(s _ {t} - 1\right)}{s _ {t}}\right) p _ {t} \\ = \frac {1}{r (p _ {t})} + \left(\frac {r ^ {\prime} (p _ {t})}{r (p _ {t}) ^ {2}} + \mathcal {O} _ {\delta , \phi} (\eta p _ {t})\right) \left(\frac {2 (s _ {t} - 1)}{s _ {t}}\right) p _ {t} \\ = \frac {1}{r (p _ {t})} + \frac {r ^ {\prime} (p _ {t})}{r (p _ {t}) ^ {2}} \left(\frac {2 (s _ {t} - 1)}{s _ {t}}\right) p _ {t} + \mathcal {O} _ {\delta , \phi} (\eta^ {2} p _ {t} ^ {2}), \\ \end{array}
$$
where we used the Taylor expansion on  $\frac{r'(p)}{r(p)^2}$  with the fact that  $\frac{d}{dp}\left(\frac{r'(p)}{r(p)^2}\right)$  is bounded on  $[-4, 4]$  and that  $|p_t| \leq 4$  to obtain the second equality. Note that  $q_{t+1} = (1 - \eta \phi(p_t)^2)^{-2} q_t = (1 + 2\eta \phi(p_t)^2) q_t + \mathcal{O}(\eta^2)$  by Eq (7). Consequently,
$$
\begin{array}{l} s _ {t + 1} = \left(1 + 2 \eta \phi \left(p _ {t}\right) ^ {2}\right) \left(s _ {t} + \frac {r ^ {\prime} \left(p _ {t}\right)}{r \left(p _ {t}\right) ^ {2}} \left(\frac {2 \left(s _ {t} - 1\right)}{s _ {t}}\right) p _ {t} q _ {t}\right) + \mathcal {O} _ {\delta , \phi} \left(\eta^ {2} p _ {t} ^ {2}\right) \\ = (1 + 2 \eta \phi (p _ {t}) ^ {2}) s _ {t} + \frac {2 p _ {t} r ^ {\prime} (p _ {t})}{r (p _ {t})} (s _ {t} - 1) + \mathcal {O} _ {\delta , \phi} (\eta^ {2} p _ {t} ^ {2}) \\ = 1 + \left(1 + \frac {2 p _ {t} r ^ {\prime} (p _ {t})}{r (p _ {t})}\right) \left(s _ {t} - 1\right) + 2 \eta \phi \left(p _ {t}\right) ^ {2} + \mathcal {O} _ {\delta , \phi} \left(\eta^ {2} p _ {t} ^ {2}\right). \\ \end{array}
$$
Note that  $h$  is even, and twice continuously differentiable function by Lemma C.13. Consequently,  $h'(0) = 0$  and  $h'(p) = \mathcal{O}_{\phi}(p)$ , since  $h''$  is bounded on closed interval. Consequently,  $h(p_{t+1}) = h((1 + \mathcal{O}_{\delta, \phi}(\eta)) p_t) = h(p_t) + \mathcal{O}_{\delta, \phi}(\eta p_t^2)$ . Hence, we can obtain the following:
$$
\begin{array}{l} s _ {t + 1} - 1 - h \left(p _ {t + 1}\right) \eta = s _ {t + 1} - 1 - h \left(p _ {t}\right) \eta + \mathcal {O} _ {\delta , \phi} \left(\eta^ {2} p _ {t} ^ {2}\right) \\ = s _ {t + 1} - 1 + \frac {\phi (p _ {t}) ^ {2} r (p _ {t})}{p _ {t} r ^ {\prime} (p _ {t})} \eta + \mathcal {O} _ {\delta , \phi} (\eta^ {2} p _ {t} ^ {2}) \\ = \left(1 + \frac {2 p _ {t} r ^ {\prime} (p _ {t})}{r (p _ {t})}\right) \left(s _ {t} - 1 + \frac {\phi (p _ {t}) ^ {2} r (p _ {t})}{p _ {t} r ^ {\prime} (p _ {t})} \eta\right) + \mathcal {O} _ {\delta , \varphi} (\eta^ {2} p _ {t} ^ {2}) \\ = \left(1 + \frac {2 p _ {t} r ^ {\prime} (p _ {t})}{r (p _ {t})}\right) \left(s _ {t} - 1 - h (p _ {t}) \eta\right) + \mathcal {O} _ {\delta , \phi} \left(\eta^ {2} p _ {t} ^ {2}\right) \\ \end{array}
$$
Note that  $r(p_{t}) = (1 + \mathcal{O}_{\delta, \phi}(\eta))q_{t} \geq (1 + \mathcal{O}_{\delta, \phi}(\eta))q_{0} \geq c_{0} \geq r(z_{0})$  for small step size  $\eta$ , where  $z_{0} = \sup \left\{\frac{zr'(z)}{r(z)} \geq -\frac{1}{2}\right\}$ . Consequently, it holds that  $1 + \frac{2p_{t}r'(p_{t})}{r(p_{t})} \geq 0$ . Therefore, we can conclude that
Now we give the proof of Theorem C.2, restated below for the sake of readability.
Proof of Theorem C.2. By Proposition C.11, there exists a time step  $t_a^* = \mathcal{O}_{\delta,\ell}(\log(\eta^{-1}))$  which satisfies:
$$
\left| s _ {t _ {a} ^ {*}} - 1 \right| = \left| \frac {q _ {t _ {a} ^ {*}}}{r \left(p _ {t _ {a} ^ {*}}\right)} - 1 \right| = \mathcal {O} _ {\delta , \phi} (\eta).
$$
By Lemma C.12, there exists a constant  $D > 0$  which depends on  $\delta, \phi$  such that if  $|s_t - 1| = \mathcal{O}_{\delta, \phi}(\eta)$ , then
$$
\left| s _ {t + 1} - 1 - h \left(p _ {t + 1}\right) \eta \right| \leq \left(1 + \frac {2 p _ {t} r ^ {\prime} \left(p _ {t}\right)}{r \left(p _ {t}\right)}\right) \left| s _ {t} - 1 - h \left(p _ {t}\right) \eta \right| + D \eta^ {2} p _ {t} ^ {2}. \tag {19}
$$
Hence, if  $|s_t - 1| = \mathcal{O}_{\delta, \phi}(\eta)$  and  $|s_t - 1 - h(p_t)\eta| \geq \left(-\frac{p_t r(p_t)}{r'(p_t)}\right)D\eta^2$ , then
$$
\left| s _ {t + 1} - 1 - h \left(p _ {t + 1}\right) \eta \right| \leq \left(1 + \frac {p _ {t} r ^ {\prime} \left(p _ {t}\right)}{r \left(p _ {t}\right)}\right) \left| s _ {t} - 1 - h \left(p _ {t}\right) \eta \right|. \tag {20}
$$
For any  $t \leq T$ , we have  $q_t < 1 - \frac{\delta}{2}$  so that if  $|s_t - 1| = \mathcal{O}_{\delta, \phi}(\eta)$ , then  $r(p_t) \leq (1 + \mathcal{O}_{\delta, \phi}(\eta)) q_t < 1 - \frac{\delta}{4}$  for small step size  $\eta$ . From Eq. (20) with  $t = t_a^*$ , we have either
$$
| s _ {t _ {a} ^ {*}} - 1 - h (p _ {t _ {a} ^ {*}}) \eta | <   \left(- \frac {p _ {t _ {a} ^ {*}} r (p _ {t _ {a} ^ {*}})}{r ^ {\prime} (p _ {t _ {a} ^ {*}})}\right) D \eta^ {2},
$$
$$
| s _ {t _ {a} ^ {*} + 1} - 1 - h (p _ {t _ {a} ^ {*} + 1}) \eta | \leq \left(1 + \frac {\hat {r} (1 - \frac {\delta}{4}) r ^ {\prime} (\hat {r} (1 - \frac {\delta}{4}))}{(1 - \frac {\delta}{4})}\right) | s _ {t _ {a} ^ {*}} - 1 - h (p _ {t _ {a} ^ {*}}) \eta |,
$$
where we used Assumption C.2 (ii) and  $|p_t| > \hat{r}(1 - \frac{\delta}{4})$ . In the latter case,  $|s_{t_a^* + 1} - 1| = \mathcal{O}_{\delta, \phi}(\eta)$  continues to hold and we can again use Eq. (20) with  $t = t_a^* + 1$ . By repeating the analogous arguments, we can obtain the time step
$$
t _ {a} \leq t _ {a} ^ {*} + \frac {\log \left(- \frac {D \eta^ {2}}{r ^ {\prime \prime} (0) | s _ {t _ {a} ^ {*}} - 1 - h (p _ {t _ {a} ^ {*}}) \eta |}\right)}{\log \left(1 + \frac {\hat {r} (1 - \frac {\delta}{4}) r ^ {\prime} (\hat {r} (1 - \frac {\delta}{4}))}{(1 - \frac {\delta}{4})}\right)} = \mathcal {O} _ {\delta , \ell} (\log (\eta^ {- 1})),
$$
$$
| s _ {t _ {a}} - 1 - h (p _ {t _ {a}}) \eta | <   \left(- \frac {p _ {t _ {a}} r (p _ {t _ {a}})}{r ^ {\prime} (p _ {t _ {a}})}\right) D \eta^ {2},
$$
$$
\left| s _ {t _ {a}} - 1 - h \left(p _ {t _ {a}}\right) \eta \right| \leq \left(- \frac {1}{r ^ {\prime \prime} (0)}\right) D \eta^ {2} \leq \left(- \frac {p _ {t _ {a}} r \left(p _ {t _ {a}}\right)}{r ^ {\prime} \left(p _ {t _ {a}}\right)}\right) D \eta^ {2} \leq \left(- \frac {4 r (4)}{r ^ {\prime} (4)}\right) D \eta^ {2},
$$
where we used  $|p_t| \leq 4$  from Lemma C.6 and  $-\frac{zr(z)}{r'(z)} \geq -\frac{1}{r''(0)}$  for any  $z$  by Assumption C.2 (iii).
By Eq. (19), if  $|s_t - 1 - h(p_t)\eta| \leq \left(-\frac{4r(4)}{r'(4)}\right)D\eta^2$  is satisfied for any time step  $t$ , then
$$
\left| s _ {t + 1} - 1 - h \left(p _ {t + 1}\right) \eta \right| \leq \left(1 + \frac {2 p _ {t} r ^ {\prime} \left(p _ {t}\right)}{r \left(p _ {t}\right)}\right) \left(- \frac {4 r (4)}{r ^ {\prime} (4)}\right) D \eta^ {2} + D \eta^ {2} p _ {t} ^ {2} \leq \left(- \frac {4 r (4)}{r ^ {\prime} (4)}\right) D \eta^ {2},
$$
by  $|p_t| \leq 4$  from Lemma C.6 and Assumption C.2 (iii).
$$
\left| s _ {t} - 1 - h \left(p _ {t}\right) \eta \right| \leq \left(- \frac {4 r (4)}{r ^ {\prime} (4)}\right) D \eta^ {2} = \mathcal {O} _ {\delta , \ell} (\eta^ {2}),
$$
by  $|p_t| \leq 4$  and Assumption C.2 (iii).
C.4 Proof of Theorem C.3
In this subsection, we prove Theorem C.3. We start by proving Lemma C.13 which provides a useful property of  $h$  defined in Theorem C.2.
Lemma C.13. Consider the function  $h$  defined in Theorem C.2, given by
Proof. By Assumption C.1,  $h$  is a positive, even function. Moreover,  $h$  is continuous since  $\lim_{p\to 0}h(p) = -\frac{1}{r''(0)} = h(0)$ . Continuous function on a compact domain is bounded, so  $h$  is bounded on the closed interval  $[-1,1]$ . Note that  $\phi (p)^2\leq 1$ , and  $\left(-\frac{r(p)}{pr'(p)}\right)$  is positive, decreasing function on  $p > 0$  by Assumption C.2 (ii). Hence,  $h$  is bounded on  $[1,\infty)$ . Since  $h$  is even,  $h$  is bounded on  $(-\infty ,1]$ . Therefore,  $h$  is a bounded on  $\mathbb{R}$ .
We finally prove that  $h$  is twice continuously differentiable. Since  $r$  is even and  $C^4$  on  $\mathbb{R}$ , we can check that for any  $p \neq 0$ ,
$$
h ^ {\prime} (p) = - \frac {2 r (p) ^ {2}}{r ^ {\prime} (p)} - \frac {\phi (p) ^ {2}}{p} + \frac {\phi (p) ^ {2} r (p) (r ^ {\prime} (p) + p r ^ {\prime \prime} (p))}{p ^ {2} r ^ {\prime} (p) ^ {2}},
$$
and  $h^\prime (p) = 0$  . Moreover, for any  $p\neq 0$
$$
\begin{array}{l} h ^ {\prime \prime} (p) = - 6 r (p) + \frac {2 \phi (p) ^ {2}}{p ^ {2}} + \frac {\phi (p) ^ {2} r ^ {\prime \prime} (p)}{p r ^ {\prime} (p)} + \frac {\phi (p) ^ {2} r (p) r ^ {(3)} (p)}{p r ^ {\prime} (p) ^ {2}} \\ + \frac {2 r (p) ^ {2} (r ^ {\prime} (p) + 2 p r ^ {\prime \prime} (p))}{p r ^ {\prime} (p) ^ {2}} - \frac {\phi (p) ^ {2} r (p) (2 r ^ {\prime} (p) + p r ^ {\prime \prime} (p))}{p ^ {3} r ^ {\prime} (p) ^ {2}} - \frac {\phi (p) ^ {2} r (p) r ^ {\prime \prime} (p) (r ^ {\prime} (p) + 2 p r ^ {\prime \prime} (p))}{p ^ {2} r ^ {\prime} (p) ^ {3}}, \\ \end{array}
$$
$$
h ^ {\prime \prime} (0) = - 1 - \frac {2 \phi^ {(3)} (0)}{3 r ^ {\prime \prime} (0)} + \frac {r ^ {(4)} (0)}{3 r ^ {\prime \prime} (0) ^ {2}}.
$$
Now we give the proof of Theorem C.3, restated below for the sake of readability.
Proof. We first prove that there exists a time step  $t \geq 0$  such that  $q_{t} > 1$ . Assume the contrary that  $q_{t} \leq 1$  for all  $t \geq 0$ . Let  $t_{a}$  be the time step obtained in Theorem C.2. Then for any  $t \geq t_{a}$ , we have
$$
r (p _ {t}) = (1 - h (p _ {t}) \eta + \mathcal {O} _ {\delta , \phi} (\eta^ {2})) q _ {t} \leq 1 - \frac {h (p _ {t}) \eta}{2},
$$
for small step size  $\eta$ . The function  $g(p) \coloneqq r(p) - 1 + \frac{h(p)\eta}{2}$  is even, continuous, and has the function value  $g(0) = \frac{\eta}{2|r''(0)|} > 0$ . Consequently, there exists a positive constant  $\epsilon > 0$  such that  $g(p) > 0$  for all  $p \in (-\epsilon, \epsilon)$ . Then, we have  $|p_t| \geq \epsilon$  for all  $t \geq t_a$ , since  $g(p_t) \leq 0$ . This implies that for any  $t \geq t_a$ , it holds that
$$
\frac {q _ {t + 1}}{q _ {t}} = (1 - \eta \phi (p _ {t}) ^ {2}) ^ {- 2} \geq (1 - \eta \phi (\epsilon) ^ {2}) ^ {- 2} > 1,
$$
Therefore, there exists a time step  $t_b$  such that  $q_{t_b} \leq 1$  and  $q_t > 1$  for any  $t > t_b$ , i.e.,  $q_t$  jumps across the value 1. This holds since the sequence  $(q_t)$  is monotonically increasing. For any  $t \leq t_b$ , we have  $q_{t + 1} \leq q_t + \mathcal{O}(\eta)$  by Lemma C.6, and this implies that  $t_b \geq \Omega((1 - q_0)\eta^{-1})$ , as desired.
$$
\left| \frac {p _ {t + 1}}{p _ {t}} \right| = \frac {2 r (p _ {t})}{q _ {t}} - 1 \leq \frac {2}{q _ {t _ {b} + 1}} - 1 <   1.
$$
Hence,  $|p_t|$  is exponentially decreasing for  $t > t_b$ . Therefore,  $p_t$  converges to 0 as  $t \to \infty$ . Since the sequence  $(q_t)_{t=0}^{\infty}$  is monotonically increasing and bounded (due to Theorem C.2, it converges. Suppose that  $(p_t, q_t)$  converges to the point  $(0, q^*)$ . By Theorem C.2, we can conclude that
$$
\left| q ^ {*} - 1 + \frac {\eta}{r ^ {\prime \prime} (0)} \right| = \mathcal {O} _ {\phi , \ell} (\eta^ {2}),
$$
C.5 Proof of Theorem C.4
In this subsection, we prove Theorem C.4. We first prove an important bound on the sharpness value provided by the Proposition C.14 stated below.
Proposition C.14. For any  $(x,y)\in \mathbb{R}^2$  with  $r(x) + x r'(x)\geq 0$ , the following inequality holds:
$$
(r (x) + x r ^ {\prime} (x)) y ^ {2} \leq \lambda_ {\max} (\nabla^ {2} \mathcal {L} (x, y)) \leq (r (x) + x r ^ {\prime} (x)) y ^ {2} + \frac {4 x ^ {2} r (x)}{r (x) + x r ^ {\prime} (x)},
$$
Proof. Let  $(x,y)\in \mathbb{R}^2$  be given. The loss Hessian at  $(x,y)$  is
$$
\nabla^ {2} \mathcal {L} (x, y) = \left[ \begin{array}{c c} (\phi (x) \phi^ {\prime \prime} (x) + \phi^ {\prime} (x) ^ {2}) y ^ {2} & 2 \phi (x) \phi^ {\prime} (x) y \\ 2 \phi (x) \phi^ {\prime} (x) y & \phi (x) ^ {2} \end{array} \right].
$$
Note that the largest absolute value of the eigenvalue of a symmetric matrix equals to its spectral norm. Since trace of the Hessian,  $\mathrm{tr}(\nabla^2\mathcal{L}(x,y)) = (\phi (x)\phi ''(x) + \phi '(x)^2)y^2 +\phi (x)^2 = (r(x) + x r'(x))y^2 +\phi (x)^2\geq 0$  , is non-negative, the spectral norm of the Hessian  $\nabla^2\mathcal{L}(x,y)$  equals to its largest eigenvalue. Hence, we have
$$
\begin{array}{l} \lambda_ {\max } \left(\nabla^ {2} \mathcal {L} (x, y)\right) = \| \nabla^ {2} \mathcal {L} (x, y) \| _ {2} \\ \geq \left\| \nabla^ {2} \mathcal {L} (x, y) \cdot \left[ \begin{array}{c} 1 \\ 0 \end{array} \right] \right\| _ {2} \\ = \left[ \left(\left(\phi (x) \phi^ {\prime \prime} (x) + \phi^ {\prime} (x) ^ {2}\right) y ^ {2}\right) ^ {2} + \left(2 \phi (x) \phi^ {\prime} (x) y\right) ^ {2} \right] ^ {\frac {1}{2}} \\ \geq (\phi (x) \phi^ {\prime \prime} (x) + \phi^ {\prime} (x) ^ {2}) y ^ {2} \\ = (r (x) + x r ^ {\prime} (x)) y ^ {2}, \\ \end{array}
$$
which is the desired lower bound. We also note that for any matrix  $\mathbf{A}$ , the inequality  $\| \mathbf{A}\| _2\leq \| \mathbf{A}\|_{\mathrm{F}}$  holds, where  $\| A\|_{\mathrm{F}}$  is the Frobenius norm. Hence, we have
$$
\begin{array}{l} \lambda_ {\max } \left(\nabla^ {2} \mathcal {L} (x, y)\right) = \| \nabla^ {2} \mathcal {L} (x, y) \| _ {2} \\ \leq \left\| \nabla^ {2} \mathcal {L} (x, y) \right\| _ {\mathrm {F}} \\ = \left[ \left(\left(\phi (x) \phi^ {\prime \prime} (x) + \phi^ {\prime} (x) ^ {2}\right) y ^ {2}\right) ^ {2} + 2 \left(2 \phi (x) \phi^ {\prime} (x) y\right) ^ {2} + \phi (x) ^ {4} \right] ^ {\frac {1}{2}} \\ = \left[ (r (x) + x r ^ {\prime} (x)) ^ {2} y ^ {4} + 8 x ^ {2} r (x) ^ {2} y ^ {2} + \phi (x) ^ {4} \right] ^ {\frac {1}{2}} \\ \leq \left[ \left((r (x) + x r ^ {\prime} (x)) y ^ {2} + \frac {4 x ^ {2} r (x)}{r (x) + x r ^ {\prime} (x)}\right) ^ {2} \right] ^ {\frac {1}{2}} \\ = (r (x) + x r ^ {\prime} (x)) y ^ {2} + \frac {4 x ^ {2} r (x)}{r (x) + x r ^ {\prime} (x)}, \\ \end{array}
$$
which is the desired upper bound.
Now we give the proof of Theorem C.4, restated below for the sake of readability.