SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. By Theorem C.2 and since  $h$  is a bounded function by Lemma C.13, we have  $s_t = 1 + \mathcal{O}_{\phi}(\eta)$  for any  $t \geq t_a$ . Note that  $r(p_t) = (1 + \mathcal{O}_{\delta, \phi}(\eta)) q_t \geq (1 + \mathcal{O}_{\delta, \phi}(\eta)) q_0 \geq c_0 \geq r(z_0)$  for small step size  $\eta$ , where  $z_0 = \sup \left\{ \frac{z r'(z)}{r(z)} \geq -\frac{1}{2} \right\}$ . Consequently, it holds that  $1 + \frac{2 p_t r'(p_t)}{r(p_t)} \geq 0$ , and this implies  $1 + \frac{p_t r'(p_t)}{r(p_t)} \geq \frac{1}{2}$ . By Proposition C.14, we can bound the sharpness  $\lambda_t := \lambda_{\max}(\nabla^2 \mathcal{L}(x_t, y_t))$  by
$$
\left(1 + \frac {p _ {t} r ^ {\prime} (p _ {t})}{r (p _ {t})}\right) \frac {2 r (p _ {t})}{\eta q _ {t}} \leq \lambda_ {t} \leq \left(1 + \frac {p _ {t} r ^ {\prime} (p _ {t})}{r (p _ {t})}\right) \frac {2 r (p _ {t})}{\eta q _ {t}} + 4 p _ {t} ^ {2} \left(1 + \frac {p _ {t} r ^ {\prime} (p _ {t})}{r (p _ {t})}\right) ^ {- 1}.
$$
By Lemma C.6,  $|p_t| \leq 4$  for any time step  $t$ . Moreover, since  $1 + \frac{p_t r'(p_t)}{r(p_t)} \geq \frac{1}{2}$  holds for any  $t \geq t_a$  with small step size  $\eta$ , we have  $4p_t^2 \left( 1 + \frac{p_t r'(p_t)}{r(p_t)} \right)^{-1} = \mathcal{O}_\phi(1)$ . Hence, for any  $t \geq t_a$ , it holds that
$$
\lambda_ {t} = \left(1 + \frac {p _ {t} r ^ {\prime} \left(p _ {t}\right)}{r \left(p _ {t}\right)}\right) \frac {2 r \left(p _ {t}\right)}{\eta q _ {t}} + \mathcal {O} _ {\phi} (1). \tag {21}
$$
For any  $t \geq t_a$ , we have  $s_t = 1 + \mathcal{O}_\phi(\eta)$  and this implies  $|s_t^{-1} - 1| = \mathcal{O}_\phi(\eta)$  and  $|r(p_t) - q_t| = \mathcal{O}_\phi(\eta)$ . For any  $0 < q < 1$ , we have
$$
\begin{array}{l} \frac {d}{d q} \left(\frac {\hat {r} (q) r ^ {\prime} (\hat {r} (q))}{q}\right) = \frac {\hat {r} ^ {\prime} (q) \left(r ^ {\prime} (\hat {r} (q)) + \hat {r} (q) r ^ {\prime \prime} (\hat {r} (q))\right)}{q} - \frac {\hat {r} (q) r ^ {\prime} (\hat {r} (q))}{q ^ {2}} \\ = \frac {1}{q} \left(1 + \frac {\hat {r} (q) r ^ {\prime \prime} (\hat {r} (q))}{r ^ {\prime} (\hat {r} (q))}\right) - \frac {\hat {r} (q) r ^ {\prime} (\hat {r} (q))}{q ^ {2}}, \\ \end{array}
$$
$$
\left| \frac {p _ {t} r ^ {\prime} (p _ {t})}{r (p _ {t})} - \frac {\hat {r} (q _ {t}) r ^ {\prime} (\hat {r} (q _ {t}))}{q _ {t}} \right| \leq \mathcal {O} _ {\phi} (| r (p _ {t}) - q _ {t} |) \leq \mathcal {O} _ {\phi} (\eta).
$$
$$
\left| \tilde {\lambda} \left(q _ {t}\right) - \left(1 + \frac {p _ {t} r ^ {\prime} \left(p _ {t}\right)}{r \left(p _ {t}\right)}\right) \frac {2 r \left(p _ {t}\right)}{\eta q _ {t}} \right| \leq \left| 1 - s _ {t} ^ {- 1} + \frac {\hat {r} \left(q _ {t}\right) r ^ {\prime} (\hat {r} \left(q _ {t}\right))}{q _ {t}} - \frac {p _ {t} r ^ {\prime} \left(p _ {t}\right)}{r \left(p _ {t}\right)} \right| \frac {2}{\eta} + \mathcal {O} _ {\phi} (1) = \mathcal {O} _ {\phi} (1), \tag {22}
$$
where we used  $\frac{p_r r'(p_t)}{q_t} = \frac{p_t r'(p_t)}{r(p_t)} (1 + \mathcal{O}_\phi (\eta)) = \frac{p_t r'(p_t)}{r(p_t)} +\mathcal{O}_\phi (\eta)$ , since  $|p_{t}r^{\prime}(p_{t})|\leq r(p_{t})$  for any  $t\geq t_a$ . Now let  $t$  be any given time step with  $q_{t}\geq 1$ . Then,  $r(p_{t}) = 1 - \mathcal{O}_{\phi}(\eta)$ , and since  $r(z) = 1 + r''(0)z^2 +\mathcal{O}_{\phi}(z^4)$  for small  $z$ , we have  $|p_t| = \mathcal{O}_{\phi}(\sqrt{\eta})$ . Hence,
$$
\left| \tilde {\lambda} \left(q _ {t}\right) - \left(1 + \frac {p _ {t} r ^ {\prime} \left(p _ {t}\right)}{r \left(p _ {t}\right)}\right) \frac {2 r \left(p _ {t}\right)}{\eta q _ {t}} \right| \leq \left| 1 - s _ {t} ^ {- 1} - \frac {p _ {t} r ^ {\prime} \left(p _ {t}\right)}{r \left(p _ {t}\right)} \right| \frac {2}{\eta} + \mathcal {O} _ {\phi} (1) = \mathcal {O} _ {\phi} (1), \tag {23}
$$
for any  $t$  with  $q_{t} \geq 1$ , where we again used  $\frac{p_{r} r^{\prime}(p_{t})}{q_{t}} = \frac{p_{t} r^{\prime}(p_{t})}{r(p_{t})} (1 + \mathcal{O}_{\phi}(\eta)) = \frac{p_{t} r^{\prime}(p_{t})}{r(p_{t})} + \mathcal{O}_{\phi}(\eta)$ . By Eqs. (21), (22), and (23), we can conclude that for any  $t \geq t_{a}$ , we have
$$
\left| \lambda_ {t} - \tilde {\lambda} (q _ {t}) \right| \leq \mathcal {O} _ {\phi} (1),
$$
the desired bound.
Finally, we can easily check that the sequence  $(\tilde{\lambda}(q_t))_{t=0}^{\infty}$  is monotonically increasing, since  $z \mapsto \frac{zr'(z)}{r(z)}$  is a decreasing function by Assumption C.2 (ii) and the sequence  $(q_t)$  is monotonically increasing.