SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} s _ {t + 1} = \left(1 - \eta^ {2} p _ {t} ^ {2} r (p _ {t}) \left(2 q _ {t} - r (p _ {t})\right)\right) ^ {- 1} \left(s _ {t} + \frac {2 p _ {t} r ^ {\prime} (p _ {t})}{r (p _ {t})} \left(s _ {t} - 1\right) + \eta^ {2} p _ {t} ^ {3} r ^ {\prime} (p _ {t}) q _ {t}\right) + \mathcal {O} _ {\delta , \ell} (\eta^ {4} p _ {t} ^ {2}) \\ = (1 + \eta^ {2} p _ {t} ^ {2} r (p _ {t}) (2 q _ {t} - r (p _ {t}))) s _ {t} + \frac {2 p _ {t} r ^ {\prime} (p _ {t})}{r (p _ {t})} (s _ {t} - 1) + \eta^ {2} p _ {t} ^ {3} r ^ {\prime} (p _ {t}) q _ {t} + \mathcal {O} _ {\delta , \ell} (\eta^ {4} p _ {t} ^ {2}) \\ = 1 + \left(1 + \frac {2 p _ {t} r ^ {\prime} (p _ {t})}{r (p _ {t})}\right) \left(s _ {t} - 1\right) + \eta^ {2} p _ {t} ^ {2} r (p _ {t}) \left(2 q _ {t} - r (p _ {t})\right) s _ {t} + \eta^ {2} p _ {t} ^ {3} r ^ {\prime} (p _ {t}) q _ {t} + \mathcal {O} _ {\delta , \ell} \left(\eta^ {4} p _ {t} ^ {2}\right). \\ \end{array}
$$
Here, since  $s_t = 1 + \mathcal{O}_{\delta,\ell}(\eta^2)$ , we can rewrite
$$
\begin{array}{l} \eta^ {2} p _ {t} ^ {2} r (p _ {t}) \left(2 q _ {t} - r (p _ {t})\right) s _ {t} + \eta^ {2} p _ {t} ^ {3} r ^ {\prime} (p _ {t}) q _ {t} \\ = \eta^ {2} p _ {t} ^ {2} r (p _ {t}) ^ {2} (2 s _ {t} - 1) s _ {t} + \eta^ {2} p _ {t} ^ {3} r ^ {\prime} (p _ {t}) r (p _ {t}) s _ {t} \\ = \eta^ {2} p _ {t} ^ {2} r (p _ {t}) ^ {2} + \eta^ {2} p _ {t} ^ {3} r ^ {\prime} (p _ {t}) r (p _ {t}) + \mathcal {O} _ {\delta , \ell} (\eta^ {4} p _ {t} ^ {2}), \\ \end{array}
$$
$$
s _ {t + 1} = 1 + \left(1 + \frac {2 p _ {t} r ^ {\prime} (p _ {t})}{r (p _ {t})}\right) (s _ {t} - 1) + \eta^ {2} p _ {t} ^ {2} r (p _ {t}) ^ {2} + \eta^ {2} p _ {t} ^ {3} r (p _ {t}) r ^ {\prime} (p _ {t}) + \mathcal {O} _ {\delta , \ell} (\eta^ {4} p _ {t} ^ {2}).
$$
Note that  $h$  is even, and twice continuously differentiable function by Lemma B.9. Consequently,  $h'(0) = 0$  and  $h'(p) = \mathcal{O}_{\ell}(p)$ , since  $h''$  is bounded on closed interval. Consequently,  $h(p_{t+1}) = h((1 + \mathcal{O}_{\delta,\ell}(\eta^2)) p_t) = h(p_t) + \mathcal{O}_{\delta,\ell}(\eta^2 p_t^2)$ . Hence, we can obtain the following:
$$
\begin{array}{l} s _ {t + 1} - 1 - h \left(p _ {t + 1}\right) \eta^ {2} = s _ {t + 1} - 1 - h \left(p _ {t}\right) \eta^ {2} + \mathcal {O} _ {\delta , \ell} \left(\eta^ {4} p _ {t} ^ {2}\right) \\ = s _ {t + 1} - 1 + \frac {1}{2} \left(\frac {p _ {t} r (p _ {t}) ^ {3}}{r ^ {\prime} (p _ {t})} + p _ {t} ^ {2} r (p _ {t}) ^ {2}\right) \eta^ {2} + \mathcal {O} _ {\delta , \ell} (\eta^ {4} p _ {t} ^ {2}) \\ = \left(1 + \frac {2 p _ {t} r ^ {\prime} (p _ {t})}{r (p _ {t})}\right) \left(s _ {t} - 1 + \frac {1}{2} \left(\frac {p _ {t} r (p _ {t}) ^ {3}}{r ^ {\prime} (p _ {t})} + p _ {t} ^ {2} r (p _ {t}) ^ {2}\right) \eta^ {2}\right) + \mathcal {O} _ {\delta , \ell} (\eta^ {4} p _ {t} ^ {2}) \\ = \left(1 + \frac {2 p _ {t} r ^ {\prime} (p _ {t})}{r (p _ {t})}\right) (s _ {t} - 1 - h (p _ {t}) \eta^ {2}) + \mathcal {O} _ {\delta , \ell} (\eta^ {4} p _ {t} ^ {2}). \\ \end{array}
$$
Note that  $r(p_{t}) = (1 + \mathcal{O}_{\delta,\ell}(\eta^{2}))q_{t}\geq (1 + \mathcal{O}_{\delta,\ell}(\eta^{2}))q_{0}\geq c_{0}\geq r(z_{0})$  for small step size  $\eta$ , where  $z_0 = \sup \{\frac{zr'(z)}{r(z)}\geq -\frac{1}{2}\}$ . Consequently, it holds that  $1 + \frac{2p_{t}r^{\prime}(p_{t})}{r(p_{t})}\geq 0$ . Therefore, we have the desired inequality:
$$
\left| s _ {t + 1} - 1 - h \left(p _ {t + 1}\right) \eta^ {2} \right| \leq \left(1 + \frac {2 p _ {t} r ^ {\prime} \left(p _ {t}\right)}{r \left(p _ {t}\right)}\right) \left| s _ {t} - 1 - h \left(p _ {t}\right) \eta^ {2} \right| + \mathcal {O} _ {\delta , \ell} \left(\eta^ {4} p _ {t} ^ {2}\right).
$$
We now provide the proof of Theorem 4.2, restated below for the sake of readability.
Theorem 4.2 (EoS regime, Phase I). Let  $\eta$  be a small enough step size and  $\ell$  be a loss function satisfying Assumptions 4.1, 4.2, and 4.3. Let  $z_0 \coloneqq \sup_z \left\{ \frac{z r'(z)}{r(z)} \geq -\frac{1}{2} \right\}$  and  $c_0 \coloneqq \max \{r(z_0), \frac{1}{2}\}$ . Let  $\delta \in (0, 1 - c_0)$  be any given constant. Suppose that the initialization  $(p_0, q_0)$  satisfies  $|p_0| \leq 1$  and  $q_0 \in (c_0, 1 - \delta)$ . Consider the reparameterized GD trajectory characterized in Eq. (6). We assume that for all  $t \geq 0$  such that  $q_t < 1$ , we have  $p_t \neq 0$ . Then, there exists a time step  $t_a = \mathcal{O}_{\delta, \ell}(\log (\eta^{-1}))$ , such that for any  $t \geq t_a$ ,
$$
\frac {q _ {t}}{r \left(p _ {t}\right)} = 1 + h \left(p _ {t}\right) \eta^ {2} + \mathcal {O} _ {\delta , \ell} \left(\eta^ {4}\right),
$$
Proof of Theorem 4.2. By Proposition B.7, there exists a time step  $t_a^* = \mathcal{O}_{\delta,\ell}(\log(\eta^{-1}))$  which satisfies:
$$
\left| s _ {t _ {a} ^ {*}} - 1 \right| = \left| \frac {q _ {t _ {a} ^ {*}}}{r \left(p _ {t _ {a} ^ {*}}\right)} - 1 \right| = \mathcal {O} _ {\delta , \ell} (\eta^ {2}).
$$
By Lemma B.8, there exists a constant  $D > 0$  which depends on  $\delta, \ell$  such that if  $|s_t - 1| = \mathcal{O}_{\delta, \ell}(\eta^2)$ , then
$$
\left| s _ {t + 1} - 1 - h \left(p _ {t + 1}\right) \eta^ {2} \right| \leq \left(1 + \frac {2 p _ {t} r ^ {\prime} \left(p _ {t}\right)}{r \left(p _ {t}\right)}\right) \left| s _ {t} - 1 - h \left(p _ {t}\right) \eta^ {2} \right| + D \eta^ {4} p _ {t} ^ {2}. \tag {10}
$$
Hence, if  $|s_t - 1| = \mathcal{O}_{\delta,\ell}(\eta^2)$  and  $|s_t - 1 - h(p_t)\eta^2| \geq \left(-\frac{p_tr(p_t)}{r'(p_t)}\right)D\eta^4$ , then
$$
\left| s _ {t + 1} - 1 - h \left(p _ {t + 1}\right) \eta^ {2} \right| \leq \left(1 + \frac {p _ {t} r ^ {\prime} \left(p _ {t}\right)}{r \left(p _ {t}\right)}\right) \left| s _ {t} - 1 - h \left(p _ {t}\right) \eta^ {2} \right|. \tag {11}
$$
For any  $t \leq T$ , we have  $q_t < 1 - \frac{\delta}{2}$  so that if  $|s_t - 1| = \mathcal{O}_{\delta, \ell}(\eta^2)$ , then  $r(p_t) \leq (1 + \mathcal{O}_{\delta, \ell}(\eta^2)) q_t < 1 - \frac{\delta}{4}$  for small step size  $\eta$ . From Eq. (11) with  $t = t_a^*$ , we have either
$$
\left| s _ {t _ {a} ^ {*}} - 1 - h \left(p _ {t _ {a} ^ {*}}\right) \eta^ {2} \right| <   \left(- \frac {p _ {t _ {a} ^ {*}} r \left(p _ {t _ {a} ^ {*}}\right)}{r ^ {\prime} \left(p _ {t _ {a} ^ {*}}\right)}\right) D \eta^ {4},
$$
$$
\left| s _ {t _ {a} ^ {*} + 1} - 1 - h \left(p _ {t _ {a} ^ {*} + 1}\right) \eta^ {2} \right| \leq \left(1 + \frac {\hat {r} \left(1 - \frac {\delta}{4}\right) r ^ {\prime} \left(\hat {r} \left(1 - \frac {\delta}{4}\right)\right)}{\left(1 - \frac {\delta}{4}\right)}\right) \left| s _ {t _ {a} ^ {*}} - 1 - h \left(p _ {t _ {a} ^ {*}}\right) \eta^ {2} \right|,
$$
where we used Assumption 4.3 (ii) and  $|p_t| > \hat{r}(1 - \frac{\delta}{4})$ . In the later case,  $|s_{t_a^* + 1} - 1| = \mathcal{O}_{\delta, \ell}(\eta^2)$  continues to hold and we can again use Eq. (11) with  $t = t_a^* + 1$ . By repeating the analogous arguments, we can obtain the time step
$$
t _ {a} \leq t _ {a} ^ {*} + \frac {\log \left(- \frac {D \eta^ {4}}{r ^ {\prime \prime} (0) | s _ {t _ {a} ^ {*}} - 1 - h (p _ {t _ {a} ^ {*}}) \eta^ {2} |}\right)}{\log \left(1 + \frac {\hat {r} (1 - \frac {\delta}{4}) r ^ {\prime} (\hat {r} (1 - \frac {\delta}{4}))}{(1 - \frac {\delta}{4})}\right)} = \mathcal {O} _ {\delta , \ell} (\log (\eta^ {- 1})),
$$
which satisfies: either
$$
\left| s _ {t _ {a}} - 1 - h \left(p _ {t _ {a}}\right) \eta^ {2} \right| <   \left(- \frac {p _ {t _ {a}} r \left(p _ {t _ {a}}\right)}{r ^ {\prime} \left(p _ {t _ {a}}\right)}\right) D \eta^ {4},
$$
$$
| s _ {t _ {a}} - 1 - h (p _ {t _ {a}}) \eta^ {2} | \leq \left(- \frac {1}{r ^ {\prime \prime} (0)}\right) D \eta^ {4} \leq \left(- \frac {p _ {t _ {a}} r (p _ {t _ {a}})}{r ^ {\prime} (p _ {t _ {a}})}\right) D \eta^ {4} \leq \left(- \frac {4 r (4)}{r ^ {\prime} (4)}\right) D \eta^ {4},
$$
where we used  $|p_t| \leq 4$  from Lemma B.2 and  $-\frac{zr(z)}{r'(z)} \geq -\frac{1}{r''(0)}$  for any  $z$  by Assumption 4.3 (iii).
By Eq. (10), if  $|s_t - 1 - h(p_t)\eta^2| \leq \left(-\frac{4r(4)}{r'(4)}\right)D\eta^4$  is satisfied for any time step  $t$ , then
$$
\left| s _ {t + 1} - 1 - h \left(p _ {t + 1}\right) \eta^ {2} \right| \leq \left(1 + \frac {2 p _ {t} r ^ {\prime} \left(p _ {t}\right)}{r \left(p _ {t}\right)}\right) \left(- \frac {4 r (4)}{r ^ {\prime} (4)}\right) D \eta^ {4} + D \eta^ {4} p _ {t} ^ {2} \leq \left(- \frac {4 r (4)}{r ^ {\prime} (4)}\right) D \eta^ {4},
$$
by  $|p_t| \leq 4$  from Lemma B.2 and Assumption 4.3 (iii).
Hence, by induction, we have the desired bound as following: for any  $t \geq t_a$ ,
$$
\left| s _ {t} - 1 - h \left(p _ {t}\right) \eta^ {2} \right| \leq \left(- \frac {4 r (4)}{r ^ {\prime} (4)}\right) D \eta^ {4} = \mathcal {O} _ {\delta , \ell} \left(\eta^ {4}\right),
$$
B.3 Proof of Theorem 4.3
In this subsection, we prove Theorem 4.3. We start by proving Lemma B.9 which provides a useful property of  $h$  defined in Theorem 4.2.
Lemma B.9. Consider the function  $h$  defined in Theorem 4.2, given by
$$
h (p) := \left\{ \begin{array}{l l} - \frac {1}{2} \left(\frac {p r (p) ^ {3}}{r ^ {\prime} (p)} + p ^ {2} r (p) ^ {2}\right) & i f p \neq 0, a n d \\ - \frac {1}{2 r ^ {\prime \prime} (0)} & i f p = 0. \end{array} \right.
$$
Then,  $h$  is a positive, even, and bounded twice continuously differentiable function.
Proof. It is clear that  $h$  is even. We first prove that  $h$  is positive. For any  $p \neq 0$ , it holds that
$$
h (p) = - \frac {p r (p) ^ {3}}{2 r ^ {\prime} (p)} \left(1 + \frac {p r ^ {\prime} (p)}{r (p)}\right) > 0,
$$
since  $\frac{pr(p)}{r'(p)} < 0$  and  $1 + \frac{pr'(p)}{r(p)} = \frac{\ell''(p)}{r(p)} > 0$  by Assumption 4.2 and convexity of  $\ell$ . The function  $h$  is continuous since  $\lim_{p \to 0} h(p) = h(0)$ . Continuous function on a compact domain is bounded, so  $h$  is bounded on the closed interval  $[-1, 1]$ . We can rewrite  $h$  as
$$
h (p) = \frac {1}{2} p ^ {2} r (p) ^ {2} \left(- \frac {r (p)}{p r ^ {\prime} (p)} - 1\right).
$$
Note that  $p^2 r(p)^2 = \ell'(p)^2 \leq 1$ , and  $\left(-\frac{r(p)}{pr'(p)} - 1\right)$  is positive, decreasing function on  $p > 0$  by Assumption 4.3 (ii). Hence,  $h$  is bounded on  $[1, \infty)$ . Since  $h$  is even,  $h$  is bounded on  $(-\infty, 1]$ . Therefore,  $h$  is a bounded function on  $\mathbb{R}$ .
We finally prove that  $h$  is twice continuously differentiable. Since  $r$  is even and  $C^4$  on  $\mathbb{R}$ , we can check that
$$
h ^ {\prime} (p) := \left\{ \begin{array}{l l} - \frac {1}{2} \left[ \frac {r (p) ^ {3} (r ^ {\prime} (p) - p r ^ {\prime \prime} (p))}{r ^ {\prime} (p) ^ {2}} + p r (p) (5 r (p) + 2 p r ^ {\prime} (p)) \right] & \text {i f} p \neq 0, \text {a n d} \\ 0 & \text {i f} p = 0. \end{array} \right.
$$
Moreover, for any  $p \neq 0$ ,
$$
\begin{array}{l} h ^ {\prime \prime} (p) = - \frac {1}{2} \left(\frac {2 r (p) ^ {2} r ^ {\prime \prime} (p) \left(p r ^ {\prime \prime} (p) - r ^ {\prime} (p)\right)}{r ^ {\prime} (p) ^ {3}} - \frac {p r (p) ^ {3} r ^ {(3)} (p)}{r ^ {\prime} (p) ^ {2}} - \frac {3 p r (p) ^ {2} r ^ {\prime \prime} (p)}{r ^ {\prime} (p)}\right) \\ - 4 r (p) ^ {2} - 7 p r (p) r ^ {\prime} (p) - p ^ {2} \left(r (p) r ^ {\prime \prime} (p) + r ^ {\prime} (p) ^ {2}\right), \\ \end{array}
$$
$$
h ^ {\prime \prime} (0) = \frac {r ^ {(4)} (0)}{6 r ^ {\prime \prime} (0) ^ {2}} - \frac {5}{2}.
$$
Since  $\lim_{p\to 0}h''(p) = h''(0)$ , we can conclude that  $h$  is a twice continuously differentiable function.
We now give the proof of Theorem 4.3, restated below for the sake of readability.
Proof. We first prove that there exists a time step  $t_b \geq 0$  such that  $q_{t_b} > 1$ . Assume the contrary that  $q_t \leq 1$  for all  $t \geq 0$ . Let  $t_a$  be the time step obtained in Theorem 4.2. Then for any  $t \geq t_a$ , we have
$$
r (p _ {t}) = (1 - h (p _ {t}) \eta^ {2} + \mathcal {O} _ {\delta , \ell} (\eta^ {4})) q _ {t} \leq 1 - \frac {h (p _ {t}) \eta^ {2}}{2},
$$
for small step size  $\eta$ . The function  $g(p) := r(p) - 1 + \frac{h(p)\eta^2}{2}$  is even, continuous, and has the function value  $g(0) = \frac{\eta^2}{4|r''(0)|} > 0$ . Consequently, there exists a positive constant  $\epsilon > 0$  such that  $g(p) > 0$  for all  $p \in (-\epsilon, \epsilon)$ . Then, we have  $|p_t| \geq \epsilon$  for all  $t \geq t_a$ , since  $g(p_t) \leq 0$ . Moreover,  $s_t \geq \frac{3}{4}$  for any  $t \geq t_a$  by Theorem 4.2 for small step size  $\eta$ . This implies that for any  $t \geq t_a$ ,
$$
\frac {q _ {t}}{q _ {t + 1}} = 1 - \eta^ {2} p _ {t} ^ {2} r (p _ {t}) ^ {2} (2 s _ {t} - 1) \leq 1 - \frac {1}{2} \eta^ {2} \ell^ {\prime} (p _ {t}) ^ {2} \leq 1 - \frac {1}{2} \eta^ {2} \ell^ {\prime} (\epsilon) ^ {2},
$$
so  $q_{t}$  grows exponentially, which results in the existence of a time step  $t_b' \geq t_a$  such that  $q_{t_b'} > 1$ , a contradiction.
Therefore, there exists a time step  $t_b$  such that  $q_{t_b} \leq 1$  and  $q_t > 1$  for any  $t > t_b$ , i.e.,  $q_t$  jumps across the value 1. This holds since the sequence  $(q_t)$  is monotonically increasing. For any  $t \leq t_b$ , we have  $q_{t + 1} \leq q_t + \mathcal{O}(\eta^2)$  by Lemma B.2, and this implies that  $t_b \geq \Omega((1 - q_0)\eta^{-2})$ , as desired.
Lastly, we prove the convergence of GD iterates  $(p_t,q_t)$ . Let  $t > t_b$  be given. Then,  $q_{t}\geq q_{t_{b} + 1} > 1$  and it holds that
$$
\left| \frac {p _ {t + 1}}{p _ {t}} \right| = \frac {2 r (p _ {t})}{q _ {t}} - 1 - \eta^ {2} p _ {t} ^ {2} r (p _ {t}) ^ {2} \leq \frac {2}{q _ {t _ {b} + 1}} - 1 <   1.
$$
Hence,  $|p_t|$  is exponentially decreasing for  $t > t_b$ . Therefore,  $p_t$  converges to 0 as  $t \to \infty$ . Since the sequence  $(q_t)_{t=0}^{\infty}$  is monotonically increasing and bounded (due to Theorem 4.2), it converges. Suppose that  $(p_t, q_t)$  converges to the point  $(0, q^*)$ . By Theorem 4.2, we can conclude that
$$
\left| q ^ {*} - 1 + \frac {\eta^ {2}}{2 r ^ {\prime \prime} (0)} \right| = \mathcal {O} _ {\delta , \ell} (\eta^ {4}),
$$
B.4 Proof of Theorem 4.4
In this subsection, we prove Theorem 4.4. We first prove a useful lemma which bounds the Hessian of the function  $(\pmb{U}, \pmb{v}) \mapsto \pmb{v}^{\top} \pmb{U} \pmb{x}$ , stated below.
Lemma B.10. For any  $\Theta = (U, v)$  with  $U \in \mathbb{R}^{m \times d}$ ,  $v \in \mathbb{R}^m$ , and  $x \in \mathbb{R}^d$  with  $\|x\|_2 = 1$ , the following equality holds:
$$
\left\| \nabla_ {(\boldsymbol {U}, \boldsymbol {v})} ^ {2} (\boldsymbol {v} ^ {\top} \boldsymbol {U} \boldsymbol {x}) \right\| _ {2} \leq 1.
$$
Moreover, if  $\lambda$  is an eigenvalue of  $\nabla_{(U,v)}^2 (\pmb{v}^\top \pmb{U}\pmb{x})$ , then  $-\lambda$  is also an eigenvalue of  $\nabla_{(U,v)}^2 (\pmb{v}^\top \pmb{U}\pmb{x})$ .
Proof. We first define the notations. We use the operator  $\otimes$  to represent tensor product, or Kronecker product between matrices. For example, for any given two matrices  $A = (a_{ij})\in \mathbb{R}^{m\times n}$  and  $B$ , we define  $A\otimes B$  by
$$
A \otimes B = \left( \begin{array}{c c c} a _ {1 1} B & \ldots & a _ {1 n} B \\ \vdots & \ddots & \vdots \\ a _ {m 1} B & \ldots & a _ {m n} B \end{array} \right).
$$
We use  $\mathbf{0}_{m\times n}$  to denote a  $m$  by  $n$  matrix with all entries filled with zero, and  $\pmb{I}_n$  denotes  $n$  by  $n$  identity matrix. Now we provide the proof of the original problem.
Let  $\pmb{U} = (u_{ij})\in \mathbb{R}^{m\times d}$ ,  $\pmb{v} = (v_{i})\in \mathbb{R}^{m}$ , and  $\pmb{x} = (x_{j})\in \mathbb{R}^{d}$  be given. Then,
$$
\boldsymbol {v} ^ {\top} \boldsymbol {U} \boldsymbol {x} = \sum_ {i, j} v _ {i} U _ {i j} x _ {j}.
$$
We vectorize the parameter  $\Theta = (U, v)$  by  $(v_{1}, \ldots, v_{m}, U_{11}, \ldots, U_{m1}, \ldots, U_{1d}, \ldots, U_{md}) \in \mathbb{R}^{m + md}$ . Then, we can represent the Hessian as
$$
\nabla_ {(\boldsymbol {U}, \boldsymbol {v})} ^ {2} (\boldsymbol {v} ^ {\top} \boldsymbol {U} \boldsymbol {x}) = \left( \begin{array}{c c} 0 & \boldsymbol {x} ^ {\top} \\ \hline \boldsymbol {x} & \boldsymbol {0} _ {d \times d} \end{array} \right) \otimes \boldsymbol {I} _ {m}. \tag {12}
$$
For any given  $c\in \mathbb{R}$  and  $\pmb {y}\in \mathbb{R}^d$  with  $c^2 +\| \pmb {y}\| _2^2 = 1$  , we have
$$
\left\| \left( \begin{array}{c c} 0 & \boldsymbol {x} ^ {\top} \\ \hline \boldsymbol {x} & \mathbf {0} _ {d \times d} \end{array} \right) \left(\frac {c}{\boldsymbol {y}}\right) \right\| _ {2} = \left\| \left( \begin{array}{c} \boldsymbol {x} ^ {\top} \boldsymbol {y} \\ \hline c \boldsymbol {x} \end{array} \right) \right\| _ {2} \leq \| \boldsymbol {x} \| _ {2} = 1.
$$
Hence, by definition of matrix operator norm, we have
$$
\left\| \left( \begin{array}{c c} 0 & \boldsymbol {x} ^ {\top} \\ \hline \boldsymbol {x} & \boldsymbol {0} _ {d \times d} \end{array} \right) \right\| _ {2} \leq 1.
$$
$$
\left\| \nabla_ {(\boldsymbol {U}, \boldsymbol {v})} ^ {2} (\boldsymbol {v} ^ {\top} \boldsymbol {U} \boldsymbol {x}) \right\| _ {2} = \left\| \left( \begin{array}{c c} 0 & \boldsymbol {x} ^ {\top} \\ \hline \boldsymbol {x} & \mathbf {0} _ {d \times d} \end{array} \right) \otimes \boldsymbol {I} _ {m} \right\| _ {2} = \left\| \left( \begin{array}{c c} 0 & \boldsymbol {x} ^ {\top} \\ \hline \boldsymbol {x} & \mathbf {0} _ {d \times d} \end{array} \right) \right\| _ {2} \leq 1.
$$
Now suppose that  $\lambda$  is an eigenvalue of  $\nabla_{(U,v)}^2 (v^\top Ux)$ . We note that for any given matrices  $A$  and  $B$ , if  $\lambda_{a}$  is an eigenvalue of  $A$  with the corresponding eigenvector  $\pmb{u}_{a}$  and  $\lambda_{b}$  is an eigenvalue of  $B$  with the corresponding eigenvector  $\pmb{u}_{b}$ , then  $\lambda_{a}\lambda_{b}$  is an eigenvalue of  $A\otimes B$  with the corresponding eigenvector  $\pmb{u}_{a}\otimes \pmb{u}_{b}$ . Moreover, any eigenvalue of  $A\otimes B$  arises as such a product of eigenvalues of  $A$  and  $B$ . Hence, using Eq. (12), we have
$$
\lambda \text {i s a n e i g e n v a l u e o f t h e m a t r i x} \left( \begin{array}{c c} 0 & \boldsymbol {x} ^ {\top} \\ \hline \boldsymbol {x} & \boldsymbol {0} _ {d \times d} \end{array} \right).
$$
We denote the corresponding eigenvector by  $(c, \pmb{y}^{\top})^{\top}$  where  $c \in \mathbb{R}$  and  $\pmb{y} \in \mathbb{R}^{d}$ , i.e., it holds that
$$
\left( \begin{array}{c c} 0 & \boldsymbol {x} ^ {\top} \\ \hline \boldsymbol {x} & \boldsymbol {0} _ {d \times d} \end{array} \right) \left( \begin{array}{c} c \\ \hline \boldsymbol {y} \end{array} \right) = \left( \begin{array}{c} \boldsymbol {x} ^ {\top} \boldsymbol {y} \\ \hline c \boldsymbol {x} \end{array} \right) = \lambda \left( \begin{array}{c} c \\ \hline \boldsymbol {y} \end{array} \right).
$$
$$
\left( \begin{array}{c c} 0 & \boldsymbol {x} ^ {\top} \\ \hline \boldsymbol {x} & \mathbf {0} _ {d \times d} \end{array} \right) \left( \begin{array}{c} - c \\ \hline \boldsymbol {y} \end{array} \right) = \left( \begin{array}{c} \boldsymbol {x} ^ {\top} \boldsymbol {y} \\ \hline - c \boldsymbol {x} \end{array} \right) = - \lambda \left( \begin{array}{c} - c \\ \hline \boldsymbol {y} \end{array} \right),
$$
and this implies that
$$
- \lambda \text {i s a n e i g e n v a l u e o f t h e m a t r i x} \left( \begin{array}{c c} 0 & \boldsymbol {x} ^ {\top} \\ \hline \boldsymbol {x} & \boldsymbol {0} _ {d \times d} \end{array} \right).
$$
Therefore, by Eq. (12),  $-\lambda$  is an eigenvalue of  $\nabla_{(U,v)}^2 (\pmb{v}^\top \pmb{U}\pmb{x})$ .
Using Lemma B.10, we prove an important bound on the sharpness value provided by the Proposition B.11 stated below.
Proposition B.11. For any  $\Theta = (U, \mathbf{v})$  with  $U \in \mathbb{R}^{m \times d}$ ,  $\mathbf{v} \in \mathbb{R}^m$ , and  $\mathbf{x} \in \mathbb{R}^d$  with  $\| \mathbf{x} \|_2 = 1$ , the following bound holds:
$$
\left| \lambda_ {\max } (\boldsymbol {\Theta}) - \ell^ {\prime \prime} (\boldsymbol {v} ^ {\top} \boldsymbol {U} \boldsymbol {x}) \left(\| \boldsymbol {U} \boldsymbol {x} \| _ {2} ^ {2} + \| \boldsymbol {v} \| _ {2} ^ {2}\right) \right| \leq 1
$$
Proof. The loss Hessian at  $\Theta = (U, v)$  can be characterized as:
$$
\nabla_ {\boldsymbol {\Theta}} ^ {2} \mathcal {L} (\boldsymbol {\Theta}) = \ell^ {\prime \prime} (\boldsymbol {v} ^ {\top} \boldsymbol {U} \boldsymbol {x}) \left(\nabla_ {\boldsymbol {\Theta}} (\boldsymbol {v} ^ {\top} \boldsymbol {U} \boldsymbol {x})\right) ^ {\otimes 2} + \ell^ {\prime} (\boldsymbol {v} ^ {\top} \boldsymbol {U} \boldsymbol {x}) \nabla_ {\boldsymbol {\Theta}} ^ {2} (\boldsymbol {v} ^ {\top} \boldsymbol {U} \boldsymbol {x}). \tag {13}
$$
We first prove that  $\lambda_{\max}(\nabla_{\Theta}^{2}\mathcal{L}(\Theta)) = \left\| \nabla_{\Theta}^{2}\mathcal{L}(\Theta)\right\|_{2}$ . Note that the largest absolute value of the eigenvalue of a symmetric matrix equals to its spectral norm. Hence,  $\left\| \nabla_{\Theta}^{2}\mathcal{L}(\Theta)\right\|_{2} = \max \{\lambda_{\max}(\nabla_{\Theta}^{2}\mathcal{L}(\Theta)), - \lambda_{\min}(\nabla_{\Theta}^{2}\mathcal{L}(\Theta))\}$ , so it suffices to prove that  $\lambda_{\max}(\nabla_{\Theta}^{2}\mathcal{L}(\Theta)) \geq -\lambda_{\min}(\nabla_{\Theta}^{2}\mathcal{L}(\Theta))$ . Let  $\boldsymbol{w}$  denote the eigenvector of  $\nabla_{\Theta}^{2}\mathcal{L}(\Theta)$  corresponding to the smallest eigenvalue  $\lambda_{\min}(\nabla_{\Theta}^{2}\mathcal{L}(\Theta))$  with  $\| \boldsymbol{w}\|_2 = 1$ . Then, using Eq. (13), we have
$$
\begin{array}{l} \lambda_ {\min } (\nabla_ {\Theta} ^ {2} \mathcal {L} (\Theta)) = \boldsymbol {w} ^ {\top} \nabla_ {\Theta} ^ {2} \mathcal {L} (\Theta) \boldsymbol {w} = \ell^ {\prime \prime} (\boldsymbol {v} ^ {\top} \boldsymbol {U x}) \boldsymbol {w} ^ {\top} \left(\nabla_ {\Theta} (\boldsymbol {v} ^ {\top} \boldsymbol {U x})\right) ^ {\otimes 2} \boldsymbol {w} + \ell^ {\prime} (\boldsymbol {v} ^ {\top} \boldsymbol {U x}) \boldsymbol {w} ^ {\top} \nabla_ {\Theta} ^ {2} (\boldsymbol {v} ^ {\top} \boldsymbol {U x}) \boldsymbol {w} \\ \geq \ell^ {\prime} (\boldsymbol {v} ^ {\top} \boldsymbol {U} \boldsymbol {x}) \boldsymbol {w} ^ {\top} \nabla_ {\boldsymbol {\Theta}} ^ {2} (\boldsymbol {v} ^ {\top} \boldsymbol {U} \boldsymbol {x}) \boldsymbol {w} \\ \geq - \left| \ell^ {\prime} \left(\boldsymbol {v} ^ {\top} \boldsymbol {U} \boldsymbol {x}\right) \right| \| \nabla_ {\boldsymbol {\Theta}} ^ {2} \left(\boldsymbol {v} ^ {\top} \boldsymbol {U} \boldsymbol {x}\right) \| _ {2}, \\ \end{array}
$$
where we used Lemma B.10 to obtain the last inequality. Note that the matrix  $\ell''(\pmb{v}^\top \pmb{U}\pmb{x})\left(\nabla_\Theta (\pmb{v}^\top \pmb{U}\pmb{x})\right)^{\otimes 2}$  is PSD, so that  $\lambda_{\max}(\nabla_\Theta^2\mathcal{L}(\Theta)) \geq \lambda_{\max}(\ell'(\pmb{v}^\top \pmb{U}\pmb{x})\nabla_\Theta^2(\pmb{v}^\top \pmb{U}\pmb{x})) = |\ell'(\pmb{v}^\top \pmb{U}\pmb{x})||\nabla_\Theta^2(\pmb{v}^\top \pmb{U}\pmb{x})|_2 \geq -\lambda_{\min}(\nabla_\Theta^2\mathcal{L}(\Theta))$ . Therefore,  $\lambda_{\max}(\nabla_\Theta^2\mathcal{L}(\Theta)) = \left\| \nabla_\Theta^2\mathcal{L}(\Theta)\right\|_2$ .
Now, we have the following triangle inequality:
$$
\begin{array}{l} \left| \lambda_ {\max } (\boldsymbol {\Theta}) - \ell^ {\prime \prime} (\boldsymbol {v} ^ {\top} \boldsymbol {U} \boldsymbol {x}) \left(\| \boldsymbol {U} \boldsymbol {x} \| _ {2} ^ {2} + \| \boldsymbol {v} \| _ {2} ^ {2}\right) \right| = \left| \left\| \nabla_ {\boldsymbol {\Theta}} ^ {2} \mathcal {L} (\boldsymbol {\Theta}) \right\| _ {2} - \left\| \ell^ {\prime \prime} (\boldsymbol {v} ^ {\top} \boldsymbol {U} \boldsymbol {x}) \left(\nabla_ {\boldsymbol {\Theta}} (\boldsymbol {v} ^ {\top} \boldsymbol {U} \boldsymbol {x})\right) ^ {\otimes 2} \right\| _ {2} \right| \\ \leq \left\| \ell^ {\prime} (\boldsymbol {v} ^ {\top} \boldsymbol {U} \boldsymbol {x}) \nabla_ {\boldsymbol {\Theta}} ^ {2} (\boldsymbol {v} ^ {\top} \boldsymbol {U} \boldsymbol {x}) \right\| _ {2} \\ = \left| \ell^ {\prime} (\boldsymbol {v} ^ {\top} \boldsymbol {U} \boldsymbol {x}) \right| \left\| \nabla_ {(\boldsymbol {U}, \boldsymbol {v})} ^ {2} (\boldsymbol {v} ^ {\top} \boldsymbol {U} \boldsymbol {x}) \right\| _ {2} \\ \leq 1, \\ \end{array}
$$
where the last inequality holds by Lemma B.10 and 1-Lipschitzness of  $\ell$ .
We now give the proof of Theorem 4.4, restated below for the sake of readability.