SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\lambda \big (\tilde {y} ^ {+}, \Phi \big) = \nabla \Phi \big (y + h \big) ^ {\top} \nabla^ {- 2} \Phi \big (y + h \big) \nabla \Phi \big (y + h \big).
$$
Now for the vector  $z$  defined below, we define the function  $F$  as
$$
z \triangleq \nabla^ {- 2} \Phi (y + h) \nabla \Phi (y + h) \quad \text {a n d} \quad F (y) := \nabla \Phi (y) ^ {\top} z. \tag {40}
$$
The partial derivative of  $F$  in direction  $h$  is given by
$$
\begin{array}{l} D F (y) [ h ] = - h ^ {\top} \nabla^ {2} \Phi (y) z, \\ = - \nabla \Phi (y) ^ {\top} H ^ {- 1} \nabla^ {2} \Phi (y) z, \\ = - \nabla \Phi (y) ^ {\top} H ^ {- 1 / 2} H ^ {- 1 / 2} \nabla^ {2} \Phi (y) H ^ {- 1 / 2} H ^ {1 / 2} z, \\ = - \nabla \Phi (y) ^ {\top} H ^ {- 1 / 2} \left(H ^ {- 1 / 2} \nabla^ {2} \Phi (y) H ^ {- 1 / 2} - I\right) H ^ {1 / 2} z - \nabla \Phi (y) ^ {\top} z. \tag {41} \\ \end{array}
$$
Now, by (21), we have
$$
\left\| H ^ {- 1 / 2} \nabla^ {2} \Phi (y) H ^ {- 1 / 2} - I \right\| \leq \frac {\alpha}{1 - \alpha}.
$$
Thus, the first term on the right-hand side of (41) can be bounded as
$$
\begin{array}{l} \nabla \Phi (y) ^ {\top} H ^ {- 1 / 2} \left(H ^ {- 1 / 2} \nabla^ {2} \Phi (y) H ^ {- 1 / 2} - I\right) H ^ {1 / 2} z \\ \leq \frac {\alpha}{1 - \alpha} \| \nabla \Phi (y) ^ {\top} H ^ {- 1 / 2} \| \| H ^ {1 / 2} z \|, \\ = \frac {\alpha}{1 - \alpha} \| \nabla \Phi (y) \| _ {H ^ {- 1}} \| z \| _ {H}, \\ \leq \frac {\alpha (1 + \alpha)}{(1 - \alpha) ^ {2}} \| \nabla \Phi (y) \| _ {\nabla^ {- 2} \Phi (y)} \| z \| _ {\nabla^ {2} \Phi (y)}. \\ = \frac {\alpha (1 + \alpha)}{(1 - \alpha) ^ {2}} \lambda (y, \Phi) \cdot \| z \| _ {\nabla^ {2} \Phi (y)}. \\ \end{array}
$$
Plugging this into (41) and using the definition  $z$  in (40), we obtain
$$
\left| D F (y) [ h ] + F (y) \right| \leq \frac {\alpha (1 + \alpha)}{(1 - \alpha) ^ {2}} \lambda (y, \Phi) \cdot \| z \| _ {\nabla^ {2} \Phi (y)}. \tag {42}
$$
Now, let  $\pmb{y}(s) \coloneqq y + sh$  and  $F \circ \pmb{y}(s) \coloneqq F(y(s))$ . With this, we have
$$
(F \circ \boldsymbol {y}) ^ {\prime} (s) - (F \circ \boldsymbol {y}) ^ {\prime} (0) = h ^ {\top} (\nabla^ {2} \Phi (y (s)) - \nabla^ {2} \Phi (y (0))) z. \tag {43}
$$
On the other hand, by Lemma 8 and our assumption on  $\lambda(y, \Phi)$ , we have
$$
\begin{array}{l} \left\| \boldsymbol {y} (s) - y \right\| _ {\nabla^ {2} \Phi (y)} = s \| h \| _ {\nabla^ {2} \Phi (y)} \leq \frac {s}{1 - \alpha} \| \nabla \Phi (y) \| _ {\nabla^ {- 2} \Phi (y)} \leq \frac {1}{1 - \alpha} \lambda (y, \Phi), (44) \\ <   \frac {1}{3 0 M _ {\Phi}}. (45) \\ \end{array}
$$
Thus, by Lemma 1, we have
$$
\left(1 - M _ {\Phi} \| \boldsymbol {y} (s) - y \| _ {\nabla^ {2} \Phi (y)} ^ {2}\right) ^ {2} \nabla^ {2} \Phi (y) \preccurlyeq \nabla^ {2} \Phi (y (s)) \preccurlyeq \frac {1}{\left(1 - M _ {\Phi} \| \boldsymbol {y} (s) - y \| _ {\nabla^ {2} \Phi (y)}\right) ^ {2}} \nabla^ {2} \Phi (y).
$$
This, together with (45) also implies that
$$
\left(1 - 3 M _ {\Phi} \| \boldsymbol {y} (s) - y \| _ {\nabla^ {2} \Phi (y)}\right) \nabla^ {2} \Phi (y) \leqslant \nabla^ {2} \Phi (y (s)) \leqslant \left(1 + 3 M _ {\Phi} \| \boldsymbol {y} (s) - y \| _ {\nabla^ {2} \Phi (y)}\right) \nabla^ {2} \Phi (y).
$$
After rearranging, this becomes
$$
- 3 M _ {\Phi} \| \boldsymbol {y} (s) - y \| _ {\nabla^ {2} \Phi (y)} \nabla^ {2} \Phi (y) \leqslant \nabla^ {2} \Phi (y (s)) - \nabla^ {2} \Phi (y) \leqslant 3 M _ {\Phi} \| \boldsymbol {y} (s) - y \| _ {\nabla^ {2} \Phi (y)} \nabla^ {2} \Phi (y).
$$
Combining this with (44) and the fact that  $c \leq \frac{1}{4}$  gives
$$
- 4 M _ {\Phi} \lambda (y, \Phi) \nabla^ {2} \Phi (y) \leqslant \nabla^ {2} \Phi (y (s)) - \nabla^ {2} \Phi (y) \leqslant 4 M _ {\Phi} \lambda (y, \Phi) \nabla^ {2} \Phi (y). \tag {46}
$$
Finally, by Lemma 9 and (46), we obtain the following bound on the right-hand side of (43):
$$
(F \circ \boldsymbol {y}) ^ {\prime} (s) - (F \circ \boldsymbol {y}) ^ {\prime} (0) \leq 6 M _ {\Phi} \lambda (y, \Phi) \| h \| _ {\nabla^ {2} \Phi (y)} \| z \| _ {\nabla^ {2} \Phi (y)}.
$$
Integrating this over  $s$  gives
$$
\begin{array}{l} \nabla \Phi (y + h) ^ {\top} z = (F \circ \mathbf {y}) (1) \\ = (F \circ \boldsymbol {y}) (0) + (F \circ \boldsymbol {y}) ^ {\prime} (0) + \int_ {0} ^ {1} ((F \circ \boldsymbol {y}) ^ {\prime} (s) - (F \circ \boldsymbol {y}) ^ {\prime} (0)) d s \\ \leq \nabla \Phi (y) ^ {\top} z + D F (y) [ z ] + 6 M _ {\Phi} \lambda (y, \Phi) \| h \| _ {\nabla^ {2} \Phi (y)} \| z \| _ {\nabla^ {2} \Phi (y)}, \\ \leq \nabla \Phi (y) ^ {\top} z + D F (y) [ z ] + \frac {6 M _ {\Phi}}{1 - \alpha} \lambda (y, \Phi) ^ {2} \| z \| _ {\nabla^ {2} \Phi (y)}, \tag {47} \\ \end{array}
$$
where the last inequality follows by (44). Now, note that from (46) (with  $s = 1$ ) and the assumption that  $\lambda(w, \Phi) \leq 1 / (40M_{\Phi})$ , we have
$$
\nabla^ {2} \Phi (y) \leqslant \frac {1 0}{9} \nabla^ {2} \Phi (y + h).
$$
$$
\left\| z \right\| _ {\nabla^ {2} \Phi (y)} \leq \frac {1 0}{9} \left\| z \right\| _ {\nabla^ {2} \Phi (y + h)} = \frac {1 0}{9} \lambda (y + h, \Phi).
$$
Plugging this into (47) and using (42), we get
$$
\begin{array}{l} \nabla \Phi (y + h) ^ {\top} z \leq \left| \nabla \Phi (y) ^ {\top} z + D F (y) [ h ] \right| + \frac {2 0 M _ {\Phi}}{3 (1 - \alpha)} \lambda (y, \Phi) ^ {2} \lambda (y + h, \Phi) \\ \leq \frac {\alpha (1 + \alpha)}{(1 - \alpha) ^ {2}} \lambda (y, \Phi) \| z \| _ {\nabla^ {2} \Phi (y)} + \frac {2 0 M _ {\Phi}}{3 (1 - \alpha)} \lambda (y, \Phi) ^ {2} \lambda (y + h, \Phi), \\ \leq \frac {1 0 \alpha (1 + \alpha)}{9 (1 - \alpha) ^ {2}} \lambda (y, \Phi) \lambda (y + h, \Phi) + \frac {2 0 M _ {\Phi}}{3 (1 - \alpha)} \lambda (y, \Phi) ^ {2} \lambda (y + h, \Phi). \\ \end{array}
$$
Now, from the definition of  $z$ , we have
$$
\nabla \Phi (y + h) ^ {\top} z = \lambda (y + h, \Phi) ^ {2}.
$$
Thus, since  $\alpha < 1 / 4$ , we finally get
$$
\lambda (y + h, \Phi) \leq 9 M _ {\Phi} \lambda (y, \Phi) ^ {2} + 2. 5 \alpha \lambda (y, \Phi).
$$
This proves the first part of the claim, i.e. (43).
Bounding the Newton decrement at  $y^{+}$ . We now bound the Newton decrement at  $y^{+} = y - H^{-1}\widehat{\nabla}_{y}$  in terms of that of  $\tilde{y}^{+} = y + h$ . Note that
$$
\tilde {y} ^ {+} - y ^ {+} = H ^ {- 1} (\nabla \Phi (y) - \widehat {\nabla} _ {y}).
$$
On the other hand, from (64) and (45), we have
$$
\nabla^ {2} \Phi (\tilde {y} ^ {+}) \leqslant (1 + 3 M _ {\Phi} \| h \|) \nabla^ {2} \Phi (y) \leqslant \frac {7}{4} \nabla^ {2} \Phi (y) \leqslant \frac {7}{4} (1 + \alpha) H, \tag {48}
$$
$$
\left\| \nabla^ {2} \Phi \left(\tilde {y} ^ {+}\right) ^ {1 / 2} H ^ {- 1} \nabla^ {2} \Phi \left(\tilde {y} ^ {+}\right) ^ {1 / 2} \right\| \leq \frac {7}{4} (1 + \alpha).
$$
$$
\begin{array}{l} \left\| \tilde {y} ^ {+} - y ^ {+} \right\| _ {\nabla^ {2} \Phi (\tilde {y} ^ {+})} ^ {2} \\ = \left(\nabla \Phi (y) - \widehat {\nabla} _ {y}\right) ^ {\top} H ^ {- 1} \nabla^ {2} \Phi (\tilde {y} ^ {+}) H ^ {- 1} \left(\nabla \Phi (y) - \widehat {\nabla} _ {y}\right), \\ = \big (\nabla \Phi (\widetilde {y} ^ {+}) - \widehat {\nabla} _ {y} \big) ^ {\top} \nabla^ {- 1 / 2} \Phi (\widetilde {y} ^ {+}) \Big (\nabla^ {2} \Phi (\widetilde {y} ^ {+}) ^ {1 / 2} H ^ {- 1} \nabla^ {2} \Phi (\widetilde {y} ^ {+}) ^ {1 / 2} \Big) ^ {2} \nabla^ {- 1 / 2} \Phi (\widetilde {y} ^ {+}) \big (\nabla \Phi (y) - \widehat {\nabla} _ {y} \big), \\ \leq \frac {4 9}{1 6} (1 + \alpha) ^ {2} \big (\nabla \Phi (y) - \widehat {\nabla} _ {y} \big) ^ {\top} \nabla^ {- 2} \Phi (\widetilde {y} ^ {+}) \big (\nabla \Phi (y) - \widehat {\nabla} _ {y} \big), \\ = \frac {4 9}{1 6} (1 + \alpha) ^ {2} \| \nabla \Phi (y) - \widehat {\nabla} _ {y} \| _ {\nabla^ {- 2} \Phi (\tilde {y} ^ {+})}. \\ \end{array}
$$
Combining this with (48) and our assumption on  $\widehat{\nabla}_y$  from (20) implies
$$
\left\| \tilde {y} ^ {+} - y ^ {+} \right\| _ {\nabla^ {2} \Phi (\tilde {y} ^ {+})} \leq \frac {7}{2} (1 + \alpha) \| \nabla \Phi (y) - \widehat {\nabla} _ {y} \| _ {\nabla^ {- 2} \Phi (y)} \leq 5 (1 + \alpha) \varepsilon . \tag {49}
$$
$$
\left(\left(1 - 5 (1 + \alpha) \varepsilon M _ {\Phi}\right) ^ {2} - 1\right) \nabla^ {2} \Phi \left(y ^ {+}\right) \preccurlyeq \nabla^ {2} \Phi \left(\tilde {y} ^ {+}\right) - \nabla^ {2} \Phi \left(y ^ {+}\right) \leq \left(\frac {1}{(1 - 5 (1 + \alpha) \varepsilon M _ {\Phi}) ^ {2}} - 1\right) \nabla^ {2} \Phi \left(y ^ {+}\right).
$$
Since  $\varepsilon < 1 / (40M_{\Phi})$  we get
$$
- 2 0 (1 + \alpha) \varepsilon \nabla^ {2} \Phi (y ^ {+}) \preccurlyeq \nabla^ {2} \Phi (\tilde {y} ^ {+}) - \nabla^ {2} \Phi (y ^ {+}) \preccurlyeq 2 0 (1 + \alpha) \varepsilon \nabla^ {2} \Phi (y ^ {+}). \tag {50}
$$
Now, by Lemma 4 instantiated with  $x = \tilde{y}^{+}$  and  $w = y^{+}$ , we have
$$
\begin{array}{l} \sqrt {\left(\nabla \Phi (\tilde {y} ^ {+}) - \nabla \Phi (y ^ {+})\right) ^ {\top} \nabla^ {- 2} \Phi (\tilde {y} ^ {+}) \left(\nabla \Phi (\tilde {y} ^ {+}) - \nabla \Phi (y ^ {+})\right)} \leq \frac {\| y ^ {+} - \tilde {y} ^ {+} \| _ {\nabla^ {2} \Phi (\tilde {y} ^ {+})}}{\left(1 - M _ {\Phi} \| y ^ {+} - \tilde {y} ^ {+} \| _ {\nabla^ {2} \Phi (\tilde {y} ^ {+})}\right)}, \\ \leq 1 0 (1 + \alpha) \varepsilon , \tag {51} \\ \end{array}
$$
where in the last inequality we used (49) and the fact that  $\varepsilon \leq 1 / (40M_{\Phi})$
Using the triangle inequality, we can bound the Newton decrement at  $y^{+}$  as
$$
\begin{array}{l} \lambda (y ^ {+}, \Phi) \leq \sqrt {\left(\nabla \Phi (\tilde {y} ^ {+}) - \nabla \Phi (y ^ {+})\right) ^ {\top} \nabla^ {- 2} \Phi (y ^ {+}) \left(\nabla \Phi (\tilde {y} ^ {+}) - \nabla \Phi (y ^ {+})\right)} \\ + \sqrt {\nabla \Phi (\tilde {y} ^ {+}) ^ {\top} \nabla^ {- 2} \Phi (y ^ {+}) \nabla \Phi (\tilde {y} ^ {+})}, \\ \leq 2 \sqrt {\left(\nabla \Phi (\tilde {y} ^ {+}) - \nabla \Phi (y ^ {+})\right) ^ {\top} \nabla^ {- 2} \Phi (\tilde {y} ^ {+}) \left(\nabla \Phi (\tilde {y} ^ {+}) - \nabla \Phi (y ^ {+})\right)} \\ + \sqrt {\nabla \Phi (\tilde {y} ^ {+}) ^ {\top} \nabla^ {- 2} \Phi (y ^ {+}) \nabla \Phi (\tilde {y} ^ {+})}, \quad \text {(b y (5 0) a n d} \varepsilon \leq 1 / (4 0 M _ {\Phi})) \\ \leq 2 0 (1 + \alpha) \varepsilon + (1 + 2 0 (1 + \alpha) \varepsilon) \cdot \lambda (\tilde {y} ^ {+}, \Phi), \\ \end{array}
$$
where the last inequality follows by (50) and (51). This completes the proof.
D.2 Proof of Lemma 6
Proof. By definition of  $(w_{t}^{m})$  in Algorithm 1, we have  $w_{t}^{1} = w_{t - 1}$  and  $w_{t} = w_{t}^{m_{\mathrm{Newton}}}$ . We show properties (23), (25), (26), and 27 using induction over  $m = 1,\ldots ,m_{\mathrm{Newton}}$ .
Base case. We start with the base case;  $m = 1$ . Note that from the assumption in (22) and definition of  $w_{t}^{1}$ , we have
$$
\left\| w _ {t} ^ {1} - u _ {t - 1} \right\| _ {\nabla^ {2} \Phi \left(u _ {t - 1}\right)} \leq 1 / \left(4 0 M _ {\phi}\right). \tag {52}
$$
Now, by definition of the Oracle  $\mathcal{O}_{\alpha}^{\mathrm{hess}}$  and the fact that  $H_{t - 1} = \mathcal{O}_{\alpha}^{\mathrm{hess}}(u_{t - 1})$  (see Algorithm 1) with  $\alpha = 0.001$ , we have
$$
(1 - 0. 0 0 1) \nabla^ {2} \Phi (u _ {t - 1}) \leqslant H _ {t - 1} \leqslant (1 + 0. 0 0 1) \nabla^ {2} \Phi (u _ {t - 1}). \tag {53}
$$
Combining this with (52) implies property (25) for the base case. Furthermore, since  $w_{t}^{1} = w_{t - 1}$  (by definition), (27) follows trivially for the base case.
Now, using that  $\Phi_t(w) = \Phi_{t-1}(w) + \eta g_{t-1}^\top w$ , we have
$$
\begin{array}{l} \lambda \left(w _ {t} ^ {1}, \Phi_ {t}\right) ^ {2} = \lambda \left(w _ {t - 1}, \Phi_ {t}\right) ^ {2} \\ = \left(\nabla \Phi_ {t - 1} \left(w _ {t - 1}\right) + \eta g _ {t - 1}\right) ^ {\top} \nabla^ {- 2} \Phi \left(w _ {t - 1}\right) \left(\nabla \Phi_ {t - 1} \left(w _ {t - 1}\right) + \eta g _ {t - 1}\right) \\ \leq 2 \nabla \Phi_ {t - 1} (w _ {t - 1}) ^ {\top} \nabla^ {- 2} \Phi (w _ {t - 1}) \nabla \Phi_ {t - 1} (w _ {t - 1}) + 2 \eta^ {2} g _ {t - 1} ^ {\top} \nabla^ {- 2} \Phi (w _ {t - 1}) g _ {t - 1} \\ = 2 \lambda \left(w _ {t - 1}, \Phi_ {t - 1}\right) ^ {2} + 2 \eta^ {2} g _ {t - 1} ^ {\top} \nabla^ {- 2} \Phi \left(w _ {t - 1}\right) g _ {t - 1} (54) \\ \leq 2 \alpha^ {2} + 2 \eta^ {2} b ^ {2} \leq 1 / \left(2 5 0 0 M _ {\Phi} ^ {2}\right), (55) \\ \end{array}
$$
where the last inequality follows by (22) and the fact that  $\| g_{t - 1}\|_{\nabla^{-2}\Phi (w_{t - 1})}\leq b$ . This shows property (23) for the base case. Thus, by Lemma 1, we have, for  $w_{t}^{\star}\in \mathrm{argmin}_{w\in \mathcal{K}}\Phi_{t}(w)$ ,
$$
\left\| w _ {t} ^ {1} - w _ {t} ^ {\star} \right\| _ {\nabla^ {2} \Phi \left(w _ {t} ^ {1}\right)} = \left\| w _ {t} ^ {1} - w _ {t} ^ {\star} \right\| _ {\nabla^ {2} \Phi_ {t} \left(w _ {t} ^ {1}\right)} \leq \lambda \left(w _ {t} ^ {1}, \Phi_ {t}\right) / \left(1 - M _ {\Phi} \lambda \left(w _ {t} ^ {1}, \Phi_ {t}\right)\right) \leq \frac {1}{4 9 M _ {\Phi}}. \tag {56}
$$
Now, combining (53) with the fact that  $\| u_{t - 1} - w_t^1\|_{\nabla^2\Phi (u_{t - 1})} = \| u_{t - 1} - w_{t - 1}\|_{\nabla^2\Phi (u_{t - 1})}\leq 1 / (40M_\Phi)$  (see (22)) and Lemma 2, we obtain
$$
\left(\frac {3 1}{3 2}\right) ^ {2} H _ {t - 1} \leqslant \nabla^ {2} \Phi \left(w _ {t} ^ {1}\right) \leqslant \left(\frac {3 2}{3 1}\right) ^ {2} H _ {t - 1}, \tag {57}
$$
Plugging (57) into (56), we get
$$
\left\| w _ {t} ^ {1} - w _ {t} ^ {\star} \right\| _ {H _ {t - 1}} \leq 1 / (4 0 M _ {\Phi}). \tag {58}
$$
Now, by the triangle inequality
$$
\begin{array}{l} \left\| u _ {t - 1} - w _ {t} ^ {\star} \right\| _ {H _ {t - 1}} \leq \left\| u _ {t - 1} - w _ {t} ^ {1} \right\| _ {H _ {t - 1}} + \left\| w _ {t} ^ {1} - w _ {t} ^ {\star} \right\| _ {H _ {t - 1}} \\ \leq 1 / \left(2 0 \Phi_ {M}\right), \tag {59} \\ \end{array}
$$
where in the last inequality we used (22) and (58). Combining (59) with (53) and Lemma 1, we get
$$
(4 / 5) \nabla^ {2} \Phi \left(w _ {t} ^ {\star}\right) \leqslant H _ {t - 1} \leqslant (5 / 4) \nabla^ {2} \Phi \left(w _ {t} ^ {\star}\right). \tag {60}
$$
Combining Equations (60) and (57) implies property (24) for the base of Induction. Furthermore, note that from Lemma 1:
$$
\left\| w _ {t} ^ {1} - w _ {t} ^ {\star} \right\| _ {\nabla^ {2} \Phi \left(w _ {t} ^ {\star}\right)} \leq \frac {5 0}{4 9} \lambda \left(w _ {t} ^ {1}, \Phi_ {t}\right) \leq \frac {1}{4 9 M _ {\Phi}},
$$
which shows property (26) for the base case.
Induction step. Now, assume that properties (23), (25), (26), and (27) hold for  $m \geq 1$ . We will show that these properties hold for  $m + 1$ . From the hypothesis of induction, we have
$$
\left\| w _ {t} ^ {m} - u _ {t - 1} \right\| _ {H _ {t - 1}} \leq \frac {1}{1 2 M _ {\Phi}},
$$
which combined with (53) and Lemma 1 implies
$$
0. 8 4 \nabla^ {2} \Phi \left(w _ {t} ^ {m}\right) \leqslant H _ {t - 1} \leqslant 1. 2 \nabla^ {2} \Phi \left(w _ {t} ^ {m}\right). \tag {61}
$$
Thus, by Lemma 5 (instantiated with  $c = 1/5$ ) and the fact that  $\lambda(w_t^m, \Phi_t) \leq 1/(40M_\Phi)$  (by the induction hypothesis), we get that for  $\tilde{w}_t^{m+1} := w_t^m - H_{t-1}^{-1}\nabla \Phi_t(w_t^m)$ :
$$
\lambda \left(\tilde {w} _ {t} ^ {m + 1}, \Phi_ {t}\right) \leq 9 M _ {\Phi} \lambda \left(w _ {t} ^ {m}, \Phi_ {t}\right) ^ {2} + 2. 5 c \lambda \left(w _ {t} ^ {m}, \Phi_ {t}\right) \leq (7 / 8) \lambda \left(w _ {t} ^ {m}, \Phi_ {t}\right).
$$
Again, by Lemma 5 with  $c = 1 / 5$ , we have
$$
\begin{array}{l} \lambda \left(w _ {t} ^ {m + 1}, \Phi_ {t}\right) \leq 2 0 (1 + c) \varepsilon + (1 + 2 0 (1 + c) \varepsilon) \lambda \left(\tilde {w} _ {t} ^ {m + 1}, \Phi\right) \\ \leq 2 5 \varepsilon + \left(\frac {1 5}{1 6}\right) \lambda \left(w _ {t} ^ {m}, \Phi_ {t}\right) \leq 1 / \left(5 0 M _ {\Phi}\right). \tag {62} \\ \end{array}
$$
By the induction hypothesis, we also have that  $\lambda(w_t^m, \Phi_t) \leq \left( \frac{15}{16} \right)^{m-1} \lambda(w_t^1, \Phi_t) + 500\varepsilon$ . Combining this with (62), we get
$$
\lambda \left(w _ {t} ^ {m + 1}, \Phi_ {t}\right) \leq \left(\frac {1 5}{1 6}\right) ^ {m} \lambda \left(w _ {t} ^ {1}, \Phi_ {t}\right) + 5 0 0 \varepsilon . \tag {63}
$$
This shows that (23) holds with  $m$  replaced by  $m + 1$ .
Next, we show that (25) holds with  $m$  replaced by  $m + 1$ . Combining (62) with Lemma 1 implies
$$
\left\| w _ {t} ^ {m + 1} - w _ {t} ^ {\star} \right\| _ {\nabla^ {2} \Phi \left(w _ {t} ^ {\star}\right)} \leq \lambda \left(w _ {t} ^ {m + 1}, \Phi_ {t}\right) / \left(1 - M _ {\Phi} \lambda \left(w _ {t} ^ {m + 1}, \Phi_ {t}\right)\right) \leq 1 / \left(4 9 M _ {\Phi}\right).
$$
This, together with (60) gives
$$
\left\| w _ {t} ^ {m + 1} - w _ {t} ^ {\star} \right\| _ {H _ {t - 1}} \leq 1 / \left(3 2 M _ {\Phi}\right). \tag {64}
$$
Combining (59) with (64) gives:
$$
\left\| w _ {t} ^ {m + 1} - u _ {t - 1} \right\| _ {H _ {t - 1}} \leq \frac {1}{1 2 M _ {\Phi}},
$$
which proves that (25) holds with  $m$  replaced by  $m + 1$ .
Next, we show that (26) holds with  $m$  replaced by  $m + 1$ . By (63) and Lemma 1, we have
$$
\begin{array}{l} \left\| w _ {t} ^ {m + 1} - w _ {t} ^ {\star} \right\| _ {\nabla^ {2} \Phi \left(w _ {t} ^ {\star}\right)} \leq \frac {5 0}{4 9} \left(\frac {1 5}{1 6}\right) ^ {m} \lambda \left(w _ {t} ^ {1}, \Phi_ {t}\right) + 2 4 0 \varepsilon , \\ \leq \frac {1}{4 9 M _ {\Phi}} \left(\frac {1 5}{1 6}\right) ^ {m} + 2 4 0 \varepsilon \leq \frac {1}{2 0 M _ {\Phi}}, \tag {65} \\ \end{array}
$$
where in the last inequality we used (55) and the bound on  $\varepsilon$  in the lemma's statement. This shows that (26) holds with  $m$  replaced by  $m + 1$ .
Next, we show that (27) holds with  $m$  replaced by  $m + 1$ . By plugging (60) into (65), we get
$$
\left\| w _ {t} ^ {m + 1} - w _ {t} ^ {\star} \right\| _ {H _ {t - 1}} \leq \frac {1}{4 0 M _ {\Phi}} \left(\frac {1 5}{1 6}\right) ^ {m} + 3 0 0 \varepsilon . \tag {66}
$$
On the other hand, by (54), we have
$$
\begin{array}{l} \lambda (w _ {t} ^ {1}, \Phi_ {t}) \leq \sqrt {\nabla \Phi_ {t - 1} (w _ {t - 1}) ^ {\top} (\nabla^ {2} \Phi (w _ {t - 1})) ^ {- 1} \nabla \Phi_ {t - 1} (w _ {t - 1})} + \sqrt {\eta^ {2} g _ {t - 1} ^ {\top} (\nabla^ {2} \Phi (w _ {t - 1})) ^ {- 1} g _ {t - 1}} \\ = \lambda \left(w _ {t - 1}, \Phi_ {t - 1}\right) + \eta \left\| g _ {t - 1} \right\| _ {\nabla^ {- 2} \Phi \left(w _ {t - 1}\right)}. \\ \end{array}
$$
Plugging this into (56) and using (55), we get
$$
\begin{array}{l} \left\| w _ {t} ^ {1} - w _ {t} ^ {\star} \right\| _ {\nabla^ {2} \Phi \left(w _ {t} ^ {1}\right)} \leq \frac {5 0}{4 9} \left(\lambda \left(w _ {t - 1}, \Phi_ {t - 1}\right) + \eta \left\| g _ {t - 1} \right\| _ {\nabla^ {- 2} \Phi \left(w _ {t - 1}\right)}\right), \\ \leq \frac {5 0}{4 9} \left(\alpha + \eta \left\| g _ {t - 1} \right\| _ {\nabla^ {- 2} \Phi \left(w _ {t - 1}\right)}\right), \tag {67} \\ \end{array}
$$
where the last inequality follows by (22). Combining (67) with (61) (instantiated with  $m = 1$ ),
$$
\left\| w _ {t} ^ {1} - w _ {t} ^ {\star} \right\| _ {H _ {t - 1}} \leq 2 \left(\alpha + \eta \left\| g _ {t - 1} \right\| _ {\nabla^ {- 2} \Phi \left(w _ {t - 1}\right)}\right). \tag {68}
$$
Now, by (66), (68), and the triangle inequality, we have
$$
\left\| w _ {t} ^ {1} - w _ {t} ^ {m + 1} \right\| _ {H _ {t - 1}} \leq 2 \eta \left\| g _ {t - 1} \right\| _ {\nabla^ {- 2} \Phi \left(w _ {t - 1}\right)} + \frac {1}{4 0 M _ {\Phi}} \left(\frac {1 5}{1 6}\right) ^ {m} + 5 0 0 \varepsilon + 2 \alpha .
$$
Via another triangle inequality, we get
$$
\begin{array}{l} \left| \left\| u _ {t - 1} - w _ {t} ^ {m + 1} \right\| _ {H _ {t - 1}} - \left\| u _ {t - 1} - w _ {t - 1} \right\| _ {H _ {t - 1}} \right| \leq 2 \eta \left\| g _ {t - 1} \right\| _ {\nabla^ {- 2} \Phi \left(w _ {t - 1}\right)} + \frac {1}{4 0 M _ {\Phi}} \left(\frac {1 5}{1 6}\right) ^ {m} \\ + 5 0 0 \varepsilon + 2 \alpha . \\ \end{array}
$$
This shows that (27) holds with  $m$  replaced by  $m + 1$ . Finally, combining (60) with (61) implies
$$
\frac {1}{2} \nabla^ {2} \Phi (w _ {t} ^ {m}) \preccurlyeq \nabla^ {2} \Phi (w _ {t} ^ {\star}) \preccurlyeq 2 \nabla^ {2} \Phi (w _ {t} ^ {m}),
$$
E Helper Lemmas
Lemma 8 (Bounding norm of the Newton step). Let  $y \in \mathcal{K}$  and  $H \in \mathbb{R}^{d \times d}$  be such that  $(1 - c)\nabla^2\Phi(y) \preccurlyeq H \preccurlyeq (1 + c)\nabla^2\Phi(y)$ . Then, for  $h := -H^{-1}\nabla \Phi(y)$ , we have
$$
\left\| h \right\| _ {\nabla^ {2} \Phi (y)} \leq \frac {1}{1 - c} \left\| \nabla \Phi (y) \right\| _ {\nabla^ {- 2} \Phi (y)}.
$$
$$
\begin{array}{l} \| h \| _ {\nabla^ {2} \Phi (y)} = \| H ^ {- 1} \nabla \Phi (y) \| _ {\nabla^ {2} \Phi (y)} \\ = \sqrt {\nabla \Phi (y) ^ {\top} H ^ {- 1} \nabla^ {2} \Phi (y) H ^ {- 1} \nabla \Phi (y)} \\ = \sqrt {\nabla \Phi (y) ^ {\top} \nabla^ {2} \Phi (y) ^ {- 1 / 2} \left(\nabla^ {2} \Phi (y) ^ {1 / 2} H ^ {- 1} \nabla^ {2} \Phi (y) ^ {1 / 2}\right) ^ {2} \nabla^ {2} \Phi (y) ^ {- 1 / 2} \nabla \Phi (y)}. \tag {69} \\ \end{array}
$$
For the middle matrix  $\nabla^2\Phi (y)^{1 / 2}H^{-1}\nabla^2\Phi (y)^{1 / 2}$  we have that
$$
\frac {1}{1 + c} I \preccurlyeq \nabla^ {2} \Phi (y) ^ {1 / 2} H ^ {- 1} \nabla^ {2} \Phi (y) ^ {1 / 2} \preccurlyeq \frac {1}{1 - c} I,
$$
since  $(1 - c)\nabla^{2}\Phi (y)\leqslant H\leqslant (1 + c)\nabla^{2}\Phi (y)$  by assumption. Plugging this back into (69), we get
$$
\left\| h \right\| _ {\nabla^ {2} \Phi (y)} \leq \frac {1}{1 - c} \left\| \nabla \Phi (y) \right\| _ {\nabla^ {2} \Phi (y) ^ {- 1}}.
$$
Lemma 9 (Cauchy-Schwarz). If  $-B \preccurlyeq A \preccurlyeq B$  are symmetric matrices and  $B$  is PSD, then
$$
x ^ {\top} A y \leq \| x \| _ {B} \| y \| _ {B}.
$$