SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left\| \nabla R _ {p, n} \left(w _ {p} ^ {*}\right) \right\| _ {H _ {p} ^ {- 1}} \leq \sqrt {2 \operatorname {E} \left[ \| \nabla \ell_ {p} \left(\langle w _ {p} ^ {*} , X \rangle - Y\right) \| _ {H _ {p} ^ {- 1}} ^ {2} \right] / (n \delta)}. \tag {30}
$$
It remains to control  $\| \hat{w}_p - w_p^*\|_{H_{p,n}}^2$  from below. Define the random vector
$$
Z = | \langle w _ {p} ^ {*}, X \rangle - Y | ^ {(p - 2) / 2} X
$$
Then, for any  $w\in \mathbb{R}^d$  , we have
$$
\begin{array}{l} \| w - w _ {p} ^ {*} \| _ {H _ {p, n}} ^ {2} = \left(w - w _ {p} ^ {*}\right) ^ {T} H _ {p, n} \left(w - w _ {p} ^ {*}\right) \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} (w - w _ {p} ^ {*}) ^ {T} \nabla^ {2} \ell_ {p} (\langle w _ {p} ^ {*}, X _ {i} \rangle - Y _ {i}) (w - w _ {p} ^ {*}) \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \left\langle w - w _ {p} ^ {*}, Z _ {i} \right\rangle^ {2} \\ \end{array}
$$
By assumption, the components of the random vector  $Z$  have finite fourth moment. Applying Proposition 1, and using the condition on  $n$  assumed in the statement of Theorem 4, we get that, with probability at least  $1 - \delta /2$ , for all  $w\in \mathbb{R}^d$
$$
\left\| w - w _ {p} ^ {*} \right\| _ {H _ {p, n}} ^ {2} \geq \frac {1}{2} \left\| w - w _ {p} ^ {*} \right\| _ {H _ {p}} ^ {2}. \tag {31}
$$
Combining (30) and (31) with (29) gives that with probability at least  $1 - \delta$ ,
$$
\begin{array}{l} R _ {p, n} (\hat {w} _ {p}) - R _ {p, n} (w _ {p} ^ {*}) \geq - \sqrt {2 \operatorname {E} \left[ \| \nabla \ell_ {p} (\langle w _ {p} ^ {*} , X \rangle - Y) \| _ {H _ {p} ^ {- 1}} ^ {2} \right] / (n \delta)} \| \hat {w} _ {p} - w _ {p} ^ {*} \| _ {H _ {p}} \\ + \frac {1}{1 6 (p - 1)} \| \hat {w} _ {p} - w _ {p} ^ {*} \| _ {H _ {p}} ^ {2}. \tag {32} \\ \end{array}
$$
Further combining (32) with (28) and rearranging yields that with probability at least  $1 - \delta$
$$
\left\| \hat {w} _ {p} - w _ {p} ^ {*} \right\| _ {H _ {p}} ^ {2} \leq \frac {5 1 2 p ^ {2} \mathrm {E} \left[ \| \nabla \ell_ {p} \left(\langle w _ {p} ^ {*} , X \rangle - Y\right) \| _ {H _ {p} ^ {- 1}} ^ {2} \right]}{n \delta} \tag {33}
$$
The last step is to bound the excess risk of the empirical risk minimizer using the bound (33) and (11) from Lemma 3. For that, we control the  $L^p$  norm term in (11) as follows
$$
\begin{array}{l} p ^ {p} \| \hat {w} _ {p} - w _ {p} ^ {*} \| _ {L ^ {p}} ^ {p} = \left(p ^ {2} \frac {\| \hat {w} _ {p} - w _ {p} ^ {*} \| _ {L ^ {p}} ^ {2}}{\| \hat {w} _ {p} - w _ {p} ^ {*} \| _ {H _ {p}} ^ {2}} \| \hat {w} _ {p} - w _ {p} ^ {*} \| _ {H _ {p}} ^ {2}\right) ^ {p / 2} \\ \leq \left(p ^ {2} \sup  _ {w \in \mathbb {R} ^ {d} \backslash \{0 \}} \left\{\frac {\| w \| _ {L ^ {p}} ^ {2}}{\| w \| _ {H _ {p}} ^ {2}} \right\} \| \hat {w} _ {p} - w _ {p} ^ {*} \| _ {H _ {p}} ^ {2}\right) ^ {p / 2} \\ = \left(p ^ {2} c _ {p} ^ {2} \| \hat {w} _ {p} - w _ {p} ^ {*} \| _ {H _ {p}} ^ {2}\right) ^ {p / 2}. \tag {34} \\ \end{array}
$$
Combining (33), (11), and (34) yields the result.
F Detailed proof of Theorem 5
This section contains the proof of Lemma 4 as well as that of Theorem 5.
F.1 Proof of Lemma 4
Both inequalities follow from Lemma 4.5 in Adil, Kyng, Peng, and Sachdeva [AKPS19]. (12) follows from a straightforward calculation using the lower bound of Lemma 4.5 in Adil, Kyng, Peng, and Sachdeva [AKPS19]; we omit it here. The upper bound requires a bit more work. We have by the quoted Lemma
$$
\ell_ {p} (t) - \ell_ {p} (s) - \ell_ {p} ^ {\prime} (s) (t - s) \leq \frac {4}{p (p - 1)} \gamma_ {p} (| s |, | t - s |).
$$
Now assume that  $|s| > 0$ . If  $|t - s| \leq |s|$ , we have
$$
\gamma_ {p} (| s |, | t - s |) = \frac {p}{2} | s | ^ {p - 2} (t - s) ^ {2} \leq | s | ^ {p - 2} (t - s) ^ {2} = \ell_ {p} ^ {\prime \prime} (s) (t - s) ^ {2}.
$$
Otherwise, if  $|t - s| > |s|$ , then we have
$$
\gamma_ {p} (| s |, | t - s |) = | t - s | ^ {p} - (1 - p / 2) | s | ^ {p} \leq (t - s) ^ {2} | t - s | ^ {p - 2} \leq | s | ^ {p - 2} (t - s) ^ {2} = \ell_ {p} ^ {\prime \prime} (s) (t - s) ^ {2}.
$$
Therefore in both cases we have  $\gamma_p(|s|, |t - s|) \leq \ell_p''(s)(t - s)^2$  as long as  $|s| > 0$ . Replacing  $t$  and  $s$  by  $\langle w, X \rangle - Y$  and  $\langle w_p^*, X \rangle - Y$  respectively we get, on the event that  $\langle w_p^*, X \rangle - Y \neq 0$
$$
\ell_ {p} (\langle w, X \rangle - Y) - \ell_ {p} (\langle w _ {p} ^ {*}, X \rangle - Y) - \langle \nabla \ell_ {p} (\langle w _ {p} ^ {*}, X \rangle - Y), w - w _ {p} ^ {*} \rangle \leq \frac {4}{p (p - 1)} \| w - w _ {p} ^ {*} \| _ {\nabla^ {2} \ell_ {p} (\langle w _ {p} ^ {*}, X \rangle - Y)}
$$
Recalling that by assumption  $\mathrm{P}\big(\langle w_p^*,X\rangle -Y\neq 0\big) = 1$ , taking expectation of both sides, and bounding  $1 / p\leq 1$  finishes the proof of (13).
We follow the same proof strategy as the one used in the proofs of Theorems 1 and 4. By definition of the empirical risk minimizer, we have
$$
R _ {p, n} \left(\hat {w} _ {p}\right) - R _ {p, n} \left(w _ {p} ^ {*}\right) \leq 0. \tag {35}
$$
Using (12) from Lemma 4 and the Cauchy-Schwarz inequality, we have the lower bound
$$
\begin{array}{l} R _ {p, n} \left(\hat {w} _ {p}\right) - R _ {p, n} \left(w _ {p} ^ {*}\right) \geq - \left\| \nabla R _ {p, n} \left(w _ {p} ^ {*}\right) \right\| _ {H _ {p} ^ {- 1}} \left\| \hat {w} _ {p} - w _ {p} ^ {*} \right\| _ {H _ {p}} \\ + \frac {1}{4 p ^ {2}} \frac {1}{n} \sum_ {i = 1} ^ {n} \gamma_ {p} \left(| \langle w _ {p} ^ {*}, X _ {i} \rangle - Y _ {i} |, | \langle w - w _ {p} ^ {*}, X _ {i} \rangle |\right) \tag {36} \\ \end{array}
$$
$$
\left\| \nabla R _ {p, n} \left(w _ {p} ^ {*}\right) \right\| _ {H _ {p} ^ {- 1}} \leq \sqrt {2 \operatorname {E} \left[ \| \nabla \ell_ {p} \left(\langle w _ {p} ^ {*} , X \rangle - Y\right) \| _ {H _ {p} ^ {- 1}} ^ {2} \right] / (n \delta)}. \tag {37}
$$
On the other hand, by Proposition 2, we have with probability  $1 - \delta /2$  , for all  $w\in \mathbb{R}^d$
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \gamma_ {p} \left(\left| \left\langle w _ {p} ^ {*}, X _ {i} \right\rangle - Y _ {i} \right|, \left| \left\langle w - w _ {p} ^ {*}, X _ {i} \right\rangle \right|\right) \geq \frac {p}{8} \min  \left\{\left\| w - w _ {p} ^ {*} \right\| _ {H _ {p}} ^ {2}, \varepsilon^ {2 - p} \| w - w _ {p} ^ {*} \| _ {H _ {p}} ^ {p} \right\}, \tag {38}
$$
where  $\varepsilon$  is as defined in Proposition 2. We now consider two cases. If  $\| \hat{w}_p - w_p^*\|_{H_p}^2\leq \varepsilon^{2 - p}\| \hat{w}_p - w_p^*\|_{H_p}^p$ , then combining (35), (36), (37), and (38) gives
$$
\left\| \hat {w} _ {p} - w _ {p} ^ {*} \right\| _ {H _ {p}} ^ {2} \leq \frac {8 1 9 2 \mathrm {E} \left[ \| \nabla \ell_ {p} \left(\langle w _ {p} ^ {*} , X \rangle - Y\right) \| _ {H _ {p} ^ {- 1}} ^ {2} \right]}{n \delta}. \tag {39}
$$
Otherwise,  $\| \hat{w}_p - w_p^* \|_{H_p}^2 > \varepsilon^{2 - p} \| \hat{w}_p - w_p^* \|_{H_p}^p$ , then again combining (35), (36), (37), and (38) gives
$$
\left\| \hat {w} _ {p} - w _ {p} ^ {*} \right\| _ {H _ {p}} ^ {2} \leq \left(\frac {8 1 9 2 \mathrm {E} \left[ \| \nabla \ell_ {p} \left(\langle w _ {p} ^ {*} , X \rangle - Y\right) \| _ {H _ {p} ^ {- 1}} ^ {2} \right] \varepsilon^ {2 (p - 2)}}{n \delta}\right) ^ {1 / (p - 1)} \tag {40}
$$
In either case, we have, using (39) and (40), with probability at least  $1 - \delta$
$$
\begin{array}{l} \| \hat {w} _ {p} - w _ {p} ^ {*} \| _ {H _ {p}} ^ {2} \leq \frac {8 1 9 2 \mathrm {E} \left[ \| \nabla \ell_ {p} (\langle w _ {p} ^ {*} , X \rangle - Y) \| _ {H _ {p} ^ {- 1}} ^ {2} \right]}{n \delta} \\ + \left(\frac {8 1 9 2 \operatorname {E} \left[ \| \nabla \ell_ {p} (\langle w _ {p} ^ {*} , X \rangle - Y) \| _ {H _ {p} ^ {- 1}} ^ {2} \right] \varepsilon^ {2 (p - 2)}}{n \delta}\right) ^ {1 / (p - 1)}. \\ \end{array}
$$
Combining this last inequality with (13) from Lemma 4 finishes the proof.