SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Taking empirical averages and expectations of both sides respectively shows that the excess empirical risk and excess risk also admit such an expansion
$$
R _ {2, n} (w) - R _ {2, n} \left(w _ {2} ^ {*}\right) = \left\langle \nabla R _ {2, n} \left(w _ {2} ^ {*}\right), w - w _ {2} ^ {*} \right\rangle + \frac {1}{2} \| w - w _ {2} ^ {*} \| _ {H _ {2, n}} ^ {2},
$$
$$
R _ {2} (w) - R _ {2} \left(w _ {2} ^ {*}\right) = \frac {1}{2} \| w - w _ {2} ^ {*} \| _ {H _ {2}} ^ {2}, \tag {6}
$$
where in the second equality we used that the gradient of the risk vanishes at the minimizer  $w_{2}^{*}$ . Therefore, to bound the excess risk, it is sufficient to bound the norm  $\| w - w_{2}^{*}\|_{H_{2}}$ . This is the goal of the second step, where we use two ideas. First, by definition, the excess empirical risk of the empirical risk minimizer satisfies the upper bound
$$
R _ {2, n} \left(\hat {w} _ {2}\right) - R _ {2, n} \left(w _ {2} ^ {*}\right) \leq 0. \tag {7}
$$
Second, we use the Cauchy-Schwartz inequality to lower bound the excess empirical risk by
$$
R _ {2, n} \left(\hat {w} _ {2}\right) - R _ {2, n} \left(w _ {2} ^ {*}\right) \geq - \left\| \nabla R _ {2, n} \left(w _ {2} ^ {*}\right) \right\| _ {H _ {2} ^ {- 1}} \left\| \hat {w} _ {2} - w _ {2} ^ {*} \right\| _ {H _ {2}} + \frac {1}{2} \left\| \hat {w} _ {2} - w _ {2} ^ {*} \right\| _ {H _ {2, n}} ^ {2}, \tag {8}
$$
and we further lower bound it by deriving high probability bounds on the two random terms  $\| \nabla R_{2,n}(w_2^*)\|_{H_2^{-1}}$  and  $\| \hat{w}_2 - w_2^*\|_{H_{2,n}}^2$ . The first can easily be bounded using Chebyshev's inequality and the elementary fact that the variance of the average of  $n$  i.i.d. random variables is the variance of their common distribution divided by  $n$ . Here we state the result for all  $p\in (1,\infty)$ ; the straightforward proof can be found in the Appendix D.
Lemma 2. Let  $p \in (1, \infty)$ . If  $p \in (1, 2)$ , let the assumptions of Theorem 5 hold. Then with probability at least  $1 - \delta / 2$
$$
\| \nabla R _ {p, n} (w _ {p} ^ {*}) \| _ {H _ {p} ^ {- 1}} \leq \sqrt {2 \operatorname {E} \left[ \| \nabla \ell_ {p} (\langle w _ {p} ^ {*} , X \rangle - Y) \| _ {H _ {p} ^ {- 1}} ^ {2} \right] / (n \delta)}.
$$
For the second random term  $\| \hat{w}_2 - w_2^*\|_{H_{2,n}}^2$ , we use Theorem 3.1 of Oliveira [Oli16], which we restate here, emphasizing that the existence of fourth moments of the components of the random vector is enough to ensure the existence of the needed norm equivalence constant.
Proposition 1 (Theorem 3.1, Oliveira [Oli16]). Let  $Z \in \mathbb{R}^d$  be a random vector satisfying  $\operatorname{E}[Z_j^4] < \infty$  for all  $j \in [d]$  and assume that  $\mathrm{P}(\langle v,Z\rangle = 0) = 1$  only if  $v = 0$ . For  $p \in [1,\infty)$  and  $v \in \mathbb{R}^d$ , define
$$
\| v \| _ {L ^ {p}} := \operatorname {E} [ (\langle v, Z \rangle) ^ {p} ] ^ {1 / p}, \quad \sigma^ {2} := \left(\sup  _ {v \in \mathbb {R} ^ {d} \setminus \{0 \}} \| v \| _ {L ^ {4}} / \| v \| _ {L ^ {2}}\right) ^ {4}.
$$
Let  $(Z_{i})_{i = 1}^{n}$  be i.i.d. samples of  $Z$ . Then, with probability at least  $1 - \delta$ , for all  $v\in \mathbb{R}^d$
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \langle v, Z _ {i} \rangle^ {2} \geq \left(1 - 7 \sigma \sqrt {\frac {d + 2 \log (2 / \delta)}{n}}\right) \| v \| _ {L ^ {2}} ^ {2}.
$$
Using this result we can immediately deduce the required high probability lower bound on the second random term  $\| \hat{w}_2 - w_2^*\|_{H_{2,n}}^2$ ; we leave the obvious proof to Appendix D.
Corollary 1. Under the assumptions of Theorem 1, if  $n \geq 196\sigma_2^2(d + 2\log(4/\delta))$ , then with probability at least  $1 - \delta/2$ , for all  $w \in \mathbb{R}^d$ ,
$$
\| w - w _ {2} ^ {*} \| _ {H _ {2, n}} ^ {2} \geq \frac {1}{2} \| w - w _ {2} ^ {*} \| _ {H _ {2}} ^ {2}.
$$
Combining Lemma 2, Corollary 1, and (8) yields that with probability at least  $1 - \delta$
$$
R _ {2, n} \left(\hat {w} _ {2}\right) - R _ {2, n} \left(w _ {2} ^ {*}\right) \geq - \sqrt {2 \mathrm {E} \left[ \| \nabla \ell_ {p} \left(\langle w _ {p} ^ {*} , X \rangle - Y\right) \| _ {H _ {p} ^ {- 1}} ^ {2} \right] / (n \delta)} \| \hat {w} _ {2} - w _ {2} ^ {*} \| _ {H _ {2}} + \frac {1}{4} \| \hat {w} _ {2} - w _ {2} ^ {*} \| _ {H _ {2}} ^ {2}. \tag {9}
$$
Finally, combining (7) and (9) gives that with probability at least  $1 - \delta$
$$
\| \hat {w} _ {2} - w _ {2} ^ {*} \| _ {H _ {2}} \leq 4 \sqrt {2 \operatorname {E} \left[ \| \nabla \ell_ {p} (\langle w _ {p} ^ {*} , X \rangle - Y) \| _ {H _ {p} ^ {- 1}} ^ {2} \right] / (n \delta)}.
$$
Replacing in (6) finishes the proof.
3.2 Proof Sketch of Theorem 4
The main challenge in moving from the case  $p = 2$  to the case  $p \in (2, \infty)$  is that the second order Taylor expansion of the loss is no longer exact. The standard way to deal with this problem is to assume that the loss is upper and lower bounded by quadratic functions, i.e. that it is smooth and strongly convex. Unfortunately, as discussed in Section 1, the  $\ell_p$  loss is not strongly convex for any  $p > 2$ , so we need to find another way to deal with this issue. Once this has been resolved however, the strategy we used in the proof of Theorem 1 can be applied almost verbatim to yield the result. Remarkably, a result of [AKPS22] allows us to upper and lower bound the  $p$ -th power loss for  $p \in (2, \infty)$  by its second order Taylor expansion around a point, up to some residual terms. An application of this result yields the following Lemma.
Lemma 3. Let  $p\in (2,\infty)$ . Then:
$$
R _ {p, n} (w) - R _ {p, n} \left(w _ {p} ^ {*}\right) \geq \frac {1}{8 (p - 1)} \| w - w _ {p} ^ {*} \| _ {H _ {p, n}} ^ {2} + \langle \nabla R _ {p, n} \left(w _ {p} ^ {*}\right), w - w _ {p} ^ {*} \rangle , \tag {10}
$$
$$
R _ {p} (w) - R _ {p} \left(w _ {p} ^ {*}\right) \leq \frac {2 p}{(p - 1)} \| w - w _ {p} ^ {*} \| _ {H _ {p}} ^ {2} + p ^ {p} \| w - w _ {p} ^ {*} \| _ {L ^ {p}} ^ {p}. \tag {11}
$$
Up to constant factors that depend only on  $p$  and an  $L^p$  norm residual term, Lemma 3 gives matching upper and lower bounds on the excess risk and excess empirical risk in terms of their second order Taylor expansions around the minimizer. We can thus use the approach taken in the proof of Theorem 1 to obtain the result. The only additional challenge is the control of the term  $\| \hat{w}_p - w_p^* \|_{L^p}$ , which we achieve by reducing it to an  $\| \hat{w}_p - w_p^* \|_{H_p}$  term using norm equivalence. A detailed proof of Theorem 4, including the proof of Lemma 3, can be found in Appendix E.
3.3 Proof Sketch of Theorem 5
The most technically challenging case is when  $p \in (1, 2)$ . Indeed as seen in the proof of Theorem 1, the most involved step is lower bounding the excess empirical risk with high probability. For the case  $p \in [2, \infty)$ , we achieved this by having access to a pointwise quadratic lower bound, which is not too surprising. Indeed, at small scales, we expect the second order Taylor expansion to be accurate, while at large scales, we expect the  $p$ -th power loss to grow at least quadratically for  $p \in [2, \infty)$ .
In the case of  $p \in (1,2)$ , we are faced with a harder problem. Indeed, as  $p \to 1$ , the  $\ell_p$  losses behave almost linearly at large scales. This means that we cannot expect to obtain a global quadratic lower bound as for the case  $p \in [2,\infty)$ , so we will need a different proof technique. Motivated by related concerns, Bubeck, Cohen, Lee, and Li [BCLL18] introduced the following approximation to the  $p$ -th power function
$$
\gamma_ {p} (t, x) := \left\{ \begin{array}{l l} \frac {p}{2} t ^ {p - 2} x ^ {2} & \quad \text {i f} \quad x \leq t \\ x ^ {p} - \left(1 - \frac {p}{2}\right) t ^ {p} & \quad \text {i f} \quad x > t, \end{array} \right.
$$
for  $t,x\in [0,\infty)$  and with  $\gamma_p(0,0) = 0$ . This function was further studied by Adil, Kyng, Peng, and Sachdeva [AKPS19], who showed in particular that for any  $t\in \mathbb{R}$ , the function  $x\mapsto \gamma_p(|t|,|x|)$  is, up to constants that depend only on  $p$ , equal to the gap between the function  $x\mapsto \ell_p(t + x)$  and its linearization around 0; see Lemma 4.5 in [AKPS19] for the precise statement. We use this result to derive the following Lemma.
Lemma 4. Let  $p \in (1,2)$ . Under the assumptions of Theorem 5, we have
$$
R _ {p, n} (w) - R _ {p, n} \left(w _ {p} ^ {*}\right) \geq \frac {1}{4 p ^ {2}} \frac {1}{n} \sum_ {i = 1} ^ {n} \gamma_ {p} \left(\left| \left\langle w _ {p} ^ {*}, X _ {i} \right\rangle - Y _ {i} \right|, \left| \left\langle w - w _ {p} ^ {*}, X _ {i} \right\rangle \right|\right) + \left\langle \nabla R _ {p, n} \left(w _ {p} ^ {*}\right), w - w _ {p} ^ {*} \right\rangle , \tag {12}
$$
$$
R _ {p} (w) - R _ {p} \left(w _ {p} ^ {*}\right) \leq \frac {4}{(p - 1)} \| w - w _ {p} ^ {*} \| _ {H _ {p}} ^ {2}. \tag {13}
$$
As expected, while we do have the desired quadratic upper bound, the lower bound is much more cumbersome, and is only comparable to the second order Taylor expansion when  $|\langle w - w_{p}^{*},X_{i}\rangle |\leq |\langle w_{p}^{*},X_{i}\rangle -Y_{i}|$ . What we need for the proof to go through is a high probability lower bound of order  $\Omega (\| w - w^{*}\|_{H_p}^2)$  on the first term in the lower bound (12). We obtain this in the following Proposition.
Proposition 2. Let  $\delta \in (0,1]$ . Under the assumptions of Theorem 5, if  $n \geq 196\sigma_p^2(d + 2\log(4/\delta))$ , then with probability at least  $1 - \delta/2$ , for all  $w \in \mathbb{R}^d$ ,
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \gamma_ {p} \left(| \langle w _ {p} ^ {*}, X _ {i} \rangle - Y _ {i} |, | \langle w - w _ {p} ^ {*}, X _ {i} \rangle |\right) \geq \frac {p}{8} \min  \left\{\| w - w _ {p} ^ {*} \| _ {H _ {p}} ^ {2}, \varepsilon^ {2 - p} \| w - w _ {p} ^ {*} \| _ {H _ {p}} ^ {p} \right\},
$$
where  $\varepsilon^{p - 2} \coloneqq 8\sigma_p^{3 - p}c_p^{(2 - p) / 2}\sqrt{c_p^*}$ , and  $c_{p}$  and  $c_{p}^{*}$  are as defined in Theorem 5.
Proof. Let  $\varepsilon > 0$  and let  $T \in (0, \infty)$  be a truncation parameter we will set later. Define
$$
\tilde {X} := X \cdot \mathbb {1} _ {[ 0, T ]} \left(\left\| X \right\| _ {H _ {p} ^ {- 1}}\right),
$$
and the constant  $\beta := T\varepsilon$ . By Lemma 3.3 in [AKPS19], we have that  $\gamma_p(t,\lambda x) \geq \min\{\lambda^2, \lambda^p\} \gamma_p(t,x)$  for all  $\lambda \geq 0$ . Furthermore, it is straightforward to verify that  $\gamma_p(t,x)$  is decreasing in  $t$  and increasing in  $x$ . Therefore, we have, for all  $w \in \mathbb{R}^d$ ,
$$
\begin{array}{l} \frac {1}{n} \sum_ {i = 1} ^ {n} \gamma_ {p} \left(| \langle w _ {p} ^ {*}, X _ {i} \rangle - Y _ {i} |, | \langle w - w _ {p} ^ {*}, X _ {i} \rangle |\right) \\ \geq \min  \left\{\varepsilon^ {- 2} \| w - w _ {p} \| _ {H _ {p}} ^ {2}, \varepsilon^ {- p} \| w - w _ {p} \| _ {H _ {p}} ^ {p} \right\} \frac {1}{n} \sum_ {i = 1} ^ {n} \gamma_ {p} \left(\left| \langle w _ {p} ^ {*}, X _ {i} \rangle - Y _ {i} \right|, \left| \left\langle \frac {\varepsilon (w - w _ {p} ^ {*})}{\| w - w _ {p} ^ {*} \| _ {H _ {p}}}, X _ {i} \right\rangle \right|\right) \\ \geq \min  \left\{\varepsilon^ {- 2} \| w - w _ {p} \| _ {H _ {p}} ^ {2}, \varepsilon^ {- p} \| w - w _ {p} \| _ {H _ {p}} ^ {p} \right\} \cdot \inf  _ {\| w \| _ {H _ {p}} = \varepsilon} \frac {1}{n} \sum_ {i = 1} ^ {n} \gamma_ {p} \left(| \langle w _ {p} ^ {*}, X _ {i} \rangle - Y _ {i} |, | \langle w, X _ {i} \rangle |\right). \tag {14} \\ \end{array}
$$
The key idea to control the infimum in (14) is to truncate  $\langle w, X_i \rangle$  from above by using the truncated vector  $\tilde{X}$ , and  $|\langle w_p^*, X_i \rangle - Y_i|$  from below by forcing it to be greater than  $\beta$ . By the monotonicity properties of  $\gamma_p$  discussed above, we get that
$$
\begin{array}{l} \inf  _ {\| w \| _ {H _ {p}} = \varepsilon} \frac {1}{n} \sum_ {i = 1} ^ {n} \gamma_ {p} \left(\left| \langle w _ {p} ^ {*}, X _ {i} \rangle - Y _ {i} \right|, \left| \langle w, X _ {i} \rangle \right|\right) \\ \geq \inf _ {\| w \| _ {H _ {p}} = \varepsilon} \frac {1}{n} \sum_ {i = 1} ^ {n} \gamma_ {p} (\max  \{| \langle w _ {p} ^ {*}, X _ {i} \rangle - Y _ {i} |, \beta \}, | \langle w, \tilde {X} _ {i} \rangle |) \\ = \frac {\varepsilon^ {2} p}{2} \inf  _ {\| w \| _ {H _ {p}} = 1} \frac {1}{n} \sum_ {i = 1} ^ {n} \max  \left\{\left| \left\langle w _ {p} ^ {*}, X _ {i} \right\rangle - Y _ {i} \right|, \beta \right\} ^ {p - 2} | \langle w, \tilde {X} _ {i} \rangle | ^ {2}, \tag {15} \\ \end{array}
$$
where the equality follows by the fact that with the chosen truncations, the second argument of  $\gamma_{p}$  is less than or equal to the first. It remains to lower bound the infimum in (15). Define
$$
Z = \max  \left\{\left| \langle w _ {p} ^ {*}, X \rangle - Y \right|, \beta \right\} ^ {(p - 2) / 2} \tilde {X}.
$$
Removing the truncations and using our assumptions, we see that the components of  $Z$  have finite fourth moment. By Proposition 1 and the condition on  $n$ , we get that with probability at least  $1 - \delta / 2$ ,
$$
\begin{array}{l} \inf _ {\| w \| _ {H _ {p}} = 1} \frac {1}{n} \sum_ {i = 1} ^ {n} \max  \left\{\left| \left\langle w _ {p} ^ {*}, X _ {i} \right\rangle - Y _ {i} \right|, \beta \right\} ^ {p - 2} | \langle w, \tilde {X} _ {i} \rangle | ^ {2} \\ = \inf  _ {\| w \| _ {H _ {p}} = 1} \frac {1}{n} \sum_ {i = 1} ^ {n} \langle w, Z _ {i} \rangle^ {2} \geq \frac {1}{2} \inf  _ {\| w \| _ {H _ {p}} = 1} \mathrm {E} \big [ \langle w, Z \rangle^ {2} \big ] \\ = \frac {1}{2} \inf  _ {\| w \| _ {H _ {p}} = 1} \operatorname {E} \left[ \max  \left\{\left| \langle w _ {p} ^ {*}, X \rangle - Y \right|, \beta \right\} ^ {(p - 2)} \langle w, \tilde {X} \rangle^ {2} \right] \\ \geq \frac {1}{2} \left(1 - \sup  _ {\| w \| _ {H _ {p}} = 1} \mathrm {E} \left[ | \langle w _ {p} ^ {*}, X \rangle - Y | ^ {p - 2} \langle w, X \rangle^ {2} \left(\mathbb {1} _ {[ 0, \beta)} \left(| \langle w _ {p} ^ {*}, X \rangle - Y |\right) + \mathbb {1} _ {(T, \infty)} \left(\| X \| _ {H _ {p} ^ {- 1}}\right)\right) \right]\right) \tag {16} \\ \end{array}
$$
We now bound the supremum in (16). We have
$$
\begin{array}{l} \sup _ {\| w \| _ {H _ {p}} = 1} \mathrm {E} \Big [ | \langle w _ {p} ^ {*}, X \rangle - Y | ^ {p - 2} \langle w, X \rangle^ {2} \Big (\mathbb {1} _ {[ 0, \beta)} \big (| \langle w _ {p} ^ {*}, X \rangle - Y | \big) + \mathbb {1} _ {(T, \infty)} \big (\| X \| _ {H _ {p} ^ {- 1}} \big) \Big) \Big ] \\ \leq \sup _ {\| w \| _ {H _ {p}} = 1} \left\{\operatorname {E} \left[ | \langle w _ {p} ^ {*}, X \rangle - Y | ^ {2 (p - 2)} \langle w, X \rangle^ {4} \right] ^ {1 / 2} \right\} \left(\operatorname {P} \left(| \langle w _ {p} ^ {*}, X \rangle - Y | <   \beta\right) + \operatorname {P} \left(\| X \| _ {H _ {p} ^ {- 1}} > T\right)\right) ^ {1 / 2} \\ = \left(\sup  _ {\| w \| _ {H _ {p}} = 1} \| w \| _ {L ^ {4}, p} ^ {2}\right) \left(\mathrm {P} \left(| \langle w _ {p} ^ {*}, X \rangle - Y | <   \beta\right) + \mathrm {P} \left(\| X \| _ {H _ {p} ^ {- 1}} > T\right)\right) ^ {1 / 2} \\ = \sigma_ {p} \left(\mathrm {P} \left(| \langle w _ {p} ^ {*}, X \rangle - Y | <   \beta\right) + \mathrm {P} \left(\| X \| _ {H _ {p} ^ {- 1}} > T\right)\right) ^ {1 / 2}, \tag {17} \\ \end{array}
$$
where the first inequality follows from Cauchy-Schwartz inequality, and the subsequent equalities by definitions of  $\| \cdot \|_{L^4,p}$  and  $\sigma_p^2$ . It remains to bound the tail probabilities. Recall that  $\beta = T\varepsilon$ , so
$$
\begin{array}{l} \mathrm {P} \left(| \langle w _ {p} ^ {*}, X \rangle - Y | <   \beta\right) = \mathrm {P} \left(| \langle w _ {p} ^ {*}, X \rangle - Y | <   T \varepsilon\right) \\ = \mathrm {P} \left(\left| \left\langle w _ {p} ^ {*}, X \right\rangle - Y \right| ^ {- 1} > (T \varepsilon) ^ {- 1}\right) \\ = \mathrm {P} \left(\left| \left\langle w _ {p} ^ {*}, X \right\rangle - Y \right| ^ {2 (p - 2)} > \left(T \varepsilon\right) ^ {2 (p - 2)}\right) \\ \leq \mathrm {E} [ | \langle w _ {p} ^ {*}, X \rangle - Y | ^ {2 (p - 2)} ] \left(T \varepsilon\right) ^ {2 (2 - p)} \\ = c _ {p} ^ {*} (T \varepsilon) ^ {2 (2 - p)}, \\ \end{array}
$$
where we applied Markov's inequality in the fourth line, and the last follows by definition of  $c_p^*$  in Theorem 5. Moreover by the finiteness of the second moment of the coordinates  $X^j$  of  $X$ , we have
$$
\operatorname {E} \left[ \| X \| _ {H _ {p} ^ {- 1}} ^ {2} \right] = \operatorname {E} \left[ X ^ {T} H _ {p} ^ {- 1} X \right] = \operatorname {E} \left[ \operatorname {T r} \left(H _ {p} ^ {- 1} X X ^ {T}\right) \right] = \operatorname {T r} \left(H _ {p} ^ {- 1} \Sigma\right) = c _ {p}
$$
where  $\Sigma = \operatorname{E}[XX^T]$ , and the last equality by definition of  $c_p$  in Theorem 5. By Markov's inequality
$$
\mathrm {P} \big (\left| \langle w _ {p} ^ {*}, X \rangle - Y \right| <   T \varepsilon \big) + \mathrm {P} \Big (\left\| X \right\| _ {H _ {p} ^ {- 1}} > T \Big) \leq c _ {p} ^ {*} T ^ {2 (2 - p)} \varepsilon^ {2 (2 - p)} + \frac {c _ {p}}{T ^ {2}}.
$$
$$
T := \left(\frac {c _ {p}}{c _ {p} ^ {*} (2 - p)}\right) ^ {1 / (6 - 2 p)}, \qquad \qquad \varepsilon^ {2 - p} := \frac {1}{8 \sigma_ {p} ^ {3 - p} \sqrt {c _ {p} ^ {*}} \cdot c _ {p} ^ {(2 - p) / 2}},
$$
ensures that
$$
\sigma_ {p} \left(\mathrm {P} \left(| \langle w _ {p} ^ {*}, X \rangle - Y | <   T ^ {*} \varepsilon\right) + \mathrm {P} \left(\| X \| _ {H _ {p} ^ {- 1}} > T ^ {*}\right)\right) ^ {1 / 2} \leq 1 / 2. \tag {18}
$$
Combining the inequalities (18), (17), (16), (15), and (14) yields the result.
A detailed proof of Theorem 5, including the proof of Lemma 4, can be found in Appendix F.
We thank Nikita Zhivotovskiy, Sushant Sachdeva, and Deeksha Adil for feedback on the manuscript. MAE was partially supported by NSERC Grant [2019-06167], CIFAR AI Chairs program, and CIFAR AI Catalyst grant.
[LM17b] G. Lecué and S. Mendelson. "Sparse Recovery under Weak Moment Assumptions". In: Journal of the European Mathematical Society (Feb. 15, 2017). DOI: 10.4171/jems/682.
[LM18] G. Lecué and S. Mendelson. “Regularization and the Small-Ball Method I: Sparse Recovery”. In: The Annals of Statistics (Apr. 2018). DOI: 10.1214/17-A0S1562.
[Men14] S. Mendelson. "Learning without Concentration". In: Proceedings of The 27th Conference on Learning Theory. May 29, 2014. URL: https://proceedings.mlr.org/press/v35/mendelson14.html.
[Men18] S. Mendelson. "Learning without Concentration for General Loss Functions". In: Probability Theory and Related Fields (June 1, 2018). DOI: 10.1007/s00440-017-0784-y.
[Men21] S. Mendelson. "Learning Bounded Subsets of  $L_p$ ". In: IEEE Transactions on Information Theory (Aug. 2021). DOI: 10.1109/TIT.2021.3083553.
[MG22] J. Mourtada and S. Gaiffas. "An Improper Estimator with Optimal Excess Risk in Misspecified Density Estimation and Logistic Regression". In: Journal of Machine Learning Research (2022). URL: http://jmlr.org/papers/v23/20-782.html.
[Mou22] J. Mourtada. "Exact Minimax Risk for Linear Least Squares, and the Lower Tail of Sample Covariance Matrices". In: The Annals of Statistics (Aug. 2022). DOI: 10.1214/22-A0S2181.
[MMVZ22] J. Mourtada, T. Vaškevicius, and N. Zhivotovsky. "Distribution-free robust linear regression". In: Mathematical Statistics and Learning (2022). DOI: 10.4171/MSL/27.
[Nie92] W. Niemiro. "Asymptotics for  $M$ -Estimators Defined by Convex Minimization". In: The Annals of Statistics (Sept. 1992). DOI: 10.1214/aos/1176348782.
[OB21] D. M. Ostrovskii and F. Bach. "Finite-Sample Analysis of  $M$ -Estimators Using Self-Concordance". In: Electronic Journal of Statistics (Jan. 2021). DOI: 10.1214/20-EJS1780.
[Oli16] R. I. Oliveira. “The Lower Tail of Random Quadratic Forms with Applications to Ordinary Least Squares”. In: Probability Theory and Related Fields (Dec. 1, 2016). DOI: 10.1007/s00440-016-0738-9.
[Ron84] A. E. Ronner. "Asymptotic Normality of  $p$ -norm Estimators in Multiple Regression". In: Zeitschrift für Wahrscheinlichkeitstheorie und Verwandte Gebiete (Sept. 1, 1984). DOI: 10.1007/BF00531893.
[RV15] M. Rudelson and R. Vershynin. "Small Ball Probabilities for Linear Images of High-Dimensional Distributions". In: International Mathematics Research Notices (2015). DOI: 10.1093/imrn/rnu243.
[Aau18] A. Saumard. "On Optimality of Empirical Risk Minimization in Linear Aggregation". In: Bernoulli (Aug. 2018). DOI: 10.3150/17-BEJ925.
[Vaa98] A. W. Van Der Vaart. Asymptotic Statistics. 1998. DOI: 10.1017/CB09780511802256.
[VW96] A. W. Van Der Vaart and J. A. Wellner. Weak Convergence and Empirical Processes. 1996. DOI: 10.1007/978-1-4757-2545-2.
[VZ23] T. Vaškevicius and N. Zhivotovskiy. "Suboptimality of Constrained Least Squares and Improvements via Non-Linear Predictors". In: Bernoulli (Feb. 2023). DOI: 10.3150/22-BEJ1465.
[Whi82] H. White. "Maximum Likelihood Estimation of Misspecified Models". In: Econometrica (1982). DOI: 10.2307/1912526.
A Differentiability of the risk
In this section, we rigorously establish the twice differentiability the risk under our assumptions. We start by showing that under a subset of our assumptions, the risk is differentiable everywhere on  $\mathbb{R}^d$ .
Lemma 5. Let  $p \in (1, \infty)$  and assume that  $\operatorname{E}[|Y|^p] < \infty$  and  $\operatorname{E}[|X_j|^p] < \infty$  for all  $j \in [d]$ . Then  $R_p$  is differentiable on  $\mathbb{R}^d$ , and
$$
\nabla R _ {p} (w) = \operatorname {E} [ \nabla \ell_ {p} (\langle w, X \rangle - Y) ].
$$
Proof. Let  $w \in \mathbb{R}^d$ . We want to show that
$$
\lim _ {\Delta \to 0} \frac {| R _ {p} (w + \Delta) - R _ {p} (w) - \mathrm {E} [ \langle \nabla \ell_ {p} (\langle w , X \rangle - Y) , \Delta \rangle ] |}{\| \Delta \|} = 0,
$$
where, for convenience, we take the norm  $\| \cdot \|$  to be the Euclidean norm. Define the function  $\phi (w,X,Y)\coloneqq \ell_p(\langle w,X\rangle -Y)$  and note that by the chain rule  $\phi$  is differentiable as a function of  $w$  on all of  $\mathbb{R}^d$ . Now let  $(\Delta_k)_{k = 1}^n$  be a sequence in  $\mathbb{R}^d$  such that  $\lim_{k\to \infty}\| \Delta_k\| = 0$ . Then
$$
\begin{array}{l} \lim  _ {k \rightarrow \infty} \frac {\left| R _ {p} (w + \Delta_ {k}) - R _ {p} (w) - \mathrm {E} [ \langle \nabla \phi (w , X , Y) , \Delta_ {k} \rangle ] \right|}{\| \Delta_ {k} \|} \\ = \lim  _ {k \rightarrow \infty} \frac {\left| \mathrm {E} [ \phi (w + \Delta_ {k} , X , Y) - \phi (w , X , Y) - \langle \nabla \phi (w , X , Y) , \Delta_ {k} \rangle ] \right|}{\| \Delta_ {k} \|} \\ \leq \lim  _ {k \rightarrow \infty} \mathrm {E} \left[ \frac {\left| \phi \left(w + \Delta_ {k} , X , Y\right) - \phi \left(w , X , Y\right) - \langle \nabla \phi (w , X , Y) , \Delta_ {k} \rangle \right|}{\left\| \Delta_ {k} \right\|} \right]. \tag {19} \\ \end{array}
$$
Our goal is to interchange the limit and expectation. For that, we will use the dominated convergence theorem. We construct our dominating function as follows. Let  $R \coloneqq \sup_{k \in \mathbb{N}} \| \Delta_k \|$ , and note that  $R < \infty$  since  $\| \Delta_k \| \to 0$  as  $k \to \infty$ . Then we have