SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \frac {\left| \phi (w + \Delta_ {k} , X , Y) - \phi (w , X , Y) - \langle \nabla \phi (w , X , Y) , \Delta_ {k} \rangle \right|}{\| \Delta_ {k} \|} \\ \leq \frac {\left| \phi (w + \Delta_ {k} , X , Y) - \phi (w , X , Y) \right|}{\| \Delta_ {k} \|} + \frac {\left| \langle \nabla \phi (w , X , Y) , \Delta_ {k} \rangle \right|}{\| \Delta_ {k} \|} \\ \leq \frac {\left\langle \int_ {0} ^ {1} \nabla \phi (w + t \Delta_ {k} , X , Y) d t , \Delta_ {k} \right\rangle}{\| \Delta_ {k} \|} + \| \nabla \phi (w, X, Y) \| \\ \leq \left\| \int_ {0} ^ {1} \nabla \phi (w + t \Delta_ {k}, X, Y) d t \right\| + \| \nabla \phi (w, X, Y) \| \\ \leq \int_ {0} ^ {1} \| \nabla \phi (w + t \Delta_ {k}, X, Y) \| d t + \| \nabla \phi (w, X, Y) \| \\ \leq 2 \sup  _ {\Delta \in B (0, R)} \| \nabla \phi (w + \Delta , X, Y) \| \\ \leq \frac{2}{p - 1}\| X\| \sup_{\Delta \in B(0,R)}|\langle w + \Delta ,X\rangle -Y|^{p - 1} \\ \leq \frac {2}{p - 1} \| X \| \sup  _ {\Delta \in B (0, R)} \max  \left\{2 ^ {p - 1}, 1 \right\} \left(\left| \langle w, X \rangle - Y \right| ^ {p - 1} + \left| \langle \Delta , X \rangle \right| ^ {p - 1}\right) \\ = \frac {2 ^ {p}}{p - 1} \left\{\left| \langle w, X \rangle - Y \right| ^ {p - 1} \| X \| + R ^ {p - 1} \| X \| ^ {p} \right\} =: g (X, Y), \\ \end{array}
$$
where the second line follows by triangle inequality, the third from the fundamental theorem of calculus applied component-wise, the fourth by Cauchy-Schwartz inequality, the fifth by Jensen's inequality and the convexity of the norm, and the eighth by the inequality  $|a + b|^q \leq \max \{2^{q - 1}, 1\} (|a|^q + |b|^q)$ .
valid for  $q > 0$ . It remains to show that  $g(X, Y)$  is integrable. We have
$$
\begin{array}{l} \operatorname {E} [ g (X, Y) ] = \frac {2 ^ {p}}{p - 1} \operatorname {E} \Big [ | \langle w, X \rangle - Y | ^ {p - 1} \| X \| + R ^ {p - 1} \| X \| ^ {p} \Big ] \\ = \frac {2 ^ {p}}{p - 1} \left\{\sum_ {j = 1} ^ {d} \mathrm {E} \left[ | \langle w, X \rangle - Y | ^ {p - 1} | X ^ {j} | \right] + R ^ {p - 1} \mathrm {E} \left[ \left(\sum_ {j = 1} ^ {d} | X ^ {j} |\right) ^ {p} \right] \right\} \\ \leq \frac {2 ^ {p}}{p - 1} \left\{\sum_ {j = 1} ^ {d} \mathrm {E} [ | \langle w, X \rangle - Y | ^ {p} ] ^ {\frac {p - 1}{p}} \mathrm {E} [ | X _ {j} | ^ {p} ] ^ {1 / p} + R ^ {p - 1} d ^ {p} \sum_ {j = 1} ^ {d} \mathrm {E} [ | X ^ {j} | ^ {p} ] \right\} \\ <   \infty , \\ \end{array}
$$
where in the second line we used that the Euclidean norm is bounded by the 1-norm, in the third we used Holder's inequality, and the last line follows from our assumptions. Applying the dominated convergence theorem, we interchange the limit and the expectation in (19). Recalling that  $\phi$  is differentiable finishes the proof.
We now turn to the twice differentiability of the risk. We start with the easy case  $p \in [2, \infty)$ . The proof is very similar to that of Lemma 5 and we omit it here.
Lemma 6. Let  $p \in [2, \infty)$  and assume that  $\operatorname{E}[|Y|^p] < \infty$  and  $\operatorname{E}[|X_j|^p] < \infty$  for all  $j \in [d]$ . Then  $R_p$  is twice differentiable on  $\mathbb{R}^d$ , and
$$
\nabla^ {2} R _ {p} (w) = \operatorname {E} [ \nabla^ {2} \ell_ {p} (\langle w, X \rangle - Y) ].
$$
The case  $p \in (1,2)$  is more complicated. The following lemma establishes the twice differentiability of the risk at its minimizer under a subset of the assumptions of Theorem 5.
Lemma 7. Let  $p \in (1,2)$ . Assume that  $\mathrm{P}\big(|\langle w_p^*,X\rangle -Y| = 0\big) = 0$  and  $\operatorname {E}[|\langle w_p^*,X\rangle - Y|^{p - 2}(X^j)^2 ] < \infty$  for all  $j\in [d]$ . Then  $R_{p}$  is twice differentiable at  $w_{p}^{*}$  and
$$
\nabla^ {2} R _ {p} (w _ {p} ^ {*}) = \operatorname {E} [ \nabla^ {2} \ell_ {p} (\langle w _ {p} ^ {*}, X \rangle - Y) ]
$$
Proof. The difficulty in the proof compared to Lemma 5 and Lemma 6 stems from the fact that the loss is not twice differentiable at zero. We still rely on the dominated convergence theorem, but the construction of the dominating function is slightly more intricate. Using the setup of the proof of Lemma 5, and following the same line of arguments, we arrive at
$$
\begin{array}{l} \lim _ {k \to \infty} \frac {\| \nabla R _ {p} (w _ {p} ^ {*} + \Delta_ {k}) - \nabla R _ {p} (w _ {p} ^ {*}) - \operatorname {E} \big [ \nabla^ {2} \phi (w _ {p} ^ {*} , X , Y) \Delta_ {k} \big ] \|}{\| \Delta_ {k} \|} \\ \leq \lim  _ {k \rightarrow \infty} \mathrm {E} \left[ \frac {\| \nabla \phi \left(w _ {p} ^ {*} + \Delta_ {k} , X , Y\right) - \nabla \phi \left(w _ {p} ^ {*} , X , Y\right) - \nabla^ {2} \phi \left(w _ {p} ^ {*} , X , Y\right) \Delta_ {k} \|}{\| \Delta_ {k} \|} \right], \tag {20} \\ \end{array}
$$
where we have used the fact that since  $\mathrm{P}\big(|\langle w_p^*,X\rangle -Y| = 0\big) = 0$ ,  $\phi (w,X,Y)$  is almost surely twice differentiable at  $w_{p}^{*}$ . To finish the proof, it remains to construct a dominating function for the above sequence to justify the interchange of the limit and expectation. We consider two cases.
Case 1:  $\| \Delta_k\| \geq \left|\langle w_p^*,X\rangle -Y\right| / (2\| X\|) = :R(X,Y)$ . Then we have
$$
\begin{array}{l} \begin{array}{c} \| \nabla \phi (w _ {p} ^ {*} + \Delta_ {k}, X, Y) - \nabla \phi (w _ {p} ^ {*}, X, Y) - \nabla^ {2} \phi (w _ {p} ^ {*}, X, Y) \Delta_ {k} \| \\ \hline \| \Delta_ {k} \| \end{array} \\ \leq \frac {\| \nabla \phi (w _ {p} ^ {*} + \Delta_ {k} , X , Y) \| + \| \nabla \phi (w _ {p} ^ {*} , X , Y) \| + \| \nabla^ {2} \phi (w _ {p} ^ {*} , X , Y) \Delta_ {k} \|}{\| \Delta_ {k} \|} \\ \leq \frac {\left(\left| \langle w _ {p} ^ {*} + \Delta , X \rangle - Y \right| ^ {p - 1} + \left| \langle w _ {p} ^ {*} , X \rangle - Y \right| ^ {p - 1}\right) \| X \|}{(p - 1) \| \Delta_ {k} \|} + \| \nabla^ {2} \phi (w _ {p} ^ {*}, X, Y) \| _ {o p} \\ \leq \frac {2 \left| \langle w _ {p} ^ {*} , X \rangle - Y \right| ^ {p - 1} \| X \|}{(p - 1) \| \Delta_ {k} \|} + \left| \langle w _ {p} ^ {*}, X \rangle - Y \right| ^ {p - 2} \| X \| ^ {2} + \frac {\left| \langle \Delta_ {k} / \| \Delta_ {k} \| , X \rangle \right| ^ {p - 1} \| X \|}{(p - 1) \| \Delta_ {k} \| ^ {2 - p}} \\ \leq \frac {4 \left| \langle w _ {p} ^ {*} , X \rangle - Y \right| ^ {p - 2} \| X \| ^ {2}}{(p - 1)} + \left| \langle w _ {p} ^ {*}, X \rangle - Y \right| ^ {p - 2} \| X \| ^ {2} + \frac {\| X \| ^ {p}}{(p - 1) \| \Delta_ {k} \| ^ {2 - p}} \\ \leq \frac {7 \left| \langle w _ {p} ^ {*} , X \rangle - Y \right| ^ {p - 2} \| X \| ^ {2}}{(p - 1)} \\ \end{array}
$$
where the second line follows by triangle inequality, the third by definition of the operator norm, the fourth by  $|a + b|^q \leq |a|^q + |b|^q$  valid for  $q \in (0,1)$ , and the fifth and sixth by Cauchy-Schwartz inequality and the assumed lower bound on  $\| \Delta_k\|$ .
Case 2:  $\| \Delta_k \| < R(X, Y)$ . We start by noting that, for all  $\Delta \in B(0, R(X, Y)) \coloneqq \{x \in \mathbb{R}^d \mid \| x \| < R(X, Y)\}$ , we have
$$
\left| \left\langle w _ {p} ^ {*} + \Delta , X \right\rangle - Y \right| \geq \left| \left\langle w _ {p} ^ {*}, X \right\rangle - Y \right| - \left| \left\langle \Delta , X \right\rangle \right| \geq \left| \left\langle w _ {p} ^ {*}, X \right\rangle - Y \right| - \| \Delta \| \| X \| > \left| \left\langle w _ {p} ^ {*}, X \right\rangle - Y \right| / 2 > 0.
$$
Therefore  $\phi (w,X,Y)$  is twice differentiable on  $B(0,R(X,Y))$ . Now
$$
\begin{array}{l} \frac {\| \nabla \phi (w _ {p} ^ {*} + \Delta_ {k} , X , Y) - \nabla \phi (w _ {p} ^ {*} , X , Y) - \nabla^ {2} \phi (w _ {p} ^ {*} , X , Y) \Delta_ {k} \|}{\| \Delta_ {k} \|} \\ \leq \frac {\| \nabla \phi (w _ {p} ^ {*} + \Delta_ {k} , X , Y) - \nabla \phi (w _ {p} ^ {*} , X , Y) \| + \| \nabla^ {2} \phi (w _ {p} ^ {*} , X , Y) \Delta_ {k} \|}{\| \Delta_ {k} \|} \\ \leq \frac {\left\| \left(\int_ {0} ^ {1} \nabla^ {2} \phi \left(w _ {p} ^ {*} + t \Delta_ {k} , X , Y\right) d t\right) \Delta_ {k} \right\|}{\| \Delta_ {k} \|} + \| \nabla^ {2} \phi \left(w _ {p} ^ {*}, X, Y\right) \| _ {o p} \\ \leq \left\| \int_ {0} ^ {1} \nabla^ {2} \phi (w + t \Delta_ {k}, X, Y) d t \right\| _ {o p} + \| \nabla^ {2} \phi \left(w _ {p} ^ {*}, X, Y\right) \| _ {o p} \\ \leq \int_ {0} ^ {1} \| \nabla^ {2} \phi (w + t \Delta_ {k}, X, Y) \| _ {o p} d t + \| \nabla^ {2} \phi (w _ {p} ^ {*}, X, Y) \| _ {o p} \\ \leq 2 \sup  _ {\Delta \in B (0, R (X, Y))} \| \nabla^ {2} \phi \left(w _ {p} ^ {*} + \Delta , X, Y\right) \| _ {o p} \\ \leq 2 \| X \| _ {2} ^ {2} \sup  _ {\Delta \in B (0, R (X, Y))} \left| \left\langle w _ {p} ^ {*} + \Delta , X \right\rangle - Y \right| ^ {p - 2} \\ \leq 4 \left| \langle w _ {p} ^ {*}, X \rangle - Y \right| ^ {p - 2} \| X \| _ {2} ^ {2} \\ \end{array}
$$
where the second line follows from the triangle inequality, the third follows from the twice differentiability of  $\phi$  on  $B(0,R(X,Y))$  and the fundamental theorem of calculus applied component-wise, the fifth by Jensen's inequality, and the last by definition of  $R(X,Y)$  and the above lower bound. We therefore define our dominating function by
$$
g (X, Y) := 8 \left| \langle w _ {p} ^ {*}, X \rangle - Y \right| ^ {p - 2} \| X \| _ {2} ^ {2}.
$$
It is then immediate from our assumptions that  $g(X, Y)$  is integrable. Interchanging the limit and the expectation in (20) and recalling that  $\phi$  is almost surely twice differentiable at  $w_{p}^{*}$  finishes the proof.
We start with the claim that  $\rho_0$  is upper-semicontinuous. We want to show that for any  $w\in \mathbb{R}^d$  and any sequence  $(w_{k})_{k = 1}^{\infty}$  converging to  $w$  (in the norm topology)
$$
\limsup_{k\to \infty}\rho_{0}(w_{k})\leq \rho_{0}(w).
$$
Fix a  $w \in \mathbb{R}^d$  and let  $(w_k)_{k=1}^\infty$  be a sequence in  $\mathbb{R}^d$  satisfying  $\lim_{k \to \infty} \| w - w_k \| = 0$ , where for convenience we take  $\| \cdot \|$  to be the Euclidean norm on  $\mathbb{R}^d$ . Then we have by (reverse) Fatou's Lemma
$$
\lim  _ {k \rightarrow \infty} \rho_ {0} (w _ {k}) = \lim  _ {k \rightarrow \infty} \operatorname {E} \left[ \mathbb {1} _ {\{0 \}} (\langle w _ {k}, X \rangle) \right] \leq \operatorname {E} \left[ \lim  _ {k \rightarrow \infty} \mathbb {1} _ {\{0 \}} (\langle w _ {k}, X \rangle) \right]. \tag {21}
$$
Now we bound the inner limsup pointwise. We split this task in two cases. If  $\langle w, X \rangle = 0$ , then
$$
\lim  _ {k \rightarrow \infty} \mathbb {1} _ {\{0 \}} (\langle w _ {k}, X \rangle) \leq 1 = \mathbb {1} _ {\{0 \}} (\langle w, X \rangle). \tag {22}
$$
Otherwise we have  $\delta := |\langle w, X \rangle| > 0$ . But then, by the convergence of  $(w_k)_{k=1}^{\infty}$  to  $w$ , there exists a  $K \in \mathbb{N}$  such that for all  $k \geq K$  we have  $\|w_k - w\| < \delta/(2\|X\|)$ . This implies that for all  $k \geq K$
$$
| \langle w _ {k}, X \rangle | = | \langle w, X \rangle - \langle w - w _ {k}, X \rangle | \geq | \langle w, X \rangle | - | \langle w - w _ {k}, X \rangle | \geq \delta - \| w _ {k} - w \| _ {2} \| X \| \geq \delta / 2 > 0.
$$
$$
\lim  _ {k \rightarrow \infty} \mathbb {1} _ {\{0 \}} (\langle w _ {k}, X \rangle) = \lim  _ {k \rightarrow \infty} \mathbb {1} _ {\{0 \}} (\langle w _ {k}, X \rangle) = 0 = \mathbb {1} _ {\{0 \}} (\langle w, X \rangle). \tag {23}
$$
Combining (21), (22), and (23) proves the upper-semicontinuity of  $\rho_0$ . Essentially the same proof shows the lower-semicontinuity of  $\rho_q(\cdot, \kappa)$  for any  $\kappa \geq 0$ ; we omit it here.
For the remaining claims, first notice that the function  $\rho_0$  is scale invariant, i.e. for all  $w\in \mathbb{R}^d$  and all  $c\in \mathbb{R}$ , we have  $\rho_0(cw) = \rho_0(w)$ . Therefore
$$
\sup_{w\in \mathbb{R}^{d}\setminus \{0\}}\rho_{0}(w) = \sup_{w\in \mathbb{S}^{d - 1}}\rho_{0}(w),
$$
where  $\mathbb{S}^{d - 1}$  is the Euclidean unit sphere. By assumption on the random vector  $X$ , we know that  $\rho_0(w) < 1$  for all  $w \in \mathbb{S}^{d - 1}$ . Furthermore since  $\rho_0$  is upper semicontinuous, and  $\mathbb{S}^{d - 1}$  is compact,  $\rho_0$  attains its supremum on  $\mathbb{S}^{d - 1}$  at some point  $w_0 \in \mathbb{S}^{d - 1}$ . From this we conclude that
$$
\rho = \sup_{w\in \mathbb{R}^{d}\setminus \{0\}}\rho_{0}(w) = \rho_{0}(w_{0}) <   1.
$$
Finally, we turn to the claim about  $\rho_q$ . Since  $\operatorname{E}[|X^j|^q] < \infty$ , the function  $\|\cdot\|_{L^q}$  is a norm on  $\mathbb{R}^d$ , from which it follows that  $\rho_q(w,\kappa)$  is also scale invariant for any  $\kappa$ . Therefore
$$
\inf_{w\in \mathbb{R}^{d}\setminus \{0\}}\rho_{q}(w,\kappa) = \inf_{w\in S_{q}}\rho_{q}(w,\kappa),
$$
where  $S_{q}$  is the unit sphere of the norm  $\| \cdot \|_{L^q}$ . Now fix  $\kappa \in [0,1)$ . We claim that  $\rho_q(w,\kappa) > 0$  for all  $w\in S_q$ . Suppose not. Then there exists a  $w\in S_q$  such that  $|\langle w,X\rangle |\leq \kappa$  with probability 1, but then we get the contradiction
$$
1 = \| w \| _ {L ^ {q}} = \operatorname {E} [ | \langle w, X \rangle | ^ {q} ] ^ {1 / q} \leq \kappa <   1.
$$
therefore  $\rho_q(w,\kappa) > 0$  for all  $w\in S_q$ . Now since  $\rho_q(\cdot ,\kappa)$  is lower-semicontinuous, and  $S_{q}$  is compact,  $\rho_q(\cdot ,\kappa)$  attains its infimum on  $S_{q}$  at some point  $w_{q}\in S_{q}$ . From this we conclude
$$
\inf_{w\in \mathbb{R}^{d}\setminus \{0\}}\rho_{q}(w,\kappa) = \rho_{q}(w_{q},\kappa) > 0.
$$
Fix  $p \in (1, \infty)$ , and let  $\hat{w} \coloneqq \hat{w}_p$ . Our goal will be to upper bound the probability  $\mathrm{P}(\hat{w} \neq w^*)$ . By assumption, we have that  $Y = \langle w^*, X \rangle$ , so that  $Y_i = \langle w^*, X_i \rangle$  for all  $i \in [n]$ . Since  $\hat{w}$  minimizes the empirical risk, we must also have  $\langle \hat{w}, X_i \rangle = Y_i = \langle w^*, X_i \rangle$  for all  $i \in [n]$ . Let  $A \in \mathbb{R}^{n \times d}$  denote the matrix whose  $i$ -th row is  $X_i$ . Then we have the following implications.
$$
\hat {w} \neq w ^ {*} \Rightarrow \left\langle \hat {w} - w ^ {*}, X _ {i} \right\rangle = 0 \forall i \in [ n ] \Rightarrow \exists w \in \mathbb {R} ^ {d} \backslash \{0 \} \mid A w = 0 \Leftrightarrow \operatorname {r o w r a n k} (A) <   d. \tag {24}
$$
Let  $r \coloneqq \operatorname{rowrank}(A)$ . We claim the following equivalence
$$
\operatorname {r o w r a n k} (A) <   d \Leftrightarrow \exists S \subset [ n ] \mid | S | = d - 1 \wedge \forall i \in [ n ] \setminus S X _ {i} \in \operatorname {s p a n} \left(\left\{X _ {k} \mid k \in S \right\}\right). \tag {25}
$$
Indeed the implication  $(\Leftarrow)$  follows by definition of the rowrank of  $A$ . For the implication  $(\Rightarrow)$ , by definition,  $\{X_{i} \mid i \in [n]\}$  is a spanning set for the row space of  $A$ , therefore it can be reduced to a basis of it  $\{X_{k} \mid k \in S_{1}\}$  for some indices  $S_{1} \subset [n]$  with  $|S_{1}| = r$ . If  $r = d - 1$ , then the choice  $S = S_{1}$  satisfies the right side of (25). Otherwise, let  $S_{2} \subset [n] \setminus S_{1}$  with  $|S_{2}| = d - 1 - r$ . Such a subset exists since by assumption  $n \geq d > d - 1$ . Then the set  $S := S_{1} \cup S_{2}$  satisfies the right side of (25). Combining (24) and (25) we arrive at:
$$
\begin{array}{l} \mathrm{P}(\hat{w}\neq w^{*})\leq \mathrm{P}\left(\bigcup_{\substack{S\subset [n]\\ |S| = d - 1}}\{\forall i\in [n]\setminus S X_{i}\in \operatorname {span}(\{X_{k}\mid k\in S\})\}\right) \\ \leq \sum_ {\substack {S \subset [ n ] \\ | S | = d - 1}} \mathrm {P} (\forall i \in [ n ] \setminus S X _ {i} \in \operatorname {span} \left(\left\{X _ {k} \mid k \in S \right\}\right)) \tag{26} \\ \end{array}
$$
where the second inequality follows from the union bound. We now bound each of the terms of the sum. Fix  $S = \{i_1, \ldots, i_{d-1}\} \subset [n]$  with  $|S| = d - 1$ . Let  $Z_S = n((X_{i_j})_{j=1}^{d-1})$  be a non-zero vector orthogonal to  $\operatorname{span}(\{X_k \mid k \in S\})$ . Such a vector must exist since  $\dim(\operatorname{span}(\{X_k \mid k \in S\})) < d$ ; see Lemma 8 below for an explicit construction of the function  $n$ . Denote by  $P_{Z_S}$  the distribution of  $Z_S$  and  $P_{(X_i)_{i \in [n] \setminus S}} = \prod_{i=1}^{n-d-1} P_X$  the distribution of  $(X_i)_{i \in [n] \setminus S}$ , where  $P_X$  is the distribution of  $X$ . Note that since  $Z_S$  is a function of  $(X_{i_j})_{j=1}^d$  only, it is independent of  $(X_i)_{i \in [n] \setminus S}$ . In particular, the joint distribution of  $(Z_S, (X_i)_{i \in [n] \setminus S})$  is given by the product  $P_{Z_S} \times P_{(X_i)_{i \in [n] \setminus S}}$ . Now if  $X_i \in \operatorname{span}(\{X_k \mid k \in S\})$ , then by definition of  $Z_S$ ,  $\langle Z_S, X_i \rangle = 0$ . Therefore
$$
\begin{array}{l} \mathrm {P} (\forall i \in [ n ] \backslash S X _ {i} \in \operatorname {s p a n} \left(\left\{X _ {k} \mid k \in S \right\}\right)) \leq \mathrm {P} (\forall i \in [ n ] \backslash S \langle Z _ {S}, X _ {i} \rangle = 0) \\ = \operatorname {E} \left[ \prod_ {i \in [ n ] \backslash S} \mathbb {1} _ {\{0 \}} \left(\left\langle Z _ {S}, X _ {i} \right\rangle\right) \right] \\ = \int \left\{\prod_ {i \in [ n ] \backslash S} \mathbb {1} _ {\{0 \}} (\langle y _ {S}, x _ {i} \rangle) \right\} P _ {Z _ {S}} (d z _ {S}) P _ {(X _ {i}) _ {i \in [ n ] \backslash S}} (d (x _ {i}) _ {i \in [ n ] \backslash S}) \\ = \int P _ {Z _ {S}} (d y _ {s}) \left\{\prod_ {i \in [ n ] \backslash S} \int \mathbb {1} _ {\{0 \}} (\langle y _ {S}, x _ {i} \rangle) P _ {X} (d x _ {i}) \right\} \\ = \int \left\{\prod_ {i \in [ n ] \backslash S} \mathrm {P} \left(\left\langle z _ {S}, X \right\rangle = 0\right) \right\} P _ {Z _ {S}} \left(d y _ {s}\right) \\ = \int \left\{\prod_ {i \in [ n ] \backslash S} \rho_ {0} \left(y _ {s}\right) \right\} P _ {Z _ {S}} \left(d y _ {s}\right) \\ \leq \rho^ {n - d + 1}, \tag {27} \\ \end{array}
$$
where in the third line we used the independence of  $Z_{S}$  and  $(X_{i})_{i\in [n]\backslash S}$ , in the fourth we used the independence of the  $(X_{i})_{i\in \backslash S}$ , in the sixth we used the definition of  $\rho_0$  in 5, and in the last line we used the fact that  $z_{S}\neq 0$  and the definition of  $\rho$  in Lemma 1. Combining the inequalities (26) and (27) yields the result.
Lemma 8. Let  $m \in \{1, \dots, d - 1\}$  and let  $(x_j)_{j=1}^m$  be a sequence of points in  $\mathbb{R}^d$ . Denote by  $A \in \mathbb{R}^{m \times d}$  the matrix whose  $j$ -th row is  $x_j$  and let  $A^+$  be its pseudo-inverse. Let  $(b_i)_{i=1}^d$  be an ordered basis of  $\mathbb{R}^d$ , and define
$$
k := \min  \left\{i \in [ n ] \mid (I - A ^ {+} A) b _ {i} \neq 0 \right\}
$$
$$
n \left(\left(x _ {j}\right) _ {j = 1} ^ {m}\right) := \left(I - A ^ {+} A\right) b _ {k}
$$
Then  $n((x_j)_{j=1}^m)$  is non-zero and is orthogonal to  $\operatorname{span}(\{x_j \mid j \in [m]\})$ .
Proof. We start by showing that  $k$  is well defined. First note that  $I - A^{+}A$  is the orthogonal projector onto the kernel of  $A$ , which is non-trivial since  $\dim(\ker(A)) = d - \dim(\operatorname{Im}(A)) \geq d - m \geq 1$ . Now we claim that there exists an  $i \in [d]$  such that  $(I - A^{+}A)b_{i} \neq 0$ . Suppose not, then for any  $w \in \mathbb{R}^d$ , we have  $(I - A^{+}A)w = (I - A^{+}A)(\sum_{i=1}^{d}c_{i}b_{i}) = \sum_{i=1}^{d}c_{i}(I - A^{+}A)b_{i} = 0$ , implying that  $I - A^{+}A = 0$ , but this contradicts the non-triviality of  $\ker(A)$ . This proves that  $k$  is well-defined, which in turn proves that  $n((x_{j})_{j=1}^{m}) \neq 0$ . It remains to prove the orthogonality claim. Let  $v \in \operatorname{span}(\{x_{j} \mid j \in [m]\})$ . Then there exists coefficients  $c \in \mathbb{R}^{m}$  such that  $v = A^{T}c$ . Therefore
$$
\langle v, n \left(\left(x _ {j}\right) _ {j = 1} ^ {m}\right) \rangle = \left\langle A ^ {T} c, n \left(\left(x _ {j}\right) _ {j = 1} ^ {m}\right) \right\rangle = c ^ {T} A \left(I - A A ^ {+}\right) b _ {k} = 0,
$$
where the last equality holds since  $(I - AA^{+})b_{k}\in \ker (A)$
D Missing proofs for Theorem 1
This section contains the proofs of Lemma 2 and Corollary 1 we used in the proof of Theorem 1.
We compute the expectation:
$$
\begin{array}{l} \operatorname {E} \left[ \| \nabla R _ {p, n} (w _ {p} ^ {*}) \| _ {H _ {p} ^ {- 1}} ^ {2} \right] = \operatorname {E} \left[ \| n ^ {- 1} \nabla \ell_ {p} (\langle w _ {p} ^ {*}, X _ {i} \rangle - Y _ {i}) \| _ {H _ {p} ^ {- 1}} ^ {2} \right] \\ = n ^ {- 2} \sum_ {i = 1} ^ {n} \mathrm {E} \left[ \| \nabla \ell_ {p} \left(\left\langle w _ {p} ^ {*}, X _ {i} \right\rangle - Y _ {i}\right) \| _ {H _ {p} ^ {- 1}} ^ {2} \right] \\ + 2 n ^ {- 2} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {i - 1} \left\langle \mathrm {E} \left[ \nabla \ell_ {p} \left(\left\langle w _ {p} ^ {*}, X _ {i} \right\rangle - Y _ {i}\right) \right], \mathrm {E} \left[ \nabla \ell_ {p} \left(\left\langle w _ {p} ^ {*}, X _ {j} \right\rangle - Y _ {j}\right) \right] \right\rangle_ {H _ {p} ^ {- 1}} \\ = n ^ {- 1} \operatorname {E} \left[ \| \nabla \ell_ {p} (\langle w _ {p} ^ {*}, X \rangle - Y) \| _ {H _ {p} ^ {- 1}} ^ {2} \right] \\ \end{array}
$$
where in the second line we expanded the inner product of the sums into its  $n^2$  terms, used linearity of expectation, and used the independence of the samples to take the expectation inside the inner product. In the last line, we used the fact that the samples are identically distributed to simplify the first term. For the second term, we used the fact that the expectation of the gradient of the loss at the risk minimizer vanishes. Applying Markov's inequality finishes the proof.
D.2 Proof of Corollary 1
$$
\begin{array}{l} \| w - w _ {2} ^ {*} \| _ {H _ {2, n}} ^ {2} = \left(w - w _ {2} ^ {*}\right) ^ {T} H _ {2, n} \left(w - w _ {2} ^ {*}\right) \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} (w - w _ {2} ^ {*}) ^ {T} \nabla^ {2} \ell_ {p} (\langle w _ {2} ^ {*}, X _ {i} \rangle - Y _ {i}) (w - w _ {2} ^ {*}) \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \langle w - w _ {2} ^ {*}, X _ {i} \rangle^ {2}. \\ \end{array}
$$
Now by assumption, the components of the vector  $X$  have finite fourth moment so that applying Proposition 1 and using the condition on  $n$  yields the result.
E Detailed proof of Theorem 4
This section contains the proof of Lemma 3 as well as that of Theorem 4.
By Lemma 2.5 in [AKPS22], we have for all  $t, s \in \mathbb{R}$
$$
\ell_ {p} (t) - \ell_ {p} (s) - \ell_ {p} ^ {\prime} (s) (t - s) \geq \frac {1}{8 (p - 1)} \ell_ {p} ^ {\prime \prime} (s) (t - s) ^ {2}.
$$
Recall that by the chain rule
$$
\nabla \ell_ {p} (\langle w, X \rangle - Y) = \ell_ {p} ^ {\prime} (\langle w, X \rangle - Y) X \qquad \nabla^ {2} \ell_ {p} (\langle w, X \rangle - Y) = \ell_ {p} ^ {\prime \prime} (\langle w, X \rangle - Y) X X ^ {T}.
$$
Replacing  $t$  and  $s$  by  $\langle w,X_i\rangle -Y_i$  and  $\langle w_p^*,X_i\rangle -Y_i$  respectively, and using the formulas for the gradient and Hessian we arrive at
$$
\begin{array}{l} \ell_ {p} (\langle w, X _ {i} \rangle - Y _ {i}) - \ell_ {p} (\langle w _ {p} ^ {*}, X _ {i} \rangle - Y _ {i}) \geq \frac {1}{8 (p - 1)} (w - w _ {p} ^ {*}) ^ {T} \nabla^ {2} \ell_ {p} (\langle w _ {p} ^ {*}, X _ {i} \rangle - Y _ {i}) (w - w _ {p} ^ {*}) \\ + \langle \nabla \ell_ {p} (\langle w _ {p} ^ {*}, X _ {i} \rangle - Y _ {i}), w - w _ {p} ^ {*} \rangle \\ \end{array}
$$
Averaging over  $i \in [n]$  yields the first inequality. The proof of the second inequality proceeds in the same way and uses instead the upper bound of Lemma 2.5 in [AKPS22]. We omit it here.
We proceed similarly to the proof of Theorem 1. By definition of the empirical risk minimizer, we have the upper bound
$$
R _ {p, n} \left(\hat {w} _ {p}\right) - R _ {p, n} \left(w _ {p} ^ {*}\right) \leq 0. \tag {28}
$$
Using (10) from Lemma 3 and the Cauchy-Schwarz inequality, we obtain the pointwise lower bound
$$
R _ {p, n} \left(\hat {w} _ {p}\right) - R _ {p, n} \left(w _ {p} ^ {*}\right) \geq - \left\| \nabla R _ {p, n} \left(w _ {p} ^ {*}\right) \right\| _ {H _ {p} ^ {- 1}} \left\| \hat {w} _ {p} - w _ {p} ^ {*} \right\| _ {H _ {p}} + \frac {1}{8 (p - 1)} \left\| \hat {w} _ {p} - w _ {p} ^ {*} \right\| _ {H _ {p, n}} ^ {2}. \tag {29}
$$
Using Lemma 2 we have that, with probability at least  $1 - \delta /2$