SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Optimal Excess Risk Bounds for Empirical Risk Minimization on  $p$ -Norm Linear Regression
Ayoub El Hanchi  
University of Toronto & Vector Institute  
aelhan@cs.toronto.edu
Murat A. Erdogdu  
University of Toronto & Vector Institute  
erdogdu@cs.toronto.edu
We study the performance of empirical risk minimization on the  $p$ -norm linear regression problem for  $p \in (1, \infty)$ . We show that, in the realizable case, under no moment assumptions, and up to a distribution-dependent constant,  $O(d)$  samples are enough to exactly recover the target. Otherwise, for  $p \in [2, \infty)$ , and under weak moment assumptions on the target and the covariates, we prove a high probability excess risk bound on the empirical risk minimizer whose leading term matches, up to a constant that depends only on  $p$ , the asymptotically exact rate. We extend this result to the case  $p \in (1, 2)$  under mild assumptions that guarantee the existence of the Hessian of the risk at its minimizer.
Real-valued linear prediction is a fundamental problem in machine learning. Traditionally, the square loss has been the default choice for this problem. The performance of empirical risk minimization (ERM) on linear regression under the square loss, as measured by the excess risk, has been studied extensively both from an asymptotic [Whi82; LC83; Vaa98] and a non-asymptotic point of view [AC11; HKZ12; Oli16; LM16; Sau18; Mou22]. An achievement of the last decade has been the development of non-asymptotic excess risk bounds for ERM on this problem under weak assumptions, and which match, up to constant factors, the asymptotically exact rate.
In this paper, we consider the more general family of  $p$ -th power losses  $t \mapsto |t|^p$  for a user-chosen  $p \in (1,\infty)$ . Under mild assumptions, the classical asymptotic theory can still be applied to ERM under these losses, yielding the asymptotic distribution of the excess risk. However, to the best of our knowledge, the problem of deriving non-asymptotic excess risk bounds for ERM for  $p \in (1,\infty) \setminus \{2\}$  remains open, and, as we discuss below, resists the application of standard tools from the literature.
Our motivation for extending the case  $p = 2$  to  $p \in (1, \infty)$  is twofold. Firstly, the freedom in the choice of  $p$  allows us to better capture our prediction goals. For example, we might only care about how accurate our prediction is on average, in which case, the choice  $p = 1$  is appropriate. At the other extreme, we might insist that we do as well as possible on a subset of inputs of probability 1, in which case the choice  $p = \infty$  is best. A choice of  $p \in (1, \infty)$  therefore allows us to interpolate between these two extremes, with the case  $p = 2$  offering a balanced choice. Secondly, different choices of  $p$  have complementary qualities. On the one hand, small values of  $p$  allow us to operate with weak moment assumptions, making them applicable in more general cases. On the other, larger values of  $p$  yield predictions whose optimality is less sensitive to changes in the underlying distribution: for  $p = \infty$ , the best predictor depends only on the support of this distribution.
To sharpen our discussion, let us briefly formalize our problem. There is an input random vector  $X \in \mathbb{R}^d$  and output random variable  $Y \in \mathbb{R}$ , and we are provided with  $n$  i.i.d. samples  $(X_i, Y_i)_{i=1}^n$ . We select our set of predictors to be the class of linear functions  $\{x \mapsto \langle w, x \rangle \mid w \in \mathbb{R}^d\}$ , and choose a value  $p \in (1, \infty)$  with the corresponding loss  $\ell_p(t) \coloneqq |t|^p / [p(p - 1)]$ . Using this loss, we define
the associated risk and empirical risk, respectively, by
$$
R _ {p} (w) := \operatorname {E} \left[ \ell_ {p} (\langle w, X \rangle - Y) \right], \quad R _ {p, n} (w) := \frac {1}{n} \sum_ {i = 1} ^ {n} \ell_ {p} (\langle w, X _ {i} \rangle - Y _ {i}).
$$
We perform empirical risk minimization  $\hat{w}_p\in \mathrm{argmin}_{w\in \mathbb{R}^d}R_{p,n}(w)$ , and our goal is to derive high probability bounds on the excess risk  $R_{p}(\hat{w}_{p}) - R_{p}(w_{p}^{*})$ , where  $w_{p}^{*}$  is the risk minimizer. For efficient algorithms for computing an empirical risk minimizer  $\hat{w}_p$ , we refer the reader to the rich recent literature dealing with this problem [BCLL18; AKPS19; APS19; JLS22].
To see why the problem we are considering is difficult, let us briefly review some of the recent literature. Most closely related to our problem are the results of [AC11; HKZ12; Oli16; LM16], who derive high probability non-asymptotic excess risk bounds for the case  $p = 2$ . The best such bounds are found in Oliveira [Oli16] and Lecué and Mendelson [LM16], who both operate under weak assumptions on  $(X,Y)$ , requiring at most the existence of fourth moments of  $Y$  and the components  $X^j$  of  $X$  for  $j \in [d]$ . Unfortunately, the analysis in Oliveira [Oli16] relies on the closed form expression of the empirical risk minimizer  $\hat{w}_2$ , and therefore cannot be extended to other values of  $p$ . Similarly, the analysis in Lecué and Mendelson [LM16] relies on an exact decomposition of the excess loss  $\ell_2(\langle w,X\rangle -Y) - \ell_2(\langle w_p^*,X\rangle -Y)$  in terms of "quadratic" and "multiplier" components, which also does not extend to other values of  $p$ .
To address these limitations, the work of Mendelson [Men18] extends the ideas of Mendelson [Men14] and Lecué and Mendelson [LM16] to work for loss functions more general than the square loss. Roughly speaking, the main result of Mendelson [Men18] states that as long as the loss is strongly convex and smooth in a neighbourhood of 0, the techniques developed by Mendelson [Men14] can still be applied to obtain high probability excess risk bounds. Unfortunately, the loss functions  $\ell_p(t)$  are particularly ill-behaved in precisely this sense, as  $\ell_p''(t) \to 0$  when  $t \to 0$  for  $p > 2$ , and  $|\ell_p''(t)| \to \infty$  as  $t \to 0$  for  $p \in (1, 2)$ . This makes the analysis of the excess risk of ERM in the case  $p \in (1, \infty) \setminus \{2\}$  particularly challenging using well-established methods.
Contrary to the non-asymptotic regime, the asymptotic properties of the excess risk of ERM under the losses  $\ell_p(t)$  are better understood [Ron84; BRW92; Nie92; Arc96; HS96; LL05], and can be derived from the more general classical asymptotic theory of  $M$ -estimators [LC83; VW96; Vaa98] under mild regularity conditions. In particular, these asymptotic results imply that the excess risk of ERM with  $n$  samples satisfies
$$
\operatorname {E} \left[ R _ {p} \left(\hat {w} _ {p}\right)\right] - R _ {p} \left(w _ {p} ^ {*}\right) = \frac {\operatorname {E} \left[ \| \nabla \ell_ {p} \left(\langle w _ {p} ^ {*} , X \rangle - Y\right) \| _ {H _ {p} ^ {- 1}} ^ {2} \right]}{2 n} + o \left(\frac {1}{n}\right) \quad \text {a s} \quad n \rightarrow \infty , \tag {1}
$$
where  $H_{p} \coloneqq \nabla^{2}R_{p}(w_{p}^{*})$  is the Hessian of the risk at its minimizer. We refer the reader to the discussions in Ostrovskii and Bach [OB21] and Mourtada and Gaiffas [MG22] for more details. As we demonstrate in Theorem 1, the rate of convergence of ERM for the square loss derived in Oliveira [Oli16] and Lecué and Mendelson [LM16] matches the asymptotic rate (1) up to a constant factor. Ideally, we would like our excess risk bounds for the cases  $p \in (1,\infty) \setminus \{2\}$  to also match the asymptotic rate (1), although it is not yet clear how to derive any meaningful such bounds.
In this paper, we prove the first high probability non-asymptotic excess risk bounds for ERM under the  $p$ -th power losses  $\ell_p(t)$  for any  $p \in (1,\infty) \setminus \{2\}$ . Our assumptions on  $(X,Y)$  are weak, arise naturally from the analysis, and reduce to the standard ones for the case  $p = 2$ . Furthermore, the rate we derive matches, up to a constant that depends only on  $p$ , the asymptotically exact rate (1).
We split the analysis in three cases. The first is when the problem is realizable, i.e.  $Y = \langle w^{*}, X \rangle$  for some  $w^{*} \in \mathbb{R}^{d}$ . This edge case is not problematic for the analysis of the case  $p = 2$ , but as discussed above, the  $\ell_{p}(t)$  losses are ill-behaved around 0 for  $p \in (1, \infty) \setminus \{2\}$ , requiring us to treat this case separately. The second case is when the problem is not realizable and  $p \in (2, \infty)$ . The final case is when the problem is not realizable and  $p \in (1, 2)$ , which turns out to be the most technically challenging. In Section 2, we present our main results and in Section 3, we provide their proofs.
Notation. We denote the components of the random vector  $X \in \mathbb{R}^d$  by  $X^j$  for  $j \in [d]$ . We assume the support of  $X$  is not contained in any hyperplane, i.e.  $\mathrm{P}(\langle w, X \rangle = 0) = 1$  only if  $w = 0$ . This is without loss of generality as discussed in Oliveira [Oli16] and Mourada [Mou22]. For a positive semi-definite matrix  $A$ , we denote the bilinear form it induces on  $\mathbb{R}^d$  by  $\langle \cdot, \cdot \rangle_A$ , and define  $\| \cdot \|_A \coloneqq \sqrt{\langle \cdot, \cdot \rangle_A}$ . We define  $H_{p,n} \coloneqq \nabla^2 R_{p,n}(w_p^*)$ .
In this section, we state our main results. We start in Section 2.1 where we introduce constants that help us formulate our theorems. In Section 2.2, we state the best known results for both the case  $p = 2$  and the realizable case where  $Y = \langle w^{*},X\rangle$ . Finally, in Section 2.3, we state our theorems.
2.1 Norm equivalence and small ball constants
To state our results, we will need to define two types of quantities first. The first kind are related to norms and their equivalence constants, which we will use in the analysis of the non-realizable case. The second are small ball probabilities, which we will use in the analysis of the realizable case.
We start by introducing the following functions on our space of coefficients  $\mathbb{R}^d$ . For  $p, q \in [1, \infty)$ , define, with the convention  $\infty^{1/p} \coloneqq \infty$  for all  $p \in [1, \infty)$ ,
$$
\| w \| _ {L ^ {p}} := \operatorname {E} [ | \langle w, X \rangle | ^ {p} ] ^ {1 / p}, \quad \| w \| _ {L ^ {q}, p} := \operatorname {E} [ \| w \| _ {\nabla^ {2} \ell_ {p} \left(\langle w _ {p} ^ {*}, X \rangle - Y\right)} ^ {q} ] ^ {1 / q}. \tag {2}
$$
As suggested by the notation, under appropriate moment assumptions on  $X$ , these functions are indeed norms on  $\mathbb{R}^d$ . In that case, we will be interested in norm equivalence constants between them
$$
C _ {a \rightarrow b} := \sup  _ {w \in \mathbb {R} ^ {d} \backslash \{0 \}} \frac {\| w \| _ {a}}{\| w \| _ {b}}, \quad \sigma_ {p} ^ {2} := C _ {(L ^ {4}, p) \rightarrow (L ^ {2}, p)} ^ {4}, \tag {3}
$$
where  $a$  and  $b$  stand for one of  $L^p$  or  $(L^q,p)$ . Let us note that since we work in a finite dimensional vector space, all norms are equivalent, so that as soon as the quantities defined in (2) are indeed norms, the constants defined in (3) are finite. Furthermore, as suggested by the notation,  $\sigma_p^2$  may be viewed as the maximum second moment of the random variables  $\| w\|_{\nabla^2\ell_p}^2 (\langle w_p^*,X\rangle -Y)$  over the unit sphere of  $\| \cdot \|_{L^2,p}$ . Finally, we record the following identities for future use
$$
\| w \| _ {L ^ {2}, p} = \| w \| _ {H _ {p}}, \quad \| w \| _ {L ^ {q}, 2} = \| w \| _ {L ^ {q}}, \quad \sigma_ {2} ^ {2} = C _ {L ^ {4}, L ^ {2}} ^ {4}. \tag {4}
$$
The first identity holds by linearity, and the second by noticing that  $\nabla^2\ell_2(\langle w,X\rangle -Y) = XX^T$
We now turn to small ball probabilities. We define the following functions on  $\mathbb{R}^d$ , for  $q \in [1, \infty)$
$$
\rho_ {0} (w) := \mathrm {P} (\langle w, X \rangle = 0), \quad \rho_ {q} (w, \kappa) := \mathrm {P} (| \langle w, X \rangle | > \kappa \| w \| _ {L ^ {q}}). \tag {5}
$$
Assumptions on the functions  $\rho_0$  and  $\rho_{2}$  have been used extensively in the recent literature, see e.g. [Men14; KM15; LM17a; LM17b; Men18; LM18; Mou22]. In particular, a standard assumption postulates the existence of strictly positive constants  $\beta_0$ , and  $(\beta_{2},\kappa_{2})$  such that  $\rho_0(w)\leq 1 - \beta_0$  and  $\rho_{2}(w,\kappa_{2})\geq \beta_{2}$  for all  $w\in \mathbb{R}^d$ . Conditions of this type are usually referred to as small ball conditions. Efforts have been made to understand when these conditions hold [Men14; RV15; LM17b] as well as reveal the dimension dependence of the constants with which they do [Sau18]. Here we prove that such conditions always hold for finite dimensional spaces. We leave the proof of Lemma 1 to Appendix B to not distract from our main development.
Lemma 1.  $\rho_0$  is upper semi-continuous. Furthermore, if for some  $q\in [1,\infty)$ ,  $\operatorname{E}[|X^j |^q ] < \infty$  for all  $j\in [d]$ , then  $\rho_q(\cdot ,\kappa)$  is lower semi-continuous for any  $\kappa \geq 0$ . Moreover, for all  $\kappa \in [0,1)$
$$
\rho := \sup  _ {w \in \mathbb {R} ^ {d} \setminus \{0 \}} \rho_ {0} (w) <   1, \qquad \qquad \inf  _ {w \in \mathbb {R} ^ {d} \setminus \{0 \}} \rho_ {q} (w, \kappa) > 0.
$$
2.2 Background
To better contextualize our results, we start by stating the best known high probability bound on ERM for the square loss, which we deduce from Oliveira [Oli16] and Lecué and Mendelson [LM16].
Theorem 1 (Theorem 4.2, Oliveira [Oli16]; Theorem 1.3, Lecue and Mendelson [LM16]). Assume that  $\operatorname{E}[Y^2] < \infty$  and  $\operatorname{E}[(X^j)^4] < \infty$  for all  $j \in [d]$ , and let  $\delta \in (0,1]$ . If
$$
n \geq 1 9 6 \sigma_ {2} ^ {2} (d + 2 \log (4 / \delta)),
$$
then, with probability at least  $1 - \delta$
$$
R _ {2} (\hat {w} _ {2}) - R _ {2} (w _ {2} ^ {*}) \leq \frac {1 6 \operatorname {E} [ \| \nabla \ell_ {2} (\langle w _ {2} ^ {*} , X \rangle - Y) \| _ {H _ {2} ^ {- 1}} ^ {2} ]}{n \delta}.
$$
Up to a constant factor and the dependence on  $\delta$ , Theorem 1 recovers the asymptotically exact rate (1). Let us briefly comment on the differences between Theorem 1 and the comparable statements in the original papers. First, the finiteness of  $\sigma_2^2$  is deduced from the finiteness of the fourth moments of the components of  $X$ , instead of being assumed as in Oliveira [Oli16] (see the discussion in Section 3.1 in Oliveira [Oli16]). Second we combine Theorem 3.1 from [Oli16] with the proof technique of Lecué and Mendelson [LM16] to achieve a slightly better bound that the one achieved by the proof technique used in the proof of Theorem 4.2 in Oliveira [Oli16], while avoiding the dependence on the small ball-constant present in the bound of Theorem 1.3 in Lecué and Mendelson [LM16], which is known to incur additional dimension dependence in some cases [Sau18].
We now move to the realizable case, where  $Y = \langle w^{*}, X \rangle$  so that  $w_{p}^{*} = w^{*}$  for all  $p \in (1, \infty)$ . We immediately note that Theorem 1 is still applicable in this case, and ensures that we recover  $w^{*}$  exactly with no more than  $n = O(\sigma_2^2 d)$  samples. However, we can do much better, while getting rid of all the moment assumptions in Theorem 1. Indeed, it is not hard to see that  $\hat{w}_{p} \neq w^{*}$  only if for some  $w \in \mathbb{R}^d \setminus \{0\}$ ,  $\langle w, X_i \rangle = 0$  for all  $i \in [n]$  (taking  $w = \hat{w}_p - w_p^*$  works). The implicit argument in Theorem 1 then uses the pointwise bound (see Lemma B.2 in Oliveira [Oli16])
$$
\mathrm {P} (\cap_ {i = 1} ^ {n} \{\langle w, X _ {i} \rangle = 0 \}) \leq \exp \left(- \frac {n}{2 \sigma_ {2} ^ {2}}\right),
$$
and uniformizes it over the  $L^2$  unit sphere in  $\mathbb{R}^d$ , where the  $L^2$  norm is as defined in (2). However, we can use the much tighter bound  $\rho^n$  where  $\rho$  is as defined in Lemma 1. To the best of our knowledge, the realizable case has not been studied explicitly before in the literature. However, with the above considerations in mind, we can deduce the following result from Lecué and Mendelson [LM17b], which uniformizes the pointwise bound we just discussed using a VC dimension argument.
Theorem 2 (Corollary 2.5, Lecué and Mendelson [LM17b]). Assume that there exists  $w^{*} \in \mathbb{R}^{d}$  such that  $Y = \langle w^{*}, X \rangle$ . Let  $\delta \in (0,1]$ . If
$$
n \geq O \left(\frac {d + \log (1 / \delta)}{(1 - \rho) ^ {2}}\right)
$$
then for any  $p\in (1,\infty)$ $\hat{w}_p = w^*$  with probability at least  $1 - \delta$
We are now in position to state our main results. As discussed in Section 1, the  $\ell_p(t)$  losses have degenerate second derivatives as  $t\to 0$ . When the problem is realizable, the risk is not twice differentiable at its minimizer for the cases  $p\in (1,2)$ , and is degenerate for the cases  $p\in (2,\infty)$ . If we want bounds of the form (1), we must exclude this case from our analysis. Our first main result is a strengthening of Theorem 2, and relies on a combinatorial argument to uniformize the pointwise estimate discussed in Section 2.2.
Theorem 3. Assume that there exists  $w^{*}\in \mathbb{R}^{d}$  such that  $\langle w^{*},X\rangle = Y$ . Then for all  $n\geq d$ , and for all  $p\in (1,\infty)$ , we have
$$
\mathrm {P} (\hat {w} _ {p} \neq w ^ {*}) \leq \left( \begin{array}{c} n \\ d - 1 \end{array} \right) \rho^ {n - d + 1},
$$
where  $\rho$  is as defined in Lemma 1. Furthermore, if
$$
n \geq \left\{ \begin{array}{l l} O \left(d + \log (1 / \delta) / \log (1 / \rho)\right) & \text {i f} 0 \leq \rho <   e ^ {- 1}, \\ O \left(\frac {d + \log (1 / \delta)}{1 - \rho}\right) & \text {i f} e ^ {- 1} \leq \rho <   e ^ {- 1 / e}, \\ O \left(\frac {d \log (1 / (1 - \rho)) + \log (1 / \delta)}{1 - \rho}\right) & \text {i f} e ^ {- 1 / e} \leq \rho <   1, \end{array} \right.
$$
then with probability at least  $1 - \delta$ ,  $\hat{w}_p = w^*$ .
Comparing Theorem 2 and Theorem 3, we see that the bound on the number of samples required to reach a confidence level  $\delta$  in Theorem 3 is uniformly smaller than the one in Theorem 2. The proof of Theorem 3 can be found in Appendix C.
We now move to the more common non-realizable case. Our first theorem here gives a non-asymptotic bound for the excess risk of ERM under a  $p$ -th power loss for  $p \in (2, \infty)$ . To the best of our knowledge, no such result is known in the literature.
Theorem 4. Let  $p \in (2, \infty)$  and  $\delta \in (0, 1]$ . Assume that no  $w \in \mathbb{R}^d$  satisfies  $Y = \langle w, X \rangle$ . Further, assume that  $\operatorname{E}[|Y|^p] < \infty$ ,  $\operatorname{E}[|X^j|^p] < \infty$ , and  $\operatorname{E}[|\langle w_p^*, X \rangle - Y|^{2(p-2)}(X^j)^4] < \infty$  for all  $j \in [d]$ . If
$$
n \geq 1 9 6 \sigma_ {p} ^ {2} (d + 2 \log (4 / \delta)),
$$
$$
\begin{array}{l} R _ {p} \left(\hat {w} _ {p}\right) - R _ {p} \left(w _ {p} ^ {*}\right) \leq \frac {2 0 4 8 p ^ {2} \operatorname {E} \left[ \| \nabla \ell_ {p} \left(\left\langle w _ {p} ^ {*} , X \right\rangle - Y\right) \| _ {H _ {p} ^ {- 1}} ^ {2} \right]}{n \delta} \\ + \left(\frac {5 1 2 p ^ {4} c _ {p} ^ {2} \operatorname {E} \left[ \| \nabla \ell_ {p} (\langle w _ {p} ^ {*} , X \rangle - Y) \| _ {H _ {p} ^ {- 1}} ^ {2} \right]}{n \delta}\right) ^ {p / 2}, \\ \end{array}
$$
where we used  $c_{p}$  to denote  $C_{L^p\to (L^2,p)}$  as defined in (3).
Up to a constant factor that depends only on  $p$  and the dependence on  $\delta$ , the bound of Theorem 4 is precisely of the form of the optimal bound (1). Indeed, as  $p > 2$ , the second term is  $o(1/n)$ . At the level of assumptions, the finiteness of the  $p$ -th moment of  $Y$  and the components of  $X$  is necessary to ensure that the risk  $R_p$  is finite for all  $w \in \mathbb{R}^d$ . The last assumption  $\operatorname{E}[|Y - \langle w_p^*, X \rangle|^{2(p-2)}(X^j)^4] < \infty$  is a natural extension of the fourth moment assumption in Theorem 1. In fact, all three assumptions in Theorem 4 reduce to those of Theorem 1 as  $p \to 2$ . It is worth noting that the constant  $c_p$  has the alternative expression  $\sup_{w \in \mathbb{R}^d \setminus \{0\}} \left\{ \| w \|_{L^p} / \| w \|_{H_p} \right\}$  by (4), i.e., it is the norm equivalence constant between the  $L^p$  norm and the norm induced by  $H_p$ . Using again (4), we see that  $c_p \to 1$  as  $p \to 2$ . As  $p \to \infty$ ,  $c_p$  grows, and we suspect in a dimension-dependent way. However, this does not affect the asymptotic optimality of our rate as  $c_p$  only enters an  $o(1/n)$  term in our bound.
We now turn to the case of  $p \in (1, 2)$  where we need a slightly stronger version of non-realizability.
Theorem 5. Let  $p \in (1,2)$  and  $\delta \in (0,1]$ . Assume that  $\mathrm{P}(|\langle w_p^*,X\rangle -Y|^{2 - p} > 0) = 1$  and  $\operatorname {E}[|\langle w_p^*,X\rangle -Y|^{2(p - 2)}] < \infty$ . Further, assume that  $\operatorname {E}[|Y|^p ] < \infty$ ,  $\operatorname {E}[(X^j)^2 ] < \infty$ ,  $\operatorname {E}[|\langle w_p^*,X\rangle - Y|^{2(p - 2)}(X^j)^4 ] < \infty$  for all  $j\in [d]$ . If
$$
\begin{array}{l} R _ {p} (\hat {w} _ {p}) - R _ {p} (w _ {p} ^ {*}) \leq \frac {3 2 7 6 8}{p - 1} \frac {\operatorname {E} \left[ \| \nabla \ell_ {p} (\langle w _ {p} ^ {*} , X \rangle - Y) \| _ {H _ {p} ^ {- 1}} ^ {2} \right]}{n \delta} \\ + \frac {1}{p - 1} \left(\frac {2 0 9 7 1 5 2 \mathrm {E} \left[ \| \nabla \ell_ {p} (\langle w _ {p} ^ {*} , X \rangle - Y) \| _ {H _ {p} ^ {- 1}} ^ {2} \right] \sigma_ {p} ^ {6 - 2 p} c _ {p} ^ {2 - p} c _ {p} ^ {*}}{n \delta}\right) ^ {1 / (p - 1)} \\ \end{array}
$$
where we used  $c_p^*$  to denote  $\operatorname{E}[|Y - \langle w_p^*, X \rangle|^{2(p-2)}]$  and  $c_p \coloneqq \operatorname{Tr}\big(H_p^{-1}\Sigma\big)$  where  $\Sigma \coloneqq \operatorname{E}\big[XX^T\big]$ .
Just like the bounds of Theorems 1 and 4, the bound of Theorem 5 is asymptotically optimal up to a constant factor that depends only on  $p$ . Indeed, since  $1 < p < 2$ ,  $1 / (p - 1) > 1$ , and the second term is  $o(1 / n)$ . At the level of assumptions, we have two additional conditions compared to Theorem 4. First, we require the existence of the second moment of the covariates instead of just the  $p$ -th moment. Second, we require a stronger version of non-realizability by assuming the existence of the  $2(2 - p)$  negative moment of  $|\langle w_p^*, X \rangle - Y|$ . In the majority of applications, an intercept variable is included as a covariate, i.e.  $X^1 = 1$ , so that this negative moment assumption is already implied by the standard assumption  $\operatorname{E}[|Y - \langle w_p^*, X \rangle|^{2(p - 2)}(X^j)^4] < \infty$ . In the rare case where an intercept variable is not included, any negative moment assumption on  $|\langle w_p^*, X \rangle - Y|$  can be used instead, at the cost of a larger factor in the  $o(1 / n)$  term.
Finally, it is worth noting that for the cases  $p \in [1, 2)$ , there are situations where the asymptotic bound (1) does not hold, as the limiting distribution of the coefficients  $\hat{w}_p$  as  $n \to \infty$  does not necessarily converge to a Gaussian, and depends heavily on the distribution of  $\langle w_p^*, X \rangle - Y$ , see e.g. Lai and Lee [LL05] and Knight [Kni98]. Overall, we suspect that perhaps a slightly weaker version of our assumptions is necessary for a fast rate like (1) to hold.
3 Proofs
3.1 Proof of Theorem 1
Here we give a detailed proof of Theorem 1. While the core technical result can be deduced by combining results from Oliveira [Oli16] and Lecué and Mendelson [LM16], here we frame the proof in a way that makes it easy to extend to the cases  $p \in (1, \infty)$ , and differently from either paper. We split the proof in three steps. First notice that since the loss is a quadratic function of  $w$ , we can express it exactly using a second order Taylor expansion around the minimizer  $w_2^*$
$$
\ell_ {2} (\langle w, X \rangle - Y) - \ell_ {2} (\langle w _ {2} ^ {*}, X \rangle - Y) = \langle \nabla \ell_ {2} (\langle w _ {2} ^ {*}, X \rangle - Y), w - w _ {2} ^ {*} \rangle + \frac {1}{2} \| w - w _ {2} ^ {*} \| _ {\nabla^ {2} \ell_ {2} (\langle w _ {2} ^ {*}, X \rangle - Y)} ^ {2}.
$$