Title: Optimal Excess Risk Bounds for Empirical Risk Minimization on $p$-Norm Linear Regression
Venue: NeurIPS 2023
Accepted: yes
Keywords: Excess risk bounds

Linear regression

Lp-norm

Fast rates
Abstract: We study the performance of empirical risk minimization on the $p$-norm linear regression problem for $p \in (1, \infty)$. We show that, in the realizable case, under no moment assumptions, and up to a distribution-dependent constant, $O(d)$ samples are enough to exactly recover the target. Otherwise, for $p \in [2, \infty)$, and under weak moment assumptions on the target and the covariates, we prove a high probability excess risk bound on the empirical risk minimizer whose leading term matches, up to a constant that depends only on $p$, the asymptotically exact rate. We extend this result to the case $p \in (1, 2)$ under mild assumptions that guarantee the existence of the Hessian of the risk at its minimizer.

Optimal Excess Risk Bounds for Empirical Risk Minimization on p -Norm Linear Regression

Abstract

We study the performance of empirical risk minimization on the p -norm linear regression problem for p ∈ (1 , ∞ ) . We show that, in the realizable case, under no moment assumptions, and up to a distribution-dependent constant, O ( d ) samples are enough to exactly recover the target. Otherwise, for p ∈ [2 , ∞ ) , and under weak moment assumptions on the target and the covariates, we prove a high probability excess risk bound on the empirical risk minimizer whose leading term matches, up to a constant that depends only on p , the asymptotically exact rate. We extend this result to the case p ∈ (1 , 2) under mild assumptions that guarantee the existence of the Hessian of the risk at its minimizer.

1 Introduction

Real-valued linear prediction is a fundamental problem in machine learning. Traditionally, the square loss has been the default choice for this problem. The performance of empirical risk minimization (ERM) on linear regression under the square loss, as measured by the excess risk, has been studied extensively both from an asymptotic [Whi82; LC83; Vaa98] and a non-asymptotic point of view [AC11; HKZ12; Oli16; LM16; Sau18; Mou22]. An achievement of the last decade has been the development of non-asymptotic excess risk bounds for ERM on this problem under weak assumptions, and which match, up to constant factors, the asymptotically exact rate.

In this paper, we consider the more general family of p -th power losses t ↦→| t | p for a user-chosen p ∈ (1 , ∞ ) . Under mild assumptions, the classical asymptotic theory can still be applied to ERM under these losses, yielding the asymptotic distribution of the excess risk. However, to the best of our knowledge, the problem of deriving non-asymptotic excess risk bounds for ERM for p ∈ (1 , ∞ ) \{ 2 } remains open, and, as we discuss below, resists the application of standard tools from the literature.

Our motivation for extending the case p = 2 to p ∈ (1 , ∞ ) is twofold. Firstly, the freedom in the choice of p allows us to better capture our prediction goals. For example, we might only care about how accurate our prediction is on average, in which case, the choice p = 1 is appropriate. At the other extreme, we might insist that we do as well as possible on a subset of inputs of probability 1 , in which case the choice p = ∞ is best. A choice of p ∈ (1 , ∞ ) therefore allows us to interpolate between these two extremes, with the case p = 2 offering a balanced choice. Secondly, different choices of p have complementary qualities. On the one hand, small values of p allow us to operate with weak moment assumptions, making them applicable in more general cases. On the other, larger values of p yield predictions whose optimality is less sensitive to changes in the underlying distribution: for p = ∞ , the best predictor depends only on the support of this distribution.

To sharpen our discussion, let us briefly formalize our problem. There is an input random vector X ∈ R d and output random variable Y ∈ R , and we are provided with n i.i.d. samples ( X i , Y i ) n i =1 . We select our set of predictors to be the class of linear functions { x ↦→⟨ w,x ⟩ | w ∈ R d } , and choose a value p ∈ (1 , ∞ ) with the corresponding loss ℓ p ( t ) := | t | p / [ p ( p -1)] . Using this loss, we define

the associated risk and empirical risk, respectively, by

We perform empirical risk minimization ˆ w p ∈ argmin w ∈ R d R p,n ( w ) , and our goal is to derive high probability bounds on the excess risk R p ( ˆ w p ) -R p ( w ∗ p ) , where w ∗ p is the risk minimizer. For efficient algorithms for computing an empirical risk minimizer ˆ w p , we refer the reader to the rich recent literature dealing with this problem [BCLL18; AKPS19; APS19; JLS22].

To see why the problem we are considering is difficult, let us briefly review some of the recent literature. Most closely related to our problem are the results of [AC11; HKZ12; Oli16; LM16], who derive high probability non-asymptotic excess risk bounds for the case p = 2 . The best such bounds are found in Oliveira [Oli16] and Lecué and Mendelson [LM16], who both operate under weak assumptions on ( X,Y ) , requiring at most the existence of fourth moments of Y and the components X j of X for j ∈ [ d ] . Unfortunately, the analysis in Oliveira [Oli16] relies on the closed form expression of the empirical risk minimizer ˆ w 2 , and therefore cannot be extended to other values of p . Similarly, the analysis in Lecué and Mendelson [LM16] relies on an exact decomposition of the excess loss ℓ 2 ( ⟨ w,X ⟩ -Y ) -ℓ 2 ( ⟨ w ∗ p , X ⟩ -Y ) in terms of 'quadratic' and 'multiplier' components, which also does not extend to other values of p .

To address these limitations, the work of Mendelson [Men18] extends the ideas of Mendelson [Men14] and Lecué and Mendelson [LM16] to work for loss functions more general than the square loss. Roughly speaking, the main result of Mendelson [Men18] states that as long as the loss is strongly convex and smooth in a neighbourhood of 0 , the techniques developed by Mendelson [Men14] can still be applied to obtain high probability excess risk bounds. Unfortunately, the loss functions ℓ p ( t ) are particularly ill-behaved in precisely this sense, as ℓ ′′ p ( t ) → 0 when t → 0 for p > 2 , and | ℓ ′′ p ( t ) |→∞ as t → 0 for p ∈ (1 , 2) . This makes the analysis of the excess risk of ERM in the case p ∈ (1 , ∞ ) \ { 2 } particularly challenging using well-established methods.

Contrary to the non-asymptotic regime, the asymptotic properties of the excess risk of ERM under the losses ℓ p ( t ) are better understood [Ron84; BRW92; Nie92; Arc96; HS96; LL05], and can be derived from the more general classical asymptotic theory of M -estimators [LC83; VW96; Vaa98] under mild regularity conditions. In particular, these asymptotic results imply that the excess risk of ERM with n samples satisfies

where H p := ∇ 2 R p ( w ∗ p ) is the Hessian of the risk at its minimizer. We refer the reader to the discussions in Ostrovskii and Bach [OB21] and Mourtada and Gaïffas [MG22] for more details. As we demonstrate in Theorem 1, the rate of convergence of ERM for the square loss derived in Oliveira [Oli16] and Lecué and Mendelson [LM16] matches the asymptotic rate (1) up to a constant factor. Ideally, we would like our excess risk bounds for the cases p ∈ (1 , ∞ ) \ { 2 } to also match the asymptotic rate (1), although it is not yet clear how to derive any meaningful such bounds.

In this paper, we prove the first high probability non-asymptotic excess risk bounds for ERM under the p -th power losses ℓ p ( t ) for any p ∈ (1 , ∞ ) \ { 2 } . Our assumptions on ( X,Y ) are weak, arise naturally from the analysis, and reduce to the standard ones for the case p = 2 . Furthermore, the rate we derive matches, up to a constant that depends only on p , the asymptotically exact rate (1).

We split the analysis in three cases. The first is when the problem is realizable, i.e. Y = ⟨ w ∗ , X ⟩ for some w ∗ ∈ R d . This edge case is not problematic for the analysis of the case p = 2 , but as discussed above, the ℓ p ( t ) losses are ill-behaved around 0 for p ∈ (1 , ∞ ) \ { 2 } , requiring us to treat this case separately. The second case is when the problem is not realizable and p ∈ (2 , ∞ ) . The final case is when the problem is not realizable and p ∈ (1 , 2) , which turns out to be the most technically challenging. In Section 2, we present our main results and in Section 3, we provide their proofs.

Notation. We denote the components of the random vector X ∈ R d by X j for j ∈ [ d ] . We assume the support of X is not contained in any hyperplane, i.e. P( ⟨ w,X ⟩ = 0) = 1 only if w = 0 . This is without loss of generality as discussed in Oliveira [Oli16] and Mourtada [Mou22]. For a positive semi-definite matrix A , we denote the bilinear form it induces on R d by ⟨· , ·⟩ A , and define ∥·∥ A := √ ⟨· , ·⟩ A . We define H p,n := ∇ 2 R p,n ( w ∗ p ) .

2 Main results

In this section, we state our main results. We start in Section 2.1 where we introduce constants that help us formulate our theorems. In Section 2.2, we state the best known results for both the case p = 2 and the realizable case where Y = ⟨ w ∗ , X ⟩ . Finally, in Section 2.3, we state our theorems.

2.1 Norm equivalence and small ball constants

To state our results, we will need to define two types of quantities first. The first kind are related to norms and their equivalence constants, which we will use in the analysis of the non-realizable case. The second are small ball probabilities, which we will use in the analysis of the realizable case.

We start by introducing the following functions on our space of coefficients R d . For p, q ∈ [1 , ∞ ) , define, with the convention ∞ 1 /p := ∞ for all p ∈ [1 , ∞ ) ,

As suggested by the notation, under appropriate moment assumptions on X , these functions are indeed norms on R d . In that case, we will be interested in norm equivalence constants between them

where a and b stand for one of L p or ( L q , p ) . Let us note that since we work in a finite dimensional vector space, all norms are equivalent, so that as soon as the quantities defined in (2) are indeed norms, the constants defined in (3) are finite. Furthermore, as suggested by the notation, σ 2 p may be viewed as the maximum second moment of the random variables ∥ w ∥ 2 ∇ 2 ℓ p ( ⟨ w ∗ p ,X ⟩-Y ) over the unit sphere of ∥·∥ L 2 ,p . Finally, we record the following identities for future use

The first identity holds by linearity, and the second by noticing that ∇ 2 ℓ 2 ( ⟨ w,X ⟩ -Y ) = XX T .

We now turn to small ball probabilities. We define the following functions on R d , for q ∈ [1 , ∞ ) ,

Assumptions on the functions ρ 0 and ρ 2 have been used extensively in the recent literature, see e.g. [Men14; KM15; LM17a; LM17b; Men18; LM18; Mou22]. In particular, a standard assumption postulates the existence of strictly positive constants β 0 , and ( β 2 , κ 2 ) such that ρ 0 ( w ) ≤ 1 -β 0 and ρ 2 ( w,κ 2 ) ≥ β 2 for all w ∈ R d . Conditions of this type are usually referred to as small ball conditions. Efforts have been made to understand when these conditions hold [Men14; RV15; LM17b] as well as reveal the dimension dependence of the constants with which they do [Sau18]. Here we prove that such conditions always hold for finite dimensional spaces. We leave the proof of Lemma 1 to Appendix B to not distract from our main development.

Lemma 1. ρ 0 is upper semi-continuous. Furthermore, if for some q ∈ [1 , ∞ ) , E[ | X j | q ] < ∞ for all j ∈ [ d ] , then ρ q ( · , κ ) is lower semi-continuous for any κ ≥ 0 . Moreover, for all κ ∈ [0 , 1)

2.2 Background

To better contextualize our results, we start by stating the best known high probability bound on ERM for the square loss, which we deduce from Oliveira [Oli16] and Lecué and Mendelson [LM16].

Theorem 1 (Theorem 4.2, Oliveira [Oli16]; Theorem 1.3, Lecué and Mendelson [LM16]) . Assume that E[ Y 2 ] < ∞ and E[( X j ) 4 ] < ∞ for all j ∈ [ d ] , and let δ ∈ (0 , 1] . If

then, with probability at least 1 -δ

Up to a constant factor and the dependence on δ , Theorem 1 recovers the asymptotically exact rate (1). Let us briefly comment on the differences between Theorem 1 and the comparable statements in the original papers. First, the finiteness of σ 2 2 is deduced from the finiteness of the fourth moments of the components of X , instead of being assumed as in Oliveira [Oli16] (see the discussion in Section 3.1 in Oliveira [Oli16]). Second we combine Theorem 3.1 from [Oli16] with the proof technique of Lecué and Mendelson [LM16] to achieve a slightly better bound that the one achieved by the proof technique used in the proof of Theorem 4.2 in Oliveira [Oli16], while avoiding the dependence on the small ball-constant present in the bound of Theorem 1.3 in Lecué and Mendelson [LM16], which is known to incur additional dimension dependence in some cases [Sau18].

̸

We now move to the realizable case, where Y = ⟨ w ∗ , X ⟩ so that w ∗ p = w ∗ for all p ∈ (1 , ∞ ) . We immediately note that Theorem 1 is still applicable in this case, and ensures that we recover w ∗ exactly with no more than n = O ( σ 2 2 d ) samples. However, we can do much better, while getting rid of all the moment assumptions in Theorem 1. Indeed, it is not hard to see that ˆ w p = w ∗ only if for some w ∈ R d \ { 0 } , ⟨ w,X i ⟩ = 0 for all i ∈ [ n ] (taking w = ˆ w p -w ∗ p works). The implicit argument in Theorem 1 then uses the pointwise bound (see Lemma B.2 in Oliveira [Oli16])

and uniformizes it over the L 2 unit sphere in R d , where the L 2 norm is as defined in (2). However, we can use the much tighter bound ρ n where ρ is as defined in Lemma 1. To the best of our knowledge, the realizable case has not been studied explicitly before in the literature. However, with the above considerations in mind, we can deduce the following result from Lecué and Mendelson [LM17b], which uniformizes the pointwise bound we just discussed using a VC dimension argument.

Theorem 2 (Corollary 2.5, Lecué and Mendelson [LM17b]) . Assume that there exists w ∗ ∈ R d such that Y = ⟨ w ∗ , X ⟩ . Let δ ∈ (0 , 1] . If

then for any p ∈ (1 , ∞ ) , ˆ w p = w ∗ with probability at least 1 -δ .

2.3 Results

We are now in position to state our main results. As discussed in Section 1, the ℓ p ( t ) losses have degenerate second derivatives as t → 0 . When the problem is realizable, the risk is not twice differentiable at its minimizer for the cases p ∈ (1 , 2) , and is degenerate for the cases p ∈ (2 , ∞ ) . If we want bounds of the form (1), we must exclude this case from our analysis. Our first main result is a strengthening of Theorem 2, and relies on a combinatorial argument to uniformize the pointwise estimate discussed in Section 2.2.

Theorem 3. Assume that there exists w ∗ ∈ R d such that ⟨ w ∗ , X ⟩ = Y . Then for all n ≥ d , and for all p ∈ (1 , ∞ ) , we have

̸

where ρ is as defined in Lemma 1. Furthermore, if

then with probability at least 1 -δ , ˆ w p = w ∗ .

Comparing Theorem 2 and Theorem 3, we see that the bound on the number of samples required to reach a confidence level δ in Theorem 3 is uniformly smaller than the one in Theorem 2. The proof of Theorem 3 can be found in Appendix C.

We now move to the more common non-realizable case. Our first theorem here gives a non-asymptotic bound for the excess risk of ERM under a p -th power loss for p ∈ (2 , ∞ ) . To the best of our knowledge, no such result is known in the literature.

Theorem 4. Let p ∈ (2 , ∞ ) and δ ∈ (0 , 1] . Assume that no w ∈ R d satisfies Y = ⟨ w,X ⟩ . Further, assume that E[ | Y | p ] < ∞ , E[ | X j | p ] < ∞ , and E[ |⟨ w ∗ p , X ⟩ -Y | 2( p -2) ( X j ) 4 ] < ∞ for all j ∈ [ d ] . If

then with probability at least 1 -δ

where we used c p to denote C L p → ( L 2 ,p ) as defined in (3).

Up to a constant factor that depends only on p and the dependence on δ , the bound of Theorem 4 is precisely of the form of the optimal bound (1). Indeed, as p > 2 , the second term is o (1 /n ) . At the level of assumptions, the finiteness of the p -th moment of Y and the components of X is necessary to ensure that the risk R p is finite for all w ∈ R d . The last assumption E[ | Y -⟨ w ∗ p , X ⟩| 2( p -2) ( X j ) 4 ] < ∞ is a natural extension of the fourth moment assumption in Theorem 1. In fact, all three assumptions in Theorem 4 reduce to those of Theorem 1 as p → 2 . It is worth noting that the constant c p has the alternative expression sup w ∈ R d \{ 0 } {∥ w ∥ L p / ∥ w ∥ H p } by (4), i.e. it is the norm equivalence constant between the L p norm and the norm induced by H p . Using again (4), we see that c p → 1 as p → 2 . As p →∞ , c p grows, and we suspect in a dimension dependent way. However, this does not affect the asymptotic optimality of our rate as c p only enters an o (1 /n ) term in our bound.

We now turn to the case of p ∈ (1 , 2) where we need a slightly stronger version of non-realizability. Theorem 5. Let p ∈ (1 , 2) and δ ∈ (0 , 1] . Assume that P( |⟨ w ∗ p , X ⟩ -Y | 2 -p > 0) = 1 and E[ |⟨ w ∗ p , X ⟩ -Y | 2( p -2) ] < ∞ . Further, assume that E[ | Y | p ] < ∞ , E[( X j ) 2 ] < ∞ , E[ |⟨ w ∗ p , X ⟩ -Y | 2( p -2) ( X j ) 4 ] < ∞ for all j ∈ [ d ] . If

then, with probability at least 1 -δ

Just like the bounds of Theorems 1 and 4, the bound of Theorem 5 is asymptotically optimal up to a constant factor that depends only on p . Indeed, since 1 < p < 2 , 1 / ( p -1) > 1 , and the second term is o (1 /n ) . At the level of assumptions, we have two additional conditions compared to Theorem 4. First, we require the existence of the second moment of the covariates instead of just the p -th moment. Second, we require a stronger version of non-realizability by assuming the existence of the 2(2 -p ) negative moment of |⟨ w ∗ p , X ⟩ -Y | . In the majority of applications, an intercept variable is included as a covariate, i.e. X 1 = 1 , so that this negative moment assumption is already implied by the standard assumption E[ | Y -⟨ w ∗ p , X ⟩| 2( p -2) ( X j ) 4 ] < ∞ . In the rare case where an intercept variable is not included, any negative moment assumption on |⟨ w ∗ p , X ⟩ -Y | can be used instead, at the cost of a larger factor in the o (1 /n ) term.

Finally, it is worth noting that for the cases p ∈ [1 , 2) , there are situations where the asymptotic bound (1) does not hold, as the limiting distribution of the coefficients ˆ w p as n →∞ does not necessarily converge to a Gaussian, and depends heavily on the distribution of ⟨ w ∗ p , X ⟩ -Y , see e.g. Lai and Lee [LL05] and Knight [Kni98]. Overall, we suspect that perhaps a slightly weaker version of our assumptions is necessary for a fast rate like (1) to hold.

3 Proofs

3.1 Proof of Theorem 1

Here we give a detailed proof of Theorem 1. While the core technical result can be deduced by combining results from Oliveira [Oli16] and Lecué and Mendelson [LM16], here we frame the proof in a way that makes it easy to extend to the cases p ∈ (1 , ∞ ) , and differently from either paper. We split the proof in three steps. First notice that since the loss is a quadratic function of w , we can express it exactly using a second order Taylor expansion around the minimizer w ∗ 2

Taking empirical averages and expectations of both sides respectively shows that the excess empirical risk and excess risk also admit such an expansion

where in the second equality we used that the gradient of the risk vanishes at the minimizer w ∗ 2 . Therefore, to bound the excess risk, it is sufficient to bound the norm ∥ w -w ∗ 2 ∥ H 2 . This is the goal of the second step, where we use two ideas. First, by definition, the excess empirical risk of the empirical risk minimizer satisfies the upper bound

Second, we use the Cauchy-Schwartz inequality to lower bound the excess empirical risk by

and we further lower bound it by deriving high probability bounds on the two random terms ∥∇ R 2 ,n ( w ∗ 2 ) ∥ H -1 2 and ∥ ˆ w 2 -w ∗ 2 ∥ 2 H 2 ,n . The first can easily be bounded using Chebyshev's inequality and the elementary fact that the variance of the average of n i.i.d. random variables is the variance of their common distribution divided by n . Here we state the result for all p ∈ (1 , ∞ ) ; the straightforward proof can be found in the Appendix D.

Lemma 2. Let p ∈ (1 , ∞ ) . If p ∈ (1 , 2) , let the assumptions of Theorem 5 hold. Then with probability at least 1 -δ/ 2

For the second random term ∥ ˆ w 2 -w ∗ 2 ∥ 2 H 2 ,n , we use Theorem 3.1 of Oliveira [Oli16], which we restate here, emphasizing that the existence of fourth moments of the components of the random vector is enough to ensure the existence of the needed norm equivalence constant.

Proposition 1 (Theorem 3.1, Oliveira [Oli16]) . Let Z ∈ R d be a random vector satisfying E[ Z 4 j ] < ∞ for all j ∈ [ d ] and assume that P( ⟨ v, Z ⟩ = 0) = 1 only if v = 0 . For p ∈ [1 , ∞ ) and v ∈ R d , define

Let ( Z i ) n i =1 be i.i.d. samples of Z . Then, with probability at least 1 -δ , for all v ∈ R d ,

Using this result we can immediately deduce the required high probability lower bound on the second random term ∥ ˆ w 2 -w ∗ 2 ∥ 2 H 2 ,n ; we leave the obvious proof to Appendix D.

Corollary 1. Under the assumptions of Theorem 1, if n ≥ 196 σ 2 2 ( d + 2log(4 /δ )) , then with probability at least 1 -δ/ 2 , for all w ∈ R d ,

Combining Lemma 2, Corollary 1, and (8) yields that with probability at least 1 -δ

Finally, combining (7) and (9) gives that with probability at least 1 -δ

Replacing in (6) finishes the proof.

3.2 Proof Sketch of Theorem 4

The main challenge in moving from the case p = 2 to the case p ∈ (2 , ∞ ) is that the second order Taylor expansion of the loss is no longer exact. The standard way to deal with this problem is to assume that the loss is upper and lower bounded by quadratic functions, i.e. that it is smooth and strongly convex. Unfortunately, as discussed in Section 1, the ℓ p loss is not strongly convex for any p > 2 , so we need to find another way to deal with this issue. Once this has been resolved however, the strategy we used in the proof of Theorem 1 can be applied almost verbatim to yield the result. Remarkably, a result of [AKPS22] allows us to upper and lower bound the p -th power loss for p ∈ (2 , ∞ ) by its second order Taylor expansion around a point, up to some residual terms. An application of this result yields the following Lemma.

Lemma 3. Let p ∈ (2 , ∞ ) . Then:

Up to constant factors that depend only on p and an L p norm residual term, Lemma 3 gives matching upper and lower bounds on the excess risk and excess empirical risk in terms of their second order Taylor expansions around the minimizer. We can thus use the approach taken in the proof of Theorem 1 to obtain the result. The only additional challenge is the control of the term ∥ ˆ w p -w ∗ p ∥ L p , which we achieve by reducing it to an ∥ ˆ w p -w ∗ p ∥ H p term using norm equivalence. A detailed proof of Theorem 4, including the proof of Lemma 3, can be found in Appendix E.

3.3 Proof Sketch of Theorem 5

The most technically challenging case is when p ∈ (1 , 2) . Indeed as seen in the proof of Theorem 1, the most involved step is lower bounding the excess empirical risk with high probability. For the case p ∈ [2 , ∞ ) , we achieved this by having access to a pointwise quadratic lower bound, which is not too surprising. Indeed, at small scales, we expect the second order Taylor expansion to be accurate, while at large scales, we expect the p -th power loss to grow at least quadratically for p ∈ [2 , ∞ ) .

In the case of p ∈ (1 , 2) , we are faced with a harder problem. Indeed, as p → 1 , the ℓ p losses behave almost linearly at large scales. This means that we cannot expect to obtain a global quadratic lower bound as for the case p ∈ [2 , ∞ ) , so we will need a different proof technique. Motivated by related concerns, Bubeck, Cohen, Lee, and Li [BCLL18] introduced the following approximation to the p -th power function

for t, x ∈ [0 , ∞ ) and with γ p (0 , 0) = 0 . This function was further studied by Adil, Kyng, Peng, and Sachdeva [AKPS19], who showed in particular that for any t ∈ R , the function x ↦→ γ p ( | t | , | x | ) is, up to constants that depend only on p , equal to the gap between the function x ↦→ ℓ p ( t + x ) and its linearization around 0 ; see Lemma 4.5 in [AKPS19] for the precise statement. We use this result to derive the following Lemma.

Lemma 4. Let p ∈ (1 , 2) . Under the assumptions of Theorem 5, we have

As expected, while we do have the desired quadratic upper bound, the lower bound is much more cumbersome, and is only comparable to the second order Taylor expansion when |⟨ w -w ∗ p , X i ⟩| ≤ |⟨ w ∗ p , X i ⟩ -Y i | . What we need for the proof to go through is a high probability lower bound of order Ω( ∥ w -w ∗ ∥ 2 H p ) on the first term in the lower bound (12). We obtain this in the following Proposition.

Proposition 2. Let δ ∈ (0 , 1] . Under the assumptions of Theorem 5, if n ≥ 196 σ 2 p ( d +2log(4 /δ )) , then with probability at least 1 -δ/ 2 , for all w ∈ R d ,

where ε p -2 := 8 σ 3 -p p c (2 -p ) / 2 p √ c ∗ p , and c p and c ∗ p are as defined in Theorem 5.

Proof. Let ε > 0 and let T ∈ (0 , ∞ ) be a truncation parameter we will set later. Define

˜ X := X · ✶ [0 ,T ] ( ∥ X ∥ H -1 p ) , and the constant β := Tε . By Lemma 3.3 in [AKPS19], we have that γ p ( t, λx ) ≥ min { λ 2 , λ p } γ p ( t, x ) for all λ ≥ 0 . Furthermore, it is straightforward to verify that γ p ( t, x ) is decreasing in t and increasing in x . Therefore, we have, for all w ∈ R d ,

The key idea to control the infimum in (14) is to truncate ⟨ w,X i ⟩ from above by using the truncated vector ˜ X , and |⟨ w ∗ p , X i ⟩ -Y i | from below by forcing it to be greater than β . By the monotonicity properties of γ p discussed above, we get that

where the equality follows by the fact that with the chosen truncations, the second argument of γ p is less than or equal to the first. It remains to lower bound the infimum in (15). Define

Removing the truncations and using our assumptions, we see that the components of Z have finite fourth moment. By Proposition 1 and the condition on n , we get that with probability at least 1 -δ/ 2 ,

We now bound the supremum in (16). We have

where the first inequality follows from Cauchy-Schwartz inequality, and the subsequent equalities by definitions of ∥·∥ L 4 ,p and σ 2 p . It remains to bound the tail probabilities. Recall that β = Tε , so

where we applied Markov's inequality in the fourth line, and the last follows by definition of c ∗ p in Theorem 5. Moreover by the finiteness of the second moment of the coordinates X j of X , we have

where Σ = E[ XX T ] , and the last equality by definition of c p in Theorem 5. By Markov's inequality

Choosing ensures that

Combining the inequalities (18), (17), (16), (15), and (14) yields the result.

A detailed proof of Theorem 5, including the proof of Lemma 4, can be found in Appendix F.

Acknowledgments and Disclosure of Funding

We thank Nikita Zhivotovskiy, Sushant Sachdeva, and Deeksha Adil for feedback on the manuscript. MAE was partially supported by NSERC Grant [2019-06167], CIFAR AI Chairs program, and CIFAR AI Catalyst grant.

References

[AC11] J.-Y. Audibert and O. Catoni. 'Robust Linear Least Squares Regression'. In: The Annals of Statistics (Oct. 2011). DOI: 10.1214/11-AOS918 .

[AKPS19] D. Adil, R. Kyng, R. Peng, and S. Sachdeva. 'Iterative Refinement for ℓ p -norm Regression'. In: Proceedings of the 2019 Annual ACM-SIAM Symposium on Discrete Algorithms (SODA) . Jan. 2019. DOI: 10.1137/1.9781611975482.86 .

[AKPS22] D. Adil, R. Kyng, R. Peng, and S. Sachdeva. Fast Algorithms for ℓ p -Regression . Nov. 7, 2022. DOI: 10.48550/arXiv.2211.03963 .

[APS19] D. Adil, R. Peng, and S. Sachdeva. 'Fast, Provably Convergent IRLS Algorithm for p-norm Linear Regression'. In: Advances in Neural Information Processing Systems . 2019. URL: https://proceedings.neurips.cc/paper_files/paper/2019/ hash/46c7cb50b373877fb2f8d5c4517bb969-Abstract.html .

[Arc96] M. A. Arcones. 'The Bahadur-Kiefer Representation of L p Regression Estimators'. In: Econometric Theory (1996). URL: https://www.jstor.org/stable/3532831 .

[AS19] D. Adil and S. Sachdeva. 'Faster p -Norm Minimizing Flows, via Smoothed q-Norm Problems'. In: Proceedings of the 2020 ACM-SIAM Symposium on Discrete Algorithms (SODA) . Dec. 23, 2019. DOI: 10.1137/1.9781611975994.54 .

[BCLL18] S. Bubeck, M. B. Cohen, Y. T. Lee, and Y. Li. 'An Homotopy Method for ℓ p Regression Provably beyond Self-Concordance and in Input-Sparsity Time'. In: Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing . June 20, 2018. DOI: 10.1145/3188745.3188776 .

[BRW92] Z. D. Bai, C. R. Rao, and Y. Wu. 'M-Estimation of Multivariate Linear Regression Parameters Under a Convex Discrepancy Function'. In: Statistica Sinica (1992). URL: https://www.jstor.org/stable/24304129 .

[CG17] O. Catoni and I. Giulini. Dimension-Free PAC-Bayesian Bounds for Matrices, Vectors, and Linear Least Squares Regression . Dec. 31, 2017. DOI: 10.48550/arXiv.1712. 02747 .

[HKZ12] D. Hsu, S. M. Kakade, and T. Zhang. 'Random Design Analysis of Ridge Regression'. In: Proceedings of the 25th Annual Conference on Learning Theory . June 16, 2012. URL: https://proceedings.mlr.press/v23/hsu12.html .

[HS96] X. He and Q.-M. Shao. 'A General Bahadur Representation of M-estimators and Its Application to Linear Regression with Nonstochastic Designs'. In: The Annals of Statistics (Dec. 1996). DOI: 10.1214/aos/1032181172 .

[JLS22] A. Jambulapati, Y. P. Liu, and A. Sidford. 'Improved Iteration Complexities for Overconstrained p -Norm Regression'. In: Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing . June 10, 2022. DOI: 10.1145/3519935.3519971 .

[KM15] V. Koltchinskii and S. Mendelson. 'Bounding the Smallest Singular Value of a Random Matrix Without Concentration'. In: International Mathematics Research Notices (2015). DOI: 10.1093/imrn/rnv096 .

[Kni98] K. Knight. 'Limiting Distributions for L 1 Regression Estimators under General Conditions'. In: The Annals of Statistics (Apr. 1998). DOI: 10.1214/aos/1028144858 .

[LC83] E. L. Lehmann and G. Casella. Theory of Point Estimation . 1983.

[LL05] P. Y. Lai and S. M. S. Lee. 'An Overview of Asymptotic Properties of L p Regression under General Classes of Error Distributions'. In: Journal of the American Statistical Association (2005). URL: https://www.jstor.org/stable/27590567 .

[LM16] G. Lecué and S. Mendelson. 'Performance of Empirical Risk Minimization in Linear Aggregation'. In: Bernoulli (Aug. 2016). DOI: 10.3150/15-BEJ701 .

[LM17a] G. Lecué and S. Mendelson. 'Regularization and the Small-Ball Method II: Complexity Dependent Error Rates'. In: Journal of Machine Learning Research (2017). URL: http: //jmlr.org/papers/v18/16-422.html .

[LM17b] G. Lecué and S. Mendelson. 'Sparse Recovery under Weak Moment Assumptions'. In: Journal of the European Mathematical Society (Feb. 15, 2017). DOI: 10.4171/jems/ 682 .

[LM18] G. Lecué and S. Mendelson. 'Regularization and the Small-Ball Method I: Sparse Recovery'. In: The Annals of Statistics (Apr. 2018). DOI: 10.1214/17-AOS1562 .

[Men14] S. Mendelson. 'Learning without Concentration'. In: Proceedings of The 27th Conference on Learning Theory . May 29, 2014. URL: https://proceedings.mlr.press/ v35/mendelson14.html .

[Men18] S. Mendelson. 'Learning without Concentration for General Loss Functions'. In: Probability Theory and Related Fields (June 1, 2018). DOI: 10.1007/s00440-017-0784-y .

[Men21] S. Mendelson. 'Learning Bounded Subsets of L p '. In: IEEE Transactions on Information Theory (Aug. 2021). DOI: 10.1109/TIT.2021.3083553 .

[MG22] J. Mourtada and S. Gaïffas. 'An Improper Estimator with Optimal Excess Risk in Misspecified Density Estimation and Logistic Regression'. In: Journal of Machine Learning Research (2022). URL: http://jmlr.org/papers/v23/20-782.html .

[Mou22] J. Mourtada. 'Exact Minimax Risk for Linear Least Squares, and the Lower Tail of Sample Covariance Matrices'. In: The Annals of Statistics (Aug. 2022). DOI: 10.1214/ 22-AOS2181 .

[MVZ22] J. Mourtada, T. Vaškeviˇ cius, and N. Zhivotovskiy. 'Distribution-free robust linear regression'. In: Mathematical Statistics and Learning (2022). DOI: 10.4171/MSL/27 .

[Nie92] W. Niemiro. 'Asymptotics for M -Estimators Defined by Convex Minimization'. In: The Annals of Statistics (Sept. 1992). DOI: 10.1214/aos/1176348782 .

[OB21] D. M. Ostrovskii and F. Bach. 'Finite-Sample Analysis of M -Estimators Using SelfConcordance'. In: Electronic Journal of Statistics (Jan. 2021). DOI: 10.1214/20EJS1780 .

[Oli16] R. I. Oliveira. 'The Lower Tail of Random Quadratic Forms with Applications to Ordinary Least Squares'. In: Probability Theory and Related Fields (Dec. 1, 2016). DOI: 10.1007/s00440-016-0738-9 .

[Ron84] A. E. Ronner. 'Asymptotic Normality of p -norm Estimators in Multiple Regression'. In: Zeitschrift für Wahrscheinlichkeitstheorie und Verwandte Gebiete (Sept. 1, 1984). DOI: 10.1007/BF00531893 .

[RV15] M. Rudelson and R. Vershynin. 'Small Ball Probabilities for Linear Images of HighDimensional Distributions'. In: International Mathematics Research Notices (2015). DOI: 10.1093/imrn/rnu243 .

[Sau18] A. Saumard. 'On Optimality of Empirical Risk Minimization in Linear Aggregation'. In: Bernoulli (Aug. 2018). DOI: 10.3150/17-BEJ925 .

[Vaa98] A. W. Van Der Vaart. Asymptotic Statistics . 1998. DOI: 10.1017/CBO9780511802256 .

[VW96] A. W. Van Der Vaart and J. A. Wellner. Weak Convergence and Empirical Processes . 1996. DOI: 10.1007/978-1-4757-2545-2 .

[VZ23] T. Vaškeviˇ cius and N. Zhivotovskiy. 'Suboptimality of Constrained Least Squares and Improvements via Non-Linear Predictors'. In: Bernoulli (Feb. 2023). DOI: 10.3150/22BEJ1465 .

[Whi82] H. White. 'Maximum Likelihood Estimation of Misspecified Models'. In: Econometrica (1982). DOI: 10.2307/1912526 .

A Differentiability of the risk

In this section, we rigorously establish the twice differentiability the risk under our assumptions. We start by showing that under a subset of our assumptions, the risk is differentiable everywhere on R d .

Lemma 5. Let p ∈ (1 , ∞ ) and assume that E[ | Y | p ] < ∞ and E[ | X j | p ] < ∞ for all j ∈ [ d ] . Then R p is differentiable on R d , and

Proof. Let w ∈ R d . We want to show that

where, for convenience, we take the norm ∥·∥ to be the Euclidean norm. Define the function ϕ ( w,X,Y ) := ℓ p ( ⟨ w,X ⟩ -Y ) and note that by the chain rule ϕ is differentiable as a function of w on all of R d . Now let (∆ k ) n k =1 be a sequence in R d such that lim k →∞ ∥ ∆ k ∥ = 0 . Then

Our goal is to interchange the limit and expectation. For that, we will use the dominated convergence theorem. We construct our dominating function as follows. Let R := sup k ∈ N ∥ ∆ k ∥ , and note that R < ∞ since ∥ ∆ k ∥ → 0 as k →∞ . Then we have

where the second line follows by triangle inequality, the third from the fundamental theorem of calculus applied component-wise, the fourth by Cauchy-Schwartz inequality, the fifth by Jensen's inequality and the convexity of the norm, and the eighth by the inequality | a + b | q ≤ max { 2 q -1 , 1 } ( | a | q + | b | q )

valid for q > 0 . It remains to show that g ( X,Y ) is integrable. We have

where in the second line we used that the Euclidean norm is bounded by the 1 -norm, in the third we used Holder's inequality, and the last line follows from our assumptions. Applying the dominated convergence theorem, we interchange the limit and the expectation in (19). Recalling that ϕ is differentiable finishes the proof.

We now turn to the twice differentiability of the risk. We start with the easy case p ∈ [2 , ∞ ) . The proof is very similar to that of Lemma 5 and we omit it here.

Lemma 6. Let p ∈ [2 , ∞ ) and assume that E[ | Y | p ] < ∞ and E[ | X j | p ] < ∞ for all j ∈ [ d ] . Then R p is twice differentiable on R d , and

The case p ∈ (1 , 2) is more complicated. The following lemma establishes the twice differentiability of the risk at its minimizer under a subset of the assumptions of Theorem 5.

Lemma 7. Let p ∈ (1 , 2) . Assume that P ( |⟨ w ∗ p , X ⟩ -Y | = 0 ) = 0 and E[ |⟨ w ∗ p , X ⟩ -Y | p -2 ( X j ) 2 ] < ∞ for all j ∈ [ d ] . Then R p is twice differentiable at w ∗ p and

Proof. The difficulty in the proof compared to Lemma 5 and Lemma 6 stems from the fact that the loss is not twice differentiable at zero. We still rely on the dominated convergence theorem, but the construction of the dominating function is slightly more intricate. Using the setup of the proof of Lemma 5, and following the same line of arguments, we arrive at

where we have used the fact that since P ( |⟨ w ∗ p , X ⟩ -Y | = 0 ) = 0 , ϕ ( w,X,Y ) is almost surely twice differentiable at w ∗ p . To finish the proof, it remains to construct a dominating function for the above sequence to justify the interchange of the limit and expectation. We consider two cases.

Case 1: ∥ ∆ k ∥ ≥ ∣ ∣ ⟨ w ∗ p , X ⟩ -Y ∣ ∣ / (2 ∥ X ∥ ) =: R ( X,Y ) . Then we have

where the second line follows by triangle inequality, the third by definition of the operator norm, the fourth by | a + b | q ≤ | a | q + | b | q valid for q ∈ (0 , 1) , and the fifth and sixth by Cauchy-Schwartz inequality and the assumed lower bound on ∥ ∆ k ∥ .

Case 2: ∥ ∆ k ∥ < R ( X,Y ) . We start by noting that, for all ∆ ∈ B (0 , R ( X,Y )) := { x ∈ R d | ∥ x ∥ < R ( X,Y ) } , we have

Therefore ϕ ( w,X,Y ) is twice differentiable on B (0 , R ( X,Y )) . Now

where the second line follows from the triangle inequality, the third follows from the twice differentiability of ϕ on B (0 , R ( X,Y )) and the fundamental theorem of calculus applied component-wise, the fifth by Jensen's inequality, and the last by definition of R ( X,Y ) and the above lower bound. We therefore define our dominating function by

It is then immediate from our assumptions that g ( X,Y ) is integrable. Interchanging the limit and the expectation in (20) and recalling that ϕ is almost surely twice differentiable at w ∗ p finishes the proof.

B Proof of Lemma 1

We start with the claim that ρ 0 is upper-semicontinuous. We want to show that for any w ∈ R d and any sequence ( w k ) ∞ k =1 converging to w (in the norm topology)

Fix a w ∈ R d and let ( w k ) ∞ k =1 be a sequence in R d satisfying lim k →∞ ∥ w -w k ∥ = 0 , where for convenience we take ∥·∥ to be the Euclidean norm on R d . Then we have by (reverse) Fatou's Lemma

Otherwise we have δ := |⟨ w,X ⟩| > 0 . But then, by the convergence of ( w k ) ∞ k =1 to w , there exists a K ∈ N such that for all k ≥ K we have ∥ w k -w ∥ < δ/ (2 ∥ X ∥ ) . This implies that for all k ≥ K

We conclude that

Combining (21), (22), and (23) proves the upper-semicontinuity of ρ 0 . Essentially the same proof shows the lower-semicontinuity of ρ q ( · , κ ) for any κ ≥ 0 ; we omit it here.

For the remaining claims, first notice that the function ρ 0 is scale invariant, i.e. for all w ∈ R d and all c ∈ R , we have ρ 0 ( cw ) = ρ 0 ( w ) . Therefore

where S d -1 is the Euclidean unit sphere. By assumption on the random vector X , we know that ρ 0 ( w ) < 1 for all w ∈ S d -1 . Furthermore since ρ 0 is upper semicontinuous, and S d -1 is compact, ρ 0 attains its supremum on S d -1 at some point w 0 ∈ S d -1 . From this we conclude that

ρ

=

sup

∈

w

R

d

\{

0

}

Finally, we turn to the claim about ρ q . Since E[ | X j | q ] < ∞ , the function ∥·∥ L q is a norm on R d , from which it follows that ρ q ( w,κ ) is also scale invariant for any κ . Therefore

where S q is the unit sphere of the norm ∥·∥ L q . Now fix κ ∈ [0 , 1) . We claim that ρ q ( w,κ ) > 0 for all w ∈ S q . Suppose not. Then there exists a w ∈ S q such that |⟨ w,X ⟩| ≤ κ with probability 1 , but then we get the contradiction

therefore ρ q ( w,κ ) > 0 for all w ∈ S q . Now since ρ q ( · , κ ) is lower-semicontinuous, and S q is compact, ρ q ( · , κ ) attains its infimum on S q at some point w q ∈ S q . From this we conclude

ρ

0

(

w

) =

ρ

0

(

w

0

)

<

1

.

C Proof of Theorem 3

Fix p ∈ (1 , ∞ ) , and let ˆ w := ˆ w p . Our goal will be to upper bound the probability P( ˆ w = w ∗ ) . By assumption, we have that Y = ⟨ w ∗ , X ⟩ , so that Y i = ⟨ w ∗ , X i ⟩ for all i ∈ [ n ] . Since ˆ w minimizes the empirical risk, we must also have ⟨ ˆ w,X i ⟩ = Y i = ⟨ w ∗ , X i ⟩ for all i ∈ [ n ] . Let A ∈ R n × d denote the matrix whose i -th row is X i . Then we have the following implications.

̸

̸

ˆ w = w ∗ ⇒⟨ ˆ w -w ∗ , X i ⟩ = 0 ∀ i ∈ [ n ] ⇒∃ w ∈ R d \ { 0 } | Aw = 0 ⇔ rowrank( A ) < d. (24) Let r := rowrank( A ) . We claim the following equivalence

Indeed the implication ( ⇐ ) follows by definition of the rowrank of A . For the implication ( ⇒ ) , by definition, { X i | i ∈ [ n ] } is a spanning set for the row space of A , therefore it can be reduced to a basis of it { X k | k ∈ S 1 } for some indices S 1 ⊂ [ n ] with | S 1 | = r . If r = d -1 , then the choice S = S 1 satisfies the right side of (25). Otherwise, let S 2 ⊂ [ n ] \ S 1 with | S 2 | = d -1 -r . Such a subset exists since by assumption n ≥ d > d -1 . Then the set S := S 1 ∪ S 2 satisfies the right side of (25). Combining (24) and (25) we arrive at:

̸

where the second inequality follows from the union bound. We now bound each of the terms of the sum. Fix S = { i 1 , . . . , i d -1 } ⊂ [ n ] with | S | = d -1 . Let Z S = n (( X i j ) d -1 j =1 ) be a non-zero vector orthogonal to span( { X k | k ∈ S } ) . Such a vector must exist since dim(span( { X k | k ∈ S } )) < d ; see Lemma 8 below for an explicit construction of the function n . Denote by P Z S the distribution of Z S and P ( X i ) i ∈ [ n ] \ S = ∏ n -d -1 i =1 P X the distribution of ( X i ) i ∈ [ n ] \ S , where P X is the distribution of X . Note that since Z S is a function of ( X i j ) d j =1 only, it is independent of ( X i ) i ∈ [ n ] \ S . In particular, the joint distribution of ( Z S , ( X i ) i ∈ [ n ] \ S ) is given by the product P Z S × P ( X i ) i ∈ [ n ] \ S . Now if X i ∈ span( { X k | k ∈ S } ) , then by definition of Z S , ⟨ Z S , X i ⟩ = 0 . Therefore

∀

P(

i

∈

[

n

]

\

S X

i

∈

span(

{

X

k

|

k

∈

S

}

))

P(

∀

i

∈

[

n

]

\

S

⟨

Z

S

, X

i

⟩

= 0)

̸

(27)

where in the third line we used the independence of Z S and ( X i ) i ∈ [ n ] \ S , in the fourth we used the independence of the ( X i ) i ∈\ S , in the sixth we used the definition of ρ 0 in 5, and in the last line we used the fact that z S = 0 and the definition of ρ in Lemma 1. Combining the inequalities (26) and (27) yields the result.

Lemma 8. Let m ∈ { 1 , . . . , d -1 } and let ( x j ) j m =1 be a sequence of points in R d . Denote by A ∈ R m × d the matrix whose j -th row is x j and let A + be its pseudo-inverse. Let ( b i ) d i =1 be an ordered basis of R d , and define

̸

Then n (( x j ) j m =1 ) is non-zero and is orthogonal to span( { x j | j ∈ [ m ] } ) .

̸

Proof. We start by showing that k is well defined. First note that I -A + A is the orthogonal projector onto the kernel of A , which is non-trivial since dim(ker( A )) = d -dim(Im( A )) ≥ d -m ≥ 1 . Now we claim that there exists an i ∈ [ d ] such that ( I -A + A ) b i = 0 . Suppose not, then for any w ∈ R d , we have ( I -A + A ) w = ( I -A + A )( ∑ d i =1 c i b i ) = ∑ d i =1 c i ( I -A + A ) b i = 0 , implying that I -A + A = 0 , but this contradicts the non-triviality of ker( A ) . This proves that k is well-defined, which in turn proves that n (( x j ) j m =1 ) = 0 . It remains to prove the orthogonality claim. Let v ∈ span( { x j | j ∈ [ m ] } ) . Then there exists coefficients c ∈ R m such that v = A T c . Therefore

̸

where the last equality holds since ( I -AA + ) b k ∈ ker( A ) .

D Missing proofs for Theorem 1

This section contains the proofs of Lemma 2 and Corollary 1 we used in the proof of Theorem 1.

D.1 Proof of Lemma 2

We compute the expectation:

where in the second line we expanded the inner product of the sums into its n 2 terms, used linearity of expectation, and used the independence of the samples to take the expectation inside the inner product. In the last line, we used the fact that the samples are identically distributed to simplify the first term. For the second term, we used the fact that the expectation of the gradient of the loss at the risk minimizer vanishes. Applying Markov's inequality finishes the proof.

D.2 Proof of Corollary 1

We have

Now by assumption, the components of the vector X have finite fourth moment so that applying Proposition 1 and using the condition on n yields the result.

E Detailed proof of Theorem 4

This section contains the proof of Lemma 3 as well as that of Theorem 4.

E.1 Proof of Lemma 3

By Lemma 2.5 in [AKPS22], we have for all t, s ∈ R

Recall that by the chain rule

Replacing t and s by ⟨ w,X i ⟩ -Y i and ⟨ w ∗ p , X i ⟩ -Y i respectively, and using the formulas for the gradient and Hessian we arrive at

Averaging over i ∈ [ n ] yields the first inequality. The proof of the second inequality proceeds in the same way and uses instead the upper bound of Lemma 2.5 in [AKPS22]. We omit it here.

E.2 Proof of Theorem 4

We proceed similarly to the proof of Theorem 1. By definition of the empirical risk minimizer, we have the upper bound

Using (10) from Lemma 3 and the Cauchy-Schwarz inequality, we obtain the pointwise lower bound

Using Lemma 2 we have that, with probability at least 1 -δ/ 2 ,

It remains to control ∥ ˆ w p -w ∗ p ∥ 2 H p,n from below. Define the random vector

Then, for any w ∈ R d , we have

By assumption, the components of the random vector Z have finite fourth moment. Applying Proposition 1, and using the condition on n assumed in the statement of Theorem 4, we get that, with probability at least 1 -δ/ 2 , for all w ∈ R d ,

Combining (30) and (31) with (29) gives that with probability at least 1 -δ ,

Further combining (32) with (28) and rearranging yields that with probability at least 1 -δ

The last step is to bound the excess risk of the empirical risk minimizer using the bound (33) and (11) from Lemma 3. For that, we control the L p norm term in (11) as follows

Combining (33), (11), and (34) yields the result.

F Detailed proof of Theorem 5

This section contains the proof of Lemma 4 as well as that of Theorem 5.

F.1 Proof of Lemma 4

Both inequalities follow from Lemma 4.5 in Adil, Kyng, Peng, and Sachdeva [AKPS19]. (12) follows from a straightforward calculation using the lower bound of Lemma 4.5 in Adil, Kyng, Peng, and Sachdeva [AKPS19]; we omit it here. The upper bound requires a bit more work. We have by the quoted Lemma

Now assume that | s | > 0 . If | t -s | ≤ | s | , we have

Otherwise, if | t -s | > | s | , then we have

̸

Therefore in both cases we have γ p ( | s | , | t -s | ) ≤ ℓ ′′ p ( s )( t -s ) 2 as long as | s | > 0 . Replacing t and s by ⟨ w,X ⟩ -Y and ⟨ w ∗ p , X ⟩ -Y respectively we get, on the event that ⟨ w ∗ p , X ⟩ -Y = 0

̸

Recalling that by assumption P ( ⟨ w ∗ p , X ⟩ -Y = 0 ) = 1 , taking expectation of both sides, and bounding 1 /p ≤ 1 finishes the proof of (13).

F.2 Proof of Theorem 5

We follow the same proof strategy as the one used in the proofs of Theorems 1 and 4. By definition of the empirical risk minimizer, we have

Using (12) from Lemma 4 and the Cauchy-Schwarz inequality, we have the lower bound

Using Lemma 2 we have that, with probability at least 1 -δ/ 2 ,

On the other hand, by Proposition 2, we have with probability 1 -δ/ 2 , for all w ∈ R d ,

where ε is as defined in Proposition 2. We now consider two cases. If ∥ ˆ w p -w ∗ p ∥ 2 H p ≤ ε 2 -p ∥ ˆ w p -w ∗ p ∥ p H p , then combining (35), (36), (37), and (38) gives

Otherwise, ∥ ˆ w p -w ∗ p ∥ 2 H p > ε 2 -p ∥ ˆ w p -w ∗ p ∥ p H p , then again combining (35), (36), (37), and (38) gives

In either case, we have, using (39) and (40), with probability at least 1 -δ ,

Combining this last inequality with (13) from Lemma 4 finishes the proof.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the excess risk of empirical risk minimization on the $p$-norm linear regression. The asymptotic bound of the excess risk of ERM for $p$ in ($1,\\infty$) is known by prior work and more recent work had given high probability excess risk bounds for $p=2$ that match that asymptotic rate. This paper expands the result to $p$ in $(1,\\infty) \\setminus \\{2\\}$. It splits the results into three parts: It first provides bounds for the realizable case for $p \\in$ (1,$\\infty$). Then, a bound for the non-realizable case with $p$ $\\in$ (2,$\\infty$) where it requires mild assumptions on the moments that are natural extensions of the assumptions needed for the $p=2$ prior work’s result. For the non-realizable case with $p$ $\\in$ (1,2), the paper requires some additional assumptions: a slightly stronger version of non-realizability, assuming that the inner product of the covariates $x$ with the optimal minimizer $w^*$ is almost surely not equal to the label, and the existence of the $2(2-p)$ negative moment of $y-\\langle w^*,x \\rangle$.\n\nThe proof technique draws inspiration from Oliveira 2016, Lecue and Mendelson 2016b, which by itself is not sufficient. However, when combined with a recent result of Adil et al. 2022, the approach becomes valid by allowing the use of the second-order Taylor expansion, despite the fact that it is not exact. For the case of $p$ $\\in$ (1,2), a different approach is employed, utilizing an approximation for the $p$-th power function from Bubeck et al. 2018, along with the analysis of Adil et al. 2019a.\n\nSTRENGTHS:\nThe paper provides the first high probability excess risk bounds of ERM for $p$-norm regression for $p \\in (1,\\infty)\\setminus \\{2\\}$ that are consistent with the existing asymptotic bounds. This is a considerable result. The paper does a very good job in describing the problem’s motivation, the problem’s background and the new results.\n\nWEAKNESSES:\nAs the authors have already described, the case $p \\in (1,2)$ requires extra assumptions that it is unclear if they are necessary, leaving the goal of the paper for that regime only partially resolved.\n\nAlthough the paper writing is good in presenting the problem and the results, the proof strategy of the main results is not really discussed until later sections of the paper.\n\nAs a minor comment on writing, it would be better to provide a definition of $\\rho$ (or a reference to its definition) inside the theorem statements that use it. Similar comment for $\\sigma_p^2$.\n\nQUESTIONS:\nWhat do the three regimes for $\\rho$ represent in Theorem 3?\n\nLIMITATIONS:\nAddressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the performance of empirical risk minimization (ERM) for $\\ell_p$-norm linear regression, in both the realizable and non-realizable (say, noisy measurement) settings. While the $p=2$ case has been studied extensively in the literature, not as much is known for the general $p \\in (1,2) \\cup (2,\\infty)$ cases. For the realizable case, the author(s) improve over prior analyses and give a $O(d)$ sample complexity bound for finding the regression coefficients with high probability. For the non-realizable case, under moment assumptions, this paper shows high probability excess risk bounds.\n\nNote: this is an emergency review, and so I did not check the math carefully.\n\nSTRENGTHS:\nThe paper is well-written and clear. The author(s) motivate the $\\ell_p$ linear regression problem well, and this paper gives the first non-asymptotic excess risk bounds for ERM in this setting. The proof ideas are also explained pretty clearly.\n\nWEAKNESSES:\nMy main concern/confusion is the message of the paper. The author(s) seem to sell the paper as having a high probability (excess) risk bound, yet the bound has a $1/\\delta$ dependence on the failure probability $\\delta$ (from using Markov, and might actually be tight) instead of logarithmic or root-logarithmic (sub-Gaussian) performance. ERM is very susceptible to heavy-tailed noise, even in the simple special case of mean estimation, so I'm a bit confused why we're studying the performance of ERM here, instead of designing better estimators that work under such mild moment assumptions. From my readthrough, it also appears that the paper is really bounding some expectation and using Markov to get the high probability bound. My question then, is, why is the paper phrased as a high probability result instead of an expectation/constant probability result, which I can imagine ERM to actually be decent at?\n\nQUESTIONS:\nBeyond the motivation question I have above:\n\n- The results are being compared to the asymptotic guarantees of ERM, but is there any evidence that ERM's asymptotic performance is statistically optimal (even when compared with other estimators)? In the special case of mean estimation, for example, we can actually write down non-asymptotic lower bounds on the estimation error in the Gaussian cases, which match the central limit theorem.\n- Can the author(s) further elaborate on the extra moment assumption $\\mathbb{E}[|Y - \\langle w_p^*,X\\rangle|^{2(p-2)} X_j^4] < \\infty$ in Theorem 4? Line 168 states that it's a \"natural extension\", but really doesn't give interpretation to this assumption, in particular, the interaction between the response and the covariate coordinates.\n\nAnother misc comment: The beginning of Section 2.2 makes it sound like Theorem 1 is an old result, but after the theorem, the author(s) state that there's novelty at least in the proof presentation. Probably worth writing that more explicitly before the theorem statement.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-24"}

comments: SUMMARY:
This paper studies the empirical risk minimization of linear regression with the p-norm loss function, where p ∈ (1,∞).

Main contributions:
- The authors provide a high probability excess risk bound on the empirical risk minimizer, for p ∈ (1,∞), where for p ∈ [2,∞) only weak moment assumptions are needed, and for p ∈ (1, 2) there are assumptions that guarantee the existence of the hessian at the minimizer of the risk.
- The authors strengthen a previous bound of perfect recovery in the realizable case.

STRENGTHS:
Some strengths are:
- This paper is the first work providing the high probability excess risk bounds for empirical risk minimization under the p-norm loss function with any p ∈ (1,∞).
- The authors strengthen a previous bound of perfect recovery in the realizable case.

WEAKNESSES:
Some weaknesses of the paper:
- The proof part is poorly written. For example, the statements of Corollary 1 and Proposition 1 don't make any sense (the inf of the left-hand side is less than the right-hand side?), where even for Proposition 1 the authors just want to cite an existing theorem in a previous paper (Theorem 3.1, Oliveira [2016]).  
- There is a lack of the argument of the importance of this work. The authors discuss the benefits of using p-norm as the loss with p != 2, but why moving from asymptotic bound to finite sample bound is important for p != 2 is missing.

Typos:
- At line 232, second line of the equation misses index i on X and Y.

QUESTIONS:
NA

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-26

comments: SUMMARY:
The paper considers linear regression with inputs in dimension d and with n data points. It consider the Lp loss |.|^p with a focus on p different from the classical value 2. The paper provides non-asymptotic risk excess bound for this Lp loss, the excess being between the optimal linear combination w^* and the one obtained by minimizing the empirical risk. The bounds are of different natures in the two cases where the problem is realizable, meaning that the output is exactly a linear function of the input, and where the problem is not realizable.

STRENGTHS:
The paper is well written. The problem is theoretically interesting. The theoretical results are well-connected to the existing literature and their novelty compared to this literature is well motivated. The proofs appear to be innovative and to combine arguments from different branches of  the literature.

WEAKNESSES:
The content is technical and so the paper is not extremely easy to follow. 
An option could be to take advantage of the appendix (with no space limit) to add more slow-paced pedagogical content.

QUESTIONS:
I do not have specific questions.

LIMITATIONS:
I do not see potential negative societal impact.
The limitations are discussed adequately
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies excess risk bounds for empirical risk minimization for linear regression under $\\ell_p$-norm. It extends existing work $p=2$ to general $p\\in (1,\\infty)$.\n\nSampling complexity bounds are given for recovering the true parameter in the realizable case.\n\nAdditive error excess risk bounds are given (under mild assumptions on $O(p)^{th}$ moments) in the non-realizable case\n\nI generally like the paper but some issues need to be discussed/resolved, based on which my rating could move either way.\n\nSTRENGTHS:\n* natural extension to $p$-norm regression\n* non-asymptotic bounds take similar forms as the asymptotic results in the p=2 case (up to dependencies on constants and $p$)\n* rigorous theory\n* good high-level discussion on the difficulties of generalizing previous proofs to general $p$ and how they are handled\n* good overview over existing work\n\nWEAKNESSES:\n* \"Optimality\", claimed already in the title is not supported by any (existing or new) lower bounds\n* some assumptions seem to 'eat' terms, that depend on the dimension d. There is some, but only very limited discussion on this\n* no result for the second most important case $p=1$\n* in a few places very 'mathematical' statements without explaining the semantic or intuition behind\n* bounds depend on Hessian forms of $\\ell_p$ loss functions, which fit _perfectly well_ to the quadratic function in the $p=2$ case but do not capture the loss as closely for other $p$, even though the bounds look similar\n* discussion on (third paragraph, ll 23-32) motivating the $\\ell_p$ loss regression problem could be improved and seems vague wrt the properties of the model. I suggest rewriting this, stating that it is the standard linear model with $p$-generalized Gaussian error distribution that has certain statistical properties parameterized by $p$. See https://arxiv.org/pdf/2203.13568.pdf and references therein.\n\nQUESTIONS:\n1. Optimality is a strong claim. In what sense can your bounds be considered _optimal_ ? I see no lower bounds at all !?!\n\n2. Theorem 3 seems (at first glance) even below known lower bounds. For instance if the generating distribution realizes standard basis vectors with equal probability and the target is not contained in any subspace spanned by a proper subset, it is known that $\\Omega(d \\log d)$ is necessary to recover the exact target (follows simply from the coupon collector's theorem). So, my question is, what is the role and the dependence of the $\\rho$ parameter on the dimension?\n\n3. Again Theorem 3: the three cases seem a bit arbitrary, why is it necessary to distinguish between them? is it only to make the upper bounds work in any case or are there stronger indications for actually separated complexities in the three cases?\n\n4. Why are there no bounds on $p=1$, would there be a major complication (in your analysis or in general) in tackling this case?\n\n5. What is a \"negative moment\"?\n\nLIMITATIONS:\nThe very important case $p=1$ is not handled or discussed at all beyond the intro/motivation", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}