Title: On the Efficiency of ERM in Feature Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Empirical risk minimization

Feature learning

Learning theory

Sparse linear regression
Abstract: Given a collection of feature maps indexed by a set $\mathcal{T}$, we study the performance of empirical risk minimization (ERM) on regression problems with square loss over the union of the linear classes induced by these feature maps. This setup aims at capturing the simplest instance of feature learning, where the model is expected to jointly learn from the data an appropriate feature map and a linear predictor. We start by studying the asymptotic quantiles of the excess risk of sequences of empirical risk minimizers. Remarkably, we show that when the set $\mathcal{T}$ is not too large and when there is a unique optimal feature map, these quantiles coincide, up to a factor of two, with those of the excess risk of the oracle procedure, which knows a priori this optimal feature map and deterministically outputs an empirical risk minimizer from the associated optimal linear class. We complement this asymptotic result with a non-asymptotic analysis that quantifies the decaying effect of the global complexity of the set $\mathcal{T}$ on the excess risk of ERM, and relates it to the size of the sublevel sets of the suboptimality of the feature maps. As an application of our results, we characterize the performance of the best subset selection procedure in sparse linear regression under general assumptions.

On the Efficiency of ERM in Feature Learning

Abstract

Given a collection of feature maps indexed by a set T , we study the performance of empirical risk minimization (ERM) on regression problems with square loss over the union of the linear classes induced by these feature maps. This setup aims at capturing the simplest instance of feature learning, where the model is expected to jointly learn from the data an appropriate feature map and a linear predictor. We start by studying the asymptotic quantiles of the excess risk of sequences of empirical risk minimizers. Remarkably, we show that when the set T is not too large and when there is a unique optimal feature map, these quantiles coincide, up to a factor of two, with those of the excess risk of the oracle procedure, which knows a priori this optimal feature map and deterministically outputs an empirical risk minimizer from the associated optimal linear class. We complement this asymptotic result with a non-asymptotic analysis that quantifies the decaying effect of the global complexity of the set T on the excess risk of ERM, and relates it to the size of the sublevel sets of the suboptimality of the feature maps. As an application of our results, we obtain new guarantees on the performance of the best subset selection procedure in sparse linear regression under general assumptions.

1 Introduction

A central idea in modern machine learning is that of data-driven feature learning. Specifically, instead of performing linear prediction on top of handcrafted features, the current dominant paradigm suggests to use models that select useful features for linear prediction in a data-dependent way [e.g. KSH12; LBH15; He+16; Vas+17]. Of course, by putting the burden of picking a feature map on the model and data, we should expect that the resulting learning problem will require more samples to be solved. But just how many more samples do we need to learn such feature-learning-based models?

In this paper, we investigate this question in a general setting. We study the performance of empirical risk minimization (ERM) on regression tasks with square loss and over model classes induced by arbitrary collections of features maps. More precisely, let X be the random input taking value in a set X , and let ( ϕ t ) t ∈T , ϕ t : X → R d , be a collection of feature maps indexed by a set T . For a given regression task and i.i.d. samples, our aim is to understand the performance of ERM over the class of predictors ∪ t ∈T { x ↦→⟨ w,ϕ t ( x ) ⟩ | w ∈ R d } as a function of the sample size, the distribution of the data, and relevant properties of the collection of feature maps ( ϕ t ) t ∈T .

Classical uniform-convergence-based analyses would suggest that the performance of ERM in this setting is determined by the size of the model class, appropriately measured. The main message of this paper is that in this case, this is wrong in a strong sense. Specifically, we prove an upper bound on the excess risk of ERM on this problem whose dependence on the size of the model class decays monotonically with the sample size, and eventually depends only on the size of the model class induced by the collection of optimal feature maps, which is typically much smaller.

Formal setup. We briefly formalize our problem here. Let X be the random input taking value in a set X , and let ( ϕ t ) t ∈T , ϕ t : X → R d , be a collection of feature maps indexed by a set T . 1 Let Y ∈ R be the output random variable, jointly distributed with the input X . Our goal is to learn to predict the output Y given the input X as well as possible within the class of predictors { x ↦→⟨ w,ϕ t ( x ) ⟩ | ( t, w ) ∈ T × R d } . We evaluate the quality of a single prediction ˆ y given the ground truth y through the loss function ℓ (ˆ y, y ) := (ˆ y -y ) 2 / 2 , and the overall quality of a predictor ( t, w ) ∈ T × R d through its risk

We assume that we have access to n i.i.d. samples ( X i , Y i ) n i =1 with the same distribution as ( X,Y ) , and perform empirical risk minimization

Our goal is to characterize the excess risk E ( ˆ t n , ˆ w n ) := R ( ˆ t n , ˆ w n ) -R ∗ .

Related work. The study of upper bounds on the excess risk of ERM in a general setting is a classical topic. It was initiated by Vapnik and Chervonenkis [VC74] who established a link between the excess risk of ERM and the uniform convergence of the underlying empirical process. More recently, and fuelled by the development of Talagrand's concentration inequality [Tal96] and its refinements [e.g. BLM00; Bou02], a literature emerged that provided more fine-grained control of the excess risk of ERM [e.g. BBM05; Kol06; BM06]. A key idea emerging from this line of work is localization. This concept, and in particular the iterative localization method of Koltchinskii [Kol06], plays an important role in our development. We refer the reader to the books [Kol11; Wai19], as well as the recent articles [LRS15; KRV22] for more on this idea.

Focusing on the task of regression with square loss, upper bounds on the excess risk of ERM are available for many classes of predictors, including finite [e.g. Aud07; JRT08; LM09], linear [e.g. LM16b; Oli16; Mou22], and convex classes [e.g. LM16a; Men14; LRS15]. A key development in this area over the last decade has been the realization that such bounds can be obtained under much weaker assumptions than previously thought, owing to the fact that only one-sided control of a certain empirical process is needed, and which can be obtained under very weak assumptions [Men14; KM15; Oli16]. The line of work most closely related to ours is the one on random-design linear regression [AC11; HKZ12; Oli16; LM16b; Sau18; Mou22; EE23], and we view our work as an extension of this literature. We review these results in more detail in Section 2.

Finally, and on a more conceptual level, our work is related to the recent effort to understand the effect of feature learning on the performance of neural networks [e.g. Bac17; Gho+20; Ba+22]. Beyond this conceptual connection however, our work is quite distinct from this literature. Among other things, our setting is more general since we consider arbitrary features maps. In the same vein, it is worth mentioning the line of work on multiple kernel learning [e.g. Lan+04; GA11; SD16], although we are not aware of results from this literature that are directly relevant to our setup.

Challenges. Our class of predictors is somewhat unstructured (e.g. it is in general non-convex), so that off-the-shelf results from the above literature are not directly applicable. Nevertheless, the analysis of the performance of ERM on linear classes provides a good starting point as we review in Section 2. Compared to that setting however, we are faced with two additional challenges. First, we need to control an additional source of error arising from the fact that ERM might select a suboptimal feature map. Second, we are lead to study the suprema of certain T -indexed empirical processes, which in the linear setting reduce to single random variables that are easily dealt with.

Organization. The rest of the paper is organized as follows. In Section 2, we review known results on the excess risk of linear regression under square loss. In Section 3, we state our main results that hold for the excess risk of ERM for general index sets T . In Section 4, we specialize our analysis to the case where the index set T is finite, obtain more explicit guarantees, and discuss their implications on the sparse linear regression problem. We conclude in Section 5 with a brief discussion.

̸

2 Background

The goal of this section is to provide more context for our results. We review known results on the excess risk of ERM over linear classes, which corresponds in our setting to the special case where the set T indexing the feature maps is a singleton. As such, to avoid introducing further notation, we use the one from the previous section, while dropping the dependence on t whenever it occurs.

In the setting of linear regression with square loss, and when the sample covariance matrix of the feature map is invertible, there is a unique empirical risk minimizer and its excess risk admits an explicit expression. Specifically, define

and let w ∗ denote the unique minimizer of the risk R ( w ) . 2 Then, an elementary calculation shows that when Σ n is invertible, there is a unique empirical risk minimizer and it satisfies

Furthermore, since the risk is a quadratic function of w whose gradient at w ∗ vanishes, replacing R ( ˆ w n ) by the equivalent exact second order Taylor expansion around w ∗ yields

While exact, this expression is not readily interpretable. For example, how fast does this excess risk go to 0 as a function of the sample size? The following classical result from asymptotic statistics [e.g. Whi82; LC06; Vaa98] makes this rate more explicit. To state it, we define

Theorem 1. Assume that for all j ∈ [ d ] , E [ ϕ 2 j ( X ) ] < ∞ , E [ Y 2 ] < ∞ , and E[ ∥ g ( X,Y ) ∥ 2 Σ -1 ] < ∞ . Then, as n →∞ ,

where Z ∼ N (0 , Σ -1 / 2 G Σ -1 / 2 ) . In particular, for any δ ∈ (0 , 0 . 1) ,

where Q X ( p ) := inf { x ∈ R | P( X ≤ x ) ≥ p } is the quantile function of a random variable X , and where we write a ≍ b to mean that there exists absolute constants C, c such that c · b ≤ a ≤ C · b . In the above statement, they can be taken as C = 1 and c = 1 / 32 .

We provide a proof in Appendix A for completeness. For our purposes, this theorem is most easily interpreted as follows: for large enough n and small enough δ , if the excess risk of ERM is bounded by some quantity with probability at least 1 -δ , then this quantity is, up to a constant, at least as large as the right-hand side of the second displayed equation divided by n . While our primary interest is in non-asymptotic bounds, this asymptotic result, by virtue of its exactness, provides us with a benchmark against which such bounds can be compared. In particular, it identifies the quantity E[ ∥ g ( X,Y ) ∥ 2 Σ -1 ] as an intrinsic parameter determining the excess risk of ERM on this problem.

For large enough n , Theorem 1 gives an interpretable expression for the excess risk. However, it says nothing about how large n needs to be for this expression to be accurate. This motivates a non-asymptotic analysis of the excess risk of ERM, which has been carried out numerous times in recent years [e.g. Oli16; LM16b; EE23]. A goal of this literature has been to obtain upper bounds on the excess risk of ERM that hold in probability under weak moment assumptions, building on the observation that this is indeed possible [Men14]. The following theorem is comparable to the best known result in this area. We leave the proof to Appendix B. To state it, we define

Theorem 2. Assume that for all j ∈ [ d ] , E [ ϕ 4 j ( X ) ] < ∞ , E [ Y 2 ] < ∞ , and E[ ∥ g ( X,Y ) ∥ 2 Σ -1 ] < ∞ . Let δ ∈ (0 , 1) . If

then with probability at least 1 -δ ,

At a high-level, this result says that above a certain explicit minimal sample size, the asymptotic expression of the excess risk of Theorem 1 is correct, up to a significantly worse dependence on δ . The restriction on the sample size is almost the best one can hope for. To see why, note that to get guarantees on the excess risk of any empirical risk minimizer, we need at least that Σ n is invertible, otherwise there exists an empirical risk minimizer arbitrarily far away from w ∗ . To get quantitative guarantees, we need slightly more control in the form of a lower bound on λ min (Σ -1 / 2 Σ n Σ -1 / 2 ) . We refer the reader to a more detailed discussion in [EME24, Section 5].

This result has two key qualities, which we aim to reproduce in our results. First, it is assumptionlean, requiring nothing more than a fourth moment assumption on the coordinates of the feature map compared to Theorem 1. Second, it recovers the right dependence on the intrinsic parameter E[ ∥ g ( X,Y ) ∥ 2 Σ -1 ] identified in Theorem 1. A downside of this generality is the bad dependence on δ . Without further assumptions, this cannot be improved; we refer the reader to the recent literature on robust linear regression for more on this issue [e.g. LM19; LL20; EME24].

3 Main Results

In this section we state our main results. They are most easily seen as extensions of Theorems 1 and 2 for general index sets T . In Section 3.1, we study the asymptotics of the excess risk of ERM in our setting, and in Section 3.2, we present a non-asymptotic upper bound on the excess risk.

To state our results, we require additional definitions and notation. We start with the population and the sample covariance matrices

We define the following collection of minimizers,

the first is uniquely defined, while the second is set-valued in general. We define the gradient of the loss at these minimizers and their corresponding covariance matrices

Finally, we introduce the following processes which play a key role in our development

as well as, for t ∗ ∈ T ∗ and t ∈ T \ T ∗ ,

We note that the process (∆ n ( t, t ∗ )) t ∈T \T ∗ is an empirical process (see [VW96] for an introduction), while (Λ n ( t )) t ∈T and ( G n ( t )) t ∈T are partial suprema of empirical processes. In the sequel, we will slightly abuse this terminology, and call all of these empirical processes, with the understanding that they can be viewed as one with more indexing. We will further assume that these processes are separable; see [BLM13, p.305-306] for a definition. This covers a wide range of applications, while avoiding delicate measurability issues. The suprema of such separable processes, which is the only way they enter our results, can be studied by taking the supremum over a countable dense subset of the index set. Therefore, without loss of generality, we assume that T is countable.

Finally, in line with the literature on the theory of empirical processes [VW96], we say that a sequence of empirical processes is Glivenko-Cantelli if, when rescaled by n -1 / 2 , the supremum of their absolute value taken over their index set converges to zero in probability as n →∞ . In other words, the weak law of large numbers holds uniformly over the index set. Similarly, we say that a sequence of empirical processes is Donsker if it converges in distribution to its limiting Gaussian process. 3 In other words, the central limit theorem holds uniformly over the index set.

3.1 Asymptotic result

Our first main result is an asymptotic characterization of the quantiles of the excess risk of any sequence of empirical risk minimizers in our setting, which vastly generalizes that of Theorem 1.

̸

Theorem 3. Assume that T ∗ = ∅ and for some t ∗ ∈ T ∗ , assume that the empirical processes (Λ n ( t )) t ∈T , (∆( t, t ∗ )) t ∈T \T ∗ and ( G n ( t )) t ∈T are Glivenko-Cantelli. Then, for all ε > 0 ,

Furthermore, if the sequence of processes ( G n ( t )) t ∈T is Donsker, then for any δ ∈ (0 , 1) ,

where Z -:= inf s ∈T ∗ ∥ Z ( s ) ∥ 2 2 , Z + := sup s ∈T ∗ ∥ Z ( s ) ∥ 2 2 , and ( Z ( t )) t ∈T is a mean-zero Gaussian process with covariance function E[ Z ( t ) Z ( s ) T ] = Σ -1 / 2 ( t ) G ( t, s )Σ -1 / 2 ( s ) for all t, s ∈ T .

We note that, up to a factor of two in the upper bound on the asymptotic quantiles, Theorem 3 reduces to Theorem 1 when T is a singleton, with the exact same assumptions. We are not aware of comparable results in the literature. The proof of Theorem 3 can be found in Appendix D.

Remark 1 . For small δ , the upper bound admits the more interpretable expression

Furthermore, if T ∗ is finite, the first term can be upper bounded as

To see why Theorem 3 is surprising, let us first focus on the case where T ∗ has a unique element t ∗ , so that Z + = Z -d = ∥ Z ∥ 2 2 where Z ∼ N (0 , Σ -1 / 2 ( t ∗ ) G ( t ∗ , t ∗ )Σ -1 / 2 ( t ∗ )) . Now consider the oracle procedure, which knows beforehand what the optimal feature map t ∗ is, and outputs t ∗ and a minimizer of R n ( t ∗ , w ) . Theorem 3 says that, up to a factor of two, the asymptotic quantiles of the excess risk of ERM, which needs to learn over the large class ∪ t ∈T { x ↦→⟨ w,ϕ t ( x ) ⟩ | w ∈ R d } , coincide with those of the oracle procedure (by Theorem 1), which only needs to learn over the linear class { x ↦→⟨ w,ϕ t ∗ ( x ) ⟩ | w ∈ R d } !

More generally, Theorem 3 establishes that asymptotically, any ERM picks a near-optimal feature map with probability one. It furthers shows that the asymptotic quantiles of the excess risk of any sequence of ERMs is controlled from above and below by those of the extrema of the limiting Gaussian process of ( G n ( t )) t ∈T on the set of optimal feature maps T ∗ . This is surprising, as it implies that asymptotically, and outside of its role in determining whether the assumptions of Theorem 3 hold, the global complexity of the set T is irrelevant to the excess risk of ERM.

Finally, we note that the Glivenko-Cantelli and Donsker assumptions in Theorem 3 can equivalently be viewed as restrictions on the size of T , for distribution and process dependent notions of size. We refer the reader to the books [VW96; GN15] for more on this connection. With this observation, the main takeaway from Theorem 3 can be stated as follows.

Asymptotically, if T is not too large, the excess risk of ERM depends, at worst, only on the complexity of the set of optimal feature maps T ∗ , and is independent of the global complexity of T .

3.2 Non-asymptotic result

The result in Theorem 3 hints at a dramatic localization phenomenon, whereby the influence of the size and complexity of the collection of feature maps ( ϕ t ) t ∈T on the excess risk of ERM vanishes as n → ∞ under appropriate assumptions. The root of this localization phenomenon is the first statement of Theorem 3: eventually, ERM picks near-optimal feature maps with probability approaching one. For small enough sample sizes however, it is clear that ERM is likely to select suboptimal feature maps, so that this localization phenomenon cannot hold uniformly over n . This raises a host of questions: (i) How fast, as measured by the sample size, does ERM learn the optimal feature map? (ii) What is the effect of this localization on the rate of decay of the excess risk of ERM non-asymptotically? (iii) What properties of the feature maps ( ϕ t ) t ∈T influence these rates?

Our answers to these questions in this very general setting are formally expressed in Theorem 4 below. To state it, we define the following parameter

where the supremum is taken over vectors ( v t ) t ∈T such that ∑ t ∈T ∥ v t ∥ 2 2 = 1 . For n ∈ N and δ ∈ (0 , 1) , we define the set function F n,δ , for any subset S ⊂ T , by

This map acts as a contraction as shown in the next lemma, whose proof is deferred to Appendix E. For a function f , we use f k to denote f k ( x ) := f ( f k -1 ( x )) with f 0 ( x ) := x .

̸

Lemma 1. Let n ∈ N , δ ∈ (0 , 1) , and assume that T ∗ = ∅ . Then for all k ∈ N ∪ { 0 } ,

F k +1 n,δ ( T ) ⊆ F k n,δ ( T ) .

̸

With these definitions, we now state the second main result of the paper. A proof is in Appendix F. Theorem 4. Assume that T ∗ = ∅ , E[ Y 2 ] < ∞ , ∀ ( t, j ) ∈ T × [ d ] , E[ ϕ 2 t,j ( X )] < ∞ , and E[ ∥ g ( t, ( X,Y )) ∥ 2 -1 ] < ∞ . Let δ ∈ (0 , 1) and k ∈ N . If, for some t ∗ ∈ T ∗ , n satisfies

Σ ( t )

then, with probability at least 1 -δ ,

where the processes Λ n , ∆ n , and G n are as in (3) and (4).

We make a few remarks before interpreting the content of the theorem. First, we note that when the index set T is a singleton, the last term in the sample size restriction vanishes, while the first matches the sample size restriction from Theorem 2 after an application of Lemma 3 below; further taking k = 1 in Theorem 4 recovers the upper bound on the excess risk of Theorem 2 up to a constant factor. Theorem 4 may therefore be viewed as a broad generalization of Theorem 2. Second, under Assumption 1 below, and by the second item of Lemma 1, the upper bound on the excess risk in Theorem 4 eventually matches the main term in the asymptotic bound of Theorem 3 as can be seen from (5), in the same way that Theorem 2 achieves this when compared with Theorem 1. Finally, the statement of Theorem 4 is very general, and in fact, too general for us to be able to interpret it precisely. As such, we will discuss it in the context of the following assumption.

Assumption 1. There exists constants C Λ , C ∆ , and C G independent of the sample size, but possibly dependent on the remaining parameters of the problem, such that for all n ∈ N ,

where Λ n , ∆ n , and G n are as in (3) and (4).

and

These assumptions can be equivalently viewed as a restriction on the appropriately measured size of the index set T [VW96; GN15], and are slightly stronger than the assumptions of Theorem 3. They always hold for finite index sets, and we will derive in Section 4 explicit estimates of the constants in Assumption 1 in terms of moments of the feature maps and target as well as the cardinality of T .

Let us now interpret the content of Theorem 4, which comes with a free parameter k , in the context of Assumption 1. We fix k here, and discuss its choice below. First, recalling the definition of F n,δ , this result says that above a certain sample size, both the suboptimality of the feature map picked by ERM and its excess risk decay at the fast rate n -1 , answering the first question we raised at the beginning of the section. Second, this result provides an upper bound on the excess risk of ERM that depends on the index set T only through the size of shrinking subsets S n,δ,k , which might be large for small n , but which by Lemma 1 converge to the set of optimal feature maps T ∗ as n →∞ . This transparently shows the effect of the localization phenomenon on the rate of decay of the excess risk of ERM, answering the second question we raised. Finally, looking at the definition of S n,δ,k , this result identifies the size of the sublevel sets of the suboptimality function R ( t, w ∗ ( t )) -R ∗ defined over feature maps as a relevant property of the collection of feature maps ( ϕ t ) t ∈T that influences the rate of convergence of the excess risk of ERM in this setting, answering the final question we raised.

Finally, let us turn to the choice of k . Practically, we select the one that minimizes the bound on the excess risk. Looking at the first item of Lemma 1, this optimal k balances the following trade-off: on the one hand, for small k , applications of F n,δ/ 2 k constrain the input set more severely, but only a few iterations are performed; on the other hand, larger values of k allow more iterations, but at the cost of more weakly constraining the input set per application.

Stepping back, there are two main takeaways from Theorem 4. Firstly, and on a conceptual level, it shows that feature learning is easy when the suboptimality function R ( t, w ∗ ( t )) -R ∗ , defined over the set of features maps, has small sublevel sets. Secondly, and on a technical level, it provides a template which can be used to derive more explicit excess risk bounds on ERM given estimates on the expected suprema of the relevant empirical processes. Deriving such accurate estimates for infinite T is a highly non-trivial task, and cannot be done at the level of generality we have been operating at. The case of finite T however is tractable in a general setting as we discuss in the next section.

4 Case study: Finite index sets

In this section, we focus on the case where the index set T is finite, and aim, among other things, at establishing explicit estimates on the various expected suprema appearing in Theorem 4 in terms of moments of the feature maps and of the target. This problem becomes tractable in the case of finite T because, roughly speaking, a worst-case analysis still yields non-trivial upper bounds. This is decidedly not the case when T is infinite, in which case these expected suprema can be infinite.

We start with a slight strengthening of Theorem 3, whose assumptions reduce to simple moments conditions when T is finite. The straightforward proof can be found in Appendix H.

Corollary 1. Assume that T is finite, for all ( t, j ) ∈ T × [ d ] , E [ ϕ 2 t,j ( X ) ] < ∞ , E [ Y 2 ] < ∞ , and for all t ∈ T , E[ ∥ g ( t, ( X,Y )) ∥ 2 Σ -1 ( t ) ] < ∞ . Then

Furthermore, for any δ ∈ (0 , 1) ,

where Z -:= min s ∈T ∗ ∥ Z s ∥ 2 2 , Z + := max s ∈T ∗ ∥ Z s ∥ 2 2 , and the random vectors ( Z t ) t ∈T are jointly Gaussian with mean zero and covariance E[ Z t Z T s ] = Σ -1 / 2 ( t ) G ( t, s )Σ -1 / 2 ( s ) for all t, s ∈ T . In particular, if T ∗ = { t ∗ } , then

The conclusions of Corollary 1 differ from those of Theorem 3 in two aspects. First, the feature map picked by ERM is guaranteed to be optimal rather than near-optimal with probability converging to

one. Second, the upper bound on the asymptotic quantiles is improved by a factor of two, yielding the exact distribution of the rescaled excess risk when T ∗ is a singleton.

Making Theorem 4 more explicit is a more laborious task. We recall here two known results that allow us to accomplish this. We start with the following bounds on the expectation of the supremum of a finitely-indexed empirical process, which we will later use to bound the suprema of the processes ( G n ( s )) s ∈S and (∆ n ( t, t ∗ )) t ∈T \T ∗ appearing in Theorem 4. A proof can be found in Appendix G.

Lemma 2. Let n, d ∈ N , and let Z be a random element taking value in a set Z , and let ( Z i ) n i =1 be i.i.d. samples with the same distribution as Z . Let F be a finite collection of R d -valued measurable functions. Define

, and let E n ( f ) := √ n · ( n -1 ∑ n i =1 f ( Z i ) -E[ f ( Z )]) . Then, we have 1 2 · σ ( F ) + 1 4 · r n ( F ) √ n ≤ E [ max f ∈F ∥ E n ( f ) ∥ 2 2 ] 1 / 2 ≤ c ( |F| ) · σ ( F ) + c 2 ( |F| ) · r n ( F ) √ n , where c ( m ) := 5 √ 1 + log m .

Lemma 2 allows us to compute the expected supremum of a finitely-indexed empirical process, up to log factors in the size of the index set. It is known that these factors cannot be removed from the upper bound nor added to the lower bound without more assumptions, we refer the reader to a related discussion in [Tro16]. Finally, while the term r n ( F ) might grow with n , by bounding the maximum with the sum, it grows at most as √ n . In many applications however, the random vectors f ( Z ) are bounded almost surely, so that r n ( F ) is of order one, which justifies our presentation choice.

The second result we recall is the expectation version of a one sided Matrix Bernstein inequality due to Tropp [Tro15]. We use it below to bound the supremum of the process (Λ n ( t )) t ∈T appearing in Theorem 4. We do not known of a matching non-asymptotic lower bound, but an asymptotic one is known [EME24, Proposition 17]. Upper and lower bounds similar to those of Lemma 2 hold if one considers the expected operator norm instead of only the maximum eigenvalue [Tro16, Section 7].

Lemma 3 ([Tro15], Theorem 6.6.1.) . Let n, d ∈ N and for each i ∈ [ n ] , let Z i ∈ R d × d be i.i.d. positive semi-definite matrices with the same distribution as Z . Define

Then, we have

Equipped with these estimates, we may now control the expected suprema of the empirical processes appearing in Theorem 4. To apply Lemma 2, define the following classes, for S ⊂ T and t ∗ ∈ T ∗

Applying Lemma 2 on G ( S ) bounds the expected supremum of the process ( G n ( s )) s ∈S while applying it on D ( t ∗ ) bounds that of (∆ n ( t, t ∗ )) t ∈T \T ∗ . To control the supremum of (Λ n ( t )) t ∈T , the key idea is to notice that it can be expressed as the maximum eigenvalue of a block diagonal matrix whose blocks are √ n ( I -Σ -1 / 2 ( t )Σ n ( t )Σ -1 / 2 ( t )) . Looking at Lemma 3, the relevant parameter is therefore a block diagonal matrix V with the following blocks

As the bound in Lemma 3 depends only on the maximum eigenvalue of V , the ordering of the blocks does not matter. Putting together these estimates, we arrive at a fully explicit version of Theorem 4.

Corollary 2. Assume that T is finite and that for all ( t, j ) ∈ T × [ d ] , E [ ϕ 4 t,j ( X ) ] < ∞ , E [ Y 4 ] < ∞ . Let δ ∈ (0 , 1) , k ∈ [1 + |T \ T ∗ | ] , and c ( · ) , σ 2 ( · ) , r n ( · ) as in Lemma 2. If, for some t ∗ ∈ T ∗ ,

then, with probability at least 1 -δ

and ˜ F n,δ ( S ) is the same as F n,δ ( S ) defined in (7) but with A ( S ) replacing E[sup s ∈S G 2 n ( s )] .

We make a few remarks about Corollary 2; a proof sketch is in Appendix I. The set function A ( S ) controlling the contraction rate of the map ˜ F n,δ as well as the excess risk, has a pleasantly simple form. To first order, and ignoring constants, it is given by

As such, as n →∞ and by Lemma 1, the upper bound on the excess risk in Corollary 2 matches the main term in the asymptotic rate derived in Theorem 3, as can be seen from (6). As the sets ˜ S n,δ,k are shrinking with n , the above expression clearly shows the decaying effect of the global complexity of T on the excess risk. Finally, we note that the restriction on k in Corollary 2 is there only because after at most that many iterations, a fixed point is reached, and further iterations worsen the bound. We conclude this section with an example of an application of our results.

Example 1 (Sparse linear regression) . Consider the sparse linear regression problem, and in particular the best subset selection (BSS) procedure [Mil02; HTF09]. This procedure corresponds to ERM over the restricted linear class { x ↦→⟨ w,ϕ ( x ) ⟩ | ∥ w ∥ 0 ≤ s } in the linear regression setup of Section 2, where ∥ w ∥ 0 is the number of non-zero entries of w and s ∈ [ d ] is a user-chosen sparsity level.

The problem of computing the BSS procedure has attracted a lot of attention recently. While NP-hard and therefore difficult in the worst case [Nat95], Bertsimas et al. [BKM16] showed that it can be tractable on practical instances of moderate size. Since then, a rich literature has emerged that devises increasingly efficient methods [e.g. Hua+18; BP20; HMS22; Guy+24]. By comparison, the statistical performance of the BSS procedure is not yet completely understood as we discuss below.

To see how the sparse linear regression problem fits in our feature learning setting, notice that { x ↦→⟨ w,ϕ ( x ) ⟩ | ∥ w ∥ 0 ≤ s } = { x ↦→⟨ v, ϕ t ( x ) ⟩ | ( t, v ) ∈ T × R s } where T is the set of all subsets of [ d ] of size s , and ϕ t ( x ) := ( ϕ j 1 ( x ) , ϕ j 2 ( x ) , . . . , ϕ j s ( x )) ∈ R s where ( j 1 , j 2 , . . . , j s ) are the elements of t in increasing order. As such, Corollaries 1 and 2 are immediately applicable and provide general statements on the performance of an arbitrary BSS procedure. To simplify the discussion, we assume for the rest of the example that there is a unique risk minimizer w ∗ satisfying ∥ w ∗ ∥ 0 = s .

On the recovery side, the first item of Corollary 1 guarantees that we asymptotically exactly recover the support of w ∗ . Non-asymptotically, the first item of Corollary 2 shows that if n further satisfies

then with probability at least 1 -δ , the BSS procedure recovers the support of w ∗ . Equivalently, these two statements say that for large enough n , the BSS procedure coincides with the oracle procedure which knows the support of w ∗ a priori and outputs an ERM from the optimal linear class.

In practice however, the interesting regime is when n is only moderately large. Corollary 2 provides our guarantee in this case, and as such, we turn our attention to the sample size restriction (8). Typically, we expect the main restriction to come from the first term, which in this case is given by λ max ( V ) · s log( d/s ) , up to constants and lower order terms. This is because if an intercept is and

where, for S ⊂ T ,

included, i.e. ϕ 1 ( X ) = 1 , then λ max ( V ) ≥ s -1 , so the first term scales as s 2 log( d/s ) at least, while the remaining terms typically grow more slowly with s . As a concrete example, when ϕ ( X ) is a Gaussian vector, λ max ( V ) = s +1 , so in this case the estimate s 2 log( d/s ) is tight. Under this sample size restriction, and if ε := Y -⟨ w ∗ , ϕ ( X ) ⟩ satisfies E [ ε 2 | X ] ≤ σ 2 , Corollary 2 upper bounds the excess risk by ( σ 2 s/n ) · a n for a sequence of decreasing distribution-dependent constants a n converging to one as n →∞ , ignoring the dependence on δ and absolute constants.

The closest existing result in the literature we are aware of is due to Shen et al. [She+13], who arrived at comparable conclusions but in a substantially different setting. In particular, their result was obtained in the setting n, d →∞ , with an implicit assumption on the distribution of ϕ ( X ) [She+13, Equation 2], and dealt with the in-sample prediction risk instead of the excess risk. Another closely related result is due to Raskutti et al. [RWY11] who showed that the minimax expected excess risk in a well-specified fixed-design setting is, up to constants, σ 2 s log( d/s ) /n ; see also [Bac24, Chapter 8]. Our results show that for moderate n , in the random-design setting, and when focusing on a single instance, the log( d/s ) factor can be replaced with another factor that decays to one as n →∞ .

Coming back to the sample size restriction discussion, we strongly suspect that the factor s log( d/s ) is suboptimal, but we are unsure what the correct dependence is, even under Gaussian ϕ ( X ) . Indeed, this factor comes from the logarithmic factor in Lemma 3, when applied to the block diagonal matrix with blocks √ n ( I -Σ -1 / 2 ( t )Σ n ( t )Σ -1 / 2 ( t ) ) . One can improve this factor by instead using versions of this inequality based on the intrinsic dimension [Tro15, Chapter 7]. However, this is also unlikely to be tight. Roughly speaking, this is because such logarithmic factors are tight only when the eigenvalues of the random matrix are near-independent. This is certainly not the case for the block diagonal matrix we are considering, since its blocks are sample covariance matrices of sub-vectors of the same random vector ϕ ( X ) . Capturing this dependence is beyond our reach and likely requires new tools; we refer the interested reader to the recent articles [vHan17; LvHY18; BBvH23].

5 Conclusion

Broadly speaking, there are two main conclusions one can draw from this work. Firstly, in the large sample regime, and if the set of candidate feature maps is not too large under an appropriate measure of size, asking a model to additionally pick a feature map on top of learning a linear predictor has a negligible effect on the excess risk of ERM on regression problems with square loss. Secondly, for moderate sample sizes, the magnitude of this effect depends on the appropriately measured size of the sublevel sets of the suboptimality function t ↦→ R ( t, w ∗ ( t )) -R ∗ . Plainly, learning feature maps is easy when only a small subset of them is good, as the bad ones can be quickly discarded.

The most tantalizing aspect of our results is their potential in explaining the experiments in [Zha+21]. It was shown there that complex neural networks trained by ERM were able to achieve good performance despite being expressive enough to fit random labels. This is paradoxical if one assumes that the performance of ERM is driven by the complexity of the model class. Our results refute this assumption for a generic collection of feature-learning-based models. While there are many works offering explanations for this apparent paradox (see e.g. [BMR21] for a survey), we are not aware of one that shows the vanishing influence of the size of the model class on the excess risk as Theorems 3 and 4 show. Formally connecting our statements to these experiments is beyond what we achieved here, yet, we believe that the new perspective we took might generate useful insights in this area.

We conclude by outlining a few limitations of our work. Firstly, we do not deal with the question of how to solve the ERM problem. Our focus is on understanding its statistical performance, and our setting is so general that such a question cannot be meaningfully tackled. Continuing on this last point, while the generality of our results is desirable in some aspects, it is detrimental in others. As an example, it would be desirable to specialize our results from Section 3 to specific infinite collections of feature maps used in practice. Let us also mention that it is a priori unclear whether ERM is an optimal procedure, in a minimax sense, for the model classes we consider; we suspect that recently developed tools might be relevant to address this question [Mou22]. Finally, while we focused on the case of regression with square loss, this was mostly done to simplify the presentation. Indeed, the only property of the loss used in the proofs is the exactness of its second order Taylor expansion. This is however not required if one can control the error term from above and below. It is known how to do this for many loss functions [e.g. OB21; EE23], and most importantly for logistic regression [Bac10; Bac14]. We have purposefully selected generic notation to make translating such arguments easier.

Acknowledgments and Disclosure of Funding

Resources used in preparing this research were provided in part by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute. CM acknowledges the support of the Natural Sciences and Engineering Research Council of Canada (NSERC), RGPIN-2021-03445. MAE was partially supported by NSERC Grant [2019-06167], CIFAR AI Chairs program, and CIFAR AI Catalyst grant.

References

[AC11] J.-Y. Audibert and O. Catoni. 'Robust Linear Least Squares Regression'. In: The Annals of Statistics (2011). URL.

[Aud07] J.-y. Audibert. 'Progressive Mixture Rules Are Deviation Suboptimal'. In: Advances in Neural Information Processing Systems . 2007. URL.

[Ba+22] J. Ba, M. A. Erdogdu, T. Suzuki, Z. Wang, D. Wu, and G. Yang. 'High-Dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representation'. In: Advances in Neural Information Processing Systems (Dec. 6, 2022). URL.

[Bac10] F. Bach. 'Self-Concordant Analysis for Logistic Regression'. In: Electronic Journal of Statistics (Jan. 2010). DOI: 10.1214/09-EJS521 .

[Bac14] F. Bach. 'Adaptivity of Averaged Stochastic Gradient Descent to Local Strong Convexity for Logistic Regression'. In: Journal of Machine Learning Research (2014). URL.

[Bac17] F. Bach. 'Breaking the Curse of Dimensionality with Convex Neural Networks'. In: Journal of Machine Learning Research (2017). URL.

[Bac24] F. Bach. Learning Theory from First Principles . Dec. 24, 2024.

[BBM05] P. L. Bartlett, O. Bousquet, and S. Mendelson. 'Local Rademacher Complexities'. In: The Annals of Statistics (Aug. 2005). DOI: 10.1214/009053605000000282 .

[BBvH23] A. S. Bandeira, M. T. Boedihardjo, and R. van Handel. 'Matrix Concentration Inequalities and Free Probability'. In: Inventiones mathematicae (Oct. 1, 2023). DOI: 10.1007/s00222-023-01204-6 .

[BKM16] D. Bertsimas, A. King, and R. Mazumder. 'Best Subset Selection via a Modern Optimization Lens'. In: The Annals of Statistics (Apr. 2016). DOI: 10.1214/15-AOS1388 .

[BLM00] S. Boucheron, G. Lugosi, and P. Massart. 'A Sharp Concentration Inequality with Applications'. In: Random Structures & Algorithms (2000). DOI: 10.1002/(SICI) 1098-2418(200005)16:3<277::AID-RSA4>3.0.CO;2-1 .

[BLM13] S. Boucheron, G. Lugosi, and P. Massart. Concentration Inequalities: A Nonasymptotic Theory of Independence . Feb. 7, 2013.

[BM06] P. L. Bartlett and S. Mendelson. 'Empirical Minimization'. In: Probability Theory and Related Fields (July 1, 2006). DOI: 10.1007/s00440-005-0462-3 .

[BMR21] P. L. Bartlett, A. Montanari, and A. Rakhlin. 'Deep Learning: A Statistical Viewpoint'. In: Acta Numerica (May 2021). DOI: 10.1017/S0962492921000027 .

[Bou02] O. Bousquet. 'A Bennett Concentration Inequality and Its Application to Suprema of Empirical Processes'. In: Comptes Rendus Mathematique (Jan. 1, 2002). DOI: 10. 1016/S1631-073X(02)02292-6 .

[BP20] D. Bertsimas and B. V. Parys. 'Sparse High-Dimensional Regression: Exact Scalable Algorithms and Phase Transitions'. In: The Annals of Statistics (Feb. 2020). DOI: 10.1214/18-AOS1804 .

[BPV20] D. Bertsimas, J. Pauphilet, and B. Van Parys. 'Sparse Regression: Scalable Algorithms and Empirical Performance'. In: Statistical Science (2020). URL.

[COB19] L. Chizat, E. Oyallon, and F. Bach. 'On Lazy Training in Differentiable Programming'. In: Advances in Neural Information Processing Systems . 2019. URL.

[DI17] G. David and Z. Ilias. 'High Dimensional Regression with Binary Coefficients. Estimating Squared Error and a Phase Transtition'. In: Proceedings of the 2017 Conference on Learning Theory . June 18, 2017. URL.

[EE23] A. El Hanchi and M. A. Erdogdu. 'Optimal Excess Risk Bounds for Empirical Risk Minimization on p -Norm Linear Regression'. In: Advances in Neural Information Processing Systems (Dec. 15, 2023). URL.

[EME24] A. El Hanchi, C. Maddison, and M. Erdogdu. 'Minimax Linear Regression under the Quantile Risk'. In: Proceedings of Thirty Seventh Conference on Learning Theory . June 30, 2024. URL.

[GA11] M. Gönen and E. Alpaydin. 'Multiple Kernel Learning Algorithms'. In: Journal of Machine Learning Research (2011). URL.

[Gho+19] B. Ghorbani, S. Mei, T. Misiakiewicz, and A. Montanari. 'Limitations of Lazy Training of Two-layers Neural Network'. In: Advances in Neural Information Processing Systems . 2019. URL.

[Gho+20] B. Ghorbani, S. Mei, T. Misiakiewicz, and A. Montanari. 'When Do Neural Networks Outperform Kernel Methods?' In: Advances in Neural Information Processing Systems . 2020. URL.

[GN15] E. Giné and R. Nickl. Mathematical Foundations of Infinite-Dimensional Statistical Models . 2015. DOI: 10.1017/CBO9781107337862 .

[GR04] E. Greenshtein and Y. Ritov. 'Persistence in High-Dimensional Linear Predictor Selection and the Virtue of Overparametrization'. In: Bernoulli (Dec. 2004). DOI: 10.3150/ bj/1106314846 .

[Gre06] E. Greenshtein. 'Best Subset Selection, Persistence in High-Dimensional Statistical Learning and Optimization under L1 Constraint'. In: The Annals of Statistics (Oct. 2006). DOI: 10.1214/009053606000000768 .

[Guy+24] T. Guyard, C. Herzet, C. Elvira, and A.-N. Arslan. 'A New Branch-and-Bound Pruning Framework for ℓ 0 -Regularized Problems'. In: Proceedings of the 41st International Conference on Machine Learning . July 8, 2024. URL.

[He+16] K. He, X. Zhang, S. Ren, and J. Sun. 'Deep Residual Learning for Image Recognition'. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . June 2016. DOI: 10.1109/CVPR.2016.90 .

[HKZ12] D. Hsu, S. M. Kakade, and T. Zhang. 'Random Design Analysis of Ridge Regression'. In: Proceedings of the 25th Annual Conference on Learning Theory . June 16, 2012. URL.

[HMS22] H. Hazimeh, R. Mazumder, and A. Saab. 'Sparse Regression at Scale: Branch-andBound Rooted in First-Order Optimization'. In: Mathematical Programming (Nov. 1, 2022). DOI: 10.1007/s10107-021-01712-4 .

[HTF09] T. Hastie, R. Tibshirani, and J. Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Second Edition . Aug. 26, 2009.

[Hua+18] J. Huang, Y. Jiao, Y. Liu, and X. Lu. 'A Constructive Approach to L 0 Penalized Regression'. In: Journal of Machine Learning Research (2018). URL.

[JRT08] A. Juditsky, P. Rigollet, and A. B. Tsybakov. 'Learning by Mirror Averaging'. In: The Annals of Statistics (Oct. 2008). DOI: 10.1214/07-AOS546 .

[Kel+23] J. Kelner, F. Koehler, R. Meka, and D. Rohatgi. 'Feature Adaptation for Sparse Linear Regression'. In: Advances in Neural Information Processing Systems (Dec. 15, 2023). URL.

[KM15] V. Koltchinskii and S. Mendelson. 'Bounding the Smallest Singular Value of a Random Matrix Without Concentration'. In: International Mathematics Research Notices (Jan. 1, 2015). DOI: 10.1093/imrn/rnv096 .

[Kol06] V. Koltchinskii. 'Local Rademacher Complexities and Oracle Inequalities in Risk Minimization'. In: The Annals of Statistics (Dec. 2006). DOI: 10 . 1214 / 009053606000001019 .

[Kol11] V. Koltchinskii. Oracle Inequalities in Empirical Risk Minimization and Sparse Recovery Problems: École D'Été de Probabilités de Saint-Flour XXXVIII-2008 . July 29, 2011.

[KRV22] V. Kanade, P. Rebeschini, and T. Vaskevicius. Exponential Tail Local Rademacher Complexity Risk Bounds Without the Bernstein Condition . Feb. 23, 2022. DOI: 10. 48550/arXiv.2202.11461 .

[KSH12] A. Krizhevsky, I. Sutskever, and G. E. Hinton. 'ImageNet Classification with Deep Convolutional Neural Networks'. In: Advances in Neural Information Processing Systems . 2012. URL.

[Lan+04] G. R. G. Lanckriet, N. Cristianini, P. Bartlett, L. E. Ghaoui, and M. I. Jordan. 'Learning the Kernel Matrix with Semidefinite Programming'. In: Journal of Machine Learning Research (2004). URL.

[LBH15] Y. LeCun, Y. Bengio, and G. Hinton. 'Deep Learning'. In: Nature (May 28, 2015). DOI: 10.1038/nature14539 .

[LC06] E. L. Lehmann and G. Casella. Theory of Point Estimation . May 2, 2006.

[LL20] G. Lecué and M. Lerasle. 'Robust Machine Learning by Median-of-Means: Theory and Practice'. In: The Annals of Statistics (Apr. 2020). DOI: 10.1214/19-AOS1828 .

[LM09] G. Lecué and S. Mendelson. 'Aggregation via Empirical Risk Minimization'. In: Probability Theory and Related Fields (Nov. 1, 2009). DOI: 10.1007/s00440-008-0180-8 .

[LM16a] G. Lecué and S. Mendelson. Learning Subgaussian Classes : Upper and Minimax Bounds . Sept. 17, 2016. DOI: 10.48550/arXiv.1305.4825 .

[LM16b] G. Lecué and S. Mendelson. 'Performance of Empirical Risk Minimization in Linear Aggregation'. In: Bernoulli (Aug. 2016). DOI: 10.3150/15-BEJ701 .

[LM19] G. Lugosi and S. Mendelson. 'Risk Minimization by Median-of-Means Tournaments'. In: Journal of the European Mathematical Society (Dec. 16, 2019). DOI: 10.4171/ jems/937 .

[LRS15] T. Liang, A. Rakhlin, and K. Sridharan. 'Learning with Square Loss: Localization through Offset Rademacher Complexity'. In: Proceedings of The 28th Conference on Learning Theory . June 26, 2015. URL.

[LvHY18] R. Latała, R. van Handel, and P. Youssef. 'The Dimension-Free Structure of Nonhomogeneous Random Matrices'. In: Inventiones mathematicae (Dec. 1, 2018). DOI: 10.1007/s00222-018-0817-x .

[Men14] S. Mendelson. 'Learning without Concentration'. In: Proceedings of The 27th Conference on Learning Theory . May 29, 2014. URL.

[Mil02] A. Miller. Subset Selection in Regression . Apr. 14, 2002. DOI: 10 . 1201 / 9781420035933 .

[Mou22] J. Mourtada. 'Exact Minimax Risk for Linear Least Squares, and the Lower Tail of Sample Covariance Matrices'. In: The Annals of Statistics (Aug. 2022). DOI: 10.1214/ 22-AOS2181 .

[Nat95] B. K. Natarajan. 'Sparse Approximate Solutions to Linear Systems'. In: SIAM Journal on Computing (Apr. 1995). DOI: 10.1137/S0097539792240406 .

[OB21] D. M. Ostrovskii and F. Bach. 'Finite-Sample Analysis of M -Estimators Using SelfConcordance'. In: Electronic Journal of Statistics (Jan. 2021). DOI: 10.1214/20EJS1780 .

[Oli16] R. I. Oliveira. 'The Lower Tail of Random Quadratic Forms with Applications to Ordinary Least Squares'. In: Probability Theory and Related Fields (Dec. 1, 2016). DOI: 10.1007/s00440-016-0738-9 .

[PG99] V. de la Peña and E. Giné. Decoupling: From Dependence to Independence . 1999.

[PWE15] M. Pilanci, M. J. Wainwright, and L. El Ghaoui. 'Sparse Learning via Boolean Relaxations'. In: Mathematical Programming (June 1, 2015). DOI: 10.1007/s10107-0150894-1 .

[RWY11] G. Raskutti, M. J. Wainwright, and B. Yu. 'Minimax Rates of Estimation for HighDimensional Linear Regression Over ℓ q -Balls'. In: IEEE Transactions on Information Theory (Oct. 2011). DOI: 10.1109/TIT.2011.2165799 .

[Sau18] A. Saumard. 'On Optimality of Empirical Risk Minimization in Linear Aggregation'. In: Bernoulli (2018). URL.

[SD16] A. Sinha and J. C. Duchi. 'Learning Kernels with Random Features'. In: Advances in Neural Information Processing Systems . 2016. URL.

[She+13] X. Shen, W. Pan, Y. Zhu, and H. Zhou. 'On Constrained and Regularized HighDimensional Regression'. In: Annals of the Institute of Statistical Mathematics (Oct. 1, 2013). DOI: 10.1007/s10463-012-0396-3 .

[SPZ12] X. Shen, W. Pan, and Y. Zhu. 'Likelihood-Based Selection and Sharp Parameter Estimation'. In: Journal of the American Statistical Association (June 11, 2012). DOI: 10.1080/01621459.2011.645783 .

[Tal96] M. Talagrand. 'New Concentration Inequalities in Product Spaces'. In: Inventiones mathematicae (Nov. 1, 1996). DOI: 10.1007/s002220050108 .

[Tro15] J. A. Tropp. 'An Introduction to Matrix Concentration Inequalities'. In: Found. Trends Mach. Learn. (May 1, 2015). DOI: 10.1561/2200000048 .

[Tro16] J. A. Tropp. 'The Expected Norm of a Sum of Independent Random Matrices: An Elementary Approach'. In: High Dimensional Probability VII . Ed. by C. Houdré, D. M. Mason, P. Reynaud-Bouret, and J. Rosi´ nski. 2016. DOI: 10.1007/978-3-319-40519-3_8 .

[Vaa98] A. W. van der Vaart. Asymptotic Statistics . 1998. DOI: 10.1017/CBO9780511802256 .

[Vas+17] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. ukasz Kaiser, and I. Polosukhin. 'Attention Is All You Need'. In: Advances in Neural Information Processing Systems . 2017. URL.

[VC74] V. Vapnik and A. Chervonenkis. Theory of Pattern Recognition . 1974.

[vHan17] R. van Handel. 'Structured Random Matrices'. In: Convexity and Concentration . 2017. DOI: 10.1007/978-1-4939-7005-6_4 .

[VW96] A. van der Vaart and J. A. Wellner. Weak Convergence and Empirical Processes: With Applications to Statistics . Mar. 14, 1996.

[Wai19] M. J. Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint . 2019. DOI: 10.1017/9781108627771 .

[Whi82] H. White. 'Maximum Likelihood Estimation of Misspecified Models'. In: Econometrica (1982). DOI: 10.2307/1912526 .

[Zha+21] C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. 'Understanding Deep Learning (Still) Requires Rethinking Generalization'. In: Commun. ACM (Feb. 22, 2021). DOI: 10.1145/3446776 .

[Zhu+20] J. Zhu, C. Wen, J. Zhu, H. Zhang, and X. Wang. 'A Polynomial Algorithm for BestSubset Selection Problem'. In: Proceedings of the National Academy of Sciences (Dec. 29, 2020). DOI: 10.1073/pnas.2014241117 .

[ZWJ14] Y. Zhang, M. J. Wainwright, and M. I. Jordan. 'Lower Bounds on the Performance of Polynomial-time Algorithms for Sparse Linear Regression'. In: Proceedings of The 27th Conference on Learning Theory . May 29, 2014. URL.

[ZWJ17] Y. Zhang, M. J. Wainwright, and M. I. Jordan. 'Optimal Prediction for Sparse Linear Models? Lower Bounds for Coordinate-Separable M-estimators'. In: Electronic Journal of Statistics (Jan. 2017). DOI: 10.1214/17-EJS1233 .

[ZZ12] C.-H. Zhang and T. Zhang. 'A General Theory of Concave Regularization for HighDimensional Sparse Estimation Problems'. In: Statistical Science (Nov. 2012). DOI: 10.1214/12-STS399 .

A Proof of Theorem 1

Let A n denote the event that Σ n is invertible. By the weak law of large numbers, Σ n converges to Σ in probability so that lim n →∞ P( A c n ) = 0 . Now on the event A n , we have by (1)

By the continuous mapping theorem, Σ -1 n converges to Σ -1 in probability and by the central limit theorem

Therefore, by Slutsky's theorem

Now since the risk is quadratic and the gradient vanishes at w ∗ ,

where Z is as in the theorem, and where the last statement follows by the continuous mapping theorem. This proves the first statement. The bounds on the quantiles are a consequence of concentration bounds for the norm of Gaussian vectors [e.g. EME24, Corollary 33].

B Proof of Theorem 2

Denote by A n the event that

We show that under the sample size restriction, P( A n ) ≥ 1 -δ/ 2 . Indeed we have the variational representation

Each element in the sum is upper bounded by 1 , and the variance parameter in Bousquet's concentration inequality [Bou02] is given by the parameter L in the statement of the theorem. Applying this inequality yields that with probability at least 1 -δ/ 2

Using Lemma 3 to upper bound the above expectation, and replacing the sample size n in the resulting inequality with the minimal allowed by the theorem proves that P( A n ) ≥ 1 -δ/ 2 for all sample sizes allowable by the theorem. Now on this event we have, using (2),

An elementary calculation shows

so that an application of Markov's inequality yields that there is an event B n that holds with probability at least 1 -δ/ 2 and on which

The union bound P( A n ∩ B n ) = 1 -P( A n ∪ B n ) ≥ 1 -δ finishes the proof.

C Main Lemma

We state here a core lemma, which we use in many of our proofs. To state it, we define, for a function F : S → R on a subset S ⊆ T ,

where the first quantity is the ℓ ∞ norm of the function F , and the remaining are one-sided variants of it. The processes appearing in the next statement are defined in (3) and (4).

̸

Lemma 4. Assume that T ∗ = ∅ and let t ∗ ∈ T ∗ . On the event that ∥ n -1 / 2 ∆ n ( · , t ∗ ) ∥ ∞ , + < 1 and ∥ n -1 / 2 Λ n ∥ ∞ , + < 1 , we have

and

Proof. To lighten the notation, we drop the dependence on n , and write ˆ t instead of ˆ t n . We start with the first statement. First, we note that if ˆ t ∈ T ∗ , then the statement holds trivially as the left-hand side is zero, so we only consider the other case in what follows. For any t ∈ T , define

where the choice of minimizer is arbitrary. With this definition, we have ˆ w n = ˆ w ( ˆ t ) . Now, by definition of ERM,

On the other hand, for any t ∈ T \ T ∗ , we have the decomposition

We study each of the terms of (10) separately, and we start with the first. Note that since we are in the event

the sample covariance matrices Σ n ( t ) are invertible for all t ∈ T , so that ˆ w ( t ) is uniquely defined and satisfies

Furthermore, since the function w ↦→ R n ( t, w ) is quadratic in w and its gradient vanishes at its minimizer ˆ w ( t ) , we have

where the last equality follows from (11). To bound this last term, define

Then we have,

Finally, the second term of (10) is lower bounded by

Combining (13) and (12) lower bounds the first term of (10), while (14) lower bounds the second. Combining the resulting lower bound on (10) with (9) and rearranging yields the first statement.

For the upper bound in the second statement, note that for all t ∈ T ,

where the second line follows from (11). In particular the inequality holds for ˆ t . The lower bound holds by a similar argument.

D Proof of Theorem 3

Consistency of ˆ t n . We want to show that, as n →∞ ,

Using the notation introduced in Appendix C, the Glivenko-Cantelli assumptions in Theorem 3 amount to the statements that, for some t ∗ ∈ T ∗ , and as n →∞ ,

Let A n denote the event that both ∥ n -1 / 2 Λ n ∥ ∞ < 1 and ∥ n -1 / 2 ∆ n ( · , t ∗ ) ∥ ∞ < 1 . The union bound and (16) show that

Furthermore, on the event A n , the first bound of Lemma 4 holds, and bounding n -1 G 2 n ( ˆ t n ) by ∥ n -1 / 2 G n ∥ 2 ∞ yields that on A n

Now let ε > 0 , and denote by B n ( ε ) the event that the right hand side of (17) is strictly larger than ε . Then the statements (16) together with the continuous mapping theorem show that lim n →∞ P( B n ( ε )) = 0 . Therefore, again by (17), we have

and taking n →∞ proves (15).

Asymptotic quantiles. We start with the upper bound. We have the simple decomposition

Now on the event A n defined above, we have, by an application of Lemma 4, combining the two bounds in the lemma along with (18), that the rescaled excess risk is upper bounded by

From the Glivenko-Cantelli assumptions (16), the first three factors converge in probability to 1 . Our aim will be to bound the upper tail of the last factor, which will imply a bound on the upper tail of the rescaled excess risk.

We briefly make explicit the Donsker assumption before deriving this bound. Both define and note

where S d -1 is the Euclidean unit sphere in R d . As pointed out in Section 3, the processes G n ( t ) are partial suprema of the empirical processes G n ( t, v ) . The Donsker assumption of the theorem states that the empirical processes G n ( t, v ) take value in the space of bounded functions on T × S d -1 , equipped with the ℓ ∞ ( T × S d -1 ) norm and the metric it induces, and converge weakly to their unique Gaussian limit ( G ( t, v )) ( t,v ) ∈T × S d -1 as n → ∞ . By inspecting their finite dimensional distributions, it is straightforward to verify that ( G ( t, v )) ( t,v ) ∈T × S d -1 d = ( ⟨ v, Z ( t ) ⟩ ) ( t,v ) ∈T × S d -1 where ( Z ( t )) t ∈T is the R d -valued Gaussian process defined in the statement of Theorem 3. Finally, we define G ( t ) := sup v ∈ S d -1 G ( t, v ) in analogy with the definition of G n ( t ) .

We now upper bound the upper tail of G 2 n ( ˆ t n ) in (19). Let ( ε k ) ∞ k =1 be a decreasing sequence of positive numbers such that ε k → 0 as k →∞ , and define the sets

as well as the function F : ℓ ( T × S ) → R by

Note on the one hand that ∩ k ≥ 1 T ∗ ( ε k ) = T ∗ , and on the other that F k is continuous for all k ∈ N , and in fact Lipschitz. Indeed, let z, z ′ ∈ ℓ ∞ ( T × S d -1 ) . Then

taking the limit as n → ∞ , the first term converges, by the continuous mapping theorem, to the probability of the event { F 2 k ( G ) > x } , where G is the limiting Gaussian process discussed above, while the second term vanishes by the first part of Theorem 3. Therefore, for all k ∈ N ,

Taking the limit as k →∞ , noticing that the events

are nested, using the continuity of probability from above, and recalling that ∩ k ≥ 1 T ∗ ( ε k ) = T ∗ gives

Using properties of the quantile function (e.g. [EME24, Lemma 20]) finishes the proof of the upper bound. For the lower bound, we make a similar argument. We have, by an application of Lemma 4,

By the Glivenko-Cantelli assumption on Λ n , the first two factors converge to 1 / 2 . For the third, we will lower bound its upper tails, which will imply a lower bound on the upper tails of the rescaled excess risk. We let ( ε k ) ∞ k =1 be a decreasing sequence of positive numbers such that ε k → 0 as k →∞ , and define H k : ℓ ∞ ( T × S d -1 ) → R by

where the subsets T ∗ ( ε k ) are as defined in (20). Clearly, for z, z ′ ∈ ℓ ∞ ( T × S d -1 ) ,

so H k is Lipschitz and therefore continuous. Now

By the same argument as above, we obtain, as n →∞ , and for all k ∈ N ,

Taking the limit as k →∞ , and noticing that

Using properties of the quantile function (e.g. [EME24, Lemma 20]) finishes the proof of the lower bound. The estimates on the quantiles of Z + in Remark 1 are a consequence of standard Gaussian concentration, see [e.g. EME24, Appendix A.3]. Finally, for the second statement in Remark 1,

where the last estimate follows from Gaussian concentration. Taking p = 1 + log |T ∗ | , and recalling that for x ∈ R d , ∥ x ∥ p ≤ d 1 /p ∥ x ∥ ∞ yields the result.

E Proof of Lemma 1

We prove the first statement by induction. For k = 0 , this follows directly from the fact that by definition F 0 n,δ ( T ) = T and F n,δ ( T ) ⊆ T . Now let k ∈ N and assume that the statement holds for k -1 . Let s ∈ F k +1 n,δ ( T ) . Then by definition

where the second inequality follows from the fact that by the induction hypothesis, F k n,δ ( T ) ⊆ F k -1 n,δ ( T ) , and that the supremum is increasing. Therefore s ∈ F k n,δ ( T ) since the last inequality is the defining inequality for F k n,δ ( T ) . We now turn to the second statement. Fix k and δ . On the one hand, T ∗ ⊆ ⋂ n ≥ 1 F k n,δ ( T ) . On the other, for any t ∈ ⋂ n ≥ 1 F k n,δ ( T ) , we have for all n ≥ n 0 , R ( t, w ∗ ( t )) -R ∗ ≤ 2 B · ( nδ ) -1 . Therefore R ( t, w ∗ ( t )) -R ∗ = 0 , and hence t ∈ T ∗ .

proves that

F Proof of Theorem 4

Recall the notation introduced in Appendix C. Let A n ( t ∗ ) be the event that:

We start by showing that under the sample size inequality stated in the theorem, there exists a t ∗ ∈ T ∗ such that P( A n ( t ∗ )) ≥ 1 -δ/ 3 . Indeed, we have

The elements of this sum are bounded by 1 , and the variance parameter of Bousquet's inequality [Bou02] is given by L as defined in Section 3.2. Applying this inequality yields that with probability at least 1 -δ/ 6

Furthermore, by Markov's inequality, with probability at least 1 -δ/ 6

Hence, when the inequality on the sample size stated in the theorem holds for some t ∗ , the event A n ( t ∗ ) holds with probability at least 1 -δ/ 3 . Now on this event, the first bound of Lemma 4 applies, and we have

Now we use the iterative localization method of Koltchinskii [Kol06]. Initially, we have no information about where ˆ t n is located aside from belonging to T , so we start with the bound

Using Markov's inequality, we have on an event B n, 1 which holds with probability at least 1 -δ/ 2 k

Replacing in (22) yields that on the event A n ( t ∗ ) ∩ B n, 1 ,

which shows that on this event, ˆ t n ∈ F n,δ/ 2 k ( T ) , by definition of the map F n,δ/ 2 k . With this knowledge, we now reuse the bound (21) to obtain that on A n ( t ∗ ) ∩ B n, 1

Iterating the procedure we just described k times, we obtain that on an event A n ( t ∗ ) ∩ ( ∩ k j =1 B n,j ) , where P( B n,j ) ≥ 1 -δ/ 2 k for all j ∈ [ k ]

Another application of Markov's inequality yields that on an event C which holds with probability at least 1 -δ/ 6

Since

equation (23) proves the first statement of the theorem. For the second statement, we have on the same event A n ( t ∗ ) ∩ ( ∩ k j =1 B n,k ) ∩ C , and combining the two upper bounds from Lemma 4,

where we used (23) and (24) in the above inequalities, concluding the proof.

G Proof of Lemma 2

We prove a slightly more general result, from which Lemma 2 can be immediately deduced.

Lemma 5. Let n, d ∈ N and let T be a finite set. For each ( i, t ) ∈ [ n ] ×T , let Z i,t ∈ R d be random vectors such that for each t ∈ T , ( Z i,t ) n i =1 are i.i.d. with the same distribution as Z t . For all t ∈ T , assume that E[ Z t ] = 0 , and define

Then

To prove Lemma 5, we need to recall a few preliminary results. The first is a classical symmetrization inequality, see e.g. [BLM13, Lemma 11.4] or [Wai19, Proposition 4.11] for a proof.

Lemma 6. For each ( i, t ) ∈ [ n ] ×T , let W i,t ∈ R d be random vectors such that for each t ∈ T , ( W i,t ) n i =1 are i.i.d. with the same distribution as W t . Let ( ε i ) n i =1 be independent Rademacher random variables, independent of the collection of random vectors W i,t . Define W i,t := W i,t -E[ W t ] . Then

The second result we recall is the Khinchin-Kahane inequality, the specific form we require is obtained from Peña and Giné [PG99, Theorem 1.3.1] by setting q = p and p = 2 in that theorem, see also Boucheron et al. [BLM13, page 141].

Lemma 7. For i ∈ [ n ] , let z i ∈ R d be fixed vectors. Let ( ε i ) n i =1 be independent Rademacher random variables. Then for all p ≥ 2 ,

Astraightforward consequence of Lemma 7 is the following result, which follows from the elementary observation that for a vector x ∈ R d , ∥ x ∥ ∞ ≤ ∥ x ∥ p ≤ d 1 /p ∥ x ∥ ∞ .

Lemma8. For ( i, t ) ∈ [ n ] ×T , let z i,t ∈ R d be fixed vectors. Let ( ε i ) n i =1 be independent Rademacher random variables. Then

Proof. Let p ≥ 1 . Then, by Jensen's inequality and Lemma 7

Recalling that ∥ x ∥ p ≤ d 1 /p ∥ x ∥ ∞ for all x ∈ R d and taking p := 1 + log |T | yields the result.

Finally, we need the following consequence of Lemmas 6 and 8. The proof idea is taken from [Tro16].

Lemma 9. For each ( i, t ) ∈ [ n ] ×T , let W i,t ∈ R be random variables such that for each t ∈ T , ( W i,t ) n i =1 are i.i.d. with the same distribution as W t , with W t ≥ 0 almost surely. Then

Proof. We have by Jensen's inequality and Lemma 6,

Conditioning on the random vectors W i,t , we have by Lemma 8 and the assumption W i,t ≥ 0 a.s.

Taking expectation with respect to W i,t , and using the Cauchy-Schwartz inequality yields

Replacing in (25) and solving the resulting quadratic inequality yields the result.

Equipped with these results, we now prove Lemma 1. The proof idea is taken from [Tro16].

Proof of Lemma 1. We start with the lower bound. We have on the one hand

On the on other hand, by Lemma 6, we have

Define the random index

Conditioning on Z i,t , we have by Jensen's inequality

̸

where in the inequality, the outer expectation is with respect to ε I , and the inner one is with respect to ( ε i ) i = I . Taking expectation with respect to Z i,t gives

Averaging the lower bounds (26) and (27) yields the desired lower bound. We now turn to the upper bound. We have by Lemmas 6 and 8.

Applying Lemma 9 on the last term yields the desired upper bound.

H Proof of Corollary 1

The Glivenko-Cantelli and Donsker assumptions of Theorem 3 follow directly from the moment assumptions of the corollary, the weak law of large numbers, and the central limit theorem, and therefore the conclusions of Theorem 3 hold. For the first statement of the corollary, we may assume without loss of generality that T ∗ = T , otherwise the statement holds trivially. Define

̸

Then ε > 0 , and by the first item of Theorem 3,

It remains to prove the improved upper bound on the asymptotic quantiles. For this, referring to the proof of Theorem 3, and in particular to (18), it is enough to show that

but this follows directly from (28).

I Proof of Corollary 2

The statement follows from the same argument as Theorem 4 with only a few simple modifications. As explained in the main text, we use Lemma 3 to bound the quantity E[max t ∈T Λ n ( t )] by constructing a block diagonal matrix. We use Lemma 2 to control, for any subset S , E [ max s ∈S G 2 n ( s ) ] . The only minor deviation from Theorem 4 is that we bound the second moment

instead of the first. This explains the slightly better dependence on δ in the sample size restriction of Corollary 2 compared to Theorem 4.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction are supported by our main results of Section 3 and 4 directly.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The last paragraph of the conclusion explicitly discusses the limitations of this work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The proofs of all the statements we claimed are new can be found in the appendix. For known statements, we provided direct references to them.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the guidelines and confirm that our work adheres to them. Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our paper is heavily theoretical, and is quite far removed from direct applications, which is why do not foresee any direct societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: We have cited all the relevant work we are aware of, but we do not use any existing code, data, or models in this work.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Not applicable.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Not applicable.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers a \"feature learning\" setting equivalent to structural risk minimization over a set of hypotheses of the form $\\langle w, \\phi_t(x)\\rangle$. It demonstrate that the statistical error of this procedure converges to a quantity that depends on a natural empirical process defined *only* in terms of the classes which contain an optimal predictor. Thus, it is claimed that ERM is \"efficient\" at feature learning in a natural way.\n\nEDIT: I raise my score from a 5 to a 6. I encourage the authors to include the counterexample presented to me in the rebuttal in the main text as a form of motivation, and as a didactic example to express \"what goes wrong\". The authors can also remark about situations under which better dependence on $\\delta$ is achievable. Thank you to authors for addressing my points.\n\nSTRENGTHS:\nThis paper has a number of strengths that make it compelling. \n\n(1) The formalism studies is **exceptionally** clean and natural. It lends itself to study well in other settings as well. \n(2) The authors provide both asymptotic and nonasymptotic results, and the asymptotic ones are rather \"sharp\" in that they provide insight into both the limiting distributions, and do so in terms of universal, Gaussian empirical processes. \n(3) The authors proofs are effective and succinct, and demonstrate great command of the relevant technical machinery. \n(4) The authors use of asymptotic bounds allows them to build considerable intuition in the presentation, before introducing the non-asymptotic bounds which are considerably more involving to parse.\n\nWEAKNESSES:\nThere are a couple weaknesses, however, that temper my excitement. \n\n(1) I apologize for the directness, but I do not find the qualitative finding particularly surprising. We know that ERM localizes, and as a consequence, if I have a predictor class of the form \n\n$\\mathcal F= \\bigcup_{t \\in \\mathcal{T}} \\mathcal{F}_t$, and all optimal predictors lie in $\\mathcal F_t, t \\in \\mathcal{T}^{\\star}$, and moreover, the risk of an  $f \\in \\mathcal{F}_t$ where, $t \\notin \\mathcal{T}^{\\star}$ is lower bound away from zero, then localization should force the limiting behavior of the problem to only depend on the statistical complexity of \n\n$\\bigcup_{t \\in \\mathcal{T}^\\star} \\mathcal{F}_t$\n\nIt is nice to quantify this rigorously, but again, the phenomenon does not seem to be to bring fundamentally new insight. \n\n(2) The non-asymptotic dependence on the probability of error $\\delta$ is quite bad. Indeed, an $O(1/\\delta)$ dependence is not even integrable, obviating in expectation bounds (perhaps in-expectation in too much to ask for, given the possibility that the covariance matrices become singular. Still, this seems like a major limitation. \n\n(3) While the problem setting is remarkably clean, Theorem 4 is not. This seems inevitable, and the authors both (a) explain its intuition and (b) instantiate it for a natural class of problems...\n\n(4) I think the authors should include some more intuition about the relevant increments and terms defined in the paper ($G_n(t)$, $\\Lambda_n(t)$) and so forth. Citing that \"such and such is just such and such in Bos[02]\" does not help with intuition building. The authors might consider adding a section in the appendix that elaborates further, and also provides formal definitions of what it means for a class to be Glivenko Cantelli (this was relatively clear) and, more importantly, Donsker (as a reader with the relevant background, I know what is meant, but this could be less accessible to others). The authors might also consider remarking on what limitations the Donkser-ity of the problem entail (e.g. sufficient restriction on metric entropy).\n\nQUESTIONS:\nCan the authors please explain why the localization phenomenon presented in this work is surprising?\n\nLIMITATIONS:\n(1) Weak dependence on error probability $\\delta$\n(2) Possible limitations due to the Donkser assumption\n(3) Restricted to the linear setting", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the learning theory of empirical risk minimization (ERM) with feature learning. Under the setting where the optimal finite-sample feature is selected by minimizing the empirical risk over a class of features, the authors show that ERM with feature learning implies convergence of the excess risk under certain assumptions. Moreover, other statistical properties, such as asymptotic normality, are also derived.\n\nSTRENGTHS:\nThe statistical theory of this paper regarding feature learning is solid, and the results of the main theorems (Theorem 3 and Theorem 4) seem to be correct, even though I have not had time to check the detailed proofs. In fact, the Glivenko-Cantelli assumptions on the empirical processes and the finite moment assumptions are widely used in learning theory, and the rate $1/n\\delta \\approx O(1/\\sqrt{n})$ seems to be correct.\n\nWEAKNESSES:\n1. The title \"On the Efficiency of ERM in Feature Learning\" seems misleading. Initially, it suggests that ERM should aid feature learning. However, after reading the paper, it appears that the authors are discussing the statistical performance of ERM in the context of feature learning, without clearly demonstrating how ERM aids in feature learning.\n\n2. The authors have reviewed many statistical theory papers. However, in terms of feature learning or representation learning, they didn't conduct a thorough literature review. Specifically, for the final-layer feature, recent studies show that the last-layer feature will converge to an Equiangular Tight Frame (optimal feature for classification problems), which is known as neural collapse, as proposed by Papyan et al. in their paper \"Prevalence of neural collapse during the terminal phase of deep learning training.\" There are some subsequent studies that show that the optimal feature is indeed the minimizer of a regularized ERM. These studies include but are not limited to \"Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training\" (Fang et al., 2021); \"A Geometric Analysis of Neural Collapse with Unconstrained Features\" (Zhu et al., 2021); and \"Neural Collapse in Multi-label Learning with Pick-all-label Loss\" (Li et al., 2024). For different loss functions, the authors may refer to \"Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path\" by Han et al. Regarding learning theory, the sample complexity under neural collapse is investigated by Wang et al. (2024) in their work \"Neural Collapse Meets Differential Privacy: Curious Behaviors of NoisyGD with Near-perfect Representation Learning.\"\n\n3. Some of the claims are confusing and may be over-claimed. Specifically, in the Conclusion, the authors claim that their theory might be used to explain the double descent phenomenon and generalization of deep learning with label noise in Zhang et al., 2021. However, the feature learning setting in this paper is far from being extended to the deep learning setting. Indeed, their assumptions here, such as the moment assumption, should hold for all feature maps in the hypothesis class, which is not verified for deep neural networks. However, existing feature or representation learning theory such as the Neural Collapse theory can partially explain the phenomenon in Zhang et al., such as the overfitted model can still generalize.\n\n4. As the theory is not as enlightening as the authors claimed in their paper since their setting is not practical, this should be regarded as a purely theoretical paper. As a theoretical paper, there should be some room for improvement, such as verifying the assumptions for all feature maps (such as the moment assumptions in Theorem 4) for a certain class. Moreover, as a purely statistical theory paper, a lower bound showing that the obtained finite-sample rate is optimal is necessary for a high-quality publication. It would also be better for the authors to emphasize more technical difficulties compared to ERM without feature learning. In fact, under the Glivenko-Cantelli assumptions and the finite sample assumptions, both the asymptotic theory (Theorem 3) and the non-asymptotic theory (Theorem 4) look like a simple extension of Theorem 1 and Theorem 2 without feature learning, while Theorem 1 and Theorem 2 are not novel in learning theory.\n\nQUESTIONS:\nPlease address my concerns in the weaknesses part.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-06"}

comments: SUMMARY:
This paper consider the problem of regression over the linear classes induced by a collection of feature maps. They study both the asymptotic and the non-asymptotic behavior of the empirical risk minimizer. Surprisingly, although the linear classes has a complexity much higher than that of only one linear map, the authors find that when there is a unique optimal feature map, ERM actually behaves similar with the oracle procedure (knows a priori the optimal feature map). General results for non-unique or even infinite feature map is also provided. The authors also apply their non-asymptotic result on finite feature map cases.

STRENGTHS:
The results in this paper is both novel and significant. They show that even though non-optimal feature map exists in training process, the actual upper bound of the excess risk depends on the size of optimal feature maps. 
From theoretical perspective, the proofs are solid.
The writing is also good, clearly states the results and the intuition, and how their results improve beyond previous classical results where the feature map set is a singleton. Case study is also provided, giving a comprehensive review of how their general framework can be applied.

WEAKNESSES:
It will be good if more case studies are provided.

QUESTIONS:
Is it possible to consider some infinite feature map set with some structure, so that you can also compute the constants in your results?

LIMITATIONS:
No limitations are stated.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the novel setting where we are give  a collection of predefined feature maps (indexed by a set T) and we choose one of these feature maps and then, learn a linear predictor on top of the chosen feature map. The authors derive upper bounds on the excess risk that depend on the number (size) of \"optimal\" feature maps and not the size of the set T.\n\nSTRENGTHS:\n- The author propose a setting for feature learning that is novel and I find it interesting.\n\n- The result in this setting very satisfying. To the best of my knowledge, this is the first result that shows that the excess risk (or at least the upper bound on the excess risk) depends on the size of \"optimal\" features.\n\n- The proof outline and strategy seems correct. I have not fully checked all the details of the proofs. But, the ones I have checked are all correct and solid. The paper is also very well written.\n\nWEAKNESSES:\n- The analysis ignores the role of the learning algorithm and looks at the problem from a purely statistical perspective. The role of implicit bias of the algorithm is not seen here. On a high level, from the set of optimal features, some features are easier to learn/ achieve than others. This might also shrink the effective size of the features. \n\n- The suprema term in the expression of Theorem 4 is not very interpretable. The case of finite features in the next section makes it more clear. However, I find the finite case not that interesting. Is there an interesting non-finite case that one can analyze to get an interpretable result?\n\n- In general, the features are also learned from data and this gives extra dependencies and goes beyond the setting of this paper (unless some sample splitting is done).\n\n- Can the authors comment on the tightness of Theorem 3 and 4? An the potential challenges of coming up with lower bounds?\n\n- The paper will greatly benefit from a simulation result to support the main finding of the paper. For example, for a simple finite feature case. A theoretical example can also help.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers a linear regression problem where an ERM learner tries to learn a linear predictor and a feature map (over a countable set of feature maps) under some specific assumptions on the set of feature maps and the property of minimizers. The paper analyzes both asymptotic and non-asymptotic behaviors of the excess risk in the problem above and suggests that it matches the rate of the case when we only learn linear predictors on top of a fixed feature map, which is surprising.\n\nSTRENGTHS:\nThe paper is well-written and easy to follow. The motivation of this paper is. The key results and directly related results in prior works, along with their intuition behinds, are explained very clearly and concisely. There are a few potential typos (see Weaknesses) but it should not be a problem. The proofs look good to me, though I did not check very carefully. \n\nOverall, I enjoy reading the paper. The only reason why I do not give a higher score is that the setting is too niche (see Weaknesses), which makes the results not so ground-breaking. However, it is still a good paper, and I advocate to accept it.\n\nWEAKNESSES:\nThe main concern about this paper.\n1. The setting is narrow: when I think about feature learning, I imagine we first learn a feature map that gives a good representation of the data. We then want to use the learned feature map that adapts it to a downstream task, which is a linear regression with squared loss in this case. The setting this paper proposed is somewhat the way around: (1) Given a feature map, learn the best linear predictor, (2) Select the best feature map. It seems to me that this paper is considering a non-linear regression problem, not feature learning. More precisely, the paper is trying to solve the problem learn the best feature map specified for the linear regression task. What can we tell about the learned feature map on other downstream tasks? Why the title of the paper is \"On the Efficiency of ERM in Feature Learning\", instead of \"On the Efficiency of ERM in Non-linear Regression\"?\n\n2. If we look at the picture from that perspective, prior results (Theorem 1, Theorem 2) seem good enough for me. Given a learned feature map, under some conditions, I have a fast rate of learning a linear predictor. Of course, it is not feature learning at all, but it tells something about the learned feature map on a specific downstream task (linear regression), which is fair enough. There is no need (at least for me) for a result explaining if I learn a linear predictor with squared loss along with a feature map specifically designed for linear regression, what the sample complexity is.\n\nComments on the Conclusion.\n1. The claims on potential explanations for generalization in DNNs: To the best of my knowledge, the reason behind that should be explained by the geometry of the loss landscape of over-parameterized models and the implicit bias of (stochastic) optimization algorithms used. I would not go into detail since it goes beyond the scope of this paper. However, this paper: (1) considers an optimization oracle for ERM, and (2) does not assume any geometry of the set of feature maps indexed by $\\mathcal{T}$. Therefore, linking the results in this paper to generalization in DNNs seems unnecessary and inappropriate for me.\n\nMinor comments\n1. In the Appendix, it might be helpful if the authors first give a proof sketch for each result for readability. \n\nMinor typos:\n 1. Line 164, a comma missing after the inequality.\n 2. Line 501, should the RHS be $\\frac{1}{2}||\\nabla{R(w^*)}||_{\\Sigma^{-1}}$? \n 3. Multiple commas, dots after (in)equalities missing in the Proof of Theorem 1, 2. \n\nAfter all, it might be too harsh to undervalue this paper based on the points above. I still think it is a good paper, and the results do not have to be connected to Feature Learning and Generalization to be meaningful.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nSee Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}