Title: On the Variance, Admissibility, and Stability of Empirical Risk Minimization
Venue: NeurIPS 2023
Accepted: yes
Keywords: empirical risk minimization

bias-variance decomposition

admissibility
Abstract: It is well known that Empirical Risk Minimization (ERM) may attain minimax suboptimal rates in terms of the mean squared error (Birgé and Massart, 1993). In this paper, we prove that, under relatively mild assumptions, the suboptimality of ERM must be due to its bias. Namely, the variance error term of ERM (in terms of the bias and variance decomposition) enjoys the minimax rate. In the fixed design setting, we provide an elementary proof of this result using the probabilistic method. Then, we extend our proof to the random design setting for various models. In addition, we provide a simple proof of Chatterjee’s admissibility theorem (Chatterjee, 2014, Theorem 1.4), which states that in the fixed design setting, ERM cannot be ruled out as an optimal method, and then we extend this result to the random design setting. We also show that our estimates imply stability of ERM, complementing the main result of Caponnetto and Rakhlin (2006) for non-Donsker classes. Finally, we highlight the somewhat irregular nature of the loss landscape of ERM in the non-Donsker regime, by showing that functions can be close to ERM, in terms of $L_2$ distance, while still being far from almost-minimizers of the empirical loss.

On the Variance, Admissibility, and Stability of Empirical Risk Minimization

Abstract

It is well known that Empirical Risk Minimization (ERM) may attain minimax suboptimal rates in terms of the mean squared error (Birgé and Massart, 1993). In this paper, we prove that, under relatively mild assumptions, the suboptimality of ERM must be due to its bias. Namely, the variance error term of ERM (in terms of the bias and variance decomposition) enjoys the minimax rate. In the fixed design setting, we provide an elementary proof of this result using the probabilistic method. Then, we extend our proof to the random design setting for various models. In addition, we provide a simple proof of Chatterjee's admissibility theorem (Chatterjee, 2014, Theorem 1.4), which states that in the fixed design setting, ERM cannot be ruled out as an optimal method, and then we extend this result to the random design setting. We also show that our estimates imply stability of ERM, complementing the main result of Caponnetto and Rakhlin (2006) for nonDonsker classes. Finally, we highlight the somewhat irregular nature of the loss landscape of ERM in the non-Donsker regime, by showing that functions can be close to ERM, in terms of L 2 distance, while still being far from almost-minimizers of the empirical loss.

1 Introduction

Maximum Likelihood and the method of Least Squares are fundamental procedures in statistics. The study of the asymptotic consistency of Maximum Likelihood has been central to the field for almost a century (Wald, 1949). Along with consistency, failures of Maximum Likelihood have been thoroughly investigated for nearly as long (Neyman and Scott, 1948; Bahadur, 1958; Ferguson, 1982). In the setting of non-parametric estimation, the seminal work of (Birgé and Massart, 1993) provided sufficient conditions for minimax optimality (in a non-asymptotic sense) of Least Squares while also presenting an example of a model class where this basic procedure is sub-optimal. Three decades later, we still do not have necessary and sufficient conditions for minimax optimality of Least Squares when the model class is large. While the present paper does not resolve this question, it makes several steps towards understanding the behavior of Least Squares - equivalently, Empirical Risk Minimization (ERM) with square loss - in large models.

Beyond intellectual curiosity, the question of minimax optimality of Least Squares is driven by the desire to understand the current practice of fitting large or overparametrized models, such as neural networks, to data (cf. (Belkin et al., 2019; Bartlett et al., 2020)). At the present moment, there is little theoretical understanding of whether such unregularized data-fitting procedures are optimal, and the study of their statistical properties may lead to new methods with improved performance.

In addition to minimax optimality, many other important properties of Least Squares on large models are yet to be understood. For instance, little is known about its stability with respect to perturbations of the data. It is also unclear whether approximate minimizers of empirical loss enjoy similar statistical

Alexnader Rakhlin BCS & IDSS MIT

properties as the exact solution. Conversely, one may ask whether in the landscape of possible solutions, a small perturbation of the minimizer output by Least Squares itself is a near-optimizer of empirical loss.

The contribution of this paper is to provide novel insights into the aforementioned questions for convex classes of functions in a quite generic setting. In detail, we show the following:

We prove that in the fixed design setting, the variance term of ERM is upper bounded by the minimax rate of estimation; thus, if the ERM is minimax suboptimal, this must be due to the bias term in the bias-variance decomposition. Then, we extend this result to random design, and provide an upper bound for the variance error term under a uniform boundedness assumption on the class. This bound also implies that under classical assumptions in empirical process theory, the variance error term is minimax optimal.

We show that under an isoperimetry assumption on the noise, the expected conditional variance error term of ERM is upper bounded by the 'lower isometry' remainder, a parameter that was introduced in Bartlett et al. (2005); Mendelson (2014). Next, under an additional isoperimetry assumption on the covariates, we prove that the variance of ERM is upper bounded by the lower isometry remainder on any robust learning architecture (namely, a class consisting of functions which are all O (1) -Lipschitz) which almost interpolates the observations (cf. Bubeck and Sellke (2023)).

It is known that ERM is always admissible in the fixed design setting (Chatterjee, 2014; Chen et al., 2017); that is, for any convex function class, there is no estimator having a lower error than ERM (up to a multiplicative absolute constant) on every regression function. We provide a short proof of this result for fixed design via a fixed-point theorem. Using the same method, we also prove a somewhat weaker result in the random design case, generalizing the main result of Chatterjee (2014).

We show that ERM is stable, in the sense that all almost-minimizers (up to the minimax rate) of the squared loss are close in the space of functions. This result is a non-asymptotic analogue of the asymptotic analysis in Caponnetto and Rakhlin (2006), and extends its scope to non-Donsker classes.

While any almost-minimizer of the squared loss is close to the minimizer with respect to the underlying population distribution, the converse is incorrect. We prove that for any non-Donsker class of functions, there exists a target regression function such that, with high probability, there exists a function with high empirical error near the ERM solution; this means that the landscape of near-solutions is, in some sense, irregular.

Conclusions Our results show that the ERM enjoys an optimal variance error term in two distinct regimes: the classical regime (van de Geer, 2000), where the function class is fixed and the number of samples is increasing, and the 'benign overfitting' setting (Belkin et al., 2019; Bartlett et al., 2020), in which the 'capacity' of the class is large compared to the number of samples. In both these settings, our work implies that the minimax optimality of ERM is only determined by its bias error term (or its implicit bias). For models with 'few' parameters, computationally efficient bias correction methods do exist and are commonly used in practice (cf. (Efron and Tibshirani, 1994)), however, these methods fail over large function classes, in which the bias causes the statistical sub-optimality. Our work reveals the importance of inventing computationally efficient debiasing methods for rich function classes, including non-parametric models and high-dimensional models. The main message of our work is that such methods, if discovered, may significantly improve the statistical performance of ERM over large models in practice.

1.1 Prior Work

Stability of ERM The stability of learning procedures, which was an active area of research in the early 2000's, has recently seen a resurgence of interest because of its connections to differential privacy and to robustness of learning methods with respect to adversarial perturbations. In the interest of space, we only compare present results to those of Caponnetto and Rakhlin (2006). In the latter paper, the authors showed that the L 1 -diameter of the set of almost-minimizers of empirical error (with respect to any loss function) asymptotically shrinks to zero as long as the perturbation is o ( n -1 / 2 ) and the function class is Donsker. The analysis there relies on passing from the empirical

process to the associated Gaussian process in the limit, and studying uniqueness of its maximum using anti-concentration properties. While the result there holds without assuming that the class is convex, it is limited by (a) its asymptotic nature and (b) the assumption that the class is not too complex. In contrast, the present paper uses more refined non-asymptotic concentration results, at the expense of additional assumptions such as convexity and minimax optimal lower and upper lower isometry remainders. Crucially, the present result, unlike that of Caponnetto and Rakhlin (2006), holds for non-Donsker classes - those for which the empirical process does not converge to the Gaussian process.

Shape-constrained regression The term 'shape-constrained regression' refers to function classes consisting of functions with a certain 'shape' property, such as convexity or monotonicity (Samworth and Sen, 2018). In these problems, a common theme is that the statistical behavior of the class exhibits a phase transition when the dimension d of the domain reaches a certain value. For instance, in convex (Lipschitz) regression, the transition happens at d = 5 : when d < 5 , the ERM procedure is minimax optimal (Seijo and Sen, 2011; Han and Wellner, 2016; Kim and Samworth, 2016; Seijo and Sen, 2011; Guntuboyina, 2012), but in higher dimensions, it is known to be suboptimal (Kur et al., 2020a,b). In other shape-constrained models, however, the ERM is minimax optimal even in high dimensions, such as isotonic regression and log-concave density estimation (Han et al., 2019; Kur et al., 2019; Carpenter et al., 2018). Our results show that the sub-optimality of ERM in shape-constrained regression can only be due to the high bias of ERM. These results also align with the empirical observation that for the problem of estimation of convex sets, the ERM has a bias towards 'smooth" convex sets (Soh and Chandrasekaran, 2019; Ghosh et al., 2021).

High-dimensional statistics In classical statistics, the Maximum Likelihood (MLE) typically has a low bias compared to its variance, and the standard approach is to introduce bias into the procedure in order to reduce the variance, overall achieving a better trade-off (see (Sur and Candès, 2019, §1) and references therein). In contrast, in high-dimensional models, the MLE may suffer from high bias even in tasks such as logistic regression and sparse linear regression (cf. Candès and Sur (2020); Javanmard and Montanari (2018)). Our results align with this line of work, showing that the problem of high bias may also arise in the task of regression over rich function classes.

2 Main Results

In §2.1, we present the setting of our model and the all required preliminaries, and in the remaining sub-sections, we present our results. In details, in §2.2-2.3, we present our results on the variance of ERM in fixed and random designs respectively, and in §3 we provide sketches of some of our proofs. For lack of space, we presents our admissibility and our landscape results on ERM in the supplementary material (§4.1-4.2).

2.1 Preliminaries

Let X be some fixed domain, F be a class of functions from X to R , and f ∗ ∈ F an unknown target regression function. We are given n ≥ 2 data points X 1 . . . , X n ∈ X and n noisy observations

which we denote by D := { ( X i , Y i ) } n i =1 ; ξ := ( ξ 1 , . . . , ξ n ) is the random noise vector.

In the fixed design setting, the observations X 1 = x 1 , . . . , X n = x n are arbitrary and fixed, and we denote the uniform measure on this set of points by P ( n ) . In the random design setting, the data points X := ( X 1 , . . . , X n ) are drawn i.i.d. from a probability distribution over X , denoted by P , and the noise vector ξ is drawn independently of X . Note that this model is general enough to cover the high-dimensional setting, as both the function class F and the distributions of ξ and X are allowed to depend on the number of samples n .

An estimator for the regression task is defined as a measurable function ¯ f n : D ↦→ {X → R } , that for any realization of the input D , outputs some real-valued measurable function on X . The risk of ¯ f n is defined as

where Q = P in the random design case, and Q = P ( n ) in the fixed design case. Note that in fixed design, the expectation E D is taken over the noise ξ , while in random design the expectation E D is

taken both over the random data points X and noise ξ . The minimax rate is defined via

In the fixed design setting, we also denote the minimax rate by M ( F , P ( n ) ) , as the dependence in n is already present in P ( n ) .

The most natural estimation procedure is the Least Squares (LS) or Empirical Risk Minimization (ERM) with squared loss, defined as

When studying fixed design, we will abuse notation and treat ̂ f n as a vector in R n . We emphasize that many of our results hold for many other estimators, including various regularized ERM procedures (see the relevant remarks below).

In both the fixed and random design settings, we shall assume the following:

Assumption 1. F is a closed convex subset of L 2 ( Q ) , where Q ∈ { P ( n ) , P } .

The convexity of F means that any f, g ∈ F and λ ∈ [0 , 1] : λf +(1 -λ ) g ∈ F ; closedness means that for any sequence { f n } ∞ n =1 ⊂ F converging to f with respect to the norm of L 2 ( Q ) , the limit f lies in F . The closedness ensures that ̂ f n is well-defined.

Assumption 1 is standard in studying the statistical performance of the ERM (cf. Lee et al. (1996); Bartlett et al. (2005); Mendelson (2014)). In particular, under this assumption, the values of ̂ f n at the observation points X 1 , . . . , X n is uniquely determined for any ξ . Note that in general, the values of a function f ∈ F at the points X 1 , . . . , X n does not uniquely identify f among all functions in the class.

In addition to ̂ f n , we analyze properties of the set of δ -approximate minimizers of empirical loss, defined for δ > 0 via

Note that O δ is a random set, in both fixed and random designs.

It is well-known that the squared error of any estimator, in particular that of LS, decomposes into variance and bias components:

where Q = P ( n ) in the fixed design setting and Q = P in the random design setting. Also, for simplicity of the presentation of our results, we denote the maximal variance error term of ¯ f n by V ( ¯ f n , F , Q ) , i.e.

In the random design setting, we also have the law of total variance:

We refer to the two terms as the expected conditional variance and the variance of the conditional expectation, respectively. We conclude this introductory section with a bit of notation and a definition.

̸

Notation We use the notation of ≍ , ≳ , ≲ to denote equality/inequality up to an absolute constant. We use ∥ · ∥ = ∥ · ∥ L 2 ( P ) to denote the L 2 ( P ) norm, and ∥ · ∥ n to denote the L 2 ( P ( n ) ) norm (that is equal to the Euclidean norm scaled by 1 / √ n ). Finally, given a function f : S → T between metric spaces, we define its Lipschitz constant as ∥ f ∥ Lip = sup a,b ∈ T,a = b d T ( f ( a ) ,f ( b )) d S ( a,b ) , and we say ' f is L -Lipschitz' when its Lipschitz constant is at most L . Finally, we denote by diam Q ( H ) the L 2 ( Q ) diameter of a set of functions H .

Definition 1. Let ϵ ≥ 0 , F ⊆ {X → R } and d ( · , · ) a pseudo-metric on F . We call a set S ⊂ F an ϵ -net of F with respect to d if for any f ∈ F there exists g ∈ S with d ( f, g ) ≤ ϵ . We denote by N ( ϵ, F , d ) the ϵ -covering number of F with respect to d , that is, the minimal positive integer N such that F admits an ϵ -net of cardinality N .

2.2 Variance of ERM in fixed design setting

In this part, we consider some fixed ( F , P ( n ) ) and assume that the noise is standard normal.

Assumption 2. The noise vector ξ is distributed as an isotropic Gaussian, i.e. ξ ∼ N (0 , I n × n ) .

Our first result provides an exact characterization of the variance (up to a multiplicative absolute constant) under Assumptions 1-2. In order to state it, for a fixed f ∗ ∈ F , we define the following set:

In words, when the underlying function f ∗ is fixed, we consider the ERM as a random vector (depending on the noise), whose expectation we denote by E D ̂ f n . H ∗ is then just a neighborhood around the expected ERM with a radius of order the square root of the variance error term of ̂ f n , when the underlying function is f ∗ ∈ F .

We can now state our first result, which uses the notion of the set O δ of δ -approximate minimizers from (5).

Theorem 1. Under Assumptions 1-2, the following holds:

and in particular V ( ̂ f n , F , P ( n ) ) ≲ M ( F , P ( n ) ) . Furthermore, for δ := δ ( f ∗ , n ) ≲ M ( H ∗ , P ( n ) ) , the event

holds with probability at least max { 1 -2 exp( -cn · M ( H ∗ , P ( n ) )) , 0 . 9 } , where c ∈ (0 , 1) is an absolute constant.

Theorem 1 establishes our first claim: the variance of ERM is bounded above (up to a multiplicative absolute constant) by the minimax rate of estimation on H ∗ . Since H ∗ is contained in F , its minimax rate is at most that of F ; in particular, the variance of ERM is bounded by M ( F , P ( n ) ) , and hence, any sub-optimality of ERM must arise from its bias. The theorem also incorporates a stability result: not only is the ERM close to its expected value E D ̂ f n with high probability, but any approximate minimizer (up to an excess error of δ 2 ) is close to E D ̂ f n as well.

Our next result complements Theorem 1 above, providing a lower bound on V ( ̂ f n , F , P ( n ) ) :

Theorem 2. Under Assumptions 1-2, the following holds:

Note that there is a multiplicative gap of order M ( F , P ( n ) ) between the bounds of Theorems 1 and 2. We leave it as an open problem whether the bound of Theorem 2 can be improved under these general assumptions.

2.3 Variance of ERM in random design setting

We now turn our attention to the random design setting. Here, we establish similar results to the previous sub-section, albeit under additional assumptions, and with significantly more effort. Unlike the fixed design case, we cannot provide an exact characterization of the variance of ERM. We shall use two different approaches to estimate the variance error term. In the first approach, we use classical tools of empirical process theory together with assumptions that are commonly used in M-estimation (van de Geer, 2000). The second approach, which is inspired by our fixed-design approach, relies heavily on isoperimetry and concentration of measure (cf. Ledoux (2001)).

Throughout this part, we assume for simplicity of presentation that the L 2 ( P ) -diameter of the function class is independent of n .

Assumption 3. There exist absolute constants C, c > 0 such that c ≤ diam P ( F ) ≤ C .

The classical work of Yang and Barron (1999) provides a characterization of the minimax rate ( n, F , P ) under appropriate assumptions (such as normal noise, and uniform boundedness of F , and richness of F ). They proved that the minimax rate is the square of the solution of the following (asymptotic) equation

where N ( ϵ, F , P ) is the ϵ -covering number of F in terms of L 2 ( P ) metric (see Def. 1 above). We denote this point by ϵ ∗ = ϵ ∗ ( n ) , and even under less restrictive assumptions, ϵ 2 ∗ also lower bounds the minimax rate, up to a multiplicative factor of O (log n ) . Also, we remark that it is well known that ERM may not achieve this optimal rate (Birgé and Massart, 1993) for large (so-called non-Donsker) function classes.

We also introduce the following additional notations and definitions: First, P n denotes the (random) uniform measure over X = ( X 1 , . . . , X n ) . Next, following Bartlett et al. (2005), we define the lower and upper isometry remainders of ( F , P ) for a given n . These remainders measure the discrepancy between L 2 ( P ) and a 'typical' L 2 ( P n ) , here 'typical' means for most of realizations of X . These remainders first emerged in the field of metric embeddings, specifically in the definition of quasi-isometries (cf. Ostrovskii (2013)).

In order to introduce these isometry remainders, we first define for each realization of the input X , the constants I L ( X ) and I U ( X ) as the minimal numbers A X , B X ≥ 0 , respectively, such that the following holds:

Note that as A X and B X increase, the geometry of L 2 ( P n ) and L 2 ( P ) over F becomes less similar. For example, in the extreme case of A X = B X = 0 , it implies the L 2 ( P ) and L 2 ( P n ) induce the same topology over F . In words, the lower isometry is the minimal threshold that satisfies the following: all f, g ∈ F that are ω ( I L ( X )) far from each other in L 2 ( P ) , must be at least Ω( ∥ f -g ∥ ) far in L 2 ( P n ) . The upper isometry remainder implies the converse. To provide further intuition on these remainders, for instance, observe that I L ( X ) upper bounds on diameter in L 2 ( P ) of possible solutions of ERM; namely, one has

Finally, the isometry remainders I L ( n ) , I U ( n ) are defined as the 'typical' values of I L ( X ) , I U ( X ) :

Definition 2. The lower and upper isometry remainders I L ( n ) , I U ( n ) are defined as the minimal constants A n , B n ≥ 0 (respectively) such that

In the classical regime (van de Geer, 2000), it is considered to be a standard assumption that max {I L ( n ) , I U ( n ) } ≲ ϵ 2 ∗ . However, in the high dimensional setting, it may happen that the lower isometry remainder is significantly smaller than the upper isometry remainder, e.g., I L ( n ) ≲ ϵ 2 ∗ and

I U ( n ) ≫ ϵ 2 ∗ (cf. Liang et al. (2020); Mendelson (2014)). We discuss these remainders further in Remark 10 below.

Finally, we remind the reader that ̂ f n is uniquely defined on the data points X when F is a convex closed function class, but it may not be unique over the entire X (as multiple functions in F may take the same values at X 1 , . . . , X n ). In §2.3.1, the results hold for any possible solution of ̂ f n over X , whereas in §2.3.2, we (implicitly) assume that ̂ f n is equipped with a selection rule such that it is also unique over the entire X (e.g., choosing the minimal norm solution (Hastie et al., 2022; Bartlett et al., 2020)); i.e, ̂ f n : D → F .

2.3.1 The empirical processes approach

Here, we assume that the function class and the noise are uniformly bounded.

Assumption 4. There exist universal constants Γ 1 , Γ 2 > 0 such that F is uniformly upper-bounded by Γ 1 , i.e. sup f ∈F ∥ f ∥ ∞ ≤ Γ 1 ; and the components of ξ = ( ξ 1 , . . . , ξ n ) are i.i.d. zero mean with variance one and are almost surely bounded by Γ 2 .

Remark 1 . The uniform boundedness assumption on the noise is taken to simplify the proof (which uses of Talagrand's inequality). This can be relaxed to the assumption that the noise is i.i.d. subGaussian, at the a price of a multiplicative factor of O (log n ) in the error term in Theorem 3 below.

Definition 3. Set ϵ U := max { ϵ ∗ , ˜ ϵ } , where ˜ ϵ is the solution of

Note that when I U ( n ) ≲ ϵ 2 ∗ , ϵ U ≍ ϵ ∗ , while if I U ( n, P ) ≫ ϵ 2 ∗ then ϵ U ≫ ϵ ∗ . The following is our main result in this approach to the random design setting:

Theorem 3. Set ϵ 2 V := max { ϵ 2 U , I L ( n ) } , then under Assumptions 1,3,4 the following holds with probability of at least 1 -n -1 :

where δ n = O ( ϵ 2 V ) ; and in particular V ( ̂ f n , F , P ) ≲ ϵ 2 V .

Theorem 3 is a generalization of Theorem 1 to the random design case, and its proof uses the strong convexity of the loss and Talagrand's inequality. In §5.1 below, we discuss this bound in the context of 'distribution unaware' estimators. We remark that this Theorem extends the scope of Caponnetto and Rakhlin (2006) to non-Donsker classes.

An immediate and useful corollary of this result is that if we have sufficient control of the upper and lower isometry remainders, the variance will be minimax optimal:

Corollary 1. Under Assumptions 1,3,4 and max {I L ( n ) , I U ( n ) } ≲ ϵ 2 ∗ , the following holds:

In the classical regime, the assumption that max {I L ( n ) , I U ( n ) } ≲ ϵ 2 ∗ is considered to be standard in the empirical process and shape constraints literature (cf. van de Geer (2000) and references therein); it holds for many classical models (see Remark 11 below).

Remark 2 . Note that Corollary 1 may also be derived directly from Theorem 1 if the noise is assumed to be standard Gaussian. Yet, this corollary holds for any isotropic sub-Gaussian noise - which is significantly more general.

2.3.2 The isoperimetry approach

In order to motivate this part, we point out that just requiring that I L ( n ) ≲ ϵ 2 ∗ is considered to be a mild assumption (see Remark 10 below). However, the upper bound of Theorem 3 depends on the upper isometry remainder as well; we would like to find some conditions under which this dependency can be removed. Moreover, note that the isometry remainders are connected to the geometry of

( F , P ) and not directly to the stability properties of the estimator . Using a different approach, based on isoperimetry, we will upper-bound the variance of ERM based on some 'interpretable' stability parameters of the estimator itself. These stability parameters will be data-dependent relatives of the lower isometry remainder. Differently from the previous part, we do not assume that the function class F is uniformly bounded by a constant independent of the sample size n .

First, we introduce the definition of Lipschitz Concentration Property (LCP):

Definition 4. Let Z = ( Z 1 , . . . , Z m ) be a random vector taking values in Z ⊗ m . Z satisfies the LCP with constant c L > 0 , with respect to a metric d : ( Z ⊗ m , Z ⊗ m ) → R + , if for all F : Z m → R is 1 -Lipschitz, the following holds:

The LCP property is also known as the isoperimetry condition (cf. (Bubeck and Sellke, 2023, §1.3)), and it is stronger than being sub-Gaussian (Boucheron et al., 2013), and yet it is significantly less restrictive than requiring normal noise (in which case c L = 1 / 2 (Ledoux, 2001)); for further details see Remark 9 below. Now, we state our first assumption:

Assumption 5. ξ is an isotropic random vector satisfying (12) with constant c L = Θ(1) , with respect to the Euclidean norm in R n .

Recall that ϵ ∗ is defined as the stationary point of nϵ 2 ≍ log N ( ϵ, F , P ) , and that the conditional variance of ̂ f n , which is a function of the realization X of the input, is defined as

that is, we fix the data points X and take expectation over the noise.

The formulation of the following definition involves a yet-to-be-defined (large) absolute constant M > 0 , which will be specified in the proof of Theorem 4 (see §3.2 below).

Definition 5. For each realization X and f ∗ ∈ F , let ρ S ( X , f ∗ ) be defined as the minimal constant δ ( n ) such that

where B n ( ξ , r ) = { ξ ′ ∈ R n : ∥ ξ -ξ ′ ∥ n ≤ r } , and c 2 > 0 is an absolute constant.

We also set ρ S ( X ) := sup f ∗ ∈F ρ S ( X , f ∗ ) . ρ S ( X ) measures the optimal radius of stability (or 'robustness') of ̂ f n to perturbations of the noise when the underlying function and data points X are fixed . This is a weaker notion than the lower isometry remainder; in fact, one can verify that ρ S ( X ) ≲ max {I L ( X ) , ϵ 2 ∗ } for every realization X (see Lemma 12 for completeness). Now, we are ready to present our first theorem:

Theorem 4. Under Assumptions 1,3,5, the following holds for every realization X of the data:

and in particular sup f ∗ ∈F E X V D ( ̂ f n | X ) ≲ max {I L ( n ) , ϵ 2 ∗ } .

Note that if I L ( n ) ≲ ϵ 2 ∗ - a very mild assumption - then we obtain that the expected conditional variance is minimax optimal. However, we believe that it is impossible to bound the total variance via the lower isometry remainder alone. Intuitively, ̂ f n only observes a given realization X , and in general, the geometry of F may 'look different' under different realizations if I L ( n ) is large (see the discussion in §5.1 for further details).

In our next result, we identify a model under which we can bound the total variance of ̂ f n by the lower isometry remainder. To state the next assumption, we fix a metric d : X × X → R + on X , and denote by d n the metric on X n given by d n ( X , X ′ ) 2 = ∑ d ( X i , X ′ i ) 2 .

Assumption 6. X ∼ P ⊗ n satisfies (12) with respect to the metric d n ( · , · ) , and with constant c X that only depends on P

Note that it is insufficient to assume that X ∼ P satisfies an LCP, since this does not imply that X satisfies an LCP with a constant independent of n (w.r.t. to d n ( · , · ) ). However, if X ∼ P satisfies a concentration inequality which tensorizes 'nicely,' such as a log-Sobolev or W 2 -transportation cost inequality (cf. (Ledoux, 2001, §5.2, §6.2)), then X ∼ P ⊗ n does satisfy this LCP property.

Next, we assume that with high probability, ̂ f n is close to interpolating the observations:

Assumption 7. There exist absolute constants c I , C I > 0 , such that the following holds:

This assumption is quite common in the study of 'rich' high dimensional models (i.e. when the function class F depends on n ; see, e.g., Belkin et al. (2019); Liang and Rakhlin (2020)) which are prominent in the recent benign overfitting literature.

Finally, we introduce another stability notion. Recall the random set O δ ⊂ F of almost-minimizers of the empirical loss, as defined in (5) above; note that, in the random design setting, O δ depends on both X and ξ . The random variable diam P ( O δ ) can be thought of as measuring the stability of the ERM with respect to imprecision in the minimization algorithm (cf. (Caponnetto and Rakhlin, 2006)). The formulation of the following definition involves another yet-to-be-defined (large) absolute constant M ′ > 0 , which will be specified in the proof of Theorem 5 (see §6.6 below), as well as the constant c I from Assumption 7.

Definition 6. ρ O ( n, P , f ∗ ) is defined as the smallest δ ( n ) ≥ 0 such that

where c I ≥ 0 is the same absolute constant defined in Assumption 7.

In order to understand the relation between this and the previous stability notions, note that under Assumption 7 and the event E of Definition 6, we have that on an event of nonnegligible probability, ρ S ( X , f ∗ ) ≤ ρ O ( n, P , f ∗ ) ; in addition, ρ O ( n, P , f ∗ ) ≲ max {I L ( n ) , ϵ 2 ∗ } (see Lemma 13 below). Under these additional two assumptions and the last definition, we state our bound for the total variance of ̂ f n :

Theorem 5. Under Assumptions 1,3,5-7, the following holds:

and in particular one has V ( ̂ f n , F , P ) ≲ c -1 X · sup f ∗ ∈F ∥ f ∗ ∥ Lip · max { ϵ 2 ∗ , I L ( n ) } .

Note that when F is a robust learning architecture (i.e. F ⊂ {X → R : ∥ f ∥ Lip = O (1) } ), our bound is optimal. Interestingly, the assumptions of Theorem 5 coincide with those of the model considered in the recent paper of Bubeck and Sellke (2023). Also note that the last theorem connects the total variance of ̂ f n to a 'probabilistic' threshold for the L 2 ( P ) -diameter of the data-dependent set of Θ( ϵ 2 ∗ ) -approximating solutions of ̂ f n - two parameters which at first sight are unrelated.

Remark 3 . One may suspect that the assumptions of almost interpolation and robustness are incompatible, which would render our theorem vacuous. However, perhaps counter-intuitively, in the high-dimensional setting these assumptions can coexist. For example, interpolation with O (1) -Lipschitz functions may be possible when the 'intrinsic' dimension of X is Ω(log( n )) (depending on the richness of F ), though it is generally impossible when the dimension is o (log( n )) (this follows from the behaviour of the entropy numbers of the class of Lipschitz functions; cf. Dudley (1999)).

Remark 4 . Using Assumptions 1,3,5-6, one may prove the same bound as in Theorem 3, i.e. that V ( ̂ f n , F , P ) ≲ ϵ 2 V , without requiring the noise or the function class to be uniformly bounded. The idea is to obtain the crucial concentration bounds in the proof of Theorem 3 by using the LCP properties of ξ and X ∼ P along with the robustness of F , rather than via Talagrand's inequality.

3 Proof sketches

In this section, we sketch the proofs of less-technical results to give the reader a flavor of our methods. The full proofs are given in the next section. For the proofs we introduce some additional notations. For m ∈ N , we set [ m ] := { 1 , . . . , m } . The inner products in L 2 ( P ) , L 2 ( P ( n ) ) are denoted by ⟨· , ·⟩ , ⟨· , ·⟩ n , respectively.

3.1 Sketch of proof of Theorem 1

Here, we sketch a simple proof of a weaker version of our result, namely V D ( ˆ f n ) ≲ M ( F , P ( n ) ) , under the stronger assumption that

where ϵ ∗ solves log N ( ϵ, F , P ( n ) ) ≍ nϵ 2 . (This holds under reasonable assumptions on F , but can be dispensed with; see Lemma 1 for the exact characterization.) In §6.1.1, we fill in the details of this sketch, and in §6.1.2, we give the full proof of Theorem 1.

The proof uses the probabilistic method (Alon and Spencer, 2016). Let f 1 , . . . , f N be centers of a minimal ϵ ∗ -cover of F with respect to L 2 ( P ( n ) ) , per Definition 1. First, since for any i ∈ [ N ] , the map ξ ↦→ √ n ∥ ̂ f n ( ξ ) -f i ∥ n is 1 -Lipschitz, (12) and a union bound ensure that with probability at least 1 -1 2 N , for all i ∈ [ N ] ,

On the other hand, by the pigeonhole principle, there exists at least one i ∗ ∈ [ N ] such that with probability at least 1 /N , ∥ ̂ f n -f i ∗ ∥ n ≤ ϵ ∗ . Hence, there exists at least one realization of ξ ∈ R n for which both bounds hold, and thus, deterministically ,

where we used the balancing equation (10). Another application of (12) and integration of tails yields

implying that the variance of ERM is minimax optimal.

3.2 Sketch of proof of Theorem 4

As is well-known, the Lipschitz concentration condition (12) is equivalent to an isoperimetric phenomenon: for any set A ⊂ R n with Pr ξ ( A ) ≥ 1 / 2 , its t -neighborhood A t = { ξ ∈ R n : inf x ∈ A ∥ x -ξ ∥ n ≤ t } satisfies

One sees quickly that this implies that if A has measure at least 2 exp( -nt 2 / 2) , then A 2 t has measure 1 -2 exp( -nt 2 / 2) .

Let E be the event of Definition 5. As in §3.1 above, one obtains via the pigeonhole principle and the definition of ϵ ∗ that there exists some f c ∈ F such that

By isoperimetry, Pr ξ ( A 2 t ) ≥ 1 -2 exp( -nt 2 / 2) , where t = Mϵ ∗ / 2 and M is chosen such that ( M/ 2) 2 ≥ 2 C 3 ; this fixes the value of the absolute constant M used in (13).

Applying (13) yields that if ξ ∈ A ⊂ E and ∥ ξ ′ -ξ ∥ n ≤ Mϵ ∗ = 2 t , ∥ ̂ f n ( ξ ) -̂ f n ( ξ ′ ) ∥ ≤ ρ S ( X , f ∗ ) and so ∥ ̂ f n ( ξ ′ ) -f c ∥ ≤ ϵ ∗ + ρ S ( X , f ∗ ) . This implies

which implies via conditional expectation that V ( ̂ f n | X ) ≲ max { ρ S ( X ) , ϵ 2 ∗ } , as desired (where we used that ϵ 2 ∗ ≳ log( n ) /n (see Lemma 3 below), and therefore exp( -nt 2 ) = O ( ϵ 2 ∗ ) ).

Acknowledgements: This work was supported by the Simons Foundation through Award 814639 for the Collaboration on the Theoretical Foundations of Deep Learning, the ERC under the European Union's Horizon 2020 research and innovation programme (grant agreement No 770127), and the NSF (awards DMS-2031883, DMS-1953181). Part of this work was carried out while the first two authors authors were in residence at the Institute for Computational and Experimental Research in Mathematics in Providence, RI, during the Harmonic Analysis and Convexity program; this residency was supported by the NSF (grant DMS-1929284). Finally, the first two authors also wish to acknowledge Prof. Shiri Artstein-Avidan for introducing them to each other.

References

Charalambos D. Aliprantis and Kim C. Border. Infinite Dimensional Analysis: a Hitchhiker's Guide . Springer, Berlin; London, 2006. ISBN 9783540326960 3540326960. doi: 10.1007/3-540-29587-9.

Noga Alon and Joel H Spencer. The probabilistic method . John Wiley & Sons, 2016.

Shiri Artstein-Avidan, Apostolos Giannopoulos, and Vitali D Milman. Asymptotic geometric analysis, Part I , volume 202 of Mathematical Surveys and Monographs . American Mathematical Society, 2015.

RR Bahadur. Examples of inconsistency of maximum likelihood estimates. Sankhy¯ a: The Indian Journal of Statistics , pages 207-210, 1958.

Peter L Bartlett, Olivier Bousquet, and Shahar Mendelson. Local Rademacher complexities. The Annals of Statistics , 33(4):1497-1537, 2005.

Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences , 117(48):30063-30070, 2020.

Mikhail Belkin, Alexander Rakhlin, and Alexandre B Tsybakov. Does data interpolation contradict statistical optimality? In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1611-1619. PMLR, 2019.

Lucien Birgé and Pascal Massart. Rates of convergence for minimum contrast estimators. Probability Theory and Related Fields , 97(1-2):113-150, 1993.

Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration inequalities: A nonasymptotic theory of independence . Oxford University Press, 2013.

Sébastien Bubeck and Mark Sellke. A universal law of robustness via isoperimetry. Journal of the ACM , 70(2):1-18, 2023.

Emmanuel J Candès and Pragya Sur. The phase transition for the existence of the maximum likelihood estimate in high-dimensional logistic regression. The Annals of Statistics , 48(1):27-42, 2020.

Andrea Caponnetto and Alexander Rakhlin. Stability properties of empirical risk minimization over Donsker classes. Journal of Machine Learning Research , 7(12), 2006.

Timothy Carpenter, Ilias Diakonikolas, Anastasios Sidiropoulos, and Alistair Stewart. Near-optimal sample complexity bounds for maximum likelihood estimation of multivariate log-concave densities. In Conference On Learning Theory , pages 1234-1262, 2018.

Sourav Chatterjee. A new perspective on least squares under convex constraint. The Annals of Statistics , 42(6):2340-2381, 2014.

Xi Chen, Adityanand Guntuboyina, and Yuchen Zhang. A note on the approximate admissibility of regularized estimators in the gaussian sequence model. arXiv preprint arXiv:1703.00542 , 2017.

Richard M Dudley. Uniform central limit theorems . Number 63. Cambridge University Press, 1999.

Bradley Efron and Robert J Tibshirani. An introduction to the bootstrap . CRC press, 1994.

Thomas S. Ferguson. An inconsistent maximum likelihood estimate. Journal of the American Statistical Association , 77(380):831-834, 1982. ISSN 01621459.

Avishek Ghosh, Ashwin Pananjady, Adityanand Guntuboyina, and Kannan Ramchandran. Max-affine regression: Parameter estimation for gaussian designs. IEEE Transactions on Information Theory , 2021.

Adityanand Guntuboyina. Optimal rates of convergence for convex set estimation from support functions. The Annals of Statistics , 40(1):385-411, 2012.

Qiyang Han and Jon A Wellner. Multivariate convex regression: global risk bounds and adaptation. arXiv preprint arXiv:1601.06844 , 2016.

Qiyang Han, Tengyao Wang, Sabyasachi Chatterjee, and Richard J Samworth. Isotonic regression in general dimensions. The Annals of Statistics , 47(5):2440-2471, 2019.

Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in highdimensional ridgeless least squares interpolation. The Annals of Statistics , 50(2):949-986, 2022.

Adel Javanmard and Andrea Montanari. Debiasing the lasso: Optimal sample size for gaussian designs. The Annals of Statistics , 46(6A):2593-2622, 2018.

Arlene KH Kim and Richard J Samworth. Global rates of convergence in log-concave density estimation. The Annals of Statistics , 44(6):2756-2779, 2016.

Vladimir Koltchinskii. Oracle Inequalities in Empirical Risk Minimization and Sparse Recovery Problems: Ecole d'Eté de Probabilités de Saint-Flour XXXVIII-2008 , volume 2033. Springer Science & Business Media, 2011.

Gil Kur and Alexander Rakhlin. On the minimal error of empirical risk minimization. In Conference on Learning Theory , pages 2849-2852. PMLR, 2021.

Gil Kur, Yuval Dagan, and Alexander Rakhlin. Optimality of maximum likelihood for log-concave density estimation and bounded convex regression. arXiv preprint arXiv:1903.05315 , 2019.

Gil Kur, Fuchang Gao, Adityanand Guntuboyina, and Bodhisattva Sen. Convex regression in multidimensions: Suboptimality of least squares estimators. arXiv preprint arXiv:2006.02044 , 2020a.

Gil Kur, Alexander Rakhlin, and Adityanand Guntuboyina. On suboptimality of least squares with application to estimation of convex bodies. In Conference on Learning Theory , pages 2406-2424. PMLR, 2020b.

Guillaume Lecué and Shahar Mendelson. Learning subgaussian classes: Upper and minimax bounds. arXiv preprint arXiv:1305.4825 , 2013.

Michel Ledoux. The concentration of measure phenomenon . Number 89 in Mathematical Surveys and Monographs. American Mathematical Soc., 2001.

Wee Sun Lee, Peter L Bartlett, and Robert C Williamson. The importance of convexity in learning with squared loss. In Proceedings of the Ninth Annual Conference on Computational Learning Theory , pages 140-146, 1996.

Tengyuan Liang and Alexander Rakhlin. Just interpolate: Kernel 'ridgeless' regression can generalize. The Annals of Statistics , 48(3):1329-1347, 2020.

Tengyuan Liang, Alexander Rakhlin, and Xiyu Zhai. On the multiple descent of minimum-norm interpolants and restricted lower isometry of kernels. In Conference on Learning Theory , pages 2683-2711. PMLR, 2020.

Shahar Mendelson. Learning without concentration. In Conference on Learning Theory , pages 25-39, 2014.

Shahar Mendelson. Extending the scope of the small-ball method. arXiv preprint arXiv:1709.00843 , 2017.

Emanuel Milman. On the role of convexity in isoperimetry, spectral gap and concentration. Inventiones mathematicae , 177(1):1-43, 2009.

Matey Neykov. On the minimax rate of the gaussian sequence model under bounded convex constraints. IEEE Transactions on Information Theory , 69(2):1244-1260, 2022.

Neyman and Elizabeth L. Scott. Consistent estimates based on partially consistent observations. Econometrica , 16(1):1-32, 1948. ISSN 00129682, 14680262.

Mikhail I Ostrovskii. Metric embeddings: Bilipschitz and coarse embeddings into Banach spaces , volume 49. Walter de Gruyter, 2013.

Alexander Rakhlin, Karthik Sridharan, and Alexandre B Tsybakov. Empirical entropy, minimax regret and minimax risk. Bernoulli , 23(2):789-824, 2017.

Richard J Samworth and Bodhisattva Sen. Special issue on "nonparametric inference under shape constraints". 2018.

Rolf Schneider. Convex bodies: the Brunn-Minkowski theory . Number 151 in Encyclopedia of Mathematics and its Applications. Cambridge University Press, 2014.

Emilio Seijo and Bodhisattva Sen. Nonparametric least squares estimation of a multivariate convex regression function. The Annals of Statistics , 39(3):1633-1657, 2011.

Yong Sheng Soh and Venkat Chandrasekaran. Fitting tractable convex sets to support function evaluations. arXiv preprint arXiv:1903.04194 , 2019.

Pragya Sur and Emmanuel J Candès. A modern maximum-likelihood theory for high-dimensional logistic regression. Proceedings of the National Academy of Sciences , 116(29):14516-14525, 2019.

Sara A van de Geer. Empirical Processes in M-estimation , volume 6. Cambridge University Press, 2000.

Roman Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge University Press, 2018.

Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge University Press, 2019.

Abraham Wald. Note on the consistency of the maximum likelihood estimate. The Annals of Mathematical Statistics , 20(4):595-601, 1949.

Yuhong Yang. Aggregating regression procedures to improve performance. Bernoulli , 10(1):25-47, 2004.

Yuhong Yang and Andrew Barron. Information-theoretic determination of minimax rates of convergence. Annals of Statistics , pages 1564-1599, 1999.

Peer reviews:
comments: SUMMARY:
This paper proves that variance for ERM enjoys a a minimax rate. The findings indicate that in scenarios where ERM is not optimal, the source of suboptimality lies within the bias component. Furthermore, these insights are extended to encompass an admissibility-type theorem for both fixed and random design, as well as a stability result for ERM.
The paper's contributions can be summarized as follows:
1)Demonstrating the optimality of the variance term in both fixed and random design situations.
2)The obtained results automatically yield a stability result for ERM.
3)Presenting a simpler proof for the admissibility theorem in the fixed and random design setting.
4)Highlighting a counterintuitive outcome concerning the ERM estimator $\widehat{f}_n$: when $f'$ is close to $\widehat{f}_n$, it can lead to a large squared error.

STRENGTHS:
The paper studies an important problem, provides convincing conclusion. The problem is not only intellectually interesting, it also advances our understanding of ERM, which is arguably the most important estimator for modern machine learning.

WEAKNESSES:
Typo: above Equation 1 $f^* \in f^*$.
Remark should be referred to in the appendix.

QUESTIONS:
Given dense results here, it would help reader to better understand the paper if the theorems could be summarized in a table.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
This paper explores the minimax optimality of ERM in terms of the bias and variance of the ERM method.
They find in some settings that the variance is always at the minimax rate, implying that suboptimality can only occur due to bias. This paper also explores stability of ERM, finding that almost-minimizers are close to the ERM w.r.t the population distribution, but not necessarily the empirical distribution.

STRENGTHS:
* Cool results and proofs in the fixed design setting
* The random design setting results are very interesting as well, albeit harder to digest all the quantities.

WEAKNESSES:
* The independence relations should be clarified in the preliminaries. I assume that $\mathbf{\xi}$ is independent from $\mathbf{X}$ in the random design case, but I don't think that's mentioned.



Minor Typos:
* In the proof of Theorem 2, $\mathcal{G}$ and $\mathcal{G}_\star$ seem to be used interchangeably.
* The statement of Lemma 2 in the proof of Theorem 2 seems to have a typo? I think it should be $f$ instead of $f^\star$.

QUESTIONS:
* Given that assumption 8 is necessary to get from a $\epsilon_V^2$ to an $\epsilon_\star^2$ bound, how restrictive is this assumption?

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
This paper studies the suboptimality of Empirical Risk minimization (ERM) of the squared loss, or equivalently, Least Squares (LS), for convex function classes, in both fixed and random design settings. In the context of non-parametric statistics, necessary and sufficient conditions for the optimality of LS are not known (though sufficient conditions were given by Birge and Massart, '93).
Roughly speaking, the main message of this paper is showing in both settings that the suboptimality is due to the bias term, where the variance term enjoys minimax rates, under some reasonable assumptions.

STRENGTHS:
This paper studies a classic and fundamental question in statistics, understanding the suboptimality of Least Squares.
This paper makes nice progress in this direction. Besides the main result (mentioned in the summary), there are several other contributions, such as the (weak) admissibility property of the ERM in the random design setting; sometimes ERM can be optimal.
Moreover, one of the results shows that ERM is stable; all approximate minimizers of the squared loss are close to each other in the functions space, and on the other end, the converse doesn't hold, sometimes the ERM is an optimal estimate and there exist close functions to it with high empirical error.

Also, the proof techniques (the isoperimetry approach for example) look interesting and elegant.

WEAKNESSES:
I don't see any.
There is a chance that I didn't understand some details, this is not my main research field and some proof techniques were new to me.

QUESTIONS:
Typos:
line 69: should be $f*\in F$.

line 16: should be "in detail".

LIMITATIONS:
I don't see any.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-08

comments: SUMMARY:
The paper considers  the Empirical Risk Minimization (ERM) problem with squared loss and shows that the suboptimality of ERM is due to the bias rather than variance.

STRENGTHS:
This paper is quite theoretical and technical. The paper provides useful insights in different aspects. The paper finds that (1) the variance term of ERM agrees with the minimax rate of estimation in both fixed and random design, (2) ERM is stable, (3) the landscape of near-solution for non-Donsker class is irregular, (4) the relation between isoperimetry and variance of ERM.

WEAKNESSES:
The main weakness is organization. Too many results wrapped in nine pages makes reader feel the paper is very crowded. 

Some definitions and results could be explained more detailedly. For example, what does it mean by the minimax rate of ERM and variance, respectively? What is local optimality? Donsker and non-donsker classes. Any specific application of current results? Technical differences between fixed and random design could be explained on a very high level.

QUESTIONS:
I found that this paper is more suitable for conference like COLT instead of NIPS. Audience from learning theory filed might be more interested in this topic.

LIMITATIONS:
More examples and applications should be included. Otherwise, it is a merely technical paper with no practical usage.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the classic problem of regression under noisy observations, with a convex and closed function class $\\mathcal{F}$. In particular, the paper considers the performance of the Empirical Risk Minimizer (ERM) under the mean squared error, in both the fixed design and random design settings.\n\nAt a high level, the author(s) show that:\n1. The variance of the ERM is comparable to the minimax rate of regression, implying that when ERM is not minimax-optimal, it must be due to large bias\n2. ERM is admissible up to constant factors, via a simpler proof using fixed point theorems\n3. ERM enjoys stability, in the sense that all almost-empirical-loss-minimizers also have similar expected loss as the actual empirical loss minimizer.\n4. For \"non-Donsker\" function classes, the converse of 3 is false, that there is always some ground truth function $f^* \\in \\mathcal{F}$ such that, with high probability over the samples and noise, there is a function $f_{\\mathrm{bad}}$ whose expected loss close to the ERM, but that its empirical loss is much larger than the ERM's.\n\nI did the following review as an emergency review, so I did not check the details/appendices carefully.\n\nSTRENGTHS:\nThe ERM is perhaps the most commonly used regression estimator. This paper furthers our understanding of its behavior, particularly for function classes where ERM isn't minimax-optimal. I found the introduction well-written, describing each result at a high level. I also find the observation that \"non-minimax-optimality must be due to bias\" to be an interesting result.\n\nWEAKNESSES:\nWhile I liked the paper, and recommend a weak accept, I think there are the following writing/presentational issues that can be improved. Overall, my comment is that the paper currently perhaps reads better for experts who have worked on ERM (or at least, in empirical process theory), but not that easy to read for a more general learning theory audience.\n\n- The paper reads a bit like a collection of related results, but without a \"main point\", and could maybe be strung together better as a story. For example, the \"variance <= minimax-rate\" result and the stability result seem somewhat disjoint in the introduction, even though later on in Theorem 1, they are shown together as one big result. I was also a bit confused about the writing in Section 2, in terms of the narrative. For example, I don't quite understand how Theorem 2 \"complements\" Theorem 1 (cf. Line 150), though Theorem 2 is a cool result itself with the local minimax optimality and is used to prove Corollary 1 (the admissibility result). Moreover, Theorem 3 also \"complements\" Theorem 1 (cf. Line 185), but there's a quadratic gap. Is the gap necessary?\n\n- I also found that the technical Section 2 is a bit too heavy on definitions and assumptions and not enough interpretation. When I was reading, I felt that the author(s) tried hard to succinctly give the most general results that are shown, but in my opinion, for readability, it might be better to start with simpler cases (and perhaps even informal versions) of statements (particularly the assumptions), and provide more interpretation.\n\n- Related to the above point, some of the assumptions/quantities in the results are stated without much interpretation or intuition (in the main body). There are also a few comments along the lines of \"this assumption is considered mild in the literature/by some other authors\" without much additional interpretation. This makes the paper not quite as self-contained as it could be. This issue is particularly prevalent in Section 2.2 (especially the isoperimetry assumptions), and it became quite hard to interpret the results. I can see that there are some remarks in the appendices, but I think a lot of them really should be in the main body for readability.\n\n- The term non-Donsker was never defined in the main body, even though it is a key element in a main result. While Donsker classes are a basic object in empirical process theory, I don't think it should be assumed knowledge for the general theory reader. There is also a lack of discussion on whether Theorem 4 only hold for non-Donsker classes, or more generally what's the significance of the assumption: whether it's a necessary condition for the result, or just that this is the result that can be proved.\n\nQUESTIONS:\nMinor questions and comments:\n1. Please consider using the same number-counter across all of definitions/theorems/assumptions. It was hard to scroll through the paper looking at backward references when reading.\n2. (Line 164) The author(s) mention that $\\mathcal{G}_\\ast$ in Theorem 2 can replace 2 with any other absolute constant. How does the replacement propagate to Theorem 2? Does it change the constants in the $\\asymp$?\n3. Theorem 6, the assumptions 1,2,4,6 have nothing to do with the instantiation of $\\mathbf{X}$?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-19"}