SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Gil Kur
gilkur@mit.edu
Eli Putterman
putterman@mail.tau.ac.il
Alexnader Rakhlin
BCS & IDSS
It is well known that Empirical Risk Minimization (ERM) may attain minimax suboptimal rates in terms of the mean squared error (Birgé and Massart, 1993). In this paper, we prove that, under relatively mild assumptions, the suboptimality of ERM must be due to its bias. Namely, the variance error term of ERM (in terms of the bias and variance decomposition) enjoys the minimax rate. In the fixed design setting, we provide an elementary proof of this result using the probabilistic method. Then, we extend our proof to the random design setting for various models. In addition, we provide a simple proof of Chatterjee's admissibility theorem (Chatterjee, 2014, Theorem 1.4), which states that in the fixed design setting, ERM cannot be ruled out as an optimal method, and then we extend this result to the random design setting. We also show that our estimates imply stability of ERM, complementing the main result of Caponnetto and Rakhlin (2006) for non-Donsker classes. Finally, we highlight the somewhat irregular nature of the loss landscape of ERM in the non-Donsker regime, by showing that functions can be close to ERM, in terms of  $L_{2}$  distance, while still being far from almost-minimizers of the empirical loss.
Maximum Likelihood and the method of Least Squares are fundamental procedures in statistics. The study of the asymptotic consistency of Maximum Likelihood has been central to the field for almost a century (Wald, 1949). Along with consistency, failures of Maximum Likelihood have been thoroughly investigated for nearly as long (Neyman and Scott, 1948; Bahadur, 1958; Ferguson, 1982). In the setting of non-parametric estimation, the seminal work of (Birgé and Massart, 1993) provided sufficient conditions for minimax optimality (in a non-asymptotic sense) of Least Squares while also presenting an example of a model class where this basic procedure is sub-optimal. Three decades later, we still do not have necessary and sufficient conditions for minimax optimality of Least Squares when the model class is large. While the present paper does not resolve this question, it makes several steps towards understanding the behavior of Least Squares — equivalently, Empirical Risk Minimization (ERM) with square loss — in large models.
Beyond intellectual curiosity, the question of minimax optimality of Least Squares is driven by the desire to understand the current practice of fitting large or overparametrized models, such as neural networks, to data (cf. [Belkin et al., 2019; Bartlett et al., 2020]). At the present moment, there is little theoretical understanding of whether such unregularized data-fitting procedures are optimal, and the study of their statistical properties may lead to new methods with improved performance.
In addition to minimax optimality, many other important properties of Least Squares on large models are yet to be understood. For instance, little is known about its stability with respect to perturbations of the data. It is also unclear whether approximate minimizers of empirical loss enjoy similar statistical
properties as the exact solution. Conversely, one may ask whether in the landscape of possible solutions, a small perturbation of the minimizer output by Least Squares itself is a near-optimizer of empirical loss.
The contribution of this paper is to provide novel insights into the aforementioned questions for convex classes of functions in a quite generic setting. In detail, we show the following:
1. We prove that in the fixed design setting, the variance term of ERM is upper bounded by the minimax rate of estimation; thus, if the ERM is minimax suboptimal, this must be due to the bias term in the bias-variance decomposition. Then, we extend this result to random design, and provide an upper bound for the variance error term under a uniform boundedness assumption on the class. This bound also implies that under classical assumptions in empirical process theory, the variance error term is minimax optimal.
2. We show that under an isoperimetry assumption on the noise, the expected conditional variance error term of ERM is upper bounded by the "lower isometry" remainder, a parameter that was introduced in Bartlett et al. (2005); Mendelson (2014). Next, under an additional isoperimetry assumption on the covariates, we prove that the variance of ERM is upper bounded by the lower isometry remainder on any robust learning architecture (namely, a class consisting of functions which are all  $O(1)$ -Lipschitz) which almost interpolates the observations (cf. Bubeck and Sellke (2023)).
3. It is known that ERM is always admissible in the fixed design setting (Chatterjee, 2014; Chen et al., 2017); that is, for any convex function class, there is no estimator having a lower error than ERM (up to a multiplicative absolute constant) on every regression function. We provide a short proof of this result for fixed design via a fixed-point theorem. Using the same method, we also prove a somewhat weaker result in the random design case, generalizing the main result of Chatterjee (2014).
4. We show that ERM is stable, in the sense that all almost-minimizers (up to the minimax rate) of the squared loss are close in the space of functions. This result is a non-asymptotic analogue of the asymptotic analysis in Caponnetto and Rakhlin (2006), and extends its scope to non-Donsker classes.
5. While any almost-minimizer of the squared loss is close to the minimizer with respect to the underlying population distribution, the converse is incorrect. We prove that for any non-Donsker class of functions, there exists a target regression function such that, with high probability, there exists a function with high empirical error near the ERM solution; this means that the landscape of near-solutions is, in some sense, irregular.
Conclusions Our results show that the ERM enjoys an optimal variance error term in two distinct regimes: the classical regime (van de Geer, 2000), where the function class is fixed and the number of samples is increasing, and the "benign overfitting" setting (Belkin et al., 2019; Bartlett et al., 2020), in which the "capacity" of the class is large compared to the number of samples. In both these settings, our work implies that the minimax optimality of ERM is only determined by its bias error term (or its implicit bias). For models with "few" parameters, computationally efficient bias correction methods do exist and are commonly used in practice (cf. (Efron and Tibshirani, 1994)), however, these methods fail over large function classes, in which the bias causes the statistical sub-optimality. Our work reveals the importance of inventing computationally efficient debiasing methods for rich function classes, including non-parametric models and high-dimensional models. The main message of our work is that such methods, if discovered, may significantly improve the statistical performance of ERM over large models in practice.
Stability of ERM The stability of learning procedures, which was an active area of research in the early 2000's, has recently seen a resurgence of interest because of its connections to differential privacy and to robustness of learning methods with respect to adversarial perturbations. In the interest of space, we only compare present results to those of Caponnetto and Rakhlin (2006). In the latter paper, the authors showed that the  $L_{1}$ -diameter of the set of almost-minimizers of empirical error (with respect to any loss function) asymptotically shrinks to zero as long as the perturbation is  $o(n^{-1/2})$  and the function class is Donsker. The analysis there relies on passing from the empirical
process to the associated Gaussian process in the limit, and studying uniqueness of its maximum using anti-concentration properties. While the result there holds without assuming that the class is convex, it is limited by (a) its asymptotic nature and (b) the assumption that the class is not too complex. In contrast, the present paper uses more refined non-asymptotic concentration results, at the expense of additional assumptions such as convexity and minimax optimal lower and upper lower isometry remainders. Crucially, the present result, unlike that of Caponnetto and Rakhlin (2006), holds for non-Donsker classes — those for which the empirical process does not converge to the Gaussian process.
Shape-constrained regression The term "shape-constrained regression" refers to function classes consisting of functions with a certain "shape" property, such as convexity or monotonicity (Samworth and Sen, 2018). In these problems, a common theme is that the statistical behavior of the class exhibits a phase transition when the dimension  $d$  of the domain reaches a certain value. For instance, in convex (Lipschitz) regression, the transition happens at  $d = 5$ : when  $d < 5$ , the ERM procedure is minimax optimal (Seijo and Sen, 2011; Han and Wellner, 2016; Kim and Samworth, 2016; Seijo and Sen, 2011; Guntuboyina, 2012), but in higher dimensions, it is known to be suboptimal (Kur et al., 2020a,b). In other shape-constrained models, however, the ERM is minimax optimal even in high dimensions, such as isotonic regression and log-concave density estimation (Han et al., 2019; Kur et al., 2019; Carpenter et al., 2018). Our results show that the sub-optimality of ERM in shape-constrained regression can only be due to the high bias of ERM. These results also align with the empirical observation that for the problem of estimation of convex sets, the ERM has a bias towards "smooth" convex sets (Soh and Chandrasekaran, 2019; Ghosh et al., 2021).
High-dimensional statistics In classical statistics, the Maximum Likelihood (MLE) typically has a low bias compared to its variance, and the standard approach is to introduce bias into the procedure in order to reduce the variance, overall achieving a better trade-off (see (Sur and Candès, 2019, §1) and references therein). In contrast, in high-dimensional models, the MLE may suffer from high bias even in tasks such as logistic regression and sparse linear regression (cf. Candès and Sur (2020); Javanmard and Montanari (2018)). Our results align with this line of work, showing that the problem of high bias may also arise in the task of regression over rich function classes.
In §2.1, we present the setting of our model and the all required preliminaries, and in the remaining sub-sections, we present our results. In details, in §2.2-2.3, we present our results on the variance of ERM in fixed and random designs respectively, and in §3 we provide sketches of some of our proofs. For lack of space, we presents our admissibility and our landscape results on ERM in the supplementary material (§4.1-4.2).
Let  $\mathcal{X}$  be some fixed domain,  $\mathcal{F}$  be a class of functions from  $\mathcal{X}$  to  $\mathbb{R}$ , and  $f^{*} \in \mathcal{F}$  an unknown target regression function. We are given  $n \geq 2$  data points  $X_{1},\ldots,X_{n} \in \mathcal{X}$  and  $n$  noisy observations
$$
Y _ {i} = f ^ {*} \left(X _ {i}\right) + \xi_ {i}, \quad i = 1, \dots , n \tag {1}
$$
which we denote by  $\mathcal{D} \coloneqq \{(X_i, Y_i)\}_{i=1}^n$ ;  $\pmb{\xi} \coloneqq (\xi_1, \dots, \xi_n)$  is the random noise vector.
In the fixed design setting, the observations  $X_{1} = x_{1},\ldots ,X_{n} = x_{n}$  are arbitrary and fixed, and we denote the uniform measure on this set of points by  $\mathbb{P}^{(n)}$ . In the random design setting, the data points  $\mathbf{X}\coloneqq (X_1,\dots,X_n)$  are drawn i.i.d. from a probability distribution over  $\mathcal{X}$ , denoted by  $\mathbb{P}$ , and the noise vector  $\xi$  is drawn independently of  $\mathbf{X}$ . Note that this model is general enough to cover the high-dimensional setting, as both the function class  $\mathcal{F}$  and the distributions of  $\xi$  and  $X$  are allowed to depend on the number of samples  $n$ .
An estimator for the regression task is defined as a measurable function  $\bar{f}_n: \mathcal{D} \mapsto \{\mathcal{X} \to \mathbb{R}\}$ , that for any realization of the input  $\mathcal{D}$ , outputs some real-valued measurable function on  $\mathcal{X}$ . The risk of  $\bar{f}_n$  is defined as
$$
\mathcal {R} \left(\bar {f} _ {n}, \mathcal {F}, \mathbb {Q}\right) := \sup  _ {f ^ {*} \in \mathcal {F}} \mathbb {E} _ {\mathcal {D}} \int \left(\bar {f} _ {n} - f ^ {*}\right) ^ {2} d \mathbb {Q}, \tag {2}
$$
where  $\mathbb{Q} = \mathbb{P}$  in the random design case, and  $\mathbb{Q} = \mathbb{P}^{(n)}$  in the fixed design case. Note that in fixed design, the expectation  $\mathbb{E}_{\mathcal{D}}$  is taken over the noise  $\pmb{\xi}$ , while in random design the expectation  $\mathbb{E}_{\mathcal{D}}$  is
taken both over the random data points  $\mathbf{X}$  and noise  $\xi$ . The minimax rate is defined via
$$
\mathcal {M} (n, \mathcal {F}, \mathbb {Q}) := \inf  _ {\bar {f} _ {n}} \mathcal {R} (\bar {f} _ {n}, \mathcal {F}, \mathbb {Q}). \tag {3}
$$
In the fixed design setting, we also denote the minimax rate by  $\mathcal{M}(\mathcal{F},\mathbb{P}^{(n)})$ , as the dependence in  $n$  is already present in  $\mathbb{P}^{(n)}$ .
The most natural estimation procedure is the Least Squares (LS) or Empirical Risk Minimization (ERM) with squared loss, defined as
$$
\widehat {f} _ {n} \in \underset {f \in \mathcal {F}} {\operatorname {a r g m i n}} \sum_ {i = 1} ^ {n} \left(f \left(X _ {i}\right) - Y _ {i}\right) ^ {2}. \tag {4}
$$
When studying fixed design, we will abuse notation and treat  $\widehat{f}_n$  as a vector in  $\mathbb{R}^n$ . We emphasize that many of our results hold for many other estimators, including various regularized ERM procedures (see the relevant remarks below).
In both the fixed and random design settings, we shall assume the following:
Assumption 1.  $\mathcal{F}$  is a closed convex subset of  $L_{2}(\mathbb{Q})$ , where  $\mathbb{Q} \in \{\mathbb{P}^{(n)}, \mathbb{P}\}$ .
The convexity of  $\mathcal{F}$  means that any  $f, g \in \mathcal{F}$  and  $\lambda \in [0,1]$ :  $\lambda f + (1 - \lambda)g \in \mathcal{F}$ ; closedness means that for any sequence  $\{f_n\}_{n=1}^{\infty} \subset \mathcal{F}$  converging to  $f$  with respect to the norm of  $L_2(\mathbb{Q})$ , the limit  $f$  lies in  $\mathcal{F}$ . The closedness ensures that  $\widehat{f}_n$  is well-defined.
Assumption 1 is standard in studying the statistical performance of the ERM (cf. Lee et al. (1996); Bartlett et al. (2005); Mendelson (2014)). In particular, under this assumption, the values of  $\widehat{f}_n$  at the observation points  $X_{1},\ldots ,X_{n}$  is uniquely determined for any  $\xi$ . Note that in general, the values of a function  $f\in \mathcal{F}$  at the points  $X_{1},\ldots ,X_{n}$  does not uniquely identify  $f$  among all functions in the class.
In addition to  $\widehat{f}_n$ , we analyze properties of the set of  $\delta$ -approximate minimizers of empirical loss, defined for  $\delta > 0$  via
$$
\mathcal {O} _ {\delta} := \left\{f \in \mathcal {F}: \frac {1}{n} \sum_ {i = 1} ^ {n} \left(Y _ {i} - f \left(X _ {i}\right)\right) ^ {2} \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \left(Y _ {i} - \widehat {f} _ {n} \left(X _ {i}\right)\right) ^ {2} + \delta \right\}. \tag {5}
$$
Note that  $\mathcal{O}_{\delta}$  is a random set, in both fixed and random designs.
It is well-known that the squared error of any estimator, in particular that of LS, decomposes into variance and bias components:
$$
\mathbb {E} _ {\mathcal {D}} \int \left(\widehat {f} _ {n} - f ^ {*}\right) ^ {2} d \mathbb {Q} = \underbrace {\mathbb {E} _ {\mathcal {D}} \int \left(\widehat {f} _ {n} - \mathbb {E} _ {\mathcal {D}} \widehat {f} _ {n}\right) ^ {2} d \mathbb {Q}} _ {V (\widehat {f} _ {n})} + \underbrace {\int \left(\mathbb {E} _ {\mathcal {D}} \widehat {f} _ {n} - f ^ {*}\right) ^ {2} d \mathbb {Q}} _ {B ^ {2} (\widehat {f} _ {n})}, \tag {6}
$$
where  $\mathbb{Q} = \mathbb{P}^{(n)}$  in the fixed design setting and  $\mathbb{Q} = \mathbb{P}$  in the random design setting. Also, for simplicity of the presentation of our results, we denote the maximal variance error term of  $\bar{f}_n$  by  $\mathcal{V}(\bar{f}_n,\mathcal{F},\mathbb{Q})$ , i.e.
$$
\mathcal {V} (\bar {f} _ {n}, \mathcal {F}, \mathbb {Q}) := \sup  _ {f ^ {*} \in \mathcal {F}} V (\bar {f} _ {n}) = \mathbb {E} _ {\mathcal {D}} \int (\bar {f} _ {n} - \mathbb {E} _ {\mathcal {D}} \bar {f} _ {n}) ^ {2} d \mathbb {Q}
$$
In the random design setting, we also have the law of total variance:
$$
V \left(\widehat {f} _ {n}\right) = \underbrace {\mathbb {E} _ {\mathbf {X}} \mathbb {E} _ {\boldsymbol {\xi}} \left[ \int \left(\widehat {f} _ {n} - \mathbb {E} _ {\boldsymbol {\xi}} \left[ \widehat {f} _ {n} | \mathbf {X} \right]\right) ^ {2} d \mathbb {P} \right]} _ {\mathbb {E} V \left(\widehat {f} _ {n} | \mathbf {X}\right)} + \underbrace {\mathbb {E} _ {\mathbf {X}} \left[ \int \left(\mathbb {E} _ {\boldsymbol {\xi}} \left[ \widehat {f} _ {n} | \mathbf {X} \right] - \mathbb {E} _ {\mathbf {X} , \boldsymbol {\xi}} \widehat {f} _ {n}\right) ^ {2} d \mathbb {P} \right]} _ {V (\mathbb {E} \left(\widehat {f} _ {n} | \mathbf {X}\right))}. \tag {7}
$$
We refer to the two terms as the expected conditional variance and the variance of the conditional expectation, respectively. We conclude this introductory section with a bit of notation and a definition.
Notation We use the notation of  $\asymp, \gtrsim, \lesssim$  to denote equality/inequality up to an absolute constant. We use  $\| \cdot \| = \| \cdot \|_{L_2(\mathbb{P})}$  to denote the  $L_2(\mathbb{P})$  norm, and  $\| \cdot \|_n$  to denote the  $L_2(\mathbb{P}^{(n)})$  norm (that is equal to the Euclidean norm scaled by  $1/\sqrt{n}$ ). Finally, given a function  $f: S \to T$  between metric spaces, we define its Lipschitz constant as  $\| f \|_{Lip} = \sup_{a,b \in T, a \neq b} \frac{d_T(f(a), f(b))}{d_S(a,b)}$ , and we say "  $f$  is  $L$ -Lipschitz" when its Lipschitz constant is at most  $L$ . Finally, we denote by  $\mathrm{diam}_{\mathbb{Q}}(\mathcal{H})$  the  $L_2(\mathbb{Q})$  diameter of a set of functions  $\mathcal{H}$ .
Definition 1. Let  $\epsilon \geq 0$ ,  $\mathcal{F} \subseteq \{\mathcal{X} \to \mathbb{R}\}$  and  $d(\cdot, \cdot)$  a pseudo-metric on  $\mathcal{F}$ . We call a set  $S \subset \mathcal{F}$  an  $\epsilon$ -net of  $\mathcal{F}$  with respect to  $d$  if for any  $f \in \mathcal{F}$  there exists  $g \in S$  with  $d(f, g) \leq \epsilon$ . We denote by  $\mathcal{N}(\epsilon, \mathcal{F}, d)$  the  $\epsilon$ -covering number of  $\mathcal{F}$  with respect to  $d$ , that is, the minimal positive integer  $N$  such that  $\mathcal{F}$  admits an  $\epsilon$ -net of cardinality  $N$ .
2.2 Variance of ERM in fixed design setting
In this part, we consider some fixed  $(\mathcal{F},\mathbb{P}^{(n)})$  and assume that the noise is standard normal.
Assumption 2. The noise vector  $\xi$  is distributed as an isotropic Gaussian, i.e.  $\xi \sim N(0,I_{n\times n})$ .
Our first result provides an exact characterization of the variance (up to a multiplicative absolute constant) under Assumptions 1-2. In order to state it, for a fixed  $f^{*} \in \mathcal{F}$ , we define the following set:
$$
\mathcal {H} _ {*} := \{f \in \mathcal {F}: \| f - \mathbb {E} _ {\mathcal {D}} \widehat {f} _ {n} \| _ {n} ^ {2} \leq 4 \cdot V (\widehat {f} _ {n}) \}. \tag {8}
$$
In words, when the underlying function  $f^{*}$  is fixed, we consider the ERM as a random vector (depending on the noise), whose expectation we denote by  $\mathbb{E}_{\mathcal{D}}\widehat{f}_n$ .  $\mathcal{H}_{*}$  is then just a neighborhood around the expected ERM with a radius of order the square root of the variance error term of  $\widehat{f}_n$  when the underlying function is  $f^{*} \in \mathcal{F}$ .
We can now state our first result, which uses the notion of the set  $\mathcal{O}_{\delta}$  of  $\delta$ -approximate minimizers from (5).
Theorem 1. Under Assumptions 1-2, the following holds:
$$
V _ {\mathcal {D}} (\hat {f} _ {n}) \asymp \mathcal {M} \left(\mathcal {H} _ {*}, \mathbb {P} ^ {(n)}\right),
$$
and in particular  $\mathcal{V}(\widehat{f}_n,\mathcal{F},\mathbb{P}^{(n)})\lesssim \mathcal{M}(\mathcal{F},\mathbb{P}^{(n)})$  . Furthermore, for  $\delta \coloneqq \delta (f^{*},n)\lesssim \mathcal{M}(\mathcal{H}_{*},\mathbb{P}^{(n)})$  the event
$$
\sup  _ {f \in \mathcal {O} _ {\delta}} \int (f - \mathbb {E} _ {\mathcal {D}} \widehat {f} _ {n}) ^ {2} d \mathbb {P} ^ {(n)} \asymp \mathcal {M} \left(\mathcal {H} _ {*}, \mathbb {P} ^ {(n)}\right). \tag {9}
$$
holds with probability at least  $\max \{1 - 2\exp (-cn\cdot \mathcal{M}(\mathcal{H}_*,\mathbb{P}^{(n)})),0.9\}$ , where  $c\in (0,1)$  is an absolute constant.
Theorem 1 establishes our first claim: the variance of ERM is bounded above (up to a multiplicative absolute constant) by the minimax rate of estimation on  $\mathcal{H}_{*}$ . Since  $\mathcal{H}_{*}$  is contained in  $\mathcal{F}$ , its minimax rate is at most that of  $\mathcal{F}$ ; in particular, the variance of ERM is bounded by  $\mathcal{M}(\mathcal{F},\mathbb{P}^{(n)})$ , and hence, any sub-optimality of ERM must arise from its bias. The theorem also incorporates a stability result: not only is the ERM close to its expected value  $\mathbb{E}_{\mathcal{D}}\widehat{f}_n$  with high probability, but any approximate minimizer (up to an excess error of  $\delta^2$ ) is close to  $\mathbb{E}_{\mathcal{D}}\widehat{f}_n$  as well.
Our next result complements Theorem 1 above, providing a lower bound on  $\mathcal{V}(\widehat{f}_n,\mathcal{F},\mathbb{P}^{(n)})$
Theorem 2. Under Assumptions 1-2, the following holds:
$$
\mathcal {V} (\widehat {f} _ {n}, \mathcal {F}, \mathbb {P} ^ {(n)}) \gtrsim \mathcal {M} (\mathcal {F}, \mathbb {P} ^ {(n)}) ^ {2}.
$$
Note that there is a multiplicative gap of order  $\mathcal{M}(\mathcal{F},\mathbb{P}^{(n)})$  between the bounds of Theorems 1 and 2. We leave it as an open problem whether the bound of Theorem 2 can be improved under these general assumptions.
2.3 Variance of ERM in random design setting
We now turn our attention to the random design setting. Here, we establish similar results to the previous sub-section, albeit under additional assumptions, and with significantly more effort. Unlike the fixed design case, we cannot provide an exact characterization of the variance of ERM. We shall use two different approaches to estimate the variance error term. In the first approach, we use classical tools of empirical process theory together with assumptions that are commonly used in M-estimation (van de Geer, 2000). The second approach, which is inspired by our fixed-design approach, relies heavily on isoperimetry and concentration of measure (cf. Ledoux (2001)).
Throughout this part, we assume for simplicity of presentation that the  $L_{2}(\mathbb{P})$ -diameter of the function class is independent of  $n$ .
Assumption 3. There exist absolute constants  $C, c > 0$  such that  $c \leq \mathrm{diam}_{\mathbb{P}}(\mathcal{F}) \leq C$ .
The classical work of Yang and Barron (1999) provides a characterization of the minimax rate  $(n, \mathcal{F}, \mathbb{P})$  under appropriate assumptions (such as normal noise, and uniform boundedness of  $\mathcal{F}$ , and richness of  $\mathcal{F}$ ). They proved that the minimax rate is the square of the solution of the following (asymptotic) equation
$$
\log \mathcal {N} (\epsilon , \mathcal {F}, \mathbb {P}) \asymp n \epsilon^ {2}, \tag {10}
$$
where  $\mathcal{N}(\epsilon, \mathcal{F}, \mathbb{P})$  is the  $\epsilon$ -covering number of  $\mathcal{F}$  in terms of  $L_2(\mathbb{P})$  metric (see Def. 1 above). We denote this point by  $\epsilon_* = \epsilon_*(n)$ , and even under less restrictive assumptions,  $\epsilon_*^2$  also lower bounds the minimax rate, up to a multiplicative factor of  $O(\log n)$ . Also, we remark that it is well known that ERM may not achieve this optimal rate (Birgé and Massart, 1993) for large (so-called non-Donsker) function classes.