Benign, Tempered, or Catastrophic: Toward a Refined Taxonomy of Overfitting
Authors: Neil Rohit Mallinar
James B Simon
Amirhesam Abedsoltan
Parthe Pandit
Misha Belkin
Preetum Nakkiran
Venue: NeurIPS 2022 Accept (2022)
Keywords: overfitting
benign overfitting
generalization
kernel regression
powerlaws
eigenspectra

Abstract: The practical success of overparameterized neural networks has motivated the recent scientific study of \emph{interpolating methods}-- learning methods which are able fit their training data perfectly. Empirically, certain interpolating methods can fit noisy training data without catastrophically bad test performance, which defies standard intuitions from statistical learning theory. Aiming to explain this, a large body of recent work has studied \emph{benign overfitting}, a behavior seen in certain asymptotic settings under which interpolating methods approach Bayes-optimality, even in the presence of noise. In this work, we argue that, while benign overfitting has been instructive to study, real interpolating methods like deep networks do not fit benignly. That is, noise in the train set leads to suboptimal generalization, suggesting that these methods fall in an intermediate regime between benign and catastrophic overfitting, in which asymptotic risk is neither is neither Bayes-optimal nor unbounded, with the confounding effect of the noise being ``tempered" but non-negligible. We call this behavior \textit{tempered overfitting}. We first provide broad empirical evidence for our three-part taxonomy, demonstrating that deep neural networks and kernel machines fit to noisy data can be reasonably well classified as benign, tempered, or catastrophic. We then specialize to kernel (ridge) regression (KR), obtaining conditions on the ridge parameter and kernel eigenspectrum under which KR exhibits each of the three behaviors, demonstrating the consequences for KR with common kernels and trained neural networks of infinite width using experiments on natural and synthetic datasets.

Benign, Tempered, or Catastrophic: A Taxonomy of Overfitting
Neil Mallinar* UC San Diego
James Simon" UC Berkeley jemesssinon@bberkeee..uuuu
Amirhesam Abedsoltan UC San Diego aabedsoltan@ucsod edu
nmallina@ucsd.edu
Parthe Pandit
Mikhail Belkin
Preetum Nakkiran
UC San Diego
UC San Diego
Apple & UC San Diego
parthepandit@ucsddeddu
mbelkin@ucsd.eddi
preetum@apple.com
Abstract
The practical success of overparameterized neural networks has motivated the recent scientific study of interpolating methods, which perfectly fit their training data. Certain interpolating methods, including neural networks, can fit noisy training data without catastrophically bad test performance, in defiance of standard intuitions from statistical learning theory. Aiming to explain this, a body of recent work has studied benign overfitring phenomenon where some interpolating methods approach Bayes optimality, even in the presence of noise. In this work we argue that while benign overfitting has been instructive and fruitful to study, many real interpolating methods like neural networks do not fir benignly: modest noise in the training set causes nonzero (but non-infinite) excess risk at test time, implying these models neither benign nor catastrophic but rather fall in intermediate regime. We call this intermediate regime tempered overfitting, and we initiate its systematic study. We first explore this phenomenon in the context of kernel (ridge) regression (KR) by obtaining conditions on the ridge parameter and kernel eigenspectrun under which KR exhibits each of the three behaviors. We find that kernels with powerlaw spectra, including Laplace kernels and ReLU neural tangent kernels, exhibit tempered overfitting. We then empirically study deep neural networks through the lens of our taxonomy, and find that those trained interpolation are tempered, while those stopped early are benign. We hope our work leads to more refined understanding of overfitting in modern learning.
Introduction
In the last decade, the dramatic success overparameterized deep neural networks (DNNs) has inspired the field to reexamine the theoretical foundations of generalization. Classical statistical learning theory suggests that an algorithm which interpolates (i.e. perfectly fits) its training data will typically catastrophically overfit at test time, generalizing no better than random function.
Figure Ic illustrates the catastrophic overfitting classically expected of interpolating method. Defying this picture, DNNs can interpolate their training data and generalize well nonetheless [Neyshabur et al., 2015, Zhang et al., 2017], suggesting the need for a new theoretical paradigm within which to understand their overfitting.
"Co-first authors.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Benign
Tempered
Catastrophic
B
trainect
prodicted f
Figure 1: As n - oc, interpolating methods can exhibit three types of overfitting. (A) In
benign overfitting, the predictor asymptotically approaches the ground-truth, Bayes-optimal function. Nadaraya- Watson kernel smoothing with singular kernel, shown here, is asymptotically benign. (B) In tempered overfitting, the regime studied in this work, the predictor approaches a constant test risk greater than the Bayes-optimal risk. Piecewise-linear interpolation is asymptotically tempered. (C) In catastrophic overfilling, the predictor generalizes arbitrarily poorly. Rank-n polynomial interpolation is asymptotically catastrophic.
This need motivated the identification and study of benign overfitting using the terminology of [Bartlett et al., 2020] (also called "harmless interpolation" [Muthukumar et al., 2020]), a phenomenon in which certain methods that perfectly fit the training data still approach Bayes-optinal generalization in the limit of large trainset size. Intuitively speaking, benignly-overfitinn methods fit the target function globally, yet fit the noise only locally, and the addition of more label noise does not asymptotically degrade generalization. Figure la illustrates a simple method that is asymptotically benign'. The study of benign overfitting has proven fruitful, leading to rich mathematical insights into high- dimensional learning', and benign overfitting is certainly closer to the real behavior of DNNs than catastrophic overfitting.
That said, it requires only simple experiments to reveal that many standard DNNs do not overfit benignly: when training noisy data, DNNs do not diverge catastrophically. but neither do they approach Bayes-optimal risk. Instead, they converge to predictor that is neither catastrophic nor optimal but rather somewhere in between, with error that increases as the noise in the data increases. Figure 2 depicts such an experiment: a ResNet trained on a binary variant of CIFAR- 10 with varying amounts of training label noise, and with increasing sample size We see from Figure 2 that greater train noise indeed results in greater test error, and this test error persists even as n grows, converging non-zero asymptotic value*. This is unlike "benign overfitting." which would produce an asymptotically-optinn
predictor at all non-trivial noise levels (depicted in blue in Figure 2). This suggests that, in the search for paradigm to understand modern interpolating methods, we should identify and study a regime intermediate between benign and catastrophic.
1.1 Summary of Contributions
In this work formally identify intermediate regime between benign and catastrophic overfitting. We call this intermediate behavior tempered overfittin because the noise's harmful effect is tempered but still nonzero. We find that both DNNs trained to interpolation and (ridgeless) kernel regression (KR) using certain common kernels fall into this intermediate regime even as the number of training examples approaches infinity, as do common methods like I-nearest-neighborss and piecewise- linear interpolation (as in Figure lb). Our tempered regime completes the taxonomy of overfitting: essentially any learning procedure is either benign, tempered, or catastrophic in the asymptotic limit.
2 As a first hint that important practical methods may not be benign (at least in low dimension), note that, in order to be benign, the predicted function in Figure la has to take this spiky shape. On the other hand, very wide and deep neural network may indeed be spiky Radhakrishnan et al. [2022].
A partial list of works here include Advani and Saxe [2017]. Bahri et al. [2020, 2021], Bartlett et al. [2020, 2021], Belkin et al. [2018a, Sa,b, 2019a], Cao et al. [2022], Chatterji and Long [2021], Chatterji et al. [2021], I]. d' Ascoli et al. [2020]. Frei et al. [2022]. Goldt et al. [2019]. Hastie et al. [2019], Koehler et al. [2021]. I]. Liang and
Rakhlin [2018. 2020]. Mei and Montanari [2019]. Muthukumar et al. [2020]. Rakhlin and Zhai [2019]. Tsigler and Bartlett [2020]. Zhang et al. [2017, 2021].
'It is knoll and is perhaps unsurprising that interpolating DNNs are harmed by label noise (e.g. Zhang al. [2017]): our new observation is that this persists even sc.

---

We begin in Section 2 with preliminaries, formal definitions of the three regimes, and a taxonomy of some common ML methods according to these regimes. In Section 3, we study these three regimes for kernel regression (KR). Using recent spectral theories characterizing the expected test error of KR, we obtain conditions on the ridge parameter and kernel eigenspectrum under which KR falls into each of the three regimes. Importandly, we find that ridgeless kernels with powerlaw spectra, including the Laplace kernel and ReLU fully-connected neural tangent kernels (NTKs), are asymptotically tempered, not benign. We confirm our theory with experiments on synthetic data. In Section 4, we empirically study overfitting for DNNs. We give evidence that standard DNNs trained to interpolation exhibit tempered overfitting, not benign overfitting, motivating the further study of tempered overfitting in the pursuit of understanding modern machine learning methods. We additionally study the time dynamics of overfitting, and the effect of early-stopping. We conclude with discussion in Sections 5 and 6.
1.2 Relation to Prior Work
DNN Noise Profile (Binary CIFAR-10)
Our work is inspired by recent developments in the theory and empirics of interpolating methods [Bartlett et al., 2020, Belkin et al., 20, 2019c,d, Devroye et al., 1998, Ji et al., 2021, Koehler et al., 2021, Liang and Rakhlin, 2018 2020, Muthukumar et al., 2020, Rakhlin and Zhai, 2019. Tsigler and Bartlett, 2020]. Many of these works prove that certain interpolating estimators are statistically consistent in certain set- tings, demonstrating benign overfitting. In contrast, we argue that a wide range of interpolators, includ- ing DNNs used in practice, are not benign. Our work is compatible with prior work because we consider different settings-erring more the side of realism- which leads us to different conclusions. The empirical observation that interpolating DNNs for classification are inconsistent, and thus not benign, was made in Nakkiran and Bansal [2020], which in part inspired the present work.
0.5
Catastrophie
0.4
Cieean Test Error
0.3
Pr Fx)
0.2
0.1
Benign
0.0
0.0
0.1
0.2
0.3
0.4
0.5
Train Label Flip Prob
Pr(g x
n 5k
n-
25k
n 10k
40k
Figure DNNs trained on image data ex- hibit tempered overfitting, not benign over- fitting. Curves show test classification error vs. training label flip probability for a Wide ResNet 28 x 10 trained to interpolation on binary CIFAR- 10 (animals vs. vehicles) for different training sizes n. These curves are noise profiles, as discussed in Section 2.
Briefly, many prior works assume that the target task always lies in "high enough dimension" relative to the number of samples n, while our work considers the limit n + oo for tasks of fixed dimension d, the real- istic asymptotic in practice. For example, in the linear regression setting, Bartlett et al. [2020] highlight that benign overfitting occurs most robustly when input dimension grows faster than the sample size. Other works explicitly scale the ambient problem dimension and sample size to infinity together at a proportional rate [Hastie et al., 2019, Liang and Rakhlin, 2018, Mei and Montanari, 2019]. This joint scaling is also often considered in statistical physics approaches to learning dynamics (see Zdeborová and Krzakala [2016] and references therein). Taking a different approach, Frei et al. [2022] prove that interpolating two-layer networks can achieve close to optimal test error on certain distributions, but require an assumption that < R(d). And Koehler et al. [2021] state generalization bound for interpolators that decays to 0 with n, but also requires n < d. In summary, a bulk of prior benign overfitting results apply in the regime where is large, but still restricted to be smaller than the dimension of the problem, whereas we do not consider such restrictions.
Our work is also compatible with prior works which observe that excess risk of certain interpolating methods decays with ambient dimension, interpreted as high -igh-dimensional problems enjoying a "blessing of dimensionality." For example, the simplicial interpolation scheme of Belkin et al. [2018a) has excess risk that decays as 0(2-4) for ambient dimension d, and Rakhlin and Zhai [2019] show that kernel ridgeless regression with the Laplace kernel is inconsistent in any fixed dimension, but with lower bound on risk that decays with dimension. We find an explicit result to this effect when studying KR: for Laplace kernels and ReLU NTKs, asymptotic excess mean squared error decays
3

---

like ((1/d). These phenomena support our claim that many interpolating methods are tempered on real distributions, which have fixed dimension.
The Three Types of Overfitting
Here formally present taxonomy of overfitting, delineating the three types of asymptotic behaviors which learning procedures can exhibit.
2.1 Definitions
We consider a fairly generic in-distribution supervised learning setting. For simplicity, we present definitions for regression, but these are readily extended to classification. We wish learn a function f X + I from a size-n dataset of i..d. samples Dn, = {(r,, 3:) ~ DD where is a joint distribution over X' R, and X is the input domain. We shall generally assume nonzero target noise, with Var[ys O. We evaluate the generalization performance of f by the mean squared error (MSE): R(J)= : Er (f(x) -) The Bayes-optimal regression function is given by f argmin, R(f). where the minimization is over all measurable functions, and has risk R which is called the irreducible risk. The excess risk of any function f is given by R()) == R(f)-Rt We say estimator achieves interpolation if (zx) y, for all (æ), y4) D,,. These definitions are readily generalized to classification, using classification error in place of MSE.
w)²
Learning Procedure. The objects of study in our taxonomy are learning procedures. Our definition of a learning procedure is quite general, allowing discussion of methods from DNNs to l-nearest- neighbors. Informally, learning procedure simply specification of which model to output given train set of given size.
Formally, a learning procedure A := {An)n is sequence of (potentially stochastic) functions, indexed by sample size n E N. each n, the function A, D, + Fn inputs train set D,, and outputs a "model" Fr : X + R. Note that this n-dependence allows learning procedures to be non-uniform, varying the learning algorithm with sample size n. For example, it allows procedures which scale up DNN or narrow kernel bandwidth as n grows. The expected risk of a learning procedure examples from distribution D is R, = EA.. D., (R(A,,))))))
2.2 The Taxonomy
We shall categorize learning procedures terms of their asymptotic expected risk. We handle regression and K-class classification settings separately, due to their different loss scalings. As the number of samples n + oo, the sequence of expected risks (R, hn can behave three different ways, as listed in Table 1. These three limiting behaviors define our taxonomy.
Regression
Classification
Benign
lim, R₂ = R.
lim R₂ = R
Tempered
tom R₂ (R', 00)
lim R₂ E (R* 1
Catastrophic
lim Rn 00
limn Rr = - 1 K
Table 1: Our taxonomy of (over)fitting.
There is technically fourth option that the limit does not exist iisst--uut but, knowledge, this does not describe any non patholllolliaal algorithms. All together, this set of behaviors is exhaustive, describing every possible learning procedure. Note that the definitions for classification and regression are identical, except for the bounds at do replaced by the error of the predictor choosing uniformly random label
We assume balanced classes throughout, for notational simplicity. The definitions can be modified appropri- ately for imbalanced classes.

---

Benign
Tempered
k-NN (k ~ log n)
One Epoch MLP
1-NN
Overfit MLP
Catastrophic
Catasteophic
Catastrophic
Catastrophic
Ckean Test Eirre
Clean Tost Exror
Clean Tost Exron
Clean Test Exror
0.3
00.3
10.3
0.2
0.2
0.1
Ronigns
Bonigns
Bonign
Banign
0.0
0.0
0.0
0.0
0.1
0.3
0.4
0.1
0.3
0.4
0.1
0.3
0.4
0.1
0.3
0.4
Train Label Flip
Train Label Plip
Train Label Plip Prob
Train Label Plip Prob
60000
Figure Examples of Benign and Tempered Fitting, Noise profiles for several different methods on the Binary-MNIST classification task, showing clean test error as a function of train label noise the train set grows. Left: two methods which exhibit benign overfitting, with performance converging to Bayes optimal + Right: two methods which exhibit tempered overfitting, with test error that remains bounded away from O. The two "tempered" methods here are interpolating, while the two "benign" methods are not. Both the benign and the tempered MLP use identical architectures; one is trained for one epoch, and the other trained interpolation. Details in Appendix C.3.
Benign
Tempered
Catastrophic
Early-stopped DNNs
Interpolating DNNs
Gaussian KR
KR with ridge
Laplace KR
k-NN log m)
ReLU NTKs
regression
Nadaraya- Hadaraya-Watton kernel smoothing with Hilbert ker- nel
k-NN (constant k)
Simplicial interpolation
Table A taxonomy models under the three types of fitting identified in this work. BOLD are results from our work, others are known or folklore results.
2.3 Noise Profiles
To study asymptotic risk, we use a tool we call a noise profile. A noise profile characterizes the sensitivity of a learning procedure to noise in the training set. For a given learning procedure A and data distribution 'D, noise profile PA describes how the asymptotic risk varies with respect to a, the level of artificial noise added to training targets. Formally, the noise profile PA Pa lim (E,.,,,.o') (o) R(A, (D)))))), where (a) denotes i.i.d. samples from the distribution D, with o-level of label noise added. The label noise o E denotes a kind of noise that depends on the problem setting: is the variance of additive Gaussian noise for regression settings, or the label-flip probability for classification settings. We cannot empirically evaluate the n - oo limit exactly, so estimate it using asymptoties from finite but large sample sizes. As shown in Figure 2, noise profiles are easily plotted and reveal at a glance whether an learning procedure is benign, tempered, or catastrophic.
2.4 Applying Our Taxonomy
Our taxonomy handles general learning procedures, even those which do not interpolate their train sets Thus, we can apply it describe many existing methods in machine learning. In doing so, we refine the language around statistical consistency: many methods were known to be statistically inconsistent (i.e. benign), but we highlight that there are distinct ways to be inconsistent: tempered and catastrophic.
To illustrate our taxonomy, give several examples of known results in Table 2. Any statistically consistent method is by definition benign: this includes non-interpolating methods such as early-
We use "overfitting" to describe interpolating methods, and "fitting" describe general methods.
5

---

stopped DNNs [Ji et al., 2021] and d--nearet-neiighoor with k ~ logn [Chaudhuri and Dasgupta, 2014, Cover and Hart, 1967], as well as interpolating methods such as Nadarya- Badarya-Watte Watson kernel smoothing [Devroye et al., 1998] and k-NN schemes [Belkin et al., 2018 1018,, 2019c].
Many catastrophic learning procedures are also known. Parametric models which are "critically- parameterized" (i.e. at the double descent peak) overfit catastrophically: this includes random feature regression with number of features p- n [Mei and Montanari, 2019], and, more generally, for a broad class of linear and random feature methods [Holzmüller, 2021]. Generic empirical--iss-miinn -iiiii-i-miiiizatiin for a hypothesis class with VC-dimension > will also overfit catastrophically in the worst case. turns out that RBF kernel regression under natural assumptions also catastrophically overfits, as we show in Section 3.
Finally, many methods, from classical to modern, exhibit tempered overfitting. First, it is well-known that 1-nearest-neighhors (I-NN) converges an asymptotic risk that is finite but bounded away from Bayes optimal', which is tempered behavior. Further, any "underfitting" method, such as empirical risk minimization over hypothesis class with VC-dimension < n, will be tempered if the hypothesis class does not contain the ground-truth function. In Section 4, we empirically demonstrate that many standard DNNs exhibit tempered overfitting when trained interpolation. We complement this with theoretical results in Section 3, showing that kernel regression with the Laplace kernel, as well as with the ReLU NTK (which describes the training of an infinite-width ReLU fully-connected network [Jacot et al., 2)), exhibits tempered overfitting. Our new category of tempered overfitting thus not merely a theoretical possibility but in fact captures many natural and widely-used learning methods.
In Figure 3 demonstrate our taxonomy experimentally for two benign methods (k-NN and early- stopped MLPs) and two tempered methods (I-NN and interpolating MLPs) on binary classification version of MNIST, with varying noise in the train labels. We plot test classification error on the clean test set against the proportion of flipped labels in the training set. As grows, the benign methods approach zero test error even at nonzero train noise, while the tempered methods converge test error bounded away from zero
3 Overfitting in Kernel Regression
We begin with study of kernel regression (KR), widely-used nonparameteric learning algorithm which, we will see, is sufficiently rich to exhibit all three regimes of overfitting, yet sufficiently simple that this can be shown analytically. Theoretical interest in this algorithm has increased significantly in recent years due to the discovery that trained DNNs converge to ridgeless KR in the infinite-width and infinite-time limit [Jacot al., 2018], implying that insights into KR simultancously shed light on overparameterized DNNs. The overparameterized linear regression setting of Bartlett et al. [2020] equivalent KR, and we will make direct comparison with their results at the end of the section.
KR is fully specified by a positive-semidefnnie kernel function K : Rdd x IRd -> IR and a ridge parameter 8 2 O. We allow the training set D, to contain samples (I;, U:) ~ D, and we assume that (xx + m with true function I" and noise 7); ~ N(000²), KR returns the predicted function f given by
- Dw) (K(P., D.)) + (I.
where (D,,,, D..) is the "data-data kernel matrix" with components K(D,,, Dn)is = ny), K(x, is row vector with components ((x, D); K(x, 2.),, and y is column vector of targets.
Existing literature provides examples of KR exhibiting all three asymptotic behaviors. Benign overfitting has been analyzed for overparameterized linear regression [Advani and Saxe, 2017, Bartlett et al., 2020, Belkin et al., 2019b, Hastie et al., 2019, Muthukumar et al., 2020), a special case of KR. It is well-known that KR with positive ridge value (with appropriate scaling conventions) is consistent and thus benign [Christmann and Steinwart, 2007]. Furthermore, in ID, a Laplace kernel approaches piecewise lincar interpolation as n co [Belkin et al., 201 8a], which is casily shown to exhibit tempered overfitting, and Rakhlin and Zhai [2019] proved that the Laplace kernel does not overfit benignly in (fixed) dimension greater than one (though did not say whether it was in fact tempered or catastrophic). Finally, it is known in experimental folklore (though not theoretically, to our knowledge) that KR with a Gaussian kernel and zero ridge tends to yield condly-conditoone
In fact, the excess test MSE converges to the variance of the observation noise.

---

kernel matrices and catastrophic behavior. Here we derive fairly general conditions under which KR falls into each regime, solidifying these various observations into a unified picture.
As our chief tool for obtaining these conditions, we use a recently-errived closed-form approximation for the expected test MSE of KR [Bordelon et al., 2020, Canatar et al., 2021, Jacot et al., 2020, Simon et al., 2021]. I]. By simply taking the n + oc limit of this expression, we can classify given kernel into one of our three regimes.
The expression we will use gives test MSE terms the eigenspectrum of the kernel (as given by the Mercer decomposition) and the eigendecomposition of the target function. The nonnegative eigenvalues \1 2 X2 2 2 0 and orthonormal eigenfunctions { are given by Ex'~pl K (x, x (x')) (x), where (x
= 8,-
We note that kernel must be positive semidefinite and that =r|k] = Ex~pp [K (2, x)}, which we assume is finite. Because the eigenfunctions form complete basis, we are free decompose the target function as (*(x)= Σ, (,),()), where {w are eigencoefficients.
The above- mentioned works derive equivalent closed-form expressions for the test MSE of KR in terms this spectral information using methods from the statistical physics literature. These methods nonrigorous and rely on approximations (see Appendix A for discussion), but they expected to become exact the large-n limit, and comparison with empirical KR generally confirms a close match even modest Here we use the framework of Simon al. [2021], which expresses the final result in terms of "modewise learnabilities" " {Ci [. set of scores in [0, 1] which indicate how well each eigenmode is learned given n. This choice of variables will simplify our proofs.
Simon et al. [2021] find that test MSE R. approximated by
Rr En Eo -1-11))0 + o² ) where En
n Σ, C3
A; and
C, A +
2 satisfies
Σ
A +
This MSE includes noise on test labels; to instead compute a value for excexs risk, one would simply subtract o².
Here we study the asymptotic behavior of e, as n + oo for varying cigenspectra and ridge values. The fitting regime of the kernel is then given by this limit: (a) if lim,- en o² (the Bayes- optimal MSE), then fitting is benign, (b) if lim &n E (o),oo) then fitting is tempered, and (c) if lim,,+oo & oc, then fitting is catastrophic. We obtain conditions the kernel eigenspectrum under which KR falls into each of these three regimes.
Our proofs rely on several (quite weak) technical assumptions on {}i}; and {ui):. the most important of which is that the target function does not place weight in zero-eigenvalue modes (i.e. outside the kernel's RKHS). We defer enumeration and discussion of these conditions to Appendix A, where they are listed as Assumption A. Our result the following:
Theorem 3.1 (KR trichotomy). For (2:) and (v:)221 satisfying Assumption A. o² > O, and En given by Eq. 3,
(a) f 5 or > = -----log" for some 1, then lim En o2. n-000
(b) If 8- 0 and X; = a for some > 1, then lim En ao².
(c) If i = and A = i log or more generally if
A X
2
(c41--lontee) for all i, then
lexi
lim E,, = xc.
n-+00
We defer the proof to Appendix A. The proof proceeds by first showing that asymptotic MSE is dominated by the noise, not the true function, and then computing limn-+oc Ep in each of the three cases.
Theorem 3.1 can be summarized as follows: a ridge parameter extremely slow eigendecay leads to benign fitting, powerlaw decay of eigenvalues leads tempered overfitting, and eigenvalue decay at least fast as i log't leads to catastrophic overfitting, This theorem strongly suggests the satisfying
7

---

heuristic that decay slower than any a > 1 powerlaw is benign, powerlaw decay itself is tempered, and decay faster than any powerlaw is catastrophic". As a grows (1. oo), powerlaw spectra fully interpolate between benign and catastrophic, suggesting we have not missed any regime of interest to our taxonomy. The fact that the asymptotic MSE from a powerlaw spectrum is simply ao² is pleasant surprise.
Theorem 3.1 has several consequences for KR with familiar kernels and the training of infinite- width networks. For illustrative purposes, we contrast the Gaussian (RBF) kernel (a( = e- la =2 with the Laplace kernel K2 (x))
e-= --= 1z1- where is bandwidth parameter. With data drawn from a d-dimensional manifold, the Gaussian and Laplace kernels have eigenspectra that decay like A - (as can be seen by taking a Fourier transform of Kς) and X ~ i- [Bietti and Bach, 2020], respectively. We note that ReL. NTKs, restricted to the hypersphere, have the same eigendecay as the Laplace kernel [Chen and Xu, 2020, Geifman et al., 2020]. The implications of Theorem 3.1 include the following:
KR with fixed positive ridge parameter will fit any function in the kernel's RKHS benignly as + 20.
Ridgeless KR with Laplace kernels or ReLU NTKs will exhibit tempered overfitting with (1/d)) excess MSE, approaching benignness as dimension grows.
Ridgeless KR with the Gaussian kernel will exhibit catastrophic overfitting.
The asymptotic behavior of KR with non-ReLU NTKs depends on the activation function. Virtually any kernel on the d-sphere (including the Gaussian kernel) can be realized as the NTK of wide network with a proper choice of activation function [Simon et al., 2022], and thus there exist choices of activation function that yield catastrophic as well as tempered overfitting.
Early stopping known to act as an effective ridge parameter for wide networks [Ali et al., 2019], and thus we should expect that early-stopped wide networks will fit benignly.
We provide experiments illustrating Theorem 3.1 with Gaussian and Laplace kernels in Section 4. We additionally check Theorem 3.lb with several subsequent experiments; Figure 5 shows that, in synthetic KR with Gaussian eigenfunctions and exact powerlaw spectra, lim, En is aa And Figure in Appendix B shows that, as predicted, Laplace kernels indeed appear to have asymptotic risk that decays like (1/d).
We note that, to our knowledge, no well-known kernel has an eigendecay slower than all a > 1 powerlaws in finite dimension, and finding one in closed form yielding ridgeless KR which overfits benignly - is an interesting open problem.
Experiments
Having demonstrated the three types fitting theoretically in KR, we now present a series of experimental results illustrating these regimes in both KR and deep neural networks (DNNs). We provide full experimental details Appendix C.
4.1 Experiments Kernel Regression
In Figure 4, we run KR on the following synthetic data distribution: the inputs x are sampled from the unit sphere Sd-1, and the targets y are zero-mean Gaussian noise (y ~ (00 1)). This is an extremely simple regression setting: we are just trying to learn the constant-( function under Gaussian observation noise. We run KR with three choices kernel: (A) Gaussian kernel with ridge, (B) Laplace kernel without ridge, and (C) Gaussian kernel without ridge. Figure 4 shows that as increase the sample size, these three settings exhibit benign, tempered, and catastrophic behavior. This agrees with the spectral predictions of Theorem 3.
$ We leave complete proof this heuristic as an open problem.
Though not reported here, find the ridged Laplace kemel also exhibits benign fitting as expected.
8

---

Ridged Gaussian Kernel (Benign)
Ridgeless Laplacian Kernel (Tempered)
Ridgeless Gaussian Kernel
(Catastrophic)
1f
18
10a
Test MSE
10a
10-2
10-2
10
109
10
102
10
10
10
20
10a
10 10'
10
10a
101
Samples
Train Samples
Train Samples
10
d
Figure 4: Kernel regression can exhibit all three fitting regimes with proper choice of ridge parameter and kernel. Plots show learning curves for KR with data {xi) sampled uniformly from the unit sphere sd-1, trained with pure noise target labels y ~ N -N(00,) 1). Test MSE is computed with respect clean test set. (a) KR with Gaussian kernel and nonzero ridge is asymptotically benign. A ridge value of = 0.1 was used. (b) Ridgeless KR with a Laplace kernel exhibits tempered overfitting. (c) Ridgeless KR with Gaussian kernel exhibits catastrophic overfitting.
In Appendix B, we report KR experiments using synthetic kernels with exact powerlaw spectra trained on noisy data. We find that as the spectral decay varies, the (asymptotic) test MSE is indeed approximately ao² in agreement with part (b) of Theorem 3.1.
4.2 Experiments on Deep Neural Networks
Interpolating DNNs. Figure 2 and Figure 8a show the noise profiles for ResNets trained to interpolation on two-class and ten-class versions of CIFAR-10, respectively. In both settings, interpolating DNNs do not approach Bayes optimality, and instead exhibit tempered overfitting. This tempered behavior is widespread across even much simpler DNN settings. For example, in Figure 8b, we train three-layer interpolating MLP for binary classification on an extremely simple synthetic dataset: with inputs drawn from S and ground-truth labels as the constant function f*(xx) = 1. Even in this simple setting, at large sample size, interpolating DNNs are not benign-the do not successfully learn the constant function in the presence of even slight label noise. Note that although this experiment is outside the technical scope of Theorem 3.1, it heuristically consistent: wide ReLU MI.s tend to have NTKs with powerlaw spectra, and thus will exhibit tempered overfitting when trained in the NTK regime.
stopy- DNNs. We now consider DNNs that have been optimally carly-stopped. Figure 9 shows noise profiles for Wide ResNets trained on a binary version of SVHN, both stopped early (Figure 9a) and trained to interpolation (Figure 9b). The early-stopped ResNets approach benign fitting as n grows, with low emror even sizable noise levels, while the ResNets trained interpolation quickly converge to a tempered noise profile. This mirrors the behavior of MLPs on binary MNIST, after one epoch of training and after interpolation, shown earlier in Figure 3. Although these benign DNN results are outside the formal scope of known theoretical results, it is heuristically consistent with results such as Ji et al. [2021]. which show that certain wide and shallow ReLU MLPs are consistent when carly-stopped.
The above discussion suggests that single DNN is trained, it exhibits benign fitting early in training (when it has not fit the noise in its train set), and then transitions to tempered overfitting late in training (as it eventually fits the noise). In Appendix E, we show simple experiment in which an MLP trained on synthetic noisy data clearly exhibits this transition between regimes.
Limitations
In this paper, we have presented a taxonomy of overfitring, presented empirical evidence that DNNs trained to interpolation exhibit tempered overfitting, and identified spectral conditions under which

---

KR, toy model for DNNs, falls into each regime. This is a first study into this regime, and many questions await careful exploration. We detail several here.
First, our DNN results are entirely empirical, and there is room for complementary theoretical studies into the overfitting regimes of shallow networks and wide networks with NTK and mean-field parameterization. Second, while we have used several real and synthetic datasets, all are of at most moderate size, and a study at large scale using correspondingly large models - is potentially interesting. Third, we have found that in some realistic settings, interpolating DNNs are tempered, but it remains open whether there might exist settings or tasks for which interpolating DNNs overfit benignly or catastrophically. Fourth, Theorem 3.1 required a "universality" assumption which ought to be checked for each kernel. Finally, our KR results suggest that input or manifold dimension should play role in the degree of tempered overfitting, and the effect of dimensionality on overfitting yet to be disentangled.
Conclusion
In this paper we study the nature of overfitting in learning methods which interpolate their training data. Much of contemporary theory and experimental work categorizes overfitting as either catastrophic or benign. In contrast, we observe that many natural learning procedures, including DNNs used in practice, overfit in a manner that is neither benign nor catastrophic- but rather in an intermediate regime. We identify and formally define this regime, which we call tempered overfitting. We present empirical evidence of learning procedures that exhibit tempered overlitting, on both synthetic and natural data, using kernel machines and deep neural networks. We show tempered overfitting can be quantified in terms of noise profiles, which measure how asymptotic performance depends on noise in the train distribution. For kernel regression, provide theoretical result in the form of trichotomy: conditions on problem parameters which yield each of the three regimes of overfitting.
Our work presents an initial study of tempered overfitting and lays the framework for future study of what we believe is a rich and relevant regime for modern learning. We hope this framework inspires further investigation into tempered overfitting for more complex models, both theoretically and experimentally. For example, it is open to understand which conditions on neural network architecture, training hyperparameterss and data distribution lead to benign, tempered, or catastrophic overfitting, in analogy to our "kernel trichotomy", and an answer might shed light on practical DNNs.
Acknowledgments
The authors thank Nikhil Ghosh, Annabelle Carrell, Spencer Frei, Daniel Beaglehole, Gil Kur, and Russ Webb for useful discussion and feedback on the manuscript. JS additionally thanks two strangers on a DC Metro for useful discussion. PN especially thanks the Simons Institute, UCSD, and Apple MLR for collaborative environments which made this project possible.
We are grateful for support from the National Science Foundation (NSF) and the Simons Foundation for the Collaboration on the Theoretical Foundations of Deep Learning through awards DMS- 2031883 and #81 4639 as well as NSF 11S-1815697 and the TILOS institute (NSF CCF-2112663 NM is thankful to be funded and supported for this research by the Eric and Wendy Schmidt Center. JS gratefully acknowledges support from the NSF Graduate Fellow Research Program (NSF. GRFP) under grant DGE 1752814. 44 This work used the Extreme Science and Engineering Discovery Environment (XSEDE) [Toowns et al., 2014], which is supported by NSF grant number ACI-1548562, Expanse CPU/GPU compute nodes, and allocations TG-CIS210104 and TG-CIS220009.
Author Contributions
NM developed and performed the bulk of the experiments, drafted the initial version of the manuscript, actively contributed to writing and editing, and led the team logistically. JS proposed and proved the theoretical KR results, aided in experimentation, and contributed to the writing throughout the paper. AA developed and executed most of the kernel experiments, provided support to other experiments, and helped edit the manuscript. PP and MB provided guidance for experiments and theory, and reviewed and edited drafts of the manuscript. PN initiated this research agenda and provided active input throughout the experimentation. theory, and writing phases of the project.
10
a1/
10

---

References
Madhu S Advani and Andrew M Saxe. High-dimensional dynamics of generalization error in neural networks. arXiv preprint arXi::1710.0366; 2017. 3,
Alnur Ali, J. Zico Kolter, and Ryan J. Tibshirani. A continuous- view of early stopping for least squares regression. In Kamalika Chaudhuri and Masashi Sugiyama, editors, International Conference on Artificial Intelligence and Statistics, AISTATS 2019, volume 89 of Proceedings of Machine Learning Research, pages 1370-1378. PMLR, 2019.
Yasaman Bahri, Jonathan Kadmon, Jeffrey Pennington, Sam S Schoenholz, Jascha Dich-Dickstein, and Surya Ganguli. Statistical mechanics of deep learning. Annual Review of Condensed Malter Physics, 1 I1)1, 2020. 3
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jachoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprini arXiv:2102.06601, 2021,
Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 2020. 3, 1.2, 3, 3, A, B
Peter L Bartlett, Andrea Montanari, and Alexander Rakhlin. Deep learning: statistical viewpoint. arXiv preprint arXiv:2103.09177 2021. A
Mikhail Belkin, Daniel J. Hsu, and Partha Mitra. Overfitting or perfect fitting? risk bounds for classification and regression rules that interpolate. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi. and Roman Garnett, editors, Ad- vances in Neural Information Processing Systems 31: Annual Conference on Neural Infor- mation Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, neurips.co. wttpcccppppppp.eee /ttpppp///////aaa///ee
pages 2306-2317, 8a. URL ttttps://proceediii Abstract htmm. 3, 1.2, 2.4,
Mikhail Belkin, Siyuan Ma, and Soumik Mandal. To understand deep learning we need to understand kernel learning. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th Interna- tional Conference on Machine Learning. ICML 2018, Stockholmsmässann, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 540-548. PMLR, 2018b. URL http: http:///rocceenn...//o/ mlr prwps/ 1r800/elkii188. .ata.. 3
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-lcarninn
practice and the classical varinncr trade-off. Proceedings of the National Academy of Sciences, 116632):15849-15554 2019a.
Mikhail Belkin, Daniel Hsu, and Ji Xu. Two models of double descent for weak features. arXiv preprint 19Xiv:9903.077// 2019b. 3
Mikhail Belkin, Alexander Rakhlin, and Alexandre B. Tsybakov. Does data interpolation contradict statistical optimality? In Kamalika Chaudhuri and Masashi Sugiyama, editors, The 22nd Interna- tional Conference on Artificial Intelligence and Statistics, AISTATS 2019, 16-18 April 2019, Naha, Okinawa, Japan, volume 89 of Proceedings of Machine Learning Research, pages 161 1611-16 PMLR, 2019c. URL http: http:///ecccce..s..////m/ mlr presss/v99/bllkin1999 html. 1.2, 2.4
Mikhail Belkin, Alexander Rakhlin, and Alexandre B Tsybakov. Does data interpolation contradict statistical optimality? In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1611- PMLR, 2019d. 1.2
Alberto Bietti and Francis Bach. Deep equals shallow for relu networks in kernel regimes. arXiv preprint arXiv:2009. 14397, 2020. 3
Blake Bordelon, Abdulkadir Canatar, and Cengiz. Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In International Conference on Machine Learning, pages 1024-1034, PMLR, 2020. 3, A
Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Spectral bias and task-model align- ment explain generalization in kernel regression and infinitely wide neural networks. Nature communications, 12(1):1-12, 2021. A
11

---

Yuan Cao. Zixiang Chen, Mikhail Belkin, and Quanquan Gu. Benign overfitting in two-layer convolutional neural networks. arXiv preprint arXiv:2202.06556, 2022. 3
Niladri Chatterji and Philip M Long. Foolish crowds support benign overfitting. arXiv preprint arXiv:2/10.02114 2021. 3
Niladri S Chatterji, Philip M Long, and Peter L Bartlett. The interplay between implicit bias and benign overfitting two-layer linear networks. arXiv preprint arXiv:2 /489, 2021. 3
Kamalika Chaudhuri and Sanjoy Dasgupta. Rates of convergence for nearest neighbor classification.
Advances in Neural Information Processing Systems, 27, 2014. 2.4
Lin Chen and Sheng Xu. Deep neural tangent kernel and laplace kernel have the same rkhs. arXiv preprint arXiv:2009. 10683, 2020. 3
Andreas Christmann and Ingo Steinwart. Consistency and robustness of kernel-based regression in convex risk minimization. Bernoulli, (33)):79-8113 2007. 3
Thomas Cover and Peter Hart. Nearest neighbor pattern classification. IEEE transactions on information theory, 13(1):21-27, 1967. 2.4
Luke Nicholas Darlow, Elliot Crowley, Antreas Antoniou, and Amos J. Storkey. Cinic- 10 is not imagenet or cifar- 10. ArXiv, abs/
2018. C.I
Stéphane d' Ascoli, Levent Sagun, and Giulio Biroli. Triple descent and the two kinds of overfitting: Where & why do they appear? Advances in Neural Information Processing Systems, 33:3058-3069, 2020. 3
Luc Devroye, Laszlo Györfi, and Adam Krzyžak. The hilbert kernel regression estimate. Journal of Multivariate Analysis, 65(2):209-227, 1998. 1.2, 2.4
Spencer Frei, Niladri Chatterji, and Peter L Bartlett. Benign overfitting without linearity: Neural net- work classifiers trained by gradient descent for noisy linear data. arXiv preprint arXiv:2202.05922. 2022. 3, 1.2
Amnon Geifman, Abhay Yadav, Yoni Kasten, Meirav Galun, David Jacobs, and Basri Ronen. On the similarity between the laplace and neural tangent kemelss. Advances in Neural Information Processing Systems, 33:1451-1461 2020.
Sebastian Goldt, Madhu Advani, Andrew M Saxe, Florent Krzakala, and Lenka Zdeborová. Dynamies of stochastic gradient descent for two-layer neural networks in the teacher-studen setup. Advances in neural information processing systems, 32, 2019. 3
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation. arXiv preprint arXiv: 2019. 3, 1.2, 3,
David Holzmüller. On the universality of the double descent peak in ridgeless regression. In International Conference on Learning Representations, 2021. URL https: /ttp::///ppeeeeeeeeemmoemee net/ forum??d--115Vddnhhall
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018. 2.4, 3
Arthur Jacot, Berfin Şimşek, Francesco Spadaro, Clément Hongler, and Franck Gabriel. Kernel alignment risk estimator: risk prediction from training data. arXiv preprint arXiv:2006.09796 2020. 3,
Ziwei Ji, Justin Li, and Matus Telgarsky. Early-stopped neural networks are consistent. Advances in Neural Information Processing Systems, 34, 2021. 1.2, 2.4, 4.2
Frederic Koehler, Lijia Zhou, Danica J Sutherland, and Nathan Srebro. Uniform convergence of interpolators: Gaussian width, norm bounds and benign overfitting. Advances in Neural Information
Processing Systems, 34, 2021. 3, 1.2
12

---

Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009. C.1
Yann LeCun. The mnist database of handwritten digits. htt:://aam. lecun. con/exdb/mmist/, 1998. C.I
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http://yamm
lecun. wwt.coo///////ee /wxpb//misms....//mm/m C.1
Tengyuan Liang and Alexander Rakhlin. Just interpolate: Kernel" ridgeless" regression can generalize.
arXiv preprint arXiv: 1800 003877 2018. 3, 1.2
Tengyuan Liang and Alexander Rakhlin. Just interpolate: Kernel "ridgeless" regression can generalize.
The Annals of Statistics, 48(3):1329-133 1347, 2020. 3, 1.2
Siyuan Ma and Mikhail Belkin. Diving into the shallows: a computational perspective on large-scale shallow learning. Advances in neural information processing systems, 30, 2017. C.2
Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and double descent curve. arXiv preprint arXiv: 05355, 2019. 3, 1.2, 2.4
Vidya Muthukumar, Kailas Vodrahalli, Vignesh Subramanian, and Anant Sahai. Harmless interpola- tion of noisy data in regression. IEEE Journal on Selected Areas in Information Theory, 2020. 1, 3, 1.2, 3
Preetum Nakkiran and Yamini Bansal. Distributional generalization: A new kind of generalization. arXiv preprint urXiv:2009.08092 2020. 1.2
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 201 1. C.1
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. search of the real inductive bias: On the role of implicit regularization in deep learning. In ICLR (Workshop), 2015. 1
Adityanarayanae anan Radhakrishnan, Mikhail Belkin, and Caroline Uhler. Wide and deep neural networks achieve optimality for classification, 2022. URL https://arxiv wwwaaaiiivi.grr.ooooooo org/ ab2//204.2222.. ..44..411 14126.
Alexander Rakhlin and Xiyu Zhai. Consistency of interpolation with laplace kernels is high- dimensional phenomenon. Alina Beygelzimer and Daniel Hsu, editors. Conference on Learning Theory, COLT 2019, 25-28 June 2019, Phoenix, AZ, USA, volume 99 of Proceedings of Machine Learning Research, pages 1099/aaklinn html. 3, 1.2,
PMLR, 2019. URL http: /ostp:/ppun /ttp///procccc...ee/ee mlr, .irrprsss/
James B Simon, Madeline Dickens, Dhruva Karkada, and Michael R WeWeese. The eigenlearning framework: A conservation law perspective on kernel ridge regression and wide neural networks. arXiv preprint arXiv:2110.03922 2021. 3, 3, A,
James B. Simon, Sajant Anand, and Michael Robert DeWeese. Reverse engineering the neural tangent kernel. arXiv preprint arxiv:2/06.000 06.03/86.. 2022.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL /att:://arxe org/ aabb/ 1556. C.2
J. Towns, T. Cockerill, M. Dahan, I. Foster, K. Gaither, A. Grimshaw, V. Hazlewood, S. Lathrop, D. Lifka, G. D. Peterson, R. Roskies, J. Scott, and N. Wilkins-Diehr. Xsede: Accelerating scientific discovery. Computing in Science & Engineering, 16(05):62-74, sep 2014, ISSN 1558-366X. doi: 10.1 109/ 6, C.4
Alexander Tsigler and Peter Bartlett. Benign overfitting in ridge regression. arXiv preprint arXiv:2009. 14286, 2020. 3, 1.2
13

---

Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Richard C. Wilson, Edwin R. Hancock, and William P. Smith, editors, Proceedings of the British Machine Vision Conference 2016, BMVC 2016, York, UK, September 19-22, 2016. BMVA Press, 2016. URL wwtp:htt:///et bmva. wtt..oowggbb.oeeoooo/oeoo wttpp////ppppppppppeeeeepppp inndex////dxxxeee html. C.2
Lenka Zdeborová and Florent Krzakala. Statistical physics of inference: Thresholds and algorithms.
Advances in Physics, 65(5):453-555 552, 2016. 1.2
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL httph:tt///// happa:///pperree /opp/eeeeeeeeeeeeeeeeeeeeeeeeeee net ner/forum? Sy=Sy8dd9xxx. 1, 3, 4, C.2
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115. 2021. 3
14


Reviews:
[{"review_id": "qDpNFBq6C12", "reviewer": "Reviewer_xyxV", "summary": "As the title already suggests, the paper proposes an extended taxonomy of overfitting, adding an additional category of “tempered” overfitting, to the previously acknowledge benign and catastrophic forms.  Motivated by the increased study of overparametrized neural networks, various existing learning procedures are first categorized into the respective three categories from a regression and classification perspective. After application of the taxonomy, two specific cases are studied further, first kernel regression, in theory and practice, and then deep neural networks form an empirical standpoint. ", "questions": "If I understand correctly, the checklist is suggesting that the authors do not wish to publish their code (please correct me if I am wrong here). I wonder what the rationale behind this decision is and whether the authors would perhaps reconsider to also open-source in favor of reproducibility? Given that there are some assumptions at play, perhaps some that we discover only later, it would be intriguing to facilitate replication of the experiments.", "limitations": "Please see first weakness point, which targets limitations and already includes constructive suggestions for improvements.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n* The paper is exceptionally well written. Whereas there are many nicely written and structured papers, the presented manuscript clearly surpasses the average presentation. The introduction starts with a crystal clear motivation, an intuitive example, and the contributions are outlined appropriately. The related work section manages to not only give credit to a myriad of prior efforts,  but also to pinpoint the key differences and how the work builds on top. Although I am personally not familiar with various of the cited works and some of the presented topics, I never started to feel lost. In parts, this is further due to the fact that the writings strikes a balance between presenting theoretical statements and also following up with intuition and more verbose statements. Similarly, I appreciated that the text clearly emphasize when a statement has been shown/proven, versus e.g. when a conjecture is based on what the authors refer to as “experimental folklore”. \n*  Subjectively, I tend to dislike footnotes, based on the rationale that footnotes are either important enough to mention directly in the text, or omit them entirely. However, in this work, I actually found them quite helpful and appreciated the additional small notes at the bottom. \n* The taxonomy itself, examples of table 1, and the follow-up theoretical and empirical work have the prospect of facilitating future discussions about overfitting, overparametrization and the various ways in which ML algorithms learn. In some sense, the paper is almost by definition limited in scope from an experimental angle, given that the topic is relevant and is concerned with the study of a plethora of models & learning procedures. With this in mind, the empirical results are quite convincing, not only on the sense that the evidence experimentally supports the presented statements/theory, but also because it spans multiple datasets, as well as simple & more complicated methods. \n\n**Weaknesses** \n\n* My only key weakness, on the one hand because of my somewhat limited expertise in the theoretical aspects of the paper, on the other hand precisely because of it, is the lack of a discussion on limitations. Particularly, I wish there existed a concise separate limitation section, such that the perhaps more generally informed ML reader can directly get an overview of what to take for granted and where to still be particularly cautious. To emphasize, I believe the paper is to an extent already adequately presenting this information, e.g. by pointing out that the DNN parts are yet to be theoretically further explored/supported similarly to the simpler perspective of KRR. Having a dedicated place to consult would however be additionally helpful.\n* This point is perhaps less of a weakness and more of a suggestion for further improvement: After reading the text and looking at the plots in figure 5, it took me a bit to grasp whether the statement on tempered vs. benign overfitting in vs. early-stopped DNNs  is as clearly exhibited as suggested by the text. Intuitively it does become apparent as the right-hand side plot does not illustrate the dependency on “n”, as the left one does. However, in absolute terms it’s also somewhat tough to compare the two plots side by side. Perhaps a single larger plot with two different colors could more clearly show the discrepancy. \n\nOverall I believe there are many strengths to the paper and it will be very beneficial to disseminate the work to the NeurIPS community.  \n\n**Minor**\n\n* Figure 3 is missing a t in “catastrophic” in the title.  \n* The references are a bit inconsistent, some including URLs & some not, sometimes including conference dates or venues, in parts referring to proceedings vs. just the conference etc. An extra pass to make them more consistent, in whichever preferred way, would be nice. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "oKd9UPz8r98", "reviewer": "Reviewer_HmMk", "summary": "Summary.\n\nThe paper is dedicated to investigating the overfitting phenomenon of overparameterized neural networks. The authors argue that real interpolating methods like deep networks do not fit benignly, although benign overfitting has been widely instructive to study. They name the behavior as tempered overfitting and demonstrate it via kernel regression and deep neural network experiments.", "questions": "Refer to the weakness section.", "limitations": "The checklist indicates that the authors do not discuss the limitation and societal impact.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros.\n\n1. This paper is well-organized.\n\n2. The formulated problem of tempered overfitting is interesting. \n\n3. Both theoretical and empirical analyses are conducted.\n\nCons.\n\n1. Poor language \"We first kernel (ridge) regression, obtaining conditions on the ridge parameter and kernel eigenspectrum under which KR exhibits each of the three behaviors, thus obtaining a relatively complete picture of overfitting for kernel regression.\"\n\n2. The term \"catastrophically overfitting\" is ambiguous. It seems the authors mean the standard overfitting in deep learning. Since continual learning (https://www.cs.uic.edu/~liub/lifelong-learning/continual-learning.pdf) and fast adversarial training (https://arxiv.org/abs/2105.02942), there are other definitions of the \"catastrophically overfitting\".\n\n3. The tempered overfitting is a very general concept but only validated in limited cases. As we know, different networks have distinctive optimization behaviors. The paper's experiments are carried out only on ResNets on CIFAR and SVHN. More network backbones like VGG, MobleNet, and more datasets like ImageNet are needed.\n\n4. What are the corresponding descriptions or phenomena of piecewise linear interpolation in the multi-dimension cases?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Q9RbUafBVQf", "reviewer": "Reviewer_PuoQ", "summary": "This paper proposes a taxonomy of overfitting based on the generalization error when the sample size goes to infinity.\nSpecifically, the authors study the case of kernel regression and derive the asymptotic generalization gap in some cases.\nBased on these cases, the authors show that (a) kernel regression with regularization is benign; (b) Some commonly-used kernels such as ReLU NTK and Laplace KR are tempered; and (c) Gaussian KR is Catastrophic.\n\nOverall, this paper provides some interesting new ideas for the deep learning community and therefore has the potential to be accepted. \n", "questions": "See the above Flaws part. ", "limitations": "No potential negative societal impact.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Contributions.\n1. The authors provide a new taxonomy of overfitting based on how the asymptotic generalization error performs. \n2. The authors derive generalization errors for some cases of kernel regression and show which category these cases belong to.\n3. The authors show that ReLU-based NTK falls in the Tempered case. \n\nFlaws:\nIt is still unclear how this taxonomy helps in deep learning. Could the authors provide more information on what if we know this taxonomy in practice?\n\nTypos:\nLine2: be able *to*\nLine13: we first kernel regression?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept