Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: As the title already suggests, the paper proposes an extended taxonomy of overfitting, adding an additional category of “tempered” overfitting, to the previously acknowledge benign and catastrophic forms.  Motivated by the increased study of overparametrized neural networks, various existing learning procedures are first categorized into the respective three categories from a regression and classification perspective. After application of the taxonomy, two specific cases are studied further, first kernel regression, in theory and practice, and then deep neural networks form an empirical standpoint. 
strengths_and_weaknesses: **Strengths**

* The paper is exceptionally well written. Whereas there are many nicely written and structured papers, the presented manuscript clearly surpasses the average presentation. The introduction starts with a crystal clear motivation, an intuitive example, and the contributions are outlined appropriately. The related work section manages to not only give credit to a myriad of prior efforts,  but also to pinpoint the key differences and how the work builds on top. Although I am personally not familiar with various of the cited works and some of the presented topics, I never started to feel lost. In parts, this is further due to the fact that the writings strikes a balance between presenting theoretical statements and also following up with intuition and more verbose statements. Similarly, I appreciated that the text clearly emphasize when a statement has been shown/proven, versus e.g. when a conjecture is based on what the authors refer to as “experimental folklore”. 
*  Subjectively, I tend to dislike footnotes, based on the rationale that footnotes are either important enough to mention directly in the text, or omit them entirely. However, in this work, I actually found them quite helpful and appreciated the additional small notes at the bottom. 
* The taxonomy itself, examples of table 1, and the follow-up theoretical and empirical work have the prospect of facilitating future discussions about overfitting, overparametrization and the various ways in which ML algorithms learn. In some sense, the paper is almost by definition limited in scope from an experimental angle, given that the topic is relevant and is concerned with the study of a plethora of models & learning procedures. With this in mind, the empirical results are quite convincing, not only on the sense that the evidence experimentally supports the presented statements/theory, but also because it spans multiple datasets, as well as simple & more complicated methods. 

**Weaknesses** 

* My only key weakness, on the one hand because of my somewhat limited expertise in the theoretical aspects of the paper, on the other hand precisely because of it, is the lack of a discussion on limitations. Particularly, I wish there existed a concise separate limitation section, such that the perhaps more generally informed ML reader can directly get an overview of what to take for granted and where to still be particularly cautious. To emphasize, I believe the paper is to an extent already adequately presenting this information, e.g. by pointing out that the DNN parts are yet to be theoretically further explored/supported similarly to the simpler perspective of KRR. Having a dedicated place to consult would however be additionally helpful.
* This point is perhaps less of a weakness and more of a suggestion for further improvement: After reading the text and looking at the plots in figure 5, it took me a bit to grasp whether the statement on tempered vs. benign overfitting in vs. early-stopped DNNs  is as clearly exhibited as suggested by the text. Intuitively it does become apparent as the right-hand side plot does not illustrate the dependency on “n”, as the left one does. However, in absolute terms it’s also somewhat tough to compare the two plots side by side. Perhaps a single larger plot with two different colors could more clearly show the discrepancy. 

Overall I believe there are many strengths to the paper and it will be very beneficial to disseminate the work to the NeurIPS community.  

**Minor**

* Figure 3 is missing a t in “catastrophic” in the title.  
* The references are a bit inconsistent, some including URLs & some not, sometimes including conference dates or venues, in parts referring to proceedings vs. just the conference etc. An extra pass to make them more consistent, in whichever preferred way, would be nice. 

questions: If I understand correctly, the checklist is suggesting that the authors do not wish to publish their code (please correct me if I am wrong here). I wonder what the rationale behind this decision is and whether the authors would perhaps reconsider to also open-source in favor of reproducibility? Given that there are some assumptions at play, perhaps some that we discover only later, it would be intriguing to facilitate replication of the experiments.
limitations: Please see first weakness point, which targets limitations and already includes constructive suggestions for improvements.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a taxonomy of overfitting based on the generalization error when the sample size goes to infinity.
Specifically, the authors study the case of kernel regression and derive the asymptotic generalization gap in some cases.
Based on these cases, the authors show that (a) kernel regression with regularization is benign; (b) Some commonly-used kernels such as ReLU NTK and Laplace KR are tempered; and (c) Gaussian KR is Catastrophic.

Overall, this paper provides some interesting new ideas for the deep learning community and therefore has the potential to be accepted. 

strengths_and_weaknesses: Contributions.
1. The authors provide a new taxonomy of overfitting based on how the asymptotic generalization error performs. 
2. The authors derive generalization errors for some cases of kernel regression and show which category these cases belong to.
3. The authors show that ReLU-based NTK falls in the Tempered case. 

Flaws:
It is still unclear how this taxonomy helps in deep learning. Could the authors provide more information on what if we know this taxonomy in practice?

Typos:
Line2: be able *to*
Line13: we first kernel regression?

questions: See the above Flaws part. 
limitations: No potential negative societal impact.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: Summary.

The paper is dedicated to investigating the overfitting phenomenon of overparameterized neural networks. The authors argue that real interpolating methods like deep networks do not fit benignly, although benign overfitting has been widely instructive to study. They name the behavior as tempered overfitting and demonstrate it via kernel regression and deep neural network experiments.
strengths_and_weaknesses: Pros.

1. This paper is well-organized.

2. The formulated problem of tempered overfitting is interesting. 

3. Both theoretical and empirical analyses are conducted.

Cons.

1. Poor language "We first kernel (ridge) regression, obtaining conditions on the ridge parameter and kernel eigenspectrum under which KR exhibits each of the three behaviors, thus obtaining a relatively complete picture of overfitting for kernel regression."

2. The term "catastrophically overfitting" is ambiguous. It seems the authors mean the standard overfitting in deep learning. Since continual learning (https://www.cs.uic.edu/~liub/lifelong-learning/continual-learning.pdf) and fast adversarial training (https://arxiv.org/abs/2105.02942), there are other definitions of the "catastrophically overfitting".

3. The tempered overfitting is a very general concept but only validated in limited cases. As we know, different networks have distinctive optimization behaviors. The paper's experiments are carried out only on ResNets on CIFAR and SVHN. More network backbones like VGG, MobleNet, and more datasets like ImageNet are needed.

4. What are the corresponding descriptions or phenomena of piecewise linear interpolation in the multi-dimension cases?
questions: Refer to the weakness section.
limitations: The checklist indicates that the authors do not discuss the limitation and societal impact.
review_title: null

Meta-review: metareview: This paper proposes a new taxonomy of overfitting that has the prospect of facilitating future discussions about overfitting, overparametrization and various mysteries of deep learning. The meta-reviewer recommends acceptance.
confidence: Less certain
Decision: Accept