Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the benign overfitting phenomenon of deep neural models, namely over-parameterized deep neural models fitting the training data can still achieve low generalization error. The authors focus on a specific regime where the model is mildly overparameterized and label noise is prevalent. They show that in such a regime, benign overfitting may fail. This explains the observation that ResNet model overfits benignly on CIFAR-10 but not on ImageNet, since ImageNet may be noisier. They also show that early stopping can help avoid overfitting in such a regime.
strength_and_weaknesses: Strength:
* This paper is well-organized and focused on an important problem in understanding the generalization of deep neural models. The claim is overall clear and the target setting is specific. Some observations made in this paper are also interesting such as deep models on ImageNet and CIFAR-10 have different overfitting phenomena.

Weakness:
* I believe the clarity of the paper needs to be improved. First, the definition of benign overfitting is rather ambiguous in this paper. I may not be an expert in this field, but based on my understanding, benign overfitting in existing works studies the generalization error as model size increases. However, in the introduction of this paper, benign overfitting is termed as 'validation performance does not drop while the model fits more training data points'. It appears to me that the authors try to study the dynamics of generalization error through model training (i.e. epoch-wisely). This is further corroborated by Figure 1 where the performance curves through training are shown. While I understand overparameterization through model size may bear similarities in training a model longer, it would be better if the authors can make it explicit. 

* In the experiment section, the above problem becomes more prominent. For example, in Figure 3 the authors show the performance curves with respect to epochs, while in Figure 4 the authors show the performance curves with respect to model size. 

* I might miss something in Theorem 3.1. But why does the lower bound to the generalization error in a noisy setting (Statement 2) show benign overfitting fails? I believe a lower bound to the generalization error should also exist in a noiseless setting. 

clarity,_quality,_novelty_and_reproducibility: Clarity:
* I believe in general the clarity of this paper needs to be improved to help the reader digest the result, as noted above.
* Some notations are not defined in this paper. For example,  $\mu$ in Section 3.1. The definition of notations is overall sloppy in this section. It is better to have a dedicated section presenting the notations.
* Some terms are not defined in this paper. For example, in Section 3.3, quote "The strange phenomenon happens because we distinguish the randomness in the training set from the randomness in the test set.". But what exactly is "randomness" here? I believe more discussion on the "difference of the randomness" is required here.

Quality and Novelty:
* I think the contribution of this work depends on whether the specific setting studied in this paper is novel compared to previous works. I am not that familiar with the related works. But I am a little bit worried that the mild-overparameterization regime may already be presented in previous works as a side result. I would be happy if the authors can correct me.













summary_of_the_review: The promises and observations made in this paper are interesting but it would be hard to judge the contribution right now given the clarity issue. I would be willing to increase my score if the authors can address my above concerns.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work studies benign overfitting with respect to the level of over-parameterization and whether under a noisy regime through implicit bias theory. This study is inspired by the phenomenon that the ResNet model overfits benignly on Cifar10 but not benignly on ImageNet. The authors identify a phase change: unlike in the previous heavy overparameterization settings, benign overfitting can now fail in the presence of label noise. The experimental simulation verifies the theoretical results.

strength_and_weaknesses: **Strength**

* This work studies benign overfitting in the mild over-parameterziation regime which is novel and interesting compared to previous works.

* The authors further study the noisy case and present the correspoding results. They find that mild-overparameterization is not enough for classification noisy problem. This result seems important to me.


**Weaknesses**

* This work directly use the result from implicit bias theory (Proposition 3.1). It is unclear what will happen when we study the gradient descent training like [1].

[1] Cao, Y., Chen, Z., Belkin, M., & Gu, Q. (2022). Benign Overfitting in Two-layer Convolutional Neural Networks. arXiv preprint arXiv:2202.06526.
clarity,_quality,_novelty_and_reproducibility: * The paper is very clear. Very organized and enjoyable to read.

* The paper is of high quality.

* The paper is quite novel to me.

* Question to authors:
   + It seems that the definition of benign overfitting in this work is different from [2], where they define benigh overfitting as  deep neural networks seem to predict well, even with a perfect fit to noisy training data. Can you provide a discussion on this? 

* Minors 
   + Page 3 missing space between assumptions and (instead of

[2] Bartlett, Peter L., et al. "Benign overfitting in linear regression." Proceedings of the National Academy of Sciences 117.48 (2020): 30063-30070
summary_of_the_review: The paper presented a relatively novel idea (to me) on benign overfitting. It presented a set of thorough theoretical analyses and experiments to validate the method. I tend to accept this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work analyzes the benign overfitting when the logistic regression is mild over-parameterization, in the setting of label noise. The core theory results are: 1) excess risk can be lower bounded at infinite SGD iteration; 2) the risk with early stopping can be upper bounded by 1/n (the number of samples. The authors also empirically verify the classification error under label noise and with different network widths.
strength_and_weaknesses: Strength:
1. Paper is well-written and easy to follow.
2. The core idea is demonstrated with both theory and experiments.

Weakness:
1. “fast convergence rate in Statement One and Statement Three”: forgive my ignorance, I only see the bound on the loss but did not see any results related to the convergence rate. For example, any bound on the difference between $L(t+1)$ and $L(t)$? Is it a linear convergence?
2. For separable data in the noisy setting, are you considering the flipped labels or their clean labels? I suppose one cannot have noisy data separable on both their flipped labels and their clean labels. If you consider the noisy data with flipped labels to be separable, is this conflict with the noiseless setting? I.e., you make separable noiseless data into noisy by flipping some labels, which will give you inseparable noisy data.
3. On page 16 case 2, “distance between $\mu$ and $\bar{\mathbf{x}}_\kappa^{\top}$ must be less than the distance from $\mu$ to its projection on the separating hyperplane”. Does that imply $|\mu^\top \mathbf{w}| \geq \| \bar{\mathbf{x}}_\kappa^{\top} − \mu \|$?
4. In proof of Thm 5.3, I suppose the bounds over $\max_{t \in [T]}$ (Eq. 10, 12, 15) may not be tight at the same $t$?
5. To complete the argument in this work, I suppose you should also show that with increasing parameterization, the overfitting becomes benign (e.g. the lower bound of risk becomes diminished)? Current benign overfitting papers do not have this conclusion in label noise classification setting.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly organized and well-written.
The core idea is also novel, to consider the label noise classification setting with mild parameterization.
I checked the proof and it generally holds, except for some issues I mentioned above.
summary_of_the_review: In general, I think this is a good paper with solid contributions to both theory and experiments.
I hope the authors could address my concerns above.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper analyzes benign overfitting when the number of parameters is not significantly larger than the number of data points. Under this mild overparameterization setup, the analysis identifies a phase that benign overfitting can fail in the presence of label noise.

Strengths:
Interesting experimental observations.

Weaknesses:
The technical clarity needs improvement. 

justification_for_why_not_higher_score: The reviewers are not very enthusiastic about this paper.
Decision: Accept: poster