Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This manuscript analyses the benign overfitting phenomenon for deep neural networks. In detail, the authors decompose the signal information and the noise during the training process and derive the corresponding convergence result according to the signal-to-noise-ratio (SNR). According to the theorems, with smaller SNR, the noise will be dropped and the harmful overfitting converts to benign overfitting.
strengths_and_weaknesses: Pros:

- This manuscript deeply investigates benign overfitting and clearly show the transition boundary between harmful overfitting and benign overfitting;

- The signal-noise decomposition is novel and can promote analysis in deep learning theory;

- The paper is well-writting and easy to follow.

Cons:

This paper comprehensively analyse the phase transition of benign overftting via SNR. However, there are two major points that can undermine the contributions of the paper: (1) Lack of ancillary experiments. The results are fully theoretical and rely on many assumptions (Condition 4.2) and Figure 1 is the schematic diagram according to theoretical results. If there are some empirical study based on real models and datasets, the conclusion will be more convincing; (2) Strong assumptions. The paper relys on many assumptions on dimiension, sample size, parameter size, etc. I recommend the authors to apply the assumtions to real empirical situation and show how the assumptions close to the real scenarios.

Some minor writing suggestions:
- Line 23: "the the classical" $\rightarrow$ "the classical";
- Line 128: "signal-to-noise ratio"should be followed by the abbr. SNR for better understandingl;
- Line 201: "that$n^{-1}$" $\rightarrow$ "that $n^{-1}$";
- Line 211: ",$n$SNR" $\rightarrow$ ", $n$SRN";

The reason for socre: Though the paper lacks empircal study, I appreciate for its solid theoretical analysis on the transition boundary between harmful and benigh overfitting. The transion boundary will be an important prespective to understand the success of neural nets.
questions: Please see Weaknesses.
limitations: Please see Weaknesses.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper studies the training dynamics and generalization behavior of GD on two-layer CNNs. In particular, it proves that when the signal-to noise ratio is large enough, GD can achieve small training and test loss; otherwise, the test loss may be large and bad overfitting occurs. Together, it shows a phase transition between benign and harmful overfitting.

strengths_and_weaknesses: The studied problem is important in understanding the generalization power of neural networks, and the result is relatively complete for the phase transition demonstrated. The technical analysis of the GD dynamics looks solid, and the decomposition approach may potentially be applied to other settings.

I have some questions in the setting of the studied problem. In Definition 3.1, the data point is composed of $x_1$ and $x_2$ representing signal and noise, separately. In this definition, the position of the signal is either $x_1$ or $x_2$, indicating that it is fixed in one of the two positions. Moreover, in the expression of CNN (Line 140), $x_1$ and $x_2$ seems to be separable, so the signal and noise seems to be separable in the setting. I am not quite sure whether this is reasonable in practice, as the position of signal usually varies (for example, the “signal” of an image may be in any corner of the image).

Another minor issue: In Section 3, the subscript of $x$ refer to both the category of signal or noise (with subscript 1, 2) and the index of data (with subscript 1, …, n). This may cause confusion.

questions: If the position of the signal is not fixed, or if the signal and noise is not separable in the CNN model, will the result keep the same?
limitations: Yes.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper characterizes when a convolutional neural network with a certain architecture and on a certain data model generalizes well while fitting the data perfectly. The analysis uses a signal-to-noise (SNR) ratio and shows that harmful overfitting can be avoided with a small sample size if SNR is large enough.
strengths_and_weaknesses: The paper attempts at an important problem and analyzes the performance of a neural network without approximating it with a convex linear model. I have not check the proofs in the appendix but the math in the main paper is correct and well explained. The two step approach in the analysis might have applications beyond the current problem. Nevertheless I am leaning towards rejection since this machinery relies on the following restrictive assumptions on both the data and the network:

1. The images have a noise patch and a signal patch. Each feature corresponds to either signal or noise through the dataset unlike a real dataset where the signal can be anywhere in the image. The label in the studied data model is a fixed vector $\mu$ for the positive class and $-\mu$ for the negative class and the noise patch is a multi-dimensional Gaussian r.v., while in a typical dataset signal and noise are entwined. Does this theory extend to a more realistic data model e.g. by replacing $\mu$ and $-\mu$ with the mean of feature vectors in the positive and negative class?

2. In the equation below line 140 it appears that the network's first layer processes the signal and noise separately, that is, it applies the filter on the first patch and then strides to the second patch and applies the filter on it. This large value of stride does not represent any common CNN architecture. A typical CNN would slide over the image in more fine-grained steps.

3. The second layers weights are initialized to $\pm 1$ and are assumed to be constant through the training. The hidden layer activation is polynomial ReLU with degree at least 2.

4. An important motivation for this work is that removing the large hidden layer width assumption in NTK. The analysis in the submission, however, still requires an overparametrization assumption (Condition 4.2). From the omega notation in the main paper it is not clear how restrictive this requirement is and how large the input and hidden layer width need to be for the results to hold.

Minor comments:
- In Line 305 the claim that small variations in loss across data points at the beginning of training matches the behavior of neural networks in practice needs citation. 
----------------------------------------------
Update: Thanks for the response. The clarifications in the rebuttal and the discussion with reviewer UZNG answered my first comment. The added experiments also show a consistency between the trend shown by theory and the one that occurs in a realistic setting without any of the unrealistic assumptions.

Regarding comment 4: My understanding is that large-width neural networks (at least some fully-connected ones) can be _well approximated_ with the NTK even if their width is finite. See theorem 3.2 in [1] for example. It is likely that under condition 4.2 one can directly analyze the NTK and then bound the difference between the behavior of a wide convolutional neural network and the NTK without the other assumptions.

Regarding asymptotic notation: The omega notation is completely fine. I was only admitting that, due to the omega notation, I do not have a clear idea if Condition 4.2 makes the behavior of the network close to NTK or not.

[1] Arora, Sanjeev, et al. "On exact computation with an infinitely wide neural net." NeurIPS 2019.
questions: See weaknesses section
limitations: See weaknesses section
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Paper studies the benign overfitting phenomenon in training a two-layer convolutional neural network (CNN).
Authors show that when the signal-to-noise ratio satisfies a certain condition, a two-layer CNN trained by gradient descent can achieve arbitrarily small training and test loss. Otherwise, overfitting becomes harmful and the obtained CNN can only achieve constant level test loss. 
Then paper shows a sharp phase transition between benign overfitting and harmful overfitting, driven by the signal-to-noise ratio.
strengths_and_weaknesses: Strengths
- intriguing property
- experiments convincing

Weaknesses
- theoretical analysis is very technical and hard to follow
- some practical example would improved the paper
questions: It would be nice to have some hint to a general property of benign overfitting.
How much do you think this result is general and holds for a wider class of function?
limitations: Some practical example would improved the quality of the paper.
review_title: null

Meta-review: metareview: This paper studies the benign overfitting phenomenon in training a two-layer convolutional neural network (CNN). Most importantly, this paper introduces SNR ratio in the analysis of benign overfitting. Under Condition 4.2, this paper provides the phase transition regime between "benign overfitting" and "harmful overfitting" in terms of SNR. A common concern from reviewers is the lack of connection to the practical settings. Still, all the reviewers agree to accept. I also think the theoretical result of this paper is interesting to the community, and I recommend accept.

confidence: Certain
Decision: Accept