Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper provided an insight into the “benign overfitting” of convolutional neural networks, by inspecting the last hidden layer representations of the networks. They observed and proposed an explanation of the phenomenon that the neurons tend to split into groups carrying duplicated information when the network is wide enough and the training error is zero.  
strengths_and_weaknesses: * Strengths

  - Originality: the paper brought a novel insight into an old problem of deep neural networks: why overfitting does not kill the generalization capability.  
  - Quality: the author(s) did comprehensive experiments and ablation studies on four datasets, pMNIST, CIFAR10, CIFAR100 and ImageNets with different variations of Wide-ResNet-28, ResNet50 and DenseNet. 
  - Clarity: The paper was well written with clear explanations of background, related works, methods, experimentations and limitations.
  - Significance: The problem that the author(s) were trying to attack is very important to both the understanding of the neural network training dynamics and the design principles of the wide and deep neural network architectures in practice. 

* Weaknesses:
  - Please refer to the Questions sections.
questions: 1. Limitations on architectures: the research is focused on convolutional neural networks. 
    - Does it generalize to Transformers or other networks with attention maps? 
    - How much does it depend on the existence of the convolution units, which have limited span? A related question: by changing the filter size of the convolutional layers from small to very large, will the following claim hold “... neurons tend to split into groups that carry identical information”?

2. In several places of the work, e.g. Figure 5, the author(s) mentioned that the training error reached zero, and they continued the training to get the claimed results. A naive question: if the training error stays zero, where does the gradient come from? 
    - Is the training error actually very small? If so, it is important to show the training curve or relevant statistics during the stage where the neurons are trying to duplicate themselves, in order to understand whether there is any phase transition or dynamical changes. 
    - If the training error is actually zero, the updates of parameters may solely be due to weight decay. If so, the weights are simply shrinking by a factor of (1-learning_rate * weight_decay_factor) per step after the training error reaches 0, and this effect alone introduces the phenomenon of duplicated neurons. Then this would be a very useful training recipe, which could be validated and applied on broader sets of architectures.

3. It is claimed that “clones appear if the training error is zero and the model is trained with weight decay”. I understand that weight decay is a way of regularization. But is there any explanation why standard regularizations, e.g. imposing L2 norm or Gaussian priors, don’t work? Or if other regularizations also work, how do they compare with using weight decay?

4. The main claim of “distinct clones provide independent measures of the same salient features of the data” seems to rely on the fact that the error decays approximately as W^{-1/2}, when the chunk width W goes beyond critical chunk size w_c. However, in Figure 2, there are only 2 data points per model beyond w_c=256; in Figure 3, the W^{-1/2} law seems not so well for CIFAR 10 and CIFAR 100; in Figure 8, it seems far away from W^{-1/2} for ImageNet. 
    - Based on these, apparently, the empirical claim of “the error of the chunks with w neurons decays as w^{-½}” is a weak claim, especially for more complex datasets CIFAR 100 and ImageNet. Are there any other experiments or insights to support the main claim?

5. Is there direct proof of the claim in line 152 “In the following we will indeed see that distinct clones differ from each other by uncorrelated random noise.” Would it be possible to verify this claim by creating statistics of the activations of neurons in the last layer?

6. Other minor issues:
    - When mentioning NTK in ln 290, I assume you are talking about neural tangent kernels. Please add proper references for that.
    - For any figures/tables in appendix, please refer then with “Appendix”, e.g. ln 245, Figure 8-c should be Appendix Figure 8-c  

limitations: Yes. Limitations are well stated and discussed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studied the statistical properties of randomly selected chunks of neurons in the last layer representations of wide deep neural networks. 

Specifically, the authors observed that when the chunk size is larger than a threshold, the test error is similar to that of a full network of the same width under a decay rate of $-1/2$. This implies that sufficiently large chunks can be viewed as statistically independent estimators of the same features of the data (referred to as "clone"), and the authors further characterize the various properties of "clones".
strengths_and_weaknesses: Strengths: 

1. The observation that sufficiently large chunks of neurons can be viewed as independent representations of the same data manifold is interesting. It may provide new insights for the connection between redundancy and generalization, model pruning and ensemble learning. 
2. The hypothesis is well-motivated by observations on multiple architectures and datasets. 
3. Extensive quantitative analysis has been conducted to verify the hypothesis and characterize the properties of "clones".

Weakness:

1. The study is only based on small scale datasets and networks. It seems that clones appear only when the model has sufficient capacity to interpolate the training data. As the properties of clones remain unknown in larger-scale settings, further implications and applications can be limited.

Minor: 

1. Missing y-axis in Figure 1.b. 


questions: Do the authors consider conducting their experiments on ImageNet with a model with higher redundancy, e.g., increase the width of the intermediate layers in addition to the final layer? It is interesting to see if clones exist and their properties.


limitations: No limitations.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies the representation of the last hidden layer of wide neural networks. The authors observe that these representations for networks that achieve zero training error contain redundant "clones". It is observed that by randomly dropping neurons of the last hidden layer, the performance is similar to training the network in the reduced size. This could serve as an explanation for the benign overfitting of over-parameterized neural networks.
strengths_and_weaknesses: Strengths:

1- The observation is novel and interesting. 

2- Some ablation studies including studying the intermediate layers have been reported.

Weaknesses:

1- A missing discussion from the paper is the connection between the presented observation and pruning neural networks. If it's the case that neurons are redundant, how can we select the redundant ones and drop them to have pruned networks? This could be a useful pruning strategy that is not covered in the paper. Overall, the importance of the outcome of the study is not clear and no application is shown.

2- The significance of the result is limited: The scope of the observation is limited to having zero training error and using weight decay. If the network is stopped early or there is no regularization the observations won't hold. Also, the last layer of the neural network should be very wide.

questions: 1- If redundant representations help generalization for wide enough neural networks, then we should observe that for the full network in Figure 1 (b), the error is lower than the chunk network. However, for large widths, we are observing the opposite. Then, either the explanation made in the introduction is not clear or the conclusion simply doesn't hold.

2- Figure 1: Why the reference for error is error_inf? Why not report the error itself? Also, what is the classification error computed on? 

3- In finding number 1, what do you mean by "w_c predicts the output with an accuracy that scales with w_c^(-0.5)"? Why is the slope of the figure of error versus net size referred to as "accuracy"? 

4- Why is w_c selected randomly? Why not select them from the correlation values to ensure the correlated neurons are not selected together and only the uncorrelated ones are selected together?  

5- The main motivation behind the work is its explainability of the bias-variance mystery. However, the exact connection has not been made. How would the bias-variance curve look in your experiments? 

6- Does the point regarding regularization only apply to weight decay or any form of regularization? If the latter, then the scope of the finding would be very limited. Also, what is the explanation behind this?

limitations: Yes, the authors mention the limitation of the observation in finding number 3.
review_title: null

Meta-review: metareview: This paper presented a perspective into explaining why overparameterized neural networks still generalize well. By studying the statistical properties of the last layer representations, the paper showed that neural networks learn redundant representations instead of overfitting to spurious correlations. This is an important topic in understanding deep learning. The reviewers agreed that the hypothesis is clearly formulated and the empirical analysis are convincing. 
confidence: Certain
Decision: Accept