Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper empirically studies the reason for the phenomenon that deep neural networks can memorize the data labels, even the labels are randomly generated. New geometric measures by replica mean-field theory are applied in the analysis. 

The findings of the paper are interesting. It shows the heterogeneity in layers and training stage of the neural net:

i) Memorization occurs in deeper layers; rewinding the final layer to the early weights mitigates memorization.

ii) When memorization happens, the early layer still learn representations that can generalize.

iii) In the training, early activations stabilize first, and deeper layers weights stabilize first. 

iv) Near initialization, the gradient is dominated by unpermuted examples.

I have the following questions/comments:

- It is better to further explain the intuition of the Manifold Geometry Metrics.  The current Figure 1(B) is not very clear.

- In Manifold Capacity, what do P and N exactly mean? Is this P the number of classes as used elsewhere?

- The paper explains that by training on permuted examples, the network can learn generalizable representations at the initial training stage because the gradient ignores permuted examples. But why in the later training stage, the early layers and later layers show different generalization properties?

In general, this paper carries well-organized experiments. One shortcoming is that the paper does not provide a methodology to solve the generalization problem or further theoretical analysis of the observations.  But the empirical discoveries are novel and can be beneficial to the deep learning community.  

###########

Updates: Thanks for the authors' response. The modified version improves clarity. I think this paper provides nice observations and initial analysis to the community and can be beneficial to future work, so I recommend this paper to be accepted.
review_title: Nice contribution in understanding generalization and memorization of deep neural networks

final_score: rating: 7: Good paper, accept
review_content: The authors apply MFTMA to DNNs trained on CIFAR with label noise to analyze their behaviors between generalization and memorization. Based on experimental results, they claim that what is involved in memorization are not lower layers but higher layers. This claim is convincing. Another claim that this is not caused by a vanishing gradient effect is plausible, too. I'm sure these results give some insights into understanding generalization and memorization by DNNs.

Questions. 
Why do the authors consider only convolutional layers, not fully-connected layers, for the analyses? In the experiment of rewinding individual layers, the three FC layers are left untouched. Why?

Is MFTMA the only method that can examine/verify the above finding?

Comments. 
At the first reading, I didn't understand what "restored examples" means, and it took me a while to understand it. The caption for Fig. A.7 has an error; CIFAR100 should be Tiny ImageNet. 
review_title: New results providing an insight to understanding of generalization and memorization by DNNs

final_score: rating: 7: Good paper, accept
review_content: ### 
Summary:
This paper investigates memorization in deep neural networks (DNNs).  Authors leverage mean field theoretic geometric analysis method (MFTMA) to analyze when and where memorization occurs in a DNN. Through empirical analysis, they show that i) generalizing feature are learned initially and that memorization happen later in training mostly in the top layers ii) we can mitigate memorization by rewinding the top layers parameters to earlier values. They also show that their MFTMA metrics can highlight the phenomena of double decent. Finally, they demonstrate that gradient descent initially ignores noisy example and focus on correctly labeled examples.

###
Reasons for score: 
I lean toward acceptance. This paper makes interesting observation regarding memorization of deep network, it performs a good empirical study which provide enough evidences for the different claims.  Although, MFTMA could be a better explained in the main paper. 
 
###
 Pros:
- As stated above, the paper makes interesting observation regarding memorization of deep network.
-  It performs a thorough empirical study.  

###
Cons: 
- I found it hard to understand MFTMA without referring to the appendix A. It would be nice to expand the explanation of MFTMA in the main paper. In addition, it would be good to further explain Fig 1. B which contains a lot of information. 
- Does the observation scale to larger dataset such as ImageNet ? 
- Experiments are run for only one seed.
 

 

review_title: Make interesting observations

final_score: rating: 7: Good paper, accept
review_content: This paper analyses memorization in DNNs, from the lens of memorization = fitting random labels, and finds that it seems to happen in later layers. These results are obtained using the MFTMA framework, a manifold analysis tool, testing geometric properties of individual layers. The analysis also attempts to explain why such a phenomenon exists, and makes a few interesting observations.
This paper does not propose any new algorithm, but instead settles some important questions by infirming or affirming past speculation on layer behaviour found in the literature.

I find three particularly interesting results in this paper:
- later layers seem to be responsible for memorization, while early layers seem to converge last but consistently learn "generalizing" features (although this may not be true for other architectures)
- increasing the dimensionality of the network to induce double descent _decreases_ the manifold dimensionality of the last layer. This is consistent with overparameterization making everything smoother/flatter and more easily disentangleable in the last layer.
- for examples with the wrong class, gradients initially vanish (due to destructive interference), which seems to be a driving force for the initial good generalization performance.

Downsides of the paper:
- The setting explored here is somewhat artificial, (1) the requirement on a high enough epsilon (random label proportion) may not represent real use of DNNs (I write this having seen Fig A.8; this is also a common criticism of double-descent results) (2) the models trained here don't seem to exceed 40% testing accuracy, again not necessarily representing real use of DNNs (this is a bit surprising considering even models from back in 2013 had above 60% accuracy on CIFAR100).
- Although the results of the paper do not hinge entirely on it, the reliance on MFTMA limits the interpretation somewhat: while an interesting tool, it's not clear to me that it allows us to make strong statements about the geometry of neural networks. In particular for the early layers, MFTMA may not be able to capture the geometry of features which might still be somewhat entangled yet possess a lot of richness.
- I have some issues with the presentation of the paper
- This paper does not really introduce a novel lens on generalization or significantly new ideas (although I'd argue it formalizes existing ideas and properly tests them empirically).

On the value of the contribution:
- I think having empirical evidence of the studied phenomena is valuable, more so than previous speculation on them.
- The empirical results presented here do open the door for new questions to be answered and may help focus the ongoing investigation of memorization and generalization in DNNs


Additional comments:
- Something seems wrong with Figure 2B-middle two columns. Aren't permuted and restored examples the same inputs X but with the corresponding Y changed? If this is the case, then their UMAP should be the same, the only difference between the second column and the third column should be the coloring of the points. I presume that the figure shows a different minibatch of Xs for these two columns; I would highly recommend not doing so and using the exact same inputs. It would be consistent with the text, and the presentation, e.g. Fig 1A.
- All Figures: the label fonts should be bigger. From the ICLR formatting guidelines: "use 10 point type [for text]", and "all artwork must be neat, clean, and legible." Having to zoom in and out to be able to read figures properly hurts accessibility and legibility, which detracts from the quality of the paper. Packing text, results, and figures in an 8-page document can be hard, but synthesizing information, including visual information contained in figures, is an essential skill in conveying knowledge.
-- Here are a few suggestions for this particular paper: Figure 1A seems unnecessary, the text conveys these 3 concepts clearly; Figure 1B is important and should take the entire width of the page, with legible fonts; Figure 2A's subplots all share the same X and Y axis, making their naming redundant and taking up space; Figure 2B's column labels are also repeated needlessly, taking up vertical space; Figure 3's X axis doesn't need individual layer name labels, and could be replaced with a single "Layer depth" label -- 3A and 3B also share this axis, leading to wasted vertical space (space that could be used to make fonts larger); idem for Figure 4A, individual layers do not need to be named, but rather the concept of layer depth can be conveyed with a properly labelled colorbar gradient -- 4CDE could be less wide and leave more horizontal space to make fonts larger.
- In Figure 5A, it's not immediately clear that the X axis are individual layers, the log(nabla) label should be on the colorbar rather than on top of the figure. I'd also suggest flipping the X and Y axis, as the X axis is typically used for time; this would allow there to have the three subplots side by side with a shared labelled colorbar on the right (matplotlib seems to be used here, see matplotlib.pyplot.subplots's sharex/sharey arguments for examples).

review_title: Review

Meta-review: The paper offers novel insights about memorization, the process by which deep neural networks are able to learn examples with incorrect labels. The core insight is that late layers are responsible for memorization. The paper presents a thorough examination of this claim from different angles. The experiments involving rewinding late layers are especially innovative.

The reviewers found the insights valuable and voted unanimously for accepting the paper. The sentiment is well summarized by R2: "The findings of the paper are interesting. It shows the heterogeneity in layers and training stage of the neural net".

I would like to bring to your attention the Coherent Gradients paper (see also R1 comment). This and other related papers already discusses the effect of label permutation on the gradient norm. Please make sure you discuss this related work. As a minor comment, please improve the resolution of all figures in the paper. 

In summary, it is my pleasure to recommend the acceptance of the paper. Thank you for submitting your work to ICLR, and please make sure you address all remarks of the reviewers in the camera-ready version. 
Decision: Accept (Poster)