Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a systematic empirical analysis of the memorization effect of DNNs w.r.t. model size and distillation. It bases the studies on the existing memorization measure 'memorisation score', and performs extensive experiments. The findings are 4-fold: (1) increasing the model complexity tends to make memorization more bi-modal where some hard examples get lower memorization scores while some ambigous examples get higher memorization scores; (2) distillation aids to inhibit memorisation; (3) some other memorization measures can strongly correlate with the used stability-based memorisation measure, but they do not capture key trends seen in the latter.
strength_and_weaknesses: # Strength
- The writing and presentation are nice. It is a pleasure to read this paper.
- The empirical studies are very thorough. I appreciate the authors' efforts in doing so.
- The paper is very relevant to ICLR community. The paper makes a step toward understanding the memorization and generalization of deep learning.
- The findings in this paper may be useful to spark new applications in scenarios such as data selection/reweighting when learning under noise and active learning.

# Weaknesses

Generally, in the aspects of displaying findings and drawing conclusions, the paper does not have significant issues. However, I would like to see improvements regarding the following points to make the paper a better contribution to the community.
- In the caption of figure 2, you state "however for architectures starting at depth 32, only a small number of points remain with
high memorisation score values". Is the used "small" correct? I see a relatively large number of points have high scores when the depth is no less than 32.
- "More interestingly, this bi-modality is exaggerated with model depth". However, as I notice, figure 2(b) cannot reflect this finding clearly. The exaggeration is very marginal. Can this be shown more confidently in some other scenarios? E.g., on mini-imagenet.
- On page 7, by inspecting figure 1(b), you state "Interestingly, none of the examples with small memorisation score from the one-hot model obtain a significant increase in memorisation from distillation." At first, it is hard for me to clearly understand figure 1(b). Are the plotted quantities the number of data? Besides, is the argument "none of" correct?
- Why does distillation have such impacts on memorization scores? Is the found phenomenon generalizable to other settings, architectures, or modalities? More explanations are desired.
- Do the different results between the stability-based score and C-score come from that C-score omits the first term in Eq 2? But as said, the first term can be ignored when the model capacity is enough. So does the conclusion in section 5 still hold when you used larger architectures like vision transformers? I want to see a clarification on this point. 
- Can you provide some extra experiments on adversarial examples? I want to know what kind of data will neural networks view the adversarial examples as？Ambigous or hard?




clarity,_quality,_novelty_and_reproducibility: The clarity is mostly good except for some minor issues. The quality, empirical novelty and reproducibility are good.
summary_of_the_review: This is a good paper with extensive empirical findings yet flaws in aspects of clarity and explanation. Currently, I am learning to accept this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work studies how and to what degree deep networks perform memorization of samples in the clean label setting. The authors build upon previous work and employ the stability-based memorization score developped in [1] which measures how much the prediction of a
model on a given sample changes if the sample is present versus not present in the training dataset. Using this measure, the authors
perform a in-depth study on how memorization is affected as a function of depth of the model as well as the effects of distillation. The 
authors find that larger depth generally decreases memorization on average, which is in-line with the fact that deeper models generalize
better. Interestingly, a more local look reveals a more fine-grained picture: There are samples that are less memorized as depth increases
but at the same time, there are examples that are memorized more. These examples often correspond to mislabeled samples. Moreover,
the authors find that distillation of a larger teacher into a smaller student reduces memorization on average.
strength_and_weaknesses: **Strengths**

1. This work provides a very broad empirical study regarding memorization in the clean label setting, which has not been performed to this degree in prior work. The results obtained in this paper might help in guiding theory to better understand the phenomenon.

2. It is very interesting that other, simpler memorization measures cannot capture the trends that the stability-inspired metric exhibits. The observed trends might serve as sanity checks and thus faciliate the design of new, computationally lighter memorization measures.

**Weaknesses**

1. While this work makes very interesting observations, a large part of the findings and the methodology have already been presented in prior works [1, 2]. The metric used in this work has been developped by [1] and initial empirical results have already been obtained on large
﻿networks such as ResNets in [2]. [2] already establishes that samples with the highest memorization scores are typically ambiguous or mislabeled, while this work builds on top of this and shows that this effect is exacerbated with depth. While some novel findings are made (i.e. that some samples are memorized less under higher capacity), large parts seem to re-confirm what has been observed in the literature.

2. A novel aspect in this work is the study of knowledge distillation, which was not considered in previous works. While certainly useful in practical settings, I am not convinced whether the understanding of memorization in this specific setting is of particular relevance. Given the fact that distilled models usually generalize better than standard ones, it does not seem so surprising that less memorization happens. However, it would be interesting to understand how memorization is reduced, but the authors  do not address this question. 

[1] Does learning require memorization? A short tale about a long tail. CoRR, abs/1906.05271, 2019.
[2] Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation
clarity,_quality,_novelty_and_reproducibility: **Quality**

The work is very well-written and easy to follow. Plots are of a high quality and simple to understand. The Appendix contains even more numerical evaluations, making this a very complete empirical study.

**Clarity**
I am listing a few points in the following which need further clarification:
1. The authors compare the stability-based metric against other, faster memorization estimators and conclude that those metrics are not precise enough to capture the trends found by the former. The authors implicitly assume that the stability-based metric is the superior estimator (other metrics should exhibit the same trends) without giving a complete reasoning as to why this is the case. A better explanation would be helpful for the reader to understand why it is not the case that the other metrics "might be right” instead.
2. In Figure 2.a), we can see the in-sample accuracy plotted as a function of model depth. I am a bit surprised that the ResNets models of depth 8, 14 and 20 cannot reach in-sample accuracy 1, which basically is the training accuracy right? Is this standard, especially since the training set is of smaller size here, if I understood correctly?

**Originality**

As already mentioned in the "Weaknesses" section, this work largely builds on top of [1,2] but scales the empirical experiments to a larger scale. Moreover, similar findings are obtained but now re-confirmed in a broader setting. Some aspects are new such as the distillation setting and the fact that some samples are memorized less with increasing capacity. 
summary_of_the_review: While some novel results are obtained in this work, it still feels more like a larger empirical study that mostly re-confirms what prior work has identified. I am not convinced of the relevance of the distillation experiments since no real insight into the workings of the method are obtained (i.e. why does a distilled network memorize less). The result that simpler memorization metrics cannot capture the same trends is interesting and certainly relevant to the community. Still, in total the novel contributions of this work are not enough in my opinion and I hence recommend the rejection of this work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper discusses memorisation of training examples in deep neural networks. 
* In particular, the paper shows how the memorisation profile of an architecture varies according to the depth of the networks.  The paper also shows what kind of examples are more or less memorised in deeper and shallower models.
* The paper discusses how this is impacted by distillation and carries out a similar study, as above, for distilled models.
* Finally, the paper studies how computationally efficiently computable metrics of memorisation vary from the exact but intractable one.
strength_and_weaknesses: ## Strengths
* Understanding the impact and diversity of memorisation in modern deep neural networks is an important topic for both understanding neural networks as well as for various notions of reliability of neural networks.
* The paper is easy to read and does a good job of balancing the mathematical notations and textual verbosity. 
* The authors ask specific questions and designs experiments to validate or negate their hypothesis. I find this approach of empirical validation or falsification quite well done.

## Weaknesses
While the paper provides an interesting empirical study on a fine-grained understanding of memorisation in deep neural networks,  I think the paper lacks in two ways (I illustrate them further in the summary below)
 * Several works have now linked memorisation to other properties like sparsity, robustness, and privacy. I think it would be good to see how the outcomes of this study relate to this. If the authors think a comprehensive study is outside the focus, I think, there should at least be a discussion with some preliminary results. This will not only inspire future works but also increase the relevance of this work.
* An empirical study where depth and architecture are pain axes of variation, the authors should discuss something more than ResNet and MobileNet. Even all the Mobilenet results have been relegated to the appendix. I think the main paper should include those but in addition, another relevant architecture type should be added.

clarity,_quality,_novelty_and_reproducibility: ## Clarity
The clarity of the paper is very good. The figures are usually clear but in some cases (pointed in the summary below) has confusing line styles and no legends.  

## Novelty and Originality. 

The paper builds of prior work, which is not a drawback but rather a plus. However, I think the paper can improve its novelty and originality by also discussing its impact on other topics like sparsity, privacy and robustness as discussed below.
summary_of_the_review: ## Main points

* The argument that _"While the study of memorisation in large neural models is fascinating, its practical relevance is stymied by a basic fact: such models are typically inadmissible for real-world settings with constraints on latency and memory. Instead, such models are typically compressed via distillation (Bucila et al., 2006; Hinton et al., 2015)"_ is rather weird and does not help with the motivation of the paper. Further, the citations provided are also not very relevant for models that are deployed in the real world today.


* One of the motivations of the model is the work of Feldman et. al. 2019 , whose main argument is that in certain distributions memorisation is necessary for generalisation. However, this paper also mentioned _"Consequently, a natural hypothesis is that memorisation of individual training samples might initially increase (as a weak model may have insufficient capacity to memorise), but should eventually saturate or reduce (as more expressive models generalise better)."_ is a contradiction of that argument and is thus, in some way, self contradicting.

* The observation that memorisation score decreases for larger models is an interesting observation. I find Figure 2(a) and 2(b) to be an excellent illustration of this. 
Does this possibly explain why adversarial robustness of deeper models is better ? For example, this is observed in Appendix B in Madry et. al. and Paleka et. al and the references therein argues that memorisation of (noisy) training data provably leads to higher adversarial error. I think this and other kinds of similar study on the impact of this observation would make a very nice addition to this paper.

* Figure 1(a) is also a good illustration of the fact that there are different kinds of examples that are memorised. Do the _increasing_ examples correspond to the human noisy labels as observed in Wei et. al ?  If that is the case, that deeper models do not need to memorise (or not as much) noisy labels, then it must also have impact on private learning as well as learning with compressed models with them ? For example, Sanyal et. al. show that private models are more inaccurate examples with higher memorisation score and similarly Hooker et. al. for compressed models but not talk about what kind of "high memorisation" scores they are inaccurate on whether _increasing_ or _decreasing_. I think this would be a great addition to this study. Similar questions also apply for adversarial training as observed in Xu et. al 2021.

* All the mathematical metrics here are expectations of random variables where the expectation is over the randomness of the training algorithm. I would like to see what is the empirical variance as well. The colour map, if I understand correctly, shows the variance across examples but not the variance due to the randomness in the algorithm.

* As this is an empirical study, I find it to be limiting to only discuss ResNet and MobileNet. It would be nice if the paper should also discuss pure convolutional architectures at least if not transformer architectures to make the study more comprehensive.

* That distillation inhibits overall memorisation and particularly those that display a higher memorisation score in non-distilled score is not very surprising but a nice thing to know. Perhaps a connection can be drawn to why distillation helps in adversarial robustness.

I find the paper interesting and would like to see the above changes made or at least argued why they should not be. Upon that, I am willing to increase my score.
 
## Minor points

* I am a bit confused my the line styles employed in Fig 3(a). Is there a particular reason for the way it is ?
* Figure 6 is missing the colour bar.
* More generally, I think the figures could be a bit smaller and more interesting results from the appendix could be brought to the main text. For example Figure 14

Madry et al. "Towards deep learning models resistant to adversarial attacks." (2017).  https://openreview.net/pdf?id=rJzIBfZAb

Paleka et. al. "A law of adversarial risk, interpolation, and label noise."  (2022).  https://arxiv.org/abs/2207.03933 

Wei et al. "Learning with noisy labels revisited: A study using real-world human annotations." (2021).  https://arxiv.org/pdf/2110.12088v2.pdf

Sanyal et. al. " How unfair is private learning?" (2022) https://proceedings.mlr.press/v180/sanyal22a.html.

Hooker et al. "What do compressed deep neural networks forget?." (2019). https://arxiv.org/pdf/1911.05248.pdf

Xu, Han, et al. "To be robust or to be fair: Towards fairness in adversarial training." (2021). https://proceedings.mlr.press/v139/xu21b.html

Papernot et al. "Distillation as a defense to adversarial perturbations against deep neural networks." (2016) . https://ieeexplore.ieee.org/document/7546524


================== post rebuttal ==================


The author's answers were satisfactory. The paper provides very useful insights and I lean towards acceptance at this stage.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Strength: (quoted sentences are reviewers', and selected by the AC. Others are AC's own opinion.)
--- “The writing and presentation are nice. It is a pleasure to read this paper.”
--- "The empirical studies are very thorough. I appreciate the authors' efforts in doing so."
--- "It is very interesting that other, simpler memorization measures cannot capture the trends that the stability-inspired metric exhibits. The observed trends might serve as sanity checks and thus facilitate the design of new, computationally lighter memorization measures."

Weakness. 

--- “While some novel findings are made (i.e., that some samples are memorized less under higher capacity), large parts seem to re-confirm what has been observed in the literature.”
--- "A novel aspect in this work is the study of knowledge distillation, which was not considered in previous works. While certainly useful in practical settings, I am not convinced whether the understanding of memorization in this specific setting is of particular relevance. "
--- Though The paper provides some form of understanding using empirical methods with some metric of memorization, there are two limitations: (1) there was no theoretical explanation for these empirical observations, (2) it's unclear what are the practical implications of these findings. The AC asks the authors the second question directly, but, unfortunately, didn't find the answers satisfactory. The authors largely seem to respond with many _potential_ applications, but there was no solid evidence that any of these implications can be eventually fleshed out. 


In general, the AC finds that the paper mostly conducted systematic experiments that do not require much novelty. Given a metric for measuring memorization, arguably, one could ask many questions of the type "does factor X affect memorization" and then throw many GPUs on them. One can do this even for multiple metrics of memorization. However, it might be more valuable to study the correct interpretation and impact of memorization metrics, e.g., what are the correct memorization metrics? Is memorization good or bad and should we avoid it in what situations? Can the amount of memorization in the system tell us some ways to improve the model？ 


justification_for_why_not_higher_score: Weakness. 

--- “While some novel findings are made (i.e., that some samples are memorized less under higher capacity), large parts seem to re-confirm what has been observed in the literature.”
--- "A novel aspect in this work is the study of knowledge distillation, which was not considered in previous works. While certainly useful in practical settings, I am not convinced whether the understanding of memorization in this specific setting is of particular relevance. "
--- Though The paper provides some form of understanding using empirical methods with some metric of memorization, there are two limitations: (1) there was no theoretical explanation for these empirical observations, (2) it's unclear what are the practical implications of these findings. The AC asks the authors the second question directly, but, unfortunately, didn't find the answers satisfactory. The authors largely seem to respond with many _potential_ applications, but there was no solid evidence that any of these implications can be eventually fleshed out. 


In general, the AC finds that the paper mostly conducted systematic experiments that do not require much novelty. Given a metric for measuring memorization, arguably, one could ask many questions of the type "does factor X affect memorization" and then throw many GPUs on them. One can do this even for multiple metrics of memorization. However, it might be more valuable to study the correct interpretation and impact of memorization metrics, e.g., what is the correct memorization metric? Is memorization good or bad and should we avoid it in what situations? Can the amount of memorization in the system tell us some ways to improve the model？ 


justification_for_why_not_lower_score: n/a
Decision: Reject