What do large networks memorize?
Authors: Michal Lukasik
Aditya Krishna Menon
Ankit Singh Rawat
Vaishnavh Nagarajan
Sanjiv Kumar
Venue: Submitted to ICLR 2023 (2023)
Keywords: memorization
overparameterization
example difficulty

Abstract: The success of modern neural models has prompted renewed study of the connection between memorisation and generalisation: such models typically generalise well, despite being able to perfectly fit ("memorise") completely random labels.
To more carefully study this issue, Feldman (2019); Feldman & Zhang (2020) provided a simple metric to quantify the degree of memorisation of a specific training example, and empirically quantified the corresponding memorisation profile of a ResNet model on image classification benchmarks.
While an exciting first glimpse into how real-world models memorise, these studies leave open several questions about memorisation of practical networks.
In particular, how is memorisation affected by increasing model size, and by distilling a large model into a smaller one?
We present a systematic empirical analysis of these questions.
On standard image classification benchmarks, we find that training examples exhibit a diverse set of memorisation trajectories across model sizes, with some samples having increased memorisation under larger models.
Further, we find that distillation tends to inhibit memorisation of the student model, while also improving generalisation.
Finally, we show that computationally tractable measures of memorisation do not capture the properties we identify for memorisation in the sense of Feldman (2019), despite highly correlating to the latter. 

Under review as a conference paper at ICLR 2023
WHAT DO LARGE NETWORKS MEMORIZE?
Anonymous authors Paper under double-blind review
ABSTRACT
The success of modern neural models has prompted renewed study of the connec- tion between memorisation and generalixation: such models typically generalise well, despite being able to perfectly fit ("memorise") completely random labels. To more carefully study this issue, Feldman (2019); Feldman & Zhang (2020) provided a simple stability-based metric to quantify the degree of memorisation of specific training example, and empirically computed the corresponding memori- sation profile of a ResNet model on image classification benchmarks. While an exciting first glimpse into how real-world models memorise, these studies leave open several questions about memorisation of practical networks. In particular, how memorisation affected by increasing model size, and by distilling a large model into smaller one? We present an empirical analysis of these questions on image classification benchmarks. We find that training examples exhibit a diverse set of memorisation trajectories across model sizes, with some samples having increased memorisation under larger models. Further, we find that distillation tends to inhibit memorisation of the student model, while also improving generalisation. Finally, we show that other memorisation measures do not capture such properties, despite highly correlating to the babblitt-bbsedd metric of Feldman (2019).
INTRODUCTION
Statistical learning is conventionally thought to involve delicate balance between memorisation of training samples, and generalisation to test samples (Hastie et al., 2001). However, the success of modern overparameterised neural models challenges this view: such models have proven successful at generalisation, despite having the capacity to memorize, e.g., by perfectly fitting completely random labels (Zhang et al., 2017). Indeed, in practice, such models typically interpolate the training set, i.e., achieve zero misclassification error. This has prompted series of analyses aiming to understand why such models can generalise (Bartlett et al., 2017; Brutzkus et al., 2018; Belkin al., 2018; Neyshabur et al., 2019; Bartlett et al., 2020; Wang et al., 2021).
Recently, Feldman (2019) established that in some settings, memorisation may be necessary for generalisation. Here, "memorisation" of sample is defined via an intuitive stability-base notion, where the high memorisation examples are the ones that the model can correctly classify only if they are present in the training set (see Equation in $2). A salient feature of this definition is that it allows the level of memorisation' of a training sample to be estimated for practical neural models trained real-worle datasets. To that end, Feldman & Zhang (2020) studied the memorisation profile of a ResNet model on standard image classification benchmarks. While an exciting first glimpse into how real-world models memorise, this study leaves open several questions about the nature of memorisation as arising in practice. We are particularly interested in two questions:
Model size and memorisation. Increasing model size (e-g., depth of a ResNet) has a well- documented effect of (unsurprisingly) improving training accuracy, and (surprisingly) test accu- racy as well (Neyshabur et al., 2019). It is unclear what impact model size has on memorisation, however; while larger models have more memorisation capacity, do they make judicious use of this and memorise fewer, more informative samples than smaller models?
Distillation and memorisation. While the study of memorisation in large neural models is fasci- nating, its practical relevance is stymied by a basic fact: such models are typically inadmissible
From hereon in, unless otherwise noted, we shall use "memorisation" to refer to the stabilitt-basedd notion of Feldman (2019); Equation

---

Under review as a conference paper at ICLR 2023
Datlled mem score
Res110 Res20
Res110 Ros22
Dari mem
10
10
127
Memorication Score
0.8
0.6
Student 10
15
Student mamorisation
Res110
10t
Dix tie mram acore
Res110
Res56
104
0.0
Dittled 15
10?
18
Resker-a
Resteel Rediet:24
Restet- -2
Rester- 44
Redtlet
Res
20
15
Student memorisstion store
Student memorisation score
(a) Example of memorisation trajectories over depth. (b) Change in memorisation under distillation.
Figure I: Illustration of how memorisation (in the sense of Equation 1) evolves with ResNet model depth on CIFAR-I00 in one-hot training (left) and distillation (right) setups. In the left plot, we show that training examples exhibit a diverse set of memorisation trajectories across model depths: fixing attention on training examples belonging to the sunfl lower class, while many examples unsurprisingly have fixed or decreasing memorisation scores (green and red curves), there are also examples with increasing memorisation (blue curves). Typically, easy and unambiguously labelled examples follow fixed trend, noisy examples follow increasing trend, while hard and ambiguously labelled examples follow either an increasing or decreasing trend: in $3 we discuss their characteristies in more detail. In the right plot, show that distillation inhibits memorisation of the student: each plot shows the joint density of memorisation scores under a standard model, and one distilled from ResNet- 10 teacher. As the gap between teacher and student models becomes wider, samples memorised by the student see a sharp decrease in memorisation score under distillation (see the vertical bar at right end of plot). Notably, distillation does not affect other samples strongly.
for wealdworld settings with constraints on latency and memory. Instead, such models are typically compressed via distillation (Bucilá et al., 2006; Hinton et al., 2015). procedure that involves fitting a smaller "student" model to the predictions of "teacher". This raises a natural question: how much of the teacher model's 's memorisation is transferred to the siuden??
In this paper, we take a step towards comprehensively exploring the memorisation behavior of modern neural networks, and in the process contribute to progress on answering the aforementioned questions. Fixing our attention on the elegant stability-based notion of Feldman (2019), we study how memorisation varies as capacity of common model architectures (ResNet, MobileNet) increases, with and without distillation. We have four main findings: first, the distribution of memorisation scores becomes increasingly bi-modal with increased model size, with relatively more samples assigned either very low or very high score. Second, training examples exhibit a diverse set of memorisation majeccooiis across model sizes, and in particular, some examples are increasingly memorised by larger models. Third, while distillation largely preserves the student model's memorisation scores, tends to inhibit memorisation in the student. Finally, we analyze computationally tractable measures of memorisation which have been shown in previous work to highly correlate to the stability-based based notion of memorisation score, and show that surprisingly they lead to significantly different properties.
To summarise, our contributions are:
(i) we present a quantitative analysis of how the degree of memorisation of standard image classifiers varies with model complexity (e.g., depth or width of ResNet). Our main findings that increasing the model complexity tends to make memorisation more bi-modal; and that there are examples of varying memorisation score trajectories across model sizes, including those where memorisation increases with model complexity.
(ii) then present a quantitative analysis of how distillation influences memorisation, and show that it tends to inhibit memorisation, particularly of samples that the one-hot (i.e., non-distillation) student memorises.
(iii) we conclude with experiments using computationally tractable measures of memorisation and example difficulty, based on behaviour across training steps (Jiang et al., 2021a), and behaviour across model layers (Baldock et al., 2021). We find that although these measures strongly correlate with -abse memorisation, they do not capture key trends seen in the latter.
2

---

Under review as a conference paper at ICLR 2023
We remark that the aim of this work is not in proposing new methods, or memorisation measures, but rather to more systematically analyse the behaviour of existing memorisation measures.
2 BACKGROUND AND RELATED WORK
The term "memorisation" often invoked when discussing supervised learning problems, but with several distinct meanings (cf. Table 1 in Appendix). A classical usage of the term refers to models that simply construct a fixed lookup table, viz. a table mapping certain keys to targets (e.g., labels); predictions are made by simply identifying given row of this table, and outputting the associated target. This includes the classic k-nearest neighbour (k-NN) algorithm (Devroye et al., 1996), and a recent construct of Chatterjee (2018). Noting that the 1-nearest neighbour algorithm is capable of interpolating the training data (i.e.. perfectly predicting every training sample), some works instead use "memorisation" et al., 2021). A more general definition is that "memorisation" occurs when the training error is lower than the best achievable error (or Bayes-error) (Bubeck et al., 2020: Cheng et al., 2022).
synonym for interpolation (Zhang et al., 2017; Arpit al., 2017; Stephenson
While each of these notions imply "memorising" the entire training set, one may also ask whether a specific sample memorised. One intuitive formalisation of this notion is that the model predictions change significantly when a sample is removed from training (Feldman, 2019; Jiang et al., 2021b), akin to the notion of algorithmic stability (Bousquet & Elisseeff, 2002). More precisely, consider training sample S comprising N samples, drawn i.i.d. from some distribution D over labelled inputs (x, v) E X x y. A learning algorithm some randomised function S): - y, where the randomness is, e.g., owing to random initialisation, ordering of mini-batches, and stochasticity in paramcter updates. The memorisation score of a sample (z, y) E S is then (Feldman, 2019):
mem( S) = P(y = S)); P(y = A(x; S - {(x,, ())))
(1)
Here, P(-) considers the randomness the learning algorithm. Intuitively, this is the excess classifica- tion accuracy on the sample (æ, y) when it is included versus excluded in the training sample. Large neural models can typically drive the first term to for any (z, y) E S (i.e., they can interpolate any training sample); however, some (I, u) may be very hard to predict when they are not in the training sample. Such examples may be considered to be "memorised", as the model could not "generalise" to these examples based on the rest of the training data alone.
Despite its strengths, Equation I has an obvious drawback: it is prohibitive to compute in most practical settings. This is owing it naïvely requiring that we retrain our learner at least N times, with every training sample excluded once; accounting for randomness in A would require further repetitions. Feldman & Zhang (2020) provided more tractable estimator, wherein for fixed integer M. one draws K sub- samples ( ]е(]]] uniformly from P (MM()) the set of all M-sized subsets of For fixed (x. y). let Kn, = {ir E [K] (a, y) S(()), and Kout - {k [K] : (x. y) & S(A) denote the sub-samples including and excluding (z, y) respectively. We then compute
mem (I., S)
lv= A(x; S(K)]
Σ keKo
I== A ) (2)
Kjn im
Kout
K
keK...
This quantity estimates mem y; S) to precision Q(1/VK) Jiang et al. (2021a) considered a closely related consistency score (or C-score), defined as
cscore y; S) = EM [Es [P(y= = S(x; ((x,,(())))) v))))]
When M is drawn from a point-mass, this is the second term in mem (I, y; S). Note that the first term in mem,,k (I, y; S) is typically 1 for overparameterised models, since they are capable of interpolation. Jiang al. (2021a) proposed effective proxies for the C-score, which study the model behaviour across training steps. Specifically, suppose we have a model that is iteratively trained for steps E 21, T). where at t-th step, the model produces a probability distribution p((); X A() over the labels. Jiang et al. (2021a) established that the metric
cprox(æ, ys S) E, ( (x))
(3)
can correlate strongly with (the point mass version of) cscore(x, y: S). We defer to Appendix A a discussion of other related notions of example difficulty.
3

---

Under review as a conference paper at ICLR 2023
Prior empirical analyses of memorisation. Several works have studied the interpolation behaviour of neural models as one varies model complexity (Zhang et al., 2017; Neyshabur et al., 2019). Empirical studies of memorisation in the sense of fitting to random (noisy) labels conducted by Arpit et al. (2017); Gu & Tresp (2019). These works demonstrated that real-world networks tend to fit "easy" samples first, and exhibit qualitatively different learning trajectories when presented with clean versus noisy samples; this was used to argue that networks tend not to simply memorise real- world datasets. Zhang al. (2020) provided an elegant study of the interplay between memorisation and generalisation for a regression problem, involving learning either constant or identity function.
Feldman & Zhang (2020) studied memorisation in the sense of the stability- based memorisation score in Feldman (2019) (cf. Equation 1), by quantifying the influence of each training example on different test examples. Based on these, they identified a subset of test examples for which the model significantly relies on the memorized (in the stability sense) training examples to make correct predictions. While the direct inspiration for our study, these experiments were for a single architecture on CIFAR-100 and ImageNet, and did not consider the impact of model distillation. Zheng & Jiang (2022) confirmed the theory of Feldman (2019) holds in the context of NLP as well, by studying memorisation of BERT models using related notion of inftuence functions (Koh & Liang, 2017). Elangovan et al. (2021) quantified how much data leakage affects a BERT model, while Tänzer et al. (2021) showed that BERT models memorise random labels only in later stages of training.
3 HOW DOES MEMORISATION EVOLVE WITH MODEL SIZE?
We now seek to quantify (via Equation 1) how memorisation varies with neural model capacity.
Setup and scope. Quantifying the nature of "memorisation" requires picking suitable definition of the term. Owing to its conceptual simplicity and intuitive alignment with the term "memorisation", we employ the stability based memorisation score of Feldman (2019), per Equation 1. For computational tractability, we employ the approximation to this score from Feldman & Zhang (2020), per Equation 2. This reduces the computational burden of estimating mem(x, y; S), but does not eliminate it: for each setting of interest, we need to draw a number of independent data sub-samples, and train a fresh model on each. This necessitates a tradeoff between the breadth of results across settings, and the precision of the memorisation scores estimated for any individual result. We favour the former, and estimate mem (z, y; 5) via K = 400 draws of sub-samples S' (5)).. with M [0.7.].
With the above setup, we examine a simple question: how is memorisation influenced by model capacity? We aim to answer this empirically. Specifically, for range of standard image classification datasets - CIFAR-10, CIFAR-100, and Tiny-ImageNet- - we empirically quantify the memorisation score as we vary the capacity of standard neural models, based on the ResNet (He et al., 2016b;c) and MobileNet-v3 (Howard et al., 2019b) family (cf. Appendix B for precise settings).
How do we expect model size affect memorisation?. Before any empirical analysis, it is worth pausing to consider how one might intuitively expect model size to affect memorisation. Recall that image classification with neural models, with increasing model capacity we see improving generalisation the test set (cf. Table in Appendix). Consequently, a natural hypothesis is that memorisation of individual training samples might initially increase with model capacity (as a weak model may have insufficient capacity to memorise), but should eventually saturate reduce (as more expressive models generalise better). We also conjecture how we expect memorisation of an individual example to evolve model size increases. For most examples, we expect that memorisation should decrease or be constant, since the larger models generalise better and thus can model even complex samples. We now assess whether this intuition borne out empirically.
Larger models -> memorisation scores reduce on average. In Figure 2(a), we visualise how ResNet model depth influences the memorisation score (Equation 1) on CIFAR-100 Specifically, for each ResNet model, we report the average memorisation score across all training samples as a coarse summary. Consistent with our hypothesis above, this score increases up to depth 20, and then starts decreasing (albeit slightly). This phenomenon may be easily understood by breaking down the two terms used to compute the memorisation score in Equation 1: the in- in-sample accuracy, and the out-of sa-pa accuracy. As expected, both quantities steadily increase with model depth. The increasing memorisation score up to depth 20 can be explained by the former increasing faster than
4

---

Under review as a conference paper at ICLR 2023
Chaambl scorvoy lome hest
cample accurty Jone not
10
mamortait
Resket-14
20
Renkett1330
Rembiel-20
10-1
Renhwel-1100
Reslict-32 32
Reshiet-1
Rebiin12 150
Model Depch
Memor fration score interval
(a) Evolution of average score.
(b) Evolution of score distribution.
Figure 2: Memorisation scores from distilled and one-hot models on CIFAR- 100 training samples. We see from the left subfigure that the average memorisation score steadily increases with depth up till the ResNet-20, and steadily decreases afterwards. The increase up to certain depth can be explained by the in-sample accuracy increasing faster than out-of-sampii accuracy, until the point where the former is ~ 100%. The average memorisation score is reduced by distillation across model depths, which can be attributed both the reduced in-sample accuracy and increased out out-of-sample accuracy. The right subfigure depicts this phenomenon in finer granularity: as the model depth increases, relatively more samples are highly memorised (notice the hump in the graphs is being progressively pushed to the right); At the same time, notice how the average memorisation goes down with higher depths. We also observe that distillation reduces memorisation scores across model depth. Overall we find a growing bi-modality of the distribution of scores with increasing depth.
memoissiion sture buckel
20
Lehot
LeVet
Mosd5
Nobi0
Mob
Mob 100
Mobi225
Architecture
Architecture
(a) CIFAR- 100 LcNet-5.
(b) Tiny ImageNet MobileNet.
Figure 3: Distributions of memorisation scores across datasets and model architectures. We divide the range of memorisation scores into 10 equally-spaced memorisation score buckets. We consider LeNet-5 (LeCun al., 1998) models Imagenet of varying sizes. We achieve multiple architecture versions for LeNet-5 by varying the widths: wk denotes model where width is scaled times compared to the original LeNet-5. We find an increasingly bi-modal distribution as the width increases.
CIFAR-100 training set and MobileNet models Tiny
the latter up to this point; beyond this point, the in-sample accuracy saturates at 100%, while the out-of-sample accuracy keeps increasing. Thus, necessarily, the memorisation score starts to drop.
Larger models -+ memorisation scores become more bi-modal. The preceding result only con- sidered the average memorisation score for a given model. But how does the distribution of memo- risation scores vary with model capacity? To this end. Figure 2(b) plots precisely this distribution for each model in consideration. We observe that the memorisation scores tend to be bi-modal, with most samples' score being close to 0 or 1. This is in agreement with observations made in Feldman & Zhang (2020) for fixed ResNet architecture,
both CIFAR-100 and ImageNet.
More interestingly, this bi-modality is exaggerated with model depth: larger models have a higher fraction of samples with both memorisation score close to 0 and 1. The former is consistent with improved generalisation of larger models, and is thus unsurprising. On the other hand, the latter denotes increasing generalisation gap subset of training points, which does not harm the test set generalisation performance. Instead, as model capacity increases, we typically see monotonically increasing test set performance (cf. Table in the Appendix). In Figure 3 we present results for
5

---

Under review as a conference paper at ICLR 2023
LeNet-5 on CIFAR- 100 and MobileNet on Tiny Imagenet, and demonstrate that our observations hold across architecture families and datasets.
(a) CIFAR-100: contrasting ResNet architectures.
(b) TinylmageNet: contrasting ResNet architectures.
Figure 4: Contrasting per-example memorisation scores across architectures in different setups. As the difference in depths becomes larger, the highly memorised examples according to small model become less memorised by the large model (note the high density in the lower-right part of the plot). On the other hand, smaller models tend to have examples which get assigned high memorisation score by a large model (note the horizontal bar at the top of a figure).
Larger models some samples are memorised more. Given the previous observation of the increasing bi-modality with model depth, we now study how exactly the memorisation score each example shifts across model depths. In Figure 4, we show how memorisation scores evolve with depth on a per-example basis. An unsurprising observation is that memorisation scores in most cases do not significantly change across depths. Beyond this, becomes larger, the highly memorised examples according to a small model become less memorised by the large model (note the high density in the lower-right part of the plot). Most interestingly, in all cases, there is non-trivial fraction of samples whose memorisation score increases with model size; note the horizontal bar at the top of Figure 4, and the bump in the right side of the marginals depicted in Figure 2(b). Increasing memorisation implies a decreasing out-of-sample accuracy on such points (as the in in-sample accuracy monotonically increases with model depth). This is perhaps surprising, since one expects increasing model depth to improve generalisation (Neyshabur et al., 2019).
that the difference in depths
To sum up, we observed that there are examples with both increasing and decreasing memorisation with model depth, as denoted by the out-of-diagonal high density regions in Figure 4. Next, inspect the specific examples exhibiting such varying trajectories.
Examining samples memorised more/less by larger models. Next, investigate examples exhibiting varying memorisation trajectories over model size. We are particularly interested in studying examples which increase and decrease in memorisation. Recall that we conjectured different trajectories of memorisation depending on the example: for the examples which are easy across model depths, we expect low, stable memorisation; for the mislabeled examples high, increasing memorisation; and for the hard but unambiguous examples, an increasing trajectory of memorisation.
In Figure ((a), we report the average memorisation score as function of depth for examples from the sunfl I ower class exhibiting varying trajectory patterns: increasing, decreasing and constant. Also, in Table (Appendix) we report predictions for the shown examples. We first notice that the least- changing memorisation examples are easy and unambigous. Further, amongst these examples, the peak memorisation score tends to be low (i.e.. such samples tend to consistently not be memorised); indeed, when measuring the change in memorisation by the difference from ResNet- 10 and ResNet- 20 models, among the least changing 1% examples the average memorisation (as measured on ResNet- 1 10) is 0.06, and among least changing 10% examples is only 0.33.
Next, we note that both the increasing and decreasing examples are hard and ambiguous. The decreasing examples arguably hard but labeled correctly. On the other hand, the increasing

---

Under review as a conference paper at ICLR 2023
examples are often multi-labeled (e.g., the first increasing example containing a bee) or mislabeled (e.g. the third increasing example), such that a human rater could be unlikely to label these images with the dataset provided label. This parallels the predictions from ResNet- 110 for these examples, which arguably are no less reasonable than the label as presented in the dataset.
This can be intuitively explained by breaking down the Feldman memorisation score into in-sample and out-of-sample accuracy components. For the examples which are easy across model depths, we find a low and not changing memorisation as the in-sample accuracy should be low and the out-of-sample accuracy should be low across depths. Next, for the mislabeled examples in the data, we find high or even increasing memorisation, as with increasing depth the correct label, which disagrees with the noisy label in the data, is being recovered better by the deeper models, and so the out-of-sample accuracy increases or remains high. Finally, for the hard but unambiguous examples, we find an increasing trajectory of memorisation, as the deeper models gradually get better at generalisating to that point, which corresponds to the sfa-pll accuracy for that example increasing, and in turn, the memorisation score decreasing.
To conclude, we indeed find examples aligned with our hypothesis, i.e., the easy and unambigous images contributing to the decreasing examples. However, we also find ambigous and mislabeled images yielding the increasing examples. More surprisingly, we also find ambiguous points among the decreasing examples. We discuss this in more detail in Appendix D.I, where we inspect more examples and further elaborate on our observations.
Summary. We have found that, contrary to our initial hypothesis, the empirical behaviour of memorisation under model size is nuanced: there are some samples which are memorised less by larger models (as expected), but there are also those for which the opposite holds. These reflect the underlying "difficulty" of the example and the sample's label. As we will see in $5, some of the other existing notions of sample difficulty which have been shown to correlate with memorisation score do not capture its properties we discussed. Most notably, we will show how these scores exhibit unimodal score distributions, while the above memorisation score exhibits bi-modal distribution.
4 DOES DISTILLATION TRANSFER MEMORISATION?
Large neural models have shown impressive predictive performance on challenging tasks in vision and NLP. However, in practical settings, is often infeasible to deploy such models. It is instead more common to compress these into more tractable models, with one popular strategy being distillation (Bucilã al., 2006; Hinton et al., 2015). Here, one feeds a large ("teacher") model's predicted probability distribution over labels as the prediction targets for small ("student") model. Compared to training the latter from scratch, distillation can provide significant performance gains; these are informally attributed to distillation performing "knowledge transfer". Knowledge distillation has been succesfuly applied across many applications, including: computer vision (Beyer et al., 202 I), language modeling (Sanh et al., 2019), information retrieval (Lin et al., 2021), machine translation (Zhou et al., 2020), and ads recommendation (Anil et al., 2022; Liu et al., 2022).
While distillation has been shown to yield significant performance gains average, train accuracy has been shown to be systematically harmed (Cho & Hariharan, 2019). Previous work also showed subset of hard examples (Lukasik et al., 2021).
how distillation can lead to worsened accuracy In a related study, model compression has been shown to harm accuracy on tail classes (Hooker et al., 2019). Given these observations, it is natural to ask what effect such distillation has memorisation of the resulting distilled models. To assess the impact of distillation on memorisation, we consider knowledge distillation as conducted using logit matching, and where the teacher is trained on the same sub sub-sample as the student for estimating the memorisation scores per Equation 2. We provide the hyperparameter details in Appendix B.
Distillation inhibits memorisation. We begin by investigating what happens to average memori- sation under distillation. Distillation is known to reduce the train accuracy compared to the one-hot models, while increasing the test accuracy (Cho & Hariharan, 2019); in Table 4 (Appendix), we report the train and test accuracies. From this, could expect distillation to inhibit memorisation. In Figure 2, we illustrate the difference in distributions of memorisation scores across models trained on the ground truth labels (which we call one-hot training) and the models distilled from ResNet-1 10
7

---

Under review as a conference paper at ICLR 2023
teacher model. As expected, we find that distillation tends reduce the number of memorised samples. We also plot in Figure 2 the decomposition of memorisation into in-sample and sample accuracies for the one-hot and distilled models. We find that across all model depths, the in sample accuracy becomes lower and out-of-sample accuracy becomes higher under distillation. This parallels the observation that train accuracy lowers and test accuracy increases under distillation.
In Figure 14 (Appendix), we report how memorisation changes when varying the teacher size. We find that across different student and teacher models, memorisation is inhibited. Interestingly, as the student depth varies, the memorisation pattern changes significantly, while with varying the teacher, there no significant change in how memorisation changes between the distilled and one-hot models.
Distillation inhibits memorisation of highly memorised examples. We next turn to analysing the distribution of per-example change in memorisation under distillation. In Figure 1(b), we report the joint density of memorisation scores under a standard model, and one distilled from a ResNet- 110 teacher. We can see that distillation inhibits memorisation particularly for the examples highly memorised by the one-hot model, and especially when the teacher-student gap is wide. Interestingly, none of the examples with small memorisation score from the one-hot model obtain a significant increase in memorisation from distillation.
In Figure 12 (Appendix). we plot trajectories of memorisation over architecture depths for examples depicted in Figure 1. We find that memorisation is overall lowered especially for the challenging and ambigous examples, which often get high memorisation score value by either the small or large models. Notice how this is consistent with the finding from Figure 1(b) of the reduction of memorisation for particularly highly memorised examples. At the same time, none of the low memorisation examples changes their trajectory. This is denoted by the area in the upper left corner of each plotted matrix, which corresponds to count O.
Do OTHER MEMORISATION SCORES BEHAVE SIMILARLY?
The preceding analysis has sought to shed light on the memorisation patterns of large neural models, building on the stability-based notion of memorisation in Equation 1. Any attempt to extend this analysis to larger datasets and models faces a fundamental stumbling block: even with the approximation of Equation 2, estimating this score imposes a non-trivial computational burden. is thus tempting to consider the viability of other, computationally lighter notions of memorisation. To this end, we now consider two such metries- the C-score proxy and prediction depth - that may also be (informally) tied to memorisation. Interestingly, we find that such metrics do not exhibit the same diversity of memorisation behaviour as the stability-based notion. Thus, quantifying memorisation of large models either requires paying a computational price, or devising new metrics.
The proxy scores. The C-score proxy cprox (Equation 3) was proposed in Jiang et al. (2021a) as a computationally efficient alternative to the C-score, a metric closely related to the stability-based memorisation score of Equation| Indeed, Jiang et al. (2021a) found this measure to have high correlation with the C-score, while cautioning that it should not be interpreted as an approximation to more fine-gras -reingrrind characteristics of the latter. The C-score also correlates with the prediction depth (Baldock et al., 2021), which computes model predictions at intermediate layers, and reports the earliest layer beyond which all predictions are consistent. These predictions were made using a k-NN classifier in Baldock et al. (2021); in Appendix F, we explain why using linear classifier is also admissible, and may even be preferable. As demonstrated in Baldock ct al. (2021), prediction depth offers an appealing balance between being well correlated with other metrics (c.g., it forms lower bound on the stability-bawe memorisation score), and being computationally efficient (it only requires training single model, with re-runs only to estimate standard deviations).
Correlation to the memorisation score. In Figure 19 (Appendix), we provide heatmaps of the joint density of these measures against stability- memorisation. We see that cprox has a visually strong relationship to the latter, and indeed the two have a high degree of correlation: e.g., for the ResNet-56 model, the correlation coefficient is 0.91. We also find correlation of prediction depth to memorisation score, albeit to a smaller extent. In particular, the correlation coefficient is 0.72 for the ResNet-56 model, and there visually regions of small density outside of the diagonal.
8

---

Under review as a conference paper at ICLR 2023
104
10"
% of sampls
101
% %afaamkss
10-4
ResNet-56
20*
ResWer
ResMet-32
Re
10-2
ResNet-20
Recien-20
Resten-33
ResNet-8
Reclen-14
Reclen-2 4
0.65..10
of P
cprox
prediction depth
interval
(a) C-score proxy (cprox).
(b) Prediction depth.
Figure 5: Marginal distribution of cprox (left) and prediction depth (right) over the CIFAR-100 training set across ResNet architectures. The marginals are unimodal, unlike the bi-modal distributions for stability- based memorisation scores (see Figure 2(b)).
Unimodal marginal distribution. Going beyond correlation, however, we find in Figure 5 that the distributions of cprox and prediction depth scores have markedly different characteristics to stability- bbsee memorisation: the former are unimodal, with most samples having a high score value (for both cprox and prediction depth). Recall that by contrast, stability- -aaled memorisation scores exhibit bi-modal distribution (Figure 2(b)), with this phenomenon exaggerated with increasing model depth. Consequently, we do not observe many of the unexpected trends exhibited by stability- based memorisation, e.g., the possibility of cprox or prediction depth systematically decreasing for some samples as depth increases. Indeed, we find that scores do not significantly change across architectures, as can be seen in Figure 18 (Appendix). This is in stark contrast to the stability-basee memorisation score, which exhibits large score changes per example when considering architectures of large depth gap. In line with Anscombe (1973), our finding illustrates that while the correlation between two distributions may be high, the nature of the distributions themselves can be significantly distinct. Thus, we conelude it is important to be cautious about conclusions drawn from a high correlation between various memorization scores and their proxies, as they may in reality be capturing different properties of data.
DISCUSSION AND FUTURE WORK
We have seen how increasing model capacity and knowledge distillation both tend to lower mem- orisation average. Intriguingly, distinct line of work has studied the impact of capacity and distillation on adversarial robustness: Madry et al. (2018) shows how model robustness increases with capacity, while Papernot et al. (2016) showed how distillation can lead to improved adversarial robustness. It is thus natural to ask how adversarial robustness relates to memorisation. As a first step, we note that intuitively, adversarially robust models ought to produce decision regions which can better generalise to even challenging examples excluded from the train set. We demonstrate this intuition on a toy example in Section G in the Appendix, where a smaller model less robust and at the same time yields higher memorisation score for a challenging example.
Sanyal al. (2022); Paleka & Sanyal (2022) show how interpolating noisy training examples can lead to a higher adversarial error. In our analysis, we found noisy examples to have an increasing trajectory of memorisation, thus hinting that models of increasing capacity may be increasingly more robust against fitting to such noisy labels. It will be of interest for future work to systematically analyse noisy training examples through the lens of stability-based memorisation.
As our analysis focussed on image classification settings, a natural question is whether one observes similar trends in NLP tasks. Building on the work of Carlini et al. (2022), which computed stability- based memorisation for single T5 model, it would be of interest to assess how model size and distillation behave in this context. Another interesting direction would be the study of techniques to improve distillation by weighting samples according to their memorisation score. Finally, contrasting the memorisation behaviour of overparameterised models to sparsely-action activated mixture of expert models (Shazeer et al., 2017) would also be of interest.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Rohan Anil, Sandra Gadanho, Da Huang, Nijith Jacob, Zhuoshu Li, Dong Lin, Todd Phillips, Cristina Pop. Kevin Regan, Gil I. Shamir, Rakesh Shivanna, and Qigi Yan. On the factory floor: MI engineering for industrial-scale ads recommendation models, 2022. URL https://arxiv... abs/2209.05344
Anscombe. Graphs in statistical analysis. The American Statistician, 27(1): 1973. doi:
10. 00.10000000033005.000 1970:0311111..11.1077.....0. 10478966. URL https://wwe www.:ttttaaoooiinnnennemm Ltnttttss//ww...mnfoonnn.e.e..o wttp./////o/ooooo.o//oooo//o/oot 00031305. 191339589977
Devansh Arpit, Stanistaw Jastrzebski. Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste- Julien. A closer look at memorization in deep networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, 17, pp. 233-242. JMLR.org, 2017.
Robert John Nicholas Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. In NeurIPS, 2021. URL https://arxiv..rrg 2226/ 09647.
Peter Bartlett, Yoav Freund, Wee Sun Lee, and Robert E. Schapire. Boosting the margin: new explanation for the effectiveness of voting methods. The Annals of Statistics, 26(5):1651 - 1686, 1998. doi: 10. 12112///////// wot.oo/////249992222222 URL https://doi. oot./org///o 10. 1224/200//02469913333
Peter L. Bartlett, Dylan J. Foster, and Matus Telgarsky. Spectrally-normalize margin bounds for neural networks. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017 December 4-9, 2017, Long Beach, CA, USA, pp. 6240-6249 2017.
Peter L. Bartlett, Philip M. Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117(48):30033-30777 2020. doi: (0000///aas.9003379809 17. URL https:
pnw.. wwtppp..gg/////aaa///oooooo//ppp
Mikhail Belkin, Daniel Hsu, and Partha P. Mitra. Overfitting or perfect fitting? risk bounds for classification and regression rules that interpolate. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, pp. 2306-2317, Red Hook, NY, USA, 2018. Curran Associates Inc.
Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distillation: A good teacher is patient and consistent, 2021.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmmann, Trevor Cai, Eliza Rutherford, Katic Mil- lican, George van den Driessche, Jean- -aptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. Improv- ing language models by retrieving from trillions of tokens. CoRR, abs/21 12.04426, 2021. URL https://arxiv.orr org abs / 2212. 04422.
Olivier Bousquet and André Elisseeff. Stability and generalization. Journal of Machine Learning Research, 2:499-526, 2002.
Gavin Brown, Mark Bun, Vitaly Feldman, Adam Smith, and Kunal Talwar. When is memorization of irrelevant training data necessary for high-accuracy learning? In Proceedings of the 53rd
Annual ACM SIGACT Symposium on Theory of Computing. STOC 2021, pp. 133-132, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450380539, doi: 24114//3406325.355111 URL httppp://dii.rrgeeeeeo
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya
10

---

Under review as a conference paper at ICLR 2023
Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, abs/2005.14165. 2020. URL https://arxiv, http://wwwwwgg///////////o 14165.
Alon Brutzkus, Amir Globerson, Eran Malach, and Shai Shalev-Shwartz. SGD learns over- parameterized networks that provably generalize on linearly separable data. In International Confer- ence on Learning Representationss 2018. URL
33wwb..
Sébastien Bubeck, Ronen Eldan, Yin Tat Lee, and Dan Mikulincer. Network size and weights size for memorization with two-layers neural networks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Cristian Bucilä, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD '06, pp- 535-541, New York, NY, USA, 2006. ACM.
Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. In 30th USENIX Security Sympo- sium (USENIX Security 21), pp. 2633-2650. USENIX Association, August 2021. 1. ISBN 978-1- www.ww.seeix.com 939133-24-3. URL http://ww....aae/eeee/eeeeeeeeer Carlini-extracton
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models, 2022. URL https://arxiv. org/aba http/bbs/222..76666
Satrajit Chatterjee. Learning and memorization. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Pro- ceedings of Machine Learning Research, pp. 755-763. PMLR, 10-15 Jul 2018. URL https: //proceedings mlr. wwarees///e/caattteeeeeeee
html.
Chen Cheng, John Duchi, and Rohith Kuditipudi. Memorize to generalize: on the necessity of interpolation in high dimensional linear regression, 2022. URL htttp::/aariv..org/bb////// 09889.
J. H. Cho and B. Hariharan. On the efficacy of knowledge distillation. In 2019 IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), pp. 3793-4801, 2019.
Jeremy M Cohen, Elan Rosenfeld, and J. Zico Kolter. Certified adversarial robustness via randomized smoothing. 2019. URL /tttss:/arxxv..ott wwwp/org/g////////o///// 02918.
L. Devroye, L. Gyorfi, and G. Lugosi. A probabilistic theory of pattern recognition. Springer, 1996.
Gintare Karolina Dziugaite and Daniel M. Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. In Proceedings of the 33rd Annual Conference on Uncertainty Artificial Intelligence (UAI), 2017.
Aparna Elangovan, Jiayuan He, and Karin Verspoor. Memorization vs. generalization Quantifying data leakage in NLP performance evaluation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 1325-1335, Online, April 2021. Association for Computational Linguistics. doi: 12265///1/0221eaccoooooo 113. URL
abs/1906.05271, 2019, URL htt.o//ooo.....o...oo..aaamaaaaa
eacl-main.
Vitaly Feldman. Does learning require memorization? A short tale about long tail. CoRR, 05271.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 2881- 1891. Curran Associates, Inc., 2020. URL fi http://abbeeeiaaeaaaaaaaaaeeeeee htw.secbbddbbuu-aae...p.. paf.r.
waarocedinngs.nuuiiss..om

---

Under review as a conference paper at ICLR 2023
Margalit Glasgow, Colin Wei, Mary Wootters, and Tengyu Ma. Max-margin works while large margin fails: Generalization without uniform convergence, 2022. URL https::/aaxxxv..rrgg///////// 2206.0788 07892.
Jindong Gu and Volker Tresp. Neural network memorization dissection. CoRR, abs/1911.09533) 2019. URL http://arxiv.on org/ htbs/1911.0999
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming- Weig-WWii Chang. Realm: Retrieval- augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org, 2020.
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning. Springer Series in Statistics. Springer New York Inc., New York, NY, USA, 2001.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016a.
doi: 10. 1 1019//VPR.2006....
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016, pp- 630-645, Cham, 2016c. Springer International Publishing. ISBN 978-3-319- 46493-0.
Geoffrey E. Hinton. Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. CoRR, abs/ 500.05533,, 2015.
Sara Hooker, Aaron Courville, Gregory Clark, Yann Dauphin, and Andrea Frome. What do com- pressed deep neural networks forget?, 2019. URL https://arxxv.... http://.orr//b///////ee/e 0524...28......
Andrew Howard, Ruoming Pang, Hartwig Adam, Quoc V. Le, Mark Sandler, Bo Chen, Weijun Wang. Blang-Chieh Chen, Mingxing Tan, Grace Chu, Vijay Vasudevan, and Yukun Zhu. Searching for mobilenerv3. In 2019 JEEF/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27- November 2, 2019, pp. 1314-1324. IEEE, 2019a. doi: 10.1109/ICCV. 2019.00140. URL https:: /ttp:://ooi.ooo/ oro/ 10.gg///... 1109/ 1109/ICCV 201CV...10.01.0...
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoe V. Le, and Hartwig Adam. Searching for Mo., In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), October 2019b.
Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C. Mozer. Characterizing structural regularities of labeled data overparameterized models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 5034-5044 PMLR, 2021a. URL httpp:/pprceeeingg mlr wtw.iinng.........s..eeeeeeeeeee pttesss/vv///j/// jaa.a///.iannikkkkaa htmml.
Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C. Mozer. Characterizing structural regularities of labeled data in overparameterizee models, 2021b.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representationss 2020. URL htttttss:///eeeeeiee..eeeeeeeeee
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' 17, pp. 1895-1894. JML,.org, 2017.
12

---

Under review as a conference paper at ICLR 2023
Guillaume Lample, Alexandre Sablayrolles. Marc' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Large memory layers with product keys. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer. Florence d' d'AccééBBu,, Emily B. Fox, and Roman Garnett (eds.), Ad- vances in Neural Information Processing Systems 32: Annual Conference on Neural In- formation Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 8546-8557, 2019. URL https: http://rceeinnnneurip... httpp/ppp..ccpppppe///////////// www.fff-bsttattt..oomm html.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-baul based learning applied to document recognition. Proceedings of the IEEE, 86( (6(11))2278-2324 1998.
Tengyuan Liang and Alexander Rakhlin. Just interpolate: Kernel "Ridgeless" regression can gen- eralize. The Annals of Statistics, 48(3):1329 1347, 2020. doi: 10.1214/19-ASS1849 URL
https://doi. wttp./..././///........ 1214/49-A051849
Sheng-Chich Lin, Jheng-Hong Yang, and Jimmy Lin. In-batch negatives for knowledge dis- tillation with tightly-coupled teachers for dense retrieval. In Proceedings of the 6th Work- shop on Representation Learning for NLP (RepL4NLP-2021), pp. 163-173, Online, August 2021, Association for Computational Linguistics. doi: 10. 10.188/311/2221 wwpp://///2 reppl4nlp 1.17. URL
https://aclanthhology wrw.y//.r./e2oooe/. http://222..rellllllpp...mmm www.-eepplpp....
Congcong Liu, Yuejiang Li, Jian Zhu, Xiwei Zhao, Changping Peng, Zhangang Lin, and Jingping Shao. Rethinking position bias modeling with knowledge distillation for ctr prediction, 2022. URL
https://arxiv. wwwooogg/bbbee /arg/ab///a /a//aabss////// 220./2224.2222 00270.
Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, and Sanjiv Kumar. Teacher's pet: understanding and mitigating biases in distillation. CoRR, abs/2106. 104944 2021. URL https:
//wpxiv.ogg/abs/////o 10494.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL id=rOZIBfZAb.
Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sehastian Farquhar, and Yarin Gal. Prioritized training on points that are learnable, worth learning, and not yet learnt. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pp. (6630-1564,, PMLR, 17-23 Jul 2022. URL mlr wttress//ve2/mmnneemaaaa
html.
Andrea Montanari and Yiqiao Zhong. The interpolation phase transition in neural networks: Memo- rization and generalization under lazy training, 2020. URL https://arxinn argg/bbs//00// 2as.////....226....... 12826.
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. The role of over-parametrization in generalization of neural networks. 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Daniel Paleka and Amartya Sanyal. law of adversarial risk, interpolation, and label noise, 2022.
URL https://arxin: http://aaxxv..rraab.././//ss arg/abs//200 0339333337.......
Rina Panigrahy, Xin Wang, and Manzil Zaheer. Sketch based memory for neural networks. In Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings of The 24th International Conference on Artificial Intelligence and Statisticcs, volume 130 of Proceedings of Machine Learning Research, pp. 3169- 3177. PMLR, 13-15 Apr 2021. html.
URL https:// http:://poooeeeig...t.es wtt.ss.ss.preesvoo/////oomo ptgg///gg......ss/.///////gg....
Nicolas Papernot, Patrick D. McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as defense to adversarial perturbations against deep neural networks. In IEEE Symposium on Security and Privacy, SP 2016, San Jose, CA, USA, May 22-26, 2016, pp. 582-597, 2016.
13

---

Under review as a conference paper at ICLR 2023
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning data diet: Finding important examples early in training. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang. and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 20596-20607. Curran Associates, Inc., 2021. URL httpp:://ppptt httpss://poceedngss/ee www.nps.c./npppeeeeeeeee ccppaprr/ Papees---Ppprppee P
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2463-2473, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653/1/DD19-1550
URL https://ac http://aaaaatthoooooooooo logy. wtg/ 1250.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-TH Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual. 2020. URL https:
http:igg.sn.urris..ccppprer//e/e hash//erssaepcogg211aaaaaaddeeee http:///ww.booeeeabaatttttttaaa. html.
Adiyyanarayanan Radhakrishnan, Mikhail Belkin, and Caroline Uhler. Overparameterizee neural networks implement associative memory. Proceedings of the National Academy of Sciences, 117 (44):27162-27170, 2020. doi: 1..1777/paas22000111111 URL https:h//ww.wt... org/doi/ abs//to.1073/pas..20000
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140): 2020. URL
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? CoRR, abs/2002.08910,, 2020. URL. http:::/s///:ii..o
2802. 08910.
Sagawa, A. Raghunathan, P. Koh, and Liang. An investigation of why overparameterization
exacerbates spurious correlations. In International Conference Machine Learning (ICML), 2020.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01100. 2019. URL http://arxiv. org/abs/1910. 01108.
Amartya Sanyal, Yaxi Hu, and Fanny Yang. How unfair is private learning? In James Cussens and Kun Zhang (eds.), Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence, volume 180 of Proceedings of Machine Learning Research, pp. 1748.174.. PMLR, html.
01-05 Aug 2022. URL https: https://proceedingg meriiinnssmlll... .ll..rr.....
Noam Shazeer, * Azalia Mirhoseini, & "Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outragcously large neural networks: The sparsely-gater mixture-of-expert layer. In International Conference Learning Representations, 2017. URL. https://openreview..rt forum?ide wiimmidddBiccMDLL..
Cory Stephenson, suchismita padhy, Abhinav Ganesh, Yue Hui, Hanlin Tang, and SueYeon Chung. On the geometry of generalization and memorization in deep neural networks. In International Conference Learning Representationss 2021. URL https::/opeerrviee..ett//ooo v8jrrnwGbuc.
Charles J. Stone. Consistent Nonparametric Regression. The Annals of Statistics, 5(4):595 620, 1977. doi: 10. 12/40//// (ataaos//117333330 176343886. URL http::://doi.rrgge/ wttp.////......//////////.....1.
Michael Tänzer, Sebastian Ruder, and Marek Rei. BERT memorisation and pitfalls low low-resource scenarios. CoRR, abs/2105.0m 00022105.00828. 2021. URL https::/arxivv.or//bb////////// 00828. 00828.
14

---

Under review as a conference paper at ICLR 2023
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study of example forgetting during deep neural network learning. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA. USA, May 6-9, 2019. OpenReview.net, 2019.
Vladimir Vapnik and Rauf Izmailov. Reinforced SVM method and aetemmrrzzationncccciiinin Puttern Recognition, 119:108018, 2021. ISSN 0031-3203, doi: https///bbiaagr//l00000.ipoooopp 108018. URL
Ke Wang, Vidya Muthukumar, and Christos Thrampoulidis. Benign overfitting in multiclass classifi- cation: All roads lead interpolation, 2021. URL https::/arxxvv.rr/aab////// 10865.
Zhen Wang and Yuan-Hai Shao. Generallzation-neeoozzzooo machines, 2022. URL https://
arxiv. www.orgg//baa.. httgg/////aee 220.//////////22//////222//22/ 03976.
Quanming Yao, Hansi Yang, Bo Han, Gang Niu, and James Yau Kwok. Searching to ex- ploit memorization effect in learning with noisy labels. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning. volume 119 of Proceedings of Machine Learning Research, pp. 10789-10798, PMLR, 13-18 Jul 2020. URL https://procceedings mlr wtt.:mppeeee//.///aoaaaaaaeee/ee
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Michael C. Mozer, and Yoram Singer. Identity crisis: Memorization and generalization under extreme overparameterrzzatin. In International Conference on Learning Representations, 2020. URL https://openrreviewweee wwwp://wwwwweeeeeeeueueee.e.eeee Bi
Xiaosen Zheng and Jing Jiang. An empirical study memorization NLP. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 6265-6278, 6278, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10.11865//112022.aal-oooo 434. URL https: htttp::/aalaatthol.gg// wwwp://woooooogooooo.eoomm acl- .434.
Chunting Zhou, Jiatao Gu, and Graham Neubig. Understanding knowledge distillation in non- autoregressive machine translation. In International Conference on Learning Representations, 2020. URL httpss//ppennev..e..tteffoooooos wwtp:///eeeeeooorum.//ggggaaaaa FVVEK..E.
Hattie Zhou, Ankit Vani, Hugo Larochelle, and Aaron Courville, Fortuitous forgetting connectionist networks. In International Conference Learning Representations, 2022. URL https:// openreview. wttt:/orrrrmiieeiisss1..........
15

---

Under review as a conference paper at ICLR 2023
A ADDITIONAL RELATED WORK
Relation to example difficulty. Equation 3 provides an interesting bridge between memorisation and example difficulty. For example, TracIn (Pruthi et al., 2020) and GRAD (Paul et al., 2021) also use the evolution of model predictions across training steps to identify samples that are difficult to lear;; roughly, these are samples which cause large loss updates when they are trained on. Another related measure of sample difficulty is the RHO-loss (Mindermann et al., 2022), wherein the training loss contrast with the irreducible loss when sample is only present a holdout set. The C-score also correlates with the prediction depth (Baldock et al., 2021), which computes model predictions at intermediate layers, and reports the earliest layer beyond which all predictions are consistent. Equation 3 is also related to the notion of forgetting (Toneva al., 2019; Zhou et al., 2022): count of the number of transitions from learned to forgotten during training an example undergoes. Another related metric is the learning speed: the earliest training iteration after which the model predicts the ground truth class for that example in all subsequent iterations.
Memorisation versus generalisation. Given that the ultimate goal of statistical learning is gen- eralisation, it is natural to ask whether this is at odds with "memorisation". classical result establishes that a lookup table as implemented by the k-NN algorithm is universally consistent (Stone, 1977). Interpolating models such as boosting with decision stumps have similarly been shown to generalise (Bartlett al., 1998), and more refined analyses have been conducted for modern interpolating neural models (Bartlett et al., 2017; Dziugaite & Roy, 2017; Brutzkus et al., 2018; Belkin al., 2018; Neyshabur al., 2019; Liang Rakhlin, 2020; Montanari & Zhong, 2020; Bartlett et al., 2020; Vapnik & Izmailov, 2021). Intriguingly, some recent works have established that under certain settings, "memorisation" may be necessary for generalisation, either in the sense of interpolation (Cheng et al., 2022), stability-based label memorisation (Feldman, 2019), or stronger cxample-level memorisation (Brown al., 2021).
Implicit versus explicit memorisation. Memorisation has received particular interest in the context of large language models (LLMs), such as GPT (Brown et al., 2020) and T5 (Raffel et al., 2020). Here, "memorisation" typically refers to the ability of model to recall factual information present in the training set (e.g.. names of individuals) (Petroni et al., 2019; Roberts et al., 2020; Carlini et al., 2021; 2022). This aligns with the notion from statistical learning of "memorisation" lookup table. While LLMs are capable of implicit memorisation, several works have shown benefits from augmenting neural models with an explicit memorisation component (Lample et al., 2019; Guu et al., 2020; Khandelwal et al., 2020; Borgeaud et al., 2021). Similar ideas have also proven useful outside of NLP (Panigrahy et al., 2021; Vapnik & Izmailov, 2021; Wang & Shao, 2022).
employing
Definition
References
Zero training error
(Zhang et al., 2017)
Zero training error + label is random Training error below Bayes error rate
(Arpit et al., 2017; Yao et al., 2020; Stephenson et al., 2021)
(Bubeck al., 2020; Cheng al., 2022)
Prediction based spurious correlations Ability to reconstruct from other training samples Inability to predict when removed from training sample
(Sagawa al., 2020; Glasgow et al., 2022)
(Radhakrishnan et al., 2020; Carlini et al., 2021)
(Feldman, 2019; Jiang et al., 2021a)
Table 1: Summary of existing definitions of "memorisation" of training sample. In this paper, we focus on the final row, which proposes stability- -ased metric quantifying sample predictability.
16

---

Under review as a conference paper at ICLR 2023
B HYPERPARAMETER SETTINGS
Our experiments use standard ResNet-v2 (He et al., 2016) and MobileNet-v3 (Howard et al., 2019a) architectures. Specifically, for CIFAR, we consider the CIFAR ResNer-(110, 56, 44, 32, 20, 14, 8} family of architectures; for Tiny-ImageNet, we consider the ResNet-(152, 101, 50, 18) and the MobileNet-v3 Large architecture with scale factors (0.35, 0.50, 0.75, 1.00, 1.25). For all ResNet models, we employ standard augmentations as per He et al. (2016a).
We train all models to minimise the softmax cross-entropy loss via minibatch SGD, with hyperparam- eter settings per Table 2.
Parameter
CIFAR1O*
Tiny- Siny-ImageNet
Weight decay
10-4
5-10-
Batch size
1024
256
Epochs
450
90
Peak learning rate
1.0
Learning rate warmup epochs
15
Learning rate decay factor
0.1
Cosine schedule
Learning rate decay epochs
300, 400
N/A
Nesterov momentum
0.9
0.9
Distillation weight
1.0
1.0
Distillation temperature
3.0
1.0
Table 2: Summary of training hyperparameter settings.
17

---

Under review as a conference paper at ICLR 2023
C SUMMARY OF TRAIN AND TEST PERFORMANCE WITH MODEL DEPTH
In Table 3 we report train and test accuracies across architectures on CIFAR- 100 from the one-hot training, while in Table 4 we report train and test accuracies from the distillation training across teachers of varying depths. We find that increasing depth results in models that interpolate the training set, while also generalising better on the test set. We also show how how distillation worsens train accuracy while improving the test accuracy.
Architecture Train
Test
ResNet-8
0.66
0.57
ResNet-14
0.82
0.65
ResNet-20
0.90
0.67
ResNet-32
0.98
0.69
ResNet-44
1.00
0.71
ResNet-56
1.00
0.71
ResNct-110
1.00
0.73
Table 3: Train and test accuracies across architectures on CIFAR-100, Consistent with a growing body of work, on the clean dataset we find that increasing depth results in models that interpolate the training set, while also generalising better on the test set.
(a) ResNet- 10 teacher.
(b) ResNet-56 teacher.
Architecture
Train
Test
Architecture
Train
Test
10-Nee-
0.65
0.59
ResNet-56-RReie 8
0.67
0.60
ResNet-110- 14
0.68
0.58
SeNNet---RReNeet-- 14
0.81
0.69
DesNet-11--Reseeeaa
0.78
0.61
ResNet-56 SesNee-56-ResNett22
0.88
0.71
BosNee-11-RRsNeeee
0.95
0.73
ResNet-56 S0NNeet-6--ReNNet-333 RReNNet-22
0.94
0.74
100bbe--10-Rsset-444
0.98
0.71
SeaNet-56-ResNiaa
0.97
0.75
ResNet-1 10-
0.99
0.75
ResNet-56 ReaNet-56--ReNeet5566 R6esNet-5
0.98
0.75
Table 4: Train and test accuracies across architectures CIFAR-100 under distillation averaged over 5 runs. We find that, compared to the model of corresponding architecture trained using the one-hot objective, distillation lowers train accuracy and in most cases improves test accuracy.
18

---

Under review as a conference paper at ICLR 2023
Faldman score bucket
samples 0000 buckat
15
16
17
resB
re114
res20
ree32
res
res56
res110
Arzhibecture
(O)
Feldman score bucket
10
simppls score bucket
15
16
17
reaD
Ten32
re5S
1ML130
Arthitecture
CIFAR-100
Feldman score bucket
10
of samples socre bucket
16
10
17
18
res18
res34
res50
res101
res152
Architecture
Tiny ImageNet.
Figure 6: Distributions of memorisation scores across datasets and ResNet model depths. A general trend is that the memorisation scores are bi-modal, with most samples' score being close to 0 or 1. Further, this bi-modality is exaggerated with model depth: larger models (unsurprisingly) assign low score ("generalise") on relatively more samples, but also (more surprisingly) assign high score ("memorise") on relatively more samples as well.
D ADDITIONAL EXPERIMENTS: STABILITY-BSSED MEMORISATION
In Figure we show the histogram of memorisation scores as a function of model depth. As in the body, see that increasing depth has the effect of exaggerating the bi-modality of the scores. Figure shows similar plot on CIFAR- -00, where we vary the width of a ResNet-32 model.
19

---

Under review as a conference paper at ICLR 2023
samples score bucket
Feldman score bucket
80
40
20
Illk.
Res32w1
Res32w2
Res32wd
Res32w8
Res32w16
Res32w32
Res32w64
Res32w128
Architecture
(a) CIFAR-100
Figure 7: Distributions of memorisation scores across model widths of a ResNet-32 on CIFAR-100. general trend is that the memorisation scores are bi-modal, with most samples' score being close to 0 or 1. Further, this bi-modality is exaggerated with model width: larger models (unsurprisingly) assign low score ("generalise") on relatively more samples, but also (more surprisingly) assign high score ("memorise") on relatively more samples as well.
20

---

Under review as a conference paper at ICLR 2023
D.1 ADDITIONAL EXPERIMENTS: PER-EXAMPLE TRAJECTORIES OF MEMORISATION OVER MODEL CAPACITY
In Figures 8,10 show additional examples across the three changes in memorisation criteria: least changing, most increasing and least decreasing in memorisation as the model capacity increases. In Tables 9, we show predictions from ResNet-20 and ResNet- 1ttt-110-00 for each example depicted across the Figures. In the discussion below, we refer to examples from Figure 8.
We find that predictions for the least changing memorisation examples are saturated in the correct class, demonstrating that these are easy across model architectures. Predictions for the examples with lowering memorisation are assigned low probability from ResNet-20 for the correct class, and high probability from ResNet-110 10. We can see that these examples are often challenging and mislabeled by reasonable classes early on (e.g., the bowl example with a high probability for clock and plate).
Predictions for the examples with increasing memorisation are often assigned high probability from ResNet-20 to categories which are also present in the image (willow tree also contains forest, and telephone also contains keyboard). We call these hard labeled ambigous examples. The other type we can see are ambigous examples, such as shark and pear, which are not clear and easily confused with other labels as predicted by ResNet- 110 (respectively, dolphin and sweet pepper labels).
We believe the presence of ambigous examples among both increasing and decreasing memorisation trajectories may be explained by the fact that the multiple labels of ambiguous points may contain labels of various difficulty, with the smaller models assigning high probability to the easy labels, while larger models shift more towards the harder labels. Then, depending whether the easy the hard label is present in the training set, different memorization trajectories are observed. If the observed label the easier label, model size gets larger, more probability gets assigned to the harder labels. Thus, increasing model size may lead to increasing memorisation trajectory. Conversely, if the observed label is the harder label, increasing model size may lead the decreasing memorisation trajectory.
21

---

Under review as a conference paper at ICLR 2023
Memonsstion
Momorication
0.4
0.4
0.0
0.0
Aestle-
RoHbet
Restie 20
Resten-22 32
Restet-4
ReHeet-5 56
Ref
Rootee-
Restet-14
Restett
Rotten 32
Restet-4d
Res 3d
Restie
Model depth
Model death
(a) Least changing.
(b) Most decreasing.
066
Momorisation
0.4
0.0
ReMee
Redtee:14
Restet-20
Austeet-2 32
Restlet-4d Add
Restee- 56
Restets 110
Model depth
Most increasing.
Figure 8: Memorisation of examples across model depths which interpolated by ResNet-20 model, and trajectories of memorisation for examples with: the change in memorisation closest to zero, the most decreasing memorisation, and the most increasing memorisation, when comparing ResNet-20 and ResNet- 10 architectures. Solid blue line denotes average trajectory according to the top 1% example selected based on the corresponding criterion, and the dashed lines denote top examples according to the corresponding criterion. We find that the fixed memorisation examples are easy and unambigous, as the case for porcupine. The decreasing memorisation example (plain) arguably more complex and gets confused with caterpillar and road classes by ResNet-20. The increasing memorisation example (willow tree) is ambigous (ResNet- 110t 10 predicts forest which is arguably also valid label). In Table 5, show predictions from ResNet-20 and ResNet- 110 for each example depicted across the subfigures.
22

---

Under review as a conference paper at ICLR 2023
Example
Label
ResNet-20 predictions
ResNet- 110 predictions
porcupine: 0.98
porcupine: shrew; 0.02
porcupine
crab: 0.01
possum: 0.01
girl: 0.00
leopard: 1.00
leopard: 1.00
Icopard
worm: 0.00
worm: 0.00
hamster: 0.00
hamster: 0.00
sweet pepper: 1.00
sweet pepper: 1.00
sweel pepper
worm: 0.00
worm: 0.00
girl: 0.00
girl: 0.00
porcupine: 1.00
porcupine: 1.00
porcupine
worm: 0.00
worm: 0.00
girl: 0.00
girl: 0.00
orange: 1.00
orange: 1.00
orange
worm: 0.00
worm: 0.00
hamster: 0.00
hamster: 0.00
caterpillar: 0.39
plain: 0.86 caterpillar:
plain
plain:
worm: 0,20
road: 0.02
lion: 0.36
lion: 0.94
lion
camel: 0.16
skyseraper: 0.02
skyscraper: 0.09
hamster: 0.01
squirrel: 0.36
squireel: 0.91
squirel
snail: 0.27
snail: 0.06
seal: 0.21
seal: 0.03
bowl:
bowl: 0.87
bowl
clock: 0.29
plate: 0.06
plate: 0.15
clock: 0.03
bee: 0.30
sunflower: 0.75
sunflower
sunflower: 0.25 hutterfly: 0.20
bee: 0.10
butterfly: 0.06
willow tree: 0.74
forest: 0.75
willow tree
forest: 0.21
willow tree: 0.20
palm tree: 0.03
oak tree: 0.02
telephone: 0.50
television: 0.88
telephone
television: 0.40
telephone: 0.09
keyboard: (1.66
keyboard: 0.03
shark: 0.78
shark: 0.40
shark
dolphin: 0.155
dolphin: 0.32
whale: 0.05
whale: 0.21
telephone: 0.52
keyboard: 0.74
telephone
keyboard 0.37
telephone: 0.16
clock: 0.08
clock: 0.09
pear: 0.71
sweet pepper: 0.45
pear
sweet pepper: 0.21
pear: 0.35
aquarium fish: 0.03
bowl: 0.12
Table 5; Predictions from ResNet-20 and ResNet- 1 10 for each example depicted in Figure 8.
23

---

Under review as a conference paper at ICLR 2023
Lcm
Sunflower class examples.
(b) Bee class examples.
(c) Willow class examples.
Forest class examples.
Figure 9: Randomly chosen examples from four classes from CIFAR-100 dataset: sunflower, bee, willow tree and forest.
24

---

Under review as a conference paper at ICLR 2023
Hemorraal
Bostieo
Feddet-dl
Resteed
Fester
Redees-2
fropee 3t2
Reshen
(a) Sunflower class
(b) Bee class.
0.5
0.8
0.6
Namorraae salinn Soae
3.6
Momorison
0.4
0.
0.
Rastet
Reater 22
Bestercad
Rectert
Racked 14
Rasteen Rasker
Rackerdf
Resfied
(c) Willow tree class.
(d) Forest class.
Figure 10: Memorisation of examples from a specific label across model depths which are interpolated by a ResNet-20 model, and trajectories of memorisation for examples with: the change in memorisa- tion closest to zero, the most decreasing memorisation, and the most increasing memorisation, when comparing ResNet-20 and ResNet- 110 architectures.
25

---

Under review as a conference paper at ICLR 2023
Example
Label
ResNet-20 predictions
ResNet-110 predictions
sunflower: 0.76
sunflower:
sunflower
bee: 0.18
bee: 0.45
poppy: 0.05
poppy: 0.06
television: 0.75 wardrobe:
television: 0.71
sunflower
wardrobe: 0.12
can: 0.08
tiger: 0.10
willow tree: 0.29
willow tree: 0.53
sunflower
forest: 0.27
forest: 0.41
maple tree: 0.22
maple tree: 0.05
sunflower: 0.38
sunflower: 0.73
sunflower
sweet pepper: 0.20
pepper: 0.09
rose: 0.06
pear: 0.03
sunflower: 0.71
sunflower: 1.00
sunflower
lobster: 0.14
worm: 0.00
erah: 0.10
girl: 0.00
sunflower: 0.70
sunflower: 0.96
sunflower
bicycle: 0.07
bicycle: 0.02
spider: 0.05
man: 0.01
sunflower: 1.00
sunflower: 1.00
sunflower
worm: 0.00
worm: 0.00
girl: 0.00
girl: 0.00
sunflower: 1.00
sunflower: 1.00
sunflower
worm: 0.00
worm: 0.00
girl: 0.00
girl: 0.00
sunflower: 1.00
sunflower: 1.00
sunflower
worm: 0.00
worm: 0.00
girl: 0.00
girl: 0.00
Table 6: Predictions from ResNet-20 and ResNet- 1 10 for each example from Figure 10.
26

---

Under review as a conference paper at ICLR 2023
Example
Label
ResNet--20 predictions
ResNet-110 predictions
mouse: 0.31
mouse: 0.44 squirrel: 0.27
bee
squirrel: 0.17
possum: 0.14
rabbit: 0.08
rocket: 0.56
rocket: 0.44
bee
lizard: 0.11
cloud: 0.21
cloud: 0.09
lizard: 0.09
lamp: 0.25
can: 0.36
bee
can: 0.23
telephone: 0.25
telephone: 0.19
lamp: 0.17
bee: 0.70
hee: 1.00
bee
shrew: 0.15
0.00
beetle: 0.05
house: 0.00
0.71
bee: 0.95
hee
wolf: 0.05
wolf: 0.01
tiger: 0.04
shrew: 0.01
0.74
bee: 0.95
bee
crab: 0.10
cockroach: 0.03
beetle: 0.06
beetle: 0.02
bee: 1.00
bee: 1.00
bee
worm: 0.00
worm: 0.00
house: 0.00
house: 0.00
bee: 1.00
bee: 1.00
bee
worm: 0.00
worm:
house: 0.00
house: 0.00
bee: 0.98
bee: 0.98
bee
caterpillar: 0.01
caterpillar: 0.02
squirrel: 0.01
worm: 0.00
Table 7: Predictions from ResNet-20 and ResNet- 10 for each example from Figure 10.
27

---

Under review as a conference paper at ICLR 2023
Example
Label
ResNet-20 predictions
ResNet- 10 predictions
oak tree: 0.65
oak tree: 0.64
willow
maple 0.34
maple tree: 0.35
pine tree: 0.01
pine tree:
tree:
oak tree: 0.87
willow tree
maple tree: 0.32
maple tree: 0.13
pine tree: 0.01
hamster: 0.00
oak tree: 0.37
oak tree: 0.41
willow tree
maple tree: 0.36
maple tree: 0.35
pine 0.27
pine tree: 0.24
willow tree: 0.81
willow
willow
forest: 0.12
forest: 0.04
pear: 0.02
bottle: 0.01
willow tree: 0.46
willow 0.61
willow tree
sunflower: 0.16
forest: 0.32
forest: 0.16
caterpillar: 0.03
willow tree: 0.81
willow tree: 0.94
willow tree
forest: 0.10
forest: 0.05
pine tree: 0,05
cloud: 0.01
willow 0.98
willow tree: 0.98
willow
forest: 0.02
forest: 0.02
worm: 0.00
worm: 0.00
willow tree: 1.00
willow tree: 1.00
willow tree
worm: 0.00
worm: 0.00
hamster: 0.00
hamster: 0.00
willow tree: 1.00
willow tree: 1.00
willow tree
worm: 0..00
worm: 0.00
hamster: 0.00
hamster: 0.00
Table 8: Predictions from ResNer-20 and ResNet- I for each example from Figure 10.
28

---

Under review as a conference paper at ICLR 2023
Example
Label
ResNet--20 predictions
ResNet-110 predictions
road: 1.00
road: 1.00
forest
worm: 0.00
worm: 0.00
girl: 0.00
girl: 0.00
train: 0.76
train: 0.82
forest
streetcar: 0.18
streetcar: 0.16
pine tree: 0.02
tank: 0.02
spider: 0.32
spider: 0.51
forest
lizard: 0.21
lohster: 0.17
dinosaur: 0.13
table: 0.12
forest: 0.80
forest: 0.98
forest
wardrobe: 0.11
wardrobe: 0.02
skyscraper: 0.02
worm: 0.00
forest: 0.86
forest: 0.98
forest
dinosaur: 0.04
bridge: 0.01
crab: 0.02
tractor: 0.01
forest: 0.90
forest: 0.99
forest
bridge: 0.03
table: 0.01
house: 0.03
worm: 0.00
forest: 1.00
forest: 1.00
forest
worm: 0.00
worm: 0.00
hamster: 0.00
hamster: 0.00
forest: 1.00
forest: 1.00
forest
worm: 0.00
worm: 0.00
hamster: 0.00
hamster: 0.00
forest: 1.00
forest: 1.00
forest
worm: 0.00
worm: 0.00
hamster: 0.00
hamster: 0.00
Table 9: Predictions from ResNet-20 and ResNet- 10 for each example from Figure 10.
29

---

Under review as a conference paper at ICLR 2023
Example
Label
ResNet-20 predictions
ResNet-110 predictions
sunflower: 0.76
sunflower:
sunflower
bee: 0.18
bee: 0.45
poppy: 0.05
poppy: 0.06
television: 0.75
television: 0.71
sunflower
wardrobe: 0.11
wardrobe: 0.12
can: 0.08
tiger: 0.10
willow tree: 0.29
willow tree: 0.53
sunflower
forest: 0.27
forest: 0.41
maple tree: 0.22
maple tree: 0.05
sunflower: 0.38
sunflower: 0.73
sunflower
sweet pepper: 0.20
pepper: 0.09
rose: 0.06
pear: 0.03
sunflower: 0.71
sunflower: 1.00
sunflower
lobster: 0.14
worm: 0.00
erah: 0.10
girl: 0.00
sunflower: 0.78
sunflower: 0.97
sunflower
bowl: 0.09
sweet pepper: 0.03
puppy: 0.07
worm: 0.00
sunflower: 1.00
sunflower: 1.00
sunflower
worm: 0.00
worm: 0.00
girl: 0.00
girl: 0.00
sunflower: 1.00
sunflower: 1.00
sunflower
worm: 0.00
worm: 0.00
girl: 0.00
girl: 0.00
sunflower: 1.00
sunflower: 1.00
sunflower
worm: 0.00
worm: 0.00
girl: 0.00
girl: 0.00
Table 10: Predictions from ResNet-20 and ResNet- 10 for each example from Figure 1.
30

---

Under review as a conference paper at ICLR 2023
E ADDITIONAL EXPERIMENTS: DISTILLATION
Figure 11 shows how memorisation distribution changes under distillation across architectures and datasets. Consistently with the one-hot training depicted in Figure we see both the high memorisation and low memorisation points to increase in number as architecture becomes deeper.
In Figure 12 plot trajectories of memorisation under distillation for examples depicted in Figure 1. We find that memorisation is overall lowered for the challenging and ambigous examples.
In Figure 13 we report additional results for how the memorisation scores change under distillation. We confirm the observations from l(b) that distillation inhibits memorisation, especially for the highly memorised examples.
Figure 14 studies the effect of varying the teacher model used for distillation. Interestingly. the choice of teacher does not have a strong influence on the results, with even the self-distillation setting for ResNet-32 resulting in an inhibition of memorisation.
We inspect examples where distillation changes memorisation the most compared to the one-hot model. In Figure 15 we contrast memorisation trajectories across examples where memorisation is most increased by distillation, most decreased, and where it least changes. We find that distillation reduces memorisation for hard and ambiguous examples, whereas the remaining groups of examples are easy and unambiguous. Notice how examples where distillation increases memorisation mostly yield similar trajectories to where distillation does not impact memorisation.
31

---

Under review as a conference paper at ICLR 2023
Feldman score bucket
11
15
16
17
res130les2
rest 10tod2
re 116c44
Architecture
CIFAR-IO.
Feldman score bucket
10
samples in 2o0re bucket
16
20
18
17
res1100ed
res1301614
rest rest10t=20
res1 res1301o32
rest
res110he56
Architecture
CIFAR 100.
Feldman score bucket
10
camples in score bucket
11
12
16
17
18
20
19
rex152tn18
1es152taM1 Architecture
Thy ImageNet ResNet.
Feldman score bucket
10
of Sang scre bucket
11
12
25
17
18
19
20
mebiL250000
mobil25to75
mob:125to100
mob:125te125
Architocture
(d) Tiny ImageNet MobileNet.
Figure 11: Distributions of memorisation under knowledge distillation datasets (different subfigures) and architectures (different bar colors in each subfigure). Similarly as under one-hot training. memorisation scores bi-modal, and this bi-modality is exaggerated with model depth.
32

---

Under review as a conference paper at ICLR 2023
1.0
Memorisation Score
0.8
0.6
0.4
0.2
0.0
ResNet-110to8 110tt-
ResNet-11ore 1esNet-110to14
ResNet-110to20
Res Restlet-110t032 110to32
ResNet- 110to44 110004
ResNet-110ual 110to56
ResNet- 110to110
Figure 12: Illustration of how distillation affects how memorisation (in the sense of Equation 1) evolves with ResNet model depth CIFAR-100 for examples depicted in Figure 1. We find that memorisation is overall lowered for the challenging and ambigous examples.
10f
Rex32
Disilled mem, score
20
Distiled mem. score
15
mom,
10
10'
Dicilled
10*
100
10"
10
15
20
10
15
10
15
10
20
Student memorisation Score
Student memorisation score
Student memorisation score
Student memorisation score
(a) CIFAR-100: contrasting
ResNet architectures.
20
Mobi125
Mobi35
10f
Distille mem, score
8
Mobi
MobIS0
10a
Disitlled mem, score
10
Mobi125 Mobi75
104
Distilled mem. score
220
Mobi125
Mab/100
10a
15
in
15
-5
10
tol
16
10a
10
202
100
10
Diziiled
10*
200
10"
15
10
10
Student memorisation score
Studentmemoriaa memorisaiion score
Student memorisation score
Student memorisation score
(b) TinyImageNet: contrasting MobileNet architectures.
Figure 13: Contrasting per-example memorisation scores across distilled and one-hot student models in different setups.
33

---

Under review as a conference paper at ICLR 2023
Disilled mem score
13
Res110 - Res8
Distilled morts score
Res110 Res14
Diszilled mom. score
Rec110 10
Res20
20
Redl 10
Res32
10"
15
Dizilles
5
Student memorisation
Student
orisation
memorisation
Distilldd mem. score
Res56 Res8
Disciled mem. score
Res56
Ros14
10
Distilled mem. score
20
Red56
Res20
10*
20
Res56
Res32
10*
15
15
10
107
Distlled
5
55
Student
Student
Student
Distilee mem. score
Res64 Res8
Disellied mam. score
Res44 + Ros
10
Distiled mam. score
Res44
Res20
10a
20
Res44
Res32
10"
15
50
20
102
10
10
Distlled
55
memorisation
memorisation
Ros32 Ras8
Res32
Ros14
Res32
Res20
Res32
Res32
Disiiled mem. score
-10
Dicai lnd mamm score
100
Distiled mem. score
10*
10*
10
10
10
Disilled
in
10 15
5
10
25
20
15
20
15
Student
Student
Student
Figure 14: Contrasting per-example memorisation scores across distilled and one-hot student models for various teachers.
1.0
Memorisation Score
Memorisation Score
0.8
0.6
0.4
0.4
0.0
0.0
Restets Restlet-11ten 12ne8
ResNet-21100 101014
Restet- Restet-110o20 10to0
Restee- Rastet-L1M622 198622
Restet- 10to44
ReaHet-2 LRtSs
ReeNet- 10to11n
ResNet-8
ResHet-14
ResNet-20
ResWet-22
ResNet-44
Resteer-56
ResHet-112
Figure 15: Evolution of memorisation score across one-hot (left) and distilled (right) models for examples across groups: where memorisation is most increased by distillation (blue), most decreased by distillation (red), and where it least changes (green). Distillation reduces memorisation most for hard and ambiguous examples. The remaining groups of examples are easy and unambiguous. version of this Figure with standard deviations is shown in Figure 16 (Appendix).
34

---

Under review as a conference paper at ICLR 2023
1.0
Nemorisation Score
Memor sation Score
0.9
0.6
0.6
0.2
0.2
0.0
Restet-1110a 110to
ResNet--10tol 110
ResNer- L100200
Rester: 110t022
Res Restet- 110tod
ResNet-1100556 1roos
10to111
ResNet-8
ReaHef-14
ResNet:20
Restet- 32
Restet-44
Restet-Se
ResNet-110 1.0
Figure 16: Evolution of memorisation score across one-hot (left) and distilled (right) models for examples across three groups: where memorisation most increased by distillation (blue), most decreased by distillation (red), and where it least changes (green). We plot standard deviations of the Feldman score estimates around the example trajectories. version of this Figure with no standard deviations for clarity is shown in Figure 15.
35

---

Under review as a conference paper at ICLR 2023
F ADDITIONAL EXPERIMENTS PREDICTION DEPTH AND CPROX
Baldock et al. (2021) make a choice between KNN and linear probes. In Appendix E the authors notice how linear probes lead all train examples having prediction depth equal O, leading to a trivial solution. Therefore, the authors settle on the KNN probes, which don't I provide a trivial solution even on the train set. However, the authors consider only 3 architectures, with the strongest model being ResNet-18. 18, and the largest dataset CIFAR-100.
Our goal is consider larger scale experiments from Baldock et al. (2021) in terms of both the architectures and datasets, for which KNN probes on large embeddings quickly become computa- tionally prohibitive. order to allieviate the computational complexity, we consider linear probes on embeddings after average pooling, leading to non trivial distribution of score values across examples. One supporting argument for this approach that in ResNet architecture, the embedding after the final layer is subjected to average pooling before passing to the classification layer. Thus, our approach to linear probes is consistent with how ResNet does classification.
In Figure 17 show the comparison between KNN and linear probes average pooled embeddings. We see how there is a significant degree of similarity between the two distribution, except the low end of depths where the KNN probes assign examples. Notice how the distribution of score values for KNN probes (leftmost figure) resembles that from Baldock et al. (2021) (see Figure 1 in Baldock al. (2021)), despite the latter being computed on non-average pooled embeddings, contrary the former.
probo vs. Linear probe over set
Prediction Depiis probes.
Prwfiction Depthe
20000
-3300
0000
1900
4000
4020
2020
2020
KDVK
Figure 17: Prediction depth statistics on train examples from CIFAR-10 for ResNet- 18 across linear or KNN probes on average pooled embeddings after each layer. Notice how linear probes lead to fewer examples with prediction depth 0 compared to KNN probes (two left-most figures). We notice how there is a high agreement between linear probes and KNN probes (right-most figure), except where KNN probes assign depth 0 or 1, where linear probes increases depth by up to 3, and in the medium range of depths, where linear probes increase the depth by up to 2.
In Figure 18 we report additional results on how prediction depth and c-score proxy scores change per example across model depths. We confirm that the shift in score values is smaller than that for memorisation score (see Figure 4).
In Figure 20 we report prediction depth trajectories model depths for examples shown in Figure 1. We find that all examples vary less in prediction depth scores compared to the memorisation scores.
Different trajectories over model depths. In Figure 20, we plot trajectories of prediction depth over architecture depths for examples depicted in Figure 1. Recall that the latter, we found a range of patterns depending the relative change of memorisation score. For prediction depth, we find that the least changing in memorisation points get the lowest depth across architectures, which may be interpreted classifying them as the easiest. Interestingly, the most and the least changing examples in terms of their memorisation score are not clearly distinguished between when considering prediction depth: most of them get assigned very high prediction depth scores across architectures.
36

---

Under review as a conference paper at ICLR 2023
1.0
15
1.0
10
0.0
128
0.0
A0
Resbet--
0.0
0.5
Resbet-56
0.6
104
0.4
0.4
101
8.4
18
022
0.2
02
0.00
9.25
0.50
0.75
1.00
10P
88.00
0.25
0.50 ResNet-88
0.75
1.00
10"
0.00
0.25
0.50
0.75
1.00
107
Restiet:8
RctNet-8
1.0
134
1.0
1.0
0.0
10
0.5
0.8
56
0.6
18
0.5
10
Re-ble-5 58
0.6
30
0.4
0.4
10
0.2
12
0.2
02
80
0.00
0.25
0.50
5.75
1.00
10
08
0.25
0.50
0.75
1.00
100
8.00
0.25
50
0.75
3.00
10
ResNet-34
ResiNoc144
(a) Memorisation.
(b) C-score proxy (cprox).
(c) Prediction depth.
Figure 18: Contrasting per-example stability-based memorisation, prediction depth and cprox scores across architectures in different setups. Notice how cprox plot is concentrated around the diagonal to greater extent than memorisation score. Prediction depth also more concentrated around the diagonal, albeit to lesser extent than cprox (notice there is more mass towards top left part of the heatmap for prediction depth than for cprox, but less than for memorisation score). Overall, the plots suggest that there relatively few samples whose prediction depth cprox significantly with increased depth. By contrast, a non-negligible fraction of samples receive low stability-basee memorisation under ResNet-8 model, but much higher score under ResNet-56 model.
changes
ResNet-8
ResNet-14
ResNet-8
ResNet-14
Cocore ro
Caccoee presy
20'
Prediction degth
10a
0.6
10?
0.6
0.6
10r
0.4
0.4
0.4
0.4
10t
101
10
10a
00
0.0
0.0
0.5
0.0
0.5
Memorisation score
Memorisstion Score
Memorisation
0.88
ResNet-20 20
prosy
0.8
ResNet-32
0.8
1.0
ResNet-20
10>
1.0
0.8
ResNet-32
10a
Cacore
10a
prediction
10a
Predction
0.4
0.4
0.4
0.4
10
10
0.2
0.2
0.2
0.2
10a
10"
0.0
1.0
10a
0g
0
1.0
10"
Merooisattoo
Memorisation
Memorisation
Memorisation
(a) C-score proxy (cprox).
(h) Prediction depth.
Figure 19: Relationship between stability based memorisation (Equation 1) and alternate measures of memorisation on CIFAR-100 Each plot shows a heatmap of the joint density under the two measures. There is strong correlation with stability-basee memorisation, particularly for the C-score proxy: the correlation to the memorisation score across examples above 90% for all compared model depths, while the correlation between prediction depth and memorisation score is above 70%.
37

---

Under review as a conference paper at ICLR 2023
1.0
Memorisation Score
0.8
0.4
0.2
ResNet-8
ResNet-114
ResNet-20
ResNet-32
ResNet-56
Figure 20; Prediction depth score trajectories for the examples from Figure We find that all examples become less changing in their score when considering prediction depth compared to memorisation score: their trajectory of score over architecture depths is usually roughly constant. This aligns with the heatmap Figure 18 where see that across various architecture depths the example prediction depth usually doesn't t change much.
38

---

Under review as a conference paper at ICLR 2023
20
20
20
28
28
10
D
20
0
0
0-
0
48
-10
-18
-60
-20
-20
-20
-20
-80
-20
28
-28
0
20
-20
28
-20
8
20
(a) Smaller model trained with different random seeds.
10
10
10
10
-10
-10
-10
-10
-20
-20
-20
-20
20
-20
20
(b) Larger model trained with different random seeds.
We show the learnt decision regions (i.e., the difference between the logits from the model assigned to the two classes) when training a smaller (1 hidden layer model with 10k dimensions) and a larger (3 hidden layers with 500 dimensions) models across multiple random seeds on an illustrative roo-dimeniionll binary classification dataset when excluding an outlier (denoted by the black cross). We observe how the larger model is more robust than the smaller model, as denoted by the learnt decision regions being more smooth, and the inner circle being connected as opposed the disconnected areas learnt by the small model. The memorisation score for the highlighted example from the smaller model is 1.0 and from the larger model is 0.65.
Figure 21:
G CONNECTION OF MEMORISATION TO ROBUSTNESS
To illustrate the connection between memorisation and robustness, we consider a two-dimensional toy binary classification dataset with two classes denoted by black points (class 0) and orange points (class 1). In Figure 21, we visualise the learnt decision regions when training a small (1 hidden layer model with 10000 dimensions) and large (3 hidden layers with 500 dimensions) models across multiple random seeds when excluding an outlier (denoted by the black cross). Concretely, we show the difference between logits returned by the model for the two classes; large positive (negative) values denote areas where the model is confident about the class being positive (negative).
We observe that the large model learns smooth decision regions, with the inner circle being connected, as opposed to the disconnected areas learnt by the small model. Thus, intuitively, the larger model more robust than the smaller model. At the same time, when calculating the stability- based memorisation score for the highlighted example, for the smaller model we obtain 1.0 and for the larger model we obtain 0.65. This hints that with greater robustness, one can expect lower memorisation of samples.
In order to measure the robustness difference between the two models more rigorously, we randomly perturb the highlighted outlier with random Gaussian corruptions of varying standard deviations and compare the accuracy on the perturbed example against the original label across the smaller and larger models (Cohen et al., 2019). We plot the decision regions of the compared models and the result of the robustness comparison in Figure 22, and find the larger model to be significantly more robust.
To summarise, on the challenging example, we find the larger model to be more robust, while having lower memorisation.
39

---

Under review as a conference paper at ICLR 2023
20
20
8
1.0
10
Sealler model
Sabutt Accurscy
20
Larger
0
0.8
50
-10
-10
8.6
-69
880
-20
-20
0.4
800
100
20
$1.ra
(a) Smaller model (all data training). (b) Larger model (all data training).
(c) Robustness trajectories.
We show the learnt decision regions (i.e., the difference between the logits from the model assigned to the two classes) when training smaller (1 hidden layer model with 10k dimensions) and larger (3 hidden layers with 500 dimensions) models on illustrative two-dimensional binary classification dataset. Note that contrary to Figure 21, here we include the highlighted outlier (denoted by the black cross). We observe how the larger model is more robust than the smaller model on the highlighted outlier, as denoted by the higher accuracy from the larger model on perturbed inputs with random Gaussian corruptions of varying standard deviations (Cohen et al., 2019).
Figure 22:
40


Reviews:
[{"review_id": "uLo6wMdJtFj", "reviewer": "Reviewer_E5os", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper discusses memorisation of training examples in deep neural networks. \n* In particular, the paper shows how the memorisation profile of an architecture varies according to the depth of the networks.  The paper also shows what kind of examples are more or less memorised in deeper and shallower models.\n* The paper discusses how this is impacted by distillation and carries out a similar study, as above, for distilled models.\n* Finally, the paper studies how computationally efficiently computable metrics of memorisation vary from the exact but intractable one.", "summary_of_the_review": "## Main points\n\n* The argument that _\"While the study of memorisation in large neural models is fascinating, its practical relevance is stymied by a basic fact: such models are typically inadmissible for real-world settings with constraints on latency and memory. Instead, such models are typically compressed via distillation (Bucila et al., 2006; Hinton et al., 2015)\"_ is rather weird and does not help with the motivation of the paper. Further, the citations provided are also not very relevant for models that are deployed in the real world today.\n\n\n* One of the motivations of the model is the work of Feldman et. al. 2019 , whose main argument is that in certain distributions memorisation is necessary for generalisation. However, this paper also mentioned _\"Consequently, a natural hypothesis is that memorisation of individual training samples might initially increase (as a weak model may have insufficient capacity to memorise), but should eventually saturate or reduce (as more expressive models generalise better).\"_ is a contradiction of that argument and is thus, in some way, self contradicting.\n\n* The observation that memorisation score decreases for larger models is an interesting observation. I find Figure 2(a) and 2(b) to be an excellent illustration of this. \nDoes this possibly explain why adversarial robustness of deeper models is better ? For example, this is observed in Appendix B in Madry et. al. and Paleka et. al and the references therein argues that memorisation of (noisy) training data provably leads to higher adversarial error. I think this and other kinds of similar study on the impact of this observation would make a very nice addition to this paper.\n\n* Figure 1(a) is also a good illustration of the fact that there are different kinds of examples that are memorised. Do the _increasing_ examples correspond to the human noisy labels as observed in Wei et. al ?  If that is the case, that deeper models do not need to memorise (or not as much) noisy labels, then it must also have impact on private learning as well as learning with compressed models with them ? For example, Sanyal et. al. show that private models are more inaccurate examples with higher memorisation score and similarly Hooker et. al. for compressed models but not talk about what kind of \"high memorisation\" scores they are inaccurate on whether _increasing_ or _decreasing_. I think this would be a great addition to this study. Similar questions also apply for adversarial training as observed in Xu et. al 2021.\n\n* All the mathematical metrics here are expectations of random variables where the expectation is over the randomness of the training algorithm. I would like to see what is the empirical variance as well. The colour map, if I understand correctly, shows the variance across examples but not the variance due to the randomness in the algorithm.\n\n* As this is an empirical study, I find it to be limiting to only discuss ResNet and MobileNet. It would be nice if the paper should also discuss pure convolutional architectures at least if not transformer architectures to make the study more comprehensive.\n\n* That distillation inhibits overall memorisation and particularly those that display a higher memorisation score in non-distilled score is not very surprising but a nice thing to know. Perhaps a connection can be drawn to why distillation helps in adversarial robustness.\n\nI find the paper interesting and would like to see the above changes made or at least argued why they should not be. Upon that, I am willing to increase my score.\n \n## Minor points\n\n* I am a bit confused my the line styles employed in Fig 3(a). Is there a particular reason for the way it is ?\n* Figure 6 is missing the colour bar.\n* More generally, I think the figures could be a bit smaller and more interesting results from the appendix could be brought to the main text. For example Figure 14\n\nMadry et al. \"Towards deep learning models resistant to adversarial attacks.\" (2017).  https://openreview.net/pdf?id=rJzIBfZAb\n\nPaleka et. al. \"A law of adversarial risk, interpolation, and label noise.\"  (2022).  https://arxiv.org/abs/2207.03933 \n\nWei et al. \"Learning with noisy labels revisited: A study using real-world human annotations.\" (2021).  https://arxiv.org/pdf/2110.12088v2.pdf\n\nSanyal et. al. \" How unfair is private learning?\" (2022) https://proceedings.mlr.press/v180/sanyal22a.html.\n\nHooker et al. \"What do compressed deep neural networks forget?.\" (2019). https://arxiv.org/pdf/1911.05248.pdf\n\nXu, Han, et al. \"To be robust or to be fair: Towards fairness in adversarial training.\" (2021). https://proceedings.mlr.press/v139/xu21b.html\n\nPapernot et al. \"Distillation as a defense to adversarial perturbations against deep neural networks.\" (2016) . https://ieeexplore.ieee.org/document/7546524\n\n\n================== post rebuttal ==================\n\n\nThe author's answers were satisfactory. The paper provides very useful insights and I lean towards acceptance at this stage.", "strength_and_weaknesses": "## Strengths\n* Understanding the impact and diversity of memorisation in modern deep neural networks is an important topic for both understanding neural networks as well as for various notions of reliability of neural networks.\n* The paper is easy to read and does a good job of balancing the mathematical notations and textual verbosity. \n* The authors ask specific questions and designs experiments to validate or negate their hypothesis. I find this approach of empirical validation or falsification quite well done.\n\n## Weaknesses\nWhile the paper provides an interesting empirical study on a fine-grained understanding of memorisation in deep neural networks,  I think the paper lacks in two ways (I illustrate them further in the summary below)\n * Several works have now linked memorisation to other properties like sparsity, robustness, and privacy. I think it would be good to see how the outcomes of this study relate to this. If the authors think a comprehensive study is outside the focus, I think, there should at least be a discussion with some preliminary results. This will not only inspire future works but also increase the relevance of this work.\n* An empirical study where depth and architecture are pain axes of variation, the authors should discuss something more than ResNet and MobileNet. Even all the Mobilenet results have been relegated to the appendix. I think the main paper should include those but in addition, another relevant architecture type should be added.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "pcaOww9A2h", "reviewer": "Reviewer_bHhw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a systematic empirical analysis of the memorization effect of DNNs w.r.t. model size and distillation. It bases the studies on the existing memorization measure 'memorisation score', and performs extensive experiments. The findings are 4-fold: (1) increasing the model complexity tends to make memorization more bi-modal where some hard examples get lower memorization scores while some ambigous examples get higher memorization scores; (2) distillation aids to inhibit memorisation; (3) some other memorization measures can strongly correlate with the used stability-based memorisation measure, but they do not capture key trends seen in the latter.", "summary_of_the_review": "This is a good paper with extensive empirical findings yet flaws in aspects of clarity and explanation. Currently, I am learning to accept this paper.", "strength_and_weaknesses": "# Strength\n- The writing and presentation are nice. It is a pleasure to read this paper.\n- The empirical studies are very thorough. I appreciate the authors' efforts in doing so.\n- The paper is very relevant to ICLR community. The paper makes a step toward understanding the memorization and generalization of deep learning.\n- The findings in this paper may be useful to spark new applications in scenarios such as data selection/reweighting when learning under noise and active learning.\n\n# Weaknesses\n\nGenerally, in the aspects of displaying findings and drawing conclusions, the paper does not have significant issues. However, I would like to see improvements regarding the following points to make the paper a better contribution to the community.\n- In the caption of figure 2, you state \"however for architectures starting at depth 32, only a small number of points remain with\nhigh memorisation score values\". Is the used \"small\" correct? I see a relatively large number of points have high scores when the depth is no less than 32.\n- \"More interestingly, this bi-modality is exaggerated with model depth\". However, as I notice, figure 2(b) cannot reflect this finding clearly. The exaggeration is very marginal. Can this be shown more confidently in some other scenarios? E.g., on mini-imagenet.\n- On page 7, by inspecting figure 1(b), you state \"Interestingly, none of the examples with small memorisation score from the one-hot model obtain a significant increase in memorisation from distillation.\" At first, it is hard for me to clearly understand figure 1(b). Are the plotted quantities the number of data? Besides, is the argument \"none of\" correct?\n- Why does distillation have such impacts on memorization scores? Is the found phenomenon generalizable to other settings, architectures, or modalities? More explanations are desired.\n- Do the different results between the stability-based score and C-score come from that C-score omits the first term in Eq 2? But as said, the first term can be ignored when the model capacity is enough. So does the conclusion in section 5 still hold when you used larger architectures like vision transformers? I want to see a clarification on this point. \n- Can you provide some extra experiments on adversarial examples? I want to know what kind of data will neural networks view the adversarial examples as？Ambigous or hard?\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fTDTb1TF9yo", "reviewer": "Reviewer_wKAi", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work studies how and to what degree deep networks perform memorization of samples in the clean label setting. The authors build upon previous work and employ the stability-based memorization score developped in [1] which measures how much the prediction of a\nmodel on a given sample changes if the sample is present versus not present in the training dataset. Using this measure, the authors\nperform a in-depth study on how memorization is affected as a function of depth of the model as well as the effects of distillation. The \nauthors find that larger depth generally decreases memorization on average, which is in-line with the fact that deeper models generalize\nbetter. Interestingly, a more local look reveals a more fine-grained picture: There are samples that are less memorized as depth increases\nbut at the same time, there are examples that are memorized more. These examples often correspond to mislabeled samples. Moreover,\nthe authors find that distillation of a larger teacher into a smaller student reduces memorization on average.", "summary_of_the_review": "While some novel results are obtained in this work, it still feels more like a larger empirical study that mostly re-confirms what prior work has identified. I am not convinced of the relevance of the distillation experiments since no real insight into the workings of the method are obtained (i.e. why does a distilled network memorize less). The result that simpler memorization metrics cannot capture the same trends is interesting and certainly relevant to the community. Still, in total the novel contributions of this work are not enough in my opinion and I hence recommend the rejection of this work. ", "strength_and_weaknesses": "**Strengths**\n\n1. This work provides a very broad empirical study regarding memorization in the clean label setting, which has not been performed to this degree in prior work. The results obtained in this paper might help in guiding theory to better understand the phenomenon.\n\n2. It is very interesting that other, simpler memorization measures cannot capture the trends that the stability-inspired metric exhibits. The observed trends might serve as sanity checks and thus faciliate the design of new, computationally lighter memorization measures.\n\n**Weaknesses**\n\n1. While this work makes very interesting observations, a large part of the findings and the methodology have already been presented in prior works [1, 2]. The metric used in this work has been developped by [1] and initial empirical results have already been obtained on large\n﻿networks such as ResNets in [2]. [2] already establishes that samples with the highest memorization scores are typically ambiguous or mislabeled, while this work builds on top of this and shows that this effect is exacerbated with depth. While some novel findings are made (i.e. that some samples are memorized less under higher capacity), large parts seem to re-confirm what has been observed in the literature.\n\n2. A novel aspect in this work is the study of knowledge distillation, which was not considered in previous works. While certainly useful in practical settings, I am not convinced whether the understanding of memorization in this specific setting is of particular relevance. Given the fact that distilled models usually generalize better than standard ones, it does not seem so surprising that less memorization happens. However, it would be interesting to understand how memorization is reduced, but the authors  do not address this question. \n\n[1] Does learning require memorization? A short tale about a long tail. CoRR, abs/1906.05271, 2019.\n[2] Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject