What do larger image classifiers memorise?
Authors: Michal Lukasik
Vaishnavh Nagarajan
Ankit Singh Rawat
Aditya Krishna Menon
Sanjiv Kumar
Venue: Accepted by TMLR ()

Abstract: The success of modern neural networks has prompted study of the connection between memorisation and generalisation: overparameterised models generalise well, despite being able to perfectly fit (“memorise”) completely random labels. To carefully study this issue, Feldman (2019) proposed a metric to quantify the degree of memorisation of individual training examples, and empirically computed the corresponding memorisation profile of a ResNet on image classification benchmarks. While an exciting first glimpse into what real-world models memorise, this leaves open a fundamental question: do larger neural models memorise more? This aligns with the common practice of training models of different sizes, each offering different cost-quality trade-offs: while larger models are typically observed to have higher quality, it is of interest to understand whether this is merely a consequence of them memorising larger numbers of input-output patterns. We present a comprehensive empirical analysis of this question on image classification benchmarks. We find that training examples exhibit an unexpectedly diverse set of memorisation trajectories across model sizes: most samples experienced decreased memorisation under larger models, while the rest exhibit cap-shaped or increasing memorisation.  We show that various proxies for the Feldman(2019) memorisation score fail to capture these fundamental trends. Lastly, we find that knowledge distillation — an effective and popular model compression technique — tends to inhibit memorisation, while also improving generalisation. Specifically, memorisation is mostly inhibited on examples with increasing memorisation trajectories, thus pointing at how distillation improves generalisation.

Published in Transactions on Machine Learning Research (4/2024)
What do larger image classifiers memorise?
Michal Lukasik
miskasskkigoonle.com
Google Research
Vaishnavh Nagarajan
waishnash@yooglee.cmm
Google Research
Ankit Singh Rawat Google Research
ankilarravna@@ggagge..oom
Aditya Krishna Menon Google Research
addtyakmenonn@ggogle..com
Sanjiv Kumar
sergjeok-@googlle.cmm
Google Research
Reviewed on OpenReview: https: openreviee. net/ forum? id= Eu73inSyhG
Abstract
The success of modern neural networks has prompted study of the connection between memorisation and generalisation: overparamcterised models generalise well, despite being able to perfectly fit ("memorise") completely random labels. To carefully study this issue, Feldman (2019) proposed metric quantify the degree of memorisation of individual training examples, and empirically computed the corresponding memorisation profile of ResNet image classification benchmarks. While exciting first glimpse into what real- world models memorise, this leaves open fundamental question: do larger newral models memorise more? This aligns with the common practice of training models of different sizes, each offering different cost-quality trade-ofls: while larger models are typically observed to have higher quality, it is of interest to understand whether this is merely a consequence of them memorising larger numbers of input-output patterns. We present a comprehensive empirical analysis of this question on image classification benchmarks. We find that training examples exhibit unxxpectedly diverse set of memorisation trajectories across model sizes: most samples experience decreased memorisation under larger models, while the rest exhibit cap-shaped or increasing memorisation. We show that various proxies for the Feldman (2019) memorisation score fail to capture these fundamental trends. Lastly, we find that knowledge distillation an effective and popular model compression technique tends to inhibit memorisation, while also improving generalisation. Specifically, memorisation mostly inhibited on examples with increasing memorisation trajectories, thus pointing at how distillation improves generalisation.
1 Introduction
Statistical learning is conventionally thought to involve delicate balance between memorisation of training samples, and generalisation to test samples (Hastic et al., 2001). However, the success of overparumeterised neural models challenges this view: such models generalise well, despite having the capacity to memorise, e.g., by perfectly fitting completely random labels (Zhang et al., 2017). Indeed, in practice, such models typically interpolate the training set, i.c., achieve zero misclassification error. This has promptod a scries of analyses aiming understand why such models can generalise (Bartlett al,, 2017; Brutzkus et al., 2018; Belkin et al., 2018; Neyshabur et al., 2019; Bartlett al., 2020; Wang et al., 2021).

---

Published in Transactions on Machine Learning Research (4/2024)
Memoritation Score
0.8
Memorisation Score
0.8
0.6
0.6
0.2
0.2
0.0
ResNet-88
RawWet-14
Restit-220
Restiet
Res
Restet-S5
ResNet-110
ResWet-8
ResNet-Je
Restiet32 RewNet-220
Restet-44
ResWet-56
ResNet-110
(a) Least changing.
Cap-shaped.
1.0
Memorisation Score
0.8
Memorisation Score
0.8
0.4
0.2
0.2
ResNet-
ResRet-14
ResNet-24 20
ResNet 32
Restett-d
Restet-Se
ResWet-110
Restet-8
ResNet-1e 14
Res 20
Restlet-3 32
Restet-dd 44
ResNet-S5 56
Rwseett111
(c) Most decreasing.
Most increasing.
Figure 1: Training examples on CIFAR-100 exhibil a diverse set of memorisalion trajeclories across model depths: fixing attention on training examples belonging to the sunflower class, while many examples unsurprisingly have fixed, decreasing or cap-shaped memorisation scores (green, red and tcal curves), there are also examples with increasing memorisation even after interpolation (blue curves). In $3.2 we discuss their characteristics in more detail, e.g. how these trajectories constitute a vast majority of trajectories seen in the data.
Recently, Feldman (2019) established that in some settings, memorisation may be necessory for generalisa- tion. Here, "memorisation" defined via a theoretically-grounnded stability-based notion, where the high memorisation examples are the ones that the model can correctly classify only if they are present in the training set (see Equation 1 in $2). This definition allows the level of memorisation' of training sample to be estimated for real-world neural models. To that end, Feldman & Zhang (2020) studied the memorisation profile of ResNet on standard image classification benchmarks.
While an exciting first glimpse into how real-world models memorise, it does not tell us how memorisation varies with model size. After all, varying the model size is an important practical consideration: training models of different sizes within a single family (c.g., ResNet (He et al., 2016b), MobileNet (Howard et al., 2019b), and T5 (Raffel al., 2020)) commonplace, forming models while respecting their computation budget for both training and inference. Is the improved performance of larger models merely result of them memorising larger number of input-label patterns? While larger models have more capacity for memorisation, it is yet not well understood whether the result of the commonly used training recipes (e.g., based on SGD) indeed result. models that increasingly memorise more as the model size increases.
enables practitioners to deploy the best per-
I From hereon in, unless otherwise nated, we shall "memorisation" refer the stability-hased notion Feldman (2019); see Equation
2

---

Published in Transactions on Machine Learning Research (4/2024)
Practitioners also employ a systematic approach to obtain high quality models of varying size: knowledge distillation. In particular, it involves developing good-quality small (student) models by drawing supervision from high-performing large (teacher) models. Fundamentally, distillation involves interaction among models of different sizes, and thus a further question can be posed: how does the interaction between the teacher and the student. impact the smaller model's memorisation?
This paper conducts systematic and comprehensive empirical study to answer the aforementioned questions. In particular, using the theoretically-goon grouuldd notion of memorisation from Feldman (2019), analyse the interplay between memorisation and generalisation across different model sizes. Based on controlled experiments, we make the following contributions:
(i) we present a quantitative analysis of how memorisation varies varies with model complexity (e.g., depth or width of ResNet) for image classifiers Our main findings are that increasing the model complexity tends to make the distribution of memorisation across examples more bi-modal (Section 3.3). At the same time, we identify that existing compputalionally-rraatal alternatives of quantifying memorisation and example difficulty do not capture this key trend (Section 3.5).
(ii) to further probe into the bi-modal memorisation trend, we present examples exhibiting varying memori- sation score trajectories across model sizes, and identify four most prominent trajectory types, including those where memorisation increases with model complexity. We find that particularly ambiguous and mislabeled examples exhibit this kind of trajectory (Section 3.4).
(iii) we conclude with quantitative analysis demonstrating that distillation tends to inhibit memorisation, particularly of samples that the one-hot (i.e., non-distilled) student memorises. Intriguingly, we find that the memorisation is mostly inhibited on the examples for which memorisation increases as the model size increases. This observation leads to conclude that distillation improves generalisation by limiting memorisation of such hard examples (Section 4).
2 Background and related work
The term "memorisation" is often invoked when discussing supervised learning, but with several distinct meanings. classical usage refers to models that simply construct a fixed lookup table, viz. a table mapping certain keys to targets (c.g., labels) (Devroye al., 1996; Chatterjce, 2018). Noting that the 1-ncarest neighbour algorithm can interpolate the training data (i.e., perfectly predict every training sample), some works instead use "memorisation" as synonym for interpolation (Zhang et al., 2017; Arpit et al., 2017: Stephenson et al., 2021). A more general definition is that "memorisation" occurs when the training error is lower than the best achievable error (or Bayes-error) (Bubeck et al., 2020; Cheng et al., 2022). We summarise more notions of memorisation in Table 1.
While each of these notions imply "memorising" the entire training set, one may also ask whether a spe- cific sample is memorised. One intuitive formalisation of this notion is that the model predictions change significantly when a sample removed from training (Feldman, 2019; Jiang et al., 2021b), akin algorith- mic stability (Bousquct & Elisscoff, 2002). More preciscly, consider training sample S comprising N í.i.d. draws from some distribution D over labelled inputs (a., y) E X y. A learning algorithem is some randomised function A(: S): X + y, where the randomness is, e.g., owing to initialisation, ordering of mini-batches, and stochasticity in parameter updates. The memorisation score of a sample (æ, y) E S is then (Feldman, 2019):
mem
(z, y: = P P(y== A(z; 5))
Ply A(1; S {1., (()))))))
(1)
in-sample acc.
out-sample acc.
Here, PP()) considers the randomness in the learning algorithm. Intuitively, this is the excess classification accuracy on the sample (a, v) when it is included versus excluded in the training sample. Large neural models can typically drive the first term to 1 for any (x,y) E S (i.e., they can interpolate any training sample); however, some (a. y) may be very hard to predict when they are not in the training sample. Such examples may be considered be "memorised", as the model could not "generalise" these examples based on the rest of the training data alone.
3

---

Published in Transactions on Machine Learning Research (4/2024)
Despite its strengths, Equation 1 has an obvious drawback: it is prohibitive to compute in most practical settings. Indeed, it naïvely requires that we retrain our learner at least N times, with every training sample excluded once; accounting for randomness in A requires further repetitions. Feldman & Zhang (2020) provided a more tractable estimator, wherein for fixed integer M, one draws K sub-samples {s(K) uniformly from PM (S), the set of all M-sized subsets of S. For fixed (æ, y), let Knn = (k E [K]: (2, v) E ) and Kout = fk E [K] (I, & ) denote the sub-samples including and excluding (I,y), respectively. We then compute
memm M,K (2, S) = Kinl
1v = A(x; 5(h)] KEKin
ly= A(a;
(2)
Kout
K
kE
This quantity estimates mem(z. y; S) to precision 0(1/VK) (Feldman & Zhang, 2020). Jiang et al. (2021a)
considered closely related quantity, namely consistency score (or C-score), defined as cscore(a; y; S) EM-H [Esr)P [P(y A(2; S' - [))]) where μ is a distribution over integers. When pu()) is a point- mass, this is the second term in mem (I. y; S). Note that the first term in mema, y; 5) is typically 1 for overparameterised models, since they are capable of interpolation. Jiang et al. (2021a) also proposed effective prozies for the C-score, which rely on the model behaviour across training steps. Suppose we have a model that is iteratively trained for steps I E (1, .T, where at I-th step, the model produces a probability distribution plo: - A(Y) over the labels. Jiang et al. (2021a) argued that the metric capturing the temporal average of the probability assigned to the true label:
cprox () = E, | (x))
can correlate strongly with (the point mass version of) cscore(ar, y; S)
Relating memorisation and example difficulty. Equation 3 provides an interesting bridge wetween memorisation and erample difficulty. For example, Tracin (Pruthi et al., 2020) and GRAD (Paul et al., 2021) also use the evolution of model predictions across training steps to identify samples that are difficult to learn; roughly, these are samples which cause large loss updates when they are trained on. Equation 3 is also related to the notion of forgetting (Toneva et al., 2019; Zhou et al., 2022; Maini et al., 2022): a count of the number of transitions from learned to forgotten during training an example undergoes. Another related metric the learning speed: the earliest training iteration after which the model predicts the ground truth class for that example in all subsequent. iterations. A different notion of difficulty introduced by Nohyun et al. (2023) is the CG score, which relics on calculating the gap in generalization error bounds of overparameterized two-layer network with ReLU activations when an example is excluded from training. Another related measure of sample difficulty is the RHO-loss (Mindermann et al., 2022), wherein the training loss is contrast with the irreducible loss when a sample is only present in a holdout set The C-score also correlates with the prediction depth (Baldock et al., 2021), which computes model predictions at intermediate layers, and reports the earliest layer beyond which all predictions are consistent.
In related line of work, Ghorbani & Zou (2019) proposed the data Shapley score to capture the value of a training example with respect to train dataset, learning algorithm and an evaluation metric. The score differs from the memorisation score in that the impact of an example is evaluated with respect to all subsets of the training set, as opposed to the entire training set. Interestingly, this is similar to Equation 2, which samples fixed-size subsets of the training set.
Memorisation versus generalisation. Given that the ultimate goal of statistical learning is generalisation, it is natural to ask whether this is at odds with "memorisation". A classical result establishes that a lookup table implemented by the k-NN algorithm is universally consistent (Stone, 1977). Interpolating models such as boosting with decision stumps have similarly been shown to generalise (Bartlett et al., 1998), and more refined analyses have been conducted for modern interpolating neural models (Bartlett et al., 2017: Dziugaite& & Roy, 2017; Brutzkus et al., 2018; Belkin et al., 2018; Neyshabur et al., 2019; Liang & Rakhlin, 2020; Montanari & Zhong, 2020; Bartlett et al., 2020; Vapnik & Izmailov, 2021). Intriguingly, some recent works have established that under certain settings, "memorisation" may be necessary for generalisation, either in the sense of interpolation (Cheng et al., 2022), stability- label memorisation (Feldman, 2019), stronger example-level memorisation (Brown et al., 2021).
4

---

Published in Transactions on Machine Learning Research (4/2024)
Implicit versus explicit memorisation. Memorisation has received particular interest in the context of large language models (II.Ms), such as GPT (Brown et al., 2020) and T5 (Raffel et al., 2020). Here, "memorisation" typically refers to the ability of a model to recall factual information present in the training set (e.g., names of individuals) (Petroni et al., 2019; Roberts et al., 2020; Carlini et al., 2021; 2022). This aligns with the notion from statistical learning of "memorisation" as employing a lookup table. While LLMs are capable of implicit memorisation, several works have shown benefits from augmenting neural models with an explicit memorisation component (Lample al., 2019; Guu et al., 2020; Khandelwal et al., 2020; Borgeaud et al., 2021; Yang et al., 2023). Similar ideas have also proven useful outside of NLP (Panigrahy et al., 2021; Vapnik & Izmailov, 2021; Wang & Shao, 2022).
Definition
References
Zero training error
(Zhang et al., 2017)
Zero training error + label is random
(Arpit et al., 2017; Yao et al., 2020; Stephenson et al., 2021)
Training error below Bayes error rate
(Bubeck et al., 2020; Cheng et al., 2022)
Prediction based on spurions correlations
(Sagawa al., 2020; Glasgow et al., 2022)
Ability to reconstruct from other training samples Inability to predict when removed from training sample
(Radhakrishnan et al., 2020; Carlini et al., 2021) (Feldman, 2019; Jiang et al., 2021a)
Summary of existing definitions of "memorisation" of a training sample. In this paper, we focus on the final row, which proposes a stability- metric quantifying sample predictability.
Table 1:
Prior empirical analyses of memorisation. Several works have studied the interpolation behaviour of neural models as one varies model complexity (Zhang et al., 2017: Neyshabur et al., 2019). Empirical studies of memorisation in the sense of fitting to random (noisy) labels was conducted in (Arpit et al., 2017; Gu & Tresp, 2019). These works demonstrated that real-world networks tend to fit "casy" samples first, and exhibit qualitatively different learning trajectories when presented with clean versus noisy samples. Zhang et al. (2020) provided an elegant study of the interplay between memorisation and generalisation for a regression problem, involving learning either constant or identity function. Feldman & Zhang (2020) studied memorisation in the sense of the trability-based memorisation score in Feldman (2019) (cf. Equation 1), by quantifying the influence of each training example on different test examples. Based on these, they identified a subset of test examples for which the model significantly relies on the memorised (in the stability sense) training examples to make correct predictions. While the direct inspiration for our study, these experiments were for a single architecture on CIFAR-100 and ImageNet. Furthermore, they did not consider the impact of model distillation.
Concurrent work (Dam et al., 2023) analyses the effect of model compression specifically for CIFAR-10 and concur with our findings that compression inhibits memorisation. However, like Feldman (2019), most other works study the memorisation behavior individually for each model rather than consolidate trends across a family of models. Han et al. (2022) examine what images are typically memorised by a large variety of image classifiers. line of work (Baldock et al., 2021; Maini et al., 2023; Stephenson et al., 2021) has investigated how to localize where an example is memorised in given network. Other works have looked at the effect of orthogonal factors such as initialization (Mchta al., 2021) on memorisation. Xu et al. (2023) examine the interplay between memorisation and robustness specifically for adversarialll--trinee models, while we study standard-trained models.
An orthogonal line of work has extended the concept of memorisation to language modeling. Zheng & Jiang (2022); Zhang et al. (2021) exlend the formulation of memorisation in Feldman (2019) and confirm that the long tail theory holds in language datasets as well. However, most work in language models (Carlini et al., 2023; Bai et al., 2021; Biderman et al., 2023) take qualitatively different approach to memorisation, wherein model is said memorise a datapoint if it can complete a prefix in the same way it was completed in the training set.
5

---

Published in Transactions on Machine Learning Research (4/2024)
3 The unexpected tale of memorisation
To demystify the excellent performance of modern neural networks, key step is developing a systematic understanding of their fundamental properties as scale their capacity. It is known that as the model size grows, both the interpolation and generalisation of the model increase. Given that memorisation is fundamental related property, it is natural to ask how the memorisation behavior of modern networks evolves with increasing model sizes. In this section, we present a detailed empirical study in this direction, while highlighting various nuanced and surprising observations. Before discussing our key findings, we begin by introducing the exact sctup and scope of our empirical study.
3.1 Setup and scope
Quantifying the nature of memorisation requires picking a suitable definition of the term. Owing to its conceptual simplicity and intuitive aligmment with the term "memorisation" we employ the stability based memorisation score of Feldman (2019), per Equation 1. For computational tractability, approximation to this score from Feldman & Zhang (2020), per Equation 2. This reduces the computational burden of estimating mem(z, y; S), but does not eliminate it: for each setting of interest, we need to draw a number of independent data sub-samples, and train a fresh model on each. This necessitates a tradcoff between the breadth of results across settings, and the precision the memorisation scores estimated for any individual result. We favour the former, and estimate mem K (2. y; S) via K 400 draws of sub-samples S ~ Pa (S)),,, with M [0.7N].
employ the
With this setup, empirically examine a simple question: how is memorisation influenced by model ca- pacity? Specifically, for a range of standard image classification datasets - CTFAR-10, CIFAR-100, and Timy-ImageNet we empirically quantify the memorisation score as we vary the capacity of standard neural models, based on the ResNet (He et al., 2016b;c) and MobileNet-v3 (Howard et al., 2019b) family (cf. Appendix D for precise settings). Here, it is worth highlighting that while Feldman (2019) studied memorisation profiles for fixed models, the change in memorisation bchavior across model sizes has not boen systematically studied before.
In the rest of the section, present key findings from our empirical study, progressively exploring mem-- orisation behavior at more granular level. We start with discussing the average memorisation scores for networks and then present the distribution of memorisation scores across training examples for different sized models. We next explore per-example memorisation trajectory across model sizes which leads an intu- itive categorization of all training examples into four categories. We conclude with inspecting whether the properties of memorisation we uncover hold under example difficulty metries which were shown in previous works to highly correlate with the stability based memorisation.
3.2 Sufficiently large models memorise less on average
Previous works have claimed that larger networks imply more memorisation, albeit based on different notion of memorisation than ours (Zhang et al., 2017; Neyshabur et al., 2019; Carlini et al., 2023). This leads one to hypothesise that average memorisation score across training set, per Equation 1 or 2, should increase with model size. We now assess whether this hypothesis is borne out empirically.
In Figure 2, we visualise how ResNet model depth influences the memorisation score (Fquation 2) on CIFAR- 100 and ImageNet. Specifically, for cach model, we report the average memorisation score across all training samples a coarse summary. For CIFAR-100 we see that this score increases up to depth 20, and then, contrary to the naive hypothesis above, starts decreasing (albeit slightly).
This (initially) puzzling phenomenon may be intuitively understood by breaking down the two terms used to compute the memorisation score in equation 2: the in-sample accuracy, and the oul-of-sample accuracy. As expected, both quantities steadily increase with model depth. The increasing memorisation score up to depth 20 can be explained by the in-sample accuracy increasing faster than the out-of-sample accuracy up to this point; beyond this point, the in-sample accuracy saturates at 100%, while the out-of-sample accuracy keeps increasing. Thus, necessarily, the memorisation score starts drop. In Appendix H.3, we

---

Published in Transactions on Machine Learning Research (4/2024)
0.8
0.8
Average value
in-sample accuracy
in-sample accuracy
-X oul-of-sample accuracy
at--sample accuracy
memonisation
memorisation score
0.4
0.4
0.2
ResNet-8 8
Reslet:14
ResNet-20
ResNet-33 32
ResNet-44
ResNet:56
ReaNet- 110
MobieNet:35
MobiteNet-S0
MobileNet: 75
MabileNet-100 100
MobileHet-123
Model Depth
Model Depth
CIFAR 100
(b) ImageNet
Figure 2: Average memorisation scores from models of varying size trained CIFAR-100 (left plot) and ImageNet (right plot) training samples. On CIFAR-100, the average memorisation score steadily increases with depth up till the ResNet-20, and steadily decreases afterwards. The increase up to certain depth can be explained by the in-sample accuracy increasing faster than out-of-sample accuracy (right plot), until the point where the former is ~ 100%. On ImageNet, the in-sample accuracy does not reach 100% and in effect the memorisation scores seen as increasing.
give an alternative explanation for why memorisation decreases after interpolation. For ImageNet, we see memorisation steadily increasing due to in-sample accuracy increasing faster than sample accuracy, while the in-sample accuracy does not saturate.
3.3
Large models have increasingly bi-modal memorisation score distributions
The result of memorisation decreasing with an increasing model size discussed in Section 3.2 only considered the average memorisation score for a given model. But how does the distribution of memorisation scores vary with model capacity? To this end, Figure 3 plots precisely this distribution for each model in consideration. We observe that the memorisation scores tend 1. This is in agreement with observations made by Feldman & Zhang (2020) for fired architecture: ResNet CIFAR-100 and MobileNet ImageNet.
bi-modal, with most samples' score being closer to or
More interestingly, our finding is that this bi-modality is exaggerated with model depth: larger models have higher fraction of samples with both memorisation score close to O and 1. The increase of samples with high memorisation score denotes an increasing generalisation gap on a subset of training points. We find this to true across various datasets and architectures, as shown in Figure (Appendix). While the increasing fraction of samples with low memorisation score is consistent with the finding on CIFAR-100 about the average memorisation score decreasing, is surprising that. there also subset of points with increasing memorisation.
3.4 On the diversity of memorisation trajectories over model sizes
The bi-modality of the memorisation scores that we observed in Section 3.3 suggests that there exist at least two kinds examples: those whose memorisation scores increase with depth, and those whose scores decrease. Next, we seek to more carefully characterize what these examples are by analysing the trajectory of memorisation score for individual examples.
Categorizing examples by their memorisation trajectorics. In Figure 1, we report the average memorisation score as function of model depth (size) for individual examples from the sunflower class of CIFAR-100 (see Appendix for presentation over more classes and across different classes, upholding

---

Published in Transactions on Machine Learning Research (4/2024)
Res-et-8
ResNet-14
ResNet-20
ResNet-32
ResNet-56
Examples count
40
Examples count
Examples count
100
Examples count
Examples count
20
20
20
20
20
1
1
Score value
Score value
Score value
Score value
Score value
(a) CIFAR-I00 100
Examples count
10
MobileNet-35
Examples count
10
MobileNet-50
Exampies count
10
MobileNet-77
Examples count
10
MobileNet-100
Examples count
10
MobileNet- 125
5
5
5
5
5
1
Score value
Score value
Score value
Score value
Score value
(b) ImageNet
Figure 3: Distributions of memorisation scores from models trained on CIFAR-100 (top row) and ImageNet (bottom row) training samples. As model size increases, relatively more samples are highly memorised (notice the hump in the graphs from the top row being progressively pushed to the right); Further, the number of examples with very low memorisation increases well. Overall, we find growing bi-modality of the distribution of memorisation scores with increasing size. In Figure 6 (Appendix). the bi-modality of the memorisation score holds across more datasets (CIFAR-10, CIFAR-100, ImageNet, Tinylmage Net and model families (ResNet, MobileNet) and in Figure 7 we show this when varying model widths (as opposed to depths) to increase model sizes.
show that
observations, as well as from the ImageNet models). We find four different kinds of trajectory patterns: increasing, decreasing, cap-shaped and constant. Also, in Table 12 (Appendix) we report predictions for the shown examples.
In Table we report counts of examples in the CIFAR-100 data broken down by the trajectory they exhibit across model sizes. For robust counting, we treat change in memorisation by less than 0.1 as no change. We can see how the four trajectory types we present cover the majority of example trajectories.
Interpreting the example trajectories. We next offer qualitative characterisation of the sam- ples from the four trajectory types we identified. Such qualitative grouping of samples is common in work studying ezample difficully Baldock et al. (2021); Jiang et al. (2021a); Feldman & Zhang (2020). We first notice that the least-changing memorisation examples are easy and unambigous. Quantitatively as well, we find that amongst. these examples, the peak memorisation score tends to be low (i.e., such samples tend to consistently not be memo- rised), suggesting that these are easy and unambiguous points. This phenomenon is also evident in Figure 9 (Appendix), where the examples with the least changing memorisation score are also least memorised scores by individual models. Next, we note that the examples corresponding to increasing, cap-shaped and de- creasing memorisation trajectories are hard and ambiguous (ef. Figure 1). The decreasing and cap-shaped examples are arguably hard but with the ground-truth label correctly assigned in the data. On the other hand, the increasing examples are often multi-labelod (c.g., the first increasing example containing a boo) or mislabeled (e.g. the third increasing example), such that a human rater could be unlikely to label these images with the dataset provided label. This parallels the predictions from ResNet-116 10 for these examples,
constant increasing decreasing cap-shaped other
0.05
24.3%
17.5%
14.5%
31.6%
12.1%
0.10
39.3%
29.8%
10.5%
18.7%
1.6%
Percentages of examples in the CIFAR-100 100 data broken down by the trajectory type. For robust counting, treat change memorisation by less than
Table 2:
change.
8

---

Published in Transactions on Machine Learning Research (4/2024)
Res-et-8
ResNet-14
ResNet-20
ResNet-32
ResNet-56
Examples count
Examples count
Examples count
Examples count
10
Examples
1
1
Score value
Score value
Score value
Score value
Score value
(a) cprox
Examples count
ResNet-8
Examples count
ResNet-14
Examples count
ResNet-20
Examples count
ResNet-32
ResNet-56
40
40
20
value
Score value
Score value
Score value
Score value
(b) Prediction depth
Figure 4: Distributions of C-proxy scores (top row; see Equation (3)) and prediction depth (bottom row; see Baldock et al. (2021) and Appendix for the details of our implementation) from models trained on CLEAR-100 training samples. Although these have high correlation with memorisation scores, as promised in prior work, they fail capture the bi-modal behavior of memorisation.
which are arguably no less reasonable than the label as presented in the dataset, as can be observed in Table 12 (Appendix). We inspect more examples and further elaborate on our observations Appendix H.
3.5
Do memorisation score proxies exhibit the same trends?
is difficult to compute. The C-score proxy As discussed in Section 2, the stability-based memorisation computationally efficient alternative to the cprox (Equation 3) proposed in Jiang et al. (2021a)
C-score, a metric closely related to the stability-based memorisation score of Equation 1. Indeed, Jiang et al. (2021a) found this measure to have high correlation with the C-score, while cautioning that it should not be interpreted as an approzimation to more fine-grained characteristics of the latter.
Going beyond correlation, however, we find Figure 4 that the distribution of cprox scores have markedly different characteristics stability-based memorisation: the former are unimodal, with most samples having high score value. Recall that by contrast, stability- based memorisation scores exhibit bi-modal distri- bution, with this phenomenon exaggerated with increasing model size (see Figure 3). Consequently, do not observe many of the unexpected trends exhibited by stability-based memorisation, e.g., the possibility of cprox systematically decreasing for some samples as depth increases. We also report distributions from the example difficult metric called prediction depth, which has been shown to be closely related to the C-score, and analogously find that prediction depth yields a very different distribution than memorisation score (see Appendix).
Overall, view the above discrepancy in statistics: it is possible for two distributions to have very similar correlation statistics, but for the dis- tributions themselves to be visually dissimilar. Thus, we conclude it is important to be cautions about conclusions drawn from a high correlation ween various memorisation scores and their proxies, as they may in reality be capturing different properties of data.
instance of the famous Anscombe's quartet (Anscombe, 1973)
4 Distillation lowers memorisation
Despite the impressive performance of large neural models on range challenging tasks in vision and NLP, practical deployment of such models is often infeasible due to their high inference cost. Recently, knowledge
9

---

Published in Transactions on Machine Learning Research (4/2024)
One-hot
inssample
Distilie mem. pcore
Res110
Res20
10
Distiteed score
20
Res110
Res32
101
out-of-sampla
Average value
memorisation
16
30"
0.6
Distil
in-sample
out-of-sample
10
15
20
10
0.4
memorisation
memorisation score
Student memorisation score
Distited mem. score
Res110 - Res44
10t
Dieited mem. score
26
Res110
ResS6
10
RewNae-B
Reshet-1d
Rushet-20
Rushet-32 32
Rechet-44
Restet-S6
Reviiet-110 110
10
304
30
10
Model Depth
Student memorisation
Student memorisation
Figure Left: Average memorisation score is reduced by distillation across model depths, which can attributed to both the reduced in-sample accuracy and increased out-of-sample accuracy. Figure shows average memorisation scores from one-hot. and distilled models CIFAR-100 training samples. Right: Distillation inhibits memorisation of the student: each plot shows the joint density of memorisation scores under a standard model, and one distilled from a ResNet-110 teacher. As the gap between teacher and student models becomes wider, samples memorised by the student see sharp decrease in memorisation score under distillation (see the vertical bar at right end of plot). Notably, distillation does not affect other samples as strongly.
distillation (Bucilă et al., 2006; Hinton et al., 2015) has emerged as a promising approach to compress these large models into more tractable models. Here, one feeds a large ("teacher") model's predicted distribution over labels as the prediction targets for a small ("student")) model. Compared the standard training on raw labels (as considered in earlier sections), distillation can provide significant performance gains; these are informally attributed to distillation performing "knowledge transfer". Knowledge distillation has been successfully applied across many applications, including: computer vision (Beyer et al., 2021), language modeling (Sanh et al., 2019), information retrieval (Lin et al., 2021), machine translation (Zhou et al., 2020), and ads recommendation (Anil et. al., 2022; Liu el al., 2022).
Compared to standard training, distillation presents us with an interesting setup where models of different sizes interact during the training procedure. This prompts us to ask: how does coupling between models of different sizes during distillation affect, if at all, the memorisation behavior of the resulting student model? Interestingly, while distillation has been shown yield significant performance gains average, training accuracy has been shown to be systematically harmed (Cho & Hariharan, 2019). Previous work also showed how distillation can lead to worsened accuracy on subset of hard examples (Lukasik et al., 2021). In a related study, model compression has been shown to harm accuracy on tail classes (Hooker et al., 2019). These observations indoed hint at the potential impact of distillation procodure on student's memorisation behavior, which we scalematically explore in this section.
Towards this, we consider knowledge distillation conducted using logit matching, and where the teacher is trained on the same sub-sample the student for estimating the memorisation scores per Equation 2. We provide the hyperparameter details in Appendix D.
Distillation inhibits memorisation on average. We begin by investigating what happens to average memorisation under distillation. As discussed earlier, distillation is known to reduce the train accuracy compared to the one-hot models, while increasing the test accuracy (Cho & Hariharan, 2019); in Table 6 (Appendix), we report the train and test accuracies. From this, one could expect distillation to inhibit memorisation. In Figure 5 (left), we illustrate the difference in distributions of memorisation scores across models trained on the ground truth labels (which we call one-hot training) and the models distilled from a ResNet-110 teacher model. As expected, we find that distillation tends reduce the number of memorised samples. From the decomposition of memorisation into in-sample and out-sample accuracies for the one- hot and distilled models, we find that the in-sample accuracy becomes lower and out-of-sample accuracy
10

---

Published in Transactions on Machine Learning Research (4/2024)
Examples set
constant
increasing
decreasing cap-shaped
other
All examples
24.3%
17.5%
14.3%
31.6%
12.1%
Reduced memorisation
0.0%
97.2%
0.0%
2.3%
0.5%
Table 3: Percentages of examples in the CIFAR-100 data broken down by the trajectory type. The examples where distillation reduces memorisation (as measured by comparing memorisation between the distilled and one-hot trained ResNet-32 models, while using ResNet-110 as the teacher model) are mostly increasing in their trajectory.
becomes higher under distillation. This parallels the observation that train accuracy lowers, and test accuracy increases under distillation.
Distillation inhibits memorisation of highly memorised examples. We next turn to analysing the distribution of per-example change in memorisation under distillation. In Figure 5 (right), we report the joint density of memorisation scores under a standard model, and one distilled from a ResNet-110 teacher. We can that distillation inhibits memorisation particularly for the examples highly memorised by the one-hot model, and especially when the teacher-student gap is wide. Interestingly, none of the examples with small memorisation score from the one-hot model obtain a significant increase in memorisation from distillation.
In the Appendix, we also show on per-example trajectories how memorisation lowers especially for the challenging and ambigous examples, which often get high memorisation score value by either the small large models (Figure 14 and Figure 15; Appendix). In the Appendix, we report further results showing how memorisation is overall lowered across different student and teacher models (Figure 19; Appendix).
Why does distillation reduce memorisation but improve generalisation? Our finding is that dis- tillation reduces memorisation but improves generalisation, which is intriguing since recent works suggest memorisation can be beneficial for generalisation (Feldman, 2019). To address this apparent conflict, below we report an analysis of what examples distillation reduces memorisation on. Our hypot is that distilla- tion reduces memorisation mostly on hardest examples, and thus releases the generalisation capacity for the model to perform better on other, easier to learn examples. To verify this hypothesis, in Table 3 we provide the breakdown of the CIFAR- 100 train data into different categories' counts over the dataset (as defined in Figure 1). In particular, of examples S where the memorisation score reduces compared to the one-hot student (one-hot ResNet-32 in the above example). We find that memorisation is mostly reduced by distillation on the examples with an increasing memorisation trajectory examples, which consist of ambiguous and noisy examples (as we note in Section 3.4). Note that this observation relates to the empirical observations about distillation being beneficial in noisy label scenarios (Lukasik et al., 2020).
fix the teacher-student pair of ResNet-110 and ResNet-32 and consider the set
5 Discussion and practical implication
Our study leads to important practical conclusions and avenues for future works. First, one should be careful with using certain statistics as proxics for memorisation. Previous works suggested that various quantities defined based on model training or model inference can provide efficient proxies for memorisation score. Although these proxies appear to yield high correlation to memorisation, are significantly different, and do not capture the key aspects of the memorisation behavior of practical models that we uncover. This points at a future direction of identifying reliable memorisation score proxies which can be efficiently computed.
find that distributionally they
Previous works have categorized the difficulty of examples (Feldman, 2019; Baldock et al., 2021; Jiang et al., 2021a) upon fixing a specific model size. Our analysis points at the importance of characterising examples by taking into account multiple model sizes. For instance, Feldman (2019) refer to examples that have high
11

---

Published in Transactions on Machine Learning Research (4/2024)
memorisation score for a particular architecture the long tail examples for given dataset. As our analysis reveals, what is memorised for one model size, may not be for a different size.
References
Rohan Anil, Sandra Gadanho, Da Huang, Nijith Jacob, Zhuoshu Li, Dong Lin, Todd Phillips, Cristina Pop, Kevin Regan, Gil 1. Shamir, Rakesh Shivanna, and Qiqi Yan. On the factory floor: MI engineering for industrial-scale ads recommendation models, 2022. URL https: hupps:/aaae waxxiv.og/abel... /2099. 05310.
F. J. Anscombe. Graphs in statistical analysis. The Americon Statistician, 27(1):17 21, 1973. doi: 10.1080 10.1080/00031300 1973. 10478966.
URL https://www tandf
doi wttp://///.....////////.....
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emnannel Bengio, Maxinder Kan- wal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look memorization in deep networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, pp. 233 233-242. JMLR.org, 2017.
Ching Yuan Bai, Hsuan- -ien Lin, Colin Raffel, and Wendy Chi-wen Kan. On training sample memorization: Lessons from benchmarking generative modeling with a large-scale competition. In Feida Zhu, Beng Chin Ooi, and Chunyan Miao (eds.), KDD '21: The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2021, pp. 2534-2555 2344. ACM, 2021.
Robert John Nicholas Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. In NewrIPS, 2021. URL https: htap:://arzee http///xxvvv////// owtp:ii.oor//aaaee
wttp://w..rrggaa///........ httpa/bb///200000 09647.
Peter Bartlett, Yoav Freund, Wee Sun Lee, and Robert E. Schapire. Boosting the margin: new explanation for the effectiveness of voting methods. The Annals of Statistics, 26(5):1651 - 1686, 1998. doi: 10.1214/ aos/ nos/1024691352. URL https: //dos.:///i....// httpe::///ooo/////////e/ 10. 1214/2222/// 2214/20s/1024691322. 1024691352.
Peter L. Bartlett, Dylan J. Foster, and Matus Telgarsky. Spectrally-normalizze margin bounds for neural networks. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 6240-6249, 2017.
Peter L. Bartlett, Philip M. Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in lin- ear regression. Proceedings of the National Academy of Sciences, 117(48):30063 300700, 2020. doi: 10.1073/ppaa..100 19073781177 URL https: http://wwwpwaaa/ org/ aooi/abs//0.110 000.17777.1....777811777 1907378117.
Mikhail Belkin, Daniel Hsu, and Partha P. Mitra. Overfitting or perfect fitting? risk bounds for classification and regression rules that interpolate. In Proceedings of the 32nd International Conference Newral Information Processing Systems, NIPS'18, pp. 2306 2317, Red Hook, NY, USA, 2018. Curran Associates Inc.
Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markoeva, Rohan Anil, and Alexander Kolesnikov. Knowl- edge distillation: A good teacher is patient and consistent, 2021.
Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raff. Emergent and predictable memorization in large language models. CoRR, hts//304.11158, 2023. URL https:///oi. org/ 10/g//1..00. 48550/arXiv. 2304. 11158.
Scbastian Borgcaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katic Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions of tokens. CoRR, hos/2112.04442 2021. URL pattps://arxter www.iii.org//baa. org/abs/2112. 04426.
12

---

Published in Transactions on Machine Learning Research (4/2024)
Olivier Bousquet and André Elissceff. Stability and generalization. Journal of Machine Learning Research, 2:499-522 526, 2002.
Gavin Brown, Mark Bun, Vitaly Feldman, Adam Smith, and Kunal Talwar. When is memorization of irrelevant training data necessary for high-accuracy learning? In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing, STOC 2021, pp. 123-132 New York, NY, USA, 2021. URL https:
Association for Computing Machinery. ISBN 9781450380539. doi: 10. //doi.org org/10. 1145/3406325 3451131.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Nya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, 22222000511115 2020. URL https: // //wxxvv./v//////////////////x/a org/ars//////eee/oooo/ooooaoa haa/abbaa200.... 14165.
Alon Brutzkus, Amir Globerson, Eran Malach, and Shai Shalev-Shwartz. SGD learns over-parameterizel networks that provably generalize on linearly separable data. In International Conference on Learning Representations, 2018. URL https://openreviis hwtppperrvieeeeee///ooeeeee htte:/ffrmmiddrr333311
Sébastien Bubeck, Ronen Eldan, Yin Tat Lee, and Dan Mikulincer. Network size and weights size for memorization with two-layers neural networks. In Proceedings of the 34th International Conference Newral Information Processing Systems, NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Cristian Bucilă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '06, pp. 535 541, New York, NY, USA, 2006. ACM.
Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert- Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2633- 2650. USENIX Association, August 2021. ISBN 978-1-933133-24-3, URL fttps://wwan www.w.........o.m./..... org/ conference/
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang, Quantifying memorization across neural language models, 2022. URL https: // //arxi/aaiiv////////e// org/ 2222 07646.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang, Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Represenlations, 2023. URL
1cK.
Satrajit Chatterjee. Learning and memorization. In Jennifer Dy and Andreas Krause (eds.), Proceed- ings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 755-763. PMLR, 10-15 Jul 2018. URL https://proceedi mlr press/v80/ chatterjeel8a.1 html.
Chen Cheng, John Duchi, and Rohith Kuditipudi. Memorize to generalizo: on the necessity of interpolation in high dimensional linear regression, 2022. URL //apiiv///xii.////a http:://axxiv.oorg/bb///2200000 09889.
J. Н. Cho and B. Hariharan. On the efficacy of knowledge distillation. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4793-4801, 2019.
Jeremy M Cohen, Elan Rosenfeld, and J. Zico Kolter. Certified adversarial robustness via randomized smoothing, 2019. URL https://arxiv. wwwpilvwworg/bbbeeee wrgp/bs//////....../////
02918
13

---

Published in Transactions on Machine Learning Research (4/2024)
Harvey Dam, Vinu Joseph, Aditya Bhaskara, Ganesh Gopalakrishnan, Saurav Muralidharan, and Michael Garland. Understanding the effect of the long tail on neural network compression. CoRR, hbs/2306.06233, 2023. URL https://doi.. org/ 10. 48550//rXiv..2300 2306. 06238..
L. Devroye, L. Gyorfi, and G. Lugosi. A probabilistic theory of pattern recognition. Springer, 1996.
Gintare Karolina Dziugaite and Daniel M. Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. In Proceedings of the 33rd Annwal Conference on Uncertainty in Artificial Intelligence (UAI), 2017.
Vitaly Feldman. Does learning require memorization? A short tale about a long tail. CoRR, abs/ 1906.05271, 2019. URL attp://arxiv.ot wwt.:oo///////aoeeeoo 1901119916......
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long lail via influence estimation. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 2881- 2891. Curran Associates, Inc., 2020. URL attps:/pproceedings http://s....iiisss.pp//e///////e pdf.
Amirata Ghorbani and Jamos Zou. Data shapley: Equitable valuation of data for machine learning. Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume of Proceedings of Machine Learning Research, pp. 2242-2251. PMLR, 09 15 Jun 2019. URL attps://proceedings mlr. www..preeaa///// / www.mii9.hh.......
Margalit Glasgow, Colin Wei, Mary Wootters, and Tengyu Ma. Max-margin works while large margin fails: Generalization without uniform convergence, 2022. URL sttps://arxiv.ooo wrg/abs//2055 278.0 07892.
Jindong Gu and Volker Tresp. Neural network memorization dissection. CoRR,, abs/ Mass/9911.09337 2019. URL
http://arae www.oogg///a/aeeoooeoooeeee//e/t abaaaaaaa/9 https//911.0000
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming- Wei Chang. Realm: Retrievallauugmented language model pre-training. In Proceediiggs of the 37th International Conference on Machine Learning, ICML'20. JMgR.org, 2020.
Junlin Han, Huangying Zhan, Jie Hong, Pengfei Fang, Hongdong Li, Lars Petersson, and lan D. Reid. What images more memorable to machines? CoRR, 2022. doi: 10.48550/
URL
https:///oi.... org/10. /rgg//..8550//aae 48550//aXiv..... 2211. 07625.
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning. Springer Series in Statistics. Springer New York Inc., New York, NY, USA, 2001.
He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pottern Recognition (CVPR), 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016a. doi: 10.1109/CVTB P0..2.0...0.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.). Computer Vision ECCV 2016, pp. 630 645, Cham, 2016c. Springer International Publishing. ISBN 978-3-319-46493-00.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. CoRR, abs/1150.002311, 2015.
Sara Hooker, Aaron Courville, Gregory Clark, Yann Dauphin, and Andrea Frome. What do compressod deep neural networks forget?, 2019. URL attps://aaree http:s:://aadd..rr//a////111111 wwtp://w.w....b.....m///. 05248.
14

---

Published in Transactions on Machine Learning Research (4/2024)
Andrew Howard, Ruoming Pang, Hartwig Adam, Quoc V. Le, Mark Sandler, Bo Chen, Weijun Wang, Liang- Chieh Chen, Mingxing Tan, Grace Chu, Vijay Vasudevan, and Yukun Zhu. Searching for mobilenetv3. In 2019 IEEE/CVE International Conference on Computer Vision, 1CCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pp. 1314-1324. IEEE, 2019a. doi: 10.1109/1CCV 2019.00140. URL attps://doi. org/ 10. 1109/ICCV 2019. 2019.00140.
Andrew Howard, Mark Sandler, Grace Chu, Liang- Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc V. Le, and Hartwig Adam. Searching for MobileNet In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), October 2019b.
Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C. Mozer. Charactorizing structural regularities of labeled data in overparameterized models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event,
volume 139 of Proceedings of Machine Learning Research, pp. 5034-5044. PMLR, 2021a. URL http: //proceedings. wwwpediiinsmmm......mmmmmrm wtt....rees//v////// jiang2ik. html.
Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C. Mozer. Charactorizing structural regularities of labeled data in overparameterized models, 2021b.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zeltlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representa- tions, 2020. URL https: /oppr//////peeeiieeeeee wwt.:/nneeeooouuuneeem wit.foorrrnndddtccCBCCCCCCCC
Guillaume Lample, Alexandre Sablayrolles, Marc' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Large memory layers with product keys. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' "'lché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Sys- tems 32: Annual Conference on Newral Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 8546 8557, 2019. URL sttps://proceedinggs www.eeedddnng...ui.s....... cc/ -watrrataaaat--taaaaatttaa html.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to docu- ment recognition. Proceedings of the IEEE, 36(11):2278-233) 1998.
Tengyuan Liang and Alexander Rakhlin. Just interpolate: Kernel "Ridgeless" regression can generalize. The Annals of Statistics, 48(3):1329 1347, 2020. doi: 10.1214/19-0051849. URL https: http://oooo.oo//.///
1214/19-A0SI1
Shong-Chich Lin, Jheng-Hong Yang, and Jimmy Lin. In-batch negatives for knowledge distillation with tightly-coupled leachers for dense retrieval. In Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021 pp. 163 163-173, Online, August 2021. Association for Computational Linguistics. doi:
1.17. URL https: atttps///aclaae h/tpa//taaaannn......./ . org/ 2w2g/222...a2 1121.repp www.eeepppp....... 1.17.
Congcong Liu, Yuejiang Li, Jian Zhu, Xiwei Zhao, Changping Peng, Zhangang Lin, and Jingping Shao. Rethinking position bias modeling with knowledge distillation for ctr prediction, 2022. URL https:
//arxiv. wwwrxiv.rr//aaee htgg/bbs/224....
Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 6448 6458. PMLR, 2020.
Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, and Sanjiv Kumar. Teacher's pet: under- standing and mitigating biases in distillation. CoRR, 0te/22006.14494 2021. URL https://arxiv.org/ hts/21066114994
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards doep learning models resistant to adversarial attacks. In 6th International Conference on Learning Repre- sentations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. Revieevvew.net, 2018. URL httpss///openrreviie http://pparneeieeeeeeeeoorrddada
15

---

Published in Transactions on Machine Learning Research (4/2024)
Pratyush Maini, Saurabh Garg, Zachary C. Lipton, and J. Zico Kolter. Characterizing datapoints via second-split forgetting. In NewrIPS, 2022. URL http://papers. wwwpppprrr..iis....mm...
#tles//aper//022
Pratyush Maini, Michael Curtis Mozer, Hanie Sedghi, Zachary Chase Lipton, J. Zico Kolter, and Chiyuan Zhang, Can neural network memorization be localized? In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, volume 202 Proceedings of Machine Learning Research, pp. 25536-23557. PMLR, 2023. URL httpps:/prroceeinng mlr. www.ppress//vvee wttpss//222mmmaa/ 20002mmaii220 htmm..
Harsh Mehta, Ashok Cutkosky, and Behnam Neyshabur. Extreme memorization via scale of initialization. In 9th International Conference on Learning Represenlations, ICLR 2021. OpenReview.nee 2021. URL petts://opperevieee
Sören Mindermann, Jan M Braumer, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, and Yarin Gal. Prioritized train- ing points that are learnable, worth learning, and not yet learnt. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th Inter- national Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 15630-15649. PMLR, 17-23 Jul 2022. URL html.
/ttp///eroee.....////// mlr, wwt..pees//v62. mnd//m/nnnd222.......m
Andrea Montanari and Yiqiao Zhong. The interpolation phase transition in neural networks: Memorization and generalization under lazy training, 2020. URL https://aaat http:/ooggg/////////e/eoes 2007 12826.
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. The role of over-parametrizatioo in generalization of neural networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.nee 2019.
Ki Nohyun, Hoyong Choi, and Hye Won Chung, Data valuation without training of a model. In The Eleventh International Conference on Learning Representations, 2023. URL https: / http//pperreeiieee www.rrewwwi..nnefoooooeeeeeeeeee wwt/teeuum/eate XIz08zr-Wblt
Rina Panigrahy, Xin Wang, and Manzil Zaheer. Sketch based memory for neural networks. In Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings of The 24th International Conference Artificial Intel- ligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pp. 3169 3177. PMLR, 13-15 Apr 2021. URL
wwainngsmmrr....mmm. wwt.s/pprssss///peeeeeee https///1////aaaaahal1a..a. html.
Nicolas Papernot, Patrick D. McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In IEEE Symposium on Security and Privacy, SP 2016, San Jose, CA, USA, May 22-26, 2016, pp.
2016.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on data diet: Find- ing important examples early in training. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 20596-20607. Curran Associates, Inc., 2021. URL https: fttps://proceedingg wwapidinnnnnnuuussss.eeee www.neeeu.ccceeeeooooeeeeooo.. pctppp/p//22////// .dff.
Fabio Petroni, Tim Rocktäschel, Scbastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexan- der Miller. Language models as knowledge bases? In Proceedinggs of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Lan- guage Processing (EMNLP-IJCNLP)) pp. 2463-2473, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653 1 1250. URL https://aclaatthooogg org/D19-1250.
Garima Pruthi, Frederick Liu, Satyen Kalc, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Had- sell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advonces in Neural Information Processing Systems 39: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020. URL https://poooedingg.neuue aG35639eec3342223334444eeeee Abstract...tt.....o...ma..a html.
16

---

Published in Transactions on Machine Learning Research (4/2024)
Adiyyanarayanan Radhakrishnan, Mikhail Belkin, and Caroline Uhler. Overparameterized neural net- works implement associative memory. Proceedings of the National Academy of Sciences, 117(44):27122- 27170, 2020. doi: 111077//paa.20050131111 CRL https: /ww. pnas /ttp//b//0..11///pmmmp.eemm 2005013117.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 211440) 2020. URL http://jmlr. wwwperrrrrrrreaeeeeeeeee pttp/ppprr//v212200000 html.
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? CoRR, abs/ 00t//202..89911 2020. URL attps://arxiv: www./rr/////2//..//a./a///// 08910.
S. Sagawa, A. Raghumathan, P. W. Koh, and P. Liang. An investigation of why overparamcterization exacerbates spurious correlations. In International Conference on Machine Learning (ICML), 2020.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, 0tss/190.00108, 2019. URL http: //arxivaaaiivv/a/aa htg//bb//911.. 01108.
Cory Stephenson, suchismita padhy, Abhinav Ganesh, Yue Hui, Hanlin Tang, and SueYeon Chung, On the geometry of generalization and memorization in deep neural networks. In International Conference on Learning Representations, 2021. URL https: //tppreeeeeperreee... wwt.:nett/fuut mett/forummt.dvaaarrwubu
Charles Stone. Consistent Nonparametric Regression. The Annals of Statistics, 5(4::595 620, 1977. doi: URL https://doi. http::////oo..o//../////// 1et.11/////1/116348888800
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geof- frey J. Gordon. An empirical study of example forgetting during deep neural network learning. In 7th International Conference on Learning Represenlations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL
Vladimir Vapnik and Rauf Izmailov. Reinforced SVM method and memorization mechanisms. Pattern Recognition, 119:108018, 2021. ISSN 0031-3203. doi: https: wwtp://wwwwiccccccwcctccc.c...ee http////cccceeettaaa////////////
patc:lg.....ooo.......... 2021..00188. URL
Ke Wang, Vidya Muthukumar, and Christos Thrampoulidis. Benign overfitting in multiclass classification: All roads lead to interpolation, 2021. URL https::/arrxv..or///bb////// 10865.
Zhen Wang and Yuan-Hai Shao. Genenliiation-ncnooiiaatooo machines, 2022. URL https://arxiv.org abs/2207. 03976.
Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. In International Conference on Learning Representations, 2022. URL
Han Xu, Xiaorui Liu, Wentao Wang, Zitao Liu, Anil K. Jain, and Jiliang Tang. How does the memorization of neural networks impact adversarial robust models? In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD. ACM, 2023.
Zitong Yang, Michal Lukasik, Vaishnavh Nagarajan, Zonglin Li, Ankit Singh Rawat, Manzil Zaheer, Aditya Krishna Menon, and Sanjiv Kumar. Resmem: Learn what you can and memorize the rest, 2023.
Quanming Yao, Hansi Yang, Bo Han, Gang Niu, and James Tin- Yau Kwok. Searching to exploit memoriza- tion effect in lcarning with noisy labels. In Hal Daumé IIT and Aarti Singh (eds.). Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. PMLR, 13 18 Jul 2020. URL https: https:// /tppa:///pooccinggs/./ mlr, wttps//e1///aae/////// htmll. httpe/diiiniml...esss/eeeee
17

---

Published in Transactions on Machine Learning Research (4/2024)
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, Prance, April 24-26, 2017, Conference Track Proceedings. OpenReview.net 2017.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Michael C. Mozer, and Yoram Singer. Identity crisis: Memo- rization and generalization under extreme overparamcterization In International Conference on Learning Represenlations, 2020. URL https://openrevi ew ww..et//oorurmiii11111yv.v
Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagiclski, Florian Tramèr, and Nicholas Carlini. Counterfactual memorization in neural language models. CoRR, abs/2112.12933, 2021. URL https: // wwwrxii.orggabb//11112 12938.
Xiaosen Zheng and Jing Jiang. An empirical study of memorization NLP. Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 6263-6278, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi:
URL
https://aclaathhology org/2022. acl-long 434.
Chunting Zhou, Jiatao Gu, and Graham Neubig. Understanding knowledge distillation in non-autoregressive machine translation. In International Conference on Learning Representations, 2020. URL https://
Hattie Zhou, Ankit Vani, Hugo Larochelle, and Aaron Courville. Fortuitous forgetting in connectionist networks. In International Conference on Learning Representations, 2022. URL attps://openreviee net/forum?iddee13VLL zYsE.
18

---

Published in Transactions on Machine Learning Research (4/2024)
A Limitations
We identify the following limitalions of our work:
While elegant and established as the memorisation score, the metric for memorisation we use (as defined in Equation 1), even with the approximation we use (as shown in Equation 2) is expensive to compute as requires training of many models. We consider finding an efficient and faithful proxy to memorisation score to be an important future work direction.
We conduct experiments in the image classification domain. While we do experiment on multiple datasots (CIFAR-100, CIFAR-10, ImageNet, Tinylmagenct) and model architectures (ResNet, Mo- bileNet, LeNet), would be of interest to analyse whether similar trends will be observed in NLP tasks.
B Societal impact
We highlight some points regarding the societal impact of our work below:
1. Understanding memorisation is important as it can manifest in undesired behavior by the underlying model. This becomes more critical given that ML models are increasingly being utilized in various areas that affect human life.
2. Even though we have provided detailed study of model memorisation on standard benchmarks, domain specific exploration is needed for individual models before they are deployed in practice.
3. As pointed out by our study, the memorisation behaviour of a model is tied to noise in the dataset. Thus, our work emphasizes the need for ensuring data quality.
4. Finally, memorisation study requires training multiple models which leads to negative envi- ronmental impact. But we hope that the insights from our findings can in the long term serve as guiding principles for future studies and reduce computation in future.
C Broader impact
Our work has multiple implications and future work directions.
First, would like to emphasise the importance of our findings for the broader goal of developing better understanding of deep learning. Both the increasing bi-modality of memorisation scores with increasing model capacity, and the existence of a variety of trajectories of memorisation scores with increasing model capacity, are lo our knowledge new observations in the community. The question of why distillation improves generalisation is still under study by the community. Our work contributes to this branch of work by studying memorisation under distillation.
Next, identify that the following practical implications can be drawn from work:
Our study leads to an important practical conclusion that one should be careful with ing eertain statistics as proxies for memorisation. Previous works suggested that certain statistics based on model training or model inference (which we discuss in Section 5) can provide efficient proxies for memorisation score. Although these proxies appear to yield high correlation to memorisation, we find that distributionally they are significantly different, and do not capture the key aspects of memorisation score that we uncover. This points a future direction of identifying better memorisation score proxies which could be efficiently computed.
19

---

Published in Transactions on Machine Learning Research (4/2024)
Our study points to potentially sound way for identifying noisy examples in the labelled data. Specifically, we find the existence of examples with increasing memorisation over model capacity even after interpolation. We qualitatively find that they are often ambiguous and mislabeled. For example, one general conclusion from this that if the average memorisation a subset of train data grows with model capacily, this can point al a possibility of poor quality of the labels in that set.
Our study points at a potential application of weighting examples based on memorisation score during distillation. Improving distillation by reweighting fittering examples is an active research direction in the community. Given the finding how distillation lowers the memorisation of train examples, and how interestingly this is often achieved by lowering the interpolation (as show in Figure 5), this suggests that memorisation could be used for improving memorisation by filtering out or downweighting examples which the model ends up memorising.
D Hyperparameter settings
Our experiments use standard ResNet-v2 (He et al., 2016) and MobileNet-v3 (Howard et al., 2019a) ar- chitectures. Specifically, for CIFAR, we consider the CIFAR ResNet-(110, 56, 44, 32, 20, 14, 8} family of architectures; for Tiny-ImageNet, we consider the ResNet- (152, 101, 50, 34, 18) and the MobileNet-v3 Targe architecture with scale factors (0.35, 0.50, 0.75, 1.00, 1.25}. For all ResNet models, we employ standard aug- mentations per He et al. (2016a).
We train all models to minimise the softmax cross-entropy loss via minibatch SGD, with hyperparameter settings per Table 4.
Parameter
CIFAR-10*
Tiny-ImageNes
Weight decay
10-4
5-10
Batch size
1024
256
Epochs
450
90
Peak learning rate
1.0
0.1
Learning rate warmup epochs
15
5
Learning rate decay factor
0.1
Cosine schedule
Learning rate decay epochs
300, 400
N/A
Nesterov momentum
0.9
0.9
Distillation weight
1.0
1.0
Distillation temperature
3.0
1.0
Table Summary of training hyperparameter seltings.
20

---

Published in Transactions on Machine Learning Research (4/2024)
E Summary of train and test performance with model depth
In Table 5 we report train and test accuracies across architectures on CIFAR-100 from the onc-hot training, while in Table we report train and test accuracies from the distillation training across teachers of varying depths. We find that increasing depth results in models that interpolate the training set, while also general- ising better on the test set. We also show how how distillation worsens train accuracy while improving the test accuracy.
Architecture
Train
Test
ResNet-8
0.66
0.57
ResNet-14
0.82
0.65
ResNet-20
0.90
0.67
ResNet-32
0.98
0.69
ResNet-44
1.00
0.71
ResNet-56
1.00
0.71
ResNet-110
1.00
0.73
Table 5: Train and test accuracies across architectures on CIFAR-100, Consistent with a growing body of work, on the clean datasct we find that incrcasing depth results in models that interpolate the training set, while also generalising better the test set.
(a) ResNet-110 110 teacher.
(b) ResNet-56 teacher.
Architecture
Train
Test
Architecture
Train
Test
ResNee--110-ReeNee-
0.65
0.59
Res.ee-556-RReseet-
0.67
0.60
Res.ee--110-RReNNet.11
0.68
0.58
ResNett-6G-ResNee-.l1
0.81
0.69
ResNet-110-PResNee-220
0.78
0.64
ResNet-55G-Res.eet.
0.88
0.71
ResNee-1100PReeNeee-332
0.95
0.73
ResNet-56--ResNeet-
0.94
0.74
ResNNet-110--RResNett44
0.98
0.74
ResNet-55O-Resseel---
0.97
0.75
ResNee-110-RReeNet--56
0.99
0.75
ResNet-566-ReNNet--55
0.98
0.75
Table 6: Train and test accuracies across architectures on CIFAR-100 under distillation averaged over 5 runs. We find that, compared to the model of corresponding architecture trained using the one-hot objective, distillation lowers train accuracy and in most cases improves test accuracy.
21

---

Published in Transactions on Machine Learning Research (4/2024)
F Additional experiments: memorisation score distributions
F.1
Memorisation score distributions across datasets
In Figure 6 we show the histogram of memorisation scores as a function of model depth. As in the body, we see that increasing depth has the effect of exaggerating the bi-modality of the scores.
F.2
Memorisation score distributions for varying ResNet width
Figure 7 shows a similar plot on CIFAR-100, where we vary the width of a ResNet-32 model.
F.3
Memorisation score distribution under distillation
Figure 16 shows how memorisation distribution changes under distillation across architectures and datasets. Consistently with the one-hot training depicted in Figure 6 we see both the high memorisation and low memorisation points to increase in number as architecture becomes deeper.
22

---

Published in Transactions on Machine Learning Research (4/2024)
80
Feldman score
% of sampler in mozore bucket
70
11
12
80
13
40
60
20
resd
re54
16520
re532
rS546
rod56
res110
Architecturee
(4) CIFAR-10.
Feldman
56 of samples in sore bucket
14
15
16
re544
res20
rea32 Architectire
reste
res56
res10
(b) CIFAR-100
Feldman soore bucket:
95 sumpies scare bucket
10
15
19
Architecture
(c) Tiny ImageNes.
corox score bucket 10
56 of samoles score bucket
Mobil-te-115 Mobleetet-S0
Mobiletet-75 75
MoNiwWe--100
Mobilk0ee-125
Depth
(d) ImageNet.
Figure 6: Distributions of memorisation scores across datasets and model sizes. A general trend is that the memorisation scores are bi-modal (for sufficiently large models), with most samples' score being close to 0 or 1. Further, Ihis bi-modality is exaggeraled with model depli: larger models (unsurprisingly) assign low score ("generalise") on relatively more samples, but also (more surprisingly) assign high score ("memorise") on relatively more samples as well.
23

---

Published in Transactions on Machine Learning Research (4/2024)
% of samples in score bucket
Feldman bucket
80
60
40
14
20
17
Res32w1
Res32w2
Res32w4
wB
Res32w16
Res32w32
Res32w64
Res32w128
Architecture
(4) CIFAR-100
Figure 7: Distributions of memorisation scores across model widths of ResNet-32 on CIFAR-100. A
general trend is that the memorisation scores are bi-modal, with most samples' score being close to or 1. Further, this bi-modality exaggerated with model width: larger models (unsurprisingly) assign low score ("generalise") on relatively more samples, but also (more surprisingly) assign high score ("memorise") on relatively more samples as well.
24

---

Published in Transactions on Machine Learning Research (4/2024)
G
Additional experiments: larger models memorise some samples more
Given the observation of the increasing bi-modality with model depth, we now study how exactly the mem- orisation score of each example shifts across model depths. In Figure 8, we show how memorisation scores evolve with depth on a per-erample basis. An unsurprising observation is that memorisation scores in most cases do not significantly change across depths. Beyond this, we can see that as the difference in depths becomes larger, the highly memorised examples according to small model become less memorised by the large model (note the high density in the lower-right, part of the plot).
Most interestingly, in all cases, there is non-trivial fraction of samples whose memorisation score increases with model size: note the horizontal bar at the top of multiple subfigures in Figure 8. Increasing memori- sation implies a decreasing out-of-sample accuracy on such points (as the in-sample accuracy monotonically increases with model depth). This is perhaps surprising, since one expects increasing model depth to improve generalisation (Neyshabur et al., 2019).
175
302
18000
(a) CIFAR-100: contrasting ResNet architectures.
(b) ThyImageNett: contrasting ResNet architectures.
Contrasting per-example memorisation scores across architectures in different setups. As the difference depths becomes larger, the highly memorised examples according to a small model become less memorised by the large model (note the high density the lower-right part the plot). On the other hand, smaller models tend have examples which get assigned a high memorisation score by large model (note the horizontal bar at the top of a figure).
Figure 8:
25

---

Published in Transactions on Machine Learning Research (4/2024)
0.3
Avg memerication
0.2
0.1
0.0
0.2
0.4
0.6
0.8
Max memorisatiin change
Figure 9: Average memorisation on CIFAR-100 examples where memorisation changes by al most Least
changing examples are less memorised.
H Additional experiments and discussion: per-example memorisation trajectories
H.1 An intuition behind the categorisations.
The categorisations we introduce can be intuitively explained by breaking down the Feldman memorisation score into in-sample and out-of-sample accuracy components. For the examples which are easy across model depths, we expect. the in-sample and out-of-sample accuracy to be high, and so find a low and unchanging memorisation across depths. Next, for the hard but unambiguous examples, we expect out-of-sample accu- racy for that example to be increasing, which corresponds to the gradually increasing generalisation, and in turn, the memorisation score decreasing after reaching the interpolation regime (i.e., where the in-sample accuracy is 100%). This corresponds to the cap-shaped and decreasing trajectories (the interpolation regime is reached even by the smaller models for the latter). Finally, for the mislabeled or ambiguous examples in the data, we find the correct label, which disagrees with the noisy or ambiguous label in the data, is being recovered better by the deeper models, and so the in-sample accuracy increases or remains high. This corresponds to high or even increasing memorisation.
Interestingly, Wei et al. (2022) find that noisy examples are eventually memorised by the model (here, memorisation was defined as the confidence in the ground truth label), however the non-noisy examples are fitted first. This parallels our observations about memorisation of noisy examples incrcasing, as the labels get interpolated, but the model does not generalise to the noisy labels.
H.2 On predicted labels and memorisation trajectories
In Figures 10 and 12 we show additional examples of per example trajectories. We focus on the following three categories: least changing. most increasing and most decreasing or U-shaped in memorisation as the model capacity increases. In Tables and we show predictions from ResNet-20 and ResNet-110 for cach example depicted across the Figures. the discussion below, we refer examples from Figure 10.
We find that predictions for the least changing memorisation examples are saturated in the correct class, demonstrating that these are easy across model architectures. Predictions for the examples with most decreasing or U-shaped are assigned low probability from ResNet-20 for the correct class, and high probability from ResNet-110. We can see that these examples are often challenging and mislabeled by reasonable classes early on (e.g,, the bowl example with a high probability for clock and plate).
Predictions for the examples with most increasing are often assigned high probability from ResNet-20 to categorics which are also present in the image (willow tree also contains forest, and telephone also contains keyboard). We call these hard labeled ambigous examples. The other type we can see are ambigous ezamples, such as shark and pear, which are not clear and easily confused with other labels as predicted by ResNet-110 (respectively, dolphin and sweet pepper labels).
We believe the presence of ambiguous examples among both increasing and decreasing memorisation tra- jectories may be explained by the fact that the multiple labels of ambiguous points may contain labels of various difficulty, with the smaller models assigning high probability to the casy labels, while larger models shift more towards the harder labels. Then, depending on whether the easy or the hard label is present in the training set, different memorisation trajectories are observed. If the observed label is the easier label,
26

---

Published in Transactions on Machine Learning Research (4/2024)
Memorisation Score
Memorisation Score
0.6
0.4
0.4
0.2
0.2
Reseet-
Restet-14
ResNet-20
Restett 32
ResHet-44 44
Res 56
ResNet-110
ResRet-
ResNet-14 14
ResHet-20
ReaNet-32 32
ResNet-44
ResHet-56
Restlet111
Model depth
Model depth
(a) Least changing.
(b) Most decressing and U-shaped.
0.8
Memoristtion Store
0.2
0.0
Restlet-
ResWet-14
RawNet-20
Rester-32 32
Reallet-4
ResWet-56 56
ResMett1100
Model depth
(c) Most increasing.
Figure 10: Memorisation of examples across model depths which are interpolated by a ResNet-20 model, and trajectories of memorisation for examples with: the change in memorisation closest to zero, the most decreasing memorisation, and the most increasing memorisation, when comparing ResNet-20 and ResNet- 110 architectures. Solid blue line denotes average trajectory according to the top 1% example selected based on the corresponding criterion, and the dashed lines denote top 5 examples according to the correspond- ing criterion. We find that the fixed memorisation examples are easy and unambigous, as is the case for porcupine. The decreasing memorisation example (plain) is arguably more complex and gets confused with caterpillar and road classes by ResNet-20 The increasing memorisation example (willow tree) is ambigous (ResNet-110 predicts forest which arguably also valid label). In Table 7, we show predictions from ResNet-20 and NesNet-110 for each example depicted across the subfigures.
model size gets larger, probability gets assigned to the harder labels. Thus, increasing model size may lead to increasing memorisation trajectory. Conversely, if the observed label is the harder label, increasing model size may lead to the decreasing memorisation trajectory.
We note that previous works offered different metrics for identifying difficult examples, with a prominent example categorisation according to prediction depth (Baldock et al., 2021). We can compare Figure with Figure 29 to see whether the categorisation by memorisation trajectories can be recovered with prediction depth. We see how the most and the least changing examples in terms of their memorisation score not clearly distinguished when considering prediction depth: most of them get assigned very high prediction depth scores across architectures.
27

---

Published in Transactions on Machine Learning Research (4/2024)
Example
Label
ResNet-20 predictions
ResNet-111 predictions
porcupine: 0.98
poreupine: 0.98
porcupine
crab: 0.01
shrew: 0.02
possum: 0.01
girl: 0.00
leopard: 1.00
leopard: 1.00
leopard
worm: 0.00
worm: 0.00
hamster: 0.00
hamster: 0.00
sweet pepper: 1.00
sweet pepper: 1.00
sweet pepper
worm: 0.00
worm: 0.00
girl: 0.00
girl: 0.00
porcupine: 1.00
porcupine: 1.00
porcupine
worm: 0.00
worm: 0.00
girl: 0.00
girl: 0.00
orange: 1.00
orange: 1.00
orange
worm: 0.00
worm: 0.00
hamster: 0.00
hamster: 0.00
caterpillar: 0.39
plain: 0.86 catorpillar: 0.08
plain
plain: 0.22
worm: 0.20
road: 0.02
lion: 0.36
lion: 0.94
lion
camel: 0.16
skyscraper: 0.02
skyscraper: 0.09
hamster: squirrel: 0.91
squirrel: 0.36
squirrel
snail: 0.27
snail: 0.06
seal: 0.21
seal: 0.03
bowl: 0.34
bowl: 0.87
bowl
clock: 0.29
plate: 0.06
plate:
clock: 0.03
bee: 0.30
sunflower: 0.75
sunflower
sunflower: 0.25
butterfly: 0.20
butterfly: 0.06
willow tree:
forest: 0.75
willow tree
forest: 0.21
willow tree: 0.20
palm tree: 0.03
oak tree: 0.02
telephone: 0.50
television: 0.88
telephone
television: 0.40
telephone: 0.09
keyboard: 0.06
keyboard: 0.03
shark: 0.78
shark: 0.40
shark
dolphin: 0.15
dolphin: 0.32
whale: 0.05
whale: 0.21
telephone: 0.52
keyboard: 0.74
telephone
keyboard: 0.37
telephone: 0.16
clock: 0.08
clock: 0.09
pear: 0.71 sweet pepper: 0.21
sweet pepper: 0.45
pear
pear: 0.35
aquarium fish: 0.03
bowl: 0.12
Table 7: Predictions from ResNet-20 and ResNet- 110 for each example depicted in Figure 10.
28

---

Published in Transactions on Machine Learning Research (4/2024)
dem
(a) Sunflower class examples.
(b) Bee class examples.
Willow class examples.
(d) Forest class examples.
Figure 11: Randomly chosen examples from four classes from CIFAR-100 dataset: sunflower, bee, willow tree and forest.
29

---

Published in Transactions on Machine Learning Research (4/2024)
0.8
0.6
0.6
Memoriattion
64
Roswer-
ReaHe-144
ResWesg0
Restest
Resteet
ReaWer-55
Rester- 1tc
Restive-
Realie:24
Reaee:20
Restee-32
Bestet 444
ResteetS6 56
Restett Lte
Sunflower class
b
Bee class.
0.8
6.8
Memerizzion Score
0.6
0.6
0.4
BesHer-
Restee-14
Restatt20
ResRat-22
ResBot-44 44
Resteet-S 56
Realee322
Resenn
Pestet-44
Pasdee-22 20
Pasdenk-22
Rasdeet
Restet122
Willow class.
(d) Forest class.
Figure 12: Memorisation of examples from specific label across model depths which interpolated by ResNet-20 model, and trajectories of memorisation for examples with: the change in memorisation closest to zero, the most decreasing memorisation, and the most increasing memorisation, when comparing ResNet-20 and ResNet-110 architectures.
30

---

Published in Transactions on Machine Learning Research (4/2024)
Example
Label
ResNet-20 predictions
ResNet-110 predictions
sunflower: 0.76
sunflower: 0.48
sunflower
bee: 0.18
bee: 0.45
poppy: 0.05
poppy: 0.06
television: 0.75
television: 0.71
sunflower
wardrobe: 0.11
wardrobe: 0.12
can: 0.08
tiger: 0.10
willow tree: 0.29
willow tree: 0.53
sunflower
forest: 0.27
forest: 0.41
maple tree: 0.22
maple tree: 0.05
sunflower: 0.38
sunflower: 0.73
sunflower
sweet pepper: 0.20
sweet pepper: 0.09
rose: 0.06
pear: 0.03
sunflower: 0.71
sunflower: 1.00
sunflower
lobster: 0.14
worm: 0.00
crab: 0.10
girl: 0.00
sunflower: 0.70
sunflower: 0.96
sunflower
bicycle: 0.07
bicycle: 0.02
spider: 0.05
man: 0.01
sunflower: 1.00
sunflower: 1.00
sunflower
worm: 0.00
worm: 0.00
girl: 0.00
girl: 0.00
sunflower: 1.00
sunflower: 1.00
sunflower
worm: 0.00
worm: 0.00
girl: 0.00
girl: 0.00
sunflower: 1.00
sunflower: 1.00
sunflower
worm: 0.00
worm: 0.00
girl: 0.00
girl: 0.00
Table 8: Predictions from ResNet-20 and ResNet-110 for each example from Figure 12.
31

---

Published in Transactions on Machine Learning Research (4/2024)
Example
Label
ResNet-20 predictions
ResNet-110 predictions
mouse: 0.31
mouse: 0.44
bee
squirrel: 0.17
squirrel: 0.27
possum: 0.14
rabbit: 0.08
rocket: 0.56
rocket: 0.44
bee
lizard: 0.11
cloud: 0.21
cloud: 0.09
lizard: 0.09
lamp: 0.25
can: 0.36
bee
can: 0.23
telephone: 0.25
telephone: 0.19
lamp: 0.17
bee: 0.70
bee: 1.00
bee
shrew: 0.15
worm: 0.00
beetle: 0.05
house: 0.00
bee: 0.71
bee: 0.95
bee
wolf: 0.05
wolf: 0.01
tiger: 0.04
shrew: 0.01
bee: 0.74
bee: 0.95
bee
crab: 0.10
cockroach: 0.03
beetle: 0.06
beetle: 0.02
bee: 1.00
bee: 1.00
bee
worm: 0.00
worm: 0.00
house: 0.00
house: 0.00
bee: 1.00
bee: 1.00
bee
0.00
0.00
house: 0.00
house: 0.00
bee: 0.98
bee: 0.98 caterpillar: 0.02
bee
caterpillar: 0.01
squirrel: 0.01
worm: 0.00
Table 9: Predictions from ResNet-20 and ResNet-110 for each example from Figure 12.
32

---

Published in Transactions on Machine Learning Research (4/2024)
Example
Label
ResNet-20 predictions
ResNet-110 predictions
oak tree: 0.65
oak tree: 0.64
willow tree
maple tree: 0.34
maple tree; 0.35
pine tree: 0.01
pine tree: 0.01
oak tree: 0.67
oak tree: 0.87
willow tree
maple tree: 0.32
maple tree: 0.13
pine tree: 0.01
hamster: 0.00
oak tree: 0.37
oak tree: 0.41
willow tree
maple tree: 0.36
maple tree: 0.35
pine tree: 0.27
pine tree: 0.24
willow tree: 0.81
willow tree: 0.95
willow tree
forest: 0.12
forest: 0.04
pear: 0.02
bottle: 0.01
willow tree: 0.46
willow tree: 0.61
willow tree
sunflower: 0.16
forest: 0.32
forest: 0.16
caterpillar: 0.03
willow tree: 0.81
willow tree: 0.94
willow tree
forest: 0.10
forest: 0.05
pine tree: 0.05
cloud: 0.01
willow tree: 0.98
willow tree: 0.98
willow tree
forest: 0.02
forest: 0.02
worm: 0.00
worm: 0.00
willow tree: 1.00
willow tree: 1.00
willow tree
worm: 0.00
worm: 0.00
hamster: 0.00
hamster: 0.00
willow tree: 1.00
willow tree: 1.00
willow tree
worm: 0.00
worm: 0.00
hamster: 0.00
hamster: 0.00
Table 10: Predictions from ResNet-20 and ResNet-1 for each example from Figure 12.
33

---

Published in Transactions on Machine Learning Research (4/2024)
Example
Label
ResNet-20 predictions
ResNet-110 predictions
road: 1.00
road: 1.00
forest
worm: 0.00
worm: 0.00
girl: 0.00
girl: 0.00
train: 0.76
train: 0.82
forest
streetcar: 0.18
streetcar: 0.16
pine tree: 0.02
tank: 0.02
spider: 0.32
spider: 0.51
forest
lizard: 0.21
lobster: 0.17
dinosaur: 0.13
table: 0.12
forest: 0.80
forest: 0.98
forest
wardrobe: 0.11
wardrobe: 0.02
skyscraper: 0.02
worm: 0.00
forest: 0.86
forest: 0.98
forest
dinosaur: 0.04
bridge: 0.01
crab: 0.02
tractor: 0.01
forest: 0.90
forest: 0.99
forest
bridge: 0.03
table: 0.01
house: 0.03
worm: 0.00
forest: 1.00
forest: 1.00
forest
worm: 0.00
worm: 0.00
hamster: 0.00
hamster: 0.00
forest: 1.00
forest: 1.00
forest
0.00
0.00
hamster: 0.00
hamster: 0.00
forest: 1.00
forest: 1.00
forest
worm: 0.00
worm: 0.00
hamster: 0.00
hamster: 0.00
Table 11: Predictions from ResNet-20 and ResNet-110 for each example from Figure 12.
34

---

Published in Transactions on Machine Learning Research (4/2024)
Example
Label
ResNet-20 predictions
ResNet-111 predictions
sunflower: 1.00
sunflower: 1.00
sunflower
worm: 0.00
worm: 0.00
girl: 0.00
girl: 0.00
sunflower: 1.00
sunflower: 1.00
sunflower
worm: 0.00
worm: 0.00
girl: 0.00
girl: 0.00
sunflower: 1.00
sunflower: 1.00
sunflower
worm: 0.00
worm: 0.00
girl: 0.00
girl: 0.00
bowl: 0.36
sunflower: 0.61
sunflower
sunflower: 0.15
bowl: 0.11
snake: 0.08
ray: 0.07
sunflower: 0.84
sunflower: 0.95
sunflower
orchid: 0.08
orchid: 0.02
poppy: 0.02
butterfly: 0.01
bee: 0.29
sunflower: 0.50
sunflower
poppy: 0.16
bee: 0.26
sunflower: 0.15
poppy: 0.12
sunflower: 0.38 sweet pepper: rose: 0.06
sunflower: 0.73
sunflower
sweet pepper: 0.09
pear: 0.03
sunflower: 0.71
sunflower: 1.00
sunflower
lobster: 0.14
worm: 0.00
crab: 0.10
girl: 0.00
sunflower: 0.78
sunflower: 0.97 sweet pepper:
sunflower
bowl: 0.09
poppy: 0.07
worm: 0.00
sunflower: 0.76
sunflower: 0.48
sunflower
bee: 0.18
bee: 0.45
poppy: 0.05
poppy: 0.06
television: 0.75
television: 0.71
sunflower
wardrobe: 0.11
wardrobe: 0.12
can: 0.08
tiger: 0.10
willow tree: 0.29
willow tree: 0.53
sunflower
forest: 0.27
forest: 0.41
maple tree: 0.22
maple tree: 0.05
Table 12: Predictions from ResNet-20 and ResNet- 110 for each example from Figure
35

---

Published in Transactions on Machine Learning Research (4/2024)
H.3 An intuition for why memorisation (mostly) lowers with depth
We have seen that increasing model capacity lends to lower memorisation on average. We have also discussed an intuition behind how this phenomenon is based on the in-sample accuracy and the out sut-of-sample accuracy. We next demonstrate a more concrete intuition on a synthetic example for why we expect that should be the case.
We consider a two-dimensional toy binary classification dataset with two classes denoted by black points (class 0) and orange points (class 1), in the form of concentric circles. We construct class 0 in way that it has few "outlier" points which harder to generalize from - here, they are handful points from an innermost circle in the distribution. In Figure 13, we visualise the decision regions learned by two models of different sizes: a smaller 1 hidden layer model with 10000 dimensions and larger 4 hidden layers with 500 dimensions. In particular, we visualise the boundaries across multiple random seeds when excluding one of the outliers (denoted by the black cross). Concretely, we show the difference between logits returned by the model for the two classes; large positive (negative) values denote arcas where the model is confident about the class being positive (negative). The decision boundary (where the logit difference is precisely zero) is denoted in solid black line.
We observe that the large model learns smooth decision regions, with the inner circle being connected, as opposed to the disconnected areas learnt by the small model. At the same time, when calculating the stability- based memorisation score for the highlighted example, as model size increases, the memorisation score decreases (see Figure 13(c)). This hints that with increasing model capacity, one can expect the model learn smoother and more contiguous regions; this can lead to better out-of-sample accuracy, and thus lower the memorisation of samples. In I (Appendix) study connection of memorisation to robustness using the same illustrative example.
36

---

Published in Transactions on Machine Learning Research (4/2024)
20
20
20
20
10
10
10
10
-2
0
0
-40
-10
-10
-10
-10
4
660
-20
-20
0
20
-20
20
-20
0
20
-20
0
20
(a) Smaller model trained with different random seeds.
20
20
20
20
10
10
10
10
0
0
0
0-
-10
-10
-10
-10-
-20
-20
-20
-20
-20
20
-20
20
-20
20
-20
20
(b) Larger model trained with different random seeds.
1. 0
X
X
X
X
Memo Memorisati ceroristioon on
0.8
0.6
x
3
layers
0.4
layers
100
200
300
400
500
2000
Hidden Dimension
(c) Memorisation score trajectory for the outlier example.
Figure 13: An illustration of an inluilion for why larger models averuge yield lower memorisation. We consider a 2D binary classification task where one class is distributed as single circle (in red-orange), and the other class (in black) distributed as an exterior circle along with a handful of "outlier" points inside the innermost circle. We visualize the decision boundary (the continuous black lines) when training a smaller model (1 hidden layer model with 10k dimensions; first row) and a larger model (4 hidden layers with 500 dimensions; second row) across multiple random seeds on this task, when excluding one of outliers (denoted by the black cross in the bottom-right of the innermost set of points). We observe that the larger model learns decision boundaries that form one contiguous region around the outliers; in contrast, the smaller model learned disconnected decision boundaries around these outliers. This means that for the larger model, out-of-sample accuracy for the excluded outlier is higher; in lurn, the memorisation score would be lower. Indeed in the last row, we show the memorisation score for the highlighted example across models with different depths and hidden dimensions. As the model size increases, the memorisation score decreases. This hints that with increasing model capacity, one can expect smoother, contiguous regions learnt by model and thus lower memorisation of samples.
37

---

Published in Transactions on Machine Learning Research (4/2024)
1.0
Memorisation Score
0.8
0.6
0.4
0.2
0.0
ResNet- ResNet-11000 110to8
ResNet-110to14 110to14
ResNet-110toz
ResNet-110to32
ResNet-110to44
ResNet- ResNet-110to56
ResNet- 110to110
110to56
Figure 14: Illustration of how distillation affects how memorisation (in the sense of Equation 1) evolves with
ResNet model depth on CIFAR-100 for examples depicted in Figure 1. We find that memorisation is overall lowered for the challenging and ambigous examples.
Memorisation Score
0.8
Memorisation Score
0.8
0.6
0.6
0.4
0.2
0.2
0.0
0.0
ResNet-110toB
Resteet-11tto14
ResNet- RerNet-110to20 110to20
ResNet- 110to32
ResMat-110to32
Restlet- 110to44
1ettee-1110o44
Restiet-1ooo556 1101b56
ResNet-110to110
ResNet-8
ResNat-14
ResNet-20
ResNet-32
ResNet-44
Restlet-56
ResNet-110
ResNet-
Figure 15: Evolution of memorisation score across one-hot (left) and distilled (right) models for examples across groups: where memorisation most increased by distillation (blue), most decreased by distillation (red), and where it least changes (green). Distillation reduces memorisation most for hard and ambiguous examples. The remaining groups of examples easy and unambiguous. A version of this Figure with standard deviations is shown in Figure 20.
H.4 Memorisation trajectories under distillation
In Figure 14, we plot trajectories of memorisation under distillation for examples depicted in Figure 1. We find that memorisation is overall lowered for the challenging and ambigous examples.
In Figure 16 we report distributions of memorisation under KD. We find that distillation preserves the increasing bimodality of the scores as the student architecture increases.
In Figure 18 we report additional results for how the memorisation scores change under distillation. We confirm the observations from 1(b) that distillation inhibits memorisation, especially for the highly memorised examples.
38

---

Published in Transactions on Machine Learning Research (4/2024)
80
Feldman score bucket
10
12
13
14
15
samples
17
18
30
20
10
res110to20
res110to32
res res110
res110to56
Architecture
(a) CIFAR-10.
Feldman score bucket
10
sammles bucket
12
13
15
16
17
18
| res110to56
re:110to8
res110to14
res110to20
res110to32
res1100o44
Architecture
(b) CIFAR-100.
Figure 16: Distributions of memorisation scores under knowledge distillation across datasets (different sub- figures) and architectures (different bar colors in each subfigure.. Similarly as under one-hot training, memorisation scores are bi-modal, and this bi- bi-modality is eraggerated with model depth.
Figure 19 studies the effect varying the teacher model used for distillation. Interestingly, the choice of teacher does not have strong influence on the results, with even the ceff-disiillatin setting for ResNet-32 resulting in inhibition of memorisation.
Figure 20 shows the evolution of memorisation scores for examples where memorisation is most increased, most decreased by distillation, and where it least changes. We find how the distillation least affects the memorisation of casy or unambiguous examples.
H.5 Memorisation trajectories on ImageNet
In Figures 22, 23, 24 we report per example memorisation trajectories for ImageNet with varying MobileNet architecture depths for selected labels: louster, lent and bubble categories. We make similar observations to those made on CIFAR-100 trajectories. First, the least changing trajectories are the easiest examples, with consistently low memorisation across model sizes. Next, both the U-shapod and the docrcasing trajectories are composed of harder examples. Finally, the most increasing trajectories are often more ambiguous or with unclear label (e-g., the first example in the most increasing category for the tend label shows dog), and some of the bubble examples don't seem to clearly show bubbles (e.g. the third most increasing example). Thus, we confirm that in the most increasing category, the examples can be mislabeled and ambiguous as for their label.
39

---

Published in Transactions on Machine Learning Research (4/2024)
30
Feldman bucket
10
25
12
13
15
16
samples
17
18
res152to18
res152to34
res152ta50
(a) Tiny ImageNet ResNet.
Feldman bucket
10
bucket
30
11
12
25
13
15
16
15
18
10
mobi125to35
mobi125tu50
mobi125to75 Architecture
mobi125to100
1ob1225t125
(b) Tiny ImageNet MobileNet.
Figure 17: Distributions of memorisation scores under knowledge distillation across datascts (different sub- figures) and architectures (different bar colors in each subfigure). Similarly as under one-hot training, memorisation scores are bi-modal, and this bi-modality is ezaggerated with model depth.
40

---

Published in Transactions on Machine Learning Research (4/2024)
Res20
104
Distilled mem. score
20
Res110
-14
Distilled mem. score
20
Res110
Res44
104
Distilled mem. score
20
Res56
10
15
15
15
15
10
10a
10
10a
10
10aa
100
10²
Distilled
11o0
10"
10
10
15
20
10 15
20
10
15
20
10
15
20
Student memorisation score
Student memorisation score
Student memorisation score
Student memorisation score
(a) CIFAR- -000: contrasting ResNet. architectures.
20
Mobi125
Mobi35
104
Distilted mem, Score
20
Mobi125
Mobi50
104
Distilled mem. score
20
Mobil25 Mobi75
104
Distilled mem. score
20
Mobi125
Mobil00
15
br
15
bs
10a
10
10a
10
10
Distiled
10e
10e
10e
10c
Student memorisation score
Student memorisation score
Student memorisation score
Student memorisation score
(b) TinylmageNet: contrasting MobileNet architectures.
Figure 18: Contrasting per-example memorisation scores across distilled and one-hot student models in
different setups.
15
Resl10
Res8
100
Res110
Res14
10
Res110
Res20
10*
20
Res110
Res32
Ln
2.
Distilled mem.
10aa
10
10
10?
Distilled
Distilled
Distilled
109
10a
10
10
5
10
20
10
20
10
20
Student memorisation score
Student memorisation score
Student memorisation score
Student memorisation score
Res56 Res8
Res56
Res14
Res56
Res20
Res56 Res32
Distilled mem. score
100
100
Distilled score
10
4.5
35
10
10?
-0
10a
10
10a
10
10
Distilled
Distilled
100
10a
10a
10
Student memorisation score
Student memorisation score
Student memorisation score
Student memorisation score
Res44
Res8
10
Res44
Res14
10*
Res44
Res20
10*
20
Res44 + Res32
10
355
Distilled mem.
20
10a
100
10
-10
10
10a
Distlleed
Distilled
Distilled
10
5
10
15
20
5
10
15
20
5
10
15
20
Student memorisation score
Student memorisation score
Student memorisation score
Student memorisation score
Res32 Res8
Res32
Res14
Res32
Res20
Res32
Res32
110
10f
10*
20
255
Distiled mem.
10
-0o2
10a
10
10
Distilted
Distilted
Distilied
100
10a
10a
10
5
10
10
10
20
Student memorisation
Student memorisation
Student memorisation
Student memorisation
Figure 19: Contrasting per-example memorisation scores across distilled and one-hot student models for various teachers.
41

---

Published in Transactions on Machine Learning Research (4/2024)
1.0
Memorisation Score
0.8
Memorisation Score
0.8
0.6
0.4
0.4
0.2
0.2
0.0
0.0
ResNet-111toB
Res Restlet-110on 110to14
ResNet- 110to20
ResNet-110to20
ResNet- 110to32
ResNet-110to32
Res ResNet-110to44 110to44
ResNet- ResMet-110tosh 110to56
Resteet11101110
ResNet-8
ResNet-14 4
ResNet-20
ResNet--2
ResNet-44
ResNet-56
ResNet-110
Figure 20: Evolution of memorisation score across one-hot (left) and distilled (right) models for examples across three groups: where memorisation most increased by distillation (blue), most decreased by distilla- tion (red), and where least changes (green). We plot standard deviations of the Feldman score estimates around the example trajectories.
of samples in busket
of samples in busket
memorisation score bucket
30
20
Mobi35
Mobi50
Mobi75
Mob1100
Mobil25
(a) CIEAR-100 LeNet-5.
(b) Tiny ImageNet MobileNet.
Figure 21: Distributions of memorisation scores datasets and model architectures. We divide the range of memorisation scores into 10 equally-spaced memorisation
buckets. We consider LeNet-5 (LeCun et al., 1998) models CIFAR-100 training set and MobileNet models Tiny Imagenet of varying sizes. We achieve multiple architecture versions for LeNet-5 by varying the widths: wk denotes a model where width is scaled & times compared to the original LeNet-5. We find an increasingly bi-modal distribution as the width increases.
42

---

Published in Transactions on Machine Learning Research (4/2024)
1.0
1.0
Memorisation Score
Memorisation Score
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
MobileNet:35
MobileNet-50
MobileNet-75
MobileNet-100 100
MobileNet-122
MobileNet:35
MobileNet-50
MobileNet-
MobileNet-100
MobiteNet-125 125
(a) Least changing.
(b)
U-shaped
1.0
Memorisation Score
0.8
0.8
Memorisation
0.6
0.6
0.2
0.2
0.0
0.0
MobileNet-35
MobileNet-50
MobileNet-75
MobileNet-100
MobileNeet-125
MobileNet-33
MobileNet-50
MobileNet-75
MobileNet-100
MobileMet-125
(c) Most. decreasing.
(d) Most increasing.
Figure 22: Per-example memorisation trajectories over depth for ImageNet examples from label toaster. We show that training cramples ezhibit diverse set of memorisation trajectories across model depths: fixing attention on training examples belonging to the sunflower class, while many examples unsurprisingly have fixed, decreasing U-shaped memorisation scores (green, red curves), teal curves), there also examples with increasing memorisation even after interpolation (blue curves). Typically, easy and unambiguously labelled examples follow a fixed trend, noisy examples follow an increasing trend, while hard and ambigu- ously labelled examples follow either an increasing, decreasing or U-shaped trend; in $3.2 we discuss their characteristics in more detail.
43

---

Published in Transactions on Machine Learning Research (4/2024)
1.0
1.0
Memorisation Score
Memorisation Score
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
MobileNet:35
MobileNet-50
MobileNet-75
MobileNett110 100
MobileNet-122
MobileNet:35
MobileNet-50
MobileNett-5
MobileNet-100
MobileNet-125
(a) Least changing.
(b)
U-shaped
1.0
Memorisation Score
0.8
0.8
Memorisation
0.6
0.6
0.2
0.2
0.0
0.0
MobileNet-35
MobileNet-50
MobileNet-75
MobileNet-100
MobileNet-125
MobileNet-35
MobileNet-50
MobileNet-75
MobileNet-100
MobileMet-125
(c) Most. decreasing.
(d) Most increasing.
Figure 23: Per-example memorisation trajectories over depth for ImageNet examples from label tent. We show that training cramples ezhibit diverse set of memorisation trajectories across model depths: fixing attention on training examples belonging to the sunflower class, while many examples unsurprisingly have fixed, decreasing U-shaped memorisation scores (green, red curves), teal curves), there also examples with increasing memorisation even after interpolation (blue curves). Typically, easy and unambiguously labelled examples follow a fixed trend, noisy examples follow an increasing trend, while hard and ambigu- ously labelled examples follow either an increasing, decreasing or U-shaped trend; in $3.2 we discuss their characteristics in more detail.
44

---

Published in Transactions on Machine Learning Research (4/2024)
1.0
1.0
Memorisation Score
Memorisation Score
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
MobileNet:35
MobileNet-50
MobileNet-75
MobileNet-100 100
MobileNet-125
MobileNet:35
MobileNet-50
MobileNett-5
MobileNet-100
MobileNet-125
(a) Least changing.
(b)
U-shaped
1.0
Memorisation Score
0.8
Memorisation Score
0.8
0.6
0.6
0.2
0.2
0.0
0.0
MobileNet-35
MobileNet-50
MobileNet-75
MobileNet-100
MobileNet-125 125
MobileNet-35
MobileNet-50
MobileNet-75
MobileNet-100
MobileMet-125
(c) Most. decreasing.
(d) Most increasing.
Figure 24: Per-example memorisation trajectories over depth for ImageNet examples from label bubble. We show that training cramples ezhibit diverse set of memorisation trajectories across model depths: fixing attention on training examples belonging to the sunflower class, while many examples unsurprisingly have fixed, decreasing U-shaped memorisation scores (green, red curves), teal curves), there also examples with increasing memorisation even after interpolation (blue curves). Typically, easy and unambiguously labelled examples follow a fixed trend, noisy examples follow an increasing trend, while hard and ambigu- ously labelled examples follow either an increasing, decreasing or U-shaped trend; in $3.2 we discuss their characteristics in more detail.
45

---

Published in Transactions on Machine Learning Research (4/2024)
20
20
1.0
10
10
-20
Robust Accuracy
Larger model
Smaller
-20
-40
0.8
0
-10
-60
-20
0.4
-80
-100
-20
20
-20
20
signa
(a) Smaller model.
(b) Larger model.
(c) Robustness trajectories.
Figure 25: We show the learnt decision regions (i.e., the difference between the logits from the model assigned the two classes) when training a smaller (1 hidden layer model with 10k dimensions) and a larger (3 hidden layers with 500 dimensions) models on an illustrative two-dimensional binary classification dataset. Note that unlike in Figure 13, here we include the highlighted outlier (denoted by the black cross). We observe how the larger model is more robust than the smaller model on the highlighted outlier, as denoted by the higher accuracy from the larger model on perturbed inputs with random Gaussian corruptions of varying standard deviations (Cohen et al., 2019).
Memorisation and robustness
Intriguingly, a distinct line of work studied the impact of capacity on adversarial robustness, e.g. Madry et al. (2018) shows how model robustness increases with capacity. while Papernot al. (2016) showed how distillation can lead to improved adversarial robustness. It is thus natural to ask how adversarial robustness relates to memorisation. In order to measure the robustness difference between the two models from our example, we revisit the example discussed in Figure 13 and randomly perturb the highlighted outlier with random Gaussian corruptions of varying standard deviations and compare the accuracy on the perturbed example against the original label across the smaller and larger models (Cohen et al., 2019). We plot the decision regions of the compared models and the result of the robustness comparison Figure 25, and find the larger model be significantly more robust. That intuitively expected, as denoted by the learnt decision regions being more smooth, and the inner circle being connected as opposed to the disconnected areas learnt by the small model.
To summarise, on the challenging example, we find the larger model to be more robust, while having lower memorisation.
46

---

Published in Transactions on Machine Learning Research (4/2024)
J Additional experiments: prediction depth and cprox
The prediction depth has been shown to be closely related to the C-score (Baldock et al., 2021). Prediction depth computes model predictions at intermediate layers, and reports the carliest layer beyond which all predictions are consistent. These predictions were made using k-NN classifier in Baldock et al. (2021). As demonstrated in Baldock et al. (2021), prediction depth forms lower bound on the stability-basod memorisation score.
J.1 On the choice of probes for prediction depth
Baldock et al. (2021) make choice between k-NN and linear probes. In Appendix E, the authors notice how linear probes lead to all train examples having prediction depth equal to 0, leading to trivial solution. Therefore, the authors settle the k-NN probes, which don't provide trivial solution even the train set. However, the authors consider only three architectures, with the strongest model being ResNct-18, and the largest dataset CIFAR-100.
Our goal is to consider larger scale experiments from Baldock et al. (2021) in terms of both the architectures and datasets, for which k-NN probes on large embeddings quickly become computationally prohibitive. order to allieviate the computational complexity, we consider linear probes embeddings after average pooling, leading to this approach is that in ResNet architecture, the embedding after the final layer is subjected to average pooling before passing to the classification layer. Thus, our approach to linear probes is consistent with how ResNet does classification.
trivial distribution of score values across examples. One supporting argument for
In Figure 26, we show the comparison between k-NN and linear probes on average pooled embeddings. We see how there is a significant degree of similarity between the two distribution, except the low end of depths where the k-NN probes assign more examples. Notice how the distribution of score values for k-NN probes (leftmost figure) resembles that from Baldock et al. (2021) (sec Figure 1 in Baldock et al. (2021)), despite the latter being compuled non-average pooled embeddings, contrary to the former.
Prediction Dapths from KNN probes
Prodiction Depthi from inear probos
IOW probe Lineer probe death over train
10000
10000
6000
Unidr probe
6000
000
2300
1000
2000
500
0
3
7
8
1
5
0
KNN arobe
Figure 26: Prediction depth statistics on train examples from CIFAR-10 for ResNet-18 across linear or K-NN probes on average pooled embeddings after each layer. Notice how linear probes lead to fewer examples with prediction depth 0 compared to k-NN probes (two left-most figures). We notice how there is a high agreement between linear probes and k-NN probes (right-most. figure), except where k-NN probes assign depth 0 where linear probes increases depth by up to 3, and in the medium range of depths, where linear probes increase the depth by up to 2.
J.2 Per-sample score trajectories
In Figure 27, report additional results how prediction depth and c-score proxy scores change per example across model depths. We confirm that the shift in score values is smaller than that for memorisation score (see Figure 8).
In Figure 28 we provide more details about the prediction depth score distributions. We see how across model architectures, the prediction depth and memorisation scores correlate to a large extent. At the same time, we see that the score marginal distributions are unimodal, contrasting with the bi-modal distributions found for memorisation score.
47

---

Published in Transactions on Machine Learning Research (4/2024)
10
1.0
10a
110
RedNet
104
ResNet-55
0.6
0.6
0.6
0.4
0.4
0.4
107
101
10
8.00
100
0.00
1.00
10f
8..00
10
1.0
10
1.0
10
10
10
ResNet-56
0.6
100
0.6
10a
0.4
0.4
0.4
10a
10*
0.2
109
10a
10
0.00
0.25
0.50
0.75
1.00
0.00
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
ResNet-14
ResNet-14
ResNet-14
(a) Memorisation.
(b) C-score proxy (cprox).
(c) Prediction depth.
Figure 27: Contrasting per-cxample stability-basod memorisation, prediction depth and cprox scores across architectures in different setups. Notice how cprox plot is concentrated around the diagonal to greater extent than memorisation score. Prediction depth is also more concentrated around the diagonal, albeit to a lesser extent than cprox (notice there is more mass towards top left part of the heatmap for prediction depth than for cprox, but less than for memorisation score). Overall, the plots suggest that there are relatively few samples whose prediction depth or cprox score changes significantly with increased depth. By contrast, a non-negligible fraction samples receive a low stability-basod memorisation score under a ResNet-8 model, but much higher score under ResNet-56 model.
ResNet-8
ResNet-14
1.0
1.0
Prediction depth
10
Prediction depth
10a
0.8
0.8
10a
0.6
10a
0.4
0.4
10
101
0.2
0.2
0.0
10c
0.0
0.0
0.5
0.0
0.5
Memorisation score
Memorisation score
% of samples
ResNet-20
ResNet-32
10"
Prediction degth
0.8
10a
Prediction depth
0.8
10a
ResNet-56
ResNet-44 44
0.6
10
0.6
10a
10-1
ResNet-20
0.4
0.4
ResNet-8
101
0.2
0.2
0.500
636-0.35
400. 45
055.0 60
0
0..0 8o
80-0 85
o 90.0 ps
no
0.0
0.5
1.0
10c
0.0
0.5
1.0
100
0
o
Memorisation
Memorisation
prediction depth score interval
(a) Joint distribution prediction depth and memorisation.
(b) Marginal distribution prediction depth.
Figure 28: Relationship between saabilitybbased memorisation (Equation 1) and prediction depth on CIFAR- 100. The left plot shows a heatmap of the joint densily under the two measures. There is a strong correlation with stability-basel memorisation: the correlation to the memorisation score across examples is above 70% for all compared model depths. The right plot shows the marginal distribution of prediction depth over the CIFAR-100 training across ResNet architectures. The marginals are unimodal, unlike the bi-modal distributions for sebbiltty-bsee memorisation scores.
48

---

Published in Transactions on Machine Learning Research (4/2024)
1.0
0.8
0.6
0.4
0.2
0.0
ResNet-8
ResNet-14
ResNet-20
ResNet-32
ResNet-56
Figure 29: Example Prediction Depth vs Model Size. We report the normalized prediction depth score trajectories for the increasing, decreasing and not-changing memorisation examples from Figure 1. We find that examples become less changing their score when considering prediction depth compared memorisation score: their trajectory of score over architecture depths is usually roughly constant. This aligns with the heatmap in Figure 27 where see that across various architecture depths the example prediction depth usually doesn't change much.
In Figure 29, we plot trajectories of prediction depth over architecture depths for examples depicted in Figure 1. Recall that in the latter, we found range of patterns depending on the relative change of memorisation score. For prediction depth, we find that the least changing in memorisation points get the lowest depth across architectures, which may interpreted as classifying them as the casiest. Interestingly, the most and the least changing examples in terms of their memorisation score not clearly distinguished between when considering prediction depth: most of them get assigned very high prediction depth scores across architectures.
49

---

Published in Transactions on Machine Learning Research (4/2024)
of samples In coore buckot
sut clrerrrrpl acortey
bucket
10
11
12
20
14
15
30
17
Mulli
re514
res20
re532 Architecture
r2544
res56
res110
Rodki Depaa
(a) Avg trajectories.
(b) Score distributions.
% of camplos scare bucket
Feldman score bucket
10
50
11
12
out of camplo siciney crbaban scan
15
16
Wentert
Ammll
/him
18
(c) Avg trajectories.
(d) Score distributions.
Figure 30: Varying the number of epochs in training ResNet models on CIFAR-100: 300 epochs (top row) and 900 epochs (bottom row). See Figure 2a and Figure 6b in paper for results on 450 epochs. We confirm both the increasing bi-modal score distributions, and the reversed U-shape trajectory of average memorisation. We also notice that we train for longer, these phenomena become more pronouneed. (i.e., there are more clearly bi-modal distributions).
50


Reviews:
[{"review_id": "msi2z6P661", "reviewer": "Reviewer_DrRU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well-written and clear in its exposition. Memorization is of interest. Knowledge distillation is used and researched. It seems it will be of interest to the community by providing a reference for empirical results. The claims seem well evidenced. The paper does not over-promise and is quite cautious with its claims.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper looks at:\n* how a memorization metric (memorization score, proposed by Feldman (2019)) changes across model sizes (# of params),\n* whether proxy metrics proposed in prior works, which correlate with this quantity, also capture other qualitative attributes (unfortunately not), and\n* how knowledge distillation affects memorization (overall less memorization).\n\nIt provides empirical evidence using different model architectures on common computer vision classification datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Potential changes:\n\nAbstract: \"While an exciting first glimpse into what real-world models memorise, this leaves open a fundamental question: do larger neural models memorise more?\" -> The introduction makes a better point for why we care about this as we do scale up models, so examining this can potentially provide insights for models that are much more expensive to train.\n\n2. Needs clarification:\n\n\"Is the improved performance realized by increasing model size merely a result of increased memorisation of training samples by the\nlarger models, or their improved generalisation?\" -> performance is measured on the test set, which is not trained on, so performance should always be independent of memorization. This might require some clarification.\n\n3. Typos:\n\np.3 \"We summarise more notions of memorisation in Table 1 (Appendix).\" <- Table 1 is on page 5, not in the appendix.\n\np. 3 You use an expectation over M to write the consistency score - what is p(M)? This seems wrong or unclear.\n\np. 5 \"A line of work Baldock et al. (2021); Maini et al. (2023); Stephenson et al. (2021)\" <- \\citep not \\citet?\n\np. 8. Please make Table 2 bigger! Also use percentages (xx.x% instead of 0.xxx).\n\np. 10: \"Why distillation reduces memorisation but improves generalisation?\" -> \"Why does distillation reduce memorisation but improve generalisation?\"\n\np. 11. Table 3: Also report percentages here please.", "broader_impact_concerns": "None - the appendix contains the broader impact statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TrDteEq2c1", "reviewer": "Reviewer_GZx9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n+ Comprehensive empirical analysis and novel insights into model size versus memorization dynamics.\n+ Identifies the nuanced behavior of knowledge distillation in reducing unwanted memorization.\n\nWeakness\n\n- the paper covers a very wide range of topics, but unfortunately didn't cover in-depth of the topics. For example, the discussion can be more interesting if these are presented with wider range of datasets and architectures. \n- It's not so clear that how does these affected by initialization and hyperprameters. For example, will Figure 1 be different if we run with different initializations and hyperparameters?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates the impact of model complexity on memorization and generalization in image classifiers, demonstrating that larger models exhibit more bi-modal memorization behaviors, particularly towards ambiguous and mislabeled examples. It highlights how knowledge distillation can mitigate memorization, suggesting its role in enhancing generalization", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. please run the models with different initializations and hyperparameters, and see if we can observe similiar behaviors of these initializations. \n\n2. It's unclear why we choose ResNet for cifar100 but MobileNet for ImageNet. It does not seem to me that MobileNet is a standard architecture for ImageNet (or as popular as ResNet). Are there other reasons (other than computing) that lead to this? \n\n3. Overall, this paper is very dense in many topics, it will be better to focus on less topics and discuss more. For example, out-of-sample accuracy does not seem to ever get defined.", "broader_impact_concerns": "none noted.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1BNgxzPyYm", "reviewer": "Reviewer_Xc8C", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\nOne of the paper's strengths is its comprehensive empirical analysis. The authors conduct numerous analyses, with many results presented in the appendix. Technically, the results appear novel and significant. As the authors note, the empirical study of how memorization varies with model scale has been understudied. The paper is well-written, and the visuals are effective, especially with the use of clear paragraph headings.\n\n**Weaknesses**\nThe paper primarily focuses on the ResNet family, and it would be beneficial to include more experiments with other settings, such as the MobileNet architecture on ImageNet. Additionally, the authors only explore one dimension of increasing model scale at a time; it would be interesting to compare the effects of increasing width versus depth, for instance. Finally, the authors make an intriguing finding that distillation can reduce memorization; it would be valuable to discuss in the conclusion how this finding can be practically applied.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper makes important contributions to the study of memorization and generalization across image classifier size scales. First, the paper demonstrates that the memorizability of individual points across model scales follows one of a few trajectories: fixed, increasing, decreasing, or cap-shaped. The fixed trajectories correspond to easy points, while other trajectories correspond to harder points. Also, the paper demonstrates that distillation can decrease memorization and improve generalization. The paper also finds that different measures of memorization have different trends.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Woud strengthen**\n\nConsider Additional Experiments on Settings Outside ResNet on CIFAR-100\n\nCompare the Effects of Increasing Model Scale Along Different Dimensions\n\nDiscuss Practical Implications of Distillation-Related Findings", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an empirical study attempting to answer the question: do larger models memorize more? The authors find that, unexpectedly, increasing model capacity results in a more bimodal distribution on memorization, that this is not always detected by memorization metrics, and that distillation helps avoid memorization. All these findings will be of interest to ML audiences, and since reviewers all agree the authors present enough evidence to support these claims, I recommend acceptance.
Decision: Accept (poster)