Memorisation in Machine Learning: A Survey of Results
Authors: Dmitrii Usynin
Moritz Knolle
Georgios Kaissis
Venue: Accepted by TMLR ()

Abstract: Quantifying the impact of individual data samples on machine learning models is an open research problem.
This is particularly relevant when complex and high-dimensional relationships have to be learned from a limited sample of the data generating distribution, such as in deep learning.
It was previously shown that, in these cases, models rely not only on extracting patterns which are helpful for generalisation, but also seem to be required to incorporate some of the training data more or less as is, in a process often termed memorisation.
This raises the question: if some memorisation is a requirement for effective learning, what are its privacy implications?
In this work we consider a broad range of previous definitions and perspectives on memorisation in ML, discuss their interplay with model generalisation and their implications of these phenomena on data privacy.
We then propose a framework to reason over what memorisation means in the context of ML training under the prism of individual sample's influence on the model.
Moreover, we systematise methods allowing practitioners to detect the occurrence of memorisation or quantify it and contextualise our findings in a broad range of ML learning settings.
Finally, we discuss memorisation in the context of privacy attacks, differential privacy and adversarial actors.

Published in Transactions on Machine Learning Research (08/2024)
Memorisation in Machine Learning: A Survey of Results
Dmitrii Usynin
du216 ac.uk
Department of Computing, Imperial College London Institute for AT in Medicine, Technical University of Munich
Moritz Knolle
m.knoole-@tmm.ee
Institute for AI in Medicine, Technical University of Munich Konrad Zuse School for Excellence in Reliable AI
Georgios Kaissis
g. g.knissis@tum.de
Institute for AT in Medicine, Technical University of Munich, Institute for Machine Learning in Biomedical Imaging, Helmholtz Munich
Reviewed on OpenReview: hhttps: opwnrevvew......... net/ forum? id= HVWODDbb
Abstract
Quantifying the impact of individual data samples machine learning models is open research problem. This is particularly relevant when complex and high-dimensional relation- ships have to be learned from limited sample of the data generating distribution, such as in deep learning. was previously shown that, in these cases, models rely not only on extract- ing patterns which are helpful for generalisation, but also seem to be required to incorporate some of the training data more or less as is, in process often tormed memorisation. This raises the question: if some memorisation is a requirement for effective learning, what are its privacy implications? In this work we consider a broad range of previous definitions and perspectives memorisation in ML, discuss their interplay with model generalisation and their implications of these phenomena on data privacy. We then propose a framework to reason over what memorisation means in the context of MT. training under the prism of individual sample's influence on the model. Morcover, we systematisc methods allowing practitioners to delect the occurrence of memorisation or quantify it and uallex our findings in a broad range of ML learning settings. Finally, we discuss memorisation in the context of privacy attacks, differential privacy and adversarial actors.
1 Introduction
Machine learning (ML) models require access to large amounts of high-quality, diverse and well-curated data to perform well in variety of tasks ranging from biomedical imaging (Seo et al., 2020; Rueckert & Schnabel, 2019) to text generation using large language models (LLMs) (Nijkamp et al., 2022; Taylor et al., 2022; Alayrac et al., 2022). Such data is often sensitive nature, mandating that its disclosure be avoided. However, with the advent of generative ML, it has become apparent that models often reproduce samples from their training datascts almost verbatim (Ippolito al.; Carlini al., 2019b; 2021), thus posing potential privacy risks for data owners. This phenomenon is not new to the ML privacy community, it has been techniques such differential privacy (DP)
known that models trained without the use of privacy-enhancee are prone to inferences about their training data, such as membership inference (MIA) (Shokri et al., 2017) or data reconstruction attacks (Zhu et al., 2019). However, these observations in generative models have led to a recent spike in research trying to uncover (1) whether and to what exlent models actually contain their training data (to then regurgitate it) and (2) which training samples are more prone to this.
The aforementioned phenomenon is (often informally) termed training data memorisation. Memorisation is often viewed (and taught)) as being the opposite pole of generalisation (that is, committing data samples to

---

Published in Transactions on Machine Learning Research (08/2024)
storage rather than learning general patterns from the training data). Obviously, this distinction is somewhat artificial, as it is all but impossible to delineate where general patterns end and the storage of actual data samples begins. Beyond this fact, it has empirically been shown that while models are able to learn data reprosentations which are useful to the task at hand- they are also entirely capable of fitting random imput- output associations such as purely random labels in the context of supervised learning (Zhang et al., 2017). In addition, while supervised MI. models are usually able perfectly fit their training data, this often does not translate to a commensurate generalisation performance on unseen data.
Additionally, certain types of ML models, such support vector machines k-nearest neighbours, exhibit learning process which is in its entirety based on storing data samples followed by a subsequent look-up process. It thus becomes evident that the delineation between memorisation and generalisation may in fact- be moot. To the contrary, it appears that both, memorisation and generalisation, are crucial to the learning process of ML. models. Prior works lend credence to this fact. An early work by Chatterjee (2018) demonstrates that network of support-limitd lookup tables can memorisc enough patterns to be able generalise to those it has never previously seen. Additionally, Brown et al. (2021) find that specifie tasks can only be successfully learned through memorisation of portions of the training data. Moreover, the seminal works of Feldman & Zhang (2020); Feldman (2020); Zhang et al. (2021a) demonstrate that memorisation and generalisation not only co-occur, but that the former is actually prerequisite for the latter if one aims to train a model of (close-to) optimal utility. Several prior works also equate memorisation to overfitting, the phenomenon where, during training, the model's performance on unseen data starts deteriorating after initially having increased, pointing to an acr-aoccomooaaioon of the model to its training data.
These introductory remarks reveal that there is not only lack of terminological clarity when discussing memorisation and generalisation in ML, but also that there exist a multitude of partially conflicting definitions. We thus identify requirement for a systematisation of prior works describing these phenomena in various sub-domains of ML. Moreover, we contend that such a systematisation must also tackle specific related points of importance:
Piipointing which samples are (more) prone to memorisation;
Quantifying, rather than merely detecting the presence of memorisation;
Identifying the implications of memorisation terms of data privacy;
Discussing techniques aimed at preventing or diminishing memorisation and their consequences on the model's performance.
Such a work will not only aid in the disambiguation of the multiple existing definitions of memorisation, but also assist practitioners in choosing the techniques to identify, measure and possibly reduce memorisation in their operational scenario or use-case. We provide a summary of all techniques discussed in this work in Table 1.
In this paper, we attempt the aforementioned systematisation. Our work is structured as follows:
We outline a formalised definition of memorisation in machine learning and discuss how it can quantified directly (Sections 2, and 4.1);
In addition, we discuss notions which are ostensibly related to (but otherwise disjoint from) memo- risation (Section 5);
We then discuss how, when and in which part of the learning protocol does memorisation get induced (Sections 6 and 7.1);
We discuss the implications of memorisation the privacy of the individuals whose data is used to Irain the model (Sections 8.1 and 9).
Finally, we provide guidelines for ML practitioners and outline promising future work directions (Sections 10 and 11).
2

---

Published in Transactions on Machine Learning Research (08/2024)
As ML encapsulates large variety of learning tasks, modalities and domains, discuss the implications these may have under the prism of memorisation. Specifically, we discuss the implication of memorisation in different modalities (imaging and textual) in Sections 8.1 and 8.2; learning settings (discriminative vs generative) in Section 8.2 and data regimes (centralised and distributed) in Appendix A. We additionally provide some practical guidelines on context-specific memorisation for the practitioners to have a better understanding of what memorisation can entail when working with sensitive data in Section 10.
Tidaiiuos
Socias
Chilleage
Mensrro
Earagla
Bestraiinn avtlads
41
Poor scalng vdta datuett abor
Voe
Pedtuan Slunng (25a0), Suang et al. (A121a)
Inforee froeiios
Ununtabble for dee kersing
sealase avdd
Pariiaty !
Kosk (1917), al, Grose (5053), Nounests (5003)
techiius
Dats valtaries
Obis was seareal acnee sctlings
Pourr erating with datort else
Partialy
Cluebau Zou (3005); Hummoookh Lond (2022))
Cesl unthoca
51
Poor wallng with the wode eie
Modeh and data depesiezzz
No
Chenoe al (20000)) Zhe (M089) 12 es al, (101)) (long & Rey (3033a), Muxlee es (1077) Agonoal et all (1923)
Ctyrus saiose plesonssa al saze
Unisuctive
No
(2az2);
(aabe]; Goultiei Geocareadi (2023); (aeat)
Choer inferstion Tou, netrecittion
Strgle dity
53
Modek dat- und indeeipaaai
Deserite acenal neated panmorona oace
No
Choa (2102);; Balriodk (1001): Gong Boy (aants)) Carlast en al. (2810)))
Obes was foreooio senen seilings
Can afret the tial mode
Ialurd mouoiatii
T1. 83
Mudek datt und indecippasion
Can tupses unceabetic continprioee
Partiaty*
>
Carliei it dl (90199)) Thomsl al (2000y, Thikkkr al, (2000) Hartey k: Twittrls (1009), Carlii et (N71))
-iiia nessum loskago, sut ancuoraation
Table 1: Summary of techniques used to identify data points which are more likely to be memorised. (1) While influence functions can be used to calculate sell-influence, these rely on assumptions which are often not satisfied in deep learning (e.g. strong convexity); (2) Shapley values were previously described in Zhang et al. (2021a) as similar to calculation of self-influence; (3) - -These can be used to measure the worst-case unintended memorisation, which frequently encapsulates memorisation, extractability and the ability of the model to leak data. Distinguishing between these three notions is a non-trivial task.
2 Preliminaries
Most ML models can be viewed as parameterised functions fa("), which, given some input x and parameter values 0, produce a corresponding output fo(x). Using a training dataset S ~ DD where 'D is the data- generating distribution, many ML algorithms can be reduced to finding an optimal parameter setting 0 (and the corresponding function fo-) ) that minimises a loss function L over S:
0" = arg min Σ I(Jo(T.).v).
(1)
which is often called empirical risk minimisation (ERM). Based on the form that S takes, we now broadly categorize MI. methods into: (I) supervised learning, where the training dataset S contains both inputs x as well as ground-truth outputs/labels y and (II) generalive Garringgmodellie example outputs are available, but we wish to learn the general structure of x. We use the terms supercised and discriminative interchangeably in this work.
where no corresponding
For the rest of this paper, will omit 0 (unless explicitly necessary) and denote the result of this procedure (commonly referred to as model training/fitting) as applying training algorithm to the dataset as f 1- (-S()).
Finally, we frame the goal of ML as obtaining model ʃ that generalises unseen data. This is measured by the generalisation performancc/error, which for supervised learning is defined as follows:
errgen (f) = E (x,y)~D
_ [L(((x)
For the generative learning setting, L takes only one input, as S contains no labels y. Note that formally, generalisation performance must be measured over the entirety of the data-generating distribution D. In practice, it can only be estimated as an empirical average over (finite) test set. Accurate estimation of errgen
3

---

Published in Transactions on Machine Learning Research (08/2024)
thus hinges the choice of representative test set that is distinct from the training dataset. Throughout, we will use the terms somple, inslance and data point synonymously.
3 A formal definition of memorisation
For long time, memorisation lacked precise definition and the term was commonly used loosely to refer to variety of phenomena. In this section, discuss the definition of Feldman (2020), who presented the first unified formulation and theory of memorisation in ML. We note that in this work we primarily concentrate on the underlying intuition behind the definition of Feldman (2020), rather than its specific operationalisation. While various other definitions have previously been proposed (and we discuss them in great detail in the following sections), we identify that the one proposed in Feldman (2020), which is based on the notion of influence as the only one which quantifies the phenomenon of memorisation itself, rather than measuring some related phenomena often associated with samples which can be memorised. We additionally note that due to the choice of the influenco-based formulation, this definition is also a) modality- b) training setting- and c) model-agnostic, making it an attraclive generic formulation of memorisation. This is primarily because the notion of influence can be easily extended to any learning scenario allowing the user to select which metric is used to quantify how the presence or absence of training point affects the resulting utility (e.g. per-sample accuracy in classification or log- perpeeelexyty in language modelling). In contrast, most of the previously proposed definitions which we discuss in this manuscript are either data- or model-specific (e-g. quantifying memorisation via canaries); only apply to specific training settings (e.g. fitting of random labels) capture different, albeit related phenomenon (e.g. memorisation through overfitting). Fundamentally, this definition is also very intuitive: if we (i.e. humans) memorise sample, are expected to be able to make a more accurate prediction on it compared to setting where we have never seen this exact sample before and only extrapolate from our knowledge of similar-looking data. Methods which do not directly measure memorisation (but, instead, certain factors leading to it) all seem to leverage this fact: the samples which fall under this behaviour are distinct and individual, (or informally "special"). This unifies the previous work on these metrics (which we later describe as methods that ostcnsibly mcasure memorisation) under this common thread. This makes Feldman (2020) formulation of memorisation a very attractive definition to describe the phenomenon of memorisation (with an intuitive method to quantify it regardless of the learning setting).
In the framework of Feldman (developed further in Feldman & Zhang (2020); Zhang et al. (2021a)), memo- risation framed as the impact a particular sample has on its own prediction (known as the self-influence). Formally, it is defined as the difference in (expected) performance on the sample (with index) when sample is included in the training dataset S:
mem mem(A,S, (A
E
[ (f())))
E (+S(W
(M((.))
(3)
(+S(S)
performance on i when E performance on when i g
Here, the expectation is taken over the randomness of the training algorithm A. M refers to some suitable performance metric, (e.g. accuracy) and S% is the training dataset with sample i removed. Note that Eq. (3) is context agnostic and can be applied to a generative settings by picking a performance metric M that only takes a single input argument (i.c. without including the ground truth labels). The algorithm here represents the training process of the model, during which memorisation occurs, but it is ultimately the model itself that. memorises. Note that a naive calculation of Eq. (3) is computationally expensive. Efficient sub-sampling estimators were proposed and are discussed in Section 4.
Key Point
Memorisation is formally defined as the inffuence a sample has on its own (correct) prediction (self- influence).
4

---

Published in Transactions on Machine Learning Research (08/2024)
Head
10-1
Tail
Self-Influence
Density
A
B
10-2
10-3
10-4
typical
atypical
max Cross-Influence
(a) long-tailed data distribution
(b) comorisationiinfueeee continuum
Figure 1: Long-tailed distributions have probability density that decays very slowly for extreme (atypical) values (a). It can thus produce samples that are so distinet from typical samples that they are required to be memorised to allow for generalisation to other similar samples. Sub-figure (b) shows schematic visualisation of the iemnorisation-infuenne continuum. While all samples from the tail of the data distribution are likely memorised during training (large self-inftuence), useless, low-quality mislabeled samples (A) are indistinguishable from useful samples that significantly influence the correct prediction of least one test sample (large max cross-influence (B).
3.1 Influence and the long-tail theory
In the same work Feldman (2020) proved that memorisation is required component of learning. This long-tail theory, contends that close-to-optimal generalisation performance on long-tailed data distributions (see Fig. Ia) necessitates the memorisation of rare and "atypical" samples from the tail of the distribution. This due to the fact that, in long-tailed distributions, samples from low-density regions (the tail) differ extremely from samples from high-density regions (the head). Thus, memorisation of these atypical samples is necessary to generalise optimally to other atypical samples at test time.
The long-tail theory (developed further in Feldman & Zhang (2020); Brown et al. (2021)), was supported by prior observations that naturally occurring data distributions commonly have long tails (Babbar & Schölkopf, 2019; Zhu et al., 2014; Van Horn & Perona, 2017: Yang et al., 2022). Further empirical evidence in direct support of this theory was presented in Feldman & Zhang (2020); Zhang et al. (2021a), which demonstrated that (for commonly used large-scale vision and text datasets), a substantial proportion of training examples have an out-sized impact on the model's performance on few specific test examples. The (sample-level) mcasure used demonstrate this phenomenon was termed cross-inftuence, that is, the impact training sample v; has on the prediction of test sample z/z
inf((A, = E (+-((S) (M(((())))) v
E 1 (
(((())))))))
(4)
performance when E performance when i
whereby self-influence (mem) is recovered when i= and M is some suitable performance metric that returns higher values for better predictions. Thus, a positive infl value indicates that training sample improves the prediction on lest. sample j when E S. M taken to be a risk/loss function, equivalent behaviour can achieved by simply negating all infl values.
Fig. 1b visualises the memorisation-infuune continuum. The continuum illustrates that while all samples from the tail of the distribution are likely to be memorised (and thus exhibit high self-influence), not all
5

---

Published in Transactions on Machine Learning Research (08/2024)
of them have significant impact at least test sample (A vs. B). Further, direct consequence of the long-tailed nature of data distributions is the fact that, at training time, low-quality mislabelled samples (A) are statistically indistinguishable from useful representative samples of rare sub-populations (B) Feldman (2020).
Key Point
When a data distribution is long-lailed, memorising samples from the tail of the distribution can help model generalise.
3.2
Prior efforts to capture memorisation
There have previously been many attempts to quantify memorisation in ML. However, we stress that while many of these phenomena have previously been labelled as "memorisation" or claim to identify samples which are "memorised" by the model, none of these can be used as a formal and modality-agnosse
definition of memorisation. In this section we outline some of the most well-known phenomena, which, while related memorisation are not suited quantify this notion. We outline additional methods which have previously been used to quantify memorisation in Table 1.
Memorisation as overfitting A number of prior works (Olatunji et al., 2021; Chen et al., 2020a; Kuppa et al., 2021; Leino & Fredrikson, 2020; Veale et al., 2018; Hilprecht et al., 2019; Mehta et al., 2020) draw direct connections between memorisation and overfitting, defining memorisation a given model as its generalisation gap (i.e. the empirical measure overfitting). Formally, the generalisation gap model f is defined as:
err errgap(]] (f) =
erigen (
L((()))))
(5)
generalisation orror
empirical (training) error on S
This quantity is task-agnostic and it can be estimated using a representative test set, making it very attractive tool for empirical memorisation measurement. However, in the light of evidence from Feldman & Zhang (2020); Zhang et al. (2021a), showing that the memorisation specific training examples actually increases generalisation performance, this definition now appears dated. In addition, the measure above does not allow for the identification of individual memorised samples, being computed an average. Finally, recent works demonstrate that memorisation usually precedes overfitting (Carlini et al., 2019b; Tirumala et al., 2022).
Key Point
Memorisation cannot be reduced overfitting. Overfitting indicates lack of generalisation, whereas memorisation was shown improve generalisation.
Fitting random labels Zhang al. (2017) demonstrated that even relatively small neural networks (by today's standards) can perfectly fit large datasets with randomised labels or even completely random data. As the correct prediction of random label is impossible without memorisation, randomised labels are commonly used to study qualitative aspects of memorisation behaviour in ML models, such as whether it can be localised to specific regions in the model or when it temporally occurs during training, These topics are discussed in more detail below in Section 6.
However, as this formulation relies on the presence of labels, it can often be challenging quantify memori- sation settings where these are absent (e.g. generative modelling). We, thus, turn inffuence functions, which allow for such quantification to be performed efficiently in the following section.

---

Published in Transactions on Machine Learning Research (08/2024)
Key Point.
Many deep neural networks have sufficient memorisation capacity to fit. large datasets of completely random input-output associations.
4 Measuring memorisation through influence estimation
Influence analysis dates back to (Cook, 1977) and aims to offer data-centric approach to explaining model's predictions by apportioning credit (positive or negative) to individual training data samples. If applied to quantify the impact that a single training data point has on the prediction of a single test sample, it is often called point-wise inffuence and is equivalent to the quantity ccross-iffluene) introduced in Eq. (4).
This canonical definition of influence (commonly known as leave-one-out (LOO) influence) is computationally expensive. Its simple Monte Carlo cstimation requires Q(n) runs of training algorithm A to convergence. Fortunately, substantial research effort has gone into developing efficient methods for influence estimation, which will be discussed next.
As covered previously in Section 3, memorisation can be quantified through self-influence. Thus, all methods discussed in this section can be used as valid approaches to estimate memorisation (if applied correctly to measure a sample's influence on its own prediction). We omit a comprehensive discussion of this topic and refer to Hammoudeh & Lowd (2022b) for an extensive survey on influence and its estimation.
4.1 Efficient influence estimation
Estimation of influence can be broadly categorised into re-training-basee methods and influence functions.
de-training-baser methods approximate Eq. (4) directly through simple Monte Carlo sampling. To circumvent the aforementioned computational complexity, Feldman & Zhang (2020) proposed sub-sampling estimalor. This method simultaneously leaves out multiple training samples randomly (instead of just one) and records each sample's membership and the result, which are subsequently averaged to obtain a smoothed estimate of a sample's influence. Correspondingly, the number of required training runs is drastically reduced, as every run of Algorithm A now contributes the influence computation of many samples, rather than just one. Authors showed that this estimator is statistically consistent and that its error is bounded (with high probability) by the sub-sampling rate and the number of repetitions per randomly sampled subsct (Feldman & Zhang, 2020). In practice, this method allows for accurate influence estimation (self/cross: cross--sssn)) in multiple hundreds or a few thousand runs on large datasets (Feldman & ZZhnng, 2020; Zhang et al., 2021a). A (somewhat) similar approach was proposed by Jiang et al. (2020), termed the per-sample generalisation, measured using a consistency score, which estimates the expected accuracy of the model on a single sample, by comparing the performances of models trained on sets of varying sizes sampled from the data distributiin. This approach, however, also suffers from computational complexity and a number of proxy metrics, such the pairwise sample distance were proposed to approximate this value.
Influence functions were first. ported from the domain of robust statistics to deep learning through the seminal work by Koh & Liang (2017). Using a classical result from robust statistics (Cook & Weisberg, 1982), Koh and Liang showed that a change in loss (due to removal of a sample) can be approximated by the inverse empirical Hessian (matrix of second-order derivatives w.r.t model parameters). This means that no model ro-training is required. Intuitively, through second-order Taylor series expansion, the effect of a small perturbation (assumed to be induced by removing a single sample) on the loss function can be approximated. Hessian matrices are infeasible to compute for large models, requiring memory which is square in the number of network parameters. However, efficient approximation techniques have been proposed to tackle this issue (Guo et al., 2020; Schioppa et al., 2022), allowing for the application of influence functions even to large language models (Grosse et al., 2023; Kounavis et al., 2023).
Re-training-basod approaches, given enough computational resources, are able to provide reasonably accurate inffuence estimates, even for large datasets such as ImageNet (Feldman & Zhang, 2020). On the other hand, influence functions have received substantial criticism (Bae et. al., 2022; Basu et al., 2020; Schioppa et al.,
7

---

Published in Transactions on Machine Learning Research (08/2024)
2023), as key assumptions of the underlying theory, e.g. strong convexity and positive definiteness of the Hessian (amongst others), are not satisfied in the context of deep neural networks. Furthermore, multiple works have shown that influence function values correlate poorly with LOO influence values (Bae et al., 2022; Basu et al., 2020), an effect that seems to be aggravated with network size and depth. Thus, based on this evidence, we deem re-training-based methods the preferred choice for estimating memorisation the available computational resources allow for it.
An alternative approach to the two above-mentioned methods relies using the so-called "representer the- orem", allowing the model owner to decompose the prediction into individual contributions (i.e. influences) from cach data point (Ych et al., 2018). This method has however been referred to as "overly reductive" (Yeh et al., 2022) in a follow-up work by the same authors as it's limited to detecting changes in the final layer only.
Key Point.
Memorisation can be measured efficiently through influence estimation techniques, but care is required to make sure such estimates are accurate.
4.2 Connections to data valuation
Data valuation is concerned with assigning value data points to determine equitable monetary compen- sation for data owners. It is closely connected to the concept of influence i.e. when influence is applied to quantify the impact of single training sample many test samples. In fact, the notion of the aforemen- tioned LOO influence is also frequently used in data valuation. Note also that, any data valuation measure can be used as a point-wise influence measure (Hammoudeh & Lowd, 2022a) which means that conversely any data valuation measure can be used to mcasure self-influence. As influence measures were not originally designed be equitable, substantial effort has gone into the game-theorelic analysis and improvement of data valuation /influence measures which satisfy a number of axioms aimed at improving data valuation equitability.
Arguably, the most popular valuation metric is the Shapley value (Ghorbani & Zou, 2019) which quantifies the expected marginal contribution of individual data points while considering all possible subsets of the datasot and their interactions. In many cases, the data samples which are "valued higher" are often the same ones that are more likely to be memorised (Zhang et al., 2021b). While such improved (re-training- based) data valuation iotnattionnifuuencee estimation methods have clear advantage for equitable data owner compensation, the relevance of such unclear. Concretely, it has been argued by Zhang et al. (2021a), that LOO influence (which does not model data point interactions). is to be regarded the preferred metric for measuring memorisation since it allows for making causal statements about the memorisation of data point. Further, the authors argue that it's the insensitivity of LOO influence to data duplicates (a behaviour previously criticised in the context of data valuation (Ghorbani & Zou, 2019)) which renders it a suitable metric for measuring the memorisation of rare data points. As a result, while there are some clear overlaps between the domains of memorisation and data valuation, there are still many open questions (particularly on the relationship of the valuation metrics, which often concentrate memorisation) making the use of data valuation techniques for quantification of memorisation problematic.
motivated axioms for the quantification of memorisation remains
the cross-influence and the self-influence discussed in the context of
Key Point
Game-theoretic data valuation metrics assign higher values to samples which often more likely to be memorised, but are not a direct proxy for memorisation.

---

Published in Transactions on Machine Learning Research (08/2024)
5 Related metrics incorrectly described as memorisation
Beyond the techniques which can be used to directly estimate memorisation introduced in Section 3 (i.e. through self-inftuence), there cxists a large number of methods which attempt to rank training samples basod on their "importance" to the model. Here the term importance encapsulates concepts such as difficulty of fitting, error/ error/gradie gradieen magnitude, etc.. While, ostensibly, some of these metrics are associated with memorisation (and are often described as its computationally incxpensive approximations), we note that none of them estimate self-influence directly. We, therefore, postulate that none of the metrics discussed below can directly quantify memorisation. We note, however, that some of the metrics from these various domains often correlate with how much memorisation individual data points can experience, these cannot be used as direct "proxies" of this phenomenon.
5.1 Gradient-based influence proxies
The methods introduced in this section leverage the gradient of the loss function evaluated at individual samples, either with respect to the model weights or to the inputs themselves. Recall that the gradient is a lincar/firstordde approximation to the effect of a sample on the loss. The main benefit of the techniques discussed below is that they are easy to implement and computationally efficient.
Gradients w.r.t. model parameters Arguably, the most widely used gradient-related metric is its magnitude, i.c. the fg-norm with respect to the model's parameters (Chen et al., 2020b; Amiri et al., 2021; Lai et. al., 2021). Some works claim that the gradient norms of individual data points are proxies for memorisation (as, similarly to loss values, they tend to decrease over the course of training, highlighting that samples get "gradually" memorised) (Zhu et al., 2022; Katharopoulos Fleuret, 2018; Li et al., 2021; Xue et al., 2021). However, while higher gradient norms (similarly to loss values) are usually associated with difficulty when making predictions on individual data points (often correlated with samples of higher influence), this is not a direct representation of memorisation (Zhu et al., 2022; Li et al., 2021). Moreover, even when viewing these metries through the lens of adversarial susceptibility, there is no clear causal relationship between the amount of information memorised about individual samples; information contained in these samples; and the information exposed by the model. The gradient norm is also central to differentially private stochastic gradient descent (Abadi et al., 2016) and especially in approaches employing individual privacy accounting (Feldman & Zrnic, 2021: Yu et al., 2023; Koskela et al., 2022), where the gradient norm is proportional to the individual's 's privacy loss. Despite that fact, and while larger gradient norms can be associated with higher attack susceptibility, this is not always observed, implying that other factors may be (wolved(Usynin et al., 2023a; Geiping et al., 2020; Balle et al., 2022).
Gradients w.r.t. inputs Orthogonal to the aforementioned methods are techniques which utilise gradients with respect to the inputs. The most notable technique in this category is the variance of gradients by Agarwal et. al. (2022), which combines tracing the gradient values with respect to the inputs during training with the computation of their variance over the time axis. A higher variance is claimed to imply that the sample is more atypical and is, hence, more difficult to learn. Thus, the authors state that samples with high variance of gradients are also more prone to memorisation. A distinct line of work (Mueller et al., 2022; Hannun et al., 2021) from the domain of privacy-preservin MI. also used gradients (or second-order derivatives) with respect to inputs to measure privacy loss. These can be used to identify samples which have "more revealing" input features and be more susceptible to attacks on privacy, allowing the data owner to identify the privacy risks associated with individual training records.
We note that, techniques based on the second-order derivatives, which estimate the curvature of the loss function with respect to individual inputs (instead of weights) have also been used to measure memorisa- tion (Garg& & Roy, 2023a). Moreover, more computationally efficient techniques not requiring second-order dcrivatives have been developed, such as techniques which trace the cvolution of the loss function's 's value or the (first order) gradient Uhrough training (Pruthi et al., 2020; Hammoudeh & Lowd, 2022a). These metrics can also be used to either A) identify samples which are more challenging for the model to learn from or B) be used to approximate the aforementioned influence functions (which are, in turn, often used to quantify memorisation). While these are conceptually similar to the efficient influence estimations proposed in Koh & Liang (2017); the subtle difference between the two concerns which type of curvature the authors discuss.
9

---

Published in Transactions on Machine Learning Research (08/2024)
For inftuence estimation, the derivatives are taken with respect to the model weights, compared to methods such as (Garg & Roy, 2023a) (or a first-order approximation proposed in Agarwal et al. (2022)), where the derivatives are taken with respect to the model inputs).
Overall, many properties of training data which can be efficiently represented through the gradients (e.g. the privacy loss) are often correlated to the magnitude of memorisation of that training sample. Therefore many of the samples that possess e-g. higher gradient norms are often those that are also more likely to be memorised. However, these links are mostly correlational as to-date there has not been any empirical theoretical evidence to show that there is a causal relationship between the characteristics of the gradient and the memorisation of individual samples. Hence, we argue that the gradient-derived metrics cannot be used as direct approximations of memorisation. Several works point to a connection between neural network geometry and memorisation, which we thus regard as a promising area for future research (Stephenson et al.; 2021; Garg & Roy, 2023a; Ravikumar et al., 2024).
Key Point
Gradient-basck metrics cannot be used to directly quantify memorisation, but often (efficiently) measure related quantities (e.g. sample difficulty).
5.2 Quantifying memorisation using information theory
The next set of techniques we discuss quantify the "flow of information" from data points to the parameters of model using the tools of information theory. subset of these works rely on the concept of Shannon mutual information (MT) (Kolmogorov, 1956), which quantifies the amount of "information" which can be deduced about a variable by monitoring another variable (i.e. the amount of "dependence" between the variables) (Shwartz-Ziv, 2022; Goldfeld et al., 2018). However, while such methods have strong theoretical foundations, they ultimately sulfer when placed in the context of ML. This is because these approaches assume that all inputs of interest including the resulting ML model's weights are random variables, which is often not the case, as most ML approaches result only in a single set of weights. Moreover, Shannon information assumes that computational abilities are generally unbounded. This means that, in the view of Shannon information theory, information cannot be increased by further processing. This stands in contrast to real-world practice, in which computation is constrained by cost or by the available time, and information is thought to be "extracted" from data by the ML model.
This motivated the development of usable information theory and its variant of information termed V- information (or V-usable information) (Xu et al., 2020). This mothod can be seen as a computationally constrained version of Shannon MI, which measures the "usable" information contained in the data, which can be extracted by functions (e.g. ML models) in a specific family (denoted V). To mitigate the aforementioned issue of non-randomness, Xu et al. (2020) proposed to use the entropy of softmax function (which is in itself distribution). This interpretation has been applied to quantify the difficulty of datasets (Ethayarajh et al., 2022) and to localise information leakage to specific components (layers) of ML. models (Mo et al., 2021). The authors of Mo et al. (2021) also correlated V-information with Jacobian sensitivity, which corresponds to the norm of the gradient with respect to the model's input, linking this study to the techniques discussed above. They deduce that there are two "types" of useful information the model can memorise: input and latent. The former represents the modcl's ability to correctly recall the input data under reconstruction atlacks (Zhao et al., 2019), while the latter corresponds to information about properties of the data (which can be exposed through attribute inference attacks, discussed below).
Orthogonal to the domain of usable information theory are approaches which attempt to enable the com- putation of information--heoreiicc quantities in high--imeensional ML models. Notable among these is the technique of estimating e.g. MI through randomised low-dimensional projections, pioneered in Goldfeld & Greenewald (2021) and termed sliced MI. Slicod MI was used in Wongso al. (2023) to mcasure memorisa- tion and captures the notion of average usable information (as averages the MI for all random projections) instead of the largest usable information. We contend that the relationship between the difficulty of learning,
10

---

Published in Transactions on Machine Learning Research (08/2024)
the complexity of the representations at hand and the memorisation properties of the network are still nascent area of research of high potential impact.
Another noteworthy approach was proposed by Harutyunyan et al. (2021) and is termed smooth unique information. This formulation considers the distinction between what data the weights of the model contain against what the model actually learns. Similarly to influence functions, authors consider how a single included/exclude instance can influence the training of the model and measure it using a smoothed KL- divergence between the two models (concretely: its expectation over the label distribution). Authors conclude that removing highly informative samples identified by their method results in stronger performance degradation compared to removing the same quantity of uninformative//radoon findings again highlight the divide between studies on informativeness of individual samples and the extent to which they are prone to memorisation. While this work studies the effect of including or excluding highly informative (in a manner of speaking influential) samples on generalisation performance, no analysis is performed in terms of the influence of training samples on themsclves.
samples instead. These
Key Point
Techniques from the domain of information theory can identify the samples which are likely to be memorised by considcring the amount of information that those samples contain (and, hence, the information learnt by the model). These approaches can, however, sulfer from unrealistic assumptions and poor computational performance.
5.3 Measuring sample difficulty
The concept of sample difficulty is often closely related to memorisation (e.g. "difficult" samples can be more prone to memorisation), but is arguably ill-defined. One can intuitively interpret difficult sample to be one which is fit poorly by the model. However, the reasons for such poor performance can vary: the sample can contain "rare" features (Agarwal et. al., 2022), have poor quality (Usynin et al., 2023b) or the sample can come from a data-generating distribution which is mismatched to the task at hand (Chen et al., 2021). A number of methods were proposed to identify at which point different "concepts" are learned from individual features of sample, showing that those that correspond to "easier" concepts are identified much earlier in the model (Baldock et al., 2021). Alternatively, the authors of Garg & Roy (2023b) rely the curvature of the loss function to assess how "clean" individual samples are (where higher cleanness significs stronger representative of a class). However, similarly to other works on example difficully, the notions of "cleanness" are closely intertwined with samples being "typical" representatives, which can have different interpretations based on the input modality (i.c. defining a typical cat image vs. a typical job description). Notably, this method through an approximation of the Hessian trace produces consistent results across different architectures and random seeds (which was a limitation of similar approaches (Agarwal et al., 2022; Pruthi et al., 2020)).
Arguably, the best example of the challenges associated with the interpretation of "sample difficulty" is discussed in Carlini et al. (2019a), as this work employs five separate metrics in order to identify the con- nections between individual image characteristies (for instance, adversarial robustness on these images) and its "difficulty". While authors conclude that most of these metrics are correlated amongst each other, there is still no cansal relationship between e.g. holdout accuracy and "difficulty" of sample. Another work of Harutyunyan al. (2021) shows that example difficulty can be estimated by observing the behaviour of the loss values on individual samples on their "corresponding" test-time points. What this definition proposes, is to disentangle concepts of train-tims performance and example difficulty, and to, instead, consider related data points, that the model has not encountered during the learning process.
Finally, another noteworthy work which tries to approximate example difficulty was proposed by Ethayarajh et al. (2022) using V-usable information (which a computationally constrained version of Shannon entropy). What we are interested in here is the similarity of this definition to the notion of sample influence. Here authors estimate the difficulty of a sample as model's predictive performance on two data distributions, one of which contains a record of interest and one does not. This metric describes the difficulty of an entire
11

---

Published in Transactions on Machine Learning Research (08/2024)
dataset, given model V, but authors also proposed pointwise V-information (PVI), which can be used to describe the difficulty of individual samples, similarly to the works we discussed above. One noteworthy conclusion of this work is that PVI, can serve as a context-agnostie measure of "difficulty" across different models and modalitics if using it as a threshold (i.e the PVI value after which samples start to be misclassified and are hence considered to be "difffcultt"). These methods establish an indirect connection between difficulty and memorisation: Difficult samples can often be those that fall on the tails of the data distribution and are, hence, more prone to memorisation. However, reasoning over sample difficulty is not straightforward, because while difficult samples may be highly self-inftuential (e.g. mislabelled data), they do not necessarily have to be (e.g. poorly acquired images).
To summarise: it is speculated by several works that the identification of "difficult" samples is similar to the identification of samples which are likely to be memorised. These methods can even be perceived as proxies to the aforementioned influence functions using only first-order information. While the definition of "sample difficulty" has not been unified yet across various works, most of the formulations that we discussed base their definitions of the concepts of "atypicality", making delection of such samples a challenging, yet important task across any input domain.
Key Point
Samples can be difficult for many reasons, but samples which are deemed to be more difficult are oflen memorised more.
Overall, we conclude that there exists a large number of formulations which are often described as "mem- orisation" of some form. We, however, highlight that while many of these measure quantities related to memorisation and identify samples which can often be more prone to memorisation, these cannot be used as direct measurements of this phenomenon.
6 Localisation and timing of memorisation
Two intuitive question which arises from the discussion above is:
Where does memorisation occur in the model?
2. When does memorisation occur during training?
To reason over the localisation of memorisation, we first need to consider the localisation of learning in ML models. One of the fundamental works presented by Bau et al. (2017) shows that individual concepts that are included in a data point (e.g- lexture or color of an object on an image) are learnt by different parts of neural network. Authors demonstrate that during the learning procedure, different parts of the model become "responsible" for learning individual concepts contained in the input. Thus, authors argue that it can be possible to identify individual neurons associated with specific concepts.
As learning of individual concepts can be localised to different parts of the model, it was previously established that this also holds for memorisation. It was shown in the works of Mo et al. (2021); Baldock et al. (2021); Maini et al. (2023) that as part of the learning process, memorisation can also be associated with specific parts of model. Specifically, authors of Baldock et al. (2021) present prediction depth, which the earliest layer in a model based on the representations of which it is possible to make a correct prediction on the input. Authors determined that "casier" concepts (i.e. those associated with generalisation) are learnt in the earlier layers, whereas the more complex ones are learnt in the last layers of the model. The work of Stephenson et al. (2021) further shows that memorisation of atypical (mostly mislabelled) samples can also be localised to the last layers of the model, where most mispredictions also occur in the early stages of training. It is of note that the work of Mo et al. (2021) shows a similar result through the lens of gradient information leakage, which we have previously discussed in Soction 5. The fact that memorisation can be localised was further supported by the work of Maini et al. (2023). However, authors of Maini et al. (2023) show that contrary to the results of Baldock et al. (2021), memorisation, while localised specific parts of
12

---

Published in Transactions on Machine Learning Research (08/2024)
the model, is not associated with the final layers. They instead show that memorisation is indeed localised to specific neurons, but these are often distributed across multiple layers. Moreover, the authors of Wongso et al. (2023) conclude that learning patterns can be different based on the architecture and the depth of the model: convolutional models exhibit more "Icarning" in the deeper convolutional layers, similar to the discussion above. In lurn, multi-layer perceplron-based models exhibit an increase in the amount of useful information the model learns approximately linearly. Additionally, Hernandez et al. (2022) previously shown that certain architecture-specifi parts of the model may even be "responsible" for memorisation altogether (in this case the attention heads). These findings highlight that learning (and memorisation) is not only inconsistent across different model types (i.e. different models can extract different information), but also that even individual layers in a model learn differently from the same data based on how deep they are (i.e. learning and memorisation can be localised to specific parts of the model).
In regards to the question of timing, authors of Arpit et al. (2017) show that memorisation is prevalent at specific stages of training. Concretely the authors showed that models tend to start learning "simple" patterns first, thus showing that generalisation can often occur before memorisation (supported by Kishida & Nakayama (2019)). Additionally, the work of Paul et al. (2021) shows that it is also possible to determine the influential samples carly on and use these to guide the training process, by removing samples of low influence from the training dataset. Therefore, it is also possible to not just link memorisation to the training phase, but to also employ the training data to force the model to change its memorisation pattern (i.e. which samples get memorised). This mirrors a finding by Baldock et al. (2021), who relate memorisation to prediction depth, i.c the depth of the layer at which representations which effectively determine the network's prediction on specific sample are formed. This highlights that a higher difficulty of forming defining representation in network is associated with a higher probability of memorisation. Another line of work has previously explored the issue of localisation of individual learning concepts (including memorisation and generalisation) relies on the tools from the domain of mechanistic interpretability (Pearce et al., 2023; Nanda et al., 2023; Hernandez et al., 2022). In Nanda et al. (2023) it was shown that using these tools it is possible to identify three distinct stages of ML training: memorisation, circuit formation (i.e. generalisation) and clean-up (i.e. replacing the memorised information with generalised patterns). This work cantradicts the findings of Arpit et al. (2017), arguing that memorisation occurs prior to generalisation, showing that while memorisation can often be temporarily localised, it is not yet possible to concretely determine when it takes place.
Overall, although there is no clear consensus regarding the specifies, evidence from multiple prior works supports that memorisation is a process that can be localised both spatially and temporally. We note, however, that while the aforementioned works have made significant progress in this field, there is, yet, no clear consensus on how and where attribute individual learning concepts (and, hence, memorisation) to.
Key Point.
Memorisation is a process that can be localised both spatially and temporally (e.g. at a specific point of training or the layer of the model).
7 Inducing deliberate memorisation
For many learning settings and in particular, for generative models, it is often very difficult to conceptualise what the model memorises (i.c. does producing an output "similar" to the training record count as mem- orisation?) One intuitive method of evaluating this involves deliberately inserting data samples which are "expected" to be memorised by the model due to their "atypicality". The capacity for memorisation is then measured with respect to how well the model can reproduce these inputs as its output (i.c. for generative models) or how performance on these data points compares to performance on the rest of the dataset (e.g. for supervised learning).
13

---

Published in Transactions on Machine Learning Research (08/2024)
Randomness space
173283949
173283948
Tested Sequence
Log-perplexity
173283947
"My secret number is:
173283946"
s|r.
"My secret number is:
173283948'
7.853
"My secret number is: 173283946"
7.930
173283945
S
173283944
"My secret number is: 173283943"
9.835
"My secret number is:
173283947"
10.85
173283943
(a)
(b)
Figure 2: Example workflow measuring memorisation through an inserted canary (shown red). First, canary is generated by combining some pre-defined structure and some randomness r drawn from some randomness space R (e-g all possible nine-digit numbers) (a). After inserting the canary into the training dataset, the ability of the model to reproduce the canary is computed determined by |/]] (the size of It) and the canary's index (rank) in table (b), ordered by log-perplexity and containing all other possible canaries from R (that could have been inserted but were not).
its erposure. This measure is
7.1 Canary memorisation
The, arguably, most widely used approach for inducing (and empirically quantifying) memorisation was proposed by Carlini et al. (2019b) and has since been employed in many machine learning contexts (Thakkar et al., 2020; Carlini et al., 2021; Tirumala al., 2022: Loe et al., 2021), particularly in the field of language processing. In essence, the authors propose to measure how much generative language model memorise by purposefully inserting crafted data samples (canaries) into the training dataset. These crafted samples are designed to require memorisation. The connection to the long-tail definition of Feldman (2020) is that such canaries are constructed to resemble data from low-density regions of the training data distribution. These samples are (often informally) called anomalous, outliers or atypical. Note that in supervised learning, a similar effect can be achieved through deliberately mislabelling the example. The "exposure" of the canaries is measured and used as a proxy for the memorisation of other alypical samples.
To define exposure, we first discuss the notion of log- oo-perplexityy it is based on. Intuitively, log-perplexity mcasures how "cxpected" a given sample is (or how much the model is "surprised" by this sample) and is defined as follows:
perp(æ, f) log2 p(m),.,,,..l.
(6)
loga (()
where f the model, and æ is the input sequence of length m. Here and below, we use p in slight abuse of notation to denote both probabílitics and likelihoods.
To generate canary (s(r)), combine predefined, fixed structure and some randomness r drawn from some predefined randomness space R. (e.g. the space of nine-digit numbers, see Fig. 2a). This generated canary is then inserted into the training datasct. After model training, we compare the perplexity of our inserted canary to all other possible canaries that we could have inserted. Perplexity, however, is a relative measure and highly dependent on the specific training setup, including model architecture, dataset composition and the application. Thus, Carlini et al. (2019b) proposed "relative" notion of perplexity called rank, i.c. the index of a canary the relative ordering of perplexities across all possible canaries (see Fig. 2b).
14

---

Published in Transactions on Machine Learning Research (08/2024)
Exposure, in turn, is then simply determined by the size of the chosen randomness space and the canary's rank. Formally it is defined as follows:
exposure log2
[R]
log, ranky(s(r)),,
(7)
where R is the size of the randomness space from which the canary s(r] was generated, thus making exposure posilive value.
The exposure definition has been successfully transferred to other machine learning modalities, such as computer vision in Hartley & Tsaftaris (2022), but due to the complexity of canary generation in other domains, most follow-up works in the area concentrated on language models (LMs) instead (Thakkar et al., 2020; Carlini et al., 2021; Tirumala et al., 2022). However, canary-based memorisation definitions are significantly less generalisable than the one discussed in Section 3, as they (1) require the data owner to generate canaries (which are specific to their setting) and (2) are particularly well-suited to language settings, compared to other generative modelling tasks.
Overall, canary-based approaches have seen popularity due to their simplicity of implementation and high applicability to generative ML tasks (Liang et al., 2023; Nijkamp et al., 2022; Taylor et al., 2022; Alayrac et al., 2022; Wei et al., 2022). One important finding that authors of Carlini et al. (2019b; 2022b; 2023)) report, is that exposure seems to increase drastically with the number of times a canary is seen during training. The authors thus suggest that if an alypical sample is included many times in the training datasel, it is much more likely to be memorised. This is in contrast to follow-up work Zhang et al. (2021a), which has argued that such a conclusion is ill-guided, as extraction extraction/ggneraaion-ba egeneration-baset memorisation quantification methods (like canaries) are biased towards identifying frequently occurring samples (Lee et al., 2021; Zhang et al., 2021a). In fact, Zhang et al. (2021a) presents contrary evidence to the prior claim: using re-training- based influence estimators (from Section 4.1) the authors show that memorisation estimates tend to actually decrease for data points that are repeated many times. This finding thus raises the question of the validity of memorisation and privacy risk conclusions drawn from extraction-based memorisation experiments, as the memorisation of froquently occurring sentences (or data points) potentially poses little privacy risk compared to rare sentences. This is because frequently occurring sentences such commonly known facts typically pose little privacy concern compared to e.g. personally identifiable information which only occurs once.
Key Point
Unintended memorisation can, in some cases, be measured by inserting samples into the training datasct which are crafted be more prone to memorisation and mcasuring how likely they are to be regurgitated by the model.
7.2 Memorisation and adversarial samples
The aforementioned canaries are deliberately crafted to resemble samples from the low-density region of the data distribution. They are, however, still valid (albeit atypical) data points and can be used to train a well-generalised model. The same concepts are exploited in works on data poisoning, where malicious actor artificially generates data points from the low-density region of the distribution, which are crafted to degrade the performance of the model. These data points produced during this process are known as adversarial samples (Goodfellow et al., 2014) and these were originally designed to showcase how tiny invisible input perturbations can be usod to cause an image classification model to mispredict, but were later adapted to other settings and modalities (Tian et al., 2022). These, unlike canaries, are applicable to both the discriminative and the generative settings (and are assumed to have high negative cross- inss-inftuence in addition to high self- self-influence.
In many cases, a very small proportion of such samples (< 1%) sulfices to severely harm the utility of the trained model (Chobola et al., 2023; Zhou et al., 2021). Adversarial data points are generated to be atypical through either incorrect labelling, embedding of features which are associated with a different class, addition of imperceptible noise etc. (Usynin et al., 2021). As result, these samples can often be used manipulate the behaviour of the model as these are A) more likely to be memorised and B) only require
15

---

Published in Transactions on Machine Learning Research (08/2024)
small perturbation, making these attacks difficult to detect. Adversarial samples also be used to aid attacks on MT. models, aiming to extract the information that the model has memorised (particularly for underrepresented samples, which were shown to be more vulnerable in Shokri et al. (2017)) about individual data points, which it would not expose otherwise (Tramèr et al., 2022; Carlini et al., 2022c; Chobola et al., 2023; Bagdasaryan & Shmatikov, 2021). For a more in-depth discussion on how these adversarial data can be generated in different learning settings, we refer to Tian al. (2022).
Key Point
Adversarial samples made to be artificially atypical and hence have higher influence the model (malicious or otherwise).
8 Privacy implications of memorisation
Memorisation is not process that occurs in vacuum: it likely affects the privacy of individuals, whose data is used to train MT. model. To assess the extent to which this phenomenon can harm these individuals, we discuss the implications of memorisation through the lens of data privacy. We note, that while our work is primarily aimed at centralised ML settings, we additionally include a detailed discussion on memorisation in decentralised settings in the Appendix. We are particularly interested in answering these questions over the course of this discussion:
What are the implications of memorisation on privacy of the individuals and which definition(s) can we use to estimate these?
What methods exist to extract the information memorised by model?
How can memorisation affect privacy of generalive models?
8.1 Memorisation and privacy attacks
Models which have memorised much of the data they have been trained on can often be perceived more "dangerous", in many cases it is possible to extract the information that has been memorised. While attacks can be exccuted in both the discriminative and the generative scttings, we are particularly interested in discrininative models. For these extraction of memorised information often takes the form of attacks on privacy, as they are not designed to output the data they were trained on. We discuss which "individual" traits of ML training can be exploited to extract the information the model memorisod about individual samples (similarly to how many of the same metrics can be used to identify influential or atypical samples). Concretely, we discuss three major privacy attack methods, namely: membership inference, attribute inference and data reconstruction (often referred as data inference or model inversion (Usynin et al., 2021)).
Inference attacks One of the more prominent privacy attacks is termed membership inference atlack (MIA) (Shokri et al., 2017) and it identifies if the model's training dataset contains specific point of interest. This attack has a flexible threat model and can be employed against shared white-box models (Sablayrolles et al., 2019) or APT-only black-box models (Choquette-Chop MIAS can successfully target both the discriminative and the generative models (Chen al., 2020a; Liu et al., 2019a; Hilprecht el al., 2019), making them versatile context-agnostic "unintended memorisation" auditing tool. There is growing evidence to support that high memorisation is associated with increased MIA vulnerability, although it scems that memorisation is not necessary and sufficient but just sufficient (i.c. where memorisation leads to being vulnerable to MIAs) and that there are other reasons for vulnerability beyond memorisation (Choi et. al., 2023; Carlini et al., 2022c).
al., 2021) with high effectiveness. Similarly,
It was previously shown that MIAs are particularly effective against models with larger generalisation gaps (Shokri et al., 2017: Usynin al., 2022; Truex et al., 2019; Salem et al., 2018) (with more examples in Dionysiou& & Athanasopoulos (2023)). For instance, suite of attacks all of which rely comparing the loss
16

---

Published in Transactions on Machine Learning Research (08/2024)
Yea' et al, 2010: Sableyrolles et at, 20 inpanamen et 2020:
Loss based
Watson et al, 2021: ui et al, 2022: Chobolo et 2023
Extracing memorised Informat.on
Eitropy-assed
Salem at, 2018, Song ct 2021 2021
Conficiece
2021: Gu et 2022, Chooola 2023
Yoom er ai, 201 8i trolla et 2019:
Infuence based
Sableyroltes at 2018: Bentley et 2000 ChooqucieecCtoo el, 2027
Attibute iyfrence
jayaramaa an etal, 2022, Mehnaz al,, 2022
Kosinss et 2013 2015: Feng 2021: etal. 2021:
Sracient meriion
2hi stal, 2019, celping et, 2078 Leyntn ee nl, 7007
Data reconstruction
Weight inverstor
Galle etall 2022 Halm at, 2022 Usymin 2022
Genorative inforrnc
al, 2015t: et ata.. Contini sl, 2022t ippetto at, 2022
Figure 3: Brief overview of varions attacks on privacy and the information they exploit.
values on previously seen and unseen samples (Chobola et al., 2023; Carlini et al., 2022a; Yeom et al., 2018; Jayaraman et al., 2020) naturally benefit from lower loss magnitudes on the data the model was trained on. However, we note that while attacks using model loss, outperform the naive prediction- based approaches (e.g- (Yeom et al., 2018; Choquette-Choo et al., 2021; Sablayrolles et al., 2019; Trolla & Châtel, 2019; Bentley et al., 2020)), they typically represent a non-membership test (i.e. are mostly effective at determining who was not a member rather than who was one) (Li ct al., 2022). Similarly, any proxy metric, such as the confidence scores, can be used to identify whether training record was previously "seen" by the model (Shokri et al., 2017; Salem et al., 2018; Watson et al., 2021; Gu et al., 2022). However, it was previously shown in Yeom et al. (2018) that overfitting is not prerequisite for a successful attack. Moreover, when one considers attacks on generative models (e.g. those described in Chen et al. (2020a); Hayes et al. (2017); Hilprecht et al. (2019)), it is not clear what overfitting even implies.
Works such as Jayaraman & Evans (2019); Song & Mittal (2021); Cohen & Giryes (2024) attempt to make a more concrete connection between how memorisation of data points affects their inference attack susceptibility (based the notion of memorisation presented in Section 7.1). This formulation permits a direct evaluation of how well individual samples can be A) memorised and B) inferred via MIA. A number of recent works, such as Cohen & Giryes (2024) and Carlini et al. (2022a) have speculated that there is a correlation between samples which are more likely to be memorised and the samples which are more vulnerable to adversarial inference (and MIAs in particular). Morcover, results from Choi et al. (2023) suggest that this is indeed a phenomenon which is applicable to memorised samples only, and not to "atypical, OOD" or "rare" samples. This is primarily because the definition of OOD is specific to an individual learning context and depends on the modality, which features are considered to be OOD etc. Additionally, works such as Cohen & Giryes (2024); Usynin et al. (2022); Leino & Fredrikson (2020) show that techniques which reduce the amount of information model can memorise about individual samples (such as differential privacy (Dwork, 2006)) have been used to effectively mitigate MIAs in the past.
So while a direct causal relationship between memorisation and susceptibilltty to MIA has not yet been established (and although seems likely), there are still two noteworthy observations. Firstly, atypical samples which are more likely to be memorised, are also more susceptible to inference attacks, showing that there is connection between how much can be memorised by the model and how much can then be "exposed" by the model (Mo et al., 2021; Cohen & Giryes, 2024; Choi et al., 2023). Secondly, similarly to the formulations of memorisation, the extent of MIA's success can also depend on memorisation and generalisation simultaneously, showing that these notions are complementary.
Another attack, which allows the adversary to determine individual features or attributes of their victim is termed as aftribute inference (Kosinski et al., 2013; Jayaraman & Evans, 2022; Mehnaz et al., 2022). For instance, it was previously shown that attribute inference attackers are capable of inferring protected demo- graphic information from parameter updates alone (Feng et al., 2021; Jourdan et al., 2021). As previously
17

---

Published in Transactions on Machine Learning Research (08/2024)
discussed in Mo et al. (2021), the risk of this attack can directly depend on the amount of information that is "embedded" in the trained models (more so in deeper layers (Maini et al., 2023)). Therefore works such as Thomas et al. (2020); Jourdan et al. (2021) hypothesise that similarly to MIAs, this attack also benefits from memorisation of individual samples and is particularly effective against data points of high influence.
Model inversion The last type of privacy attacks that we discuss model inversion attacks, which aim to extract the training data given some representation of the model (e.g. consecutive training snapshots (Usynin et al, 2022), final model (Haim et al,, 2022), gradient updates (Zhu et al., 2019; Geiping et al., 2020; Usynin et al., 2023a)). The unique trait of these attacks is the fact that they can be used against different representations of the same model in order to extract the training data.
When quantifying memorisation under these altacks it is important to clearly disentangle the notion of memorisation and "Leakage" (or exposure). Memorisation is the amount of information the model can store about individual (usually rare) samples, whereas leakage is the amount of information that the model can reproduce when queried. Therefore, while many of these attacks show "how much models leak", these leakages can often A) represent the information about a class of imput samples rather than individual samples (e.g. reconstruction in Fredrikson et al. (2015)) and B) not capture all of the information memorised by the model (e.g. when extracting an image, lot of pixel variation is possible, making it unclear what exactly was memorised by the model, while st.ill showing "some" " sensitive attributes (Usynin et al., 2023a)). These attacks typically exploit phenomena that are similar, but otherwise disjoint from memorisation (discussed in Section 5). Notions such as V-information (Mo et al., 2021) can be employed to identify the amount of useful data contained in the shared gradient, while a more direct metric (e.g. canary- based approaches (Carlini et al., 2019b)) could be used to quantify the amount of memorisation occurring over training that gets exploited by inversion attacker.
The aforementioned attacks (in particular MIAs) can be used to "audit" the model and to empirically identify information it memorised. This was previously manifested in A) works which verify is a specific training sample was memorised during model training (Jayaraman & Evans, 2019; Song & Mittal, 2021) and B) works validating the bounds on how much memorisation can occur with respect to specific samples (Ye et al., 2022; Nasr et al., 2021). The former allows privacy-conscious individuals to identify if their data was likely used (and memorised) by the model during training. The latter can help both the data and the model owners to esta establish how much information can be both memorised and exposed once trained model is made public (discussed in detail in Section 9.3) to introduce more realistic privacy bounds.
Finally, it is important to discuss how memorisation (as well as attack susceptibility) of one record can directly affect another record. We have previously established that data points which are memorised more (and have a higher influence) are typically associated with being more suscoptible to privacy attacks. Thus, intuitively we may deduce that removing these records would reduce the amount of memorisation a model can experience and, hence, make the entire settings more robust against privacy attacks. Carlini et al. (2022c) discovered that this, in fact, does not seem to be the case. They termed this phenomenon as the privacy onion effect, which states that should sensitive record be removed from the training datasct, the model can relatively quickly start memorising other records more to fill the information that becomes missing upon an exclusion of the record of interest. Hence, authors give an "onion" analogy: when the top privacy-sensitinn layer of data points gets removed (i.c. records which are most suscoptible to privacy attacks), the layer that follows is becomes more exposed and, thus, becomes more susceptible the same attacks. This phenomenon directly links to the rest of our work: the amount of memorisation that the model experiences with respect to given data point is relative i.e. all other things being equal, the rest of the data used to train this model (and in particular its distribution) can affect how much model memorises with respect to this point. It is also, to the best of our knowledge, one of the few works to identify a direct relationship between exposure, attack susceptibility and memorisation, further proving that all of these are distinct, yet complimentary processes. Note also that by design auditing measures leakage from the model and not memorisation; it thus stands to reason that auditing surfaces the points on the "outer layers of the onion", i.e. the most susceptible points, rather than the most memorised ones.
Overall, identification of the risks associated with model memorisation is non-trivial task, which relies on large number of methods. More importantly as underrepresented populations are much more likely
18

---

Published in Transactions on Machine Learning Research (08/2024)
to be memorised (Feldman & Zhang. 2020) and are often more susceptible to attacks (Kulynych et al., 2019), causal reasoning over the privacy risks associated with memorisation is an important consideration in development of secure Al systems.
Key Point
Data points which get memorised more are also often more vulnerable to most privacy attacks.
8.2 On memorisation in generative models
As generative ML becomes more prevalent (Liang et al., 2023; Nijkamp al., 2022; Taylor et al., 2022; Alayrac et al., 2022; Wei et al., 2022), we find that the existing formulations of generative memorisation (and even more so generalisation) have very different privacy implications compared to the discriminative settings. The question which naturally arises when discussing generative models is: as new data is generated based on the previously seen one, can the model compromise privacy of an individual by generating data close to the one they shared during training? This question becomes even more challenging to answer, as the community often distinguishes between the "intended" (i.e. the one that contributes towards generation of useful facts) and "unintended" (i.e. the one that reveals PII during generation) memorisation, making it difficult to identify which type of memorisation is given definition capturing (Naseh et al., 2023; Carlini et al., 2021). There exists a large number of prior studies on the privacy of generative models (Liu et al., 2019b; Triasteyn & Faltings, 2020; Wu et al., 2019; Carlini et al., 2021; 2019b), one of which we have previously discussed in detail in Section 7.1. The main takeaway from this discussion is that for individuals (and their data) which fall on the tails of the data distribution (i.c. of high influence), generative models can present greater privacy concern, as their data is more likely to be memorised (particularly in language models) (Carlini et al., 2019b; 2021; Tppolito al.). Moreover, given the recent advances in the field of LLMSs with many models exceeding billions of trainable parameters (Touvron et al., 2023; Jiang et al., 2024), it was shown that as the model's size (and its effective capacity) grows, so does the capacity for memorisation (Carlini et al., 2019b). While no formal relationship between model's effective capacity and its capacity to memorise has been established, most prior evidence shows that these are, in fact, positively correlated, presenting additional privacy challenges when training large generative models on sensitive data (Carlini et al., 2022b; 2019b; Meehan et al., 2024). Some solutions to this problem mostly rely on training data sanitation (e.g. the removal of personally-identifynn
information (PIT) (Inan et al., 2021) deduplication of the training data (Carlini et al., 2022b)), well as DP training Béguell- et al., 2020; Shi et al., 2022; Du et al., 2023). Both approaches have proven to be effective at reducing the risks of unintended data exposure, but the former requires access to the entire training dataset and the latter can often result in poor model utility.
The question which remains unanswered concerns the metrics which can be used to measure this unintended memorisation. While it is sometimes possible to generate the exact training data used to train language models (e.g. in Fig. 2), how could one mcasure the same notion in a domain, where such one-to-one mapping is unlikely and where data can be generated with "similar enough" private features corresponding to the training data? Moreover, authors of Tppolito et al. and Lee et al. (2021) argue that even for L.LMs, there is no straightforward way to describe memorisation of sensitive data (as the models are capable of producing outputs, which are syntactically different, but semantically identical, making "verbatim memorisation" a poor privacy metric). Similar conclusions were presented by Carlini et al. (2023) for images and was termed as the cidetic memorisation (closcly linked to the concept of "cpisodic" memorisation from (Zhang et al., 2021a)), which refers to model's ability to output an image it has only seen once (or k times for k-eidetic memorisation, where k is small integer). In order to quantify memorisation in this domain, authors proposed to use threshold 8, which measures the distance between an image and similarly-looking neighbours, alleviating the need to rely on an exact image reconstruction.
However, one may argue that since pixels (on their own) are not necessarily meaningful (iee change in a single pixel is unlikely to affect the semantic meaning of the image), but the features within natural images are, without further contextualisation most pixel-based metrics are meaningless. The work of Fernandez et al. (2023) tries to address this issue by using a similarity metric (proposed in Packhäuser et al. (2022))
19

---

Published in Transactions on Machine Learning Research (08/2024)
based on the re-identification ratio: can the adversary identify the private data of a specific individual based on the image generated by the adversary? Another method was previously proposed by Kuppa et al. (2021) and here authors measure generative sample memorisation through susceptibility of individuals whose data is used to train the model to privacy attacks, in particular membership inference. We discuss how memorisation is related to information leakage in Section 8.1 in detail, but as we can already see it is difficult to discntangle the two and measure the degree of memorisation directly. Therefore, we conclude that while in discriminative settings there exists a number of methods which aim identify specific features or samples the model "memorises", in the context of generative modelling, these are challenging to contextualise.
Key Point
Memorisation in generative models can be perceived either through self-influent or as the ability of model to produce outputs which are similar to the input data.
9 Preventing and reversing memorisation
As information can be memorised by the model, raises the question whether it can also be "unmemorised", This process can occur naturally over the training process and it can be induced manually by the data owner. Moreover, over the course of training it is also possible to bound the amount of information that the model can memorise through the use of DP training. In this section we discuss these phenomena and outline the processes which are responsible for reversal or reduction of memorisation.
9.1 Spontaneous reversal of memorisation
While some data points are more prone to memorisation, the information contained in the model about these memorised samples often changes over time (due to model's limited memorisation capacity). This effect is often termed as "forgetting", which intuitively means that. if the model does not encounter a previously seen training point during training (i.e. if it "disappears" " from the training set), the performance of that model on this specific training point would degrade over time (Toneva et al., 2018). The main question to consider is: "is forgetting the opposite of memorisation?" Intuitively, the answer is affirmative, as when the model forgets the data point, model's 's performance on it degrades. This effect can be particularly profound for the data points which fall on the tails of the distributions (e.g. mislabelled data points), with Jagielski et al. (2022) showing that data which is more likely to be memorised under Feldman & Zhang (2020) definition is also more likely to be forgotten.
It was previously shown by Feldman (2020) that models have to memorise certain information about in- dividual samples over the course of model training- Authors of Jagielski el al. (2022) went further and argued that similarly to how models have to memorise, they also have to forget information about indi- vidual samples if they are not encountered again during later stages of training. As deep learning often requires multiple sources of non-determinism to be present (e.g. random batch shuffling, stochastic gradient methods etc.), these are, therefore causing the models to forget. Authors additionally demonstrate that should non-determinism be eliminated in full, forgetting no longer occurs, which is "in spirit" similar to the conclusions of Feldman & Zhang (2020). In Feldman & Zhang (2020) authors formalise that if a source of non-determinism included during training (e-g. addition of noise under differential privacy), then the models would be limited in the amount of information they can memorisc (and, hence, forget).
The work of Jagielski et al. (2022), again, shows that the samples which are most vulnerable to this phe- nomenon are the atypical samples. This notion has different implications to "catastrophic forgetting", where the entire sub-distribution of data (c.g. an entire class) can be forgotten, which is not guaranteed to directly affect the previously memorised samples (but rather some population of the training set, which could have been memorised). This phenomenon can be measured using the concept of adversarial advantage, which com- pares the performance of membership-based attacks on the samples of interest at different stages of training. Morcover, the issue of forgetting captures another important question that we have previously outlined: when "forgetting", one is required to define what is that needs to be "forgotten" (i.e. what has previously
20

---

Published in Transactions on Machine Learning Research (08/2024)
been memorised)? In Tirumala et al. (2022), for instance, this concept captures verbatim memorisation, which authors of Jagielski et al. (2022) argue does not represent a more general definition of memorisation. Similarly to other works on memorisation in language models, they craft uniquely identifiable samples (i.e. canarios from Carlini et al. (2019b)) to approximate the degree of memorisation that remains on specific training records over the course of training.
Key Point
Samples that are memorised during training can be forgotten if they are not encountered again. This is particularly common for atypical data points.
9.2 Induced reversal memorisation
Finally, we briefly mention another area which is closely linked to both notions of memorisation and forget- ting: machine unlearning (Bourtoule et al., 2021). Unlearning is a set of different techniques, which given a training record(s) modifies the model to no longer include the contributions of that record(s). is of note that forgetting can be targeted as presented in Zhou al. (2022), showing that these phenomena are related, variety of factors, including the current state of the model (i.e. if
The difficulty of unlearning depends it is currently training or if the model has alrcady been trained), the features of the data rocord(s) as well as the remaining data points. Forgetting (when compared to unlearning) occurs naturally during training (it is often not possible to predict which specific records will be forgotten (Toneva et al., 2018)), whereas unlearning is a set of techniques designed to remove the contributions of specific data point(s). The final factor we outlined bears additional discussion.
As above, when performing unlearning, one tries disentangle and remove the contribution of one or more data record. If this data point is "simple" or its class is over-represented, then the removal of its contribution is unlikely to have large effect the rest of the population (e.g. their attack susceptibility). However, if the record(s) comes from the tail end of the data distribution, its unlearning can negatively affect privacy of those, whose data remains in the dataset. We have previously discussed this phenomenon, namely the "the privacy onion effect" (Carlini et al., 2022c), which shows that by deliberately removing the record (and, thus, contribution) that was more susceptible to allacks, other samples can become more prone to inference. Thus, we, again, observe that memorisation (and um-memorisation) is indced relative the data surrounding highly memorised sample. As result, this opens up number of questions, such "Does un-memorising also mean un-generalising? If yes, is it ethical to withdraw ones data if this will deteriorate the generalisation performance on her samples?". These questions must to be deliberated by the ML. community in order to devisc a unified strategy for effective model training.
Key Point
Forgetting can be artificially induced "unmemorise" individual data points.
9.3
Bounding memorisation with differential privacy
Differential privacy (Dwork & Roth, 2014), discussed previously, is the canonical privacy definition for statistical data procossing and ML. Intuitively, DP can be perceived a property of an algorithm, making it approximately invariant
inclusion inclusion/exclusion /exllusion of a single data point. More formally: The result of a computation (e.g. model training) is said to be differentially private if a probabilistic stability notion over neighbouring datasets is satisfied. Concretely, in the context of ML (and previously introduced notation), this stability notion requires that a specific model f' (or set of weights) has a similar likelihood under A(S) and A()))). This means that the value the random variable f takes, cannot change much no matter which data point we remove from the training dataset. More formally, random Algorithm A satisfies e-DP when:
(('IA(S)) < (
21

---

Published in Transactions on Machine Learning Research (08/2024)
for any i S and where A(.) can take any value in some hypothesis class H. given by e.g. the model architecture. Since the exclusion of any data point cannot significantly change the value f takes, the output of f on any sample i E S can also not be significantly impacted. This implies that changes in most metrics, due to the removal of a sample, (e.g. Eq. (3) or Eq. (4)) will be small and thus memorisation will be bounded under DP.
Concrete bounds on the memorisation ability (self-influence) of differentially private model were previously presented in Feldman (2020) and van den Burg & Williams (2021). This is also directly related to the concept of model's stobility described in Evgeniou et al. (2004), which is immediate from the results of Feldman (2020) and Brown et al. (2021). In particular, the former studies the relationship between the leavo-one-out stability of an individual sample and model's memorisation capacity, showing that under DP, stability increases and memorisation decreases. Thus, informally, stable algorithms memorise less and DP algorithms are "very stable", leading reduced memorisation. Further, using different interpretations of DP, it was also been shown that DP directly bounds the ability of an adversary to perform MIA (Wasserman & Zhou, 2010) or training data reconstruction (Hayes et al., 2023; Stock et al., 2022). Tt is known that any privacy protection scheme which does not allow for catastrophic privacy degradation incurs some utility penalty (Dinur & Nissim, 2003). While applying DP in the context of ML also results in utility penalty (which may be unavoidable due to the fact that DP limits memorisation, as shown in Feldman & Zhang (2020); Wang et al. (2024)), recent works have demonstrated that, for many real-world datasets, competitive performance can still be achieved by applying a number of training adaptations such as pre-training on large public datasets (De al., 2022; Berrada et al., 2023). Overall, there is strong evidence that DP is the tool of choice for bounding the negative privacy implications and risks of memorisation in practice.
Key Point.
Differentially private ML training can (provably) prevent the negative traits caused by memorisation.
10 Recommendations for ML practitioners
Memorisation itself is context-agnostic phenomenon which can be present in many ML tasks. However, the impact of memorisation can heavily depend upon individual training settings. Here we provide number of guidelines, which should help MI. practitioners context memorisation in different learning scenarios and adapt their training settings with these in mind:
Care should be taken when quantifying the impact of memorisation in different modalities. In language processing lasks, one should not measure memorisation verbatim (Jagielski et al., 2022; Lee et al., 2021; Ippolito et al.) and similarly, per-pixel metrics should not be used quantify memorisation in imaging settings (Fernandez et al., 2023; Packhäuser et al., 2022).
One should be careful to distinguish between memorisation, extractability and leakage. While these are all related phenomena, memorisation of a data point is not a pre-requisite for its successful extraction (Carlini et al., 2021) nor does it affect how much the model can leak about individual data samples (Ippolito al.). Therefore, these terms should not be used interchangeably.
Not all features are equally sensitive (e.g. PIIs vs common facts) or equally prone to memorisation (Carlini et al., 2022b; Inan et al., 2021). One can reduce the effectiveness of attacks exploiting unintended memorisation by curating one's training data (e.g. PII stripping) (Inan et al., 2021).
Repetition of data can lead to its higher extractability (Jagielski al., 2022; Tirumala et al., 2022). Therefore, deduplication of sensitive data can reduce the corresponding privacy risks.
Generative models can be prone to regurgitation of the training data (Carlini et al., 2019b). This effect is more profound in longer context windows and larger models (Tirumala et al., 2022). There- fore, one should consider using smaller models and context windows, while maintaining an adequate level of utility.
22

---

Published in Transactions on Machine Learning Research (08/2024)
Underrepresented populations more prone to memorisation (and to privacy attacks) (Mueller et al., 2022; Feldman & Zhang, 2020; Carlini et al., 2022b). Therefore, suitable protection techniques should be used when training models on such data.
Removal of data from the training pool can have adverse effect on the rest of the population with respect to privacy risks (Carlini et al., 2022c). Therefore, it is not recommended to remove individual data points which are perceived to be more memorisable and one should address these issues on a dataset or a model level instcad (e.g. PII stripping) (Inan et al., 2021).
Memorisation can be provably bounded by performing differentially private training, reducing the privacy risks and permitting public releases of models trained on private data (Feldman, 2020; Thakkar et al., 2020; Carlini et al., 2022c).
11 Conclusions and future directions
In this work, we systematically summarisc and discuss the existing formulations of memorisation in machine learning models. We formulate the most widely used definitions and study their implications on the privacy of the data used to train the model. As quantifying memorisation using these definitions can be infeasible for many ML settings, we additionally discuss methods which allow the practitioners to estimate memorisation instead. Furthermore, are likely to be memorised using techniques from the domains of data valuation and influence estimation. Additionally we provide a number of guidelines for ML practitioners, to help them to contextualise mem- orisation when designing ML workflows, allowing for a more safe and responsible Al training. Finally, we outline the open challenges and questions to be collectively addressed by the machine learning community in order to be able to standardise the taxonomy in the field and permit more well-designed collaboration with privacy risks associated with model memorisation in mind. Given a large number of metries related to measuring memorisation, the effects memorisation of the individuals, whose data is used to train the model, we outline number of key future directions from which, we believe, the community can benefit:
discuss methods which allow the data owners to identify which specific samples
have on both the model performance and the privacy
Standardisation of the notion: Many different definitions of memorisation are employed across various ML research fields, unifying these of uttmost importance, as the misuse of the term can have implications in broader community.
Information content and memorisation: Samples with higher information content tend to be more prone to memorisation. However, as we established, not all information contained in those samples is useful to the model. Moreover, samples which are described as "more difficult" can be either highly informative (but rare) or of poor utility because they are malformed. While both of these sample types have high positive self-influence (i.e. if included, their utility on themselves improves), the former tend to be of high positive cross-influence, (i.e. improving the utility on other samples) and the latter are often of high negative cross-influcnce, harming the overall performance on other data points. We believe that further work is required to be able to establish strong links between information content, memorisation and the resulting model utility.
Privacy implications of memorisation: Are some models or training settings more prone to "unintended" memorisation often exploited by altacks on ML? The community needs to establish clear guidelines on the implications of memorisation on the adversarial perceptibility, particularly with respect to potential privacy violations when generative models are adversarially prompted.
Memorisation in the age of AI regulation: How can (and should) memorisation be regulated with respect to copyrighted materials frequently used to train large-scale models? Copyrighted material can often be accessed publicly, thereby allowing models to learn from it and memorise some of its contents. Since there is still no clear prior definition of what memorisation is, regulating how these should be processed is open challenge.
23

---

Published in Transactions on Machine Learning Research (08/2024)
Malicious memorisation and model alignment: As we have previously seen, adversarial sam- ples can present real threats to models trained on public data and are very frequently well-memorised by ML models. As a result, the aligument of such models can be severely affected by how well they learn (and memorise) from benign/ benign/malicions /malicions data, making it a promising area of future research (with prior works such as Zou al. (2023); Carlini et al. (2024) suggesting that even in-prompt tuning can often be sufficient to override the existing alignment safeguards).
Acknowledgements and Disclosure of Funding
This wark was supported by the German Federal Ministry of Education and Research and the Bavarian State Ministry for Science and the Arts under the Munich Centre for Machine Learning (MCML), the German Ministry of Education and Research and the the Medical Informatics Initiative as part of the PrivateAIM Project, the Joint Academy of Doctoral Studies of Imperial College London and the Technical University of Munich, the Bavarian Collaborative Research Project PRIPREKI of the Free State of Bavaria Funding Programme AArtificial Intelligence - Data Science', and the German Academic Exchange Service (DAAD) under the Kondrad Zuse School of Excellence for Reliable AI (RelAT).
References
Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Tlya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communécations Security. ACM, October 2016. doi: 10. 011145/2977749229788 org/10. /0t..11///977999999797779 2978318.
URL https://dot
Chirag Agarwal, Daniel D' D'souza, and Sara Hooker. Estimating example difficulty using variance of gradients. In Proceedings of the IEEE/CVF Conference Computer Vision and Pottern Recognition, pp. 10368- 10378, 2022.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Tain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716 15-23766-23736, 2022.
Mohammad Mohammadi Amiri, Deniz Gündüz, Sanjeev R Kulkarni, and H Vincent Poor. Convergence of update aware device scheduling for federated learning at the wireless edge. IEEE Transactions on Wireless Communications, 20(6))3643-3666 2021.
Devansh Arpit, Stanislaw Jastrzkebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kan- wal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning, pp. 233 242. PMLR, 2017.
Rohit Babbar and Bernhard Schölkopf. Data scarcity, robustness and extreme multi-label classification. Machine Learning, 108(8-9):1329-1355 2019.
Juhan Bae, Nathan Ng, Alston Lo, Marzych Ghassemi, and Roger B Grosse. If influence functions are the answer, then what is the question? Advances in Neural Information Processing Systems, 35:17953 17967, 2022.
Eugene Bagdasaryan and Vitaly Shmatikov. Blind backdoors in deep learning models. In 30th USENIX Securily Symposium (USENIX Securily 21), pp. (505-1521) 2021.
Robert Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. Advances in Neural Information Processing Systems, 34:10876-10888 2021.
Borja Balle, Giovanni Cherubin, and Jamie Hayes. Reconstructing training data with informed adversaries. In 2022 IEEE Symposium Security and Privacy (SP), pp. 1138-11811111 1156. IEEE, 2022.
Samyadeep Basu, Phil Pope, and Soheil Feizi. Influence functions in deep learning are fragile. In International Conference on Learning Represendations, 2020.
24

---

Published in Transactions on Machine Learning Research (08/2024)
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 6541 6549, 2017.
Jason W Bentley, Daniel Gibney, Gary Hoppenworth, and Sumit Kumar Jha. Quantifying membership inference vulnerability via generalization gap and other model metrics. arXiv preprint or arXiv:2009.00 25560000000666, 2020.
Leonard Berrada, Soham De, Judy Hanwen Shen, Jamie Hayes, Robert Stanforth, David Stutz, Pushmeet Kohli, Samuel T, Smith, and Borja Balle. Unlocking accuracy and fairness differentially private image classification. arXiv preprint arXiv:2308. 10888, 2023.
Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pp. 141-159. IEEE, 2021.
Gavin Brown, Mark Bun, Vitaly Feldman, Adam Smith, and Kunal Talwar. When is memorization of irrelevant training data necessary for high-acouracy learning? In Proceedliggs of the 53rd annual ACM SIGACT symposium on theory of computing, pp. 123 132, 2021.
Nicholas Carlini, Ulfar Erlingsson, and Nicolas Papernot. Distribution density, tails, and outliers in machine learning: Metrics and applications. arXiv preprinl arXiv: 13427, 2019a.
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), pp. 267-284, 2019b.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Securily Symposium (USENIX Securily 21), pp. 2633 2650, 2021.
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Sympostum Security and Privacy (SP), pp. 1897-1914. IEFE, 2022a.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang, Quantifying memorization across noural language models. arXie preprint arXiv:2202.09 07646, 2022b.
Nicholas Carlini, Matthew Jagiclski, Chiyuan Zhang, Nicolas Papernot, Andreas Terzis, and Florian Tramer. The privacy onion effect: Memorization is relative. Advances in Neural Information Processing Systems, 15:18263-13276, 2022c.
Nicholas Carlini, Milad Nasr, Christopher A Choquette-Choo, Matthew Jagielski, Trena Gao, Pang Wei W Koh, Daphne Ippolito, Florian Tramer, and Ludwig Schmidt. Are aligned neural networks adversarially aligned? Advances in Neural Information Processing Systems, 36, 2024.
Nicolas Carlini, Jamic Hayes, Milad Nasr, Matthew Jagiclski, Vikash Schwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In 32nd USENIX Securily Symposium (USENIX Security 23), pp. 1253-5270 2023.
Satrajit Chatterjee. Learning and memorization. In International conference on machine learning, pp- 155. PMLR, 2018
Dingfan Chen, Ning Yu, Yang Zhang, and Mario Fritz. Gan-leaks: A taxonomy of membership inference attacks against generative models. In Proceedings of the 2020 ACM SIGSAC conference on computer and communications security, pp. 342-362, 2020a.
Richard J Chen, Ming Y Lu, Tiffany Y Chen, Drew FK Williamson, and Faisal Mahmood. Synthetic data in machine learning for medicine and healthcare. Nature Biomedical Engineering, 5(6):493-44) 497, 2021.
25

---

Published in Transactions on Machine Learning Research (08/2024)
Wenlin Chen, Samuel Horvath, and Peter Richtarik. Optimal client sampling for federated learning. arXiv preprinl ar Xrx::20010. 13723, 2020b.
Tomas Chobola, Dmitrii Usynin, and Georgios Kaissis. Membership inference attacks against semantic segmentation models. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security, pp. 43-53, 2023.
Jihye Choi, Shruti Tople, Varun Chandrasckaran, and Somesh Jha. Why train more? effective and efficient membership inference via memorization. aiv preprint orXin:2310.0800 2023.
Christopher A Choquette-Coov Choo, Florian Tramer, Nicholas Carlini, and Nicolas Papernot. Label-only mem- bership inference attacks. In International conference on machine learning, pp. 1964 1974. PMLR, 2021.
Gilad Cohen and Raja Giryes. Membership inference attack using self influence functions. In Proceedings of the IEEE/CVE Winter Conference Applications of Computer Vision, pp. 4892-4901, 2024.
R. Dennis Cook. Detection of influential observation in linear regression. Technometrics, 19(1):15-18, 1977.
R. Dennis Cook and Sanford Weisberg. Residuals and influence in regression, 1982.
Soham De, Leonard Berrada, Jamie Hayes, Samuel Smith, and Borja Balle. Unlocking agghaccuracy differentially private image classification through scale. arXiv preprint arXim:2204. 13650, 2022.
Trit Dinur and Kobbi Nissim. Revealing information while preserving privacy. In Proceedings of the twenty- second ACM SIGMOD-SIGACT-SICAA symposium on Principles of database systems, pp. 202 210, 2003.
Antreas Dionysiou and Elias Athanasopoulos. Sok: Membership inference harder than previously thought. Proceedings on Privacy Enhancing Technologies, 3:286 306, 2023.
Minxin Du, Xiang Yue, Sherman SM Chow, Tianhao Wang, Chenyu Huang, and Huan Sun. Dp-forward: Finc-tuning and inference on language models with differential privacy in forward pass. In Proceedings of the 2023 ACM SIGSAC Conference on Compuler and Communications Securily, pp. 2665 2679, 2023.
Cynthia Dwork. Differential privacy. In International colloquium on aulomala, languages, and programming, pp. 1-12. Springer, 2006.
Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Foundations and Trends@ in Theoretical Computer Science, 93344)211--077, 2014. ISSN 1551-305X. doi: 10.1561/ 0400000042. URL http:///tca doi. org/10.1155//440000044
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with v-usable information. In International Conference on Machine Learning, pp. 5988 6008. PMLR, 2022.
Theodoros Evgeniou, Massimiliano Pontil, and André Elisseeff. Leave one out error, stability, and general- ization of voting combinations of classifiers. Machine learning, 55:71-97, 2004.
Vitaly Feldman. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Anveual ACM SIGACT Symposium on Theory of Computing, pp. 954 959, 2020.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881-288 2020.
Vitaly Feldman and Tijana Zrnic. Individual privacy accounting via a renyi filter. Advances in Neural Information Processing Systems, 34:28080 28091, 2021.
Tiantian Feng, Hanich Hashemi, Rajat Hebbar, Murali Annavaram, and Shrikanth Narayanan. At- tribute inference attack of speech emotion recognition in foderated learning settings. arXiv preprint arXiv:2122. 13416, 2021.
26

---

Published in Transactions on Machine Learning Research (08/2024)
Virginia Fernandez, Pedro Sanchez, Walter Hugo Lopez Pinaya, Grzegorz Jacenków, Sotirios A Tsaftaris, and Jorge Cardoso. Privacy distillation: Reducing re-identification risk of multimodal diffusion models. arXiv preprint rrXie:2306.01320 2023.
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures In Proceedings of the 22nd ACM SIGSAC conference computer and commanicalions securily, pp. 1322 1333, 2015.
Isha Garg and Kaushik Roy. Memorization through the lens of curvature of loss function around samples. arXiv preprinl or arXic:2307.0881) 2023a.
Isha Garg and Kaushik Roy. Samples with low loss curvature improve data efficiency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 20290- -03000 2023b.
Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gradients-how easy is it to break privacy in federated learning? Advances in Neural Information Processing Systems, 33: 16937 16947, 2020.
Amirata Ghorbani and Jamos Zou. Data shapley: Equitable valuation of data for machine learning. In International conference machine learning, pp. 2242-2251. PMLR, 2019.
Ziv Goldfeld and Kristjan Greenewald. Sliced mutual information: scalable measure of statistical depen- dence. Advances in Neural Information Processing Systems, 34:17567-17577, 2021.
Ziv Goldfeld, Ewout van den Berg, Kristjan Greenewald, Igor Melnyk, Nam Nguyen, Brian Kingsbury, and Yury Polyanskiy. Fstimating information flow in deep neural networks. ar preprint arXiu:18 1ett:08100522 2018.
Tan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. 2014.
arXiv preprint ar
Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with influence 2023.
functions. Xiv preprint or arXiv:2308.0320
Yuhao Gu, Yuebin Bai, and Shubin Xu. Cs-mia: Membership inference allcc based on prediction confidence series in federated learning. Journal of Information Security and Applications, 67:103201, 2022.
Han Guo, Nazneen Fatema Rajani, Peter Hase, Mohit Bansal, and Caiming Xiong, Fastif: Scalable influence functions for efficient model interpretation and debugging. arXio preprint 2.11::201..15781, 2020.
Samyak Gupta, Yangsibo Huang, Zexuan Zhong, Tianyu Gao, Kai Li, and Danqi Chen. Recovering private text in federated learning of language models. Advances in Neural Information Processing Systems, 35: 8130-8143, 2022.
Niv Haim, Gal Vardi, Gilad Yehudai, Ohad Shamir, and Michal Irani. Reconstructing training data from trained neural networks. Advances in Neural Information Processing Systems, 09222911222222 2022.
Zayd Hammoudeh and Daniel Lowd. Identifying a training-set attack's target using renormalized influence estimation. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communiccations Security, pp. 1367 1381, 2022a.
Zayd Hammoudch and Daniel Lowd. Training data influence analysis and cstimation: A survey. arXiv preprint STTF::2222006111 2022b.
Awni Hanmun, Chuan Guo, and Laurens van der Maaten. Measuring data leakage in machine- learning models with fisher information. In Uncertainty in Artificial Intelligence, pp. 760-770, PMLR, 2021.
John Hartley and Sotirios Tsaftaris. Measuring unintended memorisation of unique private features in neural networks. Xi preprint XxXi:2202.08099 2022.
27

---

Published in Transactions on Machine Learning Research (08/2024)
Hrayr Harutyunyan, Alessandro Achille, Giovanni Paolini, Orchid Majumder, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Estimating informativeness of samples with smooth unique information. arXiv preprint arXic:2101.066400 2021.
Jamie Hayes, Luca Melis, George Danezis, and Emiliano De Cristofaro. Logan: Membership inference attacks against generative models. arXiv preprint 700.07655. 2017.
Jamie Hayes, Saeed Mahloujifar, and Borja Balle. Bounding training data reconstruction in dp-sgd. arXiv preprint arXiv:2302.0 002.002.72222 07225, 2023.
Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Halfield-Dodds, Tom Henighan, Tristan Hume, et al. Scaling laws and interpretability of learning from repeated data. arXiv preprint 2xXx:2205.10487, 2022.
Benjamin Hilprecht, Martin Härterich, and Daniel Bernau. Monte carlo and reconstruction membership inference attacks against generative models. Proc. Priv. Enhancing Technol., 2019(4):232-249) 2019.
Huseyin A Inan, Osman Ramadan, Lukas Wutschitz, Daniel Jones, Victor Rühle,, James Withers, and Robert Sim. Privacy analysis in language models via training data leakage report. ArXio, abs/2101.05405, 2021.
Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagiclski, Katherine Lec, Christo- pher A Choquette-Choo and Nicholas Carlini. Preventing generation of verbatim memorization in lan- guage models gives false sense of privacy.
Paul Irolla and Grégory Châtel. Demystifying the membership inference attack. In 2019 12th CMI Conference on Cybersecurity and Privacy (CMI), pp- 1-7. TEFE, 2019.
Matthew Jagielski, Om Thakkar, Florian Tramer, Daphne Ippolito, Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, et al. Mcasuring forgetting of memorized training examples. arXiv preprint arXiv:2207.0000 000999, 2022.
Bargav Jayaraman and David Evans. Evaluating differentially private machine learning in practice. In 28th USENIX Security Symposium (USENIX Security 19), pp. 1895-1912, 2019.
Bargav Jayaraman and David Evans. Are attribute inference attacks just imputation? In Proceedings of the 2022 ACM STGSAC Conference on Compuler and Communications Securily, pp. 1569 1582, 2022.
Bargav Jayaraman, Lingxiao Wang, Katherine Knipmeyer, Quanquan Gu, and David Evans. Revisiting membership inference under realistic assumptions. arXiv preprint arXix:2005. 10881, 2020.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint ar Xiv:22401 04088, 2024.
Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C Mozer. Charactcrizing structural regularities of labeled data in overparameterized models. arXie preprinl urXiv:2002.0322 2020.
Théo Jourdan, Antoine Boutct, and Carole Frindel. Privacy assessment of fedcrated learning using private personalized layers. In 2021 IEEE 31st International Workshop on Machine Learning for Signal Processing (MLSP), pp. 1-6. TEEE, 2021.
Angelos Katharopoulos and François Fleuret. Not all samples are created equal: Deep learning with impor- tance sampling. In International conference on machine learning, pp. 2525-2534. PMLR, 2018.
Tkki Kishida and Hideki Nakayama. Empirical study of easy and hard examples in enn training. In Neural Information Processing: 26th International Conference, ICONIP 2019, Sydney, NSW, Australia, Decemaber 12-15, 2019, Proceedings, Port IV 26, pp. 179-188. Springer, 2019.
Pang Wei Koh and Percy Liang. Understanding black- box predictions via influence functions, In International conference on machine learning, pp. 1885-1894 PMI.R, 2017.
28

---

Published in Transactions on Machine Learning Research (08/2024)
Andrey Kolmogorov. On the shamnon theory of information transmission in the case of continuous signals. TRE Transactions on Information Theory, 2(4):102-10) 108000 1956.
Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint 1xXi:1610005492. 2016.
Michal Kosinski, David Stillwell, and Thore Graepel. Private traits and attributes are predictable from digital records of human behavior. Proceedings of the national academy of sciences, 110(15):5802-55) 2013.
Antti Koskela, Marlon Tobaben, and Antti Honkela. Individual privacy accounting with gaussian differential privacy. arXiv preprint arXiv:2209. 15596, 2022.
Michael Kounavis, Ousmanc Dia, and Nqar Ramazanli. On influence functions, classification influence, relative influence, memorization and generalization. arXiv preprinl arXiv:2305. 16094, 2023.
Bogdan Kulynych, Mohammad Yaghini, Giovanni Cherubin, Michael Veale, and Carmela Troncoso. Dis- parate vulnerability to membership inference attacks. arXiv preprinl urXic:1906.000 00089, 2019.
Aditya Kuppa, Lamine Aouad, and Nhien-An Le-Khac. Towards improving privacy of syuthetic datasets. In Annual Privacy Forum, pp. 106-119. Springer, 2021.
Fan Lai, Xiangfeng Zhu, Harsha V Madhyastha, and Mosharaf Chowdhury. Oort: Efficient federated learn- ing via guided participant selection. In 15th {USENIX} Symposium on Operating Systems Design and Implementation (1 (ODDI) 21), pp. 19-35, 2021.
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint
arXiv:2107.06400 2021.
Klas Leino and Matt Fredrikson. Stolen memories: Leveraging model memorization for calibrated (White- Box} membership inference. In 29th USENIX security symposium (USENIX Security 20), pp. 1605-1622, 2020.
Anran Li, Lan Zhang, Juntao Tan, Yaxuan Qin, Junhao Wang, and Xiang-Yang Li. Sample-level data selec- tion for federated learning. In IEEE INFOCOM 2021-IERE Conference on Computer Communications. pp. 10. IEEE, 2021.
Xiao Li, Qiongxiu Li, Zhanhao Hu, and Xiaolin Hu. On the privacy effect of data enhancement via the lens of memorization. Xivi preprind X2v: 082700, 2022.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 9493 9500. IEEE, 2023.
Kin Sum Liu, Chaowei Xiao, Bo Li, and Jie Gao. Performing co-membership attacks against deep generative models. In 2019 IEEE International Conference on Data Mining (ICDM), pp. 459 467. IEEE, 2019a.
Yi Liu, Jialiang Peng, JQ Jamos, and Yi Wu. Ppgan: Privacy-preserve generative adversarial network. In 2019 IEEE 25Th international conference on parallel and distributed systems (ICPADS), pp. 985 989. TEEE, 2019b.
Pratyush Maini, Michael C Mozer, Hanie Sedghi, Zachary C Lipton, J Zico Kolter, and Chiyuan Zhang. Can neural network memorization be localized? arXiv preprint arXiv:2307.00544. 2023.
Casey Mechan, Florian Bordes, Pascal Vincent, Kamalika Chaudhuri, and Chuan Guo. Do ssl models have déjà vu? case of unintended memorization in self-supervised learning. Advonces in Neural Information Processing Systems, 36, 2024.
29

---

Published in Transactions on Machine Learning Research (08/2024)
Shagufta Mehnaz, Sayanton V Dibbo, Roberta De Viti, Ehsamul Kabir, Björn B Brandenburg. Stefan Man- your sensitive attributes private? novel model inversion attribute inference attacks on classification models. In 31st USENIX Se- curity Symposium (USENIX Security 22), pp. 4579-4596, 2022.
gard, Ninghui Li, Elisa Bertino, Michael Backes, Emiliano De Cristofaro, et al. your
Harsh Mehta, Ashok Cutkosky, and Behnam Neyshabur. Extreme memorization via scale of initialization.
arXiv preprint Xrx:20008. 18363, 2020.
Fan Mo, Anastasia Borovykh, Mohammad Malekzadeh, Hamed Haddadi, and Soteris Demetriou. Quantifying information leakage from gradients. CoRR, abs/2105. 13929, 2021.
Tamara T Mueller, Stefan Kolek, Friederike Jungmann, Alexander Ziller, Dmitrii Usynin, Moritz Knolle, Daniel Rueckert, and Georgios Kaissis. How do input attributes impact the privacy loss in differential privacy? arXiv preprint arXiv:2211 10177 2022.
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301005211 2023.
Ali Naseh, Jaechul Roh, and Amir Houmansadr. Understanding (un) intended memorization in text-to-image gencrative models. arXiv preprint arXi:22312.07550 2023.
Milad Nasr, Shuang Songi, Abhradeep Thakurta, Nicolas Papernot, and Nicholas Carlin. Adversary instan- tiation: Lower bounds for differentially private machine learning. In 2021 IEEE Symposium on security and privacy (SP), pp. 866 882. IEEE, 2021.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprinl arXiv:2203. 13474, 2022.
Tyiola E Olatunji, Wolfgang Nejdl, and Megha. Khosla. Membership inference attack on graph neural net- works. In 2021 Third IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA), pp. 11 11-20. IEEE, 2021.
Kai Packhäuser, Sebastian Gündel, Nicolas Münster, Christopher Syben, Vincent Christlein, and Andreas Maier. Deep learning-base patient re-identification is able to exploit the biometric nature of medical chest x-ray data. Scientific Reports, 12(1):14851, 2022.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Advances in Newral Information Processing Systems, 34:20596 20607, 2021.
Adam Pearce, Asma Ghandeharioun Nada Hussein, Nithum Thain, Martin Wattenberg, and Lucas Dixon. Do machine learning models memorize or generalize. People+ AI Reseorch, 2023.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33:19920-19930, 2020.
Deepak Ravikumar, Efstathia Soufteri, Abolfazl Hashemi, and Kaushik Roy. Unveiling privacy, memorization, and input curvature links. arXiv preprint arXiv:2402.18790 2024.
Daniel Rueckert and Julia A Schnabel. Model-based and data- datavdriven strategies in medical image computing. Proceedings of the IEEE, 108(1):110- 1124 2019.
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and Hervé Jégou. White- boxte-box vs black-box: Bayes optimal strategies for membership inference. In International Conference on Machine Learning, pp. 5558-5567. PMLR, 2019.
Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. MI-lcaks: Model and data independent membership inference attacks and defenses on machine learning models. arxiv preprint ar Xiv: 012.01266 2018.
30

---

Published in Transactions on Machine Learning Research (08/2024)
Andrea Schioppa, Polina Zablotskaia, David Vilar, and Artem Sokolov. Scaling up influence functions. In Proceedings of the AAAT Conference Artificial Intelligence, volume 36, pp. 8179-8186, 8186, 2022.
Andrea Schioppa, Katja Filippova, Ivan Titov, and Polina Zablotskaia. Theoretical and practical perspectives on what influence functions do. arXiv preprint arXiv:2905. 16971, 2023.
Hyunseok Seo, Masoud Badiei Khuzani, Varun Vasudevan, Charles Huang, Hongyi Ren, Ruoxiu Xiao, Xiao Jia, and Lei Xing. Machine learning techniques for biomedical image segmentation: an overview of technical aspects and introduction to state-of-art applications. Medical physics, ([))==e48-e166) 2020.
Weiyan Shi, Ryan Shea, Si Chen, Chiyuan Zhang, Ruoxi Jia, and Zhou Yu. Just fine-tune twice: Selective differential privacy for large language models. arXiv preprint arXiv:2204. 07667, 2022.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pp. 3-18. IEEE, 2017.
Ravid Shwartz-Ziv. Information flow in deep neural networks. arXie preprint or 2rx2v 06749, 2022.
Liwei Song and Prateek Mittal. Systematic evaluation of privacy risks of machine learning models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2615-2632, 2021.
Cory Stephenson, Suchismita Padhy, Abhinav Ganesh, Yue Hui, Hanlin Tang, and Yeo Chung. On the geometry of generalization and memorization in deep neural networks. arXiv preprint 1XX::22055.4602, 2021.
Pierre Stock, Igor Shilov, Nla Mironov, and Alexandre Sablayrolles. Defending against reconstruction attacks with r\ 'enyi differential privacy. arXiv preprint arXic:2202.0722 2022.
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojníc. Galactica: A large language model for science. arXiv preprinl arXiv:2211.0900 000000 2022.
Om Thakkar, Swaroop Ramaswany, Rajiv Mathews, and Françoise Beaufays. Understanding unintended memorization in federated learning. arXio preprint arXiv:2006.0744 2020.
Alcena Thomas, David Ifcoluwa Adelani, Ali Davody, Aditya Mogadala, and Dietrich Klakow. Investigat- ing the impact of pre-trained word embeddings on memorization neural networks. Tn Text, Speech, and Dialogue: 23rd International Conference, TSD 2020, Brno, Czech Republic, September 8 11, 2020, Proceedings 23, pp. 273 281. Springer, 2020.
Zhiyi Tian, Lei Cui, Jie Liang, and Shui Yu. A comprehensive survey poisoning allacks and countermea- sures in machine learning. ACM Computing Surveys, 55(8):1-35, 2022.
Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language modcls. Advances in Neural Information Processing Systems, 35:38274 38290, 2022.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Ge- offrey J Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint Xix:::1812.08 05159, 2018
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXin:2307.09988 2023.
Florian Tramèr, Reza Shokri, Ayrton San Joaquin, Hoang Le, Matthew Jagielski, Sanghyun Hong, and Nicholas Carlini. Truth serum: Poisoning machine learning models to reveal their secrets. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Securily, pp- 2779 2092277922 2022.
31

---

Published in Transactions on Machine Learning Research (08/2024)
Alcksci Triasteyn and Boi Faltings. Federated generative privacy. IEEE Intelligent Systems, 35(4):50-5) 57, 2020.
Stacey Truex, Ling Liu, Mehmet Emre Gursoy, Lei Yu, and Wenqi Wei. Demystifying membership inference attacks in machine learning as a service. IEEE Transactions on Services Computing, 14(6))2073-200) 2019.
Dmitrii Usynin, Alexander Ziller, Marcus Makowski, Rickmer Braren, Danicl Rucckert, Ben Glocker, Geor- gios Kaissis, and Jonathan Passerat-Palmbert Adversarial interference and its miligations in privacy- preserving collaborative machine learning, Nature Machine Intelligence, 3(9):749-775) 2021.
Dmitrii Usynin, Daniel Rueckert, Jonathan Passorat-Palmbach, and Georgios Kaissis. Zen and the art of model adaptation: Low-utility-cost attack mitigations in collaborative machine learning, Proc. Priv. Enhancing Technol., 2022(1):274-290 2022.
Dmitrii Usynin, Daniel Rueckert, and Georgios Kaissis. Beyond gradients: Exploiting adversarial priors in model inversion attacks. ACM Transactions on Privacy and Security, 26(3):1-30, 2023a.
Dmitrii Usynin, Daniel Rueckert, and Giorgios Kaissis. Leveraging gradient-derived metrics for data selection and valuation in differentially private training. arXiv preprint arXic:2305.029/3. 2023b.
Gerrit van den Burg and Chris Williams. On memorization probabilistic deep generative models. Advances in Neural Information Processing Systems, 34:27916 27928, 2021.
Grant Van Horn and Pictro Perona. The devil is in the tails: Fino-grained classification in the wild. arXiv preprint XrXi:: xrXi::1709.01450, 2017.
Michael Veale, Reuben Binns, and Lilian Edwards. Algorithms that remember: model inversion attacks and data protection law. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, (7662133):201800000 2018.
Wenhao Wang, Muhammad Ahmad Kaleem, Adam Dziedzic, Michael Backes, Nicolas Papernot, and Franziska Bocnisch. Memorization in self-supervised learning improves downstream generalization. arXiv preprinl Xiv:22401. 12233, 2024.
Larry Wasserman and Shuheng Zhou. A statistical framework for differential privacy. Journal of the American Statistical Association, 105(488):375--388 2010.
Lauren Watson Chuan Guo, Graham Cormode, and Alex Sablayrolles, On the importance of difficulty calibration membership inference attacks. arXiv preprint arXiv:2111.08400 2021.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Demny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint ar Xiv: 07682, 2022.
Shelvia Wongso, Rohan Ghosh, and Mehul Motani. Using sliced mutual information study memorization and generalization in deep neural networks. In International Conference Artificial Intelligence and Statistics, pp. 11608 11629. PMLR, 2023.
Bingzhe Wu, Shiwan Zhao, Chaochao Chen, Haoyang Xu, Li Wang, Xiaolu Zhang, Guangyu Sun, and Jun Zhou. Generalization generative adversarial networks: A novel perspective from privacy protection. Advances in Neural Information Processing Systems, 32, 2019.
Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable information under computational constraints. arXiv preprint arXiv:2002. 10689, 2020.
Yihao Xue, Chaoyue Niu, Zhenzhe Zheng, Shaojic Tang, Chengfei Lyu, Fan Wu, and Guihai Chen. To- ward understanding the influence of individual clients in federated learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 10560-10567, 2021.
32

---

Published in Transactions on Machine Learning Research (08/2024)
Lu Yang, He Jiang, Qing Song, and Jun Guo. A survey on long-tailed visual recognition. International Journal of Compuler Vision, 130(7):1837-1872 2022.
Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, and Reza Shokri. Enhanced membership inference attacks against machine learning models. Tn Proceedings of the 2022 ACM SIGSAC Conference Computer and Communications Security, pp. 3093-3106, 2022.
Chih-Kuan Yeh, Joon Kim, lan En-Hsu Yen, and Pradeep K Ravikumar. Representer point selection for explaining deep neural networks. Advances in neural information processing systems, 31, 2018.
Chih-Kuan Yeh, Ankur Taly, Mukund Sundararajan, Frederick Liu, and Pradeep Ravikumar. First is better than last for language data influence. Advances in Neurol Information Processing Systems, 2022.
Samuel Yeom, Trene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Ana- lyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF), pp. 268 282. IEEE, 2018
Da Yu, Gautam Kamath, Janardhan Kulkarni, Tic-Yan Liu, Jian Yin, and Huishuai Zhang. Individual privacy accounting for differentially privale stochastic gradient descent. Transaclions Machine Learning Research, 2023.
Santiago Ganella-Béguciie Lukas Wutschitz, Shruti Tople, Victor Rühle, Andrew Paverd, Olga Ohrimenko, Boris Köpf, and Mare Brockschmidt. Analyzing information leakage of updates to natural language models. In Proceediiggs of the 2020 ACM SIGSAC conference computer and communications security, pp. 363-375, 2020.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Rocht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization, 2017.
Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Counterfactual memorization neural language models. arXiv preprint arXin:2112 12. 12938, 2021a.
Hao Zhang, Yichen Xie, Longjie Zheng, Die Zhang, and Quanshi Zhang. Interpreting multivariate shapley interactions in duns. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 10877-10886, 2021b.
Benjamin Zi Hao Zhao, Hassan Jameel Asghar, Raghav Bhaskar, and Mohamed Ali Kaafar. On inferring training data altributes in machine learning models. arXiv preprinl arXiv:1908. 10558, 2019.
Hattic Zhou, Ankit Vani, Hugo Larochelle, and Aaron Courville. Fortuitous forgetting in connectionist networks. arxio preprint arXi::2202.000 00155, 2022.
Xingchen Zhou, Ming Xu, Yiming Wu, and Ning Zheng. Deep model poisoning attack on federated learning. Future Internet, 1333773,, 2021.
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. Advances in neural information processing systems, 32, 2019.
Xiangxin Zhu, Dragomir Anguelov, and Deva Ramanan. Capturing long-tail distributions of object subcate- gorics. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 915-922, 2014.
Zheqi Zhu, Pingyi Fan, Chenghui Peng, and Khaled Letaief. Isfl: Trustworthy federated learning for non-iid data with local importance sampling. arXiv preprint arXiv:2210.0211 021111.9,,.1....1 2022.
Andy Zou, Zifan Wang, J Zico Kolter, and Malt Fredrikson. Universal and transferable adversarial attacks on aligned language models, arXie preprint arXiv:2307.1504 2023.
33

---

Published in Transactions on Machine Learning Research (08/2024)
A Appendix: Memorisation in decentralised settings
While there are no "fundamental" differences between memorisation in centralised and decentralised ML (as most decentralised ML frameworks can be seen as a large number of local centralised settings) there are two factors to take into consideration when analysing the issue of memorisation in these settings. Firstly, as participating agents (or clients) are often chosen at random, it is possible that sampling bias can severely affect both the training dynamics and the memorisation patterns. Secondly, local models need to be ag- gregated once local training over, polentially affecting the memorisation that has previously occurred at local level. In this discussion employ commonly used federated learning (FL) (Konečný et al., 2016) framework for decentralised ML training. In FL it is often the case (particularly for cross-device FL, where only a handful of sites participates in joined training) that data is not independently and identically distributed (non-IID). This, in turn, means that many local models only have access to small datasets, often with highly personalised data points. In FL, each round the central server typically selects a random subset of clients. As is impossible for the server to observe the local distribution data at each client, it can be challenging to identify the submitted update corresponds to a "typical" representative of the population if it contains a large number of atypical data points. Therefore, should the central server continuously draw the clients whose data is "atypical", the selection bias would result in model which has been trained on many samples which are more likely to be memorised. As a result, in decentralised settings, there are additional considerations such as client sampling and update weighting strategies, which should be taken into account during FL model training.
In addition, as discussed in Thakkar et al. (2020), model aggregation can have a profound effect on memo- risation capacity of the global model. In particular, one of the most frequently used aggregation strategies, namely the federated averaging, was shown to reduce the unintended memorisation (measured using canary- based techniques proposed in Carlini et al. (2019b)). Additionally, the choice of aggregation can have a direct implications on the privacy of the individuals in a decentralised setting (as shown in Geiping et al. (2020); Gupla et al. (2022)); meaning that memorisation of sensitive data can have a much higher impact on the safety of the training data. These results open up a promising area of future research in the area of decentralisod memorisation and its interplay with privacy: " "Are certain aggregation strategies preferable for guiding the memorisation process on both the local and the global levels?" While there is some evidence showing that even the simplest aggregation algorithms can have effect, more concrete research is needed in order to establish clear guidelines on this matter.
34


Reviews:
[{"review_id": "mCia9dfdpc", "reviewer": "Reviewer_PFHo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "TL;DR: I learned a lot about a range of work reading this paper, which is a main goal of a review. However, I'm not sure that the attempted synthesis really landed for me.\n\nMain Critiques:\n- The paper claims to \"unify a broad range of definitions\" - I'm not sure that I agree that it successfully does that. Sections 2/3 seem to be the spots where such a synthesis might occur, leaning on a discussion of influence - however, the takeaways are only ever really specified in a way that makes sense for a supervised L, and Section 3 is really very supervised-learning specific. It's possible that some generative models which output likelihoods could fit into this framework, but the rest of the paper makes it clear that sampling-focused models are very important here too. The unsupervised aspect of this is not so important per se, but the headline claim of the paper is this unification and to me I don't totally think it happens. Even in Section 11, the authors list unifying definitions as a highly important future direction - suggesting that contrary to the claim in the abstract, this paper did not perform such unification\n- often I found the \"key points\" to not really reflect the text above them. For instance, at the end of Sec 4.2, this subtle point around whether game theoretic metrics are or are not a proxy for memorization does not seem to be discussed in detail in the text. Other examples: Sec 5.1 - the “key point” states quite strongly that gradient-based metrics are not helpful for quantifying memorization - I think this argument should be made more clearly in the text as it seems important; Sec 5.2: the “key point” around localization is not argued so clearly in the text; Sec 9.1 - the key point is not how I would summarize this section\n\n\nSecondary Notes:\n- Fig 1b is a little confusing to me - I'm not sure it's very helpful in communicating its point. Specifically, there's a \"long tail\" aspect which I think is supposed to connect to this visual which doesn't really\n- In Sec 3.1, you draw a distinction between low-quality/mislabelled samples and \"true\" long tail samples. It's not clear to me from a pure statistical perspective that this is a real distinction (mislabelled examples that trigger this metric are kind of long tail by definition), and it would be good to get more understanding here of what this really means. This clarification would also help the analogy in Sec 7 between mislabelled examples and canaries)\n- in Sec. 3.2, you give two definitions of measuring memorization: Eq 5 which is for a model, and \"fitting random labels\" which is for a learning algorithm. It seems important to distinguish these\n- in Sec 4.2 - it's not clear what LOO influence is (is this an influence function thing or a data shapley thing?)\n\n\nSmaller points:\n- sampling format throughout is quite inconsistent - make sure you're using \\citet and \\citep appropriately, as it greatly improves readability\n\n- Sec 5: I think a more precise title would be helpful here: \"Quantities ostensibly related to memorisation\" could describe the entire paper!\n- Sec 5.1 - it would be helpful to draw a clearer distinction between IF techniques and techniques around 2nd order derivatives - they're conceptually quite related\n- Sec 5.2: I don't think it's ever stated how V-estimation gets around the issue of non-random weights \n- Sec 6: in the top paragraph, the authors state: \"different parts of the model become responsible ... for learning individual concepts ... thus it is possible to identify individual neurons associated with specific concepts\". This argument doesn't quite hold to me - in fact identifying these neurons is quite hard, if they exist at all! This is a huge challenge in the field of mechanistic interpretability.\n- p21: \"uttermost\" -> I think \"utmost\" is the more common word here\n- 7.2 - I would draw a stronger distinction between adversarial examples and data poisoning (which you don't actually name but seems to be some of what you're describing)\n- \"since pixels (on their own) are independent\" - I don't understand this claim, I wouldn't say that pixels are independent in a natural image", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper reviews a number of approaches for quantifying the importance of individual data samples in deep learning. They survey a number of definitions, focusing in particular on those inspired by influence functions, and attempt to formally unify them. They also discuss a bit around questions of if memorization can be localized, and finally discuss privacy and safety implications, as well as providing some recommendations for practitioners.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The most two important things would be 1) clarification around the attempt to unify - either making this unification clearer, or adjusting the paper's claims; and 2) ensuring that the \"key points\" in each section are actually backed up by the text: clearer signposting of the arguments with these points would be helpful on reading.", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bRmfsSKRNv", "reviewer": "Reviewer_HSGv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n1. The paper provides a thorough and systematic review of the concept of memorization in machine learning, covering its definitions, their implications, and techniques for detection and prevention. The main focus is on the self and cross influence based definitions that are task and domain agnostic. \n\n\n2. In addition to the influence-based definitions and their connection to long-tail learning, the paper covers techniques for measuring memorization through influence estimation, its connections to data valuation, and quantities related to memorization such as gradient-based influence proxies and information theoretic quantities. The paper also discusses localization and timing of memorization, inducing deliberate memorization and relation to adversarial samples, and differential privacy.\n\n\n3. Memorization is a tricky topic with its pros and cons in learning a better model and privacy. The paper covers all of these aspects nicely, rather than favoring a purely negative or positive side of it.  Based on their literature review the authors provide valuable guidelines for practitioners.\n\n\n\n\n### Weaknesses\n\n1. The breadth of topics covered is appreciated. The paper does not go deeper into the definitions. \n\n\n2. The paper favors the influence-based definition and it appears as if this definition is being posed as the standard definition moving forward. This definition is quantifying the importance of the sample in learning the desired function. But importance does not imply that the sample should be “memorized” and there are several other measures of sample importance (for instance in active learning).  To me, the influence-based definition does not appear to be a satisfactory definition of memorization. Moreover, in eq (3), it is not clear if the memorization is defined for the algorithm, for the model or for both. This is also closely related to the notion of stability ( Bousquet et al. 2002), which should be discussed here.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper provides a summary of literature on memorization in machine learning, where models incorporate training data \"as is\" rather than extracting generalisable patterns. This raises data privacy concerns, as memorized data can be identified or reconstructed. The authors unify definitions and perspectives on memorisation, discuss its relationship with model generalization, and provide methods for detecting and quantifying memorisation. They also explore implications for privacy attacks and differential privacy, and give recommendations to practitioners for mitigating risks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. It would be great to give an in-depth overview of the definitions of memorization, and their connections and then argue why one is more suitable than the other. I find the discussion on fitting random labels limited. Say, for instance, one only cares about supervised learning, in that case, is the random labels based definition most appropriate? \n\n\n2. Clarify to whom is the memorization attributed to (who is said to memorize), model $f$ or algorithm $\\mathcal{A}$ or to both? Is there any result (bound) connecting model capacity and memorization? \n\n\n3. Add discussion on stability ( Bousquet et al. 2002) and its connections with the main definition of memorization.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fwVoii65gK", "reviewer": "Reviewer_TV1U", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n* The paper is mostly very nice to read and clear on the topic.\n* The paper gives a nice overview of the closely related topics, describing how they connect to memorization and what sets them apart.\n* While it does not cite all the possible papers in all of the topics covered (which I think is reasonable given the scope of the work), I cannot find many obvious omissions in the cited work.\n\n\n### Weaknesses\n\ni) The main weakness in the paper is that the term memorization has several different meanings. While the origin of the problem is in the existing literature, which does not currently agree on a single definition, this can make it convoluted at times to keep track of the exact sense in which the term is used in different parts of the manuscript; whether it is self-influence, the ability to memorize some samples exactly (which can be measured, e.g., by using canaries in generative models), or some less-well defined general sense in which, for example, MIAs can be used to quantify how much of the data a given model has memorized.\n\nii) Besides the above, I mostly have some minor questions and comments (see below in Requested changes).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper surveys memorisation in machine learning, drawing connections to several related topics, such as, influence functions, data valuation, gradient-related measures, information theory, and privacy. The paper presents a nice and clear overview of the topic and of the related subjects, highlighting connections, some disagreements and conflicting claims in existing literature, as well as some potential directions for future research.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1) Can you clarify the use of the term \"memorization\" as used in different parts of the paper; maybe a table of the possible variants could help?\n\n2) The Key points from Sec 5.2 seem unfounded considering what is actually discussed in the text. Consider either re-wording the Key points or moving this, e.g., under Sec 6, where it has more support.\n\n3) Last part of Sec 5.1, paragraph Grads w.r.t. weights \"We note that...\"might fit better under the next paragraph Grads w.r.t. inputs?\n\n4) Sec. 9.1, p.19: \"Authors of Jagielski et al. (2022) argue that similarly to the conclusions of Feldman (2020) that ML models have to memorise, they also have to forget.\" Please add some additional explanation or clarification to this: is this a theorem, a conjecture, empirical observation or what?\n\n5) Sec. 9.3, p.20: \"While applying DP to state-of-the-art ML models comes with a utility penalty (which may be unavoidable due to the fact that DP limits memorisation).\" In some sense we already know that a hit on utility is generally necessary for any privacy protection scheme that aims to avoid catastrophic privacy leakage from the results of Dinur and Nissim (2003, Revealing Information while Preserving Privacy). Please clarify this.\n\n6) Sec. 10: it might be helpful to add cites supporting the recommendation to the list, so it would be easier to check the evidence for these.\n\n7) Sec 5.3, p11: is V-usable information the same as V-information discussed in Sec 5.3? If so, please unify the spelling to make this clear.\n\nAlso, please add the following references to the discussion (or comment on why they are not relevant):\n\nChatterjee 2018: Learning and Memorization.\nJiang et al. 2021: Characterizing Structural Regularities of Labeled Data in Overparameterized Models.\nNaseh et al. 2023: Understanding (Un)Intended Memorization in Text-to-Image Generative Models.\nWang et al. 2024: Memorization in self-supervised learning improves downstream generalization.", "broader_impact_concerns": "I have no broader concerns with this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a solid survey of memorization techniques. The initial version of the paper had some overly optimistic claims, but the current version after reviewer feedback has toned it down. I support acceptance.
Decision: Accept (poster)