On Memorization in Probabilistic Deep Generative Models
Authors: Gerrit J.J. Van den Burg
Chris Williams
Venue: NeurIPS 2021 Poster (2021)
Keywords: Memorization
Generative Models
Density Estimation
Unsupervised Learning

Abstract: Recent advances in deep generative models have led to impressive results in a variety of application domains. Motivated by the possibility that deep learning models might memorize part of the input data, there have been increased efforts to understand how memorization arises. In this work, we extend a recently proposed measure of memorization for supervised learning (Feldman, 2019) to the unsupervised density estimation problem and adapt it to be more computationally efficient. Next, we present a study that demonstrates how memorization can occur in probabilistic deep generative models such as variational autoencoders. This reveals that the form of memorization to which these models are susceptible differs fundamentally from mode collapse and overfitting. Furthermore, we show that the proposed memorization score measures a phenomenon that is not captured by commonly-used nearest neighbor tests. Finally, we discuss several strategies that can be used to limit memorization in practice. Our work thus provides a framework for understanding problematic memorization in probabilistic generative models.

On Memorization in Probabilistic Deep Generative Models
Gerrit J.J. van den Burg" sertjjavaadenbbrr@@aaall com
Christopher K.I. Williams University of Edinburgh The Alan Turing Institute ckiw@inf ed. ac.uk
Abstract
Recent advances in deep generative models have led impressive results in variety of application domains. Motivated by the possibility that deep learning models might memorize part of the input data, there have been increased efforts to understand how memorization arises. In this work, we extend a recently pro- posed measure of memorization for supervised learning (Feldman, 2019) the unsupervised density estimation problem and adapt it to be more computationally efficient. Next, we present study that demonstrates how memorization can occur in probabilistic deep generative models such as variational autoencoders. This reveals that the form of memorization which these models are susceptible differs fundamentally from mode collapse and overfitting. Furthermore, we show that the proposed memorization score measures phenomenon that is not captured by commonly nearest neighbor tests. Finally, we discuss several strategies that can used to limit memorization in practice. Our work thus provides a framework for understanding problematic memorization in probabilistic generative models.
Introduction
In the last few years there have been incredible successes in generative modeling through the development of deep learning techniques such as variational autoencoders (VAEs) [1, 2], generative adversarial networks (GANs) [3], normalizing flows [4, 5], and diffusion networks [6, 7]. among others. The goal of generative modeling is to learn the data distribution a given data set, which has numerous applications such as creating realistic synthetic data, correcting data corruption, and detecting anomalies. Novel architectures for generative modeling are typically evaluated on how well complex, high dimensional data distribution samples from the model are. An important question in the evaluation of generative models is to what extent training observations are memorized by the learning algorithm, as this has implications for data privacy, model stability, and generalization performance. For example, in a medical setting is highly desirable to know if synthetic data model could produce near duplicates of the training data.
be learned by the model and how realistic the
common technique to assess memorization in deep generative models is to take samples from the model and compare these to their nearest neighbors in the training set. There several problems with this approach. First, has been well established that when using the Euclidean metric this test be easily fooled by taking image from the training set and shifting by few pixels [8]. For this reason, nearest neighbors in the feature space of a secondary model are sometimes used, as well cropping and/or downsampling before identifying nearest neighbors (e.g., (9-11)). Second, while there may not be any neighbors in the training set for a small selection of samples from the model, this does demonstrate that there are no observations that are highly memorized. Indeed, several
Work done while at The Alan Turing Institute.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

MM
(a) Memorization
(b) Overfitting
Figure I: Memorization and overfitting. In (a) the solid blue curve reflects the probability density when all observations are included, whereas the dashed yellow curve is the density when only yellow observations are included. The local change in density that occurs when an observation is removed from the training data indicates the extent which the model memorized the observation. The density typically associated with overfitting due to overtraining is shown by the solid red curve in (b).
recent publications on deep generative models it is possible to identify observations highly similar to the training set in the illustrations of generated samples (see Supplement A).
Memorization in generative models is not always surprising. When the training data set contains number of highly similar observations, such as duplicates, then it would be expected that these receive an increased weight in the model and are more likely be generated. The fact that commonly-un data sets contain numerous (near) duplicates [12] therefore provides one reason for memorization of training observations. While important. memorization due to duplicates is not the focus of this work. Instead, we are concerned with memorization that arises as an increased probability of generating sample that closely resembles the training data in regions of the input space where the algorithm has not seen sufficient observations to enable generalization. For example, we may expect that highly memorized observations are either in some way atypical or are essential for properly modeling a particular region of the data manifold.
Figure Ia illustrates this kind of local memorization in probabilistic generative models. We focus on explicit density models as these are more amenable to a direct analysis of the learned probability distribution (as opposed implicit density models such as GANs). The figure shows that in certain regions of the input space the learned probability density can be entirely supported by a single, potentially outlying, observation. When sampling from the model in these parts of the space is thus highly likely that sample similar to an input observation will be generated. The figure also illustrates that in regions of the input space that are densely supported by closely-related observations, sampling will yield observations that resemble the input data. The change in the probability density of an observation that occurs when it removed from the data forms the basis of the memorization score we propose in Section 3. This form of memorization should be contrasted with what is commonly associated with memorization due to overfitting (illustrated in Figure 1b). Overfitting property of model that typically occurs when it is trained for too long or with too high a learning rate (i.c., overtraining), so that gap develops between the training and test performance. Thus, we emphasize that in generative models memorization and generalization can occur simultaneously at distinct regions of the input space, and that memorization is not necessarily caused by overtraining.
global
To understand memorization further, consider the simple case of fitting multivariate normal distribu- tion. In this scenario. the presence or absence of a particular observation in the data set will have a small effect on the learned model unless the observation is an outlier. By contrast, kernel density estimate (KDE) [13, 14] of the probability density may be more sensitive to the presence or absence of particular observation. To see why this is the case, consider that in sparsely-populatee the imput space the KDE can be supported by a relatively small number of observations. Although deep generative models typically operate in much higher dimensional spaces than the aforementioned methods, the same problem can arise when generalizing to regions of the space that are weakly supported by the available data. Because these models are optimized globally, the model has to place some probability mass in these regions. As we will demonstrate below, is not necessarily the case that the model places low probability such observations, resulting in observations that are both highly memorized and not significantly less likely under the model than other observations.
regions of
In this work, we extend recently proposed measure of memorization for supervised learning [15, 16] to probabilistic generative models and introduce a practical estimator of this memorization score. We subsequently investigate memorization experimentally, where we focus on the variational autoencoder.

---

In our experiments we demonstrate that highly memorized observations are not necessarily outliers, that memorization can occur early during the training process, and show the connection between nearest neighbor tests for memorization and the proposed memorization score. Finally, we discuss approaches that can limit memorization practice.
2 Related Work
Here we review work on memorization in deep learning, memorization as it relates to membership inference, the evaluation of generative models, as well influence functions and stability.
Memorization in deep learning. The observation that deep learning models can learn from patterns of random data has been catalyst for recent efforts understand memorization in supervised learning [17-19]. A number of approaches have been proposed to test for memorization in specific applications. In [20] memorization in language models is evaluated using a "canary" string (e.g., if "my social security number is x" is in the training data, how often does the model complete the prompt "my social security number is" using z instead of a comparable y J that not in the training set). Unfortunately, this approach does not translate easily to other contexts, such as images. Moreover, language models often contain explicit memory cells such as LSTMs [21] that can facilitate memorization, which are absent in most generative models.
A memorization score for supervised learning was proposed in [15], which forms the inspiration for our formulation in Section 3. A related "consistency score" for supervised learning was proposed in [22]. We argue, however, that memorization in supervised learning differs fundamentally from that in generative models, as the label prediction task affects the training dynamics, and label noise is known to induce memorization in supervised learning [23, 24]. Building on earlier work by [25, 26]. a hypothesis test is proposed in [27] that is based on the premise that memorization has occurred when samples from the trained model are "closer" to the training data than observations from the test set. While this is a useful test for aggregate memorization behavior in (a region of) the input space, our proposed score function allows us to quantify the memorization of single observation.
Membership inference. A topic closely related to memorization is the problem of membership inference. Here, the goal is to recover whether particular observation was part of the unknown training data set, either using knowledge of the model, access to the model, or in a black-box setting, Membership inference is particularly important when models are deployed [28], as potentially private data could be exposed. In the supervised learning setting. [29] propose to use an attack model that learns classify whether a given sample was in the training set. Later work [30. 31] focused on generative models and proposed to train a GAN on samples from the target model. The associated diseriminator is subsequently used to classify membership of the unknown training set. related approach to recovering training images is described in [32], using an optimization algorithm that identifies for every observation the closest sample that can be generated by the network. However this requires solving highly non-convex problem, which isn't guaranteed to find the optimal solution.
Evaluating generative models. Memorization a known issue when evaluating generative models, in particular for GANs [26, 33J. Several approaches are discussed in [8], with focus on the pitfalls of relying on log- loglliikl,hood, sample quality, and nearest neighbors. Using the likelihold can be particularly problematic as it has been shown that models can assign higher likelihood to observations outside the input domain [34]. Nowadays, generative models are frequently evaluated by the quality of their samples as evaluated by other models, as is done in the Inception Score (IS) [35] and Fréchet Inception Distance (FID) [36]. Since these metrics have no concept of where the samples originate, the pathological case where a model memorizes the entire training data set will yield near-perfect score. Motivated by this observation, [37] propose to use neural network divergences to measure sample diversity and quality simultancously, but this requires training a separate evaluation model and there is no guarantee that local memorization will be detected.
Influence & Stability. The problem of memorization is also related to the concept of influence functions in statistics [38, 39J. Influence functions can be used to measure the effect of upweighting or perturbing an observation and have recently been considered as diagnostic tool for deep learning [40, 41]. However, has also been demonstrated that influence function estimates in deep learning models can be fragile [42]. Below, we therefore focus relatively simple estimator to gain
3

---

reliable understanding of memorization in probabilistic deep generative models. Concurrent work in [43] focuses on influence functions for variational autoencoders based on the "TracIn" approximation of [44] computed over the training run. An important difference is that while the method of [43] ] is computed for one partieular model, our memorization score applies a particular model architecture by averaging over multiple K'fold cross-validation fits (see Section 3). Related to influence functions is the concept of stability in learning theory [45]. In particular, the point-wise hypothesis stability is an upper bound on the expected absolute change in the loss function when an observation is removed from the training set (where the expectation over all training sets of a given size). We instead focus on the change in the density of probabilistic model when trained on a specific data set.
3 Memorization Score
We present a principled formulation of a memorization score for probabilistic generative models, inspired by the proposed recently in [15, 16] for supervised learning. Let A denote a randomized learning algorithm, and let a be an instance of the algorithm (i.e., a trained model). Here, A captures complete description of the algorithm, including the chosen hyperparameters, training epochs, and optimization method. The randomness in A arises from the particular initial conditions, the selection of mini batches during training, well as other factors. Denote the training data set by DD {x, /=11 with observations from X' C IR Let [n]] {1...., n and write Dz {x; x, E D, E I} for the subset of observations in the training data indexed by the set I = [m]. The posterior probability
assigned observation X' by model when trained on data set D is written p(x D a).
We are interested the posterior probability of an observation assigned by the algorithm A, not merely by an instantiation of the algorithm. Therefore we introduce the probability Pa (D) and its sampling estimate as
Pa(x D) = f p(x|| da
1 T
Σ p(x D.
for some number of repetitions T. We see that (x D) is the expectation of p(x a) over instances of the randomized algorithm A.
To facilitate meaningful interpretation of the memorization score we use the difference in log probabilities, in contrast to [15, 16]. Thus we define the leave-one-out (LOO) memorization score as
M1.0 D, = log (x D) - log (x Din)))))
This memorization score measures how much more likely an observation is when it is included in the training set compared to when it is not. For example, if ML00 1, 10. then Px D) exp(10) (x (1))) Moreover, when A/L00 (A, D, i) 0 removing the observation from the training data has no effect at x,, and when M (AA, D. the model when removed from the training data. We will abbreviate the LOO memorization score as M,000 (A, (A,D, when the arguments are clear from context.
< 0 the observation is more likely under
The memorization score in (2) is a leave-one-out estimator that requires fitting the learning algorithm A multiple times for each observation it is left out of the training data set. As this is computationally infeasible in general, we introduce a practical estimator that simplifies the one proposed in (2). Instead of using leave-one-out method or random sampling, we use K-fold approach as is done in cross volossvvlii,tion. Let I denote randomly sampled disjoint subsets of the indices [n] {1, (1...., ..) of size n/ K, such that _Kk Ia [n]. We then train the model each of the training sets DIj and compute the log probability for all observations in the training set and the holdout set Dr,,
Estimation.
Since there is randomness in the algorithm A and in the chosen folds I,, we repeat the cross-validation procedure L times and average the results. Writing Ie, for the k-th holdout index set in run é and

---

Algorithm Computing the Cross- Cross-Validatee Validated Memorization Score
Input: Algorithm A, data set D, repetitions L, folds K Output: M. Vi
1: f I.....,ddo
2: Ge Random partition of [n] into K disjoint subsets
3:
for Ie, E Ge with 1,...: do
4:
0t.* f Train A on [e
5:
Ke,k, Compute log l(g DinliTe (.,)). Vi E [n]
6: end for
7: end for
b LooMMANNExM(MM Ing LonSuMExe( (a))
8: U, + LOGGMEANEXP(TAAA
IL KK] i V.
9: V, + LOG-LEONME TE:. EE LL,EE K] I.
M.Keoll <- V4, Vi
abbreviating the respective training set as De,
x the memorization score becomes
K
1
M.K-fodd log
1s
p(x; De, ae.k)
(3)
L(K
1)
1=1 ==k= k=1
log
1
Σ
Lier, p(x;
D De, aext)
L
where 1, is the indicator function that equals 1 if u is true and otherwise. Each of the K folds where observation i is in the training set contributes to the first term in (2), and when observation i is in the holdout set it then contributes the second term. This approach is summarized in Algorithm where log probabilities are used for numerical accuracy. In practice, the number repetitions L and folds K will be dominated by the available computational resources.
When is memorization significant? A natural question is what values of the memorization score significant and of potential concern. The memorization scores can be directly compared between different algorithm settings on the same data set, for instance to understand whether changes in hyperparameters or model architectures increase or decrease memorization. Statistical measures such as the mean, median, and skewness of the memorization score or the location of, say, the 95th percentile, can informative when quantifying memorization of a particular model on particular data set, but can not necessarily be compared between data sets. In practice, we also find that the distribution of the memorization score can differ between modes in the data set, such as distinct object classes. This can be understood by considering that the variability of observations of distinct classes likely differs, which affects the likelihood of the objects under the model, and in turn the memorization score. We will return to this question in Section 6.
4 Experiments
We next describe several experiments that advance our understanding of memorization in probabilistic deep generative models, with a focus on the variational autoencoder setting. Additional results available in Supplement C. Code to reproduce our experiments can be found in an online repository.
4.1 Background
We employ the variational autoencoder (VAE) [1, 2] the probabilistic generative model in experiments, although it is important to emphasize that the memorization score introduced above is equally applicable methods such normalizing flows, diffusion networks, and other generative models that learn a probability density over the input space. The VAE a latent-variable model, where we model the joint distribution pe (x z) of an observation x E X' C E. and latent variable E Z C Rdd The joint distribution can be factorized po (x, z) (x z)p(z), and in the VAE the prior distribution p(z) is typically assumed to be a standard multivariate Gaussian. The posterior
See: https / /ttp//ggtttt.tt/////e/eeeeteghtt www.oommllll.. commalan--oe tan--tuiing--e inssiituts/nemoorzzatioo

---

(a) Low memorization
(b) Median memorization
(c) High memorization
Figure 2: Observations with low, median, and high memorization scores in the CIFAR- 10 data set, when learning the distribution with convolutional VAE. Memorization scores range from about -880 in the top left of figure (a) to about 900 in the bottom right of figure (c), with a median of 97.
distribution pz(z] x) is generally intractable, it is approximated using inference model, or encoder, To (z) x). Analogously, the model po [x z) is often referred to as the decoder. The VAE is trained by maximizing the lower bound on the evidence (ELBO), see (5), since
log Ps (x) 2 E E (x) [log Pe x, z) log go xx]
(4)
p(z)) + Eg x) Po z)] -DD (g x)
(5)
with (- (-1 ()- the Kullback-Leible (KL) divergence [46]. By choosing simple distribution for the encoder qo [z] x), such as a multivariate Gaussian, the KI. divergence has a closed-form expression,
resulting in efficient training algorithm.
We use importance sampling on the decoder [47] to approximate log Ps (x,) for the computation of the memorization score, and focus the MNIST [48], CIFAR- 10 [49], and CclebA [50] data sets. We use a fully connected encoder and decoder for MNIST and employ convolutional architectures for 100 and CelebA. For the optimization in PyTorch [52]. The memorization score estimated using L = 10 repetitions and K 10 folds. Additional details of the experimental setup and model architectures can be found in Supplement B.
use Adam [51] and we implement all models
4.2 Results
We first explore memorization qualitatively. Figure 2 shows examples of observations with low, median, and high memorization scores in the VAE model trained on CIFAR-10. While some of the highly-memorized observations may stand out as odd to a human observer, others appear not unlike those that receive a low memorization score. This shows that the kind of observations that are highly memorized particular model may be counterintuitive, and are not necessarily visually anomalous.
If highly memorized observations are always given a low probability when they are included in the training data, then would straightforward to dismiss them as outliers that the model recognizes as such. However, we find that this is not universally the case for highly memorized observations, and a sizable proportion of them are likely only when they are included in the training data. If consider observations with the 5% highest memorization scores to be "highly memorized", then we can check how many of these observations are considered likely by the model when they are included in the training data. Figure 3a shows the number of highly memorized and "regular" observations for bins of the log probability under the VAE model for CelebA, well as example observations from both groups for different bins. Moreover, Figure 3b shows the proportion of highly memorized observations each of the bins of the log probability under the model. While the latter figure shows that observations with low probability are more likely to be memorized, the former shows that considcrable proportion of highly memorized observations are as likely as regular observations when they are included in the training set. Indeed, more than half the highly memorized observations fall within the central 90% of log probability values (i.c., with log (x D) E [-14000, - 12000]).
The memorization score can be a useful diagnostic tool to evaluate the effect of different hyperpa- rameter settings and model architectures. For example, Figure 4c we illustrate the distribution of the memorization score for a VAE trained on MNIST with two different learning rates, and we show the train and test set losses during training in Figure 4a. With a learning rate of 1) = 10-3 (blue curves), a clear generalization gap can be seen in the loss curves, indicating the start of overtraining (note the test loss has not yet started to increase). This generalization gap disappears when training with the smaller learning rate of n 1 (yellow curves). The absence of a generalization gap is

---

60k
Regular High mom..
1.41
40k
Proportion
Count
0.4
20k
0.2
Regular
High
16000
14000
11000
150000 1,3000
11000
log (x(x D)
log Px(x (2)
(a) Counts
(b) Proportions
Figure (a) we show a histogram of the number of highly memorized (yellow) and regular (bluc) observations for bins of the log probability under VAE model trained on the CelebA data set (where n = 162, 770). The numbers above the bars correspond to the number of highly memorized observations in each bin. Randomly selected training observations from several bins are shown well, with dashed lines illustrating the bin where the images in particular column can be found. Images with yellow frame highly memorized whereas those with blue frame have low memorization scores. Figure (b) shows the proportion of highly memorized and regular observations for each bin.
sometimes used evidence for the absence of overfitting and memorization [53], but the distribution of the memorization scores for = 10-1 in Figure 4c shows that this is insufficient. While the memorization scores are reduced by lowering the learning rate, relatively high memorization values can still occur. In fact, the largest memorization score for y = 10 4 is about 80, and represents a shift from far outlier when the observation is absent from the training data to central inlier when it is present. 3
4.3 Memorization during training
To continue the relation between memorization and overtraining, look at how the memorization score evolves during training. In Figure 4b we show the 0.95 and 0.999 quantiles of the memorization score for the VAE trained on MNIST using two different learning rates. The quantiles are chosen such that they show the memorization score for the highly memorized observations. For both learning rates quantiles increase during training, can be expected. However
that the memorization we also see that for the larger learning rate of 7) 10 10-3 the memorization score quantiles already take on large values before the generalization gap in Figure 4a appears. This additional evidence that determining memorization by the generalization gap is insufficient, and implies that early stopping would not fully alleviate memorization. Moreover, memorization quantiles slows down with more training, which suggests that the memorization score stabilizes and does not keep increasing with the training epochs. This is reminiscent [20], who demonstrated that their metric for memorization in language models peaks when the test loss starts to increase. The difference that here memorization appears to stabilize even before this happens.
that the rate of increase for the peak
4.4 Nearest Neighbors
As discussed in the introduction, nearest neighbor illustrations are commonly used to argue that memorization is present in the model. Moreover, hypothesis tests and evaluation metrics have been proposed that measure memorization using distances between observations and model samples [26, 27]. Because of the prevalence of nearest neighbor tests for memorization, next demonstrate the relationship between our proposed memorization score and a nearest neighbor metric.
As example of nearest neighbor test, look the relative distance of observations from the training set to generated samples and observations from the validation set. Let S X be
For this particular observation, log (x D ())() ~ -178 when the observation excluded from the training data, and log (x D) z -97 when it is included, and the latter value is approximately equal to the average log probability of the other observations with the same digit.
7

---

125
50
0,509
120
10
Irain
0.000
115
1.00
110
100
20
95
-10
30
20
30
Epochs
Epocias
Monoctization
(a) Loss curves
(b) Memorization quantiles
(c) Memorization score
Figure 4: Loss curves, memorization score quantiles, and memorization score distributions for a VAE
trained on MNIST with different learning rates, = 10 (blue) and 77 10 (yellow), In (c) the vertical lines mark the 95th percentile of the memorization scores, and the axes are cropped for clarity (the maximum memorization score for 1j 10-3 is about 210 and for 10 it is about 80).
set of samples from the model and let V C X' be the validation set, with S| = V/ Denote by d X' X -> Ih distance metric, which choose here be the Euclidean distance between images in pixel space after downsampling by factor of 2. For all x, in the training set D. we then compute the ratio between the closest distance a member of the validation set and member of the sample set,
P: = minxev v d(x;, x)
(6)
minxes d(x;,x) x)
P; > 1, then the nearest neighbor of x; the sample set closer than the nearest neighbor the validation and vice versa. Thus pj > 1 suggests memorization is occurring, but it depends on sampling it is expected to be very noise at an individual data point. Investigating the average ratio for set of observations differs significantly from is example of using hypothesis testing approaches to measure memorization.
Figure 5 illustrates the relationship between P; and the memorization score MMK-fldd We see that in general there is no strong correlation between the two score functions, which can be explained by the fact that they measure different quantities. While the memorization score directly measures how much the model relies on the presence of x; the local probability density, nearest neighbor methods test how "close" samples from the model are to the training or validation data. They thus require a meaningful distance metric (which non-trivial for high-dimensional data) and are subject to variability in the sample and validation sets. We therefore argue that while nearest neighbor examples and hypothesis tests can be informative and may detect global memorization, to understand memorization at an instance level the proposed memorization score is to be preferred.
5 Mitigation Strategies
We describe two strategies that can be used to mitigate memorization in probabilistic generative models. First, the memorization score can be directly related the concept of Differential Privacy (DP) [54, 55). Note that the memorization score in (2) can be rewritten as
P (D) = exp(MM PA(xx (Pu)))))
(7)
and recall that a randomized algorithm A is e-differentially private if for all data sets D1, Dg that differ in only element the following inequality holds
PA(W D1) I (XP())PA D2), (2))
VWCX.
Since this must hold for all subsets W of X', it must also hold for the case where W (x,). Moreover, when x; is removed from D it can be expected that the largest change in density occurs at x;- It then follows that the memorization score can be bounded by employing cDDP estimation techniques when

---

Average distancc ratio (p)
1.4
Regular
High
1.2
Deusity
0.8
5000
0
500
1000
1500
20000
2500
3000
0
0.75
1.0
1.5
Memorizztiin scoze (J4/6660l)
(a) K-fold vs. P
(b) Distribution of
Figure 5: Nearest neighbor distance ratio in (6) compared to the memorization score for a VAE trained on CelebA. Due to the large number of observations, we present the average of p; for bins of the memorization score of width 50, and show error bars representing the confidence interval of the standard error of the mean of distance ratio measurements in each bin. The horizontal axis in figure (a) trims off one observation at M fold x 6500 for clarity. Figure (b) shows the distribution of the distance ratio for observations with a high memorization score (top 5%) and the regular ones.
training the generative model, as this will guarantee that MM.00 < E, Vi. The converse is however not true: observing a maximum memorization score of M. = for a particular model does not imply that the model is also e-DP. This connection of the memorization score to differential privacy offers additional support for the proposed formulation of the memorization score.
An alternative approach to limit memorization is to explicitly incorporate an outlier component in the model that would allow ignore atypical observations when learning the probability density. This technique has been previously used to handle outliers in factorial switching models [56] and to perform outlier detection in VAES for tabular data [57]. The intuition is that by including a model component with broad support but low probability (such Gaussian with high variance), the log probability for atypical observations will be small whether they are included the training data not, resulting in low memorization score. Other approaches such using robust divergence measures instead the KL-divergence in VAEs [58] may also be able to alleviate memorization.
Discussion
We have introduced principled formulation of memorization score for probabilistic generative models. The memorization score directly measures the impact of removing an observation on the model, and thereby allows to quantify the degree to which the model has memorized it. We explored how the memorization score evolves during training and how it relates to typical nearest neighbor tests, and we have shown that highly memorized observations are not necessarily unlikely under the model. The proposed memorization score can be used to determine regions of the input space that require additional data collection, to understand the degree of memorization that algorithm exhibits, or to identify training observations that must be pruned to avoid memorization by the model.
question that requires further study what constitutes "high" memorization score on a particular data set. One of the main difficulties with this is that density estimates returned by a model, and thus probability differences, are not necessarily comparable between data sets [34]. We expect that future work will focus this important question, and suggest that inspiration may be taken from work on choosing e in differential privacy [59, 60]. Furthermore, exploring the relationship between memorization and the double descent phenomenon [61, 62] could be worthy of investigation. Improving the efficiency of the estimator is also considered an important topic for future research.
If we want diversity in the samples created by generative models, then the model will have to learn to generalize to regions of the data manifold that are not well represented in the input data. Whether this is achieved by extrapolating from other regions of the space or fails due to memorization is an important question. Our work thus contributes to the ongoing effort to understand the balance between memorization and generalization in deep generative neural networks.

---

Acknowledgments and Disclosure of Funding
The authors would like to thank the anonymous reviewers for their helpful comments, and members of the AIDA project for helpful discussions. This work was supported in part by The Alan Turing Institute under EPSRC grant EP/N510129/1.
References
[1] Kingma, D. P. and Welling, M. Auto-Encoding Variational Bayes, In 2nd International Conference on Learning Representations, edited by Y. Bengio and Y. LeCun, 2014.
[2] Rezende, D. J., Mohamed, S., and Wierstra, D. Stochastic Backpropagation and Approximate Inference in Deep Generative Models. In Proceedings of the 3 International Conference on Machine Learning, edited by E. Xing and Jebara, volume 32 of Proceedings of Machine Learning Research, pp. 1 1286. PMLR, Beijing, China, 2014.
[3] Goodfellow, I., Pouget-Abadie, J.. Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative Adversarial Nets. In Advances in Neural Information Processing Systems, edited by Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger, volume 27. Curran Associates, Inc., 2014.
[4] Tabak, E. G. and Turner, C. V. A Family of Nonparametric Density Estimation Algorithms. Communications on Pure and Applied Muthematics, 66(2):145-164, 2013.
[5] Rezende, D. and Mohamed, S. Variational Inference with Normalizing Flows. In Proceedings of the 32nd International Conference on Machine Learning, edited by F. R. Bach and D. Blei, volume 37 of Proceedings of Machine Learning Research, pp. 1530-1538. PMLR, Lille, France, 2015.
[6] Dint-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. In Proceedings of the 32nd International Conference on Machine Learning. edited by R. Bach and D. Blei, volume 37 of Proceedings of Machine Learning Research, pp. 2265-2265 PMLR, Lille, France, 2015.
[7] Ho, J., Jain, A., and Abbeel, P. Denoising Diffusion Probabilistic Models. In Advances in Neural Information Processing Systems, edited by H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, volume 33. Curran Associates, Inc., 2020.
[8] Theis, L.,, van den Oord, A., and Bethge, M. A Note on the Evaluation of Generative Models. In 4th International Conference on Learning Representations, edited by Y. Bengio and Y. LeCun, 2016.
[9] Karras, T., Aila, T., Laine, S., and Lehtinen, J. Progressive Growing of GANs for Improved Quality, Stability, and Variation. In 6th International Conference Learning Representations, 2018.
[10] Brock, A., Donahue, J., and Simonyan. K. Large Scale GAN Training for High Fidelity Natural Image Synthesis. In 7th International Conference on Learning Representattioss 2019.
[11] Vahdat, A. and Kautz, J. NVAE: A Deep Hierarchical Variational Autoencoder. In Advances in Neural Information Processing Systems, edited by H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, volume 33, pp. 19667-19679. Curran Associates, Inc., 2020.
[12] Barz, B. and Denzler, J. Do We Train on Test Data? Purging CIFAR of dear-Duplicates. Journal of Imaging. 6(6):41, 2020.
[13] Rosenblatt, M. Remarks on Some Nonparametric Estimates of a Density Function. The Annals of Mathematical Statistics, 27(3):832 837, 1956.
[14] Parzen, E. On Estimation of Probability Density Function and Mode, The Annals of Mathematical Statistics, 33(3):1065 1076, 1962.
[15] Feldman, V. Does Learning Require Memorization? A Short Tale about Long Tail. arXiv preprint arviv: /907.0527. 2019.
[16] Feldman, and Zhang, C. What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation. In Advances in Neural Information Processing Systems, edited by H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and Lin, volume 33, pp. 2881-2891. Curran Associates, Inc., 2020.
10

---

[17] Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding Deep Learning Re- quires Rethinking Generalization. In Sth International Conference on Learning Representations. 2017.
[18] Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E.,, Kanwal, M. S., Maharaj. T., Fischer, A., Courville, A., Bengio, Y., and Lacoste-Julien, S. A Closer Look at Memorization in Deep Networks. In Proceedings of the 34th International Conference on Machine Learning, edited by D. Precup and Y. W. Teh, volume 70, pp. 233-242, 2017.
[19] Stephenson, C., Padhy, S., Ganesh, A., Hui, Y., Tang, H., and Chung, S. On the Geometry of Generalization and Memorization in Deep Neural Networks. In 9th International Conference on Learning Representations, 2021.
[20] Carlini, N., Liu, C., Erlingsson, U., Kos, J., and Song, D. The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks. In 28th USENIX Security Sympositm, pp. 267-284, 2019.
[21] Hochreiter, S. and Schmidhuber, J. Long Short-Term Memory. Neural Computation, 98): 1780, 1997.
[22] Jiang, Z., Zhang, C., Talwar, K., and Mozer, M. C. Characterizing Structural Regularities of Labeled Data in Overparameterizee Models. In Proceedings of the 38th International Conference on Machine Learning, edited by M. Meila and T. Zhang, volume 139 of Proceedings of Machine Learning Research, pp. 5034-5044. PMLR, 2021.
[23] Liu, S., Niles-Weed, J., Razavian, N., and Srrmandez-Grradeee C. Early-Learning Regularization Prevents Memorization of Noisy Labels. Advances Neural Information Processing Systems, edited by H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, volume 33, pp. 20331-20342. Curran Associates, Inc., 2020.
[24] Xia, X., Liu, T., Han, B., Gong, C.,, Wang, N., Ge, Z., and Chang, Y. Robust Early-Learning: Hindering the Memorization of Noisy Labels. In 9th International Conference on Learning Representations, 2021.
[25] Lopez-Paz, D. and Oquab, M. Revisiting Classifier Two-Sample Tests. In 5th International Conference on Learning Representations, 2017.
[26] Xu, Q., Huang, G., Yuan, Y., Guo, C., Sun, Y., Wu, F., and Weinberger, K. Q. An Empirical Study on Evaluation Metries of Generative Adversarial Networks. arXiv preprint 2018.
[27] Meehan, C., Chaudhuri, K., and Dasgupta, S. A Non-Parametric Test to Detect Data-Copying in Generative Models. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics, edited by S. Chiappa and R. Calandra, volume 108 of Proceedings of Machine Learning Research, pp. 3546-3556. PMLR, 2020.
[28] Fredrikson, M., Jha, S., and Ristenpart, T. Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, pp. 1322-1333, 2015.
[29] Shokri, R., Stronati, M., Song, C., and Shmatikov, V. Membership Inference Attacks against Machine Learning Models. In 2017 IEEE Symposium on Security and Privacy, pp. 3-18. IEEE, 2017.
[30] Hitaj. B., Atenicse, G., and Perez-Cruze F. Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 603 0613-618,66 2017.
[31] Hayes, J., Melis, L., Danezis, G., and De Cristofaro, E. LOGAN: Membership Inference Attacks Against Generative Models. Proceedings on Privacy Enhancing Technologies, 2019 1013313333-112, 2019.
[32] Webster, R., Rabin, J., Simon, L., and Jurie, F. Detecting Overfitting of Deep Generative Networks via Latent Recovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11273-1128, 2019.
[33] Arora, S., Risteski, A., and Zhang, Y. Do GANs Learn the Distribution? Some Theory and Empirics. In 6th International Conference Learning Representations, 2018.
11

---

[34] Nalisnick, E., Matsukawa, A., Teh, Y. W., Gorur. D., and Lakshminarayanan, B. Do Deep Generative Models Know What They Don't Know? In 7th International Conference on Learning Representations, 2019.
[35] Salimans, T., Goodfellow, I.. Zaremba, W., Cheun, V., Radford, A., and Chen, X. Improved Techniques for Training GANs. In Advances Neural Information Processing Systems. edited by D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Gamettt, volume 29, pp. 2234-2244 Curran Associates, Inc., 2016.
[36] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, GANs Trained by Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Advances in Neural Information Processing Systems, edited by Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, volume 30. Curran Associates, Inc., 2017.
[37] Gulrajani, I., Raffel, C., and Metz, L. Towards GAN Benchmarks Which Require Generalization. In 7th International Conference on Learning Representations, 2019.
[38] Hampel, F. R. The Influence Curve and Its Role in Robust Estimation. Journal of the American Statistical Association, 69(346):383-395 1974.
[39] Cook, R. D. and Weisberg. S. Characterizations of an Empirical Influence Function for Detecting Influential Cases in Regression. Technometrics, 22(4):495-508, 1980.
[40] Koh, P. W. and Liang, P. Understanding Black-box Predictions via Influence Functions. In Proceedings of the 34th International Conference on Machine Learning, edited by D. Precup and Y. W. Teh, volume 70, pp. 1885-1894, 2017.
[41] Terashita, N., Ohashi, H., Nonaka, Y., and Kanemaru, T. Influence Estimation for Generative Adversarial Networks. In 9th International Conference on Learning Representations, 2021.
[42] Basu, S., Pope, P., and Feizi, S. Influence Functions in Deep Learning Are Fragile. In 9th International Conference Learning Representations, 2021.
[43] Kong, Z. and Chaudhuri, K. Understanding Instance-based Interpretability of Variational Auto-Encoders. In Advances in Neurul Information Processing Systems, volume 34, 2021.
[44] Pruthi, G., Liu, F., Kale, S., and Sundararajan, M. Estimating Training Data Influence by Tracing Gradient Descent. In Advances in Neural Information Processing Systems, edited by H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, volume 33, pp. 19920-19930. Curran Associates, Inc., 2020.
[45] Bousquet, O. and Elisseeff, A. Stability and Generalization. Journal of Machine Learning Research, 2:499-526, 2002.
[46] Kullback, S. and Leibler, R. A. On Information and Sufficiency. The Annals of Mathematical Statistics, 22(1):79-88) 1951.
[47] Burda, Y., Grosse, R. B., and Salakhutdinov, R. Importance Weighted Autoencoders. In 4th International Conference on Learning Representations, edited by Y. Bengio and Y. LeCun, 2016.
[48] LeCun, Y., Cortes, C., and Burges, C. C. The MNIST Database of Handwritten Digits, 1998.
[49] Krizhevsky, A. Learning Multiple Layers of Features from Tiny Images. Master's thesis, University of Toronto, 2009.
[50] Liu, Z., Luo, P., Wang, X., and Tang, X. Deep Learning Face Attributes in the Wild. In Proceedings of International Conference on Computer Vision, 2015.
[51] Kingma, D. and Ba, J. Adam: A Method for Stochastic Optimization, In 3rd International Conference Learning Representations, edited by Y. Bengio and Y. LeCun, 2015.
[52] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., Deto, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. PyTorch: An Imperative Style, High- fligh-Performance Deep Learning Library. Advances Neural Information Processing Systems, edited by H. Wallach, H. Larochelle, A. Beygelzimer. F. d' Alché Buc, E. B. Fox, and R. Gameet, volume 32. Curran Associates, Inc., 2019.
[53] Wu, Y., Burda, Y., Salakhutdinov, R., and Grosse, R. B. On the Quantitative Analysis of Decoder- Based Generative Models. In 5th International Conference on Learning Representations, 2017.
12

---

[54]
Dwork, C., McSherry. F., Nissim, K., and Smith, A. Calibrating Noise to Sensitivity in Private Data Analysis. In Theory of Cryptography Conference, pp. 265-2844 Springer, 2006.
[55] Dwork, C. and Roth, A. The Algorithmic Foundations of Differential Privacy, Foundations and Trends in Theoretical Computer Science, (13-4):2114407 2014.
[56] Quinn, J. A., Williams, C. K. I., and McIntosh, N. Factorial Switching Linear Dynamical Systems Applied to Physiological Condition Monitoring. IEEE Transactions Pattern Analysis and Machine Intelligence, 31(9):1537-1551, 2009.
[57] Eduardo, S., Nazábal, A., Williams, C. K. I., and Sutton, C. Robust Variational Autoencoders for Outlier Detection and Repair of Mixed-Type Data. In International Conference on Artificial Intelligence and Statistics, pp. 4056-4066. PMIR, 2020.
[58] Akrami, H., Joshi, A. A., Li, J., Aydore, S., and Leahy, R. M. Robust Variational Autoencoder. arXiv preprint 1905.,995009900 2019.
[59] Lee, and Clifton, C. How Much is Enough? Choosing E for Differential Privacy. In International Conference Information Security, pp. 325-340. Springer, 2011.
[60] Hsu, J., Gaboardi, M., Haeberlen, A., Khanna, S., Narayan, A., Pierce, B. C., and Roth, A. Differential Privacy: An Economic Method for Choosing Epsilon. In 2014 IEEE 27th Computer Security Foundations Symposium, pp. 398-410. IEFF, 2014.
[61] Belkin, M., Hsu, D.. Ma, S., and Mandal, S. Reconciling Modern Machine-Learning Practice and the Classical -Vrriane Trade-off. Proceedings of the National Academy of Sciences, 116632):1884-115844 2019.
[62] Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., and Sutskever, I. Deep Double Descent: Where Bigger Models and More Data Hurt. In 8th International Conference on Learning Representations, 2020.
[63] Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-Based Generative Modeling through Stochastic Differential Equations. In 9th International Conference on Learning Representations, 2021.
[64] Salakhutdinov, R. and Murray, On the Quantitative Analysis of Deep Belief Networks. In Proceedings of the 25th International Conference on Machine Learning, edited by A. McCallum and S. Roweis, pp. 872 879. Omnipress, 2008
[65] Nair, V. and Hinton, G. E. Rectified Linear Units Improve Restricted Boltzmann Machines. In Proceedings of the 27th International Conference on Machine Learning, edited by J. Fürnkranz and T. Joachims, pp. 807-814. Omnipress, 2010.
[66] Uria, B., Murray, I., and Larochelle, H. RNADE:
The Real-Valued Neural Autoregressive
In Advances in Neural Information Processing Systems, edited by C. J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, volume 26, pp. 2175- 2183. Curran Associates, Inc., 2013.
Density-Estimant
[67] Papamakarios, G., Pavlakou, T., and Murray, I. Masked Autoregressive Flow for Density Estimation. In Advances in Neural Information Processing Systems, edited by I. Guyon, U. V. Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, volume 30. Curran Associates, Inc., 2017.
[68] Radford, A., Metz, L., and Chintala, S. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. 4th International Conference on Learning Representations, edited by Y. Bengio and Y. LeCun, 2016.
[69] loffe, S. and Szegedy, C. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proceedings of the 32nd International Conference on Machine Learning, edited by F. R. Bach and D. Blei, volume 37 of Proceedings of Machine Learning Research, pp. 448-456. PMLR, Lille, France, 2015.
[70] Maas, A. L., Hannun, A. Y., and Ng, A. Y. Rectifier Nonlinearities Improve Neural Network Acoustic Models. In Proceedings of the 30th International Conference on Machine Learning, edited by S. Dasgupta and D. McAllester, volume 28 of Proceedings of Machine Learning Research. PMLR, 2013.
13


Reviews:
[{"review_id": "QvZTozGGF_", "reviewer": "Reviewer_x5oB", "summary": "The authors propose a novel metric to measure generalization in generative models. This metric is defined to be the ratio between  the log probability of a data sample when it was included in the training set and its logP when it was not included in the training set. The authors show that this metric is reasonable and show some interesting properties of the learning dynamics of generative models that become apparent when tracking the memorization score.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The writing is clear and easy to follow. The idea is simple and, while it is adapted from another paper on supervised learning, is being applied to a novel enough domain to have relevance for further research. Furthermore, the authors illustrate interesting points about the learning dynamics of generative models such as the fact that memorization happens early on in training and plateaus, and that memorization does not seem to be correlated with the train test generalization gap. These findings are consistent with the work around double descent, where it was proposed that the double descent phenomenon occurs as networks \"memorize\" training data early on in training, and then improve performance as they move to an interpolation regime. Indeed, one would expect  larger models to \"memorize\" more and faster, and it would be interesting to see such an experiment in this paper.\n\nA few negative aspects:\n- I am not sure what point the authors are trying to make with the comparison with nearest neighbors. They argue (correctly) that their score and NN are measuring two different aspects of memorization, but I don't see what value this long discussion adds to the paper. This connection could probably be explained more briefly and the paper expended with more experiments :)\n- Related to the above, I was disappointed to see an interesting connection presented in section 5 between Differential Privacy and memorization, and how it could be used to address memorization in generative models, but no experimental results of any kind to show this point. This confuses the reader as to why this is a separate section.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss the limitation of their work, but have not addressed any potential negative societal impact for their work. Adding a paragraph to that effect would be useful.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "Di9YGZNDNu", "reviewer": "Reviewer_4FjL", "summary": "This paper proposes a point-wise memorization score for generative models, which is given by the difference of log average (under some prior over hyperparameters) probabilities assigned to the point when the model is trained with and without the observation, respectively. The authors propose to estimate this score with a sample-based estimate reminiscent of cross validation, where the data is split into folds, and the model just has to be trained over the dataset minus each of the folds, rather than over the dataset minus every single datapoint. The authors then attempt to get insights into memorization in VAEs using their score.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think this paper addresses an interesting and potentially relevant yet understudied topic in generative modeling. Measuring memorization as changes in log likelihoods after excluding a point seems sensible, and I found the connection made to differential privacy particularly interesting. Additionally, the paper is well written and easy to follow.\n\nMy main concern with the proposed method is that the authors have no discussion whatsoever about the bias and variance of their proposed memorization score estimator, as their actual memorization score cannot be computed in closed form. While a theoretical analysis of these quantities would obviously be ideal, if not available, some sanity checks should at least be performed. For example, in lines 204-205, the authors claim \"the kind of observations that are highly memorized in a particular model may be counterintuitive, and are not necessarily visually anomalous\", which I think might also be explained by high variance of the estimate. Similarly, the quantiles in Figure 4b might be dominated by the variance of the estimates, rather than memorization itself. If the authors provide evidence of the robustness of their estimate during the rebuttal, I will increase my score.\n\nIn section 5, the authors discuss a relationship between their memorization score and differential privacy; and thus how methods from differential privacy could be used to reduce memorization. While there are some settings where one can clearly think why memorization would be bad (e.g. privacy, or extreme memorization of any single data point), it is not obvious to me that it is often so. In other words, is memorization often a problem we should care about in our current models? I believe the paper would benefit from a discussion about this.\n\n======================================================================================================\n\nEDIT 1 AFTER REBUTTAL\n\nI have read the other reviews as well as the author's rebuttal, and have decided to increase my score. My concerns about the observed results being simply a consequence of high-variance estimates of the memorization score are diminished after the additional experiment done by the authors, even if I am still unsure that memorization can be problematic outside of settings where privacy matters.\n\n======================================================================================================", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations, other than the variance one I mentioned above are properly discussed in the paper, as are potential negative societal consequences.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "67zDXtmGEJC", "reviewer": "Reviewer_cz89", "summary": "This paper investigates how probabilistic deep generative models memorize training samples. They demonstrate that highly memorized observations are not necessarily outliers, that memorization occurs early in the training process, and investigate existing nearest neighbor tests for memorization on MNIST, CIFAR-10, and CelebA.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n- This paper extends a recently proposed method for assessing memorization in neural networks (Feldman et al. 2019, Feldman et al. 2020) for generative models. The authors make it clear that the problem of memorization is different in a density estimation setting rather than a classification setting, and they adequately addressed related work in Section 2.\n\nQuality:\n- The claims in the paper were technically sound and the method made sense to me. \n- However, I think what is missing is a comparison of how existing metrics for generative models (e.g. inception score, Frechet inception distance, Kernel inception distance, etc.), model architectures, model families, etc. affect memorization. Although there was a brief discussion about it in l.112-113, the paper would be much stronger if the authors could explore how we can better leverage this memorization score to draw conclusions about a trained model (or to compare sets of trained models). I’ve talked more about this point in the “Significance” section.\n- Additionally, Section 5 (“Mitigation Strategies”) doesn’t actually explore how to mitigate memorization in deep generative models, and suggests a few strategies for doing so which have yet to be explored. \n\nClarity:\n- The paper is well-written and easy to read, which I appreciated. The authors provide adequate details for their experiments both in the main text and supplementary material and have included source code to replicate their results.\n\nSignificance:\n- Memorization in deep generative models has not been explored very much, and is an important area of research. The authors’ results are interesting because their memorization score allows for quantifying whether a single observation has been memorized in the data. I liked their comparison to existing approaches (e.g. nearest neighbors tests), and how they demonstrated that their method was capturing a different kind of signal.\n- I would have appreciated a further discussion on the implications of the results. One question that I am left with is how exactly I should use this memorization score in practice. Is it more of an exploratory tool to get a sense of which examples are being memorized by my generative model? Do certain kinds of generative models (e.g. VAEs with better samples, for example according to FID score, or GANs vs. VAEs) lead to higher memorization? Given the introduction of the memorization score and the suggestions in Section 5, would the authors demonstrate how to actually reduce memorization in these models (for example, to take a model that has high memorization and to lower its memorization score)? Answers to such questions would greatly strengthen the paper.\n\nComments/Questions:\n- What are “explicit memory cells” in language models that can facilitate memorization in l.85? Is this referring to attention/LSTMs? Technically language models are a type of generative model.\n- I’m curious as to why the authors didn’t investigate GANs in more detail, since they are more prone to mode collapse and the memorization problem relative to VAEs. Was the idea to focus on likelihood-based models?\n- The phrase “using importance sampling on the decoder” is confusing (l. 194). I see that in the supplementary material the authors were referring to importance sampling of the latent z’s in the approximate posterior -- this should be changed in the main text.\n- I’m not sure what is the distinction between “global memorization” (which can be detected by nearest neighbor tests) and “instance-level memorization” (as per the proposed memorization score) in l.268-269. Would the authors elaborate between the two?\n- What is an “outlier component” in l. 283? Would it be a loss term that prevents the model from learning from outliers/discards them during training?\n- As per Figure 2, would it be safe to say that the model memorizes highly “atypical” training set examples, such as the dog with the neon green background?\n\nMinor typos:\n- z \\in \\mathcal{Z} \\in \\mathbb{R}^L in l. 186\n- Should Line 10 in Algorithm 1 be V_i - U_i as per Eq. 2?\n\n----------------------------\nUPDATE: Thank you to the authors for their response, it was helpful for clarifying some points that I was confused about. I've also read the other reviewers' responses. I think this work tries to tackle a sufficiently important, understudied problem and will be nice for the community to build on, so I've increased my score to a 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- The authors discuss the computational barriers to implementing their proposed memorization score, and propose alternative approximations that are much more tractable in Section 3. [Do they discuss the impact of t and K beyond Section 4.1/Appendix B?] They also provide guidelines for future investigations into comparing memorization scores across datasets. \n- Limiting memorization in deep generative models is important, as it is often undesirable for synthetic samples to directly mimic training examples (for privacy concerns, for example). But the authors should also provide a discussion on potential negative societal consequences of their work. \n", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "1s495e7o4_", "reviewer": "Reviewer_3dmL", "summary": "This paper proposes a new measure to quantify memorization in generative models. The proposed measure is an adapted version of a memorization measure that was recently proposed for supervised learning models. The measure reflects how much the likelihood of a training sample under the trained model would decrease if that sample would be excluded from the training set. Using a VAE model as example, the authors show that training samples that are highly memorized according to their proposed measure are not necessarily samples with low likelihood (or outliers). It is further shown that the scores of the proposed memorization measure do not strongly correlate with memorization scores from a nearest neighbour test, and the authors discuss several strategies that could be used to “alleviate memorization”.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well-written; I read it with pleasure and great interest. The memorization measure proposed by the paper is clearly defined, intuitive, well-motivated and I believe it could be a useful diagnostic tool to inspect and better understand the behaviour of deep generative models. The proposed measure is not entirely original as it is adapted from a similar, recently proposed memorization measure for supervised learning, but the authors clearly state this and I do not think this diminishes the value or contribution of the current paper much.\n\nNevertheless, I’m afraid that I lean towards rejecting this paper, for the reasons listed below. I am however open to being challenged on these in the rebuttal phase, and to re-consider my score.\n\nFirstly, a central claim of the paper is that memorization is fundamentally different from overfitting, but this distinction is not clear to me. It seems to me that both of these are on the same continuum. On this continuum one extreme is overfitting (corresponding to very high memorization scores) and the other extreme is an uninformative model that ignores the data (corresponding to a memorization score of 0). Let’s take Fig 1 as example. Panel (a) is an example of what the authors call memorization, and the model of this example thus presumably corresponds to relatively high memorization scores. Panel (b) is an example of what the authors call overfitting. Unless I’m mistaken, the model of this example will have higher memorization scores than the model from panel (a), suggesting that overfitting is (simply?) a more extreme version of memorization. I’m still happy to be convinced otherwise, but at the moment I do not see how these two concepts are fundamentally different.\n\nSecondly, a central assumption of the paper seems to be that memorization in generative model is undesired and problematic, but this is again not clear to me. It is certainly not the case that a lower memorization score is always better, because a memorization score of 0 is clearly not desirable as it means that the model ignores the data. The authors seem to be mostly concerned about the question when their memorization score is problematically high; should we not also be concerned about the question when the memorization score is problematically low?\n\nThirdly, I think there is an important issue with the experiment described in the second paragraph of section 4.2 (related to Fig. 3). In this experiment it is shown that some of the most highly memorized examples are “as likely [under the regular model] as regular observations”. This is taken as evidence that highly memorized observations are not necessarily outliers. However, an important, unaddressed confound in this analysis is that the memorization score of an example is defined as its likelihood under the regular model minus its likelihood under the model fitted with that example left out. (That is, by definition, the higher the likelihood under the regular model, the higher the memorization score.)\n\nFourthly, an important limitation of the proposed memorization score is that its absolute value is hard to interpret. For example, it is not clear when a memorization score is high or problematic. It should be said that the authors clearly discuss this limitation in the paper (e.g., L165-175). However, in the final paragraph of section 4.2 (related to Fig. 4), the authors nevertheless claim that with the VAE trained with learning rate 10e-4 “high memorization can still occur”. The authors seem to justify this interpretation with a case study of a single data point with a high memorization score, but this is not convincing to me and I think it would be good if the authors could elaborate on this more.\n\nMinor comment:\n-\tIt might be useful to briefly discuss to what extent the proposed memorization measure could be applied to GANs.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I don’t have anything additional to add here, please see my above comment.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a memorization score for generative models which quantifies how the much the log-probability of a given datapoint (under the trained model) depends on its presence in the training set, and proposes an efficient cross-validation-based estimator for it. The authors applies this metric to VAEs trained on several datasets and produce several surprising findings about memorization in such models.

This is an interesting and well written paper that sheds some light on an important problem. The approach is simple and sensible and the experimental results are quite thought-provoking. The main concerns the reviewers had about the paper were the lack of guidance about what constitutes a high memorization score and the lack of precision in the claim that memorization is different from overfitting. The authors are encouraged to address these when revising the paper, which has the potential to be an influential contribution to the field.
Decision: Accept (Poster)