On Memorization in Probabilistic Deep Generative Models

1 Introduction . In the last few years there have been incredible successes in generative modeling through the development of deep learning techniques such as variational autoencoders ( VAEs ) [ 1 , 2 ] , generative adversarial networks ( GANs ) [ 3 ] , normalizing flows [ 4 , 5 ] , and diffusion networks [ 6 , 7 ] , among others . The goal of generative modeling is to learn the data distribution of a given data set , which has numerous applications such as creating realistic synthetic data , correcting data corruption , and detecting anomalies . Novel architectures for generative modeling are typically evaluated on how well a complex , high dimensional data distribution can be learned by the model and how realistic the samples from the model are . An important question in the evaluation of generative models is to what extent training observations are memorized by the learning algorithm , as this has implications for data privacy , model stability , and generalization performance . For example , in a medical setting it is highly desirable to know if a synthetic data model could produce near duplicates of the training data . A common technique to assess memorization in deep generative models is to take samples from the model and compare these to their nearest neighbors in the training set . There are several problems with this approach . First , it has been well established that when using the Euclidean metric this test can be easily fooled by taking an image from the training set and shifting it by a few pixels [ 8 ] . For this reason , nearest neighbors in the feature space of a secondary model are sometimes used , as well as cropping and/or downsampling before identifying nearest neighbors ( e.g. , [ 9–11 ] ) . Second , while there may not be any neighbors in the training set for a small selection of samples from the model , this does not demonstrate that there are no observations that are highly memorized . Indeed , in several ∗Work done while at The Alan Turing Institute . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . recent publications on deep generative models it is possible to identify observations highly similar to the training set in the illustrations of generated samples ( see Supplement A ) . Memorization in generative models is not always surprising . When the training data set contains a number of highly similar observations , such as duplicates , then it would be expected that these receive an increased weight in the model and are more likely to be generated . The fact that commonly-used data sets contain numerous ( near ) duplicates [ 12 ] therefore provides one reason for memorization of training observations . While important , memorization due to duplicates is not the focus of this work . Instead , we are concerned with memorization that arises as an increased probability of generating a sample that closely resembles the training data in regions of the input space where the algorithm has not seen sufficient observations to enable generalization . For example , we may expect that highly memorized observations are either in some way atypical or are essential for properly modeling a particular region of the data manifold . Figure 1a illustrates this kind of local memorization in probabilistic generative models . We focus on explicit density models as these are more amenable to a direct analysis of the learned probability distribution ( as opposed to implicit density models such as GANs ) . The figure shows that in certain regions of the input space the learned probability density can be entirely supported by a single , potentially outlying , observation . When sampling from the model in these parts of the space it is thus highly likely that a sample similar to an input observation will be generated . The figure also illustrates that in regions of the input space that are densely supported by closely-related observations , sampling will yield observations that resemble the input data . The change in the probability density of an observation that occurs when it is removed from the data forms the basis of the memorization score we propose in Section 3 . This form of memorization should be contrasted with what is commonly associated with memorization due to overfitting ( illustrated in Figure 1b ) . Overfitting is a global property of a model that typically occurs when it is trained for too long or with too high a learning rate ( i.e. , overtraining ) , so that a gap develops between the training and test performance . Thus , we emphasize that in generative models memorization and generalization can occur simultaneously at distinct regions of the input space , and that memorization is not necessarily caused by overtraining . To understand memorization further , consider the simple case of fitting a multivariate normal distribution . In this scenario , the presence or absence of a particular observation in the data set will have a small effect on the learned model unless the observation is an outlier . By contrast , a kernel density estimate ( KDE ) [ 13 , 14 ] of the probability density may be more sensitive to the presence or absence of a particular observation . To see why this is the case , consider that in sparsely-populated regions of the input space the KDE can be supported by a relatively small number of observations . Although deep generative models typically operate in much higher dimensional spaces than the aforementioned methods , the same problem can arise when generalizing to regions of the space that are weakly supported by the available data . Because these models are optimized globally , the model has to place some probability mass in these regions . As we will demonstrate below , it is not necessarily the case that the model places low probability on such observations , resulting in observations that are both highly memorized and not significantly less likely under the model than other observations . In this work , we extend a recently proposed measure of memorization for supervised learning [ 15 , 16 ] to probabilistic generative models and introduce a practical estimator of this memorization score . We subsequently investigate memorization experimentally , where we focus on the variational autoencoder . In our experiments we demonstrate that highly memorized observations are not necessarily outliers , that memorization can occur early during the training process , and show the connection between nearest neighbor tests for memorization and the proposed memorization score . Finally , we discuss approaches that can limit memorization in practice . 2 Related Work . Here we review work on memorization in deep learning , memorization as it relates to membership inference , the evaluation of generative models , as well as influence functions and stability . Memorization in deep learning . The observation that deep learning models can learn from patterns of random data has been a catalyst for recent efforts to understand memorization in supervised learning [ 17–19 ] . A number of approaches have been proposed to test for memorization in specific applications . In [ 20 ] memorization in language models is evaluated using a “ canary ” string ( e.g. , if “ my social security number is x ” is in the training data , how often does the model complete the prompt “ my social security number is ” using x instead of a comparable y 6= x that is not in the training set ) . Unfortunately , this approach does not translate easily to other contexts , such as images . Moreover , language models often contain explicit memory cells such as LSTMs [ 21 ] that can facilitate memorization , which are absent in most generative models . A memorization score for supervised learning was proposed in [ 15 ] , which forms the inspiration for our formulation in Section 3 . A related “ consistency score ” for supervised learning was proposed in [ 22 ] . We argue , however , that memorization in supervised learning differs fundamentally from that in generative models , as the label prediction task affects the training dynamics , and label noise is known to induce memorization in supervised learning [ 23 , 24 ] . Building on earlier work by [ 25 , 26 ] , a hypothesis test is proposed in [ 27 ] that is based on the premise that memorization has occurred when samples from the trained model are “ closer ” to the training data than observations from the test set . While this is a useful test for aggregate memorization behavior in ( a region of ) the input space , our proposed score function allows us to quantify the memorization of a single observation . Membership inference . A topic closely related to memorization is the problem of membership inference . Here , the goal is to recover whether a particular observation was part of the unknown training data set , either using knowledge of the model , access to the model , or in a black-box setting . Membership inference is particularly important when models are deployed [ 28 ] , as potentially private data could be exposed . In the supervised learning setting , [ 29 ] propose to use an attack model that learns to classify whether a given sample was in the training set . Later work [ 30 , 31 ] focused on generative models and proposed to train a GAN on samples from the target model . The associated discriminator is subsequently used to classify membership of the unknown training set . A related approach to recovering training images is described in [ 32 ] , using an optimization algorithm that identifies for every observation the closest sample that can be generated by the network . However this requires solving a highly non-convex problem , which isn ’ t guaranteed to find the optimal solution . Evaluating generative models . Memorization is a known issue when evaluating generative models , in particular for GANs [ 26 , 33 ] . Several approaches are discussed in [ 8 ] , with a focus on the pitfalls of relying on log-likelihood , sample quality , and nearest neighbors . Using the log-likelihood can be particularly problematic as it has been shown that models can assign higher likelihood to observations outside the input domain [ 34 ] . Nowadays , generative models are frequently evaluated by the quality of their samples as evaluated by other models , as is done in the Inception Score ( IS ) [ 35 ] and Fréchet Inception Distance ( FID ) [ 36 ] . Since these metrics have no concept of where the samples originate , the pathological case where a model memorizes the entire training data set will yield a near-perfect score . Motivated by this observation , [ 37 ] propose to use neural network divergences to measure sample diversity and quality simultaneously , but this requires training a separate evaluation model and there is no guarantee that local memorization will be detected . Influence & Stability . The problem of memorization is also related to the concept of influence functions in statistics [ 38 , 39 ] . Influence functions can be used to measure the effect of upweighting or perturbing an observation and have recently been considered as a diagnostic tool for deep learning [ 40 , 41 ] . However , it has also been demonstrated that influence function estimates in deep learning models can be fragile [ 42 ] . Below , we therefore focus on a relatively simple estimator to gain a reliable understanding of memorization in probabilistic deep generative models . Concurrent work in [ 43 ] focuses on influence functions for variational autoencoders based on the “ TracIn ” approximation of [ 44 ] computed over the training run . An important difference is that while the method of [ 43 ] is computed for one particular model , our memorization score applies to a particular model architecture by averaging over multiple K-fold cross-validation fits ( see Section 3 ) . Related to influence functions is the concept of stability in learning theory [ 45 ] . In particular , the point-wise hypothesis stability is an upper bound on the expected absolute change in the loss function when an observation is removed from the training set ( where the expectation is over all training sets of a given size ) . We instead focus on the change in the density of a probabilistic model when trained on a specific data set .

Summary: This paper proposes a point-wise memorization score for generative models, which is given by the difference of log average (under some prior over hyperparameters) probabilities assigned to the point when the model is trained with and without the observation, respectively. The authors propose to estimate this score with a sample-based estimate reminiscent of cross validation, where the data is split into folds, and the model just has to be trained over the dataset minus each of the folds, rather than over the dataset minus every single datapoint. The authors then attempt to get insights into memorization in VAEs using their score.