Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a new measure to quantify memorization in generative models. The proposed measure is an adapted version of a memorization measure that was recently proposed for supervised learning models. The measure reflects how much the likelihood of a training sample under the trained model would decrease if that sample would be excluded from the training set. Using a VAE model as example, the authors show that training samples that are highly memorized according to their proposed measure are not necessarily samples with low likelihood (or outliers). It is further shown that the scores of the proposed memorization measure do not strongly correlate with memorization scores from a nearest neighbour test, and the authors discuss several strategies that could be used to “alleviate memorization”.
main_review: This paper is well-written; I read it with pleasure and great interest. The memorization measure proposed by the paper is clearly defined, intuitive, well-motivated and I believe it could be a useful diagnostic tool to inspect and better understand the behaviour of deep generative models. The proposed measure is not entirely original as it is adapted from a similar, recently proposed memorization measure for supervised learning, but the authors clearly state this and I do not think this diminishes the value or contribution of the current paper much.

Nevertheless, I’m afraid that I lean towards rejecting this paper, for the reasons listed below. I am however open to being challenged on these in the rebuttal phase, and to re-consider my score.

Firstly, a central claim of the paper is that memorization is fundamentally different from overfitting, but this distinction is not clear to me. It seems to me that both of these are on the same continuum. On this continuum one extreme is overfitting (corresponding to very high memorization scores) and the other extreme is an uninformative model that ignores the data (corresponding to a memorization score of 0). Let’s take Fig 1 as example. Panel (a) is an example of what the authors call memorization, and the model of this example thus presumably corresponds to relatively high memorization scores. Panel (b) is an example of what the authors call overfitting. Unless I’m mistaken, the model of this example will have higher memorization scores than the model from panel (a), suggesting that overfitting is (simply?) a more extreme version of memorization. I’m still happy to be convinced otherwise, but at the moment I do not see how these two concepts are fundamentally different.

Secondly, a central assumption of the paper seems to be that memorization in generative model is undesired and problematic, but this is again not clear to me. It is certainly not the case that a lower memorization score is always better, because a memorization score of 0 is clearly not desirable as it means that the model ignores the data. The authors seem to be mostly concerned about the question when their memorization score is problematically high; should we not also be concerned about the question when the memorization score is problematically low?

Thirdly, I think there is an important issue with the experiment described in the second paragraph of section 4.2 (related to Fig. 3). In this experiment it is shown that some of the most highly memorized examples are “as likely [under the regular model] as regular observations”. This is taken as evidence that highly memorized observations are not necessarily outliers. However, an important, unaddressed confound in this analysis is that the memorization score of an example is defined as its likelihood under the regular model minus its likelihood under the model fitted with that example left out. (That is, by definition, the higher the likelihood under the regular model, the higher the memorization score.)

Fourthly, an important limitation of the proposed memorization score is that its absolute value is hard to interpret. For example, it is not clear when a memorization score is high or problematic. It should be said that the authors clearly discuss this limitation in the paper (e.g., L165-175). However, in the final paragraph of section 4.2 (related to Fig. 4), the authors nevertheless claim that with the VAE trained with learning rate 10e-4 “high memorization can still occur”. The authors seem to justify this interpretation with a case study of a single data point with a high memorization score, but this is not convincing to me and I think it would be good if the authors could elaborate on this more.

Minor comment:
-	It might be useful to briefly discuss to what extent the proposed memorization measure could be applied to GANs.

limitations_and_societal_impact: I don’t have anything additional to add here, please see my above comment.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a point-wise memorization score for generative models, which is given by the difference of log average (under some prior over hyperparameters) probabilities assigned to the point when the model is trained with and without the observation, respectively. The authors propose to estimate this score with a sample-based estimate reminiscent of cross validation, where the data is split into folds, and the model just has to be trained over the dataset minus each of the folds, rather than over the dataset minus every single datapoint. The authors then attempt to get insights into memorization in VAEs using their score.
main_review: I think this paper addresses an interesting and potentially relevant yet understudied topic in generative modeling. Measuring memorization as changes in log likelihoods after excluding a point seems sensible, and I found the connection made to differential privacy particularly interesting. Additionally, the paper is well written and easy to follow.

My main concern with the proposed method is that the authors have no discussion whatsoever about the bias and variance of their proposed memorization score estimator, as their actual memorization score cannot be computed in closed form. While a theoretical analysis of these quantities would obviously be ideal, if not available, some sanity checks should at least be performed. For example, in lines 204-205, the authors claim "the kind of observations that are highly memorized in a particular model may be counterintuitive, and are not necessarily visually anomalous", which I think might also be explained by high variance of the estimate. Similarly, the quantiles in Figure 4b might be dominated by the variance of the estimates, rather than memorization itself. If the authors provide evidence of the robustness of their estimate during the rebuttal, I will increase my score.

In section 5, the authors discuss a relationship between their memorization score and differential privacy; and thus how methods from differential privacy could be used to reduce memorization. While there are some settings where one can clearly think why memorization would be bad (e.g. privacy, or extreme memorization of any single data point), it is not obvious to me that it is often so. In other words, is memorization often a problem we should care about in our current models? I believe the paper would benefit from a discussion about this.

======================================================================================================

EDIT 1 AFTER REBUTTAL

I have read the other reviews as well as the author's rebuttal, and have decided to increase my score. My concerns about the observed results being simply a consequence of high-variance estimates of the memorization score are diminished after the additional experiment done by the authors, even if I am still unsure that memorization can be problematic outside of settings where privacy matters.

======================================================================================================
limitations_and_societal_impact: Limitations, other than the variance one I mentioned above are properly discussed in the paper, as are potential negative societal consequences.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper investigates how probabilistic deep generative models memorize training samples. They demonstrate that highly memorized observations are not necessarily outliers, that memorization occurs early in the training process, and investigate existing nearest neighbor tests for memorization on MNIST, CIFAR-10, and CelebA.
main_review: Originality:
- This paper extends a recently proposed method for assessing memorization in neural networks (Feldman et al. 2019, Feldman et al. 2020) for generative models. The authors make it clear that the problem of memorization is different in a density estimation setting rather than a classification setting, and they adequately addressed related work in Section 2.

Quality:
- The claims in the paper were technically sound and the method made sense to me. 
- However, I think what is missing is a comparison of how existing metrics for generative models (e.g. inception score, Frechet inception distance, Kernel inception distance, etc.), model architectures, model families, etc. affect memorization. Although there was a brief discussion about it in l.112-113, the paper would be much stronger if the authors could explore how we can better leverage this memorization score to draw conclusions about a trained model (or to compare sets of trained models). I’ve talked more about this point in the “Significance” section.
- Additionally, Section 5 (“Mitigation Strategies”) doesn’t actually explore how to mitigate memorization in deep generative models, and suggests a few strategies for doing so which have yet to be explored. 

Clarity:
- The paper is well-written and easy to read, which I appreciated. The authors provide adequate details for their experiments both in the main text and supplementary material and have included source code to replicate their results.

Significance:
- Memorization in deep generative models has not been explored very much, and is an important area of research. The authors’ results are interesting because their memorization score allows for quantifying whether a single observation has been memorized in the data. I liked their comparison to existing approaches (e.g. nearest neighbors tests), and how they demonstrated that their method was capturing a different kind of signal.
- I would have appreciated a further discussion on the implications of the results. One question that I am left with is how exactly I should use this memorization score in practice. Is it more of an exploratory tool to get a sense of which examples are being memorized by my generative model? Do certain kinds of generative models (e.g. VAEs with better samples, for example according to FID score, or GANs vs. VAEs) lead to higher memorization? Given the introduction of the memorization score and the suggestions in Section 5, would the authors demonstrate how to actually reduce memorization in these models (for example, to take a model that has high memorization and to lower its memorization score)? Answers to such questions would greatly strengthen the paper.

Comments/Questions:
- What are “explicit memory cells” in language models that can facilitate memorization in l.85? Is this referring to attention/LSTMs? Technically language models are a type of generative model.
- I’m curious as to why the authors didn’t investigate GANs in more detail, since they are more prone to mode collapse and the memorization problem relative to VAEs. Was the idea to focus on likelihood-based models?
- The phrase “using importance sampling on the decoder” is confusing (l. 194). I see that in the supplementary material the authors were referring to importance sampling of the latent z’s in the approximate posterior -- this should be changed in the main text.
- I’m not sure what is the distinction between “global memorization” (which can be detected by nearest neighbor tests) and “instance-level memorization” (as per the proposed memorization score) in l.268-269. Would the authors elaborate between the two?
- What is an “outlier component” in l. 283? Would it be a loss term that prevents the model from learning from outliers/discards them during training?
- As per Figure 2, would it be safe to say that the model memorizes highly “atypical” training set examples, such as the dog with the neon green background?

Minor typos:
- z \in \mathcal{Z} \in \mathbb{R}^L in l. 186
- Should Line 10 in Algorithm 1 be V_i - U_i as per Eq. 2?

----------------------------
UPDATE: Thank you to the authors for their response, it was helpful for clarifying some points that I was confused about. I've also read the other reviewers' responses. I think this work tries to tackle a sufficiently important, understudied problem and will be nice for the community to build on, so I've increased my score to a 6.
limitations_and_societal_impact: - The authors discuss the computational barriers to implementing their proposed memorization score, and propose alternative approximations that are much more tractable in Section 3. [Do they discuss the impact of t and K beyond Section 4.1/Appendix B?] They also provide guidelines for future investigations into comparing memorization scores across datasets. 
- Limiting memorization in deep generative models is important, as it is often undesirable for synthetic samples to directly mimic training examples (for privacy concerns, for example). But the authors should also provide a discussion on potential negative societal consequences of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a novel metric to measure generalization in generative models. This metric is defined to be the ratio between  the log probability of a data sample when it was included in the training set and its logP when it was not included in the training set. The authors show that this metric is reasonable and show some interesting properties of the learning dynamics of generative models that become apparent when tracking the memorization score.
main_review: The writing is clear and easy to follow. The idea is simple and, while it is adapted from another paper on supervised learning, is being applied to a novel enough domain to have relevance for further research. Furthermore, the authors illustrate interesting points about the learning dynamics of generative models such as the fact that memorization happens early on in training and plateaus, and that memorization does not seem to be correlated with the train test generalization gap. These findings are consistent with the work around double descent, where it was proposed that the double descent phenomenon occurs as networks "memorize" training data early on in training, and then improve performance as they move to an interpolation regime. Indeed, one would expect  larger models to "memorize" more and faster, and it would be interesting to see such an experiment in this paper.

A few negative aspects:
- I am not sure what point the authors are trying to make with the comparison with nearest neighbors. They argue (correctly) that their score and NN are measuring two different aspects of memorization, but I don't see what value this long discussion adds to the paper. This connection could probably be explained more briefly and the paper expended with more experiments :)
- Related to the above, I was disappointed to see an interesting connection presented in section 5 between Differential Privacy and memorization, and how it could be used to address memorization in generative models, but no experimental results of any kind to show this point. This confuses the reader as to why this is a separate section.
limitations_and_societal_impact: The authors discuss the limitation of their work, but have not addressed any potential negative societal impact for their work. Adding a paragraph to that effect would be useful.
review_title: null

Meta-review: The paper introduces a memorization score for generative models which quantifies how the much the log-probability of a given datapoint (under the trained model) depends on its presence in the training set, and proposes an efficient cross-validation-based estimator for it. The authors applies this metric to VAEs trained on several datasets and produce several surprising findings about memorization in such models.

This is an interesting and well written paper that sheds some light on an important problem. The approach is simple and sensible and the experimental results are quite thought-provoking. The main concerns the reviewers had about the paper were the lack of guidance about what constitutes a high memorization score and the lack of precision in the claim that memorization is different from overfitting. The authors are encouraged to address these when revising the paper, which has the potential to be an influential contribution to the field.
Decision: Accept (Poster)