Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: **Summary**
Motivated by the observation that prevalent metrics (Inception Score, Frechet Inception Distance) used to assess the quality of samples obtained from generative models are gameable (due to either the metric not correlating well with visually assessed sample quality or the metric being susceptible to training sample memorization), the authors conduct a large scale “controlled” study to assess the gameability of said metrics. The authors conducted a competition and subsequently analyzed how approaches tend to cheat so as to obtain higher FID scores. Furthermore, to assess the extent of memorization w.r.t. the FID score, the authors propose a new metric — Memorization-Informed Frechet Inception Distance (MiFID) — which takes into account sample memorization w.r.t. a reference set. The authors conclude on a few notable observations — (1) unintentional memorization in generative models is a serious and prevalent issue; (2) the choice of latent space used to compute FID based scores can make a significant difference.

**Strengths**
- The paper is generally well-written and easy to follow for the most part. The authors do a good job of walking the reader through the design of the competition, the proposed metric and design choices adopted to ensure fair characterization of sample-memorization and cheating strategies.
- The choice of memorization assessment defined and adopted for the FID score in section 3.1 is intuitive and well-motivated. I particularly appreciated how the authors identified the fact that assigning a memorization penalty is itself subjective to the choice of feature space, reference set and the memorization threshold. Furthermore, the constraints assigned on the challenge submissions seem reasonable to ensure fair comparisons across multiple submissions. I would also like to highlight that in addition to automating “memorization” detection to the extent possible, the authors went the extra mile to manually review code-submissions to further identify false negatives — all leading to a concise characterization of the strategies adopted to cheat or memorize.
- I think this large-scale study (although operating under several constraints due to it’s controlled nature) is going to be quite beneficial to the community. Careful inspection of the submissions reveals several insights (and caveats) associated with current metrics / results for generative models that can perhaps motivate the community to take measures to establish more reliable benchmarks for generative modeling.

**Weaknesses**
I will mostly highlight weaknesses and other points that likely have clarity issues associated in the current draft.
- While the choice of constraints imposed on the challenge submissions for the study here seem reasonable, I’m concerned the study only highlights issues associated with approaches that can likely be executed / implemented under these constraints (pointing to limited computation time, isolated containerization and restricted access to pre-trained models or additional data). While this is a good starting point, it’s unclear how well insights from the study may generalize to approaches that operate outside of these constraints. Do the authors have any thoughts on this?
- In addition to other factors, the choice of the memorization margin (as described in section 3.2.1) seems to depend not only on the reference set but also on the “sets” of generated images. This likely affects all the following stages of re-calibrating the score after identifying false-positives and negatives. Can the authors comment on how dependent is this on the number and kind of submissions? This will likely inform the extent to which the analysis setup (including the metric) can be adopted / extended to other kinds of images, datasets, etc (and maybe even more relaxed constraints).
- Minor comments — (1) The paper would benefit from making the distinction between intended and unintended memorization clear early on in the introductory section; (2) I’m curious about how the constraints posed in the section 3 on challenge were verified. 
review_title: Promising study assessing how memorization can help game metrics for generative models

final_score: rating: 3: Clear rejection
review_content: This paper studied the memorization issue of generative modeling. It proposed a benchmark for a public generative modeling competition and observed how participants attempted to game the FID. 

It seems the paper only targeted FID. Actually in GAN evaluation, FID is not the only metric that are widely used. In addition, I doubt how the proposed competition can show the impact of memorization. Also, it is not clear how MiFID really evaulates the generilization ability. In addition, the title mentioning generative modeling is overclaimed (only GANs). 

The writting of the paper is not good. The introducation spent too many paragraphes on describing the background of GANs but failed to clarify the motivation/intuiation clearly. It also used several offencive words such as "cheating". 

Overall, I think this work has limit impact to the community and not provided deep insights to the audience. Based on these facts, I make my rating. 
review_title: Official Review of Paper 

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper investigates memorization/overfitting in GANs and proposes a new metric (MiFID) to identify memorization in trained GAN models automatically. The topic of memorization in GANs is an important one and certainly one that does require more work and research.

Contributions/Novelty:
- The authors introduce a new metric to automatically evaluate memorization in GANs
- A competition is held to collect data to evaluate different kinds of memorization in GANs

Background:
The background section is quite short and only looks into the IS and FID, but not into the central topic of the paper: memorization/overfitting in GANs
- What exactly is memorization in GANs? How do you define/measure it? The concept of memorization in GANs seems to be a bit obscure in some parts of the paper (see my remarks to e.g. table 1 later on)
- What about other approaches that have looked into overfitting? E.g. [1,2]

[1] Webster, R., Rabin, J., Simon, L., & Jurie, F. (2019). Detecting overfitting of deep generative networks via latent recovery. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 11273-11282).

[2] Meehan, C., Chaudhuri, K. & Dasgupta, S.. (2020). A Three Sample Hypothesis Test for Evaluating Generative Models. Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics

Competition:
The section on the competition is very sparse and I would appreciate more details:
- Why are particiapnts only allowed to train for 9h? Why not more or less? Why is the training time relevant for memorization in GANs?
- Why use a new dataset of only dog images? Why not also test with other images? Why not test with TinyImageNet or something similar?
- Why use only 20,000 images? That's a pretty small dataset (which might make memorization more likely in the first place)
- Who are the participants (academics/industry, advanced/beginner) and how were participants found/recruited?
- What was the overall goal of the competition? Identify memorization problems in GANs? Find the best model given the constraints on hardware, training time, small dataset, ...?

MiFID:
- From section 3.1: "This motivated the addition of a ”memorization-aware” metric that penalizes models producing images too similar to the training set." -> How exactly is "too similar to training set" defined?
- Why do you use the cosine similarity specifically? Why not use other metrics such as LPIPS [3] or SSIM [4]? Have you tried using other metrics?
- Technically speaking the memorization distance is not a distance since it is not symmetrical, maybe call it "divergence" instead
- For the formula/calculation of MiFID: I might misunderstand: but why use the "min" and not the "max" of the cosine distance? By using min you look for the most different sample in the training set for each generated image and use that to calculate the distances; if I'm looking for memorization wouldn't it make more sense to look for the most similar image (the memorized one) and use that for calculating the distance?

[3] Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018). The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 586-595).

[4] Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004). Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4), 600-612.

Results:
- Regarding memorization methods (Table 1): why are AEs considered cheating? Yes, they "memorize" the training set, but if trained correctly (e.g. VAEs) they can also generate novel images; maybe this shows that "memorization" in this context is not well-defined: memorization (for me) means that the generative model exhibits a mode-collapse towards the training images, i.e. no matter what kind of noise input you give the model it only generates training images; this is not the case for VAEs: they will not only generate training images for any noise input you give them...
- Similarly: I also don't understand why using data augmentation during training of generative models is "cheating" and leads to overfitting; actually, there are a lot of recent works that show how data augmentation can be helpful during GAN training, especially for smaller datasets as in this case (only 20,000 images) [5,6,7,8]

[5] Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., & Aila, T. (2020). Training generative adversarial networks with limited data. arXiv preprint arXiv:2006.06676.

[6] Tran, N. T., Tran, V. H., Nguyen, N. B., Nguyen, T. K., & Cheung, N. M. (2020). Towards Good Practices for Data Augmentation in GAN Training. arXiv preprint arXiv:2006.05338.

[7] Zhao, Z., Zhang, Z., Chen, T., Singh, S., & Zhang, H. (2020). Image Augmentations for GAN Training. arXiv preprint arXiv:2006.02595.

[8] Zhao, S., Liu, Z., Lin, J., Zhu, J. Y., & Han, S. (2020). Differentiable augmentation for data-efficient gan training. arXiv preprint arXiv:2006.10738.

Insights:
- Have you tried to use this method to also evaluate current pretrained models (BigGAN, StyleGAN, etc) for memorization roblems and visualize them?
- I don't see why the high correlation between memorizsation distance and FID is a problem; this is exactly what we would expect; I am missing experiments that clearly show that memorization is actually a problem in these models, i.e. that they only generate images from the training set

General remarks:
- I'm not convinced the results of the competition are reliable to evaluate memorization problems in GANs, especially since it is not clear who participated in the competition, why there are so many restrictions that should not directly affect memorization as such (small amount of training time and only one GPU), and the dataset for evaluation is very small (20,000 images) and only contains dogs; I would at least expect experiments on larger and different datasets and an explanation why restricting the resources is important to evaluate memorization/overfitting in GANs
- I'm also not sure I understand what exactly the authors define as "cheating"/memorization: e.g. autoencoders and data augmentation are not cheating in my perspective and also do not automatically lead to memorization; this leads back to my point that I am missing a concrete working definition on memorization in GANs in this paper
- While memorization is a serious issue for generative models I am missing a clear definition of what memorization means in this context; the authors show that there is a high correlation between the FID and their memorization distance, but this is not surprising: generative models *should* generate images close to the training set (but not "from" the training set); I feel this work lacks a clear distinction between "memorization" (only generates images from the training set) and other models (such as e.g. VAEs but also GANs) that might generate images from the training set but are also capable of generating images that do not occur in the training set (often tested by doing interpolation between two images) -> a high correlation between FID and memorization distance is therefore expected, since GANs should learn to generate images that are close to the training set -> I feel there needs to be a way to automatically detect GANs that are *not* capable of generating images that are *reasonably far* away from the training set but still realistic (if the model also generated some training images I don't think this is necessarily a big issue)

Minor:
- Why is this a table 1 a table and not a simply a bullet list/enumeration?
review_title: Review of "A Large-scale Study on Training Sample Memorization in Generative Modeling"

Meta-review: The paper proposes a competition on generative models on a new dataset to study memorization in generative models and propose a  new metric Memorization-Informed Frechet Inception Distance (MiFID). 

While this is an important topic, reviewers raised multiple issues and concerns regarding 1)  the metric definition (that it needs to be max and not min, this was acknowledged in the rebuttal but not updated in the paper) , 2)  how this competition is ran in terms of the definition of "cheating",  that the setup is not controlled and only constraining the time of training 3)  the notion of MiFID is depending on the sets of samples considered and the feature extractor used. 

Some other reviewers raised concerns that the paper is only concerned by FID and not other metrics , and that it was only verified on GANs.  We hope the authors will address those concerns and submit the paper to an upcoming venue.
Decision: Reject