A Large-scale Study on Training Sample Memorization in Generative Modeling
Authors: Ching-Yuan Bai
Hsuan-Tien Lin
Colin Raffel
Wendy Kan
Venue:  (2021)
Keywords: GAN
generative adversarial networks
generative modeling
memorization

Abstract: Many recent developments on generative models for natural images have relied on heuristically-motivated metrics that can be easily gamed by memorizing a small sample from the true distribution or training a model directly to improve the metric.
In this work, we critically evaluate the gameability of the benchmarking procedure by running a competition which ultimately resulted in participants attempting to cheat. Our competition received over 11000 submitted models which allowed us to investigate memorization-aware metrics for measuring generative model performance. Specifically, we propose the Memorization-Informed Frechet Inception Distance (MiFID) and discuss ways to ensure that winning submissions were based on genuine improvements in perceptual quality. We evaluate the effectiveness of our benchmark by manually inspecting the code for the 1000 top-performing models and labeling different forms of memorization that were intentionally or unintentionally used. To facilitate future work on benchmarking generative models, we release generated images and our labels for these models as well as code to compute the MiFID metric.

Under review as a conference paper at ICLR 2021
A LARGE-SCALE STUDY ON TRAINING SAMPLE MEMORIZATION IN GENERATIVE MODELING
Anonymous authors Paper under double-blind review
ABSTRACT
Many recent developments on generative models for natural images have relied on heuristically-motivee metrics that be easily gamed by memorizing small sample from the true distribution or training a model directly to improve the metric. In this work, we critically evaluate the gameability of such metrics by running a competition that ultimately resulted in participants attempting to cheat. Our competition received over 11000 submitted models and allowed us to investigate both intentional and unintentional memorization. To stop intentional memorization, we propose the "Memorization-Infoe Informedd Fréchet Inception Distance" (MiFID) new memorization-awee aware metric and design benchmark procedures to ensure that winning submissions made genuine improvements in perceptual quality. Fur- thermore, manually inspect the code for the 1000 top-performing models to understand and label different forms of memorization. The inspection reveals that unintentional memorization is serious and common issue in popular generative models. The generated images and our memorization labels of those models as well code compute MiFID released facilitate future studies benchmarking generative models.
INTRODUCTION
Recent work on generative models for natural images has produced huge improvements in image quality, with some models producing samples that can be indistinguishable from real images (Karras)
al. 2017: 2019a b. Brock et al. 2018 Kingma & Dhariwal 2018 Maalde et al. 2019 Menick
& Kalchbrenner 2018, Razavi et al.) 2019). Improved sample quality is important for tasks like
super-resolution Ledig et al. (017) ) and inpainting YYu et al. 2019). as well as creative applications
(Park et al. 2019; Isola et al. 2017: Zhu et al. 2017a b). These developments have also led to useful
algorithmic advances on other downstream tasks such as semi-supervised learning (Kingma et al.
2014 Odena 2016 Salimans al. 2016 Izmailov et al. [2119) representation learning (Dumoulin
et al.
2016 Donahue et al., 2016 Donahue & Simonyan 2019).
Modern generative models utilize variety of underlying frameworks, including autoregressive models (Oord et al., 2016), Generative Adversarial Networks (GANs; [Goodfellow et al., (014). flow-based models (Dinh
2014 Rezende Mohamed 2015). and Variational Autoencoders (VAEs; Kingma & Welling 2013 Rezende et al. 2014). This diversity of approaches, combined with the philosophical nature of evaluating generative performance, has prompted the development of heuristically- mutriiic metrics designed to measure the perceptual quality of generated samples such as the Inception Score (IS; (Salimanss al.) (016) or the Fréchet Inception Distance (FID; Heuseel et al. 2017). These metrics are used in a benchmarking procedure where "state-of-the-ar" results are claimed based on better score on standard datasets.
[VAEs:KKnggaa
Indeed, much recent progress in the field of machine learning as whole has relied useful benchmarks on which researchers can compare results. Specifically, improvements on the benchmark metric should reflect improvements towards useful and nontrivial goal. Evaluation of the metric should be a straightforward and well-defined procedure so that results can be reliably compared. For example, the ImageNet Large-Scale Visual Recognition Challenge (Deng et al. 2009 Russakovsky (et al., 2015)) has a useful goal (classify objects in natural images) and a well-defined evaluation procedure (top-1 and top-5 accuracy of the model's predictions). Sure enough, the ImageNet

---

Under review as a conference paper at ICLR 2021
benchmark has facilitated the development of dramatically better image classification models which have proven to be extremely impactful across a wide variety of applications.
Unfortunately, some of the commonly-use' benchmark metrics for generative models of natural images do not satisfy the aforementioned properties. For instance, although the IS is demonstrated to correlate well with human perceived image quality (Salimans et al. 2016), BBrrrttt & Sharma (2018) points out several flaws of the IS when used as a single metric for evaluating generative modeling performance, including its sensitivity to pretrained model weights which undermines generalization capability. Seperately, directly optimizing a model to improve the IS can result in extremely perealistic-looking images (Barratt & Sharma 2018) despite resulting
better score.
is also well-known that if a generative model memorizes images from the training set (i.e. producing non-novel images), it will achieve a good IS (Gulrajani et al. 2018). On the other hand, the FIID widely accepted as an improvement over IS due to its better consistency under perturbation (Heusel jt al., 2017). However, there no clear evidence of the FID resolving any of the flaws of the IS. large-scale empirical study is necessary to provide robust support for understanding quantitatively how flawed the FID is.
Motivated by these issues, we want to benchmark generative models in the "real world", i.e. outside of the research community by holding a public machine learning competition. To the extent of our knowledge, no large-scale generative modeling competitions have ever been held, possibly due to the immense difficulty of identifying training sample memorization in a efficient and scalable manner. We designed more rigorous procedure for evaluating competition submissions, including memorization-awiin variant of FID for autonomously detecting cheating via intentional memorization. We also manually inspected the code for the top 1000 submissions to reveal different forms of intentional or unintentional cheating, to ensure that the winning submissions reflect meaningful improvements, and to confirm efficacy of our proposed metric. We hope that the success of the first-ever generative modeling competition can serve as future reference and stimulate more research in developing better generative modeling benchmarks.
Our main goal in this paper is to conduct an empirical study on issues of relying on the FID as benchmark metric to guide the progression of generative modeling. In Section 2 we briefly review the metrics and challenges of evaluating generative models. In Section 3 we explain in detail the competition design choices and propose a novel benchmarking metric, the Memorization-Inforee Fréchet Inception Distance (MiFID). We show that MiFID enables fast profiling of participants that intentionally memorize the training dataset. In Section we introduce a dataset released along with this paper that includes over hundred million generated images and manual labels obtained by painstaking code review. In Section 5 we connect phenomena observed in large-scale benchmarking of generative models in the real world back to the research community and point out crucial but neglected flaws in the FID.
2 BACKGROUND
In generative modeling, our goal is to produce a model po(x)) (parameterized by 0) of some true distribution p(z). We are not given direct access to p(z); instead, we are provided only with samples drawn from it x ~ p(r). In this paper, we will assume that samples æ from p(x) are 64-by-64 pixel natural images, i.e. E [66x64 A common approach is to optimize 0 so that pa (x) assigns high likelihood to samples from p(z). This provides a natural evaluation procedure which measures the likelihood assigned by pz(z) to samples from p(x) that were held out during the optimization of f. However, not all models facilitate exact computation of likelihoods. Notably, Generative Adversarial Networks (GANs) (Goodfellow et al. 2014) learn an "implicit" model of p(x:) from which we can draw samples but that does not provide an exact (or even an estimate) of the likelihood for a given sample. The GAN framework has proven particularly successful at learning models which can generate extremely realistic and high-resolution images, which leads to a natural question: How should we evaluate the quality of generative model if we can't 't compute the likelihood assigned to held-out samples?
This question has led the development of many alternative ways evaluate generative models (Borji 2019). A historically popular metric, proposed in (Salimans et al. 2016), is the Inception Score (IS) which computes
TS(po) Ex (x) (IN()l)))))) [N()))
2

---

Under review as a conference paper at ICLR 2021
where (N) is the conditional probability of a class label y assigned datapoint by a pre- trained Inception Network (Szegedy et al. 2015). More recently, (Heusel et al. 2017) proposed the Fréchet Inception Distance (FID) which better correlates with perceptual quality. The FID uses the estimated mean and covariance of the Inception Network feature space distribution to calculate the distance between the real and fake distributions up second order. The FID between the real images r and generated images is computed as:
F) ||,, Hal2+ TT -
where My and Mg are the mean of the real and generated images in latent space, and E,, and Eg are the covariance matrices for the real and generated feature vectors. A drawback of both the IS and FII) is that they assign a very good score model which simply memorizes small and finite sample from p(x) (Guraajani et al. 2018), an issue we address section 3.1
GENERATIVE MODELING COMPETITION DESIGN
We designed the first generative model competition where participants were invited generate realis- tic dog images given 20,579 images of dogs from ImageNet (Russakovsky al.) 2015). Participants were required to implement their generative model in a constrained computation environment to prevent them from obtaining unfair advantages. The computation environment was designed with:
Limited computation resource (9 hours on a NVIDIA P100 GPU for each submission) since generative model performance is known be highly related to the amount of computational resources used (Brock et al. (2118)
Isolated containerization avoid continuous training by reloading model checkpoints from previous sessions
No access to external resources (i.e. the internet) to avoid usage of pre-trained models or additional data
Fach submission is required to provide 10,000 generated images of dimension 64 64 x 3 and receives a public score return. Participants are allowed to submit any number of submissions during the two-month competition. Before the end of the competition, each team is required to choose two submissions, and the final ranking is determined by the better private score (described below) out of the two selected submissions.
In the following sections, we discuss how the final decisions were made regarding pretrained model selection (for FID feature projection) and how we enforced penalties to ensure the fairness of the competition.
3.1 MEMORIZATIOON FRÉCHET INCEPTION DISTANCE (MIFID)
The most crucial part of the competition is the performance evaluation metric to score the submissions, To assess the quality of generated images, we adopted the Fréchet Inception Distance (Heusel et al.) 2017) which a widely used metric for benchmarking generative tasks. Compared the Inception Score (Salimans et al. 2016), the FID) has the benefits of better robustness against noise and distortion and more efficient computation (Borji 2019).
For a generative modeling competition, a good metric not only needs to reflect the quality of generated samples but must also allow easy identification of cheating with as little manual intervention as possible. Many forms of cheating were prevented by setting up the aforementioned computation environment, but even with these safeguards it would be possible to "game" the FID score. Specifically, predicted that memorization of training data would be major issue, since current generative model evaluation metrics such as IS or FID are prone to reward high scores to memorized instances (Gulrajani al.) 2018). This motivated the addition of a "memorizationnaware" metric that penalizes models producing images too similar to the training set.
Combining memorization-awwe and generation quality components, we introduced the Memorization- Informed Fréchet Inception Distance (MiFID) as the metric used for the competition:
MiFID St) m- S.) FID(S,, St)
3

---

Under review as a conference paper at ICLR 2021
where Sy is the generated set, St is the original training set, FID is the Fréchet Inception Distance, and m, is the memorization penalty which we discuss in the following section.
3.1.1 MEMORIZATION PENALTY
To capture the similarity between two sets of data - in our case, generated images and original training images started by measuring similarity between individual images. Cosine similarity, the inner product of two vectors, is commonly used similarity measure. It is easy to implement with high computational efficiency (with existing optimized BLAS libraries) which is ideal when running competition with hundreds of submissions each day. The value is also bounded, making it possible to intuitively understand and compare the degree of similarity.
We define the memorization distance of a target projected generated set Sg S C with respect to reference projected training set S, S IRd as subtracted by the mean of minimum (signed cosine) similarity of all elements Sy and Sr. Intuitively, lower memorization distance is associated with more severe training sample memorization. Note that the distance is asymmetric i.e. s(S.. St) & s(S,, Sg). but this is irrelevant for our use-case.
min (xig: mi) [zyl [a2a
SSgr S.) =
Σ E.
S
#.ES.
We hypothesize that cheating submissions with intentional memorization would generate images with significantly lower memorization distance. To leverage this idea, only submissions with distance lower than specific threshold are penalized. Thus, the memorization penalty m, is defined
(e < 1); if (Sg; St) <T
m, (S.z St) = s1S,
otherwise
More memorization (subceeding the predefined threshold r) will result in higher penalization. Dealing with false positives and negatives under this penalty scheme further discussed in Section 3.2
3.1.2 PREVENTING OVERFITTING
In order prevent participants of the competition from overfitting to the public leaderboard, used different data for calculating the public and private score and we generalized the FID to use any visually-relevant latent space for feature projection. Specifically, we selected different pre-trained ImageNet classification models for public and private score calculation. For the same score, the same pre-trained model used for both feature projection for the memorization penalty and for standard FID calculation. Inception V3 was used for public score following past literature, while the private score used NASNet (Zoph et al. 2018). We will discuss how NASNet was selected in Section 3.2.1
3.2 DETERMINING FINAL RANKS
After the competition was closed submission there two-week window re-process all the submissions and remove ones violating the competition rules (e.g. by intentionally memorizing the training set) before the final private leaderboard was announced. The memorization penalty term in MiFID was efficiently configured for re- re-running with a change of the parameter allowing finalizing of results within a short time frame.
3.2. SELECTING TRA-IRDN MODEL FOR THE PRIVATE SCORE
As commonly assumed that FID generally invariant to the projection space, the pre-trained model for private score was selected to best combat cheating via training set memorization. The goal is to separate cheating and non-cheating submissions as cleanly as possible. We calculate the memorization distance for subset of submissions projected with the chosen pre-trained model and coarsely label whether the submission intentionally memorized training samples. Coarse labeling of submissions achieved by exploiting competition-related clues obtain noisy labels.
There exists a threshold 7* that best separates memorized versus non memorized submissions via the memorization distance (see Figure Here define the memorization margin d of pre-trained
4

---

Under review as a conference paper at ICLR 2021
model M as
d(M) = min Σ (s(Sg, St) - =)2
45
The pre-trained model with largest memorization margin then selected for calculation of the private score, in this case, NASNet (Zoph et al. 2018), and the optimal corresponding memorization penalty m, where =
100
100
memorized
memorized
non-memorized
non-memarized
memorization
memorization distance
Figure 1: Histogram of memorization distance for private (left) and public (right) leaderboards (using NASNet and Inception). As shown the figure, the two classes (legitimate models and memorizing/cheating models) are well separated.
3.2.2 HANDLING FALSE PENALIZATION
While MiFID was designed to handle penalization automatically, in practice we observed minor mixing cheating and non-cheating submissions between the well-separated peaks (Figure [. While it is well accepted that model can be perfect, it was necessary to ensure that competition was fair. Therefore, different strategies were adopted to resolve false positives and negatives. For legitimate submissions that are falsely penalized (false positives), participants are allowed to actively submit rebuttals for the result. For cheating submissions that are dissimilar enough to the training set to dodge penalization (false negatives), the code manually reviewed to determine if intentional memorization was present. This manual reviewing process of code submissions was labor intensive, it required expert knowledge of generative modeling. The goal was review enough submissions such that the top 100 teams on the leaderboard would be free of cheaters, since we reward the top 100 ranked teams. Thanks our design of MiFID, it is possible that we were comfortable that most users ranked lower than 100 on the leaderboard who cheated with memorization were penalized by MiFID. This configuration of MiFID significantly reduced the time needed to finish the review, approximately by 5x. The results of the manual review is presented in Section
set the penalty threshold such
RESULTS AND DATA RELEASE
total of 924 teams joined the competition, producing over 11,192 submissions. Visual samples from submitted images are shown in the appendix.
4.1 DATA RELEASE
The complete dataset will be released with the publication of this paper to facilitate future work on benchmarking generative modeling. It includes:
A total of 11,192 submissions, each containing 10,000 generated dog images with dimension
64 x 64 3.
5

---

Under review as a conference paper at ICLR 2021
Manual labels for the top 1000 ranked submissions of whether the code is legitimate generative method and the type of illegitimacy involved if it is not. This was extremely to obtain.
Crowd-labeled image quality: 50,000 human labeled quality and diversity of images generated from the top 100 teams (non-memorized submissions).
will also release the code to reproduce results in the paper.
4.2
MEMORIZATION METHODS SUMMARY
The 1000 top submissions are manually labeled as to whether or not (and how) they cheated. As we previously predicted, the most pronounced way of cheating was training sample memorization. We observed different levels sophistication in these methods from very naive (submitting the training images) to highly complex (designing GAN to memorize). The labeling results summarized in Table 1
Table 1: Training sample memorization methods
METHOD
DESCRIPTION
MEMORIZATION GAN (MGAN)
MEMORIZATION GANS ARE PURPOSELY TRAINED TO MEMORIZE THE TRAINING SET WHILE MAINTAINING THE ARCHITECTURE OF GENER- ATOR AND DISCRIMINATOR. THE TRAINING PROCESS IS SPLIT INTO TWO PARTS: (1) TRAIN DISCRIMINATOR ONLY WITH REAL TRAINING IMAGES EDEGENERATES TO CLASSIFER OF TRAINING SET MEMBER- DISCRIMINATOR. CONSTRUCTS PAIRS OF LABELED DATA CONSISTING OF NOISE VEC-
TYPICAL GAN MODIEYING
SHIP), (2) TRAIN GENERATOR ONLY WITH
SUPERVISED MAPPING (SUP)
TORS AND TRAINING SAMPLES AND TRAIN A NEURAL NETWORK TO LEARN THE MAPPING.
AUTOENCODER (AE)
AUTOENCODERS RELATIVELY STRAIGHTFORWARE TRAIN-
ING DIRECTLY ON TRAINING SAMPLES AND RECONSTRUCTING THEM.
AUGMENTATION (AUG)
SUCH CROPPING, MORPHING, BLENDING AND TECHNIQUES NOISE. THE NAIVETY OF THIS APPROACH MAKES IT THE EASIEST TO IDEN-
COMBINATIONS OF TYPICAL IMAGE AUGMENTATION
TIFY AND GENERALLY CAN BE FILTERED OUT WITH THE MIFID.
4.3 COMPETITION RESULTS SUMMARY
In Figure (left), we observe that non-generative methods score extremely good (low) FID scores on both the public and private leaderboard. Specifically, memorization GAN achieved top tier performance and it was highly-debated topic for long time whether it should be allowed in the competition. Ultimately, memorization GAN banned, but serves as good reminder that generative-lookie looking models may not actually be generative. In Figure 2 (right), we observe that the range of memorization calculated by NASNet (private) spans twice as wide as Inception (public), allowing easier profiling of cheating submissions by memorization penalty. It reflects the effectiveness of our strategy selecting the model for calculating private score.
Participants generally started with basic generative models such as DCGAN (Radford et al. 2015) and moved to more complex ones as they grow familiar with the framework. Most notably BigGAN (Brock et al. 2018), SAGAN (Zhang al. 2018) and StyleGAN (Karras al. 2019a) achieved the most success. Interestingly, one submission using DCGAN (Radford et al. 2015) with spectral- normalization (Miyato et al. 2018) made it into top 10 in the private leaderboard, suggesting that different variations of GANs with proper tuning might all be able to achieve good FID scores (Lucic)
al. 2017).

---

Under review as a conference paper at ICLR 2021
non-mem
migann
120
mem--iisarcc NA (private)
eug
Sup
FID NASNIT (priiate)
1.00
0.35
80
0.30
mgan
aug
sup
0:15
20
40
D0
60
100
120
0.00
0.05
0.10
0.25
FDD Inception (public)
memn-iistance Inception (public)
Figure 2: Distribution of FID (left) and memorization distances (right) for public vs private scores with manual labels. The better (lower) FIDs are the ones using various memorization techniques.
INSIGHTS
5.1 UNINTENTIONAL CHEATING: MODELS WITH BETTER FID MEMORIZE MORE
In our observation, almost all cheating submissions attempted to cheat by memorizing the training set. This is likely because it is well-known that memorization achieves a good FID score. The research community has long been aware that memorization can be an issue for the FID metric. However, there has been formal studies on the impact of IS or FID scores affected by memorization. This can pose a serious problem when researchers continue to claim state-of-the- results based on improvements to the FID score if there is not systematic way to measure and address training set memorization. With disturbing findings from our study, we caution the danger of ignoring memorization in research benchmark metrics, especially with unintentional memorization of training data.
In Figure (right) we plot the relationship between FID and memorization distance for all 500 non-cheating models in the public and private leaderboard, respectively. Note that these models non-cheating, most of which popular variants of state-of-the-an generative models such as DCGAN and SAGAN recently published in top machine learning conferences. Disturbingly, the Pearson correlation between FID and memorization distance is above 0.95 for both leaderboards. High correlation does not imply that memorization solely enables good model performance evaluated by FID but it is reasonable to suspect that generation of images close to the training set can result in high FID score.
It is important for to take memorization more seriously, given how easy it is for memorization to occur unintentionally. The research community needs to better study and understand the limitation of current generative model benchmark metrics. When proposing to adopt rigorous inspections of model quality, especially regarding training sample memorization. Existing methods such visualizing pairs of generated image and their nearest neighbors in the training dataset should be mandatory in benchmarks. Furthermore, other methods such as the FID and memorization distance correlation (Figure 3) for different model parameters also be helpful to include in publications.
generative techniques, it is crucial
5.2 DEBUNKING FID: CHOICE OF LATENT SPACE FOR FEATURE PROJECTION IS NON-TRIVIAL
In the original paper where FID is proposed (Heusel et al. 2017), features from the coding layer of an Inception model are used as the projected latent space to obtain "vision-relevant" features. is generally assumed that Fréchet Distance is invariant to the chosen latent space for projection long as the space "information-rich" which is why the arbitrary choice of the Inception model has been widely accepted. Interestingly, there has not been much study the extent of our knowledge to whether the assumption holds true even though relatively large amount new generative model architectures are being proposed (many of which rely heavily FID for performance benchmarking).
7

---

Under review as a conference paper at ICLR 2021
public
500
0.45
com(Pearson) 0.9642
NASNet (priiatt)
memorization distonnc
600
0.35
890
0.59
Speaman)) 0.0353
corrPet 0.9551
0.15
100
2:00
300
200
500
600
20
40
60
50
100
120
140
HD Inception (public)
FID
Figure 3: Public FID Inception vs private FID NASNet (left) and FID vs memorization distance distribution with non-memorized submissions (right). It shows that FID is highly correlated to memorization.
In our competition, we used different models for the public and private leaderboards in an attempt to avoid models which "overfit" some particular feature space.
In Figure (left), we examine the relationship between Fréchet Distance calculated by two different pre-trained image models that achieved close state-of-the-ee art performance on ImageNet classi- fication (specifically. Inception (Szegedy et al. (016)) and NasNet (Zoph & Le (016)). At first glance, a Spearman correlation of 0.93 seems to support the assumption FID being invariant the projection space. However, on closer inspection we noticed that the mean absolute rank difference 124.6 between public and private leaderboards for all 1675 effective submissions. we take out the consistency of rank contributed by intentional memorization by considering the top 500 labeled, non-memorized submissions only, the mean absolute rank difference is as large as 94.7 (18.9 %). To put into perspective, only the top 5 places receive monetary awards and there is only common member between the top 5 evaluated by FID projected with the two models.
It's common to publications claiming state-of-an art performance with less than 5% improvement compared to others. As summarized in the Introduction section of this paper, generated model evalua- tion, compared other well-studied tasks such classification, is extremely difficult. Observing that model performance measured by FID fluctuates in such great amplitude relative to the improvement of many newly proposed generation techniques, would suggest taking progression the FID metric with grain of salt.
6 CONCLUSIONS
We summarized our design of the first ever generative modeling competition and shared insights obtained regarding FID generative modeling benchmark metric. By running a public generative modeling competition we observed how participants attempted to game the FID, specifically with memorization, when incentivized with monetary awards. Our proposed Memorization Informedd Fréchet Inception Distance (MiFID) effectively punished models that intentionally memorize the training set which current popular generative modeling metrics do not take into consideration.
We shared two main insights from analyzing the 11,000+ submissions. First, unintentional training sample memorization a serious and possibly widespread issue. Careful inspection of the models and analysis on memorization should be mandatory when proposing new generative model techniques. Second, contrary to popular belief, the choice of pre-trained model latent space when calculating FID is non-trivial. The top 500 labeled, non-memorized submission mean absolute rank difference percentage between our two models 18.9 %, suggesting that FID is rather unstable to serve as the benchmark metric for studies to claim minor improvement past methods.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Shane Barratt and Rishi Sharma. A note on the inception score. arXiv preprint arXiv:1801.0197 2018.
Ali Borji. Pros and cons of gan evaluation measures. Computer Vision and Image Understanding, 179:41-65, 2019.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1096, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE conference on computer vision and paltern recognition, 2009.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516, 2014.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. In Advances in Neural Information Processing Systems, 2019.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. arXiv preprint 1rXiv:1605.09788 2016.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Arjovsky, and Aaron Courville. Adversarially learned inference. arXiv preprint arXiv: (60:::6000000 00704, 2016.
Ian Goodfellow, Jean Poudet-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems, pp. 2672-26002 2680, 2014.
Ishaan Gulrajani, Colin Raffel, and Luke Metz. Towards gan benchmarks which require generalization. 2018.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 0222-6637, 2017.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
Pavel Izmailov, Polina Kirichenko, Mare Finzi, and Andrew Gordon Wilson. Semi-supervised learning with normalizing flows. arXiv preprint arXiv:1912.13023. 2019.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2019a.
Tero Karras, Samuli Laine, Miik Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. arXiv preprint arXiv:19/2.04998, 2019b.
Diederik P Kingma and Max Welling.
Auto-encoding variational bayes.
arXiv preprint
1312v 14, 2013.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems, 2018.
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling, Semi-supervised learning with deep generative models. In Advances in neural information processing systems, pp. 3581-3589, 2014.

---

Under review as a conference paper at ICLR 2021
Christian Ledig, Lucas Theis, Ferene Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. Photo-realistic single image super-resolution using generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? large-scale study, 2017.
Lars Maaløe, Marco Fraccaro, Valentin Liévin, and Ole Winther. Biva: A very deep hierarchy of latent variables for generative modeling. arXiv preprint arXiv:/902.02 2019.
Jacob Menick and Nal Kalchbrenner. Generating high fidelity images with subscale pixel networks and multidimensional upscaling. arXiv preprint arXiv: (...vv/8//..0 1 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks, 2018.
Augustus Odena. Semi-supervised learning with generative adversarial networks. arXiv preprint arXiv:1606.01555 168.0015000.... 2016.
Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. arXiv preprint arXiv: 001.660..667999 2016.
Taesung Park, Ming-Yu Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks, 2015.
Ali Razavi, Aaron den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. arXiv preprint arXiv: 1901::90000000 00446, 2019.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. arXiv preprint arXiv:1505.05771 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference deep generative models. arXiv preprint arXiv: DXXiv:1401.4088 2014.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3), 2015.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Chen. Improved techniques for training gans. In Advances in neural information processing systems, 2016.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference computer vision and pattern recognition, 2016.
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S. Huang. Free-form image inpainting with gated convolution. In Proceedings of the IEEE International Conference Computer Vision, 2019.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks, 2018.
Jun Yan Zhu, Taesung Park. Phillip Isola, and Alexei A. Efros. Unpaired image-to-imann translation using cycle-consistent adversarial networks. Proceedings of the IEEE international conference computer vision, 2017 7a.
10

---

Under review as a conference paper at ICLR 2021
Shizhan Zhu, Raquel Urtasun, Sanja Fidler, Dahua Lin, and Chen Change Loy. Be your own prada: Fashion synthesis with structural coherence. In Proceedings of the IEEE International Conference Computer Vision, 2017b.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning, arXiv preprint 1110055,, 2016.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018.
A
APPENDIX
Table 2: Public and Private leaderboards MiFID configurations
PUBLIC
PRIVATE
MODEL
INCEPTION
NASNET
IMAGENET DOGS
IMAGENET DOGS
PRIVATE
DATASET
120 BREEDS,
20579 IMAGES
DOGS +
INTERNET DOGS
Figure Submissions from ranks (first row), 2, 3, 5, 10, 50, 100 (last row) on the private leaderboard. Each row is random sample of 10 images from the same team. Visually, the quality of the generated images gets lower as the ranks get higher.


Reviews:
[{"review_id": "qb4dC31hbU", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studied the memorization issue of generative modeling. It proposed a benchmark for a public generative modeling competition and observed how participants attempted to game the FID. \n\nIt seems the paper only targeted FID. Actually in GAN evaluation, FID is not the only metric that are widely used. In addition, I doubt how the proposed competition can show the impact of memorization. Also, it is not clear how MiFID really evaulates the generilization ability. In addition, the title mentioning generative modeling is overclaimed (only GANs). \n\nThe writting of the paper is not good. The introducation spent too many paragraphes on describing the background of GANs but failed to clarify the motivation/intuiation clearly. It also used several offencive words such as \"cheating\". \n\nOverall, I think this work has limit impact to the community and not provided deep insights to the audience. Based on these facts, I make my rating. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "j7tdwf426T7", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper investigates memorization/overfitting in GANs and proposes a new metric (MiFID) to identify memorization in trained GAN models automatically. The topic of memorization in GANs is an important one and certainly one that does require more work and research.\n\nContributions/Novelty:\n- The authors introduce a new metric to automatically evaluate memorization in GANs\n- A competition is held to collect data to evaluate different kinds of memorization in GANs\n\nBackground:\nThe background section is quite short and only looks into the IS and FID, but not into the central topic of the paper: memorization/overfitting in GANs\n- What exactly is memorization in GANs? How do you define/measure it? The concept of memorization in GANs seems to be a bit obscure in some parts of the paper (see my remarks to e.g. table 1 later on)\n- What about other approaches that have looked into overfitting? E.g. [1,2]\n\n[1] Webster, R., Rabin, J., Simon, L., & Jurie, F. (2019). Detecting overfitting of deep generative networks via latent recovery. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 11273-11282).\n\n[2] Meehan, C., Chaudhuri, K. & Dasgupta, S.. (2020). A Three Sample Hypothesis Test for Evaluating Generative Models. Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics\n\nCompetition:\nThe section on the competition is very sparse and I would appreciate more details:\n- Why are particiapnts only allowed to train for 9h? Why not more or less? Why is the training time relevant for memorization in GANs?\n- Why use a new dataset of only dog images? Why not also test with other images? Why not test with TinyImageNet or something similar?\n- Why use only 20,000 images? That's a pretty small dataset (which might make memorization more likely in the first place)\n- Who are the participants (academics/industry, advanced/beginner) and how were participants found/recruited?\n- What was the overall goal of the competition? Identify memorization problems in GANs? Find the best model given the constraints on hardware, training time, small dataset, ...?\n\nMiFID:\n- From section 3.1: \"This motivated the addition of a ”memorization-aware” metric that penalizes models producing images too similar to the training set.\" -> How exactly is \"too similar to training set\" defined?\n- Why do you use the cosine similarity specifically? Why not use other metrics such as LPIPS [3] or SSIM [4]? Have you tried using other metrics?\n- Technically speaking the memorization distance is not a distance since it is not symmetrical, maybe call it \"divergence\" instead\n- For the formula/calculation of MiFID: I might misunderstand: but why use the \"min\" and not the \"max\" of the cosine distance? By using min you look for the most different sample in the training set for each generated image and use that to calculate the distances; if I'm looking for memorization wouldn't it make more sense to look for the most similar image (the memorized one) and use that for calculating the distance?\n\n[3] Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018). The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 586-595).\n\n[4] Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004). Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4), 600-612.\n\nResults:\n- Regarding memorization methods (Table 1): why are AEs considered cheating? Yes, they \"memorize\" the training set, but if trained correctly (e.g. VAEs) they can also generate novel images; maybe this shows that \"memorization\" in this context is not well-defined: memorization (for me) means that the generative model exhibits a mode-collapse towards the training images, i.e. no matter what kind of noise input you give the model it only generates training images; this is not the case for VAEs: they will not only generate training images for any noise input you give them...\n- Similarly: I also don't understand why using data augmentation during training of generative models is \"cheating\" and leads to overfitting; actually, there are a lot of recent works that show how data augmentation can be helpful during GAN training, especially for smaller datasets as in this case (only 20,000 images) [5,6,7,8]\n\n[5] Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., & Aila, T. (2020). Training generative adversarial networks with limited data. arXiv preprint arXiv:2006.06676.\n\n[6] Tran, N. T., Tran, V. H., Nguyen, N. B., Nguyen, T. K., & Cheung, N. M. (2020). Towards Good Practices for Data Augmentation in GAN Training. arXiv preprint arXiv:2006.05338.\n\n[7] Zhao, Z., Zhang, Z., Chen, T., Singh, S., & Zhang, H. (2020). Image Augmentations for GAN Training. arXiv preprint arXiv:2006.02595.\n\n[8] Zhao, S., Liu, Z., Lin, J., Zhu, J. Y., & Han, S. (2020). Differentiable augmentation for data-efficient gan training. arXiv preprint arXiv:2006.10738.\n\nInsights:\n- Have you tried to use this method to also evaluate current pretrained models (BigGAN, StyleGAN, etc) for memorization roblems and visualize them?\n- I don't see why the high correlation between memorizsation distance and FID is a problem; this is exactly what we would expect; I am missing experiments that clearly show that memorization is actually a problem in these models, i.e. that they only generate images from the training set\n\nGeneral remarks:\n- I'm not convinced the results of the competition are reliable to evaluate memorization problems in GANs, especially since it is not clear who participated in the competition, why there are so many restrictions that should not directly affect memorization as such (small amount of training time and only one GPU), and the dataset for evaluation is very small (20,000 images) and only contains dogs; I would at least expect experiments on larger and different datasets and an explanation why restricting the resources is important to evaluate memorization/overfitting in GANs\n- I'm also not sure I understand what exactly the authors define as \"cheating\"/memorization: e.g. autoencoders and data augmentation are not cheating in my perspective and also do not automatically lead to memorization; this leads back to my point that I am missing a concrete working definition on memorization in GANs in this paper\n- While memorization is a serious issue for generative models I am missing a clear definition of what memorization means in this context; the authors show that there is a high correlation between the FID and their memorization distance, but this is not surprising: generative models *should* generate images close to the training set (but not \"from\" the training set); I feel this work lacks a clear distinction between \"memorization\" (only generates images from the training set) and other models (such as e.g. VAEs but also GANs) that might generate images from the training set but are also capable of generating images that do not occur in the training set (often tested by doing interpolation between two images) -> a high correlation between FID and memorization distance is therefore expected, since GANs should learn to generate images that are close to the training set -> I feel there needs to be a way to automatically detect GANs that are *not* capable of generating images that are *reasonably far* away from the training set but still realistic (if the model also generated some training images I don't think this is necessarily a big issue)\n\nMinor:\n- Why is this a table 1 a table and not a simply a bullet list/enumeration?", "extra_scores": {}, "extra_text": {}}, {"review_id": "PGlYq2EiQ-t", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\nMotivated by the observation that prevalent metrics (Inception Score, Frechet Inception Distance) used to assess the quality of samples obtained from generative models are gameable (due to either the metric not correlating well with visually assessed sample quality or the metric being susceptible to training sample memorization), the authors conduct a large scale “controlled” study to assess the gameability of said metrics. The authors conducted a competition and subsequently analyzed how approaches tend to cheat so as to obtain higher FID scores. Furthermore, to assess the extent of memorization w.r.t. the FID score, the authors propose a new metric — Memorization-Informed Frechet Inception Distance (MiFID) — which takes into account sample memorization w.r.t. a reference set. The authors conclude on a few notable observations — (1) unintentional memorization in generative models is a serious and prevalent issue; (2) the choice of latent space used to compute FID based scores can make a significant difference.\n\n**Strengths**\n- The paper is generally well-written and easy to follow for the most part. The authors do a good job of walking the reader through the design of the competition, the proposed metric and design choices adopted to ensure fair characterization of sample-memorization and cheating strategies.\n- The choice of memorization assessment defined and adopted for the FID score in section 3.1 is intuitive and well-motivated. I particularly appreciated how the authors identified the fact that assigning a memorization penalty is itself subjective to the choice of feature space, reference set and the memorization threshold. Furthermore, the constraints assigned on the challenge submissions seem reasonable to ensure fair comparisons across multiple submissions. I would also like to highlight that in addition to automating “memorization” detection to the extent possible, the authors went the extra mile to manually review code-submissions to further identify false negatives — all leading to a concise characterization of the strategies adopted to cheat or memorize.\n- I think this large-scale study (although operating under several constraints due to it’s controlled nature) is going to be quite beneficial to the community. Careful inspection of the submissions reveals several insights (and caveats) associated with current metrics / results for generative models that can perhaps motivate the community to take measures to establish more reliable benchmarks for generative modeling.\n\n**Weaknesses**\nI will mostly highlight weaknesses and other points that likely have clarity issues associated in the current draft.\n- While the choice of constraints imposed on the challenge submissions for the study here seem reasonable, I’m concerned the study only highlights issues associated with approaches that can likely be executed / implemented under these constraints (pointing to limited computation time, isolated containerization and restricted access to pre-trained models or additional data). While this is a good starting point, it’s unclear how well insights from the study may generalize to approaches that operate outside of these constraints. Do the authors have any thoughts on this?\n- In addition to other factors, the choice of the memorization margin (as described in section 3.2.1) seems to depend not only on the reference set but also on the “sets” of generated images. This likely affects all the following stages of re-calibrating the score after identifying false-positives and negatives. Can the authors comment on how dependent is this on the number and kind of submissions? This will likely inform the extent to which the analysis setup (including the metric) can be adopted / extended to other kinds of images, datasets, etc (and maybe even more relaxed constraints).\n- Minor comments — (1) The paper would benefit from making the distinction between intended and unintended memorization clear early on in the introductory section; (2) I’m curious about how the constraints posed in the section 3 on challenge were verified. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a competition on generative models on a new dataset to study memorization in generative models and propose a  new metric Memorization-Informed Frechet Inception Distance (MiFID). 

While this is an important topic, reviewers raised multiple issues and concerns regarding 1)  the metric definition (that it needs to be max and not min, this was acknowledged in the rebuttal but not updated in the paper) , 2)  how this competition is ran in terms of the definition of "cheating",  that the setup is not controlled and only constraining the time of training 3)  the notion of MiFID is depending on the sets of samples considered and the feature extractor used. 

Some other reviewers raised concerns that the paper is only concerned by FID and not other metrics , and that it was only verified on GANs.  We hope the authors will address those concerns and submit the paper to an upcoming venue.
Decision: Reject