A Large-scale Study on Training Sample Memorization in Generative Modeling

Many recent developments on generative models for natural images have relied on heuristically-motivated metrics that can be easily gamed by memorizing a small sample from the true distribution or training a model directly to improve the metric . In this work , we critically evaluate the gameability of such metrics by running a competition that ultimately resulted in participants attempting to cheat . Our competition received over 11000 submitted models and allowed us to investigate both intentional and unintentional memorization . To stop intentional memorization , we propose the “ Memorization-Informed Fréchet Inception Distance ” ( MiFID ) as a new memorization-aware metric and design benchmark procedures to ensure that winning submissions made genuine improvements in perceptual quality . Furthermore , we manually inspect the code for the 1000 top-performing models to understand and label different forms of memorization . The inspection reveals that unintentional memorization is a serious and common issue in popular generative models . The generated images and our memorization labels of those models as well as code to compute MiFID are released to facilitate future studies on benchmarking generative models . 1 INTRODUCTION . Recent work on generative models for natural images has produced huge improvements in image quality , with some models producing samples that can be indistinguishable from real images ( Karras et al. , 2017 ; 2019a ; b ; Brock et al. , 2018 ; Kingma & Dhariwal , 2018 ; Maaløe et al. , 2019 ; Menick & Kalchbrenner , 2018 ; Razavi et al. , 2019 ) . Improved sample quality is important for tasks like super-resolution ( Ledig et al. , 2017 ) and inpainting ( Yu et al. , 2019 ) , as well as creative applications ( Park et al. , 2019 ; Isola et al. , 2017 ; Zhu et al. , 2017a ; b ) . These developments have also led to useful algorithmic advances on other downstream tasks such as semi-supervised learning ( Kingma et al. , 2014 ; Odena , 2016 ; Salimans et al. , 2016 ; Izmailov et al. , 2019 ) or representation learning ( Dumoulin et al. , 2016 ; Donahue et al. , 2016 ; Donahue & Simonyan , 2019 ) . Modern generative models utilize a variety of underlying frameworks , including autoregressive models ( Oord et al. , 2016 ) , Generative Adversarial Networks ( GANs ; Goodfellow et al. , 2014 ) , flow-based models ( Dinh et al. , 2014 ; Rezende & Mohamed , 2015 ) , and Variational Autoencoders ( VAEs ; Kingma & Welling , 2013 ; Rezende et al. , 2014 ) . This diversity of approaches , combined with the philosophical nature of evaluating generative performance , has prompted the development of heuristically-motivated metrics designed to measure the perceptual quality of generated samples such as the Inception Score ( IS ; Salimans et al. , 2016 ) or the Fréchet Inception Distance ( FID ; Heusel et al. , 2017 ) . These metrics are used in a benchmarking procedure where “ state-of-the-art ” results are claimed based on a better score on standard datasets . Indeed , much recent progress in the field of machine learning as a whole has relied on useful benchmarks on which researchers can compare results . Specifically , improvements on the benchmark metric should reflect improvements towards a useful and nontrivial goal . Evaluation of the metric should be a straightforward and well-defined procedure so that results can be reliably compared . For example , the ImageNet Large-Scale Visual Recognition Challenge ( Deng et al. , 2009 ; Russakovsky et al. , 2015 ) has a useful goal ( classify objects in natural images ) and a well-defined evaluation procedure ( top-1 and top-5 accuracy of the model ’ s predictions ) . Sure enough , the ImageNet benchmark has facilitated the development of dramatically better image classification models which have proven to be extremely impactful across a wide variety of applications . Unfortunately , some of the commonly-used benchmark metrics for generative models of natural images do not satisfy the aforementioned properties . For instance , although the IS is demonstrated to correlate well with human perceived image quality ( Salimans et al. , 2016 ) , Barratt & Sharma ( 2018 ) points out several flaws of the IS when used as a single metric for evaluating generative modeling performance , including its sensitivity to pretrained model weights which undermines generalization capability . Seperately , directly optimizing a model to improve the IS can result in extremely unrealistic-looking images ( Barratt & Sharma , 2018 ) despite resulting in a better score . It is also well-known that if a generative model memorizes images from the training set ( i.e . producing non-novel images ) , it will achieve a good IS ( Gulrajani et al. , 2018 ) . On the other hand , the FID is widely accepted as an improvement over IS due to its better consistency under perturbation ( Heusel et al. , 2017 ) . However , there is no clear evidence of the FID resolving any of the flaws of the IS . A large-scale empirical study is necessary to provide robust support for understanding quantitatively how flawed the FID is . Motivated by these issues , we want to benchmark generative models in the “ real world ” , i.e . outside of the research community by holding a public machine learning competition . To the extent of our knowledge , no large-scale generative modeling competitions have ever been held , possibly due to the immense difficulty of identifying training sample memorization in a efficient and scalable manner . We designed a more rigorous procedure for evaluating competition submissions , including a memorization-aware variant of FID for autonomously detecting cheating via intentional memorization . We also manually inspected the code for the top 1000 submissions to reveal different forms of intentional or unintentional cheating , to ensure that the winning submissions reflect meaningful improvements , and to confirm efficacy of our proposed metric . We hope that the success of the first-ever generative modeling competition can serve as future reference and stimulate more research in developing better generative modeling benchmarks . Our main goal in this paper is to conduct an empirical study on issues of relying on the FID as a benchmark metric to guide the progression of generative modeling . In Section 2 , we briefly review the metrics and challenges of evaluating generative models . In Section 3 , we explain in detail the competition design choices and propose a novel benchmarking metric , the Memorization-Informed Fréchet Inception Distance ( MiFID ) . We show that MiFID enables fast profiling of participants that intentionally memorize the training dataset . In Section 4 , we introduce a dataset released along with this paper that includes over one hundred million generated images and manual labels obtained by painstaking code review . In Section 5 , we connect phenomena observed in large-scale benchmarking of generative models in the real world back to the research community and point out crucial but neglected flaws in the FID . 2 BACKGROUND . In generative modeling , our goal is to produce a model pθ ( x ) ( parameterized by θ ) of some true distribution p ( x ) . We are not given direct access to p ( x ) ; instead , we are provided only with samples drawn from it x ∼ p ( x ) . In this paper , we will assume that samples x from p ( x ) are 64-by-64 pixel natural images , i.e . x ∈ R64×64×3 . A common approach is to optimize θ so that pθ ( x ) assigns high likelihood to samples from p ( x ) . This provides a natural evaluation procedure which measures the likelihood assigned by pθ ( x ) to samples from p ( x ) that were held out during the optimization of θ . However , not all models facilitate exact computation of likelihoods . Notably , Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) learn an “ implicit ” model of p ( x ) from which we can draw samples but that does not provide an exact ( or even an estimate ) of the likelihood for a given sample . The GAN framework has proven particularly successful at learning models which can generate extremely realistic and high-resolution images , which leads to a natural question : How should we evaluate the quality of a generative model if we can ’ t compute the likelihood assigned to held-out samples ? This question has led to the development of many alternative ways to evaluate generative models ( Borji , 2019 ) . A historically popular metric , proposed in ( Salimans et al. , 2016 ) , is the Inception Score ( IS ) which computes IS ( pθ ) = Ex∼pθ ( x ) [ DKL ( IN ( y|x ) ‖ IN ( y ) ) ] where IN ( y|x ) is the conditional probability of a class label y assigned to a datapoint x by a pretrained Inception Network ( Szegedy et al. , 2015 ) . More recently , ( Heusel et al. , 2017 ) proposed the Fréchet Inception Distance ( FID ) which better correlates with perceptual quality . The FID uses the estimated mean and covariance of the Inception Network feature space distribution to calculate the distance between the real and fake distributions up to second order . The FID between the real images r and generated images g is computed as : FID ( r , g ) = ‖µr − µg‖22 + Tr ( Σr + Σg − 2 ( ΣrΣr ) 1 2 ) where µr and µg are the mean of the real and generated images in latent space , and Σr and Σg are the covariance matrices for the real and generated feature vectors . A drawback of both the IS and FID is that they assign a very good score to a model which simply memorizes a small and finite sample from p ( x ) ( Gulrajani et al. , 2018 ) , an issue we address in section 3.1 . 3 GENERATIVE MODELING COMPETITION DESIGN . We designed the first generative model competition where participants were invited to generate realistic dog images given 20,579 images of dogs from ImageNet ( Russakovsky et al. , 2015 ) . Participants were required to implement their generative model in a constrained computation environment to prevent them from obtaining unfair advantages . The computation environment was designed with : • Limited computation resource ( 9 hours on a NVIDIA P100 GPU for each submission ) since generative model performance is known to be highly related to the amount of computational resources used ( Brock et al. , 2018 ) • Isolated containerization to avoid continuous training by reloading model checkpoints from previous sessions • No access to external resources ( i.e . the internet ) to avoid usage of pre-trained models or additional data Each submission is required to provide 10,000 generated images of dimension 64 × 64 × 3 and receives a public score in return . Participants are allowed to submit any number of submissions during the two-month competition . Before the end of the competition , each team is required to choose two submissions , and the final ranking is determined by the better private score ( described below ) out of the two selected submissions . In the following sections , we discuss how the final decisions were made regarding pretrained model selection ( for FID feature projection ) and how we enforced penalties to ensure the fairness of the competition . 3.1 MEMORIZATION-INFORMED FRÉCHET INCEPTION DISTANCE ( MIFID ) . The most crucial part of the competition is the performance evaluation metric to score the submissions . To assess the quality of generated images , we adopted the Fréchet Inception Distance ( Heusel et al. , 2017 ) which is a widely used metric for benchmarking generative tasks . Compared to the Inception Score ( Salimans et al. , 2016 ) , the FID has the benefits of better robustness against noise and distortion and more efficient computation ( Borji , 2019 ) . For a generative modeling competition , a good metric not only needs to reflect the quality of generated samples but must also allow easy identification of cheating with as little manual intervention as possible . Many forms of cheating were prevented by setting up the aforementioned computation environment , but even with these safeguards it would be possible to “ game ” the FID score . Specifically , we predicted that memorization of training data would be a major issue , since current generative model evaluation metrics such as IS or FID are prone to reward high scores to memorized instances ( Gulrajani et al. , 2018 ) . This motivated the addition of a ” memorization-aware ” metric that penalizes models producing images too similar to the training set . Combining memorization-aware and generation quality components , we introduced the MemorizationInformed Fréchet Inception Distance ( MiFID ) as the metric used for the competition : MiFID ( Sg , St ) = mτ ( Sg , St ) · FID ( Sg , St ) where Sg is the generated set , St is the original training set , FID is the Fréchet Inception Distance , and mτ is the memorization penalty which we discuss in the following section .

Summary: Motivated by the observation that prevalent metrics (Inception Score, Frechet Inception Distance) used to assess the quality of samples obtained from generative models are gameable (due to either the metric not correlating well with visually assessed sample quality or the metric being susceptible to training sample memorization), the authors conduct a large scale “controlled” study to assess the gameability of said metrics. The authors conducted a competition and subsequently analyzed how approaches tend to cheat so as to obtain higher FID scores. Furthermore, to assess the extent of memorization w.r.t. the FID score, the authors propose a new metric — Memorization-Informed Frechet Inception Distance (MiFID) — which takes into account sample memorization w.r.t. a reference set. The authors conclude on a few notable observations — (1) unintentional memorization in generative models is a serious and prevalent issue; (2) the choice of latent space used to compute FID based scores can make a significant difference.