Random Network Distillation as a Diversity Metric for Both Image and Text Generation

Generative models are increasingly able to produce remarkably high quality images and text . The community has developed numerous evaluation metrics for comparing generative models . However , these metrics do not effectively quantify data diversity . We develop a new diversity metric that can readily be applied to data , both synthetic and natural , of any type . Our method employs random network distillation , a technique introduced in reinforcement learning . We validate and deploy this metric on both images and text . We further explore diversity in few-shot image generation , a setting which was previously difficult to evaluate . 1 INTRODUCTION . State-of-the-art generative adversarial networks ( GANs ) are able to synthesize such high quality images that humans may have a difficult time distinguishing them from natural images ( Brock et al. , 2018 ; Karras et al. , 2019 ) . Not only can GANs produce pretty pictures , but they are also useful for applied tasks from projecting noisy images onto the natural image manifold to generating training data ( Samangouei et al. , 2018 ; Sixt et al. , 2018 ; Bowles et al. , 2018 ) . Similarly , massive transformer models are capable of performing question-answering and translation ( Brown et al. , 2020 ) . In order for GANs and text generators to be valuable , they must generate diverse data rather than memorizing a small number of samples . Diverse data should contain a wide variety of semantic content , and its distribution should not concentrate around a small subset of modes from the true image distribution . A number of metrics have emerged for evaluating GAN-generated images and synthetic text . However , these metrics do not effectively quantify data diversity , and they work on a small number of specific benchmark tasks ( Salimans et al. , 2016 ; Heusel et al. , 2017 ) . Diversity metrics for synthetic text use only rudimentary tools and only measure similarity of phrases and vocabulary rather than semantic meaning ( Zhu et al. , 2018 ) . Our novel contributions can be summarized as follows : • We design a framework ( RND ) for comparing diversity of datasets using random network distillation . Our framework can be applied to any type of data , from images to text and beyond . RND does not suffer from common problems that have plagued evaluation of generative models , such as vulnerability to memorization , and it can even be used to evaluate the diversity of natural data ( not synthetic ) since it does not require a reference dataset . • We validate the effectiveness of our method in a controlled setting by synthetically manipulating the diversity of GAN-generated images . We use the same truncation strategy employed by BigGAN to increase FID scores , and we confirm that this strategy indeed decreases diversity . This observation calls into question the usefulness of such popular metrics as FID scores for measuring diversity . • We benchmark data , both synthetic and natural , using our random distillation method . In addition to evaluating the most popular ImageNet-trained generative models and popular language models , we evaluate GANs in the data scarce regime , i.e . single-image GANs , which were previously difficult to evaluate . We also evaluate the diversity of natural data . 2 DESIGNING A GOOD DIVERSITY METRIC . Formally defining “ diversity ” is a difficult problem ; human perception is hard to understand and does not match standard mathematical norms . Thus , we first define desiderata for a useful diversity metric , and we explore the existing literature on evaluation of generative models . 2.1 WHAT DO WE WANT FROM A DIVERSITY METRIC ? . Diversity should increase as the data distribution ’ s support includes more data . For example , the distribution of images containing brown dogs should be considered less diverse than the distribution of images containing brown , black , or white dogs . While this property might seem to be a good stand-alone definition of diversity , we have not yet specified what types of additional data should increase diversity measurements . Diversity should reflect signal rather than noise . If a metric is to agree with human perception of diversity , it must not be highly sensitive to noise . Humans looking at static on their television screen do not recognize that this noise is different than the last time they saw static on their screen , yet these two static noises are likely far apart with respect to lp metrics . The need to measure semantic signal rather than noise precludes using entropy-based measurements in image space without an effective perceptual similarity metric . Similarly , diversity metrics for text that rely on counting unique tokens may be sensitive to randomly exchanging words with their synonyms , or even random word swaps , without increasing the diversity of semantic content . Quality 6= diversity . While some GANs can consistently produce realistic images , we do not want to assign their images a high diversity measurement if they produce very little variety . In contrast , other GANs may produce a large variety of unrealistic images and should receive high diversity marks . The quality and diversity of data are not the same , and we want a measurement that disentangles the two . Metrics should be agnostic to training data . Recent single-image GANs and few-shot GANs are able to generate many distinct images from very few training images ( sometimes just one ) ( Shaham et al. , 2019b ; Clouâtre & Demers , 2019 ) . Thus , a good metric should be capable of producing diversity scores for synthetic data that are higher than those of the training set . Likewise , simply memorizing the training data should not allow a generative model to achieve a maximal diversity score . Moreover , two companies may deploy face-generating models trained on two disjoint proprietary datasets , and we should still be able to compare the diversity of faces generated by these models without having training set access . An ideal diversity metric would allow one to collect data and measure its diversity outside of the setting of generative models . Diversity should be measureable on many kinds of data . Measurements based on hand-crafted perceptual similarity metrics or high-performance neural networks trained carefully on large datasets can only be used for the single type of data for which they are designed . We develop a diversity concept that is adaptable to various domain , including both images and text . 2.2 EXISTING METRICS . We now review existing metrics for generative models to check if any already satisfy the above criteria . We focus on the most popular metrics before briefly discussing additional examples . Inception Score ( IS ) ( Salimans et al. , 2016 ) . The Inception Score is a popular metric that rewards having high confidence class labels for each generated example , according to an ImageNet trained InceptionV3 network , while also producing a diversity of softmax outputs across the overall set of generated images ( Deng et al. , 2009 ; Szegedy et al. , 2016 ) . While this metric does encourage generated data to be class-balanced and is not fooled by noise , IS suffers from several disqualifying problems when considered as a measure of diversity . First , it does not significantly reward diversity within classes ; a generative model that memorizes one image from each class in ImageNet may achieve a very strong score . Second , IS often fails when used on classes not in ImageNet and is not adaptable to settings outside of natural image classification ( Barratt & Sharma , 2018 ) . Finally , IS does not disentangle diversity from quality . The Inception Score can provide a general evaluation of GANs trained on ImageNet , but it has limited utility in other settings . Fréchet Inception Distance ( FID ) ( Heusel et al. , 2017 ) . The FID score measures the Fréchet distance between a Gaussian distribution fit to InceptionV3 features on generated data and a Gaussian distribution fit to features on ground-truth data , for example natural images on which the generative model was trained . Unlike IS , FID scores compare generated data to real data , and they do not explicitly rely on ImageNet classes . Thus , FID more effectively discourages a generator from memorizing one image per class . However , FID assumes that the training data is “ sufficient ” and does not reward producing more diversity than the training data . A second problem with FID is that it relies on either ( 1 ) ImageNet models being useful for the problem at hand or ( 2 ) the ability to acquire a reference dataset and train a high-performance model on the problem . Like IS , FID does not disentangle diversity from quality and suffers from similar problems . Several additional metrics have attempted to address these problems . For example , precision and recall have been suggested to tease apart different aspects of a generative model ’ s performance ( Sajjadi et al. , 2018 ; Kynkäänniemi et al. , 2019 ) . However , these metrics ( as used to measure diversity ) , along with other earlier metrics all aim to estimate the likelihood of real data under the generated distribution , and thus , require access to a reference ground-truth dataset . This makes these metrics are inflexible , and these metrics achieve optimal values when a generator simply memorizes the reference dataset . Other examples of metrics include the Modified Inception Score ( m-IS ) uses a crossentropy term to resolve the fact that IS rewards models for memorizing one image per ImageNet class ( Gurumurthy et al. , 2017 ) . However , m-IS is still beholden to the InceptionV3 label space and still prevents a user from discerning diversity from quality . Boundary distortion is a method for detecting covariate shift in GAN distributions by comparing how well classifiers trained on synthetic data perform on ground-truth data ( Santurkar et al. , 2018 ) . This measurement indicates similarity to the true data distribution rather than purely measuring diversity , and it ignores the possibility of models generating even more diverse data than the data on which they are trained . Additionally , this method assumes the user has a large ground-truth dataset . Recently , classification accuracy score ( CAS ) was introduced to measure the performance of generative models by training classifiers on synthetic data produced by the generative models and evaluating the performance of the classifier ( Ravuri & Vinyals , 2019 ) . Similar to the boundary distortion method , this measurement focuses its comparison on the likeness of the synthetic data distribution to the ground-truth distribution and is dependent upon having access to labeled ground-truth data . In natural language processing , several metrics exist for evaluating text generation for dialogue , conversational AI systems , and machine translation . Papineni et al . ( 2002 ) introduced the now widely used Bilingual Evaluation Understudy ( BLEU ) score as a metric for evaluating the quality of machine translated text . BLEU uses a modified form of precision to compare a candidate translation against multiple reference translations , where diversity ideally can be evaluated by including all plausible translations as references when computing the score . However , this requires massive annotation cost , and it remains difficult to capture all viable translations for a given sentence . Li et al . ( 2015 ) and Xu et al . ( 2018 ) propose counting the number of unique n-grams as a measure for evaluating the diversity of text generation tasks in conversational models , however , this metric does not account for the semantic meaning of different tokens and fails to capture paraphrases of semantically similar text . Montahaei et al . ( 2019 ) propose a joint metric for assessing both quality and diversity for text generation systems by approximating the distance of the learned generative model and the real data distribution . This metric couples both diversity and quality in a single metric . Zhu et al . ( 2018 ) introduce Self-BLEU to evaluate sentence variety . Self-BLEU measures BLEU score for each generated sentence by considering other generations as references . By averaging these BLEU scores , a metric called Self-BLEU is computed where lower values indicate more diversity . However , Self-BLEU remains very sensitive to local syntax , and it fails to capture global consistency and diverse semantic information in generated text . Many other metrics exist for generative models which do not approach the problem of diversity . Our work is not the first to recognize this gap in the literature ( Borji , 2019 ) .

Summary: This paper applies random network distillation (RND) as a method for quantifying how diverse samples from a generative model are. Samples from the generative model (or any dataset) are used to train a neural network to mimic a randomly initialized network. Intuitively, this is a more difficult task on a more diverse dataset, and so the distillation loss can be interpreted as a measure of diversity. The authors argue that this approach has advantages over other diversity metrics because it can capture semantic diversity and does not require a second reference dataset.