Random Network Distillation as a Diversity Metric for Both Image and Text Generation

Generative models are increasingly able to produce remarkably high quality images and text . The community has developed numerous evaluation metrics for comparing generative models . However , these metrics do not effectively quantify data diversity . We develop a new diversity metric that can readily be applied to data , both synthetic and natural , of any type . Our method employs random network distillation , a technique introduced in reinforcement learning . We validate and deploy this metric on both images and text . We further explore diversity in few-shot image generation , a setting which was previously difficult to evaluate . 1 INTRODUCTION . State-of-the-art generative adversarial networks ( GANs ) are able to synthesize such high quality images that humans may have a difficult time distinguishing them from natural images ( Brock et al. , 2018 ; Karras et al. , 2019 ) . Not only can GANs produce pretty pictures , but they are also useful for applied tasks from projecting noisy images onto the natural image manifold to generating training data ( Samangouei et al. , 2018 ; Sixt et al. , 2018 ; Bowles et al. , 2018 ) . Similarly , massive transformer models are capable of performing question-answering and translation ( Brown et al. , 2020 ) . In order for GANs and text generators to be valuable , they must generate diverse data rather than memorizing a small number of samples . Diverse data should contain a wide variety of semantic content , and its distribution should not concentrate around a small subset of modes from the true image distribution . A number of metrics have emerged for evaluating GAN-generated images and synthetic text . However , these metrics do not effectively quantify data diversity , and they work on a small number of specific benchmark tasks ( Salimans et al. , 2016 ; Heusel et al. , 2017 ) . Diversity metrics for synthetic text use only rudimentary tools and only measure similarity of phrases and vocabulary rather than semantic meaning ( Zhu et al. , 2018 ) . Our novel contributions can be summarized as follows : • We design a framework ( RND ) for comparing diversity of datasets using random network distillation . Our framework can be applied to any type of data , from images to text and beyond . RND does not suffer from common problems that have plagued evaluation of generative models , such as vulnerability to memorization , and it can even be used to evaluate the diversity of natural data ( not synthetic ) since it does not require a reference dataset . • We validate the effectiveness of our method in a controlled setting by synthetically manipulating the diversity of GAN-generated images . We use the same truncation strategy employed by BigGAN to increase FID scores , and we confirm that this strategy indeed decreases diversity . This observation calls into question the usefulness of such popular metrics as FID scores for measuring diversity . • We benchmark data , both synthetic and natural , using our random distillation method . In addition to evaluating the most popular ImageNet-trained generative models and popular language models , we evaluate GANs in the data scarce regime , i.e . single-image GANs , which were previously difficult to evaluate . We also evaluate the diversity of natural data . 2 DESIGNING A GOOD DIVERSITY METRIC . Formally defining “ diversity ” is a difficult problem ; human perception is hard to understand and does not match standard mathematical norms . Thus , we first define desiderata for a useful diversity metric , and we explore the existing literature on evaluation of generative models . 2.1 WHAT DO WE WANT FROM A DIVERSITY METRIC ? . Diversity should increase as the data distribution ’ s support includes more data . For example , the distribution of images containing brown dogs should be considered less diverse than the distribution of images containing brown , black , or white dogs . While this property might seem to be a good stand-alone definition of diversity , we have not yet specified what types of additional data should increase diversity measurements . Diversity should reflect signal rather than noise . If a metric is to agree with human perception of diversity , it must not be highly sensitive to noise . Humans looking at static on their television screen do not recognize that this noise is different than the last time they saw static on their screen , yet these two static noises are likely far apart with respect to lp metrics . The need to measure semantic signal rather than noise precludes using entropy-based measurements in image space without an effective perceptual similarity metric . Similarly , diversity metrics for text that rely on counting unique tokens may be sensitive to randomly exchanging words with their synonyms , or even random word swaps , without increasing the diversity of semantic content . Quality 6= diversity . While some GANs can consistently produce realistic images , we do not want to assign their images a high diversity measurement if they produce very little variety . In contrast , other GANs may produce a large variety of unrealistic images and should receive high diversity marks . The quality and diversity of data are not the same , and we want a measurement that disentangles the two . Metrics should be agnostic to training data . Recent single-image GANs and few-shot GANs are able to generate many distinct images from very few training images ( sometimes just one ) ( Shaham et al. , 2019b ; Clouâtre & Demers , 2019 ) . Thus , a good metric should be capable of producing diversity scores for synthetic data that are higher than those of the training set . Likewise , simply memorizing the training data should not allow a generative model to achieve a maximal diversity score . Moreover , two companies may deploy face-generating models trained on two disjoint proprietary datasets , and we should still be able to compare the diversity of faces generated by these models without having training set access . An ideal diversity metric would allow one to collect data and measure its diversity outside of the setting of generative models . Diversity should be measureable on many kinds of data . Measurements based on hand-crafted perceptual similarity metrics or high-performance neural networks trained carefully on large datasets can only be used for the single type of data for which they are designed . We develop a diversity concept that is adaptable to various domain , including both images and text . 2.2 EXISTING METRICS . We now review existing metrics for generative models to check if any already satisfy the above criteria . We focus on the most popular metrics before briefly discussing additional examples . Inception Score ( IS ) ( Salimans et al. , 2016 ) . The Inception Score is a popular metric that rewards having high confidence class labels for each generated example , according to an ImageNet trained InceptionV3 network , while also producing a diversity of softmax outputs across the overall set of generated images ( Deng et al. , 2009 ; Szegedy et al. , 2016 ) . While this metric does encourage generated data to be class-balanced and is not fooled by noise , IS suffers from several disqualifying problems when considered as a measure of diversity . First , it does not significantly reward diversity within classes ; a generative model that memorizes one image from each class in ImageNet may achieve a very strong score . Second , IS often fails when used on classes not in ImageNet and is not adaptable to settings outside of natural image classification ( Barratt & Sharma , 2018 ) . Finally , IS does not disentangle diversity from quality . The Inception Score can provide a general evaluation of GANs trained on ImageNet , but it has limited utility in other settings . Fréchet Inception Distance ( FID ) ( Heusel et al. , 2017 ) . The FID score measures the Fréchet distance between a Gaussian distribution fit to InceptionV3 features on generated data and a Gaussian distribution fit to features on ground-truth data , for example natural images on which the generative model was trained . Unlike IS , FID scores compare generated data to real data , and they do not explicitly rely on ImageNet classes . Thus , FID more effectively discourages a generator from memorizing one image per class . However , FID assumes that the training data is “ sufficient ” and does not reward producing more diversity than the training data . A second problem with FID is that it relies on either ( 1 ) ImageNet models being useful for the problem at hand or ( 2 ) the ability to acquire a reference dataset and train a high-performance model on the problem . Like IS , FID does not disentangle diversity from quality and suffers from similar problems . Several additional metrics have attempted to address these problems . For example , precision and recall have been suggested to tease apart different aspects of a generative model ’ s performance ( Sajjadi et al. , 2018 ; Kynkäänniemi et al. , 2019 ) . However , these metrics ( as used to measure diversity ) , along with other earlier metrics all aim to estimate the likelihood of real data under the generated distribution , and thus , require access to a reference ground-truth dataset . This makes these metrics are inflexible , and these metrics achieve optimal values when a generator simply memorizes the reference dataset . Other examples of metrics include the Modified Inception Score ( m-IS ) uses a crossentropy term to resolve the fact that IS rewards models for memorizing one image per ImageNet class ( Gurumurthy et al. , 2017 ) . However , m-IS is still beholden to the InceptionV3 label space and still prevents a user from discerning diversity from quality . Boundary distortion is a method for detecting covariate shift in GAN distributions by comparing how well classifiers trained on synthetic data perform on ground-truth data ( Santurkar et al. , 2018 ) . This measurement indicates similarity to the true data distribution rather than purely measuring diversity , and it ignores the possibility of models generating even more diverse data than the data on which they are trained . Additionally , this method assumes the user has a large ground-truth dataset . Recently , classification accuracy score ( CAS ) was introduced to measure the performance of generative models by training classifiers on synthetic data produced by the generative models and evaluating the performance of the classifier ( Ravuri & Vinyals , 2019 ) . Similar to the boundary distortion method , this measurement focuses its comparison on the likeness of the synthetic data distribution to the ground-truth distribution and is dependent upon having access to labeled ground-truth data . In natural language processing , several metrics exist for evaluating text generation for dialogue , conversational AI systems , and machine translation . Papineni et al . ( 2002 ) introduced the now widely used Bilingual Evaluation Understudy ( BLEU ) score as a metric for evaluating the quality of machine translated text . BLEU uses a modified form of precision to compare a candidate translation against multiple reference translations , where diversity ideally can be evaluated by including all plausible translations as references when computing the score . However , this requires massive annotation cost , and it remains difficult to capture all viable translations for a given sentence . Li et al . ( 2015 ) and Xu et al . ( 2018 ) propose counting the number of unique n-grams as a measure for evaluating the diversity of text generation tasks in conversational models , however , this metric does not account for the semantic meaning of different tokens and fails to capture paraphrases of semantically similar text . Montahaei et al . ( 2019 ) propose a joint metric for assessing both quality and diversity for text generation systems by approximating the distance of the learned generative model and the real data distribution . This metric couples both diversity and quality in a single metric . Zhu et al . ( 2018 ) introduce Self-BLEU to evaluate sentence variety . Self-BLEU measures BLEU score for each generated sentence by considering other generations as references . By averaging these BLEU scores , a metric called Self-BLEU is computed where lower values indicate more diversity . However , Self-BLEU remains very sensitive to local syntax , and it fails to capture global consistency and diverse semantic information in generated text . Many other metrics exist for generative models which do not approach the problem of diversity . Our work is not the first to recognize this gap in the literature ( Borji , 2019 ) .

Summary: In this paper, the authors introduce a new quantitative diversity measure advocating its usage for generative models evaluation. In a nutshell, to measure the diversity of a particular set, the authors split it into disjoint train/val subsets and learn a DNN to predict the outputs of another randomly initialized DNN on the train set. Then the generalization gap of the trained DNN is computed on the unseen val subset, and the normalized value of this gap (averaged over several splits/initializations) is considered as a diversity measure.