On Self-Supervised Image Representations for GAN Evaluation

1 INTRODUCTION . Generative adversarial networks ( GANs ) are an extremely active research direction in machine learning . The intensive development of the field requires established quantitative measures to assess constantly appearing models . While a large number of evaluation protocols were proposed ( Borji , 2019 ; Xu et al. , 2018 ; Zhou et al. , 2019 ; Naeem et al. , 2020 ) , there is still no consensus regarding the best evaluation measure . Across the existing measures , the Fréchet Inception Distance ( FID ) ( Heusel et al. , 2017 ) and Precision/Recall ( Kynkäänniemi et al. , 2019 ) are the most widely adopted due to their simplicity and decent consistency with human judgments . FID and Precision/Recall quantify the discrepancy between distributions of real and generated images . Since these distributions are complicated to describe in the original RGB space , the images are represented by embeddings , typically extracted with CNNs pretrained on the Imagenet classification ( Deng et al. , 2009 ) . While FID computed with these embeddings was shown to correlate with human evaluation ( Heusel et al. , 2017 ) , these observations were mostly obtained on datasets , semantically close to Imagenet . Meanwhile , on non-Imagenet datasets , FID can result in inadequate evaluation , as widely reported in the literature ( Rosca et al. , 2017 ; Barratt & Sharma , 2018 ; Zhou et al. , 2019 ) . In this work , we propose to employ the state-of-the-art self-supervised models ( Chen et al. , 2020a ; He et al. , 2020 ; Caron et al. , 2020 ) to extract image embeddings for GAN evaluation . These models were shown to produce features that transfer better to new tasks , hence , they become a promising candidate to provide a more universal representation . Intuitively , classification-pretrained embeddings by design can suppress the information , irrelevant for the Imagenet class labels , which , however , can be crucial for other domains , like human faces . On the contrary , self-supervised models , mostly trained via contrastive or clustering-based learning , do not have such a bias since their main goal is typically to learn invariances to common image augmentations . To justify the usage of self-supervised embeddings , we perform a thorough comparison of the recent GAN models trained on the five most common benchmark datasets . We demonstrate that classification-pretrained embeddings can lead to incorrect ranking in terms of FID , Precision , and Recall , which are the most popular metrics . On the other hand , self-supervised representations produce more sensible ranking , advocating their advantage over “ classification-oriented ” counterparts . Since all the checkpoints needed to compute self-supervised embeddings are publicly available , they can serve as a handy instrument for GAN comparison , consistent between different papers . We release the code for the “ self-supervised ” GAN evaluation along with data and human labeling reported in the paper online1 . To sum up , the contributions of this paper are as follows : 1 . To the best of our knowledge , our work is the first to employ self-supervised image representations to evaluate GANs trained on natural images . 2 . By extensive experiments on the standard non-Imagenet benchmarks , we demonstrate that the usage of self-supervised representations provides a more reliable GAN comparison . 3 . We show that the FID measure computed with self-supervised representations often has higher sample-efficiency and analyze the sources of this advantage . 2 RELATED WORK . GAN evaluation measures . Over the last years , a variety of quantitative GAN evaluation methods have been developed by the community , and the development process has yet to converge since all the measures possess specific weaknesses ( Borji , 2019 ; Xu et al. , 2018 ) . The Inception Score ( Salimans et al. , 2016 ) was the first widely adopted measure but was shown to be hardly applicable for non-Imagenet domains ( Barratt & Sharma , 2018 ) . The Fréchet Inception Distance ( FID ) ( Heusel et al. , 2017 ) quantifies dissimilarity of real and generated distributions , computing the Wasserstein distance between their Gaussian approximations , and is currently the most popular scalar measure of GAN ’ s quality . Several recent measures were proposed ( Sajjadi et al. , 2018 ; Kynkäänniemi et al. , 2019 ; Naeem et al. , 2020 ) that separately evaluate fidelity and diversity of GAN-produced images . All of them mostly use the embeddings produced by the Imagenet classification CNN . A recent work ( Zhou et al. , 2019 ) has introduced a human-in-the-loop measure , which is more reliable compared to automated ones but can not be used , e.g. , for monitoring the training process . We focus on three the most widely used measures : FID , Precision , and Recall , which are discussed briefly below . Fréchet Inception Distance quantifies the discrepancy between the distributions of real and generated images , denoted by pD and pG . Both pD and pG are defined on the high-dimensional image space forming nontrivial manifolds , which are challenging to approximate by simple functions . To be practical , FID operates in the lower-dimensional space of image embeddings . Formally , the embeddings are defined by a map f : RN → Rd , where N and d correspond to the dimensionalities of the images and embeddings spaces , respectively . By design , FID measures the dissimilarity between the induced distributions fpD , fpG as follows . First , fpD and fpG are approximated by Gaussian distributions . Then the Wasserstein distance between these distributions is evaluated . As was shown in ( Dowson & Landau , 1982 ) , for distributions defined by the means µD , µG and the covariance matrices ΣD , ΣG , this quantity equals to ‖µD − µG‖22 + tr ( ΣD + ΣG − 2 ( ΣDΣG ) 1 2 ) . Lower FID values correspond to higher similarity between pG and pD ; hence , can be used to evaluate the performance of generative models . As a common practice in the FID computation , one typically uses the activations from the InceptionV3 ( Szegedy et al. , 2016 ) pretrained on Imagenet classification . Precision and Recall . When assessing generative models , it is important to quantify both the visual quality of generated images and the model diversity , e.g. , to diagnose mode collapsing . However , the scalar FID values were shown ( Sajjadi et al. , 2018 ; Kynkäänniemi et al. , 2019 ) to sacrifice diversity in favor of visual quality , therefore FID can not serve as the only sufficient metric . To this end , ( Sajjadi et al. , 2018 ) introduced Precision and Recall , which aim to measure the image realism and the model diversity , respectively . A recent follow-up ( Kynkäänniemi et al. , 2019 ) elaborates on these metrics and proposes a reasonable procedure to quantify both precision and recall based only on the image embeddings . In a nutshell , ( Kynkäänniemi et al. , 2019 ) assumes that the visual quality of a particular sample is high if its embedding is neighboring for the embeddings of the real images . On 1https : //github.com/stanis-morozov/self-supervised-gan-eval the other hand , a given real image is considered covered by the model if its embedding belongs to the neighborhood of embeddings of the generated images . Self-supervised representations . Self-supervised learning is currently attracting much research attention , especially to contrastive learning and clustering-based methods ( Chen et al. , 2020a ; He et al. , 2020 ; Caron et al. , 2020 ) . The common idea behind these methods is to construct representations that are invariant to a wide range of common image augmentations . The recent self-supervised methods were shown to provide more transferrable ( He et al. , 2020 ; Caron et al. , 2020 ) and robust ( Hendrycks et al. , 2019 ) features , which implies their usage as more universal representations . In this paper , we show them being a better alternative compared to established classifier-produced embeddings in the context of GAN assessment . 3 GAN EVALUATION . Here we systematically compare the publicly available GANs to highlight the cases of misleading comparison with classification-pretrained embeddings . Our goal is to demonstrate that selfsupervised embeddings are a better alternative in these cases , while in other cases , the rankings with both types of embeddings are mostly consistent . We examine open-sourced GAN models2 trained on five popular benchmarks : • CelebaHQ 1024x1024 ( Karras et al. , 2017 ) with the following GAN models : StyleGAN with truncation 0.7 ( Karras et al. , 2019a ) and without it , MSG ( Karnewar & Wang , 2020 ) with truncation 0.6 and without it , PGGAN ( Karras et al. , 2017 ) . To compute the metrics , we use 30k real and synthetic images ; • FFHQ 1024x1024 ( Karras et al. , 2019a ) with the following GAN models : StyleGAN ( Karras et al. , 2019a ) , StyleGAN2 ( Karras et al. , 2019b ) , MSG ( Karnewar & Wang , 2020 ) with truncation 0.6 and without it . To compute the metrics , we use 30k real and synthetic images ; • LSUN Bedroom 256x256 ( Yu et al. , 2015 ) with the following GAN models : StyleGAN ( Karras et al. , 2019a ) with truncation 0.7 and without it , PGGAN ( Karras et al. , 2017 ) , COCO-GAN ( Lin et al. , 2019 ) , RPGAN ( Voynov & Babenko , 2019 ) , RPGAN with high diversity ( RPGAN div. ) . RPGAN generates 128x128 images , so we upscale them to 256x256 . To compute the metrics , we use 30k real and synthetic images ; • LSUN Church 256x256 ( Yu et al. , 2015 ) with the models : StyleGAN2 ( Karras et al. , 2019b ) with truncation 0.5 and without it , MSG ( Karnewar & Wang , 2020 ) with truncation 0.6 and without it , PGGAN ( Karras et al. , 2017 ) , SNGAN ( Miyato et al. , 2018 ) . SNGAN generates 128x128 images , so we upscale them to 256x256 . To compute the metrics , we use 100k real and synthetic images ; • Imagenet 128x128 ( Deng et al. , 2009 ) with the following GAN models : BigGAN ( Brock et al. , 2019 ) , BigGAN-deep ( Brock et al. , 2019 ) ( both with truncation 2.0 ) , S3GAN ( Lucic et al. , 2019 ) , Your Local GAN ( YLG ) ( Daras et al. , 2020 ) . To compute the metrics , we use 50k images ( 50 per class ) . We include this dataset to demonstrate that for Imagenet , the proposed self-supervised representations provide consistent ranking with commonly used InceptionV3 embeddings . To compute image embeddings , we use the following publicly available models : • InceptionV3 ( Szegedy et al. , 2016 ) pretrained on the ILSVRC-2012 task ( Deng et al. , 2009 ) ; • Resnet50 ( He et al. , 2016 ) pretrained on the ILSVRC-2012 task . We include this model since self-supervised models employ Resnet50 , therefore , it is important to demonstrate that better GAN ranking comes from the training objective rather than the deeper architecture ; • Imagenet21k ( Kolesnikov et al. , 2019 ) pretrained on the multi-label classification task on approximately 14M images from the full Imagenet . Kolesnikov et al . ( 2019 ) have shown that supervised pretraining on huge datasets provides more transferrable features , therefore , Imagenet21k can also potentially provide more universal representations . The model architecture is Resnet50 ; • SwAV ( Caron et al. , 2020 ) is the state-of-the-art self-supervised image representation model trained on ILSVRC-2012 . The idea of SwAV is to simultaneously cluster the images while enforcing consistency between cluster assignments produced for different augmentations of the same image . The model architecture is Resnet50 ; 2The URLs for all models are provided in Appendix . • DeepClusterV2 ( Caron et al. , 2020 ) is another self-supervised model obtained by alternating between pseudo-labels generation via k-means clustering and training the network with a classification loss supervised by these pseudo-labels . The model architecture is Resnet50 ; • MoCoV2 ( Chen et al. , 2020b ) is the state-of-the-art contrastive learning approach , which training objective enforces the closeness of representations produced for different augmentations of the same image while pushing apart the representations of unrelated images . The model architecture is Resnet50 . Three self-supervised models listed above outperform supervised pretraining on a number of transfer tasks ( He et al. , 2020 ; Caron et al. , 2020 ) , which implies that their embeddings capture more information relevant for these tasks , compared to supervised models pretrained on Imagenet . Below , for a large number of publicly available GANs , we present the values of FID , Precision , and Recall metrics computed with different embeddings . For the cases where the GANs ranking is inconsistent , we aim to show that the ranking obtained with the self-supervised representations is more reasonable .

Summary: The papers looks at the problem of evaluating GAN samples. Current methods, such as FID/PR with Inception v3 are problematic because they generally depend on using the features of a model discriminatively trained on (a super set of) ImageNet. The authors show that these type of models ignore details that are meaningful when for example comparing results on CelebaHQ.