On Self-Supervised Image Representations for GAN Evaluation

1 INTRODUCTION . Generative adversarial networks ( GANs ) are an extremely active research direction in machine learning . The intensive development of the field requires established quantitative measures to assess constantly appearing models . While a large number of evaluation protocols were proposed ( Borji , 2019 ; Xu et al. , 2018 ; Zhou et al. , 2019 ; Naeem et al. , 2020 ) , there is still no consensus regarding the best evaluation measure . Across the existing measures , the Fréchet Inception Distance ( FID ) ( Heusel et al. , 2017 ) and Precision/Recall ( Kynkäänniemi et al. , 2019 ) are the most widely adopted due to their simplicity and decent consistency with human judgments . FID and Precision/Recall quantify the discrepancy between distributions of real and generated images . Since these distributions are complicated to describe in the original RGB space , the images are represented by embeddings , typically extracted with CNNs pretrained on the Imagenet classification ( Deng et al. , 2009 ) . While FID computed with these embeddings was shown to correlate with human evaluation ( Heusel et al. , 2017 ) , these observations were mostly obtained on datasets , semantically close to Imagenet . Meanwhile , on non-Imagenet datasets , FID can result in inadequate evaluation , as widely reported in the literature ( Rosca et al. , 2017 ; Barratt & Sharma , 2018 ; Zhou et al. , 2019 ) . In this work , we propose to employ the state-of-the-art self-supervised models ( Chen et al. , 2020a ; He et al. , 2020 ; Caron et al. , 2020 ) to extract image embeddings for GAN evaluation . These models were shown to produce features that transfer better to new tasks , hence , they become a promising candidate to provide a more universal representation . Intuitively , classification-pretrained embeddings by design can suppress the information , irrelevant for the Imagenet class labels , which , however , can be crucial for other domains , like human faces . On the contrary , self-supervised models , mostly trained via contrastive or clustering-based learning , do not have such a bias since their main goal is typically to learn invariances to common image augmentations . To justify the usage of self-supervised embeddings , we perform a thorough comparison of the recent GAN models trained on the five most common benchmark datasets . We demonstrate that classification-pretrained embeddings can lead to incorrect ranking in terms of FID , Precision , and Recall , which are the most popular metrics . On the other hand , self-supervised representations produce more sensible ranking , advocating their advantage over “ classification-oriented ” counterparts . Since all the checkpoints needed to compute self-supervised embeddings are publicly available , they can serve as a handy instrument for GAN comparison , consistent between different papers . We release the code for the “ self-supervised ” GAN evaluation along with data and human labeling reported in the paper online1 . To sum up , the contributions of this paper are as follows : 1 . To the best of our knowledge , our work is the first to employ self-supervised image representations to evaluate GANs trained on natural images . 2 . By extensive experiments on the standard non-Imagenet benchmarks , we demonstrate that the usage of self-supervised representations provides a more reliable GAN comparison . 3 . We show that the FID measure computed with self-supervised representations often has higher sample-efficiency and analyze the sources of this advantage . 2 RELATED WORK . GAN evaluation measures . Over the last years , a variety of quantitative GAN evaluation methods have been developed by the community , and the development process has yet to converge since all the measures possess specific weaknesses ( Borji , 2019 ; Xu et al. , 2018 ) . The Inception Score ( Salimans et al. , 2016 ) was the first widely adopted measure but was shown to be hardly applicable for non-Imagenet domains ( Barratt & Sharma , 2018 ) . The Fréchet Inception Distance ( FID ) ( Heusel et al. , 2017 ) quantifies dissimilarity of real and generated distributions , computing the Wasserstein distance between their Gaussian approximations , and is currently the most popular scalar measure of GAN ’ s quality . Several recent measures were proposed ( Sajjadi et al. , 2018 ; Kynkäänniemi et al. , 2019 ; Naeem et al. , 2020 ) that separately evaluate fidelity and diversity of GAN-produced images . All of them mostly use the embeddings produced by the Imagenet classification CNN . A recent work ( Zhou et al. , 2019 ) has introduced a human-in-the-loop measure , which is more reliable compared to automated ones but can not be used , e.g. , for monitoring the training process . We focus on three the most widely used measures : FID , Precision , and Recall , which are discussed briefly below . Fréchet Inception Distance quantifies the discrepancy between the distributions of real and generated images , denoted by pD and pG . Both pD and pG are defined on the high-dimensional image space forming nontrivial manifolds , which are challenging to approximate by simple functions . To be practical , FID operates in the lower-dimensional space of image embeddings . Formally , the embeddings are defined by a map f : RN → Rd , where N and d correspond to the dimensionalities of the images and embeddings spaces , respectively . By design , FID measures the dissimilarity between the induced distributions fpD , fpG as follows . First , fpD and fpG are approximated by Gaussian distributions . Then the Wasserstein distance between these distributions is evaluated . As was shown in ( Dowson & Landau , 1982 ) , for distributions defined by the means µD , µG and the covariance matrices ΣD , ΣG , this quantity equals to ‖µD − µG‖22 + tr ( ΣD + ΣG − 2 ( ΣDΣG ) 1 2 ) . Lower FID values correspond to higher similarity between pG and pD ; hence , can be used to evaluate the performance of generative models . As a common practice in the FID computation , one typically uses the activations from the InceptionV3 ( Szegedy et al. , 2016 ) pretrained on Imagenet classification . Precision and Recall . When assessing generative models , it is important to quantify both the visual quality of generated images and the model diversity , e.g. , to diagnose mode collapsing . However , the scalar FID values were shown ( Sajjadi et al. , 2018 ; Kynkäänniemi et al. , 2019 ) to sacrifice diversity in favor of visual quality , therefore FID can not serve as the only sufficient metric . To this end , ( Sajjadi et al. , 2018 ) introduced Precision and Recall , which aim to measure the image realism and the model diversity , respectively . A recent follow-up ( Kynkäänniemi et al. , 2019 ) elaborates on these metrics and proposes a reasonable procedure to quantify both precision and recall based only on the image embeddings . In a nutshell , ( Kynkäänniemi et al. , 2019 ) assumes that the visual quality of a particular sample is high if its embedding is neighboring for the embeddings of the real images . On 1https : //github.com/stanis-morozov/self-supervised-gan-eval the other hand , a given real image is considered covered by the model if its embedding belongs to the neighborhood of embeddings of the generated images . Self-supervised representations . Self-supervised learning is currently attracting much research attention , especially to contrastive learning and clustering-based methods ( Chen et al. , 2020a ; He et al. , 2020 ; Caron et al. , 2020 ) . The common idea behind these methods is to construct representations that are invariant to a wide range of common image augmentations . The recent self-supervised methods were shown to provide more transferrable ( He et al. , 2020 ; Caron et al. , 2020 ) and robust ( Hendrycks et al. , 2019 ) features , which implies their usage as more universal representations . In this paper , we show them being a better alternative compared to established classifier-produced embeddings in the context of GAN assessment . 3 GAN EVALUATION . Here we systematically compare the publicly available GANs to highlight the cases of misleading comparison with classification-pretrained embeddings . Our goal is to demonstrate that selfsupervised embeddings are a better alternative in these cases , while in other cases , the rankings with both types of embeddings are mostly consistent . We examine open-sourced GAN models2 trained on five popular benchmarks : • CelebaHQ 1024x1024 ( Karras et al. , 2017 ) with the following GAN models : StyleGAN with truncation 0.7 ( Karras et al. , 2019a ) and without it , MSG ( Karnewar & Wang , 2020 ) with truncation 0.6 and without it , PGGAN ( Karras et al. , 2017 ) . To compute the metrics , we use 30k real and synthetic images ; • FFHQ 1024x1024 ( Karras et al. , 2019a ) with the following GAN models : StyleGAN ( Karras et al. , 2019a ) , StyleGAN2 ( Karras et al. , 2019b ) , MSG ( Karnewar & Wang , 2020 ) with truncation 0.6 and without it . To compute the metrics , we use 30k real and synthetic images ; • LSUN Bedroom 256x256 ( Yu et al. , 2015 ) with the following GAN models : StyleGAN ( Karras et al. , 2019a ) with truncation 0.7 and without it , PGGAN ( Karras et al. , 2017 ) , COCO-GAN ( Lin et al. , 2019 ) , RPGAN ( Voynov & Babenko , 2019 ) , RPGAN with high diversity ( RPGAN div. ) . RPGAN generates 128x128 images , so we upscale them to 256x256 . To compute the metrics , we use 30k real and synthetic images ; • LSUN Church 256x256 ( Yu et al. , 2015 ) with the models : StyleGAN2 ( Karras et al. , 2019b ) with truncation 0.5 and without it , MSG ( Karnewar & Wang , 2020 ) with truncation 0.6 and without it , PGGAN ( Karras et al. , 2017 ) , SNGAN ( Miyato et al. , 2018 ) . SNGAN generates 128x128 images , so we upscale them to 256x256 . To compute the metrics , we use 100k real and synthetic images ; • Imagenet 128x128 ( Deng et al. , 2009 ) with the following GAN models : BigGAN ( Brock et al. , 2019 ) , BigGAN-deep ( Brock et al. , 2019 ) ( both with truncation 2.0 ) , S3GAN ( Lucic et al. , 2019 ) , Your Local GAN ( YLG ) ( Daras et al. , 2020 ) . To compute the metrics , we use 50k images ( 50 per class ) . We include this dataset to demonstrate that for Imagenet , the proposed self-supervised representations provide consistent ranking with commonly used InceptionV3 embeddings . To compute image embeddings , we use the following publicly available models : • InceptionV3 ( Szegedy et al. , 2016 ) pretrained on the ILSVRC-2012 task ( Deng et al. , 2009 ) ; • Resnet50 ( He et al. , 2016 ) pretrained on the ILSVRC-2012 task . We include this model since self-supervised models employ Resnet50 , therefore , it is important to demonstrate that better GAN ranking comes from the training objective rather than the deeper architecture ; • Imagenet21k ( Kolesnikov et al. , 2019 ) pretrained on the multi-label classification task on approximately 14M images from the full Imagenet . Kolesnikov et al . ( 2019 ) have shown that supervised pretraining on huge datasets provides more transferrable features , therefore , Imagenet21k can also potentially provide more universal representations . The model architecture is Resnet50 ; • SwAV ( Caron et al. , 2020 ) is the state-of-the-art self-supervised image representation model trained on ILSVRC-2012 . The idea of SwAV is to simultaneously cluster the images while enforcing consistency between cluster assignments produced for different augmentations of the same image . The model architecture is Resnet50 ; 2The URLs for all models are provided in Appendix . • DeepClusterV2 ( Caron et al. , 2020 ) is another self-supervised model obtained by alternating between pseudo-labels generation via k-means clustering and training the network with a classification loss supervised by these pseudo-labels . The model architecture is Resnet50 ; • MoCoV2 ( Chen et al. , 2020b ) is the state-of-the-art contrastive learning approach , which training objective enforces the closeness of representations produced for different augmentations of the same image while pushing apart the representations of unrelated images . The model architecture is Resnet50 . Three self-supervised models listed above outperform supervised pretraining on a number of transfer tasks ( He et al. , 2020 ; Caron et al. , 2020 ) , which implies that their embeddings capture more information relevant for these tasks , compared to supervised models pretrained on Imagenet . Below , for a large number of publicly available GANs , we present the values of FID , Precision , and Recall metrics computed with different embeddings . For the cases where the GANs ranking is inconsistent , we aim to show that the ranking obtained with the self-supervised representations is more reasonable .

Summary: This paper proposes to use image representations from trained self-supervised models to evaluate GANs more accurately. Compared to the currently used representations from supervised-pretrained models e.g. InceptionV3, the authors claim, that such embeddings suppress information not critical for the classification process which, however, are assumed to be crucial for assessing the full distributions of real and generated images. The authors use 5 datasets and their respective representations from 5 models, 3 supervised and 2 self-supervised, to show that representations from self-supervised models lead to better GAN evaluations. The representations were used to evaluate 6 GAN models with 3 metrics, namely FID, Precision and Recall. A ranking of the GAN models shows inconsistencies between supervised and self-supervised based representations. By visual inspection, prediction accuracy tests, and a comparison of representation invariances the authors show that rankings via self-supervised embeddings are more plausible.