On Self-Supervised Image Representations for GAN Evaluation
Authors: Stanislav Morozov
Andrey Voynov
Artem Babenko
Venue: ICLR 2021 Spotlight (2021)
Keywords: GAN
evaluation
embedding

Abstract: The embeddings from CNNs pretrained on Imagenet classification are de-facto standard image representations for assessing GANs via FID, Precision and Recall measures. Despite broad previous criticism of their usage for non-Imagenet domains, these embeddings are still the top choice in most of the GAN literature.

In this paper, we advocate the usage of the state-of-the-art self-supervised representations to evaluate GANs on the established non-Imagenet benchmarks. These representations, typically obtained via contrastive learning, are shown to provide better transfer to new tasks and domains, therefore, can serve as more universal embeddings of natural images. With extensive comparison of the recent GANs on the common datasets, we show that self-supervised representations produce a more reasonable ranking of models in terms of FID/Precision/Recall, while the ranking with classification-pretrained embeddings often can be misleading.

Published as a conference paper at ICLR 2021
ON SELF-SUPERVISED IMAGE REPRESENTATIONS FOR GAN EVALUATION
Stanislav Morozov
Andrey Voynov Yandex
Marchuk Institute of Numerical Mathematics RAS stanis-mit meweiiissoooooooooaaoooooo ru
avoinéyandex-tem.
Artem Babenko
Yandex
HSE University
artem. batambbbbbenkkphysttcch. edu
ABSTRACT
The embeddings from CNNs pretrained on Imagenet classification are de-facto standard image representations for assessing GANs via FID, Precision and Re- call measures. Despite broad previous criticism of their usage for non-Imagenet domains, these embeddings are still the top choice in most of the GAN literature. In this paper, we advocate using state-of-the-ar self-supervised representations to evaluate GANs on the established non-Imagenet benchmarks. These representa- tions, typically obtained via contrastive or clustering- approaches, provide better transfer new tasks and domains, therefore, can serve as more universal embeddings of natural images. With extensive comparison of the recent GANs on the standard datasets, we demonstrate that supersueeee representations pro- duce more reasonable ranking of models in terms of FID//PrecisionRRecall while the ranking with classification- clasiffcation-pretrainee pretraiind embeddings often can be misleading. Furthermore, using self-supervised representations often improves the sample- efficiency of FID, which makes it more reliable in limited-data regimes.
INTRODUCTION
Generative adversarial networks (GANs) are an extremely active research direction in machine learning. The intensive development of the field requires established quantitative measures assess constantly appearing models. While large number of evaluation protocols were proposed (Borji 2019 Xu al. 2018 Zhou al. 2019 Naeem et al. 2020), there still no consensus regarding the best evaluatiin measure. Across the existing measures, the Fréchet Inception Distance (FID) (Heusel ct al. 2017) and Precision/Recall (Kynkäänniemi et al. 2019) are the most widely adopted due to their simplicity and decent consistency with human judgments. FID and Precision/Recal quantify the discrepancy between distributions of real and generated images. Since these distributions are complicated describe in the original RGB space, the images are represented by embeddings. typ- ically extracted with CNNs pretrained on the Imagenet classification (Deng et al. 2009). While FID computed with these embeddings was shown correlate with human evaluation (Heusel al. 2017). these observations were mostly obtained on datasets, semantically close to Imagenet. Mean- while, on non- mon-enetenet datasets, FID can result inadequate evaluation, as widely reported in the literature (Rosca et al. 2017 Barratt & Sharma 2018 Zhou et al. 2019).
In this work, we propose to employ the state-of-the-an self-supervised models (Chen et al.. 2020a He et al.][020) [ Caron et al. (020)) to extract image embeddings for GAN evaluation. These models were shown to produce features that transfer better to new tasks, hence, they become promising candidate to provide a more universal representation. Intuitively, classification-preraiin dings by design can suppress the information, irrelevant for the Imagenet class labels, which, how- ever, be crucial for other domains, like human faces. On the contrary, self- self-supervised models, mostly trained via contrastive or clustering blusedring-based learning, do not have such a bias since their main goal is typically to learn invariances to common image augmentations.
embed-

---

Published as a conference paper at ICLR 2021
To justify the usage of self-supervised embeddings, we perform thorough comparison of the cent GAN models trained on the five most common benchmark datasets. We demonstrate that Dussiifiattinnpretaaiin
embeddings can lead to incorrect ranking in terms of FID, Precision, and Recall, which are the most popular metrics. On the other hand, self-supervised representations pro- duce more sensible ranking, advocating their advantage over "classification-riinnee Since all the checkpoints needed to compute self-supervised embeddings are publicly available, they can serve as a handy instrument for GAN comparison, consistent between different papers. We release the code for the "self="sspeeereeee supervised" GAN evaluation along with data and human labeling reported in the paper online" To sum up, the contributions of this paper are as follows:
counterparts.
1. To the best of our knowledge, our work is the first to employ self-supervised image repre- sentations to evaluate GANs trained on natural images.
2. By extensive experiments the standard non-Imagenet benchmarks, we demonstrate that the usage of self-supervised representations provides a more reliable GAN comparison.
3. We show that the FID measure computed with self-supervised representations often has higher sample-efficiency and analyze the sources of this advantage.
2 RELATED WORK
GAN evaluation measures. Over the last years, a variety of quantitative GAN evaluation methods have been developed by the community, and the development process has yet to converge since all the measures possess specific weaknesses (Borj] 2019 Xu al.] 2018). The Inception Score (Slliaan et al. 2016) was the first widely adopted measure but was shown to be hardly applicable for non- -on-emmteentt domains (Barratt& & Sharma 2018). The Fréchet Inception Distance (FID) (Heusel et al., 2017) quantifies dissimilarity of real and generated distributions, computing the Wasserstein distance between their Gaussian approximations, and is currently the most popular scalar measure of GAN's quality. Several recent measures were proposed (Sajjadi et al. 2018 Kynkäänniemi al., 2019 Naeem et al., 2020) that separately evaluate fidelity and diversity of GAN-produced images. All of them mostly use the embeddings produced by the Imagenet classification CNN. A recent work (Zhou et al. 2019) has introduced human-ing memtan-in-theee tn-the-ooop measure, which is more reliable compared to automated ones but cannot be used, e.g., for monitoring the training process. We focus on three the most widely used measures: FID, Precision, and Recall, which are discussed briefly below.
Fréchet Inception Distance quantifies the discrepancy between the distributions of real and gener- ated images, denoted by PD and PG. Both pD and PG are defined on the high- nigh-dimensional image space forming nontrivial manifolds, which are challenging to approximate by simple functions. To be practical, FID operates in the lower-dimensional space of image embeddings. Formally, the em- beddings are defined by a map f R + Rd, where N and d correspond to the dimensionalities of the images and embeddings spaces, respectively. By design, FID measures the dissimilarity between the induced distributions fpD, f as follows. First, fPD and fPG are approximated by Gaussian distributions. Then the Wasserstein distance between these distributions is evaluated. As was shown in (Dowson & Landau 1982). for distributions defined by the means PD: Mg and the covariance matrices Ep Eg. this quantity equals to ||μ - -Mall3 + tr(Ep + Σg 2 Eg) Lower FII) values correspond to higher similarity between PG and pD: hence, can be used to evaluate the per- formance of generative models. As a common practice in the FID computation, one typically uses the activations from the Inception (Szegedy et al. 2016) pretrained on Imagenet classification.
Precision and Recall. When assessing generative models, it is important to quantify both the visual quality of generated images and the model diversity, e.g.. to diagnose mode collapsing. However, the scalar FID values were shown (Sajjadi et al. 2018 Kynkäänniemi et al.) (019) to sacrifice diversity in favor of visual quality, therefore FID cannot serve as the only sufficient metric. To this end, (Sajaadi et al.] 2018) introduced Precision and Recall, which aim to measure the image realism and the model diversity, respectively. A recent follow-up (Kynkäänniemi et all. 2019) elaborates on these metrics and proposes a reasonable procedure to quantify both precision and recall based only on the image embeddings. In a nutshell, (Kynkäänniemi et al.) 2019) assumes that the visual quality of a particular sample is high if its embedding is neighboring for the embeddings of the real images. On
https://githubs cttoo///taaiie
susprvissedeeeeee sed-gan-evv" sen
2

---

Published as a conference paper at ICLR 2021
the other hand, given real image is considered covered by the model if its embedding belongs to the neighborhood of embeddings of the generated images.
Self-supervised representations. Self-supervised learning is currently attracting much research attention, especially to contrastive learning and clustering -aaerinn--bsedd methods (Chen et al. 2020a He et al. 2020 Caron al. 2020). The common idea behind these methods is construct representa- tions that are invariant to a wide range of common image augmentations. The recent supf-suee
methods were shown to provide more transferrable (He al. 2020 Caron al. 2020) and ro-
bust (Hendrycks et al. 2019) features, which implies their usage as more universal representations. In this paper, we show them being a better alternative compared established fassifier-producee embeddings in the context of GAN assessment.
3 GAN EVALUATION
Here we systematically compare the publicly available GANs to highlight the cases of mislead- ing comparison with daasification-pretrainee embeddings. Our goal is to demonstrate that self- supervised embeddings are better alternative in these cases, while other cases, the rankings with both types of embeddings are mostly consistent. We examine open- open-sourced GAN model trained on five popular benchmarks:
CelebaHQ 1024x1024 (Krrras et al. 2017) with the following GAN models: StyleGAN with truncation 0.7 (Karras et al. (019a) and without it, MSG (Karnewar & Wang (020) with trunca- tion 0.6 and without it, PGGAN (Karras et al. 2017). To compute the metrics, we use 30k real and synthetic images;
FFHQ 1024x 1024 (Karras et al. 2019a) with the following GAN models: StyleGAN (Karras
et al. 2019a), StyleGAN2 (Karras et al. (2019)), MSG (Karnewar & Wang, (2020) with truncation 0.6 and without it. To compute the metrics, we use 30k real and synthetic images;
LSUN Bedroom 256x256 (Yu et al. 2015) with the following GAN models; StyleGAN (Karaas 2019a) with truncation 0.7 and without PGGAN (Karras et al. 2017), COCO-GAN (Lin
ct al.
et al.
2019) RPGAN (Voynov & Babenko 2019), RPGAN with high diversity (RPGAN div.). RPGAN generates 128x images, so we upscale them to 256x256. To compute the metrics, we
use 30k real and synthetic images;
LSUN Church 256x256 (Yu et al. 2015) with the models: StyleGAN2 (Karras et al. 2019b) with truncation 0.5 and without it, MSG (Karnewar & Wang (020) with truncation 0.6 and without it, PGGAN Karras et al. 2017). 7). SNGAN (Miyato et 2018). SNGAN generates 128x 128 images, upscale them to 256x256. To compute the metries, we use 100k real and synthetic images;
Imagenet 128x128 Deng et al.] 2009) with the following GAN models: BigGAN (Brock et al. 2019) BigGANNdeey (Brock al. 20019] (both with truncation 2.0), S3GAN (Lucic al. [019)). Your Local GAN (YLG) (Daras et al. 2020). To compute the metrics, we use 50k images (50 per class). We include this dataset to demonstrate that for Imagenet, the proposed self-supervised representations provide consistent ranking with commonly used Inception V3 embeddings.
To compute image embeddings, we use the following publicly available models:
InceptionV2 V (Szegedy et al. 2016) pretrained on the ILSVRC-2012 task (Deng et al. 2009);
Resnet50 (He et al. 2016) pretrained the ILSVRC-2012 task. We include this model since self-supervised models employ Resnet50, therefore, it is important to demonstrate that better GAN ranking comes from the training objective rather than the deeper architecture;
Imagenet21k [Kolesnikov et al.) 2019) pretrained on the multi-label classification task on approx- imately 14M images from the full Imagenet. Kolesnikov et al. (2019) have shown that supervised pretraining on huge datasets provides more transferrable features, therefore, Imagenet21k 1k can also potentially provide more universal representations. The model architecture is Resnet50;
SwAV (Caron et al., 2020) is the state-of-the- self-supervised image representation model trained on ILSVRC-2012. The idea of SwAV is to simultaneously cluster the images while en- forcing consistency between cluster assignments produced for different augmentations of the same image. The model architecture Resnet50;
"The URLs for all models are provided in Appendix.
3

---

Published as a conference paper at ICLR 2021
Table 1: FID values computed with different embeddings. The *** symbol indicates models with
truncation. The inconsistencies between Inception V3 and SwAV rankings are highlighted in color.
CELEBAHQ
FFH
StyleGAN
MSG
PGGAN
StyleGAN
MSG'
StyleGAN2 MSG StyleGAN MSG
InceptionV3
5.958
7.041
7.747
12.761
18.845
3.355
6.560
6.89
22.552
StyleGAN
MSG
PGGAN
StyleGAN
MSG
StyleGAN2 MSG StyleGAN MSG
Resnet50
5.981
7.427
9.395
13.650
19.048
3.813
7.252
8.230
25.076
MSG
StyleGAN 301.3
PGGAN 451.6
StyleGAN
MSG
StyleGAN2 MSG StyleGAN MSG
Imagenet21k
295.3
635.5
874.4
177.7
334.7
389.1
1092
MSG
StyleGAN 1,304
StyleGAN
MSG" 1.832
PGGAN
StyleGAN2 MSG StyleGAN MSG"
SwAV
1.206
1.473
1.898
0.634
1.275
1.482
2.461
MSG
StyleGAN 2,255
StyleGAN 2.680
PGGAN
MSG
StyleGAN2 MSG StyleGAN MSG
DeepClusterV2
1.847
2.865
2.935
0.978
1890
2.076
3,926
MSG
StyleGAN 0.009
PGGAN 0.012
StyleGAN
MSG"
StyleGAN2 MSG StyleGAN MSG
MoCoV2
0.008
0.016
0.023
0.005
0.009 0.010 0.035
-SUN-BEDROOM
StyleGAN 2.986
PGGAN 8.658
StyleGAN COCO-GAN
RPGAN
RPGAN div. 40.165
Inception V3
9.655
18.612
37.924188
Resnet50 StyleGAN
PGGAN
StyleGAN
COCO-GAN
RPGAN
RPGAN div.
6,263
17.689
20.042
34.045
44.850
51.850
StyleGAN StyleGAN 321.7
PGGAN COCO-GAN RPGAN div.
RPGAN
Imagene121K k
633.3
940.6
1270
1589
1593
SwAV StyleGAN StyleGAN 1.475
PGGAN
RPGAN
COCO-GAN RPGAN div.
1.776
4.160
5.608
6.289
6,460
DeepClusterV2 StyleGAN StyleGAN 2,095
PGGAN
RPGAN
COCO-GAN RPGAN div.
2.958
5,855
8.757
9.151
10.13
StyleGAN StyleGAN 0.012
PGGAN 0.037
RPGAN 0.072
RPGAN div. COCO-GAN 0.083
MoCoV2
0.031
0.085
LSUN-CHURCH
Inception StyleGAN2 3,652
MSG
PGGAN MSG StyleGAN2
SNGAN
5.009
6.296
13.854
24.966
32.661
Resnet50 StyleGAN2
MSG
PGGAN MSG StyleGAN2
SNGAN
6.650
9.673
10.850
22.670
55.571
56.114
StyleGAN2 558.1
MSG
PGGAN MSG
SNGAN
StyleGAN2
Imagenet21K
856.8
946.0 1305.3
1715
2165
SWAV StyleGAN2 PGGAN
MSG 3.578
MSG StyleGAN2 5.194
SNGAN
1.898
3.233
4.062
6.043
DeepClusterV2 StyleGAN2 PGGAN 2.599
MSG
MSG
SNGAN
StyleGAN2
4.149
4.586
5.395
7,830
8.612
MoCov2 StyleGAN2
MSG
PGGAN
MSG
SNGAN
StyleGAN2
0.019
0.030
0.030
0.048
0.067
0.083
DeepClusterV2 (Caron et al. 2020) another self-supervised model obtained by alternating between pseudo-labels generation via k-means clustering and training the network with classifi- cation loss supervised by these pseudo-labels. The model architecture is Resnet50;
MoCoV2 (Chen al. 2020b) is the state-of-the-art contrastive learning approach, which training objective enforces the closeness of representations produced for different augmentations of the same image while pushing apart the representations of unrelated images. The model architecture is Resnet50.
Three self-supervised models listed above outperform supervised pretraining on number transfer tasks (He al. 2020; Caron et al.) 2020), which implies that their embeddings capture more infor- mation relevant for these tasks, compared to supervised models pretrained on Imagenet. Below, for a large number of publicly available GANs, present the values of FID, Precision, and Recall met- rics computed with different embeddings. For the cases where the GANs ranking is inconsistent, we aim show that the ranking obtained with the self-supervised representations is more reasonable.
4

---

Published as a conference paper at ICLR 2021
Table 2: Prediction accuracy of CelebaHQ attributes from Inception and SwAV embeddings.
Model Inception V3
Mouth Slightly Open
No Beard
High Checkbones
Smiling
0.802
0.906
0,811 0.851
0.837
SwAV
0.868
0.938
0.893
3.1 FRÉCHET INCEPTION DISTANCE
The FID values for the non- non-Imagenet datasets computed with different embeddings are shown in The cases of inconsistent ranking with supervised Inception V3 and self-supervised SwAV embeddings are highlighted in color. The key observations are listed below:
Table [
(a) On CelebaHQ, SwAV ranks StyleGAN* higher, while Inception V3/Resnet50 prefer PGGAN. Figure [ shows random samples from both StyleGAN and PGGAN and clearly demonstrates the superiority of StyleGAN*. To investigate the reasons why SwAV produces a more adequate rank- ing compared to Inception/Resnet50, we perform two additional experiments. (I) First, we verify that SwAV embeddings capture more information relevant for face images. The Celeba dataset (Liu iet al. 2018) provides labels of 40 attributes for each image, describing various person properties (gender, age, hairstyle, etc.). For each attribute, we train 4-layer feedforward neural network with 2048 neurons on each layer with cross-entropy loss, which learns to predict the attribute from the SwAV/Incption embedding. For all attributes, the predictions from SwAV embeddings appear to be more accurate compared to Inception (several examples are given in Table 2). It confirms the intuition that Inception representations partially suppress the information about small facial details, which, however, is critical to identify more realistic images. (If) As qualitative experiment, we compare SwAV and supervised Resnet50 embeddings visually via recent technique described in Rombach et al. (2020). In nutshell, this technique reveals the invariances learned by the par- ticular representation model: for given image, visualizes several images having approximately the same embedding. By inspecting these images, one can analyze what factors of variations are not captured in the embedding (see the details in Section A.2). Two illustrative examples of such visu- alization for SwAV and Resnet50 are shown in Figure demonstrating that Resnet50 embeddings are more invariant to sensitive information, like gender or race, compared to SWAV. Such ignorance universal representa-
of sensitive information makes supervised embeddings less appealing tions. One of the key ingredients of the visualization method is an autoencoder, which is expected to capture all relevant information from an image. However, we argue that autoencoder representations are not well-suited for evaluating generative models and elaborate on this in detail in Section D
(b) On Bedroom, there are two inconsistencies in Inception and SWAV ranking. The first that SwAV ranks StyleGAN higher than PGGAN and the second is that SWAV ranks RPGAN higher than COCO-GAN. Figure 3 shows the samples from StyleGAN, PGGAN, RPGAN, and COCO- GAN and demonstrates that the ranking according to SWAV embeddings is more adequate. Namely, the quality of StyleGAN generatedd images is substantially higher. Also, it is difficult to identify favorite among RPGAN and COCO-GAN visually, while Inception embeddings claim strong superiority of the COCO-GAN model. On the other hand, self-supervised embeddings consider these models comparable, which is better aligned with human perception.
(c) There are also cases of the inconsistent ranking of MSG and PGGAN on Church, and StyleGAN and MSG CelebaHQ. But since the difference of the FID values are small for both Inception and SwAV, we do not consider it as a strong disagreement.
StyleGAN-CelebaHQ
PGGAN-CelebaHQ
Figure 1:
Samples generated by StyleGAN and PGGAN trained on CelebaHQ. The quality of images generated by StyleGAN* is substantially higher.
5

---

Published as a conference paper at ICLR 2021
Resnet50
SWAV
Resnet50
SWAV
Figure 2:
The reference image is top-left denoted by green frame, while 8 others constitute diverse sample of images that have approximately the same embeddings as the reference one.
(d) The rankings with the supervised Inception V 33/eenet550 embeddings are consistent with each other while being different from the rankings with the self-supervised ones. It indicates that the model architecture does not affect GAN ranking, and the training objective that matters.
(e) Imagener21k corrects some cases of misleading ranking with Inception V3, but not all of them. Namely, it correctly ranks StyleGAN and PGGAN on Bedroom while being wrong on CelebaHQ.
(f) SwAV and DeepClusterV2 have minimal inconsistencies in the ranking of MSG* vs PGGAN on CelebaHQ and StyleGAN2* vs SNGAN on Church, but the differences in the absolute values FID metric are negligible, so we consider these embedding models mostly consistent.
(g) MoCoV2 fixes some ranking mistakes with Inception V3,, but not of them. While it fixes the ranking of StyleGAN and PGGAN on Bedroom and reduces the gap between RPGAN and COCO- GAN, the ranking of PGGAN and StyleGAN on CelebaHQ still incorrect. Overall, the most reasonable rankings obtained using SwAV/DeepCluster which have significantly higher transfer performance compared to MoCoV2. In further experiments, we focus on the most transferable SwWVV/Deeplluster models.
Overall, self-supervised embeddings provide a more reasonable FID ranking across existing non- Imagenet benchmarks. For completeness, also report the FID values for the Imagenet dataset in 'able In this case, rankings with all embeddings are the same, which confirms that the SWAV representations can be used for Imagenet as well, while is not the main focus of our work.
3.2 PRECISION
The values of the Precision metric are reported in Table The main observations are listed below:
StyleGAN Bedrroom
PGGAN Bedroom
RPGAN Bedroom
COCO-GAN- Bedroom
Figure 3: Samples generated by StyleGAN PGGAN, RPGAN and COCO-GAN trained Bed- room. The quality images generated by StyleGAN* is substantially higher, while the quality of the images generated by RPGAN and COCO-GAN is approximately the same.

---

Published as a conference paper at ICLR 2021
Table 3: Precision (k=5) for different embedding models. The *** symbol indicates models with
truncation. Inconsistencies between Inception V3 and SwAV models are highlighted in color.
CELEBAHQ
FFHQ
MSG
StyleGAN
StyleGAN
MSG
PGGAN
MSG
StyleGAN2 StyleGAN
MSG
InceptionV3
0.894
0,882
0.803
0.7963
0.778
0.843
0.778
0.778
0.776
Resnet50 StyleGAN
MSG'
StyleGAN
MSG
PGGAN
MSG
StyleGAN2
StyleGAN
MSG
0.921
0,916
0.857
0.839
0.839
0.865
0.821
0.807
0.789
1magener21k k StyleGAN
MSG'
PGGAN
StyleGAN
MSG
StyleGAN2 0.883
MSG
StyleGAN
MSG
0,923
0.912
0.904
0.896
0.868
0.875
0,871
0.851
MSG
StyleGAN 0,928
MSG
StyleGAN 0.875
PGGAN
MSG
StyleGAN StyleGAN2 MSG 0.905
SwAV
0.935
0.879
0.851
0.911
0.889
0.870
MSG' 0.945
StyleGAN 0.937
StyleGAN 0.892
PGGAN 0.890
MSG
MSG 0.920
StyleGAN StyleGAN2 MSG
DeepClusterV2
0.887
0.908
0.882 0.873
LSUN-BEDROOM
StyleGAN StyleGAN 0,799
PGGAN COCO-GAN div. RPGAN
V3
0.649
0.541
0.443
0.092
0.086
Resnet50 StyleGAN StyleGAN 0,849
PGGAN
COCO-GANRPGAN div. RPGAN
0.722
0.646
0.596
0.545
0.515
StyleGAN COCO-GAA PGGAN
RPGAN div. RPGAN
Imagenet21K
0,723
0.653
0.622
0.434
0.171
0.114
SwAV StyleGAN StyleGAN 0.849
PGGAN COCO-GANRPGAN div. RPGAN
0.760
0.732
0.725
0.540
0.440
DeepClusterV2 StyleGAN StyleGAN 0.817
PGGAN 0.722
COCO-GANRPGAT div. RPGAN
0.728
0.706
0.540
0.423
LSUN-CHURCH
StyleGAN2 0,909
MSG'
MSG
PGGAN
StylGGANSSNGAR
Inception V3
0.837
0.698
0.692
0.689
0.193
Resnet50 StyleGAN2
MSG'
MSG
StyleGAN2
PGGAN
SNGAN
0.910
0.828
0.679
0.672
0.639
0.403
Imagenet21K StyleGAN2 StyleGAN2
MSG
MSG
PGGAN 0.399
SNGAN
0.567
0.492
0.458
0.448
0.183
SwAV StyleGAN2 0.929
MSG
StyleGAN2 0.619
PGGAN
MSG 0.491
SNGAN
0.647
0.533
0.465
DeepClusterV2 StyleGAN2 0.936
MSG
StyleGAN2
MSG
PGGAN
SNGAN
0.779
0.630
0.593
0.589
0.438
(a) As with FID, all supervised V3/Reset50 embeddings provide the same ranking, except minor differences between MSG with truncation and StyleGAN on CelebaHQ, and StyleGAN2 and PGGAN on Church. Self-supervised SwAV and DeepClusterV2 are also consistent except for the negligible difference in the ranking of PGGAN and MSG on CelebaHQ and Church;
(b) The most notable inconsistency between supervised and self-supervised embeddings is revealed on LSUN-Church, where Inception V3 considers MSG to be comparable to StyleGAN2, while SWAV ranks StyleGAN2 significantly higher. (I) To analyze which ranking of two GANs is more reasonable, we perform the following. On the synthetic data from the first GAN, we train classifier that aims distinguish between real and synthetic images. This classifier is then evaluated on the train a classifier to detect synthetic images
synthetic data from the second GAN. Concretely, real LSUN-Church and the images generated by MSG. Then we evaluate this model on hold-out real images and images produced by StyleGAN2. Intuitively, if a model was trained on high-quality synthetic samples, it will easily detect lower-quality ones. On the other hand, if the model learns to detect only low-quality synthetics, it will be harder to discriminate real images from the high-qualit ones. In this experiment, we employ a Resnet50 classifier with a binary cross-entropy loss. The re- sults for Church are provided in Table 4 meaning that the StyleGAN2 images of higher quality, therefore, the SwAV ranking is more reasonable. (II) We also conduct a human study to determine which of the generative models gives more realistic images in terms of human perception. For each generative model, we show ten assessors a real or randomly generated (fake) image and ask them to choose whether is real fake. The error rate reflects the visual quality of the generative model. For both models, MSG and StyleGAN2, we demonstrate to assessors 500 images, and the error rate 0.4% for MSG and 2. 2.8% for StyleGAN2, which clearly shows the superiority of StyleGAN2.
7

---

Published as a conference paper at ICLR 2021
Table Recall (k=5) for different GAN and embedding models. The ** symbol indicates models
with truncation. Inconsistencies between Inception V3 and SwAV models are highlighted in color.
CELEBAHQ
FFHQ
StGSeyAA
PGGAN
StyleGAN
MSG
Inception V3
0.477 0.405
0.398
0.281
0.276
0.632
0.538
0.474
0.339
PGGAN StyleGAN
MSG
StylGGAN2MGGI SIGGGGAN
Resnet50
0.502
0.419
0.330
0,284
0.277
0.624 0.547 0.456 0,299
Imagenet2 1k MyGGSAy
MSG
StyleGAN 0,315
PGGAN
0,546
0.414
0.321
0.287
0.688 0.551 0,452 0.335
SwAV MSGSStyleGAN
MSG
StyleGAN 0.035
PGGANI
StyleGAN2
MSGSSNGGAMG 0.399
0.151
0.061
0.055
0.027
0.095 0.193
0.063
MSG
MSG
StyleGANStyleGAN
SYGGANSSy SIGYEGGANNS 0.039
DeepClusterV2, 0.185
0.074
0.072
0.042
0.409
0.220 0.132 0.078
LSUN-BEDROOM
Inception StyleGAN 0.592
PGGAN
COCO-GAN StyleGAN
RPGANRPGAN div.
0.516
0.476
0.397
0.190
0.110
Resnet50 StyleGAN
PGGAN
StyleGAN
COCO-GANRPGANRGAAN div.
0.519
0.352
0.304
0.258
0.058
0.030
StyleGANStyleGAN
PGGAN 0.240
COCO-GANRPGANRPGAA div.
Imagener21k
0.498
0.293
0.157
0.008
0.003
SWAV StyleGANStyleCC 0.153
PGGAN 0.015
COCO.GARRFGANPGANN
0.069
0.003
0.0003
0.00003
DeepClusterV2 StyllGAN
PGGAN 0.039
COCOGGARPFANRFANN div.
0.210
0.109
0.009
0.001
0.0
LSUN-CHURCH
Inception StyleGAN2 0.519
MSG
PGGAN
SNGAN StyleGAN2
0.491
0.461 0.271
0.090
0.071
Resnet50 SGGeGAANPPGANN
MSG
MSG
SNGAN
StyleGAN2
0.460
0.413
0.406
0.182
0.038
0.021
Imagenet2 1k StyleGAN2
MSG
MGG StyleGAN2' 0.013
SNGAN
0.350
0.260
0.251
0.105
0.010
SwAV StyleGAN2
MSG
MSG' StyleGAN2 0.002
SNGAN
0.073
0.019
0.008
0.0014
0.0002
DeepClusterV2 StyleGAN2PGGAN
MSG MSG SNGAN StyleGAN2 0.002
0.112
0.058
0.031
0.009
0.002
Table 4:
The accuracy of fake images detection Church. The rows correspond GANs producing the train synthetics, while the columns correspond to GANs producing the test.
(c) Imagenet21k Ik ranks GANs less reliably com- pared SwAV. The most notable mistake ranking of COCO-GAN and PGGAN on Bed- room, where PGGAN produces more visually appealing images, see 3] Another case is comparison on CelebaHH,, where Imagenet2 ranks PGGAN higher than the more powerful MSG, see samples in Section
Train/Test
MSG
StyleGAN2
MSG
0.999
0.610
StyleGAN2 0.967
0.979
3.3 RECALL
The values of the Recall metric are shown in Table and the main observations are provided below:
(a) As in previous experiments, there are only minor inconsistencies between supervised Incep- t3 and Resnet50 models, namely, StyleGAN vs COCO-GAN on Bedroom and MSG vs PG- GAN on Church. The only insignificant difference between the self-supervised methods is the rank- ing of StyleGAN with truncation vs SNGAN on Church, however, Recall values for both models negligible. In terms of Recall, Imagenet2 Ik ranking always coincides with the ranking obtained by sulf-super methods, except for the negligible discrepancy between MSG and PGGAN on Church;
8

---

Published as a conference paper at ICLR 2021
(b) The absolute Recall values for SwAV/DeepClusterV2 V We attribute this behavior to the fact that GANs tend to simplify images omitting the details (Bau et al., 2019), e.g., people in front of buildings, cars, fences, etc. The classifier- pretrained embeddings are less sensitive to these details since they are not crucial for correct classi- fication. In contrast, self-supervised embeddings are more susceptible small details (see Figure ] and Table hence, more images are considered "not covered". Figure 6 Jection[Csshwwe exam- ples of real LSUN-Church images that are "definitely covered" by StyleGAN2 from the standpoint of Inception V3 embeddings, but are "not covered" if SWAV embeddings are used. More formally, we say that real image covered by a synthetic one with the neighborhood size k if the distance between their embeddings does not exceed the distance from the embedding of the synthetic image to its k-th nearest neighbor in the set of all synthetic embeddings. The images from Figure 6 are covered by at least 10 synthetic images with neighborhood size 5 with Inception embeddings, while being not covered even by the neighborhood of size 100 for SWAV embeddings. These im-
smaller compared to Incep-
ages possess many small details, such monuments, cars, people, branches in the foreground, and so on, that GANs usually omit to generate.
2.-
SwAV
(c) Interestingly, the Precision values for SwAV are quite high, therefore, SwAV considers the state- of-the-art generative models being able to produce high-fidelity images, but failing to generate diverse images with small details.
Inceptiani3
18-
(d) There are two significant inconsistencies in rank- ings based on Inception and SwAV. First, on CelebaHQ. SwAV prefers MSG* over PGGAN. Sec- ond, on LSUN-Bedroom, Inception ranks PG- GAN higher than StyleGAN. Since StyleGAN is a more powerful model compared to PGGAN, we also consider it as case of more reliable ranking with SwAV, even though is difficult to confirm quan- titatively due to the lack of an "oracle" measure of generation diversity.
12-
1.0
0
20000
40000
60000
80000
100000
Figure 4: FID values for different sample sizes for StyleGAN2 on Church. Since FID values for SwAV and Inception have dif- ferent scales, they normalized by the FID value computed for sample of size 100k.
3.4 SAMPLE EFFICIENCY OF SWAV-BASED FID
As an additional practical advantage of SwAV, we highlight that computing FID becomes more sample-efficient compared to I33eptionV3. Namely, to obtain a reliable estimation of FID values, one requires much fewer samples when using SWAV embeddings. We illustrate this effect in Fig- 4 which plots FID values w.r.. sample size for StyleGAN2 trained on Church. Since the FID values for SWAV and Inception have different typical scales, we normalize both curves by the corresponding FID value computed for a sample of size 100k. FID based on SwAV embeddings converges faster, i.e., using SwAV always achieves more reliable FID estimates for fixed sample size.
We attribute this benefit of SwAV to the fact that its representations capture more information needed to distinguish between real and fake distributions. Intuitively, the covariance matrices for real and synthetic data computed from SwAV embeddings are more dissimilar compared to Inception based ones. Quantitatively, the magnitude of the covariance term in FID tr(CR + Cs 2VCRCS) larger for SwAV, which leads to smaller relative errors of its stochastic estimates. We elaborate on this issue more rigorously in Section E
4 CONCLUSION
In this paper, we have investigated if the state-of-the-ar self-supervised models can produce more appropriate representations for GAN evaluation. With extensive experiments, we have shown that using these representations often corrects the cases of misleading ranking obtained with cenifficatio--rrerrinee embeddings. Overall, self-supervised representations provide a more ad- equate GAN comparison the four established non-Imagenet benchmarks of natural images. Of course, we do not claim that they should be used universally for all areas, e.g., for spectrograms or medical images. But our work indicates that obtaining good representations needed for proper GAN evaluation does not require supervision, therefore, domain-specifer self-supervised learning becomes promising direction for further study.

---

Published as a conference paper at ICLR 2021
5 ACKNOWLEDGEMENTS
We thank the Anonymous Reviewers for their reviews. We also thank Xun Huang for commenting on his experience with SwAV on OpenReview.
REFERENCES
Shane Barratt and Rishi Sharma. note on the inception score. arXiv preprint arXiv:1801.0197 2018.
David Bau, Jun-Yan Zhu, Jonas Wulff, William Peebles, Hendrik Strobelt, Bolei Zhou, and An- tonio Torralba. Seeing what a gan cannot generate. In Proceedings of the IEEE International Conference Computer Vision, pp. 4502-4511, 2019.
Ali Borji. Pros and cons of gan evaluation measures. Computer Vision and Image Understanding, 179:41-65, 2019.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference Learning Representations, 2019. URL
net/ http://ooommmdd1lfs000000
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint wrXiv:2006.09882, 2020.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint wxXiv:2022.05709 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04220 2020b.
Giannis Daras, Augustus Odena, Han Zhang, and Alexandros G Dimakis. Your local gan: Design- ing two dimensional local attention mechanisms for generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14531-14539, 2020.
Deng, Dong, R. Socher, L.-J. Li, Li, and L.. Fei-Fei. ImageNet: Large-Scale Hierarchical Image Database. In CVPR09, 2009.
D.C. Dowson and B.V. Landau. The fréchet distance between multivariate normal distributions. Journal of multivariate analysis, 12(3):450-455, 1982.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. In Advances in Neural Information Processing Systems, pp.
2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in Neural Information Processing Systems, pp.
2017.
Animesh Karnewar and Oliver Wang. Msg-gan: Multi-scale gradients for generative adversarial networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recog- nition, pp. 7799-7808, 2020.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint 1700 2017.
10

---

Published as a conference paper at ICLR 2021
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4401-4410, 2019a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. arXiv preprint urXiv:1912.04958, 2019b.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. arXiv preprint 1rXiv:1912.1137,, 2019.
Tuomas Kynkäänniemi. Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In Advances in Neural Information Processing Systems, pp. 3929-3938, 2019.
Chang, Yu- lu-Sheng Chen, Da-Cheng Juan, Wei Wei, and Hwann-Tzong Chieh Hubert Lin, generation by parts via conditional coordinating. In IEEE International Chen. COCO-GAN:
Conference on Computer Vision (ICCV), 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August, 15:2018, 2018.
Mario Lucic, Michael Tschannen, Marvin Ritter, Xiaohua Zhai, Olivier Bachem, and Sylvain Gelly. High-fidelity image generation with fewer labels. arXiv preprint vrXiv:1903.02277 2019.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018. URL httpttttoppeeereiieeeeeeeo net www.forrumiioooomom d=B QRgziT-
Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. arXiv preprint (XXi:20022.0979. 2020.
Stanislav Pidhorskyi, Donald A Adjeroh, and Gianfranco Doretto. Adversarial latent autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 14104-14113, 2020.
Robin Rombach, Patrick Esser, and Björn Ommer. Making sense cnns: Interpreting deep repre- sentations & their invariances with inns. arXiv preprint arXiv:2008.01777, 2020.
Mihaela Rosca, Balaji Lakshminarayanan, David Warde-Farley, and Shakir Mohamed. Variational approaches for auto-encoding generative adversarial networks. arXiv preprint arXiv:1706.0499 049877 2017.
Mehdi SM Sajjadi, Olivier Bachem, Mario Lucie, Olivier Bousquet, and Sylvain Gelly. Assess- ing generative models via precision and recall. In Advances Neural Information Processing Systems, pp. 5228-52377 2018.
Tim Salimans. Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp.
2016.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826. 2016.
Andrey Voynov and Artem Babenko. Rpgan: Gans interpretability via random routing, arXiv preprint arXiv:1912.10922 2019.
Qiantong Xu, Gao Huang, Yang Yuan, Chuan Guo, Yu Sun, Felix Wu, and Kilian Weinberger. An empirical study on evaluation metrics of generative adversarial networks. arXiv preprint 2018.
Fisher Yu, Ari Seff. Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: /506. 003600 2015.
11

---

Published as a conference paper at ICLR 2021
Sharon Zhou, Mitchell Gordon, Ranjay Krishna, Austin Narcomey, Li F Fei-Fei, and Michael Bemn- stein. Hype: A benchmark for human eye perceptual evaluation of generative models. In Advances in Neural Information Processing Systems, pp. 3449-3461, 2019.
A APPENDIX
A.1 IMAGENET
Table 6: FID values for GANs on Imagenet.
IMAGNNT
Inception BigGAAN-depp
BigGAN S3GAN YLG
6.923
7.659
10.85 20.64
Resnet50 BigGAN-deep BigGAN S3GAN YLG
11.22
19.50
20.05 34.11
Imagenet21k BigGAN-deep BigGAN S3GAN YLG
334.8
601.5
1202. 1697
SWAV BigGAN deep BigGAN S3GAN YLG 4.162
2.374
4.282 5.965
DeepClusterV2 BigGAN deep BigGAN S3GAN YLG
3.937
6.240
6,555 9.101
A.2 THE DETAILS OF THE VISUALIZATION TECHNIQUE IN SECTION
The technique from Rombach et al. (020) proposes to construct the embedding visualization in the following way. Let have autoencoder A(x), whose latent representations are denoted by z. It is assumed that a latent representation ==A(r) captures all the information from the image I, since the autoencoder's goal to fully reconstruct from 2. Let us denote the embedding model we want to study by E(x). For a particular image Rombach et al. (2020) compute an autoencoder representation
A(z) and aims to disentangle the information contained in the autoencoder representation z into the information contained in the image embedding e F(x) and the information e to which the embedding model E(x) is invariant. To this end, an invertible neural network (INN) is trained, which predicts the invariant part e INN e) based representation and embedding. Moreover, is trained such a way that the invariant part has normal distribution
the autoencoder
1). Finally, bunch of images having the same but different invariant parts v are produced. Namely, several samples invariant parts & are produced from (O 1) and the corresponding latent representations are computed from 7 and embedding e, that is S = INN.reverse(w.ee) Then 2 can be decoded into an image using the autoencoder. As a result, for given embedding model Fi(x:) and given image æ, it allows obtain set of images with approximately the same embeddings but varying invariant part. For the CelebaHQ dataset, we use ALAE autoencoder (Pidhorskyi et al. 2020 and train INN with the hyperparameters provided in
Rombach et al. 2020).
B CELEBAHQ SAMPLES
MSG-CelebaHQ
PGGAN-CelebaHQ
OC
Figure 5: Random sample of images generated by MSG without truncation and PGGAN trained on CelebaHQ dataset. One that the quality of images generated by MSG is substantially higher.
12

---

Published as a conference paper at ICLR 2021
C
CHURCH IMAGES
Figure 6: Examples of real images that confidently covered by StyleGANv2 in terms of Incep-
tionV3 embeddings, but not covered in terms of SwAV.
D AUTOENCODER EMBEDDINGS
Query
SWAV
ALAE
PS
IIT
e
Figure 7: Examples of nearest neighbors in terms of SwAV and ALAE representations.
In this section, we compare SwAV and ALAE autoencoder (Pichorskyi et al.) 2020) embeddings. ALAE was trained on Celeba (Liu et al. 2018) dataset, therefore, is expected to work correctly for the ranking of generative models CelebaHQ, To investigate what information is important for each of the embeddings. we build dataset containing 30k real images from CelebaHQ and 30k synthetic images generated by PGGAN. Then select 1.5k real and 1.5k synthetic images as queries and leave the remaining 57k pictures as a database. For each query, we compute three nearest neighbors from the database. Typical examples of the nearest neighbors shown Figure 7
13

---

Published as a conference paper at ICLR 2021
One can that ALAE places strong emphasis the exact spatial arrangement, while sparsely sampled manifolds rarely include near-exact matches in terms of spatial structure (Kynkäänniemi et al.) 2019). This makes ALAE embeddings poor for GAN evaluation, for instance, via Precision and Recall metrics. We also perform real/fake classification of queries using 1-NN classification on the constructed datahase. The classification accuracy is 0.729 for ALAE embeddings and 0.839 for SwAV. Overall, the distance in the space of the autoencoder's embeddings is less informative to distinguish between the real/fake distributions.
E ON THE ESTIMATE OF FID VALUES
Let us denote CR the covariance matrix for real data, Cs for synthetic data and ÜR and Čs their estimates with finite number of samples. We also denote
d(CR. Cs) = tr + Cs -VS5-2CCCCS
(1)
In (Dowson Landau 1982) it was shown that d(CR,, Cs) defines a metric on the space of all covariance matrices of order Our goal is assess the relative error of the FID estimate. Under the assumption that the means of real and synthetic data distributions are estimated quite accurately (which is the case already for few thousand of samples) we need to estimate only
d(CRR, Cs) (ČRR Os)
(2)
d(CR, Cs)
Due to the metric properties
(Cn, Cs) s sd (Cnn.CC) + d dCCn, Cs) + (Cs,Cs)
(3)
Then
d Cn) -d(Cr, Cs) d cd d(Cn, Cs)
d(Cn, Cs)
Due to the symmetry the same inequality can be obtained with the opposite sign and as the result we get
d (Ōn. (
-d(CR. Cs) I (Cnz.Ci) Cn) + +d(Cs.Css
(5)
((CR.Cs)
d(CR, Cs)
where the numerator corresponds to the accuracy of the estimation of covariance matrices and the denominator corresponds to the distance between covariance matrices for real and synthetic data. Thus, larger values of d(Cn, Cs)) result in smaller relative error of the FID estimation. Experi- mentally, we compute the distances d(CR, Cs) tr(CR Cs 2 CvCCCs) based on SwAV and Inception V3 embeddings for StyleGAN2 trained on the Church dataset We obtain 0.083 for SWAV and 0.028 for Inception which confirms the validity of calculations above and explains the better sample-efficiency of SWAV presented in Figure
F HUMAN EVALUATION
As an additional evidence that inter-image distances induced by SwAV are better aligned with human perception compared Inception V3, we perform two crowdsourcing experiments. All the data and labellings are released the GitHub
SwAV-based Precision and Recall have higher agreement with human judgements. The key step of computing Recall is checking, if for given real embedding there exists generated em- bedding that is closer to r than its k-th real neighbor. To verify, a particular embedding agrees well with human perception, perform the following procedure. We form triplet of an anchor
Since the absolute values d(Cn, Cs) depend on the scale SwAV/Inception?? activations, we nor- malize them by the geometric mean of norms CR and Cs. wwwppriised-geeeee
https:: // thub. com/ stani m-rrrooov//
14

---

Published as a conference paper at ICLR 2021
CELEBAHQ, StyleGAN
CELEBAHQ, PGGAN
Precision
Recall
Precision
Recall
Inception V3
0.396
0.497
0.399
0.482
SWAV
0.418
0.525
0.482
0.479
LSUN-Bedroom, StyleGAN
LSUN-Bedroom, PGGAN
Precision
Recall
Precision
Recall
Inception
0.417
0.522
0.375
0.444
SWAV
0.450
0.530
0.446
0.429
LSUN-Church, MSG
LSUN-Church, PGGAN
Precision
Recall
Precision
Recall
Inception V3
0.386
0.473
0.400
0.449
SwAV
0.477
0.497
0.445
0.458
Table 7: Human evaluation of the precision and recall agreement with different embeddings (higher is better).
real image lanchor that contributes to the Recall value, its 5-th closest neighbor among the real im- ages 15th and the generated image /gea that appears to be closer to Jacchor in terms of the considered embedding. A human assessor is then asked to choose an image between Igen and Isth that is more similar to anchorr Once the assessor chooses the generated one, we consider it as case of agreement with the embedding. The embeddings with higher agreement rate are more suitable for computing Recall.
For Precision, we similarly form the triplets consisting of generated image, its 5-th neighbor among the generated images and real image [reat closer to it lanchor, 1eh Leeat Once assessor answers that reat is more similar to anchor then 15th. we consider this is as an agreement with the embedding.
Here we always use the same real and generated samples for the evaluation of the metrics in Section 3 We label three datasets with two GAN models for each. For each pair of a datasct and generator, we label 200 different triplets, each by ten different assessors. An assessor is also able to choose the options "'qually similar" or "both completely dissimilar". Once the "equally similar" chosen, suppose that the agreement happens with the probability 0.5. The user interface is illustrated on Figure (left). All the labeling was performed in Yandex Toloka [ The results are presented on Table and confirm that SwAV emebddings mostly have higher agreement with human perception.
Quality of neighbors. As more simple experiment, we also ask human assessors compare quality of top-5 neighbors produced by Inception and SwAV embeddings. Namely, we take a set of N real images Z, same as Section 3 For given real image r E we form two lists of its 5 nearest neighbors Bivs C II and BS C I based on Inception V3 and SwAV embeddings. An assessor is asked to assign either Biv3 or BSwAV. Same above, the assessor may also label it "equal" which is treated as an equal probability of each set to be chosen. The user interface illustrated on Figure g (right). For each dataset we form 500 different triplets r, BswAv, Biva, each labeled by ten different assessors. Table Breseetsseesettss based set of neighbors, indicating that SwAV-induced distances better capture perceptual similarity compared to InceptionVV.
presents probabilities that assessors prefer SwAV-
InceptionV3
SwAV
CELEBAHQ
0.37
0.63
-edrrom
0.39
0.61
LSUN-Chure
0.41
0.59
Table 8: Comparison of 5 neighbor lists quality based on SwAVVInception representations.
https://toloka.at
15

---

Ecally servie Published as a conference paper at ICLR 2021
Wiech image moce simelar to upper one?
Imagee which are more siriler top image?
both can
Aigh
Figure 8: User interface for precision / recall agreement labeling (left) and top-5 neighbours labeling (right).
model
source
truncation
CELEBAHO
StyleGAN
StyleGAN MSG
0.7
MSG
htte://fttt...ccm/aarrrrrrprgrrp 0.6
PGGAN
of gans
FFHQ
StyleGAN2 StyleGAN
pytorch
MSG
MSG*
0.6
LSUN-Bedroom
StyleGAN
StyleGAN
httrss//ghhhbbbbcom/////a///222f
0.7
PGGAN
of_gans
RPGAN
model: 2eneeatt::_1smmm22221-1
RPGAN-div
https httes://htthb..cc///oonno////oor
model: nude://geccaaeello.......eeeeeer
COCO-GAN
https: htteae///ttttauooo////orr//////2
model: 2UMNN__2266256_222222222
LSUN-Church
StyleGAN2
StyleGAN2
0.5
PGGAN
httea///tttbbbccn//aarrrgggggeee http:///ooooesiiieeeeoooooooggo growing of gans
MSG
MSGs
SNGAN
Submission authors implementation
Table 9: The URLs with GAN checkpoints and truncation levels used in experiments.
16

---

Published as a conference paper at ICLR 2021
Representation
Dim
URL
Checkpoint
InceptionV3
2048
https:
https://gittub..cmm
com/ con/mseitzer/
fieiteez/pytorch-fia/
pytorch-fid
teleases/cownicad/ffi
welghhss/p__Ineepcion-00033
12-05-6726625d.p.hh
Resnet-50
2048
https://pytorch.
attps:/boonllsslppttorrc..
org/docs//table/
-
Eorchvision/
Isc8e357.pta
models.atml
Imagenet21k Ik
2048
https://tfhub.
http:://tfhab..der/ggglgs
dev tev/google/bit/m
bit/n-r50x1/1
r50x1/1
SWAV
2048
https://
https://da..baaipubllicffff
github.com
comp/eeeocluster//vaav..o _00epp___0_p______
facebookresearch/
pretrain.pth.tar
Swav
DeepClusterV2
2048
https://
2ttps///ll.hbojpullccfeeee
github.com/
com/deepcluster/
facebookresearch/
ddepcllsster2__80ep
Swav
pret.ran..pth.te
MoCoV2
2048
https://
attps///ll.fhaipublicifeee.
github.com
facebookresearch/
moco_vv__00ee//moco_v2 800ep_rretraln.phhhtar
moco
Table 10: The URLs with checkpoints used in our experiments.
17


Reviews:
[{"review_id": "sHFHhpP26tO", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to use image representations from trained self-supervised models to evaluate GANs more accurately. Compared to the currently used representations from supervised-pretrained models e.g. InceptionV3, the authors claim, that such embeddings suppress information not critical for the classification process which, however, are assumed to be crucial for assessing the full distributions of real and generated images. The authors use 5 datasets and their respective representations from 5 models, 3 supervised and 2 self-supervised, to show that representations from self-supervised models lead to better GAN evaluations. The representations were used to evaluate 6 GAN models with 3 metrics, namely FID, Precision and Recall. A ranking of the GAN models shows inconsistencies between supervised and self-supervised based representations. By visual inspection, prediction accuracy tests, and a comparison of representation invariances the authors show that rankings via self-supervised embeddings are more plausible.\n\nPros: Interesting proposal to better evaluate GANs and generative models for image data. The paper is well written and easy to understand. The experiments are extensive and support the claim of the authors. Testing for invariances of representations is an interesting idea and the results support the use of embeddings from self-supervised models.  \n\nCons: The authors argue that latent representations from an autoencoder capture all the information from images. It would be interesting to see how such representations, e.g. from the autoencoder used to show the invariances described in section A.1, behave compared to the proposed self-supervised representations. I would like to see them to be included in the experiments.\n\nMinor comment: Typo in A.1: corrseponding\n\nEdit: The authors have not responded to any of the reviews, i lower my rating to 4 \n\nEdit2: Oh there was a misunderstanding, i probably was not logged in and didn't see any comments and reviews. I raised the rating and will read the answers and will rate again.\n\nEdit3: After reading the rebuttals, i raise my rating to 7 ", "extra_scores": {}, "extra_text": {}}, {"review_id": "eeXz27JhyR", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nThe papers looks at the problem of evaluating GAN samples. Current methods, such as FID/PR with Inception v3 are problematic because they generally depend on using the features of a model discriminatively trained on (a super set of) ImageNet. The authors show that these type of models ignore details that are meaningful when for example comparing results on CelebaHQ.\n\nInstead the authors propose to use a recent self-supervised trained model which have been shown to provide more general representations. They take a selection of recent powerful GANs, and compare the ranking of their results based on FID/PR with discriminative imagine features vs self-supervised features and show that there are indeed differences.\n\nTo evaluate the ground truth, the authors device a number of small experiments that attempt to establish these facts: retrieving celebA labels from features of each model and an additional classifier trained on one GAN output evaluated on another. In all experiments the authors show that the self-supervised trained model produces a GAN ranking that is closer to the truth.\n\n## Review\n\nThe paper is well written and provides a very nice overview of recent advances in GANs and description of their evaluation methods. While the proposed method is a simple improvement over previous work (replace the feature extractor, keeping most else constant), the empirical evaluation is very thorough and well done.\n\nIn particular I found the additional experiments based on the results in table 1 and 3 very informative and welcome. Results in table 2 and 4 give a very interesting confirmation that self-supervised embeddings are indeed more informative.\n\nThe visualisations are well done and relevant, and the markings in the table make finding comparisons straightforward. \n\n## Discussion \n\nWhile the ordering between SwAV and DeepClusterV2 is the same, the actual numbers are significantly different, do the authors know why this might happen?\n\nThe authors compare to two specific self-supervised algorithms, which are additionally trained using clustering, is there a particular reason those models were chosen? Does the type of contrastive learning impact the results, or is generally better representations (as measured by fine-tuning for imagenet) better for GAN evaluation too? Would it be worthwhile to attempt to add the types of artifacts in GANs to the set of augmentations done for the contrastive learning?\n\nIn general, I think the paper can benefit from more analysis around the choice for the right self-supervised network and trade offs.\n\n## Post answer\n\nMy questions are appropriately answered and I appreciate the addition of section 3.4. I think my current score accurately reflects my evaluation of the paper, with the remaining concern being the magnitude of the contribution.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Q0Y-LkwmDL0", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview of paper: this work compares supervised feature extractors vs. two types of self-supervised feature extractors for the task of GAN model evaluation. It shows that the ranking provided by self-supervised features is different from that of supervised features, and claims it corresponds better with human judgement. Experiments are conducted of multiple large GANs and datasets.\n\nNovelty: I am not aware of previous works investigating self-supervised features for GAN evaluation, but note that [1] evaluated self-supervised features as a perceptual loss which is highly related.  \n\nSignificance: the current metrics used to evaluate GANs are well-known to be problematic and the search for better measures is important. On the other hand, I am not certain that this paper is conclusive enough be able to shift the community towards different metrics. This is a hard thing to do as even changing the evaluation from using InceptionV3 features (which are fairly outdated) to more modern ResNets has not happened yet.\n\nMethodology: I have a few issues with the method - although the authors claim to have better agreement with human and objective judgements, this is not extremely well justified. E.g. showing that Swav can classify facial attributes between InceptionV3 is not by itself very indicative - it uses a better architecture and more generally transfer learning of self-supervise vs. supervised methods was extensively validated in the original papers (and is about equal). Also, supervised ImageNet features are particularly poor for faces, I guess that for other objects types results might be different. The evidence for the \"groundtruth\" ranking for precision and recall is not particularly strong.\n\nEvaluation:  + many gans and datasets evaluated - why only SWAV and DeepClusters, why not use the other popular contrastive learning methods e.g. MoCo, SimCLR, BYOL? \n\nOverall: the investigation of better ways of evaluating GANs is important. The main criticism is that (in my opinion) not enough effort was taken to establish the groundtruth ranking between models, making the results of this investigation less significant.\n\n########################################################################################\n\nThe rebuttal addressed my concerns - I increased the score\n\n[1] Zhang et al. ,The unreasonable effectiveness of deep features as a perceptual metric, CVPR'18", "extra_scores": {}, "extra_text": {}}, {"review_id": "NAuWcCxCfTK", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides an interesting empirical study of self-supervised image embeddings as an alternative to pre-trained ImageNet classification models, for the purpose of evaluating the quality and diversity of GAN generative image models. I always found it a little odd that a model trained on ImageNet with cross-entropy loss should somehow be a magical universal quality metric, and I am happy to see that this paper provides good evidence that this is not the case. The authors select 2 self-supervised models and compare them against a number of supervised models. The metrics used are FID and Precision/Recall. I am curious why Inception Score was not also compared? \n\nThe paper does quite a thorough job of selecting and comparing models, by normalizing for architecture and changing dataset or loss function. It shows clearly that self-supervised methods outperform the supervised methods for ranking various GAN models.\n\nIt would have been interesting to train the self-supervised model on the dataset itself e.g. LSUN or CelebA to see whether that provides an even more useful signal. Given that deep networks find it hard to generalize across datasets, I would expect that directly training an embedding on the target dataset would do better. Did the authors try something along these lines?\n\nA minor comment is that the layout of the results and comments is a bit confusing: due to the very long number of points that refer to a particular figure and needing lots of scrolling back and forth. Some better way to organize the information and comments would be appreciated.\n\nI would also find it insightful to better understand *why* self-supervision works better for evaluating representations? Any comments to this regard would be interesting. Lastly, I am curious why the authors did not consider self-supervised methods such as SimCLR?\n\nI have read the rebuttals and other comments and maintain my rating of the paper. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All four reviewers unanimously recommended for an acceptance (four 7s). They generally appreciated that the proposed idea is novel and experiments are convincing. I think the paper tackles an important problem of evaluating GANs, and the idea of using self-supervised representations, as opposed to the conventional ImageNet-based representations, would lead to interesting discussions and follow-ups. 
Decision: Accept (Spotlight)