Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper proposes to use image representations from trained self-supervised models to evaluate GANs more accurately. Compared to the currently used representations from supervised-pretrained models e.g. InceptionV3, the authors claim, that such embeddings suppress information not critical for the classification process which, however, are assumed to be crucial for assessing the full distributions of real and generated images. The authors use 5 datasets and their respective representations from 5 models, 3 supervised and 2 self-supervised, to show that representations from self-supervised models lead to better GAN evaluations. The representations were used to evaluate 6 GAN models with 3 metrics, namely FID, Precision and Recall. A ranking of the GAN models shows inconsistencies between supervised and self-supervised based representations. By visual inspection, prediction accuracy tests, and a comparison of representation invariances the authors show that rankings via self-supervised embeddings are more plausible.

Pros: Interesting proposal to better evaluate GANs and generative models for image data. The paper is well written and easy to understand. The experiments are extensive and support the claim of the authors. Testing for invariances of representations is an interesting idea and the results support the use of embeddings from self-supervised models.  

Cons: The authors argue that latent representations from an autoencoder capture all the information from images. It would be interesting to see how such representations, e.g. from the autoencoder used to show the invariances described in section A.1, behave compared to the proposed self-supervised representations. I would like to see them to be included in the experiments.

Minor comment: Typo in A.1: corrseponding

Edit: The authors have not responded to any of the reviews, i lower my rating to 4 

Edit2: Oh there was a misunderstanding, i probably was not logged in and didn't see any comments and reviews. I raised the rating and will read the answers and will rate again.

Edit3: After reading the rebuttals, i raise my rating to 7 
review_title: Interesting proposal with some missing experiments 

final_score: rating: 7: Good paper, accept
review_content: ## Summary

The papers looks at the problem of evaluating GAN samples. Current methods, such as FID/PR with Inception v3 are problematic because they generally depend on using the features of a model discriminatively trained on (a super set of) ImageNet. The authors show that these type of models ignore details that are meaningful when for example comparing results on CelebaHQ.

Instead the authors propose to use a recent self-supervised trained model which have been shown to provide more general representations. They take a selection of recent powerful GANs, and compare the ranking of their results based on FID/PR with discriminative imagine features vs self-supervised features and show that there are indeed differences.

To evaluate the ground truth, the authors device a number of small experiments that attempt to establish these facts: retrieving celebA labels from features of each model and an additional classifier trained on one GAN output evaluated on another. In all experiments the authors show that the self-supervised trained model produces a GAN ranking that is closer to the truth.

## Review

The paper is well written and provides a very nice overview of recent advances in GANs and description of their evaluation methods. While the proposed method is a simple improvement over previous work (replace the feature extractor, keeping most else constant), the empirical evaluation is very thorough and well done.

In particular I found the additional experiments based on the results in table 1 and 3 very informative and welcome. Results in table 2 and 4 give a very interesting confirmation that self-supervised embeddings are indeed more informative.

The visualisations are well done and relevant, and the markings in the table make finding comparisons straightforward. 

## Discussion 

While the ordering between SwAV and DeepClusterV2 is the same, the actual numbers are significantly different, do the authors know why this might happen?

The authors compare to two specific self-supervised algorithms, which are additionally trained using clustering, is there a particular reason those models were chosen? Does the type of contrastive learning impact the results, or is generally better representations (as measured by fine-tuning for imagenet) better for GAN evaluation too? Would it be worthwhile to attempt to add the types of artifacts in GANs to the set of augmentations done for the contrastive learning?

In general, I think the paper can benefit from more analysis around the choice for the right self-supervised network and trade offs.

## Post answer

My questions are appropriately answered and I appreciate the addition of section 3.4. I think my current score accurately reflects my evaluation of the paper, with the remaining concern being the magnitude of the contribution.
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: This paper provides an interesting empirical study of self-supervised image embeddings as an alternative to pre-trained ImageNet classification models, for the purpose of evaluating the quality and diversity of GAN generative image models. I always found it a little odd that a model trained on ImageNet with cross-entropy loss should somehow be a magical universal quality metric, and I am happy to see that this paper provides good evidence that this is not the case. The authors select 2 self-supervised models and compare them against a number of supervised models. The metrics used are FID and Precision/Recall. I am curious why Inception Score was not also compared? 

The paper does quite a thorough job of selecting and comparing models, by normalizing for architecture and changing dataset or loss function. It shows clearly that self-supervised methods outperform the supervised methods for ranking various GAN models.

It would have been interesting to train the self-supervised model on the dataset itself e.g. LSUN or CelebA to see whether that provides an even more useful signal. Given that deep networks find it hard to generalize across datasets, I would expect that directly training an embedding on the target dataset would do better. Did the authors try something along these lines?

A minor comment is that the layout of the results and comments is a bit confusing: due to the very long number of points that refer to a particular figure and needing lots of scrolling back and forth. Some better way to organize the information and comments would be appreciated.

I would also find it insightful to better understand *why* self-supervision works better for evaluating representations? Any comments to this regard would be interesting. Lastly, I am curious why the authors did not consider self-supervised methods such as SimCLR?

I have read the rebuttals and other comments and maintain my rating of the paper. 

review_title: Interesting empirical evaluation

final_score: rating: 7: Good paper, accept
review_content: Overview of paper: this work compares supervised feature extractors vs. two types of self-supervised feature extractors for the task of GAN model evaluation. It shows that the ranking provided by self-supervised features is different from that of supervised features, and claims it corresponds better with human judgement. Experiments are conducted of multiple large GANs and datasets.

Novelty: I am not aware of previous works investigating self-supervised features for GAN evaluation, but note that [1] evaluated self-supervised features as a perceptual loss which is highly related.  

Significance: the current metrics used to evaluate GANs are well-known to be problematic and the search for better measures is important. On the other hand, I am not certain that this paper is conclusive enough be able to shift the community towards different metrics. This is a hard thing to do as even changing the evaluation from using InceptionV3 features (which are fairly outdated) to more modern ResNets has not happened yet.

Methodology: I have a few issues with the method - although the authors claim to have better agreement with human and objective judgements, this is not extremely well justified. E.g. showing that Swav can classify facial attributes between InceptionV3 is not by itself very indicative - it uses a better architecture and more generally transfer learning of self-supervise vs. supervised methods was extensively validated in the original papers (and is about equal). Also, supervised ImageNet features are particularly poor for faces, I guess that for other objects types results might be different. The evidence for the "groundtruth" ranking for precision and recall is not particularly strong.

Evaluation:  + many gans and datasets evaluated - why only SWAV and DeepClusters, why not use the other popular contrastive learning methods e.g. MoCo, SimCLR, BYOL? 

Overall: the investigation of better ways of evaluating GANs is important. The main criticism is that (in my opinion) not enough effort was taken to establish the groundtruth ranking between models, making the results of this investigation less significant.

########################################################################################

The rebuttal addressed my concerns - I increased the score

[1] Zhang et al. ,The unreasonable effectiveness of deep features as a perceptual metric, CVPR'18
review_title: Review

Meta-review: All four reviewers unanimously recommended for an acceptance (four 7s). They generally appreciated that the proposed idea is novel and experiments are convincing. I think the paper tackles an important problem of evaluating GANs, and the idea of using self-supervised representations, as opposed to the conventional ImageNet-based representations, would lead to interesting discussions and follow-ups. 
Decision: Accept (Spotlight)