Venue: ICLR 2018 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: In the paper, the authors discuss several GAN evaluation metrics.
Specifically, the authors pointed out some desirable properties that GANS evaluation metrics should satisfy.
For those properties raised, the authors experimentally evaluated whether existing metrics satisfy those properties or not.
Section 4 summarizes the results, which concluded that the Kernel MMD and 1-NN classifier in the feature space are so far recommended metrics to be used.

I think this paper tackles an interesting and important problem, what metrics are preferred for evaluating GANs.
In particular, the authors showed that Inception Score, which is one of the most popular metric, is actually not preferred for several reasons.
The result, comparing data distributions and the distribution of the generator would be the preferred choice (that can be attained by Kernel MMD and 1-NN classifier), seems to be reasonable.
This would not be a surprising result as the ultimate goal of GAN is mimicking the data distribution.
However, the result is supported by exhaustive experiments making the result highly convincing.

Overall, I think this paper is worthy for acceptance as several GAN methods are proposed and good evaluation metrics are needed for further improvements of the research field.

review_title: A good survey of GAN evaluation metrics with exhaustive experimental evaluations.

final_score: rating: 7: Good paper, accept
review_content: Thanks for an interesting paper. 

The paper evaluates popular GAN evaluation metrics to better understand their properties. The "novelty" of this paper is a bit hard to assess. However, I found their empirical evaluation and experimental observations to be very interesting. If the authors release their code as promised, the off-the-shelf tool would be a very valuable contribution to the GAN community. 

In addition to existing metrics, it would be useful to add Frechet Inception Distance (FID) and Multi-scale structural similarity (MS-SSIM). 

Have you considered approximations to Wasserstein distance? E.g. Danihelka et al proposed using an independent Wasserstein critic to evaluate GANs: 
Comparison of Maximum Likelihood and GAN-based training of Real NVPs
https://arxiv.org/pdf/1705.05263.pdf

How sensitive are the results to hyperparameters? It would be interesting to see some sensitivity analysis as well as understand the correlations between different metrics for different hyperparameters (cf. Appendix G in https://arxiv.org/pdf/1706.04987.pdf)

Do you think it would be useful to compare other generative models (e.g. VAEs) using these evaluation metrics? Some of the metrics don't capture perceptual similarity, but I'm curious to hear what you think. 

review_title: Evaluation of GAN evaluation metrics

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper describes an empirical evaluation of some of the most common metrics to evaluate GANs (inception score, mode score, kernel MMD, Wasserstein distance and LOO accuracy). 

The paper is well written, clear, organized and easy to follow.

Given that the underlying application is image generation, the authors move from a pixel representation of images to using the feature representation given by a pre-trained ResNet, which is key in their results and further comparisons. They analyzed discriminability, mode collapsing and dropping, robustness to transformations, efficiency and overfitting. 

Although this work and its results are very useful for practitioners, it lacks in two aspects. First, it only considers a single task for which GANs are very popular. Second, it could benefit from a deeper (maybe theoretical analysis) of some of the questions. Some of the conclusions could be further clarified with additional experiments (e.g., Sec 3.6 ‘while the reason that RMS also fails to detect overfitting may again be its lack of generalization to datasets with classes not contained in the ImageNet dataset’).

review_title: An empirical comparison of 4 metrics for GANs

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper introduces a comparison between several approaches for evaluating GANs. The authors consider the setting of a pre-trained image models as generic representations of generated and real images to be compared. They compare the evaluation methods based on five criteria termed disciminability, mode collapsing and mode dropping, sample efficiency,computation efficiency, and robustness to transformation. This paper has some interesting insights and a few ideas of how to validate an evaluation method. The topic is an important one and a very difficult one. However, the work has some problems in rigor and justification and the conclusions are overstated in my view.

Pros
-Several interesting ideas for evaluating evaluation metrics are proposed
-The authors tackle a very challenging subject

Cons
-It is not clear why GANs are the only generative model considered
-Unprecedented visual quality as compared to other generative models has brought the GAN to prominence and yet this is not really a big factor in this paper.
-The evaluations rely on using a pre-trained imagenet model as a representation. The authors point out that different architectures yield similar results for their analysis, however it is not clear how the biases of the learned representations affect the results. The use of learned representations needs more rigorous justification
-The evaluation for discriminative metric, increased score when mix of real and unreal increases, is interesting but it is not convincing as the sole evaluation for “discriminativeness” and seems like something that can be gamed. 
- The authors implicitly contradict the argument of Theis et al against monolithic evaluation metrics for generative models, but this is not strongly supported.

Several references I suggest:
https://arxiv.org/abs/1706.08500 (FID score)
https://arxiv.org/abs/1511.04581 (MMD as evaluation)

review_title: An empirical comparison of metrics for GAN

Meta-review: The problem addressed here is an important one: What is a good evaluation metric for generative models?  A good selection of popular metrics are analyzed for their appropriateness for model selection of GANs.  Two popular approaches are recommended: the kernel Maximum Mean Discrepancy (MMD) and the 1-Nearest-Neighbour (1-NN) two-sample test.  This seems reasonable, but the present work was not recommended for acceptance by 2 reviewers who raised valid concerns.

From a readability perspective, it would be nice to simply list the answer to question (1) directly in the introduction.  One must read more than a few pages to get to the answer of why the metrics that are advocated were picked.  It need not read like a mystery.

R4: "The evaluations rely on using a pre-trained imagenet model as a representation. The authors point out that different architectures yield similar results for their analysis, however it is not clear how the biases of the learned representations affect the results. The use of learned representations needs more rigorous justification"

R2: "First, it only considers a single task for which GANs are very popular. Second, it could benefit from a deeper (maybe theoretical analysis) of some of the questions." - the first point of which is also related to a concern of R4.

Given the overall high selectivity of ICLR, the present submission falls short.
Decision: Reject