Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes using divergence and distance functions typically used for generative model training to evaluate the performance of various types of GANs. Through numerical evaluation, the authors observed that the behavior is consistent across various proposed metrics and the test-time metrics do not favor networks that use the same training-time criterion. 

More specifically, the evaluation metric used in the paper are: 1) Jensen-Shannon divergence, 2) Constrained Pearson chi-squared, 3) Maximum Mean Discrepancy, 4) Wasserstein Distance, and 5) Inception Score. They applied those metrics to compare three different GANs: the standard DCGAN, Wasserstein DCGAN, and LS-DCGAN on MNIST and CIFAR-10 datasets. 

Summary:
——
In summary, it is an interesting topic, but I think that the paper does not have sufficient novelty. Some empirical results are still preliminary. It is hard to judge the effectiveness of the proposed metrics for model selection and is not clear that those metrics are better qualitative descriptors to replace visual assessment. In addition, the writing should be improved. See comments below for details and other points.

Comments:
——
1.	In Section 3, the evaluation metrics are existing metrics and some of them have already been used in comparing GAN models.  Maximum mean discrepancy has been used before in work by Yujia Li et al. (2016, 2017)

2.	In the experiments, the proposed metrics were only tested on small scale datasets; the authors should evaluate on larger datasets such as CIFAR-100, Toronto Faces, LSUN bedrooms or CelebA.

3.	In the experiments, the authors noted that “Gaussian observable model might not be the ideal assumption for GANs. Moreover, we observe a high log-likelihood at the beginning of training, followed by a drop in likelihood, which then returns to the high value, and we are unable to explain why this happens.” Could the authors give explanation to this phenomenon? The authors should look into this more carefully.

4.	In algorithm 1, it seems that the distance is computed via gradient decent. Is it possible to show that the optimization always converges? Is it meaningful to compare the metrics if some of them cannot be properly computed?

5.     With many different metrics for assessing GANs, how should people choose? How do we trust the scores? Recently, Fréchet Inception Distance (FID) was proposed to evaluate the samples generated from GANs (Heusel et al. 2017), how are the above scores compared with FID?

Minor Comments:
——
1.	Writing should be fixed: “It seems that the common failure case of MMD is when the mean pixel intensities are a better match than texture matches (see Figure 5), and the common failure cases of IS happens to be when the samples are recognizable textures, but the intensity of the samples are either brighter or darker (see Figure 2).”

review_title: review

final_score: rating: 7: Good paper, accept
review_content: the paper proposes an evaluation method for training GANs using four standard distribution distances in literature namely:
- JSD
- Pearson-chi-square
- MMD
- Wasserstein-1

For each distance, a critic is initialized with parameters p. The critic is a neural network with the same architecture as the discriminator.
The critic then takes samples from the trained generator model, and samples from the groundtruth dataset. It trains itself to maximize the distance measure between these two distributions (trained via gradient descent).

These critics after convergence will then give a measure of the quality of the generator (lower the better).

The paper is easy to read, experiments are well thought out.
Figure 3 is missing (e) and (f) sub-figures.

When proposing a distance measure for GANs (which is a high standard, because everyone is looking forward to a robust measure), one has to have enough convincing to do. The paper only does experiments on two small datasets MNIST and CIFAR. If the paper has to convince me that this metric is good and should be used, I need to see experiments on one large-scale datset, such as Imagenet or LSUN. If one can clearly identify the good generators from bad generators using a weighted-sum of these 4 distances on Imagenet or LSUN, this is a metric that is going to stand well.
If such experiments are constructed in the rebuttal period, I shall raise my rating.
review_title: new evaluation metrics for GANs

final_score: rating: 7: Good paper, accept
review_content: Through evaluation of current popular GAN variants. 
  * useful AIS figure
  * useful example of failure mode of inception scores
   * interesting to see that using a metric based on a model’s distance does not make the model better at that distance
the main criticism that can be given to the paper is that the proposed metrics are based on trained models which do not have an independent clear evaluation metrics (as classifiers do for inception scores). However, the authors do show that the results are consistent when changing the critic architecture. Would be nice to see if this also holds for changes in learning rates. 
 * nice to see an evaluation on how models scale with the increase in training data.

Using an Independent critic for evaluation has been proposed and used in practice before, see “Comparison of Maximum Likelihood and GAN-based training of Real NVPs”, Danihelka et all, as well as Variational Approaches for Auto-Encoding Generative Adversarial Networks, Rosca at all.

Improvements to be added to the paper:
   * How about overfitting? Would be nice to mention whether the proposed metrics are useful at detecting overfitting. From algorithm 1 one can see that the critic is trained on training data, but at evaluation time test data is used. However, if the generator completely memorizes the training set, the critic will not be able to learn anything useful. In that case, the test measure will not provide any information either. A way to go around this is to use validation data to train the critic, not training data. In that case, the critic can learn the difference between training and validation data and at test time the test set can be used. 
  * Using the WGAN with weight clipping is not a good baseline. The improved WGAN method is more robust to hyper parameters and is the one currently  used by the community. The WGAN with weight clipping is quite sensitive to the clipping hyperparameter, but the authors do not report having changed it from the original paper, both for the critic or for the discriminator used during training. 
  *  Is there a guidance for  which metric should be used? 

Figure 3 needs to be made a bit larger, it is quite hard to read in the current set up. 
review_title: Through evaluation of current popular GAN variants. 

Meta-review: + clearly written and thorough empirical comparison of several metrics/divergences for evaluating GANs, prominently parametric-critic based divergences.
 - little technical novelty with respect to prior work. As noted by reviewers and an anonymous commentator:  using an Independent critic for evaluation has been proposed and used in practice before.
 +  the contribution of the work thus lies primarily in its well-done and extensive empirical comparisons of multiple metrics and models    
Decision: Accept (Poster)