Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Overall comments: Trying to shed light at comparison between different GAN variants, but the metrics introduced are not very novel, results are not comparable with prior work and older version of certain models are used (WGAN instead of Improved WGAN)

Section 2.1: quantifying mode collapse
* This section should mention Inception scores. A model which collapses on only one class will have a low inception score, and this metric also uses a conv net classifier, as the approach is very similar (the method is only mentioned briefly in section 2.3)
* The authors might not be aware of concurrent work published before the ICLR deadline, which introduces a very similar metric: https://arxiv.org/abs/1706.08500

Section 2.2: measuring diversity:
* There is an inherent flaw in this metric, namely it trains one GAN per class. One cannot generalize from this metric on how different GAN models will perform when trained on the entire dataset. One model might be able to capture more diverse distributions, but lose a bit of quality, while another model might be able to create good samples when train on low diversity data. We already know that when looking at other generative models, we can find such examples. VAEs can obtain very good samples on celebA, a dataset with relative low diversity, but not so good samples on cifar. 
* The authors compare their experiment with Radford et al. (2015), but that needs to be done with caution. In Radford et al. (2015), the authors use a conditional generative model trained on the entire dataset. In that setting, this test is more suitable since one can test how good well the model has learned the conditioning. For example, for a conditional model trained on cats and dogs, a failure mode is that the model generates only cats. This failure mode can then be captured by this metric. However, when training two models, one on cats and one on dogs, this failure mode is not present since the data is already split into classes. 
* The proposed metric is not necessarily a diversity metric, it is also a quality metric:  in a situation where all the models diverge and generate random noise, with high diversity, but without any structure. This metric would capture this issue, because a classifier will not be able to learn the classes, because there is no correlation between the classes and the generated images. 

Experimental results:
* Positive insights regarding labels and celeba. Looks like subtle labels on faces are not being captured by GAN models. 
Figure 1 is hard to read. 
* ALI having higher diversity on celeba is explicitly mentioned in a paper the authors cite, namely “Do GANs actually learn the distribution? An empirical study”. Would be nice to mention that in the paper.

Would like to see:
* A comparison with the Improved Wasserstein GAN model. This model is now the one used by the community, as opposed to the original Wasserstein GAN.
* Models trained on cifar, with the reported inception scores of the models on cifar. That makes the paper comparable with previous work and is a test against bugs in model implementations or other parts of the code. This would also allow to test for claims such as the fact that the Improved GAN has more mode collapse than DCGAN, while the Improved GAN paper says the opposite. 
The reason why the authors chose the models they did for comparison. In the BiGAN (same model as ALI) paper the authors report a low inception score, which suggests that their model is not able to capture the subtleties of the Cifar dataset, and this seems to be correlated with the results obtained in this work. 



review_title: Trying to shed light at comparison between different GAN variants, but the metrics introduced are not very novel, results are not comparable with prior work and older version of certain models are used

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a new evaluation measure for evaluating GANs. Specifically, the paper proposes generating synthetic images using GAN, training a classifier (for an auxiliary task, not the real vs fake discriminator) and measuring the performance of this classifier on held out real data. 

While the idea of using a downstream classification task to evaluate the quality of generative models has been explored before  (e.g. semi-supervised learning), I think that this is the first paper to evaluate GANs using such an evaluation metric.

I'm not super convinced that this is an useful evaluation metric as the absolute number is somewhat to interpret and dependent on the details of the classifier used. The results in Table 1 change quite a bit depending on the classifier. 

It would be useful to add a discussion of the failure modes of the proposed metric. It seems like a generator which generates samples close to the classification boundary (but drops examples far away from the boundary) could still achieve a high score under this metric. 

In the experiments, were different architectures used for different GAN variants?

I think the mode-collapse evaluation metrics in MR-GAN are worth discussing in Section 2.1
Mode Regularized Generative Adversarial Networks
https://arxiv.org/abs/1612.02136
review_title: Training a classifier to evaluate GANs

final_score: rating: 3: Clear rejection
review_content: This paper propose to evaluate the distributions learned by GAN using classification-based methods. As two examples, the authors evaluates the mode collapse effect and measure the diversity for GAN distributions. The proposed approaches are experimental but does not require human inspection. The main idea is to fit a classifier on the training data and also learn a GAN model using the training data. Then generate simulated data using GAN and use the classifier to predict the labels of the simulated data. The distribution of predicted labels  and the labels of the true data can be easily compared.

Detailed comments:

1. The proposed method is purely experimental. It  would be better to gain some theoretical insights of this methodology. Moreover, in terms of experiments, it would be nice to consider more examples except for mode collapse and diversity, since these problems are well-known for GAN.

2. Since mode collapse is a well-known phenomenon, the novelty of this paper is not sufficient.

3. There are other measures for the quality of GAN. For example, the inception scores and mode scores (Salimans et al. 2016, Che et al. 2017). It would be nice to compare the method here with other related work.

References:
1. Improved Techniques for Training GANs https://arxiv.org/abs/1606.03498

2. Mode Regularized Generative Adversarial Networks https://arxiv.org/abs/1612.02136


review_title: Neat idea. But not worth publishing

Meta-review: The paper proposes a new metric to measure GAN performance by training a classifier on the true labeled dataset and then comparing the distribution of the labels of the generated samples to the true label distribution. Reviewers find that the paper is well written but lacks novelty and is quite experimental does not present any new insights. The paper investigates well-known model collapse and diversity issues. Reviewers are not convinced that this is a good metric to measure sample quality or diversity as the generator can drop examples far away from the boundary and still achieve a good score on this metric.
Decision: Reject