Venue: ICLR 2018 conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: The main idea is to use the accuracy of a classifier trained on synthetic training examples produced by a generative model to define an evaluation metric for the generative model. Specifically, compare the accuracy of a classifier trained on a noise-perturbed version of the real dataset to that of a classifier trained on a mix of real data and synthetic data generated by the model being evaluated. Results are shown on MNIST and Fashion MNIST.

The paper should discuss the assumptions needed for classifier accuracy to be a good proxy for the quality of a generative model that generated the classifier's training data. It may be the case that even a "bad" generative model (according to some other metric) can still result in a classifier that produces reasonable test accuracy. Since a classifier can be a highly nonlinear function, it can potentially ignore many aspects of its input distribution such that even poor approximations (as measured by, say, KL) lead to similar test accuracy as good approximations.

The sensitivity of the evaluation metric defined in equation 2 to the choice of hyperparameters of the classifier and the metric itself (e.g., alpha) is not evaluated. Is it possible that a different choice of hyperparameters can change the model ranking? Should the hyperparameters be tuned separately for each generative model being evaluated?

The intuition behind comparing against a classifier trained on a noise-perturbed version of the data is not explained clearly. Why not compare a classifier trained on only (unperturbed) real data to a classifier trained on both real and synthetic data?

Evaluation on two datasets is not sufficient to provide insight into whether the proposed metric is useful. Other datasets such as ImageNet, Cifar10/100, Celeb A, etc., should also be included.
review_title: insufficient evaluation

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes a technique for analysis of generative models.

The main idea is to (1) define a classification task on the underlying data, (2) use the generative model to produce a training set for this classification task, and (3) compare performance on the classification task when training on generated and real training data. 

Note that in step (2) is it required to assign class labels to the generated samples. In this paper this is achieved by learning a separate generative model for each class label. 

Summary:

I think the proposed technique is useful, but needs to be combined with other techniques to exclude the possibility that model just memorized the training set. To be stronger the paper needs to consider other more realistic tasks from the literature and directly compare to other evaluation protocols. 

Positive:
+ the technique operates directly on the samples from the model. It is not required to compute the likelihood of the test set as for example is needed in  the "perplexity" measure). This makes the technique applicable for evaluation of a wider class of techniques.  

+ I like the result in Fig. 1. There is a clear difference between results by WGAN and by other models. This experiment convinces me that the peroposed analysis by augmentation is a valuable tool. 

+ I think the technique is particularly valuable verify that samples are capturing variety of modes in the data. Verifying this via visual inspection is difficult.

Negative: 

- I think this metric can be manipulated by memorizing training data, isn't it? The model that reproduces the training set will still achieve good performance at \tau = 1, and the model that does simple augmentation like small shifts / rotations / scale and contrast changes might even improve over training data alone. So the good performance on the proposed task does not mean that the model generalized over the training dataset.

- I believe that for tasks such as image generating none of the existing models generate samples that would be realistic enough to help in classification. Still some methods produce images that are more realistic than others. I am not sure if the proposed evalaution protocol would be useful for this type of tasks. 

- The paper does not propose an actual metric. Is the metric for performance of generative model defined by the best relative improvement over baseline after tuning \tau as in Tab. 1?  Wouldn't it be better to fix \tau, e.g. \tau = 1?

- Other datasets / methods and comparison to other metrics. This is perhaps the biggest limitation for me right now. To establish a new comparison method the paper needs to demonstrate it on relevant tasks (e.g. image generation?), and compare to existing metrics (e.g. "visual inspection" and "average log-likelihood"). 


review_title: Potentially useful technique for evaluation of generative models (need experiments on real tasks) 

final_score: rating: 3: Clear rejection
review_content: The authors propose to evaluate how well generative models fit the training set by analysing their data augmentation capacity, namely the benefit brought by training classifiers on mixtures of real/generated data, compared to training on real data only. Despite the the idea of exploiting generative models to perform data augmentation is interesting, using it as an evaluation metric does not constitute an innovative enough contribution. 

In addition, there is a fundamental matter which the paper does not address: when evaluating a generative model, one should always ask himself what purpose the data is generated for. If the aim is to have realistic samples, a visual turing test is probably the best metric. If instead the purpose is to exploit the generated data for classification, well, in this case an evaluation of the impact of artificial data over training is a good option.

PROS:
The idea is interesting. 

CONS:
1. The authors did not relate the proposed evaluation metric to other metrics cited (e.g., the inception score, or a visual turing test, as discussed in the introduction). It would be interesting to understand how the different metrics relate. Moreover, the new metric is introduced with the following motivation “[visual Turing test and Inception Score] do not indicate if the generator collapses to a particular mode of the data distribution”. The mode collapse issue is never discussed elsewhere in the paper. 

2. Only two datasets were considered, both extremely simple: generating MNIST digits is nearly a toy task nowadays. Different works on GANs make use of CIFAR-10 and SVHN, since they entail more variability: those two could be a good start. 

3. The authors should clarify if the method is specifically designed for GANs and VAEs. If not, section 2.1 should contain several other works (as in Table 1). 

4. One of the main statements of the paper “Our approach imposes a high entropy on P(Y) and gives unbiased indicator about entropy of both P(Y|X) and P(X|Y)” is never proved, nor discussed.

5. Equation 2 (the proposed metric) is not convincing: taking the maximum over tau implies training many models with different fractions of generated data, which is expensive. Further, how many tau’s one should evaluate? In order to evaluate a generative model one should test on the generated data only (tau=1) I believe. In the worst case, the generator experiences mode collapse and performs badly. Differently, it can memorize the training data and performs as good as the baseline model. If it does actual data augmentation, it should perform better.

6. The protocol of section 3 looks inconsistent with the aim of the work, which is to evaluate data augmentation capability of generative models. In fact, the limit of training with a fixed dataset is that the model ‘sees’ the data multiple times across epochs with the risk of memorizing. In the proposed protocol, the model ‘sees’ the generated data D_gen (which is fixed before training) multiple time across epochs. This clearly does not allow to fully evaluate the capability of the generative model to generate newer and newer samples with significant variability.


Minor: 
Section 2.2 might be more readable it divided in two (exploitation and evaluation).   

review_title: Not very novel, weak analysis and justification.

Meta-review: Given that the paper proposes a new evaluation scheme for generative models, I agree with the reviewers that it is essential that the paper compare with existing metrics (even if they are imperfect). The choice of datasets was very limited as well, given the nature of the paper. I acknowledge that the authors took care to respond in detail to each of the reviews.
Decision: Reject