Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposed a procedure for assessing the performance of GANs by re-considering the key of observation. And using the procedure to test and improve current version of GANs. It demonstrated some interesting stuff. 

It is not easy to follow the main idea of the paper. The paper just told difference stories section by section. Based on my understanding, the claims are 1) the new formalization of the goal of GAN training and 2) using this test to evaluate the success of GAN algorithms empirically?  I suggested that the author should reform the structure, ignore some unrelated content and make the clear claims about the contributions on the introduction part.  

Regarding the experimental part, it can not make strong support for all the claims. Figure 2 showed almost similar plots for all the varieties. Meanwhile, the results are performed on some specific model configurations (like ResNet) and settings. It is difficult to justify whether it can generalize to other cases. Some of the figures do not have the notations of curvey, making people hard to compare. 

Therefore, I think the current version is not ready to be published. The author can make it stronger and consider next venue. 
review_title: Official review for the paper 

final_score: rating: 6: Marginally above acceptance threshold
review_content: The quality of the paper is good, and clarity is mostly good. The proposed metric is interesting, but it is hard to judge the significance without more thorough experiments demonstrating that it works in practice.

Pros:
 - clear definitions of terms
 - overall outline of paper is good
 - novel metric

Cons
 - text is a bit over-wordy, and flow/meaning sometimes get lost. A strict editor would be helpful, because the underlying content is good
 - odd that your definition of generalization in GANs appears immediately preceding the section titled "Generalisation in GANs"
 - the paragraph at the end of the "Generalisation in GANs" section is confusing. I think this section and the previous ("The objective of unsupervised learning") could be combined, removing some repetition, adding some subtitles to improve clarity. This would cut down the text a bit to make space for more experiments.
 - why is your definition of generalization that the test set distance is strictly less than training set ? I would think this should be less-than-or-equal
 - there is a sentence that doesn't end at the top of p.3: "... the original GAN paper showed that [ends here]"
 - should state in the abstract what your "notion of generalization" for gans is, instead of being vague about it
 - more experiments showing a comparison of the proposed metric to others (e.g. inception score, Mturk assessments of sample quality, etc.) would be necessary to find the metric convincing
 - what is a "pushforward measure"? (p.2)
 - the related work section is well-written and interesting, but it's a bit odd to have it at the end. Earlier in the work (e.g. before experiments and discussion) would allow the comparison with MMD to inform the context of the introduction
 - there are some errors in figures that I think were all mentioned by previous commentators.
review_title: Good paper, a bit wordy and lacking a few experiments

final_score: rating: 5: Marginally below acceptance threshold
review_content: `The papers aims to provide a quality measure/test for GANs.   The objective is ambitious an deserve attention. As GANs are minimizing some f-divergence measure, the papers remarks that computing a  Wasserstein distance between two distributions made of a sum of Diracs is not a degenerate case and is tractable. So they propose evaluate the current approximation of a distribution learnt by a GAN by using this distance as a baseline performance (in terms of W distance and computed on a hold out dataset). 

A first remark is that the papers does not clearly develop the interest of puting things a trying to reach a treshold of performance in W distance rather than just trying to minimize the desired f-divergence. More specifically as they assess the performance in terms of W distance I would would be tempted to just minimize the given criterion. This would be very interesting to have arguments on why being better than the "Dirac estimation" in terms of W2 distance would lead to better performance for others tasks (as other f-divergences or image generation).

According to the authors the core claims are:
"1/ We suggest a formalisation of the goal of GAN training (/generative modelling more broadly) in terms of divergence minimisation. This leads to a natural, testable notion of generalisation. "
Formalization in terms of divergence minimization is not new (see O. Bousquet & all https://arxiv.org/pdf/1701.02386.pdf ) and I do not feel like this paper actually performs any "test" (in a statistical sense). In my opinion the contribution is more about exhibiting a baseline which has to be defeated for any algorithm interesting is learning the distribution in terms of W2 distance.

"2/ We use this test to evaluate the success of GAN algorithms empirically, with the Wasserstein distance as our divergence."
Here the distance does not seems so good because the performance in generation does not seems to only be related to W2 distance. Nevertheless, there is interesting observations in the paper about the sensitivity of this metric to the bluring of pictures. I would enjoyed more digging in this direction. The authors proposes to solve this issue by relying to an embedded space where the L2 distance makes more sense for pictures (DenseNet). This is of course very reasonable but I would expect anyone working on distribution over picture to work with such embeddings. Here I'm not sure if this papers opens a new way to improve the embedding making use on non labelled data. One could think about allowing the weights of the embeddings to vary while f-divergence is minimized but this is not done in the submitted work.

 "3/ We find that whether our proposed test matches our intuitive sense of GAN quality depends heavily on the ground metric used for the Wasserstein distance."
This claim is highly biased by who is giving the "intuitive sense". It would be much better evaluated thought a mechanical turk test.

 "4/ We discuss how to use these insights to improve the design of WGANs more generally."
As our understanding of the GANs dynamics are very coarse, I feel this is not a good thing to claim that "doing xxx should improve things" without actually trying it. 

review_title: Interesting idea but realization could be improved

final_score: rating: null
review_content: 
- I'd like the claims of the paper to be more clearly apparent. As an example it seems that the papers claims that using the (1) criterion is a good idea for testing the generalization quality of a generative model but fig 3 clearly shows that it is not always the case. 

- In section (4) What is special about using a W distance in (1) to perform the GAN quality assertion?  Is it just because it can be estimated easily and is meaningful when the two distributions are a sum of Diracs?
 
-  There is a direct link between the cost optimized by a GAN and the minimization of f-divergences (see §2.1 of https://arxiv.org/pdf/1701.02386.pdf for details). Why would it not be enough to estimate this divergence on the test set ? In other word,  what is the interest to set a threshold on the divergence rather than just minimizing it?

- Is the idea proposed in §5 consist in solving the optimal transport problem in an embedding space (ResNet one as an example) rather than on the image space?

- About the form of the paper, I usually think that following the convention of ML papers and stating core claims as theorems whenever possible and highlighting proposed algorithms. 
review_title: Some questions

Meta-review: This paper proposes a method for quantitatively evaluating GANs. Better quantitative metrics for GANs are badly needed, as the field is being held back by excessive focus on generated samples. This paper proposes to estimate the Wasserstein distance to the data distribution. A paper which does this well would be a significant contribution, but unfortunately (as the reviewers point out) the experimental validation in this paper seems insufficient.

To be convincing, a paper would first need to demonstrate the ability to accurately estimate Wasserstein distance -- not an easy task, but one which receives little mention in this paper. Then it would need to validate that the method can either quantitatively confirm known results about GANs or uncover previously unknown phenomena. As it stands, I don't think this submission is ready for publication in ICLR, but I'd encourage resubmission after more careful experimental validation along the lines suggested by the reviewers.


Decision: Reject