Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper claims to demystify MMD-GAN, a generative adversarial network with the maximum mean discrepancy (MMD) as a critic, by showing that the usual estimator for MMD yields unbiased gradient estimates (Theorem 1). It was noted by the authors that biased gradient estimate can cause problem when performing stochastic gradient descent, as also noted previously by Bellemare et al. The authors also proposed a kernel inception distance (KID) as a quantitative evaluation metric for GAN. The KID is defined to be the squared MMD between inception representation of the distributions. In experiments, the authors compared the quality of samples generated by MMD-GAN with various kernels with the ones generated from WGAN-GP (Gulrajani et al., 2017) and Cramer GAN (Bellemare et al., 2017). The empirical results show the benefits of using the MMD on top of deep convolutional features. 

The major flaw of this paper is that its contribution is not really clear. Showing that the expectation and gradient can be interchanged (Theorem 1) does not seem to provide sufficient significance. Unbiasedness of the gradient alone does not guarantee that training will be successful and that the resulting models will better reflect the underlying data distribution, as evident by other successful variants of GANs, e.g., WGAN, which employ biased estimate. Indeed, since the training process relies on a small mini-batch, a small bias could help counteract the potentially high variance of the gradient estimate. The key is rather a good balance of both bias and variance during the training process and a guarantee that the estimate is asymptotically unbiased wrt the training iterations. Lastly, I do not see how the empirical results would demystify MMD-GANs, as claimed by the paper.

The paper is clearly written. 

Some minor comments:

- The proof of the main result, Theorem 1, should be placed in the main paper.
- Page 7, 2nd paragraph: later --> layer
review_title: The contribution is too incremental!

final_score: rating: 6: Marginally above acceptance threshold
review_content: The main contribution of the paper is that authors extend some work of Bellemare: they show that MMD GANs [which includes the Cramer GAN as a subset] do possess unbiased gradients. They provide a lot of context for the utility of this claim, and in the experiments section they provide a few different metrics for comparing GANs [as this is a known tricky problem]. The authors finally show that an MMD GAN can achieve comparable performance with a much smaller network used in the discriminator.

As previously mentioned, the big contribution of the paper is the proof that MMD GANs permit unbiased gradients. This is a useful result; however, given the lack of other outstanding theoretical or empirical results, it almost seems like this paper would be better shaped as a theory paper for a journal. I could be swayed to accept this paper however if others feel positive about it.


review_title: Good overview; main contribution is theoretical proof

final_score: rating: 7: Good paper, accept
review_content: The quality and clarity of this work are very good. The introduction of the kernel inception metric is well-motivated and novel, to my knowledge. With the mention of a bit more related work (although this is already quite good), I believe that this could be a significant resource for understanding MMD GANs and how they fit into the larger model zoo.

Pros
 - best description of MMD GANs that I have encountered
 - good contextualization of related work and descriptions of relationships, at least among the works surveyed
 - reasonable proposed metric (KID) and comparison with other scores
 - proof of unbiased gradient estimates is a solid contribution

Cons
 - although the review of related work is very good, it does focus on ~3 recent papers. As a review, it would be nice to see mention (even just in a list with citations) of how other models in the zoo fit in
 - connection between IPMs and MMD gets a bit lost; a figure (e.g.  flow chart) would help
 - wavers a bit between proposing/proving novel things vs. reviewing and lacks some overall structure/storyline
 - Figure 1 is a bit confusing; why is KID tested without replacement, and FID with? Why 100 vs 10 samples? The comparison is good to have, but it's hard to draw any insight with these differences in the subfigures. The figure caption should also explain what we are supposed to get out of looking at this figure.

Specific comments:
 - I suggest bolding terms where they are defined; this makes it easy for people to scan/find (e.g. Jensen-Shannon divergence, Integral Probability Metrics, witness functions, Wasserstein distance, etc.) 
 - Although they are common knowledge in the field, because this is a review it could be helpful to provide references or brief explanations of e.g. JSD, KL, Wasserstein distance, RKHS, etc.
 - a flow chart (of GANs, IPMs, MMD, etc., mentioning a few more models than are discussed in depth here, would be *very* helpful.
 - page 2, middle paragraph, you mention "...constraints to ensure the kernel distribution embeddings remained injective"; it would be helpful to add a sentence here to explain why that's a good thing.

review_title: Clearly written review of MMD gans with some good insights

Meta-review: This paper does an excellent job at helping to clarify the relationship between various, recently proposed GAN models. The empirical contribution is small, but the KID metric will hopefully be a useful one for researchers. It would be really useful to show that it maintains its advantage when the dimensionality of the images increases (e.g., on Imagenet 128x128).
Decision: Accept (Poster)