Demystifying MMD GANs

Abstract: We investigate the training and performance of generative adversarial networks using the Maximum Mean Discrepancy (MMD) as critic, termed MMD GANs. As our main theoretical contribution, we clarify the situation with bias in GAN loss functions raised by recent work: we show that gradient estimators used in the optimization process for both MMD GANs and Wasserstein GANs are unbiased, but learning a discriminator based on samples leads to biased gradients for the generator parameters. We also discuss the issue of kernel choice for the MMD critic, and characterize the kernel corresponding to the energy distance used for the Cramér GAN critic. Being an integral probability metric, the MMD benefits from training strategies recently developed for Wasserstein GANs. In experiments, the MMD GAN is able to employ a smaller critic network than the Wasserstein GAN, resulting in a simpler and faster-training algorithm with matching performance. We also propose an improved measure of GAN convergence, the Kernel Inception Distance, and show how to use it to dynamically adapt learning rates during GAN training.

Reviews and author responses:

This paper claims to demystify MMD-GAN, a generative adversarial network with the maximum mean discrepancy (MMD) as a critic, by showing that the usual estimator for MMD yields unbiased gradient estimates (Theorem 1). It was noted by the authors that biased gradient estimate can cause problem when performing stochastic gradient descent, as also noted previously by Bellemare et al. The authors also proposed a kernel inception distance (KID) as a quantitative evaluation metric for GAN. The KID is defined to be the squared MMD between inception representation of the distributions. In experiments, the authors compared the quality of samples generated by MMD-GAN with various kernels with the ones generated from WGAN-GP (Gulrajani et al., 2017) and Cramer GAN (Bellemare et al., 2017). The empirical results show the benefits of using the MMD on top of deep convolutional features. 

The major flaw of this paper is that its contribution is not really clear. Showing that the expectation and gradient can be interchanged (Theorem 1) does not seem to provide sufficient significance. Unbiasedness of the gradient alone does not guarantee that training will be successful and that the resulting models will better reflect the underlying data distribution, as evident by other successful variants of GANs, e.g., WGAN, which employ biased estimate. Indeed, since the training process relies on a small mini-batch, a small bias could help counteract the potentially high variance of the gradient estimate. The key is rather a good balance of both bias and variance during the training process and a guarantee that the estimate is asymptotically unbiased wrt the training iterations. Lastly, I do not see how the empirical results would demystify MMD-GANs, as claimed by the paper.

The paper is clearly written. 

Some minor comments:

- The proof of the main result, Theorem 1, should be placed in the main paper.
- Page 7, 2nd paragraph: later --> layer

The quality and clarity of this work are very good. The introduction of the kernel inception metric is well-motivated and novel, to my knowledge. With the mention of a bit more related work (although this is already quite good), I believe that this could be a significant resource for understanding MMD GANs and how they fit into the larger model zoo.

Pros
 - best description of MMD GANs that I have encountered
 - good contextualization of related work and descriptions of relationships, at least among the works surveyed
 - reasonable proposed metric (KID) and comparison with other scores
 - proof of unbiased gradient estimates is a solid contribution

Cons
 - although the review of related work is very good, it does focus on ~3 recent papers. As a review, it would be nice to see mention (even just in a list with citations) of how other models in the zoo fit in
 - connection between IPMs and MMD gets a bit lost; a figure (e.g.  flow chart) would help
 - wavers a bit between proposing/proving novel things vs. reviewing and lacks some overall structure/storyline
 - Figure 1 is a bit confusing; why is KID tested without replacement, and FID with? Why 100 vs 10 samples? The comparison is good to have, but it's hard to draw any insight with these differences in the subfigures. The figure caption should also explain what we are supposed to get out of looking at this figure.

Specific comments:
 - I suggest bolding terms where they are defined; this makes it easy for people to scan/find (e.g. Jensen-Shannon divergence, Integral Probability Metrics, witness functions, Wasserstein distance, etc.) 
 - Although they are common knowledge in the field, because this is a review it could be helpful to provide references or brief explanations of e.g. JSD, KL, Wasserstein distance, RKHS, etc.
 - a flow chart (of GANs, IPMs, MMD, etc., mentioning a few more models than are discussed in depth here, would be *very* helpful.
 - page 2, middle paragraph, you mention "...constraints to ensure the kernel distribution embeddings remained injective"; it would be helpful to add a sentence here to explain why that's a good thing.


The main contribution of the paper is that authors extend some work of Bellemare: they show that MMD GANs [which includes the Cramer GAN as a subset] do possess unbiased gradients. They provide a lot of context for the utility of this claim, and in the experiments section they provide a few different metrics for comparing GANs [as this is a known tricky problem]. The authors finally show that an MMD GAN can achieve comparable performance with a much smaller network used in the discriminator.

As previously mentioned, the big contribution of the paper is the proof that MMD GANs permit unbiased gradients. This is a useful result; however, given the lack of other outstanding theoretical or empirical results, it almost seems like this paper would be better shaped as a theory paper for a journal. I could be swayed to accept this paper however if others feel positive about it.



We just uploaded a new revision with a minor improvement: Table 3 now contains test set metrics for the LSUN bedrooms dataset, for context, as for the other tables. All GAN models we considered obtain higher Inception scores than the test set, highlighting the inappropriateness of the Inception score for this dataset.

Thanks to all for their comments. We just posted a new revision addressing many of the comments, as well as the following general improvements:

First, a note on the bias situation. After submission, we cleaned up the proof of unbiasedness and, in doing so, noticed that we were able to generalize it significantly. Our unbiasedness result now covers nearly all feedforward neural network structures used in practice, rather than just ReLU networks as before. We also realized in this process that, with very little extra work, we could cover not just MMD GANs but also WGANs and even original GANs (with bounded discriminator outputs to avoid the logs blowing up). This at first seems counterintuitive, since of course the Cramér GAN paper showed that Wasserstein has biased sample gradients. We have thus added a detailed description of the relationship to the theory section and to the new Appendix B. In short: with a fixed kernel, the MMD estimator is unbiased, but the estimator of the supremum over kernels of the MMD is biased. Likewise, with a fixed critic function, the Wasserstein estimator (such as it is) is unbiased, but the estimator of the supremum over critic functions (the actual Wasserstein) is biased. Thus the bias situation is more analogous between the two models than had been previously thought, which our paper now helps to substantially clarify.

We also cleaned up the experimental results somewhat, including new results on LSUN that we didn't have time to finish for the initial submission. While doing that, we also used the KID in a new way: to dynamically adapt the learning rate based on the similarity of the generator's model to the training set, using the relative similarity test of https://arxiv.org/abs/1511.04581. This is similar to popular schemes used in supervised learning based on validation set accuracy, and allows for less manual tuning of the learning rate decay (which can be very important, and differ between models).

Thanks for your comments. We've posted a new revision addressing most of them; see also our separate comment describing other significant improvements.

- Review of related work: thanks for the suggestion. We have added a brief section 2.4 with some more related work; we would be happy to add more if you have some other suggestions.

- We have attempted to slightly clarify the description of IPMs in this revision, and will further consider better ways to do this.

- KID/FID comparison figure: We agree that this difference is confusing. It was done because the standard KID estimator becomes biased when there are repeated points due to sampling with replacement, but of course when sampling 10,000 / 10,000 points without replacement, it is unsurprising that there is no variance in the estimate, so it made more sense for the point we were trying to make to evaluate FID with replacement. The difference in number of samples was due to the relatively higher computational expense of the FID (which requires the SVD of a several thousand dimensional-matrix), but we have increased that to the same number of samples as well. The figures look essentially identical changing either of these issues; we have changed to using a variant of the KID estimator which is still unbiased for samples with replacement and clarified the caption.

- We have added a footnote on why injectivity of the distribution embeddings is desirable.

Thanks for your comments, and please also see our comments about improvements in the new revision above.

You are certainly correct that an unbiased gradient does not guarantee that training will succeed; our recent revision also substantially clarifies the bias situation. However, in SGD the bias-variance tradeoff is somewhat different than the situation in e.g. ridge regression, where the regularization procedure adds some bias but also reduces variance enough that it is worthwhile. There doesn't seem to be any reason to think that the gradient variance is any higher for MMD GANs than for WGANs, and so a direct analogy doesn't quite apply. Also, when performing SGD, the biases of each step might add up over time, and so – as in Bellemare et al.'s example – following biased gradients is worth at least some level of concern.

With regards to the rest of the contribution: the title "demystifying" was intended more for the earlier parts of the paper, which elucidate the relationship of MMD GANs to other models and (especially in the revision) clarify the nature of the bias argument of Bellemare et al. The empirical results perhaps do not directly "demystify," but rather bring another level of understanding of these models.

Thanks for your comments. We do feel that this paper has contributions outside just the proof of unbiased gradients, in particular clarifying the relationship among various slightly-different GAN models, the KID score, and the new experimental results about success with smaller critic networks, which are of interest to the ICLR community.

Please also see our general comments about the new revision above, which includes substantial improvements.


It's true that the model we consider here is also described in Section 5.5 of the latest version of Li et al. This result was not in the original version of the paper, however, and only appeared in the revised arXiv submission of November 6, a week and a half after the ICLR deadline; we were not aware of the new version until you pointed it out (thanks for doing so).

That said, the main point of our paper is not to propose yet another GAN variation (YAGAN?). The idea of using an MMD as critic with a kernel defined on deep convolutional features is not new, nor is the idea of regularizing the gradient of the critic witness function: we cite the papers where (to our knowledge) these ideas were first proposed.  The point of our paper is to understand (and "demystify") the MMD GAN, and its relation with other integral probability metric-based GANs. In this direction, our new results are in three areas:

* We clarify the relationship between MMD GANs and Cramér GANs, and the relationship of the MMD GAN critic and witness function to those of WGAN-GPs (thus explaining why the gradient penalty makes sense for the MMD GAN).

* We formally show the unbiasedness of the gradient of the MMD estimator wrt the network parameters, in Theorem 1.  This is our main theoretical result, and an important property to establish when the MMD is used as a GAN critic.

* Our main new experimental finding is that MMD GANs seem to work about as well as WGAN-GPs that use much larger critic networks. Thus, for a given generator, MMD GANs will be simpler and faster to train than WGAN-GPs. Our understanding of why this happens is described in detail in the paper.

Along the way, we also proposed the KID score, which is a more natural metric of generative model convergence than the Inception score, and inherits many of the nice properties of the previously-proposed FID, but is much easier and more intuitive to estimate.

What is the main difference between this paper and the MMD GAN paper [Li et al. 2017]. For my understanding, it is just the MMD GAN work which replaced clipping with gradient penalty. GP was also mentioned and tried in [Li et al. 2017] (last version on arXiv) https://arxiv.org/pdf/1705.08584.pdf  

Meta-review: This paper does an excellent job at helping to clarify the relationship between various, recently proposed GAN models. The empirical contribution is small, but the KID metric will hopefully be a useful one for researchers. It would be really useful to show that it maintains its advantage when the dimensionality of the images increases (e.g., on Imagenet 128x128).

Decision: accepted-poster-papers