Title: Generative Models and Model Criticism via Optimized Maximum Mean Discrepancy
Authors: Danica J. Sutherland
Hsiao-Yu Tung
Heiko Strathmann
Soumyajit De
Aaditya Ramdas
Alex Smola
Arthur Gretton
Venue: ICLR 2017
Keywords: Unsupervised Learning

Abstract: We propose a method to optimize the representation and distinguishability of samples from two probability distributions, by maximizing the estimated power of a statistical test based on the maximum mean discrepancy (MMD). This optimized MMD is applied to the setting of unsupervised learning by generative adversarial networks (GAN), in which a model attempts to generate realistic samples, and a discriminator attempts to tell these apart from data samples. In this context, the MMD may be used in two roles: first, as a discriminator, either directly on the samples, or on features of the samples. Second, the MMD can be used to evaluate the performance of a generative model, by testing the model’s samples against a reference data set. In the latter role, the optimized MMD is particularly helpful, as it gives an interpretable indication of how the model and data distributions differ, even in cases where individual model samples are not easily distinguished either by eye or by classifier.

TL;DR: A way to optimize the power of an MMD test, to use it for evaluating generative models and training GANs

Review discussion:
1) It is known that the discretisation method (e.g. hard-thresholding vs. Bernoulli-sampling proportional to the grey level) has a big influence on the performance of generative models;
at least when training latent variable models with MLL objective. What discretisation method did you use? Do you see different results for different methods? Why did you discretised at all? The kernels used in the experimental section should work with real valued input, shouldn’t they?
2) The section on MMD as a training criterion is very short. To me it is not obvious how training was performed (size of sample sets; joint learning of all MMD parameters? tuning of learning rates; etc.) Please elaborate.
Thanks for the questions.
1) Yes, our methods work with continuous values. We discretized the images for the test because otherwise the task is too trivially easy: looking at just marginal pixel values, a very small-bandwidth Gaussian kernel would immediately be able to distinguish the GAN pixel values (which are continuous but unlikely to hit exact 0s or 1s) from the MNIST pixel values (which are discretized into 256 discrete values).
In fact, the histograms of pixel values for true and generated models are noticeably distinct even rounded to the nearest decimal. We used hard thresholds; in this case, it seems unlikely that proportional sampling would be significantly different, but we will run it to make sure the results are similar.
2) We can add some more details on the exact training procedure to the paper. Full details are available in our code: https://github.com/dougalsutherland/opt-mmd
Thank you for this nice submission.
First, when you choose the kernel of MMD, you need to split your data into a "training sample" and a "test sample". How should split my data? Half and half? It seems that using a larger sample set will let me choose my kernel better, but I will loose power in the test sample.
On the other hand, using a smaller training sample may result in a sub-optimal kernel choice, but will lead to higher power in the test sample.
Second, the paper mostly discusses MMD on raw pixels with the RBF kernel. How would these results translate into more complicated datasets, such as CIFAR, LSUN, CelebA, ImageNET...?
Third, I do not see any difference between Figures 4a, 4b, 4c, or any qualitative discussion in the text.
Thank you for your questions.
In this submission, we used a 50-50 split for training and testing. In general, this tradeoff is hard to quantify; (4) provides a partial picture in terms of its dependence on m. In the generative model criticism case, we can take as many samples from the generated model as needed, and so available sample sizes will often be relatively large, making careful consideration less necessary.
These methods are certainly applicable to more complex datasets, using e.g. convolutional networks, with the t statistic as a loss. We ran some initial experiments on the Salimans et al. models for CIFAR-10 and found that the model samples were in fact easy to distinguish even with median-heuristic RBF kernels, so that our machinery was not needed for powerful tests. Presumably this would be true for current GAN-type models on more complex datasets as well, but our methods should apply to tests on better models in those settings (or to other testing problems on images).
In terms of the GAN sample qualities: to our eyes, it seems that the feature matching-based samples of 4(c) are somewhat better than those of (a) and (b). The GMMN and t-GMMN results are more similar. It seems that, at least on MNIST with this particular fixed kernel, direct classification loss gives a better distribution to match against than the raw image structure. This demonstrates that adaptivity in the matching is helpful.
Our initial attempts to directly optimize the kernel resulted in instability in training. We conjecture, inspired by the simultaneous work of Arjovksy and Bottou (https://openreview.net/forum?id=Hk4_qw5xe), that this is because the two-sample test discriminator becomes too powerful in distinguishing the generator and true data manifolds. With a fixed kernel, this effect is avoided.
This is an interesting paper containing three contributions:
1) An expression for the variance of the quadratic-time MMD estimate, which can be efficiently minimized for kernel selection.
2) Advanced computational optimizations for permutation tests for the quadratic MMD statistic.
3) Crystal-clear examples on the importance of reducing the variance in two-sample testing (Figure 2)
Regarding my criticisms,
1) The conceptual advances in this submission are modest: the use of MMD to train generative models, as well as the importance of variance-reduction in MMD were already known.
2) The quadratic-time MMD test may lead to a discriminator that is "too good" in practical applications. Since MMD quickly picks up on pixel-level artifacts, I wonder if its use would be possible to train generators properly on realistic (non-binarized) data. This of course could be addressed by regularizing (smoothing) the kernel bandwidths, and for sure raises an interesting question/trade-off in generative modeling.
Overall, the submission is technically sound and well-written: I recommend it for publication in ICLR 2017.
Thank you for your helpful comments. Responses to a few points follow:
“1) the use of MMD to train generative models, as well as the importance of variance-reduction in MMD were already known.”
The test power maximization strategy for the quadratic-time MMD is a new result, although a related result for a linear-time MMD statistic was published in Gretton et al. (NIPS 2012). The argument for the quadratic-time MMD presented in the present paper is a little more delicate, due to the different forms of the null and alternative distributions of the quadratic MMD estimate, and the far more complex expression for the variance.
It’s correct that the MMD is established in training generative models (e.g., we reference the generative moment matching networks of Li et al. 2015 and the related model of Dziugaite et al. 2015). Our main purpose in this paper was rather to reveal statistically significant and interpretable differences in distributions of the trained models from  that of the reference data set.
That said, we have also proposed some additional strategies when using MMD to train GANs: for instance, using a sum of kernels with different bandwidths in our MMD, which helps ensure at least one of the kernel bandwidths in our “dictionary” gives a good gradient signal to the generator, and using an MMD defined on discriminator features as the signal to our generator.
2)  "I wonder if its use would be possible to train generators properly on realistic (non-binarized) data. This of course could be addressed by regularizing (smoothing) the kernel bandwidths ..."
We agree that training a GAN (which gives continuous outputs) against the MNIST reference data (which is quantized into 256 bins) would benefit from using a broader kernel bandwidth which might "blur over" this quantization effect. It then becomes an interesting question as to how to choose the best kernel to control for perceptually relevant differences, and to ignore differences in distribution due to quantization/pixel level artifacts.
It is also perhaps worth clarifying that we binarized the MNIST digits only for the model criticism experiments;
when using MMD as a GAN objective, we used the original digits.
A well written paper that proposes to use MMD to distinguish generated and reference data. The primary contribution of this paper is to derive a way to optimize the MMD kernels to maximize the test power of the two sample test.
Pros
Principled approach; derivations start from first principles and the theoretical results will probably be applicable to other applications of two sample tests.
Well written; puts the contributions and related approaches into context and explains connections to previous work; especially to GANs.
Cons: I don’t expect that this work will have a big impact in the field:
The two sample test are still quadratic in the number of samples.
Experiments only on toy data sets and on binarized MNIST
It would be interesting to know in what way this approach fails on e.g. image data (or other complex, high dimensional data where neural network generalize well). I could imagine that the neural network based discriminators in GANs generalize better than kernel based MMD methods. I would like to see follow up work that investigates this in more detail (and potentially profes my intuition wrong).
Thank you for your helpful comments. We'll briefly respond to a couple of points:
“The two sample test are still quadratic in the number of samples. “
In respect of computational cost: this is less of an issue when using MMD to reveal statistically significant differences in the distribution of GAN and reference digits, since such an evaluation occurs as a one-off step after the GAN is trained.  In training, the situation is less severe than one might expect, since we use minibatches, which results in speeds comparable to regular GANs (bearing in mind that the discriminator requires no training in the GMMN cases).
“It would be interesting to know in what way this approach fails on e.g. image data (or other complex, high dimensional data where neural network generalize well). I could imagine that the neural network based discriminators in GANs generalize better than kernel based MMD methods. I would like to see follow up work that investigates this in more detail (and potentially profes my intuition wrong).”
Yes, we agree this is the next step. Note that we already have looked at using MMD in training using discriminator features, however a very interesting direction would be to use MMD defined on convolutional features to troubleshoot the output of a GAN, and to reveal interpretable differences between more complex generated data and reference images.
This paper provides an interesting idea to use the optimized MMD for generative model evaluation and learning. Starting from the test power, the authors justified the criterion.
Moreover, they also provided an efficient implementation of perturbation tests for empirical MMD.
Pros:
1) The criterion is principled which is derived from the test power.
2) The criterion can be used to detect the difference template by incorporating ARD technique.
3) By exploiting kernel in the objective, the generated algorithm, t-GMMN, training can be improved from the GMMN.
Cons:
1) How to train the provided t objective is not clear.
2) The algorithm is only tested on MNIST dataset as model criticism and learning objective. Comprehensive empirical comparison to the state-of-the-art criteria, e.g., log-likelihood, and other learning objectives is missing.
Thank you for your comments. A few notes:
"How to train the provided t objective is not clear."
The estimator, which is given by dividing (2) by the square root of (5), is differentiable with respect to the kernel, and indeed can be easily handled by automatic differentiation systems such as Theano and TensorFlow. We then merely need to use it as a loss either at the end of an ARD-computing network (for the model criticism experiments) or within a GAN framework, and optimize with standard gradient methods. Implementations of the estimator in both Theano and TensorFlow are provided at https://github.com/dougalsutherland/opt-mmd, which also demonstrates its use in both setups.
"The algorithm is only tested on MNIST dataset as model criticism and learning objective. Comprehensive empirical comparison to the state-of-the-art criteria, e.g., log-likelihood, and other learning objectives is missing."
Our paper proposes a method for model criticism and a new training objective in the GAN setting, where true likelihoods are intractable; to compare to log-likelihoods, we would need to instead use models where those are evaluable.
Additionally, as pointed out by Theis et al. (ICLR 2016), log-likelihood in fact is nearly independent of sample quality. As they demonstrate, models can have excellent-looking samples with terrible log-likelihoods, or very good log-likelihoods with terrible samples; log-likelihood is a useful measure for understanding the quality of models for some applications, but is not very useful for measuring sample quality. MMD and the related measures we consider, by contrast, are entirely sample-based and model-independent; they are good for understanding the quality of samples from a model, but not for directly understanding the quality of a model for e.g. regression purposes. Our MMD-based measures thus help in providing a quantitative measure of something akin to visual sample quality, rather than the very different log-likelihood criterion generally considered in previous evaluations of generative models.
Hello to posterity,
We've just uploaded a revision fixing some constants in the variance estimator; the former version was accidentally slightly biased.

Meta-review:
role: PC
data: comment: This paper presents two ways that MMDs can be used to aid the GAN training framework. The relation to current literature is clearly explained, and the paper has illuminating side-experiments. The main con is that it's not clear if MMD-based training will be competitive in the long run with more flexible, but harder-to-use, neural network based approaches. However, this paper gives us a conceptual framework to evaluate new proposals for related GAN training procedures.

Decision: Accept (Poster)