Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper provides an interesting idea to use the optimized MMD for generative model evaluation and learning. Starting from the test power, the authors justified the criterion. Moreover, they also provided an efficient implementation of perturbation tests for empirical MMD. 

Pros:

1) The criterion is principled which is derived from the test power. 
2) The criterion can be used to detect the difference template by incorporating ARD technique. 
3) By exploiting kernel in the objective, the generated algorithm, t-GMMN, training can be improved from the GMMN.

Cons:

1) How to train the provided t objective is not clear. 
2) The algorithm is only tested on MNIST dataset as model criticism and learning objective. Comprehensive empirical comparison to the state-of-the-art criteria, e.g., log-likelihood, and other learning objectives is missing. 
review_title: interesting paper

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: A well written paper that proposes to use MMD to distinguish generated and reference data. The primary contribution of this paper is to derive a way to optimize the MMD kernels to maximize the test power of the two sample test. 

Pros

Principled approach; derivations start from first principles and the theoretical results will probably be applicable to other applications of two sample tests.

Well written; puts the contributions and related approaches into context and explains connections to previous work; especially to GANs.

Cons: I don’t expect that this work will have a big impact in the field:

The two sample test are still quadratic in the number of samples. 

Experiments only on toy data sets and on binarized MNIST

It would be interesting to know in what way this approach fails on e.g. image data (or other complex, high dimensional data where neural network generalize well). I could imagine that the neural network based discriminators in GANs generalize better than kernel based MMD methods. I would like to see follow up work that investigates this in more detail (and potentially profes my intuition wrong).

review_title: Interesting paper with a good theoretical section

final_score: rating: null
review_content: Thank you for this nice submission.

First, when you choose the kernel of MMD, you need to split your data into a "training sample" and a "test sample". How should split my data? Half and half? It seems that using a larger sample set will let me choose my kernel better, but I will loose power in the test sample. On the other hand, using a smaller training sample may result in a sub-optimal kernel choice, but will lead to higher power in the test sample.

Second, the paper mostly discusses MMD on raw pixels with the RBF kernel. How would these results translate into more complicated datasets, such as CIFAR, LSUN, CelebA, ImageNET...?

Third, I do not see any difference between Figures 4a, 4b, 4c, or any qualitative discussion in the text.
review_title: On training and testing, RBF kernels, and qualitative comparisons

final_score: rating: null
review_content: 1) It is known that the discretisation method (e.g. hard-thresholding vs. Bernoulli-sampling proportional to the grey level) has a big influence on the performance of generative models; at least when training latent variable models with MLL objective. What discretisation method did you use? Do you see different results for different methods? Why did you discretised at all? The kernels used in the experimental section should work with real valued input, shouldn’t they?

2) The section on MMD as a training criterion is very short. To me it is not obvious how training was performed (size of sample sets; joint learning of all MMD parameters? tuning of learning rates; etc.) Please elaborate.


review_title: Discretisation & MMD training

final_score: rating: 7: Good paper, accept
review_content: This is an interesting paper containing three contributions:

1) An expression for the variance of the quadratic-time MMD estimate, which can be efficiently minimized for kernel selection.

2) Advanced computational optimizations for permutation tests for the quadratic MMD statistic.

3) Crystal-clear examples on the importance of reducing the variance in two-sample testing (Figure 2)

Regarding my criticisms,

1) The conceptual advances in this submission are modest: the use of MMD to train generative models, as well as the importance of variance-reduction in MMD were already known.

2) The quadratic-time MMD test may lead to a discriminator that is "too good" in practical applications. Since MMD quickly picks up on pixel-level artifacts, I wonder if its use would be possible to train generators properly on realistic (non-binarized) data. This of course could be addressed by regularizing (smoothing) the kernel bandwidths, and for sure raises an interesting question/trade-off in generative modeling. 

Overall, the submission is technically sound and well-written: I recommend it for publication in ICLR 2017.
review_title: Good paper

Meta-review: This paper presents two ways that MMDs can be used to aid the GAN training framework. The relation to current literature is clearly explained, and the paper has illuminating side-experiments. The main con is that it's not clear if MMD-based training will be competitive in the long run with more flexible, but harder-to-use, neural network based approaches. However, this paper gives us a conceptual framework to evaluate new proposals for related GAN training procedures.
Decision: Accept (Poster)