Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper aims to improve the kernel selection issue of the MMD-based generative models. The author formulates the kernels via inverse Fourier transform and the goal is to learn the optimal N finite random Fourier features (RFF). The RFF samples are optimized by the proposed kernel alignment loss where the positive and negative labels are defined as samples coming from real and negative data distributions, respectively. Some theoretical analysis regarding the consistency of the learned kernel is provided. Experiment results on the IS score and FID on CIFAR-10 show improvement of the proposed methods over MMD-GAN baselines, while the results are not comparable to the original MMD-GAN due to unknown results.

While motivated from the mean-field theory, the algorithm 1 is essentially doing stochastic gradient on the RFF samples with fixed learning rate. Learning spectral distribution of kernel via optimising RFF samples is also not entirely new, as [0] presented in the Appendix C4 of [1]. they show the difference between two different realization of kernel learning.

I would love to increase my score if the author could address the following comments:
(1) Can you explain why the IS and FID results of MMD-GAN presented in Table 1 is inconsistent (i.e. considerably worse) with other papers [1,2,3]?
(2) In experiment setting, the learning rate eta of learning RFF samples is fixed to 10. Does this guarantee the learned spectral distribution lying in the constraint set P as specify in Eq (8)?

[1] Implicit kernel learning, AISTATS 2019
[2] DEMYSTIFYING MMD GANS, ICLR 2018
[3] MMD GAN: Towards Deeper Understanding of Moment Matching Network, NIPS 2017

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The paper addresses the problem of kernel learning in MMD GAN using particle stochastic gradient descent to solve an approximation of the intractable distributional optimization problem for random features. The paper provides theoretical guarantees for the consistency of approximations, although proofs are deferred to Appendix.
It seems to be a good result theoretically thanks to the consistency guarantees for the particle SGD approximation of the optimization problem. However, its practical efficacy is not completely clear.
1. There is no discussion on how the method fares in terms of time/space complexity and if it is scalable to higher-dimensional datasets or larger batch sizes. How many steps T for good results are needed? How much time does it take to learn the model compared to the Implicit Kernel Learning or original MMD GAN?
2. For a more detailed analysis of performance, it would be helpful to see the benefits of the kernel learned with the proposed method on synthetic data and its performance on supervised learning tasks compared with other kernel learning methods on supervised tasks.

Some minor remarks:
1. Scaling parameter alpha has become parameter beta on page ix.
2. Some citations and equation references should be fixed.
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: This paper proposes to learn a kernel for training MMD-GAN by optimizing over the probability distribution that defines the kernel by means of random features. This is unlike the usual setting of MMD-GAN where the kernel is parametrized by composing a fixed top-kernel with a discriminator network that is optimized during training. The main motivation for this approach is to avoid having to 'manually' fix some parameters of the top-level kernel like the bandwidth of the RBF kernel. They provide an algorithm to achieve such optimization in probability space along with some consistency results and perform experiments on MNIST and Cifar10 to demonstrate empirically the advantage of such an approach over those that fix the top-level kernel.

Theory:
	Theorem 4.1 provides a convergence result of an oracle finite-sample estimator: that is the one obtained by exactly solving the optimization problem in 19b. In that case, they show the consistency of the proposed estimator. The result is somehow expected but the proof relies on nice duality results for measures and is very technical.	
The clarity of the proof could be improved:
- Currently, the structure of the main proof mixes direct lemma (lemma B.7) with less obvious ones (lemma B.6). Also, some concepts are introduced in the main proof but not necessary for its understanding: The notion of Orlicz norm is in introduced in Definition B.3 on the fly to state lemma B.6, but only equations 50 and 51 are used in this lemma which does not make use of the notion of Orlicz norm at all.

	Theorem 4.1 doesn't say anything about the consistency of the algorithm itself. To partly address this, the authors show in theorem 4.2 that as the number of particles grows the empirical process converges to a McKean Vlasov PDE (equation 22). This means that the proposed algorithm is approximating some gradient flow in metric space (25). 
- However, this gradient flow is a non-convex optimization problem and there is no guarantee that a global solution is reached.  Recent work provides cases when global convergence occurs [Chizat2018] but it is not the case in general. Some further clarification about the connection between Thm 4.1 and Thm 4.2 would be therefore useful.
	
Thm 4.2 is also curious in the sense that the process defined by equation 16, which is noisy since it relies on one sample from the data, would converge towards (22) which is a Mc-valsov equation with a drift only (no diffusion or other noise).  What happened to the noise coming from sampling from P_v  and P_w in equation 16? Wouldn't there be some sort of diffusion term as in [Hu2018]?
	
More generally it would be nice to have a discussion of the assumptions and results in the paper as they seem to rely on methods that people in the machine learning community are not totally familiar with.
	
Experiments:  The experiments are not convincing for several reasons:
 - The comparison with the other methods is somehow unfair since the bandwidth is manually tuned for the competing methods. It is easy to adaptively learn the bandwidth as well:  in this case, it will be just an additional parameter of a discriminator network. This was done in [Arbel2018] where a single gaussian kernel is used and a regularization of the critic allows to learn the bandwidth without manual tuning. Does the proposed method offer an additional advantage compared to those?
- In practice and for a scaling parameter alpha=1, isn't the algorithm strictly equivalent to considering an MMD-GAN with a dot product kernel and a discriminator given by the feature \phi(x,\zeta)?
- Mnist and Cifar10 are somehow very simple, what would happen on more complicated datasets (CelebA or imagenet)?

- I also think there is an ablation that is missing: If the auto-encoder also needs to be optimized then does it also help to optimize over the particles as well or is optimizing the auto-encoder discriminator enough to achieve a similar performance? In other words, does optimizing the auto-encoder compensate for the need to learn the distribution mu? Of course, this would depend on how the auto-encoder is parametrized but I don't see why it wouldn't in many cases.


Overall, I'm not convinced that the proposed approach would lead to any substantial improvement for MMD-GAN in practice, and the experiments are not really convincing as they are now. However, I find the theoretical results interesting and might be used to better analyse the dynamics of GAN's. But as the paper is currently framed, it is hard to put these theoretical results in perspective.

review_title: Official Blind Review #2

Meta-review: This paper was assessed by three reviewers who scored it as 6/1/6. The main criticism included somewhat weak experiments due to the manual tuning of bandwidth, the use of old (and perhaps mostly solved/not challenging) datasets such as Mnist and Cifar10, lack of ablation studies. The other issue voiced in the review is that the proposed method is very close to a MMD-GAN with a kernel plus random features. Taking into account all positives and negatives, we regret to conclude that this submission falls short of the quality required by ICLR2020, thus it cannot be accepted at this time.


Decision: Reject