MMD GAN with Random-Forest Kernels

Abstract: In this paper, we propose a novel kind of kernel, random forest kernel, to enhance the empirical performance of MMD GAN. Different from common forests with deterministic routings, a probabilistic routing variant is used in our innovated random-forest kernel, which is possible to merge with the CNN frameworks. Our proposed random-forest kernel has the following advantages: From the perspective of random forest, the output of GAN discriminator can be viewed as feature inputs to the forest, where each tree gets access to merely a fraction of the features, and thus the entire forest benefits from ensemble learning. In the aspect of kernel method, random-forest kernel is proved to be characteristic, and therefore suitable for the MMD structure. Besides, being an asymmetric kernel, our random-forest kernel is much more flexible, in terms of capturing the differences between distributions. Sharing the advantages of CNN, kernel method, and ensemble learning, our random-forest kernel based MMD GAN obtains desirable empirical performances on CIFAR-10, CelebA and LSUN bedroom data sets. Furthermore, for the sake of completeness, we also put forward comprehensive theoretical analysis to support our experimental results.

Reviews and author responses:

Theorem 2 and its proof are plagiarised: they are rephrased and reorganized formulation and proof of Theorem 1 of [1], while being presented as authors' own work. 

Although the assumptions are slightly different (random forest kernels vs general kernels), core of the proof is the same, including notation and its split into Lemmas and helper Theorems. In particular:
- formulation is the same (even use of MMD_u is copied, while not being defined before),
- main proof of Theorem 2 (p.25-26) is the proof of Corollary 3 of [1] followed by the proof of Theorem 5 of [1],
- Lemma 13 is Lemma 3 of [1],
- Definition 3 in Appendix B.2 is the same as Assumption D of [1] (Appendix C.2),
- Proposition 4 and it's proof (p. 21) are the same as Lemma 2 of [1].

[1] Mikołaj Bińkowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. International Conference on Learning Representations, 2018.

This paper proposed to utilize the random-forest kernel into MMD GAN.  The experiments are conducted on CIFAR-10, CelebA and LSUN datasets. 

The method is not novel. Both MMD GAN and the random-forest kernel have been well explored. Combining them together is considered as an extension. For the theory, the paper only provides the unbiasedness analysis. It is not clear to me whether this kernel is better than other MMD GAN variations. It is not clear how the claimed flexibility comes from. 

Regarding the experiments, it only compares with very basic baselines and the results are not significantly better. It would be better to include stronger baselines (Wang et al., 2019, Binkowski et al., 2018).

The writing of the paper is poor. with several typos. Moreover, as mentioned by reviewer #1,  theorem 2 and its proof are plagiarised. 

Overall, I think the paper is a clear reject. 

Overview: 
The paper propose an MMD GAN extension via using Random forest Kernel.  Instead of using Gaussian kernel on the top of the learned embeddings from the discriminator, it combines existing deep forests kernels. The theory of being differentiable is carefully studied (to prove zero measure) and the experiments are well conducted.  

1.  Some important  references are missing.  One very related paper is 

* Li et al., Implicit Kernel Learning,  AISTATS 2019. 

That paper is using the same idea to learn to manipulate the random features on the top of the learned embedding.  The main difference between it and the proposed algorithm is they use MLP parameterization instead of the tree-based model.    Also, the deep forest model can be treated as a sparse neural network, does it have more advantage over Li et al., (2019)? given they use simple dense MLP.   Please at least discuss the similarity and difference in the rebuttal and update the draft correspondingly.  I would even encourage the author to empirically compare with it in the camera ready version.  It would be interesting to see which parameterization is better in this space. 

There are also other recent MMD GAN extensions should be cited in the discussion, such as 
* On gradient regularizers for MMD GANs. 

2. For the theory part, based on Binkowski (2018), the gradients for the generator parameters should be biased. Could you discuss it with Theorem 2? 

3. For most MMD GAN results, one important property in Li et al., (2017),  Arbel et al., (2018) and Li et al., (2019) is weak* topology.  Does the proposed Random Forest MMD GAN also has that property? In Li et al., (2019), they need some condition to ensure that, how's case in the proposed algorithm?  

Meta-review: Reviewers raise the serious issue that the proof of Theorem 2 is plagiarized from Theorem 1 of "Demystifying MMD GANs" (https://arxiv.org/abs/1801.01401). With no response from the authors, this is a clear reject.


Decision: reject