MMD GAN with Random-Forest Kernels
Authors: Tao Huang
Zhen Han
Xu Jia
Hanyuan Hang
Venue:  (2020)
Keywords: GANs
MMD
kernel
random forest
unbiased gradients

Abstract: In this paper, we propose a novel kind of kernel, random forest kernel, to enhance the empirical performance of MMD GAN. Different from common forests with deterministic routings, a probabilistic routing variant is used in our innovated random-forest kernel, which is possible to merge with the CNN frameworks. Our proposed random-forest kernel has the following advantages: From the perspective of random forest, the output of GAN discriminator can be viewed as feature inputs to the forest, where each tree gets access to merely a fraction of the features, and thus the entire forest benefits from ensemble learning. In the aspect of kernel method, random-forest kernel is proved to be characteristic, and therefore suitable for the MMD structure. Besides, being an asymmetric kernel, our random-forest kernel is much more flexible, in terms of capturing the differences between distributions. Sharing the advantages of CNN, kernel method, and ensemble learning, our random-forest kernel based MMD GAN obtains desirable empirical performances on CIFAR-10, CelebA and LSUN bedroom data sets. Furthermore, for the sake of completeness, we also put forward comprehensive theoretical analysis to support our experimental results.

Under review conference paper at ICLR 2020
MMD GAN WITH RANDOM-FOREST KERNELS CONFERENCE SUBMISSIONS
Anonymous authors
Paper under double-blind review
ABSTRACT
In this paper, we propose novel kind of kernel, random forest kernel, to hance the empirical performance of MMD GAN. Different from common forests with deterministic routings, a probabílistic routing variant used in our innovated random-foree kernel, which is possible to merge with the CNN frameworks. Our proposed random- forest kernel has the following advantages: From the perspec- tive of random forest, the output of GAN discriminator can be viewed as feature inputs to the forest, where each tree gets access to merely a fraction of the features, and thus the entire forest benefits from ensemble learning. In the aspect of kernel method, random-forest kernel is proved to be characteristic, and therefore suitable for the MMD structure. Besides, being an asymmetric kernel, random- forest kernel is much more flexible, terms of capturing the differences between distri- butions. Sharing the advantages of CNN, kernel method, and ensemble learning, our random-fore -orestt kernel based MMD GAN obtains desirable empirical perfor- mances CIFAR-IO, CelebA and LSUN bedroom data sets. Furthermore, for the sake of completeness, we also put forward comprehensive theoretical analysis to support our experimental results.
INTRODUCTION
Generative adversarial nets (GANs; Goodfellow et al., 2014) are well-known generative models, which largely attribute to the sophisticated design of a generator and a discriminator which are trained jointly in an adversarial fashion. Nowadays GANs are intensely used in a variety of practical tasks, such as image-to-image translation (Tang et al., 2019; Mo et al., 2019); 3D reconstruction (Gecer et al., 2019); video prediction (Kwon & Park, 2019); text-to-image generation (Zhu et al., 2019); just to name a few.
However, it's well-known that the training of GANs is little tricky, e.g. (Salimans et al., 2016). One reason of instability of GAN training lies the distance used in discriminator to measure the divergence between the generated distribution and the target distribution. For instance, concerning with the Jensen-Shannon divergence based GANs proposed in Goodfellow et al. (2014), Arjovsky & Bottou (2017) points out that if the generated distribution and the target distribution are supported on manifolds where the measure of intersection is zero, the KL divergences be infinite. Consequently, the generator fails obtain enough useful gradient to update, which undermines GAN training. Moreover, two non-overlapping distributions may be judged be quite different by the densen-Shannow divergence, even if they are nearby with high probability.
divergence will be constant and
As result, to better measure the difference between two distributions, Integral Probability Metrics (IPM) based GANs have been proposed. For instance, Arjovsky et al. (2017) utilizes Wasserstein distance in GAN discriminator, while Li et al. (2017) adopts maximum mean discrepancy (MMD), managing to project and discriminate data in reproducing kernel Hilbert space (RKHS). To mention, the RKHS with characteristic kernels including Gaussian RBF kernel (Li et al., 2017) and rational quadratic kernel (Binkowski al., 2018) has strong power in the discrimination of two distributions, see e.g. (Sriperumbudur et al., 2010).
In this paper, inspired by non-line monelinear discriminating power of decision forests, we propose new type of kernel named random-fore forest kernel to improve the performance of MMD GAN discriminator. In order to fit with back-propagation training procedure, borrow the decision forest model with

---

Under review as a conference paper at ICLR 2020
stochastic and differentiable decision trees from Kontschieder et al. (2015) in our random-forest kernel. To be specific, each dimension of the GAN discriminator outputs is randomly connected to one internal node of soft decision forest, serving as the candidate to-be-split dimension. Then, the tree is split with a soft decision function through a probabilistic routing. Other than the typical decision forest used in classification tasks where the value of each leaf node is a label, the leaf value of our random forest is the probability of sample x; falling into certain leaf node of the forest. If the output of the discriminator is denoted as how (2;) and the probability output of the t-th tree is denoted as μ² (hax (z:): FF), the random forest kernel RRF can be formulated as
KRE(Zi, Izj 0F) 0F)
+ Σ t=1
(u' (1s); (F)); (haw (e));OF))
where T is the total number of trees in the forest, ON and 0p denote the parameters of the GAN discriminator and the random forest respectively.
Recall that random forest and deep neural networks are first combined in Kontschieder al. (2015), where differentiable decision tree model and deep convolutional networks are trained together in an end-to-end manner to solve classification tasks. Then, Shen et al. (2017) extends the idea to label distribution learning, and Shen et al. (2018) makes further extensions in regression regime. Moreover, Zuo & Drummond (2017). Zuo et al. (2018) and Avraham et al. (2019) also introduce deep decision forests. Apart from the typical ensemble method that averages the results across trees, they aggregate the results by multiplication. As for the combination of random forest and GAN, Zuo et al. (2018) introduce forests structure in GAN discriminator, combining CNN network and forest as a composited classifier, while Avraham et al. (2019) uses forest structure as one of non-linear mapping functions in regularization part.
On the other hand, in the aspect of relationship between random forest and kernel method, Breiman (2000) initiates the literature concerning the link. He shows the fact that a purely random tree partition is equivalent to a kernel acting on the true margin, of which form can be viewed as the probability of two samples falling into the same terminal node. Shen & Vogelstein (2018) proves that random forest kernel is characteristic. Some more theoretical analysis can be found in Davies & Ghahramani (2014), Arlot & Genuer (2014), Scornet (2016). However, despite their theoretical breakthroughs, forest decision functions used in these forest kernels are non-differentiabble hard margins rather than differentiable soft ones, and thus cannot be directly used in back propagation regime.
To the best of our knowledge, MMD GAN with our proposed random-forest kernel is the first to combine random forest with deep neural network in the form of kernel MMD GAN. Through the- oretical analysis and numerical experiments, we evaluate the effectiveness of MMD GAN with our modoo--oreett kernel. From the theoretical point of view, our random-forest kernel enjoys the prop- erty of being characteristic, and the gradient estimators used in the training process of random-forest kernel GAN are unbiased. In numerical experiments, we evaluate our random-forest kernel under the setting of both the original MMD GAN (Li et al., 2017) and the one with repulsive loss (Wang et al., 2019). Besides, we also compare our random- forest kernel with Gaussian RBF kernel (Li et al., 2017), rational quadratic kernel (Biikowski et al., 2018), and bounded RBF kernel (Wang et al., 2019). As a result, MMD GAN with our random- forest kernel outperforms its counterparts with respect to both accuracy and training stability.
This paper organized as follows. First of all, we introduce some preliminaries of MMD GAN in Section 2. Then we review the concept of deep random forest and show how it is embedded within a CNN in 3.1. After that, random-forest kernels and MMD GAN with random-forest kernels are proposed in 3.2 and 3.3 respectively. Besides, the training techniques of MMD GAN with uundom-forest kernel are demonstrated in Section 3.4 and the theoretical results are shown in Section 3.5. Eventually, Section 4 presents the experimental setups and results, including the comparison between our proposed random-forest kernel and other kernels. In addition, all detailed theoretical proofs are included in the Appendices.
2 MMD GAN
GANs are recently introduced as a novel way of training a generative model. To learn a distribution Px x. we build an adversarial model composed of two parts, the generator G and the discriminator D.
2

---

Under review conference paper at ICLR 2020
The generative model captures the data distribution Px. , by building a mapping function G 2 from prior noise distribution Pz to data space. While the discriminative model D) X R is used distinguish generated distribution Py from real data distribution Px-
Taking X' ~ Px and Y, Y' ~ Py == Pc(Z) where Y := G(Z) and Y' = G(Z)). the squared MMD expressed as
MMD Py] Ex xx,x - 2Ex, 2 + By, Y'). min
The loss generator and discriminator in MMD GAN proposed Li al. (2017) is:
min La MMD3 Pyl. G
Ln
P
Wang et al. (2019) proposed MMD GAN with repulsive loss, where the objective functions for G and D are:
min LG = (M, min LD Ex Ex -y,,) D
Given i.d. samples Dx f.1...., Fnf Px and Dy (G(z)) ..... G(em)) Py, we can write an unbiased estimator of the squared MMD in terms of as
MMD', Dyl
n(n 1) i=1 jitt
m
2 mum
k(Ui,
M( Uj).
m(m 1) 1=1 jpti
When & is characteristic kernel, we have Px, 0 with equality applies if and only Px -yy. The best-known characteristic kernels are gaussian RBF kernel and rational quadratic kernel (Binkowski et al., 2018).
3 RANDOM-FOREST KERNEL
In this section, we review a stochastic and differentiable variant random forest and how it is embedded within deep convolutional neural network proposed in Kontschieder al. (2015). Then we propose random-forest kernel and we apply it in MMD GAN. We illustrate the advantages of our random-foree kernel, show the training technique of MMD GAN with random-forte kernel, and study its theoretical properties.
3.1 DEEP RANDOM FOREST
random trees. For the 1-th tree the forest, t E (1.....T), we denote N₁ := {d' Jj=1 as the set of its internal nodes and if T trees have the same structure, then we have [W] [W], see Figure 1. Furthermore, we denote L. the set of its leaf nodes and 0% as the parameters of the 1-th tree.
Suppose that random forest consists of [GG]
Here we introduce the routing function pz; 0%p) which indicates the probability of the sample falling into the f-th leaf node of the 1-th tree. In order to provide an explicit form for the routing function ( 0f-), e.g. the thick black line in Figure I, we introduce the following binary relations that depend on the tree structure: / d; is true, if belongs to the left subtree of node dy and > di is true, function of the j-th internal node in the t-th tree, that is the probability of the sample falling into the left child of node de in the 1-th tree. Then, pf can be expressed by these relations as
belongs to the right subtree of node d'. Moreover, let (x; ; 08.)) be the decision
(1(2; 0z)
II () 0% 1ec ( (1-)((10
djER
where R4 denotes the unique path from node 1 to node & of the 1-th tree.

---

Under review conference paper at ICLR 2020
--1
1+1
t
d
d2
d3
d4
ds
dB
d'
L1
l2
L3
L
LB
L
LB
Figure Example of a random forest with 7 trees: blue nodes denote the internal nodes N {d1, ..., dr} while purple nodes are the leaf nodes C := {f1, ....... 6.} J. The black thick path illustrates the route of sample falling into the é leaf node of the 1-th tree.
Now, us derive the explicit form of the decision function p (1;; 0z)) Here, to utilize the power of deep learning, consider using the convolutional neural network to construct decision functions of random forest.
To be specific, given the parameter ON trained from CNN network, we denote h(:0x) as the d'-dimension output of convolutional neural network, which is the unit the last fully- connected layer in the CNN, and h- (N) is the i-th element of the CNN output. We denote {1. (1....,T N[} + (1.....d'} as the connection function, which represents the connec- tion between the internal node d'i and the former CNN output h; Note that during the whole training process, the form of the connection function C is not changed and every internal node dy is randomly assigned to an element he(r(e=1)+)) (x)) = (1+e as the decision function, and the parameters the i-th tree 0f == (w², b²) (6f....,b(N) (W)) then the decision functions p delivering
If we choose the sigmoid functioon
with w' = (wi.... w(wi) and b² stochastic routing be defined as
() ON 03p) = (wy heuree + b2).
For example, we have the probability (2; ON 03) ow 0N) + 6}) for the node in the first layer, see Figure 2 for explicit example of the random forest.
d'-dim h 0,)
h
h
di
T trees with depth 2
d
Figure 2: Example of connection between CNN network dimeensioonl output h(:; 0x) and random forest with T trees and depth 2. The internal node d: in the left tree has the probability (x; 0x 0%) a (wshhi (2; 0x) b)) and therefore the routing function of Icaf node é) is M4, (z: 0N 0+)
a | ha (x; 0x) + b1) awshh (z;; 0N) + b5))-
4

---

Under review conference paper at ICLR 2020
Every leaf node in each tree has a unique road R2 from node 1 node & with length |Ry] ( Then, for the every leaf node l of the 1-th tree, we have
p((x) = II (wigherre-Dns)t + + lec (1- oluhhhorr- (x) b5)) 1sess
JER:
(3)
where Lf denotes the set of all left son nodes of its father node.
3.2 RANDOM-FOREST KERNEL
Here, we propose the random-fort forest kernel as follows:
Definition (Random-Forre Forest Kernel) Let æ, be a pair of kernel input, let 0z. = (w", b²) denotes the weights and bias of the t-th tree of the random forest, and 0p := (0%))==- The random-forest kernel can be defined as
EgF x. y: (x:
Σ (1)
EEC.
1
Σ (4 p(: (z)
T
(x;(x;))) (v:
where Li denotes the set of leaf nodes the 1-th tree, μ (ul)eec, and (() (10))
3.3 MMD GAN WITH RANDOM FORESTT KERNELS
We write
k(x,x') = knr (how (r),ho,, (r)
and introduce the objective functions of MMD GAN with random-foree kernel by min LG = MMD Pr]
= Ex x') -Xx,yx(XXX)) +x,yy,(Y,,
min LD = -MDD P +R
0x.0p
-x.x X') + 2Ex,Y ( -yy "K,Y"'+7 +R,
where y = (2), noise vector, and is the regularizer of random-fored foorest kernel (the detail shown in Section 3.4). In addition, the objective functions of MMD GAN with repulsive loss are
min LG 2MD Pxx Pyl
Ex X1X X') - 2E 2Ex,y Ey,y:
min Ln (MM P +
0x
Ex X') Ey,r R.
Random-forest kernel MMD GAN enjoys the following advantages:
Our proposed random-forest kernel used in MMD GAN benefits from ensemble learning. From the perspective of random forest, the output of MMD GAN discriminator (x:0xx) can be viewed feature inputs to the forest. To mention, each tree only gets access to merely fraction of the features by random connection functions, and thus the entire forest benefits from ensemble learning.
5

---

Under review conference paper at ICLR 2020
Our random-forest kernel MMD GAN enjoys the advantages three powerful discrimi- native methods, which are CNN, kernel method, and ensemble learning. To be specific, CNN is good extracting useful features from images; Kernel method utilize RKHS for discrimination; Ensemble learning utilizes the power of randomness and ensemble.
Our proposed random-forest kernel has some good theoretical properties. one aspect, random-forest kernel is proved to be characteristic in Shen & Vogelstein (2018). In another, in Section 3.5, the unbiasedness of the gradients of MMD GAN with random-forest kernel is proved.
3.4 RANDOM-FOREST KERNEL. TRAINING TECHNIQUE
In Frosst & Hinton (2017), the authors mention that the tree may get stuck on plateaus if internal nodes always assign the most probability to one of its subtree. The gradients will vanish because the gradients of the logistic-type decision function will be very closed to zero. In order to stabi- lize the training of random-fore forest kernel and avoid the stuck on bad solutions, we add penalty that encourage each internal node to split in balanced style as Frosst & Hinton (2017) does, that is, we penalize the cross entropy between the desired 0.5, 0.5 average probabilities of falling into two subtrees and the actual average probability 1
The actual average probability of the i-th internal node ax is
Ezen (1)) ExenPi(s
where Pi (x) the routing probability x from root node to internal node p;()) ) is the probability of x falling into the left subtree of the i-th internal node, and Ω is the collection of mini-batch samples.
Then, the formulation of the regularizer is:
R(2) = - Σ
0.5 lo5 (o:) + 0.5 loglog (1 a0)),
*Enneeraal Nodes
where À is exponentially decayed with the depth of d of the internal node by multiplying the coeffi- cient 2 2- for the intuition that less balanced split in deeper internal node may increase the non-linear discrimination power.
When training aadom-foress kernel, mini-batch of real samples and generated pictures Y both fed into the discriminator, and then &(X, XX) A X, Y) and k(Y, Y) are calculated, where k == RR o h(: vy). Here, notify, we find that the in the regularizer formulation does matter in forest-kernel setting. i's better to calculate a and R(2) in the case of Ω X, Ω Y, Ω XUY respectively, and then sum up three parts of regularizer as final regularizer R.
Therefore, the formulation of regularizer R. added in the training of random-forest kernel is
R = R(X) + 2 R(X U UY) + R(Y)
(4)
3.5 THEORETICAL RESULTS
In this subsection, we present our main theoretical results.
Theorem (Unbiasedness) Let X be the true data on X with the distribution Px and Z be the noise on Z with the distribution Pz satisfying Epx I| < oo and 1Z 3o for some a 2 1. Moreover, let G 2 - X be a generator network, hey X - IRd be a discriminator network, kar be the random-forest kernel, and OD (0x, 0r) be the parameters of the GAN discriminator. Then, for y-almost all OD Rloo and 1b ERIV there holds
EDX~Pa. DD D2-P2 [0p MMD2 (hay (Dx), hay (Dz))) D. MMD? (Px), (G (P2)))
In other words, during the training process of Random-Forest Kernel MMD GAN, the estimated gradients of MMD with respect to the parameters eb and AD are unbiased, that is, the expectation and the differential exchangeable.

---

Under review conference paper at ICLR 2020
4 EXPERIMENTS
In this section, evaluate our proposed random-forest kernel in the setting of MMD GAN in (Li et al., 2017) and the MMD GAN with repulsive loss (Wang et al., 2019). To illustrate the efficacy of our random-foree forest kemnll, compare 2017), rational quadratic kernel (Binkowski et al., 2018) in the setting of the original MMD GAN loss, and compare random-forte kernel with bounded Gaussian kernel (Wang et al., 2019) in the setting of MMD GAN with repulsive loss.
undom-foress kernel with Gaussian kemel (Li et al.,
4.1 EXPERIMENTAL SETUP
4.1.1 DATASETS
The experiments are evaluated on three benchmark datasets:
(1) the Cifarl0 dataset of 32 32 pictures (Krizhevsky et al., 2009);
(2) the LSUN dataset of bedroom pictures resized to 64 x 64 (Yu et al., 2015);
(3) the CelebA dataset of celebrity faces pictures randomly cropped and resized to 160 x 160 (Liu al., 2015). The images are scaled to range [0, 1].
4.1.2 KERNELS AND LOSSES
Under the setting of MMD GAN loss proposed in Li al. (2017), compare proposed random- forest kernel with the Gaussian REF kernel (Li et al., 2017) with mixture of kernel scales o and the Rational quadratic kernel (Biníkowsii al., 2018) with mixture of kernel scales a. Moreover, under another setting of MMD GAN with repulsive loss proposed in Wang et al. (2019), we compare our random-forest kernel with the bounded RBF kernel (Wang al., 2019).
134
13
1.9
435
23
128
23
(a) Gaussian RBF kernel with mixture of kernel scales
(b) Rational quadratic kernel with mixture kernel scales a
(c) Random-forest kemel with T = 10 and dep
Figure 3: Visualization of three kernels by drawing filled contours: (a) & (b) A direct visualization of 2-dimensional kernels with reference (); (c) A 2-dimensional visualization of the multi- dimensional random-fores forest kernel with the help of t-SNE (Maaten Hinton, 2008). The details of visualization are shown in Appendix A.2.
As is shown in Figure 3, the shapes of the Gaussian RBF kernel and the rational quadratic kernel are both symmetric. However, the local structure of random-foress kernel (w.rtt reference points except 70-dimensional zero vector) is asymmetric and very complex. The asymmetry and complexity of random-forest kernel may be helpful to discriminate two distributions in MMD GAN training.
4.1.3 NETWORK ARCHITECTURE
For dataset Cifar10 and dataset LSUN bedroom, DCGAN (Radford et al., 2016) architecture with hyper parameters from Miyato et al. (2018) is used for both generator and discriminator; and for dataset Celebl. we use a 5-layer DCGAN discriminator and a 10-layer ResNet generator. Further details of the network architecture batch normalization (loffe & Szegedy, 2015) is used in the generator and spectral normalization (Miyato et al., 2018) is used in the discriminator. The hyper-parameter details of kernels used in
given in Appendix A.3. We mention that in all experiments,

---

Under review conference paper at ICLR 2020
Table 1: Score evaluation results for three datasets: Inception Score (IS). Fréchet Inception Distance (FID) and Kernel Inception Distance (KID). The score within two setups are marked bold
Kernel
Loss
Dimension of Output Layer
CIFAR-10
ColebA
LSUN
KID x10
x10
FID
x10
Real data
11.28
0.217
0.00
0.589
0.00
0.670
0.00
mix-rbf
mmd
6.91
28.99
20.92
22.08
14.07
68.36
41,67
mix-rq -r
mind
7.12
28.23
20.13
34.02
16.96
82.08
56.41
mix thf
mmnd
6.96
27.56
19.51
21.64
14.39
65,48
40.34
mix-rq
mmd
7.19
27.68
18.88
29.78
11.91
78.89
48.82
forest
7.16
24.74
17.29
25.07
24.99
24.35
ibf-b rbf-b mmd-rep
mand-rep
7.37
23.54
17.27
27.81
15.53
17.99
16.04
7.60
24.80
16.38
30.82
18:36
18.28
16.57
forest
mand-rep
70
7.42
22.32
15.46
21.11
14.29
17.60
16.94
the discriminator shown in Appendix A.1. For the sake of comparison with forest kernel, the dimension of discriminator output layer & set to be 70 for random-forest kernel and to be 16 for other kernels following the previous setting of Binkowski al. (2018); Wang et al. (2019).
4.1.4 TRAINING HYPER PIEER-PARAMEEERS
We set the initial learning rate 10-4 and decrease the learning rate by coefficient 0.8 in iteration 30000, 60000, 90000, and 120000. Adam optimizer (Kingma & Ba, 2015) used with momentum parameters 31 0.5 and f2 0.999. The batch size of each model is 64. All models trained for 150000 iterations on CIFAR- -RRR-10,0 10, CelebA, and LSUN bedroom datasets, with five discriminator updates per generator update.
4.1.5 EVALUATION METRICS
The following three metrics used for quantitative evaluation: Inception score (IS) (Salimans et al., 2016), Fréchet inception distance (FID) (Heusel et al., 2017), and Kernel inception distance (KID) (Bińkkowski al., 2018). In general, higher IS and Lower FID, KID means better quality. However, outside the dataset Imagenet, the metric IS has some problem, especially for datasets celebA and LSUN bedroom. Therefore, for inception score, only report the inception CIFAR-10. Quantitative scores are calculated based on 50000 generator samples and 50000 real samples.
of
4.2 EXPERIMENTAL RESULTS
We compare proposed random-ford forest kernel with mix-rbf kernel and mix-rq kernel in the setting of the MMD GAN loss, and compare our proposed random-forest kernel with rbf-b kernel in the setting with MMD GAN repulsive loss. The Inception Score, the Fréchet Inception Distance and the Kernel Inception Distance of applying different kernels and different loss functions on three benchmark datasets shown in table 1.
We find that, in the perspective of the original MMD GAN loss, our newly proposed random-foredt kernel shows better performance than the mix-rbf kernel and the mix-rq kernel in CIFAR-11 dataset and LSUN bedroom dataset; and in the perspective of the repulsive loss, the performance of our newly proposed random-foo forest kernel is comparable or better than the rbf-b kernel. The efficacy of our newly proposed random-forest kernel is shown under the setting of both MMD GAN loss and MMD GAN repulsive loss.
Some randomly generated pictures of model learned with various kemels and two different loss functions are visualized in Appendix D.
8

---

Under review conference paper at ICLR 2020
REFERENCES
Martin Arjovsky and Leon Bottou. Towards principled methods for training generative adversarial networks, In International Conference on Learning Representations, 2017.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference machine learning, pp. 214-223, 2017.
Sylvain Arlot and Robin Genuer. Analysis of purely random forests bias. arXiv preprint arXiv:1407.3933 393,9 2014.
Gil Avraham, Yan Zuo, and Tom Drummond. Parallel optimal transport gan. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 441444444444 1-4400, 2019.
Mikolaj Binkowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In International Conference Learning Representations, 2018.
Leo Breiman. Some infinity theory for predictor ensembles. Technical report, Technical Report 579, Statistics Dept. UCB, 2000.
Djork-Aréé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (ELUs). In International Conference on Learning Represen- tations, pp. 1 1-14, 2016.
Alex Davies and Zoubin Ghahramani. The random forest kernel and other kernels for big data from random partitions. arXiv preprint arXiv:1402 4293, 2014.
Gerald B. Folland. Real analysis. John Wiley & Sons, Inc., New York, second edition, 1999.
Nicholas Frosst and Geoffrey Hinton. Distilling neural network into soft decision tree. arXiv preprint arXiv: 1111/7111100777 1,09884, 2017.
Baris Gecer, Stylianos Ploumpis. Irene Kotsia, and Stefanos Zafeiriou. Ganfit: Generative adversar- ial network fitting for high fidelity 3d face reconstruction. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1155-1164, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp.
2014.
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13:723-773, 2012.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. In Advances in neural information processing systems, pp. 5767-5777. 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456, 2015.
Diederik P. Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations, 2015.
Peter Kontschieder, Madalina Fiterau, Antonio Criminisi, and Samuel Rota Bulo. Deep neural decision forests. Proceedings of the IEEE international conference on computer vision, pp- 1467-1475, 2015.

---

Under review conference paper at ICLR 2020
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Hoon Kwon and Min-Gyu Park. Predicting future frames using retrospective cycle gan. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1811- 1820, 2019.
Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabás Póczos. Mmd gan: Towards deeper understanding moment matching network. Advances in neural information processing systems, pp. 2203-2213. 2213, 2017.
Ziwei Liu, Ping Luo. Xiaogang Wang, and Xiaoou Tang. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
Deep learning face attributes in the wild.
Laurens der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, (Nov):2579-2605, 2008.
Boris Mityagin. The zero set of a real analytic function. arXiv preprint arXiv: 2015.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference Learning Representations, 2018.
Sangwoo Mo, Minsu Cho, and Jinwoo Shin. Instance-aware image-to-imas translation. In Inter- national Conference Learning Representations, 2019.
George Piranian. The set of nondifferentiability of continuous function. The American Mathemat- ical Monthly, (3(4):57-61, 1966.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In International Conference Learning Repre- sentations, 2016.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 2234-2242, 2016.
Erwan Scornet. Random forests and kernel methods. IEEE Transactions on Information Theory, 62 -5000, 2016.
Cencheng Shen and Joshua T Vogelstein. Decision forests induce characteristic kernels. arXiv preprint arXiv:1812.0000: 2018.
Wei Shen, Kai Zhao, Yilu Guo, and Alan L Yuille. Label distribution learning forests. In Advances in Neural Information Processing Systems, pp. 834 843, 2017.
Wei Shen, Yilu Guo, Yan Wang, Kai Zhao, Bo Wang, and Alan L Yuille. Deep regression forests for age estimation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2304-2313, 2018
Bharath K Sriperumbudur, Arthur Gretton, Kenji Fukumizu, Bernhard Schölkopf, and Gert RG Lanckriet. Hilbert space embeddings and metrics probability measures. Journal of Machine Learning Research, 1(Apr))1517-1561, 2010.
Hao Tang, Dan Xu, Nicu Sebe, Yanzhi Wang, Jason Corso, and Yan Yan. Multi-channel attention selection gan with cascaded semantic guidance for cross-view image translation. In Proceedings of the IEEE Conference on Computer Vision and Patler Recognition, pp. 2417-2426, 2019.
Wei Wang, Yuan Sun, and Saman Halgamuge. Improving MMD-GAN training with repulsive loss function. In International Conference on Learning Representations, 2019.
10

---

Under review conference paper at ICLR 2020
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv 2015.
preprint
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks, In European conference on computer vision, pp. 818-833. Springer, 2014.
Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. Dm-gan: Dynamic memory generative adver- sarial networks for text-to-image synthesis. Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 5802-5810, 2019.
Yan Zuo and Tom Drummond. Fast residual forests: Rapid ensemble learning for semantic segmen- tation. In Conference on Robot Learning, pp. 27-36, 2017.
Yan Zuo, Gil Avraham, and Tom Drummond. Generative adversarial forests for better conditioned adversarial learning. arXiv preprint /805.05 2018.

---

Under review conference paper at ICLR 2020
A SUPPLEMENTARY METHODOLOGY
A.1 KERNELS AND LOSSES
The formulation of our proposed random-forest kernel
1 kRF 0F) (a (x; (y: ( 0F))
T
In experiments, we take the number trees in the forest = 10 and the depth of trees dep = 3. Thus, the total number of internal nodes is 70. To notify, in general, the parameters O. (0)) are trainable, where 0z = (w', b%), w = every internal nodes of tree. However, for the sake of experimental simplification, we fix each w/ = 1, bf untrainable.
w()) =
and / is the set of
In the setting of MMD GAN loss proposed in Li et al. (2017), we compare our random-forest kernel with the following kernels: Gaussian kernel (Li et al., 2017) with mixture of kernel scales o, that is.
||
3/?
(x, (z. 3) = Σ
exp
2x²
.E
where Σ {2, 5, 10, 20, 40, 80}, and rational quadratic kernel (Bííkowski et al., 2018) with mixture of kernel scale a, that is,
1+ |
v11 2x
-a
Kn
(2, Σ aEA
where (,,
In the setting of the MMD GAN with repulsive loss proposed in Wang et al. (2019), compare forest kernel with bounded RBF kernel (Wang et al., 2019), that is,
max
| - y1| b1}
exp
if (Py]
20
(a (z, 8)
min { || y 1/2,,
exp
if (Px).
20
with a = 1, b, = 0.25 and by 4.
A.2 VISUALIZATION DETAILS OF KERNELS
In figure 3, we compare the contours of three different kernels (the detail of kernels is shown in A.1). We directly plot the filled contours of 2-dimensional Gaussian kernel and rational quadratic kernel with reference to (0, 0); As for random-foree kernel with T = 10 and dep = 3, where the input dimension is 70, the steps of 2-dimensional visualization are follows:
1) We randomly generate 25000 points from the uniform distribution u[-1, and set (0.5.... 0.5) E 70 as the reference point. To notify, if the reference point is 70- dimensional zero vector, the values random-forest kernel will be constant;
2) We calculate the 25000 output values of candom-foress kernel;
3) We transform 70-dimensional 25000 randomly generated points and reference point to- gether by t-SNE to 2-dimensional points;
4) We show the filled contour of the neighborhood of transformed reference point. We try to visualize the local structure random-forest kernel.
A.3 NETWORK ARCHITECTURE
In the experiments of the CIFAR-1 10 and LSUN bedroom datasets, use the DCGAN architecture following Miyato et al. (2018), and for the experiments of the CelebA dataset, we use 5-layer DCGAN discriminator and 10-layer ResNet generator in Birikowski et al. (2018). The first few layers the ResNet generator consist linear layer and residual blocks as Guirajani et al. (2017). The network architecture details are shown in Table 2 and 3.
12

---

Under review conference paper at ICLR 2020
Table Network architecture used the image generation on CIFAR-10 dataset LSUN bedroom dataset. In terms the shape parameter and w in the table, we take h = w 4 for CIFAR-10 and It = 8 for LSUN bedroom. As for the output dimension of discriminator s, we take = 70 for random-foree kernel and 16, 70 for other kernels. To notify. 16 is the setting in Binkkoskii et al. (2018); Wang et al. (2019) and 70 set the comparison of random-forest kernel.
(a) Generator
(b) Discriminator
noise input E 128 U[-1,1]
RGB picture a E [0, [0,1]"xwxx
128 + h X w x 512, dense, BN, ReLU
3 3, stride conv, 64, LReLU
4 x 4, stride 2 deconv, 256, BN, ReLU
x stride cony, 128, LReLU
x 4, stride 2 deconv, 128, BN, ReLU
3, stride conv, 128 LReLU
4 x 4, stride 2 deconv, 64, BN, ReLU
4 4, stride conv, 256, LReLU
3 x 3, stride 1 deconv, 3, Sigmoid
3 3, stride cony, 256 LReLU
4, stride conv, 512, LReLU
3 3, stride conv, 512 LReLU
dense -
Table 3: Network architecture used in the image generation on CelebA dataset, For the output dimension of discriminator we take = 70 for random-forest kemel and 16, 70 for other kernels.
(a) Generator
(b) Discriminator
noise input E R 128
1111,
RGB picture x E 1| 1
128 5 1024, dense
x 5, stride conv, 64, LReLU
ResBlock up. 512
5 5, stride conv, 128, LReLU
ResBlock up, 256
5 5, stride conv, 256, LReLU
ResBlock up, 128
5 stride conv, 512, LReLU
ResBlock up, 64, BN
5 5, stride conv, 1024, LReLU
5 x 5, stride 2 deconv, 3, Sigmoid
dense +
13

---

Under review a conference paper at ICLR 2020
B THEORETICAL ANALYSIS
In Section B, we will show the main propositions used to prove the Theorem 2. To be specific, in Section B.I, we represent neural networks computation graphs. Section B.2, we consider general class of piecewise analytic functions as the non-linear part of neural networks. In Section B.3, we prove the Lipschitz property the whole discriminators. In Section B.4, we discover that for Px almost surely, the network is not differential for its parameters 02 and vb. Fortunately, we prove that the measure of bad parameters set is zero.
In Section C, we will show the explicit proof of main propositions in Section B and Theorem 2.
B.1 NEURAL NETWORKS (NN)
B.1.1 CONVOLUTIONAL NEURAL NETWORKS (CNN)
Historical attempts to scale up GANs using CNNs to model images have been unsuccessful. The original CNN architecture is made up of convolution, non-linear and pooling. Now for our model, we udopt the deconvolution (Zeiler & Fergus, 2014) net to generate the new data with spatial upsam- pling. Moreover, batch normalization (loffe & Szegedy, 2015) is a regular method which stabilizes learning by normalizing the input to each unit to have zero mean and unit variance. Furthermore, relu functions are used both generator and discriminator networks as non-linear part. Here we avoid spatial pooling such as max-pooling and global average pooling.
B.1.2 COMPUTATIONAL GRAPH REPRESENTATION OF NEURAL NETWORKS
Throughout this paper, we always denote by
h(0)= := ((;;; 0) = (
(5)
the output of fully connected layer, where d' represents the number of neurons the output.
The general feed-forward networks including CNN and FC can be formulated as directed acyclic computation graph consisting of L + layers, with a root node = and a leaf node i For a fixed node i, we use the following notations:
x (i): the set parent nodes of
j < i: is a parent node of i.
Each node i > 0 corresponds to function f; that receives a IR input vector, which is the concatenation of the outputs of each layer in (i)), and outputs -valued vector, where d=(s)= = -jEn()) d, and do = d > 0. According to the construction of the graph G, the feed-forward network that factorizes with functions f; recursively can therefore be defined by h = X, and for all L.
:= -((h(()))))
where (() is the concatenation of the vectors h², j (1). Here, the functions fi can be of the following different types:
(i) Linear/Affine: If the weights W are middinensional vectors, and the function fi mh. x +1) is linear operator on the weights W W.e. convolutions, then the functions fi are of the linear form
M(Y =:("") (Y 1)" =: g:
(ii) Non-linear: Such functions including ReLU, max-pooling and ELU, have learnable weights, can potentially be non-differentiable, and only satisfy some weak regularization conditions, Definition 3 and the related Examples 2, 3, and 4.
In the following, we denote by the set of nodes such that fi is non-linear, and its complement by Ie (1,..., T.} I, that is, the set of all of all linear modules. We write 0y the concatenation of parameters
N = (W heze E Rlex!
14

---

Under review conference paper at ICLR 2020
where 10 denotes the total number of parameters of 0y. Moreover, the feature vector of the network corresponds to the output neurons Gr. of the last layer I. and will be denoted by
hg h/ I
where the subscript 0 stands for the parameters of the network. If X is random, we will use hav (X) denote explicit dependence on X, and otherwise when X fixed, it will be omitted.
B.2 PIECEWISE ANALYTIC NONLINEAR LAYERS
Throughout this paper, in the construction of neural networks, we only consider activation functions that are piecewise analytic which can be defined as follows:
Definition (Piecewise Analytic Functions) Let (filiez be non-linear layers in neural networks, then fo is said be an piecewise analytic function if there exists a partition of B with Ja pieces, that is, - D = Bd-0 and Di nD! = for $ j', such that there exist
(i) S real analytic functions P : 1Rd - RR such that
=Y R (.) > for all E {I....,,!!. S
(ii) Jj real analytic functionn (ff))e(),....)) such that f:(Y) = (Y)
for all Y D..
The sets Dj are called analytic domains and the functions ft are called analytic value.
The following examples show that in practice, the vast majority of deep networks satisfy the condi- tions in the above definition, and therefore are piecewise analytic.
Example 1 (Sigmoid) Let Si outputs the sigmoid activation function on the inputs. Then need no partition, and hence there exist
(i) s = real analytic functions gi,(Y) = such that
D. 1o00
corresponding the whole space;
(ii) Jj 1 real analytic function ft : fx.
Here, mention that the case in Example corresponds functions used in deep learning, more examples include the softmax, hyperbolic tangent, and batch normalization functions. On the other hand, in the case that the functions are not differentiable. as is shown in the following examples, many commonly used activation functions including the ReLU activation function are least piecewise analytic.
of the differentiable activation
Example 2 (ReLU) Let ft outputs the ReLU activation function on two inputs. Then we have partition of Jj pieces, with each D., j (1,2,3,4} corresponding to quadrant of the real plane, such that there exist
(i) S 2 real analytic functions (Y) +Y and (Y, +Y₂ such that
DJ. = (Y ( 0 and 5. (Y) 0};
(ii) J, real analytic functions ff = (Yr, Y2), ft (Y,, ff = (0, 0), and f
(0, Y2) such that
1(Y)=$((Y)
for all Y D..
15

---

Under review conference paper at ICLR 2020
Besides the ReLU activation function, other activation functions, such as the Max-Pooling and the ELU (Clevert et al., 2016) also satisfy the form of Definition 3 are therefore piecewise analytic.
Example 3 (Max-Pooling) Let fs outputs max- max-pooling on two inputs. Then we have a partition of J4 pieces with each domain Di. j == 1.2, corresponding to half plane, such that there exist
(i) S = S? = real functiion (X) == 11 - Y2 and 97,1 (Y) = Y22 - Yi such that D. = (Y 9f, 0} and
DJ (Y 1 > 0):
(ii) J, = 2 real analytic functions fi == Y1 and ft = Y2 such that f((Y) (Y)
for all Y D..
Example 4 (ELU) Let fi output the ELU activation auttttotionuunction on two inputs. Then we have a parti- tion of J; = 4 pieces, with each Di, j E (1,2,3.4) 4), corresponding to a quadrant of the real plane, such that there exist
(i) S real analytic functions (i,(Y) == +Y, and g²,( =Y2 such that
DJ (Y ()) (Y) 0 and 2²2 0}:
(ii) Jj 4 real analytic functions ft (Y,, Y2), f. = (Y,,0), fa == (0, 0). and ft
(0, Y2) such that
f((Y) = P(
for all E D..
B.3 BOUNDING THE NETWORK GROWTH
In this section, we investigate the Lipschitz property of the proposed discriminative networks, which is formulated as follows:
Denote == (z, y), then the linear kernel (z, y) can be written univariate function (z)u,, that is,
K(u) == : :
(6)
Proposition 4 Let On be the parameters of discriminators and B, (0n) C BRoo be the ball with center OD and radius E (0. oo). Then, for all O', E (0D) and all x E R.d, there exists regular
function c(z) with Ex [c(x)] 00 such that
K
Pep (x:) K (10g
(T)
(T) (x)) s c(x)|10> 6'l1-
B.4 EFFECTLESS OF CRITICAL PARAMETERS
In the analysis section B.3, we intentionally ignore the situation when samples fall into the "bad sets", where the network is not differentiable for data sets z with nonzero measure. And in proposi- tion 5, we show that the measure of the so called "bad sets" is zero.
To better illustrate this proposition, first denote notations follows:
For fixed 0N. E RR we denote
x,, {a Rd 0N + hew (x) is not differentiable at 0
(7)
the set of input vectors E Rd such that hoy (x) is not differentiable with respect to 0y the
point N.o Then we call
Opx (Ox Px(((()))) >
the set of critical parameters, where the network is not differentiable for data sets x with nonzero measure.
Proposition 5 Let the set Opx be as in equation 8. Then, for any distribution Px, we have
1(Orx) ) 0.
16

---

Under review a conference paper at ICLR 2020
C PROOFS
C.1 PROOFS OF SECTION B.3
To prove Proposition 4, we first introduce Lemma and Lemma 7.
Lemma 6 describes the growth and Lipschitz properties of general networks including convolutional neural networks and fully connected neural networks introduced in Section B.1.
Let bay be the neural networks defined as in Section B. 1. Then there exist continuous functions a, b Bl0xl -> and a, B TR21 - R such that for all x E Red and all A 0'y Ox there hold
Lemma
I1ow (x)]l - b(Ox) + [(=x)
(9)
I (x) hoy, (+)]l = ||ON 03 + a((O,, [x)]==]).
(10)
Proof [of Lemma 6] We proceed the proof by induction on the nodes of the network.
Obviously, for 0, the inequalities hold with bo 0, do 1, Po 0 and ag 0. Then, for the
induction step, let fix index i. Assume that for all x E Rd and all @N. 8'N- R fxrl, there hold
( (x) S (x(e((xx) + am( (1x)
| (x) - (0)( (x) S 10, 0x | (B (1) (ON, 02x) + an (xxx0(x)))]]=
where (ON) (n( (0x), (i))) (ON. 0'x) and (z(() (ON. 8yy) are some continuous functions. (i) lffi I, that is, if is linear layer, then the following growth condition
+ 1) s (0x) + a,(0 || holds with as (0x) = ll l (ON)) and b;(0x) = |9 | (1) Moreover, concern- ing with the Lipschitz property, have
| || s (9
A x3(00 px s |1
[| W [|
| (x) Nag (z)]] = |9 [9 (W. ) (2((0 (x) - g. (W,)) (-( (z)]
< (w:) 7()) 8y (x) n (z)) + [a - (x)]
< |1 W: || (1)0 (a) Now (z) + ||; - W (z) + 1))
1 0'x11 0'x) as (Ov, 0x)))=
where we used the notations
(0,0 0'w) = ||0: (I ((] + [[][)) (0xx 03y) + (ax(1) ax(1) (6x)) (,,0N 0')) ||9: WG]] + ([[[)) (ON, xx) (bx()(ON) - (0)) 1).
(ii) if Is, that is, is not linear layer, here we only consider the sigmoid and ReLU functions. We first show that both of them are Lipschitz continuous. Concerning the sigmoid function.
(xx) 1te
we obviously have
1
d'xx) 2
(e (e- + c=/2 I 4'
for all x E RR Consequently, the sigmoid function is Lipschitz continuous with Lipschitz constant |ol| = 1/4. Next, for the ReLU function,
(xx) 2).
we have for all æ E R.
o' d'(x) max(0, 1] < 1.
17

---

Under review conference paper at ICLR 2020
Therefore, the ReLU function is Lipschitz continuous with Lipschitz constant 10/2 == 1. Thus, non-linear layer Ji is Lipschitz continuous with Lipschitz constant M = [/il
Consequently, by recursion, we obtain continuous functions 0; = Max()) Pr MB a; Mar()), and b; = Mbx()))
Next, we investigate the growth conditions and Lipschitz property of the random forest. For the ease of notations, we write the function (T()) as
a a (1;; NN, 0p) =: (h (h(a; (x
Moreover, it is easily seen that T [ equals the number of internal nodes in the random forest.
Lemma Let Ma:; 0N) be the input vector of random trees and Or = (w, b) where w and denote the weights and bias the the random forests, respectively. Then, for all h E Ted and fr, 0'r E 22TW there exist continuous functions CT C2. and constants Cg2 Ca. G3 such that
(N): (M(a;))))
ea (b)b)
(11)
(M( S callw calb bl]:
(12)
- (x) 0x) <=|lw]]]|b< h(2; f'x)
(13)
Proof [of Lemma 7] For = =1.1.... 1..... T, denote وز as the f-th leaf node of the t-th tree. By convenient abuse of notation, we write the function jul as
(=;2;; w', bb) (h(x,; ON); , (e))).
First of all, let us verify the growth condition equation 11. From the definition of the funetion we immediately know that for any node i in the random forest, there exists an element j T() such that
(h(h) + exp (wihy-b.)
Obviously, we have
max(pi, 1- pı}
1 + exp -lol, +b,
For any leaf node E L., that is, for any node in the 1-th tree, in the following, for the ease of notation, we always set ji = 1re w; w/, hg h2y b, b4, if not otherwise mentioned. Then fe in equation 3 can be upper bounded by
1
He -
+
luch, + byl
(1+e-luibytby
2
iER;
ER
IN
I
1
ww,h + bul -
(lwhhyl + (6.l).
2
Taking the summation on both sides of the above inequality with respect to all of the nodes in the random forest, that is, w.r.t. É E L, obtain
L
||
+ W
b9
+
W
Σ
2
[C]
1/2
1/2
+
+|
+ LA1
Σ
Σ
(er())
2
3,4
2,t
[
+ W ||||||| Will
||bl|
2
(b) + (
18

---

Under review a conference paper at ICLR 2020
where the second inequality follows from the Cauchy-Schwarts inequality and the third inequality is due to the fact that the number that the nodes p; in the random forest assigned to the corresponding node h equals [T or [T[A]/d'] + which are less than /
Now, we show the Lipschitz properties equation 12 and equation 13 of the random forest. From the equivalent form equation concerning the value of the leaf node, we easily see that μıg can be written as product of probability functions p; or 1 - Pi. Therefore, without loss of generality, we can assume lig are of the product form
He PR((I) ..... (R((H))
(14)
For fixed = 1,...: T, recall that T₃ denotes the connection function of the t-th random tree. Then, the Lipsehitz property of the sigmoid function and the continuously differentiability of the linear transform yield
lp, p/1 < (): (h; 07) S lwgh, + b. - w/h 1/1 s lw; w/llb;1 +
bb; 1/1. (15)
Then, equation 14 together with equation 15 implies
(k)(h; - pb(h;; (PR((1) x x PRI(H) - PR) x ... x PR.j
$ (PR,(1) - -'RR(1)) X (R, ... x (7))
P(RE(1) X... PRe(H(-1) x |PRE(H)) PR.
W
PRe()) P'R.OO S
PP rkl
k=1
W
I - w + 1b4 - 1/2l
A=1
1/2
$
||
wt
Σ kT))))))
+ +WIlIB - 8z 11,
where the last inequality follows from the Cauchy-Schwarr inequality. Consequently, concerning the random forest, obtain
A
1/2
(T
Σ 1 - w01l
Σ
hT((())
+ W/1 Σ 11b² 5t
1/2
< W| - w'll
Σ
hT
+ +WR/TII - 811
h2t
< Wi]l - w/ _ ||bl|| + | - l1
d'
ca w w/ |||||||||| + ca b Bl
where the second inequality is again due to then Cauchy-Schward inequality. Analogously, for fixed t 1,..., T and any we have
(h: (xh':: ( < |wshts + by -wah's bal < |ws| | 131.
19

---

Under review conference paper at ICLR 2020
and consequently we obtain
ph(h; OF) (h(: = (1) ... x PR((H) PRI(1) .... PRR)CO))
S (1) (RR(1)) (R)(2) X (R)
+.. + (I) x.. x (R x (PRE(H)) - PRR)
H
I
PRe()) PR) - Σ
1ot - pt
2=1
[
( - (()(
(nr, - 1erccol9 1/2
||w0|| ||
and for the random forest, there holds
1/2
(h: - (h':
S W Σ ||00|| Σ IT
1/2
Wi
| 1!r
d' 1// k'll
=: c||||||| |h h I1.
which completes the proof.
The next proposition presents the growth condition and the Lipschitz property of the composition of the neural network and the random forest.
Lemma Let (fp)) C Rl be the ball with center 0D and radius r E (0, oc). Then, for all 8D (fD)), all x E IR,, there exist continuous functions Co, C7. Ca, and cy such that
||000 (x; 0D) - (a; S cr(0p) + co(OD)]l=|]:
< || - (c 0)) co (0)0,,)))lll))
Proof [of Lemma 8] Combining equation Lemma with equation in Lemma we obtain the growth condition of the form
(1( (x; 1N 0r)
s (b) + cc(])]])(;;
S ci (b) + (()
= ci (b) + +(W))
c7(b, w. 0N) + cc(,
= <>(0p) + cs (0p) |z||-
Concerning the Lipschitz property, using equation 12 and equation 13, we get
(x; On, 0F) - (T (x;; 0')]
(T; 0N, 0r) (x; 0'y 0r)] + (x; 0r) n (x;
(x; U'y ,
< [,(= Mx;; calw - w'11i(a (x) )l + cs lb 1--1'11
< + +
(x) +a(0x.0)))1l
ca lo'l/=
+ callb b'1l.
20

---

Under review conference paper at ICLR 2020
where the last inequality follows from equation 9 and equation 10 established in Proposition 6. With the concatenation OD == NO,, 0r) = (ON, w. b) we obtain
I 0p) n TT) (z; Pb - |0D 0))l(c((0:00)) + co(0
and thus the assertion is proved.
Proof [of Proposition 4] First give the growth conditions for the linear kernel. Let &s be the linear kernel ky (z. y) = (x, y). then we have
(L, 3) < || + |vll +
I KI, w)1l < (1==|| + | 12) 1/2 + +1.
If we denote :- (x, v). then the above linear kernel (r, can be written univariate function (u), that is,
((u) == (2(4) == ky (z. y)-
(16)
Therefore we have K is continuously differentiable satisfying the following growth conditions:
[K(x) ||u|| 1,
(17)
V. s ||||| +1.
(18)
Let us define the function f: [0, 1] + by
f(t Ktuu (11 -1-11 f)u), Then we have f(0) K()) and f(1) = Kuu). Moreover, is differentiable with derivative
E [0, 1].
f'(4) (K(u (1 () v).
The growth condition equation 18 implies
lf' [V(((tu + (1 - t)o)(u v) ||V + (1 - t)v) v)
S (llt(u - v) + all + 1) vl
$ ((I)u w/l + ||=|]) + 1) ||u r|]-
Using the mean value theorem, we obtain that for all u, E R there holds
(u) K (w)) < ((1 wll + [|-l]) + 1) |/ w/|- (T)
With = pop (2) and := p (z), one gets
(2) K [K(2 (x)) K K(p0 (x)) < 1 (x)
(T) no, (x) + 1100000000000
(x)||+11000 (x)] + +1) 1)]1" (x) - -10p (x)
Proposition 8 tells us that
|| (3) (x) Mep
(T) (z)] < (c. 0b) + co(0D, ()1==)))) 0'bl.
(x)f(()) S (7(0 co =01)
holds for some continuous functions Ca, C7, Cs. and Cy, which are also bounded by certain constant B > 0 on the ball (yp)). Some elementary algebra shows that
(x)) - K (m (=)) < (B²(r 1)(1 ||>|)² B(1 + || [1) - -pll- [[
(T) 0 [(
Since t + (22+ is concave
> 0, Jensen's incquality together with the moment assumption
E x xo implies
E(1 |-]|)² (1 (1+ (E12
<
and also B(1+ + |[=l]) < oo by the moment assumption. Therefore, the regular function c(x) defined by
<(=) = (B + 1) + ||=]]) + B(1 + |
is integrable and thus our assertion is proved.
21

---

Under review conference paper at ICLR 2020
C.2 PROOFS OF SECTION B.4
To reach the proof of Proposition 5, we first introduce Lemma 9 and Lemma 11.
Let a fixed node. To describe paths through the network's computational graph, we need to introduce the following notations:
P {(,,,,s) E N* i 10.. E (1,...., Sti.. (
A(t) {i' is an ancestor i);
-i = (P A(1) or i
EPI
-(1) = U 1= EP | A(i)):
(EE((()
0i = ((6,,,))
Obviously, we always have di S -i and -i di U - We define backward trajectory starting from node by
q (t', Jo (e)AACC))))
je Je)
The set of all backward trajectories for node will be denoted by Q(i).
Lemma Let i be fixed node in the network graph. If ON RR( \ 08 then there exist constant y > O and a trajectory q E 2(1) such that for all 0' E (fx))),, there holds
hlg (((())
where fa is the real analyticfunction on (6 with the same structure as bax only replacing each nonlinear fy with the analytic function fbr for (i', jv)
Proof [of Lemma 9] We proceed by induction the nodes of the network. If 0, we obviously which is real analytic on Rl For the induction step assume that the assertion holds for - -m(()) (i) and let 0,y R 10ll \ as- Then there exist constant 17 > 0 and trajectory q E (()(i)) such that for all O'y E (,(0x), there holds
have (g
h (+) (0'x) %
(19)
with fa [0 + being a real analytic function.
(i) If 0N & 8 then there exists a sufficiently small constant ry' > 0 such that (, n 0. Therefore, there exists j E JT. such that for all 8'x- E (G (f))). ), there holds
h = (1
where f is one of the real analytic functions in the definition of f.. Then, equation 19 implies that for all 8/y- E (Bmin(nor) (fx), there holds
hig = (f
(20)
(ii) Consider the case ON SOi By assumption, have ON as Then there exists small enough constant n/ > 0 such that (0x) Soi If we denote
A (p (i.j.8) di ON E EP))
then A + 9 since Oy E S0i Therefore, we have
0N E n SP
and
ON & U SP.
PEΛ
PEA-
22

---

Under review conference paper at ICLR 2020
Now we show by contradiction that for 7/ small enough, there holds
(ON) S 0 S".
PEA
To this end, we assume that there exists a sequence of parameter and index pairs (0 such Since A is finite, there exists a constant
that Ph. E A", NN.. n E S"r, and nN NN subsequence {Pnz} C {pr} and some constant po E A with Pn; = Po for all i. Then the continuity of the network and Ypo imply that 8 is closed set and consequently we obtain IN E SPo by taking the limit, which contradicts the fact that V & Upe SP. Therefore, for n/' small enough, there holds (Ox) ΠEEA S", which contradicts the assumption ON E as Consequently, there exists j {l,..., J+} satisfying equation 20. By setting f = [, with go ((i,j) @ 4) E Q(1). where + denotes concatenation, then for for all 0' E Bmin(n-rr) ) NO)), there holds
R0ys r00000))
and the assertion is proved.
Now, for fixed p (1,j,8) P, denote the set of network parameters Oy that lie the boundary of by
S (ON R (1700)) = =],
where the functions gh are as in Definition As usual, the boundary of the set SP is denoted by OSe and the set of the boundaries denoted by
aS U aS..
(21)
PEP
Finally, for the ease of notations, if C.P, we write
SP
S
aS
U asa
(22)
P'EP'
P'EP
Obviously, we have IN n" 0S (+)) To prove Lemma 11, need the following lemma which follows directly from Mityagin (2015) and hence we omit the proof.
Lemma 10 Let ON + F(NN): R ON + R be real analytic function and define
M = (On E ON! F(Ox) = 0}.
Then we have either μ(M) = 0 or F 0.
Lemma 11 Let the set of boundaries äSP be as in equation 21. Then we have
u(asp)= =.
Proof [of Lemma 11] We proceed the proof by induction. For i = 0, we obviously have as and therefore u(as -0) 0. For the induction step let us assume that
u(as = 0.
For (p, q), the pair of an index p di and trajectory g Q(1), we define
M, = (0x E R Ovl ((x)) = 0},
where the analytic function f is defined as Proposition Then we prove by contradiction that for any O2 0s as m()) there exists an s di Q(1) such that 0 E M, and
((M.) = 0.
23

---

Under review conference paper at ICLR 2020
To this end, let 0,y as as (s)*()) then for small enough 17 > O, there holds (, R
as -(), By Proposition 9, there exists a trajectory E (()(1)) such that for all 0y E (,(O there holds
(+) xx
(00x)
(23)
Moreover, since dy E as there exists an index p di such that 9p (1) for = (p. q). we have v E M,, Therefore, we have
(p(
This means that
as a))) (+) C
U
(24)
SEA
where A C P x 0i 2() is finite. Suppose that p(M,) 0, then by Lemma 10, have
M, = Ii and hence B, c Mas By equation 23, we then have B,((x) C S which contradicts the fact that d E th and hence we have
m(M,)= =0
(25)
Combing equation 24 and equation 25, we obtain
1 " (0) S Σ m(M.) 0.
SEA
By the assumption ((aSS=()) 0, conclude that p(88~t) 0. Since for the last node I,, have - P and therefore (()SP) = 0.
Note that the random forest y (-) can be considered as the composition of affine transformations and sigmoid functions and hence always continuously differentiable with respect to 6y, we only need to investigate whether the neural network he (x) is differentiable with respect to d For fixed Rd write
θ (0 Rllx
0N hox (x) not differentiable at 0x.03
(26)
as the set parameters for which the network is not differentiable.
Lemma 12 Let the set O₂ be as equation 26. Then, for any R2d we have
u(Oa)) O.
Proof [of Lemma 12] Let the boundary PSP be defined in equation 22 and ON,o Q.. Obviously, we have On E SP We proceed the proof of the inclusion Q C aS by contradiction. To this end, we assume that dDD & aSP. When Lemma 9 applied to the output layer, there exist some 7) > 0 and real analytic function f(Ow) such that for all ON (Ox,,,)), there holds
hew = f(Ox).
Consequently, the network is differentiable at 0,.u. contradicting the fact that AN.o E O Therefore, we have θ S as and hence p(O,)) = since y(as") = 0 according to Lemma 11.
Proof [of Proposition 5] Let the sets ((xx), Opx, and O,, be as in equation 7, equation 8, and equation 26, respectively. Consider the sets
Si ((ON,x) x) E R 1xvl
Rdd ON Orx and x E ((O))),
S₂ {(ON, x) E 10vl
x gd 10 O.).
Since the network is continuous and not differentiable, Theorem in Piranian (1966) implies that the sets S and S2 are measurable. Obviously, have S, C S2 and therefore obtain v(Si) S v(22), where v = Px & μ. On the one hand, Fubini's theorem implies
{ 2g S.
du(Ox) dPx(a)) / ((O.)dPxx
((22)
24

---

Under review conference paper at ICLR 2020
By Lemma 12, we have μ(())) and therefore v(S2) 0 and hence v(S1) 0. On the other hand, Fubini's theorem again yields
v(S1) = Jeex JN(ew) dP dP(()) = Px(N(ON))dH(ON)
Jerx
By the definition of Opx we have Px(N((N)) > 0 for all ON E Opx Therefore, v(Si) = implies that p((px) =
C.3 PROOFS OF SECTION 3.5
To prove Theorem 2, we also need the following lemma.
Lemma 13 Let non)NEN be sequence in IR' converging towards 8, i.e. On # Gu as Moreover, let f Rmm - R be function and g be a vector in R If
If(0)) - f(0n) - g (0n - 0o) +
00, Poll
holds for all sequences (0n)neN, then J is differentiable at Og with differential g.
Proof (of Lemma 13] The definition of differential tell us that g is the differential of f 0o. if
1 [f(00 + +6 -(()) - 0. |8||
lim
By the sequential characterization of limits, we immediately obtain the assertion.
Proof [of Theorem 2] Consider the following augmented networks:
((D. Z) = (160) (X), Hop (G
(1)
h (2)
(2, Z') =(40 (G (Z)) (T) (G,
(6D w)
hgp (X, X') (uu (X), Pop (X)).
(3)
(T)
Without loss of generality, in the following, we only consider the network i1 (1) from Px @ Pz, which satisfies E [XX, 2) oc.
(0n.v) (X, 2) with inputs
By the expression of definition 1, we have
1
kRF (ha (X), how (G (Z)):
(how T (han, (how (Ge)
T (Pop (X),
1
(x),
(.) (G,(2)))
(T) where we denote Pap (x) == (1o (I); Due to the linear kernel (I v) = (I, v), there holds
KRF (hox (o) (G) kz (X), (T
(
=: =:((hxuu)),
where u (), (0p. v), and the second equation due to equation 16. Suppose that there exists a λo (0p,o, vo)) such that the function (0p, v) + (u (T') (xx), (z)) is differentiable at Ao for almostt all æ and Pl-almost z. Then, according Proposition 5, this statement holds for y-almost all On Rlool and all g Rlol, Consider sequence (An)neN that converges to Ao then
25

---

Under review conference paper at ICLR 2020
there exists 8 such that | Aoll for all n E N. For a fixed a (x, E IR R!
Proposition 4 states that there exists regular function c(u) with Ep c(X) < oo such that
|K (hs, (u)) K (hea - c(u)llA - Aoll
and consequently we have
( (1)) 5 c(u)
for Px @ Pg-almost all u E Rll
For N, define the sequence 9n (a:) by
gn (w) (K(hx,(w)) K K(hr, (u))
- -do1
AA Aoll
Obviously, the sequence 9n. (3r) converges pointwise to 0 and is bounded by the integrable function 2c(u). By the dominated convergence theorem, see c.g. Theorem 2.24 in Folland (1999), we have
&x SP (n) -
Moreover, for n E we define the sequence Gn (xx) by
gn Ep K(hs,, Ep K(ha (u)) Ep O2 (gK(h,o (u))(An Au)
|| - doll
Clearly, the sequence gn.2) (z) is upper bounded by Epx xx In (u) and therefore converges 0. By Lemma 13, x [K (K)(()) is differentiable at Do with differential
(u)
Exx O2 (h)())
Since similar results above hold also for the networks (2) and (3) and Lemma 6 in Gretton et al. (2012) states that MMD is unbiased, our assertion follows then from the linearity of the form of MMD;, in equation 1. a
D SAMPLES OF GENERATED PICTURES
Generated samples on the datasets CIFAR-10, C, CelebA, and LSUNN bedroom are shown in Figure
4, 5, and respectively.
26

---

Under review as a conference paper at ICLR 2020
(a) Real Samples
(b) MMD loss with mix-rbf kemel
MMD loss with mix-rg kernel
(d) MMD loss with random-forest kernel
(e) MMD repulsive loss with rbf-b kernel
(1) MMD repulsive loss with random-forest kernel
Figure 4: Generated Pictures with different kernels and two different loss functions on 32x32 CIFAR-10 dataset.
27

---

Under review conference paper at ICLR 2020
(a) Real Samples
(h) MMD loss with mix-rbf kernel
(c) MMD loss with mix-rq kernel
(d) MMD loss with anndm-forrst kernel
J.
(c) MMD repulsive loss with rbf-b kernel
(f) MMD repulsive loss with random-fore kernel
Figure 5: Generated Pictures with different kernels and different loss functions 160x 160 CelebA dataset.
28

---

Under review conference paper at ICLR 2020
(a) Real Samples
(b) MMD loss with mix-rbf kemel
(c) MMD loss with mix-rg kernel
(d) MMD loss with random- forest kernel
(e) MMD repulsive loss with rbf-b kernel
(i) MMD repulsive loss with random-forest kernel
Figure 6: Generated Pictures with different kernels and two different loss functions on 64x64 LSUN bedrooms dataset.
29


Reviews:
[{"review_id": "rkxC0wKhFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Theorem 2 and its proof are plagiarised: they are rephrased and reorganized formulation and proof of Theorem 1 of [1], while being presented as authors' own work. \n\nAlthough the assumptions are slightly different (random forest kernels vs general kernels), core of the proof is the same, including notation and its split into Lemmas and helper Theorems. In particular:\n- formulation is the same (even use of MMD_u is copied, while not being defined before),\n- main proof of Theorem 2 (p.25-26) is the proof of Corollary 3 of [1] followed by the proof of Theorem 5 of [1],\n- Lemma 13 is Lemma 3 of [1],\n- Definition 3 in Appendix B.2 is the same as Assumption D of [1] (Appendix C.2),\n- Proposition 4 and it's proof (p. 21) are the same as Lemma 2 of [1].\n\n[1] Mikołaj Bińkowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. International Conference on Learning Representations, 2018.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "S1geYsQnYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview: \nThe paper propose an MMD GAN extension via using Random forest Kernel.  Instead of using Gaussian kernel on the top of the learned embeddings from the discriminator, it combines existing deep forests kernels. The theory of being differentiable is carefully studied (to prove zero measure) and the experiments are well conducted.  \n\n1.  Some important  references are missing.  One very related paper is \n\n* Li et al., Implicit Kernel Learning,  AISTATS 2019. \n\nThat paper is using the same idea to learn to manipulate the random features on the top of the learned embedding.  The main difference between it and the proposed algorithm is they use MLP parameterization instead of the tree-based model.    Also, the deep forest model can be treated as a sparse neural network, does it have more advantage over Li et al., (2019)? given they use simple dense MLP.   Please at least discuss the similarity and difference in the rebuttal and update the draft correspondingly.  I would even encourage the author to empirically compare with it in the camera ready version.  It would be interesting to see which parameterization is better in this space. \n\nThere are also other recent MMD GAN extensions should be cited in the discussion, such as \n* On gradient regularizers for MMD GANs. \n\n2. For the theory part, based on Binkowski (2018), the gradients for the generator parameters should be biased. Could you discuss it with Theorem 2? \n\n3. For most MMD GAN results, one important property in Li et al., (2017),  Arbel et al., (2018) and Li et al., (2019) is weak* topology.  Does the proposed Random Forest MMD GAN also has that property? In Li et al., (2019), they need some condition to ensure that, how's case in the proposed algorithm?  ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJx3j8eaFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed to utilize the random-forest kernel into MMD GAN.  The experiments are conducted on CIFAR-10, CelebA and LSUN datasets. \n\nThe method is not novel. Both MMD GAN and the random-forest kernel have been well explored. Combining them together is considered as an extension. For the theory, the paper only provides the unbiasedness analysis. It is not clear to me whether this kernel is better than other MMD GAN variations. It is not clear how the claimed flexibility comes from. \n\nRegarding the experiments, it only compares with very basic baselines and the results are not significantly better. It would be better to include stronger baselines (Wang et al., 2019, Binkowski et al., 2018).\n\nThe writing of the paper is poor. with several typos. Moreover, as mentioned by reviewer #1,  theorem 2 and its proof are plagiarised. \n\nOverall, I think the paper is a clear reject. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers raise the serious issue that the proof of Theorem 2 is plagiarized from Theorem 1 of "Demystifying MMD GANs" (https://arxiv.org/abs/1801.01401). With no response from the authors, this is a clear reject.

Decision: Reject