Generative Ratio Matching Networks
Authors: Akash Srivastava
Kai Xu
Michael U. Gutmann
Charles Sutton
Venue:  (2020)
Keywords: deep generative model
deep learning
maximum mean discrepancy
density ratio estimation

Abstract: Deep generative models can learn to generate realistic-looking images, but many of the most effective methods are adversarial and involve a saddlepoint optimization, which requires a careful balancing of training between a generator network and a critic network. Maximum mean discrepancy networks (MMD-nets) avoid this issue by using kernel as a fixed adversary, but unfortunately, they have not on their own been able to match the generative quality of adversarial training. In this work, we take their insight of using kernels as fixed adversaries further and present a novel method for training deep generative models that does not involve saddlepoint optimization. We call our method generative ratio matching or GRAM for short. In GRAM, the generator and the critic networks do not play a zero-sum game against each other, instead, they do so against a fixed kernel. Thus GRAM networks are not only stable to train like MMD-nets but they also match and beat the generative quality of adversarially trained generative networks.

Published as a conference paper at ICLR 2020
GENERATIVE RATIO MATCHING NETWORKS
Akash Srivastava"
Kai Xu"
MIT-IBM Watson Al Labt
University of Edinburgh kai, xußed. aebeddaccuu uk
akash. s srivastaval ibm. com
Michael U. Gutmann
Charles Sutton
University of Edinburgh mi michael www.gggaaeeeaa.....
Google AI!
charlessuttordgoogle...om
ABSTRACT
Deep generative models can learn to generate realistic-looking images, but many of the most effective methods are adversarial and involve a saddlepoint optimization, which requires a careful balancing of training between generator network and a critic network. Maximum mean discrepancy networks (MMD-nets) avoid this issue by using kernel as a fixed adversary, but unfortunately, they have not on their own been able to match the generative quality of adversarial training. In this work, we take their insight of using kernels as fixed adversaries further and present a novel method for training deep generative models that does involve saddlepoint optimization. We call our method generative ratio matching or GRAM for short. GRAM, the generator and the critic networks do play a zero-sum game against each other, instead they do so against a fixed kernel. Thus GRAM networks are not only stable to train like MMD-nets but they also match and beat the generative quality of ad adversarially trained generative networks.
INTRODUCTION
Deep generative models (Kngmma & Welling 2013 Goodfellow et al. 2014 Kingma & Dhariwal 2018) have been shown learn generate realistic-looking images. These methods train deep neural network, called a generator, to transform samples from noise distribution to samples from the data distribution. Most methods use adversarial learning (Goodfellow et al. 2014),, which the generator is pitted against critic function, also called discriminator, which is trained to distinguish between the samples from the data distribution and from the generator. Upon successful training the two sets of samples become indistinguishable with respect to the critic.
Maximum mean discrepancy networks (MMD-nets) (Li et al. 2013) Dziugaite et al. 2015) are class of generative models that trained to minimize the MMD (Gretton et al.) 2012) between the true data distribution and the model distribution. MMD-nets are similar in spirit to generative adversarial networks (GANs) (Goodfellow et al.) 2014 Nowozin et al. 2016), in the sense that the MMD is defined by maximizing over a class of critic functions. However, in contrast to GANs, where finding the right balance between generator and critic difficult, training is simpler for MMD-nets because using the kernel trick the MMD can be estimated without the need to numerically optimize over the critic function. This avoids the need in GANs numerically solve a saddlepoint problem.
Unfortunately, although MMD-nets work well on low dimensional data, these networks have not on their own matched the generative performance of adversarial methods on higher dimensional datasets, such as natural images (Dziugaite et al.) 2015). Several authors (Li et al. 2017 Binkowski et al.) 201 Sa) suggest that a reason is that MMD is sensitive to the choice of kernel. Li et al. (2017) propose method called MMD-GAN, in which the critic maps the samples from the generator and the data into lower-dimensions representation, and MMD is applied in this transformed space. This
Equual contributions.
Additional affiliations: The University of Edinburgh and IBM Research.
Additional affiliations: The University of Edinhurgh and The Alan Turing Institute Official implementations are available httths://ggtthhh/ com
www.///AA/. GRAM/

---

Published as a conference paper at ICLR 2020
can be interpreted as a method for learning the kernel in MMD. The critic is learned adversarially by maximizing the MMD at the same time as it is minimized with respect to the generator. This is much more effective than MMD-nets, but training MMD-GAN can be challenging, because in this saddlepoint optimization problem, the need to balance training of the learned kemnel and the generator can create sensitivity hyperparameter settings. In practice, it is necessary to introduce several additional penalties to the loss function in order for training to be effective.
In this work, we present a novel training method that builds MMD-nets' insight use kernels as fixed adversaries in order to avoid saddlepoint optimization based training for the critic and the generator. Our goal is for the critic to map the samples into nower-dimensional space in which the MMD-net estimator will be more effective. Our proposal is that the critic should be trained to preserve density ratios, namely, the ratio of the true density to the model density. the critic is successful in this, then matching the generator to the true data in the lower dimensional space will also match the distributions in the original space. We call networks that have been trained using this criterion generative ratio matching (GRAM) networks, or GRAM-net We show empirically that our method is not only able to generate high quality images but by virtue of avoiding a zero-sum game (in critic and generator) it avoids saddlepoint optimization and hence is more stable to train and robust to the choice of hyperparameters.
1.1 RELATED WORK
Li et al. 2015) and Dziugaite al. (2015) independently proposed MMD-nets, which use the
MMD criterion to train a deep generative model. Unlike f-divergences, MMD is well defined even for distributions that do not have overlapping support, which is an important consideration for training generative models (Arjovsky et al. (017)). Therefore, MMD-nets use equation ( in order to minimize the discrepancy between the distributions Is and Ps with respect to G.,- However, the sample quality of MMD-nets generally degrades for higher dimensional or color image datasets Li al. 2015).
To address this problem. Li et al. 2017) introduce MMD-GANs, which use a critic fo R - R to map the samples to a lower dimensional space RK and train the generator to minimize MMD in this reduced space. This can be interpreted as learning the kernel function for MMD, because if So injective and kg is a kernel in RR then k(r, x²) = ko fo(x), fo(('') )) is a kernel in R.D This injectivity constraint on fo is imposed by introducing another deep neural network fag. which is trained to approximately invert fo using an auto-encoding penalty. Though it has been shown before that inverting fo is not necessary for the method to work. We also confirm this in experiments (See appendix).
The critic So is trained using an adversarial criterion (maximizing equation ( that the generator minimizes), which requires numerical saddlepoint optimization, and avoiding this was one of the main attractions of MMD in the first place. Due to this, successfully training fe in practice required penalty term called feasible set reduction on the class of functions that fo can learn to represent. Furthermore, f is restricted to be k-Lipschitz continuous by using a low learning rate and explicitly clipping the gradients during update steps of akin to WGAN (Arjovsky et al.) 2017). Recently, Binkowski et al. (2018b); Li et al. (201)) have proposed improvements to stabilize the training of the MMD-GAN method. But these method still rely on solving the same saddle-point problem train the critic.
2 BACKGROUND
Given data x, E RDD for i E (1...N) N} from a distribution of interest with density Pz. the goal of deep generative modeling is learn a parameterized function G, : IR* + RD, called generator network, that maps samples = E R where h D from a noise distribution p= to samples from the model distribution. Since G. defines a new random variable, we denote its density function by qa. and also write N (2), where we suppress the dependency of x" on y. The parameters ? of the generator are chosen to minimize a loss criterion which encourages 92 to match pz.
Interestingly, the training of GRAM-nets heavily relies the use of kernel Gram matrices.
2

---

Published as a conference paper at ICLR 2020
2.1 MAXIMUM MEAN DISCREPANCY
Maximum mean discrepancy measures the discrepancy between two distributions as the maximum difference between the expectations of class of functions F, that is,
MMD sup [E)] -B, fer
(1)
where E denotes expectation. F is chosen to be rich enough class, then MMD q) 0 implies that p = q. Gretton et al. (2012) show that it is sufficient to choose FF to be a unit ball in an reproducing kernel Hilbert space R with characteristic kernel k. Given samples 21 IN p and Yi yM ~ 9. we can estimate MMDR as
M
M
MMD
2 NM
1 M
k My)
k(y, 25)) (2)
N
j=1
2.2 DENSITY RATIO ESTIMATION
Sugiyama et al. (012 present an elegant MMD-based estimator for the ratio between the densities p
and gi r(x)
p
that only needs samples from p and q. This estimator is derived by optimizing
{(z)
S M(rs)))(()) - /
min rE PER
(3)
where k is a kernel function. It is easy to see that at the minimum, we have r p/ This estimator not guaranteed be non-negative. As such, positivity constraint should be imposed if needed.
Sugiyama et al. 2011) suggest that density ratio estimation for distributions p and over RR. can be more accurately done lower dimensional sub-spaces K They propose first learn linear projection to a lower dimensional space by maximizing an f-divergence between the distributions p and q of the projected data and then estimate the ratio of p and q (using direct density ratio estimation). They showed that the projected distributions preserve the original density ratio.
3 METHOD
Our aim will be to enjoy the advantages of MMD-nets, but to improve their sample quality by mapping the data (R²)) into Jower-dimensional space (R*), using critic network fa. before computing the MMD criterion. Because MMD with fixed kernel performs well for lower-dimensional data
(Li et al. 2015; Dziugaite et al. 2015), we hope that by choosing < D, we will improve the performance of the MeD-net. Instead of training fa using adversarial criterion like we aim at a stable training method that avoids the saddle-point optimization for training the critic.
More specifically, unlike the MMD-GAN type methods, instead of maximizing the same MMD criterion that the generator is trained to minimize, we train fo to minimize the squared ratio difference, that is, the difference between density ratios in the original space and in the low-dimensional space induced by fo (Section I. More specifically, let q be the density of the transformed simulated data, i.e., the density of the random variable f' (=)),, where = ~ Ps. Similarly let ý be the density of the transformed data, i.e., the density of the random variable fe(x). The squared ratio difference minimized when 0 is such that P=/9z equals p/q. The motivation is that if density ratios are preserved by fo. then matching the generator to the data in the transformed space will also match in the original space (Section (.2).. The reduced dimension of fo should be chosen to strike trade-off between dimensionality reduction and ability to approximate the ratio. If the data lie on lower-dimensional manifold DR which is the case for e.g. natural images, then it is reasonable to suppose that we can find a critic that strikes good trade-off. To compute this criterion, we need to estimate density ratio p/q. which can be done in closed form using MMD (Section 2.2 The generator is trained from the generator {f [f, (2,)} in the lower dimensional space (Section 3.2.. Our method performs stochastic gradient (SG) optimization on the critic and the generator jointly (Section 3.3
MMD-net to match the transformed data {fo(x.) with transformed outputs
3

---

Published as a conference paper at ICLR 2020
3.1 TRAINING THE CRITIC USING SQUARED RATIO DIFFERENCE
Our principle is to choose fo so that the resulting densities f and & preserve the density ratio between P> and qu. We will choose fa minimize the distance between the two density ratio functions
Ta (z) pa(4)//oo1))
TØ(x) p(fe(z))/((f((z))
One way to measure how well fe preserves density ratios is to use the squared distance
(x(x) p(fo())) 2
(7 - yz (x))
dr.
(4)
(x) (f)()))
This objective is minimized only when the ratios match exactly, that is, when To re for all in the support of Qr- Clearly a distance of zero can be trivially achieved K = D and if so is the identity function.But nontrivial optima can exist as well. For example, suppose that Pa and dr are "intrinsically low dimensional" in the following sense. Suppose K < D, and consider two distributions Po and do over R.K and an injective map T R.K - RRD. Suppose that T maps samples from Po and Yo to samples from Pr and lr. by which we mean (x) = ((DT)po(TT((()) and similarly for Gz- Here JDD) denotes the Jacobian (DDT) v V of T. Then we have that D' is minimized to 0 when ʃe = T-1
However, it is difficult to optimize (0) directly because density ratio estimation in high dimension, where data lives, is known to be hard, i.e. the term (=(wr in is difficult to estimate. We will show how to alleviate this issue next.
Avoiding density ratio estimation in data space: To avoid computing the term P=(x) in (4), we
(x)
expand the square in (4), apply the law of the unconscious statistician and cancel terms out (See appendix A for detailed steps), which yields
P(f((E)) dfo(1) - 2 pll((t))) p(fa(z)) dfo(z), ((Jo(x))
0-0)=C44 ((oo()))
((aa()))
(5)
C -CC[faaoio [alndee)
((fo(z)) ((o(c))
dfo())-1)
where C and C' = C - 1 does not depend on 0. This implies that minimizing D) is equivalent to maximizing the Pearson divergence
PD(p. 1000=aaas (ALSOCE) ( dfo(w) - 1- dfo(t)-1=
a(o(()) d6(o)-1-f(((o(a)
(PJo(x)) - - (f((()))
dfo
between ý and which justifies our terminology of referring to fe as critic function. So we can alternatively interpret our squared ratio distance objective as preferring fe so that the low-dimensional distributions and maximally separated under Pearson divergence. Therefore D can be minimized empirically using samples x
x2 ~ Qz, to maximize the critic loss function M
((
[ro(2))) 1
M
Optimizing this requires way estimate r=())) For this purpose use the density ratio
estimator introduced in Section 2.2, Notice that to compute (7). we need the value of ro only for the points x ...." 2y In other words,
need to approximate the vector rg.0 [re(x?)... ro. (2() IT
Following Sugiyama et al. (2012), we replace the integrals in (3) with Monte Carlo averages over the FollowingSSgiyyma
points f= fe(x%)) and over points fo(=?) ... fo(xh) P. the minimizing values of rg.0 can then be computed as
K
(8)
Here Kngg and Kg denote the Gram matrices defined by [Kggli. ((o(())) and [K 1Kpplis k(fo(=?)) fo(z)))
4

---

Published as a conference paper at ICLR 2020
Substituting these estimates into 7 and adding positivity constraint F7 for using the MMD density ratio estimator (Sugiyama et al. (012), we get
M Σ
1 [(0) M
Iro(x!) 112 + AE 1
(9)
where À a parameter control the constraint, being can be maximised to learn the critic fo. We see that this is an estimator of the Pearson divergence PD(p. ?) in that we are both, averaging over samples from fr. and approximating the density ratio. Thus maximising this objective leads to the preservation of density ratio (Sugiyama et al. 2011.
in all our experiments This objective
3.2 GENERATOR Loss
To train the generator network Gy, we minimize the MMD in the low-dimensional space, where both the generated data and the true data are transformed by fo. In other words, we minimize the MMD between and q. We sample points ...2M ~ Ps from the input distribution of the generator. Then using the empirical estimate 2 of the MMD, we define the generator loss function
N
N
N
M fool
(7)
((o((.))
N
NM
(10)
M
which we minimize with respect to
3.3 THE GENERATIVE RATIO MATCHING ALGORITHM
Finally, the overall training of the critic and the generator proceeds by jointly performing SG optimization on (() and [()). Unlike WGAN (Arjovsky et al. 2017) and MMD-GAN (Li al. 2017), we do not require the use of gradient clipping, feasible set reduction and autoencoding regularization terms. Our algorithm is a simple iterative process.
while converged do
Sample minibatch of data {x Pa and generated samples {x]] qai Using fe to transform data as ) and generated samples (fo
Compute the Gram matrix under Gaussian kernels in the transformed space;
Compute (()) via (8) and (9) and ((7) via (TO) using the same K;
Compute the gradients ge VeL( and gy = v,)(
0 - 0 7799; 7 79
// Perform SG optimisation for 0 and y
end
Algorithm 1: Generative ratio matching
Convergence: If we succeed in matching the generator to the true data in the low-dimensional space, then we have also matched the generator the data the original space, in the limit of infinite data. To see this, suppose that we have 7 and f* such that D (0*) O and that M, MMD(p. 9) = 0. Then for all x, we have Tz (x) = Tg- (x) because (0*) = O, and that ra- (x) 1, because M, = This means that Tx (z) = 1, so we have that Pa: Gz.
3.4 STABILITY GRAM-NETS
Unlike GANs, the GRAM formulation avoids the saddle-point optimization which leads to very stable training of the model. In this section we provide a battery of controlled experiments to
As it is pointed out by one of the reviewer, instead of adding the regularization term, another way to resolve the negativity issue of the ratio estimator to simply clipping it the estimator be positive. This in fact works well in practice and can further improves the stability of training.
5

---

Published as a conference paper at ICLR 2020
empirically demonstrate that GRAM training relying the empirical estimators of MMD and Pearson Divergence (PD), i.e. equations (7) and (9). While the MMD estimator is very extensively studied, novel empirical estimator of PD is not. Therefore now show that in fact maximizing our estimator for PD can be reliably used for training the critie, i.e. minimizes equation
Stability of adversarial methods has been extensively studied before by using simple dataset of eight axis-aligned Gaussian distributions with standard deviations of 0.01, arranged in ring shape (Roth al. 2017 Mescheder et al., 2017; Srivastava et al. 2017). Therefore we train a simple generator using GRAM on this 2D ring dataset to study the stability and accuracy of GRAM-nets. We set the projection dimension K D 2 in order facilitate visualization. Both the generator and the projection networks are implemented as two-layer feed-forward neural networks with two hidden layers of size of 100 and Rel.U activations.
The generator output (orange) is visualized over the real data (blue) in Figure la at 10, 100,10000000 and 10,000th training iterations. The top row visualizes the observation space (R (R²) and the bottom row visualizes the projected space (R*). Note, as the training progress the critic (bottom row) tries to find projections that better separate the data and the generator distributions (especially noticeable in columns and 3). This provides clear and strong gradient signal to the generator optimization that successfully learns to minimize the MMD of the projected distributions and eventually the data and the model distributions well (as shown in the last column). Notice, how in the final column the critic is no longer able to separate the distributions. Throughout this process the critic and the generator objectives consistently minimized. This is clearly shown in Figure Tb which records the values of equations (10) for the generator (orange) loss and equation (5) for the critic objective (blue).
6, povict
10'
105
(a) Data and samples in the original (top) and pro- jected space (bottom) during training: four plots are at iteration 10, 100, 1000 and 10, 000 respec- tively. Notice how the projected space separates p and ς.
20 40 03 00
100
100 1000 2000
3020
4000
5000
(b) Trace of E. and D (equation 55 during train- ing. The left plot is for iteration to 100 and the right plot is for 100 to 5,000, with the same y-axes in the log scale.
Figure 1: Training results with projected dimension fixed to 2.
Next, we compare GRAM-based training against classical adversarial training that involves a zero- sum game. For this purpose we train a GAN with the same exact architecture and hyperparameter as those used for the GRAM-net in the previous experiments. Adversarial models are known to be unstable on the 2D ring dataset (Roth et al. 2017 : Mescheder et al., (017) as the critic can often easily outperform the generator in this simple setting. Figure 2uummarizes the results of this comparison. Both models of the input noise variable (h). GANS are known to be unstable at both low and high dimensional noise but they additionally tend mode-collapse (Srivastava et al. 2017) on high dimensional noise. This is confirmed in our experiments; GANs failed to learn the true data distribution in every case and in most cases the optimization also diverged. In sharp contrast to this, GRAM training successfully converged in all 12 settings. Adversarial training needs a careful balance between the generator and the critic capacities. training become more unstable in GAN. This is not the case for GRAM-nnes, which train well without requiring to make any adjustments in other hyperparameters. This enlists as an important advantage of GRAM, one can use the most powerful model (generator and critic) given their computational budget instead of worrying about balancing the optimization. We also provide the corresponding results for MMD-nets and MMD-GAN in Appendix B.3 in which one can see MMD-nets can also capture all modes, but the learned distribution less separated (or sharp) compared to GRAM-nets and MMD-GANs tend to produces distributions which are too sharp.
trained three different levels of generator capacity and four different dimensions
the plot we can see that, as the capacity of the generator becomes larger, the

---

Published as a conference paper at ICLR 2020
4
16
2
16
20
100
200
(a) GAN
(b) GRAM-net
Figure 2: Training after 2,000 epochs by varying noise dimension h and the hidden layer size of critic model. For each model, each row is different layer size in [20, 100, 200] and each column is different in [2, 4, 8, 16]. Half of the GAN training diverges while GRAM training converges.
2 ~
x Pz
2 Ps
x ~ Pz
fo
x7
fo
(x))
2 ~
x
7
pz
K
fo
, K
fo()
K
Co
by
by
Lo
Lo
GAN
(b) MMD-net
MMD-GAN
(d) GRAM-net
Figure 3: Computation graphs of GAN, MMD-net, MMD-GAN and GRAM-net. K is the kernel Gram matrix. Solid arrows represents the flow the computation and dashed lines represents min-max relationship between the losses, i.e. saddle-poing optimization in which minimizing one loss maximizes the other. Therefore in the zero-sum game case (GAN, MMD-GAN) the two objectives (b, and Lg) cannot be optimized simultaneously (Mescheder et al. 2017)
3.4.1 COMPUTATION GRAPHI
Figure B visualizes the computational graph for GAN, MMD-net, MMD-GAN and GRAM-net. Solid arrows describe the direction of the computation, K denotes the kernel gram matrix and most importantly dashed lines represent the presence of saddlepoint optimization. case GAN and MMD-GAN, the dashed lines imply that by formulation, training the critic adversarially affects the training of the generator as they are trained by minimizing and maximizing the same objective i.c. L, Lg. Both MMD-ne nets and GRAM-nets avoid this saddle- poilee problem. In GRAM-nets the critic and generator do not play a zero-sum game as they are trained to optimize different objectives, i.e. the critic learns find a projection in which the density ratios of the pair of input distributions preserved after the projection and the generator tries to minimize the MMD in the projected space.
4 EXPERIMENTS
In this section we empirically compare GRAM-nets against MMD-GANs and vanilla GANs, on the Ci and CelebA image datasets. Please note that while have tried include maximum number of evaluations in this section itself, due to space limitations, some of the results are made available in the appendix. To evaluate the sample quality and resilience against mode dropping, used Frechet
7

---

Published as a conference paper at ICLR 2020
Inception Distance (FID) (Heusel et al. 2017) Like the Inception Score (IS), FID also leverages pre-trained Inception Net to quantify the quality of the generated samples. but is more robust to noise than IS and can also detect intra-class mode dropping (Lucic et al. 2017). FID first embeds both the real and the generated samples into the feature space of a specific layer of the pre-trained Inception Net. It further assumes this feature space to follow multivariate Gaussian distribution and calculates the mean and covariance for both sets of embeddings. The sample quality is then defined as the Frechet distance between the two Gaussian distributions, which is
(ID Rgg = l1xx, - Mz, 1 + Tr(Es,, + Exz - 2( 2 Sx,
where (Uzz, Eng ). and (Hzzg Ex,) are the mean and covariance of the sample embeddings from the data distribution and model distribution. We report FID on a held-out set that was not used to train the models. We run all the models three times from random initializations and report the mean and standard deviation of FID over the initializations.
Architecture: We test all the methods on the same architectures for the generator and the critic, namely a four-layer DCGAN architecture (Radford al., 2015), because this has been consistently shown to perform well for the datasets that we use. Additionally, to study the effect of changing architecture, we also evaluate a slightly weaker critic, with the same number of layers but half the number of hidden units. Details of the architectures are provided in Appendix [
To facilitate fair comparison with MMD-GAN we set all the hyperparameters shared across the three methods to the values used in et al. (2017). Therefore, we use a learning rate of 5e-b and set the batch size to 64. For the MMD-GAN and GRAM-nets, we used the same set of RBF kernels that were used in Li et al. (2017). We used the implementation of MMD-GANs from Li et al. y We leave all the hyper-parameters that are only used by MMD-GAN to the settings in the authors' original code. For GRAM-nets, we choose K = h, i.e. the projected dimension equals the input noise dimension. We present an evaluation of hyperparameter sensitivity in Section
Hyperparameters:
Table 1:
Sample quality (measured by FID; lower is better) of GRAM-nets compared to GANs.
Table 2: FID with fully convo- lutional architecture originally used by Li et al (017))
Areh.
Dataset
MMD-GAN
GAN
GRAM-net
DCGAN
Cifar10
40.00 1 0.56
26.82 1 0.49
24.85 0.94
Dataset
MMD-GAN
Weaker
Cifar10
210.85 + 8.92
31.64 2.10
24.82 0.62
Cifar10
38.39 + 0.28
DCGAN
CelebA
41.105 = 1.42
30.97 = 5.32
27.04 4.24
CclebA 40.27 + 1.32
4.1
IMAGE QUALITY
We now look at how our method competes against GANs and MMD-GANS on sample quality and mode dropping on Cifar10 and Celeb datasets. Results are shown Table [ Clearly, GRet- outperform both baselines. For CelebA, we do not run experiments using the weaker critic, because this is much larger and higher-dimensional dataset, so a low-capycity critic is unlikely to work well.
It is worth noting that while the difference between the FIDs of GAN and GRAM-net is relatively smaller, it is quite significant that GRAM-net outperforms GAN on both datasets. As shown in large scale study of adversarial generative models (Lucic et al.) 2017), GANs in general perform very well on FID when compared to the state-of-ther the-art methods such as WGAN (Arjovsky et al. (017). Interestingly, in the case CIFAR1O. GANs are the state-of-ane art on FID performance.
To provide evidence that GRAM-nets are not simply memorizing the training set, we note that we measure FID held-out set, so network that memorized the training set would be likely have poor performance. For additional qualitative evidence of this, see Figure [ This figure shows the five nearest neighbors from the training set for 20 randomly generated samples from the trained generator of our GRAM-net. None of the generated images have an exact copy in the training set, and qualitatively the 20 images appear to be fairly diverse.
Note that our architecture is different from that used in the results of Li et al. (0017). That work uses a fully convolutional architecture for both the critic and the generator, which results in an
from "We use standard implementation wailn com/bi finn-jjo/rrUR
AAvallable htttps::/ / / thub. wwwp:///ttobbbreeaan///naaaaaaaa
8

---

Published as a conference paper at ICLR 2020
Figure 4: Nearest training images samples from GRAM-net trained on Cifar10. each column, the top image is sample from the generator, and the images below it are the nearest neighbors.
300
UMG
MND-GA
CAN
292
2206
2000
2000
300
(a) FID vs Learning Rate
(b) FID vs Batch Size
(c) FID vs critic output dimension for GR M-neet.
Figure 5: Hyper- paper-parameter sensitivity of MMD-GAN, GAN and GRAM-net on Cifar10 dataset.
Sample quality measured by FID.
order of magnitude more weights. This makes a large comparison more expensive, and also risks overfitting on a small dataset like Cifar10. However, for completeness, and verify the fairness of our comparison, we also report the FID that we were able to obtain with MMD-GAN on this fully-convolutional architecture in Table Compared to our experiments using MMD-GAN to train the DCGAN architecture, the performance of MMD-GAN with the fully convolutional architecture remains unchanged for the larger CelebA dataset. On Cifar 10, not surprisingly, the larger fully convolutional architecture performs slightly better than the DCGAN architecture trained using MMD- GAN. The difference in FID between the two different architectures is relatively small, justifying our decision to compare the generative training methods on the DCGAN architecture.
4.2 SENSITIVITY TO HYPERPARAMETERS AND EFFECT OF THE CRITIC DIMENSIONALITY
GAN training can be sensitive to the learning rate and the batch size used for training (Lucic et al. 2017). We examine the effect of learning rates and batch sizes on the performance of all three methods. Figure [5a compares the performance as a function of the learning rates. We see that GRAM-nets are much less sensitive the learning rate than MMD-GAN, and are about as robust to changes in the learning rate vanilla GAN. MMD-GAN seems to be especially sensitive to this hyperparameter. We suggest that this might be the case since the critic in MMD GAN restricted to the set of k-Lipschitz continuous functions using gradient clipping, and hence needs lower learning rates. Similarly, Figure 55 shows the effect of batch size on three models. We notice that all models are slightly sensitive to the batch size, and lower batch size is in general better for all methods.
We examine how changing the dimensionality of the critic affects the performance our method. We use the Cifarl0 data. Results are shown in Figure 5c Interestingly, we find that there are two regimes: the output dimensionality steadily improves the FID until K 1000, but larger values sharply degrade performance. This agrees with the intuition in Section B.1 that dimensionality reduction is especially useful for an "intrinsically low dimensional" distribution. For more inspections of the stability of MMD-GANE Appendix C

---

Published as a conference paper at ICLR 2020
5 SUMMARY
We propose a new algorithm for training MMD-nets. While MMD-nets in their original formulation fail generate high dimensional images in good quality, their performance can be greatly improved by training them under a low dimensional mapping. Unlike the alternative adversarial (MMD-GAN) for learning such a mapping, our training method, GRAM, learns this mapping while avoiding the saddle-pon optimization by being trained to match density ratios of the input and the projected pair of distributions. This leads sizable improvement in stability and generative quality, that is par with or better than adversarial generative methods.
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXiv:1701.07875. 2017.
Wasserstein GAN.
arXiv preprint
Mikotaj Binkowski, Dougal J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXiv:1801.01 2018a.
Mikolaj Binkowski, Dougal J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint (7X1v:8801.0140) 2018b.
Vladimir Bogachev. Measure theory, volume 1. Springer Science & Business Media, 2007.
Gintare Karolina Dziugaite, Daniel M Roy, and Zoubin Ghahramani. Training generative neural networks via maximum mean discrepancy optimization. arXiv preprint (505:::5500000000 2015.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- fion processing systems, pp. 2672-2680, 2014.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(Mar):723-773 2012.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. In I. Guyon, U. V. Luxburg, Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 6626-6637. 001-6663.2111 2017.
Diederik P Kingma and Max Welling.
Auto-encoding variational bayes.
arXiv preprint
1312.61 14, 2013.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.
Chun-Liang Li, Wei- Vei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabás Póczos. Mmd gan: Towards deeper understanding of moment matching network. In Advances in Neural Information Processing Systems, pp. 2200-2210, 2017.
Chun-Liang Li, Wei-Cheng Chang, Youssef Mroueh, Yiming Yang, and Barnabás Póczos. Implicit kernel learning. arXiv preprint arXiv: 2019.
Yujia Li, Kevin Swersky, and Rich Zemel. Generative moment matching networks. In International Conference on Machine Learning, pp. 11111777,2015.
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? large-scale study, arXiv preprint arXiv:/711. 10337, 2017.
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. The numerics of gans. In Advances in Neural Information Processing Systems, pp. 1825-1835, 2017.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Advances in Neural Information Processing Systems, pp. 271-279, 2016.
10

---

Published as a conference paper at ICLR 2020
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06444, 2015.
Kevin Roth, Aurelien Lucchi, Sebastian Nowozin, and Thomas Hofmann. Stabilizing training of generative adversarial networks through regularization. In Advances in neural information processing systems, pp. 2018-2028, 2017.
Akash Srivastava, Lazar Valkoz, Chris Russell, Michael U Gutmann, and Charles Sutton. Veegan: Reducing mode collapse in gans using implicit variational learning. In Advances in Neural Information Processing Systems, pp. 3310-3320, 2017.
Masashi Sugiyama, Makoto Yamada, Paul Bünau, Taiji Suzuki, Takafumi Kanamori, and Motoaki Kawanabe. Direct density-ratio estimation with dimensionality reduction via least-squares hetero-distributionaal subspace search. Neural Networks, 24 2:183 98, 2011.
Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density ratio estimation in machine learning. Cambridge University Press, 2012.

---

Published as a conference paper at ICLR 2020
A DERIVATION DETAILS
The derivation from 4 relies on the law of unconscious statistician (LOTUS), and therefore no Jacobian correction needed. Here we show how we applied LOTUS in the derivation, with more detailed steps.
Let F(z)
Fla) [())
Expanding ( we have,
D² / (x(x))
Pa(x)) (x)x))
dx -2 2 2/ / (x(x) Pola) (o) ((o(())
2x(x) ((fo(z)) dr (=(()) ((lo(z))
dr
((fo(z))
f (z(
(x( dr 2 Pa(x) P(Fo(T)) de + (o(x))
(P(f(x)))
dx
(x)
a(fo(z))
((fo(z))
f Po(x))
(x) dr 2 (a( / / (((t)) Qn
We obtain the second line by canceling and rearranging terms and the third line, by plugging in F(z)). Applying Theorem 3.6.1 from Bogachev (2007) (pg. 190) on the third term, J (f(o))))f((z))) with fa. we get
g 22 and
/ (olo))(f(())dd=
(
Similarly, for the second term, with g and for obtain
2 ʃ (=()) p(oo(a))(((()))6600))
Thus,
Po(x) dx 2 ((006)))(((())))000000) fala:aaaaaaaa/aaaooaolaaaa (6166)))
D²( (00 (r)()) P
(x(x)
which is the first line of (
Note that we use LOTUS "in reverse" of its usual application.
B EXPERIMENTAL DETAILS AND MORE RESULTS FOR SECTION 3.4
B.1 EXPERIMENTAL DETAILS
Below list the syper-parameters used in Section 3.4 except from those being varied in the experiments.
Number of epochs: 2,000
Noise distribution: Gaussian
Activation between hidden layers: ReLU
Batch normalisation: not used
Batch size: 200
Batch size for generated samples: 200
GAN
Optimizer: ADAM
Learning rate: le-4
12

---

Published as a conference paper at ICLR 2020
Momentum decay: 0.5
Critic architecture: 2-100-100-10
MMD-net
Optimizer: RMSprop
Learning rate: le-3
RBF kernel bandwidth:
GRAM-net
Optimizer: ADAM
Learning rate: 1e-3
Momentum decay: 0.5
RBF kernel bandwidth:
Critic architecture: 2-100-100-10
Note that we also provide the experimental details for MMD-nets where we will show the results in Appendix B.3
All parameter settings are also available in the examp Sttamples/Hiee weleeeseyyyye... tom] file of our submitted code. All parameters heing varied also available in the examp le es/paralllallla petrraalle__eeee 31 file of our submitted code.
B.2 3D RING DATASET
We also conduct the same experiment as in Figure Ta with a 3-dimensional data space and 2- dimensional projection space. We create 3D ring dataset by augmenting the third dimension with Gaussian noise (0 mean and 0.1 standard deviation), which is then rotated by 60 degrees along the second axis. We repeat the experiments in Figure Ia using this dataset. The results are shown in Figure Unlike the 2D ring example, the optimal choice of the projection function learned here is no
(a) Data and samples in the original (top) and projected space (bottom) during training; four plots iteration 0, 100, 1000 and 10, 000 respectively. Notice how the projected space separates p and q.
Figure 6:
Training results of GRAM-nets with projected dimension fixed to 2 on the 3D ring dataset.
longer the identity function. However, method can stil, easily learn a low-dimensional manifold that tends to preserve the density ratio.
B.3 STABILITY OF MMD-NETS AND MMD-GANS
In this section we repeat the same stability-relater experiments that we conducted on GANs and GRAM-nets in Section 3.4 for MMD-nets and MMD-GANs. Results are shown in Figures 7a and
7b. One can see MMD-nets can capture all the modes, but the learned distribution less separated (or sharp) compared GRAM-nets. This is because the MMD is computed with fixed kernel and in fixed space, which can only distinguish distributions, subject to the set kernels being used.
Figure 7b shows the results MMD-GAN modles that trained for 4,000 epochs, each with
13

---

Published as a conference paper at ICLR 2020
4
16
2
16
20
100
200
(a) MMD-nett nets
(b) MMD-GAN
Figure 7: Corresponding plots Figure for MMD-nets and MMD-GANS.
steps for the critic network and step for the generative network Compared Figure 2b which GRAM-nets only takes 2,000 training epochs with joint step for both networks, even with longer training, the quality of MMD-GAN Notice, how the generated samples of MMD-GAN are similar to the successful runs for GAN in Figure 2a generated samples tend be too concentrated around the mode of the individual clusters. Our method on the other hand, with shorter training time, is clearly able to recover all the 8 clusters along with their spreads.
visually worse than GRAM-nets in this synthetic example.
Note that, unlike GRAM-nets which are robust to changing dimensionality of the projected spaces (2, 8, 16), MMD-GAN training easily diverges for 2-dimensional projected spaces. As a result, for MMD-GAN we only show the results up the iterations before the training diverges. It's likely that any change in the neural network size (output dimension) requires a different set of hyperparameters to make MMD-GA converge. On the other hand, GRAM-nets are robust to these changes, out of the box.
B.4 GRAM-NETS ON THE MNIST DATASET
In this section, we show the results of training GRAM-nets on the MNIST dataset. Figure Sa shows how the generated samples change during the phase of training. Figure 8b shows how the generator loss and the ratio matching objective are being optimized simultancously. The generator 11-6000600-80007775 with ReLU
used in this experiment activations between hidden layers and sigmoid activation for the output layer; the noise distribution multivariate uniform distribution between [-1, with 100 dimensions. For the critic, we use convolution architecture specified in Table 3 For the optimisation, we use ADAM with a learning rate of 1e-3, momentum decay of 0.5, and batch size of 200 for both data and generated samples. For the RBF kernels, we use bandwidths of [0.1, 1, 10, 100]
multilayer perceptron (MLP) of size
C STABILITY OF MMD-GANS
In addition to Figure 7b which evaluates the stability of MMD-GANs in terms of the projected dimension and the generator capacity, we perform qualitative evaluation by visualizing the projected space during training as well the effect of stabilization techniques in this section.
C. PROJECTED SPACE DURING TRAINING
We first perform the qualitative evaluation (same as Figure la and Figure 6 for MMD-GANs on the 2D and 3D ring datasets. The original space and the projected space during training are visualized
When training MMD-GAN, both the auto-encoding loss and the feasible set reduction loss are used. Parameter clipping is done with -0.1 (lower) and 0.1 (upper). Learning rate is set to 5e-5 and other parameters the same GRAM-nets.
14

---

Published as a conference paper at ICLR 2020
0
6
3
3
4
8
0
7
6
3
H
7
8
0
3
3
H
8
0
O
0
7
4
4
3
0
O
0
4
4
3
0
0
7
3
6
6
3
8
1
6
2
8
6
6
5
8
/
3
7
9
/
1
/
4
9
3
3
8
X
&
3
4
8
3-
0
7
G
7
&
1
7
6
7
i/
5
5
9
8
#
+
0
G
7
7
C
9
4
5
0
7
3
3
7
0
7
6
H
7
%
0
7
3
3
%
0
0
7
4
0
0
0
7
4
4
0
0
7
6
6
2
3
8
1
0
6
6
2
8
O
6
6
2
3
8
O
/
8
3
4
9 [
8
/
1
/
8
3
4
9 3 3
8
/
8
3
4
8
/ 7
7
/
2
5
2
2
5
0
5
Y
3
7
6
/
7
2
4
7
4
6
7
7
7
5
4
7
2
7
3
2
0
Q
O
Y 7
8
7
3 0
3
7
Q
0
7
6
9
2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
10
0
6
3
3
H
%
0
7
6
3
H
7 4
%
0
3
3
H
g
0
0
0
7
4
4
3
0
0
0
7
4
3
0
0
9
0
7
3
6
6
3
8
1
6
2
8
O
6
6
3
8
/
8
3
4
9
/
1
/
4
9
8
/
8
3
2
9
0
6
7
/
0
3
9 6 6 7
3
S
O
5
7
6
5
G
9
3 7
9
0
0
6
{ 3
7
7
3
O
7
O
S
S
9
1
4
/
6
7
5
0
4 0
7
6
S
b
O
5
7
3
9
5
0
7
0
7
(a) Data samples (top and bottom half in each plot) during training iteration 100, 250, 500 (first row), 750, 1,000, 2,000 (second row), and 3,000, 4,000 and 5,000 (last row). The orders for each row are from left to right.
195
131
135
(b) Trace of É., and D" (equation 55) during training. The left plot is for iteration 100 and the right plot for 100 to 5,000, with the same y-axes the log scale.
Figure 8: Training results of GRAM-nets the MNIST dataset.
in Figure 9 It is clear that the projected spaces quite different between MMD-GAN and our method. Unlike GRAM-nets, the projected space, the generated samples do not overlap with the data samples.
15

---

Published as a conference paper at ICLR 2020
Op
Input
Oufput
Filter
Pooling
Padding
Reshape
784
(-1,28,28,1)
Conv2D + BatchNorm
1
16
3
2
Conv2D + BatchNorm
16
32
3
2
Conv2D + BatchNorm
32
32
3
2
Reshape
(-1,3,3,32)
(-1,288)
Linear
288
100
Table 3: Critic architecture for MNIST. All BatchNorm are followed by ReLU activation.
(a) 2D Ring
(b) 3D Ring
Figure 9: Training of MMD-GAN with projected dimension fixed to before diverging. Data and samples in the original (top) and projected space (bottom) during training, four plots at iteration 100, 500 and 1, 000 respectively. Notice how the projected space separates p and Ø.
C.2 STABILIZATION TECHNIQUES
We also evaluate the effect of the various stabilization techniques used for training, namely the autoencoder penalty (AE) and the feasible set reduction (FSR) techniques from (Li et al. (017)) Cifarl0 data over two settings the batch size. Table 4 shows the results. The performance of
Table Performance of MMD-GAN (Inception scores; larger is better) for MMD-GAN with and without additional penalty terms: feasible set reduction (FSR) and the autoencoding loss (AE), The full MMD-GAN method is MMD+ISR+AE.
Batch Size
MMD+FSR
MMD+AE
MMD
64
5.35 + 0.05
5,40 0.04
3.26
0.03
3.51 0.03
300
5.43 + 0.03
5.15 + 0.06
3.68 + 0.22
3.87 + 0.03
MMD-GAN training clearly relics heavily on FSR. This penalty not only stabilizes the critic but also provides additional learning signal to the generator. Because these penalties are important to the performance MMD-GANs, requires tuning several weighting parameters, which need to be set carefully for successful training.
We would like to re-emphasize the stability of GRAM-nets with respect to different settings of network sizes, noise dimension, projected dimension, learning rate, batch size without relying regularization terms with additional hyperparameters.
D ARCHITECTURE
For the generator used in Section 4 used the following DCGAN architecture,
16

---

Published as a conference paper at ICLR 2020
Op
Input
Output
Filter
Stride
Padding
Linear
128
2048
Reshape
2048
(-1,4,4,128)
Conn2D_transpose 128
64
2
SAME
Conv2D
Con>2D_transpose transpose
64
32
2
SAME
32
2
SAME
Table DCGAN generator architecture for Cifar10.
We used two different architectures for the experiments on Cifar 10 dataset. Table 6 shows the standard DCGAN discriminator that was used. Table shows the architecture of the weaker DCGAN
Op
Input
Output
Filter
Stride
Padding
Conv2I)
32
4
SAME
Conv2D
32
64
4
SAME
Conv2D
64
128
4
SAME
Flatten
128
2048
Linear
2048
128
Table 6: DCGAN discriminator architecture for Cifar10.
discriminator architecture that also used for Cifarl0 experiments. While leaky-ReLU was used
Op
Input
Output
Filter
Stride
Padding
Conv2D
3
32
4
2
SAME
Conv2D
32
32
4
2
SAME
Conv2D
32
64
4
SAME
Flatten
64
1024
Linear
1024
128
Table 7: Shallow DCGAN discriminator architecture.
as non-linearee non-linearity in the discriminator, ReLU was used in the generator, except for the last layer, where was tanh. Batchnorm was used in both the generator and the discriminator.
E
SAMPLES
We show some the randomly generated samples from our method on CIFAR1O and CelebA in Figure T0
F INCEPTION SCORE
Inception score (IS) is another evaluation metric for quantifying the sample quality in GANs. Com- pared FID, IS is not very robust to noise and cannot account for mode dropping. In addition the FID scores that we provide in the paper, here we also report IS for all the methods on CIFAR10 for completeness since the MMD-GAN paper used it as their evaluation criteria.
17

---

Published as a conference paper at ICLR 2020
CIFAR10
(b) CelebA
Figure 10: Random Samples from randomly selected epoch (>100).
Table 8: Inception Scores for MMD-GAN, GAN, GRAM-net and MMD-nets on CIFAR10 for three random initializations.
MMD-GAN
GAN
GRAM-net 5.73 1:2 0.10
5.35 0.12
5.17
Inception Score
5.21 0.14
4.94
5.44 + 0.12
5.31 +.10
5.27 0.05
5.45 + 0.18
18


Reviews:
[{"review_id": "HkebyY-VFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The proposed approach amounts to a non-adversarial way to update the parameters theta of a kernel K_theta(x, y) = k(f_theta(x), f_theta(y)) in an MMD-net, rather than adversarially as in MMD-GANs. Avoiding the saddle point problem clearly gives a more stable optimization problem, as shown in Section 3.4, which is very nice.\n\n\nTheory/motivation:\n\nThe motivation based on density ratio estimation, however, very much assumes that these densities exist. As convincingly argued by Arjovsky and Bottou (ICLR 17, \"Towards principled methods for training generative adversarial networks\"), the densities p_x, q_x of (4) typically do not exist (w.r.t. Lebesgue measure), and moreover the support of the distributions are typically almost disjoint. Algorithmically, this doesn't seem to be a major issue for your approach: the estimator (8) corresponds to some kind of smoothing by the kernel. But it would be nice from a motivation point of view to have a better understanding of:\n\n- What assumptions exactly does (8) make about the existence of densities? For example, in the case of a Gaussian kernel, does it correspond to the distance between distributions convolved with Gaussian noises with a smaller Gaussian kernel? Does the algorithm \"make sense\" when distributions don't exist?\n\n- Can we say something about when a density exists for the dimensionality-reduced data f_theta(x) even when it doesn't exist for the data x? \n\n- Relatedly, the dimension-reduced Pearson divergence is clearly somehow different from the plain Pearson divergence. Can we understand a little bit more how it's different? How does the choice of architecture of f_theta affect the distance metric?\n\n- Is it the case, as in Arjovsky et al. (2017) or in Arbel et al. (NeurIPS 2018, \"On gradient regularizers for MMD GANs\"), that the overall loss function with an optimal f_theta is continuous with respect to the GAN distribution?\n\nCertainly not all of these theoretical questions require answers for an initial ICLR paper, but they would be illuminating for understanding the method.\n\n\nMotivating experiments:\n\n- In Figure 2, it seems that except in the first image (h=2) the mapping f_theta becomes essentially the identity. Is this expected to be generally true in your 2d cases, because the density ratio can already be well-represented there? A more interesting example for this aspect of the model might involve higher-dimensional distributions with some low-dimensional manifold structure; would f_theta pick up some reasonable representation of the manifold in those cases?\n\n- Can you plot the density ratio estimate in a simple 2d example like this? How does it look visually? Is a good estimate necessary for good performance of your generative model?\n\n\nExperiments:\n\n- As shown by Binkowski et al., it's extremely important to specify the sample size for FID scores, which you don't do.\n\n- That said, the FID scores of the MMD GAN models of Binkowski et al using DCGAN discriminators (which replace the FSR and AE penalties of Li et al. with a single gradient penalty) found substantially smaller penalties for performance of MMD-GANs with using a smaller discriminator (1/4 as many hidden units rather than 1/2 as you use) than you did. They also reported substantially better numbers on CelebA than you did. If not too difficult code-wise, it might be worth trying either their code or that of the followup Arbel et al. (citation above) to compare to your approach.\n\n- The results of, for example, Mescheder et al. (ICML-18, \"Which Training Methods for GANs do actually Converge?\") and Arbel et al. (NeurIPS-18, \"On gradient regularizers for MMD GANs\") seem to be much better visually with comparably-sized models than the results of Figure 8 (and of course StyleGAN/etc are *far* better but with drastically larger models). It would be good to compare empirically to some more recent models than original MMD GANs (whose empirical results were substantially improved with only minor tweaks by Binkowski et al. mere months later) or DCGAN, which are fairly old approaches in this space.\n\n- Two more extremely relevant methods it would be worth comparing to or at least mentioning:\n\n   - Ravuri et al. (ICML 2018, \"Learning Implicit Generative Models with the Method of Learned Moments\") can be interpreted as learning f_theta for an MMD-net as you do by taking the gradients of an auxiliary classification network. Although it is still somewhat \"adversarial,\" they need to update the kernel function extremely rarely (every 2,000 generator update steps) and the optimization seems overall much more stable.\n\n   - dos Santos et al. (https://arxiv.org/abs/1904.02762 \"Learning Implicit Generative Models by Matching Perceptual Features\") effectively use an MMD-net (with some slight tweaks) with a *fixed* kernel. Though results aren't quite as good, it's worth at least mentioning.\n\n\nMinor typos/etc:\n\n- Top of page 3: it is sufficient to choose F a unit ball in an RKHS with a _characteristic_ kernel.\n- Page 4: \"this issue in next.\"\n- Figure 1 caption: (both) should probably be (bottom).\n\n\nOverall initial thoughts:\n\nThis seems like a nice alternative to adversarial methods, but it does not compare to more recent (last 2 years) models in this space or very thoroughly establish the applicability of its motivation. I think it's worthy of a weak accept as is, but could be much more convinced with some additional work.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BylSbyR3YH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes GRAM-nets using MMD as the critic to train GANs. Similar to MMD-GAN, the MMD is computed on a projected lower dimensional space to prevent the kernel struggle in the high dimensional observed space. On the other hand, contrary to MMD-GAN, GRAM-nets trains the projection f_{\\theta} trying to matching the density ratio of p/q between the observed and the latent space. The proposed density ratio matching criterion avoids the adversarial training in MMD-GAN, thus can potentially fix the two-player optimization problem. The paper shows improved FID scores and nice-looking CIFAR10 generations. \n\nStrengths, \n1, Matching density ratios is a novel and interesting idea. If the data lives in a lower dimensional subspace, matching density ratio could probably reveal the subspace. Compared to adversarially training f_theta, the proposed approach could lead to more stable training potentially. \n2, By manipulating E (px/qx - pz/qz)^2, they avoid estimating the high-dimensional px/qx and only estimate pz/qz.\n\n\nWeakness,\n1, The paper needs to be more careful with mathematical expressions. 1) Eq(2) should be MMD^2, instead of MMD. 2) Eq(5) and Eq(6) are wrong, although the used Eq(7) becomes true again. In Eq(5), Eq(6), the integration should be over f(x) instead of x, that is to say $\\int ..... df(x)$.\n2, It is unclear why one needs the regularization in Eq(9). In fact, the major problem of the density ratio estimator lies in that r(x) might be negative, so a clipping might be useful.\n3, The major contribution of GRAM-nets lies in removing the adversarial training in MMD-GAN. Therefore, more empirical comparisons should be made with MMD-GAN. For example, how MMD-GAN evolves in Figure 2 is necessary.\n4, In real image generation tasks, it is beneficial to show the stability of training GRAM-nets, compared with GAN, MMD-GAN as well as WGAN-GP; And Inception scores should also be reported for better validating the effectiveness of the proposed method.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BkgbEu5dKS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, authors propose a new generative adversarial networks based on density-ratio estimation. More specifically, the squared-loss between the true ratio function and the model  (a.k.a., Pearson divergence) is employed to train model. Through experiments, authors demonstrate that the proposed density-ratio based approach compares favorably with existing algorithm.\n\nExperimentally, the proposed model performs pretty nicely. However, the derivation is not certain and some important density-ratio based GAN is missing.\n\nDetailed comments:\n1. Some important density-ratio based GAN methods are missing. For instance, https://arxiv.org/abs/1610.02920 https://arxiv.org/abs/1610.03483\n2. Some derivation of density-ratio estimation is not clear. To the best of my knowledge, the ratio model is obtained by minimizing the distance between true ratio function and its ratio model (Something similar to (4)). But, in this paper, the authors describe the ratio is obtained by using MMD density ratio estimator. Could you explain the derivation in detail?  \n3. I guess, (8) is derived in similar way of  http://www.jmlr.org/papers/volume10/kanamori09a/kanamori09a.pdf In this case, (8) is not correct.\n4. The ratio is defined by using kernel. Thus, I guess the kernel model is employed. However, this information is missing. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a training method for generative adversarial network that avoids solving a zero-sum game between the generator and the critic, hence leading to more stable optimization problems. It is similar to MMD-GAN, in which MMD is computed on a projected low-dim space, but the projection is trained to match the density ratio between the observed and the latent space.
The reviewers raised several questions. Most of them have been addressed after several rounds of discussions. Overall, they are all positive about this paper, so I recommend acceptance. I encourage the authors to incorporate those discussions in their revised paper.
Decision: Accept (Poster)