Title: IMPROVING MMD-GAN TRAINING WITH REPULSIVE LOSS FUNCTION

Abstract: Generative adversarial nets (GANs) are widely used to learn the data sampling process and their performance may heavily depend on the loss functions, given a limited computational budget. This study revisits MMD-GAN that uses the maximum mean discrepancy (MMD) as the loss function for GAN and makes two contributions. First, we argue that the existing MMD loss function may discourage the learning of fine details in data as it attempts to contract the discriminator outputs of real data. To address this issue, we propose a repulsive loss function to actively learn the difference among the real data by simply rearranging the terms in MMD. Second, inspired by the hinge loss, we propose a bounded Gaussian kernel to stabilize the training of MMD-GAN with the repulsive loss function. The proposed methods are applied to the unsupervised image generation tasks on CIFAR-10, STL-10, CelebA, and LSUN bedroom datasets. Results show that the repulsive loss function significantly improves over the MMD loss at no additional computational cost and outperforms other representative loss functions. The proposed methods achieve an FID score of 16.21 on the CIFAR-10 dataset using a single DCGAN network and spectral normalization.

1. INTRODUCTION:

Generative adversarial nets (GANs) (Goodfellow et al. (2014) ) are a branch of generative models that learns to mimic the real data generating process. GANs have been intensively studied in recent years, with a variety of successful applications (Karras et al. (2018) ; Li et al. (2017b) ; Lai et al. (2017) ; Zhu et al. (2017) ; Ho & Ermon (2016) ). The idea of GANs is to jointly train a generator network that attempts to produce artificial samples, and a discriminator network or critic that distinguishes the generated samples from the real ones. Compared to maximum likelihood based methods, GANs tend to produce samples with sharper and more vivid details but require more efforts to train.

Recent studies on improving GAN training have mainly focused on designing loss functions, network architectures and training procedures. The loss function, or simply loss, defines quantitatively the difference of discriminator outputs between real and generated samples. The gradients of loss functions are used to train the generator and discriminator. This study focuses on a loss function called maximum mean discrepancy (MMD), which is well known as the distance metric between two probability distributions and widely applied in kernel two-sample test (Gretton et al. (2012) ). Theoretically, MMD reaches its global minimum zero if and only if the two distributions are equal. Thus, MMD has been applied to compare the generated samples to real ones directly (Li et al. (2015) ; Dziugaite et al. (2015) ) and extended as the loss function to the GAN framework recently (Unterthiner et al. (2018) ; Li et al. (2017a) ; Bińkowski et al. (2018) ).

In this paper, we interpret the optimization of MMD loss by the discriminator as a combination of attraction and repulsion processes, similar to that of linear discriminant analysis. We argue that the existing MMD loss may discourage the learning of fine details in data, as the discriminator attempts to minimize the within-group variance of its outputs for the real data. To address this issue, we propose a repulsive loss for the discriminator that explicitly explores the differences among real data. The proposed loss achieved significant improvements over the MMD loss on image generation k•D (P X , P G ) reaches its minimum if and only if P X = P G (Li et al. (2017a) ). Thus, the objective functions for G and D could be (Li et al. (2017a) ; Bińkowski et al. (2018)) :

min G L mmd G = M 2 k•D (P X , P G ) = E P G [k D (y, y )] -2E P X ,P G [k D (x, y)] + E P X [k D (x, x )] (2) min D L att D = -M 2 k•D (P X , P G ) = 2E P X ,P G [k D (x, y)] -E P X [k D (x, x )] -E P G [k D (y, y )]

(3) MMD-GAN has been shown to be more effective than the model that directly uses MMD as the loss function for the generator G (Li et al. (2017a) ). Liu et al. (2017) showed that MMD and Wasserstein metric are weaker objective functions for GAN than the Jensen-Shannon (JS) divergence (related to minimax loss) and total variation (TV) distance (related to hinge loss). The reason is that convergence of P G to P X in JS-divergence and TV distance also implies convergence in MMD and Wasserstein metric. Weak metrics are desirable as they provide more information on adjusting the model to fit the data distribution (Liu et al. (2017) ). Nagarajan & Kolter (2017) proved that the GAN trained using the minimax loss and gradient updates on model parameters is locally exponentially stable near equilibrium, while the GAN using Wasserstein loss is not. In Appendix A, we demonstrate that the MMD-GAN trained by gradient descent is locally exponentially stable near equilibrium.

3. REPULSIVE LOSS FUNCTION:

In this section, we interpret the training of MMD-GAN (using L att D and L mmd G ) as a combination of attraction and repulsion processes, and propose a novel repulsive loss function for the discriminator by rearranging the components in L att D . First, consider a linear discriminant analysis (LDA) model as the discriminator. The task is to find a projection w to maximize the between-group variance w T µ x -w T µ y and minimize the withingroup variance w T (Σ x + Σ y )w, where µ and Σ are group mean and covariance.

In MMD-GAN, the neural-network discriminator works in a similar way as LDA. By minimizing L att D , the discriminator D tackles two tasks: 1) D reduces E P X ,P G [k D (x, y)], i.e., causes the two groups {D(x)} and {D(y)} to repel each other (see Fig. 1a orange arrows), or maximize betweengroup variance; and 2) D increases E P X [k D (x, x )] and E P G [k(y, y )], i.e. contracts {D(x)} and {D(y)} within each group (see Fig. 1a blue arrows), or minimize the within-group variance. We refer to loss functions that contract real data scores as attractive losses.

We argue that the attractive loss L att D (Eq. 3) has two issues that may slow down the GAN training:

1. The discriminator D may focus more on the similarities among real samples (in order to contract {D(x)}) than the fine details that separate them. Initially, G produces low-quality samples and it may be adequate for D to learn the common features of {x} in order to distinguish between {x} and {y}. Only when {D(y)} is sufficiently close to {D(x)} will D learn the fine details of {x} to be able to separate {D(x)} from {D(y)}. Consequently, D may leave out some fine details in real samples, thus G has no access to them during training.

2. As shown in Fig. 1a , the gradients on D(y) from the attraction (blue arrows) and repulsion (orange arrows) terms in L att D (and thus L mmd G ) may have opposite directions during training. Their summation may be small in magnitude even when D(y) is far away from D(x), which may cause G to stagnate locally.

Therefore, we propose a repulsive loss for D to encourage repulsion of the real data scores {D(x)}:

EQUATION

The generator G uses the same MMD loss L mmd G as before (see Eq. 2). Thus, the adversary lies in the fact that D contracts {D(y)} via maximizing E P G [k D (y, y )] (see Fig. 1b ) while G expands {D(y)} (see Fig. 1c ). Additionally, D also learns to separate the real data by minimizing E P X [k D (x, x )], which actively explores the fine details in real samples and may result in more meaningful gradients for G. Note that in Eq. 4, D does not explicitly push the average score of {D(y)} away from that of {D(x)} because it may have no effect on the pair-wise sample distances. But G aims to match the average scores of both groups. Thus, we believe, compared to the model using L mmd G and L att D , our model of L mmd G and L rep D is less likely to yield opposite gradients when {D(y)} and {D(x)} are distinct (see Fig. 1c ). In Appendix A, we demonstrate that GAN trained using gradient descent and the repulsive MMD loss (L rep D , L mmd G ) is locally exponentially stable near equilibrium.

At last, we identify a general form of loss function for the discriminator D:

L D,λ = λE P X [k D (x, x )] -(λ -1)E P X ,P G [k D (x, y)] -E P G [k D (y, y )]

(5) where λ is a hyper-parameter2 . When λ < 0, the discriminator loss L D,λ is attractive, with λ = -1 corresponding to the original MMD loss L att D in Eq. 3; when λ > 0, L D,λ is repulsive and λ = 1 corresponds to L rep D in Eq. 4. It is interesting that when λ > 1, the discriminator explicitly contracts {D(x)} and {D(y)} via maximizing E P X ,P G [k D (x, y)], which may work as a penalty that prevents the pairwise distances of {D(x)} from increasing too fast. Note that L D,λ has the same computational cost as L att D (Eq. 3) as we only rearranged the terms in L att D .

4. REGULARIZATION ON MMD AND DISCRIMINATOR:

In this section, we propose two approaches to stabilize the training of MMD-GAN: 1) a bounded kernel to avoid the saturation issue caused by an over-confident discriminator; and 2) a generalized power iteration method to estimate the spectral norm of a convolutional kernel, which was used in spectral normalization on the discriminator in all experiments in this study unless specified otherwise.

4.1. KERNEL IN MMD:

For MMD-GAN, the following two kernels have been used:

• Gaussian radial basis function (RBF), or Gaussian kernel (Li et al. (2017a) ),

k rbf σ (a, b) = exp(-1 2σ 2 a -b 2 )

where σ > 0 is the kernel scale or bandwidth.

• Rational quadratic kernel (Bińkowski et al. (2018) ),

k rq α (a, b) = (1 + 1 2α a -b 2 ) -α

, where the kernel scale α > 0 corresponds to a mixture of Gaussian kernels with a Gamma(α, 1) prior on the inverse kernel scales σ -1 . It is interesting that both studies used a linear combination of kernels with five different kernel scales, i.e., k rbf = 5 i=1 k rbf σi and k rq = 5 i=1 k rq αi , where σ i ∈ {1, 2, 4, 8, 16}, α i ∈ {0.2, 0.5, 1, 2, 5} (see Fig. 2a and 2c for illustration). We suspect the reason is that a single kernel k(a, b) is saturated when the distance ab is either too large or too small compared to the kernel scale (see Fig. 2b and 2d ), which may cause diminishing gradients during training. Both Li et al. (2017a) and Bińkowski et al. (2018) applied penalties on the discriminator parameters but not to the MMD loss itself. Thus the saturation issue may still exist. Using a linear combination of kernels with different kernel scales may alleviate this issue but not eradicate it.

Inspired by the hinge loss (see Appendix B.1), we propose a bounded RBF (RBF-B) kernel for the discriminator. The idea is to prevent D from pushing {D(x)} too far away from {D(y)} and causing saturation. For L att D in Eq. 3, the RBF-B kernel is:

EQUATION

For L rep D in Eq. 4, the RBF-B kernel is:

k rbf-b σ (a, b) = exp(-1 2σ 2 max( a -b 2 , b l )) if a, b ∈ {D(y)} exp(-1 2σ 2 min( a -b 2 , b u )) if a, b ∈ {D(x)} (7)

where b l and b u are the lower and upper bounds. As such, a single kernel is sufficient and we set σ = 1, b l = 0.25 and b u = 4 in all experiments for simplicity and leave their tuning for future work. It should be noted that, like the case of hinge loss, the RBF-B kernel is used only for the discriminator to prevent it from being over-confident. The generator is always trained using the original RBF kernel, thus we retain the interpretation of MMD loss L mmd G as a metric.

RBF-B kernel is among many methods to address the saturation issue and stabilize MMD-GAN training. We found random sampling kernel scale, instance noise (Sønderby et al. ( 2017)) and label smoothing (Szegedy et al. (2016) ; Salimans et al. (2016) ) may also improve the model performance and stability. However, the computational cost of RBF-B kernel is relatively low.

4.2. SPECTRAL NORMALIZATION IN DISCRIMINATOR:

Without any Lipschitz constraints, the discriminator D may simply increase the magnitude of its outputs to minimize the discriminator loss, causing unstable training3 . Spectral normalization divides the weight matrix of each layer by its spectral norm, which imposes an upper bound on the magnitudes of outputs and gradients at each layer of D (Miyato et al. (2018) ). However, to estimate the spectral norm of a convolution kernel, Miyato et al. (2018) reshaped the kernel into a matrix. We propose a generalized power iteration method to directly estimate the spectral norm of a convolution kernel (see Appendix C for details) and applied spectral normalization to the discriminator in all experiments. In Appendix D.1, we explore using gradient penalty to impose the Lipschitz constraint (Gulrajani et al. (2017) ; Bińkowski et al. (2018) ; Arbel et al. (2018) ) for the proposed repulsive loss.

5. EXPERIMENTS:

In this section, we empirically evaluate the proposed 1) repulsive loss L rep D (Eq. 4) on unsupervised training of GAN for image generation tasks; and 2) RBF-B kernel to stabilize MMD-GAN training. The generalized power iteration method is evaluated in Appendix C.3. To show the efficacy of L rep D , we compared the loss functions (L rep D , L mmd G ) using Gaussian kernel (MMD-rep) with (L att D , L mmd G ) using Gaussian kernel (MMD-rbf) (Li et al. (2017a) ) and rational quadratic kernel (MMD-rq) (Bińkowski et al. (2018) ), as well as non-saturating loss (Goodfellow et al. (2014) ) and hinge loss (Tran et al. (2017) ). To show the efficacy of RBF-B kernel, we applied it to both L att D and L rep D , resulting in two methods MMD-rbf-b and MMD-rep-b. The Wasserstein loss was excluded for comparison because it cannot be directly used with spectral normalization ( Miyato et al. (2018) ).

5.1. EXPERIMENT SETUP:



Dataset::

The loss functions were evaluated on four datasets: 1) CIFAR-10 (50K images, 32 × 32 pixels) (Krizhevsky & Hinton (2009) ); 2) STL-10 (100K images, 48 × 48 pixels) (Coates et al. (2011)) ; 3) CelebA (about 203K images, 64 × 64 pixels) (Liu et al. (2015) ); and 4) LSUN bedrooms (around 3 million images, 64×64 pixels) (Yu et al. (2015) ). The images were scaled to range [-1, 1] to avoid numeric issues.

Network architecture: The DCGAN (Radford et al. (2016) ) architecture was used with hyperparameters from Miyato et al. (2018) (see Appendix B.2 for details). In all experiments, batch normalization (BN) (Ioffe & Szegedy (2015) ) was used in the generator, and spectral normalization with the generalized power iteration (see Appendix C) in the discriminator. For MMD related losses, the dimension of discriminator output layer was set to 16; for non-saturating loss and hinge loss, it was 1. In Appendix D.2, we investigate the impact of discriminator output dimension on the performance of repulsive loss. The models here differ only by the loss functions and dimension of discriminator outputs. See Section 5.1.

2 For CelebA and LSUN-bedroom, IS is not meaningful (Bińkowski et al. (2018) ) and thus omitted.

3 On LSUN-bedroom, MMD-rbf and MMD-rq did not achieve reasonable results and thus are omitted.

Hyper-parameters: We used Adam optimizer (Kingma & Ba ( 2015)) with momentum parameters β 1 = 0.5, β 2 = 0.999; two-timescale update rule (TTUR) (Heusel et al. (2017) ) with two learning rates (ρ D , ρ G ) chosen from {1e-4, 2e-4, 5e-4, 1e-3} (16 combinations in total); and batch size 64.

Fine-tuning on learning rates may improve the model performance, but constant learning rates were used for simplicity. All models were trained for 100K iterations on CIFAR-10, STL-10, CelebA and LSUN bedroom datasets, with n dis = 1, i.e., one discriminator update per generator update4 . For MMD-rbf, the kernel scales

σ i ∈ {1, √ 2, 2, 2 √ 2, 4}

were used due to a better performance than the original values used in Li et al. (2017a) . For MMD-rq, α i ∈ {0.2, 0.5, 1, 2, 5}. For MMD-rbf-b, MMD-rep, MMD-rep-b, a single Gaussian kernel with σ = 1 was used.

Evaluation metrics: Inception score (IS) (Salimans et al. (2016) ), Fréchet Inception distance (FID) (Heusel et al. (2017) ) and multi-scale structural similarity (MS-SSIM) (Wang et al. (2003) ) were used for quantitative evaluation. Both IS and FID are calculated using a pre-trained Inception model (Szegedy et al. (2016) ). Higher IS and lower FID scores indicate better image quality. MS-SSIM calculates the pair-wise image similarity and is used to detect mode collapses among images of the same class (Odena et al. (2017) ). Lower MS-SSIM values indicate perceptually more diverse images. For each model, 50K randomly generated samples and 50K real samples were used to calculate IS, FID and MS-SSIM.

5.2. QUANTITATIVE ANALYSIS:

Table 1 shows the Inception score, FID and MS-SSIM of applying different loss functions on the benchmark datasets with the optimal learning rate combinations tested experimentally. Note that the same training setup (i.e., DCGAN + BN + SN + TTUR) was applied for each loss function. We observed that: 1) MMD-rep and MMD-rep-b performed significantly better than MMD-rbf and MMD-rbf-b respectively, showing the proposed repulsive loss L rep D (Eq. 4) greatly improved over the attractive loss L att D (Eq. 3); 2) Using a single kernel, MMD-rbf-b performed better than MMD-rbf and MMD-rq which used a linear combination of five kernels, indicating that the kernel saturation may be an issue that slows down MMD-GAN training; 3) MMD-rep-b performed comparable or better than MMD-rep on benchmark datasets where we found the RBF-B kernel managed to stabilize MMD-GAN training using repulsive loss. 4) MMD-rep and MMD-rep-b performed significantly better than the non-saturating and hinge losses, showing the efficacy of the proposed repulsive loss.

Additionally, we trained MMD-GAN using the general loss L D,λ (Eq. 5) for discriminator and L mmd G (Eq. 2) for generator on the CIFAR-10 dataset. Fig. 3 shows the influence of λ on the performance ), in the order of (1e-4, 1e-4), (1e-4, 2e-4),...,(1e-3, 1e-3). The discriminator was trained using L D,λ (Eq. 5) with λ ∈ {-1, -0.5, 0, 0.5, 1, 2}, and generator using L mmd G (Eq. 2). We use the FID> 30 to indicate that the model diverged or produced poor results.

λ = -1 λ = -0.5 λ = 0 λ = 0.5 λ = 1 λ = 2

of MMD-GAN with RBF and RBF-B kernel5 . Note that when λ = -1, the models are essentially MMD-rbf (with a single Gaussian kernel) and MMD-rbf-b when RBF and RBF-B kernel are used respectively. We observed that: 1) the model performed well using repulsive loss (i.e., λ ≥ 0), with λ = 0.5, 1 slightly better than λ = -0.5, 0, 2; 2) the MMD-rbf model can be significantly improved by simply increasing λ from -1 to -0.5, which reduces the attraction of discriminator on real sample scores; 3) larger λ may lead to more diverged models, possibly because the discriminator focuses more on expanding the real sample scores over adversarial learning; note when λ 1, the model would simply learn to expand all real sample scores and pull the generated sample scores to real samples', which is a divergent process; 4) the RBF-B kernel managed to stabilize MMD-rep for most diverged cases but may occasionally cause the FID score to rise up.

The proposed methods were further evaluated in Appendix A, C and D. In Appendix A.2, we used a simulation study to show the local stability of MMD-rep trained by gradient descent, while its global stability is not guaranteed as bad initialization may lead to trivial solutions. The problem may be alleviated by adjusting the learning rate for generator. In Appendix C.3, we showed the proposed generalized power iteration (Section 4.2) imposes a stronger Lipschitz constraint than the method in Miyato et al. (2018) , and benefited MMD-GAN training using the repulsive loss. Moreover, the RBF-B kernel managed to stabilize the MMD-GAN training for various configurations of the spectral normalization method. In Appendix D.1, we showed the gradient penalty can also be used with the repulsive loss. In Appendix D.2, we showed that it was better to use more than one neuron at the discriminator output layer for the repulsive loss.

5.3. QUALITATIVE ANALYSIS:

The discriminator outputs may be interpreted as a learned representation of the input samples. Fig. 4 visualizes the discriminator outputs learned by the MMD-rbf and proposed MMD-rep methods on CIFAR-10 dataset using t-SNE (van der Maaten (2014)). MMD-rbf ignored the class structure in data (see Fig. 4a ) while MMD-rep learned to concentrate the data from the same class and separate different classes to some extent (Fig. 4b ). This is because the discriminator D has to actively learn Published as a conference paper at ICLR 2019 the data structure in order to expands the real sample scores {D(x)}. Thus, we speculate that techniques reinforcing the learning of cluster structures in data may further improve the training of MMD-GAN.

In addition, the performance gain of proposed repulsive loss (Eq. 4) over the attractive loss (Eq. 3) comes at no additional computational cost. In fact, by using a single kernel rather than a linear combination of kernels, MMD-rep and MMD-rep-b are simpler than MMD-rbf and MMD-rq. Besides, given a typically small batch size and a small number of discriminator output neurons (64 and 16 in our experiments), the cost of MMD over the non-saturating and hinge loss is marginal compared to the convolution operations.

In Appendix D.3, we provide some random samples generated by the methods in our study.

6. DISCUSSION:

This study extends the previous work on MMD-GAN (Li et al. (2017a) ) with two contributions. First, we interpreted the optimization of MMD loss as a combination of attraction and repulsion processes, and proposed a repulsive loss for the discriminator that actively learns the difference among real data. Second, we proposed a bounded Gaussian RBF (RBF-B) kernel to address the saturation issue. Empirically, we observed that the repulsive loss may result in unstable training, due to factors including initialization (Appendix A.2), learning rate (Fig. 3b ) and Lipschitz constraints on the discriminator (Appendix C.3). The RBF-B kernel managed to stabilize the MMD-GAN training in many cases. Tuning the hyper-parameters in RBF-B kernel or using other regularization methods may further improve our results.

The theoretical advantages of MMD-GAN require the discriminator to be injective. The proposed repulsive loss (Eq. 4) attempts to realize this by explicitly maximizing the pair-wise distances among the real samples. Li et al. (2017a) achieved the injection property by using the discriminator as the encoder and an auxiliary network as the decoder to reconstruct the real and generated samples, which is more computationally extensive than our proposed approach. On the other hand, Bińkowski et al. (2018) ; Arbel et al. (2018) imposed a Lipschitz constraint on the discriminator in MMD-GAN via gradient penalty, which may not necessarily promote an injective discriminator.

The idea of repulsion on real sample scores is in line with existing studies. It has been widely accepted that the quality of generated samples can be significantly improved by integrating labels (Odena et al. (2017) ; Miyato & Koyama (2018) ; Zhou et al. (2018) ) or even pseudo-labels generated by k-means method (Grinblat et al. (2017) ) in the training of discriminator. The reason may be that the labels help concentrate the data from the same class and separate those from different classes. Using a pre-trained classifier may also help produce vivid image samples (Huang et al. (2017) ) as the learned representations of the real samples in the hidden layers of the classifier tend to be well separated/organized and may produce more meaningful gradients to the generator.

At last, we note that the proposed repulsive loss is orthogonal to the GAN studies on designing network structures and training procedures, and thus may be combined with a variety of novel techniques. For example, the ResNet architecture (He et al. (2016) ) has been reported to outperform the plain DCGAN used in our experiments on image generation tasks (Miyato et al. (2018) ; Gulrajani et al. (2017) ) and self-attention module may further improve the results (Zhang et al. (2018) ). On the other hand, Karras et al. (2018) proposed to progressively grows the size of both discriminator and generator and achieved the state-of-the-art performance on unsupervised training of GANs on the CIFAR-10 dataset. Future work may explore these directions.

Appendices:

A STABILITY ANALYSIS OF MMD-GAN This section demonstrates that, under mild assumptions, MMD-GAN trained by gradient descent is locally exponentially stable at equilibrium. It is organized as follows. The main assumption and proposition are presented in Section A.1, followed by simulation study in Section A.2 and proof in Section A.3. We discuss the indications of assumptions on the discriminator of GAN in Section A.4.

A.1 MAIN PROPOSITION:

We consider GAN trained using the MMD loss L mmd G for generator G and either the attractive loss L att D or repulsive loss L rep D for discriminator D, listed below:

EQUATION

where For loss functions like minimax and Wasserstein, D θ D (x) may be interpreted as how plausible a sample is real. Thus at equilibrium, it may be reasonable to assume all real and generated samples are equally plausible. However, D θ * D (x) = 0 also indicates that D θ * D may have no discrimination power (see Appendix A.4 for discussion). For MMD loss in Eq. S1, D θ D (x)| x∼P may be interpreted as a learned representation of the distribution P . As long as two distributions P and Q match, M 2 k•D θ D (P, Q) = 0. On the other hand, D θ D (x) = 0 is a minima solution for D but D is trained to find local maxima. Thus in contrast to Assumption 1, we assume Assumption 2. For GANs using MMD loss in Eq. S1, and random initialization on parameters, at equilibrium, D θ * D (x) is injective on S(P X ) S(P θ * G ).

Assumption 2 indicates that D θ * D (x) is not constant almost everywhere. We use a simulation study in Section A.2 to show that D θ * D (x) = 0 does not hold in general for MMD loss. Based on Assumption 2, we propose the following proposition and prove it in Appendix A.3: Proposition 1. If there exists θ * G ∈ Θ G such that P θ * G = P X , then GANs with MMD loss in Eq. S1 has equilibria (θ * G , θ D ) for any θ D ∈ Θ D . Moreover, the model trained using gradient descent methods is locally exponentially stable at (θ * G , θ D ) for any θ D ∈ Θ D .

There may exist non-realizable cases where the mapping between P Z and P X cannot be represented by any generator G θ G with θ G ∈ Θ G . In Section A.2, we use a simulation study to show that both the attractive MMD loss L att D (Eq. S1b) and the proposed repulsive loss L rep D (Eq. S1c) may be locally stable and leave the proof for future work.

A.2 SIMULATION STUDY:

In this section, we reused the example from Nagarajan & Kolter (2017) to show that GAN trained using the MMD loss in Eq. S1 is locally stable. Consider a two-parameter MMD-GAN with uniform latent distribution (a) the data distribution P X is the same as P Z , i.e., uniform over [-1, 1], thus P X is realizable; (b) P X is standard Gaussian, thus non-realizable for any w 1 ∈ R.

P Z over [-1, 1], generator G(z) = w 1 z, discriminator D(x) = w 2 x 2

Fig. S1 shows that MMD-GAN are locally stable in both cases and D θ * D (x) = 0 does not hold in general for MMD loss. However, MMD-rep may not be globally stable for the tested cases: initialization of (w 1 , w 2 ) in some regions may lead to the trivial solution w 2 = 0 (see Fig. S1b and S1d ). We note that by decreasing the learning rate for G, the area of such regions decreased. At last, it is interesting to note that both MMD-rbf and MMD-rep had the same nontrivial solution w 1 ≈ 1.55 for generator in the non-realizable cases (see Fig. S1c and S1d ).

A.3 PROOF OF PROPOSITION 1:

This section divides the proof for Proposition 1 into two parts. First, we show that GAN with the MMD loss in Eq. S1 has equilibria for any parameter configuration of discriminator D; second, we prove the model is locally exponentially stable. For convenience, we consider the general form of discriminator loss in Eq. 5:

L D,λ = λE P X [k D (x, x )] -(λ -1)E P X ,P G [k D (x, y)] -E P G [k D (y, y )] (S2)

which has L att D and L rep D as the special cases when λ equals -1 and 1 respectively. Consider real data X r ∼ P X , latent variable Z ∼ P Z and generated variable Y g = G θ G (Z). Let x r , z, y g be their samples. Denote ∇ a b = ∂a ∂b ;

.

θ D = -∇ L D θ D ,

. where k is the kernel of MMD. The gradients of MMD loss are .

θ G = -∇ L G θ G ; d g = D(G(z)), d r = D(x

EQUATION

.

θ G = 2E P θ G ,P X [∇ k eg,r ∇ dg xg ∇ xg θ G ] -E P θ G [∇ k eg1,g2 (∇ dg1 xg1 ∇ xg1 θ G -∇ dg2 xg2 ∇ xg2 θ G )] (S3b)

Note that, given i.i.d. drawn samples X = {x i } n i=1 ∼ P X and Y = {y i } n i=1 ∼ P G , an unbiased estimator of the squared MMD is (Gretton et al. (2012) )

M 2 k (P X , P G ) = 1 n(n -1) n i =j k(x i , x j )+ 1 n(n -1) n i =j k(y i , y j )- 2 n(n -1) n i =j k(x i , y j ) (S4) At equilibrium, consider a sequence of N samples d ri = d gi = d i with N → ∞, we have . θ D ∝ (λ -1) i =j ∇ k ei,j ∇ ei,j θ D -λ i =j ∇ k ei,j ∇ ei,j θ D + i =j ∇ k ei,j ∇ ei,j θ D = 0 . θ * G ∝ - i =j ∇ k ei,j (∇ di xi ∇ xi θ G -∇ dj xj ∇ xj θ G ) + 2 i =j ∇ k ei,j ∇ di xi ∇ xi θ G = i =j ∇ k ei,j (∇ di xi ∇ xi θ G + ∇ dj xj ∇ xj θ G ) = 0

where for .

θ * G we have used the fact that for each term in the summation, there exists an term with i, j reversed and ∇ k ei,j = -∇ k ej,i thus the summation is zero. Since we have not assumed the status of θ D , .

θ D = 0 for any θ D ∈ Θ D .

We proceed to prove the model stability. First, following Theorem 5 in Gretton et al. (2012) and Theorem 4 in Li et al. (2017a) , it is straightforward to see: 2017)). Consider a non-linear system of parameters (θ, γ): θ = h 1 (θ, γ), γ = h 2 (θ, γ) with an equilibrium point at (0, 0). Let there exist such that ∀γ ∈ B (0), (0, γ) is an equilibrium. If J = ∂h1(θ,γ) ∂θ (0,0) is a Hurwitz matrix, the non-linear system is exponentially stable.

Lemma A.1. Under Assumption 2, M 2 k•D θ D (P X , P θ G ) ≥

Now we can prove::

Proposition 1 (Part 2). At equilibrium P θ * G = P X , the GAN trained using MMD loss and gradient descent methods is locally exponentially stable at (θ * G , θ D ) for any θ D ∈ Θ D .

Proof. Inspired by Nagarajan & Kolter (2017), we first derive the Jacobian of the system

J = J DD J DG J GD J GG ∂ . θ T D /∂θ D ∂ . θ T D /∂θ G ∂ . θ T G /∂θ D ∂ . θ T G /∂θ G Denote ∆ a b = ∂ 2 a ∂b 2 and ∆ a bc = ∂ 2 a ∂b∂c .

Based on Eq. S3, we have

EQUATION

J GG = -E P θ G [(∆ eg1,g2 θ G ) T ⊗ (∇ k eg1,g2 ) T + (∇ eg1,g2 θ G ) T ∆ k eg1,g2 ∇ eg1,g2 θ G ] + 2E P X ,P θ G [(∆ dg θ G ) T ⊗ (∇ k er,g ) T + (∇ dg θ G ) T ∆ k er,g ∇ dg θ G ] (S5d)

where ⊗ is the kronecker product. At equilibrium, consider a sequence of N samples d ri = d gi = d i with N → ∞, we have J DD = 0, J GD = 0 and

J DG ∝ (λ + 1) i<j [(∆ ei,j θ D θ G ) T ⊗ (∇ k ei,j ) T + (∇ ei,j θ D ) T ∆ k ei,j ∇ ei,j θ G ] J GG = E P θ G [(∇ dg1 θ G ) T ∆ k eg1,g2 ∇ dg1 θ G + (∇ dg2 θ G ) T ∆ k eg1,g2 ∇ dg2 θ G -(∇ eg1,g2 θ G ) T ∆ k eg1,g2 ∇ eg1,g2 θ G ] = E P θ G [(∇ dg1 θ G ) T ∆ k eg1,g2 ∇ dg2 θ G + (∇ dg2 θ G ) T ∆ k eg1,g2 ∇ dg1 θ G ]

Given Lemma A.1 and fact that J GG is the Hessian matrix of M 2 k•D θ D (P X , P θ G ), J GG is negative semidefinite. The eigenvectors of J GG corresponding to zero eigenvalues form null(J GG ). There may exist small distortion δθ G ∈ null(J GG ) such that P θ * G +δθ G = P θ * G . That is, P θ * G is locally constant along some directions in the parameter space of G. As a result, null(J GG ) ⊆ null(J DG ) because varying θ * G along these directions has no effect on D. Following Lemma C.3 of Nagarajan & Kolter (2017), we consider eigenvalue decomposition

J GG = U G Λ G U T G and J DG J T DG = U D Λ D U T D . Let U G = [T T G , T T G ], U D = [T T D , T T D ] such that Col(T T G ) = null(J GG ), Col(T T D ) = null(J T DG )

. Thus, the projections γ G = T G θ G are orthogonal to null(J GG ). Then, the Jacobian corresponding to the projected system has the form J = [0, J DG ; 0, J GG ] with block J DG = T D J DG T T G and J GG = T G J GG T T G , where J GG is negative definite. Moreover, on all directions exclude those described by J GG , the system is surrounded by a neighborhood of equilibia at least locally. According to Lemma A.2, the system is exponentially stable.

A.4 DISCUSSION ON ASSUMPTION 1:

This section shows that constant discriminator output D θ * D (x) = c for x ∈ S(P X ) S(P θ * G ) indicates that D θ * D may have no discrimination power. First, we make the following assumptions: Assumption 3. 1. D is a multilayer perceptron where each layer l can be factorized into an affine transform and an element-wise activation function f l . 2. Each activation function f l ∈ C 0 ; furthermore, f l has a finite number of discontinuities and f l ∈ C 06 . 3. Input data to D is continuous and its support S is compact in R d with non-zero measure in each dimension and d > 17 .

Based on Assumption 3, we have the following proposition:

Proposition 2. If ∀x ∈ S, D(x) = c

, where c is constant, then there always exists distortion δx such that x + δx ∈ S and D(x + δx) = c.

Proof. Without loss of generality, we consider D

(x) = W 2 h(x) + b 2 and h(x) = f (W 1 x + b 1 ), where W 1 ∈ R d h ×d , W 2 , b 1 , b 2

are model weights and biases, f is an activation function satisfying Assumption 3. For x ∈ S, since D(x) = c, we have h(x) ∈ null(W 2 ). Furthermore:

(a) If rank(W 1 ) < d, for any δx ∈ null(W 1 ), h(x + δx) ∈ null(W 2 ). (b) If rank(W 1 ) = d = d h , the problem h(x + δx) = k • h(x) has unique solution for any k ∈ R

as long as k • h(x) is within the output range of f .

(c) If rank(W 1 ) = d < d h , let U and V be two basis matrices of R d h such that W 1 x = U xT 0 T T and any vector in null(W 2 ) can be represented as V z T 0 T T , where

x ∈ R d h ×d , z ∈ R d h ×n

and n is the nullity of W 2 . Let the projected support be Ŝ. Thus, ∀x ∈ Ŝ, there exists z such that f

(U xT 0 T T + b 1 ) = V z T z T c T with z c = 0.

Consider the Jacobian:

J = ∂ z T z T c T ∂ xT 0 T T = V -1 ∇ΣU (S6)

where • For linear activation, ∇Σ = I and J is constant. Then z c remains 0 for xb + δ xb , i.e., there exists z such that h(x + δ x) ∈ null(W 2 ).

∇Σ = diag( d f d ai ) and a = [a i ] d h i=1 is

• For nonlinear activations, assume f has N discontinuities. Since U xT 0 T T + b 1 = c has unique solution for any vector c, the boundary points {x b } cannot yield pre-activations {a b } that all lie on the discontinuities in any of the d h directions. Though we might need to sample d N +1 h points in the worst case to find an exception, there are infinite number of exceptions. Let xb be a sample where {a b } does not lie on the discontinuities in any direction. Because f is continuous, z c remains 0 for xb + δ xb , i.e., there exists z such that h(x + δ x) ∈ null(W 2 ).

In conclusion, we can always find δx such that x + δx / ∈ S and D(x + δx) = c.

Proposition 2 indicates that if D θ * D (x) = 0 for x ∈ S(P X ) S(P θ * G ), D θ * D cannot discriminate against fake samples with distortions to the original data. In contrast, Assumption 2 and Lemma A.1 guarantee that, at equilibrium, the discriminator trained using MMD loss function is effective against such fake samples given a large number of i.i.d. test samples (Gretton et al. (2012) ).

B SUPPLEMENTARY METHODOLOGY B.1 REPRESENTATIVE LOSS FUNCTIONS IN LITERATURE:

Several loss functions have been proposed to quantify the difference between real and generated sample scores, including: (assume linear activation is used at the last layer of D)

• The Minimax loss (Goodfellow et al. (2014) ):

L D = E P X [Softplus(-D(x))] + E P Z [Softplus(D(G(z)))

] and L G = -L D , which can be derived from the Jensen-Shannon (JS) divergence between P X and the model distribution P G .

• The non-saturating loss (Goodfellow et al. (2014) ), which is a variant of the minimax loss with the same L D and L G = E P Z [Softplus (-D(G(z) ))].

• The Hinge loss (Tran et al. (2017) ):

L D = E P X [ReLU(1 -D(x))] + E P Z [ReLU(1 + D(G(z)))], L G = E P Z [-D(G(z))],

which is notably known for usage in support vector machines and is related to the total variation (TV) distance (Nguyen et al. (2009) ).

• The Wasserstein loss (Arjovsky et al. (2017) ; Gulrajani et al. (2017) ), which is derived from the Wasserstein distance between P X and P G :

L G = -E P Z [D(G(z))], L D = E P Z [D(G(z))] - E P X [D(x)],

where D is subject to some Lipschitz constraint.

• The maximum mean discrepancy (MMD) (Li et al. (2017a) ; Bińkowski et al. (2018) ), as described in Section 2.

B.2 NETWORK ARCHITECTURE:

For unsupervised image generation tasks on CIFAR-10 and STL-10 datasets, the DCGAN architecture from Miyato et al. (2018) was used. For CelebA and LSUN bedroom datasets, we added more layers to the generator and discriminator accordingly. See Table S1 and S2 for details.

Table S1 : DCGAN models for image generation on CIFAR-10 (h = w = 4, H = W = 32) and STL-10 (h = w = 6, H = W = 48) datasets. For non-saturating loss and hinge loss, s = 1; for MMD-rand, MMD-rbf, MMD-rq, s = 16.

(a) Generator √ hw s σ -1 PICO ], depending on the kernel W c . For a typical kernel of size 3 × 3 and stride 1, σ -1 PIM may vary from σ -1 PICO to 3σ -1 PICO . Thus, LIP PIM is indefinite and may vary during training. In deep convolutional networks, PIM could potentially result in a very loose constraint on the Lipschitz constant of the network. In Appendix Section C.3, we experimentally compare the performance of PICO and PIM with several loss functions.

C.3 EXPERIMENTS:

In this section, we empirically evaluate the effects of coefficient C K on the performance of PICO and compare PICO against PIM using several loss functions.

Experiment setup: We used a similar setup as Section 5.1 with the following adjustments. Four loss functions were tested: hinge, MMD-rbf, MMD-rep and MMD-rep-b. Either PICO or PIM was used at each layer of the discriminator. For PICO, five coefficients C K were tested: 16, 32, 64, 128 and 256 (note this is the overall coefficient for K layers; K = 8 for CIFAR-10 and STL-10; K = 10 for CelebA and LSUN-bedroom; see Appendix B.2). FID was used to evaluate the performance of each combination of loss function and power iteration method, e.g., hinge + PICO with C K = 16.

Results: For each combination of loss function and power iteration method, the distribution of FID scores over 16 learning rate combinations is shown in Fig. S2 . We separated well-performed learning rate combinations from diverged or poorly-performed ones using a threshold τ as the diverged cases often had non-meaningful FID scores. The boxplot shows the distribution of FID scores for goodperformed cases while the number of diverged or poorly-performed cases was shown above each box if it is non-zero. Fig. S2 shows that: 1) When PICO was used, the hinge, MMD-rbf and MMD-rep methods were sensitive to the choices of C K while MMD-rep-b was robust. For hinge and MMD-rbf, higher C K may result in better FID scores and less diverged cases over 16 learning rate combinations. For MMD-rep, higher C K may cause more diverged cases; however, the best FID scores were often achieved with C K = 64 or 128.

2) For CIFAR-10, STL-10 and CelebA datasets, PIM performed comparable to PICO with C K = 128 or 256 on four loss functions. For LSUN bedroom dataset, it is likely that the performance of PIM corresponded to that of PICO with C K > 256. This implies that PIM may result in a relatively loose Lipschitz constraint on deep convolutional networks.

3) MMD-rep-b performed generally better than hinge and MMD-rbf with tested power iteration methods and hyper-parameter configurations. Using PICO, MMD-rep also achieved generally better FID scores than hinge and MMD-rbf. This implies that, given a limited computational budget, the proposed repulsive loss may be a better choice than the hinge and MMD loss for the discriminator. Table S3 shows the best FID scores obtained by PICO and PIM where C K was fixed at 128 for hinge and MMD-rbf, and 64 for MMD-rep and MMD-rep-b. For hinge and MMD-rbf, PICO performed significantly better than PIM on the LSUN-bedroom dataset and comparably on the rest datasets.

For MMD-rep and MMD-rep-b, PICO achieved consistently better FID scores than PIM.

However, compared to PIM, PICO has a higher computational cost which roughly equals the additional cost incurred by increasing the batch size by two (Tsuzuku et al. (2018) ). This may be problematic when a small batch has to be used due to memory constraints, e.g., when handling high resolution images on a single GPU. Thus, we recommend using PICO when the computational cost is less of a concern.

D SUPPLEMENTARY EXPERIMENTS:

D.1 LIPSCHITZ CONSTRAINT VIA GRADIENT PENALTY Gradient penalty has been widely used to impose the Lipschitz constraint on the discriminator arguably since Wasserstein GAN (Gulrajani et al. (2017) ). This section explores whether the proposed repulsive loss can be applied with gradient penalty.

Several gradient penalty methods have been proposed for MMD-GAN. Bińkowski et al. (2018) penalized the gradient norm of witness function f

EQUATION

)] w.r.t. the interpolated sample z = ux + (1 -u)y to one, where u ∼ U(0, 1)9 . More recently, Arbel et al. (2018) proposed to impose the Lipschitz constraint on the mapping φ • D directly and derived the Scaled MMD (SMMD) as SM k (P, Q) = σ µ,k,λ M k (P, Q), where the scale σ µ,k,λ incorporates gradient and smooth penalties. Using the Gaussian kernel and measure µ = P X leads to the discriminator loss:

L SMMD D = L att D 1 + λE P X [ ∇D(x) 2 F ] (S7)

We apply the same formation of gradient penalty to the repulsive loss:

L rep-gp D = L rep D -1 1 + λE P X [ ∇D(x) 2 F ] (S8)

where the numerator L rep D -1 ≤ 0 so that the discriminator will always attempt to minimize both L rep D and the Frobenius norm of gradients ∇D(x) w.r.t. real samples. Meanwhile, the generator is trained using the MMD loss L mmd G (Eq. 2).

Experiment setup: The gradient-penalized repulsive loss L rep-gp D (Eq. S8, referred to as MMD-repgp) was evaluated on the CIFAR-10 dataset. We found λ = 10 in Arbel et al. (2018) too restrictive and used λ = 0.1 instead. Same as Arbel et al. (2018) , the output dimension of discriminator was set to one. Since we entrusted the Lipschitz constraint to the gradient penalty, spectral normalization was not used. The rest experiment setup can be found in Section 5.1.

Results: Table S4 shows that the proposed repulsive loss can be used with gradient penalty to achieve reasonable results on CIFAR-10 dataset. For comparison, we cited the Inception score and FID for Scaled MMD-GAN (SMMDGAN) and Scaled MMD-GAN with spectral normalization (SN-SMMDGAN) from Arbel et al. (2018) . Note that SMMDGAN and SN-SMMDGAN used the same DCGAN architecture as MMD-rep-gp, but were trained for 150k generator updates and 750k discriminator updates, much more than that of MMD-rep-gp (100k for both G and D). Thus, the repulsive loss significantly improved over the attractive MMD loss for discriminator.

D.2 OUTPUT DIMENSION OF DISCRIMINATOR:

In this section, we investigate the impact of the output dimension of discriminator on the performance of repulsive loss.

Experiment setup: We used a similar setup as Section 5.1 with the following adjustments. The repulsive loss was tested on the CIFAR-10 dataset with a variety of discriminator output dimensions: d ∈ {1, 4, 16, 64, 256}. Spectral normalization was applied to discriminator with the proposed PICO method (see Appendix C) and the coefficients C K selected from {16, 32, 64, 128, 256}.

Results: Table S5 shows that using more than one output neuron in the discriminator D significantly improved the performance of repulsive loss over the one-neuron case on CIFAR-10 dataset. The reason may be that using insufficient output neurons makes it harder for the discriminator to learn an injective and discriminative representation of the data (see Fig. 4b ). However, the performance gain diminished when more neurons were used, perhaps because it becomes easier for D to surpass the generator G and trap it around saddle solutions. The computation cost also slightly increased due to more output neurons.

D.3 SAMPLES OF UNSUPERVISED IMAGE GENERATION:

Generated samples on CelebA dataset are given in Fig. S3 and LSUN bedrooms in Fig. S4 . : 16, 32, 64, 128, and 256 . A learning rate combination was considered diverged or poorly-performed if the FID score exceeded a threshold τ , which is 50, 80, 50, 90 for CIFAR-10, STL-10, CelebA and LSUN-bedroom respectively. The box quartiles were plotted based on the cases with FID < τ while the number of diverged or poorly-performed cases (out of 16 learning rate combinations) was shown above each box if it is non-zero. We introduced τ because the diverged cases often had arbitrarily large and non-meaningful FID scores. 

:

The weights for the three terms in L D,λ sum up to zero. This is to make sure the ∂L D,λ /∂θD is zero at equilibrium P X = PG, where θD is the parameters of D.

Note that training stability is different from the local stability considered in Appendix A. Training stability often refers to the ability of model converging to a desired state measured by some criterion. Local stability means that if a model is initialized sufficiently close to an equilibrium, it will converge to the equilibrium.

Increasing or decreasing n dis may improve the model performance in some cases, but it has significant impact on the computation cost. For simple and fair comparison, we set n dis = 1 in all cases.

For λ < 0, the RBF-B kernel in Eq.

was used in L D,λ .

This include many commonly used activations like linear, sigmoid, tanh, ReLU and ELU.

For distributions with semi-infinite or infinite support, we consider the effective or truncated support S (P ) = {x ∈ X |P (x) ≥ }, where > 0 is a small scalar. This is practical, e.g., univariate Gaussian has support in (-∞, +∞) yet a sample five standard deviations away from the mean is unlikely to be valid.

This was obtained by optimizing σPICO/σPIM w.r.t. a variety of randomly initialized kernel Wc. Both gradient descent and Adam methods were tested with a small learning rate 1e -5 so that the error of spectral norm estimation may be ignored at each iteration.

Empirically, we found this gradient penalty did not work with the repulsive loss. The reason may be the attractive loss L att D (Eq. 3) is symmetric in the sense that swapping P X and PG results in the same loss; while the repulsive loss is asymmetric and naturally results in varying gradient norms in data space.

Figures and Tables:

In MMD-GAN, the discriminator D can be interpreted as forming a new kernel with k: k•D(a, b) = k(D(a), D(b)) = k D (a, b). If D is injective, k • D is characteristic and M 2

Figure 1: Illustration of the gradient directions of each loss on the real sample scores {D(x)} ("r" nodes) and generated sample scores {D(y)} ("g" nodes). The blue arrows stand for attraction and the orange arrows for repulsion. When L mmd G is paired with L att D , the gradient directions of L mmd G on {D(y)} can be obtained by reversing the arrows in (a), thus are omitted.

Figure 2: (a) Gaussian kernels {k rbf σi (a, b)} and their mean as a function of e = ab , where σ i ∈ {1, √ 2, 2, 2 √ 2, 4} were used in our experiments; (b) derivatives of {k rbf σi (a, b)} in (a); (c) rational quadratic kernel {k rq αi (a, b)} and their mean, where α i ∈ {0.2, 0.5, 1, 2, 5}; (d) derivatives of {k rq αi (a, b)} in (c).

Figure3: FID scores of MMD-GAN using (a) RBF kernel and (b) RBF-B kernel in L D,λ on CIFAR-10 dataset for 16 learning rate combinations. Each color bar represents the FID score using a learning rate combination (ρ D , ρ G ), in the order of (1e-4, 1e-4), (1e-4, 2e-4),...,(1e-3, 1e-3). The discriminator was trained using L D,λ (Eq. 5) with λ ∈ {-1, -0.5, 0, 0.5, 1, 2}, and generator using L mmd

Figure 4: t-SNE visualization of discriminator outputs {D(x)} learned by (a) MMD-rbf and (b) MMD-rep for 2560 real samples from the CIFAR-10 dataset, colored by their class labels.

k D (a, b) = k(D(a), D(b)). Let S(P ) be the support of distribution P ; let θ G ∈ Θ G , θ D ∈ Θ D be the parameters of the generator G and discriminator D respectively. To prove that GANs trained using the minimax loss and gradient updates is locally stable at the equilibrium point (θ * D , θ * G ), Nagarajan & Kolter (2017) made the following assumption: Assumption 1 (Nagarajan & Kolter (2017)). P θ * G = P X and ∀x ∈ S(P X ), D θ * D (x) = 0.

, and Gaussian kernel k rbf 0.5 . The MMD-rbf model (L mmd G and L att D from Eq. S1b) and the MMD-rep model (L mmd G and L rep D from Eq. S1c) were tested. Each model was applied to two cases:

Figure S1: Streamline plots of MMD-GAN using the MMD-rbf and the MMD-rep model on distributions: P Z = U(-1, 1), P X = U(-1, 1) or P X = N (0, 1). In (a) and (b), the equilibria satisfying P G = P X lie on the line w 1 = 1. In (c), the equilibrium lies around point (1.55, 0.74); in (d), it is around (1.55, 0.32).

r ) where L D and L G are the losses for D and G respectively. Assume an isotropic stationary kernel k(a, b) = k I ( ab ) (Genton (2002)) is used in MMD. We first show: Proposition 1 (Part 1). If there exists θ * G ∈ Θ G such that P θ * G = P X , the GAN with the MMD loss in Eq. S1a and Eq. S2 has equilibria (θ * G , θ D ) for any θ D ∈ Θ D .Proof. Denote e i,j = a i -b j and ∇ k ei,j = ∂k(a i ,b j ) ∂e

0 with the equality if and only if P X = P θ G . Lemma A.1 and Proposition 1 (Part 1) state that at equilibrium P θ * G = P X , every discriminator D θ D and kernel k will give M 2 k•D θ D (P θ * G , P X ) = 0, thus no discriminator can distinguish the two distributions. On the other hand, we cite Theorem A.4 from Nagarajan & Kolter (2017): Lemma A.2 (Nagarajan & Kolter (

the input to activation, or pre-activations. Since Ŝ is continuous and compact, it has infinite number of boundary points {x b } for d > 1. Consider one boundary point xb and its normal line δ xb . Let > 0 be a small scalar such that xb -δ xb ∈ Ŝ and xb + δ xb ∈ Ŝ.

FigureS2: Boxplot of the FID scores for 16 learning rate combinations on four datasets: (a) CIFAR-10, (b) STL-10, (c) CelebA, (d) LSUN-bedroom, using four loss functions, Hinge, MMD-rbf, MMD-rep and MMD-rmb. Spectral normalization was applied to discriminator with two power iteration methods: PICO and PIM. For PICO, five coefficients C K were tested: 16, 32, 64, 128, and 256. A learning rate combination was considered diverged or poorly-performed if the FID score exceeded a threshold τ , which is 50, 80, 50, 90 for CIFAR-10, STL-10, CelebA and LSUN-bedroom respectively. The box quartiles were plotted based on the cases with FID < τ while the number of diverged or poorly-performed cases (out of 16 learning rate combinations) was shown above each box if it is non-zero. We introduced τ because the diverged cases often had arbitrarily large and non-meaningful FID scores.

Figure S3: Image generation using different loss functions on 64 × 64 CelebA dataset.

Figure S4: Image generation using different loss functions on 64 × 64 LSUN bedroom dataset.

Inception score (IS), Fréchet Inception distance (FID) and multi-scale structural similarity (MS-SSIM) on image generation tasks using different loss functions

Table content: <table><tr><td>Methods 1</td><td colspan="2">CIFAR-10</td><td colspan="2">STL-10</td><td colspan="2">CelebA 2</td><td colspan="2">LSUN-bedrom 2</td></tr><tr><td/><td>IS</td><td>FID</td><td>IS</td><td>FID</td><td>FID</td><td>MS-SSIM</td><td>FID</td><td>MS-SSIM</td></tr><tr><td>Real data</td><td colspan="4">11.31 2.09 26.37 2.10</td><td>1.09</td><td>0.2678</td><td>1.24</td><td>0.0915</td></tr><tr><td>Non-saturating</td><td colspan="5">7.39 23.23 8.25 48.53 10.64</td><td>0.2895</td><td>23.66</td><td>0.1027</td></tr><tr><td>Hinge</td><td colspan="5">7.33 23.46 8.24 49.44 8.60</td><td>0.2894</td><td>16.73</td><td>0.0946</td></tr><tr><td>MMD-rbf 3</td><td colspan="5">7.05 28.38 8.13 57.52 13.03</td><td>0.2937</td><td/></tr><tr><td>MMD-rq 3</td><td colspan="5">7.22 27.00 8.11 54.05 12.74</td><td>0.2935</td><td/></tr><tr><td>MMD-rbf-b</td><td colspan="5">7.18 25.25 8.07 51.86 10.09</td><td>0.3090</td><td>32.29</td><td>0.1001</td></tr><tr><td>MMD-rep</td><td colspan="5">7.99 16.65 9.36 36.67 7.20</td><td>0.2761</td><td>16.91</td><td>0.0901</td></tr><tr><td>MMD-rep-b</td><td colspan="5">8.29 16.21 9.34 37.63 6.79</td><td>0.2659</td><td>12.52</td><td>0.0908</td></tr><tr><td>1</td><td/><td/><td/><td/><td/><td/><td/></tr></table>

Fréchet Inception distance (FID) on image generation tasks using spectral normalization with two power iteration methods PICO and PIM

Table content: <table><tr><td>Methods</td><td>C K in PICO</td><td colspan="2">CIFAR-10 PIM PICO PIM PICO PIM PICO PIM STL-10 CelebA LSUN-bedrom PICO</td></tr><tr><td>Hinge</td><td>128</td><td colspan="2">23.60 22.89 47.10 47.24 10.02 9.08 27.38 17.20</td></tr><tr><td>MMD-rbf</td><td>128</td><td colspan="2">26.56 26.50 53.17 54.23 13.06 12.81</td></tr><tr><td>MMD-rep</td><td>64</td><td>19.98 17.00 40.40 37.15 8.51</td><td>6.81 74.03 16.01</td></tr><tr><td>MMD-rep-b</td><td>64</td><td>18.24 16.65 39.78 37.31 7.09</td><td>6.42 20.12 11.22</td></tr><tr><td colspan="3">found 8 that σ -1 PIM varies in the range [σ -1 PICO ,</td><td/></tr></table>



Table content: <table><tr><td colspan="3">: Inception score (IS) and</td></tr><tr><td colspan="3">Fréchet Inception distance (FID) on</td></tr><tr><td colspan="3">CIFAR-10 dataset using gradient</td></tr><tr><td colspan="3">penalty and different loss functions</td></tr><tr><td>Methods 1</td><td>IS</td><td>FID</td></tr><tr><td>Real data</td><td colspan="2">11.31 2.09</td></tr><tr><td>SMMDGAN 2</td><td>7.0</td><td>31.5</td></tr><tr><td>SN-SMMDGAN 2</td><td>7.3</td><td>25.0</td></tr><tr><td>MMD-rep-gp</td><td colspan="2">7.26 23.01</td></tr><tr><td colspan="3">1 All methods used the same DCGAN ar-</td></tr><tr><td>chitecture.</td><td/><td/></tr><tr><td colspan="3">2 Results from Arbel et al. (2018) Table 1.</td></tr></table>



Table content: <table><tr><td colspan="4">: Inception score (IS), Fréchet</td></tr><tr><td colspan="4">Inception distance (FID) on CIFAR-10</td></tr><tr><td colspan="4">dataset using MMD-rep and different di-</td></tr><tr><td colspan="3">mensions of discriminator outputs</td><td/></tr><tr><td>Methods</td><td>C K</td><td>IS</td><td>FID</td></tr><tr><td>Real data</td><td/><td colspan="2">11.31 2.09</td></tr><tr><td>MMD-rep-1</td><td>64</td><td colspan="2">7.43 22.43</td></tr><tr><td>MMD-rep-4</td><td>64</td><td colspan="2">7.81 17.87</td></tr><tr><td>MMD-rep-16</td><td>32</td><td colspan="2">8.20 16.99</td></tr><tr><td>MMD-rep-64</td><td>32</td><td colspan="2">8.08 15.65</td></tr><tr><td>MMD-rep-256</td><td>32</td><td colspan="2">7.96 16.61</td></tr></table>

Peer review (iclr2019):
 OVERALL COMMENTS: I haven*t had much time to write this, so I*m giving a low confidence score and you should feel free to correct me. I didn*t think this paper was very clear. I had trouble grasping what the contributions were supposed to be and I had trouble judging the significance of the experiments. That said, now that (I think) I understand what*s going on, the idea seems well motivated, the connection between the repulsion and the use of label information in other GAN variants makes sense to me, and the statements you are making seem (as much as I had time to check them) correct. This leaves the issue of scientific significance. I feel like I need to understand what specifically contributed to the improvements in table 1 to evaluate significance. First of all, it seems like there are a lot of other *good-scoring* models left out of this table. I understand that you make the claim that your improvement is orthogonal, but that seems like something that needs to be tested empirically. You have orthogonal motivation but it might be that in practice your technique works for a reason similar to the reason other techniques work. I would like to see more exploration of this. Second, are the models below the line the only models using spectral norm? I can*t tell. Overall, it*s hard for me to envision this work really seriously changing the course of research on GANs, but that*s perhaps too high a bar for poster acceptance. For these reasons, I am giving a score of 6. DETAILED COMMENTS ON TEXT: > their performance heavily depends on the loss functions used in training. This is not true, IMO. See [1] > may discourage the learning of data structures What does *data structures* mean in this case? It has another more common usage that makes this confusing. > Several loss functions have been proposed IMO this list doesn*t belong in the main body of the text. I would move it to an appendix. > We assume linear activation is used at the last layer of D I*m not sure what this means? My best guess is just that you*re saying there is no activation function applied to the logits. > Arjovsky et al. (2017) showed that, if the supports of PX and PG do not overlap, there exists a perfect discriminator... This doesn*t affect your paper that much, but was this really something that needed to be shown? If the discriminator has finite capacity it*s not true in general and if it has infinite capacity its vacuous. > We propose a generalized power iteration method... Why do this when we can explicitly compute the singular values as in [2]? Genuine question. > MS-SSIM is not compatible with CIFAR-10 and STL-10 which have data from many classes; Just compute the intra-class MS-SSIM as in [3]. > Higher IS and lower FID scores indicate better image quality I*m a bit worried about using the FID to evaluate a model that*s been trained w/ an MMD loss where the discriminator is itself a neural network w/ roughly the same architecture as the pre-trained image classifier used to compute the FID. What can you say about this? Am I wrong to be worried? > Table 1: Which models use spectral norm? My understanding is that this has a big influence on the scores. This seems like a very important point. REFERENCES: [1] Are GANs Created Equal? A Large-Scale Study [2] The Singular Values of Convolutional Layers [3] Conditional Image Synthesis With Auxiliary Classifier GANs