Title: HOW DOES LIPSCHITZ REGULARIZATION INFLUENCE GAN TRAINING?

Abstract: Despite the recent success of Lipschitz regularization in stabilizing GAN training, the exact reason of its effectiveness remains poorly understood. It is commonly believed that the main function of K-Lipschitz regularization is to restrict the L2-norm of the neural network gradient to be smaller than a threshold K (e.g. K = 1) such that ∇f ≤ K. While in this work, we uncover a counter-intuitive fact that under typical GAN setups, the choice of K does not matter. This finding suggests that instead of keeping the neural network gradients small, an even more important function of Lipschitz regularization is its restriction on the domain and interval of attainable gradient values of the loss function. This avoids the bias of the loss function over input samples. Empirically, we verify our proposition on the MNIST, CIFAR10 and CelebA datasets.

1. INTRODUCTION:

Generative Adversarial Networks (GANs) are a class of generative models successfully applied to various applications, e.g., pose-guided image generation (Ma et al., 2017) , image-to-image translation (Zhu et al., 2017) , super-resolution (Ledig et al., 2017) , high resolution image synthesis (Wang et al., 2018) , urban modeling (Kelly et al., 2018) , etc. Theoretically, Goodfellow et al. (2014) proved the convergence of GAN training by assuming that the generator is always updated according to the temporarily optimal discriminator at each training step. In practice, this assumption is too difficult to satisfy and GANs remain notoriously difficult to train. To stabilize the training of the GANs, various techniques have been proposed regarding the choices of architectures (Radford et al., 2015; He et al., 2016) , loss functions (Arjovsky et al., 2017; Mao et al., 2017a) , regularization and normalization (Arjovsky et al., 2017; Gulrajani et al., 2017; Mescheder et al., 2018; Miyato et al., 2018) . We refer interested audiences to (Lucic et al., 2017; Kurach et al., 2018) for empirical studies.

Among them, the Lipschitz regularization (Gulrajani et al., 2017; Miyato et al., 2018) has shown great success in stabilizing the training of various GANs recently. For example, Mao et al. (2017b) and Fedus et al. (2018) observed that the gradient penalty Lipschitz regularizer helps to improve the training of the LS-GAN (Mao et al., 2017a) and the NS-GAN (Goodfellow et al., 2014) respectively; Miyato et al. (2018) observed that the NS-GAN with their spectral normalization Lipschitz regularizer works better than WGAN-GP (Gulrajani et al., 2017) .

In this paper, we provide an analysis to better understand the impact of Lipschitz regularization and the loss function. This analysis is summarized below. i) How does the Lipschitz constant K influence GAN training?

• For the neural network, under typical GAN setups, the choice of K does not matter. The only exception is that a very small K leads to numerical robustness issues and slow training. • For the loss function, K restricts its domain and thereby the range of values of the attainable gradients. This means that, as K → 0, all loss functions degenerate to linear functions with constant gradients that are similar to the Wasserstein loss.

In other words, K does not matter for the network itself, but it matters for the loss function.

ii) How does the loss function influence GAN training? There are actually two conflicting goals:

• The loss function should be strictly convex for better convergence than linear ones.

• The loss function should be linear to avoid mode collapse.

So our main insight is that the rule of thumb of using small Lipschitz constants is mainly to degenerate loss functions to almost linear ones. These degenerate losses improve GAN training by achieving a balance between the two conflicting goals mentioned above. Because of this, the exact shapes of the loss functions before degeneration do not seem to matter that much. We demonstrate this by two experiments. First, we show that when K is sufficiently small, even GANs trained with ridiculous loss functions (e.g. cosine) give comparable results to all other loss functions. Second, we can directly degenerate loss functions by introducing loss function scaling. This enables successful GAN training for a wide range of K for all loss functions, which only worked for the Wasserstein loss before. Our analysis is made more formal and expanded on in the following sections.

2. LIPSCHITZ REGULARIZATION AND GAN LOSS FUNCTIONS:

Traditionally, Lipschitz regularization and GAN loss functions have been studied separately.

GAN Loss Functions. Under the assumption of optimal discriminators, the GAN loss function is usually interpreted as a measure of some sort of statistical divergence between the model distribution and the data distribution. Based on this understanding, a variety of GAN loss functions have been proposed. Goodfellow et al. (2014) first proposed to use the sigmoid cross-entropy loss function that minimizes the Jensen-Shannon (JS) divergence. However, because of the saturation at both ends of the sigmoid function, such a loss function can lead to vanishing gradients and thus fail to update the generator. To compensate for it, Goodfellow et al. (2014) proposed a variant named the non-saturating loss (NS-GAN), which heuristically amplifies the gradients when updating the generator. Observing that the JS divergence is a special case of the f -divergence, Nowozin et al. ( 2016) extended the idea of Goodfellow et al. (2014) and showed that any f -divergence can be used to train GANs. Their work suggested that the performance of GANs can be improved by employing "better" divergence measures. Following this direction, Arjovsky & Bottou (2017) first pointed out the flaws of the JS divergence used in GANs and then proposed to use the Wasserstein distance (WGAN) instead (Arjovsky et al., 2017) . In addition, they employed the Kantorovich-Rubinstein dual of the Wasserstein distance to improve the computational efficiency, which simplifies the WGAN loss function to a simple identity function but at the cost of requiring the discriminator neural network to be 1-Lipschitz continuous. Meanwhile, Mao et al. (2017a) proposed to use the quadratic loss function that minimizes the Pearson χ 2 divergence, which is called the Least-Square GAN (LS-GAN).

Lipschitz Regularization. Instead of stabilizing GAN training, Lipschitz regularization is first designed to enforce the 1-Lipschitz continuity of the discriminator neural network, which is required by the Kantorovich-Rubinstein duality applied in WGAN (Arjovsky et al., 2017) . Since then, several algorithms are proposed to implement Lipschitz regularization. Arjovsky et al. (2017) first proposed the weight clipping, whose main idea is to clamp the weights of each neural network layer to a small fixed range [-c, c] , where c is a small positive constant. Although weight clipping guarantees the Lipschitz continuity of the discriminator, the choice of parameter c is difficult and prone to invalid gradients. To this end, Gulrajani et al. (2017) proposed the gradient penalty (GP) which incorporates an extra regularization term of discriminator's gradient magnitude into the loss function. In their original setting, GP is one-centered to enforce 1-Lipschitz continuity. While Mescheder et al. (2018) showed that the zero-centered one is more reasonable for the convergence of GAN training. However, one major problem of GP is that it is computed with finite samples, which makes it intractable to be applied to the entire output space. To sidestep this problem, Gulrajani et al. (2017) proposed to heuristically sample from the straight lines connecting model distribution and data distribution. However, this makes their approach heavily dependent on the support of the model distribution (Miyato et al., 2018) . Addressing this issue, Miyato et al. (2018) proposed the spectral normalization (SN) that enforces the Lipschitz continuity of a neural network in the operator space.

Observing that the Lipschitz constant of the entire neural network is bounded by the product of those of its layers, they break down the problem to applying the Lipschitz regularization onto each neural network layer. These simplified sub-problems can then be solved by normalizing the weight matrix of each layer according to its largest singular value.

While recently, a tendency of applying 1-Lipschitz regularization to GANs without Lipschitz constraint requirements appears and links the two fields together. For example, Fedus et al. (2018) and Mao et al. (2017b) observed that applying the gradient penalty Lipschitz regularizer designed for the WGAN also helps to improve the training of the NS-GAN and the LS-GAN respectively. Similarly, 

(•) = sigmoid[f (•)]. GAN types Discriminator Loss Generator Loss NS-GAN -E[log(f * (x))] -E[log(1 -f * (g(z)))] -E[log(f * (g(z)))] LS-GAN E[(f (x) -1) 2 ] + E[f (g(z)) 2 ] E[(f (g(z)) -1) 2 ] WGAN E[f (x)] -E[f (g(z))] E[f (g(z))] COS-GAN -E[cos(f (x) -1)] -E[cos(f (g(z)) + 1)] -E[cos(f (g(z)) -1)] EXP-GAN E[exp(f (x))] + E[exp(-f (g(z)))] E[exp(f (g(z)))]

Miyato et al. ( 2018) observed that the NS-GAN with their spectral normalization regularizer works better than WGAN-GP (Gulrajani et al., 2017) . All these observations imply that: Hypothesis 1. Keeping neural network gradients small by enforcing 1-Lipschitz regularization is the key to successful GAN training while the choice of loss functions contributes little.

However, due to its complex nature, deep learning is usually criticized as "alchemy" (Hutson, 2018) and prone to attribution error, i.e. the credit of success is often incorrectly attributed. To this end, we would like to question the above hypothesis: is it true?

2.1. EMPIRICAL EVALUATIONS ON LIPSCHITZ REGULARIZED GANS:

To answer the above question, we empirically evaluate how the choice of Lipschitz constants influence the performance of various Lipschitz regularized GANs. First, we regularize the discriminator neural network with the state-of-the-art spectral normalization and enforce K-Lipschitz regularization by tuning the hyper-parameter k SN (Miyato et al., 2018) ,

EQUATION

where W is the weight matrix of a neural network layer, σ(W ) is its largest singular value. The relationship between k SN and K can be quantitatively approximated as K ≈ k n SN (Miyato et al., 2018) , where n is the number of neural network layers in the discriminator. Then, we train these Lipschitz regularized GANs with a variety of GAN loss functions (Table 1 ) consisting of three most widely-used ones: NS-GAN, LS-GAN, WGAN, and two arbitrary ones: cos(•) and exp(•). Following the best practice, we employ the Fréchet Inception Distance (FID) metric (Heusel et al., 2017) to quantitatively measure the performance of these GANs. The smaller the FID score, the better the performance of the GAN. Please see Appendix B for the detailed neural network architectures and other experimental setups.

Table 2 shows the FID scores of different GAN loss functions with different k SN . Qualitative results are shown in Appendix E. At the first glance, the results seem to be a solid support of Hypothesis 1. For example, when k SN ≤ 1.0, all the loss functions (including the arbitrary ones) can be used to train GANs stably; the FID scores of all loss functions (except the WGAN) are similar; the FID scores of all loss functions slightly worsen as k SN further decreases. When k SN ≥ 5.0, the performance of all the GANs (except the WGAN) worsen and even break (very high FID scores, e.g. ≥ 100). However, the history tells us that significant scientific progresses are usually inspired by outliers instead of the observations as expected. In our case, the WGAN is such an outlier as it works normally when k SN ≥ 5.0. This suggests that the performance of WGAN is insensitive to the choice of Lipschitz constants, which contradicts Hypothesis 1. So does the choice of Lipschitz constant really matter?

3. DO LIPSCHITZ CONSTANTS MATTER?:

In a nutshell, the choice of Lipschitz constants does not matter when i) scale insensitive activation functions (e.g. ReLU) and ii) optimizers using normalized gradients (e.g. RMSProp) are used. Note that both techniques are dominating ones in practical applications due to their effectiveness in stabilizing neural network training (Glorot et al., 2011; Tieleman & Hinton, 2012) .

Table 2 : FID scores vs. k SN on different datasets. k SN is typically fixed as 1 to enforce 1-Lipschitz regularization (Miyato et al., 2018) . The WGAN is an outlier that works normally even when k SN ≥ 5.0. For the line plots, x-axis shows k SN (in log scale) and y-axis shows the FID scores. From left to right, the seven points on each line have k SN = 0.2, 0.25, 0.5, 1.0, 5.0, 10.0, 50.0 respectively. Lower FID scores are better. SN: spectral normalization (Miyato et al., 2018) Let us consider a simple discriminator D(x) = L(f (x)), where x is the input, f is the discriminator neural network with scalar output, L is the loss function. Following Miyato et al. (2018) , we omit the bias term for simplicity and describe a spectral normalized f (x) as:

EQUATION

where, a n is the activation function, W n SN is the normalized weight matrix of the n-th layer (Eq.1). Forward Pass. As Glorot et al. (2011) and Hoffer et al. (2018) suggested, scale-invariant activation functions allow the scaling parameter like k SN to move freely within the neural network without influencing its function. For example, let W 1-Lip = W/σ(W ), Eq.2 can be rewritten as

EQUATION

where K = (k SN ) N +1 is the Lipschitz constant. Thus, it can be concluded that K only works as a scaling hyper-parameter and is independent of the overall network function.

Backpropagation. Similar to the case of forward pass, scale-insensitive activation functions allow k SN to move freely during backpropagation.

∇a(k SN • W 1-Lip h in )) = k SN ∇a(W 1-Lip h in ) (4)

where h in is the input of a neural network layer. Eq.4 implies that the scaling parameter k SN will be accumulated when the gradients are backpropagated through neural network layers, which may produce exploding or vanishing gradients that can heavily influence the training. However, for optimizers with normalized gradients (e.g. RMSProp), each gradient is divided by an exponentially decaying average of the past gradients, which largely eliminates the influence of the gradient scale. For example in RMSProp (Tieleman & Hinton, 2012) ,

w t+1 = w t -η k m SN • g t E[(k m SN • g) 2 ] t + = w t -η g t E[g 2 ] t + /k 2m SN (5)

where k m SN is a scaling parameter, w is the neural network weight, η is the learning rate, g is the gradient, E[g 2 ] is the exponentially decaying average that E[g 2 ] t = γE[g 2 ] t1 + (1γ)g 2 t , > 0 is a small constant for numerical stability that can usually be ignored when E[(k m SN • g) 2 ] t . Summarizing the above analysis, the major influence of k SN is only a simple scaling effect applied to the neural network output during the forward pass, which has nothing to do with the overall neural network function. Thus, the choice of Lipschitz constant does not matter.

The above conclusion clarifies the "strange" robustness of WGAN against large k SN (k SN ≥ 5) in Table 2 : the WGAN loss functions are linear and have constant gradients which are independent of the scale of neural network outputs that is controlled by k SN . Thus, the choice of k SN does not influence the performance of WGAN. Similarly, it can be easily inferred that the big impact of Lipschitz constants on the performance of other GANs (Table 2 ) originates from its interaction with the loss functions.

Remark. The above analysis also clarifies the observation that when k SN ≤ 1, the FID scores of all loss functions slightly worsen as k SN decreases, which is actually a numerical problem. Eq.5 shows that the effect of is heavily dependent on the scale of gradients. When k SN is small (e.g. k SN = 0.2), the gradients are dramatically scaled down by a factor of k m SN . As a result, cannot be ignored and reduces the magnitude of the gradients, which accounts for the degenerated performance.

4. RESTRICTIONS OF GAN LOSS FUNCTIONS:

In this section, we will show that: compared to the choice of Lipschitz constants, the dominating factor of Lipschitz regularization is its side-effect of restricting the domain and interval of attainable gradient values of the loss functions. To support our claim, we first theoretically analyze why a K-Lipschitz regularized discriminator restricts the domain and interval of attainable gradient values of the loss function to intervals bounded by K. Then, an empirical study is performed accordingly to verify our theoretical analysis. Finally, we show that such restrictions prevent the backpropagation of vanishing or exploding gradients as well as avoid the bias of loss functions over input samples, which stabilizes GAN training.

4.1. THEORETICAL DERIVATION:

Let us consider a simple discriminator D(x) = L(f (x)), where x is the input, f is a neural network with scalar output, L is the loss function. During training, the loss function L works by backpropagating the gradient ∇L = ∂L(f (x))/∂f (x) to update the neural network weights:

∂D(x) ∂W n = ∂L(f (x)) ∂f (x) ∂f (x) ∂W n (6)

where W n is the weight matrix of the n-th layer. Let X and Ω be the domain and the range of f respectively (i.e. f : X → Ω), it can be easily derived that the attainable values of ∇L is determined by Ω (i.e. ∇L : Ω → Ψ). Without loss of generality, we assume that x ∈ X = [-1, 1] m×n×3 are normalized images and derive the bound of the size of Ω as follows:

Theorem 1. If the discriminator neural network f satisfies the k-Lipschitz continuity condition, we have f : X → Ω ⊂ R satisfying | min(Ω) -max(Ω)| ≤ k √ 12mn.

Please see Appendix A for the proof of Theorem 1. Theorem 1 shows that the size of Ω is bounded by k. However, k can be unbounded when Lipschitz regularization is not enforced during training, which results in an unbounded Ω and a large interval of attainable gradient values. On the contrary, when K-Lipschitz regularization is applied (i.e. k ≤ K), the loss function L is restricted as follows:

Corollary 1.1 (Restriction of Loss Function). Assume that f is a Lipschitz regularized neural network whose Lipschitz constant k ≤ K, the loss function L is C 2 -continuous with M as the maximum absolute value of its second derivatives in its domain. Let Ψ be the interval of attainable gradient values that ∇L : Ω → Ψ, we have 1 ) on the CelebA dataset. NS-GAN: Non-Saturating GAN; LS-GAN: Least-Square GAN; WGAN: Wasserstein GAN. k SN is a parameter of spectral normalization (Eq.1).

EQUATION

kSN NS-GAN-SN, L(•) = -log(sigmoid(•)) LS-GAN-SN, L(•) = (•) 2 Ω Ψ Ω Ψ 5.0 [-

Corollary 1.1 shows that under a mild condition (C 2 -continuous), applying K-Lipschitz regularization restricts the domain Ω and thereby the interval of attainable gradient values Ψ of the loss function L to intervals bounded by K. When K is small, e.g., K = 1 (Gulrajani et al., 2017; Fedus et al., 2018; Mao et al., 2017b; Miyato et al., 2018) , the interval of attainable gradient values of the loss function is considerably reduced, which prevents the backpropagation of vanishing or exploding gradients and thereby stabilizes the training.

-5

0 5 0 2 4 a = -4.0 b = -3.0 c = -0.4 d = 0.2 [a, b] shifts to [c, d].

Remark on the Changes of Ω i During Training. So far we analyzed the restriction of the loss function by a static discriminator. However, the discriminator neural network f is dynamically updated during training and thus its range

Ω ∪ = ∪ i Ω i ,

where Ω i is the discriminator range at each training step i. Therefore, we need to analyze two questions:

1. How does the size of Ω i change during training?

2. Does Ω i shift during training (figure on the right)?

For question 1, the size of Ω i is always bounded by the Lipschitz constant K throughout the training. While for question 2, the answer depends on the discriminator loss function. In a nutshell, common loss functions (e.g. NS-GAN, LS-GAN) are strictly convex and have a unique minimum respectively, which forces Ω i to be positioned around it and prevents Ω i from shifting. Thus, Ω ∪ is still roughly bounded by K. However, Ω i may be allowed to shift when the discriminator loss functions is not strictly convex (e.g. WGAN). While for WGAN, its attainable gradient value Ψ is a constant that is independent of Ω i . Thus, regarding to Ψ, we can view it as a degenerate loss function that still fits in our discussion.

4.2. EMPIRICAL VERIFICATION:

Figure 1 shows the corresponding empirical results on the restriction of different loss functions. Note that the K-Lipschitz regularization is enforce by applying the spectral normalization (Miyato et al., 2018) that K ≈ k n SN , where n is the number of neural network layers (Eq.1).

Restriction on Ω. In Figure 1 (a), we plot the domain Ω as intervals for different iterations under different k SN . It can be observed that the interval Ω shrinks rapidly as k SN decreases, which verifies that the domain Ω is restricted by the Lipschitz regularization (Eq.7).

Changes of Ω i During Training:

Figure 1 (a) also shows that the domains Ω i of the NS-GAN and the LS-GAN are rather fixed during training while those of the WGAN shift, which verifies our remark on the changes of Ω i during training. More specifically, it can be observed that Ω i of NS-GAN (Table 1 ) are relatively fixed around zero, which is the unique minimum of its loss function. A similar phenomenon can be observed for LS-GAN (Table 1 ) around 0.5. Interestingly, we empirically observed that the Ω i of WGAN also get relatively fixed at late stages of the training.

Restriction on Ψ. In Figure 1 (b), we show the corresponding intervals of attainable gradient values Ψ when the domain Ω (Ω = ∪ i Ω i ) is restricted as in Figure 1 . In particular, we compute the bounds of Ψ as follows, min(Ψ) = ∇L min(Ω) , max(Ψ) = ∇L max(Ω) .

(9) because ∇L is monotonically increasing in Ω for common GAN loss functions that are convex (e.g., NS-GAN, LS-GAN, WGAN). Note that the Ψ of WGAN is not included as it is a constant whose size is always zero. Similar to the domain Ω, Ψ also shrinks with the increasing strength of Lipschitz regularization, which avoids the saturating and exploding parts of the loss function. For example when k SN = 5.0, the gradient of the NS-GAN loss function saturates to a value around 0 while that of the LS-GAN loss function explodes to 24.041. However, such problems do not happen when k SN ≤ 1.0. Note that we only compute Ψ on one of the two symmetric loss terms used in the discriminator loss function (Table 1 ). The interval of attained gradient values of the other loss term follows similar patterns.

4.3. HOW DOES THE RESTRICTION OF LOSS FUNCTIONS HELP TRAINING?:

At this point, we know that instead of influencing the overall neural network function, the choice of Lipschitz constant K only works by restricting the domain and thereby the interval of attainable gradient values of the loss function, which is implemented by scaling the neural network outputs by K (Eq.3). Thus, for a K-Lipschitz regularized neural network f , we have

f : X → Ω ⇐⇒ f : X → K • Ω 1-Lip (10)

where Ω 1-Lip is the range of a corresponding 1-Lipschitz continuous neural network. This suggests that we can restrict the domain Ω of loss function L by a positive scaling hyper-parameter α directly,

EQUATION

Compared to tuning k SN , the proposed domain scaling is more numerically stable as it does not influence the scale of the backpropagated gradients and thus does not suffer from the numerical problem caused by (discussed in section 3, Remark). This also motivates us to include α in the denominator of Eq.11, which helps to preserve the gradient scale of the loss function. Please see Appendix D for the extensive experiments on domain scaling.

The equivalency between tuning Lipschitz constants and domain scaling (Eq.11) implies that the performance improvements of various Lipschitz regularized GANs (Table 2 ) originates from the restriction of the loss function. But what kinds of restrictions are beneficial to the GAN training? Qualitatively, we hypothesize that:

1. The interval of attainable gradients values should be small. This prevents the backpropagation of exploding or vanishing gradients and guarantees that there is no significant bias over input samples from the same category (i.e. real or synthesized). 2. For better convergence, the restricted loss function should be strictly convex and has a unique minimum locally in its restricted domain.

For the first point, we omit the discussion of the well-studied exploding or vanishing gradients problem and focus on the bias of loss functions that may cause mode collapse (Figure 2 ). As we know, neural networks are naturally biased over inputs: they may "prefer" some input samples over the others by thinking they are "more real" and outputting higher/lower values, even though all the inputs are sample from unbiased real data. Biased (scale-sensitive) loss functions like NS-GAN backpropagate different gradients according to the biased network outputs, which further enhances the bias and may cause mode collapse (Figure 2 ). On the contrary, using unbiased (e.g. WGAN) or weakly biased loss functions (e.g. restricted loss functions) can sidestep the problem by backpropagating same or similar gradients for input samples within the same category (i.e. real or synthesized).

- The second point makes sense because strictly convex loss functions with unique minima usually have better convergence. We observe that any loss function degenerates as its domain Ω shrinks to a single value. According to Taylor's expansion, let ω, ω + ∆ω ∈ Ω, we have:

2 -1 0 1 2 -2.5

L(ω + ∆ω) = L(ω) + L (ω) 1! ∆ω + L (ω) 2! (∆ω) 2 + • • • . (12)

As | max(Ω)min(Ω)| shrinks to zero, we have L(ω + ∆ω) ≈ L(ω)+L (ω)∆ω showing that we can approximate any loss function by a linear function with constant gradient as its domain Ω shrinks to a single value (figure on the right). Let ω ∈ Ω, we implement the degeneration of a loss function by scaling its domain Ω with an extremely small constant α:

EQUATION

)

We use α = 1e A PROOF OF THEOREM 1

Theorem 1. If the discriminator neural network f satisfies the k-Lipschitz continuity condition, we have f :

X → Ω ⊂ R satisfying | min(Ω) -max(Ω)| ≤ k √ 12mn.

Proof. Given a k-Lipschitz continuous neural newtork f , for all x 1 , x 2 ∈ X, we have:

EQUATION

)

Let x b , x w ∈ X be the pure black and pure white images that maximize the Euclidean distance:

EQUATION

Thus, we have:

EQUATION

Thus, the range of f is restricted to Ω, which satisfies:

EQUATION

B DETAILED EXPERIMENTAL SETUPS

In our experiments, we use two variants of the standard CNN architecture (Radford et al., 2015; Arjovsky et al., 2017; Miyato et al., 2018) for the GANs to learn the distributions of the MNIST, CIFAR10 datasets at 32 × 32 resolution and the CelebA dataset (Liu et al., 2015) at 64 × 64 resolution. We use a batch size of 64 to train the GANs. We use an RMSProp optimizer with learning rate 0.00005. To make a fair comparison, we fix the number of discriminator updates in each iteration n dis = 1 for all the GANs tested, i.e. we do not use multiple discriminator updates like (Arjovsky & Bottou, 2017; Arjovsky et al., 2017) . Unless specified, we stop the training after 10 5 iterations. Details of the architectures are shown as follows. BN batch normalization; SN: spectral normalization; c = 3 for CIFAR10 dataset and c = 1 for MNIST dataset in input 

x ∈ [-1, 1] 64×64×c . z ∈ R 100 ∼ N (0, 1) Reshape → 1 × 1 × 100 4 × 4,

C RESTRICTIONS OF GAN LOSS FUNCTIONS (GRADIENT PENALTY):

Before spectral normalization, the best-performing Lipschitz regularizer is the gradient penalty (Gulrajani et al., 2017) :

EQUATION

where L GAN is the GAN loss function without gradient penalty, λ is the weight of the gradient penalty term, P x is the distribution of linearly interpolated samples between the target distribution and the model distribution, k GP is the target gradient norm that controls the Lipschitz constant of the discriminator K. In general, the smaller k GP , the smaller K. However, it is challenging to find a quantitative approximation between k GP and K because the gradient penalty term λ Ex∈P x [(

x D(x) -k GP ) 2

] has no upper bound during training. Such a challenge makes gradient penalty less impactful in restricting the loss function.

As Figure 3 shows, we plot the domain Ω as intervals for different iterations under different k GP and k SN for the gradient penalty and the spectral normalization regularizers respectively. Similar to Gulrajani et al. (2017) and Fedus et al. (2018) , we use λ = 10 for the gradient penalty. It can be observed that for both regularizers, the interval Ω shrinks as k GP and k SN decreases, which shows that the domain of the loss function is also restricted when gradient penalty is applied. However, k SN is much more impactful than k GP in restricting Ω, which is another reason for our choice of using spectral normalization to alter the strength of the Lipschitz regularization in this paper. The details of the architectures are shown below. (Arjovsky et al., 2017) ; GP: gradient penalty (Gulrajani et al., 2017) ; SN: spectral normalization (Miyato et al., 2018) . Note that the y-axis is in log scale. 

z ∈ R 100 ∼ N (0, 1) Reshape → 1 × 1 × 100 4 × 4,

D EMPIRICAL RESULTS ON DOMAIN SCALING:

Table 5 shows the FID scores against α on the MNIST, CIFAR10 and CelebA datasets, where α is the hyper-parameter for domain scaling,

EQUATION

)

Table 5 (a) shows that the FID scores of different loss functions generally improve with decreasing α, which is similar to those of k SN . To further illustrate the similarity between α and k SN , we show that a small α (e.g. α ≤ 10 -9 ) can compensate for the influence of an extremely large Lipschtiz constant (K ≈ k n SN = 50 4 = 6.25 × 10 6 ) and thereby stabilizes GAN training. Also, the FID scores when α ≤ 10 -11 are slightly worse than those when α ≤ 10 -9 . As discussed in the main paper, the reason for this phenomenon is that restricting the domain of the loss function converges towards the performance of WGAN whose loss function does not have a unique minimum.

Table 5 (b) shows that the FID scores of different loss functions generally worsen with less restricted domains. Note that when α ≥ 10 5 , the common practice of 1-Lipschitz regularization fails to stabilize the GAN training, which also implies that Lipschitz regularization only works as a scaling hyper-parameter of the neural network output.

Remark on LS-GAN and LS-GAN # . Note that the LS-GAN-SN has some abnormal behavior (e.g. α = 1e -11 in Table 5 (a) and α = 1e 1 in Table 5 (b)) due to the conflict between its 0.5centered domain and our zero-centered domain scaling method (Eq.11). This can be easily fixed by using the zero-centered LS-GAN # -SN (Mao et al., 2017b) .

Table 5 : FID scores vs. α. For the line plots, the x-axis shows α (in log scale) and the y-axis shows the FID scores. Lower FID scores are better. With varying k SN on the MNIST dataset,

• Figure 4 shows sample images of the NS-GAN-SN;

• Figure 5 shows sample images of the LS-GAN-SN;

• Figure 6 shows sample images of the WGAN-SN;

• Figure 7 shows sample images of the COS-GAN-SN;

• Figure 42 shows sample images of the EXP-GAN-SN;

• Figure 43 shows sample images of the COS-GAN-SN.

With varying α, k SN = 1.0 and on CelebA dataset,

• Figure 44 shows sample images of the NS-GAN-SN;

• Figure 45 shows sample images of the LS-GAN-SN;

• Figure 46 shows sample images of the LS-GAN # -SN;

• Figure 47 shows sample images of the EXP-GAN-SN;

• Figure 48 shows sample images of the COS-GAN-SN.

E.3 SAMPLES OF THE DEGENERATE LOSS FUNCTION:

Figure 49 shows the sample images of WGAN-SN and some extremely degenerate loss functions, which corresponds to the following 

:



Figures and Tables:

The domain Ω shrinks with decreasing kSN for different loss functions. The x-axis shows the number of training iterations while the y-axis shows the domain Ω. Note that the y-axis is in log scale.

Figure1: Empirical results regarding the restrictions of different GAN loss functions (Table1) on the CelebA dataset. NS-GAN: Non-Saturating GAN; LS-GAN: Least-Square GAN; WGAN: Wasserstein GAN. k SN is a parameter of spectral normalization (Eq.1).

Figure 2: Samples of randomly generated images with NS-GAN against α (k SN = 50.0, MNIST).

Figure 3: Relationship between domain Ω and k GP , k SN for different loss functions on CelebA dataset, where k GP , k SN are the parameters controlling the strength of the Lipschitz regularizers. The domain Ω shrinks with decreasing k GP or k SN . Each column shares the same loss function while each row shares the same Lipschitz regularizer. NS-GAN: Non-Saturating GAN (Goodfellow et al., 2014); LS-GAN: Least-Square GAN (Mao et al., 2017a); WGAN: Wasserstein GAN(Arjovsky et al., 2017); GP: gradient penalty(Gulrajani et al., 2017); SN: spectral normalization(Miyato et al., 2018). Note that the y-axis is in log scale.

Figure 4: Samples of randomly generated images with NS-GAN-SN of varying k SN (MNIST). For the line plot, x-axis shows k SN (in log scale) and y-axis shows the FID scores.

Figure 5: Samples of randomly generated images with LS-GAN-SN of varying k SN (MNIST). For the line plot, x-axis shows k SN (in log scale) and y-axis shows the FID scores.

Figure 6: Samples of randomly generated images with WGAN-SN of varying k SN (MNIST). For the line plot, x-axis shows k SN (in log scale) and y-axis shows the FID scores.

Figure 7: Samples of randomly generated images with COS-GAN-SN of varying k SN (MNIST). For the line plot, x-axis shows k SN (in log scale) and y-axis shows the FID scores.

Figure 8: Samples of randomly generated images with EXP-GAN-SN of varying k SN (MNIST). For the line plot, x-axis shows k SN (in log scale) and y-axis shows the FID scores.

Figure 10: Samples of randomly generated images with LS-GAN-SN of varying k SN (CIFAR10). For the line plot, x-axis shows k SN (in log scale) and y-axis shows the FID scores.

Figure 11: Samples of randomly generated images with WGAN-SN of varying k SN (CIFAR10). For the line plot, x-axis shows k SN (in log scale) and y-axis shows the FID scores.

Figure 12: Samples of randomly generated images with COS-GAN-SN of varying k SN (CIFAR10). For the line plot, x-axis shows k SN (in log scale) and y-axis shows the FID scores.

Figure 13: Samples of randomly generated images with EXP-GAN-SN of varying k SN (CIFAR10). For the line plot, x-axis shows k SN (in log scale) and y-axis shows the FID scores.

Figure 14: Samples of randomly generated images with NS-GAN-SN of varying k SN (CelebA). For the line plot, x-axis shows k SN (in log scale) and y-axis shows the FID scores.

Figure 15: Samples of randomly generated images with LS-GAN-SN of varying k SN (CelebA). For the line plot, x-axis shows k SN (in log scale) and y-axis shows the FID scores.

Figure 16: Samples of randomly generated images with WGAN-SN of varying k SN (CelebA). For the line plot, x-axis shows k SN (in log scale) and y-axis shows the FID scores.

Figure 17: Samples of randomly generated images with COS-GAN-SN of varying k SN (CelebA). For the line plot, x-axis shows k SN (in log scale) and y-axis shows the FID scores.

Figure 18: Samples of randomly generated images with EXP-GAN-SN of varying k SN (CelebA). For the line plot, x-axis shows k SN (in log scale) and y-axis shows the FID scores.

Figure 20: Samples of randomly generated images with LS-GAN-SN of varying α (k SN = 50.0, MNIST). For the line plot, x-axis shows α (in log scale) and y-axis shows the FID scores.

Figure 21: Samples of randomly generated images with LS-GAN # -SN of varying α (k SN = 50.0, MNIST). For the line plot, x-axis shows α (in log scale) and y-axis shows the FID scores.

Figure 22: Samples of randomly generated images with EXP-GAN-SN of varying α (k SN = 50.0, MNIST). For the line plot, x-axis shows α (in log scale) and y-axis shows the FID scores.

Figure 23: Samples of randomly generated images with COS-GAN-SN of varying α (k SN = 50.0, MNIST). For the line plot, x-axis shows α (in log scale) and y-axis shows the FID scores.

Figure 25: Samples of randomly generated images with LS-GAN-SN of varying α (k SN = 50.0, CIFAR10). For the line plot, x-axis shows α (in log scale) and y-axis shows the FID scores.

Figure 26: Samples of randomly generated images with LS-GAN # -SN of varying α (k SN = 50.0, CIFAR10). For the line plot, x-axis shows α (in log scale) and y-axis shows the FID scores.

Figure 27: Samples of randomly generated images with EXP-GAN-SN of varying α (k SN = 50.0, CIFAR10). For the line plot, x-axis shows α (in log scale) and y-axis shows the FID scores.

Figure 28: Samples of randomly generated images with COS-GAN-SN of varying α (k SN = 50.0, CIFAR10). For the line plot, x-axis shows α (in log scale) and y-axis shows the FID scores.

Figure 30: Samples of randomly generated images with LS-GAN-SN of varying α (k SN = 50.0, CelebA). For the line plot, x-axis shows α (in log scale) and y-axis shows the FID scores.

Figure 31: Samples of randomly generated images with LS-GAN # -SN of varying α (k SN = 50.0, CelebA). For the line plot, x-axis shows α (in log scale) and y-axis shows the FID scores.

Figure 32: Samples of randomly generated images with EXP-GAN-SN of varying α (k SN = 50.0, CelebA). For the line plot, x-axis shows α (in log scale) and y-axis shows the FID scores.

Figure 33: Samples of randomly generated images with COS-GAN-SN of varying α (k SN = 50.0, CelebA). For the line plot, x-axis shows α (in log scale) and y-axis shows the FID scores.

Figure 35: Samples of randomly generated images with LS-GAN-SN of varying α (k SN = 1.0, MNIST). For the line plot, x-axis shows α (in log scale) and y-axis shows the FID scores.

Figure 36: Samples of randomly generated images with LS-GAN # -SN of varying α (k SN = 1.0, MNIST). For the line plot, x-axis shows α (in log scale) and y-axis shows the FID scores.

Figure 37: Samples of randomly generated images with EXP-GAN-SN of varying α (k SN = 1.0, MNIST). For the line plot, x-axis shows α (in log scale) and y-axis shows the FID scores.

Figure 38: Samples of randomly generated images with COS-GAN-SN of varying α (k SN = 1.0, MNIST). For the line plot, x-axis shows α (in log scale) and y-axis shows the FID scores.

Figure 40: Samples of randomly generated images with LS-GAN-SN of varying α (k SN = 1.0, CIFAR10). For the line plot, x-axis shows α (in log scale) and y-axis shows the FID scores.

Figure 41: Samples of randomly generated images with LS-GAN # -SN of varying α (k SN = 1.0, CIFAR10). For the line plot, x-axis shows α (in log scale) and y-axis shows the FID scores.

Figure 42: Samples of randomly generated images with EXP-GAN-SN of varying α (k SN = 1.0, CIFAR10). For the line plot, x-axis shows α (in log scale) and y-axis shows the FID scores.

Figure 43: Samples of randomly generated images with COS-GAN-SN of varying α (k SN = 1.0, CIFAR10). For the line plot, x-axis shows α (in log scale) and y-axis shows the FID scores.

Figure 45: Samples of randomly generated images with LS-GAN-SN of varying α (k SN = 1.0, CelebA). For the line plot, x-axis shows α (in log scale) and y-axis shows the FID scores.

Figure 46: Samples of randomly generated images with LS-GAN # -SN of varying α (k SN = 1.0, CelebA). For the line plot, x-axis shows α (in log scale) and y-axis shows the FID scores.

Figure 47: Samples of randomly generated images with EXP-GAN-SN of varying α (k SN = 1.0, CelebA). For the line plot, x-axis shows α (in log scale) and y-axis shows the FID scores.

Figure 48: Samples of randomly generated images with COS-GAN-SN of varying α (k SN = 1.0, CelebA). For the line plot, x-axis shows α (in log scale) and y-axis shows the FID scores.

Figure 49: Samples of randomly generated images with WGAN-SN and some extremely degenerate loss functions (α = 1e -25 ) on different datasets. We use k SN = 50 for all our experiments.

The GAN loss functions used in our experiments. f (•) is the output of the discriminator neural network; g(•) is the output of the generator neural network; x is the sample from the training dataset; z is the sample from the noise distribution. For the NS-GAN, f *

Table content: <table/>

.

Table content: <table><tr><td colspan="2">Dataset</td><td/><td>GANs</td><td/><td/><td colspan="3">kSN = 0.2</td><td/><td>0.25</td><td/><td>0.5</td><td colspan="2">FID Scores 1.0</td><td/><td colspan="2">5.0</td><td colspan="2">10.0</td><td>50.0</td></tr><tr><td/><td/><td/><td colspan="3">NS-GAN-SN</td><td/><td>5.41</td><td/><td/><td>3.99</td><td/><td>4.20</td><td>3.90</td><td/><td colspan="3">144.28</td><td colspan="2">156.60</td><td>155.41</td></tr><tr><td/><td/><td/><td colspan="3">LS-GAN-SN</td><td/><td>5.14</td><td/><td/><td>3.96</td><td/><td>3.90</td><td>4.42</td><td/><td/><td colspan="2">36.26</td><td colspan="2">59.04</td><td>309.35</td></tr><tr><td colspan="2">MNIST</td><td/><td colspan="3">WGAN-SN</td><td/><td>6.35</td><td/><td/><td>6.06</td><td/><td>4.44</td><td>4.70</td><td/><td/><td colspan="2">3.58</td><td colspan="2">3.50</td><td>3.71</td></tr><tr><td/><td/><td/><td colspan="3">COS-GAN-SN</td><td/><td>5.41</td><td/><td/><td>4.83</td><td/><td>4.05</td><td>3.86</td><td/><td colspan="3">291.44</td><td colspan="2">426.62</td><td>287.23</td></tr><tr><td/><td/><td/><td colspan="3">EXP-GAN-SN</td><td/><td>4.38</td><td/><td/><td>4.93</td><td/><td>4.25</td><td>3.69</td><td/><td colspan="3">286.96</td><td colspan="2">286.96</td><td>286.96</td></tr><tr><td/><td/><td/><td colspan="3">NS-GAN-SN</td><td/><td>29.23</td><td/><td colspan="2">24.37</td><td colspan="2">23.29</td><td>15.81</td><td/><td/><td colspan="2">41.04</td><td colspan="2">49.67</td><td>48.03</td></tr><tr><td/><td/><td/><td colspan="3">LS-GAN-SN</td><td/><td>28.04</td><td/><td colspan="2">26.85</td><td colspan="2">23.14</td><td>17.30</td><td/><td/><td colspan="2">33.53</td><td colspan="2">39.90</td><td>349.35</td></tr><tr><td colspan="3">CIFAR10</td><td colspan="3">WGAN-SN</td><td/><td>29.20</td><td/><td colspan="2">25.07</td><td colspan="2">26.61</td><td>21.75</td><td/><td/><td colspan="2">21.63</td><td colspan="2">21.45</td><td>23.36</td></tr><tr><td/><td/><td/><td colspan="3">COS-GAN-SN</td><td/><td>29.45</td><td/><td colspan="2">25.31</td><td colspan="2">20.73</td><td>15.88</td><td/><td colspan="3">309.96</td><td colspan="2">327.20</td><td>370.13</td></tr><tr><td/><td/><td/><td colspan="3">EXP-GAN-SN</td><td/><td>30.89</td><td/><td colspan="2">24.74</td><td colspan="2">20.90</td><td>16.66</td><td/><td colspan="3">401.24</td><td colspan="2">401.24</td><td>401.24</td></tr><tr><td/><td/><td/><td colspan="3">NS-GAN-SN</td><td/><td>18.59</td><td/><td colspan="2">12.71</td><td/><td>8.04</td><td>6.11</td><td/><td/><td colspan="2">18.95</td><td colspan="2">17.04</td><td>184.06</td></tr><tr><td/><td/><td/><td colspan="3">LS-GAN-SN</td><td/><td>20.34</td><td/><td colspan="2">12.14</td><td/><td>8.85</td><td>5.69</td><td/><td/><td colspan="2">12.40</td><td colspan="2">13.14</td><td>399.39</td></tr><tr><td colspan="2">CelebA</td><td/><td colspan="3">WGAN-SN</td><td/><td>23.26</td><td/><td colspan="2">17.93</td><td/><td>8.48</td><td>9.41</td><td/><td/><td colspan="2">9.03</td><td colspan="2">7.37</td><td>7.82</td></tr><tr><td/><td/><td/><td colspan="3">COS-GAN-SN</td><td/><td>20.59</td><td/><td colspan="2">13.93</td><td/><td>8.88</td><td>5.20</td><td/><td colspan="3">356.70</td><td colspan="2">265.53</td><td>256.44</td></tr><tr><td/><td/><td/><td colspan="3">EXP-GAN-SN</td><td/><td>18.23</td><td/><td colspan="2">13.65</td><td/><td>9.18</td><td>5.88</td><td/><td colspan="3">328.94</td><td colspan="2">328.94</td><td>328.94</td></tr><tr><td>400</td><td colspan="3">NS-GAN-SN</td><td/><td/><td/><td>400</td><td/><td colspan="2">NS-GAN-SN</td><td/><td/><td colspan="2">400</td><td/><td colspan="2">NS-GAN-SN</td><td/><td/><td/></tr><tr><td/><td colspan="3">LS-GAN-SN</td><td/><td/><td/><td/><td/><td colspan="2">LS-GAN-SN</td><td/><td/><td/><td/><td/><td colspan="2">LS-GAN-SN</td><td/><td/><td/></tr><tr><td>300</td><td colspan="3">WGAN-SN COS-GAN-SN</td><td/><td/><td/><td>300</td><td/><td colspan="2">WGAN-SN COS-GAN-SN</td><td/><td/><td colspan="2">300</td><td/><td colspan="2">WGAN-SN COS-GAN-SN</td><td/><td/><td/></tr><tr><td>200</td><td colspan="3">EXP-GAN-SN</td><td/><td/><td/><td>200</td><td/><td colspan="2">EXP-GAN-SN</td><td/><td/><td colspan="2">200</td><td/><td colspan="2">EXP-GAN-SN</td><td/><td/><td/></tr><tr><td>100</td><td/><td/><td/><td/><td/><td/><td>100</td><td/><td/><td/><td/><td/><td colspan="2">100</td><td/><td/><td/><td/><td/><td/></tr><tr><td>0</td><td>0.20 0.25</td><td>0.50</td><td>1.00</td><td>5.00</td><td>10.00</td><td>50.00</td><td>0</td><td>0.20 0.25</td><td>0.50</td><td>1.00</td><td>5.00</td><td>10.00</td><td>50.00</td><td>0</td><td>0.20 0.25</td><td>0.50</td><td>1.00</td><td>5.00</td><td>10.00</td><td>50.00</td></tr><tr><td/><td/><td/><td colspan="2">(a) MNIST</td><td/><td/><td/><td/><td/><td colspan="3">(b) CIFAR10</td><td/><td/><td/><td/><td colspan="2">(c) CelebA</td><td/><td/></tr></table>

-25 in our experiments. Smaller values are not used due to numerical errors (N aN ).More specifically, we scale the domain by α = 1e -25 and show the FID scores of WGAN and those of different loss functions in the table on the right. To verify that α is an alternative to k SN and can compensate its effect, we use a large k SN = 50. The LS-GAN# (Mao et al., 2017b) is the zero-centered version of LS-GAN, whose rationale is discussed in Appendix D. It can be observed that all loss functions have similar performance as WGAN. However, due to the loss of strict convexity, their performance is slightly worsen than the best ones in Table2and in Appendix D.5 CONCLUSIONIn this paper, we studied the influence of Lipschitz regularization on the GAN training. First, we point out a counter-intuitive fact that under typical GAN setups, the choice of Lipschitz constants is independent of the overall neural network function but just works as a scaling hyper-parameter of the neural network output. Second, we demonstrate that instead of keeping the neural network gradients small, the dominating factor of Lipschitz regularization is its restriction on the domain and interval of attainable gradient values of the loss function. Such restrictions stabilize GAN training by preventing the backpropagation of vanishing or exploding gradients and avoiding the bias of the loss function over input samples, which is a new step in understanding the exact reason for Lipschitz regularization's effectiveness. Xudong Mao, Qing Li, Haoran Xie, Raymond Y.K. Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In The IEEE International Conference on Computer Vision (ICCV), Oct 2017a. Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. On the effectiveness of least squares generative adversarial networks. arXiv preprint arXiv:1712.06391, 2017b. Lars M. Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, pp. 3478-3487, 2018. Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018. Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Advances in Neural Information Processing Systems 29, pp. 271-279. Curran Associates, Inc., 2016. Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.

Table content: <table><tr><td>GANs</td><td colspan="3">FID Scores MNIST CIFAR10 CELEBA</td></tr><tr><td>WGAN</td><td>3.71</td><td>23.36</td><td>7.82</td></tr><tr><td>NS-GAN</td><td>3.74</td><td>21.92</td><td>8.10</td></tr><tr><td colspan="2">LS-GAN # 3.81</td><td>21.47</td><td>8.51</td></tr><tr><td colspan="2">COS-GAN 3.96</td><td>23.65</td><td>8.30</td></tr><tr><td colspan="2">EXP-GAN 3.86</td><td>21.91</td><td>8.22</td></tr></table>

SAMPLES OF FID SCORES vs. k SN EXPERIMENT This subsection corresponds to the FID scores vs. k SN experiment (Table 2 in the main paper).

Table content: <table><tr><td>Dataset</td><td>GANs</td><td>α = 1e -11</td><td>1e -9</td><td colspan="2">FID Scores 1e -7 1e -5</td><td>1e -3</td><td>1e -1</td><td/><td colspan="2">Line Plot</td></tr><tr><td/><td>NS-GAN-SN</td><td>3.67</td><td>3.61</td><td>3.81</td><td colspan="3">33.48 154.14 155.54</td><td/><td>NS-GAN-SN</td></tr><tr><td>MNIST</td><td>LS-GAN-SN LS-GAN # -SN</td><td>281.82 4.25</td><td>6.48 4.15</td><td>3.74 3.99</td><td>24.93 27.62</td><td colspan="2">29.31 285.46 55.64 90.00</td><td/><td>LS-GAN-SN LS-GAN#-SN EXP-GAN-SN COS-GAN-SN</td></tr><tr><td/><td>COS-GAN-SN</td><td>3.74</td><td>3.93</td><td colspan="4">3.66 445.15 306.09 263.85</td><td/><td/></tr><tr><td/><td>EXP-GAN-SN</td><td>4.14</td><td>4.01</td><td colspan="4">3.54 134.76 286.96 286.96</td><td>10-11</td><td>10-9</td><td>10-7</td><td>10-5</td><td>10-3</td><td>10-1</td></tr><tr><td/><td>NS-GAN-SN</td><td>19.58</td><td>22.46</td><td>18.73</td><td>24.57</td><td>49.56</td><td>43.42</td><td/><td>NS-GAN-SN</td></tr><tr><td>CIFAR10</td><td>LS-GAN-SN LS-GAN # -SN</td><td colspan="2">400.91 127.38 20.16 19.96</td><td>18.68 18.13</td><td>34.78 31.03</td><td colspan="2">33.17 282.11 35.06 254.88</td><td/><td>LS-GAN-SN LS-GAN#-SN EXP-GAN-SN COS-GAN-SN</td></tr><tr><td/><td>COS-GAN-SN</td><td>22.16</td><td>22.69</td><td colspan="4">20.19 356.10 369.11 445.37</td><td/><td/></tr><tr><td/><td>EXP-GAN-SN</td><td>21.93</td><td>21.70</td><td colspan="4">18.50 236.77 401.24 401.24</td><td>10-11</td><td>10-9</td><td>10-7</td><td>10-5</td><td>10-3</td><td>10-1</td></tr><tr><td/><td>NS-GAN-SN</td><td>9.08</td><td>7.05</td><td>7.84</td><td>18.51</td><td colspan="2">18.41 242.64</td><td/><td>NS-GAN-SN</td></tr><tr><td>CelebA</td><td>LS-GAN-SN LS-GAN # -SN</td><td>135.17 6.66</td><td>6.57 5.68</td><td>10.67 8.72</td><td>13.39 11.13</td><td colspan="2">17.42 311.93 14.90 383.61</td><td/><td>LS-GAN-SN LS-GAN#-SN EXP-GAN-SN COS-GAN-SN</td></tr><tr><td/><td>COS-GAN-SN</td><td>8.00</td><td colspan="5">6.31 300.55 280.84 373.31 318.53</td><td/><td/></tr><tr><td/><td>EXP-GAN-SN</td><td>8.85</td><td colspan="5">6.09 264.49 375.32 375.32 375.32</td><td>10-11</td><td>10-9</td><td>10-7</td><td>10-5</td><td>10-3</td><td>10-1</td></tr><tr><td/><td/><td/><td colspan="3">(a) kSN = 50.0</td><td/><td/><td/><td/></tr><tr><td>Dataset</td><td>GANs</td><td>α = 1e 1</td><td>1e 3</td><td colspan="2">FID Scores 1e 5 1e 7</td><td>1e 9</td><td>1e 11</td><td/><td colspan="2">Line Plot</td></tr><tr><td/><td>NS-GAN-SN</td><td colspan="6">6.55 148.97 134.44 133.82 130.21 131.87</td><td>400</td><td/></tr><tr><td/><td>LS-GAN-SN</td><td>23.37</td><td colspan="5">26.96 260.05 255.73 256.96 265.76</td><td>300</td><td/></tr><tr><td>MNIST</td><td>LS-GAN # -SN</td><td>13.43</td><td colspan="5">26.51 271.85 212.74 274.63 269.96</td><td>200</td><td/><td>NS-GAN-SN LS-GAN-SN</td></tr><tr><td/><td>COS-GAN-SN</td><td colspan="6">11.79 377.62 375.72 363.45 401.12 376.39</td><td>100</td><td/><td>LS-GAN#-SN EXP-GAN-SN</td></tr><tr><td/><td>EXP-GAN-SN</td><td colspan="6">11.02 286.96 286.96 286.96 286.96 286.96</td><td>0</td><td>103</td><td>105</td><td>107</td><td>109 COS-GAN-SN 1011</td></tr><tr><td/><td>NS-GAN-SN</td><td>17.63</td><td>47.31</td><td>46.85</td><td>45.44</td><td>45.67</td><td>39.90</td><td>400</td><td/></tr><tr><td/><td>LS-GAN-SN</td><td>25.55</td><td colspan="5">34.44 373.07 171.18 309.55 312.96</td><td>300</td><td/></tr><tr><td>CIFAR10</td><td>LS-GAN # -SN</td><td>20.45</td><td colspan="5">36.18 429.21 269.63 291.55 297.71</td><td>200</td><td/><td>NS-GAN-SN LS-GAN-SN</td></tr><tr><td/><td>COS-GAN-SN</td><td colspan="6">18.59 386.24 259.83 268.89 293.29 318.65</td><td>100</td><td/><td>LS-GAN#-SN EXP-GAN-SN COS-GAN-SN</td></tr><tr><td/><td>EXP-GAN-SN</td><td colspan="6">21.56 401.24 401.24 401.24 401.24 401.24</td><td>0</td><td>103</td><td>105</td><td>107</td><td>109</td><td>1011</td></tr><tr><td/><td>NS-GAN-SN</td><td>5.88</td><td>16.14</td><td>17.75</td><td>17.67</td><td>16.87</td><td>18.81</td><td>400</td><td/></tr><tr><td/><td>LS-GAN-SN</td><td>8.41</td><td colspan="5">12.09 201.22 312.83 299.30 321.84</td><td>300</td><td/></tr><tr><td>CelebA</td><td>LS-GAN # -SN</td><td>7.21</td><td colspan="5">13.13 221.41 248.48 311.21 315.94</td><td>200</td><td/><td>NS-GAN-SN LS-GAN-SN</td></tr><tr><td/><td>COS-GAN-SN EXP-GAN-SN</td><td colspan="6">6.62 450.57 233.42 390.40 306.17 335.87 6.91 375.32 375.32 375.32 375.32 375.32</td><td>100 0</td><td>103</td><td>105</td><td>107</td><td>109 LS-GAN#-SN 1011 EXP-GAN-SN COS-GAN-SN</td></tr><tr><td/><td/><td/><td colspan="3">(b) kSN = 1.0</td><td/><td/><td/><td/></tr><tr><td colspan="5">E ADDITIONAL QUALITATIVE RESULTS</td><td/><td/><td/><td/><td/></tr><tr><td colspan="11">In this section, we show the qualitative results (sample images) corresponding to the quantitative</td></tr><tr><td colspan="11">experiments in the main paper. The FID scores and line plots are shown together with the samples.</td></tr><tr><td>E.1</td><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>

table in the main paper. It can be observed that all loss functions have similar performance.

Table content: <table><tr><td>GANs</td><td colspan="3">FID Scores MNIST CIFAR10 CELEBA</td></tr><tr><td>WGAN-SN</td><td>3.71</td><td>23.36</td><td>7.82</td></tr><tr><td>NS-GAN-SN</td><td>3.74</td><td>21.92</td><td>8.10</td></tr><tr><td colspan="2">LS-GAN # -SN 3.81</td><td>21.47</td><td>8.51</td></tr><tr><td colspan="2">COS-GAN-SN 3.96</td><td>23.65</td><td>8.30</td></tr><tr><td colspan="2">EXP-GAN-SN 3.86</td><td>21.91</td><td>8.22</td></tr></table>

Peer review (iclr2020):
Summary The present work investigates the mechanism by which Lipschitz regularization influence GAN training. It makes the observation that under scale invariant activation functions and optimizers (for instance, RELU and RMSProp), the value of the Lipschitz constant in GAN variants like WGAN or spectrally normalized GAN only results in a multiplicative constant of the generator output. Therefore, it argues that the role of the commonly used 1-Lipschitz penalty is not to ensure small discriminator gradient, but rather to restrict the relevant regions of the loss function and the size of the gradients. To support this claim, it is shown that in the limit of small Lipschitz constants the size of the domain of attained discriminator outputs shrinks. For large Lipschitz constants, however, the performance of all GAN variants other than WGAN detoriates, which is explained with the fact that the relevant domain of the loss function is not restricted any more. Decision I believe that it is an interesting pursuit to better understand the role that Lipschitz regularization plays in GAN training and I much appreciate the thorough experiments provided by the authors. However, in the present form, key points of their analysis too vague (see examples below), which is why I vote for rejecting the paper It is unclear why the paragraph above Hypothesis 1 implies that *the choice of loss functions contributes little*, as claimeed just before the hypothesis. Hypothesis 1 inadequatly summarizes the introduction of Liptschitz regularization which was largely motivated by the fact that if the data and generated distribution are mutually singular (for instance, because the data is concentrated on a low-dimensional structure), loss of the optimal discriminator does not depend on the choice of generator. In the original WGAN paper, WGAN is motivated as computing an approximate minimizer of the wasserstein distance between the data and the generated measure. As is evident from the Kantorovich duality that changing the Lipschitz constraints on the discriminator only amounts to applying a scaling to the resulting distance. The paper sometimes mentions the role of Lipschitz regularization and sometimes the role of the choice of Lipschitz constant. However the interpretation of WGAN as approximating the Wasserstein distance readily shows that the Lipschitz regularization can matter even in cases where the exact choice of Lipschitz constant does not. For instance, it claims *The equivalency between tuning Lipschitz constants and domain scaling (Eq.11) implies that the performance improvements of various Lipschitz regularized GANs (Table 2) originates from the restriction of the loss function* Tuning the Lipschitz constant is equivalent to rescaling of the loss function. However, imposing a Lipschitz constraint at all is not equivalent to restricting the output range of the discriminator, since a function of arbitrary small range can have arbitrary large Lipschitz constant. Thus, it can not be concluded that the improvement due to Lipschitz regularization is due to a restriction of the loss function. Suggestions for improvement The observation that spectral regularization often amounts to rescaling the entire network and that this in turn can be seen as a restriction of the relevant domain of the loss function might still be a useful observation, but as outlined above important parts of the conclusions are unclear in the present form.