GENERATIVE ADVERSARIAL NETWORKS
Year: 2019

Abstract:
Generative Adversarial Networks (GANs) are one of the most popular tools for learning complex high dimensional distributions. However, generalization properties of GANs have not been well understood. In this paper, we analyze the generalization of GANs in practical settings. We show that discriminators trained on discrete datasets with the original GAN loss have poor generalization capability and do not approximate the theoretically optimal discriminator. We propose a zero-centered gradient penalty for improving the generalization of the discriminator by pushing it toward the optimal discriminator. The penalty guarantees the generalization and convergence of GANs. Experiments on synthetic and large scale datasets verify our theoretical analysis.

text: Generative Adversarial Networks (GANs) are one of the most popular tools for learning complex high dimensional distributions. However, generalization properties of GANs have not been well understood. In this paper, we analyze the generalization of GANs in practical settings. We show that discriminators trained on discrete datasets with the original GAN loss have poor generalization capability and do not approximate the theoretically optimal discriminator. We propose a zero-centered gradient penalty for improving the generalization of the discriminator by pushing it toward the optimal discriminator. The penalty guarantees the generalization and convergence of GANs. Experiments on synthetic and large scale datasets verify our theoretical analysis.

heading: 1 INTRODUCTION
text: GANs (Goodfellow et al., 2014) are one of the most popular tools for modeling high dimensional data. The original GAN is, however, highly unstable and often suffers from mode collapse. Much of recent researches has focused on improving the stability of GANs (Radford et al., 2015; Arjovsky et al., 2017; Heusel et al., 2017; Miyato et al., 2018; Karras et al., 2018). On the theoretical aspect, Nagarajan & Kolter (2017) proved that gradient based training of the original GAN is locally stable. Heusel et al. (2017) further proved that GANs trained with Two Timescale Update Rule (TTUR) converge to local equilibria. However, the generalization of GANs at local equilibria is not discussed in depth in these papers.
Arora et al. (2017) showed that the generator can win by remembering a polynomial number of training examples. The result implies that a low capacity discriminator cannot detect the lack of diversity. Therefore, it cannot teach the generator to approximate the target distribution. In section 4, we discuss the generalization capability of high capacity discriminators. We show that high capacity discriminators trained with the original GAN loss tends to overfit to the mislabeled samples in training dataset, guiding the generator toward collapsed equilibria (i.e. equilibria where the generator has mode collapse).
Arora et al. (2018) proposed to measure the generalization capability of GAN by estimating the number of modes in the model distribution using the birthday paradox. Experiments on several datasets showed that the number of modes in the model distribution is several times greater than the number of training examples. The author concluded that although GANs might not be able to learn distributions, they do exhibit some level of generalization. Our analysis shows that poor generalization comes from the mismatch between discriminators trained on discrete finite datasets and the theoretically optimal discriminator. We propose a zero-centered gradient penalty for improving the generalization capability of (high capacity) discriminators. Our zero-centered gradient penalty pushes the discriminator toward the optimal one, making GAN to converge to equilibrium with good generalization capability.
Our contributions are as follow:
1. We show that discriminators trained with the original GAN loss have poor generalization capability. Poor generalization in the discriminator prevents the generator from learning the target distribution.
2. We show that the original GAN objective encourages gradient exploding in the discriminator. Gradient exploding in the discriminator can lead to mode collapse in the generator.
3. We propose a zero-centered gradient penalty (0-GP) for improving the generalization capability of the discriminator. We show that non-zero centered GP and the zero-centered GP proposed in Mescheder et al. (2018) cannot make the discriminator generalize. Our 0-GP helps GANs to converge to generalizable equilibria. Theoretical results are verified on real world datasets.
4. We show that 0-GP helps the discriminator to distribute its capacity more equally between regions of the space, effectively preventing mode collapse. Experiments on synthetic and real world datasets verify that 0-GP can prevent mode collapse. GANs with 0-GP is much more robust to changes in hyper parameters, optimizers, and network architectures than the original GAN and GANs with other gradient penalties.
Table 1 compares the key properties of our 0-GP with one centered GP (1-GP) (Gulrajani et al., 2017) and zero centered GP on real/fake samples only (0-GP-sample) (Mescheder et al., 2018).
NOTATIONS
pr the target distribution pg the model distribution pz the noise distribution dx the dimensionality of a data sample (real or fake) dz the dimensionality of a noise sample supp(p) the support of distribution p x ∼ pr a real sample z ∼ pz a noise vector drawn from the noise distribution pz y = G(z) a generated sample Dr = {x1, ...,xn} the set of n real samples D(t)g = { y (t) 1 , ...,y (t) m } the set of m generated samples at step t
D(t) = Dr ∪ D(t)g the training dataset at step t

heading: 2 RELATED WORKS
text: Gradient penalties are widely used in GANs literature. There are a plethora of works on using gradient penalty to improve the stability of GANs (Mescheder et al., 2018; Gulrajani et al., 2017; Petzka et al., 2018; Roth et al., 2017; Qi, 2017). However, these works mostly focused on making the training of GANs stable and convergent. Our work aims to improve the generalization capability of GANs via gradient regularization.
Arora et al. (2018) showed that the number of modes in the model distribution grows linearly with the size of the discriminator. The result implies that higher capacity discriminators are needed for better approximation of the target distribution. Zhang et al. (2018) studied the tradeoff between generalization and discrimination in GANs. The authors showed that generalization is guaranteed if the discriminator set is small enough. In practice, rich discriminators are usually used for better discriminative power. Our GP makes rich discriminators generalizable while remaining discriminative.
Although less mode collapse is not exactly the same as generalization, the ability to produce more diverse samples implies better generalization. There are a large number of papers on preventing mode collapse in GANs. Radford et al. (2015); Salimans et al. (2016) introduced a number of empirical tricks to help stabilizing GANs. Arjovsky & Bottou (2017) showed the importance of divergences in GAN training, leading to the introduction of Wasserstein GAN (Arjovsky et al., 2017). The use of weak divergence is further explored by Mroueh & Sercu (2017); Mroueh et al. (2018). Lucas et al. (2018) advocated the use of mixed-batches, mini-batches of real and fake data,
to smooth out the loss surface. The method exploits the distributional information in a mini-batch to prevent mode collapse. VEEGAN (Srivastava et al., 2017) uses an inverse of the generator to map the data to the prior distribution. The mismatch between the inverse mapping and the prior is used to detect mode collapse. If the generator can remember the entire training set, then the inverse mapping can be arbitrarily close the the prior distribution. It suggests that VEEGAN might not be able to help GAN to generalize outside of the training dataset. Our method helps GANs to discover unseen regions of the target distribution, significantly improve the diversity of generated samples.

heading: 3 BACKGROUND
text: In the original GAN, the discriminator D maximizes the following objective
L = Ex∼pr [log(D(x))] + Ez∼pz [log(1−D(G(z)))] (1) Goodfellow et al. (2014) showed that if the density functions pg and pr are known, then for a fixed generator G the optimal discriminator is
D∗(v) = pr(v)
pr(v) + pg(v) ,∀v ∈ supp(pr) ∪ supp(pg) (2)
In the beginning of the training, pg is very different from pr so we have pr(x) pg(x), for x ∈ Dr and pg(y) pr(y), for y ∈ Dg . Therefore, in the beginning of the training D∗(x) ≈ 1, for x ∈ Dr and D∗(y) ≈ 0, for y ∈ Dg . As the training progresses, the generator will bring pg closer to pr. The game reaches the global equilibrium when pr = pg . At the global equilibrium, D∗(v) = 1 2 ,∀v ∈ supp(pr) ∪ supp(pg). One important result of the original paper is that, if the discriminator is optimal at every step of the GAN algorithm, then pg converges to pr.
In practice, density functions are not known and the optimal discriminator is approximated by optimizing the classification performance of a parametric discriminator D(·;θD) on a discrete finite dataset D = Dr ∪ Dg . We call a discriminator trained on a discrete finite dataset an empirical discriminator. The empirically optimal discriminator is denoted by D̂∗.
Arora et al. (2017) defined generalization of a divergence d as follow: A divergence d is said to have generalization error if |d(Dg,Dr)− d(pg, pr)| ≤ (3) A discriminator D defines a divergence between two distributions. The performance of a discriminator with good generalization capability on the training dataset should be similar to that on the entire data space. In practice, generalization capability of D can be estimated by measuring the difference between its performance on the training dataset and a held-out dataset.

heading: 4 GENERALIZATION CAPABILITY OF DISCRIMINATORS

heading: 4.1 THE EMPIRICALLY OPTIMAL DISCRIMINATOR DOES NOT APPROXIMATE THE THEORETICALLY OPTIMAL DISCRIMINATOR
text: It has been observed that if the discriminator is too good at discriminating real and fake samples, the generator cannot learn effectively (Goodfellow et al., 2014; Arjovsky & Bottou, 2017). The phenomenon suggests that D̂∗ does not well approximate D∗, and does not guarantee the convergence of pg to pr. In the following, we clarify the mismatch between D̂∗ and D∗, and its implications.
Proposition 1. The two datasetsDr andD(t)g are disjoint with probability 1 regardless of how close the two distributions pr and p (t) g are.
Proof. See appendix A.
Dr and D(t)g are disjoint with probability 1 even when pg and pr are exactly the same. D̂∗ perfectly classifies the real and the fake datasets, and D̂∗(x) = 1,∀x ∈ Dr , D̂∗(y) = 0,∀y ∈ D(t)g . The value of D̂∗ on D(t) does not depend on the distance between the two distributions and does not reflect the learning progress. The value of D̂∗ on the training dataset approximates that of D∗ in the beginning of the learning process but not when the two distributions are close. When trained using gradient descent on a discrete finite dataset with the loss in Eqn. 1, the discriminator D is pushed toward D̂∗, not D∗. This behavior does not depend on the size of training set (see Fig. 1a, 1b), implying that the original GAN is not guaranteed to converge to the target distribution even when given enough data.

heading: 4.2 EMPIRICAL DISCRIMINATORS HAVE POOR GENERALIZATION CAPABILITY
text: When the generator gets better, generated samples are more similar to samples from the target distribution. However, regardless of their quality, generated samples are still labeled as fake in Eqn. 1. The training dataset D is a bad dataset as it contains many mislabeled examples. A discriminator trained on such dataset will overfit to the mislabeled examples and has poor generalization capability. It will misclassify unseen samples and cannot teach the generator to generate these samples.
Figure 1a and 1b demonstrate the problem on a synthetic dataset consisting of samples from two Gaussian distributions. The discriminator in Fig. 1a overfits to the small dataset and does not generalize to new samples in Fig. 1b. Although the discriminator in Fig. 1b was trained on a larger dataset which is sufficient to characterize the two distributions, it still overfits to the data and its value surface is very different from that of the theoretically optimal discriminator in Fig. 1f.
An overfitted discriminator does not guide the model distribution toward target distribution but toward the real samples in the dataset. This explains why the original GAN usually exhibits mode collapse behavior. Finding the empirically optimal discriminator using gradient descent usually requires many iterations. Heuristically, overfitting can be alleviated by limiting the number of discriminator updates per generator update. Goodfellow et al. (2014) recommended to update the discriminator once every generator update. In the next subsection, we show that limiting the number of discriminator updates per generator update prevents the discriminator from overfitting.

heading: 4.2.1 -OPTIMAL DISCRIMINATORS
text: D̂∗ is costly to find and maintain. We consider here a weaker notion of optimality which can be achieved in practical settings.
Definition 1 ( -optimal discriminator). Given two disjoint datasetsDr andDg , and a number > 0, a discriminator D is -optimal if
D(x) ≥ 1 2 + 2 ,∀x ∈ Dr D(y) ≤ 1 2 − 2 ,∀y ∈ Dg
As observed in Goodfellow et al. (2014), D̂∗ does not generate usable gradient for the generator. Goodfellow et al. proposed the non-saturating loss for the generator to circumvent this vanishing gradient problem. For an -optimal discriminator, if is relatively small, then the gradient of the discriminator w.r.t. fake datapoints might not vanish and can be used to guide the model distribution toward the target distribution. Proposition 2. Given two disjoint datasets Dr and Dg , and a number > 0, an -optimal discriminator D exists and can be constructed as a one hidden layer MLP withO(dx(m+n)) parameters.
Proof. See appendix B.
Because deep networks are more powerful than shallow ones, the size of a deep -optimal discriminator can be much smaller than O(dx(m + n)). From the formula, the size of a shallow -optimal discriminator for real world datasets ranges from a few to hundreds of millions parameters. That is comparable to the size of discriminators used in practice. Arjovsky & Bottou (2017) showed that even when the generator can generate realistic samples, a discriminator that can perfectly classify real and fake samples can be found easily using gradient descent. The experiment verified that -optimal discriminator can be found using gradient descent in practical settings.
We observe that the norm of the gradient w.r.t. the discriminator’s parameters decreases as fakes samples approach real samples. If the discriminator’s learning rate is fixed, then the number of gradient descent steps that the discriminator has to take to reach -optimal state should increase. Proposition 3. Alternating gradient descent with the same learning rate for discriminator and generator, and fixed number of discriminator updates per generator update (Fixed-Alt-GD) cannot maintain the (empirical) optimality of the discriminator.
Fixed-Alt-GD decreases the discriminative power of the discriminator to improve its generalization capability. The proof for linear case is given in appendix C.
In GANs trained with Two Timescale Update Rule (TTUR) (Heusel et al., 2017), the ratio between the learning rate of the discriminator and that of the generator goes to infinity as the iteration number goes to infinity. Therefore, the discriminator can learn much faster than the generator and might be able to maintain its optimality throughout the learning process.

heading: 4.2.2 GRADIENT EXPLODING IN -OPTIMAL DISCRIMINATORS
text: Let’s consider a simplified scenario where the real and the fake datasets each contains a single datapoint: Dr = {x}, D(t)g = { y(t) } . Updating the generator according to the gradient from the discriminator will push y(t) toward x. The absolute value of directional derivative of D in the direction u = x− y(t), at x is
|(∇uD)x| = lim y(t) u−→x ∣∣D(x)−D(y(t))∣∣∥∥x− y(t)∥∥ If D is always -optimal, then
∣∣D(x)−D(y(t))∣∣ ≥ ,∀t ∈ N, and |(∇uD)x| ≥ lim
y(t) u−→x ∥∥x− y(t)∥∥ =∞ The directional derivate of the -optimal discriminator explodes as the fake datapoint approaches the real datapoint. Directional derivative exploding implies gradient exploding at datapoints on the line
segment connecting x and y(t). If in the next iteration, the generator produces a sample in a region where the gradient explodes, then the gradient w.r.t. the generator’s parameters explodes.
Let’s consider the following line integral
∫ C (∇D)v · ds = D(x)−D(y(t)) (4)
where C is the line segment from y(t) to x. As the model distribution gets closer to the target distribution, the length of C should be non increasing. Therefore, maximizing D(x) −D(y(t)), or the discriminative power of D, leads to the maximization of the directional derivative of D in the direction ds. The original GAN loss makes D to maximize its discriminative power, encouraging gradient exploding to occur.
Gradient exploding happens in the discriminator trained with TTUR in Fig. 2c and 2d. Because TTUR can help the discriminator to maintain its optimality, gradient exploding happens and persists throughout the training process. Without TTUR, the discriminator cannot maintain its optimality so gradient exploding can happen sometimes during the training but does not persist (Fig. 2a and 2b). Because of the saturated regions in the sigmoid function used in neural network based discriminators, the gradient w.r.t. datapoints in the training set could vanishes. However, gradient exploding must happen at some datapoints on the path between a pair of samples, where the sigmoid function does not saturate. In Fig. 1a, gradient exploding happens near the decision boundary.
In practice, Dr and Dg contain many datapoints and the generator is updated using the average of gradients of the discriminator w.r.t. fake datapoints in the mini-batch. If a fake datapoint y0 is very close to a real datapoint x0, the gradient (∇D)y0 might explode. When the average gradient is computed over the mini-batch, (∇D)y0 outweighs other gradients. The generator updated with this average gradient will move many fake datapoints in the direction of (∇D)y0 , toward x0, making mode collapse visible.

heading: 5 IMPROVING GENERALIZATION CAPABILITY OF EMPIRICAL DISCRIMINATORS
text: Although the theoretically optimal discriminator D∗ is generalizable, the original GAN loss does not push empirical discriminators toward D∗. We aim to improve the generalization capability of empirical discriminators by pushing them toward D∗.

heading: 5.1 PUSHING EMPIRICAL DISCRIMINATORS TOWARD D∗
text: For any input v ∈ supp(pr) ∪ supp(pg), the value of D∗(v) goes to 12 and the gradient (∇D)v goes to 0 as pg approaches pr. Consider again the line integral in Eqn. 4. As D∗(x) and D∗(y) approach 12 for all x ∈ supp(pr) and y ∈ supp(pg), we have
D∗(x)−D∗(y) = ∫ C (∇D∗)v · ds→ 0 (5)
for all pairs of x and y and all paths C from y to x. That means, the discriminative power of D∗ must decrease as the two distributions become more similar.
To push an empirical discriminator D toward D∗, we force D to satisfy two requirements:
1. (∇D)v → 0,∀ v ∈ supp(pr) ∪ supp(pg) 2. D(x)−D(y) = ∫ C (∇D)v · ds→ 0,∀ x ∼ pr,y ∼ pg, C from y to x

heading: 5.2 ZERO-CENTERED GRADIENT PENALTY
text: The first requirement can be implemented by sampling some datapoints v ∈ supp(pr) ∪ supp(pg) and force (∇D)v to be 0. The second requirement can be implemented by sampling pairs of real and fake datapoints (x,y) and force D(x) −D(y) to be 0. The two requirements can be added to the discriminator’s objective as follows
L̂ = L − λ1Ev[‖(∇D)v‖2]− λ2Ex,y[(D(x)−D(y))2]
whereL is the objective in Eqn. 1. However, as discussed in section 4.2.2, an -optimal discriminator can have zero gradient on the training dataset and have gradient exploding outside of the training dataset. The gradient norm could go to infinity even when D(x) − D(y) is small. Regulating the difference between D(x) and D(y) is not an efficient way to prevent gradient exploding.
We want to prevent gradient exploding on every path in supp(pr) ∪ supp(pg). Because (∇D∗)v → 0 for all v ∈ supp(pr) ∪ supp(pg) as pg approach pr, we could push the gradient w.r.t. every datapoint on every path C ∈ supp(pr) ∪ supp(pg) toward 0. We note that, if (∇D)v → 0,∀ v ∈ C then ∫ C (∇D)v · ds→ 0. Therefore, the two requirements can be enforced by a single zero-centered gradient penalty of the form λEv∈C [‖(∇D)v‖2]
The remaining problem is how to find the path C from a fake to a real sample which lies inside supp(pr) ∪ supp(pg). Because we do not have access to the full supports of pr and pg , and the supports of two distributions could be disjoint in the beginning of the training process, finding a path which lies completely inside the support is infeasible.
In the current implementation, we approximate C with the straight line connecting a pair of samples, although there is no guarantee that all datapoints on that straight line are in supp(pr) ∪ supp(pg). That results in the following objective
L0−GP = L − λEx̃[‖(∇D)x̃‖2] (6)
where x̃ = αx+ (1− α)y, x ∼ pr, y ∼ pg , and α ∼ U(0, 1) 1. We describe a more sophisticated way of finding a better path in appendix F.
1Wu et al. (2018) independently proposed the Wasserstein divergence for WGAN which uses a gradient penalty of similar form. Although the two penalties have similar approximate form, they have different motivations and addresses different problems in GANs.
The larger λ is, the stronger (∇D)x̃ is pushed toward 0. If λ is 0, then the discriminator will only focus on maximizing its discriminative power. If λ approaches infinity, then the discriminator has maximum generalization capability and no discriminative power. λ controls the tradeoff between discrimination and generalization in the discriminator.

heading: 5.3 GENERALIZATION CAPABILITY OF DIFFERENT GRADIENT PENALTIES
text: Mescheder et al. (2018) proposed to force the gradient w.r.t. datapoints in the real and/or fake dataset(s) to be 0 to make the training of GANs convergent. In section 4, we showed that for discrete training dataset, an empirically optimal discriminator D̂∗ always exists and could be found by gradient descent. Although (∇D̂∗)v = 0,∀ v ∈ D, D̂∗ does not satisfy the requirement in Eqn. 5 and have gradient exploding when some fake datapoints approach a real datapoint. The discriminators in Fig. 1a, 1b, 1d, 2c and 2d have vanishingly small gradients on datapoints in the training dataset and very large gradients outside. They have poor generalization capability and cannot teach the generator to generate unseen real datapoints. Therefore, zero-centered gradient penalty on samples from pr and pg only cannot help improving the generalization of the discriminator.
Non-zero centered GPs do not push an empirical discriminator toward D∗ because the gradient does not converge to 0. A commonly used non-zero centered GP is the one-centered GP (1-GP) (Gulrajani et al., 2017) which has the following form
λEx̃[(‖(∇D)x̃‖ − 1)2] (7) where x̃ = αx+(1−α)y, x ∼ pr, y ∼ pg , and α ∼ U(0, 1). Although the initial goal of 1-GP was to enforce Lipschitz constraint on the discriminator 2, Fedus et al. (2018) found that 1-GP prevents gradient exploding, making the original GAN more stable. 1-GP forces the norm of gradients w.r.t. datapoints on the line segment connecting x and y to be 1. If all gradients on the line segment have norm 1, then the line integral in Eqn. 4 could be as large as ‖x− y‖. Because the distance between random samples grows with the dimensionality, in high dimensional space ‖x− y‖ is greater than 1 with high probability. The discriminator could maximize the value of the line integral without violating the Lipschitz constraint. The discriminator trained with 1-GP, therefore, can overfit to the training data and have poor generalization capability.

heading: 5.4 CONVERGENCE ANALYSIS FOR ZERO-CENTERED GRADIENT PENALTY
text: Mescheder et al. (2018) showed that zero-centered GP on real and/or fake samples (0-GP-sample) makes GANs convergent. The penalty is based on the convergence analysis for the Dirac GAN, an 1-dimensional linear GAN which learns the Dirac distribution. The intuition is that when pg is the same as pr, the gradient of the discriminator w.r.t. the fake datapoints (which are also real datapoints) should be 0 so that generator will not move away when being updated using this gradient. If the gradient from the discriminator is not 0, then the generator will oscillate around the equilibrium.
Our GP forces the gradient w.r.t. all datapoints on the line segment between a pair of samples (including the two endpoints) to be 0. As a result, our GP also prevents the generator from oscillating. Therefore, our GP has the same convergence guarantee as the 0-GP-sample.

heading: 5.5 ZERO-CENTERED GRADIENT PENALTY IMPROVES CAPACITY DISTRIBUTION
text: Discriminators trained with the original GAN loss tends to focus on the region of the where fake samples are close to real samples, ignoring other regions. The phenomenon can be seen in Fig. 2a, 2b, 2c, 2d, 2h and 2i. Gradients in the region where fake samples are concentrated are large while gradients in other regions, including regions where real samples are located, are very small. The generator cannot discover and generate real datapoints in regions where the gradient vanishes.
When trained with the objective in Eqn. 6, the discriminator will have to balance between maximizing L and minimizing the GP. For finite λ, the GP term will not be exactly 0. Let γ = λEx̃[‖(∇D)x̃‖2]. Among discriminators with the same value of γ, gradient descent will find the discriminator that maximizes L. As discussed in section 4.2.2, maximizing L leads to the
2Petzka et al. (2018) pointed out that 1-GP is based on the wrong intuition that the gradient of the optimal critic must be 1 everywhere under pr and pg . The corrected GP is based on the definition of Lipschitzness.
maximization of norms of gradients on the path from y to x. The discriminator should maximize the value η = Ex̃[‖(∇D)x̃‖]. If γ is fixed then η is maximized when ‖∇Dx̃(i)‖ = ‖∇Dx̃(j)‖,∀ i, j (Cauchy-Schwarz inequality). Therefore, our zero-centered GP encourages the gradients at different regions of the real data space to have the same norm. The capacity of D is distributed more equally between regions of the real data space, effectively reduce mode collapse. The effect can be seen in Fig. 2e and 2f.
1-GP encourages |‖∇Dx̃(i)‖ − 1| = |‖∇Dx̃(j)‖ − 1|,∀ i, j. That allows gradient norms to be smaller than 1 in some regions and larger than 1 in some other regions. The problem can be seen in Fig. 2h.

heading: 6 EXPERIMENTS
text: The code is made available at https://github.com/htt210/ GeneralizationAndStabilityInGANs.

heading: 6.1 ZERO-CENTERED GRADIENT PENALTY PREVENTS OVERFITTING
text: To test the effectiveness of gradient penalties in preventing overfitting, we designed a dataset with real and fake samples coming from two Gaussian distributions and trained a MLP based discriminator on that dataset. The result is shown in Fig. 1. As predicted in section 5.3, 0-GP-sample does not help to improve generalization. 1-GP helps to improve generalization. The value surface in Fig. 1c is smoother than that in Fig. 1a. However, as discussed in section 5.3, 1-GP cannot help much in higher dimensional space where the pair-wise distances are large. The discriminator trained with our 0-GP has the best generalization capability, with a value surface which is the most similar to that of the theoretically optimal one.
We increased the number of discriminator updates per generator update to 5 to see the effect of GPs in preventing overfitting. On the MNIST dataset, GAN without GP and with other GPs cannot learn anything after 10,000 iterations. GAN with our 0-GP can still learn normally and start produce recognizable digits after only 1,000 iterations. The result confirms that our GP is effective in preventing overfitting in the discriminator.

heading: 6.2 ZERO-CENTERED GRADIENT PENALTY IMPROVES GENERALIZATION AND ROBUSTNESS
text: OF GANS
SYNTHETIC DATA
We tested different gradient penalties on a number of synthetic datasets to compare their effectiveness. The first dataset is a mixture of 8 Gaussians. The dataset is scaled up by a factor of 10 to simulate the situation in high dimensional space where random samples are far from each other. The result is shown in Fig. 2. GANs with other gradient penalties all fail to learn the distribution and exhibit mode collapse problem to different extents. GAN with our 0-GP (GAN-0-GP) can successfully learn the distribution. Furthermore, GAN-0-GP can generate datapoints on the circle, demonstrating good generalization capability. The original GAN collapses to some disconnected modes and cannot perform smooth interpolation between modes: small change in the input result in large, unpredictable change in the output. GAN with zero-centered GP on real/fake samples only also exhibits the same ”mode jumping” behavior. The behavior suggests that these GANs tend to remember the training dataset and have poor generalization capability. Fig. 9 in appendix D demonstrates the problem on MNIST dataset.
We observe that GAN-0-GP behaves similar to Wasserstein GAN as it first learns the overall structure of the distribution and then focuses on the modes. An evolution sequence of GAN-0-GP is shown in Fig. 5 in appendix D. Results on other synthetic datasets are shown in appendix D.
MNIST DATASET
The result on MNIST dataset is shown in Fig. 3. After 1,000 iterations, all other GANs exhibit mode collapse or cannot learn anything. GAN-0-GP is robust to changes in hyper parameters such
as learning rate and optimizers. When Adam is initialized with large β1, e.g. 0.9, GANs with other GPs cannot learn anything after many iterations. More samples are given in appendix D.
We observe that higher value of λ improves the diversity of generated samples. For λ = 50, we observe some similar looking samples in the generated data. This is consistent with our conjecture that larger λ leads to better generalization.
IMAGENET
When trained on ImangeNet (Deng et al., 2009), GAN-0-GP can produce high quality samples from all 1,000 classes. We compared our method with GAN with 0-GP-sample and WGAN-GP. GAN-0GP-sample is able to produce samples of state of the art quality without using progressive growing trick (Karras et al., 2018). The result in Fig. 4 shows that our method consistently outperforms GAN-0-GP-sample. GAN-0-GP and GAN-0-GP-sample outperform WGAN-GP by a large margin. Image samples are given in appendix D.

heading: 7 CONCLUSION
text: In this paper, we clarify the reason behind the poor generalization capability of GAN. We show that the original GAN loss does not guide the discriminator and the generator toward a generalizable equilibrium. We propose a zero-centered gradient penalty which pushes empirical discriminators toward the optimal discriminator with good generalization capability. Our gradient penalty provides better generalization and convergence guarantee than other gradient penalties. Experiments on diverse datasets verify that our method significantly improves the generalization and stability of GANs.

heading: A PROOF FOR PROPOSITION 1
text: For continuous random variable V , P(V = v) = 0 for any v. The probability of finding a noise vector z such that G(z) is exactly equal to a real datapoint x ∈ Dr via random sampling is 0. Therefore, the probability of a real datapoint xi being in the fake dataset Dg is 0. Similarly, the probability of any fake datapoint being in the real dataset is 0.
P(x ∈ D(t)g ) = 0,∀x ∈ Dr, t ∈ N
P(y ∈ Dr) = 0,∀y ∈ D(t)g , t ∈ N
P(Dr ∩ D(t)g = ∅) = 1,∀t ∈ N (8)
Furthermore, due to the curse of dimensionality, the probability of sampling a datapoint which is close to another datapoint in high dimensional space also decrease exponentially. The distances between datapoints are larger in higher dimensional space. That suggests that it is easier to separate Dr and D(t)g in higher dimensional space.

heading: B CONSTRUCTING -OPTIMAL DISCRIMINATORS
text: To make the construction process simpler, let’s assume that samples are normalized:
‖xi‖ = ‖yj‖ = 1,∀xi ∈ Dr,yj ∈ Dg Let’s use the following new notations for real and fake samples:
D = Dr ∪ Dg = {v1, ...,vm+n}
vi = { xi, for i = 1, ..., n yi−n, for i = n+ 1, ..., n+m
We construct the -optimal discriminator D as a MLP with 1 hidden layer. Let W1 ∈ R(m+n)×dx and W2 ∈ Rm+n be the weight matrices of D. The total number of parameters in D is dx(m + n) + (m+ n) = O(dx(m+ n)). We set the value ofW1 as
W1 = k  v > 1 ...
v>n+m  andW2 as
W2,i = 1
2 + 2 + α, for i = 1, ..., n
W2,i = 1
2 − 2 − α, for i = n+ 1, ..., n+m
α > 0
Given an input v ∈ D, the output is computed as:
D(v) =W>2 σ(W1v)
where σ is the softmax function. Let a =W1v, we have
ai = kv > i v = { k, if v = vi < k, if v 6= vi
As k → ∞, σ(W1vi) becomes a one-hot vector with the i-th element being 1, all other elements being 0. Thus, for large enough k, for any vj ∈ D, the output of the network is
D(vj) =W > 2 σ(W1vj) ≈W2,j =
{ > 12 + 2 , for j = 1, ..., n
< 12 − 2 , for j = n+ 1, ..., n+m
D is a -optimal discriminator for dataset D.

heading: C FIXED-ALT-GD CANNOT MAINTAIN THE OPTIMALITY OF
text: -DISCRIMINATORS
Let’s consider the case where the real and the fake dataset each contain a single datapointDr = {x}, D(t)g = y(t), and the discriminator and the generator are linear:
D(v) = θ>Dv
G(z) = θGz
and the objective is also linear (Wasserstein GAN’s objective):
LW = Ex∈Dr [D(x)]− Ey∈D(t)g [D(y)]
= D(x)−D(y(t))
The same learning rate α is used for D and G.
At step t, the discriminator is -optimal D(x)−D(y(t)) = θ>D ( x− y(t) ) ≥ (9)
‖θD‖ ≥ ∥∥x− y(t)∥∥ (10)
The gradients w.r.t. θD and θG are
∂L ∂θD = x− y(t) (11)
∂L ∂θG = ∂L ∂y(t) × z>
= θD × z> (12) If the learning rate α is small enough, ∥∥x− y(t)∥∥ should decrease as t increases. As the empirical
fake distribution converges to the empirical real distribution, ∥∥x− y(t)∥∥→ 0. The norm of gradient w.r.t. θD, therefore, decreases as t increases and vanishes when the two empirical distributions are the same. From Eqn. 10, we see that, in order to maintain D’s -optimality when
∥∥x− y(t)∥∥ decreases, ‖θD‖ has to increase. From Eqn. 10 and 12, we see that the gradient w.r.t. θG grows as the two empirical distributions are more similar. As
∥∥x− y(t)∥∥→ 0,∥∥∥∂LW∂θD ∥∥∥∥∥∥∂LW∂θG ∥∥∥ → 0 (13) Because the same learning rate α is used for both G and D, G will learn much faster than D. Furthermore, because
∥∥x− y(t)∥∥ decreases as t increases, the difference ∥∥x− y(t+1)∥∥ − ∥∥x− y(t)∥∥
increases with t. The number of gradient steps that D has to take to reach the next -optimal state increases, and goes to infinity as ∥∥x− y(t)∥∥ → 0. Therefore, gradient descent with fixed number of updates to θD cannot maintain the optimality of D.
The derivation for the objective in Eqn. 1 is similar.

heading: D RESULTS ON DIFFERENT DATASETS
text: E IMPLEMENTATION DETAILS
We used Pytorch (Paszke et al., 2017) for development.
E.1 SYNTHETIC AND MNIST DATASETS
Generator architecture in synthetic and MNIST experiments
Fully connected layer 2× nhidden ↓
ReLU ↓
Fully connected layer nhidden× nhidden ↓
ReLU ↓
Fully connected layer nhidden× nhidden ↓
Fully connected layer nhidden× 2
Discriminator architecture in synthetic and MNIST experiments
Fully connected layer 2× nhidden ↓
ReLU ↓
Fully connected layer nhidden× nhidden ↓
ReLU ↓
Fully connected layer nhidden× nhidden ↓
Fully connected layer nhidden× 1 ↓
Sigmoid
Hyper parameters for synthetic and MNIST experiments
Learning rate 0.003 for both G and D Learning rate TTUR 0.003 for G, 0.009 for D
E.2 IMAGENET
The entire ImageNet dataset with all 1000 classes was used in the experiment. Because of our hardware limits, we used images of size 64× 64. We used the code from Mescheder et al. (2018), available at https://github.com/LMescheder/GAN stability, for our experiment. Generator and Discriminator are ResNets, each contains 5 residual blocks. All GANs in our experiment have the same architectures and hyper parameters. The configuration for WGAN-GP5 is as follows.
generator: name: resnet2 kwargs:
nfilter: 32 nfilter_max: 512 embed_size: 128
discriminator: name: resnet2 kwargs:
nfilter: 32 nfilter_max: 512 embed_size: 128
z_dist: type: gauss dim: 128
training: out_dir: ../output/imagenet_wgangp5_TTUR gan_type: wgan reg_type: wgangp reg_param: 10. batch_size: 64 nworkers: 32 take_model_average: true model_average_beta: 0.999 model_average_reinit: false monitoring: tensorboard sample_every: 1000 sample_nlabels: 20 inception_every: 10000 save_every: 900 backup_every: 100000 restart_every: -1 optimizer: adam lr_g: 0.0001 lr_d: 0.0003 lr_anneal: 1. lr_anneal_every: 150000 d_steps: 5 equalize_lr: false

heading: F FINDING A BETTER PATH BETWEEN A PAIR OF SAMPLES
text: Because the set of real data is unlikely to be convex, a linear interpolation between two datapoints is unlikely to be in the set. For example, the weighted average of two real images is often not a real image. If the generator is good enough then its output is likely to be on the real data manifold. A path C from y to x can be found by transforming the straight line between the two corresponding latent codes, to the data space using the generator G. To get the latent code of real datapoints, we use an encoder E which is trained to map real data to normally distributed latent codes. The pseudo code of the path finding algorithm is shown in Alg. 1.
Data: an encoder E; a pair of samples x,y = G(z); Result: interpolated datapoint x̃
1 Get the latent code of x: zx = E(x) 2 Calculate the interpolated latent code: z̃ = αzx + (1− α)z 3 Generate the interpolated datapoint: x̃ = G(z̃)
Algorithm 1: Path finding algorithm

Review summaries:
The paper proposes to add to the original GAN (2014) loss a zero-centered gradient penalty as the one defined in the WGAN-GP paper. It also provides an analysis on the mode collapse and lack of stability of classical GANs. The authors compare results using their penalty on a few synthetic examples and on image net dogs generations to results using the classical GAN loss with or without gradient penalties.