Improving Generalization and Stability of Generative Adversarial Networks

Generative Adversarial Networks ( GANs ) are one of the most popular tools for learning complex high dimensional distributions . However , generalization properties of GANs have not been well understood . In this paper , we analyze the generalization of GANs in practical settings . We show that discriminators trained on discrete datasets with the original GAN loss have poor generalization capability and do not approximate the theoretically optimal discriminator . We propose a zero-centered gradient penalty for improving the generalization of the discriminator by pushing it toward the optimal discriminator . The penalty guarantees the generalization and convergence of GANs . Experiments on synthetic and large scale datasets verify our theoretical analysis . 1 INTRODUCTION . GANs ( Goodfellow et al. , 2014 ) are one of the most popular tools for modeling high dimensional data . The original GAN is , however , highly unstable and often suffers from mode collapse . Much of recent researches has focused on improving the stability of GANs ( Radford et al. , 2015 ; Arjovsky et al. , 2017 ; Heusel et al. , 2017 ; Miyato et al. , 2018 ; Karras et al. , 2018 ) . On the theoretical aspect , Nagarajan & Kolter ( 2017 ) proved that gradient based training of the original GAN is locally stable . Heusel et al . ( 2017 ) further proved that GANs trained with Two Timescale Update Rule ( TTUR ) converge to local equilibria . However , the generalization of GANs at local equilibria is not discussed in depth in these papers . Arora et al . ( 2017 ) showed that the generator can win by remembering a polynomial number of training examples . The result implies that a low capacity discriminator can not detect the lack of diversity . Therefore , it can not teach the generator to approximate the target distribution . In section 4 , we discuss the generalization capability of high capacity discriminators . We show that high capacity discriminators trained with the original GAN loss tends to overfit to the mislabeled samples in training dataset , guiding the generator toward collapsed equilibria ( i.e . equilibria where the generator has mode collapse ) . Arora et al . ( 2018 ) proposed to measure the generalization capability of GAN by estimating the number of modes in the model distribution using the birthday paradox . Experiments on several datasets showed that the number of modes in the model distribution is several times greater than the number of training examples . The author concluded that although GANs might not be able to learn distributions , they do exhibit some level of generalization . Our analysis shows that poor generalization comes from the mismatch between discriminators trained on discrete finite datasets and the theoretically optimal discriminator . We propose a zero-centered gradient penalty for improving the generalization capability of ( high capacity ) discriminators . Our zero-centered gradient penalty pushes the discriminator toward the optimal one , making GAN to converge to equilibrium with good generalization capability . Our contributions are as follow : 1 . We show that discriminators trained with the original GAN loss have poor generalization capability . Poor generalization in the discriminator prevents the generator from learning the target distribution . 2 . We show that the original GAN objective encourages gradient exploding in the discriminator . Gradient exploding in the discriminator can lead to mode collapse in the generator . 3 . We propose a zero-centered gradient penalty ( 0-GP ) for improving the generalization capability of the discriminator . We show that non-zero centered GP and the zero-centered GP proposed in Mescheder et al . ( 2018 ) can not make the discriminator generalize . Our 0-GP helps GANs to converge to generalizable equilibria . Theoretical results are verified on real world datasets . 4 . We show that 0-GP helps the discriminator to distribute its capacity more equally between regions of the space , effectively preventing mode collapse . Experiments on synthetic and real world datasets verify that 0-GP can prevent mode collapse . GANs with 0-GP is much more robust to changes in hyper parameters , optimizers , and network architectures than the original GAN and GANs with other gradient penalties . Table 1 compares the key properties of our 0-GP with one centered GP ( 1-GP ) ( Gulrajani et al. , 2017 ) and zero centered GP on real/fake samples only ( 0-GP-sample ) ( Mescheder et al. , 2018 ) . NOTATIONS pr the target distribution pg the model distribution pz the noise distribution dx the dimensionality of a data sample ( real or fake ) dz the dimensionality of a noise sample supp ( p ) the support of distribution p x ∼ pr a real sample z ∼ pz a noise vector drawn from the noise distribution pz y = G ( z ) a generated sample Dr = { x1 , ... , xn } the set of n real samples D ( t ) g = { y ( t ) 1 , ... , y ( t ) m } the set of m generated samples at step t D ( t ) = Dr ∪ D ( t ) g the training dataset at step t 2 RELATED WORKS . Gradient penalties are widely used in GANs literature . There are a plethora of works on using gradient penalty to improve the stability of GANs ( Mescheder et al. , 2018 ; Gulrajani et al. , 2017 ; Petzka et al. , 2018 ; Roth et al. , 2017 ; Qi , 2017 ) . However , these works mostly focused on making the training of GANs stable and convergent . Our work aims to improve the generalization capability of GANs via gradient regularization . Arora et al . ( 2018 ) showed that the number of modes in the model distribution grows linearly with the size of the discriminator . The result implies that higher capacity discriminators are needed for better approximation of the target distribution . Zhang et al . ( 2018 ) studied the tradeoff between generalization and discrimination in GANs . The authors showed that generalization is guaranteed if the discriminator set is small enough . In practice , rich discriminators are usually used for better discriminative power . Our GP makes rich discriminators generalizable while remaining discriminative . Although less mode collapse is not exactly the same as generalization , the ability to produce more diverse samples implies better generalization . There are a large number of papers on preventing mode collapse in GANs . Radford et al . ( 2015 ) ; Salimans et al . ( 2016 ) introduced a number of empirical tricks to help stabilizing GANs . Arjovsky & Bottou ( 2017 ) showed the importance of divergences in GAN training , leading to the introduction of Wasserstein GAN ( Arjovsky et al. , 2017 ) . The use of weak divergence is further explored by Mroueh & Sercu ( 2017 ) ; Mroueh et al . ( 2018 ) . Lucas et al . ( 2018 ) advocated the use of mixed-batches , mini-batches of real and fake data , to smooth out the loss surface . The method exploits the distributional information in a mini-batch to prevent mode collapse . VEEGAN ( Srivastava et al. , 2017 ) uses an inverse of the generator to map the data to the prior distribution . The mismatch between the inverse mapping and the prior is used to detect mode collapse . If the generator can remember the entire training set , then the inverse mapping can be arbitrarily close the the prior distribution . It suggests that VEEGAN might not be able to help GAN to generalize outside of the training dataset . Our method helps GANs to discover unseen regions of the target distribution , significantly improve the diversity of generated samples . 3 BACKGROUND . In the original GAN , the discriminator D maximizes the following objective L = Ex∼pr [ log ( D ( x ) ) ] + Ez∼pz [ log ( 1−D ( G ( z ) ) ) ] ( 1 ) Goodfellow et al . ( 2014 ) showed that if the density functions pg and pr are known , then for a fixed generator G the optimal discriminator is D∗ ( v ) = pr ( v ) pr ( v ) + pg ( v ) , ∀v ∈ supp ( pr ) ∪ supp ( pg ) ( 2 ) In the beginning of the training , pg is very different from pr so we have pr ( x ) pg ( x ) , for x ∈ Dr and pg ( y ) pr ( y ) , for y ∈ Dg . Therefore , in the beginning of the training D∗ ( x ) ≈ 1 , for x ∈ Dr and D∗ ( y ) ≈ 0 , for y ∈ Dg . As the training progresses , the generator will bring pg closer to pr . The game reaches the global equilibrium when pr = pg . At the global equilibrium , D∗ ( v ) = 1 2 , ∀v ∈ supp ( pr ) ∪ supp ( pg ) . One important result of the original paper is that , if the discriminator is optimal at every step of the GAN algorithm , then pg converges to pr . In practice , density functions are not known and the optimal discriminator is approximated by optimizing the classification performance of a parametric discriminator D ( · ; θD ) on a discrete finite dataset D = Dr ∪ Dg . We call a discriminator trained on a discrete finite dataset an empirical discriminator . The empirically optimal discriminator is denoted by D̂∗ . Arora et al . ( 2017 ) defined generalization of a divergence d as follow : A divergence d is said to have generalization error if |d ( Dg , Dr ) − d ( pg , pr ) | ≤ ( 3 ) A discriminator D defines a divergence between two distributions . The performance of a discriminator with good generalization capability on the training dataset should be similar to that on the entire data space . In practice , generalization capability of D can be estimated by measuring the difference between its performance on the training dataset and a held-out dataset . 4 GENERALIZATION CAPABILITY OF DISCRIMINATORS . 4.1 THE EMPIRICALLY OPTIMAL DISCRIMINATOR DOES NOT APPROXIMATE THE THEORETICALLY OPTIMAL DISCRIMINATOR . It has been observed that if the discriminator is too good at discriminating real and fake samples , the generator can not learn effectively ( Goodfellow et al. , 2014 ; Arjovsky & Bottou , 2017 ) . The phenomenon suggests that D̂∗ does not well approximate D∗ , and does not guarantee the convergence of pg to pr . In the following , we clarify the mismatch between D̂∗ and D∗ , and its implications . Proposition 1 . The two datasetsDr andD ( t ) g are disjoint with probability 1 regardless of how close the two distributions pr and p ( t ) g are . Proof . See appendix A. Dr and D ( t ) g are disjoint with probability 1 even when pg and pr are exactly the same . D̂∗ perfectly classifies the real and the fake datasets , and D̂∗ ( x ) = 1 , ∀x ∈ Dr , D̂∗ ( y ) = 0 , ∀y ∈ D ( t ) g . The value of D̂∗ on D ( t ) does not depend on the distance between the two distributions and does not reflect the learning progress . The value of D̂∗ on the training dataset approximates that of D∗ in the beginning of the learning process but not when the two distributions are close . When trained using gradient descent on a discrete finite dataset with the loss in Eqn . 1 , the discriminator D is pushed toward D̂∗ , not D∗ . This behavior does not depend on the size of training set ( see Fig . 1a , 1b ) , implying that the original GAN is not guaranteed to converge to the target distribution even when given enough data .

Summary: The paper proposes to add to the original GAN (2014) loss a zero-centered gradient penalty as the one defined in the WGAN-GP paper. It also provides an analysis on the mode collapse and lack of stability of classical GANs. The authors compare results using their penalty on a few synthetic examples and on image net dogs generations to results using the classical GAN loss with or without gradient penalties.