ROBUST GENERATIVE ADVERSARIAL NETWORK

INTRODUCTION . Generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) have been enjoying much attention recently due to their great success on different tasks and datasets ( Radford et al. , 2015 ) ( Salimans et al. , 2016 ) ( Ho & Ermon , 2016 ) ( Li et al. , 2017 ) ( Chongxuan et al. , 2017 ) . The framework of GANs can be formulated as the game between generator and discriminator . The generator tries to produce the fake distribution which approximates the real data distribution , while the discriminator attempts to distinguish the fake distribution from the real distribution . These two players compete with each other iteratively . GANs are also popular for their theoretical value . Training the discriminator is showed to equivalent to training a good estimator for the density ratio between the fake distribution and the real one ( Nowozin et al. , 2016 ) ( Uehara et al. , 2016 ) ( Mohamed & Lakshminarayanan , 2016 ) . The discriminator generally measures the departure between the model distribution and real data distribution with certain divergence measure , e.g . Jensen-Shannon divergence or f - divergence ( Nowozin et al. , 2016 ) . Arjovsky et al . proved that the supports of the fake and real distributions are typically disjoint on low dimensional manifolds and there is a nearly trivial discriminator which can correctly classify the real and fake data ( Arjovsky et al. , 2017 ) . The loss of such discriminator converges quickly to zero which causes the vanishing gradient for generator . To alleviate such problem , Arjovsky et al . proposed the Wasserstein GAN based on Wasserstein metric requiring no joint supports . Since it is inconvenient to minimize directly the Wasserstein distance , they solve the dual problem by clipping the weights to ensure the Lipschitz condition for discriminator . Later Gulrajani et al . proposed the gradient penalty to guarantee the Lipschitz condition ( Gulrajani et al. , 2017 ) . Spectral normalization is also proposed to stabilize the training of the discriminator ( Miyato et al. , 2018 ) . Most existing methods try to improve the stability of GANs by controlling the discriminator . However , the robustness of GANs have not adequately been considered . When the discriminator is not robust to noise ( i.e. , the discriminator can not measure the distance between the fake and real distribution accurately ) , some examples might be mis-classified which consequently misleads the training of generator . Meanwhile , the poor generalization performance of generator might cause the “ blurry ” generated images for some potential input noise . Robust Conditional Generative Adversarial Net- works is proposed to improve the robustness of conditional GAN for noised data . However , this method can merely implement on conditional GAN which improves the ability of generator only to defend the noise ( Chrysos et al. , 2018 ) . Some other researchers focus on the robustness of GAN to label noise ( Thekumparampil et al. , 2018 ) ( Kaneko et al. , 2019 ) . In this paper , we attempt to improve the robustness of GANs in a systematical way by promoting the robustness of both discriminator and generator . We propose a novel robust method called robust generative adversarial network ( RGAN ) where the generator and discriminator still compete with each other iteratively , but in a worst-case setting . Specifically , a robust optimization is designed with considering the worst distribution within the small Wasserstein ball . The generator tries to map the worst input distribution ( rather than a specific distribution ) to the real data distribution , while the discriminator attempts to distinguish the real and fake distribution with the worst perturbation . We provide some theoretical analysis for the proposed Robust GAN including generalization . We also implement our robust framework on different baseline GANs ( i.e. , DCGAN , WGAN-GP , and BWGAN ) ( Radford et al. , 2015 ) ( Adler & Lunz , 2018 ) , observing substantial improvements consistently on all the datasets used in this paper . GENERATIVE ADVERSARIAL NETWORK . The principle of GAN is a game between two players : generator and discriminator , both of which are usually formulated as the deep neural networks . The generator tries to generate a fake example to fool discriminator , while the discriminator attempts to distinguish between fake and real images . Formally , the training procedure of GAN can be formulated as : min G max D S ( G , D ) , Ex∼Pr [ logD ( x ) ] + Ex̃∼Pg [ ( 1− logD ( G ( zi ) ) ) ] ( 1 ) where , x and x̃ = G ( z ) are real and fake examples sampled from the real data distribution Pr and generation distribution Pg respectively . The generation distribution is defined by G ( z ) where z ∼ Pz ( Pz is a specific input noise distribution ) . The minmax problem can not be solved directly since the expectation of the real and generation distribution is usually intractable . Therefore , the approximation problem is defined as : min G max D Sm ( G , D ) , 1 m m∑ i=1 [ logD ( xi ) ] + 1 m m∑ i=1 [ ( 1− logD ( G ( zi ) ) ) ] ( 2 ) where m examples of xi and zi are sampled from distributions Pr and Pz and the mean value of loss is used to approximate the original problem . However , such a way might not ensure a good robustness of discriminator and generator . Some noised images might not be classified correctly and potential input noise points will cause degraded generation . In this paper , for alleviating such problem , we design a distributionally robust optimization . Particularly , we consider the worst distribution ( rather than a specific single distribution ) within the small range . ROBUST GENERATIVE ADVERSARIAL NETWORK . As we discussed in the previous sections , although most existing GAN methods can stabilize the training of the discriminator , the robustness might not be adequately considered . In other words , the discriminator might not perform well on some noised data which consequently misleads the training of generator . Similarly , the generator might produce poor generations for certain input noise points if its robustnessis not good . To alleviate such problem , we design the distributionally robust optimization on GAN . Before we discuss how we can achieve this , we first elaborate the distributionally robust optimization . DISTRIBUTIONALLY ROBUST OPTIMIZATION . Let d : X ×X → R+ ∪ { ∞ } . The departure between x and x0 can then be represented by d ( x , x0 ) . For distributionally robust optimization , the robustness region P = { P : D ( P , P0 ) ≤ ρ } is considered , a ρ -neighborhood of the distribution P0 under the divergence D ( . , . ) instead of a single distribution1 . The distributionally robust optimization can be formulated as ( Sinha et al. , 2017 ) : min θ sup P∈P EP [ l ( X ; θ ) ] ( 3 ) where l ( . ) is a loss function parameterized by θ . The problem of ( 3 ) is typically intractable for arbitrary ρ . In order to solve this problem , we first present a proposition : Proposition 0.1 Let l : θ ×X → R and d : X ×X → R+ be continuous . Then , for any distribution P0 and ρ > 0 we have sup P∈P { EP [ l ( X ; θ ) ] − γW ( P , P0 ) } = EP0 [ sup x∈X { l ( x ; θ ) − γd ( x , x0 ) } ] ( 4 ) ( Proof is provided in ( Sinha et al. , 2017 ) ) . With Proposition 0.1 , we can reformulate ( 3 ) with the Lagrangian relaxation as follows : min θ EP0 sup x∈X [ l ( x ; θ ) − λd ( x , x0 ) ] ( 5 ) where the second term d ( x , x0 ) is to restrict the distance between two points . ROBUST TRAINING OVER GENERATOR . With the distributionally robust optimization , we first discuss how we can perform robust training over generator . The generator of GAN tries to map a noise distribution Pz to the image distribution Pr . The objective of generator is described as follows : min G 1 m m∑ i=1 [ log ( 1−D ( G ( zi ) ) ) ] , where zi ∼ Pz ( 6 ) Typically , Pz is a Gaussian distribution . For improving the robustness , we consider all the possible distributions within the robust region Pz = { P : W ( P , Pz ) ≤ ρz } rather than a single specific distribution ( typically a Gaussain in most existing GANs ) . Here we use the Wasserstein metric to measure the distance between P and Pz , where P is the ρz-neighbor of the original distribution Pz . However , it is difficult to consider all the distributions in this small region , the alternative way is to consider their upper bound ( the worst distribution ) . The robust optimization problem for G is then described as follows : min G sup P∈Pz 1 m m∑ i=1 [ log ( 1−D ( G ( zi ) ) ) ] , where zi ∼ P ( 7 ) According to Proposition 0.1 , we can relax ( 7 ) as : min G max r 1 m m∑ i=1 [ log ( 1−D ( G ( zi + ri ) ) ) − λz‖r‖22 ] , where zi ∼ Pz ( 8 ) Different from those previous methods , our method attempts to map the worst distribution ( in the ρz-neighborhood of the original distribution Pz ) to the image distribution . Intuitively , we sample the noise points which are most likely ( or the worst ) to generate the blurry images and optimize the generator based on these risky points . Therefore , such generator would be robust against poor input noises and might be less likely to generate the low-quality images . ROBUST TRAINING OVER DISCRIMINATOR . In traditional GANs described by ( 2 ) , the generator attempts to generate a fake distribution to approximate the real data distribution , while the discriminator tries to learn the decision boundary to separate real and fake distributions . Apparently , a discriminator with a poor robustness would inevitably mislead the training of generator . In this section , we utilize the popular adversarial learning 1Normally , the Wasserstein metric W ( . , . ) is used and corresponding d ( x , x0 ) = ‖x− x0‖2p where p > 0 method and propose the robust optimization method to improve the discriminator ’ s robustness both for clean and noised data . Specifically , we define the robust regions for both the fake distribution Pg = { P : W ( P , Pg ) ≤ ρg } and real distribution Pr = { P : W ( P , Pr ) ≤ ρr } . The generator tries to reduce the distance between the fake distribution Pg and real distribution Pr . The discriminator attempts to separate the worst distributions in Pg and Pr . Intuitively , the worst distributions are closer to decision boundary ( less discriminative ) and they are able to guide the training of discriminator to perform well on ” confusing ” data points near the classification boundary ( such discriminator can be more robust than original one ) . We can reformulate ( 2 ) in the robust version : max D sup P1∈Pr 1 m m∑ i=1 [ logD ( x′i ) ] + sup P2∈Pg 1 m m∑ i=1 [ log ( 1−D ( G′ ( zi ) ) ) ] ( 9 ) where zi ∼ Pz , x′i ∼ P1 and G′ ∼ P2 . Using Proposition 0.1 , we can relax the alternate problem as : max D min r1 , r2 1 m m∑ i=1 [ logD ( xi + r i 1 ) ] + 1 m m∑ i=1 [ log ( 1−D ( G ( zi ) + ri2 ) ) ] + λd m m∑ i=1 [ ‖ri1‖22 ] + ‖ri2‖22 ] with zi ∼ Pz , xi ∼ Pr ( 10 ) Here r1 = { ri1 } mi=1 is the set of small perturbations for the points sampled from real distribution Pd which tries to make the real distribution closer to the fake distribution . r2 = { ri2 } mi=1 tries to make fake distribution closer to real one . Intuitively , these perturbations try to enhance the difficulty of classification task for discriminator by making real and fake data less distinguishable and it can help promote the robustness of discriminator .

Summary: Developing stable GAN training method has gained much attention these years.  This paper propose to tackle this issue via involving distributionally robust optimization into GAN training. Its main contribution is to combine Sinha et al with GAN, proposing a new GAN training method on the basis of vanilla GAN. Relative theory results are proved and detailed experiments are conducted.