Local Stability and Performance of Simple Gradient Penalty $\mu$-Wasserstein GAN

Wasserstein GAN ( WGAN ) is a model that minimizes the Wasserstein distance between a data distribution and sample distribution . Recent studies have proposed stabilizing the training process for the WGAN and implementing the Lipschitz constraint . In this study , we prove the local stability of optimizing the simple gradient penalty µ-WGAN ( SGP µ-WGAN ) under suitable assumptions regarding the equilibrium and penalty measure µ . The measure valued differentiation concept is employed to deal with the derivative of the penalty terms , which is helpful for handling abstract singular measures with lower dimensional support . Based on this analysis , we claim that penalizing the data manifold or sample manifold is the key to regularizing the original WGAN with a gradient penalty . Experimental results obtained with unintuitive penalty measures that satisfy our assumptions are also provided to support our theoretical results . 1 INTRODUCTION . Deep generative models reached a turning point after generative adversarial networks ( GANs ) were proposed by Goodfellow et al . ( 2014 ) . GANs are capable of modeling data with complex structures . For example , DCGAN can sample realistic images using a convolutional neural network ( CNN ) structure ( Radford et al. , 2015 ) . GANs have been implemented in many applications in the field of computer vision with good results , such as super-resolution , image translation , and text-to-image generation ( Ledig et al. , 2017 ; Isola et al. , 2017 ; Zhang et al. , 2017 ; Reed et al. , 2016 ) . However , despite these successes , GANs are affected by training instability and mode collapse problems . GANs often fail to converge , which can result in unrealistic fake samples . Furthermore , even if GANs successfully synthesize realistic data , the fake samples exhibit little variability . A common solution to this instability problem is injecting an instance noise and finding different divergences . The injection of instance noise into real and fake samples during the training procedure was proposed by Sønderby et al . ( 2017 ) , where its positive impact on the low dimensional support for the data distribution was shown to be a regularizing factor based on the Wasserstein distance , as demonstrated analytically by Arjovsky & Bottou ( 2017 ) . In f -GAN , f -divergence between the target and generator distributions was suggested which generalizes the divergence between two distributions ( Nowozin et al. , 2016 ) . In addition , a gradient penalty term which is related with Sobolev IPM ( Integral Probability Metric ) between data distribution and sample distribution was suggested by Mroueh et al . ( 2018 ) . The Wasserstein GAN ( WGAN ) is known to resolve the problems of generic GANs by selecting the Wasserstein distance as the divergence ( Arjovsky et al. , 2017 ) . However , WGAN often fails with simple examples because the Lipschitz constraint on discriminator is rarely achieved during the optimization process and weight clipping . Thus , mimicking the Lipschitz constraint on the discriminator by using a gradient penalty was proposed by Gulrajani et al . ( 2017 ) . Noise injection and regularizing with a gradient penalty appear to be equivalent . The addition of instance noise in f -GAN can be approximated to adding a zero centered gradient penalty ( Roth et al. , 2017 ) . Thus , regularizing GAN with a simple gradient penalty term was suggested by Mescheder et al . ( 2018 ) who provided a proof of its stability . Based on a theoretical analysis of the dynamic system , Nagarajan & Kolter ( 2017 ) proved the local exponential stability of the gradient-based optimization dynamics in GANs by treating the simultaneous gradient descent algorithm with a dynamic system approach . These previous studies were useful because they showed that the local behavior of GANs can be explained using dynamic system tools and the related Jacobian ’ s eigenvalues . In this study , we aim to prove the convergence property of the simple gradient penalty µ-Wasserstein GAN ( SGP µ-WGAN ) dynamic system under general gradient penalty measures µ . To the best of our knowledge , our study is the first theoretical approach to GAN stability analysis which deals with abstract singular penalty measure . In addition , measure valued differentiation ( Heidergott & Vázquez-Abad , 2008 ) is applied to take the derivative on the integral with a parametric measure , which is helpful for handling an abstract measure and its integral in our proof . The main contributions of this study are as follows . • We prove the regularized effect and local stability of the dynamic system for a general penalty measure under suitable assumptions . The assumptions are written as both a tractable strong version and intractable weak version . To prove the main theorem , we also introduce the measure valued differentiation concept to handle the parametric measure . • Based on the proof of the stability , we explain the reason for the success of previous penalty measures . We claim that the support of a penalty measure will be strongly related to the stability , where the weight on the limiting penalty measure might affect the speed of convergence . • We experimentally examined the general convergence results by applying two test penalty measures to several examples . The proposed test measures are unintuitive but they still satisfy the assumptions and similar convergence results were obtained in the experiment . 2 PRELIMINARIES . First , we introduce our notations and basic measure-theoretic concepts . Second , we define our SGP µ-WGAN optimization problem and treat this problem as a continuous dynamic system . Preliminary measure theoretic concepts are required to justify that the dynamic system changes in a sufficiently smooth manner as the parameter changes , so it is possible to use linearization theorem . They are also important for dealing with the parametric measure and its derivative . The problem setting with a simple gradient term is also discussed . The squared gradient size and simple gradient penalty term are used to build a differentiable dynamic system and to apply soft regularization as a resolving constraint , respectively . The continuous dynamic system approach , which is a so-called ODE method , is used to analyze the GAN optimization problem with the simultaneous gradient descent algorithm , as described by Nagarajan & Kolter ( 2017 ) . 2.1 NOTATIONS AND PRELIMINARIES REGARDING MEASURE THEORY . D ( x ; ψ ) : X → R is a discriminator function with its parameter ψ and G ( z ; θ ) : Z → X is a generator function with its parameter θ. pd is the distribution of real data and pg = pθ is the distribution of the generated samples in X , which is induced from the generator function G ( z ; θ ) and a known initial distribution platent ( z ) in the latent space Z . ‖·‖ denotes the L2 Euclidean norm if no special subscript is present . The concept of weak convergence for finite measures is used to ensure the continuity of the integral term over the measure in the dynamic system , which must be checked before applying the theorems related to stability . Throughout this study , we assume that the measures in the sample space are all finite and bounded . Definition 1 . For a set of finite measures { µi } i∈I in ( Rn , d ) with euclidean distance d , { µi } i∈I is referred to as bounded if there exists some M > 0 such that for all i ∈ I , µi ( Rn ) ≤M For instance , M can be set as 1 if { µi } are probability measures on Rn . Assuming that the penalty measures are bounded , Portmanteau theorem offers the equivalent definition of the weak conver- gence for finite measures . This definition is important for ensuring that the integrals over pθ and µ in the dynamic system change continuously . Definition 2 . ( Portmanteau Theorem ) For a bounded sequence of finite measures { µn } n∈N on the Euclidean space Rn with a σ-field of Borel subsets B ( Rn ) , µn converges weakly to µ if and only if for every continuous bounded function φ on Rn , its integrals with respect to µn converge to ∫ φdµ , i.e. , µn → µ⇐⇒ ∫ φdµn → ∫ φdµ The most challenging problem in our analysis with the general penalty measure is taking the derivative of the integral , where the measure depends on the variable that we want to differentiate . If our penalty measure is either absolutely continuous or discrete , then it is easy to deal with the integral . However , in the case of singular penalty measure , dealing with the integral term is not an easy task . Therefore , we introduce the concept of a weak derivative of a probability measure in the following ( Heidergott & Vázquez-Abad , 2008 ) . The weak derivative of a measure is useful for handling a parametric measure that is not absolutely continuous with low dimensional support . Definition 3 . ( Weak Derivatives of a Probability Measure ) Consider the Euclidean space and its σ-field of Borel subsets ( Rd , B ( Rd ) ) . The probability measure Pθ is called weakly differentiable at θ if a signed finite measure P ′θ exists where d dθ ∫ φ ( x ) dPθ = lim ∆→0 1 ∆ { ∫ φ ( x ) dPθ+∆ − ∫ φ ( x ) dPθ } = ∫ φ ( x ) dP ′θ is satisfied for every continuous bounded function φ on Rn . For the multidimensional parameter θ , this can be defined similar manner . We can show that the positive part and negative part of P ′θ have the same mass by putting φ ( x ) = 1 and the Hahn–Jordan decomposition on P ′θ . Therefore , the following triple ( cθ , P + θ , P − θ ) is called a weak derivative of Pθ , where P±θ are probability measures and P ′ θ is rewritten as : P ′θ = cθP + θ − cθP − θ Therefore , d dθ ∫ φ ( x ) dPθ = ∫ φ ( x ) dP ′θ = cθ ( ∫ φ ( x ) dP+θ − ∫ φ ( x ) dP−θ ) holds for every continuous bounded function φ on Rn . It is known that the representation of ( cθ , P + θ , P − θ ) for P ′ θ is not unique because ( cθ + Cθ , P + θ + qθ , P − θ + qθ ) is also another representation of P ′θ . For the general finite measure Qθ , a normalizing coefficient M ( θ ) < ∞ can be introduced . The product rule for differentiating can also be applied in a similar manner to calculus . d dθ ∫ φ ( x ; θ ) dPθ = ∫ ∇θφ ( x ; θ ) dPθ + ∫ φ ( x ; θ ) dP ′θ Therefore , for the general finite measure Qθ = M ( θ ) Pθ , its derivative Q′θ can be represented as below . Q′θ = M ′ ( θ ) Pθ +M ( θ ) P ′ θ = M ′ ( θ ) Pθ + cθM ( θ ) P + θ − cθM ( θ ) P − θ 2.2 PROBLEM SETTING AS A DYNAMIC SYSTEM . Previous work of Mescheder et al . ( 2018 ) showed that the dynamic system of WGAN-GP is not necessarily stable at equilibrium by demonstrating that the sequence of parameters is not Cauchy sequence . This is mainly due to the term ‖x‖ in the dynamic system which has a derivative x‖x‖ that is not defined at x = 0 . WGAN-GP has a penalty term EµGP [ ( ‖∇xD ( x ; ψ ) ‖ − 1 ) 2 ] that can lead to a discontinuity in its dynamic system . These problems can be avoided by using the squared value of the gradient ’ s norm ‖∇xD‖2 , which is a differentiable function . In contrast to the WGAN-GP , recent methods based on a gradient penalty such as the simple gradient penalty employed by Mescheder et al . ( 2018 ) and the Sobolev GAN used the average of the squared values for the penalty area , whereas the WGAN-GP penalizes the size of the discriminator ’ s gradient ‖∇xD‖ away from 1 in a pointwise manner . This advantage of squared gradient term1 , Eµ [ ‖∇xD‖2 ] , makes the dynamic system differentiable and we define the WGAN problem with the square of the gradient ’ s norm as a simple gradient penalty . This simple gradient penalty can be treated as soft regularization based on the size of the discriminator ’ s gradient , especially in case where µ is the probability measure ( Roth et al. , 2017 ) . It is convenient to determine whether the system is stable by observing the spectrum of the Jacobian matrix . In the following , ( D ( x ; ψ ) , pd , pθ , µ ) is defined as an SGP µ-WGAN optimization problem ( SGP-form ) with a simple gradient penalty term on the penalty measure µ . Definition 4 . The WGAN optimization problem with a simple gradient penalty term ‖∇xD‖2 , penalty measure µ , and penalty weight hyperparameter ρ > 0 is given as follows , where the penalty term is only introduced to update the discriminator . max ψ : Epd [ D ( x ; ψ ) ] − Epθ [ D ( x ; ψ ) ] − ρ 2 Eµ [ ‖∇xD ( x ; ψ ) ‖2 ] min θ : Epd [ D ( x ; ψ ) ] − Epθ [ D ( x ; ψ ) ] According to Nagarajan & Kolter ( 2017 ) and many other optimization problem studies , the simultaneous gradient descent algorithm for GAN updating can be viewed as an autonomous dynamic system of discriminator parameters and generator parameters , which we denote as ψ and θ . As a result , the related dynamic system is given as follows . ψ̇ = Epd [ ∇ψD ] − Epθ [ ∇ψD ] − ρ 2 ∇ψEµ [ ∇TxD∇xD ] θ̇ = ∇θEpθ [ D ]

Summary: In the paper, WGAN with a squared zero centered gradient penalty term w.r.t. to a general measure is studied. Under strong assumptions, local stability of a time-continuous gradient ascent/descent dynamical system near an equilibrium point are proven for the new GP term. Experiments show comparable results to the original WGAN-GP formulation w.r.t. FID and inception score.