Wasserstein-Bounded Generative Adversarial Networks

1 INTRODUCTION . Over the past few years , Generative Adversarial Networks ( GANs ) have shown impressive results in many generative tasks . They are inspired by the game theory , that two models compete with each other : a generator which seeks to produce samples from the same distribution as the data , and a discriminator whose job is to distinguish between real and generated data . Both models are forced stronger simultaneously during the training process . GANs are capable of producing plausible synthetic data across a wide diversity of data modalities , including natural images ( Karras et al. , 2017 ; Brock et al. , 2018 ; Lucic et al. , 2019 ) , natural language ( Press et al. , 2017 ; Lin et al. , 2017 ; Rajeswar et al. , 2017 ) , music ( Yang et al. , 2017 ; Mogren , 2016 ; Dong et al. , 2017 ; Dong & Yang , 2018 ) , etc . Despite their success , it is often difficult to train a GAN model in a fast and stable way , and researchers are facing issues like vanishing gradients , training instability , mode collapse , etc . This has led to a proliferation of works that focus on improving the quality of GANs by stabilizing the training procedure ( Radford et al. , 2015 ; Salimans et al. , 2016 ; Zhao et al. , 2016 ; Nowozin et al. , 2016 ; Chen et al. , 2016 ; Qi , 2017 ; Deshpande et al. , 2018 ) . In particular , Arjovsky et al . ( 2017 ) introduced a variant of GANs based on the Wasserstein distance , and releases the problem of gradient disappearance to some extent . However , WGANs limit the weight within a range to enforce the continuity of Lipschitz , which can easily cause over-simplified critic functions ( Gulrajani et al. , 2017 ) . To solve this issue , Gulrajani et al . ( 2017 ) proposed a gradient penalty method termed WGAN-GP , which replaces the weight clipping in WGANs with a gradient penalty term . As such , WGAN-GP provides a more stable training procedure and succeeds in a variety of generating tasks . Based on WGAN-GP , more works ( Wei et al. , 2018 ; Petzka et al. , 2017 ; Wu et al. , 2018 ; Mescheder et al. , 2018 ; Thanh-Tung et al. , 2019 ; Kodali et al. , 2017 ; Kim et al. , 2018 ) adopt different forms of gradient penalty terms to further improve training stability . However , it is often observed that such gradient penalty strategy sometimes generate samples with unsatisfying quality , or even do not always converge to the equilibrium point ( Mescheder et al. , 2018 ) . In this paper , we propose a general framework named Wasserstein-Bounded GAN ( WBGAN ) , which improve the stability of WGAN training by bounding the Wasserstein term . The highlight is that the instability of WGANs also resides in the dramatic changes of the estimated Wasserstein distance during the initial iterations . Many previous works just focused on improving the gradient penalty term for stable training , while they ignored the bottleneck of the Wasserstein term . The proposed training strategy is able to adaptively enforce the Wasserstein term within a certain value , so as to balance the Wasserstein loss and gradient penalty loss dynamically and make the training process more stable . WBGANs are generalized , which can be instantiated using different kinds of bound estimations , and incorporated into any variant of WGANs to improve the training stability and accelerate the convergence . Specifically , with Sinkhorn distance ( Cuturi , 2013 ; Genevay et al. , 2017 ) for bound estimation , we test three representative variants of WGANs ( WGAN-GP ( Gulrajani et al. , 2017 ) , WGANdiv ( Wu et al. , 2018 ) , and WGAN-GPReal ( Mescheder et al. , 2018 ) ) on the CelebA dataset ( Liu et al. , 2015 ) . As shown in Fig . 1 , WBGANs outperform the corresponding counterparts , which demonstrates that the bounded strategy results in more stable training and accelerates the convergence . 2 BACKGROUNDS . Wasserstein GANs ( WGANs ) . WGANs ( Arjovsky et al. , 2017 ) were primarily motivated by unstable training caused by the gradient vanishing problem of the original GANs ( Goodfellow et al. , 2014 ) . They proposed to use 1-Wasserstein distance W1 ( Pr , Pg ) to measure the difference between Pr and Pg , the real and generated distributions , given that W1 ( Pr , Pg ) is continuous everywhere and differentiable almost everywhere under mild assumptions . The objective of WGAN is formulated using the KantorovichRubinstein duality ( Villani , 2008 ) : min G max D∈L1 Ex∼Pr [ D ( x ) ] − Ex̃∼Pg [ D ( x̃ ) ] , ( 1 ) where L1 is the function space of all D satisfying the 1-Lipschitz constraint ‖D‖L ≤ 1 . D is a critic and G is the generator , both of which are parameterized by a neural network . Under an optimal critic , minimizing the objective with respect to G is to minimize W1 ( Pr , Pg ) . To enforce the 1-Lipschitz constraint on the critic , WGAN used a weight clipping on the critic to constrain the weights within a compact range , [ −c , c ] , which guarantees the set of critic functions is a subset of the k-Lipschitz functions for some k. With weight clipping , the critic tends to learn over-simplified functions ( Gulrajani et al. , 2017 ) , which may lead to unsatisfying results . Gulrajani et al . ( 2017 ) ; Wei et al . ( 2018 ) ; Petzka et al . ( 2017 ) ; Wu et al . ( 2018 ) proposed different forms of gradient penalty as a regularization term , so that a generalized loss function with respect to the critic can be written as : LD = − [ Ex∼Pr [ D ( x ) ] − Ex̃∼Pg [ D ( x̃ ) ] ] + λGP ·GP , ( 2 ) where Ex∼Pr [ D ( x ) ] −Ex̃∼Pg [ D ( x̃ ) ] stands for the Wasserstein term , and GP for the gradient penalty term . LD is actually posing a tradeoff between these two objectives . Wasserstein Distance between Empirical Distributions . In practice , we approximate W1 ( Pr , Pg ) using W1 ( P̂r , P̂g ) , where P̂r and P̂g denote the empirical version of Pr and Pg with N samples , i.e. , P̂r = 1N ∑N i=1 δyi , and P̂g = 1 N ∑N i=1 δG ( zi ) . Here , yi is randomly sampled from the real image dataset , and δyi is the Dirac delta function at location yi . Computing W1 ( P̂r , P̂g ) is a typical problem named discrete optimal transport . We denote B as the set of probabilistic couplings between two empirical distributions defined as : B : = { Γ ∈ RN×N+ | Γ1N = P̂g , Γ > 1N = P̂r } , ( 3 ) where 1N is a N -dimensional all-one vector . Then we have W1 ( P̂r , P̂g ) = minγ∈B〈Γ , C〉F , where 〈· , ·〉F is the Frobenius dot-product and C is the cost matrix , with each element Ci , j = c ( G ( zi ) , yj ) denoting the cost to move a probability mass from G ( zi ) to yj . The optimal coupling is the solution of this minimization problem : Γ0 = arg minΓ∈B〈Γ , C〉F . The Sinkhorn Algorithm . Despite Wasserstein distance has appealing theoretical properties in measuring the difference between distributions , its computational costs for linear programming are often high in particular when the problem size becomes large . To alleviate this burden , Sinkhorn distance ( Cuturi , 2013 ) was proposed to approximate Wasserstein distance : dα ( P̂r , P̂g ) : = min P∈Uα ( P̂r , P̂g ) 〈P , C〉 , ( 4 ) where Uα ( P̂r , P̂g ) is a subset of B defined in Eq . 3 : Uα ( P̂r , P̂g ) : = { Γ ∈ B|H ( Γ ) > H ( P̂r ) + H ( P̂g ) − α } ⊂ B , ( 5 ) where H ( · ) is the entropy defined as H ( Γ ) = − ∑N i , j=1 Γi , j log Γi , j and H ( P̂r ) = − ∑N n=1 p̂n log p̂n where p̂n is the probability of the n-th sample . Compared to Wasserstein distance , Sinkhorn distance restricts the search space of joint probabilities to those with sufficient smoothness . To compute Sinkhorn distance , a Lagrange multiplier was used : dλ ( P̂r , P̂g ) = 〈 Γλ , C 〉 , Γλ = arg min Γ∈B 〈Γ , C〉 − 1 λ H ( Γ ) . ( 6 ) Each α corresponds a λ ∈ [ 0 , ∞ ) such that dα ( P̂r , P̂g ) = dλ ( P̂r , P̂g ) holds for that pair ( P̂r , P̂g ) . dλ ( · , · ) can be computed with a much cheaper cost than the original Wasserstein distance using matrix scaling algorithms . For λ > 0 , the solution Γλ is unique and has the form Γλ = diag ( u ) K diag ( v ) , where K is the element-wise exponential of −λC . u and v are two non-negative vectors uniquely defined up to a multiplicative factor ( Cuturi , 2013 ) . 3 WASSERSTEIN-BOUNDED GANS . 3.1 BOUND CONSTRAINT ON 1-WASSERSTEIN DISTANCE . We start with Eq . 2 and denote W = Ex∼Pr [ D ( x ) ] − Ex̃∼Pg [ D ( x̃ ) ] . W is often referred to as the Wasserstein term , which is unbounded during the training process . In a wide range of WGAN ’ s variants such as WGAN-GP ( Gulrajani et al. , 2017 ) , the critic defined by LD is to maximize the Wasserstein term W while satisfying the gradient penalty GP . However , in practice , we find that W often rises rapidly to a tremendous value which is far from rational during the initial training procedure . A possible reason may lie in that the critic function does not satisfy the Lipschitz constraint during the initial training stage . As shown in Fig . 2 , this leads to dramatic instability in optimization and finally results in unsatisfying performance in image generation . Our idea is thus straightforward , i.e. , setting an upper-bound for W . The modified critic loss function is written as : LD = −W + [ W −ĎW ] + + λGP ·GP , ( 7 ) where [ · ] + = max { · , 0 } is the ramp function that ignores negative inputs , and ĎW denotes the upper bound of W , and will be discussed later . If W 6 ĎW , the term [ W −ĎW ] + simply vanishes and Eq . 7 is equivalent to Eq . 2 ; otherwise , we have−W + [ W −ĎW ] + = −ĎW , which implies that the modified Wasserstein term is bounded by ĎW . Our formulation brings a benefit to the numerical stability of the Wasserstein term . In practice , it remains comparable to the other term , λGP ·GP , so that both W and GP can be optimized in a ‘ mild ’ manner , i.e. , without any one of them dominating or being ignored during training . Note that the ĎW term can not be chosen arbitrarily . Setting it too small , ĎW will limit the capacity of the critic function , resulting in a poor generation . Setting it too large , there will be no effect of bounding the W term . The proposed bounded strategy is a general framework . We name it general in two folds : First , WBGAN can be applied to almost all gradient penalty based WGANs , such as WGAN-GP ( Gulrajani et al. , 2017 ) , WGAN-GPReal ( Mescheder et al. , 2018 ) , etc . Moreover , there are different ways to estimate the value of ĎW . For example , the linear programming was applied successfully to some existing WGANs like WGAN-TS ( Liu et al. , 2018 ) . In what follows , we present an example which uses Sinkhorn distance to estimate ĎW , while we believe other ways of estimation are also possible .

Summary: The paper proposes a new way of stabilizing Wasserstein GANs by using Sinkhorn distance to upper-bound the objective of WGAN's critic's loss during the training. GAN stabilization is a well-motivated problem and limiting the dramatic changes of discriminator loss clearly helps achieving this goal. Experiments show that in few settings the proposed method successfully addresses this issue. However, the theoretical insights have multiple flaws and incorrect proofs, while the choice of experiments raises few questions. Finally, the resolution limitations suggest the overall contribution is rather incremental.