Smoothness and Stability in GANs

1 INTRODUCTION : TAMING INSTABILITY WITH SMOOTHNESS . Generative adversarial networks ( Goodfellow et al. , 2014 ) , or GANs , are a powerful class of generative models defined through minimax game . GANs and their variants have shown impressive performance in synthesizing various types of datasets , especially natural images . Despite these successes , the training of GANs remains quite unstable in nature , and this instability remains difficult to understand theoretically . Since the introduction of GANs , there have been many techniques proposed to stabilize GANs training , including studies of new generator/discriminator architectures , loss functions , and regularization techniques . Notably , Arjovsky et al . ( 2017 ) proposed Wasserstein GAN ( WGAN ) , which in principle avoids instability caused by mismatched generator and data distribution supports . In practice , this is enforced by Lipschitz constraints , which in turn motivated developments like gradient penalties ( Gulrajani et al. , 2017 ) and spectral normalization ( Miyato et al. , 2018 ) . Indeed , these stabilization techniques have proven essential to achieving the latest state-of-the-art results ( Karras et al. , 2018 ; Brock et al. , 2019 ) . On the other hand , a solid theoretical understanding of training stability has not been established . Several empirical observations point to an incomplete understanding . For example , why does applying a gradient penalty together spectral norm seem to improve performance ( Miyato et al. , 2018 ) , even though in principle they serve the same purpose ? Why does applying only spectral normalization with the Wasserstein loss fail ( Miyato , 2018 ) , even though the analysis of Arjovsky et al . ( 2017 ) suggests it should be sufficient ? Why is applying gradient penalties effective , even outside their original context of the Wasserstein GAN ( Fedus et al. , 2018 ) ? In this work , we develop a framework to analyze the stability of GAN training that resolves these apparent contradictions and clarifies the roles of these regularization techniques . Our approach considers the smoothness of the loss function used . In optimization , smoothness is a well-known condition that ensures that gradient descent and its variants become stable ( see e.g. , Bertsekas ( 1999 ) ) . For example , the following well-known proposition is the starting point of our stability analysis : Proposition 1 ( Bertsekas ( 1999 ) , Proposition 1.2.3 ) . Suppose f : Rp → R is L-smooth and bounded below . Let xk+1 : = xk − 1L∇f ( xk ) . Then ||∇f ( xk ) || → 0 as k →∞ . This proposition says that under a smoothness condition on the function , gradient descent with a constant step size 1L approaches stationarity ( i.e. , the gradient norm approaches zero ) . This is a rather weak notion of convergence , as it does not guarantee that the iterates converge to a point , and even if the iterates do converge , the limit is a stationary point and not necessarily an minimizer . Nevertheless , empirically , not even this stationarity is satisfied by GANs , which are known to frequently destabilize and diverge during training . To diagnose this instability , we consider the smoothness of the GAN ’ s loss function . GANs are typically framed as minimax problems of the form inf θ sup ϕ J ( µθ , ϕ ) , ( 1 ) where J is a loss function that takes a generator distribution µθ and discriminator ϕ , and θ ∈ Rp denotes the parameters of the generator . Unfortunately , the minimax nature of this problem makes stability and convergence difficult to analyze . To make the analysis more tractable , we define J ( µ ) = supϕ J ( µ , ϕ ) , so that ( 1 ) becomes simply inf θ J ( µθ ) . ( 2 ) This choice corresponds to the common assumption that the discriminator is allowed to reach optimality at every training step . Now , the GAN algorithm can be regarded as simply gradient descent on the Rp → R function θ 7→ J ( µθ ) , which may be analyzed using Proposition 1 . In particular , if this function θ 7→ J ( µθ ) satisfies the smoothness assumption , then the GAN training should be stable in that it should approach stationarity under the assumption of an optimal discriminator . In the remainder of this paper , we investigate whether the smoothness assumption is satisfied for various GAN losses . Our analysis answers two questions : Q1 . Which existing GAN losses , if any , satisfy the smoothness condition in Proposition 1 ? Q2 . Are there choices of loss , regularization , or architecture that enforce smoothness in GANs ? As results of our analysis , our contributions are as follows : 1 . We derive sufficient conditions for the GAN algorithm to be stationary under certain assumptions ( Theorem 1 ) . Our conditions relate to the smoothness of GAN loss used as well as the parameterization of the generator . 2 . We show that most common GAN losses do not satisfy the all of the smoothness conditions , thereby corroborating their empirical instability . 3 . We develop regularization techniques that enforce the smoothness conditions . These regularizers recover common GAN stabilization techniques such as gradient penalties and spectral normalization , thereby placing their use on a firmer theoretical foundation . 4 . Our analysis provides several practical insights , suggesting for example the use of smooth activation functions , simultaneous spectral normalization and gradient penalties , and a particular learning rate for the generator . 1.1 RELATED WORK . Divergence minimization Our analysis regards the GAN algorithm as minimizing a divergence between the current generator distribution and the desired data distribution , under the assumption of an optimal discriminator at every training step . This perspective originates from the earliest GAN paper , in which Goodfellow et al . ( 2014 ) show that the original minimax GAN implicitly minimizes the Jensen–Shannon divergence . Since then , the community has introduced a large number of GAN or GAN-like variants that learn generative models by implicitly minimizing various divergences , including f -divergences ( Nowozin et al. , 2016 ) , Wasserstein distance ( Arjovsky et al. , 2017 ) , and maximum-mean discrepancy ( Li et al. , 2015 ; Unterthiner et al. , 2018 ) . Meanwhile , the non-saturating GAN ( Goodfellow et al. , 2014 ) has been shown to minimize a certain Kullback– Leibler divergence ( Arjovsky & Bottou , 2017 ) . Several more theoretical works consider the topological , geometric , and convexity properties of divergence minimization ( Arjovsky & Bottou , 2017 ; Liu et al. , 2017 ; Bottou et al. , 2018 ; Farnia & Tse , 2018 ; Chu et al. , 2019 ) , perspectives that we draw heavily upon . Sanjabi et al . ( 2018 ) also prove smoothness of GAN losses in the specific case of the regularized optimal transport loss . Their assumption for smoothness is entangled in that it involves a composite condition on generators and discriminators , while our analysis addresses them separately . Other approaches Even though many analyses , including ours , operate under the assumption of an optimal discriminator , this assumption is unrealistic in practice . Li et al . ( 2017b ) contrast this optimal discriminator dynamics with first-order dynamics , which assumes that the generator and discriminator use alternating gradient updates and is what is used computationally . As this is a differing approach from ours , we only briefly mention some results in this area , which typically rely on game-theoretic notions ( Kodali et al. , 2017 ; Grnarova et al. , 2018 ; Oliehoek et al. , 2018 ) or local analysis ( Nagarajan & Kolter , 2017 ; Mescheder et al. , 2018 ) . Some of these results rely on continuous dynamics approximations of gradient updates ; in contrast , our work focuses on discrete dynamics . 1.2 NOTATION . Let R̄ : = R ∪ { ∞ , −∞ } . We let P ( X ) denote the set of all probability measures on a compact set X ⊆ Rd . We letM ( X ) and C ( X ) denote the dual pair consisting of the set of all finite signed measures onX and the set of all continuous functionsX → R. For any statementA , we let χ { A } be 0 if A is true and∞ if A is false . For a Euclidean vector x , its Euclidean norm is denoted by ‖x‖2 , and the operator norm of a matrix A is denoted by ‖A‖2 , i.e. , ‖A‖2 = sup‖x‖2≤1 ‖Ax‖2/‖x‖2 . A function f : X → Y between two metric spaces is α-Lipschitz if dY ( f ( x1 ) , f ( x2 ) ) ≤ αdX ( x1 , x2 ) . A function f : Rd → R is β-smooth if its gradients are β-Lipschitz , that is , for all x , y ∈ Rd , ‖∇f ( x ) −∇f ( y ) ‖2 ≤ β‖x− y‖2 . 2 SMOOTHNESS OF GAN LOSSES . This section presents Theorem 1 , which provides concise criteria for the smoothness of GAN losses . In order to keep our analysis agnostic to the particular GAN used , let J : P ( X ) → R̄ be an arbitrary convex loss function , which takes a distribution over X ⊂ Rd and outputs a real number . Note that the typical minimax formulation of GANs can be recovered from just the loss function J using convex duality . In particular , recall that the convex conjugate J ? : C ( X ) → R̄ of J satisfies the following remarkable duality , known as the Fenchel–Moreau theorem : J ? ( ϕ ) : = sup µ∈M ( X ) ∫ ϕ ( x ) dµ− J ( µ ) , J ( µ ) = sup ϕ∈C ( X ) ∫ ϕ ( x ) dµ− J ? ( ϕ ) . ( 3 ) Based on this duality , minimizing J can be framed as the minimax problem inf µ∈P ( X ) J ( µ ) = inf µ∈P ( X ) sup ϕ∈C ( X ) ∫ ϕ ( x ) dµ− J ? ( ϕ ) : = inf µ∈P ( X ) sup ϕ∈C ( X ) J ( µ , ϕ ) , ( 4 ) recovering the well-known adversarial formulation of GANs . We now define the notion of an optimal discriminator for an arbitrary loss function J , based on this convex duality : Definition 1 ( Optimal discriminator ) . Let J : M ( X ) → R̄ be a convex , l.s.c. , proper function . An optimal discriminator for a probability distribution µ ∈ P ( X ) is a continuous function Φµ : X → R that attains the maximum of the second equation in ( 3 ) , i.e. , J ( µ ) = ∫ Φµ ( x ) dµ− J ? ( Φµ ) . This definition recovers the optimal discriminators of many existing GAN and GAN-like algorithms ( Farnia & Tse , 2018 ; Chu et al. , 2019 ) , most notably those in Table 1 . Our analysis will apply to any algorithm in this family of algorithms . See Appendix B for more details on this perspective . We also formalize the notion of a family of generators : Definition 2 ( Family of generators ) . A family of generators is a set of pushforward probability measures { µθ = fθ # ω : θ ∈ Rp } , where ω is a fixed probability distribution on Z ( the latent variable ) and fθ : Z → X is a measurable function ( the generator ) . Now , in light of Proposition 1 , we are interested in the smoothness of the mapping θ 7→ J ( µθ ) , which would guarantee the stationarity of gradient descent on this objective , which in turn implies stationarity of the GAN algorithm under the assumption of an optimal discriminator . The following theorem is our central result , which decomposes the smoothness of θ 7→ J ( µθ ) into conditions on optimal discriminators and the family of generators . Theorem 1 ( Smoothness decomposition for GANs ) . Let J : M ( X ) → R̄ be a convex function whose optimal discriminators Φµ : X → R satisfy the following regularity conditions : ( D1 ) x 7→ Φµ ( x ) is α-Lipschitz , ( D2 ) x 7→ ∇xΦµ ( x ) is β1-Lipschitz , ( D3 ) µ 7→ ∇xΦµ ( x ) is β2-Lipschitz w.r.t . the 1-Wasserstein distance . Also , let µθ = fθ # ω be a family of generators that satisfies : ( G1 ) θ 7→ fθ ( z ) is A-Lipschitz in expectation for z ∼ ω , i.e. , Ez∼ω [ ‖fθ1 ( z ) − fθ2 ( z ) ‖2 ] ≤ A‖θ1 − θ2‖2 , and ( G2 ) θ 7→ Dθfθ ( z ) is B-Lipschitz in expectation for z ∼ ω , i.e. , Ez∼ω [ ‖Dθ1fθ1 ( z ) − Dθ2fθ2 ( z ) ‖2 ] ≤ B‖θ1 − θ2‖2 . Then θ 7→ J ( µθ ) is L-smooth , with L = αB +A2 ( β1 + β2 ) . Theorem 1 connects the smoothness properties of the loss function J with the smoothness properties of the optimal discriminator Φµ , and once paired with Proposition 1 , it suggests a quantitative value 1 L for a stable generator learning rate . In order to obtain claims of stability for practically sized learning rates , it is important to tightly bound the relevant constants . In Sections 4 to 6 , we carefully analyze which GAN losses satisfy ( D1 ) , ( D2 ) , and ( D3 ) , and with what constants . We summarize our results in Table 2 : it turns out that none of the listed losses , except for one , satisfy ( D1 ) , ( D2 ) , and ( D3 ) simultaneously with a finite constant . The MMD-based loss satisfies the three conditions , but its constant for ( D1 ) grows as α = O ( √ d ) , which is an unfavorable dependence on the data dimension d that forces an unacceptably small learning rate . See for complete details of each condition . This failure of existing GANs to satisfy the stationarity conditions corroborates the observed instability of GANs . Theorem 1 decomposes smoothness into conditions on the generator and conditions on the discriminator , allowing a clean separation of concerns . In this paper , we focus on the discriminator conditions ( D1 ) , ( D2 ) , and ( D3 ) and only provide an extremely simple example of a generator that satisfies ( G1 ) and ( G2 ) , in Section 7 . Because analysis of the generator conditions may become quite complicated and will vary with the choice of architecture considered ( feedforward , convolutional , ResNet , etc . ) , we leave a detailed analysis of the generator conditions ( G1 ) and ( G2 ) as a promising avenue for future work . Indeed , such analyses may lead to new generator architectures or generator regularization techniques that stabilize GAN training .

Summary: This paper provides a unified theoretical framework for regularizing GAN losses. It accounts for most regularization technics especially spectral normalization and gradient penalty and explains how those two methods are in fact complementary. So far this was only observed experimentally but without any theoretical insight. The result goes beyond that as the criterion could be applied to general convex cost functional.