SGD Learns One-Layer Networks in WGANs

1 INTRODUCTION . Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) are a prominent framework for learning generative models of complex , real-world distributions given samples from these distributions . GANs and their variants have been successfully applied to numerous datasets and tasks , including image-to-image translation ( Isola et al. , 2017 ) , image super-resolution ( Ledig et al. , 2017 ) , domain adaptation ( Tzeng et al. , 2017 ) , probabilistic inference ( Dumoulin et al. , 2016 ) , compressed sensing ( Bora et al. , 2017 ) and many more . These advances owe in part to the success of Wasserstein GANs ( WGANs ) ( Arjovsky et al. , 2017 ; Gulrajani et al. , 2017 ) , leveraging the neural net induced integral probability metric to better measure the difference between a target and a generated distribution . Along with the afore-described empirical successes , there have been theoretical studies of the statistical properties of GANs—see e.g . ( Zhang et al. , 2018 ; Arora et al. , 2017 ; 2018 ; Bai et al. , 2018 ; Dumoulin et al. , 2016 ) and their references . These works have shown that , with an appropriate design of the generator and discriminator , the global optimum of the WGAN objective identifies the target distribution with low sample complexity . On the algorithmic front , prior work has focused on the stability and convergence properties of gradient descent-ascent ( GDA ) and its variants in GAN training and more general min-max optimization problems ; see e.g . ( Nagarajan & Kolter , 2017 ; Heusel et al. , 2017 ; Mescheder et al. , 2017 ; 2018 ; Daskalakis et al. , 2017 ; Daskalakis & Panageas , 2018a ; b ; Gidel et al. , 2019 ; Liang & Stokes , 2019 ; Mokhtari et al. , 2019 ; Jin et al. , 2019 ; Lin et al. , 2019 ) and their references . It is known that , even in min-max optimization problems with convex-concave objectives , GDA may fail to compute the min-max solution and may even exhibit divergent behavior . Hence , these works have studied conditions under which GDA converges to a globally optimal solution under a convex-concave objective , or different types of locally optimal solutions under nonconvex-concave or nonconvex-nonconcave objectives . They have also identified variants of GDA with better stability properties in both theory and practice , most notably those using negative momentum . In the context of GAN training , Feizi et al . ( 2017 ) show that for WGANs with a linear generator and quadratic discriminator GDA succeeds in learning a Gaussian using polynomially many samples in the dimension . In the same vein , we are the first to our knowledge to study the global convergence properties of stochastic GDA in the GAN setting , and establishing such guarantees for non-linear generators . In particular , we study the WGAN formulation for learning a single-layer generative model with some reasonable choices of activations including tanh , sigmoid and leaky ReLU . Our contributions . For WGAN with a one-layer generator network using an activation from a large family of functions and a quadratic discriminator , we show that stochastic gradient descent-ascent learns a target distribution using polynomial time and samples , under the assumption that the target distribution is realizable in the architecture of the generator . This is achieved by a ) analysis of the dynamics of stochastic gradient-descent to show it attains a global optimum of the minmax problem , and b ) appropriate design of the discriminator to ensure a parametric O ( 1√ n ) statistical rate ( Zhang et al. , 2018 ; Bai et al. , 2018 ) . Related Work . We briefly review relevant results in GAN training and learning generative models : - Optimization viewpoint . For standard GANs and WGANs with appropriate regularization , Nagarajan & Kolter ( 2017 ) , Mescheder et al . ( 2017 ) and Heusel et al . ( 2017 ) establish sufficient conditions to achieve local convergence and stability properties for GAN training . At the equilibrium point , if the Jacobian of the associated gradient vector field has only eigenvalues with negative real-part at the equilibrium point , GAN training is verified to converge locally for small enough learning rates . A follow-up paper by ( Mescheder et al. , 2018 ) shows the necessity of these conditions by identifying a prototypical counterexample that is not always locally convergent with gradient descent based GAN optimization . However , the lack of global convergence prevents the analysis to provide any guarantees of learning the real distribution . The work of ( Feizi et al. , 2017 ) described above has similar goals as our paper , namely understanding the convergence properties of basic dynamics in simple WGAN formulations . However , they only consider linear generators , which restrict the WGAN model to learning a Gaussian . Our work goes a step further , considering WGANs whose generators are one-layer neural networks with a broad selection of activations . We show that with a proper gradient-based algorithm , we can still recover the ground truth parameters of the underlying distribution . More broadly , WGANs typically result in nonconvex-nonconcave min-max optimization problems . In these problems , a global min-max solution may not exist , and there are various notions of local min-max solutions , namely local min-local max solutions Daskalakis & Panageas ( 2018b ) , and local min solutions of the max objective Jin et al . ( 2019 ) , the latter being guaranteed to exist under mild conditions . In fact , Lin et al . ( 2019 ) show that GDA is able to find stationary points of the max objective in nonconvex-concave objectives . Given that GDA may not even converge for convexconcave objectives , another line of work has studied variants of GDA that exhibit global convergence to the min-max solution Daskalakis et al . ( 2017 ) ; Daskalakis & Panageas ( 2018a ) ; Gidel et al . ( 2019 ) ; Liang & Stokes ( 2019 ) ; Mokhtari et al . ( 2019 ) , which is established for GDA variants that add negative momentum to the dynamics . While the convergence of GDA with negative momentum is shown in convex-concave settings , there is experimental evidence supporting that it improves GAN training ( Daskalakis et al. , 2017 ; Gidel et al. , 2019 ) . - Statistical viewpoint . Several works have studied the issue of mode collapse . One might doubt the ability of GANs to actually learn the distribution vs just memorize the training data ( Arora et al. , 2017 ; 2018 ; Dumoulin et al. , 2016 ) . Some corresponding cures have been proposed . For instance , Zhang et al . ( 2018 ) ; Bai et al . ( 2018 ) show for specific generators combined with appropriate parametric discriminator design , WGANs can attain parametric statistical rates , avoiding the exponential in dimension sample complexity ( Liang , 2018 ; Bai et al. , 2018 ; Feizi et al. , 2017 ) . Recent work of Wu et al . ( 2019 ) provides an algorithm to learn the distribution of a single-layer ReLU generator network . While our conclusion appears similar , our focus is very different . Our paper targets understanding when a WGAN formulation trained with stochastic GDA can learn in polynomial time and sample complexity . Their work instead relies on a specifically tailored algorithm for learning truncated normal distributions Daskalakis et al . ( 2018 ) . 2 PRELIMINARIES . We consider GAN formulations for learning a generatorGA : Rk → Rd of the form z 7→ x = φ ( Az ) , where A is a d × k parameter matrix and φ some activation function . We consider discriminators Dv : Rd → R or DV : Rd → R that are linear or quadratic forms respectively for the different purposes of learning the marginals or the joint distribution . We assume latent variables z are sampled from the normal N ( 0 , Ik×k ) , where Ik×k denotes the identity matrix of size k. The real/target distribution outputs samples x ∼ D = GA∗ ( N ( 0 , Ik0×k0 ) ) , for some ground truth parameters A∗ , where A∗ is d× k0 , and we take k ≥ k0 for enough expressivity , taking k = d when k0 is unknown . The Wasserstain GAN under our choice of generator and discriminator is naturally formulated as : min A∈Rd×k max v∈Rd { f ( A , v ) ≡ Ex∼DDv ( x ) − Ez∼N ( 0 , Ik×k ) Dv ( GA ( z ) ) } .1 1We will replace v by V ∈ Rd×d when necessary . We use ai to denote the i-th row vector of A . We sometimes omit the 2 subscript , using ‖x‖ to denote the 2-norm of vector x , and ‖X‖ to denote the spectral norm of matrix X . Sn ⊂ Rn×n represents all the symmetric matrices of dimension n× n. We use Df ( X0 ) [ B ] to denote the directional derivative of function f at point X0 with direction B : Df ( X0 ) [ B ] = limt→0 f ( X0+tB ) −f ( X0 ) t . 3 WARM-UP : LEARNING THE MARGINAL DISTRIBUTIONS . As a warm-up , we ask whether a simple linear discriminator is sufficient for the purposes of learning the marginal distributions of all coordinates of D. Notice that in our setting , the i-th output of the generator is φ ( x ) where x ∼ N ( 0 , ‖ai‖2 ) , and is thus solely determined by ‖ai‖2 . With a linear discriminator Dv ( x ) = v > x , our minimax game becomes : min A∈Rd×k max v∈Rd { f1 ( A , v ) ≡ Ex∼D [ v > x ] − Ez∼N ( 0 , Ik×k ) [ v > φ ( Az ) ] } . ( 1 ) Notice that when the activation φ is an odd function , such as the tanh activation , the symmetric property of the Gaussian distribution ensures that Ex∼D [ v > x ] = 0 , hence the linear discriminator in f1 reveals no information about A∗ . Therefore specifically for odd activations ( or odd plus a constant activations ) , we instead use an adjusted rectified linear discriminator Dv ( x ) ≡ v > R ( x−C ) to enforce some bias , where C = 12 ( φ ( x ) + φ ( −x ) ) for all x , and R denotes the ReLU activation . Formally , we slightly modify our loss function as : f̄1 ( A , v ) ≡ Ex∼D [ v > R ( x− C ) ] − Ez∼N ( 0 , Ik×k ) [ v > R ( φ ( Az ) − C ) ] . ( 2 ) We will show that we can learn each marginal of D if the activation function φ satisfies the following . Assumption 1 . The activation function φ satisfies either one of the following : 1. φ is an odd function plus constant , and φ is monotone increasing ; 2 . The even component of φ , i.e . 12 ( φ ( x ) +φ ( −x ) ) , is positive and monotone increasing on x ∈ [ 0 , ∞ ) . Remark 1 . All common activation functions like ( Leaky ) ReLU , tanh or sigmoid function satisfy Assumption 1 . Lemma 1 . Suppose A∗ 6= 0 . Consider f1 with activation that satisfies Assumption 1.2 and f̄1 with activation that satisfies Assumption 1.1 . The stationary points of such f1 and f̄1 yield parameters A satisfying ‖ai‖ = ‖a∗i ‖ , ∀i ∈ [ d ] . To bound the capacity of the discriminator , similar to the Lipschitz constraint in WGAN , we regularize the discriminator . For the regularized formulation we have : Theorem 1 . In the same setting as Lemma 1 , alternating gradient descent-ascent with proper learning rates on min A max v { f1 ( A , v ) − ‖v‖2/2 } or respectively min A max v { f̄1 ( A , v ) − ‖v‖2/2 } recovers A such that ‖ai‖ = ‖a∗i ‖ , ∀i ∈ [ d ] . All the proofs of the paper can be found in the appendix . We show that all local min-max points in the sense of ( Jin et al. , 2019 ) of the original problem are global min-max points and recover the correct norm of a∗i , ∀i . Notice for the source data distribution x = ( x1 , x2 , · · ·xd ) ∼ D with activation φ , the marginal distribution of each xi follows φ ( N ( 0 , ‖a∗i ‖2 ) ) and is determined by ‖a∗i ‖ . Therefore we have learned the marginal distribution for each entry i . It remains to learn the joint distribution .

Summary: In this paper, the authors attempt to prove that the Stochastic Gradient Descent-Ascent could converge to a global solution to the min-max problem of WGAN, in the setting of a one-layer generator and simple discriminator. They also show that the linear discriminator could be used to learn the marginal distributions of each coordinate, while a quadratic one could obtain joint distributions of every two coordinates. Since the linear discriminator and the quadratic one could be solved in one step Gradient Ascent, the author applied the standard analysis method to reveal the property of the Gradient Descent method. Experiments are also carried out to justify their theory that the WGAN could recover the distribution.