user: text: Write a concise one-sentence TLDR summary of the scientific paper content below.

Title: SGD Learns One-Layer Networks in WGANs

Paper content:
Generative adversarial networks (GANs) are a widely used framework for learning generative models.
Wasserstein GANs (WGANs), one of the most successful variants of GANs, require solving a minmax problem to global optimality, but in practice, are successfully trained with stochastic gradient descent-ascent.
In this paper, we show that, when the generator is a one-layer network, stochastic gradient descent-ascent converges to a global solution in polynomial time and sample complexity.
Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) are a prominent framework for learning generative models of complex, real-world distributions given samples from these distributions.
GANs and their variants have been successfully applied to numerous datasets and tasks, including image-to-image translation (Isola et al., 2017) , image super-resolution (Ledig et al., 2017) , domain adaptation (Tzeng et al., 2017) , probabilistic inference (Dumoulin et al., 2016) , compressed sensing (Bora et al., 2017) and many more.
These advances owe in part to the success of Wasserstein GANs (WGANs) Gulrajani et al., 2017) , leveraging the neural net induced integral probability metric to better measure the difference between a target and a generated distribution.
Along with the afore-described empirical successes, there have been theoretical studies of the statistical properties of GANs-see e.g. (Zhang et al., 2018; Arora et al., 2017; Bai et al., 2018; Dumoulin et al., 2016) and their references.
These works have shown that, with an appropriate design of the generator and discriminator, the global optimum of the WGAN objective identifies the target distribution with low sample complexity.
On the algorithmic front, prior work has focused on the stability and convergence properties of gradient descent-ascent (GDA) and its variants in GAN training and more general min-max optimization problems; see e.g. (Nagarajan & Kolter, 2017; Heusel et al., 2017; Mescheder et al., 2017; Daskalakis et al., 2017; Daskalakis & Panageas, 2018a; b; Gidel et al., 2019; Liang & Stokes, 2019; Mokhtari et al., 2019; Jin et al., 2019; Lin et al., 2019) and their references.
It is known that, even in min-max optimization problems with convex-concave objectives, GDA may fail to compute the min-max solution and may even exhibit divergent behavior.
Hence, these works have studied conditions under which GDA converges to a globally optimal solution under a convex-concave objective, or different types of locally optimal solutions under nonconvex-concave or nonconvex-nonconcave objectives.
They have also identified variants of GDA with better stability properties in both theory and practice, most notably those using negative momentum.
In the context of GAN training, Feizi et al. (2017) show that for WGANs with a linear generator and quadratic discriminator GDA succeeds in learning a Gaussian using polynomially many samples in the dimension.
In the same vein, we are the first to our knowledge to study the global convergence properties of stochastic GDA in the GAN setting, and establishing such guarantees for non-linear generators.
In particular, we study the WGAN formulation for learning a single-layer generative model with some reasonable choices of activations including tanh, sigmoid and leaky ReLU.
Our contributions.
For WGAN with a one-layer generator network using an activation from a large family of functions and a quadratic discriminator, we show that stochastic gradient descent-ascent learns a target distribution using polynomial time and samples, under the assumption that the target distribution is realizable in the architecture of the generator.
This is achieved by
a) analysis of the dynamics of stochastic gradient-descent to show it attains a global optimum of the minmax problem, and
b) appropriate design of the discriminator to ensure a parametric O( 1 √ n ) statistical rate (Zhang et al., 2018; Bai et al., 2018) .
Related Work.
We briefly review relevant results in GAN training and learning generative models: -Optimization viewpoint.
For standard GANs and WGANs with appropriate regularization, Nagarajan & Kolter (2017) , Mescheder et al. (2017) and Heusel et al. (2017) establish sufficient conditions to achieve local convergence and stability properties for GAN training.
At the equilibrium point, if the Jacobian of the associated gradient vector field has only eigenvalues with negative real-part at the equilibrium point, GAN training is verified to converge locally for small enough learning rates.
A follow-up paper by (Mescheder et al., 2018) shows the necessity of these conditions by identifying a prototypical counterexample that is not always locally convergent with gradient descent based GAN optimization.
However, the lack of global convergence prevents the analysis to provide any guarantees of learning the real distribution.
The work of (Feizi et al., 2017) described above has similar goals as our paper, namely understanding the convergence properties of basic dynamics in simple WGAN formulations.
However, they only consider linear generators, which restrict the WGAN model to learning a Gaussian.
Our work goes a step further, considering WGANs whose generators are one-layer neural networks with a broad selection of activations.
We show that with a proper gradient-based algorithm, we can still recover the ground truth parameters of the underlying distribution.
More broadly, WGANs typically result in nonconvex-nonconcave min-max optimization problems.
In these problems, a global min-max solution may not exist, and there are various notions of local min-max solutions, namely local min-local max solutions Daskalakis & Panageas (2018b) , and local min solutions of the max objective Jin et al. (2019) , the latter being guaranteed to exist under mild conditions.
In fact, Lin et al. (2019) show that GDA is able to find stationary points of the max objective in nonconvex-concave objectives.
Given that GDA may not even converge for convexconcave objectives, another line of work has studied variants of GDA that exhibit global convergence to the min-max solution Daskalakis et al. (2017) ; Daskalakis & Panageas (2018a); Gidel et al. (2019); Liang & Stokes (2019) ; Mokhtari et al. (2019) , which is established for GDA variants that add negative momentum to the dynamics.
While the convergence of GDA with negative momentum is shown in convex-concave settings, there is experimental evidence supporting that it improves GAN training (Daskalakis et al., 2017; Gidel et al., 2019 ).
-Statistical viewpoint.
Several works have studied the issue of mode collapse.
One might doubt the ability of GANs to actually learn the distribution vs just memorize the training data (Arora et al., 2017; Dumoulin et al., 2016) .
Some corresponding cures have been proposed.
For instance, Zhang et al. (2018) ; Bai et al. (2018) show for specific generators combined with appropriate parametric discriminator design, WGANs can attain parametric statistical rates, avoiding the exponential in dimension sample complexity (Liang, 2018; Bai et al., 2018; Feizi et al., 2017) .
Recent work of Wu et al. (2019) provides an algorithm to learn the distribution of a single-layer ReLU generator network.
While our conclusion appears similar, our focus is very different.
Our paper targets understanding when a WGAN formulation trained with stochastic GDA can learn in polynomial time and sample complexity.
Their work instead relies on a specifically tailored algorithm for learning truncated normal distributions Daskalakis et al. (2018) .

assistant: text: Attempts to prove that the Stochastic Gradient Decent-Ascent could converge to a global solution for the min-max problem of WGAN.