Understanding Over-parameterization in Generative Adversarial Networks

A broad class of unsupervised deep learning methods such as Generative Adversarial Networks ( GANs ) involve training of overparameterized models where the number of parameters of the model exceeds a certain threshold . Indeed , most successful GANs used in practice are trained using overparameterized generator and discriminator networks , both in terms of depth and width . A large body of work in supervised learning have shown the importance of model overparameterization in the convergence of the gradient descent ( GD ) to globally optimal solutions . In contrast , the unsupervised setting and GANs in particular involve non-convex concave mini-max optimization problems that are often trained using Gradient Descent/Ascent ( GDA ) . The role and benefits of model overparameterization in the convergence of GDA to a global saddle point in non-convex concave problems is far less understood . In this work , we present a comprehensive analysis of the importance of model overparameterization in GANs both theoretically and empirically . We theoretically show that in an overparameterized GAN model with a 1-layer neural network generator and a linear discriminator , GDA converges to a global saddle point of the underlying non-convex concave min-max problem . To the best of our knowledge , this is the first result for global convergence of GDA in such settings . Our theory is based on a more general result that holds for a broader class of nonlinear generators and discriminators that obey certain assumptions ( including deeper generators and random feature discriminators ) . Our theory utilizes and builds upon a novel connection with the convergence analysis of linear timevarying dynamical systems which may have broader implications for understanding the convergence behavior of GDA for non-convex concave problems involving overparameterized models . We also empirically study the role of model overparameterization in GANs using several large-scale experiments on CIFAR-10 and Celeb-A datasets . Our experiments show that overparameterization improves the quality of generated samples across various model architectures and datasets . Remarkably , we observe that overparameterization leads to faster and more stable convergence behavior of GDA across the board . 1 INTRODUCTION . In recent years , we have witnessed tremendous progress in deep generative modeling with some state-of-the-art models capable of generating photo-realistic images of objects and scenes ( Brock et al. , 2019 ; Karras et al. , 2019 ; Clark et al. , 2019 ) . Three prominent classes of deep generative models include GANs ( Goodfellow et al. , 2014 ) , VAEs ( Kingma & Welling , 2014 ) and normalizing flows ( Dinh et al. , 2017 ) . Of these , GANs remain a popular choice for data synthesis especially in the image domain . GANs are based on a two player min-max game between a generator network that generates samples from a distribution , and a critic ( discriminator ) network that discriminates real distribution from the generated one . The networks are optimized using Gradient Descent/Ascent ( GDA ) to reach a saddle-point of the min-max optimization problem . ∗First two authors contributed equally . Correspondence to yogesh @ cs.umd.edu , sajedi @ usc.edu One of the key factors that has contributed to the successful training of GANs is model overparameterization , defined based on the model parameters count . By increasing the complexity of discriminator and generator networks , both in depth and width , recent papers show that GANs can achieve photo-realistic image and video synthesis ( Brock et al. , 2019 ; Clark et al. , 2019 ; Karras et al. , 2019 ) . While these works empirically demonstrate some benefits of overparameterization , there is lack of a rigorous study explaining this phenomena . In this work , we attempt to provide a comprehensive understanding of the role of overparameterization in GANs , both theoretically and empirically . We note that while overparameterization is a key factor in training successful GANs , other factors such as generator and discriminator architectures , regularization functions and model hyperparameters have to be taken into account as well to improve the performance of GANs . Recently , there has been a large body of work in supervised learning ( e.g . regression or classification problems ) studying the importance of model overparameterization in gradient descent ( GD ) ’ s convergence to globally optimal solutions ( Soltanolkotabi et al. , 2018 ; Allen-Zhu et al. , 2019 ; Du et al. , 2019 ; Oymak & Soltanolkotabi , 2019 ; Zou & Gu , 2019 ; Oymak et al. , 2019 ) . A key observation in these works is that , under some conditions , overparameterized models experience lazy training ( Chizat et al. , 2019 ) where optimal model parameters computed by GD remain close to a randomly initialized model . Thus , using a linear approximation of the model in the parameter space , one can show the global convergence of GD in such minimization problems . In contrast , training GANs often involves solving a non-convex concave min-max optimization problem that fundamentally differs from a single minimization problem of classification/regression . The key question is whether overparameterized GANs also experience lazy training in the sense that overparameterized generator and discriminator networks remain sufficiently close to their initializations . This may then lead to a general theory of global convergence of GDA for such overparameterized non-convex concave min-max problems . In this paper we first theoretically study the role of overparameterization for a GAN model with a 1-hidden layer generator and a linear discriminator . We study two optimization procedures to solve this problem : ( i ) using a conventional training procedure in GANs based on GDA in which generator and discriminator networks perform simultaneous steps of gradient descent to optimize their respective models , ( ii ) using GD to optimize generator ’ s parameters for the optimal discriminator . The latter case corresponds to taking a sufficiently large number of gradient ascent steps to update discriminator ’ s parameters for each GD step of the generator . In both cases , our results show that in an overparameterized regime , the GAN optimization converges to a global solution . To the best of our knowledge , this is the first result showing the global convergence of GDA in such settings . While in our results we focus on one-hidden layer generators and linear discriminators , our theory is based on analyzing a general class of min-max optimization problems which can be used to study a much broader class of generators and discriminators potentially including deep generators and deep random feature-based discriminators . A key component of our analysis is a novel connection to exponential stability of non-symmetric time varying dynamical systems in control theory which may have broader implications for theoretical analysis of GAN ’ s training . Ideas from control theory have also been used for understanding and improving training dynamics of GANs in ( Xu et al. , 2019 ; An et al. , 2018 ) . Having analyzed overparameterized GANs for relatively simple models , we next provide a comprehensive empirical study of this problem for practical GANs such as DCGAN ( Radford et al. , 2016 ) and ResNet GAN ( Gulrajani et al. , 2017 ) trained on CIFAR-10 and Celeb-A datasets . For example , the benefit of overparamterization in training DCGANs on CIFAR-10 is illustrated in Figure 1 . We have three key observations : ( i ) as the model becomes more overparameterized ( e.g . using wider networks ) , the training FID scores that measure the training error , decrease . This phenomenon has been observed in other studies as well ( Brock et al. , 2019 ) . ( ii ) overparameterization does not hurt the test FID scores ( i.e . the generalization gap remains small ) . This improved test-time performance can also be seen qualitatively in the center panel of Figure 1 , where overparameterized models produce samples of improved quality . ( iii ) Remarkably , overparameterized GANs , with a lot of parameters to optimize over , have significantly improved convergence behavior of GDA , both in terms of rate and stability , compared to small GAN models ( see the right panel of Figure 1 ) . In summary , in this paper • We provide the first theoretical guarantee of simultaneous GDA ’ s global convergence for an overparameterized GAN with one-hidden neural network generator and a linear discriminator ( Theorem 2.1 ) . • By establishing connections with linear time-varying dynamical systems , we provide a theoretical framework to analyze simultaneous GDA ’ s global convergence for a general overparameterized GAN ( including deeper generators and random feature discriminators ) , under some general conditions ( Theorems 2.3 and A.4 ) . • We provide a comprehensive empirical study of the role of model overparameterization in GANs using several large-scale experiments on CIFAR-10 and Celeb-A datasets . We observe overparameterization improves GANs ’ training error , generalization error , sample qualities as well as the convergence rate and stability of GDA . 2 THEORETICAL RESULTS . 2.1 PROBLEM FORMULATION . Given n data points of the form x1 , x2 , . . . , xn ∈ Rm , the goal of GAN ’ s training is to find a generator that can mimic sampling from the same distribution as the training data . More specifically , the goal is to find a generator mapping Gθ ( z ) : Rd → Rm , parameterized by θ ∈ Rp , so that Gθ ( z1 ) , Gθ ( z2 ) , . . . , Gθ ( zn ) with z1 , z2 , . . . , zn generated i.i.d . according toN ( 0 , Id ) has a similar empirical distribution to x1 , x2 , . . . , xn1 . To measure the discrepancy between the data points and the GAN outputs , one typically uses a discriminator mapping Dθ̃ : R m → R parameterized with θ̃ ∈ Rp̃ . The overall training approach takes the form of the following min-max optimization problem which minimizes the worst-case discrepancy detected by the discriminator min θ max θ̃ 1 n n∑ i=1 Dθ̃ ( xi ) − 1 n n∑ i=1 Dθ̃ ( Gθ ( zi ) ) +R ( θ̃ ) . ( 1 ) Here , R ( θ̃ ) is a regularizer that typically ensures the discriminator is Lipschitz . This formulation mimics the popular Wasserstein GAN ( Arjovsky et al. , 2017 ) ( or , IPM GAN ) formulations . This optimization problem is typically solved by running Gradient Descent Ascent ( GDA ) on the minimization/maximization variables . The generator and discriminator mappings G and D used in practice are often deep neural networks . Thus , the min-max optimization problem above is highly nonlinear and non-convex concave . Saddle point optimization is a classical and fundamental problem in game theory ( Von Neumann & Morgenstern , 1953 ) and control ( Gutman , 1979 ) . However , most of the classical results apply to the 1In general , the number of observed and generated samples can be different . However , in practical GAN implementations , batch sizes of observed and generated samples are usually the same . Thus , for simplicity , we make this assumption in our setup . convex-concave case ( Arrow et al. , 1958 ) while the saddle point optimization of GANs is often non convex-concave . If GDA converges to the global ( local ) saddle points , we say it is globally ( locally ) stable . For a general min-max optimization , however , GDA can be trapped in a loop or even diverge . Except in some special cases ( e.g . ( Feizi et al. , 2018 ) for a quadratic GAN formulation or ( Lei et al. , 2019 ) for the under-parametrized setup when the generator is a one-layer network ) , GDA is not globally stable for GANs in general ( Nagarajan & Kolter , 2017 ; Mescheder et al. , 2018 ; Adolphs et al. , 2019 ; Mescheder et al. , 2017 ; Daskalakis et al. , 2020 ) . None of these works , however , study the role of model overparameterization in the global/local convergence ( stability ) of GDA . In particular , it has been empirically observed ( as we also demonstrate in this paper ) that when the generator/discriminator contain a large number of parameters ( i.e . are sufficiently overparameterized ) GDA does indeed find ( near ) globally optimal solutions . In this section we wish to demystify this phenomenon from a theoretical perspective .

Summary: This paper studies the effect of model over-parametrization in GANs. While there is a lot of work on this in the supervised learning setting of classification/regression there is not much in the GAN framework where the minimax objective function complicates such an analysis. This paper considers two types of training of the GAN model, one with the simultaneous gradient descent ascent and one where the discriminator is trained to optimality for every generator update. It provides global convergence results under both algorithms in the case of a generator network with one hidden layer that is large enough and a linear discriminator.