A Closer Look at the Optimization Landscapes of Generative Adversarial Networks

1 INTRODUCTION . Deep neural networks have exhibited remarkable success in many applications ( Krizhevsky et al. , 2012 ) . This success has motivated many studies of their non-convex loss landscape ( Choromanska et al. , 2015 ; Kawaguchi , 2016 ; Li et al. , 2018b ) , which , in turn , has led to many improvements , such as better initialization and optimization methods ( Glorot and Bengio , 2010 ; Kingma and Ba , 2015 ) . While most of the work on studying non-convex loss landscapes has focused on single objective minimization , some recent class of models require the joint minimization of several objectives , making their optimization landscape intrinsically different . Among these models is the generative adversarial network ( GAN ) ( Goodfellow et al. , 2014 ) which is based on a two-player game formulation and has achieved state-of-the-art performance on some generative modeling tasks such as image generation ( Brock et al. , 2019 ) . On the theoretical side , many papers studying multi-player games have argued that one main optimization issue that arises in this case is the rotation due to the adversarial component of the game ( Mescheder et al. , 2018 ; Balduzzi et al. , 2018 ; Gidel et al. , 2019b ) . This has been extensively studied on toy examples , in particular on the so-called bilinear example ( Goodfellow , 2016 ) ( a.k.a Dirac GAN ( Mescheder et al. , 2018 ) ) . However , those toy examples are very far from the standard realistic setting of image generation involving deep networks and challenging datasets . To our knowledge it remains an open question if this rotation phenomenon actually occurs when training GANs in more practical settings . In this paper , we aim at closing this gap between theory and practice . Following Mescheder et al . ( 2017 ) and Balduzzi et al . ( 2018 ) , we argue that instead of studying the loss surface , we should study the game vector field ( i.e. , the concatenation of each player ’ s gradient ) , which can provide ∗Equal contributions . Correspondence to firstname.lastname @ umontreal.ca . †Canada CIFAR AI Chair ( held at Mila ) 1Code available at https : //bit.ly/2kwTu87 better insights to the problem . To this end , we propose a new visualization technique that we call Path-angle which helps us observe the nature of the game vector field close to a stationary point for high dimensional models , and carry on an empirical investigation of the properties of the optimization landscape of GANs . The core questions we want to address may be summarized as the following : Is rotation a phenomenon that occurs when training GANs on real world datasets , and do existing training methods find local Nash equilibria ? . To answer this question we conducted extensive experiments by training different GAN formulations ( NSGAN and WGAN-GP ) with different optimizers ( Adam and ExtraAdam ) on three datasets ( MoG , MNIST and CIFAR10 ) . Based on our experiments and using our visualization techniques we observe that the landscape of GANs is fundamentally different from the standard loss surfaces of deep networks . Furthermore , we provide evidence that existing GAN training methods do not converge to a local Nash equilibrium . Contributions More precisely , our contributions are the following : ( i ) We propose studying empirically the game vector field ( as opposed to studying the loss surfaces of each player ) to understand training dynamics in GANs using a novel visualization tool , which we call Path-angle and that captures the rotational and attractive behaviors near local stationary points ( ref . §4.2 ) . ( ii ) We observe experimentally on both a mixture of Gaussians , MNIST and CIFAR10 datasets that a variety of GAN formulations have a significant rotational behavior around their locally stable stationary points ( ref . §5.1 ) . ( iii ) We provide empirical evidence that existing training procedures find stable stationary points that are saddle points , not minima , for the loss function of the generator ( ref . § 5.2 ) . 2 RELATED WORK . Improving the training of GANs has been an active research area in the past few years . Most efforts in stabilizing GAN training have focused on formulating new objectives ( Arjovsky et al. , 2017 ) , or adding regularization terms ( Gulrajani et al. , 2017 ; Mescheder et al. , 2017 ; 2018 ) . In this work , we try to characterize the difference in the landscapes induced by different GAN formulations and how it relates to improving the training of GANs . Recently , Nagarajan and Kolter ( 2017 ) ; Mescheder et al . ( 2018 ) show that a local analysis of the eigenvalues of the Jacobian of the game can provide guarantees on local stability properties . However , their theoretical analysis is based on some unrealistic assumptions such as the generator ’ s ability to fully capture the real distribution . In this work , we assess experimentally to what extent these theoretical stability results apply in practice . Rotations in differentiable games has been mentioned and interpreted by ( Mescheder et al. , 2018 ; Balduzzi et al. , 2018 ) and Gidel et al . ( 2019b ) . While these papers address rotations in games from a theoretical perspective , it was never shown that GANs , which are games with highly non-convex losses , suffered from these rotations in practice . To our knowledge , trying to quantify that GANs actually suffer from this rotational component in practice for real world dataset is novel . The stable points of the gradient dynamics in general games have been studied independently by Mazumdar and Ratliff ( 2018 ) and Adolphs et al . ( 2018 ) . They notice that the locally stable stationary point of some games are not local Nash equilibria . In order to reach a local Nash equilibrium , Adolphs et al . ( 2018 ) ; Mazumdar et al . ( 2019 ) develop techniques based on second order information . In this work , we argue that reaching local Nash equilibria may not be as important as one may expect and that we do achieve good performance at a locally stable stationary point . Several works have studied the loss landscape of deep neural networks . Goodfellow et al . ( 2015 ) proposed to look at the linear path between two points in parameter space and show that neural networks behave similarly to a convex loss function along this path . Draxler et al . ( 2018 ) proposed an extension where they look at nonlinear paths between two points and show that local minima are connected in deep neural networks . Another extension was proposed by ( Li et al. , 2018a ) where they use contour plots to look at the 2D loss surface defined by two directions chosen appropriately . In this paper , we use a similar approach of following the linear path between two points to gain insight about GAN optimization landscapes . However , in this context , looking at the loss of both players along that path may be uninformative . We propose instead to look , along a linear path from initialization to best solution , at the game vector field , particularly at its angle w.r.t . the linear path , the Path-angle . Another way to gain insight into the landscape of deep neural networks is by looking at the Hessian of the loss ; this was done in the context of single objective minimization by ( Dauphin et al. , 2014 ; Sagun et al. , 2016 ; 2017 ; Alain et al. , 2019 ) . Compared to linear path visualizations which can give global information ( but only along one direction ) , the Hessian provides information about the loss landscape in several directions but only locally . The full Hessian is expensive to compute and one often has to resort to approximations such has computing only the top-k eigenvalues . While , the Hessian is symmetric and thus has real eigenvalues , the Jacobian of a game vector field is significantly different since it is in general not symmetric , which means that the eigenvalues belong to the complex plane . In the context of GANs , Mescheder et al . ( 2017 ) introduced a gradient penalty and use the eigenvalues of the Jacobian of the game vector field to show its benefits in terms of stability . In our work , we compute these eigenvalues to assess that , on different GAN formulations and datasets , existing training procedures find a locally stable stationary point that is a saddle point for the loss function of the generator . 3 FORMULATIONS FOR GAN OPTIMIZATION AND THEIR PRACTICAL IMPLICATIONS . 3.1 THE STANDARD GAME THEORY FORMULATION . From a game theory point of view , GAN training may be seen as a game between two players : the discriminator Dϕ and the generator Gθ , each of which is trying to minimize its loss LD and LG , respectively . Using the same formulation as Mescheder et al . ( 2017 ) , the GAN objective takes the following form ( for simplicity of presentation , we focus on the unconstrained formulation ) : θ∗ ∈ arg min θ∈Rp LG ( θ , ϕ∗ ) and ϕ∗ ∈ arg min ϕ∈Rd LD ( θ∗ , ϕ ) . ( 1 ) The solution ( θ∗ , ϕ∗ ) is called a Nash equilibrium ( NE ) . In practice , the considered objectives are non-convex and we typically can not expect better than a local Nash equilibrium ( LNE ) , i.e . a point at which ( 1 ) is only locally true ( see e.g . ( Adolphs et al. , 2018 ) for a formal definition ) . Ratliff et al . ( 2016 ) derived some derivative-based necessary and sufficient conditions for being a LNE . They show that , for being a local NE it is sufficient to be a differential Nash equilibrium : Definition 1 ( Differential NE ) . A point ( θ∗ , ϕ∗ ) is a differential Nash equilibrium ( DNE ) iff ‖∇θLG ( θ∗ , ϕ∗ ) ‖ = ‖∇ϕLD ( θ∗ , ϕ∗ ) ‖ = 0 , ∇2θLG ( θ∗ , ϕ∗ ) 0 and ∇2ϕLD ( θ∗ , ϕ∗ ) 0 ( 2 ) where S 0 if and only if S is positive definite . Being a DNE is not necessary for being a LNE because a local Nash equilibrium may have Hessians that are only semi-definite . NE are commonly used in GANs to describe the goal of the learning procedure ( Goodfellow et al. , 2014 ) : in this definition , θ∗ ( resp . ϕ∗ ) is seen as a local minimizer of LG ( · , ϕ∗ ) ( resp . LD ( θ∗ , · ) ) . Under this view , however , the interaction between the two networks is not taken into account . This is an important aspect of the game stability that is missed in the definition of DNE ( and Nash equilibrum in general ) . We illustrate this point in the following section , where we develop an example of a game for which gradient methods converge to a point which is a saddle point for the generator ’ s loss and thus not a DNE for the game . 3.2 AN ALTERNATIVE FORMULATION BASED ON THE GAME VECTOR FIELD . In practice , GANs are trained using first order methods that compute the gradients of the losses of each player . Following Gidel et al . ( 2019a ) , an alternative point of view on optimizing GANs is to jointly consider the players ’ parameters θ and ϕ as a joint state ω : = ( θ , ϕ ) , and to study the vector field associated with these gradients,2 which we call the game vector field v ( ω ) : = [ ∇θLG ( ω ) > ∇ϕLD ( ω ) > ] > where ω : = ( θ , ϕ ) . ( 3 ) 2Note that , in practice , the joint vector field ( 3 ) is not a gradient vector field , i.e. , it can not be rewritten as the gradient of a single function . With this perspective , the notion of DNE is replaced by the notion of locally stable stationary point ( LSSP ) . Verhulst ( 1989 , Theorem 7.1 ) defines a LSSP ω∗ using the eigenvalues of the Jacobian of the game vector field∇v ( ω∗ ) at that point . Definition 2 ( LSSP ) . A point ω∗ is a locally stable stationary point ( LSSP ) iff v ( ω∗ ) = 0 and < ( λ ) > 0 , ∀λ ∈ Sp ( ∇v ( ω∗ ) ) . ( 4 ) where < denote the real part of the eigenvalue λ belonging to the spectrum of∇v ( ω∗ ) . This definition is not easy to interpret but one can intuitively understand a LSSP as a stationary point ( a point ω∗ where v ( ω∗ ) = 0 ) to which all neighbouring points are attracted . We will formalize this intuition of attraction in Proposition 1 . In our two-player game setting , the Jacobian of the game vector field around the LSSP has the following block-matrices form : ∇v ( ω∗ ) = [ ∇2θLG ( ω∗ ) ∇ϕ∇θLG ( ω∗ ) ∇θ∇ϕLD ( ω∗ ) ∇2ϕLD ( ω∗ ) ] = [ S1 B A S2 ] . ( 5 ) WhenB = −A > , being a DNE is a sufficient condition for being of LSSP ( Mazumdar and Ratliff , 2018 ) . However , some LSSP may not be DNE ( Adolphs et al. , 2018 ) , meaning that the optimal generator θ∗ could be a saddle point of LG ( · , ϕ∗ ) , while the optimal joint state ( θ∗ , ϕ∗ ) may be a LSSP of the game . We summarize these properties in Table 1 . In order to illustrate the intuition behind this counter-intuitive fact , we study a simple example where the generator is 2D and the discriminator is 1D . Example 1 . Let us consider LG as a hyperbolic paraboloid ( a.k.a. , saddle point function ) centered in ( 1 , 1 ) where ( 1 , ϕ ) is the principal descent direction and ( −ϕ , 1 ) is the principal ascent direction , while LD is a simple bilinear objective . LG ( θ1 , θ2 , ϕ ) = ( θ2 − ϕθ1 − 1 ) 2 − 12 ( θ1 + ϕθ2 − 1 ) 2 , LD ( θ1 , θ2 , ϕ ) = ϕ ( 5θ1 + 4θ2 − 9 ) We plot LG in Fig . 1b . Note that the discriminator ϕ controls the principal descent direction of LG . We show ( see § A.2 ) that ( θ∗1 , θ∗2 , ϕ∗ ) = ( 1 , 1 , 0 ) is a locally stable stationary point but is not a DNE : the generator loss at the optimum ( θ1 , θ2 ) 7→ LG ( θ1 , θ2 , ϕ∗ ) = θ22 − 12θ21 is not at a DNE because it has a clear descent direction , ( 1 , 0 ) . However , if the generator follows this descent direction , the dynamics will remain stable because the discriminator will update its parameter , rotating the saddle and making ( 1 , 0 ) an ascent direction . We call this phenomenon dynamic stability : the loss LG ( · , ϕ∗ ) is unstable for a fixed ϕ∗ but becomes stable when ϕ dynamically interacts with the generator around ϕ∗ . A mechanical analogy for this dynamic stability phenomenon is a ball in a rotating saddle—even though the gravity pushes the ball to escape the saddle , a quick enough rotation of the saddle would trap the ball at the center ( see ( Thompson et al. , 2002 ) for more details ) . This analogy has been used to explain Paul ’ s trap ( Paul , 1990 ) : a counter-intuitive way to trap ions using a dynamic electric field . In Example 1 , the parameter ϕ explicitly controls the rotation of the saddle . This example illustrates the fact that the DNE corresponds to a notion of static stability : it is the stability of one player ’ s loss given the other player is fixed . Conversely , LSSP captures a notion of dynamic stability that considers both players jointly . By looking at the game vector field we capture these interactions . Fig . 1b only captures a snapshot of the generator ’ s loss surface for a fixed ϕ and indicates static instability ( the generator is at a saddle point of its loss ) . In Fig . 1a , however , one can see that , starting from any point , we will rotate around the stationary point ( ϕ∗ , θ∗1 ) = ( 0 , 1 ) and eventually converge to it . The visualization of the game vector field reveals an interesting behavior that does not occur in single objective minimization : close to a LSSP , the parameters rotate around it . Understanding this phenomenon is key to grasp the optimization difficulties arising in games . In the next section , we formally characterize the notion of rotation around a LSSP and in §4 we develop tools to visualize it in high dimensions . Note that gradient methods may converge to saddle points in single objective minimization , but these are not stable stationary points , unlike in our game example .

Summary: This paper tries to provide a deeper understanding of the training dynamics of GANs in practice via characterizing and visualizing the rotation and attraction phenomena nearby a locally stable stationary point (LSSP) and questions the necessity to access a differential/local Nash equilibrium (LNE). In particular, this paper first discusses the difference between LSSP and LNE and formalize the notions of rotation and attraction around LSSP in games. Then, this paper proposes the path angle to visualize the rotation and attraction nearby an LSSP. The path angle is a function that maps linearly distributed points in the line, which is determined by an initial parameter set and a well-trained parameter set, to the angles between the line and the gradient of a given point in that line. The rotation and attraction phenomena can be observed in the plot of the path angle as  "a quick sign switch" and "a bump" nearby 1, respectively. The experiments empirically demonstrate that: 1. rotation exists in the training dynamics of practical GANs; 2. GANs often converge to an LSSP than an LNE, but still, achieve good results.