A Closer Look at the Optimization Landscapes of Generative Adversarial Networks
Authors: Hugo Berard
Gauthier Gidel
Amjad Almahairi
Pascal Vincent
Simon Lacoste-Julien
Venue:  (2020)
Keywords: Deep Learning
Generative models
GANs
Optimization
Visualization

Abstract: Generative adversarial networks have been very successful in generative modeling, however they remain relatively challenging to train compared to standard deep neural networks. In this paper, we propose new visualization techniques for the optimization landscapes of GANs that enable us to study the game vector field resulting from the concatenation of the gradient of both players.   Using these visualization techniques we try to bridge the gap between theory and practice by showing empirically that the training of GANs exhibits significant rotations around LSSP, similar to the one predicted by theory on toy examples. Moreover, we provide empirical evidence that GAN training seems to converge to a stable stationary point which is a saddle point for the generator loss, not a minimum, while still achieving excellent performance.

Published as a conference paper at ICLR 2020
A CLOSER LOOK AT THE OPTIMIZATION LANDSCAPES OF GENERATIVE ADVERSARIAL NETWORKS
Hugo Berard
Gauthier Gidel
Mila, Université de Montréal Facehook AI Research
Mila, Université de Montréal Element Al
Amjad Almahairi Element Al
Pascal Vincent
Simon Lacoste-Julien'
Mila. Université de Montréal
Mila, Université de Montréal Element Al
Facebook Al Research
ABSTRACT
Generative adversarial networks have been very successful generative modeling, however they remain relatively challenging to train compared standard deep neural networks. In this paper, we propose new visualization techniques for the optimization landscapes of GANs that enable us study the game vector field resulting from the concatenation of the gradient of both players. Using these visualization techniques we try bridge the gap between theory and practice by showing empirically that the training of GANs exhibits significant rotations around Local Stable Stationary Points (LSSP), similar to the one predicted by theory on toy examples. Moreover, we provide empirical evidence that GAN training converge to a stable stationary point which a saddle point for the generator loss, not a minimum, while still achieving excellent performance.
INTRODUCTION
Deep neural networks have exhibited remarkable success in many applications (Krizhevsky et al., 2012). This success has motivated many studies of their non-convex loss landscape (Choromanska al., 2015; Kawaguchi, 2016; Li al., 2018b), which, in turn, has led to many improvements, such as better initialization and optimization methods (Glorot and Bengio, 2010; Kingma and Ba, 2015).
While most of the work on studying non-convex loss landscapes has focused on single objective minimization, some recent class of models require the joint minimization of several objectives, making their optimization landscape intrinsically different. Among these models is the generative adversarial network (GAN) (Goodfellow al., 2014) which is based on two-player game formula- tion and has achieved state-of-the-arr performance on some generative modeling tasks such as image generation (Brock et al., 2019).
On the theoretical side, many papers studying multi-player games have argued that one main op- timization issue that arises in this case is the rotation due the adversarial component of the game (Mescheder et al., 2018; Balduzzi et al., 2018; Gidel et al., 2019b). This has been extensively studied toy examples, in particular Dirac GAN (Mescheder et al., 2018)). However, those toy examples are very far from the standard realistic setting of image generation involving deep networks and challenging datasets. To our knowledge it remains an open question if this rotation phenomenon actually occurs when training GANs in more practical settings.
the so-called bilinear example (Goodfellow, 2016) (a.k.a
In this paper, we aim at closing this gap between theory and practice. Following Mescheder et al. (2017) and Balduzzi et (2018), we argue that instead of studying the loss surface, we should study the game vector field (i.e., the concatenation of each player's gradient), which can provide
" "Equal contributions. Correspondence firstname wat.tme...sttnnee Lastnaneeumnnt 1 CCandda CIFAR AI Chair (held at Mila) Coode available https: /btt. 1y 12/2k

---

Published as a conference paper at ICLR 2020
better insights to the problem. To this end. we propose a new visualization technique that we call Path-angle which helps us observe the nature of the game vector field close to a stationary point for high dimensional models, and carry on empirical investigation of the properties of the optimization landscape of GANs. The core questions we want to address may be summarized as the following:
Is rotation a phenomenon that occurs when training GANs on real world datasets, and do existing training methods find local Nash equilibria?
To answer this question we conducted extensive experiments by training different GAN formulations (NSGAN and WGAN-GP) with different optimizers (Adam and ExtraAdam) on three datasets (MoG, MNIST and CIFARIO). Based our experiments and using our visualization techniques we observe that the landscape of GANs is fundamentally different from the standard loss surfaces of deep networks. Furthermore, we provide evidence that existing GAN training methods do not converge to a local Nash equilibrium.
Contributions More preciscly, our contributions are the following: ii We propose studying empirically the game vector field (as opposed to studying the loss surfaces of each player) to understand training dynamics in GANs using novel visualization tool, which we call Path-angle and that captures the rotational and attractive behaviors near local stationary points (ref. $4.2). (ii) We both a mixture of Gaussians. MNIST and CIFARIO datasets that variety of GAN formulations have a significant rotational behavior around their locally stable stationary points (ref. $5.1). (iii) We provide empirical evidence that existing training procedures find stable stationary points that are saddle points, not minima, for the loss function of the generator (ref. 5.2).
observe experimentally
2 RELATED WORK
Improving the training of GANs has been an active research area in the past few years. Most efforts in stabilizing GAN training have focused on formulating new objectives (Arjovsky et al., 2017), or adding regularization terms (Gulrajani et al., 2017; Mescheder et al., 2017: 2018). In this work, we try to characterize the difference in the landscapes induced by different GAN formulations and how it relates to improving the training of GANs.
Recently, Nagarajan and Kolter (2017); Mescheder et al. (2018) show that a local analysis of the eigenvalues of the Jacobian of the game can provide guarantees on local stability properties. However, their theoretical analysis is based on some unrealistic assumptions such as the generator's ability to fully capture the real distribution. In this work, we assess experimentally to what extent these theoretical stability results apply practice.
Rotations in differentiable games has been mentioned and interpreted by (Mescheder et al., 2018; Balduzzi et al., 2018) and Gidel et al. (2019b). While these papers address rotations in games from a theoretical perspective, was never shown that GANs, which are games with highly non-convex losses, suffered from these rotations in practice. To our knowledge, trying to quantify that GANs actually suffer from this rotational component in practice for real world dataset is novel.
The stable points of the gradient dynamies in general games have been studied independently by Mazumdar and Ratliff (2018) and Adolphs et al. (2018). They notice that the locally stable stationary point of some games are not local Nash equilibria. In order reach a local Nash equilibrium, Adolphs et al. (2018); Mazumdar et al. (2019) develop techniques based on second order information. In this work, we argue that reaching local Nash equilibria may not be as important as one may expect and that we do achieve good performance at a locally stable stationary point.
Several works have studied the loss landscape of deep neural networks. Goodfellow al. (2015) proposed to look at the linear path between two points in parameter space and show that neural networks behave similarly to convex loss function along this path. Draxler et (2018) proposed an extension where they look at nonlinear paths between two points and show that local minima are connected in deep neural networks. Another extension was proposed by (Li et al., 2018a) where they use contour plots to look at the 2D loss surface defined by two directions chosen appropriately. In this paper, we use a similar approach of following the linear path between two points to gain insight about GAN optimization landscapes. However, in this context, looking at the loss of both players along that path may be uninformative. We propose instead to look, along a linear path from initialization to best solution, at the game vector field, particularly at its angle w.r.t. the linear path, the Path-angle.
2

---

Published as a conference paper at ICLR 2020
Another way to gain insight into the landscape of deep neural networks is by looking at the Hessian of the loss; this was done in the context of single objective minimization by (Dauphin et al., 2014; Sagun al., 2016; 2017; Alain al., 2019). Compared to linear path visualizations which can give global information (but only along one direction), the Hessian provides information about the loss landscape in several directions but only locally. The full Hessian is expensive to compute and one often has to resort to approximations such has computing only the top-k eigenvalues. While, the Hessian is symmetric and thus has real eigenvalues, the Jacobian of game vector field is significantly different since it is in general not symmetric, which means that the eigenvalues belong to the complex plane. In the context of GANs, Mescheder al. (2017) introduced a gradient penalty and use the eigenvalues of the Jacobian of the game vector field to show its benefits in terms of stability. In our work, we compute these eigenvalues to assess that, on different GAN formulations and datasets, existing training procedures find locally stable stationary point that is saddle point for the loss function of the generator.
3. FORMULATIONS FOR GAN OPTIMIZATION AND THEIR PRACTICAL IMPLICATIONS
3.1 THE STANDARD GAME THEORY FORMULATION
From a discriminator Dup and the generator Gg, each of which is trying to minimize its loss CD and Lc. respectively. Using the same formulation as Mescheder et al. (2017), the GAN objective takes the following form (for simplicity of presentation, we focus on the unconstrained formulation):
game theory point of view, GAN training may be seen as game between two players: the
arg min La (0,4')))))))))))()))) BER*
and
"0" E arg min LD(0", ,4)
yERd
The solution (²") non-convex and we typically cannot expect better than a local Nash equilibrium (LNE), i.c. a point at which (1) is only locally true (see e.g. (Adolphs et al., 2018) for formal definition). Ratliff et al. (2016) derived some derivative-based necessary and sufficient conditions for being a LNE. They show that, for being local NE it is sufficient to be differential Nash equilibrium:
) is called a Nash equilibrium (NE). In practice, the considered objectives are
Definition 1 (Differential NE). A point (0', 4p*) is differential Nash equilibrium (DNE) iff
(o ,4'))1 VV .4'))l (0')l==0, 0, V2La(0", ;0) > 0 and 4 (0"
,4¹ > 0 (2)
where S > O if and only if S is positive definite.
Being DNE is not necessary for being LNE because local Nash equilibrium may have Hessians that are only semi-definite. NE are commonly used in GANs describe the goal of the learning procedure (Goodfellow et al., 2014): in this definition, 6* (resp. sp") is seen as a local minimizer of (o(: (resp. (D(0",))
Under this view, however, the interaction between the two networks is not taken into account. This is an important aspect of the game stability that missed in the definition of DNE (and Nash equilibrum in general). We illustrate this point in the following section, where we develop an example of a game for which gradient methods converge to point which a saddle point for the generator's loss and thus not DNE for the game.
3.2 AN ALTERNATIVE FORMULATION BASED ON THE GAME VECTOR FIELD
In practice, GANs are trained using first order methods that compute the gradients of the losses of each player. Following Gidel al. (2019a), an alternative point of view on optimizing GANs is to
jointly consider the players' parameters O and &o a joint state w = (0, sp), and to study the vector field associated with these gradients, which we call the game vector field
v(w) == [Ve
VT where w := (0, ())
(3)
Note that, in practice the joint vector field (3) is not gradient vector field, i.c., it cannot he rewritten as the gradient of single function.
3

---

Published as a conference paper at ICLR 2020
Zero-sum game
Non-zero-sum game
NE + LSSE (Mescheder et al., 2018) NE # LSSE (Adolphs et al., 2018)
NE A LSSE (Example 2, SA.2)
NE & LSSE (Example 1)
Table 1: Summary of the implications between Differentiable Nash Equilibrium (DNE) and a locally stable stationnary point (LSSP): in general, being DNE is neither necessary or sufficient for being LSSP.
With this perspective, the notion of DNE is replaced by the notion of locally stable stationary point (LSSP). Verhulst (1989, Theorem 7.1) defines a LSSP wn using the eigenvalues of the Jacobian of the game vector field Vo(w*) at that point.
Definition 2 (LSSP). A point wn is locally stable stationary point (LSSP) iff
v(w") = 0 and
R((A) 00, Aλ Sp(Vo(w")),
(4)
where R denote the real part of the eigenvalue X belonging to the spectrum of Vo(w").
This definition is not easy to interpret but one can intuitively understand a LSSP as a stationary point (a point w" where v(w²) ) = 0) which all neighbouring points attracted. We will formalize this intuition of attraction in Proposition 1. In our two-player game setting, the Jacobian of the game
vector field around the LSSP has the following block-matrices form:
V Vo Ve(w")- = Vo VoVoLol (DD("")
V V PoLD(W>
S
B S
(5)
A
When == AT, being a DNE is sufficient condition for being of LSSP (Mazumdar and Ratliff, 2018). However, some LSSP may not be DNE (Adolphs et al., 2018), meaning that the optimal generator 0* could be saddle point of La(: while the optimal joint state (0", may be a LSSP of the game. We summarize these properties in Table I. In order to illustrate the intuition behind this counter-intuitive fact, we study simple example where the generator is 2D and the discriminator ID.
Example 1. Let us consider LG as hyperbolic paraboloid (a.k.a,, saddle point function) centered in (1, 1) where (1,4) is the principal descent direction and (-9p, 1) is the principal ascent direction, while LD a simple bilinear objective.
Lo Oz. sp) (02 -00 1)² ( + :00 1)², Ln 02, u) c(50) + 402 -02
We plot LG in Fig. lb. Note that the discriminator yp controls the principal descent direction of Lo-
We show (see 3 A.2) that (87,02, 10,02,000 )0*) 1, is locally stable stationary point but is not a DNE: the generator loss at the optimum (0,, + La (01, 02, ) = 03 O is not at a DNE because it has a clear descent direction, 0). However, if the generator follows this descent direction, the dynamics will remain stable because the discriminator will update its parameter, rotating the saddle and making (1, 0) an ascent direction. We call this phenomenon dynamic stability: the loss Ca sp") is unstable for a fixed sp but becomes stable when dynamically interacts with the generator around pp"
A mechanical analogy for this dynamic stability phenomenon is ball in a rotating saddle- even though the gravity pushes the ball to escape the saddle, quick enough rotation of the saddle would trap the ball at the center (see (Thompson et al., 2002) for more details). This analogy has been used to explain Paul's trap (Paul, 1990): a counter-intuitive way to trap ions using a dynamic electric field. In Example the parameter yp explicitly controls the rotation of the saddle.
This example illustrates the fact that the DNE corresponds to a notion of static stability: is the stability of one player's loss given the other player is fixed. Conversely, LSSP captures a notion of dynamic stability that considers both players jointly.
By looking the game vector field capture these interactions. Fig. 1b only captures snapshot of the generator's s loss surface for a fixed yp and indicates static instability (the generator is at a saddle point of its loss). In Fig. 1a, however, one the stationary point (4"", (7) = (0, 1) and eventually converge to it.
that, starting from any point, we will rotate around
The visualization of the game vector field reveals an interesting behavior that does not occur in single objective minimization: close to a LSSP, the parameters rotate around it. Understanding this phenomenon is key to grasp the optimization difficulties arising in games. In the next section,
4

---

Published as a conference paper at ICLR 2020
0.5-
0.0-
-0.5
9
02
0.5
1.0
1.5
(a) 2D projection of the vector field.
(b) Landscape of the generator loss.
Figure 1: Visualizations of Example 1. Left: projection of the game vector field on the plane 0₂ = 1. Right: Generator loss. The descent direction is (1, sp) (in grey). As the generator follows this descent direction, the discriminator changes the value of se. making the saddle rotate, indicated by the circular black arrow
formally characterize the notion of rotation around LSSP and in 34 we develop tools visualize in high dimensions. Note that gradient methods may converge to saddle points in single objective minimization, but these are not stable stationary points, unlike in our game example.
3.3 ROTATION AND ATTRACTION AROUND LOCALLY STABLE STATIONARY POINTS IN GAMES
In this section, we formalize the notions of rotation and attraction around LSSP in games, which we believe may explain some difficulties in GAN training. The local stability of a LSSP is characterized by the eigenvalues of the Jacobian Vv(w") because we can linearize v(w) around w
v(w) ~ Vo(w") (w(w")(w-wi - w*)
(6)
If we assume that (6) is an equality, we have the following theorem.
Proposition 1. Let us assume that (6) is an equality and that Ve(w") is diagonalizable, then there exists a basis P such that the coordinates ()) ;= [P(w(l) w") }l where w(t) is solution (6)
have the following behavior: for Xg E Sp Vv(w") we have,
1. Af E. R, observe pure attraction:
(t(() - (
(t({)
cos |Aytl
sin Ayt
3 (0)
2. If ((A))) = O, we observe pure rotation:
(0+1
sin |yyt
cos |yyt|
3. Otherwise, we observe both:
(t(() (++z(())
= Re(A,)e -
cos Im Im (Ayt) sin Im Xyt)
sin Im (Axt)) cos Im (Aytt)
0,100
Note that we re-ordered the eigenvalues such that the complex conjugate eigenvalues form pairs: if Xg & R then Aj+1 = Xj.
Matrices in 2. and 3. are rotations matrices. They induce a rotational behavior illustrated in Fig la.
This proposition shows that the dynamics of w(t) can be decomposed in a particular basis into attractions and rotations over components that do not interact between each other. Rotation does not appear in single objective minimization around a local minimum, because the eigenvalues of the Hessian of the objective are always real. Mescheder et al. (2017) discussed that difficulties in training result of the imaginary part of the eigenvalues of the Jacobian of the game vector field and Gidel et al. (2019b) mentioned that games have natural oscillatory behavior. This cyclic behavior has been explained in (Balduzzi et al., 2018) by a non- zero Hamiltonian component in the Helmholtz decomposition of the Jacobian of the game vector field. All these explanations are related to the spectral properties of this Jacobian. The goal of Proposition is to provide a formal definition to the notions of rotation and attraction we dealing with in this paper.
GANs may
In the following section, we introduce a new tool order to assess the magnitude of the rotation around L.SP compared to the attraction this point.

---

Published as a conference paper at ICLR 2020
4 VISUALIZATION FOR THE VECTOR FIELD LANDSCAPE
Neural networks are parametrized by a large number of variables and visualizations are only possible using low dimensional plots (1D or 2D). We first present standard visualization tool for deep neural network loss surfaces that we will exploit in $4.2.
4.1 STANDARD VISUALIZATIONS FOR THE LOSS SURFACE
One way to visualize neural network's 's loss landscape follow a parametrized path w())) that connects two parameters w, w (often one is chosen early in learning and another one is chosen late in learning, close solution). path is a continuous function w-) such that w(0) (11) = w' Goodfellow et al. (2015) considered linear path w(a) = aw + (1 - a'w'' More complex paths can be considered to assess whether different minima are connected (Draxler et al., 2018).
and
4.2 PROPOSED VISUALIZATION PATII-ANGLE
We propose to study the linear path between parameters early in learning and parameters late in learning. We illustrate the extreme cases for the game vector field along this path in simple examples in Figure 2(a-c): pure attraction occurs when the vector field perfectly points to the optimum (Fig. 2a) and pure rotation when the vector field is orthogonal to the direction the optimum (Fig. 2b). In practice, we expect the vector field to be in between these two extreme cases (Fig. 2c). In order to determine in which case we are, around: a LSSP, in practice, we propose the following tools.
Path-norm. We first ensure that we are in a neighborhood of stationary point by computing the norm of the vector field. Note that considering independently the norm of each player may be misleading: even though the gradient of one player may be close to zero, it does not mean that we are at stationary point since the other player might still be updating parameters.
Path-angle. Once we are close to a final point w', i.e., in neighborhood of a LSSP, we propose to look at the angle between the vector field (3) and the linear path from w to w'. Specifically, we monitor the cosine of this angle, quantity we call Path-angle:
ca)
-.) where va vlaw' (1 a)w), a [a, 6]. (wo -
Usually (a. b] [0, 1]. but since we are interested in the landscape around a LSSP, it might be more informative to also consider further extrapolated points around w' with b > 1.
Eigenvalues of the Jacobian. Another important tool to gain insights on the behavior close to a LSSP, as discussed in 53.2, is to look at the eigenvalues of Vv(w"). We propose to compute the top-k eigenvalues of this Jacobian. When all the eigenvalues have positive real parts, we conclude that we have reached LSSP, and if some eigenvalues have large imaginary parts, then the game has a strong rotational behavior (Thm. 1). Similarly, we can also compute the top-k eigenvalues of the diagonal blocks of the Jacobian, which correspond the Hessian of each player. These eigenvalues can inform us on whether we have converged to LSSP that is not a LNE.
An important advantage of the Path- angle relative to the computation of the eigenvalues of Vv(w"" is that it only requires computing gradients (and not second order derivatives, which may be prohibitively computationally expensive for deep networks). Also, it provides information along a whole path between two points and thus, more global information than the Jacobian computed at single point. In the following section, we use the Path-angle to study the archetypal behaviors presented Thm
4.3 ARCHETYPAL BEHAVIORS OF THE PATH-ANGLE AROUND LSSP
Around a LSSP, we have seen in (6) that the behavior of the vector field is mainly dictated by the Jacobian matrix Vo(w"). This motivates the study of the behavior of the Path-angle c(a) where the Jacobian is a constant matrix:
B S₂
S A
B
v(w)= =
(w w-)
and thus Vo(w) =
Ve.
(8)
A
S2

---

Published as a conference paper at ICLR 2020
0.75
0.05
Path 011210
00
0.50
101
0.5
-005
Attraction only
(b) Rotation only
Rotation and attraction
Figure 2: Above: game vector field (in grey) for different archetypal behaviors. The equilibrium of the game is (0, 0). Black arrows correspond to the directions of the vector field at different linear interpolations between two points: and *. Below: path-angle c(a:) for different archetypal behaviors (right y-axis, in blue). The left y-axis in orange correspond to the norm of the gradients. Notice the "bump" in path-angle (close to 1), characteristic of rotational dynamics.
Depending on the choice of S1, S2, A and B, we cover the following cases:
B eigenvalues are real. Thm. ensures that we only have attraction. Far from w", the gradient points to w" (See Fig. 2a) and thus c(a) for < 1 and c(a) -1 for a >> 1. Since w' not exactly w", we observe a quick sign switch of the Path-angle around
S1, S2 >
1. We plotted the average Path-angle over different approximate optima in Fig. 2a (see appendix for details).
T T eigenvalues are pure imaginary. Thm. ensures that we only have rotations. Far from the optimum the gradient is orthogonal to the direction that points to w (See Fig. 2b). Thus, c((x) vanishes for a & and a >> 1. Because w' not exactly w", around = 1, the gradient is tangent to the circles induced by the rotational dynamics and thus c(a) = +1. That is why in Fig. 2b we observe a bump in c(a: when a is close to 1.
S1 =
General high dimensional LSSP (4). The dynamics display both attraction and rotation. We observe combination of the sign switch due to the attraction and the bump due to the rotation. The higher the bump, the closer we are to pure rotations. Since we are performing low dimensional visualization, we actually project the gradient onto our direction of interest. That is why the Path-angle is significantly smaller than in Fig. 2c.
NUMERICAL RESULTS ON GANS
Losses. We focus on two common GAN loss formulations: we consider both the original non- saturating GAN (NSGAN) formulation proposed in Goodfellow et al. (2014) and the WGAN-GP objective described in Gulrajani al. (2017).
Datasets. We first propose to train GAN on a toy task composed of a 1D mixture of Gaussians (MoG) with 10,000 samples. For this task both the generator and discriminator are neural networks with 1 hidden layer and ReLU activations. We also train GAN MNIST, where DCGAN architecture (Radford et al., 2016) with spectral normalization(sec $C.2 for details), Finally also look the optimization landscape of state of the ResNet CIFAR10 (Krizhevsky and Hinton, 2009).
the
Optimization methods. For the mixture of Gaussian (MoG) dataset, used the full-batch extragra- dient method (Korpelevich. 1976; Gidel et al., 2019a). We also tried to use standard batch gradient descent, but this led to unstable results indicating that gradient descent might indeed be unable to
7

---

Published as a conference paper at ICLR 2020
Arga
NSGAN
Gradient
Utter
(a) MoG
(b) MNIST, IS = 8.97
(c) CIFARIO, IS = 7.33
Gret keem
6.2
Gral Amat
WGAN-GP
00000
1959 Parl
Liea Path
Leneor Fath
(d) MaG
(c) MNIST, 9.46
(f) CIFARIO, IS 7.65
Figure 3: Path-angle for NSGAN (top row) and WGAN-GP (bottom row) trained on the different datasets, see Appendix C.3 for details on how the path-angle is computed. For MoG the ending point generator which has learned the distribution. For MNIST and CIFAR10 indicate the Inception score (IS) at the ending point of the interpolation. Notice the "bump" in path-angle (close to a = 1.0), characteristic of games rotational dynamics, and absent the minimization problem (d). Details on error bars in SC.3.
SK
NSGAN
(wag mary
mmag Moryy
-35
26
50x 1000 356 20000 2500 XXiX
(a) MoG
(b) MNIST, IS =.97
(c) CIFARIO, IS = 7.33
Imsggnary hart
WGAN-GP
08
1500C 2000E Vilio E000 SCEC0S
Reall
Real Part
(d) MoG
(c) MNIST, IS 9.46
(1) CIFARIO, IS 7.65
Figure 4: Eigenvalues of the Jacobian of the game for NSGAN (top row) and WGAN-GP (bottom row) trained on the different datasets. Large imaginary eigenvalues are characteristic of rotational behavior. Notice that NSGAN and WGAN-GP objectives lead to very different landscapes (see how the eigenvalues of WGAN-GP are shifted to the right of the imaginary axis). This could explain the difference in performance between NSGAN and WGAN-GP.
converge to stable stationary points due the rotations (see SC.4). On MNIST and CIFARIO, we tested both Adam (Kingma and Ba, 2015) and ExtraAdam (Gidel et al., 2019a). The observations made on models trained with both methods are very similar. ExtraAdam gives slightly better perfor- mance in terms of inception score (Salimans et al., 2016), and Adam sometimes converge to unstable points, thus we decided to only include the observations on ExtraAdam, for more details on the observations Adam (see &C.5). As recommended by Heusel et al. (2017), chose different learning rates for the discriminator and the generator. All the parer--rrrmeters and precise details about the experiments can be found in SC.11
8

---

Published as a conference paper at ICLR 2020
5.1 EVIDENCE OF ROTATION AROUND LOCALLY STABLE STATIONARY POINTS IN GANS
We first look, for all the different models and datasets, at the path-angles between a random initial- ization (initial point) and the set of parameters during training achieving the best performance (end point) (Fig. 3), and at the eigenvalues of the Jacobian of the game vector field for the same end point (Fig. 4). We're mostly interested in looking at the optimization landscape around LSSPs, so we first check if we are actually close to one. To do so we look at the gradient norm around the end point, this is shown by the orange curves in Fig.3, we can see that the norm of the gradient is quite small for all the models meaning that point is stable, to do so we look at the eigenvalues of the Game in Fig. 4, if all the eigenvalues have positive real parts then the point is also stable. We observe that most of the time, the model has reached a LSSP. However we can see that this is not always the case, for example in Fig. 4d some of the eigenvalues have a negative real part. We still include those results since although the point is unstable it gives similar performance to a LSSP.
close to a stationary point. We also need to check that the
Our first observation is that all the GAN objectives on both datasets have non zero rotational component. This can be seen by looking at the Path-angle in Fig. 3, where we always observe bump, and this is also confirmed by the large imaginary part in the eigenvalues of the Jacobian in Fig. 4. The rotational component is clearly visible in Fig. 3d, where we see no sign switch and clear bump similar to Fig. 2b. On MNIST and CIFARIO, with NSGAN and WGAN-GP (see Fig. 3), we observe combination of a bump and a sign switch similar to Fig. 2c, Also Fig. 4 clearly shows the existence of imaginary eigenvalues with large magnitude. Fig. 4c and 4c. We can see that while almost all models exhibit rotations, the distribution of the eigenvalues the complex eigenvalues for NSGAN seems to be much more concentrated on the imaginary axis while WGAN-GP tends to spread the eigenvalues towards the right of the imaginary axis Fig. 4e. This shows that different GAN objectives can lead to very different landscapes, and has implications in terms of optimization, in particular that might explain why WGAN-GP performs slightly better than NSGAN.
very different. In particular
Generator
Mightals
To5-100 E Egenvalaes
22 Elgervalues
Too 20 Elprwallus
X55
Mogieaat
Discriminator
15
Top-100 Elgenwatues
75
90
Top Elgenvaluss
(a) MoG
(b) MNIST, IS 8.97
(c) CIFARIO, IS 7.33
Figure 5: NSGAN. Top k k-Eigenvalues of the Hessian of each player (in terms of magnitude) in descending order. Top Eigenvalues indicate that the Generator does not reach a local minimum but a saddle point (for CIFAR10 10 actually both the generator and discriminator are at saddle points). Thus the training algorithms converge to LSSPs which are not Nash equilibria.
5.2 THE LOCALLY STABLE STATIONARY POINTS OF GANS ARE NOT LOCAL NASH EQUILIBRIA
As mentioned at the beginning of $5.1, 1, the points we are considering are most of the times LSSP. To check if these points are also local Nash equilibria (LNE) we compute the eigenvalues of the Hessian of each player independently. If all the eigenvalues of each player are positive, it means that we have reached DNE. Since the computation of the full spectrum of the Hessians is expensive, we restrict ourselves to the top-k cigenvalues with largest magnitude: exhibiting one significant negative eigenvalue is enough to indicate that the point considered is not in the neighhorhood of

---

Published as a conference paper at ICLR 2020
R
Generator
Maggh burde
Negn
-23
15
Top-100 Egenvalues
Top Egenneluss
50000
Mogiitate
Discriminator
M4getuue
30000
30000
Tep-100 Exgervalues
Top 20 Eigeralies
(a) MoG
(b) MNIST, 9.46
(c) CIFARIO, IS 7.65
Figure 6: WGAN-GP. Top k-Eigenvalues of the Hessian of each player (in terms of magnitude) in descending order. Top Eigenvalues indicate that the Generator does not reach local minimum but saddle point. Thus the training algorithms converge to LSSPs which are not Nash equilibria.
LNE. Results are shown in Fig. 5 and Fig. 6, from which we make several observations. First, we that the generator never reaches local minimum but instead finds saddle point. This means that the algorithm converges to LSSP which is not LNE, while achieving good results with respect to our evaluation metrics. This raises the question whether convergence to a LNE is actually needed or if converging to LSSP is sufficient reach good solution. We also observe large difference in the eigenvalues of the discriminator when using the WGAN-GP v.s. the NSGAN objective. In particular, we find that the discriminator in NSGAN converges to solution with very large positive cigenvalues compared to WGAN-GP, This shows that the discriminator in NSGAN converges to much sharper minimum. This is consistent with the fact that the gradient penalty acts regularizer on the discriminator and prevents it from becoming too sharp.
DISCUSSION
Across different GAN formulations, standard optimization methods and datasets, we consistently observed that GANs do not converge to local Nash equilibria. Instead the generator often ends up being saddle point of the generator loss function. However, in practice, these LSSP achieve really good generator performance metrics, which leads us to question whether we need a Nash equilibrium get generator with good performance in GANs and whether such DNE with good performance does actually exist. Moreover, we have provided evidence that the optimization landscapes of GANs typically have rotational components specific to games. We argue that these rotational components are part of the reason why GANs are challenging to train, in particular that the instabilities observed during training may come from such rotations close LSSP. shows that simple low dimensional examples, such as for instance Dirac GAN, does capture some of the arising challenges for training large scale GANs, thus, motivating the practical use of method able to handle strong rotational components, such extragradient (Gidel et al., 2019a), averaging (Yazıcı al., 2019), optimism (Daskalakis et al., 2018) or gradient penalty based methods (Mescheder et al., 2017; Gulrajani et al., 2017).
ACKNOWLEDGGENNTS.
The contribution to this research by Mila, Université de Montréal authors was partially supported by the Canada CIFAR AI Chair Program (held at Mila), the Canada Excellence Research Chair in "Data by the NSERC Discovery Grant RGPIN-2017-06995 (held at Université de Montréal), by a Borealis AI fellowship and by a Google Focused Research award. The authors would like to thank Tatjana Chavdarova for fruitful discussions.
Science for Realtime Decision-makin
10

---

Published as a conference paper at ICLR 2020
REFERENCES
I. Adolphs, H. Daneshmand, A. Lucchi, and T. Hofmann. Local saddle point optimization: A curvature exploitation approach. arXiv, 2018.
G. Alain, N. Le Roux, and P.-A. Manzagol. Negative eigenvalues of the hessian in deep neural networks. arXiv, 2019.
M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein generative adversarial networks. In ICML, 2017.
D. Balduzzi, Racaniere, J. Martens, J. Foerster, K. Tuyls, and T. Graepel. The mechanics of n-player differentiable games. In ICML, 2018.
A. Brock, J. Donahue, and K. Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR, 2019.
A. Choromanska, M. Henaff, M. Mathieu, G. B. Arous, and Y. LeCun. The loss surfaces of multilayer networks. In Artificial Intelligence and Statistics, 2015.
C. Daskalakis, A. Ilyss, V. Syrgkanis, and H. Zeng. Training GANs with optimism. In ICLR, 2018.
Y. N. Dauphin, Pascanu, C. Gulcehre, Cho. S. Ganguli, and Y. Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In NeurlPS, 2014.
F. Draxler, K. Veschgini, M. Salmhofer, and F. Hamprecht. Essentially no barriers in neural network energy landscape. In ICMI., 2018.
G. Gidel, Berard, Vincent, and S. Lacoste-Julien. variational inequality perspective on generative adversarial nets. ICLR, 2019a.
G. Gidel, R. A. Hemmat, M. Pezeshki, G. Huang, R. Lepriol, S. Lacoste-Julien, and I. Mitliagkas. Negative momentum for improved game dynamics. In AISTATS, 2019b.
X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, 2010.
I. Goodfellow. Neurips 2016 tutorial:
Generative adversarial networks. arXiv:1701.00160, 2016.
I. Goodfellow, Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In NeurlPS, 2014.
I.J. Goodfellow, O. Vinyals, and A. M. Saxe. Qualitatively characterizing neural network optimization problems. In ICLR, 2015.
I. Gulrajani, F. Ahmed, M. Arjovsky, V, Dumoulin, and A. Courville. Improved training of wasserstein GANs. In NeurIPS, 2017.
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs trained by two time-scale update rule converge to a local nash equilibrium. In NeurIPS, 2017.
K. Kawaguchi. Deep learning without poor local minima. In NeurIPS, 2016.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
G. Korpelevich. The extragradient method for finding saddle points and other problems. Matecon, 1976.
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NewrIPS, 2012.
Y. LeCun, C. Cortes, and C. Burges. MNIST handwritten digit database. AT&T Labs [Online]. Available: http://yann. lecun. com/exdb/mmist, 2010.

---

Published as a conference paper at ICLR 2020
H. Li, Z. Xu, G. Taylor, C. Studer, and T. Goldstein. Visualizing the loss landscape of neural nets. In NeurIPS, 2018a.
J. Li, A. Madry, J. Peebles, and L. Schmidt. On the limitations of first order approximation in gan dynamics. In ICML, 2018b.
Mazumdar and J. Ratliff. On the convergence of gradient-based learning in continuous games. ArXiv, 2018.
E. V. Mazumdar, M. I. Jordan, and S. S. Sastry. On finding local nash equilibria (and only local nash equilibria) in zero-sum games. arXiv, 2019.
L. Mescheder, S. Nowozin, and A. Geiger. The numerics of GANs. In NeurlPS, 2017.
L. Mescheder, A. Geiger, and S. Nowozin. Which Training Methods for GANs do actually Converge? In ICML, 2018.
Miyato, Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. In ICLR, 2018.
V. Nagarajan and J. Z. Kolter. Gradient descent GAN optimization is locally stable. In NeurIPS, 2017.
W. Paul. Electromagnetic traps for charged and neutral particles. Reviews of modern physics, 1990.
B. A. Pearlmutter. Fast exact multiplication by the hessian. Neural computation, 1994.
A. Radford, I.. Metz, and S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR, 2016.
L. J. Ratliff, S. A. Burden, and S. Sastry. On the characterization of local nash equilibria in continuous games. In IEEE Transactions on Automatic Control, 2016.
L. Sagun, Bottou, and LeCun. Eigenvalues of the hessian in deep learning: Singularity and beyond. arXiv, 2016.
L. Sagun, U. Evci, V. U. Guney, Y. Dauphin, and L. Bottou. Empirical analysis of the hessian of over-parametrized neural networks. arXiv, 2017.
Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training GANs. In NeurlPS, 2016.
R. Thompson, T. Harmon, and M. Ball. The rotating-saddie trap: mechanical analogy rf-electric- quadrupole ion trapping? Canadian journal of physics, 2002.
F. Verhulst. Nonlinear differential equations and dynamical systems. Springer Science & Business Media, 1989.
Y. Yazıeı, C.-S. Foo, S. Winkler, K.-H. Yap, G. Piliouras, and V. Chandrasekhar. The unusual effectiveness of averaging in GAN training. In ICLR, 2019.
12

---

Published as a conference paper at ICLR 2020
A PROOF OF THEOREMS AND PROPOSITIONS
A. PROOF OF THEOREM 1
Let us recall the theorem of interest:
Proposition' 1. Let assume that (6) is equality and that Ve(w") is diagonalizable, then there exists a basis P such that the coordinates ((t) == P(w(t) w²) have the following behavior,
1. For Ay Sp Vo(w"), A5 R. we observe pure attraction:
3 5 - (=,))(;0000)))
55)(t)
W (
2. For A3 E Sp Vo(w"), R((A)) = 0, we observe pure rotation:
= Ragzt
[1+1
(,(() (1)
5,(0) [
3. Otherwise, we observe both:
Re(A;)t Rim(As)
The matrix R. corresponds rotation of angle ip Note that, re-ordered the eigenvalues such that the complex conjugate eigenvalues form pairs: x3 & R then As+1 Xg.
Proof. The ODE we consider is,
dw)(t) (w()))(((()) w*)
(9)
The solution of this ODE is
w t)) = (t- (w)(t) w") wt
(10)
Let us now consider \ an eigenvalue of Sp( Sp(Ve(w")) such that Re(A) > and Im(A) * Since Vo(w") is real matrix and Im(A) # we know that the complex conjugate À of À belongs to (P(()(W")) Let uig complex eigenvector of X, then we have that,
Volw")uo Auo
Ve(w") Jun Xuoo
(11)
and thus un is a eigenvector of λ. Now if we set in == uo + uo and iue == 240 To, we have that
(-vll( u = un U
+ Im(e-(A)e
(12)
etVolw") = - - 17 un = i(Re(e -1 Juz Im(e (A)a))))
(13)
Thus we consider the basis that diagonalizes Ve(w*) and modify the complex conjugate eigenval- ues in the way we described right after 11 we get the expected diagonal form in a real basis. Thus there exists such that
Vo(w²)- PDP-1
(14)
where D is the block diagonal matrix with the block described in Theorem 1.
A.2 BEING A DNE IS NEITHER NECESSARY OR SUFFICIENT FOR BEING A LSSP
Let us first recall Example
Example' 1. Let consider La hyperbolic paraboloid (a.k.a., saddle point function) centered in (1, 1) where (1, up) is the principal descent direction and (-ip, 1) is the principal ascent direction, while LD is simple bilinear objective.
La(01, 02 pp) = (02 -00 1)2 - 1 (01 + 402 - 1)² Lp(01,02. = (050, + 402 - -9)
We want show that (1, 1, 0) is locally stable stationary point.
Proof. The game vector field has the following form,
(2p2² 1)0, 3:002 + 24p
v 02. = (2 4² 02) 3,01 - 2 +
(15)
50, 402
13

---

Published as a conference paper at ICLR 2020
Thus, (0T, 05, x0" (1,1,0) is stationary point (i.e., v(07, 08 :0*) 0). The Jacobian of the game vector field
22p² 11
- 3p
302
V2(01,02.14))
-3ip
2 - x² 301
(16)
5
4
and thus,
1 -2
1o,00,02,99
0
2
(17)
We can verify that the eigenvalues of this matrix have positive real part with any solver (the eigenvalues of a 3 x 3 always have a closed form). For completeness we provide proof without using the closed form of the eigenvalues. The eigenvalues Vo(0i, 02 x0*) are given by the roots of its characteristic polynomial,
X
0
x())=
-
2 = X3 - X² 11X - 2.
(18)
This polynomial has real root in (0, because x(0) -2 < 0< 9 x(1.. Thus know that, there exists a E (0, 1) such that,
2 x2 11X - -x A1)(X -2))
(19)
Then we have the equalities,
ad1d =2
(20)
a \1 + A2 =1
(21)
Thus, since 0 a < L, we have that,
If \ and A2 are real, they have the same sign \1 -2/a 0) and thus are positive (A + X2 = > 0).
If X is complex then x2 Xı and thus, 28((1) A1 A2 a 0.
Example showed that LSSP did not imply DNE. Let construct an example where a game have DNE which not locally stable.
Example Consider the non-zero-sun game with the following respective losses for each player.
(1 () 40 +
1). -1) and ()) (40 - 1) 0
(22)
This game has two stationary points for and +1. The Jacobian of the dynamics these two points are
11 1/2 2 1/2
and Vu(0, -1) = (1 -1/2
Vo(0,
(23)
-1/2
Thus,
The stationary point (0, 1) is DNE but Sp(Ve 1)) = [3ty17]
contains an eigenvalue
with negative real part and so not a L.SSP.
The statioanry point (0, -1) is not a DNE but Sp(Vu 1)) = ( contains only eigenvalue with positive real part and is I.SPP
14

---

Published as a conference paper at ICLR 2020
B COMPUTATION OF THE TOP-K EIGENVALUES OF THE JACOBIAN
Neural networks usually have large number of parameters, this usually makes the storing of the full Jacobian matrix impossible. However the Jacobian vector product can be efficiently computed by using the trick from (Pearlmutter, 1994). Indeed it's easy to show that Vv(w)u ((v(w)* u).
To compute the eigenvalues of the Jacobian of the Game, we first compute the gradient v(w) over subset of the dataset. We then define a function that computes the Jacobian vector product using automatic differentiation. We can then use this function to compute the top-k cigenvalues of the Jacobian using the sparse www.alinni.... eigs functions of the Scipy library.
C EXPERIMENTAL DETAILS
C.I MIXTURE OF GAUSSIAN EXPERIMENT
Dataset. The Mixture of Gaussian dataset composed of 10,000 points sampled independently from the following distribution (2) = [2, 0.5) + (-,, where (H,, σ²) is the probability
density function of a ID-Gaussian distribution with mean f and variance o². The latent variables 2 E I2 are sampled from a standard Normal distribution (0, Ia). Because we want to use full-batch methods, we sample 10,000 points that we re-use for each iteration during training.
Neural Networks Architecture. Both the generator and discriminator are one hidden layer neural networks with 100 hidden units and ReLU activations.
WGAN Clipping. Because of the clipping of the discriminator parameters some components of the gradient of the discriminator's 's gradient should no be taken into account. In order to compute the relevant path angle we apply the following filter to the gradient:
1 {1p] = and V = signe)}
(24)
where is clipped between - and e. this condition holds for a coordinate of the gradient then mean that after gradient step followed by a clipping the value of the coordinate will not change.
Hyperparameters for WGAN-GP on MoG
Batch size
10, 000 (Full-Batch)
Number of iterations
= 330, 000
Learning rate for generator
= X 10-2
Learning rate for discriminator Gradient Penalty coefficient
=1 X 10
= 1 X 10-3
Hy Hyperparameters for NSGAN on MoG
Batch size
10, 000 (Full-Batch)
Number of iterations
30, 000
Learning rate for generator
1 X 10
Learning rate for discriminator
=1 10-1
C.2 MNIST EXPERIMENT
Dataset We use the training part of MNIST dataset LeCun al. (2010) (50K examples) for training our models, and scale each image to the range [-1,1] 1].
Architecture We use the DCGAN architecture Radford et al. (2016) for our generator and discrimi- nator, with both the NSGAN and WGAN-GP objectives. The only change we make is that we replace the Batch-norm layer in the discriminator with Spectral-norm layer Miyato et al. (2018), which find to stabilize training.
Training Details
15

---

Published as a conference paper at ICLR 2020
Hyperparameters for NSGAN with Adam
Batch size
=100
Number of iterations
000 000
Learning rate for generator
=2 10-1
Learning rate for discriminator B1
=5 x 10-5
= =0.5
Hyperparameters for NSGAN with ExtraAdam
Batch size
=100
Number of iterations
=100, 000
Learning rate for generator
=2 x 10-4
Leamingg rate for discriminator $1
10-5 5
0.9
Hyperparameters for WGAN-GP with Adam
Batch size
= =100
Number of iterations
=200, 000
Learning rate for generator Learning rate for discriminator
=8.6 X 10 10-5
=8.6 10-5
B1
0.5
Gradient penalty À Critic per Gen. iterations X
10
Hyperparameters for WGAN-GP with ExtraAdam
Batch size
=100
Number iterations
200, 000
Learning rate for generator Learning rate for discriminator
=8.6 10
= 8.6 x 10-5
31
0.9
Gradient penalty À Critic per Gen. iterations À
10
=
Computing Inception Score on MNIST We compute the inception score (IS) for our models using a LeNet classifier pretrained on MNIST, The average IS score of real MNIST data is 9.9.
C.3 PATH-ANGLE PLOT
We use the path-angle plot to illustrate the dynamics close to a LSSP. To compute this plot, we need to choose an initial point w and an end point w'. We choose the be the parameters at initialization, but w' can more subtle to choose. In practice, when we use stochastic gradient methods we typically reach a neighborhood of LSSP where the norm of the gradient is small. However, due to the stochastic noise, we keep moving around the LSSP. In order to be robust to the choice of the end point w', we take multiple close-by points during training that have good performance (e.g., high IS in MNIST). In all of figures, we compute the path-angle (and path-norm) for all these end points (with the same start point), and we plot the median path-angle (middle line) and interquartile range (shaded area).
C.4 INSTABILITY OF GRADIENT DESCENT
For the MoG dataset we tried both the extragradient method (Korpelevich, 1976: Gidel et al., 2019a) and the standard gradient descent. We observed that gradient descent leads to unstable results. In
16

---

Published as a conference paper at ICLR 2020
particular the norm of the gradient has very large variance compared to extragradient this is shown in Fig.
Gradien Descent
Extragradient
10
Gradient Norm
10-3
0.0
0.5
1.5
2.0
2.5
Nb of iterations
Figure 7: The norm of gradient during training for the standard GAN objective. We observe that while extra-gradient reaches low norm which indicates that has converged, the gradient descent the contrary doesn't seem to converge.
C.5 ADDITIONAL RESULTS WITH ADAM
0,005
0.004
Gradient Norm
30*
0.003
Path Angle
Gradlent Norm
Path Angle
0.002
201
0.001
30
0.000
10a
0.5
1.0
0.4
Linear Path
Linear Path
200
end
end
150
Imaginary Part
Imaginary Part
100
-10
-50
-20
-100
30
-150
-200
500
1000
1500
2000
-75
-50
-25
25
50
75
Real Part
Real Part
NSGAN on MNIST, IS: 8.95
(b) WGAN-GP on MNIST, IS: 9.30
Figure 8: Path-angle and Eigenvalues computed on MNIST with Adam.
17

---

Published as a conference paper at ICLR 2020
Inst
Gradient Norm
0.002
Imaginary Part
Angle
0,001
307
-0.001
-500
-0.002
-1000
0.5
0.7
1.1
-200
-100
100
200
Real Part
Figure 9: Path-angle and Eigenvalues for NSGAN CIFARIO computed CIFARIO with Adam. We that the model has eigenvalues with negative real part, this means that we actually reached unstable point.
18


Reviews:
[{"review_id": "S1xyowehYr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present a study of GAN dynamics in training with the goal of understanding whether rotational behavior occurs when training GANs on real world datasets, and whether training methods find local Nash equilibria. \n\n\nThe authors start by motivating the study of a game vector field with a toy example in which we can see all relevant behavior in the relevant directions. The work investigates the game vector field with a visualization technique called “path-angle” that attempts to alleviate the problem of high dimensionality by only looking at the cosine similarity (between the linear interpolation between the two points versus the true gradient at the point) along a path between two (concatenated) weight vectors at a time. The work also investigates by looking at the gradient norms of weights in these optimization trajectories.\n\n\nThe work uses these techniques to visualize the dynamics of GANs trained on standard datasets. The authors find that GANs do not converge to local Nash equilibria, that each player ends at a saddle point, and state evidence for “rotational behavior” in GAN dynamics.\n\n\nThe finding that GAN training methods do not find local Nash equilibria is interesting. However, it is unclear to me what the experiments presented show about GAN training dynamics (in particular, it is unclear to me what rotational dynamics are in the context of GANs and what consequences they have for training). I have listed more detailed feedback below.\n\n\nDetailed feedback:\n* Section 3.1: What is the formulation of Mescheder et al 2017? This should be explicitly stated in the paper.\n* The authors never explicitly, formally define what it means for there to be rotational behavior. However, this terminology is used frequently throughout the paper, particularly in the empirical section (5.1). What does rotational component mean in this Section?\n* What is the motivation for completing the path based landscape visualization methods between a random initialization and the final weight vector? Is there a reason why the actual iterates were not investigated with this method?\n* Why does the bump in Figure 3 imply that there is a non zero rotational component?\n* It would be good to complete a more thorough understanding of the spectra of hessians at convergence; the extent of the experiments in Figure 5 and 6 appears to just be 3 training runs; a larger sample size would be good to establish trends.\n* What is the motivation for visualizing via the path based methods? Why does interpolating between initialization and the final learned weight vector tell us about the rotational dynamics in high dimensions? This line may not be representative of the optimization trajectory followed by the actual iterates.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "HyeEo544tB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tries to provide a deeper understanding of the training dynamics of GANs in practice via characterizing and visualizing the rotation and attraction phenomena nearby a locally stable stationary point (LSSP) and questions the necessity to access a differential/local Nash equilibrium (LNE). In particular, this paper first discusses the difference between LSSP and LNE and formalize the notions of rotation and attraction around LSSP in games. Then, this paper proposes the path angle to visualize the rotation and attraction nearby an LSSP. The path angle is a function that maps linearly distributed points in the line, which is determined by an initial parameter set and a well-trained parameter set, to the angles between the line and the gradient of a given point in that line. The rotation and attraction phenomena can be observed in the plot of the path angle as  \"a quick sign switch\" and \"a bump\" nearby 1, respectively. The experiments empirically demonstrate that: 1. rotation exists in the training dynamics of practical GANs; 2. GANs often converge to an LSSP than an LNE, but still, achieve good results.\n\nGenerally, this paper is interesting and well-written. The contribution is clearly presented and the literature is well discussed. However, I have some questions to be clarified by the authors as follows.\n\n1. In Sec 3.2., this paper tries to motivate the readers to notice the difference between the LSSP and DNE by introducing Example 1. However, I notice that there is a gap that hasn't been presented clearly: Example 1 is a general game but does not correspond to a GAN, which is of the most interest in the paper. Besides, the generator loss at the optimum should be (theta_2 - 1)^2 - 1/2(theta_1 - 1)^2 instead of theta_2^2 - 1/2theta_1^2. \n\n2. For the rotation around LSSP, existing work, including Mescheder et al. (2018), Gidel et al. (2019b),  has a prior discussion. Besides, it is intuitive that an LSSP is not an LNE in practical GANs with high probability because finding a descent direction is easy given such a high-dimensional space. It is also possible to find a sharp descent direction nearby an LSSP because the norm of the gradient is averaged across all dimensions. It is good to formulate these observations in a precise way but it would be better to see further implications of the two observations. If so, the paper quality will be significantly improved. \n\n3. A minor thing is why (c) and (f) in Figure 3 and Figure 4 use different metrics, i.e. FID and IS, respectively? \n\nI also note that this paper has 10 pages and should be expected at a higher level than other accepted papers. Given all these conditions, I think I make it clear why I give a rating 6 currently. \n\nBy the way, I'm not absolutely confident about the comments because I didn't work on analyzing the dynamics of GANs. I'll appreciate it if my issues can be addressed or a potential misunderstanding can be corrected. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HkxNcJcqYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \n\nThis paper proposes visualization techniques for the optimization landscape in GANs. The primary tool presented in this paper is a quantity called path-angle, which looks at the angle between the game vector field and the linear path between a point away from a stationary point and a point near a stationary point. The paper present examples of the visualization for dynamics with pure attraction, pure rotation, and a mix of attraction and rotation. Along with this, the authors propose to look at the eigenvalues of the game Jacobian and the individual player Hessian’s to evaluate convergence in GANs. The paper presents application of the tools on GANs trained with NSGAN and WGAN-GP objectives on a mixture of Gaussians, MNIST, and CIFAR10. The primary observation is that the generator performance is good, but the algorithms converge to non-Nash stable attractors. Moreover, it is shown using the path-angle plots that GANs exhibit rotational behavior around stable points.\n\nReview: \n\nThere has been a lot of work in the past few years (and ongoing) on principled training approaches for GANs. The objective of the algorithms is typically to converge a differential Nash equilibrium and/or to reach a stable point of the dynamics quickly. In my view, this work fills some of the gap on the empirical side of things with respect to each goal. \n\nNotably, a main idea to speed up convergence in GANs is to change the gradient play dynamics so rotational components are neutralized. The path angle visualization provides a novel tool to evaluate the empirical ability of any dynamics proposed for GANs to cancel out rotational components. Since it is generally known that gradient play dynamics are susceptible to cycling, I would have been interesting in seeing the path angle plots for some recently proposed algorithms such as consensus, symplectic gradient adjustment, stable opponent shaping, local symplectic surgery, etc to see how they compare. This would have made the experiments using the path angle visualization stronger in my view. Nonetheless, the path angle tool is useful and I can foresee it being commonly used in the future. \n\nAside from neutralizing rotational components, dynamics have been proposed with the goal of avoiding non-Nash stable attractors and converging only to differential Nash equilibria. However, to my knowledge, there has not been much, if any, evaluation in GANs to see if the methods are in fact converging to Nash equilibria as theory may predict. While simple, I found it interesting to evaluate the eigenvalues of the relevant quantities at convergence. I am curious why the authors evaluate the top-k eigenvalues in terms of magnitude? The scipy package referenced in the appendix can compute the largest and smallest real eigenvalues, which is what it seems like you would want to evaluate the definiteness of the game Jacobian and the individual player Hessians. The most interesting empirical result in the paper to me was that it is common to converge to non-Nash stable attractors using standard training techniques and at such stable points the generator performance is strong. This is an important observation and  may cause some consideration of what points should be sought in GANs. I am not fully convinced this is always what the dynamics would always converge to depending on the network, learning rates, optimization methods, etc, but showing that it can be the case is useful. \n\nOverall, I think this paper introduces some useful tools to interpret the performance in GANs and to help understand the behavior of training dynamics. The main tool introduced was the path angle visualization and the primary empirical result was that standard GAN methods may reach non-Nash stable attractors and perform well. The paper probably be condensed in the first 4 pages, so that more experimental results could be presented and this would make the paper stronger.  \n\nPost Response: Thanks for the response. I believe this paper should be accepted.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is an interesting contribution that sheds some light on a well-studied but still poorly understood problem. I think it might be of interest to the community.
Decision: Accept (Poster)