Which Training Methods for GANs do actually Converge?

Lars Mescheder 1 Andreas Geiger 1 2 Sebastian Nowozin 3

Abstract
Recent work has shown local convergence of
GAN training for absolutely continuous data andgenerator distributions. In this paper, we showthat the requirement of absolute continuity is necessary: we describe a simple yet prototypicalcounterexample showing that in the more realistic case of distributions that are not absolutely
continuous, unregularized GAN training is notalways convergent. Furthermore, we discuss regularization strategies that were recently proposed
to stabilize GAN training. Our analysis showsthat GAN training with instance noise or zerocentered gradient penalties converges. On theother hand, we show that Wasserstein-GANs and
WGAN-GP with a finite number of discriminator
updates per generator update do not always converge to the equilibrium point. We discuss theseresults, leading us to a new explanation for thestability problems of GAN training. Based on
our analysis, we extend our convergence results
to more general GANs and prove local convergence for simplified gradient penalties even if thegenerator and data distributions lie on lower dimensional manifolds. We find these penalties towork well in practice and use them to learn highresolution generative image models for a variety
of datasets with little hyperparameter tuning.

1. Introduction
Generative Adversarial Networks (GANs) (Goodfellow
et al., 2014) are powerful latent variable models that can beused to learn complex real-world distributions. Especiallyfor images, GANs have emerged as one of the dominant
approaches for generating new realistically looking samples
after the model has been trained on some dataset.
1

MPI Tübingen, Germany 2 ETH Zürich, Switzerland
Microsoft Research, Cambridge, UK. Correspondence to: Lars
Mescheder <lars.mescheder@tue.mpg.de>.
3

Proceedings of the 35 th International Conference on Machine
Learning, Stockholm, Sweden, PMLR 80, 2018. Copyright 2018
by the author(s).

Methodunregularized (Goodfellow et al., 2014)
WGAN (Arjovsky et al., 2017)
WGAN-GP (Gulrajani et al., 2017)
DRAGAN (Kodali et al., 2017)
Instance noise (Sønderby et al., 2016)
ConOpt (Mescheder et al., 2017)
Gradient penalties (Roth et al., 2017)
Gradient penalty on real data only
Gradient penalty on fake data only

Local
convergence(a.c. case)Local
convergence(general case)

3
7
7
3
3
3
3
3
3

7
7
7
7
3
3
3
3
3

Table 1. Convergence properties of different GAN training algorithms for general GAN-architectures. Here, we distinguish between the case where both the data and generator distributions are
absolutely continuous (a.c.) and the general case where they may
lie on lower dimensional manifolds.

However, while very powerful, GANs can be hard to train
and in practice it is often observed that gradient descentbased GAN optimization does not lead to convergence. As
a result, a lot of recent research has focused on finding
better training algorithms (Arjovsky et al., 2017; Gulrajani
et al., 2017; Kodali et al., 2017; Sønderby et al., 2016; Rothet al., 2017) for GANs as well as gaining better theoreticallyunderstanding of their training dynamics (Arjovsky et al.,2017; Arjovsky & Bottou, 2017; Mescheder et al., 2017;
Nagarajan & Kolter, 2017; Heusel et al., 2017).
Despite practical advances, the training dynamics of GANs
are still not completely understood. Recently, Mescheder
et al. (2017) and Nagarajan & Kolter (2017) showed that
local convergence and stability properties of GAN training can be analyzed by examining the eigenvalues of theJacobian of the the associated gradient vector field: if theJacobian has only eigenvalues with negative real-part at theequilibrium point, GAN training converges locally for smallenough learning rates. On the other hand, if the Jacobian has
eigenvalues on the imaginary axis, it is generally not locallyconvergent. Moreover, Mescheder et al. (2017) showed that
if there are eigenvalues close but not on the imaginary axis,
the training algorithm can require intractably small learning
rates to achieve convergence. While Mescheder et al. (2017)
observed eigenvalues close to the imaginary axis in practice,
this observation does not answer the question if eigenvalues
close to the imaginary axis are a general phenomenon and if
yes, whether they are indeed the root cause for the training

Which Training Methods for GANs do actually Converge?

instabilities that people observe in practice.

2. Instabilities in GAN training

A partial answer to this question was given by Nagarajan& Kolter (2017), who showed that for absolutely continuous data and generator distributions1 all eigenvalues of theJacobian have negative real-part. As a result, GANs are locally convergent for small enough learning rates in this case.
However, the assumption of absolute continuity is not true
for common use cases of GANs, where both distributions
may lie on lower dimensional manifolds (Sønderby et al.,2016; Arjovsky & Bottou, 2017).

2.1. BackgroundIn this paper we show that this assumption is indeed necessary: by considering a simple yet prototypical exampleof GAN training we analytically show that (unregularized)
GAN training is not always locally convergent. We also
discuss how recent techniques for stabilizing GAN training affect local convergence on our example problem. Ourfindings show that neither Wasserstein GANs (WGANs) (Arjovsky et al., 2017) nor Wasserstein GANs with GradientPenalty (WGAN-GP) (Gulrajani et al., 2017) nor DRAGAN(Kodali et al., 2017) converge on this simple example for a
fixed number of discriminator updates per generator update.
On the other hand, we show that instance noise (Sønderby
et al., 2016; Arjovsky & Bottou, 2017), zero-centered gradient penalties (Roth et al., 2017) and consensus optimization(Mescheder et al., 2017) lead to local convergence.
Based on our analysis, we give a new explanation for the
instabilities commonly observed when training GANs basedon discriminator gradients orthogonal to the tangent spaceof the data manifold. We also introduce simplified gradientpenalties for which we prove local convergence. We findthat these gradient penalties work well in practice, allowing
us to learn high-resolution image based generative models
for a variety of datasets with little hyperparameter tuning.
In summary, our contributions are as follows:• We identify a simple yet prototypical counterexample
showing that (unregularized) gradient descent based
GAN optimization is not always locally convergent• We discuss if and how recently introduced regularization techniques stabilize the training• We introduce simplified gradient penalties and prove
local convergence for the regularized GAN training
dynamics
All proofs can be found in the supplementary material.
1
Nagarajan & Kolter (2017) also proved local convergence fora slightly more general family of probability distributions where
the support of the generator distribution is equal to the support of
the true data distribution near the equilibrium point. Alternatively,
they showed that their results also hold when the discriminatorsatisfies certain (strong) smoothness conditions. However, these
conditions are usually hard to satisfy in practice without priorknowledge about the support of the true data distribution.

GANs are defined by a min-max two-player game between
a discriminative network Dψ (x) and generative networkGθ (z). While the discriminator tries to distinguish between
real data point and data points produced by the generator,
the generator tries to fool the discriminator. It can be shown(Goodfellow et al., 2014) that if both the generator anddiscriminator are powerful enough to approximate any realvalued function, the unique Nash-equilibrium of this twoplayer game is given by a generator that produces the true
data distribution and a discriminator which is 0 everywhereon the data distribution.
Following the notation of Nagarajan & Kolter (2017), thetraining objective for the two players can be described byan objective function of the formL(θ, ψ) = Ep(z) [f (Dψ (Gθ (z)))]+ EpD (x) [f (−Dψ (x))] (1)for some real-valued function f . The common choicef (t) = − log(1 + exp(−t)) leads to the loss function considered in the original GAN paper (Goodfellow et al., 2014).
For technical reasons we assume that f is continuously differentiable and satisfies f 0 (t) 6= 0 for all t ∈ R.
The goal of the generator is to minimize this loss whereas
the discriminator tries to maximize it. Our goal when training GANs is to find a Nash-equilibrium, i.e. a parameter
assignment (θ∗ , ψ ∗ ) where neither the discriminator nor thegenerator can improve their utilities unilaterally.
GANs are usually trained using Simultaneous or Alternating
Gradient Descent (SimGD and AltGD). Both algorithms
can be described as fixed point algorithms (Mescheder et al.,2017) that apply some operator Fh (θ, ψ) to the parametervalues (θ, ψ) of the generator and discriminator, respectively.
For example, simultaneous gradient descent corresponds tothe operator Fh (θ, ψ) = (θ, ψ) + h v(θ, ψ), where v(θ, ψ)denotes the gradient vector field−∇θ L(θ, ψ)v(θ, ψ) :=.
(2)∇ψ L(θ, ψ)
Similarly, alternating gradient descent can be described by
an operator Fh = F2,h ◦ F1,h where F1,h and F2,h perform
an update for the generator and discriminator, respectively.
Recently, it was shown (Mescheder et al., 2017) that local
convergence of GAN training near an equilibrium point(θ∗ , ψ ∗ ) can be analyzed by looking at the spectrum of theJacobian Fh0 (θ∗ , ψ ∗ ) at the equilibrium: if Fh0 (θ∗ , ψ ∗ ) has
eigenvalues with absolute value bigger than 1 , the training
algorithm will generally not converge to (θ∗ , ψ ∗ ). On theother hand, if all eigenvalues have absolute value smaller

Which Training Methods for GANs do actually Converge?
yyDefinition 2.1. The Dirac-GAN consists of a (univariate)
generator distribution pθ = δθ and a linear discriminatorDψ (x) = ψ · x. The true data distribution pD is given by a
Dirac-distribution concentrated at 0.

Dψ (x)
p D = δ0pθ = δθ

pD = δ0p θ = δθDψ (x)xx(a) t = t0(b) t = t1

Figure 1. Visualization of the counterexample showing that gradient descent based GAN optimization is not always convergent:(a) In the beginning, the discriminator pushes the generator towards
the true data distribution and the discriminator’s slope increases.
(b) When the generator reaches the target distribution, the slope ofthe discriminator is largest, pushing the generator away from thetarget distribution. This results in oscillatory training dynamics
that never converge.

than 1, the training algorithm will converge to (θ∗ , ψ ∗ ) withlinear rate O(|λmax |k ) where λmax is the eigenvalue ofF 0 (θ∗ , ψ ∗ ) with the biggest absolute value. If all eigenvalues of F 0 (θ∗ , ψ ∗ ) are on the unit circle, the algorithm canbe convergent, divergent or neither, but if it is convergent
it will generally converge with a sublinear rate. A similar
result (Khalil, 1996; Nagarajan & Kolter, 2017) also holds
for the (idealized) continuous system

 

θ̇(t)
−∇ψ L(θ, ψ)=(3)∇θ L(θ, ψ)ψ̇(t)which corresponds to training the GAN with infinitely small
learning rate: if all eigenvalues of the Jacobian v 0 (θ∗ , ψ ∗ )at a stationary point (θ∗ , ψ ∗ ) have negative real-part, thecontinuous system converges locally to (θ∗ , ψ ∗ ) with linear convergence rate. On the other hand, if v 0 (θ∗ , ψ ∗ ) has
eigenvalues with positive real-part, the continuous system
is not locally convergent. If all eigenvalues have zero realpart, it can be convergent, divergent or neither, but if it isconvergent, it will generally converge with a sublinear rate.
For simultaneous gradient descent linear convergence canbe achieved if and only if all eigenvalues of the Jacobianof the gradient vector field v(θ, ψ) have negative real part
(Mescheder et al., 2017). This situation was also consideredby Nagarajan & Kolter (2017) who examined the asymptotic
case of step sizes h that go to 0 and proved local convergence
for absolutely continuous generator and data distributions
under certain regularity assumptions.

Note that for the Dirac-GAN, both the generator and thediscriminator have exactly one parameter. This situation
is visualized in Figure 1. In this setup, the GAN training
objective (1) is given byL(θ, ψ) = f (ψθ) + f (0)(4)

While using linear discriminators might appear restrictive,
the class of linear discriminators is in fact as powerful as
the class of all real-valued functions for this example: when
we use f (t) = − log(1 + exp(−t)) and we take the supremum over ψ in (4), we obtain (up to scalar and additive
constants) the Jensen-Shannon divergence between pθ andpD . The same holds true for the Wasserstein-divergence,
when we use f (t) = t and put a Lipschitz constraint on thediscriminator (see Section 3.1).
We show that the training dynamics of GANs do not converge in this simple setup.
Lemma 2.2. The unique equilibrium point of the training
objective in (4) is given by θ = ψ = 0. Moreover, theJacobian of the gradient vector field at the equilibrium point
has the two eigenvalues ±f 0 (0) i which are both on theimaginary axis.
We now take a closer look at the training dynamics producedby various algorithms for training the Dirac-GAN. First, weconsider the (idealized) continuous system in (3): while
Lemma 2.2 shows that the continuous system is generally
not linearly convergent to the equilibrium point, it couldin principle converge with a sublinear convergence rate.
However, this is not the case as the next lemma shows:
Lemma 2.3. The integral curves of the gradient vector fieldv(θ, ψ) do not converge to the Nash-equilibrium. Morespecifically, every integral curve (θ(t), ψ(t)) of the gradientvector field v(θ, ψ) satisfies θ(t)2 + ψ(t)2 = const for allt ∈ [0, ∞).

Simple experiments, simple theorems are the building
blocks that help us understand more complicated systems.
Ali Rahimi - Test of Time Award speech, NIPS 2017

Note that our results do not contradict the results of Nagarajan & Kolter (2017) and Heusel et al. (2017): our example
violates Assumption IV in Nagarajan & Kolter (2017) thatthe support of the generator distribution is equal to the support of the true data distribution near the equilibrium. It
also violates the assumption2 in Heusel et al. (2017) thatthe optimal discriminator parameter vector is a continuousfunction of the current generator parameters. In fact, unlessIn this section, we describe a simple yet prototypical counterexample which shows that in the general case unregularized GAN training is neither locally nor globally convergent.

2This assumption is usually even violated by WassersteinGANs, as the optimal discriminator parameter vector as a functionof the current generator parameters can have discontinuities near
the Nash-equilibrium. See Section 3.1 for details.

2.2. The Dirac-GAN

Which Training Methods for GANs do actually Converge?

2.3. Where do instabilities come from?
Our simple example shows that naive gradient based GAN
optimization does not always converge to the equilibrium
point. To get a better understanding of what can go wrong
for more complicated GANs, it is instructive to analyze
these instabilities in depth for this simple example problem.
(a) SimGD(b) AltGD

Figure 2. Training behavior of the Dirac-GAN. The starting iterate
is marked in red.

θ = 0, there is not even an optimal discriminator parameterfor the Dirac-GAN. Indeed, we found that two-time scale
updates as suggested by Heusel et al. (2017) do not help convergence towards the Nash-equilibrium (see Figure 22 in the
supplementary material). However, our example seems to
be a prototypical situation for (unregularized) GAN training
which usually deals with distributions that are concentrated
on lower dimensional manifolds (Arjovsky & Bottou, 2017).
We now take a closer look at the discretized system.
Lemma 2.4. For simultaneous gradient descent, the Jacobian of the update operator Fh (θ, ψ) has
p eigenvalues
λ1/2 = 1 ± hf 0 (0)i with absolute values 1 + h2 f 0 (0)2
at the Nash-equilibrium. Independently of the learning rate,
simultaneous gradient descent is therefore not stable near
the equilibrium. Even stronger, for every initial conditionand learning rate h > 0, the norm of the iterates (θk , ψk )obtained by simultaneous gradient descent is monotonicallyincreasing.
The behavior of simultaneous gradient descent for our example problem is visualized in Figure 2a.
Similarly, for alternating gradient descent we have
Lemma 2.5. For alternating gradient descent with ng generator and nd discriminator updates, the Jacobian of the
update operator Fh (θ, ψ) has eigenvalues
α2
λ1/2 = 1 −
±
2

s

α2
1−
2

2
− 1.

(5)

√
with α := ng nd hf 0 (0). For α ≤ 2, all eigenvalues arehence on the unit circle. Moreover for α > 2, there are
eigenvalues outside the unit circle.
Even though Lemma 2.5 shows that alternating gradientdescent does not converge linearly to the Nash-equilibrium,
it could in principle converge with a sublinear convergencerate. However, this is very unlikely because – as Lemma 2.3
shows – even the continuous system does not converge. Indeed, we empirically found that alternating gradient descentoscillates in stable cycles around the equilibrium and showsno sign of convergence (Figure 2b).

To understand the instabilities, we have to take a closer
look at the oscillatory behavior that GANs exhibit both for
the Dirac-GAN and for more complex systems. An intuitive explanation for the oscillations is given in Figure 1:
when the generator is far from the true data distribution,
the discriminator pushes the generator towards the true data
distribution. At the same time, the discriminator becomes
more certain, which increases the discriminator’s slope (Figure 1a). Now, when the generator reaches the target distribution (Figure 1b), the slope of the discriminator is largest,pushing the generator away from the target distribution. As
a result, the generator moves away again from the true data
distribution and the discriminator has to change its slope
from positive to negative. After a while, we end up with asimilar situation as in the beginning of training, only on theother side of the true data distribution. This process repeats
indefinitely and does not converge.
Another way to look at this is to consider the local behaviorof the training algorithm near the Nash-equilibrium. Indeed,near the Nash-equilibrium, there is nothing that pushes thediscriminator towards having zero slope on the true data
distribution. Even if the generator is initialized exactly on
the target distribution, there is no incentive for the discriminator to move to the equilibrium discriminator. As a result,training is unstable near the equilibrium point.
This phenomenon of discriminator gradients orthogonal tothe data distribution can also arise for more complex examples: as long as the data distribution is concentrated on alow dimensional manifold and the class of discriminators
is big enough, there is no incentive for the discriminator to
produce zero gradients orthogonal to the tangent space ofthe data manifold and hence converge to the equilibrium
discriminator. Even if the generator produces exactly thetrue data distribution, there is no incentive for the discriminator to produce zero gradients orthogonal to the tangent
space. When this happens, the discriminator does not provide useful gradients for the generator orthogonal to the data
distribution and the generator does not converge.
Note that these instabilities can only arise if the true data
distribution is concentrated on a lower dimensional manifold. Indeed, Nagarajan & Kolter (2017) showed that under some suitable assumptions - gradient descent based
GAN optimization is locally convergent for absolutely continuous distributions. Unfortunately, this assumption may
not be satisfied for data distributions like natural images toWhich Training Methods for GANs do actually Converge?

posed in the original GAN paper (Goodfellow et al., 2014)
leads to convergence of the continuous system, albeit withan extremely slow convergence rate. A more detailed discussion and an analysis of consensus optimization (Mescheder
et al., 2017) can be found in the supplementary material.
(b) Non-saturating GAN

3.1. Wasserstein GAN(c) WGAN (nd = 5)(d) WGAN-GP (nd = 5)

The two-player GAN game can be interpreted as minimizing
a probabilistic divergence between the true data distribution
and the distribution produced by the generator (Nowozin
et al., 2016; Goodfellow et al., 2014). This divergence isobtained by considering the best-response strategy for thediscriminator, resulting in an objective function that only
contains the generator parameters. Many recent regularization techniques for GANs are based on the observation(Arjovsky & Bottou, 2017) that this divergence may be discontinuous with respect to the parameters of the generator
or may even take on infinite values if the support of the data
distribution and the generator distribution do not match.

(e) Consensus optimization(f) Instance noise(g) Gradient penalty(h) Gradient penalty (CR)(a) Standard GAN

Figure 3. Convergence properties of different GAN training algorithms using alternating gradient descent with recommended
number of discriminator updates per generator update (nd = 1
if not noted otherwise). The shaded area in Figure 3c visualizes
the set of forbidden values for the discriminator parameter ψ. The
starting iterate is marked in red.

which GANs are commonly applied (Arjovsky & Bottou,2017). Moreover, even if the data distribution is absolutely
continuous but concentrated along some lower dimensional
manifold, the eigenvalues of the Jacobian of the gradientvector field will be very close to the imaginary axis, resulting in a highly ill-conditioned problem. This was observed
by Mescheder et al. (2017) who examined the spectrumof the Jacobian for a data distribution given by a circular
mixture of Gaussians with small variance.

To make the divergence continuous with respect to the parameters of the generator, Wasserstein GANs (WGANs)
Arjovsky et al. (2017) replace the Jensen-Shannon divergence used in the original derivation of GANs (Goodfellow
et al., 2014) with the Wasserstein-divergence. As a result,
Arjovsky et al. (2017) propose to use f (t) = t and restrict
the class of discriminators to Lipschitz continuous functions
with Lipschitz constant equal to some g0 > 0. While a
WGAN converges if the discriminator is always trained until convergence, in practice WGANs are usually trained byrunning only a fixed finite number of discriminator updates
per generator update. However, near the Nash-equilibrium
the optimal discriminator parameters can have a discontinuity as a function of the current generator parameters: forthe Dirac-GAN, the optimal discriminator has to move fromψ = −1 to ψ = 1 when θ changes signs. As the gradients
get smaller near the equilibrium point, the gradient updates
do not lead to convergence for the discriminator. Overall,
the training dynamics are again determined by the Jacobianof the gradient vector field near the Nash-equilibrium:
Lemma 3.1. A WGAN trained with simultaneous or alternating gradient descent with a fixed number of discriminator updates per generator update and a fixed learning rateh > 0 does generally not converge to the Nash equilibrium
for the Dirac-GAN.

3. Regularization strategies
As we have seen in Section 2, unregularized GAN training
does not always converge to the Nash-equilibrium. In thissection, we discuss how several regularization techniquesthat have recently been proposed, influence convergence of
the Dirac-GAN.
Interestingly, we also find that the non-saturating loss pro-

The training behavior of the WGAN is visualized in Figure 3c. We stress that this analysis only holds if the discriminator is trained with a fixed number of discriminator updates(as it is usually done in practice). More careful training that
ensures that the discriminator is kept exactly optimal or
two-timescale training (Heusel et al., 2017) might be able
to ensure convergence for WGANs.

Which Training Methods for GANs do actually Converge?

yDψ (x)x(a) Example with instance noise(b) Eigenvalues

Figure 4. Dirac-GAN with instance noise. While unregularized
GAN training is inherently unstable, instance noise can stabilize it:(a) Near the Nash-equilibrium, the discriminator is pushed towards
the zero discriminator. (b) As we increase the noise level σ from0 to σcritical , the real part of the eigenvalues at the equilibrium
point becomes negative and the absolute value of the imaginary
part becomes smaller. For noise levels bigger than σcritical all
eigenvalues are real-valued and GAN training hence behaves likea normal optimization problem.

The convergence properties of WGANs were also considered by Nagarajan & Kolter (2017) who showed that evenfor absolutely continuous densities and infinitesimal learning rates, WGANs are not always locally convergent.
We also found that WGAN-GP (Gulrajani et al., 2017) does
not converge for the Dirac-GAN (Figure 3d). Please see thesupplementary material for details.3
3.2. Instance noise
A common technique to stabilize GANs is to add instancenoise (Sønderby et al., 2016; Arjovsky & Bottou, 2017), i.e.
independent Gaussian noise, to the data points. While theoriginal motivation was to make the probabilistic divergence
between data and generator distribution well-defined for distributions that do not have common support, this does notclarify the effects of instance noise on the training algorithm
itself and its ability to find a Nash-equilibrium. Interestingly,
however, it was recently shown (Nagarajan & Kolter, 2017)that in the case of absolutely continuous distributions, gradient descent based GAN optimization is - under suitable
assumptions - locally convergent.
Indeed, for the Dirac-GAN we have:
Lemma 3.2. When using Gaussian instance noise with standard deviation σ, the eigenvalues of the Jacobian of thegradient vector field are given byp
λ1/2 = f 00 (0)σ 2 ± f 00 (0)2 σ 4 − f 0 (0)2 .
(6)
In particular, all eigenvalues of the Jacobian have negative
real-part at the Nash-equilibrium if f 00 (0) < 0 and σ > 0.
Hence, simultaneous and alternating gradient descent areboth locally convergent for small enough learning rates.
3
Despite these negative results, WGAN-GP has been successfully applied in practice (Gulrajani et al., 2017; Karras et al., 2017)
and we leave a theoretical analysis of these empirical results tofuture research.

Interestingly, Lemma 3.2 shows that there is a critical noise2level given by σcritical= |f 0 (0)|/|f 00 (0)|. If the noise levelis smaller than the critical noise level, the eigenvalues of
the Jacobian have non-zero imaginary part which results
in a rotational component in the gradient vector field near
the equilibrium point. If the noise level is larger than thecritical noise level, all eigenvalues of the Jacobian becomereal-valued and the rotational component in the gradientvector field disappears. The optimization problem is best
behaved when we select σ = σcritical : in this case we caneven achieve quadratic convergence for h = |f 0 (0)|−1 . The
effect of instance noise on the eigenvalues is visualized in
Figure 4b, which shows the traces of the two eigenvalues as
we increase σ from 0 to 2σcritical .
Figure 3f shows the training behavior of the GAN withinstance noise, showing that instance noise indeed creates astrong radial component in the gradient vector field whichmakes the training algorithm converge.
3.3. Zero-centered gradient penalties
Motivated by the success of instance noise to make the f divergence between two distributions well-defined, Rothet al. (2017) derived a local approximation to instance noisethat results in a zero-centered4 gradient penalty for the discriminator.
For the Dirac-GAN, a penalty on the squared norm of thegradients of the discriminator (no matter where) results in
the regularizerγ
R(ψ) = ψ 2 .
(7)
2This regularizer does not include the weighting terms considered by Roth et al. (2017). However, the same analysis
can also be applied to the regularizer with the additional
weighting, yielding almost exactly the same results (seeSection D.2 of the supplementary material).
Lemma 3.3. The eigenvalues of the Jacobian of the gradient vector field for the gradient-regularized Dirac-GAN at
the equilibrium point are given byrγγ2
λ1/2 = − ±− f 0 (0)2 .
(8)
2
4
In particular, for γ > 0 all eigenvalues have negative realpart. Hence, simultaneous and alternating gradient descentare both locally convergent for small enough learning rates.
Like for instance noise, there is a critical regularization parameter γcritical = 2|f 0 (0)| that results in a locally rotation
free vector field. A visualization of the training behavior of
the Dirac-GAN with gradient penalty is shown in Figure 3g.
Figure 3h illustrates the training behavior of the GAN with
4
In contrast to the gradient regularizers used in WGAN-GP andDRAGAN which are not zero-centered.

Which Training Methods for GANs do actually Converge?

gradient penalty and critical regularization (CR). In particular, we see that near the Nash-equilibrium the vector field
does not have a rotational component anymore and hence
behaves like a normal optimization problem.

4.2. Convergence

4. General convergence results

Let (θ∗ , ψ ∗ ) denote an equilibrium point of the regularizedtraining dynamics. In our convergence analysis, we consider
the realizable case, i.e. we assume that there are generator
parameters that make the generator produce the true data
distribution:

In Section 3 we analyzed the convergence properties of various regularization strategies for the Dirac-GAN. In thissection, we consider general GANs. First, we introduce two
simplified versions of the zero-centered gradient penalty
proposed by Roth et al. (2017). We then show that these gradient penalties allow us to extend the convergence proof by
Nagarajan & Kolter (2017) to the case where the generator
and data distribution do not locally have the same support.5
As a result, our convergence proof for the regularized training dynamics also holds for the more realistic case where
both the generator and data distributions may lie on lower
dimensional manifolds.
4.1. Simplified gradient penalties
Our analysis suggests that the main effect of the zerocentered gradient penalties proposed by Roth et al. (2017)
on local stability is to penalize the discriminator for deviating from the Nash-equilibrium. The simplest way to achieve
this is to penalize the gradient on real data alone: when thegenerator distribution produces the true data distribution
and the discriminator is equal to 0 on the data manifold, thegradient penalty ensures that the discriminator cannot createa non-zero gradient orthogonal to the data manifold without
suffering a loss in the GAN game.
This leads to the following regularization term:
R1 (ψ) :=γ
EpD (x) k∇Dψ (x)k2 .
2(9)In this section we present convergence results for the regularized GAN-training dynamics for both regularization terms
R1 (ψ) and R2 (ψ) under some suitable assumptions.6

Assumption I. We have pθ∗ = pD and Dψ∗ (x) = 0 in
some local neighborhood of supp pD .
Like Nagarajan & Kolter (2017), we assume that f satisfies
the following property:
Assumption II. We have f 0 (0) 6= 0 and f 00 (0) < 0.
An extension of our convergence proof to f (t) = t (as in
WGANs) can be found in the supplementary material.
The convergence proof is complicated by the fact that forneural networks, there generally is not a single equilibrium
point (θ∗ , ψ ∗ ), but a submanifold of equivalent equilibria
corresponding to different parameterizations of the samefunction. We therefore define the reparameterization manifolds MG and MD . To this end, leth(ψ) := EpD (x) |Dψ (x)|2 + k∇x Dψ (x)k2 .
(11)
The reparameterization manifolds are then defined as
MG := {θ | pθ = pD }MD := {ψ | h(ψ) = 0}. (12)To prove local convergence, we have to assume some regularity properties for MG and MD near the equilibrium
point. To state these assumptions, we needg(θ) := Epθ (x) [∇ψ Dψ (x)|ψ=ψ∗ ] .

Note that this regularizer is a simplified version of to theregularizer derived by Roth et al. (2017). However, ourregularizer does not contain the additional weighting terms
and penalizes the discriminator gradients only on the true
data distribution.
We also consider a similar regularization term given byR2 (θ, ψ) :=γEpθ (x) k∇Dψ (x)k22(10)

where we penalize the discriminator gradients on the current
generator distribution instead of the true data distribution.
Note that for the Dirac-GAN from Section 2, both regularizers reduce to the gradient penalty from Section 3.3 whose
behavior is visualized in Figure 3g and Figure 3h.
5Assumption IV in Nagarajan & Kolter (2017)(13)

Assumption III. There are -balls B (θ∗ ) and B (ψ ∗ )around θ∗ and ψ ∗ so that MG ∩ B (θ∗ ) and MD ∩ B (ψ ∗ )define C 1 - manifolds. Moreover, the following holds:(i) if v ∈ Rn is not in the tangent space of MD at ψ ∗ ,then ∂v2 h(ψ ∗ ) 6= 0.
(ii) if w ∈ Rm is not in the tangent space of MG at θ∗ ,then ∂w g(θ∗ ) 6= 0.
While formally similar, the two conditions in Assumption III
have very different meanings: the first condition is a simple
regularity property that means that the geometry of MD
can be locally described by the second derivative of h. The
second condition implies that the discriminator is strong6

Our results also hold for any convex combination of R1 andR2 and the regularizer with the additional weighting terms derived
by Roth et al. (2017). See the supplementary material for details.

Which Training Methods for GANs do actually Converge?

enough so that it can detect any deviation from the equilibrium generator distribution. Indeed, this is the only point
where we assume that the class of representable discriminators is sufficiently expressive (and excludes, for example,
the trivial case Dψ = 0 for all ψ).
We are now ready to state our main convergence result. To
this end, consider the regularized gradient vector field−∇θ L(θ, ψ)ṽi (θ, ψ) :=.
(14)∇ψ L(θ, ψ) − ∇ψ Ri (θ, ψ)
Theorem 4.1. Assume Assumption I, II and III hold for(θ∗ , ψ ∗ ). For small enough learning rates, simultaneousand alternating gradient descent for ṽ1 and ṽ2 are bothconvergent to MG × MD in a neighborhood of (θ∗ , ψ ∗ ).
Moreover, the rate of convergence is at least linear.
Theorem 4.1 shows that GAN training with our gradientpenalties is convergent when initialized sufficiently close
to the equilibrium point. While this does not show that themethod is globally convergent, it at least shows that near theequilibrium the method is well-behaved.
4.3. Stable equilibria for unregularized GAN training
As we have seen in Section 2, unregularized GAN training
does not always converge to the Nash-equilibrium. However,this does not rule out the existence of stable equilibria forevery GAN architecture. In Section E of the supplementarymaterial, we identify two forms of stable equilibria that may
exist for unregularized GAN training (energy solutions andfull-rank solutions). However, it is not yet clear under what
conditions such solutions exist for high dimensional data
distributions.

5. Experiments
2D-Problems Measuring convergence for GANs is hardfor high dimensional problems, because we lack a metricthat can reliably detect non-convergent behavior. We therefore first examine the behavior of the different regularizers
on simple 2D examples where we can assess convergence
using an estimate of the Wasserstein-1-distance.
To this end, we run 5 different training algorithms on 4 different 2D-examples for 6 different GAN architectures. For
each method, we try both stochastic gradient descent and
RMS-Prop with 4 different learning rates. For the R1 -, R2 and WGAN-GP-regularizers we try 3 different regularization parameters. We train all methods for 50k iterations andreport the results for the best hyperparameter setup. Please
see the supplementary material for details.
The results are shown in Figure 5. We see that the R1 - andR2 -regularizers perform similarly and they achieve slightly
better results than unregularized training or training with(a) 2D Gaussian(b) Line segment(c) Circle(d) Four line segments

Figure 5. Wasserstein-1-distance to true data distribution for 4 different 2D-data-distributions, 6 different architectures (small bars)
and 5 different training methods. Here, we abbreviate WGANGP with 1 and 5 discriminator update(s) per generator update as
WGP-1 and WGP-5.

WGAN-GP. In the supplementary material we show that the
R1 - and R2 -regularizers find solutions where the discriminator is 0 in a neighborhood of the true data distribution,
whereas unregularized training and WGAN-GP converge toenergy solutions.
Images To test how well the gradient penalties from Section 4.1 perform on more complicated tasks, we train convolutional GANs on a variety of datasets, including a generative model for all 1000 Imagenet classes and a generativemodel for the celebA-HQ dataset (Karras et al., 2017) at resolution 1024×1024. While we find that unregularized GAN
training quickly leads to mode-collapse for these problems,
our simple R1 -regularizer enables stable training. Random
samples from the models and more details on the experimental setup can be found in the supplementary material.

6. Conclusion
In this paper, we analyzed the stability of GAN training ona simple yet prototypical example. Due to the simplicity ofthe example, we were able to analyze the convergence properties of the training dynamics analytically and we showedthat (unregularized) gradient based GAN optimization is
not always locally convergent. Our findings also show that
WGANs and WGAN-GP do not always lead to local convergence whereas instance noise and zero-centered gradientpenalties do. Based on our analysis, we extended our results
to more general GANs and we proved local convergence forsimplified zero-centered gradient penalties under suitable
assumptions. In the future, we would like to extend ourtheory to the non-realizable case and examine the effect offinite sampling sizes on the GAN training dynamics.

Which Training Methods for GANs do actually Converge?

Acknowledgements
We would like to thank Vaishnavh Nagarajan and Kevin
Roth for insightful discussions. We also thank Vaishnavh
Nagarajan for giving helpful feedback on an early draft ofthis manuscript. We thank NVIDIA for donating the GPUs
for the experiments presented in the supplementary material.
This work was supported by Microsoft Research through its
PhD Scholarship Programme.

References
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean,
J., Devin, M., Ghemawat, S., Irving, G., Isard, M., Kudlur, M., Levenberg, J., Monga, R., Moore, S., Murray,
D. G., Steiner, B., Tucker, P. A., Vasudevan, V., Warden,P., Wicke, M., Yu, Y., and Zheng, X. Tensorflow: A
system for large-scale machine learning. In 12th USENIX
Symposium on Operating Systems Design and Implementation, OSDI 2016, Savannah, GA, USA, November 2-4,
2016., pp. 265–283, 2016.
Arjovsky, M. and Bottou, L. Towards principled methods for training generative adversarial networks. CoRR,
abs/1701.04862, 2017.
Arjovsky, M., Chintala, S., and Bottou, L. Wasserstein
GAN. CoRR, abs/1701.07875, 2017.
Barratt, S. and Sharma, R. A note on the inception score.
CoRR, abs/1801.01973, 2018.
Berthelot, D., Schumm, T., and Metz, L. BEGAN: boundary equilibrium generative adversarial networks. CoRR,
abs/1703.10717, 2017.
Bertsekas, D. P. Nonlinear programming. Athena scientificBelmont, 1999.
Gidel, G., Berard, H., Vincent, P., and Lacoste-Julien, S. A
variational inequality perspective on generative adversarial nets. CoRR, abs/1802.10551, 2018.

He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE
conference on computer vision and pattern recognition,
pp. 770–778, 2016.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., andHochreiter, S. Gans trained by a two time-scale update
rule converge to a local nash equilibrium. In Advancesin Neural Information Processing Systems 30: Annual
Conference on Neural Information Processing Systems
2017, 4-9 December 2017, Long Beach, CA, USA, pp.
6629–6640, 2017.
Hjelm, R. D., Jacob, A. P., Che, T., Cho, K., and Bengio,
Y. Boundary-seeking generative adversarial networks.
CoRR, abs/1702.08431, 2017.
Karras, T., Aila, T., Laine, S., and Lehtinen, J. Progressive growing of gans for improved quality, stability, andvariation. CoRR, abs/1710.10196, 2017.
Khalil, H. K. Nonlinear systems. Prentice-Hall, New Jersey,
2(5):5–1, 1996.
Kodali, N., Abernethy, J. D., Hays, J., and Kira, Z. How to
train your DRAGAN. CoRR, abs/1705.07215, 2017.
Krizhevsky, A. and Hinton, G. Learning multiple layers offeatures from tiny images. 2009.
Liu, Z., Luo, P., Wang, X., and Tang, X. Deep learning face
attributes in the wild. In Proceedings of International
Conference on Computer Vision (ICCV), 2015.
Mescheder, L. M., Nowozin, S., and Geiger, A. The numerics of gans. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long
Beach, CA, USA, pp. 1823–1833, 2017.
Miyato, T., Kataoka, T., Koyama, M., and Yoshida, Y. Spectral normalization for generative adversarial networks.
CoRR, abs/1802.05957, 2018.

Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B.,
Warde-Farley, D., Ozair, S., Courville, A. C., and Bengio,
Y. Generative adversarial nets. In Advances in Neural Information Processing Systems 27: Annual Conference on
Neural Information Processing Systems 2014, December
8-13 2014, Montreal, Quebec, Canada, pp. 2672–2680,
2014.

Nagarajan, V. and Kolter, J. Z. Gradient descent GAN
optimization is locally stable. In Advances in Neural
Information Processing Systems 30: Annual Conference
on Neural Information Processing Systems 2017, 4-9
December 2017, Long Beach, CA, USA, pp. 5591–5600,
2017.

Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., andCourville, A. C. Improved training of wasserstein gans. In
Advances in Neural Information Processing Systems 30:
Annual Conference on Neural Information Processing
Systems 2017, 4-9 December 2017, Long Beach, CA,
USA, pp. 5769–5779, 2017.

Nowozin, S., Cseke, B., and Tomioka, R. f-gan: Training
generative neural samplers using variational divergence
minimization. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016,
Barcelona, Spain, pp. 271–279, 2016.

Which Training Methods for GANs do actually Converge?

Odena, A., Olah, C., and Shlens, J. Conditional image
synthesis with auxiliary classifier gans. In Proceedings ofthe 34th International Conference on Machine Learning,
ICML 2017, Sydney, NSW, Australia, 6-11 August 2017,
pp. 2642–2651, 2017.
Radford, A., Metz, L., and Chintala, S. Unsupervised representation learning with deep convolutional generative
adversarial networks. CoRR, abs/1511.06434, 2015.
Roth, K., Lucchi, A., Nowozin, S., and Hofmann, T. Stabilizing training of generative adversarial networks throughregularization. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017,
Long Beach, CA, USA, pp. 2015–2025, 2017.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S.,
Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein,M., Berg, A. C., and Fei-Fei, L. ImageNet Large Scale
Visual Recognition Challenge. International Journal ofComputer Vision (IJCV), 115(3):211–252, 2015. doi:
10.1007/s11263-015-0816-y.
Salimans, T., Goodfellow, I. J., Zaremba, W., Cheung, V.,
Radford, A., and Chen, X. Improved techniques fortraining gans. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016,
Barcelona, Spain, pp. 2226–2234, 2016.
Sønderby, C. K., Caballero, J., Theis, L., Shi, W., and
Huszár, F. Amortised MAP inference for image superresolution. CoRR, abs/1610.04490, 2016.
Tieleman, T. and Hinton, G. Lecture 6.5-rmsprop: Divide
the gradient by a running average of its recent magnitude,
2012.
Yazici, Y., Foo, C. S., Winkler, S., Yap, K., Piliouras, G., andChandrasekhar, V. The unusual effectiveness of averaging
in GAN training. CoRR, abs/1806.04498, 2018.
Yu, F., Zhang, Y., Song, S., Seff, A., and Xiao, J. LSUN:
construction of a large-scale image dataset using deeplearning with humans in the loop. CoRR, abs/1506.03365,
2015.
Zhao, J. J., Mathieu, M., and LeCun, Y. Energy-based
generative adversarial network. CoRR, abs/1609.03126,
2016.