Which Training Methods for GANs do
actually Converge?
Lars Mescheder1, Sebastian Nowozin2, Andreas Geiger1
1

MPI-IS Tübingen 2University of Tübingen 3Microsoft Research Cambridge

Introduction
Generative neural networks:noise

Key challenge:
have to learn high dimensional probability distribution

2

Introduction
Generative Adversarial networks (GANs):

real or fake?

Generator
Discriminator3Generative Adversarial Networks
Alternating gradient descent4

Generative Adversarial Networks
Simultaneous gradient descent5

Generative Adversarial Networks●Does a (pure) Nash-equilibrium exist?
–●with(Goodfellow et al., 2014)

Does it solve the min-max problem?
–●Yes, if there isYes, if(Goodfellow et al., 2014)

Do simultaneous and / or alternating gradient descent converge
to the Nash-equilbrium?

6Ist GAN training locally asymptotically stable?
Mescheder et al. (2017): No, if Jacobian of gradientvector field has purely imaginary eigenvalues
Nagarajan and Kolter (2017): Yes, if generator
and data distributions locally have the same support
Heusel et al. (2017): Yes, if optimal discriminatorparameters are continuous function of generator parameters
and two-timescale annealing scheme is adopted

Is GAN training locally asymptotically stable in thegeneral case?
7

Our Contributions●

Dirac-GAN:–●

Analysis of common regularizers:––●

WGAN and WGAN-GP not always stable
Instance noise & zero-centered gradient penalties are stable

Simplified gradient penalties–●

Unregularized GAN-training is not always stable
when distributions do not have the same support

Convergence proof for realizable case

Empirical results:–

High resolution (1024x1024) generative models
without progressively growing architectures8

Convergence theory“Simple experiments, simple theorems are the building
blocks that help us understand more complicated systems.”
Ali Rahimi – Test of Time Award speech, NIPS 20179

Convergence theory

The Dirac-GAN:

10

Convergence theory

Unregularized GAN training:11

Convergence theory
Understanding the gradient vector field:

Local convergence of simultaneous and alternating gradientdescent determined by eigenvalues of Jacobian

12

Convergence theory
Continuous system:

13

Convergence theory
Continuous system:

14

Convergence theory
Discretized system:

15

Convergence theory
Discretized system:

16

Which training methods converge?

Unregularized GAN training:Eigenvalues:
17

Which training methods converge?

Wasserstein-GAN1 training:

Eigenvalues:
Arjovsky et al. - Wasserstein GAN (2017)

118

Which training methods converge?

Wasserstein-GAN1 training(5 discriminator updates / generator update):

Eigenvalues:
Arjovsky et al. - Wasserstein GAN (2017)

119Which training methods converge?

Zero-centered Gradient Penalties

Eigenvalues:
20

Which training methods converge?

Zero-centered Gradient Penalties (critical)

Eigenvalues:
21

Which training methods converge?

Zero-centered Gradient Penalties
fast convergenceno convergenceslow convergence22General convergence results●Regularizers for discriminator●

Regularized gradient vector field

23

General convergence results

Assumption I: the generator can represent thetrue data distribution
Assumption II:

andAssumption III: the discriminator can detect when
the generator deviates from equilibrium
Assumption IV: the generator and data distribution
have locally the same support (Nagarajan & Kolter)
24General convergence results
For the Dirac-GAN:

Assumption I: the generator can represent thetrue data distribution
Assumption II:

andAssumption III: the discriminator can detect when
the generator deviates from equilibrium
Assumption IV: the generator and data distribution
have locally the same support (Nagarajan & Kolter)
25General convergence results
For GANs in the wild:

Assumption I: the generator can represent thetrue data distribution
Assumption II:

andAssumption III: the discriminator can detect when
the generator deviates from equilibrium
Assumption IV: the generator and data distribution
have locally the same support (Nagarajan & Kolter)
26

General convergence results

Theorem: under Assumption I, II, III and some mild
technical assumptions the GAN training dynamics
for the regularized training objective are locallyasymptotically stable near the equilibrium point

27

General convergence results
Proof (idea):(Extends prior work by Nagarajan & Kolter1)

Full column rank

Negative definite(orthogonal toAll eigenvalues of)have negative real part

Nagarajan & Kolter - Gradient descent GAN optimization is locally stable (2017)

128

Experiments

Imagenet (128 x 128, 1k classes)

29Experiments

LSUN bedrooms (256 x 256)30

Experiments

LSUN churches (256 x 256)31

Experiments

LSUN towers (256 x 256)32

Experiments

celebA-HQ (1024 x 1024)

33

Practical recommendations

●
●
●
●

●

use alternating instead of simultaneous gradient descent
don’t use momentum
use regularization to stabilize the training
simple zero-centered gradient penalties for the discriminatoryield excellent results
progressively growing architectures might be not all thatimportant when using a good regularizer

34

Poster #77

35