Which Training Methods for GANs do actually Converge?
Supplementary Material

A. Preliminaries
In this section we first summarize some results from thetheory of discrete dynamical systems. We also prove a discrete version of a basic convergence theorem for continuous
dynamical systems from Nagarajan & Kolter (2017) whichallows us to make statements about training algorithms for
GANs for finite learning rates. Afterwards, we summarize
some results from Mescheder et al. (2017) about the convergence properties of simultaneous and alternating gradientdescent. Moreover, we state some eigenvalue bounds thatwere derived by Nagarajan & Kolter (2017) which we needto prove Theorem 4.1 on the convergence of the regularized
GAN training dynamics.
A.1. Discrete dynamical systems
In this section, we recall some basic definitions from the theory of discrete nonlinear dynamical systems. For a similar
description of the theory of continuous nonlinear dynamicalsystems see for example Khalil (1996) and Nagarajan &Kolter (2017).
In this paper, we consider continuously differentiable operators F : Ω → Ω acting on an open set Ω ⊂ Rn . A fixed
point of F is a point x̄ ∈ Ω such that F (x̄) = x̄. We areinterested in stability and convergence of the fixed point iteration F (k) (x) near the fixed point. To this end, we first haveto define what we mean by stability and local convergence:
Definition A.1. Let x̄ ∈ Ω be a fixed point of a continuously
differentiable operator F : Ω → Ω. We call x̄
• stable if for every  > 0 there is δ > 0 such thatkx − x̄k < δ implies kF (k) (x) − x̄k <  for all k ∈ N.
• asymptotically stable if it is stable and there is δ > 0such that kx − x̄k < δ implies that F (k) (x) converges
to x̄• exponentially stable if there is λ ∈ [0, 1), δ > 0 and
C > 0 such that kx − x̄k < δ implies
kF (k) (x) − x̄k < Ckx − x̄kλk

responding algorithm linearly convergent. Moreover, if x̄
is exponentially stable, we call the infimum of all λ so that(15) holds for some C > 0 the convergence rate of the fixed
point iteration.
As it turns out, local convergence of fixed point iterations
can be analyzed by examining the spectrum of the Jacobianof the fixed point operator. We have the following central
Theorem:
Theorem A.2. Let F : Ω → Ω be a C 1 -mapping on an
open subset Ω of Rn and x̄ ∈ Ω be a fixed point of F .
Assume that the absolute values of the eigenvalues of theJacobian F 0 (x̄) are all smaller than 1. Then the fixed point
iteration F (k) (x) is locally convergent to x̄. Moreover, therate of convergence is at least linear with convergence rate
|λmax | where λmax denotes the eigenvalue of F 0 (x̄) withthe largest absolute value.
Proof. See Bertsekas (1999), Proposition 4.4.1.
For the proof of Theorem 4.1 in Section D, we need a generalization of Theorem A.2 that takes into account submanifolds of fixed points. The next theorem is a discrete versionof Theorem A.4 from Nagarajan & Kolter (2017) and we
prove it in a similar way:
Theorem A.3. Let F (α, γ) define a C 1 -mapping that maps
some domain Ω to itself. Assume that there is a local neighborhood U of 0 such that F (0, γ) = (0, γ) for γ ∈ U . Moreover, assume that all eigenvalues of J := ∇α F (α, 0) |α=0have absolute value smaller than 1. Then the fixed point iteration defined by F is locally convergent to M := {(0, γ) |γ ∈ U } with linear convergence rate in a neighborhood of(0, 0). Moreover, the convergence rate is |λmax | with λmax
the eigenvalue of J with largest absolute value.
Proof. In the following, we write F (α, γ)=(F1 (α, γ), F2 (α, γ)), so that the fixed point iteration
can be written as(15)

for all k ∈ N.
If x̄ is asymptotically stable fixed point of F , we call thealgorithm obtained by iteratively applying F locally convergent to x̄. If x̄ is exponentially stable, we call the cor-

αk+1 = F1 (αk , γk ) γk+1 = F2 (αk , γk ).

(16)

We first examine the behavior of F1 near (0, 0). To this end,
we develop F1 into a Taylor-series
F1 (α, γ) = Jα + g1 (α, γ)(17)Which Training Methods for GANs do actually Converge?

We first show that for any c > 0 we have kg1 (α, γ)k ≤ckαk sufficiently close to (0, 0): because F1 (0, γ) = 0 for
all γ close to 0, g1 (α, γ) must be of the form g1 (α, γ) =h1 (α, γ)α with h1 (0, 0) = 0. This shows that for any c > 0
there is indeed an open neighborhood V of (0, 0) so that
|g1 (α, γ)| ≤ ckαk for all (α, γ) ∈ V .
According to Bertsekas (1999), Proposition A 15, we canselect for every  > 0 a norm k · kQ on Rn such thatkJαkQ < (|λmax | + )kαkQ(18)

for α ∈ Rn where |λmax | denotes the eigenvalue of J withthe largest absolute value.
Hence, for (α, γ) ∈ V ,kF1 (α, γ)kQ ≤ kJαkQ + kg1 (α, γ)kQ
< (|λmax | +  + c)kαkQ(19)Because we can make c +  as small as we want, this showsthat kαk k ≤ Cλk kα0 k for some C > 0 and λ ∈ [0, 1), if
α0 and all γl for l = 0, . . . , k − 1 are sufficiently close to 0 .
We therefore have to show that the iterates γk stay in a given
local neighborhood of 0, i.e. kγk k < δ for some δ > 0,
when α0 and γ0 are initialized sufficiently close to 0.
To show this, we develop F2 into a Taylor-series around 0:
F2 (α, γ) = γ + g2 (α, γ).

(20)Again, we see that g2 must be of the form g2 (α, γ) =h2 (α, γ)α, showing that kg2 (α, γ)k ≤ c0 kαkQ for somefixed constant c0 > 0 (note that in general h2 (0, 0) 6= 0).
We therefore havewhere L(θ, ψ) is the GAN training objective defined in (1).
Similarly, alternating gradient descent can be described by
an update operator of the form Fh = F2,h ◦ F1,h where F1,hand F2,h are given byθ − h∇θ L(θ, ψ)
F1,h (θ, ψ) =
(23)
ψ


θ
F2,h (θ, ψ) =.
(24)
ψ + h∇ψ L(θ, ψ)Moreover, we defined the gradient vector field−∇θ L(θ, ψ)v(θ, ψ) =.
∇ψ L(θ, ψ)(25)

To understand convergence of simultaneous and alternating
gradient descent, we have to understand when the Jacobianof the corresponding update operator has only eigenvalues
with absolute value smaller than 1.
Lemma A.4. The eigenvalues of the Jacobian of the update
operator for simultaneous gradient descent are given by
λ = 1 + hµ with µ the eigenvalues of v 0 (θ∗ , ψ ∗ ). Assume
that v 0 (θ∗ , ψ ∗ ) has only eigenvalues with negative real part.
The eigenvalues of the Jacobian of the update operator Fhfor simultaneous gradient descent are then all in the unit
circle if and only ifh<1| Re(λ)|

2
1+



Im(λ)
Re(λ)

2

(26)

for all eigenvalues λ of v 0 (θ∗ , ψ ∗ ).
Proof. For simultaneous gradient descent we havekγk − γ0 k ≤k−1
Xkg2 (αl , γl )k ≤

l=0

≤

k−1
X
l=0

k−1
X

c0 kαl kQ

Fh (θ, ψ) = (θ, ψ) + hv(θ, ψ)

l=0

Cc0
Cc λ kα0 kQ ≤kα0 kQ
1−λ
0 l

(21)

(27)

and hence Fh0 (θ∗ , ψ ∗ ) = I + hv 0 (θ∗ , ψ ∗ ). Therefore theeigenvalues are given by λ = 1 + hµ with µ the eigenvaluesof v 0 (θ∗ , ψ ∗ ).

1−λHence, if we initialize α0 within kα0 kQ < 2CC
0 δ andδ
γ0 within kγ0 k < 2 , we have kγk k < δ for all k ∈ N,
concluding the proof.

To see when |λ| < 1, we write µ = −a + ib with a, b ∈ R
and a > 0. Then

A.2. Simultaneous and Alternating Gradient Descent

which is smaller than 1 if and only if

In this section, we recall some results from Mescheder et al.
(2017) about the convergence properties of simultaneousand alternating gradient descent as algorithms for training
generative adversarial networks.
Recall that simultaneous gradient descent can be describedby an update operator of the form
θ − h∇θ L(θ, ψ)Fh (θ, ψ) =(22)
ψ + h∇ψ L(θ, ψ)|λ|2 = (1 − ha)2 + h2 b2

h<

2a
.
a2 + b2

(28)

(29)

Dividing both the numerator and denominator by a2 shows
the assertion.
Lemma A.5. Assume that v 0 (θ∗ , ψ ∗ ) has only eigenvalues with negative real part. For h > 0 small enough, theeigenvalues of the Jacobian of the update operator Fh foralternating gradient descent are then all in the unit circle.

Which Training Methods for GANs do actually Converge?

Proof. The Jacobian of the update operator Fh = Fh,2 ◦Fh,1
at an equilibrium point (θ∗ , ψ ∗ ) is00
Fh0 (θ∗ , ψ ∗ ) = Fh,2(θ∗ , ψ ∗ ) · Fh,1(θ∗ , ψ ∗ ).

(30)

However, we have

As we will see in the proof of Theorem 4.1, Remark A.6
allows us to apply Theorem A.3 to situations where the stationary points lie on a lower dimensional manifold insteadof a space of the form {0}k × Rn+m−k .
A.3. Eigenvalue bounds

0Fh,i(θ∗ , ψ ∗ ) = I + hvi0 (θ∗ , ψ ∗ )(31)

for i ∈ {1, 2} where−∇θ L(θ, ψ)00v2 (θ, ψ) =∇ψ L(θ, ψ)
v1 (θ, ψ) =

(32)
(33)

denote the components of the gradient vector field. Hence
Fh0 (θ∗ , ψ ∗ ) = I + h(v10 (θ∗ , ψ ∗ ) + v20 (θ∗ , ψ ∗ ))+ h2 v20 (θ∗ , ψ ∗ )v10 (θ∗ , ψ ∗ )= I + h(v 0 (θ∗ , ψ ∗ ) + h R(θ∗ , ψ ∗ )). (34)with R(θ∗ , ψ ∗ ) := v20 (θ∗ , ψ ∗ )v10 (θ∗ , ψ ∗ ). For h > 0 smallenough, all eigenvalues of v 0 (θ∗ , ψ ∗ ) + h R(θ∗ , ψ ∗ ) will be
arbitrarily close to the eigenvalues of v 0 (θ∗ , ψ ∗ ). Because
all eigenvalues of v 0 (θ∗ , ψ ∗ ) have negative real-part, alleigenvalues of Fh0 (θ∗ , ψ ∗ ) will hence lie inside the unit
circle for h > 0 small enough.
In the proof of Theorem 4.1 we will use local coordinates,
i.e. a diffeomorphism φ that maps a local neighborhood of(θ∗ , ψ ∗ ) to an open subset of Rn+m . The vector field v andthe update operator Fh then have the following representation in the local coordinates:
Fhφ (α) := φ ◦ Fh ◦ φ−1 (α)(35)

v φ (α) = φ0 (θ, ψ) · (v ◦ φ−1 (α))(36)

While in local coordinates, the simple relationships between Fhφ (α) and v φ (α) needed to prove Lemma A.4 andLemma A.5 do not hold anymore, the spectrum can be described in the same way:
Remark A.6. Assume (θ∗ , ψ ∗ ) is a fixed point of Fh and astationary point of v. Let α∗ = φ(θ∗ , ψ ∗ ). Then(Fhφ )0 (α∗ ) = φ0 (θ∗ , ψ ∗ )Fh0 (θ∗ , ψ ∗ )φ0 (θ∗ , ψ ∗ )−1
φ 0
∗
0 ∗
∗ 0 ∗
∗ 0 ∗
∗ −1
(v ) (α ) = φ (θ , ψ )v (θ , ψ )φ (θ , ψ )

(37)
(38)

Hence, (Fhφ )0 (α∗ ) and Fh0 (θ∗ , ψ ∗ ) have the same spectrum.
The same also holds for (v φ )0 (α∗ ) and v 0 (θ∗ , ψ ∗ ).
Proof. This follows from the chain and product rules byusing the fact that Fh (θ∗ , ψ ∗ ) = (θ∗ , ψ ∗ ) and v(θ∗ , ψ ∗ ) =0.

When analyzing the convergence properties of GANs, wehave to analyze the spectrum of real-valued matrices of thefrom
0 −B T(39)
B −Q
with Q symmetric positive definite. To this end, we need
the following important theorem from Nagarajan & Kolter(2017) which gives explicit bounds on the real part of theeigenvalues:
Theorem A.7. Assume J ∈ R(n+m)×(n+m) is of the following form:
0 −B TJ=(40)
B −Q
where Q ∈ Rm×m is a symmetric positive definite matrix
and B ∈ Rm×n has full column rank. Then all eigenvalues
λ of J satisfy Re(λ) < 0. More precisely• if Im(λ) = 0
Re(λ) ≤ −λmin (Q)λmin (B T B)
(41)
λmax (Q)λmin (Q) + λmin (B T B)

• if Im(λ) 6= 0Re(λ) ≤ −λmin (Q)2(42)

Proof. See Nagarajan & Kolter (2017), Lemma G.2.
In Section E.1, we need a generalization of Theorem A.7.
Using almost exactly the same proof as for Theorem A.7,
we obtain
Theorem A.8. Assume J ∈ R(n+m)×(n+m) is of the following form:
−P −B T
J=
(43)
B
−Q
where P ∈ Rn×n is a symmetric positive semi-definite
matrix, Q ∈ Rm×m is a symmetric positive definite matrix
and B ∈ Rm×n has full column rank. Then all eigenvalues
λ of J satisfy Re(λ) < 0.
Proof. Let v T = (aT , bT ) denote some eigenvector of
J with corresponding eigenvalues λ = λr + iλi , where
λr , λi ∈ R. Then
λr =

1 T
v̄ (J + J T )v = −āT P a − b̄T Qb.
2(44)

Which Training Methods for GANs do actually Converge?

Because both P and Q are positive semi-definite, we haveλr ≤ 0. Because Q is positive definite, it suffices to showthat b 6= 0 to prove λr < 0.
Assume that b = 0. Because v is an eigenvector of J, we
have Ba − Qb = λb and therefore Ba = 0. Because B
has full-column rank, this shows a = 0 and hence v = 0.
However, this contradicts the fact that v is an eigenvector of
J. All in all, this show that b 6= 0 and thus λr ≤ −b̄T Qb <0 as required.

B. Proofs for the Dirac-GAN
This section contains the proofs for our results from Section 2 and Section 3 on the properties of the Dirac-GAN.
Lemma 2.2. The unique equilibrium point of the training
objective in (4) is given by θ = ψ = 0. Moreover, theJacobian of the gradient vector field at the equilibrium point
has the two eigenvalues ±f 0 (0) i which are both on theimaginary axis.
Proof. The loss in (4) can be rewritten as

For applying Theorems A.2, we have to show that the Jacobian of the update operator Fh only has eigenvalues withabsolute value smaller than 1. For simultaneous and alternating gradient descent this can be achieved (Lemma A.4
and A.5), if the Jacobian of the gradient vector field v only
has eigenvalues with negative real-part. While this condition suffices to prove convergence for small learning rates,
Mescheder et al. (2017) showed that simultaneous and alternating gradient descent might still require intractably small
learning rates if the imaginary part of the eigenvalues islarge. However, in our case we have the following simple
bound on the imaginary part of the eigenvalues:
Lemma A.9. Let

J=

−P
B

−B T
−Q


(45)

where P ∈ Rn×n and Q ∈ Rm×m are symmetric. All
eigenvalues λ of J then satisfyq(46)| Im(λ)| ≤ λmax (B T B).

Proof. Assume v, kvk = 1, is an eigenvector of J witheigenvalue λ. Then(47)

1with Ja := 2i(J − J T ). Hence, by the Cauchy-Schwarz
inequality| Im(λ)| ≤ kvkkJa vk = kJa vk.

(51)

It is easy to check that the gradient vector field is given by−f 0 (θψ)ψ
v(θ, ψ) =.
(52)
f 0 (θψ)θ
Because L(θ, 0) = L(0, ψ) = const for all θ, ψ ∈ R,
(θ, ψ) = (0, 0) is indeed a Nash-equilibrium for the gamedefined by (51). Because we assume f 0 (t) 6= 0 for all t ∈ R,
we have v(θ, ψ) = 0 if and only if (θ, ψ) = (0, 0), showingthat (0, 0) is indeed the unique Nash-equilibrium.
Moreover, the Jacobian v 0 (θ, ψ) of v is given by−f 00 (θψ)ψ 2−f 0 (θψ) − f 00 (θψ)θψ. (53)
f 0 (θψ) + f 00 (θψ)θψf 00 (θψ)θ2
Evaluating it at the Nash equilibrium θ = ψ = 0, we obtain
0−f 0 (0)v 0 (0, 0) =(54)
f 0 (0)0which has the eigenvalues ±f 0 (0)i.

Note that this bound is independent from P and Q.

Im(λ) = v̄ T Ja v.

L(θ, ψ) = f (θψ) + const(48)

Lemma 2.3. The integral curves of the gradient vector fieldv(θ, ψ) do not converge to the Nash-equilibrium. Morespecifically, every integral curve (θ(t), ψ(t)) of the gradientvector field v(θ, ψ) satisfies θ(t)2 + ψ(t)2 = const for allt ∈ [0, ∞).
Proof. Let R(θ, ψ) := 12 (θ2 + ψ 2 ). Thend
R(θ(t), ψ(t))
dt= θ(t)v1 (θ(t), ψ(t)) + ψ(t)v2 (θ(t), ψ(t)) = 0, (55)But, if v T = (aT , bT ),showing that R(θ, ψ) is indeed constant for all t ∈ [0, ∞).
kJa vk2 = bBB T b + aB T Ba ≤ λmax (B T B).

(49)

This shows| Im(λ)| ≤q

λmax (B T B).

(50)

Lemma 2.4. For simultaneous gradient descent, the Jacobian of the update operator Fh (θ, ψ) has
p eigenvalues
λ1/2 = 1 ± hf 0 (0)i with absolute values 1 + h2 f 0 (0)2
at the Nash-equilibrium. Independently of the learning rate,
simultaneous gradient descent is therefore not stable near

Which Training Methods for GANs do actually Converge?

the equilibrium. Even stronger, for every initial conditionand learning rate h > 0, the norm of the iterates (θk , ψk )obtained by simultaneous gradient descent is monotonicallyincreasing.
Proof. The first part is a direct consequence of Lemma A.4
and Lemma 2.2.
To see the the norms of the iterates (θk , ψk ) is monotonicallyincreasing, we calculate22
θk+1+ ψk+1= (θk − hf 0 (θk ψk )ψk )2 + (ψk + hf 0 (θk ψk )θk )2= θk2 + ψk2 + h2 f 0 (θk ψk )2 (θk2 + ψk2 )≥ θk2 + ψk2 . (56)

Lemma 2.5. For alternating gradient descent with ng generator and nd discriminator updates, the Jacobian of the
update operator Fh (θ, ψ) has eigenvalues
s
2
α2
α2
±
− 1.
(5)
λ1/2 = 1 −1−22√
with α := ng nd hf 0 (0). For α ≤ 2, all eigenvalues arehence on the unit circle. Moreover for α > 2, there are
eigenvalues outside the unit circle.
Proof. The update operators for alternating gradient descentare given byθ − hf 0 (θψ)ψ
F1 (θ, ψ) =
(57)
ψ


θ
F2 (θ, ψ) =.
(58)
ψ + hf 0 (θψ)θ
Hence, the Jacobians of these operators at 0 are given by
1 −hf 0 (0)0F1 (0, 0) =
(59)
0
1


1
0
F20 (0, 0) =.
(60)
hf 0 (0) 1
As a result, the Jacobian of the combined update operator isn(F2nd ◦ F1 g )0 (0, 0) = F20 (0, 0)nd · F10 (0, 0)ng
1−ng hf 0 (0)=. (61)nd hf 0 (0) −ng nd h2 f 0 (0)2 + 1
An easy calculation shows that the eigenvalues of this matrix
are
s
2
α2
α2
λ1/2 = 1 −
±
1−
−1
(62)
2
2

√
with α = ng nd hf 0 (0) which are on the unit circle if andonly if α ≤ 2.

Lemma 3.1. A WGAN trained with simultaneous or alternating gradient descent with a fixed number of discriminator updates per generator update and a fixed learning rateh > 0 does generally not converge to the Nash equilibrium
for the Dirac-GAN.
Proof. First, consider simultaneous gradient descent. Assume that the iterates (θk , ψk ) converge towards the equilibrium point (0, 0). Note that (θk+1 , ψk+1 ) 6= 0 if (θk , ψk ) 6=
0. We can therefore assume without loss of generality that(θk , ψk ) 6= 0 for all k ∈ N.
Because limk→∞ ψk = 0, there exists k0 such that for allk ≥ k0 we have |ψk | < 1. For k ≥ k0 we therefore have

 
 
θk+1
1 −h
θk
=
.
(63)
ψk+1
h 1
ψk
For k ≥ k0 , the iterates are therefore given by
 




θk
θk0
1 −h
= Ak−k0
with A =. (64)
ψkψk0
h 1
However, the eigenvalues of A are given by λ1/2 = 1 ±√hi which both have absolute value 1 + h2 > 1. This
contradicts the assumption that (θk , ψk ) converges to (0, 0).
A similar argument also hold for alternating gradient descent.
In this case, A is given by

n 
n


1 0 d 1 −h g1−hng=
. (65)
h 1
0 1
hnd 1 − h2 ng nd
The eigenvalues of A as in (65) are given bys
2h2 ng nhh2 ng nh
1−
±
1−
− 1.
2
2

(66)

At least one of these eigenvalues has absolute value greateror equal to 1. Note that for almost all initial conditions(θ0 , ψ0 ), the the inner product between the eigenvector corresponding to the eigenvalue with modulus bigger than 1
will be nonzero for all k ∈ N. Since the recursion in (63)is linear, this contradicts the fact that (θk , ψk ) → (0, 0),
showing that alternating gradient descent generally does notconverge to the Nash-equilibrium either.
Lemma 3.2. When using Gaussian instance noise with standard deviation σ, the eigenvalues of the Jacobian of thegradient vector field are given byp
λ1/2 = f 00 (0)σ 2 ± f 00 (0)2 σ 4 − f 0 (0)2 .
(6)Which Training Methods for GANs do actually Converge?

In particular, all eigenvalues of the Jacobian have negative
real-part at the Nash-equilibrium if f 00 (0) < 0 and σ > 0.
Hence, simultaneous and alternating gradient descent areboth locally convergent for small enough learning rates.
Proof. When using instance noise, the GAN training objective (1) is given byhiEθ̃∼N (θ,σ2 ) f (θ̃ψ) + Ex∼N (0,σ2 ) [f (−xψ)] . (67)The corresponding gradient vector field is hence given by−ψf 0 (θ̃ψ)
ṽ(θ, ψ) = Eθ̃,x.
(68)
θ̃f 0 (θ̃ψ) − xf 0 (−xψ)
The Jacobian ṽ 0 (θ, ψ) is therefore−f 00 (θ̃ψ)ψ 2−f 0 (θ̃ψ) − f 00 (θ̃ψ)θ̃ψEθ̃,xf 0 (θ̃ψ) + f 00 (θ̃ψ)θ̃ψ f 00 (θ̃ψ)θ̃2 + x2 f (−xψ)(69)
Evaluating it at θ = ψ = 0 yields
0−f 0 (0)ṽ 0 (0, 0) =(70)
f 0 (0) 2f 00 (0)σ 2
whose eigenvalues are given byp
λ1/2 = f 00 (0)σ 2 ± f 00 (0)2 σ 4 − f 0 (0)2 .

(71)

Lemma 3.3. The eigenvalues of the Jacobian of the gradient vector field for the gradient-regularized Dirac-GAN at
the equilibrium point are given byrγγ2− f 0 (0)2 .
(8)
λ1/2 = − ±24
In particular, for γ > 0 all eigenvalues have negative realpart. Hence, simultaneous and alternating gradient descentare both locally convergent for small enough learning rates.
Proof. The regularized gradient vector field becomes−f 0 (θψ)ψ
ṽ(θ, ψ) =.
(72)
f 0 (θψ)θ − γψ
The Jacobian ṽ 0 (θ, ψ) is therefore given by−f 00 (θψ)ψ 2−f 0 (θψ) − f 00 (θψ)θψ. (73)
f 0 (θψ) + f 00 (θψ)θψf 00 (θψ)θ2 − γ
Evaluating it at θ = ψ = 0 yields
0−f 0 (0)0ṽ (0, 0) =f 0 (0)−γ
whose eigenvalues are given byrγγ2
λ1/2 = − ±− f 0 (0)2 .
2
4

(74)

C. Other regularization strategies
In this section we discuss further regularization techniques
for GANs on our example problem that were omitted in themain text due to space constraints.
C.1. Nonsaturating GAN
Especially in the beginning of training, the discriminator
can reject samples produced by the generator with highconfidence (Goodfellow et al., 2014). When this happens,
the loss for the generator may saturate so that the generator
receives almost no gradient information anymore.
To circumvent this problem Goodfellow et al. (2014) introduced a nonsaturating objective for the generator. In
nonsaturating GANs, the generator objective is replaced
with7
max Epθ (x) f (−Dψ (x)).
(76)θ

In our example, this is maxθ f (−ψθ).
While the nonsaturating generator objective was originallymotivated by global stability considerations, we investigate
its effect on local convergence. A linear analysis similar tonormal GANs yields
Lemma C.1. The unique Nash-equilibrium for the nonsaturating GAN on the example problem is given by θ = ψ = 0.
The eigenvalues of the Jacobian of the gradient vector fieldat the equilibrium are ±f 0 (0)i which are both on the imaginary axis.
Proof. The gradient vector field for the nonsaturating GAN
is given by 0
−f (−θψ)ψ
v(θ, ψ) =.
(77)
f 0 (θψ)θ
As in the proof of Lemma 2.2, we see that (ψ, θ) = (0, 0)
defines the unique Nash-equilibrium for the nonsaturating
GAN.
Moreover, the Jacobian v 0 (θ, ψ) is
f 00 (−θψ)ψ 2−f 0 (−θψ) + f 00 (−θψ)θψ.
f 0 (θψ) + f 00 (θψ)θψf 00 (θψ)θ2(78)
At θ = ψ = 0 we therefore have0
−f 0 (0)0v (0, 0) =.
(79)
f 0 (0)0with eigenvalues λ1/2 = ±f 0 (0)i.

(75)

Lemma C.1 implies that simultaneous gradient descent is
not locally convergent for a nonsaturating GAN and any
7

Goodfellow et al. (2014) used f (t) = − log(1 + exp(−t)).

Which Training Methods for GANs do actually Converge?

learning rate h > 0, because the eigenvalues of the Jacobianof the corresponding update operator Fh all have absolutevalue larger than 1 (Lemma A.4). While Lemma C.1 also
rules out linear convergence towards the Nash-equilibriumin the continuous case (i.e. for h → 0), the continuous
training dynamics could in principle still converge with asublinear convergence rate. Indeed, we find this to be thecase for the Dirac-GAN. We have
Lemma C.2. For every integral curve of the gradient vector
field of the nonsaturating Dirac-GAN we haved(θ(t)2 + ψ(t)2 ) = 2 [f 0 (θψ) − f 0 (−θψ)] θψ.
dt

(80)

tem analytically8 for φ and w:
φ(t) = f 0 (0)(t − t0 )w(t) =00f (0)0−f 00 (0)t + 4f
0 (0) sin(4f (0)(t − t0 )) + c(90)

The training dynamics are hence convergent with logarith1√
mic convergence rate O.
t
For a more formal proof, first note that w is nonincreasing
by the first part of the proof. Moreover, for every  > 0
there is δ > 0 such that for w < δ:
f 0 (0) −  ≤ φ̇ ≤ f 0 (0) + 
00

For concave f this is nonpositive. Moreover, for f 00 (0) <0, the continuous training dynamics of the nonsaturating
Dirac-GAN converge with logarithmic convergence rate.

(89)
2

2

2

ẇ ≤ (f (0) sin (2φ) + )w .

(91)
(92)

This implies that for every time interval [0, T ], φ(t) is in

[ π
π 3π
π
+k ,
+k
(93)
8
2 8
2
k∈Z

Proof. The gradient vector field for the nonsaturating DiracGAN is given by−f 0 (−θψ)ψ
.
f 0 (θψ)θ

for t in a union of intervals QT ⊆ [0, T ] with total length
at least βbαT c with some constants α, β > 0 which are
independent of T .

(81)

For these t ∈ QT we have sin2 (2φ(t)) ≥ 12 . Because
f 00 (0) < 0, this shows
1 00
f (0) +  w(t)2(94)
ẇ(t) ≤
2d(θ(t)2 + ψ(t)2 ) = v1 (θ, ψ)θ + v2 (θ, ψ)ψ
dt= 2θψ [f 0 (θψ) − f 0 (−θψ)] . (82)for t ∈ QT and  small enough. Solving the right handformally yieldsv(θ, ψ) =Hence, we havew(t) ≤For concave f , we havef 0 (θψ) − f 0 (−θψ)
≤0
2θψ

(83)

d
(θ(t)2 + ψ(t)2 ) ≤ 0.
dt

(95)

As w(t) is nonincreasing for t ∈/ QT and the total length of
QT is at least βbαT c this shows thatw(T ) ≤and hence1.
−( 21 f 00 (0) + )t + c

1.
−( 12 f 00 (0) + )βbαT c + c

(96)

(84)

The training dynamics
hence converge with logarithmic 
1convergence rate O √t .

To intuitively understand why the continuous system converges with logarithmic convergence rate, note that near theequilibrium point
have in polar coordi√ we asymptotically√nates (θ, ψ) = ( w cos(φ), w sin(φ)):

Note that the standard choice f (t) = − log(1 + exp(−t))
is concave and satisfies f 00 (0) = − 14 < 0. Lemma C.1 ishence applicable and shows that the GAN training dynamics for the standard choice of f converge with logarithmic
convergence rate in the continuous case. The training behavior of the nonsaturating GAN on our example problem
is visualized in Figure 3b.

Now assume that f 0 (0) 6= 0 and f 00 (0) < 0.

φ̇ = f 0 (0) + O(|w|1/2 )
00

2

2

(85)
4

ẇ = 4f (0)θ ψ + O(|θψ| )= f 00 (0)w2 sin2 (2φ) + O(|w|4 ).

(86)
(87)

When we ignore higher order terms, we can solve this sys-

8
For solving the ODE we use the separation of variablestechnique and the identityZsin(2ax)
2 sin2 (ax)dx = x −
.
(88)
2a

Which Training Methods for GANs do actually Converge?

C.2. Wasserstein GAN-GP
In practice, it can be hard to enforce the Lipschitz-constraint
for WGANs. A practical solution to this problem was given
by Gulrajani et al. (2017), who derived a simple gradientpenalty with a similar effect as the Lipschitz-constraint.
The resulting training objective is commonly referred to as
WGAN-GP.
Similarly to WGANs, we find that WGAN-GP does not converge for the Dirac-GAN. A similar analysis also applies tothe DRAGAN-regularizer proposed by Kodali et al. (2017).
The regularizer proposed by Gulrajani et al. (2017) is givenbyγ
2
R(ψ) = Ex̂ (k∇x Dψ (x̂)k − g0 )(97)
2
where x̂ is sampled uniformly on the line segment between
two random points x1 ∼ pθ (x1 ), x2 ∼ pD (x2 ).
For the Dirac-GAN, it simplifies toR(ψ) =

γ(|ψ| − g0 )2
2

The corresponding gradient vector field is given by−ψṽ(θ, ψ) =.
θ − sign(ψ)γ(|ψ| − g0 )

(98)

(99)

Lemma C.3. WGAN-GP trained with simultaneous or alternating gradient descent with a fixed number of generator
and discriminator updates and a fixed learning rate h > 0
does not converge locally to the Nash equilibrium for the
Dirac-GAN.
Proof. First, consider simultaneous gradient descent. Assume that the iterates (θk , ψk ) converge towards the equilibrium point (0, 0). For almost all initial conditions9 we have(θk , ψk ) 6= (0, 0) for all k ∈ N. This implies(100)

and hence limk→∞ |ψk+1 − ψk | = h|g0 | 6= 0, showingthat (θk , ψk ) is not a Cauchy sequence. This contradicts theassumption that (θk , ψk ) converges to the equilibrium point(0, 0).
A similar argument also holds for alternating gradient descent.
The training behavior of WGAN-GP on our example problem is visualized in Figure 3d.
9Depending on γ, h and g0 modulo a set of measure 0.

C.3. Consensus optimization
Consensus optimization (Mescheder et al., 2017) is an algorithm that attempts to solve the problem of eigenvalues withzero real-part by introducing a regularization term that explicitly moves the eigenvalues to the left. The regularization
term in consensus optimization is given byγkv(θ, ψ)k22γ
= (k∇θ L(θ, ψ)k2 + k∇ψ L(θ, ψ)k2 ). (101)2

R(θ, ψ) =As was proved by Mescheder et al. (2017), consensus optimization converges locally for small learning rates h > 0
provided that the Jacobian v 0 (θ∗ , ψ ∗ ) is invertible.10
Indeed, for the Dirac-GAN we have

Note that the gradient vector field has a discontinuity atthe equilibrium point, as the gradient vector field takes onvalues with norm bigger than some fixed constant in every
neighborhood of the equilibrium point. As a result, we have|ψk+1 − ψk | = h|θk − γψk − sign(ψk )g0 |As for WGANs, we stress that this analysis only holds if thediscriminator is trained with a fixed number of discriminator
updates per generator update. Again, more careful trainingthat ensures that the discriminator is kept exactly optimal or
two-timescale training (Heusel et al., 2017) might be able
to ensure convergence for WGAN-GP.

Lemma C.4. The eigenvalues of the Jacobian of the gradient vector field for consensus optimization at the equilibrium
point are given by
λ1/2 = −γf 0 (0)2 ± if 0 (0)(102)

In particular, all eigenvalues have a negative real part−γf 0 (0)2 . Hence, simultaneous and alternating gradientdescent are both locally convergent using consensus optimization for small enough learning rates.
Proof. As was shown by Mescheder et al. (2017), the Jacobian of the modified vector field ṽ at the equilibrium pointisṽ 0 (0, 0) = v 0 (0, 0) − γv 0 (0, 0)| v 0 (0, 0).
In our case, this is−γf 0 (0)2
f 0 (0)−f 0 (0)−γf 0 (0)2 .

(103)

(104)

A simple calculation shows that the eigenvalues of ṽ 0 (0, 0)
are given by
λ1/2 = −γf 0 (0)2 ± if 0 (0).

(105)

This concludes the proof.
10
Mescheder et al. (2017) considered only the case of isolated
equilibrium points. However, by applying Theorem A.3, it isstraightforward to generalize their proof to the case where we areconfronted with a submanifold of equivalent equilibrium points.

Which Training Methods for GANs do actually Converge?

A visualization of consensus optimization for the DiracGAN is given in Figure 3e.
Unfortunately, consensus optimization has the drawbackthat it can introduce new spurious points of attraction to the
GAN training dynamics. While this is usually not a problem
for simple examples, it can be a problem for more complex
ones like deep neural networks.

Similarly, the gradient of L(θ, ψ) with respect to ψ is givenby∇ψ L(θ, ψ) = Epθ (x) [f 0 (Dψ (x))∇ψ Dψ (x)]− EpD (x) [f 0 (−Dψ (x))∇ψ Dψ (x)] . (109)
Proof. This is just the chain rule.

A similar regularization term as in consensus optimization
was also independently proposed by Nagarajan & Kolter(2017). However, Nagarajan & Kolter (2017) proposed toonly regularize the component ∇ψ L(θ, ψ) of the gradientvector field corresponding to the discriminator parameters.
Moreover, the regularization term is only added to the generator objective to give the generator more foresight. It canbe shown (Nagarajan & Kolter, 2017) that this simplified
regularization term can in certain situations also make thetraining dynamics locally convergent, but might be better
behaved at stationary points of the GAN training dynamics
that do not correspond to a local Nash-equilibrium. Indeed, a more detailed analysis shows that this simplified
regularization term behaves similarly to instance noise andgradient penalties (which we discussed in Section 3.2 andSection 3.3) for the Dirac-GAN.

Lemma D.2. Assume that (θ∗ , ψ ∗ ) satisfies Assumption I.
The Jacobian of the gradient vector field v(θ, ψ) at (θ∗ , ψ ∗ )is then


T
0
−KDG
0 ∗
∗
v (θ , ψ ) =.
(110)
KDG KDD

D. General convergence results

By Assumption I, Dψ∗ (x) = 0 in some neighborhoodof supp pD . Hence, we also have ∇x Dψ∗ (x) = 0 and∇2x Dψ∗ (x) = 0 for x ∈ supp pD . By taking the derivative of (108) with respect to θ and using ∇x Dψ∗ (x) =0 and ∇2x Dψ∗ (x) = 0 for x ∈ supp pD we see that∇2θ L(θ∗ , ψ ∗ ) = 0.

In this section, we prove Theorem 4.1. To this end, we
extend the convergence proof by Nagarajan & Kolter (2017)
to our setting. We show that by introducing the gradientpenalty terms Ri (θ, ψ), we can get rid of the assumption thatthe generator and data distributions locally have the same
support. As we have seen, this makes the theory applicable
to more realistic cases, where both the generator and data
distributions typically lie on lower dimensional manifolds.

The terms KDD and KDG are given by
KDD = 2f 00 (0) EpD (x) [∇ψ Dψ∗ (x)∇ψ Dψ∗ (x)T ] (111)
KDG = f 0 (0)∇θ Epθ (x) [∇ψ Dψ∗ (x)] |θ=θ∗(112)

Proof. First note that by the definition of v(θ, ψ) in (106),
the Jacobian v 0 (θ∗ , ψ ∗ ) of v(θ, ψ) is given by−∇2θ L(θ∗ , ψ ∗ ) −∇2θ,ψ L(θ∗ , ψ ∗ ).
(113)∇2θ,ψ L(θ∗ , ψ ∗ )∇2ψ L(θ∗ , ψ ∗ )To show (111) and (112), simply take the derivative of(109) with respect to θ and ψ and evaluate at it at (θ, ψ) =(θ∗ , ψ ∗ ).
We now take a closer look at the regularized vector field.
Recall that we consider the two regularization terms

D.1. Convergence proof
To prove Theorem 4.1, we first need to understand the local
structure of the gradient vector field v(θ, ψ). Recall that thegradient vector field v(θ, ψ) is defined as−∇θ L(θ, ψ)v(θ, ψ) :=(106)∇ψ L(θ, ψ)withL(θ, ψ) = Ep(z) [f (Dψ (Gθ (z)))]+ EpD (x) [f (−Dψ (x))] . (107)Lemma D.1. The gradient of L(θ, ψ) with respect to θ isgiven by∇θ L(θ, ψ) = Ep(z) f 0 (Dψ (Gθ (z)) [∇θ Gθ (z)]T· ∇x Dψ (Gθ (z)) . (108)γ
EpD (x) k∇x Dψ (x)k22γ
R2 (θ, ψ) := Epθ (x) k∇x Dψ (x)k2 .
2
R1 (θ, ψ) :=

(114)
(115)

As discussed in Section 4.1, the regularization is only applied to the discriminator. The regularized vector field ishence given by−∇θ L(θ, ψ)ṽ(θ, ψ) :=.
(116)∇ψ L(θ, ψ) − ∇ψ Ri (θ, ψ)
Lemma D.3. The gradient ∇ψ Ri (θ, ψ) of the regularization terms Ri , i ∈ {1, 2}, with respect to ψ are∇ψ R1 (θ, ψ) = γ EpD(x) [∇ψ,x Dψ (x)∇x Dψ (x)] (117)∇ψ R2 (θ, ψ) = γ Epθ (x) [∇ψ,x Dψ (x)∇x Dψ (x)] . (118)

Which Training Methods for GANs do actually Converge?

Proof. These equations can be derived by taking the derivative of (114) and (115) with respect to ψ.

Using the fact that Dψ (x) = 0 and ∇x Dψ (x) = 0 forx ∈ supp pD , we see that the Hessian of h(ψ) at ψ ∗ is

Lemma D.4. The second derivatives ∇2ψ Ri (θ∗ , ψ ∗ ) of theregularization terms Ri , i ∈ {1, 2}, with respect to ψ at(θ∗ , ψ ∗ ) are both given by∇2ψ h(ψ ∗ ) = 2 EpD (x) ∇ψ Dψ (x)∇ψ Dψ (x)TLDD := γ EpD (x) [∇ψ,x Dψ∗ (x)∇ψ,x Dψ∗ (x)T ] . (119)The second directional derivate ∂v2 h(ψ) is therefore

Moreover,bothregularization∇θ,ψ Ri (θ∗ , ψ ∗ ) = 0.

terms

satisfy+ ∇ψ,x Dψ (x)∇ψ,x Dψ (x)T



(125)


∂v2 h(ψ) = 2 EpD (x) |∇ψ Dψ (x)T v|2+ k∇x,ψ Dψ (x)vk2 = 0. (126)

Proof. ∇2ψ Ri (θ∗ , ψ ∗ ), i ∈ {1, 2}, can be computed by taking the derivative of (117) and (118) with respect to ψ andusing the fact that ∇x Dψ∗ (x) = 0 in a neighborhood of
supp pD .
∗∗Moreover, we clearly have ∇θ,ψ R1 (θ , ψ ) = 0, because
R1 does not depend on θ. To see that ∇θ,ψ R2 (θ∗ , ψ ∗ ) = 0,
take the derivative of (118) with respect to θ and use thefact that ∇x Dψ∗ (x) = 0 and ∇2x Dψ∗ (x) = 0 for x ∈supp pD .

By Assumption III, this can only hold if v is in the tangent
space of MD at ψ ∗ .
Lemma D.6. Assume that Assumption III holds. If w 6= 0
is not in the tangent space of MG at θ∗ , then KDG w 6= 0.
Proof. By Lemma D.2, we have
KDG w = f 0 (0) ∇θ Epθ (x) [∇ψ Dψ∗ (x)] |θ=θ∗ w= f 0 (0)∂w g(θ). (127)

0

∗

∗

As a result, the Jacobian ṽ (θ , ψ ) of the regularized gradient vector field at the equilibrium point is given by


T
0
−KDG
ṽ 0 (θ∗ , ψ ∗ ) =.
(120)
KDG KDD − LDD
For brevity, we define MDD := KDD − LDD .
To prove Theorem 4.1, we have to show that ṽ 0 (θ∗ , ψ ∗ ) is
well behaved when restricting it to the space orthogonal to
the tangent space of MG × MD at (θ∗ , ψ ∗ ):Lemma D.5. Assume that Assumptions II and III hold.
If v 6= 0 is not in the tangent space of MD at ψ ∗ , thenv̄ T MDD v < 0.
Proof. By Lemma D.2, we havehi2v T KDD v = 2f 00 (0) EpD (x) (∇ψ Dψ∗ (x)T v)(121)

and by Lemma D.4
hv T LDD v = γ EpD (x) k∇x,ψ Dψ∗ (x)vk
00

2

i

.

(122)

By Assumption II, we have f
0 and v T MDD v = 0 implies(0) < 0. Hence, v T M

DD v ≤∇ψ Dψ∗ (x)T v = 0 and∇x,ψ Dψ∗ (x)v = 0(123)

for all x ∈ supp pD .

forg(θ) := Epθ (x) [∇ψ Dψ∗ (x)] .

(128)

By Assumption III, this implies KDG w 6= 0 if w is not in
the tangent space of MG at θ∗ .
We are now ready to prove Theorem 4.1:
Theorem 4.1. Assume Assumption I, II and III hold for(θ∗ , ψ ∗ ). For small enough learning rates, simultaneousand alternating gradient descent for ṽ1 and ṽ2 are bothconvergent to MG × MD in a neighborhood of (θ∗ , ψ ∗ ).
Moreover, the rate of convergence is at least linear.
Proof. First note that by Lemma D.1 and Lemma D.3v(θ, ψ) = 0 for all points (θ, ψ) ∈ MG × MD , becauseDψ (x) = 0 and ∇x Dψ (x) = 0 for all x ∈ supp pD andψ ∈ MD . Hence, MG × MD consists only of equilibrium
points of the regularized gradient vector fields.
Let Tθ∗ MG and Tψ∗ MD denote the tangent spaces of MG
and MD at θ∗ and ψ ∗ .
We now want to show that both simultaneous and alternating
gradient descent are locally convergent to MG × MD forthe regularized gradient vector field ṽ(θ, ψ). To this end,
we want to apply Theorem A.3. By choosing local coordinates θ(α, γG ) and ψ(β, γD ) for MG and MD and using
Remark A.6, we can assume without loss of generality thatθ∗ = 0, ψ ∗ = 0 as well as
MG = Tθ∗ MG = {0}k × Rn−kLeth(ψ) := EpD (x) |Dψ (x)|2 + k∇x Dψ (x)k2 .

(124)

l

MD = Tψ∗ MD = {0} × R

m−l

(129)
.

(130)

Which Training Methods for GANs do actually Converge?

This allows us to write11 ṽ(θ, ψ) = ṽ(α, γG , β, γD )In order to apply Theorem A.3, we have to show that∇(α,β) ṽ(θ∗ , ψ ∗ ) only has eigenvalues with negative realpart.
By Lemma D.2, ∇(α,β) ṽ(θ∗ , ψ ∗ ) is of the form


T
0
−K̃DG
K̃DG K̃DD − L̃DD

(131)

where K̃DD , K̃DG and L̃DD denote the submatrices of
KDD , KDG and LDD corresponding to the (α, β) coordinates.
We now show that M̃DD := K̃DD − L̃DD is negative definite and K̃DG has full column rank.
To this end, first note thatṽ T M̃DD ṽ = v T MDD v(132)

with v T := (ṽ T , 0). Note that v ∈/ Tψ∗ MD for ṽ 6= 0.
Hence, by Lemma D.5 we have that ṽ T M̃DD ṽ < 0 if ṽ 6= 0.
As a result, we see that M̃DD is symmetric negative definite.
Similarly, for wT := (w̃T , 0), the components of KDG w
corresponding to the β-coordinates are given by K̃DG w̃.
Again, we have w ∈/ Tθ∗ MG for w̃ 6= 0. Hence, byLemma D.6 we have that KDG w 6= 0 if w̃ 6= 0. Because thecomponents of KDG w corresponding to the γD coordinates
are 0, this shows that K̃DG w̃ 6= 0. K̃DG therefore has fullcolumn rank.
Theorem A.7 now implies that all eigenvalues of∇(α,β) ṽ(θ∗ , ψ ∗ ) have negative real part. By Lemma A.4,
Lemma A.5 and Theorem A.3, simultaneous and alternating gradient descent are therefore both convergent to
MG × MD near (θ∗ , ψ ∗ ) for small enough learning rates.
Moreover, the rate of convergence is at least linear.
D.2. Extensions
In the proof of Theorem 4.1 we have assumed that f 00 (0) <0. This excludes the function f (t) = t which is used in
Wasserstein-GANs. We now show that our convergence
proof extends to the case where f (t) = t when we modify
Assumption III as little bit:Remark D.7. When we replace h(ψ) withh̃(ψ) := EpD (x) k∇x Dψ (x)k2(133)

and MD with M̃D := {ψ | h̃(ψ) = 0} the results ofTheorem 4.1 still hold for f (t) = t.
Proof. Almost everything in the proof of Theorem 4.1 stillholds for these modified assumptions. The only thing that11
By abuse of notation, we simply write θ = (α, γG ) andψ = (β, γD ).

we have to show is that MG × MD still consists only ofequilibrium points and that Lemma D.5 still holds in thissetting.
To see the former, note that by Lemma D.1 we still have∇θ L(θ, ψ) = 0 for (θ, ψ) ∈ MG × MD , because we have∇x Dψ (x) = 0 for ψ ∈ MD and x ∈ supp pD . On theother hand, for f (t) = t we also have ∇ψ L(θ, ψ) = 0
if θ ∈ MG , because for θ ∈ MG the definition of MG
implies that pθ = pD and hence, by Lemma D.1,
∇ψ L(θ, ψ) = Ex∼pD [∇ψ Dψ (x)]− Ex∼pD [∇ψ Dψ (x)] = 0. (134)
To see why Lemma D.5 still holds, first note that for f (t) =t, we have f 00 (0) = 0, so that by Lemma D.2 KDD = 0.
Hence,v T MDD v = −v T LDD v.
(135)
We therefore have to show that v T LDD v 6= 0 if v is not in
the tangent space of MD .
However, we have seen in the proof of Lemma D.5 thathi2v T LDD v = γ EpD (x) k∇x,ψ Dψ∗ (x)vk .
(136)
Hence v T LDD v = 0 implies ∇x,ψ Dψ∗ (x)v = 0 for x ∈supp pD and thus∂v2 h(ψ) = 2 EpD (x) k∇x,ψ Dψ (x)vk2 = 0. (137)
By Assumption III, this can only be the case if v is in thetangent space of MD . This concludes the proof.
In Section D.1, we showed that both regularizers R1 and R2from Section 4.1 make the GAN training dynamics locallyconvergent. A similar, but slightly more complex regularizer
was also proposed by Roth et al. (2017) who tried to find acomputationally efficient approximation to instance noise.
The regularizer proposed by Roth et al. (2017) is given bya linear combination of R1 and R2 where the weighting is
adaptively chosen depending on the logits of Dψ (x) of thecurrent discriminator at a data point x:
RRoth (θ, ψ) = Epθ (x) (1 − σ(Dψ (x)))2 k∇x Dψ (x)k2+ EpD (x) σ(Dψ (x)))2 k∇x Dψ (x)k2 (138)
Indeed, we can show that our convergence proof extends
to this regularizer (and a slightly more general class ofregularizers):
Remark D.8. When we replace the regularization terms R1
and R2 with

R3 (θ, ψ) = Epθ (x) w1 (Dψ (x))k∇x Dψ (x)k2+ EpD (x) w2 (Dψ (x))k∇x Dψ (x)k2 (139)

Which Training Methods for GANs do actually Converge?

so that w1 (0) > 0 and w2 (0) > 0, the results of Theorem 4.1 still hold.
Proof. Again, we have to show that MG × MD still consists only of equilibrium points and that Lemma D.5 stillholds in this setting.
However, by using ∇x Dψ (x) = 0 for x ∈ supp pD andψ ∈ MD , it is easy to see that ∇ψ R3 (θ, ψ) = 0 for all(θ, ψ) ∈ MG × MD , which implies that MG × MD still
consists only of equilibrium points.
To see why Lemma D.5 still holds in this setting, note that(after a little bit of algebra) we still have ∇θ,ψ R3 (θ∗ , ψ ∗ ) =0 and∇2ψ R3 (θ∗ , ψ ∗ ) =1(w1 (0) + w2 (0))LDD .
γ

(140)

The proof of Lemma D.5 therefore still applies in this setting.

E. Stable equilibria for unregularized GAN
training
In Section 2, we have seen that unregularized GAN training
is not always locally convergent to the equilibrium point.
Moreover, in Section 4, we have shown that zero-centeredgradient penalties make general GANs locally convergent
under some suitable assumptions.
While our results demonstrate that we cannot expect unregularized GAN training to lead to local convergence forgeneral GAN architectures, there can be situations where
unregularized GAN training has stable equilibria. Such equilibria usually require additional assumptions on the class ofrepresentable discriminators.
In this section, we identify two types of stable equilibria.
For the first class of stable equilibria, which we call energy
solutions, the equilibrium discriminator forms an energyfunction for the true data distributions and might be a partial
explanation for the success of autoencoder-based discriminators (Zhao et al., 2016; Berthelot et al., 2017). For thesecond class, which we call full-rank solutions, the discriminator learns a representation of the data distribution withcertain properties and might be a partial explanation for the
success of batch-normalization for training GANs (Radford
et al., 2015).
E.1. Energy Solutions
For technical reasons, we assume that supp pD defines a
C 1 -manifold in this section.
Energy solutions are solutions where the discriminator forms
a potential function for the true data distribution. Suchsolutions (θ∗ , ψ ∗ ) satisfy the following property:

Assumption I0 . We have pθ∗ = pD , Dψ∗ (x) = 0,
∇x Dψ∗ (x) = 0 and v T ∇2x Dψ∗ (x)v > 0 for all x ∈supp pD and v not in the tangent space of supp pD at x.
We also need a modified version of Assumption III which
ensures certain regularity properties of the reparameterization manifolds MG and MD near the equilibrium (θ∗ , ψ ∗ ).
To formulate Assumption III0 , we needg̃(ψ) := ∇θ Epθ (x) [Dψ (x)] θ=θ∗ .

(141)Assumption III0 . There are -balls B (θ∗ ) and B (ψ ∗ )around θ∗ and ψ ∗ so that MG ∩ B (θ∗ ) and MD ∩ B (ψ ∗ )define C 1 - manifolds. Moreover, the following holds:(i) if v is not in the tangent space of MD at ψ ∗ , then∂v g̃(ψ ∗ ) 6= 0.
(ii) if w is not in the tangent space of MG at θ∗ , then there
is a latent code z ∈ Rk so that ∇θ Gθ∗ (z)w is not in
the tangent space of supp pD at Gθ∗ (z) ∈ supp pD .
The first part of Assumption III0 implies that the generator gradients become nonzero whenever the discriminatormoves away from an equilibrium discriminator. The second
part of Assumption III0 means that every time the generator
leaves the equilibrium, it pushes some data point aways
from supp pD , i.e. the generator is not simply redistributing
mass on supp pD .
In Theorem E.2 we show that energy solutions lead to local
convergence of the unregularized GAN training dynamics.
For the proof, we first need a generalization of Lemma D.2:
Lemma E.1. Assume that (θ∗ , ψ ∗ ) satisfies Assumption I0 .
The Jacobian of the gradient vector field v(θ, ψ) at (θ∗ , ψ ∗ )is then given by
T
KGG −KDG
0 ∗∗v (θ , ψ ) =.
(142)
KDG KDD
The terms KDD and KDG are given by
KGG = −f 0 (0) Ep(z) [∇θ Gθ∗ (z)]T(143)∇2x Dψ∗ (Gθ∗ (z))∇θ Gθ∗ (z)KDD = 2f 00 (0) EpD (x) [∇ψ Dψ∗ (x)∇ψ Dψ∗ (x)T ] (144)T
KDG = f 0 (0) ∇θ Epθ (x) [∇ψ Dψ∗ (x)] |θ=θ∗(145)
Proof. Almost all parts of the proof of Lemma D.2 arestill valid. The only thing that remains to show is that∇2θ L(θ∗ , ψ ∗ ) = −KGG . To see this, just take the derivativeof (108) with respect to θ and use the fact that ∇x Dψ (x) =0 for x ∈ supp pD .
We are now ready to formulate our convergence result forenergy solutions:Which Training Methods for GANs do actually Converge?

Theorem E.2. Assume Assumption I0 , II and III0 hold for(θ∗ , ψ ∗ ). Moreover, assume that f 0 (0) > 0. For smallenough learning rates, simultaneous and alternating gradient descent for the (unregularized) gradient vector fieldv are both convergent to MG × MD in a neighborhoodof (θ∗ , ψ ∗ ). Moreover, the rate of convergence is at least
linear.
Proof (Sketch). The proof is similar to the proof of Theorem 4.1.
First, note that MG × MD still only consists of equilibrium points. Next, we introduce local coordinates and showthat for v not in the tangent space of MG at θ∗ , we havev T KGG v < 0. This can be shown using Lemma E.1, Assumption I0 and the second part of Assumption III0 .
Moreover, we need to show that for w not in the tangent
T w 6= 0. This can bespace of MD at ψ ∗ , we have KDG
shown by applying the first part of Assumption III0 .
The rest of the proof is the same as the proof of Theorem 4.1,
except that we have to apply Theorem A.8 instead of Theorem A.7.
Note that energy solutions are only possible, if the discriminator is able to satisfy Assumption I0 . This is not the casefor the Dirac-GAN from Section 2. However, if we use aquadratic discriminator instead, there are also energy solutions to the unregularized GAN training dynamics for the
Dirac-GAN. To see this, we can parameterize Dψ (x) asDψ (x) := ψ1 x2 + ψ2 x.

(146)

It is easy to check that the Dirac-GAN with a discriminator as in (146) indeed has energy solutions: every (θ, ψ)with θ = 0 and ψ2 = 0 defines an equilibrium point of
the Dirac-GAN and the GAN-training dynamics are locallyconvergent near this point if ψ1 > 0. Note however, thateven though all equilbria with ψ1 > 0 are points of attraction for the continuous GAN training dynamics, they may
not be attractors for the discretized system when ψ1 is large
and the learning rate h is fixed. In general, the conditioningof energy solutions depends on the condition numbers ofthe Hessians ∇2x Dψ∗ (x) at all x ∈ supp pD . Indeed, thepresence of ill-conditioned energy solutions might be one
possible explanation why WGAN-GP often works well inpractice although it is not even locally convergent for the
Dirac-GAN.
E.2. Full-Rank Solutions
In practice, Dψ (x) is usually implemented by a deep neuralnetwork. Such discriminators can be described by functionsof the formDψ (x) = ψ1T ηψ2 (x)(147)

with a vector-valued C 1 -functions ηψ2 and ψ = (ψ1 , ψ2 ).
ηψ2 can be regarded as a feature-representation of the data
point x.
We now state several assumptions that lead to local convergence in this situation.
The first assumption can be seen as a variant of Assumption I
adapted to this specific situation:
Assumption I00 . We have pθ∗ = pD and ψ1∗ = 0.
We again consider reparameterization manifolds, which we
define as follows in this section:
MG := {θ | pθ = pD }M0D := {ψ | ψ1 = 0}. (148)

Moreover, letg(θ) = Epθ (x) ηψ2∗ (x) .

(149)

Assumption III now becomes:
Assumption III00 . There is an -ball B (θ∗ ) around θ∗ sothat MG defines a C 1 - manifold12 . Moreover, the following
holds:(i) The matrix EpD (x) ηψ2∗ (x)ηψ2∗ (x)T has full rank.
(ii) if w is not in the tangent space of MG at θ∗ , then∂w g(θ∗ ) 6= 0.
We call a function ηψ2∗ that satisfies the first part of Assumption III00 a full-rank representation of pD . Moreover,if ηψ2∗ satisfies the second part of Assumption III00 , we callηψ2∗ a complete representation, because the second part ofAssumption III00 implies that every deviation from the Nashequilibrium pθ∗ = pD is detectable using ηψ2∗ .
In practice, complete full-rank representations might only
exist if the class of discriminators is very powerful or theclass of generators is limited. Especially the second part ofAssumption III00 might be hard to satisfy in practice. Moreover, finding such representations might be much harderthan finding equilibria for the regularized GAN-training
dynamics from Section 4.
Nonetheless, we have the following convergence result for
GANs that allow for complete full-rank representations:
Theorem E.3. Assume Assumption I0 , Assumption II and
III0 hold for (θ∗ , ψ ∗ ). For small enough learning rates,
simultaneous and alternating gradient descent for the (unregularized) gradient vector field v are both convergent to
MG × M0D in a neighborhood of (θ∗ , ψ ∗ ). Moreover, therate of convergence is at least linear.
Proof (Sketch). The proof is again similar to the proof of
Theorem 4.1. We again introduce local coordinates and
12

Note that M0D is a C 1 -manifold by definition in this setup.

Which Training Methods for GANs do actually Converge?

show that for w not in the tangent space of M0D at ψ ∗ , wehave wT KDD w < 0. To see this, note that w must have anonzero ψ1 component if it is not in the tangent space of
M0D at ψ ∗ . However, using (111), we see that the submatrix
of KDD corresponding to the ψ1 coordinates is given byK̃DD = 2f 00 (0) EpD (x) ηψ2∗ (x)ηψ2∗ (x)T .

(150)

This matrix is negative definite by Assumption II and thefirst part of Assumption III00 .
Moreover, by applying (112), we see that the componentof KDG w, w ∈ Rn , corresponding to the ψ1 coordinates isgiven by
∂w g(θ∗ ) = f 0 (0)∇θ Epθ (x) ηψ2∗ (x) θ=θ∗ w.

(151)

Using the second part of Assumption III00 , we therefore seethat for w not in the tangent space of MG at θ∗ , we have
KDG w 6= 0.
The rest of the proof is the same as the proof of Theorem 4.1.

For the Dirac-GAN from Section 2, we can obtain a complete full-rank representation, when we parameterize thediscriminator Dψ as Dψ (x) = ψ exp(x), i.e. if we set
ψ1 := ψ and ηψ2 (x) := exp(x). It is easy to check that ηψ2indeed defines a complete full-rank representation and thatthe Dirac-GAN is locally convergent to (θ∗ , ψ ∗ ) = (0, 0)
for this parameterization of Dψ (x).

F. Experiments
In this section, we describe additional experiments and give
more details on our experimental setup. If not noted otherwise, we always use the nonsaturating GAN-objective introduced by Goodfellow et al. (2014) for training the generator.
For WGAN-GP we use the generator and discriminator objectives introduced by Gulrajani et al. (2017).13
2D-Problems For the 2D-problems, we run unregularized
GAN training, R1 -regularized and R2 -regularized GAN
training as well WGAN-GP with 1 and 5 discriminator
update per generator update. We run each method on 4
different 2D-examples for 6 different GAN architectures.
The 4 data-distributions are visualized in Figure 8. All 6
GAN architectures consist of 4-layer fully connected neuralnetworks for both the generator and discriminator, where
we select the number of hidden units from {8, 16, 32} anduse select either leaky RELUs (i.e. ϕ(t) = max(t, 0.2t)) or
Tanh-activation functions.
13

The code to reproduce the experiments presented inthis section can be found under https://github.com/
LMescheder/GAN_stability.

For each method, we try both Stochastic Gradient Descent (SGD) and RMS-Prop with 4 different learning
rates: for SGD, we select the learning rate from {5 ·10−3 , 10−2 , 2 · 10−2 , 5 · 10−2 }. For RMSProp, we selectit from {5 · 10−5 , 10−4 , 2 · 10−4 , 5 · 10−4 }. For the R1 -,R2 - and WGAN-GP-regularizers we try the regularization
parameters γ = 1, γ = 3 and γ = 10. For each method
and architecture, we pick the hyperparameter setting which
achieves the lowest Wasserstein-1-distance to the true data
distribution. We train all methods for 50k iterations and wereport the Wasserstein-1-distance averaged over the last 10k
iterations. We estimate the Wasserstein-1-distance using
the Python Optimal Transport package14 by drawing 2048
samples from both the generator distribution and the true
data distribution.
The best solution found by each method for the “Circle”distribution is shown in Figure 9. We see that the R1 - andR2 -regularizers converge to solutions for which the discriminator is 0 in a neighborhood of the true data distribution.
On the other hand, unregularized training and WGAN-GP
converge to energy solutions where the discriminator forms
a potential function for the true data distribution. Please see
Section E.1 for details.
CIFAR-10 To test our theory on real-world tasks, we traina DC-GAN architecture (Radford et al., 2015) with 3 convolutional layers and no batch-normalization on the CIFAR-10
dataset (Krizhevsky & Hinton, 2009). We apply different
regularization strategies to stabilize the training. To compare the different regularization strategies, we measure theinception score (Salimans et al., 2016) over Wall-clock-time.
We implemented the network in the Tensorflow framework(Abadi et al., 2016). For all regularization techniques, we
use the RMSProp optimizer (Tieleman & Hinton, 2012)
with α = 0.9 and a learning rate of 10−4 .
For the R1 and R2 regularizers from Section 4.1 we use aregularization parameter of γ = 10. For the WGAN-GP
regularizer we also use a regularization parameter of γ = 10
as suggested by Gulrajani et al. (2017). We train all methods
using 1 discriminator update per generator update except
for WGAN-GP, for which we try both 1 and 5 discriminator
updates
The inception score over time for the different regularization
strategies is shown in Figure 6. As predicted by our theory,
we see that the R1 - and R2 -regularizers from Section 4.1
lead to stable training whereas unregularized GAN training
is not stable. We also see that WGAN-GP with 1 or 5 discriminator updates per generator update lead to similar final
inception scores on this architecture. The good behavior of
WGAN-GP is surprising considering the fact that it does
not even converge locally for the Dirac-GAN. One possible
14

http://pot.readthedocs.io

Which Training Methods for GANs do actually Converge?

explanation is that WGAN-GP oscillates in narrow circles
around the equilibrium which might be enough to produce
images of sufficiently high quality. Another possible explanation is that WGAN-GP converges to an energy or full-ranksolution (Section E) for this example.

Imagenet In this experiment, we use the R1 -regularizer tolearn a generative model of all 1000 Imagenet (Russakovsky
et al., 2015) classes at resolution 128 × 128 in a single GAN.
Because of the high variability of this dataset, this is known
to be a challenging task and only few prior works havemanaged to obtain recognizable samples for this dataset.
Prior works that report results for this dataset either showa high amount of mode collapse (Salimans et al., 2016;
Odena et al., 2017), report results only at a lower resolution(Hjelm et al., 2017) or use advanced normalization layers tostabilize the training (Miyato et al., 2018).
For the Imagenet experiment, we use ResNet-architectures15
for the generator and discriminator, both having 26 layers intotal. Both the generator and discriminator are conditionedon the labels of the input data. The architectures for the generator and discriminator are shown in Table 3. We use preactivation ResNet-blocks and Leaky RELU-nonlinearities
everywhere. We also multiply the output of the ResNet
blocks with 0.1. For the generator, we sample a latent variable z from a 256-dimensional unit Gaussian distribution
and concatenate it with a 256 dimensional embedding of
the labels, which we normalize to the unit sphere. The resulting 512-dimensional vector is then fed into the first fully
connected layer of the generator. The discriminator takes
as input an image and outputs a 1000 dimensional vector.
Depending on the label of the input, we select the corresponding index in this vector and use it as the logits for the
GAN-objective.
For training, we use the RMSProp optimizer with α = 0.99,
 = 10−8 and an initial learning rate of 10−4 . We use abatch size of 128 and we train the networks on 4 GeForceGTX 1080 Ti GPUs for 500.000 iterations. Similarly toprior work (Karras et al., 2017; Yazici et al., 2018; Gidel
et al., 2018), we use an exponential moving average16 withdecay 0.999 over the weights to produce the final model.
We find that while training this GAN without any regularization quickly leads to mode collapse, using the R1 regularizers from Section 4.1 leads to stable training.
15
We used more complicated architectures for the generator
and discriminator in an earlier version of this manuscript. The
simplified architectures presented in this version are more efficientand perform slightly better.
16
In an earlier version of this manuscript, we instead annealed
the learning rate which has a similar effect. However, taking
a moving average introduces fewer hyperparameters and led tohigher inception scores in our experiments.

Some random (unconditional) samples can be seen in Figure 10. Moreover, Figure 11 and Figure 12 show conditional
samples for some selected Imagenet classes. While not completely photorealistic, we find that our model can produce
convincing samples from all 1000 Imagenet classes.
We also compare the R1 -regularizer with WGAN-GP (with
1 discriminator update per generator update). For this comparison, we did not use the exponential moving average over
the weights and also did not normalize the embeddings ofthe labels to the unit sphere. The resulting inception score17
over the number of iterations is visualized in Figure 7. We
find that for this dataset and architecture we can achieve
higher inception scores when using the R1 -regularizer inplace of the WGAN-GP regularizer.
celebA and LSUN To see if the R1 -regularizers helps totrain GANs for high-resolution image distributions, we apply our method to the celebA dataset (Liu et al., 2015) and to
4 subsets of the LSUN dataset (Yu et al., 2015) with resolution 256×256. We use a similar training setup as for the Imagenet experiment, but we use a slightly different architecture(Table 4). As in the Imagenet-experiment, we use preactivation ResNet-blocks and Leaky RELU-nonlinearities everywhere and we multiply the output of the ResNet-blocks with
0.1. We implemented the network in the PyTorch framework and use the RMSProp optimizer with α = 0.99 and alearning rate of 10−4 . We again found that results can be(slightly) improved by using an exponential moving averagewith decay 0.999 over the weights to produce the final models. As a regularization term, we use the R1 -regularizer with
γ = 10. For the latent code z, we use a 256 dimensional
Gaussian distribution. The batch size is 64. We trained eachmodel for about 300.000 iterations on 2 GeForce GTX 1080
Ti GPUs.
We find that the R1 -regularizer successfully stabilizes training of this architecture. Some random samples can be seenin Figures 13, 14, 15, 16 and 17.
celebA-HQ In addition to the generative model for celebA
with resolution 256 × 256, we train a GAN on the celebAHQ dataset (Karras et al., 2017) with resolution 1024×1024.
We use almost the same architecture as for celebA (Table 4),
but add two more levels to increase the resolution from256 × 256 to 1024 × 1024 and decrease the number offeatures from 64 to 16. Because of memory constraints,
we also decrease the batch size to 24. As in the previous
experiments, we use an exponential moving average withdecay 0.999 over the weights to produce the final model.
In contrast to Karras et al. (2017), we train our model endto-end during the whole course of training, i.e. we do not
17

For measuring the inception score, we use the public implementation from http://github.com/sbarratt/inception-score-pytorch.

Which Training Methods for GANs do actually Converge?

use progressively growing GAN-architectures (nor any of
the other techniques used by Karras et al. (2017) to stabilize the training). We trained the model for about 300.000
iterations on 4 GeForce GTX 1080 Ti GPUs. We find thatthe simple R1 -regularizer stabilizes the training, allowing
our model to converge to a good solution without using
a progressively growing GAN. Some random samples areshown in Figure 18.

Figure 6. Inception score over time for various regularization strategies when training on CIFAR-10. While the inception score can beproblematic for evaluating probabilistic models (Barratt & Sharma,2018), it still gives a rough idea about the convergence and stability
properties of different training methods.

Figure 7. Inception score over the number of iterations for GAN
training with R1 - and WGAN-GP-regularization when training on
Imagenet. We find that R1 -regularization leads to higher inception
scores for this dataset and GAN-architecture.
Layer

output size

filterFully Connected
Reshape
TransposedConv2D
TransposedConv2D
TransposedConv2D

256 · 4 · 4
256 × 4 × 4
128 × 8 × 8
64 × 16 × 16
3 × 32 × 32

256 → 256 · 4 · 4
256 → 128
128 → 64
64 → 3(a) Generator architecture
Layer

output size

filter

Conv2D
Conv2D
Conv2D
Reshape
Fully Connected

64 × 16 × 16
128 × 8 × 8
256 × 4 × 4
256 · 4 · 4
256 · 4 · 4

3 → 6464 → 128
128 → 256
256 · 4 · 4 → 1(b) Discriminator architecture
Table 2. Architectures for CIFAR-10-experiment.

Which Training Methods for GANs do actually Converge?

(a) 2D-Gaussian(b) Line(c) Circle(d) Four lines

Figure 8. The four 2D-data distributions on which we test the different algorithms.

(a) unregularized(b) R1(c) R2(d) WGAN-GP-1(e) WGAN-GP-5

Figure 9. Best solutions found by the different algorithms for learning a circle. The blue points are samples from the true data distribution,
the orange points are samples from the generator distribution. The colored areas visualize the gradient magnitude of the equilibrium
discriminator. We find that while the R1 - and R2 -regularizers converge to equilibrium discriminators that are 0 in a neighborhood of thetrue data distribution, unregularized training and WGAN-GP converge to energy solutions (Section E.1).

(a) Training distribution(b) Random samples

Figure 10. Unconditional random samples for a GAN trained on the ILSVRC dataset (Russakovsky et al., 2015) with resolution 128 × 128.
The final inception score is 30.2 ± 0.5.

Which Training Methods for GANs do actually Converge?

(a) tench(b) papillon(c) weevil(d) admiral(e) lighthouse(f) bell cote(g) castle(h) dam(i) dock

Figure 11. Class conditional random samples for a GAN trained on the Imagenet dataset.

Which Training Methods for GANs do actually Converge?

(a) home theater(b) jack-o’-lantern(c) passenger car(d) police van(e) rugby ball(f) ski(g) trifle(h) pizza(i) valley

Figure 12. Class conditional random samples for a GAN trained on the Imagenet dataset.

Which Training Methods for GANs do actually Converge?

Figure 13. Random samples for a GAN trained on the celebA dataset (Liu et al., 2015) (256 × 256) for a DC-GAN (Radford et al., 2015)
based architecture with additional residual connections (He et al., 2016). For both the generator and the discriminator, we do not use batchnormalization.

Which Training Methods for GANs do actually Converge?

fFigure 14. Random samples for a GAN trained on the LSUN-bedroom dataset (Yu et al., 2015) (256 × 256) for a DC-GAN (Radford
et al., 2015) based architecture with additional residual connections (He et al., 2016). For both the generator and the discriminator, we do
not use batch normalization.

Which Training Methods for GANs do actually Converge?

Figure 15. Random samples for a GAN trained on the LSUN-church dataset (Yu et al., 2015) (256 × 256) for a DC-GAN (Radford et al.,2015) based architecture with additional residual connections (He et al., 2016). For both the generator and the discriminator, we do notuse batch normalization.

Which Training Methods for GANs do actually Converge?

Figure 16. Random samples for a GAN trained on the LSUN-bridge dataset (Yu et al., 2015) (256 × 256) for a DC-GAN (Radford et al.,2015) based architecture with additional residual connections (He et al., 2016). For both the generator and the discriminator, we do notuse batch normalization.

Which Training Methods for GANs do actually Converge?

Figure 17. Random samples for a GAN trained on the LSUN-tower dataset (Yu et al., 2015) (256 × 256) for a DC-GAN (Radford et al.,2015) based architecture with additional residual connections (He et al., 2016). For both the generator and the discriminator, we do notuse batch normalization.

Which Training Methods for GANs do actually Converge?

Figure 18. Random samples for a GAN trained on the celebA-HQ dataset (Karras et al., 2017) (1024 × 1024) for a DC-GAN (Radford
et al., 2015) based architecture with additional residual connections (He et al., 2016). During the whole course of training, we directlytrain the full-resolution generator and discriminator end-to-end, i.e. we do not use any of the techniques described in Karras et al. (2017)
to stabilize the training.

Which Training Methods for GANs do actually Converge?

Layeroutput size

filterFully Connected
Reshape1024 · 4 · 41024 × 4 × 4

512 → 1024 · 4 · 4-Resnet-BlockResnet-Block
NN-Upsampling

1024 × 4 × 41024 × 4 × 41024 × 8 × 81024 → 1024 → 1024
1024 → 1024 → 1024-

Resnet-BlockResnet-Block
NN-Upsampling

1024 × 8 × 81024 × 8 × 81024 × 16 × 16

1024 → 1024 → 1024
1024 → 1024 → 1024-

Resnet-BlockResnet-Block
NN-Upsampling

512 × 16 × 16
512 × 16 × 16
512 × 32 × 32

1024 → 512 → 512
512 → 512 → 512-

Resnet-BlockResnet-Block
NN-Upsampling

256 × 32 × 32
256 × 32 × 32
256 × 64 × 64

512 → 256 → 256
256 → 256 → 256-

Resnet-BlockResnet-Block
NN-Upsampling

128 × 64 × 64
128 × 64 × 64
128 × 128 × 128

256 → 128 → 128
128 → 128 → 128-

Resnet-BlockResnet-Block
Conv2D

64 × 128 × 128
64 × 128 × 128
3 × 128 × 128

128 → 64 → 64
64 → 64 → 64
64 → 3Layer

output size

filterFully Connected
Reshape1024 · 4 · 41024 × 4 × 4

512 → 1024 · 4 · 4-Resnet-Block
NN-Upsampling

1024 × 4 × 41024 × 8 × 81024 → 1024 → 1024-

Resnet-Block
NN-Upsampling

1024 × 8 × 81024 × 16 × 16

1024 → 1024 → 1024-

Resnet-Block
NN-Upsampling

512 × 16 × 16
512 × 32 × 32

1024 → 512 → 512-

Resnet-Block
NN-Upsampling

256 × 32 × 32
256 × 64 × 64

512 → 256 → 256-

Resnet-Block
NN-Upsampling

128 × 64 × 64
128 × 128 × 128

256 → 128 → 128-

Resnet-Block
NN-Upsampling

64 × 128 × 128
64 × 256 × 256

128 → 64 → 64-

Resnet-Block
Conv2D

64 × 256 × 256
3 × 256 × 256

64 → 64 → 64
64 → 3(a) Generator architecture(a) Generator architecture

Layer

output size

filter

Conv2D

64 × 128 × 128

3 → 64

Resnet-BlockResnet-Block
Avg-Pool2D

64 × 128 × 128
128 × 128 × 128
128 × 64 × 64

64 → 64 → 64
64 → 64 → 128-

Resnet-BlockResnet-Block
Avg-Pool2D

128 × 64 × 64
256 × 64 × 64
256 × 32 × 32

128 → 128 → 128
128 → 128 → 256-

Resnet-BlockResnet-Block
Avg-Pool2D

256 × 32 × 32
512 × 32 × 32
512 × 16 × 16

256 → 256 → 256
256 → 256 → 512-

Resnet-BlockResnet-Block
Avg-Pool2D

512 × 16 × 16
1024 × 16 × 16
1024 × 8 × 8

512 → 512 → 512
512 → 512 → 1024-

Resnet-BlockResnet-Block
Avg-Pool2D

1024 × 8 × 81024 × 8 × 81024 × 4 × 41024 → 1024 → 1024
1024 → 1024 → 1024-

Resnet-BlockResnet-Block
Fully Connected1024 × 4 × 41024 × 4 × 41024 · 4 · 41024 → 1024 → 1024
1024 → 1024 → 1024
1024 · 4 · 4 → 1000

Layer

output size

filter

Conv2D

64 × 256 × 256

3 → 64

Resnet-Block
Avg-Pool2D

64 × 256 × 256
64 × 128 × 128

64 → 64 → 64-

Resnet-Block
Avg-Pool2D

128 × 128 × 128
128 × 64 × 64

64 → 64 → 128-

Resnet-Block
Avg-Pool2D

256 × 64 × 64
256 × 32 × 32

128 → 128 → 256-

Resnet-Block
Avg-Pool2D

512 × 32 × 32
512 × 16 × 16

256 → 256 → 512-

Resnet-Block
Avg-Pool2D

1024 × 16 × 16
1024 × 8 × 8

512 → 512 → 1024-

Resnet-Block
Avg-Pool2D

1024 × 8 × 81024 × 4 × 41024 → 1024 → 1024-

Fully Connected1024 · 4 · 41024 · 4 · 4 → 1(b) Discriminator architecture

Table 4. Architectures for LSUN- and celebA-experiments.
(b) Discriminator architecture

Table 3. Architectures for Imagenet-experiment.

Which Training Methods for GANs do actually Converge?

(a) Standard GAN(b) Non-saturating GAN(c) WGAN(d) WGAN-GP(e) Consensus optimization(f) Instance noise(g) Gradient penalty(h) Gradient penalty (CR)

Figure 19. Convergence properties of different GAN training algorithms using simultaneous gradient descent. The shaded area in
Figure 19c visualizes the set of forbidden values for the discriminator parameter ψ. The starting iterate is marked in red.

(a) Standard GAN(b) Non-saturating GAN(c) WGAN(d) WGAN-GP(e) Consensus optimization(f) Instance noise(g) Gradient penalty(h) Gradient penalty (CR)

Figure 20. Convergence properties of different GAN training algorithms using alternating gradient descent with 1 discriminator
update per generator update The shaded area in Figure 20c visualizes the set of forbidden values for the discriminator parameter ψ.
The starting iterate is marked in red.

Which Training Methods for GANs do actually Converge?

(a) Standard GAN(b) Non-saturating GAN(c) WGAN(d) WGAN-GP(e) Consensus optimization(f) Instance noise(g) Gradient penalty(h) Gradient penalty (CR)

Figure 21. Convergence properties of different GAN training algorithms using alternating gradient descent with 5 discriminator
updates per generator update. The shaded area in Figure 21c visualizes the set of forbidden values for the discriminator parameter
ψ. The starting iterate is marked in red.

Which Training Methods for GANs do actually Converge?

(a) SimGD(b) AltGD (nd = 1)(c) AltGD (nd = 5)

Figure 22. Convergence properties of our GAN using two time-scale training as proposed by Heusel et al. (2017). For the Dirac-GAN we
do not see any sign of convergence when training with two time-scales. The starting iterate is marked in red.