The Numerics of GANs: Supplementary Material

Lars Mescheder
Autonomous Vision GroupMPI Tübingenlars.mescheder@tuebingen.mpg.de

Sebastian Nowozin
Machine Intelligence and Perception Group
Microsoft Researchsebastian.nowozin@microsoft.com

Andreas Geiger
Autonomous Vision GroupMPI Tübingen
andreas.geiger@tuebingen.mpg.de

Abstract
This document contains proofs that were omitted in the main text of the paper“The Numerics of GANs” as well as additional theoretical results. We also include
additional experimental results and demonstrate that our method leads to stabletraining of GANs on a variety of architectures and divergence measures.

ProofsThis section contains proofs that were omitted in the main text.
Smooth two player games
Lemma 1. For zero-sum games, v 0 (x) is negative (semi-)definite if and only if ∇2φ f (φ, θ) is negative(semi-)definite and ∇2θ f (φ, θ) is positive (semi-)definite.
Proof. We have for any w = (w1 , w2 ) 6= 0
wT v 0 (x)w = w1T ∇2φ f (φ, θ)w1 − w2T ∇2θ f (φ, θ)w2 .
Hence, we have wT v 0 (x)w < 0 for all vectors w 6= 0 if and only if w1T ∇2φ f (φ, θ)w1
w2T ∇2θ f (φ, θ)w2 > 0 for all vectors w1 , w2 6= 0.

(19)< 0 andThis shows that v 0 (x) is negative definite if and only if ∇2φ f (φ, θ) is negative definite and ∇2θ f (φ, θ)is positive definite.
A similar proof shows that v 0 (x) is negative semi-definite if and only if ∇2φ f (φ, θ) is negative
semi-definite and ∇2θ f (φ, θ) is positive semi-definite.
Corollary 2. For zero-sum games, v 0 (x̄) is negative semi-definite for any local Nash-equilibriumx̄. Conversely, if x̄ is a stationary point of v(x) and v 0 (x̄) is negative definite, then x̄ is a local
Nash-equilibrium.
Proof. If x̄ is a local Nash-equilibrium, ∇2φ f (φ̄, θ̄) is negative semi-definite and ∇2θ f (φ̄, θ̄) is positive
semi-definite, so v 0 (x̄) is negative definite by Lemma 1.
Conversely, if v 0 (x̄) is negative definite, ∇2φ f (φ̄, θ̄) is negative definite and ∇2θ f (φ̄, θ̄) positive
definite by Lemma 1. This implies that x̄ is a local Nash-equilibrium of the two-player game definedby f .
31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.

Convergence theory
Proposition 3. Let F : Ω → Ω be a continuously differential function on an open subset Ω of Rn
and let x̄ ∈ Ω be so that
1. F (x̄) = x̄, and2. the absolute values of the eigenvalues of the Jacobian F 0 (x̄) are all smaller than 1.
Then there is an open neighborhood U of x̄ so that for all x0 ∈ U , the iterates F (k) (x0 ) converge
to x̄. The rate of convergence is at least linear. More precisely, the error kF (k) (x0 ) − x̄k is inO(|λmax |k ) for k → ∞ where λmax is the eigenvalue of F 0 (x̄) with the largest absolute value.
Proof. See [6], Proposition 4.4.1.
Lemma 4. Assume that A ∈ Rn×n only has eigenvalues with negative real-part and let h > 0. Thenthe eigenvalues of the matrix I + h A lie in the unit ball if and only ifh<1|<(λ)|

2
1+



=(λ)
<(λ)

(10)

2

for all eigenvalues λ of A.
Proof. For λ = −a + b i with a > 0 we have
|1 + h λ|2 = (1 − h a)2 + h2 b2 = 1 + h2 b2 + h2 a2 − 2 h a.

(20)

For h > 0, this is smaller than 1 if and only ifh<2ab2 + a2=2a−12(b/a) + 1.

(21)

Corollary 5. If v 0 (x̄) only has eigenvalues with negative real-part at a stationary point x̄, then
Algorithm 1 is locally convergent to x̄ for h > 0 small enough.
Proof. This is a direct consequence of Proposition 3 and Lemma 4.
Lemma 6. Assume h > 0 and A(x) invertible for all x. Then x̄ is a fixed point of (15) if and only if
it is a stationary point of v. Moreover, if x̄ is a stationary point of v, we haveF 0 (x̄) = I + hA(x̄)v 0 (x̄).

(16)

Proof. If v(x̄) = 0, then F (x̄) = x̄, so x̄ is a fixed point of F . Conversely, if x̄ satisfies F (x̄) = x̄,
we have A(x̄) v(x̄) = 0. Because we assume A(x̄) to be invertible, this shows v(x̄) = 0.
Now, the ith partial derivative of F (x) is given by∂xi F (x) = bi + h ∂xi A(x) v(x) + hA(x)∂xi v(x)(22)

where bi denotes the ith unit basis vector. For a fixed point x̄ we have by the first part of the proofv(x̄) = 0 and therefore∂xi F (x̄) = bi + hA(x̄)∂xi v(x̄).
(23)
This shows
F 0 (x̄) = I + hA(x̄) v 0 (x̄).
(24)

Lemma 7. Let A(x) = I − γv 0 (x)T and assume that v 0 (x̄) is negative semi-definite and invertible6 .
Then A(x̄)v 0 (x̄) is negative definite.
Note that v 0 (x̄) is usually not symmetric and therefore it is possible that v 0 (x̄) is negative semi-definite andinvertible but not negative-definite.
613

Proof. We have for all w 6= 0:
wT A(x̄)v 0 (x̄)w = wT v 0 (x̄)w − γkv 0 (x̄)wk2 ≤ −γkv 0 (x̄)wk2 < 0

(25)

0

as v (x̄)w 6= 0 for w 6= 0.
Corollary 8. Let v(x) be the associated gradient vector field of a two-player zero-sum game andA(x) = I − γv 0 (x)T . If x̄ is a local Nash-equilibrium, then there is an open neighborhood U of x̄ sothat for all x0 ∈ U , the iterates F (k) (x0 ) converge to x̄ for h > 0 small enough.
Proof. This is a direct consequence of Proposition 3, Lemma 6 and Lemma 7.
Lemma 9. Assume that A ∈ Rn×n is negative semi-definite. Let q(γ) be the maximum of |=(λ)||<(λ)|(possibly infinite) with respect to λ where λ denotes the eigenvalues of A − γAT A and <(λ) and=(λ) denote their real and imaginary part respectively. Moreover, assume that A is invertible with
|Av| ≥ ρ|v| for ρ > 0 and let|v̄ T (A + AT )v|
c = minn T(17)v∈S(C ) |v̄ (A − AT )v|
where S(Cn ) denotes the unit sphere in Cn . Thenq(γ) ≤

1.
c + 2ρ2 γ(18)

Proof. Let v ∈ Cn \ {0} be any eigenvector of B := A − γAT A and λ ∈ C the corresponding
eigenvalue. We can assume w.l.o.g. that kvk = 1.
Thenλ = λv̄ T v = v̄ T Bv.

(26)<(λ) =

λ + λ̄
1= v̄ T (B + B T ) v
2
2

(27)

=(λ) =λ − λ̄1 T=v̄ (B − B T ) v.
2i
2i

(28)

This implies that

and, similarly,

Consequently, we have=(λ)|v T (B − B T ) v|
= T<(λ)|v̄ (B + B T ) v|
and thus|v T (B − B T ) v|.
v∈S(C ) |v̄ T (B + B T ) v|

q(γ) ≤ maxn

(29)

(30)

However, we haveB − B T = A − ATand

B + B T = A + AT − 2γAT A.

(31)

This implies, because A is negative semi-definite, that|v T (B − B T ) v|
|v T (A − AT ) v|=.
|v̄ T (B + B T ) v|
|v̄ T (A + AT ) v| + 2 γkAvk2(32)

Because kAvk2 ≥ ρkvk2 = ρ this implies the assertion.

Additional Theoretical Results
This section contains some additional theoretical results. On the one hand, we demonstrate how theconvergence of gradient ascent and common modifications like momentum and gradient rescaling
can be analyzed naturally using Proposition 3. On the other hand, we analyze alternating gradientascent for two-player games and show that for small step sizes h > 0 it is locally convergent towards
a Nash-equilibrium if all the eigenvalues of the Jacobian of the associated gradient vector field havenegative real-part. We also discuss the bias of consensus optimization in a mini-batch setting andpresent a possible way to debias it.
14

Gradient AscentLetv(x) = ∇f (x).
Then v 0 (x) = ∇2 f (x) is the Hessian-matrix of f at x. Let(33)

F (x) = x + h v(x).

(34)

A direct consequence of Proposition 3 is
Proposition 10. Any fixed point of (34) is a stationary point of the gradient vector field v(x).
Moreover, if ∇2 f (x) is negative definite, the fixed point iteration defined by F is locally convergent
towards x̄ for small h > 0.
Proof. This follows directly from Proposition 3.
MomentumLet v(x) be a vector field. Using momentum, the operator F can be written as
F (x, m) = (x, m) + h G(x, m)(35)

with G(x, m) = (m, v(x) − γ m). The Jacobian of G is given by0I
G0 (x, m) =.
v 0 (x) −γI(36)

Lemma 11. λ is an eigenvalue of G0 (x, m) if and only if λ(λ + γ) is an eigenvalue of v 0 (x).
Proof. Let (w1 , w2 ) be an eigenvector of G0 (x, m) as in (36) with associated eigenvalue λ. Thenλw1 = w2(37)

λw2 = v 0 (x̄)w1 − γw2 ,

(38)

0

showing that λ(λ+γ)w1 = v (x̄)w1 . As w1 = 0 implies w2 = 0, this shows that w1 is an eigenvectorof v 0 (x̄) with associated eigenvalue λ(λ + γ).
Conversely, let λ(λ + γ) be an eigenvalue of v 0 (x) to the eigenvector w. We have  

 
0
I
w
λw
w
=
=
λ
v 0 (x) −γI
λw
λ(λ + γ)w − γλw
λw

(39)

showing that λ is an eigenvalue of G0 (x, m) to the eigenvector (w, λw).
Corollary 12. Any fixed point (x̄, m̄) of (35) satisfies m̄ = 0 and v(x̄) = 0. Moreover, assume that
γ > 0 and that v 0 (x̄) only has real negative eigenvalues. Then the fixed point iteration defined by(35) is locally convergent towards (x̄, m̄) for small h > 0.
Proof. It is easy to see that any fixed point of (35) must satisfy m̄ = 0 and v(x̄) = 0. If all
eigenvalues µ of v 0 (x̄) are real and non-positive, then all solutions to λ(λ + γ) = µ have negative
real part, showing local convergence by Proposition 3.
Note that the proof of Corollary 12 breaks down if v 0 (x̄) has complex eigenvalues, as it is often thecase for the associated gradient vector field of two-player games.
Gradient Rescaling
In this section we investigate the effect of gradient rescaling as used in ADAM and RMSProp on
local convergence. In particular, lethx + √β+v(x)F (x, β) =(40)
(1 − α)β + αkv(x)k2
for some α > 0. The Jacobian of (40) is then given byhh
I + √β+
v 0 (x) − 2(β+)3/2 v(x)F 0 (x, β) =.
α v 0 (x)T v(x)
1−α
15

(41)

Algorithm 3 Alternating Gradient Ascent
1: while not converged do2:
φ ← φ + h ∇φ f (θ, φ)3:θ ← θ + h ∇θ g(θ, φ)
4: end while

Proposition 13. Any fixed point (x̄, β̄) of (40) satisfies β̄ = 0 and v(x̄) = 0. Moreover, assume thatα ∈ (0, 1) and that all eigenvalues of I + √h v 0 (x̄) lie in the unit ball. Then the fixed point iteration
defined by (40) is locally convergent towards (x̄, β̄).
Proof. It is easy to see that any fixed point (x̄, β̄) of (40) must satisfy β̄ = 0 and v(x̄) = 0. We
therefore haveI + √h v 0 (x)00F (x̄, β̄) =
.
(42)
0
1−α
The eigenvalues of (42) are just the eigenvalues of I + √h v 0 (x) and 1 − α which all lie in the unit
ball by assumption.
Alternating Gradient Ascent
Alternating Gradient Ascent (AltGA) applies gradient ascent-updates for the two players in analternating fashion, see Algorithm 3. For a theoretical analysis, we more generally regard fixed-point
methods that iteratively apply a function of the form
F (x) = F2 (F1 (x))(43)

to x. By the chain rule, the Jacobian of F at x is given by
F 0 (x) = F20 (F1 (x)))F10 (x).

(44)

Assume now that F1 (x) = x + hG1 (x) and F2 (x) = x + hG2 (x). Then, if x̄ is a fixed point of both
F1 and F2 , we haveF 0 (x̄) = (I + hG02 (x̄))(I + hG01 (x̄)) = I + hG01 (x̄) + hG02 (x̄) + h2 G02 (x̄)G01 (x̄).

(45)

Let A(x) := G02 (x)+G01 (x). Notice that for alternating gradient ascent, A(x) is equal to the Jacobianv 0 (x) of the gradient vector field v(x). The eigenvalues of Ãh (x) := A(x) + hG02 (x)G01 (x) depend
continuously on h. Hence, for h small enough, all eigenvalues of Ãh (x) will be arbitrarily close to
the eigenvalues of A(x).
As a consequence, we the following analogue of Corollary 5:
Proposition 14. If the Jacobian v 0 (x̄) of the gradient vector field v(x) at a fixed point x̄ only has
eigenvalues with negative real part, Algorithm 3 is locally convergent to x̄ for h > 0 small enough.
Proof. This follows directly from the derivation above and Proposition 3.

16

The bias in consensus optimization
To calculate ∇L(x) in consensus optimization, we need estimates of L(x), which is defined as
L(x) :=

1
kv(x)k2 .
2(46)

In a mini-batch setting, we can obtain unbiased estimates of v(x) by averaging all gradients vi (x) forthe examples in the mini-batch:Bv(x) ≈ vB (x) =1 X
vi (x).
B i=1

(47)

However, when we substitute this estimate for v(x) in the definition of L(x), we obtain a biased
estimator of L(x) as the next lemma shows. The lemma also gives an explicit expression for the bias.
Lemma 15. Assume that vi (x), i = 1, . . . , B, are independent and identically distributed unbiased
estimates of v(x), i.e. E[vi (x)] = v(x) for all i. Let
B

LB (x) :=1 1 X
2vi (x) .
2 B i=1(48)

Then LB (x) is a biased estimator of L(x) with
E[LB (x)] = L(x) +1
Var[v1 (x)].
2B

(49)

Proof. We have111
Var[vB (x)] = E[kvB (x)k2 ] − [k E vB (x)k2 ] = E[LB (x)] − L(x).
2
2
2

(50)

Moreover,B

Var[vB (x)] =1
1 X
Var[vi (x)] = Var[v1 (x)].
B 2 i=1
B(51)

Together, this yields the assertion.
Note that the bias is going down linearly with the batch size. In practice, we found that a batch sizeof 64 is usually sufficient to make consensus optimization work.
Alternatively, as was also noted by Ferenc Huszár in a blog post on www.inference.vc, it is possible
to obtain unbiased estimates of L(x) by subtracting 1/(2B − 2) times the sample variance of thevi (x) from LB (x).
We leave a closer investigation of the benefits and disadvantages of this variant of consensus optimization to future research.

17

Additional Experimental Results
SimGA

AltGA

Consensus
Optimization

DC-GAN+ no batchnormalization+ constant numberof filters in eachlayer

+ Jensen-Shannon
objective

Figure 6: Comparison of the results obtained by training different architectures with simultaneous
gradient ascent, alternating gradient ascent and consensus optimization: we train a standard DC-GAN
architecture (with 4 convolutional layers), a DC-GAN architecture without batch-normalization inneither the discriminator nor the generator, a DC-GAN architecture that additionally has a constantnumber of filters in each layer and a DC-GAN architecture that additionally uses the Jensen-Shannon
objective for the generator that was originally derived in [10], but usually not used in practice as
is was found to hamper convergence. While our algorithm struggles with the original DC-GAN
architecture (with batch-normalization), it successfully trains all other architectures.

18

(a) DC-GAN(b) + no batch-normalization(c) + constant number of filters in each layer(d) + Jensen-Shannon objective

Figure 7: Inception score over the number of iterations for the results in Figure 6. To investigate if
consensus optimization can alternatively be interpreted as a way of smoothing the discriminator, we
also conducted experiments where we removed the regularization term from the generator loss but
keep it for the discriminator loss. We call the corresponding algorithm smoothing optimizer. While
the smoothing optimizer trains a DC-GAN with batch-normalization successfully where consensus
optimization fails, it usually performs worse than consensus optimization.

(a) + constant number of filters in each layer(b) + Jensen-Shannon objective

Figure 8: Inception score over the number of iterations for training a the architecture from Figure 7
with an additional fully connected layer in the discriminator on cifar-10. We observe similar results
as in Figure 7, but the smoothing optimizer fails for the architecture with a constant number of filtersin each layer where consensus optimization succeeds.

19(a) Standard GAN objective(b) Jensen-Shannon divergence(c) Indicator divergence

Figure 9: Using consensus optimization to train models with different generator objectives corresponding to different divergence functions: all samples were generated by a DC-GAN model withoutbatch-normalization and with a constant number of filters in each layer. We see that consensus
optimization can be used to train GANs with a large variety of divergence functions.

(a) Simultaneous Gradient Ascent(b) Alternating Gradient Ascent(c) Consensus Optimization

Figure 10: Samples generated from a model where both the generator and discriminator are based ona DC-GAN [21]. However, we use no batch-normalization, a constant number of filters in each layer
and additional RESNET-layers. We see that only our method results in visually compelling results.
While simultaneous gradient ascent completely fails to train the model, alternating gradient ascentresults in a bad solution. Although alternating gradient ascent does better than simultaneous gradientascent on this example, there is a significant amount of mode collapse and the results keep changing
from iteration to iteration.

(a) Discriminator loss(b) Generator loss(c) Inception score

Figure 11: (a) and (b): Comparison of the generator and discriminator loss on a DC-GAN architecture with 4 convolutional layers trained on cifar-10 for consensus optimization (without batchnormalization) and alternating gradient ascent (with batch-normalization). (c): Comparison of the
inception score over time which was computed using 6400 samples. We see that on this architecture
consensus optimization achieves much better end results.
20

Effect of Hyperparameters(a) 3 convolutional layers(b) 4 convolutional layers

Figure 12: Effect of the learning rate on training a DC-GAN architecture with alternating gradientascent. We use RMSProp [25] as an optimizer for all experiments.

(a) 3 convolutional layers(b) 4 convolutional layers

Figure 13: Effect of the learning rate on consensus optimization. We use the RMSProp-optimizer [25]for all experiments with a regularization parameter γ = 0.1 for the architecture with 3 convolutional
layers and γ = 10 for the architecture with 4 convolutional layers.

(a) 3 convolutional layers(b) 4 convolutional layers

Figure 14: Effect of the regularization parameter γ on consensus optimization. We use the RMSPropoptimizer [25] with a learning rate of 2 · 10−4 for all experiments.

21