Supplementary Material for
Adversarial Variational Bayes: Unifying Variational Autoencoders andGenerative Adversarial NetworksLars Mescheder 1

Sebastian Nowozin 2

Abstract

Andreas Geiger 1 3

To apply our method in practice, we need to obtain unbiased gradients of the ELBO. As it turns out, this can beachieved by taking the gradients w.r.t. a fixed optimal discriminator. This is a consequence of the following Proposition:
Proposition 2. We have

In the main text we derived Adversarial Variational Bayes (AVB) and demonstrated its usefulness both for black-box Variational Inference andfor learning latent variable models. This document contains proofs that were omitted in themain text as well as some further details about
the experiments and additional results.

Eqφ (z|x) (∇φ T ∗ (x, z)) = 0.

(3.6)

Proof. By Proposition 1,I. Proofs
This section contains the proofs that were omitted in themain text.
The derivation of AVB in Section 3.1 relies on the fact that
we have an explicit representation of the optimal discriminator T ∗ (x, z). This was stated in the following Proposition:
Proposition 1. For pθ (x | z) and qφ (z | x) fixed, the optimal discriminator T ∗ according to the objective in (3.3) isgiven by∗T (x, z) = log qφ (z | x) − log p(z).

(3.4)

Proof. As in the proof of Proposition 1 in Goodfellow et al.
(2014), we rewrite the objective in (3.3) as
ZpD (x)qφ (z | x) log σ(T (x, z))
+ pD (x)p(z) log(1 − σ(T (x, z)) dxdz. (I.1)
This integral is maximal as a function of T (x, z) if and only
if the integrand is maximal for every (x, z). However, thefunction
t 7→ a log(t) + b log(1 − t)
(I.2)
aattains its maximum at t = a+b, showing thatσ(T ∗ (x, z)) =qφ (z | x)qφ (z | x) + p(z)(I.3)

or, equivalently,
T ∗ (x, z) = log qφ (z | x) − log p(z).

(I.4)

Eqφ (z|x) (∇φ T ∗ (x, z))= Eqφ (z|x) (∇φ log qφ (z | x)) . (I.5)
For an arbitrary family of probability densities qφ we haveZ∇φ qφ (z)dzEqφ (∇φ log qφ ) = qφ (z)qφ (z)
Z= ∇φ qφ (z)dz = ∇φ 1 = 0. (I.6)
Together with (8.5), this implies (3.6).
In Section 3.3 we characterized the Nash-equilibria of thetwo-player game defined by our algorithm. The following Proposition shows that in the nonparametric limit for
T (x, z) any Nash-equilibrium defines a global optimum of
the variational lower bound:
Proposition 3. Assume that T can represent any functionof two variables. If (θ∗ , φ∗ , T ∗ ) defines a Nash-equilibriumof the two-player game defined by (3.3) and (3.7), thenT ∗ (x, z) = log qφ∗ (z | x) − log p(z)(3.8)

and (θ∗ , φ∗ ) is a global optimum of the variational lower
bound in (2.4).
Proof. If (θ∗ , φ∗ , T ∗ ) defines a Nash-equilibrium, Proposition 1 shows (3.8). Inserting (3.8) into (3.5) shows that(φ∗ , θ∗ ) maximizes
EpD (x) Eqφ (z|x) − log qφ∗ (z | x) + log p(z)+ log pθ (x | z)(I.7)

Adversarial Variational Bayes

as a function of φ and θ. A straightforward calculationshows that (8.7) is equal toL(θ, φ) + EpD (x) KL(qφ (z | x), qφ∗ (z | x))(I.8)

wherehL(θ, φ) := EpD (x) − KL(qφ (z | x), p(z))
i+ Eqφ (z|x) log pθ (x | z)

(I.9)

is the variational lower bound in (2.4).
Notice that (8.8) evaluates to L(θ∗ , φ∗ ) when we insert(θ∗ , φ∗ ) for (θ, φ).
Assume now, that (θ∗ , φ∗ ) does not maximize the variational lower bound L(θ, φ). Then there is (θ0 , φ0 ) withL(θ0 , φ0 ) > L(θ∗ , φ∗ ).

(I.10)

Inserting (θ0 , φ0 ) for (θ, φ) in (8.8) we obtain
L(θ0 , φ0 ) + EpD (x) KL(qφ0 (z | x), qφ∗ (z | x)),II. Adaptive Contrast
In Section 4 we derived a variant of AVB that contrasts thecurrent inference model with an adaptive distribution rather
than the prior. This leads to Algorithm 2. Note that we do
not consider the µ(k) and σ (k) to be functions of φ andtherefore do not backpropagate gradients through them.
Algorithm 2 Adversarial Variational Bayes with AdaptiveConstrast (AC)1: i ← 0
2: while not converged do
3:
Sample {x(1) , . . . , x(m) } from data distrib. pD (x)4:
Sample {z (1) , . . . , z (m) } from prior p(z)5:Sample {(1) , . . . , (m) } from N (0, 1)
6:
Sample {η (1) , . . . , η (m) } from N (0, 1)
7:
for k = 1, . . . , m do(k)8:zφ , µ(k) , σ (k) ← encoderφ (x(k) , (k) )
z(k)

−µ(k)

z̄φ ← φ σ(k)end forCompute θ-gradient (eq. 3.7):
Pm
1(k) (k)gθ ← m∇logpx,zθ
k=1 θφ12:Compute φ-gradient (eq. 3.7):

Pm(k)
1(k) (k)gφ ← m, z̄φ + 12 kz̄φ k2
k=1 ∇φ −Tψ x(k) + log pθ x(k) , zφCompute ψ-gradient (eq. 3.3) :h Pm
1(k) (k)gψ ← m∇logσ(T(x,z̄ψψ
k=1φ
i+ log 1 − σ(Tψ (x(k) , η (k) )(I.11)

which is strictly bigger than L(θ∗ , φ∗ ), contradicting thefact that (θ∗ , φ∗ ) maximizes (8.8). Together with (3.8), thisproves the theorem.

(k)

9:
10:
11:

13:

Perform SGD-updates for θ, φ and ψ:θ ← θ + hi gθ , φ ← φ + hi gφ , ψ ← ψ + hi gψ
15:
i←i+1
16: end while
14:

III. Architecture for MNIST-experiment
To apply Adaptive Contrast to our method, we have to beable to efficiently estimate the moments of the current inference model qφ (z | x). To this end, we propose a networkarchitecture like in Figure 8. The final output z of the network is a linear combination of basis noise vectors where
the coefficients depend on the data point x, i.e.
zk =mXvi,k (i )ai,k (x).

(III.1)i=1

The noise basis vectors vi (i ) are defined as the output of small fully-connected neural networks fi acting on
normally-distributed random noise i , the coefficient vec-

Adversarial Variational Bayes

1

f1

v1

∗

a1

..
.

..
.

..
.

..
.

..
.

m

fmvm

∗

am+gxz(a) Training dataFigure 8. Architecture of the network used for the MNISTexperiment(b) Random samples

Figure 9. Independent samples for a model trained on celebA.

tors ai (x) are defined as the output of a deep convolutional
neural network g acting on x.
The moments of the zi are then given byE(zk ) =Var(zk ) =mX
i=1m
X

E[vi,k (i )]ai,k (x).

(III.2)Var[vi,k (i )]ai,k (x)2 .

(III.3)i=1

By estimating E[vi,k (i )] and Var[vi,k (i )] via sampling
once per mini-batch, we can efficiently compute the moments of qφ (z | x) for all the data points x in a single
mini-batch.

IV. Additional Experiments
celebA We also used AVB (without AC) to train a deep
convolutional network on the celebA-dataset (Liu et al.,2015) for a 64-dimensional latent space with N (0, 1)-prior.
For the decoder and adversary we use two deep convolutional neural networks acting on x like in Radford et al.
(2015). We add the noise  and the latent code z to eachhidden layer via a learned projection matrix. Moreover, inthe encoder and decoder we use three RESNET-blocks (Heet al., 2015) at each scale of the neural network. We add
the log-prior log p(z) explicitly to the adversary T (x, z),
so that it only has to learn the log-density of the inference
model qφ (z | x).
The samples for celebA are shown in Figure 9. We seethat our model produces visually sharp images of faces. To
demonstrate that the model has indeed learned an abstract
representation of the data, we show reconstruction results
and the result of linearly interpolating the z-vector in the latent space in Figure 10. We see that the reconstructions arereasonably sharp and the model produces realistic images
for all interpolated z-values.

Figure 10. Interpolation experiments for celebA

MNIST To evaluate how AVB with adaptive contrast
compares against other methods on a fixed decoder architecture, we reimplemented the methods from Maaløe et al.
(2016) and Kingma et al. (2016). The method from Maaløeet al. (2016) tries to make the variational approximation
to the posterior more flexible by using auxiliary variables,
the method from Kingma et al. (2016) tries to improve thevariational approximation by employing an Inverse Autoregressive Flow (IAF), a particularly flexible instance of anormalizing flow (Rezende & Mohamed, 2015). In our experiments, we compare AVB with adaptive contrast to astandard VAE with diagonal Gaussian inference model aswell as the methods from Maaløe et al. (2016) and Kingmaet al. (2016).
In our first experiment, we evaluate all methods on training
a decoder that is given by a fully-connected neural networkwith ELU-nonlinearities and two hidden layers with 300
units each. The prior distribution p(z) is given by a 32dimensional standard-Gaussian distribution.
The results are shown in Table 3a. We observe, thatboth AVB and the VAE with auxiliary variables achievea better (approximate) ELBO than a standard VAE. Whenevaluated using AIS, both methods result in similar log-Adversarial Variational Bayes

AVB + ACVAE
auxiliary VAE
VAE + IAF

ELBO
≈ −85.1 ± 0.2−88.9 ± 0.2−88.0 ± 0.2−88.9 ± 0.2

AIS−83.7 ± 0.3−85.0 ± 0.3−83.8 ± 0.3−84.9 ± 0.3

reconstr. error
59.3 ± 0.2
62.2 ± 0.2
62.1 ± 0.2
62.3 ± 0.2(a) fully-connected decoder (dim(z) = 32)

AVB + AC
VAE
auxiliary VAE
VAE + IAF

ELBO
≈ −93.8 ± 0.2−94.9 ± 0.2−95.0 ± 0.2−94.4 ± 0.2

AIS−89.7 ± 0.3−89.9 ± 0.4−89.7 ± 0.3−89.7 ± 0.3

reconstr. error
76.4 ± 0.2
76.7 ± 0.2
76.8 ± 0.2
76.1 ± 0.2(b) convolutional decoder (dim(z) = 8)

AVB + AC
VAE
auxiliary VAE
VAE + IAF

ELBO
≈ −82.7 ± 0.2−85.7 ± 0.2−85.6 ± 0.2−85.5 ± 0.2

AIS−81.7 ± 0.3−81.9 ± 0.3−81.6 ± 0.3−82.1 ± 0.4

reconstr. error
57.0 ± 0.2
59.4 ± 0.2
59.6 ± 0.2
59.6 ± 0.2(c) convolutional decoder (dim(z) = 32)

likelihoods. However, AVB results in a better reconstruction error than an auxiliary variable VAE and a better (approximate) ELBO. We observe that our implementation ofa VAE with IAF did not improve on a VAE with diagonal
Gaussian inference model. We suspect that this due to optimization difficulties.
In our second experiment, we train a decoder that is givenby the shallow convolutional neural network described inSalimans et al. (2015) with 800 units in the last fullyconnected hidden layer. The prior distribution p(z) is given
by either a 8-dimensional or a 32-dimensional standardGaussian distribution.
The results are shown in Table 3b and Table 3c. Even
though AVB achieves a better (approximate) ELBO anda better reconstruction error for a 32-dimensional latent
space, all methods achieve similar log-likelihoods for thisdecoder-architecture, raising the question if strong inference models are always necessary to obtain a good generative model. Moreover, we found that neither auxiliary variables nor IAF did improve the ELBO. Again, we believe
this is due to optimization challenges.