Adversarial Variational Bayes:
Unifying Variational Autoencoders and Generative Adversarial Networks

Lars Mescheder 1

Sebastian Nowozin 2

Andreas Geiger 1 3

Abstract
Variational Autoencoders (VAEs) are expressive
latent variable models that can be used to learncomplex probability distributions from training
data. However, the quality of the resulting model
crucially relies on the expressiveness of the inference model. We introduce Adversarial Variational Bayes (AVB), a technique for training
Variational Autoencoders with arbitrarily expressive inference models. We achieve this by introducing an auxiliary discriminative networkthat allows to rephrase the maximum-likelihoodproblem as a two-player game, hence establishing a principled connection between VAEs andGenerative Adversarial Networks (GANs). We
show that in the nonparametric limit our method
yields an exact maximum-likelihood assignment
for the parameters of the generative model, as
well as the exact posterior distribution over thelatent variables given an observation. Contrary
to competing approaches which combine VAEs
with GANs, our approach has a clear theoreticaljustification, retains most advantages of standard
Variational Autoencoders and is easy to implement.

1. Introduction
Generative models in machine learning are models that canbe trained on an unlabeled dataset and are capable of generating new data points after training is completed. As generating new content requires a good understanding of the
training data at hand, such models are often regarded as a
key ingredient to unsupervised learning.
In recent years, generative models have become more and1Autonomous Vision Group, MPI Tübingen 2 Microsoft
Research Cambridge 3 Computer Vision and Geometry
Group, ETH Zürich. Correspondence to: Lars Mescheder<lars.mescheder@tuebingen.mpg.de>.

Proceedings of the 34 th International Conference on Machine
Learning, Sydney, Australia, PMLR 70, 2017. Copyright 2017by the author(s).

fFigure 1. We propose a method which enables neural samplers
with intractable density for Variational Bayes and as inference
models for learning latent variable models. This toy example demonstrates our method’s ability to accurately approximate
complex posterior distributions like the one shown on the right.

more powerful. While many model classes such as PixelRNNs (van den Oord et al., 2016b), PixelCNNs (van den
Oord et al., 2016a), real NVP (Dinh et al., 2016) and Plug
& Play generative networks (Nguyen et al., 2016) havebeen introduced and studied, the two most prominent onesare Variational Autoencoders (VAEs) (Kingma & Welling,2013; Rezende et al., 2014) and Generative Adversarial
Networks (GANs) (Goodfellow et al., 2014).
Both VAEs and GANs come with their own advantages
and disadvantages: while GANs generally yield visually
sharper results when applied to learning a representationof natural images, VAEs are attractive because they naturally yield both a generative model and an inference model.
Moreover, it was reported, that VAEs often lead to betterlog-likelihoods (Wu et al., 2016). The recently introduced
BiGANs (Donahue et al., 2016; Dumoulin et al., 2016) add
an inference model to GANs. However, it was observedthat the reconstruction results often only vaguely resemble
the input and often do so only semantically and not in termsof pixel values.
The failure of VAEs to generate sharp images is often attributed to the fact that the inference models used during
training are usually not expressive enough to capture thetrue posterior distribution. Indeed, recent work shows thatusing more expressive model classes can lead to substantially better results (Kingma et al., 2016), both visuallyand in terms of log-likelihood bounds. Recent work (Chenet al., 2016) also suggests that highly expressive inference
models are essential in presence of a strong decoder to allow the model to make use of the latent space at all.

Adversarial Variational Bayes

In this paper, we present Adversarial Variational Bayes(AVB) 1 , a technique for training Variational Autoencoders
with arbitrarily flexible inference models parameterized byneural networks. We can show that in the nonparametriclimit we obtain a maximum-likelihood assignment for thegenerative model together with the correct posterior distribution.
While there were some attempts at combining VAEs and
GANs (Makhzani et al., 2015; Larsen et al., 2015), mostof these attempts are not motivated from a maximumlikelihood point of view and therefore usually do not leadto maximum-likelihood assignments. For example, in Adversarial Autoencoders (AAEs) (Makhzani et al., 2015) the
Kullback-Leibler regularization term that appears in the
training objective for VAEs is replaced with an adversarial
loss that encourages the aggregated posterior to be close to
the prior over the latent variables. Even though AAEs do
not maximize a lower bound to the maximum-likelihood
objective, we show in Section 6.2 that AAEs can be interpreted as an approximation to our approach, thereby establishing a connection of AAEs to maximum-likelihood
learning.
Outside the context of generative models, AVB yields a
new method for performing Variational Bayes (VB) withneural samplers. This is illustrated in Figure 1, where weused AVB to train a neural network to sample from a nontrival unnormalized probability density. This allows to accurately approximate the posterior distribution of a probabilistic model, e.g. for Bayesian parameter estimation.
The only other variational methods we are aware of thatcan deal with such expressive inference models are basedon Stein Discrepancy (Ranganath et al., 2016; Liu & Feng,
2016). However, those methods usually do not directly target the reverse Kullback-Leibler-Divergence and can therefore not be used to approximate the variational lower boundfor learning a latent variable model. Our contributions are
as follows:• We enable the usage of arbitrarily complex inference
models for Variational Autoencoders using adversarial
training.
• We give theoretical insights into our method, showing that in the nonparametric limit our method recovers the true posterior distribution as well as a true
maximum-likelihood assignment for the parametersof the generative model.

x
1

x
1

f

f
+/∗
Encoder

Encoder

z

g

z

g

2

+/∗

2

+/∗

x

x

Decoder

Decoder(a) Standard VAE(b) Our model

Figure 2. Schematic comparison of a standard VAE and a VAE
with black-box inference model, where 1 and 2 denote samples
from some noise distribution. While more complicated inference
models for Variational Autoencoders are possible, they are usually
not as flexible as our black-box approach.

• We empirically demonstrate that our model is able
to learn rich posterior distributions and show that themodel is able to generate compelling samples for complex data sets.

2. BackgroundAs our model is an extension of Variational Autoencoders(VAEs) (Kingma & Welling, 2013; Rezende et al., 2014),
we start with a brief review of VAEs.
VAEs are specified by a parametric generative model pθ (x |z) of the visible variables given the latent variables, a priorp(z) over the latent variables and an approximate inference
model qφ (z | x) over the latent variables given the visible
variables. It can be shown thatlog pθ (x) ≥ −KL(qφ (z | x), p(z))+ Eqφ (z|x) log pθ (x | z). (2.1)
The right hand side of (2.1) is called the variational lower
bound or evidence lower bound (ELBO). If there is φ suchthat qφ (z | x) = pθ (z | x), we would havelog pθ (x) = max −KL(qφ (z | x), p(z))φ+ Eqφ (z|x) log pθ (x | z). (2.2)

1

Concurrently to our work, several researchers have described similar ideas. Some ideas of this paper were describedindependently by Huszár in a blog post on http://www.
inference.vc and in Huszár (2017). The idea to use adversarial training to improve the encoder network was also suggested by
Goodfellow in an exploratory talk he gave at NIPS 2016 and by Li& Liu (2016). A similar idea was also mentioned by Karaletsos(2016) in the context of message passing in graphical models.

However, in general this is not true, so that we only havean inequality in (2.2).
When performing maximum-likelihood training, our goal
is to optimize the marginal log-likelihood
EpD (x) log pθ (x),(2.3)

Adversarial Variational Bayes

where pD is the data distribution. Unfortunately, computing log pθ (x) requires marginalizing out z in pθ (x, z)
which is usually intractable. Variational Bayes uses inequality (2.1) to rephrase the intractable problem of optimizing (2.3) intohmax max EpD (x) −KL(qφ (z | x), p(z))θφ

i+ Eqφ (z|x) log pθ (x | z) . (2.4)
Due to inequality (2.1), we still optimize a lower bound to
the true maximum-likelihood objective (2.3).
Naturally, the quality of this lower bound depends on theexpressiveness of the inference model qφ (z | x). Usually,qφ (z | x) is taken to be a Gaussian distribution with diagonal covariance matrix whose mean and variance vectors areparameterized by neural networks with x as input (Kingma& Welling, 2013; Rezende et al., 2014). While this model
is very flexible in its dependence on x, its dependence onz is very restrictive, potentially limiting the quality of theresulting generative model. Indeed, it was observed that
applying standard Variational Autoencoders to natural images often results in blurry images (Larsen et al., 2015).

3. Method
In this work we show how we can instead use a black-box
inference model qφ (z | x) and use adversarial training toobtain an approximate maximum likelihood assignment θ∗to θ and a close approximation qφ∗ (z | x) to the true posterior pθ∗ (z | x). This is visualized in Figure 2: on the left
hand side the structure of a typical VAE is shown. The righthand side shows our flexible black-box inference model. In
contrast to a VAE with Gaussian inference model, we include the noise 1 as additional input to the inference model
instead of adding it at the very end, thereby allowing the inference network to learn complex probability distributions.
3.1. Derivation
To derive our method, we rewrite the optimization problem
in (2.4) as
max max EpD (x) Eqφ (z|x) log p(z)θφ− log qφ (z | x) + log pθ (x | z) . (3.1)
When we have an explicit representation of qφ (z | x) suchas a Gaussian parameterized by a neural network, (3.1) canbe optimized using the reparameterization trick (Kingma &Welling, 2013; Rezende & Mohamed, 2015) and stochastic
gradient descent. Unfortunately, this is not the case when
we define qφ (z | x) by a black-box procedure as illustrated
in Figure 2b.

The idea of our approach is to circumvent this problem by
implicitly representing the termlog p(z) − log qφ (z | x)

(3.2)

as the optimal value of an additional real-valued discriminative network T (x, z) that we introduce to the problem.
More specifically, consider the following objective for thediscriminator T (x, z) for a given qφ (x | z):
max EpD (x) Eqφ (z|x) log σ(T (x, z))T+ EpD (x) Ep(z) log (1 − σ(T (x, z))) . (3.3)
Here, σ(t) := (1 + e−t )−1 denotes the sigmoid-function.
Intuitively, T (x, z) tries to distinguish pairs (x, z) that were
sampled independently using the distribution pD (x)p(z)from those that were sampled using the current inference
model, i.e., using pD (x)qφ (z | x).
To simplify the theoretical analysis, we assume that themodel T (x, z) is flexible enough to represent any function of the two variables x and z. This assumption is often
referred to as the nonparametric limit (Goodfellow et al.,2014) and is justified by the fact that deep neural networks
are universal function approximators (Hornik et al., 1989).
As it turns out, the optimal discriminator T ∗ (x, z) according to the objective in (3.3) is given by the negative of (3.2).
Proposition 1. For pθ (x | z) and qφ (z | x) fixed, the optimal discriminator T ∗ according to the objective in (3.3) isgiven byT ∗ (x, z) = log qφ (z | x) − log p(z).

(3.4)

Proof. The proof is analogous to the proof of Proposition
1 in Goodfellow et al. (2014). See the Supplementary Material for details.
Together with (3.1), Proposition 1 allows us to write theoptimization objective in (2.4) asmax EpD (x) Eqφ (z|x) − T ∗ (x, z) + log pθ (x | z) , (3.5)θ,φ

where T ∗ (x, z) is defined as the function that maximizes(3.3).
To optimize (3.5), we need to calculate the gradients of(3.5) with respect to θ and φ. While taking the gradientwith respect to θ is straightforward, taking the gradient withrespect to φ is complicated by the fact that we have definedT ∗ (x, z) indirectly as the solution of an auxiliary optimization problem which itself depends on φ. However, the following Proposition shows that taking the gradient with respect to the explicit occurrence of φ in T ∗ (x, z) is not necessary:

Adversarial Variational Bayes

Algorithm 1 Adversarial Variational Bayes (AVB)
1: i ← 0
2: while not converged do
3:
Sample {x(1) , . . . , x(m) } from data distrib. pD (x)4:
Sample {z (1) , . . . , z (m) } from prior p(z)5:Sample {(1) , . . . , (m) } from N (0, 1)
6:Compute θ-gradient (eq. 3.7):Pm

1gθ ← mk=1 ∇θ log pθ

x(k) | zφ x(k) , (k)algorithm converges, any fix point of this algorithm yields
a stationary point of the objective in (2.4).
Note that optimizing (3.5) with respect to φ while keeping θ and T fixed makes the encoder network collapse toa deterministic function. This is also a common problem
for regular GANs (Radford et al., 2015). It is therefore
crucial to keep the discriminative T network close to optimality while optimizing (3.5). A variant of Algorithm 1
therefore performs several SGD-updates for the adversary
for one SGD-update of the generative model. However,throughout our experiments we use the simple 1-step version of AVB unless stated otherwise.

Compute φ-gradient (eq. 3.7):


Pm
1
(k)
gφ ← m, zφ (x(k) , (k) )k=1 ∇φ −Tψ x+ log pθ x(k) | zφ (x(k) , (k) )8:Compute ψ-gradient (eq. 3.3) :h 3.3. Theoretical resultsPm
1∇
log σ(Tψ (x(k) , zφ (x(k) , (k) )))gψ ← mψ
k=1
i
In Sections 3.1 we derived AVB as a way of performing+ log 1 − σ(Tψ (x(k) , z (k) )stochastic gradient descent on the variational lower boundin (2.4). In this section, we analyze the properties of Algo9:
Perform SGD-updates for θ, φ and ψ:rithm 1 from a game theoretical point of view.
θ ← θ + hi gθ , φ ← φ + hi gφ , ψ ← ψ + hi gψ
As the next proposition shows, global Nash-equilibria of
10:
i←i+1
Algorithm 1 yield global optima of the objective in (2.4):
11: end while
Proposition 3. Assume that T can represent any functionof two variables. If (θ∗ , φ∗ , T ∗ ) defines a Nash-equilibrium
Proposition 2. We haveof the two-player game defined by (3.3) and (3.7), then7:

Eqφ (z|x) (∇φ T ∗ (x, z)) = 0.

(3.6)

Proof. The proof can be found in the Supplementary Material.
Using the reparameterization trick (Kingma & Welling,2013; Rezende et al., 2014), (3.5) can be rewritten in theform

T ∗ (x, z) = log qφ∗ (z | x) − log p(z)(3.8)

and (θ∗ , φ∗ ) is a global optimum of the variational lower
bound in (2.4).
Proof. The proof can be found in the Supplementary Material.

(3.7)

Our parameterization of qφ (z | x) as a neural network allows qφ (z | x) to represent almost any probability density
on the latent space. This motivatesfor a suitable function zφ (x, ). Together with Proposition
1, (3.7) allows us to take unbiased estimates of the gradientsof (3.5) with respect to φ and θ.

Corollary 4. Assume that T can represent any function of
two variables and qφ (z | x) can represent any probability
density on the latent space. If (θ∗ , φ∗ , T ∗ ) defines a Nashequilibrium for the game defined by (3.3) and (3.7), thenmax EpD (x) E − T ∗ (x, zφ (x, ))θ,φ+ log pθ (x | zφ (x, ))

3.2. Algorithm
In theory, Propositions 1 and 2 allow us to apply Stochastic
Gradient Descent (SGD) directly to the objective in (2.4).
However, this requires keeping T ∗ (x, z) optimal which iscomputationally challenging. We therefore regard the optimization problems in (3.3) and (3.7) as a two-player game.
Propositions 1 and 2 show that any Nash-equilibrium of
this game yields a stationary point of the objective in (2.4).
In practice, we try to find a Nash-equilibrium by applying
SGD with step sizes hi jointly to (3.3) and (3.7), see Algorithm 1. Here, we parameterize the neural network T witha vector ψ. Even though we have no guarantees that this1. θ∗ is a maximum-likelihood assignment
2. qφ∗ (z | x) is equal to the true posterior pθ∗ (z | x)
3. T ∗ is the pointwise mutual information between x andz, i.e.
pθ∗ (x, z)
T ∗ (x, z) = log.
(3.9)
pθ∗ (x)p(z)Proof. This is a straightforward consequence of Proposition 3, as in this case (θ∗ , φ∗ ) optimizes the variational
lower bound in (2.4) if and only if 1 and 2 hold. Inserting the result from 2 into (3.8) yields 3.

Adversarial Variational Bayes

4. Adaptive Contrast
While in the nonparametric limit our method yields the correct results, in practice T (x, z) may fail to become sufficiently close to the optimal function T ∗ (x, z). The reason for this problem is that AVB calculates a contrast between the two densities pD (x)qφ (z | x) to pD (x)p(z)which are usually very different. However, it is known thatlogistic regression works best for likelihood-ratio estimation when comparing two very similar densities (Friedman
et al., 2001).
To improve the quality of the estimate, we therefore propose to introduce an auxiliary conditional probability distribution rα (z | x) with known density that approximatesqφ (z | x). For example, rα (z | x) could be a Gaussian distribution with diagonal covariance matrix whose mean andvariance matches the mean and variance of qφ (z | x).
Using this auxiliary distribution, we can rewrite the variational lower bound in (2.4) ash
EpD (x) −KL (qφ (z | x), rα (z | x))i+ Eqφ (z|x) (− log rα (z | x) + log pφ (x, z)) . (4.1)
As we know the density of rα (z | x), the second term in(4.1) is amenable to stochastic gradient descent with respect to θ and φ. However, we can estimate the first termusing AVB as described in Section 3. If rα (z | x) approximates qφ (z | x) well, KL (qφ (z | x), rα (z | x)) is usually much smaller than KL (qφ (z | x), p(z)), which makes
it easier for the adversary to learn the correct probability
ratio.
We call this technique Adaptive Contrast (AC), as we are
now contrasting the current inference model qφ (z | x) toan adaptive distribution rα (z | x) instead of the prior p(z).
Using Adaptive Contrast, the generative model pθ (x | z)
and the inference model qφ (z | x) are trained to maximize
EpD (x) Eqφ (z|x) − T ∗ (x, z)− log rα (z | x) + log pθ (x, z) , (4.2)
where T ∗ (x, z) is the optimal discriminator distinguishingsamples from rα (z | x) and qφ (z | x).
Consider now the case that rα (z | x) is given by a Gaussian
distribution with diagonal covariance matrix whose meanµ(x) and variance vector σ(x) match the mean and variance of qφ (z | x). As the Kullback-Leibler divergence isinvariant under reparameterization, the first term in (4.1)
can be rewritten as
EpD (x) KL (q̃φ (z̃ | x), r0 (z̃))(4.3)

where q̃φ (z̃ | x) denotes the distribution of the normalized
vector z̃ := z−µ(x)σ(x) and r0 (z̃) is a Gaussian distribution

Figure 3. Comparison of KL to ground truth posterior obtained by
Hamiltonian Monte Carlo (HMC).

with mean 0 and variance 1. This way, the adversary only
has to account for the deviation of qφ (z | x) from a Gaussian distribution, not its location and scale. Please see theSupplementary Material for pseudo code of the resulting
algorithm.
In practice, we estimate µ(x) and σ(x) using a MonteCarlo estimate. In the Supplementary Material we describe
a network architecture for qφ (z | x) that makes the computation of this estimate particularly efficient.

5. Experiments
We tested our method both as a black-box method for variational inference and for learning generative models. The
former application corresponds to the case where we fix thegenerative model and a data point x and want to learn theposterior qφ (z | x).
An additional experiment on the celebA dataset (Liu et al.,2015) can be found in the Supplementary Material.
5.1. Variational Inference
When the generative model and a data point x is fixed, AVB
gives a new technique for Variational Bayes with arbitrarily
complex approximating distributions. We applied this tothe “Eight School” example from Gelman et al. (2014). In
this example, the coaching effects yi , i = 1, . . . , 8 for eightschools are modeled asyi ∼ N (µ + θ · ηi , σi ),where µ, τ and the ηi are the model parameters to be inferred. We place a N (0, 1) prior on the parameters of themodel. We compare AVB against two variational methods
with Gaussian inference model (Kucukelbir et al., 2015)as implemented in STAN (Stan Development Team, 2016).
We used a simple two layer model for the posterior and apowerful 5-layer network with RESNET-blocks (He et al.,2015) for the discriminator. For every posterior update step
we performed two steps for the adversary. The groundtruth data was obtained by running Hamiltonian Monte-

Adversarial Variational Bayes(µ, τ )(τ, η1 )AVB

VB(fullrank)

Figure 5. Training examples in the synthetic dataset.

(a) VAE(b) AVB

Figure 6. Distribution of latent code for VAE and AVB trained onsynthetic dataset.

HMC

Figure 4. Comparison of AVB to VB on the “Eight Schools” example by inspecting two marginal distributions of the approximation to the 10-dimensional posterior. We see that AVB accurately
captures the multi-modality of the posterior distribution. In contrast, VB only focuses on a single mode. The ground truth isshown in the last row and has been obtained using HMC.

Carlo (HMC) for 500000 steps using STAN. Note that
AVB and the baseline variational methods allow to draw
an arbitrary number of samples after training is completed
whereas HMC only yields a fixed number of samples.
We evaluate all methods by estimating the KullbackLeibler-Divergence to the ground-truth data using the ITEpackage (Szabo, 2013) applied to 10000 samples from theground-truth data and the respective approximation. The
resulting Kullback-Leibler divergence over the number of
iterations for the different methods is plotted in Figure 3.
We see that our method clearly outperforms the methods
with Gaussian inference model. For a qualitative visualization, we also applied Kernel-density-estimation to the 2dimensional marginals of the (µ, τ )- and (τ, η1 )-variables
as illustrated in Figure 4. In contrast to variational Bayeswith Gaussian inference model, our approach clearly captures the multi-modality of the posterior distribution. We
also observed that Adaptive Contrast makes learning morerobust and improves the quality of the resulting model.
5.2. Generative Models
Synthetic Example To illustrate the application of our
method to learning a generative model, we trained the neural networks on a simple synthetic dataset containing only
the 4 data points from the space of 2 × 2 binary images
shown in Figure 5 and a 2-dimensional latent space. Boththe encoder and decoder are parameterized by 2-layer fully
connected neural networks with 512 hidden units each. The

log-likelihood
reconstruction error
ELBO
KL(qφ (z), p(z))

VAE-1.56888.5 ·10−3-1.697≈ 0.165

AVB-1.4035.77 ·10−3≈ -1.421≈ 0.026

Table 1. Comparison of VAE and AVB on synthetic dataset.
The optimal log-likelihood score on this dataset is − log(4) ≈−1.386.

encoder network takes as input a data point x and a vector of Gaussian random noise  and produces a latent codez. The decoder network takes as input a latent code z andproduces the parameters for four independent Bernoullidistributions, one for each pixel of the output image. The
adversary is parameterized by two neural networks withtwo 512-dimensional hidden layers each, acting on x and z
respectively, whose 512-dimensional outputs are combinedusing an inner product.
We compare our method to a Variational Autoencoder witha diagonal Gaussian posterior distribution. The encoderand decoder networks are parameterized as above, but theencoder does not take the noise  as input and produces amean and variance vector instead of a single sample.
We visualize the resulting division of the latent space inFigure 6, where each color corresponds to one state in thex-space. Whereas the Variational Autoencoder divides thespace into a mixture of 4 Gaussians, the Adversarial Variational Autoencoder learns a complex posterior distribution.
Quantitatively this can be verified by computing the KLdivergence between
R the prior p(z) and the aggregated posterior qφ (z) := qφ (z | x)pD (x)dx, which we estimate
using the ITE-package (Szabo, 2013), see Table 1. Notethat the variations for different colors in Figure 6 are solely
due to the noise  used in the inference model.
The ability of AVB to learn more complex posterior models leads to improved performance as Table 1 shows. In
particular, AVB leads to a higher likelihood score that isclose to the optimal value of − log(4) compared to a standard VAE that struggles with the fact that it cannot divide

Adversarial Variational Bayes

1000 intermediate distributions and 5 parallel chains on
2048 test examples. The results are reported in Table 2.
Using AIS, we see that AVB without AC overestimates thetrue ELBO which degrades its performance. Even though
the results suggest that AVB with AC can also overestimate
the true ELBO in higher dimensions, we note that the loglikelihood estimate computed by AIS is also only a lower
bound to the true log-likelihood (Wu et al., 2016).
(a) Training data(b) Random samples

Figure 7. Independent samples for a model trained on MNISTAVB (8-dim)
AVB + AC (8-dim)
AVB + AC (32-dim)
VAE (8-dim)
VAE (32-dim)
VAE + NF (T=80)
VAE + HVI (T=16)
convVAE + HVI (T=16)
VAE + VGP (2hl)
DRAW + VGP
VAE + IAF
Auxiliary VAE (L=2)

log p(x) ≥(≈ −83.6 ± 0.4)
≈ −96.3 ± 0.4≈ −79.5 ± 0.3−98.1 ± 0.5−87.2 ± 0.3
−85.1
−88.3
−84.1
−81.3
−79.9
−80.8
−83.0

log p(x) ≈−91.2 ± 0.6−89.6 ± 0.6−80.2 ± 0.4−90.9 ± 0.6−81.9 ± 0.4−−85.51−81.94−−−79.10−(Rezende & Mohamed, 2015)(Salimans et al., 2015)(Salimans et al., 2015)(Tran et al., 2015)(Tran et al., 2015)(Kingma et al., 2016)(Maaløe et al., 2016)

Table 2. Log-likelihoods on binarized MNIST for AVB and othermethods improving on VAEs. We see that our method achieves
state of the art log-likelihoods on binarized MNIST. The approximate log-likelihoods in the lower half of the table were not obtained with AIS but with importance sampling.

the latent space appropriately. Moreover, we see that thereconstruction error given by the mean cross-entropy between an input x and its reconstruction using the encoderand decoder networks is much lower when using AVB instead of a VAE with diagonal Gaussian inference model.
We also observe that the estimated variational lower boundis close to the true log-likelihood, indicating that the adversary has learned the correct function.
MNIST In addition, we trained deep convolutional networks based on the DC-GAN-architecture (Radford et al.,2015) on the binarized MNIST-dataset (LeCun et al.,1998). For the decoder network, we use a 5-layer deep convolutional neural network. For the encoder network, we use
a network architecture that allows for the efficient computation of the moments of qφ (z | x). The idea is to define theencoder as a linear combination of learned basis noise vectors, each parameterized by a small fully-connected neuralnetwork, whose coefficients are parameterized by a neuralnetwork acting on x, please see the Supplementary Material for details. For the adversary, we replace the fully
connected neural network acting on z and x with a fully
connected 4-layer neural networks with 1024 units in eachhidden layer. In addition, we added the result of neural networks acting on x and z alone to the end result.
To validate our method, we ran Annealed Importance Sampling (AIS) (Neal, 2001), the gold standard for evaluating
decoder based generative models (Wu et al., 2016) with

Using AVB with AC, we see that we improve both on astandard VAE and AVB without AC. When comparing toother state of the art methods, we see that our method
achieves state of the art results on binarized MNIST2 . For
an additional experimental evaluation of AVB and threebaselines for a fixed decoder architecture see the Supplementary Material. Some random samples for MNIST are
shown in Figure 7. We see that our model produces random samples that are perceptually close to the training set.

6. Related Work6.1. Connection to Variational Autoencoders
AVB strives to optimize the same objective as a standard
VAE (Kingma & Welling, 2013; Rezende et al., 2014), but
approximates the Kullback-Leibler divergence using an adversary instead of relying on a closed-form formula.
Substantial work has focused on making the class of approximate inference models more expressive. Normalizing
flows (Rezende & Mohamed, 2015; Kingma et al., 2016)
make the posterior more complex by composing a simple
Gaussian posterior with an invertible smooth mapping forwhich the determinant of the Jacobian is tractable. Auxiliary Variable VAEs (Maaløe et al., 2016) add auxiliary
variables to the posterior to make it more flexible. However, no other approach that we are aware of allows to use
black-box inference models to optimize the ELBO.
6.2. Connection to Adversarial Autoencoders
Makhzani et al. (Makhzani et al., 2015) introduced the concept of Adversarial Autoencoders. The idea is to replace
the term
KL(qφ (z | x), p(z))(6.1)
in (2.4) with an adversarial loss that tries to enforce thatupon convergenceZqφ (z | x)pD (x)dx ≈ p(z).
(6.2)
While related to our approach, the approach by Makhzani
et al. modifies the variational objective while our approach
2
Note that the methods in the lower half of Table 2 weretrained with different decoder architectures and therefore only
provide limited information regarding the quality of the inference
model.

Adversarial Variational Bayes

retains the objective.
The approach by Makhzani et al. can be regarded as an
approximation to our approach, where T (x, z) is restricted
to the class of functions that do not depend on x. Indeed,
an ideal discriminator that only depends on z maximizes
Z Z
pD (x)q(z | x) log σ(T (z))dxdz
Z Z+
pD (x)p(z) log (1 − σ(T (z)))dxdz (6.3)
which is the case if and only ifZ
T (z) = log q(z | x)pD (x)dx − log p(z).

= sup Eq(z|x) T (x, z) − Ep(z) f ∗ (T (x, z)), (6.8)
T∗with f (ξ) = exp(ξ − 1) the convex conjugate of f (t) =t log t.
All in all, this yields
θ

=

(6.4)

but qφ (z | x) need not be close to the true posterior pθ (z |x). Intuitively, while mapping pD (x) through qφ (z | x)
results in the correct marginal distribution, the contributionof each x to this distribution can be very inaccurate.
In contrast to Adversarial Autoencoders, our goal is to improve the ELBO by performing better probabilistic inference. This allows our method to be used in a more general
setting where we are only interested in the inference network itself (Section 5.1) and enables further improvements
such as Adaptive Contrast (Section 4) which are not possible in the context of Adversarial Autoencoders.
6.3. Connection to f-GANs
Nowozin et al. (Nowozin et al., 2016) proposed to generalize Generative Adversarial Networks (Goodfellow et al.,2014) to f-divergences (Ali & Silvey, 1966) based on results by Nguyen et al. (Nguyen et al., 2010). In this paragraph we show that f-divergences allow to represent AVB
as a zero-sum two-player game.

(6.9)

max min EpD (x) Ep(z) f ∗ (T (x, z))θ,qT+EpD (x) Eq(z|x) (log pθ (x | z) − T (x, z)).
By replacing the objective (3.3) for the discriminator withhimin EpD (x) Ep(z) eT (x,z)−1 − Eq(z|x) T (x, z) , (6.10)
T

we can reformulate the maximum-likelihood-problem as amini-max zero-sum game. In fact, the derivations from
Section 3 remain valid for any f -divergence that we use
to train the discriminator. This is similar to the approachtaken by Poole et al. (Poole et al., 2016) to improve the
GAN-objective. In practice, we observed that the objective(6.10) results in unstable training. We therefore used thestandard GAN-objective (3.3), which corresponds to theJensen-Shannon-divergence.
6.4. Connection to BiGANs
BiGANs (Donahue et al., 2016; Dumoulin et al., 2016)
are a recent extension to Generative Adversarial Networks
with the goal to add an inference network to the generativemodel. Similarly to our approach, the authors introduce
an adversary that acts on pairs (x, z) of data points and latent codes. However, whereas in BiGANs the adversary
is used to optimize the generative and inference networks
separately, our approach optimizes the generative and inference model jointly. As a result, our approach obtains goodreconstructions of the input data, whereas for BiGANs weobtain these reconstructions only indirectly.

7. Conclusion(6.6)

for some convex functional f : R → R∞ with f (1) = 0.
Nguyen et al. (2010) show that by using the convex conjugate f ∗ of f , (Hiriart-Urruty & Lemaréchal, 2013), we
obtainDf (pkq) = sup Eq(x) [T (x)] − Ep(x) [f ∗ (T (x))] , (6.7)
T

KL(q(z | x), p(z)) = Df (p(z), q(z | x))

max EpD (x) log pθ (x)Clearly, this simplification is a crude approximation to ourformulation from Section 3, but Makhzani et al. (2015)
show that this method can still lead to good sampling results. In theory, restricting T (x, z) in this way ensures thatupon convergence we approximately haveZ
qφ (z | x)pD (x)dx = p(z),(6.5)

The family of f-divergences is given by


q(x)
.
Df (pkq) = Ep fp(x)where T is a real-valued function. In particular, this is true
for the reverse Kullback-Leibler divergence with f (t) =t log t. We therefore obtain

We presented a new training procedure for Variational Autoencoders based on adversarial training. This allows us tomake the inference model much more flexible, effectively
allowing it to represent almost any family of conditional
distributions over the latent variables.
We believe that further progress can be made by investigating the class of neural network architectures used for theadversary and the encoder and decoder networks as well as
finding better contrasting distributions.

Adversarial Variational Bayes

Acknowledgements
This work was supported by Microsoft Research throughits PhD Scholarship Programme.

References
Ali, Syed Mumtaz and Silvey, Samuel D. A general classof coefficients of divergence of one distribution from another. Journal of the Royal Statistical Society. Series B(Methodological), pp. 131–142, 1966.
Chen, Xi, Kingma, Diederik P, Salimans, Tim, Duan, Yan,
Dhariwal, Prafulla, Schulman, John, Sutskever, Ilya, andAbbeel, Pieter. Variational lossy autoencoder. arXiv
preprint arXiv:1611.02731, 2016.
Dinh, Laurent, Krueger, David, and Bengio, Yoshua. Nice:
Non-linear independent components estimation. arXiv
preprint arXiv:1410.8516, 2014.
Dinh, Laurent, Sohl-Dickstein, Jascha, and Bengio, Samy.
Density estimation using real nvp. arXiv preprint
arXiv:1605.08803, 2016.

Hornik, Kurt, Stinchcombe, Maxwell, and White, Halbert.
Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359–366, 1989.
Huszár, Ferenc. Variational inference using implicit distributions. arXiv preprint arXiv:1702.08235, 2017.
Karaletsos, Theofanis. Adversarial message passing forgraphical models. arXiv preprint arXiv:1612.05048,
2016.
Kingma, Diederik P and Welling, Max. Auto-encoding
variational bayes. arXiv preprint arXiv:1312.6114,
2013.
Kingma, Diederik P, Salimans, Tim, and Welling, Max. Improving variational inference with inverse autoregressive
flow. arXiv preprint arXiv:1606.04934, 2016.
Kucukelbir, Alp, Ranganath, Rajesh, Gelman, Andrew, andBlei, David. Automatic variational inference in stan. In
Advances in neural information processing systems, pp.
568–576, 2015.

Donahue, Jeff, Krähenbühl, Philipp, and Darrell,
Trevor. Adversarial feature learning. arXiv preprint
arXiv:1605.09782, 2016.

Larsen, Anders Boesen Lindbo, Sønderby, Søren Kaae,and Winther, Ole.
Autoencoding beyond pixels
using a learned similarity metric.
arXiv preprint
arXiv:1512.09300, 2015.

Dumoulin, Vincent, Belghazi, Ishmael, Poole, Ben, Lamb,
Alex, Arjovsky, Martin, Mastropietro, Olivier, andCourville, Aaron. Adversarially learned inference. arXiv
preprint arXiv:1606.00704, 2016.

LeCun, Yann, Bottou, Léon, Bengio, Yoshua, and Haffner,
Patrick. Gradient-based learning applied to document
recognition. Proceedings of the IEEE, 86(11):2278–
2324, 1998.

Friedman, Jerome, Hastie, Trevor, and Tibshirani, Robert.
The elements of statistical learning, volume 1. Springer
series in statistics Springer, Berlin, 2001.
Gelman, Andrew, Carlin, John B, Stern, Hal S, and Rubin,
Donald B. Bayesian data analysis, volume 2. Chapman& Hall/CRC Boca Raton, FL, USA, 2014.
Goodfellow, Ian, Pouget-Abadie, Jean, Mirza, Mehdi, Xu,
Bing, Warde-Farley, David, Ozair, Sherjil, Courville,
Aaron, and Bengio, Yoshua. Generative adversarial nets.
In Advances in Neural Information Processing Systems,
pp. 2672–2680, 2014.
He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun,
Jian. Deep residual learning for image recognition. arXiv
preprint arXiv:1512.03385, 2015.
Hiriart-Urruty, Jean-Baptiste and Lemaréchal, Claude.
Convex analysis and minimization algorithms I: fundamentals, volume 305. Springer science & business media, 2013.

Li, Yingzhen and Liu, Qiang. Wild variational approximations. In NIPS workshop on advances in approximate
Bayesian inference, 2016.
Liu, Qiang and Feng, Yihao. Two methods for wild variational inference. arXiv preprint arXiv:1612.00081,
2016.
Liu, Ziwei, Luo, Ping, Wang, Xiaogang, and Tang, Xiaoou.
Deep learning face attributes in the wild. In Proceedingsof International Conference on Computer Vision (ICCV),2015.
Maaløe, Lars, Sønderby, Casper Kaae, Sønderby,
Søren Kaae, and Winther, Ole. Auxiliary deep generative models. arXiv preprint arXiv:1602.05473, 2016.
Makhzani, Alireza, Shlens, Jonathon, Jaitly, Navdeep, andGoodfellow, Ian. Adversarial autoencoders. arXiv
preprint arXiv:1511.05644, 2015.
Neal, Radford M. Annealed importance sampling. Statistics and Computing, 11(2):125–139, 2001.

Adversarial Variational Bayes

Nguyen, Anh, Yosinski, Jason, Bengio, Yoshua, Dosovitskiy, Alexey, and Clune, Jeff. Plug & play generative
networks: Conditional iterative generation of images inlatent space. arXiv preprint arXiv:1612.00005, 2016.
Nguyen, XuanLong, Wainwright, Martin J, and Jordan,
Michael I. Estimating divergence functionals and thelikelihood ratio by convex risk minimization. IEEE
Transactions on Information Theory, 56(11):5847–5861,
2010.
Nowozin, Sebastian, Cseke, Botond, and Tomioka, Ryota. f-gan: Training generative neural samplers using variational divergence minimization. arXiv preprint
arXiv:1606.00709, 2016.
Poole, Ben, Alemi, Alexander A, Sohl-Dickstein, Jascha,and Angelova, Anelia. Improved generator objectives
for gans. arXiv preprint arXiv:1612.02780, 2016.
Radford, Alec, Metz, Luke, and Chintala, Soumith. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint
arXiv:1511.06434, 2015.
Ranganath, Rajesh, Tran, Dustin, Altosaar, Jaan, and Blei,
David. Operator variational inference. In Advances in
Neural Information Processing Systems, pp. 496–504,
2016.
Rezende, Danilo Jimenez and Mohamed, Shakir. Variational inference with normalizing flows. arXiv preprint
arXiv:1505.05770, 2015.
Rezende, Danilo Jimenez, Mohamed, Shakir, and Wierstra, Daan. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint
arXiv:1401.4082, 2014.
Salimans, Tim, Kingma, Diederik P, Welling, Max, et al.
Markov chain monte carlo and variational inference:
Bridging the gap. In ICML, volume 37, pp. 1218–1226,
2015.
Stan Development Team. Stan modeling language users
guide and reference manual, Version 2.14.0, 2016. URL
http://mc-stan.org.
Szabo, Zoltán. Information theoretical estimators (ite) toolbox. 2013.
Tran, Dustin, Ranganath, Rajesh, and Blei, David M.
The variational gaussian process.
arXiv preprint
arXiv:1511.06499, 2015.
van den Oord, Aaron, Kalchbrenner, Nal, Espeholt, Lasse,
Vinyals, Oriol, Graves, Alex, et al. Conditional image generation with pixelcnn decoders. In Advances In

Neural Information Processing Systems, pp. 4790–4798,
2016a.
van den Oord, Aaron van den, Kalchbrenner, Nal, andKavukcuoglu, Koray. Pixel recurrent neural networks.
arXiv preprint arXiv:1601.06759, 2016b.
Wu, Yuhuai, Burda, Yuri, Salakhutdinov, Ruslan, andGrosse, Roger.
On the quantitative analysis of
decoder-based generative models.
arXiv preprint
arXiv:1611.04273, 2016.