Auto-Encoding Variational Bayes
Authors: Diederik P. Kingma, Max Welling
ICLR 2014

Abstract: Can we efficiently learn the parameters of directed probabilistic models, in the presence of continuous latent variables with intractable posterior distributions? We introduce an unsupervised on-line learning method that efficiently optimizes the variational lower bound on the marginal likelihood and that, under some mild conditions, even works in the intractable case. The method optimizes a probabilistic encoder (also called a recognition network) to approximate the intractable posterior distribution of the latent variables. The crucial element is a reparameterization of the variational bound with an independent noise variable, yielding a stochastic objective function which can be jointly optimized w.r.t. variational and generative parameters using standard gradient-based stochastic optimization methods. Theoretical advantages are reflected in experimental results.


Peer review:
[Review]:
This paper adresses the problem of learning the parameters of directed probabilistic models with continuous latent variables when the latent posterior is intractable.

The proposed method (Auto-Encoding Variational Bayes or AEVB for short) can be summarized as:
 1 - Maximization of the variational lower bound as a proxy for the marginal log-likelihood.
 2 - Estimation of this lower-bound using a Monte-Carlo estimation.
 3 - Using a re-parametrization trick for the parametric variational inference approximation q_{	heta}(z|x), i.e. expressing it as a deterministic function f(x,e) of x and e, where e is a random variable.

The AEVB method is in theory applicable to a wide range of choices for p(x|z), p(z) and q(z|x). In their study, the authors propose a specific application where the prior p(z) is a simple gaussian centered on 0 and where p(x|z) and q(z|x) are both Gaussian distributions with their means and covariance matrices being determined by the output of a neural network. For instance q(z|x) is defined as
	q(z|x) = N(z;mu,sigma^2*I) with mu and sigma being determined from x using a neural network, i.e. mu = f_1(x) and sigma = f_2(x), with f_1 and f_2 deterministic.

This work is closely related to several recent contributions (Generative Stochastic Networks,Denoising Auto-Encoder theory) which are properly discussed.

Several experiments are run on both the Frey-faces dataset and the Mnist dataset:
 - quantitatively comparing the lower bound with AEVB vs with the wake sleep algorithm.
 - quantitatively comparing the marginal likelihood (with small latent space as is to be expected) with AEVB vs with the wake sleep algorithm.
 - showing visualizations of 2D manifolds learned with AEVB
 - showing random samples obtained by sampling from the learned generative model
 
The proposed approach (AEVB) offers a new and exciting solution to the well known problem of estimating the parameters of a graphical model. Both the variational inference and generative distribution can be quite complex as they are both represented using neural networks with hidden layers, however the training criterion is tractable and includes hyper-parameter-free regularization terms which can prevent overfitting, as supported by the experiments.

pros:
 - Very good summary of previous work / very good discussion of related work.
 - The approach is tractable even with intractable latent posteriors.
 - The training criterion contains hyper-parameter free regularization.
 - The experiments fully support the practical applicability of the approach.
cons:
 - Experiments on more complex datasets would have made for an even more convincing argument.