Auto-Encoding Variational Bayes
Authors: Diederik P. Kingma, Max Welling
ICLR 2014

Abstract: Can we efficiently learn the parameters of directed probabilistic models, in the presence of continuous latent variables with intractable posterior distributions? We introduce an unsupervised on-line learning method that efficiently optimizes the variational lower bound on the marginal likelihood and that, under some mild conditions, even works in the intractable case. The method optimizes a probabilistic encoder (also called a recognition network) to approximate the intractable posterior distribution of the latent variables. The crucial element is a reparameterization of the variational bound with an independent noise variable, yielding a stochastic objective function which can be jointly optimized w.r.t. variational and generative parameters using standard gradient-based stochastic optimization methods. Theoretical advantages are reflected in experimental results.


Peer review:
[Review]:
We have just submitted a new version to arXiv.

In previous versions, we treated both the MNIST and Frey Face datasets as binary, where the pixel values (normalized to the interval (0,1)) were treated as probabilities, using the negative binary cross-entropy function in place of log p(x|z). This use of the binary cross-entropy function is common practice, and is equivalent (in expectation of the objective and gradients) to log p(x|z) with binary samples drawn from the data according to the probabilities indicated by the pixel values. While this treatment of pixel values makes sense for MNIST (the dataset is almost binary anyway), it makes less sense for Frey Face.
We therefore ran new experiments where the Frey Face data was treated appropriately as continuous data. We chose log p(x|z) to be an isotropic Gaussian, with a conditional distribution identical to log q(z|x), with a minor difference that the predicted means were constrained to the interval (0,1) by appending a sigmoid activation function to the MLP output units that correspond to the predicted mean of log p(x|z). 

As shown in the new version of the paper, the wake-sleep algorithm had much more difficulty with learning this (more appropriate) model of the Frey Face data.

Another difference is that models were allowed to train for a longer period of time, and results for higher dimensional latent space were included. Most interestingly, superfluous dimensionality of latent space did not result in increased overfitting, which is explained by the regularizing effect of the prior and entropy terms of the objective function.