Auto-Encoding Variational Bayes
Authors: Diederik P. Kingma, Max Welling
ICLR 2014

Abstract: Can we efficiently learn the parameters of directed probabilistic models, in the presence of continuous latent variables with intractable posterior distributions? We introduce an unsupervised on-line learning method that efficiently optimizes the variational lower bound on the marginal likelihood and that, under some mild conditions, even works in the intractable case. The method optimizes a probabilistic encoder (also called a recognition network) to approximate the intractable posterior distribution of the latent variables. The crucial element is a reparameterization of the variational bound with an independent noise variable, yielding a stochastic objective function which can be jointly optimized w.r.t. variational and generative parameters using standard gradient-based stochastic optimization methods. Theoretical advantages are reflected in experimental results.


Peer review:
[Review]:
This paper presents a variational learning algorithm  for probabilistic graphical model embedded with a latent representation that is continuous-valued. The proposed algorithm is stochastic in nature. It is able to scale to large datasets and deal with models for which tractable EM is not an option. It is based on gradient descent on a Monte Carlo approximation of the standard variational lower-bound on the marginal likelihood. Crucially, in order to obtain an unbiased gradient estimate, the Monte Carlo approximation is taken such that the sampling of the MC points is independent of the parameters of the model, thanks to an 'alternative parametrization trick' of the variational posterior. An interesting connection is also made with autoencoders. Experiments show that the algorithm can improve over the wake-sleep algorithm.  

I really like this paper. The idea is simple and non-trivial. The alternative parametrization trick is a nice tool to have in one's toolbox. The connection with autoencoders is also really nice.

The only 'cons' I see is that the experiments are somewhat limited, in the sense that they are performed only on MNIST and Frey Faces, and only log-likelihood or visualization results are reported (as opposed to experiments in the context of a real application). But I still think this is good first step in a potentially promising direction for training complex non-linear continuous representation models. So I'd argue that this work deserves to get in.

Minor comments:
- Couldn't an online Monte Carlo EM approach be followed, where each update of the M step would be performed on a minibatch?
- This is super minor, but NADE isn't just applicable to binary observations and actually has a real-valued extension (RNADE: The real-valued neural autoregressive density-estimator, by Uria, Murray and Larochelle, NIPS 2013). Otherwise however, as the authors mention, it's not directly related to the approach in this paper.