Auto-Encoding Variational Bayes
Authors: Diederik P. Kingma, Max Welling
ICLR 2014

Abstract: Can we efficiently learn the parameters of directed probabilistic models, in the presence of continuous latent variables with intractable posterior distributions? We introduce an unsupervised on-line learning method that efficiently optimizes the variational lower bound on the marginal likelihood and that, under some mild conditions, even works in the intractable case. The method optimizes a probabilistic encoder (also called a recognition network) to approximate the intractable posterior distribution of the latent variables. The crucial element is a reparameterization of the variational bound with an independent noise variable, yielding a stochastic objective function which can be jointly optimized w.r.t. variational and generative parameters using standard gradient-based stochastic optimization methods. Theoretical advantages are reflected in experimental results.


Peer review:
[Review]:
This paper proposes a variational approach for training directed graphical models with continuous latent variables and intractable posteriors/marginals. The idea is to reparameterize the latent variables so that they can be written as a deterministic mapping followed by a stochastic perturbation. This allows Monte Carlo estimators of the variational lower bound to be differentiated with respect to the variational parameters.

The proposed method is novel and interesting, and seemingly more practical than previous approaches. The experiments favour comparably and show that this method is quite promising, however they still feel preliminary and don’t give the reader a good sense of the trade-offs and difficulties one might encounter using this approach. A simple way to rectify this would be a discussion of the training: how sensitive the model is to various hyperparameter choices, how robust the model is to overfitting, etc. Another way would be to compare against a tractable model such as PPCA, and show how the approach compares to EM and exact inference. The results here would give a sense of how the method might perform on intractable cases.

On a high level, the auto-encoder presented here reminds me of the back-constrained GP-LVM [1]. The goals are different, but the models seem similar.

[1] Local Distance Preservation in the GP-LVM through Back Constraints, Neil D. Lawrence and Joaquin Quinonero-Candela, ICML 2006

Questions:
-Different noise distributions are discussed, but they are not compared. What are the implications of choosing different classes of perturbation?

-How powerful is the regularization effect of the variational bound? At what point does overfitting occur, or do things simply plateau when more latent variables are added?

-An important baseline would be to measure the effectiveness of this approach on held-out data. For example, rather than training this variational bound, one could perhaps use a regular auto-encoder with a similar architecture and train for reconstruction error. How would the authors expect the test-set reconstruction error of their approach to compare to this baseline? Is the only difference between the proposed approach and a standard auto-encoder the variational regularizer (first term of equation 10)?

-Intuitively, what is the regularizer provided by the variational bound doing to the encoding parameters? What would the regularization provided by the full variational approach do to the decoding parameters?

Typos and grammar:
-The first sentence could perhaps be reworded slightly.

-Please remove “is” from “is even works” in the sentence ”Conversely, we are here interested in a general algorithm that is even works efficiently in the case of…”

-In equation (6), should z^{(l)} be z^{(i,l)}?