Auto-Encoding Variational Bayes
Authors: Diederik P. Kingma, Max Welling
ICLR 2014

Abstract: Can we efficiently learn the parameters of directed probabilistic models, in the presence of continuous latent variables with intractable posterior distributions? We introduce an unsupervised on-line learning method that efficiently optimizes the variational lower bound on the marginal likelihood and that, under some mild conditions, even works in the intractable case. The method optimizes a probabilistic encoder (also called a recognition network) to approximate the intractable posterior distribution of the latent variables. The crucial element is a reparameterization of the variational bound with an independent noise variable, yielding a stochastic objective function which can be jointly optimized w.r.t. variational and generative parameters using standard gradient-based stochastic optimization methods. Theoretical advantages are reflected in experimental results.


Peer review:
[Review]:
Reviewers, thank you for your overall very positive and constructive reviews. Your points of critique are highly appreciated and will be incorporated into a revised version.

Below we listed your key points, followed by our responses.

==> Reviewer bd7f: “Couldn’t an online Monte Carlo EM approach be followed?”

We tried an online version of Monte Carlo EM algorithm, but found that the online version didn’t compare favourably to the batch Monte Carlo EM algorithm used in experiments. The main reason is that the computational cost per E-step grows with the size of the dataset. With larger datasets, each minibatch gets processed more rarely, and the posterior distribution changes to a higher degree between epochs, such that more MCMC samples are required to get an approximately unbiased sample from the new posterior given a sample from the old posterior computed at the previous epoch.

==> Reviewer 79a6: “The experiments compare favourably […] but don’t give the reader a good sense of the trade-offs. A simple way to rectify this would be a discussion of the training: how sensitive the model is to various hyperparameter choices, how robust the model is to overfitting, etc.”

We’ll include a discussion of the trade-offs. Some of our responses below also relevant to this point.

==> Reviewer 79a6: “Different noise distributions are discussed, but they are not compared. What are the implications of choosing different classes of perturbation?”

Interesting question. One important aim is to chose an approximate posterior such that KL(q||p), the divergence of the approximate posterior q(z|x) from the true posterior p(z|x), can be minimized well. We chose a Gaussian since modes of posteriors are often locally approximately quadratic, in which case a Gaussian q(z|x) and leads to a low KL(q||p). We did perform some experiments with student-T (heavy-tailed) noise distributions, which did not improve the reported performance. Performance of this experiment were not included, since the main aim of the experiments is to compare to other approaches, rather than finding the optimal hyper-parameter setting.

==> Reviewer 79a6: “How powerful is the regularization effect of the variational bound? At what point does overfitting occur, or do things simply plateau when more latent variables are added?”

Train and test performances indeed plateau when increasing the dimensionality of latent space beyond necessity. For the models of MNIST, only about ~20 dimensions were used by the model; the others were 'switched off' automatically: after training their incoming and outgoing weights were approximately zero, and their posterior distributions were always (practically) equal to their prior distributions. Everything else being equal, a model with a latent dimensionality of 200 performed almost identical to a model with latent dimensionality of 20 (see figure 2).

==> Reviewer 79a6: “An important baseline would be to measure the effectiveness of this approach on held-out data. For example, rather than training this variational bound, one could perhaps use a regular auto-encoder with a similar architecture and train for reconstruction error. How would the authors expect the test-set reconstruction error of their approach to compare to this baseline? Is the only difference between the proposed approach and a standard auto-encoder the variational regularizer (first term of equation 10)?”

Besides the regularizer, an important distinctive feature of the variational auto-encoder is the noisy activation of the central hidden layer, distributed as q(z|x). The noise does not contributing to the reconstruction error, and can be tuned down by changing the encoder weights. The effect of removing the regularization term would be that the best solution is to ignore the noise (i.e. the variance of q is set to approximately zero). This would lead to a high KL divergence between the true and approximate posteriors, consequently a low marginal likelihood of the model, and most probably a high reconstruction error on held-out data.

==> Reviewer 79a6: “Intuitively, what is the regularizer provided by the variational bound doing to the encoding parameters? What would the regularization provided by the full variational approach do to the decoding parameters?”

The regularizer makes sure that q(z|x) does not diverge too much from the prior p(z). In practice this means that excess latent dimensions are “switched off”; for the fully connected models in the paper this means that all the incoming and outgoing connection weights corresponding to excess dimensions are set to zero. Therefore, the dimensionality of latent space does not require much tuning, and should simply be set to a high enough number.

Important to note is that overfitting can still occur simply by choosing too many units for the hidden layers of the encoder and/or decoder (i.e. the layers sandwiched between ‘x’ and ‘z’). In our comparative experiments we did not tune the amount of hidden units, and simply chose a regime educated by settings that work well for other auto-encoder architectures; we did not encounter overfitting problems in these experiments.

==> Reviewer 79a6: “In equation (6), should z^{(l)} be z^{(i,l)}?”

Indeed.