Title: Discrete Variational Autoencoders
Authors: Jason Tyler Rolfe
Venue: ICLR 2017
Keywords: Deep learning
Unsupervised Learning

Abstract: Probabilistic models with discrete latent variables naturally capture datasets composed of discrete classes. However, they are difficult to train efficiently, since backpropagation through discrete variables is generally not possible. We present a novel method to train a class of probabilistic models with discrete latent variables using the variational autoencoder framework, including backpropagation through the discrete latent variables. The associated class of probabilistic models comprises an undirected discrete component and a directed hierarchical continuous component. The discrete component captures the distribution over the disconnected smooth manifolds induced by the continuous component. As a result, this class of models efficiently learns both the class of objects in an image, and their specific realization in pixels, from unsupervised data; and outperforms state-of-the-art methods on the permutation-invariant MNIST, Omniglot, and Caltech-101 Silhouettes datasets.

TL;DR: We present a novel method to train a class of probabilistic models with discrete latent variables using the variational autoencoder framework, including backpropagation through the discrete latent variables.

Review discussion:
It seems like the derivatives of F^{-1} could have high variance in a multimodal setting.
For instance, consider this simplified example. We let z be a Bernoulli random variable with mean parameter \theta, and
\zeta | z=0 ~ N(0, 1)
\zeta | z=0 ~ N(10, 1)
Suppose \theta = 0.5 and \rho = 0.5, so that \zeta = 5 is in between the two modes. By my calculation, dF^{-1}(\rho) / d\theta is approximately 1/r(\zeta), which is 6.72e5. Since the derivative in this region is about 1/r(\zeta), the derivatives get large as fast as the probabilities get small, so maybe these low probability regions could still contribute significantly to the variance.
Because derivatives of dF^{-1} appear in the backprop computations, this could lead to high variance stochastic gradients in general. Is there some reason (either theoretical or empirical) to believe this isn't a problem?
Thank you for your question.  I believe you are correct that, given the particular smoothing distributions r(zeta | z) that you propose, much of the expected value of the gradients will occur in a region of low probability, and the stochastic approximation to the gradient will have high variance.
While this is a problem for some smoothing distributions r(zeta | z), it can be addressed by engineering a better r(zeta | z).  Viewed another way, I believe the difficulty you identify results because the smoothing distribution in question is poorly matched to the discrete variational autoencoder.  This is analogous to the way in which the transfer function logistic(c*x) is poorly matched to feedforward neural networks as c goes to infinity, even though it is fine when c=1.
The smoothing provided by the continuous random variables zeta is only effective if there is a region of meaningful overlap between r(zeta | z=0) and r(zeta | z=1).
In particular, there must be some zeta' such that r(zeta' | z=0) is not approximately zero, and r(zeta' | z=1) is not approximately zero.  As you observe, this does not hold in your example.
However, it is possible to significantly increase the amount of overlap, while maintaining a distinctly multimodal marginal distribution over zeta, by simply shifting your normal distributions closer together.
For instance, consider:
r(zeta | z=0) ~ N(0,1)
r(zeta | z=1) ~ N(4,1)
In this case, both the probabilities and the derivatives remain moderate in the region of overlap.  This overlap property can be ensured by either choosing a fixed r(zeta | z), or bounding the allowed parameters of r(zeta | z).  In the spike-and-exponential distribution described in Section 2.1, this overlap can be ensured by fixing or bounding beta.
This is an interesting and mathematically rather dense paper. I am slowly making my way through it. No specific questions at this time.
This is an interesting paper on how to handle reparameterization in VAEs when you have discrete variables. The idea is to introduce a smoothing transformation that is shared between the generative model and the recognition model (leading to cancellations).
A second contribution is to introduce an RBM as the prior model P(z) and to use autoregressive connections in generative and recognition models. The whole package becomes a bit entangled and complex and it is hard to figure out what causes the claimed good performance. Experiments that study these contributions separately would have been nice.
The framework does become a little complex but this should not be a problem if nice software is delivered that can be used in a plug and play mode.
Overall, the paper is very rich with ideas so I think it would be a great contribution to the conference.
Thank you very much for your review.  We have added the results of some experiments on simpler probabilistic models in Appendix J.  We are currently working on refactoring our code into a clear, clean implementation, and plan to publicly release it upon completion.
This paper presents a way of training deep generative models with discrete hidden variables using the reparameterization trick. It then applies it to a particular DBN-like architecture, and shows that this architecture achieves state-of-the-art density modeling performance on MNIST and similar datasets.
The paper is well written, and the exposition is both thorough and precise. There are several appendices which justify various design decisions in detail. I wish more papers in our field would take this degree of care with the exposition!
The log-likelihood results are quite strong, especially given that most of the competitive algorithms are based on continuous latent variables. Probably the main thing missing from the experiments is some way to separate out the contributions of the architecture and the inference algorithm. (E.g., what if a comparable architecture is trained with VIMCO, or if the algorithm is applied to a previously published discrete architecture?)
I’m a bit concerned about the variance of the gradients in the general formulation of the algorithm. See my comment “variance of the derivatives of F^{-1}” below. I think the response is convincing, but the problem (as well as “engineering principles” for the smoothing distribution) are probably worth pointing out in the paper itself, since the problem seems likely to occur unless the user is aware of it. (E.g., my proposal of widely separated normals would be a natural distribution to consider until one actually works through the gradients — something not commonly done in the age of autodiff frameworks.)
Another concern is how many sequential operations are needed for inference in the RBM model. (Note: is this actually an RBM, or a general Boltzmann machine?) The q distribution takes the form of an autoregressive model where the variables are processed one at a time. Section 3 mentions the possibility of grouping together variables in the q distribution, and this is elaborated in detail in Appendix A.
But the solution requires decomposing the joint into a product of conditionals and applying the CDFs sequentially.
So either way, it seems like we’re stuck handling all the variables sequentially, which might get expensive.
Minor: the second paragraph of Section 3 needs a reference to Appendix A.
Thank you very much for your review.
We always use a bipartite Boltzmann machine (i.e., an RBM) for our experiments.  Our derivations are consistent with fully connected Boltzmann machines, but sampling from fully connected Boltzmann machines is much more computationally demanding.
We use an approximating posterior (q distribution) that is autoregressive with respect to groups.  Group j depends upon all preceding groups i < j.  In the limit of each group comprising a single variable, the variables are processed one at a time.  In our experiments, we generally use four groups, each containing 32 variables, for a total of 128 binary variables.  We investigate the effect of the number of groups, holding the number of binary variables constant, in Figure 6c.
Within each group, the approximating posterior is a product of independent distributions, conditioned on the preceding groups.
As a result, all the approximating posteriors in a group can be sampled in parallel, by applying the inverse CDF (e.g., Equation 9) to each variable in parallel.  The number of sequential operations required is only equal to the number of groups, rather than the number of variables.
I have added Appendix D.3 to discuss the issue you identified regarding the variance of the gradient estimates when the smoothing transformation has insufficient overlap between the modes.  Thank you for pointing this out; I'm sure it will help anyone who tries to build upon this work.  I have also added the results of experiments on a few simplified versions of the architecture in Appendix J, which should help the reader to evaluate the contributions of the various components.
Paper proposes a novel Variational Encoder architecture that contains discrete variables. Model contains an undirected discrete component that captures distribution over disconnected manifolds and a directed hierarchical continuous component that models the actual manifolds (induced by the discrete variables). In essence the model clusters the data and at the same time learns a continuous manifold representation for the clusters. The training procedure for such models is also presented and is quite involved. Experiments illustrate state-of-the-art performance on public datasets (including MNIST, Omniglot, Caltech-101).
Overall the model is interesting and could be useful in a variety of applications and domains. The approach is complex and somewhat mathematically involved. It's not exactly clear how the model compares or relates to other RBM formulations, particularly those that contain discrete latent variables and continuous outputs. As a prime example:
Graham Taylor and Geoffrey Hinton. Factored conditional restricted Boltzmann machines for modeling motion style. In Proc. of the 26th International Conference on Machine Learning (ICML), 1025–1032, 2009.
Discussion of this should certainly be added.
Thank you very much for your review.
The generative model underlying the discrete variational autoencoder resembles a deep belief network DBN.  A DBN comprises a sigmoid belief network, the top layer of which is conditioned on the visible units of an RBM.
In contrast to a DBN, we use a bipartite Boltzmann machine, with both sides of the bipartite split connected to the rest of the model.
Moreover, all hidden layers below the bipartite Boltzmann machine are composed of continuous latent variables with a fully autoregressive layer-wise connection architecture.  Each layer j receives connections from all previous layers i < j, with connections from the bipartite Boltzmann machine mediated by a set of smoothing variables.
However, these architectural differences are secondary to those in the gradient estimation technique.
Whereas DBNs are traditionally trained by unrolling a succession of RBMs, discrete variational autoencoders use the reparameterization trick to backpropagate through the evidence lower bound.  I have added this discussion to Section 1.2 of the paper.
The factored conditional restricted Boltzmann machine for modeling motion style (Taylor & Hinton, 2009) is an interesting connection.
However, their one-hot style variables are always observed.
As a result, they train a family of models with discrete latent variables (traditional Gaussian-Bernoulli RBMs), conditioned on a pseudo-continuous component (as well as a buffer of previous time steps).
In response to the helpful comments from the peer reviewers and other readers, I have made some small additions to the paper.  The new version contains:
* A discussion of the relationship to deep belief networks to Section 1.2
* A note in Section 2.1 that a model without continuous smoothing variables zeta can easily be produced by taking the limit beta -> \infty, in which case zeta_i = z_i almost surely
* A discussion of variance of derivatives of the inverse CDF in Appendix D.3
* Some results on simplified probabilistic models in Appendix J
* A few small clarifications.
In particular, I have clarified the use of the term "restricted Boltzmann machine."  The Boltzmann machine used in the prior is bipartite, but all Boltzmann machine variables are directly connected to the smoothing variables zeta, and through them to the rest of the model.

Meta-review:
role: PC
data: comment: The authors present a novel reparameterization framework for VAEs with discrete random variables. The idea is to carry out symmetric projections of the approximate posterior and the prior into a continuous space and evaluating the autoencoder term in that space by marginalizing out the discrete variables. They consider the KL divergence between the approximating posterior and the true prior in the original discrete space and show that due to the symmetry of the projection into the continuous space, it does not
 contribute to the KL term. 
 
 One question that warrants further investigation is whether this framework can be extended to GANs and what empirical success they would have.
 
 The reviewers have presented a strong case for the acceptance of the paper and I go with their recommendation.

Decision: Accept (Poster)