Venue: ICLR 2017 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper introduces the notion of a "variational lossy autoencoder", where a powerful autoregressive conditional distribution on the inputs x given the latent code z is crippled in a way that forces it to use z in a meaningful way. Its three main contributions are:

(1) It gives an interesting information-theoretical insight as to why VAE-type models don't tend to take advantage of their latent representation when the conditional distribution on x given z is powerful enough.

(2) It shows that this insight can be used to efficiently train VAEs with powerful autoregressive conditional distributions such that they make use of the latent code.

(3) It presents a powerful way to parametrize the prior in the form of an autoregressive flow transformation which is equivalent to using an inverse autoregressive flow transformation on the approximate posterior.

By itself, I think the information-theoretical explanation of why VAEs do not use their latent code when the conditional distribution on x given z is powerful enough constitutes an excellent addition to our understanding of VAE-related approaches.

However, the way this intuition is empirically evaluated is a bit weak. The "crippling" method used feels hand-crafted and very task-dependent, and the qualitative evaluation of the "lossyness" of the learned representation is carried out on three datasets (MNIST, OMNIGLOT and Caltech-101 Silhouettes) which feature black-and-white images with little-to-no texture. Figures 1a and 2a do show that reconstructions discard low-level information, as observed in the slight variations in strokes between the input and the reconstruction, but such an analysis would have been more compelling with more complex image datasets. Have the authors tried applying VLAE to such datasets?

I think the Caltech101 Silhouettes benchmark should be treated with caution, as no comparison is made against other competitive approaches like IAF VAE, PixelRNN and Conv DRAW. This means that VLAE significantly outperforms the state-of-the-art in only one of the four settings examined.

A question which is very relevant to this paper is "Does a latent representation on top of an autoregressive model help improve the density modeling performance?" The paper touches this question, but very briefly: the only setting in which VLAE is compared against recent autoregressive approaches shows that it wins against PixelRNN by a small margin.

The proposal to transform the latent code with an autoregressive flow which is equivalent to parametrizing the approximate posterior with an inverse autoregressive flow transformation is also interesting. There is, however, one important distinction to be made between the two approaches: in the former, the prior over the latent code can potentially be very complex whereas in the latter the prior is limited to be a simple, factorized distribution.

It is not clear to me that having a very powerful prior is necessarily a good thing from a representation learning point of view: oftentimes we are interested in learning a representation of the data distribution which is untangled and composed of roughly independent factors of variation. The degree to which this can be achieved using something as simple as a spherical gaussian prior is up for discussion, but finding a good balance between the ability of the prior to fit the data and its usefulness as a high-level representation certainly warrants some thought. I would be interested in hearing the authors' opinion on this.

Overall, the paper introduces interesting ideas despite the flaws outlined above, but weaknesses in the empirical evaluation prevent me from recommending its acceptance.

UPDATE: The rating has been revised to a 7 following the authors' reply.
review_title: Interesting ideas, weak evaluation.

final_score: rating: null
review_content: Have you tried different numbers of dimensions for the latent code?
Have you looked at how changing only a particular dimension of the latent code influences the decompressed image?
review_title: Latent code dimensions and interpretability

final_score: rating: null
review_content: In Table 1, when comparing the model with the IAF approximate posterior and the model with the equivalent AF prior, did you do a proper hyperparameter search for each model or did you use the same hyperparameters for both experiments?
review_title: Table 1 question

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a Variational Autoencoder model that can discard information found irrelevant, in order to learn interesting global representations of the data. This can be seen as a lossy compression algorithm, hence the name Variational Lossy Autoencoder. To achieve such model, the authors combine VAEs with neural autoregressive models resulting in a model that has both a latent variable structure and a powerful recurrence structure.

The authors first present an insightful Bits-Back interpretation of VAE to show when and how the latent code is ignored. As it was also mentioned in the literature, they say that the autoregressive part of the model ends up explaining all structure in the data, while the latent variables are not used. Then, they propose two complementary approaches to force the latent variables to be used by the decoder. The first one is to make sure the autoregressive decoder only uses small local receptive field so the model has to use the latent code to learn long-range dependency. The second is to parametrize the prior distribution over the latent code with an autoregressive model.

They also report new state-of-the-art results on binarized MNIST (both dynamical and statically binarization), OMNIGLOT and Caltech-101 Silhouettes.

Review:
The bits-Back interpretation of VAE is a nice contribution to the community. Having novel interpretations for a model helps to better understand it and sometimes, like in this paper, highlights how it can be improved.

Having a fine-grained control over the kind of information that gets included in the learned representation can be useful for a lot of applications. For instance, in image retrieval, such learned representation could be used to retrieve objects that have similar shape no matter what texture they have.

However, the authors say they propose two complementary classes of improvements to VAE, that is the lossy code via explicit information placement (Section 3.1) and learning the prior with autoregressive flow (Section 3.2). However, they never actually showed how a VAE without AF prior but that has a PixelCNN decoder performs. What would be the impact on the latent code is no AF prior is used?

Also, it is not clear if WindowAround(i) represents only a subset of x_{<i} or it can contain any data other than x_i. The authors mentioned the window can be represented as a small rectangle adjacent to a pixel x_i, must it only contains pixels above and to the left of x_i (similar to PixelCNN)

Minor:
In Equation 8, should there be an expectation over the data distribution?
review_title: Having control over the kind of information learned is interesting and would be useful in several applications

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper motivates the combination of autoregressive models with Variational Auto-Encoders and how to control the amount the amount of information stored in the latent code. The authors provide state-of-the-art results on MNIST, OMNIGLOT and Caltech-101.
I find that the insights provided in the paper, e.g. with respect to the effect of having a more powerful decoder on learning the latent code, the bit-back coding, and the lossy decoding are well-written but are not novel.
The difference between an auto-regressive prior and the inverse auto-regressive posterior is new and interesting though.
The model presented combines the recent technique of PixelRNN/PixelCNN and Variational Auto-Encoders with Inverse Auto-Regressive Flows, which enables the authors to obtain state-of-the-art results on MNIST, OMNIGLOT and Caltech-101. Given the insights provided in the paper, the authors are also able to control the amount of information contained in the latent code to an extent.
This paper gather several insight on Variational Auto-Encoders scattered through several publications in a well-written way. From these, the authors are able to obtain state-of-the-art models on small complexity datasets. Larger scale experiments will be necessary.
review_title: null

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: The AR prior and its equivalent - the inverse AR posterior - is one of the more elegant ways to improve the unfortunately poor generative qualities of VAE-s. It is only an incremental but important step. Incremental, because, judging by the lack of, say, CIFAR10 pictures of the VLAE in its "creative" regime ( i.e., when sampling from prior), it will not answer many of the questions hanging over. We hope to see the paper accepted: in relative terms, the paper shines in the landscape of the other papers which are rich on engineering hacks but lacking on theoretical insights.

Some disagreements with the theoretical suppositions in the paper:

i) The VAE-s posterior converges to the prior faster than we would like because the gradients of the "generative" error (the KL divergence of prior and posterior) w.r.t. mu & sigma are simple, inf differentiable functions and their magnitude far exceeds the magnitude of the resp. gradients of the reconstruction error. Especially when more "hairy" decoders like pixelCNN are used. We always considered this obvious and certainly not worthy of one page of CS mumbo-jumbo to explain. Dumbing-down the decoder via variations of dropout or "complexifying" the sampler as in here, or slapping the generative error with a "DeepMind" constant (beta_VAE), are the natural band-aids, but seem to fail in the "creative" regime, for real-life sets like CIFAR10 or more complex ones. Other conceptual solutions are needed, some are discussed in [2].

ii) The claim near the end of section 2.2 that "the extra coding cost a.k.a. variational error will exist and will not be negligible" is a speculation, which, in our empirical experience at least, is incorrect. The variational error is quantifiable for the Gibbs/exponential family of priors/posteriors, as described in [1], section 3.8, and as Tim Salimans knows from his own previous work. In the case of CIFAR10 for example, the variational error is negligible, even for simple sampling families like Gaussian, Laplacian, etc. Moreover, in hindsight, using the closed-form for generative error (the KL divergence of prior and posterior) in the pioneering VAE papers, was likely a mistake inherited by the unnecessary Bayseanism which inspired them (beautiful but a mistake nonetheless): The combo of generative and variational error should together be approximated by the same naive Monte Carlo used for the reconstruction error (easily follows from equation (3.13) in [1]) i.e. arithmetic average over observations.

On the lighter side, guys, please do not recycle ridiculous terms like "optimizationally challenged", as in section 2.2! The English language has recently acquired "mentally-challenged", "emotionally-challenged", etc, and now political correctness has sadly found its way to machines?

[1] https://arxiv.org/pdf/1508.06585v5.pdf
[2] https://arxiv.org/pdf/1511.02841v3.pdf
review_title: Important contribution to the ongoing research on rich VAE posteriors. Timely demo of the equivalence of certain rich posteriors to rich priors. Disputable theoretical suppositions but they are a tiny hole in a strong armor!

Meta-review: The reviewers agree that this is a well executed paper, and should be accepted and will make a positive contribution to the conference. In any final version please try to make a connection to the other paper at this conference with the same aims and execution.
Decision: Accept (Poster)