Title: Variational Lossy Autoencoder
Authors: Xi Chen
Diederik P. Kingma
Tim Salimans
Yan Duan
Prafulla Dhariwal
John Schulman
Ilya Sutskever
Pieter Abbeel
Venue: ICLR 2017
Keywords: Deep learning
Unsupervised Learning

Abstract: Representation learning seeks to expose certain aspects of observed data in a learned representation that's amenable to downstream tasks like classification. 
For instance, a good representation for 2D images might be one that describes only global structure and discards information about detailed texture. 
In this paper, we present a simple but principled method to learn such global representations by combining Variational Autoencoder (VAE) with neural autoregressive models such as RNN, MADE and PixelRNN/CNN. 
Our proposed VAE model allows us to have control over what the global latent code can learn and , by designing the architecture accordingly, we can force the global latent code to discard irrelevant information such as texture in 2D images, and hence the code only ``autoencodes'' data in a lossy fashion.
In addition, by leveraging autoregressive models as both prior distribution $p(z)$ and decoding distribution $p(x|z)$, we can greatly improve generative modeling performance of VAEs, achieving new state-of-the-art results on MNIST, OMNIGLOT and Caltech-101 as well as competitive results on CIFAR10.  


TL;DR: A VAE that provably learns global structure of images with a local PixelCNN decoder.

Review discussion:
In Table 1, when comparing the model with the IAF approximate posterior and the model with the equivalent AF prior, did you do a proper hyperparameter search for each model or did you use the same hyperparameters for both experiments?
Thanks for your question! I apologize for the late response due to NIPS.
- did you do a proper hyperparameter search for each model or did you use the same hyperparameters for both experiments?
We swept and chose best hyperparameters for the IAF model while working on the camera-ready version of its corresponding paper [1]. We then used the same set of hyperparameters for AF model with little further tweaking.
[1]: Kingma, Diederik P., Tim Salimans, and Max Welling. "Improving variational inference with inverse autoregressive flow." arXiv preprint arXiv:1606.04934 (2016).
Have you tried different numbers of dimensions for the latent code?
Have you looked at how changing only a particular dimension of the latent code influences the decompressed image?
Thanks for your questions! I apologize for the late response due to NIPS.
- Have you tried different numbers of dimensions for the latent code?
We have tried a couple numbers of dimensions like 16, 32, ... 128.
In terms of density estimation, a higher-dimensional latent code usually gives better performance, possibly because optimization is easier in those cases.
- Have you looked at how changing only a particular dimension of the latent code influences the decompressed image?
We haven't, however, looked at the case of using an low-dimensional latent codes that have a higher chance of being interpretable. I think investigating interpretable latent codes is a good way to gain more understanding of lossy latent codes and we will perform some experiments with techniques in [1,2].
[1]: Chen, Xi, et al. "Infogan: Interpretable representation learning by information maximizing generative adversarial nets." Advances in Neural Information Processing Systems. 2016.
[2]: Higgins, Irina, et al. "Early visual concept learning with unsupervised deep learning." arXiv preprint arXiv:1606.05579 (2016).
This paper motivates the combination of autoregressive models with Variational Auto-Encoders and how to control the amount the amount of information stored in the latent code. The authors provide state-of-the-art results on MNIST, OMNIGLOT and Caltech-101.
I find that the insights provided in the paper, e.g.
with respect to the effect of having a more powerful decoder on learning the latent code, the bit-back coding, and the lossy decoding are well-written but are not novel.
The difference between an auto-regressive prior and the inverse auto-regressive posterior is new and interesting though.
The model presented combines the recent technique of PixelRNN/PixelCNN and Variational Auto-Encoders with Inverse Auto-Regressive Flows, which enables the authors to obtain state-of-the-art results on MNIST, OMNIGLOT and Caltech-101. Given the insights provided in the paper, the authors are also able to control the amount of information contained in the latent code to an extent.
This paper gather several insight on Variational Auto-Encoders scattered through several publications in a well-written way. From these, the authors are able to obtain state-of-the-art models on small complexity datasets. Larger scale experiments will be necessary.
Dear AnonReviewer1,
Thank you for your thoughtful review! I'd like to ask a few clarification questions that can help us better prepare next revision:
- "I find that the insights provided in the paper, e.g.
with respect to the effect of having a more powerful decoder on learning the latent code, the bit-back coding, and the lossy decoding are well-written but are not novel.
I'd like to agree with you that the connection between bits-back coding and variational inference is widely known, as acknowledged in original submission.
Nevertheless, to the best of my knowledge, there is no prior work that discusses lossy decoding and information placement. If you could point us to related work, we would include proper citations and discussions.
- "Larger scale experiments will be necessary."
Is this the main concern you have about the current revision? If so, what kind of evaluations do you want to see on datasets with higher complexity?
Thanks again for your time. I hope that, through more discussion enabled an open review format, we can improve the manuscript to make it more useful to our community.
"Nevertheless, to the best of my knowledge, there is no prior work that discusses lossy decoding and information placement. If you could point us to related work, we would include proper citations and discussions."
I think Luke Vilnis comment was referring to that: the necessity of "crippling" the decoder distribution as to learn any informative latent code has already been pointed out. This crippling in their paper has been done through the use of their "word dropout".
"Is this the main concern you have about the current revision? If so, what kind of evaluations do you want to see on datasets with higher complexity?"
You state that you already have done experiments on CIFAR-10.
While you do not achieve state-of-the-art, you can do similar experiments regarding the information contained in the code. A fair amount of limitation in placing information in the code is already present in Omniglot, does it break down in CIFAR-10 ?
It seems that one of the objectives of this paper is to show that you can boost performance of VAEs while still keeping a meaningful latent code. Showing that it's the case on more challenging datasets like CIFAR-10 or small Imagenet would strengthen this claim.
- "the necessity of "crippling" the decoder distribution as to learn any informative latent code has already been pointed out"
We acknowledged that [1,2,3] discussed the phenomenon that the latent code (or part of it) will be ignored when there is a strong decoder distribution and hence either crippling decoder or other tricks like KL annealing or freebits is necessary. The new insight that we offer is "why this happens" rather than pointing out "this phenomenon can happen". More importantly, this work includes the key discussion that what information will be explained by the decoder distribution and what will be explained by the latent code, which we have not found in literature. We believe that our discussion on lossy coding provides new analysis but again we are happy to include any relevant previous work that we overlooked.
While the current revision hasn't reflected this change, the next revision will include discussion of "word dropout" as well.
- large-scale experiments
The limitation of Omniglot is intentionally shown since the Omniglot experiment uses the same receptive field size as MNIST while this dataset contains meaningful variation that's finer. We can definitely provide additional lossy coding experiments on richer datasets in next revision to show we can similarly control information placement.
In addition, spatial locality (limited-depth PixelCNN as decoder) is just one way to construct a lossy code and we used this as an example to illustrate the ideas that we discussed. We believe how to do it differently is an interesting future research direction.
Thank you again for participating in this discussion - it's much better than a one-off rebuttal. I hope we will be able to reach an understanding.
[1]: Bowman, Samuel R., et al. "Generating sentences from a continuous space." arXiv preprint arXiv:1511.06349 (2015).
[2]: Sønderby, Casper Kaae, et al. "Ladder variational autoencoders." Advances In Neural Information Processing Systems. 2016.
[3]: Kingma, Diederik P., Tim Salimans, and Max Welling. "Improving variational inference with inverse autoregressive flow." arXiv preprint arXiv:1606.04934 (2016).
This paper introduces the notion of a "variational lossy autoencoder", where a powerful autoregressive conditional distribution on the inputs x given the latent code z is crippled in a way that forces it to use z in a meaningful way. Its three main contributions are:
(1) It gives an interesting information-theoretical insight as to why VAE-type models don't tend to take advantage of their latent representation when the conditional distribution on x given z is powerful enough.
(2) It shows that this insight can be used to efficiently train VAEs with powerful autoregressive conditional distributions such that they make use of the latent code.
(3) It presents a powerful way to parametrize the prior in the form of an autoregressive flow transformation which is equivalent to using an inverse autoregressive flow transformation on the approximate posterior.
By itself, I think the information-theoretical explanation of why VAEs do not use their latent code when the conditional distribution on x given z is powerful enough constitutes an excellent addition to our understanding of VAE-related approaches.
However, the way this intuition is empirically evaluated is a bit weak. The "crippling" method used feels hand-crafted and very task-dependent, and the qualitative evaluation of the "lossyness" of the learned representation is carried out on three datasets (MNIST, OMNIGLOT and Caltech-101 Silhouettes) which feature black-and-white images with little-to-no texture. Figures 1a and 2a do show that reconstructions discard low-level information, as observed in the slight variations in strokes between the input and the reconstruction, but such an analysis would have been more compelling with more complex image datasets. Have the authors tried applying VLAE to such datasets?
I think the Caltech101 Silhouettes benchmark should be treated with caution, as no comparison is made against other competitive approaches like IAF VAE, PixelRNN and Conv DRAW. This means that VLAE significantly outperforms the state-of-the-art in only one of the four settings examined.
A question which is very relevant to this paper is "Does a latent representation on top of an autoregressive model help improve the density modeling performance?" The paper touches this question, but very briefly: the only setting in which VLAE is compared against recent autoregressive approaches shows that it wins against PixelRNN by a small margin.
The proposal to transform the latent code with an autoregressive flow which is equivalent to parametrizing the approximate posterior with an inverse autoregressive flow transformation is also interesting. There is, however, one important distinction to be made between the two approaches: in the former, the prior over the latent code can potentially be very complex whereas in the latter the prior is limited to be a simple, factorized distribution.
It is not clear to me that having a very powerful prior is necessarily a good thing from a representation learning point of view: oftentimes we are interested in learning a representation of the data distribution which is untangled and composed of roughly independent factors of variation. The degree to which this can be achieved using something as simple as a spherical gaussian prior is up for discussion, but finding a good balance between the ability of the prior to fit the data and its usefulness as a high-level representation certainly warrants some thought. I would be interested in hearing the authors' opinion on this.
Overall, the paper introduces interesting ideas despite the flaws outlined above, but weaknesses in the empirical evaluation prevent me from recommending its acceptance.
UPDATE: The rating has been revised to a 7 following the authors' reply.
Dear AnonReviewer3,
Thank you for reading our manuscript carefully! Below are some answers to questions in your review. I'd like to also ask a few clarification questions that can help us better prepare next revision:
- "However, the way this intuition is empirically evaluated is a bit weak ....
but such an analysis would have been more compelling with more complex image datasets"
Do you want to see qualitative results on colored datasets that have richer variations? We do have some initial results on CIFAR10 that show characteristics qualitatively similar to current binary experiments: global shape being preserved whilst local color patterns and exact details change from one decompression to another. We can also run similar experiments on higher-resolution datasets if this is what you are looking for.
- "This means that VLAE significantly outperforms the state-of-the-art in only one of the four settings examined."
We'd like to reiterate that on three out of four settings, we kept the hyperparameters fixed for VLAE and we kept VLAE at a disadvantage to assess its ability to generalize across datasets. We do out-perform previous SOTA (ConvDraw)  on OMNIGLOT, for instance, more significantly when we tune hyperparameters specifically for this dataset. Do you want to see this kind of evaluation?
- ""Does a latent representation on top of an autoregressive model help improve the density modeling performance?" The paper touches this question, but very briefly: the only setting in which VLAE is compared against recent autoregressive approaches shows that it wins against PixelRNN by a small margin."
I apologize for the confusion as the question that we try to ask and answer is "Does an autoregressive model as decoder help improve latent-code models' density estimation performance". Since a latent-code model like VAE suffers some penalty in code length for using approximate posterior, PixelCNN/RNN seems to be a better fit for just getting good logprob.
Nevertheless, it's usually hard to get good representations from this kind of autoregressive models, unlike latent-code models. We believe part of this work's value is improving density estimation performance for latent-code models specifically.
- "It is not clear to me that having a very powerful prior is necessarily a good thing ....."
A prior that's transformed by an autoregressive flow can indeed become quite complicated and less useful for learning disentangled representations. If we are interested in obtaining a latent code that has factorized distribution, we can treat the noise source \epsilon in (14) as the true latent code.
- "weaknesses in the empirical evaluation prevent me from recommending its acceptance."
We thought the current set of experiments, though on smaller scale datasets, do validate the 3 main contributions that you summarized excellently, but we are definitely open to suggestions. Is there any additional evaluation not covered in questions above that you think is important?
Thanks again for your time. I hope that, through more discussion enabled an open review format, we can improve the manuscript to make it more useful to our community.
Thank you for the clarifications.
Regarding the evaluation, my issues have been reasonably addressed by the revised version, both in terms of NLL comparison and qualitative evaluation of the effect of "crippling" on how information is discarded.
I also feel like my other points have been addressed satisfyingly.
As such, I updated my rating to 7.
This paper proposes a Variational Autoencoder model that can discard information found irrelevant, in order to learn interesting global representations of the data. This can be seen as a lossy compression algorithm, hence the name Variational Lossy Autoencoder. To achieve such model, the authors combine VAEs with neural autoregressive models resulting in a model that has both a latent variable structure and a powerful recurrence structure.
The authors first present an insightful Bits-Back interpretation of VAE to show when and how the latent code is ignored. As it was also mentioned in the literature, they say that the autoregressive part of the model ends up explaining all structure in the data, while the latent variables are not used.
Then, they propose two complementary approaches to force the latent variables to be used by the decoder. The first one is to make sure the autoregressive decoder only uses small local receptive field so the model has to use the latent code to learn long-range dependency. The second is to parametrize the prior distribution over the latent code with an autoregressive model.
They also report new state-of-the-art results on binarized MNIST (both dynamical and statically binarization), OMNIGLOT and Caltech-101 Silhouettes.
Review:
The bits-Back interpretation of VAE is a nice contribution to the community. Having novel interpretations for a model helps to better understand it and sometimes, like in this paper, highlights how it can be improved.
Having a fine-grained control over the kind of information that gets included in the learned representation can be useful for a lot of applications.
For instance, in image retrieval, such learned representation could be used to retrieve objects that have similar shape no matter what texture they have.
However, the authors say they propose two complementary classes of improvements to VAE, that is the lossy code via explicit information placement (Section 3.1) and learning the prior with autoregressive flow (Section 3.2).
However, they never actually showed how a VAE without AF prior but that has a PixelCNN decoder performs. What would be the impact on the latent code is no AF prior is used?
Also, it is not clear if WindowAround(i) represents only a subset of x_{<i} or it can contain any data other than x_i. The authors mentioned the window can be represented as a small rectangle adjacent to a pixel x_i, must it only contains pixels above and to the left of x_i (similar to PixelCNN)
Minor:
In Equation 8, should there be an expectation over the data distribution?
Dear AnonReviewer2,
Thank you for your thoughtful review and we are glad that you think learning lossy features that are relevant is important！Below are our answers to your questions:
- "What would be the impact on the latent code is no AF prior is used?"
We addressed this question indirectly in text "A similar gain carries over when an autoregressive decoder is used: on statically binarized MNIST, using AF prior instead of IAF posterior reduces train NLL by $0.8$ nat and test NLL by $0.6$ nat." And the effect of introducing IAF posterior to a VAE that uses PixelCNN decoder is similar to that of a VAE that uses factorized decoder (Table 1 in [1]). We will address this more directly in next revision by giving direct ablation experiments to show the complementary benefits of both modifications.
- "The authors mentioned the window can be represented as a small rectangle adjacent to a pixel x_i, must it only contains pixels above and to the left of x_i (similar to PixelCNN)"
The small window will be exacly the local receptive field of a depth-limited PixelCNN. We will include visualizations to make the exact definition used more explicit.
- "In Equation 8, should there be an expectation over the data distribution?"
Yes, you are right! It will be fixed in next revision.
Thanks again for taking time to review.
[1]: http://papers.nips.cc/paper/6581-improving-variational-autoencoders-with-inverse-autoregressive-flow
Dear Reviewers & AC:
We'd like to thank all the reviewers & commenters for their useful suggestions. By taking into account discussion so far, we have significantly updated the manuscript to address questions/concerns.
Minor clarification questions have been answered in discussion below. Here we summarize the main concerns of reviewers and subsequently explain how our latest revision address these concerns:
*** Larger scale experiments (Reviewer1, Reviewer3)
This is the main focus of the latest revision. In Section 4.3 of the latest revision, we have shown:
1. VLAE has the current best density estimation performance on CIFAR10 among latent-code models. It also outperforms PixelCNN/PixelRNN [1] and is only second to PixelCNN++ [2].
2. We show several different ways to encode different kinds of information into latent code to demonstrate that one can similarly control information placement on CIFAR10.
*** "the necessity of "crippling" the decoder ... has already been pointed out" (Reviewer1)
Thanks for making the distinction clear. We have revised relevant text (last paragraph of introduction for example) to highlight that the analysis of "why crippling is necessary" is our main contribution as opposed to "pointing out it's necessary" [3].
*** "However, they never actually showed how a VAE without AF prior but that has a PixelCNN decoder performs. What would be the impact on the latent code is no AF prior is used?" (Reviewer2)
Thanks for your suggestion and we have added related experiments in Appendix. C (edit: should be Appendix. D) of the latest revision of the paper. These experiments show removing AF prior will result in worse performance and make the latent code carry less information.
Next we briefly summarize additional contributions in the latest revision.
In the initial revision, we hypothesized that by improving VAE training procedure, VLAE's modelling performance can be improved. In initial experiments, we used the “free-bits” technique [5] and this turned out to create optimization challenges. In the latest revision, we proposed a new technique "soft free-bits" (described in Appendix.C) which has resulted in more robust optimization.
We also investigated replacing ResNet VAEs [4,5] with DenseNet [6] VAEs and this change reduces overfitting. “Soft free-bits” and DenseNet VAEs are novel changes and enable VLAE models to outperform previous latent-code models like ResNet VAE w/ IAF [5] as well as competitive autoregressive density estimators like PixelRNN [1] on CIFAR10. These techniques are not unique to VLAE and we suspect they will also be useful to other types of VAEs.
We also have added one more way to separate information -- by restricting the PixelCNN decoder to condition only on grayscale image, a lossy latent code can be forced to learn color information.
We believe we have addressed reviewers' concerns and overall strengthened this submission. We hope reviewers can take a look at the latest revision and possibly adjust rating in light of new evidence.
We are happy to answer any additional questions!
PS: We also plan to include experiments with 64x64 datasets but due to time constraints we haven't been able to finish them in time for inclusion in the current revision. We will include them in a later version and release all code when experiments are finished.
[1]: van den Oord, Aaron, Nal Kalchbrenner, and Koray Kavukcuoglu. "Pixel Recurrent Neural Networks." arXiv preprint arXiv:1601.06759 (2016).
[2]: https://openreview.net/forum?id=BJrFC6ceg
[3]: Bowman, Samuel R., et al. "Generating sentences from a continuous space." arXiv preprint arXiv:1511.06349 (2015).
[4]: Salimans, Tim, Diederik P. Kingma, and Max Welling. "Markov chain Monte Carlo and variational inference: Bridging the gap." International Conference on Machine Learning. 2015.
[5]: Kingma, Diederik P., Tim Salimans, and Max Welling. "Improving variational inference with inverse autoregressive flow." arXiv preprint arXiv:1606.04934 (2016).
[6]: Huang, Gao, et al. "Densely connected convolutional networks." arXiv preprint arXiv:1608.06993 (2016).
When you say "...
when Kl is too much higher than λ, γ is increased, and when Kl is too low, ...", how do you determine "too much" and "too low"? I guess those thresholds are hyperparameters, what kind of values did you try and work for you?
Also, just to clarify in your previous comment when you say "experiments in Appendix. C of the latest revision of the paper", you are in fact referring to Appendix D, right?
Thank you for following up!
- "When you say "...
when Kl is too much higher than λ, γ is increased, and when Kl is too low, ...", how do you determine "too much" and "too low"? I guess those thresholds are hyperparameters, what kind of values did you try and work for you?"
Thresholds are hyperparameters and we tried a wide range from 3%~30%, all of which yield improved results. We have added more details to that section (PDF updated).
- "Also, just to clarify in your previous comment when you say "experiments in Appendix. C of the latest revision of the paper", you are in fact referring to Appendix D, right?"
Yes you are right, thanks for catching that (comment above edited).

Meta-review:
role: PC
data: comment: The reviewers agree that this is a well executed paper, and should be accepted and will make a positive contribution to the conference. In any final version please try to make a connection to the other paper at this conference with the same aims and execution.

Decision: Accept (Poster)