Title: PixelVAE: A Latent Variable Model for Natural Images
Authors: Ishaan Gulrajani
Kundan Kumar
Faruk Ahmed
Adrien Ali Taiga
Francesco Visin
David Vazquez
Aaron Courville
Venue: ICLR 2017
Keywords: Deep learning
Unsupervised Learning

Abstract: Natural image modeling is a landmark challenge of unsupervised learning. Variational Autoencoders (VAEs) learn a useful latent representation and model global structure well but have difficulty capturing small details. PixelCNN models details very well, but lacks a latent code and is difficult to scale for capturing large structures. We present PixelVAE, a VAE model with an autoregressive decoder based on PixelCNN. Our model requires very few expensive autoregressive layers compared to PixelCNN and learns latent codes that are more compressed than a standard VAE while still capturing most non-trivial structure. Finally, we extend our model to a hierarchy of latent variables at different scales. Our model achieves state-of-the-art performance on binarized MNIST, competitive performance on 64 × 64 ImageNet, and high-quality samples on the LSUN bedrooms dataset.


TL;DR: VAE with an autoregressive PixelCNN-based decoder with strong performance on binarized MNIST, ImageNet 64x64, and LSUN bedrooms.

Review discussion:
- What exactly are the architectures for the VAE and PixelCNN layers used in the LSUN and ImageNet experiments (in terms of number of layers and units per layer)?
- Do you have any guarantees for the robustness of the estimate of the NLL on the MNIST dataset?
- Could you please further explain in what sense “The KL divergence term can be interpreted as a measure of the information content in the posterior distribution q(z|x)”
Thanks for your questions. Our answers are below; we'll also revise the paper to include these.
> What exactly are the architectures for the VAE and PixelCNN layers used in the LSUN and ImageNet experiments (in terms of number of layers and units per layer)?
The LSUN and ImageNet models use the same architecture: all encoders and decoders are residual networks; we use pre-activation residual blocks (He et al. 2016) with two 3x3 conv layers each and ELU nonlinearity. Some residual blocks perform either downsampling (using a 2x2 stride in the second convolutional layer) or upsampling (using subpixel convolution in the first convolutional layer). Weight normalization is used throughout the model. We optimize using Adam with learning rate 5e-4 for LSUN and 3e-4 for ImageNet. Training proceeds for 400K iterations using batch size 64.
The residual blocks in the encoders are as follows (by number of output channels):
x -> h1 encoder: 128 -> 256 [downsample] -> 256 -> 512 [downsample] -> 512 -> 512 [downsample] -> 512 -> 512
h1 -> h2 encoder: 512 -> 512 -> 512 [downsample] -> 512 -> 512
The decoder architectures are symmetrical to the encoders, with the addition of the following extra masked convolutions at the end (see figure 2):
z2 -> z1 decoder: 512 5x5 linear conv -> 512 residual block -> 512 residual block -> 512 residual block with 1x1 convs
z1 -> x decoder: 256 5x5 linear conv -> 256 residual block -> 256 residual block -> 256 residual block
> Do you have any guarantees for the robustness of the estimate of the NLL on the MNIST dataset?
We estimate the marginal likelihood of our MNIST model using the importance sampling technique in Importance Weighted Autoencoders (Burda et al. 2016), which computes a lower bound on the likelihood whose tightness increases with the number of importance samples per datapoint. We use N=1000 samples per datapoint; higher values don't appear to significantly affect the likelihood estimate.
> Could you please further explain in what sense “The KL divergence term can be interpreted as a measure of the information content in the posterior distribution q(z|x)”
We mean it in the sense that in expectation, samples from q(z|x) require KL(q||p) fewer bits to code under a code optimized for q than under one optimized for p. Burnham & Anderson ("Model Selection and Multimodal Inference", 2nd ed., p78) refer to the KL divergence KL(g||f) as "information lost when f is used to approximate g";
conversely it is also the information gained when revising a belief in the opposite direction.
I think it would be great if you could add an appendix section to the paper and explain the exact architecture details of all the experiment.
Also open sourcing the code would help me and other people to understand the paper better. I read the answer to AnonReviewer5, but I still find some parts confusing:
Could you explain more about how PixelCNN is used in the hierarchical posterior case: "At each level, the generator is a conditional PixelCNN over the latent features in the level below." Is it like the normalizing flow or inverse autoregressive flow idea? Have you included the log det-Jacobian term? What is the exact architecture? Isn't the diagonal Gaussian output layers limiting for pixelcnn as it can't capture multi-modal distributions like the way softmax does?
What do you exactly mean by "Gated PixelVAE without upsampling"? I suppose this must be using the bias as a function of conditioning vector. If it works better than "concatenated feature map" version of PixelVAE on MNIST, how does it work on other datasets? How does the two different ways of conditioning compare in general?
What do you mean by spatial resolution in "two-level PixelVAE with latent variables at 1×1 and 8×8 spatial resolutions"?
It would be great to also include two sets of MNIST samples maybe in an appendix section. One with PixelCNN and the other with PixelVAE with the same pixelcnn depth to illustrate the hidden code in PixelVAE actually captures the global structure.
Thanks for your comments.
> I think it would be great if you could add an appendix section to the paper and explain the exact architecture details of all the experiment.
Also open sourcing the code would help me and other people to understand the paper better.
We agree and are in the process of doing both of these.
> Could you explain more about how PixelCNN is used in the hierarchical posterior case: "At each level, the generator is a conditional PixelCNN over the latent features in the level below." Is it like the normalizing flow or inverse autoregressive flow idea? Have you included the log det-Jacobian term? What is the exact architecture? Isn't the diagonal Gaussian output layers limiting for pixelcnn as it can't capture multi-modal distributions like the way softmax does?
Our latent variables are split into multiple levels z_1, . , z_n. The joint posterior over all of these is a simple fully factorized Gaussian (e. g. conditioned on x, z_2 is independent of z_1), unlike normalizing flows (and IAF, which is a special case of NF) which are used to make the posterior distribution more flexible. Our contribution is complementary to NF/IAF, and using them together would likely further improve performance. Our hierarchy exists entirely in the generator (a. k. a. decoder, a. k. a. conditional likelihood function): if we have n levels of latent variables, we (jointly end-to-end) train n "decoders", each of which models a prior over the next level of latent variables downward conditioned on the current level (e. g. if our latent variables are z_1 and z_2, with z_2 at the top, one decoder models p(z_1 | z_2) and the other models p(x | z_1) ).
Each of these functions is a PixelVAE decoder (see figure 2). At sampling time, we first sample from p(z_2) [which in our experiments is just unit Gaussian], then sample from p(z_1 | z2) autoregressively, then sample from p(x | z_1). The log det-Jacobian term used in NF comes from the rule for change of variables, which our model doesn't use (we don't transform a simple posterior into a more flexible one, rather we directly output a flexible *prior*) hence that term isn't needed in our likelihood lower bound. As you mention, for all but the lowest layer we use a diagonal Gaussian output across channels instead of a softmax (we found that using a more flexible distribution here didn't help much).
> What do you exactly mean by "Gated PixelVAE without upsampling"? I suppose this must be using the bias as a function of conditioning vector. If it works better than "concatenated feature map" version of PixelVAE on MNIST, how does it work on other datasets? How does the two different ways of conditioning compare in general?
That's correct; we bias a Gated PixelCNN with a linear transform of the conditioning vector.
Though it achieves slightly better likelihood on MNIST (where models tend to overfit easily), we find this architecture doesn't scale as well to large datasets (where we are mainly compute-bound) because it requires a large number of PixelCNN layers. We also suspect it would perform less well with higher image resolutions;
for this reason we choose to use upsampling layers in our large-scale models, and demonstrate that they work well.
> What do you mean by spatial resolution in "two-level PixelVAE with latent variables at 1×1 and 8×8 spatial resolutions"?
That model has two levels of latent variables, z_1 and z_2. z_2 is a set of 1x1 feature maps with 512 channels (i.e. a tensor with shape n,512,1,1 in "NCHW" format).
Similarly z_1 is a set of 8x8 feature maps with 64 channels (i.e. a tensor with shape n,64,8,8 in "NCHW" format). We didn't specify the number of channels because we found our model wasn't very sensitive to that choice.
> It would be great to also include two sets of MNIST samples maybe in an appendix section. One with PixelCNN and the other with PixelVAE with the same pixelcnn depth to illustrate the hidden code in PixelVAE actually captures the global structure.
Good idea! We'll do that.
All in all this is a nice paper.
I think the model is quite clever, attempting to get the best of latent variable models and auto-regressive models. The implementation and specific architecture choices (as discussed in the pre-review) also seem reasonable.
On the experimental side, I would have liked to see something more than NLL measurements and samples - maybe show this is useful for other tasks such as classification?
Though I don't think this is a huge leap forward this is certainly a nice paper and I recoemmend acceptance.
The paper combines a hierarchical Variational Autoencoder with PixelCNNs to model the distribution of natural images.
They report good (although not state of the art) likelihoods on natural images and briefly start to explore what information is encoded by the latent representations in the hierarchical VAE.
I believe that combining the PixelCNN with a VAE, as was already suggested in the PixelCNN paper, is an important and interesting contribution.
The encoding of high-, mid- and low-level variations at the different latent stages is interesting but seems not terribly surprising, since the size of the image regions the latent variables model is also at the corresponding scale. Showing that the PixelCNN improves the latent representation of the VAE with regard to some interesting task would be a much stronger result.
Also, while the paper claims, that combining the PixelCNN with the VAE reduces the number of computationally expensive autoregressive layers, it remains unclear how much more efficient their whole model is than an PixelCNN with comparable likelihood.
In general, I find the clarity of the presentation wanting.
For example, I agree with reviewer1 that the exact structure of their model remains unclear from the paper and would be difficult to reproduce.
Thanks for your thoughtful comments. We've taken your suggestions into account and updated our paper accordingly:
- Re. our model's efficiency compared to PixelCNN with comparable likelihood, we've updated Table 2 to list each model's computational complexity alongside its likelihood (for those models whose papers contained enough details for us to calculate this).
In particular, we show that our model achieves comparable likelihood at a fraction of the computational cost of PixelRNN/PixelCNN. All current models underfit on this dataset and performance is mostly limited by computational cost (as observed by both us and PixelRNN); we think it's quite likely that further increasing model size could improve our likelihood.
- Re. details of our model's structure, we've added tables detailing specifics of our architecture in an appendix, and we've also made code to reproduce our experiments available online and referenced it within the paper.
- Re.
further investigation of our model's latent representation, we've added experiments to demonstrate our model's ability to learn more useful low-dimensional representations than VAE (section 4.1.3 and figure 5).
- Re. our visualization of sample variations at different levels, our purpose is to provide a qualitative demonstration of our model behaving the way we design it to perform -- we use fewer autoregressive layers (with a smaller receptive field) which should force the model to only focus on low-level details such as precise positioning and color, and leave the more semantic features to be captured by the latent layers. In a sense, this should not be "surprising" since the model has been designed to capture variations according to this hierarchy, but it does act as a qualitative check for alignment of model design with actual model behavior.
UPDATE: The authors addressed all my concerns in the new version of the paper, so I raised my score and now recommend acceptance.
This paper combines the recent progress in variational autoencoder and autoregressive density modeling in the proposed PixelVAE model. The paper shows that it can match the NLL performance of a PixelCNN with a PixelVAE that has a much shallower PixelCNN decoder.
I think the idea of capturing the global structure with a VAE and modeling the local structure with a PixelCNN decoder makes a lot of sense and can prevent the blurry reconstruction/samples of VAE. I specially like the hierarchical image generation experiments.
I have the following suggestions/concerns about the paper:
1) Is there any experiment showing that using the PixelCNN as the decoder of VAE will result in better disentangling of high-level factors of variations in the hidden code?
For example, the authors can train a PixelVAE and VAE on MNIST with 2D hidden code and visualize the 2D hidden code for test images and color code each hidden code based on the digit and show that the digits have a better separation in the PixelVAE representation. A semi-supervised classification comparison between VAE and the PixelVAE will also significantly improve the quality of the paper.
2) A similar idea is also presented in a concurrent ICLR submission "Variational Lossy Autoencoder". It would be interesting to have a discussion included in the paper and compare these works.
3) The answer to the pre-review questions made the architecture details of the paper much more clear, but I still ask the authors to include the exact architecture details of all the experiments in the paper and/or open source the code. The clarity of the presentation is not satisfying and the experiments are difficult to reproduce.
4) As pointed out in my pre-review question, it would be great to include two sets of MNIST samples maybe in an appendix section. One with PixelCNN and the other with PixelVAE with the same pixelcnn depth to illustrate the hidden code in PixelVAE actually captures the global structure.
I will gladly raise the score if the authors address my concerns.
Thanks for your thoughtful comments; we agree that all of these are good suggestions. We've updated our paper to address them as follows:
1) We've run this experiment and included the resulting plots in the paper (figure 5).
2) We've added a discussion to our 'Related Work' section and also included VLAE in our MNIST results table for comparison.
3) We've added tables of our model architecture, along with more detailed descriptions, in an appendix; we've also made our code open-source (https://github.com/igul222/PixelVAE) and referenced it in the paper.
4) We've added plots of MNIST samples from PixelVAE and an equal-depth PixelCNN in Appendix B. We've also included samples from a PixelCNN with greater depth, but roughly equal computational complexity.

Meta-review:
role: PC
data: comment: The paper provides a solution that combines best of latent variable models and auto-regressive models. The concept is executed well and will make a positive contribution to the conference.

Decision: Accept (Poster)