Venue: ICLR 2017 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: UPDATE: The authors addressed all my concerns in the new version of the paper, so I raised my score and now recommend acceptance.
--------------
This paper combines the recent progress in variational autoencoder and autoregressive density modeling in the proposed PixelVAE model. The paper shows that it can match the NLL performance of a PixelCNN with a PixelVAE that has a much shallower PixelCNN decoder.
I think the idea of capturing the global structure with a VAE and modeling the local structure with a PixelCNN decoder makes a lot of sense and can prevent the blurry reconstruction/samples of VAE. I specially like the hierarchical image generation experiments.

I have the following suggestions/concerns about the paper:

1) Is there any experiment showing that using the PixelCNN as the decoder of VAE will result in better disentangling of high-level factors of variations in the hidden code? For example, the authors can train a PixelVAE and VAE on MNIST with 2D hidden code and visualize the 2D hidden code for test images and color code each hidden code based on the digit and show that the digits have a better separation in the PixelVAE representation. A semi-supervised classification comparison between VAE and the PixelVAE will also significantly improve the quality of the paper.

2) A similar idea is also presented in a concurrent ICLR submission "Variational Lossy Autoencoder". It would be interesting to have a discussion included in the paper and compare these works.

3) The answer to the pre-review questions made the architecture details of the paper much more clear, but I still ask the authors to include the exact architecture details of all the experiments in the paper and/or open source the code. The clarity of the presentation is not satisfying and the experiments are difficult to reproduce.

4) As pointed out in my pre-review question, it would be great to include two sets of MNIST samples maybe in an appendix section. One with PixelCNN and the other with PixelVAE with the same pixelcnn depth to illustrate the hidden code in PixelVAE actually captures the global structure.

I will gladly raise the score if the authors address my concerns.
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: All in all this is a nice paper.

I think the model is quite clever, attempting to get the best of latent variable models and auto-regressive models. The implementation and specific architecture choices (as discussed in the pre-review) also seem reasonable.
On the experimental side, I would have liked to see something more than NLL measurements and samples - maybe show this is useful for other tasks such as classification?

Though I don't think this is a huge leap forward this is certainly a nice paper and I recoemmend acceptance.
review_title: Nice paper

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper combines a hierarchical Variational Autoencoder with PixelCNNs to model the distribution of natural images. 
They report good (although not state of the art) likelihoods on natural images and briefly start to explore what information is encoded by the latent representations in the hierarchical VAE.

I believe that combining the PixelCNN with a VAE, as was already suggested in the PixelCNN paper, is an important and interesting contribution. 
The encoding of high-, mid- and low-level variations at the different latent stages is interesting but seems not terribly surprising, since the size of the image regions the latent variables model is also at the corresponding scale. Showing that the PixelCNN improves the latent representation of the VAE with regard to some interesting task would be a much stronger result. 
Also, while the paper claims, that combining the PixelCNN with the VAE reduces the number of computationally expensive autoregressive layers, it remains unclear how much more efficient their whole model is than an PixelCNN with comparable likelihood.

In general, I find the clarity of the presentation wanting. For example, I agree with reviewer1 that the exact structure of their model remains unclear from the paper and would be difficult to reproduce. 

review_title: null

final_score: rating: null
review_content: - What exactly are the architectures for the VAE and PixelCNN layers used in the LSUN and ImageNet experiments (in terms of number of layers and units per layer)?

- Do you have any guarantees for the robustness of the estimate of the NLL on the MNIST dataset? 

- Could you please further explain in what sense “The KL divergence term can be interpreted as a measure of the information content in the posterior distribution q(z|x)” 


review_title: Model architectures and questions

final_score: rating: null
review_content: I think it would be great if you could add an appendix section to the paper and explain the exact architecture details of all the experiment. Also open sourcing the code would help me and other people to understand the paper better. I read the answer to AnonReviewer5, but I still find some parts confusing:

Could you explain more about how PixelCNN is used in the hierarchical posterior case: "At each level, the generator is a conditional PixelCNN over the latent features in the level below." Is it like the normalizing flow or inverse autoregressive flow idea? Have you included the log det-Jacobian term? What is the exact architecture? Isn't the diagonal Gaussian output layers limiting for pixelcnn as it can't capture multi-modal distributions like the way softmax does?

What do you exactly mean by "Gated PixelVAE without upsampling"? I suppose this must be using the bias as a function of conditioning vector. If it works better than "concatenated feature map" version of PixelVAE on MNIST, how does it work on other datasets? How does the two different ways of conditioning compare in general?

What do you mean by spatial resolution in "two-level PixelVAE with latent variables at 1×1 and 8×8 spatial resolutions"?

It would be great to also include two sets of MNIST samples maybe in an appendix section. One with PixelCNN and the other with PixelVAE with the same pixelcnn depth to illustrate the hidden code in PixelVAE actually captures the global structure.
review_title: some questions

Meta-review: The paper provides a solution that combines best of latent variable models and auto-regressive models. The concept is executed well and will make a positive contribution to the conference.
Decision: Accept (Poster)