Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The proposed approach is straight forward, experimental results are good, but don’t really push the state of the art. But the empirical analysis (e.g. decomposition of different cost terms) is detailed and very interesting.   
review_title: Good paper

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:

This paper attempts to solve the problem of meaningfully combining variational autoencoders (VAEs) and PixelCNNs. It proposes to do this by simultaneously optimizing a VAE with PixelCNN++ decoder, and a VAE with factorial decoder. The model is evaluated in terms of log-likelihood (with no improvement over a PixelCNN++) and the visual appearance of samples and reconstructions.

Review:

Combining density networks (like VAEs) and autoregressive models is an unsolved problem and potentially very useful. To me, the most interesting bit of information in this paper was the realization that you can weight the reconstruction and KL terms of a VAE and interpret it as variational inference in a generative model with multiple copies of pixels (below Equation 7). Unfortunately the authors were unable to make any good use of this insight, and I will explain below why I don’t see any evidence of an improved generative model in this paper.

As the paper is written now, it is not clear what the goal of the authors is. Is it density estimation? Then the addition of the VAE had no measurable effect on the PixelCNN++’s performance, i.e., it seems like a bad idea due to the added complexity and loss of tractability. Is it representation learning? Then the paper is missing experiments to support the idea that the learned representations are in any way an improvement. Is it image synthesis (not a real application by itself), then the paper should have demonstrated the usefulness of the model on a real task and probably involve human subjects in a quantitative evaluation.

Much of the authors’ analysis is based on a qualitative evaluation of samples. However, samples can be very misleading. A lookup table storing the training data generates samples containing objects and perfect details, but obviously has not learned anything about either objects or the low-level statistics of natural images. 

In contrast to the authors, I fail to see a meaningful difference between the groups of samples in Figure 1.

The VAE samples in Figure 3b) look quite smooth. Was independent Gaussian noise added to the VAE samples or are those (as is sometimes done) sampled means? If the former, what was sigma and how was it chosen?

On page 7, the authors conclude that “the pixelCNN clearly takes into account the output of the VAE decoder” based on the samples. Being a mixture model, a PixelCNN++ could easily represent the following mixture:

p(x | z) = 0.01 \prod_i p(x_i | x_{<i}) + 0.99 \prod_i p(x_i | z)

The first term is just like a regular PixelCNN++, ignoring the latent variables. The second term is just like a variational autoencoder with factorial decoder. The samples in this case would be dominated by the VAE, which depends on the latent state. The log-likelihood would be dominated by the first term and would be minimally effected (see Theis et al., 2016). Note that I am not saying that this is exactly what the model has learned. I am merely providing a possible counter example to the notion that the PixelCNN++ has learned to use of the latent representation in a meaningful way.

What happens if the KL term is simply downweighted but the factorial decoder is not included? This seems like it would be a useful control to include.

The paper is well written and clear.
review_title: Important problem but no evidence of progress

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors present Auxiliary Guided Autoregressive Variational autoEncoders (AGAVE), a hybrid approach that combines the strengths of variational autoencoders (global statistics) and autorregressive models (local statistics) for improved image modeling. This is done by controlling the capacity of the autorregressive component within an auxiliary loss function.

The proposed approach is a straightforward combination of VAE and PixelCNN that although empirically better than PixelCNN, and presumably VAE, does not outperform PixelCNN++. Provided that the authors use PixelCNN++ in their approach, quantitively speaking, it is difficult to defend the value of adding a VAE component to the model. The authors do not describe how \lambda was selected, which is critical for performance, provided the results in Figure 4. That being said, the contribution from the VAE is likely to be negligible given the performance of PixelCNN++ alone.

- The KL divergence in (3) does more than simply preventing the approximation q() from becoming a point mass distribution.
review_title: AUXILIARY GUIDED AUTOREGRESSIVE VARIATIONAL AUTOENCODERS

Meta-review: To ensure that a VAE with a powerful autoregressive decoder does not ignore its latent variables, the authors propose adding an extra term to the ELBO, corresponding to a reconstruction with an auxiliary non-autoregressive decoder. This does indeed produce models that use latent variables and (with some tuning of the weight on the KL term) perform as well as the underlying autoregressive model alone. However, as the reviewers pointed out, the paper does not demonstrate the value of the resulting models. If the goal is learning meaningful latent representations, then the quality of the representations should be evaluated empirically. Currently it is not clear whether that the proposed approach would yield better representations than a VAE with a non-autoregressive decoder or a VAE with an autoregressive decoder trained using the "free bits" trick of Kingma et al. (2016). This is certainly an interesting idea, but without a proper evaluation it is impossible to judge its value.
Decision: Reject