Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

The authors highlight an important problem in VAE - the prior-hole problem - which is that the approximate posterior and the simple gaussian prior do not match in spite of the KL term in the ELBO which makes sampling an issue - leading to the prior putting probability mass on latents that are not decoded to high probability mass regions in data manifold. Prior approaches have overcome this problem by increasing the expressivity of the prior through autoregressive models, and/or using hierarchical latents, EBMs with MCMC sampling. This paper proposes a very simple two stage method - (1) train a regular VAE, (2) train a binary classifier in NCE style to distinguish samples from prior and approx. posterior; use the re-weighting term from the NCE score to sample from a better re-weighted prior - either through langevin dynamics or re-sampling. The authors combine this approach with the use of hierarchical latents and produce really good performing generative models on a host of benchmarks with good looking samples.

Pros:

very simple and neat approach that produces really good results.
looks easy enough to reimplement and adopt widely for future research in VAEs.
samples look great, FID scores are good.
ablations on LD and SIR are very useful.
Cons:

would have been nice not to have a 2-stage approach. figuring out a way to have an online way of trying to train the re-weighting classifier in a GAN-like manner would be much more preferred.
paper can do a better job at citing related work - ex - should cite Variational Lossy Autoencoder - Chen et al 2016 - on using autoregressive priors and pointing out a connection to IAF posterior.
would be nice to show results even w/o hierarchical latents - how much improvement can be obtained over vanilla deep VAE with this two-stage approach
report results on log-likelihood benchmarks - CIFAR10, ImageNet-32, ImageNet-64 - compare to Flow models, Autoregressive Models, etc.

Post Rebuttal:
I believe the positioning of the paper could be improved but at the same time empirical results in the paper are strong. I am adjusting my score to 6 with a confidence of 4.
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: The main concerns are,
* The idea is somewhat novel but very similar to ideas developed in  [1,2] and (Bauer & Mnih, 2019). Note that, in [1] rejection sampling is performed on samples from the prior distribution based on the likelihood ratio obtained from a discriminator. While the results reported in [1] are not competitive to those reported in this work, it is probably be due to differences in model architecture. Therefore, a fair comparison to prior works [1,2] and (Bauer & Mnih, 2019) is required using a common model architecture. E.g. experiments using a simple DCGAN style architecture on CIFAR-10  would be sufficient to indicate efficacy.

* Prior resampling vs more complex priors: While the proposed NCP sampling scheme is applied on a Gaussian base prior with the NVAE architecture, it is unclear whether a similar gain in performance can be obtained using a more complex prior e.g. flow [3] or autoregressive (Van Den Oord et al., 2017; Razavi et al., 2019), [4]. A simple experiment which could help here is: keeping the architecture constant compare NCP with Gaussian base vs a flow or autoregressive prior.

* Additionally, no comparison to state of the art VAE based image synthesis approaches like VQ-VAE-2 (Razavi et al., 2019) are provided e.g. on ImageNet 256 × 256.  

* No comparison to VAE-GAN based approaches [5,6], which achieves FIDs of  23.4 (vs 24.08 of this paper). It it also unclear whether the proposed approach can be applied on top of VAE-GAN based approaches.

* The current work uses the proposed NCP sampling scheme on a Gaussian base prior. Can the proposed prior be applied to more complex priors e.g. flow or autoregressive priors?

* No qualitative comparison of sample quality: Figure 3 only shows samples for the NCP approach. The paper should qualitative demonstrate that NCP leads to reduction in the number of "corrupted" images. To do this one possibility would be to demonstrate that the worst sample image in a batch from the NCP approach is on average significantly better than that from the plain NVAE approach. Alternatively, an even simpler experiment can be performed on the 2d Ring and Grid [7] datasets to demonstrate gains in sample quality.

[1] Dual Rejection Sampling for Wasserstein Auto-Encoders, ECAI 2020.

[2] Variational Rejection Sampling, AISTATS 2018.

[3] Latent Normalizing Flows for Many-to-Many Cross-Domain Mappings, ICLR 2020.

[4] Normalizing Flows with Multi-scale Autoregressive Priors, CVPR 2020.

[5] Variational Approaches for Auto-Encoding Generative Adversarial Networks, arXiv 2017.

[6] "Best-of-Many-Samples" Distribution Matching, NeurIPS Workshop 2019.

[7] GDPP: Learning Diverse Generations using Determinantal Point Process, ICML 2019.
review_title: Novelity is unclear and additional experimental evidence needed

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: **GENERAL**
The goal of the paper is to model the marginal over latents in VAEs in such a way to minimize the mismatch with the aggregated posterior. The paper proposes a new class of marginal distributions over the latent space that is a product of two experts: the first expert is a non-trainable probability distribution, and the second expert is an unnormalized probability distribution parameterized using neural networks. Since training a product of experts requires to apply an approximate inference (e.g., MCMC sampling), the authors propose to use the likelihood ratio trick. Eventually, a VAE is trained in two stages. First, they assume the marginal over z's to be simply the non-trainable distribution, and the VAE is trained. At the second stage, they propose to train the second expert (i.e., the binary classifier that distinguishes z ~ q(z) and z ~ p(z)) in order to obtain the final NCP that better matches the aggregated posterior. Further, the idea is extended to hierarchical VAEs, and a separate binary classifier is trained per each stochastic level.

**Strengths:**
S1: The idea is very interesting and allows to enrich the marginal distribution over z's in the VAE framework.

S2: The paper is well positioned in current trends in generative modeling. I find the combination of energy-based models and VAEs as a very appealing research direction.

S3: The proposed two-stage learning procedure is very logical and seem to be efficient. Its simplicity increases its reproducibility.

S4: Obviously, introducing an energy-based component results in intractability of the likelihood function due to the partition function. However, the FID scores and the quality of generated images are very convincing.

**Deficiencies:**
D1: In this review, I kept using "a non-trainable expert" or "a non-trainable marginal" for the base prior (the term used by the authors). However, I am not quite sure whether the base prior is non-trainable. I was unable to find any information about it in the paper. Therefore, I assumed it is a standard Gaussian after inspecting Figure 1. It would be easier for a reader, if this information was included in the text.


**Remarks:**
R1: The Appendix B.1 is closely related to the following papers:
- Rezende, D. J., & Viola, F. (2018). Taming vaes. arXiv preprint arXiv:1810.00597.
- Matthew D Hoffman and Matthew J Johnson. Elbo surgery: yet another way to carve up the variational evidence lower bound. In Workshop in Advances in Approximate Bayesian Inference, NIPS, 2016.
- Jakub Tomczak and Max Welling. Vae with a vampprior. In International Conference on Artificial Intelli- gence and Statistics, pp. 1214–1223, 2018.
It is maybe worth to mention these papers there. Otherwise, the text may sound as a new contribution.

R2: Section 3, first paragraph: The authors stated that: "Recently, energy-based models have shown promising results in representing complex distributions". This statement is very misleading, because the energy-based models (e.g., Boltzmann machines) have been used in ML for over 30 years, e.g.:
- Ackley, D. H., Hinton, G. E., & Sejnowski, T. J. (1985). A learning algorithm for Boltzmann machines. Cognitive science, 9(1), 147-169.
- Hinton, G. E., & Sejnowski, T. J. (1986). Learning and relearning in Boltzmann machines. Parallel distributed processing: Explorations in the microstructure of cognition, 1(282-317), 2.
- Salakhutdinov, R., & Hinton, G. (2009, April). Deep boltzmann machines. In Artificial intelligence and statistics (pp. 448-455).
- Larochelle, H., & Bengio, Y. (2008, July). Classification using discriminative restricted Boltzmann machines. In Proceedings of the 25th international conference on Machine learning (pp. 536-543).

The word "recently" suggests that this is a new invention that is simply not true.

R3: The proposed prior (NCP) could be seen as a specific form of a product of experts (e.g., Hinton, G. E. (2002). Training products of experts by minimizing contrastive divergence. Neural computation, 14(8), 177). In my opinion, it is an interesting connection.

R4: I wonder whether it is feasible to use some sort of importance sampling (e.g., Annealed Importance Sampling, Salakhutdinov, R., & Murray, I. (2008, July). On the quantitative analysis of deep belief networks. In Proceedings of the 25th international conference on Machine learning (pp. 872-879).) or other procedure (e.g., Perturb-and-MAP, Hazan, T., & Jaakkola, T. (2012, June). On the partition function and random maximum a-posteriori perturbations. In Proceedings of the 29th International Conference on International Conference on Machine Learning (pp. 1667-1674).) to estimate the partition function.

**AFTER REBUTTAL**
I would like to thank the authors for their rebuttal and all updates. The paper is related to other ideas in the literature, however, it constitutes an interesting contribution to the field. I appreciate all new results and discussions. I keep my original score.
review_title: A new energy-based marginal distribution over latents for VAEs

final_score: rating: 5: Marginally below acceptance threshold
review_content: Authors approach the "hole problem" of variational autoencoders where the aggregate posterior fails to match the prior, causing some areas of the prior distribution to be left out. Consequently, the decoder is not trained properly to operate in such regions, and the whole generate models is then subject to suboptimal performance. To attack this problem authors introduce two changes:
- Once a standard VAE is trained, a post-hoc learning of the prior distribution is performed. This is expected to move the prior closer to the aggregate posterior.
- In order to increase prior's expressivity, authors employ Energy Based Models (EBMs). To sidestep the difficulties of the likelihood-based training of EBMs authors switch to Noise Contrastive Estimation (NCE) procedure. Two techniques are proposed to perform approximate sampling from the EBM.

**Strong points**:
The paper is clearly written, technically correct and presents a new state-of-the-art method that significantly improves sample quality of Variational Autoencoders.

**Weak points**:
My major problem with the paper is the seeming disparity between the models in the experimental comparison. In particular, authors build NCP-VAE upon NVAE [1], a state-of-the-art VAE architecture notorious for its large size and use of many stochastic layers (groups of latent variables in authors' terms). At the same time, for both multistage VAE baselines (*2 stage VAE* and *RAE*) the FID metrics were taken from the corresponding papers, which were based on much simpler architectures with a single group of latent variables.

A more fair comparison would be to take the NVAE architecture and apply techniques from *RAE* [2] / *2 stage VAE* [3] to it. In the end of the sec. 4 authors say "It is not clear how 2s-VAE or RAE are applied to state-of-the-art hierarchical models". I believe this is a doable task. For example, consider (perhaps appropriately downscaled) NVAE's encoder (fig. 2a from [1]) and replace the input x with a random noise ε – the resulting latent variable model could be used as a second stage VAE [3]. For the *RAE* remove the red bottom-up path completely and make distribution over z1 a mixture. It might also be interesting to consider a mixture of size 1 -- a single gaussian distribution. In this case the 2nd stage training would be moving the prior towards the aggregate posterior. While in theory one might expect p(z) already be a decent approximation for q(z) (as shown in appendix B), in practice suboptimality of stochastic joint optimization might prevent p(z) from matching the aggregated q(z) to the fullest. The NVAE shares some weights between the prior and the encoder, which could either alleviate this problem by ensuring both encoder and prior change in a similar fashion, or hurt prior's expressivity since the top-down path has to be able to work in two somewhat different scenarios.

Finally, the qualitative evaluation seems disconnected from the problem being solved. It is my understanding that lessening the prior hole problem should result in a fewer unappealing / blurry samples generated by the model. The additional samples in appendix H still contain some such samples. Unfortunately, no such samples are presented for the baseline models (in [1] the samples were curated). It's therefore hard to tell if the frequency of such samples was reduced.

**Conclusion**: the results presented in the paper are impressive and the quality of samples is indeed very high. Unfortunately, I have to vote for rejection of this paper in its current form for its lack of rigorous experimental evaluation. It feels like the authors did overly focus on presenting a new method and have not given the prior work their best effort.

My suggestions to authors:
- Evaluate two stage approach to the NVAE: 1) nested VAE prior as in [3], 2) mixture over z1 as in [2], 3) fine-tune NVAE's prior. The later will also disentangle the benefits of an energy-based prior from those of the two-stage training.
- Perform qualitative evaluation based on frequency of visually unappealing samples from NCP-VAE versus that of the NVAE.

[1]: Arash Vahdat and Jan Kautz.  NVAE: A deep hierarchical variational autoencoder. 

[2]: Partha Ghosh, Mehdi S. M. Sajjadi, Antonio Vergari, Michael Black, and Bernhard Scholkopf.  From variational to deterministic autoencoders.

[3]: Bin Dai and David Wipf.  Diagnosing and enhancing vae models.



## Post-rebuttal update

Authors have addresses some of the issues outlined above, in particular the additional comparison in sec. 5.4 and table 7 is informative. However, the RAE numbers indicate that a simple 10-component Gaussian Mixture is superior to the complex model of NCP-VAE with a Gaussian prior on a small VAE, and the superiority of NCP with GMM prior base provides little information as RAE with 50-components GMM could have performed even better. It's also not clear how 2s-VAE, RAE and WAE would scale to bigger architectures such as NVAE, which raises the question if the NCP was the optimal one.

It's surprising to see the 2-stage VAE to perform worse than the standard VAE, whereas (Dai & Wipf, 2018) have shown a significant improvement in the original paper. I think, this result needs further elaboration and validation.

Regarding the qualitative evaluation: figure 12 does show that the NCP-NVAE has fever poor samples, although it's hard to tell whether the difference is significant. Another issue is that the comparison does not include prior works.

As a result I bump by score to 5, but I think the paper still needs more work to make a sound argument for the proposed idea.

review_title: Issues with experimental setup

Meta-review: This paper is rejected.

All of the reviewers found the empirical results strong. However, R3 and R4 pointed out concerns with the positioning of the work relative to prior work and that their approach is conceptually similar to previous work. The authors have tried to address these concerns in their rebuttal. Both reviewers appreciate the changes, but still have remaining concerns that I agree with. Based on these concerns, it is unclear if the strong empirical results are mostly due to using the NVAE architecture, rather than a methodological improvement over previous methods. The authors should work on positioning their paper in the context of prior work and the comparisons requested by R3 and R4 for a resubmission.

Decision: Reject