Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a way to improve the performance of a trained VAE by modifying the learned prior with a reweighting factor. The reweighting factor is computed using a network trained to discriminate between samples from the learned prior and samples from the encoder given real data. To sample from the VAE, the latent variables are sampled from the reweighted prior using MC inference (e.g. sampling-importance-resampling) and fed through the decoder. This leads to better samples (versus not using reweighting) but also greater computational cost due to the MC inference.
main_review: # Novelty
The authors situate their work well in relation to prior work. Other (cited) papers use reweighted priors, so the novelty comes mainly from using noise contrastive estimation to train the reweighting factor (via learning to discriminate between samples from the prior and from the encoder). This is a significant improvement as it makes training simpler and scalable to larger VAE architectures.

# Strengths
- Clearly written and a pleasure to read.
- Thorough experimental comparisons and ablation studies.
- Applicable to modern hierarchical VAE architectures, unlike much prior work.

# Concerns
- Sampling from the VAE with the suggested parameters (e.g. using 5000 proposal samples) is very slow, taking 10s to create a single output (based on Table 8). This might limit the practical applicability of this approach. It would also be interesting to see these times compared to those for the VAEBM baseline, which also involves MC inference.
- The experimental results are not overly convincing, with the reported FIDs/NLLs often being similar to, or worse than, the baselines. However, given that the authors address this (e.g. pointing out that VAEBM baseline is more complex and not applicable to discrete data), I think the results are good enough for acceptance.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In trained VAEs, mismatch between the prior and the variational posterior can lead to low-quality generated samples.

To address this issue, the authors suggest the following procedure:
1) given a trained VAE, train a classifier to distinguish images generated from the variational posterior vs images generated from the prior;
2) adjust the prior to minimize the accuracy of the trained classifier from step 1. This is done by reweighting the sampling procedure using the predictions from the classifier.

Main contribution:
a practical method to improve the generative quality of several forms of VAE by reweighing the prior to diminish the mismatch from the variational posterior
main_review: Originality:
Overall, the combination of ideas/methods to improve image generation in VAEs is original.

Quality:
The submission is technically sound.

Clarity:
Generally well written. I have two minor comments:
- I think the manuscript would read better if the authors first discuss the main steps of the procedure (train classifier given a trained VAE, then inference with sampling reweighting) as done in Section 3.2 and then describe the specifics of contrastive estimation in Section 3.1;
- the authors state that prior hole problem is when "the prior distribution fails to match the aggregate approximate posterior". I would say that the contrary happens: after training, the aggregate approximate posterior may not match the prior sufficiently well (this may due to limited expressivity of the posterior, trainability, low sample size, etc);

Significance:
This is my only major concern. While I see the practicality of this procedure, the technical contribution is narrow and I wonder why one would prefer this procedure over applying state-of-the-art VAE methods such as VDVAE and VQVAE. Could the authors provide some guidance on the use cases where their method should be preferred?

After revision:
After reading authors' reply and other reviewers' comments, I decided to change my score to a 6 (+2 difference from previous score). I think my original assessment on significance was incomplete. As Reviewer C3kP notices (i) "the idea is very simple and easy to implement, and "The method can be plugged with many different VAE variants". After more consideration, these pros address my original concern on the significance of this work. The authors may consider making this more explicit in their manuscript, should it be accepted.
limitations_and_societal_impact: NA
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes to learn an energy based model using noise contrastive estimation on the latent of a VAE to mitigate the prior hole problem. To do so, the paper proposes a very simple method that consists in first training a standard VAE, then, given the frozen VAE, training a discriminator in latent space to discriminate between samples from the prior, and samples from encodings of real images. The discriminator can then be used to reweigh the prior, and obtain a better, but non algebraically computable, prior distribution. The implicit distribution can still be sampled using importance resampling or langevin dynamics. The paper proceeds to show that this technique improves the performance of the model in term of FID on various datasets. 
main_review: Pros:
- The idea is very simple and easy to implement.
- The method can be plugged with many different VAE variants.
- The method seems to improve sample quality as measured by FID across the board.
- The paper provides quite thorough empirical validations, on many different datasets, with many different architectures.
- The ablations, while mainly confirming intuitions, are informative.
- The paper is well written, and most claims are well substantiated.

Cons:
- The method cannot reliably be used to evaluate a bound on the log-likelihood.
- The only performance metric used is the FID, which arguably favors quality of samples over generalization (as mentionned in (Razavi et al. 2019). As a consequence, I am not sure that the claim that 'diversity' of samples is improved (l.319) is well substantiated. Evaluating performance using a more diverse set of metrics, and notably the metrics used in (Razavi et al. 2019) could help in substantiating this claim.

Other comments:
- I don't understand the statement l.240. Fig 1. from Kingma et al. 2016 seems to indicate that IAF exactly attend to the prior holes problem.
- l294. Could you also give the number of proposal samples, so that we can compare it to the effective sample size?

Overall, I think the pros outweigh the cons. This is a simple and interesting contribution. I recommend acceptance.

References:
_Generating Diverse High-Fidelity Images with VQ-VAE-2_ Ali Razavi, Aäron van den Oord, Oriol Vinyals
limitations_and_societal_impact: The authors provide a statement assessing the potential societal impact of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Update after Response:

The response of the authors resolved some of my concerns and I am therefore improving my score by 1 point, provided that the authors add the additional information mentioned in their reply. I think the work is relevant and presents a potential solution to solve some of the problems that plague autoencoder sample generation. 

Old:
The authors propose a novel method of adapting the prior p(z) of VAE to the experimentally observed approximate posterior q_{\theta}(z) over the training samples. The modification of the prior is performed as a separate step, after model training. This improves the visual quality of generated samples drawn from the modified prior. The article includes an extensive evaluation and the appendix includes further results with important studies.
main_review: Strengths:
- very clearly written method
- simple, straightforward method
- detailed evaluation on multiple datasets and with multiple baselines
- High visual quality of qualitative samples shown.
- topic highly relevant to the neurips community

Weaknesses:
- implementation not accessible
- On initial reading, I feared that by adapting the prior to the aggregate approximate posterior of the training dataset, we are teaching the model memorization of the training set. The nearest-neighbor evaluation given in Appendix F has convinced me of the opposite. Sad, that I had to reach to the appendix for this very relevant experimental result.
- evaluation by FID: What has been used as the comparison data distribution? If the training set is used, I'm not surprised that this method achieves high FID scores, as the NCE training step is focused on aligning the prior to the approximate posterior of the training dataset. How do the FID metric on the validation/test set compare?
- In the experiments, it is unclear how the log partition function is computed. It is well known that for many estimators, a small standard deviation is not a reliable estimate for correctness of the mean since the distribution of estimates can be highly skewed.
- I feel the use of the t-value during image generation, e.g. Figure 3, undermines the message of the paper. If it is beneficial to accurately represent the marginal posterior, why would we then sample from a different distribution than the marginal posterior?  Also, it is unclear how the temperature was selected.  

Correctness:
The experimental evaluation is mostly appropriate to determine the correctness of the work. However, details are unclear.


Clarity: Is the paper well written?
- The introduction and method sections are very clearly written
- The Experimental section could explain the total model size, used compute, and evaluation metrics used in more detail/higher clarity.
- Consider moving required background knowledge for the method from the related work section (after the method) to the background section (before the method).


Additional feedback:

The authors phrase the problem as the inability of the prior to model the posterior correctly. But wouldn't it rather be the inability of the posterior to correctly model the prior or our inability to train a posterior that accurately models the prior? It is well known that GANs provide very good results when sampling from a standard normal prior, thus it should be possible to train a VAE that uses the same prior. I think the authors should provide more evidence to the claim that the prior is at fault here and not the posterior/ELBO or rephrase the relevant parts of the paper. 

The checklist includes several answers that are debatable
- 1b) I do not see section 4 discussing limitations of the presented work but only limitations of the presented work. I would expect those to be discussed in the discussion/conclusion of experiments and results.
- 3a) asks for code to which the authors reply yes, without making the code accessible. A description of the method is not equivalent to making the code public, e.g. to check for correctness or reproduction of the experiments. 
- 3b) I think number of training iterations/stopping criteria are missing.
- 3d) asks for total estimated compute time used for the experiments. The paper only includes partial measurements and it is unclear how costly a full reproduction of the results would be.


Typos:
line 196, prior art -> prior work
line 199, In most our experiments -> In most of our experiments


limitations_and_societal_impact: The authors do not discuss limitations of their work. An important limitation of their work is that their proposed SIR sampling can fail. By proposing samples using the prior it can be difficult to sample regions which have high probability under the marginal posterior but low probability under the prior. While the authors explore Langevin sampling as alternative, the study does not compare how well the samples drawn by the samples reflect the marginal posterior.

While the impact contains an impact statement, it is only of limited value since it is mostly used to praise the work presented. The societal impact of generating images of high fidelity could include for example the dangers of creating deep fakes. I do appreciate however the reference to bias in image generation.
review_title: null

Meta-review: The paper proposes a new class of priors for Variational Auto-Encoders (VAEs). The main idea is to use Noise Contrastive Estimation and a two-stage training method. The reviewers highlighted that:
- The paper is clearly written.
- The idea is simple (in the positive sense!) and easy to implement.
- The proposed prior could be plugged in any VAE.

However, the reviewers raised some concerns:
- Sampling from the VAE with the suggested parameters is very slow.
- The experimental results are not overly convincing.
- The organization of the paper could be improved.
- The method cannot reliably be used to evaluate a bound on the log-likelihood.
- The only performance metric used is the FID, which arguably favors the quality of samples overgeneralization.

Moreover, I find it rather surprising that the authors do not compare their approach to at least one of the following two methods:
- Tomczak, J., & Welling, M. (2018). VAE with a VampPrior. In International Conference on Artificial Intelligence and Statistics (pp. 1214-1223). PMLR.
- Norouzi, S., Fleet, D. J., & Norouzi, M. (2020). Exemplar VAE: Linking Generative Models, Nearest Neighbor Retrieval, and Data Augmentation. arXiv preprint arXiv:2004.04795.

It seems rather natural to focus on a thorough comparison against other priors for VAE. Instead, the authors have a limited comparison in Table 2. Moreover, the authors decided to compare their method to various deep generative models. I totally agree that it is also important, however, without a proper comparison against various priors, it is hard to properly evaluate their idea.

The authors provided a thorough rebuttal and they promised to improve the paper. Overall, the paper is solid and the idea is neat, therefore, I tend to accept the paper.

Decision: Accept (Poster)