Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper tackles the important problem of finding suitable learnable prior for VAE. The paper provides some improvements of Exemplar VAE that increase the efficiency of the model. The proposed model is simply aiming at reducing the examples for nearest neighbours used by Exemplar VAE by application of bayesian pseudocoresets  for efficiently approximating the entire original dataset. The proposed approach was evaluated using standard benchmark datasets in a number of tasks, including density estimation, representation learning and generative data augmentation.
main_review: The paper is well-written and easy to follow. The main contribution of the paper is well-expressed and seems to be well-motivated. I like the application for data augmentation - however it requires more experimental analysis. 

The proposed approach seems to be a bit incremental compared to Exemplar VAE  and VAMPrior. Conceptually, the paper presents the alternative way (referring to VamPrior) of learning pseudo inputs for the Exemplar VAE model. It would be beneficial to make some deeper comparison (both theoretical nad empirical) between the nature of pseudo inputs from VAMPrior and ByPE-VAE and express the novelty. 

After reading the introduction and method description I would expect more convincing results of experiments. It is not clear, what type of criterion is provided in table 1. It seems that it is test NLL estimated using the Importance Sampling. Why pure ELBO value is not used for evolution? For CelebA there are no quantitative results reported in Table 1. but this dataset was used in qualitative analysis. 

For the remaining experiments the range of datasets is reduced to MNIST and related. Having the models trained on CIFAR10 and CelebA it would be beneficial to see complete results for augmentation and representation tasks, even some of them are not SOTA compared to reference approaches. Similar issue is observed with table 2. - why results for VAMP (assuming the same number of pseudo inputs) are not reported here. 

Because I serve as emergency reviewer I did not have to chance to deeply go through the math formulas but they seems to be correct. 
limitations_and_societal_impact: I did not notice the limitations of proposed approach - it should be expressed in more visible way. The negative aspects are not discussed by authors - also needs including.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a Bayesian pseudocoreset based exemplar VAE algorithm which utilizes pseudocoresets to improve the computational complexity and reduce over-fitting of Exemplar-VAE (a related work), by conditioning the prior on a subset of data. The paper proposes variational inference to find the optimal pseudocoreset, by minimizing the KL divergence between the posteriors defined on the pseudocoreset and the training data. Detailed experimental evaluation is provided against baselines and competitors and shows improved performance of the proposed approach in running time and quality of the learned latent space.

The main contributions of the paper are:
- Bayesian pseudocoresets proposed instead of using all the training data as exemplars in  Exemplar-VAE
- Simultaneous optimization of the optimal pseudocoreset and VAE training, with the optimal pseudocoreset selected through variational inference
- Detailed experimental evaluation against competitors showing improved running time and latent space quality.
main_review: The paper proposes Bayesian pseudocoresets to improve the computational complexity of Exemplar-VAE and to reduce overfitting.  The paper is well-written, and extensive experimental evaluation is provided. Variational inference to find optimal pseudocoresets, and the joint optimization along with VAE are strong technical contributions. However, the paper very closely follows the structure, formatting, and experimental evaluation of Exemplar-VAE (a related work). New experimental evaluation or insights on the pseudocoresets are not provided but a repetition of the same experiments conducted by Exemplar-VAE on the proposed approach. The novelty of the paper is thus somewhat limited and incremental. The pseudocode for the algorithm optimization is not explained in the text.

Detailed comments:

- The optimization process in Algorithm 2 is not explained in the paper. What is the significance of subsets S and M in Algorithm 2?
- Equations (9)-(14) are similar or same as the ones in Exemplar-VAE. The equations defined in Exemplar-VAE should be cited.  
- Statistical significance of results on KNN is not provided. The results on density estimation show marginal improvement for the proposed approach.
- The t-SNE plots should also compare the latent space of Exemplar-VAE. Currently only the proposed approach and a baseline VAE are compared.
- Experiments to show how well the learned pseudocoresets approximate the training data, would be a useful addition.
- A comparison of KNN/log-likelihood for different coreset sizes would be useful. How does the proposed approach perform if the coreset size is set equal to the exemplar size in Exemplar-VAE?




limitations_and_societal_impact: Limitations are addressed, no negative societal impact observed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes the use of pseudo-coresets to accelerate inference in Examplar VAE.
The Examplar VAE ELBO (Eq.4/5 in [ExamplarVAE]) is updated to use pseudo-coreset (Eq.8/9 in this paper). Other quantities of interest are updated accordingly and a two-stage approach is proposed to update two types of parameters (VAE and pseudo-coresets).

[ExamplarVAE] Norouzi et al (2020) "Exemplar VAE: Linking Generative Models, Nearest Neighbor Retrieval, and Data Augmentation", NeurIPS 2020.
main_review: The authors do not report the sensitivity for the parameter 'k' (pseudocoreset are updated every 'k' epochs).
It would be interesting to show a few different values to understand how important these updates are, for example how data-dependent. 
In KNN the number of neighboors is also 'K' which can be confusing.

The dynamics of the two-stage optimization approach are not examined. This would be important as it is a key novel component of the proposed method. Is convergence stable across experimental settings? does it requires any tuning?

Fig.5: are the differences in log-likelihood statistically significant?

Overall, the goal is clear and the methodology is rigorously detailed, including all equations of interests. 
My main concern is novelty: the method is a relatively straightforward extension of Examplar VAE using Pseudo-coresets - both of these techniques are pre-existent.
Additionally, it would have been good to demonstrate this approach on a larger/more challenging dataset.
limitations_and_societal_impact: -
review_title: null

Meta-review: The paper proposes a new way of learning a prior in the Variational Auto-Encoder framework. The paper is a follow-up on the VampPrior VAE and the Exemplar VAE. In a nutshell, the idea is to use a Bayesian pseudocoreset to learn pseudoinputs (in the context of the VampPrior). Overall, the reviewers appreciate the paper because:
- It is easy to follow.
- All concepts are clearly explained.
- The idea of using the Byesian pseudocoreset is interesting.

The main concern is about the experimental section. The reviewers would prefer to see a deepr comparison of learned pseudoinputs using backpropagation (like in the VampPrior) or by the proposed approach, and how learned pseudoinputs compared to real data.
The rebuttal contains many new results and addresses many concerns raised by the reviewers. Therefore, I tend to accept the paper.

Decision: Accept (Poster)