Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a variational inference method (BB PSVI) based on PSVI, which learns a variational approximation over model parameters given pseudo data that approximates the true posterior over model parameters given real data by maximizing a variational lower on the true data likelihood (ELBO-PSVI). 
A valid but generally intractable choice for the variational approximation is the true posterior over model-parameters given pseudo-data. To work around the tractability issues the authors propose to perform (adaptive) importance sampling which leverages another variational proposal over model-parameters by maximizing a lower bound on the pseudo-data likelihood (ELBO-IP).
The authors show that replacing the pseudo-data likelihood term in the PSVI-IS ELBO with the IP ELBO results again in a lower bound on the data likelihood. This variational lower bound (ELBO-PSVI-IS-BB) constitutes their final objective used to optimize the pseudo-observations. The variational approximation over model-parameters given pseudo-data can be modified to incorporate an weighting factor for individual pseudo-datapoint that can be learned jointly by maximizing the PSVI-IS-BB ELBO.

strengths_and_weaknesses: **Originality:**
The idea of using an importance sampling based approximation of the *pseudo posterior* to derive a black box version of PSVI is original and methodologically interesting and allows the use of PSVI for previously intractable black box models. As such I believe that this is a potentially interesting paper for both methodologically interested readers and application-oriented ML partitioners alike.

**Quality:**
The methodology and technical argument are sound.

**Clarity:**
The paper is well structured and clearly written. Given that the authors are using a nested variational inference step using self-normalized importance sampling, they should definitely cite recent work on Nested Variational Inference [1] in their related work subsection discussion Variational inference, importance weighting, and Monte Carlo objectives.

[1] Zimmermann, H., Wu, H., Esmaeili, B., & van de Meent, J.-W. (2021, December). Nested Variational Inference. 35th Conference on Neural Information Processing Systems (NeurIPS 2021).

**Significance:**
The presented results show significant improvements in accuracy and test log-likelihood for the presented models and datasets. I am however skeptical how the proposed approach performs on data with complex structure and high intrinsic dimensionality as the importance sampling approach is likely to fail here regardless of smart position-initialization and acquisition schemes. This trend seems to be already visible in the evaluation of the presented datasets (webspam, phishing, adult) where the effective sample size is non-trivial but rather low. I believe that additional discussion and an additional experiment on synthetic data which systematically evaluates the performance of the proposed method under increasing dimensionality would be great to gain further insights.

questions: The authors observe that existing approaches are limited by the data dimensionality while their approach allows for accurate approximate inference on the full data set regardless of dimensionality. While seemingly true for the presented models and datasets, intuitively the quality of the importance sampling estimate scales badly with dimensionality and is heavily reliant on good proposals for the position of the pseudo data points. The mechanisms to acquire positional proposals (pruning, incremental selection, or even potentially smarter ones like Bayesian optimization) seem to also scale badly with dimensionality as exponentially more positions have to be considered as the dimensionality increases. Is this an issue you'd expect to see in practice on different datasets? Why isn't this an issue for the webspam dataset (d=128)? Can you please elaborate a bit on that topic.

**Some minor things I noticed:**                                                                                               
- Line 64-66: the posterior distributions $p(\theta \mid u ,z, v)$ and $p(\theta \mid x, y)$ are over model parameters and not over the pseudo-dataset and dataset as stated in the text.

limitations: See questions regarding scaling with dimensionality
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a general pseudo-data sparse black-box variational inference method. Using the idea of Bayesian pseud-ocoresets from [25], the paper proposes the ELBO formulation of the sparse pseudo-data variational inference. Importance sampling is used for the variational distribution. A nested variational inference objective is further developed over the pseudo points and the variational parameters. How to update or select the coreset data is also discussed. Experiments are carried out on two different models: logistic regression and Bayesian neural networks. 
strengths_and_weaknesses: Strengths:

- The proposed black-box pseudo-coreset variational inference formulation is novel. And the derivation follows through.

Weaknesses:

- The experiment results on MNIST do not seem very competitive. With 10 data points in the coreset, the inducing point type of approaches can achieve 90+% accuracy. Also, there is some discrepancy with the reported accuracy of Dataset Condens.[44] in the original paper. Also, for Dataset Condens.[44], SLDD[37], and Dataset Distill.[42] the model uses 100 condensed data to train NN from scratch and achieve the reported accuracy. While for the proposed method, it is also using the labels of all training data for learning $\phi$, $u$, and $z$.
- The motivation behind importance sampling is not very clear to me. It does not exactly explain why not to use the ELBO defined by the approximate posterior ($q(\theta|u,z;\phi)$) directly.
- For the regression tasks and MNIST, it would be more interesting to look at the pseudo-dataset learned and analyze if they make sense, just as what is done with the simulated datasets.
questions: 1. I would hope to get more explanation on why importance sampling is used instead of directly using ELBO defined by the approximate posterior ($q(\theta|u,z;\phi)$).
2. Also it would be more convincing if there is some experiment that shows the promises of using this approach to achieve something that other methods cannot.
limitations: N.A.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors present a new method to build coresets for inference tasks. That is, a method to build a "smaller" dataset so that the posterior over this new dataset is close to the true posterior. Their initial formulation follows the previously used approach finding the coreset so that the resulting distribution is closest (in KL divergence) to the true target. Since this in often intractable, the authors propose two approximations based on the use of a variational approximation (to approximate the coreset posterior) and importance sampling (to estimate expectations with respect to the coreset posterior).
strengths_and_weaknesses: Strengths:

S1: The method is presented in a clear way.

S2: The method is novel and simple to implement.

S3: The method appears to work well in the empirical evaluation presented.

Weaknesses:

W1: The proposed algorithm produces two outputs: (1) a coreset, and (2) a variational approximation (which approximates the coreset posterior, which in turn approximates the true target posterior). Regarding (2), a typical application of VI is as efficient (or even more efficient, since it is also compatible with subsampling and does not involve a nested optimization problem) and leads to similar or better results. Thus, the method's main benefit comes from producing a coreset (1), which may be used to enable other methods, such as efficient MCMC. But the paper does not explore this. I think it would be quite interesting to see how this method enables other algorithms (that would otherwise be very slow, e.g. MCMC) and how the resulting performance compares against plain VI.

W2: The use of a simple variational distribution may hurt the quality of the coreset built. I think this would happen because of two "forces" that may be pushing in different directions. The first term of the objective (eq. 9) is pushing towards a coreset that leads to a coreset-posterior that's close to the true posterior, while the second term to a coreset that leads to a coreset-posterior that is not too far from the variational approximation $q(\theta\vert \psi)$. A large increase in either of these terms may lead to an increase in the loss, and thus have to be balanced, possibly sacrificing coreset quality. Simply put, the use of a variational approximation leads to a tractable but biased approximation of the "ideal" objective to build the coreset from eq. 5, where the quality of the coreset may depend on the amount of bias. While the use of importance sampling as a debiasing tool as presented in the paper may ameliorate this issue, importance sampling is known to work poorly in high dimensions. I think that adding a more explicit discussion about this bias should be included somewhere in the paper.

W3: Related to W2, another way to improve the bias issue would involve the use of a more powerful posterior (e.g. full-covariance Gaussian, flows). It would be interesting to see how the performance of the method improves in this case, and if the use of the resulting coreset leads to better performance when used with other methods, such as MCMC (related to W1).

I think W1 is quite important, specially since VI is already giving an approximate solution. As I see it, a motivation for building coresets is that they may enable more accurate but otherwise more inefficient methods, such as MCMC. Thus, exploring whether the use of the resulting coresets with more accurate methods (e.g. MCMC) yield better results than the plain VI solution seems particularly relevant. (This is also related to W2, as the quality of the variational approximation may affect the quality of the coreset obtained.)


A few additional comments:
- Equation 4 defines $q(\theta\vert u,z) = p(\theta\vert u,z)$. But then in equation 5 both $p$ and $q$ are used. I think it would be better to just decide to use one of them, and stick to it throughout.
- I think the paragraph between lines 97-102 may need some editing? It seems that the line starting from the textbf second is redundant with what is said before?

questions: The most important point to be addressed is, in my opinion, W1. Simply put, convincingly show the benefits of using the resulting coreset over other naturally scalable approaches, such as plain VI. Addressing W2-W3 would be nice but not strictly necessary.

A few additional questions (these do not affect my score):

- Have you tried optimizing the full objective jointly, instead of using a nested optimization procedure?

- In table 1 the random coreset approach appears to work significantly better than Sparse VI. Do you have any intuition regarding why this may be? (I appears that the incremental/greedy approaches perform poorly, since Sparse BBVI also works worse than the random approach).
limitations: Are the limitations of the work explicitly discussed somewhere?
review_title: null

Meta-review: metareview: This paper bridges black-box probabilistic modeling to the use of variational 'pseudo coresets'. The reviewers seem to have reached a positive consensus and have engaged in a fruitful discussion during the rebuttal period. Overall, the contributions are solid and of interest to the community, but I recommend the reviewers take into consideration the remaining concerns raised by reviewer vKEG in preparing a thoughtful revision. 
confidence: Less certain
Decision: Accept