Black-box coreset variational inference
Authors: Dionysios Manousakas
Hippolyt Ritter
Theofanis Karaletsos
Venue: NeurIPS 2022 Accept (2022)
Keywords: Bayesian coresets
approximate inference
variational inference
probabilistic machine learning

Abstract: Recent advances in coreset methods have shown that a selection of representative datapoints can replace massive volumes of data for Bayesian inference, preserving the relevant statistical information and significantly accelerating subsequent downstream tasks. Existing variational coreset constructions rely on either selecting subsets of the observed datapoints, or jointly performing approximate inference and optimizing pseudodata in the observed space akin to inducing points methods in Gaussian Processes. So far, both approaches are limited by complexities in evaluating their objectives for general purpose models, and require generating samples from a typically intractable posterior over the coreset throughout inference and testing.  In this work, we present a black-box variational inference framework for coresets that overcomes these constraints and enables principled application of variational coresets to intractable models, such as Bayesian neural networks. We apply our techniques to supervised learning problems, and compare them with existing approaches in the literature for data summarization and inference.

Black-box Coreset Variational Inference
Dionysis Manousakas" Metal
Hippolyt Ritter* Meta hippolyt@meta.com com
Theofanis Karaletsos Insitro
dm7548cantab.ac.. ac..aa..a.................
theofanis@@araleetsoo.u
Abstract
Recent advances coreset methods have shown that selection of representative datapoints can replace massive volumes of data for Bayesian inference, preserving the relevant statistical information and significantly accelerating subsequent down- stream tasks. Existing variational coreset constructions rely on either selecting subsets of the observed datapoints, or jointly performing approximate inference and optimizing pseudodata in the observed space akin to inducing points methods in Gaussian Processes. So far, both approaches are limited by complexities evaluating their objectives for general purpose models, and require generating samples from a typically intractable posterior over the coresct throughout inference and testing. In this work, we present black-box variational inference framework for coresets that overcomes these constraints and enables principled application of variational coresets to intractable models, such as Bayesian neural networks. We apply our techniques to supervised learning problems, and compare them with existing approaches in the literature for data summarization and inference.
Introduction
Machine learning models widely trained using mini-batches of data [8, 22], enabling practitioners to leverage large scale data for increasingly accurate models. However, repeatedly processing these accumulating amounts of data becomes and more hardware-intensive over time and therefore costly. Thus, efficient techniques for extracting and representing the relevant information of a dataset for given model are urgently needed.
Recent work on coresets for probabilistic models has demonstrated that Bayesian posteriors on large scale datasets can be sparsely represented via surrogate densities defined through weighted subset of the training data (Sparse Variational Inference; Sparse VI) [10] or set of learnable weighted pseudo-observations in data space (Pseudodata Sparse Variational Inference; PSVI) [31]. Similar to inducing points in Gaussian Processes [50], these target to parsimoniously represent the sufficient statisties of the observed dataset that required for inference in the model. Replacing the original data via the reduced set (pseudo-) observations unlocks the potential of scaling up downstream learning tasks, compressing for efficient storage and visualisation, and accelerating model exploration.
Unfortunately, Sparse VI and PSVI rely on access to exact samples from the model posterior and closed form gradients of it to be evaluated and updated. Yet, the posterior for most common probabilistic models is intractable, limiting their use to the class of tractable models or necessitating the of heuristics deviating from the core objective. Dealing with intractable models typically necessitates the use of approximate methods such as variational inference, which has been increasingly popular with the development of black-box techniques for arbitrary models [43].
In this work, we overcome these challenges and introduce a principled framework for performing approximate inference with variational families based on weighted coreset posteriors. This leads
Equal contribution Now at Amazon 'Research supporting this publication done at Meta
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

on full data
HMC on extracted coreset
HMC on random coreset
Figure 1: Posterior inference results via Hamiltonian Monte Carlo [35] on the full training dataset, 16- 16-point coreset learned via BB PSVI, and 16-point coreset comprised of randomly selected, uniformly weighted data points for half-moon binary classification with a feedforward neural network. Coreset locations marked in crosses with the size indicating the relative weight. Regions of high predictive uncertainty are visualized by dark blue shaded areas.
to extensions of Sparse VI and PSVI to black-box models, termed BB Sparse VI and BB PSVI. respectively. As illustrated in Fig. 11 (and further detailed in the supplement) on synthetic binary classification problem with a Bayesian neural network - an intractable model-class for prior ap- proaches such as PSVI relying on exact posterior samples - BB PSVI is able to learn pseudodata locations and weights inducing a posterior that more faithfully represents the statistical information in the original dataset compared to random selection. However, emphasize that the utility of coresets goes beyond applying expensive but accurate inference methods to large-scale datasets, and may drive progress in fields such continual learning
demonstrate later on.
Specifically, we make the following contributions in this work:
1. We derive principled interpretation of coresets as rich variational families,
2. we modify the objective to obtain a black-box version of it using importance sampling and nested variational inference step.
3. utilizzng this objective, we propose BB PSVI and BB Sparse VI, two novel suites of variational inference algorithms for black-box probabilistic models,
4. we empirically compare to prior Bayesian and non-Bayesian coreset techniques, and
5. we showcase these techniques on previously infeasible models, Bayesian neural networks.
2 Background
2.1 Variational Inference
Consider modeling problem where we are given standard model:
p(æ, y, 0) p(y xx, ()p(O).
(I)
Given a dataset D = {(x, y) the task of inference and maximization of the marginal likelihood p(ylar) (a)) entails estimation the posterior distribution over model parameters 0. p(0) y). In general, the posterior p(0) æ., y) is not tractable. A common solution is to resort to approximate inference, for instance via variational inference (VI). In VI, we assume that we can approximate the true posterior within a variational family q(; X) with free variational parameters X, by minimizing the Kullback- Leibler (KL) divergence between the approximate and the true posterior D kL. (g(0; A)llp(O) x. y)). or equivalently by maximizing the Evidence Lower Bound (ELBO):
log x:) ELBO(A) = B.((:A)) log plyl2 22, 0 0)p(O)
(2)
(0:
2.2 (Bayesian) coresets and pseudocoresets
Bayesian coreset constructions aim to provide a highly automated, data austere approach to the problem of inference, given dataset D and a probabilistic model Eq. (1). Agnostic to the particular

---

inference method used, Bayesian coresets aim to construct a dataset smaller than the original one such that the posterior of the model of interest given the coreset closely matches the posterior over the original dataset, i.e. p(0 ru, 2, v) R p(0| ar y). The coreset is a weighted dataset representation (v, u, 2), where v denote a set of positive weights per datapoint, || = M < N and (um; 2m are the input and output vectors for the coreset. Initial Bayesian coreset constructions formulated the approximation of the true data likelihood function uni- formly over the parameter space. This perspective has admitted solutions via convexifications relying on importance sampling and conditional gradient methods [12, 23], greedy methods for geodesic ascent [11], well non-convex approaches employing iterative hard thresholding [56]. Recently, Sparse VI [10] reformulated the problem as performing directly VI within a sparse exponential family of approximate posteriors defined on the coreset datapoints.
problem as yielding a sparsity-constraiinn
PSVI The idea of using a set of learnable weighted pseudodata (or inducing points) (v, u, 2) := {Vmm Mmm &m Mm } to summarize a dataset for the purpose of posterior inference was introduced in [31], as the following objective:
0² u. arg min DKL (, 2, v) (p(0 v))-
0,8,
This objective facilitates learning weighted pseudodata such that the exact posterior of model parame- ters given pseudodata p(0) u,, 2, v) approximates the exact posterior given the true dataset p( a y). and was demonstrated on variety of models as a generally applicable objective. However, in [31] it was assumed that this objective is tractable, as demonstrated in experiments on models admitting analytical posteriors, or via the use of heuristics to sample from the coreset posterior, as is the case in a logistic regression employing a Laplace approximation.
Sparse VI Sparse [I[ targets to incrementally minimize the rhs of Eq. (3) via selecting a subset of the dataset as a coreset, thereby fixing the locations to observed data, and only optimizing their weights In its original construction, the coreset points are initialized the empty set and v O. In the general case of an intractable model, the coresct posterior is computed via Monte Carlo (and in the code optionally via Laplace approximation) on the weighted datapoints of the coreset (v. u, z). The next point selection step involves: (i) computing approximations of the centered log- lieliielihood function of each datapoint (I,, yn.) for model parameters 0, sampled from the current coreset poste- rior, via MC estimate f(z.. Ya, 0z) (log (log [z,,, 0.) 1/3 E poygp(yllaa [m,,, 0.t)) E SR..
O2 p00 u, v), (ii) computing correlations with the residual error of the total data ligelikelioood r(v) = (1 - v). and (iii) making a greedy selection of the point that [Corr(,, r)] en > 0
maximizes the correlation between the two: n² E arg max { nE[N]
where
Corr(f, en =
ro diag [ 1 - (+Eir!). Subsequently, the vector e gets optimized via minimizing the KL divergence between the coreset and the true posterior. For the variational family of coresets, under the assumption that θ is sampled from the true coresct posterior, the formula of the gradient of the objective in Eq. (3) can be derived, and approximated via resampling from the coreset over the course of e optimization. At the end of the optimization, the extracted points and weights are given input an approximate inference algorithm algorthm-commmnjj Laplace approximation ipproximation-whiihh which provides the final estimate of the posterior.
3 Black-bor coreset Variational Inference
In order derive a general PSVI objective, we observe that we first need to posit variational family involving the pseudodata appropriately. Let's assume the (intractable) variational family (0/u,, z) p(O 2), which has the following particular structure: it is parametrized as the posterior over 0 as estimated given inducing points u. 2. In this case, q(0 u, z) can be thought of as a variational program (see [44]) with variational parameters & = {u, 2 and expressed as:
q00uu u, 2) == p(0)u, u, 2) p(z/u, ())
(4)
p(z/u)
where the marginal likelihood term p(z|u) is typically intractable.
3

---

Utilizing this for the formulation of pseudo-coresed variational inference algorithm leads to the following lower bound on the marginal likelihood of the true data:
ELBOpsvi (u, z) By(O u.=) log plyl ()(())
(5)
q(0lu, 2)
Performing VI involves maximizing Eq. (5) with respect to the parameters {u, 2}. Scrutinizing this objective reveals that the posterior q(0) u. =) == p00 (2 2) is used in two locations: first, the sampling distribution for 0, second to evaluate log-densities of samples (scoring). This objective only be evaluated if the posterior p(0/u, [u, =) can be readily evaluated in closed form and sampled from directly. This causes this expression to remain intractable but for the simplest of cases, and reveals the need make it computable for any sample
3.1 Bayesian coresets for intractable posteriors
We identified two key problems to evaluate Eq. (5) involving q u.) z).
Sampling While we cannot sample from the expectation over q0)uu, =), we can draw K samples from tractable parametrized distribution O₃ rO; ub) with variational parameters +b (i.e. any suitable variational family). However, we cannot plug these samples from r into Eq. (5) directly, as they do not follow the exact distribution called for. To overcome this, we can leverage self- normalized importance sampling (IS) correction [39] to obtain the desired approximate samples from the coreset posterior. We thus estimate (0)/u, z) via importance sampling by re-weighting samples from the tractable distribution ((0z/)) We denote the resulting implicit distribution q(0 u, z; v)).
We assign the samples unnormalized weights wy (noting that p(= u,, 0&) == [] p(z u, 0x)):
(6)
r(0s; 1b)
and denote the corresponding normalized weights कx /// E3 wj. Using q0)uu, 2) from Eq. (4), we rewrite Eq. (5) as:
ELBOpsvIa = S <(() u, z) log
p(yl ())( q u, z)
do
(7)
/ 9 u, z) r(0;
log ply x 0)p(() ( u. z)
wy log ply x. 0z On)p(On)
Σ
(0k
Ox~r(0.b))
Scoring After having tackled sampling, shift our attention the second problem, evaluating q0)uu, 2) inside the logarithm. The key problem with scoring log q( O u. 2) as defined in Eq. (4) is the intractable marginal probability of the inducing points (IP) log p(z u). We overcome this by introducing a variational approximation to log (00 u, 2) using the same distribution r(0; ap) used above and obtain evidence lower bound ELBOpp of the IP:
EL (+b) (r(0))) log ((z(0,0)p(O) r(0;
pzgp(z)u)
Putting it all together We return to Eq. (7) and substitute (0x u, 2), per Eq. (4) to derive the following multi-sample importance sampling lower bound on the evidence of the observed data and expose the term log p(2
ply x. On (x)p(zz)) p(z|u,00 0x)
Ty log p 0x) p(z|u, 0,) + log p(=|u).
ELBOpsvi-SS (u, 2) ~ Σ Wk log
Σ Owr

---

By lower bounding logp(z/u u) with IL bb), we now propose our final black box objective ELBOpsvI BB as lower bound to ELBOPsvIIS(U, z), under which we can tractably score u, z:
ELBOPsVIIS x) 2 ELBOPsVLIS-BBB 2,
Σ wx log ply a Ox)
+ Eewr log p(z| u. ()p(()
(9)
p(z u, 0k)
r(0; 1b)
P
ply x 0k) u(z 0k)
1
p(z 0% ((x))
wg log
log
K
r(0s: 2b)
We now remind the reader out that by introducing ELBOpsvi-s we lower bounded log p(y)))) and that ELBOpsvI-S-B rigorously lower bounds ELBOpsvi-Is. showing that our proposed fully tractable black box objective ELBOpsvI-S is rigorous variational objective for approximating log p(ylar)
log logp(y)))) 2 ELBOpsvr.s (u, z) 2 ELBOPSVEIS.BBR 2b).
Maximizing EL ELBOPsvIIS Ba leads to optimizing parameters sp and thus adapting the proposals for the importance weighting throughout inference, within the confines of the chosen variational family. This is common approach when blending variational inference and Monte Carlo corrections and can be thought of as specifying an implicit variational family. Depending on the structure of the model one might elect to replace importance sampling with richer Monte Carlo schemes here. We also note that parameters vó serve maximization of evidence for inducing points given fixed {u, =}. while parameters {u, } are adapted to maximize evidence over observed data fær, y) which need to account for during optimization of said parameters in Section 3.4. Studying this objective further reveals that replacing our pseudodata dependent importance weights with a uniform distribution Wi = 1/K, k = I,..., K, or using a single-sample objective (i.e. in the degenerate case when K 1), allows us to recover the classical ELBO of Eq. (2), cancelling the pseudodata likelihood terms in the variational objective.? 2 The setting where we apply this black-box estimator to the basic PSVI algorithm for inferring locations & and model variables vp will be denoted BB PSVI.
3.2 Variational families: (un)weighted inducing points
When studying the objective in Eq. (5) quickly becomes evident that the exact amount of the selected coreset datapoints defines the available evidence for that particular posterior construction and as such is a quantity of interest. We will consider this quantity pseudo-evidence.
For notational brevity so far we have considered the variational parameters & to represent pseudodata directly, while rescaling using the fixed data compression ratio to maintain invariance of the total pseudo-evidence to the coreset size. We will be calling this setting unweighted pseudodata.
One choice we can make now ensure that the available pseudo-evidence is invariant to the chosen size of the coreset support and potentially equal to the regular dataset. To achieve that, we set:
(0 2) == p(0 =) p(z/u, u. 0) NIM p(0)
Ip(Zz 0) N/M p(0)
(10)
p(z(u)
p(zlu)
with p(2 u) similarly reweighted via the selected observational data compression ratio N/M, i.e. for continuous parameter spaces p(z|u) == p(z p())ddd
We may also consider a scenario where we have more generally weighted pseudodata, which involves an additional learnable parameter v; per datapoint u, and permits the following interpretation: v. ~ Mu)ti( Sampling repeatedly from this distribution yields a collection of inducing points with their frequencies governed by their respective weights:
p()) II p u4, ")
q( 10 v u, z) == p(0 u,
(11)
p(z u)
Simplifying the notation for weighted log- liglliiess throughout, we denote:
log ((( 2 U2 u. 0) := U; log p(z; /4, 0).
(12)
"The nested variational program Eq. (8) still allows propagation of non-zero gradients w.r.t. the pseudodata.
5

---

We can now also combine the ideas of learning the weights of pseudodata and controlling the amount of pseudoevidence by posing parametrizations where v; is non-negative and sums (e.g. == sof (mma((3;) = ..... M), and adding global variational parameters to optimize the magnitude of total evidence over the pseudodata (e-g. log ((zu,, u, 0, == av log pl (2|u,, 0)), where a can be set by hand (i.e. to N/M) or be an extra learnable quantity. We will explore various such choices for coreset variational family parametrizations & = {u, 2, u, a in our experiments and the effects they have on learning.
3.3 Incremental and Batch Black-Bou Sparse VI
The estimator we derive can also be used to design a black-box version of the Sparse VI algorithm introduced in [10] (see Sec. 2.2), by the insight that it requires only the variational parameters for coreset weights to be updated while using copies of real datapoints as coreset locations. This unlocks various choices to the overall algorithmic flow. The original incremental construction scheme can become black-box via modifying how the approximate posterior on the coreset data gets computed, and introducing generalized objective for the optimization of model evidence involving the coreset parameters. In this construction (BB Sparse VT), we posit a variational family r(0: 1)) (e.g. mean-field variational distributions) and maximize the corresponding ELBO computed on the weighted datapoints of the coreset. Using the extracted variational approximation, we draw samples from the coreset and correct for the model. Next, over the greedy selection step, we use the samples and importance weights compute the centered log-likelihood vectors f(zn, In. 0.)
log p(y, (.,, 0.)-E. War log yn En, O2), ). and select the next datapoint via the greedy correlation maximization criterion. Subsequently, we maximize w.r.t. v. rb an extension of the PSVI ELBO per Eq. (9), where the coreset data log- loglliiel is multiplied by the weight vector in Eq. (12):
refine the weight vector towards the true data posterior,
=E ih log ply ar Ox) Pcrrr
1
log p(ylo, x. 0k)p(Ox)] r(0x: 2/)
Spu (1, ab)
plyoo x 0x) + K
(13)
The importance sampling scheme for the posterior is defined identically to Eq. (6), after substituting the likelihood term with p(y|U, x. 02) for each sample 0₂ ~ r(0s: v)) At each iteration, this variational objective (which, following similar reasoning with the previous section, be a lower bound of the evidence), can maximized wrt the coreset and variational parameters v, sb. Moreover, we can omit the incremental inclusion of points to the coreset, and consider a batch version of this construction, where we initialise at a random subset of the original dataset, keep coreset point locations fixed, and optimize only the weights attached to the coreset support using Eq. (13) with non-negativity constraints. Finally, we use the variational approximation fit on the coreset to compute predictive posteriors on unseen datapoints correcting for our importance sampling.
be shown to
We also define variant of the algorithm without incremental selection through pruning based on the weighting assigned to the coresct points when optimizing the ELBO of Eq. (13). We can start from large coreset size and, after training, reduce the size of the summary by keeping K samples from a multinomial defined the coreset points via their far learned weights, and re-initialising. We provide pseudo-code for the respective methods in Algs. 3-5 in Supplement B.
3.4 Nested optimization
When using a nested optimizer for the maximization of the ELBOpsvi rearranging the terms of the objective places our method in the generic arrdiaaity-conssrainee bilevel optimization framework for data distillation and coreset constructions [6, 26). VI maximizing the objective of Eq. (5) inside parametric family r(0; v)) can be reformulated as a bilevel optimization problem as:
v²
u
arg min L(v, u. 2, 2b" (v, u. 2): y)
s.t. 26* = arg min Elb, u,,,)), (14)
where
L(v, u(,, ELBOPsvIIS u, z. v)
(15)
and
ple, z O2 (2)p(())))
( U, u) --==))=EEBBBO u., : -Barrr log
(16)
r(0x; +b)

---

We optimize via iterative differentiation [29], i.e. trace the gradient computation over the optimization of the variational parameters v when solving the inner problem, and use this information at the computation of the outer objective gradient wrt the variational coreset parameters & {u, u, 2, a).
Experiments
In this section we evaluate the performance of our inference framework intractable models and compare against standard variational inference methods and earlier Bayesian coreset constructions, as well as black-box extensions of existing variational coresets that rely on our generalized ELBO Eq. (9).
As running baseline with unrestricted access to the training data, we use the standard meant-field VI with diagonal covariance matrix. To capture the implications of enforeing data cardinality constraints, also construct random coreset using randomly selected data subset with fixed data locations and fixed likelihood multiplicities correcting for the dataset size reduction, which we then use to optimize a mean-field approximate posterior. As carlier work relied on Laplace approximations on coreset data as a black-box approach for sampling for intractable coreset posteriors, we also experiment with Laplace approximations random data subsets extra baseline. We make code available at www. www. www.hb. com/ffcabbokreeeeaccch wwwpcchhhhaaacocccccoeooo Cow.rboooo-.rrres--V......
4.1 Logistic regression
First, we perform inference on logistic regression fitting 3 publicly available binary classification datasets [3] with sizes ranging between 10k: and 150k: datapoints, and and 128 dimensions. We posit normal priors 0 ~ (0, 7) and consider mean-field variational approximations with diagonal covariance. the following, all presented metrics are averaged across 3 independent trials, and we show the means along with the corresponding standard errors.
Impact of variational parameters We present the predictive metrics of accuracy and log-likelihood on the test set in Tables I and 2. We observe that coreset methods relying on optimizing pseudodata have the capacity to better approximate inference on the full dataset for small coreset sizes regardless of data dimension, reaching the performance of unrestricted meennffield VI with the use of less than 50 weighted points. contrast, the approximations that rely on existing datapoints are limited by data dimensionality and need to acquire larger support to reach the performance of full-data VI (with effects being more evident in webspam, which has a data dimensionality of 128). Including variational parameterisations for optimal scaling of the coreset likelihood aids inference, offering faster convergence. Omitting correct for dataset reduction (o = 1) has detrimental effects on predictive performance. As opposed earlier constructions limited by heuristics on coreset posterior approximation throughout optimization, our variational framework enables approximating the full data posterior within the assumed variational family. Evaluation of predictive metrics and computation time requirements across a wider range of coreset sizes is included in the supplement.
Impact of importance weighting In Fig. 2 we plot the normalized effective sample size (ESS) of our importance weighting scheme for 10 Monte Carlo samples computed on test data. We observe that the posteriors can benefit from IW, achieving non-trivial ESSs, which tend to converge to close values for large coreset sizes. This finding applies the testing phase also on ablations of the PSVI variational training that replace the IW scheme with uniform sampling.
Importance sampling in high dimensions. Importance sampling suffer from large variance in high dimensional spaces [39]. Hence, one may be skeptical about the approximation quality the scheme can provide. To this end, in Supplement G conduct complementary experiment with a synthetic logistic regression problem. As dimensionality increases, we find that while ESS goes down, it remains non-trivial. More involved sampling methods such as [58] a promising avenue for leveraging the modular nature of BB PSVI to achieve further performance improvements.
4.2 Bayesian Neural Networks
In this section we present inference results on Bayesian neural networks (BNNs), model class that previous work on Bayesian coresets did not consider due to the absence of a black-box variational estimator. In the first part perform inference via black-box PSVI on 2-dimensional synthetic data
7

---

wabspam
phishing
aduat
ESS (num. of IW samples)
0.200
ESS (num. of IW samples
0.35
ESS (num. of IW samples)
0.4
PSVI w/ uniform
0.175
sim plmp
0.150
0.25
0.3
BB PSVI w/ on simplex,
BB Sparse Vi [Batch]
0.125
0.20
0.100
0.075
0.10
50
100
150
200
0
50
100
150
200
0
50
100
150
200
Coreset size
Coreset size
Coreset size
Figure 2: Normalized effective sample size for predictions on test data using 10 Monte Carlo samples
from the coreset posterior over increasing coresct size.
Table 1:
Test accuracy on 3 logistic regression datasets. (+) denotes use of softmax parameterisation for the weights N tmooxtax(( tmax (3).
vobepan
phicting
adult
M
100
BB PSVI
u(*).
92.2 = 0.1
92.4 0.1
92.3 2 0.0
90.1 1: 0.5
91.2 1.0
90.4 =: 0.0
82.2
83.5 2:
$3.5 = 0.0
nis),
92
92.2
6 0.9
83.6 0.0
M.
92.3
92.2
89.8
90.3
80.3
81.5
un unit. oi(a),
90.9 0.3
91.9 0.1
92.2 00.20:.3 0.5
90.2 0.3
88.9 0.7
90.5 2 0.0
81.0004
0.1
83.5 0.5
02.1=001
90.8
90.6
90.4
83.5
IW, u(x).
v=l.
61.1 = 1.5
81.2
61.2 2.2
85.5 10.4
86.3 11.1
87.3
11.5
75.0 3.7
75.1
75.8
BB Spane VI
Barch
80.1 0.9
86.9=02
88.9 0.4
87.6
0.3
90.2
78.1=11
826=044
83.5 1155+000 0.7
Incremental
0.0
76.9=331
$
2.1
38.9
$8.1 2.2
78.6=0.1
$1.9
8.74
P3V]
Sparse VI (10)
-
74.2 74.2-47
74.6
87.0 + 166
89.3 0..
90.0
78.1-21
79.0
80.7
Subset Laplace
60.0
=8
82.7 = 1.3
72.7 4.1
84.7 221
$8.0
64.
71.0
812 1.2
Rand. Coneset
8..4
9
using single-hidden layer BNNs, while in the latter part we evaluate the performance our methods on compressing the MNIST dataset using the LeNet architecture [25].
Simulated datasets We generate two synthetic datasets with size 1 datapoints, corresponding noisy samples from half-moon shaped 2-class dataset, and mixture of 4 unimodal clusters of data each belonging to different class [24], and use layer with 20 and 50 units respectively. To evaluate the representation ability of the pseudocoresets we consider we initialise the pseudo locations on a random subset equally split across categories, and a random initialization using Gaussian centered on the means of the empirical distributions. Fig. visualize the inferred predictive posterior entropy, along with coreset points locations and relative weights. Regardless of the initialization, BB PSVI has the capacity to optimize the locations of the coreset support according to the statistics of the true data distribution, yielding correct decision boundaries. Importantly, the coreset support consists itself of separable summarizing pseudo points, while the likelihood reweighting variational parameters enable emphasizing on critical regions of the
initialization schemes:
Table 2: Test negative log. likelihood on 3 logistic regression datasets.
gebopax
phishing
sd:1t
M
10
10
100
10
BB PSVI
e(+).
21.7
21.9 0.2
21.22 0.2
38.1 0.8
34.8
20.0
36 1.6.1.. = 118
34.8 + 3.8
26.0 0.1
e(+).
22.1
0.5
21.6 0.2
21.0 0.0
33.6 0.6
342 1:11
25.0 0.1
332
34.0::00
$3.9-1...0
01.6+0..
26.0+0..1
78.0-66
31.++0.1
26.4
0.3
1.5
80.8
57.7 10.9
2..
70.5 14.5
33.910.1
$3.9
und. JW, eC M
21.2 0.2
20.8
20.8 2.1
31.
$1.2227
26.9
48.1 2.2
$3.9
3151
wlo JW, ((+),
29.8
2.1
41880.
2..
63.8555
15.8 6.3
26.9...
74.0-02 10.2
3422
310
00.4
78.4+44
7..4
37.5=24.4 24.4
34.7 +11T+22300 23.6
38.3 24.8
07.1=7.1
03.1 13.1+35
40.1
BB Spurse VI
Batch
65.0 8.6
39.1
1.2
60.2 2.0
1..2
2..9.12.11
70.8
18.3.1.6
$3.9 10.2
Incremental
86.1
13.=16
34.8131
2.
36.
[34]
Sparse [10]
9..6
38.3+13
37.7
45.3-48
Subwei Laplace
06.5
16.0.1..
37.8
55.1 47
4.44
10.9
119.2.16.
65.0
39.11.11
Rand. Corexet
175.1 25.3
76.9 + 15.5
K3.1
122.0 13.4
166.0 63.0
47.6 3.4
251.4 32.1
178.5 43.55
51
Full MFVI
19.8 0.00
26.1
33.9 1.00

---

BB PSVI (random init)
BB PSVI (subset init)
Batch BB Sparse VI (pruning)
Mean-field Vi (full data)
Figure 3: Predictive posterior entropy simulated datasets. Coreset points are visualised in crosses, with size proportional their weights.
Table 3: Test accuracy on the MNIST dataset using NNs with the LeNet architecture for our black-box coreset constructions and other learning baselines. Full MFVI accuracy: 99.13 + 0.07
10
30
50
80
100
250
(++).
53.65 1 2.48
76.78
2.1
85 65
0.0.
89.43 1 0.56
91. 91.28 0.74
93.85 0.0
u(+). u,
52.97 1 3.32
76.07
2.06
85.05
0.3
89
0.95
90.77 : 0.92
93.51
0.14
BB PSVI
u(t).
40.43 1 0.66
63.15
0.14
77.54
1.32
84.52 0.188
65.47 1 0.21
90.98
0.33
m(*). x
40.69 + 0.84
62.8
0.02
76.85
0.86
84.52
0.31
86.03
0.6
91.3 + 0.37
Sparse VI [Batch]
43.7 2.14
68.41
0.6
77.21 2.43
83.88
0.44
65.65 1. 0.17
91.54 0.16
Dataset Condens. [57]
93.9
SLDD
22.7#28.
Dataset Distill. [54]
79.5 01.5:100 8.1
Random Coresct
42.17 1.74
3.26
2.11
06.21+
87.44
0
landscape, assigning larger weights to pseudopoints lying close to the boundaries among the classes. Moreover, our pruning scheme which makes of batch BB Sparse VI is able gradually compress large coreset of 250 datapoints to compact performant subset of 20 datapoints arranged in critical locations for learning the Bayesian posterior.
this part we assess the approximation quality of large-scale dataset compression for BNNs via coresets. We compare the predictive performance of black-box PSVI against standard mean-field VI, random coresets and frequentist methods relying on learnable synthetic data, namely dataset distillation w/ and w/o learnable soft labels [49, 54], and data condensation [57]. We can that BB PSVI consistently outperforms the random coreset baseline for coreset sizes in the orders of few tens or hundreds, however the gap narrows towards 250 coreset points (Table 3). Moreover, at size 100 our construction provides better test accuracy compared to two frequentist approaches, enjoying the additional advantage of providing uncertainty estimates.
MNIST
Continual learning with Bayesian coresets As discussed in the introduction, envision coresets being leveraged as a tool beyond scaling sampling methods to large datasets. One such example setting is continual learning, where past methods have encoded statistical information in the form of approximate posterior distributions over the weights [36, 47J. While [36] leverages (randomly selected) coresets, these necessarily lose information about the data due the approximate nature of their closed-fome posteriors. Bayesian coresets, in contrast do not make use of an explicit closed- form posterior, as they rely on the implicit posterior induced by the coreset, representing the prior information by mixing the pseudo data at frequency equal to their weight with the new data. We showcase such continual learning method on the four-class synthetic problem from above Fig. 10 with the difference that each class arrives sequentially after the initial two and prior classes are not revisited except for the coresets (details in Supplement D). Non- Non-Bayesian approaches to continual learning with coresets have been investigated in more depth in [4, 6, 55].

---

Task
Task
Task 3
Figure 4: Incremental learning: coreset is constructed by incrementally fitting a BNN 3 classifi- cation tasks, starting with 10 coreset points and subsequently increasing to 15 and 20.
Related work and discussion
Coresets and data pruning beyond Bayesian inference The paradigm of coresets emerged from seminal work computational geometry [1, 18), and has since found broad use in data-intensive machine learning, including least mean square solvers [28]. dimensionality reduction [20], clus- tering [2, 19, 27] and maximum likelihood estimation [3, 32). Coreset ideas have recently found applications in active learning [42, 7], continual learning [4, 6] and robustness [30, 33]. Recent works in neural networks have investigated heuristics for pruning datasets, including the misclassification events of datapoints over learning [52], or gradient scores in early stages of training [41].
Pseudodata for learning Despite the breakthroughs of classical coreset methods, the common confine of selecting the coresets among the constituent datapoints might lead to large summary sizes in practice. Exempt from this limitation are methods that synthesize informative eseudl-examples for training that can summarize statistics lying away from the data manifold. These appear in the literature under various terms, including dataset distillation [26, 29, 54), condensation [57], dataset meta-learning [37] and inducing points [38]. Moreover, recent work [51] has demonstrated that incorporating learnable pseudo-inputs as hyperparameters of the model prior can enhance inference in variational auto-encoders. Learnable pseudodata that resemble the sufficient statistics of the original dataset have also been used in herding [13] and scalable GP inference methods [48, 50].
Variational inference, importance weighting, and Monte Carlo objectives Importance weight- ing can provide tighter lower bounds of the data log-likelihood for training variational auto- encoders [9], resulting in richer latent representations [14]. In later work, [15] propose an importance weighting VI scheme for general purpose probabilistic inference, while [44, 45] introduce variational programs as programmatic ways specify variational distribution. Higher dimensional proposals are commonly handled via Sequential Monte Carlo techniques [16, 34), while [58] develops a family of methods for learning proposals for importance samplers in nested variational approximations. Two forces are at play when considering our approximate inference scheme: the variational family r and the sampling scheme used estimate q( u, x; v)) These two interact nontrivially, since forms the proposal for the importance sampling scheme, and this inference yields gradients for the formation of the coreset. For hard inference problems richer sampling strategies (e.g. SMC) and variational families may yield useful coreset and posterior beliefs over the model faster and with more accuracy.
6 Conclusion
We have developed a framework for novel black-box constructions of variational coresets that can be applied at scale to large datasets well as richly structured models, and result in effective model- conditional dataset summarization. We overcome the intractability of previous inference algorithms with our proposed black-box objectives, which we develop for both Sparse VI as well as PSVI, and propose a suite of algorithms utilizing them.
Through experiments on intractable models, we show that the inferred coresets form a useful summary of a dataset with respect to the learning problem hand, and are capable to efficiently compress information for Bayesian posterior approximations. In future work, we plan to extend this inference toolbox for privacy purposes and non-traditional learning settings, e.g. continual learning, explore alternative objectives, and richer models of coresets.
10

---

References
[1] K. Agarwal, Har-Peled, K. R. Varadarajan, et al. Geometric approximation via coresets. Combinatorial and computational geometry, 2005.
[2] O. Bachem, M. Lucic, and A. Krause. Coresets for nonparametric estimation the case of De-mes.s. In ICML, 2015.
[3] O. Bachem, M. Lucic, and A. Krause. Practical coreset constructions for machine learning. arXiv preprint, 2017.
[4] L. Balles, G. Zappella, and C. Archambeau. Gradient-matching coresets for continual learning. arXiv preprint, 2021.
[5] E. Bingham, J. P. Chen, M. Jankowiak, F. Obermeyer, N. Pradhan, T. Karaletsos, R. Singh,
P. Szerlip, P. Horsfall, and N. D. Goodman. Pyro: Deep universal probabilistic programming. JMLR, 2019.
[6] Z. Borsos, M. Mutny, and A. Krause. Coresets via bilevel optimization for continual learning and streaming. In NeurIPS, 2020.
[7] Z. Borsos, Tagliasacchi, and A. Krause. Semi-supervised batch active learning via bilevel optimization. In ICASSP, 2021.
[8] L. Bottou. Large-scale machine learning with stochastic gradient descent. In COMPSTAT: 2010.
[9] Y. Burda, R. B. Grosse, and R. Salakhutdinov. Importance weighted autoencoders. In ICLR. 2016.
[10] Campbell and B. Beronov. Sparse variational inference: Bayesian coresets from scratch. In NeurlPS, 2019.
[11] T. Campbell and T. Broderick. Bayesian coreset construction via greedy iterative geodesic ascent. In ICML, 2018.
[12] T. Campbell and Broderick. Automated scalable Bayesian inference via Hilbert coresets. JMLR, 2019.
[13] Y. Chen, M. Welling, and A. J. Smola. Super-samples from kernel herding. In UAI, 2010.
[14] C. Cremer, Q. Morris, and D. Duvenaud. Reinterpreting importance-weighted autoencoders. In ICLR, 2017.
[15] J. Domke and D. R. Sheldon. Importance weighting and variational inference. In NeurlPS, 2018.
[16] A. Doucet, A. M. Johansen, al. A tutorial particle filtering and smoothing: Fifteen years later. Handbook of nonlinear filtering, 12(656-704)3, 2009.
[17] D. Dua and C. Graff. UCI Machine Learning Repository. Irvine, CA: University of California. School of Information and Computer Science, 2019.
[18] D. Feldman and M. Langberg. A unified framework for approximating and clustering data. In STOC, 1.
[19] D. Feldman, M. Faulkner, and A. Krause. Scalable training of mixture models via coresets. In NeurlPS, 2011.
[20] D. Feldman, M. Schmidt, and C. Sohler. Turning big data into tiny data: Constant-size coresets for k-means, PCA, and projective clustering. SICOMP, 2020.
[21] E. Grefenstette, B. Amos, D. Yarats, P. M. Htut, A. Molchanov, F. Meier, D. Kiela, K. Cho, and Chintala. Generalized inner loop meta- learnie.. arXiv preprint 10000299/..07777. 2019.
[22] M. Hoffman, D. M. Blei, C. Wang, and J. Paisley. Stochastic variational inference. JMI.R, 2013.

---

[23] J. Huggins, T. Campbell, and T. Broderick. Coresets for scalable Bayesian logistic regression. 2016.
[24] Kapoor, T. Karaletsos, and D. Bui. Variational auto-regressive Gaussian processes for continual learning, In ICML, 2021.
[25] Y. LeCun and C. Cortes. MNIST handwritten digit database. 2010. URL http://yam..eeuu com/ exdb/umiat/.
[26] J. Lorraine, Vicol, and D. Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In AISTATS, 2020.
[27] M. Lucic, O. Bachem, and A. Krause. Strong coresets for hard and soft Bregman clustering with applications to exponential family mixtures. In AISTATS, 2016.
[28] A. Maalouf, I. Jubran, and D. Feldman. Fast and accurate least-mean-squares solvers. In NeurIPS, 2019.
[29] D. Maclaurin, D. Duvenaud, and R. Adams. Gradient-based hyperparameter optimization through reversible learning. In ICML, 2015.
[30] D. Manousakas and C. Mascolo. 8-Cores: Robust large-scale Bayesian data summarization in the presence of outliers. WSDM, 2021.
[31] D. Manousakas, Xu, C. Mascolo, and T. Campbell. Bayesian pseudocoresets. NeurIPS, 2020.
[32] B. Mirzasoleiman, J. Bilmes, and J. Leskovec. Coresets for data-cfficient training of machine learning models. In ICML, 2020.
[33] B. Mirzasoleiman, K. Cao, and J. Leskovec. Coresets for robust training of deep neural networks against noisy labels. In NeurlPS, 2020.
[34] C. A. Naesseth, F. Lindsten, T. B. Schön, et al. Elements of sequential monte carlo. Foundations and Trends@ in Machine Learning, 12(3):37-392, 2019.
[35] R. M. Neal. Bayesian learning, for neural networks. PhD thesis, University of Toronto, 1995.
[36] C. Nguyen, Li, T.
Bui, and R. E. Turner. Variational continual learning. In ICLR, 2018.
[37] T. Nguyen, Z. Chen, and J. Lee. Dataset meta-leaming from kemel ridge-regression. In ICLR, 2021.
[38] Nguyen, Novak, L. Xiao, and J. Lee. Dataset distillation with infinitely wide convolutional networks. In NeurIPS, 2021.
[39] A. B. Owen. Monte Carlo theory, methods and examples. 2013.
[40] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. Vetooo, Z. Lin, A. Desmaison.
L. Antiga, and A. Lerer. Automatic differentiation in Pytorch. 2017.
[41] M. Paul, S. Ganguli, and G. K. Dziugaite. Deep learning on a data diet: Finding important examples early in training. In NeurIPS, 2021.
[42] R. Pinsler, J. Gordon, E. Nalisnick, and J. M. Hernández-Lobato Bayesian batch active learning as sparse subset approximation. In NeurIPS, 2019.
[43] R. Ranganath, S. Gerrish, and D. Blei. Black box variational inference. In AISTATS, 2014.
[44] R. Ranganath, D. Tran, J. Altosaar, and D. Blei. Operator variational inference. In NeurIPS,
2016.
[45] R. Ranganath, D. Tran, and D. Blei. Hierarchical variational models. ICML, 2016.
[46] H. Ritter and T. Karaletsos. TyXe: Pyro-based Bayesian neural nets for Pytorch, Proceedings of Machine Learning and Systems, 2022.
12

---

[47] H. Ritter, A. Botev, and D. Barber. Online structured Laplace approximations for overcoming catastrophic forgetting. NeurIPS, 2018.
[48] E. Snelson and Z. Ghahramani. Sparse Gaussian processes using pseudo-inputs. NeurIPS, 2006.
[49] I. Sucholutsky and M. Schonlau. Soft-label dataset distillation and text dataset distillation. In IJCNN, 2021.
[50] M. Titsias. Variational learning of inducing variables in sparse Gaussian processes. AISTATS, 2009.
[51] Tomezak and M. Welling. VAE with a VampPrior. In AISTATS, 2018.
[52] M. Toneva, A. Sordoni, R. T. des Combes, A. Trischler, Y. Bengio, and G. Gordon. An empirical study of example forgetting during deep neural network learning. In ICLR, 2019.
[53] D. Wang, D. Irani, and C. Pu. Evolutionary study of web spam: Webb spam corpus 201 versus webb spam corpus 2006. In ICCCNAW, 2012.
[54] Wang, J.-Y. Zhu, A. Torralba, and A. A. Efros. Dataset distillation. arXiv preprint, 2018.
[55] J. Yoon, D. Madaan, E. Yang, and S. J. Hwang. Online coreset selection for rehearsal- based continual learning. In ICLR, 2022.
[56] Zhang, R. Khanna, Kyrillidis, and S. Koyejo. Bayesian coresets: Revisiting the nonconvex optimization perspective. In AISTATS, 2021.
[57] B. Zhao, K. R. Mopuri, and H. Bilen. Dataset condensation with gradient matching. In ICLR, 2021.
[58] H. Zimmermann, H. Wu, B. Esmaeili, and J.-W. van de Meent. Nested variational inference. In NeurlPS, 2021.
13


Reviews:
[{"review_id": "xj4T_RG4EK0", "reviewer": "Reviewer_etFX", "summary": "The authors propose a variational inference method (BB PSVI) based on PSVI, which learns a variational approximation over model parameters given pseudo data that approximates the true posterior over model parameters given real data by maximizing a variational lower on the true data likelihood (ELBO-PSVI). \nA valid but generally intractable choice for the variational approximation is the true posterior over model-parameters given pseudo-data. To work around the tractability issues the authors propose to perform (adaptive) importance sampling which leverages another variational proposal over model-parameters by maximizing a lower bound on the pseudo-data likelihood (ELBO-IP).\nThe authors show that replacing the pseudo-data likelihood term in the PSVI-IS ELBO with the IP ELBO results again in a lower bound on the data likelihood. This variational lower bound (ELBO-PSVI-IS-BB) constitutes their final objective used to optimize the pseudo-observations. The variational approximation over model-parameters given pseudo-data can be modified to incorporate an weighting factor for individual pseudo-datapoint that can be learned jointly by maximizing the PSVI-IS-BB ELBO.\n", "questions": "The authors observe that existing approaches are limited by the data dimensionality while their approach allows for accurate approximate inference on the full data set regardless of dimensionality. While seemingly true for the presented models and datasets, intuitively the quality of the importance sampling estimate scales badly with dimensionality and is heavily reliant on good proposals for the position of the pseudo data points. The mechanisms to acquire positional proposals (pruning, incremental selection, or even potentially smarter ones like Bayesian optimization) seem to also scale badly with dimensionality as exponentially more positions have to be considered as the dimensionality increases. Is this an issue you'd expect to see in practice on different datasets? Why isn't this an issue for the webspam dataset (d=128)? Can you please elaborate a bit on that topic.\n\n**Some minor things I noticed:**                                                                                               \n- Line 64-66: the posterior distributions $p(\\theta \\mid u ,z, v)$ and $p(\\theta \\mid x, y)$ are over model parameters and not over the pseudo-dataset and dataset as stated in the text.\n", "limitations": "See questions regarding scaling with dimensionality", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Originality:**\nThe idea of using an importance sampling based approximation of the *pseudo posterior* to derive a black box version of PSVI is original and methodologically interesting and allows the use of PSVI for previously intractable black box models. As such I believe that this is a potentially interesting paper for both methodologically interested readers and application-oriented ML partitioners alike.\n\n**Quality:**\nThe methodology and technical argument are sound.\n\n**Clarity:**\nThe paper is well structured and clearly written. Given that the authors are using a nested variational inference step using self-normalized importance sampling, they should definitely cite recent work on Nested Variational Inference [1] in their related work subsection discussion Variational inference, importance weighting, and Monte Carlo objectives.\n\n[1] Zimmermann, H., Wu, H., Esmaeili, B., & van de Meent, J.-W. (2021, December). Nested Variational Inference. 35th Conference on Neural Information Processing Systems (NeurIPS 2021).\n\n**Significance:**\nThe presented results show significant improvements in accuracy and test log-likelihood for the presented models and datasets. I am however skeptical how the proposed approach performs on data with complex structure and high intrinsic dimensionality as the importance sampling approach is likely to fail here regardless of smart position-initialization and acquisition schemes. This trend seems to be already visible in the evaluation of the presented datasets (webspam, phishing, adult) where the effective sample size is non-trivial but rather low. I believe that additional discussion and an additional experiment on synthetic data which systematically evaluates the performance of the proposed method under increasing dimensionality would be great to gain further insights.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PdJP6Q3nrxX", "reviewer": "Reviewer_iNe6", "summary": "The paper proposes a general pseudo-data sparse black-box variational inference method. Using the idea of Bayesian pseud-ocoresets from [25], the paper proposes the ELBO formulation of the sparse pseudo-data variational inference. Importance sampling is used for the variational distribution. A nested variational inference objective is further developed over the pseudo points and the variational parameters. How to update or select the coreset data is also discussed. Experiments are carried out on two different models: logistic regression and Bayesian neural networks. ", "questions": "1. I would hope to get more explanation on why importance sampling is used instead of directly using ELBO defined by the approximate posterior ($q(\\theta|u,z;\\phi)$).\n2. Also it would be more convincing if there is some experiment that shows the promises of using this approach to achieve something that other methods cannot.", "limitations": "N.A.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The proposed black-box pseudo-coreset variational inference formulation is novel. And the derivation follows through.\n\nWeaknesses:\n\n- The experiment results on MNIST do not seem very competitive. With 10 data points in the coreset, the inducing point type of approaches can achieve 90+% accuracy. Also, there is some discrepancy with the reported accuracy of Dataset Condens.[44] in the original paper. Also, for Dataset Condens.[44], SLDD[37], and Dataset Distill.[42] the model uses 100 condensed data to train NN from scratch and achieve the reported accuracy. While for the proposed method, it is also using the labels of all training data for learning $\\phi$, $u$, and $z$.\n- The motivation behind importance sampling is not very clear to me. It does not exactly explain why not to use the ELBO defined by the approximate posterior ($q(\\theta|u,z;\\phi)$) directly.\n- For the regression tasks and MNIST, it would be more interesting to look at the pseudo-dataset learned and analyze if they make sense, just as what is done with the simulated datasets.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8AYp0n82vOp", "reviewer": "Reviewer_vKEG", "summary": "The authors present a new method to build coresets for inference tasks. That is, a method to build a \"smaller\" dataset so that the posterior over this new dataset is close to the true posterior. Their initial formulation follows the previously used approach finding the coreset so that the resulting distribution is closest (in KL divergence) to the true target. Since this in often intractable, the authors propose two approximations based on the use of a variational approximation (to approximate the coreset posterior) and importance sampling (to estimate expectations with respect to the coreset posterior).", "questions": "The most important point to be addressed is, in my opinion, W1. Simply put, convincingly show the benefits of using the resulting coreset over other naturally scalable approaches, such as plain VI. Addressing W2-W3 would be nice but not strictly necessary.\n\nA few additional questions (these do not affect my score):\n\n- Have you tried optimizing the full objective jointly, instead of using a nested optimization procedure?\n\n- In table 1 the random coreset approach appears to work significantly better than Sparse VI. Do you have any intuition regarding why this may be? (I appears that the incremental/greedy approaches perform poorly, since Sparse BBVI also works worse than the random approach).", "limitations": "Are the limitations of the work explicitly discussed somewhere?", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nS1: The method is presented in a clear way.\n\nS2: The method is novel and simple to implement.\n\nS3: The method appears to work well in the empirical evaluation presented.\n\nWeaknesses:\n\nW1: The proposed algorithm produces two outputs: (1) a coreset, and (2) a variational approximation (which approximates the coreset posterior, which in turn approximates the true target posterior). Regarding (2), a typical application of VI is as efficient (or even more efficient, since it is also compatible with subsampling and does not involve a nested optimization problem) and leads to similar or better results. Thus, the method's main benefit comes from producing a coreset (1), which may be used to enable other methods, such as efficient MCMC. But the paper does not explore this. I think it would be quite interesting to see how this method enables other algorithms (that would otherwise be very slow, e.g. MCMC) and how the resulting performance compares against plain VI.\n\nW2: The use of a simple variational distribution may hurt the quality of the coreset built. I think this would happen because of two \"forces\" that may be pushing in different directions. The first term of the objective (eq. 9) is pushing towards a coreset that leads to a coreset-posterior that's close to the true posterior, while the second term to a coreset that leads to a coreset-posterior that is not too far from the variational approximation $q(\\theta\\vert \\psi)$. A large increase in either of these terms may lead to an increase in the loss, and thus have to be balanced, possibly sacrificing coreset quality. Simply put, the use of a variational approximation leads to a tractable but biased approximation of the \"ideal\" objective to build the coreset from eq. 5, where the quality of the coreset may depend on the amount of bias. While the use of importance sampling as a debiasing tool as presented in the paper may ameliorate this issue, importance sampling is known to work poorly in high dimensions. I think that adding a more explicit discussion about this bias should be included somewhere in the paper.\n\nW3: Related to W2, another way to improve the bias issue would involve the use of a more powerful posterior (e.g. full-covariance Gaussian, flows). It would be interesting to see how the performance of the method improves in this case, and if the use of the resulting coreset leads to better performance when used with other methods, such as MCMC (related to W1).\n\nI think W1 is quite important, specially since VI is already giving an approximate solution. As I see it, a motivation for building coresets is that they may enable more accurate but otherwise more inefficient methods, such as MCMC. Thus, exploring whether the use of the resulting coresets with more accurate methods (e.g. MCMC) yield better results than the plain VI solution seems particularly relevant. (This is also related to W2, as the quality of the variational approximation may affect the quality of the coreset obtained.)\n\n\nA few additional comments:\n- Equation 4 defines $q(\\theta\\vert u,z) = p(\\theta\\vert u,z)$. But then in equation 5 both $p$ and $q$ are used. I think it would be better to just decide to use one of them, and stick to it throughout.\n- I think the paragraph between lines 97-102 may need some editing? It seems that the line starting from the textbf second is redundant with what is said before?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept