Title: General bounds on the quality of Bayesian coresets
Venue: NeurIPS 2024
Accepted: yes
Keywords: Bayesian

coreset

Kullback Leibler divergence

error bounds
Abstract: Bayesian coresets speed up posterior inference in the large-scale data regime by approximating the full-data log-likelihood function with a surrogate log-likelihood based on a small, weighted subset of the data. But while Bayesian coresets and methods for construction are applicable in a wide range of models, existing theoretical analysis of the posterior inferential error incurred by coreset approximations only apply in restrictive settings---i.e., exponential family models, or models with strong log-concavity and smoothness assumptions. This work presents general upper and lower bounds on the Kullback-Leibler (KL) divergence of coreset approximations that reflect the full range of applicability of Bayesian coresets. The lower bounds require only mild model assumptions typical of Bayesian asymptotic analyses, while the upper bounds require the log-likelihood functions to satisfy a generalized subexponentiality criterion that is weaker than conditions used in earlier work. The lower bounds are applied to obtain fundamental limitations on the quality of coreset approximations, and to provide a theoretical explanation for the previously-observed poor empirical performance of importance sampling-based construction methods. The upper bounds are used to analyze the performance of recent subsample-optimize methods. The flexibility of the theory is demonstrated in validation experiments involving multimodal, unidentifiable, heavy-tailed Bayesian posterior distributions.

General bounds on the quality of Bayesian coresets

Abstract

Bayesian coresets speed up posterior inference in the large-scale data regime by approximating the full-data log-likelihood function with a surrogate log-likelihood based on a small, weighted subset of the data. But while Bayesian coresets and methods for construction are applicable in a wide range of models, existing theoretical analysis of the posterior inferential error incurred by coreset approximations only apply in restrictive settings-i.e., exponential family models, or models with strong log-concavity and smoothness assumptions. This work presents general upper and lower bounds on the Kullback-Leibler (KL) divergence of coreset approximations that reflect the full range of applicability of Bayesian coresets. The lower bounds require only mild model assumptions typical of Bayesian asymptotic analyses, while the upper bounds require the log-likelihood functions to satisfy a generalized subexponentiality criterion that is weaker than conditions used in earlier work. The lower bounds are applied to obtain fundamental limitations on the quality of coreset approximations, and to provide a theoretical explanation for the previously-observed poor empirical performance of importance sampling-based construction methods. The upper bounds are used to analyze the performance of recent subsample-optimize methods. The flexibility of the theory is demonstrated in validation experiments involving multimodal, unidentifiable, heavy-tailed Bayesian posterior distributions.

1 Introduction

Large-scale data is now commonplace in scientific and commerical applications of Bayesian statistics. But despite its prevalence, and the corresponding wealth of research dedicated to scalable Bayesian inference, there are still suprisingly few general methods that provably provide inferential results, within some reasonable tolerated error, at a significant computational cost savings. Exact Markov chain Monte Carlo (MCMC) methods require many full passes over the data [1, Ch. 6-12, 2, Ch. 1112], limiting the utility of these methods when even a single pass is expensive. A wide range of MCMCmethods that access only a subset of data per iteration, e.g., via delayed acceptance [3-6], pseudomarginal or auxiliary variable methods [7-9], and basic subsampling [10-13], provide at most a minor improvement over full-data MCMC [14-16]. On the other hand, methods including carefully constructed log-likelihood function control variates can provide substantial gains [17-19]. However, black-box control variate constructions for large-scale data often rely on assumptions such as posterior density differentiability and unimodality that do not hold in many popular models, e.g., those with discrete variables or multimodality. See [15, 20] for a survey of scalable MCMC methods. Parametric approximations via variational inference [21] or the Laplace approximation [22, 23] can be obtained scalably using stochastic optimization methods, but existing general theoretical guarantees for these methods again typically rely on posterior normality assumptions [24, p. 141-144,25-30] (see [21, 31] for a review).

Although many existing methods rely on asymptotic normality or unimodality in the large-scale data regime, the problem of handling large-scale data in Bayesian inference does not fundamentally require this structure. Instead, one can more generally exploit redundancy in the data (i.e., the existence of good approximate sufficient statistics), which can be used to draw principled conclusions about a large data set based only on a small fraction of examples. Indeed, while approximate posterior normality often does not hold in models with latent discrete or combinatorial objects, weakly identifiable or unidentifiable parameters, persisting heavy tails, multimodality, etc., such models can and regularly do exhibit significant redundancy in the data that can be exploited for faster large-scale inference. Bayesian coresets [32]-which involve replacing the full dataset during inference with a sparse weighted subset-are based on this notion of exploiting data redundancy. Empirical studies have shown the existence of high-quality coreset posterior approximations constructed from a small fraction of the data, even in models that violate posterior normality assumptions and for which standard control variate techniques work poorly [33-37]. However, existing theoretical support for Bayesian coresets in the literature is limited. There exist no lower bounds on Bayesian coreset approximation error, and while upper bounds do exist, they currently impose restrictive assumptions. In particular, the best available theoretical upper bounds to date apply to exponential family models [36, 38] and models with strongly log-concave and locally smooth log-densities [37].

This article presents new theoretical techniques and results regarding the quality of Bayesian coreset approximations. The main results are two general large-data asymptotic lower bounds on the KL divergence (Theorems 3.3 and 3.5), as well as a general upper bound on the KL divergence (Theorem 5.3) under the assumption that the log-likelihoods satisfy a multivariate generalization of subexponentiality (Definition 5.2). The main general results in this paper lead to various novel insights about specific Bayesian coreset construction methods. Under mild assumptions,

common importance-weighted coreset constructions (e.g. [32]) require a coreset size M proportional to the dataset size N (Corollary 4.1), even with post-hoc optimal weight scaling (Corollary 4.2), and thus yield a negligible improvement over full-data inference;

any construction algorithm requires a coreset size M > d when the log-likelihood function is determined by d parameters locally around a point of concentration (Corollary 4.3);

subsample-optimize coreset construction algorithms (e.g. [36-39]) achieve an asymptotically bounded error with a coreset size polylog N in a wide variety of models (Corollary 6.1).

The paper includes empirical validation of the main theoretical claims on two models that violate common assumptions made in the literature: a multimodal, unidentifiable Cauchy location model with a heavy-tailed prior, and an unidentifiable logistic regression model with a heavy-tailed prior and persisting posterior heavy tails. Experiments were performed on a computer with an Intel Core i7-8700K and 32GB of RAM. Proofs of all theoretical results may be found in Appendix A.

Notation. We use standard asymptotic growth symbols O, Ω , Θ , o, ω (see, e.g., [40, Sec. 3.3]), and their probabilistic variants O p , Ω p , Θ p , o p , ω p (see, e.g., [24, Sec. 2.2]). We use the same symbol to denote a measure π and its density π ( · ) with respect to a specified dominating measure. We also regularly suppress integration variables and differential symbols in integrals throughout for notational brevity when these are clear from context; for example, ∫ π exp( ℓ ) is shorthand for ∫ π (d θ ) exp( ℓ ( θ )) . Finally, the pushforward of a measure π by a map η is denoted simply ηπ .

2 Background

Define a target probability distribution π on a space Θ comprised of a sum of N potentials ℓ n : Θ → R , n = 1 , . . . , N and a base distribution π 0 (d θ ) ,

where the normalization constant Z is not known. In the Bayesian context, this distribution corresponds to a Bayesian posterior distribution for a statistical model with prior π 0 and conditionally i.i.d. data X n , where ℓ n ( θ ) = log p ( X n | θ ) . The goal is to compute or approximate expectations under π ; but the likelihood ℓ (and its gradient) becomes expensive to evaluate when N is large. To

avoid this cost, Bayesian coresets [32-37] involve replacing the target with a surrogate density

where w ∈ R N , w ≥ 0 are a set of weights, and Z ( w ) is the new normalizing constant. If w has at most M ≪ N nonzeros, the O ( M ) cost of evaluating ∑ n w n ℓ n (and its gradient) is a significant improvement upon the original O ( N ) cost. In this work, the problem of coreset construction is formulated in the data-asymptotic limit; a coreset construction method should

run in o ( N ) time and memory (or at most O ( N ) with a small leading constant),

produce a small coreset of size M = o ( N ) ,

produce a coreset with O (1) posterior forward/reverse KL divergence as N →∞ .

These three desiderata ensure that the effort spent constructing and sampling from the coreset posterior is worthwhile: the coreset provides a meaningful reduction in computational cost compared with standard Markov chain Monte Carlo algorithms, and has a bounded approximation error.

3 Lower bounds on approximation error

This section presents lower bounds on the KL divergence of coreset approximations for general models and data generating processes. The first key steps in the analysis are to write all expectations in terms of distributions that do not depend on w , and to remove the difficult-to-control influence of the tails of π and π w by restricting certain integrals to some small subset B ⊆ Θ of the parameter space. Lemma 3.1, the key theoretical tool used in this section, achieves both of these two goals; note that the result has no major assumptions and applies generally in any setting that a Bayesian coreset can be used. For convenience, define

and the decreasing, nonnegative function f : R + → R + ,

Lemma 3.1 (Basic KL Lower Bound) . For all measurable B ⊆ Θ and coreset weights w ,

Note that while the integrals in the fraction denominator in J B ( w ) range over the whole Θ space, a further lower bound on KL( w ) can be obtained by restricting their domains arbitrarily. Also, crucially, the bound in Lemma 3.1 does not depend on π w ( B c ) , which would be difficult to analyze without detailed knowledge of the tail behaviour of π w as a function of the coreset weights w . Although the bound in Lemma 3.1 applies generally, it is most useful when B is small (so that simple local approximations of ℓ and ℓ w can be used), π concentrates on B (so that π ( B c ) ≈ 0 ), and π and π w are very different when restricted to B ; the behaviour of the bound in this case is roughly (see the proof in Appendix A) f ( J B ( w )) ≈ -log(1 -TV( π, π w )) . Finally, note that Lemma 3.1 remains valid if one replaces ℓ w with ℓ w -c and ℓ with ℓ -c ′ for any constants c, c ′ that do not depend on θ but may depend on the data and coreset weights w .

For the remainder of this section, consider the setting where Θ is a measurable subset of R d for some d ∈ N , fix some θ 0 ∈ Θ , and assume each ℓ n is differentiable in a neighbourhood of θ 0 . Let

where

Theorems 3.3 and 3.5 characterize KL divergence lower bounds in terms of the sum of the coreset weights w and the log-likelihood gradients g, g w . Intuitively for the full data set where all w n = 1 and w = N , and an i.i.d. data generating process from the likelihood with parameter θ 0 , the central limit theorem asserts under mild conditions that g w /w p → 0 at a rate of N -1 / 2 . Theorems 3.3 and 3.5 below provide KL lower bounds when the coreset construction algorithm does not match this behavior. In particular, Theorem 3.3 provides results that are useful when g w /w p → 0 occurs reasonably quickly but slower than N -1 / 2 , while Theorem 3.5 strengthens the conclusion when g w /w p → 0 very slowly or not at all. The major benefit of Theorems 3.3 and 3.5 for analyzing coreset construction methods is that they reduce the problem of analyzing posterior KL divergence to the much easier problem of analyzing the 2-norm ∥ · ∥ 2 of a weighted sum of random vectors in R d .

Consider a sequence r → 0 as N →∞ , and for a fixed matrix H ≻ 0 let

be a sequence of neighbourhoods around θ 0 ; these will appear in Assumptions 3.2 and 3.4 and Theorems 3.3 and 3.5 below. Note that throughout, all asymptotics will be taken as N →∞ , and various sequences (e.g., r and B ) are implicitly indexed by N . To simplify notation, this dependence is suppressed. Assumption 3.2 makes some weak assumptions about the model and data generating process: it intuitively asserts that the potential functions are sufficiently smooth around θ 0 , that r → 0 slowly, and that π concentrates at θ 0 at a usual rate. Note that Assumption 3.2 does not assume data are generated i.i.d. and places no conditions on the coreset construction algorithm.

Assumption 3.2. π 0 has a density with respect to the Lebesgue measure, π 0 ( θ 0 ) > 0 , each ℓ n ( θ ) and π 0 ( θ ) are twice differentiable in B for sufficiently large N , and

Two additional assumptions related to the coreset construction algorithm-namely, that it works well enough that 1 w ∑ n w n ∇ 2 ℓ n ( θ ) p → H and g w /w p → 0 at a rate faster than r → 0 -lead to asymptotic lower bounds on the best possible quality of coresets produced by the algorithm, as well as lower bounds even after optimal post-hoc scaling of the weights.

Theorem 3.3. Suppose Assumption 3.2 holds. If

then

Theorem 3.3 is restricted to the case where the coreset algorithm is performing reasonably well. Theorem 3.5 extends the bounds to the case where the algorithm is performing poorly, in the sense that it is unable to make g w w p → 0 at a rate faster than r → 0 (or perhaps g w w does not converge to 0 at all). In order to draw conclusions in this setting, we need a weak global assumption on the potential functions. A function f : Θ → R is L -smooth below at θ 0 if

Note that L -smoothness below is weaker than Lipschitz smoothness and does not imply concavity; Eq. (1) restricts the growth of the function only in the negative direction, and only when the expansion is taken at θ 0 . Assumption 3.4 asserts that the potential functions are smooth below.

Assumption 3.4. There exist L 0 , . . . , L N , L > 0 such that log π 0 is L 2 0 -smooth below at θ 0 , for each n ∈ [ N ] ℓ n is L 2 n -smooth below at θ 0 , and 1 N ∑ N n =1 L 2 n p → L 2 .

Theorem 3.5 uses Assumptions 3.2 and 3.4 and additional assumptions related to the coreset construction algorithm to obtain lower bounds in a setting that relaxes the 'performance' conditions in Theorem 3.3: -1 w ∑ n w n ∇ 2 ℓ n ( θ ) no longer needs to converge to H in probability, and g w /w can converge to 0 slowly or not at all.

Theorem 3.5. Suppose Assumptions 3.2 and 3.4 hold. If there exist α, β > 0 such that

then

An important final note in this section is that while Theorems 3.3 and 3.5, as stated, require choosing Θ to be some measurable subset of R d and that the posterior π concentrates around some point of interest θ 0 ∈ R d , these results can be generalized to a wider class of models and spaces. In particular, Corollary 3.6 demonstrates that if Θ is arbitrary, but the potential functions ℓ n only depend on θ through some other function η : Θ → R d , that the conclusions of Theorems 3.3 and 3.5 still hold.

Corollary 3.6. Suppose Θ is an arbitrary measurable space, and the potential functions take the form ℓ n ( η ( θ )) for some measurable function η : Θ → R d . Then if the assumptions of Theorems 3.3 and 3.5 hold for potentials ( ℓ n ) N n =1 as functions on R d and pushforward prior ηπ 0 on R d , the stated lower bounds also hold for min { KL( π || π w ) , KL( π w || π ) } .

4 Lower bound applications

In this section, the general theoretical results from Section 3 are applied to specific algorithms, Bayesian models, and data generating processes to explain previously observed empirical behaviour of coreset construction, as well as to place fundamental limits on the necessary size of coresets. Consider a setting where the data X n arise as an i.i.d. sequence drawn from some probability distribution ν , ℓ n ( η ( θ )) = log p ( X n | η ( θ )) for η : Θ → R d , η 0 = η ( θ 0 ) , and the following technical criteria hold (where E denotes expectation under the data generating process):

(A4) ηπ 0 is twice differentiable a neighbourhood of η 0 , and π ( η 0 ) > 0 .

These conditions apply to a wide range of models, e.g., an unidentifiable, multimodal location model posterior with heavy tails on Θ = R , where the Bayesian model is specified by

and the data are generated from the likelihood with parameter θ 0 = 5 , and an unidentifiable logistic regression posterior with heavy tails on R 2 , where the Bayesian model is specified by

the covariates are generated via X n iid ∼ Unif( { x ∈ R 2 : ∥ x ∥ 2 ≤ 1 } ) , and the observations Y n are generated from the likelihood with parameter θ 0 = [1 6] T . See Proposition A.6 in Appendix A for the verification of (A1-5) for these two models. Example posterior log-densities for these models are displayed in Fig. 1.

(a)

(b)

250

-100

20

500

750

150

15

1000

density

10

1250

-200

1500

posterior

250

Parameter

5

1750

0

2000

Log

5

693

4

2

Parameter value (blue slice)

0

2

6

8

10

-350

10

694

400

15

suap

As

695

450

-20

696

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

-20

-15-10

-5

0

5

10

15

20

Parameter

Parameter 2

697

698

10.0

7.5

5.02.5

Parameter value (orange slice)

7.5

10.0

Algorithm 1 Importance-weighted coreset construction

Compute probabilities ( p n ) N n =1 (may depend on the data and model)

Draw I 1 , . . . , I M iid ∼ Categorical( p 1 , . . . , p N )

For each n , set w n = 1 Mp n ∑ M m =1 1 [ I m = n ] .

return ( w n ) N n =1

Algorithm 2 Scaled importance-weighted coreset construction

Obtain coreset weights ( w n ) N n =1 via Algorithm 1 Compute α ⋆ = arg min α ≥ 0 KL( π αw || π ) return ( α ⋆ w n ) N n =1

4.1 Minimum coreset size for importance-weighted coresets

A popular algorithm for coreset construction that has appeared in a wide variety of domains-e.g., Bayesian inference [32, 33, Section 4.1], frequentist inference (e.g., [41-45]), and optimization (see [46] for a recent survey)-involves subsampling of the data followed by an importance-weighting correction. The pseudocode is given in Algorithm 1. Note that E [ w n ] = 1 , and so E [ ℓ w ] = ℓ ; the coreset potential is an unbiased estimate of the exact potential. The advantage of this method is that it is straightforward and computationally efficient. If the sampling probabilities are uniform p n = 1 / N , then Algorithm 1 constructs a coreset in O ( M ) time and O ( M ) memory. Nonuniform probabilities p n require Ω( N ) time, as they require a pass over all N data points to compute each p n [32, 42] followed by sampling the coreset, e.g., via an alias table [47, 48]. However, empirical results produced by this methodology have generally been underwhelming, even with carefully chosen sampling probabilities; see, e.g., Figure 2 of [32].

Corollary 4.1 explains these poor results: Bayesian coresets constructed via Algorithm 1 must satisfy M ∝ N in order to maintain a bounded KL( w ) in the data-asymptotic limit. In other words, such coresets do not satisfy the desiderata in Section 2. The only restriction is that there exist constants c, C > 0 such that for all N ∈ N , the sampling probabilities ( p n ) N n =1 satisfy

The lower threshold ensures that the variance of the importance-weighted log-likelihood is not too large, while the upper threshold ensures sufficient diversity in the draws from subsampling. The condition in Eq. (4) is not a major restriction, in the sense that performance should deteriorate even further when it does not hold. The ( p n ) N n =1 may otherwise depend arbitrarily on the data and model.

Corollary 4.1. Given (A1-6), M →∞ , and M = o ( N ) , coresets produced by Algorithm 1 satisfy

The intuition behind Corollary 4.1 is that both the true posterior and the importance-weighted coreset posterior are asymptotically approximately normal with variance ∝ 1 /N as N →∞ ; however, the coreset posterior mean is roughly ∝ M -1 / 2 away from the posterior mean, because the subsample is of size M . The KL divergence between two Gaussians is lower-bounded by the inverse variance times the mean difference squared, yielding ≈ N/M as in Eq. (5).

Given the intuition that the coreset posterior mean is far from the posterior mean relative to their variances, it is worth asking whether one can apply a small amount of effort to 'correct' the importance-weighted coreset by scaling the weights (and hence the variance) down, as shown in Algorithm 2. Unfortunately, Corollary 4.2 demonstrates that even with optimal scaling, M ∝ N is still required in order to maintain a bounded KL divergence as N →∞ .

Corollary 4.2. Given (A1-6), M →∞ , and M = o ( N ) , coresets produced by Algorithm 1 satisfy

Fig. 2 provides empirical confirmation of Corollaries 4.1 and 4.2 on the Cauchy location and logistic regression models in Eqs. (2) and (3). In particular, these figures show that the empirical rates of growth of KL as a function of N closely matches Ω p ( N M ) for importance-weighted coresets, and Ω p (log N M ) for the same with post-hoc scaling, for a wide range of coreset sizes M ∈ { log N, √ N , 1 / 2 N } . Thus, importance weighted coreset construction methods do not satisfy the desiderata in Section 2 for a wide range of models, and alternate methods should be considered.

4.2 Minimum coreset size for any coreset construction

This section extends the minimum coreset size results from importance-weighted schemes to any coreset construction algorithm. In particular, Corollary 4.3 shows that under (A7)-a strengthening of (A3) and Assumption 3.4-and (A8)-which asserts that ∇ ℓ 1 ( η 0 ) , . . . , ∇ ℓ M ( η 0 ) are linearly independent a.s. and satisfy a technical moment condition-at least d coreset points are required to keep the KL divergence bounded as N →∞ .

(A7) Assumption 3.4 holds and there exists γ > 0 such that for all sufficiently large N ∈ N ,

(A8) For all coreset sizes M < d , there exists a δ > 0 such that

Corollary 4.3. For a fixed coreset size M < d , given (A1-5,7,8),

5 Upper bounds on approximation error

This section presents upper bounds on the KL divergence of coreset approximations. As in Section 3, the first step is to write all expectations in terms of distributions that do not depend on w . Lemma 5.1 does so without imposing any major assumptions; the result again applies generally in any setting that a Bayesian coreset can be used. For convenience, define

Lemma 5.1 (Basic KL Upper Bound) . For all coreset weights w ,

The upper bound in Lemma 5.1 is nonvacuous (i.e., finite) as long as there exists a α > 1 such that the α R´ enyi divergence D α ( π w || π ) [49, p. 3799] is finite. Note that as in Lemma 3.1, the bound in

M=sqrt(N)

M=sqrt(N)

M=N/2

M=N/2

M=log N

M=log N

102

102

Minimum KL

Minimum KL

101

101

100

100

102

103

104

102

103

104

Dataset size (N)

Dataset size (N)

3.5

M=sqrt(N)

5

M=sqrt(N)

M=N/2

M=N/2

十

M=log N

M=log N

3.0

2.5

Minimum KL

Minimum KL

2.0

1.5

1.0

0.5

102

103

104

102

103

104

Dataset size (N)

Dataset size (N)

Lemma 5.1 remains valid if one replaces ℓ w with ℓ w -c and ℓ with ℓ -c ′ for any constants c, c ′ that do not depend on θ but may depend on the coreset weights w and data.

More practical bounds necessitate an assumption about the behaviour of the potentials ( ℓ n ) N n =1 . Definition 5.2 below asserts that the multivariate moment generating function of ( ℓ n ) N n =1 is bounded when the vector is close to 0. This definition is a generalization of the usual definition of subexponentiality for the univariate setting (e.g., [50, Sec. 2.7]). Theorem 5.3 subsequently shows that Definition 5.2 is sufficient to obtain simple bounds on KL .

Definition 5.2. For A ∈ R N × N , A ⪰ 0 , and monotone function h : R + → R + such that lim x → 0 h ( x ) = h (0) = 0 , the potentials ( ℓ n ) N n =1 are ( h, A ) -subexponential if

Theorem 5.3. If the potentials ( ℓ n ) N n =1 are ( h, A ) -subexponential, then

Definition 5.2, the key assumption in Theorem 5.3, is satisfied by a wide range of models when choosing h ( x ) = x and A ∝ Cov π ( ( ℓ n ) N n =1 ) , as demonstrated by Proposition 5.4. Because this case applies widely, let A -subexponential be shorthand for ( h, A ) -subexponentiality with h ( x ) = x .

∫ exp( ¯

Proposition 5.4. If for all w in a ball centered at the origin, π ℓ w ) < ∞ , then there exists β > 0 such that the potentials ( ℓ n ) N n =1 are β Cov π ( ( ℓ n ) N n =1 ) -subexponential.

Algorithm 3 Subsample-optimize coreset construction

̸

In other words, intuitively, if a coreset construction algorithm produces weights such that Var π ( ¯ ℓ w -¯ ℓ ) is small, then KL( w ) is also small. That being said, the generality of Definition 5.2 to allow arbitrary h, A is still helpful in obtaining upper bounds in specific cases; see, e.g., Propositions A.1 and A.2.

6 Upper bound application: subsample-optimize coresets

A strategy to construct Bayesian coresets that has recently emerged in the literature, shown in Algorithm 3, is to first subsample the data to select M data points, and subsequently optimize the weights for those selected data points [36-38]. The subsampling step serves to pick a reasonably flexible basis of log-likelihood functions for coreset approximation, and avoids the slow greedy selection routines from earlier work [33-35]. The optimization step tunes the weights for the selected basis, avoiding the poor approximations of importance-weighting methods. Indeed, Algorithm 3 creates exact coresets π w ⋆ = π with high probability in Gaussian location models [36, Prop. 3.1] and finite-dimensional exponential family models [37, Thm. 4.1], and near-exact coresets with high probability in strongly log-concave models [37, Thm. 4.2] and Bayesian linear regression [38, Prop. 3].

Corollary 6.1 generalizes these results substantially, and demonstrates that coresets of size M = O (polylog( N )) produced by the subsample-optimize method in Algorithm 3 maintain a bounded KL divergence as N → ∞ . Two key assumptions are subexponentiality of the potentials and a polynomial (in N ) growth of Var π ( ℓ ( θ )) ; these conditions are not stringent and should hold for a wide range of Bayesian models and i.i.d. data generating processes. The last key assumption in Eq. (6) is that a randomly-chosen potential function ℓ I , I ∼ Categorical( p 1 , . . . , p N ) (with probabilities as in Algorithm 3) is well-aligned with the residual coreset error function. Similar alignment conditions have appeared in past results for more restrictive settings (see, e.g., J ( δ ) in [37, Thm. 4.1]).

Corollary 6.1. Suppose there exist β, α > 0 and 0 ≤ ρ, ϵ < 1 such that the potential functions ( ℓ n ) N n =1 are β Cov π (( ℓ n ) N n =1 ) -subexponential with probability increasing to 1 as N → ∞ , Var π ( ℓ ( θ )) = O p ( N α ) , M = (log N ) 1 1 -ρ , and

Then Algorithm 3 produces a coreset with KL( w ) = O p (1) as N →∞ .

Fig. 3 confirms that subsample-optimize coreset construction methods applied to the logistic regression and Cauchy location models in Eqs. (2) and (3) (which both violate the conditions of past upper bounds in the literature) are able to provide high-quality posterior approximations for very small coresets-in this case, M ∝ log N .

7 Conclusions

This article presented new general lower and upper bounds on the quality of Bayesian coreset approximations, as measured by the KL divergence. These results were used to draw novel conclusions regarding importance-weighted and subsample-optimize coreset methods, which align with simulation experiments on two synthetic models that violate the assumptions of past theoretical results. Avenues for future work include general bounds on the subexponentiality constant β in Proposition 5.4, as well as the alignment probability in Eq. (6), in the setting of Bayesian models with i.i.d. data generating processes. A limitation of this work is that both quantities currently require case-by-case analysis.

M=5 + 2log(N)

M=5 + 2log(N)

10-1

10-2

10-2

Maximum I

103

10-3

10-4

10-4

10-5

102

103

104

102

103

104

Dataset size (N)

Dataset size (N)

Acknowledgments and Disclosure of Funding

The author gratefully acknowledges the support of an NSERC Discovery Grant (RGPIN-2019-03962).

References

Christian Robert and George Casella. Monte Carlo Statistical Methods . Springer, 2nd edition, 2004.

Andrew Gelman, John Carlin, Hal Stern, David Dunson, Aki Vehtari, and Donald Rubin. Bayesian data analysis . CRC Press, 3rd edition, 2013.

J. Andr´ es Christen and Colin Fox. Markov chain Monte Carlo using an approximation. Journal of Computational and Graphical Statistics , 14(4):795-810, 2005.

Marco Banterle, Clara Grazian, Anthony Lee, and Christian P. Robert. Accelerating Metropolis-Hastings algorithms by delayed acceptance. Foundations of Data Science , 1(2):103-128, 2019.

Richard Payne and Bani Mallick. Bayesian big data classification: a review with complements. arXiv:1411.5653 , 2014.

Chris Sherlock, Andrew Golightly, and Daniel Henderson. Adaptive, delayed-acceptance MCMC for targets with expensive likelihoods. Journal of Computational and Graphical Statistics , 26(2):434-444, 2017.

Arnaud Doucet, Michael Pitt, George Deligiannidis, and Robert Kohn. Efficient implementation of Markov chain Monte Carlo when using an unbiased likelihood estimator. Biometrika , 102(2):295-313, 2015.

Dougal Maclaurin and Ryan Adams. Firefly Monte Carlo: exact MCMC with subsets of data. In Conference on Uncertainty in Artificial Intelligence , 2014.

Matias Quiroz, Minh-Ngoc Tran, Mattias Villani, Robert Kohn, and Khue-Dung Dang. The block-Poisson estimator for optimally tuned exact subsampling MCMC. Journal of Computational and Graphical Statistics , 30(4):877-888, 2021.

Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient Langevin dynamics. In International Conference on Machine Learning , 2011.

Sungjin Ahn, Anoop Korattikara, and Max Welling. Bayesian posterior sampling via stochastic gradient Fisher scoring. In International Conference on Machine Learning , 2012.

Anoop Korattikara, Yutian Chen, and Max Welling. Austerity in MCMC land: cutting the MetropolisHastings budget. In International Conference on Machine Learning , 2014.

Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient Hamiltonian Monte Carlo. In International Conference on Machine Learning , 2015.

James Johndrow, Natesh Pillai, and Aaron Smith. No free lunch for approximate MCMC. arXiv:2010.12514 , 2020.

R´ emi Bardenet, Arnaud Doucet, and Chris Holmes. On Markov chain Monte Carlo methods for tall data. Journal of Machine Learning Research , 18:1-43, 2017.

Tigran Nagapetyan, Andrew Duncan, Leonard Hasenclever, Sebastian Vollmer, Lukasz Szpruch, and Konstantinos Zygalakis. The true cost of stochastic gradient Langevin dynamics. arXiv:1706.02692 , 2017.

Jack Baker, Paul Fearnhead, Emily Fox, and Christopher Nemeth. Control variates for stochastic gradient MCMC. Statistics and Computing , 29:599-615, 2019.

Christopher Nemeth and Paul Fearnhead. Stochastic gradient Markov Chain Monte Carlo. Journal of the American Statistical Association , 116(533):433-450, 2021.

Matias Quiroz, Robert Kohn, Mattias Villani, and Minh-Ngoc Tran. Speeding up MCMC by efficient data subsampling. Journal of the American Statistical Association , 114(526):831-843, 2019.

Matias Quiroz, Robert Kohn, and Khue-Dung Dang. Subsampling MCMC-an introduction for the survey statistician. Sankhya: The Indian Journal of Statistics , 80-A:S33-S69, 2018.

David Blei, Alp Kucukelbir, and Jon McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association , 112(518):859-877, 2017.

Zhenming Shun and Peter McCullagh. Laplace approximation of high dimensional integrals. Journal of the Royal Statistical Society: Series B , 57(4):749-760, 1995.

Peter Hall, Tung Pham, Matt Wand, and Shen S.J. Wang. Asymptotic normality and valid inference for gaussian variational approximation. The Annals of Statistics , 39(5):2502-2532, 2011.

Aad van der Vaart. Asymptotic Statistics . Cambridge University Press, 2000.

Yixin Wang and David Blei. Frequentist consistency of variational Bayes. Journal of the American Statistical Association , 114(527):1147-1161, 2018.

Badr-Eddine Ch´ erief-Abdellatif and Pierre Alquier. Consistency of variational Bayes inference for estimation and model selection in mixtures. Electronic Journal of Statistics , 12:2995-3035, 2018.

Yun Yang, Debdeep Pati, and Anirban Bhattacharya. α -variational inference with statistical guarantees. The Annals of Statistics , 2018.

Pierre Alquier and James Ridgway. Concentration of tempered posteriors and of their variational approximations. The Annals of Statistics , 48(3):1475-1497, 2020.

Zuheng Xu and Trevor Campbell. The computational asymptotics of Gaussian variational inference and the Laplace approximation. Statistics and Computing , 32(63), 2022.

Jeffrey Miller. Asymptotic normality, concentration, and coverage of generalized posteriors. Journal of Machine Learning Research , 22:1-53, 2021.

Cheng Zhang, Judith B¨ utepage, Hedvig Kjellstr¨ om, and Stephan Mandt. Advances in variational inference. IEEE Transactions on Pattern Analysis and Machine Intelligence , 41(8):2008-2026, 2019.

Jonathan Huggins, Trevor Campbell, and Tamara Broderick. Coresets for scalable Bayesian logistic regression. In Advances in Neural Information Processing Systems , 2016.

Trevor Campbell and Tamara Broderick. Automated scalable Bayesian inference via Hilbert coresets. Journal of Machine Learning Research , 20(15):1-38, 2019.

Trevor Campbell and Boyan Beronov. Sparse variational inference: Bayesian coresets from scratch. In Advances in Neural Information Processing Systems , 2019.

Trevor Campbell and Tamara Broderick. Bayesian coreset construction via greedy iterative geodesic ascent. In International Conference on Machine Learning , 2018.

Naitong Chen, Zuheng Xu, and Trevor Campbell. Bayesian inference via sparse Hamiltonian flows. In Advances in Neural Information Processing Systems , 2022.

Cian Naik, Judith Rousseau, and Trevor Campbell. Fast Bayesian coresets via subsampling and quasiNewton refinement. In Advances in Neural Information Processing Systems , 2022.

Martin Jankowiak and Du Phan. Surrogate likelihoods for variational annealed importance sampling. In International Conference on Machine Learning , 2022.

Naitong Chen and Trevor Campbell. Coreset Markov chain Monte Carlo. In International Conference on Artificial Intelligence and Statistics , 2024.

Thomas Cormen, Charles Leiserson, Ronald Rivest, and Clifford Stein. Introduction to Algorithms . The MIT Press, 4 th edition, 2022.

Ping Ma, Michael Mahoney, and Bin Yu. A statistical perspective on algorithmic leveraging. Journal of Machine Learning Research , 16:861-911, 2015.

HaiYing Wang, Rong Zhu, and Ping Ma. Optimal subsampling for large sample logistic regression. Journal of the American Statistical Association , 113(522):829-844, 2018.

HaiYing Wang. More efficient estimation for logistic regression with optimal subsamples. Journal of Machine Learning Research , 20:1-59, 2019.

Mingyao Ai, Jun Yu, Huiming Zhang, and HaiYing Wang. Optimal subsampling algorithms for big data regressions. Statistica Sinica , 31(2):749-772, 2021.

HaiYing Wang and Yanyuan Ma. Optimal subsampling for quantile regression in big data. Biometrika , 108(1):99-112, 2021.

Dan Feldman. Introduction to core-sets: an updated survey. arXiv:2011.09384 , 2020.

Alastair Walker. New fast method for generating discrete random numbers with arbitrary frequency distributions. Electronics Letters , 10(8):127-128, 1974.

Alastair Walker. An efficient method for generating discrete random variables with general distributions. ACM Transactions on Mathematical Software , 3(3):253-256, 1977.

Tim van Erven and Peter Harr¨ emos. R´ enyi divergence and Kullback-Leibler divergence. IEEE Transactions on Information Theory , 60(7):3797-3820, 2014.

Roman Vershynin. High-dimensional probability: an introduction with applications in data science . Cambridge University Press, 2020.

Igor Vajda. Note on discrimination information and variation. IEEE Transactions on Information Theory , 16(6):771-773, 1970.

David Pollard. A user's guide to probability theory . Cambridge series in statistical and probabilistic mathematics. Cambridge University Press, 7 th edition, 2002.

Robert Keener. Theoretical statistics: topics for a core course . Springer, 2010.

Andre Bulinski. Conditional central limit theorem. Theory of Probability & its Applications , 61(4):613-631, 2017.

Lorraine Schwartz. On Bayes procedures. Z. Wahrscheinlichkeitstheorie und Verwandte Gebiete , 4:10-26, 1965.

A Proofs

Proof of Lemma 3.1. By Vajda's inequality [51],

The bound is monotone increasing in TV( π, π w ) ; therefore because the squared Hellinger distance satisfies the inequality [52, p. 61],

we have that

We substitute the value of the squared Hellinger distance to find that

Note that ∫ √ ππ w ≤ 1 , so

The bound is monotone decreasing in ∫ √ ππ w , so we require an upper bound on ∫ √ ππ w . To obtain the required bound, we split the integral into two parts-one on the set B , and the other on B c -and then use the Cauchy-Schwarz inequality to bound the part on B c . Note that by definition π and π w are mutually dominating, so the density ratio π w /π is well-defined and measurable.

The result follows.

Proof of Lemma 5.1. We first consider the forward KL divergence. By definition,

Since the KL is positive, for λ > 0 ,

̸

by Jensen's inequality. Next we consider the reverse KL divergence. For any λ = 0 ,

By Jensen's inequality, for λ > 0 ,

This is the same bound as in the forward KL divergence case. Since the bound applies for all λ > 0 , we can take the infimum.

Proof of Theorem 3.3. By replacing the integrals over the whole space Θ in the denominator of J B ( w ) in Lemma 3.1 with integrals over the subset B ,

So to obtain the stated lower bound on the KL divergence, we require an upper bound on log ∫ B π 0 exp((1 / 2)( ℓ + ℓ w )) , and lower bounds on log ∫ B π 0 exp( ℓ ) and log ∫ B π 0 exp( ℓ w ) . By Taylor's theorem, Assumption 3.2, and the assumption on ∇ 2 ℓ w ( θ ) , for all θ ∈ B ,

We shift the exponential arguments in G B ( w ) by (1 / 2)( ℓ ( θ 0 ) + ℓ w ( θ 0 )) , note that π 0 is continuous and positive around θ 0 , and and apply the Taylor expansions in Eq. (7) to obtain an upper bound on the first term:

where ( ∼ 1) denotes a quantity that converges in probability to 1 as N →∞ . We can transform variables to x = C T ( θ -θ 0 ) , where H = CC T is the Cholesky factorization of H , and subsequently complete the square:

We can obtain lower bounds on the other two terms using a similar technique:

It remains to analyze the three log ∫ . . . terms. We bound the integral term in Eq. (8) with the integral over the whole space:

For the integral term in Eq. (9), note that since Nr 2 = ω (1) and ∥ C -1 g/N ∥ = O p ( N -1 / 2 ) , we have

For the integral term in Eq. (10), we consider two cases: one where w is large, and one where it is small. First assume wr 2 > 8 d log 2 ; then by a similar technique as used in the first lower bound, since ∥ C -1 g w /w ∥ = o p ( r ) ,

When wr 2 ≤ 8 d log 2 , we transform variables y = x/r to find that since ∥ C -1 g w /w ∥ = o p ( r ) ,

Therefore regardless of the value of w ,

So therefore combining all previous results,

We now consider the minimum over α ≥ 0 . Since neither O p (1) or Ω p (1) above depends on w , we have that

On the 1 /r 2 branch of the objective function, the derivative in α is always positive, and hence the minimum occurs at α = 0 , and so

On the αw branch of the objective function,

For a, b > 0 and x ≥ 0 , the function ax -b log x is convex in x with minimum at x ⋆ = b/a , and so

By assumption, ∥ g N ∥ = o p ( r ) and ∥ g w w ∥ = o p ( r ) , and hence the αw branch has the asymptotic minimum:

Proof of Theorem 3.5. By Lemma 3.1,

Note that G B in this proof is subtly different from the G B used in the proof of Theorem 3.3; the latter two integrals are over the whole space (directly from Lemma 3.1), rather than B . We shift the exponential arguments in G B ( w ) by (1 / 2)( ℓ ( θ 0 ) + ℓ w ( θ 0 )) . We first provide lower bounds on two of the integral terms via Assumption 3.4:

where ( ∼ 1) denotes a quantity that converges in probability to 1, g 0 = ∇ log π 0 ( θ 0 ) , and L ′ 2 = NL 2 + L 2 0 N +1 . Transforming variables via x = L ′ ( θ -θ 0 ) ,

wL 2 + L 2

For the upper bound on the first term, we use a local quadratic expansion around θ 0 , where H 0 = -∇ 2 log π 0 ( θ 0 ) ,

Because H ≻ 0 , we have ( N + αw ) H +2 H 0 ≻ 0 eventually; we can transform variables to x = C T ( θ -θ 0 ) , where ( N + αw ) H +2 H 0 N + w +2 = CC T is the Cholesky factorization, and subsequently complete the square. Note that

so

and therefore

Suppose first that w +1 ≤ N/ (4 ∥ C -1 ∥ 2 max { βL 2 , L 2 0 } ) . In this case we bound the integral in Eq. (11) by integrating over the whole space:

Combining this with the previous results yields

Bounding the last term below by 0 and minimizing over w such that w ≤ √ N yields

Bounding ( N + w +2) / ( N +1) ≥ 1 and minimizing over w such that w ≥ √ N yields

where the second line follows because for a, b > 0 and x ≥ 0 , the function ax -b log x is convex in x with minimum at x ⋆ = b/a . Therefore for w +1 ≤ N/ ( . . . ) ,

Next suppose w +1 ≥ N/ (4 ∥ C -1 ∥ 2 max { βL 2 , L 2 0 } ) . A second upper bound on Eq. (11) can be obtained by taking the maximum of the integrand over the integration region ∥ x ∥ 2 ≤ r 2 . Note that since ∥ g w /w ∥ = ω p ( r ) , w = Ω p ( N ) , g/N = O p ( N -1 / 2 ) , and Nr 2 = ω p (1) , we have that ∥ ( g + g w +2 g 0 ) / ( N + w +2) ∥ = ω p ( r ) , and so

So therefore combining this result with the previous bounds and minimizing over w yields

then

By assumption, the condition holds when λ = 1 ; the result follows.

Proof of Proposition 5.4. Let C ( w ) = log ∫ π exp( ¯ ℓ w ) . By the finiteness condition, [53, Theorem 2.4] asserts that C ( w ) is continuous, and has derivatives of all orders that can be obtained by passing differentiation through the integral within the set ∥ w ∥ 2 < α . Let U = Cov π (( ℓ n ) N n =1 ) , and S = span { w ∈ R N : w T ( ¯ ℓ n ) N n =1 = 0 π -a.s. } . Note that S = ker U : since w T Uw = Var π ( w T ( ℓ n ) N n =1 ) , w T Uw = 0 if and only if w T ( ¯ ℓ n ) N n =1 = 0 π -a.s.; and since U is symmetric positive semidefinite, w T Uw = 0 if and only if w ∈ ker U . Therefore C ( w ) is continuous, has derivatives of all orders, and derivatives can be passed through the integral within the set { w ∈ R N : w = v + u, ∥ v ∥ 2 < α/ 2 , u ∈ ker U } . For a vector w = v + u , v ⊥ ker U , u ∈ ker U , and minimum positive eigenvalue λ + of U ,

and so C ( w ) is continuous, has derivatives of all orders, and derivatives can be passed through the integral within the set { w ∈ R N : w T Uw ≤ α 2 λ + 4 } . By Taylor's theorem, for any w in this set, there exists a distribution ν w with density proportional to π exp( ¯ ℓ w ′ ) for some w ′ on the segment from the origin to w such that

By definition of ν w , w ∈ ker U implies that w T ( ¯ ℓ n ) N n =1 = 0 ν w -a.s. and hence 1 6 E ν w [ ( w T ( ¯ ℓ n ) N n =1 ) 3 ] = 0 . Therefore, for w T Uw ≤ α 2 λ + 4 ,

where ⊗ denotes outer products to form a tensor. By continuity of derivatives of all orders within the neighbourhood ∥ w ∥ 2 < α , the result follows by selecting a sufficiently small α .

Combining with the earlier bound and noting that N min {∥ g w /w ∥ , 1 } = ω p (1) yields the final result.

Proof of Corollary 3.6. The proof follows directly from Theorems 3.3 and 3.5 by the data processing inequality applied to KL( w ) .

Proof of Theorem 5.3. By Lemma 5.1,

Since ( ℓ n ) N n =1 are ( f, A ) -subexponential, if

Proposition A.1. Suppose there exist c ∈ R , α, δ > 0 , and 0 < ϵ < 1 such that ℓ ≤ c and for all coreset weights w satisfying αw T Cov π (( ℓ n ) N n =1 ) w ≤ 1 , | ¯ ℓ w | ≤ ϵ | ℓ -c | + δ . Then the potentials ( ℓ n ) N n =1 are ( h, α Cov π (( ℓ n ) N n =1 )) -subexponential with h ( x ) = 1 2 x + e δ + cϵ ∫ π 0 e ϵℓ x 1 -ϵ .

Proof of Proposition A.1. Let ℓ ′ = ℓ -c . Since ℓ ′ ≤ 0 and | ¯ ℓ w | ≤ ϵ | ℓ ′ | + δ for some ϵ < 1 , δ > 0 ,

Proposition A.2. Suppose Θ = R d , ¯ ℓ is G -strongly concave, and there exists L < G , α > 0 , and θ 0 ∈ Θ such that for all coreset weights w satisfying αw T Cov π (( ℓ n ) N n =1 ) w ≤ 1 , ¯ ℓ w is L -Lipschitz smooth, and both ∥∇ ℓ w ( θ 0 ) ∥ and ¯ ℓ w ( θ 0 ) are bounded. Then for any ( L/G ) < ϵ < 1 , there exists c ∈ R , δ > 0 such that the potentials ( ℓ n ) N n =1 are ( h, α Cov π (( ℓ n ) N n =1 )) -subexponential with the same h as in Proposition A.1.

Proof of Proposition A.2. Since ¯ ℓ is G -strongly concave and ¯ ℓ w is L -Lipschitz smooth, we can write

So setting c = ℓ ( θ 0 ) + G 2 ∥ ∥ G -1 ∇ ℓ ( θ 0 ) ∥ ∥ 2 implies ℓ -c is a nonpositive function as required. Then

For 0 < a < G -L , setting ϵ = L + a and then maximizing over ∥ θ -θ 0 ∥ yields

¯

By the boundedness of ℓ w ( θ 0 ) and ∇ ℓ w ( θ 0 ) , maximizing over w yields a value of δ < ∞ .

Lemma A.3. Let X 1 , X 2 , . . . be i.i.d. random variables in R with E X n = 0 , and define the resampled sum

where ( M 1 , . . . , M N ) ∼ Multi( M, ( p 1 , . . . , p N )) , with strictly positive resampling probabilities p 1 , . . . , p N that may depend on X 1 , . . . , X N and N . If there exists a δ > 0 such that as N →∞ , then

Proof. We can rewrite and hence by Slutsky's theorem,

where I m iid ∼ Categorical( p 1 , . . . , p N ) . Consider S N + B N where B N is independent of S N , B N = ± 1 with probability 1 2( NM ) 1+ δ , and B N = 0 otherwise. So if we set A N = σ ( X 1 , . . . , X N ) , [54, Cor. 3] asserts that

as long as for all N large enough,

and as N →∞ ,

Note that the conditional mean and variance have the form

which implies that Var [ 1 M X Im p Im |A N ] < ∞ a.s., since p 1 , . . . , p N are strictly nonnegative and E X n = 0 implies X n is finite almost surely. Next, note that

The above expression converges in probability to 0 by the technical assumptions in the statement of the result as well as the fact that 1 N ∑ n X n a.s. → 0 by the law of large numbers. Once again by the technical assumptions, Var[ S N |A N ] = Ω p ( N 2 /M ) , so

Using Slutsky's theorem again with 1 N ∑ n X n p → 0 and rearranging yields the final result.

Lemma A.4. Suppose coreset weights are generated using the importance weighted construction in Algorithm 1. Let g = ∇ ℓ ( η 0 ) , g w = ∇ ℓ w ( η 0 ) , and H = -E [ ∇ 2 ℓ n ( η 0 ) ] . If conditions A(1-3) and (A6) in Section 4 hold, M = o ( N ) , and M = ω (1) , then

and

Proof. First, since w = ∑ n M n Mp n , E w = N , and

̸

where the last line follows by assumption A6. Therefore by Chebyshev's inequality and M →∞ , w/N p → 1 . Since the data are i.i.d., by conditions A1 and A2, the central limit theorem holds for the sum of ∇ ℓ n ( η 0 ) such that g/ √ N converges in distribution to a normal, and hence ∥ ∥ g N ∥ ∥ = Θ p ( N -1 / 2 ) . By conditions A1, A2, and A6, Lemma A.3 holds such that for any t ∈ R d ,

Since condition A6 asserts that C > Np n ≥ c > 0 , the law of large numbers, condition A1, and M/N → 0 imply that

Summing over a basis of vectors t 1 , . . . , t d shows that

This completes the first three results. Next, by condition A3, for sufficiently large N such that the neighbourhood contains the ball of radius r around η 0 ,

and

so that we have that both

by Markov's inequality. Finally, by the bounded variance in A2, sampling probability bounds in A6, and M →∞ , the variances of 1 N ∇ 2 ℓ w ( η 0 ) and 1 N ∇ 2 ℓ ( η 0 ) both converge to 0 as N →∞ , and since both of these quantities are unbiased estimates of E [ ∇ 2 ℓ n ( η 0 ) ] , Chebyshev's inequality yields the desired convergence in probability.

Lemma A.5. Suppose ( X n ) N n =1 are N i.i.d. random vectors in R d . Fix M ∈ N , M < d and define X = [ X 1 X 2 . . . X M ] ∈ R d × M . If there exists δ > 0 such that

where 1 denotes a vector of all 1 entries, then as N →∞ ,

Proof. For any ϵ > 0 , by the union bound over subsets of [ N ] of size M ,

By Markov's inequality and ( N M ) ≤ ( eN/M ) M ,

Setting ϵ = N -M + δ/ 2 M + δ yields

The right-hand side converges to 0 as N →∞ , yielding the stated result.

Proof of Corollary 4.1 and Corollary 4.2. Set r = (log M ) -1 / 2 . Then since M = o ( N ) , M = ω (1) , and assumptions (A1-3) and (A6) hold, Lemma A.4 holds. Note that ∥ g w /w ∥ = Θ p ( M -1 / 2 ) = o p ( r ) , ηπ 0 is positive at η 0 and twice differentiable by (A4), and Nr 2 = N/ log M = ω (1) since M = o ( N ) . Thus the conditions of Theorem 3.3 are verified. Substitution into the right term in the minimum of Theorem 3.3 yields the stated lower bound of Ω p ( N/M ) . For the left term in Theorem 3.3, define B = { ( η -η 0 ) T H ( η -η 0 ) ≤ r 2 } . Then since H ≻ 0 , r → 0 , and r 2 = 1 / log M = ω (log N/N ) , (A5) guarantees that -log( ηπ )( B c ) = Ω p ( Nr 2 ) = Ω p ( N/ log M ) . Therefore the minimum is Ω p ( N/M ) , and we complete the proof by transferring from KL( w ) on the η -pushforward model to KL( w ) on the original model using Corollary 3.6.

Proof of Corollary 4.3. Fix the δ > 0 guaranteed by (A8), and set r = N -M +3 δ/ 4 2( M + δ ) . Note that Nr 2 = N δ/ 4 M + δ = ω (1) , ηπ 0 is positive at η 0 and twice differentiable by (A4), and by (A1-3) the results pertaining to ∥ ∥ g N ∥ ∥ 2 and sup ∥ η -η 0 ∥ 2 ≤ r ∥ ∥ -1 N ∇ 2 ℓ ( η ) -H ∥ ∥ 2 in Lemma A.4 hold; thus Assumption 3.2 holds. By (A7), Assumption 3.4 holds as well as the conditions on 1 w ∇ 2 ℓ w ( θ ) and 1 w ∑ n w n L 2 n in Theorem 3.5. Finally by (A8), Lemma A.5 holds such that

and hence ∥ ∥ g w w ∥ ∥ = ω p ( r ) . Therefore all conditions of Theorem 3.5 hold. For the left term in the minimum in Theorem 3.5, define B = { ( η -η 0 ) T H ( η -η 0 ) ≤ r 2 } . Then since H ≻ 0 , r → 0 , and r 2 = N -M +3 δ/ 4 M + δ = ω (log N/N ) , (A5) guarantees that -log( ηπ )( B c ) = Ω p ( Nr 2 ) = Ω p ( N δ/ 4 M + δ ) . For the right term,

The minimum of these two is from the right term, so

We complete the proof by transferring from KL( w ) on the η -pushforward model to KL( w ) on the original model using Corollary 3.6.

Proposition A.6. The models specified in Eqs. (2) and (3) satisfy assumptions (A1-5).

Proof. The exact same technique applies to both models, so here we will just demonstrate it for the Cauchy model. In the Cauchy model, θ ∈ R , η : R → R + , η ( θ ) = θ 2 , and

where Z n = X n . Property (A1) holds by routine interchange of differentiation and integration. Property (A2) holds (for any δ > 0 ) because ∇ ℓ n ( η ) and ∇ 2 ℓ n ( η ) are bounded functions of η and X n jointly. Property (A3) holds (for any neighbourhood of η 0 ) because ∇ 3 ℓ n ( η ) is a bounded function of η and X n jointly. Property (A4) holds because the pushforward of Cauchy(0 , 1) through η ( θ ) = θ 2 has full support on R + . In order to verify assumption (A5), suppose there exists a sequence of bounded measurable functions ϕ r ( Z 1 , . . . , Z N ) ∈ [0 , 1] of the data and constants c, c ′ > 0 such that for all r → 0 , r 2 = ω (log N/N ) ,

The functions ϕ r are similar to the test functions of Schwartz [55]. Then defining µ = ηπ and µ 0 = ηπ 0 ,

Using the same proof technique as in Theorem 3.3, the denominator satisfies

By assumption, there exists c > 0 such that

and a c ′ > 0 such that

Therefore µ ( ∥ η -η 0 ∥ ≥ r ) = O p ( e -cNr 2 + N d/ 2 e -c ′ Nr 2 ) = O p ( e ( d/ 2) log N -c ′′ Nr 2 ) ; and since r 2 = ω (log N/N ) , -log µ ( ∥ η -η 0 ∥ ≥ r ) = Ω p ( Nr 2 ) as required by (A5). So to complete the proof of (A5) we need to find a suitable ϕ r . Fix ϵ > 0 , and set

Under p η 0 , Hoeffding's inequality yields

And under p η for ∥ η -η 0 ∥ > r , for small enough ϵ > 0 , P η 0 ( | Z -η 0 | ≤ 1) -P η ( | Z -η 0 | ≤ 1) ≥ 2 ϵr . Therefore

at which point we can again apply Hoeffding's inequality, completing the result.

Lemma A.7. Fix vectors u, u 1 , . . . , u N in a separable Hilbert space with inner product denoted a · b and norm denoted ∥ ∥ . Let v 1 , . . . , v M be drawn from { u 1 , . . . , u N } with probabilities p 1 , . . . , p N either with or without replacement (if without replacement, the probabilities are renormalized after every draw). Then for all ϵ ≥ 0 ,

where

Proof. First note that it suffices to analyze the case with replacement, since this case provides an upper bound on the case without replacement. To demonstrate this, we couple two probability spaces-one that draws v 1 , . . . , v M with replacement, and one without replacement. First, draw an identical vector v 1 for both copies. On each subsequent iteration m> 1 , the 'with replacement' copy first draws whether or not it selects a vector that was previously selected by the 'without replacement' copy. If it does, it draws that vector independently; if it does not, it selects the same vector as the 'without replacement' copy. In any case, at each iteration m , the vectors drawn by the 'with replacement' copy are always a subset of the vectors drawn by the 'without replacement' copy, and hence the minimum over w ≥ 0 is greater for that copy. It therefore suffices to analyze the case with replacement.

To obtain an upper bound on the probability when sampling with replacement, instead of minimizing over all w ≥ 0 jointly, suppose we use the following iterative algorithm. Set x 0 = 0 . At the first iteration, we draw v 1 and set the weight w 1 by optimizing over w 1 ≥ 0 :

Set x 1 = w 1 v 1 , and note that ( u -x 1 ) · x 1 = 0 . Then at each subsequent iteration k , assume the previous iterate is optimized over all nonnegative weights, and hence satisfies ( u -x k -1 ) · x k -1 = 0 . We draw another vector v k , and bound the erorr of the next iterate x k by optimizing over only the weight w k for the new vector v k . Then

Therefore,

where

So for all 0 ≤ K ≤ M ,

Using the Chernoff bound on the binomial CDF, for all K -1 ≤ Mq ,

Substituting K -1 = Mq/ 2 yields

Proof of Corollary 6.1. Since the potentials are β Cov π (( ℓ n ) N n =1 ) subexponential, Theorem 5.3 guarantees that ∀ w ∈ R N + : 4 β ( w -1) T Cov π (( ℓ n ) N n =1 )( w -1) ≤ 1 , KL( w ) ≤ 4 β ( w -1) T Cov π (( ℓ n ) N n =1 )( w -1) . We apply Lemma A.7 with vectors ℓ 1 , . . . , ℓ N (in equivalence classes specified up to a additive constant) and inner product between ℓ i , ℓ j defined by Cov π ( ℓ i , ℓ j ) . In the notation of Lemma A.7, by assumption, ∥ u ∥ 2 = O p ( N α ) and q ( M,ϵ ) = ω p ( M -ρ ) . Substituting M = (log N ) 1 / (1 -ρ ) , we find that

Combining this result with the KL bound above yields the final result.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction state that the paper produces new general upper and lower bounds for Bayesian coreset approximations, that these bounds are applied to particular cases of interest, and that empirical results align with the theory. The paper does indeed contain these results, with proofs in the supplemental material, and empirical results in the figures do match the theory well.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In the conclusions section of the work, two main limitations of the theory are mentioned: the need to perform case-by-case analysis of subexponentiality constants and alignment probabilities in Definition 5.2 and Corollary 6.1.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Theoretical results are numbered, and proofs of all results are included in the appendix. Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All details needed to reproduce the experimental results in Figure 2 and 3 are included in the text. Algorithms used in the experiments exist in the cited literature.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: There are no new algorithms presented in this work; the experiments involve only existing methods for which public code is available. The code is not central to the contributions of the paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All details needed to reproduce the experimental results in Figure 2 and 3 are included in the text. Algorithms used in the experiments exist in the cited literature.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All empirical results show the mean over a number of trials, as well as error bars indicating standard error.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: These details are not important for this paper, as there are no new methods or algorithms presented or claims related to computational performance. However, the introduction does list that simulations were performed on a desktop computer with a Core i7 processor and 32GB RAM.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This paper presents a new theoretical analysis of error bounds for Bayesian coresets methods. It does not present any new methodology or data with potential harmful consequences.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no potential negative societal impact of this work. The paper provides new theory regarding existing methodology.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This does not apply.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This does not apply.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This does not apply.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This does not apply.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This does not apply.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper studies the quality of posterior likelihood based on coreset, a subset of the full data. The quality is quantified through KL divergence between the posterior likelihoods based on coreset and full data, so the main theorems demonstrate the upper and lower bounds of KL which are of great concern.

STRENGTHS:
The Bayesian coreset approach is interesting and has been studied recently. Upper and lower bounds of the KL divergence are interesting and useful.

WEAKNESSES:
1. Tightness of the upper and lower bounds. I am not sure how different are the bounds from each other. If upper and lower bounds are different from each other, their applications might be limited.

2. There seem to be insufficient experimental results to support the results.

QUESTIONS:
1. Please describe the gaps between the upper and lower bounds of the KL divergence.

2. The paper seems to be related subsampling in frequent statistics. Is coreset method a Bayesian counterpart?

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-06-30

comments: SUMMARY:
This paper provides new lower and upper asymptotic bounds on the KL divergence between the true posterior and the posterior obtained from common classes of coreset construction algorithms, under milder assumptions than previously used.

My own research is in the area of Bayesian statistics but overall less theoretical. Yet, I could follow along thanks to the authors clear structure and writing. I did not have time to check all the proofs in detail but the authors gave me no reason to disbelieve their results.

STRENGTHS:
The paper provides important new insights on the asymptotic performance we can can expect from Bayesian coreset approaches. It is very well written and provides good intuition about their theorems, thus facilitating understanding of the results even for readers you do not have the time or background to go through all the technicalities of the theorems. The two empirical examples are simple but provide good intuition through relating the empirical results with the theory.

WEAKNESSES:
I don’t see any major weaknesses. Of course, it would have been desirable to see the results illustrated on more complicated empirical examples, but I understand why the authors used the examples they did.

QUESTIONS:
Can the authors provide more intuition on why (close to Eq 4) “The lower threshold ensures that the variance of the importance-weighted log-likelihood is not too large, while the upper threshold ensures sufficient diversity in the draws from subsampling.”?

LIMITATIONS:
The provided limitations appear sensible to me.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper derives bounds on the quality of coresets, as measured by forward and reverse KL divergence. The lower bound is used to study importance weighted coresets, leading to the conclusion that importance weighting leads to a large (forward or reverse) KL divergence between the approximate posterior and the posterior unless $\\Omega(N)$ points are sampled into the coreset. They also use these results to show that under additional conditions, any coreset must be of size at least $d$, where $d$ is the dimensionality of the parametric class to avoid a large KL divergence. The upper bound is used to show that coresets  of size $O(\\log N)$ are sufficient to maintain bounded KL divergences for the subsample-optimize approach.\n\nSTRENGTHS:\n- The corollaries of the main results have consequences for the choice of method for coreset construction that seem insightful.\n- Conditions are carefully stated throughout the main text and examples are given to show that the conditions assumed in results do not exclude all interesting cases.\n\nWEAKNESSES:\n- Several statements are made in text that seem overly strong. For example, the authors claim that “handling large-scale data in Bayesian inference…”requires exploiting redundancy in the data”. This seems overly strong (without proof). Further, I think there are large-scale data problems in Bayesian inference that can be solved via exploring structure that cannot be solved via exploiting data redundancy. For example, the likelihood in Gaussian process inference with a regularly spaced one-dimensional data can be computed quickly by exploiting structure in the prior/posterior, but I don’t think would be well-addressed by coreset methods (since there is very little data redundancy). Of course, this isn’t really meant as a criticism of coresets, or to suggest this is the type of problem the authors try to solve. I simply think strong claims should be made specific and supported by results.\n- (Minor) I find the notation used around measures to be somewhat confusing and I think unorthodox. For example, my understanding is that $\\pi_0$ is a measure, but in assumption 3.2, it is also used for a density. Similarly, in assumption 3.2, using $\\pi_0(d \\theta)$ for the measure itself (without an integral) seems unusual. And the integration notation without a differential element seems unorthodox (I would expect either $d\\pi$ or maybe $\\pi(d\\theta)$). I have also not seen the push forward measure written without either a $\\\\#$ or $\\*$, or more directly as a composition. Perhaps the authors can point me to a standard reference using these notations.\n- (Minor) The definition of f in Lemma 3.1 seems unnecessarily complicated. In particular, if $x \\leq 1$, f is just 0. Defining f in cases would likely make it more interpretable if space allows.\n\nQUESTIONS:\n- Could the authors provide proof that $L$ smoothness from below implies Lipschitz smoothness? At a minimum it seems differentiability is required in $L$-smoothness. But also it looks to me more like a Lipschitz condition on the derivate of f instead of f itself.\n- The authors claim that (2) and (3) satisfy the earlier assumptions, but no reference is provided. I didn’t see a checking of assumptions in the appendix. Could the authors either include this, or point me to the correct section if it is already included?\n- Line 191: “Nonuniform probabilities require at least $O(N)$ time and memory…”, should this be $\\Omega(N)$? I’m not convinces at least $O(N)$ makes sense as a notion, since $O(N)$ includes, for example, constant time algorithms.\n- In section 6, the authors make several references to “exact coresets” without previously introducing the concept. What is meant by this? Does this mean a coreset that results in exactly recovering the posterior, or something else?\n- This is certainly beyond the scope of the current work, but I am curious. Are the approximation results presented strong enough to establish contraction for the approximate posterior produced by Bayesian coreset methods? (perhaps following a similar argument to Ray and Szabo, 2020 Theorem 5).\n\n### Reference\nRay and Szabo. Variational Bayes for high-dimensional linear regression with sparse priors. 2020.\n\nLIMITATIONS:\nLimitations seem well-adressed.\n\n### Other Comments:\nConsider defining the various big-O etc notation used in a footnote or appendix. These are all reasonably standard, but some come up less than others and I think it would be useful for many readers. Relatedly, $\\omega$ and $w$ are quite hard to tell apart. I don’t know if there is a fix for this, but perhaps it is worth considering.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present general upper and lower bounds on the Kullback-Leibler (KL) divergence of coreset approximations. The lower bounds require only mild model assumptions typical of Bayesian asymptotic analyses, while the upper bounds require the log-likelihood functions to satisfy a generalized subexponentiality criterion that is weaker than conditions used in earlier work. The lower bounds are applied to explain the the poor performance of importance sampling-based construction methods. The upper bounds are used to analyze the performance of recent subsample-optimize methods.\n\nSTRENGTHS:\n1. The paper is well-written, and the main technical results are clearly presented.\n2. The paper addresses several gaps in the analysis of Bayesian coreset such as providing a theoretical explanation for the previously-observed poor empirical performance of importance sampling-based construction methods through a lower bound on KL divergence.\n3. To the best of my knowledge, the theorems are sound and the derivations are accurate.\n\nWEAKNESSES:\nThe paper offers a theoretical explanation for the suboptimal performance of importance-weighted coreset construction, and the conclusions of Corollary 4.1 and 4.2 are further validated by simulation results (Figure 2). \nHowever, [32] presents extensive experimental results on real-world datasets, which raises questions about the effectiveness of the lower bounds in scenarios where the model is misspecified or when working with real-world data (where models are often misspecified to some extent).\n\nQUESTIONS:\n1. Could you please provide some discussions about whether the theories can be extended to misspecified regime?\n2. Providing formal definitions for some notations such as $\\Omega_p$ and $\\omega_p$ would be helpful.\n\nLIMITATIONS:\nThe paper clearly states its limitation.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work contributes to the approximation theory of Bayesian coresets. It establishes asymptotic lower bounds (in KL divergence) of Bayesian coreset approximation that do not require posterior normality assumptions. It also provides an upper bound of the approximation (in KL divergence) when the potentials satisfy the generalized subexponentiality condition. The theories are applied to some recent coreset construction algorithms in the literature to corroborate their observed empirical performances.\n\nSTRENGTHS:\n-  I must preface this review by saying that this paper is not in my expertise. However, as someone unfamiliar with Bayesian corset, I found it well-written, has a good flow, and I believe it presents very solid theoretical results.\n- The theoretical results are novel, presenting the first established lower bound for coreset approximation in the literature.\n- The upper bound of coreset approximation in this work relaxes the assumptions found in previous studies. I appreciate the author(s) providing examples of subexponential potentials in Propositions A.1 and A.2.\n\nWEAKNESSES:\n- Since this manuscript studies both the lower and upper bounds of coreset approximation, a natural question that arises is how well do the lower and upper bounds match each other? Can the author(s) provide an example to see if there is a gap between these two bounds? That is, if we consider a specific set of subexponential potentials with a specific $f$ and $A$, can we show that its upper bound matches its lower bound when $N$ is sufficiently large? Or, are the lower and upper bounds incomparable? If these bounds are optimal, they can guide researchers in designing corset construction algorithms and greatly benefit the community. \n- In the second lower bound in Theorem 3.3, is there a trade-off between $N$ and $\\|\\frac{g}{N} - \\frac{g_w}{\\bar{w}}\\|^2_2$ as $N \\to \\infty$ while $\\|\\frac{g}{N} - \\frac{g_w}{\\bar{w}}\\|^2_2 \\to 0$? If $\\|\\frac{g}{N} - \\frac{g_w}{\\bar{w}}\\|^2_2 \\to 0$ faster than $N \\to \\infty$, do we end up having a lower bound that's close to $0$? \n- It seems like the role of the sequence $r$ (i.e., how fast does $r \\to 1$) is very significant in establishing the lower bound. Specifically, in Corollary 4.1, 4.2, and 4.3, $r$ is carefully chosen as shown in the proofs. I find it necessary to specify these choices of $r$ in the statements of Corollary 4.1, 4.2, and 4.3.\n\nQUESTIONS:\nSee weaknesses\n\nLIMITATIONS:\nYes, the author(s) have addressed the limitations in the conclusion section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-15"}