Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper makes a contribution to speeding up Bayesian Quadrature (BQ), a popular method for computing the posterior and evidence. The proposed method builds on the weighted kernel quadrature method of Hayakawa et al. and uses the (proven) advantage of kernel recombination to improve sample selection for worst case error reduction and hence improve the accuracy of the quantities estimated by standard BQ methodology. It is shown that the method performs favourably compared to the state of the art WSABI.

strengths_and_weaknesses: In my view, this paper is an obvious improvement on the WSABI methodology with few drawbacks (other than those intrinsic to BQ utilizing Gaussian processes in general, such an upper bound of applicability to 10-15 dimensions) and a very natural alternative to it that I can see being used in practice. It is well-written, complete and easy to read.

questions: To me, the weakest point of the paper is a somewhat unclear treatment of the relationship between KQ and BQ, w.r.t reference [40] and the explicit contribution made. It is said that "the point configuration in KQ with a small worst-case error gives a good way to select points to reduce the integral variance in Bayesian quadrature." Now, to the best of my knowledge, (naive) kernel herding is limited by the computation of certain integrals required when sequentially selecting points. As far as I understand, the reference [37] is a practical way of circumventing this limitation? It would be nice to expand a bit more on this crucial point upon which the paper is ultimately based. It would also be good to mention some potential areas, if any, where the proposed method performs worse than existing ones. 

limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a novel quadrature procedure that combines elements of warped Bayesian quadrature and random convex hull quadrature. The resulting method enjoys benefits from both component aspects: the computational tractability of RCHQ and the integrand-modeling/active learning techniques of square-root warped BQ (WSABI). The authors empirically evaluate their method against a batch version of WSABI and various implements of nested sampling; they also provide some theoretical analysis on the convergence rate of their method. 
strengths_and_weaknesses: Overall, I think this paper presents some interesting ideas and represents a promising line of inquiry. However, in its current state, I believe there are some issues (detailed below) which keep this version from being a strong candidate for acceptance. 

1. This work makes a litany of smaller, practical contributions which are certainly novel, although the core approach is a combination of existing ideas. However, I do think the methods described in Section 4.2 are sufficient in terms of getting this work above the bar in terms of originality. 

2. My primary concern with this work is the level of analysis performed: given that this work combines many complex components, hyperparameters, design choices, heuristics, etc... an ablation study/sensitivity analysis of some sort seems warranted to assess which aspects are providing what benefit or if the sum is greater than the whole of its parts. Some notable components that I feel warrant a deeper dive include the hyperparameters M, N and r; the GP model on the integrand; sampling methodology (some of these are expanded on in my questions below). 

Relatedly the subsample proposal distributions are somewhat poorly motivated heuristics: it is not clear to me why the uncertainty sampling acquisition function need be mixed with the prior to ensure global optimization. Calling this mixing the "integral variance reduction" distribution is also a bit strange: there is a well-known connection between DPPs and kernel quadrature and the correctly defined DPP would in some sense be the actual integral variance reducing distribution (although I suspect that sampling from this distribution would likely be computationally intractable for this purpose). 

3. Overall, I think this manuscript is reasonably clear/well-written: there are a few minor grammatical errors but they did not hinder my ability to comprehend the material being presented. As a minor nitpick, I would have liked to see more main paper real estate devoted to the figures, both in terms of actual size and interpretation: Figure 1(e) and (f) are a bit difficult to parse and the text doesn't provide much insight but ultimately, they did add to my appreciation of the proposed method. Similarly, the bottom-right quadrant of Figure 2 provides little compelling evidence in favor of their method. 

4. My impression of this work is that it is of low to middling impact, given the somewhat niche use case of low-dimensional, non-negative batch quadrature; this is somewhat validated by the limited existing work/baselines to compare against in this space (of course, the lack of existing work is not being held against the authors, I bring it up merely as a measure of potential significance). Furthermore, related to my notes about quality, the myriad design choices represent a nontrivial barrier for other practitioners/researchers looking to experiment with/improve upon the work presented here. 
questions: - Did the authors consider alternative low-rank approximation methods for the kernel matrix? Or is there something fundamental about the Nyström approximation that would invalidate this method if an alternative were used?

- Similarly, is there something about the WSABI-L approximation that makes it particularly well suited for the method proposed by the authors? Did they also experiment with the WSABI-M approximation or other previously-proposed transforms?

- In the empirical evaluation, are confidence intervals omitted for the real-word experiments or are they just imperceptibly small? In general, are there any meaningful takeaways from the fact that batch WSABI performs identically to the BASQ variants on the real-world benchmarks in terms of log MAE? 
limitations: The authors' discussion of limitations in Section 7 is sufficiently detailed in my opinion.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper combines the batch-version of WSABI Bayesian quadrature method [30], which uses a square-root transform to encode non-negativity of the integrand function (in this case a likelihood function), with the kernel recombination method for sampling of quadrature points from [37] and the Nyström approximation of the kernel.
strengths_and_weaknesses: I do not find this paper particularly exciting: the proposed BASQ method is a conceptually rather straightforward combination of a few existing methods. As far as I can see, the paper contains no interesting new ideas of any generality. The main contribution is that the proposed method outperforms some existing approaches in a number of examples. While the examples do seem impressive, I am not sure if this is sufficient and find the paper borderline, though tend to recommend rejection. However, I have no serious objections to acceptance if the consensus is that a contribution like this is sufficient for this conference.

I have certain concerns with the convergence theory in Section 6 and how it is advertised, which I detail in the next section of this review. Whether or not these concerns are valid has little impact on my assessment of this paper because I do not expect a paper like this to have any useful convergence theory.

The paper is generally reasonably well written and effortless to read. However, there are many unclear statements and undefined notation, examples of which I give in the next section. In general, I find that it would be helpful to replace many of the verbal descriptions with mathematics.
questions: ON THEORY:

1. Theorem 1, if correct (see the next two points), guarantees exponential convergence over a single iteration in the sense that the BQ integral variance decays exponentially fast. This is a consequence of the use of the extremely strong squared exponential GP prior. But no discussion or elaboration on the practical meaning of this statement can be found in the paper. Does the integral estimate produced by the method converge exponentially to the integral of the true likelihood function $\ell_\text{true}$? If so, what assumptions need to be imposed on $\ell_\text{true}$? To include an an unconditional statement that the proposed method "possesses a provably-exponential convergence rate" in the abstract is quite misleading.
2. It is claimed after Theorem 1 that the convergence of the variance is exponential if the kernel has exponentially decaying Mercer eigenvalues. But what about the second term on the right-hand side of Equation (12)? I do not see how this term is to decay exponentially.
3. Equation (12) in the supplement uses the property in Equation (6) of the main paper which says that BQ variance equals the minimal squared worst-case error in the RKHS of $K$. But this characterization only applies to the variance of the vanilla BQ, not the variance of WSABI. Therefore it seems to me that Theorem 1 does not apply to BASQ as proposed in Section 4 but to a variant which uses vanilla BQ instead of WSABI. Is that correct?

MINOR COMMENTS:

- p2, lines 38-39: "Choosing an appropriate kernel and prior makes the integrals analytical." I suppose this refers to the kernel mean embeddings etc. that need to be computed in BQ. However, how the sentence in inserted in the introduction makes it sound like the likelihood integrals that are being computed with BQ are available analytically. There is a similar issue on p2 line 54, "In particular, the integral becomes analytic...".
- Section 2: In this section, BQ is presented as a method specifically for computation of quantities related to posteriors ("The objective of BQ is to compute the following integrals and obtain surrogate function of posterior:") while in reality the integrals computed by BQ do not have to have anything to do with posteriors or Bayesian inference. It would be good to clarify this, e.g. saying that this is just what BQ is used for in this particular paper.
- p2, line 45: "following integrals" There is only one integral in Equation (1).
- p2, lines 60-61: "WSABI ... sets ..." It is unclear what this sentence is trying to say. What is being "set" to something?
- Section 2, in particular "Vanilla Bayesian quadrature": One unacquianted with GPs would probably have a very hard time trying to understand what happens in this section. For example, it is not said that the likelihood function is being modelled as GP(0, K) and that Equation (2) is the resulting posterior GP. The section also contains a lot of undefined notation.
- p2, line 62: Why is the coefficient 0.8 used in the definition of alpha?
- Section 2: I do not see why different notation for point sets is used in "Vanilla Bayesian quadrature" and "Kernel quadrature in general".
- p5, lines 160-1: A word must be missing ("selects sparse samples than batch WSABI"). Or "sparser"?
- Eq (9): dx in the denominator looks different than elsewhere in the paper.
- p6, line 178: Isn't $C_y^L$ the warped kernel?
- Lines 243-4: "that can earn per wall time" (?)
- Lines 246-7: "Still, the time becomes much longer when it increases for accuracy." (?)
- Line 248: "Yet, the original data size is roughly 1,000 times." (?)
- Line 268: "explorative UB proposal distribution exceeds" (?)
- Line 299: "It also works well with a moderate M in practise." What works?
- Ref [9]: "F. X. Briol" should be "F.-X. Briol"
- Ref [58]: "Sarkka" should be "Särkkä"
- Supplement, line 15 has \propto while Line 21 has \approx.

limitations: One of the main limitations of the proposed method (which is shared by all methods that utilize WSABI-BQ) is that it really only applies to $\pi$ being a mixture of Gaussian densities and $K$ the squared exponential kernel because in other cases the integrals of the functions in Equation (4) have not been computed analytically. This limitation should be stated more openly as at the moment the only allusion to it in the abstract and introduction is the sentence "Choosing an appropriate kernel and prior makes the integrals analytical." on p2, lines 38-9.
review_title: null

Meta-review: metareview: The initial round of reviews for the submitted manuscript was mostly positive in tone, but this enthusiasm was tempered by a number of deep technical issues raised by the reviewers. Fortunately, the author rebuttal and author--reviewer discussion phases went a long way toward clearing up some initial confusion and clarifying the contributions of the authors, which swayed the prevailing opinion of the reviewers toward acceptance.

I want to commend the authors for their enlightening contributions to that discussion, which assuaged most of the reviewers' initial complaints. 

_However, I would also like to stress that it is critical that the fruits of this discussion be incorporated into a revised version of this manuscript._ The reviewers are unanimous in this opinion.

In particular, I direct the authors to the conversation with reviewer CnRy and the points raised about:

- the manner in which the theoretical results were initially presented in the discussion/abstract, and
- more clarity regarding the assumptions made Theorem 1 and the notation used to communicate these assumptions and the theorem.

confidence: Certain
Decision: Accept