Title: Kernel Quadrature with Randomly Pivoted Cholesky
Venue: NeurIPS 2023
Accepted: yes
Keywords: kernel quadrature

Nyström approximation

reproducing kernel Hilbert space

randomly pivoted Cholesky
Abstract: This paper presents new quadrature rules for functions in a reproducing kernel Hilbert space using nodes drawn by a sampling algorithm known as randomly pivoted Cholesky. The resulting computational procedure compares favorably to previous kernel quadrature methods, which either achieve low accuracy or require solving a computationally challenging sampling problem. Theoretical and numerical results show that randomly pivoted Cholesky is fast and achieves comparable quadrature error rates to more computationally expensive quadrature schemes based on continuous volume sampling, thinning, and recombination. Randomly pivoted Cholesky is easily adapted to complicated geometries with arbitrary kernels, unlocking new potential for kernel quadrature.

Kernel Quadrature with Randomly Pivoted Cholesky

Abstract

This paper presents new quadrature rules for functions in a reproducing kernel Hilbert space using nodes drawn by a sampling algorithm known as randomly pivoted Cholesky. The resulting computational procedure compares favorably to previous kernel quadrature methods, which either achieve low accuracy or require solving a computationally challenging sampling problem. Theoretical and numerical results show that randomly pivoted Cholesky is fast and achieves comparable quadrature error rates to more computationally expensive quadrature schemes based on continuous volume sampling, thinning, and recombination. Randomly pivoted Cholesky is easily adapted to complicated geometries with arbitrary kernels, unlocking new potential for kernel quadrature.

1 Introduction

Quadrature is one of the fundamental problems in computational mathematics, with applications in Bayesian statistics [35], probabilistic ODE solvers [27], reinforcement learning [32], and modelbased machine learning [30]. The task is to approximate an integral of a function f by the weighted sum of f 's values at judiciously chosen quadrature points s 1 , . . . , s n :

Here, and throughout, X denotes a topological space equipped with a Borel measure µ , and g ∈ L 2 ( µ ) denotes a square-integrable function. The goal of kernel quadrature is to select quadrature weights w 1 , . . . , w n ∈ R and nodes s 1 , . . . , s n ∈ X which minimize the error in the approximation (1) for all f drawn from a reproducing kernel Hilbert space (RKHS) H of candidate functions.

The ideal kernel quadrature scheme would satisfy three properties:

Spectral accuracy. The error of the approximation (1) decreases at a rate governed by the eigenvalues of the reproducing kernel k of H , with rapidly decaying eigenvalues guaranteeing rapidly decaying quadrature error.

Efficiency. The nodes s 1 , . . . , s n and weights w 1 , . . . , w n can be computed by an algorithm which is efficient in both theory and practice.

The first two goals may be more easily achieved if one has access to a Mercer decomposition

where e 1 , e 2 , . . . form an orthonormal basis of L 2 ( µ ) and the eigenvalues λ 1 ≥ λ 2 ≥ ·· · are decreasing. Fortunately, the Mercer decomposition is known analytically for many RKHS's H on simple sets X such as boxes [0 , 1] d . However, such a decomposition is hard to compute for general kernels k and domains X , leading to the third of our desiderata:

Mercer-free. The quadrature scheme can be efficiently implemented without access to an explicit Mercer decomposition (2).

Despite significant progress in the probabilistic and kernel quadrature literature (see, e.g., [3-8, 17, 18, 21-23, 26, 41]), the search for a kernel quadrature scheme meeting all three criteria remains ongoing.

Contributions. Wepresent a new kernel quadrature method based on the randomly pivoted Cholesky (RPCHOLESKY) sampling algorithm that achieves all three of the goals. Our main contributions are

Generalizing RPCHOLESKY sampling (introduced in [10] for finite kernel matrices) to the continuum setting and demonstrating its effectiveness for kernel quadrature.

Establishing theoretical results (see theorem 1) which show that RPCHOLESKY kernel quadrature achieves near-optimal quadrature error rates.

Developing efficient rejection sampling implementations (see algorithms 2 and 4) of RPCHOLESKY in the continuous setting, allowing kernel quadrature to be applied to general spaces X , measures µ , and kernels k with ease.

The remainder of this introduction will sketch our proposal for RPCHOLESKY kernel quadrature. A comparison with existing kernel quadrature approaches appears in section 2.

Randomly pivoted Cholesky. Let H be an RKHS with a kernel k that is integrable on the diagonal

RPCHOLESKY uses the kernel diagonal k ( x, x ) as a sampling distribution to pick quadrature nodes. The first node s 1 is chosen to be a sample from the distribution k ( x, x ) d µ ( x ) , properly normalized:

Having selected s 1 , we remove its influence on the kernel, updating the entire kernel function:

Linear algebraically, the update (4) can be interpreted as Gaussian elimination, eliminating 'row' and 'column' s 1 from the 'infinite matrix' k . Probabilistically, if we interpret k as the covariance function of a Gaussian process, the update (4) represents conditioning on the value of the process at s 1 . We use the updated kernel k ′ to select the next quadrature node:

whose influence is then subtracted off k ′ as in (4). RPCHOLESKY continues along these lines until n nodes have been selected. The resulting algorithm is shown in algorithm 1. Having chosen the nodes s 1 , . . . , s n , our choice of weights w 1 , . . . , w n is standard and is discussed in section 3.2.

RPCHOLESKY sampling is more flexible than many kernel quadrature methods, easily adapting to general spaces X , measures µ , and kernels k . To demonstrate this flexibility, we apply RPCHOLESKY to the region X in fig. 1a equipped with the Matérn 5 / 2 -kernel with bandwidth 2 and the measure

A set of n = 20 quadrature nodes produced by RPCHOLESKY sampling (using algorithm 2) is shown in fig. 1a. The cyan-pink shading shows the diagonal of the kernel after updating for the selected nodes. We see that near the selected nodes, the updated kernel is very small, meaning that future steps of the algorithm will avoid choosing nodes in those regions. Nodes far from any currently selected nodes have a much larger kernel value, making them more likely to be chosen in future RPCHOLESKY iterations.

(a)

(b)

2

0.035

1.5

0.03

100

1

0.025

10-1

0.5

0.02

enb

a

0

0.015

e

10-2

-0.5

-1

0.01

Tean

10-3

IID

-1.5

0.005

M

O(n-5/2)

10-4

-2

101

102

-2

-1

0

1

2

Number of nodes n

The quadrature error for f ( x, y ) = sin( x ) exp( y ) , g ≡ 1 , and different numbers n of quadrature nodes for RPCHOLESKY kernel quadrature, kernel quadrature with nodes drawn iid from µ/µ ( X ) , and Monte Carlo quadrature are shown in fig. 1b. Other spectrally accurate kernel quadrature methods would be difficult to implement in this setting because they would require an explicit Mercer decomposition (projection DPPs and leverage scores) or an expensive sampling procedure (continuous volume sampling). A comparison of RPCHOLESKY with more kernel quadrature methods on a benchmark problem is provided in fig. 2.

2 Related work

Here, we overview past work on kernel quadrature and discuss the history of RPCHOLESKY sampling.

2.1 Kernel quadrature

The goal of kernel quadrature is to provide a systematic means for designing quadrature rules on RKHS's. Relevant points of comparison are Monte Carlo [36] and quasi-Monte Carlo [14] methods, which have O (1 / √ n ) and O (polylog( n ) /n ) convergence rates, respectively.

The literature on probabilistic and kernel approaches to quadrature is vast, so any short summary is necessarily incomplete. Here, we briefly summarize some of the most prominent kernel quadrature

methods, highlighting limitations that we address with our RPCHOLESKY approach. We refer the reader to [22, Tab. 1] for a helpful comparison of many of the below-discussed methods.

Herding. Kernel herding schemes [4, 11, 26, 41] iteratively select quadrature nodes using a greedy approach. These methods face two limitations. First, they require the solution of a (typically) nonconvex global optimization problem at every step, which may be computationally costly. Second, these methods exhibit slow O (1 /n ) quadrature error rates [11, Prop. 4] (or even slower [16]). Optimally weighted herding is known as sequential Bayesian quadrature [26].

Thinning. Thinning methods [17, 18, 38] try to select n good quadrature nodes from a larger set of N ≫ n candidate nodes drawn from a simple distribution. While these methods have other desirable theoretical properties, they do not benefit from spectral accuracy.

Leverage scores and projection DPPs. With optimal weights (section 3.2), quadrature with nodes sampled using a projection determintal point process (DPP) [7] (see also [5, 6, 21]) or iid from the (ridge) leverage score distribution [3] achieve spectral accuracy. However, known efficient sampling algorithms require access to the Mercer decomposition (2), limiting the applicability of these schemes to simple spaces X , measures µ , and kernels k , where the decomposition is known analytically.

Continuous volume sampling. Continuous volume sampling is the continuous analog of n -DPP sampling [28], providing quadrature nodes that achieve spectral accuracy [8, Prop. 5]. Unfortunately, continuous volume sampling is computationally challenging. In the finite setting, the best-known algorithm [9] for exact n -DPP sampling requires a costly O ( | X | · n 6 . 5 + n 9 . 5 ) operations. Inexact samplers based on Markov chain Monte Carlo (MCMC) (e.g., [1, 2, 34]) may be more competitive, but the best-known samplers in the continuous setting still require an expensive O ( n 5 log n ) MCMCsteps [34, Thms. 1.3-1.4]. RPCHOLESKY sampling achieves similar theoretical guarantees to continuous volume sampling (see theorem 1) and can be efficiently and exactly sampled (algorithm 2).

Recombination and convex weights. The paper [22] (see also [23]) proposes two ideas for kernel quadrature when µ is a probability measure and g ≡ 1 . First, they suggest using a recombination algorithm (e.g., [29]) to subselect good quadratures from N ≫ n candidate nodes iid sampled from µ . All of the variants of their method either fail to achieve spectral accuracy or require an explicit Mercer decomposition [22, Tab. 1]. Second, they propose choosing weights ( w 1 , . . . , w n ) that are convex combination coefficients. This choice makes the quadrature scheme robust against misspecification of the RKHS H ̸∋ f , among other benefits. It may be worth investigating a combination of RPCHOLESKY quadrature nodes with convex weights in future work.

2.2 Randomly pivoted Cholesky

RPCHOLESKY was proposed, implemented, and analyzed in [10] for the task of approximating an M × M positive-semidefinite matrix A . The algorithm is algebraically equivalent to applying an earlier algorithm known as adaptive sampling [12, 13] to A 1 / 2 . Despite their similarities, RPCHOLESKY and adaptive sampling are different algorithms: To produce a rankn approximation to an M × M matrix, RPCHOLESKY requires O ( n 2 M ) operations, while adaptive sampling requires a much larger O ( nM 2 ) operations. See [10, §4.1] for further discussion on RPCHOLESKY's history. In this paper, we introduce a continuous extension of RPCHOLESKY and analyze its effectiveness for kernel quadrature.

3 Theoretical results

In this section, we prove our main theoretical result for RPCHOLESKY kernel quadrature. We first establish the mathematical setting (section 3.1) and introduce kernel quadrature (section 3.2). Then, we present our main theorem (section 3.3) and discuss its proof (section 3.4).

3.1 Mathematical setting and notation

Let µ be a Borel measure supported on a topological space X and let H be a RKHS on X with continuous kernel k : X × X → R . We assume that x ↦→ k ( x, x ) is integrable (3) and that H

is dense in L 2 ( µ ) . These assumptions imply that H is compactly embedded in L 2 ( µ ) , the Mercer decomposition (2) converges pointwise, and the Mercer eigenfunctions form an orthonormal basis of L 2 ( µ ) and an orthogonal basis of H [39, Thm. 3.1].

Define the integral operator provided

Viewed as an operator T : L 2 ( µ ) → L 2 ( µ ) , T is self-adjoint, positive semidefinite, and trace-class.

One final piece of notation: For a function δ : X → R and an ordered (multi)set S = { s 1 , . . . , s n } , we let δ ( S ) be the column vector with i th entry δ ( s i ) . Similarly, for a bivariate function h : X × X → R , h ( · , S ) (resp. h ( S , · ) ) denotes the row (resp. column) vector-valued function with i th entry h ( · , s i ) (resp. h ( s i , · ) ), and h ( S , S ) denotes the matrix with ij entry h ( s i , s j ) .

3.2 Kernel quadrature

Following earlier works (e.g., [3, 7]), let us describe the kernel quadrature problem more precisely. Given a function g ∈ L 2 ( µ ) , we seek quadrature weights w = ( w 1 , . . . , w n ) ∈ R n and nodes S = { s 1 , . . . , s n } ⊆ X that minimize the maximum quadrature error over all f ∈ H with ∥ f ∥ H ≤ 1 :

A short derivation ([3, Eq. (7)] or appendix C) yields the simple formula

This equation reveals that the quadrature rule minimizing Err( S , w ; g ) is the least-squares approximation of Tg ∈ H by a linear combination of kernel function evaluations ∑ n i =1 w i k ( · , s i ) .

If we fix the quadrature nodes S = { s 1 , . . . , s n } , the optimal weights w ⋆ = ( w ⋆ 1 , . . . , w ⋆n ) ∈ R n minimizing Err( S , w ; g ) are the solution of the linear system of equations

We use (8) to select the weights throughout this paper, and denote the error with optimal weights by

3.3 Error bounds for randomly pivoted Cholesky kernel quadrature

Our main result for RPCHOLESKY kernel quadrature is as follows:

Theorem 1. Let S = { s 1 , . . . , s n } be generated by the RPCHOLESKY sampling algorithm. For any function g ∈ L 2 ( µ ) , nonnegative integer r ≥ 0 , and real number δ > 0 , we have

To illustrate this result, consider an RKHS with eigenvalues λ i = O ( i -2 s ) for s > 1 / 2 . An example is the periodic Sobolev space H s per ([0 , 1]) (see section 5). By setting δ = 1 /r , we see that

The optimal scheme requires n = Θ( r ) nodes to achieve this bound [8, Prop. 5 & §2.5]. Thus, to achieve an error of O ( r -2 s ) , RPCHOLESKY requires just logarithmically more nodes than

the optimal quadrature scheme for such spaces. This compares favorably to continuous volume sampling, which achieves the slightly better optimal rate but is much more difficult to sample.

Having established that RPCHOLESKY achieves nearly optimal error rates for interesting RKHS's, we make some general comments on theorem 1. First, observe that the error depends on the sum ∑ ∞ i = r +1 λ i of the tail eigenvalues. This tail sum is characteristic of spectrally accurate kernel quadrature schemes [7, 8]. The more distinctive feature in (10) is the logarithmic factor. Fortunately, for double precision computation, the achieveable accuracy is bounded by the machine precision 2 -52 , so this logarithmic factor is effectively a modest constant:

3.4 Connection to Nyström approximation and idea of proof

We briefly outline the proof of theorem 1. Following previous works (e.g., [3, 23]), we first utilize the connection between kernel quadrature and the Nyström approximation [31, 42] of the kernel k .

Definition 2. For nodes S = { s 1 , . . . , s n } ⊆ X , the Nyström approximation to the kernel k is

Here, † denotes the Moore-Penrose pseudoinverse. The Nyström approximate integral operator is

This definition leads to a formula for the quadrature rule ∑ n i =1 w i k ( · , s i ) with optimal weights (8).

Proposition 3. Fix nodes S = { s 1 , . . . , s n } . With the optimal weights w ⋆ ∈ R n (8) , we have

To finish the proof of theorem 1, we develop and solve a recurrence for an upper bound on the largest eigenvalue of E [ T -T S ] . See appendix A for details and for the proof of proposition 3.

4 Efficient randomly pivoted Cholesky by rejection sampling

To efficiently perform RPCHOLESKY sampling in the continuous setting, we can use rejection sampling . (A similar rejection sampling idea is used in the MCMC continuous volume sampler of [34].) Assume for simplicity that the measure µ is normalized so that

We assume two forms of access to the kernel k :

Entry evaluations. For any x, y ∈ X , we can evaluate k ( x, y ) .

Sampling the diagonal. We can produce samples from the measure k ( x, x ) d µ ( x ) .

To sample from the RPCHOLESKY distribution, we use k ( x, x ) d µ ( x ) as a proposal distribution and accept proposal s with probability

(Recall k S from (11).) The resulting algorithm for RPCHOLESKY sampling is shown in algorithm 2.

Consequently,

Algorithm 2 RPCHOLESKY with rejection sampling

Algorithm 3 Helper subroutine to evaluate residual kernel

Input:

Point

s

∈

X

, set

S

⊆

X

, Cholesky factor

L

of

k

(

S

,

S

)

, and kernel

k

Output:

d

=

k

(

s, s

)

-

k

S

(

s, s

)

and

c

=

L

-

1

k

(

S

, s

)

1:

2:

3:

procedure

c

L

d

-

k

(

1

RESIDUALKERNEL(

k

(

S

, s

)

s, s

)

4:

return

(

d,

-∥

c

5: end procedure

Theorem 4. Algorithm 2 produces exact RPCHOLESKY samples. Let η i denote the trace-error of the best ranki approximation to T :

The expected runtime of algorithm 2 is at most O ( ∑ n i =1 i 2 /η i ) ≤ O ( n 3 /η n -1 ) .

This result demonstrates that algorithm 2 suffers from the curse of smoothness : The faster the eigenvalues λ 1 , λ 2 , . . . decrease, the smaller η n -1 will be and, consequently, the slower algorithm 2 will be in expectation. While this curse is an unfortunate limitation, it is also a common one. The curse of smoothness affects all known Mercer-free, spectrally accurate kernel quadrature schemes. In fact, the situation is worse for other algorithms. The CVS sampler of [34], for example, requires as many as O ( n 5 log n ) MCMC steps, each of which has cost O ( n 2 /η n -1 ) . According to current analysis, algorithm 2 is O ( n 4 log n ) -times faster than the CVS sampler of [34].

Notwithstanding the curse of smoothness, algorithm 2 is useful in practice. The algorithm works under minimal assumptions and can reach useful accuracies η = 10 -5 while sampling n = 1000 nodes on a laptop. Algorithm 2 may also be interesting for RPCHOLESKY sampling for a large finite set | X | ≥ 10 9 , since its runtime has no explicit dependence on the size of the space X .

To achieve higher accuracies and blunt the curse of smoothness, we can improve algorithm 2 with optimization. Indeed, the optimal acceptance probability for algorithm 2 would be

This suggests the following scheme: Initialize with α = 1 and run the algorithm with acceptance probability p ( s i ; α ) . If we perform many loop iterations without an acceptance, we then recompute the optimal α by solving an optimization problem. The resulting procedure is shown in algorithm 4.

If the optimization problem for α is solved to global optimality, then algorithm 4 produces exact RPCHOLESKY samples. The downside of algorithm 4 is the need to solve a (typically) nonconvex global optimization problem to compute the optimal α . Fortunately, in our experiments (section 5), only a small number of optimization problems ( ≤ 10 ) are needed to produce a sample of n = 1000 nodes. In the setting where the optimization problem is tractable, the speedup of algorithm 4 can

c

)

∥

2

s

S

,

,

L

,

k

)

Algorithm 4 RPCHOLESKY with optimized rejection sampling

be immense. To produce n = 200 RPCHOLESKY samples for the space [0 , 1] 3 with the kernel (13), algorithm 4 requires just 0 . 19 seconds compared to 7 . 47 seconds for algorithm 2.

5 Comparison of methods on a benchmark example

Having demonstrated the versatility of RPCHOLESKY for general spaces X , measures µ , and kernels k in fig. 1, we now present a benchmark example from the kernel quadrature literature to compare RPCHOLESKY with other methods. Consider the periodic Sobolev space H s per ([0 , 1]) with kernel

where B 2 s is the (2 s ) th Bernoulli polynomial and {·} reports the fractional part of a real number [3, p. 7]. We also consider [0 , 1] 3 equipped with the product kernel

We quantify the performance of nodes S and weights w using Err( S , w ; g ) , which can be computed in closed form [22, Eq. (14)]. We set µ := UNIF [0 , 1] d and g ≡ 1 .

We compare the following schemes:

Monte Carlo, IID kernel quadrature. Nodes s 1 , . . . , s n iid ∼ µ with uniform weights w i = 1 /n (Monte Carlo) and optimal weights (8) (IID).

Thinning. Nodes s 1 , . . . , s n thinned from n 2 iid samples from µ = UNIF [0 , 1] using kernel thinning [17, 18] with the COMPRESS ++ algorithm [38] with optimal weights (8).

Continuous volume sampling (CVS). Nodes s 1 , . . . , s n drawn from the volume sampling distribution [8, Def. 1] by Markov chain Monte Carlo with optimal weights (8).

RPCHOLESKY . Nodes s 1 , . . . , s n sampled by RPCHOLESKY using algorithm 4 with the optimal weights (8).

Positively weighted kernel quadrature (PWKQ). Nodes and weights computed by the Nyström+empirical+opt method as described on [22, p. 9].

See appendix D for more details about our numerical experiments. Experiments were run on a MacBook Pro with a 2.4 GHz 8-Core Intel Core i9 CPU and 64 GB 2667 MHz DDR4 RAM. Our code is available at https://github.com/eepperly/RPCholesky-Kernel-Quadrature .

Errors for the different methods are shown in fig. 2 (left panels). We see that RPCHOLESKY consistently performs among the best methods at every value of n , numerically achieving the rate

(a)

s

= 3

,

d

= 1

(b)

s

= 3

,

d

= 3

10°

10-2

Time (sec)

100

Err(S, w; 9

·ID

10-4

+Thinning

10~2

CVS

←PWKQ

10-6

10-4

101

102

100

101

102

103

Number of nodes n

Number of nodes n

10

102

Err(S, w; g9)

10%

Time (sec)

100

10

10-2

10-2

10-4

101

102

100

101

102

103

Number of nodes n

Number of nodes n

of convergence of the fastest method for each problem. Particularly striking is the result that RPCHOLESKY sampling's performance is either very close to or better than that of continuous volume sampling, despite the latter's slightly stronger theoretical properties.

Figure 2 (right panels) show the sampling times for each of the nonuniform sampling methods. RPCHOLESKY sampling is the fastest by far. To sample 128 nodes for s = 3 and d = 1 , RPCHOLESKY was 17 × faster than continuous volume sampling, 52 × faster than thinning, and 170 × faster than PWKQ.

To summarize our numerical evaluation (comprising fig. 2 and fig. 1 from the introduction), we find that RPCHOLESKY is among the most accurate kernel quadrature methods tested. To us, the strongest benefits of RPCHOLESKY (supported by these experiments) are the method's speed and flexibility. These virtues make it possible to apply spectrally accurate kernel quadrature in scenarios where it would have been computationally intractable before.

6 Application: Analyzing large chemistry datasets

While our focus thusfar has been on infinite domains X , the kernel quadrature formalism can also be applied to a finite set X of data points. Let µ = UNIF X be the uniform distribution and set g ≡ 1 . Here, the task is to exhibit n nodes S = { s 1 , . . . , s n } ⊆ X and weights w ∈ R n such that the average of every 'smooth' function f : X → R over the whole dataset is well-approximated by a sum:

Here is an application to chemistry. Let X denote a large set of compounds of interest, and let f : X → R denote a target chemical property such as specific heat capacity. We are interested

in computing mean( f ) . Unfortunately, evaluating f on each x ∈ X requires an expensive density functional theory computation, so it can be prohibitively expensive to evaluate f on every x ∈ X . Fortunately, we can obtain fast approximations to mean( f ) using (14), which only require evaluating f on a much smaller set of size | S | ≪| X | .

Our experimental setup is as follows. For X , we use 2 × 10 4 randomly selected points from the QM9 dataset [33, 37, 40]. We represent each compound as a vector in R 1500 using the many-body tensor representation [25] computed with the DScribe package [24]. Choose k to be a Gaussian kernel with bandwidth chosen by median heuristic [20] on a further subsample of 10 3 random points. We omit continuous volume sampling, thinning, and positively weighted kernel quadrature because of their computational cost. In their place, we add the greedy Nyström method [19] with optimal weights (8).

(a)

(b)

polarizability error

10

10-1

Err(S, w; g)

10-2

Isotropic

Greedy

10-2

10-3

IID

101

102

101

102

Number of nodes n

Number of nodes n

Results are shown in fig. 3. Figure 3a shows the worst-case quadrature error (9). For this metric, IID and randomly pivoted Cholesky are the definitive winners, with randomly pivoted Cholesky being slightly better. Figure 3b shows the mean relative error for the kernel quadrature estimates of the mean isotropic polarizability of the compounds in X . For sufficiently large n , randomly pivoted Cholesky has the smallest error, beating IID and Monte Carlo by a factor of three at n = 512 .

7 Conclusions, Limitations, and Possibilites for Future Work

In this article, we developed continuous RPCHOLESKY sampling for kernel quadrature. Theorem 1 demonstrates RPCHOLESKY kernel quadrature achieves near-optimal error rates. Numerical results (fig. 2) hint that its practical performance might be even better than that suggested by our theoretical analysis and fully comparable with kernel quadrature based on the much more computationally expensive continuous volume sampling distribution. RPCHOLESKY supports performant rejection sampling algorithms (algorithms 2 and 4), which facilitate implementation for general spaces X , measures µ , and kernels k with ease.

We highlight three limitations of RPCHOLESKY kernel quadrature that would be worth addressing in future work. First, given the comparable performance of RPCHOLESKY and continuous volume sampling in practice, it would be desirable to prove stronger error bounds for RPCHOLESKY sampling or find counterexamples which demonstrate a separation between the methods. Second, it would be worth developing improved sampling algorithms for RPCHOLESKY which avoid the need for global optimization steps. Third, all known spectrally accurate kernel quadrature methods require integrals of the form ∫ X k ( x, y ) g ( y ) d µ ( y ) , which may not be available. RPCHOLESKY kernel quadrature requires them for the computation of the weights (8). Developing spectrally accurate kernel quadrature schemes that avoid such integrals remains a major open problem for the field.

References

Nima Anari, Yang P. Liu, and Thuy-Duong Vuong. Optimal sublinear sampling of spanning trees and determinantal point processes via average-case entropic independence. In 63rd Annual Symposium on Foundations of Computer Science , pages 123-134. IEEE, October 2022.

Nima Anari, Shayan Oveis Gharan, and Alireza Rezaei. Monte Carlo Markov chain algorithms for sampling strongly Rayleigh distributions and determinantal point processes. In Proceedings of the 29th Conference on Learning Theory , pages 103-115. PMLR, June 2016.

Francis Bach. On the equivalence between kernel quadrature rules and random feature expansions. Journal of Machine Learning Research , 18(1):714-751, January 2017.

Francis Bach, Simon Lacoste-Julien, and Guillaume Obozinski. On the equivalence between herding and conditional gradient algorithms. Proceedings of the 29th International Conference on Machine Learning , March 2012.

Rémi Bardenet and Adrien Hardy. Monte Carlo with determinantal point processes. The Annals of Applied Probability , 30(1):368-417, February 2020.

Ayoub Belhadji. An analysis of Ermakov-Zolotukhin quadrature using kernels. In Advances in Neural Information Processing Systems , volume 34, pages 27278-27289. Curran Associates, Inc., 2021.

Ayoub Belhadji, Rémi Bardenet, and Pierre Chainais. Kernel quadrature with DPPs. In Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Ayoub Belhadji, Rémi Bardenet, and Pierre Chainais. Kernel interpolation with continuous volume sampling. In Proceedings of the 37th International Conference on Machine Learning , pages 725-735. PMLR, November 2020.

Daniele Calandriello, Michal Derezinski, and Michal Valko. Sampling from a k -DPP without looking at all items. In Advances in Neural Information Processing Systems , volume 33, pages 6889-6899. Curran Associates, Inc., 2020.

Yifan Chen, Ethan N. Epperly, Joel A. Tropp, and Robert J. Webber. Randomly pivoted Cholesky: Practical approximation of a kernel matrix with few entry evaluations, February 2023. arXiv:2207.06503.

Yutian Chen, Max Welling, and Alex Smola. Super-samples from kernel herding. In Proceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence , page 109-116. AUAI Press, 2010.

Amit Deshpande, Luis Rademacher, Santosh Vempala, and Grant Wang. Matrix approximation and projective clustering via volume sampling. In Proceedings of the seventeenth annual ACM-SIAM symposium on Discrete algorithm , pages 1117-1126. ACM, 2006.

Amit Deshpande and Santosh Vempala. Adaptive sampling and fast low-rank matrix approximation. In Josep Díaz, Klaus Jansen, José D. P. Rolim, and Uri Zwick, editors, Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques , pages 292-303. Springer Berlin Heidelberg, 2006.

Josef Dick and Friedrich Pillichshammer. Quasi-Monte Carlo integration, discrepancy and reproducing kernel Hilbert spaces , page 16-45. Cambridge University Press, 2010.

Tobin A. Driscoll, Nicholas Hale, and Lloyd N. Trefethen. Chebfun guide, 2014.

J. C. Dunn. Convergence rates for conditional gradient sequences generated by implicit step length rules. SIAM Journal on Control and Optimization , 18(5):473-487, 1980.

Raaz Dwivedi and Lester Mackey. Kernel thinning. In Proceedings of Thirty Fourth Conference on Learning Theory , pages 1753-1753. PMLR, August 2021.

Raaz Dwivedi and Lester Mackey. Generalized kernel thinning. In Tenth International Conference on Learning Representations , April 2022.

Shai Fine and Katya Scheinberg. Efficient SVM Training training using low-rank kernel representations. Journal of Machine Learning Research , 2(Dec):243-264, 2001.

Damien Garreau, Wittawat Jitkrittum, and Motonobu Kanagawa. Large sample analysis of the median heuristic, October 2018. arXiv:1707.07269.

Guillaume Gautier, Rémi Bardenet, and Michal Valko. On two ways to use determinantal point processes for Monte Carlo integration. In Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Satoshi Hayakawa, Harald Oberhauser, and Terry Lyons. Positively weighted kernel quadrature via subsampling. In Advances in Neural Information Processing Systems , volume 35, pages 6886-6900. Curran Associates, Inc., 2022.

Satoshi Hayakawa, Harald Oberhauser, and Terry Lyons. Sampling-based Nyström approximation and kernel quadrature, January 2023. arXiv:2301.09517.

Lauri Himanen, Marc O. J. Jäger, Eiaki V. Morooka, Filippo Federici Canova, Yashasvi S. Ranawat, David Z. Gao, Patrick Rinke, and Adam S. Foster. DScribe: Library of descriptors for machine learning in materials science. Computer Physics Communications , 247:106949, February 2020.

Haoyan Huo and Matthias Rupp. Unified representation of molecules and crystals for machine learning. Machine Learning: Science and Technology , 3:045017, November 2022.

Ferenc Huszár and David Duvenaud. Optimally-weighted herding is Bayesian quadrature. In Proceedings of the Twenty-Eighth Conference on Uncertainty in Artificial Intelligence , page 377-386. AUAI Press, 2012.

Hans Kersting and Philipp Hennig. Active uncertainty calibration in Bayesian ODE solvers. In Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence , page 309-318. AUAI Press, 2016.

Alex Kulesza and Ben Taskar. k -DPPs: Fixed-size determinantal point processes. In Proceedings of the 28th International Conference on Machine Learning , pages 1193-1200. Omnipress, January 2011.

C. Litterer and T. Lyons. High order recombination and an application to cubature on Wiener space. The Annals of Applied Probability , 22(4):1301-1327, August 2012.

Kevin P. Murphy. Machine Learning: A Probabilistic Perspective . The MIT Press, 2012.

E. J. Nyström. Über Die Praktische Auflösung von Integralgleichungen mit Anwendungen auf Randwertaufgaben. Acta Mathematica , 54(0):185-204, 1930.

Supratik Paul, Konstantinos Chatzilygeroudis, Kamil Ciosek, Jean-Baptiste Mouret, Michael A. Osborne, and Shimon Whiteson. Alternating optimisation and quadrature for robust control. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18) , pages 3925-3933. AAAI Press, February 2018.

Raghunathan Ramakrishnan, Pavlo O. Dral, Matthias Rupp, and O. Anatole von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data , 1(1):140022, December 2014.

Alireza Rezaei and Shayan Oveis Gharan. A polynomial time MCMC method for sampling from continuous determinantal point processes. In Proceedings of the 36th International Conference on Machine Learning , pages 5438-5447. PMLR, May 2019.

Christian P. Robert. The Bayesian Choice: From Decision-Theoretic Foundations to Computational Implementation . Springer Texts in Statistics. Springer, second edition, 2007.

Christian P. Robert and George Casella. Monte Carlo Statistical Methods . Springer Texts in Statistics. Springer, second edition, 2004.

Lars Ruddigkeit, Ruud van Deursen, Lorenz C. Blum, and Jean-Louis Reymond. Enumeration of 166 billion organic small molecules in the chemical universe database GDB-17. Journal of Chemical Information and Modeling , 52(11):2864-2875, November 2012.

Abhishek Shetty, Raaz Dwivedi, and Lester Mackey. Distribution compression in near-linear time. In Tenth International Conference on Learning Representations , April 2022.

Ingo Steinwart and Clint Scovel. Mercer's theorem on general domains: On the interaction between measures, kernels, and RKHSs. Constructive Approximation , 35(3):363-417, June 2012.

Annika Stuke, Milica Todorovi´ c, Matthias Rupp, Christian Kunkel, Kunal Ghosh, Lauri Himanen, and Patrick Rinke. Chemical diversity in molecular orbital energy predictions with kernel ridge regression. The Journal of Chemical Physics , 150(20):204121, May 2019.

Kazuma Tsuji, Ken'ichiro Tanaka, and Sebastian Pokutta. Pairwise conditional gradients without swap steps and sparser kernel herding. In Proceedings of the International Conference on Machine Learning , 2022.

Christopher K. I. Williams and Matthias Seeger. Using the Nyström method to speed up kernel machines. In Proceedings of the 13th International Conference on Neural Information Processing Systems , pages 661-667. MIT Press, January 2000.

A Proof of theorem 1

We start off by proving proposition 3:

Proof of proposition 3. Equation (12) follows from a direct calculation:

To establish the second claim, use the fact [3, §2.1] that T -1 / 2 : H → L 2 ( µ ) is an isometry and invoke the definition (9) of Err( S ; g ) :

The last inequality follows because T -T S and T S = T -( T -T S ) are both positive semidefinite.

For the rest of our proof of theorem 1, we employ the following notation. Let ⪯ denote the Loewner order for bounded, linear operators on L 2 ( µ ) . That is, A ⪯ B if B -A is positive semidefinite (psd). Let k i denote the residual

for the approximate kernel associated with the first i nodes. (See proposition 8 below.) The associated integral operator is

To analyze RPCHOLESKY, we begin by analyzing a single step of the procedure. The expected value of the residual kernel after one step, k 1 , is

Consequently, the expected value of the integral operator T 1 is

The map Φ enjoys several useful properties [10, Lem. 3.2].

Proposition 5. The map Φ is positive, monotone, and concave with respect to the order ⪯ on the set of trace-class psd operators on L 2 ( µ ) . That is, for trace-class psd operators A,B ,

Proof. We reproduce the proof from [10, Lem. 3.2] for completeness.

Positivity. Assume A ⪰ 0 . Observe that Id ⪰ A/ Tr A , where Id denotes the identity operator. Thus

This establishes that Φ is positive: A ⪰ 0 implies Φ( A ) ⪰ 0 .

Concavity. For θ ∈ [0 , 1] , denote θ := 1 -θ . Then

This establishes the concavity claim (17).

and, for θ ∈ [0 , 1] ,

Monotonicity. Suppose 0 ⪯ A ⪯ B . Observe the map Φ is positive homogeneous: For θ > 0 , Φ( θA ) = θ Φ( A ) . Thus,

Invoke concavity (17) to obtain

In the last inequality, we use the positivity Φ( B -A ) ⪰ 0 . This completes the proof of (16).

We can use the map Φ to bound the residual integral operator T i :

Proposition 6. For i ≥ 0 , we have the bound

where Φ i denotes the i -fold composition of Φ .

Proof. The i th step of RPCHOLESKY applies one step of RPCHOLESKY to k i -1 . Thus,

Now, we can apply concavity (17) to obtain

Applying this result twice and using monotonicity (16) yields:

Iterating this argument, we obtain the desired conclusion (18).

We now have the tools to prove theorem 1. We actually prove the following slight refinement:

Theorem 7. Let S = { s 1 , . . . , s n } be generated by the RPCHOLESKY sampling algorithm. For any function g ∈ L 2 ( µ ) , nonnegative integer r ≥ 0 , and real numbers a, ε > 0 , we have provided

Theorem 1 immediately follows from theorem 7 by taking ε = δ/ 2 and a = ( δ/ 2) ∑ ∞ i = r +1 λ i . We now prove theorem 7.

Proof of theorem 7. In the standing notation, T -T S corresponds to T n . Thus, by proposition 6, we have

Thus, it is sufficient to show that holds under condition (19).

Using the definition (15) of Φ , we see that the eigenvalues λ ( i ) 1 ≥ λ ( i ) 2 ≥ · · · of Φ i ( T ) are given by the following recurrence

with initial condition λ (0) j = λ j for all j . From this formula, we immediately conclude that each λ ( i ) j is nonnegative and nonincreasing in i . In addition, the ordinal ranking λ ( i ) 1 ≥ λ ( i ) 2 ≥ · · · is preserved for each i . Indeed, if λ ( i -1) j +1 -λ ( i -1) j ≥ 0 , then

and the claim follows by induction on i . To bound λ max (Φ i ( T )) = λ ( i ) 1 , we bound the sum of eigenvalues appearing in the recurrence (21):

To bound the first r terms, we used the ordering of the eigenvalues λ ( i -1) 1 ≥ · · · ≥ λ ( i -1) r . To bound the tail, we used the fact that for each ℓ ∈ { r +1 , r +2 , . . . } , the eigenvalue λ ( i -1) ℓ is nonincreasing in i and thus bounded by λ (0) ℓ = λ ℓ . Substituting (22) in (21), we obtain

To bound the solution to the recurrence (23), we pass to continuous time. Specifically, for any t ∈ Z + , λ ( t ) 1 is bounded by the process x ( t ) solving the initial value problem

The bound λ ( t ) 1 ≤ x ( t ) holds for all t ∈ Z + because x ↦→ -x 2 / ( rx + ∑ ∞ ℓ = r +1 λ ℓ ) is decreasing for x ∈ (0 , ∞ ) . We are interested in the time t = t ⋆ at which x ( t ⋆ ) = a + ε ∑ ∞ ℓ = r +1 λ ℓ , which we obtain by separation of variables

Since x is decreasing, the following holds for n satisfying (10):

This shows (20), completing the proof.

B Proof of theorem 4

To see why algorithm 2 produces samples from the RPCHOLESKY sampling distribution, observe that at each effective iteration i the matrix L (1 : i -1 , 1 : i -1) is the Cholesky factor of k ( S , S ) . Therefore, d is equal to the residual kernel k res S := k -k S evaluated at s i

At each effective iteration i , the candidate point s i is sampled from k ( x, x ) d µ ( x ) and accepted with probability k res S ( s i , s i ) /k ( s i , s i ) . Therefore, conditional on acceptance and previous steps of the algorithm, s i is sampled from the probability measure

Since this is a probability measure which must integrate to one, the probability of acceptance is

Therefore, conditional on acceptance and previous steps of the algorithm, s i is sampled from the probability measure k res S ( x, x ) d µ ( x ) / ∫ X k res S ( x, x ) d µ ( x ) . This corresponds exactly with the RPCHOLESKY sampling procedure described in algorithm 1 (see proposition 8 below).

Now, we bound the runtime. For each iteration i , the expected acceptance probability is bounded from below by the smallest possible trace error achievable by a rank ( i -1) approximation

Therefore, the expected number of rejections at step i is bounded by the expectation of a geometric random variable with parameter η i -1 , i.e., by η -1 i -1 . Since each loop iteration takes O ( i 2 ) operations (attributable to line 4) and at most η -1 i -1 expected iterations are required to accept at step i , the total expected runtime is at most O ( ∑ n i =1 i 2 /η i -1 ) ≤ O ( n 3 /η n -1 ) . □

C Derivation of (7)

For completeness, we provide a derivation of (7). This result is standard; see, e.g., [3, Eq. (7)]. Recall our definition (6) for Err( S , w ; g ) :

Use the reproducing property of k and linearity to write

Now apply linearity of ⟨· , ·⟩ H to obtain

Use the dual characterization of the RKHS norm ∥·∥ H and the definition (5) of T to conclude

D Details for numerical experiments

In this section, we provide additional details about our numerical experiments.

Continuous volume sampling. We are unaware of a provably correct n -DPP/continuous volume distribution sampler that is sufficiently efficient to facillitate numerical testing and comparisons. The best-known sampler [34] for the continuous setting is inexact and requires O ( n 5 log n ) MCMC steps, which would be prohibitive for us. To produce samples from the continuous volume sampling distribution for this paper, we use a continuous analog of the MCMC algorithm [2]. Given a set S = { s 1 , . . . , s n } and assuming µ is a probability measure, one MCMC step proceeds as follows:

Sample s new ∼ µ and s old ∼ UNIF S .

Define S ′ = S ∪ { s new } \ { s old } .

With probability

accept and set S ← S ′ . Otherwise, leave S unchanged.

In our experiments, we initialize with s 1 , . . . , s n drawn iid from µ and run for 10 n MCMCsteps.

Running for just 10 n MCMCsteps is aggressive, even in the finite setting. Indeed, the theoretical analysis of [2, Thm. 2, Eq. (1.2)] for µ = UNIF { 1 , 2 , . . . , | X |} , proves the sampler mixes to stationarity in at most ˜ O ( n 2 | X | ) MCMCsteps when properly initialized. Even with just 10 n MCMCsteps, continuous volume sampling is dramatically more expensive than RPCHOLESKY in our experiments for n ≥ 64 .

Finally, we notice that the performance of continuous volume sampling degrades to that of iid sampling for sufficiently large n . We believe that the cause for this are numerical issues in evaluating the determinants of the kernel matrices k ( S , S ) . This suggests that, in addition to being faster, RPCHOLESKY may be more numerically robust than continuous volume sampling.

Thinning and positively weighted kernel quadrature. Our kernel quadrature experiments with thinning and PWKQ is based on code from the authors of [22] available online at https://github. com/satoshi-hayakawa/kernel-quadrature , which uses the goodpoints package ( https: //github.com/microsoft/goodpoints ) by the authors of [17, 18, 38] to perform thinning. We use g = 4 , δ = 0 . 5 , and four bins for the COMPRESS ++ algorithm.

Miscellaneous. Our code is available online at https://github.com/eepperly/RPCholesky-Kernel-Quadrature .

To evaluate exact integrals

for fig. 1, we use the 2D functionality in ChebFun [15]. To compute the optimal weights (8), we add a small multiple of the identity to regularize the system:

Here, ε mach = 2 -52 is the double precision machine epsilon.

E Randomly pivoted Cholesky and Nyström approximation

In this section, we describe the connection between the RPCHOLESKY algorithm and Nyström approximation. We hope that this explanation provides context that will be helpful in understanding the RPCHOLESKY algorithm and the proofs of theorems 1 and 4.

To make the following discussion more clear, we begin by rewriting the basic RPCHOLESKY algorithm (see algorithm 1) in algorithm 5 to avoid overwriting the kernel k at each step:

From this rewriting, we see that in addition to selecting quadrature nodes s 1 , . . . , s n , the RPCHOLESKY algorithm builds an approximation ˆ k n to the kernel k . Indeed, one can verify that the approximation ˆ k n produced by this algorithm is precisely the Nyström approximation (definition 2).

Proposition 8. The output ˆ k n of algorithm 5 is equal to the Nyström approximation k { s 1 ,...,s n } .

Proof. Proceed by induction on n . The base case n = 0 is immediate. Fix n and let S := { s 1 , . . . , s n } , s ′ := s n +1 , and S ′ := { s 1 , . . . , s n , s ′ } . Write the Nyström approximation k S ′ as

Algorithm 5 RPCHOLESKY: unoptimized implementation without overwriting

Input: Kernel k : X × X → R and number of quadrature points n

Output: Quadrature points s 1 , . . . , s n and Nyström approximation ˆ k n = k { s 1 ,...,s n } to k

1: Initialize k 0 ← k , ˆ k 0 ← 0

2: for i = 1 , 2 , . . . , n do

3: Sample s i from the probability measure k ( x, x ) d µ ( x ) / ∫ X k ( x, x ) d µ ( x )

5: Update kernel k i ← k i -1 -k i -1 ( · , s i ) k i -1 ( s i , s i ) -1 k i -1 ( s i , · )

4: Update Nyström approximation ˆ k i ← ˆ k i -1 + k i -1 ( · , s i ) k i -1 ( s i , s i ) -1 k i -1 ( s i , · )

6: end for

Compute a block LDL ⊤ factorization of the matrix in the middle of this expression:

Here, we've used the induction hypothesis

Thus,

For (25), we used the induction hypothesis (24) again. Having established that k S ′ = k n +1 , the proposition is proven.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe kernel quadrature problem computes quadrature weights and nodes that minimize the maximum quadrature error over all functions in RKHS with the norm bounded by 1. This paper proposes new quadrature rules for functions in RKHS using nodes drawn by randomly pivoted Cholesky algorithm. The randomly pivoted Cholesky used in the paper is an extension of the finite variant introduced in [10]. The authors provide the unoptimized implementation followed by the variant with efficient rejection sampling.\n\nSTRENGTHS:\n1. Quadrature problem is one of the fundamental problems in computational mathematics and the paper proposes an efficient method for it.\n\n2. The experiments offer comprehensive comparisons against other state-of-the-art methods.\n\n3. Theorem 1 provides error bounds which achieve spectral accuracy with just logarithmic number of nodes.\n\n4. The proposed method (both the unoptimized and the optimized variants) is easy to implement from the description.\n\nWEAKNESSES:\n1. There are no empirical results on real datasets despite earlier work such as [19] having results on real datasets.\n\n2. Discussion on the application of the proposed method in machine learning is largely missing.\n\nQUESTIONS:\n1. For the right side of the Figure 2 (time vs number of nodes): what is the error level considered for all methods?\n\n2. Could the authors comment on how the results could change for other Mercer kernels?\n\n3. How about the effect of dimensionality on the empirical results? The reference [19] does offer more parameter settings used for evaluation.\n\n4. Seeing that Algorithm 2 and Algorithm 3 stores Cholesky factors, could the authors comment on scalability of the method as the number of quadrature points is increased?\n\nLIMITATIONS:\nThere are no results on real datasets and even on synthetic datasets, the results are limited to one type of kernel. Although it is clear the proposed method has wide applications in machine learning, the authors should demonstrate a usecase in which the proposed method can be applied (e.g. scaling kernel machines).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new method for kernel quadrature which relies on a randomly pivoted Cholesky decomposition.\nThe sampling of the quadrature points is done thanks to a rejection sampling algorithm. To improve performance, an optimized rejection sampling procedure is also introduced, so that the acceptance probability is updated if the number of rejections becomes too large. Numerical simulations show that this method is fast and often as accurate as other performant methods.\nAn error bound for the expectation of the quadrature method is also derived.\n\nSTRENGTHS:\n- This paper is very well written. It mentions other important methods for kernel quadrature.\n- The main idea of this research work is simple and elegant. \n- The theoretical results are well introduced and the proofs are clear although non-trivial.\n- From the practical perspective, the numerical simulations are convincing. There is a clear gain in sampling time, in particular with respect to continuous volume sampling which is a closely related method.\n\nWEAKNESSES:\nThere are very few weaknesses in this paper.\n- It seems that the proof techniques are mainly inspired from [10] to which the authors refer. This is not an issue and does not limit the significance of this contribution.\n- The end of the proof of proposition 3 is not very explicit. However, the authors provide the main elements to finish the proof so that the reader can complete the argument on a sheet of paper.\n\nQUESTIONS:\n- Concerning the simulations reported in Fig 2 (a) (left-hand side), I wonder if there is a numerical issue in the computation of the curve corresponding to continuous volume sampling (CVS). It seems to me that CVS should have at least a similar performance as RPCholesky.\nHowever, for the last two data points, CVS suddenly has a worse performance (following the IID case). This phenomenon is not happening in Fig 2 (b) (left-hand side). Hence, I wonder if there could be an issue in the implementation of CVS such as a numerical instability. Could you comment about that ?\n\nLIMITATIONS:\nThe authors mention possible limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

comments: SUMMARY:
The authors provide theoretical analysis and show that RPCholesky has near optimal error rates.
Meanwhile, the authors propose to combine RPCholesky with rejection sampling in order to use it in the continuous setting. Experiments show that the proposed method is compared favorably to continuous volume sampling which is much more difficult to sample.

STRENGTHS:
The paper is clearly written.

Both assumptions and theorems are well presented and discussed.

The experiments demonstrate the good performance of the proposed method.

WEAKNESSES:
For ''Err'' under line 140 and ''Err'' under line 141, it’s better to include the derivation in the appendix to make the paper self-contained.

The description in Line 226 and Line 227 is unclear. For instance, “opt” is imprecise.

The proof of the theorem in Section 3.4 can be moved to the appendix. It seems not necessary to include it in the main paper.

I think the biggest weakness of the paper is: the experiments are very limited. I would like to see more experiments as the authors claim that their method is very general.

QUESTIONS:
Can the authors include additional experiments and show that the proposed method can also perform well in practice, e.g. real-world experiments? Is this something difficult to implement?

LIMITATIONS:
The limitations are discussed in Section 6.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-19

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe recently proposed Randomly Pivoted Cholesky procedure is an efficient and accurate method for approximating symmetric PSD (e.g. kernel) matrices. It builds a Nystrom approximation via randomizing the choice of pivot in the classical Cholesky decomposition procedure.\nThe authors of the present submission transport the same sampling idea to the continuous case and develop new kernel quadrature rules.\nThe proposed algorithm is based on rejection sampling, it's simple to implement and it's much faster than many other kernel quadrature procedures. Rigorous theoretical guarantees and experimental analysis with synthetic data accompany the algorithm.\n\nSTRENGTHS:\n1) Solid theory. The error of the quadrature depends on the eigenvalue distribution of the kernel (it's spectrally accurate).\n2) The rejection sampling algorithm is practical to implement and it's efficient.\n3) Empirical evaluation supports the claims.\n\nWEAKNESSES:\n1) As the authors themselves point out in Section 6 all spectrally accurate quadratures, including this, currently require the exact computation of the Hilbert–Schmidt integral of function $g$, which may limit the applicability of the method. E.g. the experiments are all based on $g\\equiv 1$.\n2) Although the first sentence of the introduction in lines 12-13 refers to several applications all the experiments in Figures 1 and 2 rely on computing the worst case error of kernel integrals in low dimensions on a given domain with a particular (complicated) kernel. Albeit this style of synthetic evaluation seems common in the cited prior work I believe it would increase the impact of the paper if the authors demonstrated end to end improvements on an actual machine learning tasks that requires computing these integrals. Including a concrete application would boost the accessibility and motivation for non experts reading the paper as well in my opinion.\n\nQUESTIONS:\n1) The analysis and experiment focus on and minimize the worst case error (line 140). What, if anything, would change if you considered some expected error (loss) when $f$ is drawn from some distribution?\n2) Since the experiments in Figure 2) are rather low dimensional ($d=1$ and $d=3$) could you also include a Quasi-Monte Carlo baseline with equal and optimal (eq (5)) weights similarly to current Monte Carlo and IID? Since QMC is not spectrally accurate one would still expect RPCholesky to have much lower error, the gap might shrink though. Could you also consider running experiments with much higher $d$?\n\nLIMITATIONS:\nTheory paper, societal impact is not applicable. The authors do a thorough job in disclosing and discussing technical limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article studies a family of quadrature rules suited for functions that belongs to an RKHS. The proposed construction is based on an algorithm called randomly pivoted Cholesky (RPCholesky). The latter was studied for the task of approximating positive-semidefinite matrices. In particular, the authors extended RPC to continuous domain, and investigated its effectiveness for kernel quadrature. More specifically, the authors studied the quadrature rule obtained by taking the nodes to be the configuration of points that RPCholesky and by taking the weights that minimises the worst-case integration error on the unit ball of the RKHS (WCE). In particular, they proved that the expected value of the squared WCE of this quadrature rule converges to 0, with a rate of convergence that depends on the eigenvalues of the integration operator. Moreover, they studied the expected runtime of the proposed algorithmes.\n\nThe contributions of the article may be summarised as follows: \n* An extension of the RPCholesky algorithm to a continuous domain (Algorithm 1) along with a detailed implementation of it using rejection sampling (Algorithm 2).\n* An implementation of RPCholesky with optimized rejection sampling (Algorithm 4).\n* A generic result (Theorem 1) gives the upper bound of the expected squared WCE for the proposed quadrature rule.\n* The study of the expected runtime of Algorithm 2 (Theorem 4).\n\nSTRENGTHS:\n- The article is very well written. \n- The article expands the application of the randomly pivoted Cholesky algorithm into the continuous domain, establishing another bridge between the numerical linear algebra community and the kernel quadrature community.\n- The proof of the main result (Theorem 1) is original and may be beneficial for the community of kernel quadrature in the future.\n\nWEAKNESSES:\nAt first sight, the proposed algorithm seems to bring a significant algorithmic improvement upon continuous volume sampling. This is at least what we may understand reading the comments of Figure 2 (line 235-line 238). While, the relative improvement is undeniable, RPCholesky suffers from the same limitation of continuous volume sampling that is poor scalability for large values of $N$, especially for smooth kernels. Figure 2 of the article is corroborating this claim: the dependence of the empirical expected time as a function of $N$ seems to be polynomial. This is particularly the case in dimension 1,  where we have $\\lambda_m = \\mathcal{O}(m^{-2s})$ so that $1/\\sum_{m=N+1}^{+\\infty}\\lambda_{m} = \\mathcal{O}(N^{2s+1})$. In other words, the smoother is the kernel the longer it will takes to sample from the RPCholesky, which is exactly the same problem with Gibbs sampling proposed in [1], which provably approximate continuous volume sampling. It turns out that Algorithm 1 is nothing but the algorithm proposed in [1] to initialise an MCMC algorithm that approximate continuous volume sampling; see Algorithm 1 in [1].  The difference between the two works is in the subroutine that consist in sampling from the probability measure $x \\mapsto k(x,x) \\mathrm{d}\\mu(x)/\\int_{\\mathcal{X}}k(u,u)\\mathrm{d}\\mu(u)$: [1] proposed to use a ‘stochastic greedy’ algorithm using rejection sampling, while this article proposed to use RPCholesky with rejection sampling. Interestingly, the two algorithms have an expected time that depends on $1/\\sum_{m=N+1}^{+\\infty}\\lambda_{m}$, where is the number of nodes in the quadrature rule, see Lemma 5.1 in [1]. In other words, this work does not break the ‘curse of smoothness’ that hinders the scalability of continuous volume sampling. \n\n[1] Rezaei, A., & Gharan, S. O. (2019, May). A polynomial time MCMC method for sampling from continuous determinantal point processes. In International Conference on Machine Learning (pp. 5438-5447). PMLR.\n\n[2] Anari, N., Gharan, S. O., & Rezaei, A. (2016, June). Monte Carlo Markov chain algorithms for sampling strongly Rayleigh distributions and determinantal point processes. In Conference on Learning Theory (pp. 103-115). PMLR.\n\nQUESTIONS:\n- The empirical comparison in section 5 was limited to Sobolev spaces of order s=3. It may be interesting to compare RPCholesky to other algorithms on other RKHSs. \n- The empirical experiments make use of a continuous analog of a ’basis-exchange’ MCMC algorithm proposed in [2] instead of the MCMC algorithm based on Gibbs sampling proposed in [1]. Up to my knowledge, only the latter was studied theoretically, and it is possible that the convergence of ’basis-exchange’ MCMC algorithm is slower than the Gibbs sampling.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-28"}