SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ethan N. Epperly and Elvira Moreno*  
partment of Computing and Mathematical Sciences  
California Institute of Technology  
Pasadena, CA 91125  
{eeppperly, emoreno2}@caltech.edu
This paper presents new quadrature rules for functions in a reproducing kernel Hilbert space using nodes drawn by a sampling algorithm known as randomly pivoted Cholesky. The resulting computational procedure compares favorably to previous kernel quadrature methods, which either achieve low accuracy or require solving a computationally challenging sampling problem. Theoretical and numerical results show that randomly pivoted Cholesky is fast and achieves comparable quadrature error rates to more computationally expensive quadrature schemes based on continuous volume sampling, thinning, and recombination. Randomly pivoted Cholesky is easily adapted to complicated geometries with arbitrary kernels, unlocking new potential for kernel quadrature.
Quadrature is one of the fundamental problems in computational mathematics, with applications in Bayesian statistics [35], probabilistic ODE solvers [27], reinforcement learning [32], and model-based machine learning [30]. The task is to approximate an integral of a function  $f$  by the weighted sum of  $f$ 's values at judiciously chosen quadrature points  $s_1, \ldots, s_n$ :
$$
\int_ {\mathsf {X}} f (x) g (x) \mathrm {d} \mu (x) \approx \sum_ {i = 1} ^ {n} w _ {i} f \left(s _ {i}\right). \tag {1}
$$
Here, and throughout,  $\mathsf{X}$  denotes a topological space equipped with a Borel measure  $\mu$ , and  $g \in \mathsf{L}^2(\mu)$  denotes a square-integrable function. The goal of kernel quadrature is to select quadrature weights  $w_1, \ldots, w_n \in \mathbb{R}$  and nodes  $s_1, \ldots, s_n \in \mathsf{X}$  which minimize the error in the approximation (1) for all  $f$  drawn from a reproducing kernel Hilbert space (RKHS)  $\mathcal{H}$  of candidate functions.
The ideal kernel quadrature scheme would satisfy three properties:
1. Spectral accuracy. The error of the approximation (1) decreases at a rate governed by the eigenvalues of the reproducing kernel  $k$  of  $\mathcal{H}$ , with rapidly decaying eigenvalues guaranteeing rapidly decaying quadrature error.
2. Efficiency. The nodes  $s_1, \ldots, s_n$  and weights  $w_1, \ldots, w_n$  can be computed by an algorithm which is efficient in both theory and practice.
*ENE acknowledges support by NSF FRG 1952777, Carver Mead New Horizons Fund, and the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, Department of Energy Computational Science Graduate Fellowship under Award Number DE-SC0021110. EM was supported in part by Air Force Office of Scientific Research grant FA9550-22-1-0225.
The first two goals may be more easily achieved if one has access to a Mercer decomposition
$$
k (x, y) = \sum_ {i = 1} ^ {\infty} \lambda_ {i} e _ {i} (x) e _ {i} (y), \tag {2}
$$
where  $e_1, e_2, \ldots$  form an orthonormal basis of  $\mathsf{L}^2(\mu)$  and the eigenvalues  $\lambda_1 \geq \lambda_2 \geq \dots$  are decreasing. Fortunately, the Mercer decomposition is known analytically for many RKHS's  $\mathcal{H}$  on simple sets  $\mathsf{X}$  such as boxes  $[0, 1]^d$ . However, such a decomposition is hard to compute for general kernels  $k$  and domains  $\mathsf{X}$ , leading to the third of our desiderata:
3. Mercer-free. The quadrature scheme can be efficiently implemented without access to an explicit Mercer decomposition (2).
Despite significant progress in the probabilistic and kernel quadrature literature (see, e.g., [3-8, 17, 18, 21-23, 26, 41]), the search for a kernel quadrature scheme meeting all three criteria remains ongoing.
Contributions. We present a new kernel quadrature method based on the randomly pivoted Cholesky (RPCHOLESKY) sampling algorithm that achieves all three of the goals. Our main contributions are
1. Generalizing RPCHOLESKY sampling (introduced in [10] for finite kernel matrices) to the continuum setting and demonstrating its effectiveness for kernel quadrature.
2. Establishing theoretical results (see theorem 1) which show that RPCHOLESKY kernel quadrature achieves near-optimal quadrature error rates.
3. Developing efficient rejection sampling implementations (see algorithms 2 and 4) of RP-CHOLESKY in the continuous setting, allowing kernel quadrature to be applied to general spaces  $X$ , measures  $\mu$ , and kernels  $k$  with ease.
The remainder of this introduction will sketch our proposal for RPCHOLESKY kernel quadrature. A comparison with existing kernel quadrature approaches appears in section 2.
Randomly pivoted Cholesky. Let  $\mathcal{H}$  be an RKHS with a kernel  $k$  that is integrable on the diagonal
$$
\int_ {X} k (x, x) \mathrm {d} \mu (x) <   + \infty . \tag {3}
$$
RPCHOLESKY uses the kernel diagonal  $k(x, x)$  as a sampling distribution to pick quadrature nodes. The first node  $s_1$  is chosen to be a sample from the distribution  $k(x, x) \, \mathrm{d}\mu(x)$ , properly normalized:
$$
s _ {1} \sim \frac {k (x , x) \mathrm {d} \mu (x)}{\int_ {\mathsf {X}} k (x , x) \mathrm {d} \mu (x)}.
$$
Having selected  $s_1$ , we remove its influence on the kernel, updating the entire kernel function:
$$
k ^ {\prime} (x, y) = k (x, y) - \frac {k (x , s _ {1}) k (s _ {1} , y)}{k (s _ {1} , s _ {1})}. \tag {4}
$$
Linear algebraically, the update (4) can be interpreted as Gaussian elimination, eliminating "row" and "column"  $s_1$  from the "infinite matrix"  $k$ . Probabilistically, if we interpret  $k$  as the covariance function of a Gaussian process, the update (4) represents conditioning on the value of the process at  $s_1$ . We use the updated kernel  $k'$  to select the next quadrature node:
$$
s _ {2} \sim \frac {k ^ {\prime} (x , x) \mathrm {d} \mu (x)}{\int_ {\mathsf {X}} k ^ {\prime} (x , x) \mathrm {d} \mu (x)},
$$
whose influence is then subtracted off  $k'$  as in (4). RPCHOLESKY continues along these lines until  $n$  nodes have been selected. The resulting algorithm is shown in algorithm 1. Having chosen the nodes  $s_1, \ldots, s_n$ , our choice of weights  $w_1, \ldots, w_n$  is standard and is discussed in section 3.2.
RPCHOLESKY sampling is more flexible than many kernel quadrature methods, easily adapting to general spaces  $X$ , measures  $\mu$ , and kernels  $k$ . To demonstrate this flexibility, we apply RPCHOLESKY to the region  $X$  in fig. 1a equipped with the Matérn 5/2-kernel with bandwidth 2 and the measure
$$
\mathrm {d} \mu (x, y) = \left(x ^ {2} + y ^ {2}\right) \mathrm {d} x \mathrm {d} y.
$$
Algorithm 1 RPCHOLESKY: unoptimized implementation
Input: Kernel  $k:\mathsf{X}\times \mathsf{X}\to \mathbb{R}$  and number of quadrature points  $n$
Output: Quadrature points  $s_1, \ldots, s_n$
1: for  $i = 1,2,\dots ,n$  do
2: Sample  $s_i$  from the probability measure  $k(x, x) \, \mathrm{d}\mu(x) / \int_{\mathsf{X}} k(x, x) \, \mathrm{d}\mu(x)$
3: Update kernel  $k \gets k - k(\cdot, s_i)k(s_i, s_i)^{-1}k(s_i, \cdot)$
4: end for
A set of  $n = 20$  quadrature nodes produced by RPCHOLESKY sampling (using algorithm 2) is shown in fig. 1a. The cyan-pink shading shows the diagonal of the kernel after updating for the selected nodes. We see that near the selected nodes, the updated kernel is very small, meaning that future steps of the algorithm will avoid choosing nodes in those regions. Nodes far from any currently selected nodes have a much larger kernel value, making them more likely to be chosen in future RPCHOLESKY iterations.
(b)
Figure 1: RPCHOLESKY kernel quadrature on oddly shaped region. Left: Black dots show 20 points selected by RPCHOLESKY on a crescent-shaped region. Shading shows kernel values  $k((x,y),(x,y))$  after removing influence of selected points. Right: Mean relative error for RPCHOLESKY kernel quadrature, iid kernel quadrature, and Monte Carlo quadrature for  $f(x,y) = \sin (x)\exp (y)$  with 100 trials. Shaded regions show  $10\% /90\%$  quantiles.
The quadrature error for  $f(x,y) = \sin (x)\exp (y), g\equiv 1$ , and different numbers  $n$  of quadrature nodes for RPCHOLESKY kernel quadrature, kernel quadrature with nodes drawn iid from  $\mu /\mu (\mathsf{X})$  and Monte Carlo quadrature are shown in fig. 1b. Other spectrally accurate kernel quadrature methods would be difficult to implement in this setting because they would require an explicit Mercer decomposition (projection DPPs and leverage scores) or an expensive sampling procedure (continuous volume sampling). A comparison of RPCHOLESKY with more kernel quadrature methods on a benchmark problem is provided in fig. 2.
Here, we overview past work on kernel quadrature and discuss the history of RPCHOLESKY sampling.
2.1 Kernel quadrature
The goal of kernel quadrature is to provide a systematic means for designing quadrature rules on RKHS's. Relevant points of comparison are Monte Carlo [36] and quasi-Monte Carlo [14] methods, which have  $\mathcal{O}(1 / \sqrt{n})$  and  $\mathcal{O}(\mathrm{polylog}(n) / n)$  convergence rates, respectively.
The literature on probabilistic and kernel approaches to quadrature is vast, so any short summary is necessarily incomplete. Here, we briefly summarize some of the most prominent kernel quadrature
methods, highlighting limitations that we address with our RPCHOLESKY approach. We refer the reader to [22, Tab. 1] for a helpful comparison of many of the below-discussed methods.
Herding. Kernel herding schemes [4, 11, 26, 41] iteratively select quadrature nodes using a greedy approach. These methods face two limitations. First, they require the solution of a (typically) nonconvex global optimization problem at every step, which may be computationally costly. Second, these methods exhibit slow  $\mathcal{O}(1/n)$  quadrature error rates [11, Prop. 4] (or even slower [16]). Optimally weighted herding is known as sequential Bayesian quadrature [26].
Thinning. Thinning methods [17, 18, 38] try to select  $n$  good quadrature nodes from a larger set of  $N \gg n$  candidate nodes drawn from a simple distribution. While these methods have other desirable theoretical properties, they do not benefit from spectral accuracy.
Leverage scores and projection DPPs. With optimal weights (section 3.2), quadrature with nodes sampled using a projection determinantal point process (DPP) [7] (see also [5, 6, 21]) or iid from the (ridge) leverage score distribution [3] achieve spectral accuracy. However, known efficient sampling algorithms require access to the Mercer decomposition (2), limiting the applicability of these schemes to simple spaces  $\mathsf{X}$ , measures  $\mu$ , and kernels  $k$ , where the decomposition is known analytically.
Continuous volume sampling. Continuous volume sampling is the continuous analog of  $n$ -DPP sampling [28], providing quadrature nodes that achieve spectral accuracy [8, Prop. 5]. Unfortunately, continuous volume sampling is computationally challenging. In the finite setting, the best-known algorithm [9] for exact  $n$ -DPP sampling requires a costly  $\mathcal{O}(|\mathsf{X}| \cdot n^{6.5} + n^{9.5})$  operations. Inexact samplers based on Markov chain Monte Carlo (MCMC) (e.g., [1, 2, 34]) may be more competitive, but the best-known samplers in the continuous setting still require an expensive  $\mathcal{O}(n^5 \log n)$  MCMC steps [34, Thms. 1.3-1.4]. RPCHOLESKY sampling achieves similar theoretical guarantees to continuous volume sampling (see theorem 1) and can be efficiently and exactly sampled (algorithm 2).
Recombination and convex weights. The paper [22] (see also [23]) proposes two ideas for kernel quadrature when  $\mu$  is a probability measure and  $g\equiv 1$ . First, they suggest using a recombination algorithm (e.g., [29]) to subselect good quadratures from  $N\gg n$  candidate nodes iid sampled from  $\mu$ . All of the variants of their method either fail to achieve spectral accuracy or require an explicit Mercer decomposition [22, Tab. 1]. Second, they propose choosing weights  $(w_{1},\ldots ,w_{n})$  that are convex combination coefficients. This choice makes the quadrature scheme robust against misspecification of the RKHS  $\mathcal{H}\not\geq f$ , among other benefits. It may be worth investigating a combination of RPCHOLESKY quadrature nodes with convex weights in future work.
2.2 Randomly pivoted Cholesky
RPCHOLESKY was proposed, implemented, and analyzed in [10] for the task of approximating an  $M \times M$  positive-semidefinite matrix  $\mathbf{A}$ . The algorithm is algebraically equivalent to applying an earlier algorithm known as adaptive sampling [12, 13] to  $A^{1/2}$ . Despite their similarities, RPC-CHOLESKY and adaptive sampling are different algorithms: To produce a rank- $n$  approximation to an  $M \times M$  matrix, RPCCHOLESKY requires  $\mathcal{O}(n^2 M)$  operations, while adaptive sampling requires a much larger  $\mathcal{O}(n M^2)$  operations. See [10, §4.1] for further discussion on RPCCHOLESKY's history. In this paper, we introduce a continuous extension of RPCCHOLESKY and analyze its effectiveness for kernel quadrature.
In this section, we prove our main theoretical result for RPCHOLESKY kernel quadrature. We first establish the mathematical setting (section 3.1) and introduce kernel quadrature (section 3.2). Then, we present our main theorem (section 3.3) and discuss its proof (section 3.4).
3.1 Mathematical setting and notation
Let  $\mu$  be a Borel measure supported on a topological space  $X$  and let  $\mathcal{H}$  be a RKHS on  $X$  with continuous kernel  $k: X \times X \to \mathbb{R}$ . We assume that  $x \mapsto k(x, x)$  is integrable (3) and that  $\mathcal{H}$
is dense in  $\mathsf{L}^2 (\mu)$ . These assumptions imply that  $\mathcal{H}$  is compactly embedded in  $\mathsf{L}^2 (\mu)$ , the Mercer decomposition (2) converges pointwise, and the Mercer eigenfunctions form an orthonormal basis of  $\mathsf{L}^2 (\mu)$  and an orthogonal basis of  $\mathcal{H}$  [39, Thm. 3.1].
Define the integral operator
$$
T f (x) = \int_ {\mathsf {X}} k (x, y) f (y) \mathrm {d} \mu (y). \tag {5}
$$
Viewed as an operator  $T: \mathsf{L}^2(\mu) \to \mathsf{L}^2(\mu)$ ,  $T$  is self-adjoint, positive semidefinite, and trace-class.
One final piece of notation: For a function  $\delta : \mathsf{X} \to \mathbb{R}$  and an ordered (multi)set  $\mathsf{S} = \{s_1, \ldots, s_n\}$ , we let  $\delta(\mathsf{S})$  be the column vector with  $i$ th entry  $\delta(s_i)$ . Similarly, for a bivariate function  $h : \mathsf{X} \times \mathsf{X} \to \mathbb{R}$ ,  $h(\cdot, \mathsf{S})$  (resp.  $h(\mathsf{S}, \cdot)$ ) denotes the row (resp. column) vector-valued function with  $i$ th entry  $h(\cdot, s_i)$  (resp.  $h(s_i, \cdot)$ ), and  $h(\mathsf{S}, \mathsf{S})$  denotes the matrix with  $ij$  entry  $h(s_i, s_j)$ .
3.2 Kernel quadrature
Following earlier works (e.g., [3, 7]), let us describe the kernel quadrature problem more precisely. Given a function  $g \in \mathsf{L}^2(\mu)$ , we seek quadrature weights  $\boldsymbol{w} = (w_1, \ldots, w_n) \in \mathbb{R}^n$  and nodes  $S = \{s_1, \ldots, s_n\} \subseteq \mathsf{X}$  that minimize the maximum quadrature error over all  $f \in \mathcal{H}$  with  $\| f \|_{\mathcal{H}} \leq 1$ :
$$
\operatorname {E r r} (\mathbf {S}, \boldsymbol {w}; g) := \max  _ {\| f \| _ {\mathcal {H}} \leq 1} \left| \int_ {\mathbf {X}} f (x) g (x) \mathrm {d} \mu (x) - \sum_ {i = 1} ^ {n} w _ {i} f \left(s _ {i}\right) \right|. \tag {6}
$$
A short derivation ([3, Eq. (7)] or appendix C) yields the simple formula
$$
\operatorname {E r r} (\mathsf {S}, \boldsymbol {w}; g) = \left\| T g - \sum_ {i = 1} ^ {n} w _ {i} k (\cdot , s _ {i}) \right\| _ {\mathcal {H}}. \tag {7}
$$
This equation reveals that the quadrature rule minimizing  $\mathrm{Err}(\mathsf{S},\boldsymbol {w};g)$  is the least-squares approximation of  $Tg\in \mathcal{H}$  by a linear combination of kernel function evaluations  $\sum_{i = 1}^{n}w_{i}k(\cdot ,s_{i})$
If we fix the quadrature nodes  $\mathsf{S} = \{s_1, \ldots, s_n\}$ , the optimal weights  $\pmb{w}_{\star} = (w_{\star 1}, \dots, w_{\star n}) \in \mathbb{R}^n$  minimizing  $\mathrm{Err}(\mathsf{S}, \pmb{w}; g)$  are the solution of the linear system of equations
$$
k (\mathsf {S}, \mathsf {S}) \boldsymbol {w} _ {\star} = T g (\mathsf {S}). \tag {8}
$$
We use (8) to select the weights throughout this paper, and denote the error with optimal weights by
$$
\operatorname {E r r} (\mathsf {S}; g) := \operatorname {E r r} (\mathsf {S}, \boldsymbol {w} _ {\star}; g). \tag {9}
$$
3.3 Error bounds for randomly pivoted Cholesky kernel quadrature
Our main result for RPCHOLESKY kernel quadrature is as follows:
Theorem 1. Let  $\mathsf{S} = \{s_1, \ldots, s_n\}$  be generated by the RPCHOLESKY sampling algorithm. For any function  $g \in \mathsf{L}^2(\mu)$ , nonnegative integer  $r \geq 0$ , and real number  $\delta > 0$ , we have
$$
\mathbb {E} \operatorname {E r r} ^ {2} (\mathsf {S}; g) \leq \delta \cdot \| g \| _ {\mathsf {L} ^ {2} (\mu)} ^ {2} \cdot \sum_ {i = r + 1} ^ {\infty} \lambda_ {i}
$$
$$
n \geq r \log \left(\frac {2 \lambda_ {1}}{\delta \sum_ {i = r + 1} ^ {\infty} \lambda_ {i}}\right) + \frac {2}{\delta}. \tag {10}
$$
To illustrate this result, consider an RKHS with eigenvalues  $\lambda_{i} = \mathcal{O}(i^{-2s})$  for  $s > 1 / 2$ . An example is the periodic Sobolev space  $H_{\mathrm{per}}^s ([0,1])$  (see section 5). By setting  $\delta = 1 / r$ , we see that
$$
\mathbb {E} \operatorname {E r r} ^ {2} (\mathrm {S}; g) \leq \mathcal {O} (r ^ {- 2 s}) \| g \| _ {\mathrm {L} ^ {2} (\mu)} ^ {2} \quad \text {f o r R P C H O L E S K Y w i t h} n = \Omega (r \log r).
$$
The optimal scheme requires  $n = \Theta(r)$  nodes to achieve this bound [8, Prop. 5 & §2.5]. Thus, to achieve an error of  $\mathcal{O}(r^{-2s})$ , RPCHOLESKY requires just logarithmically more nodes than
the optimal quadrature scheme for such spaces. This compares favorably to continuous volume sampling, which achieves the slightly better optimal rate but is much more difficult to sample.
Having established that RPCHOLESKY achieves nearly optimal error rates for interesting RKHS's, we make some general comments on theorem 1. First, observe that the error depends on the sum  $\sum_{i=r+1}^{\infty} \lambda_i$  of the tail eigenvalues. This tail sum is characteristic of spectrally accurate kernel quadrature schemes [7, 8]. The more distinctive feature in (10) is the logarithmic factor. Fortunately, for double precision computation, the achievable accuracy is bounded by the machine precision  $2^{-52}$ , so this logarithmic factor is effectively a modest constant:
$$
\log \left(\frac {2 \lambda_ {1}}{\delta \sum_ {i = r + 1} ^ {\infty}}\right) \stackrel {{\leqslant}} {{\approx}} \log (2 ^ {5 2}) <   3 7.
$$
3.4 Connection to Nyström approximation and idea of proof
We briefly outline the proof of theorem 1. Following previous works (e.g., [3, 23]), we first utilize the connection between kernel quadrature and the Nyström approximation [31, 42] of the kernel  $k$ .
Definition 2. For nodes  $S = \{s_1, \ldots, s_n\} \subseteq X$ , the Nyström approximation to the kernel  $k$  is
$$
k _ {\mathsf {S}} (x, y) = k (x, \mathsf {S}) k (\mathsf {S}, \mathsf {S}) ^ {\dagger} k (\mathsf {S}, y). \tag {11}
$$
Here,  $\dagger$  denotes the Moore-Penrose pseudoinverse. The Nyström approximate integral operator is
$$
T _ {\mathsf {S}} f := \int_ {\mathsf {X}} k _ {\mathsf {S}} (\cdot , x) f (x) \mathrm {d} \mu (x).
$$
This definition leads to a formula for the quadrature rule  $\sum_{i=1}^{n} w_i k(\cdot, s_i)$  with optimal weights (8).
Proposition 3. Fix nodes  $\mathsf{S} = \{s_1, \dots, s_n\}$ . With the optimal weights  $\boldsymbol{w}_{\star} \in \mathbb{R}^{n}$  (8), we have
$$
\sum_ {i = 1} ^ {n} w _ {i \star} k (\cdot , s _ {i}) = T _ {\mathrm {S}} g. \tag {12}
$$
$$
\operatorname {E r r} ^ {2} (\mathsf {S}; g) = \| (T - T _ {\mathsf {S}}) g \| _ {\mathcal {H}} ^ {2} \leq \langle g, (T - T _ {\mathsf {S}}) g \rangle_ {\mathsf {L} ^ {2} (\mu)}.
$$
To finish the proof of theorem 1, we develop and solve a recurrence for an upper bound on the largest eigenvalue of  $\mathbb{E}[T - T_{\mathbb{S}}]$ . See appendix A for details and for the proof of proposition 3.
4 Efficient randomly pivoted Cholesky by rejection sampling
To efficiently perform RPCHOLESKY sampling in the continuous setting, we can use rejection sampling. (A similar rejection sampling idea is used in the MCMC continuous volume sampler of [34].) Assume for simplicity that the measure  $\mu$  is normalized so that
$$
\int_ {\mathbb {X}} k (x, x) \mathrm {d} \mu (x) = \operatorname {T r} T = 1.
$$
We assume two forms of access to the kernel  $k$ :
1. Entry evaluations. For any  $x, y \in \mathsf{X}$ , we can evaluate  $k(x, y)$ .
2. Sampling the diagonal. We can produce samples from the measure  $k(x,x) \, \mathrm{d}\mu(x)$ .
To sample from the RPCHOLESKY distribution, we use  $k(x,x) \mathrm{d}\mu (x)$  as a proposal distribution and accept proposal  $s$  with probability
$$
1 - \frac {k _ {\mathsf {S}} (s , s)}{k (s , s)}.
$$
(Recall  $k_{5}$  from (11).) The resulting algorithm for RPCHOLESKY sampling is shown in algorithm 2.
Algorithm 2 RPCHOLESKY with rejection sampling
Input: Kernel  $k:\mathsf{X}\times \mathsf{X}\to \mathbb{R}$  and number of quadrature points  $n$    
Output: Quadrature points  $s_1,\ldots ,s_n$    
1: Initialize  $\pmb {L}\gets \pmb{0}_{n\times n},i\gets 1,\mathsf{S}\gets \emptyset$ $\triangleright L$  stores a Cholesky decomposition of  $k(S,S)$    
2: while  $i\leq n$  do   
3: Sample  $s_i$  from the probability measure  $k(x,x)\mathrm{d}\mu (x)$    
4:  $(d,c)\gets \mathrm{RESIDUALKERN}(s_i,\mathsf{S},\mathbf{L},k)$ $\triangleright$  Helper subroutine algorithm 3   
5: Draw a uniform random variable  $U\sim \mathrm{UNIF}[0,1]$    
6: if  $U <   d / k(s_i,s_i)$  then  $\triangleright$  Accept with probability  $d / k(s_i,s_i)$    
7: Induct  $s_i$  into the selected set:  $\mathsf{S}\gets \mathsf{S}\cup \{s_i\} ,i\gets i + 1$    
8:  $L(i,1:i)\gets [c^{\top}\quad \sqrt{d} ]$    
9: end if   
10: end while
Algorithm 3 Helper subroutine to evaluate residual kernel
Input: Point  $s\in \mathsf{X}$  , set  $\mathsf{S}\subseteq \mathsf{X}$  , Cholesky factor  $\pmb{L}$  of  $k(S,S)$  , and kernel  $k$    
Output:  $d = k(s,s) - k_{\mathsf{S}}(s,s)$  and  $c = L^{-1}k(S,s)$    
1: procedure RESIDUALKERNEL(s,S,L,k)   
2:  $c\gets L^{-1}k(S,s)$    
3:  $d\gets k(s,s) - \| c\|^2\qquad \triangleright d = k(s,s) - k_{\mathsf{S}}(s,s)$    
4: return  $(d,c)$    
5: end procedure
Theorem 4. Algorithm 2 produces exact RPCHOLESKY samples. Let  $\eta_{i}$  denote the trace-error of the best rank- $i$  approximation to  $T$ :
$$
\eta_ {i} = \sum_ {j = i + 1} ^ {\infty} \lambda_ {i}.
$$
The expected runtime of algorithm 2 is at most  $\mathcal{O}(\sum_{i=1}^{n} i^2 / \eta_i) \leq \mathcal{O}(n^3 / \eta_{n-1})$ .
This result demonstrates that algorithm 2 suffers from the curse of smoothness: The faster the eigenvalues  $\lambda_1, \lambda_2, \ldots$  decrease, the smaller  $\eta_{n-1}$  will be and, consequently, the slower algorithm 2 will be in expectation. While this curse is an unfortunate limitation, it is also a common one. The curse of smoothness affects all known Mercer-free, spectrally accurate kernel quadrature schemes. In fact, the situation is worse for other algorithms. The CVS sampler of [34], for example, requires as many as  $\mathcal{O}(n^5 \log n)$  MCMC steps, each of which has cost  $\mathcal{O}(n^2 / \eta_{n-1})$ . According to current analysis, algorithm 2 is  $\mathcal{O}(n^4 \log n)$ -times faster than the CVS sampler of [34].
Notwithstanding the curse of smoothness, algorithm 2 is useful in practice. The algorithm works under minimal assumptions and can reach useful accuracies  $\eta = 10^{-5}$  while sampling  $n = 1000$  nodes on a laptop. Algorithm 2 may also be interesting for RPCHOLESKY sampling for a large finite set  $|\mathsf{X}|\geq 10^{9}$ , since its runtime has no explicit dependence on the size of the space  $\mathsf{X}$ .
To achieve higher accuracies and blunt the curse of smoothness, we can improve algorithm 2 with optimization. Indeed, the optimal acceptance probability for algorithm 2 would be