SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \lambda_ {j + 1} ^ {(i)} - \lambda_ {j} ^ {(i)} = \lambda_ {j + 1} ^ {(i - 1)} - \lambda_ {j} ^ {(i - 1)} - \frac {\left(\lambda_ {j + 1} ^ {(i - 1)}\right) ^ {2} - \left(\lambda_ {j} ^ {(i - 1)}\right) ^ {2}}{\sum_ {\ell = 1} ^ {\infty} \lambda_ {\ell} ^ {(i - 1)}} \\ = \left(\lambda_ {j + 1} ^ {(i - 1)} - \lambda_ {j} ^ {(i - 1)}\right) \left(1 - \frac {\lambda_ {j + 1} ^ {(i - 1)} + \lambda_ {j} ^ {(i - 1)}}{\sum_ {\ell = 1} ^ {\infty} \lambda_ {\ell} ^ {(i - 1)}}\right) \geq 0, \\ \end{array}
$$
and the claim follows by induction on  $i$ . To bound  $\lambda_{\max}(\Phi^i (T)) = \lambda_1^{(i)}$ , we bound the sum of eigenvalues appearing in the recurrence (21):
$$
\sum_ {\ell = 1} ^ {\infty} \lambda_ {\ell} ^ {(i - 1)} = \lambda_ {1} ^ {(i - 1)} + \dots + \lambda_ {r} ^ {(i - 1)} + \sum_ {\ell = r + 1} ^ {\infty} \lambda_ {\ell} ^ {(i - 1)} \leq r \lambda_ {1} ^ {(i - 1)} + \sum_ {\ell = r + 1} ^ {\infty} \lambda_ {\ell}. \tag {22}
$$
To bound the first  $r$  terms, we used the ordering of the eigenvalues  $\lambda_1^{(i-1)} \geq \dots \geq \lambda_r^{(i-1)}$ . To bound the tail, we used the fact that for each  $\ell \in \{r+1, r+2, \ldots\}$ , the eigenvalue  $\lambda_\ell^{(i-1)}$  is nonincreasing in  $i$  and thus bounded by  $\lambda_\ell^{(0)} = \lambda_\ell$ . Substituting (22) in (21), we obtain
$$
\lambda_ {1} ^ {(i)} \leq \lambda_ {1} ^ {(i - 1)} - \frac {\left(\lambda_ {1} ^ {(i - 1)}\right) ^ {2}}{r \lambda_ {1} ^ {(i - 1)} + \sum_ {\ell = r + 1} ^ {\infty} \lambda_ {\ell}} \quad \text {f o r} i = 1, 2, \dots . \tag {23}
$$
To bound the solution to the recurrence (23), we pass to continuous time. Specifically, for any  $t \in \mathbb{Z}_+$ ,  $\lambda_1^{(t)}$  is bounded by the process  $x(t)$  solving the initial value problem
$$
\frac {\mathrm {d}}{\mathrm {d} t} x (t) = - \frac {x (t) ^ {2}}{r x (t) + \sum_ {\ell = r + 1} ^ {\infty} \lambda_ {\ell}}, \quad x (0) = \lambda_ {1}.
$$
The bound  $\lambda_1^{(t)}\leq x(t)$  holds for all  $t\in \mathbb{Z}_+$  because  $x\mapsto -x^2 /(rx + \sum_{\ell = r + 1}^{\infty}\lambda_{\ell})$  is decreasing for  $x\in (0,\infty)$ . We are interested in the time  $t = t_{\star}$  at which  $x(t_{\star}) = a + \varepsilon \sum_{\ell = r + 1}^{\infty}\lambda_{\ell}$ , which we obtain by separation of variables
$$
\begin{array}{l} t _ {\star} = \int_ {\lambda_ {1}} ^ {a + \varepsilon \sum_ {\ell = r + 1} ^ {\infty} \lambda_ {\ell}} - \frac {r x + \sum_ {\ell = r + 1} ^ {\infty} \lambda_ {\ell}}{x ^ {2}} d x = \left[ - r \log x + \frac {\sum_ {\ell = r + 1} ^ {\infty} \lambda_ {\ell}}{x} \right] _ {x = \lambda_ {1}} ^ {x = a + \varepsilon \sum_ {\ell = r + 1} ^ {\infty} \lambda_ {\ell}} \\ = r \log \frac {\lambda_ {1}}{a + \varepsilon \sum_ {\ell = r + 1} ^ {\infty} \lambda_ {\ell}} + \frac {\sum_ {\ell = r + 1} \lambda_ {\ell}}{a + \varepsilon \sum_ {\ell = r + 1} \lambda_ {\ell}} - \frac {\sum_ {\ell = r + 1} \lambda_ {\ell}}{\lambda_ {1}} \leq r \log \frac {\lambda_ {1}}{a} + \frac {1}{\varepsilon}. \\ \end{array}
$$
Since  $x$  is decreasing, the following holds for  $n$  satisfying (10):
$$
\lambda_ {\max } (\Phi^ {n} (T)) = \lambda_ {1} ^ {(n)} \leq x (n) \leq x (t _ {\star}) = a + \varepsilon \sum_ {\ell = r + 1} ^ {\infty} \lambda_ {\ell}.
$$
This shows (20), completing the proof.
B Proof of theorem 4
To see why algorithm 2 produces samples from the RPCHOLESKY sampling distribution, observe that at each effective iteration  $i$  the matrix  $\pmb{L}(1:i - 1,1:i - 1)$  is the Cholesky factor of  $k(\mathsf{S},\mathsf{S})$ . Therefore,  $d$  is equal to the residual kernel  $k_{\mathsf{S}}^{\mathrm{res}}\coloneqq k - k_{\mathsf{S}}$  evaluated at  $s_i$
$$
d = k _ {S} ^ {\mathrm {r e s}} \left(s _ {i}, s _ {i}\right).
$$
At each effective iteration  $i$ , the candidate point  $s_i$  is sampled from  $k(x,x)\mathrm{d}\mu (x)$  and accepted with probability  $k_{\mathsf{S}}^{\mathrm{res}}(s_i,s_i) / k(s_i,s_i)$ . Therefore, conditional on acceptance and previous steps of the algorithm,  $s_i$  is sampled from the probability measure
$$
\frac {1}{\mathbb {P} \{\text {A c c e p t a n c e} \mid \mathsf {S} \}} \cdot k (x, x)   \mathrm {d} \mu (x) \cdot \frac {k _ {\mathsf {S}} ^ {\mathrm {r e s}} (x , x)}{k (x , x)} = \frac {k _ {\mathsf {S}} ^ {\mathrm {r e s}} (x , x)   \mathrm {d} \mu (x)}{\mathbb {P} \{\text {A c c e p t a n c e} \mid \mathsf {S} \}}.
$$
Since this is a probability measure which must integrate to one, the probability of acceptance is
$$
\mathbb {P} \{\text {A c c e p t a n c e} \mid \mathrm {S} \} = \int_ {\mathrm {X}} k _ {\mathrm {S}} ^ {\text {r e s}} (x, x) \mathrm {d} \mu (x).
$$
Therefore, conditional on acceptance and previous steps of the algorithm,  $s_i$  is sampled from the probability measure  $k_{\mathsf{S}}^{\mathrm{res}}(x,x)\mathrm{d}\mu (x) / \int_{\mathsf{X}}k_{\mathsf{S}}^{\mathrm{res}}(x,x)\mathrm{d}\mu (x)$ . This corresponds exactly with the RPCHOLESKY sampling procedure described in algorithm 1 (see proposition 8 below).
Now, we bound the runtime. For each iteration  $i$ , the expected acceptance probability is bounded from below by the smallest possible trace error achievable by a rank  $(i - 1)$  approximation
$$
\mathbb {P} \{\text {A c c e p t a n c e} \} = \mathbb {E} [ \mathbb {P} \{\text {A c c e p t a n c e} | \mathsf {S} \} ] = \mathbb {E} \left[ \int_ {\mathsf {X}} k _ {\mathsf {S}} ^ {\text {r e s}} (x, x) \mathrm {d} \mu (x) \right] \geq \eta_ {i - 1}.
$$
Therefore, the expected number of rejections at step  $i$  is bounded by the expectation of a geometric random variable with parameter  $\eta_{i - 1}$ , i.e., by  $\eta_{i - 1}^{-1}$ . Since each loop iteration takes  $\mathcal{O}(i^2)$  operations (attributable to line 4) and at most  $\eta_{i - 1}^{-1}$  expected iterations are required to accept at step  $i$ , the total expected runtime is at most  $\mathcal{O}(\sum_{i = 1}^{n}i^{2} / \eta_{i - 1})\leq \mathcal{O}(n^{3} / \eta_{n - 1})$ .
C Derivation of (7)
For completeness, we provide a derivation of (7). This result is standard; see, e.g., [3, Eq. (7)]. Recall our definition (6) for  $\mathrm{Err}(\mathsf{S},\boldsymbol {w};g)$ :
$$
\operatorname {E r r} (\mathsf {S}, \boldsymbol {w}; g) = \max  _ {\| f \| _ {\mathcal {H}} \leq 1} \left| \int_ {\mathsf {X}} f (x) g (x) \mathrm {d} \mu (x) - \sum_ {i = 1} ^ {n} w _ {i} f (s _ {i}) \right|.
$$
Use the reproducing property of  $k$  and linearity to write
$$
\operatorname {E r r} (\mathsf {S}, \boldsymbol {w}; g) = \max  _ {\| f \| _ {\mathcal {H}} \leq 1} \left| \int_ {\mathsf {X}} \langle f, k (\cdot , x) \rangle_ {\mathcal {H}} g (x)   \mathrm {d} \mu (x) - \sum_ {i = 1} ^ {n} w _ {i} \langle f, k (\cdot , s _ {i}) \rangle_ {\mathcal {H}} \right|.
$$
Now apply linearity of  $\langle \cdot ,\cdot \rangle_{\mathcal{H}}$  to obtain
$$
\mathrm {E r r} (\mathsf {S}, \boldsymbol {w}; g) = \max _ {\| f \| _ {\mathcal {H}} \leq 1} \left| \left\langle f, \int_ {\mathsf {X}} k (\cdot , x) g (x) \mathrm {d} \mu (x) - \sum_ {i = 1} ^ {n} w _ {i} k (\cdot , s _ {i}) \right\rangle_ {\mathcal {H}} \right|.
$$
Use the dual characterization of the RKHS norm  $\| \cdot \|_{\mathcal{H}}$  and the definition (5) of  $T$  to conclude
$$
\operatorname {E r r} (\mathsf {S}, \boldsymbol {w}; g) = \left\| T g - \sum_ {i = 1} ^ {n} w _ {i} k (\cdot , s _ {i}) \right\| _ {\mathcal {H}}.
$$
D Details for numerical experiments
In this section, we provide additional details about our numerical experiments.
Continuous volume sampling. We are unaware of a provably correct  $n$ -DPP/continuous volume distribution sampler that is sufficiently efficient to facilitate numerical testing and comparisons. The best-known sampler [34] for the continuous setting is inexact and requires  $\mathcal{O}(n^5\log n)$  MCMC steps, which would be prohibitive for us. To produce samples from the continuous volume sampling distribution for this paper, we use a continuous analog of the MCMC algorithm [2]. Given a set  $S = \{s_1,\dots ,s_n\}$  and assuming  $\mu$  is a probability measure, one MCMC step proceeds as follows:
1. Sample  $s_{\mathrm{new}} \sim \mu$  and  $s_{\mathrm{old}} \sim \mathrm{UNIF} \mathsf{S}$ .
2. Define  $\mathsf{S}' = \mathsf{S} \cup \{s_{\mathrm{new}}\} \setminus \{s_{\mathrm{old}}\}$ .
3. With probability
$$
\frac {1}{2} \min  \left\{1, \frac {\det  k \left(S ^ {\prime} , S ^ {\prime}\right)}{\det  k (S , S)} \right\},
$$
accept and set  $S \gets S'$ . Otherwise, leave  $S$  unchanged.
In our experiments, we initialize with  $s_1, \ldots, s_n$  drawn iid from  $\mu$  and run for 10n MCMC steps.
Running for just  $10n$  MCMC steps is aggressive, even in the finite setting. Indeed, the theoretical analysis of [2, Thm. 2, Eq. (1.2)] for  $\mu = \mathrm{UNIF}\{1,2,\ldots ,|\mathsf{X}|\}$ , proves the sampler mixes to stationarity in at most  $\tilde{\mathcal{O}}(n^2|\mathsf{X}|)$  MCMC steps when properly initialized. Even with just  $10n$  MCMC steps, continuous volume sampling is dramatically more expensive than RPCHOLESKY in our experiments for  $n\geq 64$ .
Finally, we notice that the performance of continuous volume sampling degrades to that of iid sampling for sufficiently large  $n$ . We believe that the cause for this are numerical issues in evaluating the determinants of the kernel matrices  $k(\mathsf{S},\mathsf{S})$ . This suggests that, in addition to being faster, RPCHOLESKY may be more numerically robust than continuous volume sampling.
Thinning and positively weighted kernel quadrature. Our kernel quadrature experiments with thinning and PWKQ is based on code from the authors of [22] available online at https://github.com/satoshi-hayakawa/kernel-quadrature, which uses the goodpoints package (https://github.com/microsoft/goodpoints) by the authors of [17, 18, 38] to perform thinning. We use  $\mathfrak{g} = 4$ ,  $\delta = 0.5$ , and four bins for the COMPRESS++ algorithm.
Miscellaneous. Our code is available online at
https://github.com/eepperly/RPCholesky-Kernel-Quadrature.
To evaluate exact integrals
$$
T g (x) = \int_ {\mathbf {X}} k (x, y) g (y) \mathrm {d} \mu (y)
$$
for fig. 1, we use the 2D functionality in ChebFun [15]. To compute the optimal weights (8), we add a small multiple of the identity to regularize the system:
$$
\boldsymbol {w} _ {\star , \text {r e g}} = (k (\mathsf {S}, \mathsf {S}) + 1 0 \varepsilon_ {\text {m a c h}} \operatorname {t r a c e} (k (\mathsf {S}, \mathsf {S})) \cdot \mathbf {I}) ^ {- 1} T g (\mathsf {S}).
$$
Here,  $\varepsilon_{\mathrm{mach}} = 2^{-52}$  is the double precision machine epsilon.
E Randomly pivoted Cholesky and Nyström approximation
In this section, we describe the connection between the RPCHOLESKY algorithm and Nyström approximation. We hope that this explanation provides context that will be helpful in understanding the RPCHOLESKY algorithm and the proofs of theorems 1 and 4.
To make the following discussion more clear, we begin by rewriting the basic RPCHOLESKY algorithm (see algorithm 1) in algorithm 5 to avoid overwriting the kernel  $k$  at each step:
From this rewriting, we see that in addition to selecting quadrature nodes  $s_1,\ldots ,s_n$ , the RPC-HOLESKY algorithm builds an approximation  $\hat{k}_n$  to the kernel  $k$ . Indeed, one can verify that the approximation  $\hat{k}_n$  produced by this algorithm is precisely the Nyström approximation (definition 2).
Proposition 8. The output  $\hat{k}_n$  of algorithm 5 is equal to the Nyström approximation  $k_{\{s_1,\dots ,s_n\}}$ .
Proof. Proceed by induction on  $n$ . The base case  $n = 0$  is immediate. Fix  $n$  and let  $S := \{s_1, \ldots, s_n\}$ ,  $s' := s_{n+1}$ , and  $S' := \{s_1, \ldots, s_n, s'\}$ . Write the Nyström approximation  $k_{S'}$  as
$$
k _ {\mathsf {S} ^ {\prime}} = [ k (\cdot , \mathsf {S}) \quad k (\cdot , s ^ {\prime}) ] \left[ \begin{array}{c c} k (\mathsf {S}, \mathsf {S}) & k (\mathsf {S}, s ^ {\prime}) \\ k (s ^ {\prime}, \mathsf {S}) & k (s ^ {\prime}, s ^ {\prime}) \end{array} \right] ^ {- 1} \left[ \begin{array}{c} k (\mathsf {S}, \cdot) \\ k (s ^ {\prime}, \cdot) \end{array} \right]
$$
Algorithm 5 RPCHOLESKY: unoptimized implementation without overwriting
Output: Quadrature points  $s_1, \ldots, s_n$  and Nyström approximation  $\hat{k}_n = k_{\{s_1, \ldots, s_n\}}$  to  $k$
1: Initialize  $k_{0}\gets k,\tilde{k}_{0}\gets 0$
2: for  $i = 1,2,\dots ,n$  do
3: Sample  $s_i$  from the probability measure  $k(x, x) \, \mathrm{d}\mu(x) / \int_{\mathsf{X}} k(x, x) \, \mathrm{d}\mu(x)$
4: Update Nyström approximation  $\hat{k}_i\gets \hat{k}_{i - 1} + k_{i - 1}(\cdot ,s_i)k_{i - 1}(s_i,s_i)^{-1}k_{i - 1}(s_i,\cdot)$
5: Update kernel  $k_{i}\gets k_{i - 1} - k_{i - 1}(\cdot ,s_{i})k_{i - 1}(s_{i},s_{i})^{-1}k_{i - 1}(s_{i},\cdot)$
6: end for
Compute a block  $LDL^{\top}$  factorization of the matrix in the middle of this expression:
$$
\left[ \begin{array}{c c} k (\mathsf {S}, \mathsf {S}) & k (\mathsf {S}, s ^ {\prime}) \\ k (s ^ {\prime}, \mathsf {S}) & k (s ^ {\prime}, s ^ {\prime}) \end{array} \right] = \left[ \begin{array}{c c} \mathbf {I} & \mathbf {0} \\ k (s ^ {\prime}, \mathsf {S}) k (\mathsf {S}, \mathsf {S}) ^ {- 1} & 1 \end{array} \right] \left[ \begin{array}{c c} k (\mathsf {S}, \mathsf {S}) & \mathbf {0} \\ \mathbf {0} ^ {\top} & k _ {n} (s ^ {\prime}, s ^ {\prime}) \end{array} \right] \left[ \begin{array}{c c} \mathbf {I} & k (\mathsf {S}, \mathsf {S}) ^ {- 1} k (\mathsf {S}, s ^ {\prime}) \\ \mathbf {0} ^ {\top} & 1 \end{array} \right].
$$
Here, we've used the induction hypothesis
$$
k _ {n} \left(s ^ {\prime}, s ^ {\prime}\right) = k \left(s ^ {\prime}, s ^ {\prime}\right) - k _ {\mathbb {S}} \left(s ^ {\prime}, s ^ {\prime}\right) = k \left(s ^ {\prime}, s ^ {\prime}\right) - k \left(s ^ {\prime}, \mathbb {S}\right) k (\mathbb {S}, \mathbb {S}) ^ {- 1} k (\mathbb {S}, s ^ {\prime}). \tag {24}
$$
$$
\begin{array}{l} k _ {\mathsf {S} ^ {\prime}} = \left[ k (\cdot , \mathsf {S}) \quad k (\cdot , s ^ {\prime}) \right] \left[ \begin{array}{c c} k (\mathsf {S}, \mathsf {S}) & k (\mathsf {S}, s ^ {\prime}) \\ k (s ^ {\prime}, \mathsf {S}) & k (s ^ {\prime}, s ^ {\prime}) \end{array} \right] ^ {- 1} \left[ \begin{array}{c} k (\mathsf {S}, \cdot) \\ k (s ^ {\prime}, \cdot) \end{array} \right] \\ = [ k (\cdot , \mathsf {S}) \quad k (\cdot , s ^ {\prime}) ] \left[ \begin{array}{c c} \mathbf {I} & - k (\mathsf {S}, \mathsf {S}) ^ {- 1} k (\mathsf {S}, s ^ {\prime}) \\ \mathbf {0} ^ {\top} & 1 \end{array} \right] \left[ \begin{array}{c c} k (\mathsf {S}, \mathsf {S}) ^ {- 1} & \mathbf {0} \\ \mathbf {0} ^ {\top} & k _ {n} (s ^ {\prime}, s ^ {\prime}) ^ {- 1} \end{array} \right] \\ \cdot \left[ \begin{array}{c c} \mathbf {I} & \mathbf {0} \\ - k (s ^ {\prime}, \mathsf {S}) k (\mathsf {S}, \mathsf {S}) ^ {- 1} & 1 \end{array} \right] \left[ \begin{array}{c} k (\mathsf {S}, \cdot) \\ k (s ^ {\prime}, \cdot) \end{array} \right] \\ = \left[ k (\cdot , \mathsf {S}) \quad k _ {n} (\cdot , s ^ {\prime}) \right] \left[ \begin{array}{c c} k (\mathsf {S}, \mathsf {S}) ^ {- 1} & \mathbf {0} \\ \mathbf {0} ^ {\top} & k _ {n} \left(s ^ {\prime}, s ^ {\prime}\right) ^ {- 1} \end{array} \right] \left[ \begin{array}{l} k (\mathsf {S}, \cdot) \\ k _ {n} (\cdot , s ^ {\prime}) \end{array} \right] \tag {25} \\ = \hat {k} _ {n} + k _ {n} (\cdot , s ^ {\prime}) k _ {n} \left(s ^ {\prime}, s ^ {\prime}\right) ^ {- 1} k _ {n} \left(s ^ {\prime}, \cdot\right) = \hat {k} _ {n + 1}. \\ \end{array}
$$
For (25), we used the induction hypothesis (24) again. Having established that  $k_{S'} = k_{n+1}$ , the proposition is proven.