SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
p (s _ {i}; \alpha) = \frac {1}{\alpha} \cdot \frac {k (s _ {i} , s _ {i}) - k _ {\mathsf {S}} (s _ {i} , s _ {i})}{k (s _ {i} , s _ {i})} \quad \text {w h e r e} \quad \alpha = \max  _ {x \in \mathsf {X}} \frac {k (x , x) - k _ {\mathsf {S}} (x , x)}{k (x , x)}.
$$
This suggests the following scheme: Initialize with  $\alpha = 1$  and run the algorithm with acceptance probability  $p(s_i; \alpha)$ . If we perform many loop iterations without an acceptance, we then recompute the optimal  $\alpha$  by solving an optimization problem. The resulting procedure is shown in algorithm 4.
If the optimization problem for  $\alpha$  is solved to global optimality, then algorithm 4 produces exact RPCHOLESKY samples. The downside of algorithm 4 is the need to solve a (typically) nonconvex global optimization problem to compute the optimal  $\alpha$ . Fortunately, in our experiments (section 5), only a small number of optimization problems ( $\leq 10$ ) are needed to produce a sample of  $n = 1000$  nodes. In the setting where the optimization problem is tractable, the speedup of algorithm 4 can
Algorithm 4 RPCHOLESKY with optimized rejection sampling
Input: Kernel  $k:\mathsf{X}\times \mathsf{X}\to \mathbb{R}$  and number of quadrature points  $n$    
Output: Quadrature points  $s_1,\ldots ,s_n$    
1: Initialize  $\pmb {L}\gets \mathbf{0}_{n\times n},i\gets 1,\mathsf{S}\gets \emptyset ,$  trials  $\leftarrow 0,\alpha \leftarrow 1$    
2: while  $i\leq n$  do   
3: trials  $\leftarrow$  trials + 1   
4: Sample  $s_i$  from the probability measure  $k(x,x)\mathrm{d}\mu (x)$    
5:  $(d,c)\gets \mathrm{RESIDUALKERN}(s_i,\mathsf{S},\mathbf{L},k)$  >Helper subroutine algorithm 3   
6: Draw a uniform random variable  $U\sim \mathrm{UNIF}[0,1]$    
7: if  $U <   (1 / \alpha)\cdot d / k(s_i,s_i)$  then Accept with probability  $d / k(s_i,s_i)$    
8: S  $\leftarrow$  S U  $\{s_i\} ,i\leftarrow i + 1$  ,trials  $\leftarrow 0$    
9: L(i,1:i)  $\leftarrow \left[c^{\top}\quad \sqrt{d}\right]$    
10: end if   
11: if trials  $\geq$  TRIALS_MAX then We use TRIALS_MAX  $\in$  [25, 1000]   
12:  $\alpha \leftarrow \max_{x\in X}\mathrm{RESIDUALKERN}(x,\mathsf{S},\mathbf{L},k) / k(x,x)$    
13: trials  $\leftarrow 0$    
14: end if   
15: end while
be immense. To produce  $n = 200$  RPCHOLESKY samples for the space  $[0,1]^3$  with the kernel (13), algorithm 4 requires just 0.19 seconds compared to 7.47 seconds for algorithm 2.
5 Comparison of methods on a benchmark example
Having demonstrated the versatility of RPCHOLESKY for general spaces  $X$ , measures  $\mu$ , and kernels  $k$  in fig. 1, we now present a benchmark example from the kernel quadrature literature to compare RPCHOLESKY with other methods. Consider the periodic Sobolev space  $H_{\mathrm{per}}^{s}([0,1])$  with kernel
$$
k (x, y) = 1 + 2 \sum_ {m = 1} ^ {\infty} m ^ {- 2 s} \cos (2 \pi m (x - y)) = 1 + \frac {(- 1) ^ {s - 1} (2 \pi) ^ {2 s}}{(2 s) !} B _ {2 s} (\{x - y \}),
$$
where  $B_{2s}$  is the (2s)th Bernoulli polynomial and  $\{\cdot\}$  reports the fractional part of a real number [3, p. 7]. We also consider  $[0,1]^3$  equipped with the product kernel
$$
k ^ {\otimes 3} \left(\left(x _ {1}, x _ {2}, x _ {3}\right), \left(y _ {1}, y _ {2}, y _ {3}\right)\right) = k \left(x _ {1}, y _ {1}\right) k \left(x _ {2}, y _ {2}\right) k \left(x _ {3}, y _ {3}\right). \tag {13}
$$
We quantify the performance of nodes  $S$  and weights  $w$  using  $\operatorname{Err}(S, w; g)$ , which can be computed in closed form [22, Eq. (14)]. We set  $\mu := \mathrm{UNIF}[0, 1]^d$  and  $g \equiv 1$ .
We compare the following schemes:
- Monte Carlo, IID kernel quadrature. Nodes  $s_1, \ldots, s_n \stackrel{\mathrm{iid}}{\sim} \mu$  with uniform weights  $w_i = 1/n$  (Monte Carlo) and optimal weights (8) (IID).
- Thinning. Nodes  $s_1, \ldots, s_n$  thinned from  $n^2$  iid samples from  $\mu = \mathrm{UNIF}[0,1]$  using kernel thinning [17, 18] with the COMPRESS++ algorithm [38] with optimal weights (8).
- Continuous volume sampling (CVS). Nodes  $s_1, \ldots, s_n$  drawn from the volume sampling distribution [8, Def. 1] by Markov chain Monte Carlo with optimal weights (8).
- RPCHOLESKY. Nodes  $s_1, \ldots, s_n$  sampled by RPCHOLESKY using algorithm 4 with the optimal weights (8).
- Positively weighted kernel quadrature (PWKQ). Nodes and weights computed by the Nystrom+empirical+opt method as described on [22, p. 9].
See appendix D for more details about our numerical experiments. Experiments were run on a MacBook Pro with a 2.4 GHz 8-Core Intel Core i9 CPU and 64 GB 2667 MHz DDR4 RAM. Our code is available at https://github.com/epperly/RPCholesky-Kernel-Quadrature.
Errors for the different methods are shown in fig. 2 (left panels). We see that RPCHOLESKY consistently performs among the best methods at every value of  $n$ , numerically achieving the rate
(a)  $s = 3,d = 1$
(b)  $s = 3, d = 3$
Figure 2: Benchmark example: Sobolev space. Mean quadrature error  $\mathrm{Err}(\mathsf{S},\boldsymbol {w};g)$  (left) and sampling time (right) for different methods (100 trials) for  $s = 1$ $d = 3$  (top) and  $s = d = 3$  (bottom). Shaded regions show  $10\% /90\%$  quantiles.
of convergence of the fastest method for each problem. Particularly striking is the result that RPCHOLESKY sampling's performance is either very close to or better than that of continuous volume sampling, despite the latter's slightly stronger theoretical properties.
Figure 2 (right panels) show the sampling times for each of the nonuniform sampling methods. RPCHOLESKY sampling is the fastest by far. To sample 128 nodes for  $s = 3$  and  $d = 1$ , RPCHOLESKY was  $17 \times$  faster than continuous volume sampling,  $52 \times$  faster than thinning, and  $170 \times$  faster than PWKQ.
To summarize our numerical evaluation (comprising fig. 2 and fig. 1 from the introduction), we find that RPCHOLESKY is among the most accurate kernel quadrature methods tested. To us, the strongest benefits of RPCHOLESKY (supported by these experiments) are the method's speed and flexibility. These virtues make it possible to apply spectrally accurate kernel quadrature in scenarios where it would have been computationally intractable before.
6 Application: Analyzing large chemistry datasets
While our focus thus far has been on infinite domains  $\mathsf{X}$ , the kernel quadrature formalism can also be applied to a finite set  $\mathsf{X}$  of data points. Let  $\mu = \mathrm{UNIF}\mathsf{X}$  be the uniform distribution and set  $g \equiv 1$ . Here, the task is to exhibit  $n$  nodes  $\mathsf{S} = \{s_1,\dots ,s_n\} \subseteq \mathsf{X}$  and weights  $\boldsymbol {w}\in \mathbb{R}^n$  such that the average of every "smooth" function  $f:\mathsf{X}\to \mathbb{R}$  over the whole dataset is well-approximated by a sum:
$$
\operatorname {m e a n} (f) := \frac {1}{| \mathrm {X} |} \sum_ {x \in \mathrm {X}} f (x) \approx \sum_ {i = 1} w _ {i} f \left(s _ {i}\right). \tag {14}
$$
Here is an application to chemistry. Let  $\mathsf{X}$  denote a large set of compounds of interest, and let  $f: \mathsf{X} \to \mathbb{R}$  denote a target chemical property such as specific heat capacity. We are interested
in computing  $\mathrm{mean}(f)$ . Unfortunately, evaluating  $f$  on each  $x\in \mathsf{X}$  requires an expensive density functional theory computation, so it can be prohibitively expensive to evaluate  $f$  on every  $x\in \mathsf{X}$ . Fortunately, we can obtain fast approximations to  $\mathrm{mean}(f)$  using (14), which only require evaluating  $f$  on a much smaller set of size  $|\mathsf{S}|\ll |\mathsf{X}|$ .
Our experimental setup is as follows. For  $\mathsf{X}$ , we use  $2 \times 10^{4}$  randomly selected points from the QM9 dataset [33, 37, 40]. We represent each compound as a vector in  $\mathbb{R}^{1500}$  using the many-body tensor representation [25] computed with the DScribe package [24]. Choose  $k$  to be a Gaussian kernel with bandwidth chosen by median heuristic [20] on a further subsample of  $10^{3}$  random points. We omit continuous volume sampling, thinning, and positively weighted kernel quadrature because of their computational cost. In their place, we add the greedy Nyström method [19] with optimal weights (8).
(b)
Figure 3: Application: chemistry. Worst-case quadrature error (left) and mean relative error for estimation of the average value of the isotropic polarizability function  $f(x)$  (right) for different methods (100 trials). Shaded regions show  $10\% / 90\%$  quantiles.
Results are shown in fig. 3. Figure 3a shows the worst-case quadrature error (9). For this metric, IID and randomly pivoted Cholesky are the definitive winners, with randomly pivoted Cholesky being slightly better. Figure 3b shows the mean relative error for the kernel quadrature estimates of the mean isotropic polarizability of the compounds in X. For sufficiently large  $n$ , randomly pivoted Cholesky has the smallest error, beating IID and Monte Carlo by a factor of three at  $n = 512$ .
7 Conclusions, Limitations, and Possibilites for Future Work
In this article, we developed continuous RPCHOLESKY sampling for kernel quadrature. Theorem 1 demonstrates RPCHOLESKY kernel quadrature achieves near-optimal error rates. Numerical results (fig. 2) hint that its practical performance might be even better than that suggested by our theoretical analysis and fully comparable with kernel quadrature based on the much more computationally expensive continuous volume sampling distribution. RPCHOLESKY supports performant rejection sampling algorithms (algorithms 2 and 4), which facilitate implementation for general spaces  $X$ , measures  $\mu$ , and kernels  $k$  with ease.
We highlight three limitations of RPCHOLESKY kernel quadrature that would be worth addressing in future work. First, given the comparable performance of RPCHOLESKY and continuous volume sampling in practice, it would be desirable to prove stronger error bounds for RPCHOLESKY sampling or find counterexamples which demonstrate a separation between the methods. Second, it would be worth developing improved sampling algorithms for RPCHOLESKY which avoid the need for global optimization steps. Third, all known spectrally accurate kernel quadrature methods require integrals of the form  $\int_{\mathsf{X}}k(x,y)g(y)\mathrm{d}\mu (y)$ , which may not be available. RPCHOLESKY kernel quadrature requires them for the computation of the weights (8). Developing spectrally accurate kernel quadrature schemes that avoid such integrals remains a major open problem for the field.
We thank Lester Mackey, Eliza O'Reilly, Joel Tropp, and Robert Webber for helpful discussions and feedback.
Disclaimer. This report was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government nor any agency thereof, nor any of their employees, makes any warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government or any agency thereof. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof.
[1] Nima Anari, Yang P. Liu, and Thuy-Duong Vuong. Optimal sublinear sampling of spanning trees and determinantal point processes via average-case entropic independence. In 63rd Annual Symposium on Foundations of Computer Science, pages 123-134. IEEE, October 2022.
[2] Nima Anari, Shayan Oveis Gharan, and Alireza Rezaei. Monte Carlo Markov chain algorithms for sampling strongly Rayleigh distributions and determinantal point processes. In Proceedings of the 29th Conference on Learning Theory, pages 103-115. PMLR, June 2016.
[3] Francis Bach. On the equivalence between kernel quadrature rules and random feature expansions. Journal of Machine Learning Research, 18(1):714-751, January 2017.
[4] Francis Bach, Simon Lacoste-Julien, and Guillaume Obozinski. On the equivalence between herding and conditional gradient algorithms. Proceedings of the 29th International Conference on Machine Learning, March 2012.
[5] Rémi Bardenet and Adrien Hardy. Monte Carlo with determinantal point processes. The Annals of Applied Probability, 30(1):368-417, February 2020.
[6] Ayoub Belhadji. An analysis of Ermakov-Zolotukhin quadrature using kernels. In Advances in Neural Information Processing Systems, volume 34, pages 27278-27289. Curran Associates, Inc., 2021.
[7] Ayoub Belhadji, Rémi Bardenet, and Pierre Chainais. Kernel quadrature with DPPs. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[8] Ayoub Belhadji, Rémi Bardenet, and Pierre Chainais. Kernel interpolation with continuous volume sampling. In Proceedings of the 37th International Conference on Machine Learning, pages 725-735. PMLR, November 2020.
[9] Daniele Calandriello, Michal Derezinski, and Michal Valko. Sampling from a  $k$ -DPP without looking at all items. In Advances in Neural Information Processing Systems, volume 33, pages 6889-6899. Curran Associates, Inc., 2020.
[10] Yifan Chen, Ethan N. Epperly, Joel A. Tropp, and Robert J. Webber. Randomly pivoted Cholesky: Practical approximation of a kernel matrix with few entry evaluations, February 2023. arXiv:2207.06503.
[11] Yutian Chen, Max Welling, and Alex Smola. Super-samples from kernel herding. In Proceedings of the Twenty-Sixth Conference on Uncertainty in Artificial Intelligence, page 109-116. AUAI Press, 2010.
[12] Amit Deshpande, Luis Rademacher, Santosh Vempala, and Grant Wang. Matrix approximation and projective clustering via volume sampling. In Proceedings of the seventeenth annual ACM-SIAM symposium on Discrete algorithm, pages 1117-1126. ACM, 2006.
[32] Supratik Paul, Konstantinos Chatzilygeroudis, Kamil Ciosek, Jean-Baptiste Mouret, Michael A. Osborne, and Shimon Whiteson. Alternating optimisation and quadrature for robust control. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), pages 3925-3933. AAAI Press, February 2018.
[33] Raghunathan Ramakrishnan, Pavlo O. Dral, Matthias Rupp, and O. Anatole von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data, 1(1):140022, December 2014.
[34] Alireza Rezaei and Shayan Oveis Gharan. A polynomial time MCMC method for sampling from continuous determinantal point processes. In Proceedings of the 36th International Conference on Machine Learning, pages 5438-5447. PMLR, May 2019.
[35] Christian P. Robert. The Bayesian Choice: From Decision-Theoretic Foundations to Computational Implementation. Springer Texts in Statistics. Springer, second edition, 2007.
[36] Christian P. Robert and George Casella. Monte Carlo Statistical Methods. Springer Texts in Statistics. Springer, second edition, 2004.
[37] Lars Ruddigkeit, Ruud van Deursen, Lorenz C. Blum, and Jean-Louis Reymond. Enumeration of 166 billion organic small molecules in the chemical universe database GDB-17. Journal of Chemical Information and Modeling, 52(11):2864–2875, November 2012.
[38] Abhishek Shetty, Raaz Dwivedi, and Lester Mackey. Distribution compression in near-linear time. In Tenth International Conference on Learning Representations, April 2022.
[39] Ingo Steinwart and Clint Scovel. Mercer's theorem on general domains: On the interaction between measures, kernels, and RKHSs. Constructive Approximation, 35(3):363-417, June 2012.
[40] Annika Stuke, Milica Todorovic, Matthias Rupp, Christian Kunkel, Kunal Ghosh, Lauri Hima-nen, and Patrick Rinke. Chemical diversity in molecular orbital energy predictions with kernel ridge regression. The Journal of Chemical Physics, 150(20):204121, May 2019.
[41] Kazuma Tsuji, Ken'ichiro Tanaka, and Sebastian Pokutta. Pairwise conditional gradients without swap steps and sparser kernel herding. In Proceedings of the International Conference on Machine Learning, 2022.
[42] Christopher K. I. Williams and Matthias Seeger. Using the Nyström method to speed up kernel machines. In Proceedings of the 13th International Conference on Neural Information Processing Systems, pages 661-667. MIT Press, January 2000.
A Proof of theorem 1
We start off by proving proposition 3:
Proof of proposition 3. Equation (12) follows from a direct calculation:
$$
\sum_ {i = 1} ^ {n} w _ {i} k (\cdot , s _ {i}) = k (\cdot , \mathsf {S}) \boldsymbol {w} = \int_ {\mathsf {X}} k (\cdot , \mathsf {S}) k (\mathsf {S}, \mathsf {S}) ^ {- 1} k (\mathsf {S}, x) g (x) \mathrm {d} \mu (x) = T _ {\mathsf {S}} g.
$$
To establish the second claim, use the fact [3, §2.1] that  $T^{-1/2} : \mathcal{H} \to \mathsf{L}^2(\mu)$  is an isometry and invoke the definition (9) of  $\operatorname{Err}(\mathsf{S}; g)$ :
$$
\begin{array}{l} \operatorname {E r r} ^ {2} (\mathsf {S}; g) = \| (T - T _ {\mathsf {S}}) g \| _ {\mathcal {H}} ^ {2} = \left\| T ^ {- 1 / 2} (T - T _ {\mathsf {S}}) g \right\| _ {\mathsf {L} ^ {2} (\mu)} ^ {2} \\ = \langle g, (T - T _ {\mathfrak {S}}) T ^ {- 1} (T - T _ {\mathfrak {S}}) g \rangle_ {\mathrm {L} ^ {2} (\mu)} \leq \langle g, (T - T _ {\mathfrak {S}}) g \rangle_ {\mathrm {L} ^ {2} (\mu)}. \\ \end{array}
$$
The last inequality follows because  $T - T_{\mathsf{S}}$  and  $T_{\mathsf{S}} = T - (T - T_{\mathsf{S}})$  are both positive semidefinite.
For the rest of our proof of theorem 1, we employ the following notation. Let  $\preceq$  denote the Loewner order for bounded, linear operators on  $\mathsf{L}^2 (\mu)$ . That is,  $A\preceq B$  if  $B - A$  is positive semidefinite (psd). Let  $k_{i}$  denote the residual
$$
k _ {i} := k - k _ {\{s _ {1}, \dots , s _ {i} \}}
$$
for the approximate kernel associated with the first  $i$  nodes. (See proposition 8 below.) The associated integral operator is
$$
T _ {i} f := \int_ {\mathsf {X}} k _ {i} (\cdot , x) f (x) \mathrm {d} \mu (x).
$$
To analyze RPCHOLESKY, we begin by analyzing a single step of the procedure. The expected value of the residual kernel after one step,  $k_{1}$ , is
$$
\mathbb {E} k _ {1} (x, y) = k (x, y) - \frac {\int_ {\mathsf {X}} k (x , s) k (s , y) \mathrm {d} \mu (s)}{\int_ {\mathsf {X}} k (z , z) \mathrm {d} \mu (z)}.
$$
Consequently, the expected value of the integral operator  $T_{1}$  is
$$
\Phi (T) := \mathbb {E} \left[ T _ {1} \right] = T - \frac {T ^ {2}}{\operatorname {T r} T}. \tag {15}
$$
The map  $\Phi$  enjoys several useful properties [10, Lem. 3.2].
Proposition 5. The map  $\Phi$  is positive, monotone, and concave with respect to the order  $\preceq$  on the set of trace-class psd operators on  $\mathsf{L}^2 (\mu)$ . That is, for trace-class psd operators  $A,B$ ,
$$
0 \preceq A \preceq B \Longrightarrow 0 \preceq \Phi (A) \preceq \Phi (B) \tag {16}
$$
and, for  $\theta \in [0,1]$
$$
\theta \Phi (A) + (1 - \theta) \Phi (B) \preceq \Phi (\theta A + (1 - \theta) B). \tag {17}
$$
Proof. We reproduce the proof from [10, Lem. 3.2] for completeness.
Positivity. Assume  $A \succeq 0$ . Observe that  $\mathrm{Id} \succeq A / \operatorname{Tr} A$ , where  $\mathrm{Id}$  denotes the identity operator. Thus
$$
\Phi (A) = A \left(\operatorname {I d} - \frac {A}{\operatorname {T r} A}\right) \succeq 0,
$$
This establishes that  $\Phi$  is positive:  $A\succeq 0$  implies  $\Phi (A)\succeq 0$
Concavity. For  $\theta \in [0,1]$ , denote  $\overline{\theta} \coloneqq 1 - \theta$ . Then
$$
\Phi (\theta A + \bar {\theta} B) - \theta \Phi (A) - \bar {\theta} \Phi (B) = \frac {\theta \bar {\theta}}{\theta \operatorname {T r} A + \bar {\theta} \operatorname {T r} B} \left(\sqrt {\frac {\operatorname {T r} B}{\operatorname {T r} A}} A - \sqrt {\frac {\operatorname {T r} A}{\operatorname {T r} B}} B\right) \succeq 0.
$$
This establishes the concavity claim (17).
Monotonicity. Suppose  $0 \preceq A \preceq B$ . Observe the map  $\Phi$  is positive homogeneous: For  $\theta > 0$ ,  $\Phi(\theta A) = \theta \Phi(A)$ . Thus,
$$
\Phi (B) = \Phi (A + (B - A)) = 2 \Phi \left(\frac {A + (B - A)}{2}\right).
$$
Invoke concavity (17) to obtain
$$
\Phi (B) = 2 \Phi \left(\frac {A + (B - A)}{2}\right) \succeq \Phi (A) + \Phi (B - A) \succeq \Phi (A).
$$
In the last inequality, we use the positivity  $\Phi(B - A) \succeq 0$ . This completes the proof of (16).
We can use the map  $\Phi$  to bound the residual integral operator  $T_{i}$ :
Proposition 6. For  $i \geq 0$ , we have the bound
$$
\mathbb {E} \left[ T _ {i} \right] \preceq \Phi^ {i} (T), \tag {18}
$$
where  $\Phi^i$  denotes the  $i$ -fold composition of  $\Phi$ .
Proof. The  $i$ th step of RPCHOLESKY applies one step of RPCHOLESKY to  $k_{i-1}$ . Thus,
$$
\mathbb {E} \left[ T _ {i} \right] = \mathbb {E} \left[ \mathbb {E} \left[ T _ {i} \mid s _ {1}, \dots , s _ {i - 1} \right] \right] = \mathbb {E} \left[ \Phi \left(T _ {i - 1}\right) \right].
$$
Now, we can apply concavity (17) to obtain
$$
\mathbb {E} \left[ T _ {i} \right] = \mathbb {E} \left[ \Phi \left(T _ {i - 1}\right) \right] \preceq \Phi \left(\mathbb {E} \left[ T _ {i - 1} \right]\right).
$$
Applying this result twice and using monotonicity (16) yields:
$$
\mathbb {E} [ T _ {i} ] \preceq \Phi (\mathbb {E} [ T _ {i - 1} ]) \text {a n d} \mathbb {E} [ T _ {i - 1} ] \preceq \Phi (\mathbb {E} [ T _ {i - 2} ]) \Longrightarrow \mathbb {E} [ T _ {i} ] \preceq \Phi (\Phi (\mathbb {E} [ T _ {i - 2} ]).
$$
Iterating this argument, we obtain the desired conclusion (18).
We now have the tools to prove theorem 1. We actually prove the following slight refinement:
Theorem 7. Let  $\mathsf{S} = \{s_1, \ldots, s_n\}$  be generated by the RPCHOLESKY sampling algorithm. For any function  $g \in \mathsf{L}^2(\mu)$ , nonnegative integer  $r \geq 0$ , and real numbers  $a, \varepsilon > 0$ , we have
$$
\mathbb {E} \operatorname {E r r} ^ {2} (\mathsf {S}; g) \leq \| g \| _ {\mathsf {L} ^ {2} (\mu)} ^ {2} \cdot \left(a + \varepsilon \sum_ {i = r + 1} ^ {\infty} \lambda_ {i}\right)
$$
$$
n \geq r \log \left(\frac {\lambda_ {1}}{a}\right) + \frac {1}{\varepsilon}. \tag {19}
$$
Theorem 1 immediately follows from theorem 7 by taking  $\varepsilon = \delta /2$  and  $a = (\delta /2)\sum_{i = r + 1}^{\infty}\lambda_{i}$ . We now prove theorem 7.
Proof of theorem 7. In the standing notation,  $T - T_{\mathsf{S}}$  corresponds to  $T_{n}$ . Thus, by proposition 6, we have
$$
\mathbb {E} \langle g, (T - T _ {\mathsf {S}}) g \rangle_ {\mathsf {L} ^ {2} (\mu)} = \langle g, \mathbb {E} [ T _ {n} ] g \rangle_ {\mathsf {L} ^ {2} (\mu)} \leq \langle g, \Phi^ {n} (T) g \rangle_ {\mathsf {L} ^ {2} (\mu)} \leq \| g \| _ {\mathsf {L} ^ {2} (\mu)} ^ {2} \cdot \lambda_ {\max } (\Phi^ {n} (T)).
$$
Thus, it is sufficient to show that
$$
\lambda_ {\max } \left(\Phi^ {n} (T)\right) \leq a + \varepsilon \sum_ {i = r + 1} ^ {\infty} \lambda_ {i} \tag {20}
$$
holds under condition (19).
Using the definition (15) of  $\Phi$ , we see that the eigenvalues  $\lambda_1^{(i)} \geq \lambda_2^{(i)} \geq \dots$  of  $\Phi^i(T)$  are given by the following recurrence
$$
\lambda_ {j} ^ {(i)} = \lambda_ {j} ^ {(i - 1)} - \frac {\left(\lambda_ {j} ^ {(i - 1)}\right) ^ {2}}{\sum_ {\ell = 1} ^ {\infty} \lambda_ {\ell} ^ {(i - 1)}} \quad \text {f o r} i, j = 1, 2, \dots \tag {21}
$$
with initial condition  $\lambda_j^{(0)} = \lambda_j$  for all  $j$ . From this formula, we immediately conclude that each  $\lambda_j^{(i)}$  is nonnegative and nonincreasing in  $i$ . In addition, the ordinal ranking  $\lambda_1^{(i)} \geq \lambda_2^{(i)} \geq \dots$  is preserved for each  $i$ . Indeed, if  $\lambda_{j+1}^{(i-1)} - \lambda_j^{(i-1)} \geq 0$ , then