SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We now turn to the proofs of our main results, Theorems 5, 6 and 8, which for convenience we restate below. The idea of these proofs is to extend the result of Lemma 22 from input locations to the whole manifold  $\mathcal{M}$  using an appropriate concentration inequality. To this end, the proof closely follows the one of Theorem 2 in van der Vaart and van Zanten [56], but relies on Lemma 22 proved above and on the concentration inequality we prove in Appendix D along with some important sample differentiability properties of the prior processes.
$$
\mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \mathbb {E} _ {f \sim \Pi (\cdot | \boldsymbol {x}, \boldsymbol {y})} \| f - f _ {0} \| _ {L ^ {2} (p _ {0})} ^ {2} \leq C n ^ {- \frac {2 \min  (\beta , \nu)}{2 \nu + d}}. \tag {9}
$$
Theorem 6. Let  $f$  be a Riemannian Matérn Gaussian process prior on  $\mathcal{M}$  with smoothness parameter  $\nu > d/2$ , modified to truncate the infinite sum to at least  $J_{n} \geq cn^{\frac{d(\min(1,\nu/\beta))}{2\nu + d}}$  terms, and let  $f_{0}$  satisfy Assumption 3. Then there is a  $C > 0$  such that
$$
\mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \mathbb {E} _ {f \sim \Pi (\cdot | \boldsymbol {x}, \boldsymbol {y})} \| f - f _ {0} \| _ {L ^ {2} \left(p _ {0}\right)} ^ {2} \leq C n ^ {- \frac {2 \min  (\beta , \nu)}{2 \nu + d}}. \tag {10}
$$
Proof of Theorems 5, 6 and 8. To ease notation, we denote the expectations under the true data generating process by  $\mathbb{E}_{\boldsymbol{x},\boldsymbol{y}} = \mathbb{E}_{\boldsymbol{x},\boldsymbol{y}}^{(f_0)}$  and  $\mathbb{E}_{\boldsymbol{y}|\boldsymbol{x}} = \mathbb{E}_{\boldsymbol{y}|\boldsymbol{x}}^{(f_0)}$ , omitting the superscript  $(\cdot)^{f_0}$ . Take  $\varepsilon_n \propto n^{-\frac{\min(\beta,\nu)}{2\nu + d}}$  satisfying  $\varphi_{f_0}(\varepsilon_n / 2) \leq n\varepsilon_n^2$ , noting that such a rate exists by Theorem 21. Then, for each  $n$ , there exists an element  $f_n \in \mathbb{H}_n$ , where  $\mathbb{H}_n$  is the RKHS corresponding to  $\Pi_n$ , satisfying
$$
\left\| f _ {n} \right\| _ {\mathbb {H} _ {n}} ^ {2} \leq n \varepsilon_ {n} ^ {2} \quad \left\| f _ {n} - f _ {0} \right\| _ {L ^ {\infty} (\mathcal {M})} \leq \varepsilon_ {n} / 2. \tag {61}
$$
$$
\begin{array}{l} \varepsilon_ {n} ^ {- 2} \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \mathbb {E} _ {f \sim \Pi_ {n} (\cdot | \boldsymbol {x}, \boldsymbol {y})} \| f - f _ {0} \| _ {L ^ {2} (p _ {0})} ^ {2} \lesssim \varepsilon_ {n} ^ {- 2} \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \mathbb {E} _ {f \sim \Pi_ {n} (\cdot | \boldsymbol {x}, \boldsymbol {y})} \| f _ {n} - f _ {0} \| _ {L ^ {2} (p _ {0})} ^ {2} (62) \\ + \varepsilon_ {n} ^ {- 2} \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \mathbb {E} _ {f \sim \Pi_ {n} (\cdot | \boldsymbol {x}, \boldsymbol {y})} \| f - f _ {n} \| _ {L ^ {2} (p _ {0})} ^ {2} (63) \\ \lesssim 1 + \varepsilon_ {n} ^ {- 2} \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \mathbb {E} _ {f \sim \Pi_ {n} (\cdot | \boldsymbol {x}, \boldsymbol {y})} \| f - f _ {n} \| _ {L ^ {2} \left(p _ {0}\right)} ^ {2}. (64) \\ \end{array}
$$
Thus, we can work with  $f - f_{n}$  instead of  $f - f_{0}$ . Define  $\mathcal{B}(r) = \left\{\|f - f_{n}\|_{L^{2}(p_{0})} > \varepsilon_{n}r\right\}$ . Then
$$
\varepsilon_ {n} ^ {- 2} \mathbb {E} _ {f \sim \Pi_ {n} (\cdot | \boldsymbol {x}, \boldsymbol {y})} \| f - f _ {n} \| _ {L ^ {2} \left(p _ {0}\right)} ^ {2}. = \int_ {0} ^ {\infty} 2 r \mathbb {P} _ {f \sim \Pi_ {n} (\cdot | \boldsymbol {x}, \boldsymbol {y})} (\mathcal {B} (r)) \mathrm {d} r. \tag {65}
$$
Fix a  $\gamma$  such that  $d / 2 < \gamma < \nu, \gamma \notin \mathbb{Z}_{>0}$  and  $s > 0, \tau > 0$  and define
$$
\mathcal {B} ^ {(1)} (r) = \left\{2 \| f - f _ {n} \| _ {n} > \varepsilon_ {n} r \right\} \tag {66}
$$
$$
\mathcal {B} ^ {(\mathrm {I I})} (r) = \left\{\| f \| _ {\mathcal {C H} ^ {\gamma} (\mathcal {M})} > \tau \sqrt {n} \varepsilon_ {n} r ^ {s} \right\} \tag {67}
$$
$$
\mathcal {B} ^ {\text {(I I I)}} (r) = \mathcal {B} (r) \backslash \left(\mathcal {B} ^ {\text {(I)}} (r) \cup \mathcal {B} ^ {\text {(I I)}} (r)\right). \tag {68}
$$
Then  $\mathcal{B}(r) \subseteq \mathcal{B}^{(\mathrm{I})}(r) \cup \mathcal{B}^{(\mathrm{II})}(r) \cup \mathcal{B}^{(\mathrm{III})}(r)$ , and thus for an indexed family of events  $\mathcal{A}_r$  to be chosen later, we have
$$
\begin{array}{l} \varepsilon_ {n} ^ {- 2} \mathbb {E} _ {f \sim \Pi_ {n} (\cdot | \boldsymbol {x}, \boldsymbol {y})} \| f - f _ {n} \| _ {L ^ {2} (p _ {0})} ^ {2} \lesssim \int_ {0} ^ {\infty} r \mathbb {P} _ {f \sim \Pi_ {n} (\cdot | \boldsymbol {x}, \boldsymbol {y})} \left(\mathcal {B} ^ {(1)} (r)\right) d r + \int_ {0} ^ {\infty} r \mathbb {1} _ {\mathcal {A} _ {r} ^ {c}} d r (69) \\ + \int_ {0} ^ {\infty} r \mathbb {1} _ {\mathcal {A} _ {r}} \mathbb {P} _ {f \sim \Pi_ {n} (\cdot | \boldsymbol {x}, \boldsymbol {y})} \left(\mathcal {B} ^ {(\mathrm {I I})} (r)\right) \mathrm {d} r + \int_ {0} ^ {\infty} r \mathbb {1} _ {\mathcal {A} _ {r}} \mathbb {P} _ {f \sim \Pi_ {n} (\cdot | \boldsymbol {x}, \boldsymbol {y})} \left(\mathcal {B} ^ {(\mathrm {I I I})} (r)\right) \mathrm {d} r. (70) \\ \end{array}
$$
For the first term, by Lemma 22 applied conditionally on  $x$ , for which we got a bound on the integrated empirical  $L^2$ -norm uniformly on the design points, we have
$$
\mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \int_ {0} ^ {\infty} r \mathbb {P} _ {f \sim \Pi_ {n} (\cdot | \boldsymbol {x}, \boldsymbol {y})} \left(\mathcal {B} ^ {(1)} (r)\right) \mathrm {d} r \lesssim \varepsilon_ {n} ^ {- 2} \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \mathbb {E} _ {f \sim \Pi_ {n} (\cdot | \boldsymbol {x}, \boldsymbol {y})} \| f - f _ {0} \| _ {n} ^ {2} \lesssim \varepsilon_ {n} ^ {- 2} \varepsilon_ {n} ^ {2} = 1. \tag {71}
$$
Moreover, by Lemma 14 of van der Vaart and van Zanten [56] applied with  $r$  in the notation of the reference being equal to  $\sqrt{n}\varepsilon_{n}r^{s}$ , for each  $r > 0$ , the event
$$
\begin{array}{l} \mathcal {A} _ {r} (\boldsymbol {x}) = \left\{\boldsymbol {u} \in \mathbb {R} ^ {n}: \int \frac {p _ {\boldsymbol {y} \mid \boldsymbol {x}} ^ {(f)} (\boldsymbol {u})}{p _ {\boldsymbol {y} \mid \boldsymbol {x}} ^ {(f _ {0})} (\boldsymbol {u})} \mathrm {d} \Pi_ {n} (f) \geq e ^ {- n \varepsilon_ {n} ^ {2} r ^ {2 s}} \mathbb {P} _ {f \sim \Pi_ {n}} \left(\| f - f _ {0} \| _ {L ^ {\infty} (\mathcal {M})} <   \varepsilon_ {n} r ^ {s}\right) \right\} (72) \\ \supseteq \left\{\boldsymbol {u} \in \mathbb {R} ^ {n}: \int \frac {p _ {\boldsymbol {y} | \boldsymbol {x}} ^ {(f)} (\boldsymbol {u})}{p _ {\boldsymbol {y} | \boldsymbol {x}} ^ {(f _ {0})} (\boldsymbol {u})} \mathrm {d} \Pi_ {n} (f) \geq e ^ {- n \varepsilon_ {n} ^ {2} r ^ {2 s}} \mathbb {P} _ {f \sim \Pi_ {n}} \left(\| f - f _ {0} \| _ {n} <   \varepsilon_ {n} r ^ {s}\right) \right\} (73) \\ \end{array}
$$
is such that
$$
p _ {\boldsymbol {y} \mid \boldsymbol {x}} ^ {(f _ {0})} [ \mathcal {A} _ {r} ^ {c} (\boldsymbol {x}) ] \leq e ^ {- n \varepsilon_ {n} ^ {2} r ^ {2 s} / 8}. \tag {74}
$$
It should be noted that the  $\sqrt{n}$  factor disappears because of the discrepancy between the empirical  $L^2$  norm  $\| \cdot \| _n$  and the Euclidean norm used in van der Vaart and van Zanten [56]. By Fubini's Theorem and since  $n\varepsilon_n^2\geq n^{\frac{d}{2\nu + d}}\geq 1$ , the second term is bounded by
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \int_ {0} ^ {\infty} r \mathbb {1} _ {\mathcal {A} _ {r} ^ {c} (\boldsymbol {x})} \mathrm {d} r = \int_ {0} ^ {\infty} r \mathbb {E} _ {\boldsymbol {x}} \left[ \mathbb {E} _ {\boldsymbol {y} | \boldsymbol {x}} \left[ \mathbb {1} _ {\mathcal {A} _ {r} ^ {c} (\boldsymbol {x})} \right] \right] \mathrm {d} r (75) \\ \leq \int_ {0} ^ {\infty} r e ^ {- n \varepsilon_ {n} ^ {2} r ^ {2 s} / 8} d r (76) \\ \leq \int_ {0} ^ {\infty} r e ^ {- r ^ {2 s} / 8} d r = C _ {s} <   \infty . (77) \\ \end{array}
$$
It remains to bound the last two terms. By Bayes' Rule, we have the equality
$$
\begin{array}{l} \mathbb {P} _ {f \sim \Pi_ {n} (\cdot | \boldsymbol {x}, \boldsymbol {y})} \left(\| f \| _ {\mathcal {C H} ^ {\gamma} (\mathcal {M})} > \tau \sqrt {n} \varepsilon_ {n} r ^ {s}\right) = \frac {\int_ {\| f \| _ {\mathcal {C H} ^ {\gamma} (\mathcal {M})} > \tau \sqrt {n} \varepsilon_ {n} r ^ {s}} p _ {\boldsymbol {y} | \boldsymbol {x}} ^ {(f)} (\boldsymbol {y}) \mathrm {d} \Pi_ {n} (f)}{\int p _ {\boldsymbol {y} | \boldsymbol {x}} ^ {(f)} (\boldsymbol {y}) \mathrm {d} \Pi_ {n} (f)} (78) \\ = \frac {\int_ {\| f \| _ {\mathcal {C H} ^ {\gamma} (\mathcal {M})}} > \tau \sqrt {n} \varepsilon_ {n} r ^ {s} \frac {p _ {\boldsymbol {y} | \boldsymbol {x}} ^ {(f)} (\boldsymbol {y})}{p _ {\boldsymbol {y} | \boldsymbol {x}} ^ {(f _ {0})} (\boldsymbol {y})} \mathrm {d} \Pi_ {n} (f)}{\int \frac {p _ {\boldsymbol {y} | \boldsymbol {x}} ^ {(f)} (\boldsymbol {y})}{p _ {\boldsymbol {y} | \boldsymbol {x}} ^ {(f _ {0})} (\boldsymbol {y})} \mathrm {d} \Pi_ {n} (f)} (79) \\ \end{array}
$$
therefore for  $\pmb {y}\in \mathcal{A}_r(\pmb {x})$  , we have
$$
\begin{array}{l} \mathbb {P} _ {f \sim \Pi_ {n} (\cdot | \boldsymbol {x}, \boldsymbol {y})} \left(\| f \| _ {\mathcal {C H} ^ {\gamma} (\mathcal {M})} > \tau \sqrt {n} \varepsilon_ {n} r ^ {s}\right) (80) \\ \leq \frac {e ^ {n \varepsilon_ {n} ^ {2} r ^ {2 s}}}{\mathbb {P} _ {f \sim \Pi_ {n}} \left(\| f - f _ {0} \| _ {L ^ {\infty} (\mathcal {M})} <   \varepsilon_ {n} r ^ {s}\right)} \int_ {\| f \| _ {\mathcal {C H} ^ {\gamma} (\mathcal {M})} > \tau \sqrt {n} \varepsilon_ {n} r ^ {s}} \frac {p _ {\boldsymbol {y} | \boldsymbol {x}} ^ {(f)} (\boldsymbol {y})}{p _ {\boldsymbol {y} | \boldsymbol {x}} ^ {(f _ {0})} (\boldsymbol {y})} d \Pi_ {n} (f). (81) \\ \end{array}
$$
Hence taking expectations, using Tonelli's Theorem, and  $\mathbb{E}_{\boldsymbol{x},\boldsymbol{y}}\frac{p_{\boldsymbol{y}|\boldsymbol{x}}^{(f)}(\boldsymbol{y})}{p_{\boldsymbol{y}|\boldsymbol{x}}^{(f_0)}(\boldsymbol{y})} = 1$  gives
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \left[ \mathbb {1} _ {\mathcal {A} _ {r} (\boldsymbol {x})} \mathbb {P} _ {f \sim \Pi_ {n} (\cdot | \boldsymbol {x}, \boldsymbol {y})} \left(\| f \| _ {\mathcal {C H} ^ {\gamma} (\mathcal {M})} > \tau \sqrt {n} \varepsilon_ {n} r ^ {s}\right) \right] (82) \\ \leq \frac {e ^ {n \varepsilon_ {n} ^ {2} r ^ {2 s}}}{\mathbb {P} _ {f \sim \Pi_ {n}} \left(\| f - f _ {0} \| _ {L ^ {\infty} (\mathcal {M})} <   \varepsilon_ {n} r ^ {s}\right)} \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \int_ {\| f \| _ {\mathcal {C H} ^ {\gamma} (\mathcal {M})} > \tau \sqrt {n} \varepsilon_ {n} r ^ {s}} \frac {p _ {\boldsymbol {y} | \boldsymbol {x}} ^ {(f)} (\boldsymbol {y})}{p _ {\boldsymbol {y} | \boldsymbol {x}} ^ {(f _ {0})} (\boldsymbol {y})} d \Pi_ {n} (f) (83) \\ = \frac {e ^ {n \varepsilon_ {n} ^ {2} r ^ {2 s}}}{\mathbb {P} _ {f \sim \Pi_ {n}} \left(\| f - f _ {0} \| _ {L ^ {\infty} (\mathcal {M})} <   \varepsilon_ {n} r ^ {s}\right)} \int_ {\| f \| _ {\mathcal {C H} ^ {\gamma} (\mathcal {M})} > \tau \sqrt {n} \varepsilon_ {n} r ^ {s}} \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \frac {p _ {\boldsymbol {y} | \boldsymbol {x}} ^ {(f)} (\boldsymbol {y})}{p _ {\boldsymbol {y} | \boldsymbol {x}} ^ {(f _ {0})} (\boldsymbol {y})} d \Pi_ {n} (f) (84) \\ = \frac {e ^ {n \varepsilon_ {n} ^ {2} r ^ {2 s}}}{\mathbb {P} _ {f \sim \Pi_ {n}} \left(\| f - f _ {0} \| _ {L ^ {\infty} (\mathcal {M})} <   \varepsilon_ {n} r ^ {s}\right)} \mathbb {P} _ {f \sim \Pi_ {n}} \left(\| f \| _ {\mathcal {C H} ^ {\gamma} (\mathcal {M})} > \tau \sqrt {n} \varepsilon_ {n} r ^ {s}\right). (85) \\ \end{array}
$$
Therefore, the third term can be bounded as
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \int_ {0} ^ {\infty} r \mathbb {1} _ {\mathcal {A} _ {r}} \mathbb {P} _ {f \sim \Pi_ {n} (\cdot | \boldsymbol {x}, \boldsymbol {y})} \left(\mathcal {B} ^ {(\mathrm {I I})} (r)\right) \mathrm {d} r (86) \\ \leq \int_ {0} ^ {\infty} r \frac {e ^ {n \varepsilon_ {n} ^ {2} r ^ {2 s}}}{\mathbb {P} _ {f \sim \Pi_ {n}} \left(\left\| f - f _ {0} \right\| _ {L ^ {\infty} (\mathcal {M})} <   \varepsilon_ {n} r ^ {s}\right)} \mathbb {P} _ {f \sim \Pi_ {n}} \left(\left\| f \right\| _ {\mathcal {C H} ^ {\gamma} (\mathcal {M})} > \tau \sqrt {n} \varepsilon_ {n} r ^ {s}\right) d r. (87) \\ \end{array}
$$
Now, using Lemma 31, for a possibly small constant  $c > 0$  independent of  $n$ , we have
$$
\mathbb {P} _ {f \sim \Pi_ {n} (\cdot | \boldsymbol {x}, \boldsymbol {y})} \left(\| f \| _ {\mathcal {C H} ^ {\gamma} (\mathcal {M})} > \tau \sqrt {n} \varepsilon_ {n} r ^ {s}\right) \leq e ^ {- c \tau^ {2} n \varepsilon_ {n} ^ {2} r ^ {2 s}}. \tag {88}
$$
Moreover, by using the bound on the concentration function in Theorem 21 and Ghosal and van der Vaart [20], Proposition 11.19, we can assume that
$$
\mathbb {P} _ {f \sim \Pi_ {n}} \left[ \| f - f _ {0} \| _ {L ^ {\infty} (\mathcal {M})} <   \varepsilon_ {n} r ^ {s} \right] \geq e ^ {- c ^ {- 1} n \varepsilon_ {n} ^ {2} r ^ {2 s}}. \tag {89}
$$
Therefore, the third term is bounded by
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \int_ {0} ^ {\infty} r \mathbb {1} _ {\mathcal {A} _ {r} (\boldsymbol {x})} \mathbb {P} _ {f \sim \Pi_ {n} (\cdot | \boldsymbol {x}, \boldsymbol {y})} \left(\mathcal {B} ^ {(\mathrm {I I})} (r)\right) \mathrm {d} r \leq \int_ {0} ^ {\infty} r e ^ {- n \left(c \tau^ {2} - 1\right) \varepsilon_ {n} ^ {2} r ^ {2 s}} e ^ {c ^ {- 1} n \varepsilon_ {n} ^ {2} r ^ {2 s}} \mathrm {d} r (90) \\ \leq \int_ {0} ^ {\infty} r e ^ {- r ^ {2 s}} d r <   \infty (91) \\ \end{array}
$$
if  $\tau^2 c > 2 + c^{-1}$ . It remains to bound the last term. We have, by the same arguments as above, that
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \int_ {0} ^ {\infty} r \mathbb {1} _ {\mathcal {A} _ {r} (\boldsymbol {x})} \mathbb {P} _ {f \sim \Pi_ {n} (\cdot | \boldsymbol {x}, \boldsymbol {y})} \left(\mathcal {B} ^ {(\mathrm {I I I})} (r)\right) \mathrm {d} r (92) \\ = \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \int_ {0} ^ {\infty} r \mathbb {1} _ {\mathcal {A} _ {r} (\boldsymbol {x})} (93) \\ \times \mathbb {P} _ {f \sim \Pi_ {n} (\cdot | \boldsymbol {x}, \boldsymbol {y})} \left(\| f \| _ {\mathcal {C H} ^ {\gamma} (\mathcal {M})} \leq \tau \sqrt {n} \varepsilon_ {n} r ^ {s}, 2 \| f - f _ {n} \| _ {n} \leq \varepsilon_ {n} r \leq \| f - f _ {n} \| _ {L ^ {2} \left(p _ {0}\right)}\right) d r (94) \\ \leq \int_ {0} ^ {\infty} r \frac {e ^ {n \varepsilon_ {n} ^ {2} r ^ {2 s}}}{\mathbb {P} _ {f \sim \Pi_ {n}} \left(\left\| f - f _ {0} \right\| _ {L ^ {\infty} (\mathcal {M})} <   \varepsilon_ {n} r ^ {s}\right)} (95) \\ \times \mathbb {E} _ {\boldsymbol {x}} \mathbb {P} _ {f \sim \Pi_ {n}} \left(\| f \| _ {\mathcal {C H} ^ {\gamma} (\mathcal {M})} \leq \tau \sqrt {n} \varepsilon_ {n} r ^ {s}, 2 \| f - f _ {n} \| _ {n} \leq \varepsilon_ {n} r \leq \| f - f _ {n} \| _ {L ^ {2} \left(p _ {0}\right)}\right) d r (96) \\ \leq \int_ {0} ^ {\infty} r e ^ {(c + 1) n \varepsilon_ {n} ^ {2} r ^ {2 s}} (97) \\ \times \int_ {\| f \| _ {\mathcal {C H} ^ {\gamma} (\mathcal {M})} \leq \tau \sqrt {n} \varepsilon_ {n} r ^ {s}, \varepsilon_ {n} r \leq \| f - f _ {n} \| _ {L ^ {2} \left(p _ {0}\right)}} \mathbb {E} _ {\boldsymbol {x}} \mathbb {1} _ {\| f - f _ {n} \| _ {L ^ {2} \left(p _ {0}\right)} \geq 2 \| f - f _ {n} \| _ {n}} \mathrm {d} \Pi_ {n} (f) \mathrm {d} r. (98) \\ \end{array}
$$
As the squared empirical  $L^2$ -norm is a Monte Carlo approximation of the true  $L^2$ -norm, the probability in the integrand can be controlled via a concentration inequality. As in van der Vaart and van Zanten [56], we use Bernstein's inequality [58, Lemma 2.2.9]. For a collection  $Y_{1},\ldots Y_{n}$  of random variables such that  $\mathbb{E}Y_{i} = 0$  and  $Y_{i}\in [-M,M]$  almost surely for some constant  $M > 0$ , this inequality asserts
$$
\mathbb {P} \left(\left| Y _ {1} + \dots + Y _ {n} \right| > x\right) \leq 2 \exp \left(- \frac {1}{2} \frac {x ^ {2}}{v + M x / 3}\right) \tag {99}
$$
where  $v \geq \operatorname{Var}(Y_1 + \ldots + Y_n)$ . We put  $Y_i = \frac{1}{n} (f(x_i) - f_n(x_i))^2 - \frac{1}{n} \| f - f_n \|_{L^2(p_0)}^2$  where  $x_i$  are IID with  $x_i \sim p_0$ . It is easy to check that  $\mathbb{E} Y_i = 0$  and  $Y_i \in [-M, M]$  for  $M = \frac{1}{n} \| f - f_n \|_{L^\infty(\mathcal{M})}$ . Furthermore,  $v = \frac{1}{n} \| f - f_n \|_{L^\infty(\mathcal{M})}^2 \| f - f_n \|_{L^2(\mathcal{M})}^2$  upper-bounds the variance of the respective sum, since
$$
\begin{array}{l} \operatorname {V a r} \left(Y _ {1} + \dots + Y _ {n}\right) = \frac {1}{n} \operatorname {V a r} _ {x \sim p _ {0}} \left(f (x) - f _ {n} (x)\right) ^ {2} \leq \frac {1}{n} \mathbb {E} _ {x \sim p _ {0}} \left(f (x) - f _ {n} (x)\right) ^ {4} (100) \\ \leq \frac {\left\| f - f _ {n} \right\| _ {L ^ {\infty} (\mathcal {M})} ^ {2}}{n} \mathbb {E} _ {x \sim p _ {0}} (f (x) - f _ {n} (x)) ^ {2} = v. (101) \\ \end{array}
$$
Using Bernstein's inequality with  $Y_{i}$ ,  $M$ ,  $v$  as above and with  $x = \frac{3}{4}\| f - f_n\|_{L^2 (p_0)}^2$ , we have
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {x}} \mathbb {1} _ {\| f - f _ {n} \| _ {L ^ {2} (p _ {0})} \geq 2 \| f - f _ {n} \| _ {n}} = \mathbb {E} _ {\boldsymbol {x}} \mathbb {1} _ {\| f - f _ {n} \| _ {L ^ {2} (p _ {0})} ^ {2} \geq 4 \| f - f _ {n} \| _ {n} ^ {2}} (102) \\ = \mathbb {E} _ {\boldsymbol {x}} \mathbb {1} _ {\| f - f _ {n} \| _ {n} ^ {2} - \| f - f _ {n} \| _ {L ^ {2} (p _ {0})} ^ {2}} \leq - \frac {3}{4} \| f - f _ {n} \| _ {L ^ {2} (p _ {0})} ^ {2} (103) \\ \lesssim \exp \left(- \frac {1}{2} \frac {x ^ {2}}{v + M x / 3}\right) (104) \\ = \exp \left(- \frac {\frac {9}{3 2} \| f - f _ {n} \| _ {L ^ {2} \left(p _ {0}\right)} ^ {4}}{v + \frac {1}{n} \| f - f _ {n} \| _ {L ^ {\infty} (\mathcal {M})} ^ {2} \frac {3}{4} \| f - f _ {n} \| _ {L ^ {2} \left(p _ {0}\right)} ^ {2} / 3}\right) (105) \\ = \exp \left(- \frac {9 n}{3 2} \frac {4}{5} \frac {\| f - f _ {n} \| _ {L ^ {2} \left(p _ {0}\right)} ^ {2}}{\| f - f _ {n} \| _ {L ^ {\infty} (\mathcal {M})} ^ {2}}\right) (106) \\ = \exp \left(- \frac {9 n}{4 0} \frac {\| f - f _ {n} \| _ {L ^ {2} \left(p _ {0}\right)} ^ {2}}{\| f - f _ {n} \| _ {L ^ {\infty} (\mathcal {M})} ^ {2}}\right). (107) \\ \end{array}
$$
Moreover, by Lemma 19, since  $\gamma \notin \mathbb{Z}_{>0}$ , we have
$$
\| f - f _ {n} \| _ {L ^ {\infty} (\mathcal {M})} \lesssim \| f - f _ {n} \| _ {\mathcal {C H} ^ {\gamma} (\mathcal {M})} ^ {\frac {d}{2 \gamma + d}} \| f - f _ {n} \| _ {L ^ {2} \left(p _ {0}\right)} ^ {\frac {2 \gamma}{2 \gamma + d}}. \tag {108}
$$
Using the Sobolev Embedding Theorem, given in De Vito et al. [16], Theorem 4,  $\| f - f_n\|_{\mathcal{CH}^\gamma (\mathcal{M})}\lesssim$ $\| f_n\|_{\mathbb{H}} + \| f\|_{\mathcal{CH}^\gamma (\mathcal{M})}\lesssim \tau \sqrt{n}\varepsilon_nr^s$  whenever  $\| f\|_{\mathcal{CH}^\gamma (\mathcal{M})}\leq \tau \sqrt{n}\varepsilon_nr^s$  . Therefore, for a constant  $c > 0$  , when  $\| f\|_{\mathcal{CH}^\gamma (\mathcal{M})}\leq \tau \sqrt{n}\varepsilon_nr^s$  and  $\varepsilon_{n}r\leq \| f - f_{n}\|_{L^{2}(p_{0})}$  , we have that
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {x}} \mathbb {1} _ {\| f - f _ {n} \| _ {L ^ {2} (\mathcal {M})} \geq 2 \| f - f _ {n} \| _ {n}} \lesssim \exp \left(- c n \frac {\| f - f _ {n} \| _ {L ^ {2} \left(p _ {0}\right)} ^ {2}}{\| f - f _ {n} \| _ {\mathcal {C H} ^ {\gamma} (\mathcal {M})} ^ {\frac {2 d}{2 \gamma + d}} \| f - f _ {n} \| _ {L ^ {2} \left(p _ {0}\right)} ^ {\frac {4 \gamma}{2 \gamma + d}}}\right) (109) \\ \leq e ^ {- c \tau^ {- \frac {2 d}{2 \gamma + d}} n ^ {\frac {2 \gamma}{2 \gamma + d}} r ^ {\frac {2 d}{2 \gamma + d} (1 - s)}}. (110) \\ \end{array}
$$
Hence, we can bound the last term as
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \int_ {0} ^ {\infty} r \mathbb {1} _ {\mathcal {A} _ {r}} \mathbb {P} _ {f \sim \Pi_ {n} (\cdot | \boldsymbol {x}, \boldsymbol {y})} \left(\mathcal {B} ^ {(\mathrm {I I I})} (r)\right) \mathrm {d} r (111) \\ \lesssim \int_ {0} ^ {\infty} r e ^ {(c + 1) n \varepsilon_ {n} ^ {2} r ^ {2 s}} e ^ {- c \tau^ {- \frac {2 d}{2 \gamma + d}} n ^ {\frac {2 \gamma}{2 \gamma + d}} r ^ {\frac {2 d}{2 \gamma + d} (1 - s)}} \mathrm {d} r. (112) \\ \end{array}
$$
We have  $n^{\frac{2\gamma}{2\gamma + d}} = n\left(n^{-\frac{d / 2}{2\gamma + d}}\right)^2$ . Since  $\varepsilon_{n} \lesssim n^{-\frac{\min(\nu,\beta)}{2\nu + d}}$  and  $\min (\nu ,\beta) > d / 2$ , we have  $n\varepsilon_n^2 \lesssim n^{\frac{2\gamma}{2\gamma + d}}$  for some  $\gamma \in (d / 2,\nu)$ . Moreover, for this choice of  $\gamma$  and  $s$  small enough we have  $\frac{2d}{2\gamma + d} (1 - s) \geq 2s$ , which proves that for some constants  $C,C^{\prime},C^{\prime \prime} > 0$  the fourth term is bounded by
$$
C \int_ {0} ^ {\infty} r e ^ {- C ^ {\prime} r ^ {C ^ {\prime \prime}}} \mathrm {d} r <   \infty . \tag {113}
$$
Remark 23. Following the proof, it is easy to see that  $\| f - f_0\|_{L^2 (p_0)}^2$  on the left-hand side of Equations (9), (10) and (12) can be replaced with  $\| f - f_0\|_{L^2 (p_0)}^q$  for any  $q > 1$ , changing the exponent  $-\frac{2\min(\beta,\nu)}{2\nu + d}$  on the right-hand side to  $-\frac{q\min(\beta,\nu)}{2\nu + d}$ .
C Characterizing Reproducing Kernel Hilbert Spaces of Matérn Kernels
We start by describing the reproducing kernel Hilbert spaces (RKHSs) of the (truncated) intrinsic Matérn processes, proving that they coincide with (certain subspaces of) the manifold Sobolev spaces. We follow the ideas of Borovitskiy et al. [11], where the same was shown somewhat implicitly. We consider the more-involved case of the extrinsic Matérn processes immediately after.
The next lemma describes the RKHS of the intrinsic Matérn processes, including truncated variants. This result is easy to obtain since we have defined them in terms of the Karhunen-Loève expansions.
Lemma 24. Let  $\mathbb{H}_J$  be the RKHS of the intrinsic Matérn Gaussian process with smoothness parameter  $\nu$  truncated at the level  $J\in \mathbb{Z}_{>0}\cup \{\infty \}$ , and let  $\{f_j\}_{j = 0}^{\infty}$  be the orthonormal basis of Laplace-Beltrami eigenfunctions. The space  $\mathbb{H}_J$  is norm-equivalent—with constants depending only on  $\nu, \kappa$  and  $\sigma_f^2$ —to the set of functions  $f = \sum_{j = 1}^{J}b_{j}f_{j}$  with  $b_{j}\in \mathbb{R}$ , equipped with the inner product
$$
\left\langle \sum_ {j = 1} ^ {J} b _ {j} f _ {j}, \sum_ {j = 1} ^ {J} b _ {j} ^ {\prime} f _ {j} \right\rangle_ {\mathbb {H} _ {J}} = \sum_ {j = 1} ^ {J} \left(1 + \lambda_ {j}\right) ^ {\nu + d / 2} b _ {j} b _ {j} ^ {\prime}. \tag {114}
$$
In particular,  $\mathbb{H}_J\subset H^{\nu +d / 2}(\mathcal{M})$  for all  $J$ , and for every  $h\in \mathbb{H}_J$  we have  $\| h\|_{\mathbb{H}_J} = \| h\|_{H^{\nu +d / 2}(\mathcal{M})}$ .
Proof. By direct computation, the covariance  $k$  of the (truncated) intrinsic Gaussian process is
$$
k \left(x, x ^ {\prime}\right) = \frac {\sigma_ {f} ^ {2}}{C _ {\nu , \kappa}} \sum_ {j = 1} ^ {J} \left(\frac {2 \nu}{\kappa^ {2}} + \lambda_ {j}\right) ^ {- (\nu + d / 2)} f _ {j} (x) f _ {j} \left(x ^ {\prime}\right). \tag {115}
$$
Hence, the covariance operator  $K:L^{2}(\mathcal{M})\to L^{2}(\mathcal{M})$  defined by
$$
(K f) (x) = \int_ {\mathcal {M}} k \left(x, x ^ {\prime}\right) f \left(x ^ {\prime}\right) d x ^ {\prime} \tag {116}
$$
is diagonal in the basis  $\{f_j\}_{j=1}^J$ , with  $Kf_j = \frac{\sigma_f^2}{C_{\nu,\kappa}} \left( \frac{2\nu}{\kappa^2} + \lambda_j \right)^{-(\nu + d/2)} f_j$ . Then, Kanagawa et al. [28], Theorem 4.2 implies that  $\mathbb{H}_J$  consists of functions of form  $f = \sum_{j=1}^J a_j f_j$  satisfying
$$
\left\| f \right\| _ {\mathbb {H} _ {J}} ^ {2} = \frac {\sigma_ {f} ^ {2}}{C _ {\nu , \kappa}} \sum_ {j = 1} ^ {J} \left(\frac {2 \nu}{\kappa^ {2}} + \lambda_ {j}\right) ^ {\nu + d / 2} \left| a _ {j} \right| ^ {2} <   \infty . \tag {117}
$$
Using the elementary inequality  $\min \left(\frac{2\nu}{\kappa^2},1\right)\leq \frac{\frac{2\nu}{\kappa^2} + \lambda}{1 + \lambda}\leq \max \left(\frac{2\nu}{\kappa^2},1\right)$ , we find that this space is norm-equivalent to the space  $H_J^{\nu +d / 2}(\mathcal{M})$  of functions  $f = \sum_{j = 1}^{J}a_{j}f_{j}$  satisfying