SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Having established the existence of an  $L^2(\Omega)$ -derivative  $\dot{h}_l$  of the process  $h_l$ , we would like now to show that  $\dot{h}_l$  possesses a  $(\gamma - 1)$ -regular version for every  $\gamma < \nu$ . For this, we would like to apply Lemma 28 to  $\dot{h}_l$ . Notice that, by isometry, for all  $h > 0$  we have
$$
\begin{array}{l} \left. \mathbb {E} _ {f \sim \Pi_ {n}} \left| \dot {h} _ {l} (x) - \dot {h} _ {l} (y) \right| ^ {2} = \left\| \frac {\partial \widetilde {K} _ {l}}{\partial x} (y, \cdot) - \frac {\partial \widetilde {K} _ {l}}{\partial x} (x, \cdot) \right\| _ {\widetilde {\mathbb {H}} _ {l}} ^ {2} \right. (168) \\ \leq 3 \left\| \frac {\widetilde {K} _ {l} (y + \delta , \cdot) - \widetilde {K} _ {l} (y , \cdot)}{h} - \frac {\partial \widetilde {K} _ {l}}{\partial x} (y, \cdot) \right\| _ {\widetilde {\mathbb {H}} _ {l}} ^ {2} (169) \\ + 3 \left\| \frac {\widetilde {K} _ {l} (x + \delta , \cdot) - \widetilde {K} _ {l} (x , \cdot)}{h} - \frac {\partial \widetilde {K} _ {l}}{\partial x} (x, \cdot) \right\| _ {\widetilde {\mathbb {H}} _ {l}} ^ {2} (170) \\ + 3 \left\| \frac {\widetilde {K} _ {l} (x + \delta , \cdot) - \widetilde {K} _ {l} (x , \cdot)}{h} - \frac {\widetilde {K} _ {l} (y + \delta , \cdot) - \widetilde {K} _ {l} (y , \cdot)}{h} \right\| _ {\widetilde {\mathbb {H}} _ {l}} ^ {2}. (171) \\ \end{array}
$$
See for instance Aliprantis and Border [1] for a review of Cauchy nets.
Therefore, by the same arguments as above, we have
$$
\begin{array}{l} \left(\mathbb {E} _ {f \sim \Pi_ {n}} \left| \dot {h} _ {l} (x) - \dot {h} _ {l} (y) \right| ^ {2}\right) \stackrel {1 / 2} {\lesssim} \lim  _ {h \rightarrow 0} \inf  _ {h \rightarrow 0} \left\| \frac {\widetilde {K} _ {l} (x + \delta , \cdot) - \widetilde {K} _ {l} (x , \cdot)}{h} - \frac {\widetilde {K} _ {l} (y + \delta , \cdot) - \widetilde {K} _ {l} (y , \cdot)}{h} \right\| _ {\widetilde {\mathbb {H}} _ {l}} (172) \\ \leq \lim  _ {h \rightarrow 0} \inf  _ {\| \varphi \| _ {\mathbb {B} _ {l}} = 1} \int_ {0} ^ {1} | \varphi^ {\prime} (x + t h) - \varphi^ {\prime} (y + t h) | \mathrm {d} t (173) \\ \leq \lim  _ {h \rightarrow 0} \inf  C _ {\nu} | x - y | ^ {\nu - 1} = C _ {\nu} | x - y | ^ {\nu - 1} (174) \\ \end{array}
$$
where the transition from the second-to-last line to the last line is similar to (163)-(165).
Therefore, we can apply Lemma 28 to  $\dot{h}_l$ , and find a version  $\bar{h}_l'$  of  $\dot{h}_l$  with sample paths in  $\mathcal{CH}^{\alpha -1}(\mathcal{V}_l)$  almost surely for all  $\alpha < \nu$ , such that
$$
\mathbb {E} _ {f \sim \Pi_ {n}} \| \tilde {h} _ {l} ^ {\prime} \| _ {\mathcal {C H} ^ {\alpha - 1} \left(\nu_ {l}\right)} ^ {2} \leq C _ {\nu , \alpha} <   + \infty \quad \alpha <   \nu . \tag {175}
$$
This gives Hölder regularity of the respective derivatives: we now integrate these to obtain a Hölder-regular version of the process itself. Take any  $c_{l} \in (0,1)$  and consider  $\tilde{h}_{l} = h_{l}(c_{l}) + \int_{c_{l}}^{\cdot}\tilde{h}_{l}^{\prime}(t)\mathrm{d}t$ . Then since  $\tilde{h}_l^\prime$  is almost surely in  $\mathcal{CH}^{\alpha -1}(\mathcal{V}_l)$ ,  $\tilde{h}_l$  is has sample paths almost surely in  $\mathcal{CH}^{\alpha}(\mathcal{V}_l)$ . Moreover, it is easy to check using our previous results that  $\tilde{h}_l$  has an  $L^2 (\Omega)$ -derivative given by  $\tilde{h}_l^\prime$ . This implies that  $\tilde{h}_l$  is a version of  $h_l$ .
To conclude the argument, we construct  $\tilde{f}$  from the obtained parts, by pulling  $\tilde{h}_l$  back from the charts to the manifold. Consider  $\tilde{f} = \sum_{l=1}^{L} \tilde{h}_l \circ \phi_l$ . Arguing as in the case  $\nu \leq 1$ , we see that  $\tilde{f}$  is a version of  $f$  with  $\mathcal{CH}^{\alpha}(\mathcal{M})$  sample paths for every  $\alpha < \nu$ , and for every  $\alpha < \nu$  we have  $\mathbb{E}_{f \sim \Pi_n} \| \tilde{f} \|_{\mathcal{CH}^{\alpha}(\mathcal{M})}^2 \leq C_{\alpha, \nu} < +\infty$ . This gives the claim.
With this, it is easy to prove that all Matérn Gaussian processes considered in this paper can be seen as Gaussian random elements in the Banach space  $\left(\mathcal{C}(\mathcal{M}),\| \cdot \|_{\mathcal{C}(\mathcal{M})}\right)$  of continuous functions on  $\mathcal{M}$ . This allows us to use the same proof scheme as in van der Vaart and van Zanten [56] through the control of the concentration functions defined in Appendix B. It is also important that we work with Gaussian random elements in  $\mathcal{C}(\mathcal{M})$  and not only with the classical notion of Gaussian process, as the concentration functions are defined using the Gaussian random element RKHS defined in van Zanten and van der Vaart [59], which can potentially be different from the classical RKHS. Fortunately, when the process is a Gaussian random element in  $\mathcal{C}(\mathcal{M})$ , van Zanten and van der Vaart [59], Theorem 2.1 implies that the two notions of RKHS coincide.
Corollary 30. The intrinsic Matérn Gaussian processes of Definition 4, their truncated versions as in Theorem 6 as well as the extrinsic Matérn Gaussian processes of Definition 7 are Gaussian random elements in  $(\mathcal{C}(\mathcal{M}),\| \cdot \|_{\mathcal{C}(\mathcal{M})})$
Proof. By Lemma 18, it suffices to show that the processes have almost surely continuous sample paths. Since Euclidean Matérn Gaussian processes have continuous sample paths, this implies the same for their restrictions, the extrinsic Matérn Gaussian processes on  $\mathcal{M}$ . For the intrinsic Matérn process, we apply Lemma 29.
Using Lemma 29 and known properties of Euclidean Matérn processes, we now show, in a sense, that all of the Matérn processes presented in this paper are sub-Gaussian, in a manner which holds uniformly with respect to the truncation parameter in the case of the truncated intrinsic Matérn process, and live in Hölder spaces with appropriate exponents. This result is used to control Hölder norms when going from the error at input locations to the  $L^2(p_0)$ -error. We use the notation  $\Pi_n$  to emphasize that the prior depends on  $n$  when we consider a truncated intrinsic Matérn process.
Lemma 31. For  $f \sim \Pi_n$  with  $\Pi_n$  the prior in either Definition 4, Theorem 6 or Definition 7, for every  $\nu > 0$  and  $\gamma < \nu, \gamma \notin \mathbb{Z}_{>0}$ , there exists a constant  $\sigma(f) = \sigma_{\gamma}(f)$  independent of  $n$  we have for  $x > 0$  that
$$
\mathbb {P} \left(\| f \| _ {\mathcal {C H} ^ {\gamma} (\mathcal {M})} > (x + 1) \sigma (f)\right) \leq 2 e ^ {- x ^ {2} / 2}. \tag {176}
$$
Proof. We start with the restriction  $f$  of an extrinsic Matérn process  $\tilde{f}$  to  $\mathcal{M}$ , as in Definition 7. By van der Vaart and van Zanten [56], Section 3.1, for every  $\gamma < \nu$  we have  $\tilde{f} \in \mathcal{CH}^{\gamma}([0,1]^D)$  almost surely. By Ghosal and van der Vaart [20], Lemma I.7, for every  $\gamma < \nu$ ,  $\tilde{f}$  is a Gaussian random element in the Banach space  $\mathcal{CH}^{\gamma}([0,1]^D)$ . In particular, by the Borel-TIS inequality [20, Proposition I.8] we have for  $x > 0$  that
$$
\mathbb {P} \left(\left\| \tilde {f} \right\| _ {\mathcal {C H} ^ {\gamma} ([ 0, 1 ] ^ {D})} > (x + 1) \sigma (\tilde {f})\right) \leq 2 e ^ {- x ^ {2} / 2} \tag {177}
$$
where  $\sigma (\tilde{f}) = \left(\mathbb{E}\left\| \tilde{f}\right\|_{\mathcal{CH}^{\gamma}([0,1]^{D})}^{2}\right)^{1 / 2} < \infty$ . Since  $\mathcal{M}$  is smooth, the restriction  $f$  also satisfies for  $x > 0$  the expression
$$
\mathbb {P} \left(\| f \| _ {\mathcal {C H} ^ {\gamma} (\mathcal {M})} > (x + 1) \sigma (f)\right) \leq 2 e ^ {- x ^ {2} / 2} \tag {178}
$$
perhaps for a possibly larger constant  $\sigma(f)$ . Finally, the case of the intrinsic Matérn process  $f \sim \Pi_{n}$  truncated at  $J_{n} \in \mathbb{Z}_{>0} \cup \{\infty\}$  follows in the same manner, as we have shown in Lemma 29 that  $\sup_{n \geq 1} \mathbb{E}_{f \sim \Pi_{n}} \|f\|_{\mathcal{CH}^{\alpha}(\mathcal{M})}^{2} \leq C_{\alpha, \nu}$ .
E Small Ball Asymptotics
Here, we bound the probability that a Matérn Gaussian process lies in the  $\varepsilon$ -ball with respect to the  $L^{\infty}$ -norm for small  $\varepsilon$ . For a Banach space  $\mathbb{B}$ , an element  $x \in \mathbb{B}$  and a number  $r \in \mathbb{R}_{>0}$ , let us denote the closed  $r$ -ball around  $x$  by  $B_r^\mathbb{B}(x)$ . We start by an upper bound on the metric entropy of Sobolev balls on  $\mathcal{M}$  with respect to the uniform norm.
Lemma 32 (Entropy of Sobolev balls). For any  $s > 0$  define the  $\varepsilon$ -covering number of  $A \subseteq H^s(\mathcal{M})$  with respect to the norm  $\| \cdot \|_{L^\infty(\mathcal{M})}$  by
$$
N \left(\varepsilon , A, \| \cdot \| _ {L ^ {\infty} (\mathcal {M})}\right) = \underset {J \in \mathbb {Z} > 0} {\arg \min } \left\{\exists h _ {1},.., h _ {J} \in A: A \subset \bigcup_ {j = 1} ^ {J} B _ {\varepsilon} ^ {L ^ {\infty} (\mathcal {M})} \left(h _ {j}\right) \right\}. \tag {179}
$$
Then for any  $s > d / 2$ , there exist  $C, \varepsilon_0 > 0$  such that for every  $\varepsilon \leq \varepsilon_0$
$$
\ln N \left(\varepsilon , B _ {1} ^ {H ^ {s} (\mathcal {M})} (0), \| \cdot \| _ {L ^ {\infty} (\mathcal {M})}\right) \leq C \varepsilon^ {- \frac {d}{s}}, \tag {180}
$$
where the left-hand side of the inequality above, as a function of  $\varepsilon$ , is called the METRIC ENTROPY of the Sobolev ball  $B_1^{H^s (\mathcal{M})}(0)$  with respect to the uniform norm  $\| \cdot \|_{L^{\infty}(\mathcal{M})}$ .
Proof. Using charts we will reduce the problem to the entropy of the unit ball of the Sobolev space  $H^{s}([0,1]^{d})$  for which the upper bound is known. Take  $f \in B_{1}^{H^{s}(\mathcal{M})}(0)$  and consider approximations of  $f$  by  $g$  of the form
$$
g = \sum_ {l = 1} ^ {L} \chi_ {l} \left(g _ {l} \circ \phi_ {l}\right) \tag {181}
$$
for some functions  $g_{l}:\mathcal{V}_{l}\to \mathbb{R}$  where  $\mathcal{V}_l\subseteq \mathbb{R}^d$  .We have
$$
\begin{array}{l} \left\| f - g \right\| _ {L ^ {\infty} (\mathcal {M})} = \left\| \sum_ {l = 1} ^ {L} \chi_ {l} \left(g _ {l} \circ \phi_ {l} - f\right) \right\| _ {L ^ {\infty} (\mathcal {M})} \leq \sum_ {l = 1} ^ {L} \left\| \chi_ {l} \left(g _ {l} \circ \phi_ {l} - f\right) \right\| _ {L ^ {\infty} \left(\mathcal {U} _ {l}\right)} (182) \\ \leq \sum_ {l = 1} ^ {L} \left\| g _ {l} \circ \phi_ {l} - f \right\| _ {L ^ {\infty} \left(\mathcal {U} _ {l}\right)} \leq \sum_ {l = 1} ^ {L} \left\| g _ {l} - f \circ \phi_ {l} ^ {- 1} \right\| _ {L ^ {\infty} \left(\mathcal {V} _ {l}\right)} (183) \\ \leq L \max  _ {1 \leq l \leq L} \left\| g _ {l} - f \circ \phi_ {l} ^ {- 1} \right\| _ {L ^ {\infty} ([ 0, 1 ] ^ {d})}. (184) \\ \end{array}
$$
This means that to approximate  $f$  by  $g$  uniformly on  $\mathcal{M}$ , we need to choose the functions  $g_{l}$  that approximate  $f\circ \phi_l^{-1}$  well with respect to the uniform norm on  $[0,1]^d$ .
Next, we show that the functions  $f \circ \phi_l^{-1}$  are contained in an Euclidean Sobolev ball of radius  $R$ , with  $R$  depending only on  $\nu$  and the atlas. To do this we use Große and Schneider [24], Lemma 2.1 to write
$$
\begin{array}{l} \left\| f \circ \phi_ {l} ^ {- 1} \right\| _ {H ^ {s} ([ 0, 1 ] ^ {d})} = \left\| \sum_ {l ^ {\prime} = 1} ^ {L} \left(\chi_ {l ^ {\prime}} f\right) \circ \phi_ {l} ^ {- 1} \right\| _ {H ^ {s} ([ 0, 1 ] ^ {d})} \leq \sum_ {l ^ {\prime} = 1} ^ {L} \left\| \left(\chi_ {l ^ {\prime}} f\right) \circ \phi_ {l} ^ {- 1} \right\| _ {H ^ {s} ([ 0, 1 ] ^ {d})} (185) \\ = \sum_ {l ^ {\prime} = 1} ^ {L} \left\| \left(\chi_ {l ^ {\prime}} f\right) \circ \phi_ {l ^ {\prime}} ^ {- 1} \circ \phi_ {l ^ {\prime}} \circ \phi_ {l} ^ {- 1} \right\| _ {H ^ {s} ([ 0, 1 ] ^ {d})} (186) \\ \lesssim \sum_ {l ^ {\prime} = 1} ^ {L} \left\| \left(\chi_ {l ^ {\prime}} f\right) \circ \phi_ {l ^ {\prime}} ^ {- 1} \right\| _ {H ^ {s} ([ 0, 1 ] ^ {d})} \lesssim \| f \| _ {H ^ {s} (\mathcal {M})}. (187) \\ \end{array}
$$
Note, importantly, that the remark just above Große and Schneider [24], Lemma 2.1 allows us to consider Besov spaces  $B_{2,2}^{s}$  coinciding with the Sobolev spaces  $H^{s}$  instead of the Besov spaces  $B_{2,\infty}^{s}$  — to get from the second line to the third. Note also that the constant hidden behind the notation  $\lesssim$  in the last line where we use Theorem 16 is the radius  $R$ . Without loss of generality we assume  $R = 1$ . By the Euclidean counterpart of the result we are proving [21, Theorem 4.3.36], we have
$$
\ln N \left(\varepsilon , B _ {1} ^ {H ^ {s} [ 0, 1 ] ^ {d}} (0), \| \cdot \| _ {L ^ {\infty} ([ 0, 1 ] ^ {d})}\right) \lesssim \varepsilon^ {- \frac {d}{s}}. \tag {188}
$$
Let  $h_1, \ldots, h_J \in B_1^{H^s ([0,1]^d)}(0)$  be such that  $B_1^{H^s ([0,1]^d)}(0) \subset \bigcup_{j=1}^J B_{\varepsilon/L}^{L^\infty ([0,1]^d)}(h_k)$ . Then for any  $f \in B_1^{H^s(\mathcal{M})}(0)$  there exists a sequence  $\{j_l\}_{l=1}^L \subseteq \{1, \ldots, J\}$  such that
$$
\left\| f - \sum_ {l = 1} ^ {L} \chi_ {l} \left(h _ {j _ {l}} \circ \phi_ {l}\right) \right\| _ {L ^ {\infty} (\mathcal {M})} <   L \frac {\varepsilon}{L} = \varepsilon . \tag {189}
$$
This shows that  $N\big(\varepsilon, B_1^{H^s(\mathcal{M})}(0), \| \cdot \|_{L^\infty(\mathcal{M})}\big) \leq LJ$ , where  $L$  is just the number of charts, thereby proving the claim.
For the related diffusion spaces [16], the RKHS corresponding to the heat (diffusion) kernels, Castillo et al. [12] uses the results of Coulhon et al. [14] to bound the entropy in terms of a wavelet frame instead of relying on charts. We believe this alternative proof scheme should work in our case as well. However, we could not, to the best of our effort, get a tight-enough bound for the Sobolev spaces by directly using the results of Coulhon et al. [14] and therefore we chose to rely on charts instead.
Having established regularity properties for our prior processes, we now turn to the small ball problem: we want to find sharp lower bounds on  $\mathbb{P}(\| f\|_{L^{\infty}(\mathcal{M})} < \varepsilon)$  where  $f\sim \Pi$  is our prior process. This will be crucial in order to control the concentration functions used in Appendix B. In fact, it is well-known that this problem is closely related to the estimation of the metric entropy of the unit ball of the RKHS of  $f$  with respect to the uniform norm: see Li and Linde [31] for details. Since we have already characterized the RKHS of our processes in Proposition 27 and Lemma 24, we are able to lower bound the small-ball probabilities. The technicality here involves getting a bound uniform in the truncation parameter for the truncated intrinsic Matérn process, as the truncated Matérn process is a sequence of priors rather than a fixed prior.
Lemma 33. If  $f \sim \Pi_n$  where  $\Pi_n$  is the prior in either Definition 4 and Theorem 6 or Definition 7 with smoothness parameter  $\nu > d/2$ , then there exist two constants  $C, \varepsilon_0 > 0$  that do not depend on  $n$  such that for all  $\varepsilon \leq \varepsilon_0$  we have  $-\ln \mathbb{P}\left(\|f\|_{L^\infty(\mathcal{M})} < \varepsilon\right) \leq C\varepsilon^{-\frac{d}{\nu}}$ .
Proof. Because the processes are Gaussian random elements in  $\mathcal{C}(\mathcal{M})$  by Lemma 18, their stochastic process RKHS given by Proposition 27 coincide with their Gaussian random element RKHS defined in van Zanten and van der Vaart [59]. Hence, for the non-truncated intrinsic and the extrinsic Matérn processes, the result follows by a direct application of Lemma 32 and Li and Linde [31], Theorem 1.2. For the intrinsic Matérn process truncated at  $J_{n}$  it is not immediately clear that the constants  $C,\varepsilon_0$  can be taken independent of  $n$ , so we go through the proof of Li and Linde [31], Proposition 3.1 to see that this is, in fact, the case. We first need a crude upper bound of the form
$$
- \ln \mathbb {P} \left(\| f \| _ {L ^ {\infty} (\mathcal {M})} <   \varepsilon\right) \leq c \varepsilon^ {- c} \tag {190}
$$
for some possibly large constant  $c > 0$ . To get such a bound, we use Castillo et al. [12], Proposition 3 which shows the existence of a universal constant  $C > 0$  such that for all  $\varepsilon \leq \min(1, 4\sigma(f))$
$$
- \ln \mathbb {P} \left(\| f \| _ {L ^ {\infty} (\mathcal {M})} <   \varepsilon\right) \leq C n (\varepsilon) \ln \left(\frac {6 n (\varepsilon) \max  (1 , \sigma (f))}{\varepsilon}\right) \tag {191}
$$
where  $\sigma(f) = \left(\mathbb{E}_{f\sim \Pi_n}\| f\|_{L^\infty (\mathcal{M})}^2\right)^{1 / 2}$  and  $n(\varepsilon)$  is defined in the following way in Castillo et al. [12], page 684 using auxiliary quantities  $l_J$  that are defined in Li and Linde [31], page 1562, namely
$$
n (\varepsilon) = \max  \{J \geq 0: 4 l _ {J} (f) \geq \varepsilon \}, \tag {192}
$$
$$
l _ {J} (f) = \inf  \left\{\left(\mathbb {E} _ {f \sim \Pi_ {n}} \left\| \sum_ {j \geq J} \varepsilon_ {j} h _ {j} \right\| _ {L ^ {\infty} (\mathcal {M})} ^ {2}\right) ^ {1 / 2}: f \stackrel {(d)} {=} \sum_ {j \geq 1} \varepsilon_ {j} h _ {j} \right\} \tag {193}
$$
with  $\stackrel{(d)}{=}^{\prime}$  standing for the equality in distributions and the infimum being taken over all possible decompositions  $\sum_{j\geq 1}\varepsilon_jh_j$  with  $h_j\in \mathcal{C}(\mathcal{M}),\varepsilon_j$  being a sequence of IID  $\mathrm{N}(0,1)$  random variables, and the series being required to converge uniformly almost surely.
The function  $f = \sum_{j=1}^{J_n+1}\left(\frac{2\nu}{\kappa^2} + \lambda_{j-1}\right)^{-\frac{\nu+d/2}{2}} \varepsilon_j f_{j-1}$  is a valid decomposition. Therefore
$$
l _ {J} (f) \leq \left(\mathbb {E} _ {f \sim \Pi_ {n}} \left\| \sum_ {j = J} ^ {J _ {n} + 1} \left(\frac {2 \nu}{\kappa^ {2}} + \lambda_ {j - 1}\right) ^ {- \frac {\nu + d / 2}{2}} \varepsilon_ {j} f _ {j - 1} \right\| _ {L ^ {\infty} (\mathcal {M})} ^ {2}\right) ^ {1 / 2}. \tag {194}
$$
By the Sobolev Embedding Theorem and by Weyl's Law, given in Result 10, for every  $d / 2 < \gamma < \nu$  there exists a constant  $C = C_{\gamma ,\mathcal{M}}$  such that for all  $J\in \mathbb{Z}_{>0}$ , allowing  $C$  to change from line to line, we have
$$
\begin{array}{l} \mathbb {E} _ {f \sim \Pi_ {n}} \left\| \sum_ {j = J} ^ {J _ {n} + 1} (1 + \lambda_ {j - 1}) ^ {- \frac {\nu + d / 2}{2}} \varepsilon_ {j} f _ {j - 1} \right\| _ {L ^ {\infty} (\mathcal {M})} ^ {2} \leq C ^ {2} \mathbb {E} _ {f \sim \Pi_ {n}} \left\| \sum_ {j = J} ^ {J _ {n} + 1} (1 + \lambda_ {j - 1}) ^ {- \frac {\nu + d / 2}{2}} \varepsilon_ {j} f _ {j - 1} \right\| _ {H ^ {\gamma} (\mathcal {M})} ^ {2} \\ = C ^ {2} \sum_ {j = J} ^ {J _ {n} + 1} \left(1 + \lambda_ {j - 1}\right) ^ {- (\nu + d / 2 - \gamma)} (196) \\ \leq C ^ {2} \sum_ {j = J} ^ {J _ {n} + 1} j ^ {- (1 + 2 (\nu - \gamma) / d)} (197) \\ \leq C ^ {2} \sum_ {j \geq J} j ^ {- (1 + 2 (\nu - \gamma) / d)} (198) \\ \leq C ^ {2} J ^ {- 2 (\nu - \gamma) / d}. (199) \\ \end{array}
$$
By choosing  $J = 1$  this gives us  $\sigma(f) \leq C$  independent of  $n$ . Moreover, by choosing  $J \geq C \varepsilon^{-\frac{d}{2(\nu - \gamma)}}$ , again for a comparison constant  $C$  independent of  $n$ , this gives us  $n(\varepsilon) \leq C \varepsilon^{-\frac{d}{2(\nu - \gamma)}}$  for  $C$  independent of  $n$ . This implies using Castillo et al. [12], Proposition 3 that
$$
- \ln \mathbb {P} \left(\| f \| _ {L ^ {\infty} (\mathcal {M})} <   \varepsilon\right) \leq c \varepsilon^ {- c} \tag {200}
$$
for  $c > 0$  independent of  $n$ .
With this crude bound, we can now continue the proof of Li and Linde [31], Proposition 3.1. For this, we need a metric entropy estimate. For this notice that for all  $J \in \mathbb{Z}_{>0} \cup \{\infty\}$  we have  $B_1^{\mathbb{H}^J}(0) \subset B_1^{\mathbb{H}^\infty}(0) = B_1^{H^{\nu + d/2}(\mathcal{M})}(0)$ , and therefore using Lemma 32, we have the metric entropy estimate
$$
\ln N \left(\varepsilon , B _ {1} ^ {\mathbb {H} ^ {J}} (0), \| \cdot \| _ {L ^ {\infty} (\mathcal {M})}\right) \leq C \varepsilon^ {- \frac {d}{\nu + d / 2}} \tag {201}
$$
for a constant  $C > 0$  independent of  $J$ . Therefore following the proof of proposition 3.1 in Li and Linde [31] (with  $J \equiv 1$ ) we find  $-\ln \mathbb{P}\left(\|f\|_{L^{\infty}(\mathcal{M})} < \varepsilon\right) \leq C \varepsilon^{-\frac{d}{\nu}}$  for every  $\varepsilon \leq \varepsilon_0$ , where  $C, \varepsilon_0 > 0$  are constants independent of  $n$ .
<sup>9</sup>We consider  $\sum_{j\geq 1}$ , unlike  $\sum_{j\geq 0}$  frequently used above, in order to follow the respective references.
F Expressions for Pointwise Worst-case Errors
Let  $k$  be a kernel on some abstract input domain  $\mathcal{X}$ , and let  $\mathbb{H}_k$  be the respective RKHS. Consider  $n$  input values  $\mathbf{X} \subseteq \mathcal{X}$  and let  $\sigma_{\varepsilon}^{2} > 0$  be the noise variance. Define
$$
m _ {k, \mathbf {X}, f, \varepsilon} (t) = \mathbf {K} _ {t \mathbf {X}} \left(\mathbf {K} _ {\mathbf {X X}} + \sigma_ {\varepsilon} ^ {2} \mathbf {I}\right) ^ {- 1} (f (\mathbf {X}) + \varepsilon), \tag {202}
$$
$$
v ^ {(\mathrm {i})} (t) = v _ {k, \mathbf {X}} (t) = k (t, t) - \mathbf {K} _ {t \mathbf {X}} \left(\mathbf {K} _ {\mathbf {X X}} + \sigma_ {\varepsilon} ^ {2} \mathbf {I}\right) ^ {- 1} \mathbf {K} _ {\mathbf {X} t}. \tag {203}
$$
Proposition 34. With notation above
$$
v ^ {(\mathrm {i})} (t) = \sup  _ {f \in \mathcal {H} _ {k}, \| f \| _ {\mathcal {H} _ {k}} \leq 1} \mathbb {E} _ {\boldsymbol {\epsilon} \sim \mathrm {N} (\mathbf {0}, \sigma_ {\varepsilon} ^ {2} \mathbf {I})} | f (t) - m _ {k, \mathbf {X}, f, \varepsilon} (t) | ^ {2}. \tag {204}
$$
Proof. To simplify notation, we shorten  $\mathbb{E}_{\varepsilon \sim \mathrm{N}(\mathbf{0},\sigma_{\varepsilon}^{2}\mathbf{I})}$  to  $\mathbb{E}$  and denote  $\alpha = \mathbf{K}_{t\mathbf{X}}(\mathbf{K}_{\mathbf{XX}} + \sigma_{\varepsilon}^{2}\mathbf{I})^{-1}$ . First of all, by direct computation,
$$
\mathbb {E} m _ {k, \mathbf {X}, f, \varepsilon} (t) = \boldsymbol {\alpha} f (\mathbf {X}), \tag {205}
$$
$$
\mathbb {E} m _ {k, \mathbf {X}, f, \varepsilon} (t) ^ {2} = \boldsymbol {\alpha} f (\mathbf {X}) f (\mathbf {X}) ^ {\top} \boldsymbol {\alpha} ^ {\top} + \sigma_ {\varepsilon} ^ {2} \boldsymbol {\alpha} \boldsymbol {\alpha} ^ {\top}. \tag {206}
$$
$$
\begin{array}{l} \mathbb {E} | f (t) - m _ {k, \mathbf {X}, f, \varepsilon} (t) | ^ {2} = f (t) ^ {2} - 2 f (t) \mathbb {E} m _ {k, \mathbf {X}, f, \varepsilon} (t) + \mathbb {E} m _ {k, \mathbf {X}, f, \varepsilon} (t) ^ {2} (207) \\ = f (t) ^ {2} - 2 f (t) \boldsymbol {\alpha} f (\mathbf {X}) + \boldsymbol {\alpha} f (\mathbf {X}) f (\mathbf {X}) ^ {\top} \boldsymbol {\alpha} ^ {\top} + \sigma_ {\varepsilon} ^ {2} \boldsymbol {\alpha} \boldsymbol {\alpha} ^ {\top} (208) \\ = (f (t) - \boldsymbol {\alpha} f (\mathbf {X})) ^ {2} + \sigma_ {\varepsilon} ^ {2} \boldsymbol {\alpha} \boldsymbol {\alpha} ^ {\top} (209) \\ = \left\langle k (t, \cdot) - \sum_ {j = 1} ^ {n} \alpha_ {j} k \left(x _ {j}, \cdot\right), f \right\rangle_ {\mathbb {H} _ {k}} ^ {2} + \sigma_ {\varepsilon} ^ {2} \boldsymbol {\alpha} \boldsymbol {\alpha} ^ {\top}. (210) \\ \end{array}
$$
As  $\| g\|_{\mathbb{H}_k} = \sup_{f\in \mathbb{H}_k,\| f\|_{\mathbb{H}_k}\leq 1}\langle g,f\rangle_{\mathbb{H}_k}$ , implying  $\sup_{f\in \mathbb{H}_k,\| f\|_{\mathbb{H}_k}\leq 1}\langle g,f\rangle_{\mathbb{H}_k}^2 = \| g\|_{\mathbb{H}_k}^2$ , we have