SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left\| f \right\| _ {H _ {J} ^ {\nu + d / 2} (\mathcal {M})} ^ {2} = \sum_ {j = 1} ^ {J} \left(1 + \lambda_ {j}\right) ^ {\nu + d / 2} \left| a _ {j} \right| ^ {2} <   \infty \tag {118}
$$
where the comparison constants  $\sqrt{\frac{\sigma_f^2}{C_{\nu,\kappa}}\min\left(1,\frac{2\nu}{\kappa^2}\right)}$  and  $\sqrt{\frac{\sigma_f^2}{C_{\nu,\kappa}}\max\left(1,\frac{2\nu}{\kappa^2}\right)}$  only depend on the parameters  $\nu, \kappa, \sigma_f^2$ .
The next two theorems will be useful to characterize the RKHS of the extrinsic Matérn process on  $\mathcal{M}$ . We start by a lemma relating the RKHS of the restriction of a Gaussian process to the original one.
Lemma 25. Assume that  $k$  is a kernel on  $\mathbb{R}^d$ ,  $f \sim \mathrm{GP}(0, k)$  with almost surely continuous sample paths and  $\widetilde{\mathbb{H}}$  is the RKHS of  $k$ . If  $\mathcal{M} \subseteq \mathbb{R}^d$  is a submanifold, then the RKHS  $\mathbb{H}$  corresponding to the restricted process  $f_{|\mathcal{M}}$  is the set of all restrictions  $g_{|\mathcal{M}}$  of functions  $g \in \widetilde{\mathbb{H}}$  equipped with the norm
$$
\| h \| _ {\mathbb {H}} = \inf  _ {g \in \widetilde {\mathbb {H}}, g | _ {\mathcal {M}} = h} \| g \| _ {\widetilde {\mathbb {H}}}. \tag {119}
$$
Moreover there always exists an element  $g \in \widetilde{\mathbb{H}}$  such that  $g|_{\mathcal{M}} = f$  and  $\| g \|_{\widetilde{\mathbb{H}}} = \| f \|_{\mathbb{H}}$ .
Proof. Lemma 5.1 in Yang and Dunson [63].
The last result will be used to characterize the RKHS of the extrinsic Matérn Gaussian processes using trace and extension operators. The second ingredient for this is the following.
Theorem 26. If  $s > \frac{D - d}{2}$  then the restriction operator extends to a bounded linear map  $\operatorname{Tr}: H^{s}\big(\mathbb{R}^{D}\big)\to H^{s - \frac{D - d}{2}}(\mathcal{M})$ . Moreover, for every  $u > 0$  there exists a bounded right inverse  $\operatorname{Ex}: H^{u}(\mathcal{M})\to H^{u + \frac{D - d}{2}}\big(\mathbb{R}^{D}\big)$  such that  $\operatorname{Tr}\circ \operatorname{Ex} = \operatorname{id}_{H^u (\mathcal{M})}$  where  $\operatorname{Tr}$  corresponds to  $s = u + \frac{D - d}{2}$ .
Proof. Theorem 4.10 in GroBe and Schneider [24].
The last two results allow us to characterize the RKHS of the extrinsic Matérn process on  $\mathcal{M}$ .
Proposition 27. The RKHS  $\mathbb{H}$  of a restricted extrinsic Matérn process  $f$  with smoothness parameter  $\nu$  on  $\mathcal{M}$  is norm-equivalent to the Sobolev space  $H^{\nu +d / 2}(\mathcal{M})$ .
Proof. Using Lemma 25, the RKHS  $\mathbb{H}$  can be characterized as the set of functions  $f:\mathcal{M}\to \mathbb{R}$  that are the restrictions of some  $g\in \tilde{\mathbb{H}}$  , where  $\tilde{\mathbb{H}}$  is the RKHS of the ambient Matern process  $\tilde{f}$  , with
$$
\| f \| _ {\mathbb {H}} = \inf  _ {g \in \widetilde {\mathbb {H}}, g | _ {\mathcal {M}} = f} \| g \| _ {\widetilde {\mathbb {H}}}. \tag {120}
$$
Since  $\widetilde{\mathbb{H}}$  is norm-equivalent to the Sobolev space  $H^{\nu +D / 2}\bigl (\mathbb{R}^D\bigr)$  see for instance Kanagawa et al. [28]by Theorem 26, for every  $f\in \mathbb{H}$  we have
$$
\| f \| _ {\mathbb {H}} \lesssim \| \operatorname {E x} (f) \| _ {H ^ {\nu + D / 2} (\mathbb {R} ^ {D})} \lesssim \| f \| _ {H ^ {\nu + D / 2 - \frac {D - d}{2}} (\mathcal {M})} = \| f \| _ {H ^ {\nu + d / 2} (\mathcal {M})}. \tag {121}
$$
Similarly, for every  $g \in \widetilde{\mathbb{H}}$  with  $g_{|\mathcal{M}} = f$ , we have
$$
\left\| f \right\| _ {H ^ {\nu + d / 2} (\mathcal {M})} = \left\| g _ {| \mathcal {M}} \right\| _ {H ^ {\nu + d / 2} (\mathcal {M})} \lesssim \left\| g \right\| _ {H ^ {\nu + D / 2} \left(\mathbb {R} ^ {D}\right)} \lesssim \left\| g \right\| _ {\widetilde {\mathbb {H}}}. \tag {122}
$$
Hence, taking the infimum, we obtain
$$
\| f \| _ {H ^ {\nu + d / 2} (\mathcal {M})} \lesssim \inf  _ {g \in \widetilde {\mathbb {H}}, g | _ {\mathcal {M}} = f} \| g \| _ {\widetilde {\mathbb {H}}} = \| f \| _ {\mathbb {H}}. \tag {123}
$$
D Concentration and Sample Path Regularity
In this section, we prove that intrinsic, truncated intrinsic and extrinsic Matérn processes are sub-Gaussian, uniformly with respect to the truncation parameter in the case of the truncated intrinsic Matérn process, and live in Hölder spaces with appropriate exponents. On our way to proving this, we characterize sample path regularity of (truncated) intrinsic Matérn processes, which is of independent interest. A simple way to do this would be to build on the results of Appendix C, using Driscoll's Theorem—given in Kanagawa et al. [28], Theorem 4.9—and the Sobolev Embedding Theorem—De Vito et al. [16], Theorem 4—but that would only give us that the sample paths are almost surely in  $\mathcal{CH}^{\gamma}(\mathcal{M})$  for every  $0 < \gamma < \nu - d/2$ ,  $\gamma \notin \mathbb{N}$ , whereas here we improve the range of index to  $\gamma < \nu$ .
Kolmogorov's continuity criterion is a standard tool to show that a given stochastic process has a Hölder continuous version: we re-prove it here because we will need a form of the result which gives explicit control of the Hölder norms, which is not usually included in the respective statement.
Lemma 28 (Kolmogorov's continuity criterion). If  $g \sim \Pi$  is a zero-mean Gaussian process on  $[0,1]^d$  with
$$
\mathbb {E} \left| g (x) - g (y) \right| ^ {2} \leq C \| x - y \| _ {\mathbb {R} ^ {d}} ^ {2 \rho} \tag {124}
$$
for some  $0 < \rho \leq 1$  and  $C > 0$ , then there exists a version of  $g$  with samples paths in  $\mathcal{CH}^{\alpha}([0,1]^d)$  for every  $0 < \alpha < \rho$ . Moreover for every  $\alpha < \rho$  this version satisfies  $\mathbb{E}\| g\|_{\mathcal{CH}^{\alpha}([0,1]^d)}^2 \leq C'$  where  $C' < +\infty$  depends only on  $C, \rho, \alpha$  and  $d$ .
Proof. Take  $x, y \in [0,1]^d$ ,  $M > 0$  and  $q \in \mathbb{N}$ . Since the random variable  $g(x) - g(y)$  is Gaussian we have
$$
\mathbb {E} \left| g (x) - g (y) \right| ^ {2 q} = \frac {(2 q) !}{2 ^ {q} q !} \left(\mathbb {E} \left| g (x) - g (y) \right| ^ {2}\right) ^ {q} \leq C _ {q} \| x - y \| _ {\mathbb {R} ^ {d}} ^ {2 \rho q} \tag {125}
$$
where we have defined  $C_q = C^q \frac{(2q)!}{2^q q!}$ . The reason for considering the the 2qth power will become clear later in the proof. By Markov's inequality, for every  $x, y \in [0,1]^d$  we have
$$
\mathbb {P} \left(\left| g (x) - g (y) \right| > u\right) \leq C _ {q} u ^ {- 2 q} \| x - y \| _ {\mathbb {R} ^ {d}} ^ {2 q \rho}. \tag {126}
$$
Now, take  $X = \cup_{k\geq 0}X_k$ ,  $X_k = 2^{-k}\mathbb{Z}^d \cap [0,1]^d$ . Then, the previous inequality applied to any adjacent  $x,y \in X_k$ , where we see  $X_k$  as a graph where two vertices are connected if they differ by  $2^{-k}$ , and  $u = M2^{-k\alpha}$ , implies
$$
\mathbb {P} \left(| g (x) - g (y) | > M 2 ^ {- k \alpha}\right) \leq C _ {q} M ^ {- 2 q} 2 ^ {- 2 k q (\rho - \alpha)}. \tag {127}
$$
Note that this norm-equivalence is the only property of the Gaussian process we use in the proofs. Any other Gaussian process satisfying this, including ones different from the Matérn processes of Borovitskiy et al. [11], would also work. This is of potential interest since other Euclidean kernels, such as Wendland kernels [61], are known to possess RKHSs which are norm-equivalent to those of the Matérn kernel.
Denote  $K = \sup_{x,y\in X\text{adjacent}}\frac{|g(x) - g(y)|}{\|x - y\|^{\alpha}}$ . Summing over  $k\geq 1$  and adjacent points in  $X$  and there are at most  $C2^{k(d + 1)}$  of them where  $C > 0$  is an absolute constant—gives us for  $q > \frac{d + 1}{2(\rho - \alpha)}$  that
$$
\begin{array}{l} \mathbb {P} (K > M) \leq \sum_ {k \geq 0} \sum_ {x, y \in X \text {a d j a c e n t}} \mathbb {P} \left(| g (x) - g (y) | > M 2 ^ {- k \alpha}\right) (128) \\ \leq C \sum_ {k \geq 0} 2 ^ {k (d + 1)} C _ {q} M ^ {- 2 q} 2 ^ {- 2 k q (\rho - \alpha)} = \frac {C C _ {q}}{1 - 2 ^ {2 q (\rho - \alpha) - d - 1}} M ^ {- 2 q}. (129) \\ \end{array}
$$
In particular, for all  $q > \max \left(1,\frac{d + 1}{2(\rho - \alpha)}\right)$ , we have
$$
\begin{array}{l} \mathbb {E} \left(K ^ {2}\right) = 2 \int_ {0} ^ {\infty} M \mathbb {P} (K > M) \mathrm {d} M = 2 \int_ {0} ^ {1} M \mathbb {P} (K > M) \mathrm {d} M + 2 \int_ {1} ^ {\infty} M \mathbb {P} (k > M) \mathrm {d} M (130) \\ \leq 2 + 2 \int_ {1} ^ {\infty} M \mathbb {P} (K > M) \mathrm {d} M \leq C _ {C, \alpha , \rho , d} (131) \\ \end{array}
$$
for some constant  $C_{C,\alpha,\rho,d} < +\infty$ . This means that  $K$  is finite almost surely. Since  $X$  is dense in  $[0,1]^d$  and  $g$  is almost surely uniformly continuous on  $X$  by the classical version of the Kolmogorov's continuity criterion—see for instance Lototsky and Rozovsky [34]— $g$  admits a unique continuous extension to  $[0,1]^d$  on a probability one event  $\mathcal{A}$ . Let us define
$$
\bar {g} (x) = \left\{\begin{array}{l l}\lim  _ {y \rightarrow x, y \in X} g (y)&\text {o n} \mathcal {A},\\0&\text {o t h e r w i s e},\end{array}\right. \tag {132}
$$
for  $x\in [0,1]^d$  . For any  $x,y\in [0,1]^d$  and  $x_{n}\to x,y_{n}\to y,x_{n},y_{n}\in X$  we have on  $\mathcal{A}$
$$
\begin{array}{l} \left| \bar {g} (x) - \bar {g} (y) \right| \leq \lim  _ {n \rightarrow \infty} \inf  \left(\left| \bar {g} (x) - \bar {g} \left(x _ {n}\right)\right| + \left| \bar {g} \left(x _ {n}\right) - \bar {g} \left(y _ {n}\right)\right| + \left| \bar {g} \left(y _ {n}\right) - \bar {g} (y) \right|\right) (133) \\ \leq \lim  _ {n \rightarrow \infty} \inf  \left(| \bar {g} (x) - \bar {g} (x _ {n}) | + K \| x _ {n} - y _ {n} \| _ {\mathbb {R} ^ {d}} ^ {\alpha} + | \bar {g} (y _ {n}) - \bar {g} (y) |\right) (134) \\ = K \| x - y \| _ {\mathbb {R} ^ {d}} ^ {\alpha}. (135) \\ \end{array}
$$
On the complement of  $\mathcal{A}$ , we have  $\overline{g} = 0$ . Hence,  $\overline{g}$  is  $\alpha$ -Hölder continuous on  $[0,1]^d$  with the same (random) constant  $K$ , since
$$
\mathbb {E} \| \bar {g} \| _ {\mathcal {C H} ^ {\alpha} ([ 0, 1 ] ^ {d})} ^ {2} = \mathbb {E} \left(\sup  _ {x, y \in X} \frac {| \bar {g} (x) - \bar {g} (y) |}{\| x - y \| ^ {\alpha}}\right) ^ {2} \leq \mathbb {E} K ^ {2} \leq C _ {C, \alpha , \rho , d} <   \infty . \tag {136}
$$
Since  $\overline{g}$  is a version of  $g$ , the claim follows.
The next lemma applies our version of Kolmogorov's criterion, Lemma 28, to the intrinsic Matérn processes on  $\mathcal{M}$  by considering charts. This allows us to show that the sample paths are almost surely in  $\mathcal{CH}^{\gamma}(\mathcal{M})$  for every  $\gamma < \nu$ , which is both used in our arguments and also of independent interest. For the claims in Appendix B, we need to ensure that this property holds somewhat uniformly with respect to the truncation parameter, which is why we tracked the constants in our proof of Kolmogorov's criterion. As we will see, the main difficulty in the proof of the next result will be to tackle the case of regularity strictly larger than 1.
Lemma 29. Let  $f \sim \Pi_n$  be an intrinsic Matérn process with smoothness parameter  $\nu > 0$  truncated at  $J_n \in \mathbb{N} \cup \{\infty\}$ . Then for every  $\gamma < \nu$  we have
$$
\sup  _ {n} \mathbb {E} _ {f \sim \Pi_ {n}} \left[ \| f \| _ {\mathcal {C H} ^ {\gamma} (\mathcal {M})} ^ {2} \right] <   \infty . \tag {137}
$$
Proof. We start with the case  $\nu \leq 1$ . Take  $1 \leq l \leq L$  and define  $h_l = (\chi_l f) \circ \phi_l^{-1}$ . Then  $h_l$  is a Gaussian process with covariance kernel  $\widetilde{K}_l$  given by
$$
\widetilde {K} _ {l} (x, y) = \left(\chi_ {l} \circ \phi_ {l} ^ {- 1}\right) (x) K \left(\phi_ {l} ^ {- 1} (x), \phi_ {l} ^ {- 1} (y)\right) \left(\chi_ {l} \circ \phi_ {l} ^ {- 1}\right) (y) \tag {138}
$$
where  $x, y \in \mathcal{V}_l$  and  $K(x, y) = \mathrm{Cov}(f(x), f(y))$  is the covariance kernel of  $f$ . Let  $\widetilde{\mathbb{H}}_l$  be the RKHS induced by  $\widetilde{K}_l$ . We seek to apply Lemma 28 to  $h_l$ . For all  $x, y \in \mathcal{V}_l$ , where we recall that we can assume that  $\mathcal{V}_l = (0,1)^d$ , we have
$$
\begin{array}{l} \mathbb {E} _ {f \sim \Pi_ {n}} | h _ {l} (x) - h _ {l} (y) | ^ {2} = \widetilde {K} _ {l} (x, x) + \widetilde {K} _ {l} (y, y) - 2 \widetilde {K} _ {l} (x, y) (139) \\ = \left\| \widetilde {K} _ {l} (x, \cdot) - \widetilde {K} _ {l} (y, \cdot) \right\| _ {\widetilde {\mathbb {H}} _ {l}} ^ {2} (140) \\ = \sup  _ {\| \varphi \| _ {\widetilde {\mathbb {H}} _ {l}} = 1} \left| \left\langle \widetilde {K} _ {l} (x, \cdot) - \widetilde {K} _ {l} (y, \cdot), \varphi \right\rangle_ {\widetilde {\mathbb {H}} _ {l}} \right| ^ {2} (141) \\ = \sup  _ {\| \varphi \| _ {\mathbb {H} _ {l}} = 1} | \varphi (x) - \varphi (y) | ^ {2} (142) \\ \leq \sup  _ {\| \varphi \| _ {\widetilde {\mathbb {H}} _ {l}} = 1} \| \varphi \| _ {\mathcal {C H} ^ {\nu} (\mathcal {V} _ {l})} ^ {2} \| x - y \| _ {\mathbb {R} ^ {d}} ^ {2 \nu} (143) \\ \end{array}
$$
where  $\| \varphi \|_{\mathcal{CH}^{\nu}(\mathcal{V}_l)}^2$  can potentially be infinite or unbounded: we will show this is not the case. To do this it suffices to show that we have a continuous embedding  $\widetilde{\mathbb{H}}_l \hookrightarrow \mathcal{CH}^\nu(\mathcal{V}_l)$ , that is  $\| \cdot \|_{\mathcal{CH}^\nu(\mathcal{V}_l)} \lesssim \| \cdot \|_{\widetilde{\mathbb{H}}_l}$ . The RKHS  $\widetilde{\mathbb{H}}_l$  is by definition the completion of
$$
\begin{array}{l} \left\{\sum_ {i = 1} ^ {p} \alpha_ {i} \widetilde {K} _ {l} \left(x _ {i}, \cdot\right): p \geq 1, \alpha_ {i} \in \mathbb {R}, x _ {i} \in \mathcal {V} _ {l} \right\} (144) \\ = \left\{\sum_ {i = 1} ^ {p} \alpha_ {i} \left(\chi_ {l} \circ \phi_ {l} ^ {- 1}\right) \left(x _ {i}\right) \left(\chi_ {l} \circ \phi_ {l} ^ {- 1}\right) (\cdot) K \left(\phi_ {l} ^ {- 1} \left(x _ {i}\right), \phi_ {l} ^ {- 1} (\cdot)\right): p \geq 1, \alpha_ {i} \in \mathbb {R}, x _ {i} \in \mathcal {V} _ {l} \right\} (145) \\ \end{array}
$$
with respect to the topology induced by the RKHS norm
$$
\left\| \sum_ {i = 1} ^ {p} \alpha_ {i} \widetilde {K} _ {l} \left(x _ {i}, \cdot\right) \right\| _ {\widetilde {\mathbb {H}} _ {l}} ^ {2} = \sum_ {i, j = 1} ^ {p} \alpha_ {i} \alpha_ {j} \left(\chi_ {l} \circ \phi_ {l} ^ {- 1}\right) \left(x _ {i}\right) \left(\chi_ {l} \circ \phi_ {l} ^ {- 1}\right) \left(x _ {j}\right) K \left(\phi_ {l} ^ {- 1} \left(x _ {i}\right), \phi_ {l} ^ {- 1} \left(x _ {j}\right)\right). \tag {146}
$$
Denote the RKHS of  $K$  by  $\mathbb{H}$ . By Theorem 16, and by the equality  $\| \cdot \|_{\mathbb{H}} = \| \cdot \|_{H^{\nu +d / 2}(\mathcal{M})}$  on  $\mathbb{H}$  which follows by Lemma 24, we have
$$
\begin{array}{l} \left\| \sum_ {i = 1} ^ {p} \alpha_ {i} \widetilde {K} _ {l} \left(x _ {i}, \cdot\right) \right\| _ {H ^ {\nu + d / 2} \left(\mathbb {R} ^ {d}\right)} ^ {2} (147) \\ = \left\| \sum_ {i = 1} ^ {p} \alpha_ {i} \left(\chi_ {l} \circ \phi_ {l} ^ {- 1}\right) \left(x _ {i}\right) \left(\chi_ {l} \circ \phi_ {l} ^ {- 1}\right) (\cdot) K \left(\phi_ {l} ^ {- 1} \left(x _ {i}\right), \phi_ {l} ^ {- 1} (\cdot)\right) \right\| _ {H ^ {\nu + d / 2} \left(\mathbb {R} ^ {d}\right)} ^ {2} (148) \\ \lesssim \left\| \sum_ {i = 1} ^ {p} \alpha_ {i} \left(\chi_ {l} \circ \phi_ {l} ^ {- 1}\right) \left(x _ {i}\right) K \left(\phi_ {l} ^ {- 1} \left(x _ {i}\right), \cdot\right) \right\| _ {H ^ {\nu + d / 2} (\mathcal {M})} ^ {2} (149) \\ = \left\| \sum_ {i = 1} ^ {p} \alpha_ {i} \left(\chi_ {l} \circ \phi_ {l} ^ {- 1}\right) \left(x _ {i}\right) K \left(\phi_ {l} ^ {- 1} \left(x _ {i}\right), \cdot\right) \right\| _ {\mathbb {H}} ^ {2} (150) \\ = \sum_ {i, j = 1} ^ {p} \alpha_ {i} \alpha_ {j} \left(\chi_ {l} \circ \phi_ {l} ^ {- 1}\right) \left(x _ {i}\right) \left(\chi_ {l} \circ \phi_ {l} ^ {- 1}\right) \left(x _ {j}\right) K \left(\phi_ {l} ^ {- 1} \left(x _ {i}\right), \phi_ {l} ^ {- 1} \left(x _ {j}\right)\right) (151) \\ = \left\| \sum_ {i = 1} ^ {p} \alpha_ {i} \widetilde {K} _ {l} \left(x _ {i}, \cdot\right) \right\| _ {\widetilde {\mathbb {H}} _ {l}} ^ {2}. (152) \\ \end{array}
$$
Therefore, we have a continuous embedding  $\widetilde{\mathbb{H}}_l\hookrightarrow H^{\nu +d / 2}\big(\mathbb{R}^d\big)$  with  $\| \cdot \|_{H^{\nu +d / 2}(\mathbb{R}^d)}\lesssim \| \cdot \|_{\widetilde{\mathbb{H}}_l}$  on  $\widetilde{\mathbb{H}}_l$ . By the Sobolev Embedding Theorem in  $\mathbb{R}^d$ —see for instance Triebel [50], Section 2.7.1,
Remark 2—we have  $B_{2,2}^{\nu +d / 2}\big(\mathbb{R}^d\big) = H^{\nu +d / 2}\big(\mathbb{R}^d\big)\hookrightarrow \mathcal{CH}^\nu \big(\mathbb{R}^d\big)$ , which implies  $\widetilde{\mathbb{H}}_l\hookrightarrow \mathcal{CH}^\nu \big(\mathbb{R}^d\big)$  by composition. Thus, there exists a constant  $C = C_{\nu}$  such that
$$
\mathbb {E} _ {f \sim \Pi_ {n}} \left| h _ {l} (x) - h _ {l} (y) \right| ^ {2} \leq C \| x - y \| _ {\mathbb {R} ^ {d}} ^ {2 \nu} \tag {153}
$$
for  $x,y\in \mathcal{V}_l$ . Hence, by applying Lemma 28, there exists a version  $\tilde{h}_l$  of  $h_l$  with almost surely  $\alpha$ -Hölder continuous sample paths for every  $\alpha < \nu$ . Now consider  $\tilde{f} = \sum_{l=1}^{L}\tilde{h}_l\circ \phi_l$ . Then  $\tilde{f}$  is a version of  $f$ . We proceed to bound  $\mathbb{E}_{f\sim \Pi_n}\left[\|\tilde{f}\|_{\mathcal{CH}^\alpha (\mathcal{M})}^2\right]$ . For any  $1\leq l,r\leq L$  write
$$
\left| \tilde {h} _ {r} \left(\phi_ {r} \left(\phi_ {l} ^ {- 1} (x)\right)\right) - \tilde {h} _ {r} \left(\phi_ {r} \left(\phi_ {l} ^ {- 1} (y)\right)\right) \right| \leq K \left| \phi_ {r} \left(\phi_ {l} ^ {- 1} (x)\right) - \phi_ {r} \left(\phi_ {l} ^ {- 1} (y)\right) \right| ^ {\alpha} \leq C K | x - y | ^ {\alpha} \tag {154}
$$
where  $C$  is the Lipshitz constant of  $\phi_r\circ \phi_l^{-1}$  which is well defined and finite because this composition is a diffeomorphism and  $K$  is a random constant with  $\mathbb{E}_{f\sim \Pi_n}K^2\leq C_{\alpha ,\nu ,d}$ . Hence
$$
\begin{array}{l} \mathbb {E} _ {f \sim \Pi_ {n}} \| \tilde {f} \| _ {\mathcal {C H} ^ {\alpha} (\mathcal {M})} ^ {2} = \mathbb {E} _ {f \sim \Pi_ {n}} \sum_ {l = 1} ^ {L} \left\| \left(\chi_ {l} \tilde {f}\right) \circ \phi_ {l} ^ {- 1} \right\| _ {\mathcal {C H} ^ {\alpha} (\mathbb {R} ^ {d})} ^ {2} (155) \\ = \mathbb {E} _ {f \sim \Pi_ {n}} \sum_ {l = 1} ^ {L} \left\| \left(\chi_ {l} \sum_ {r = 1} ^ {L} \tilde {h} _ {r} \circ \phi_ {r}\right) \circ \phi_ {l} ^ {- 1} \right\| _ {\mathcal {C H} ^ {\alpha} (\mathbb {R} ^ {d})} ^ {2} (156) \\ \lesssim \sum_ {l = 1} ^ {L} \sum_ {r = 1} ^ {L} \mathbb {E} _ {f \sim \Pi_ {n}} \left\| \tilde {h} _ {r} \circ \phi_ {r} \circ \phi_ {l} ^ {- 1} \right\| _ {\mathcal {C H} ^ {\alpha} (\mathbb {R} ^ {d})} ^ {2} \lesssim \mathbb {E} _ {f \sim \Pi_ {n}} K ^ {2} \leq C _ {\alpha , \nu , d}. (157) \\ \end{array}
$$
which gives the  $\nu \leq 1$  case.
We now turn to the general case. The proof will be similar to the one of Ghosal and van der Vaart [20], Proposition I.3 although we need to control the Hölder norms and work through charts since our Gaussian processes are supported on manifolds. Assume for simplicity that  $d = 1, 1 < \nu \leq 2$ , otherwise it suffices to introduce coordinates and to proceed by induction on  $\lfloor \nu \rfloor$ . Let  $l \in \{1, \dots, L\}$  and as before define  $\widetilde{K}_l(x,y) = (\chi_l \circ \phi_l^{-1})(x)(\chi_l \circ \phi_l^{-1})(y)K(\phi_l^{-1}(x),\phi_l^{-1}(y))$  the covariance kernel of  $h_l = (\chi_l f) \circ \phi_l^{-1}$  as well as  $\widetilde{\mathbb{H}}_l$  its RKHS.
First, let us construct an  $L^2(\Omega)$ -derivative  $\dot{h}_l$  of  $h_l$ , where  $L^2(\Omega)$  is the space of random variables with finite variance with  $\langle a, b \rangle_{L^2(\Omega)} = \mathbb{E}(ab)$ . This derivative is a square integrable process on  $\mathcal{V}_l$  such that
$$
\mathbb {E} _ {f \sim \Pi_ {n}} \left| \frac {h _ {l} (x + \delta) - h _ {l} (x)}{h} - \dot {h} _ {l} (x) \right| ^ {2} \rightarrow 0 \tag {158}
$$
as  $h\to 0$  , for all  $x\in \mathcal{V}_l$  . For this, we will first show that  $\frac{\partial\widetilde{K}_l}{\partial x} (x,\cdot)\in \widetilde{\mathbb{H}}_l$  for every  $x\in \mathcal{V}_l$  -here  $\frac{\partial\widetilde{K}_l}{\partial x}$  denotes the derivative of the function  $\widetilde{K}_l(\cdot ,\cdot ')$  with respect to the first argument-and that
$$
\left\| \frac {\partial \widetilde {K} _ {l}}{\partial x} (x, \cdot) - \frac {\partial \widetilde {K} _ {l}}{\partial x} \left(x ^ {\prime}, \cdot\right) \right\| _ {\widetilde {\mathbb {H}} _ {l}} \leq C _ {\nu} | x - x ^ {\prime} | ^ {\nu - 1}. \tag {159}
$$
We first show that  $\frac{\widetilde{K}_l(x + \delta,\cdot) - \widetilde{K}_l(x,\cdot)}{h}$  is a Cauchy net in  $\widetilde{\mathbb{H}}_l$ . We have
$$
\begin{array}{l} \left\| \frac {\widetilde {K} _ {l} (x + \delta , \cdot) - \widetilde {K} _ {l} (x , \cdot)}{h} - \frac {\widetilde {K} _ {l} (x + \delta^ {\prime} , \cdot) - \widetilde {K} _ {l} (x , \cdot)}{h ^ {\prime}} \right\| _ {\widetilde {\mathbb {H}} _ {l}} (160) \\ = \sup  _ {\| \varphi \| _ {\mathbb {H} _ {l}} = 1} \left\langle \frac {\widetilde {K} _ {l} (x + \delta , \cdot) - \widetilde {K} _ {l} (x , \cdot)}{h} - \frac {\widetilde {K} _ {l} (x + \delta^ {\prime} , \cdot) - \widetilde {K} _ {l} (x , \cdot)}{h ^ {\prime}}, \varphi \right\rangle_ {\widetilde {\mathbb {H}} _ {l}} (161) \\ = \sup  _ {\| \varphi \| _ {\widetilde {\mathbb {H}} _ {l}} = 1} \left(\frac {\varphi (x + \delta) - \varphi (x)}{h} - \frac {\varphi (x + \delta^ {\prime}) - \varphi (x)}{h ^ {\prime}}\right) (162) \\ = \sup  _ {\| \varphi \| _ {\mathbb {H} _ {l}} = 1} \int_ {0} ^ {1} \left[ \varphi^ {\prime} (x + t h) - \varphi^ {\prime} (x + t h ^ {\prime}) \right] d t (163) \\ \leq \sup  _ {\| \varphi \| _ {\widetilde {\mathbb {H}} _ {l}} = 1} \| \varphi^ {\prime} \| _ {\mathcal {C H} ^ {\nu - 1} (\mathcal {V} _ {l})} | h - h ^ {\prime} | ^ {\nu - 1} (164) \\ \leq \sup  _ {\| \varphi \| _ {\tilde {\mathbb {H}} _ {l}} = 1} \| \varphi \| _ {\mathcal {C H} ^ {\nu} \left(\mathcal {V} _ {l}\right)} \left| h - h ^ {\prime} \right| ^ {\nu - 1} (165) \\ \end{array}
$$
where in (163) the derivative  $\varphi'$  exists because, exactly as in the case  $\nu \leq 1$ , we can show that  $\widetilde{\mathbb{H}}_l \hookrightarrow \mathcal{CH}^\nu(\mathbb{R}^d)$ . This also implies that for a constant  $C = C_\nu$  we have
$$
\left\| \frac {\widetilde {K} _ {l} (x + \delta , \cdot) - \widetilde {K} _ {l} (x , \cdot)}{h} - \frac {\widetilde {K} _ {l} (x + \delta^ {\prime} , \cdot) - \widetilde {K} _ {l} (x , \cdot)}{h ^ {\prime}} \right\| _ {\widetilde {\mathbb {H}} _ {l}} \leq C | h - h ^ {\prime} | ^ {\nu - 1}. \tag {166}
$$
As  $|h - h'|^{\nu - 1} \to 0$  when  $h, h' \to 0$ , because  $\nu > 1$ , this proves that  $\frac{\widetilde{K}_l(x + \delta, \cdot) - \widetilde{K}_l(x, \cdot)}{h}$  is a Cauchy net in  $\widetilde{\mathbb{H}}_l$ : by completeness of  $\widetilde{\mathbb{H}}_l$  it converges in  $\widetilde{\mathbb{H}}_l$  to a limit. Since by general properties of RKHSs, convergence in  $\widetilde{\mathbb{H}}_l$  implies pointwise convergence, the limit satisfies
$$
\lim  _ {h \rightarrow 0} \frac {\widetilde {K} _ {l} (x + \delta , y) - \widetilde {K} _ {l} (x , y)}{h} = \frac {\partial \widetilde {K} _ {l}}{\partial x} (x, y). \tag {167}
$$
Hence the partial derivative  $\frac{\partial\widetilde{K}_l}{\partial x} (x,y)$  exists for all  $y$  and  $\frac{\partial\widetilde{K}_l}{\partial x} (x,\cdot)\in \widetilde{\mathbb{H}}_l$ . Moreover, by the isometry  $L^2 (\Omega)\ni h_l(x)\mapsto \mathbb{E}_{f\sim \Pi_n}h_l(x)h_l(\cdot) = \widetilde{K}_l(x,\cdot)\in \widetilde{\mathbb{H}}_l$ , we deduce that  $h_l$  is actually  $L^2 (\Omega)$ -differentiable, with an  $L^2 (\Omega)$ -derivative denoted as  $\dot{h}_l$ , and that the derivative process  $\dot{h}_l$  is Gaussian, as it is an  $L^2 (\Omega)$ -limit of Gaussian random variables, satisfying  $\mathbb{E}_{f\sim \Pi_n}\dot{h}_l(x)\dot{h}_l(y) = \left\langle \frac{\partial\widetilde{K}_l}{\partial x} (x,\cdot),\frac{\partial\widetilde{K}_l}{\partial x} (y,\cdot)\right\rangle_{\widetilde{\mathbb{H}}_l}$