SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \sup  _ {\substack {f \in \mathcal {H} _ {k} \\ \| f \| _ {\mathcal {H} _ {k}} \leq 1}} \mathbb {E} | f (t) - m _ {k, \mathbf {X}, f, \varepsilon} (t) | ^ {2} = \left\| k (t, \cdot) - \sum_ {j = 1} ^ {n} \alpha_ {j} k \left(x _ {j}, \cdot\right) \right\| _ {\mathbb {H} _ {k}} ^ {2} + \sigma_ {\varepsilon} ^ {2} \boldsymbol {\alpha} \boldsymbol {\alpha} ^ {\top} (211) \\ = k (t, t) - 2 \boldsymbol {\alpha} \mathbf {K} _ {\mathbf {X} t} + \underbrace {\underline {{\boldsymbol {\alpha}}} \mathbf {K} _ {\mathbf {X X}} \boldsymbol {\alpha} ^ {\top} + \sigma_ {\varepsilon} ^ {2} \boldsymbol {\alpha} \boldsymbol {\alpha} ^ {\top}} _ {\boldsymbol {\alpha} \mathbf {K} _ {\mathbf {X} t}} (212) \\ = k (t, t) - \alpha \mathbf {K} _ {\mathbf {X} t} = \underbrace {k (t , t) - \mathbf {K} _ {t \mathbf {X}} \left(\mathbf {K} _ {\mathbf {X X}} + \sigma_ {\varepsilon} ^ {2} \mathbf {I}\right) ^ {- 1} \mathbf {K} _ {\mathbf {X} t}} _ {v _ {k, \mathbf {X}} (t)}. (213) \\ \end{array}
$$
We now move to the misspecified case. Consider the RKHS  $\mathcal{H}_c$  for some other kernel  $c: \mathcal{X} \times \mathcal{X} \to \mathbb{R}$  instead of  $\mathbb{H}_k$ . Then, continuing from (210), write
$$
\sup  _ {\substack {f \in \mathcal {H} _ {c} \\ \| f \| _ {\mathcal {H} _ {c}} \leq 1}} \mathbb {E} | f (t) - m _ {k, \mathbf {X}, f, \varepsilon} (t) | ^ {2} = \left\| c (t, \cdot) - \sum_ {j = 1} ^ {n} \alpha_ {j} c \left(x _ {j}, \cdot\right) \right\| _ {\mathbb {H} _ {c}} ^ {2} + \sigma_ {\varepsilon} ^ {2} \boldsymbol {\alpha} \boldsymbol {\alpha} ^ {\top}. \tag{214}
$$
The next question is how to compute the norm on the right-hand side. There is not much hope of doing this exactly in the misspecified case: thus, we consider approximations. To this end, we take some large set of locations  $\mathbf{X}' \subseteq \mathcal{X}$ . Then we use  $\|g\|_{\mathbb{H}_c}^2 \approx g(\mathbf{X}')^\top \mathbf{C}_{\mathbf{X}'\mathbf{X}'}^{-1} g(\mathbf{X}')$  for  $g(\cdot) = c(t,\cdot) - \sum_{j=1}^{n} \alpha_j c(x_j,\cdot)$ . As a result, we obtain the approximation
$$
\sup  _ {\substack {f \in \mathcal {H} _ {c} \\ \| f \| _ {\mathcal {H} _ {c}} \leq 1}} \mathbb {E} | f (t) - m _ {k, \mathbf {X}, f, \varepsilon} (t) | ^ {2} \approx g \left(\mathbf {X} ^ {\prime}\right) ^ {\top} \mathbf {C} _ {\mathbf {X} ^ {\prime} \mathbf {X} ^ {\prime}} ^ {- 1} g \left(\mathbf {X} ^ {\prime}\right) + \sigma_ {\varepsilon} ^ {2} \boldsymbol {\alpha} \boldsymbol {\alpha} ^ {\top} = \tilde {v} _ {k, c, \mathbf {X}} (t) = v ^ {(e)} (t) \tag{215}
$$
where  $v^{(\mathrm{e})}(t)$  was first introduced in Section 4
To compute spatial averages of this quantity, let  $g_{t}(\cdot) = c(t,\cdot) - \sum_{j = 1}^{n}\alpha_{j}c(x_{j},\cdot)$ , the same as  $g$  before, but now with explicit dependence on  $t$ . Similarly, put  $\alpha_{t} = \mathbf{K}_{t\mathbf{X}}(\mathbf{K}_{\mathbf{XX}} + \sigma_{\varepsilon}^{2}\mathbf{I})^{-1}$ . Then
$$
g _ {t} \left(\mathbf {X} ^ {\prime}\right) = \mathbf {C} _ {\mathbf {X} ^ {\prime} t} - \mathbf {C} _ {\mathbf {X} ^ {\prime} \mathbf {X}} \boldsymbol {\alpha} _ {t} ^ {\top} = \mathbf {C} _ {\mathbf {X} ^ {\prime} t} - \mathbf {C} _ {\mathbf {X} ^ {\prime} \mathbf {X}} \left(\mathbf {K} _ {\mathbf {X X}} + \sigma_ {\varepsilon} ^ {2} \mathbf {I}\right) ^ {- 1} \mathbf {K} _ {\mathbf {X} t} \tag {216}
$$
$$
g _ {t} \left(\mathbf {X} ^ {\prime}\right) ^ {\top} \mathbf {C} _ {\mathbf {X} ^ {\prime} \mathbf {X} ^ {\prime}} ^ {- 1} g _ {t} \left(\mathbf {X} ^ {\prime}\right) = \left(\mathbf {C} _ {t \mathbf {X} ^ {\prime}} - \boldsymbol {\alpha} _ {t} \mathbf {C} _ {\mathbf {X} \mathbf {X} ^ {\prime}}\right) \mathbf {C} _ {\mathbf {X} ^ {\prime} \mathbf {X} ^ {\prime}} ^ {- 1} \left(\mathbf {C} _ {\mathbf {X} ^ {\prime} t} - \mathbf {C} _ {\mathbf {X} ^ {\prime} \mathbf {X}} \boldsymbol {\alpha} _ {t} ^ {\top}\right). \tag {217}
$$
From here we can also deduce that
$$
\frac {1}{| \mathbf {X} ^ {\prime} |} \sum_ {t \in \mathbf {X} ^ {\prime}} \tilde {v} _ {k, c, \mathbf {X}} (t) = \frac {1}{| \mathbf {X} ^ {\prime} |} \sum_ {t \in \mathbf {X} ^ {\prime}} g _ {t} \left(\mathbf {X} ^ {\prime}\right) ^ {\top} \mathbf {C} _ {\mathbf {X} ^ {\prime} \mathbf {X} ^ {\prime}} ^ {- 1} g _ {t} \left(\mathbf {X} ^ {\prime}\right) \tag {218}
$$
$$
= \frac {1}{| \mathbf {X} ^ {\prime} |} \operatorname {t r} \left(g _ {\mathbf {X} ^ {\prime}} \left(\mathbf {X} ^ {\prime}\right) ^ {\top} \mathbf {C} _ {\mathbf {X} ^ {\prime} \mathbf {X} ^ {\prime}} ^ {- 1} g _ {\mathbf {X} ^ {\prime}} \left(\mathbf {X} ^ {\prime}\right)\right) \tag {219}
$$
where  $g_{\mathbf{X}^{\prime}}(\mathbf{X}^{\prime}) = \mathbf{C}_{\mathbf{X}^{\prime}\mathbf{X}^{\prime}} - \mathbf{C}_{\mathbf{X}^{\prime}\mathbf{X}}(\mathbf{K}_{\mathbf{XX}} + \sigma_{\varepsilon}^{2}\mathbf{I})^{-1}\mathbf{K}_{\mathbf{XX}^{\prime}}$
G Full Experimental Details
All of our kernels were computed using GPJAX [38] and the GEOMETRIC Kernels library. We use three manifolds, each represented by a mesh: (i) a dumbbell-shaped manifold represented as a mesh with 1556 nodes, (ii) a sphere represented by an icosahedral mesh with 2562 nodes, and (iii) the Stanford dragon mesh, preprocessed to keep only its largest connected component, which has 100179 nodes. For the sphere, we also considered a finer icosahedral mesh with 10242 nodes, but this was found to have virtually no effect on the computed pointwise expected errors.
We use extrinsic Matérn and Riemannian Matérn kernels with the following hyperparameters:  $\sigma_f^2 = 1$  and  $\sigma_{\varepsilon}^{2} = 0.0005$ . For the truncated Karhunen-Loève expansion, we used  $J = 500$  eigenpairs obtained from the mesh. We selected smoothness values to ensure norm-equivalence of the intrinsic and extrinsic kernels' reproducing kernel Hilbert spaces, which was  $\nu = 5/2$  for the intrinsic model, and  $\nu = 5/2 + d/2$  for the extrinsic model, where  $d$  is the manifold's dimension. We used different length scales for each manifold:  $\kappa = 200$  for the dumbbell,  $\kappa = 0.25$  for the sphere, and  $\kappa = 0.05$  for the dragon, selected to ensure that the Gaussian processes were neither approximately constant, nor white-noise-like. We considered data sizes of  $N = 50$ ,  $N = 500$ , and  $N = 1000$ , respectively, for the dumbbell, sphere, and dragon, sampled uniformly from the mesh's nodes, which in each case resulted in a reasonably-uniform distribution of points across the manifold. Finally, for the extrinsic pointwise error approximation, we used a subset  $\mathbf{X}'$  uniformly sampled from each mesh's nodes, of size equal to the data size. For each respective test set, we used the full mesh. Each experiment was repeated for 10 different seeds.
To set the length scales for the extrinsic process, we used maximum marginal likelihood optimization on the full data, except for the dumbbell whose full data size is small and for which we instead generated a larger set consisting of 500 points. We optimized only the length scale, leaving all other hyperparameters fixed. We used ADAM with a learning rate of 0.005, and an initialization equal to the length scale  $\kappa$  of the intrinsic model, except for the dumbbell where this leads to divergence and we instead used an initial value of  $\kappa /4$ . We ran the optimizer for a total of 1000 steps. With these settings, we found empirically that maximum marginal likelihood optimization always converged.
10See HTTPS://GPJAX.READHEDOCS.IO and HTTPS://GEOMETRIC-Kernels.GITHB.IO.