SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Note:  $\tilde{\mathbf{A}}_{ii} = 1$ , as if the graph has loops. Assuming  $\tilde{\mathbf{A}}_{ii} = 0$  would lead to discontinuities at later stages.
(a) Extended eigenvector  $\pmb{f}_3(\cdot)$
Figure 3: Different quantities connected to kernel extension. Notice that the values on subfigures (a) and (b) are artificially restricted to the set  $\mathrm{dist}(\cdot, \mathcal{M}) < 3\alpha$  to maintain numerical stability.
(b) Extended kernel  $k_{\nu ,\kappa ,\sigma^2}^{\mathbf{X}}(\cdot ,\cdot)$
(c) Function  $\gamma (\mathrm{dist}(\cdot ,\mathcal{M}))$
semi-definiteness. To work around it, we extend the features  $f_{l}$ . For this, we use Nyström method: we allow the first argument of  $S_{K}$  to be an arbitrary vector from  $\mathbb{R}^d$ , defining  $S_{K}(\pmb{x},\pmb{x}_{j}) = 1$  if  $\pmb{x}_{j}$  is one of the  $K$  nearest neighbors of  $\pmb{x}$  among  $\pmb{x}_1,\dots,\pmb{x}_N$ . This allows us to extend  $\tilde{\mathbf{A}}$ ,  $\tilde{\mathbf{D}}$ ,  $\mathbf{A}$  and  $\mathbf{D}$  as
$$
\tilde {A} (\boldsymbol {x}, \boldsymbol {x} _ {j}) = S _ {K} (\boldsymbol {x}, \boldsymbol {x} _ {j}) \exp \left(- \frac {\left\| \boldsymbol {x} - \boldsymbol {x} _ {j} \right\| ^ {2}}{4 \alpha^ {2}}\right), \quad \tilde {D} (\boldsymbol {x}) = \sum_ {j = 1} ^ {N} \tilde {A} (\boldsymbol {x}, \boldsymbol {x} _ {j}) = \sum_ {\boldsymbol {x} _ {j} \in \operatorname {K N N} (\boldsymbol {x})} \tilde {A} (\boldsymbol {x}, \boldsymbol {x} _ {j}), \tag {9}
$$
$$
A (\boldsymbol {x}, \boldsymbol {x} _ {j}) = \frac {\tilde {A} (\boldsymbol {x} , \boldsymbol {x} _ {j})}{\tilde {D} (\boldsymbol {x}) \tilde {D} (\boldsymbol {x} _ {j})}, \quad D (\boldsymbol {x}) = \sum_ {j = 1} ^ {N} A (\boldsymbol {x}, \boldsymbol {x} _ {j}) = \sum_ {\boldsymbol {x} _ {j} \in \operatorname {K N N} (\boldsymbol {x})} A (\boldsymbol {x}, \boldsymbol {x} _ {j}), \tag {10}
$$
where  $\mathrm{KNN}(\pmb{x})$  is the set of the  $K$  nearest neighbors from  $\pmb{x}$  among  $\pmb{x}_1, \dots, \pmb{x}_N$ . With this, we define
$$
f _ {l} (\boldsymbol {x}) = \frac {1}{1 - \lambda_ {l}} \sum_ {j = 1} ^ {N} \frac {A (\boldsymbol {x} , \boldsymbol {x} _ {j})}{D (\boldsymbol {x})} f _ {l} (\boldsymbol {x} _ {j}) = \frac {1}{1 - \lambda_ {l}} \sum_ {\boldsymbol {x} _ {j} \in \operatorname {K N N} (\boldsymbol {x})} \frac {A (\boldsymbol {x} , \boldsymbol {x} _ {j})}{D (\boldsymbol {x})} f _ {l} (\boldsymbol {x} _ {j}). \tag {11}
$$
It is easy to check that this extension perfectly reproduces the values  $f_{l}(\pmb{x}_{j})$ , simply because  $A(\pmb{x}, \pmb{x}_{j})$  coincides with  $\mathbf{A}_{ij}$  when  $\pmb{x} = \pmb{x}_{i}$  and because  $(f_{l}(\pmb{x}_{1}),.., f_{l}(\pmb{x}_{N}))^{\top}$  is the eigenvector of  $\mathbf{A}$  corresponding to the eigenvalue  $1 - \lambda_{l}$ . It also allows us to extend the kernel  $k_{\nu,\kappa,\sigma^{2}}^{\mathbf{X}}$  as well, such that  $k_{\nu,\kappa,\sigma^{2}}^{\mathbf{X}}(\pmb{x}, \pmb{x}^{\prime})$  is defined for arbitrary  $\pmb{x}, \pmb{x}^{\prime} \in \mathbb{R}^{d}$  and Equation (8) still holds for the nodes  $\pmb{x}_{1},.., \pmb{x}_{N}$ . We visualize an extended eigenvector and an extended kernel in Figures 3a and 3b.
For  $\pmb{x}$  far away from the nodes  $\pmb{x}_j$  the values of  $\tilde{D}(\pmb{x})$  become very small, making the extension procedure numerically unstable. Furthermore, the geometric model is generally not so relevant far away from the manifold. Because of this, the final predictive model  $f^{(p)} \sim \mathrm{GP}(m^{(p)}, k^{(p)})$  combines the geometric model  $f^{(m)} \sim \mathrm{GP}(m^{(m)}, k^{(m)})$  the posterior under the kernel  $k_{\nu, \kappa, \sigma^2}^{\mathbf{X}}$  we defined just above—with the standard Euclidean model  $f^{(e)} \sim \mathrm{GP}(m^{(e)}, k^{(e)})$  the posterior under the standard Euclidean Gaussian process in  $\mathbb{R}^d$ , for instance with the squared exponential kernel:
$$
f ^ {(p)} (\boldsymbol {x}) = \gamma (\boldsymbol {x}) f ^ {(m)} (\boldsymbol {x}) + (1 - \gamma (\boldsymbol {x})) f ^ {(e)} (\boldsymbol {x}), \quad \gamma (\boldsymbol {x}) = \exp \left(1 - \frac {(3 \alpha) ^ {2}}{(3 \alpha) ^ {2} - \operatorname {d i s t} (\boldsymbol {x} , \mathcal {M}) ^ {2}}\right), \tag {12}
$$
where  $\gamma$  is a bump function that is zero outside the  $3\alpha$  neighborhood of the manifold, illustrated in Figure 3c. Here  $\mathrm{dist}(\pmb{x},\mathcal{M})$  can be computed as the distance from  $\pmb{x}$  to its nearest neighbor node  $\pmb{x}_j$ .<sup>5</sup>
4 Efficient Training of the Implicit Manifold Gaussian Processes
Here we describe how to perform the implicit manifold Gaussian process regression efficiently, being able to handle hundreds of thousands of points, in both supervised and semi-supervised regimes.
In all cases we need efficient (approximate) KNN to build a graph, extend the kernel beyond the nodes  $\boldsymbol{x}_i$  and combine the geometric model with the standard Euclidean one. For this we use FAISS (Johnson et al., 2019). The resulting sparse matrices, such as the Laplacian  $\Delta_{\mathrm{rw}}$ , we represent as black box functions capable of performing matrix-vector multiplications for any given input vector.
In practice it makes sense to compute  $\mathrm{dist}(\pmb{x},\mathcal{M})$  as the average of distances between  $\pmb{x}$  and its  $K$  nearest neighbors to smoothen the resulting  $\gamma (\pmb {x})$  —this is computationally cheap given an efficient KNN implementation.
After hyperparameters are found—we will return to their search later—we need to compute the eigenpairs  $\lambda_l, f_l$  of  $\Delta_{\mathrm{rw}}$ . For this we run Lanczos algorithm (Meurant, 2006) to evaluate the eigenpairs  $\lambda_l^{\mathrm{sym}}, f_l^{\mathrm{sym}}$  of the symmetric matrix  $\Delta_{\mathrm{sym}}$ , putting  $\lambda_l = \lambda_l^{\mathrm{sym}}$  and  $f_l = \mathbf{D}^{-1/2} f_l^{\mathrm{sym}}$  because the matrices  $\Delta_{\mathrm{rw}}$  and  $\Delta_{\mathrm{sym}}$  are similar, i.e.  $\Delta_{\mathrm{rw}} = \mathbf{D}^{-1/2} \Delta_{\mathrm{sym}} \mathbf{D}^{1/2}$ . Importantly, Lanczos algorithm only relies on matrix-vector products with  $\Delta_{\mathrm{sym}}$ . We only compute a few hundred of eigenpairs, asking Lanczos to provide twice or thrice as many and disregarding the rest.
When there is a lot of labeled data, we approximate the classical Euclidean (e.g. squared exponential) kernel using random Fourier features approximation (Rahimi and Recht, 2007), this allows linear computational complexity scaling with respect to the number of data points. The number of Fourier features is taken to be equal to  $L$ , with the same  $L$  as in Equation (8).
As it was already mentioned, we need to find hyperparameters  $\hat{\pmb{\theta}} = (\hat{\alpha},\hat{\kappa},\hat{\sigma}^2,\hat{\sigma}_\varepsilon^2)$  that determine the graph, Gaussian process prior and the noise variance that fit the observations  $\pmb{y}$  best. The  $\nu$  parameter we assume manually fixed. To avoid nonsensical parameter values—a common difficulty often occurring when the data is scarce—one might want to assume a prior  $p(\pmb {\theta})$  on  $\pmb{\theta}$ . Some specific choices of  $p(\pmb {\theta})$  are discussed in Appendix C. Then  $\hat{\pmb{\theta}}$  is a maximum a posteriori (MAP) estimate:
$$
\hat {\boldsymbol {\theta}} = \arg \max  _ {\boldsymbol {\theta}} \log p (\boldsymbol {y} \mid \boldsymbol {\theta}, \mathbf {X}) + \log p (\boldsymbol {\theta}). \tag {13}
$$
To simplify hyperparameter initialization and align with zero prior mean assumption it makes sense to preprocess  $y_{i}$  to be centered and normalized.
To solve the optimization problem in Equation (13) we use restarted gradient descent. Repeatedly evaluating the gradient of  $\log p(\pmb{y} \mid \pmb{\theta}, \mathbf{X})$  is the main computational bottleneck. The key idea for doing this efficiently—viable for integer values of  $\nu$ —is to reduce matrix-vector products with Matérn kernels' precision to iterated matrix-vector products with the Laplacian, which is sparse. First, we describe this in detail in the noiseless supervised setting, where the idea is most directly applicable.
4.1 Noiseless Supervised Learning
Here we assume that all inputs are labeled, i.e.  $N = n$ , and all observations are noiseless, i.e.  $\sigma_{\varepsilon}^{2} = 0$ .
Denoting by  $\mathbf{P}_{\mathbf{XX}} = \mathbf{K}_{\mathbf{XX}}^{-1}$  the precision matrix, the log-likelihood  $\log p(\pmb {y}\mid \pmb {\theta},\mathbf{X})$  , up to a multiplicative constant and an additive constant irrelevant for optimization, is given by
$$
\mathcal {L} (\boldsymbol {\theta}) = - \log \det  (\mathbf {K} _ {\mathbf {X X}}) - \boldsymbol {y} ^ {\top} \mathbf {K} _ {\mathbf {X X}} ^ {- 1} \boldsymbol {y} = \log \det  (\mathbf {P} _ {\mathbf {X X}}) - \boldsymbol {y} ^ {\top} \mathbf {P} _ {\mathbf {X X}} \boldsymbol {y}. \tag {14}
$$
Its gradient may be given and then subsequently approximated (Gardner et al., 2018) by
$$
\frac {\partial \mathcal {L} (\boldsymbol {\theta})}{\partial \boldsymbol {\theta}} = \operatorname {t r} \left(\mathbf {P} _ {\mathbf {X X}} ^ {- 1} \frac {\partial \mathbf {P} _ {\mathbf {X X}}}{\partial \boldsymbol {\theta}}\right) - \boldsymbol {y} ^ {\top} \frac {\partial \mathbf {P} _ {\mathbf {X X}}}{\partial \boldsymbol {\theta}} \boldsymbol {y} \approx \boldsymbol {z} ^ {\top} \mathbf {P} _ {\mathbf {X X}} ^ {- 1} \frac {\partial \mathbf {P} _ {\mathbf {X X}}}{\partial \boldsymbol {\theta}} \boldsymbol {z} - \boldsymbol {y} ^ {\top} \frac {\partial \mathbf {P} _ {\mathbf {X X}}}{\partial \boldsymbol {\theta}} \boldsymbol {y}, \tag {15}
$$
where  $z$  is a random vector consisting of IID variables that are either 1 or -1 with probability  $1/2$ . The first term on the right-hand side is the stochastic estimate of the trace of Hutchinson (1989). Since the kernels from Section 3.3 coincide with graph Matérn kernels on the nodes  $x_{i}$ , we have
$$
\mathbf {K} _ {\mathbf {X X}} = \frac {\sigma^ {2}}{C _ {\nu , \kappa}} \sum_ {l = 0} ^ {L - 1} \Phi_ {\nu , \kappa} (\lambda_ {l}) \boldsymbol {f} _ {l} \boldsymbol {f} _ {l} ^ {\top}, \quad \Delta_ {\mathrm {r w}} \boldsymbol {f} _ {l} = \lambda_ {l} \boldsymbol {f} _ {l}, \quad \boldsymbol {f} _ {l} ^ {\top} \mathbf {D} \boldsymbol {f} _ {m} = \delta_ {l m}. \tag {16}
$$
However, as the graph bandwidth  $\alpha$  is one of the hyperparameters we optimize over, using Equation (16) would entail repeated eigenpair computations and differentiating through this procedure. Because of this, we use an alternative way to compute matrix-vector products  $\mathbf{P}_{\mathbf{XX}}\mathbf{u}$  detailed below. $^6$
Proposition 2. Assuming  $\nu \in \mathbb{N}$ , the precision matrix  $\mathbf{P}_{\mathbf{X}\mathbf{X}}$  of  $k_{\nu, \kappa, \sigma^2}^{\mathbf{X}}(\pmb{x}_i, \pmb{x}_j)$  can be given by
$$
\mathbf {P} _ {\mathbf {X X}} = \frac {\sigma^ {- 2}}{C _ {\nu , \kappa} ^ {- 1}} \mathbf {D} \underbrace {\left(\frac {2 \nu}{\kappa^ {2}} \mathbf {I} + \boldsymbol {\Delta} _ {\mathrm {r w}}\right) \cdots \cdots \left(\frac {2 \nu}{\kappa^ {2}} \mathbf {I} + \boldsymbol {\Delta} _ {\mathrm {r w}}\right)} _ {\nu t i m e s}. \tag {17}
$$
Though automatic differentiability could in principle work for iterative methods like the Lanczos algorithm, the amount of memory required for storing the gradients of the intermediate steps quickly becomes prohibitive.
Using Proposition 2 to evaluate matrix-vector products  $\mathbf{P}_{\mathbf{XX}}\mathbf{u}$  and conjugate gradients (Meurant, 2006) to solve  $z^{\top}\mathbf{P}_{\mathbf{XX}}^{-1}$  using only the matrix-vector products, we can efficiently evaluate the right-hand side of Equation (15), with linear costs with respect to  $N$ , assuming that the graph is sparse. Preconditioning (Wenger et al., 2022) can be used to further improve the efficiency of the solve.
4.2 Noiseless Semi-Supervised Learning
Here we assume that inputs are partly unlabeled, i.e.  $N \neq n$ , while observations are still noiseless, i.e.  $\sigma_{\varepsilon}^{2} = 0$ . Denote  $\mathbf{Z}$  to be the labeled part of  $\mathbf{X}$ . Then the matrix  $\mathbf{K}_{\mathbf{XX}}$  in the log-likelihood given by Equation (14) should be substituted with  $\mathbf{K}_{\mathbf{ZZ}}$ . However, while  $\mathbf{P}_{\mathbf{XX}}$  can be represented using Equation (17), the precision  $\mathbf{P}_{\mathbf{ZZ}} = \mathbf{K}_{\mathbf{ZZ}}^{-1}$  cannot. We thus compute it as the Schur complement:
$$
\mathbf {P} _ {\mathbf {Z Z}} = \mathbf {A} - \mathbf {B D} ^ {- 1} \mathbf {C}, \quad \mathbf {P} _ {\mathbf {X X}} = \left( \begin{array}{l l} \mathbf {A} & \mathbf {B} \\ \mathbf {C} & \mathbf {D} \end{array} \right), \tag {18}
$$
where partitioning of  $\mathbf{P}_{\mathbf{XX}}$  corresponds to partitioning  $\mathbf{X}$  into  $\mathbf{Z}$  and the rest. Evaluating a matrix-vector product  $\mathbf{P}_{\mathbf{ZZ}}\mathbf{u}$  requires a solve of  $\mathbf{D}^{-1}(\mathbf{C}\mathbf{u})$ . This solve can also be performed using conjugate gradients, keeping the computational complexity linear in  $N$  but increasing the constants.
4.3 Handling Noisy Observations
Finally, we assume noisy observations, i.e.  $\sigma_{\varepsilon}^{2} > 0$ . The inputs can be partially unlabeled, i.e.  $N \neq n$ . In this case, matrix  $\mathbf{K}_{\mathbf{XX}}$  in the log-likelihood given by Equation (14) should be substituted with  $\mathbf{K}_{\mathbf{ZZ}} + \sigma_{\varepsilon}^{2}\mathbf{I}$ . To reduce this to the previously considered cases, we use the Taylor expansion
$$
\left(\mathbf {K} _ {\mathbf {Z} \mathbf {Z}} + \sigma_ {\varepsilon} ^ {2} \mathbf {I}\right) ^ {- 1} \approx \mathbf {K} _ {\mathbf {Z} \mathbf {Z}} ^ {- 1} - \sigma_ {\varepsilon} ^ {2} \mathbf {K} _ {\mathbf {Z} \mathbf {Z}} ^ {- 2} + \sigma_ {\varepsilon} ^ {4} \mathbf {K} _ {\mathbf {Z} \mathbf {Z}} ^ {- 3} - \dots = \mathbf {P} _ {\mathbf {Z} \mathbf {Z}} - \sigma_ {\varepsilon} ^ {2} \mathbf {P} _ {\mathbf {Z} \mathbf {Z}} ^ {2} + \sigma_ {\varepsilon} ^ {4} \mathbf {P} _ {\mathbf {Z} \mathbf {Z}} ^ {3} - \dots \tag {19}
$$
In practice, we only use the first two terms on the right-hand side as an approximation. This allows to retain linear computational complexity scaling with respect to  $N$  but increases the constants.
4.4 Resulting Algorithm
Here we provide a concise summary of the implicit manifold Gaussian process regression algorithm.
Step 1: KNN-index. Construct the KNN index on the points  $x_{1}, \ldots, x_{N}$ . This allows linear time evaluation of any matrix-vector product with  $\tilde{\mathbf{A}}$ , and thus also with  $\mathbf{A}$ ,  $\Delta_{rw}$ ,  $\mathbf{P}_{\mathbf{XX}}$  for  $\nu \in \mathbb{N}$ , etc.
Step 2: hyperparameter optimization. Find the hyperparameters  $\hat{\theta}$  that solve Equation (13). Assuming  $\nu = \hat{\nu} \in \mathbb{N}$  is manually fixed, this relies only on matrix-vector products with  $\Delta_{rw}$ .
Step 3: computing the eigenpairs. Fixing the graph bandwidth  $\hat{\alpha}$  found on Step 2, compute the eigenpairs  $\lambda_l, f_l$  corresponding to the  $L$  smallest eigenvalues  $\lambda_l$ . For large  $N$ , use Lanczos algorithm.
After the steps above are finished, Equations (8) and (11) define the geometric kernel  $k_{\hat{\nu}, \hat{\kappa}, \hat{\sigma}^2}^{\mathbf{X}}(\boldsymbol{x}, \boldsymbol{x}')$  for arbitrary  $\boldsymbol{x}, \boldsymbol{x}' \in \mathbb{R}^d$ . Then the respective prior  $\mathrm{GP}(0, k_{\hat{\nu}, \hat{\kappa}, \hat{\sigma}^2}^{\mathbf{X}})$  can be conditioned by the labeled data in the standard way, yielding the posterior  $f^{(m)} \sim \mathrm{GP}(m^{(m)}, k^{(m)})$ . To get sensible predictions far away from the data, the geometric model  $f^{(m)}$  is convexly combined with an independently trained classical Gaussian process model, as given by Equation (12). The resulting predictive model is still a Gaussian process, sum of two appropriately weighted independent Gaussian processes.
Remark. The number of neighbors  $K$ , the number of eigenpairs  $L$  and the smoothness  $\nu$  are assumed to be manually fixed parameters. Higher values of  $K$  and  $L$  improve the quality of approximation of the manifold kernel, which is often linked to better predictive performance, but requires more computational resources. The parameter  $\nu$  can be picked using cross validation or prior knowledge. Small integer values of  $\nu$  reduce computational costs, but may be inadequate for higher dimensions of the assumed manifold due to the  $\nu' = \nu - \dim(\mathcal{M}) / 2$  link with the manifold kernel smoothness  $\nu'$ .
We start in Section 5.1 by examining a simple synthetic example to gain intuition on how noise-sensitive the technique is. Then in Section 5.2 we consider real datasets, showing improvements in higher dimensions. More experiments, results, and additional discussion can be found in Appendix B
Figure 4: The ground truth function on the dumbbell manifold and the predictions of the implicit manifold Gaussian process regression (IMGP) under different levels of noise.
(b) Noiseless, prediction
(c) Low noise, prediction
(d) High noise, prediction
5.1 Synthetic Examples
We consider a one-dimensional manifold resembling the shape of a dumbbell which already appeared in Figures 1 and 3. The unknown function  $f_{*}$  is defined by fixing a point  $x^{*}$  in the top left part of the dumbbell, and computing  $\sin(\mathrm{d}(x^{*},\cdot))$  where  $\mathrm{d}(\cdot,\cdot)$  denotes the geodesic (intrinsic) distance between a pair of points on the manifold. This function is illustrated in Figure 4a.
To measure performance we primarily rely on measuring negative log-likelihood (NLL) on the dense mesh of test locations. We do this because such metric is able to combine accuracy and calibration simultaneously. Additionally, we present the root mean square error (RMSE).
We investigate a semi-supervised setting where the number of unlabeled points is large ( $N - n = 1546$ ) and the number of labeled points is small ( $n = 10$ ). We contaminate the inputs with noise, putting  $\mathbf{X} = \mathbf{X}_{\mathrm{noiseless}} + \mathrm{N}(0, \sigma_{\mathbf{X}}^{2}\mathbf{I})$  and do the same with the outputs, putting  $\pmb{y} = f_{*}(\mathbf{X}) + \mathrm{N}(0, \sigma_{\mathbf{y}}^{2}\mathbf{I})$  for various values of  $\sigma_{\mathbf{X}}, \sigma_{\mathbf{y}} > 0$ . Specifically, we consider  $\sigma_{\mathbf{X}} = \sigma_{\mathbf{y}} = \beta \in \{0, 0.01, 0.05\}$  to which we refer to as the noiseless setting, the low noise setting and the high noise setting, respectively.
The results for these are visualized in Figures 4b to 4d with performance metrics reported in Table 1. The implicit manifold Gaussian process regression is referred to as IMGP (we use  $\nu = 1$ ) and it is compared with the standard Euclidean Matérn-5/2 Gaussian process. IMGP performs much better in the noiseless and the low noise settings. The high noise is enough to damage the calibration of IMGP, as it ties with the baseline model: NLL is slightly worse and RMSE is slightly better.
In Appendix B.1 we show how performance depends on the fraction  $n / N$  of labeled data points, the truncation level  $L$  and we discuss the choice of the  $K$  parameter in KNN. Additionally, in Appendix B.2 we consider noise-sensitivity for a 2D manifold.
5.2 High Dimensional Datasets
For the high-dimensional setting, we considered predicting rotation angles for MNIST-based datasets. Additionally, we examined a high-dimensional dataset from the UCI ML Repository, CT slices.
5.2.1 Setup
Datasets. We consider two MNIST-based datasets. The first one is created by extracting a single image per digit from the complete MNIST dataset. By randomly rotating these 10 images we obtained  $N = 10000$  training samples and 1000 testing samples. We call it Single Rotated MNIST (SR-MNIST). For the second dataset, we select 100 random samples from MNIST. By randomly rotating these, we generate  $N = 100000$  training samples, most will be unlabeled, and 10000 testing samples. We call it Multiple Rotated MNIST (MR-MNIST). The last dataset, CT slices, has dimensionality of  $d = 385$ , we split it to have  $N = 24075$  training samples and 24075 testing samples. Dataset names can be complemented by the fraction of labeled samples, e.g. MR-MNIST-10% refers to  $n = 10\% N$ .
RMSE	NLL
β = 0	β = 0.01	β = 0.05	β = 0	β = 0.01	β = 0.05
Euclidean Matérn-5/2	0.98	0.99 ± 0.02	1.02 ± 0.03	-2.17	-2.09 ± 0.03	-1.91 ± 0.10
IMGP	0.33	0.34 ± 0.02	1.00 ± 0.02	-5.02	-4.19 ± 0.1	-1.91 ± 1.88
Table 1: Performance metrics for the dumbbell manifold with varying magnitude of noise  $\beta$  .
Method	MNIST	CT slices
SR - 10%	MR - 1%	MR - 10%	5%	10%	25%
EGP	-0.54 ± 0.01	-0.20 ± 0.01	-0.43 ± 0.01	-0.80 ± 0.02	-0.96 ± 0.00	-1.20 ± 0.09
S-IMGP	-1.42 ± 0.01	2.24 ± 0.20	-0.68 ± 0.08	0.47 ± 0.06	-0.59 ± 0.08	-0.08 ± 0.01
SS-IMGP	-1.52 ± 0.01	-0.59 ± 0.01	-0.79 ± 0.00	26.1 ± 12.7	1.03 ± 0.09	-0.72 ± 0.68
S-IMGP (full)	-	-	-	0.64 ± 0.83	0.88 ± 0.29	-0.42 ± 0.10
SS-IMGP (full)	-	-	-	-2.48 ± 0.08	-2.35 ± 0.04	-1.99 ± 0.04
Table 2: Negative log likelihood on test samples for real datasets. For RMSE see Tables 4 and 5.
Methods. We consider implicit manifold Gaussian processes in the supervised regime (S-IMGP) and in the semi-supervised regime (SS-IMGP). We compare them to the GPyTorch implementation of the Euclidean Matérn-5/2 Gaussian Process. We refer to it as the Euclidean Gaussian Process (EGP).
Additional details. We run 100 iterations of hyperparameter optimization using Adam with a fixed learning rate of 0.01. For MNIST, with use IMGP with  $\nu = 2$ ; for CT slices—with  $\nu = 3$ .
5.2.2 Results
Table 2 shows the negative log-likelihood metric for different datasets and methods on the test set. The respective RMSEs are presented in Appendix B.3. On SR-MNIST, IMGP outperforms EGP in both supervised and semi-supervised scenarios. MR-MNIST is more challenging. In the supervised setting for  $n = 1\% N$ , S-IMGP is incapable of inferring the underlying manifold structure, performing worse than EGP. However, SS-IMGP, with more data to infer manifold from, performs best. For  $n = 10\% N$ , IMGP gets a better grip of the dataset's geometry, outperforming EGP in both regimes.
For CT slices, regardless of  $n$ , both S-IMGP and SS-IMGP performed poorly. Looking for an explanation, we considered two modifications. First, we fixed the graph bandwidth  $\hat{\alpha}$  found in the algorithm's Step 2 (cf. Section 4.4), and re-optimized the other hyperparameters  $\kappa, \sigma^2, \sigma_{\varepsilon}^2$  by maximizing the likelihood of the eigenpair-based model (truncated to  $L = 2000$  eigenpairs) computed in the algorithm's Step 3. This resulted in limited improvement but did not change the big picture—in fact, values for S-IMGP and SS-IMGP in Table 2 for CT slices already include this modification.
Second, on top of this hyperparameter re-optimization, we tried computing the eigenpairs using torch.linalg.eigh instead of the Lanczos implementation in GPyTorch, taking the same number  $L = 2000$  of eigenpairs. The resulting methods S-IMGP (full) and SS-IMGP (full) showed considerable improvement over the baseline, as shown in Table 2. This indicated an issue with the quality of eigenpairs derived from the Lanczos method which requires further investigation. We discuss this in Appendix B.3, together with the aforementioned hyperparameter re-optimization procedure.