SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this work, we propose the implicit manifold Gaussian process regression technique. It is able to use unlabeled data to improve predictions and uncertainty calibration by learning the implicit manifold upon which the data lies, being inspired by the convergence of graph Matérn Gaussian processes to their manifold counterparts. This helps building better probabilistic models in higher dimensional settings where the standard Euclidean Gaussian processes usually struggle. This is supported by our experiments in a synthetic low-dimensional setting and for high-dimensional datasets. Leveraging sparse structure of graph Matérn precision matrices and efficient approximate KNN, the technique is able to scale to large datasets of hundreds of thousands points, which is especially important in high dimension, where a large number of unlabeled points is often needed to learn the implicit manifold. The model is fully differentiable, making it possible to infer hyperparameters in the usual way.
Limitations. The quality of the constructed graph significantly influences the technique's performance. When dealing with data from complex manifolds or exhibiting highly non-uniform density, simplistic KNN strategies might fail to capture the manifold structure due to their reliance on a single graph bandwidth. In such scenarios, larger values of parameters  $K$  and  $L$ , or in high dimensions, of parameter  $\nu$ , may be beneficial but could substantially increase computational costs. Furthermore, larger datasets coupled with high parameter values can lead to numerical stability issues, for instance, in the Lanczos algorithm, calling for further improvements and research. Despite these challenges, our method shows promise for advancing probabilistic modeling in higher dimensions.
We express our gratitude to Alexander Shulzhenko (St. Petersburg University, mentored by VB), whose initial work on a similar problem and accessible source material at https://github.com/ AlexanderShulzhenko/MPLICIT-Manifold-Gaussian-Processes, while not included in the paper, sparked inspiration for this project. We thank Dr. Alexander Terenin (Cornell University) for generously making his Blender rendering scripts accessible to the public. These scripts, which aided us in creating Figure 2, can be found in his Ph.D. thesis repository at https://github.com/aterenin/phdthesis. BF and AB acknowledge support by the European Research Council (ERC), Advanced Grant agreement No 741945, Skill Acquisition in Humans and Robots. VB acknowledges support by an ETH Zürich Postdoctoral Fellowship.
I. Azangulov, A. Smolensky, A. Terenin, and V. Borovitskiy. Stationary Kernels and Gaussian Processes on Lie Groups and their Homogeneous Spaces I: the compact case. arXiv preprint arXiv:2208.14960, 2022. Cited on pages 2, 3.
I. Azangulov, A. Smolensky, A. Terenin, and V. Borovitskiy. Stationary Kernels and Gaussian Processes on Lie Groups and their Homogeneous Spaces II: non-compact symmetric spaces. arXiv preprint arXiv:2301.13088, 2023. Cited on page 2.
M. Binois and N. Wycoff. A survey on high-dimensional Gaussian process modeling with application to Bayesian optimization. ACM Transactions on Evolutionary Learning and Optimization, 2(2):1-26, 2022. Cited on page 2.
V. Borovitskiy, I. Azangulov, A. Terenin, P. Mostowsky, M. Deisenroth, and N. Durrande. Matérn Gaussian Processes on Graphs. In International Conference on Artificial Intelligence and Statistics, 2021. Cited on pages 2, 3.
V. Borovitskiy, M. R. Karimi, V. R. Somnath, and A. Krause. Isotropic Gaussian Processes on Finite Spaces of Graphs. In International Conference on Artificial Intelligence and Statistics, 2023. Cited on page 2.
V. Borovitskiy, A. Terenin, P. Mostowsky, and M. P. Deisenroth. Matérn Gaussian Processes on Riemannian Manifolds. In Advances in Neural Information Processing Systems, 2020. Cited on pages 2, 3, 14.
J. Calder and N. G. Trillos. Improved spectral convergence rates for graph Laplacians on  $\varepsilon$ -graphs and k-NN graphs. Applied and Computational Harmonic Analysis, 60:123-175, 2022. Cited on page 4.
J.-P. Chiles and P. Delfiner. *Geostatistics: Modeling Spatial Uncertainty*. John Wiley & Sons, 2012. Cited on page 1.
R. R. Coifman and S. Lafon. Diffusion Maps. Applied and Computational Harmonic Analysis. Special Issue: Diffusion Maps and Wavelets, 21(1):5-30, 2006. Cited on pages 2, 4.
E. De Vito, N. Mücke, and L. Rosasco. Reproducing kernel Hilbert spaces on manifolds: Sobolev and diffusion spaces. Analysis and Applications, 19(03):363-396, 2021. Cited on page 14.
M. Deisenroth and C. E. Rasmussen. PILCO: A model-based and data-efficient approach to policy search. In International Conference on Machine Learning, 2011. Cited on page 1.
H. Donnelly. Eigenfunctions of the Laplacian on compact Riemannian manifolds. Asian Journal of Mathematics, 10(1):115-126, 2006. Cited on page 14.
D. B. Dunson, H.-T. Wu, and N. Wu. Graph based Gaussian processes on restricted domains. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(2):414-439, 2022. Cited on page 2.
D. B. Dunson, H.-T. Wu, and N. Wu. Spectral convergence of graph Laplacian and heat kernel reconstruction in  $L^{\infty}$  from random samples. Applied and Computational Harmonic Analysis, 55:282-336, 2021. Cited on pages 2, 4.
P. Whittle. Stochastic Processes in Several Dimensions. Bulletin of the International Statistical Institute, 40:974-994, 1963. Cited on page 3.
J. T. Wilson, V. Borovitskiy, A. Terenin, P. Mostowsky, and M. P. Deisenroth. Efficiently Sampling Functions from Gaussian Process Posteriors. In International Conference on Machine Learning, 2020. Cited on page 2.
J. T. Wilson, V. Borovitskiy, A. Terenin, P. Mostowsky, and M. P. Deisenroth. Pathwise Conditioning of Gaussian Processes. Journal of Machine Learning Research, 22(105):1-47, 2021. Cited on page 2.
$$
k _ {\nu , \kappa , \sigma^ {2}} ^ {N, \alpha , L} \left(\boldsymbol {x} _ {i}, \boldsymbol {x} _ {j}\right) = \frac {\sigma^ {2}}{C _ {\nu , \kappa}} \sum_ {l = 0} ^ {L - 1} \left(\frac {2 \nu}{\kappa^ {2}} + \lambda_ {l}\right) ^ {- \nu - \dim (\mathcal {M}) / 2} f _ {l} \left(\boldsymbol {x} _ {i}\right) f _ {l} \left(\boldsymbol {x} _ {j}\right). \tag {6}
$$
Proof. Fix small  $\varepsilon > 0$ . We will prove that for  $\alpha$  small enough and  $N, L$  large enough we have  $|k_{\nu, \kappa, \sigma_f^2}(\pmb{x}_i, \pmb{x}_j) - k_{\nu, \kappa, \sigma_f^2}^{N, \alpha, L}(\pmb{x}_i, \pmb{x}_j)| < C\varepsilon$  for some  $C > 0$  with probability at least  $1 - \delta$ . Since the assumption holds on the same event of probability  $1 - \delta$  for all  $\varepsilon$ , this directly translates to the convergence on the same event. In fact, a probabilistic narrative is nonessential for what we actually prove, and we do not use it below. To simplify notation, we replace  $\sum_{l=0}^{L-1}$  by  $\sum_{l=0}^{L}$ .
First, for any  $L \in \mathbb{Z}_{>0}$  define the truncated version  $k_{\nu,\kappa,\sigma_f^2}^L$  of the manifold kernel  $k_{\nu,\kappa,\sigma_f^2}$  by
$$
k _ {\nu , \kappa , \sigma_ {f} ^ {2}} ^ {L} (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) = \frac {\sigma_ {f} ^ {2}}{C _ {\nu , \kappa}} \sum_ {l = 0} ^ {L} \left(\frac {2 \nu}{\kappa^ {2}} + \lambda_ {l} ^ {\mathcal {M}}\right) ^ {- \nu - \dim (\mathcal {M}) / 2} f _ {l} ^ {\mathcal {M}} (\boldsymbol {x}) f _ {l} ^ {\mathcal {M}} \left(\boldsymbol {x} ^ {\prime}\right). \tag {20}
$$
The manifold Matérn kernels are the reproducing kernels of Sobolev spaces, if the latter are defined appropriately (Borovitskiy et al., 2020). These are Mercer kernels (De Vito et al., 2021), hence, by Mercer's theorem,  $k_{\nu,\kappa,\sigma_f^2}^L \to k_{\nu,\kappa,\sigma_f^2}$  uniformly on  $\mathcal{M}$ , i.e. for  $L$  large enough we have
$$
\left| k _ {\nu , \kappa , \sigma_ {f} ^ {2}} ^ {L} (\boldsymbol {x} _ {i}, \boldsymbol {x} _ {j}) - k _ {\nu , \kappa , \sigma_ {f} ^ {2}} (\boldsymbol {x} _ {i}, \boldsymbol {x} _ {j}) \right| \leq \sup  _ {\boldsymbol {x}, \boldsymbol {x} ^ {\prime} \in \mathcal {M}} \left| k _ {\nu , \kappa , \sigma_ {f} ^ {2}} ^ {L} (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) - k _ {\nu , \kappa , \sigma_ {f} ^ {2}} (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) \right| <   \varepsilon . \tag {21}
$$
Now suppose that  $\alpha$  is small enough and  $N$  is large enough so that
$$
\left| \lambda_ {l} - \lambda_ {l} ^ {\mathcal {M}} \right| <   \varepsilon^ {\prime}, \quad \left| f _ {l} (\boldsymbol {x} _ {i}) - f _ {l} ^ {\mathcal {M}} (\boldsymbol {x} _ {i}) \right| <   \varepsilon^ {\prime}, \quad \varepsilon^ {\prime} = \min  \left(1, \left(\lambda_ {L} ^ {\mathcal {M}}\right) ^ {- \frac {d - 1}{4}}, \left(\lambda_ {L} ^ {\mathcal {M}}\right) ^ {- \frac {d - 1}{2}}\right) \frac {\varepsilon}{L} \tag {22}
$$
for all  $l \in \{1,..,L\}$  and for all  $i \in \{1,..,N\}$  with probability at least  $1 - \delta$ .
Assuming the manifold is connected, by Donnelly (2006) we have  $|f_{l}^{\mathcal{M}}| \leq C_{\lambda}(\lambda_{l}^{\mathcal{M}})^{\frac{d - 1}{4}}$  for  $l > 0$ , where  $C_{\lambda} > 0$  is a constant that depends on the geometry of the manifold. The case  $l = 0$  is special because  $\lambda_0^{\mathcal{M}} = 0$ . Since  $f_{0}^{\mathcal{M}}$  is a constant function, we have
$$
\left| f _ {l} ^ {\mathcal {M}} \right| \leq C _ {\lambda} \max  \left(\left(\lambda_ {l} ^ {\mathcal {M}}\right) ^ {\frac {d - 1}{4}}, 1\right) \tag {23}
$$
for all  $l \geq 0$ , where  $C_{\lambda}$  here is potentially different from the  $C_{\lambda}$  before. Assuming, without loss of generality,  $\varepsilon < 1$ , we have
$$
\begin{array}{l} \left| f _ {l} \left(\boldsymbol {x} _ {i}\right) f _ {l} \left(\boldsymbol {x} _ {j}\right) - f _ {l} ^ {\mathcal {M}} \left(\boldsymbol {x} _ {i}\right) f _ {l} ^ {\mathcal {M}} \left(\boldsymbol {x} _ {j}\right) \right| \leq \left| f _ {l} \left(\boldsymbol {x} _ {i}\right) \right| \left| f _ {l} \left(\boldsymbol {x} _ {j}\right) - f _ {l} ^ {\mathcal {M}} \left(\boldsymbol {x} _ {j}\right) \right| (24) \\ + \left| f _ {l} ^ {\mathcal {M}} \left(\boldsymbol {x} _ {j}\right) \right| \left| f _ {l} \left(\boldsymbol {x} _ {i}\right) - f _ {l} ^ {\mathcal {M}} \left(\boldsymbol {x} _ {i}\right) \right| (25) \\ \leq \varepsilon^ {\prime} \cdot \left(| f _ {l} (\boldsymbol {x} _ {i}) | + | f _ {l} ^ {\mathcal {M}} (\boldsymbol {x} _ {j}) |\right) (26) \\ \leq \varepsilon^ {\prime} \cdot \left(\left| f _ {l} \left(\boldsymbol {x} _ {i}\right) - f _ {l} ^ {\mathcal {M}} \left(\boldsymbol {x} _ {i}\right) \right| + \left| f _ {l} ^ {\mathcal {M}} \left(\boldsymbol {x} _ {i}\right) \right| + \left| f _ {l} ^ {\mathcal {M}} \left(\boldsymbol {x} _ {j}\right) \right|\right) (27) \\ \leq \varepsilon^ {\prime} \cdot \left(\varepsilon^ {\prime} + 2 C _ {\lambda} \max  \left(\left(\lambda_ {l} ^ {\mathcal {M}}\right) ^ {\frac {d - 1}{4}}, 1\right)\right) \leq \frac {(1 + 2 C _ {\lambda}) \varepsilon}{L}. (28) \\ \end{array}
$$
The function  $\Phi (\lambda) = \left(\frac{2\nu}{\kappa^2} +\lambda\right)^{-\nu -\dim (\mathcal{M}) / 2}$  is Lipschitz:  $|\Phi (\lambda) - \Phi (\lambda ')|\leq C_{\Phi}|\lambda -\lambda '|$  where
$$
C _ {\Phi} = \sup  _ {\lambda \geq 0} | \Phi^ {\prime} (\lambda) | = \sup  _ {\lambda \geq 0} \left(\nu + \dim (\mathcal {M}) / 2\right) \left(\frac {2 \nu}{\kappa^ {2}} + \lambda\right) ^ {- \nu - \dim (\mathcal {M}) / 2 - 1} = \left(\nu + \dim (\mathcal {M}) / 2\right) \left(\frac {2 \nu}{\kappa^ {2}}\right) ^ {- \nu - \dim (\mathcal {M}) / 2 - 1}. \tag {29}
$$
Define an auxiliary kernel with manifold eigenvalues and graph eigenfunctions by
$$
\tilde {k} _ {\nu , \kappa , \sigma_ {f} ^ {2}} ^ {L} (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) = \frac {\sigma_ {f} ^ {2}}{C _ {\nu , \kappa}} \sum_ {l = 0} ^ {L} \left(\frac {2 \nu}{\kappa^ {2}} + \lambda_ {l} ^ {\mathcal {M}}\right) ^ {- \nu - \dim (\mathcal {M}) / 2} f _ {l} (\boldsymbol {x}) f _ {l} \left(\boldsymbol {x} ^ {\prime}\right). \tag {30}
$$
$$
\begin{array}{l} \left. \right. \frac {C _ {\nu , \kappa}}{\sigma_ {f} ^ {2}} \left| k _ {\nu , \kappa , \sigma_ {f} ^ {2}} ^ {L} \left(\boldsymbol {x} _ {i}, \boldsymbol {x} _ {j}\right) - \tilde {k} _ {\nu , \kappa , \sigma_ {f} ^ {2}} ^ {L} \left(\boldsymbol {x} _ {i}, \boldsymbol {x} _ {j}\right)\right| \leq \sum_ {l = 0} ^ {L} \left(\frac {2 \nu}{\kappa^ {2}} + \lambda_ {l} ^ {\mathcal {M}}\right) ^ {- \nu - \dim (\mathcal {M}) / 2} \frac {(1 + 2 C _ {\lambda}) \varepsilon}{L} (31) \\ \leq \Phi (0) \left(1 + 2 C _ {\lambda}\right) \varepsilon . (32) \\ \end{array}
$$
Also, noting that  $|f_{l}(\pmb{x}_{i})|\leq \left|f_{l}^{\mathcal{M}}(\pmb{x}_{i}) - f_{l}(\pmb{x}_{i})\right| + \left|f_{l}^{\mathcal{M}}(\pmb{x}_{i})\right|\leq \varepsilon^{\prime} + C_{\lambda}\max ((\lambda_{l}^{\mathcal{M}})^{\frac{d - 1}{4}},1)$ , write
$$
\begin{array}{l} \left. \right. \frac {C _ {\nu , \kappa}}{\sigma_ {f} ^ {2}} \left| \tilde {k} _ {\nu , \kappa , \sigma_ {f} ^ {2}} ^ {L} \left(\boldsymbol {x} _ {i}, \boldsymbol {x} _ {j}\right) - k _ {\nu , \kappa , \sigma_ {f} ^ {2}} ^ {N, \alpha , L} \left(\boldsymbol {x} _ {i}, \boldsymbol {x} _ {j}\right)\right| \leq \sum_ {l = 0} ^ {L} \left| \Phi \left(\lambda_ {l} ^ {\mathcal {M}}\right) - \Phi (\lambda_ {l}) \right| | f _ {l} \left(\boldsymbol {x} _ {i}\right) | | f _ {l} \left(\boldsymbol {x} _ {j}\right) | (33) \\ \leq \sum_ {l = 0} ^ {L} C _ {\Phi} \left| \lambda_ {l} ^ {\mathcal {M}} - \lambda_ {l} \right| \left| f _ {l} \left(\boldsymbol {x} _ {i}\right) \right| \left| f _ {l} \left(\boldsymbol {x} _ {j}\right) \right| (34) \\ \leq \sum_ {l = 0} ^ {L} 2 C _ {\Phi} \varepsilon^ {\prime} \left(\left(\varepsilon^ {\prime}\right) ^ {2} + C _ {\lambda} ^ {2} \max  \left(\left(\lambda_ {l} ^ {\mathcal {M}}\right) ^ {\frac {d - 1}{2}}, 1\right)\right) (35) \\ \leq 2 C _ {\Phi} \left(1 + C _ {\lambda} ^ {2}\right) \varepsilon . (36) \\ \end{array}
$$
$$
\begin{array}{l} \left| k _ {\nu , \kappa , \sigma_ {f} ^ {2}} (\boldsymbol {x} _ {i}, \boldsymbol {x} _ {j}) - k _ {\nu , \kappa , \sigma_ {f} ^ {2}} ^ {N, \alpha , L} (\boldsymbol {x} _ {i}, \boldsymbol {x} _ {j}) \right| \leq \left| k _ {\nu , \kappa , \sigma_ {f} ^ {2}} (\boldsymbol {x} _ {i}, \boldsymbol {x} _ {j}) - k _ {\nu , \kappa , \sigma_ {f} ^ {2}} ^ {L} (\boldsymbol {x} _ {i}, \boldsymbol {x} _ {j}) \right| (37) \\ + \left| k _ {\nu , \kappa , \sigma_ {f} ^ {2}} ^ {L} \left(\boldsymbol {x} _ {i}, \boldsymbol {x} _ {j}\right) - \tilde {k} _ {\nu , \kappa , \sigma_ {f} ^ {2}} ^ {L} \left(\boldsymbol {x} _ {i}, \boldsymbol {x} _ {j}\right) \right| (38) \\ + \left| \tilde {k} _ {\nu , \kappa , \sigma_ {f} ^ {2}} ^ {L} \left(\boldsymbol {x} _ {i}, \boldsymbol {x} _ {j}\right) - k _ {\nu , \kappa , \sigma_ {f} ^ {2}} ^ {N, \alpha , L} \left(\boldsymbol {x} _ {i}, \boldsymbol {x} _ {j}\right) \right| (39) \\ \leq \varepsilon + \frac {\sigma_ {f} ^ {2}}{C _ {\nu , \kappa}} \left(\Phi (0) \left(1 + 2 C _ {\lambda}\right) + 2 C _ {\Phi} \left(1 + C _ {\lambda} ^ {2}\right)\right) \varepsilon . (40) \\ \end{array}
$$
This proves the claim.
Proposition 2. Assuming  $\nu \in \mathbb{N}$ , the precision matrix  $\mathbf{P}_{\mathbf{XX}}$  of  $k_{\nu, \kappa, \sigma^2}^{\mathbf{X}}(\boldsymbol{x}_i, \boldsymbol{x}_j)$  can be given by
$$
\mathbf {P} _ {\mathbf {X X}} = \frac {\sigma^ {- 2}}{C _ {\nu , \kappa} ^ {- 1}} \underbrace {\mathbf {D} \left(\frac {2 \nu}{\kappa^ {2}} \mathbf {I} + \boldsymbol {\Delta} _ {\mathrm {r w}}\right) \cdot \dots \cdot \left(\frac {2 \nu}{\kappa^ {2}} \mathbf {I} + \boldsymbol {\Delta} _ {\mathrm {r w}}\right)} _ {\nu t i m e s}. \tag {17}
$$
Proof. The covariance matrix  $\mathbf{K}_{\mathbf{XX}}$  is given by
$$
\mathbf {K} _ {\mathbf {X X}} = \frac {\sigma^ {2}}{C _ {\nu , \kappa}} \sum_ {l = 0} ^ {L - 1} \Phi (\lambda_ {l}) \boldsymbol {f} _ {l} \boldsymbol {f} _ {l} ^ {\top}, \quad \Phi (\lambda) = \left(\frac {2 \nu}{\kappa^ {2}} + \lambda\right) ^ {- \nu} \tag {41}
$$
where  $\pmb{f}_l = \mathbf{D}^{-1/2}\pmb{f}_l^{\mathrm{sym}}$  and  $\pmb{f}_l^{\mathrm{sym}}$  are the orthonormal eigenvectors of the symmetric normalized Laplacian  $\Delta_{\mathrm{sym}}$ . Denote
$$
\mathbf {K} _ {\mathbf {X X}} ^ {\text {s y m}} = \frac {\sigma^ {2}}{C _ {\nu , \kappa}} \sum_ {l = 0} ^ {L - 1} \Phi (\lambda_ {l}) \boldsymbol {f} _ {l} ^ {\text {s y m}} \left(\boldsymbol {f} _ {l} ^ {\text {s y m}}\right) ^ {\top} \tag {42}
$$
then  $\left(\mathbf{K}_{\mathbf{X}\mathbf{X}}^{\mathrm{sym}}\right)^{-1} = \frac{\sigma^{-2}}{C_{\nu,\kappa}^{-1}}\sum_{l = 0}^{L - 1}\left(\frac{2\nu}{\kappa^2} +\lambda_l\right)^\nu \pmb {f}_l^{\mathrm{sym}}(\pmb {f}_l^{\mathrm{sym}})^\top = \frac{\sigma^{-2}}{C_{\nu,\kappa}^{-1}}\left(\frac{2\nu}{\kappa^2}\mathbf{I} + \pmb {\Delta}_{\mathrm{sym}}\right)^\nu$  We have
$$
\mathbf {K} _ {\mathbf {X X}} = \frac {\sigma^ {2}}{C _ {\nu , \kappa}} \sum_ {l = 0} ^ {L - 1} \Phi (\lambda_ {l}) \mathbf {D} ^ {- 1 / 2} \boldsymbol {f} _ {l} ^ {\text {s y m}} \left(\boldsymbol {f} _ {l} ^ {\text {s y m}}\right) ^ {\top} \mathbf {D} ^ {- 1 / 2} = \mathbf {D} ^ {- 1 / 2} \mathbf {K} _ {\mathbf {X X}} ^ {\text {s y m}} \mathbf {D} ^ {- 1 / 2}. \tag {43}
$$
(a) RMSE depending on  $L$
Figure 5: Root Mean Square Error (RMSE) and Negative Log-Likelihood (NLL) for increasing number of eigenpairs  $L$  (left panels) and increasing fraction  $f = n\% N$  of labeled points (right panels). The legend in (a) and (b) refers to the number of hyperparameter optimization iterations.
(b) NLL depending on  $L$
(c) RMSE depending on  $f$
(d) NLL depending on  $f$
$$
\begin{array}{l} \frac {\sigma_ {f} ^ {- 2}}{C _ {\nu , \kappa} ^ {- 1}} \mathbf {D} \left(\frac {2 \nu}{\kappa^ {2}} \mathbf {I} + \boldsymbol {\Delta} _ {\mathrm {r w}}\right) ^ {\nu} = \frac {\sigma_ {f} ^ {- 2}}{C _ {\nu , \kappa} ^ {- 1}} \mathbf {D} \left(\frac {2 \nu}{\kappa^ {2}} \mathbf {I} + \mathbf {D} ^ {- 1 / 2} \boldsymbol {\Delta} _ {\mathrm {s y m}} \mathbf {D} ^ {1 / 2}\right) ^ {\nu} (44) \\ = \frac {\sigma_ {f} ^ {- 2}}{C _ {\nu , \kappa} ^ {- 1}} \mathbf {D} ^ {1 / 2} \left(\frac {2 \nu}{\kappa^ {2}} \mathbf {I} + \boldsymbol {\Delta} _ {\text {s y m}}\right) ^ {\nu} \mathbf {D} ^ {1 / 2} = \mathbf {D} ^ {1 / 2} \left(\mathbf {K} _ {\mathbf {X X}} ^ {\text {s y m}}\right) ^ {- 1} \mathbf {D} ^ {1 / 2} (45) \\ = \mathbf {K} _ {\mathbf {X X}} ^ {- 1} = \mathbf {P} _ {\mathbf {X X}}. (46) \\ \end{array}
$$
B Additional Experimental Results and Details
Here we provide additional results and details for synthetic examples and high-dimensional datasets.
B.1 1D Dumbbell Manifold
In this section, we analyze our method's sensitivity to the number of labeled points, spectrum truncation, and neighbor count in graph construction, offering deeper insights into its behavior.
Eigenpairs Truncation. From a theoretical standpoint, utilizing the complete set of eigenpairs to construct the kernel should be beneficial. However, this assumption may not hold true in cases where the optimization problem is not fully converged. The trends of RMSE and NLL, as depicted in Figures 5a and 5b, illustrate the impact of increasing the number of eigenpairs for 100, 200, and 1000 iterations. In situations where hyperparameter optimization does not converge fully, the length scale parameter  $\kappa$  fails to reach sufficiently high values necessary to generate appropriate spectral density decay, which in turn would properly weigh higher frequency eigenpairs. In such scenarios, truncating the spectrum is similar to increasing the length scale, which might improve results in certain cases. In the 1D scenario, due to its simplicity, we opted for a modest number of eigenpairs, namely  $L = 50$ . Exceeding this count did not yield any discernible improvements.
Dataset Size. We analyze the sensitivity to dataset size of our method (IMGP) against the Euclidean case (EGP) in the semi-supervised learning scenario. For fixed number of eigenpairs,  $L = 50$ , Figures 5c and 5d show performance metrics (RMSE and NLL) depending on the percentage  $n\% N$  of labeled points. In a typical scenario where we have at our disposal fewer labeled points compared to the number of unlabeled points, our method outperforms EGP in both accuracy (RMSE) and uncertainty quantification (NLL). As  $n$  increases EGP starts to match the performance of IMGP.
Number of neighbors. For the one dimensional case considered in this section, only three neighbors are necessary to capture the essential features of the manifold's geometry. Choosing a number less than three would hinder the algorithm's ability to capture the underlying manifold structure. On the other hand, increasing the number of neighbors beyond this threshold does not affect the solution, provided that sufficient time is allocated for hyperparameter optimization to converge and the graph bandwidth becomes small enough to "correct" for all undesired edges in the graph structure (for instance in the central region of the dumbbell). In high-dimensional problems where the dimension of the underlying manifold is unknown, this suggests to incrementally increase the number of neighbors until the loss function stops improving, if it is computationally feasible to try multiple values of  $K$ .
(e) Euclidean, prediction
(b) IMGP pred.  $\beta = 0$
(c)IMGP pred.  $\beta = 0.01$
(f) Euclidean, uncertainty
(d)IMGP pred.  $\beta = 0.05$
(g) IMGP, uncertainty
Figure 6: (a) Ground truth function on the complex 2D manifold and (b)-(d) predictions of the implicit manifold Gaussian process regression (IMGP) for increasing level of sampling noise  $\beta$ . (e)-(d) Euclidean GP prediction and uncertainty and (g) IMGP uncertainty in noiseless scenario.
B.2 2D Dragon Manifold
We consider another synthetic setting, a complex 2D manifold, depicted in Figure 6. The ground truth function is visualized in Figure 6a, it is the same as in the one-dimensional case, the sine of the geodesic distance to a point, which is located in the green area.
In the semi-supervised learning scenario, Figures 6e and 6b offer a comparison of the posterior mean between the Euclidean GP and IMGP, while Figures 6f and 6g illustrate the posterior standard deviation for both models—quite similar to each other, in this regime. Similar to what we did for the 1D compact manifold in Section 5.1, we evaluated the performance of IMGP under varying levels of sampling noise. Figures 6b to 6d display the IMGP predictions in the semi-supervised learning scenario as sampling noise increases. Similar to the 1D case, our approach consistently outperforms the standard Euclidean GP, as evidenced in Table 3.
Remark. We observed that for higher levels of sampling noise, the linear combination of posteriors, as described by Equation (12), significantly outperform the single geometric model.
B.3 High Dimensional Datasets
Rotated MNIST. For IMGP, we use  $\nu = 2$ . As discussed in Appendix B.1, the optimal number of eigenpairs  $L$  varies considerably depending on the convergence of the optimization problem. Given the limited number of iterations per run we opted to fix the number of eigenpairs at  $20\% N$  and  $2\% N$ , for SR-MNIST and MR-MNIST, respectively. Table 4 reports the complete results obtained for the rotated MNIST dataset, including both RMSE and NLL. Notably, these emphasize the importance of unlabeled points, as S-IMGP performs worse than both SS-IMGP and EGP on MR-MNIST-1%.
RMSE	NLL
β = 0	β = 0.01	β = 0.05	β = 0	β = 0.01	β = 0.05
Euclidean Matérn-5/2	0.24 ± 0.02	0.24 ± 0.01	0.26 ± 0.00	-0.85 ± 0.46	0.16 ± 1.58	1.26 ± 1.80
IMGP	0.12 ± 0.01	0.21 ± 0.01	0.22 ± 0.01	-2.14 ± 0.13	-1.51 ± 0.03	-1.30 ± 0.09
Table 3: Results for a complex 2D manifold with varying magnitude of sampling noise.