SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Bernardo Fichera
Viacheslav Borovitskiy
Andreas Krause2
Aude Billard
$^{1}$  EPFL
Gaussian process regression is widely used because of its ability to provide well-calibrated uncertainty estimates and handle small or sparse datasets. However, it struggles with high-dimensional data. One possible way to scale this technique to higher dimensions is to leverage the implicit low-dimensional manifold upon which the data actually lies, as postulated by the manifold hypothesis. Prior work ordinarily requires the manifold structure to be explicitly provided though, i.e. given by a mesh or be known to be one of the well-known manifolds like the sphere. In contrast, in this paper we propose a Gaussian process regression technique capable of inferring implicit structure directly from data (labeled and unlabeled) in a fully differentiable way. For the resulting model, we discuss its convergence to the Matérn Gaussian process on the assumed manifold. Our technique scales up to hundreds of thousands of data points, and improves the predictive performance and calibration of the standard Gaussian process regression in some high-dimensional settings.
Gaussian processes are among the most adopted models for learning unknown functions within the Bayesian framework. Their data efficiency and aptitude for uncertainty quantification make them appealing for modeling and decision-making applications in the fields like robotics (Deisenroth and Rasmussen, 2011), geostatistics (Chiles and Delfiner, 2012), numerics (Hennig et al., 2015), etc.
The most widely used Gaussian process models, like squared exponential and Matérn (Rasmussen and Williams, 2006), impose the simple assumption of differentiability of the unknown function while also respecting the geometry of  $\mathbb{R}^d$  by virtue of being stationary or isotropic. Such simple assumptions make uncertainty estimates reliable, albeit too conservative at times. The same simplicity makes these models struggle from the curse of dimensionality. We hypothesize that it is still possible to leverage these simple priors for real world high-dimensional problems granted that they are adapted to the implicit low-dimensional submanifolds where the data actually lies, as illustrated by Figure 1.
(a) Euclidean, prediction
(b) Ours, prediction
(c)Euclidean, uncertainty
(d) Ours, uncertainty
Figure 1: Euclidean (standard Matérn-5/2 kernel) vs ours (implicit manifold) Gaussian process regression for data that lies on a dumbbell-shaped curve (1-dimensional manifold) assumed unknown. The data contains a small set of labeled points and a large set of unlabeled points. Our technique recognizes that the two lines in the middle are intrinsically far away from each other, giving a much better model on and near the manifold. Far away from the manifold it reverts to the Euclidean model.
Code available at https://github.com/nash169/manifold-gp.
Recent works in machine learning generalized Matérn Gaussian processes for modeling functions on non-Euclidean domains such as manifolds or graphs (Azangulov et al., 2022; 2023; Borovitskiy et al., 2021; 2023; 2020). Crucially, this line of work assumes known geometry (e.g. a manifold or a graph) beforehand. In this work we aim to widen the applicability of Gaussian processes for higher dimensional problems by automatically learning the implicit low-dimensional manifold upon which the data lies, the existence of which is suggested by the manifold hypothesis. We propose a new model which learns this structure and approximates the Matérn kernel on the implicit manifold.
Our approach can operate in both supervised and semi-supervised settings, with the emphasis on the latter: uncovering the implicit manifold may require a lot of samples from it, however these samples need not be labeled, and unlabeled data is usually more abundant. Taking inspiration in the manifold learning results of Coifman and Lafon (2006), Dunson et al. (2021) and others we approximate the unknown manifold by an appropriately weighted nearest neighbor graph. Then we use graph Matérn kernels thereon as approximations to the manifold Matérn kernels of Borovitskiy et al. (2020), extending them to the vicinity of the manifold in the ambient  $\mathbb{R}^d$  in an appropriate way.
1.1 Related Work and Contribution
High-dimensional Gaussian process regression is an area of active research, primarily motivated by decision making applications like Bayesian optimization. There are three main directions in this area: (1) selecting a small subset of input dimensions, (2) learning a small number of new features by linearly projecting the inputs and (3) learning non-linear features. Our technique belongs to the third direction. Further details on the area can be found in the recent review by Binois and Wycoff (2022).
A close relative of our technique in the literature is described in Dunson et al. (2022). It targets the low-dimensional setting where the inputs are densely sampled on the underlying surface. It is based on the heat (diffusion) kernels on graphs as in Kondor and Lafferty (2002) and uses the Nyström method to extend kernels to  $\mathbb{R}^d$ , both of which may incur a high computational cost. Another close relative is the concurrent work by Peach et al. (2023) on modeling vector fields on implicit manifolds.
We are targeting the high-dimensional setting. Here, larger datasets of partly labeled points are often needed to infer geometry. Because of this, we emphasize computational efficiency by leveraging sparse precision matrix structure of Matérn kernels (as opposed to the heat kernels) and use KNN for sparsifying the graph and accelerating the Nyström method. This results in linear computational complexity with respect to the number of data points. Furthermore, the model we propose is fully differentiable, which may be used to find both kernel and geometry hyperparameters by maximizing the marginal likelihood. Finally, to get reasonable predictions on the whole ambient space  $\mathbb{R}^d$ , we combine the prediction of the geometric model with the prediction of a classical Euclidean Gaussian process, weighting these by the relative distance to the manifold.
The geometric model is differentiable with respect to its kernel-, likelihood- and geometry-related hyperparameters, with gradient evaluation cost being linear with respect to the number of data points. After training, we can efficiently compute the predictive mean and kernel as well as sample the predictive model, providing the basic computational primitives needed for the downstream applications like Bayesian optimization. We evaluate our technique on a synthetic low-dimensional example and test it in a high-dimensional large dataset setting of predicting rotation angles of rotated MNIST images, improving over the standard Gaussian process regression.
2 Gaussian Processes
A Gaussian process  $f \sim \mathrm{GP}(m, k)$  is a distribution over functions on a set  $\mathcal{X}$ . It is determined by the mean function  $m(\pmb{x}) = \mathbb{E}f(\pmb{x})$  and the covariance function (kernel)  $k(\pmb{x}, \pmb{x}^{\prime}) = \operatorname{Cov}(f(\pmb{x}), f(\pmb{x}^{\prime}))$ .
Given data  $\mathbf{X},\pmb{y}$ , where  $\mathbf{X} = (x_{1},..,x_{n})^{\top}$  and  $\pmb{y} = (y_{1},..,y_{n})^{\top}$  with  $\pmb{x}_i\in \mathcal{X}^d$ ,  $y_{i}\in \mathbb{R}$ , one usually assumes  $y_{i} = f(\pmb{x}_{i}) + \varepsilon_{i}$  where  $\varepsilon_{i}\sim \mathrm{N}(0,\sigma_{\varepsilon}^{2})$  is IID noise and  $f\sim \mathrm{GP}(0,k)$  is some prior Gaussian process, whose mean is assumed to be zero in order to simplify notation. The posterior distribution  $f\mid \pmb{y}$  is then another Gaussian process  $f\mid \pmb{y}\sim \mathrm{GP}(\hat{m},\hat{k})$  with (Rasmussen and Williams, 2006)
$$
\hat {m} (\cdot) = \mathbf {K} _ {(\cdot) \mathbf {X}} \left(\mathbf {K} _ {\mathbf {X X}} + \sigma_ {\varepsilon} ^ {2} \mathbf {I}\right) ^ {- 1} \boldsymbol {y}, \quad \hat {k} (\cdot , \cdot^ {\prime}) = \mathbf {K} _ {(\cdot , \cdot^ {\prime})} - \mathbf {K} _ {(\cdot) \mathbf {X}} \left(\mathbf {K} _ {\mathbf {X X}} + \sigma_ {\varepsilon} ^ {2} \mathbf {I}\right) ^ {- 1} \mathbf {K} _ {\mathbf {X} (\cdot^ {\prime})}, \tag {1}
$$
where the matrix  $\mathbf{K}_{\mathbf{X}\mathbf{X}}$  has entries  $k(\pmb{x}_i,\pmb{x}_j)$ , the vector  $\mathbf{K}_{\mathbf{X}(\cdot)} = \mathbf{K}_{(\cdot)\mathbf{X}}^\top$  has components  $k(\pmb{x}_i,\cdot)$ . If needed, one can efficiently sample  $f\mid \pmb{y}$  using pathwise conditioning (Wilson et al., 2020; 2021)
(a) Kernel, sphere  $\mathbb{S}_2$
Figure 2: Kernel values  $k(\cdot, \cdot)$  and samples for the Matérn-3/2 Gaussian processes on the sphere manifold  $\mathbb{S}_2$  and for the approximating Matérn-5/2 process on a geodesic polyhedron graph  $\mathrm{P} \mathbb{S}_2$ .
(b) Kernel, graph  $\mathrm{P}\mathbb{S}_2$
(c) Sample, sphere  $\mathbb{S}_2$
(d) Sample, graph  $\mathbb{P}\mathbb{S}_2$
Matérn Gaussian processes—including the limiting  $\nu \to \infty$  case, squared exponential Gaussian processes—are the most popular family of models for  $\mathcal{X} = \mathbb{R}^d$ . These have zero mean and kernels
$$
k _ {\nu , \kappa , \sigma^ {2}} (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) = \sigma^ {2} \frac {2 ^ {1 - \nu}}{\Gamma (\nu)} \left(\sqrt {2 \nu} \frac {\| \boldsymbol {x} - \boldsymbol {x} ^ {\prime} \|}{\kappa}\right) ^ {\nu} K _ {\nu} \left(\sqrt {2 \nu} \frac {\| \boldsymbol {x} - \boldsymbol {x} ^ {\prime} \|}{\kappa}\right) \tag {2}
$$
where  $K_{\nu}$  is the modified Bessel function of the second kind (Gradshteyn and Ryzhik, 2014) and  $\nu ,\kappa ,\sigma^2$  are the hyperparameters responsible for smoothness, length scale and variance, respectively. We proceed to describe how Matern processes can be generalized to inputs  $\pmb{x}$  lying on explicitly given manifolds or graphs instead of the Euclidean space  $\mathbb{R}^d$
2.1 Matérn Gaussian Processes on Explicit Manifolds and Graphs
For a domain which is a Riemannian manifold, an obvious and natural idea for generalizing Matérn Gaussian processes could be to substitute the Euclidean distances  $\| x - x' \|$  in Equation (2) with the geodesic distance. However, this approach results in ill-defined kernels that fail to be positive semi-definite (Feragen et al., 2015; Gneiting, 2013).
Another direction for generalization is based on the stochastic partial differential equation (SPDE) characterization of Matérn processes first described by Whittle (1963):  $f \sim \mathrm{GP}(0, k_{\nu,\kappa,\sigma^2})$  solves
$$
\left(\frac {2 \nu}{\kappa^ {2}} - \Delta_ {\mathbb {R} ^ {d}}\right) ^ {\frac {\nu}{2} + \frac {d}{4}} f = \mathcal {W}, \tag {3}
$$
where  $\Delta_{\mathbb{R}^d}$  is the standard Laplacian operator and  $\mathcal{W}$  is the Gaussian white noise with variance proportional to  $\sigma^2$ . If taken to be the definition, this characterization can be easily extended to general Riemannian manifolds  $\mathcal{X} = \mathcal{M}$  by substituting  $\Delta_{\mathbb{R}^d}$  with the Laplace-Beltrami operator  $\Delta_{\mathcal{M}}$ , taking  $d = \dim \mathcal{M}$  and substituting  $\mathcal{W}$  with the appropriate generalization of the Gaussian white noise (Lindgren et al., 2011). Based on this idea, Borovitskiy et al. (2020) showed that on compact Riemannian manifolds, Matérn Gaussian processes are the zero-mean processes with kernels
$$
k _ {\nu , \kappa , \sigma^ {2}} (x, x ^ {\prime}) = \frac {\sigma^ {2}}{C _ {\nu , \kappa}} \sum_ {l = 0} ^ {\infty} \left(\frac {2 \nu}{\kappa^ {2}} + \lambda_ {l}\right) ^ {- \nu - d / 2} f _ {l} (x) f _ {l} \left(x ^ {\prime}\right), \tag {4}
$$
where  $-\lambda_{l},f_{l}$  are eigenvalues and eigenfunctions of the Laplace-Beltrami operator and  $C_{\nu ,\kappa}$  is the normalizing constant ensuring that  $\frac{1}{\mathcal{X}}\int_{\mathcal{X}}k_{\nu ,\kappa ,\sigma^2}(x,x)\mathrm{d}x = \sigma^2$  . This, alongside with considerations from Azangulov et al. (2022) allows one to practically compute  $k_{\nu ,\kappa ,\sigma^2}$  for many compact manifolds.
If the domain  $\mathcal{X}$  is a weighted undirected graph  $\mathcal{G}$ , we can also use Equation (3) to define Matérn Gaussian processes on  $\mathcal{G}$  (Borovitskiy et al., 2021). In this case,  $\Delta_{\mathbb{R}^d}$  is substituted with the minus graph Laplacian  $-\Delta_{\mathcal{G}}$  and  $\mathcal{W} \sim \mathrm{N}(0, \sigma_{\mathcal{W}}^2 \mathbf{I})$  is the vector of IID Gaussians. Here, SPDE transforms into a stochastic linear system, whose solution is of the same form as Equation (4) but with a finite sum instead of the infinite series, with  $d = 0$  because there is no canonical notion of dimension for graphs and with  $\lambda_l, f_l$  being the eigenvalues and eigenvectors—as functions on the node set—of the matrix  $\Delta_{\mathcal{G}}$ . These processes are illustrated on Figure 2.
3 Implicit Manifolds and Gaussian Processes on Them
Consider a dataset  $\mathbf{X} = (\pmb{x}_1,..,\pmb{x}_N)^\top$ ,  $\pmb{x}_i \in \mathbb{R}^d$  partially labeled with labels  $y_1,..,y_n \in \mathbb{R}$ ,  $n \leq N$ . Assume that  $\pmb{x}_i$  are IID randomly sampled from a compact Riemannian submanifold  $\mathcal{M} \subseteq \mathbb{R}^d$ . As by Section 2.1, the manifold  $\mathcal{M}$  is associated to a family of Matérén Gaussian processes tailored to its geometry. We do not assume to know  $\mathcal{M}$ , only the fact that it exists, hence the question is: how can we recover the kernels of the aforementioned geometry-aware processes from the observed dataset?
It is clear from Equation (4) that to recover  $k_{\nu,\kappa,\sigma^2}$  we need to get the eigenpairs  $-\lambda_l, f_l$  of the Laplace-Beltrami operator on  $\mathcal{M}$ . Naturally, for a finite dataset this can only be done approximately. We proceed to discuss the relevant theory of Laplace-Beltrami eigenpair approximation.
3.1 Background on Approximating the Eigenpairs of the Laplace-Beltrami Operator
There exists a number of theoretical and empirical results on eigenpair approximation. Virtually all of them study approximating the implicit manifold by some kind of a weighted undirected graph<sup>1</sup> with node set  $\{\pmb{x}_1, \dots, \pmb{x}_N\}$  and weights that are somehow determined by the Euclidean distances  $\| \pmb{x}_i - \pmb{x}_j \|$ . The eigenvalues of the graph Laplacian on this graph are supposed to approximate the eigenvalues of the Laplace-Beltrami operator, while the eigenvectors—regarded as functions on the node set—approximate the values of the eigenfunctions of the Laplace-Beltrami operator at  $\pmb{x}_i \in \mathcal{M}$ . To approximate eigenfunctions elsewhere, any sort of continuous (smooth) interpolation suffices.
There are three popular notions of graph Laplacian. Let us denote the adjacency matrix of the weighted graph by  $\mathbf{A}$  and define  $\mathbf{D}$  to be the diagonal degree matrix with  $\mathbf{D}_{ii} = \sum_{j}\mathbf{A}_{ij}$ . Then
$$
\underbrace {\Delta_ {\mathrm {u n}} = \mathbf {D} - \mathbf {A}} _ {\text {u n n o r m a l i z e d}}, \quad \underbrace {\Delta_ {\mathrm {s y m}} = \mathbf {I} - \mathbf {D} ^ {- 1 / 2} \mathbf {A D} ^ {- 1 / 2}} _ {\text {s y m m e t r i c n o r m a l i z e d}}, \quad \underbrace {\Delta_ {\mathrm {r w}} = \mathbf {I} - \mathbf {D} ^ {- 1} \mathbf {A}} _ {\text {r a n d o m w a l k n o r m a l i z e d}}. \tag {5}
$$
The first two of these are symmetric positive semi-definite matrices, the third is, generally speaking, non-symmetric. However, from the point of view of linear operators, all of them can be considered symmetric (self-adjoint) positive semi-definite: the first two with respect to the standard Euclidean inner product  $\langle \cdot ,\cdot \rangle$ , and the third one with respect to the modified inner product  $\langle v,u\rangle_{\mathbf{D}} = \langle \mathbf{D}v,u\rangle$ . Thus for each there exists an orthonormal basis of eigenvectors and eigenvalues are non-negative.2
The most common way to define the graph is by setting  $\mathbf{A}_{ij} = \exp \left(-\| x_i - x_j\|^2 /4\alpha^2\right)$  for an  $\alpha >0$  If  $x_{i}$  are IID samples from the uniform distribution on the manifold  $\mathcal{M}$ , then all of the graph Laplacians, each multiplied by an appropriate power of  $\alpha$ , converge to the Laplace-Beltrami operator, both pointwise (Hein et al., 2007) and spectrally (Garcia Trillos et al., 2020), i.e. in the sense of eigenpair convergence, at least at the node set of the graph. However, if the inputs  $x_{i}$  are sampled non-uniformly, graph Laplacians, at best, converge to different continuous limits, none of which coincides with the Laplace-Beltrami operator (Hein et al., 2007).
Coifman and Lafon (2006) proposed a clever trick to handle non-uniformly sampled data  $\boldsymbol{x}_1, \dots, \boldsymbol{x}_N$ . Starting with  $\tilde{\mathbf{A}}$  and  $\tilde{\mathbf{D}}$  defined in the same way as  $\mathbf{A}$  and  $\mathbf{D}$  before, they define  $\mathbf{A} = \tilde{\mathbf{D}}^{-1}\tilde{\mathbf{A}}\tilde{\mathbf{D}}^{-1}$ . Intuitively, this corresponds to normalizing by the kernel density estimator to cancel out the unknown density. The corresponding  $\Delta_{\mathrm{rw}}$  then converges pointwise to the Laplace-Beltrami operator (Hein et al., 2007), though  $\Delta_{\mathrm{un}}$  and  $\Delta_{\mathrm{sym}}$  do not: they converge to different continuous limits. Dunson et al. (2021, Theorem 2) show that, under technical regularity assumptions, eigenvalues  $\lambda_k$  and renormalized eigenvectors of  $\Delta_{\mathrm{rw}}$  converge to the respective eigenvalues and eigenfunctions of the Laplace-Beltrami operator, regardless of the sampling density of  $x_i$ .
Both in the simple case and in the sampling density independent case, the graphs and their respective Laplacians turn out to be dense, requiring a lot of memory to store and being inefficient to operate with. To make computations efficient, sparse graphs such as KNN graphs are much more preferable over the dense graphs. Spectral convergence for KNN graphs is studied, for example, in Calder and Trillos (2022), for  $\mathbf{A}_{ij} = h\big(\| \pmb{x}_i - \pmb{x}_j\| /\alpha \big)$  with a compactly supported regular function  $h$  , and with
<sup>1</sup>Other possibilities include linear (classical PCA) or quadratic (Pavutnitskiy et al., 2022) approximations. See the books by Lee and Verleysen (2007) and Ma and Fu (2011) for additional context.
${}^{2}$  Naturally,for the random walk normalized Laplacian  ${\Delta }_{\mathrm{{rw}}}$  the orthonormality is with respect to  $\left\langle  {\cdot , \cdot  }\right\rangle  {}_{\mathbf{D}}$  .
$^3$ García Trillos et al. (2020) do not explicitly study  $\Delta_{\mathrm{sym}}$ . Since  $\Delta_{\mathrm{sym}}$  and  $\Delta_{\mathrm{rw}}$  are similar matrices, they share eigenvalues, so eigenvalue convergence for  $\Delta_{\mathrm{sym}}$  is trivial, the eigenvector convergence, however, is not.
limit depending on the sampling density. Unfortunately, we are unaware of any spectral convergence results in the literature that hold for KNN graphs and are independent of the data sampling density.
3.2 Approximating Matérn Kernels on Manifolds
Here we incorporate various convergence results, including but not limited to the ones described in Section 3.1, proving that all spectral convergence results imply the convergence of graph Matérn kernels to the respective manifold Matérn kernels.
Proposition 1. Denote the eigenpairs by  $\lambda_l, f_l$  for a graph Laplacian and by  $\lambda_l^{\mathcal{M}}, f_l^{\mathcal{M}}$  for the Laplace-Beltrami operator. Fix  $\delta > 0$ . Assume that, with probability at least  $1 - \delta$ , for all  $\varepsilon > 0$ , for  $\alpha$  small enough and for  $N$  large enough we have  $|\lambda_l - \lambda_l^{\mathcal{M}}| < \varepsilon$  and  $|f_l(\pmb{x}_i) - f_l^{\mathcal{M}}(\pmb{x}_i)| < \varepsilon$ . Then, with probability at least  $1 - \delta$ , we have  $k_{\nu,\kappa,\sigma^2}^{N,\alpha,L}(\pmb{x}_i, \pmb{x}_j) \to k_{\nu,\kappa,\sigma^2}(\pmb{x}_i, \pmb{x}_j)$  as  $\alpha \to 0$ ,  $N, L \to \infty$ , where
$$
k _ {\nu , \kappa , \sigma^ {2}} ^ {N, \alpha , L} (\boldsymbol {x} _ {i}, \boldsymbol {x} _ {j}) = \frac {\sigma^ {2}}{C _ {\nu , \kappa}} \sum_ {l = 0} ^ {L - 1} \left(\frac {2 \nu}{\kappa^ {2}} + \lambda_ {l}\right) ^ {- \nu - \dim (\mathcal {M}) / 2} f _ {l} (\boldsymbol {x} _ {i}) f _ {l} (\boldsymbol {x} _ {j}). \tag {6}
$$
Proof. First prove that the tail of the series in Equation (4) converges uniformly to zero, then combine this with eigenpair bounds. See details in Appendix A.
Remark. The convergence in  $\pmb{x}_i \in \mathcal{M}$  can be lifted to pointwise convergence for all  $\pmb{x} \in \mathcal{M}$  if eigenvectors are interpolated Lipschitz-continuously, simply because the eigenfunctions are smooth.
Inspired by this theory, we proceed to present the implicit manifold Gaussian process model.
3.3 Implicit Manifold Gaussian Process
Guided by the theory we described in the previous section we are now ready to formulate the implicit manifold Gaussian process model. Given the dataset  $\mathbf{x}_1, \dots, \mathbf{x}_N \in \mathbb{R}^d$  and  $y_1, \dots, y_n \in \mathbb{R}$ , we put
$$
\mathbf {A} = \tilde {\mathbf {D}} ^ {- 1} \tilde {\mathbf {A}} \tilde {\mathbf {D}} ^ {- 1}, \quad \tilde {\mathbf {A}} _ {i j} = S _ {K} \left(\boldsymbol {x} _ {i}, \boldsymbol {x} _ {j}\right) \exp \left(- \frac {\left\| \boldsymbol {x} _ {i} - \boldsymbol {x} _ {j} \right\| ^ {2}}{4 \alpha^ {2}}\right), \quad \tilde {\mathbf {D}} _ {i j} = \left\{ \begin{array}{l l} \sum_ {m} \tilde {\mathbf {A}} _ {i m} & i = j, \\ 0 & i \neq j. \end{array} \right. \tag {7}
$$
Here  $S_K(\pmb{x}_i, \pmb{x}_j) = 1$  if  $\pmb{x}_i$  is one of the  $K$  nearest neighbors of  $\pmb{x}_j$  or vice versa and  $S_K(\pmb{x}_i, \pmb{x}_j) = 0$  otherwise; all matrices are of size  $N \times N$  and depend on  $\alpha$  and  $K$  as hyperparameters. Thanks to the coefficient  $S_K(\pmb{x}_i, \pmb{x}_j)$  that performs KNN sparsification, the matrix  $\mathbf{A}$  is sparse when  $K \ll N$ .<sup>4</sup>
Then we consider the operator  $\pmb{\Delta}_{\mathrm{rw}} = \mathbf{I} - \mathbf{D}^{-1}\mathbf{A}$  defined by Equation (5), whose matrix is also sparse. Denoting its eigenvalues—ordered from the smallest to the largest—by  $0 = \lambda_0 \leq \lambda_1 \leq \ldots \leq \lambda_{N-1}$ , and its eigenvectors—orthonormal under the modified inner product  $\langle \cdot, \cdot \rangle_D$  and regarded as functions on the node set of the graph—by  $f_0, f_1, \ldots, f_{N-1}$ , we define Matérn kernel on graph nodes  $\pmb{x}_i$  by
$$
k _ {\nu , \kappa , \sigma^ {2}} ^ {\mathbf {X}} \left(\boldsymbol {x} _ {i}, \boldsymbol {x} _ {j}\right) = \frac {\sigma^ {2}}{C _ {\nu , \kappa}} \sum_ {l = 0} ^ {L - 1} \Phi_ {\nu , \kappa} \left(\lambda_ {l}\right) f _ {l} \left(\boldsymbol {x} _ {i}\right) f _ {l} \left(\boldsymbol {x} _ {j}\right), \quad \Phi_ {\nu , \kappa} (\lambda) = \left(\frac {2 \nu}{\kappa^ {2}} + \lambda\right) ^ {- \nu}, \tag {8}
$$
where  $L$  does not need to be equal to the actual number  $N$  of eigenpairs. Doing so means truncating the high frequency eigenvectors  $(f_{l}$  for  $l$  large), which always contribute less to the sum because they correspond to smaller values of  $\Phi_{\nu ,\kappa}(\lambda_l)$ . This can massively reduce the computational costs.
By Proposition 1, Equation (8) approximates the manifold Matérn kernel with smoothness  $\nu' = \nu - \dim(\mathcal{M}) / 2$ . We adopt such a reparametrization because it does not require estimating the a priori unknown  $\dim(\mathcal{M})$ . This, however, makes the typical assumption of  $\nu \in \{1/2, 3/2, 5/2\}$  inadequate. We chose a particular graph Laplacian normalization, namely the random walk normalized graph Laplacian  $\Delta_{\mathrm{rw}}$ , to approximate the true Laplace-Beltrami operator regardless of the potential non-uniform sampling of  $x_1, \ldots, x_N$ , based on the theoretical insights described in Section 3.1.
The kernel in Equation (8) is only defined on the set of nodes  $\boldsymbol{x}_i$ , next step is to extend it to the whole space  $\mathbb{R}^d$ . Extending kernels is usually a difficult problem because one has to worry about positive