SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Paul Rosa
Alexander Terenin
University of Cambridge and Cornell University
Judith Rousseau
Gaussian processes are used in many machine learning applications that rely on uncertainty quantification. Recently, computational tools for working with these models in geometric settings, such as when inputs lie on a Riemannian manifold, have been developed. This raises the question: can these intrinsic models be shown theoretically to lead to better performance, compared to simply embedding all relevant quantities into  $\mathbb{R}^d$  and using the restriction of an ordinary Euclidean Gaussian process? To study this, we prove optimal contraction rates for intrinsic Matérn Gaussian processes defined on compact Riemannian manifolds. We also prove analogous rates for extrinsic processes using trace and extension theorems between manifold and ambient Sobolev spaces: somewhat surprisingly, the rates obtained turn out to coincide with those of the intrinsic processes, provided that their smoothness parameters are matched appropriately. We illustrate these rates empirically on a number of examples, which, mirroring prior work, show that intrinsic processes can achieve better performance in practice. Therefore, our work shows that finer-grained analyses are needed to distinguish between different levels of data-efficiency of geometric Gaussian processes, particularly in settings which involve small data set sizes and non-asymptotic behavior.
Gaussian processes provide a powerful way to quantify uncertainty about unknown regression functions via the formulation of Bayesian learning. Motivated by applications in the physical and engineering sciences, a number of recent papers [11, 9, 10, 37] have studied how to extend this model class to spaces with geometric structure, in particular Riemannian manifolds including important special cases such as spheres and Grassmannians [4], hyperbolic spaces and spaces of positive definite matrices [5], as well as general manifolds approximated numerically by a mesh [11].
These Riemannian Gaussian process models are starting to be applied for statistical modeling, and decision-making settings such as Bayesian optimization. For example, in a robotics setting, Jaquier et al. [27] has shown that using Gaussian processes with the correct geometric structure allows one to learn quantities such as the orientation of a robotic arm with less data compared to baselines. The same model class has also been used by Coveney et al. [15] to perform Gaussian process regression on a manifold which models the geometry of a human heart for downstream applications in medicine.
Given these promising empirical results, it is important to understand whether these learning algorithms have good theoretical properties, as well as their limitations. Within the Bayesian framework, a natural way to quantify data-efficiency and generalization error is to posit a data-generating mech-
(a) Intrinsic
(b) Extrinsic
(c) Intrinsic
Figure 1: Samples from different Matérn Gaussian processes on different manifolds, namely a one-dimensional dumbbell-shaped manifold and a two-dimensional sphere. Notice that the values across the dumbbell's bottleneck can be very different for the intrinsic process in (a), despite being very close in the ambient Euclidean distance and in contrast to the situation for the extrinsic model in (b). On the other hand, there is little qualitative difference between (c) and (d), since the embedding produces a reasonably-good global approximation to geodesic distances on the sphere.
(d) Extrinsic
anism model and study if—and how fast—the posterior distribution concentrates around the true regression function as the number of observations goes to infinity.
Within the Riemannian setting, it is natural to compare intrinsic methods, which are formulated directly on the manifold of interest, with extrinsic ones, which require one to embed the manifold within a higher-dimensional Euclidean space. For example, the two-dimensional sphere can be embedded into the Euclidean space  $\mathbb{R}^3$ : intrinsic Gaussian processes model functions on the sphere while extrinsic ones model functions on  $\mathbb{R}^3$ , which are then restricted to the sphere. Are the former more efficient than the latter? Since embeddings—even isometric ones—at best only preserve distances locally, they can induce spurious dependencies, as points can be close in the ambient space but far away with respect to the intrinsic geodesic distance: this is illustrated in Figure 1. In cases where embeddings significantly alter distances, one can expect intrinsic models to perform better, and it is therefore interesting to quantify such differences.
In other settings, the manifold on which the data lies can be unknown, which makes using intrinsic methods directly no longer possible. There, one would like to understand how well extrinsic methods can be expected to perform. According to the manifold hypothesis [18], it is common for perceptual data such as text and images to concentrate on a lower-dimensional submanifold within, for instance, pixel space or sequence space. It is therefore also interesting to investigate how Gaussian process models—which, being kernel-based, are simpler than for instance deep neural networks—perform in such scenarios, at least in the asymptotic regime.
In this work, we develop geometric analogs of the Gaussian process posterior contraction theorems of van der Vaart and van Zanten [56]. More specifically, we derive posterior contraction rates for three main geometric model classes: (1) the intrinsic Riemannian Matérn Gaussian processes, (2) truncated versions of the intrinsic Riemannian Matérn Gaussian processes, which are used in practice to avoid infinite sums, and (3) the extrinsic Euclidean Matérn Gaussian processes under the assumption that the data lies on a compact Riemannian manifold. In all cases, we focus on IID randomly-sampled input points—commonly referred to as random design in the literature—and contraction in the sense of the  $L^2(p_0)$  distance, defined in Section 2. We focus on compact Riemannian manifolds: this allows one to define Matérn Gaussian processes through their Karhunen-Loève expansions, which requires a discrete spectrum for the Laplace-Beltrami operator—see for instance Borovitskiy et al. [11] and Chavel [13], Chapter 1—and is a common setting in statistics [39].
Contributions. We show that all three classes of Gaussian processes lead to optimal procedures, in the minimax sense, as long as the smoothness parameter of the kernel is aligned with the regularity of the unknown function. While this result is natural—though non trivial—in the case of intrinsic Matérn processes, it is rather remarkable that it also holds for extrinsic ones. This means that in order to understand their differences better, finite-sample considerations are necessary. We therefore present experiments that compute the worst case errors numerically. These experiments highlight that intrinsic models are capable of achieving better performance in the small-data regime. We conclude with a discussion of why these results—which might at first seem counterintuitive—are very natural
when viewed from an appropriate mathematical perspective: they suggest that optimality is perhaps best seen as a basic property or an important guarantee that any sensible model should satisfy.
Gaussian process regression is a Bayesian approach to regression where the modeling assumptions are  $y_{i} = f(x_{i}) + \varepsilon_{i}$ , with  $\varepsilon_{i} \sim \mathrm{N}(0, \sigma_{\varepsilon}^{2})$ ,  $x_{i} \in X$ , and  $f$  is assigned a Gaussian process prior. A Gaussian process is a random function  $f: X \to \mathbb{R}$  for which all finite-dimensional marginal distributions are multivariate Gaussian. The distribution of such a process is uniquely determined by its mean function  $m(\cdot) = \mathbb{E}(f(\cdot))$  and covariance kernel  $k(\cdot, \cdot') = \operatorname{Cov}(f(\cdot), f(\cdot'))$ , hence we write  $f \sim \mathrm{GP}(m, k)$ .
For Gaussian process regression, the posterior distribution given the data is also a Gaussian process with probability kernel  $\Pi (\cdot \mid x,y) = \mathrm{GP}(m_{\Pi (\cdot \mid x,y)},k_{\Pi (\cdot \mid x,y)})$ , see Rasmussen and Williams [41],
$$
m _ {\Pi (\cdot | \boldsymbol {x}, \boldsymbol {y})} (\cdot) = \mathbf {K} _ {(\cdot) \boldsymbol {x}} \left(\mathbf {K} _ {\boldsymbol {x x}} + \sigma_ {\varepsilon} ^ {2} \mathbf {I}\right) ^ {- 1} \boldsymbol {y}, \tag {1}
$$
$$
k _ {\Pi (\cdot | \boldsymbol {x}, \boldsymbol {y})} (\cdot , \cdot^ {\prime}) = \mathbf {K} _ {(\cdot , \cdot^ {\prime})} - \mathbf {K} _ {(\cdot) \boldsymbol {x}} (\mathbf {K} _ {\boldsymbol {x x}} + \sigma_ {\varepsilon} ^ {2} \mathbf {I}) ^ {- 1} \mathbf {K} _ {\boldsymbol {x} (\cdot^ {\prime})}. \qquad (2)
$$
These quantities describe how incorporating data updates the information contained within the Gaussian process. We will be interested studying the case where  $X$  is a Riemannian manifold, but first review the existing theory on the asymptotic behaviour of the posterior when  $X = [0,1]^d$ .
2.1 Posterior Contraction Rates
Posterior contraction results describe how the posterior distribution concentrates around the true data generating process, as the number of observations increases, so that it eventually uncovers the true data-generating mechanism. The area of posterior asymptotics is concerned with understanding conditions under which this does or does not occur, with questions of posterior contraction rates—how fast such convergence occurs—being of key interest. At present, there is a well-developed literature on posterior contraction rates, see Ghosal and van der Vaart [20] for a review.
In the context of Gaussian process regression with random design, which is the focus of this paper, the true data generating process is assumed to be of the form
$$
y _ {i} \mid x _ {i} \sim \mathrm {N} \left(f _ {0} \left(x _ {i}\right), \sigma_ {\varepsilon} ^ {2}\right) \quad x _ {i} \sim p _ {0} \tag {3}
$$
where  $f_0 \in \mathcal{F} \subseteq \mathbb{R}^X$ , a class of real-valued functions, and  $\mathrm{N}(\mu, \sigma^2)$  denotes the Gaussian with moments  $\mu, \sigma^2$ . Note that, in this particular variant, these equations exactly mirror those of the Gaussian process model's likelihood, including the use of the same noise variance  $\sigma_{\varepsilon}^2$  in both cases: in this paper, we focus on the particular case where  $\sigma_{\varepsilon}$  is known in advance. This setting is restrictive, one can extend to an unknown  $\sigma_{\varepsilon} > 0$  using techniques that are not specific to our geometric setting: for instance, the approach of [55] allows to handle an unknown  $\sigma_{\varepsilon}$  if one assumes an upper and lower bound on it and keep the same contraction rates. In practice, more general priors, including ones that do not assume an upper or lower bound on  $\sigma_{\varepsilon}$ , can be used, such as a conjugate one like in Banerjee [6]—these can also be analyzed to obtain contraction rates, albeit with additional considerations. The generalization error for prediction in such models is strongly related to the weighted  $L^2$  loss given by
$$
\left\| f - f _ {0} \right\| _ {L ^ {2} \left(p _ {0}\right)} = \left(\int_ {X} | f (x) - f _ {0} (x) | ^ {2} \mathrm {d} p _ {0} (x)\right) ^ {1 / 2} \tag {4}
$$
which is arguably the natural way of measuring discrepancy between  $f$  and  $f_0$ , given the fact that the covariates  $x_i$  are sampled from  $p_0$ . The posterior contraction rate is then defined as
$$
\mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \mathbb {E} _ {f \sim \Pi (\cdot | \boldsymbol {x}, \boldsymbol {y})} \| f - f _ {0} \| _ {L ^ {2} \left(p _ {0}\right)} ^ {2} \tag {5}
$$
where  $\mathbb{E}_{f\sim \Pi (\cdot |x,y)}(\cdot)$  denotes expectation under the posterior distribution while  $\mathbb{E}_{x,y}(\cdot)$  denotes expectation under the true data generating process. In the case of covariates distributed on  $[0,1]^d$  posterior contraction rates have been derived under Matérn Gaussian process priors [47] in van der Vaart and van Zanten [56], who showed the following result.
<sup>1</sup>Note that other notions of posterior contraction can be found in the literature, see Ghosal and van der Vaart [20] and Rousseau [42] for examples that are slightly weaker than the definition we work with.
Result 1 (Theorem 2 of van der Vaart and van Zanten [56]). In the Bayesian regression model, let  $f$  be a mean-zero Matérn Gaussian process prior on  $\mathbb{R}^d$  with amplitude  $\sigma_f^2$ , length scale  $\kappa$ , and smoothness  $\nu > d/2$ . Assume that the true data generating process is given by (3), where  $p_0$  has a Lebesgue density on  $X = [0,1]^d$  which is bounded from below and above by  $0 < c_{p_0} < C_{p_0} < \infty$ , respectively. Let  $f_0 \in H^\beta \cap \mathcal{CH}^\beta$  with  $\beta > d/2$ , where  $H^\beta$  and  $\mathcal{CH}^\beta$  the Sobolev and Hölder spaces, respectively. Then there exists a constant  $C > 0$ , which does not depend on  $n$  but does depend on  $d$ ,  $\sigma_f^2$ ,  $\nu$ ,  $\kappa$ ,  $\beta$ ,  $p_0$ ,  $\sigma_\varepsilon^2$ ,  $\|f_0\|_{H^\beta(\mathcal{M})}$ , and  $\|f_0\|_{\mathcal{CH}^\beta(\mathcal{M})}$ , such that
$$
\mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \mathbb {E} _ {f \sim \Pi (\cdot | \boldsymbol {x}, \boldsymbol {y})} \| f - f _ {0} \| _ {L ^ {2} (p _ {0})} ^ {2} \leq C n ^ {- \frac {2 \min  (\beta , \nu)}{2 \nu + d}} \tag {6}
$$
and, moreover, the posterior mean satisfies
$$
\mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \| m _ {\Pi (\cdot | \boldsymbol {x}, \boldsymbol {y})} - f _ {0} \| _ {L ^ {2} (p _ {0})} ^ {2} \leq C n ^ {- \frac {2 \min  (\beta , \nu)}{2 \nu + d}}. \tag {7}
$$
Note that  $m_{\Pi(\cdot|\mathbf{x}, \mathbf{y})}$  is the Bayes estimator [52] of  $f$  associated to the weighted  $L^2$  loss and that the second inequality above is a direct consequence of the first. Therefore the posterior contraction rate implies the same convergence rate for  $m_{\Pi(\cdot|\mathbf{x}, \mathbf{y})}$ . The best rate is attained when  $\beta = \nu$ : that is, when true smoothness and prior smoothness match—which is known to be minimax optimal in the problem of estimating  $f_0$ : see Tsybakov [52]. In this paper, we extend this result to the manifold setting.
2.2 Related Work and Current State of Affairs
The formalization of posterior contraction rates of Bayesian procedures dates back to the work of Schwartz [46] and Le Cam [29], but has been extensively developed since the seminal paper of Ghosal et al. [19] for various sampling and prior models, see for instance [20, 42] for reviews. This includes, in particular, work on Gaussian process priors [54, 56, 57, 43, 49]. Most of the results in the literature, however, assume Euclidean data: as a consequence, contraction properties of Bayesian models under manifold assumptions are still poorly understood, with exception of some recent developments in both density estimation [7, 8, 60] and regression [63, 60].
The results closest to ours are those of Yang and Dunson [63] and Castillo et al. [12]. In the former, the authors use an extrinsic length-scale-mixture of squared exponential Gaussian processes to achieve optimal contraction rates with respect to the weighted  $L^2$  norm, using a completely different proof technique compared to us, and their results are restricted to  $f_0$  having Hölder smoothness of order less than or equal to two. On the other hand Castillo et al. [12] consider, as an intrinsic process on the manifold, a hierarchical Gaussian process based on its heat kernel and provide posterior contraction rates. For the Matérn class, Li et al. [30] presents results which characterize the asymptotic behavior of kernel hyperparameters: our work complements these results by studying contraction of the Gaussian process itself toward the unknown ground-truth function. One can also study analogous discrete problems: Dunson et al. [17] and Sanz-Alonso and Yang [45] present posterior contraction rates for a specific graph Gaussian process model in a semi-supervised setting. In the next section, we present our results on Matérn processes, defined either by restriction of an ambient process or by an intrinsic construction, and discuss their implications.
3 Posterior Contraction Rates on Compact Riemannian Manifolds
We now study posterior contraction rates for Matérn Gaussian processes on manifolds, which are arguably the most-widely-used Gaussian process priors in both the Euclidean and Riemannian settings. We begin by more precisely describing our geometric setting before stating our key results and discussing their implications. From now on, we write  $X = \mathcal{M}$ , to emphasize that the covariate space is a manifold.
Assumption 2. Assume that  $\mathcal{M} \subset \mathbb{R}^D$  is a smooth, compact submanifold (without boundary) of dimension  $d < D$  equipped with the standard Riemannian volume measure  $\mu$ .
We denote  $|\mathcal{M}| = \int_{\mathcal{M}} d\mu(x)$  for volume of  $\mathcal{M}$ . With this geometric setting defined, we will need to describe regularity assumptions in terms of functional spaces on the manifold  $\mathcal{M}$ . We work with Hölder spaces  $\mathcal{CH}^{\gamma}(\mathcal{M})$ , defined using charts via the usual Euclidean Hölder spaces, the Sobolev spaces  $H^{s}(\mathcal{M})$ , and Besov spaces  $B_{\infty, \infty}^{s}(\mathcal{M})$  which are one of the ways of generalizing
(c) Intrinsic
Figure 2: Different Matérn kernels  $k(\bullet, x)$  on different manifolds.
the Euclidean Hölder spaces of smooth functions to manifolds. We follow Coulhon et al. [14] and Castillo et al. [12], and define these spaces using the Laplace-Beltrami operator on  $\mathcal{M}$  in Appendix A.
Recall that the data-generating process is given by (3), with  $f_{0}$  as the true regression function and  $p_0$  as the distribution of the covariates.
Assumption 3. Assume that  $p_0$  is absolutely continuous with respect to  $\mu$ , and that its density, denoted by  $p_0$ , satisfies  $c \leq p_0 \leq C$  for  $0 < c, C < \infty$ . Assume the regression function  $f_0$  satisfies  $f_0 \in H^\beta(\mathcal{M}) \cap B_{\infty, \infty}^\beta(\mathcal{M})$  for some  $\beta > d/2$ , and that  $\sigma_\varepsilon^2 > 0$  is fixed and known.
This setting can be extended to handle unknown variance  $\sigma_{\varepsilon}$  by putting a prior on  $\sigma_{\varepsilon}$ , following the strategy of Salomond [44] and Naulet and Barat [36]. Since we are focused primarily on the impact of the manifold, we do not pursue this here. With the setting fully defined, we proceed to develop posterior contraction results for different types of Matérn Gaussian process priors: intrinsic, intrinsic truncated and extrinsic.
3.1 Intrinsic Matérn Gaussian Processes
We now introduce the first geometric Gaussian process prior under study—the Riemannian Matérn kernel of Whittle [62], Lindgren et al. [33], and Borovitskiy et al. [11]. This process was originally defined using stochastic partial differential equations: here, we present it by its Karhunen-Loève expansion, to facilitate comparisons with its truncated analogs presented in Section 3.2.
Definition 4 (Intrinsic Matérn prior). Let  $\nu > 0$ , and let  $(\lambda_j, f_j)_{j \geq 0}$  be the eigenvalues and orthonormal eigenfunctions of the Laplace-Beltrami operator on  $\mathcal{M}$ , in increasing order. Define the intrinsic Riemannian MATÉRN GAUSSIAN PROCESS through its Karhunen-Loève expansion to be
$$
f (\cdot) = \frac {\sigma_ {f} ^ {2}}{C _ {\nu , \kappa}} \sum_ {j = 1} ^ {\infty} \left(\frac {2 \nu}{\kappa^ {2}} + \lambda_ {j}\right) ^ {- \frac {\nu + d / 2}{2}} \xi_ {j} f _ {j} (\cdot) \quad \xi_ {j} \sim \mathrm {N} (0, 1) \tag {8}
$$
where  $\nu, \kappa, \sigma_f^2$  are positive parameters and  $C_{\nu,\kappa}$  is the normalization constant, chosen such that  $\frac{1}{|\mathcal{M}|} \int_{M} \operatorname{Var}(f(x)) \, \mathrm{d}\mu(x) = \sigma_f^2$ , where  $\operatorname{Var}$  denotes the variance.
The covariance kernels of these processes are visualized in Figure 2. With this prior, and the setting defined in Section 3, we are ready to present our first result: this model attains the desired optimal posterior contraction rate as soon as the regularity of the ground-truth function matches the regularity of the Gaussian process, as described by the parameter  $\nu$ .
Theorem 5. Let  $f$  be a Riemannian Matérn Gaussian process prior of Definition 4 with smoothness parameter  $\nu > d/2$  and let  $f_0$  satisfy Assumption 3. Then there is a  $C > 0$  such that
$$
\mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \mathbb {E} _ {f \sim \Pi (\cdot | \boldsymbol {x}, \boldsymbol {y})} \| f - f _ {0} \| _ {L ^ {2} \left(p _ {0}\right)} ^ {2} \leq C n ^ {- \frac {2 \min  (\beta , \nu)}{2 \nu + d}}. \tag {9}
$$
All proofs are given in Appendix B. Our proof follows the general approach of van der Vaart and van Zanten [56], by first proving a contraction rate with respect to the distance  $n^{-1/2} \| f(\pmb{x}) - f_0(\pmb{x}) \|_{\mathbb{R}^n}$  at input locations  $\pmb{x}$ , and then extending the result to the true  $L^2$ -distance by applying a suitable concentration inequality. The first part is obtained by studying the concentration function, which is known to be the key quantity to control in order to derive contraction rates of Gaussian process priors—see Ghosal and van der Vaart [20] and van der Vaart and van Zanten [57] for an overview.
Given our regularity assumptions on  $f_0$ , the most difficult part lies in controlling the small-ball probabilities  $\Pi\big[\|f\|_{\mathcal{C}(\mathcal{M})} < \varepsilon\big]$ : we handle this by using results relating this quantity with the entropy of an RKHS unit ball with respect to the uniform norm. Since our process' RKHS is related to the Sobolev space  $H^{\nu + d/2}(\mathcal{M})$  which admits a description in terms of charts, we apply results on the entropy of Sobolev balls in the Euclidean space to conclude the first part. Finally, to extend the rate to the true  $L^2(p_0)$  norm, following van der Vaart and van Zanten [56], we prove a Hölder-type property for manifold Matérn processes, and apply Bernstein's inequality. Together, this gives the claim.
This result is good news for the intrinsic Matérn model: it tells us that asymptotically it incorporates the data as efficiently as possible at least in terms of posterior contraction rates, given that its regularity matches the regularity of  $f_0$ . An inspection of the proof shows that the constant  $C > 0$  can be seen to depend on  $d, \sigma_f^2, \nu, \kappa, \beta, p_0, \sigma_\varepsilon^2, \|f_0\|_{H^\beta(\mathcal{M})}, \|f_0\|_{B_{\infty \infty}^\beta(\mathcal{M})}$ , and  $\|f_0\|_{\mathcal{CH}^\beta(\mathcal{M})}$ . Theorem 5 extends to the case where the norm is raised to any power  $q > 1$  rather than the second power, with the right-hand side raised to the same power: see Appendix B for details. We now consider variants of this prior that can be implemented in practice.
3.2 Truncated Matérn Gaussian Processes
The Riemannian Matérn prior's covariance kernel cannot in general be computed exactly, since Definition 4 involves an infinite sum. Arguably the simplest way to implement these processes numerically is to truncate the respective infinite series in the Karhunen-Loève expansion by taking the first  $J$  terms, which is also optimal in an  $L^2(\mathcal{M})$ -sense.
Note that the truncated prior is a randomly-weighted finite sum of Laplace-Beltrami eigenfunctions, which have different smoothness properties compared to the original prior: the truncated prior takes its values in  $\mathcal{C}^{\infty}(\mathcal{M})$  since the eigenfunctions of  $\mathcal{M}$  are smooth—see for instance De Vito et al. [16]. Nevertheless, if the truncation level is allowed to grow as the sample size increases, then the regularity of the process degenerates and one gets a function with essentially-finite regularity in the limit.