Vector-valued Gaussian Processes on Riemannian Manifolds via Gauge Independent Projected Kernels
Authors: Michael John Hutchinson
Alexander Terenin
Viacheslav Borovitskiy
So Takao
Yee Whye Teh
Marc Peter Deisenroth
Venue: NeurIPS 2021 Poster (2021)
Keywords: Gaussian processes
Riemannian manifolds
gauge independence

Abstract: Gaussian processes are machine learning models capable of learning unknown functions in a way that represents uncertainty, thereby facilitating construction of optimal decision-making systems. Motivated by a desire to deploy Gaussian processes in novel areas of science, a rapidly-growing line of research has focused on constructively extending these models to handle non-Euclidean domains, including Riemannian manifolds, such as spheres and tori. We propose techniques that generalize this class to model vector fields on Riemannian manifolds, which are important in a number of application areas in the physical sciences. To do so, we present a general recipe for constructing gauge independent kernels, which induce Gaussian vector fields, i.e. vector-valued Gaussian processes coherent with
geometry, from scalar-valued Riemannian kernels. We extend standard Gaussian process training methods, such as variational inference, to this setting. This enables vector-valued Gaussian processes on Riemannian manifolds to be trained using standard methods and makes them accessible to machine learning practitioners.

Vector-valued Gaussian Processes on Riemannian Manifolds via Gauge Independant Projected Kernels
Michael Hutchinson" So Takao*3 +5
Alexander Terenin +2.3 Yee Whye Teh!
Viacheslav Borovitskiy"* Marc Peter Deisenroth
University of Oxford St. Petersburg State University
University of Cambridge 3 Imperial College London Centre for Artificial Intelligence, University College London
Abstract
Gaussian processes machine learning models capable of learning unknown functions a way that represents uncertainty, thereby facilitating construction optimal decision-making systems. Motivated by a desire to deploy Gaussian processes in novel areas of science, rapidly-ggooing line of research has focused constructively extending these models handle non-Euclidean domains, in- cluding Riemannian manifolds, such as spheres and tori. We propose techniques that generalize this class to model vector fields on Riemannian manifolds, which are important in number of application areas in the physical sciences. To do so, we present a general recipe for constructing gauge independent kernels, which induce Gaussian vector fields, i.e. vector-valued Gaussian processes coherent with geometry, from scalar-valued Riemannian kernels. We extend standard Gaussian process training methods, such as variational inference, to this setting. This enables vector-valued Gaussian processes on Riemannian manifolds to be trained using standard methods and makes them accessible to machine learning practitioners.
Introduction
Gaussian processes are an effective model class for learning unknown functions. They are particularly attractive for within data-efficient decision systems, including Bayesian optimization 32, 39).
model-based reinforcement learning [34] 7], and active learning [21). In these settings, Gaussian processes can represent and propagate uncertainty, information in order to drive data efficiency. A key aspect of prior information is the geometry of the domain on which the Gaussian process is defined, which often encodes key properties, such as symmetry. Following the growing deployment of Gaussian processes, a number of recent works have focused on how to define Gaussian processes on non-Fuclidean domains in ways that reflect their geometric structure [.
well encode inductive biases prior
In many applications, such as climate science, quantities of interest vector-valued. For example, global wind velocity modeling must take into account both speed and direction, and is represented by vector field. On geometric domains, the mathematical properties of vector fields can differ noticeably from their Euclidean counterparts: for instance, one can prove that every smooth vector field on sphere must vanish in at least one point (26). Behavior such as this simultaneously highlights the need to represent geometry correctly when modeling vector-vall valued data, and presents number of non-tri non-trivial technical challenges in constructing models that are mathematically sound.
"Equal contribution. Code: wttp:///wiil.llllllliiioeeeeeeae general implementation, see wttp://////oooooooooooo/oooooooe
35th Conference on Neural Information Processing Systems (NeurIPS 2021).

---

In particular, even the classical definition of a vector-valued Gaussian process that is, a random function with multivariate Gaussian marginals at any finite set of shint--aready fails to be a fully satisfactory notion when considering smooth vector fields on a sphere. This is because tangent vectors at distinct points live within different tangent spaces, and it is not clear how to construct a rrossccoaaiiacce between them that does not depend on a completely arbitrary choice of basis vectors within each space. Constructions that are independent of this choice basis are called gauge independent, and recent work [45] 5 15 in geometric machine learning has focused on satisfying this key property for convolutional neural networks that deal with non-Euclidean data.
Our contributions include the following. We (a) present a differential- -geometric formalism for coordinate-free way, suitable for Gaussian process practitioners with minimal familiarity with differential geometry. (b) present universal and fully constructive technique for defining prior Gaussian vector fields on Riemannian manifolds, which we term the projected kernel construction, and (c) discuss how to adapt key components in the computational Gaussian process toolkit, such as inducing point methods, to the vector field setring.
defining Gaussian vector fields on manifolds
The structure of the paper is as follows. In Section 2 we define vector-valued Gaussian processes on smooth manifolds. We start by reviewing the multi-output Gaussian process set-up, which is typically used in machine learning. We then detail a differential-geomeed valued Gaussian processes on smooth manifolds. In Section we provide a concrete construction for these Gaussian processes on Riemannian manifolds and discuss how they can be trained using variational sparse approximations. Section I showcases Gaussian vector fields on two tasks, namely weather imputation from satellite observations and learning the dynamics of a mechanical system.
formalism for defining vector-
2 Vector-valued Gaussian Processes on Smooth Manifolds
A vector-valued Gaussian process (GP) is a random function f X + Bd such that, for any finite set of points æ E X", the random variable f(x E Ruxd is jointly Gaussian. Every such GP is characterized by its mean function μ : X + Rd and matrix-valued covariance kernel : x X + gaxd which is a positive-definite function in the matrix sense. These functions satisfy E(f())) = (xx) and (f)( f(z)) = k(x, x²) for any x, x' X. Here, dependence between function values is encoded in the kernel's 's variability along its input domain, and correlations between different dimensions of the vector vector-valued output are encoded in the matrix that the kernel outputs.
Consider a function f with y = f()) + e, where e N(O, o and training data (æ, y). Placing a GP prior f GP(O, k) on the unknown function results in a GP posterior, whose mean and covariance are given by
(1) Here, (.) denotes an arbitrary set of test locations, K k(ar,, a:) is the kernel matrix, and
E(f v) = (( (Kaz + --o))'aa Cov(f 3) = K(.) - K (() (Kea + + Ka())
K Kej = k(xz..) is the cross-covariance matrix between function values evaluated at the training and test inputs. The GP posterior can also be written as
(f)))) = f(-) + K() (Kæz + +021) - E).
-~ 020
(2)
where f(-) is the prior GP, and equality holds in distribution (47 48 These expressions form the foundation upon which Gaussian prrraan--roees---baee based methods in machine learning are built.
Recent works have studied techniques for working with the expressions ( and (2) when the input domain X a Riemannian manifold, focusing both on defining general classes of kernels (2). and on efficient computational techniques [47 [88). In this setting, namely for f X + R, defining kernels already presents technical challenges: the scemingly-obviout first choice one might consider, namely the geodesic squared exponential kernel, is ill-defined in general [13]. We build on these recent developments model vector fields on manifolds using GPs. We do not consider manifold-valuee generalizations of Gaussian processes, for instance f: E + X: various constructions in this setting are instead studied by Stroock (40), Émery [11). Mallasto and Feragen [29]. and Mallasto et al. [30]
To begin, we review what a vector field on manifold actually is.
2.1 Vector Fields on Manifolds
Let X be a d-dimensional smooth manifold with T,X denoting its tangent space at x. Let TX= (Z,,)) I E X, T,X} be its tangent bundle, and let 1 X ((z,0) x X, E XE
2

---

Basis coefficients
Frame
Vector field
X
X
Illustration 2 Here we illustrate two possible bases (also called frames), consisting of green and orange basis vectors (center), that can be chosen locally on the manifold $ The vector field on g (right) can be produced by taking two scalar fields (left) in each respective color, and combining them with the basis vectors (center) to form the vector field.
Figure 1:
be its cotangent bundle endow both spaces with the structure of smooth manifolds. Define the pro- jection map projx TX + X by projx (z, v) = z. A vector field on X is a map that assigns each point in X to a tangent vector attached to that point. More formally, vector field is crosy-section, or simply a section, of the tangent bundle, which is map f: - TX, such that projx f(x) for all o A vector field is called smooth if this map is smooth.
To represent a vector field on a manifold numerically, one must choose basis in each tangent space, which serves as a coordinate system for vectors in the tangent space. On many manifolds it is impossible choose these basis vectors in way that they vary smoothly in space P This can be handled by working with local coordinates, or with bases that are non-smooth. Any chosen set of basis vectors is arbitrary, so objects constructed using them should not depend on this choice. Constructions that satisfy this property are called gauge independent. This notion is illustrated in Figure and will play a key role in the sequel.
2.2 Gaussian Vector Fields
Upon reflecting the above considerations in the context of GPs, the first issue encounters is that, for a random vector field f X + TXX it is not clear what means for finte-dimensional marginal distributions to be multivariate Gaussian given that ff takes its values in bundle rather than a vector space. The first step towards constructing Gaussian vector fields, therefore, involves adapting the notion of finite- fifte-dimesionall marginal distributions appropriately.
Definition 1. Let X be a smooth manifold. We say that a random vector field f is Gaussian for any finite set of locations ,n,... E X, the random vector (f(æ))..... J. (xxx) Tr, X @... .. Tx X
is Gaussian (either in the sense of duality or in any basis: see Appendix A for details).
This definition is near-identical to the Euclidean case: the only difference is that finite- dineesiaminii marginals are now supported in a direct sum of tangent spaces, instead of gnxd, With this definition, the standard multi-output GP properties, such conditioning, carry over, virtually unmodified. Definition is a natural choice: if we embed our manifold into Euclidean space, the induced GP is a vector-valued GP as defined in the beginning of Section
AA vector field is not the same as map f X - Re: an output value fxx E TX formally consists of both a copy of the input point x. and vector within the tangent space T,-X at this point. This encodes the geometric structure of the underlying manifold. The algebraic requirement projx f(x) tangent vector chosen correctly corresponds to the point at which is attached.
for all æ ensures that the
"If a smooth choice of basis vectors existed, it would define smooth non-vanishing vector field. On the sphere, by the hairy ball theorem, all smooth vector fields vanish in at least point, such bases exist.
3

---

Proposition 2. Let X be a manifold and emb : X + Rs be a smooth embedding. Let f be Gaussian vector field (as defined in Definition [j. and let femb : emb(X) + R2 be its pushforward along the embedding. Then femb is vector-valued Gaussian process in the Euclidean sense.
All proofs in this work can found in Appendix A Having established the notion of a vector- valued Gaussian process smooth manifold, proceed to deduce what mathematical objects play the role of a mean function and kernel, so that it is clear what ingredients are needed to construct and determine such process.
The former is clear: the mean of Gaussian vector field should be an ordinary vector field, and will determine the mean vector at all mnite-dimensionll marginals. The kernel, on the other hand, is less obvious: because distinct tangent vectors live in different tanget spaces, is unclear whether or not Gaussian vector field is characterized by appropriate notion of a matrix-valued kernel, or by something else. Now, we define the right notion for kernel this setting.
Definition 3. We call a symmelric function k T T of points x, x'' E X
R fiberwise bilinear for all pairs
k(Aazz + p32. (wr) = Ak(az: (k:) + pk(33 7ax')
(3)
holds for any ag, 32 E X, Ye T and X, p E R. and positive semi-definite if for any set of covectors Or1..... ,a., E X, we huve Ei=1 Ej=1 k(az,, 2 0. We call a symmetric fiberwise bilinear positive semi semi-definite function a cross-covariance kernel.
This coordinate- free function should be viewed as analogous to ((x, v), (x', o²)) + d² g o' in the Euclidean setting, where v, v' multiply the matrix-valued kernel from both sides. Its coordinate representation, which more closely matches the Euclidean case, will be explored in the sequel. To show that this is indeed the right notion, we prove the following result.
Theorem 4. The system of marginal distributions of a Gaussian vector field on a smooth manifold X is uniquely determined by a mean vector field μ : X + TX and a cross-covariance kernel k: T x X + R. Moreover, this correspondence is one-to-one.
By virtue of defining and characterizing all Gaussian vector fields, Theorem assures us the definition of a kernel introduced is the correct mathematical notion. The constructions presented here are all intrinsic or, in other words, coordinate-free, and do not involve the use of bases. To understand how to perform numerical calculations with these kernels we proceed to study their coordinate representations with respect to a specific choice of basis.
2.3 Matrix-valued Kernels
we defined what a Gaussian vector field on a manifold is. However, by nature of the manifold setting. the resulting objects are more abstract than usual and do not describe how can be represented numerically. We now develop a point of view suitable for this task.
In Section 2.2
To this end, we introduce frame F on X, also known as gauge in physical literature, which is collection of (not necessarily smooth) vector fields e1.-.., edd on such that at each point X, the set of vectors e1 (z), ... ea(z) forms basis of T.X. The frame allows us express vector field f on X as simply vector-valued function f (f ..... fa) : X - Rd. such that ( (x) for all x E X. The corresponding coframe F* is defined as a collection el e....., ed of covector fields (one-forms) on X such that (<))x,,,,z = d:₃ for all x E X, where 8.j is the Kronecker delta. In the following proposition, we show that if is Gaussian vector field on X (in the sense of Definition T), then the corresponding vector representation f expressed in a given frame is vector- vector-valued GP in the standard sense.
Proposition 5. Let f be Gaussian vector field defined on X with cross-covariance kernel k: T T + IR. Given a frame F (e1..... ed) on X, define f X + Rd as above. Then is vector-valued GP in the usual sense with kernel Kp X x X Raxd given by
e
KF(z,z')=
(4)
(((
((((

---

where (e"), with raised indices, is the coframe corresponding to (et). Conversely, given a vector- valued GP f = (f',...,,f)) Ei (-)) defines Gaussian vector on X.
+ Thd and a frame = (e1,..., ea) on X, f() =
This result shows precisely how numerical representations of a Gaussian vector field depends on the choice of frame. While this representation is not invariant under this choice, it is equivariant, meaning that transformation in the frame results in an appropriate transformation the kernel. To make this notion precise, we introduce a matrix subgroup G GL(d, R), called the guuge group, that acts on F by a standard matrix-vector multiplication. Given two frames F, F² on X, an abstract vector f= E TxX X has two vector representations fa. f'r in the respective frames. We say that F' is obtained from F by gauge transformation with respect matrix field A X G Rdxd if
f/ = Ac)f
(5)
holds for all a E X, and we write F' = A.. Note that A need not be smooth in x. We see that the gauge transformation is therefore just linear change of basis of the frame F at each point for which one can identify vectors in x X as elements RR. The corresponding gauge dependant matrix-valued kernels must also respect this transformation rule, a statement of gauge independence.
Let F be frame on X and KF X x X + Rddd be the corresponding matrix representation (4) of a cross-covariance kernel k T X T RR. This satisfies the equivariance condition
Corollary 6.
Kar(2,x)) Kr) Az")
where A + S Iddd Proposition 4 faiise this way.
d is a gauge transformation. All cross-covariance kernels in the sense of
Hence, one way define Gaussian vector field manifold is to find gauge independent kernel. In summary, we have described Gaussian vector fields in a coordinate- language, and deduced enough properties confirm the objects defined truly deserve to be called GPs. In doing so, we have both introduced the necessary formalism to the GP community, and obtained recipe for defining kernels, through a simple condition atop standard matrix-valued kernels. To proceed towards practical machine learning methods, we therefore study techniques for constructing such kernels explicitly.
differential -geommtrrcc
3 Model Construction and Bayesian Learning for Riemannian Manifolds
In Section 2 we introduced a notion of a Gaussian vector field. We now study how to use vector fields for machine learning purposes. This entails two primary issues: (a) how to construct practically useful kernels, and (b) once kernel is constructed, how to train Gaussian processes.
To construct Gaussian vector field prior, the preceding theory tells that need to specify mean vector field and a cross-covariance kernel. From the definition, it is not at all obvious how to specify natural kernel, and experience with the scalar-valued case-whert the innocuous-looking geodesic squared exponential kernel is generally not positive semi-definite on most Riemannian manifolds [13 suggests that the problem is delicate, i.e., simply guessing the kernel's form is unlikely to succeed. Our goal, therefore, is to introduce a general construction for building wide classes of kernels from simple building blocks.
The same issues are present if we consider variational approximations to posterior GPs, such as the inducing point framework of Titsias (33: these formulated using matrix-vector vector expressions involving kernel matrices, and is important for the approximate posterior covariance to be gauge independent in order to lead to valid approximate process. We proceed to address these issues.
3.1 Projected Kernels
Here, we introduce general technique for defining cross- covariance kernels k T X
and for working with such functions numerically. Section[]giver us promising strategy to construct a suitable kernel-namely, it suffices to find a gauge independent matrix-valued kernel. At first glance, it is not obvious how to construct such kernel in the manifold setting. On many manifolds, such as the sphere, owing to the hairy ball theorem, every frame must be discontinuous: therefore, constructing continuous kernel in such choice of frame appears difficult.
5

---

Scalar processes
Embedded process
Projected process
Illustration of the construction process of the projected process, The manifold S is embedded into RR Three identical scalar GPs (left) are placed on the manifold. These three scalar GPs are combined to construct a vector-valued GP in the ambient Euclidean space (center). This GP then projected onto the tangent space of S as subspace of the tangent space R (right).
Figure 2:
To both get around these obstacles, and aid numerical implementation, we propose to isometrically embed the manifold into Euclidean space. Doing so greatly simplifies these issues by virtue of making it possible to represent the manifold using single global coordinate system. On the other hand, the main trade-off from this choice is that by its extrinsic nature, the construction can make theoretical analysis more difficult. To proceed, we need two ingredients.
An isometric embedding emb + Rdd of the manifold.
2. A vector-valued Gaussian process f' : X + 1dd in the standard sense
simple choice which reflects the geometry of the manifold is to take f' to be d' independent scalar-valued GPs on X.
By standard results in differential geometry, any smooth map Ø X + X' between two manifolds induces a corresponding linear map on the tangent spaces d : TrX - X', which can loosely be thought of as mapping to its first-order Taylor expansion at Thus, an embedding emb: X + Rdd, induces map d.embb T,X- Temb(=) Now fixing frame F on X, each tangent space T&X can be identified with 1dd so without loss of generality, the map dmemb can be expressed simply as a deetedent matrix P7 R xd Taking the transpose. we obtain Pa E dg which we call the projection matrix. The desired Gaussian vector field on X, with respect F, is then constructed f(x) f'xr). This procedure is illustrated in Figure 2 there we see that to get a vector field on an IR -embedded sphere, we may take a vector-valued function on it and project its values make vectors tangential to this sphere, thus obtaining valid vector field. Since the projection operator preserves smoothness and since we can take a smooth vector-vale valued GP to begin with, it is clear that this approach may be used build smooth vector fields.
We prove that (a) the resulting expression is, indeed, a kernel, and that (b) no expressivity is lost via the construction because all cross-covariance kernels arise this way.
Proposition 7. Let (X, g) be Riemannian manifold, emb X - 12dd be isometric embedding and F be frame on X. We denote by P()) X Ddxd' the associated projection matrix under F, and let f' X + IRd be any vector-valued Gaussian process with matrix-valued kernel K. X X - Rdd xd' Then, the vector-valued function f = PS' defines a Gaussian vector field f X using the construction in Proposition whose kernel under the frame F has matrix representation
Kr(o,2)) = P2
An embedding emb: X R² is called isometric it preserves the metric tensor. By Nash's Theorem such embeddings exist any d-dimensional manifold, an embedded dimension d' <
26
We emphasize again that f is not Gaussian vector field because it is not a random section. In particular, note that d' > for most embeddings.

---

Random samples from Gaussian processes with gauge independent projected kernels, on the torus and Klein bottle, respectively. The latter is non-orientable manifold: the ability to handle such cases highlights the generality of projected kernels.
Figure 3:
Moreover, all cross-covariance kernels ks T X + arise this way. We call a kernel defined this way projected kernel.
To construct these kernels we require scalar-valued kernels manifolds to use basic building block. These are studied in the general Riemannian setting by Lindgren al. (28) and Borovitskiy et al. (2): relying on these kernels is the only reason we require the Riemannian structure. It is also possible to obtain such kernels using embeddings, following Lin et al. [27]. Similar techniques to those we consider
used by Freeden and Schreiner (14) to construct vector-valued zonal kernels on the sphere: in contrast, we work with arbitrary manifolds. The projection kernel idea a very general way to build kernels for vector fields by combining scalar kernels, but effective scalar kernels, naturally, rely on Riemannian structure Figure 3 shows random samples from Gaussian processes constructed with the described kernels.
The projected kernel construction both makes easy to define cross-covariance kernels on general manifolds, and describes straightforward way to implement them numerically by representing the embedded manifold in coordinates and calculating the resulting matrix veetorr expressions. The constructed kernel depends on the embedding, but can be transformed appropriately if switching to a different embedding. Embeddings, turn, are available for most manifolds of practical interest, and are obtained automatically for manifolds approximated numerically as meshes. Everything described is constructive and fully compatible with the modern machine learning toolkit, and most operations for constructing and/or sampling from specialized
based
priors 44 22 23 including on spaces such as the sphere where specific analytic tools are available With these kernels in hand, we thus proceed to study training methods.
3.2
Gauge Independent Variational Approximations
We now discuss variational inference for training GPs in the Riemannian vector field setting. Approx- imations, such as the inducing-point framework by Titsias (43) and Hensman et [17). approximate the posterior GP with another GP, termed the variational approximation. 'he latter is typically constructed by specifying multivariate Gaussian at a set of test locations with parameterized mean and kernel matrix. For example, Opper and Archambeau [3] consider N(m, S), where
m ())s (Kzz + Σ)ー 1
s= K ()) ()xx((22 + Σ) (Kzz + E) ")-KK()) (8)
The variational parameters include set inducing locations z, a mean vector μ4, and a block- diagonal cross-covariance matrix Σ. Training proceeds by optimizing these parameters to minimize the BetbbackkLeiblee divergence of the variational distribution from the true posterior, typically using mini-batcl stochastic gradient descent.
In the last decade, wide and diverse range of inducing point approximations suited for many different
settings have been proposed 43
31
25
44 49) The vast majority of them employ coordinate-
'The structure of a smooth manifold not rigid enough to define natural kernels. For instance, the smooth structure of the sphere is indistinguishable from the smooth structure of an ellipsoid or even of the dragon manifold from Borovitskiy al. (2). but their Riemannian structures differ considerably.
7

---

True dynamics
Forecast
Uncertainty
2T
Momenium
Position
Euclidean
π
33
Observed data
2T
Momentum
Riemannian
Position
-3
0
2m
0
2m
200
400
600
800
1000
Position
Position
Timestep
Figure 4: Upper left: pendulum with friction state space. Lower left: Two rollouts used to train GP. Upper middle: State space plot of rollouts from standard Euclidean GP condtioned the training data. Upper right: temporal plot rollouts from a standard Euclidean vector GP. Solid line the true rollout, dashed line and shade is the mean and = 1 std of the GP rollouts. Lower middle and right: same for geometric manifold vector field kernel on S R.
dependent matrix-vector expressions. This raises the question, which of these constructions can be adapted define valid variational approximations in the vector field setting?
To proceed, one can choose a frame and formulate given variational approximation using matrices defined with respect this frame. To ensure well-definednees,, one must ensure that all these matrices, such as the kernel matrix and the variational parameter in (8), are gauge independent. These considerations can be simplified adopting the pathwise view of GPs, and examining the random variables directly. For example, the variational approximation of Opper and Archambeau [3]] shown previously in (8) can be reinterpreted pathwise the GP
(f v-) f(-) + K())=(KK + -E f(=)== =)
E N(O, Σ)
where we view the matrices K(-jz, Kez, E as linear operators between direct sums of tangent spaces: K()= : Tay X + ... Te.. X + T.,xX and Kzz, Σ T2, ... TT.. X Tr,X X ... T:., X. By virtue of being defined at the level of vector fields using components that are all intrinsically valid, the posterior covariance of the resulting variational approximation automatically gauge independent. Hence, checking gauge independence is then equivalent to deducing the domains and ranges of these operators from their coordinate representations, and checking if they are compatible. This applies to any variational family that can be constructed in the given manner.
The vast majority of inducing point constructions readily to the Riemannian vector field setting by simply representing the necessary matrices in chosen frame. In particular, the classical approach of Titsias (33 is gauge independent.
be interpreted in this manner and thus extend
4 Illustrated Examples
Here, we showcase number of examples that illustrate potential use cases of the models developed.
4.1 Dynamical Systems Modeling
Here, we show how Gaussian vector fields can be used to learn the equations of motion of physical system-an important task in imitation learning, model-based reinforcement learning, and robotics. GPs are attractive model class in this area owing to their ability to represent and propagate

---

uncertainty, which enables them to separate what is known about an environment from what is not, thereby driving data-efficient exploration.
For prototype physical system, we consider an ideal pendulum, whose configuration space is the circle 81, representing the angle of the pendulum, with zero being at the bottom of the loop, and whose pasition- momentum state-space is the cylinder S² x F. We consider conservative dynamics with additional friction applied at the pivot. Since this system is non-conservative, we cannot just learn the Hamiltonian of the system, but must learn the vector field over the state space that defines the dynamics of the system. The true dynamics of the system are given by the differential equations
p²
dq a7
dp dt
07 b 3q m p,
H
2ml2 + mgl(1 cos(4))
(10)
dt Op
where H is the Hamiltonian of the system defining the conservative part of the dynamics, q and p are the position and momentum of the pendulum, m is the mass, 1 is the length, y is the gravitational field strength and is friction parameter. Experimental details can be found in Appendix B
To learn this model, we initialise the system at two start points, and evolve the system using leapfrog integration. From these observations of position, we backward Euler integrate the momentum of the system, and from these position-monet momentum trajectories we estimate observations of the dynamics field. Using these observations, we condition sparse GP. The result is estimate of the system dynamics with suitable uncertainty estimates. In order to compute rollouts of these dynamics, we use pathwise sampling of this sparse GP
[47 48) for speed together with leapfrog integration.
Results can be seen in Figure While the Euclidean GP performs reasonably well at the start of the rollouts, once the trajectory crosses the discontinuity caused by looping the angle back around to zero, the system starts to make incoherent predictions: this is due to the discontinuity arising from wrap-around condition of the angle. The manifold vector-valued GP does not have this issue as the learned and sampled dynamics fields are continuous throughout the state-space.
4.2 Weather Modeling
In this experiment, we show how vector-valued GPs on manifolds can be used in the context of meteorology, where geometric information often plays an important role in accurately modeling global weather fields [46 38 19I. Data assimilation in numerical weather forecasting refers to the practice of using observed data to update predictions of the atmosphere closer to the truth. Uncertainty plays a critical role here: it is not usually possible to observe the weather at all locations on the globe simultaneously, and taking into account observation uncertainty crucial in numerical weather forecasting during the data assimilation step (24] (3). In this section, we explore Gaussian processes as a tool for carrying out global interpolation of wind fields, while simultaneously performing uncertainty quantification, mirroring optimal interpolation techniques in data assimilation (20).
We consider a simplified setting, where the goal is to interpolate the wind velocity observed by the Acolus satellite (37), which uses LiDAR sensors to measure wind velocity directly. To mimic this setting, we use an hour of the Acolus satellite track during the period 2019/01/01 09:00-10:00 for the input locations and the wind velocity data (10m above ground) from the ERA5 atmospheric reanalysis data [18) interpolated these locations, to simulate measurements taken from the Aeolus satellite. We subtract the weekly historical average wind velocity from the observations, before training the GP models, where the historical mean is computed from the hourly wind data (10 m above ground) from the WeatherBench dataset (35), available from 1979-2018. Further details can be found in Appendix We compare the results of a Matérn-3/2 manifold vector-valued GP regression model fitted on the wind anomaly observations along the Acolus trajectory, with the results from a Euclidean Matérn-3/2 multi-output GP trained on the same data, except projected onto a latitude-longitude map.
Results are shown in Figure 5 where the benefits of using a manifold valuer GP become clear. When the satellite crosses the left/right boundary in the lat/lon projection, the outputs from the Euclidean vector- vector-valued GP give rise to a spurious discontinuity in the uncertainty along the solid pink line. In addition. predictions become less certain in the Euclidean case as the satellite approaches the poles, which is simply an artifact of the distortion caused by projecting the spherical data onto the plane. By construction, the manifold vector-valued GP is able to avoid both of these issues, resulting in more realistic prediction with much more uniform uncertainty along the satellite trajectory from pole to pole. In addition, the predictions from the manifold GP are more certain overall, due to the useful structural bias embedded in the kernel.

---

Figure 5: Top row: Euclidean GP trained on wind measurements along the chosen Acolus satellite trajectory, viewed as deviation from normal with respect to the historical average vector field. White arrows are the satellite measurements, black arrows and ellipsoids are the posterior mean and cross- covariance of the vector field, colors indicate the posterior standard deviation norm, and the solid red line indicates the latitudinal boundary when the sphere projected onto the plane using the lat/lon projection. Bottom row: Same as above except using a manifold kernel on S².
Conclusion
In this paper, propose techniques that generalize Gaussian processes to model vector fields Riemannian manifolds. This is done by first providing a well-defined notion of such processes manifolds and then introducing the underlying geometry. By virtue of satisfying the key condition of gauge independence, our construction is coordinate-free and thus meaningful manifolds. In addition to this, we extend standard Gaussian process training methods, such as variational inference, to this setting, and verify that such methods are also compatible with gauge independence. This theoretical work gives practitioners additional tools for stochastic modeling of vector fields on manifolds. As such, its societal impact will be mainly determined by the applications that belong the domain of future work. We demonstrate our techniques on a series of examples in modeling dynamical systems and weather science, and show that incorporating geometric structural bias into probabilistic modeling beneficial in these settings to obtain coherent predictions and uncertainties.
explicit method construct them in way that respects
6
Acknowledgements and Funding Disclosure
Michael supported by the EPSRC Centre for Doctoral Training in Modern Statistics and Statistical Machine Learning (EP/S023151/I), Section and Appendix A of this paper were solely financially supported by the RSF grant N 1-00047 contributed to by all authors in equal proportion with the remaining work. Other sections were supported by the remaining funding. We also acknowledge support from Huawei Research. We thank Peter Dueben for useful suggestions in Section 4.2 including, in particular, making us of the Acolus satellite. We are grateful to Nick Sharp for introducing us to the process used for making the dine-dimensiooal figures appearing in this work.
http:://scc..ueee/pojjoccccc 10000////)1110 but was
10


Reviews:
[{"review_id": "oTwRI7Uvxx", "reviewer": "Reviewer_UAbr", "summary": "The authors propose a class of vector Gaussian processes (GP) on a class of Riemannian manifolds.  An important feature of the proposed GP is its ability to model vector fields on the Riemannian manifold. This is a broad application in physical sciences. One of the main novelties of the work is the development of gauge-equivariant kernels operators. Variation inference algorithms are extended for posterior inference.  Numerical examples demonstrate the practical performance of the proposed GP.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Novel contributions are as follows:\n* The authors propose a class of vector Gaussian processes (GP) on a class of Riemannian manifolds.  \n* The authors successfully apply techniques from differential geometry for flexible GP modeling.\n* The authors develop gauge-equivariant kernels operators.\n* Outline the approach for posterior inference using variational Bayes.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Comparisons with deep GP are missing.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "gIiXqa2HP6R", "reviewer": "Reviewer_aB3f", "summary": "This paper proposed a way to define vector-valued Gaussian processes on Riemannian manifolds. A computational framework is provided to facilitate statistical inference using these models. The main contribution is an explicit methodology for characterizing vector-valued Gaussian processes through extrinsic kernels satisfying a gauge constraint, thus bypassing abstract mechanisms often employed to define stochastic processes on Riemannian manifolds. Two examples are provided to illustrate the power of the proposed models.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: Extending previous work on Gaussian processes on Riemannian manifolds to vector fields is an interesting and valuable contribution. It is not clear why previous work can not be more straightforwardly generalized to this setting, and a lot of literature on stochastic processes on Riemannian manifolds are not mentioned.\n\nQuality: This submission is technically sound and complete.\n\nClarity: This submission is reasonably clearly written, well organized, and easy to follow.\n\nSignificance: The results presented in this paper have great practical value. It can be imagined that this work can potentially lead to future work along the same lines but larger societal impacts, as the authors suggested in the conclusion section.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "While the authors emphasized the value of the theoretical constructions in this paper, I found the computational ingredients and illustrated examples much more valuable since the theoretical construction does not appear novel given the extensive existing literature on stochastic differential geometry. More specifically:\n\n- It is not immediately clear to me why the authors indicated in the introduction that defining vector-valued Gaussian process on Riemannian manifold is harder than defining scalar-valued Gaussian processes on manifolds or manifold valued Gaussian processes. Manifold-valued diffusion processes are well-understood for at least several decades [1, Sec. III]; the extrinsic characterization and projection approach can be traced back to at least [2, Ch. 4]; many recent work in statistical learning have addressed similar and more general themes covered in this submission, e.g. [3], [4], [5], [6]. The equivariant viewpoint played a central role in [4], and the projection of extrinsic processes to tangent spaces is known to [1], [2] and more generally the stochastic geometry community. The gauge-equivariance of the kernel and the variational approximations can be derived from these results without too much extra technicality. It is true that this submission phrased many of these ideas in a different context, but the novelty is limited due to the vast existing literature covering this topic, and relation of proposed work with these previous contributions are not fully elaborated.\n\n[1] Émery, M. (2012). Stochastic calculus in manifolds. Springer Science & Business Media.\n\n[2] Stroock, D. W. (2000). An introduction to the analysis of paths on a Riemannian manifold (No. 74). American Mathematical Soc.\n\n[3] Mallasto, A., & Feragen, A. (2018). Wrapped Gaussian process regression on Riemannian manifolds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 5580-5588).\n\n[4] Lin, L., Mu, N., Cheung, P., & Dunson, D. (2019). Extrinsic Gaussian processes for regression and classification on manifolds. Bayesian Analysis, 14(3), 887-906.\n\n[5] Mallasto, A., Hauberg, S., & Feragen, A. (2019, April). Probabilistic Riemannian submanifold learning with wrapped Gaussian process latent variable models. In The 22nd International Conference on Artificial Intelligence and Statistics (pp. 2368-2377). PMLR.\n\n[6] Guhaniyogi, R., & Dunson, D. B. (2016). Compressed gaussian process for manifold regression. The Journal of Machine Learning Research, 17(1), 2472-2497.\n\n- I believe the proposed computational methodology will have significant impact, but more examples will need to be provided to illustrate the computational advantage. Approximation errors incurred in the variational approximation and their dependence on the manifold geometry may need further elaboration as well.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8 hours"}, "extra_text": {}}, {"review_id": "7AzmhDI41iE", "reviewer": "Reviewer_RZVP", "summary": "The paper suggests a construction of Covariance functions of multi-output Gaussian processes for vector fields on manifolds. The construction is independent of coordinate choices, but does depend on a Nash embedding of the manifold. In addition, the paper describes a clear mathematical framework of coordinate independent constructions of Gaussian processes. The approach is extended to variational approximations of such Gaussian processes, where the independence of the approach w.r.t. coordinate choices is again taken into consideration.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**edit:**\nThe answer to criticism (1) is very convincing. The answer to criticism (2) is as of yet rather vague. I have increased the score form 6 to 7.\n---\n\nThe paper is very clear in its mathematical content (with one exception, see below), including the proofs in the appendix. In particular, both measure theory and differential geometry are correctly represented and cleverly combined. The results are correct, meaningfull and important, to construct regression models of vector fields on manifolds, e.g. the surface of the earth. Experiments are conducted, clearly described, and code is given.\nIn general, I think the paper deserves to be published.\n\nHowever, I have two major criticisms, which I would like to see fixed. These issues decrease my rating. I am willig to increase it again after a suitable author response.\n (1) Are your constructions independent of the choise of the Nash embedding? (Probably not.) How do you justify working with these choices in a paper that is so much about naturality (=independence of choices)? Furthermore: how do you get these embeddings in practice? \n (2) Your examples (RxS^1 and S^2) are rather trivial, compared to what is possible with your approach. Can you give geometrically more complicated examples, e.g. a Klein bottle or a torus?\n \nSome minor comments, which should be easily fixed:\n - The cited literature is coming from a background of differential geometry in neural networks. There are already paper using constructions of Gaussian processes on manifolds (in particular spheres), often including vector fields. You might want to compare your approach at least to some of those, e.g. [CL, DDH, EP, EFP, L1,L2] (Note that your approach is probably supperior to those) \n - I guess the typo spoken about on page 18 (appendix) will be fixed in the final version.\n - Line 26: the hairy ball theorem only holds for even-dimensional spheres. One could mention that the existence of smooth non-vanishing vectors fields is a rare occurence for generic manifolds.\n - Line 75: regarding \"spaces with the structure of smooth manifolds\", these space usually have more than one smooth structure, they only have a unique natural smooth structure constructed from the smooth structure of X.\n - You might want to give clear references to the proofs of your theorems into the appendix.\n - Line 258: \"The vast majority of...\" please name at least 3 that do and at least one that does not.\n - Section 4.2: Wind fields should be approximately divergence free. Could the divergence free fields on a sphere from [LH1] help?\n - Appendix: Definition 11: what are algebraic sections? I would call sections algebraic, if they are given by polynomial (=algebraic) maps. You probably mean something different. This should be cleared up.\n\nReferences\n\n  [CL] Creasey, Lang, Fast generation of isotropic Gaussian random fields on the sphere, Monte Carlo Methods and Applications 2018;\n\n  [DDH] Dutordoir, Durrande, Hensman, Sparse Gaussian Processes with Spherical Harmonic Features, ICML 2020;\n\n  [EP] Emery, Porcu, Simulating isotropic vector-valued Gaussian random fields on the sphere through finite harmonics approximations, Stochastic Environmental Research and Risk Assessment 2019;\n\n  [EFP] Emery, Furrer, Porcu, A turning bands method for simulating isotropic Gaussian random fields on the sphere, Statistics & Probability Letters 2019;\n\n  [L1] Lange-Hegermann, Algorithmic Linearly Constrained Gaussian Processes, Neurips 2018; \n\n  [L2] Lange-Hegermann, Linearly Constrained Gaussian Processes with Boundary Conditions, AISTATS 2021.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "2sUWj6CL3CF", "reviewer": "Reviewer_mmXd", "summary": "The paper proposes a natural extension of vector-valued GPs to data residing on Riemannian manifolds. The construction is straight-forward (replace matrix products with bilinear forms), which I consider a good thing. This also imply that it is relatively easy to apply the developed theory in practice. Toy demonstrations are provided to show that this is indeed the case. The paper is exceptionally well-written.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The task considered in this paper is that of fitting GPs to vector fields over Riemannian manifolds. The first task is then to define the notion of GPs in this setting. The paper follows the obvious path forward and phrase this in the tangent bundle (note: 'obvious' is a positive word; I merely mean 'non-silly'), and formally verify that this is a sensible approach. Next the paper provides a scheme for extending any scalar kernel on the manifold to a vector-valued kernel. This is the main technical contribution.\n\nThe paper is a joy to read; things are clearly spelled out most of the time, and the authors have chosen a nice balance between rigor and intuition when explaining the concepts.\n\n== Comments and questions (in arbitrary order) ==\n*) In lines 117-120, one could consider mentioning that you just view matrix multiplication as the application of a bilinear operator. For readers familiar with such, a large bulk of the paper follows directly once this view is in place.\n\n*) I found it somewhat annoying to have to go to the appendix to see a definition of a 'coframe' in Proposition 5 -- is that really needed? It would be nice of the paper was more self-contained.\n\n*) In general, I enjoyed that the paper does not try to make things sound more complicated than they really are. This paper could easily have been written using as scary-as-possible math, and the authors did a nice job of avoiding this. One place where I found this didn't quite work out was lines 180-183 where the text reads (to me) as if it is a big challenge to apply variational approximations within the proposed setup. I'd flip this around and say that part of the elegance of the proposed approach is that this is mostly trivial.\n\n*) I appreciate that footnote 3 makes it explicit that Nash's Theorem is about the metric tensor; all too many machine learning papers have been written assuming that Nash's theorem is concerned with the distance function.\n\n*) I wonder if the construction of tangential projection (lines 200-211) is related to the 'projected normal distribution' (see e.g. Mardia & Jupp's book on directional statistics) ? Can I essentially think of the proposed machinery as the extension of this distribution to the infinite dimensional (i.e. stochastic process) case (and for general Riemannian manifolds) ?\n\n*) It seems to me that the approach should also be applicable to deep kernels. These are often phrased as k'(x, y) = k(f(x), f(y)), where k is a kernel of interest, and f is a neural network. My intuition is that if f has a full-rank Jacobian everywhere, then one could still apply the machinery proposed in this paper. If so, that would be kinda neat.\n\n*) It's a shame that the authors does not seem to plan on releasing code. I understand that for a paper of this sort, the developed code is probably messy, and not suitable as a library. But just having a Jupyter notebook that illustrated how one might do this on a sphere using GPFlow, would make it much easier for others to pick up the work. I recommend that the authors consider this.\n\n*) It might also be good to mention other works on GPs on manifolds, e.g. \"Wrapped Gaussian process regression on Riemannian manifolds\", Mallasto & Aasa Feragen, CVPR 2018, comes to mind.\n\n== After rebuttal ==\nThe authors give very satisfying answers to all my questions. I'm happy, and still recommend acceptance.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper is foundational, so the societal impact is left as future work.\n\nAs to limitations, then I would have appreciated a deeper discussion of the reliance on an embedding of the manifold. This potentially prevents a great deal of applications, yet it seems that what is actually needed are operators for tangential projections, which perhaps can be found even when embeddings are not easily available. Either way, I feel that this is a significant limitation and it would be good to more explicitly acknowledge this.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The majority of the reviewers argued rather strongly for accepting this paper. One reviewer saw the paper as more borderline, but the concerns were addressed in the rebuttal. This paper is to most parts well written and, as one of the reviewers say, a joy to read. However, the reviewers have all made a number of remarks for improving the paper, and I recommend that you take care in incorporating them in the camera-ready.
Decision: Accept (Poster)