Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: The authors propose a class of vector Gaussian processes (GP) on a class of Riemannian manifolds.  An important feature of the proposed GP is its ability to model vector fields on the Riemannian manifold. This is a broad application in physical sciences. One of the main novelties of the work is the development of gauge-equivariant kernels operators. Variation inference algorithms are extended for posterior inference.  Numerical examples demonstrate the practical performance of the proposed GP.
main_review: Novel contributions are as follows:
* The authors propose a class of vector Gaussian processes (GP) on a class of Riemannian manifolds.  
* The authors successfully apply techniques from differential geometry for flexible GP modeling.
* The authors develop gauge-equivariant kernels operators.
* Outline the approach for posterior inference using variational Bayes.
limitations_and_societal_impact: Comparisons with deep GP are missing.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposed a way to define vector-valued Gaussian processes on Riemannian manifolds. A computational framework is provided to facilitate statistical inference using these models. The main contribution is an explicit methodology for characterizing vector-valued Gaussian processes through extrinsic kernels satisfying a gauge constraint, thus bypassing abstract mechanisms often employed to define stochastic processes on Riemannian manifolds. Two examples are provided to illustrate the power of the proposed models.
main_review: Originality: Extending previous work on Gaussian processes on Riemannian manifolds to vector fields is an interesting and valuable contribution. It is not clear why previous work can not be more straightforwardly generalized to this setting, and a lot of literature on stochastic processes on Riemannian manifolds are not mentioned.

Quality: This submission is technically sound and complete.

Clarity: This submission is reasonably clearly written, well organized, and easy to follow.

Significance: The results presented in this paper have great practical value. It can be imagined that this work can potentially lead to future work along the same lines but larger societal impacts, as the authors suggested in the conclusion section.
limitations_and_societal_impact: While the authors emphasized the value of the theoretical constructions in this paper, I found the computational ingredients and illustrated examples much more valuable since the theoretical construction does not appear novel given the extensive existing literature on stochastic differential geometry. More specifically:

- It is not immediately clear to me why the authors indicated in the introduction that defining vector-valued Gaussian process on Riemannian manifold is harder than defining scalar-valued Gaussian processes on manifolds or manifold valued Gaussian processes. Manifold-valued diffusion processes are well-understood for at least several decades [1, Sec. III]; the extrinsic characterization and projection approach can be traced back to at least [2, Ch. 4]; many recent work in statistical learning have addressed similar and more general themes covered in this submission, e.g. [3], [4], [5], [6]. The equivariant viewpoint played a central role in [4], and the projection of extrinsic processes to tangent spaces is known to [1], [2] and more generally the stochastic geometry community. The gauge-equivariance of the kernel and the variational approximations can be derived from these results without too much extra technicality. It is true that this submission phrased many of these ideas in a different context, but the novelty is limited due to the vast existing literature covering this topic, and relation of proposed work with these previous contributions are not fully elaborated.

[1] Émery, M. (2012). Stochastic calculus in manifolds. Springer Science & Business Media.

[2] Stroock, D. W. (2000). An introduction to the analysis of paths on a Riemannian manifold (No. 74). American Mathematical Soc.

[3] Mallasto, A., & Feragen, A. (2018). Wrapped Gaussian process regression on Riemannian manifolds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 5580-5588).

[4] Lin, L., Mu, N., Cheung, P., & Dunson, D. (2019). Extrinsic Gaussian processes for regression and classification on manifolds. Bayesian Analysis, 14(3), 887-906.

[5] Mallasto, A., Hauberg, S., & Feragen, A. (2019, April). Probabilistic Riemannian submanifold learning with wrapped Gaussian process latent variable models. In The 22nd International Conference on Artificial Intelligence and Statistics (pp. 2368-2377). PMLR.

[6] Guhaniyogi, R., & Dunson, D. B. (2016). Compressed gaussian process for manifold regression. The Journal of Machine Learning Research, 17(1), 2472-2497.

- I believe the proposed computational methodology will have significant impact, but more examples will need to be provided to illustrate the computational advantage. Approximation errors incurred in the variational approximation and their dependence on the manifold geometry may need further elaboration as well.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper suggests a construction of Covariance functions of multi-output Gaussian processes for vector fields on manifolds. The construction is independent of coordinate choices, but does depend on a Nash embedding of the manifold. In addition, the paper describes a clear mathematical framework of coordinate independent constructions of Gaussian processes. The approach is extended to variational approximations of such Gaussian processes, where the independence of the approach w.r.t. coordinate choices is again taken into consideration.
main_review: **edit:**
The answer to criticism (1) is very convincing. The answer to criticism (2) is as of yet rather vague. I have increased the score form 6 to 7.
---

The paper is very clear in its mathematical content (with one exception, see below), including the proofs in the appendix. In particular, both measure theory and differential geometry are correctly represented and cleverly combined. The results are correct, meaningfull and important, to construct regression models of vector fields on manifolds, e.g. the surface of the earth. Experiments are conducted, clearly described, and code is given.
In general, I think the paper deserves to be published.

However, I have two major criticisms, which I would like to see fixed. These issues decrease my rating. I am willig to increase it again after a suitable author response.
 (1) Are your constructions independent of the choise of the Nash embedding? (Probably not.) How do you justify working with these choices in a paper that is so much about naturality (=independence of choices)? Furthermore: how do you get these embeddings in practice? 
 (2) Your examples (RxS^1 and S^2) are rather trivial, compared to what is possible with your approach. Can you give geometrically more complicated examples, e.g. a Klein bottle or a torus?
 
Some minor comments, which should be easily fixed:
 - The cited literature is coming from a background of differential geometry in neural networks. There are already paper using constructions of Gaussian processes on manifolds (in particular spheres), often including vector fields. You might want to compare your approach at least to some of those, e.g. [CL, DDH, EP, EFP, L1,L2] (Note that your approach is probably supperior to those) 
 - I guess the typo spoken about on page 18 (appendix) will be fixed in the final version.
 - Line 26: the hairy ball theorem only holds for even-dimensional spheres. One could mention that the existence of smooth non-vanishing vectors fields is a rare occurence for generic manifolds.
 - Line 75: regarding "spaces with the structure of smooth manifolds", these space usually have more than one smooth structure, they only have a unique natural smooth structure constructed from the smooth structure of X.
 - You might want to give clear references to the proofs of your theorems into the appendix.
 - Line 258: "The vast majority of..." please name at least 3 that do and at least one that does not.
 - Section 4.2: Wind fields should be approximately divergence free. Could the divergence free fields on a sphere from [LH1] help?
 - Appendix: Definition 11: what are algebraic sections? I would call sections algebraic, if they are given by polynomial (=algebraic) maps. You probably mean something different. This should be cleared up.

References

  [CL] Creasey, Lang, Fast generation of isotropic Gaussian random fields on the sphere, Monte Carlo Methods and Applications 2018;

  [DDH] Dutordoir, Durrande, Hensman, Sparse Gaussian Processes with Spherical Harmonic Features, ICML 2020;

  [EP] Emery, Porcu, Simulating isotropic vector-valued Gaussian random fields on the sphere through finite harmonics approximations, Stochastic Environmental Research and Risk Assessment 2019;

  [EFP] Emery, Furrer, Porcu, A turning bands method for simulating isotropic Gaussian random fields on the sphere, Statistics & Probability Letters 2019;

  [L1] Lange-Hegermann, Algorithmic Linearly Constrained Gaussian Processes, Neurips 2018; 

  [L2] Lange-Hegermann, Linearly Constrained Gaussian Processes with Boundary Conditions, AISTATS 2021.
limitations_and_societal_impact: None.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper proposes a natural extension of vector-valued GPs to data residing on Riemannian manifolds. The construction is straight-forward (replace matrix products with bilinear forms), which I consider a good thing. This also imply that it is relatively easy to apply the developed theory in practice. Toy demonstrations are provided to show that this is indeed the case. The paper is exceptionally well-written.
main_review: The task considered in this paper is that of fitting GPs to vector fields over Riemannian manifolds. The first task is then to define the notion of GPs in this setting. The paper follows the obvious path forward and phrase this in the tangent bundle (note: 'obvious' is a positive word; I merely mean 'non-silly'), and formally verify that this is a sensible approach. Next the paper provides a scheme for extending any scalar kernel on the manifold to a vector-valued kernel. This is the main technical contribution.

The paper is a joy to read; things are clearly spelled out most of the time, and the authors have chosen a nice balance between rigor and intuition when explaining the concepts.

== Comments and questions (in arbitrary order) ==
*) In lines 117-120, one could consider mentioning that you just view matrix multiplication as the application of a bilinear operator. For readers familiar with such, a large bulk of the paper follows directly once this view is in place.

*) I found it somewhat annoying to have to go to the appendix to see a definition of a 'coframe' in Proposition 5 -- is that really needed? It would be nice of the paper was more self-contained.

*) In general, I enjoyed that the paper does not try to make things sound more complicated than they really are. This paper could easily have been written using as scary-as-possible math, and the authors did a nice job of avoiding this. One place where I found this didn't quite work out was lines 180-183 where the text reads (to me) as if it is a big challenge to apply variational approximations within the proposed setup. I'd flip this around and say that part of the elegance of the proposed approach is that this is mostly trivial.

*) I appreciate that footnote 3 makes it explicit that Nash's Theorem is about the metric tensor; all too many machine learning papers have been written assuming that Nash's theorem is concerned with the distance function.

*) I wonder if the construction of tangential projection (lines 200-211) is related to the 'projected normal distribution' (see e.g. Mardia & Jupp's book on directional statistics) ? Can I essentially think of the proposed machinery as the extension of this distribution to the infinite dimensional (i.e. stochastic process) case (and for general Riemannian manifolds) ?

*) It seems to me that the approach should also be applicable to deep kernels. These are often phrased as k'(x, y) = k(f(x), f(y)), where k is a kernel of interest, and f is a neural network. My intuition is that if f has a full-rank Jacobian everywhere, then one could still apply the machinery proposed in this paper. If so, that would be kinda neat.

*) It's a shame that the authors does not seem to plan on releasing code. I understand that for a paper of this sort, the developed code is probably messy, and not suitable as a library. But just having a Jupyter notebook that illustrated how one might do this on a sphere using GPFlow, would make it much easier for others to pick up the work. I recommend that the authors consider this.

*) It might also be good to mention other works on GPs on manifolds, e.g. "Wrapped Gaussian process regression on Riemannian manifolds", Mallasto & Aasa Feragen, CVPR 2018, comes to mind.

== After rebuttal ==
The authors give very satisfying answers to all my questions. I'm happy, and still recommend acceptance.
limitations_and_societal_impact: The paper is foundational, so the societal impact is left as future work.

As to limitations, then I would have appreciated a deeper discussion of the reliance on an embedding of the manifold. This potentially prevents a great deal of applications, yet it seems that what is actually needed are operators for tangential projections, which perhaps can be found even when embeddings are not easily available. Either way, I feel that this is a significant limitation and it would be good to more explicitly acknowledge this.
review_title: null

Meta-review: The majority of the reviewers argued rather strongly for accepting this paper. One reviewer saw the paper as more borderline, but the concerns were addressed in the rebuttal. This paper is to most parts well written and, as one of the reviewers say, a joy to read. However, the reviewers have all made a number of remarks for improving the paper, and I recommend that you take care in incorporating them in the camera-ready.
Decision: Accept (Poster)