Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a Bayesian optimisation framework for optimisation problems over Riemannian manifolds commonly appearing in robotics problems. This work extends the Geometry-aware Bayesian optimisation (GABO) framework [17] by including techniques to derive Matérn kernels for manifolds where a squared-exponential kernel is defined. The paper includes expressions for the sphere, SO(d), spaces of positive-definite matrices and hyperbolic spaces, which are common in robotics planning problems. Experimental results demonstrate the performance of the proposed methodology on synthetic functions and simulated robotics problems.
strengths_and_weaknesses: The paper is mostly well written and clear in its presentation structure. I believe the expressions provided for Matérn on common robotics manifolds can be very useful to the community, especially to those working with kernel methods and planning over manifolds. However, the paper comes with a few issues which make it hard to assess the significance of its contribution.
summary_of_recommendation: My main concern with this paper is that it seems incremental when compared to the original GABO framework [17] and the experimental results do not seem to show much improvement.
issues: 1. The contrast with Jaquier et al. [17]'s original GABO is not clear at first. Since the proposed framework is mostly similar, I believe a more clear statement of contributions and differences with respect to the original framework would help assessing the novelty in this paper. Since Jaquier et al. only used the squared exponential kernel, I wonder if the main difference lies in the new kernels formulation?

2. Performance differences w.r.t. standard Euclidean kernels in robotics experiments (Figure 3) do not seem very evident, except for the planning problem, which only had one baseline and is apparently in a very simplified setting.

3. Experimental comparisons against the original GABO are restricted to the orientation sampler (sphere manifold), though that framework had also been tested at least on the positive-definite matrices manifold [17].
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper is concerned with endowing GP models the ability to consider the geometry of the search space (e.g. the domain of the function being optimised lies on a manifold). Developing these geometry aware GP models is non-trivial, thus the authors leverage existing tools for formulating valid kernels that respect various problem geometries. The main contribution is extending the work of a previous paper [17] by applying these geometry-aware GP models to additional robotics problems and formulating their appropriate kernels.
strengths_and_weaknesses: - The robotic experimental results are not as convincing, for example some of the euclidean kernels seem to perform very similarly.
- It would be useful to show computation time to assess whether these algorithms could be run in real-time.
summary_of_recommendation: This paper provides an incremental contribution to [17] by extending its kernel formulations to additional manifold types relevant to various robotics problems. In particular, these formulations are tailored to their Riemannian structure and overcome the issue of having to choose a small enough length scale; for example, required when using the more general geodesic SE kernel. Although the paper does not present any new fundamental ideas, the derivations are interesting and non-trivial and could be useful for practitioners working on similar problems.
issues: - The robotic experimental results are not as convincing, for example some of the euclidean kernels seem to perform very similarly.
- It would be useful to show computation time to assess whether these algorithms could be run in real-time.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper focuses on the problem of designing geometry-based kernels for Gaussian Process optimization. The kernels are derived for several well-known manifolds that appears naturally in several robotic applications. The geometry-based kernels are compared against the Euclidean-based kernels to validate the performance of the proposed kernels.
strengths_and_weaknesses: Strengths:
The paper is well written paper, in general, with solid ideas and contributions. In particular, the proposed kernels are sound and easy to compute with detailed analysis provided in the appendix. The numerical results provide a clear evidence of the gained performance when geometry-based kernels are used.

summary_of_recommendation: Solid paper with clear application to robotic applications.
issues: N/A.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper studies the application of Bayesian optimization with Riemannian Matern kernels in various robotics applications where the search domain for BO lies in a Riemannian manifold instead of the commonly assumed Euclidean space.
strengths_and_weaknesses: Strengths:
- The paper covers a large variety of robotics applications with various Riemannian manifolds, and hence can potentially contribute useful ideas to a large number of robotics tasks.
- The paper is clearly motivated and in general well written.

Weaknesses:
- My most major concern is regarding the significance of the contribution of the current paper. In particular, this paper feels more like a case study on the use of existing Riemannian Matern kernels in different robotics applications. So I wonder what are the novel contributions of this paper, especially when compared with the most related works of [17] and [7]?
- line 170: if I understand correctly, the expressions of all the kernels introduced in the remainder of this section are based on the approach of heat kernels (equation 3)? So the method of equation (2) is not used at all? Please clarify.
- Experiments, lines 269-271: which method in the figures correspond to this baseline? "Approx. Riem. SE"?
- Figure 2: for the SO(3) and $\mathcal{H}^3$ spaces, why only two methods are compared here?
- Figure 2 and 3: It seems that the "Riem. SE" method works the best in general across all figure. Does this suggest maybe there is no need for Matern kernels in these applications?
- Figure 2: I think the figure in its current form is very hard to read. Since you compare with different baselines in different sub-figures, I would suggest having a separate legend for every sub-figure

Minor issues:
- line 71: symmetric-positive-definite: I think it's usually semi-definite instead of definite
summary_of_recommendation: This paper provides a comprehensive study of the use of Bayesian optimization with Riemannian kernels in a variety of robotics tasks. My most major concern is regarding the technical novelty of the paper. Please clarify this, as well as my other concerns listed in the section "Strengths and Weaknesses" above.
issues: As mentioned above, my most major concern is regrading the technical contribution. In particular, it's unclear to me which part of the paper can be considered as novel contributions from this paper compared with previous works (especially the most related works of [17] and [7]).
review_title: null

Meta-review: The paper describes a Bayesian optimisation methodology on manifolds with Matern Kernels. The main contribution is the derivation of BO for Riemannian Matern kernels with interesting applications in robotics. 

As pointed out by reviewers 6VZw, 3LxP and f3EN, there are two issues with the paper the authors should try to address in the rebuttal: 

1. The novelty in the paper is not clear given [17] and the derivation of the GPs on Riemannian Manifolds with Matern kernels in [7]. What are the specifics of the contribution that goes beyond applying ideas in [7] into [17]?

2. The experiments do not demonstrate a significant advantage of Matern kernels, despite their generality, compared to squared exponential on manifolds. Given that the SE kernel is a special case within the Matern family, would you be able to provide examples of problems where the cost (implicit) function requires a less smooth kernel to be modelled properly?  

========== Post discussion update ============

I thank you the authors for addressing the reviewers concerns and in improving the paper with a better explanation of the novelty in this work. The reviewers now agree that this paper can be accepted. This is also my recommendation.  

Decision: Accept (Poster)