Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors propose the use of a so-called vector-valued distance to extract geometric information from the space of symmetric positive definite matrices (SPD), which is a manifold that is ubiquitous in a wide range of applications, and present the accompanying gyrovector calculus which constructs analogs of vector space operations in this curved space. These operations are then applied to a number of tasks including knowledge graph completion, item recommendation, and question answering, with favorable results compared to the corresponding Euclidean and hyperbolic spaces. 
main_review: SPD manifolds have been used in a very wide range of applications in computer vision, medical imaging, and machine learning and the study of their geometries and research into their applications is highly active. The work under review fits naturally within this context and should be of interest to a large community of researchers. The paper is very well-written with a clear and thoughtful structure that includes high quality and enlightening illustrations as well as a clear mathematical exposition. The background literature is also quite thorough and comprehensive with the exception of a few very important omissions which I have noted below. The main contribution of the paper is in the examples considered and the experiments that demonstrate the versatility of the approach and the superior expressivity of SPD manifolds equipped with geometries based on the vector-valued distances compared to Euclidean or hyperbolic baselines. As a result, I am pleased to support acceptance of the paper subject to revision based on the comments and suggestions provided below, some of which I consider to be necessary.

Major Comments:

1. On first reading I understood that the authors had introduced the gyro calculus based on the vector-valued distance as a novel mathematical contribution inspired by the hyperbolic analog. After searching the literature, it became clear that the same gyro vector calculus presented here has already been introduced and discussed in the pure mathematics literature. This needs to be made more clear in the text and relevant papers need to be cited and given due credit. Examples include references [1-3] below. Others may be appropriate as well.

[1] T Abe, O Hatori, Generalized gyrovector spaces and a Mazur-Ulam theorem, published in Publ. Math. Debrecen, 87 (2015), 393--413.

[2] O Hatori, Examples and Applications of Generalized Gyrovector Spaces, Results in Mathematics, 71 (2017), 295-317.

[3] S Kim, Gyrovector Spaces on the Open Convex Cone of Positive Definite Matrices, Mathematics Interdisciplinary Research, (2016).

2. Similar to the above comment, the particular vector-valued distance (equation (1)) has been noted in several works as being a fundamental geometric object for a pair of SPD matrices, which very naturally generates a family of Finsler structures that include the Fisher-Rao (affine-invariant) Riemannian metric and Thompson’s part metric on the positive definite cone as special cases. Relevant papers include the standard reference [4] as well as more recent work such as [5], which studies Thompson geometry in this framework.

[4] R Bhatia, On the exponential metric increasing property, Linear Algebra and its Applications, Volume 375, (2003).

[5] C Mostajeran, C Grussler, R Sepulchre, Geometric Matrix Midranges, SIAM Journal on Matrix Analysis and Applications, (2020).

3. Could the authors elaborate on the issues they faced in using Riemannian optimization, which led them to use tangent space optimization instead? Is an explanation available for these observations?

4. Standard references for Riemannian optimization on manifolds include [6] and [7], which are worth citing on line 201.

[6] P.-A Absil, R Mahony, R Sepulchre, Optimization Algorithms on Matrix Manifolds (2008).

[7] N Boumal, B Mishra, P.-A Absil, R Sepulchre, Manopt, a Matlab Toolbox for Optimization on Manifolds, Journal of Machine Learning Research, (2014).

5. The observed superiority of SPD models compared to hyperbolic models in the experiments is interesting and deserves a deeper discussion in my view. Are particular aspects of the geometry relevant for each of the specific examples/tasks considered, or is the geometry useful in the same way in each case?

6. Perhaps relevant to point 5 above, the authors note on lines 75-77 the contents of SPDn including the number of Euclidean subspaces, hyperbolic subspaces, and hyperbolic planes. Please include a citation for this statement.

7. The authors briefly note the computational limitations of their approach towards the end of the paper. I think this should be expanded and discussed in more detail. Particularly, the distance and gyro calculus operations represent quite a significant computational cost in large dimensions due to eigenvalue computations and matrix operations, which may be missed by a quick glance from a casual reader who may get the wrong idea from the word ‘vector’ in the title!

Minor comments:

1. On line 53, the authors state that the Log-Euclidean geometry (Euclidean-based tools in tangent space) ‘distorts the manifold structure in regions far from the tangent space affecting the performance.’ While I understand what the authors intend to communicate here, it is not the ‘manifold geometry’ that is distorted, since the Log-Euclidean geometry is perfectly valid as a metric structure (Riemannian indeed) on this manifold. It just happens not to be the ‘right geometry’ in many cases, so it is the metric structure that is distorted for many practical applications and not the manifold nature of the space.

2. Remove ‘of’ on line 215.  
limitations_and_societal_impact: Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the authors studied representation learning on SPD manifold by proposing a vector-valued distance function that helps to visualize the embeddings and also generalizing the arithmetic operations to SPD manifold via the Gyrocalculus. They showed good representation power in knowledge graph completion, item recommendation, and question answering.
main_review: Overall, the paper is well-written with nice visualizations. With the increasing attention on geometric deep learning models, the proposed arithmetic operations on SPD matrices can be a good contribution. 

However, the major concern I have is on the experiments. In all three experiments, the authors mainly consider representation learning of different objects on SPD manifold, including node entities in knowledge graph and word embeddings. These objects are primarily modelled on hyperbolic space in the literature with a clear justification, i.e., hierarchical structures that are known to exist in graph entities and between words are captured. However, it is unclear why embedding these objects onto SPD manifold is beneficial. 

In addition, as mentioned by the authors, the computational cost can be high to perform the arithmetic operations compared to the hyperbolic counterparts. It would be helpful to report and compare the runtime in the experiments. Because in some applications, the improvements over the baselines are marginal. If the time required to train the models on SPD manifold take much longer, embedding onto SPD manifold is not well-justified. 

Instead, it would be better if the authors could compare with other SPD representation learning methods. Also, it would be interesting to see how to construct a neural network architecture on SPD manifold using the proposed arithmetic operations.

limitations_and_societal_impact: NA
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a generalized representation on the SPD manifold. The author leverages the vector-valued distance to help obtain unified metrics form on the manifold of SPD and visualize the high-dimensional data. In addition, the author also proposes to equip the SPD space with the arithmetic operations under the gyrovector space structure, which could help implement the calculations directly on the SPD manifold and maintain the essential structural information. Experiments are conducted on three different downstream tasks and demonstrate the effectiveness of the proposed method.
main_review: Strengths:

1. The paper is mostly well presented. 
2. Concise description of the proposed method and detailed theoretical analysis are both provided. 
3. The experiments cover different kinds of tasks and data types.  

Weaknesses:

1. In Sec. 1, the author mentions that "the representational power of SPD is not fully exploited...". It is desirable to provide some details or examples to illustrate why/how the representational power of SPD is under exploited. 

2. In Sec. 3.1, on line 101, it is claimed that "In Euclidean or hyperbolic spaces, the relative position between two points is completely  determined by their distance, ...", it is ideal to explain this more clearly.

3. In the experiment section, the under-performance of the proposed method under the rotation and reflection operations are quite significant and thus some insightful explanation will be much helpful.  

5. Some typos. On line 215, "the task of is to predict..."



limitations_and_societal_impact: No potential negative societal impact of this work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper firstly defines a vector valued metric for SPD matrices and then it addresses the issue of defining operations for a set of SPD matrices (addition, matrix multiplication, scalar multiplication) using gyrocalculus. This operations allow to make computations on a set of SPD matrices, showing how the setting combined with different methods outperform the state of the art in three case studies. 
main_review: Positive aspects: the papar address the urgent and important problem of analyzing non-euclidean objects such as spd matrices in a correct geometrical and algebraic framework. In addition, SPD matrices are broadly used in different applications, so the potential impacts of the work are high. The paper is overall well written.

Negative aspects: If I have correctly understood, I am concerns about the fact that the operations defined (Section 4) relies on a basepoint, thus they are local. This is causing two main problems: (a) how the framework behave if the points are not close to each other, i.e. is the tangent space a good approximation? (b) how the framework behave close to the singularity points of the space? If it is solved by the gyrocalculus approach, it should be better explained.

Major comments:
Section 3. There are several works regarding the comparison between metrices on SPD matrices. Is the vector valued metric intrisic or extrinsic? Please detailed it.

Section 2. How does this work relate with the work on Equivariant neural networks?

Section 4. The space of SPDn have singularities as well as positive curvature. In this area of the space, the approximation of the space with tangent space is not a good choice (e.g. the origin has not one unique tangent space). Could the author clarify how the framework interact with this problem?

Section 4. The gyro-vector space structure is based on the choice of a basepoint. The authors should better state where is the improvement in definition the operation in Section 4 - which relies anyway on the tangent space approach and the base point choice - over the usage of the standard operation on the tangent space (R^(n-1)n/2).


Section 6 As a common behave in the literature, the methodology used are given for granted. It should be useful to understand how the operations well behave with respect to different models used in the section, by enlarging this introduction.

Section 6.1 The example is good but describing a graph in the space of SPDn is a choice which could rise some concerns. In addition, it is not clear to me which is the graph representation. Is the graph represented as a Laplacian? if so then it is an SPD matrix, otherwise the example it is not clear to me.


Minor comments:
Section 3.1 why vector-valued distance is better? it is not clear to all readers and it could sound like an unusual choice. 

Section 3.1 why 'the VVD provides much more information than just the distance'?

Section 6.1 Baselines: the results reported for the other methods are taken from the original papers. Is the setting exactly the same? It is often a risky choice in terms of reproducibility.

Tables in Section 6 are hard to read
limitations_and_societal_impact: Th authors listed some limitation of their work but it should be a bit enlarged in the conclusion.
review_title: null

Meta-review: The paper proposes to use vector-valued distances (and calculus) on the manifold of symmetric positive definite matrices to use in machine learning applications.  As all reviewers agree, the paper contains novel insights and interesting applications (with good analysis). While some theoretical insights are well-known in some previous works, the paper does a great job of putting things cleanly for the NeurIPS community.

Decision: Accept (Spotlight)