Vector-valued Distance and Gyrocalculus on the Space of Symmetric Positive Definite Matrices
Authors: Federico Lopez
Maria Beatrice Pozzetti
Steve J Trettel
Michael Strube
Anna Wienhard
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: symmetric spaces
spd space
spd manifold
symmetric positive definite matrices
spd
riemannian manifold
rotations
reflections
translations
scaling
gyro vector
gyro calculus
gyro groups
gyrocalculus
tangent space optimization
non euclidean optimization
hyperbolic geometry
hyperbolic space
matrix models
non-euclidean geometry
finsler metrics
finsler distance
finsler geometry
vector valued distance
vector valued distance function
riemannian manifold learning
manifold learning
geometric deep learning
graph embeddings
knowledge graph embeddings
item recommendations
question answering

Abstract: We propose the use of the vector-valued distance to compute distances and extract geometric information from the manifold of symmetric positive definite matrices (SPD), and develop gyrovector calculus, constructing analogs of vector space operations in this curved space. We implement these operations and showcase their versatility in the tasks of knowledge graph completion, item recommendation, and question answering. In experiments, the SPD models outperform their equivalents in Euclidean and hyperbolic space. The vector-valued distance allows us to visualize embeddings, showing that the models learn to disentangle representations of positive samples from negative ones.

Vector-valued Distance and Gyrocalculus on the Space of Symmetric Positive Definite Matrices
Federico López* HITS
Beatrice Pozzetti Heidelberg University
Steve Trettel Stanford University
Michael Strube
Anna Wienhard
HITS
Heidelberg University
HITS
Abstract
We propose the use of the vector-valued distance to compute distances and extract geometric information from the manifold of symmetric positive definite matrices (SPD), and develop gyrovector calculus, constructing analogs of vector space op- erations in this curved space. We implement these operations and showcase their versatility in the tasks of knowledge graph completion, item recommendation, and question answering. In experiments, the SPD models outperform their equivalents in Euclidean and hyperbolic space. The vector-valued distance allows us to visual- ize embeddings, showing that the models learn disentangle representations of positive samples from negative ones.
Introduction
Symmetric Positive Definite (SPD) matrices have been applied in many tasks in computer vision
such pedestrian detection 75 77), action [36] 51 60) face recognition [4] 42). object (43 90]
and image set classification 86 visual tracking 88). and medical imaging analysis 5 65 among
others. They have been used to capture statistical notions (Gaussian distributions [68 covariance (78).. while respecting the Riemannian geometry of the underlying SPD manifold, which offers a
convenient trade-off between structural richness and computational tractability (27). Previous work has applied approximation methods that locally flatten the manifold by projecting it to its tangent
space 22 83
by embedding the manifold into higher dimensional Hilbert spaces [35 90)
These methods face problems such as distortion of the geometrical structure of the manifold and other known concerns with regard to nigh-dimensional spaces (29]. To overcome these issues, several distances on SPD manifolds have been proposed, such as the Affine Invariant metric [65], the Stein metric (71). the Hures-Wassersteii metric 13 or the Log- Eugl-dull metric [5] 6). with their respective geometric properties. However, the representational power of SPD not fully exploited in many cases 66 65). At the same time, it is hard to translate operations into their non-Euclidean domain given the lack of closed-form expressions. There has been a growing need to generalize basic operations, such addition, rotation, reflection or scalar multiplication, to their Riemannian geometric counterparts to leverage this structure in the context of Geometric Deep Learning [19
SPD manifolds have rich geometry that contains both Fuclidean and hyperbolic subspaces. Thus, embeddings into SPD manifolds are beneficial, since they can accommodate hierarchical structure in data sets in the hyperbolic subspaces while at the same time represent Euclidean features. This makes them more versatile than using only hyperbolic or Euclidean spaces, and in fact, their different submanifold geometries be used identify and disentangle such substructures in graphs.
"Conresoondence to federico. www.:oo..oppehhhhh..h.omo
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

In this work, we introduce the vector-valued distance function to exploit the full representation power of SPD (S3.J). While Euclidean or hyperbolic space the relative position between two points is completely captured by their distance (and this is the only invariant), in SPD this invariant is a vector, encoding much more information than single scalar. This vector reflects the higher expressivity of SPD due to its richer geometry encompassing Euclidean as well as hyperbolic spaces. We develop algorithms using the vector-valued distance and showcase two main advantages: its versatility to implement universal models, and its use in explaining and visualizing what the model has learned.
Furthermore, we bridge the gap between Euclidean and SPD geometry by developing gyrocalculus in SPD ) which yields closed-form expressions of arithmetic operations, such as addition, scalar multiphccation and matrix scaling. This provides means to translate previously implemented ideas in different metric spaces to their analog notions in SPD. These arithmetic operations are also useful to adapt neural network architectures to SPD manifolds.
We showcase this on knowledge graph completion, item recommendation, and question answering. In the experiments, the proposed SPD models outperform their equivalents in Euclidean and hyperbolic space ( These results reflect the superior expressivity of SPD, and show the versatility of the approach and ease of integration with downstream tasks.
The vector-valued distance allows us to develop a new tool for the analysis of the structural properties of the learned representations. With this tool, we visualize nigh-dimensional SPD embeddings, providing better explainability on what the models learn (66.4). We show that the knowledge graph models capable of disentangling and clustering positive triples from negative ones.
2 Related Work
Symmetric positive definite matrices are not new in the Machine Learning literature. They have been
used in a plethora of applications (5 29 36 40 51 60 65 68
75
78
79
86
87
90), although
not always respecting the intrinsic structure or the positive definiteness constraint 22
31
35 83 90].
The alternative has been to map manifold points onto a tangent space and employ Euclidean- based tools. Unfortunately, this mapping distorts the metric structure in regions far from the origin of the tangent space affecting the performance 4 96
Previous work has proposed alternatives to the basic neural building blocks respecting the geometry alternate convolutional layers based
of the space. For example, transformation layers [29] 33 40,
on SPDs (94) and Riemannian means 233, or appended after the convolution (21. recurrent models [24], projections onto Euclidean spaces [50 557 and batch normalization (20] Our work follows this
line, providing explicit formulas for translating Euclidean arithmetic notions into SPDs.
Our general view, using the vector-valued distance function, allows us to treat Riemannian and Finsler metrics on SPD a unified framework. Finsler metrics have previously been applied in compressed sensing [30). information geometry [69], for clustering categorical distributions [63], and in robotics [67]. With regard to optimization, matrix backpropagation techniques have been explored 2 43 with of them accounting for different Riemannian geometries 20 40). Nonetheless, opt for tangent space optimization (66 by exploiting the explicit formulations the exponential and logarithmic map.
3 The Space SPD,
The space SPD,, is Riemannian manifold of non-positive curvature of n(n + 1)/2 dimen- sions. Points in SPD,, are positive definite real symmetric n matrices, with the identity ma- trix I being a natural basepoint. The tangent space to any point of SPD,, can be identified with the vector space S, of all real symmetric n X n matrices. SPD,, contains vi-dimensional Euclidean subspaces, (n 1)-dimensional hy- perbolic subspaces as well as products of [
Figure 1: SPD2 is foliated by hyperboloids, each of which is copy of the hyperbolic plane.
hyperbolic planes (see Helgason [39] for an in-depth introduction).

---

P
P
P
K
MPM M E SPD,
KPKT K E SO(n)
RPR R E O(n)
Some isometries of SPD, have analogous Euclidean counterparts. Translation (left), rotation (center) and reflection (right).
Figure 2:
In Figure D we visualize the smallest nontrivial example. SPD₂ identifies with the inside of cone in IR, cut out by requiring both eigenvalues of the matrix (# / to positive. It carries the product geometry of the hyperbolic plane times line.
The exponential map, exp: Sn - SPDn, is a homeomor- phism which links the Euclidean geometry of the tangent space S, and the curved geometry of SPD,, Its inverse is the logarithm map, log: SPD, + Sn. This pair of functions give diffeomorphisms that allows one to freely move between 'tangent space coordinates or the original 'manifold coordinates' We apply both maps based E SPD,. The reason for this that while mathematically any two equivalent, and we could obtain different concrete expression for any other
Exponential and logarithmic maps:
points on SPD, choice of basepoint B SPD,, the resulting formulas would be more complicated, and thus / the best choice from computational point of view. We prove this in Appendix
Symmetries: The prototypical symmetries of SPD,, are parameterized by elements of GL(n; R): any invertible matrix M defines the symmetry P + MPM" acting on all points P E SPD,, Thus many geometric transformations SPD,, can completed using standard optimized matrix algorithms as opposed to custom-built procedures. See Appendix C.2 for brief review of these symmetries.
Among these, may find gennraaeeneallizationn of familiar symmetries of Fuclidean geometry. When also M is an element of SPD,, the symmetry P + MPM is generalization of an Euclidean translation, fixing no points of SPD,, When M an orthogonal matrix, the symmetry P + MPM" R is conjugation by M, and thus fixes the basepoint / = M MIM" = MM -1 1. We think of elements fixing the basepoint as being D,' -rotation or SPDD.-reflections, when the matrix M is familiar rotation or reflection (see Figure 2
The Euclidean symmetry of reflecting in point also has natural generalization to SPD,, Euclidean reflection in the origin is given by + -p; and its SPD, -analog, reflection in the basepoint 7, is matrix inversion P + p-1, The general this by an SPD, translation, given by P - QP-1Q.
in point Q ,,D,, is conjugate of
3.1 Vector-valued Distance Function
In Euclidean or hyperbolic spaces, the relative position between two points completely determined by their distance, which is given by scalar. For the space SPD... it is determined by vector.
The VVD vector:
To assign this vector in SPD introduce the vector-valued distance (VVD) function d SPD,, x SPD, + R., For two points P.Q E SPD,, the VVD is defined as:
den (P.Q) 1.)
where X1 (P-IQQ)
2 X, (P-1Q) the eigenvalues of P- sorted in descending order. This vector is an invariant of the relative position of two points up to isometry. This means that in SPD,..
only if the VVD between two points A and B is the vector E IR.. and the VVD between P and Q is also then there exists an isometry mapping A to P and B to Q. Thus, we can recover completely the relative position of two points in SPD,, from this vector. For example, the Riemannian metrie is obtained by using the standard l2 norm on the VVD vector. This is: d"(P,Q Q) = ||dvo (P, Q) See 46 $2.6 and Appendix C.4 for review of VVDs in symmetric spaces.

---

Finsler metries: Any norm on I" that is invariant under per- mutation of the entries induces metric on SPD,, Moreover, SPD, do not only support Rie- mannian metric, but also Finsler metrics, a whole family of dis- tances with the same symme- try group (group of isometries). These metrics are of special im- portance since distance minimiz- ing geodesics are not necessarily unique in Finsler geometry. Two different paths can have the same minimal length. This is particularly valuable when embedding graphs in SPD,, since in graphs there are generally several shortest paths. We obtain the Finsler metrics F, or Fo by taking the respective & or foo norms of the VVD in IR" (see Figure 3 See Planche (66) and Appendix C.6 review of the theory of Finsler metrics, [12] for the study of some Finsler metrics on SPD,, and [22 for applications of Finsler metrics on symmetric spaces in representation learning.
WD
Finsler Norms
F 1.9
F
F
Metric Distances
0.8
L.5
1.7
1.3
2.0
Figure 3: The vector-valued distance allows reconstruct the Riemannian, or any Finsler distance.
The proposed metric learning framework based on the VVD offers several advantages. First, single model can be run with different metries, according to the chosen norm. The VVD contains the full information of the Riemannian distance and of all invariant Finsler distances, hence
Advantages of VVD:
easily recover the Riemannian metric and extend the approach to many other alternatives (in Appendix C.7 we detail how the VVD generalizes other SPD,, metrics). Second, the VVD provides much more information than just the distance, and can be used analyze the learned representation in SPD,. independent of the choice of the metric. Out of the VVD between two points, one can immediately read the regularity of the unique geodesics joining these two points. Geodesics in SPD,, have different regularity. which is related with the number of maximal Euclidean subspaces that contain the geodesic. The Riemannian or Finsler distances cannot distinguish the differences between these geodesics of different regularity, but the VVD function can. Third, the VVD function can be leveraged as tool visualize and analyze the learned dithemminiiin representations (see $6.4).
4 Gyrocalculus
To build an analog of many Euclidean operators in SPD,, we require also a translation of operations internal Euclidean geometry, chief among these being the vector space operations of addition and scalar multiplication. By means of tools introduced in pure mathematics literature we describe a gyro-vector space structure on SPD,, which provides geometrically meaningful extensions of these vector space operations, extending successful applications of this framework in geometric deep learning to hyperbolic space [32] (7), These operations provide a template for translation, where one may attempt to replace +, x in formulas familiar from Euclidean spaces with the analogous operations + G & on SPD,, While straightforward, such translation requires some care, as gyro-addition is neither commutative nor associative. See Appendix D for review of the underlying general theory of gyrogroups and additional guidelines for accurate formula translation.
Addition and Subtraction: Given a fixed choice I of basepoint and two points P.Q E SPD,, we define the gyroaddition of and Q to be the point P & Q E SPD, which the image Q under the isometry which translates / to P along the geodesic connecting them. This directly generalizes via the geometric interpretation of the gyroaddition hyperbolic space exploited by 32 70 Vermeer [84] (see Figure
Fixing P E SPD,, we may compute the value of P Q for arbitrary Q the result of applying the transatranslitionn moving the bascpoint to P, evaluated on Q. We see also the additive inverse of a point with respect this operation must then be given by its geodesic reflection in
P + Q =BQ=PPQVP
"See 11 37 for abstract treatment of gyrocaleulus, and 47 for the specific examples discussed here.

---

P
P
P Q
@ P
M x P
P
Figure 4: Gyro-addition (left), gyro-scalar multiplication (center) and matrix scaling (right).
As this operation encodes a symmetry of SPD,, it is possible recast certain geometric statements purely in the gyrovector formalism. In particular, the vector-valued distance d may be computed as the logarithm of the eigenvalues of e @ Q (see Appendix (C.))
For a fixed basepoint 7, we define the scalar multiplica- tion of point P SPD, by scalar E R he the point which lies at distance ad(J, P) from / in the direction of P, where d(.,.) the metric distance on SPD,- Geometrically, this is a transfer of the vector-space scalar multiplication on the tangent space to SPD,:
Scalar Multiplication and Matrix Scaling:
P pa = exp(a
where exp, log the matrix exponential and logarithm. We further generalize the notion of scalar multiplication to allow for different relative expansion rates in different directions. For a fixed basepoint I and point P SPD,, symmetric matrix Sn.. We define this matrix scaling by:
replace the scalar from Eq. 3 with an arbitrary real
A P exp(A A @ log(P))
where X denotes the Hadamard product. We denote the matrix scaling with @, extending the previous usage: for any a E I,, we have [a] @ P = a @ P where [a] is the matrix with every entry
5 Implementation
In this section we detail how we learn representations in SPD., and implement different linear mappings so that they conform the premises of each operator, yielding SPD neural layers.
Embeddings in SPD,, and Sn We are interested in learning embeddings in SPD,, To do so exploit the connection between SPD,, , and its tangent space S, through the exponential and logarithmic maps. To learn point P E SPD,, we first model as a symmetric matrix E S,, We impose symmetry U by learning triangular matrix X Rax with n(x+1)/2 parameters, such that U = X + X". To obtain the matrix P SPD,, we employ the exponential map: P = exp(U). Modeling points the tangent space offers advantages for optimization, explained in $5 For the matrix scaling A @ P, we impose symmetry on the factor matrix A E S, in the same way that we learn the symmetric matrix U.
Isometries: Rotations and Reflections: Rotations in n dimensions are described as collections of pairwise orthogonal 2-dimensional rotations in planes (with a leftover 1- dimensional "axis of rotation" " in odd dimensions). We utilize this observation to ef- ficiently build elements of (nn) out of two-dimensional rotations in coordinate planes. More precisely, for any E [0,2x) and choice of sign {+.-} we let R= de- note the 2-dimensional rotation (+) or reflection (-) as R²(())
(
cos 0 F sin cos
Then for any pair the transformation which applies R=((0) the plane of TR.", and leaves all other coordinates fixed. For example, in O(5) the element (0) (see on the right) denotes the transformation where we replace the entries (ii, ij. ji. of I, with the corresponding values R+ (0). More gen- eral, rotations and reflections are built by taking products
in I...n, we denote by (0)
0
0
0
0
cos
0
0
-
sin
0
(5)
0
0
1
0
0
0
sin 0
cos
0
of these basic transformations, Given p(n- -aa-1)/2-dimemnieeee 21/2-dimensional vector of angles Ø (012..... ...........
5

---

and choice of sign, we define the rotation and reflection corresponding to & by:
Rot(O)) II R (
Refl( = II (0.g
(5)
where Rot(O), (e) E R are the isometry matrices, and the vector of angles 0 can be regarded learnable parameter of the model. Finally, denote the application of the transformation M to the point P E SPD, by:
M の MPM
(6)
Optimization:
For the proposed rotations and reflections, the learnable weights are vectors of angles 0 E IR "(u-") which do not pose an optimization challenge. On the other hand, embeddings in SPD have to be optimized respecting the geometry of the manifold, but as already explained, we model them on the space of symmetric matrices S,,, and then we apply the exponential map. In this manner, we are able to perform tangent space optimization [26] using standard Euclidean techniques, and
circumvent the need for Riemannian optimization [15] [T], which we found be less numerically stable. Due to the geometry of SPD,, (see Appendix C.3 this is an exact procedure, which does not incur losses representational power.
Complexity: The most frequently utilized operation when learning embeddings is the distance calculation, thus analyze its complexity. In Appendix [ we detail the complexity of different operations. Calculating the distance between two points in SPD,, implies computing multiplications, inversions and diagonalizations of with respect to the matrix dimensions is One)). Although matrix of rank n implies n + 1)/2 dimensions thus large n value is usually not required, the cost of many operations polynomial instead of linear.
matrices. We find that the cost of the distance computation
Towards neural network architectures: We employ the proposed mappings along with the gyro- vector operations as building blocks for SPD neural layers. This is showcased in the experiments presented below. Scalings, rotations and reflections can be gyro-addition allows us to define the equivalent of bias addition. Finally, although we do not employ non-linearities, approach can be seamlessly integrated with the ReEig layer (adaptation of ReLU layer for SPD) proposed in 40
feature transformations. Moreover,
6 Experiments
In this section employ the transformations developed on SPD to build neural models for knowledge graph completion, item recommendation and question answering. Task-specific models in different geometries have been developed in the three cases, hence consider them adequate benchmarks for representation learning
6.1 Knowledge Graph Completion
Knowledge graphs (KG) represent heterogeneous knowledge in the shape of (head, relation, tail) triples, where head and tail are entities and relation represents relationships among entities. KG exhibit an intricate and varying structure where entities can be connected by symmetric, anti- symmetric, or hierarchical relations. To capture these non-trivial patterns more expressive modelling techniques become necessary (25), thus we choose this application to showcase the capabilities of our transformations on SPD manifolds. Given an incomplete KG, the task to predict which unknown links valid.
Problem formulation: Let (E, R, T) be a knowledge graph where & the set of entities, R is the set of relations and T C E x R x E is the set of triples stored in the graph. The usual approach is learn a scoring function Ø E R with the goal of scoring all missing triples correctly. To do so, we propose to learn representations of entities as embeddings in SPD,,, and relation- specific transformation in the manifold, such that the KG structure is preserved.
+ R that measures the likelihood of a triple be true,
Code available htttp://ggihuub.co///eeeeeee

---

We follow the base hyperbolic model MuRP 8 and adapt it into SPD,, by means of the matrix sealing. Its scoring function has shown success in the task given that it combines multiplicative and additive components, which are fundamental to model different properties of KG relations We translate into SPD, as:
Scaling model:
rh,, d(( & H) R T) +
where H, T E SPD,, are embeddings and 6n, by E R are scalar biases for the head and tail entities respectively. R SPD,, and M, are matrices that depend on the relation. For d(.,-), experiment with the Riemannian and the Finsler One metric distances.
Isometric model: A possible alternative is to embed the relation-specific transformations as elements of the (n) group (i.e., rotations and reflections). This technique has proven effective in different metric spaces 25 [8]. In this case, M, is rotation reflection matrix as in Eq. [ and the scoring function is defined as:
(h,, r, t) = (M, M @
H +
R. T) + by + by
Datasets: We employ two standard benchmarks, namely WN18RR 16 28 and FB15k-237 [6 76
WNI8RR subset of WordNet (59) containing 11 lexical relationships between 40, 943 word senses, FB15k-237 is subset of Freebase [1). with 14, 541 entities and 237 relationships.
Training: We follow the standard data augmentation protocol by adding inverse relations to the datasets [48]. We optimize the cross- cross-yntropy loss with uni-
L=
Σ log(1 + rx, r)))
form negative sampling defined in Equation 9 where
(NiDET
T is the set of training triples, and Y2 -1 if f. is factual triple or Y, 1 if / is a negative sample. We employ the Adam optimizer [55]. We conduct a grid search with matrices of dimension n x n where n E (14, 20, 24) (this is the equívalent of (105, 210, 300) degrees of freedom respectively) to select optimal dimensions, learning rate and weight decay, using the validation set. More details and set of hyperparameters in Appendix B.]
Evaluation metrics: At test time, we rank the correct tail or head entity against all possible entities using the scoring function, and use inverse relations for head prediction [48). Following previous work, we compute two ranking-bass mean of inverse ranks assigned to correct entities, and hits at K (H@ K, K E 3, 10)). which measures the proportion of correct triples among the top K predicted triples. We follow the standard evaluation protocol of filtering out all true triples in the KG during evaluation [16]
metrics: mean reciprocal rank (MRR). which measures the
We compare our models with their respective equivalents in different metric spaces, which are also state-of-the-ard models for the task. For the scaling model, these are MURE and MURP (8), which perform the scaling operation in Euclidean and hyperbolic space respectively. For the isometric models, we compare to RoTC [3], ROTE and ROTH, (25] (rotations in Complex, Euclidean and hyperbolic space respectively), and REFE and REFH (25 (reflections in Euclidean and hyperbolic space). Baseline results are taken from the original papers. We do not compare to previous work
Baselines:
on SPD given that they lack the definition of an arithmetic operation in the space, thus a vis-a-vis comparison is not possible.
We report the performance for all analyzed models, segregated by operation, in Table On both dataset, the scaling model SPDsca outperforms its direct competitors MuRE and MuRP, and this is specially notable in HR @ 10 for WNI8RR: 59.0 for SPD vs 55.4 and 56.6 respectively, SPD reflections are very effective on WN18RR as well. They outperform their Euclidean and hyperbolic counterparts RefE. and RefH, in particular when equipped with the Finsler metric. Rotations on the SPD manifold, on the other hand, seem to be less effective. However, Euclidean and hyperbolic rotations require 500 dimensions whereas the SPDRot models are trained on matrices of rank 14 (cquivalent to 105 dims). Moreover, the underperformance observed in some of the analyzed cases for rotations and reflections does not repeat in the following experiments ($6.2 & 6..3.. Hence, we consider this is due to overfitting in some particular setups. Although we tried different regularization methods, regard sub-optimal configuration rather than geometric reason to be the cause for the underperformance.
Results:
Regarding the choice of a distance metric, the Finsler One metric is better suited with respect to HR @ and HR @ 10 when using scalings and reflections on WN18RR. For the FB15k-237 dataset, SPD models operating with the Riemannian metric outperform their Finsler counterparts. This
7

---

Table I: Results for Knowledge graph completion.
WN18RR
FB15k-237
Operation
Model
MRR
HR@1 l
HR@3
HR@ 10
MRR
HR@1 I
HR@3
HR @ 10
MURE
47.5
43.6
48.7
55.4
33.6
24.5
37.0
52.1
Scaling
MURP
48.1
44.0
49.5
56.6
33.5
24.3
36.7
51.8
SPD
Sea
48.1
43.1
50.1
57.6
34,5
25.1
38.0
53,5
SPD
48.4
42.6
51.0
59.0
32.9
23.6
36.3
51.5
Sca
RoTC
47.6
42.8
49.2
57.1
33.8
24.1
37.5
53.3
RoTE
49.4
44.6
51.2
58.5
34,6
25.1
38.1
53.8
Rotations
ROTH
49.6
44.9
51.4
58.6
34.4
24.6
38.0
53.5
SPD
46.2
39.7
49.6
57.8
32.9
23.6
36.3
51.6
SPD Rot
40.9
30.5
48.2
57.3
32.1
22.9
35.4
50.5
REFE
47.3
43.0
48.5
56.1
35.1
25,6
39,0
54,1
REFH
46.1
40.4
48.5
56.8
34.6
25.2
38.3
53.6
Reflections
SPD
48.3
44.0
49.7
56.7
32.5
23.4
35.6
$1.0
SPD Ref
48.7
44.3
50.1
57.4
31.6
22.5
34.6
50.0
suggests that the Riemannian metric is capable of disentangling the large number of relationships in
this dataset to a better extent.
In these experiments we have evaluated models applying equivalent operations and scoring functions in different geometries, thus they can be thought vis-a-vis comparison of the metrie spaces. We observe that SPD models tie or outperform baselines in most instances. This showcases the improved representation capacity of the SPD manifold when compared to Euclidean and hyperbolic spaces. Moreover, it demonstrates the effectiveness of the proposed metrics and operations in this manifold.
6.2 Knowledge Graph Recommender Systems
Recommender systems (RS) model user preferences provide personalized recommendations 93 KG embedding methods have been widely adopted into the recommendation problem as an effective tool model side information and enhance the performance 99 34]. For instance,
reason for recommending movie to a particular user is that the user has already watched many movies from the same genre or director (56). Given multiple relations between users, items, and heterogencous entities, the goal is to predict the user's next item purchase or preference.
We model the recommendation problem In addition, we aim to incorporate side information between users, items and other entities. Hence we apply our KG embedding method from $ as is, to embed this multi-relational graph. We evaluate the capabilities of the approach by only measuring the performance over user-item interactions.
Model:
link prediction task over users and items 40
Datasets: To investigate the recommendation problem endowed with added relationships, employ the Amazon dataset [58 61 (branches "Software", "Luxury & Beauty" and "Prime Pantry"), with users' purchases of products, and the MindReader dataset [18) of movie recommendations. Both datasets provide additional relationships between users, items and entities such as product brands, or movie directors and actors. To generate evaluation splits, the penultimate and last item the user has interacted with are withheld as dev and test sets respectively.
Training: In this setup we also augment the data by adding inverse relations and optimize the loss from Equation g We the size of the matrices to 10 x 10 dimensions (equivalent 55 free parameters). More details about relationships and set of hyperparameters in Appendix B.2
Evaluation and metries: We follow the standard procedure of evaluating against 100 randomly selected samples the user has not interacted with 38 533 To evaluate the recommendation perfor- mance we focus on the buys likes relation. For each user u we rank the items i according to the scoring function of u, buys, iy). We adopt MRR and H@@ 10, as ranking metrics for recommendations.
Baselines: We compare to TransE [6], RotC 73 MuRE and MuRP trained with 55 dimensions.
Results: In Table we observe that the SPD models tie outperform the baselines in both MRR and HR @ 10 across all analyzed datasets. Rotations in both, Riemannian and Finsler metrics, are more effective in this task, achieving the best performance scaling models. Overall, this shows the capabilities of the systems to effectively represent user-item interactions enriched with relations between items and their attributes, thus learning to better model users' preferences. Furthermore, it displays the versatility of the approach to diverse data domains.
out cases, followed by the

---

Table 2: Results for Knowledge graph-based recommender systems.
SOFTWARE
LUXURY
PANTRY
MINDREADER
Model
MRR
H@ 10
MRR
10@10
MRR
H@@0
MRR
H@ 10
TRANSE
28.5++0.1
47.2+0.5
35.6+0.1
52.3=0.1
16.6+0.0
35.3++0.1
19.1=0.4
37.6+0.1
ROTC
28.3400.
15.4=1.4
33.0:0.1
49.8=0.2
14.5+0.0
31.340.2
25.3=0.3
50.31:06
MuRE
29.4:1:0.4
47.110.4
35.6:1..7
54.0:0.3
1944:0.1
39.5:0022
25.2::0.3
49.9::.6
MURP
29.6=0033
47.9=0.3
37.5+0.11
55.2-0.3
19.400.1
39,8+0.2
25.3=0.3
49.3+0.2
SPD
29.410..
48.110.8
37.5.1.22
55.110.2
19.510.0
39.610.3
25.410.1
49.8.10.3
SPD 28.810.11
46.910.5
37.310.3
54.110.9
19.010.11
388800.2
2..7.0..5
49.5.10.1
SPD 30.310.2
48.6.10.9
37.2.10.1
54.8104
20.0110.1
40.310.1
25.3.10.
50,5100,
SPD Bor
30.110.11
09.1
36.910.11
54.510.6
19.210.0
39.310.11
25.7
49.510.2
SPD te
29.6+0.2
48.0+-0.5
37.3+0.2
55.0=0.2
19.3+0.0
39.7++0.3
25.3+000
49.1+0.1
SPD
Ref
29.3++0.1
47.5+-0.6
36.8+0.0
54.8+0.11
18.6+0.2
38.3++033
24.8+0022
47.9+188
6.3 Question Answering
We evaluate our approach on the task of Question Answering (QA). In this manner we also showease the capabilities of our methods to train word embeddings.
Model: We adapt the model from HyperQA 74 SPD. We model word embeddings 4 E SPD,, and represent question/answers
sim(g, a) -wyd(QQA. wb:
summation of the embeddings of their corresponding tokens. We apply a feature transforma- tion T(+) followed by a bias addition, an equivalent of neural linear layer. (-)) can be a scaling, rotation or reflec- tion. Finally we compute a distance- representations as defined in Equation 10 where wy, wy E R, B E SPD, and the transformation T are parameters of the model.
(10)
where Q = TT ( #
similarity function between the resulting question/answer
Datasets: We analyze two popular benchmarks for QA: TrecQA 85 (clean version) and WikiQA 89 filtering questions with multiple answers from the dev and test sets.
Training: We optimize the cross-entropy loss from Eq. 9 where we replace for sim(g, a) and for each question we use wrong answers as negative samples. We set the size of the matrices to 14 x 14 dimensions (equivalent to 105 free parameters). The set of hyperparameters can be found in Appendix [.3]
Table 3: Results for Question Answering.
TREEQA
WIKIQA
Model
MRR
H@1
MRR
Euelidean
55.9=2.0
41.0+20.
43.4-1..
1..4+
Hyperbolic
58.0=1.3
39.322.0
44.0:0.4
22.8:0.6
SPD
05.4=0.1
37.110.1
45.5:10.5
2..4::111
SPD
57.1=0.7
38.6:0..
0.5
24.0-10.6
test time, for each ques- tion we rank its candidate answers according to
Evaluation metrics:
SPD
5..7=1.5
41.44229
44.6:10.6
23.6-10.6
SPD
58.1=0.5
43.6:::00
43.7400.4
23.8:0088
SPD
57.3-0.3
40.7+1.1
48.9400.7
23.442000
Eq. TO We adopt MRR and H@@ 1 as evaluation metries.
SPD
59.6=0.5
25.0+:.55
We compare against Euclidean and hyperbolic spaces of 105 dimensions. For the Euclidean model we employ a linear layer as feature transformation. For the hyperbolic model, operate on the tangent space and project the points into the Poincaré ball to compute distances.
Baselines:
We present results in Table In both datasets, transformations learned by the SPD models are able to place questions and answers representations in the space such that they outperform Euclidean and hyperbolic baselines. Finsler metrics seem be very effective in this scenario, improving the performance of their Riemannian counterparts in many cases. Overall, this suggests that embeddings in SPD manifolds learn meaningful representations that can be exploited into downstream tasks. Moreover, we showcase how to employ different operations feature transformations and bias additions, replicating the behavior of linear layers in classical deep learning architectures that can be seamlessly integrated with different distance metrics.
Results:
that the word embeddings and
6.4 Analysis
One reason to embed data into Riemannian manifolds, such as SPD, to use geometric properties of the manifold to analyze the structure of the data [52]. Visualizations in SPD are difficult due their

---

2.0
HR@ 10: 97.21%
0.00
0.00
0.05
0.10
0.15
0.20
0.00
0.2
0.6
0.0
0.5
1.0
1,5
2.0
0.10
4.0
HR@ 10: 37.19%
0.5
0.2
0.00
0.05
0.30
0.15
0.20
0.0
0.2
0.4
0.6
0.8
0.0
0.5
1.0
1.5
2.0
Figure 5: Train, negative and validation triples for relationships 'derivationally related form' (top) and 'hypernym' (bottom) for 5 (left), 50 (center) and 3000 (right) epochs for the SPD Fi model. The red dot corresponds to the relation addition R.
high dimensionality. As solution we use the vector- distance function to develop a new tool visualize and analyze structural properties of the learned representations.
We adopt the vector (n - 1, n 3, ... -n + 3, -n + 1), as the barycenter of the space in TR" where the VVD is contained. Then, plot the norm of the VVD vector and its angle with respect to this barycenter. In 5 we compute and plot the VVD corresponding to d(M,- & H, T) and R as defined in Eq. 7 for KG models trained on WNISRR. In early stages of the training, all points fall near the origin (left side of the plots). As training evolves, the model learns to separate true (, t) triples from corrupted ones (center part). When the training converges, the model is able to clearly disentangle and cluster positive and negative samples. We observe how the position of the validation triples (green points, not seen during training) directly correlates with the performance of each relation. Plots for more relations in Appendix B.I
7 Conclusions
Riemannian geometry has gained attention due to its capacity to represent non-Euclidean data arising in several domains. In this work introduce the vector-valued distance function, which allows to implement universal models (generalizing previous metrics on SPD), and can be leveraged to provide geometric interpretation on what the models learn. Moreover, bridge the gap between Euclidean and SPD geometry under the lens of the gyrovector theory, providing means to transfer standard arithmetic operations from the Fuclidean setting their analog notions in SPD, These tools enable practitioners to exploit the full representation power of SPD, and profit from the enhanced expressivity of this manifold. We propose and evaluate SPD models on three tasks and eight datasets, which showcases the versatility of the approach and ease of integration with downstream tasks. The results reflect the superior expressivity of SPD when compared to Euclidean or hyperbolic baselines.
This work is not without limitations. We consider the computational complexity of working with spaces of matrices to be the main drawback, since the cost of many operations is polynomial instead of linear. Nevertheless, a matrix of rank n implies n + 1)/2 dimensions thus large n value is usually not required.
Acknowledgments and Disclosure of Funding
This work has been supported by the German Research Foundation (DFG) as part of the Research Training Group Adaptive Preparation of Information from Heterogeneous Sources (AIPHES) under grant No. GRK 1994/1, as well as under Germany's Excellence Strategy EXC-2181/1 390900948 (the Heidelberg STRUCTURES Cluster of Excellence), and by the Klaus Tschira Foundation, Heidelberg, Germany.
10

---

References
[1] Abe, T. and Hatori, O. (2015). Generalized gyrovector spaces and a mazur-ulam theorem. Publicationes Mathematicae Debrecen, 87:393-413.
[2] Absil, P.-A., Mahony, R., and Sepulchre, R. (2009). aenpiiiiattoonAgooiiiiiii on Matrix Manifolds. Princeton University Press.
[3] Ai, Q., Azizi, V., Chen, X., and Zhang, Y. (2018). Learning heterogeneous knowledge base embeddings for explainable recommendation. Algorithms, 11(9).
[4] Allen, C., Balazevic, I., and Hospedales, T. (2021). Interpreting knowledge graph relation representation from word embeddings. In International Conference on Learning Representations.
[5] Arsigny, V., Fillard, P., Pennec, X., and Ayache, N. (2006a). Geometric means in novel vector space structure on symmetric positive-definit matrices. SIAM J. Matrix Anal. Appl., 29(1):328 347. -47.
[6] Arsigny, V., Fillard, P., Pennec, X., and Ayache, N. (2006b). Log-euclidean metrics for fast and simple calculus diffusion tensors. Magnetic Resonance in Medicine, 56(2):411-421.
[7] Bachmann, G., Becigneul, G., and Ganea, O.-E. (2020). Constant curvature graph convolutional networks. In 37th International Conference on Machine Learning (ICML).
[8] Balazevic, I., Allen, C., and Hospedales, T. (2019a). Multi-relational poincaré graph embeddings. Wallach, H., Larochelle, H., Beygelzimer, A., d' d'Alché-Buc, F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 32, pages 4463 44734 Curran Associates, Inc.
[9] Balazevie, I., Allen, C., and Hospedales, T. (2019b). TuckER: Tensor factorization for knowledge graph complction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process- ing (EMNLP-IJCNIP), pages 3185-5194, Hong Kong, China. Association for Computational Linguisties.
[10] Ballmann, W., Brin, M., and Eberlein, P. (1985). Structure manifolds of nonpositive curvature. I. Annals of Mathematics, 122(1):171-233,
[11] Bécigneul, G. and Ganea, O. (2019). Riemannian adaptive optimization methods. In 7th International Conference on Learning Representations, ICLR, New Orleans, LA, USA.
[12] Bhatia, R. (2003). On the exponential metric increasing property. Linear Algebra and its Applications, 375:211-22
[13] Bhatia, R., Jain, T., and Lim, Y. (2019). On the Mures-Wasssrsteii distance between positive definite matrices. Expositiones Mathematicae, 1652)1165-1
[14] Bollacker, K., Evans, C., Paritosh, P., Sturge, T., and Taylor, J. (2008). Freebase: A collabora- tively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD International Conference Management of Data, SIGMOD '08, page 1247-1250. New York, NY, USA. Association for Computing Machinery.
[15] Bonnabel, S. (2011). Stochastic gradient descent Riemannian manifolds. IEEE Transactions on Automatic Control, 58.
[16] Bordes, A., Usunier, N., Gareia-Duran, A., Weston, J., and Yakhnenko, O. (2013). Translating embeddings for modeling multi-relational data. In Burges, C. J. C., Bottou, L., Welling, M., Ghahramani. Z., and Weinberger, K. Q., editors, Advances in Neural Information Processing Systems, volume 26, pages 2787-2795. Curran Associates, Inc.
[17] Boumal, N., Mishra, B., Absil, P.-A., and Sepulchre, R. (2014). Manopt, a matlab toolbox for optimization manifolds. Journal of Machine Learning Research, 1454599445

---

[18] Brams, A. H.. Jakobsen, A. L., Jendal, T. E., Lissandrini, M., Dolog, P., and Hose, K. (2020). Mindreader: Recommendation over knowledge graph entities with explicit user ratings. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, CIKM '20, page 2975-2982, New York, NY, USA. Association for Computing Machinery.
[19] Bronstein, M. M., Bruna, J., LeCun, Y., Szlam, A., and Vandergheynst, P. (2017). Geometric deep learning: Going beyond Euclidean data. IEEE Signal Processing Magazine, 34(4):18-42,
[20] Brooks, D., Schwander, O., Barbaresco, F., Schneider, J.-Y., and Cord, M. (2019a). Riemannian batch normalization for SPD neural networks. In Wallach, H., Larochelle, H., Beygelzimer, A., d' d'Alché-Buc, F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 32, pages 15589-15500. Curran Associates, Inc.
[21] Brooks, D. A., Schwander, O., Barbaresco, F., Schneider, J.-Y., and Cord, M. (2019b). Exploring complex time-series representations for riemannian machine learning of radar data. In ICASSP 2019- 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1672-3676.
[22] Carreira, J., Caseiro, R., Batista, J., and Sminchisescu, C. (2012). Semantic segmentation with second-order pooling. In Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., and Schmid, C., editors, Computer Vision ECCV 2012, pages 430 443, Berlin, Heidelberg. Springer Berlin Heidelberg.
[23] Chakraborty, R., Bouza, J., Manton, J., and Vemuri, B. C. (2020). Manifoldnet: A deep neural network for manifold-valued data with applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-1.
[24] Chakraborty, R., Yang, C.-H., Zhen, X.,, Banerjee, M., Archer, D., Vaillancourt, D., Singh, V., and Vemuri, B. (2018). A statistical recurrent model on the manifold of symmetric positive definite matrices. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.
[25] Chami, I., Wolf, A., Juan, D.-C., Sala, F., Ravi, S., and Ré, C. (2020). Low-dimensional hyperbolic knowledge graph embeddings. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 6901-6914, 14, Online. Association for Computational Linguistics.
[26] Chami, I., Ying, Z., Ré, C., and Leskovec, J. (2019). Hyperbolic graph convolutional neural networks. In Advances in Neural Information Processing Systems 32, pages 4869 48800 Curran Associates, Inc.
[27] Cruceru, C., Becigneul, G., and Ganea, O.-E. (2020). Computationally tractable Riemannian manifolds for graph embeddings. In 37th International Conference on Machine Learning (ICML).
[28] Dettmers, T., Pasquale, M., Pontus, S., and Riedel, S. (2018). Convolutional 2d knowledge graph embeddings. In Proceedings of the 32th AAAI Conference on Artificial Intelligence, pages 1811-1818.
[29] Dong, Z., Jia. S., Zhang, C., Pei, M., and Wu, Y. (2017). Deep manifold learning of symmetric positive definite matrices with application to face recognition. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI' 17, page 4005-4015. AAAI Press.
[30] Donoho, D. L. and Tsaig, Y. (2008). Fast solution of -nom minimization problems when the solution may be sparse. IEFE Trans. Information Theory, 54(1) (111)):87994811
[31] Feragen, A., Lauze, F., and Hauberg. S. (2015). Geodesic exponential kernels: When curvature and linearity conflict. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3032-3042.
[32] Ganea, O., Becigneul, G., and Hofmann, T. (2018). Hyperbolic neural networks. In Advances in Neural Information Processing Systems 31, pages 5345-5355. Curran Associates, Inc.
[33] Gao, 7., Wu, Y., Bu, X., Yu, T., Yuan, J., and Jia, (2019). Learning robust representation via deep network on symmetric positive definite manifolds. Pattern Recognition, 92:1-12.
12

---

[34] Guo, Q.. Zhuang, F., Qin, C.. Zhu, H., Xie, X., Xiong, H., and He. Q. (2020). A survey on knowledge graph-based recommender systems. IEEE Transactions on Knowledge and Data Engineering, pages 1-1.
[35] Ha Quang. M., San Biagio, M., and Murino, V. (2014). Sog-etiibert-thhiiitt metric between positive definite operators on Hilbert spaces. In Ghahramani, Z., Welling, M.. Cortes, C., Lawrence, N., and Weinberger, K. Q., editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc.
[36] Harandi, M. T., Salzmann, M., and Hartley, R. (2014). From manifold manifold: Geometry- aware dimensionality reduction for SPD matrices. In Fleet, D., Pajdla, T., Schiele, B., and Tuytelaars, T., editors, Computer Vision - ECCV 2014, pages 17-32, Cham. Springer International Publishing.
[37] Hatori, O. (2017). Examples and applications of generalized gyrovector spaces. Results in Mathematics, 71:295-317.
[38] He, X., Liao, L., Zhang, H., Nie, L.,, Hu, X., and Chua, T.-S. (2017). Neural collaborative filtering. In Proceedings of the 26th International Conference World Wide Web, WWW '17, page 173-182, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
[39] Helgason, S. (1978). Differential geometry, Lie groups, and symmetric spaces. Academic Press New York.
[40] Huang, Z. and Gool, L. V. (2017). A Riemannian network for SPD matrix learning. In Proceed- ings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI' page 2036-2042. AAAI Press.
[41] Huang, Z. Wang, R., Shan, S., and Chen, X. (2014). Learning Auclidean-to-Riemamiia metric for point-to-set classification. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 1677-1684.
[42] Huang, Z., Wang, R., Shan, S., Li. X., and Chen, X. (2015). Log-Euclidean metric learning on symmetric positive definite manifold with application to image set classification. In Proceedings of the 32nd International Conference on International Conference Machine Learning Volume 37, ICML'CI 15. page 720-729. JMLR.org.
[43] Ionescu, C., Vantzos, O., and Sminchisescu, C. (2015). Matrix backpropagation for deep networks with structured layers. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 2965-2973.
[44] Jayasumana, S., Hartley, R., Salzmann, M., Li, H., and Harandi, M. (2013). Kernel methods on the Riemannian manifold of symmetric positive definite matrices. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
[45] Kapovich, M. Leeb, B., and Millson, J. (2009). Convex functions on symmetric spaces, side lengths of polygons and the stability inequalities for weighted configurations infinity. J. Differential Geom.
[46] Kapovich, M., Leeb, B., and Porti, J. (2017). Anosov subgroups: dynamical and geometric characterizations, European Journal of Mathematics. 8(3):808-899
[47] Kim, S. (2016), Gyrovector spaces on the open convex cone of positive definite matrices. Mathematics Interdisciplinary Research, 1(1):173-185.
[48] Lacroix, T., Usumier, N., and Obozinski, G. (2018). Canonical tensor decomposition for knowl- edge base completion. In Dy. J. and Krause, A., editors, Proceedings of Machine Learning Research, volume 80 of Proceedings of Machine Learning Research, pages 2863-2872, Stock- holmsmässan, Stockholm Sweden. PMLR.
[49] Li, J., Zhang, L., Meng, F., and Li, F. (2014). Recommendation algorithm based on link prediction and domain knowledge in retail transactions. Procedia Computer Science, 31:875- 881. 2nd International Conference on Information Technology and Quantitative Management, ITQM 2014.
13

---

[50] Li, P., Xie, J., Wang, Q., and Gao, Z. (2018). Towards faster training of global covariance pooling networks by iterative matrix square root normalization. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 947-955.
[51] Li, Y. and Lu, R. (2018). Locality preserving projection on SPD matrix lie group: algorithm and analysis. Sci. China Inf. Sci., 6119))092104411-02210411
[52] López, F., Pozzetti, B., Trettel, S., Strube, M., and Wienhard, A. (2021a). Symmetric spaces for graph embeddings: A finsler-riemannian approach. In Meila, M. and Zhang, T., editors, Proceed- ings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 7090-7101. PMLR.
[53] López, F,, Scholz, M., Yung, J., Pellat, M., Strube, M., and Dixon, L. (2021b). Augmenting the user-item graph with textual similarity models.
[54] López, F. and Strube, M. (2020). A fully hyperbolic neural model for hierarchical multi-class classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 460-475, Online. Association for Computational Linguistics.
[55] Loshchilov, I. and Hutter, (2019). Decoupled weight decay regularization. In International Conference on Learning Representations.
[56] Ma, W., Zhang, M., Cao, Y., Jin, W., Wang, C., Liu, Y., Ma, S., and Ren, X. (2019). Jointly learning explainable rules for recommendation with knowledge graph. In The World Wide Web Conference, WWW '19, page 1210-1221, New York, NY, USA. Association for Computing Machinery.
[57] Mao, Y., Wang. R., Shan, S., and Chen, X. (2019). Cosonet: Compact second-order network for video face recognition. In Jawahar. C. V., Li, H., Mori, G., and Schindler, K., editors, Computer 2018, pages 51-67, Cham. Springer International Publishing.
[58] McAuley, J. and Leskovec, J. (2013). Hidden factors and hidden topics: Understanding rating dimensions with review text. In Proceedings of the 7th ACM Conference on Recommender Systems, RecSys 13. page 165-172, New York, NY, USA. Association for Computing Machinery.
[59] Miller, G. A. (1992). WordNet: A lexical database for English. In Speech and Natural Language: Proceedings of a Workshop Held at Harriman, New York, February 23-26, 1992.
[60] Nguyen, X. S., Brun, L., Lezoray, O., and Bougleux, S. (2019). A neural network based on SPD manifold learning for skeleton- hand gesture recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[61] Ni, J., Li, J., and McAuley, J. (2019). Justifying recommendations using listantly-labeled reviews and fine-grained aspects. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 188-197, Hong Kong, China. Association for Computational Linguisties.
[62] Nickel, M. and Kiela, D. (2017). Poincaré embeddings for learning hierarchical representations. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems 30, pages 6341-6350. Curran Associates, Inc.
[63] Nielsen, F. and Sun, K. (2019). Clustering in Hilbert's Projective Geometry: The Case Studies of the Probability Simplex and the Elliptope of Correlation Matrices, pages 297-331. Springer International Publishing, Cham.
[64] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., De Z., Raison M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. (2019). Pytorch: An imperative style, performace deep learning library. In Wallach, H., Larochelle, H., Beygelzimer, A., Alché-Buc, F,, Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems 32, pages
Curran Associates, Inc
14

---

[65] Pennec, X., Fillard, P., and Ayache, N. (2006). A Riemannian framework for tensor computing. Int. J. Comput. Vision, 66(1):41-66
[66] Planche, P. (1995). Géométrie de Finsler sur les espaces symétriques. PhD thesis, Université de Genève, Geneve, Switzerland.
[67] Ratliff, N. D., Wyk, K. V., Xie, M.. Li, A., and Rana, M. A. (2020). Generalized nonlinear and Finsler geometry for roboties. CoRR, abs/2010.14745.
[68] Said, S., Bombrun, L., Berthoumiee, Y., and Manton, H. (2017). Riemannian Gaussian distri- butions on the space of symmetric positive definite matrices. IEEE Transactions Information Theory, 6354)22153-2
[69] Shen, (2006). Riemann-
Feemann-Finsler Finsler geometry with applications to information geometry. Chinese Annals of Mathematics, Series B, 27:73-99 94.
[70] Shimizu, R., Mukuta, Y., and Harada, T. (2021). Hyperbolic neural networks++. In International Conference on Learning Representations.
[71] Sra, S. (2012). new metric on the manifold of kernel matrices with application matrix geometric means. In Pereira, F., Burges, C. J. C., Bottou, I.., and Weinberger, K. Q., editors, Advances Neural Information Processing Systems, volume 25. Curran Associates, Inc.
[72] Sra, S. (2015). Positive definite matrices and the S-divergence. Proceedings of the American Mathematical Society. Published electronically: October 22, 2015.
[73] Sun, Z., Deng, Z.-H., Nie, J.-Y., and Tang, J. (2019). Rotate: Knowledge graph embedding by relational rotation in complex space. International Conference on Learning Representations.
[74] Tay, Y., Tuan, L. A., and Hui, S. C. (2018). Hyperbolic representation learning for fast and efficient neural question answering. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM 18, pages 583-591, New York, NY, USA. ACM.
[75] Tosato, D., Farenzena, M., Spera, M., Murino, V., and Cristani, M. (2010). Multi-class classification on Riemannian manifolds for video surveillance. In Danillidis, K., Maragos, P., and Paragios, N., editors, Computer Vision ECCV 2010, 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010 Proceedings, Part 11, volume 6312 of Lecture Notes in Computer Science, pages 378-391. Springer.
[76] Toutanova, K. and Chen, D. (2015). Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop Continuous Vector Space Models and their Compositionaliit, pages 57-66, Beijing, China. Association for Computational Linguistics.
[77] Trouillon, T., Welbl, J., Riedel, S., Gaussier, E., and Bouchard, G. (2016). Complex embeddings for simple link prediction. In Proceedings of the 33rd International Conference on International Conference on Machine Learning Volume 48, ICML' 16, page 2071 12000 JMLR.org.
[78] Tuzel, O., Porikli, F,, and Meer, (2006). Region covariance: fast descriptor for detection and classification. In Leonardis, A., Bischof, H., and Pinz, A., editors, Computer Vision ECCV 2006, pages 589-600, Berlin, Heidelberg. Springer Berlin Heidelberg.
[79] Tuzel, O.,, Porikli, F., and Meer, P. (2008). Pedestrian detection via classification on Riemannian manifolds. IEEE Trans. Pattern Anal. Mach. Intell., 10):1111311117
[80] Ungar, A. (2005). Gyrovector spaces and their differential geometry. Nonlinear Functional Analysis and Applications, 10.
[81] Ungar, A. A. (2008). Gyrovector Space Approach to Hyperbolic Geometry. Morgan & Claypool.
[82] Ungar, A. A. (2018). Beyond Sreudo-Rotations in Analysis and its Applications. Academic Press.
Spaces. Mathematical
[83] Vemulapalli, and Jacobs, D. (2015). Riemannian metric learning for symmetric positive definite matrices. ArXiv, 0bs//550100239
15

---

[84] Vermeer, J. (2005). A geometric interpretation of Ungar's addition and of gyration in the hyperbolic plane. Topology and Its Applications: a journal devoted general, geometric, set- theoretic and algebraic topology, 152(3):226-242.
[85] Wang, M., Smith, N. A., and Mitamura, T. (2007). What is the Jeopardy model? A quasi- synchronous grammar for QA. In Proceedings of the 2007 Joint Conference on Empirical Meth- ods in Natural Language Processing and Computational Natural Language Learning (EMNLP. CoNLL), pages 22-32, Prague, Czech Republic. Association for Computational Linguistics.
[86] Wang, W. Wang, R., Huang, Z., Shan, S., and Chen, X. (2018). Discriminant analysis on Riemannian manifold of Gaussian distributions for face recognition with image sets. IEEE Transactions on Image Processing, 27(1):151-663.
[87] Wu, Y., Jia, Y., Li. P., Zhang, J., and Yuan, J. (2015). Manifold kernel sparse representation of symmetric positive-definit matrices and its applications. IEEE Transactions on Image Processing, 24(11)3729-3741.
[88] Yang, T., Sha, L., and Hong, P. (2020). NagE: Non-Abelian Group Embedding for Knowledge Graphs, page 1742. Association for Computing Machinery, New York, NY, USA.
[89] Yang, Y., Yih. W.-t. and Meek, C. (2015). WikiQA: A challenge dataset for open-domain question answering. In Proceedings the 2015 Conference on Empirical Methods in Natu- ral Language Processing, pages 2013-2018, Lisbon, Portugal. Association for Computational Linguistics.
[90] Yin, M., Guo, Y., Gao, J., He, Z., and Xie, S. (2016). Kernel sparse subspace clustering on symmetric positive definite manifolds. In CVPR, pages 5157-5164. IEEE Computer Society.
[91] Zhang, F., Yuan, J., Lian, D.. Xie, X.,, and Ma, W.-Y (2016). Collaborative knowledge base embedding for recommender systems. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16, page 353-362, New York, NY, USA. Association for Computing Machinery.
[92] Zhang, S., Tay, Y.. Yao, L., and Liu, Q. (2019a). Quaternion knowledge graph embeddings, In Wallach, H., Larochelle, H., Beygelzimer. A., d'Alché-Buc, F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 32, pages 2735-2745. Curran Associates, Inc.
[93] Zhang, S., Yao, L., Sun, A., and Tay, Y. (2019b). Deep learning based recommender system: A survey and new perspectives. ACM Comput. Surv., 52(1).
[94] Zhang, T., Zheng, W., Cui, Z., and Li, C. (2018a). Deep manifold-to-manifoll transforming network. In 2018 25th IEEE International Conference on Image Processing (ICIP), pages 4098- 4102.
[95] Zhang, Y. Ai, Q., Chen, X., and Wang. P. (2018b). Learning over knowledge-bard base embeddings for recommendation. CoRR, abs/ 0ass/8803.06. 803.06540.
[96] Zhao, K., Wiliem, A., Chen, S., and Lovell, B. C. (2019). Convex class model on symmetric positive definite manifolds. Image and Vision Computing, 87:57-67.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] the conclusions ($7) and in Appendix A. we highlight the computational complexity as the main limitation.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] The authors consider that the main contributions of this work are Representation Learning tools, and they do not pose societal risks by themselves.
16

---

(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Assumptions and proofs can be found on Appendix C
and D
(b) Did you include complete proofs of all theoretical results?
Yes] Assumptions and
proofs can be found Appendix Cand
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as a URL)? [Yes] Datasets are
well-known and publicly accessible. For experiment reproducibility see {6 and Ap-
pendix B The code available at battps://githie com httpf/fdelllle77//oom hot.dlolooo77/yrsppppp gyrospd
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See previous answer.
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes] For KG recommender systems and QA experiments, we report mean and standard deviation of the three best runs for all models. For Knowledge graph completion we do not report standard deviation, given that none of the previous work for this task has reported either, hence a comparison of these values is not possible.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix B
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Corresponding citations are made to the creators of PyTorch and works that introduced the different
datasets.
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either the supplemental material or as URL? [Yes] Code is available at htttt:://sg /gi thub wwt///deelooee////////ooo/ gyrospdd
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [No] Datasets and baselines models used in this work are publicly available. We cite the corresponding work where they were published for the first time.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
17


Reviews:
[{"review_id": "aNuuzBpBmEx", "reviewer": "Reviewer_rEf9", "summary": "The paper firstly defines a vector valued metric for SPD matrices and then it addresses the issue of defining operations for a set of SPD matrices (addition, matrix multiplication, scalar multiplication) using gyrocalculus. This operations allow to make computations on a set of SPD matrices, showing how the setting combined with different methods outperform the state of the art in three case studies. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positive aspects: the papar address the urgent and important problem of analyzing non-euclidean objects such as spd matrices in a correct geometrical and algebraic framework. In addition, SPD matrices are broadly used in different applications, so the potential impacts of the work are high. The paper is overall well written.\n\nNegative aspects: If I have correctly understood, I am concerns about the fact that the operations defined (Section 4) relies on a basepoint, thus they are local. This is causing two main problems: (a) how the framework behave if the points are not close to each other, i.e. is the tangent space a good approximation? (b) how the framework behave close to the singularity points of the space? If it is solved by the gyrocalculus approach, it should be better explained.\n\nMajor comments:\nSection 3. There are several works regarding the comparison between metrices on SPD matrices. Is the vector valued metric intrisic or extrinsic? Please detailed it.\n\nSection 2. How does this work relate with the work on Equivariant neural networks?\n\nSection 4. The space of SPDn have singularities as well as positive curvature. In this area of the space, the approximation of the space with tangent space is not a good choice (e.g. the origin has not one unique tangent space). Could the author clarify how the framework interact with this problem?\n\nSection 4. The gyro-vector space structure is based on the choice of a basepoint. The authors should better state where is the improvement in definition the operation in Section 4 - which relies anyway on the tangent space approach and the base point choice - over the usage of the standard operation on the tangent space (R^(n-1)n/2).\n\n\nSection 6 As a common behave in the literature, the methodology used are given for granted. It should be useful to understand how the operations well behave with respect to different models used in the section, by enlarging this introduction.\n\nSection 6.1 The example is good but describing a graph in the space of SPDn is a choice which could rise some concerns. In addition, it is not clear to me which is the graph representation. Is the graph represented as a Laplacian? if so then it is an SPD matrix, otherwise the example it is not clear to me.\n\n\nMinor comments:\nSection 3.1 why vector-valued distance is better? it is not clear to all readers and it could sound like an unusual choice. \n\nSection 3.1 why 'the VVD provides much more information than just the distance'?\n\nSection 6.1 Baselines: the results reported for the other methods are taken from the original papers. Is the setting exactly the same? It is often a risky choice in terms of reproducibility.\n\nTables in Section 6 are hard to read", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Th authors listed some limitation of their work but it should be a bit enlarged in the conclusion.", "review_text": "", "extra_scores": {"ethical_concerns": "No Ethical concerns", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "XDK7lHI8kCd", "reviewer": "Reviewer_5T8d", "summary": "This paper presents a generalized representation on the SPD manifold. The author leverages the vector-valued distance to help obtain unified metrics form on the manifold of SPD and visualize the high-dimensional data. In addition, the author also proposes to equip the SPD space with the arithmetic operations under the gyrovector space structure, which could help implement the calculations directly on the SPD manifold and maintain the essential structural information. Experiments are conducted on three different downstream tasks and demonstrate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The paper is mostly well presented. \n2. Concise description of the proposed method and detailed theoretical analysis are both provided. \n3. The experiments cover different kinds of tasks and data types.  \n\nWeaknesses:\n\n1. In Sec. 1, the author mentions that \"the representational power of SPD is not fully exploited...\". It is desirable to provide some details or examples to illustrate why/how the representational power of SPD is under exploited. \n\n2. In Sec. 3.1, on line 101, it is claimed that \"In Euclidean or hyperbolic spaces, the relative position between two points is completely  determined by their distance, ...\", it is ideal to explain this more clearly.\n\n3. In the experiment section, the under-performance of the proposed method under the rotation and reflection operations are quite significant and thus some insightful explanation will be much helpful.  \n\n5. Some typos. On line 215, \"the task of is to predict...\"\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No potential negative societal impact of this work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "WtDBE4uTUgi", "reviewer": "Reviewer_ErZJ", "summary": "In this paper, the authors studied representation learning on SPD manifold by proposing a vector-valued distance function that helps to visualize the embeddings and also generalizing the arithmetic operations to SPD manifold via the Gyrocalculus. They showed good representation power in knowledge graph completion, item recommendation, and question answering.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is well-written with nice visualizations. With the increasing attention on geometric deep learning models, the proposed arithmetic operations on SPD matrices can be a good contribution. \n\nHowever, the major concern I have is on the experiments. In all three experiments, the authors mainly consider representation learning of different objects on SPD manifold, including node entities in knowledge graph and word embeddings. These objects are primarily modelled on hyperbolic space in the literature with a clear justification, i.e., hierarchical structures that are known to exist in graph entities and between words are captured. However, it is unclear why embedding these objects onto SPD manifold is beneficial. \n\nIn addition, as mentioned by the authors, the computational cost can be high to perform the arithmetic operations compared to the hyperbolic counterparts. It would be helpful to report and compare the runtime in the experiments. Because in some applications, the improvements over the baselines are marginal. If the time required to train the models on SPD manifold take much longer, embedding onto SPD manifold is not well-justified. \n\nInstead, it would be better if the authors could compare with other SPD representation learning methods. Also, it would be interesting to see how to construct a neural network architecture on SPD manifold using the proposed arithmetic operations.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "NA", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "5eXVNKMuY-", "reviewer": "Reviewer_PQuu", "summary": "The authors propose the use of a so-called vector-valued distance to extract geometric information from the space of symmetric positive definite matrices (SPD), which is a manifold that is ubiquitous in a wide range of applications, and present the accompanying gyrovector calculus which constructs analogs of vector space operations in this curved space. These operations are then applied to a number of tasks including knowledge graph completion, item recommendation, and question answering, with favorable results compared to the corresponding Euclidean and hyperbolic spaces. ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "SPD manifolds have been used in a very wide range of applications in computer vision, medical imaging, and machine learning and the study of their geometries and research into their applications is highly active. The work under review fits naturally within this context and should be of interest to a large community of researchers. The paper is very well-written with a clear and thoughtful structure that includes high quality and enlightening illustrations as well as a clear mathematical exposition. The background literature is also quite thorough and comprehensive with the exception of a few very important omissions which I have noted below. The main contribution of the paper is in the examples considered and the experiments that demonstrate the versatility of the approach and the superior expressivity of SPD manifolds equipped with geometries based on the vector-valued distances compared to Euclidean or hyperbolic baselines. As a result, I am pleased to support acceptance of the paper subject to revision based on the comments and suggestions provided below, some of which I consider to be necessary.\n\nMajor Comments:\n\n1. On first reading I understood that the authors had introduced the gyro calculus based on the vector-valued distance as a novel mathematical contribution inspired by the hyperbolic analog. After searching the literature, it became clear that the same gyro vector calculus presented here has already been introduced and discussed in the pure mathematics literature. This needs to be made more clear in the text and relevant papers need to be cited and given due credit. Examples include references [1-3] below. Others may be appropriate as well.\n\n[1] T Abe, O Hatori, Generalized gyrovector spaces and a Mazur-Ulam theorem, published in Publ. Math. Debrecen, 87 (2015), 393--413.\n\n[2] O Hatori, Examples and Applications of Generalized Gyrovector Spaces, Results in Mathematics, 71 (2017), 295-317.\n\n[3] S Kim, Gyrovector Spaces on the Open Convex Cone of Positive Definite Matrices, Mathematics Interdisciplinary Research, (2016).\n\n2. Similar to the above comment, the particular vector-valued distance (equation (1)) has been noted in several works as being a fundamental geometric object for a pair of SPD matrices, which very naturally generates a family of Finsler structures that include the Fisher-Rao (affine-invariant) Riemannian metric and Thompson’s part metric on the positive definite cone as special cases. Relevant papers include the standard reference [4] as well as more recent work such as [5], which studies Thompson geometry in this framework.\n\n[4] R Bhatia, On the exponential metric increasing property, Linear Algebra and its Applications, Volume 375, (2003).\n\n[5] C Mostajeran, C Grussler, R Sepulchre, Geometric Matrix Midranges, SIAM Journal on Matrix Analysis and Applications, (2020).\n\n3. Could the authors elaborate on the issues they faced in using Riemannian optimization, which led them to use tangent space optimization instead? Is an explanation available for these observations?\n\n4. Standard references for Riemannian optimization on manifolds include [6] and [7], which are worth citing on line 201.\n\n[6] P.-A Absil, R Mahony, R Sepulchre, Optimization Algorithms on Matrix Manifolds (2008).\n\n[7] N Boumal, B Mishra, P.-A Absil, R Sepulchre, Manopt, a Matlab Toolbox for Optimization on Manifolds, Journal of Machine Learning Research, (2014).\n\n5. The observed superiority of SPD models compared to hyperbolic models in the experiments is interesting and deserves a deeper discussion in my view. Are particular aspects of the geometry relevant for each of the specific examples/tasks considered, or is the geometry useful in the same way in each case?\n\n6. Perhaps relevant to point 5 above, the authors note on lines 75-77 the contents of SPDn including the number of Euclidean subspaces, hyperbolic subspaces, and hyperbolic planes. Please include a citation for this statement.\n\n7. The authors briefly note the computational limitations of their approach towards the end of the paper. I think this should be expanded and discussed in more detail. Particularly, the distance and gyro calculus operations represent quite a significant computational cost in large dimensions due to eigenvalue computations and matrix operations, which may be missed by a quick glance from a casual reader who may get the wrong idea from the word ‘vector’ in the title!\n\nMinor comments:\n\n1. On line 53, the authors state that the Log-Euclidean geometry (Euclidean-based tools in tangent space) ‘distorts the manifold structure in regions far from the tangent space affecting the performance.’ While I understand what the authors intend to communicate here, it is not the ‘manifold geometry’ that is distorted, since the Log-Euclidean geometry is perfectly valid as a metric structure (Riemannian indeed) on this manifold. It just happens not to be the ‘right geometry’ in many cases, so it is the metric structure that is distorted for many practical applications and not the manifold nature of the space.\n\n2. Remove ‘of’ on line 215.  ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to use vector-valued distances (and calculus) on the manifold of symmetric positive definite matrices to use in machine learning applications.  As all reviewers agree, the paper contains novel insights and interesting applications (with good analysis). While some theoretical insights are well-known in some previous works, the paper does a great job of putting things cleanly for the NeurIPS community.

Decision: Accept (Spotlight)