Vector-valued Distance and Gyrocalculus on the Space of Symmetric Positive Definite Matrices

1 Introduction . Symmetric Positive Definite ( SPD ) matrices have been applied in many tasks in computer vision such as pedestrian detection [ 75 , 79 ] , action [ 36 , 51 , 60 ] or face recognition [ 41 , 42 ] , object [ 43 , 90 ] and image set classification [ 86 ] , visual tracking [ 87 ] , and medical imaging analysis [ 5 , 65 ] among others . They have been used to capture statistical notions ( Gaussian distributions [ 68 ] , covariance [ 78 ] ) , while respecting the Riemannian geometry of the underlying SPD manifold , which offers a convenient trade-off between structural richness and computational tractability [ 27 ] . Previous work has applied approximation methods that locally flatten the manifold by projecting it to its tangent space [ 22 , 83 ] , or by embedding the manifold into higher dimensional Hilbert spaces [ 35 , 90 ] . These methods face problems such as distortion of the geometrical structure of the manifold and other known concerns with regard to high-dimensional spaces [ 29 ] . To overcome these issues , several distances on SPD manifolds have been proposed , such as the Affine Invariant metric [ 65 ] , the Stein metric [ 71 ] , the Bures–Wasserstein metric [ 13 ] or the Log-Euclidean metric [ 5 , 6 ] , with their respective geometric properties . However , the representational power of SPD is not fully exploited in many cases [ 6 , 65 ] . At the same time , it is hard to translate operations into their non-Euclidean domain given the lack of closed-form expressions . There has been a growing need to generalize basic operations , such as addition , rotation , reflection or scalar multiplication , to their Riemannian geometric counterparts to leverage this structure in the context of Geometric Deep Learning [ 19 ] . SPD manifolds have a rich geometry that contains both Euclidean and hyperbolic subspaces . Thus , embeddings into SPD manifolds are beneficial , since they can accommodate hierarchical structure in data sets in the hyperbolic subspaces while at the same time represent Euclidean features . This makes them more versatile than using only hyperbolic or Euclidean spaces , and in fact , their different submanifold geometries can be used to identify and disentangle such substructures in graphs . ∗Correspondence to federico.lopez @ h-its.org 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this work , we introduce the vector-valued distance function to exploit the full representation power of SPD ( §3.1 ) . While in Euclidean or hyperbolic space the relative position between two points is completely captured by their distance ( and this is the only invariant ) , in SPD this invariant is a vector , encoding much more information than a single scalar . This vector reflects the higher expressivity of SPD due to its richer geometry encompassing Euclidean as well as hyperbolic spaces . We develop algorithms using the vector-valued distance and showcase two main advantages : its versatility to implement universal models , and its use in explaining and visualizing what the model has learned . Furthermore , we bridge the gap between Euclidean and SPD geometry by developing gyrocalculus in SPD ( §4 ) , which yields closed-form expressions of arithmetic operations , such as addition , scalar multiplication and matrix scaling . This provides means to translate previously implemented ideas in different metric spaces to their analog notions in SPD . These arithmetic operations are also useful to adapt neural network architectures to SPD manifolds . We showcase this on knowledge graph completion , item recommendation , and question answering . In the experiments , the proposed SPD models outperform their equivalents in Euclidean and hyperbolic space ( §6 ) . These results reflect the superior expressivity of SPD , and show the versatility of the approach and ease of integration with downstream tasks . The vector-valued distance allows us to develop a new tool for the analysis of the structural properties of the learned representations . With this tool , we visualize high-dimensional SPD embeddings , providing better explainability on what the models learn ( §6.4 ) . We show that the knowledge graph models are capable of disentangling and clustering positive triples from negative ones . 2 Related Work . Symmetric positive definite matrices are not new in the Machine Learning literature . They have been used in a plethora of applications [ 5 , 29 , 36 , 40–43 , 51 , 60 , 65 , 68 , 75 , 78 , 79 , 86 , 87 , 90 ] ) , although not always respecting the intrinsic structure or the positive definiteness constraint [ 22 , 31 , 35 , 83 , 90 ] . The alternative has been to map manifold points onto a tangent space and employ Euclidean-based tools . Unfortunately , this mapping distorts the metric structure in regions far from the origin of the tangent space affecting the performance [ 44 , 96 ] . Previous work has proposed alternatives to the basic neural building blocks respecting the geometry of the space . For example , transformation layers [ 29 , 33 , 40 ] , alternate convolutional layers based on SPDs [ 94 ] and Riemannian means [ 23 ] , or appended after the convolution [ 21 ] , recurrent models [ 24 ] , projections onto Euclidean spaces [ 50 , 57 ] and batch normalization [ 20 ] . Our work follows this line , providing explicit formulas for translating Euclidean arithmetic notions into SPDs . Our general view , using the vector-valued distance function , allows us to treat Riemannian and Finsler metrics on SPD in a unified framework . Finsler metrics have previously been applied in compressed sensing [ 30 ] , information geometry [ 69 ] , for clustering categorical distributions [ 63 ] , and in robotics [ 67 ] . With regard to optimization , matrix backpropagation techniques have been explored [ 2 , 17 , 43 ] , with some of them accounting for different Riemannian geometries [ 20 , 40 ] . Nonetheless , we opt for tangent space optimization [ 26 ] by exploiting the explicit formulations of the exponential and logarithmic map . 3 The Space SPDn The space SPDn is a Riemannian manifold of non-positive curvature of n ( n + 1 ) /2 dimensions . Points in SPDn are positive definite real symmetric n× n matrices , with the identity matrix I being a natural basepoint . The tangent space to any point of SPDn can be identified with the vector space Sn of all real symmetric n× n matrices . SPDn contains n-dimensional Euclidean subspaces , ( n − 1 ) -dimensional hyperbolic subspaces as well as products of bn2 c hyperbolic planes ( see Helgason [ 39 ] for an in-depth introduction ) . In Figure 1 we visualize the smallest nontrivial example . SPD2 identifies with the inside of a cone in R3 , cut out by requiring both eigenvalues of the matrix ( x yy z ) to be positive . It carries the product geometry of the hyperbolic plane times a line . Exponential and logarithmic maps : The exponential map , exp : Sn → SPDn , is a homeomorphism which links the Euclidean geometry of the tangent space Sn and the curved geometry of SPDn . Its inverse is the logarithm map , log : SPDn → Sn . This pair of functions give diffeomorphisms that allows one to freely move between ’ tangent space coordinates ’ or the original ’ manifold coordinates ’ . We apply both maps based at I ∈ SPDn . The reason for this is that while mathematically any two points on SPDn are equivalent , and we could obtain a different concrete expression for any other choice of basepoint B ∈ SPDn , the resulting formulas would be more complicated , and thus I is the best choice from a computational point of view . We prove this in Appendix C.3 . Symmetries : The prototypical symmetries of SPDn are parameterized by elements of GL ( n ; R ) : any invertible matrix M defines the symmetry P 7→MPMT acting on all points P ∈ SPDn . Thus many geometric transformations SPDn can be completed using standard optimized matrix algorithms as opposed to custom-built procedures . See Appendix C.2 . for a brief review of these symmetries . Among these , we may find SPDn-generalizations of familiar symmetries of Euclidean geometry . When also M is an element of SPDn , the symmetry P 7→ MPMT is a generalization of an Euclidean translation , fixing no points of SPDn . When M is an orthogonal matrix , the symmetry P 7→MPMT is conjugation by M , and thus fixes the basepoint I = MIMT = MM−1 = I . We think of elements fixing the basepoint as being SPDn-rotations or SPDn-reflections , when the matrix M is a familiar rotation or reflection ( see Figure 2 ) . The Euclidean symmetry of reflecting in a point also has a natural generalization to SPDn . Euclidean reflection in the origin is given by p 7→ −p ; and its SPDn-analog , reflection in the basepoint I , is matrix inversion P 7→ P−1 . The general SPDn-reflection in a point Q ∈ SPDn is a conjugate of this by an SPDn translation , given by P 7→ QP−1Q . 3.1 Vector-valued Distance Function . In Euclidean or hyperbolic spaces , the relative position between two points is completely determined by their distance , which is given by a scalar . For the space SPDn , it is determined by a vector . The VVD vector : To assign this vector in SPD we introduce the vector-valued distance ( VVD ) function dvv : SPDn×SPDn → Rn . For two points P , Q ∈ SPDn , the VVD is defined as : dvv ( P , Q ) = log ( λ1 ( P −1Q ) , . . . , λn ( P −1Q ) ) ( 1 ) where λ1 ( P−1Q ) ≥ . . . ≥ λn ( P−1Q ) are the eigenvalues of P−1Q sorted in descending order . This vector is an invariant of the relative position of two points up to isometry . This means that in SPDn , only if the VVD between two points A and B is the vector v ∈ Rn , and the VVD between P and Q is also v , then there exists an isometry mapping A to P and B to Q . Thus , we can recover completely the relative position of two points in SPDn from this vector . For example , the Riemannian metric is obtained by using the standard l2 norm on the VVD vector . This is : dR ( P , Q ) = ||dvv ( P , Q ) ||2 . See [ 46 ] §2.6 and Appendix C.4 for a review of VVDs in symmetric spaces . Finsler metrics : Any norm on Rn that is invariant under permutation of the entries induces a metric on SPDn . Moreover , SPDn do not only support a Riemannian metric , but also Finsler metrics , a whole family of distances with the same symmetry group ( group of isometries ) . These metrics are of special importance since distance minimizing geodesics are not necessarily unique in Finsler geometry . Two different paths can have the same minimal length . This is particularly valuable when embedding graphs in SPDn , since in graphs there are generally several shortest paths . We obtain the Finsler metrics F1 or F∞ by taking the respective ` 1 or ` ∞ norms of the VVD in Rn ( see Figure 3 ) . See Planche [ 66 ] and Appendix C.6 for a review of the theory of Finsler metrics , [ 12 ] for the study of some Finsler metrics on SPDn , and [ 52 ] for applications of Finsler metrics on symmetric spaces in representation learning . Advantages of VVD : The proposed metric learning framework based on the VVD offers several advantages . First , a single model can be run with different metrics , according to the chosen norm . The VVD contains the full information of the Riemannian distance and of all invariant Finsler distances , hence we can easily recover the Riemannian metric and extend the approach to many other alternatives ( in Appendix C.7 , we detail how the VVD generalizes other SPDn metrics ) . Second , the VVD provides much more information than just the distance , and can be used to analyze the learned representation in SPDn , independent of the choice of the metric . Out of the VVD between two points , one can immediately read the regularity of the unique geodesics joining these two points . Geodesics in SPDn have different regularity , which is related with the number of maximal Euclidean subspaces that contain the geodesic . The Riemannian or Finsler distances can not distinguish the differences between these geodesics of different regularity , but the VVD function can . Third , the VVD function can be leveraged as a tool to visualize and analyze the learned high-dimensional representations ( see §6.4 ) .

Summary: This paper presents a generalized representation on the SPD manifold. The author leverages the vector-valued distance to help obtain unified metrics form on the manifold of SPD and visualize the high-dimensional data. In addition, the author also proposes to equip the SPD space with the arithmetic operations under the gyrovector space structure, which could help implement the calculations directly on the SPD manifold and maintain the essential structural information. Experiments are conducted on three different downstream tasks and demonstrate the effectiveness of the proposed method.