Lorentzian Distance Learning

1 INTRODUCTION . Generalizations of Euclidean space are important forms of data representation in machine learning . For instance , kernel methods rely on Hilbert spaces that possess the structure of the inner product and can therefore be used to compare examples . The properties of such spaces are well-known and closed-form relations are often exploited to obtain efficient , scalable , and interpretable training algorithms . While representing examples in a Euclidean space is appropriate to compare lengths and angles , non-Euclidean representations are useful when the task requires specific structure . A common and natural non-Euclidean representation space is the spherical model ( e.g . ( Wang et al. , 2017 ) ) where the data lies on a unit hypersphere Sd = { x ∈ Rd : ‖x‖ = 1 } and angles are compared with the cosine similarity function . Recently , some machine learning approaches ( Nickel & Kiela , 2017 ; 2018 ; Ganea et al. , 2018 ) have considered representing hierarchical datasets with the hyperbolic model . The motivation is that any finite tree can be mapped into a finite hyperbolic space while approximately preserving distances ( Gromov , 1987 ) . Since hierarchies can be formulated as trees , hyperbolic spaces can be used to represent hierarchically structured data where the high-level nodes of the hierarchy are represented close to the origin ( i.e . with small Euclidean norm ) whereas leaves are further away from the origin . Since their first formulation in the early nineteenth century by Lobachevsky and Bolyai , hyperbolic spaces have been used in many domains . In particular , they became popular in mathematics ( e.g . space theory and differential geometry ( Ratcliffe , 2006 ) ) , and physics when Varicak ( 1908 ) discovered that special relativity theory ( Einstein , 1905 ) had a natural interpretation in hyperbolic geometry . Various hyperbolic geometries and related distances have been studied since then . Among them are the Poincaré metric , the Lorentzian distance ( Ratcliffe , 2006 ) , and the gyrodistance ( Ungar , 2010 ) . In the case of hierarchical datasets , machine learning approaches that learn hyperbolic representations designed to preserve the hierarchical similarity order have typically employed the Poincaré metric . Usually , the optimization problem is formulated so that the representation of a node in a hierarchy should be closer to the representation of its children and other descendants than to any other node in the hierarchy . Based on Gromov ( 1987 ) , the Poincaré metric is a sensible dissimilarity function as it satisfies all the properties of a distance metric and is thus natural to interpret . In this paper , we propose to exploit the squared Lorentzian distance instead of the usual Poincaré distance . The squared Lorentzian distance is linear in the hyperbolic representations and differentiable on the whole domain , which avoids numerical instabilities and exploding gradients . Moreover , the order of the Poincaré distances between pairs of points is the same as the order of the ( squared ) Lorentzian distances in the unit hyperboloid model . We may therefore directly adapt existing approaches by replacing Poincaré distances with squared Lorentzian distances . The main property of interest is that the center of mass w.r.t . the squared Lorentzian distance can be written in closed form and is then easy to interpret . In particular , we show that the Euclidean norm of the center of mass of a set of different points decreases as the curvature of the exploited hyperboloid model1 decreases and tends to −∞ . This property makes it ideal to represent hierarchical structures since we would like the root of a subtree to minimize the distance to all of its descendants ( i.e . it should be their centroid w.r.t . to the chosen distance function ) and have smaller Euclidean norm than them . Our contributions are both analytic and experimental . We study properties of the chosen dissimilarity function and explain why they make it appropriate for the task of interest . In particular , we show that the Euclidean norm of the center of mass in the Poincaré ball representation decreases as the curvature −1/β ∈ R− of the hyperboloid model decreases ( i.e . as β > 0 decreases ) . Curvature can then be used as a hyperparameter to implicitly enforce the root of a subtree to have smaller Euclidean norm than the other nodes . We experimentally show on different problems that exploiting the squared Lorentzian distance improves retrieval and classification performance . 2 BACKGROUND . In this section , we provide some technical background about hyperbolic geometry and introduce relevant notation . The interested reader may refer to ( Ratcliffe , 2006 ) for more detail . 2.1 NOTATION AND DEFINITIONS . To simplify the notation , we consider that vectors are row vectors and ‖ · ‖ is the ` 2-norm . In the following , we consider three important spaces . Poincaré ball : The Poincaré ball Pd is defined as the set of d-dimensional vectors with Euclidean norm smaller than 1 ( i.e . Pd = { x ∈ Rd : ‖x‖ < 1 } ) . Its associated distance is the Poincaré distance metric defined in Eq . ( 3 ) . Hyperboloid model : We consider some specific hyperboloid modelsHd , β defined as follows : Hd , β : = { a = ( a0 , · · · , ad ) ∈ Rd+1 : ‖a‖2L = −β , a0 > 0 , β > 0 } ( 1 ) where ‖a‖2L = 〈a , a〉L is the squared Lorentzian norm of a . The squared Lorentzian norm is derived from the Lorentzian inner product defined as : ∀a = ( a0 , · · · , ad ) ∈ Hd , β , b = ( b0 , · · · , bd ) ∈ Hd , β , 〈a , b〉L : = −a0b0 + d∑ i=1 aibi ≤ −β ( 2 ) It is worth noting that 〈a , b〉L = −β iff a = b . Otherwise , 〈a , b〉L < −β for all pair ( a , b ) ∈ ( Hd , β ) 2 . Vectors inHd , β are a subset of positive time-like vectors2 . Moreover , every vector a ∈ Hd , β satisfies a0 = √ β + ∑d i=1 a 2 i . We note Hd : = Hd,1 the space obtained when β = 1 ; it is usually called the ( positive sheet of the ) unit hyperboloid model and is the main hyperboloid model considered in the literature . Model space : Finally , we noteFd ⊆ Rd the output space of our model ( e.g . the output representation of some linear model or neural network ) . In the following , we consider that Fd = Rd . 2.2 OPTIMIZING THE POINCARÉ DISTANCE METRIC . Most methods that compare hyperbolic representations ( Nickel & Kiela , 2017 ; 2018 ; Ganea et al. , 2018 ; Gulcehre et al. , 2018 ) consider the Poincaré distance metric defined as : ∀c ∈ Pd , d ∈ Pd , dP ( c , d ) = arcosh ( 1 + 2 ‖c− d‖2 ( 1− ‖c‖2 ) ( 1− ‖d‖2 ) ) ( 3 ) 1The curvature of the considered hyperboloid models is negative and is -1 for the unit hyperboloid model . 2A vector a that satisfies 〈a , a〉L < 0 is called time-like and it is called positive iff a0 > 0. which satisfies all the properties of a distance metric and is therefore natural to interpret . Direct optimization of problems using the distance formulation in Eq . ( 3 ) is numerically instable for two main reasons ( see for instance ( Nickel & Kiela , 2018 ) or ( Ganea et al. , 2018 , Section 4 ) ) . First , the denominator depends on the norm of examples , so optimizing over c and d when either of their norms is close to 1 leads to numerical instability . Second , elements have to be re-projected onto the Poincaré ball at each iteration with a fixed maximum norm . To increase the numerical stability of their solver , Nickel & Kiela ( 2018 ) propose to use an equivalent formulation of dP in the unit hyperboloid model . They use the fact that there exists an invertible mapping h : Hd , β → Pd defined as : ∀a = ( a0 , · · · , ad ) ∈ Hd , β , h ( a ) : = 1 1 + √ 1 + ∑d i=1 a 2 i ( a1 , · · · , ad ) ∈ Pd ( 4 ) When β = 1 , we have the following equivalence : ∀a ∈ Hd , b ∈ Hd , dH ( a , b ) = dP ( h ( a ) , h ( b ) ) = arcosh ( −〈a , b〉L ) ( 5 ) which corresponds to the ( Lorentzian ) angle between a and b . It is shown in ( Nickel & Kiela , 2018 ) that optimizing the formulation in Eq . ( 5 ) is more stable numerically . Nonetheless , the gradient of Eq . ( 5 ) still tends to +∞ as dH ( a , b ) tends to 0 . Interestingly , one can observe from Eq . ( 5 ) that preserving the order of Poincaré distances is equivalent to preserving the order of the opposite of Lorentzian inner products since the arcosh function is monotonically increasing on its domain [ 1 , +∞ ) . In the same way as kernel methods that consider inner products in Hilbert spaces as similarity measures , we consider in this paper the Lorentzian inner product . 3 PROPOSED METHOD . We present in this section the ( squared ) Lorentzian distance function dL ( Ratcliffe , 2006 ) and mention the properties that make it appropriate for our task . We give the formulation of its center of mass and observe that its Euclidean norm depends on the curvature −1/β which can then be used as a hyper-parameter of the model to implicitly enforce centroids to have small Euclidean norm . We also discuss optimization details . 3.1 LORENTZIAN DISTANCE AND MAPPINGS . The squared Lorentzian distance ( Ratcliffe , 2006 ) is defined as : ∀a ∈ Hd , β , b ∈ Hd , β , d2L ( a , b ) = ‖a−b‖2L = ‖a‖2L+ ‖b‖2L− 2〈a , b〉L = −2β− 2〈a , b〉L ( 6 ) It satisfies all the axioms of a distance metric except the triangle inequality ( i.e . non-negativity , symmetry , identity of indiscernibles ) . One interesting property is that d2L is linear in both a and b . Moreover , when β = 1 , it defines the hyperbolic length of a curve inHd ( Ratcliffe , 2006 ) . Mapping : Current hyperbolic machine learning models re-project at each iteration their learned representations onto the Poincaré ball ( Nickel & Kiela , 2017 ) or use some exponential map ( Nickel & Kiela , 2018 ; Gulcehre et al. , 2018 ) to directly optimize on the unit hyperboloid model ( i.e . when β = 1 ) . Since our approach does not necessarily consider that β = 1 , we consider the invertible mapping gβ : Fd → Hd , β defined as : ∀f = ( f1 , · · · , fd ) ∈ Fd , gβ ( f ) : = ( √ ‖f‖2 + β , f1 , · · · , fd ) ∈ Hd , β ( 7 ) As mentioned in Section 2.1 , Fd is the space of our model ; it is Rd in practice . To be fair with baselines , the examples in Fd in our experiments are non-parametric embeddings . Fd may also be the output space of some parametric model such as a neural network . We compare two examples f1 ∈ Fd and f2 ∈ Fd with d2L by calculating : ∀f1 ∈ Fd , f2 ∈ Fd , d2L ( gβ ( f1 ) , gβ ( f2 ) ) = −2β − 2〈gβ ( f1 ) , gβ ( f2 ) 〉L ( 8 ) = −2 [ β + 〈f1 , f2〉 − √ ‖f1‖2 + β √ ‖f2‖2 + β ] ( 9 ) Preserved order of Euclidean norms : Although examples are compared with d2L in the hyperbolic space Hd , β where all the points have the same Lorentzian norm , it is worth noting that the order of the Euclidean norms of examples is preserved along the three spaces Fd , Hd , β and Pd with the mappings gβ and h. The preservation with gβ is straightforward : ∀f1 , f2 ∈ Fd , ‖f1‖ < ‖f2‖ ⇐⇒√ 2‖f1‖2 + β = ‖gβ ( f1 ) ‖ < ‖gβ ( f2 ) ‖ . The proof of the following theorem is given in the appendix : Theorem 3.1 ( Order of Euclidean norms ) . The following order of norms is preserved with h ◦ g : ∀a ∈ Fd , b ∈ Fd , ‖a‖ < ‖b‖ ⇐⇒ ‖h ( gβ ( a ) ) ‖ < ‖h ( gβ ( b ) ) ‖ ( 10 ) In conclusion , the Euclidean norms of examples can be compared equivalently in any space . This is particularly useful if we want to study the Euclidean norm of centroids w.r.t . examples , and also understand the significance of the following section .

Summary: Learning embeddings of graphs in hyperbolic space have become popular and yielded promising results. A core reason for that is learning hierarchical representations of the graphs is easier in hyperbolic space due to the curvature and the geometrical properties of the hyperbolic space. Similar to [1, 2], this paper uses Lorentzian model of the hyperbolic space in order to learn embeddings of the graph. The main difference of the proposed approach in this paper is that  they come up with a closed-form solution such that each node representation close to the centroid of their descendants. A curious property of the equation for the centroids proposed to learn the embeddings of each node also encodes information related to the specificity in the Euclidean norm of the centroid. Also this paper introduces two additional hyperparameters. Beta hyperparameter is selected to control the curvature of the space. Depending on the task the optimal curvature can be tuned to be a different value. This also ties closely with the de-Sitter spaces. Authors provide results on different graph embedding benchmark tasks. The paper claims that, an advantage of the proposed approach is that the embedding of the model can be tuned with regular SGD without needing to use Riemannian optimization techniques.