Riemannian TransE: Multi-relational Graph Embedding in Non-Euclidean Space

1 INTRODUCTION . 1.1 BACKGROUND . Multi-relational graphs , such as social networks and knowledge bases , have a variety of applications , and embedding methods for these graphs are particularly important for these applications . For instance , multi-relational graph embedding has been applied to social network analysis ( KrohnGrimberghe et al. , 2012 ) and knowledge base completion ( Bordes et al. , 2013 ) . A multi-relational graph consists of entities V , a set R of relation types , and a collection of real data triples , where each triple ( h , r , t ) ∈ V × R × V represents some relation r ∈ R between a head entity h ∈ V and a tail entity t ∈ V . Embedding a multi-relational graph refers to a map from the entity and the relation set to some space . Mathematical operations in this space enable many tasks , including clustering of entities and completion , prediction , or denoising of triples . Indeed , completion tasks for knowledge bases attract considerable attention , because knowledge bases are known to be far from complete , as discussed in ( West et al. , 2014 ) ( Krompaß et al. , 2015 ) . Multi-relational graph embedding can help its completion and improve the performance of applications that use the graph . This is the reason why much work focuses on multi-relational graph embedding . Figure 1 shows an example of a multi-relational graph and a completion task . In multi-relational graph embedding , reducing the number of parameters is an important problem in the era of big data . Many parameters are needed with tensor-factorization-based methods , such as Bayesian clustered tensor factorization ( BCTF ) ( Sutskever et al. , 2009 ) , RESCAL ( Nickel et al. , 2011 ) , and a neural tensor network ( NTN ) ( Socher et al. , 2013 ) , where each relation has a dense matrix or tensors ( O ( D2 ) or more parameters , where D is dimensionality of the space ) . Thus , TransE ( Bordes et al. , 2013 ) was proposed to reduce the number of parameters , to overcome this problem . In TransE , each entity is mapped to a point in Euclidean space and each relation is no more than a vector addition ( O ( D ) parameters ) , rather than a matrix operation . The successors to TransE , TransH ( Wang et al. , 2014 ) and TransD ( Ji et al. , 2016 ) , also use only a small number of parameters . Some methods succeeded in reducing parameters using diagonal matrices instead of dense matrices : e.g . DISTMULT ( Yang et al. , 2015 ) , ComplEx ( Trouillon et al. , 2016 ) , HolE ( through the Fourier transform ) ( Nickel et al. , 2016 ) , and ANALOGY ( Liu et al. , 2017 ) . In these methods , all relations share one space for embedding , but each relation uses its own dissimilarity criterion . The success of these methods implies that one common space underlies whole data , and each relation can be regarded as a dissimilarity criterion in the space . Whereas these methods use distances or inner products in Euclidean space as dissimilarity criteria , recent work has shown that using non-Euclidean space can further reduce the number of parameters . One typical example of this is Poincaré Embedding ( Nickel & Kiela , 2017 ) for hierarchical data , where a hyperbolic space is used as a space for embedding . Here , the tree structure of hierarchical data has good compatibility with the exponential growth of hyperbolic space . Recall the circumference with radius R is given by 2π sinhR ( ≈ 2π expR ) in a hyperbolic plane . As a result , Poincaré embedding achieved good graph completion accuracy , even in low dimensionality such as 5 or 10 . On the other hand , spheres ( circumference : 2π sinR ) are compatible with cyclic structures . Since Poincaré embedding , several methods have been proposed for single-relational graph embedding in non-Euclidean space ( e.g . ( Ganea et al. , 2018b ) , ( Nickel & Kiela , 2018 ) ) and shown good results . The success of these methods suggests that the appropriate choice of a manifold ( i.e. , space ) can retain low dimensionality , although these methods are limited to single-relational graph embedding . According to the success of the TransE and its derivation and Poincaré embedding , it is reasonable in multi-relational graph embedding to assume the existence of a single structure compatible with a non-Euclidean manifold . For example , we can consider a single tree-like structure , which contains multiple hierarchical structures , where root selection gives multiple hierarchical structures from a single tree , which is compatible with hyperbolic spaces ( See Figure 2 ) . Therefore , embedding in a single shared non-Euclidean manifold with multiple dissimilarity criteria used in TransE is promising . Taking Poincaré embedding ’ s success with low dimensionality into consideration , this method should work well ( e.g. , in graph completion tasks ) with small number of parameters . This is the main idea of this paper . 1.2 CONTRIBUTIONS . We propose a novel method , called Riemannian TransE , for multi-relation graph embedding using a non-Euclidean manifold . In Riemannian TransE , the relations share one non-Euclidean space and the entities are mapped to the space , whereas each relation has its own dissimilarity criterion based on the distance in the space . Specifically , the dissimilarity criteria in Riemannian TransE are similar to those in TransE ( Bordes et al. , 2013 ) based on vector addition , which is known to be effective . Unfortunately , we can not straightforwardly use TransE ’ s dissimilarity criteria . This is due to non-existence of a parallel vector field ( See Figure 4 ) , which is implicitly but essentially used in “ vector addition. ” However , the parallel condition is not essential in TransE ’ s idea . For example , hierarchical bottom to top relations should be regarded as attraction to the top in the hierarchy , which is not parallel but has an attractive point . Moreover , parallel vector fields can be regarded as a vector field attracted to a point at infinity . Therefore , we replace parallel vector fields in TransE by vector fields with an attractive point that are well-defined in Riemannian manifolds , and as a result , we obtain Riemannian TransE . Advantages of non-Euclidean spaces enable our Riemannian TransE to achieve good performance ( e.g . in graph completion ) with low-dimensional parameters . Riemannian TransE further exploits the advantages of TransE : that is , the method needs only O ( D ) parameters for each relation . Numerical experiments on graph completion tasks show that with an appropriate choice of manifold , our method can improve the performance of multi-relational graph embedding with few parameters . 2 RELATED WORK . 2.1 MULTI-RELATIONAL GRAPH EMBEDDING . Let V and R denote the entities and relations in a multi-relational graph , and let T ⊂ V × R × V denote the triples in the graph . Multi-relational graph embedding refers to a pair of maps from V andR intoMe andMr , respectively . Particularly , learning multi-relational graph embedding refers to obtaining an appropriate pair of maps v 7→ pv ( v ∈ V , pv ∈Me ) and r 7→ wr ( r ∈ R , wr ∈Mr ) from the triples T . In this paper , we call pv the planet of entity v , wr the launcher of relation r , and Me and Mr the planet manifold and launcher manifold , respectively . The quality of embedding is measured through a score function f : ( Me ×Me ) × Mr → R , which is designed by each method . Embedding is learned such that the value score function f ( ph , pt ; wr ) will be low when ph , pt ; wr ∈ T and high when ph , pt ; wr /∈ T . For specific loss functions designed from the score function , see Subsection 2.3 . We interpret the score function of multi-relational graph embedding as dissimilarity in a manifold , which we call a satellite manifoldMs . We rewrite the score function f in multi-relational graph embedding using two maps H , T : Me ×Mr →Ms and the dissimilarity measure function D : Ms ×Ms → R as follows : f ( ph , pt ; wr ) : = D ( sHh ; r , s T t ; r ) , where sHh ; r = H ( ph ; wr ) , s T t ; r = T ( pt ; wr ) . ( 1 ) We call H and T the head and tail launch map , respectively , and call sHv ; r and s T v ; r the head and tail satellite of entity v ( or of planet pv ) with respect to relation r. The idea of this formulation is embedding in one shared space with multiple dissimilarity criteria . Specifically , each entity has only one planet and their satellite pairs give multiple dissimilarity criteria , each of which corresponds to a relation . In other words , all of the relations shares one space and the planets in it , and the differences among the relations are reduced to the difference of their launcher maps and the satellites given by them . We regard the planets as the embeddings of the entities , whereas dissimilarity between entities with respect to a relation is evaluated through their satellites which correspond to the relation . A simple example of this is TransE ( Bordes et al. , 2013 ) , where all of the planets , satellites , and launchers share the same Euclidean space , i.e.Me =Ms =Mr = RD , the launch maps are given by vector addition as H ( p ; w ) = p +w and T ( p ; w ) = p , and the distance in a norm space— i.e . the norm of the difference—is used as a dissimilarity criterion i.e . D ( sH , sT ) = ∥∥sT − sH∥∥ ( the L1 or L2 norm is often used in practice ) . See Figure 5 ( left ) . As Nguyen ( 2017 ) suggested , one can associate the idea of representing relations as vector additions with the fact that we can find a relation through a substraction operator in Word2Vec Mikolov et al . ( 2013 ) . That is , we can find relations such as pFrance−pParis ≈ pItaly−pRome in Word2Vec . As explained above , TransE is based on the distance between satellites , and each satellite is given by simple vector addition . Regardless of this simplicity , the performance of TransE has been exemplified in review papers ( Nickel et al. , 2016 ) ( Nguyen , 2017 ) . Indeed , the addition operation in a linear space is essential in the launcher map , and hence TransE can easily be extended to a Lie group , which is a manifold equipped with an addition operator , as suggested in Ebisu & Ichise ( 2017 ) . Some methods , such as TransH ( Wang et al. , 2014 ) , TransR ( Lin et al. , 2015 ) , and TransD ( Ji et al. , 2016 ) , also use a norm in linear space as a dissimilarity measure , integrating a linear map into a latent space . Another simple example is RESCAL ( Nickel et al. , 2011 ) , which uses the negative inner product as a dissimilarity measure . In RESCAL , the launcher of relation r is a matrixW ∈Mr = RD×D , the launch maps are given by a linear map , i.e . H ( p ; ( W , w ) ) = Wp and T ( p ; ( W , w ) ) = p , and the dissimilarity measure is the negative inner product D ( sH , sT ) = − ( sH ) > sT. Other methods are also based on the ( negative ) inner product dissimilarity : e.g. , DISTMULT ( Yang et al. , 2015 ) , ComplEx ( Trouillon et al. , 2016 ) , HolE ( through the Fourier transform ) ( Nickel et al. , 2016 ) , and ANALOGY ( Liu et al. , 2017 ) . Table 1 shows score functions of these methods . Whereas some methods are based on a neural network ( e.g. , the neural tensor network ( Socher et al. , 2013 ) and ConvE ( Dettmers et al. , 2017 ) ) , their score function consists of linear operations and element-wise nonlinear functions .

Summary: In this paper, authors focus on the problem of efficiently embedding Knowledge Graphs in low-dimensional embedding spaces, a task where models are commonly evaluated via downstream link prediction and triple classification tasks. The proposed model - Riemannian TransE, based on TransE [Bordes et al. 2013] - maps entities to points in a non-Euclidean space, by minimising a loss based on the geodesic distance in such space. This paper is especially interesting, since extends previous approaches - such as Poincare embeddings - to the multi-relational setting. Results look promising on WN11 and FB13, but authors mention results on the more commonly used WN18 and FB15k are less accurate than those obtained by the baselines (without reporting them). It is worth mentioning that WN18 and FB15k were found to be solvable by very simple baselines (e.g. see [1]). Furthermore, authors do not report any finding on the geometry of the learned spaces.