Riemannian TransE: Multi-relational Graph Embedding in Non-Euclidean Space

1 INTRODUCTION . 1.1 BACKGROUND . Multi-relational graphs , such as social networks and knowledge bases , have a variety of applications , and embedding methods for these graphs are particularly important for these applications . For instance , multi-relational graph embedding has been applied to social network analysis ( KrohnGrimberghe et al. , 2012 ) and knowledge base completion ( Bordes et al. , 2013 ) . A multi-relational graph consists of entities V , a set R of relation types , and a collection of real data triples , where each triple ( h , r , t ) ∈ V × R × V represents some relation r ∈ R between a head entity h ∈ V and a tail entity t ∈ V . Embedding a multi-relational graph refers to a map from the entity and the relation set to some space . Mathematical operations in this space enable many tasks , including clustering of entities and completion , prediction , or denoising of triples . Indeed , completion tasks for knowledge bases attract considerable attention , because knowledge bases are known to be far from complete , as discussed in ( West et al. , 2014 ) ( Krompaß et al. , 2015 ) . Multi-relational graph embedding can help its completion and improve the performance of applications that use the graph . This is the reason why much work focuses on multi-relational graph embedding . Figure 1 shows an example of a multi-relational graph and a completion task . In multi-relational graph embedding , reducing the number of parameters is an important problem in the era of big data . Many parameters are needed with tensor-factorization-based methods , such as Bayesian clustered tensor factorization ( BCTF ) ( Sutskever et al. , 2009 ) , RESCAL ( Nickel et al. , 2011 ) , and a neural tensor network ( NTN ) ( Socher et al. , 2013 ) , where each relation has a dense matrix or tensors ( O ( D2 ) or more parameters , where D is dimensionality of the space ) . Thus , TransE ( Bordes et al. , 2013 ) was proposed to reduce the number of parameters , to overcome this problem . In TransE , each entity is mapped to a point in Euclidean space and each relation is no more than a vector addition ( O ( D ) parameters ) , rather than a matrix operation . The successors to TransE , TransH ( Wang et al. , 2014 ) and TransD ( Ji et al. , 2016 ) , also use only a small number of parameters . Some methods succeeded in reducing parameters using diagonal matrices instead of dense matrices : e.g . DISTMULT ( Yang et al. , 2015 ) , ComplEx ( Trouillon et al. , 2016 ) , HolE ( through the Fourier transform ) ( Nickel et al. , 2016 ) , and ANALOGY ( Liu et al. , 2017 ) . In these methods , all relations share one space for embedding , but each relation uses its own dissimilarity criterion . The success of these methods implies that one common space underlies whole data , and each relation can be regarded as a dissimilarity criterion in the space . Whereas these methods use distances or inner products in Euclidean space as dissimilarity criteria , recent work has shown that using non-Euclidean space can further reduce the number of parameters . One typical example of this is Poincaré Embedding ( Nickel & Kiela , 2017 ) for hierarchical data , where a hyperbolic space is used as a space for embedding . Here , the tree structure of hierarchical data has good compatibility with the exponential growth of hyperbolic space . Recall the circumference with radius R is given by 2π sinhR ( ≈ 2π expR ) in a hyperbolic plane . As a result , Poincaré embedding achieved good graph completion accuracy , even in low dimensionality such as 5 or 10 . On the other hand , spheres ( circumference : 2π sinR ) are compatible with cyclic structures . Since Poincaré embedding , several methods have been proposed for single-relational graph embedding in non-Euclidean space ( e.g . ( Ganea et al. , 2018b ) , ( Nickel & Kiela , 2018 ) ) and shown good results . The success of these methods suggests that the appropriate choice of a manifold ( i.e. , space ) can retain low dimensionality , although these methods are limited to single-relational graph embedding . According to the success of the TransE and its derivation and Poincaré embedding , it is reasonable in multi-relational graph embedding to assume the existence of a single structure compatible with a non-Euclidean manifold . For example , we can consider a single tree-like structure , which contains multiple hierarchical structures , where root selection gives multiple hierarchical structures from a single tree , which is compatible with hyperbolic spaces ( See Figure 2 ) . Therefore , embedding in a single shared non-Euclidean manifold with multiple dissimilarity criteria used in TransE is promising . Taking Poincaré embedding ’ s success with low dimensionality into consideration , this method should work well ( e.g. , in graph completion tasks ) with small number of parameters . This is the main idea of this paper . 1.2 CONTRIBUTIONS . We propose a novel method , called Riemannian TransE , for multi-relation graph embedding using a non-Euclidean manifold . In Riemannian TransE , the relations share one non-Euclidean space and the entities are mapped to the space , whereas each relation has its own dissimilarity criterion based on the distance in the space . Specifically , the dissimilarity criteria in Riemannian TransE are similar to those in TransE ( Bordes et al. , 2013 ) based on vector addition , which is known to be effective . Unfortunately , we can not straightforwardly use TransE ’ s dissimilarity criteria . This is due to non-existence of a parallel vector field ( See Figure 4 ) , which is implicitly but essentially used in “ vector addition. ” However , the parallel condition is not essential in TransE ’ s idea . For example , hierarchical bottom to top relations should be regarded as attraction to the top in the hierarchy , which is not parallel but has an attractive point . Moreover , parallel vector fields can be regarded as a vector field attracted to a point at infinity . Therefore , we replace parallel vector fields in TransE by vector fields with an attractive point that are well-defined in Riemannian manifolds , and as a result , we obtain Riemannian TransE . Advantages of non-Euclidean spaces enable our Riemannian TransE to achieve good performance ( e.g . in graph completion ) with low-dimensional parameters . Riemannian TransE further exploits the advantages of TransE : that is , the method needs only O ( D ) parameters for each relation . Numerical experiments on graph completion tasks show that with an appropriate choice of manifold , our method can improve the performance of multi-relational graph embedding with few parameters . 2 RELATED WORK . 2.1 MULTI-RELATIONAL GRAPH EMBEDDING . Let V and R denote the entities and relations in a multi-relational graph , and let T ⊂ V × R × V denote the triples in the graph . Multi-relational graph embedding refers to a pair of maps from V andR intoMe andMr , respectively . Particularly , learning multi-relational graph embedding refers to obtaining an appropriate pair of maps v 7→ pv ( v ∈ V , pv ∈Me ) and r 7→ wr ( r ∈ R , wr ∈Mr ) from the triples T . In this paper , we call pv the planet of entity v , wr the launcher of relation r , and Me and Mr the planet manifold and launcher manifold , respectively . The quality of embedding is measured through a score function f : ( Me ×Me ) × Mr → R , which is designed by each method . Embedding is learned such that the value score function f ( ph , pt ; wr ) will be low when ph , pt ; wr ∈ T and high when ph , pt ; wr /∈ T . For specific loss functions designed from the score function , see Subsection 2.3 . We interpret the score function of multi-relational graph embedding as dissimilarity in a manifold , which we call a satellite manifoldMs . We rewrite the score function f in multi-relational graph embedding using two maps H , T : Me ×Mr →Ms and the dissimilarity measure function D : Ms ×Ms → R as follows : f ( ph , pt ; wr ) : = D ( sHh ; r , s T t ; r ) , where sHh ; r = H ( ph ; wr ) , s T t ; r = T ( pt ; wr ) . ( 1 ) We call H and T the head and tail launch map , respectively , and call sHv ; r and s T v ; r the head and tail satellite of entity v ( or of planet pv ) with respect to relation r. The idea of this formulation is embedding in one shared space with multiple dissimilarity criteria . Specifically , each entity has only one planet and their satellite pairs give multiple dissimilarity criteria , each of which corresponds to a relation . In other words , all of the relations shares one space and the planets in it , and the differences among the relations are reduced to the difference of their launcher maps and the satellites given by them . We regard the planets as the embeddings of the entities , whereas dissimilarity between entities with respect to a relation is evaluated through their satellites which correspond to the relation . A simple example of this is TransE ( Bordes et al. , 2013 ) , where all of the planets , satellites , and launchers share the same Euclidean space , i.e.Me =Ms =Mr = RD , the launch maps are given by vector addition as H ( p ; w ) = p +w and T ( p ; w ) = p , and the distance in a norm space— i.e . the norm of the difference—is used as a dissimilarity criterion i.e . D ( sH , sT ) = ∥∥sT − sH∥∥ ( the L1 or L2 norm is often used in practice ) . See Figure 5 ( left ) . As Nguyen ( 2017 ) suggested , one can associate the idea of representing relations as vector additions with the fact that we can find a relation through a substraction operator in Word2Vec Mikolov et al . ( 2013 ) . That is , we can find relations such as pFrance−pParis ≈ pItaly−pRome in Word2Vec . As explained above , TransE is based on the distance between satellites , and each satellite is given by simple vector addition . Regardless of this simplicity , the performance of TransE has been exemplified in review papers ( Nickel et al. , 2016 ) ( Nguyen , 2017 ) . Indeed , the addition operation in a linear space is essential in the launcher map , and hence TransE can easily be extended to a Lie group , which is a manifold equipped with an addition operator , as suggested in Ebisu & Ichise ( 2017 ) . Some methods , such as TransH ( Wang et al. , 2014 ) , TransR ( Lin et al. , 2015 ) , and TransD ( Ji et al. , 2016 ) , also use a norm in linear space as a dissimilarity measure , integrating a linear map into a latent space . Another simple example is RESCAL ( Nickel et al. , 2011 ) , which uses the negative inner product as a dissimilarity measure . In RESCAL , the launcher of relation r is a matrixW ∈Mr = RD×D , the launch maps are given by a linear map , i.e . H ( p ; ( W , w ) ) = Wp and T ( p ; ( W , w ) ) = p , and the dissimilarity measure is the negative inner product D ( sH , sT ) = − ( sH ) > sT. Other methods are also based on the ( negative ) inner product dissimilarity : e.g. , DISTMULT ( Yang et al. , 2015 ) , ComplEx ( Trouillon et al. , 2016 ) , HolE ( through the Fourier transform ) ( Nickel et al. , 2016 ) , and ANALOGY ( Liu et al. , 2017 ) . Table 1 shows score functions of these methods . Whereas some methods are based on a neural network ( e.g. , the neural tensor network ( Socher et al. , 2013 ) and ConvE ( Dettmers et al. , 2017 ) ) , their score function consists of linear operations and element-wise nonlinear functions .

Summary: The paper proposes a new approach to compute embeddings of multi-relational data such as knowledge graphs. For this purpose, the paper introduces a variant of TransE that operates on Riemannian manifolds, in particular, Euclidean, Spherical, and Hyperbolic space. This approach is motivated by the results of Nickel & Kiela (2017), who showed tha  Hyperbolic space can provide important advantages for embedding graphs with hierarchical structure.