Modeling Heterogeneous Hierarchies with Relation-specific Hyperbolic Cones

1 Introduction . Knowledge graph ( KG ) is a prevalent data structure that stores factual knowledge in the form of triplets , which connect two entities ( nodes ) with a relation ( edge ) [ 1 ] . Knowledge graphs play an important role in many scientific and machine learning applications , including question answering [ 2 ] , information retrieval [ 3 ] and discovery in biomedicine [ 4 ] . Knowledge graph completion is the problem of predicting missing relations in the graph , and is crucial in many real-world applications . Knowledge graph embedding ( KGE ) models [ 5 , 6 , 7 ] approach the task by embedding entities and relations into low-dimensional vector space and then use the embeddings to learn a function that given a head entity h and a relation r predicts the tail entity t. Hierarchical information is ubiquitous in real-world KGs , such as WordNet [ 8 ] or Gene Ontology [ 9 ] , since much human knowledge is organized hierarchically . The relations in these KGs can be separated into non-hierarchical relations ( e.g. , likes , friendOf ) and hierarchical relations ( e.g. , isA , ∗Equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . partOf ) , where non-hierarchical relations capture interactions between the entities at the same level while hierarchical relations induce a tree-like partial ordering structure of entities . Recent works propose the use of a variety of embedding geometries such as hyperbolic embeddings , box embeddings , and cone embeddings [ 10 , 11 , 12 ] to model partial ordering property of hierarchical relations , but two important challenges remain : ( 1 ) Existing works that consider hierarchical relations [ 13 ] do not take into account existing non-hierarchical relations [ 14 ] . ( 2 ) These methods can only be applied to graphs with a single hierarchical relation type , and are thus not suitable to real-world knowledge graphs that simultaneously encode multiple hierarchies using many different relations . For example , in Figure 1 , subClass and partOf each define a unique hierarchy over the same set of entities . However , existing models treat all relations in a KG as part of one single hierarchy , limiting the ability to reason with different types of heterogeneous hierarchical relations . While there are methods for reasoning over KGs that use hyperbolic space ( MuRP [ 15 ] , RotH [ 16 ] ) , which is suitable for modeling tree-like graphs , the choice of relational transformations used in these works ( rotation ) prevents them from faithfully capturing all the properties of hierarchical relations . For example , they can not model transitivity of hierarchical relations : if there exist triplets ( h1 , r , h2 ) and ( h2 , r , h3 ) , then ( h1 , r , h3 ) exists , i.e . h1 and h3 are also related by relation r. Here we propose a novel hyperbolic knowledge graph embedding model ConE . ConE is motivated by the transitivity of nested angular cones [ 12 ] that naturally model the partial ordering defined by hierarchical relations . Our proposed approach embeds entities into the product space of hyperbolic planes , where the coordinate in each hyperbolic plane corresponds to a 2D hyperbolic cone . To address challenge ( 1 ) , we model non-hierarchical relations as hyperbolic cone rotations from head entity to tail entity , while we model hierarchical relations as a restricted rotation which guarantees cone containment ( Figure 1 ( b ) ) . To address challenge ( 2 ) , we assign distinct embedding subspaces corresponding to product spaces of a different set of hyperbolic planes for each hierarchical relation , to enforce cone containment constraints . By doing so , multiple heterogeneous hierarchies are preserved simultaneously in unique subspaces , allowing ConE to perform multiple hierarchical reasoning tasks accurately . We evaluate the performance of ConE on the KG completion task and hierarchical reasoning task . A single trained ConE model can achieve remarkable performance on both tasks simultaneously . On KG completion task , ConE achieves new state-of-the-art results on two benchmark knowledge graph datasets including WN18RR [ 5 , 17 ] , DDB14 [ 18 ] ( outperforming by 0.9 % and 4.5 % on Hits @ 1 metric ) . We also develop a novel biological knowledge graph GO21 from biomedical domain and show that ConE successfully models multiple hierarchies induced by different biological processes . We also evaluate our model against previous hierarchical modeling approaches on ancestordescendant prediction task . Results show that ConE significantly outperforms baseline models ( by 20 % on average when missing links are included ) , suggesting that it effectively models multiple heterogeneous hierarchies . Moreover , ConE performs well on the lowest common ancestor ( LCA ) prediction task , improving over previous methods by 100 % in Hits @ 3 metric . 2 Related Work . Hierarchical reasoning . The most related line of work is learning structured embeddings to perform hierarchical reasoning on graphs and ontologies : order embedding , probabilistic order embedding , box embedding , Gumbel-box embedding and hyperbolic embedding [ 10 , 11 , 12 , 19 , 20 , 21 , 22 ] . These embedding-based methods map entities to various geometric representations that can capture the transitivity and entailment of hierarchical relations . These methods aim to perform hierarchical reasoning ( transitive closure completion ) , such as predicting if an entity is an ancestor of another entity . However , the limitation of the above works is that they can only model a single hierarchical relation , and it remains unexplored how to extend them to multiple hierarchical relations in heterogeneous knowledge graphs . Recently , [ 23 ] builds upon the box embedding and further models joint ( two ) hierarchies using two boxes as entity embeddings . However , the method is not scalable since the model needs to learn a quadratic number of transformation functions between all pairs of hierarchical relations . Furthermore , the missing part is that these methods do not leverage nonhierarchical relations to further improve the hierarchy modeling . For example in Figure 1 ( a ) , with the sisterTerm ( PalmTree , MapleTree ) and subClass ( PalmTree , Tree ) , we may infer subClass ( MapleTree , Tree ) . In contrast to prior methods , ConE is able to achieve exactly this type of reasoning as it can simultaneously model multiple hierarchical as well as non-hierarchical relations . Knowledge graph embedding . Various embedding methods have been proposed to model entities and relations in heterogeneous knowledge graphs . Prominent examples include TransE [ 5 ] , DistMult [ 24 ] , ComplEx [ 25 ] , RotatE [ 7 ] and TuckER [ 14 ] . These methods often require high embedding dimensionality to model all the triples . Recently KG embeddings based on hyperbolic space have shown success in modeling hierarchical knowledge graphs . MuRP [ 15 ] learns relation-specific parameters in the Poincaré ball model . RotH [ 16 ] uses rotation and reflection transformation in n-dimensional Poincaré space to model relational patterns , and achieves state-of-the-art for the KG completion task , especially under low-dimensionality . However , transformations used in MuRP and RotH can not capture transitive relations which hierarchical relations naturally are . To the best of our knowledge , ConE is the first model that can faithfully model multiple hierarchical as well as non-hierarchical relations in a single embedding framework . 3 ConE Model Framework . 3.1 Preliminaries . Knowledge graphs and knowledge graph embeddings . We denote the entity set and the relation set in knowledge graph as E and R respectively . Each edge in the graph is represented by a triplet ( h , r , t ) , connecting the head entity h ∈ E and the tail entity t ∈ E with relation r ∈ R. In KG embedding models , entities and relations are mapped to vectors : E → RdE , R → RdR . Here dE , dR refer to the dimensionality of entity and relation embeddings , respectively . Specifically , the mapping is learnt via optimizing a defined scoring function RdE × RdR × RdE → R measuring the likelihood of triplets [ 16 ] , while maximizing such likelihood only for true triplets . Hierarchies in knowledge graphs . Many real-world knowledge graphs contain hierarchical relations [ 10 , 11 , 26 ] . Such hierarchical structure is characterized by very few top-level nodes corresponding to general and abstract concepts and a vast number of bottom-level nodes corresponding to concrete instances or components of the concept . Examples of hierarchical relations include isA , partOf . Note that there may exist multiple ( heterogeneous ) hierarchical relations in the same graph , which induce several different potentially incompatible hierarchies ( i.e. , partial orderings ) over the same set of entities ( Figure 1 ( a ) ) . In contrast to prior work , our approach is able to model many simultaneous hierarchies over the same set of entities . Hyperbolic embeddings . Hyperbolic embeddings can naturally capture hierarchical structures . Hyperbolic geometry is a non-Euclidean geometry with a constant negative curvature , where curvature measures how a geometric manifold deviates from Euclidean space . In this work , we use Poincaré ball model with constant curvature c = −1 as the hyperbolic space for entity embeddings [ 10 ] . We also investigate on more flexible curvatures , see Appendix B , results show that our model is robust enough with constant curvature c = −1 . In particular , we denote d-dimensional Poincaré ball centered at origin as Bd = { x ∈ Rd : ∥x∥ < 1 } , where ∥·∥ is the Euclidean norm . The Poincaré ball model of hyperbolic space is equipped with Riemannian metric : gB = ( 2 1− ∥x∥2 ) 2gE ( 1 ) where gE denotes the Euclidean metric , i.e. , gE = Id . The mobius addition ⊕ [ 27 ] defined on Poincaré ball model with −1 curvature is given by : x⊕ y = ( 1 + 2⟨x , y⟩+ ∥y∥ 2 ) x+ ( 1− ∥x∥2 ) y 1 + 2⟨x , y⟩+ ∥x∥2 ∥y∥2 ( 2 ) For each point x ∈ Bd , the tangent space TxB is the Euclidean vector space containing all tangent vectors at x . One can map vectors in TxB to vectors in Bd through exponential map expx ( · ) : TxB → Bd as follows : expx ( u ) = x⊕ tanh ( ∥u∥ 1− ∥x∥ ) u ∥u∥ ( 3 ) Conversely , the logarithmic map logx ( · ) : Bd → TxB maps vectors in Bd back to vectors in TxB , in particular : logx ( u ) = ( 1− ∥x∥ ) · tanh −1 ( ∥−x⊕ v∥ ) −x⊕ v ∥−x⊕ v∥ ( 4 ) Also , the hyperbolic distance between x , y ∈ Bd is : dB ( x , y ) = 2 tanh −1 ( ∥−x⊕ y∥ ) ( 5 ) A key property of hyperbolic space is that the amount of space covered by a ball of radius r in hyperbolic space increases exponentially with respect to r , rather than polynomially as in Euclidean space . This property contributes to the fact that hyperbolic space can naturally model hierarchical tree-like structure . Hyperbolic entailment cones . Each hierarchical relation induces a partial ordering over the entities . To capture a given partial ordering , we use the hyperbolic entailment cones [ 12 ] . Figure 1 ( b ) gives an example of 2D hyperbolic cones . Let Cx denotes the cone at apex x . The goal is to model partial order by containment relationship between cones , in particular , the entailment cones satisfy transitivity : ∀x , y ∈ Bd\ { 0 } : y ∈ Cx ⇒ Cy ⊆ Cx ( 6 ) Also , for x , y ∈ Bd , we define the angle of y at x to be the angle between the half-lines −→ox and −→xy and denote it as ∠xy . It can be expressed as : ∠xy = cos −1 ( ⟨x , y⟩ ( 1 + ∥x∥2 ) − ∥x∥2 ( 1 + ∥y∥2 ) ∥x∥ ∥x− y∥ √ 1 + ∥x∥2 ∥y∥2 − 2⟨x , y⟩ ) ( 7 ) To satisfy transitivity of nested angular cones and symmetric conditions [ 12 ] , we have the following expression of Poincaré entailment cone at apex x ∈ Bd : Cx = { y ∈ Bd|∠xy ≤ sin−1 ( K 1− ∥x∥2 ∥x∥ ) } ( 8 ) where K ∈ R is a hyperparameter ( we take K = 0.1 ) . This implies that the half aperture ϕx of cone Cx is as follows : ϕx = sin −1 ( K 1− ∥x∥2 ∥x∥ ) ( 9 )

Summary: This work proposes ConE, a knowledge graph embedding model which embeds entities into hyperbolic cones and represents relations as transformations between the cones. Non-hierarchical relations are modelled using rotations and hierarchical relations using restricted rotations, which impose the cone containment constraint. The proposed method is evaluated on the KG completion and ancestor-descendant prediction tasks.