Modeling Heterogeneous Hierarchies with Relation-specific Hyperbolic Cones

1 Introduction . Knowledge graph ( KG ) is a prevalent data structure that stores factual knowledge in the form of triplets , which connect two entities ( nodes ) with a relation ( edge ) [ 1 ] . Knowledge graphs play an important role in many scientific and machine learning applications , including question answering [ 2 ] , information retrieval [ 3 ] and discovery in biomedicine [ 4 ] . Knowledge graph completion is the problem of predicting missing relations in the graph , and is crucial in many real-world applications . Knowledge graph embedding ( KGE ) models [ 5 , 6 , 7 ] approach the task by embedding entities and relations into low-dimensional vector space and then use the embeddings to learn a function that given a head entity h and a relation r predicts the tail entity t. Hierarchical information is ubiquitous in real-world KGs , such as WordNet [ 8 ] or Gene Ontology [ 9 ] , since much human knowledge is organized hierarchically . The relations in these KGs can be separated into non-hierarchical relations ( e.g. , likes , friendOf ) and hierarchical relations ( e.g. , isA , ∗Equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . partOf ) , where non-hierarchical relations capture interactions between the entities at the same level while hierarchical relations induce a tree-like partial ordering structure of entities . Recent works propose the use of a variety of embedding geometries such as hyperbolic embeddings , box embeddings , and cone embeddings [ 10 , 11 , 12 ] to model partial ordering property of hierarchical relations , but two important challenges remain : ( 1 ) Existing works that consider hierarchical relations [ 13 ] do not take into account existing non-hierarchical relations [ 14 ] . ( 2 ) These methods can only be applied to graphs with a single hierarchical relation type , and are thus not suitable to real-world knowledge graphs that simultaneously encode multiple hierarchies using many different relations . For example , in Figure 1 , subClass and partOf each define a unique hierarchy over the same set of entities . However , existing models treat all relations in a KG as part of one single hierarchy , limiting the ability to reason with different types of heterogeneous hierarchical relations . While there are methods for reasoning over KGs that use hyperbolic space ( MuRP [ 15 ] , RotH [ 16 ] ) , which is suitable for modeling tree-like graphs , the choice of relational transformations used in these works ( rotation ) prevents them from faithfully capturing all the properties of hierarchical relations . For example , they can not model transitivity of hierarchical relations : if there exist triplets ( h1 , r , h2 ) and ( h2 , r , h3 ) , then ( h1 , r , h3 ) exists , i.e . h1 and h3 are also related by relation r. Here we propose a novel hyperbolic knowledge graph embedding model ConE . ConE is motivated by the transitivity of nested angular cones [ 12 ] that naturally model the partial ordering defined by hierarchical relations . Our proposed approach embeds entities into the product space of hyperbolic planes , where the coordinate in each hyperbolic plane corresponds to a 2D hyperbolic cone . To address challenge ( 1 ) , we model non-hierarchical relations as hyperbolic cone rotations from head entity to tail entity , while we model hierarchical relations as a restricted rotation which guarantees cone containment ( Figure 1 ( b ) ) . To address challenge ( 2 ) , we assign distinct embedding subspaces corresponding to product spaces of a different set of hyperbolic planes for each hierarchical relation , to enforce cone containment constraints . By doing so , multiple heterogeneous hierarchies are preserved simultaneously in unique subspaces , allowing ConE to perform multiple hierarchical reasoning tasks accurately . We evaluate the performance of ConE on the KG completion task and hierarchical reasoning task . A single trained ConE model can achieve remarkable performance on both tasks simultaneously . On KG completion task , ConE achieves new state-of-the-art results on two benchmark knowledge graph datasets including WN18RR [ 5 , 17 ] , DDB14 [ 18 ] ( outperforming by 0.9 % and 4.5 % on Hits @ 1 metric ) . We also develop a novel biological knowledge graph GO21 from biomedical domain and show that ConE successfully models multiple hierarchies induced by different biological processes . We also evaluate our model against previous hierarchical modeling approaches on ancestordescendant prediction task . Results show that ConE significantly outperforms baseline models ( by 20 % on average when missing links are included ) , suggesting that it effectively models multiple heterogeneous hierarchies . Moreover , ConE performs well on the lowest common ancestor ( LCA ) prediction task , improving over previous methods by 100 % in Hits @ 3 metric . 2 Related Work . Hierarchical reasoning . The most related line of work is learning structured embeddings to perform hierarchical reasoning on graphs and ontologies : order embedding , probabilistic order embedding , box embedding , Gumbel-box embedding and hyperbolic embedding [ 10 , 11 , 12 , 19 , 20 , 21 , 22 ] . These embedding-based methods map entities to various geometric representations that can capture the transitivity and entailment of hierarchical relations . These methods aim to perform hierarchical reasoning ( transitive closure completion ) , such as predicting if an entity is an ancestor of another entity . However , the limitation of the above works is that they can only model a single hierarchical relation , and it remains unexplored how to extend them to multiple hierarchical relations in heterogeneous knowledge graphs . Recently , [ 23 ] builds upon the box embedding and further models joint ( two ) hierarchies using two boxes as entity embeddings . However , the method is not scalable since the model needs to learn a quadratic number of transformation functions between all pairs of hierarchical relations . Furthermore , the missing part is that these methods do not leverage nonhierarchical relations to further improve the hierarchy modeling . For example in Figure 1 ( a ) , with the sisterTerm ( PalmTree , MapleTree ) and subClass ( PalmTree , Tree ) , we may infer subClass ( MapleTree , Tree ) . In contrast to prior methods , ConE is able to achieve exactly this type of reasoning as it can simultaneously model multiple hierarchical as well as non-hierarchical relations . Knowledge graph embedding . Various embedding methods have been proposed to model entities and relations in heterogeneous knowledge graphs . Prominent examples include TransE [ 5 ] , DistMult [ 24 ] , ComplEx [ 25 ] , RotatE [ 7 ] and TuckER [ 14 ] . These methods often require high embedding dimensionality to model all the triples . Recently KG embeddings based on hyperbolic space have shown success in modeling hierarchical knowledge graphs . MuRP [ 15 ] learns relation-specific parameters in the Poincaré ball model . RotH [ 16 ] uses rotation and reflection transformation in n-dimensional Poincaré space to model relational patterns , and achieves state-of-the-art for the KG completion task , especially under low-dimensionality . However , transformations used in MuRP and RotH can not capture transitive relations which hierarchical relations naturally are . To the best of our knowledge , ConE is the first model that can faithfully model multiple hierarchical as well as non-hierarchical relations in a single embedding framework . 3 ConE Model Framework . 3.1 Preliminaries . Knowledge graphs and knowledge graph embeddings . We denote the entity set and the relation set in knowledge graph as E and R respectively . Each edge in the graph is represented by a triplet ( h , r , t ) , connecting the head entity h ∈ E and the tail entity t ∈ E with relation r ∈ R. In KG embedding models , entities and relations are mapped to vectors : E → RdE , R → RdR . Here dE , dR refer to the dimensionality of entity and relation embeddings , respectively . Specifically , the mapping is learnt via optimizing a defined scoring function RdE × RdR × RdE → R measuring the likelihood of triplets [ 16 ] , while maximizing such likelihood only for true triplets . Hierarchies in knowledge graphs . Many real-world knowledge graphs contain hierarchical relations [ 10 , 11 , 26 ] . Such hierarchical structure is characterized by very few top-level nodes corresponding to general and abstract concepts and a vast number of bottom-level nodes corresponding to concrete instances or components of the concept . Examples of hierarchical relations include isA , partOf . Note that there may exist multiple ( heterogeneous ) hierarchical relations in the same graph , which induce several different potentially incompatible hierarchies ( i.e. , partial orderings ) over the same set of entities ( Figure 1 ( a ) ) . In contrast to prior work , our approach is able to model many simultaneous hierarchies over the same set of entities . Hyperbolic embeddings . Hyperbolic embeddings can naturally capture hierarchical structures . Hyperbolic geometry is a non-Euclidean geometry with a constant negative curvature , where curvature measures how a geometric manifold deviates from Euclidean space . In this work , we use Poincaré ball model with constant curvature c = −1 as the hyperbolic space for entity embeddings [ 10 ] . We also investigate on more flexible curvatures , see Appendix B , results show that our model is robust enough with constant curvature c = −1 . In particular , we denote d-dimensional Poincaré ball centered at origin as Bd = { x ∈ Rd : ∥x∥ < 1 } , where ∥·∥ is the Euclidean norm . The Poincaré ball model of hyperbolic space is equipped with Riemannian metric : gB = ( 2 1− ∥x∥2 ) 2gE ( 1 ) where gE denotes the Euclidean metric , i.e. , gE = Id . The mobius addition ⊕ [ 27 ] defined on Poincaré ball model with −1 curvature is given by : x⊕ y = ( 1 + 2⟨x , y⟩+ ∥y∥ 2 ) x+ ( 1− ∥x∥2 ) y 1 + 2⟨x , y⟩+ ∥x∥2 ∥y∥2 ( 2 ) For each point x ∈ Bd , the tangent space TxB is the Euclidean vector space containing all tangent vectors at x . One can map vectors in TxB to vectors in Bd through exponential map expx ( · ) : TxB → Bd as follows : expx ( u ) = x⊕ tanh ( ∥u∥ 1− ∥x∥ ) u ∥u∥ ( 3 ) Conversely , the logarithmic map logx ( · ) : Bd → TxB maps vectors in Bd back to vectors in TxB , in particular : logx ( u ) = ( 1− ∥x∥ ) · tanh −1 ( ∥−x⊕ v∥ ) −x⊕ v ∥−x⊕ v∥ ( 4 ) Also , the hyperbolic distance between x , y ∈ Bd is : dB ( x , y ) = 2 tanh −1 ( ∥−x⊕ y∥ ) ( 5 ) A key property of hyperbolic space is that the amount of space covered by a ball of radius r in hyperbolic space increases exponentially with respect to r , rather than polynomially as in Euclidean space . This property contributes to the fact that hyperbolic space can naturally model hierarchical tree-like structure . Hyperbolic entailment cones . Each hierarchical relation induces a partial ordering over the entities . To capture a given partial ordering , we use the hyperbolic entailment cones [ 12 ] . Figure 1 ( b ) gives an example of 2D hyperbolic cones . Let Cx denotes the cone at apex x . The goal is to model partial order by containment relationship between cones , in particular , the entailment cones satisfy transitivity : ∀x , y ∈ Bd\ { 0 } : y ∈ Cx ⇒ Cy ⊆ Cx ( 6 ) Also , for x , y ∈ Bd , we define the angle of y at x to be the angle between the half-lines −→ox and −→xy and denote it as ∠xy . It can be expressed as : ∠xy = cos −1 ( ⟨x , y⟩ ( 1 + ∥x∥2 ) − ∥x∥2 ( 1 + ∥y∥2 ) ∥x∥ ∥x− y∥ √ 1 + ∥x∥2 ∥y∥2 − 2⟨x , y⟩ ) ( 7 ) To satisfy transitivity of nested angular cones and symmetric conditions [ 12 ] , we have the following expression of Poincaré entailment cone at apex x ∈ Bd : Cx = { y ∈ Bd|∠xy ≤ sin−1 ( K 1− ∥x∥2 ∥x∥ ) } ( 8 ) where K ∈ R is a hyperparameter ( we take K = 0.1 ) . This implies that the half aperture ϕx of cone Cx is as follows : ϕx = sin −1 ( K 1− ∥x∥2 ∥x∥ ) ( 9 )

Summary: Methods for embedding hypergraph-based knowledge bases, where different edge types encode different relation types, have a long history in machine learning. Alternately, there has been a more recent interest in geometric embedding approaches for knowledge bases, which associate more complex geometric objects, such as hyperbolic cones, with knowledge base entities, due to the favorable geometric properties of hyperbolic space for modeling hierarchies. However, the latter approaches have usually been limited to representing only single relation types, due to representing the relation of interest with some geometric property of the embedding space such as enclosure between entity cones. This work attempts to bridge the gap between the two approaches by modeling relations as transformations between hyperbolic cones, specifically containment in different subspaces.