A Group-Theoretic Framework for Knowledge Graph Embedding

1 INTRODUCTION . Knowledge graphs ( KGs ) are prominent structured knowledge bases for many downstream semantic tasks ( Hao et al. , 2017 ) . A KG contains an entity set E = { ei } , which correspond to vertices in the graph , and a relation set R = { rk } , which forms edges . The entity and relation sets form a collection of factual triplets , each of which has the form ( ei , rk , ej ) where rk is the relation between the head entity ei and the tail entity ej . Since large scale KGs are usually incomplete due to missing links ( relations ) amongst entities , an increasing amount of recent works ( Bordes et al. , 2013 ; Yang et al. , 2014 ; Trouillon et al. , 2016 ; Lin et al. , 2015 ) have devoted to the graph completion ( i.e. , link prediction ) problem by exploring a low-dimensional representation of entities and relations . More formally , each relation r acts as a mapping Or [ · ] from its head entity e1 to its tail entity e2 : r : e1 7→ Or [ e1 ] = : e2 . ( 1 ) The original KG dataset represents these mappings in a tabular form , and the task of KG embedding is to find a better representation for these abstract mappings . For example , in the TransE model ( Bordes et al. , 2013 ) , relations and entities are embedded in the same vector space , and the operation Or [ · ] is simply a vector summation : Or [ e ] = e + r. In general , the operation could be either linear or nonlinear , either pre-defined or learned . Since the entity and relation sets in KGs play conceptually different roles , it is reasonable to represent them differently , which demands a more proper operation Or [ · ] for bridging the entity and relation sets . On the other hand , the graph completion task relies on the fact that relations are not independent . For example , the hypernym and hyponym are inverse to each other ; while kinship relations usually support mutual inferences . Previous studies ( Sun et al. , 2019 ; Xu & Li , 2019 ) have concerned some specific cases of inter-relation dependencies , including ( anti- ) symmetry and compositional relations . However , one may ask a more general question : could we establish certain fundamental principles to guide the operation design , such that any inter-relation dependency could be naturally captured ? In this work , we find the answer is positive , and we term these principles as hyper-relations . Group theory emerges as a natural description tool that satisfies the need for finding better operations and hyper-relations . In the following sections , we demonstrate the emergence of group definition , namely closure , identity , inverse , and associativity , from a study of knowledge graph itself , and explain in detail an implementation of group theory to tackle general relational embedding problems . One main contribution of this work is it provides a framework for addressing the KG embedding problem from a novel and more rigorous perspective : the group-theoretic perspective . This frame- work arises from our investigation of the graph reconstruction problem . We prove that the intrinsic structure of this task automatically produces the complete definition of groups . To our best knowledge , this is the first proof that rigorously legitimates the application of group theory in KG embedding . With this framework , we reconstructed several existing models using group theory language ( see Sec 3.3 ) , including : TransE ( Bordes et al. , 2013 ) , TransR ( Lin et al. , 2015 ) , TorusE ( Ebisu & Ichise , 2018 ) , RotatE ( Sun et al. , 2019 ) , ComplEx ( Trouillon et al. , 2016 ) , DisMult ( Yang et al. , 2014 ) . 2 RELATED WORKS . From the group theory perspective , our work may be related to the TorusE ( Ebisu & Ichise , 2018 ) , the RotatE ( Sun et al. , 2019 ) and DihEdral ( Xu & Li , 2019 ) models . The TorusE model tries to frame the KG embedding in a Lie-group and deals with the compactness problem . Its authors proved that the additive nature of the operation in the TransE model contradicts the entity regularization . However , if a non-compact embedding space is used , entities must be regularized to prevent the divergence of negative scores . Therefore the TorusE model used n-torus , a compact manifold , as the embedding space . In other words , a group is regarded as a manifold , rather than as a set of operations where algebraic structures are more emphasized . Besides , the compactness issue was solved in many later works by bounding the training objective using Softplus function ( Sun et al. , 2019 ) . In the DihEdral model , the D4 group the author used plays the same role as in our work : depicting the transformation of entities by assigning each relation a group element . The motivation of DihEdral is to resolve the non-abelian composition ( i.e. , the compositional relation formed by r1 and r2 would change if the two are switched ) . Nevertheless , DihEdral applies a discrete group D8 for relation embedding while using a continuous entity embedding space , which may suffer two problems as discussed in the later Section 3.3 . The RotatE model was designed to accommodate symmetric , inversion , and ( abelian ) composition of triplets at the same time . Different from those previous works , this work does not target at one or few specific cases but aims at answering the more general question : finding the generative principle of all possible cases , and thus to provide guidance for model designs that can accommodate all cases . More importantly , compared with most preceding works connecting to groups ( Xu & Li , 2019 ; Cai , 2019 ) , the analysis in our current work does not focus on impose an implementation of groups . Rather , we start merely by studying the graph reconstruction problem and prove that the intrinsic structure of this task itself automatically produces the complete definition of groups . To our the best of our knowledge , this is the first proof that rigorously legitimates the application of group theory in KG embeddings . 3 GROUP THEORY IN RELATIONAL EMBEDDINGS . In this section , we formulate the group-theoretic analysis for relational embedding problems . For simplicity , our discussion would start from 1-to-1 mappings and discuss the generalization to more complicated cases in the Appendix D. Firstly , as most embedding models represent objects ( including entities and relations ) as vectors , the task of operation design thus can be stated as finding proper transformations of vectors . Secondly , as we mentioned in the introduction , our ultimate goal of reproducing the whole knowledge graph using atomic triplets further requires certain types of local patterns to be accommodated . We now discuss these structures , which in the end naturally leads to the definition of groups in mathematics . 3.1 HYPER-RELATION PATTERNS : RELATION-OF-RELATIONS . One difficulty of generating the whole knowledge graph from atomic triplets lies in the fact that different relations are not independent of each other . The task of relation inference relies exactly on their mutual dependency . In other words , there exist certain relation of relations in the graph structure , which we term as hyper-relation patterns . A proper relation embedding method and the associated operations should be able to capture these hyper-relations . Now instead of studying exampling cases one by one , we ask the most general question : what are the most fundamental hyper-relations ? The answer is quite simple and only contains two types , namely , inversion and composition : • Inversion : given a relation r , there may exist an inversion r̄ , such that : r : e1 7→ e2 −→ r̄ : e2 7→ e1 , ∀e1 , e2 ∈ E . ( 2 ) The inversion captures any relation path with a length equal to 1 ( in the unit of relations ) . • Composition : given two relations r1 and r2 , there may exist a third relation r3 , such that : { r1 : e1 7→ e2 r2 : e2 7→ e3 −→ r3 : e1 7→ e3 , ∀e1 , e2 , e3 ∈ E . ( 3 ) Any relation paths longer than 1 can be captured by a sequence of compositions . One may notice the phrase may exist in the above definition , this simply emphasizes that the existence of these derived conceptual relations r̄ and r3 depends on the specific KG dataset ; while , on the other hand , to accommodate general KG datasets , the embedding space should always contains the mathematical representations of these conceptual relations . An important feature of KG is that with the above two hyper-relations , one could generate any local graph pattern and eventually the whole graph , as relational paths with arbitrary length have been captured . Note the term of inversion and composition might have different meanings from ones in other works : most existing works study triplets to analyze hyper relations , while the definition we provide above is based purely on relations . This is more general in the sense that any conclusion derived would not depend on entities at all , and some different hyper relations could , therefore , be summarized as a single one . For example , there are enormous discussions on symmetric triplets and anti-symmetric triplets ( Sun et al. , 2019 ) , which are defined as : symmetric : ( e1 , r , e2 ) −→ ( e2 , r , e1 ) , anti-symmetric : ( e1 , r , e2 ) −→ ¬ ( e2 , r , e1 ) . ( 4 ) In fact , if for any choice of e1,2 , one could produce a symmetric pair of true triplets using r , this would imply a property of r itself , and in which case , one could then simply derive : r̄ = r. ( 5 ) This is a special case of the inversion hyper-relation ; and similarly , the anti-symmetric case simply implies r̄ 6= r , which is quite common , and does not require extra design . The deep reason for discussing hyper-relations which relies merely on relations rather than triplets is that the logic of relation inference problem itself is not entity-dependent . 3.2 EMERGENT GROUP THEORY . To accommodate both general inversions and general compositions , we now derive explicit requirements on the relation embedding model . We start by defining the product of two relations r1 and r2 : r1 · r2 , as subsequently ” finding the tail ” twice according to the two relations , i.e . Or1·r2 [ · ] : = Or1 [ Or2 [ · ] ] . ( 6 ) With the above definition , equation 3 can be rewritten as : r3 = r1 · r2 . One would realize that the following properties should be supported by a proper embedding model : 1 . Inverse element : to allow the possible existence of inversion , the elements r̄ should also be an element living in the same relation-embedding space1 . 2 . Closure : to allow the possible existence of composition , in general , the elements r1 · r2 should also be an element living in the same relation-embedding space2 . 3 . Identity element : the possibly existing inverse and composition together define another special and unique relation : i = r · r̄ , ∀r ∈ R. ( 7 ) This element should map any entity to itself , and thus we call it identity element . 1Given a graph , not all inversions correspond to meaningful relations , but an embedding model should be able to capture this possibility in general . 2Given a graph , not all compositions correspond to meaningful relations , but an embedding model should be able to capture this possibility in general . 4 . Associativity : In a relational path with the length longer than three ( containing three or more relations { r1 , r2 , r3 , ... } ) , as long as the sequential order does not change , the following two compositions should produce the same result : ( r1 · r2 ) · r3 = r1 · ( r2 · r3 ) . ( 8 ) See Appendix A for the proof . 5 . Commutativity/Nonconmmutativity : In general , commuting two relations in a composition , i.e . r1 · r2 ↔ r2 · r1 , may compose either the same or different results . In real graphs , any cases may exist , and a proper embedding method should be able to accommodate both.3 The first four properties are exactly the definition of a group . In other words , the group theory automatically emerges from the relational embedding problem itself , rather than being applied manually . This is quite convincing evidence that group theory is indeed the most natural language for relational embeddings if one aims at ultimately reproducing all possible local patterns in graphs . Besides , the fifth property on commutativity/nonconmmutativity are actually termed as abelian/nonabelian in the group theory language . Since abelian is more special , to accommodate both possibilities , one should , in general , consider a nonabelian group for the relation embedding , and guarantee at the same time it contains at least one nontrivial abelian subgroup . More explicitly , given a graph , to implement a group structure in embedding , one should embed all relations as group elements , which are parametrized by certain group parameters . For instance : the translation group T can be parametrized by a real number δ . And correspondingly , due to its vector nature , the embedding of entities could be regarded as a representation ( rep ) space of the same group . For the translation group , R ( the real field ) is a rep space of T . This suggests the group representation theory is useful in knowledge graph embedding problems when talking about entity embeddings , and we leave this as a separate topic for subsequent works later . In the later section , we provide a general recipe for the graph embedding implementation .

Summary: This paper start merely by studying the graph reconstruction problem and prove that the intrinsic structure of this task itself automatically produces the complete definition of groups. it seems to be a novel result. Based on this result, one could construct embedding models that naturally accommodate all possible local graph patterns, and the paper also shows a few simulations.