TransINT: Embedding Implication Rules in Knowledge Graphs with Isomorphic Intersections of Linear Subspaces

1 . Introduction . Learning distributed vector representations of multi-relational knowledge is an active area of research [ Bordes et al. , 2013 , Nickel et al. , 2011 , Kazemi and Poole , 2018a , Wang et al. , 2014 , Bordes et al. , 2011 ] . These methods map components of a KG ( entities and relations ) to elements of Rdand capture statistical patterns , regarding vectors close in distance as representing similar concepts . One focus of current research is to bring logical rules to KG embeddings [ Guo et al. , 2016 , Wang et al. , 2015a , Wei et al. , 2015 ] . While existing methods impose hard geometric constraints and embed asymmetric orderings of knowledge [ Nickel and Kiela , 2017 , Vendrov et al. , 2015 , Vilnis et al. , 2018 ] , many of them only embed hierarchy ( unary Is a relations ) , and can not embed binary or n-ary relations in KG ’ s . On the other hand , other methods that integrate binary and n-ary rules [ Guo et al. , 2016 , Fatemi et al. , 2018 , Rocktäschel et al. , 2015 , Demeester et al. , 2016 ] do not bring significant enough performance gains . We propose TransINT , a new and extremely powerful KG embedding method that isomorphically preserves the implication ordering among relations in the embedding space . Given pre-defined implication rules , TransINT restricts entities tied by a relation to be embedded to vectors in a particular region of Rd included isomorphically to the order of relation implication . For example , we map any entities tied by is father of to vectors in a region that is part of the region for is parent of ; thus , we can automatically know that if John is a father of Tom , he is also his parent even if such a fact is missing in the KG . Such embeddings are constructed by sharing and rank-ordering the basis of the linear subspaces where the vectors are required to belong . Mathematically , a relation can be viewed as sets of entities tied by a constraint [ Stoll , 1979 ] . We take such a view on KG ’ s , since it gives consistency and interpretability to model behavior . We show that angles between embedded relation sets can identify semantic patterns and implication rules - an extension of the line of thought as in word/ image embedding methods such as Mikolov et al . [ 2013 ] , Frome et al . [ 2013 ] to relational embedding . The main contributions of our work are : ( 1 ) A novel KG embedding such that implication rules in the original KG are guaranteed to unconditionally , not approximately , hold . ( 2 ) Our model suggests possibilities of learning semantic relatedness between groups of objects . ( 3 ) We significantly outperform state-of-the-art rule integration embedding methods , [ Guo et al. , 2016 ] and [ Fatemi et al. , 2018 ] , on two benchmark datasets , FB122 and NELL Sport/ Location . 2 . TransINT . In this section , we describe the intuition and justification of our method . We first define relation as sets , and revisit TransH [ Wang et al. , 2014 ] as mapping relations to sets in Rd . Finally , we propose TransINT . We put ∗ next to definitions and theorems we propose/ introduce . Otherwise , we use existing definitions and cite them . 2.1 Sets as Relations . We define relations as sets and implication as inclusion of sets , as in set-theoretic logic . Definition ( Relation Set ) : Let ri be a binary relation and x , y entities . Then , a set Ri such that ri ( x , y ) if and only if ( x , y ) ∈ Ri always exists [ Stoll , 1979 ] . We call Ri the relation set of ri . For example , consider the distinct relations in Figure 1a , and their corresponding sets in Figure 1b ; Is Father Of ( Tom , Harry ) is equivalent to ( Tom , Harry ) ∈ RIs Father Of . Definition ( Logical Implication ) : For two relations , r1 implies r2 ( or r1 ⇒ r2 ) iff ∀x , y , ( x , y ) ∈ R1 ⇒ ( x , y ) ∈ R2 or equivalently , R1 ⊂ R2 . [ Stoll , 1979 ] For example , Is Father Of ⇒ Is Parent Of . ( In Figure 1b , RIs Father Of ⊂ RIs Parent Of ) . h⊥ + # » rj ≈ # » t⊥ ( b ) : difference first , then projection - first subtracting # » h from # » t , and then projecting the difference ( # » t− h ) to His parent of and requiring ( # » t− h ) ⊥ ≈ rj . All ( # » t− h ) ⊥ belong to the red line , which is unique because it is when # » ris parent of is translated to the origin . 2.2 Background : TransH . Given a fact triple ( h , rj , t ) in a KG ( i.e . ( Harry , is father of , Tom ) ) , TransH maps each entity to a vector , and each relation rj to a relation-specific hyperplane Hj and a fixed vector # » rj on Hj ( Figure 2a ) . For each fact triple ( h , rj , t ) , TransH wants # » h⊥ + # » rj ≈ # » t⊥ · · · · · ( Eq . 1 ) where # » h⊥ , # » t⊥ are projections on # » h , # » t onto Hj ( Figure 2a ) . Revisiting TransH We interpret TransH in a novel perspective . An equivalent way to put Eq.1 is to change the order of subtraction and projection ( Figure 2b ) : Projection of ( # » t− h ) onto Hj ≈ # » rj . This means that all entity vectors ( # » h , # » t ) such that their difference # » t− h belongs to the red line are considered to be tied by relation rj ( Figure 2b ) ; Rj ≈ the red line , which is the set of all vectors whose projection onto Hj is the fixed vector # » rj . Thus , upon a deeper look , TransH actually embeds a relation set in KG ( figure 1b ) to a particular set in Rd . We call such sets relation space for now ; in other words , a relation space of some relation ri is the space where each ( h , ri , t ) ’ s # » t− h can exist . We formally visit it later in Section 3.1 . Thus , in TransH , ri ( x , y ) ≡ ( x , y ) ∈ Ri ( relation in KG ) ∼= # » y − x ∈ relation space of ri ( relation in Rd ) t− h ’ s of h , t tied by these relations can exist . The blue and the red line lie on the green plane - is parent of ’ s relation space includes the other two ’ s . 2.3 TransINT . We propose TransINT , which , given pre-defined implication rules , guarantees isomorphic ordering of relations in the embedding space . Like TransH , TransINT embeds a relation rj to a ( subspace , vector ) pair ( Hj , # » rj ) . However , TransINT modifies the relation embeddings ( Hj , # » rj ) so that the relation spaces ( i.e . red line of Figure 2b ) are ordered by implication ; we do so by intersecting the Hj ’ s and projecting the # » rj ’ s ( Figure 3a ) . We explain with familial relations as a running example . Intersecting the Hj ’ s TransINT assigns distinct hyperplanesHis father of andHis mother of to is father of and is mother of . However , because is parent of is implied by the aforementioned relations , we assign His parent of = His father of ∩His mother of . TrainsINT ’ s His parent of is not a hyperplane but a linear subspace of rank 2 ( Figure 3a ) , unlike in TransH where all Hj ’ s are hyperplanes ( whose ranks are 1 ) . Projecting the # » rj ’ s TransINT constrains the # » rj ’ s with projections ( Figure 3a ’ s dotted orange lines ) . First , # » ris father of and # » ris mother of are required to have the same projection onto His parent of . Second , # » ris parent of is that same projection onto His parent of . We connect the two above constraints to ordering relation spaces . Figure 3b graphically illustrates that is parent of ’ s relation space ( green hyperplane ) includes those of is father of ( blue line ) and is mother of ( red line ) . More generally , TransINT requires that For distinct relations ri , rj , require the following if and only if ri ⇒ rj : Intersection Constraint : Hj = Hi ∩Hj . Projection Constraint : Projection of # » r1 to Hj is # » rj . where # » Hi , # » Hj and # » ri , # » rj are distinct . We prove that these two constraints guarantee that an ordering isomorphic to implication holds in the embedding space : ( ri ⇒ rj ) iff ( ri ’ s rel . space ⊂ rj ’ s rel . space ) or equivalently , ( Ri ⊂ Rj ) iff ( ri ’ s rel . space ⊂ rj ’ s rel . space ) . 3 . TransINT ’ s Isomorphic Guarantee . In this section , we formally state TransINT ’ s isomorphic guarantee . We denote all d × d matrices with capital letters ( e.g . A ) and vectors with arrows on top ( e.g . # » b ) . 3.1 Projection and Relation Space . In Rd , there is a bijection between each linear subspace Hi and a projection matrix Pi ; ∀ # » x ∈ Rd , Pix ∈ Hi [ Strang , 2006 ] . A random point # » a ∈ Rd is projected onto Hi iff multiplied by Pi ; i.e . Pia = # » b ∈ Hi . In the rest of the paper , we denote P ( or Pi ) as the projection matrix onto a linear subspace H ( or Hi ) . Now , we formally define a general concept that subsumes relation space ( Figure 3b ) . Definition∗ ( Sol ( P , # » k ) ) : Let H be a linear subspace and P its projection matrix . Then , given # » k on H , the set of vectors that become # » k when projected on to H , or the solution space of P # » x = # » k , is denoted as Sol ( P , # » k ) . With this definition , relation space ( Figure 3b ) is ( Sol ( Pi , # » ri ) ) , where Pi is the projection matrix of Hi ( subspace for relation ri ) ; it is the set of points # » t− h such that Pi ( # » t− h ) = # » ri .

Summary: This work describes a new approach to learn KG embeddings by preserving the "implication ordering" among relations in the embedding space. The paper provides a cute new interpretation of TransH and then uses it to extend TransH to learn entity and relation representations. The crucial novelty of the approach is to map relations into linear subspaces whose parameters are tied using the implication ordering of relations.