Rot-Pro: Modeling Transitivity by Projection in Knowledge Graph Embedding

1 Introduction . Knowledge graph embedding ( KGE ) aims to learn low-dimensional dense vectors to express the entities and relations in the knowledge graphs ( KG ) . It is widely used in recommendation system , question answering , dialogue systems [ 5 , 17 , 7 ] . The general intuition of KGE is to model and infer relations between entities in knowledge graphs , which has complex patterns such as symmetry , asymmetry , inversion , composition , and transitivity as shown in Table 1 . Many studies dedicate to find a method , which is able to model various relation patterns [ 3 , 2 , 21 , 20 ] . TransE models relations as translations , aims to model the inversion and composition patterns ; DisMult can model symmetric relations by capturing interactions between head and tail entities and relations . One representative model proposed recently is RotatE [ 20 ] , which is proved to be able to model symmetry , asymmetry , inversion and composition patterns by modeling relation as a rotation in the complex plane . However , none of them can model all the five relation patterns , especially the transitivity pattern . This paper focus on modeling the transitivity pattern . We theoretically show that the transitive relations can be modeled with idempotent transformations , i.e . projections [ 25 ] . Any projection matrix is similar to a diagonal matrix with elements in the diagonal being 0 or 1 . We design the projection by constraining the similarity matrix to be a rotation matrix , which has less parameters to learn . In order to model not only transitivity but also other relation patterns shown in Table 1 , we propose the Rot-Pro model which combines the projection and relational rotation together . We theoretically prove that Rot-Pro can infer the symmetry , asymmetry , inversion , composition , and transitivity ∗Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . patterns . Experimental results show that the Rot-Pro model can effectively learn the transitivity pattern . The Rot-Pro model achieves the state-of-the-art results on the link prediction task in the Countries dataset containing transitive relations and outperforms other models in the YAGO3-10 and FB15k-237 dataset . 2 Related work . There are mainly two types of knowledge graph embedding models , either using translation transformation or linear transformation between head and tail entities . Trans-series models . Trans-series models , which is well-known in KGE area , are essentially translation transformation based models . TransE [ 3 ] proposed a pure translation distance-based score function , which assumes the added embedding of head entity h and relation r should be close to the embedding of tail entity t. This simple approach is effective in capturing composition , asymmetric and inversion relations , but is hard to handle the 1-to-N , N-to-1 and N-N relations . To overcome these issues , many variants and extensions of TransE have been proposed . TransH [ 26 ] projects entities and relations into a relation-specific hyperplanes and enables different projections of an entity in different relations . TransR [ 28 ] introduces relation-specific spaces , which builds entity and relation embeddings in different spaces separately . TransD [ 9 ] simplifies TransR by constructs dynamic mapping matrices . For the purpose of model optimization , some models relax the requirement for translational distance . TransA [ 10 ] replaces Euclidean distance by Mahalanobis distance to enable more adaptive metric learning , and a recent model TransMS [ 18 ] transmits multidirectional semantics by complex relations . The variants of TransE improve the capability of the models to handle 1-to-N , N-to-1 and N-N relations as well as effectively modeling symmetric and asymmetric relations , but they are no longer able to model composition and inversion relations as they do linear transformation on head and tail entities separately before modeling the relation as translation . BoxE [ 1 ] , a recent trans-series model , embeds entities as points , and relations as a set of boxes , for yielding a model that could express multiple relation patterns including composition and inversion , but it can not express transitivity . Bilinear models . Models of bilinear series model relations as linear transformation matrix Mr from head to tail entity . The type of relation patterns that a linear transformation based model can infer depends on the property of Mr. RESCAL [ 14 ] proposes the transformation of relation as a matrix that models the pairwise interactions between entities and relation . The score of a fact is defined by a bilinear function : fr = hTMrt . DistMult [ 2 ] simplifies RESCAL by restricting Mr to diagonal matrices . Therefore , it can not handle other types of relations except symmetry . HolE [ 12 ] combines the expressive power of RESCAL with the simplicity of DistMult which introduces circular correlations . HolE can express multiple types of relations , since cyclic correlation operations are not commutative . Recently , some KGE models begin to model relation patterns explicitly . Dihedral [ 27 ] models relations in KGs with the representation of dihedral group that has properties to support the relation as symmetry . To expand Euclidean space , ComplEx [ 21 ] firstly introduces complex vector space which can capture both symmetric and asymmetric relations . RotatE [ 20 ] also models in complex space and can capture additional inversion and composition patterns by introducing rotational Hadmard product . QuatE [ 29 ] extends RotatE , using a quaternion inner product and gains more expressive semantic learning capability . ATTH [ 4 ] proposes a low-dimensional hyperbolic knowledge graph embedding method , which capture logical patterns such as symmetry and asymmetrical . However , none of existing models is capable of modeling transitivity relation pattern . We are the first to show that the transitivity can be modeled with projections , and we prove that the proposed model is able to infer all the five relation patterns shown in Table 2 . 3 Rot-Pro : Modeling Relation Patterns by Projection and Rotation . 3.1 Preliminary . RotatE is a representative approach that models a relation as an element-wise rotation from the embedding eh of head entity to the embedding et of tail entity in complex vector space . This can be denoted as et ( k ) ≈ rot ( eh ( k ) , θr ( k ) ) , where θr is the embedding of relation r and rot ( eh ( k ) , θr ( k ) ) is the rotation function that rotates the kth element of eh with a phase of the kth element of θr . For each embedding e , let Re ( e ( k ) ) and Im ( e ( k ) ) be the real number and imaginary number of its kth dimension , the rotation transformation in the kth dimension is defined by an orthogonal matrix whose determinant is 1 , as follows : [ Re ( et ( k ) ) Im ( et ( k ) ) ] ≈ [ Re ( rot ( eh ( k ) , θr ( k ) ) ) Im ( rot ( eh ( k ) , θr ( k ) ) ) ] = [ cos θr ( k ) − sin θr ( k ) sin θr ( k ) cos θr ( k ) ] [ Re ( eh ( k ) ) Im ( eh ( k ) ) ] . ( 1 ) It has been proved that RotatE can infer symmetry , asymmetry , inversion , and composition relation patterns [ 20 ] . However , it can not infer the transitivity pattern , and we will explain in what follows . 3.2 Representation of transitive relation . Relation r is a transitive relation , if for any instances ( e1 , r , e2 ) and ( e2 , r , e3 ) of relation r , ( e1 , r , e3 ) is also an instance of r. For convenience of illustration , we define the transitive chain of a transitive relation as follows . Definition 1 . A transitive chain of r is defined as a chain of instances ( e1 , r , e2 ) , . . . , ( em−1 , r , em ) of r , where e1 , . . . , em are different entities . For the transitive closure of a transitive chain , every two entities in the chain should be connected by the transitive relation r. Hence , it can be represented as a fully connected directed graph with m ( m−1 ) 2 edges . It can be proved that a transitive relation can be represented as the union of transitive closures of all transitive chains . Thus , the representation of transitive relations can be reduced to the representation of transitive chains . An example of a transitive chain and its transitive closure are shown in Figure 1 ( a ) , where ( e1 , r , e2 ) , ( e2 , r , e3 ) , ( e3 , r , e4 ) form a transitive chain , ( e1 , r , e3 ) and ( e2 , r , e4 ) are instances derived by transitivity via one-hop , and ( e1 , r , e4 ) is the only instance derived via two-hops . Due to the speciality of transitivity , current models are unable to effectively model such transformation in vector space . For instance , in TransE ( Figure 1 ( b ) ) , where a relation is regarded as a translation between the head and tail entities , it requires the translation to be a zero vector to model transitivity , which forces the embeddings of entities in a transitive chain to be the same . Thus , it can not model transitivity . In RotatE ( Figure 1 ( c ) ) , it requires the relational rotation phase θr ( k ) in each dimension to be 2nπ ( n = 0 , 1 , . . . ) to model transitivity , which also forces the embeddings of entities to be the same in a transitive chain . Our solution . Based on the observation on transitive closures of transitive chains , in each transitive chain ( e1 , r , e2 ) , . . . , ( em−1 , r , em ) , for each entity ej , ( ej , r , ej+l ) can be derived by transitivity via l − 1-hops ( 2 6 l 6 m − j ) . If we model each relation r as a kind of transformation Tr , then it requires Tr ( eh ) = et for each relation instance ( h , r , t ) . Therefore , the transformation of a transitive relation must satisfies that T lr ( ej ) = Tr ( ej ) ( 1 6 j 6 m , 1 6 l 6 m − j ) , i.e . the result of transforming an entity embedding multiple times is equivalent to that of transforming it once . This inspires us to model the transitivity pattern in terms of the idempotent transformation ( projection [ 25 ] ) which has the same property . For each relation r , let Sr ( k ) be an invertible matrix on the kth dimension , a general orthogonal projection is defined by the idempotent matrix : Mr ( k ) = Sr ( k ) −1 [ ar ( k ) 0 0 br ( k ) ] Sr ( k ) , ( ar ( k ) , br ( k ) ∈ { 0 , 1 } ) . ( 2 ) Without loss of generality , we simply set Sr ( k ) = [ cos θp ( k ) − sin θp ( k ) sin θp ( k ) cos θp ( k ) ] to be a rotation matrix , which rotates the original axis by a phase θp ( k ) . The orthogonal projection pr ( k ) defined by Mr ( k ) is performed in the new axis after rotation : pr ( k ) ( x+ yi ) = [ 1 i ] Mr ( k ) [ x y ] . ( 3 ) In the rest of paper , we will omit the dimensional indices ( k ) in Mr ( k ) and pr ( k ) for simplicity . In this way , for entities e1 , . . . , em in a transitive chain , we have plr ( ej ( k ) ) = pr ( ej ( k ) ) ( 1 6 j 6 m , 1 6 l 6 m− j ) . This implies that pr ( e1 ( k ) ) = · · · = pr ( em ( k ) ) , which does not force the entity embeddings e1 , . . . , em to be the same . The embeddings e1 , . . . , em can be different to each other and have the same projected vector under pr .

Summary: Summary: This study proposes a new geometric model Rot-Pro that enables the learning of transitive inference patterns by combining rotation and projection. The core idea is to involve a learnable idempotent projection onto a reduced basis. This achieves the transitivity pattern without forcing the entity embeddings to be the same as it is in previous models. The study shows the effectiveness in experimental results mostly on the small Countries dataset, but also shows that the model performs well on larger KG benchmarks especially on the Yago3-10 dataset. The authors also report their investigation into initialization schemes that are required for successfully training the model.