Rot-Pro: Modeling Transitivity by Projection in Knowledge Graph Embedding
Year: 2022

Abstract:
Knowledge graph embedding models learn the representations of entities and relations in the knowledge graphs for predicting missing links (relations) between entities. Their effectiveness are deeply affected by the ability of modeling and inferring different relation patterns such as symmetry, asymmetry, inversion, composition and transitivity. Although existing models are already able to model many of these relations patterns, transitivity, a very common relation pattern, is still not been fully supported. In this paper, we first theoretically show that the transitive relations can be modeled with projections. We then propose the Rot-Pro model which combines the projection and relational rotation together. We prove that Rot-Pro can infer all the above relation patterns. Experimental results show that the proposed Rot-Pro model effectively learns the transitivity pattern and achieves the state-of-the-art results on the link prediction task in the datasets containing transitive relations.

heading: 1 Introduction
text: Knowledge graph embedding (KGE) aims to learn low-dimensional dense vectors to express the entities and relations in the knowledge graphs (KG). It is widely used in recommendation system, question answering, dialogue systems [5, 17, 7]. The general intuition of KGE is to model and infer relations between entities in knowledge graphs, which has complex patterns such as symmetry, asymmetry, inversion, composition, and transitivity as shown in Table 1.
Many studies dedicate to find a method, which is able to model various relation patterns [3, 2, 21, 20]. TransE models relations as translations, aims to model the inversion and composition patterns; DisMult can model symmetric relations by capturing interactions between head and tail entities and relations. One representative model proposed recently is RotatE [20], which is proved to be able to model symmetry, asymmetry, inversion and composition patterns by modeling relation as a rotation in the complex plane. However, none of them can model all the five relation patterns, especially the transitivity pattern.
This paper focus on modeling the transitivity pattern. We theoretically show that the transitive relations can be modeled with idempotent transformations, i.e. projections [25]. Any projection matrix is similar to a diagonal matrix with elements in the diagonal being 0 or 1. We design the projection by constraining the similarity matrix to be a rotation matrix, which has less parameters to learn.
In order to model not only transitivity but also other relation patterns shown in Table 1, we propose the Rot-Pro model which combines the projection and relational rotation together. We theoretically prove that Rot-Pro can infer the symmetry, asymmetry, inversion, composition, and transitivity ∗Corresponding author.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
patterns. Experimental results show that the Rot-Pro model can effectively learn the transitivity pattern. The Rot-Pro model achieves the state-of-the-art results on the link prediction task in the Countries dataset containing transitive relations and outperforms other models in the YAGO3-10 and FB15k-237 dataset.

heading: 2 Related work
text: There are mainly two types of knowledge graph embedding models, either using translation transformation or linear transformation between head and tail entities.
Trans-series models. Trans-series models, which is well-known in KGE area, are essentially translation transformation based models. TransE [3] proposed a pure translation distance-based score function, which assumes the added embedding of head entity h and relation r should be close to the embedding of tail entity t. This simple approach is effective in capturing composition, asymmetric and inversion relations, but is hard to handle the 1-to-N, N-to-1 and N-N relations.
To overcome these issues, many variants and extensions of TransE have been proposed. TransH [26] projects entities and relations into a relation-specific hyperplanes and enables different projections of an entity in different relations. TransR [28] introduces relation-specific spaces, which builds entity and relation embeddings in different spaces separately. TransD [9] simplifies TransR by constructs dynamic mapping matrices. For the purpose of model optimization, some models relax the requirement for translational distance. TransA [10] replaces Euclidean distance by Mahalanobis distance to enable more adaptive metric learning, and a recent model TransMS [18] transmits multidirectional semantics by complex relations.
The variants of TransE improve the capability of the models to handle 1-to-N, N-to-1 and N-N relations as well as effectively modeling symmetric and asymmetric relations, but they are no longer able to model composition and inversion relations as they do linear transformation on head and tail entities separately before modeling the relation as translation. BoxE [1], a recent trans-series model, embeds entities as points, and relations as a set of boxes, for yielding a model that could express multiple relation patterns including composition and inversion, but it cannot express transitivity.
Bilinear models. Models of bilinear series model relations as linear transformation matrix Mr from head to tail entity. The type of relation patterns that a linear transformation based model can infer depends on the property of Mr. RESCAL [14] proposes the transformation of relation as a matrix that models the pairwise interactions between entities and relation. The score of a fact is defined by a bilinear function: fr = hTMrt. DistMult [2] simplifies RESCAL by restricting Mr to diagonal matrices. Therefore, it cannot handle other types of relations except symmetry. HolE
[12] combines the expressive power of RESCAL with the simplicity of DistMult which introduces circular correlations. HolE can express multiple types of relations, since cyclic correlation operations are not commutative.
Recently, some KGE models begin to model relation patterns explicitly. Dihedral [27] models relations in KGs with the representation of dihedral group that has properties to support the relation as symmetry. To expand Euclidean space, ComplEx [21] firstly introduces complex vector space which can capture both symmetric and asymmetric relations. RotatE [20] also models in complex space and can capture additional inversion and composition patterns by introducing rotational Hadmard product. QuatE [29] extends RotatE, using a quaternion inner product and gains more expressive semantic learning capability. ATTH [4] proposes a low-dimensional hyperbolic knowledge graph embedding method, which capture logical patterns such as symmetry and asymmetrical.
However, none of existing models is capable of modeling transitivity relation pattern. We are the first to show that the transitivity can be modeled with projections, and we prove that the proposed model is able to infer all the five relation patterns shown in Table 2.

heading: 3 Rot-Pro: Modeling Relation Patterns by Projection and Rotation

heading: 3.1 Preliminary
text: RotatE is a representative approach that models a relation as an element-wise rotation from the embedding eh of head entity to the embedding et of tail entity in complex vector space. This can be denoted as et(k) ≈ rot(eh(k), θr(k)), where θr is the embedding of relation r and rot(eh(k), θr(k)) is the rotation function that rotates the kth element of eh with a phase of the kth element of θr. For each embedding e, let Re(e(k)) and Im(e(k)) be the real number and imaginary number of its kth dimension, the rotation transformation in the kth dimension is defined by an orthogonal matrix whose determinant is 1, as follows:[
Re(et(k)) Im(et(k)) ] ≈ [ Re(rot(eh(k), θr(k))) Im(rot(eh(k), θr(k))) ] = [ cos θr(k) − sin θr(k) sin θr(k) cos θr(k) ] [ Re(eh(k)) Im(eh(k)) ] . (1)
It has been proved that RotatE can infer symmetry, asymmetry, inversion, and composition relation patterns [20]. However, it cannot infer the transitivity pattern, and we will explain in what follows.

heading: 3.2 Representation of transitive relation
text: Relation r is a transitive relation, if for any instances (e1, r, e2) and (e2, r, e3) of relation r, (e1, r, e3) is also an instance of r. For convenience of illustration, we define the transitive chain of a transitive relation as follows.
Definition 1. A transitive chain of r is defined as a chain of instances (e1, r, e2), . . . , (em−1, r, em) of r, where e1, . . . , em are different entities.
For the transitive closure of a transitive chain, every two entities in the chain should be connected by the transitive relation r. Hence, it can be represented as a fully connected directed graph with m(m−1)
2 edges. It can be proved that a transitive relation can be represented as the union of transitive closures of all transitive chains. Thus, the representation of transitive relations can be reduced to the representation of transitive chains.
An example of a transitive chain and its transitive closure are shown in Figure 1(a), where (e1, r, e2), (e2, r, e3), (e3, r, e4) form a transitive chain, (e1, r, e3) and (e2, r, e4) are instances derived by transitivity via one-hop, and (e1, r, e4) is the only instance derived via two-hops.
Due to the speciality of transitivity, current models are unable to effectively model such transformation in vector space. For instance, in TransE (Figure 1(b)), where a relation is regarded as a translation between the head and tail entities, it requires the translation to be a zero vector to model transitivity, which forces the embeddings of entities in a transitive chain to be the same. Thus, it cannot model transitivity. In RotatE (Figure 1(c)), it requires the relational rotation phase θr(k) in each dimension to be 2nπ (n = 0, 1, . . .) to model transitivity, which also forces the embeddings of entities to be the same in a transitive chain.
Our solution. Based on the observation on transitive closures of transitive chains, in each transitive chain (e1, r, e2), . . . , (em−1, r, em), for each entity ej , (ej , r, ej+l) can be derived by transitivity via l − 1-hops (2 6 l 6 m − j). If we model each relation r as a kind of transformation Tr, then it requires Tr(eh) = et for each relation instance (h, r, t). Therefore, the transformation of a transitive relation must satisfies that T lr(ej) = Tr(ej) (1 6 j 6 m, 1 6 l 6 m − j), i.e. the result of transforming an entity embedding multiple times is equivalent to that of transforming it once. This inspires us to model the transitivity pattern in terms of the idempotent transformation (projection [25]) which has the same property. For each relation r, let Sr(k) be an invertible matrix on the kth dimension, a general orthogonal projection is defined by the idempotent matrix:
Mr(k) = Sr(k) −1 [ ar(k) 0 0 br(k) ] Sr(k), (ar(k), br(k) ∈ {0, 1}). (2)
Without loss of generality, we simply set Sr(k) = [ cos θp(k) − sin θp(k) sin θp(k) cos θp(k) ] to be a rotation matrix, which rotates the original axis by a phase θp(k). The orthogonal projection pr(k) defined by Mr(k) is performed in the new axis after rotation:
pr(k)(x+ yi) = [1 i]Mr(k) [ x y ] . (3)
In the rest of paper, we will omit the dimensional indices (k) in Mr(k) and pr(k) for simplicity. In this way, for entities e1, . . . , em in a transitive chain, we have plr(ej(k)) = pr(ej(k)) (1 6 j 6 m, 1 6 l 6 m− j). This implies that pr(e1(k)) = · · · = pr(em(k)), which does not force the entity embeddings e1, . . . , em to be the same. The embeddings e1, . . . , em can be different to each other and have the same projected vector under pr.

heading: 3.3 The Rot-Pro model
text: Model formulation. In order to model not only transitivity but also other relation patterns shown in Table 1, we combine the above projection based representation for transitivity and the relational rotation based representation for symmetry, asymmetry, inversion, and composition together. We propose Rot-Pro to model relations as relational rotations on the projected entity embeddings on complex space Cd. For each triple (h, r, t), the Rot-Pro model requires that
rot(pr(eh(k)), θr(k)) = pr(et(k)). (4) We demonstrate in the following theorem that Rot-Pro enables the modeling and inferring of all the five types of relation patterns introduced above. Theorem 1. Rot-Pro can infer the symmetry, asymmetry, inversion, composition, and transitivity patterns.
Proof. (1) Let ar = 1 and br = 1, M becomes an identity matrix and p becomes an identity transformation, and our model is reduced to the RotatE model. Thus, Rot-Pro can also infer the symmetry, asymmetry, inversion and composition patterns as RotatE does [20].
(2) Here we will construct the solutions in Rot-Pro model for transitive relations. Let ar = 1 and br = 0, pr is a projection to the real axis x′ as shown in Figure 2 2. As discussed in previous section, to model the transitivity of relation r, the projected entity embeddings in a transitive chain must satisfy that rot(p(rot(p(ej(k)), θr(k))), θr(k)) = rot(pr(ej(k)), θr(k)) = pr(ej+2(k)) = pr(ej+1(k)). Therefore, the phase of relational rotation θr(k) can only be 2nπ (n = 0, 1, 2, . . .) and pr(ej(k)) = pr(e1(k)) for any 1 < j 6 m.
According to Equation 4, the following equation is expected to hold.
Mr [ Re(ej(k)) Im(ej(k)) ] = [ cos θr(k) − sin θr(k) sin θr(k) cos θr(k) ]( Mr [ Re(e1(k)) Im(e1(k)) ]) =Mr [ Re(e1(k)) Im(e1(k)) ] (5)
The above equation holds iff
cos θp(k)Re(ej(k))− sin θp(k)Im(ej(k)) = cos θp(k)Re(e1(k))− sin θp(k)Im(e1(k)). (6)
This equation holds if for any ej in the transitive chain,
cos θp(i)Re(ej(k))− sin θp(k)Im(ej(k)) = ck,
where ck = cos θp(k)Re(e1(k)) − sin θp(k)Im(e1(k)) is a constant. That is, all these entity embeddings ej(k) = x+ yi in the transitive chain are located in the line defined by Equation (7) on the complex plane as shown in Figure 2.
cos θp(k)x− sin θp(k)y = ck. (7)
Here, different value of ck can represent different transitive chain. In summary, we construct the solutions for representing transitivity in Rot-Pro model, i.e. ar(k) = 1, br(k) = 0, θr(k) = 2nπ, and for any entity embedding ej , it satisfies that cos θp(i)Re(ej(k)) − sin θp(k)Im(ej(k)) = ck, where ck is a constant.
Score function. For each triple (h, r, t), the distance function of the Rot-Pro model is defined as following: dr(eh, et) = ‖rot(pr(eh), θr)− pr(et)‖. (8) The score function fr(eh, et) = −dr(eh, et).

heading: 3.4 Optimization objective
text: In the training process, we adopt the self-adversarial negative sampling, which has been proved as an effective optimization approach to KGE [20]. The negative sampling loss Ls with self-adversarial training is defined as:
Ls = − log σ(γ − dr(h, t))− n∑ j=1 p(h′j , r, t ′ j) log σ(dr(h ′ j , t ′ j)− γ) (9)
2We can also set a = 0 and b = 1, then p is a projection to the imaginary axis y′.
where γ is a fixed margin, σ is the sigmoid function, (h′j , r, t ′ j) is the jth negative instance and p(h′j , r, t ′ j) is the distribution for negative sampling [20].
In addition, to ensure the learned matrix to be a projection, the values of a and b in Equation 2 should be restricted to 0 or 1. To enforce such constraint, we proposed a projection penalty loss as follows:
Lp = |R|∑ j=1 (||(aj − 1.0) aj qj ||2 + ||(bj − 1.0) bj qj ||2). (10)
Here |R| is the number of relations, is the Hadamard product, and qj = {qj(k)}dk=1, where qj(k) = 1 if [(xj(k)− 1.0) (xj(k)− 0.0)] < γm, otherwise qj(k) = β > 1. Here γm and β are hyper-parameters. We define qj to impose more penalty to values which are far from 0 or 1 than that of values which are close to 0 or 1.
Let α be a hyper-parameter, the total loss is defined as the weighted average of the above two losses.
L = Ls + α · Lp. (11)

heading: 4 Experiments

heading: 4.1 Datasets
text: We evaluate the Rot-Pro model on four well-known benchmarks. In general, FB15k-237 and WN18RR are two widely-used benchmarks and YAGO3-10 and Countries are two benchmarks with abundant relation patterns including transitivity.
• FB15k-237: Freebase [11] contains information including people, media, geographical and locations. FB15k is a subset of Freebase and FB15k-237 [24] is a modified version of FB15k, which excludes inverse relations to resolve a flaw with FB15k [23]. It contains 14,541 entities, 237 relations, and 272,115 training triples.
• WN18RR: WN18RR [23] is a subset of WN18 [3] from WordNet [15]. WordNet is a dataset that characterizes associations between English words. Compared with WN18, WN18RR retains most of the symmetric, asymmetric and compositional relations, while removing the inversion relations. It contains 40,943 entities, 11 relations, and 86,835 training triples.
• YAGO3-10: YAGO [19] is a dataset which integrates vocabulary definitions of WordNet with classification system of Wikipedia. YAGO3-10 [13] is a subset of YAGO, which contains 123,182 entities, 37 relations and 1,079,040 training triples. According to the ontology of YAGO3, it contains almost all common relation patterns.
• Countries: Countries [8] is a relatively small-scale dataset, which contains 2 relations and 272 entities (244 countries, 5 regions and 23 sub-regions). The two relations of Countries are locatedIn and neighborOf, which are transitive and symmetric relations respectively. The Countries dataset has 3 tasks, each requiring inferring a composition pattern with increasing length and difficulty.

heading: 4.2 Evaluation protocol
text: We evaluate the KGE models on three common evaluation metrics: mean rank (MR), mean reciprocal rank (MRR), and top-k Hit Ratio (Hit@k). For each valid triples (h, r, t) in the test set, we replace either h or t with every other entities in the dataset to create corrupted triples in the link prediction task. Following previous work [3, 23, 6, 29, 16], all the models are evaluated in a filtered setting, i.e, corrupt triples that appear in training, validation, or test sets are removed during ranking. The valid triple and filtered corrupted triples are ranked in ascending order based on their prediction scores. Lower MR, higher MRR or higher Hit@k indicate better performance.

heading: 4.3 Experiment setup
text: With the hyper-parameters introduced, we train Rot-Pro using a grid search of hyper-parameters: fixed margin γ in Equation 9 ∈ {0.1, 4.0, 6.0, 9.0, 16.0, 20.0}, weights tuning hyper-parameters for loss, α ∈ {0.0001, 0.0005, 0.0008}, value of γm in Equation 10 ∈ {1e−6, 5e−6, 1e−5}, value
of β in Equation 10 ∈ {1.3, 1.5, 2.0}. Both the real and imaginary parts of the entity embeddings are uniformly initialized, and the phases of the relational rotations are initialized between {(−π, π), (−π2 , π 2 )}. In some settings, the phases of the relational rotations are also normalized to between {(−π, π), (−π2 , π 2 )} during training.

heading: 4.4 Main results
text: We compare Rot-Pro with several state-of-the-art models, including TransE [3], DistMult [2], ComplEx [21], ConvE [23], as well as RotatE [20] and BoxE [1], to empirically show the importance of being able to model and infer more relation patterns for the task of predicting missing links. Table 3 summarizes our results on FB15k-237 and WN18RR, where results of baseline models are taken from Sun et al [20] and Ralph et al [1]. We can see that Rot-Pro outperforms the baseline models on most evaluation metrics. Compared to RotatE, the improvement of Rot-Pro is limited since there is no sufficient transitive relation defined on these two datasets, but the results are still comparable with other baseline models.
Table 4 summarizes our results on YAGO3-10 and Countries, which contain transitive relations. Hence the improvement of Rot-Pro over RotatE and other linear transformation models is much more significant. Specifically, Rot-Pro obtains better AUC-PR result than existing state-of-the-art approaches, which indicates that Rot-Pro could effectively infer relation patterns such as transitivity, symmetry and composition. As a translation transformation model, BoxE outperforms Rot-Pro on YAGO3-10 on most evaluation metrics, which indicates it is also a strong KGE model for inferring multiple relation patterns. However, the performance of BoxE on specific transitivity test sets is still not comparable with Rot-Pro, where additional experiments can be found in the appendix.

heading: 4.5 Validation of learned representation of transitive relations
text: We conduct further analysis on the the Rot-Pro model to verify that the model can actually learn the representations of transitive relations and have the theoretical property as expected. To do this, we first investigate the distributions of relational rotation phases in all dimensions of entity embeddings obtained by training on YAGO3-10. According to our theoretical analysis, we expect the model could learn to represent transitivity, i.e. for any non-trivial projection (i.e. a = 1, b = 0 or a = 0, b = 1), the corresponding phase of relational rotation should be 2nπ. The experimental results are shown in Figure 3(a). It can be observed that the Rot-Pro model does learn the relational rotation phases 0 and 2π as expected. However, it also learns the unexpected relational rotation phase π. Further
experiments reveal that by turning the initialization range of the relational rotation phases, the problem of learning unexpected relational rotation phase π could be mitigated. By changing the initialization range of relational rotation phases from (−π, π) to (−π2 , π 2 ), the number of relational rotation phases π becomes significantly less. When we further restrict the relational rotation phases to (−π2 , π 2 ) during training, almost all relational rotation phases become 0 or 2π.
The results above are also reflected in the quantitative test. To fully understand the impact of changing initialization range on the performance of the Rot-Pro model on modelling transitive relations, we construct three sub-test sets S1, S2, S3 of YAGO3-10 for evaluation, which consist of a single transitive relation isLocatedIn. Test set S1 contains instances of isLocatedIn in the original test set. Test set S2 is obtained by applying the transitivity once on instances of isLocatedIn in the YAGO3-10 dataset. Test set S3 is constructed similarly to S2, except by applying the transitivity at least twice. We take the RotatE model as baseline, and compare it with three variant of Rot-Pro models with different settings: the first one with relational rotation phase initialized in (−π, π), the second one with relational rotation phase initialized in (−π2 , π 2 ), the third one with relational rotation phase restricted in (−π2 , π 2 ) during training. The experimental result is shown in Figure 4(a). It shows that tuning of initialization range also largely improves the performance of the Rot-Pro model, which
coincides with the improvement of learning correct representations of transitive relations. All the variant of Rot-Pro models outperform RotatE significantly, especially when the relational rotation phase is restricted to (−π2 , π 2 ) during training.
We also visualize one dimension of embeddings of three entities connected by a transitive relation isLocatedIn in YAGO3-10. Figure 4(c) shows the visualization of entity embeddings of the Rot-Pro model trained with initialization range (−π, π), which contains a miss placed entity embedding. While Figure 4(d) is the visualization of embeddings of entities of the Rot-Pro model trained by restricting the relational rotation phase to (−π2 , π 2 ) during training, where all entity embeddings in the transitive chain are placed correctly as expected. We can see that these vectors are basically fit in a line and can almost be projected to the same vector in the rotated axis.
Explanation. The Rot-Pro model with no additional restrictions on the relational rotation phase may learn a phase π which does not exactly meet our expectation. A possible representations of four entities in a transitive chain with relational rotation phase π is illustrated in Figure 5(a), in which four out of the six instances of transitive relation are correctly represented, while the other two instances (e1, r, e3) and (e2, r, e4) are not. Obviously, this is not an optimal solution for the model, and the reason is likely to be that the model falls into a local optimum during the learning process. To demonstrate this, we plot the variation of loss for three triples in a transitive chain with the relational rotation phase range over (0, 2π). The result is shown in Figure 5(b). We can find that there is indeed a local optimum at π, and the global optimum is at 0 and 2π, which is consistent with our conjecture.

heading: 4.6 Limitation
text: According to the experimental results, Rot-Pro is sensitive to the range of relational rotation phases, and hence prone to fall into the local optimum solution. Though it can learn the idempotency
of transitivity correctly by enforcing addition constraints on training, however, such constraints have negative impact on the learning of other relation patterns, such as symmetry. We can find in Figure 6(a) that for a symmetric relation, the relational rotation phases learned by a Rot-Pro without phase constraint are either 0, π or 2π, which indicates that it has similar capability of modeling and inferring symmetry relation pattern as RotatE. By narrowing of the range of relational rotation phases, the histogram on the symmetry relation is gradually disrupted as shown in Figure 6(b) and 6(c). Therefore, a trade-off should be made between the better modeling and inferring of transitivity and the other relation patterns. Such limitation might be further optimized through learning each relation pattern separately and integrate through mechanisms such as attention, which we will study in future works.

heading: 5 Conclusion and Future Work
text: In this paper, we theoretically showed that the transitive relations can be modeled with projections that is an idempotent transformation. We also theoretically proved that the proposed Rot-Pro model is able to infer the symmetry, asymmetry, inversion, composition, and transitivity patterns. Our experimental results empirically showed that the Rot-Pro model can effectively learn the transitivity pattern. Our model also has the potential to be improved by extending the complex space to higher dimension space, such as quaternion space [29, 22]. While the proof of expressiveness in many previous works is focused on the expressiveness of each relation pattern separately, it is worthwhile to further investigate whether a model can handle all common relation patterns simultaneously, considering that a single relation may exhibit multiple relation patterns and different relation patterns may have complex interactions in knowledge graphs.
Acknowledgments and Disclosure of Funding
This work was supported by the National Key Research and Development Plan of China (Grant No. 2018AAA0102301) and the National Natural Science Foundation of China (Grant No. 61690202).

Review summaries:
This paper is related to the problem of Knowledge Graph (KG) Embedding. It seems that none of the existing approaches for KG embedding is capable of modeling the transitivity property of the relation which is an important property for many relations to have in real-world KGs. The aim of this paper, therefore, is to bridge this specific research gap.   This paper proposes a KG embedding model called __Rot-Pro__ which not only is capable of modeling the transitive property of a relation but also other properties including symmetry, asymmetry, inversion, and composition. The approach is based on applying a projection operator on the embedding of the head and tail entities before applying RotatE so as to be able to model transitive relations. The experiments are performed on two well-known datasets FB15k-237 and WN18RR. The experiments are also performed on two other datasets called YACO3-10 and Countries because they have lots of transitive relations present in them.   On the datasets, where transitive relations are dominant, the approach seems to do justice but the approach seems to be facing some hiccups in the presence of other types of relations such as symmetry.

When embedding entities and relations in knowledge graphs, operations such as translations, matrix-vector products, quadratic forms, etc are used to represent relations between entity vectors. While these vector-based methods are flexible, they do not specifically encode an inductive bias towards transitive relations that is necessary for modeling certain hierarchies. This paper proposes an approach based on projections (idempotent transformations) and rotations taken together, which allows it to exactly model transitivity in relations.

Summary: This study proposes a new geometric model Rot-Pro that enables the learning of transitive inference patterns by combining rotation and projection. The core idea is to involve a learnable idempotent projection onto a reduced basis. This achieves the transitivity pattern without forcing the entity embeddings to be the same as it is in previous models. The study shows the effectiveness in experimental results mostly on the small Countries dataset, but also shows that the model performs well on larger KG benchmarks especially on the Yago3-10 dataset. The authors also report their investigation into initialization schemes that are required for successfully training the model.