Introduction
Knowledge graphs are usually collections of factual triples—(head entity, relation, tail entity), which represent human knowledge in a structured way. In the past few years, we have witnessed the great achievement of knowledge graphs in many areas, such as natural language processing BIBREF0, question answering BIBREF1, and recommendation systems BIBREF2.
Although commonly used knowledge graphs contain billions of triples, they still suffer from the incompleteness problem that a lot of valid triples are missing, as it is impractical to find all valid triples manually. Therefore, knowledge graph completion, also known as link prediction in knowledge graphs, has attracted much attention recently. Link prediction aims to automatically predict missing links between entities based on known links. It is a challenging task as we not only need to predict whether there is a relation between two entities, but also need to determine which relation it is.
Inspired by word embeddings BIBREF3 that can well capture semantic meaning of words, researchers turn to distributed representations of knowledge graphs (aka, knowledge graph embeddings) to deal with the link prediction problem. Knowledge graph embeddings regard entities and relations as low dimensional vectors (or matrices, tensors), which can be stored and computed efficiently. Moreover, like in the case of word embeddings, knowledge graph embeddings can preserve the semantics and inherent structures of entities and relations. Therefore, other than the link prediction task, knowledge graph embeddings can also be used in various downstream tasks, such as triple classification BIBREF4, relation inference BIBREF5, and search personalization BIBREF6.
The success of existing knowledge graph embedding models heavily relies on their ability to model connectivity patterns of the relations, such as symmetry/antisymmetry, inversion, and composition BIBREF7. For example, TransE BIBREF8, which represent relations as translations, can model the inversion and composition patterns. DistMult BIBREF9, which models the three-way interactions between head entities, relations, and tail entities, can model the symmetry pattern. RotatE BIBREF7, which represents entities as points in a complex space and relations as rotations, can model relation patterns including symmetry/antisymmetry, inversion, and composition. However, many existing models fail to model semantic hierarchies in knowledge graphs.
Semantic hierarchy is a ubiquitous property in knowledge graphs. For instance, WordNet BIBREF10 contains the triple [arbor/cassia/palm, hypernym, tree], where “tree” is at a higher level than “arbor/cassia/palm” in the hierarchy. Freebase BIBREF11 contains the triple [England, /location/location/contains, Pontefract/Lancaster], where “Pontefract/Lancaster” is at a lower level than “England” in the hierarchy. Although there exists some work that takes the hierarchy structures into account BIBREF12, BIBREF13, they usually require additional data or process to obtain the hierarchy information. Therefore, it is still challenging to find an approach that is capable of modeling the semantic hierarchy automatically and effectively.
In this paper, we propose a novel knowledge graph embedding model—namely, Hierarchy-Aware Knowledge Graph Embedding (HAKE). To model the semantic hierarchies, HAKE is expected to distinguish entities in two categories: (a) at different levels of the hierarchy; (b) at the same level of the hierarchy. Inspired by the fact that entities that have the hierarchical properties can be viewed as a tree, we can use the depth of a node (entity) to model different levels of the hierarchy. Thus, we use modulus information to model entities in the category (a), as the size of moduli can reflect the depth. Under the above settings, entities in the category (b) will have roughly the same modulus, which is hard to distinguish. Inspired by the fact that the points on the same circle can have different phases, we use phase information to model entities in the category (b). Combining the modulus and phase information, HAKE maps entities into the polar coordinate system, where the radial coordinate corresponds to the modulus information and the angular coordinate corresponds to the phase information. Experiments show that our proposed HAKE model can not only clearly distinguish the semantic hierarchies of entities, but also significantly and consistently outperform several state-of-the-art methods on the benchmark datasets.
Notations Throughout this paper, we use lower-case letters $h$, $r$, and $t$ to represent head entities, relations, and tail entities, respectively. The triplet $(h,r,t)$ denotes a fact in knowledge graphs. The corresponding boldface lower-case letters $\textbf {h}$, $\textbf {r}$ and $\textbf {t}$ denote the embeddings (vectors) of head entities, relations, and tail entities. The $i$-th entry of a vector $\textbf {h}$ is denoted as $[\textbf {h}]_i$. Let $k$ denote the embedding dimension.
Let $\circ :\mathbb {R}^n\times \mathbb {R}^n\rightarrow \mathbb {R}^n$ denote the Hadamard product between two vectors, that is,
and $\Vert \cdot \Vert _1$, $\Vert \cdot \Vert _2$ denote the $\ell _1$ and $\ell _2$ norm, respectively.
Related Work
In this section, we will describe the related work and the key differences between them and our work in two aspects—the model category and the way to model hierarchy structures in knowledge graphs.
Related Work ::: Model Category
Roughly speaking, we can divide knowledge graph embedding models into three categories—translational distance models, bilinear models, and neural network based models. Table TABREF2 exhibits several popular models.
Translational distance models describe relations as translations from source entities to target entities. TransE BIBREF8 supposes that entities and relations satisfy $\textbf {h}+\textbf {r}\approx \textbf {t}$, where $\textbf {h}, \textbf {r}, \textbf {t} \in \mathbb {R}^n$, and defines the corresponding score function as $f_r(\textbf {h},\textbf {t})=-\Vert \textbf {h}+\textbf {r}-\textbf {t}\Vert _{1/2}$. However, TransE does not perform well on 1-N, N-1 and N-N relations BIBREF14. TransH BIBREF14 overcomes the many-to-many relation problem by allowing entities to have distinct representations given different relations. The score function is defined as $f_r(\textbf {h},\textbf {t})=-\Vert \textbf {h}_{\perp }+\textbf {r}-\textbf {t}_{\perp }\Vert _2$, where $\textbf {h}_{\perp }$ and $\textbf {t}_{\perp }$ are the projections of entities onto relation-specific hyperplanes. ManifoldE BIBREF15 deals with many-to-many problems by relaxing the hypothesis $\textbf {h}+\textbf {r}\approx \textbf {t}$ to $\Vert \textbf {h}+\textbf {r}-\textbf {t}\Vert _2^2\approx \theta _r^2$ for each valid triple. In this way, the candidate entities can lie on a manifold instead of exact point. The corresponding score function is defined as $f_r(\textbf {h},\textbf {t})=-(\Vert \textbf {h}+\textbf {r}-\textbf {t}\Vert _2^2-\theta _r^2)^2$. More recently, to better model symmetric and antisymmetric relations, RotatE BIBREF7 defines each relation as a rotation from source entities to target entities in a complex vector space. The score function is defined as $f_r(\textbf {h},\textbf {t})=-\Vert \textbf {h}\circ \textbf {r}-\textbf {t}\Vert _1$, where $\textbf {h},\textbf {r},\textbf {t}\in \mathbb {C}^k$ and $|[\textbf {r}]_i|=1$.
Bilinear models product-based score functions to match latent semantics of entities and relations embodied in their vector space representations. RESCAL BIBREF16 represents each relation as a full rank matrix, and defines the score function as $f_r(\textbf {h},\textbf {t})=\textbf {h}^\top \textbf {M}_r \textbf {t}$, which can also be seen as a bilinear function. As full rank matrices are prone to overfitting, recent works turn to make additional assumptions on $\textbf {M}_r$. For example, DistMult BIBREF9 assumes $\textbf {M}_r$ to be a diagonal matrix, and ANALOGY BIBREF19 supposes that $\textbf {M}_r$ is normal. However, these simplified models are usually less expressive and not powerful enough for general knowledge graphs. Differently, ComplEx BIBREF17 extends DistMult by introducing complex-valued embeddings to better model asymmetric and inverse relations. HolE BIBREF20 combines the expressive power of RESCAL with the efficiency and simplicity of DistMult by using the circular correlation operation.
Neural network based models have received greater attention in recent years. For example, MLP BIBREF21 and NTN BIBREF22 use a fully connected neural network to determine the scores of given triples. ConvE BIBREF18 and ConvKB BIBREF23 employ convolutional neural networks to define score functions. Recently, graph convolutional networks are also introduced, as knowledge graphs obviously have graph structures BIBREF24.
Our proposed model HAKE belongs to the translational distance models. More specifically, HAKE shares similarities with RotatE BIBREF7, in which the authors claim that they use both modulus and phase information. However, there exist two major differences between RotatE and HAKE. Detailed differences are as follows.
The aims are different. RotatE aims to model the relation patterns including symmetry/antisymmetry, inversion, and composition. HAKE aims to model the semantic hierarchy, while it can also model all the relation patterns mentioned above.
The ways to use modulus information are different. RotatE models relations as rotations in the complex space, which encourages two linked entities to have the same modulus, no matter what the relation is. The different moduli in RotatE come from the inaccuracy in training. Instead, HAKE explicitly models the modulus information, which significantly outperforms RotatE in distinguishing entities at different levels of the hierarchy.
Related Work ::: The Ways to Model Hierarchy Structures
Another related problem is how to model hierarchy structures in knowledge graphs. Some recent work considers the problem in different ways. BIBREF25 embed entities and categories jointly into a semantic space and designs models for the concept categorization and dataless hierarchical classification tasks. BIBREF13 use clustering algorithms to model the hierarchical relation structures. BIBREF12 proposed TKRL, which embeds the type information into knowledge graph embeddings. That is, TKRL requires additional hierarchical type information for entities.
Different from the previous work, our work
considers the link prediction task, which is a more common task for knowledge graph embeddings;
can automatically learn the semantic hierarchy in knowledge graphs without using clustering algorithms;
does not require any additional information other than the triples in knowledge graphs.
The Proposed HAKE
In this section, we introduce our proposed model HAKE. We first introduce two categories of entities that reflect the semantic hierarchies in knowledge graphs. Afterwards, we introduce our proposed HAKE that can model entities in both of the categories.
The Proposed HAKE ::: Two Categories of Entities
To model the semantic hierarchies of knowledge graphs, a knowledge graph embedding model must be capable of distinguishing entities in the following two categories.
Entities at different levels of the hierarchy. For example, “mammal” and “dog”, “run” and ”move”.
Entities at the same level of the hierarchy. For example, “rose” and “peony”, “truck” and ”lorry”.
The Proposed HAKE ::: Hierarchy-Aware Knowledge Graph Embedding
To model both of the above categories, we propose a hierarchy-aware knowledge graph embedding model—HAKE. HAKE consists of two parts—the modulus part and the phase part—which aim to model entities in the two different categories, respectively. Figure FIGREF13 gives an illustration of the proposed model.
To distinguish embeddings in the different parts, we use $\textbf {e}_m$ ($\textbf {e}$ can be $\textbf {h}$ or $\textbf {t}$) and $\textbf {r}_m$ to denote the entity embedding and relation embedding in the modulus part, and use $\textbf {e}_p$ ($\textbf {e}$ can be $\textbf {h}$ or $\textbf {t}$) and $\textbf {r}_p$ to denote the entity embedding and relation embedding in the phase part.
The modulus part aims to model the entities at different levels of the hierarchy. Inspired by the fact that entities that have hierarchical property can be viewed as a tree, we can use the depth of a node (entity) to model different levels of the hierarchy. Therefore, we use modulus information to model entities in the category (a), as moduli can reflect the depth in a tree. Specifically, we regard each entry of $\textbf {h}_m$ and $\textbf {t}_m$, that is, $[\textbf {h}_m]_i$ and $[\textbf {t}_m]_i$, as a modulus, and regard each entry of $\textbf {r}_m$, that is, $[\textbf {r}]_i$, as a scaling transformation between two moduli. We can formulate the modulus part as follows:
The corresponding distance function is:
Note that we allow the entries of entity embeddings to be negative but restrict the entries of relation embeddings to be positive. This is because that the signs of entity embeddings can help us to predict whether there exists a relation between two entities. For example, if there exists a relation $r$ between $h$ and $t_1$, and no relation between $h$ and $t_2$, then $(h, r, t_1)$ is a positive sample and $(h, r, t_2)$ is a negative sample. Our goal is to minimize $d_r(\textbf {h}_m, \textbf {t}_{1,m})$ and maximize $d_r(\textbf {h}_m, \textbf {t}_{2,m})$, so as to make a clear distinction between positive and negative samples. For the positive sample, $[\textbf {h}]_i$ and $[\textbf {t}_1]_i$ tend to share the same sign, as $[\textbf {r}_m]_i>0$. For the negative sample, the signs of $[\textbf {h}_m]_i$ and $[\textbf {t}_{2,m}]_i$ can be different if we initialize their signs randomly. In this way, $d_r(\textbf {h}_m, \textbf {t}_{2,m})$ is more likely to be larger than $d_r(\textbf {h}_m, \textbf {t}_{1,m})$, which is exactly what we desire. We will validate this argument by experiments in Section 4 of the supplementary material.
Further, we can expect the entities at higher levels of the hierarchy to have smaller modulus, as these entities are more close to the root of the tree.
If we use only the modulus part to embed knowledge graphs, then the entities in the category (b) will have the same modulus. Moreover, suppose that $r$ is a relation that reflects the same semantic hierarchy, then $[\textbf {r}]_i$ will tend to be one, as $h\circ r\circ r=h$ holds for all $h$. Hence, embeddings of the entities in the category (b) tend to be the same, which makes it hard to distinguish these entities. Therefore, a new module is required to model the entities in the category (b).
The phase part aims to model the entities at the same level of the semantic hierarchy. Inspired by the fact that points on the same circle (that is, have the same modulus) can have different phases, we use phase information to distinguish entities in the category (b). Specifically, we regard each entry of $\textbf {h}_p$ and $\textbf {t}_p$, that is, $[\textbf {h}_p]_i$ and $[\textbf {t}_p]_i$ as a phase, and regard each entry of $\textbf {r}_p$, that is, $[\textbf {r}_p]_i$, as a phase transformation. We can formulate the phase part as follows:
The corresponding distance function is:
where $\sin (\cdot )$ is an operation that applies the sine function to each element of the input. Note that we use a sine function to measure the distance between phases instead of using $\Vert \textbf {h}_p+\textbf {r}_p-\textbf {t}_p\Vert _1$, as phases have periodic characteristic. This distance function shares the same formulation with that of pRotatE BIBREF7.
Combining the modulus part and the phase part, HAKE maps entities into the polar coordinate system, where the radial coordinate and the angular coordinates correspond to the modulus part and the phase part, respectively. That is, HAKE maps an entity $h$ to $[\textbf {h}_m;\textbf {h}_p]$, where $\textbf {h}_m$ and $\textbf {h}_p$ are generated by the modulus part and the phase part, respectively, and $[\,\cdot \,; \,\cdot \,]$ denotes the concatenation of two vectors. Obviously, $([\textbf {h}_m]_i,[\textbf {h}_p]_i)$ is a 2D point in the polar coordinate system. Specifically, we formulate HAKE as follows:
The distance function of HAKE is:
where $\lambda \in \mathbb {R}$ is a parameter that learned by the model. The corresponding score function is
When two entities have the same moduli, then the modulus part $d_{r,m}(\textbf {h}_m,\textbf {t}_m)=0$. However, the phase part $d_{r,p}(\textbf {h}_p,\textbf {t}_p)$ can be very different. By combining the modulus part and the phase part, HAKE can model the entities in both the category (a) and the category (b). Therefore, HAKE can model semantic hierarchies of knowledge graphs.
When evaluating the models, we find that adding a mixture bias to $d_{r,m}(\textbf {h},\textbf {t})$ can help to improve the performance of HAKE. The modified $d_{r,m}(\textbf {h},\textbf {t})$ is given by:
where $0<\textbf {r}^{\prime }_m<1$ is a vector that have the same dimension with $\textbf {r}_m$. Indeed, the above distance function is equivalent to
where $/$ denotes the element-wise division operation. If we let $\textbf {r}_m\leftarrow (1-\textbf {r}_m^{\prime })/(\textbf {r}_m+\textbf {r}_m^{\prime })$, then the modified distance function is exactly the same as the original one when compare the distances of different entity pairs. For notation convenience, we still use $d_{r,m}(\textbf {h},\textbf {t})=\Vert \textbf {h}_m\circ \textbf {r}_m-\textbf {t}_m\Vert _2$ to represent the modulus part. We will conduct ablation studies on the bias in the experiment section.
The Proposed HAKE ::: Loss Function
To train the model, we use the negative sampling loss functions with self-adversarial training BIBREF7:
where $\gamma $ is a fixed margin, $\sigma $ is the sigmoid function, and $(h^{\prime }_i,r,t^{\prime }_i)$ is the $i$th negative triple. Moreover,
is the probability distribution of sampling negative triples, where $\alpha $ is the temperature of sampling.
Experiments and Analysis
This section is organized as follows. First, we introduce the experimental settings in detail. Then, we show the effectiveness of our proposed model on three benchmark datasets. Finally, we analyze the embeddings generated by HAKE, and show the results of ablation studies. The code of HAKE is available on GitHub at https://github.com/MIRALab-USTC/KGE-HAKE.
Experiments and Analysis ::: Experimental Settings
We evaluate our proposed models on three commonly used knowledge graph datasets—WN18RR BIBREF26, FB15k-237 BIBREF18, and YAGO3-10 BIBREF27. Details of these datasets are summarized in Table TABREF18.
WN18RR, FB15k-237, and YAGO3-10 are subsets of WN18 BIBREF8, FB15k BIBREF8, and YAGO3 BIBREF27, respectively. As pointed out by BIBREF26 and BIBREF18, WN18 and FB15k suffer from the test set leakage problem. One can attain the state-of-the-art results even using a simple rule based model. Therefore, we use WN18RR and FB15k-237 as the benchmark datasets.
Evaluation Protocol Following BIBREF8, for each triple $(h,r,t)$ in the test dataset, we replace either the head entity $h$ or the tail entity $t$ with each candidate entity to create a set of candidate triples. We then rank the candidate triples in descending order by their scores. It is worth noting that we use the “Filtered” setting as in BIBREF8, which does not take any existing valid triples into accounts at ranking. We choose Mean Reciprocal Rank (MRR) and Hits at N (H@N) as the evaluation metrics. Higher MRR or H@N indicate better performance.
Training Protocol We use Adam BIBREF28 as the optimizer, and use grid search to find the best hyperparameters based on the performance on the validation datasets. To make the model easier to train, we add an additional coefficient to the distance function, i.e., $d_{r}(\textbf {h},\textbf {t})=\lambda _1d_{r,m}(\textbf {h}_m,\textbf {t}_m)+\lambda _2 d_{r,p}(\textbf {h}_p,\textbf {t}_p)$, where $\lambda _1,\lambda _2\in \mathbb {R}$.
Baseline Model One may argue that the phase part is unnecessary, as we can distinguish entities in the category (b) by allowing $[\textbf {r}]_i$ to be negative. We propose a model—ModE—that uses only the modulus part but allow $[\textbf {r}]_i<0$. Specifically, the distance function of ModE is
Experiments and Analysis ::: Main Results
In this part, we show the performance of our proposed models—HAKE and ModE—against existing state-of-the-art methods, including TransE BIBREF8, DistMult BIBREF9, ComplEx BIBREF17, ConvE BIBREF18, and RotatE BIBREF7.
Table TABREF19 shows the performance of HAKE, ModE, and several previous models. Our baseline model ModE shares similar simplicity with TransE, but significantly outperforms it on all datasets. Surprisingly, ModE even outperforms more complex models such as DistMult, ConvE and Complex on all datasets, and beats the state-of-the-art model—RotatE—on FB15k-237 and YAGO3-10 datasets, which demonstrates the great power of modulus information. Table TABREF19 also shows that our HAKE significantly outperforms existing state-of-the-art methods on all datasets.
WN18RR dataset consists of two kinds of relations: the symmetric relations such as $\_similar\_to$, which link entities in the category (b); other relations such as $\_hypernym$ and $\_member\_meronym$, which link entities in the category (a). Actually, RotatE can model entities in the category (b) very well BIBREF7. However, HAKE gains a 0.021 higher MRR, a 2.4% higher H@1, and a 2.4% higher H@3 against RotatE, respectively. The superior performance of HAKE compared with RotatE implies that our proposed model can better model different levels in the hierarchy.
FB15k-237 dataset has more complex relation types and fewer entities, compared with WN18RR and YAGO3-10. Although there are relations that reflect hierarchy in FB15k-237, there are also lots of relations, such as “/location/location/time_zones” and “/film/film/prequel”, that do not lead to hierarchy. The characteristic of this dataset accounts for why our proposed models doesn't outperform the previous state-of-the-art as much as that of WN18RR and YAGO3-10 datasets. However, the results also show that our models can gain better performance so long as there exists semantic hierarchies in knowledge graphs. As almost all knowledge graphs have such hierarchy structures, our model is widely applicable.
YAGO3-10 datasets contains entities with high relation-specific indegree BIBREF18. For example, the link prediction task $(?, hasGender, male)$ has over 1000 true answers, which makes the task challenging. Fortunately, we can regard “male” as an entity at higher level of the hierarchy and the predicted head entities as entities at lower level. In this way, YAGO3-10 is a dataset that clearly has semantic hierarchy property, and we can expect that our proposed models is capable of working well on this dataset. Table TABREF19 validates our expectation. Both ModE and HAKE significantly outperform the previous state-of-the-art. Notably, HAKE gains a 0.050 higher MRR, 6.0% higher H@1 and 4.6% higher H@3 than RotatE, respectively.
Experiments and Analysis ::: Analysis on Relation Embeddings
In this part, we first show that HAKE can effectively model the hierarchy structures by analyzing the moduli of relation embeddings. Then, we show that the phase part of HAKE can help us to distinguish entities at the same level of the hierarchy by analyzing the phases of relation embeddings.
In Figure FIGREF20, we plot the distribution histograms of moduli of six relations. These relations are drawn from WN18RR, FB15k-237, and YAGO3-10. Specifically, the relations in Figures FIGREF20a, FIGREF20c, FIGREF20e and FIGREF20f are drawn from WN18RR. The relation in Figure FIGREF20d is drawn from FB15k-237. The relation in Figure FIGREF20b is drawn from YAGO3-10. We divide the relations in Figure FIGREF20 into three groups.
Relations in Figures FIGREF20c and FIGREF20d connect the entities at the same level of the semantic hierarchy;
Relations in Figures FIGREF20a and FIGREF20b represent that tail entities are at higher levels than head entities of the hierarchy;
Relations in Figures FIGREF20e and FIGREF20f represent that tail entities are at lower levels than head entities of the hierarchy.
As described in the model description section, we expect entities at higher levels of the hierarchy to have small moduli. The experiments validate our expectation. For both ModE and HAKE, most entries of the relations in the group (A) take values around one, which leads to that the head entities and tail entities have approximately the same moduli. In the group (B), most entries of the relations take values less than one, which results in that the head entities have smaller moduli than the tail entities. The cases in the group (C) are contrary to that in the group (B). These results show that our model can capture the semantic hierarchies in knowledge graphs. Moreover, compared with ModE, the relation embeddings' moduli of HAKE have lower variances, which shows that HAKE can model hierarchies more clearly.
As mentioned above, relations in the group (A) reflect the same semantic hierarchy, and are expected to have the moduli of about one. Obviously, it is hard to distinguish entities linked by these relations only using the modulus part. In Figure FIGREF22, we plot the phases of the relations in the group (A). The results show that the entities at the same level of the hierarchy can be distinguished by their phases, as many phases have the values of $\pi $.
Experiments and Analysis ::: Analysis on Entity Embeddings
In this part, to further show that HAKE can capture the semantic hierarchies between entities, we visualize the embeddings of several entity pairs.
We plot the entity embeddings of two models: the previous state-of-the-art RotatE and our proposed HAKE. RotatE regards each entity as a group of complex numbers. As a complex number can be seen as a point on a 2D plane, we can plot the entity embeddings on a 2D plane. As for HAKE, we have mentioned that it maps entities into the polar coordinate system. Therefore, we can also plot the entity embeddings generated by HAKE on a 2D plane based on their polar coordinates. For a fair comparison, we set $k=500$. That is, each plot contains 500 points, and the actual dimension of entity embeddings is 1000. Note that we use the logarithmic scale to better display the differences between entity embeddings. As all the moduli have values less than one, after applying the logarithm operation, the larger radii in the figures will actually represent smaller modulus.
Figure FIGREF29 shows the visualization results of three triples from the WN18RR dataset. Compared with the tail entities, the head entities in Figures FIGREF29a, FIGREF29b, and FIGREF29c are at lower levels, similar levels, higher levels in the semantic hierarchy, respectively. We can see that there exist clear concentric circles in the visualization results of HAKE, which demonstrates that HAKE can effectively model the semantic hierarchies. However, in RotatE, the entity embeddings in all three subfigures are mixed, making it hard to distinguish entities at different levels in the hierarchy.
Experiments and Analysis ::: Ablation Studies
In this part, we conduct ablation studies on the modulus part and the phase part of HAKE, as well as the mixture bias item. Table TABREF26 shows the results on three benchmark datasets.
We can see that the bias can improve the performance of HAKE on nearly all metrics. Specifically, the bias improves the H@1 score of $4.7\%$ on YAGO3-10 dataset, which illustrates the effectiveness of the bias.
We also observe that the modulus part of HAKE does not perform well on all datasets, due to its inability to distinguish the entities at the same level of the hierarchy. When only using the phase part, HAKE degenerates to the pRotatE model BIBREF7. It performs better than the modulus part, because it can well model entities at the same level of the hierarchy. However, our HAKE model significantly outperforms the modulus part and the phase part on all datasets, which demonstrates the importance to combine the two parts for modeling semantic hierarchies in knowledge graphs.
Experiments and Analysis ::: Comparison with Other Related Work
We compare our models with TKRL models BIBREF12, which also aim to model the hierarchy structures. For the difference between HAKE and TKRL, please refer to the Related Work section. Table TABREF27 shows the H@10 scores of HAKE and TKRLs on FB15k dataset. The best performance of TKRL is .734 obtained by the WHE+STC version, while the H@10 score of our HAKE model is .884. The results show that HAKE significantly outperforms TKRL, though it does not require additional information.
Conclusion
To model the semantic hierarchies in knowledge graphs, we propose a novel hierarchy-aware knowledge graph embedding model—HAKE—which maps entities into the polar coordinate system. Experiments show that our proposed HAKE significantly outperforms several existing state-of-the-art methods on benchmark datasets for the link prediction task. A further investigation shows that HAKE is capable of modeling entities at both different levels and the same levels in the semantic hierarchies.
Appendix
In this appendix, we will provide analysis on relation patterns, negative entity embeddings, and moduli of entity embeddings. Then, we will give more visualization results on semantic hierarchies.
A. Analysis on Relation Patterns
In this section, we prove that our HAKE model can infer the (anti)symmetry, inversion and composition relation patterns. Detailed propositions and their proofs are as follows.
Proposition 1 HAKE can infer the (anti)symmetry pattern.
If $r(x, y)$ and $r(y, x)$ hold, we have
Then we have
Otherwise, if $r(x, y)$ and $\lnot r(y, x)$ hold, we have
Proposition 2 HAKE can infer the inversion pattern.
If $r_1(x, y)$ and $r_2(y, x)$ hold, we have
Then, we have
Proposition 3 HAKE can infer the composition pattern.
If $r_1(x, z)$, $r_2(x, y)$ and $r_3(y, z)$ hold, we have
Then we have
B. Analysis on Negative Entity Embeddings
We denote the linked entity pairs as the set of entity pairs linked by some relation, and denote the unlinked entity pairs as the set of entity pairs that no triple contains in the train/valid/test dataset. It is worth noting that the unlinked paris may contain valid triples, as the knowledge graph is incomplete. For both the linked and the unlinked entity pairs, we count the embedding entries of two entities that have different signs. Figure FIGREF34 shows the result.
For the linked entity pairs, as we expected, most of the entries have the same sign. Due to the large amount of unlinked entity pairs, we randomly sample a part of them for plotting. For the unlinked entity pairs, around half of the entries have different signs, which is consistent with the random initialization. The results support our hypothesis that the negative signs of entity embeddings can help our model to distinguish positive and negative triples.
C. Analysis on Moduli of Entity Embeddings
Figure FIGREF37 shows the modulus of entity embeddings. We can observe that RotatE encourages the modulus of embeddings to be the same, as the relations are modeled as rotations in a complex space. Compared with RotatE, the modulus of entity embeddings in HAKE are more dispersed, making it to have more potential to model the semantic hierarchies.
D. More Results on Semantic Hierarchies
In this part, we visualize more triples from WN18RR. We plot the head and tail entities on 2D planes using the same method as that in the main text. The visualization results are in Figure FIGREF41, where the subcaptions demonstrate the corresponding triples. The figures show that, compared with RotatE, our HAKE model can better model the entities both in different hierarchies and in the same hierarchy.

Question: What benchmark datasets are used for the link prediction task?
Answer: WN18RR, FB15k-237, YAGO3-10

WN18RR BIBREF26, FB15k-237 BIBREF18, YAGO3-10 BIBREF27

Question: What are state-of-the art models for this task?
Answer: TransE, DistMult, ComplEx, ConvE, RotatE

Question: How better does HAKE model peform than state-of-the-art methods?
Answer: 0.021 higher MRR, a 2.4% higher H@1, and a 2.4% higher H@3 against RotatE, respectively, doesn't outperform the previous state-of-the-art as much as that of WN18RR and YAGO3-10, HAKE gains a 0.050 higher MRR, 6.0% higher H@1 and 4.6% higher H@3 than RotatE, respectively

Question: How are entities mapped onto polar coordinate system?
Answer: radial coordinate and the angular coordinates correspond to the modulus part and the phase part, respectively