Experiments demonstrate that HAKE can effectively model the semantic hierarchies in knowledge graphs, and significantly outperforms existing state-of-the-art methods on benchmark datasets for the link prediction task.

Conclusion	To model the semantic hierarchies in knowledge graphs, we propose a novel hierarchy-aware knowledge graph embedding model—HAKE—which maps entities into the polar coordinate system. Experiments show that our proposed HAKE significantly outperforms several existing state-of-the-art methods on benchmark datasets for the link prediction task. A further investigation shows that HAKE is capable of modeling entities at both different levels and the same levels in the semantic hierarchies.

YAGO3-10 datasets contains entities with high relation-specific indegree . For example, the link prediction task $(?, hasGender, male)$ has over 1000 true answers, which makes the task challenging. Fortunately, we can regard “male” as an entity at higher level of the hierarchy and the predicted head entities as entities at lower level. In this way, YAGO3-10 is a dataset that clearly has semantic hierarchy property, and we can expect that our proposed models is capable of working well on this dataset. Table TABREF19 validates our expectation. Both ModE and HAKE significantly outperform the previous state-of-the-art. Notably, HAKE gains a 0.050 higher MRR, 6.0% higher H@1 and 4.6% higher H@3 than RotatE, respectively.

Introduction	Knowledge graphs are usually collections of factual triples—(head entity, relation, tail entity), which represent human knowledge in a structured way. In the past few years, we have witnessed the great achievement of knowledge graphs in many areas, such as natural language processing , question answering , and recommendation systems . Although commonly used knowledge graphs contain billions of triples, they still suffer from the incompleteness problem that a lot of valid triples are missing, as it is impractical to find all valid triples manually. Therefore, knowledge graph completion, also known as link prediction in knowledge graphs, has attracted much attention recently. Link prediction aims to automatically predict missing links between entities based on known links. It is a challenging task as we not only need to predict whether there is a relation between two entities, but also need to determine which relation it is. Inspired by word embeddings  that can well capture semantic meaning of words, researchers turn to distributed representations of knowledge graphs (aka, knowledge graph embeddings) to deal with the link prediction problem. Knowledge graph embeddings regard entities and relations as low dimensional vectors (or matrices, tensors), which can be stored and computed efficiently.

Related Work ::: The Ways to Model Hierarchy Structures	Another related problem is how to model hierarchy structures in knowledge graphs. Some recent work considers the problem in different ways.  embed entities and categories jointly into a semantic space and designs models for the concept categorization and dataless hierarchical classification tasks.  use clustering algorithms to model the hierarchical relation structures.  proposed TKRL, which embeds the type information into knowledge graph embeddings. That is, TKRL requires additional hierarchical type information for entities. Different from the previous work, our work considers the link prediction task, which is a more common task for knowledge graph embeddings; can automatically learn the semantic hierarchy in knowledge graphs without using clustering algorithms; does not require any additional information other than the triples in knowledge graphs.

RotatE aims to model the relation patterns including symmetry/antisymmetry, inversion, and composition. HAKE aims to model the semantic hierarchy, while it can also model all the relation patterns mentioned above. The ways to use modulus information are different. RotatE models relations as rotations in the complex space, which encourages two linked entities to have the same modulus, no matter what the relation is. The different moduli in RotatE come from the inaccuracy in training. Instead, HAKE explicitly models the modulus information, which significantly outperforms RotatE in distinguishing entities at different levels of the hierarchy.

Moreover, like in the case of word embeddings, knowledge graph embeddings can preserve the semantics and inherent structures of entities and relations. Therefore, other than the link prediction task, knowledge graph embeddings can also be used in various downstream tasks, such as triple classification , relation inference , and search personalization . The success of existing knowledge graph embedding models heavily relies on their ability to model connectivity patterns of the relations, such as symmetry/antisymmetry, inversion, and composition . For example, TransE , which represent relations as translations, can model the inversion and composition patterns. DistMult , which models the three-way interactions between head entities, relations, and tail entities, can model the symmetry pattern. RotatE , which represents entities as points in a complex space and relations as rotations, can model relation patterns including symmetry/antisymmetry, inversion, and composition. However, many existing models fail to model semantic hierarchies in knowledge graphs. Semantic hierarchy is a ubiquitous property in knowledge graphs. For instance, WordNet  contains the triple [arbor/cassia/palm, hypernym, tree], where “tree” is at a higher level than “arbor/cassia/palm” in the hierarchy.

Experiments and Analysis ::: Experimental Settings	We evaluate our proposed models on three commonly used knowledge graph datasets—WN18RR , FB15k-237 , and YAGO3-10 . Details of these datasets are summarized in Table TABREF18. WN18RR, FB15k-237, and YAGO3-10 are subsets of WN18 , FB15k , and YAGO3 , respectively. As pointed out by  and , WN18 and FB15k suffer from the test set leakage problem. One can attain the state-of-the-art results even using a simple rule based model. Therefore, we use WN18RR and FB15k-237 as the benchmark datasets. Evaluation Protocol Following , for each triple $(h,r,t)$ in the test dataset, we replace either the head entity $h$ or the tail entity $t$ with each candidate entity to create a set of candidate triples. We then rank the candidate triples in descending order by their scores. It is worth noting that we use the “Filtered” setting as in , which does not take any existing valid triples into accounts at ranking.

Experiments and Analysis ::: Ablation Studies	In this part, we conduct ablation studies on the modulus part and the phase part of HAKE, as well as the mixture bias item. Table TABREF26 shows the results on three benchmark datasets. We can see that the bias can improve the performance of HAKE on nearly all metrics. Specifically, the bias improves the H@1 score of $4.7\%$ on YAGO3-10 dataset, which illustrates the effectiveness of the bias. We also observe that the modulus part of HAKE does not perform well on all datasets, due to its inability to distinguish the entities at the same level of the hierarchy. When only using the phase part, HAKE degenerates to the pRotatE model . It performs better than the modulus part, because it can well model entities at the same level of the hierarchy. However, our HAKE model significantly outperforms the modulus part and the phase part on all datasets, which demonstrates the importance to combine the two parts for modeling semantic hierarchies in knowledge graphs.

Experiments and Analysis	This section is organized as follows. First, we introduce the experimental settings in detail. Then, we show the effectiveness of our proposed model on three benchmark datasets. Finally, we analyze the embeddings generated by HAKE, and show the results of ablation studies. The code of HAKE is available on GitHub at https://github.com/MIRALab-USTC/KGE-HAKE.

Experiments and Analysis ::: Main Results	In this part, we show the performance of our proposed models—HAKE and ModE—against existing state-of-the-art methods, including TransE , DistMult , ComplEx , ConvE , and RotatE . Table TABREF19 shows the performance of HAKE, ModE, and several previous models. Our baseline model ModE shares similar simplicity with TransE, but significantly outperforms it on all datasets. Surprisingly, ModE even outperforms more complex models such as DistMult, ConvE and Complex on all datasets, and beats the state-of-the-art model—RotatE—on FB15k-237 and YAGO3-10 datasets, which demonstrates the great power of modulus information. Table TABREF19 also shows that our HAKE significantly outperforms existing state-of-the-art methods on all datasets. WN18RR dataset consists of two kinds of relations: the symmetric relations such as $\_similar\_to$, which link entities in the category (b); other relations such as $\_hypernym$ and $\_member\_meronym$, which link entities in the category (a).

Under the above settings, entities in the category (b) will have roughly the same modulus, which is hard to distinguish. Inspired by the fact that the points on the same circle can have different phases, we use phase information to model entities in the category (b). Combining the modulus and phase information, HAKE maps entities into the polar coordinate system, where the radial coordinate corresponds to the modulus information and the angular coordinate corresponds to the phase information. Experiments show that our proposed HAKE model can not only clearly distinguish the semantic hierarchies of entities, but also significantly and consistently outperform several state-of-the-art methods on the benchmark datasets.  Notations Throughout this paper, we use lower-case letters $h$, $r$, and $t$ to represent head entities, relations, and tail entities, respectively. The triplet $(h,r,t)$ denotes a fact in knowledge graphs. The corresponding boldface lower-case letters $\textbf {h}$, $\textbf {r}$ and $\textbf {t}$ denote the embeddings (vectors) of head entities, relations, and tail entities.

Learning Hierarchy-Aware Knowledge Graph Embeddings for Link Prediction	Knowledge graph embedding, which aims to represent entities and relations as low dimensional vectors (or matrices, tensors, etc.), has been shown to be a powerful technique for predicting missing links in knowledge graphs. Existing knowledge graph embedding models mainly focus on modeling relation patterns such as symmetry/antisymmetry, inversion, and composition. However, many existing approaches fail to model semantic hierarchies, which are common in real-world applications. To address this challenge, we propose a novel knowledge graph embedding model---namely, Hierarchy-Aware Knowledge Graph Embedding (HAKE)---which maps entities into the polar coordinate system. HAKE is inspired by the fact that concentric circles in the polar coordinate system can naturally reflect the hierarchy. Specifically, the radial coordinate aims to model entities at different levels of the hierarchy, and entities with smaller radii are expected to be at higher levels; the angular coordinate aims to distinguish entities at the same level of the hierarchy, and these entities are expected to have roughly the same radii but different angles.

B. Analysis on Negative Entity Embeddings	We denote the linked entity pairs as the set of entity pairs linked by some relation, and denote the unlinked entity pairs as the set of entity pairs that no triple contains in the train/valid/test dataset. It is worth noting that the unlinked paris may contain valid triples, as the knowledge graph is incomplete. For both the linked and the unlinked entity pairs, we count the embedding entries of two entities that have different signs. Figure FIGREF34 shows the result. For the linked entity pairs, as we expected, most of the entries have the same sign. Due to the large amount of unlinked entity pairs, we randomly sample a part of them for plotting. For the unlinked entity pairs, around half of the entries have different signs, which is consistent with the random initialization. The results support our hypothesis that the negative signs of entity embeddings can help our model to distinguish positive and negative triples.

Our goal is to minimize $d_r(\textbf {h}_m, \textbf {t}_{1,m})$ and maximize $d_r(\textbf {h}_m, \textbf {t}_{2,m})$, so as to make a clear distinction between positive and negative samples. For the positive sample, $[\textbf {h}]_i$ and $[\textbf {t}_1]_i$ tend to share the same sign, as $[\textbf {r}_m]_i>0$. For the negative sample, the signs of $[\textbf {h}_m]_i$ and $[\textbf {t}_{2,m}]_i$ can be different if we initialize their signs randomly. In this way, $d_r(\textbf {h}_m, \textbf {t}_{2,m})$ is more likely to be larger than $d_r(\textbf {h}_m, \textbf {t}_{1,m})$, which is exactly what we desire. We will validate this argument by experiments in Section 4 of the supplementary material.

We choose Mean Reciprocal Rank (MRR) and Hits at N (H@N) as the evaluation metrics. Higher MRR or H@N indicate better performance. Training Protocol We use Adam  as the optimizer, and use grid search to find the best hyperparameters based on the performance on the validation datasets. To make the model easier to train, we add an additional coefficient to the distance function, i.e., $d_{r}(\textbf {h},\textbf {t})=\lambda _1d_{r,m}(\textbf {h}_m,\textbf {t}_m)+\lambda _2 d_{r,p}(\textbf {h}_p,\textbf {t}_p)$, where $\lambda _1,\lambda _2\in \mathbb {R}$. Baseline Model One may argue that the phase part is unnecessary, as we can distinguish entities in the category (b) by allowing $[\textbf {r}]_i$ to be negative. We propose a model—ModE—that uses only the modulus part but allow $[\textbf {r}]_i<0$. Specifically, the distance function of ModE is

Moreover, compared with ModE, the relation embeddings' moduli of HAKE have lower variances, which shows that HAKE can model hierarchies more clearly. As mentioned above, relations in the group (A) reflect the same semantic hierarchy, and are expected to have the moduli of about one. Obviously, it is hard to distinguish entities linked by these relations only using the modulus part. In Figure FIGREF22, we plot the phases of the relations in the group (A). The results show that the entities at the same level of the hierarchy can be distinguished by their phases, as many phases have the values of $\pi $.

Therefore, we use modulus information to model entities in the category (a), as moduli can reflect the depth in a tree. Specifically, we regard each entry of $\textbf {h}_m$ and $\textbf {t}_m$, that is, $[\textbf {h}_m]_i$ and $[\textbf {t}_m]_i$, as a modulus, and regard each entry of $\textbf {r}_m$, that is, $[\textbf {r}]_i$, as a scaling transformation between two moduli. We can formulate the modulus part as follows: The corresponding distance function is: Note that we allow the entries of entity embeddings to be negative but restrict the entries of relation embeddings to be positive. This is because that the signs of entity embeddings can help us to predict whether there exists a relation between two entities. For example, if there exists a relation $r$ between $h$ and $t_1$, and no relation between $h$ and $t_2$, then $(h, r, t_1)$ is a positive sample and $(h, r, t_2)$ is a negative sample.

Actually, RotatE can model entities in the category (b) very well . However, HAKE gains a 0.021 higher MRR, a 2.4% higher H@1, and a 2.4% higher H@3 against RotatE, respectively. The superior performance of HAKE compared with RotatE implies that our proposed model can better model different levels in the hierarchy. FB15k-237 dataset has more complex relation types and fewer entities, compared with WN18RR and YAGO3-10. Although there are relations that reflect hierarchy in FB15k-237, there are also lots of relations, such as “/location/location/time_zones” and “/film/film/prequel”, that do not lead to hierarchy. The characteristic of this dataset accounts for why our proposed models doesn't outperform the previous state-of-the-art as much as that of WN18RR and YAGO3-10 datasets. However, the results also show that our models can gain better performance so long as there exists semantic hierarchies in knowledge graphs. As almost all knowledge graphs have such hierarchy structures, our model is widely applicable.

Figure FIGREF29 shows the visualization results of three triples from the WN18RR dataset. Compared with the tail entities, the head entities in Figures FIGREF29a, FIGREF29b, and FIGREF29c are at lower levels, similar levels, higher levels in the semantic hierarchy, respectively. We can see that there exist clear concentric circles in the visualization results of HAKE, which demonstrates that HAKE can effectively model the semantic hierarchies. However, in RotatE, the entity embeddings in all three subfigures are mixed, making it hard to distinguish entities at different levels in the hierarchy.

Experiments and Analysis ::: Comparison with Other Related Work	We compare our models with TKRL models , which also aim to model the hierarchy structures. For the difference between HAKE and TKRL, please refer to the Related Work section. Table TABREF27 shows the H@10 scores of HAKE and TKRLs on FB15k dataset. The best performance of TKRL is .734 obtained by the WHE+STC version, while the H@10 score of our HAKE model is .884. The results show that HAKE significantly outperforms TKRL, though it does not require additional information.

The Proposed HAKE ::: Hierarchy-Aware Knowledge Graph Embedding	To model both of the above categories, we propose a hierarchy-aware knowledge graph embedding model—HAKE. HAKE consists of two parts—the modulus part and the phase part—which aim to model entities in the two different categories, respectively. Figure FIGREF13 gives an illustration of the proposed model. To distinguish embeddings in the different parts, we use $\textbf {e}_m$ ($\textbf {e}$ can be $\textbf {h}$ or $\textbf {t}$) and $\textbf {r}_m$ to denote the entity embedding and relation embedding in the modulus part, and use $\textbf {e}_p$ ($\textbf {e}$ can be $\textbf {h}$ or $\textbf {t}$) and $\textbf {r}_p$ to denote the entity embedding and relation embedding in the phase part. The modulus part aims to model the entities at different levels of the hierarchy. Inspired by the fact that entities that have hierarchical property can be viewed as a tree, we can use the depth of a node (entity) to model different levels of the hierarchy.

For example, DistMult  assumes $\textbf {M}_r$ to be a diagonal matrix, and ANALOGY  supposes that $\textbf {M}_r$ is normal. However, these simplified models are usually less expressive and not powerful enough for general knowledge graphs. Differently, ComplEx  extends DistMult by introducing complex-valued embeddings to better model asymmetric and inverse relations. HolE  combines the expressive power of RESCAL with the efficiency and simplicity of DistMult by using the circular correlation operation. Neural network based models have received greater attention in recent years. For example, MLP  and NTN  use a fully connected neural network to determine the scores of given triples. ConvE  and ConvKB  employ convolutional neural networks to define score functions. Recently, graph convolutional networks are also introduced, as knowledge graphs obviously have graph structures . Our proposed model HAKE belongs to the translational distance models. More specifically, HAKE shares similarities with RotatE , in which the authors claim that they use both modulus and phase information. However, there exist two major differences between RotatE and HAKE. Detailed differences are as follows. The aims are different.

For notation convenience, we still use $d_{r,m}(\textbf {h},\textbf {t})=\Vert \textbf {h}_m\circ \textbf {r}_m-\textbf {t}_m\Vert _2$ to represent the modulus part. We will conduct ablation studies on the bias in the experiment section.

Further, we can expect the entities at higher levels of the hierarchy to have smaller modulus, as these entities are more close to the root of the tree. If we use only the modulus part to embed knowledge graphs, then the entities in the category (b) will have the same modulus. Moreover, suppose that $r$ is a relation that reflects the same semantic hierarchy, then $[\textbf {r}]_i$ will tend to be one, as $h\circ r\circ r=h$ holds for all $h$. Hence, embeddings of the entities in the category (b) tend to be the same, which makes it hard to distinguish these entities. Therefore, a new module is required to model the entities in the category (b). The phase part aims to model the entities at the same level of the semantic hierarchy. Inspired by the fact that points on the same circle (that is, have the same modulus) can have different phases, we use phase information to distinguish entities in the category (b).

Specifically, we regard each entry of $\textbf {h}_p$ and $\textbf {t}_p$, that is, $[\textbf {h}_p]_i$ and $[\textbf {t}_p]_i$ as a phase, and regard each entry of $\textbf {r}_p$, that is, $[\textbf {r}_p]_i$, as a phase transformation. We can formulate the phase part as follows: The corresponding distance function is: where $\sin (\cdot )$ is an operation that applies the sine function to each element of the input. Note that we use a sine function to measure the distance between phases instead of using $\Vert \textbf {h}_p+\textbf {r}_p-\textbf {t}_p\Vert _1$, as phases have periodic characteristic. This distance function shares the same formulation with that of pRotatE . Combining the modulus part and the phase part, HAKE maps entities into the polar coordinate system, where the radial coordinate and the angular coordinates correspond to the modulus part and the phase part, respectively.

The Proposed HAKE ::: Loss Function	To train the model, we use the negative sampling loss functions with self-adversarial training : where $\gamma $ is a fixed margin, $\sigma $ is the sigmoid function, and $(h^{\prime }_i,r,t^{\prime }_i)$ is the $i$th negative triple. Moreover, is the probability distribution of sampling negative triples, where $\alpha $ is the temperature of sampling.

Freebase  contains the triple [England, /location/location/contains, Pontefract/Lancaster], where “Pontefract/Lancaster” is at a lower level than “England” in the hierarchy. Although there exists some work that takes the hierarchy structures into account , , they usually require additional data or process to obtain the hierarchy information. Therefore, it is still challenging to find an approach that is capable of modeling the semantic hierarchy automatically and effectively. In this paper, we propose a novel knowledge graph embedding model—namely, Hierarchy-Aware Knowledge Graph Embedding (HAKE). To model the semantic hierarchies, HAKE is expected to distinguish entities in two categories: (a) at different levels of the hierarchy; (b) at the same level of the hierarchy. Inspired by the fact that entities that have the hierarchical properties can be viewed as a tree, we can use the depth of a node (entity) to model different levels of the hierarchy. Thus, we use modulus information to model entities in the category (a), as the size of moduli can reflect the depth.

D. More Results on Semantic Hierarchies	In this part, we visualize more triples from WN18RR. We plot the head and tail entities on 2D planes using the same method as that in the main text. The visualization results are in Figure FIGREF41, where the subcaptions demonstrate the corresponding triples. The figures show that, compared with RotatE, our HAKE model can better model the entities both in different hierarchies and in the same hierarchy.

Experiments and Analysis ::: Analysis on Entity Embeddings	In this part, to further show that HAKE can capture the semantic hierarchies between entities, we visualize the embeddings of several entity pairs. We plot the entity embeddings of two models: the previous state-of-the-art RotatE and our proposed HAKE. RotatE regards each entity as a group of complex numbers. As a complex number can be seen as a point on a 2D plane, we can plot the entity embeddings on a 2D plane. As for HAKE, we have mentioned that it maps entities into the polar coordinate system. Therefore, we can also plot the entity embeddings generated by HAKE on a 2D plane based on their polar coordinates. For a fair comparison, we set $k=500$. That is, each plot contains 500 points, and the actual dimension of entity embeddings is 1000. Note that we use the logarithmic scale to better display the differences between entity embeddings. As all the moduli have values less than one, after applying the logarithm operation, the larger radii in the figures will actually represent smaller modulus.

The score function is defined as $f_r(\textbf {h},\textbf {t})=-\Vert \textbf {h}_{\perp }+\textbf {r}-\textbf {t}_{\perp }\Vert _2$, where $\textbf {h}_{\perp }$ and $\textbf {t}_{\perp }$ are the projections of entities onto relation-specific hyperplanes. ManifoldE  deals with many-to-many problems by relaxing the hypothesis $\textbf {h}+\textbf {r}\approx \textbf {t}$ to $\Vert \textbf {h}+\textbf {r}-\textbf {t}\Vert _2^2\approx \theta _r^2$ for each valid triple. In this way, the candidate entities can lie on a manifold instead of exact point. The corresponding score function is defined as $f_r(\textbf {h},\textbf {t})=-(\Vert \textbf {h}+\textbf {r}-\textbf {t}\Vert _2^2-\theta _r^2)^2$.

Related Work	In this section, we will describe the related work and the key differences between them and our work in two aspects—the model category and the way to model hierarchy structures in knowledge graphs.

Related Work ::: Model Category	Roughly speaking, we can divide knowledge graph embedding models into three categories—translational distance models, bilinear models, and neural network based models. Table TABREF2 exhibits several popular models. Translational distance models describe relations as translations from source entities to target entities. TransE  supposes that entities and relations satisfy $\textbf {h}+\textbf {r}\approx \textbf {t}$, where $\textbf {h}, \textbf {r}, \textbf {t} \in \mathbb {R}^n$, and defines the corresponding score function as $f_r(\textbf {h},\textbf {t})=-\Vert \textbf {h}+\textbf {r}-\textbf {t}\Vert _{1/2}$. However, TransE does not perform well on 1-N, N-1 and N-N relations . TransH  overcomes the many-to-many relation problem by allowing entities to have distinct representations given different relations.

That is, HAKE maps an entity $h$ to $[\textbf {h}_m;\textbf {h}_p]$, where $\textbf {h}_m$ and $\textbf {h}_p$ are generated by the modulus part and the phase part, respectively, and $[\,\cdot \,; \,\cdot \,]$ denotes the concatenation of two vectors. Obviously, $([\textbf {h}_m]_i,[\textbf {h}_p]_i)$ is a 2D point in the polar coordinate system. Specifically, we formulate HAKE as follows: The distance function of HAKE is: where $\lambda \in \mathbb {R}$ is a parameter that learned by the model. The corresponding score function is When two entities have the same moduli, then the modulus part $d_{r,m}(\textbf {h}_m,\textbf {t}_m)=0$. However, the phase part $d_{r,p}(\textbf {h}_p,\textbf {t}_p)$ can be very different.

More recently, to better model symmetric and antisymmetric relations, RotatE  defines each relation as a rotation from source entities to target entities in a complex vector space. The score function is defined as $f_r(\textbf {h},\textbf {t})=-\Vert \textbf {h}\circ \textbf {r}-\textbf {t}\Vert _1$, where $\textbf {h},\textbf {r},\textbf {t}\in \mathbb {C}^k$ and $|[\textbf {r}]_i|=1$. Bilinear models product-based score functions to match latent semantics of entities and relations embodied in their vector space representations. RESCAL  represents each relation as a full rank matrix, and defines the score function as $f_r(\textbf {h},\textbf {t})=\textbf {h}^\top \textbf {M}_r \textbf {t}$, which can also be seen as a bilinear function. As full rank matrices are prone to overfitting, recent works turn to make additional assumptions on $\textbf {M}_r$.

The Proposed HAKE	In this section, we introduce our proposed model HAKE. We first introduce two categories of entities that reflect the semantic hierarchies in knowledge graphs. Afterwards, we introduce our proposed HAKE that can model entities in both of the categories.

The Proposed HAKE ::: Two Categories of Entities	To model the semantic hierarchies of knowledge graphs, a knowledge graph embedding model must be capable of distinguishing entities in the following two categories. Entities at different levels of the hierarchy. For example, “mammal” and “dog”, “run” and ”move”. Entities at the same level of the hierarchy. For example, “rose” and “peony”, “truck” and ”lorry”.

C. Analysis on Moduli of Entity Embeddings	Figure FIGREF37 shows the modulus of entity embeddings. We can observe that RotatE encourages the modulus of embeddings to be the same, as the relations are modeled as rotations in a complex space. Compared with RotatE, the modulus of entity embeddings in HAKE are more dispersed, making it to have more potential to model the semantic hierarchies.

By combining the modulus part and the phase part, HAKE can model the entities in both the category (a) and the category (b). Therefore, HAKE can model semantic hierarchies of knowledge graphs. When evaluating the models, we find that adding a mixture bias to $d_{r,m}(\textbf {h},\textbf {t})$ can help to improve the performance of HAKE. The modified $d_{r,m}(\textbf {h},\textbf {t})$ is given by: where $0<\textbf {r}^{\prime }_m<1$ is a vector that have the same dimension with $\textbf {r}_m$. Indeed, the above distance function is equivalent to where $/$ denotes the element-wise division operation. If we let $\textbf {r}_m\leftarrow (1-\textbf {r}_m^{\prime })/(\textbf {r}_m+\textbf {r}_m^{\prime })$, then the modified distance function is exactly the same as the original one when compare the distances of different entity pairs.

Experiments and Analysis ::: Analysis on Relation Embeddings	In this part, we first show that HAKE can effectively model the hierarchy structures by analyzing the moduli of relation embeddings. Then, we show that the phase part of HAKE can help us to distinguish entities at the same level of the hierarchy by analyzing the phases of relation embeddings. In Figure FIGREF20, we plot the distribution histograms of moduli of six relations. These relations are drawn from WN18RR, FB15k-237, and YAGO3-10. Specifically, the relations in Figures FIGREF20a, FIGREF20c, FIGREF20e and FIGREF20f are drawn from WN18RR. The relation in Figure FIGREF20d is drawn from FB15k-237. The relation in Figure FIGREF20b is drawn from YAGO3-10. We divide the relations in Figure FIGREF20 into three groups.

Relations in Figures FIGREF20c and FIGREF20d connect the entities at the same level of the semantic hierarchy; Relations in Figures FIGREF20a and FIGREF20b represent that tail entities are at higher levels than head entities of the hierarchy; Relations in Figures FIGREF20e and FIGREF20f represent that tail entities are at lower levels than head entities of the hierarchy. As described in the model description section, we expect entities at higher levels of the hierarchy to have small moduli. The experiments validate our expectation. For both ModE and HAKE, most entries of the relations in the group (A) take values around one, which leads to that the head entities and tail entities have approximately the same moduli. In the group (B), most entries of the relations take values less than one, which results in that the head entities have smaller moduli than the tail entities. The cases in the group (C) are contrary to that in the group (B). These results show that our model can capture the semantic hierarchies in knowledge graphs.

Appendix	In this appendix, we will provide analysis on relation patterns, negative entity embeddings, and moduli of entity embeddings. Then, we will give more visualization results on semantic hierarchies.

A. Analysis on Relation Patterns	In this section, we prove that our HAKE model can infer the (anti)symmetry, inversion and composition relation patterns. Detailed propositions and their proofs are as follows. Proposition 1 HAKE can infer the (anti)symmetry pattern. If $r(x, y)$ and $r(y, x)$ hold, we have Then we have Otherwise, if $r(x, y)$ and $\lnot r(y, x)$ hold, we have Proposition 2 HAKE can infer the inversion pattern. If $r_1(x, y)$ and $r_2(y, x)$ hold, we have Then, we have  Proposition 3 HAKE can infer the composition pattern. If $r_1(x, z)$, $r_2(x, y)$ and $r_3(y, z)$ hold, we have Then we have

The $i$-th entry of a vector $\textbf {h}$ is denoted as $[\textbf {h}]_i$. Let $k$ denote the embedding dimension. Let $\circ :\mathbb {R}^n\times \mathbb {R}^n\rightarrow \mathbb {R}^n$ denote the Hadamard product between two vectors, that is, and $\Vert \cdot \Vert _1$, $\Vert \cdot \Vert _2$ denote the $\ell _1$ and $\ell _2$ norm, respectively.

Question: What benchmark datasets are used for the link prediction task?
Answer: WN18RR, FB15k-237, YAGO3-10