# Do Similar Entities have Similar Embeddings?

Nicolas Hubert, Heiko Paulheim, Armelle Brun, Davy Monticolo

## ► To cite this version:

Nicolas Hubert, Heiko Paulheim, Armelle Brun, Davy Monticolo. Do Similar Entities have Similar Embeddings?. ESWC 2024 - 21st International Conference on Semantic Web, May 2024, HERSONIS-SOS (CRETE), Greece. hal-04520741

HAL Id: hal-04520741

https://hal.science/hal-04520741

Submitted on 25 Mar 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Do Similar Entities have Similar Embeddings?

Nicolas Hubert, Heiko Paulheim, Armelle Brun, Davy Monticolo

## ► To cite this version:

Nicolas Hubert, Heiko Paulheim, Armelle Brun, Davy Monticolo. Do Similar Entities have Similar Embeddings?. ESWC 2024 - 21st International Conference on Semantic Web, May 2024, HERSONIS-SOS (CRETE), Greece. hal-04520741

HAL Id: hal-04520741

https://hal.science/hal-04520741v1

Submitted on 25 Mar 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Do Similar Entities have Similar Embeddings?

Nicolas Hubert¹,²[0000-0002-4682-422X], Heiko Paulheim³[0000-0003-4386-8195], Armelle Brun²[0000-0002-9876-6906], and Davy Monticolo¹[0000-0002-4244-684X]

¹ Université de Lorraine, ERPI, Nancy, France

² Université de Lorraine, CNRS, LORIA, Nancy, France

³ Data and Web Science Group, University of Mannheim, Germany
{nicolas.hubert,armelle.brun,davy.monticolo}@univ-lorraine.fr
heiko@informatik.uni-mannheim.de

**Abstract.** Knowledge graph embedding models (KGEMs) developed for link prediction learn vector representations for entities in a knowledge graph, known as embeddings. A common tacit assumption is the *KGE entity similarity assumption*, which states that these KGEMs retain the graph's structure within their embedding space, i.e., position similar entities within the graph close to one another. This desirable property make KGEMs widely used in downstream tasks such as recommender systems or drug repurposing. Yet, the relation of entity similarity and similarity in the embedding space has rarely been formally evaluated. Typically, KGEMs are assessed based on their sole link prediction capabilities, using ranked-based metrics such as Hits@K or Mean Rank. This paper challenges the prevailing assumption that entity similarity in the graph is inherently mirrored in the embedding space. Therefore, we conduct extensive experiments to measure the capability of KGEMs to cluster similar entities together, and investigate the nature of the underlying factors. Moreover, we study if different KGEMs expose a different notion of similarity. Datasets, pre-trained embeddings and code are available at: https://github.com/nicolas-hbt/similar-embeddings/.

**Keywords:** Knowledge Graph · Embedding · Representation Learning · Entity Similarity

## 1 Introduction

Knowledge Graphs (KGs) such as DBpedia [3] and YAGO [31] represent facts as triples (*s*, *p*, *o*) consisting of a subject *s* and an object *o* connected by a predicate *p* defining their relationship. Common learning tasks with KGs include entity clustering, node classification, and link prediction.

These tasks are predominantly tackled using Knowledge Graph Embedding Models (KGEMs), which generate dense vector representations for entities and relations of a KG, a.k.a. Knowledge Graph Embeddings (KGEs). The dense numerical vectors that are learnt for entities and relations are expected to preserve the intrinsic semantics of the KG [38].

As KGEMs take into account the semantic relationship between two entities to learn embeddings, it is often taken for granted that the resulting embeddings
# Do Similar Entities have Similar Embeddings?

Nicolas Hubert¹,²[0000-0002-4682-422X], Heiko Paulheim³[0000-0003-4386-8195], Armelle Brun²[0000-0002-9876-6906], and Davy Monticolo¹[0000-0002-4244-684X]

¹ Université de Lorraine, ERPI, Nancy, France

² Université de Lorraine, CNRS, LORIA, Nancy, France

³ Data and Web Science Group, University of Mannheim, Germany
{nicolas.hubert,armelle.brun,davy.monticolo}@univ-lorraine.fr
heiko@informatik.uni-mannheim.de

**Abstract.** Knowledge graph embedding models (KGEMs) developed for link prediction learn vector representations for entities in a knowledge graph, known as embeddings. A common tacit assumption is the *KGE entity similarity assumption*, which states that these KGEMs retain the graph's structure within their embedding space, i.e., position similar entities within the graph close to one another. This desirable property make KGEMs widely used in downstream tasks such as recommender systems or drug repurposing. Yet, the relation of entity similarity and similarity in the embedding space has rarely been formally evaluated. Typically, KGEMs are assessed based on their sole link prediction capabilities, using ranked-based metrics such as Hits@K or Mean Rank. This paper challenges the prevailing assumption that entity similarity in the graph is inherently mirrored in the embedding space. Therefore, we conduct extensive experiments to measure the capability of KGEMs to cluster similar entities together, and investigate the nature of the underlying factors. Moreover, we study if different KGEMs expose a different notion of similarity. Datasets, pre-trained embeddings and code are available at: https://github.com/nicolas-hbt/similar-embeddings/.

**Keywords:** Knowledge Graph · Embedding · Representation Learning · Entity Similarity

## 1 Introduction

Knowledge Graphs (KGs) such as DBpedia [3] and YAGO [31] represent facts as triples (*s*, *p*, *o*) consisting of a subject *s* and an object *o* connected by a predicate *p* defining their relationship. Common learning tasks with KGs include entity clustering, node classification, and link prediction.

These tasks are predominantly tackled using Knowledge Graph Embedding Models (KGEMs), which generate dense vector representations for entities and relations of a KG, a.k.a. Knowledge Graph Embeddings (KGEs). The dense numerical vectors that are learnt for entities and relations are expected to preserve the intrinsic semantics of the KG [38].

As KGEMs take into account the semantic relationship between two entities to learn embeddings, it is often taken for granted that the resulting embeddings
capture both the semantics and attributes of entities and their relationships in the KG. Embeddings are thus widely used to measure the semantic similarity between entities and relations, facilitating data integration through entity or relation alignments [7,17]. They are also used in various similarity-based tasks including entity similarity [33] and conceptual clustering [9].

However, the widespread assumption that KGEMs create semantically meaningful representations of the underlying entities (i.e., project similar entities closer than dissimilar ones) has been challenged recently [14]. Jain et al. [14] demonstrate that entity embeddings learnt with KGEMs are not well-suited to identify the concepts or classes for a vast majority of KG entities, while simple statistical approaches provide comparable or better performance. Concerned with word embeddings, Ilievski et al. [13] also point out that KGEMs are consistently outperformed by simpler heuristics for similarity-based tasks. The authors argue that many properties on which KGEMs heavily rely on are not useful for determining similarity, which introduces noise and subsequently decreases performance. Our work falls within this line of thought. More specifically, we formulate the following research question:

**RQ1.** To what extent does proximity in the embedding space align with the notion of entity similarity in the KG?

We call this the *KGE entity similarity assumption*. Notably, there is no universally accepted definition for entity similarity. In this work, we follow a straightforward approach: two entities in a KG are similar if we make similar statements about them. This aligns with the assumption of distributional semantics: words appearing in similar contexts are semantically similar. Answering **RQ1** requires remembering that most embedding-based models are trained to maximize rank-based metrics for link prediction, which disregards semantics. One could then argue that maximizing such metrics is at least partially decoupled from the task of learning similar vectors for similar entities. However, as related entities are more likely to appear in similar triples (e.g., featuring the same predicate), it is reasonable to believe that semantically close entities – especially those connected to other entities through a shared set of predicates – are also more likely to be assigned similar vectors [23]. This raises our second research question:

**RQ2.** How do traditional rank-based metrics correlate with entity similarity?

In other words, we ask whether KGEMs with good link prediction performance w.r.t. metrics such as Mean Reciprocal Rank (MRR) and Hits@K necessarily group similar entities close in the embedding space. If so, it would be sensible to study which of the two is most likely to influence the other, and whether a form of causality (rather than just correlation) between these two aspects exists.

To delve further, it is worth noting that many link prediction train sets have been shown to suffer from extremely skewed distributions, especially in the occurrence of a subset of predicates [26]. Rossi et al. [26] demonstrate that relying on global metrics (e.g. Hits@K and MRR) over such heavily skewed distributions hinders our understanding of KGEMs. Consequently, we ultimately
capture both the semantics and attributes of entities and their relationships in the KG. Embeddings are thus widely used to measure the semantic similarity between entities and relations, facilitating data integration through entity or relation alignments [7,17]. They are also used in various similarity-based tasks including entity similarity [33] and conceptual clustering [9].

However, the widespread assumption that KGEMs create semantically meaningful representations of the underlying entities (i.e., project similar entities closer than dissimilar ones) has been challenged recently [14]. Jain et al. [14] demonstrate that entity embeddings learnt with KGEMs are not well-suited to identify the concepts or classes for a vast majority of KG entities, while simple statistical approaches provide comparable or better performance. Concerned with word embeddings, Ilievski et al. [13] also point out that KGEMs are consistently outperformed by simpler heuristics for similarity-based tasks. The authors argue that many properties on which KGEMs heavily rely on are not useful for determining similarity, which introduces noise and subsequently decreases performance. Our work falls within this line of thought. More specifically, we formulate the following research question:

**RQ1.** To what extent does proximity in the embedding space align with the notion of entity similarity in the KG?

We call this the *KGE entity similarity assumption*. Notably, there is no universally accepted definition for entity similarity. In this work, we follow a straightforward approach: two entities in a KG are similar if we make similar statements about them. This aligns with the assumption of distributional semantics: words appearing in similar contexts are semantically similar. Answering **RQ1** requires remembering that most embedding-based models are trained to maximize rank-based metrics for link prediction, which disregards semantics. One could then argue that maximizing such metrics is at least partially decoupled from the task of learning similar vectors for similar entities. However, as related entities are more likely to appear in similar triples (e.g., featuring the same predicate), it is reasonable to believe that semantically close entities – especially those connected to other entities through a shared set of predicates – are also more likely to be assigned similar vectors [23]. This raises our second research question:

**RQ2.** How do traditional rank-based metrics correlate with entity similarity?

In other words, we ask whether KGEMs with good link prediction performance w.r.t. metrics such as Mean Reciprocal Rank (MRR) and Hits@K necessarily group similar entities close in the embedding space. If so, it would be sensible to study which of the two is most likely to influence the other, and whether a form of causality (rather than just correlation) between these two aspects exists.

To delve further, it is worth noting that many link prediction train sets have been shown to suffer from extremely skewed distributions, especially in the occurrence of a subset of predicates [26]. Rossi et al. [26] demonstrate that relying on global metrics (e.g. Hits@K and MRR) over such heavily skewed distributions hinders our understanding of KGEMs. Consequently, we ultimately
distance ourselves from analyzing rank-based metrics, and dive deeper into entity embeddings with the sole consideration of studying how and why they may differ between models. In line with Rossi et al. [26] findings that a given subset of predicates is often likely to heavily influence entity representations – we formulate our last research question as follows:

**RQ3.** Do different KGEMs focus on different predicates to capture the notion of similarity in the embedding space?

The top-K neighbors in the embedding space for a given entity may differ between KGEMs. However, this does not tell us much about why this is the case. We posit that studying the distribution of predicates in the K-hop subgraph centered around each neighboring entity can provide insights into the relevance of certain predicates for particular KGEMs. In other words, the subset of predicates that Rossi et al. [26] found out to influence entity representations and KGEM performance w.r.t. rank-based metrics might not just be dataset-dependent. Different KGEMs might also implicitly overweigh different subsets of predicates.

The main contributions of our work are summarized as follows.

- We show that different KGEMs fulfill the *KGE entity similarity assumption* only to a limited extent. Notably, even for a given KGEM, results can vary substantially on a per-class basis. Moreover, the semantics of classes is inequally captured by different KGEMs, thereby highlighting that different KGEMs expose different notions of similarity.

- We show that in most cases, performance in link prediction does not correlate with a model's adherence to the KGE entity similarity assumption. This demonstrates that rank-based metrics cannot be used as a reliable proxy for assessing the semantic consistency of the embedding space.

- We show that different KGEMs turn their attention to different predicate subsets for learning similar embeddings for related entities. This suggests that the notion of similarity in the embedding space is partially influenced by the predicate distribution in the close neighborhood around KG entities.

The remainder of the paper is structured as follows. Related work about KGEMs and their use in semantic-related tasks is presented in Section 2. Section 3 elaborates on our approach for measuring similarity. Section 4 details our experimental setting. Results are provided and discussed in Section 5. Lastly, Section 6 summarizes the key findings and outlines directions for future research.

## 2 Related Work

**Knowledge graph embeddings.** KGEMs have garnered significant attention in recent years due to their capacity to represent structured knowledge within a continuous vector space. The seminal translational model TransE [6] represents entities and relations as low-dimensional vectors and establishes the relationship between a head, a relation, and a tail through a translation operation in the embedding space. Subsequent KGEMs have primarily aimed to address its limitations and enhance the representational expressiveness of knowledge
distance ourselves from analyzing rank-based metrics, and dive deeper into entity embeddings with the sole consideration of studying how and why they may differ between models. In line with Rossi et al. [26] findings that a given subset of predicates is often likely to heavily influence entity representations – we formulate our last research question as follows:

**RQ3.** Do different KGEMs focus on different predicates to capture the notion of similarity in the embedding space?

The top-K neighbors in the embedding space for a given entity may differ between KGEMs. However, this does not tell us much about why this is the case. We posit that studying the distribution of predicates in the K-hop subgraph centered around each neighboring entity can provide insights into the relevance of certain predicates for particular KGEMs. In other words, the subset of predicates that Rossi et al. [26] found out to influence entity representations and KGEM performance w.r.t. rank-based metrics might not just be dataset-dependent. Different KGEMs might also implicitly overweigh different subsets of predicates.

The main contributions of our work are summarized as follows.

- We show that different KGEMs fulfill the *KGE entity similarity assumption* only to a limited extent. Notably, even for a given KGEM, results can vary substantially on a per-class basis. Moreover, the semantics of classes is inequally captured by different KGEMs, thereby highlighting that different KGEMs expose different notions of similarity.

- We show that in most cases, performance in link prediction does not correlate with a model's adherence to the KGE entity similarity assumption. This demonstrates that rank-based metrics cannot be used as a reliable proxy for assessing the semantic consistency of the embedding space.

- We show that different KGEMs turn their attention to different predicate subsets for learning similar embeddings for related entities. This suggests that the notion of similarity in the embedding space is partially influenced by the predicate distribution in the close neighborhood around KG entities.

The remainder of the paper is structured as follows. Related work about KGEMs and their use in semantic-related tasks is presented in Section 2. Section 3 elaborates on our approach for measuring similarity. Section 4 details our experimental setting. Results are provided and discussed in Section 5. Lastly, Section 6 summarizes the key findings and outlines directions for future research.

## 2 Related Work

**Knowledge graph embeddings.** KGEMs have garnered significant attention in recent years due to their capacity to represent structured knowledge within a continuous vector space. The seminal translational model TransE [6] represents entities and relations as low-dimensional vectors and establishes the relationship between a head, a relation, and a tail through a translation operation in the embedding space. Subsequent KGEMs have primarily aimed to address its limitations and enhance the representational expressiveness of knowledge
graph embeddings [38]. Representative models are DistMult [40], ComplEx [37], ConvE [8], and TuckER [4]. The embeddings learnt by these KGEMs have shown potential applications in tasks such as link prediction [38], entity clustering [12], and node classification [12].

**Using KGEs for semantic-related tasks.** As relational models, KGEMs are widely used for predicting links in KGs [38,25]. However, the vector representations learnt by these models can also be used for other tasks [16]. For example, pretrained language models with knowledge graphs have been used for Named Entity Recognition (NER) [32,19]. Other tasks aiming at discovering rich information about entities through their embeddings include entity typing [20] and entity alignment [34]. Many works also explore the use of KGEs for drug repurposing [30] and recommender systems [10]. These works are based on the premise that the distance between entities in the embedding space should reflect their intrinsic similarity, and can be leveraged, e.g. for recommending a common set of items to similar users [10].

**Analzying the KGE entity similarity assumption.** While many of the approaches *implicitly* rely on the KGE entity similarity assumption to hold, there are only few works actually *explicitly* validating this assumption. Portisch *et al.* [23] provide anecdotic examples for a few embedding approaches, also suggesting that the underlying notions of similarity might differ, but do not conduct any formal evaluation. Jain *et al.* [14] analyze KGEMs on the basis of class assignments, showing that the original class assignments can only be reconstructed to a limited extent with classification and clustering methods, which rather questions the assumption of similar entities being close in the vector space. A similar study is conducted by Alshargi *et al.* [2], also concluding that “the current quality of the embeddings for ontological concepts is not in a satisfactory state”. While Portisch *et al.* provide anecdotic examples, and Jain *et al.* and Alshargi *et al.* only look at class assignments, this work is the first one to empirically study the relation of entity similarity and similarity in the KGE space. Moreover, our study is more fine-grained than the previous ones, which end at the class level: while those only inspect whether entities of the same class have similar embeddings (e.g., two movies are embedded closer than a movie and a person), we also consider similarity *within* a class (e.g., analyze whether similar movies are embedded closer than less similar ones).

## 3 Approach

In this section, we detail our proposal for quantifying the KGE entity similarity assumption. While similarity in embedding spaces is usually measured using cosine similarity, there is no uniform definition of entity similarity in KGs. In Section 3.1, we elaborate on the metrics used to capture the notion of similarity between entities in the KG. Section 3.2 discusses how the aforementioned notions of similarity can be compared. It is worth noting that the line of research closest to ours is the one of Jain *et al.* [14]. Both [14] and our work question common assumptions that are taken for granted in the KG community. However, [14] looks at the capability of KGEMs to learn the semantics of classes, and considers the
graph embeddings [38]. Representative models are DistMult [40], ComplEx [37], ConvE [8], and TuckER [4]. The embeddings learnt by these KGEMs have shown potential applications in tasks such as link prediction [38], entity clustering [12], and node classification [12].

**Using KGEs for semantic-related tasks.** As relational models, KGEMs are widely used for predicting links in KGs [38,25]. However, the vector representations learnt by these models can also be used for other tasks [16]. For example, pretrained language models with knowledge graphs have been used for Named Entity Recognition (NER) [32,19]. Other tasks aiming at discovering rich information about entities through their embeddings include entity typing [20] and entity alignment [34]. Many works also explore the use of KGEs for drug repurposing [30] and recommender systems [10]. These works are based on the premise that the distance between entities in the embedding space should reflect their intrinsic similarity, and can be leveraged, e.g. for recommending a common set of items to similar users [10].

**Analzying the KGE entity similarity assumption.** While many of the approaches *implicitly* rely on the KGE entity similarity assumption to hold, there are only few works actually *explicitly* validating this assumption. Portisch *et al.* [23] provide anecdotic examples for a few embedding approaches, also suggesting that the underlying notions of similarity might differ, but do not conduct any formal evaluation. Jain *et al.* [14] analyze KGEMs on the basis of class assignments, showing that the original class assignments can only be reconstructed to a limited extent with classification and clustering methods, which rather questions the assumption of similar entities being close in the vector space. A similar study is conducted by Alshargi *et al.* [2], also concluding that “the current quality of the embeddings for ontological concepts is not in a satisfactory state”. While Portisch *et al.* provide anecdotic examples, and Jain *et al.* and Alshargi *et al.* only look at class assignments, this work is the first one to empirically study the relation of entity similarity and similarity in the KGE space. Moreover, our study is more fine-grained than the previous ones, which end at the class level: while those only inspect whether entities of the same class have similar embeddings (e.g., two movies are embedded closer than a movie and a person), we also consider similarity *within* a class (e.g., analyze whether similar movies are embedded closer than less similar ones).

## 3 Approach

In this section, we detail our proposal for quantifying the KGE entity similarity assumption. While similarity in embedding spaces is usually measured using cosine similarity, there is no uniform definition of entity similarity in KGs. In Section 3.1, we elaborate on the metrics used to capture the notion of similarity between entities in the KG. Section 3.2 discusses how the aforementioned notions of similarity can be compared. It is worth noting that the line of research closest to ours is the one of Jain *et al.* [14]. Both [14] and our work question common assumptions that are taken for granted in the KG community. However, [14] looks at the capability of KGEMs to learn the semantics of classes, and considers the
specific tasks of entity classification and clustering. In contrast, we are concerned
with similarity measures of different embedding models and how they reflect
entity similarity. As such, even though our work fits within an existing literature
on the concept of semantic capture in embeddings, to the best of our knowledge
this paper is the first one to present an approach to thoroughly investigate the
extent to which entity similarity in KGs is mirrored in the embedding space.

## 3.1 Towards a Graph-based Notion of Similarity

In this section, we detail our attempt at capturing the notion of similarity in
the original KG. Unlike the notion of similarity in the embedding space, which
is typically measured as the cosine similarity of two vectors, it should be noted
that graph-based similarity cannot be measured in a single, uniform way. As
previsouly said, there is no universal definition for entity similarity in KGs.
Multiple metrics and approaches can be used, and the choice between one or
another largely depends on the particular aspect to be measured. In what follows,
we intend to explain the rationale behind our modeling choices, and briefly
mention alternatives that we ultimately discarded.

To measure the similarity between two entities $e_1$ and $e_2$ in one KG, we (1) determine the set of common statements about $e_1$ and $e_2$, i.e., relations to other entities that $e_1$ and $e_2$ have in common. While many entities will have such relations in common with central entities (i.e., those with many ingoing and outgoing edges), we also (2) need to make sure that centrality does not skew our similarity measure. Finally, (3) since an entity cannot be fully described by its immediate neighbors, indirect dependencies also need to be captured.

To satisfy our first desideratum (1), we compare the subgraphs around $e_1$ and $e_2$. We experiment with the 1-hop only vs. 2-hop subgraph neighborhood. The latter option addresses desideratum (3), as we also consider indirect dependencies. In Fig. 1, we give a concrete example of the 1-hop and 2-hop subgraphs for entities Bob ($e_1$) and Julie ($e_2$).<fnref n="4" /> The similarity of $e_1$ and $e_2$ can now be measured by the similarity of their respective subgraphs. Graph Edit Distance (GED) [28] has been used for this intent. However, GED is NP-hard and thus computationally demanding, it comes with the need for arbitrarily defining weights for vertex and edge insertion/deletion/substitution, and we experimentally found it was sensitive to subgraphs' sizes, which is detrimental to desideratum (2). We considered other metrics such as Katz centrality and the common-neighbors metric. However, as the name suggests, Katz is a measure of centrality, not similarity. Besides, it takes into account all the paths between two entities and is therefore sensitive to the absolute number of paths. Common neighbors does not consider predicates and is only suited for unlabeled graphs.<fnref n="5" />

In our experiments, we use the Jaccard coefficient is used to measure the
overlap in the 1-hop and 2-hop subgraphs of entities. In particular, its value

<footnote n="4">⁴ We consider both ingoing and outgoing edges to define those neighborhoods.</footnote>

<footnote n="5">⁵ Given, e.g., (SAP, headquarter, Germany), (BOSCH, headquarter, Germany), (Berlin, capitalOf, Germany), with common neighbors, SAP, BOSCH, and Berlin would be equally similar.</footnote>
specific tasks of entity classification and clustering. In contrast, we are concerned
with similarity measures of different embedding models and how they reflect
entity similarity. As such, even though our work fits within an existing literature
on the concept of semantic capture in embeddings, to the best of our knowledge
this paper is the first one to present an approach to thoroughly investigate the
extent to which entity similarity in KGs is mirrored in the embedding space.

## 3.1 Towards a Graph-based Notion of Similarity

In this section, we detail our attempt at capturing the notion of similarity in
the original KG. Unlike the notion of similarity in the embedding space, which
is typically measured as the cosine similarity of two vectors, it should be noted
that graph-based similarity cannot be measured in a single, uniform way. As
previsouly said, there is no universal definition for entity similarity in KGs.
Multiple metrics and approaches can be used, and the choice between one or
another largely depends on the particular aspect to be measured. In what follows,
we intend to explain the rationale behind our modeling choices, and briefly
mention alternatives that we ultimately discarded.

To measure the similarity between two entities $e_1$ and $e_2$ in one KG, we (1) determine the set of common statements about $e_1$ and $e_2$, i.e., relations to other entities that $e_1$ and $e_2$ have in common. While many entities will have such relations in common with central entities (i.e., those with many ingoing and outgoing edges), we also (2) need to make sure that centrality does not skew our similarity measure. Finally, (3) since an entity cannot be fully described by its immediate neighbors, indirect dependencies also need to be captured.

To satisfy our first desideratum (1), we compare the subgraphs around $e_1$ and $e_2$. We experiment with the 1-hop only vs. 2-hop subgraph neighborhood. The latter option addresses desideratum (3), as we also consider indirect dependencies. In Fig. 1, we give a concrete example of the 1-hop and 2-hop subgraphs for entities Bob ($e_1$) and Julie ($e_2$).<fnref n="4" /> The similarity of $e_1$ and $e_2$ can now be measured by the similarity of their respective subgraphs. Graph Edit Distance (GED) [28] has been used for this intent. However, GED is NP-hard and thus computationally demanding, it comes with the need for arbitrarily defining weights for vertex and edge insertion/deletion/substitution, and we experimentally found it was sensitive to subgraphs' sizes, which is detrimental to desideratum (2). We considered other metrics such as Katz centrality and the common-neighbors metric. However, as the name suggests, Katz is a measure of centrality, not similarity. Besides, it takes into account all the paths between two entities and is therefore sensitive to the absolute number of paths. Common neighbors does not consider predicates and is only suited for unlabeled graphs.<fnref n="5" />

In our experiments, we use the Jaccard coefficient is used to measure the
overlap in the 1-hop and 2-hop subgraphs of entities. In particular, its value

<footnote n="4">⁴ We consider both ingoing and outgoing edges to define those neighborhoods.</footnote>

<footnote n="5">⁵ Given, e.g., (SAP, headquarter, Germany), (BOSCH, headquarter, Germany), (Berlin, capitalOf, Germany), with common neighbors, SAP, BOSCH, and Berlin would be equally similar.</footnote>
denotes how similar the respective subgraphs of two entities are, which gives us insight into how much two entities are related based on graph-based information. The central entity ($e_1$ or $e_2$) is replaced by a unique token, e.g. :dummy. Triples forming the 1-hop neighborhood are extracted as is, while for the 2-hop paths the intermediate entity is ignored (written as □ below).

More formally: let $\mathcal{T}_1(A)$ be the set of all triples $\{(A, p, o) \cup (s, p, A)\}$, where A is replaced by a dummy identifier. Then, $J_1(e_1, e_2)$ is the Jaccard overlap of $\mathcal{T}_1(e_1)$ and $\mathcal{T}_1(e_2)$, i.e., $\frac{|\mathcal{T}_1(e_1) \cap \mathcal{T}_1(e_2)|}{|\mathcal{T}_1(e_1) \cup \mathcal{T}_1(e_2)|}$.

Let $\mathcal{T}_2(A)$ be the set of all paths $\{(A, p1, \square, p2, o) \cup (s, p1, \square, p2, A)\}$, where A is replaced by a dummy identifier. Then $J_2(e_1, e_2)$ is the Jaccard overlap of $\mathcal{T}_2(e_1)$ and $\mathcal{T}_2(e_2)$, i.e., $\frac{|\mathcal{T}_2(e_1) \cap \mathcal{T}_2(e_2)|}{|\mathcal{T}_2(e_1) \cup \mathcal{T}_2(e_2)|}$.

Following Fig. 1, triples in the two 1-hop subgraphs are:

<table><thead><tr><th>Bob</th><th>Julie</th></tr></thead><tbody><tr><td>(:dummy, plays, Guitar)</td><td>(:dummy, plays, Guitar)</td></tr><tr><td>(:dummy, livesIn, Mannheim)</td><td>(:dummy, livesIn, Karlsruhe)</td></tr><tr><td>(:dummy, friend, Anna)</td><td>(:dummy, friend, Roger)</td></tr></tbody></table>

When considering 2-hop information, we also consider the following paths which are shared between Bob's and Julie's subgraphs (reverse edges included):

(:dummy, livesIn, □, inCountry, Germany)
(:dummy, livesIn, □, inRegion, Baden-Württemberg)
(:dummy, friend, □, staffMember⁻¹, Vodafone)

The similarity of Bob and Julie considering the 1-hop subgraphs only is $J_1(\text{Bob}, \text{Julie}) = 0.2$ (since they share one out of five triples). Considering the 2-hop neighborhood graphs: $J_2(\text{Bob}, \text{Julie}) = 0.5$ (since they share three out of six two-hop paths). For instance, we consider (:dummy, livesIn, □, inCountry, Germany) as the full path. This allows for capturing indirect dependencies, such as both Julie and Bob live in a city in Germany, or both Julie and Bob know someone who works for Vodafone. This approach better fits the general and com-monsense idea that two entities should already be similar if they share relations to entities which are similar themselves according to their respective 1-hop sub-graphs. Moreover, keeping the intermediate entity in the path would outweigh information already counted as part of the 1-hop subgraph, especially if the intermediate entity were a high degree entity (desideratum (3)). For any entity in the 1-hop neighborhood of a central entity, its own 1-hop neighborhood is known and fixed. Consequently, whenever there is a match between triples in the 1-hop neighborhood of two central entities being compared, there would be as many matches in their respective 2-hop subgraphs as there are relations connecting their shared 1-hop neighbors. For example, if Bob and Julie were living in the same city, e.g. Mannheim, this information would be counted multiple times – in (:dummy, livesIn, Mannheim, inCountry, Germany), (:dummy, livesIn, Mannheim, inRegion, Baden-Württemberg), etc.. The resulting relatedness between Bob and Julie would thus be overly influenced by that single
denotes how similar the respective subgraphs of two entities are, which gives us insight into how much two entities are related based on graph-based information. The central entity ($e_1$ or $e_2$) is replaced by a unique token, e.g. :dummy. Triples forming the 1-hop neighborhood are extracted as is, while for the 2-hop paths the intermediate entity is ignored (written as □ below).

More formally: let $\mathcal{T}_1(A)$ be the set of all triples $\{(A, p, o) \cup (s, p, A)\}$, where A is replaced by a dummy identifier. Then, $J_1(e_1, e_2)$ is the Jaccard overlap of $\mathcal{T}_1(e_1)$ and $\mathcal{T}_1(e_2)$, i.e., $\frac{|\mathcal{T}_1(e_1) \cap \mathcal{T}_1(e_2)|}{|\mathcal{T}_1(e_1) \cup \mathcal{T}_1(e_2)|}$.

Let $\mathcal{T}_2(A)$ be the set of all paths $\{(A, p1, \square, p2, o) \cup (s, p1, \square, p2, A)\}$, where A is replaced by a dummy identifier. Then $J_2(e_1, e_2)$ is the Jaccard overlap of $\mathcal{T}_2(e_1)$ and $\mathcal{T}_2(e_2)$, i.e., $\frac{|\mathcal{T}_2(e_1) \cap \mathcal{T}_2(e_2)|}{|\mathcal{T}_2(e_1) \cup \mathcal{T}_2(e_2)|}$.

Following Fig. 1, triples in the two 1-hop subgraphs are:

<table><thead><tr><th>Bob</th><th>Julie</th></tr></thead><tbody><tr><td>(:dummy, plays, Guitar)</td><td>(:dummy, plays, Guitar)</td></tr><tr><td>(:dummy, livesIn, Mannheim)</td><td>(:dummy, livesIn, Karlsruhe)</td></tr><tr><td>(:dummy, friend, Anna)</td><td>(:dummy, friend, Roger)</td></tr></tbody></table>

When considering 2-hop information, we also consider the following paths which are shared between Bob's and Julie's subgraphs (reverse edges included):

(:dummy, livesIn, □, inCountry, Germany)
(:dummy, livesIn, □, inRegion, Baden-Württemberg)
(:dummy, friend, □, staffMember⁻¹, Vodafone)

The similarity of Bob and Julie considering the 1-hop subgraphs only is $J_1(\text{Bob}, \text{Julie}) = 0.2$ (since they share one out of five triples). Considering the 2-hop neighborhood graphs: $J_2(\text{Bob}, \text{Julie}) = 0.5$ (since they share three out of six two-hop paths). For instance, we consider (:dummy, livesIn, □, inCountry, Germany) as the full path. This allows for capturing indirect dependencies, such as both Julie and Bob live in a city in Germany, or both Julie and Bob know someone who works for Vodafone. This approach better fits the general and com-monsense idea that two entities should already be similar if they share relations to entities which are similar themselves according to their respective 1-hop sub-graphs. Moreover, keeping the intermediate entity in the path would outweigh information already counted as part of the 1-hop subgraph, especially if the intermediate entity were a high degree entity (desideratum (3)). For any entity in the 1-hop neighborhood of a central entity, its own 1-hop neighborhood is known and fixed. Consequently, whenever there is a match between triples in the 1-hop neighborhood of two central entities being compared, there would be as many matches in their respective 2-hop subgraphs as there are relations connecting their shared 1-hop neighbors. For example, if Bob and Julie were living in the same city, e.g. Mannheim, this information would be counted multiple times – in (:dummy, livesIn, Mannheim, inCountry, Germany), (:dummy, livesIn, Mannheim, inRegion, Baden-Württemberg), etc.. The resulting relatedness between Bob and Julie would thus be overly influenced by that single
<figure><img src="image_2.png" /><figcaption>Fig. 1. 1-hop and 2-hop subgraphs for Bob and Julie.</figcaption></figure>

fact. Similarly, if they were living in distinct cities – which is the case in our example – their relatedness score would be unduly penalized.

## 3.2 Embedding vs. Graph: a Different Notion of Similarity?

Based on a given entity e, it is possible to get its N closest neighbors in the KG (using the $J_1$ and $J_2$ metrics defined above) and in the embedding space (using cosine similarity). To assess if the two notions of similarity are different, we need to measure how much the two lists of closest neighbors between these two approaches actually overlap.

A common measure to compare two ranked lists is Kendall’s Tau [18]. However, it suffers from two important caveats: the two lists should be of the same size and they should contain the same set of items. Considering our experimental purposes, the latter limitation is undesirable, as there is no reason why these two lists (of the top N similar entities according to $J_1$ or $J_2$, and in the embedding space) should share the same set of entities.

Consequently, we use the Rank-Biased Overlap (RBO) [39] to compare the similarity of two ranked lists. Unlike Kendall’s Tau which is correlation-based, RBO is intersection-based. Most importantly, RBO can handle lists containing different items. It also allows for the weighting of rankings, giving higher importance to items at the top of the lists through tweaking the *persistence* parameter *p*. A small value for *p* will only consider the first few items, whereas a larger value will encompass more items. However, we experimentally found that results are very sensitive to the choice of *p*. Besides, when comparing two ranked lists, we want to consider all of their items. Consequently, we stick to the default parameter strategy *p* = 1 as proposed in an open-source implementation of RBO<fnref n="6" /> that we used in our experiments. Formally, RBO is expressed as follows.

Let *S* and *T* be two ranking lists, and let $S_i$ (resp. $T_i$) be the element at rank *i* in list *S* (resp. *T*). Then, $S_{c:d}$ (resp. $T_{c:d}$) denotes the set of the elements from position *c* to position *d* in list *S* (resp. *T*). At each depth *d*, the intersection of lists *S* and *T* to depth *d* is defined as $I_{S,T,d} = S_{1:d} \cap T_{1:d}$, and their overlap up

<footnote n="6">⁶ https://github.com/changyaochen/rbo/</footnote>
<figure><img src="image_2.png" /><figcaption>Fig. 1. 1-hop and 2-hop subgraphs for Bob and Julie.</figcaption></figure>

fact. Similarly, if they were living in distinct cities – which is the case in our example – their relatedness score would be unduly penalized.

## 3.2 Embedding vs. Graph: a Different Notion of Similarity?

Based on a given entity e, it is possible to get its N closest neighbors in the KG (using the $J_1$ and $J_2$ metrics defined above) and in the embedding space (using cosine similarity). To assess if the two notions of similarity are different, we need to measure how much the two lists of closest neighbors between these two approaches actually overlap.

A common measure to compare two ranked lists is Kendall’s Tau [18]. However, it suffers from two important caveats: the two lists should be of the same size and they should contain the same set of items. Considering our experimental purposes, the latter limitation is undesirable, as there is no reason why these two lists (of the top N similar entities according to $J_1$ or $J_2$, and in the embedding space) should share the same set of entities.

Consequently, we use the Rank-Biased Overlap (RBO) [39] to compare the similarity of two ranked lists. Unlike Kendall’s Tau which is correlation-based, RBO is intersection-based. Most importantly, RBO can handle lists containing different items. It also allows for the weighting of rankings, giving higher importance to items at the top of the lists through tweaking the *persistence* parameter *p*. A small value for *p* will only consider the first few items, whereas a larger value will encompass more items. However, we experimentally found that results are very sensitive to the choice of *p*. Besides, when comparing two ranked lists, we want to consider all of their items. Consequently, we stick to the default parameter strategy *p* = 1 as proposed in an open-source implementation of RBO<fnref n="6" /> that we used in our experiments. Formally, RBO is expressed as follows.

Let *S* and *T* be two ranking lists, and let $S_i$ (resp. $T_i$) be the element at rank *i* in list *S* (resp. *T*). Then, $S_{c:d}$ (resp. $T_{c:d}$) denotes the set of the elements from position *c* to position *d* in list *S* (resp. *T*). At each depth *d*, the intersection of lists *S* and *T* to depth *d* is defined as $I_{S,T,d} = S_{1:d} \cap T_{1:d}$, and their overlap up

<footnote n="6">⁶ https://github.com/changyaochen/rbo/</footnote>
to depth *d* is the size of the intersection, i.e. |*I*<sub>S,T,d</sub>|. RBO relies on the notion of agreement between *S* and *T* to depth *d*:

$$A_{S,T,d} = \frac{|I_{S,T,d}|}{d} \qquad (1)$$

In the case where the persistence parameter *p* = 1, the RBO formula is expressed as follows:

$$RBO(S, T, k) = \frac{1}{k} \sum_{d=1}^{k} A_{S,T,d} \qquad (2)$$

where *k* is the size of the two lists *S* and *T* being compared. For the case where *p* ≠ 1, we refer the reader to [39].

In Fig. 2, we illustrate how RBO@5 is calculated for two lists of top-5 neighbors, where one relies on similarity in the embedding space whereas the other one depends on the Jaccard overlap between entities' subgraphs. We see that (i) RBO can be calculated on lists with different sets of entities, (ii) RBO weighs more matches that occur at the top of the lists, and (iii) RBO is sensitive to the order of matching entities in the respective lists.

<figure><img src="image_3.png" /><figcaption>Fig. 2. Illustration of RBO@5 values in four cases.</figcaption></figure>

# 4 Experiments

In this section, we detail our experimental setting, i.e. the KGEMs (Section 4.1) and datasets (Section 4.2) used in the experiments.

## 4.1 Knowledge Graph Embedding Models

In our experiments, we use seven popular KGEMs from different families of models: geometric-based (TransE [6], TransD [15], BoxE [1]), multiplicative
to depth *d* is the size of the intersection, i.e. |*I*<sub>S,T,d</sub>|. RBO relies on the notion of agreement between *S* and *T* to depth *d*:

$$A_{S,T,d} = \frac{|I_{S,T,d}|}{d} \qquad (1)$$

In the case where the persistence parameter *p* = 1, the RBO formula is expressed as follows:

$$RBO(S, T, k) = \frac{1}{k} \sum_{d=1}^{k} A_{S,T,d} \qquad (2)$$

where *k* is the size of the two lists *S* and *T* being compared. For the case where *p* ≠ 1, we refer the reader to [39].

In Fig. 2, we illustrate how RBO@5 is calculated for two lists of top-5 neighbors, where one relies on similarity in the embedding space whereas the other one depends on the Jaccard overlap between entities' subgraphs. We see that (i) RBO can be calculated on lists with different sets of entities, (ii) RBO weighs more matches that occur at the top of the lists, and (iii) RBO is sensitive to the order of matching entities in the respective lists.

<figure><img src="image_3.png" /><figcaption>Fig. 2. Illustration of RBO@5 values in four cases.</figcaption></figure>

# 4 Experiments

In this section, we detail our experimental setting, i.e. the KGEMs (Section 4.1) and datasets (Section 4.2) used in the experiments.

## 4.1 Knowledge Graph Embedding Models

In our experiments, we use seven popular KGEMs from different families of models: geometric-based (TransE [6], TransD [15], BoxE [1]), multiplicative
(RESCAL [22], DistMult [40], TuckER [4]), and convolutional-based (ConvE [8]) models. Other models have been considered, e.g. ComplEx [37] and RotatE [35]. However, these models generate complex-valued embeddings, which requires a different approach than using simple cosine similarity. We trained these KGEMs using PyKEEN<sup>7</sup> with the provided configuration files, when available. For those datasets with no reported best hyperparameters, we used the hyperparameters reported in the original paper (e.g. for YAGO4-19K, with best hyperparameters found in [11]) or performed manual hyperparameter search and kept the sets of hyperparameters leading to the best results on the validation sets (e.g. for AIFB). We also trained RDF2Vec [24], which is a versatile embedding approach that can be adapted for different downstream applications and that relies on path-based information to encode entities (only). Even though RDF2Vec is not specifically designed to handle the link prediction task and is not evaluated w.r.t. it, its entity embeddings are expected to reflect their similarity or relatedness in the KG to some extent [23]. We used the implementation provided by pyRDF2Vec<sup>8</sup> with the default hyperparameters reported in [23]: embeddings of dimension 200, with 2,000 walks maximum, a depth of 4, a window size of 5, and 25 epochs for training word2vec [21] with the continuous skip-gram architecture.

## 4.2 Datasets

Since we also want to analyze results on a per-class basis, we consider only benchmark datasets which also come with a schema of multiple classes and relations. We use the following datasets in our evaluation: AIFB [5], Codex-S and Codex-M [27], DBpedia50 [29], FB15K-237 [36], and YAGO4-19K [11]. Entity types are not directly available in the original repository of DBpedia50<sup>9</sup> and FB15k-237<sup>10</sup>. However, we ran SPARQL queries against DBpedia to get entity types for DBpedia50. The resulting class hierarchy is a subset of the DBpedia ontology, with a maximum depth of 8, an average fan-out (branching factor) of 3.40, and each entity being typed with an average of 4.55 classes. For FB15K-237, we reused the entity-typed version presented in [11]. Table 1 provides finer-grained statistics for these datasets.

# 5 Results and Discussion

## 5.1 Different Notions of Similarity (RQ1)

Table 2 reports results w.r.t. rank-based metrics (MRR, Hits@K) and RBO@K for all the KGEMs and datasets considered in this work. Similar to MRR and Hits@K, RBO is computed for all entities in a KG and the average is reported.

When retaining 1-hop subgraphs (from R1@3 to R1@100 in Table 2), in most cases TuckER has the highest RBO values. In other words, it appears to fulfill

<footnote n="7">⁷ https://github.com/pykeen/pykeen/
⁸ https://github.com/IBCNServices/pyRDF2Vec/
⁹ https://github.com/bxshi/ConMask/
¹⁰ https://www.microsoft.com/en-us/download/details.aspx?id=52312/</footnote>
(RESCAL [22], DistMult [40], TuckER [4]), and convolutional-based (ConvE [8]) models. Other models have been considered, e.g. ComplEx [37] and RotatE [35]. However, these models generate complex-valued embeddings, which requires a different approach than using simple cosine similarity. We trained these KGEMs using PyKEEN<sup>7</sup> with the provided configuration files, when available. For those datasets with no reported best hyperparameters, we used the hyperparameters reported in the original paper (e.g. for YAGO4-19K, with best hyperparameters found in [11]) or performed manual hyperparameter search and kept the sets of hyperparameters leading to the best results on the validation sets (e.g. for AIFB). We also trained RDF2Vec [24], which is a versatile embedding approach that can be adapted for different downstream applications and that relies on path-based information to encode entities (only). Even though RDF2Vec is not specifically designed to handle the link prediction task and is not evaluated w.r.t. it, its entity embeddings are expected to reflect their similarity or relatedness in the KG to some extent [23]. We used the implementation provided by pyRDF2Vec<sup>8</sup> with the default hyperparameters reported in [23]: embeddings of dimension 200, with 2,000 walks maximum, a depth of 4, a window size of 5, and 25 epochs for training word2vec [21] with the continuous skip-gram architecture.

## 4.2 Datasets

Since we also want to analyze results on a per-class basis, we consider only benchmark datasets which also come with a schema of multiple classes and relations. We use the following datasets in our evaluation: AIFB [5], Codex-S and Codex-M [27], DBpedia50 [29], FB15K-237 [36], and YAGO4-19K [11]. Entity types are not directly available in the original repository of DBpedia50<sup>9</sup> and FB15k-237<sup>10</sup>. However, we ran SPARQL queries against DBpedia to get entity types for DBpedia50. The resulting class hierarchy is a subset of the DBpedia ontology, with a maximum depth of 8, an average fan-out (branching factor) of 3.40, and each entity being typed with an average of 4.55 classes. For FB15K-237, we reused the entity-typed version presented in [11]. Table 1 provides finer-grained statistics for these datasets.

# 5 Results and Discussion

## 5.1 Different Notions of Similarity (RQ1)

Table 2 reports results w.r.t. rank-based metrics (MRR, Hits@K) and RBO@K for all the KGEMs and datasets considered in this work. Similar to MRR and Hits@K, RBO is computed for all entities in a KG and the average is reported.

When retaining 1-hop subgraphs (from R1@3 to R1@100 in Table 2), in most cases TuckER has the highest RBO values. In other words, it appears to fulfill

<footnote n="7"><sup>7</sup> https://github.com/pykeen/pykeen/
<sup>8</sup> https://github.com/IBCNServices/pyRDF2Vec/
<sup>9</sup> https://github.com/bxshi/ConMask/
<sup>10</sup> https://www.microsoft.com/en-us/download/details.aspx?id=52312/</footnote>
<table><caption>**Table 1.** Datasets used in the experiments. Column header from left to right: number of entities, relations (predicates), classes, train triples, validation triples, and test triples.</caption><thead><tr><th>Dataset</th><th>|E|</th><th>|R|</th><th>|C|</th><th>|T<sub>train</sub>|</th><th>|T<sub>valid</sub>|</th><th>|T<sub>test</sub>|</th></tr></thead><tbody><tr><td>AIFB</td><td>2,389</td><td>16</td><td>18</td><td>14,170</td><td>745</td><td>785</td></tr><tr><td>Codex-S</td><td>2,034</td><td>42</td><td>502</td><td>32,888</td><td>1,827</td><td>1,827</td></tr><tr><td>Codex-M</td><td>17,050</td><td>51</td><td>1,503</td><td>185,584</td><td>10,310</td><td>10,311</td></tr><tr><td>DBpedia50</td><td>24,624</td><td>351</td><td>285</td><td>32,388</td><td>123</td><td>2,098</td></tr><tr><td>FB15k237</td><td>14,541</td><td>237</td><td>643</td><td>272,115</td><td>17,535</td><td>20,466</td></tr><tr><td>YAGO4-19k</td><td>18,960</td><td>74</td><td>1,232</td><td>27,447</td><td>485</td><td>463</td></tr></tbody></table>

the KGE entity similarity assumption best, having the highest tendency to position similar entities closer in the vector space. When moving to 2-hop subgraphs, DistMult and TuckER are the models with the best alignment capabilities (Table 2). BoxE, RDF2Vec, RESCAL, and TransD fare worse than other models. The substantially lower RBO values for RDF2Vec can be related to the fact that the vector distance in RDF2vec space mixes similarity and relatedness, where the latter does not show a strong overlap in common paths [23]. For example, it will position USA close to Washington D.C., but they have only few common paths starting/ending in the respective entities.

A crucial observation from our study is that the behavior of KGEMs varies not only across datasets but also across different classes within these datasets. Table 3 illustrates this insight on classes that were selected to provide a more intuitive understanding of how different KGEMs are differently able to capture similarities between instances of the same class. In particular, we clearly see that on YAGO4-19K, TransE captures the notion of SpanishMunicipality quite well compared to other models, but does not do well for MusicPlaylist entities (highlighted in blue). This means that different models are better suited for capturing the nuances of specific classes. Jain et al. [14] demonstrated that semantic representation in embeddings is not consistent across all KG entities, but is restricted to a small subset of them. Our results point in the same direction and suggest that KGEMs are not equipped with consistent capabilities to learn similar embeddings for entities within the same class. This problem appears at two different levels: First, we noted a general misalignment in the representation of certain classes across various datasets, regardless of the KGEM used. Second, there is noticeable inconsistency within specific classes, where entities might align well with their graph-based proximity in one KGEM but not in another.

To generalize this observation, we computed correlation matrices for RBO values across models and datasets, focusing on class-specific performance. Our analysis, averaging these matrices, gives us a broader view of how different KGEMs perceive similarity on a per-class basis. Fig. 4 shows the rank correlations between models when considering 2-hop subgraphs, averaged over all datasets and RBO values. Notably, RDF2Vec stands out as it seems to capture a distinct notion of similarity compared to other KGEMs (cf. [23]). It demonstrates that RDF2Vec fares better on different classes than other KGEMs. We also ob-
<table><caption>**Table 1.** Datasets used in the experiments. Column header from left to right: number of entities, relations (predicates), classes, train triples, validation triples, and test triples.</caption><thead><tr><th>Dataset</th><th>|E|</th><th>|R|</th><th>|C|</th><th>|T<sub>train</sub>|</th><th>|T<sub>valid</sub>|</th><th>|T<sub>test</sub>|</th></tr></thead><tbody><tr><td>AIFB</td><td>2,389</td><td>16</td><td>18</td><td>14,170</td><td>745</td><td>785</td></tr><tr><td>Codex-S</td><td>2,034</td><td>42</td><td>502</td><td>32,888</td><td>1,827</td><td>1,827</td></tr><tr><td>Codex-M</td><td>17,050</td><td>51</td><td>1,503</td><td>185,584</td><td>10,310</td><td>10,311</td></tr><tr><td>DBpedia50</td><td>24,624</td><td>351</td><td>285</td><td>32,388</td><td>123</td><td>2,098</td></tr><tr><td>FB15k237</td><td>14,541</td><td>237</td><td>643</td><td>272,115</td><td>17,535</td><td>20,466</td></tr><tr><td>YAGO4-19k</td><td>18,960</td><td>74</td><td>1,232</td><td>27,447</td><td>485</td><td>463</td></tr></tbody></table>

the KGE entity similarity assumption best, having the highest tendency to position similar entities closer in the vector space. When moving to 2-hop subgraphs, DistMult and TuckER are the models with the best alignment capabilities (Table 2). BoxE, RDF2Vec, RESCAL, and TransD fare worse than other models. The substantially lower RBO values for RDF2Vec can be related to the fact that the vector distance in RDF2vec space mixes similarity and relatedness, where the latter does not show a strong overlap in common paths [23]. For example, it will position USA close to Washington D.C., but they have only few common paths starting/ending in the respective entities.

A crucial observation from our study is that the behavior of KGEMs varies not only across datasets but also across different classes within these datasets. Table 3 illustrates this insight on classes that were selected to provide a more intuitive understanding of how different KGEMs are differently able to capture similarities between instances of the same class. In particular, we clearly see that on YAGO4-19K, TransE captures the notion of SpanishMunicipality quite well compared to other models, but does not do well for MusicPlaylist entities (highlighted in blue). This means that different models are better suited for capturing the nuances of specific classes. Jain et al. [14] demonstrated that semantic representation in embeddings is not consistent across all KG entities, but is restricted to a small subset of them. Our results point in the same direction and suggest that KGEMs are not equipped with consistent capabilities to learn similar embeddings for entities within the same class. This problem appears at two different levels: First, we noted a general misalignment in the representation of certain classes across various datasets, regardless of the KGEM used. Second, there is noticeable inconsistency within specific classes, where entities might align well with their graph-based proximity in one KGEM but not in another.

To generalize this observation, we computed correlation matrices for RBO values across models and datasets, focusing on class-specific performance. Our analysis, averaging these matrices, gives us a broader view of how different KGEMs perceive similarity on a per-class basis. Fig. 4 shows the rank correlations between models when considering 2-hop subgraphs, averaged over all datasets and RBO values. Notably, RDF2Vec stands out as it seems to capture a distinct notion of similarity compared to other KGEMs (cf. [23]). It demonstrates that RDF2Vec fares better on different classes than other KGEMs. We also ob-
<table><caption>**Table 3.** RBO@10 values per model for selected classes. Numbers in parenthesis indicate the rank of a class for a given model.</caption><thead><tr><th>Dataset</th><th>Class</th><th>RDF2Vec</th><th>TransE</th><th>TransD</th><th>BoxE</th><th>RESCAL</th><th>DistMult</th><th>Tucker</th><th>ConvE</th></tr></thead><tbody><tr><td>AIFB</td><td>Book<br>inProceedings</td><td>0.383 (1)<br>0.048 (13)</td><td>0.560 (1)<br>0.419 (3)</td><td>0.314 (7)<br>0.227 (11)</td><td>0.551 (1)<br>0.461 (3)</td><td>0.613 (1)<br>0.412 (2)</td><td>0.621 (1)<br>0.453 (3)</td><td>0.374 (5)<br>0.332 (11)</td><td>0.214 (13)<br>0.326 (3)</td></tr><tr><td>Codex-S</td><td>Republic<br>NationalAcademy</td><td>0.000 (26)<br>0.000 (27)</td><td>0.556 (3)<br>0.558 (2)</td><td>0.236 (12)<br>0.217 (14)</td><td>0.463 (5)<br>0.446 (8)</td><td>0.363 (3)<br>0.383 (2)</td><td>0.339 (10)<br>0.323 (11)</td><td>0.399 (9)<br>0.366 (12)</td><td>0.311 (12)<br>0.301 (14)</td></tr><tr><td>Codex-M</td><td>UrbanDistrict<br>CollegeTown</td><td>0.002 (66)<br>0.000 (88)</td><td>0.538 (9)<br>0.615 (2)</td><td>0.178 (40)<br>0.205 (30)</td><td>0.482 (11)<br>0.653 (2)</td><td>0.057 (66)<br>0.133 (30)</td><td>0.494 (14)<br>0.670 (1)</td><td>0.504 (8)<br>0.640 (2)</td><td>0.327 (12)<br>0.337 (27)</td></tr><tr><td>DBpedia50</td><td>Album<br>EthnicGroup</td><td>0.134 (52)<br>0.028 (136)</td><td>0.336 (33)<br>0.469 (9)</td><td>0.270 (23)<br>0.227 (36)</td><td>0.351 (30)<br>0.452 (9)</td><td>0.300 (9)<br>0.400 (1)</td><td>0.310 (42)<br>0.458 (9)</td><td>0.387 (27)<br>0.434 (16)</td><td>0.315 (29)<br>0.388 (12)</td></tr><tr><td>FB15k237</td><td>Periodicals<br>BaseballPlayer</td><td>0.002 (414)<br>0.001 (420)</td><td>0.115 (109)<br>0.146 (78)</td><td>0.255 (64)<br>0.336 (31)</td><td>0.150 (163)<br>0.236 (93)</td><td>0.124 (181)<br>0.143 (160)</td><td>0.139 (280)<br>0.277 (104)</td><td>0.590 (11)<br>0.631 (4)</td><td>0.638 (3)<br>0.598 (12)</td></tr><tr><td>YAGO-19k</td><td>MusicPlaylist<br>SpanishMunicipality</td><td>0.233 (35)<br>0.226 (40)</td><td><b>0.325 (94)</b><br><b>0.636 (1)</b></td><td>0.322 (37)<br>0.460 (4)</td><td>0.411 (10)<br>0.327 (33)</td><td>0.350 (41)<br>0.236 (95)</td><td>0.366 (83)<br>0.531 (20)</td><td>0.627 (4)<br>0.551 (20)</td><td>0.554 (7)<br>0.505 (17)</td></tr></tbody></table>

<figure><img src="image_4.png" /><figcaption>**Fig. 3.** Correlations between MRR and RBO values with 1-hop and 2-hop subgraphs.</figcaption></figure>

## 5.2 Correlation between Rank-based Metrics and RBO (RQ2)

As previously mentioned, some KGEMs fare better than others in terms of RBO values, e.g. DistMult and TuckER. It is important to note that RBO values are designed to measure how well a KGEM entity embedding aligns with the concept of proximity within the original KG. This goal is distinct from the aim of rank-based metrics, which assess how effectively a KGEM assigns higher plausibility scores to ground-truth triples. In what follows, we aim to study the relationship between these two types of metrics: how does KGEM performance in rank-based metrics correlate with RBO values?

We analyzed this using Pearson's correlation coefficient, comparing MRR and RBO values for 1-hop and 2-hop subgraphs across our 6 datasets, as depicted in Fig. 3. The results revealed notable disparities among the models. For instance, TransE and TuckER consistently show moderate to high correlation between MRR and RBO values. This implies that for these models, higher plausibility scoring of triples often aligns with better representation of entity proximity in the KG. However, this trend is not uniform across all models, and we do not observe general trends across families of models: while TransE and TransD are both translational models, correlation results are quite different.
<table><caption>**Table 3.** RBO@10 values per model for selected classes. Numbers in parenthesis indicate the rank of a class for a given model.</caption><thead><tr><th>Dataset</th><th>Class</th><th>RDF2Vec</th><th>TransE</th><th>TransD</th><th>BoxE</th><th>RESCAL</th><th>DistMult</th><th>Tucker</th><th>ConvE</th></tr></thead><tbody><tr><td>AIFB</td><td>Book<br>inProceedings</td><td>0.383 (1)<br>0.048 (13)</td><td>0.560 (1)<br>0.419 (3)</td><td>0.314 (7)<br>0.227 (11)</td><td>0.551 (1)<br>0.461 (3)</td><td>0.613 (1)<br>0.412 (2)</td><td>0.621 (1)<br>0.453 (3)</td><td>0.374 (5)<br>0.332 (11)</td><td>0.214 (13)<br>0.326 (3)</td></tr><tr><td>Codex-S</td><td>Republic<br>NationalAcademy</td><td>0.000 (26)<br>0.000 (27)</td><td>0.556 (3)<br>0.558 (2)</td><td>0.236 (12)<br>0.217 (14)</td><td>0.463 (5)<br>0.446 (8)</td><td>0.363 (3)<br>0.383 (2)</td><td>0.339 (10)<br>0.323 (11)</td><td>0.399 (9)<br>0.366 (12)</td><td>0.311 (12)<br>0.301 (14)</td></tr><tr><td>Codex-M</td><td>UrbanDistrict<br>CollegeTown</td><td>0.002 (66)<br>0.000 (88)</td><td>0.538 (9)<br>0.615 (2)</td><td>0.178 (40)<br>0.205 (30)</td><td>0.482 (11)<br>0.653 (2)</td><td>0.057 (66)<br>0.133 (30)</td><td>0.494 (14)<br>0.670 (1)</td><td>0.504 (8)<br>0.640 (2)</td><td>0.327 (12)<br>0.337 (27)</td></tr><tr><td>DBpedia50</td><td>Album<br>EthnicGroup</td><td>0.134 (52)<br>0.028 (136)</td><td>0.336 (33)<br>0.469 (9)</td><td>0.270 (23)<br>0.227 (36)</td><td>0.351 (30)<br>0.452 (9)</td><td>0.300 (9)<br>0.400 (1)</td><td>0.310 (42)<br>0.458 (9)</td><td>0.387 (27)<br>0.434 (16)</td><td>0.315 (29)<br>0.388 (12)</td></tr><tr><td>FB15k237</td><td>Periodicals<br>BaseballPlayer</td><td>0.002 (414)<br>0.001 (420)</td><td>0.115 (109)<br>0.146 (78)</td><td>0.255 (64)<br>0.336 (31)</td><td>0.150 (163)<br>0.236 (93)</td><td>0.124 (181)<br>0.143 (160)</td><td>0.139 (280)<br>0.277 (104)</td><td>0.590 (11)<br>0.631 (4)</td><td>0.638 (3)<br>0.598 (12)</td></tr><tr><td>YAGO-19k</td><td>MusicPlaylist<br>SpanishMunicipality</td><td>0.233 (35)<br>0.226 (40)</td><td><b>0.325 (94)</b><br><b>0.636 (1)</b></td><td>0.322 (37)<br>0.460 (4)</td><td>0.411 (10)<br>0.327 (33)</td><td>0.350 (41)<br>0.236 (95)</td><td>0.366 (83)<br>0.531 (20)</td><td>0.627 (4)<br>0.551 (20)</td><td>0.554 (7)<br>0.505 (17)</td></tr></tbody></table>

<figure><img src="image_4.png" /><figcaption>**Fig. 3.** Correlations between MRR and RBO values with 1-hop and 2-hop subgraphs.</figcaption></figure>

## 5.2 Correlation between Rank-based Metrics and RBO (RQ2)

As previously mentioned, some KGEMs fare better than others in terms of RBO values, e.g. DistMult and TuckER. It is important to note that RBO values are designed to measure how well a KGEM entity embedding aligns with the concept of proximity within the original KG. This goal is distinct from the aim of rank-based metrics, which assess how effectively a KGEM assigns higher plausibility scores to ground-truth triples. In what follows, we aim to study the relationship between these two types of metrics: how does KGEM performance in rank-based metrics correlate with RBO values?

We analyzed this using Pearson's correlation coefficient, comparing MRR and RBO values for 1-hop and 2-hop subgraphs across our 6 datasets, as depicted in Fig. 3. The results revealed notable disparities among the models. For instance, TransE and TuckER consistently show moderate to high correlation between MRR and RBO values. This implies that for these models, higher plausibility scoring of triples often aligns with better representation of entity proximity in the KG. However, this trend is not uniform across all models, and we do not observe general trends across families of models: while TransE and TransD are both translational models, correlation results are quite different.
<figure><img src="image_5.png" /><figcaption>**Fig. 4.** Rank correlations between KGEMs averaged over all datasets and all RBO metrics ($K = 3, 10, 100$).</figcaption></figure>

From a coarse-grained viewpoint, we note that MRR *generally* does not correlate well with RBO values. Two specific cases are that MRR is heavily (and positively) correlated with RBO@10 (1-hop) and RBO@100 (2-hop), which means that MRR can be seen as a reliable proxy measure for those two metrics (and vice versa). However, it still remains to be explained why the correlation between MRR and RBO@10 (1-hop) is so high, while a substantial drop in correlation between MRR and RBO@100 (1-hop) is observed. Another result to be assessed is why MRR correlates more with RBO@K with lower $K$ values when considering 1-hop subgraph and higher $K$ values when considering 2-hop subgraphs. Therefore, **RQ2** cannot be answered in a conclusive way: while in the general case, the correlation between performance in grouping similar entities and link prediction performance is moderate at best, some metrics show a higher correlation. This has a severe practical implication: the commonly observed practice to pick a KGE model which is good at link prediction for a task heavily relying on entity similarity has to be considered a suboptimal strategy.

## 5.3 Analyzing Predicate Importance (RQ3)

We previously highlighted that different KGEMs are equipped with different notions of similarity in their respective embedding spaces. We also demonstrated that the adherence to the KGE entity similarity assumption can vary a lot between classes: for a given combination of dataset and model, how much the top-
<figure><img src="image_5.png" /><figcaption>**Fig. 4.** Rank correlations between KGEMs averaged over all datasets and all RBO metrics ($K = 3, 10, 100$).</figcaption></figure>

From a coarse-grained viewpoint, we note that MRR *generally* does not correlate well with RBO values. Two specific cases are that MRR is heavily (and positively) correlated with RBO@10 (1-hop) and RBO@100 (2-hop), which means that MRR can be seen as a reliable proxy measure for those two metrics (and vice versa). However, it still remains to be explained why the correlation between MRR and RBO@10 (1-hop) is so high, while a substantial drop in correlation between MRR and RBO@100 (1-hop) is observed. Another result to be assessed is why MRR correlates more with RBO@K with lower $K$ values when considering 1-hop subgraph and higher $K$ values when considering 2-hop subgraphs. Therefore, **RQ2** cannot be answered in a conclusive way: while in the general case, the correlation between performance in grouping similar entities and link prediction performance is moderate at best, some metrics show a higher correlation. This has a severe practical implication: the commonly observed practice to pick a KGE model which is good at link prediction for a task heavily relying on entity similarity has to be considered a suboptimal strategy.

## 5.3 Analyzing Predicate Importance (RQ3)

We previously highlighted that different KGEMs are equipped with different notions of similarity in their respective embedding spaces. We also demonstrated that the adherence to the KGE entity similarity assumption can vary a lot between classes: for a given combination of dataset and model, how much the top-
K neighbors of an entity in the embedding space align with its top-K neighbors in the graph largely depends on which class this entity belongs to.

To examine this behavior further, we pick random classes and count the predicates in the common triples of the top-K neighbors (in the embedding space) for entities of this class differs between models.

<table><caption>**Table 4.** Top 5 predicates for entities of classes MusicComposition and MusicGroup on YAGO4-19K, along with their importance score.</caption><thead><tr><th rowspan="2"></th><th colspan="2">MusicComposition</th><th colspan="2">MusicGroup</th></tr><tr><th>K=3</th><th>K=10</th><th>K=3</th><th>K=10</th></tr></thead><tbody><tr><td rowspan="5">RDF2Vec</td><td>isPartOf (0.52)</td><td>isPartOf (0.46)</td><td>genre (0.53)</td><td>genre (0.60)</td></tr><tr><td>inLanguage (0.18)</td><td>inLanguage (0.30)</td><td>foundingLocation (0.38)</td><td>foundingLocation (0.32)</td></tr><tr><td>hasPart (0.17)</td><td>hasPart (0.15)</td><td>award (0.10)</td><td>award (0.07)</td></tr><tr><td>composer (0.09)</td><td>composer (0.07)</td><td>memberOf (0.00)</td><td>memberOf (0.01)</td></tr><tr><td>lyricist (0.02)</td><td>lyricist (0.02)</td><td>byArtist (0.00)</td><td>byArtist (0.00)</td></tr><tr><td rowspan="5">TransE</td><td>isPartOf (0.40)</td><td>isPartOf (0.36)</td><td>genre (0.79)</td><td>genre (0.85)</td></tr><tr><td>hasPart (0.28)</td><td>hasPart (0.27)</td><td>foundingLocation (0.17)</td><td>foundingLocation (0.12)</td></tr><tr><td>inLanguage (0.16)</td><td>inLanguage (0.26)</td><td>award (0.04)</td><td>award (0.02)</td></tr><tr><td>composer (0.09)</td><td>composer (0.06)</td><td>memberOf (0.00)</td><td>memberOf (0.00)</td></tr><tr><td>citation (0.05)</td><td>citation (0.04)</td><td>byArtist (0.00)</td><td>knowsLanguage (0.00)</td></tr><tr><td rowspan="5">TransD</td><td>isPartOf (0.40)</td><td>inLanguage (0.37)</td><td>genre (0.84)</td><td>genre (0.87)</td></tr><tr><td>inLanguage (0.24)</td><td>isPartOf (0.31)</td><td>foundingLocation (0.13)</td><td>foundingLocation (0.11)</td></tr><tr><td>hasPart (0.23)</td><td>hasPart (0.23)</td><td>award (0.03)</td><td>award (0.02)</td></tr><tr><td>composer (0.08)</td><td>composer (0.06)</td><td>memberOf (0.00)</td><td>memberOf (0.00)</td></tr><tr><td>citation (0.03)</td><td>citation (0.03)</td><td>byArtist (0.00)</td><td>knowsLanguage (0.00)</td></tr><tr><td rowspan="5">RESCAL</td><td>isPartOf (0.42)</td><td>inLanguage (0.33)</td><td>genre (0.74)</td><td>genre (0.81)</td></tr><tr><td>inLanguage (0.23)</td><td>isPartOf (0.33)</td><td>foundingLocation (0.22)</td><td>foundingLocation (0.16)</td></tr><tr><td>hasPart (0.21)</td><td>hasPart (0.25)</td><td>award (0.03)</td><td>award (0.02)</td></tr><tr><td>composer (0.09)</td><td>composer (0.06)</td><td>memberOf (0.01)</td><td>memberOf (0.00)</td></tr><tr><td>citation (0.04)</td><td>citation (0.02)</td><td>knowsLanguage (0.00)</td><td>knowsLanguage (0.00)</td></tr><tr><td rowspan="5">DistMult</td><td>isPartOf (0.38)</td><td>inLanguage (0.39)</td><td>genre (0.81)</td><td>genre (0.86)</td></tr><tr><td>hasPart (0.26)</td><td>isPartOf (0.29)</td><td>foundingLocation (0.14)</td><td>foundingLocation (0.11)</td></tr><tr><td>inLanguage (0.22)</td><td>hasPart (0.22)</td><td>award (0.04)</td><td>award (0.02)</td></tr><tr><td>composer (0.08)</td><td>composer (0.05)</td><td>memberOf (0.00)</td><td>memberOf (0.00)</td></tr><tr><td>citation (0.05)</td><td>citation (0.03)</td><td>knowsLanguage (0.00)</td><td>knowsLanguage (0.00)</td></tr><tr><td rowspan="5">TuckER</td><td>isPartOf (0.38)</td><td>inLanguage (0.37)</td><td>genre (0.85)</td><td>genre (0.90)</td></tr><tr><td>hasPart (0.25)</td><td>isPartOf (0.30)</td><td>foundingLocation (0.11)</td><td>foundingLocation (0.08)</td></tr><tr><td>inLanguage (0.24)</td><td>hasPart (0.23)</td><td>award (0.04)</td><td>award (0.02)</td></tr><tr><td>composer (0.08)</td><td>composer (0.05)</td><td>memberOf (0.01)</td><td>memberOf (0.00)</td></tr><tr><td>citation (0.04)</td><td>citation (0.03)</td><td>byArtist (0.00)</td><td>byArtist (0.00)</td></tr><tr><td rowspan="5">BoxE</td><td>isPartOf (0.48)</td><td>isPartOf (0.40)</td><td>genre (0.73)</td><td>genre (0.83)</td></tr><tr><td>hasPart (0.29)</td><td>hasPart (0.30)</td><td>foundingLocation (0.23)</td><td>foundingLocation (0.15)</td></tr><tr><td>inLanguage (0.11)</td><td>inLanguage (0.19)</td><td>award (0.04)</td><td>award (0.02)</td></tr><tr><td>composer (0.10)</td><td>composer (0.07)</td><td>byArtist (0.00)</td><td>byArtist (0.00)</td></tr><tr><td>lyricist (0.02)</td><td>citation (0.02)</td><td>memberOf (0.00)</td><td>memberOf (0.00)</td></tr><tr><td rowspan="5">ConvE</td><td>isPartOf (0.39)</td><td>isPartOf (0.33)</td><td>genre (0.83)</td><td>genre (0.86)</td></tr><tr><td>hasPart (0.25)</td><td>inLanguage (0.31)</td><td>foundingLocation (0.14)</td><td>foundingLocation (0.12)</td></tr><tr><td>inLanguage (0.22)</td><td>hasPart (0.25)</td><td>award (0.02)</td><td>award (0.02)</td></tr><tr><td>composer (0.08)</td><td>composer (0.06)</td><td>memberOf (0.01)</td><td>memberOf (0.00)</td></tr><tr><td>citation (0.04)</td><td>citation (0.04)</td><td>byArtist (0.00)</td><td>byArtist (0.00)</td></tr></tbody></table>

Results for two picked classes are reported in Table 4, where each predicate is weighted in accordance to its frequency in the set of common triples between the 2-hop subgraphs of top-K neighbors in the embedding space (for K = 3 and K =
K neighbors of an entity in the embedding space align with its top-K neighbors in the graph largely depends on which class this entity belongs to.

To examine this behavior further, we pick random classes and count the predicates in the common triples of the top-K neighbors (in the embedding space) for entities of this class differs between models.

<table><caption>**Table 4.** Top 5 predicates for entities of classes MusicComposition and MusicGroup on YAGO4-19K, along with their importance score.</caption><thead><tr><th rowspan="2"></th><th colspan="2">MusicComposition</th><th colspan="2">MusicGroup</th></tr><tr><th>K=3</th><th>K=10</th><th>K=3</th><th>K=10</th></tr></thead><tbody><tr><td rowspan="5">RDF2Vec</td><td>isPartOf (0.52)</td><td>isPartOf (0.46)</td><td>genre (0.53)</td><td>genre (0.60)</td></tr><tr><td>inLanguage (0.18)</td><td>inLanguage (0.30)</td><td>foundingLocation (0.38)</td><td>foundingLocation (0.32)</td></tr><tr><td>hasPart (0.17)</td><td>hasPart (0.15)</td><td>award (0.10)</td><td>award (0.07)</td></tr><tr><td>composer (0.09)</td><td>composer (0.07)</td><td>memberOf (0.00)</td><td>memberOf (0.01)</td></tr><tr><td>lyricist (0.02)</td><td>lyricist (0.02)</td><td>byArtist (0.00)</td><td>byArtist (0.00)</td></tr><tr><td rowspan="5">TransE</td><td>isPartOf (0.40)</td><td>isPartOf (0.36)</td><td>genre (0.79)</td><td>genre (0.85)</td></tr><tr><td>hasPart (0.28)</td><td>hasPart (0.27)</td><td>foundingLocation (0.17)</td><td>foundingLocation (0.12)</td></tr><tr><td>inLanguage (0.16)</td><td>inLanguage (0.26)</td><td>award (0.04)</td><td>award (0.02)</td></tr><tr><td>composer (0.09)</td><td>composer (0.06)</td><td>memberOf (0.00)</td><td>memberOf (0.00)</td></tr><tr><td>citation (0.05)</td><td>citation (0.04)</td><td>byArtist (0.00)</td><td>knowsLanguage (0.00)</td></tr><tr><td rowspan="5">TransD</td><td>isPartOf (0.40)</td><td>inLanguage (0.37)</td><td>genre (0.84)</td><td>genre (0.87)</td></tr><tr><td>inLanguage (0.24)</td><td>isPartOf (0.31)</td><td>foundingLocation (0.13)</td><td>foundingLocation (0.11)</td></tr><tr><td>hasPart (0.23)</td><td>hasPart (0.23)</td><td>award (0.03)</td><td>award (0.02)</td></tr><tr><td>composer (0.08)</td><td>composer (0.06)</td><td>memberOf (0.00)</td><td>memberOf (0.00)</td></tr><tr><td>citation (0.03)</td><td>citation (0.03)</td><td>byArtist (0.00)</td><td>knowsLanguage (0.00)</td></tr><tr><td rowspan="5">RESCAL</td><td>isPartOf (0.42)</td><td>inLanguage (0.33)</td><td>genre (0.74)</td><td>genre (0.81)</td></tr><tr><td>inLanguage (0.23)</td><td>isPartOf (0.33)</td><td>foundingLocation (0.22)</td><td>foundingLocation (0.16)</td></tr><tr><td>hasPart (0.21)</td><td>hasPart (0.25)</td><td>award (0.03)</td><td>award (0.02)</td></tr><tr><td>composer (0.09)</td><td>composer (0.06)</td><td>memberOf (0.01)</td><td>memberOf (0.00)</td></tr><tr><td>citation (0.04)</td><td>citation (0.02)</td><td>knowsLanguage (0.00)</td><td>knowsLanguage (0.00)</td></tr><tr><td rowspan="5">DistMult</td><td>isPartOf (0.38)</td><td>inLanguage (0.39)</td><td>genre (0.81)</td><td>genre (0.86)</td></tr><tr><td>hasPart (0.26)</td><td>isPartOf (0.29)</td><td>foundingLocation (0.14)</td><td>foundingLocation (0.11)</td></tr><tr><td>inLanguage (0.22)</td><td>hasPart (0.22)</td><td>award (0.04)</td><td>award (0.02)</td></tr><tr><td>composer (0.08)</td><td>composer (0.05)</td><td>memberOf (0.00)</td><td>memberOf (0.00)</td></tr><tr><td>citation (0.05)</td><td>citation (0.03)</td><td>knowsLanguage (0.00)</td><td>knowsLanguage (0.00)</td></tr><tr><td rowspan="5">TuckER</td><td>isPartOf (0.38)</td><td>inLanguage (0.37)</td><td>genre (0.85)</td><td>genre (0.90)</td></tr><tr><td>hasPart (0.25)</td><td>isPartOf (0.30)</td><td>foundingLocation (0.11)</td><td>foundingLocation (0.08)</td></tr><tr><td>inLanguage (0.24)</td><td>hasPart (0.23)</td><td>award (0.04)</td><td>award (0.02)</td></tr><tr><td>composer (0.08)</td><td>composer (0.05)</td><td>memberOf (0.01)</td><td>memberOf (0.00)</td></tr><tr><td>citation (0.04)</td><td>citation (0.03)</td><td>byArtist (0.00)</td><td>byArtist (0.00)</td></tr><tr><td rowspan="5">BoxE</td><td>isPartOf (0.48)</td><td>isPartOf (0.40)</td><td>genre (0.73)</td><td>genre (0.83)</td></tr><tr><td>hasPart (0.29)</td><td>hasPart (0.30)</td><td>foundingLocation (0.23)</td><td>foundingLocation (0.15)</td></tr><tr><td>inLanguage (0.11)</td><td>inLanguage (0.19)</td><td>award (0.04)</td><td>award (0.02)</td></tr><tr><td>composer (0.10)</td><td>composer (0.07)</td><td>byArtist (0.00)</td><td>byArtist (0.00)</td></tr><tr><td>lyricist (0.02)</td><td>citation (0.02)</td><td>memberOf (0.00)</td><td>memberOf (0.00)</td></tr><tr><td rowspan="5">ConvE</td><td>isPartOf (0.39)</td><td>isPartOf (0.33)</td><td>genre (0.83)</td><td>genre (0.86)</td></tr><tr><td>hasPart (0.25)</td><td>inLanguage (0.31)</td><td>foundingLocation (0.14)</td><td>foundingLocation (0.12)</td></tr><tr><td>inLanguage (0.22)</td><td>hasPart (0.25)</td><td>award (0.02)</td><td>award (0.02)</td></tr><tr><td>composer (0.08)</td><td>composer (0.06)</td><td>memberOf (0.01)</td><td>memberOf (0.00)</td></tr><tr><td>citation (0.04)</td><td>citation (0.04)</td><td>byArtist (0.00)</td><td>byArtist (0.00)</td></tr></tbody></table>

Results for two picked classes are reported in Table 4, where each predicate is weighted in accordance to its frequency in the set of common triples between the 2-hop subgraphs of top-K neighbors in the embedding space (for K = 3 and K =
10) and a given entity of class `MusicComposition` and `MusicGroup` (YAGO4-19K). A class-based analysis reveals that predicates are differently ranked, i.e. different KGEMs *may* rely on different sets of predicates. For example, for *K* = 10 and entities of the class `MusicComposition`, either `isPartOf` or `inLanguage` appears as the most frequent predicate depending on the KGEM considered.

We additionally observe that even when the ranking is the same between KGEMs – especially since the predicate distribution can be severely skewed towards one or few predicates – the relative importance of predicates can still differ. For instance, although *genre* is consistently ranked as the most frequent predicate for `MusicGroup` entities, results for RDF2Vec suggest a more balanced predicate distribution and a lesser emphasis put on this single predicate (highlighted in blue in Table 4). It is the only model to put a significant emphasis on *award* (i.e., considers two `MusicGroups` as similar if they have won the same (1-hop) or similar (2-hop) awards). In some cases, a few KGEMs seem to assign a lower relevance to specific predicates. For instance, under BoxE the top-3 and top-10 neighbors of entities that are `MusicCompositions` contain the predicate `inLanguage` at a much lesser frequency in their subgraphs' intersection (highlighted in brown in Table 4), compared to other models. Answering `RQ3`, we observe that the different notions of entity similarity reflected by different KGEMs are comparable, as most of them put a focus on the same set of predicates when determining entity similarity.

# 6 Conclusion and Outlook

This work delved into the intricate relationship between entity similarity in KGs and their respective embeddings in KGEMs, questioning the widespread *KGE entity similarity assumption*. Contrary to this belief, we showed that the choice of KGEM significantly influences the notion of entity similarity encoded in the resulting vector space. This finding has profound implications for a variety of downstream tasks where accurate entity similarity is crucial, e.g. recommender systems and semantic searches. For instance, if the proximity of graph embeddings does not align with our proposed metric for measuring entity similarity, the tacit assumption of downstream systems exploiting embeddings for recommender systems does not hold and caution is needed when deploying such systems.

A common practice is to evaluate KGEMs by their performance in link prediction, then picking the one with the best performance for a downstream task relying on entity similarity. A critical takeaway from our study is that this practice might not yield the best results in terms of capturing true entity similarity. Instead, our results advocate for cautiousness. Moreover, in scenarios where the similarity of specific classes of entities is of paramount importance, a per-class analysis becomes essential. This approach allows for a tailored selection of KGEMs that are more adept at capturing the nuances and semantics of particular classes, thereby ensuring a more accurate and meaningful representation of entity similarity.
10) and a given entity of class `MusicComposition` and `MusicGroup` (YAGO4-19K). A class-based analysis reveals that predicates are differently ranked, i.e. different KGEMs *may* rely on different sets of predicates. For example, for *K* = 10 and entities of the class `MusicComposition`, either `isPartOf` or `inLanguage` appears as the most frequent predicate depending on the KGEM considered.

We additionally observe that even when the ranking is the same between KGEMs – especially since the predicate distribution can be severely skewed towards one or few predicates – the relative importance of predicates can still differ. For instance, although *genre* is consistently ranked as the most frequent predicate for `MusicGroup` entities, results for RDF2Vec suggest a more balanced predicate distribution and a lesser emphasis put on this single predicate (highlighted in blue in Table 4). It is the only model to put a significant emphasis on *award* (i.e., considers two `MusicGroups` as similar if they have won the same (1-hop) or similar (2-hop) awards). In some cases, a few KGEMs seem to assign a lower relevance to specific predicates. For instance, under BoxE the top-3 and top-10 neighbors of entities that are `MusicCompositions` contain the predicate `inLanguage` at a much lesser frequency in their subgraphs' intersection (highlighted in brown in Table 4), compared to other models. Answering `RQ3`, we observe that the different notions of entity similarity reflected by different KGEMs are comparable, as most of them put a focus on the same set of predicates when determining entity similarity.

# 6 Conclusion and Outlook

This work delved into the intricate relationship between entity similarity in KGs and their respective embeddings in KGEMs, questioning the widespread *KGE entity similarity assumption*. Contrary to this belief, we showed that the choice of KGEM significantly influences the notion of entity similarity encoded in the resulting vector space. This finding has profound implications for a variety of downstream tasks where accurate entity similarity is crucial, e.g. recommender systems and semantic searches. For instance, if the proximity of graph embeddings does not align with our proposed metric for measuring entity similarity, the tacit assumption of downstream systems exploiting embeddings for recommender systems does not hold and caution is needed when deploying such systems.

A common practice is to evaluate KGEMs by their performance in link prediction, then picking the one with the best performance for a downstream task relying on entity similarity. A critical takeaway from our study is that this practice might not yield the best results in terms of capturing true entity similarity. Instead, our results advocate for cautiousness. Moreover, in scenarios where the similarity of specific classes of entities is of paramount importance, a per-class analysis becomes essential. This approach allows for a tailored selection of KGEMs that are more adept at capturing the nuances and semantics of particular classes, thereby ensuring a more accurate and meaningful representation of entity similarity.
References

1. Abboud, R., Ceylan, İ.İ., Lukasiewicz, T., Salvatori, T.: Boxe: A box embedding model for knowledge base completion. In: Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual (2020)

2. Alshargi, F., Shekarpour, S., Soru, T., Sheth, A.: Concept2vec: Metrics for evaluating quality of embeddings for ontological concepts. arXiv preprint arXiv:1803.04488 (2018)

3. Auer, S., Bizer, C., Kobilarov, G., Lehmann, J., Cyganiak, R., Ives, Z.G.: Dbpedia: A nucleus for a web of open data. In: The Semantic Web, 6th International Semantic Web Conference, 2nd Asian Semantic Web Conference, ISWC + ASWC. Lecture Notes in Computer Science, vol. 4825, pp. 722-735. Springer (2007)

4. Balazevic, I., Allen, C., Hospedales, T.M.: Tucker: Tensor factorization for knowledge graph completion. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019. pp. 5184-5193. Association for Computational Linguistics (2019). https://doi.org/10.18653/v1/D19-1522

5. Bloehdorn, S., Sure, Y.: Kernel methods for mining instance data in ontologies. In: The Semantic Web, 6th International Semantic Web Conference, 2nd Asian Semantic Web Conference, ISWC 2007 + ASWC 2007, Busan, Korea, November 11-15, 2007. Lecture Notes in Computer Science, vol. 4825, pp. 58-71. Springer (2007). https://doi.org/10.1007/978-3-540-76298-0_5

6. Bordes, A., Usunier, N., García-Durán, A., Weston, J., Yakhnenko, O.: Translating embeddings for modeling multi-relational data. In: Conference on Neural Information Processing Systems (NeurIPS). pp. 2787-2795 (2013)

7. Chen, W., Zhu, H., Han, X., Liu, Z., Sun, M.: Quantifying similarity between relations with fact distribution. In: Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers. pp. 2882-2894. Association for Computational Linguistics (2019). https://doi.org/10.18653/V1/P19-1278

8. Dettmers, T., Minervini, P., Stenetorp, P., Riedel, S.: Convolutional 2d knowledge graph embeddings. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (IAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018. pp. 1811-1818. AAAI Press (2018)

9. Gad-Elrab, M.H., Stepanova, D., Tran, T., Adel, H., Weikum, G.: Excut: Explainable embedding-based clustering over knowledge graphs. In: The Semantic Web - ISWC 2020 - 19th International Semantic Web Conference, Athens, Greece, November 2-6, 2020, Proceedings, Part I. Lecture Notes in Computer Science, vol. 12506, pp. 218-237. Springer (2020). https://doi.org/10.1007/978-3-030-62419-4_13

10. Guo, Q., Zhuang, F., Qin, C., Zhu, H., Xie, X., Xiong, H., He, Q.: A survey on knowledge graph-based recommender systems. IEEE Trans. Knowl. Data Eng. 34(8), 3549-3568 (2022). https://doi.org/10.1109/TKDE.2020.3028705

11. Hubert, N., Monnin, P., Brun, A., Monticolo, D.: Sem@k: Is my knowledge graph embedding model semantic-aware? Semantic Web 14, 1-37 (12 2023). https://doi.org/10.3233/SW-233508
References

1. Abboud, R., Ceylan, İ.İ., Lukasiewicz, T., Salvatori, T.: Boxe: A box embedding model for knowledge base completion. In: Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual (2020)

2. Alshargi, F., Shekarpour, S., Soru, T., Sheth, A.: Concept2vec: Metrics for evaluating quality of embeddings for ontological concepts. arXiv preprint arXiv:1803.04488 (2018)

3. Auer, S., Bizer, C., Kobilarov, G., Lehmann, J., Cyganiak, R., Ives, Z.G.: Dbpedia: A nucleus for a web of open data. In: The Semantic Web, 6th International Semantic Web Conference, 2nd Asian Semantic Web Conference, ISWC + ASWC. Lecture Notes in Computer Science, vol. 4825, pp. 722–735. Springer (2007)

4. Balazevic, I., Allen, C., Hospedales, T.M.: Tucker: Tensor factorization for knowledge graph completion. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019. pp. 5184–5193. Association for Computational Linguistics (2019). https://doi.org/10.18653/v1/D19-1522

5. Bloehdorn, S., Sure, Y.: Kernel methods for mining instance data in ontologies. In: The Semantic Web, 6th International Semantic Web Conference, 2nd Asian Semantic Web Conference, ISWC 2007 + ASWC 2007, Busan, Korea, November 11-15, 2007. Lecture Notes in Computer Science, vol. 4825, pp. 58–71. Springer (2007). https://doi.org/10.1007/978-3-540-76298-0_5

6. Bordes, A., Usunier, N., García-Durán, A., Weston, J., Yakhnenko, O.: Translating embeddings for modeling multi-relational data. In: Conference on Neural Information Processing Systems (NeurIPS). pp. 2787–2795 (2013)

7. Chen, W., Zhu, H., Han, X., Liu, Z., Sun, M.: Quantifying similarity between relations with fact distribution. In: Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers. pp. 2882–2894. Association for Computational Linguistics (2019). https://doi.org/10.18653/V1/P19-1278

8. Dettmers, T., Minervini, P., Stenetorp, P., Riedel, S.: Convolutional 2d knowledge graph embeddings. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (IAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018. pp. 1811–1818. AAAI Press (2018)

9. Gad-Elrab, M.H., Stepanova, D., Tran, T., Adel, H., Weikum, G.: Excut: Explainable embedding-based clustering over knowledge graphs. In: The Semantic Web - ISWC 2020 - 19th International Semantic Web Conference, Athens, Greece, November 2-6, 2020, Proceedings, Part I. Lecture Notes in Computer Science, vol. 12506, pp. 218–237. Springer (2020). https://doi.org/10.1007/978-3-030-62419-4_13

10. Guo, Q., Zhuang, F., Qin, C., Zhu, H., Xie, X., Xiong, H., He, Q.: A survey on knowledge graph-based recommender systems. IEEE Trans. Knowl. Data Eng. 34(8), 3549–3568 (2022). https://doi.org/10.1109/TKDE.2020.3028705

11. Hubert, N., Monnin, P., Brun, A., Monticolo, D.: Sem@k: Is my knowledge graph embedding model semantic-aware? Semantic Web 14, 1–37 (12 2023). https://doi.org/10.3233/SW-233508
12. Hubert, N., Paulheim, H., Monnin, P., Brun, A., Monticolo, D.: Schema first! learn versatile knowledge graph embeddings by capturing semantics with maschine. In: K-CAP '23: Knowledge Capture Conference, Pensacola, Florida, USA, December 5-7, 2023. ACM (2023). https://doi.org/10.48550/ARXIV.2306.03659

13. Ilievski, F., Shenoy, K., Chalupsky, H., Klein, N., Szekely, P.: A study of concept similarity in wikidata. Semantic Web Journal (2023)

14. Jain, N., Kalo, J.C., Balke, W.T., Krestel, R.: Do embeddings actually capture knowledge graph semantics? In: The Semantic Web: 18th International Conference, ESWC 2021, Virtual Event, June 6–10, 2021, Proceedings 18. pp. 143–159. Springer (2021)

15. Ji, G., He, S., Xu, L., Liu, K., Zhao, J.: Knowledge graph embedding via dynamic mapping matrix. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 1: Long Papers. pp. 687–696. The Association for Computer Linguistics (2015). https://doi.org/10.3115/v1/p15-1067

16. Ji, S., Pan, S., Cambria, E., Marttinen, P., Yu, P.S.: A survey on knowledge graphs: Representation, acquisition, and applications. IEEE Trans. Neural Networks Learn. Syst. 33(2), 494–514 (2022). https://doi.org/10.1109/TNNLS.2021.3070843

17. Kalo, J., Ehler, P., Balke, W.: Knowledge graph consolidation by unifying synonymous relationships. In: The Semantic Web - ISWC 2019 - 18th International Semantic Web Conference, Auckland, New Zealand, October 26-30, 2019, Proceedings, Part I. Lecture Notes in Computer Science, vol. 11778, pp. 276–292. Springer (2019). https://doi.org/10.1007/978-3-030-30793-6_16

18. Kendall, M.G.: A new measure of rank correlation. Biometrika 30(1/2), 81–93 (1938)

19. Liu, W., Zhou, P., Zhao, Z., Wang, Z., Ju, Q., Deng, H., Wang, P.: K-BERT: enabling language representation with knowledge graph. In: The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020. pp. 2901–2908. AAAI Press (2020). https://doi.org/10.1609/AAAI.V34I03.5681

20. Ma, Y., Cambria, E., Gao, S.: Label embedding for zero-shot fine-grained named entity typing. In: Calzolari, N., Matsumoto, Y., Prasad, R. (eds.) COLING 2016, 26th International Conference on Computational Linguistics, Proceedings of the Conference: Technical Papers, December 11-16, 2016, Osaka, Japan. pp. 171–180. ACL (2016), https://aclanthology.org/C16-1017/

21. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. In: 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings (2013)

22. Nickel, M., Tresp, V., Kriegel, H.: A three-way model for collective learning on multi-relational data. In: Proc. of the 28th International Conference on Machine Learning, ICML. pp. 809–816 (2011)

23. Portisch, J., Heist, N., Paulheim, H.: Knowledge graph embedding for data mining vs. knowledge graph embedding for link prediction–two sides of the same coin? Semantic Web 13(3), 399–422 (2022)

24. Ristoski, P., Paulheim, H.: Rdf2vec: RDF graph embeddings for data mining. In: The Semantic Web - ISWC 2016 - 15th International Semantic Web Conference,
12. Hubert, N., Paulheim, H., Monnin, P., Brun, A., Monticolo, D.: Schema first! learn versatile knowledge graph embeddings by capturing semantics with maschine. In: K-CAP '23: Knowledge Capture Conference, Pensacola, Florida, USA, December 5-7, 2023. ACM (2023). https://doi.org/10.48550/ARXIV.2306.03659

13. Ilievski, F., Shenoy, K., Chalupsky, H., Klein, N., Szekely, P.: A study of concept similarity in wikidata. Semantic Web Journal (2023)

14. Jain, N., Kalo, J.C., Balke, W.T., Krestel, R.: Do embeddings actually capture knowledge graph semantics? In: The Semantic Web: 18th International Conference, ESWC 2021, Virtual Event, June 6–10, 2021, Proceedings 18. pp. 143–159. Springer (2021)

15. Ji, G., He, S., Xu, L., Liu, K., Zhao, J.: Knowledge graph embedding via dynamic mapping matrix. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 1: Long Papers. pp. 687–696. The Association for Computer Linguistics (2015). https://doi.org/10.3115/v1/p15-1067

16. Ji, S., Pan, S., Cambria, E., Marttinen, P., Yu, P.S.: A survey on knowledge graphs: Representation, acquisition, and applications. IEEE Trans. Neural Networks Learn. Syst. 33(2), 494–514 (2022). https://doi.org/10.1109/TNNLS.2021.3070843

17. Kalo, J., Ehler, P., Balke, W.: Knowledge graph consolidation by unifying synonymous relationships. In: The Semantic Web - ISWC 2019 - 18th International Semantic Web Conference, Auckland, New Zealand, October 26-30, 2019, Proceedings, Part I. Lecture Notes in Computer Science, vol. 11778, pp. 276–292. Springer (2019). https://doi.org/10.1007/978-3-030-30793-6\_16

18. Kendall, M.G.: A new measure of rank correlation. Biometrika 30(1/2), 81–93 (1938)

19. Liu, W., Zhou, P., Zhao, Z., Wang, Z., Ju, Q., Deng, H., Wang, P.: K-BERT: enabling language representation with knowledge graph. In: The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020. pp. 2901–2908. AAAI Press (2020). https://doi.org/10.1609/AAAI.V34I03.5681

20. Ma, Y., Cambria, E., Gao, S.: Label embedding for zero-shot fine-grained named entity typing. In: Calzolari, N., Matsumoto, Y., Prasad, R. (eds.) COLING 2016, 26th International Conference on Computational Linguistics, Proceedings of the Conference: Technical Papers, December 11-16, 2016, Osaka, Japan. pp. 171–180. ACL (2016), https://aclanthology.org/C16-1017/

21. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. In: 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings (2013)

22. Nickel, M., Tresp, V., Kriegel, H.: A three-way model for collective learning on multi-relational data. In: Proc. of the 28th International Conference on Machine Learning, ICML. pp. 809–816 (2011)

23. Portisch, J., Heist, N., Paulheim, H.: Knowledge graph embedding for data mining vs. knowledge graph embedding for link prediction–two sides of the same coin? Semantic Web 13(3), 399–422 (2022)

24. Ristoski, P., Paulheim, H.: Rdf2vec: RDF graph embeddings for data mining. In: The Semantic Web - ISWC 2016 - 15th International Semantic Web Conference,
Kobe, Japan, October 17-21, 2016, Proceedings, Part I. Lecture Notes in Computer Science, vol. 9981, pp. 498-514 (2016). https://doi.org/10.1007/978-3-319-46523-4_30

25. Rossi, A., Barbosa, D., Firmani, D., Matinata, A., Merialdo, P.: Knowledge graph embedding for link prediction: A comparative analysis. ACM Transactions on Knowledge Discovery from Data **15**(2), 14:1-14:49 (2021)

26. Rossi, A., Matinata, A.: Knowledge graph embeddings: Are relation-learning models learning relations? In: Proceedings of the Workshops of the EDBT/ICDT 2020 Joint Conference, Copenhagen, Denmark, March 30, 2020. CEUR Workshop Proceedings, vol. 2578. CEUR-WS.org (2020)

27. Safavi, T., Koutra, D.: Codex: A comprehensive knowledge graph completion benchmark. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020. pp. 8328-8350. Association for Computational Linguistics (2020). https://doi.org/10.18653/v1/2020.emnlp-main.669

28. Sanfeliu, A., Fu, K.: A distance measure between attributed relational graphs for pattern recognition. IEEE Trans. Syst. Man Cybern. **13**(3), 353-362 (1983). https://doi.org/10.1109/TSMC.1983.6313167

29. Shi, B., Weninger, T.: Open-world knowledge graph completion. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018. pp. 1957-1964. AAAI Press (2018). https://doi.org/10.1609/AAAI.V32I1.11535

30. Sosa, D.N., Derry, A., Guo, M.G., Wei, E., Brinton, C., Altman, R.B.: A literature-based knowledge graph embedding method for identifying drug repurposing opportunities in rare diseases. In: Pacific Symposium on Biocomputing 2020, Fairmont Orchid, Hawaii, USA, January 3-7, 2020. pp. 463-474 (2020)

31. Suchanek, F.M., Kasneci, G., Weikum, G.: Yago: a core of semantic knowledge. In: Proc. of the 16th International Conference on World Wide Web, WWW. pp. 697-706. ACM (2007)

32. Sun, Y., Wang, S., Li, Y., Feng, S., Tian, H., Wu, H., Wang, H.: ERNIE 2.0: A continual pre-training framework for language understanding. In: The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020. pp. 8968-8975. AAAI Press (2020). https://doi.org/10.1609/AAAI.V34I05.6428

33. Sun, Z., Zhang, Q., Hu, W., Wang, C., Chen, M., Akrami, F., Li, C.: A benchmarking study of embedding-based entity alignment for knowledge graphs. Proc. VLDB Endow. **13**(11), 2326-2340 (2020)

34. Sun, Z., Zhang, Q., Hu, W., Wang, C., Chen, M., Akrami, F., Li, C.: A benchmarking study of embedding-based entity alignment for knowledge graphs. Proc. VLDB Endow. **13**(11), 2326-2340 (2020)

35. Sun, Z., Deng, Z., Nie, J., Tang, J.: Rotate: Knowledge graph embedding by rotational rotation in complex space. In: 7th International Conference on Learning Representations, ICLR (2019)

36. Toutanova, K., Chen, D.: Observed versus latent features for knowledge base and text inference. In: Proc. of the 3rd Workshop on Continuous Vector Space Models and their Compositionality. pp. 57-66. Association for Computational Linguistics (2015)
Kobe, Japan, October 17-21, 2016, Proceedings, Part I. Lecture Notes in Computer Science, vol. 9981, pp. 498-514 (2016). https://doi.org/10.1007/978-3-319-46523-4_30

25. Rossi, A., Barbosa, D., Firmani, D., Matinata, A., Merialdo, P.: Knowledge graph embedding for link prediction: A comparative analysis. ACM Transactions on Knowledge Discovery from Data 15(2), 14:1-14:49 (2021)

26. Rossi, A., Matinata, A.: Knowledge graph embeddings: Are relation-learning models learning relations? In: Proceedings of the Workshops of the EDBT/ICDT 2020 Joint Conference, Copenhagen, Denmark, March 30, 2020. CEUR Workshop Proceedings, vol. 2578. CEUR-WS.org (2020)

27. Safavi, T., Koutra, D.: Codex: A comprehensive knowledge graph completion benchmark. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020. pp. 8328-8350. Association for Computational Linguistics (2020). https://doi.org/10.18653/v1/2020.emnlp-main.669

28. Sanfeliu, A., Fu, K.: A distance measure between attributed relational graphs for pattern recognition. IEEE Trans. Syst. Man Cybern. 13(3), 353-362 (1983). https://doi.org/10.1109/TSMC.1983.6313167

29. Shi, B., Weninger, T.: Open-world knowledge graph completion. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018. pp. 1957-1964. AAAI Press (2018). https://doi.org/10.1609/AAAI.V32I1.11535

30. Sosa, D.N., Derry, A., Guo, M.G., Wei, E., Brinton, C., Altman, R.B.: A literature-based knowledge graph embedding method for identifying drug repurposing opportunities in rare diseases. In: Pacific Symposium on Biocomputing 2020, Fairmont Orchid, Hawaii, USA, January 3-7, 2020. pp. 463-474 (2020)

31. Suchanek, F.M., Kasneci, G., Weikum, G.: Yago: a core of semantic knowledge. In: Proc. of the 16th International Conference on World Wide Web, WWW. pp. 697-706. ACM (2007)

32. Sun, Y., Wang, S., Li, Y., Feng, S., Tian, H., Wu, H., Wang, H.: ERNIE 2.0: A continual pre-training framework for language understanding. In: The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020. pp. 8968-8975. AAAI Press (2020). https://doi.org/10.1609/AAAI.V34I05.6428

33. Sun, Z., Zhang, Q., Hu, W., Wang, C., Chen, M., Akrami, F., Li, C.: A benchmarking study of embedding-based entity alignment for knowledge graphs. Proc. VLDB Endow. 13(11), 2326-2340 (2020)

34. Sun, Z., Zhang, Q., Hu, W., Wang, C., Chen, M., Akrami, F., Li, C.: A benchmarking study of embedding-based entity alignment for knowledge graphs. Proc. VLDB Endow. 13(11), 2326-2340 (2020)

35. Sun, Z., Deng, Z., Nie, J., Tang, J.: Rotate: Knowledge graph embedding by relative rotation in complex space. In: 7th International Conference on Learning Representations, ICLR (2019)

36. Toutanova, K., Chen, D.: Observed versus latent features for knowledge base and text inference. In: Proc. of the 3rd Workshop on Continuous Vector Space Models and their Compositionality. pp. 57-66. Association for Computational Linguistics (2015)
37. Trouillon, T., Welbl, J., Riedel, S., Gaussier, É., Bouchard, G.: Complex embeddings for simple link prediction. In: Proceedings of the 33rd International Conference on Machine Learning, ICML. vol. 48, pp. 2071–2080 (2016)

38. Wang, M., Qiu, L., Wang, X.: A survey on knowledge graph embeddings for link prediction. Symmetry **13**(3), 485 (2021)

39. Webber, W., Moffat, A., Zobel, J.: A similarity measure for indefinite rankings. ACM Trans. Inf. Syst. **28**(4), 20:1–20:38 (2010).
https://doi.org/10.1145/1852102.1852106

40. Yang, B., Yih, W., He, X., Gao, J., Deng, L.: Embedding entities and relations for learning and inference in knowledge bases. In: 3rd International Conference on Learning Representations, ICLR (2015)
37. Trouillon, T., Welbl, J., Riedel, S., Gaussier, É., Bouchard, G.: Complex embeddings for simple link prediction. In: Proceedings of the 33rd International Conference on Machine Learning, ICML. vol. 48, pp. 2071–2080 (2016)

38. Wang, M., Qiu, L., Wang, X.: A survey on knowledge graph embeddings for link prediction. Symmetry **13**(3), 485 (2021)

39. Webber, W., Moffat, A., Zobel, J.: A similarity measure for indefinite rankings. ACM Trans. Inf. Syst. **28**(4), 20:1–20:38 (2010).
https://doi.org/10.1145/1852102.1852106

40. Yang, B., Yih, W., He, X., Gao, J., Deng, L.: Embedding entities and relations for learning and inference in knowledge bases. In: 3rd International Conference on Learning Representations, ICLR (2015)