dataesr/scientific-paragraphs-categorization huggingface dataset:
• D_Y_15K_V1 and D_Y_15K_V2 are the sparse and the dense datasets, respectively, that were constructed from DBpedia and YAGO3. They contain the same entity types with D_W_15K but unlike D_W_15K, they have both high number of entities that have long textual descriptions and high entity name similarity. Five additional datasets, from  Efthymiou et al. (2015) ,  Obraczka et al. (2021) , were employed, that originate from the following eight KGs: the BTC2012 version of DBpedia, 4 BBCmusic (originating from Kasabi 5 ), IMDb, 6 TVDB, 7 TMDb, 8 and two restaurant KGs. 9 None of these datasets originally cover the assumptions of embedding-based entity alignment methods, thus we pre-process them, as described in Sect. 3. 4. The five extra datasets, built from those KGs, are the following: • BBC-DB  (Efthymiou et al.
2015)   average attributes per entity and the highest number of entities that have textual descriptions among other datasets. In addition, this dataset is better suited for the methods that perform schema alignment, since these KGs follow similar predicate naming conventions. • imdb-tmdb, imdb-tvdb, and tmdb-tvdb  (Obraczka et al. 2021)  were constructed from IMDb, TMDb, and TVDB that contain descriptions about people, movies, tv series, episodes and companies. Those datasets have a low number of entities, which makes them interesting for examining the performance of supervised vs unsupervised methods with a small training dataset. In addition, they are much sparser than the previous datasets, with low average attributes per entity, while they exhibit high homogeneity in terms of literals and predicate names.
tmdb-tvdb features higher homogeneity in terms of literals and lower similarity of long-text descriptions, compared to imdb-tvdb and tmdb-tvdb. • Restaurants 9 contains descriptions of real restaurants and their addresses from two different KGs. It is the smallest dataset in terms of number of entities, relation types and attribute types, but its main challenge is that it does not cover the assumptions of all the embedding-based entity alignment methods. Specifically, not all entities of KG1 are matched with all the entities KG2 and vice versa. We chose this dataset to examine whether the evaluated methods can be generalized beyond datasets that satisfy the 1-to-1 assumption.
Statistics and meta-features In order to quantify useful information about the knowledge graphs, we calculated some basic dataset statistics regrading the type and the number of relations and attributes that are presented in Table  3  and some additional dataset statistics that are shown in Table  4 , categorized in the three following categories: seed alignment size, density and heterogeneity that were used for constructing the meta-features of Table  5. Seed alignment size refers to the number of entity pairs in the seed alignment. Density consists of the average relations per entity and the average attributes per entity, which indicate the density of the knowledge graphs on structural and attribute level, respectively. In addition, it consists of the proportion of sole and hyper relation types of the two knowledge graphs, where a sole relation type is a relation type that does not co-occur with another relation type in any entity pair  (Zhang et al.
2019). Specifically, we report the average of MAX similarities of each predicate (relation or attribute). The meta-features of Table  5  were constructed by applying the aggregation functions of Table  6  on the statistics of Table  4 , in order to construct features for each individual dataset instead of each individual KG of the dataset and use them in Sect. 4. 3. More precisely, for Avg_Rels_per_Entity, Avg_Attrs_per_Entity, Sole_Rels, Hyper_Rels and #Ents_Descr (based on KDCoE that proposes possibly aligned entities of the entities of the first KG) were used aggregation functions, while #Entity_Pairs, Descr_Sim, Ents_Name_Sim, Lit_Sim and Pred_Name_Sim were used as meta-features without the application of any aggregation function.
(2020) , there are different behaviors for this phenomenon that are categorized as optimistic, pessimistic, non-deterministic and realistic. In optimistic behavior, it is assumed that the true entity is ranked first among other entities with the same score, in pessimistic, it is assumed that it is ranked last, and realistic is the mean of optimistic and pessimistic. In our case, all of the evaluated methods are non-deterministic, which means that handling ties depends on the inner working of the sorting algorithm that the methods use to sort each similarity list. In the context of ensuring a reliable and unbiased experimental process, we conducted the experiments with 5-fold cross-validation  (Sun et al. 2020) , where each fold is split into train, validation and test 10. Finally, the performance of each method, per dataset, for a specific metric is calculated by the average scores that the method achieves in the 5 folds for this metric.
Pre-processing pipelines The embedding-based entity alignment methods are working under some strict assumptions that make them inapplicable to the five new datasets. However, for a reliable experimental evaluation of the methods under the same conditions and data characteristics, we pre-process those datasets in order to conform to the assumptions described in Sect. 2. 1-to-1 and structural-information assumptions Firstly, we processed the seed alignment, in order to satisfy the 1-to-1 mapping assumption, as follows. If there are two entity pairs (e i , e j ) and (e i , e � j ) in the seed alignment, we need to keep one of them. In the dilemma of choosing which entity pair to remove, we examined two factors. The first factor is to keep the entity pair in which the true entity, e j or e ′ j in this example, is involved in at least one relation edge (one of the other assumptions of these methods).
11  For example, given the attributes :name and :hasName, the second one is replaced by :name. Sampling. For sampling real-world KGs, the authors of OpenEA proposed an iterative degree-based sampling (IDS) algorithm which simultaneously removes entities of the two KGs using seed alignment. The objective of this algorithm is to generate a dataset of a specific size from real-world KGs, such that the difference of (34) MRR = 1 | I | ∑ r∈I 1 r , 1 3 Knowledge graph embedding methods for entity alignment: an… the entity degree distribution, between the original KG and the sampled KG, does not exceed a certain value. In order to determine which entities to remove without greatly affecting the entity degree distribution, they delete entities with low Pag-eRank scores, while assessing the divergence of the two degree distributions of the KGs using the Jensen-Shannon (JS)  (Lin 1991)  measure.
Implementation details All experiments were performed in a server with 16 cores (AMD EPYC 7232P @ 3. 1 GHz), 64 GB RAM, one RTX-4090 GPU (24 GB) and Ubuntu 18. 04. 5 LTS. We used the code of OpenEA 12 (2020-07-28) for the experiments of methods MTransE, MTransE + RotatE, KDCoE, MultiKE and RDGCN. For AttrE, we noticed that the implementation of OpenEA utilizes the seed alignment for training, which contradicts the unsupervised learning nature of AttrE  (Trisedya et al. 2019). For this reason, we used the code provided by the authors of AttrE 11 (2019-01-11). Finally, for RREA (basic and semi), BERT_INT and PARIS, we used the code provided by the authors. 13,14,15  For conducting a fair experimental process, we used the same hyperparameter values as described in the works of OpenEA, AttrE, RREA and BERT_INT, and are shown in Table  7.
Dim is the dimensions of the embeddings (length of the vector representations). The higher dimensions an embedding has, the more features (expressiveness) it has. Batch size is the number of relation edges that we used in each iteration during training. All the examined methods adopt a batch size of 5,000, except AttrE which adopts 100 and BERT_INT 128. Max epochs refers to the times the learning algorithm will work in the entire dataset. In our experiments, there are two ways to define the number of epochs. The first one is to give a fixed number (50, 205, 1,200 or 6,000 epochs), as AttrE, BERT_INT, RREA(basic) and RREA(semi) do. The second one, early stopping, is to terminate the training process when Hits@1 starts dropping, based on the validation set, checked every 10 epochs. The latter ensures that the model does not overfit.
Analysis of experimental results In this section, we report and analyze the results of a series of experiments we conducted to answer the open questions Q1-Q4 posed in Sect. 1, regarding embeddingbased entity alignment methods. More precisely, to shed light regarding Q1 (characteristics of methods) and Q2 (families of methods), we analyze the performance of the methods described in Sect. 2 across all datasets (Sect. 4. 1). To answer Q3 (effectiveness-efficiency trade-off), we rank the methods in a statistically significant way, while we analyze their corresponding execution time over all datasets of our testbed, and compare the time they require to reach a relative effectiveness threshold (Sect. 4. 2). Finally, in Sect. 4. 3, we address Q4 (characteristics of datasets), by conducting a meta-level analysis to identify correlations between the methods and the various meta-features extracted from the KGs of our testbed.
Effectiveness of EA methods using different metrics To answer Q1 and Q2, we experimentally compare five relation-based (MTransE, MTransE+RotatE, RDGCN, RREA(basic) and RREA(semi)) and three attribute-based methods (KDCoE, MultiKE and AttrE), using the protocol and metrics described in Sect. 3. 3. Table  8  reports the different evaluation metrics per method over all datasets of our testbed.
We set k=1000, which is the largest k value for which we did not run intomemory issues. space. In case that the embedding space has limited dimensions and the KGs are dense, dissimilar entities end up close in the embedding space. The lack of negative sampling (that distances dissimilar entities in the embedding space) is the reason why MTransE is not improved in dense datasets. In the new datasets (BBC-DB, imdb-tmdb, imdb-tvdb and tmdb-tvdb) that are all sparse, fewer entity pairs in seed alignment and fewer relation edges are used for training, resulting in worse performance, too, for this method (when comparing BBC-DB to tmdb-tvdb). For example, Hits@1 for BBC-DB with 61,039 relation edges is 24. 98, while Hits@1 for imdb-tmdb with 9,604 relation edges is 15. On the contrary, MR is improved as can been seen in Table  8.
This is attributed to the dependency of MR on the size of the similarity lists (see the evaluation protocol in Sect. 3. 3. 1). Specifically, as the entity pairs of seed alignment are reduced, the size of similarity lists is reduced too. The smaller the similarity lists, the lower the ranks for true entity pairs in the test set, resulting to an improved MR metric. MTransE+RotatE is a variation of MTransE that exploits RotatE as embedding module, as well as negative sampling, and was originally proposed for better handling symmetrical relations in KGs  (Sun et al. 2020). Since such relations are not encountered in datasets imdb_tmdb, imdb_tvdb, and tmdb_tvdb (see Table  3 ), we investigate only the effect of negative sampling on the performance of the method. MTransE+RotatE exhibits a similar performance to MTransE in the sparse versions of D_W_15K_V1 and D_Y_15K_V1 (e.
g. , in dataset D_W_15K_V1, MTransE Hits@10 is 72. 78 and MTransE+RotatE Hits@10 is 72. 94). However, the negative sampling used by MTransE+RotatE significantly improves its performance in dense datasets compared to MTransE. For example, for D_Y_15K_V2, the MR of MTransE is 79. 47 and that of MTransE+RotatE is 3. 46, and for D_W_15K_V2, MTransE Hits@1 is 26. 27 and MTransE+RotatE Hits@1 is 49. 22. This means that MTransE+RotatE represents similar entities closer in the embedding space and dissimilar entities farther, exploiting the rich semantic information that dense KGs offer. In the new datasets, MTransE+RotatE is outperformed by MTransE (e. g. , for tmdb-tvdb Hits@1 of MTransE is 10. 63, while Hits@1 of MTransE+RotatE drops to 5. 60). This is due to the small size of the new datasets in terms of seed entity pairs (e.
g. , 982 in tmdb-tvdb) and relation edges (e. g. , 3,396 for KG 1 vs only 938 for KG 2 in tmdb-tvdb) that, in conjunction with the high dimensionality of the embed- ding space, place similar entities far away. Negative sampling seems to amplify this behavior. RDGCN exploits multi-hop entity neighborhoods, as well as a higher number of negative samples and dimensions in the embedding space than the other methods (see Table  7 ). As shown in Table  8 , RDGCN has the highest Hits@1 and Hits@10, compared to MTransE and MTransE+RotatE in datasets D_W_15K and D_Y_15K, both for sparse (V1) and dense (V2) versions (e. g. , for D_W_15K_ V2 and D_Y_15K_V1, RDGCN Hits@1 is respectively 63. 71 and 94. 10, while MTransE Hits@1 is 26. 27 and 45. 54 and MTransE+RotatE Hits@1 is 49. 22 and 46. 25). This behavior can be attributed to the fact that RDGCN exploits an extended neighborhood of entities (multi-hop) for constructing the embeddings.
Of course, this richer semantic context incurs the risk of incorporating noisy information into 1 3 Knowledge graph embedding methods for entity alignment: an… the embeddings, due to potentially irrelevant or less important distant neighbors for aligning entities. For these reasons, RDGCN requires both high-dimensional embeddings to capture more features, as well as increased negative samples per positive edge that place irrelevant neighbors far away in the embedding space. It is worth mentioning that RDGCN outperforms the other methods in all OpenEA datasets, demonstrating the added value of multi-hop neighbors in embeddings. On the other hand, RDGCN is outperformed by MTransE+RotatE in datasets BBC-DB, imdbtmdb, imdb-tvdb and tmdb-tvdb (e. g. , in BBC-DB Hits@1 drops from 18. 41 to 6. 38).
This behavior stems from the same problem that negative sampling causes to MTransE+RotatE in these datasets. As RDGCN uses more negative samples and higher dimensions in the embedding space than MTransE+RotatE, it aggravates the problem. RREA(basic), like RDGCN, exploits multi-hop entity neighborhoods, having the same issue about irrelevant or less important neighbors. Contrary to RDGCN, that needs higher number of negative samples and dimensions to solve the aforementioned issue, RREA(basic) utilizes an attention mechanism that assigns high weights to relevant and important neighbors, outperforming RDGCN, but also MTransE and MTransE+RotatE in all datasets, regardless of the metric (e. g. , in D_W_15K_V2, RREA(basic)'s Hits@1 is 87. 86 and for MTransE it is 26. 27, in D_W_15K_V1, RREA(basic)'s Hits@10 is 88.
49, while for MTransE+RotatE it is 56. 98, and in D_Y_15K_V2, RREA(basic)'s MR is 1. 32, while the corresponding MR for RDGCN is 11. 50). As we can see from D_Y_15K_V1, RREA(basic) fails to differentiate the importance of relevant and irrelevant neighbors, and RDGCN achieves higher performance in all metrics (e. g. , Hits@1 for RDGCN is 94. 10, while for RREA(basic) it is 76. 19). The fact that RREA(basic) solves the issue of noisy information and irrelevant neighbors by utilizing the attention mechanism, without requiring higher number of negative samples, leads to outperforming MTransE, MTransE+RotatE and RDGCN in all new datasets (e. g. , RREA(basic)'s Hits@1 for imdb_tmdb is 27. 06, while for MTransE it is 15, RREA(basic)'s Hits@10 for bbc_db is 65. 28, while for MTransE+RotatE it is 41.
66 and RREA(basic)'s MR for imdb_tvdb is 25. 86, while for RDGCN it is 377. 50). RREA(semi) runs in 5 iterations, where the first iteration is RREA(basic), which means that the 4 extra iterations, increase the performance of RREA(semi) compared to RREA(basic) and consequently to MTransE, MTransE+RotatE and RDGCN in all datasets, in Hits@1, Hits@10 and MRR (e. g. , RREA(semi)'s Hits@1 for D_Y_15K_V2 is 97. 53, while for RREA(basic) it is 96. 56, RREA(semi)'s Hits@10 for imdb_tmdb is 73. 65, while for RDGCN it is 0. 73, and RREA(semi)'s MRR for imdb_tvdb is 0. 29, while for MTransE it is 0. 23). RREA(semi), like RREA(basic) fails to distinguish relevant and irrelevant neighbors in D_Y_15K_ V1, where RDGCN achieves better performance in all metrics (e. g. , Hits@1 for RDGCN is 94. 10, while for RREA(semi) it is 81.
79). The fact that RREA(semi) enriches the seed alignment with new entity pairs leads to increasing the candidatespace (denominator of MR) and also the probability of wrong alignment (compared to RREA(basic)), making RREA(semi) exhibit worse MR in all datasets (e. g. , in D_Y_15K_V2, RREA(semi)'s MR is 1. 39, while for RREA(basic) it is 1. 32).
Attribute-based EA methods KDCoE in addition to one-hop neighborhood and negative sampling, exploits the textual descriptions of entities, when available. Negative sampling also improves the performance of KDCoE in dense versions of the datasets, compared to their sparse versions (e. g. , in dataset D_W_15K_V1 Hits@10 is 46. 29, while in D_W_15K_V2 Hits@10 is 64. 38, and in dataset D_Y_15K_V1 MR is 143. 78, while in D_Y_15K_ V2 MR is 2. 28). In dense entity graphs, negative sampling helps KDCoE to increase the distance of dissimilar entities in an embedding space with limited dimensions. The use of textual descriptions as an additional source of similarity evidence favors KDCoE to outperform the other methods in datasets that are rich in terms of textual descriptions. As shown in Table  8 , this is the case of BBC_DB, which features the highest number of entities with textual descriptions (2,430), compared to the size of the seed alignment (9,396), and where KDCoE exhibits better performance compared to MTransE, MTransE+RotatE and RDGCN.
The higher ratio of entities with similar textual descriptions in a dataset boosts the performance of KDCoE. It is worth mentioning that although KDCoE outperforms the aforementioned methods in BBC_DB, both versions of RREA exhibit higher performance in all metrics, due to the attention mechanism that both utilize and the rule that RREA(semi) uses. In case of lower ratio of entities with similar textual descriptions in a dataset, KDCoE's performance drops as the training set is flooded with falsely similar entity pairs in terms of textual descriptions. This behavior is observed in datasets imdb_tmdb, imdb_tvdb, tmdb_tvdb, where textual descriptions of entities exhibit the minimum similarity among all datasets (e. g. , for imdb-tmdb descriptions similarity is 0. 35). Statistically meaningful correlations of the method with the specific data characteristics (#Ents_Descr and Descr_Sim) are reported in Sect.
4. 3. Note that the disfavored effect of negative sampling in these datasets still persists (as in MTransE+RotatE and RDGCN), but with an attenuated strength, due the smaller number of negative samples this method uses. MultiKE exploits multiple sources of similarity evidence of entities (i. e. , from attribute names, literals and relations) along with a high number of negative samples. In this respect, MultiKE embeddings are less sensitive to the number of entity relations than in the previous methods. For this reason, MultiKE, in contrary to KDCoE, is not improving in all dense versions of the datasets, even if it uses negative sampling (see Table  8 ). Specifically, in D_W_15K, when comparing the dense to the sparse version, Hits@1 increases from 40. 49 to 49. 54 as both the number (average attributes per entity is 3 for sparse and 5.
58 for dense) and the similarity score of literals (i. e. , 81. 24 for sparse and 83. 79 for dense) increase. In contrary, from the sparse D_Y_15K_V1 to the dense D_Y_15K_V2, there is a small decrease ( ≈ 8% ) in both entity names similarity (from 0. 99 to 0. 91), causing Hits@1 to drop from 90. 18 to 85. 64. Finally, in datasets imdb_tmdb, imdb_tvdb and tmdb_tvdb, MultiKE has a lower Hits@1 than KDCoE (i. e. , for imdb_tmdb Hits@1 of KDCoE is 20. 02, while for MultiKE it is 15. 28), due to the increased number of negative samples. The correlation of MultiKE with the similarity of entity names will be further investigated in Sect. 4. 3. 3.
3 Knowledge graph embedding methods for entity alignment: an… AttrE is an unsupervised method that consumes fewer negative samples (like KDCoE) and exploits the similarity of literal values. On the other hand, it requires the alignment of predicates in KGs as a pre-processing step. The highest the number of aligned predicates, the better the performance of the method. In datasets without aligned predicates, AttrE is unable to run. Specifically, AttrE is not applicable to datasets D_W_15K_V1 and D_W_15K_V2, while on D_Y_15K_V1 and D_Y_15K_V2, it exhibits a very low performance (e. g. , Hits@1 is 0. 0003 for D_Y_15K_V1) due to the low predicate similarity (e. g. , 0 in D_W_15K_V1 and 47. 37% in D_Y_15K_V1). In datasets like BBC_DB, where both the similarity of predicates (80. 93) and the similarity of literals (0.
87) are high, AttrE's performance is significantly improved. In datasets imdb-tmdb, imdb-tvdb and tmdb-tvdb, exhibiting the highest similarity of predicates and literals, AttrE outperforms all other methods. The correlation of the method with specific data characteristics (Pred_Sim and Lit_Sim) will be presented in Sect. 4. 3. Finally, AttrE is the only method that is able to run on the Restaurants datasets, where not all the entities of KG 1 are aligned with entities of KG 2 and vice versa. BERT_INT is an attribute-based method that exploits multiple sources of similarity evidence between entities (i. e. , long-text descriptions, literals, names, and relations). More precisely, BERT_INT, rather than aggregating the information of multi-hop neighbors that in some cases introduces noise (as in RDGCN), even by the usage of attention mechanisms (as in RREA), it employs interactions between the different views (i.
e. , description/name-view, neighbor-view and attribute-views), considering multi-hops and consuming very few negative samples compared to RDGCN. Regarding the factual information, it prioritizes textual descriptions over entity names, in order to leverage as much textual information as possible (textual descriptions are typically longer than entity names). BERT_INT has the advantage, compared to MultiKE, that it also exploits multiple resources but without prioritizing them, since the noise of irrelevant factual information is not incorporated in the embeddings. BERT_INT also has an advantage over AttrE and KDCoE, that they exclusively exploit either longer textual descriptions or other literals. Table  8  shows that BERT_INT outperforms all methods in all datasets and measures, except the two versions of D_W_15K.
For example, in D_Y_15K_V1, BERT_ INT achieves 99. 23 H@1 when RDGCN and RREA(semi) achieve 94. 10 and 81. 79. In D_Y_15K_V2, we observe similar behavior with BERT_INT exhibiting 99. 37 when RDGCN and RREA(semi) achieve 94. 15 and 97. 53. This happens, because in both versions of D_Y_15K the number of the entities that have textual descriptions, the descriptions similarity and the entity names are high (e. g. , in D_Y_15K_V1 3,455 entities have textual descriptions with 0. 88 similarity and 0. 99 similarity in names). The opposite behavior is observed in both versions of D_W_15K, where RREA outperforms BERT_INT with H@10 90. 03 instead of 48. 96 that BERT_INT achieves, since in these datasets the number of the entities that have descriptions is low (only 278 in D_W_15K_V2), with low similarity (0.
73) and low similarity in entity names (0. 78). In new datasets, the similarity of long-text descriptions and entity names is very low, the literal similarity is very high (0. 94 in tmdb-tvdb); as a result, the performance of BERT_INT is even better, outperforming even AttrE (e. g. , in tmdb-tvdb, BERT_INT finds all the matches correctly, with an impressive MRR 1, while the corresponding MRR of the second best method, AttrE, is only 0. 44).